2024-03-06T07:09:47.892 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:09:47.892 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063 --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --owner scheduled_leonidus@teuthology --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/orig.config.yaml 2024-03-06T07:09:47.934 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:09:48.017 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583063' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDw56Hh3KyspNyvaqv217HKBRutd3W/q/7CoyAjsp0tCJ6aA5Dm6IB+gQXPBBW+Zp/Gm+67VWFbbWwOFy5KGYOw= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNzfSidiz0rkDkJfLzZf7THIJ4NstMbuBEpwm7P8rMqb9+nhELtBXuADDX0j2ZoCV4097QnzClIx3awlnqAeAeQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:09:48.017 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:09:48.018 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:09:48.018 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:09:48.020 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:09:48.044 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:09:48.045 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:09:48.045 INFO:teuthology.packaging:ref: None 2024-03-06T07:09:48.045 INFO:teuthology.packaging:tag: None 2024-03-06T07:09:48.045 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:09:48.045 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:09:48.045 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:09:48.200 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:09:48.202 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:09:48.207 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:09:48.207 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:09:48.227 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:09:48.243 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:09:48.248 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:09:48.269 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.776244', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDw56Hh3KyspNyvaqv217HKBRutd3W/q/7CoyAjsp0tCJ6aA5Dm6IB+gQXPBBW+Zp/Gm+67VWFbbWwOFy5KGYOw='} 2024-03-06T07:09:48.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.777318', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNzfSidiz0rkDkJfLzZf7THIJ4NstMbuBEpwm7P8rMqb9+nhELtBXuADDX0j2ZoCV4097QnzClIx3awlnqAeAeQ='} 2024-03-06T07:09:48.293 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:09:48.299 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:09:48.299 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:09:48.299 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:09:48.375 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6007e1cf70>, signals=[15]) 2024-03-06T07:09:48.375 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:09:48.380 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:09:48.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:09:48.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:48.472 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:09:48.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:48.561 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:09:48.566 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-03-06T07:09:48.584 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-03-06T07:09:48.584 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:09:48.640 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:09:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:09:48.642 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-03-06T07:09:48.664 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-06T07:09:48.682 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-06T07:09:48.682 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-06T07:09:48.738 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:09:48.739 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:09:48.740 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-06T07:09:48.763 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:09:48.772 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:09:48.776 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:09:48.777 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:09:48.779 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:09:48.795 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:09:48.801 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:09:48.801 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:09:48.835 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:09:49.141 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:09:49.189 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:09:49.205 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-06T07:09:49.205 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-06T07:09:49.206 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:09:49.206 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:09:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:09:49.222 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-06T07:09:49.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:09:49.224 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-06T07:09:49.279 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:09:49.280 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-03-06T07:09:49.280 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-03-06T07:09:49.281 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:09:49.281 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-03-06T07:09:49.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-06T07:09:51.569 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-06T07:09:51.574 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-06T07:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-06T07:09:53.250 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-06T07:09:54.232 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-06T07:09:56.975 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-06T07:09:57.976 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-06T07:10:01.003 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-03-06T07:10:09.641 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 682 kB/s | 6.1 MB 00:09 2024-03-06T07:10:11.183 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-06T07:10:12.978 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 737 kB/s | 6.1 MB 00:08 2024-03-06T07:10:14.462 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-03-06T07:10:15.713 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-06T07:10:20.647 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-06T07:10:22.887 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:22.887 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:22.981 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-06T07:10:22.986 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-06T07:10:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-03-06T07:10:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:10:22.988 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 100 M 2024-03-06T07:10:22.988 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-03-06T07:10:22.988 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-06T07:10:31.390 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 736 kB/s | 6.0 MB 00:08 2024-03-06T07:10:34.806 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 892 kB/s | 20 MB 00:23 2024-03-06T07:10:39.773 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-03-06T07:10:42.030 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:42.030 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:42.123 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-06T07:10:42.134 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-03-06T07:10:42.135 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:10:42.136 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 100 M 2024-03-06T07:10:42.136 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-03-06T07:10:42.136 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-06T07:10:43.882 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 6.0 MB 00:01 2024-03-06T07:10:46.324 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 5.3 MB/s | 21 MB 00:03 2024-03-06T07:10:47.316 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 9.7 MB/s | 33 MB 00:03 2024-03-06T07:10:49.458 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.6 MB/s | 39 MB 00:07 2024-03-06T07:10:49.458 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:49.458 INFO:teuthology.orchestra.run.smithi169.stdout:Total 14 MB/s | 100 MB 00:07 2024-03-06T07:10:50.349 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-06T07:10:50.710 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-06T07:10:50.711 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-06T07:10:50.855 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 768 kB/s | 21 MB 00:27 2024-03-06T07:10:52.081 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-06T07:10:52.082 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-06T07:10:55.014 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-06T07:10:55.698 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:10:55.833 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:58.645 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:58.820 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:03.756 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:08.543 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:29.410 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 607 kB/s | 39 MB 01:06 2024-03-06T07:11:29.585 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 588 kB/s | 33 MB 00:58 2024-03-06T07:11:29.586 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:11:29.586 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.5 MB/s | 100 MB 01:06 2024-03-06T07:11:30.466 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-06T07:11:30.862 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-06T07:11:30.862 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-06T07:11:32.488 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-06T07:11:32.489 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-06T07:11:35.418 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-06T07:11:36.201 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:36.347 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:37.763 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:37.763 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:37.763 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:11:37.849 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:38.330 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:38.330 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:38.330 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:38.330 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:38.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:38.772 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:11:38.772 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-06T07:11:38.772 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:38.772 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:38.773 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:38.773 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:38.773 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:11:38.773 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-06T07:11:39.128 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:39.295 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:39.320 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:39.489 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:39.489 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:39.490 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:39.490 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:39.490 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:39.490 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-06T07:11:39.490 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:39.490 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:39.611 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:39.615 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:39.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-03-06T07:11:39.662 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-03-06T07:11:39.662 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-03-06T07:11:39.662 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-03-06T07:11:39.663 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:11:39.664 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:39.665 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:39.665 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:39.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:40.256 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-03-06T07:11:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:41.656 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-03-06T07:11:41.659 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-06T07:11:41.674 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.5qYKwO7q40 2024-03-06T07:11:41.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5qYKwO7q40 2024-03-06T07:11:41.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.5qYKwO7q40 2024-03-06T07:11:41.881 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.5qYKwO7q40 is 6KB 2024-03-06T07:11:41.929 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.5qYKwO7q40 2024-03-06T07:11:41.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:42.008 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:42.008 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:42.008 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:42.008 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:42.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:42.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-03-06T07:11:44.039 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:44.164 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:48.916 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:12.147 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-06T07:12:12.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-06T07:12:12.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:17.998 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:17.998 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:12:17.998 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:12:18.082 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:18.580 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:18.580 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:12:18.581 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:12:18.581 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:12:19.026 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-06T07:12:19.544 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:19.686 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:19.687 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:19.687 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:12:19.687 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:12:19.687 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:12:19.687 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-06T07:12:19.687 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:12:19.687 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:19.808 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:19.809 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:19.809 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-03-06T07:12:19.843 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-03-06T07:12:19.843 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-03-06T07:12:19.843 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-03-06T07:12:19.843 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-03-06T07:12:19.843 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-03-06T07:12:19.843 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:12:19.844 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:12:19.845 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:12:19.845 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:12:19.845 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-03-06T07:12:19.845 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:12:19.845 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:12:19.846 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:12:19.846 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:12:19.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:12:20.471 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-03-06T07:12:21.847 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:12:21.876 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-03-06T07:12:21.879 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-06T07:12:21.895 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.XPhCPdy6dr 2024-03-06T07:12:21.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XPhCPdy6dr 2024-03-06T07:12:21.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.XPhCPdy6dr 2024-03-06T07:12:22.108 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.XPhCPdy6dr is 6KB 2024-03-06T07:12:22.156 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.XPhCPdy6dr 2024-03-06T07:12:22.170 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:12:22.240 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:12:22.240 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:12:22.240 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:12:22.240 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:12:22.241 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:12:22.382 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-03-06T07:12:46.653 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-06T07:12:52.391 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-06T07:12:52.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-06T07:12:52.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:55.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-06T07:12:55.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:14.048 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-06T07:13:26.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-06T07:13:26.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:27.409 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-06T07:13:27.895 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-06T07:13:27.895 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:27.896 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-06T07:13:27.909 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:27.909 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:27.910 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:27.910 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:28.910 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-06T07:13:28.910 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:28.992 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-03-06T07:13:29.027 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:52.395 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:14:01.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-06T07:14:01.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:02.002 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-06T07:14:02.480 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-06T07:14:02.480 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:14:02.481 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-06T07:14:02.495 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:14:02.495 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:14:02.495 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:14:02.495 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:14:03.501 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-06T07:14:03.501 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:14:03.590 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-03-06T07:14:03.635 DEBUG:teuthology.parallel:result is None 2024-03-06T07:14:03.635 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:14:03.643 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:14:03.643 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:14:03.647 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:14:03.668 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:14:03.690 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:14:03.702 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:14:03.703 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:14:03.705 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:14:03.738 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:14:03.746 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:14:03.746 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:14:03.762 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:14:03.796 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:03.806 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:03.812 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:03.820 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:03.822 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:14:03.833 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:14:03.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:14:03.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:14:03.892 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:14:03.904 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:14:03.905 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:14:03.931 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:14:03.948 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:04.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:04.070 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:04.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:04.201 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:14:04.201 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:14:04.273 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:04.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:04.374 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:04.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:04.500 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:14:04.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:14:04.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-03-06T07:14:04.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-06T07:14:04.624 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:14:04.653 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:14:04.946 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:14:04.954 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:14:04.955 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:14:04.965 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:14:04.975 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-03-06T07:14:05.061 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-03-06T07:14:05.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-06T07:14:05.122 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-06T07:14:05.123 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:14:05.123 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-03-06T07:14:05.154 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-03-06T07:14:05.154 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-06T07:14:05.178 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-06T07:14:05.178 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi122.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:14:05.178 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:14:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:14:05.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:14:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:14:05.250 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:14:05.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:14:05.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:14:05.339 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:14:05.353 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:14:05.398 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:14:05.399 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi122.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:18:20.048 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi122.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-03-06T07:18:20.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-06T07:18:20.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:20.141 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-06T07:18:20.218 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-06T07:18:20.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-06T07:18:20.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:20.308 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-06T07:18:20.382 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-06T07:18:20.383 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:18:20.395 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:18:20.395 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:20.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:20.398 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:20.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:20.443 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:20.454 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:20.470 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:20.472 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-03-06T07:18:20.485 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-03-06T07:18:20.499 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-06T07:18:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:20.513 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-06T07:18:20.518 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:20.530 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:20.546 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:20.569 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:20.572 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:20.572 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-06T07:18:20.572 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.572 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.572 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.572 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.604 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:20.607 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:20.609 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-06T07:18:20.609 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.609 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.609 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.609 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:20.610 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:18:20.618 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:18:20.618 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:18:20.618 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:18:20.618 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:18:20.624 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:18:20.624 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:18:20.625 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:20.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:20.627 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:20.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:20.774 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:20.774 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:20.951 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:20.951 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:21.072 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:21.072 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:18:21.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:21.101 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:21.101 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:21.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:21.231 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:21.231 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:18:21.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:21.250 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:21.260 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:21.261 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:21.316 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-06T07:18:21.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-06T07:18:21.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:21.412 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:21.478 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-03-06T07:18:21.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-03-06T07:18:21.557 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-03-06T07:18:21.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:21.720 INFO:teuthology.orchestra.run.smithi122.stdout:74 files removed 2024-03-06T07:18:21.748 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:22.787 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 94 kB/s | 83 kB 00:00 2024-03-06T07:18:22.924 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 96 kB/s | 83 kB 00:00 2024-03-06T07:18:23.554 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-06T07:18:23.758 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-06T07:18:24.338 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-06T07:18:24.658 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-06T07:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-06T07:18:25.174 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-03-06T07:18:26.582 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-06T07:18:26.586 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-06T07:18:28.732 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-06T07:18:29.276 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-06T07:18:30.624 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-06T07:18:31.336 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-06T07:18:34.625 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-06T07:18:36.235 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2024-03-06T07:18:38.570 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-06T07:18:40.054 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-03-06T07:18:41.297 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-06T07:18:42.297 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 21 MB 00:05 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-06T07:18:47.327 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-06T07:18:48.727 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:48.727 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:48.740 INFO:teuthology.orchestra.run.smithi122.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:18:48.740 INFO:teuthology.orchestra.run.smithi122.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:18:48.741 INFO:teuthology.orchestra.run.smithi122.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:18:48.742 INFO:teuthology.orchestra.run.smithi122.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:48.744 INFO:teuthology.orchestra.run.smithi122.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:48.747 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:18:48.747 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:18:48.830 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================ 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================ 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:18:48.852 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:18:48.853 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:18:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:18:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:48.856 INFO:teuthology.orchestra.run.smithi122.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:18:48.857 INFO:teuthology.orchestra.run.smithi122.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:18:48.858 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:48.859 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:18:48.860 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:18:48.861 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:18:48.862 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:18:48.863 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:18:48.864 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:18:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:18:48.866 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================ 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout:Install 150 Packages 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 26 Packages 2024-03-06T07:18:48.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:18:48.868 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 224 M 2024-03-06T07:18:48.868 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-06T07:18:49.816 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:49.817 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:49.831 INFO:teuthology.orchestra.run.smithi169.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:18:49.832 INFO:teuthology.orchestra.run.smithi169.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:18:49.833 INFO:teuthology.orchestra.run.smithi169.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:18:49.834 INFO:teuthology.orchestra.run.smithi169.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:49.836 INFO:teuthology.orchestra.run.smithi169.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:49.839 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:18:49.840 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:18:49.920 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-06T07:18:49.942 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-06T07:18:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-06T07:18:49.942 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-06T07:18:49.942 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-06T07:18:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:18:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:18:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:18:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:18:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:18:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:18:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:18:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:18:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:18:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:18:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:18:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout:Install 150 Packages 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 26 Packages 2024-03-06T07:18:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:18:49.957 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 224 M 2024-03-06T07:18:49.957 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-06T07:18:50.065 INFO:teuthology.orchestra.run.smithi122.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-06T07:18:50.612 INFO:teuthology.orchestra.run.smithi169.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-06T07:18:51.332 INFO:teuthology.orchestra.run.smithi122.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 702 kB/s | 888 kB 00:01 2024-03-06T07:18:51.732 INFO:teuthology.orchestra.run.smithi122.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 363 kB/s | 145 kB 00:00 2024-03-06T07:18:52.179 INFO:teuthology.orchestra.run.smithi169.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 567 kB/s | 888 kB 00:01 2024-03-06T07:18:52.704 INFO:teuthology.orchestra.run.smithi169.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 276 kB/s | 145 kB 00:00 2024-03-06T07:18:53.924 INFO:teuthology.orchestra.run.smithi122.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.3 MB/s | 5.3 MB 00:04 2024-03-06T07:18:55.049 INFO:teuthology.orchestra.run.smithi122.stdout:(5/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 713 kB/s | 2.3 MB 00:03 2024-03-06T07:18:55.291 INFO:teuthology.orchestra.run.smithi122.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 1.1 MB/s | 1.5 MB 00:01 2024-03-06T07:18:57.111 INFO:teuthology.orchestra.run.smithi169.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 789 kB/s | 5.3 MB 00:06 2024-03-06T07:18:57.729 INFO:teuthology.orchestra.run.smithi169.stdout:(5/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 471 kB/s | 2.3 MB 00:05 2024-03-06T07:18:59.096 INFO:teuthology.orchestra.run.smithi169.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 771 kB/s | 1.5 MB 00:01 2024-03-06T07:19:01.341 INFO:teuthology.orchestra.run.smithi122.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 763 kB/s | 4.7 MB 00:06 2024-03-06T07:19:05.095 INFO:teuthology.orchestra.run.smithi169.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 652 kB/s | 4.7 MB 00:07 2024-03-06T07:19:07.020 INFO:teuthology.orchestra.run.smithi169.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 20 MB 00:16 2024-03-06T07:19:07.137 INFO:teuthology.orchestra.run.smithi169.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 216 kB/s | 25 kB 00:00 2024-03-06T07:19:13.732 INFO:teuthology.orchestra.run.smithi122.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 808 kB/s | 9.8 MB 00:12 2024-03-06T07:19:13.832 INFO:teuthology.orchestra.run.smithi122.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 252 kB/s | 25 kB 00:00 2024-03-06T07:19:15.508 INFO:teuthology.orchestra.run.smithi122.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 852 kB/s | 17 MB 00:20 2024-03-06T07:19:15.933 INFO:teuthology.orchestra.run.smithi122.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 519 kB/s | 220 kB 00:00 2024-03-06T07:19:16.033 INFO:teuthology.orchestra.run.smithi122.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 270 kB/s | 27 kB 00:00 2024-03-06T07:19:16.125 INFO:teuthology.orchestra.run.smithi122.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 342 kB/s | 31 kB 00:00 2024-03-06T07:19:16.642 INFO:teuthology.orchestra.run.smithi122.stdout:(14/176): ceph-common-19.0.0-1739.g1bc250c7.el9 776 kB/s | 20 MB 00:26 2024-03-06T07:19:16.825 INFO:teuthology.orchestra.run.smithi122.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 892 kB/s | 163 kB 00:00 2024-03-06T07:19:16.992 INFO:teuthology.orchestra.run.smithi122.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 760 kB/s | 127 kB 00:00 2024-03-06T07:19:17.070 INFO:teuthology.orchestra.run.smithi169.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 958 kB/s | 17 MB 00:17 2024-03-06T07:19:17.301 INFO:teuthology.orchestra.run.smithi122.stdout:(17/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 635 kB/s | 747 kB 00:01 2024-03-06T07:19:17.312 INFO:teuthology.orchestra.run.smithi169.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 913 kB/s | 220 kB 00:00 2024-03-06T07:19:17.476 INFO:teuthology.orchestra.run.smithi122.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 1.0 MB/s | 501 kB 00:00 2024-03-06T07:19:17.585 INFO:teuthology.orchestra.run.smithi122.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 417 kB/s | 45 kB 00:00 2024-03-06T07:19:17.687 INFO:teuthology.orchestra.run.smithi169.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 72 kB/s | 27 kB 00:00 2024-03-06T07:19:17.787 INFO:teuthology.orchestra.run.smithi169.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 313 kB/s | 31 kB 00:00 2024-03-06T07:19:17.827 INFO:teuthology.orchestra.run.smithi122.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 543 kB/s | 131 kB 00:00 2024-03-06T07:19:18.002 INFO:teuthology.orchestra.run.smithi122.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 924 kB/s | 162 kB 00:00 2024-03-06T07:19:18.336 INFO:teuthology.orchestra.run.smithi122.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 962 kB/s | 321 kB 00:00 2024-03-06T07:19:18.404 INFO:teuthology.orchestra.run.smithi169.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 747 kB 00:00 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi169.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 891 kB/s | 163 kB 00:00 2024-03-06T07:19:18.652 INFO:teuthology.orchestra.run.smithi122.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 941 kB/s | 298 kB 00:00 2024-03-06T07:19:18.755 INFO:teuthology.orchestra.run.smithi169.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 761 kB/s | 127 kB 00:00 2024-03-06T07:19:18.819 INFO:teuthology.orchestra.run.smithi122.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 595 kB/s | 99 kB 00:00 2024-03-06T07:19:18.920 INFO:teuthology.orchestra.run.smithi122.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 853 kB/s | 85 kB 00:00 2024-03-06T07:19:19.155 INFO:teuthology.orchestra.run.smithi169.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.2 MB/s | 501 kB 00:00 2024-03-06T07:19:21.955 INFO:teuthology.orchestra.run.smithi169.stdout:(18/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 593 kB/s | 9.8 MB 00:16 2024-03-06T07:19:22.173 INFO:teuthology.orchestra.run.smithi169.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 209 kB/s | 45 kB 00:00 2024-03-06T07:19:22.389 INFO:teuthology.orchestra.run.smithi169.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 606 kB/s | 131 kB 00:00 2024-03-06T07:19:22.681 INFO:teuthology.orchestra.run.smithi169.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 554 kB/s | 162 kB 00:00 2024-03-06T07:19:22.720 INFO:teuthology.orchestra.run.smithi122.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 833 kB/s | 3.1 MB 00:03 2024-03-06T07:19:22.945 INFO:teuthology.orchestra.run.smithi122.stdout:(27/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 929 kB/s | 5.1 MB 00:05 2024-03-06T07:19:23.012 INFO:teuthology.orchestra.run.smithi122.stdout:(28/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 585 kB/s | 171 kB 00:00 2024-03-06T07:19:23.029 INFO:teuthology.orchestra.run.smithi122.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 377 kB/s | 31 kB 00:00 2024-03-06T07:19:23.156 INFO:teuthology.orchestra.run.smithi169.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 675 kB/s | 321 kB 00:00 2024-03-06T07:19:23.179 INFO:teuthology.orchestra.run.smithi122.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 826 kB/s | 138 kB 00:00 2024-03-06T07:19:23.323 INFO:teuthology.orchestra.run.smithi169.stdout:(23/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.2 MB/s | 5.1 MB 00:04 2024-03-06T07:19:23.499 INFO:teuthology.orchestra.run.smithi169.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 567 kB/s | 99 kB 00:00 2024-03-06T07:19:23.566 INFO:teuthology.orchestra.run.smithi169.stdout:(25/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 729 kB/s | 298 kB 00:00 2024-03-06T07:19:23.599 INFO:teuthology.orchestra.run.smithi169.stdout:(26/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 850 kB/s | 85 kB 00:00 2024-03-06T07:19:23.791 INFO:teuthology.orchestra.run.smithi169.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 890 kB/s | 171 kB 00:00 2024-03-06T07:19:23.891 INFO:teuthology.orchestra.run.smithi169.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 314 kB/s | 31 kB 00:00 2024-03-06T07:19:24.058 INFO:teuthology.orchestra.run.smithi169.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 829 kB/s | 138 kB 00:00 2024-03-06T07:19:24.638 INFO:teuthology.orchestra.run.smithi122.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.1 MB/s | 1.8 MB 00:01 2024-03-06T07:19:24.972 INFO:teuthology.orchestra.run.smithi122.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 746 kB/s | 248 kB 00:00 2024-03-06T07:19:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 538 kB/s | 49 kB 00:00 2024-03-06T07:19:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 184 kB/s | 15 kB 00:00 2024-03-06T07:19:25.375 INFO:teuthology.orchestra.run.smithi169.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.4 MB/s | 1.8 MB 00:01 2024-03-06T07:19:25.506 INFO:teuthology.orchestra.run.smithi122.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 736 kB/s | 264 kB 00:00 2024-03-06T07:19:26.556 INFO:teuthology.orchestra.run.smithi122.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 712 kB/s | 748 kB 00:01 2024-03-06T07:19:26.648 INFO:teuthology.orchestra.run.smithi122.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 199 kB/s | 18 kB 00:00 2024-03-06T07:19:26.790 INFO:teuthology.orchestra.run.smithi122.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-06T07:19:26.840 INFO:teuthology.orchestra.run.smithi122.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-06T07:19:26.874 INFO:teuthology.orchestra.run.smithi122.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:19:26.908 INFO:teuthology.orchestra.run.smithi122.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 613 kB/s | 20 kB 00:00 2024-03-06T07:19:26.950 INFO:teuthology.orchestra.run.smithi122.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:26.983 INFO:teuthology.orchestra.run.smithi122.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-06T07:19:27.092 INFO:teuthology.orchestra.run.smithi122.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-06T07:19:27.126 INFO:teuthology.orchestra.run.smithi122.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 517 kB/s | 17 kB 00:00 2024-03-06T07:19:27.159 INFO:teuthology.orchestra.run.smithi122.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 304 kB/s | 10 kB 00:00 2024-03-06T07:19:27.201 INFO:teuthology.orchestra.run.smithi122.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-06T07:19:27.235 INFO:teuthology.orchestra.run.smithi122.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:27.285 INFO:teuthology.orchestra.run.smithi122.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-06T07:19:27.335 INFO:teuthology.orchestra.run.smithi122.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-06T07:19:27.369 INFO:teuthology.orchestra.run.smithi122.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 735 kB/s | 24 kB 00:00 2024-03-06T07:19:27.402 INFO:teuthology.orchestra.run.smithi122.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-06T07:19:27.444 INFO:teuthology.orchestra.run.smithi122.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:19:27.495 INFO:teuthology.orchestra.run.smithi122.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-06T07:19:27.529 INFO:teuthology.orchestra.run.smithi122.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-06T07:19:27.654 INFO:teuthology.orchestra.run.smithi122.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-06T07:19:27.754 INFO:teuthology.orchestra.run.smithi122.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 30 MB/s | 3.0 MB 00:00 2024-03-06T07:19:27.821 INFO:teuthology.orchestra.run.smithi122.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-06T07:19:27.850 INFO:teuthology.orchestra.run.smithi169.stdout:(31/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 739 kB/s | 3.1 MB 00:04 2024-03-06T07:19:27.871 INFO:teuthology.orchestra.run.smithi122.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-06T07:19:27.914 INFO:teuthology.orchestra.run.smithi122.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-06T07:19:28.131 INFO:teuthology.orchestra.run.smithi122.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-06T07:19:28.181 INFO:teuthology.orchestra.run.smithi122.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-06T07:19:28.223 INFO:teuthology.orchestra.run.smithi122.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-06T07:19:28.415 INFO:teuthology.orchestra.run.smithi122.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-06T07:19:28.482 INFO:teuthology.orchestra.run.smithi122.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 3.3 MB/s | 227 kB 00:00 2024-03-06T07:19:28.523 INFO:teuthology.orchestra.run.smithi122.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 586 kB/s | 24 kB 00:00 2024-03-06T07:19:28.542 INFO:teuthology.orchestra.run.smithi169.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 359 kB/s | 248 kB 00:00 2024-03-06T07:19:28.557 INFO:teuthology.orchestra.run.smithi122.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-06T07:19:28.599 INFO:teuthology.orchestra.run.smithi122.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-06T07:19:28.633 INFO:teuthology.orchestra.run.smithi122.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 949 kB/s | 32 kB 00:00 2024-03-06T07:19:28.659 INFO:teuthology.orchestra.run.smithi169.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 423 kB/s | 49 kB 00:00 2024-03-06T07:19:28.800 INFO:teuthology.orchestra.run.smithi122.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-06T07:19:28.809 INFO:teuthology.orchestra.run.smithi169.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 102 kB/s | 15 kB 00:00 2024-03-06T07:19:28.842 INFO:teuthology.orchestra.run.smithi122.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-06T07:19:28.875 INFO:teuthology.orchestra.run.smithi122.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-06T07:19:28.909 INFO:teuthology.orchestra.run.smithi122.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:28.943 INFO:teuthology.orchestra.run.smithi122.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:28.985 INFO:teuthology.orchestra.run.smithi122.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-06T07:19:29.035 INFO:teuthology.orchestra.run.smithi122.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-06T07:19:29.068 INFO:teuthology.orchestra.run.smithi122.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 6.2 MB/s | 211 kB 00:00 2024-03-06T07:19:29.102 INFO:teuthology.orchestra.run.smithi122.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-06T07:19:29.136 INFO:teuthology.orchestra.run.smithi122.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-06T07:19:29.386 INFO:teuthology.orchestra.run.smithi122.stdout:(80/176): ceph-mgr-diskprediction-local-19.0.0- 1.2 MB/s | 7.4 MB 00:06 2024-03-06T07:19:29.545 INFO:teuthology.orchestra.run.smithi122.stdout:(81/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-06T07:19:29.570 INFO:teuthology.orchestra.run.smithi122.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 213 kB/s | 39 kB 00:00 2024-03-06T07:19:29.595 INFO:teuthology.orchestra.run.smithi122.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 820 kB/s | 41 kB 00:00 2024-03-06T07:19:29.629 INFO:teuthology.orchestra.run.smithi122.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 215 kB 00:00 2024-03-06T07:19:29.655 INFO:teuthology.orchestra.run.smithi122.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 575 kB/s | 34 kB 00:00 2024-03-06T07:19:29.668 INFO:teuthology.orchestra.run.smithi169.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 307 kB/s | 264 kB 00:00 2024-03-06T07:19:29.680 INFO:teuthology.orchestra.run.smithi122.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 859 kB/s | 43 kB 00:00 2024-03-06T07:19:29.705 INFO:teuthology.orchestra.run.smithi122.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-06T07:19:29.730 INFO:teuthology.orchestra.run.smithi122.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-06T07:19:29.756 INFO:teuthology.orchestra.run.smithi122.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-06T07:19:29.781 INFO:teuthology.orchestra.run.smithi122.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-06T07:19:29.806 INFO:teuthology.orchestra.run.smithi122.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-03-06T07:19:29.831 INFO:teuthology.orchestra.run.smithi122.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:19:29.857 INFO:teuthology.orchestra.run.smithi122.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-06T07:19:29.890 INFO:teuthology.orchestra.run.smithi122.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:29.893 INFO:teuthology.orchestra.run.smithi169.stdout:(36/176): ceph-mgr-diskprediction-local-19.0.0- 1.6 MB/s | 7.4 MB 00:04 2024-03-06T07:19:29.916 INFO:teuthology.orchestra.run.smithi122.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-03-06T07:19:29.941 INFO:teuthology.orchestra.run.smithi122.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-06T07:19:29.967 INFO:teuthology.orchestra.run.smithi122.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-06T07:19:29.985 INFO:teuthology.orchestra.run.smithi169.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 199 kB/s | 18 kB 00:00 2024-03-06T07:19:30.000 INFO:teuthology.orchestra.run.smithi122.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-06T07:19:30.025 INFO:teuthology.orchestra.run.smithi122.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-06T07:19:30.051 INFO:teuthology.orchestra.run.smithi122.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-06T07:19:30.084 INFO:teuthology.orchestra.run.smithi122.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 1.9 MB/s | 113 kB 00:00 2024-03-06T07:19:30.110 INFO:teuthology.orchestra.run.smithi122.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.1 MB/s | 187 kB 00:00 2024-03-06T07:19:30.127 INFO:teuthology.orchestra.run.smithi169.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-06T07:19:30.187 INFO:teuthology.orchestra.run.smithi169.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-06T07:19:30.219 INFO:teuthology.orchestra.run.smithi169.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:19:30.235 INFO:teuthology.orchestra.run.smithi122.stdout:(103/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.6 MB/s | 245 kB 00:00 2024-03-06T07:19:30.252 INFO:teuthology.orchestra.run.smithi169.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 611 kB/s | 20 kB 00:00 2024-03-06T07:19:30.294 INFO:teuthology.orchestra.run.smithi169.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:30.320 INFO:teuthology.orchestra.run.smithi169.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-06T07:19:30.412 INFO:teuthology.orchestra.run.smithi169.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-06T07:19:30.437 INFO:teuthology.orchestra.run.smithi169.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 684 kB/s | 17 kB 00:00 2024-03-06T07:19:30.462 INFO:teuthology.orchestra.run.smithi169.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 404 kB/s | 10 kB 00:00 2024-03-06T07:19:30.496 INFO:teuthology.orchestra.run.smithi169.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-06T07:19:30.521 INFO:teuthology.orchestra.run.smithi169.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-06T07:19:30.572 INFO:teuthology.orchestra.run.smithi169.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-06T07:19:30.614 INFO:teuthology.orchestra.run.smithi169.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-06T07:19:30.639 INFO:teuthology.orchestra.run.smithi169.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 971 kB/s | 24 kB 00:00 2024-03-06T07:19:30.665 INFO:teuthology.orchestra.run.smithi169.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-06T07:19:30.694 INFO:teuthology.orchestra.run.smithi122.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-03-06T07:19:30.698 INFO:teuthology.orchestra.run.smithi169.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-06T07:19:30.732 INFO:teuthology.orchestra.run.smithi169.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-06T07:19:30.757 INFO:teuthology.orchestra.run.smithi169.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-06T07:19:30.891 INFO:teuthology.orchestra.run.smithi169.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-03-06T07:19:30.999 INFO:teuthology.orchestra.run.smithi169.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 28 MB/s | 3.0 MB 00:00 2024-03-06T07:19:31.058 INFO:teuthology.orchestra.run.smithi169.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 26 MB/s | 1.5 MB 00:00 2024-03-06T07:19:31.108 INFO:teuthology.orchestra.run.smithi169.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-06T07:19:31.150 INFO:teuthology.orchestra.run.smithi169.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-06T07:19:31.300 INFO:teuthology.orchestra.run.smithi169.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-06T07:19:31.326 INFO:teuthology.orchestra.run.smithi169.stdout:(62/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 451 kB/s | 748 kB 00:01 2024-03-06T07:19:31.360 INFO:teuthology.orchestra.run.smithi169.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-06T07:19:31.385 INFO:teuthology.orchestra.run.smithi169.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-06T07:19:31.535 INFO:teuthology.orchestra.run.smithi169.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-06T07:19:31.543 INFO:teuthology.orchestra.run.smithi122.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 645 kB/s | 548 kB 00:00 2024-03-06T07:19:31.560 INFO:teuthology.orchestra.run.smithi169.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 227 kB 00:00 2024-03-06T07:19:31.586 INFO:teuthology.orchestra.run.smithi169.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 489 kB/s | 24 kB 00:00 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi169.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 358 kB/s | 18 kB 00:00 2024-03-06T07:19:31.636 INFO:teuthology.orchestra.run.smithi169.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-06T07:19:31.661 INFO:teuthology.orchestra.run.smithi169.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-06T07:19:31.702 INFO:teuthology.orchestra.run.smithi122.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.9 MB/s | 309 kB 00:00 2024-03-06T07:19:31.778 INFO:teuthology.orchestra.run.smithi169.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-06T07:19:31.804 INFO:teuthology.orchestra.run.smithi169.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-06T07:19:31.811 INFO:teuthology.orchestra.run.smithi122.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-06T07:19:31.829 INFO:teuthology.orchestra.run.smithi169.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-06T07:19:31.855 INFO:teuthology.orchestra.run.smithi169.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-06T07:19:31.871 INFO:teuthology.orchestra.run.smithi169.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 983 kB/s | 41 kB 00:00 2024-03-06T07:19:31.897 INFO:teuthology.orchestra.run.smithi169.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-06T07:19:31.922 INFO:teuthology.orchestra.run.smithi169.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-06T07:19:31.936 INFO:teuthology.orchestra.run.smithi122.stdout:(108/176): protobuf-compiler-3.14.0-13.el9.x86_ 507 kB/s | 863 kB 00:01 2024-03-06T07:19:31.947 INFO:teuthology.orchestra.run.smithi169.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:19:31.964 INFO:teuthology.orchestra.run.smithi169.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-06T07:19:31.981 INFO:teuthology.orchestra.run.smithi169.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-06T07:19:32.132 INFO:teuthology.orchestra.run.smithi169.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 258 kB/s | 39 kB 00:00 2024-03-06T07:19:32.286 INFO:teuthology.orchestra.run.smithi122.stdout:(109/176): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-03-06T07:19:32.357 INFO:teuthology.orchestra.run.smithi169.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-06T07:19:32.370 INFO:teuthology.orchestra.run.smithi122.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 60 kB/s | 26 kB 00:00 2024-03-06T07:19:32.374 INFO:teuthology.orchestra.run.smithi169.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 170 kB/s | 41 kB 00:00 2024-03-06T07:19:32.396 INFO:teuthology.orchestra.run.smithi122.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-06T07:19:32.399 INFO:teuthology.orchestra.run.smithi169.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.0 MB/s | 215 kB 00:00 2024-03-06T07:19:32.419 INFO:teuthology.orchestra.run.smithi169.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 813 kB/s | 34 kB 00:00 2024-03-06T07:19:32.435 INFO:teuthology.orchestra.run.smithi169.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:19:32.459 INFO:teuthology.orchestra.run.smithi169.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-06T07:19:32.476 INFO:teuthology.orchestra.run.smithi169.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-06T07:19:32.501 INFO:teuthology.orchestra.run.smithi169.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-06T07:19:32.504 INFO:teuthology.orchestra.run.smithi122.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-06T07:19:32.518 INFO:teuthology.orchestra.run.smithi169.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-06T07:19:32.535 INFO:teuthology.orchestra.run.smithi169.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:32.552 INFO:teuthology.orchestra.run.smithi169.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-06T07:19:32.577 INFO:teuthology.orchestra.run.smithi169.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-03-06T07:19:32.604 INFO:teuthology.orchestra.run.smithi122.stdout:(113/176): libunwind-1.6.2-1.el9.x86_64.rpm 288 kB/s | 67 kB 00:00 2024-03-06T07:19:32.611 INFO:teuthology.orchestra.run.smithi169.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:32.636 INFO:teuthology.orchestra.run.smithi169.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-03-06T07:19:32.647 INFO:teuthology.orchestra.run.smithi122.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.7 MB/s | 838 kB 00:00 2024-03-06T07:19:32.661 INFO:teuthology.orchestra.run.smithi169.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-06T07:19:32.687 INFO:teuthology.orchestra.run.smithi169.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-06T07:19:32.720 INFO:teuthology.orchestra.run.smithi169.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-06T07:19:32.737 INFO:teuthology.orchestra.run.smithi169.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-06T07:19:32.755 INFO:teuthology.orchestra.run.smithi122.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 402 kB/s | 43 kB 00:00 2024-03-06T07:19:32.762 INFO:teuthology.orchestra.run.smithi169.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.9 MB/s | 124 kB 00:00 2024-03-06T07:19:32.788 INFO:teuthology.orchestra.run.smithi169.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:19:32.805 INFO:teuthology.orchestra.run.smithi169.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 4.4 MB/s | 187 kB 00:00 2024-03-06T07:19:32.880 INFO:teuthology.orchestra.run.smithi122.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-03-06T07:19:32.914 INFO:teuthology.orchestra.run.smithi169.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-06T07:19:32.964 INFO:teuthology.orchestra.run.smithi169.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-03-06T07:19:32.989 INFO:teuthology.orchestra.run.smithi122.stdout:(117/176): python3-asyncssh-2.13.2-4.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-03-06T07:19:33.006 INFO:teuthology.orchestra.run.smithi122.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 119 kB/s | 15 kB 00:00 2024-03-06T07:19:33.098 INFO:teuthology.orchestra.run.smithi122.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-06T07:19:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-06T07:19:33.197 INFO:teuthology.orchestra.run.smithi169.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-03-06T07:19:33.232 INFO:teuthology.orchestra.run.smithi122.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-06T07:19:33.248 INFO:teuthology.orchestra.run.smithi169.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 6.0 MB/s | 309 kB 00:00 2024-03-06T07:19:33.273 INFO:teuthology.orchestra.run.smithi169.stdout:(107/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-03-06T07:19:33.274 INFO:teuthology.orchestra.run.smithi122.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 282 kB/s | 35 kB 00:00 2024-03-06T07:19:33.341 INFO:teuthology.orchestra.run.smithi122.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-06T07:19:33.357 INFO:teuthology.orchestra.run.smithi169.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-06T07:19:33.458 INFO:teuthology.orchestra.run.smithi122.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-06T07:19:33.604 INFO:teuthology.orchestra.run.smithi122.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 73 kB/s | 11 kB 00:00 2024-03-06T07:19:33.658 INFO:teuthology.orchestra.run.smithi122.stdout:(126/176): python3-grpcio-1.46.7-10.el9.x86_64. 5.3 MB/s | 2.0 MB 00:00 2024-03-06T07:19:33.709 INFO:teuthology.orchestra.run.smithi122.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 169 kB/s | 18 kB 00:00 2024-03-06T07:19:33.767 INFO:teuthology.orchestra.run.smithi122.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-06T07:19:33.818 INFO:teuthology.orchestra.run.smithi122.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-06T07:19:33.870 INFO:teuthology.orchestra.run.smithi169.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 51 kB/s | 26 kB 00:00 2024-03-06T07:19:33.876 INFO:teuthology.orchestra.run.smithi122.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-06T07:19:33.941 INFO:teuthology.orchestra.run.smithi169.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 713 kB/s | 49 kB 00:00 2024-03-06T07:19:33.985 INFO:teuthology.orchestra.run.smithi122.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-03-06T07:19:34.010 INFO:teuthology.orchestra.run.smithi122.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 346 kB/s | 46 kB 00:00 2024-03-06T07:19:34.094 INFO:teuthology.orchestra.run.smithi122.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2024-03-06T07:19:34.127 INFO:teuthology.orchestra.run.smithi122.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 494 kB/s | 58 kB 00:00 2024-03-06T07:19:34.211 INFO:teuthology.orchestra.run.smithi122.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-06T07:19:34.236 INFO:teuthology.orchestra.run.smithi122.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-06T07:19:34.320 INFO:teuthology.orchestra.run.smithi122.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 828 kB/s | 90 kB 00:00 2024-03-06T07:19:34.353 INFO:teuthology.orchestra.run.smithi122.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-06T07:19:34.429 INFO:teuthology.orchestra.run.smithi122.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-06T07:19:34.487 INFO:teuthology.orchestra.run.smithi122.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-03-06T07:19:34.554 INFO:teuthology.orchestra.run.smithi122.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-06T07:19:34.605 INFO:teuthology.orchestra.run.smithi122.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-03-06T07:19:34.663 INFO:teuthology.orchestra.run.smithi122.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-03-06T07:19:34.730 INFO:teuthology.orchestra.run.smithi122.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-06T07:19:34.781 INFO:teuthology.orchestra.run.smithi122.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 767 kB/s | 90 kB 00:00 2024-03-06T07:19:34.807 INFO:teuthology.orchestra.run.smithi169.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 78 kB/s | 67 kB 00:00 2024-03-06T07:19:34.856 INFO:teuthology.orchestra.run.smithi122.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-06T07:19:34.874 INFO:teuthology.orchestra.run.smithi169.stdout:(112/176): libarrow-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 4.4 MB 00:01 2024-03-06T07:19:34.889 INFO:teuthology.orchestra.run.smithi122.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-06T07:19:34.899 INFO:teuthology.orchestra.run.smithi169.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-06T07:19:34.983 INFO:teuthology.orchestra.run.smithi169.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-03-06T07:19:34.990 INFO:teuthology.orchestra.run.smithi122.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-06T07:19:35.073 INFO:teuthology.orchestra.run.smithi122.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-06T07:19:35.967 INFO:teuthology.orchestra.run.smithi169.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 513 kB/s | 548 kB 00:01 2024-03-06T07:19:35.992 INFO:teuthology.orchestra.run.smithi169.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 43 kB/s | 43 kB 00:01 2024-03-06T07:19:36.017 INFO:teuthology.orchestra.run.smithi169.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-06T07:19:36.043 INFO:teuthology.orchestra.run.smithi169.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-06T07:19:36.285 INFO:teuthology.orchestra.run.smithi169.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 643 kB/s | 172 kB 00:00 2024-03-06T07:19:36.818 INFO:teuthology.orchestra.run.smithi169.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 462 kB/s | 358 kB 00:00 2024-03-06T07:19:37.773 INFO:teuthology.orchestra.run.smithi169.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 204 kB/s | 303 kB 00:01 2024-03-06T07:19:37.835 INFO:teuthology.orchestra.run.smithi169.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 35 kB/s | 35 kB 00:01 2024-03-06T07:19:38.077 INFO:teuthology.orchestra.run.smithi169.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 666 kB/s | 203 kB 00:00 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi122.stdout:(150/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 865 kB/s | 3.1 MB 00:03 2024-03-06T07:19:38.832 INFO:teuthology.orchestra.run.smithi122.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 68 kB 00:00 2024-03-06T07:19:38.932 INFO:teuthology.orchestra.run.smithi122.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-03-06T07:19:39.091 INFO:teuthology.orchestra.run.smithi122.stdout:(153/176): librados2-19.0.0-1739.g1bc250c7.el9. 847 kB/s | 3.4 MB 00:04 2024-03-06T07:19:39.141 INFO:teuthology.orchestra.run.smithi122.stdout:(154/176): NetworkManager-libnm-1.46.0-1.el9.x8 8.7 MB/s | 1.8 MB 00:00 2024-03-06T07:19:39.166 INFO:teuthology.orchestra.run.smithi122.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 557 kB/s | 41 kB 00:00 2024-03-06T07:19:39.200 INFO:teuthology.orchestra.run.smithi122.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.1 MB/s | 247 kB 00:00 2024-03-06T07:19:39.225 INFO:teuthology.orchestra.run.smithi122.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2024-03-06T07:19:39.242 INFO:teuthology.orchestra.run.smithi122.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 853 kB/s | 35 kB 00:00 2024-03-06T07:19:39.267 INFO:teuthology.orchestra.run.smithi122.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:19:39.301 INFO:teuthology.orchestra.run.smithi122.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 254 kB 00:00 2024-03-06T07:19:39.343 INFO:teuthology.orchestra.run.smithi122.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 10 MB/s | 802 kB 00:00 2024-03-06T07:19:39.377 INFO:teuthology.orchestra.run.smithi122.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 5.7 MB/s | 433 kB 00:00 2024-03-06T07:19:39.402 INFO:teuthology.orchestra.run.smithi122.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 381 kB/s | 22 kB 00:00 2024-03-06T07:19:39.427 INFO:teuthology.orchestra.run.smithi122.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-06T07:19:39.469 INFO:teuthology.orchestra.run.smithi122.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.2 MB/s | 154 kB 00:00 2024-03-06T07:19:39.494 INFO:teuthology.orchestra.run.smithi122.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 135 kB 00:00 2024-03-06T07:19:39.536 INFO:teuthology.orchestra.run.smithi122.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 942 kB/s | 63 kB 00:00 2024-03-06T07:19:39.570 INFO:teuthology.orchestra.run.smithi122.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 367 kB/s | 28 kB 00:00 2024-03-06T07:19:39.629 INFO:teuthology.orchestra.run.smithi122.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-06T07:19:39.704 INFO:teuthology.orchestra.run.smithi122.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 16 MB/s | 2.2 MB 00:00 2024-03-06T07:19:39.754 INFO:teuthology.orchestra.run.smithi122.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-06T07:19:39.780 INFO:teuthology.orchestra.run.smithi122.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.9 MB/s | 438 kB 00:00 2024-03-06T07:19:39.802 INFO:teuthology.orchestra.run.smithi169.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 1.0 MB/s | 2.0 MB 00:01 2024-03-06T07:19:39.805 INFO:teuthology.orchestra.run.smithi122.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 607 kB/s | 30 kB 00:00 2024-03-06T07:19:39.827 INFO:teuthology.orchestra.run.smithi169.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 83 kB/s | 144 kB 00:01 2024-03-06T07:19:39.839 INFO:teuthology.orchestra.run.smithi122.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-06T07:19:39.853 INFO:teuthology.orchestra.run.smithi169.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-03-06T07:19:39.872 INFO:teuthology.orchestra.run.smithi122.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-06T07:19:39.878 INFO:teuthology.orchestra.run.smithi169.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-06T07:19:39.903 INFO:teuthology.orchestra.run.smithi169.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-06T07:19:39.929 INFO:teuthology.orchestra.run.smithi169.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-06T07:19:39.954 INFO:teuthology.orchestra.run.smithi169.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-03-06T07:19:40.346 INFO:teuthology.orchestra.run.smithi169.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-03-06T07:19:40.371 INFO:teuthology.orchestra.run.smithi169.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 111 kB/s | 46 kB 00:00 2024-03-06T07:19:40.413 INFO:teuthology.orchestra.run.smithi169.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-06T07:19:40.439 INFO:teuthology.orchestra.run.smithi169.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 863 kB/s | 58 kB 00:00 2024-03-06T07:19:40.789 INFO:teuthology.orchestra.run.smithi122.stdout:(176/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.8 MB/s | 47 MB 00:26 2024-03-06T07:19:40.794 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:40.794 INFO:teuthology.orchestra.run.smithi122.stdout:Total 4.3 MB/s | 224 MB 00:51 2024-03-06T07:19:40.841 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-06T07:19:41.153 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-06T07:19:41.154 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-06T07:19:41.447 INFO:teuthology.orchestra.run.smithi169.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 263 kB/s | 272 kB 00:01 2024-03-06T07:19:41.498 INFO:teuthology.orchestra.run.smithi169.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 15 kB/s | 16 kB 00:01 2024-03-06T07:19:41.948 INFO:teuthology.orchestra.run.smithi169.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 180 kB/s | 90 kB 00:00 2024-03-06T07:19:42.015 INFO:teuthology.orchestra.run.smithi169.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 55 kB/s | 28 kB 00:00 2024-03-06T07:19:42.107 INFO:teuthology.orchestra.run.smithi169.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 194 kB/s | 31 kB 00:00 2024-03-06T07:19:43.126 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-06T07:19:43.126 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-06T07:19:43.448 INFO:teuthology.orchestra.run.smithi169.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 132 kB/s | 188 kB 00:01 2024-03-06T07:19:43.607 INFO:teuthology.orchestra.run.smithi169.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 39 kB/s | 59 kB 00:01 2024-03-06T07:19:43.774 INFO:teuthology.orchestra.run.smithi169.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 110 kB/s | 36 kB 00:00 2024-03-06T07:19:43.933 INFO:teuthology.orchestra.run.smithi169.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 157 kB/s | 51 kB 00:00 2024-03-06T07:19:44.968 INFO:teuthology.orchestra.run.smithi169.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 193 kB/s | 230 kB 00:01 2024-03-06T07:19:45.417 INFO:teuthology.orchestra.run.smithi169.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 60 kB/s | 90 kB 00:01 2024-03-06T07:19:45.908 INFO:teuthology.orchestra.run.smithi169.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 453 kB/s | 427 kB 00:00 2024-03-06T07:19:45.933 INFO:teuthology.orchestra.run.smithi169.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 39 kB/s | 20 kB 00:00 2024-03-06T07:19:46.084 INFO:teuthology.orchestra.run.smithi169.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-06T07:19:48.767 INFO:teuthology.orchestra.run.smithi169.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 572 kB/s | 1.6 MB 00:02 2024-03-06T07:19:50.034 INFO:teuthology.orchestra.run.smithi169.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 879 kB/s | 3.4 MB 00:03 2024-03-06T07:19:50.067 INFO:teuthology.orchestra.run.smithi169.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:19:50.143 INFO:teuthology.orchestra.run.smithi169.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 30 MB/s | 2.3 MB 00:00 2024-03-06T07:19:50.210 INFO:teuthology.orchestra.run.smithi169.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-06T07:19:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-06T07:19:50.268 INFO:teuthology.orchestra.run.smithi169.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 710 kB/s | 41 kB 00:00 2024-03-06T07:19:50.314 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:19:50.318 INFO:teuthology.orchestra.run.smithi169.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-06T07:19:50.349 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:19:50.352 INFO:teuthology.orchestra.run.smithi169.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:50.377 INFO:teuthology.orchestra.run.smithi169.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:50.411 INFO:teuthology.orchestra.run.smithi169.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 5.6 MB/s | 191 kB 00:00 2024-03-06T07:19:50.414 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:50.453 INFO:teuthology.orchestra.run.smithi169.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.9 MB/s | 254 kB 00:00 2024-03-06T07:19:50.465 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:19:50.507 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:19:50.520 INFO:teuthology.orchestra.run.smithi169.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 12 MB/s | 802 kB 00:00 2024-03-06T07:19:50.562 INFO:teuthology.orchestra.run.smithi169.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:19:50.566 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:19:50.596 INFO:teuthology.orchestra.run.smithi169.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 22 kB 00:00 2024-03-06T07:19:50.607 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:19:50.630 INFO:teuthology.orchestra.run.smithi169.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 107 kB 00:00 2024-03-06T07:19:50.642 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:19:50.707 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:19:50.913 INFO:teuthology.orchestra.run.smithi169.stdout:(164/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 47 MB 00:43 2024-03-06T07:19:50.939 INFO:teuthology.orchestra.run.smithi169.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 498 kB/s | 154 kB 00:00 2024-03-06T07:19:50.964 INFO:teuthology.orchestra.run.smithi169.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-06T07:19:50.989 INFO:teuthology.orchestra.run.smithi169.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-03-06T07:19:51.014 INFO:teuthology.orchestra.run.smithi169.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 551 kB/s | 28 kB 00:00 2024-03-06T07:19:51.028 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:19:51.048 INFO:teuthology.orchestra.run.smithi169.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-03-06T07:19:51.084 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:51.115 INFO:teuthology.orchestra.run.smithi169.stdout:(170/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-03-06T07:19:51.165 INFO:teuthology.orchestra.run.smithi169.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-06T07:19:51.206 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:51.207 INFO:teuthology.orchestra.run.smithi169.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-06T07:19:51.233 INFO:teuthology.orchestra.run.smithi169.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 3.7 MB/s | 438 kB 00:00 2024-03-06T07:19:51.235 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:51.258 INFO:teuthology.orchestra.run.smithi169.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 333 kB/s | 30 kB 00:00 2024-03-06T07:19:51.283 INFO:teuthology.orchestra.run.smithi169.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:19:51.317 INFO:teuthology.orchestra.run.smithi169.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-06T07:19:51.322 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:51.322 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.6 MB/s | 224 MB 01:01 2024-03-06T07:19:51.327 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:51.365 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:51.365 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-06T07:19:51.525 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:19:51.582 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:19:51.606 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:51.682 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-06T07:19:51.683 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-06T07:19:51.709 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:51.741 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:51.789 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:51.837 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:19:51.860 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:51.874 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:52.018 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:52.049 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:19:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:19:52.410 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:52.479 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:52.593 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:52.647 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:52.859 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:19:53.072 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:19:53.110 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:19:53.178 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:19:53.232 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:19:53.263 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:19:53.299 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:19:53.347 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:19:53.410 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:19:53.454 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:19:53.498 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:19:53.531 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:19:53.558 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:19:53.627 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:19:53.664 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:19:53.707 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:19:53.735 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:19:53.761 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-06T07:19:53.761 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-06T07:19:53.804 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:19:53.850 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:19:53.931 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:19:54.072 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:19:54.127 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:19:54.641 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:19:54.722 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:19:54.808 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:19:54.868 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:54.876 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:54.915 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:19:54.953 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:19:54.999 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:19:55.114 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:19:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:19:55.494 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:19:55.669 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:19:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:19:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:19:55.954 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:19:56.002 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:19:56.056 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:19:56.120 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:19:56.153 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:19:56.189 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:19:56.235 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:19:56.280 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:19:56.323 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:19:56.365 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:19:56.497 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:19:56.537 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:19:56.568 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:19:56.614 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:19:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:19:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:19:56.791 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:19:56.817 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:19:56.842 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:19:56.934 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:19:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:19:57.157 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:19:57.200 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:19:57.257 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:19:57.320 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:19:57.502 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:19:57.546 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:19:57.680 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:19:57.734 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:19:57.768 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:19:57.809 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:19:57.844 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:19:57.875 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:19:57.903 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:19:57.937 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:19:57.976 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:19:58.008 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:19:58.046 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:19:58.130 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:19:58.165 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:19:58.240 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:19:58.276 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:19:58.364 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:19:58.446 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:19:58.490 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:19:58.522 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:19:58.577 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:19:58.681 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:19:58.740 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:19:59.457 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:19:59.512 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:19:59.546 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:19:59.594 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:19:59.681 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:19:59.722 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:19:59.898 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:19:59.928 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:00.006 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:00.637 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:00.707 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-06T07:20:00.749 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:00.796 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:20:00.843 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:20:00.894 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:00.923 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:00.984 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:20:01.026 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:20:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:20:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:20:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:20:01.197 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:20:01.524 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:20:01.570 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:01.693 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:01.722 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:01.812 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:01.845 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:02.009 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:20:02.065 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:20:02.093 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:02.124 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:02.151 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:02.196 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:02.219 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:02.284 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:02.323 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:20:02.345 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:02.361 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:02.566 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:02.602 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:02.606 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:20:02.638 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:02.780 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:20:02.891 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:02.951 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:03.069 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:03.097 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:03.118 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:03.136 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:03.171 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:20:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:20:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:20:03.644 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:20:03.697 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:20:03.728 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:20:03.763 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:20:03.803 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:20:03.867 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:20:03.902 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:20:03.946 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:20:03.970 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:20:03.997 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:20:04.067 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:20:04.102 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:20:04.155 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:20:04.183 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:04.253 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:04.297 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:04.372 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:04.514 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:04.567 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:05.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:05.162 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:05.255 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:05.306 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:05.315 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:05.353 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:05.385 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:05.391 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:05.424 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:05.424 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:05.528 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:05.608 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:05.873 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:05.965 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:06.011 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:06.045 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:06.108 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:06.108 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:06.109 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:06.109 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:06.109 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:06.153 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:06.154 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:06.298 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:06.467 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:06.498 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:06.535 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:06.582 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:06.618 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:06.661 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:06.703 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:06.838 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:06.875 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:06.905 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:06.947 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:07.041 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:07.188 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:07.268 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:07.388 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:07.492 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:07.546 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:07.603 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:07.659 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:07.844 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:07.892 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:08.032 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:08.073 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:08.106 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:08.138 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:08.173 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:08.203 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:08.240 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:08.265 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:08.305 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:08.336 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:08.367 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:08.443 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:08.477 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:08.510 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:08.568 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:08.603 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:08.693 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:08.791 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:08.843 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:08.873 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:08.920 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:09.019 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:09.067 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:09.802 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:09.848 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:09.882 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:09.930 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:10.019 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:10.049 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:10.222 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:10.255 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:10.343 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:10.988 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:11.113 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:12.426 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:12.453 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:12.881 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:12.912 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:12.949 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:13.404 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:13.436 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:13.507 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:13.529 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:15.702 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:15.734 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:16.271 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:16.304 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:16.385 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:16.385 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:16.385 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:16.385 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:16.422 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-03-06T07:20:17.856 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:18.017 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:18.033 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:18.033 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:19.530 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:19.555 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:19.555 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:19.555 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:19.555 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:19.555 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:19.555 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:19.781 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:19.802 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:19.803 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:19.803 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:19.803 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:19.803 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:19.803 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:20.239 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:20.261 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:20.261 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:20.262 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:20.262 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:20.262 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:20.262 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:20.349 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:20.732 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:20.764 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:20.856 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:20.879 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:22.131 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:22.230 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:22.447 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.447 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:22.447 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:22.447 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:22.566 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:23.449 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:23.477 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:23.500 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:23.500 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.500 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:23.500 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:23.500 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:23.500 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:23.567 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.567 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:23.679 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.679 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:23.679 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:23.680 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:23.938 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:23.959 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:23.959 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.959 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:23.959 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:23.959 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:23.959 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:28.103 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-06T07:20:28.103 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-06T07:20:28.104 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-06T07:20:28.104 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-06T07:20:28.104 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-06T07:20:28.104 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-03-06T07:20:28.104 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-06T07:20:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:28.259 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:28.273 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:28.274 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:28.274 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:28.274 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:28.475 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:28.675 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:28.791 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:28.831 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:29.012 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:29.066 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:29.110 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:29.158 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:29.187 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:29.264 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:29.349 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:29.779 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.779 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:29.779 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:29.779 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:30.004 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:30.054 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:30.474 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:30.592 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:30.872 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:30.939 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:30.980 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:31.009 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:31.040 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:31.066 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:31.071 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:31.101 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:31.126 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:31.132 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:31.152 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:31.154 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:31.178 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:31.179 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:31.271 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:31.299 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:31.329 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:31.330 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:31.384 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:31.398 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:31.424 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:31.431 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:31.457 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:31.482 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:31.539 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:31.584 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:31.624 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:31.665 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:31.697 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:31.722 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:31.747 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:31.773 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:31.798 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:31.804 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:31.984 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:31.984 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:32.051 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:32.082 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:32.109 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:32.134 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:32.160 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:32.225 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:32.402 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:32.422 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:32.522 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:32.730 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:32.730 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:32.730 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:32.730 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:32.730 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:32.730 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:32.775 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:32.786 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:32.841 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:33.701 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:33.727 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:33.749 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:33.749 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:33.750 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:33.750 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:33.750 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:33.750 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:33.795 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:33.816 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:33.816 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:33.816 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:33.817 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:33.870 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:33.892 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:33.892 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:33.892 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:33.892 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:33.892 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:33.892 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:34.204 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:34.227 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:34.227 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:34.227 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:34.227 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:34.227 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:34.227 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:37.355 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:37.356 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:37.357 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:37.358 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:37.374 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:37.375 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:37.376 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:37.377 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:37.378 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:37.379 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:37.379 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:37.379 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:37.379 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:37.379 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:37.380 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:37.380 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:37.380 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:37.381 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:37.381 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:37.381 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:37.381 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:37.382 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:37.382 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:37.382 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:37.383 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:37.384 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:37.385 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:37.386 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:37.387 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:37.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:37.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:37.388 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:37.389 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:37.390 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:37.391 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:37.392 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:37.393 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:37.394 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:37.395 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:37.395 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:38.684 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:38.733 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:38.772 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:38.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:38.990 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:39.098 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:39.147 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:39.207 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:39.289 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:39.377 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:39.490 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:39.519 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:39.605 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:39.665 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:39.702 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:40.330 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:40.376 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:40.396 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:40.428 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:40.429 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:40.430 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:40.430 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:40.430 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-06T07:20:41.044 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:41.218 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:41.296 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:41.337 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:41.434 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:41.467 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:41.492 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:41.517 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:41.544 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:41.544 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:41.628 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:41.656 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:41.695 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:41.695 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:41.788 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:41.815 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:41.848 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:41.873 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:41.904 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:41.941 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:41.983 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:42.062 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:42.087 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:42.113 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:42.146 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:42.172 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:42.178 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:42.178 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:42.367 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:42.367 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:42.450 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:42.481 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:42.508 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:42.533 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:42.559 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:47.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:47.439 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:47.455 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:47.456 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:47.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:47.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:47.459 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:47.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:47.463 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:47.464 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:47.465 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:47.466 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:47.467 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.446 INFO:teuthology.orchestra.run.smithi169.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:50.447 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.448 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.449 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:50.450 INFO:teuthology.orchestra.run.smithi169.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:50.451 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:50.452 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.453 INFO:teuthology.orchestra.run.smithi169.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:50.454 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:50.455 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:50.456 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:50.457 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:50.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:50.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.460 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:50.461 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:50.462 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:50.462 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-06T07:20:51.100 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:51.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:51.322 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:51.349 INFO:teuthology.orchestra.run.smithi122.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:51.349 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:51.350 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:51.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:51.575 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:51.599 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:51.599 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:51.599 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:51.601 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:20:51.601 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:51.602 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:51.637 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:51.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:51.671 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:20:51.671 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:51.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:51.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:51.779 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:51.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:51.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:51.876 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:20:51.876 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:51.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:51.905 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:51.975 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:51.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:52.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:52.069 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:20:52.070 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:52.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:52.098 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:52.167 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:52.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:52.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:52.261 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:20:52.340 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:20:52.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:52.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:52.368 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:20:52.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:52.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:52.436 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:20:52.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:52.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:52.510 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:20:52.510 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:20:52.510 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:20:52.511 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:52.511 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:52.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:52.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:52.705 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:52.705 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:52.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:52.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:52.871 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:20:52.871 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:20:52.871 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583063', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDw56Hh3KyspNyvaqv217HKBRutd3W/q/7CoyAjsp0tCJ6aA5Dm6IB+gQXPBBW+Zp/Gm+67VWFbbWwOFy5KGYOw=', 'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNzfSidiz0rkDkJfLzZf7THIJ4NstMbuBEpwm7P8rMqb9+nhELtBXuADDX0j2ZoCV4097QnzClIx3awlnqAeAeQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:20:52.872 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:52.874 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:52.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:52.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:52.987 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:52.987 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:53.007 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:53.008 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:53.062 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:53.062 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-06 07:20:32.578947770 +0000 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-06 07:17:56.081935356 +0000 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-06 07:17:56.081935356 +0000 2024-03-06T07:20:53.063 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-06 07:17:56.081935356 +0000 2024-03-06T07:20:53.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:53.127 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-06T07:20:53.127 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-06T07:20:53.127 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00017428 s, 2.9 MB/s 2024-03-06T07:20:53.128 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:53.185 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-06 07:20:32.579947747 +0000 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-06 07:17:56.434927305 +0000 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-06 07:17:56.434927305 +0000 2024-03-06T07:20:53.239 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-06 07:17:56.434927305 +0000 2024-03-06T07:20:53.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:53.304 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-06T07:20:53.304 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-06T07:20:53.304 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000187695 s, 2.7 MB/s 2024-03-06T07:20:53.305 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:53.361 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-06 07:20:32.579947747 +0000 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-06 07:17:56.786919278 +0000 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-06 07:17:56.786919278 +0000 2024-03-06T07:20:53.416 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-06 07:17:56.786919278 +0000 2024-03-06T07:20:53.417 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:53.481 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-06T07:20:53.481 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-06T07:20:53.481 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000228226 s, 2.2 MB/s 2024-03-06T07:20:53.482 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:53.538 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:53.593 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-06 07:20:32.579947747 +0000 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-06 07:17:57.125911547 +0000 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-06 07:17:57.125911547 +0000 2024-03-06T07:20:53.594 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-06 07:17:57.125911547 +0000 2024-03-06T07:20:53.594 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-06T07:20:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-06T07:20:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000203951 s, 2.5 MB/s 2024-03-06T07:20:53.660 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:53.716 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:53.716 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:53.716 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:53.733 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:53.733 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:53.789 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:53.789 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-06 07:20:42.975970702 +0000 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-06 07:17:56.071972887 +0000 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-06 07:17:56.071972887 +0000 2024-03-06T07:20:53.790 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-06 07:17:56.071972887 +0000 2024-03-06T07:20:53.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:53.857 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-06T07:20:53.857 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-06T07:20:53.857 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.00021368 s, 2.4 MB/s 2024-03-06T07:20:53.858 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:53.915 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:53.971 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:53.971 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:53.971 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-06T07:20:53.971 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:53.972 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:53.972 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-06 07:20:42.976970680 +0000 2024-03-06T07:20:53.972 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-06 07:17:56.428965030 +0000 2024-03-06T07:20:53.972 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-06 07:17:56.428965030 +0000 2024-03-06T07:20:53.973 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-06 07:17:56.428965030 +0000 2024-03-06T07:20:53.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:54.038 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-06T07:20:54.038 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-06T07:20:54.038 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000187393 s, 2.7 MB/s 2024-03-06T07:20:54.039 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:54.096 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:54.151 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-06 07:20:42.976970680 +0000 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-06 07:17:56.780957282 +0000 2024-03-06T07:20:54.152 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-06 07:17:56.780957282 +0000 2024-03-06T07:20:54.153 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-06 07:17:56.780957282 +0000 2024-03-06T07:20:54.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:54.217 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-06T07:20:54.218 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-06T07:20:54.218 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000233397 s, 2.2 MB/s 2024-03-06T07:20:54.219 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:54.276 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:54.330 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-06 07:20:42.976970680 +0000 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-06 07:17:57.114949932 +0000 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-06 07:17:57.114949932 +0000 2024-03-06T07:20:54.331 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-06 07:17:57.114949932 +0000 2024-03-06T07:20:54.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-06T07:20:54.423 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-06T07:20:54.423 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.00019134 s, 2.7 MB/s 2024-03-06T07:20:54.425 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:54.481 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:54.481 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:54.481 INFO:tasks.ceph:Generating config... 2024-03-06T07:20:54.482 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:20:54.482 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:20:54.482 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:20:54.482 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:20:54.483 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:20:54.483 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:20:54.483 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:20:54.484 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:20:54.484 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:20:54.484 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:20:54.484 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:20:54.484 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:20:54.485 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:54.485 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:20:54.485 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:20:54.485 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:20:54.485 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:20:54.485 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:20:54.485 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:20:54.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:54.536 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:20:54.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:20:54.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:54.609 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.122'), ('mon.b', '172.21.15.169'), ('mon.c', '[v2:172.21.15.169:3301,v1:172.21.15.169:6790]')] 2024-03-06T07:20:54.609 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:20:54.610 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:54.610 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:54.664 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.122 --add b 172.21.15.169 --addv c '[v2:172.21.15.169:3301,v1:172.21.15.169:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: generated fsid 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:setting min_mon_release = quincy 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:epoch 0 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:fsid 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:last_changed 2024-03-06T07:20:54.747453+0000 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:created 2024-03-06T07:20:54.747453+0000 2024-03-06T07:20:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:min_mon_release 17 (quincy) 2024-03-06T07:20:54.749 INFO:teuthology.orchestra.run.smithi122.stdout:election_strategy: 1 2024-03-06T07:20:54.749 INFO:teuthology.orchestra.run.smithi122.stdout:0: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.a 2024-03-06T07:20:54.749 INFO:teuthology.orchestra.run.smithi122.stdout:1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.b 2024-03-06T07:20:54.749 INFO:teuthology.orchestra.run.smithi122.stdout:2: [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] mon.c 2024-03-06T07:20:54.749 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:20:54.750 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:54.764 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 12542aa0-5a71-4218-ae26-0a8da740ed3f... 2024-03-06T07:20:54.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:54.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:54.854 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:20:54.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.856 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.857 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.858 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:20:54.859 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:20:54.860 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:20:54.861 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:54.862 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:54.863 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:20:54.864 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:20:54.865 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:20:54.866 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:20:54.867 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:54.868 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:54.869 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:20:54.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:20:54.871 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:54.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:20:54.873 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:54.874 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:20:54.876 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:20:54.877 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:20:54.878 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:20:54.880 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:20:54.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:20:54.975 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:20:54.976 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:54.976 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:20:54.991 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:54.991 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:20:55.045 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:20:55.045 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:55.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:20:55.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:55.125 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:20:55.125 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:55.179 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:20:55.180 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:55.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:20:55.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:55.219 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:20:55.219 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:55.274 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:20:55.274 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:20:55.275 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:20:55.329 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:20:55.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:20:55.399 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:20:55.401 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:20:55.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:20:55.456 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:20:55.458 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:20:55.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:20:55.579 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:20:55.581 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:20:55.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:20:55.663 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:20:55.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:20:55.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:20:55.786 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:20:55.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:20:55.817 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:20:55.817 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:20:55.859 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:20:55.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:20:55.956 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:20:55.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:20:56.013 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:20:56.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:20:56.112 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:20:56.127 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:20:56.128 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:56.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:56.154 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:56.154 INFO:tasks.ceph:role: osd.0 2024-03-06T07:20:56.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:20:56.154 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:20:56.228 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:56.229 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:56.716 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-06T07:20:56.718 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-06T07:20:56.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:56.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:56.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:56.947 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:56.947 INFO:tasks.ceph:role: osd.1 2024-03-06T07:20:56.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:20:56.947 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:57.022 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:57.023 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:57.023 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:57.483 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-06T07:20:57.485 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-06T07:20:57.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:57.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:57.608 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:57.676 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:57.676 INFO:tasks.ceph:role: osd.2 2024-03-06T07:20:57.676 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:20:57.676 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:57.749 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:57.750 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:58.236 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-06T07:20:58.238 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-06T07:20:58.238 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:58.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:58.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:58.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:58.488 INFO:tasks.ceph:role: osd.3 2024-03-06T07:20:58.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:20:58.488 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:58.562 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:58.563 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:59.067 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-06T07:20:59.069 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-06T07:20:59.069 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:59.190 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:59.221 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:59.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:20:59.319+0000 7fdb19216640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:20:59.319+0000 7fdb19216640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:20:59.319+0000 7fdb19216640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:20:59.319+0000 7fdb19216640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:20:59.319+0000 7fdb19216640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:20:59.320+0000 7fdb19216640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:21:02.167 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:02.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:02.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:02.281+0000 7fe64649b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:21:02.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:02.281+0000 7fe64649b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:21:02.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:02.281+0000 7fe64649b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:02.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:02.281+0000 7fe64649b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:02.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:02.281+0000 7fe64649b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:02.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:02.282+0000 7fe64649b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:21:05.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:05.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:05.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:05.248+0000 7f351d585640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:21:05.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:05.248+0000 7f351d585640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:21:05.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:05.248+0000 7f351d585640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:05.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:05.248+0000 7f351d585640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:05.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:05.248+0000 7f351d585640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:05.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:05.249+0000 7f351d585640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:21:08.106 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:08.135 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:08.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:08.222+0000 7f4548268640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:21:08.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:08.222+0000 7f4548268640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:21:08.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:08.222+0000 7f4548268640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:08.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:08.222+0000 7f4548268640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:08.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:08.222+0000 7f4548268640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:08.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:08.223+0000 7f4548268640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:21:11.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:11.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:11.132 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:11.132 INFO:tasks.ceph:role: osd.4 2024-03-06T07:21:11.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:21:11.133 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:11.210 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:11.210 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:11.210 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:11.211 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:11.700 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-06T07:21:11.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-06T07:21:11.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:11.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:11.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:11.915 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:11.915 INFO:tasks.ceph:role: osd.5 2024-03-06T07:21:11.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:21:11.915 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:11.989 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:11.989 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:11.990 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:12.457 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-06T07:21:12.459 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-06T07:21:12.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:12.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:12.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:12.672 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:12.673 INFO:tasks.ceph:role: osd.6 2024-03-06T07:21:12.673 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:21:12.673 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:12.747 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:12.748 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:12.748 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:13.236 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-06T07:21:13.238 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-06T07:21:13.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:13.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:13.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:13.441 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:13.441 INFO:tasks.ceph:role: osd.7 2024-03-06T07:21:13.441 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:21:13.441 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:13.516 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:14.035 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-06T07:21:14.037 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-06T07:21:14.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:14.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:14.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:14.327+0000 7efe5532a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:21:14.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:14.332+0000 7efe5532a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:21:14.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:14.332+0000 7efe5532a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:14.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:14.332+0000 7efe5532a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:14.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:14.332+0000 7efe5532a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:14.341+0000 7efe5532a640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:21:17.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:17.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:17.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:17.313+0000 7f8db2cf9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:21:17.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:17.314+0000 7f8db2cf9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:21:17.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:17.314+0000 7f8db2cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:17.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:17.314+0000 7f8db2cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:17.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:17.314+0000 7f8db2cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:17.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:17.314+0000 7f8db2cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:21:20.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:20.203 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:20.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:20.289+0000 7f2f65c13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:21:20.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:20.289+0000 7f2f65c13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:21:20.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:20.289+0000 7f2f65c13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:20.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:20.289+0000 7f2f65c13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:20.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:20.289+0000 7f2f65c13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:20.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:20.290+0000 7f2f65c13640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:21:22.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:22.876 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:23.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:23.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:23.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:23.249+0000 7f7e32413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:21:23.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:23.249+0000 7f7e32413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:21:23.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:23.249+0000 7f7e32413640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:23.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:23.249+0000 7f7e32413640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:23.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:23.249+0000 7f7e32413640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:23.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:23.249+0000 7f7e32413640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:21:26.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:26.134 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:21:26.135 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.135 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:21:26.167 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.167 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:21:26.238 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:21:26.310 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.310 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:21:26.382 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.382 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:21:26.453 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:21:26.525 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:26.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:21:26.595 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:21:26.627 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:21:26.697 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:21:26.768 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:21:26.836 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:21:26.904 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:21:26.972 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:26.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:21:27.040 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:27.040 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:21:27.058 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:27.058 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:21:27.115 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:27.115 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:21:27.130 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:27.130 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:21:27.185 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:21:27.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:27.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:27.228 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.y] 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA== 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ== 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw== 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w== 2024-03-06T07:21:27.229 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ== 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A== 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg== 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ== 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg== 2024-03-06T07:21:27.230 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.4] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.5] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.6] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.7] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[client.2] 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ== 2024-03-06T07:21:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-03-06T07:21:27.232 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg== 2024-03-06T07:21:27.232 INFO:teuthology.orchestra.run.smithi122.stdout:[client.3] 2024-03-06T07:21:27.232 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w== 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.y] 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA== 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ== 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw== 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w== 2024-03-06T07:21:27.251 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ== 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A== 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.3] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg== 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ== 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg== 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw== 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.4] 2024-03-06T07:21:27.252 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.5] 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.6] 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.7] 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[client.2] 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg== 2024-03-06T07:21:27.253 INFO:teuthology.orchestra.run.smithi169.stdout:[client.3] 2024-03-06T07:21:27.254 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w== 2024-03-06T07:21:27.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:27.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:27.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.403 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:27.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:27.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.787 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:27.843 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:27.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:28.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:28.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:28.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.174 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:28.288 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:21:28.288 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:28.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:28.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:28.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:28.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:28.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:28.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:28.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:28.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:28.781 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:28.783 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:28.838 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:21:28.839 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:21:28.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:21:28.842 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:21:28.842 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:21:28.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:21:28.881 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:21:28.881 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:21:28.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:21:28.883 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:21:28.883 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:21:28.883 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:21:28.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:21:28.886 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:21:28.886 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:21:28.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:21:28.888 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:21:28.888 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:21:28.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:21:28.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.987+0000 7f963a916640 1 Processor -- start 2024-03-06T07:21:28.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.987+0000 7f963a916640 1 -- start start 2024-03-06T07:21:28.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f963a916640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 0x7f96340cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f963a916640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f963a916640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96340ceb30 con 0x7f96340c72a0 2024-03-06T07:21:28.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f963a916640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96340c6ff0 con 0x7f963406b6c0 2024-03-06T07:21:28.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f963a916640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96340ce570 con 0x7f9634161980 2024-03-06T07:21:28.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52240/0 (socket says 172.21.15.122:52240) 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 -- 172.21.15.122:0/4117649019 learned_addr learned my addr 172.21.15.122:0/4117649019 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 -- 172.21.15.122:0/4117649019 >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 msgr2=0x7f963406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 -- 172.21.15.122:0/4117649019 >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 msgr2=0x7f963406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.988+0000 7f9638eb3640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.989+0000 7f9633fff640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 0x7f96340cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.989+0000 7f9633fff640 1 -- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 msgr2=0x7f96340cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.989+0000 7f9633fff640 1 -- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 msgr2=0x7f96340cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.989+0000 7f9633fff640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 0x7f96340cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:28.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:28.989+0000 7f9633fff640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 0x7f96340cf1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.188+0000 7f9632ffd640 1 -- 172.21.15.122:0/4117649019 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349022484 0 0) 0x7f9628001110 con 0x7f96340c72a0 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.188+0000 7f9632ffd640 1 -- 172.21.15.122:0/4117649019 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c003520 con 0x7f96340c72a0 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.189+0000 7f9638eb3640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.122:3300/0 conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.189+0000 7f9638eb3640 1 -- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 msgr2=0x7f96340cf1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.189+0000 7f9638eb3640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 0x7f96340cf1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.189+0000 7f9638eb3640 1 -- 172.21.15.122:0/4117649019 >> v1:172.21.15.169:6789/0 conn(0x7f96340c72a0 legacy=0x7f963406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.189+0000 7f9638eb3640 1 -- 172.21.15.122:0/4117649019 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340ce850 con 0x7f963406b6c0 2024-03-06T07:21:29.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.190+0000 7f9638eb3640 1 --2- 172.21.15.122:0/4117649019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963406b6c0 0x7f963406e380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9634074f50 tx=0x7f962c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c529021ef9f6e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.190+0000 7f9632ffd640 1 -- 172.21.15.122:0/4117649019 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c032070 con 0x7f963406b6c0 2024-03-06T07:21:29.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.190+0000 7f9632ffd640 1 -- 172.21.15.122:0/4117649019 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c02cdb0 con 0x7f963406b6c0 2024-03-06T07:21:29.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.190+0000 7f9632ffd640 1 -- 172.21.15.122:0/4117649019 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c036600 con 0x7f963406b6c0 2024-03-06T07:21:29.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 -- 172.21.15.122:0/4117649019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963406b6c0 msgr2=0x7f963406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 --2- 172.21.15.122:0/4117649019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963406b6c0 0x7f963406e380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9634074f50 tx=0x7f962c02c000 comp rx=0 tx=0).stop 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 -- 172.21.15.122:0/4117649019 shutdown_connections 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 --2- 172.21.15.122:0/4117649019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963406b6c0 0x7f963406e380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 --2- 172.21.15.122:0/4117649019 >> v2:172.21.15.169:3300/0 conn(0x7f9634161980 0x7f96340cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 -- 172.21.15.122:0/4117649019 >> 172.21.15.122:0/4117649019 conn(0x7f96340cd420 msgr2=0x7f96340cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 -- 172.21.15.122:0/4117649019 shutdown_connections 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.191+0000 7f963a916640 1 -- 172.21.15.122:0/4117649019 wait complete. 2024-03-06T07:21:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.192+0000 7f963a916640 1 Processor -- start 2024-03-06T07:21:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.192+0000 7f963a916640 1 -- start start 2024-03-06T07:21:29.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f963a916640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f963406b6c0 0x7f9634154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f963a916640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96340c72a0 0x7f963415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f963a916640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 0x7f963415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f963a916640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96340c7700 con 0x7f9634161980 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f963a916640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96340c6c80 con 0x7f96340c72a0 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f963a916640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340ce090 con 0x7f963406b6c0 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f9638eb3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 0x7f963415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f9633fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f963406b6c0 0x7f9634154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f96337fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96340c72a0 0x7f963415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f9638eb3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 0x7f963415d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52286/0 (socket says 172.21.15.122:52286) 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.193+0000 7f9638eb3640 1 -- 172.21.15.122:0/1021124143 learned_addr learned my addr 172.21.15.122:0/1021124143 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9638eb3640 1 -- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f963406b6c0 msgr2=0x7f9634154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9638eb3640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f963406b6c0 0x7f9634154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9638eb3640 1 -- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96340c72a0 msgr2=0x7f963415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9638eb3640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96340c72a0 0x7f963415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9638eb3640 1 -- 172.21.15.122:0/1021124143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963415e740 con 0x7f9634161980 2024-03-06T07:21:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f96337fe640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96340c72a0 0x7f963415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9638eb3640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 0x7f963415d070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f962c009c10 tx=0x7f962c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c02cbb0 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c03f5a0 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963415ea20 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c03f950 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963415efe0 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f962c047020 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.194+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f962c040b60 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.195+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963404f330 con 0x7f9634161980 2024-03-06T07:21:29.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.197+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f962c02cd50 con 0x7f9634161980 2024-03-06T07:21:29.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.264+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f96340cb620 con 0x7f9634161980 2024-03-06T07:21:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.265+0000 7f9630ff9640 1 -- 172.21.15.122:0/1021124143 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f962c040db0 con 0x7f9634161980 2024-03-06T07:21:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:adjusted tunables profile to default 2024-03-06T07:21:29.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 msgr2=0x7f963415d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 0x7f963415d070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f962c009c10 tx=0x7f962c004ab0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 shutdown_connections 2024-03-06T07:21:29.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9634161980 0x7f963415d070 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96340c72a0 0x7f963415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 --2- 172.21.15.122:0/1021124143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f963406b6c0 0x7f9634154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 >> 172.21.15.122:0/1021124143 conn(0x7f96340cd420 msgr2=0x7f96340cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 shutdown_connections 2024-03-06T07:21:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:29.266+0000 7f963a916640 1 -- 172.21.15.122:0/1021124143 wait complete. 2024-03-06T07:21:29.279 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:21:29.279 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:21:29.279 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:29.279 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:21:29.306 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:29.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:21:29.372 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:29.372 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:21:29.439 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:21:29.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:21:29.506 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:29.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:21:29.535 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:29.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:21:29.603 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:29.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:21:29.670 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:29.668+0000 7f1d64e2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:29.672 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:21:29.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:21:29.723 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:29.721+0000 7f1d64e2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:29.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 2d8c299b-b0ee-4721-950f-7e9b86285b56 0 2024-03-06T07:21:29.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.879+0000 7f705144e640 1 Processor -- start 2024-03-06T07:21:29.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.880+0000 7f705144e640 1 -- start start 2024-03-06T07:21:29.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.880+0000 7f705144e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f704c166d40 0x7f704c169130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.880+0000 7f705144e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c0cf430 con 0x7f704c0b42f0 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.880+0000 7f705144e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c06e240 con 0x7f704c0701a0 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.880+0000 7f705144e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704c15f6e0 con 0x7f704c166d40 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f704c166d40 0x7f704c169130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f704c166d40 0x7f704c169130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46088/0 (socket says 172.21.15.169:46088) 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704b7fe640 1 -- 172.21.15.169:0/3845879093 learned_addr learned my addr 172.21.15.169:0/3845879093 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704affd640 1 --1- 172.21.15.169:0/3845879093 >> v1:172.21.15.169:6789/0 conn(0x7f704c0701a0 0x7f704c06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:40582/0 (socket says 172.21.15.169:40582) 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704b7fe640 1 -- 172.21.15.169:0/3845879093 >> v1:172.21.15.169:6789/0 conn(0x7f704c0701a0 legacy=0x7f704c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704b7fe640 1 -- 172.21.15.169:0/3845879093 >> v1:172.21.15.122:6789/0 conn(0x7f704c0b42f0 legacy=0x7f704c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.881+0000 7f704b7fe640 1 -- 172.21.15.169:0/3845879093 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c0cae80 con 0x7f704c166d40 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.882+0000 7f704b7fe640 1 --2- 172.21.15.169:0/3845879093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c166d40 0x7f704c169130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7044009e50 tx=0x7f704402c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99fdda26ea056dc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.882+0000 7f704a7fc640 1 -- 172.21.15.169:0/3845879093 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704402cbb0 con 0x7f704c166d40 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.882+0000 7f704a7fc640 1 -- 172.21.15.169:0/3845879093 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704402cd50 con 0x7f704c166d40 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.882+0000 7f704a7fc640 1 -- 172.21.15.169:0/3845879093 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044034610 con 0x7f704c166d40 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 -- 172.21.15.169:0/3845879093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c166d40 msgr2=0x7f704c169130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 --2- 172.21.15.169:0/3845879093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c166d40 0x7f704c169130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7044009e50 tx=0x7f704402c010 comp rx=0 tx=0).stop 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 -- 172.21.15.169:0/3845879093 shutdown_connections 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 --2- 172.21.15.169:0/3845879093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c166d40 0x7f704c169130 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 -- 172.21.15.169:0/3845879093 >> 172.21.15.169:0/3845879093 conn(0x7f704c0b3800 msgr2=0x7f704c0b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 -- 172.21.15.169:0/3845879093 shutdown_connections 2024-03-06T07:21:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.883+0000 7f705144e640 1 -- 172.21.15.169:0/3845879093 wait complete. 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.884+0000 7f705144e640 1 Processor -- start 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.884+0000 7f705144e640 1 -- start start 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f704c0701a0 0x7f704c0b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 0x7f704c0b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f704c166d40 0x7f704c14f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0cec30 con 0x7f704c0701a0 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0ce950 con 0x7f704c0b42f0 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0cef10 con 0x7f704c166d40 2024-03-06T07:21:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 0x7f704c0b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f704b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f704c166d40 0x7f704c14f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 0x7f704c0b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46102/0 (socket says 172.21.15.169:46102) 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 -- 172.21.15.169:0/430629143 learned_addr learned my addr 172.21.15.169:0/430629143 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f704affd640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f704c0701a0 0x7f704c0b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 -- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f704c166d40 msgr2=0x7f704c14f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f704c166d40 0x7f704c14f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 -- 172.21.15.169:0/430629143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f704c0701a0 msgr2=0x7f704c0b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f704c0701a0 0x7f704c0b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 -- 172.21.15.169:0/430629143 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c0b5280 con 0x7f704c0b42f0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f7043fff640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 0x7f704c0b1f80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f703400c430 tx=0x7f703400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70340052d0 con 0x7f704c0b42f0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7034005490 con 0x7f704c0b42f0 2024-03-06T07:21:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c0b5560 con 0x7f704c0b42f0 2024-03-06T07:21:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034007260 con 0x7f704c0b42f0 2024-03-06T07:21:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.885+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c0b5b20 con 0x7f704c0b42f0 2024-03-06T07:21:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.886+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7034007400 con 0x7f704c0b42f0 2024-03-06T07:21:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.886+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7034003b30 con 0x7f704c0b42f0 2024-03-06T07:21:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.886+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c09fe60 con 0x7f704c0b42f0 2024-03-06T07:21:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.888+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7034003d80 con 0x7f704c0b42f0 2024-03-06T07:21:29.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.961+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2d8c299b-b0ee-4721-950f-7e9b86285b56", "id": 0} v 0) v1 -- 0x7f704c06c240 con 0x7f704c0b42f0 2024-03-06T07:21:29.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.966+0000 7f70437fe640 1 -- 172.21.15.169:0/430629143 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d8c299b-b0ee-4721-950f-7e9b86285b56", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f70340035f0 con 0x7f704c0b42f0 2024-03-06T07:21:29.969 INFO:teuthology.orchestra.run.smithi169.stdout:0 2024-03-06T07:21:29.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 msgr2=0x7f704c0b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 0x7f704c0b1f80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f703400c430 tx=0x7f703400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 shutdown_connections 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f704c166d40 0x7f704c14f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f704c0b42f0 0x7f704c0b1f80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 --2- 172.21.15.169:0/430629143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f704c0701a0 0x7f704c0b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 >> 172.21.15.169:0/430629143 conn(0x7f704c0b3800 msgr2=0x7f704c0b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 shutdown_connections 2024-03-06T07:21:29.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:29.967+0000 7f705144e640 1 -- 172.21.15.169:0/430629143 wait complete. 2024-03-06T07:21:29.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 49c5e29b-a496-4025-a0fa-d8122f00ae45 1 2024-03-06T07:21:30.047 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.044+0000 7f440242b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:30.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 Processor -- start 2024-03-06T07:21:30.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 -- start start 2024-03-06T07:21:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd4b81455d0 0x7fd4b8072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd4b806b3f0 0x7fd4b806e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b81459b0 con 0x7fd4b806afb0 2024-03-06T07:21:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b80cfdf0 con 0x7fd4b81455d0 2024-03-06T07:21:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.080+0000 7fd4beab5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b80cf1e0 con 0x7fd4b806b3f0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd4b806b3f0 0x7fd4b806e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd4b806b3f0 0x7fd4b806e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46118/0 (socket says 172.21.15.169:46118) 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 -- 172.21.15.169:0/178752934 learned_addr learned my addr 172.21.15.169:0/178752934 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bc851640 1 --2- 172.21.15.169:0/178752934 >> v2:172.21.15.122:3300/0 conn(0x7fd4b81455d0 0x7fd4b8072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 -- 172.21.15.169:0/178752934 >> v2:172.21.15.122:3300/0 conn(0x7fd4b81455d0 msgr2=0x7fd4b8072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 --2- 172.21.15.169:0/178752934 >> v2:172.21.15.122:3300/0 conn(0x7fd4b81455d0 0x7fd4b8072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 -- 172.21.15.169:0/178752934 >> v1:172.21.15.169:6789/0 conn(0x7fd4b806afb0 legacy=0x7fd4b806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.081+0000 7fd4bd052640 1 -- 172.21.15.169:0/178752934 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80cf4c0 con 0x7fd4b806b3f0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4bd052640 1 --2- 172.21.15.169:0/178752934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b806e5e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80d09a0 tx=0x7fd4b002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f964a580981439 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4b77fe640 1 -- 172.21.15.169:0/178752934 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b002cb20 con 0x7fd4b806b3f0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4b77fe640 1 -- 172.21.15.169:0/178752934 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b002ccc0 con 0x7fd4b806b3f0 2024-03-06T07:21:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4b77fe640 1 -- 172.21.15.169:0/178752934 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0034610 con 0x7fd4b806b3f0 2024-03-06T07:21:30.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 -- 172.21.15.169:0/178752934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 msgr2=0x7fd4b806e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 --2- 172.21.15.169:0/178752934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b806e5e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80d09a0 tx=0x7fd4b002bf80 comp rx=0 tx=0).stop 2024-03-06T07:21:30.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 -- 172.21.15.169:0/178752934 shutdown_connections 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 --2- 172.21.15.169:0/178752934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b806e5e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 --2- 172.21.15.169:0/178752934 >> v2:172.21.15.122:3300/0 conn(0x7fd4b81455d0 0x7fd4b8072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 -- 172.21.15.169:0/178752934 >> 172.21.15.169:0/178752934 conn(0x7fd4b80b2660 msgr2=0x7fd4b80b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 -- 172.21.15.169:0/178752934 shutdown_connections 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.082+0000 7fd4beab5640 1 -- 172.21.15.169:0/178752934 wait complete. 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 Processor -- start 2024-03-06T07:21:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 -- start start 2024-03-06T07:21:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 0x7fd4b81375a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b8141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4b81455d0 0x7fd4b8141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80729c0 con 0x7fd4b81455d0 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80cfa80 con 0x7fd4b806b3f0 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.084+0000 7fd4beab5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80b3400 con 0x7fd4b806afb0 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 0x7fd4b81375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b8141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 0x7fd4b81375a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.169:41998/0 (socket says 172.21.15.169:41998) 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 -- 172.21.15.169:0/3147155168 learned_addr learned my addr 172.21.15.169:0/3147155168 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bd052640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4b81455d0 0x7fd4b8141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 -- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 msgr2=0x7fd4b8141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b8141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 -- 172.21.15.169:0/3147155168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4b81455d0 msgr2=0x7fd4b8141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4b81455d0 0x7fd4b8141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 -- 172.21.15.169:0/3147155168 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b813aaa0 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4b7fff640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b8141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bd052640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4b81455d0 0x7fd4b8141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4bc851640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 0x7fd4b81375a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00e870 tx=0x7fd4ac00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac017070 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac00c940 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac00cd30 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b813ad80 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.085+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b813c5d0 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.086+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd4ac01a070 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.086+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fd4ac01c840 con 0x7fd4b806afb0 2024-03-06T07:21:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.087+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b8073390 con 0x7fd4b806afb0 2024-03-06T07:21:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.089+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd4ac01ca40 con 0x7fd4b806afb0 2024-03-06T07:21:30.105 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.102+0000 7f440242b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:30.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.162+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd new", "uuid": "49c5e29b-a496-4025-a0fa-d8122f00ae45", "id": 1} v 0) v1 -- 0x7fd4b8068ff0 con 0x7fd4b806afb0 2024-03-06T07:21:30.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.165+0000 7fd4b57fa640 1 -- 172.21.15.169:0/3147155168 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49c5e29b-a496-4025-a0fa-d8122f00ae45", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd4ac01cc20 con 0x7fd4b806afb0 2024-03-06T07:21:30.168 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 msgr2=0x7fd4b81375a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 0x7fd4b81375a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00e870 tx=0x7fd4ac00ed30 comp rx=0 tx=0).stop 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 shutdown_connections 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4b81455d0 0x7fd4b8141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b806b3f0 0x7fd4b8141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 --2- 172.21.15.169:0/3147155168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4b806afb0 0x7fd4b81375a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 >> 172.21.15.169:0/3147155168 conn(0x7fd4b80b2660 msgr2=0x7fd4b806f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 shutdown_connections 2024-03-06T07:21:30.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.166+0000 7fd4beab5640 1 -- 172.21.15.169:0/3147155168 wait complete. 2024-03-06T07:21:30.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db 2 2024-03-06T07:21:30.262 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.260+0000 7f1d64e2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:30.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.278+0000 7f212a3b5640 1 Processor -- start 2024-03-06T07:21:30.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.278+0000 7f212a3b5640 1 -- start start 2024-03-06T07:21:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f212a3b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 0x7f21240b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f212a3b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 0x7f2124157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f212a3b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21240b3a00 con 0x7f212406b600 2024-03-06T07:21:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f212a3b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21240b3ce0 con 0x7f212415ecb0 2024-03-06T07:21:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f212a3b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212415e4b0 con 0x7f212406b220 2024-03-06T07:21:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 0x7f21240b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2128952640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 0x7f2124157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 0x7f21240b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.169:42014/0 (socket says 172.21.15.169:42014) 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2128952640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 0x7f2124157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46148/0 (socket says 172.21.15.169:46148) 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 -- 172.21.15.169:0/107586381 learned_addr learned my addr 172.21.15.169:0/107586381 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 -- 172.21.15.169:0/107586381 >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 msgr2=0x7f2124157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 --2- 172.21.15.169:0/107586381 >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 0x7f2124157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 -- 172.21.15.169:0/107586381 >> v1:172.21.15.122:6789/0 conn(0x7f212406b600 legacy=0x7f2124072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2123fff640 1 -- 172.21.15.169:0/107586381 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212415e790 con 0x7f212406b220 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.279+0000 7f2128952640 1 --2- 172.21.15.169:0/107586381 >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 0x7f2124157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:30.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.280+0000 7f2123fff640 1 --2- 172.21.15.169:0/107586381 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 0x7f21240b58f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2114009870 tx=0x7f211402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d9b7358faf738ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.280+0000 7f2122ffd640 1 -- 172.21.15.169:0/107586381 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114035020 con 0x7f212406b220 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.280+0000 7f2122ffd640 1 -- 172.21.15.169:0/107586381 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2114031640 con 0x7f212406b220 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.280+0000 7f2122ffd640 1 -- 172.21.15.169:0/107586381 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211402d040 con 0x7f212406b220 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 -- 172.21.15.169:0/107586381 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 msgr2=0x7f21240b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 --2- 172.21.15.169:0/107586381 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 0x7f21240b58f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2114009870 tx=0x7f211402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 -- 172.21.15.169:0/107586381 shutdown_connections 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 --2- 172.21.15.169:0/107586381 >> v2:172.21.15.169:3300/0 conn(0x7f212415ecb0 0x7f2124157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 --2- 172.21.15.169:0/107586381 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b220 0x7f21240b58f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 -- 172.21.15.169:0/107586381 >> 172.21.15.169:0/107586381 conn(0x7f212406c0c0 msgr2=0x7f212406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 -- 172.21.15.169:0/107586381 shutdown_connections 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.281+0000 7f212a3b5640 1 -- 172.21.15.169:0/107586381 wait complete. 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.282+0000 7f212a3b5640 1 Processor -- start 2024-03-06T07:21:30.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.282+0000 7f212a3b5640 1 -- start start 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f212a3b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f212406b220 0x7f2124144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f212a3b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b600 0x7f2124161cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f212a3b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 0x7f2124162200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f212a3b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212415f2a0 con 0x7f212406b220 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f212a3b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212415f120 con 0x7f212415ecb0 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f212a3b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2124158390 con 0x7f212406b600 2024-03-06T07:21:30.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 0x7f2124162200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f21237fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b600 0x7f2124161cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 0x7f2124162200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46160/0 (socket says 172.21.15.169:46160) 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 -- 172.21.15.169:0/3271153147 learned_addr learned my addr 172.21.15.169:0/3271153147 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2123fff640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f212406b220 0x7f2124144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 -- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b600 msgr2=0x7f2124161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b600 0x7f2124161cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 -- 172.21.15.169:0/3271153147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f212406b220 msgr2=0x7f2124144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f212406b220 0x7f2124144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 -- 172.21.15.169:0/3271153147 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124147820 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.283+0000 7f2128952640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 0x7f2124162200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f211c009e60 tx=0x7f211c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c018030 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c00cdf0 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2124144990 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c003bb0 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2124144f50 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f211c005d10 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f211c01d910 con 0x7f212415ecb0 2024-03-06T07:21:30.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.284+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21240733b0 con 0x7f212415ecb0 2024-03-06T07:21:30.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.287+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f211c022070 con 0x7f212415ecb0 2024-03-06T07:21:30.316 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.314+0000 7f1d64e2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:30.360 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.358+0000 7f1d64e2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:30.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.359+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db", "id": 2} v 0) v1 -- 0x7f21240722e0 con 0x7f212415ecb0 2024-03-06T07:21:30.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.364+0000 7f2120ff9640 1 -- 172.21.15.169:0/3271153147 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f211c010040 con 0x7f212415ecb0 2024-03-06T07:21:30.366 INFO:teuthology.orchestra.run.smithi169.stdout:2 2024-03-06T07:21:30.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 msgr2=0x7f2124162200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 0x7f2124162200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f211c009e60 tx=0x7f211c00c490 comp rx=0 tx=0).stop 2024-03-06T07:21:30.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 shutdown_connections 2024-03-06T07:21:30.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f212415ecb0 0x7f2124162200 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f212406b600 0x7f2124161cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 --2- 172.21.15.169:0/3271153147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f212406b220 0x7f2124144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 >> 172.21.15.169:0/3271153147 conn(0x7f212406c0c0 msgr2=0x7f212406ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 shutdown_connections 2024-03-06T07:21:30.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.365+0000 7f212a3b5640 1 -- 172.21.15.169:0/3271153147 wait complete. 2024-03-06T07:21:30.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 17483d4d-a374-4a1a-b0cf-6765bcc28ad0 3 2024-03-06T07:21:30.415 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.413+0000 7f1d64e2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:30.457 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.455+0000 7f1d64e2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.480+0000 7ff5fa6d4640 1 Processor -- start 2024-03-06T07:21:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.480+0000 7ff5fa6d4640 1 -- start start 2024-03-06T07:21:30.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5fa6d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f4072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5fa6d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5f406b3f0 0x7ff5f406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5fa6d4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f41459b0 con 0x7ff5f406afb0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5fa6d4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f40cfdf0 con 0x7ff5f406b3f0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5fa6d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cf1e0 con 0x7ff5f41455d0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f4072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f37fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff5f406afb0 0x7ff5f406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:40598/0 (socket says 172.21.15.169:40598) 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f37fe640 1 -- 172.21.15.169:0/914330680 learned_addr learned my addr 172.21.15.169:0/914330680 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f8c71640 1 --2- 172.21.15.169:0/914330680 >> v2:172.21.15.122:3300/0 conn(0x7ff5f406b3f0 0x7ff5f406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f3fff640 1 -- 172.21.15.169:0/914330680 >> v2:172.21.15.122:3300/0 conn(0x7ff5f406b3f0 msgr2=0x7ff5f406e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f3fff640 1 --2- 172.21.15.169:0/914330680 >> v2:172.21.15.122:3300/0 conn(0x7ff5f406b3f0 0x7ff5f406e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f2ffd640 1 -- 172.21.15.169:0/914330680 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259728125 0 0) 0x7ff5f41459b0 con 0x7ff5f406afb0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f3fff640 1 -- 172.21.15.169:0/914330680 >> v1:172.21.15.169:6789/0 conn(0x7ff5f406afb0 legacy=0x7ff5f406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f3fff640 1 -- 172.21.15.169:0/914330680 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40cf4c0 con 0x7ff5f41455d0 2024-03-06T07:21:30.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.481+0000 7ff5f8c71640 1 --2- 172.21.15.169:0/914330680 >> v2:172.21.15.122:3300/0 conn(0x7ff5f406b3f0 0x7ff5f406e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5f3fff640 1 --2- 172.21.15.169:0/914330680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f4072480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40097a0 tx=0x7ff5e402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77437919007a8ece server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5f2ffd640 1 -- 172.21.15.169:0/914330680 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4035020 con 0x7ff5f41455d0 2024-03-06T07:21:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5f2ffd640 1 -- 172.21.15.169:0/914330680 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e4031830 con 0x7ff5f41455d0 2024-03-06T07:21:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5f2ffd640 1 -- 172.21.15.169:0/914330680 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4031c00 con 0x7ff5f41455d0 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/914330680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 msgr2=0x7ff5f4072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/914330680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f4072480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40097a0 tx=0x7ff5e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/914330680 shutdown_connections 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/914330680 >> v2:172.21.15.122:3300/0 conn(0x7ff5f406b3f0 0x7ff5f406e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/914330680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f4072480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/914330680 >> 172.21.15.169:0/914330680 conn(0x7ff5f40b2660 msgr2=0x7ff5f40b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/914330680 shutdown_connections 2024-03-06T07:21:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.482+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/914330680 wait complete. 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 Processor -- start 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 -- start start 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5f406afb0 0x7ff5f41f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 0x7ff5f413a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f413a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f406b880 con 0x7ff5f406afb0 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cfa80 con 0x7ff5f406b3f0 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.484+0000 7ff5fa6d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40729c0 con 0x7ff5f41455d0 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 0x7ff5f413a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f413a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 0x7ff5f413a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46172/0 (socket says 172.21.15.169:46172) 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f37fe640 1 -- 172.21.15.169:0/1021620926 learned_addr learned my addr 172.21.15.169:0/1021620926 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f3fff640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5f406afb0 0x7ff5f41f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 -- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 msgr2=0x7ff5f413a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 0x7ff5f413a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 -- 172.21.15.169:0/1021620926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5f406afb0 msgr2=0x7ff5f41f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5f406afb0 0x7ff5f41f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 -- 172.21.15.169:0/1021620926 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f41f42e0 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f37fe640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 0x7ff5f413a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f8c71640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f413a5e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec007e60 tx=0x7ff5ec00d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec01a070 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec003b90 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f415cea0 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec004030 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f415d460 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff5ec01d070 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.485+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7ff5ec0158d0 con 0x7ff5f41455d0 2024-03-06T07:21:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.486+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f4073390 con 0x7ff5f41455d0 2024-03-06T07:21:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.488+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff5ec024020 con 0x7ff5f41455d0 2024-03-06T07:21:30.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.561+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd new", "uuid": "17483d4d-a374-4a1a-b0cf-6765bcc28ad0", "id": 3} v 0) v1 -- 0x7ff5f406ed80 con 0x7ff5f41455d0 2024-03-06T07:21:30.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.566+0000 7ff5f0ff9640 1 -- 172.21.15.169:0/1021620926 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17483d4d-a374-4a1a-b0cf-6765bcc28ad0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5ec010070 con 0x7ff5f41455d0 2024-03-06T07:21:30.569 INFO:teuthology.orchestra.run.smithi169.stdout:3 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 msgr2=0x7ff5f413a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f413a5e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec007e60 tx=0x7ff5ec00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 shutdown_connections 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5f41455d0 0x7ff5f413a5e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5f406b3f0 0x7ff5f413a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 --2- 172.21.15.169:0/1021620926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5f406afb0 0x7ff5f41f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 >> 172.21.15.169:0/1021620926 conn(0x7ff5f40b2660 msgr2=0x7ff5f40b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 shutdown_connections 2024-03-06T07:21:30.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.567+0000 7ff5fa6d4640 1 -- 172.21.15.169:0/1021620926 wait complete. 2024-03-06T07:21:30.573 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.571+0000 7f1d64e2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:30.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new a0dacb4c-2138-4ddc-a686-2f38ac64f9dd 4 2024-03-06T07:21:30.641 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.639+0000 7f1d64e2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:30.676 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.673+0000 7f440242b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:30.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.677+0000 7f68da86d640 1 Processor -- start 2024-03-06T07:21:30.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.678+0000 7f68da86d640 1 -- start start 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68da86d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f68d415f2a0 0x7f68d40b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68da86d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4073190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68da86d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d406ba60 con 0x7f68d40d0a40 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68da86d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d4072590 con 0x7f68d415f2a0 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68da86d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d406bd00 con 0x7f68d4072d00 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4073190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4073190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.169:42054/0 (socket says 172.21.15.169:42054) 2024-03-06T07:21:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 -- 172.21.15.169:0/2995057725 learned_addr learned my addr 172.21.15.169:0/2995057725 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d3fff640 1 --2- 172.21.15.169:0/2995057725 >> v2:172.21.15.122:3300/0 conn(0x7f68d415f2a0 0x7f68d40b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 -- 172.21.15.169:0/2995057725 >> v2:172.21.15.122:3300/0 conn(0x7f68d415f2a0 msgr2=0x7f68d40b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 --2- 172.21.15.169:0/2995057725 >> v2:172.21.15.122:3300/0 conn(0x7f68d415f2a0 0x7f68d40b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 -- 172.21.15.169:0/2995057725 >> v1:172.21.15.122:6789/0 conn(0x7f68d40d0a40 legacy=0x7f68d40b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.679+0000 7f68d8e0a640 1 -- 172.21.15.169:0/2995057725 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d406bfe0 con 0x7f68d4072d00 2024-03-06T07:21:30.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.680+0000 7f68d8e0a640 1 --2- 172.21.15.169:0/2995057725 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4073190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009e50 tx=0x7f68cc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b0fcd0348b3488 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.680+0000 7f68d2ffd640 1 -- 172.21.15.169:0/2995057725 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc02cb00 con 0x7f68d4072d00 2024-03-06T07:21:30.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.680+0000 7f68d2ffd640 1 -- 172.21.15.169:0/2995057725 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc02cca0 con 0x7f68d4072d00 2024-03-06T07:21:30.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.680+0000 7f68d2ffd640 1 -- 172.21.15.169:0/2995057725 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc034610 con 0x7f68d4072d00 2024-03-06T07:21:30.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 -- 172.21.15.169:0/2995057725 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 msgr2=0x7f68d4073190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 --2- 172.21.15.169:0/2995057725 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4073190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009e50 tx=0x7f68cc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 -- 172.21.15.169:0/2995057725 shutdown_connections 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 --2- 172.21.15.169:0/2995057725 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4073190 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 --2- 172.21.15.169:0/2995057725 >> v2:172.21.15.122:3300/0 conn(0x7f68d415f2a0 0x7f68d40b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 -- 172.21.15.169:0/2995057725 >> 172.21.15.169:0/2995057725 conn(0x7f68d40b4fe0 msgr2=0x7f68d40b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 -- 172.21.15.169:0/2995057725 shutdown_connections 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.681+0000 7f68da86d640 1 -- 172.21.15.169:0/2995057725 wait complete. 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.683+0000 7f68da86d640 1 Processor -- start 2024-03-06T07:21:30.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.683+0000 7f68da86d640 1 -- start start 2024-03-06T07:21:30.686 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.684+0000 7f1d64e2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:30.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68da86d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68da86d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68d40d0a40 0x7f68d4156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68da86d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68d415f2a0 0x7f68d41565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68da86d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d415f800 con 0x7f68d415f2a0 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68da86d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d415f680 con 0x7f68d40d0a40 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68da86d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40b10f0 con 0x7f68d4072d00 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68d3fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68d3fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4157530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.169:42064/0 (socket says 172.21.15.169:42064) 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68d3fff640 1 -- 172.21.15.169:0/1530884644 learned_addr learned my addr 172.21.15.169:0/1530884644 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.684+0000 7f68d37fe640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68d40d0a40 0x7f68d4156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d8e0a640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68d415f2a0 0x7f68d41565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d3fff640 1 -- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68d40d0a40 msgr2=0x7f68d4156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d3fff640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68d40d0a40 0x7f68d4156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d3fff640 1 -- 172.21.15.169:0/1530884644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68d415f2a0 msgr2=0x7f68d41565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d3fff640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68d415f2a0 0x7f68d41565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d3fff640 1 -- 172.21.15.169:0/1530884644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d415b7c0 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d3fff640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4157530 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68c400e9b0 tx=0x7f68c400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c4017070 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c400c9a0 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d4158af0 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c400cd90 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d4157a70 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f68c401a070 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.685+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f68c401caa0 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.686+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d409e4e0 con 0x7f68d4072d00 2024-03-06T07:21:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.688+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f68c401cdb0 con 0x7f68d4072d00 2024-03-06T07:21:30.734 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.732+0000 7f440242b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:30.738 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.736+0000 7f1d64e2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:30.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.760+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a0dacb4c-2138-4ddc-a686-2f38ac64f9dd", "id": 4} v 0) v1 -- 0x7f68d40d0100 con 0x7f68d4072d00 2024-03-06T07:21:30.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.763+0000 7f68d0ff9640 1 -- 172.21.15.169:0/1530884644 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a0dacb4c-2138-4ddc-a686-2f38ac64f9dd", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f68c401c660 con 0x7f68d4072d00 2024-03-06T07:21:30.766 INFO:teuthology.orchestra.run.smithi169.stdout:4 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 msgr2=0x7f68d4157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4157530 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68c400e9b0 tx=0x7f68c400ee70 comp rx=0 tx=0).stop 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 shutdown_connections 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68d415f2a0 0x7f68d41565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68d40d0a40 0x7f68d4156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 --2- 172.21.15.169:0/1530884644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68d4072d00 0x7f68d4157530 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 >> 172.21.15.169:0/1530884644 conn(0x7f68d40b4fe0 msgr2=0x7f68d40af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 shutdown_connections 2024-03-06T07:21:30.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.764+0000 7f68da86d640 1 -- 172.21.15.169:0/1530884644 wait complete. 2024-03-06T07:21:30.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new eeb05427-e45f-42bd-9e82-bcf250302eef 5 2024-03-06T07:21:30.782 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.779+0000 7f440242b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:30.785 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:30.784+0000 7f1d64e2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:30.839 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.837+0000 7f440242b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:30.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.876+0000 7f4a68d56640 1 Processor -- start 2024-03-06T07:21:30.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.876+0000 7f4a68d56640 1 -- start start 2024-03-06T07:21:30.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a68d56640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4a6415e0e0 0x7f4a6406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a68d56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a64161bd0 con 0x7f4a6415dd00 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a68d56640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a64070220 con 0x7f4a6406b6c0 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a68d56640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a64161e90 con 0x7f4a6415e0e0 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a61d9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4a6415e0e0 0x7f4a6406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a61d9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4a6415e0e0 0x7f4a6406e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46184/0 (socket says 172.21.15.169:46184) 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a61d9b640 1 -- 172.21.15.169:0/1342750058 learned_addr learned my addr 172.21.15.169:0/1342750058 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a61d9b640 1 -- 172.21.15.169:0/1342750058 >> v1:172.21.15.169:6789/0 conn(0x7f4a6406b6c0 legacy=0x7f4a640b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a61d9b640 1 -- 172.21.15.169:0/1342750058 >> v1:172.21.15.122:6789/0 conn(0x7f4a6415dd00 legacy=0x7f4a6406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.877+0000 7f4a61d9b640 1 -- 172.21.15.169:0/1342750058 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64162170 con 0x7f4a6415e0e0 2024-03-06T07:21:30.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.878+0000 7f4a61d9b640 1 --2- 172.21.15.169:0/1342750058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6415e0e0 0x7f4a6406e620 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009870 tx=0x7f4a4c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e82b6f4d568f4d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.878+0000 7f4a6159a640 1 -- 172.21.15.169:0/1342750058 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c02f840 con 0x7f4a6415e0e0 2024-03-06T07:21:30.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.878+0000 7f4a6159a640 1 -- 172.21.15.169:0/1342750058 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c02f9e0 con 0x7f4a6415e0e0 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.878+0000 7f4a6159a640 1 -- 172.21.15.169:0/1342750058 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c02fdb0 con 0x7f4a6415e0e0 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.878+0000 7f4a68d56640 1 -- 172.21.15.169:0/1342750058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6415e0e0 msgr2=0x7f4a6406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.878+0000 7f4a68d56640 1 --2- 172.21.15.169:0/1342750058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6415e0e0 0x7f4a6406e620 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009870 tx=0x7f4a4c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.879+0000 7f4a68d56640 1 -- 172.21.15.169:0/1342750058 shutdown_connections 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.879+0000 7f4a68d56640 1 --2- 172.21.15.169:0/1342750058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6415e0e0 0x7f4a6406e620 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.879+0000 7f4a68d56640 1 -- 172.21.15.169:0/1342750058 >> 172.21.15.169:0/1342750058 conn(0x7f4a640b2660 msgr2=0x7f4a640b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.879+0000 7f4a68d56640 1 -- 172.21.15.169:0/1342750058 shutdown_connections 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.879+0000 7f4a68d56640 1 -- 172.21.15.169:0/1342750058 wait complete. 2024-03-06T07:21:30.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.880+0000 7f4a68d56640 1 Processor -- start 2024-03-06T07:21:30.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.880+0000 7f4a68d56640 1 -- start start 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a68d56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 0x7f4a640d0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a68d56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6415dd00 0x7f4a640c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a68d56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a640c7b00 0x7f4a64147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a68d56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640b0180 con 0x7f4a6415dd00 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a68d56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640b0000 con 0x7f4a6406b6c0 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a68d56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640b0300 con 0x7f4a640c7b00 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a6259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 0x7f4a640d0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a6259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 0x7f4a640d0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46188/0 (socket says 172.21.15.169:46188) 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a6259c640 1 -- 172.21.15.169:0/1388211460 learned_addr learned my addr 172.21.15.169:0/1388211460 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:30.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a640c7b00 0x7f4a64147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a61d9b640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6415dd00 0x7f4a640c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 -- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 msgr2=0x7f4a640d0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 0x7f4a640d0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 -- 172.21.15.169:0/1388211460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6415dd00 msgr2=0x7f4a640c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6415dd00 0x7f4a640c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 -- 172.21.15.169:0/1388211460 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6415f980 con 0x7f4a640c7b00 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a6259c640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 0x7f4a640d0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a62d9d640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a640c7b00 0x7f4a64147ff0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00c700 tx=0x7f4a5c00cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.881+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c003b70 con 0x7f4a640c7b00 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.882+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c003d10 con 0x7f4a640c7b00 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.882+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6415fc60 con 0x7f4a640c7b00 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.882+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c005310 con 0x7f4a640c7b00 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.882+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a64160220 con 0x7f4a640c7b00 2024-03-06T07:21:30.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.882+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4a5c0054b0 con 0x7f4a640c7b00 2024-03-06T07:21:30.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.882+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f4a5c0129a0 con 0x7f4a640c7b00 2024-03-06T07:21:30.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.883+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a64073390 con 0x7f4a640c7b00 2024-03-06T07:21:30.887 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:30.883+0000 7f440242b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:30.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.885+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4a5c012c80 con 0x7f4a640c7b00 2024-03-06T07:21:30.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.957+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd new", "uuid": "eeb05427-e45f-42bd-9e82-bcf250302eef", "id": 5} v 0) v1 -- 0x7f4a64068ff0 con 0x7f4a640c7b00 2024-03-06T07:21:30.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.963+0000 7f4a5affd640 1 -- 172.21.15.169:0/1388211460 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eeb05427-e45f-42bd-9e82-bcf250302eef", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4a5c012420 con 0x7f4a640c7b00 2024-03-06T07:21:30.965 INFO:teuthology.orchestra.run.smithi169.stdout:5 2024-03-06T07:21:30.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a640c7b00 msgr2=0x7f4a64147ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a640c7b00 0x7f4a64147ff0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00c700 tx=0x7f4a5c00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 shutdown_connections 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a640c7b00 0x7f4a64147ff0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6415dd00 0x7f4a640c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 --2- 172.21.15.169:0/1388211460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6406b6c0 0x7f4a640d0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 >> 172.21.15.169:0/1388211460 conn(0x7f4a640b2660 msgr2=0x7f4a641599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 shutdown_connections 2024-03-06T07:21:30.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:30.964+0000 7f4a68d56640 1 -- 172.21.15.169:0/1388211460 wait complete. 2024-03-06T07:21:30.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 60c966b2-ab66-42b5-826b-8282fb747a5f 6 2024-03-06T07:21:31.008 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.005+0000 7f440242b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:31.083 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.081+0000 7f440242b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:31.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.107+0000 7f5d36e46640 1 Processor -- start 2024-03-06T07:21:31.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.107+0000 7f5d36e46640 1 -- start start 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d36e46640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d3006fd10 0x7f5d300cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d36e46640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d300cd4b0 con 0x7f5d300d0af0 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d36e46640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d300cd790 con 0x7f5d3015c050 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d36e46640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d300cda70 con 0x7f5d3006fd10 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d34be2640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5d3015c050 0x7f5d3006f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:40608/0 (socket says 172.21.15.169:40608) 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d34be2640 1 -- 172.21.15.169:0/1155760048 learned_addr learned my addr 172.21.15.169:0/1155760048 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-06T07:21:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d2ffff640 1 --2- 172.21.15.169:0/1155760048 >> v2:172.21.15.122:3300/0 conn(0x7f5d3006fd10 0x7f5d300cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.108+0000 7f5d2f7fe640 1 -- 172.21.15.169:0/1155760048 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439035392 0 0) 0x7f5d300cd790 con 0x7f5d3015c050 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.109+0000 7f5d2f7fe640 1 -- 172.21.15.169:0/1155760048 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d20003520 con 0x7f5d3015c050 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.109+0000 7f5d2ffff640 1 -- 172.21.15.169:0/1155760048 >> v1:172.21.15.169:6789/0 conn(0x7f5d3015c050 legacy=0x7f5d3006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.109+0000 7f5d2ffff640 1 -- 172.21.15.169:0/1155760048 >> v1:172.21.15.122:6789/0 conn(0x7f5d300d0af0 legacy=0x7f5d300ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.109+0000 7f5d2f7fe640 1 -- 172.21.15.169:0/1155760048 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894923194 0 0) 0x7f5d300cd4b0 con 0x7f5d300d0af0 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.109+0000 7f5d2ffff640 1 -- 172.21.15.169:0/1155760048 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d300cdd50 con 0x7f5d3006fd10 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.109+0000 7f5d2ffff640 1 --2- 172.21.15.169:0/1155760048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3006fd10 0x7f5d300cc760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0097a0 tx=0x7f5d1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5dd89c92f8f5c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d2f7fe640 1 -- 172.21.15.169:0/1155760048 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c035020 con 0x7f5d3006fd10 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d2f7fe640 1 -- 172.21.15.169:0/1155760048 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c031830 con 0x7f5d3006fd10 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d2f7fe640 1 -- 172.21.15.169:0/1155760048 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c031c00 con 0x7f5d3006fd10 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 -- 172.21.15.169:0/1155760048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3006fd10 msgr2=0x7f5d300cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 --2- 172.21.15.169:0/1155760048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3006fd10 0x7f5d300cc760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0097a0 tx=0x7f5d1c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 -- 172.21.15.169:0/1155760048 shutdown_connections 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 --2- 172.21.15.169:0/1155760048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3006fd10 0x7f5d300cc760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 -- 172.21.15.169:0/1155760048 >> 172.21.15.169:0/1155760048 conn(0x7f5d300b3800 msgr2=0x7f5d300b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 -- 172.21.15.169:0/1155760048 shutdown_connections 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.110+0000 7f5d36e46640 1 -- 172.21.15.169:0/1155760048 wait complete. 2024-03-06T07:21:31.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.111+0000 7f5d36e46640 1 Processor -- start 2024-03-06T07:21:31.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 -- start start 2024-03-06T07:21:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 0x7f5d300b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d300d0af0 0x7f5d300b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 0x7f5d3014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d30166e50 con 0x7f5d300d0af0 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d30166cd0 con 0x7f5d3006fd10 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d36e46640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d30166fd0 con 0x7f5d3015c050 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d34be2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 0x7f5d300b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.112+0000 7f5d353e3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 0x7f5d3014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d34be2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 0x7f5d300b1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46204/0 (socket says 172.21.15.169:46204) 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d353e3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 0x7f5d3014b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.169:42084/0 (socket says 172.21.15.169:42084) 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d34be2640 1 -- 172.21.15.169:0/1195383288 learned_addr learned my addr 172.21.15.169:0/1195383288 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d300d0af0 0x7f5d300b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 -- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 msgr2=0x7f5d3014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 0x7f5d3014b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 -- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 msgr2=0x7f5d300b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 0x7f5d300b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 -- 172.21.15.169:0/1195383288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d301552b0 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d353e3640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 0x7f5d3014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d34be2640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 0x7f5d300b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2ffff640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d300d0af0 0x7f5d300b2220 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5d30071e40 tx=0x7f5d1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c040070 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c03e070 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d301565d0 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c035040 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d30155810 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.113+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5d1c038020 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.114+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f5d1c03aab0 con 0x7f5d300d0af0 2024-03-06T07:21:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.115+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3009fe60 con 0x7f5d300d0af0 2024-03-06T07:21:31.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.117+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5d301552b0 con 0x7f5d300d0af0 2024-03-06T07:21:31.142 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.140+0000 7f440242b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:31.160 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.158+0000 7f1d64e2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:31.206 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.203+0000 7f440242b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.213+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "60c966b2-ab66-42b5-826b-8282fb747a5f", "id": 6} v 0) v1 -- 0x7f5d30068ff0 con 0x7f5d300d0af0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.216+0000 7f5d2d7fa640 1 -- 172.21.15.169:0/1195383288 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "60c966b2-ab66-42b5-826b-8282fb747a5f", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5d1c031800 con 0x7f5d300d0af0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi169.stdout:6 2024-03-06T07:21:31.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.217+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d300d0af0 msgr2=0x7f5d300b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.217+0000 7f5d36e46640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d300d0af0 0x7f5d300b2220 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5d30071e40 tx=0x7f5d1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 shutdown_connections 2024-03-06T07:21:31.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d3015c050 0x7f5d3014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d300d0af0 0x7f5d300b2220 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 --2- 172.21.15.169:0/1195383288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d3006fd10 0x7f5d300b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 >> 172.21.15.169:0/1195383288 conn(0x7f5d300b3800 msgr2=0x7f5d300cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 shutdown_connections 2024-03-06T07:21:31.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.218+0000 7f5d36e46640 1 -- 172.21.15.169:0/1195383288 wait complete. 2024-03-06T07:21:31.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 1fd2b82a-556e-4d70-bde1-39ac144dca46 7 2024-03-06T07:21:31.259 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.256+0000 7f440242b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:31.295 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.293+0000 7f1d64e2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:31.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 Processor -- start 2024-03-06T07:21:31.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 -- start start 2024-03-06T07:21:31.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 0x7f9bc415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40d0ac0 0x7f9bc40b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9bc40b4d10 0x7f9bc40cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc40ced70 con 0x7f9bc40b4d10 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc40d47f0 con 0x7f9bc40d06e0 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.331+0000 7f9bc9c9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40d1ab0 con 0x7f9bc40d0ac0 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc37fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 0x7f9bc415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc2ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40d0ac0 0x7f9bc40b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc37fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 0x7f9bc415fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46220/0 (socket says 172.21.15.169:46220) 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc37fe640 1 -- 172.21.15.169:0/1083894103 learned_addr learned my addr 172.21.15.169:0/1083894103 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc3fff640 1 --2- 172.21.15.169:0/1083894103 >> v2:172.21.15.122:3300/0 conn(0x7f9bc40b4d10 0x7f9bc40cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc2ffd640 1 -- 172.21.15.169:0/1083894103 >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 msgr2=0x7f9bc415fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc2ffd640 1 --2- 172.21.15.169:0/1083894103 >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 0x7f9bc415fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc2ffd640 1 -- 172.21.15.169:0/1083894103 >> v2:172.21.15.122:3300/0 conn(0x7f9bc40b4d10 msgr2=0x7f9bc40cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc2ffd640 1 --2- 172.21.15.169:0/1083894103 >> v2:172.21.15.122:3300/0 conn(0x7f9bc40b4d10 0x7f9bc40cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc2ffd640 1 -- 172.21.15.169:0/1083894103 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40d1d90 con 0x7f9bc40d0ac0 2024-03-06T07:21:31.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.332+0000 7f9bc37fe640 1 --2- 172.21.15.169:0/1083894103 >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 0x7f9bc415fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc2ffd640 1 --2- 172.21.15.169:0/1083894103 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40d0ac0 0x7f9bc40b47d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40b6850 tx=0x7f9bb802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81eef0c654493c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc27fc640 1 -- 172.21.15.169:0/1083894103 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb80315a0 con 0x7f9bc40d0ac0 2024-03-06T07:21:31.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc27fc640 1 -- 172.21.15.169:0/1083894103 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb8037070 con 0x7f9bc40d0ac0 2024-03-06T07:21:31.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc27fc640 1 -- 172.21.15.169:0/1083894103 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8031960 con 0x7f9bc40d0ac0 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/1083894103 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40d0ac0 msgr2=0x7f9bc40b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/1083894103 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40d0ac0 0x7f9bc40b47d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40b6850 tx=0x7f9bb802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/1083894103 shutdown_connections 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/1083894103 >> v2:172.21.15.122:3300/0 conn(0x7f9bc40b4d10 0x7f9bc40cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/1083894103 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40d0ac0 0x7f9bc40b47d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/1083894103 >> v2:172.21.15.169:3300/0 conn(0x7f9bc40d06e0 0x7f9bc415fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/1083894103 >> 172.21.15.169:0/1083894103 conn(0x7f9bc40d3cd0 msgr2=0x7f9bc40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/1083894103 shutdown_connections 2024-03-06T07:21:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.333+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/1083894103 wait complete. 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.334+0000 7f9bc9c9d640 1 Processor -- start 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 -- start start 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 0x7f9bc414ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bc40d06e0 0x7f9bc40b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bc40d0ac0 0x7f9bc40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40b44d0 con 0x7f9bc40d0ac0 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40ce990 con 0x7f9bc40d06e0 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc9c9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40d0f00 con 0x7f9bc40b4d10 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.335+0000 7f9bc37fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 0x7f9bc414ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bc40d06e0 0x7f9bc40b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 0x7f9bc414ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.169:42102/0 (socket says 172.21.15.169:42102) 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 -- 172.21.15.169:0/2886512285 learned_addr learned my addr 172.21.15.169:0/2886512285 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-06T07:21:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc3fff640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bc40d0ac0 0x7f9bc40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 -- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bc40d06e0 msgr2=0x7f9bc40b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bc40d06e0 0x7f9bc40b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 -- 172.21.15.169:0/2886512285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bc40d0ac0 msgr2=0x7f9bc40b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bc40d0ac0 0x7f9bc40b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 -- 172.21.15.169:0/2886512285 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40ccb20 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc2ffd640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bc40d06e0 0x7f9bc40b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc37fe640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 0x7f9bc414ac90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400e9e0 tx=0x7f9bb400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4017070 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb400c9d0 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400cd80 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc40cce00 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc40cd3c0 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9bb401a070 con 0x7f9bc40b4d10 2024-03-06T07:21:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.336+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9bb40065a0 con 0x7f9bc40b4d10 2024-03-06T07:21:31.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.337+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc415f970 con 0x7f9bc40b4d10 2024-03-06T07:21:31.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.339+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9bc415f970 con 0x7f9bc40b4d10 2024-03-06T07:21:31.402 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.400+0000 7f1d64e2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.411+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1fd2b82a-556e-4d70-bde1-39ac144dca46", "id": 7} v 0) v1 -- 0x7f9bc40a1c80 con 0x7f9bc40b4d10 2024-03-06T07:21:31.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.417+0000 7f9babfff640 1 -- 172.21.15.169:0/2886512285 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1fd2b82a-556e-4d70-bde1-39ac144dca46", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9bb40067f0 con 0x7f9bc40b4d10 2024-03-06T07:21:31.420 INFO:teuthology.orchestra.run.smithi169.stdout:7 2024-03-06T07:21:31.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 msgr2=0x7f9bc414ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 0x7f9bc414ac90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400e9e0 tx=0x7f9bb400eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 shutdown_connections 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bc40d0ac0 0x7f9bc40b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bc40d06e0 0x7f9bc40b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 --2- 172.21.15.169:0/2886512285 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bc40b4d10 0x7f9bc414ac90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 >> 172.21.15.169:0/2886512285 conn(0x7f9bc40d3cd0 msgr2=0x7f9bc40d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 shutdown_connections 2024-03-06T07:21:31.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:21:31.418+0000 7f9bc9c9d640 1 -- 172.21.15.169:0/2886512285 wait complete. 2024-03-06T07:21:31.432 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:21:31.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:21:31.434 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:21:31.434 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:21:31.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:21:31.436 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:21:31.437 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:21:31.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:21:31.439 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:21:31.439 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:21:31.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:21:31.441 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:21:31.441 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:21:31.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:21:31.453 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.452+0000 7f1d64e2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:31.475 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:21:31.475 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:21:31.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:21:31.482 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:21:31.482 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:21:31.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:21:31.484 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:21:31.484 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:21:31.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:21:31.487 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:21:31.487 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:31.589 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.587+0000 7f1d64e2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:31.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.599+0000 7fc5e7847640 1 Processor -- start 2024-03-06T07:21:31.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.600+0000 7fc5e7847640 1 -- start start 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.600+0000 7fc5e7847640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 0x7fc5e00afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.600+0000 7fc5e7847640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 0x7fc5e015bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.600+0000 7fc5e7847640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e00b4e30 con 0x7fc5e015ab00 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.600+0000 7fc5e7847640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00b0060 con 0x7fc5e00af6c0 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.600+0000 7fc5e7847640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00b5110 con 0x7fc5e015bb10 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e4de2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 0x7fc5e00afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e5de4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 0x7fc5e015bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e55e3640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc5e015ab00 0x7fc5e00b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:42226/0 (socket says 172.21.15.122:42226) 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e5de4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 0x7fc5e015bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44302/0 (socket says 172.21.15.122:44302) 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e4de2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 0x7fc5e00afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46204/0 (socket says 172.21.15.122:46204) 2024-03-06T07:21:31.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e55e3640 1 -- 172.21.15.122:0/4035549736 learned_addr learned my addr 172.21.15.122:0/4035549736 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:31.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e5de4640 1 -- 172.21.15.122:0/4035549736 >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 msgr2=0x7fc5e00afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e5de4640 1 --2- 172.21.15.122:0/4035549736 >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 0x7fc5e00afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e5de4640 1 -- 172.21.15.122:0/4035549736 >> v1:172.21.15.169:6789/0 conn(0x7fc5e015ab00 legacy=0x7fc5e00b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e5de4640 1 -- 172.21.15.122:0/4035549736 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e015b310 con 0x7fc5e015bb10 2024-03-06T07:21:31.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.601+0000 7fc5e4de2640 1 --2- 172.21.15.122:0/4035549736 >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 0x7fc5e00afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5e5de4640 1 --2- 172.21.15.122:0/4035549736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 0x7fc5e015bfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009d90 tx=0x7fc5d802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a35bafaf238d6de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5d7fff640 1 -- 172.21.15.122:0/4035549736 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8039070 con 0x7fc5e015bb10 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5d7fff640 1 -- 172.21.15.122:0/4035549736 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d802cc50 con 0x7fc5e015bb10 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5d7fff640 1 -- 172.21.15.122:0/4035549736 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8034440 con 0x7fc5e015bb10 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5e7847640 1 -- 172.21.15.122:0/4035549736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 msgr2=0x7fc5e015bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5e7847640 1 --2- 172.21.15.122:0/4035549736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 0x7fc5e015bfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009d90 tx=0x7fc5d802bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5e7847640 1 -- 172.21.15.122:0/4035549736 shutdown_connections 2024-03-06T07:21:31.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5e7847640 1 --2- 172.21.15.122:0/4035549736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e015bb10 0x7fc5e015bfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.602+0000 7fc5e7847640 1 --2- 172.21.15.122:0/4035549736 >> v2:172.21.15.169:3300/0 conn(0x7fc5e00af6c0 0x7fc5e00afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.603+0000 7fc5e7847640 1 -- 172.21.15.122:0/4035549736 >> 172.21.15.122:0/4035549736 conn(0x7fc5e006c0c0 msgr2=0x7fc5e006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.603+0000 7fc5e7847640 1 -- 172.21.15.122:0/4035549736 shutdown_connections 2024-03-06T07:21:31.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.603+0000 7fc5e7847640 1 -- 172.21.15.122:0/4035549736 wait complete. 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 Processor -- start 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 -- start start 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e00af6c0 0x7fc5e00cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e015ab00 0x7fc5e01545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 0x7fc5e0155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e0071b20 con 0x7fc5e015bb10 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00b4ac0 con 0x7fc5e015ab00 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.604+0000 7fc5e7847640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e0071ca0 con 0x7fc5e00af6c0 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e5de4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 0x7fc5e0155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e5de4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 0x7fc5e0155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52336/0 (socket says 172.21.15.122:52336) 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e5de4640 1 -- 172.21.15.122:0/918455597 learned_addr learned my addr 172.21.15.122:0/918455597 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e015ab00 0x7fc5e01545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e55e3640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e00af6c0 0x7fc5e00cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 -- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e00af6c0 msgr2=0x7fc5e00cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e00af6c0 0x7fc5e00cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 -- 172.21.15.122:0/918455597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 msgr2=0x7fc5e0155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 0x7fc5e0155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 -- 172.21.15.122:0/918455597 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00cffe0 con 0x7fc5e015ab00 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e5de4640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 0x7fc5e0155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e4de2640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e015ab00 0x7fc5e01545c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00e930 tx=0x7fc5dc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc017070 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc00ca00 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc00cdd0 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.605+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e0154b00 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.606+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e00ce010 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.606+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc5dc01a070 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.606+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc5dc013b40 con 0x7fc5e015ab00 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.607+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e004f330 con 0x7fc5e015ab00 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.610+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc5dc009380 con 0x7fc5e015ab00 2024-03-06T07:21:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.672+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5e015c1c0 con 0x7fc5e015ab00 2024-03-06T07:21:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.673+0000 7fc5d5ffb640 1 -- 172.21.15.122:0/918455597 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc5dc0064d0 con 0x7fc5e015ab00 2024-03-06T07:21:31.675 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:31.675 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":10,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:31.414268+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:31.676 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.672+0000 7f440242b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e015ab00 msgr2=0x7fc5e01545c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e015ab00 0x7fc5e01545c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00e930 tx=0x7fc5dc00edf0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 shutdown_connections 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5e015bb10 0x7fc5e0155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5e015ab00 0x7fc5e01545c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 --2- 172.21.15.122:0/918455597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5e00af6c0 0x7fc5e00cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 >> 172.21.15.122:0/918455597 conn(0x7fc5e006c0c0 msgr2=0x7fc5e00d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 shutdown_connections 2024-03-06T07:21:31.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:31.674+0000 7fc5e7847640 1 -- 172.21.15.122:0/918455597 wait complete. 2024-03-06T07:21:31.689 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:21:31.689 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:21:31.690 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.687+0000 7f1d64e2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:31.810 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.808+0000 7f440242b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:31.845 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.843+0000 7f1d64e2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:31.886 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:31.885+0000 7f1d64e2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:31.920 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.918+0000 7f440242b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:31.967 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:31.964+0000 7f440242b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:32.041 INFO:tasks.ceph.osd.0.smithi122.stderr:2024-03-06T07:21:32.039+0000 7f5ce3a22640 -1 Falling back to public interface 2024-03-06T07:21:32.042 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:32.040+0000 7f1d64e2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:32.046 INFO:tasks.ceph.osd.1.smithi122.stderr:2024-03-06T07:21:32.044+0000 7f3910c1d640 -1 Falling back to public interface 2024-03-06T07:21:32.055 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-03-06T07:21:32.053+0000 7f73d8619640 -1 Falling back to public interface 2024-03-06T07:21:32.070 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-03-06T07:21:32.068+0000 7f037ec99640 -1 Falling back to public interface 2024-03-06T07:21:32.080 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:32.078+0000 7f440242b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:32.088 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-06T07:21:32.085+0000 7f436b60d640 -1 Falling back to public interface 2024-03-06T07:21:32.088 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-06T07:21:32.086+0000 7f3adff06640 -1 Falling back to public interface 2024-03-06T07:21:32.088 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-06T07:21:32.086+0000 7f3adeeb5640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2024-03-06T07:21:32.097 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-06T07:21:32.094+0000 7f75aba13640 -1 Falling back to public interface 2024-03-06T07:21:32.105 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-06T07:21:32.102+0000 7f83d6ea0640 -1 Falling back to public interface 2024-03-06T07:21:32.183 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:32.180+0000 7f440242b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:32.338 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:32.336+0000 7f440242b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:32.382 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:32.380+0000 7f440242b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:32.495 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:32.539 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:32.537+0000 7f440242b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:32.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.581+0000 7f1237095640 1 Processor -- start 2024-03-06T07:21:32.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.581+0000 7f1237095640 1 -- start start 2024-03-06T07:21:32.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1237095640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12300d08e0 0x7f12300d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1237095640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d0cc0 0x7f12300cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1237095640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12300d2810 con 0x7f12300d1950 2024-03-06T07:21:32.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1237095640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12300cf210 con 0x7f12300d08e0 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1237095640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300d2a90 con 0x7f12300d0cc0 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1235632640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f12300d1950 0x7f12301611a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:60540/0 (socket says 172.21.15.122:60540) 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1235632640 1 -- 172.21.15.122:0/1697454139 learned_addr learned my addr 172.21.15.122:0/1697454139 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122ffff640 1 --2- 172.21.15.122:0/1697454139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d0cc0 0x7f12300cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1234e31640 1 --2- 172.21.15.122:0/1697454139 >> v2:172.21.15.169:3300/0 conn(0x7f12300d08e0 0x7f12300d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122f7fe640 1 -- 172.21.15.122:0/1697454139 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843406775 0 0) 0x7f1228001490 con 0x7f12300d1950 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122f7fe640 1 -- 172.21.15.122:0/1697454139 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1220003520 con 0x7f12300d1950 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122ffff640 1 -- 172.21.15.122:0/1697454139 >> v2:172.21.15.169:3300/0 conn(0x7f12300d08e0 msgr2=0x7f12300d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122ffff640 1 --2- 172.21.15.122:0/1697454139 >> v2:172.21.15.169:3300/0 conn(0x7f12300d08e0 0x7f12300d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122ffff640 1 -- 172.21.15.122:0/1697454139 >> v1:172.21.15.122:6789/0 conn(0x7f12300d1950 legacy=0x7f12301611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f122ffff640 1 -- 172.21.15.122:0/1697454139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12300cebc0 con 0x7f12300d0cc0 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.582+0000 7f1234e31640 1 --2- 172.21.15.122:0/1697454139 >> v2:172.21.15.169:3300/0 conn(0x7f12300d08e0 0x7f12300d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.583+0000 7f122ffff640 1 --2- 172.21.15.122:0/1697454139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d0cc0 0x7f12300cf6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f121c0097a0 tx=0x7f121c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae4211622e28ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.583+0000 7f122f7fe640 1 -- 172.21.15.122:0/1697454139 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c0315a0 con 0x7f12300d0cc0 2024-03-06T07:21:32.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.583+0000 7f122f7fe640 1 -- 172.21.15.122:0/1697454139 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c037070 con 0x7f12300d0cc0 2024-03-06T07:21:32.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.583+0000 7f122f7fe640 1 -- 172.21.15.122:0/1697454139 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c031960 con 0x7f12300d0cc0 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 -- 172.21.15.122:0/1697454139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d0cc0 msgr2=0x7f12300cf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 --2- 172.21.15.122:0/1697454139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d0cc0 0x7f12300cf6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f121c0097a0 tx=0x7f121c02b840 comp rx=0 tx=0).stop 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 -- 172.21.15.122:0/1697454139 shutdown_connections 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 --2- 172.21.15.122:0/1697454139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d0cc0 0x7f12300cf6a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 --2- 172.21.15.122:0/1697454139 >> v2:172.21.15.169:3300/0 conn(0x7f12300d08e0 0x7f12300d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 -- 172.21.15.122:0/1697454139 >> 172.21.15.122:0/1697454139 conn(0x7f123009d050 msgr2=0x7f12300d3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 -- 172.21.15.122:0/1697454139 shutdown_connections 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.584+0000 7f1237095640 1 -- 172.21.15.122:0/1697454139 wait complete. 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.586+0000 7f1237095640 1 Processor -- start 2024-03-06T07:21:32.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.586+0000 7f1237095640 1 -- start start 2024-03-06T07:21:32.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1237095640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d08e0 0x7f1230160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1237095640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12300d0cc0 0x7f12300c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1237095640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 0x7f12300c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1237095640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12300ce9a0 con 0x7f12300d1950 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1237095640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12300ce820 con 0x7f12300d0cc0 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1237095640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300cfbe0 con 0x7f12300d08e0 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 0x7f12300c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 0x7f12300c89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52352/0 (socket says 172.21.15.122:52352) 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 -- 172.21.15.122:0/2885298366 learned_addr learned my addr 172.21.15.122:0/2885298366 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1234e31640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d08e0 0x7f1230160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f122ffff640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12300d0cc0 0x7f12300c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 -- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d08e0 msgr2=0x7f1230160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d08e0 0x7f1230160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 -- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12300d0cc0 msgr2=0x7f12300c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12300d0cc0 0x7f12300c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 -- 172.21.15.122:0/2885298366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123015f1c0 con 0x7f12300d1950 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.587+0000 7f1235632640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 0x7f12300c89d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f122800cb80 tx=0x7f122800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228017070 con 0x7f12300d1950 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122800ad40 con 0x7f12300d1950 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228012620 con 0x7f12300d1950 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123015f4a0 con 0x7f12300d1950 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123015fa60 con 0x7f12300d1950 2024-03-06T07:21:32.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f122801a070 con 0x7f12300d1950 2024-03-06T07:21:32.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.588+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f1228007740 con 0x7f12300d1950 2024-03-06T07:21:32.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.589+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123004f330 con 0x7f12300d1950 2024-03-06T07:21:32.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.591+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1228021020 con 0x7f12300d1950 2024-03-06T07:21:32.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.653+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12300a1a70 con 0x7f12300d1950 2024-03-06T07:21:32.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.654+0000 7f122d7fa640 1 -- 172.21.15.122:0/2885298366 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f12300d2810 con 0x7f12300d1950 2024-03-06T07:21:32.656 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-06T07:21:32.656 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:31.414268+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:32.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 msgr2=0x7f12300c89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 0x7f12300c89d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f122800cb80 tx=0x7f122800a430 comp rx=0 tx=0).stop 2024-03-06T07:21:32.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 shutdown_connections 2024-03-06T07:21:32.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12300d1950 0x7f12300c89d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12300d0cc0 0x7f12300c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 --2- 172.21.15.122:0/2885298366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12300d08e0 0x7f1230160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 >> 172.21.15.122:0/2885298366 conn(0x7f123009d050 msgr2=0x7f12300b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 shutdown_connections 2024-03-06T07:21:32.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:32.655+0000 7f1237095640 1 -- 172.21.15.122:0/2885298366 wait complete. 2024-03-06T07:21:32.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:32.867 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:32.865+0000 7f1d64e2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:33.141 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:33.139+0000 7f1d64e2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:33.193 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:33.191+0000 7f1d64e2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:33.308 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:33.306+0000 7f1d64e2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:33.381 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:33.378+0000 7f440242b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:33.663 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:33.660+0000 7f440242b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:33.714 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:33.712+0000 7f440242b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:33.834 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-06T07:21:33.831+0000 7f440242b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:34.333 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:34.331+0000 7f1d0531a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:21:34.335 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-06T07:21:34.334+0000 7f1d0531a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:21:35.497 INFO:tasks.ceph.osd.0.smithi122.stderr:2024-03-06T07:21:35.495+0000 7f5ce3a22640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:21:35.536 INFO:tasks.ceph.osd.1.smithi122.stderr:2024-03-06T07:21:35.534+0000 7f3910c1d640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:21:35.563 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-03-06T07:21:35.561+0000 7f037ec99640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:21:35.573 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-03-06T07:21:35.571+0000 7f73d8619640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:21:35.573 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-06T07:21:35.571+0000 7f3adff06640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:21:35.624 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-06T07:21:35.622+0000 7f436b60d640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:21:35.671 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-06T07:21:35.668+0000 7f83d6ea0640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:21:35.671 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-06T07:21:35.668+0000 7f75aba13640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:21:39.247 INFO:tasks.ceph.osd.0.smithi122.stderr:2024-03-06T07:21:39.245+0000 7f5cdead8640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:21:39.261 INFO:tasks.ceph.osd.0.smithi122.stderr:2024-03-06T07:21:39.259+0000 7f5cd710a640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.309 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-03-06T07:21:39.307+0000 7f73d385a640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:21:39.322 INFO:tasks.ceph.osd.1.smithi122.stderr:2024-03-06T07:21:39.320+0000 7f390bcdb640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:21:39.322 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-03-06T07:21:39.321+0000 7f73cbe8c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.333 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-03-06T07:21:39.332+0000 7f037a40c640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:21:39.335 INFO:tasks.ceph.osd.1.smithi122.stderr:2024-03-06T07:21:39.334+0000 7f390430d640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.343 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-03-06T07:21:39.342+0000 7f0372a3e640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.444 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-06T07:21:39.442+0000 7f75a6b25640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:21:39.458 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-06T07:21:39.456+0000 7f4366717640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:21:39.458 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-06T07:21:39.456+0000 7f759f157640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.464 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-06T07:21:39.461+0000 7f3adb679640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:21:39.471 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:39.472 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-06T07:21:39.469+0000 7f83d2613640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:21:39.473 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-06T07:21:39.471+0000 7f435ed49640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.473 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-06T07:21:39.471+0000 7f3ad3cab640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.489 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-06T07:21:39.487+0000 7f83cac45640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:39.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.558+0000 7f9d9c57a640 1 Processor -- start 2024-03-06T07:21:39.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.558+0000 7f9d9c57a640 1 -- start start 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.559+0000 7f9d9c57a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d940d0790 0x7f9d940b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.559+0000 7f9d9c57a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9415c350 con 0x7f9d9415bf70 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.559+0000 7f9d9c57a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d940d2810 con 0x7f9d940ceab0 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.559+0000 7f9d9c57a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940d25c0 con 0x7f9d940d0790 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99b15640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d940d0790 0x7f9d940b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99b15640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d940d0790 0x7f9d940b19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48648/0 (socket says 172.21.15.122:48648) 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99b15640 1 -- 172.21.15.122:0/3071336092 learned_addr learned my addr 172.21.15.122:0/3071336092 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:39.562 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d9a316640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9d9415bf70 0x7f9d940d0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:60816/0 (socket says 172.21.15.122:60816) 2024-03-06T07:21:39.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99b15640 1 -- 172.21.15.122:0/3071336092 >> v1:172.21.15.169:6789/0 conn(0x7f9d940ceab0 legacy=0x7f9d940b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99b15640 1 -- 172.21.15.122:0/3071336092 >> v1:172.21.15.122:6789/0 conn(0x7f9d9415bf70 legacy=0x7f9d940d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99b15640 1 -- 172.21.15.122:0/3071336092 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940b11e0 con 0x7f9d940d0790 2024-03-06T07:21:39.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.560+0000 7f9d99314640 1 -- 172.21.15.122:0/3071336092 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866541697 0 0) 0x7f9d9415c350 con 0x7f9d9415bf70 2024-03-06T07:21:39.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d99b15640 1 --2- 172.21.15.122:0/3071336092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940d0790 0x7f9d940b19e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009870 tx=0x7f9d8402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3dbfefa111ad13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d99314640 1 -- 172.21.15.122:0/3071336092 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84034070 con 0x7f9d940d0790 2024-03-06T07:21:39.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d99314640 1 -- 172.21.15.122:0/3071336092 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d84030860 con 0x7f9d940d0790 2024-03-06T07:21:39.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d99314640 1 -- 172.21.15.122:0/3071336092 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84030c30 con 0x7f9d940d0790 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 -- 172.21.15.122:0/3071336092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940d0790 msgr2=0x7f9d940b19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/3071336092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940d0790 0x7f9d940b19e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009870 tx=0x7f9d8402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 -- 172.21.15.122:0/3071336092 shutdown_connections 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/3071336092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940d0790 0x7f9d940b19e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 -- 172.21.15.122:0/3071336092 >> 172.21.15.122:0/3071336092 conn(0x7f9d940d1aa0 msgr2=0x7f9d940d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 -- 172.21.15.122:0/3071336092 shutdown_connections 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.561+0000 7f9d9c57a640 1 -- 172.21.15.122:0/3071336092 wait complete. 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.562+0000 7f9d9c57a640 1 Processor -- start 2024-03-06T07:21:39.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 -- start start 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 0x7f9d940c4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940d0790 0x7f9d94158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9415bf70 0x7f9d940c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940b1550 con 0x7f9d940ceab0 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940b0f90 con 0x7f9d9415bf70 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9c57a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940cf880 con 0x7f9d940d0790 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9a316640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 0x7f9d940c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9a316640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 0x7f9d940c4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48650/0 (socket says 172.21.15.122:48650) 2024-03-06T07:21:39.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9a316640 1 -- 172.21.15.122:0/66173380 learned_addr learned my addr 172.21.15.122:0/66173380 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d99b15640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940d0790 0x7f9d94158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.563+0000 7f9d9ab17640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9415bf70 0x7f9d940c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9a316640 1 -- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940d0790 msgr2=0x7f9d94158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9a316640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940d0790 0x7f9d94158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9a316640 1 -- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9415bf70 msgr2=0x7f9d940c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9a316640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9415bf70 0x7f9d940c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9a316640 1 -- 172.21.15.122:0/66173380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940c4ab0 con 0x7f9d940ceab0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9a316640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 0x7f9d940c4570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00c780 tx=0x7f9d8c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c018070 con 0x7f9d940ceab0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00a640 con 0x7f9d940ceab0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00aa70 con 0x7f9d940ceab0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d941589c0 con 0x7f9d940ceab0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.564+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9406c570 con 0x7f9d940ceab0 2024-03-06T07:21:39.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.565+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9404f330 con 0x7f9d940ceab0 2024-03-06T07:21:39.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.565+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f9d8c01b070 con 0x7f9d940ceab0 2024-03-06T07:21:39.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.565+0000 7f9d8affd640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7806b470 0x7f9d7806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.566+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f9d8c088770 con 0x7f9d940ceab0 2024-03-06T07:21:39.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.566+0000 7f9d99b15640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7806b470 0x7f9d7806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.566+0000 7f9d99b15640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7806b470 0x7f9d7806d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9d84036000 tx=0x7f9d84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.570+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d8c057ea0 con 0x7f9d940ceab0 2024-03-06T07:21:39.744 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.742+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d940af140 con 0x7f9d940ceab0 2024-03-06T07:21:39.745 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.743+0000 7f9d8affd640 1 -- 172.21.15.122:0/66173380 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f9d8c05bd00 con 0x7f9d940ceab0 2024-03-06T07:21:39.745 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-06T07:21:39.745 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":12,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:37.335638+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:39.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7806b470 msgr2=0x7f9d7806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7806b470 0x7f9d7806d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9d84036000 tx=0x7f9d84008030 comp rx=0 tx=0).stop 2024-03-06T07:21:39.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 msgr2=0x7f9d940c4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 0x7f9d940c4570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00c780 tx=0x7f9d8c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 shutdown_connections 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7806b470 0x7f9d7806d930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9415bf70 0x7f9d940c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940d0790 0x7f9d94158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 --2- 172.21.15.122:0/66173380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d940ceab0 0x7f9d940c4570 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 >> 172.21.15.122:0/66173380 conn(0x7f9d940d1aa0 msgr2=0x7f9d940d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 shutdown_connections 2024-03-06T07:21:39.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:39.745+0000 7f9d9c57a640 1 -- 172.21.15.122:0/66173380 wait complete. 2024-03-06T07:21:39.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:46.565 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:46.664 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 Processor -- start 2024-03-06T07:21:46.664 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 -- start start 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8ff40d0790 0x7f8ff40b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8ff40ceab0 0x7f8ff40b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff415c350 con 0x7f8ff415bf70 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40d25c0 con 0x7f8ff40d0790 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.662+0000 7f8ff91f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40b0f90 con 0x7f8ff40ceab0 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8febfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8ff40d0790 0x7f8ff40b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8ff2d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8ff415bf70 0x7f8ff40d0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54612/0 (socket says 172.21.15.122:54612) 2024-03-06T07:21:46.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8febfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8ff40d0790 0x7f8ff40b19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44558/0 (socket says 172.21.15.122:44558) 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8ff2d9d640 1 -- 172.21.15.122:0/3474472016 learned_addr learned my addr 172.21.15.122:0/3474472016 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8ff359e640 1 --2- 172.21.15.122:0/3474472016 >> v2:172.21.15.169:3300/0 conn(0x7f8ff40ceab0 0x7f8ff40b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8febfff640 1 -- 172.21.15.122:0/3474472016 >> v2:172.21.15.169:3300/0 conn(0x7f8ff40ceab0 msgr2=0x7f8ff40b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8febfff640 1 --2- 172.21.15.122:0/3474472016 >> v2:172.21.15.169:3300/0 conn(0x7f8ff40ceab0 0x7f8ff40b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8febfff640 1 -- 172.21.15.122:0/3474472016 >> v1:172.21.15.122:6789/0 conn(0x7f8ff415bf70 legacy=0x7f8ff40d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8febfff640 1 -- 172.21.15.122:0/3474472016 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40d2810 con 0x7f8ff40d0790 2024-03-06T07:21:46.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.663+0000 7f8ff259c640 1 -- 172.21.15.122:0/3474472016 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435516668 0 0) 0x7f8ff415c350 con 0x7f8ff415bf70 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8febfff640 1 --2- 172.21.15.122:0/3474472016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff40d0790 0x7f8ff40b19e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8ff415f9a0 tx=0x7f8fdc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bd021d18b9304b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff259c640 1 -- 172.21.15.122:0/3474472016 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc0315a0 con 0x7f8ff40d0790 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff259c640 1 -- 172.21.15.122:0/3474472016 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc037070 con 0x7f8ff40d0790 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff259c640 1 -- 172.21.15.122:0/3474472016 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc031960 con 0x7f8ff40d0790 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 -- 172.21.15.122:0/3474472016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff40d0790 msgr2=0x7f8ff40b19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/3474472016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff40d0790 0x7f8ff40b19e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8ff415f9a0 tx=0x7f8fdc02b840 comp rx=0 tx=0).stop 2024-03-06T07:21:46.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 -- 172.21.15.122:0/3474472016 shutdown_connections 2024-03-06T07:21:46.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/3474472016 >> v2:172.21.15.169:3300/0 conn(0x7f8ff40ceab0 0x7f8ff40b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/3474472016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff40d0790 0x7f8ff40b19e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 -- 172.21.15.122:0/3474472016 >> 172.21.15.122:0/3474472016 conn(0x7f8ff40d1aa0 msgr2=0x7f8ff40d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 -- 172.21.15.122:0/3474472016 shutdown_connections 2024-03-06T07:21:46.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.664+0000 7f8ff91f6640 1 -- 172.21.15.122:0/3474472016 wait complete. 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 Processor -- start 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 -- start start 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ff40ceab0 0x7f8ff40cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ff40d0790 0x7f8ff40c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 0x7f8ff41f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40cd830 con 0x7f8ff415bf70 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b1550 con 0x7f8ff40ceab0 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.666+0000 7f8ff91f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40cd9b0 con 0x7f8ff40d0790 2024-03-06T07:21:46.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 0x7f8ff41f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 0x7f8ff41f1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44564/0 (socket says 172.21.15.122:44564) 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 -- 172.21.15.122:0/2185325508 learned_addr learned my addr 172.21.15.122:0/2185325508 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8febfff640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ff40d0790 0x7f8ff40c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff2d9d640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ff40ceab0 0x7f8ff40cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 -- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ff40d0790 msgr2=0x7f8ff40c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ff40d0790 0x7f8ff40c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 -- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ff40ceab0 msgr2=0x7f8ff40cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ff40ceab0 0x7f8ff40cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 -- 172.21.15.122:0/2185325508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff41f2240 con 0x7f8ff415bf70 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff2d9d640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ff40ceab0 0x7f8ff40cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff359e640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 0x7f8ff41f1730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8fec007e90 tx=0x7f8fec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec014420 con 0x7f8ff415bf70 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec019070 con 0x7f8ff415bf70 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff41f2520 con 0x7f8ff415bf70 2024-03-06T07:21:46.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec014800 con 0x7f8ff415bf70 2024-03-06T07:21:46.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.667+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff40c8fd0 con 0x7f8ff415bf70 2024-03-06T07:21:46.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.668+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff404f330 con 0x7f8ff415bf70 2024-03-06T07:21:46.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.669+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fec0149a0 con 0x7f8ff415bf70 2024-03-06T07:21:46.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.669+0000 7f8feb7fe640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fd406b4d0 0x7f8fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.669+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8fec08a570 con 0x7f8ff415bf70 2024-03-06T07:21:46.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.669+0000 7f8ff2d9d640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fd406b4d0 0x7f8fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.670+0000 7f8ff2d9d640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fd406b4d0 0x7f8fd406d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4000f60 tx=0x7f8fe4007cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.674+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fec0597d0 con 0x7f8ff415bf70 2024-03-06T07:21:46.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.850+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ff40b5390 con 0x7f8ff415bf70 2024-03-06T07:21:46.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.851+0000 7f8feb7fe640 1 -- 172.21.15.122:0/2185325508 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12884 (secure 0 0 0) 0x7f8fec0590d0 con 0x7f8ff415bf70 2024-03-06T07:21:46.853 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-06T07:21:46.854 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":16,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:43.507224+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:46.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fd406b4d0 msgr2=0x7f8fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fd406b4d0 0x7f8fd406d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4000f60 tx=0x7f8fe4007cc0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 msgr2=0x7f8ff41f1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 0x7f8ff41f1730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8fec007e90 tx=0x7f8fec00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 shutdown_connections 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fd406b4d0 0x7f8fd406d990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ff415bf70 0x7f8ff41f1730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ff40d0790 0x7f8ff40c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 --2- 172.21.15.122:0/2185325508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ff40ceab0 0x7f8ff40cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 >> 172.21.15.122:0/2185325508 conn(0x7f8ff40d1aa0 msgr2=0x7f8ff40cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 shutdown_connections 2024-03-06T07:21:46.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-06T07:21:46.853+0000 7f8ff91f6640 1 -- 172.21.15.122:0/2185325508 wait complete. 2024-03-06T07:21:46.865 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:21:46.865 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:21:46.865 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:46.866 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:46.868 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:46.868 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:46.868 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:46.870 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:46.870 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:46.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:46.873 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:46.873 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:46.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:46.875 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:46.875 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:21:46.878 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:46.943 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:21:46.944 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:21:46.958 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:21:46.959 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:21:46.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.981+0000 7f4d0eb0c640 1 Processor -- start 2024-03-06T07:21:46.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.981+0000 7f4d0eb0c640 1 -- start start 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.982+0000 7f4d0eb0c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 0x7f4d080b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.982+0000 7f4d0eb0c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080b3430 con 0x7f4d0815dfe0 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.982+0000 7f4d0eb0c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080b3710 con 0x7f4d080d22b0 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.982+0000 7f4d0eb0c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080b31e0 con 0x7f4d080d2710 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.982+0000 7f4d0c8a8640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4d0815dfe0 0x7f4d080cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54628/0 (socket says 172.21.15.122:54628) 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.982+0000 7f4d0c8a8640 1 -- 172.21.15.122:0/1381693353 learned_addr learned my addr 172.21.15.122:0/1381693353 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 0x7f4d080b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653138205 0 0) 0x7f4d080b3430 con 0x7f4d0815dfe0 2024-03-06T07:21:46.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf8003520 con 0x7f4d0815dfe0 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952286628 0 0) 0x7f4cf8003520 con 0x7f4d0815dfe0 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 msgr2=0x7f4d080b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 0x7f4d080b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 >> v1:172.21.15.169:6789/0 conn(0x7f4d080d22b0 legacy=0x7f4d080b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d080b39f0 con 0x7f4d0815dfe0 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4cfc001670 con 0x7f4d0815dfe0 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 >> v1:172.21.15.122:6789/0 conn(0x7f4d0815dfe0 legacy=0x7f4d080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cf80039e0 0x7f4cf8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 0x7f4cf8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf8004d40 0x7f4cf80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080b31e0 con 0x7f4cf8004d40 2024-03-06T07:21:46.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf800ba70 con 0x7f4cf80039e0 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf800bc70 con 0x7f4d080d2710 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf8004d40 0x7f4cf80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0c8a8640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cf80039e0 0x7f4cf8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d07fff640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 0x7f4cf8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 msgr2=0x7f4cf8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080d2710 0x7f4cf8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cf80039e0 msgr2=0x7f4cf8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cf80039e0 0x7f4cf8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.983+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1381693353 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf8003520 con 0x7f4cf8004d40 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.984+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf8004d40 0x7f4cf80051c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009fc0 tx=0x7f4cf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ceaefe914af3be8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.984+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00039070 con 0x7f4cf8004d40 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.984+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0002cc20 con 0x7f4cf8004d40 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.984+0000 7f4d077fe640 1 -- 172.21.15.122:0/1381693353 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00009c10 con 0x7f4cf8004d40 2024-03-06T07:21:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1381693353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf8004d40 msgr2=0x7f4cf80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1381693353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf8004d40 0x7f4cf80051c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009fc0 tx=0x7f4cf8005830 comp rx=0 tx=0).stop 2024-03-06T07:21:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1381693353 reap_dead start 2024-03-06T07:21:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1381693353 shutdown_connections 2024-03-06T07:21:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1381693353 >> 172.21.15.122:0/1381693353 conn(0x7f4d080d3cd0 msgr2=0x7f4d080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1381693353 shutdown_connections 2024-03-06T07:21:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.985+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1381693353 wait complete. 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.986+0000 7f4d0eb0c640 1 Processor -- start 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.986+0000 7f4d0eb0c640 1 -- start start 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08157040 0x7f4d080ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080cd470 0x7f4d080b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 0x7f4d081480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080cfa00 con 0x7f4d080b98d0 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080b3d60 con 0x7f4d08157040 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080d1c70 con 0x7f4d080cd470 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 0x7f4d081480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 0x7f4d081480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44610/0 (socket says 172.21.15.122:44610) 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1266642272 learned_addr learned my addr 172.21.15.122:0/1266642272 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:46.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d07fff640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080cd470 0x7f4d080b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0c8a8640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08157040 0x7f4d080ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080cd470 msgr2=0x7f4d080b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080cd470 0x7f4d080b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08157040 msgr2=0x7f4d080ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08157040 0x7f4d080ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 -- 172.21.15.122:0/1266642272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d080cbe90 con 0x7f4d080b98d0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d07fff640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080cd470 0x7f4d080b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0c8a8640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08157040 0x7f4d080ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0d0a9640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 0x7f4d081480d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000e230 tx=0x7f4d00009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00039030 con 0x7f4d080b98d0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d080cc170 con 0x7f4d080b98d0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0003a040 con 0x7f4d080b98d0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00009c10 con 0x7f4d080b98d0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.987+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d080cc730 con 0x7f4d080b98d0 2024-03-06T07:21:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.988+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d080b54c0 con 0x7f4d080b98d0 2024-03-06T07:21:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.989+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d00045020 con 0x7f4d080b98d0 2024-03-06T07:21:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.989+0000 7f4d057fa640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b500 0x7f4cec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.989+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4d000ab9c0 con 0x7f4d080b98d0 2024-03-06T07:21:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.989+0000 7f4d0c8a8640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b500 0x7f4cec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.990+0000 7f4d0c8a8640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b500 0x7f4cec06d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc000c00 tx=0x7f4cfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:46.994+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d080b54c0 con 0x7f4d080b98d0 2024-03-06T07:21:47.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.166+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d080d1a90 con 0x7f4d080b98d0 2024-03-06T07:21:47.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.167+0000 7f4d057fa640 1 -- 172.21.15.122:0/1266642272 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12884 (secure 0 0 0) 0x7f4d080d1a90 con 0x7f4d080b98d0 2024-03-06T07:21:47.168 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":16,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:43.507224+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:47.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b500 msgr2=0x7f4cec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b500 0x7f4cec06d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc000c00 tx=0x7f4cfc009040 comp rx=0 tx=0).stop 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 msgr2=0x7f4d081480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 0x7f4d081480d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000e230 tx=0x7f4d00009750 comp rx=0 tx=0).stop 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 shutdown_connections 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b500 0x7f4cec06d9c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080b98d0 0x7f4d081480d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080cd470 0x7f4d080b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 --2- 172.21.15.122:0/1266642272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08157040 0x7f4d080ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.168+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 >> 172.21.15.122:0/1266642272 conn(0x7f4d080d3cd0 msgr2=0x7f4d080b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.169+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 shutdown_connections 2024-03-06T07:21:47.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.169+0000 7f4d0eb0c640 1 -- 172.21.15.122:0/1266642272 wait complete. 2024-03-06T07:21:47.181 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:47.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:47.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.282+0000 7f007120e640 1 Processor -- start 2024-03-06T07:21:47.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.282+0000 7f007120e640 1 -- start start 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f007120e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c15c050 0x7f006c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f007120e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f006c0d0af0 0x7f006c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f007120e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c0cd4b0 con 0x7f006c06fb40 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f007120e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c06ff80 con 0x7f006c0d0af0 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f007120e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0cd790 con 0x7f006c15c050 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f006c0d0af0 0x7f006c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f006c0d0af0 0x7f006c0cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44626/0 (socket says 172.21.15.122:44626) 2024-03-06T07:21:47.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 -- 172.21.15.122:0/2774469329 learned_addr learned my addr 172.21.15.122:0/2774469329 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006ad9d640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c15c050 0x7f006c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 -- 172.21.15.122:0/2774469329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c15c050 msgr2=0x7f006c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c15c050 0x7f006c06f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 -- 172.21.15.122:0/2774469329 >> v1:172.21.15.169:6789/0 conn(0x7f006c06fb40 legacy=0x7f006c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006b59e640 1 -- 172.21.15.122:0/2774469329 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0cda70 con 0x7f006c0d0af0 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.283+0000 7f006ad9d640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c15c050 0x7f006c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.284+0000 7f006b59e640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c0d0af0 0x7f006c0cce70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0064009870 tx=0x7f006402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fcc4e97bd82a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.284+0000 7f0069d9b640 1 -- 172.21.15.122:0/2774469329 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064035020 con 0x7f006c0d0af0 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.284+0000 7f0069d9b640 1 -- 172.21.15.122:0/2774469329 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064031640 con 0x7f006c0d0af0 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.284+0000 7f0069d9b640 1 -- 172.21.15.122:0/2774469329 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006402d040 con 0x7f006c0d0af0 2024-03-06T07:21:47.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 -- 172.21.15.122:0/2774469329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c0d0af0 msgr2=0x7f006c0cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c0d0af0 0x7f006c0cce70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0064009870 tx=0x7f006402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 -- 172.21.15.122:0/2774469329 shutdown_connections 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c0d0af0 0x7f006c0cce70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 --2- 172.21.15.122:0/2774469329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c15c050 0x7f006c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 -- 172.21.15.122:0/2774469329 >> 172.21.15.122:0/2774469329 conn(0x7f006c0b3800 msgr2=0x7f006c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 -- 172.21.15.122:0/2774469329 shutdown_connections 2024-03-06T07:21:47.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.285+0000 7f007120e640 1 -- 172.21.15.122:0/2774469329 wait complete. 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.286+0000 7f007120e640 1 Processor -- start 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.286+0000 7f007120e640 1 -- start start 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f007120e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f006c06fb40 0x7f006c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f007120e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c0d0af0 0x7f006c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f007120e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 0x7f006c0cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f007120e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c165c30 con 0x7f006c15c050 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f007120e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c165ab0 con 0x7f006c06fb40 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f007120e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c165db0 con 0x7f006c0d0af0 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 0x7f006c0cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 0x7f006c0cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44638/0 (socket says 172.21.15.122:44638) 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 -- 172.21.15.122:0/460506539 learned_addr learned my addr 172.21.15.122:0/460506539 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006ad9d640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f006c06fb40 0x7f006c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006a59c640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c0d0af0 0x7f006c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 -- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c0d0af0 msgr2=0x7f006c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c0d0af0 0x7f006c0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 -- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f006c06fb40 msgr2=0x7f006c155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f006c06fb40 0x7f006c155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006b59e640 1 -- 172.21.15.122:0/460506539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0b7000 con 0x7f006c15c050 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.287+0000 7f006ad9d640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f006c06fb40 0x7f006c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.288+0000 7f006b59e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 0x7f006c0cb3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0064009870 tx=0x7f00640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.288+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064040070 con 0x7f006c15c050 2024-03-06T07:21:47.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.288+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00640399f0 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.288+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c0b48f0 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.288+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064035040 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.288+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c0b4eb0 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.289+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c09fe60 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.289+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0064038020 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.289+0000 7f00537fe640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005806b4d0 0x7f005806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.290+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f00640ab7d0 con 0x7f006c15c050 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.290+0000 7f006ad9d640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005806b4d0 0x7f005806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.290+0000 7f006ad9d640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005806b4d0 0x7f005806d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f005c009b40 tx=0x7f005c003a30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.294+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f006407aa60 con 0x7f006c15c050 2024-03-06T07:21:47.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.466+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f006c068ff0 con 0x7f006c15c050 2024-03-06T07:21:47.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.466+0000 7f00537fe640 1 -- 172.21.15.122:0/460506539 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f006407a360 con 0x7f006c15c050 2024-03-06T07:21:47.469 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005806b4d0 msgr2=0x7f005806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005806b4d0 0x7f005806d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f005c009b40 tx=0x7f005c003a30 comp rx=0 tx=0).stop 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 msgr2=0x7f006c0cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 0x7f006c0cb3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0064009870 tx=0x7f00640046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 shutdown_connections 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005806b4d0 0x7f005806d990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f006c15c050 0x7f006c0cb3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f006c0d0af0 0x7f006c0cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 --2- 172.21.15.122:0/460506539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f006c06fb40 0x7f006c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 >> 172.21.15.122:0/460506539 conn(0x7f006c0b3800 msgr2=0x7f006c0aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 shutdown_connections 2024-03-06T07:21:47.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.468+0000 7f007120e640 1 -- 172.21.15.122:0/460506539 wait complete. 2024-03-06T07:21:47.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:47.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.580+0000 7f83c333a640 1 Processor -- start 2024-03-06T07:21:47.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.580+0000 7f83c333a640 1 -- start start 2024-03-06T07:21:47.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c333a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f83bc06b6c0 0x7f83bc072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c333a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f83bc0cd390 0x7f83bc0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c333a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0d0610 con 0x7f83bc15baf0 2024-03-06T07:21:47.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c333a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc072c40 con 0x7f83bc06b6c0 2024-03-06T07:21:47.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c333a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc0d08f0 con 0x7f83bc0cd390 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f83bc06b6c0 0x7f83bc072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f83bc06b6c0 0x7f83bc072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44650/0 (socket says 172.21.15.122:44650) 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 -- 172.21.15.122:0/2619376454 learned_addr learned my addr 172.21.15.122:0/2619376454 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 -- 172.21.15.122:0/2619376454 >> v2:172.21.15.169:3300/0 conn(0x7f83bc0cd390 msgr2=0x7f83bc0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 --2- 172.21.15.122:0/2619376454 >> v2:172.21.15.169:3300/0 conn(0x7f83bc0cd390 0x7f83bc0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 -- 172.21.15.122:0/2619376454 >> v1:172.21.15.169:6789/0 conn(0x7f83bc15baf0 legacy=0x7f83bc06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.581+0000 7f83c08d5640 1 -- 172.21.15.122:0/2619376454 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0d0bd0 con 0x7f83bc06b6c0 2024-03-06T07:21:47.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.582+0000 7f83c08d5640 1 --2- 172.21.15.122:0/2619376454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc06b6c0 0x7f83bc072700 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0ce310 tx=0x7f83ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a4ce17df1a88f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.582+0000 7f83b3fff640 1 -- 172.21.15.122:0/2619376454 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac035020 con 0x7f83bc06b6c0 2024-03-06T07:21:47.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.582+0000 7f83b3fff640 1 -- 172.21.15.122:0/2619376454 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac031870 con 0x7f83bc06b6c0 2024-03-06T07:21:47.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.582+0000 7f83b3fff640 1 -- 172.21.15.122:0/2619376454 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac031c40 con 0x7f83bc06b6c0 2024-03-06T07:21:47.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 -- 172.21.15.122:0/2619376454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc06b6c0 msgr2=0x7f83bc072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 --2- 172.21.15.122:0/2619376454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc06b6c0 0x7f83bc072700 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0ce310 tx=0x7f83ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 -- 172.21.15.122:0/2619376454 shutdown_connections 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 --2- 172.21.15.122:0/2619376454 >> v2:172.21.15.169:3300/0 conn(0x7f83bc0cd390 0x7f83bc0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 --2- 172.21.15.122:0/2619376454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc06b6c0 0x7f83bc072700 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 -- 172.21.15.122:0/2619376454 >> 172.21.15.122:0/2619376454 conn(0x7f83bc06c0c0 msgr2=0x7f83bc06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 -- 172.21.15.122:0/2619376454 shutdown_connections 2024-03-06T07:21:47.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.583+0000 7f83c333a640 1 -- 172.21.15.122:0/2619376454 wait complete. 2024-03-06T07:21:47.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.584+0000 7f83c333a640 1 Processor -- start 2024-03-06T07:21:47.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.584+0000 7f83c333a640 1 -- start start 2024-03-06T07:21:47.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83bc06b6c0 0x7f83bc14f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83bc0cd390 0x7f83bc142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 0x7f83bc15dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc15bed0 con 0x7f83bc15baf0 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d1190 con 0x7f83bc06b6c0 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d16b0 con 0x7f83bc0cd390 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 0x7f83bc15dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 0x7f83bc15dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44656/0 (socket says 172.21.15.122:44656) 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 -- 172.21.15.122:0/2449578330 learned_addr learned my addr 172.21.15.122:0/2449578330 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c08d5640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83bc0cd390 0x7f83bc142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c10d6640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83bc06b6c0 0x7f83bc14f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 -- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83bc0cd390 msgr2=0x7f83bc142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83bc0cd390 0x7f83bc142290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 -- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83bc06b6c0 msgr2=0x7f83bc14f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83bc06b6c0 0x7f83bc14f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 -- 172.21.15.122:0/2449578330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc15cc70 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c18d7640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 0x7f83bc15dd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f83b400b3e0 tx=0x7f83b4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4015020 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b4011830 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc15cf50 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.585+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4011c00 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.586+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc15d510 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.586+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc04f330 con 0x7f83bc15baf0 2024-03-06T07:21:47.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.587+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83b4018020 con 0x7f83bc15baf0 2024-03-06T07:21:47.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.587+0000 7f83b1ffb640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.587+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f83b4089d10 con 0x7f83bc15baf0 2024-03-06T07:21:47.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.587+0000 7f83c10d6640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.588+0000 7f83c10d6640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83b8007dc0 tx=0x7f83b80072f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.591+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83b401f020 con 0x7f83bc15baf0 2024-03-06T07:21:47.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.800+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83bc071f00 con 0x7f83bc15baf0 2024-03-06T07:21:47.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.800+0000 7f83b1ffb640 1 -- 172.21.15.122:0/2449578330 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7f83b40172c0 con 0x7f83bc15baf0 2024-03-06T07:21:47.803 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:47.803 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:47.803 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 2 2024-03-06T07:21:47.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f83a006b4d0 msgr2=0x7f83a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83b8007dc0 tx=0x7f83b80072f0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 msgr2=0x7f83bc15dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 0x7f83bc15dd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f83b400b3e0 tx=0x7f83b4009f90 comp rx=0 tx=0).stop 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 shutdown_connections 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc15baf0 0x7f83bc15dd10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83bc0cd390 0x7f83bc142290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 --2- 172.21.15.122:0/2449578330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83bc06b6c0 0x7f83bc14f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.802+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 >> 172.21.15.122:0/2449578330 conn(0x7f83bc06c0c0 msgr2=0x7f83bc06d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.803+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 shutdown_connections 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:47.803+0000 7f83c333a640 1 -- 172.21.15.122:0/2449578330 wait complete. 2024-03-06T07:21:49.816 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.924+0000 7fd3b0cd4640 1 Processor -- start 2024-03-06T07:21:49.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.924+0000 7fd3b0cd4640 1 -- start start 2024-03-06T07:21:49.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3b0cd4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd3ac0b4f70 0x7fd3ac0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3b0cd4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd3ac06c6d0 0x7fd3ac15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3b0cd4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0b5890 con 0x7fd3ac066d90 2024-03-06T07:21:49.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3b0cd4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac15e9f0 con 0x7fd3ac06c6d0 2024-03-06T07:21:49.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3b0cd4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac0cda10 con 0x7fd3ac0b4f70 2024-03-06T07:21:49.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3a1d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd3ac06c6d0 0x7fd3ac15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3a1d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd3ac06c6d0 0x7fd3ac15ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44660/0 (socket says 172.21.15.122:44660) 2024-03-06T07:21:49.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.925+0000 7fd3a1d9b640 1 -- 172.21.15.122:0/120301721 learned_addr learned my addr 172.21.15.122:0/120301721 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:49.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3a1d9b640 1 -- 172.21.15.122:0/120301721 >> v2:172.21.15.169:3300/0 conn(0x7fd3ac0b4f70 msgr2=0x7fd3ac0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3aa59c640 1 --2- 172.21.15.122:0/120301721 >> v2:172.21.15.169:3300/0 conn(0x7fd3ac0b4f70 0x7fd3ac0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3a1d9b640 1 --2- 172.21.15.122:0/120301721 >> v2:172.21.15.169:3300/0 conn(0x7fd3ac0b4f70 0x7fd3ac0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3a1d9b640 1 -- 172.21.15.122:0/120301721 >> v1:172.21.15.169:6789/0 conn(0x7fd3ac066d90 legacy=0x7fd3ac0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3a1d9b640 1 -- 172.21.15.122:0/120301721 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0cd450 con 0x7fd3ac06c6d0 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3aa59c640 1 --2- 172.21.15.122:0/120301721 >> v2:172.21.15.169:3300/0 conn(0x7fd3ac0b4f70 0x7fd3ac0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.926+0000 7fd3a1d9b640 1 --2- 172.21.15.122:0/120301721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac06c6d0 0x7fd3ac15ee80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac06cac0 tx=0x7fd39402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebc4a253c610db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3a9d9b640 1 -- 172.21.15.122:0/120301721 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394035020 con 0x7fd3ac06c6d0 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3a9d9b640 1 -- 172.21.15.122:0/120301721 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd394031870 con 0x7fd3ac06c6d0 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3a9d9b640 1 -- 172.21.15.122:0/120301721 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394031c40 con 0x7fd3ac06c6d0 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/120301721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac06c6d0 msgr2=0x7fd3ac15ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/120301721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac06c6d0 0x7fd3ac15ee80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac06cac0 tx=0x7fd39402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:49.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/120301721 shutdown_connections 2024-03-06T07:21:49.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/120301721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac06c6d0 0x7fd3ac15ee80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/120301721 >> v2:172.21.15.169:3300/0 conn(0x7fd3ac0b4f70 0x7fd3ac0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/120301721 >> 172.21.15.122:0/120301721 conn(0x7fd3ac0724d0 msgr2=0x7fd3ac0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/120301721 shutdown_connections 2024-03-06T07:21:49.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.927+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/120301721 wait complete. 2024-03-06T07:21:49.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 Processor -- start 2024-03-06T07:21:49.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 -- start start 2024-03-06T07:21:49.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 0x7fd3ac0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd3ac0b4f70 0x7fd3ac15a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd3ac159660 0x7fd3ac15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac15f3c0 con 0x7fd3ac066d90 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac15e680 con 0x7fd3ac0b4f70 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.929+0000 7fd3b0cd4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac161ad0 con 0x7fd3ac159660 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 0x7fd3ac0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 0x7fd3ac0ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44674/0 (socket says 172.21.15.122:44674) 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 -- 172.21.15.122:0/2315733591 learned_addr learned my addr 172.21.15.122:0/2315733591 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3a1d9b640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd3ac0b4f70 0x7fd3ac15a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aad9d640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd3ac159660 0x7fd3ac15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 -- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd3ac159660 msgr2=0x7fd3ac15bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd3ac159660 0x7fd3ac15bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 -- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd3ac0b4f70 msgr2=0x7fd3ac15a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd3ac0b4f70 0x7fd3ac15a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 -- 172.21.15.122:0/2315733591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0ae160 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aad9d640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd3ac159660 0x7fd3ac15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3aa59c640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 0x7fd3ac0ad910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00c820 tx=0x7fd39c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c015070 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c0079d0 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac0ae440 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c007d80 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.930+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac0aece0 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.931+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac0ae610 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.932+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd39c018070 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.932+0000 7fd3a37fe640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38c06b4d0 0x7fd38c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.932+0000 7fd3a1d9b640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38c06b4d0 0x7fd38c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.932+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd39c089d70 con 0x7fd3ac066d90 2024-03-06T07:21:49.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.933+0000 7fd3a1d9b640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38c06b4d0 0x7fd38c06d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd394036000 tx=0x7fd39403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:49.937+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd39c058860 con 0x7fd3ac066d90 2024-03-06T07:21:50.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.145+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3ac04f330 con 0x7fd3ac066d90 2024-03-06T07:21:50.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.145+0000 7fd3a37fe640 1 -- 172.21.15.122:0/2315733591 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3706 (secure 0 0 0) 0x7fd39c05c900 con 0x7fd3ac066d90 2024-03-06T07:21:50.148 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:50.148 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:21:50.148 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 3 2024-03-06T07:21:50.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38c06b4d0 msgr2=0x7fd38c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38c06b4d0 0x7fd38c06d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd394036000 tx=0x7fd39403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 msgr2=0x7fd3ac0ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 0x7fd3ac0ad910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00c820 tx=0x7fd39c00cce0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 shutdown_connections 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38c06b4d0 0x7fd38c06d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd3ac159660 0x7fd3ac15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd3ac0b4f70 0x7fd3ac15a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 --2- 172.21.15.122:0/2315733591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3ac066d90 0x7fd3ac0ad910 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 >> 172.21.15.122:0/2315733591 conn(0x7fd3ac0724d0 msgr2=0x7fd3ac06bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 shutdown_connections 2024-03-06T07:21:50.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.147+0000 7fd3b0cd4640 1 -- 172.21.15.122:0/2315733591 wait complete. 2024-03-06T07:21:50.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:50.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.258+0000 7f6b62cc9640 1 Processor -- start 2024-03-06T07:21:50.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.258+0000 7f6b62cc9640 1 -- start start 2024-03-06T07:21:50.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b62cc9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6b5c1424f0 0x7f6b5c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b62cc9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6b5c0cd7a0 0x7f6b5c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b62cc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0cc980 con 0x7f6b5c06d330 2024-03-06T07:21:50.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b62cc9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c0cd220 con 0x7f6b5c1424f0 2024-03-06T07:21:50.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b62cc9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c0ccc60 con 0x7f6b5c0cd7a0 2024-03-06T07:21:50.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b60a65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6b5c1424f0 0x7f6b5c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b61266640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6b5c06d330 0x7f6b5c071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54630/0 (socket says 172.21.15.122:54630) 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b61266640 1 -- 172.21.15.122:0/2606192770 learned_addr learned my addr 172.21.15.122:0/2606192770 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b60a65640 1 -- 172.21.15.122:0/2606192770 >> v2:172.21.15.169:3300/0 conn(0x7f6b5c0cd7a0 msgr2=0x7f6b5c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b60a65640 1 --2- 172.21.15.122:0/2606192770 >> v2:172.21.15.169:3300/0 conn(0x7f6b5c0cd7a0 0x7f6b5c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b60a65640 1 -- 172.21.15.122:0/2606192770 >> v1:172.21.15.122:6789/0 conn(0x7f6b5c06d330 legacy=0x7f6b5c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b5b7fe640 1 -- 172.21.15.122:0/2606192770 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293357170 0 0) 0x7f6b5c0cc980 con 0x7f6b5c06d330 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.259+0000 7f6b60a65640 1 -- 172.21.15.122:0/2606192770 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c0ccf40 con 0x7f6b5c1424f0 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b60a65640 1 --2- 172.21.15.122:0/2606192770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c0cc2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0b2a00 tx=0x7f6b5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c715455f6ec2ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b5b7fe640 1 -- 172.21.15.122:0/2606192770 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50035020 con 0x7f6b5c1424f0 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b5b7fe640 1 -- 172.21.15.122:0/2606192770 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b50031830 con 0x7f6b5c1424f0 2024-03-06T07:21:50.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b5b7fe640 1 -- 172.21.15.122:0/2606192770 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50031c00 con 0x7f6b5c1424f0 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2606192770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 msgr2=0x7f6b5c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2606192770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c0cc2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0b2a00 tx=0x7f6b5002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2606192770 shutdown_connections 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2606192770 >> v2:172.21.15.169:3300/0 conn(0x7f6b5c0cd7a0 0x7f6b5c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2606192770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c0cc2f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2606192770 >> 172.21.15.122:0/2606192770 conn(0x7f6b5c0cf8a0 msgr2=0x7f6b5c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2606192770 shutdown_connections 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.260+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2606192770 wait complete. 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 Processor -- start 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 -- start start 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b5c06d330 0x7f6b5c1412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b5c0cd7a0 0x7f6b5c1f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c06d920 con 0x7f6b5c1424f0 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c06d7a0 con 0x7f6b5c0cd7a0 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.262+0000 7f6b62cc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c06daa0 con 0x7f6b5c06d330 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44690/0 (socket says 172.21.15.122:44690) 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 -- 172.21.15.122:0/2934660073 learned_addr learned my addr 172.21.15.122:0/2934660073 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b60a65640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b5c06d330 0x7f6b5c1412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 -- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b5c06d330 msgr2=0x7f6b5c1412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b5c06d330 0x7f6b5c1412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 -- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b5c0cd7a0 msgr2=0x7f6b5c1f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b5c0cd7a0 0x7f6b5c1f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 -- 172.21.15.122:0/2934660073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c1417e0 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b61266640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c155730 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400c7a0 tx=0x7f6b5400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54018070 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5400a640 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c1f5960 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400aa70 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.263+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c1f5f20 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.264+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c04f330 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.264+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b5401b070 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.265+0000 7f6b597fa640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b4006b500 0x7f6b4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.265+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6b54089380 con 0x7f6b5c1424f0 2024-03-06T07:21:50.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.265+0000 7f6b60a65640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b4006b500 0x7f6b4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.265+0000 7f6b60a65640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b4006b500 0x7f6b4006d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b50036000 tx=0x7f6b50008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.269+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b540585e0 con 0x7f6b5c1424f0 2024-03-06T07:21:50.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.442+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b5c0d0620 con 0x7f6b5c1424f0 2024-03-06T07:21:50.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.442+0000 7f6b597fa640 1 -- 172.21.15.122:0/2934660073 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12884 (secure 0 0 0) 0x7f6b54057ee0 con 0x7f6b5c1424f0 2024-03-06T07:21:50.444 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:50.445 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":16,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:43.507224+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:50.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b4006b500 msgr2=0x7f6b4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b4006b500 0x7f6b4006d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b50036000 tx=0x7f6b50008030 comp rx=0 tx=0).stop 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 msgr2=0x7f6b5c155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c155730 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400c7a0 tx=0x7f6b5400cc60 comp rx=0 tx=0).stop 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 shutdown_connections 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b4006b500 0x7f6b4006d9c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5c1424f0 0x7f6b5c155730 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b5c0cd7a0 0x7f6b5c1f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 --2- 172.21.15.122:0/2934660073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b5c06d330 0x7f6b5c1412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 >> 172.21.15.122:0/2934660073 conn(0x7f6b5c0cf8a0 msgr2=0x7f6b5c15a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 shutdown_connections 2024-03-06T07:21:50.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.444+0000 7f6b62cc9640 1 -- 172.21.15.122:0/2934660073 wait complete. 2024-03-06T07:21:50.457 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:50.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:50.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.558+0000 7fdca45d0640 1 Processor -- start 2024-03-06T07:21:50.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.558+0000 7fdca45d0640 1 -- start start 2024-03-06T07:21:50.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca45d0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdc9c0d4860 0x7fdc9c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca45d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c1618b0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca45d0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0cf400 con 0x7fdc9c1614d0 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca45d0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c0d4cf0 con 0x7fdc9c0d4860 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca1b6b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdc9c0d4860 0x7fdc9c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca2b6d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdc9c0b0e30 0x7fdc9c0b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54646/0 (socket says 172.21.15.122:54646) 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.559+0000 7fdca2b6d640 1 -- 172.21.15.122:0/1277032419 learned_addr learned my addr 172.21.15.122:0/1277032419 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.560+0000 7fdca1b6b640 1 -- 172.21.15.122:0/1277032419 >> v1:172.21.15.169:6789/0 conn(0x7fdc9c1614d0 legacy=0x7fdc9c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.560+0000 7fdca1b6b640 1 -- 172.21.15.122:0/1277032419 >> v1:172.21.15.122:6789/0 conn(0x7fdc9c0b0e30 legacy=0x7fdc9c0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.560+0000 7fdca1b6b640 1 -- 172.21.15.122:0/1277032419 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0cf6a0 con 0x7fdc9c0d4860 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.560+0000 7fdca136a640 1 -- 172.21.15.122:0/1277032419 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910516917 0 0) 0x7fdc9c1618b0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.560+0000 7fdca1b6b640 1 --2- 172.21.15.122:0/1277032419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0d4860 0x7fdc9c0d2020 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0097a0 tx=0x7fdc8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b37dc0b33c03dde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca136a640 1 -- 172.21.15.122:0/1277032419 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c034070 con 0x7fdc9c0d4860 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca136a640 1 -- 172.21.15.122:0/1277032419 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c030610 con 0x7fdc9c0d4860 2024-03-06T07:21:50.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca136a640 1 -- 172.21.15.122:0/1277032419 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c0309e0 con 0x7fdc9c0d4860 2024-03-06T07:21:50.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 -- 172.21.15.122:0/1277032419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0d4860 msgr2=0x7fdc9c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1277032419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0d4860 0x7fdc9c0d2020 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0097a0 tx=0x7fdc8c02b960 comp rx=0 tx=0).stop 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 -- 172.21.15.122:0/1277032419 shutdown_connections 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1277032419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0d4860 0x7fdc9c0d2020 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 -- 172.21.15.122:0/1277032419 >> 172.21.15.122:0/1277032419 conn(0x7fdc9c0d3d40 msgr2=0x7fdc9c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 -- 172.21.15.122:0/1277032419 shutdown_connections 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.561+0000 7fdca45d0640 1 -- 172.21.15.122:0/1277032419 wait complete. 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.562+0000 7fdca45d0640 1 Processor -- start 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.562+0000 7fdca45d0640 1 -- start start 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca45d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 0x7fdc9c0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca45d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc9c0d4860 0x7fdc9c156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca45d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc9c1614d0 0x7fdc9c1f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca45d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0b0910 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca45d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0cfc60 con 0x7fdc9c1614d0 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca45d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0d2560 con 0x7fdc9c0d4860 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 0x7fdc9c0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 0x7fdc9c0cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44714/0 (socket says 172.21.15.122:44714) 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 -- 172.21.15.122:0/1789977372 learned_addr learned my addr 172.21.15.122:0/1789977372 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca1b6b640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc9c0d4860 0x7fdc9c156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca2b6d640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc9c1614d0 0x7fdc9c1f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 -- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc9c0d4860 msgr2=0x7fdc9c156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc9c0d4860 0x7fdc9c156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 -- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc9c1614d0 msgr2=0x7fdc9c1f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc9c1614d0 0x7fdc9c1f7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca236c640 1 -- 172.21.15.122:0/1789977372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0ca020 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.563+0000 7fdca1b6b640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc9c0d4860 0x7fdc9c156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.564+0000 7fdca236c640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 0x7fdc9c0cb7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c430 tx=0x7fdc9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.564+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98017070 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.564+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc980052d0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.564+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98003830 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.564+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c0cbee0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.564+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c0cc4a0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.565+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c068ff0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.565+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc9801a070 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.566+0000 7fdc92ffd640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.566+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdc980891a0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.566+0000 7fdca1b6b640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.566+0000 7fdca1b6b640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8006b4d0 0x7fdc8006d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.570+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc98058400 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.739+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc9c0d0af0 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.740+0000 7fdc92ffd640 1 -- 172.21.15.122:0/1789977372 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc98057d00 con 0x7fdc9c0b0e30 2024-03-06T07:21:50.742 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:21:50.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8006b4d0 msgr2=0x7fdc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8006b4d0 0x7fdc8006d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 msgr2=0x7fdc9c0cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 0x7fdc9c0cb7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c430 tx=0x7fdc9800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 shutdown_connections 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc9c1614d0 0x7fdc9c1f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc9c0d4860 0x7fdc9c156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 --2- 172.21.15.122:0/1789977372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc9c0b0e30 0x7fdc9c0cb7d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 >> 172.21.15.122:0/1789977372 conn(0x7fdc9c0d3d40 msgr2=0x7fdc9c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 shutdown_connections 2024-03-06T07:21:50.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.742+0000 7fdca45d0640 1 -- 172.21.15.122:0/1789977372 wait complete. 2024-03-06T07:21:50.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:50.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.852+0000 7f967850d640 1 Processor -- start 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.852+0000 7f967850d640 1 -- start start 2024-03-06T07:21:50.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f967850d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f967015ab00 0x7f96700b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f967850d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700afdd0 0x7f967015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f967850d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96700b4e30 con 0x7f96700b48f0 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f967850d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96700b5110 con 0x7f967015ab00 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f967850d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967015b310 con 0x7f96700afdd0 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f96762a9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f967015ab00 0x7f96700b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f96762a9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f967015ab00 0x7f96700b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44716/0 (socket says 172.21.15.122:44716) 2024-03-06T07:21:50.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.853+0000 7f96762a9640 1 -- 172.21.15.122:0/1051579220 learned_addr learned my addr 172.21.15.122:0/1051579220 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:50.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f9676aaa640 1 --2- 172.21.15.122:0/1051579220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700afdd0 0x7f967015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f96762a9640 1 -- 172.21.15.122:0/1051579220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700afdd0 msgr2=0x7f967015bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f96762a9640 1 --2- 172.21.15.122:0/1051579220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700afdd0 0x7f967015bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f96762a9640 1 -- 172.21.15.122:0/1051579220 >> v1:172.21.15.169:6789/0 conn(0x7f96700b48f0 legacy=0x7f96700af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f96762a9640 1 -- 172.21.15.122:0/1051579220 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967015b5f0 con 0x7f967015ab00 2024-03-06T07:21:50.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f96762a9640 1 --2- 172.21.15.122:0/1051579220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f966c009e50 tx=0x7f966c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e49ca7f43f59124 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.854+0000 7f96752a7640 1 -- 172.21.15.122:0/1051579220 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c02cb00 con 0x7f967015ab00 2024-03-06T07:21:50.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f96752a7640 1 -- 172.21.15.122:0/1051579220 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c02cca0 con 0x7f967015ab00 2024-03-06T07:21:50.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f96752a7640 1 -- 172.21.15.122:0/1051579220 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c034610 con 0x7f967015ab00 2024-03-06T07:21:50.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 -- 172.21.15.122:0/1051579220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 msgr2=0x7f96700b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 --2- 172.21.15.122:0/1051579220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f966c009e50 tx=0x7f966c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:50.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 -- 172.21.15.122:0/1051579220 shutdown_connections 2024-03-06T07:21:50.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 --2- 172.21.15.122:0/1051579220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700afdd0 0x7f967015bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 --2- 172.21.15.122:0/1051579220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b43b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 -- 172.21.15.122:0/1051579220 >> 172.21.15.122:0/1051579220 conn(0x7f967006c0c0 msgr2=0x7f967006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 -- 172.21.15.122:0/1051579220 shutdown_connections 2024-03-06T07:21:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.855+0000 7f967850d640 1 -- 172.21.15.122:0/1051579220 wait complete. 2024-03-06T07:21:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.856+0000 7f967850d640 1 Processor -- start 2024-03-06T07:21:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.856+0000 7f967850d640 1 -- start start 2024-03-06T07:21:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f967850d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96700afdd0 0x7f9670149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f967850d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700b48f0 0x7f96701564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f967850d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f967850d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967015b060 con 0x7f967015ab00 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f967850d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967015aee0 con 0x7f96700afdd0 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f967850d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967015c050 con 0x7f96700b48f0 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44728/0 (socket says 172.21.15.122:44728) 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 -- 172.21.15.122:0/1729806722 learned_addr learned my addr 172.21.15.122:0/1729806722 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f96762a9640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96700afdd0 0x7f9670149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9675aa8640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700b48f0 0x7f96701564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 -- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700b48f0 msgr2=0x7f96701564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700b48f0 0x7f96701564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 -- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96700afdd0 msgr2=0x7f9670149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96700afdd0 0x7f9670149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9676aaa640 1 -- 172.21.15.122:0/1729806722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96701541c0 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f96762a9640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96700afdd0 0x7f9670149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.857+0000 7f9675aa8640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700b48f0 0x7f96701564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.858+0000 7f9676aaa640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b28b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f966800d580 tx=0x7f966800da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.858+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680125f0 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.858+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9668019070 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.858+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680129f0 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.858+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96701580e0 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.858+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96701586a0 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.859+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967004f330 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.859+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9668012b90 con 0x7f967015ab00 2024-03-06T07:21:50.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.859+0000 7f9666ffd640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.860+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f966808a250 con 0x7f967015ab00 2024-03-06T07:21:50.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.860+0000 7f96762a9640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.860+0000 7f96762a9640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f965406b4d0 0x7f965406d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f966c035000 tx=0x7f966c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:50.864+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9668059430 con 0x7f967015ab00 2024-03-06T07:21:51.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.037+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f96700b1dd0 con 0x7f967015ab00 2024-03-06T07:21:51.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.037+0000 7f9666ffd640 1 -- 172.21.15.122:0/1729806722 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9668058d30 con 0x7f967015ab00 2024-03-06T07:21:51.039 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:51.039 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:21:51.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:21:51.039 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:21:51.040 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:21:51.040 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:21:51.040 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f965406b4d0 msgr2=0x7f965406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f965406b4d0 0x7f965406d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f966c035000 tx=0x7f966c030000 comp rx=0 tx=0).stop 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 msgr2=0x7f96700b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b28b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f966800d580 tx=0x7f966800da40 comp rx=0 tx=0).stop 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 shutdown_connections 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f967015ab00 0x7f96700b28b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96700b48f0 0x7f96701564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 --2- 172.21.15.122:0/1729806722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96700afdd0 0x7f9670149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 >> 172.21.15.122:0/1729806722 conn(0x7f967006c0c0 msgr2=0x7f967015d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 shutdown_connections 2024-03-06T07:21:51.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.039+0000 7f967850d640 1 -- 172.21.15.122:0/1729806722 wait complete. 2024-03-06T07:21:51.051 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:51.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:51.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.153+0000 7f1bad6aa640 1 Processor -- start 2024-03-06T07:21:51.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.153+0000 7f1bad6aa640 1 -- start start 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1bad6aa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba815dfa0 0x7f1ba80d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1bad6aa640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1ba80d4630 0x7f1ba80d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1bad6aa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1ba80d2820 0x7f1ba80d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1bad6aa640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba80d2100 con 0x7f1ba80d2820 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1bad6aa640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba815e380 con 0x7f1ba80d4630 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1bad6aa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80d1e30 con 0x7f1ba815dfa0 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1ba80d2820 0x7f1ba80d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1ba80d2820 0x7f1ba80d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44740/0 (socket says 172.21.15.122:44740) 2024-03-06T07:21:51.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 -- 172.21.15.122:0/371095537 learned_addr learned my addr 172.21.15.122:0/371095537 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 -- 172.21.15.122:0/371095537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba815dfa0 msgr2=0x7f1ba80d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba6ffd640 1 --2- 172.21.15.122:0/371095537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba815dfa0 0x7f1ba80d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba67fc640 1 --2- 172.21.15.122:0/371095537 >> v2:172.21.15.169:3300/0 conn(0x7f1ba80d4630 0x7f1ba80d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 --2- 172.21.15.122:0/371095537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba815dfa0 0x7f1ba80d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 -- 172.21.15.122:0/371095537 >> v2:172.21.15.169:3300/0 conn(0x7f1ba80d4630 msgr2=0x7f1ba80d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 --2- 172.21.15.122:0/371095537 >> v2:172.21.15.169:3300/0 conn(0x7f1ba80d4630 0x7f1ba80d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.154+0000 7f1ba77fe640 1 -- 172.21.15.122:0/371095537 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80cf470 con 0x7f1ba80d2820 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.155+0000 7f1ba77fe640 1 --2- 172.21.15.122:0/371095537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba80d05f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ba808e150 tx=0x7f1ba002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9a7a3f4537f107 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.155+0000 7f1ba5ffb640 1 -- 172.21.15.122:0/371095537 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba002c9e0 con 0x7f1ba80d2820 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.155+0000 7f1ba5ffb640 1 -- 172.21.15.122:0/371095537 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba002cb80 con 0x7f1ba80d2820 2024-03-06T07:21:51.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.155+0000 7f1ba5ffb640 1 -- 172.21.15.122:0/371095537 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba00343f0 con 0x7f1ba80d2820 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 -- 172.21.15.122:0/371095537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 msgr2=0x7f1ba80d05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/371095537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba80d05f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ba808e150 tx=0x7f1ba002be40 comp rx=0 tx=0).stop 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 -- 172.21.15.122:0/371095537 shutdown_connections 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/371095537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba80d05f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/371095537 >> v2:172.21.15.169:3300/0 conn(0x7f1ba80d4630 0x7f1ba80d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/371095537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba815dfa0 0x7f1ba80d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 -- 172.21.15.122:0/371095537 >> 172.21.15.122:0/371095537 conn(0x7f1ba809d000 msgr2=0x7f1ba80d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 -- 172.21.15.122:0/371095537 shutdown_connections 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.156+0000 7f1bad6aa640 1 -- 172.21.15.122:0/371095537 wait complete. 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.157+0000 7f1bad6aa640 1 Processor -- start 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.157+0000 7f1bad6aa640 1 -- start start 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba80d4630 0x7f1ba809da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ba815dfa0 0x7f1ba809dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80d0b30 con 0x7f1ba80d2820 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80cfa30 con 0x7f1ba815dfa0 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80d0cb0 con 0x7f1ba80d4630 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba815c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44752/0 (socket says 172.21.15.122:44752) 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 -- 172.21.15.122:0/523922999 learned_addr learned my addr 172.21.15.122:0/523922999 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:51.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba67fc640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba80d4630 0x7f1ba809da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 -- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba80d4630 msgr2=0x7f1ba809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba80d4630 0x7f1ba809da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 -- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ba815dfa0 msgr2=0x7f1ba809dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ba815dfa0 0x7f1ba809dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 -- 172.21.15.122:0/523922999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba809f980 con 0x7f1ba80d2820 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1ba6ffd640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba815c4f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c7f0 tx=0x7f1b9800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98015070 con 0x7f1ba80d2820 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98004220 con 0x7f1ba80d2820 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98007620 con 0x7f1ba80d2820 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba80b87b0 con 0x7f1ba80d2820 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.158+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba80b8d70 con 0x7f1ba80d2820 2024-03-06T07:21:51.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.159+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba804f330 con 0x7f1ba80d2820 2024-03-06T07:21:51.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.160+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b980079d0 con 0x7f1ba80d2820 2024-03-06T07:21:51.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.160+0000 7f1b8f7fe640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b9406b500 0x7f1b9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.160+0000 7f1ba67fc640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b9406b500 0x7f1b9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.160+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1b98089e80 con 0x7f1ba80d2820 2024-03-06T07:21:51.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.161+0000 7f1ba67fc640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b9406b500 0x7f1b9406d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ba815ca30 tx=0x7f1b9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.165+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b98059050 con 0x7f1ba80d2820 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.337+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1ba80b5470 con 0x7f1ba80d2820 2024-03-06T07:21:51.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.530+0000 7f1b8f7fe640 1 -- 172.21.15.122:0/523922999 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b98058950 con 0x7f1ba80d2820 2024-03-06T07:21:51.533 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b9406b500 msgr2=0x7f1b9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b9406b500 0x7f1b9406d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ba815ca30 tx=0x7f1b9c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 msgr2=0x7f1ba815c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba815c4f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c7f0 tx=0x7f1b9800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 shutdown_connections 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b9406b500 0x7f1b9406d9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ba815dfa0 0x7f1ba809dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ba80d4630 0x7f1ba809da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 --2- 172.21.15.122:0/523922999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba80d2820 0x7f1ba815c4f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 >> 172.21.15.122:0/523922999 conn(0x7f1ba809d000 msgr2=0x7f1ba80d3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 shutdown_connections 2024-03-06T07:21:51.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.533+0000 7f1bad6aa640 1 -- 172.21.15.122:0/523922999 wait complete. 2024-03-06T07:21:51.549 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:51.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.654+0000 7f8458355640 1 Processor -- start 2024-03-06T07:21:51.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.654+0000 7f8458355640 1 -- start start 2024-03-06T07:21:51.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f8458355640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84501614d0 0x7f84500d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f8458355640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d24b0 0x7f84500b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f8458355640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84501618b0 con 0x7f84500d1910 2024-03-06T07:21:51.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f8458355640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84500cf400 con 0x7f84501614d0 2024-03-06T07:21:51.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f8458355640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84500cf6a0 con 0x7f84500d24b0 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84560f1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84501614d0 0x7f84500d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84558f0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f84500d1910 0x7f84500d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54658/0 (socket says 172.21.15.122:54658) 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84558f0640 1 -- 172.21.15.122:0/3627309046 learned_addr learned my addr 172.21.15.122:0/3627309046 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84560f1640 1 -- 172.21.15.122:0/3627309046 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d24b0 msgr2=0x7f84500b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84568f2640 1 --2- 172.21.15.122:0/3627309046 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d24b0 0x7f84500b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84560f1640 1 --2- 172.21.15.122:0/3627309046 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d24b0 0x7f84500b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84550ef640 1 -- 172.21.15.122:0/3627309046 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021957698 0 0) 0x7f84501618b0 con 0x7f84500d1910 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84560f1640 1 -- 172.21.15.122:0/3627309046 >> v1:172.21.15.122:6789/0 conn(0x7f84500d1910 legacy=0x7f84500d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.655+0000 7f84560f1640 1 -- 172.21.15.122:0/3627309046 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500cf980 con 0x7f84501614d0 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.656+0000 7f84560f1640 1 --2- 172.21.15.122:0/3627309046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f84500d4860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f844c0097a0 tx=0x7f844c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8570b88b413ef6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.656+0000 7f84550ef640 1 -- 172.21.15.122:0/3627309046 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c035020 con 0x7f84501614d0 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.656+0000 7f84550ef640 1 -- 172.21.15.122:0/3627309046 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c031830 con 0x7f84501614d0 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.656+0000 7f84550ef640 1 -- 172.21.15.122:0/3627309046 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c031c00 con 0x7f84501614d0 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.656+0000 7f8458355640 1 -- 172.21.15.122:0/3627309046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 msgr2=0x7f84500d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.656+0000 7f8458355640 1 --2- 172.21.15.122:0/3627309046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f84500d4860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f844c0097a0 tx=0x7f844c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.657+0000 7f8458355640 1 -- 172.21.15.122:0/3627309046 shutdown_connections 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.657+0000 7f8458355640 1 --2- 172.21.15.122:0/3627309046 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d24b0 0x7f84500b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.657+0000 7f8458355640 1 --2- 172.21.15.122:0/3627309046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f84500d4860 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.657+0000 7f8458355640 1 -- 172.21.15.122:0/3627309046 >> 172.21.15.122:0/3627309046 conn(0x7f84500d3d40 msgr2=0x7f84500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.657+0000 7f8458355640 1 -- 172.21.15.122:0/3627309046 shutdown_connections 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.657+0000 7f8458355640 1 -- 172.21.15.122:0/3627309046 wait complete. 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 Processor -- start 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 -- start start 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d1910 0x7f84500ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d24b0 0x7f8450152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f8450152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84500b1370 con 0x7f84501614d0 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84500b0910 con 0x7f84500d24b0 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f8458355640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84500b14f0 con 0x7f84500d1910 2024-03-06T07:21:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.658+0000 7f84568f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f8450152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f8450152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44784/0 (socket says 172.21.15.122:44784) 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 -- 172.21.15.122:0/245997156 learned_addr learned my addr 172.21.15.122:0/245997156 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84560f1640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d1910 0x7f84500ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84558f0640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d24b0 0x7f8450152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 -- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d1910 msgr2=0x7f84500ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d1910 0x7f84500ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 -- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d24b0 msgr2=0x7f8450152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d24b0 0x7f8450152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 -- 172.21.15.122:0/245997156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500ccb20 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f84568f2640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f8450152770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f844800cd70 tx=0x7f844800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448003b90 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8448003d30 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8450152dc0 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448004240 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.659+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8450153380 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.660+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84500d2190 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.661+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84480043e0 con 0x7f84501614d0 2024-03-06T07:21:51.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.661+0000 7f8446ffd640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.661+0000 7f84560f1640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.661+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f844808aa20 con 0x7f84501614d0 2024-03-06T07:21:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.661+0000 7f84560f1640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f844c036000 tx=0x7f844c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.666+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8448059430 con 0x7f84501614d0 2024-03-06T07:21:51.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:51.836+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8450068ff0 con 0x7f84501614d0 2024-03-06T07:21:52.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.535+0000 7f8446ffd640 1 -- 172.21.15.122:0/245997156 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f844805d4d0 con 0x7f84501614d0 2024-03-06T07:21:52.537 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.537+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 msgr2=0x7f843406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.537+0000 7f8458355640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f844c036000 tx=0x7f844c03c040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 msgr2=0x7f8450152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f8450152770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f844800cd70 tx=0x7f844800a660 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 shutdown_connections 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84501614d0 0x7f8450152770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d24b0 0x7f8450152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 --2- 172.21.15.122:0/245997156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84500d1910 0x7f84500ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 >> 172.21.15.122:0/245997156 conn(0x7f84500d3d40 msgr2=0x7f84500d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 shutdown_connections 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.538+0000 7f8458355640 1 -- 172.21.15.122:0/245997156 wait complete. 2024-03-06T07:21:52.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:52.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.665+0000 7f10e8593640 1 Processor -- start 2024-03-06T07:21:52.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.665+0000 7f10e8593640 1 -- start start 2024-03-06T07:21:52.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e8593640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0158420 0x7f10e00b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e8593640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f10e015ba80 0x7f10e00b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e8593640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e015e8a0 con 0x7f10e0158800 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e8593640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e015e650 con 0x7f10e015ba80 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e8593640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e015eb80 con 0x7f10e0158420 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f10e015ba80 0x7f10e00b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f10e015ba80 0x7f10e00b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44788/0 (socket says 172.21.15.122:44788) 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e5b2e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f10e0158800 0x7f10e015b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54668/0 (socket says 172.21.15.122:54668) 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 -- 172.21.15.122:0/4176837470 learned_addr learned my addr 172.21.15.122:0/4176837470 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e632f640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0158420 0x7f10e00b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 -- 172.21.15.122:0/4176837470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0158420 msgr2=0x7f10e00b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0158420 0x7f10e00b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 -- 172.21.15.122:0/4176837470 >> v1:172.21.15.122:6789/0 conn(0x7f10e0158800 legacy=0x7f10e015b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e6b30640 1 -- 172.21.15.122:0/4176837470 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e015ee60 con 0x7f10e015ba80 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e532d640 1 -- 172.21.15.122:0/4176837470 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504878819 0 0) 0x7f10e015e8a0 con 0x7f10e0158800 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.666+0000 7f10e632f640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0158420 0x7f10e00b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.667+0000 7f10e6b30640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e015ba80 0x7f10e00b3b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009e50 tx=0x7f10d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13530ede169e3ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.667+0000 7f10e532d640 1 -- 172.21.15.122:0/4176837470 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d802ca70 con 0x7f10e015ba80 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.667+0000 7f10e532d640 1 -- 172.21.15.122:0/4176837470 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d802cc10 con 0x7f10e015ba80 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.667+0000 7f10e532d640 1 -- 172.21.15.122:0/4176837470 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80343f0 con 0x7f10e015ba80 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.667+0000 7f10e8593640 1 -- 172.21.15.122:0/4176837470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e015ba80 msgr2=0x7f10e00b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.667+0000 7f10e8593640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e015ba80 0x7f10e00b3b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009e50 tx=0x7f10d802bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.668+0000 7f10e8593640 1 -- 172.21.15.122:0/4176837470 shutdown_connections 2024-03-06T07:21:52.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.668+0000 7f10e8593640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e015ba80 0x7f10e00b3b00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.668+0000 7f10e8593640 1 --2- 172.21.15.122:0/4176837470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0158420 0x7f10e00b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.668+0000 7f10e8593640 1 -- 172.21.15.122:0/4176837470 >> 172.21.15.122:0/4176837470 conn(0x7f10e006afb0 msgr2=0x7f10e006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.668+0000 7f10e8593640 1 -- 172.21.15.122:0/4176837470 shutdown_connections 2024-03-06T07:21:52.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.668+0000 7f10e8593640 1 -- 172.21.15.122:0/4176837470 wait complete. 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 Processor -- start 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 -- start start 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 0x7f10e0140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10e015ba80 0x7f10e0143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0150be0 0x7f10e01435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0071ba0 con 0x7f10e0158420 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0071a20 con 0x7f10e015ba80 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e8593640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0071d20 con 0x7f10e0150be0 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e632f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 0x7f10e0140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e632f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 0x7f10e0140ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44798/0 (socket says 172.21.15.122:44798) 2024-03-06T07:21:52.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.669+0000 7f10e632f640 1 -- 172.21.15.122:0/2546813134 learned_addr learned my addr 172.21.15.122:0/2546813134 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e6b30640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0150be0 0x7f10e01435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e5b2e640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10e015ba80 0x7f10e0143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e632f640 1 -- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0150be0 msgr2=0x7f10e01435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e632f640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0150be0 0x7f10e01435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e632f640 1 -- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10e015ba80 msgr2=0x7f10e0143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e632f640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10e015ba80 0x7f10e0143060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e632f640 1 -- 172.21.15.122:0/2546813134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e006caa0 con 0x7f10e0158420 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e6b30640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0150be0 0x7f10e01435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e5b2e640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10e015ba80 0x7f10e0143060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e632f640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 0x7f10e0140ac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00c7f0 tx=0x7f10dc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc015070 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc007a30 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e006cd80 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc007e00 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.670+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e006d340 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.671+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e004f330 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.671+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10dc018070 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.672+0000 7f10d6ffd640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f10c406b4d0 0x7f10c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.672+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f10dc08a150 con 0x7f10e0158420 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.672+0000 7f10e5b2e640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f10c406b4d0 0x7f10c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.672+0000 7f10e5b2e640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f10c406b4d0 0x7f10c406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f10e015e2c0 tx=0x7f10d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.676+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10dc059030 con 0x7f10e0158420 2024-03-06T07:21:52.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:52.885+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f10e0071520 con 0x7f10e0158420 2024-03-06T07:21:52.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:52.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:53.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.546+0000 7f10d6ffd640 1 -- 172.21.15.122:0/2546813134 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f10dc058930 con 0x7f10e0158420 2024-03-06T07:21:53.549 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:21:53.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.557+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f10c406b4d0 msgr2=0x7f10c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.557+0000 7f10e8593640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f10c406b4d0 0x7f10c406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f10e015e2c0 tx=0x7f10d0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:53.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.557+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 msgr2=0x7f10e0140ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.557+0000 7f10e8593640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 0x7f10e0140ac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00c7f0 tx=0x7f10dc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.559+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 shutdown_connections 2024-03-06T07:21:53.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.560+0000 7f10e8593640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f10c406b4d0 0x7f10c406d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.560+0000 7f10e8593640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10e0150be0 0x7f10e01435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.560+0000 7f10e8593640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10e015ba80 0x7f10e0143060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.560+0000 7f10e8593640 1 --2- 172.21.15.122:0/2546813134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e0158420 0x7f10e0140ac0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.560+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 >> 172.21.15.122:0/2546813134 conn(0x7f10e006afb0 msgr2=0x7f10e006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.562+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 shutdown_connections 2024-03-06T07:21:53.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.562+0000 7f10e8593640 1 -- 172.21.15.122:0/2546813134 wait complete. 2024-03-06T07:21:53.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:53.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.676+0000 7ff50d071640 1 Processor -- start 2024-03-06T07:21:53.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.676+0000 7ff50d071640 1 -- start start 2024-03-06T07:21:53.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50d071640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5080cf850 0x7ff5080cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50d071640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50815e380 con 0x7ff50806cf20 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50d071640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080cdc10 con 0x7ff5080d00c0 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50d071640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50806ca90 con 0x7ff5080cf850 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5080cf850 0x7ff5080cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5080cf850 0x7ff5080cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44804/0 (socket says 172.21.15.122:44804) 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50759e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff50806cf20 0x7ff50806d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54672/0 (socket says 172.21.15.122:54672) 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50659c640 1 -- 172.21.15.122:0/3104638150 learned_addr learned my addr 172.21.15.122:0/3104638150 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50659c640 1 -- 172.21.15.122:0/3104638150 >> v1:172.21.15.169:6789/0 conn(0x7ff5080d00c0 legacy=0x7ff5080d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50659c640 1 -- 172.21.15.122:0/3104638150 >> v1:172.21.15.122:6789/0 conn(0x7ff50806cf20 legacy=0x7ff50806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff50659c640 1 -- 172.21.15.122:0/3104638150 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50806c160 con 0x7ff5080cf850 2024-03-06T07:21:53.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.677+0000 7ff505d9b640 1 -- 172.21.15.122:0/3104638150 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848051488 0 0) 0x7ff50815e380 con 0x7ff50806cf20 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.678+0000 7ff50659c640 1 --2- 172.21.15.122:0/3104638150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5080cf850 0x7ff5080cd3c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009870 tx=0x7ff4fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a566ba7b637504 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.678+0000 7ff505d9b640 1 -- 172.21.15.122:0/3104638150 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc034070 con 0x7ff5080cf850 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.678+0000 7ff505d9b640 1 -- 172.21.15.122:0/3104638150 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc030860 con 0x7ff5080cf850 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.678+0000 7ff505d9b640 1 -- 172.21.15.122:0/3104638150 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc030c30 con 0x7ff5080cf850 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 -- 172.21.15.122:0/3104638150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5080cf850 msgr2=0x7ff5080cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 --2- 172.21.15.122:0/3104638150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5080cf850 0x7ff5080cd3c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009870 tx=0x7ff4fc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 -- 172.21.15.122:0/3104638150 shutdown_connections 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 --2- 172.21.15.122:0/3104638150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5080cf850 0x7ff5080cd3c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 -- 172.21.15.122:0/3104638150 >> 172.21.15.122:0/3104638150 conn(0x7ff5080b35a0 msgr2=0x7ff5080b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 -- 172.21.15.122:0/3104638150 shutdown_connections 2024-03-06T07:21:53.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.679+0000 7ff50d071640 1 -- 172.21.15.122:0/3104638150 wait complete. 2024-03-06T07:21:53.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.680+0000 7ff50d071640 1 Processor -- start 2024-03-06T07:21:53.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.680+0000 7ff50d071640 1 -- start start 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50d071640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 0x7ff5080c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50d071640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5080cf850 0x7ff508150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50d071640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5080d00c0 0x7ff508151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50d071640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080cd900 con 0x7ff50806cf20 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50d071640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50806c720 con 0x7ff5080cf850 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50d071640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080b1030 con 0x7ff5080d00c0 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 0x7ff5080c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 0x7ff5080c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44814/0 (socket says 172.21.15.122:44814) 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 -- 172.21.15.122:0/206394582 learned_addr learned my addr 172.21.15.122:0/206394582 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50659c640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5080cf850 0x7ff508150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50759e640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5080d00c0 0x7ff508151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 -- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5080d00c0 msgr2=0x7ff508151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5080d00c0 0x7ff508151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 -- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5080cf850 msgr2=0x7ff508150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5080cf850 0x7ff508150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 -- 172.21.15.122:0/206394582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508154810 con 0x7ff50806cf20 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50659c640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5080cf850 0x7ff508150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff50759e640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5080d00c0 0x7ff508151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff506d9d640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 0x7ff5080c6b70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800b410 tx=0x7ff4f800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.681+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8015020 con 0x7ff50806cf20 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.682+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f8011830 con 0x7ff50806cf20 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.682+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8011c20 con 0x7ff50806cf20 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.682+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5080c55a0 con 0x7ff50806cf20 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.682+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080c9fa0 con 0x7ff50806cf20 2024-03-06T07:21:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.682+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50804f330 con 0x7ff50806cf20 2024-03-06T07:21:53.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.683+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4f8018020 con 0x7ff50806cf20 2024-03-06T07:21:53.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.683+0000 7ff4ef7fe640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4f406b4d0 0x7ff4f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.683+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4f808a110 con 0x7ff50806cf20 2024-03-06T07:21:53.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.683+0000 7ff50659c640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4f406b4d0 0x7ff4f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.684+0000 7ff50659c640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4f406b4d0 0x7ff4f406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036000 tx=0x7ff4fc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.688+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4f801f020 con 0x7ff50806cf20 2024-03-06T07:21:53.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.863+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5080ceea0 con 0x7ff50806cf20 2024-03-06T07:21:53.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.864+0000 7ff4ef7fe640 1 -- 172.21.15.122:0/206394582 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16618 (secure 0 0 0) 0x7ff4f80172c0 con 0x7ff50806cf20 2024-03-06T07:21:53.866 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:53.867 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":19,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:52.887353+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:53.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4f406b4d0 msgr2=0x7ff4f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4f406b4d0 0x7ff4f406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036000 tx=0x7ff4fc03b040 comp rx=0 tx=0).stop 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 msgr2=0x7ff5080c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 0x7ff5080c6b70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800b410 tx=0x7ff4f800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 shutdown_connections 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4f406b4d0 0x7ff4f406d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5080d00c0 0x7ff508151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5080cf850 0x7ff508150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 --2- 172.21.15.122:0/206394582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff50806cf20 0x7ff5080c6b70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 >> 172.21.15.122:0/206394582 conn(0x7ff5080b35a0 msgr2=0x7ff50806d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 shutdown_connections 2024-03-06T07:21:53.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.866+0000 7ff50d071640 1 -- 172.21.15.122:0/206394582 wait complete. 2024-03-06T07:21:53.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:53.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.978+0000 7f5d750c2640 1 Processor -- start 2024-03-06T07:21:53.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.978+0000 7f5d750c2640 1 -- start start 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d750c2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d700cf470 0x7f5d700cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d750c2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d7006bb70 0x7f5d7006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d750c2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7015b160 con 0x7f5d700730d0 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d750c2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d700b4f70 con 0x7f5d700cf470 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d750c2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700b16f0 con 0x7f5d7006bb70 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d700cf470 0x7f5d700cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6e59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5d700730d0 0x7f5d700b59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54682/0 (socket says 172.21.15.122:54682) 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d700cf470 0x7f5d700cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44818/0 (socket says 172.21.15.122:44818) 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6e59c640 1 -- 172.21.15.122:0/204105929 learned_addr learned my addr 172.21.15.122:0/204105929 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6f59e640 1 --2- 172.21.15.122:0/204105929 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d7006bb70 0x7f5d7006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/204105929 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d7006bb70 msgr2=0x7f5d7006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6ed9d640 1 --2- 172.21.15.122:0/204105929 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d7006bb70 0x7f5d7006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/204105929 >> v1:172.21.15.122:6789/0 conn(0x7f5d700730d0 legacy=0x7f5d700b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/204105929 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700b19d0 con 0x7f5d700cf470 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.979+0000 7f5d6dd9b640 1 -- 172.21.15.122:0/204105929 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996038153 0 0) 0x7f5d7015b160 con 0x7f5d700730d0 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.980+0000 7f5d6ed9d640 1 --2- 172.21.15.122:0/204105929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d700cf470 0x7f5d700cfb70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d600097a0 tx=0x7f5d6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52eb86fc7e66e02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.980+0000 7f5d6dd9b640 1 -- 172.21.15.122:0/204105929 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60035020 con 0x7f5d700cf470 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.980+0000 7f5d6dd9b640 1 -- 172.21.15.122:0/204105929 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d60031830 con 0x7f5d700cf470 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.980+0000 7f5d6dd9b640 1 -- 172.21.15.122:0/204105929 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60031c00 con 0x7f5d700cf470 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 -- 172.21.15.122:0/204105929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d700cf470 msgr2=0x7f5d700cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 --2- 172.21.15.122:0/204105929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d700cf470 0x7f5d700cfb70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d600097a0 tx=0x7f5d6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 -- 172.21.15.122:0/204105929 shutdown_connections 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 --2- 172.21.15.122:0/204105929 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d7006bb70 0x7f5d7006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 --2- 172.21.15.122:0/204105929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d700cf470 0x7f5d700cfb70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 -- 172.21.15.122:0/204105929 >> 172.21.15.122:0/204105929 conn(0x7f5d700724d0 msgr2=0x7f5d700728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 -- 172.21.15.122:0/204105929 shutdown_connections 2024-03-06T07:21:53.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.981+0000 7f5d750c2640 1 -- 172.21.15.122:0/204105929 wait complete. 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.982+0000 7f5d750c2640 1 Processor -- start 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.982+0000 7f5d750c2640 1 -- start start 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d750c2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 0x7f5d701457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d750c2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d700730d0 0x7f5d701f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d750c2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d700cf470 0x7f5d700d0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d750c2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700b1330 con 0x7f5d7006bb70 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d750c2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700b54a0 con 0x7f5d700730d0 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d750c2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700b14b0 con 0x7f5d700cf470 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 0x7f5d701457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 0x7f5d701457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44834/0 (socket says 172.21.15.122:44834) 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/2443555545 learned_addr learned my addr 172.21.15.122:0/2443555545 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6f59e640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d700cf470 0x7f5d700d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6e59c640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d700730d0 0x7f5d701f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d700cf470 msgr2=0x7f5d700d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d700cf470 0x7f5d700d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d700730d0 msgr2=0x7f5d701f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d700730d0 0x7f5d701f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 -- 172.21.15.122:0/2443555545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70145d00 con 0x7f5d7006bb70 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d6ed9d640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 0x7f5d701457c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5d60034040 tx=0x7f5d600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60040070 con 0x7f5d7006bb70 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d600399a0 con 0x7f5d7006bb70 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.983+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60035040 con 0x7f5d7006bb70 2024-03-06T07:21:53.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.984+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d70153290 con 0x7f5d7006bb70 2024-03-06T07:21:53.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.984+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d701524d0 con 0x7f5d7006bb70 2024-03-06T07:21:53.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.985+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7004f330 con 0x7f5d7006bb70 2024-03-06T07:21:53.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.985+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d60038020 con 0x7f5d7006bb70 2024-03-06T07:21:53.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.985+0000 7f5d577fe640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d5c06b500 0x7f5d5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.986+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d600aba80 con 0x7f5d7006bb70 2024-03-06T07:21:53.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.986+0000 7f5d6e59c640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d5c06b500 0x7f5d5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.986+0000 7f5d6e59c640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d5c06b500 0x7f5d5c06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5d700af520 tx=0x7f5d64007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:53.990+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d70145d00 con 0x7f5d7006bb70 2024-03-06T07:21:54.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.162+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d7006ec20 con 0x7f5d7006bb70 2024-03-06T07:21:54.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.163+0000 7f5d577fe640 1 -- 172.21.15.122:0/2443555545 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16618 (secure 0 0 0) 0x7f5d6007a910 con 0x7f5d7006bb70 2024-03-06T07:21:54.165 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:54.165 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":19,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:52.887353+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d5c06b500 msgr2=0x7f5d5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d5c06b500 0x7f5d5c06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5d700af520 tx=0x7f5d64007b50 comp rx=0 tx=0).stop 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 msgr2=0x7f5d701457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 0x7f5d701457c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5d60034040 tx=0x7f5d600046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 shutdown_connections 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d5c06b500 0x7f5d5c06d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d700cf470 0x7f5d700d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d700730d0 0x7f5d701f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 --2- 172.21.15.122:0/2443555545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d7006bb70 0x7f5d701457c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 >> 172.21.15.122:0/2443555545 conn(0x7f5d700724d0 msgr2=0x7f5d700ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 shutdown_connections 2024-03-06T07:21:54.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.165+0000 7f5d750c2640 1 -- 172.21.15.122:0/2443555545 wait complete. 2024-03-06T07:21:54.178 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:54.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.278+0000 7fe24b94b640 1 Processor -- start 2024-03-06T07:21:54.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.278+0000 7fe24b94b640 1 -- start start 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe24b94b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe24415def0 0x7fe2440b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe24b94b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2440ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe24b94b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24406d030 con 0x7fe2440ce880 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe24b94b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2440cecf0 con 0x7fe24415def0 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe24b94b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440cf1c0 con 0x7fe24406cab0 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe2496e7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe24415def0 0x7fe2440b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe2496e7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe24415def0 0x7fe2440b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44840/0 (socket says 172.21.15.122:44840) 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.279+0000 7fe2496e7640 1 -- 172.21.15.122:0/3044724690 learned_addr learned my addr 172.21.15.122:0/3044724690 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe248ee6640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2440ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe2496e7640 1 -- 172.21.15.122:0/3044724690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 msgr2=0x7fe2440ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe2496e7640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2440ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe2496e7640 1 -- 172.21.15.122:0/3044724690 >> v1:172.21.15.169:6789/0 conn(0x7fe2440ce880 legacy=0x7fe2440cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe2496e7640 1 -- 172.21.15.122:0/3044724690 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24406d310 con 0x7fe24415def0 2024-03-06T07:21:54.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe248ee6640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2440ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe2496e7640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24415def0 0x7fe2440b41b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe234009e50 tx=0x7fe23402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3ed86266e65b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe243fff640 1 -- 172.21.15.122:0/3044724690 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23402cb90 con 0x7fe24415def0 2024-03-06T07:21:54.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.280+0000 7fe243fff640 1 -- 172.21.15.122:0/3044724690 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23402cd30 con 0x7fe24415def0 2024-03-06T07:21:54.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe243fff640 1 -- 172.21.15.122:0/3044724690 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234034610 con 0x7fe24415def0 2024-03-06T07:21:54.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 -- 172.21.15.122:0/3044724690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24415def0 msgr2=0x7fe2440b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24415def0 0x7fe2440b41b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe234009e50 tx=0x7fe23402bff0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 -- 172.21.15.122:0/3044724690 shutdown_connections 2024-03-06T07:21:54.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2440ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 --2- 172.21.15.122:0/3044724690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24415def0 0x7fe2440b41b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 -- 172.21.15.122:0/3044724690 >> 172.21.15.122:0/3044724690 conn(0x7fe2440b3690 msgr2=0x7fe2440b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 -- 172.21.15.122:0/3044724690 shutdown_connections 2024-03-06T07:21:54.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.281+0000 7fe24b94b640 1 -- 172.21.15.122:0/3044724690 wait complete. 2024-03-06T07:21:54.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.282+0000 7fe24b94b640 1 Processor -- start 2024-03-06T07:21:54.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.282+0000 7fe24b94b640 1 -- start start 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe24b94b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2441510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe24b94b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 0x7fe24406ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe24b94b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe24415def0 0x7fe24406f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe24b94b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440cc3f0 con 0x7fe2440ce880 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe24b94b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440cc760 con 0x7fe24415def0 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe24b94b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24415e2d0 con 0x7fe24406cab0 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 0x7fe24406ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 0x7fe24406ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44848/0 (socket says 172.21.15.122:44848) 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 -- 172.21.15.122:0/248692998 learned_addr learned my addr 172.21.15.122:0/248692998 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe2496e7640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2441510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe249ee8640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe24415def0 0x7fe24406f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 -- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 msgr2=0x7fe2441510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2441510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 -- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe24415def0 msgr2=0x7fe24406f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe24415def0 0x7fe24406f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.283+0000 7fe248ee6640 1 -- 172.21.15.122:0/248692998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2441f8150 con 0x7fe2440ce880 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.284+0000 7fe248ee6640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 0x7fe24406ecf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe23800e730 tx=0x7fe23800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.284+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23800f040 con 0x7fe2440ce880 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.284+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe238004230 con 0x7fe2440ce880 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.284+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2440c8b90 con 0x7fe2440ce880 2024-03-06T07:21:54.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.284+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23800ee40 con 0x7fe2440ce880 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.284+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2441f7670 con 0x7fe2440ce880 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.285+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24409fcf0 con 0x7fe2440ce880 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.285+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe23800c6d0 con 0x7fe2440ce880 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.286+0000 7fe241ffb640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23006b500 0x7fe23006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.286+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe238089e00 con 0x7fe2440ce880 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.286+0000 7fe2496e7640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23006b500 0x7fe23006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.286+0000 7fe2496e7640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23006b500 0x7fe23006d9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe234035000 tx=0x7fe234030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.291+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe238017020 con 0x7fe2440ce880 2024-03-06T07:21:54.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.498+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe2440d1a30 con 0x7fe2440ce880 2024-03-06T07:21:54.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.928+0000 7fe241ffb640 1 -- 172.21.15.122:0/248692998 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe2380586c0 con 0x7fe2440ce880 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23006b500 msgr2=0x7fe23006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23006b500 0x7fe23006d9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe234035000 tx=0x7fe234030000 comp rx=0 tx=0).stop 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 msgr2=0x7fe24406ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 0x7fe24406ecf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe23800e730 tx=0x7fe23800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 shutdown_connections 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23006b500 0x7fe23006d9c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe24415def0 0x7fe24406f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2440ce880 0x7fe24406ecf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 --2- 172.21.15.122:0/248692998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe24406cab0 0x7fe2441510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.930+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 >> 172.21.15.122:0/248692998 conn(0x7fe2440b3690 msgr2=0x7fe2440d2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.931+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 shutdown_connections 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:54.931+0000 7fe24b94b640 1 -- 172.21.15.122:0/248692998 wait complete. 2024-03-06T07:21:54.944 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:54.944 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:21:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.046+0000 7f77ff13e640 1 Processor -- start 2024-03-06T07:21:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.046+0000 7f77ff13e640 1 -- start start 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77ff13e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f77f815def0 0x7f77f80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77ff13e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f806cab0 0x7f77f80ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77ff13e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f77f80ce880 0x7f77f80cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77ff13e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f80cecf0 con 0x7f77f80ce880 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77ff13e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f80cf1c0 con 0x7f77f815def0 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77ff13e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80cc760 con 0x7f77f806cab0 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f77f80ce880 0x7f77f80cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f77f80ce880 0x7f77f80cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54370/0 (socket says 172.21.15.122:54370) 2024-03-06T07:21:55.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 -- 172.21.15.122:0/3122524816 learned_addr learned my addr 172.21.15.122:0/3122524816 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77f7fff640 1 --2- 172.21.15.122:0/3122524816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f806cab0 0x7f77f80ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fceda640 1 --2- 172.21.15.122:0/3122524816 >> v2:172.21.15.169:3300/0 conn(0x7f77f815def0 0x7f77f80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 -- 172.21.15.122:0/3122524816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f806cab0 msgr2=0x7f77f80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 --2- 172.21.15.122:0/3122524816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f806cab0 0x7f77f80ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 -- 172.21.15.122:0/3122524816 >> v2:172.21.15.169:3300/0 conn(0x7f77f815def0 msgr2=0x7f77f80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 --2- 172.21.15.122:0/3122524816 >> v2:172.21.15.169:3300/0 conn(0x7f77f815def0 0x7f77f80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.047+0000 7f77fd6db640 1 -- 172.21.15.122:0/3122524816 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f806d030 con 0x7f77f80ce880 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.048+0000 7f77fd6db640 1 --2- 172.21.15.122:0/3122524816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f80ce880 0x7f77f80cfd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77f8071f60 tx=0x7f77f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c3426d6b72a6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.048+0000 7f77f77fe640 1 -- 172.21.15.122:0/3122524816 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f002cb00 con 0x7f77f80ce880 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.048+0000 7f77f77fe640 1 -- 172.21.15.122:0/3122524816 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f002cca0 con 0x7f77f80ce880 2024-03-06T07:21:55.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.048+0000 7f77f77fe640 1 -- 172.21.15.122:0/3122524816 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0034610 con 0x7f77f80ce880 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 -- 172.21.15.122:0/3122524816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f80ce880 msgr2=0x7f77f80cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 --2- 172.21.15.122:0/3122524816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f80ce880 0x7f77f80cfd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77f8071f60 tx=0x7f77f002bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 -- 172.21.15.122:0/3122524816 shutdown_connections 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 --2- 172.21.15.122:0/3122524816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f80ce880 0x7f77f80cfd00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 --2- 172.21.15.122:0/3122524816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f806cab0 0x7f77f80ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 --2- 172.21.15.122:0/3122524816 >> v2:172.21.15.169:3300/0 conn(0x7f77f815def0 0x7f77f80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 -- 172.21.15.122:0/3122524816 >> 172.21.15.122:0/3122524816 conn(0x7f77f80b3690 msgr2=0x7f77f80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 -- 172.21.15.122:0/3122524816 shutdown_connections 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.049+0000 7f77ff13e640 1 -- 172.21.15.122:0/3122524816 wait complete. 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.050+0000 7f77ff13e640 1 Processor -- start 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.050+0000 7f77ff13e640 1 -- start start 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77f806cab0 0x7f77f81f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f80ce880 0x7f77f81f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 0x7f77f80b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80cc3f0 con 0x7f77f815def0 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80cc110 con 0x7f77f806cab0 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f815e2d0 con 0x7f77f80ce880 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 0x7f77f80b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 0x7f77f80b6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54382/0 (socket says 172.21.15.122:54382) 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 -- 172.21.15.122:0/800926154 learned_addr learned my addr 172.21.15.122:0/800926154 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77f7fff640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f80ce880 0x7f77f81f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fceda640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77f806cab0 0x7f77f81f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 -- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f80ce880 msgr2=0x7f77f81f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f80ce880 0x7f77f81f81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 -- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77f806cab0 msgr2=0x7f77f81f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77f806cab0 0x7f77f81f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 -- 172.21.15.122:0/800926154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f81f4860 con 0x7f77f815def0 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77fd6db640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 0x7f77f80b6920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f77f0036040 tx=0x7f77f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0043020 con 0x7f77f815def0 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f002ce50 con 0x7f77f815def0 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0009510 con 0x7f77f815def0 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.051+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f81f4b40 con 0x7f77f815def0 2024-03-06T07:21:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.052+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f80b48b0 con 0x7f77f815def0 2024-03-06T07:21:55.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.053+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f809fcf0 con 0x7f77f815def0 2024-03-06T07:21:55.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.053+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77f0046020 con 0x7f77f815def0 2024-03-06T07:21:55.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.053+0000 7f77f57fa640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77e406b4d0 0x7f77e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.053+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77f00ac4e0 con 0x7f77f815def0 2024-03-06T07:21:55.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.053+0000 7f77fceda640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77e406b4d0 0x7f77e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.054+0000 7f77fceda640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77e406b4d0 0x7f77e406d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009a20 tx=0x7f77e8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.058+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77f004d030 con 0x7f77f815def0 2024-03-06T07:21:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.265+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f77f80b4a80 con 0x7f77f815def0 2024-03-06T07:21:55.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.936+0000 7f77f57fa640 1 -- 172.21.15.122:0/800926154 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f77f007aec0 con 0x7f77f815def0 2024-03-06T07:21:55.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77e406b4d0 msgr2=0x7f77e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77e406b4d0 0x7f77e406d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009a20 tx=0x7f77e8009190 comp rx=0 tx=0).stop 2024-03-06T07:21:55.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 msgr2=0x7f77f80b6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 0x7f77f80b6920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f77f0036040 tx=0x7f77f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 shutdown_connections 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77e406b4d0 0x7f77e406d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77f815def0 0x7f77f80b6920 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77f80ce880 0x7f77f81f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 --2- 172.21.15.122:0/800926154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77f806cab0 0x7f77f81f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 >> 172.21.15.122:0/800926154 conn(0x7f77f80b3690 msgr2=0x7f77f80d21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 shutdown_connections 2024-03-06T07:21:55.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:55.939+0000 7f77ff13e640 1 -- 172.21.15.122:0/800926154 wait complete. 2024-03-06T07:21:55.952 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:21:55.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.055+0000 7f3ca0b8a640 1 Processor -- start 2024-03-06T07:21:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.055+0000 7f3ca0b8a640 1 -- start start 2024-03-06T07:21:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3ca0b8a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3c9c166af0 0x7f3c9c168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3ca0b8a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c15e380 con 0x7f3c9c0cf850 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3ca0b8a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c0cdc10 con 0x7f3c9c0d00c0 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3ca0b8a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9c06ca90 con 0x7f3c9c166af0 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3c99d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3c9c0cf850 0x7f3c9c0cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59568/0 (socket says 172.21.15.122:59568) 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3c99d9b640 1 -- 172.21.15.122:0/1604399195 learned_addr learned my addr 172.21.15.122:0/1604399195 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3c9ad9d640 1 --2- 172.21.15.122:0/1604399195 >> v2:172.21.15.169:3300/0 conn(0x7f3c9c166af0 0x7f3c9c168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3c9959a640 1 -- 172.21.15.122:0/1604399195 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886514892 0 0) 0x7f3c9c15e380 con 0x7f3c9c0cf850 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3c9959a640 1 -- 172.21.15.122:0/1604399195 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c84003520 con 0x7f3c9c0cf850 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.056+0000 7f3c9ad9d640 1 -- 172.21.15.122:0/1604399195 >> v1:172.21.15.169:6789/0 conn(0x7f3c9c0d00c0 legacy=0x7f3c9c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9959a640 1 -- 172.21.15.122:0/1604399195 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692737639 0 0) 0x7f3c9c0cdc10 con 0x7f3c9c0d00c0 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9ad9d640 1 -- 172.21.15.122:0/1604399195 >> v1:172.21.15.122:6789/0 conn(0x7f3c9c0cf850 legacy=0x7f3c9c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9ad9d640 1 -- 172.21.15.122:0/1604399195 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c06c160 con 0x7f3c9c166af0 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9ad9d640 1 --2- 172.21.15.122:0/1604399195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c168ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3c940097a0 tx=0x7f3c9402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27c953d7864f443 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9959a640 1 -- 172.21.15.122:0/1604399195 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94034070 con 0x7f3c9c166af0 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9959a640 1 -- 172.21.15.122:0/1604399195 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c94030610 con 0x7f3c9c166af0 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.057+0000 7f3c9959a640 1 -- 172.21.15.122:0/1604399195 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c940309c0 con 0x7f3c9c166af0 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/1604399195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 msgr2=0x7f3c9c168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/1604399195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c168ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3c940097a0 tx=0x7f3c9402b960 comp rx=0 tx=0).stop 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/1604399195 shutdown_connections 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/1604399195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c168ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/1604399195 >> 172.21.15.122:0/1604399195 conn(0x7f3c9c0b35a0 msgr2=0x7f3c9c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/1604399195 shutdown_connections 2024-03-06T07:21:56.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.058+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/1604399195 wait complete. 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.059+0000 7f3ca0b8a640 1 Processor -- start 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.059+0000 7f3ca0b8a640 1 -- start start 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3ca0b8a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 0x7f3c9c0c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3ca0b8a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c9c0d00c0 0x7f3c9c157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3ca0b8a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3ca0b8a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c06c720 con 0x7f3c9c0cf850 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3ca0b8a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c06df30 con 0x7f3c9c166af0 2024-03-06T07:21:56.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3ca0b8a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0b1030 con 0x7f3c9c0d00c0 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 0x7f3c9c0c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 0x7f3c9c0c7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54396/0 (socket says 172.21.15.122:54396) 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 -- 172.21.15.122:0/4009306972 learned_addr learned my addr 172.21.15.122:0/4009306972 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9ad9d640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c99d9b640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c9c0d00c0 0x7f3c9c157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 -- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c9c0d00c0 msgr2=0x7f3c9c157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c9c0d00c0 0x7f3c9c157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 -- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 msgr2=0x7f3c9c157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.060+0000 7f3c9a59c640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3c9a59c640 1 -- 172.21.15.122:0/4009306972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c0c6050 con 0x7f3c9c0cf850 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3c9ad9d640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3c9a59c640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 0x7f3c9c0c7a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c8c0 tx=0x7f3c8c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c017070 con 0x7f3c9c0cf850 2024-03-06T07:21:56.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c003dc0 con 0x7f3c9c0cf850 2024-03-06T07:21:56.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c0c62c0 con 0x7f3c9c0cf850 2024-03-06T07:21:56.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c0052b0 con 0x7f3c9c0cf850 2024-03-06T07:21:56.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.061+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c0c6880 con 0x7f3c9c0cf850 2024-03-06T07:21:56.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.062+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c04f330 con 0x7f3c9c0cf850 2024-03-06T07:21:56.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.062+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c8c01a070 con 0x7f3c9c0cf850 2024-03-06T07:21:56.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.063+0000 7f3c8affd640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.063+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c8c089880 con 0x7f3c9c0cf850 2024-03-06T07:21:56.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.063+0000 7f3c99d9b640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.063+0000 7f3c99d9b640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c06f020 tx=0x7f3c90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.067+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c8c0580b0 con 0x7f3c9c0cf850 2024-03-06T07:21:56.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.274+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c9c0d0d00 con 0x7f3c9c0cf850 2024-03-06T07:21:56.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.275+0000 7f3c8affd640 1 -- 172.21.15.122:0/4009306972 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f3c8c05c150 con 0x7f3c9c0cf850 2024-03-06T07:21:56.276 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:56.276 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:21:56.276 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.276+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c7c06b4d0 msgr2=0x7f3c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.276+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c06f020 tx=0x7f3c90009040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.276+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 msgr2=0x7f3c9c0c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 0x7f3c9c0c7a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c8c0 tx=0x7f3c8c00cd80 comp rx=0 tx=0).stop 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 shutdown_connections 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c9c166af0 0x7f3c9c157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c9c0d00c0 0x7f3c9c157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 --2- 172.21.15.122:0/4009306972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0cf850 0x7f3c9c0c7a60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 >> 172.21.15.122:0/4009306972 conn(0x7f3c9c0b35a0 msgr2=0x7f3c9c15d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 shutdown_connections 2024-03-06T07:21:56.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.277+0000 7f3ca0b8a640 1 -- 172.21.15.122:0/4009306972 wait complete. 2024-03-06T07:21:56.288 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.387+0000 7f05996a4640 1 Processor -- start 2024-03-06T07:21:56.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.387+0000 7f05996a4640 1 -- start start 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05996a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940cf850 0x7f05940cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05996a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059415e380 con 0x7f059406cf20 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05996a4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05940cdc10 con 0x7f05940d00c0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05996a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059406ca90 con 0x7f05940cf850 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05937fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f059406cf20 0x7f059406d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59578/0 (socket says 172.21.15.122:59578) 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05937fe640 1 -- 172.21.15.122:0/466350537 learned_addr learned my addr 172.21.15.122:0/466350537 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05927fc640 1 --2- 172.21.15.122:0/466350537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940cf850 0x7f05940cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f0591ffb640 1 -- 172.21.15.122:0/466350537 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886991398 0 0) 0x7f059415e380 con 0x7f059406cf20 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f0591ffb640 1 -- 172.21.15.122:0/466350537 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057c003520 con 0x7f059406cf20 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f0591ffb640 1 -- 172.21.15.122:0/466350537 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235004625 0 0) 0x7f05940cdc10 con 0x7f05940d00c0 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05927fc640 1 -- 172.21.15.122:0/466350537 >> v1:172.21.15.169:6789/0 conn(0x7f05940d00c0 legacy=0x7f05940d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05927fc640 1 -- 172.21.15.122:0/466350537 >> v1:172.21.15.122:6789/0 conn(0x7f059406cf20 legacy=0x7f059406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.388+0000 7f05927fc640 1 -- 172.21.15.122:0/466350537 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059406c160 con 0x7f05940cf850 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.389+0000 7f05927fc640 1 --2- 172.21.15.122:0/466350537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940cf850 0x7f05940cd3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05880097d0 tx=0x7f058802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb270684f1a1b4ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.389+0000 7f0591ffb640 1 -- 172.21.15.122:0/466350537 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588035020 con 0x7f05940cf850 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.389+0000 7f0591ffb640 1 -- 172.21.15.122:0/466350537 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0588031890 con 0x7f05940cf850 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.389+0000 7f0591ffb640 1 -- 172.21.15.122:0/466350537 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588031c40 con 0x7f05940cf850 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 -- 172.21.15.122:0/466350537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940cf850 msgr2=0x7f05940cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 --2- 172.21.15.122:0/466350537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940cf850 0x7f05940cd3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05880097d0 tx=0x7f058802b930 comp rx=0 tx=0).stop 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 -- 172.21.15.122:0/466350537 shutdown_connections 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 --2- 172.21.15.122:0/466350537 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940cf850 0x7f05940cd3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 -- 172.21.15.122:0/466350537 >> 172.21.15.122:0/466350537 conn(0x7f05940b35a0 msgr2=0x7f05940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 -- 172.21.15.122:0/466350537 shutdown_connections 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.390+0000 7f05996a4640 1 -- 172.21.15.122:0/466350537 wait complete. 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.391+0000 7f05996a4640 1 Processor -- start 2024-03-06T07:21:56.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.391+0000 7f05996a4640 1 -- start start 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05996a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f059406cf20 0x7f0594144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05996a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 0x7f05940c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05996a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940d00c0 0x7f05940b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05996a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05940cd900 con 0x7f05940cf850 2024-03-06T07:21:56.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05996a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059406c720 con 0x7f059406cf20 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05996a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05940b1030 con 0x7f05940d00c0 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 0x7f05940c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 0x7f05940c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54400/0 (socket says 172.21.15.122:54400) 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 -- 172.21.15.122:0/498387644 learned_addr learned my addr 172.21.15.122:0/498387644 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05937fe640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940d00c0 0x7f05940b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f0592ffd640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f059406cf20 0x7f0594144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 -- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940d00c0 msgr2=0x7f05940b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940d00c0 0x7f05940b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 -- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f059406cf20 msgr2=0x7f0594144c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f059406cf20 0x7f0594144c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 -- 172.21.15.122:0/498387644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940b04a0 con 0x7f05940cf850 2024-03-06T07:21:56.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.392+0000 7f05927fc640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 0x7f05940c7d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0588000c00 tx=0x7f05880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.393+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588035040 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.393+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0588030d20 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.393+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05880466e0 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.393+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05940b0780 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.393+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0594158510 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.394+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059404f330 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.394+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0588038020 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.394+0000 7f057b7fe640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f058006b4d0 0x7f058006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.394+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05880ac7c0 con 0x7f05940cf850 2024-03-06T07:21:56.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.394+0000 7f0592ffd640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f058006b4d0 0x7f058006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.395+0000 7f0592ffd640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f058006b4d0 0x7f058006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0584000f60 tx=0x7f0584009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.399+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f058802d040 con 0x7f05940cf850 2024-03-06T07:21:56.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.570+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05940ca910 con 0x7f05940cf850 2024-03-06T07:21:56.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.571+0000 7f057b7fe640 1 -- 172.21.15.122:0/498387644 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f058807aff0 con 0x7f05940cf850 2024-03-06T07:21:56.572 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:56.572 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f058006b4d0 msgr2=0x7f058006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f058006b4d0 0x7f058006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0584000f60 tx=0x7f0584009040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 msgr2=0x7f05940c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 0x7f05940c7d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0588000c00 tx=0x7f05880046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 shutdown_connections 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f058006b4d0 0x7f058006d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05940d00c0 0x7f05940b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05940cf850 0x7f05940c7d10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 --2- 172.21.15.122:0/498387644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f059406cf20 0x7f0594144c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 >> 172.21.15.122:0/498387644 conn(0x7f05940b35a0 msgr2=0x7f05940b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 shutdown_connections 2024-03-06T07:21:56.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.573+0000 7f05996a4640 1 -- 172.21.15.122:0/498387644 wait complete. 2024-03-06T07:21:56.585 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:21:56.585 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.683+0000 7fd277e34640 1 Processor -- start 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.683+0000 7fd277e34640 1 -- start start 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd277e34640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2700d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd277e34640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2700cc8b0 0x7fd2700ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd277e34640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270161980 con 0x7fd27015bb20 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd277e34640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2700d2500 con 0x7fd2700cc8b0 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd277e34640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd270161c60 con 0x7fd2700cfff0 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2700cc8b0 0x7fd2700ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2700cc8b0 0x7fd2700ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54412/0 (socket says 172.21.15.122:54412) 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd275bd0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd27015bb20 0x7fd2700d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59590/0 (socket says 172.21.15.122:59590) 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 -- 172.21.15.122:0/3216978708 learned_addr learned my addr 172.21.15.122:0/3216978708 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2753cf640 1 --2- 172.21.15.122:0/3216978708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2700d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 -- 172.21.15.122:0/3216978708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 msgr2=0x7fd2700d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 --2- 172.21.15.122:0/3216978708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2700d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 -- 172.21.15.122:0/3216978708 >> v1:172.21.15.122:6789/0 conn(0x7fd27015bb20 legacy=0x7fd2700d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd274bce640 1 -- 172.21.15.122:0/3216978708 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239629221 0 0) 0x7fd270161980 con 0x7fd27015bb20 2024-03-06T07:21:56.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.684+0000 7fd2763d1640 1 -- 172.21.15.122:0/3216978708 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd270161f40 con 0x7fd2700cc8b0 2024-03-06T07:21:56.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd2763d1640 1 --2- 172.21.15.122:0/3216978708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700ccd40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd268009e50 tx=0x7fd26802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36644be90c115f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd274bce640 1 -- 172.21.15.122:0/3216978708 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26802cb00 con 0x7fd2700cc8b0 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd274bce640 1 -- 172.21.15.122:0/3216978708 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26802cca0 con 0x7fd2700cc8b0 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd274bce640 1 -- 172.21.15.122:0/3216978708 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268034610 con 0x7fd2700cc8b0 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd277e34640 1 -- 172.21.15.122:0/3216978708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 msgr2=0x7fd2700ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd277e34640 1 --2- 172.21.15.122:0/3216978708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700ccd40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd268009e50 tx=0x7fd26802bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd277e34640 1 -- 172.21.15.122:0/3216978708 shutdown_connections 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd277e34640 1 --2- 172.21.15.122:0/3216978708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700ccd40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd277e34640 1 --2- 172.21.15.122:0/3216978708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2700d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.685+0000 7fd277e34640 1 -- 172.21.15.122:0/3216978708 >> 172.21.15.122:0/3216978708 conn(0x7fd27009a9f0 msgr2=0x7fd27009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.686+0000 7fd277e34640 1 -- 172.21.15.122:0/3216978708 shutdown_connections 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.686+0000 7fd277e34640 1 -- 172.21.15.122:0/3216978708 wait complete. 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 Processor -- start 2024-03-06T07:21:56.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 -- start start 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2701f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd27015bb20 0x7fd270159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2700d1980 con 0x7fd2700cc8b0 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd270162500 con 0x7fd27015bb20 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd277e34640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2700d1b00 con 0x7fd2700cfff0 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd275bd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd275bd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54414/0 (socket says 172.21.15.122:54414) 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd275bd0640 1 -- 172.21.15.122:0/3905890253 learned_addr learned my addr 172.21.15.122:0/3905890253 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.687+0000 7fd2753cf640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2701f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd2763d1640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd27015bb20 0x7fd270159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd275bd0640 1 -- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 msgr2=0x7fd2701f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd275bd0640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2701f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd275bd0640 1 -- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd27015bb20 msgr2=0x7fd270159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd275bd0640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd27015bb20 0x7fd270159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd275bd0640 1 -- 172.21.15.122:0/3905890253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2701f75d0 con 0x7fd2700cc8b0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd275bd0640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700caae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00c460 tx=0x7fd26c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c018070 con 0x7fd2700cc8b0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c007500 con 0x7fd2700cc8b0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2701f78b0 con 0x7fd2700cc8b0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c004a20 con 0x7fd2700cc8b0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.688+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27015a4c0 con 0x7fd2700cc8b0 2024-03-06T07:21:56.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.689+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27004f330 con 0x7fd2700cc8b0 2024-03-06T07:21:56.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.689+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd26c01b070 con 0x7fd2700cc8b0 2024-03-06T07:21:56.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.690+0000 7fd2667fc640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd25406b4d0 0x7fd25406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.690+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd26c0893b0 con 0x7fd2700cc8b0 2024-03-06T07:21:56.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.690+0000 7fd2753cf640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd25406b4d0 0x7fd25406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.690+0000 7fd2753cf640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd25406b4d0 0x7fd25406d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd27006df30 tx=0x7fd260009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.694+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd26c057be0 con 0x7fd2700cc8b0 2024-03-06T07:21:56.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.860+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2701f7dd0 con 0x7fd25406b4d0 2024-03-06T07:21:56.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.866+0000 7fd2667fc640 1 -- 172.21.15.122:0/3905890253 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237079 (secure 0 0 0) 0x7fd2600119d0 con 0x7fd25406b4d0 2024-03-06T07:21:56.870 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:56.871 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd25406b4d0 msgr2=0x7fd25406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd25406b4d0 0x7fd25406d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd27006df30 tx=0x7fd260009190 comp rx=0 tx=0).stop 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 msgr2=0x7fd2700caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700caae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00c460 tx=0x7fd26c00c920 comp rx=0 tx=0).stop 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 shutdown_connections 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd25406b4d0 0x7fd25406d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd27015bb20 0x7fd270159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2700cfff0 0x7fd2701f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 --2- 172.21.15.122:0/3905890253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2700cc8b0 0x7fd2700caae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 >> 172.21.15.122:0/3905890253 conn(0x7fd27009a9f0 msgr2=0x7fd2700b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 shutdown_connections 2024-03-06T07:21:56.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:56.869+0000 7fd277e34640 1 -- 172.21.15.122:0/3905890253 wait complete. 2024-03-06T07:21:56.881 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:21:55.330497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219300,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755420,"statfs":{"total":773094113280,"available":772869550080,"internally_reserved":0,"allocated":2433024,"data_stored":1546872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000751"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-03-06T07:21:43.848299+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:43.848299+0000","last_peered":"2024-03-06T07:21:43.848299+0000","last_clean":"2024-03-06T07:21:43.848299+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:43.848299+0000","last_undegraded":"2024-03-06T07:21:43.848299+0000","last_fullsized":"2024-03-06T07:21:43.848299+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:56.885 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:57.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.018+0000 7efd58a0e640 1 Processor -- start 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.018+0000 7efd58a0e640 1 -- start start 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd58a0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540d0320 0x7efd540b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd58a0e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd540b54b0 0x7efd540cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd58a0e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5415c270 con 0x7efd540d0700 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd58a0e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd540b04e0 con 0x7efd540b54b0 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd58a0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd540d0e90 con 0x7efd540d0320 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd52d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd540b54b0 0x7efd540cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd5259c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540d0320 0x7efd540b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd52d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd540b54b0 0x7efd540cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54426/0 (socket says 172.21.15.122:54426) 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd5259c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540d0320 0x7efd540b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44354/0 (socket says 172.21.15.122:44354) 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.019+0000 7efd52d9d640 1 -- 172.21.15.122:0/4084397841 learned_addr learned my addr 172.21.15.122:0/4084397841 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.020+0000 7efd52d9d640 1 -- 172.21.15.122:0/4084397841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540d0320 msgr2=0x7efd540b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.020+0000 7efd52d9d640 1 --2- 172.21.15.122:0/4084397841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540d0320 0x7efd540b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.020+0000 7efd52d9d640 1 -- 172.21.15.122:0/4084397841 >> v1:172.21.15.169:6789/0 conn(0x7efd540d0700 legacy=0x7efd540ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.020+0000 7efd52d9d640 1 -- 172.21.15.122:0/4084397841 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd540b5ae0 con 0x7efd540b54b0 2024-03-06T07:21:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.020+0000 7efd52d9d640 1 --2- 172.21.15.122:0/4084397841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540b54b0 0x7efd540cd1f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009e50 tx=0x7efd4c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7e1a09afc90ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd5159a640 1 -- 172.21.15.122:0/4084397841 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c02cb00 con 0x7efd540b54b0 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd5159a640 1 -- 172.21.15.122:0/4084397841 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c02cca0 con 0x7efd540b54b0 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd5159a640 1 -- 172.21.15.122:0/4084397841 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c034610 con 0x7efd540b54b0 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 -- 172.21.15.122:0/4084397841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540b54b0 msgr2=0x7efd540cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 --2- 172.21.15.122:0/4084397841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540b54b0 0x7efd540cd1f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009e50 tx=0x7efd4c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 -- 172.21.15.122:0/4084397841 shutdown_connections 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 --2- 172.21.15.122:0/4084397841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540b54b0 0x7efd540cd1f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 --2- 172.21.15.122:0/4084397841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540d0320 0x7efd540b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 -- 172.21.15.122:0/4084397841 >> 172.21.15.122:0/4084397841 conn(0x7efd540724d0 msgr2=0x7efd540728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 -- 172.21.15.122:0/4084397841 shutdown_connections 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.021+0000 7efd58a0e640 1 -- 172.21.15.122:0/4084397841 wait complete. 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.022+0000 7efd58a0e640 1 Processor -- start 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.022+0000 7efd58a0e640 1 -- start start 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd58a0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540b54b0 0x7efd54145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd58a0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 0x7efd54157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd58a0e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd540d0700 0x7efd54158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd58a0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540b60a0 con 0x7efd540d0320 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd58a0e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540b0170 con 0x7efd540d0700 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd58a0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd540ca790 con 0x7efd540b54b0 2024-03-06T07:21:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 0x7efd54157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 0x7efd54157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54440/0 (socket says 172.21.15.122:54440) 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 -- 172.21.15.122:0/2812869762 learned_addr learned my addr 172.21.15.122:0/2812869762 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd5259c640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540b54b0 0x7efd54145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd52d9d640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd540d0700 0x7efd54158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 -- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540b54b0 msgr2=0x7efd54145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540b54b0 0x7efd54145530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 -- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd540d0700 msgr2=0x7efd54158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.023+0000 7efd51d9b640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd540d0700 0x7efd54158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd51d9b640 1 -- 172.21.15.122:0/2812869762 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54145a70 con 0x7efd540d0320 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd51d9b640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 0x7efd54157f50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd4800c400 tx=0x7efd4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48017070 con 0x7efd540d0320 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd480052d0 con 0x7efd540d0320 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd54146120 con 0x7efd540d0320 2024-03-06T07:21:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48003830 con 0x7efd540d0320 2024-03-06T07:21:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.024+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd541466e0 con 0x7efd540d0320 2024-03-06T07:21:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.025+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5404f330 con 0x7efd540d0320 2024-03-06T07:21:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.025+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd4801a070 con 0x7efd540d0320 2024-03-06T07:21:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.026+0000 7efd42ffd640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd3406b4d0 0x7efd3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.026+0000 7efd5259c640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd3406b4d0 0x7efd3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.026+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd480894b0 con 0x7efd540d0320 2024-03-06T07:21:57.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.026+0000 7efd5259c640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd3406b4d0 0x7efd3406d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd44009b40 tx=0x7efd44009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.030+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd54145a70 con 0x7efd540d0320 2024-03-06T07:21:57.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.200+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd54073190 con 0x7efd3406b4d0 2024-03-06T07:21:57.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.205+0000 7efd42ffd640 1 -- 172.21.15.122:0/2812869762 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237079 (secure 0 0 0) 0x7efd54073190 con 0x7efd3406b4d0 2024-03-06T07:21:57.209 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:57.210 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:21:57.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd3406b4d0 msgr2=0x7efd3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd3406b4d0 0x7efd3406d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd44009b40 tx=0x7efd44009190 comp rx=0 tx=0).stop 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 msgr2=0x7efd54157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 0x7efd54157f50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd4800c400 tx=0x7efd4800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 shutdown_connections 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd3406b4d0 0x7efd3406d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd540d0700 0x7efd54158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd540d0320 0x7efd54157f50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 --2- 172.21.15.122:0/2812869762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd540b54b0 0x7efd54145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.208+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 >> 172.21.15.122:0/2812869762 conn(0x7efd540724d0 msgr2=0x7efd540ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.209+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 shutdown_connections 2024-03-06T07:21:57.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.209+0000 7efd58a0e640 1 -- 172.21.15.122:0/2812869762 wait complete. 2024-03-06T07:21:57.220 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:21:55.330497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219300,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755420,"statfs":{"total":773094113280,"available":772869550080,"internally_reserved":0,"allocated":2433024,"data_stored":1546872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000751"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-03-06T07:21:43.848299+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:43.848299+0000","last_peered":"2024-03-06T07:21:43.848299+0000","last_clean":"2024-03-06T07:21:43.848299+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:43.848299+0000","last_undegraded":"2024-03-06T07:21:43.848299+0000","last_fullsized":"2024-03-06T07:21:43.848299+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:52.532052+0000","last_change":"2024-03-06T07:21:52.532052+0000","last_active":"2024-03-06T07:21:52.532052+0000","last_peered":"2024-03-06T07:21:52.532052+0000","last_clean":"2024-03-06T07:21:52.532052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:52.532052+0000","last_undegraded":"2024-03-06T07:21:52.532052+0000","last_fullsized":"2024-03-06T07:21:52.532052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:51.527333+0000","last_change":"2024-03-06T07:21:51.527333+0000","last_active":"2024-03-06T07:21:51.527333+0000","last_peered":"2024-03-06T07:21:51.527333+0000","last_clean":"2024-03-06T07:21:51.527333+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:51.527333+0000","last_undegraded":"2024-03-06T07:21:51.527333+0000","last_fullsized":"2024-03-06T07:21:51.527333+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:57.224 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:57.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.360+0000 7f2fdbcc2640 1 Processor -- start 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.360+0000 7f2fdbcc2640 1 -- start start 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fdbcc2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2fd40b4f70 0x7f2fd40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fdbcc2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40b5890 con 0x7f2fd406c6d0 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fdbcc2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40cd450 con 0x7f2fd4066d90 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fdbcc2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd415e9f0 con 0x7f2fd40b4f70 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fd9a5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2fd40b4f70 0x7f2fd40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fd925d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2fd406c6d0 0x7f2fd415ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59606/0 (socket says 172.21.15.122:59606) 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.361+0000 7f2fd925d640 1 -- 172.21.15.122:0/2190615663 learned_addr learned my addr 172.21.15.122:0/2190615663 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd9a5e640 1 -- 172.21.15.122:0/2190615663 >> v1:172.21.15.169:6789/0 conn(0x7f2fd4066d90 legacy=0x7f2fd40ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd9a5e640 1 -- 172.21.15.122:0/2190615663 >> v1:172.21.15.122:6789/0 conn(0x7f2fd406c6d0 legacy=0x7f2fd415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd8a5c640 1 -- 172.21.15.122:0/2190615663 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774222054 0 0) 0x7f2fd40b5890 con 0x7f2fd406c6d0 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd9a5e640 1 -- 172.21.15.122:0/2190615663 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40cd730 con 0x7f2fd40b4f70 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd9a5e640 1 --2- 172.21.15.122:0/2190615663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40b5350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009d90 tx=0x7f2fd002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f92615c08616e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd8a5c640 1 -- 172.21.15.122:0/2190615663 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0039070 con 0x7f2fd40b4f70 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd8a5c640 1 -- 172.21.15.122:0/2190615663 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd002cd00 con 0x7f2fd40b4f70 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.362+0000 7f2fd8a5c640 1 -- 172.21.15.122:0/2190615663 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0034660 con 0x7f2fd40b4f70 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/2190615663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd40b4f70 msgr2=0x7f2fd40b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/2190615663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40b5350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009d90 tx=0x7f2fd002bf50 comp rx=0 tx=0).stop 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/2190615663 shutdown_connections 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/2190615663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40b5350 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/2190615663 >> 172.21.15.122:0/2190615663 conn(0x7f2fd40724d0 msgr2=0x7f2fd40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/2190615663 shutdown_connections 2024-03-06T07:21:57.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.363+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/2190615663 wait complete. 2024-03-06T07:21:57.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.364+0000 7f2fdbcc2640 1 Processor -- start 2024-03-06T07:21:57.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.364+0000 7f2fdbcc2640 1 -- start start 2024-03-06T07:21:57.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fdbcc2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2fd4066d90 0x7f2fd40cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fdbcc2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fdbcc2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 0x7f2fd41606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fdbcc2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4161ad0 con 0x7f2fd41601f0 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fdbcc2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40cda10 con 0x7f2fd40b4f70 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fdbcc2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4161c50 con 0x7f2fd4066d90 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 0x7f2fd41606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 0x7f2fd41606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54468/0 (socket says 172.21.15.122:54468) 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 -- 172.21.15.122:0/1180248763 learned_addr learned my addr 172.21.15.122:0/1180248763 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fd9a5e640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2fd4066d90 0x7f2fd40cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fd925d640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 -- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2fd4066d90 msgr2=0x7f2fd40cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2fd4066d90 0x7f2fd40cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 -- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fd40b4f70 msgr2=0x7f2fd40cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 -- 172.21.15.122:0/1180248763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40cf890 con 0x7f2fd41601f0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.365+0000 7f2fda25f640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 0x7f2fd41606a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00c3a0 tx=0x7f2fcc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.366+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc017070 con 0x7f2fd41601f0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.366+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc0052d0 con 0x7f2fd41601f0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.366+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd40ce140 con 0x7f2fd41601f0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.366+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc003830 con 0x7f2fd41601f0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.366+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd40ce700 con 0x7f2fd41601f0 2024-03-06T07:21:57.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.367+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4068280 con 0x7f2fd41601f0 2024-03-06T07:21:57.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.367+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fcc01a070 con 0x7f2fd41601f0 2024-03-06T07:21:57.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.367+0000 7f2fca7fc640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2fb806b500 0x7f2fb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.368+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fcc089390 con 0x7f2fd41601f0 2024-03-06T07:21:57.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.368+0000 7f2fd9a5e640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2fb806b500 0x7f2fb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.368+0000 7f2fd9a5e640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2fb806b500 0x7f2fb806d9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0036000 tx=0x7f2fd0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.372+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd4068280 con 0x7f2fd41601f0 2024-03-06T07:21:57.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.540+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fd4160be0 con 0x7f2fb806b500 2024-03-06T07:21:57.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.546+0000 7f2fca7fc640 1 -- 172.21.15.122:0/1180248763 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251944 (secure 0 0 0) 0x7f2fd4160be0 con 0x7f2fb806b500 2024-03-06T07:21:57.551 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:57.553 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:21:57.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2fb806b500 msgr2=0x7f2fb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2fb806b500 0x7f2fb806d9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0036000 tx=0x7f2fd0030000 comp rx=0 tx=0).stop 2024-03-06T07:21:57.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 msgr2=0x7f2fd41606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 0x7f2fd41606a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00c3a0 tx=0x7f2fcc00c860 comp rx=0 tx=0).stop 2024-03-06T07:21:57.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 shutdown_connections 2024-03-06T07:21:57.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2fb806b500 0x7f2fb806d9c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fd41601f0 0x7f2fd41606a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fd40b4f70 0x7f2fd40cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 --2- 172.21.15.122:0/1180248763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2fd4066d90 0x7f2fd40cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 >> 172.21.15.122:0/1180248763 conn(0x7f2fd40724d0 msgr2=0x7f2fd406d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 shutdown_connections 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.549+0000 7f2fdbcc2640 1 -- 172.21.15.122:0/1180248763 wait complete. 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-06T07:21:57.330908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218948,"kb_used_data":2752,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755772,"statfs":{"total":773094113280,"available":772869910528,"internally_reserved":0,"allocated":2818048,"data_stored":1715228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.798716"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778717+0000","last_change":"2024-03-06T07:21:53.563999+0000","last_active":"2024-03-06T07:21:54.778717+0000","last_peered":"2024-03-06T07:21:54.778717+0000","last_clean":"2024-03-06T07:21:54.778717+0000","last_became_active":"2024-03-06T07:21:53.563557+0000","last_became_peered":"2024-03-06T07:21:53.563557+0000","last_unstale":"2024-03-06T07:21:54.778717+0000","last_undegraded":"2024-03-06T07:21:54.778717+0000","last_fullsized":"2024-03-06T07:21:54.778717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562641+0000","last_change":"2024-03-06T07:21:53.560952+0000","last_active":"2024-03-06T07:21:54.562641+0000","last_peered":"2024-03-06T07:21:54.562641+0000","last_clean":"2024-03-06T07:21:54.562641+0000","last_became_active":"2024-03-06T07:21:53.560651+0000","last_became_peered":"2024-03-06T07:21:53.560651+0000","last_unstale":"2024-03-06T07:21:54.562641+0000","last_undegraded":"2024-03-06T07:21:54.562641+0000","last_fullsized":"2024-03-06T07:21:54.562641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777337+0000","last_change":"2024-03-06T07:21:53.547890+0000","last_active":"2024-03-06T07:21:54.777337+0000","last_peered":"2024-03-06T07:21:54.777337+0000","last_clean":"2024-03-06T07:21:54.777337+0000","last_became_active":"2024-03-06T07:21:53.547631+0000","last_became_peered":"2024-03-06T07:21:53.547631+0000","last_unstale":"2024-03-06T07:21:54.777337+0000","last_undegraded":"2024-03-06T07:21:54.777337+0000","last_fullsized":"2024-03-06T07:21:54.777337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778291+0000","last_change":"2024-03-06T07:21:53.565535+0000","last_active":"2024-03-06T07:21:54.778291+0000","last_peered":"2024-03-06T07:21:54.778291+0000","last_clean":"2024-03-06T07:21:54.778291+0000","last_became_active":"2024-03-06T07:21:53.565256+0000","last_became_peered":"2024-03-06T07:21:53.565256+0000","last_unstale":"2024-03-06T07:21:54.778291+0000","last_undegraded":"2024-03-06T07:21:54.778291+0000","last_fullsized":"2024-03-06T07:21:54.778291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778186+0000","last_change":"2024-03-06T07:21:53.558864+0000","last_active":"2024-03-06T07:21:54.778186+0000","last_peered":"2024-03-06T07:21:54.778186+0000","last_clean":"2024-03-06T07:21:54.778186+0000","last_became_active":"2024-03-06T07:21:53.558582+0000","last_became_peered":"2024-03-06T07:21:53.558582+0000","last_unstale":"2024-03-06T07:21:54.778186+0000","last_undegraded":"2024-03-06T07:21:54.778186+0000","last_fullsized":"2024-03-06T07:21:54.778186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777685+0000","last_change":"2024-03-06T07:21:53.567354+0000","last_active":"2024-03-06T07:21:54.777685+0000","last_peered":"2024-03-06T07:21:54.777685+0000","last_clean":"2024-03-06T07:21:54.777685+0000","last_became_active":"2024-03-06T07:21:53.567054+0000","last_became_peered":"2024-03-06T07:21:53.567054+0000","last_unstale":"2024-03-06T07:21:54.777685+0000","last_undegraded":"2024-03-06T07:21:54.777685+0000","last_fullsized":"2024-03-06T07:21:54.777685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777444+0000","last_change":"2024-03-06T07:21:53.563005+0000","last_active":"2024-03-06T07:21:54.777444+0000","last_peered":"2024-03-06T07:21:54.777444+0000","last_clean":"2024-03-06T07:21:54.777444+0000","last_became_active":"2024-03-06T07:21:53.562611+0000","last_became_peered":"2024-03-06T07:21:53.562611+0000","last_unstale":"2024-03-06T07:21:54.777444+0000","last_undegraded":"2024-03-06T07:21:54.777444+0000","last_fullsized":"2024-03-06T07:21:54.777444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.766995+0000","last_change":"2024-03-06T07:21:53.566285+0000","last_active":"2024-03-06T07:21:54.766995+0000","last_peered":"2024-03-06T07:21:54.766995+0000","last_clean":"2024-03-06T07:21:54.766995+0000","last_became_active":"2024-03-06T07:21:53.565429+0000","last_became_peered":"2024-03-06T07:21:53.565429+0000","last_unstale":"2024-03-06T07:21:54.766995+0000","last_undegraded":"2024-03-06T07:21:54.766995+0000","last_fullsized":"2024-03-06T07:21:54.766995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778770+0000","last_change":"2024-03-06T07:21:53.565916+0000","last_active":"2024-03-06T07:21:54.778770+0000","last_peered":"2024-03-06T07:21:54.778770+0000","last_clean":"2024-03-06T07:21:54.778770+0000","last_became_active":"2024-03-06T07:21:53.565555+0000","last_became_peered":"2024-03-06T07:21:53.565555+0000","last_unstale":"2024-03-06T07:21:54.778770+0000","last_undegraded":"2024-03-06T07:21:54.778770+0000","last_fullsized":"2024-03-06T07:21:54.778770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:23.946424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562690+0000","last_change":"2024-03-06T07:21:53.557656+0000","last_active":"2024-03-06T07:21:54.562690+0000","last_peered":"2024-03-06T07:21:54.562690+0000","last_clean":"2024-03-06T07:21:54.562690+0000","last_became_active":"2024-03-06T07:21:53.557224+0000","last_became_peered":"2024-03-06T07:21:53.557224+0000","last_unstale":"2024-03-06T07:21:54.562690+0000","last_undegraded":"2024-03-06T07:21:54.562690+0000","last_fullsized":"2024-03-06T07:21:54.562690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562612+0000","last_change":"2024-03-06T07:21:53.565698+0000","last_active":"2024-03-06T07:21:54.562612+0000","last_peered":"2024-03-06T07:21:54.562612+0000","last_clean":"2024-03-06T07:21:54.562612+0000","last_became_active":"2024-03-06T07:21:53.565390+0000","last_became_peered":"2024-03-06T07:21:53.565390+0000","last_unstale":"2024-03-06T07:21:54.562612+0000","last_undegraded":"2024-03-06T07:21:54.562612+0000","last_fullsized":"2024-03-06T07:21:54.562612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777702+0000","last_change":"2024-03-06T07:21:53.558368+0000","last_active":"2024-03-06T07:21:54.777702+0000","last_peered":"2024-03-06T07:21:54.777702+0000","last_clean":"2024-03-06T07:21:54.777702+0000","last_became_active":"2024-03-06T07:21:53.558065+0000","last_became_peered":"2024-03-06T07:21:53.558065+0000","last_unstale":"2024-03-06T07:21:54.777702+0000","last_undegraded":"2024-03-06T07:21:54.777702+0000","last_fullsized":"2024-03-06T07:21:54.777702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:44.794765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767222+0000","last_change":"2024-03-06T07:21:53.567165+0000","last_active":"2024-03-06T07:21:54.767222+0000","last_peered":"2024-03-06T07:21:54.767222+0000","last_clean":"2024-03-06T07:21:54.767222+0000","last_became_active":"2024-03-06T07:21:53.566826+0000","last_became_peered":"2024-03-06T07:21:53.566826+0000","last_unstale":"2024-03-06T07:21:54.767222+0000","last_undegraded":"2024-03-06T07:21:54.767222+0000","last_fullsized":"2024-03-06T07:21:54.767222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777131+0000","last_change":"2024-03-06T07:21:53.552931+0000","last_active":"2024-03-06T07:21:54.777131+0000","last_peered":"2024-03-06T07:21:54.777131+0000","last_clean":"2024-03-06T07:21:54.777131+0000","last_became_active":"2024-03-06T07:21:53.552654+0000","last_became_peered":"2024-03-06T07:21:53.552654+0000","last_unstale":"2024-03-06T07:21:54.777131+0000","last_undegraded":"2024-03-06T07:21:54.777131+0000","last_fullsized":"2024-03-06T07:21:54.777131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:04.669227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562518+0000","last_change":"2024-03-06T07:21:53.562294+0000","last_active":"2024-03-06T07:21:54.562518+0000","last_peered":"2024-03-06T07:21:54.562518+0000","last_clean":"2024-03-06T07:21:54.562518+0000","last_became_active":"2024-03-06T07:21:53.561996+0000","last_became_peered":"2024-03-06T07:21:53.561996+0000","last_unstale":"2024-03-06T07:21:54.562518+0000","last_undegraded":"2024-03-06T07:21:54.562518+0000","last_fullsized":"2024-03-06T07:21:54.562518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777264+0000","last_change":"2024-03-06T07:21:53.563516+0000","last_active":"2024-03-06T07:21:54.777264+0000","last_peered":"2024-03-06T07:21:54.777264+0000","last_clean":"2024-03-06T07:21:54.777264+0000","last_became_active":"2024-03-06T07:21:53.563237+0000","last_became_peered":"2024-03-06T07:21:53.563237+0000","last_unstale":"2024-03-06T07:21:54.777264+0000","last_undegraded":"2024-03-06T07:21:54.777264+0000","last_fullsized":"2024-03-06T07:21:54.777264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778868+0000","last_change":"2024-03-06T07:21:53.561815+0000","last_active":"2024-03-06T07:21:54.778868+0000","last_peered":"2024-03-06T07:21:54.778868+0000","last_clean":"2024-03-06T07:21:54.778868+0000","last_became_active":"2024-03-06T07:21:53.561189+0000","last_became_peered":"2024-03-06T07:21:53.561189+0000","last_unstale":"2024-03-06T07:21:54.778868+0000","last_undegraded":"2024-03-06T07:21:54.778868+0000","last_fullsized":"2024-03-06T07:21:54.778868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560237+0000","last_change":"2024-03-06T07:21:53.560903+0000","last_active":"2024-03-06T07:21:54.560237+0000","last_peered":"2024-03-06T07:21:54.560237+0000","last_clean":"2024-03-06T07:21:54.560237+0000","last_became_active":"2024-03-06T07:21:53.560624+0000","last_became_peered":"2024-03-06T07:21:53.560624+0000","last_unstale":"2024-03-06T07:21:54.560237+0000","last_undegraded":"2024-03-06T07:21:54.560237+0000","last_fullsized":"2024-03-06T07:21:54.560237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778508+0000","last_change":"2024-03-06T07:21:53.561059+0000","last_active":"2024-03-06T07:21:54.778508+0000","last_peered":"2024-03-06T07:21:54.778508+0000","last_clean":"2024-03-06T07:21:54.778508+0000","last_became_active":"2024-03-06T07:21:53.560799+0000","last_became_peered":"2024-03-06T07:21:53.560799+0000","last_unstale":"2024-03-06T07:21:54.778508+0000","last_undegraded":"2024-03-06T07:21:54.778508+0000","last_fullsized":"2024-03-06T07:21:54.778508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562750+0000","last_change":"2024-03-06T07:21:53.558851+0000","last_active":"2024-03-06T07:21:54.562750+0000","last_peered":"2024-03-06T07:21:54.562750+0000","last_clean":"2024-03-06T07:21:54.562750+0000","last_became_active":"2024-03-06T07:21:53.558543+0000","last_became_peered":"2024-03-06T07:21:53.558543+0000","last_unstale":"2024-03-06T07:21:54.562750+0000","last_undegraded":"2024-03-06T07:21:54.562750+0000","last_fullsized":"2024-03-06T07:21:54.562750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777302+0000","last_change":"2024-03-06T07:21:53.560684+0000","last_active":"2024-03-06T07:21:54.777302+0000","last_peered":"2024-03-06T07:21:54.777302+0000","last_clean":"2024-03-06T07:21:54.777302+0000","last_became_active":"2024-03-06T07:21:53.560428+0000","last_became_peered":"2024-03-06T07:21:53.560428+0000","last_unstale":"2024-03-06T07:21:54.777302+0000","last_undegraded":"2024-03-06T07:21:54.777302+0000","last_fullsized":"2024-03-06T07:21:54.777302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777813+0000","last_change":"2024-03-06T07:21:53.562732+0000","last_active":"2024-03-06T07:21:54.777813+0000","last_peered":"2024-03-06T07:21:54.777813+0000","last_clean":"2024-03-06T07:21:54.777813+0000","last_became_active":"2024-03-06T07:21:53.562351+0000","last_became_peered":"2024-03-06T07:21:53.562351+0000","last_unstale":"2024-03-06T07:21:54.777813+0000","last_undegraded":"2024-03-06T07:21:54.777813+0000","last_fullsized":"2024-03-06T07:21:54.777813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767070+0000","last_change":"2024-03-06T07:21:53.564304+0000","last_active":"2024-03-06T07:21:54.767070+0000","last_peered":"2024-03-06T07:21:54.767070+0000","last_clean":"2024-03-06T07:21:54.767070+0000","last_became_active":"2024-03-06T07:21:53.564008+0000","last_became_peered":"2024-03-06T07:21:53.564008+0000","last_unstale":"2024-03-06T07:21:54.767070+0000","last_undegraded":"2024-03-06T07:21:54.767070+0000","last_fullsized":"2024-03-06T07:21:54.767070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777401+0000","last_change":"2024-03-06T07:21:53.565197+0000","last_active":"2024-03-06T07:21:54.777401+0000","last_peered":"2024-03-06T07:21:54.777401+0000","last_clean":"2024-03-06T07:21:54.777401+0000","last_became_active":"2024-03-06T07:21:53.564878+0000","last_became_peered":"2024-03-06T07:21:53.564878+0000","last_unstale":"2024-03-06T07:21:54.777401+0000","last_undegraded":"2024-03-06T07:21:54.777401+0000","last_fullsized":"2024-03-06T07:21:54.777401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767433+0000","last_change":"2024-03-06T07:21:53.558718+0000","last_active":"2024-03-06T07:21:54.767433+0000","last_peered":"2024-03-06T07:21:54.767433+0000","last_clean":"2024-03-06T07:21:54.767433+0000","last_became_active":"2024-03-06T07:21:53.558454+0000","last_became_peered":"2024-03-06T07:21:53.558454+0000","last_unstale":"2024-03-06T07:21:54.767433+0000","last_undegraded":"2024-03-06T07:21:54.767433+0000","last_fullsized":"2024-03-06T07:21:54.767433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562735+0000","last_change":"2024-03-06T07:21:53.563894+0000","last_active":"2024-03-06T07:21:54.562735+0000","last_peered":"2024-03-06T07:21:54.562735+0000","last_clean":"2024-03-06T07:21:54.562735+0000","last_became_active":"2024-03-06T07:21:53.563493+0000","last_became_peered":"2024-03-06T07:21:53.563493+0000","last_unstale":"2024-03-06T07:21:54.562735+0000","last_undegraded":"2024-03-06T07:21:54.562735+0000","last_fullsized":"2024-03-06T07:21:54.562735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562575+0000","last_change":"2024-03-06T07:21:53.552128+0000","last_active":"2024-03-06T07:21:54.562575+0000","last_peered":"2024-03-06T07:21:54.562575+0000","last_clean":"2024-03-06T07:21:54.562575+0000","last_became_active":"2024-03-06T07:21:53.551826+0000","last_became_peered":"2024-03-06T07:21:53.551826+0000","last_unstale":"2024-03-06T07:21:54.562575+0000","last_undegraded":"2024-03-06T07:21:54.562575+0000","last_fullsized":"2024-03-06T07:21:54.562575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767379+0000","last_change":"2024-03-06T07:21:53.563089+0000","last_active":"2024-03-06T07:21:54.767379+0000","last_peered":"2024-03-06T07:21:54.767379+0000","last_clean":"2024-03-06T07:21:54.767379+0000","last_became_active":"2024-03-06T07:21:53.562833+0000","last_became_peered":"2024-03-06T07:21:53.562833+0000","last_unstale":"2024-03-06T07:21:54.767379+0000","last_undegraded":"2024-03-06T07:21:54.767379+0000","last_fullsized":"2024-03-06T07:21:54.767379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777387+0000","last_change":"2024-03-06T07:21:53.550849+0000","last_active":"2024-03-06T07:21:54.777387+0000","last_peered":"2024-03-06T07:21:54.777387+0000","last_clean":"2024-03-06T07:21:54.777387+0000","last_became_active":"2024-03-06T07:21:53.550562+0000","last_became_peered":"2024-03-06T07:21:53.550562+0000","last_unstale":"2024-03-06T07:21:54.777387+0000","last_undegraded":"2024-03-06T07:21:54.777387+0000","last_fullsized":"2024-03-06T07:21:54.777387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777942+0000","last_change":"2024-03-06T07:21:53.564528+0000","last_active":"2024-03-06T07:21:54.777942+0000","last_peered":"2024-03-06T07:21:54.777942+0000","last_clean":"2024-03-06T07:21:54.777942+0000","last_became_active":"2024-03-06T07:21:53.564250+0000","last_became_peered":"2024-03-06T07:21:53.564250+0000","last_unstale":"2024-03-06T07:21:54.777942+0000","last_undegraded":"2024-03-06T07:21:54.777942+0000","last_fullsized":"2024-03-06T07:21:54.777942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560202+0000","last_change":"2024-03-06T07:21:53.555673+0000","last_active":"2024-03-06T07:21:54.560202+0000","last_peered":"2024-03-06T07:21:54.560202+0000","last_clean":"2024-03-06T07:21:54.560202+0000","last_became_active":"2024-03-06T07:21:53.555337+0000","last_became_peered":"2024-03-06T07:21:53.555337+0000","last_unstale":"2024-03-06T07:21:54.560202+0000","last_undegraded":"2024-03-06T07:21:54.560202+0000","last_fullsized":"2024-03-06T07:21:54.560202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562445+0000","last_change":"2024-03-06T07:21:53.552447+0000","last_active":"2024-03-06T07:21:54.562445+0000","last_peered":"2024-03-06T07:21:54.562445+0000","last_clean":"2024-03-06T07:21:54.562445+0000","last_became_active":"2024-03-06T07:21:53.551907+0000","last_became_peered":"2024-03-06T07:21:53.551907+0000","last_unstale":"2024-03-06T07:21:54.562445+0000","last_undegraded":"2024-03-06T07:21:54.562445+0000","last_fullsized":"2024-03-06T07:21:54.562445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777543+0000","last_change":"2024-03-06T07:21:53.554149+0000","last_active":"2024-03-06T07:21:54.777543+0000","last_peered":"2024-03-06T07:21:54.777543+0000","last_clean":"2024-03-06T07:21:54.777543+0000","last_became_active":"2024-03-06T07:21:53.553811+0000","last_became_peered":"2024-03-06T07:21:53.553811+0000","last_unstale":"2024-03-06T07:21:54.777543+0000","last_undegraded":"2024-03-06T07:21:54.777543+0000","last_fullsized":"2024-03-06T07:21:54.777543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:11.236013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767399+0000","last_change":"2024-03-06T07:21:52.551262+0000","last_active":"2024-03-06T07:21:54.767399+0000","last_peered":"2024-03-06T07:21:54.767399+0000","last_clean":"2024-03-06T07:21:54.767399+0000","last_became_active":"2024-03-06T07:21:52.550349+0000","last_became_peered":"2024-03-06T07:21:52.550349+0000","last_unstale":"2024-03-06T07:21:54.767399+0000","last_undegraded":"2024-03-06T07:21:54.767399+0000","last_fullsized":"2024-03-06T07:21:54.767399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777450+0000","last_change":"2024-03-06T07:21:53.562215+0000","last_active":"2024-03-06T07:21:54.777450+0000","last_peered":"2024-03-06T07:21:54.777450+0000","last_clean":"2024-03-06T07:21:54.777450+0000","last_became_active":"2024-03-06T07:21:53.561933+0000","last_became_peered":"2024-03-06T07:21:53.561933+0000","last_unstale":"2024-03-06T07:21:54.777450+0000","last_undegraded":"2024-03-06T07:21:54.777450+0000","last_fullsized":"2024-03-06T07:21:54.777450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562398+0000","last_change":"2024-03-06T07:21:52.547490+0000","last_active":"2024-03-06T07:21:54.562398+0000","last_peered":"2024-03-06T07:21:54.562398+0000","last_clean":"2024-03-06T07:21:54.562398+0000","last_became_active":"2024-03-06T07:21:52.547168+0000","last_became_peered":"2024-03-06T07:21:52.547168+0000","last_unstale":"2024-03-06T07:21:54.562398+0000","last_undegraded":"2024-03-06T07:21:54.562398+0000","last_fullsized":"2024-03-06T07:21:54.562398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767143+0000","last_change":"2024-03-06T07:21:52.551838+0000","last_active":"2024-03-06T07:21:54.767143+0000","last_peered":"2024-03-06T07:21:54.767143+0000","last_clean":"2024-03-06T07:21:54.767143+0000","last_became_active":"2024-03-06T07:21:52.551599+0000","last_became_peered":"2024-03-06T07:21:52.551599+0000","last_unstale":"2024-03-06T07:21:54.767143+0000","last_undegraded":"2024-03-06T07:21:54.767143+0000","last_fullsized":"2024-03-06T07:21:54.767143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767031+0000","last_change":"2024-03-06T07:21:53.557369+0000","last_active":"2024-03-06T07:21:54.767031+0000","last_peered":"2024-03-06T07:21:54.767031+0000","last_clean":"2024-03-06T07:21:54.767031+0000","last_became_active":"2024-03-06T07:21:53.556320+0000","last_became_peered":"2024-03-06T07:21:53.556320+0000","last_unstale":"2024-03-06T07:21:54.767031+0000","last_undegraded":"2024-03-06T07:21:54.767031+0000","last_fullsized":"2024-03-06T07:21:54.767031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777454+0000","last_change":"2024-03-06T07:21:52.553279+0000","last_active":"2024-03-06T07:21:54.777454+0000","last_peered":"2024-03-06T07:21:54.777454+0000","last_clean":"2024-03-06T07:21:54.777454+0000","last_became_active":"2024-03-06T07:21:52.552991+0000","last_became_peered":"2024-03-06T07:21:52.552991+0000","last_unstale":"2024-03-06T07:21:54.777454+0000","last_undegraded":"2024-03-06T07:21:54.777454+0000","last_fullsized":"2024-03-06T07:21:54.777454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562572+0000","last_change":"2024-03-06T07:21:53.552639+0000","last_active":"2024-03-06T07:21:54.562572+0000","last_peered":"2024-03-06T07:21:54.562572+0000","last_clean":"2024-03-06T07:21:54.562572+0000","last_became_active":"2024-03-06T07:21:53.552377+0000","last_became_peered":"2024-03-06T07:21:53.552377+0000","last_unstale":"2024-03-06T07:21:54.562572+0000","last_undegraded":"2024-03-06T07:21:54.562572+0000","last_fullsized":"2024-03-06T07:21:54.562572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562577+0000","last_change":"2024-03-06T07:21:52.556094+0000","last_active":"2024-03-06T07:21:54.562577+0000","last_peered":"2024-03-06T07:21:54.562577+0000","last_clean":"2024-03-06T07:21:54.562577+0000","last_became_active":"2024-03-06T07:21:52.555825+0000","last_became_peered":"2024-03-06T07:21:52.555825+0000","last_unstale":"2024-03-06T07:21:54.562577+0000","last_undegraded":"2024-03-06T07:21:54.562577+0000","last_fullsized":"2024-03-06T07:21:54.562577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778464+0000","last_change":"2024-03-06T07:21:53.561469+0000","last_active":"2024-03-06T07:21:54.778464+0000","last_peered":"2024-03-06T07:21:54.778464+0000","last_clean":"2024-03-06T07:21:54.778464+0000","last_became_active":"2024-03-06T07:21:53.561204+0000","last_became_peered":"2024-03-06T07:21:53.561204+0000","last_unstale":"2024-03-06T07:21:54.778464+0000","last_undegraded":"2024-03-06T07:21:54.778464+0000","last_fullsized":"2024-03-06T07:21:54.778464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778543+0000","last_change":"2024-03-06T07:21:52.557218+0000","last_active":"2024-03-06T07:21:54.778543+0000","last_peered":"2024-03-06T07:21:54.778543+0000","last_clean":"2024-03-06T07:21:54.778543+0000","last_became_active":"2024-03-06T07:21:52.556955+0000","last_became_peered":"2024-03-06T07:21:52.556955+0000","last_unstale":"2024-03-06T07:21:54.778543+0000","last_undegraded":"2024-03-06T07:21:54.778543+0000","last_fullsized":"2024-03-06T07:21:54.778543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777433+0000","last_change":"2024-03-06T07:21:53.559108+0000","last_active":"2024-03-06T07:21:54.777433+0000","last_peered":"2024-03-06T07:21:54.777433+0000","last_clean":"2024-03-06T07:21:54.777433+0000","last_became_active":"2024-03-06T07:21:53.558256+0000","last_became_peered":"2024-03-06T07:21:53.558256+0000","last_unstale":"2024-03-06T07:21:54.777433+0000","last_undegraded":"2024-03-06T07:21:54.777433+0000","last_fullsized":"2024-03-06T07:21:54.777433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562655+0000","last_change":"2024-03-06T07:21:52.557898+0000","last_active":"2024-03-06T07:21:54.562655+0000","last_peered":"2024-03-06T07:21:54.562655+0000","last_clean":"2024-03-06T07:21:54.562655+0000","last_became_active":"2024-03-06T07:21:52.557619+0000","last_became_peered":"2024-03-06T07:21:52.557619+0000","last_unstale":"2024-03-06T07:21:54.562655+0000","last_undegraded":"2024-03-06T07:21:54.562655+0000","last_fullsized":"2024-03-06T07:21:54.562655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777844+0000","last_change":"2024-03-06T07:21:53.557112+0000","last_active":"2024-03-06T07:21:54.777844+0000","last_peered":"2024-03-06T07:21:54.777844+0000","last_clean":"2024-03-06T07:21:54.777844+0000","last_became_active":"2024-03-06T07:21:53.556848+0000","last_became_peered":"2024-03-06T07:21:53.556848+0000","last_unstale":"2024-03-06T07:21:54.777844+0000","last_undegraded":"2024-03-06T07:21:54.777844+0000","last_fullsized":"2024-03-06T07:21:54.777844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562841+0000","last_change":"2024-03-06T07:21:52.548388+0000","last_active":"2024-03-06T07:21:54.562841+0000","last_peered":"2024-03-06T07:21:54.562841+0000","last_clean":"2024-03-06T07:21:54.562841+0000","last_became_active":"2024-03-06T07:21:52.548095+0000","last_became_peered":"2024-03-06T07:21:52.548095+0000","last_unstale":"2024-03-06T07:21:54.562841+0000","last_undegraded":"2024-03-06T07:21:54.562841+0000","last_fullsized":"2024-03-06T07:21:54.562841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778266+0000","last_change":"2024-03-06T07:21:53.557821+0000","last_active":"2024-03-06T07:21:54.778266+0000","last_peered":"2024-03-06T07:21:54.778266+0000","last_clean":"2024-03-06T07:21:54.778266+0000","last_became_active":"2024-03-06T07:21:53.557536+0000","last_became_peered":"2024-03-06T07:21:53.557536+0000","last_unstale":"2024-03-06T07:21:54.778266+0000","last_undegraded":"2024-03-06T07:21:54.778266+0000","last_fullsized":"2024-03-06T07:21:54.778266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777481+0000","last_change":"2024-03-06T07:21:52.555948+0000","last_active":"2024-03-06T07:21:54.777481+0000","last_peered":"2024-03-06T07:21:54.777481+0000","last_clean":"2024-03-06T07:21:54.777481+0000","last_became_active":"2024-03-06T07:21:52.555657+0000","last_became_peered":"2024-03-06T07:21:52.555657+0000","last_unstale":"2024-03-06T07:21:54.777481+0000","last_undegraded":"2024-03-06T07:21:54.777481+0000","last_fullsized":"2024-03-06T07:21:54.777481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777191+0000","last_change":"2024-03-06T07:21:53.556393+0000","last_active":"2024-03-06T07:21:54.777191+0000","last_peered":"2024-03-06T07:21:54.777191+0000","last_clean":"2024-03-06T07:21:54.777191+0000","last_became_active":"2024-03-06T07:21:53.556137+0000","last_became_peered":"2024-03-06T07:21:53.556137+0000","last_unstale":"2024-03-06T07:21:54.777191+0000","last_undegraded":"2024-03-06T07:21:54.777191+0000","last_fullsized":"2024-03-06T07:21:54.777191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:33.561906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778025+0000","last_change":"2024-03-06T07:21:52.556426+0000","last_active":"2024-03-06T07:21:54.778025+0000","last_peered":"2024-03-06T07:21:54.778025+0000","last_clean":"2024-03-06T07:21:54.778025+0000","last_became_active":"2024-03-06T07:21:52.555844+0000","last_became_peered":"2024-03-06T07:21:52.555844+0000","last_unstale":"2024-03-06T07:21:54.778025+0000","last_undegraded":"2024-03-06T07:21:54.778025+0000","last_fullsized":"2024-03-06T07:21:54.778025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778299+0000","last_change":"2024-03-06T07:21:53.567432+0000","last_active":"2024-03-06T07:21:54.778299+0000","last_peered":"2024-03-06T07:21:54.778299+0000","last_clean":"2024-03-06T07:21:54.778299+0000","last_became_active":"2024-03-06T07:21:53.566788+0000","last_became_peered":"2024-03-06T07:21:53.566788+0000","last_unstale":"2024-03-06T07:21:54.778299+0000","last_undegraded":"2024-03-06T07:21:54.778299+0000","last_fullsized":"2024-03-06T07:21:54.778299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562338+0000","last_change":"2024-03-06T07:21:52.547444+0000","last_active":"2024-03-06T07:21:54.562338+0000","last_peered":"2024-03-06T07:21:54.562338+0000","last_clean":"2024-03-06T07:21:54.562338+0000","last_became_active":"2024-03-06T07:21:52.547163+0000","last_became_peered":"2024-03-06T07:21:52.547163+0000","last_unstale":"2024-03-06T07:21:54.562338+0000","last_undegraded":"2024-03-06T07:21:54.562338+0000","last_fullsized":"2024-03-06T07:21:54.562338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562392+0000","last_change":"2024-03-06T07:21:53.552210+0000","last_active":"2024-03-06T07:21:54.562392+0000","last_peered":"2024-03-06T07:21:54.562392+0000","last_clean":"2024-03-06T07:21:54.562392+0000","last_became_active":"2024-03-06T07:21:53.551529+0000","last_became_peered":"2024-03-06T07:21:53.551529+0000","last_unstale":"2024-03-06T07:21:54.562392+0000","last_undegraded":"2024-03-06T07:21:54.562392+0000","last_fullsized":"2024-03-06T07:21:54.562392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560082+0000","last_change":"2024-03-06T07:21:52.552286+0000","last_active":"2024-03-06T07:21:54.560082+0000","last_peered":"2024-03-06T07:21:54.560082+0000","last_clean":"2024-03-06T07:21:54.560082+0000","last_became_active":"2024-03-06T07:21:52.552010+0000","last_became_peered":"2024-03-06T07:21:52.552010+0000","last_unstale":"2024-03-06T07:21:54.560082+0000","last_undegraded":"2024-03-06T07:21:54.560082+0000","last_fullsized":"2024-03-06T07:21:54.560082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777580+0000","last_change":"2024-03-06T07:21:53.571841+0000","last_active":"2024-03-06T07:21:54.777580+0000","last_peered":"2024-03-06T07:21:54.777580+0000","last_clean":"2024-03-06T07:21:54.777580+0000","last_became_active":"2024-03-06T07:21:53.571350+0000","last_became_peered":"2024-03-06T07:21:53.571350+0000","last_unstale":"2024-03-06T07:21:54.777580+0000","last_undegraded":"2024-03-06T07:21:54.777580+0000","last_fullsized":"2024-03-06T07:21:54.777580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777653+0000","last_change":"2024-03-06T07:21:52.553990+0000","last_active":"2024-03-06T07:21:54.777653+0000","last_peered":"2024-03-06T07:21:54.777653+0000","last_clean":"2024-03-06T07:21:54.777653+0000","last_became_active":"2024-03-06T07:21:52.553693+0000","last_became_peered":"2024-03-06T07:21:52.553693+0000","last_unstale":"2024-03-06T07:21:54.777653+0000","last_undegraded":"2024-03-06T07:21:54.777653+0000","last_fullsized":"2024-03-06T07:21:54.777653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777984+0000","last_change":"2024-03-06T07:21:53.567545+0000","last_active":"2024-03-06T07:21:54.777984+0000","last_peered":"2024-03-06T07:21:54.777984+0000","last_clean":"2024-03-06T07:21:54.777984+0000","last_became_active":"2024-03-06T07:21:53.566157+0000","last_became_peered":"2024-03-06T07:21:53.566157+0000","last_unstale":"2024-03-06T07:21:54.777984+0000","last_undegraded":"2024-03-06T07:21:54.777984+0000","last_fullsized":"2024-03-06T07:21:54.777984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560201+0000","last_change":"2024-03-06T07:21:52.547686+0000","last_active":"2024-03-06T07:21:54.560201+0000","last_peered":"2024-03-06T07:21:54.560201+0000","last_clean":"2024-03-06T07:21:54.560201+0000","last_became_active":"2024-03-06T07:21:52.547436+0000","last_became_peered":"2024-03-06T07:21:52.547436+0000","last_unstale":"2024-03-06T07:21:54.560201+0000","last_undegraded":"2024-03-06T07:21:54.560201+0000","last_fullsized":"2024-03-06T07:21:54.560201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778797+0000","last_change":"2024-03-06T07:21:53.567657+0000","last_active":"2024-03-06T07:21:54.778797+0000","last_peered":"2024-03-06T07:21:54.778797+0000","last_clean":"2024-03-06T07:21:54.778797+0000","last_became_active":"2024-03-06T07:21:53.567134+0000","last_became_peered":"2024-03-06T07:21:53.567134+0000","last_unstale":"2024-03-06T07:21:54.778797+0000","last_undegraded":"2024-03-06T07:21:54.778797+0000","last_fullsized":"2024-03-06T07:21:54.778797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:57.048252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562696+0000","last_change":"2024-03-06T07:21:52.556768+0000","last_active":"2024-03-06T07:21:54.562696+0000","last_peered":"2024-03-06T07:21:54.562696+0000","last_clean":"2024-03-06T07:21:54.562696+0000","last_became_active":"2024-03-06T07:21:52.556507+0000","last_became_peered":"2024-03-06T07:21:52.556507+0000","last_unstale":"2024-03-06T07:21:54.562696+0000","last_undegraded":"2024-03-06T07:21:54.562696+0000","last_fullsized":"2024-03-06T07:21:54.562696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778541+0000","last_change":"2024-03-06T07:21:53.563768+0000","last_active":"2024-03-06T07:21:54.778541+0000","last_peered":"2024-03-06T07:21:54.778541+0000","last_clean":"2024-03-06T07:21:54.778541+0000","last_became_active":"2024-03-06T07:21:53.563490+0000","last_became_peered":"2024-03-06T07:21:53.563490+0000","last_unstale":"2024-03-06T07:21:54.778541+0000","last_undegraded":"2024-03-06T07:21:54.778541+0000","last_fullsized":"2024-03-06T07:21:54.778541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560265+0000","last_change":"2024-03-06T07:21:52.557535+0000","last_active":"2024-03-06T07:21:54.560265+0000","last_peered":"2024-03-06T07:21:54.560265+0000","last_clean":"2024-03-06T07:21:54.560265+0000","last_became_active":"2024-03-06T07:21:52.557275+0000","last_became_peered":"2024-03-06T07:21:52.557275+0000","last_unstale":"2024-03-06T07:21:54.560265+0000","last_undegraded":"2024-03-06T07:21:54.560265+0000","last_fullsized":"2024-03-06T07:21:54.560265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778386+0000","last_change":"2024-03-06T07:21:53.565629+0000","last_active":"2024-03-06T07:21:54.778386+0000","last_peered":"2024-03-06T07:21:54.778386+0000","last_clean":"2024-03-06T07:21:54.778386+0000","last_became_active":"2024-03-06T07:21:53.563789+0000","last_became_peered":"2024-03-06T07:21:53.563789+0000","last_unstale":"2024-03-06T07:21:54.778386+0000","last_undegraded":"2024-03-06T07:21:54.778386+0000","last_fullsized":"2024-03-06T07:21:54.778386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778063+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:54.778063+0000","last_peered":"2024-03-06T07:21:54.778063+0000","last_clean":"2024-03-06T07:21:54.778063+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:54.778063+0000","last_undegraded":"2024-03-06T07:21:54.778063+0000","last_fullsized":"2024-03-06T07:21:54.778063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777283+0000","last_change":"2024-03-06T07:21:52.557611+0000","last_active":"2024-03-06T07:21:54.777283+0000","last_peered":"2024-03-06T07:21:54.777283+0000","last_clean":"2024-03-06T07:21:54.777283+0000","last_became_active":"2024-03-06T07:21:52.556594+0000","last_became_peered":"2024-03-06T07:21:52.556594+0000","last_unstale":"2024-03-06T07:21:54.777283+0000","last_undegraded":"2024-03-06T07:21:54.777283+0000","last_fullsized":"2024-03-06T07:21:54.777283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767209+0000","last_change":"2024-03-06T07:21:53.565607+0000","last_active":"2024-03-06T07:21:54.767209+0000","last_peered":"2024-03-06T07:21:54.767209+0000","last_clean":"2024-03-06T07:21:54.767209+0000","last_became_active":"2024-03-06T07:21:53.565356+0000","last_became_peered":"2024-03-06T07:21:53.565356+0000","last_unstale":"2024-03-06T07:21:54.767209+0000","last_undegraded":"2024-03-06T07:21:54.767209+0000","last_fullsized":"2024-03-06T07:21:54.767209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778021+0000","last_change":"2024-03-06T07:21:52.555927+0000","last_active":"2024-03-06T07:21:54.778021+0000","last_peered":"2024-03-06T07:21:54.778021+0000","last_clean":"2024-03-06T07:21:54.778021+0000","last_became_active":"2024-03-06T07:21:52.555485+0000","last_became_peered":"2024-03-06T07:21:52.555485+0000","last_unstale":"2024-03-06T07:21:54.778021+0000","last_undegraded":"2024-03-06T07:21:54.778021+0000","last_fullsized":"2024-03-06T07:21:54.778021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777097+0000","last_change":"2024-03-06T07:21:53.559225+0000","last_active":"2024-03-06T07:21:54.777097+0000","last_peered":"2024-03-06T07:21:54.777097+0000","last_clean":"2024-03-06T07:21:54.777097+0000","last_became_active":"2024-03-06T07:21:53.558925+0000","last_became_peered":"2024-03-06T07:21:53.558925+0000","last_unstale":"2024-03-06T07:21:54.777097+0000","last_undegraded":"2024-03-06T07:21:54.777097+0000","last_fullsized":"2024-03-06T07:21:54.777097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:54.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560174+0000","last_change":"2024-03-06T07:21:52.546620+0000","last_active":"2024-03-06T07:21:54.560174+0000","last_peered":"2024-03-06T07:21:54.560174+0000","last_clean":"2024-03-06T07:21:54.560174+0000","last_became_active":"2024-03-06T07:21:52.546297+0000","last_became_peered":"2024-03-06T07:21:52.546297+0000","last_unstale":"2024-03-06T07:21:54.560174+0000","last_undegraded":"2024-03-06T07:21:54.560174+0000","last_fullsized":"2024-03-06T07:21:54.560174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562488+0000","last_change":"2024-03-06T07:21:53.553218+0000","last_active":"2024-03-06T07:21:54.562488+0000","last_peered":"2024-03-06T07:21:54.562488+0000","last_clean":"2024-03-06T07:21:54.562488+0000","last_became_active":"2024-03-06T07:21:53.552911+0000","last_became_peered":"2024-03-06T07:21:53.552911+0000","last_unstale":"2024-03-06T07:21:54.562488+0000","last_undegraded":"2024-03-06T07:21:54.562488+0000","last_fullsized":"2024-03-06T07:21:54.562488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778113+0000","last_change":"2024-03-06T07:21:52.556096+0000","last_active":"2024-03-06T07:21:54.778113+0000","last_peered":"2024-03-06T07:21:54.778113+0000","last_clean":"2024-03-06T07:21:54.778113+0000","last_became_active":"2024-03-06T07:21:52.555510+0000","last_became_peered":"2024-03-06T07:21:52.555510+0000","last_unstale":"2024-03-06T07:21:54.778113+0000","last_undegraded":"2024-03-06T07:21:54.778113+0000","last_fullsized":"2024-03-06T07:21:54.778113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778491+0000","last_change":"2024-03-06T07:21:53.561767+0000","last_active":"2024-03-06T07:21:54.778491+0000","last_peered":"2024-03-06T07:21:54.778491+0000","last_clean":"2024-03-06T07:21:54.778491+0000","last_became_active":"2024-03-06T07:21:53.561435+0000","last_became_peered":"2024-03-06T07:21:53.561435+0000","last_unstale":"2024-03-06T07:21:54.778491+0000","last_undegraded":"2024-03-06T07:21:54.778491+0000","last_fullsized":"2024-03-06T07:21:54.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:24.631055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562673+0000","last_change":"2024-03-06T07:21:52.553733+0000","last_active":"2024-03-06T07:21:54.562673+0000","last_peered":"2024-03-06T07:21:54.562673+0000","last_clean":"2024-03-06T07:21:54.562673+0000","last_became_active":"2024-03-06T07:21:52.553436+0000","last_became_peered":"2024-03-06T07:21:52.553436+0000","last_unstale":"2024-03-06T07:21:54.562673+0000","last_undegraded":"2024-03-06T07:21:54.562673+0000","last_fullsized":"2024-03-06T07:21:54.562673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779356+0000","last_change":"2024-03-06T07:21:53.556153+0000","last_active":"2024-03-06T07:21:54.779356+0000","last_peered":"2024-03-06T07:21:54.779356+0000","last_clean":"2024-03-06T07:21:54.779356+0000","last_became_active":"2024-03-06T07:21:53.555896+0000","last_became_peered":"2024-03-06T07:21:53.555896+0000","last_unstale":"2024-03-06T07:21:54.779356+0000","last_undegraded":"2024-03-06T07:21:54.779356+0000","last_fullsized":"2024-03-06T07:21:54.779356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778511+0000","last_change":"2024-03-06T07:21:52.557084+0000","last_active":"2024-03-06T07:21:54.778511+0000","last_peered":"2024-03-06T07:21:54.778511+0000","last_clean":"2024-03-06T07:21:54.778511+0000","last_became_active":"2024-03-06T07:21:52.556688+0000","last_became_peered":"2024-03-06T07:21:52.556688+0000","last_unstale":"2024-03-06T07:21:54.778511+0000","last_undegraded":"2024-03-06T07:21:54.778511+0000","last_fullsized":"2024-03-06T07:21:54.778511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:23.626336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562632+0000","last_change":"2024-03-06T07:21:53.552698+0000","last_active":"2024-03-06T07:21:54.562632+0000","last_peered":"2024-03-06T07:21:54.562632+0000","last_clean":"2024-03-06T07:21:54.562632+0000","last_became_active":"2024-03-06T07:21:53.552434+0000","last_became_peered":"2024-03-06T07:21:53.552434+0000","last_unstale":"2024-03-06T07:21:54.562632+0000","last_undegraded":"2024-03-06T07:21:54.562632+0000","last_fullsized":"2024-03-06T07:21:54.562632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562589+0000","last_change":"2024-03-06T07:21:52.553671+0000","last_active":"2024-03-06T07:21:54.562589+0000","last_peered":"2024-03-06T07:21:54.562589+0000","last_clean":"2024-03-06T07:21:54.562589+0000","last_became_active":"2024-03-06T07:21:52.553394+0000","last_became_peered":"2024-03-06T07:21:52.553394+0000","last_unstale":"2024-03-06T07:21:54.562589+0000","last_undegraded":"2024-03-06T07:21:54.562589+0000","last_fullsized":"2024-03-06T07:21:54.562589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767449+0000","last_change":"2024-03-06T07:21:53.555702+0000","last_active":"2024-03-06T07:21:54.767449+0000","last_peered":"2024-03-06T07:21:54.767449+0000","last_clean":"2024-03-06T07:21:54.767449+0000","last_became_active":"2024-03-06T07:21:53.555436+0000","last_became_peered":"2024-03-06T07:21:53.555436+0000","last_unstale":"2024-03-06T07:21:54.767449+0000","last_undegraded":"2024-03-06T07:21:54.767449+0000","last_fullsized":"2024-03-06T07:21:54.767449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562639+0000","last_change":"2024-03-06T07:21:52.550448+0000","last_active":"2024-03-06T07:21:54.562639+0000","last_peered":"2024-03-06T07:21:54.562639+0000","last_clean":"2024-03-06T07:21:54.562639+0000","last_became_active":"2024-03-06T07:21:52.550179+0000","last_became_peered":"2024-03-06T07:21:52.550179+0000","last_unstale":"2024-03-06T07:21:54.562639+0000","last_undegraded":"2024-03-06T07:21:54.562639+0000","last_fullsized":"2024-03-06T07:21:54.562639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777421+0000","last_change":"2024-03-06T07:21:53.553031+0000","last_active":"2024-03-06T07:21:54.777421+0000","last_peered":"2024-03-06T07:21:54.777421+0000","last_clean":"2024-03-06T07:21:54.777421+0000","last_became_active":"2024-03-06T07:21:53.552759+0000","last_became_peered":"2024-03-06T07:21:53.552759+0000","last_unstale":"2024-03-06T07:21:54.777421+0000","last_undegraded":"2024-03-06T07:21:54.777421+0000","last_fullsized":"2024-03-06T07:21:54.777421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777184+0000","last_change":"2024-03-06T07:21:52.555781+0000","last_active":"2024-03-06T07:21:54.777184+0000","last_peered":"2024-03-06T07:21:54.777184+0000","last_clean":"2024-03-06T07:21:54.777184+0000","last_became_active":"2024-03-06T07:21:52.555469+0000","last_became_peered":"2024-03-06T07:21:52.555469+0000","last_unstale":"2024-03-06T07:21:54.777184+0000","last_undegraded":"2024-03-06T07:21:54.777184+0000","last_fullsized":"2024-03-06T07:21:54.777184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777797+0000","last_change":"2024-03-06T07:21:53.558488+0000","last_active":"2024-03-06T07:21:54.777797+0000","last_peered":"2024-03-06T07:21:54.777797+0000","last_clean":"2024-03-06T07:21:54.777797+0000","last_became_active":"2024-03-06T07:21:53.558199+0000","last_became_peered":"2024-03-06T07:21:53.558199+0000","last_unstale":"2024-03-06T07:21:54.777797+0000","last_undegraded":"2024-03-06T07:21:54.777797+0000","last_fullsized":"2024-03-06T07:21:54.777797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562638+0000","last_change":"2024-03-06T07:21:52.556952+0000","last_active":"2024-03-06T07:21:54.562638+0000","last_peered":"2024-03-06T07:21:54.562638+0000","last_clean":"2024-03-06T07:21:54.562638+0000","last_became_active":"2024-03-06T07:21:52.556629+0000","last_became_peered":"2024-03-06T07:21:52.556629+0000","last_unstale":"2024-03-06T07:21:54.562638+0000","last_undegraded":"2024-03-06T07:21:54.562638+0000","last_fullsized":"2024-03-06T07:21:54.562638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767159+0000","last_change":"2024-03-06T07:21:53.554655+0000","last_active":"2024-03-06T07:21:54.767159+0000","last_peered":"2024-03-06T07:21:54.767159+0000","last_clean":"2024-03-06T07:21:54.767159+0000","last_became_active":"2024-03-06T07:21:53.554321+0000","last_became_peered":"2024-03-06T07:21:53.554321+0000","last_unstale":"2024-03-06T07:21:54.767159+0000","last_undegraded":"2024-03-06T07:21:54.767159+0000","last_fullsized":"2024-03-06T07:21:54.767159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562564+0000","last_change":"2024-03-06T07:21:52.547548+0000","last_active":"2024-03-06T07:21:54.562564+0000","last_peered":"2024-03-06T07:21:54.562564+0000","last_clean":"2024-03-06T07:21:54.562564+0000","last_became_active":"2024-03-06T07:21:52.547285+0000","last_became_peered":"2024-03-06T07:21:52.547285+0000","last_unstale":"2024-03-06T07:21:54.562564+0000","last_undegraded":"2024-03-06T07:21:54.562564+0000","last_fullsized":"2024-03-06T07:21:54.562564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767314+0000","last_change":"2024-03-06T07:21:53.557506+0000","last_active":"2024-03-06T07:21:54.767314+0000","last_peered":"2024-03-06T07:21:54.767314+0000","last_clean":"2024-03-06T07:21:54.767314+0000","last_became_active":"2024-03-06T07:21:53.556568+0000","last_became_peered":"2024-03-06T07:21:53.556568+0000","last_unstale":"2024-03-06T07:21:54.767314+0000","last_undegraded":"2024-03-06T07:21:54.767314+0000","last_fullsized":"2024-03-06T07:21:54.767314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778329+0000","last_change":"2024-03-06T07:21:52.558820+0000","last_active":"2024-03-06T07:21:54.778329+0000","last_peered":"2024-03-06T07:21:54.778329+0000","last_clean":"2024-03-06T07:21:54.778329+0000","last_became_active":"2024-03-06T07:21:52.558486+0000","last_became_peered":"2024-03-06T07:21:52.558486+0000","last_unstale":"2024-03-06T07:21:54.778329+0000","last_undegraded":"2024-03-06T07:21:54.778329+0000","last_fullsized":"2024-03-06T07:21:54.778329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562402+0000","last_change":"2024-03-06T07:21:53.558230+0000","last_active":"2024-03-06T07:21:54.562402+0000","last_peered":"2024-03-06T07:21:54.562402+0000","last_clean":"2024-03-06T07:21:54.562402+0000","last_became_active":"2024-03-06T07:21:53.557916+0000","last_became_peered":"2024-03-06T07:21:53.557916+0000","last_unstale":"2024-03-06T07:21:54.562402+0000","last_undegraded":"2024-03-06T07:21:54.562402+0000","last_fullsized":"2024-03-06T07:21:54.562402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779232+0000","last_change":"2024-03-06T07:21:53.567503+0000","last_active":"2024-03-06T07:21:54.779232+0000","last_peered":"2024-03-06T07:21:54.779232+0000","last_clean":"2024-03-06T07:21:54.779232+0000","last_became_active":"2024-03-06T07:21:53.567223+0000","last_became_peered":"2024-03-06T07:21:53.567223+0000","last_unstale":"2024-03-06T07:21:54.779232+0000","last_undegraded":"2024-03-06T07:21:54.779232+0000","last_fullsized":"2024-03-06T07:21:54.779232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777697+0000","last_change":"2024-03-06T07:21:52.559682+0000","last_active":"2024-03-06T07:21:54.777697+0000","last_peered":"2024-03-06T07:21:54.777697+0000","last_clean":"2024-03-06T07:21:54.777697+0000","last_became_active":"2024-03-06T07:21:52.559430+0000","last_became_peered":"2024-03-06T07:21:52.559430+0000","last_unstale":"2024-03-06T07:21:54.777697+0000","last_undegraded":"2024-03-06T07:21:54.777697+0000","last_fullsized":"2024-03-06T07:21:54.777697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778415+0000","last_change":"2024-03-06T07:21:53.568862+0000","last_active":"2024-03-06T07:21:54.778415+0000","last_peered":"2024-03-06T07:21:54.778415+0000","last_clean":"2024-03-06T07:21:54.778415+0000","last_became_active":"2024-03-06T07:21:53.568127+0000","last_became_peered":"2024-03-06T07:21:53.568127+0000","last_unstale":"2024-03-06T07:21:54.778415+0000","last_undegraded":"2024-03-06T07:21:54.778415+0000","last_fullsized":"2024-03-06T07:21:54.778415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778429+0000","last_change":"2024-03-06T07:21:52.558983+0000","last_active":"2024-03-06T07:21:54.778429+0000","last_peered":"2024-03-06T07:21:54.778429+0000","last_clean":"2024-03-06T07:21:54.778429+0000","last_became_active":"2024-03-06T07:21:52.558616+0000","last_became_peered":"2024-03-06T07:21:52.558616+0000","last_unstale":"2024-03-06T07:21:54.778429+0000","last_undegraded":"2024-03-06T07:21:54.778429+0000","last_fullsized":"2024-03-06T07:21:54.778429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777374+0000","last_change":"2024-03-06T07:21:53.558956+0000","last_active":"2024-03-06T07:21:54.777374+0000","last_peered":"2024-03-06T07:21:54.777374+0000","last_clean":"2024-03-06T07:21:54.777374+0000","last_became_active":"2024-03-06T07:21:53.558612+0000","last_became_peered":"2024-03-06T07:21:53.558612+0000","last_unstale":"2024-03-06T07:21:54.777374+0000","last_undegraded":"2024-03-06T07:21:54.777374+0000","last_fullsized":"2024-03-06T07:21:54.777374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767442+0000","last_change":"2024-03-06T07:21:52.553080+0000","last_active":"2024-03-06T07:21:54.767442+0000","last_peered":"2024-03-06T07:21:54.767442+0000","last_clean":"2024-03-06T07:21:54.767442+0000","last_became_active":"2024-03-06T07:21:52.552785+0000","last_became_peered":"2024-03-06T07:21:52.552785+0000","last_unstale":"2024-03-06T07:21:54.767442+0000","last_undegraded":"2024-03-06T07:21:54.767442+0000","last_fullsized":"2024-03-06T07:21:54.767442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767397+0000","last_change":"2024-03-06T07:21:53.563091+0000","last_active":"2024-03-06T07:21:54.767397+0000","last_peered":"2024-03-06T07:21:54.767397+0000","last_clean":"2024-03-06T07:21:54.767397+0000","last_became_active":"2024-03-06T07:21:53.562797+0000","last_became_peered":"2024-03-06T07:21:53.562797+0000","last_unstale":"2024-03-06T07:21:54.767397+0000","last_undegraded":"2024-03-06T07:21:54.767397+0000","last_fullsized":"2024-03-06T07:21:54.767397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777504+0000","last_change":"2024-03-06T07:21:52.557097+0000","last_active":"2024-03-06T07:21:54.777504+0000","last_peered":"2024-03-06T07:21:54.777504+0000","last_clean":"2024-03-06T07:21:54.777504+0000","last_became_active":"2024-03-06T07:21:52.556808+0000","last_became_peered":"2024-03-06T07:21:52.556808+0000","last_unstale":"2024-03-06T07:21:54.777504+0000","last_undegraded":"2024-03-06T07:21:54.777504+0000","last_fullsized":"2024-03-06T07:21:54.777504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:57.565 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:21:57.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.698+0000 7f4fc7ddd640 1 Processor -- start 2024-03-06T07:21:57.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.698+0000 7f4fc7ddd640 1 -- start start 2024-03-06T07:21:57.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc7ddd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc006ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc7ddd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc006d370 con 0x7f4fc0070fe0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc7ddd640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc006d650 con 0x7f4fc015b2d0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc7ddd640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc01448a0 con 0x7f4fc01444c0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc637a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4fc0070fe0 0x7f4fc00cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59608/0 (socket says 172.21.15.122:59608) 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc637a640 1 -- 172.21.15.122:0/4220202751 learned_addr learned my addr 172.21.15.122:0/4220202751 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc5b79640 1 --2- 172.21.15.122:0/4220202751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc006ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464880893 0 0) 0x7f4fc006d370 con 0x7f4fc0070fe0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fac003520 con 0x7f4fc0070fe0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791375229 0 0) 0x7f4fc006d650 con 0x7f4fc015b2d0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc006d370 con 0x7f4fc015b2d0 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc5b79640 1 -- 172.21.15.122:0/4220202751 >> v1:172.21.15.169:6789/0 conn(0x7f4fc015b2d0 legacy=0x7f4fc00708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc5b79640 1 -- 172.21.15.122:0/4220202751 >> v1:172.21.15.122:6789/0 conn(0x7f4fc0070fe0 legacy=0x7f4fc00cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.699+0000 7f4fc5b79640 1 -- 172.21.15.122:0/4220202751 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc006d930 con 0x7f4fc01444c0 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.700+0000 7f4fc5b79640 1 --2- 172.21.15.122:0/4220202751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc006ca00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0097d0 tx=0x7f4fbc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a526237a44182 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.700+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc035020 con 0x7f4fc01444c0 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.700+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc031890 con 0x7f4fc01444c0 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.700+0000 7f4fc4b77640 1 -- 172.21.15.122:0/4220202751 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc031c40 con 0x7f4fc01444c0 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/4220202751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 msgr2=0x7f4fc006ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/4220202751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc006ca00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0097d0 tx=0x7f4fbc02b930 comp rx=0 tx=0).stop 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/4220202751 shutdown_connections 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/4220202751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc006ca00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/4220202751 >> 172.21.15.122:0/4220202751 conn(0x7f4fc00b1c80 msgr2=0x7f4fc00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/4220202751 shutdown_connections 2024-03-06T07:21:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.701+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/4220202751 wait complete. 2024-03-06T07:21:57.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.702+0000 7f4fc7ddd640 1 Processor -- start 2024-03-06T07:21:57.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.702+0000 7f4fc7ddd640 1 -- start start 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 0x7f4fc013b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc0149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fc015b2d0 0x7f4fc014bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006cf40 con 0x7f4fc0070fe0 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006def0 con 0x7f4fc015b2d0 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006d0c0 con 0x7f4fc01444c0 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 0x7f4fc013b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 0x7f4fc013b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54478/0 (socket says 172.21.15.122:54478) 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 -- 172.21.15.122:0/232966176 learned_addr learned my addr 172.21.15.122:0/232966176 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc637a640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fc015b2d0 0x7f4fc014bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5378640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc0149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 -- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 msgr2=0x7f4fc0149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc0149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 -- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fc015b2d0 msgr2=0x7f4fc014bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fc015b2d0 0x7f4fc014bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 -- 172.21.15.122:0/232966176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc014ca10 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc5b79640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 0x7f4fc013b5c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc000c00 tx=0x7f4fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc035040 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc030d20 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc014ccf0 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.703+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc006e10 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.704+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc013dc80 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.705+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc014cec0 con 0x7f4fc0070fe0 2024-03-06T07:21:57.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.705+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fbc038020 con 0x7f4fc0070fe0 2024-03-06T07:21:57.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.705+0000 7f4fb67fc640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fa406b4d0 0x7f4fa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.705+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fbc0ac780 con 0x7f4fc0070fe0 2024-03-06T07:21:57.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.705+0000 7f4fc5378640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fa406b4d0 0x7f4fa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.706+0000 7f4fc5378640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fa406b4d0 0x7f4fa406d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00afd60 tx=0x7f4fb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.710+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fc014cec0 con 0x7f4fc0070fe0 2024-03-06T07:21:57.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.946+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4fc004f330 con 0x7f4fc0070fe0 2024-03-06T07:21:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.946+0000 7f4fb67fc640 1 -- 172.21.15.122:0/232966176 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1011 (secure 0 0 0) 0x7f4fbc02d040 con 0x7f4fc0070fe0 2024-03-06T07:21:57.948 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:57.948 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":28,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":20,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":224202752,"bytes_avail":772869910528,"bytes_total":773094113280,"write_bytes_sec":2472,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":7,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"d","status":"up:active","gid":4287}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-03-06T07:21:29.173777+0000","services":{}},"progress_events":{}} 2024-03-06T07:21:57.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fa406b4d0 msgr2=0x7f4fa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fa406b4d0 0x7f4fa406d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00afd60 tx=0x7f4fb0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 msgr2=0x7f4fc013b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 0x7f4fc013b5c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc000c00 tx=0x7f4fbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 shutdown_connections 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fa406b4d0 0x7f4fa406d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fc015b2d0 0x7f4fc014bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fc01444c0 0x7f4fc0149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 --2- 172.21.15.122:0/232966176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fc0070fe0 0x7f4fc013b5c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 >> 172.21.15.122:0/232966176 conn(0x7f4fc00b1c80 msgr2=0x7f4fc00ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 shutdown_connections 2024-03-06T07:21:57.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:57.948+0000 7f4fc7ddd640 1 -- 172.21.15.122:0/232966176 wait complete. 2024-03-06T07:21:57.961 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:21:57.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:57.961 INFO:tasks.ceph.mds.d.smithi169.stderr:2024-03-06T07:21:57.959+0000 7f571cae9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:58.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 Processor -- start 2024-03-06T07:21:58.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 -- start start 2024-03-06T07:21:58.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f449c15fa10 0x7f449c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0cebe0 con 0x7f449c15bdd0 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c0ceec0 con 0x7f449c15fa10 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.059+0000 7f44a258a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c0cf1a0 con 0x7f449c0d16e0 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f449c15fa10 0x7f449c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f449c15fa10 0x7f449c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54480/0 (socket says 172.21.15.122:54480) 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 -- 172.21.15.122:0/2491695016 learned_addr learned my addr 172.21.15.122:0/2491695016 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f44a0b27640 1 --2- 172.21.15.122:0/2491695016 >> v2:172.21.15.169:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 -- 172.21.15.122:0/2491695016 >> v2:172.21.15.169:3300/0 conn(0x7f449c0d16e0 msgr2=0x7f449c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 --2- 172.21.15.122:0/2491695016 >> v2:172.21.15.169:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 -- 172.21.15.122:0/2491695016 >> v1:172.21.15.169:6789/0 conn(0x7f449c15bdd0 legacy=0x7f449c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 -- 172.21.15.122:0/2491695016 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0cc8e0 con 0x7f449c15fa10 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.060+0000 7f449b7fe640 1 --2- 172.21.15.122:0/2491695016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c0ce270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f449c0c5520 tx=0x7f448c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e0ff4049abdc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f449affd640 1 -- 172.21.15.122:0/2491695016 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c035020 con 0x7f449c15fa10 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f449affd640 1 -- 172.21.15.122:0/2491695016 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c031870 con 0x7f449c15fa10 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f449affd640 1 -- 172.21.15.122:0/2491695016 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c031c40 con 0x7f449c15fa10 2024-03-06T07:21:58.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 -- 172.21.15.122:0/2491695016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 msgr2=0x7f449c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 --2- 172.21.15.122:0/2491695016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c0ce270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f449c0c5520 tx=0x7f448c02b910 comp rx=0 tx=0).stop 2024-03-06T07:21:58.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 -- 172.21.15.122:0/2491695016 shutdown_connections 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 --2- 172.21.15.122:0/2491695016 >> v2:172.21.15.169:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 --2- 172.21.15.122:0/2491695016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c0ce270 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 -- 172.21.15.122:0/2491695016 >> 172.21.15.122:0/2491695016 conn(0x7f449c09a9f0 msgr2=0x7f449c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 -- 172.21.15.122:0/2491695016 shutdown_connections 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.061+0000 7f44a258a640 1 -- 172.21.15.122:0/2491695016 wait complete. 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.062+0000 7f44a258a640 1 Processor -- start 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 -- start start 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f449c0d16e0 0x7f449c09f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f449c15bdd0 0x7f449c09ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0d2030 con 0x7f449c15fa10 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c15c1b0 con 0x7f449c15bdd0 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a258a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0ce7b0 con 0x7f449c0d16e0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a0b27640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f449bfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f449c0d16e0 0x7f449c09f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a0b27640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c15ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54484/0 (socket says 172.21.15.122:54484) 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f44a0b27640 1 -- 172.21.15.122:0/3861098221 learned_addr learned my addr 172.21.15.122:0/3861098221 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.063+0000 7f449b7fe640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f449c15bdd0 0x7f449c09ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a0b27640 1 -- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f449c0d16e0 msgr2=0x7f449c09f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a0b27640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f449c0d16e0 0x7f449c09f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a0b27640 1 -- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f449c15bdd0 msgr2=0x7f449c09ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a0b27640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f449c15bdd0 0x7f449c09ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a0b27640 1 -- 172.21.15.122:0/3861098221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c09fa40 con 0x7f449c15fa10 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f449bfff640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f449c0d16e0 0x7f449c09f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f449b7fe640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f449c15bdd0 0x7f449c09ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a0b27640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c15ca10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4494009dd0 tx=0x7f449400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494018020 con 0x7f449c15fa10 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4494014480 con 0x7f449c15fa10 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c15d5e0 con 0x7f449c15fa10 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494014870 con 0x7f449c15fa10 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.064+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c15dba0 con 0x7f449c15fa10 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.065+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c04f330 con 0x7f449c15fa10 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.065+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4494005d10 con 0x7f449c15fa10 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.066+0000 7f4498ff9640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f447c06b500 0x7f447c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.066+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f449408aac0 con 0x7f449c15fa10 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.066+0000 7f449bfff640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f447c06b500 0x7f447c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.066+0000 7f449bfff640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f447c06b500 0x7f447c06d9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f449c0b77d0 tx=0x7f4484009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.070+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4494022070 con 0x7f449c15fa10 2024-03-06T07:21:58.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.237+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f449c0b7a40 con 0x7f447c06b500 2024-03-06T07:21:58.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.243+0000 7f4498ff9640 1 -- 172.21.15.122:0/3861098221 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251944 (secure 0 0 0) 0x7f449c0b7a40 con 0x7f447c06b500 2024-03-06T07:21:58.247 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:21:58.249 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:21:58.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f447c06b500 msgr2=0x7f447c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f447c06b500 0x7f447c06d9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f449c0b77d0 tx=0x7f4484009040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 msgr2=0x7f449c15ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c15ca10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4494009dd0 tx=0x7f449400c490 comp rx=0 tx=0).stop 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 shutdown_connections 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f447c06b500 0x7f447c06d9c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f449c15fa10 0x7f449c15ca10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f449c15bdd0 0x7f449c09ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 --2- 172.21.15.122:0/3861098221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f449c0d16e0 0x7f449c09f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 >> 172.21.15.122:0/3861098221 conn(0x7f449c09a9f0 msgr2=0x7f449c06fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 shutdown_connections 2024-03-06T07:21:58.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:21:58.246+0000 7f44a258a640 1 -- 172.21.15.122:0/3861098221 wait complete. 2024-03-06T07:21:58.258 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-06T07:21:57.330908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218948,"kb_used_data":2752,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755772,"statfs":{"total":773094113280,"available":772869910528,"internally_reserved":0,"allocated":2818048,"data_stored":1715228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.798716"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778717+0000","last_change":"2024-03-06T07:21:53.563999+0000","last_active":"2024-03-06T07:21:54.778717+0000","last_peered":"2024-03-06T07:21:54.778717+0000","last_clean":"2024-03-06T07:21:54.778717+0000","last_became_active":"2024-03-06T07:21:53.563557+0000","last_became_peered":"2024-03-06T07:21:53.563557+0000","last_unstale":"2024-03-06T07:21:54.778717+0000","last_undegraded":"2024-03-06T07:21:54.778717+0000","last_fullsized":"2024-03-06T07:21:54.778717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562641+0000","last_change":"2024-03-06T07:21:53.560952+0000","last_active":"2024-03-06T07:21:54.562641+0000","last_peered":"2024-03-06T07:21:54.562641+0000","last_clean":"2024-03-06T07:21:54.562641+0000","last_became_active":"2024-03-06T07:21:53.560651+0000","last_became_peered":"2024-03-06T07:21:53.560651+0000","last_unstale":"2024-03-06T07:21:54.562641+0000","last_undegraded":"2024-03-06T07:21:54.562641+0000","last_fullsized":"2024-03-06T07:21:54.562641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777337+0000","last_change":"2024-03-06T07:21:53.547890+0000","last_active":"2024-03-06T07:21:54.777337+0000","last_peered":"2024-03-06T07:21:54.777337+0000","last_clean":"2024-03-06T07:21:54.777337+0000","last_became_active":"2024-03-06T07:21:53.547631+0000","last_became_peered":"2024-03-06T07:21:53.547631+0000","last_unstale":"2024-03-06T07:21:54.777337+0000","last_undegraded":"2024-03-06T07:21:54.777337+0000","last_fullsized":"2024-03-06T07:21:54.777337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778291+0000","last_change":"2024-03-06T07:21:53.565535+0000","last_active":"2024-03-06T07:21:54.778291+0000","last_peered":"2024-03-06T07:21:54.778291+0000","last_clean":"2024-03-06T07:21:54.778291+0000","last_became_active":"2024-03-06T07:21:53.565256+0000","last_became_peered":"2024-03-06T07:21:53.565256+0000","last_unstale":"2024-03-06T07:21:54.778291+0000","last_undegraded":"2024-03-06T07:21:54.778291+0000","last_fullsized":"2024-03-06T07:21:54.778291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778186+0000","last_change":"2024-03-06T07:21:53.558864+0000","last_active":"2024-03-06T07:21:54.778186+0000","last_peered":"2024-03-06T07:21:54.778186+0000","last_clean":"2024-03-06T07:21:54.778186+0000","last_became_active":"2024-03-06T07:21:53.558582+0000","last_became_peered":"2024-03-06T07:21:53.558582+0000","last_unstale":"2024-03-06T07:21:54.778186+0000","last_undegraded":"2024-03-06T07:21:54.778186+0000","last_fullsized":"2024-03-06T07:21:54.778186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777685+0000","last_change":"2024-03-06T07:21:53.567354+0000","last_active":"2024-03-06T07:21:54.777685+0000","last_peered":"2024-03-06T07:21:54.777685+0000","last_clean":"2024-03-06T07:21:54.777685+0000","last_became_active":"2024-03-06T07:21:53.567054+0000","last_became_peered":"2024-03-06T07:21:53.567054+0000","last_unstale":"2024-03-06T07:21:54.777685+0000","last_undegraded":"2024-03-06T07:21:54.777685+0000","last_fullsized":"2024-03-06T07:21:54.777685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777444+0000","last_change":"2024-03-06T07:21:53.563005+0000","last_active":"2024-03-06T07:21:54.777444+0000","last_peered":"2024-03-06T07:21:54.777444+0000","last_clean":"2024-03-06T07:21:54.777444+0000","last_became_active":"2024-03-06T07:21:53.562611+0000","last_became_peered":"2024-03-06T07:21:53.562611+0000","last_unstale":"2024-03-06T07:21:54.777444+0000","last_undegraded":"2024-03-06T07:21:54.777444+0000","last_fullsized":"2024-03-06T07:21:54.777444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.766995+0000","last_change":"2024-03-06T07:21:53.566285+0000","last_active":"2024-03-06T07:21:54.766995+0000","last_peered":"2024-03-06T07:21:54.766995+0000","last_clean":"2024-03-06T07:21:54.766995+0000","last_became_active":"2024-03-06T07:21:53.565429+0000","last_became_peered":"2024-03-06T07:21:53.565429+0000","last_unstale":"2024-03-06T07:21:54.766995+0000","last_undegraded":"2024-03-06T07:21:54.766995+0000","last_fullsized":"2024-03-06T07:21:54.766995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778770+0000","last_change":"2024-03-06T07:21:53.565916+0000","last_active":"2024-03-06T07:21:54.778770+0000","last_peered":"2024-03-06T07:21:54.778770+0000","last_clean":"2024-03-06T07:21:54.778770+0000","last_became_active":"2024-03-06T07:21:53.565555+0000","last_became_peered":"2024-03-06T07:21:53.565555+0000","last_unstale":"2024-03-06T07:21:54.778770+0000","last_undegraded":"2024-03-06T07:21:54.778770+0000","last_fullsized":"2024-03-06T07:21:54.778770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:23.946424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562690+0000","last_change":"2024-03-06T07:21:53.557656+0000","last_active":"2024-03-06T07:21:54.562690+0000","last_peered":"2024-03-06T07:21:54.562690+0000","last_clean":"2024-03-06T07:21:54.562690+0000","last_became_active":"2024-03-06T07:21:53.557224+0000","last_became_peered":"2024-03-06T07:21:53.557224+0000","last_unstale":"2024-03-06T07:21:54.562690+0000","last_undegraded":"2024-03-06T07:21:54.562690+0000","last_fullsized":"2024-03-06T07:21:54.562690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562612+0000","last_change":"2024-03-06T07:21:53.565698+0000","last_active":"2024-03-06T07:21:54.562612+0000","last_peered":"2024-03-06T07:21:54.562612+0000","last_clean":"2024-03-06T07:21:54.562612+0000","last_became_active":"2024-03-06T07:21:53.565390+0000","last_became_peered":"2024-03-06T07:21:53.565390+0000","last_unstale":"2024-03-06T07:21:54.562612+0000","last_undegraded":"2024-03-06T07:21:54.562612+0000","last_fullsized":"2024-03-06T07:21:54.562612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777702+0000","last_change":"2024-03-06T07:21:53.558368+0000","last_active":"2024-03-06T07:21:54.777702+0000","last_peered":"2024-03-06T07:21:54.777702+0000","last_clean":"2024-03-06T07:21:54.777702+0000","last_became_active":"2024-03-06T07:21:53.558065+0000","last_became_peered":"2024-03-06T07:21:53.558065+0000","last_unstale":"2024-03-06T07:21:54.777702+0000","last_undegraded":"2024-03-06T07:21:54.777702+0000","last_fullsized":"2024-03-06T07:21:54.777702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:44.794765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767222+0000","last_change":"2024-03-06T07:21:53.567165+0000","last_active":"2024-03-06T07:21:54.767222+0000","last_peered":"2024-03-06T07:21:54.767222+0000","last_clean":"2024-03-06T07:21:54.767222+0000","last_became_active":"2024-03-06T07:21:53.566826+0000","last_became_peered":"2024-03-06T07:21:53.566826+0000","last_unstale":"2024-03-06T07:21:54.767222+0000","last_undegraded":"2024-03-06T07:21:54.767222+0000","last_fullsized":"2024-03-06T07:21:54.767222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777131+0000","last_change":"2024-03-06T07:21:53.552931+0000","last_active":"2024-03-06T07:21:54.777131+0000","last_peered":"2024-03-06T07:21:54.777131+0000","last_clean":"2024-03-06T07:21:54.777131+0000","last_became_active":"2024-03-06T07:21:53.552654+0000","last_became_peered":"2024-03-06T07:21:53.552654+0000","last_unstale":"2024-03-06T07:21:54.777131+0000","last_undegraded":"2024-03-06T07:21:54.777131+0000","last_fullsized":"2024-03-06T07:21:54.777131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:04.669227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562518+0000","last_change":"2024-03-06T07:21:53.562294+0000","last_active":"2024-03-06T07:21:54.562518+0000","last_peered":"2024-03-06T07:21:54.562518+0000","last_clean":"2024-03-06T07:21:54.562518+0000","last_became_active":"2024-03-06T07:21:53.561996+0000","last_became_peered":"2024-03-06T07:21:53.561996+0000","last_unstale":"2024-03-06T07:21:54.562518+0000","last_undegraded":"2024-03-06T07:21:54.562518+0000","last_fullsized":"2024-03-06T07:21:54.562518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777264+0000","last_change":"2024-03-06T07:21:53.563516+0000","last_active":"2024-03-06T07:21:54.777264+0000","last_peered":"2024-03-06T07:21:54.777264+0000","last_clean":"2024-03-06T07:21:54.777264+0000","last_became_active":"2024-03-06T07:21:53.563237+0000","last_became_peered":"2024-03-06T07:21:53.563237+0000","last_unstale":"2024-03-06T07:21:54.777264+0000","last_undegraded":"2024-03-06T07:21:54.777264+0000","last_fullsized":"2024-03-06T07:21:54.777264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778868+0000","last_change":"2024-03-06T07:21:53.561815+0000","last_active":"2024-03-06T07:21:54.778868+0000","last_peered":"2024-03-06T07:21:54.778868+0000","last_clean":"2024-03-06T07:21:54.778868+0000","last_became_active":"2024-03-06T07:21:53.561189+0000","last_became_peered":"2024-03-06T07:21:53.561189+0000","last_unstale":"2024-03-06T07:21:54.778868+0000","last_undegraded":"2024-03-06T07:21:54.778868+0000","last_fullsized":"2024-03-06T07:21:54.778868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560237+0000","last_change":"2024-03-06T07:21:53.560903+0000","last_active":"2024-03-06T07:21:54.560237+0000","last_peered":"2024-03-06T07:21:54.560237+0000","last_clean":"2024-03-06T07:21:54.560237+0000","last_became_active":"2024-03-06T07:21:53.560624+0000","last_became_peered":"2024-03-06T07:21:53.560624+0000","last_unstale":"2024-03-06T07:21:54.560237+0000","last_undegraded":"2024-03-06T07:21:54.560237+0000","last_fullsized":"2024-03-06T07:21:54.560237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778508+0000","last_change":"2024-03-06T07:21:53.561059+0000","last_active":"2024-03-06T07:21:54.778508+0000","last_peered":"2024-03-06T07:21:54.778508+0000","last_clean":"2024-03-06T07:21:54.778508+0000","last_became_active":"2024-03-06T07:21:53.560799+0000","last_became_peered":"2024-03-06T07:21:53.560799+0000","last_unstale":"2024-03-06T07:21:54.778508+0000","last_undegraded":"2024-03-06T07:21:54.778508+0000","last_fullsized":"2024-03-06T07:21:54.778508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562750+0000","last_change":"2024-03-06T07:21:53.558851+0000","last_active":"2024-03-06T07:21:54.562750+0000","last_peered":"2024-03-06T07:21:54.562750+0000","last_clean":"2024-03-06T07:21:54.562750+0000","last_became_active":"2024-03-06T07:21:53.558543+0000","last_became_peered":"2024-03-06T07:21:53.558543+0000","last_unstale":"2024-03-06T07:21:54.562750+0000","last_undegraded":"2024-03-06T07:21:54.562750+0000","last_fullsized":"2024-03-06T07:21:54.562750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777302+0000","last_change":"2024-03-06T07:21:53.560684+0000","last_active":"2024-03-06T07:21:54.777302+0000","last_peered":"2024-03-06T07:21:54.777302+0000","last_clean":"2024-03-06T07:21:54.777302+0000","last_became_active":"2024-03-06T07:21:53.560428+0000","last_became_peered":"2024-03-06T07:21:53.560428+0000","last_unstale":"2024-03-06T07:21:54.777302+0000","last_undegraded":"2024-03-06T07:21:54.777302+0000","last_fullsized":"2024-03-06T07:21:54.777302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777813+0000","last_change":"2024-03-06T07:21:53.562732+0000","last_active":"2024-03-06T07:21:54.777813+0000","last_peered":"2024-03-06T07:21:54.777813+0000","last_clean":"2024-03-06T07:21:54.777813+0000","last_became_active":"2024-03-06T07:21:53.562351+0000","last_became_peered":"2024-03-06T07:21:53.562351+0000","last_unstale":"2024-03-06T07:21:54.777813+0000","last_undegraded":"2024-03-06T07:21:54.777813+0000","last_fullsized":"2024-03-06T07:21:54.777813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767070+0000","last_change":"2024-03-06T07:21:53.564304+0000","last_active":"2024-03-06T07:21:54.767070+0000","last_peered":"2024-03-06T07:21:54.767070+0000","last_clean":"2024-03-06T07:21:54.767070+0000","last_became_active":"2024-03-06T07:21:53.564008+0000","last_became_peered":"2024-03-06T07:21:53.564008+0000","last_unstale":"2024-03-06T07:21:54.767070+0000","last_undegraded":"2024-03-06T07:21:54.767070+0000","last_fullsized":"2024-03-06T07:21:54.767070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777401+0000","last_change":"2024-03-06T07:21:53.565197+0000","last_active":"2024-03-06T07:21:54.777401+0000","last_peered":"2024-03-06T07:21:54.777401+0000","last_clean":"2024-03-06T07:21:54.777401+0000","last_became_active":"2024-03-06T07:21:53.564878+0000","last_became_peered":"2024-03-06T07:21:53.564878+0000","last_unstale":"2024-03-06T07:21:54.777401+0000","last_undegraded":"2024-03-06T07:21:54.777401+0000","last_fullsized":"2024-03-06T07:21:54.777401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767433+0000","last_change":"2024-03-06T07:21:53.558718+0000","last_active":"2024-03-06T07:21:54.767433+0000","last_peered":"2024-03-06T07:21:54.767433+0000","last_clean":"2024-03-06T07:21:54.767433+0000","last_became_active":"2024-03-06T07:21:53.558454+0000","last_became_peered":"2024-03-06T07:21:53.558454+0000","last_unstale":"2024-03-06T07:21:54.767433+0000","last_undegraded":"2024-03-06T07:21:54.767433+0000","last_fullsized":"2024-03-06T07:21:54.767433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562735+0000","last_change":"2024-03-06T07:21:53.563894+0000","last_active":"2024-03-06T07:21:54.562735+0000","last_peered":"2024-03-06T07:21:54.562735+0000","last_clean":"2024-03-06T07:21:54.562735+0000","last_became_active":"2024-03-06T07:21:53.563493+0000","last_became_peered":"2024-03-06T07:21:53.563493+0000","last_unstale":"2024-03-06T07:21:54.562735+0000","last_undegraded":"2024-03-06T07:21:54.562735+0000","last_fullsized":"2024-03-06T07:21:54.562735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562575+0000","last_change":"2024-03-06T07:21:53.552128+0000","last_active":"2024-03-06T07:21:54.562575+0000","last_peered":"2024-03-06T07:21:54.562575+0000","last_clean":"2024-03-06T07:21:54.562575+0000","last_became_active":"2024-03-06T07:21:53.551826+0000","last_became_peered":"2024-03-06T07:21:53.551826+0000","last_unstale":"2024-03-06T07:21:54.562575+0000","last_undegraded":"2024-03-06T07:21:54.562575+0000","last_fullsized":"2024-03-06T07:21:54.562575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767379+0000","last_change":"2024-03-06T07:21:53.563089+0000","last_active":"2024-03-06T07:21:54.767379+0000","last_peered":"2024-03-06T07:21:54.767379+0000","last_clean":"2024-03-06T07:21:54.767379+0000","last_became_active":"2024-03-06T07:21:53.562833+0000","last_became_peered":"2024-03-06T07:21:53.562833+0000","last_unstale":"2024-03-06T07:21:54.767379+0000","last_undegraded":"2024-03-06T07:21:54.767379+0000","last_fullsized":"2024-03-06T07:21:54.767379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777387+0000","last_change":"2024-03-06T07:21:53.550849+0000","last_active":"2024-03-06T07:21:54.777387+0000","last_peered":"2024-03-06T07:21:54.777387+0000","last_clean":"2024-03-06T07:21:54.777387+0000","last_became_active":"2024-03-06T07:21:53.550562+0000","last_became_peered":"2024-03-06T07:21:53.550562+0000","last_unstale":"2024-03-06T07:21:54.777387+0000","last_undegraded":"2024-03-06T07:21:54.777387+0000","last_fullsized":"2024-03-06T07:21:54.777387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777942+0000","last_change":"2024-03-06T07:21:53.564528+0000","last_active":"2024-03-06T07:21:54.777942+0000","last_peered":"2024-03-06T07:21:54.777942+0000","last_clean":"2024-03-06T07:21:54.777942+0000","last_became_active":"2024-03-06T07:21:53.564250+0000","last_became_peered":"2024-03-06T07:21:53.564250+0000","last_unstale":"2024-03-06T07:21:54.777942+0000","last_undegraded":"2024-03-06T07:21:54.777942+0000","last_fullsized":"2024-03-06T07:21:54.777942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560202+0000","last_change":"2024-03-06T07:21:53.555673+0000","last_active":"2024-03-06T07:21:54.560202+0000","last_peered":"2024-03-06T07:21:54.560202+0000","last_clean":"2024-03-06T07:21:54.560202+0000","last_became_active":"2024-03-06T07:21:53.555337+0000","last_became_peered":"2024-03-06T07:21:53.555337+0000","last_unstale":"2024-03-06T07:21:54.560202+0000","last_undegraded":"2024-03-06T07:21:54.560202+0000","last_fullsized":"2024-03-06T07:21:54.560202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562445+0000","last_change":"2024-03-06T07:21:53.552447+0000","last_active":"2024-03-06T07:21:54.562445+0000","last_peered":"2024-03-06T07:21:54.562445+0000","last_clean":"2024-03-06T07:21:54.562445+0000","last_became_active":"2024-03-06T07:21:53.551907+0000","last_became_peered":"2024-03-06T07:21:53.551907+0000","last_unstale":"2024-03-06T07:21:54.562445+0000","last_undegraded":"2024-03-06T07:21:54.562445+0000","last_fullsized":"2024-03-06T07:21:54.562445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777543+0000","last_change":"2024-03-06T07:21:53.554149+0000","last_active":"2024-03-06T07:21:54.777543+0000","last_peered":"2024-03-06T07:21:54.777543+0000","last_clean":"2024-03-06T07:21:54.777543+0000","last_became_active":"2024-03-06T07:21:53.553811+0000","last_became_peered":"2024-03-06T07:21:53.553811+0000","last_unstale":"2024-03-06T07:21:54.777543+0000","last_undegraded":"2024-03-06T07:21:54.777543+0000","last_fullsized":"2024-03-06T07:21:54.777543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:11.236013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767399+0000","last_change":"2024-03-06T07:21:52.551262+0000","last_active":"2024-03-06T07:21:54.767399+0000","last_peered":"2024-03-06T07:21:54.767399+0000","last_clean":"2024-03-06T07:21:54.767399+0000","last_became_active":"2024-03-06T07:21:52.550349+0000","last_became_peered":"2024-03-06T07:21:52.550349+0000","last_unstale":"2024-03-06T07:21:54.767399+0000","last_undegraded":"2024-03-06T07:21:54.767399+0000","last_fullsized":"2024-03-06T07:21:54.767399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777450+0000","last_change":"2024-03-06T07:21:53.562215+0000","last_active":"2024-03-06T07:21:54.777450+0000","last_peered":"2024-03-06T07:21:54.777450+0000","last_clean":"2024-03-06T07:21:54.777450+0000","last_became_active":"2024-03-06T07:21:53.561933+0000","last_became_peered":"2024-03-06T07:21:53.561933+0000","last_unstale":"2024-03-06T07:21:54.777450+0000","last_undegraded":"2024-03-06T07:21:54.777450+0000","last_fullsized":"2024-03-06T07:21:54.777450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562398+0000","last_change":"2024-03-06T07:21:52.547490+0000","last_active":"2024-03-06T07:21:54.562398+0000","last_peered":"2024-03-06T07:21:54.562398+0000","last_clean":"2024-03-06T07:21:54.562398+0000","last_became_active":"2024-03-06T07:21:52.547168+0000","last_became_peered":"2024-03-06T07:21:52.547168+0000","last_unstale":"2024-03-06T07:21:54.562398+0000","last_undegraded":"2024-03-06T07:21:54.562398+0000","last_fullsized":"2024-03-06T07:21:54.562398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767143+0000","last_change":"2024-03-06T07:21:52.551838+0000","last_active":"2024-03-06T07:21:54.767143+0000","last_peered":"2024-03-06T07:21:54.767143+0000","last_clean":"2024-03-06T07:21:54.767143+0000","last_became_active":"2024-03-06T07:21:52.551599+0000","last_became_peered":"2024-03-06T07:21:52.551599+0000","last_unstale":"2024-03-06T07:21:54.767143+0000","last_undegraded":"2024-03-06T07:21:54.767143+0000","last_fullsized":"2024-03-06T07:21:54.767143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767031+0000","last_change":"2024-03-06T07:21:53.557369+0000","last_active":"2024-03-06T07:21:54.767031+0000","last_peered":"2024-03-06T07:21:54.767031+0000","last_clean":"2024-03-06T07:21:54.767031+0000","last_became_active":"2024-03-06T07:21:53.556320+0000","last_became_peered":"2024-03-06T07:21:53.556320+0000","last_unstale":"2024-03-06T07:21:54.767031+0000","last_undegraded":"2024-03-06T07:21:54.767031+0000","last_fullsized":"2024-03-06T07:21:54.767031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777454+0000","last_change":"2024-03-06T07:21:52.553279+0000","last_active":"2024-03-06T07:21:54.777454+0000","last_peered":"2024-03-06T07:21:54.777454+0000","last_clean":"2024-03-06T07:21:54.777454+0000","last_became_active":"2024-03-06T07:21:52.552991+0000","last_became_peered":"2024-03-06T07:21:52.552991+0000","last_unstale":"2024-03-06T07:21:54.777454+0000","last_undegraded":"2024-03-06T07:21:54.777454+0000","last_fullsized":"2024-03-06T07:21:54.777454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562572+0000","last_change":"2024-03-06T07:21:53.552639+0000","last_active":"2024-03-06T07:21:54.562572+0000","last_peered":"2024-03-06T07:21:54.562572+0000","last_clean":"2024-03-06T07:21:54.562572+0000","last_became_active":"2024-03-06T07:21:53.552377+0000","last_became_peered":"2024-03-06T07:21:53.552377+0000","last_unstale":"2024-03-06T07:21:54.562572+0000","last_undegraded":"2024-03-06T07:21:54.562572+0000","last_fullsized":"2024-03-06T07:21:54.562572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562577+0000","last_change":"2024-03-06T07:21:52.556094+0000","last_active":"2024-03-06T07:21:54.562577+0000","last_peered":"2024-03-06T07:21:54.562577+0000","last_clean":"2024-03-06T07:21:54.562577+0000","last_became_active":"2024-03-06T07:21:52.555825+0000","last_became_peered":"2024-03-06T07:21:52.555825+0000","last_unstale":"2024-03-06T07:21:54.562577+0000","last_undegraded":"2024-03-06T07:21:54.562577+0000","last_fullsized":"2024-03-06T07:21:54.562577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778464+0000","last_change":"2024-03-06T07:21:53.561469+0000","last_active":"2024-03-06T07:21:54.778464+0000","last_peered":"2024-03-06T07:21:54.778464+0000","last_clean":"2024-03-06T07:21:54.778464+0000","last_became_active":"2024-03-06T07:21:53.561204+0000","last_became_peered":"2024-03-06T07:21:53.561204+0000","last_unstale":"2024-03-06T07:21:54.778464+0000","last_undegraded":"2024-03-06T07:21:54.778464+0000","last_fullsized":"2024-03-06T07:21:54.778464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778543+0000","last_change":"2024-03-06T07:21:52.557218+0000","last_active":"2024-03-06T07:21:54.778543+0000","last_peered":"2024-03-06T07:21:54.778543+0000","last_clean":"2024-03-06T07:21:54.778543+0000","last_became_active":"2024-03-06T07:21:52.556955+0000","last_became_peered":"2024-03-06T07:21:52.556955+0000","last_unstale":"2024-03-06T07:21:54.778543+0000","last_undegraded":"2024-03-06T07:21:54.778543+0000","last_fullsized":"2024-03-06T07:21:54.778543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777433+0000","last_change":"2024-03-06T07:21:53.559108+0000","last_active":"2024-03-06T07:21:54.777433+0000","last_peered":"2024-03-06T07:21:54.777433+0000","last_clean":"2024-03-06T07:21:54.777433+0000","last_became_active":"2024-03-06T07:21:53.558256+0000","last_became_peered":"2024-03-06T07:21:53.558256+0000","last_unstale":"2024-03-06T07:21:54.777433+0000","last_undegraded":"2024-03-06T07:21:54.777433+0000","last_fullsized":"2024-03-06T07:21:54.777433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562655+0000","last_change":"2024-03-06T07:21:52.557898+0000","last_active":"2024-03-06T07:21:54.562655+0000","last_peered":"2024-03-06T07:21:54.562655+0000","last_clean":"2024-03-06T07:21:54.562655+0000","last_became_active":"2024-03-06T07:21:52.557619+0000","last_became_peered":"2024-03-06T07:21:52.557619+0000","last_unstale":"2024-03-06T07:21:54.562655+0000","last_undegraded":"2024-03-06T07:21:54.562655+0000","last_fullsized":"2024-03-06T07:21:54.562655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777844+0000","last_change":"2024-03-06T07:21:53.557112+0000","last_active":"2024-03-06T07:21:54.777844+0000","last_peered":"2024-03-06T07:21:54.777844+0000","last_clean":"2024-03-06T07:21:54.777844+0000","last_became_active":"2024-03-06T07:21:53.556848+0000","last_became_peered":"2024-03-06T07:21:53.556848+0000","last_unstale":"2024-03-06T07:21:54.777844+0000","last_undegraded":"2024-03-06T07:21:54.777844+0000","last_fullsized":"2024-03-06T07:21:54.777844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562841+0000","last_change":"2024-03-06T07:21:52.548388+0000","last_active":"2024-03-06T07:21:54.562841+0000","last_peered":"2024-03-06T07:21:54.562841+0000","last_clean":"2024-03-06T07:21:54.562841+0000","last_became_active":"2024-03-06T07:21:52.548095+0000","last_became_peered":"2024-03-06T07:21:52.548095+0000","last_unstale":"2024-03-06T07:21:54.562841+0000","last_undegraded":"2024-03-06T07:21:54.562841+0000","last_fullsized":"2024-03-06T07:21:54.562841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778266+0000","last_change":"2024-03-06T07:21:53.557821+0000","last_active":"2024-03-06T07:21:54.778266+0000","last_peered":"2024-03-06T07:21:54.778266+0000","last_clean":"2024-03-06T07:21:54.778266+0000","last_became_active":"2024-03-06T07:21:53.557536+0000","last_became_peered":"2024-03-06T07:21:53.557536+0000","last_unstale":"2024-03-06T07:21:54.778266+0000","last_undegraded":"2024-03-06T07:21:54.778266+0000","last_fullsized":"2024-03-06T07:21:54.778266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777481+0000","last_change":"2024-03-06T07:21:52.555948+0000","last_active":"2024-03-06T07:21:54.777481+0000","last_peered":"2024-03-06T07:21:54.777481+0000","last_clean":"2024-03-06T07:21:54.777481+0000","last_became_active":"2024-03-06T07:21:52.555657+0000","last_became_peered":"2024-03-06T07:21:52.555657+0000","last_unstale":"2024-03-06T07:21:54.777481+0000","last_undegraded":"2024-03-06T07:21:54.777481+0000","last_fullsized":"2024-03-06T07:21:54.777481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777191+0000","last_change":"2024-03-06T07:21:53.556393+0000","last_active":"2024-03-06T07:21:54.777191+0000","last_peered":"2024-03-06T07:21:54.777191+0000","last_clean":"2024-03-06T07:21:54.777191+0000","last_became_active":"2024-03-06T07:21:53.556137+0000","last_became_peered":"2024-03-06T07:21:53.556137+0000","last_unstale":"2024-03-06T07:21:54.777191+0000","last_undegraded":"2024-03-06T07:21:54.777191+0000","last_fullsized":"2024-03-06T07:21:54.777191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:33.561906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778025+0000","last_change":"2024-03-06T07:21:52.556426+0000","last_active":"2024-03-06T07:21:54.778025+0000","last_peered":"2024-03-06T07:21:54.778025+0000","last_clean":"2024-03-06T07:21:54.778025+0000","last_became_active":"2024-03-06T07:21:52.555844+0000","last_became_peered":"2024-03-06T07:21:52.555844+0000","last_unstale":"2024-03-06T07:21:54.778025+0000","last_undegraded":"2024-03-06T07:21:54.778025+0000","last_fullsized":"2024-03-06T07:21:54.778025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778299+0000","last_change":"2024-03-06T07:21:53.567432+0000","last_active":"2024-03-06T07:21:54.778299+0000","last_peered":"2024-03-06T07:21:54.778299+0000","last_clean":"2024-03-06T07:21:54.778299+0000","last_became_active":"2024-03-06T07:21:53.566788+0000","last_became_peered":"2024-03-06T07:21:53.566788+0000","last_unstale":"2024-03-06T07:21:54.778299+0000","last_undegraded":"2024-03-06T07:21:54.778299+0000","last_fullsized":"2024-03-06T07:21:54.778299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562338+0000","last_change":"2024-03-06T07:21:52.547444+0000","last_active":"2024-03-06T07:21:54.562338+0000","last_peered":"2024-03-06T07:21:54.562338+0000","last_clean":"2024-03-06T07:21:54.562338+0000","last_became_active":"2024-03-06T07:21:52.547163+0000","last_became_peered":"2024-03-06T07:21:52.547163+0000","last_unstale":"2024-03-06T07:21:54.562338+0000","last_undegraded":"2024-03-06T07:21:54.562338+0000","last_fullsized":"2024-03-06T07:21:54.562338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562392+0000","last_change":"2024-03-06T07:21:53.552210+0000","last_active":"2024-03-06T07:21:54.562392+0000","last_peered":"2024-03-06T07:21:54.562392+0000","last_clean":"2024-03-06T07:21:54.562392+0000","last_became_active":"2024-03-06T07:21:53.551529+0000","last_became_peered":"2024-03-06T07:21:53.551529+0000","last_unstale":"2024-03-06T07:21:54.562392+0000","last_undegraded":"2024-03-06T07:21:54.562392+0000","last_fullsized":"2024-03-06T07:21:54.562392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560082+0000","last_change":"2024-03-06T07:21:52.552286+0000","last_active":"2024-03-06T07:21:54.560082+0000","last_peered":"2024-03-06T07:21:54.560082+0000","last_clean":"2024-03-06T07:21:54.560082+0000","last_became_active":"2024-03-06T07:21:52.552010+0000","last_became_peered":"2024-03-06T07:21:52.552010+0000","last_unstale":"2024-03-06T07:21:54.560082+0000","last_undegraded":"2024-03-06T07:21:54.560082+0000","last_fullsized":"2024-03-06T07:21:54.560082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777580+0000","last_change":"2024-03-06T07:21:53.571841+0000","last_active":"2024-03-06T07:21:54.777580+0000","last_peered":"2024-03-06T07:21:54.777580+0000","last_clean":"2024-03-06T07:21:54.777580+0000","last_became_active":"2024-03-06T07:21:53.571350+0000","last_became_peered":"2024-03-06T07:21:53.571350+0000","last_unstale":"2024-03-06T07:21:54.777580+0000","last_undegraded":"2024-03-06T07:21:54.777580+0000","last_fullsized":"2024-03-06T07:21:54.777580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777653+0000","last_change":"2024-03-06T07:21:52.553990+0000","last_active":"2024-03-06T07:21:54.777653+0000","last_peered":"2024-03-06T07:21:54.777653+0000","last_clean":"2024-03-06T07:21:54.777653+0000","last_became_active":"2024-03-06T07:21:52.553693+0000","last_became_peered":"2024-03-06T07:21:52.553693+0000","last_unstale":"2024-03-06T07:21:54.777653+0000","last_undegraded":"2024-03-06T07:21:54.777653+0000","last_fullsized":"2024-03-06T07:21:54.777653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777984+0000","last_change":"2024-03-06T07:21:53.567545+0000","last_active":"2024-03-06T07:21:54.777984+0000","last_peered":"2024-03-06T07:21:54.777984+0000","last_clean":"2024-03-06T07:21:54.777984+0000","last_became_active":"2024-03-06T07:21:53.566157+0000","last_became_peered":"2024-03-06T07:21:53.566157+0000","last_unstale":"2024-03-06T07:21:54.777984+0000","last_undegraded":"2024-03-06T07:21:54.777984+0000","last_fullsized":"2024-03-06T07:21:54.777984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560201+0000","last_change":"2024-03-06T07:21:52.547686+0000","last_active":"2024-03-06T07:21:54.560201+0000","last_peered":"2024-03-06T07:21:54.560201+0000","last_clean":"2024-03-06T07:21:54.560201+0000","last_became_active":"2024-03-06T07:21:52.547436+0000","last_became_peered":"2024-03-06T07:21:52.547436+0000","last_unstale":"2024-03-06T07:21:54.560201+0000","last_undegraded":"2024-03-06T07:21:54.560201+0000","last_fullsized":"2024-03-06T07:21:54.560201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778797+0000","last_change":"2024-03-06T07:21:53.567657+0000","last_active":"2024-03-06T07:21:54.778797+0000","last_peered":"2024-03-06T07:21:54.778797+0000","last_clean":"2024-03-06T07:21:54.778797+0000","last_became_active":"2024-03-06T07:21:53.567134+0000","last_became_peered":"2024-03-06T07:21:53.567134+0000","last_unstale":"2024-03-06T07:21:54.778797+0000","last_undegraded":"2024-03-06T07:21:54.778797+0000","last_fullsized":"2024-03-06T07:21:54.778797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:57.048252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562696+0000","last_change":"2024-03-06T07:21:52.556768+0000","last_active":"2024-03-06T07:21:54.562696+0000","last_peered":"2024-03-06T07:21:54.562696+0000","last_clean":"2024-03-06T07:21:54.562696+0000","last_became_active":"2024-03-06T07:21:52.556507+0000","last_became_peered":"2024-03-06T07:21:52.556507+0000","last_unstale":"2024-03-06T07:21:54.562696+0000","last_undegraded":"2024-03-06T07:21:54.562696+0000","last_fullsized":"2024-03-06T07:21:54.562696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778541+0000","last_change":"2024-03-06T07:21:53.563768+0000","last_active":"2024-03-06T07:21:54.778541+0000","last_peered":"2024-03-06T07:21:54.778541+0000","last_clean":"2024-03-06T07:21:54.778541+0000","last_became_active":"2024-03-06T07:21:53.563490+0000","last_became_peered":"2024-03-06T07:21:53.563490+0000","last_unstale":"2024-03-06T07:21:54.778541+0000","last_undegraded":"2024-03-06T07:21:54.778541+0000","last_fullsized":"2024-03-06T07:21:54.778541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560265+0000","last_change":"2024-03-06T07:21:52.557535+0000","last_active":"2024-03-06T07:21:54.560265+0000","last_peered":"2024-03-06T07:21:54.560265+0000","last_clean":"2024-03-06T07:21:54.560265+0000","last_became_active":"2024-03-06T07:21:52.557275+0000","last_became_peered":"2024-03-06T07:21:52.557275+0000","last_unstale":"2024-03-06T07:21:54.560265+0000","last_undegraded":"2024-03-06T07:21:54.560265+0000","last_fullsized":"2024-03-06T07:21:54.560265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778386+0000","last_change":"2024-03-06T07:21:53.565629+0000","last_active":"2024-03-06T07:21:54.778386+0000","last_peered":"2024-03-06T07:21:54.778386+0000","last_clean":"2024-03-06T07:21:54.778386+0000","last_became_active":"2024-03-06T07:21:53.563789+0000","last_became_peered":"2024-03-06T07:21:53.563789+0000","last_unstale":"2024-03-06T07:21:54.778386+0000","last_undegraded":"2024-03-06T07:21:54.778386+0000","last_fullsized":"2024-03-06T07:21:54.778386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778063+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:54.778063+0000","last_peered":"2024-03-06T07:21:54.778063+0000","last_clean":"2024-03-06T07:21:54.778063+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:54.778063+0000","last_undegraded":"2024-03-06T07:21:54.778063+0000","last_fullsized":"2024-03-06T07:21:54.778063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777283+0000","last_change":"2024-03-06T07:21:52.557611+0000","last_active":"2024-03-06T07:21:54.777283+0000","last_peered":"2024-03-06T07:21:54.777283+0000","last_clean":"2024-03-06T07:21:54.777283+0000","last_became_active":"2024-03-06T07:21:52.556594+0000","last_became_peered":"2024-03-06T07:21:52.556594+0000","last_unstale":"2024-03-06T07:21:54.777283+0000","last_undegraded":"2024-03-06T07:21:54.777283+0000","last_fullsized":"2024-03-06T07:21:54.777283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767209+0000","last_change":"2024-03-06T07:21:53.565607+0000","last_active":"2024-03-06T07:21:54.767209+0000","last_peered":"2024-03-06T07:21:54.767209+0000","last_clean":"2024-03-06T07:21:54.767209+0000","last_became_active":"2024-03-06T07:21:53.565356+0000","last_became_peered":"2024-03-06T07:21:53.565356+0000","last_unstale":"2024-03-06T07:21:54.767209+0000","last_undegraded":"2024-03-06T07:21:54.767209+0000","last_fullsized":"2024-03-06T07:21:54.767209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778021+0000","last_change":"2024-03-06T07:21:52.555927+0000","last_active":"2024-03-06T07:21:54.778021+0000","last_peered":"2024-03-06T07:21:54.778021+0000","last_clean":"2024-03-06T07:21:54.778021+0000","last_became_active":"2024-03-06T07:21:52.555485+0000","last_became_peered":"2024-03-06T07:21:52.555485+0000","last_unstale":"2024-03-06T07:21:54.778021+0000","last_undegraded":"2024-03-06T07:21:54.778021+0000","last_fullsized":"2024-03-06T07:21:54.778021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777097+0000","last_change":"2024-03-06T07:21:53.559225+0000","last_active":"2024-03-06T07:21:54.777097+0000","last_peered":"2024-03-06T07:21:54.777097+0000","last_clean":"2024-03-06T07:21:54.777097+0000","last_became_active":"2024-03-06T07:21:53.558925+0000","last_became_peered":"2024-03-06T07:21:53.558925+0000","last_unstale":"2024-03-06T07:21:54.777097+0000","last_undegraded":"2024-03-06T07:21:54.777097+0000","last_fullsized":"2024-03-06T07:21:54.777097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:54.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560174+0000","last_change":"2024-03-06T07:21:52.546620+0000","last_active":"2024-03-06T07:21:54.560174+0000","last_peered":"2024-03-06T07:21:54.560174+0000","last_clean":"2024-03-06T07:21:54.560174+0000","last_became_active":"2024-03-06T07:21:52.546297+0000","last_became_peered":"2024-03-06T07:21:52.546297+0000","last_unstale":"2024-03-06T07:21:54.560174+0000","last_undegraded":"2024-03-06T07:21:54.560174+0000","last_fullsized":"2024-03-06T07:21:54.560174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562488+0000","last_change":"2024-03-06T07:21:53.553218+0000","last_active":"2024-03-06T07:21:54.562488+0000","last_peered":"2024-03-06T07:21:54.562488+0000","last_clean":"2024-03-06T07:21:54.562488+0000","last_became_active":"2024-03-06T07:21:53.552911+0000","last_became_peered":"2024-03-06T07:21:53.552911+0000","last_unstale":"2024-03-06T07:21:54.562488+0000","last_undegraded":"2024-03-06T07:21:54.562488+0000","last_fullsized":"2024-03-06T07:21:54.562488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778113+0000","last_change":"2024-03-06T07:21:52.556096+0000","last_active":"2024-03-06T07:21:54.778113+0000","last_peered":"2024-03-06T07:21:54.778113+0000","last_clean":"2024-03-06T07:21:54.778113+0000","last_became_active":"2024-03-06T07:21:52.555510+0000","last_became_peered":"2024-03-06T07:21:52.555510+0000","last_unstale":"2024-03-06T07:21:54.778113+0000","last_undegraded":"2024-03-06T07:21:54.778113+0000","last_fullsized":"2024-03-06T07:21:54.778113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778491+0000","last_change":"2024-03-06T07:21:53.561767+0000","last_active":"2024-03-06T07:21:54.778491+0000","last_peered":"2024-03-06T07:21:54.778491+0000","last_clean":"2024-03-06T07:21:54.778491+0000","last_became_active":"2024-03-06T07:21:53.561435+0000","last_became_peered":"2024-03-06T07:21:53.561435+0000","last_unstale":"2024-03-06T07:21:54.778491+0000","last_undegraded":"2024-03-06T07:21:54.778491+0000","last_fullsized":"2024-03-06T07:21:54.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:24.631055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562673+0000","last_change":"2024-03-06T07:21:52.553733+0000","last_active":"2024-03-06T07:21:54.562673+0000","last_peered":"2024-03-06T07:21:54.562673+0000","last_clean":"2024-03-06T07:21:54.562673+0000","last_became_active":"2024-03-06T07:21:52.553436+0000","last_became_peered":"2024-03-06T07:21:52.553436+0000","last_unstale":"2024-03-06T07:21:54.562673+0000","last_undegraded":"2024-03-06T07:21:54.562673+0000","last_fullsized":"2024-03-06T07:21:54.562673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779356+0000","last_change":"2024-03-06T07:21:53.556153+0000","last_active":"2024-03-06T07:21:54.779356+0000","last_peered":"2024-03-06T07:21:54.779356+0000","last_clean":"2024-03-06T07:21:54.779356+0000","last_became_active":"2024-03-06T07:21:53.555896+0000","last_became_peered":"2024-03-06T07:21:53.555896+0000","last_unstale":"2024-03-06T07:21:54.779356+0000","last_undegraded":"2024-03-06T07:21:54.779356+0000","last_fullsized":"2024-03-06T07:21:54.779356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778511+0000","last_change":"2024-03-06T07:21:52.557084+0000","last_active":"2024-03-06T07:21:54.778511+0000","last_peered":"2024-03-06T07:21:54.778511+0000","last_clean":"2024-03-06T07:21:54.778511+0000","last_became_active":"2024-03-06T07:21:52.556688+0000","last_became_peered":"2024-03-06T07:21:52.556688+0000","last_unstale":"2024-03-06T07:21:54.778511+0000","last_undegraded":"2024-03-06T07:21:54.778511+0000","last_fullsized":"2024-03-06T07:21:54.778511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:23.626336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562632+0000","last_change":"2024-03-06T07:21:53.552698+0000","last_active":"2024-03-06T07:21:54.562632+0000","last_peered":"2024-03-06T07:21:54.562632+0000","last_clean":"2024-03-06T07:21:54.562632+0000","last_became_active":"2024-03-06T07:21:53.552434+0000","last_became_peered":"2024-03-06T07:21:53.552434+0000","last_unstale":"2024-03-06T07:21:54.562632+0000","last_undegraded":"2024-03-06T07:21:54.562632+0000","last_fullsized":"2024-03-06T07:21:54.562632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562589+0000","last_change":"2024-03-06T07:21:52.553671+0000","last_active":"2024-03-06T07:21:54.562589+0000","last_peered":"2024-03-06T07:21:54.562589+0000","last_clean":"2024-03-06T07:21:54.562589+0000","last_became_active":"2024-03-06T07:21:52.553394+0000","last_became_peered":"2024-03-06T07:21:52.553394+0000","last_unstale":"2024-03-06T07:21:54.562589+0000","last_undegraded":"2024-03-06T07:21:54.562589+0000","last_fullsized":"2024-03-06T07:21:54.562589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767449+0000","last_change":"2024-03-06T07:21:53.555702+0000","last_active":"2024-03-06T07:21:54.767449+0000","last_peered":"2024-03-06T07:21:54.767449+0000","last_clean":"2024-03-06T07:21:54.767449+0000","last_became_active":"2024-03-06T07:21:53.555436+0000","last_became_peered":"2024-03-06T07:21:53.555436+0000","last_unstale":"2024-03-06T07:21:54.767449+0000","last_undegraded":"2024-03-06T07:21:54.767449+0000","last_fullsized":"2024-03-06T07:21:54.767449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562639+0000","last_change":"2024-03-06T07:21:52.550448+0000","last_active":"2024-03-06T07:21:54.562639+0000","last_peered":"2024-03-06T07:21:54.562639+0000","last_clean":"2024-03-06T07:21:54.562639+0000","last_became_active":"2024-03-06T07:21:52.550179+0000","last_became_peered":"2024-03-06T07:21:52.550179+0000","last_unstale":"2024-03-06T07:21:54.562639+0000","last_undegraded":"2024-03-06T07:21:54.562639+0000","last_fullsized":"2024-03-06T07:21:54.562639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777421+0000","last_change":"2024-03-06T07:21:53.553031+0000","last_active":"2024-03-06T07:21:54.777421+0000","last_peered":"2024-03-06T07:21:54.777421+0000","last_clean":"2024-03-06T07:21:54.777421+0000","last_became_active":"2024-03-06T07:21:53.552759+0000","last_became_peered":"2024-03-06T07:21:53.552759+0000","last_unstale":"2024-03-06T07:21:54.777421+0000","last_undegraded":"2024-03-06T07:21:54.777421+0000","last_fullsized":"2024-03-06T07:21:54.777421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777184+0000","last_change":"2024-03-06T07:21:52.555781+0000","last_active":"2024-03-06T07:21:54.777184+0000","last_peered":"2024-03-06T07:21:54.777184+0000","last_clean":"2024-03-06T07:21:54.777184+0000","last_became_active":"2024-03-06T07:21:52.555469+0000","last_became_peered":"2024-03-06T07:21:52.555469+0000","last_unstale":"2024-03-06T07:21:54.777184+0000","last_undegraded":"2024-03-06T07:21:54.777184+0000","last_fullsized":"2024-03-06T07:21:54.777184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777797+0000","last_change":"2024-03-06T07:21:53.558488+0000","last_active":"2024-03-06T07:21:54.777797+0000","last_peered":"2024-03-06T07:21:54.777797+0000","last_clean":"2024-03-06T07:21:54.777797+0000","last_became_active":"2024-03-06T07:21:53.558199+0000","last_became_peered":"2024-03-06T07:21:53.558199+0000","last_unstale":"2024-03-06T07:21:54.777797+0000","last_undegraded":"2024-03-06T07:21:54.777797+0000","last_fullsized":"2024-03-06T07:21:54.777797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562638+0000","last_change":"2024-03-06T07:21:52.556952+0000","last_active":"2024-03-06T07:21:54.562638+0000","last_peered":"2024-03-06T07:21:54.562638+0000","last_clean":"2024-03-06T07:21:54.562638+0000","last_became_active":"2024-03-06T07:21:52.556629+0000","last_became_peered":"2024-03-06T07:21:52.556629+0000","last_unstale":"2024-03-06T07:21:54.562638+0000","last_undegraded":"2024-03-06T07:21:54.562638+0000","last_fullsized":"2024-03-06T07:21:54.562638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767159+0000","last_change":"2024-03-06T07:21:53.554655+0000","last_active":"2024-03-06T07:21:54.767159+0000","last_peered":"2024-03-06T07:21:54.767159+0000","last_clean":"2024-03-06T07:21:54.767159+0000","last_became_active":"2024-03-06T07:21:53.554321+0000","last_became_peered":"2024-03-06T07:21:53.554321+0000","last_unstale":"2024-03-06T07:21:54.767159+0000","last_undegraded":"2024-03-06T07:21:54.767159+0000","last_fullsized":"2024-03-06T07:21:54.767159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562564+0000","last_change":"2024-03-06T07:21:52.547548+0000","last_active":"2024-03-06T07:21:54.562564+0000","last_peered":"2024-03-06T07:21:54.562564+0000","last_clean":"2024-03-06T07:21:54.562564+0000","last_became_active":"2024-03-06T07:21:52.547285+0000","last_became_peered":"2024-03-06T07:21:52.547285+0000","last_unstale":"2024-03-06T07:21:54.562564+0000","last_undegraded":"2024-03-06T07:21:54.562564+0000","last_fullsized":"2024-03-06T07:21:54.562564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767314+0000","last_change":"2024-03-06T07:21:53.557506+0000","last_active":"2024-03-06T07:21:54.767314+0000","last_peered":"2024-03-06T07:21:54.767314+0000","last_clean":"2024-03-06T07:21:54.767314+0000","last_became_active":"2024-03-06T07:21:53.556568+0000","last_became_peered":"2024-03-06T07:21:53.556568+0000","last_unstale":"2024-03-06T07:21:54.767314+0000","last_undegraded":"2024-03-06T07:21:54.767314+0000","last_fullsized":"2024-03-06T07:21:54.767314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778329+0000","last_change":"2024-03-06T07:21:52.558820+0000","last_active":"2024-03-06T07:21:54.778329+0000","last_peered":"2024-03-06T07:21:54.778329+0000","last_clean":"2024-03-06T07:21:54.778329+0000","last_became_active":"2024-03-06T07:21:52.558486+0000","last_became_peered":"2024-03-06T07:21:52.558486+0000","last_unstale":"2024-03-06T07:21:54.778329+0000","last_undegraded":"2024-03-06T07:21:54.778329+0000","last_fullsized":"2024-03-06T07:21:54.778329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562402+0000","last_change":"2024-03-06T07:21:53.558230+0000","last_active":"2024-03-06T07:21:54.562402+0000","last_peered":"2024-03-06T07:21:54.562402+0000","last_clean":"2024-03-06T07:21:54.562402+0000","last_became_active":"2024-03-06T07:21:53.557916+0000","last_became_peered":"2024-03-06T07:21:53.557916+0000","last_unstale":"2024-03-06T07:21:54.562402+0000","last_undegraded":"2024-03-06T07:21:54.562402+0000","last_fullsized":"2024-03-06T07:21:54.562402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779232+0000","last_change":"2024-03-06T07:21:53.567503+0000","last_active":"2024-03-06T07:21:54.779232+0000","last_peered":"2024-03-06T07:21:54.779232+0000","last_clean":"2024-03-06T07:21:54.779232+0000","last_became_active":"2024-03-06T07:21:53.567223+0000","last_became_peered":"2024-03-06T07:21:53.567223+0000","last_unstale":"2024-03-06T07:21:54.779232+0000","last_undegraded":"2024-03-06T07:21:54.779232+0000","last_fullsized":"2024-03-06T07:21:54.779232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777697+0000","last_change":"2024-03-06T07:21:52.559682+0000","last_active":"2024-03-06T07:21:54.777697+0000","last_peered":"2024-03-06T07:21:54.777697+0000","last_clean":"2024-03-06T07:21:54.777697+0000","last_became_active":"2024-03-06T07:21:52.559430+0000","last_became_peered":"2024-03-06T07:21:52.559430+0000","last_unstale":"2024-03-06T07:21:54.777697+0000","last_undegraded":"2024-03-06T07:21:54.777697+0000","last_fullsized":"2024-03-06T07:21:54.777697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778415+0000","last_change":"2024-03-06T07:21:53.568862+0000","last_active":"2024-03-06T07:21:54.778415+0000","last_peered":"2024-03-06T07:21:54.778415+0000","last_clean":"2024-03-06T07:21:54.778415+0000","last_became_active":"2024-03-06T07:21:53.568127+0000","last_became_peered":"2024-03-06T07:21:53.568127+0000","last_unstale":"2024-03-06T07:21:54.778415+0000","last_undegraded":"2024-03-06T07:21:54.778415+0000","last_fullsized":"2024-03-06T07:21:54.778415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778429+0000","last_change":"2024-03-06T07:21:52.558983+0000","last_active":"2024-03-06T07:21:54.778429+0000","last_peered":"2024-03-06T07:21:54.778429+0000","last_clean":"2024-03-06T07:21:54.778429+0000","last_became_active":"2024-03-06T07:21:52.558616+0000","last_became_peered":"2024-03-06T07:21:52.558616+0000","last_unstale":"2024-03-06T07:21:54.778429+0000","last_undegraded":"2024-03-06T07:21:54.778429+0000","last_fullsized":"2024-03-06T07:21:54.778429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777374+0000","last_change":"2024-03-06T07:21:53.558956+0000","last_active":"2024-03-06T07:21:54.777374+0000","last_peered":"2024-03-06T07:21:54.777374+0000","last_clean":"2024-03-06T07:21:54.777374+0000","last_became_active":"2024-03-06T07:21:53.558612+0000","last_became_peered":"2024-03-06T07:21:53.558612+0000","last_unstale":"2024-03-06T07:21:54.777374+0000","last_undegraded":"2024-03-06T07:21:54.777374+0000","last_fullsized":"2024-03-06T07:21:54.777374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767442+0000","last_change":"2024-03-06T07:21:52.553080+0000","last_active":"2024-03-06T07:21:54.767442+0000","last_peered":"2024-03-06T07:21:54.767442+0000","last_clean":"2024-03-06T07:21:54.767442+0000","last_became_active":"2024-03-06T07:21:52.552785+0000","last_became_peered":"2024-03-06T07:21:52.552785+0000","last_unstale":"2024-03-06T07:21:54.767442+0000","last_undegraded":"2024-03-06T07:21:54.767442+0000","last_fullsized":"2024-03-06T07:21:54.767442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767397+0000","last_change":"2024-03-06T07:21:53.563091+0000","last_active":"2024-03-06T07:21:54.767397+0000","last_peered":"2024-03-06T07:21:54.767397+0000","last_clean":"2024-03-06T07:21:54.767397+0000","last_became_active":"2024-03-06T07:21:53.562797+0000","last_became_peered":"2024-03-06T07:21:53.562797+0000","last_unstale":"2024-03-06T07:21:54.767397+0000","last_undegraded":"2024-03-06T07:21:54.767397+0000","last_fullsized":"2024-03-06T07:21:54.767397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777504+0000","last_change":"2024-03-06T07:21:52.557097+0000","last_active":"2024-03-06T07:21:54.777504+0000","last_peered":"2024-03-06T07:21:54.777504+0000","last_clean":"2024-03-06T07:21:54.777504+0000","last_became_active":"2024-03-06T07:21:52.556808+0000","last_became_peered":"2024-03-06T07:21:52.556808+0000","last_unstale":"2024-03-06T07:21:54.777504+0000","last_undegraded":"2024-03-06T07:21:54.777504+0000","last_fullsized":"2024-03-06T07:21:54.777504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:01.264 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:01.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.374+0000 7f61e0743640 1 Processor -- start 2024-03-06T07:22:01.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.374+0000 7f61e0743640 1 -- start start 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61e0743640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61d80af6c0 0x7f61d80afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61e0743640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815bb10 0x7f61d815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61e0743640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d80b4e30 con 0x7f61d815ab00 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61e0743640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d80b0060 con 0x7f61d80af6c0 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61e0743640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80b5110 con 0x7f61d815bb10 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61ddcde640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61d80af6c0 0x7f61d80afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61dece0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815bb10 0x7f61d815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61de4df640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f61d815ab00 0x7f61d80b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:33932/0 (socket says 172.21.15.122:33932) 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61de4df640 1 -- 172.21.15.122:0/2743790676 learned_addr learned my addr 172.21.15.122:0/2743790676 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.375+0000 7f61ddcde640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61d80af6c0 0x7f61d80afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54492/0 (socket says 172.21.15.122:54492) 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.376+0000 7f61ddcde640 1 -- 172.21.15.122:0/2743790676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815bb10 msgr2=0x7f61d815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.376+0000 7f61ddcde640 1 --2- 172.21.15.122:0/2743790676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815bb10 0x7f61d815bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.376+0000 7f61ddcde640 1 -- 172.21.15.122:0/2743790676 >> v1:172.21.15.169:6789/0 conn(0x7f61d815ab00 legacy=0x7f61d80b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.376+0000 7f61dd4dd640 1 -- 172.21.15.122:0/2743790676 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111200657 0 0) 0x7f61d80b4e30 con 0x7f61d815ab00 2024-03-06T07:22:01.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.376+0000 7f61ddcde640 1 -- 172.21.15.122:0/2743790676 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d815b310 con 0x7f61d80af6c0 2024-03-06T07:22:01.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.376+0000 7f61ddcde640 1 --2- 172.21.15.122:0/2743790676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d80af6c0 0x7f61d80afb20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f61c80097a0 tx=0x7f61c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42800eb4c08be87c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61dd4dd640 1 -- 172.21.15.122:0/2743790676 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8035020 con 0x7f61d80af6c0 2024-03-06T07:22:01.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61dd4dd640 1 -- 172.21.15.122:0/2743790676 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c8031830 con 0x7f61d80af6c0 2024-03-06T07:22:01.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61dd4dd640 1 -- 172.21.15.122:0/2743790676 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8031c00 con 0x7f61d80af6c0 2024-03-06T07:22:01.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 -- 172.21.15.122:0/2743790676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d80af6c0 msgr2=0x7f61d80afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 --2- 172.21.15.122:0/2743790676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d80af6c0 0x7f61d80afb20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f61c80097a0 tx=0x7f61c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 -- 172.21.15.122:0/2743790676 shutdown_connections 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 --2- 172.21.15.122:0/2743790676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815bb10 0x7f61d815bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 --2- 172.21.15.122:0/2743790676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d80af6c0 0x7f61d80afb20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 -- 172.21.15.122:0/2743790676 >> 172.21.15.122:0/2743790676 conn(0x7f61d806c0c0 msgr2=0x7f61d806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 -- 172.21.15.122:0/2743790676 shutdown_connections 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.377+0000 7f61e0743640 1 -- 172.21.15.122:0/2743790676 wait complete. 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.378+0000 7f61e0743640 1 Processor -- start 2024-03-06T07:22:01.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.378+0000 7f61e0743640 1 -- start start 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61e0743640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d80af6c0 0x7f61d8148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61e0743640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815ab00 0x7f61d81476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61e0743640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 0x7f61d80cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61e0743640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d8071b20 con 0x7f61d815bb10 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61e0743640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80b4ac0 con 0x7f61d80af6c0 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61e0743640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d8071ca0 con 0x7f61d815ab00 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61dece0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 0x7f61d80cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61dece0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 0x7f61d80cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54498/0 (socket says 172.21.15.122:54498) 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61dece0640 1 -- 172.21.15.122:0/1042540547 learned_addr learned my addr 172.21.15.122:0/1042540547 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:01.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61de4df640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d80af6c0 0x7f61d8148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.379+0000 7f61ddcde640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815ab00 0x7f61d81476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61dece0640 1 -- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815ab00 msgr2=0x7f61d81476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61dece0640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815ab00 0x7f61d81476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61dece0640 1 -- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d80af6c0 msgr2=0x7f61d8148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61dece0640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d80af6c0 0x7f61d8148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61dece0640 1 -- 172.21.15.122:0/1042540547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d80ce740 con 0x7f61d815bb10 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61dece0640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 0x7f61d80cf8e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f61d000e820 tx=0x7f61d000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0017070 con 0x7f61d815bb10 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d000c730 con 0x7f61d815bb10 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d80cea20 con 0x7f61d815bb10 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d000cb00 con 0x7f61d815bb10 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.380+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d80cefe0 con 0x7f61d815bb10 2024-03-06T07:22:01.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.381+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d804f330 con 0x7f61d815bb10 2024-03-06T07:22:01.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.381+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61d001a070 con 0x7f61d815bb10 2024-03-06T07:22:01.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.382+0000 7f61ceffd640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.382+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61d008a310 con 0x7f61d815bb10 2024-03-06T07:22:01.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.382+0000 7f61de4df640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.382+0000 7f61de4df640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f61d40012e0 tx=0x7f61d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.386+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61d0058b40 con 0x7f61d815bb10 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.555+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61d8147be0 con 0x7f61bc06b4d0 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.562+0000 7f61ceffd640 1 -- 172.21.15.122:0/1042540547 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7f61d4001fc0 con 0x7f61bc06b4d0 2024-03-06T07:22:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:01.568 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:01.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 msgr2=0x7f61bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f61d40012e0 tx=0x7f61d4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 msgr2=0x7f61d80cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 0x7f61d80cf8e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f61d000e820 tx=0x7f61d000ece0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 shutdown_connections 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d815bb10 0x7f61d80cf8e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d815ab00 0x7f61d81476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 --2- 172.21.15.122:0/1042540547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d80af6c0 0x7f61d8148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 >> 172.21.15.122:0/1042540547 conn(0x7f61d806c0c0 msgr2=0x7f61d80b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 shutdown_connections 2024-03-06T07:22:01.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.565+0000 7f61e0743640 1 -- 172.21.15.122:0/1042540547 wait complete. 2024-03-06T07:22:01.578 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:22:01.331851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218996,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755724,"statfs":{"total":773094113280,"available":772869861376,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001737"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778717+0000","last_change":"2024-03-06T07:21:53.563999+0000","last_active":"2024-03-06T07:21:54.778717+0000","last_peered":"2024-03-06T07:21:54.778717+0000","last_clean":"2024-03-06T07:21:54.778717+0000","last_became_active":"2024-03-06T07:21:53.563557+0000","last_became_peered":"2024-03-06T07:21:53.563557+0000","last_unstale":"2024-03-06T07:21:54.778717+0000","last_undegraded":"2024-03-06T07:21:54.778717+0000","last_fullsized":"2024-03-06T07:21:54.778717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562641+0000","last_change":"2024-03-06T07:21:53.560952+0000","last_active":"2024-03-06T07:21:54.562641+0000","last_peered":"2024-03-06T07:21:54.562641+0000","last_clean":"2024-03-06T07:21:54.562641+0000","last_became_active":"2024-03-06T07:21:53.560651+0000","last_became_peered":"2024-03-06T07:21:53.560651+0000","last_unstale":"2024-03-06T07:21:54.562641+0000","last_undegraded":"2024-03-06T07:21:54.562641+0000","last_fullsized":"2024-03-06T07:21:54.562641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777337+0000","last_change":"2024-03-06T07:21:53.547890+0000","last_active":"2024-03-06T07:21:54.777337+0000","last_peered":"2024-03-06T07:21:54.777337+0000","last_clean":"2024-03-06T07:21:54.777337+0000","last_became_active":"2024-03-06T07:21:53.547631+0000","last_became_peered":"2024-03-06T07:21:53.547631+0000","last_unstale":"2024-03-06T07:21:54.777337+0000","last_undegraded":"2024-03-06T07:21:54.777337+0000","last_fullsized":"2024-03-06T07:21:54.777337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778291+0000","last_change":"2024-03-06T07:21:53.565535+0000","last_active":"2024-03-06T07:21:54.778291+0000","last_peered":"2024-03-06T07:21:54.778291+0000","last_clean":"2024-03-06T07:21:54.778291+0000","last_became_active":"2024-03-06T07:21:53.565256+0000","last_became_peered":"2024-03-06T07:21:53.565256+0000","last_unstale":"2024-03-06T07:21:54.778291+0000","last_undegraded":"2024-03-06T07:21:54.778291+0000","last_fullsized":"2024-03-06T07:21:54.778291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778186+0000","last_change":"2024-03-06T07:21:53.558864+0000","last_active":"2024-03-06T07:21:54.778186+0000","last_peered":"2024-03-06T07:21:54.778186+0000","last_clean":"2024-03-06T07:21:54.778186+0000","last_became_active":"2024-03-06T07:21:53.558582+0000","last_became_peered":"2024-03-06T07:21:53.558582+0000","last_unstale":"2024-03-06T07:21:54.778186+0000","last_undegraded":"2024-03-06T07:21:54.778186+0000","last_fullsized":"2024-03-06T07:21:54.778186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777685+0000","last_change":"2024-03-06T07:21:53.567354+0000","last_active":"2024-03-06T07:21:54.777685+0000","last_peered":"2024-03-06T07:21:54.777685+0000","last_clean":"2024-03-06T07:21:54.777685+0000","last_became_active":"2024-03-06T07:21:53.567054+0000","last_became_peered":"2024-03-06T07:21:53.567054+0000","last_unstale":"2024-03-06T07:21:54.777685+0000","last_undegraded":"2024-03-06T07:21:54.777685+0000","last_fullsized":"2024-03-06T07:21:54.777685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777444+0000","last_change":"2024-03-06T07:21:53.563005+0000","last_active":"2024-03-06T07:21:54.777444+0000","last_peered":"2024-03-06T07:21:54.777444+0000","last_clean":"2024-03-06T07:21:54.777444+0000","last_became_active":"2024-03-06T07:21:53.562611+0000","last_became_peered":"2024-03-06T07:21:53.562611+0000","last_unstale":"2024-03-06T07:21:54.777444+0000","last_undegraded":"2024-03-06T07:21:54.777444+0000","last_fullsized":"2024-03-06T07:21:54.777444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.766995+0000","last_change":"2024-03-06T07:21:53.566285+0000","last_active":"2024-03-06T07:21:54.766995+0000","last_peered":"2024-03-06T07:21:54.766995+0000","last_clean":"2024-03-06T07:21:54.766995+0000","last_became_active":"2024-03-06T07:21:53.565429+0000","last_became_peered":"2024-03-06T07:21:53.565429+0000","last_unstale":"2024-03-06T07:21:54.766995+0000","last_undegraded":"2024-03-06T07:21:54.766995+0000","last_fullsized":"2024-03-06T07:21:54.766995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778770+0000","last_change":"2024-03-06T07:21:53.565916+0000","last_active":"2024-03-06T07:21:54.778770+0000","last_peered":"2024-03-06T07:21:54.778770+0000","last_clean":"2024-03-06T07:21:54.778770+0000","last_became_active":"2024-03-06T07:21:53.565555+0000","last_became_peered":"2024-03-06T07:21:53.565555+0000","last_unstale":"2024-03-06T07:21:54.778770+0000","last_undegraded":"2024-03-06T07:21:54.778770+0000","last_fullsized":"2024-03-06T07:21:54.778770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:23.946424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562690+0000","last_change":"2024-03-06T07:21:53.557656+0000","last_active":"2024-03-06T07:21:54.562690+0000","last_peered":"2024-03-06T07:21:54.562690+0000","last_clean":"2024-03-06T07:21:54.562690+0000","last_became_active":"2024-03-06T07:21:53.557224+0000","last_became_peered":"2024-03-06T07:21:53.557224+0000","last_unstale":"2024-03-06T07:21:54.562690+0000","last_undegraded":"2024-03-06T07:21:54.562690+0000","last_fullsized":"2024-03-06T07:21:54.562690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562612+0000","last_change":"2024-03-06T07:21:53.565698+0000","last_active":"2024-03-06T07:21:54.562612+0000","last_peered":"2024-03-06T07:21:54.562612+0000","last_clean":"2024-03-06T07:21:54.562612+0000","last_became_active":"2024-03-06T07:21:53.565390+0000","last_became_peered":"2024-03-06T07:21:53.565390+0000","last_unstale":"2024-03-06T07:21:54.562612+0000","last_undegraded":"2024-03-06T07:21:54.562612+0000","last_fullsized":"2024-03-06T07:21:54.562612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777702+0000","last_change":"2024-03-06T07:21:53.558368+0000","last_active":"2024-03-06T07:21:54.777702+0000","last_peered":"2024-03-06T07:21:54.777702+0000","last_clean":"2024-03-06T07:21:54.777702+0000","last_became_active":"2024-03-06T07:21:53.558065+0000","last_became_peered":"2024-03-06T07:21:53.558065+0000","last_unstale":"2024-03-06T07:21:54.777702+0000","last_undegraded":"2024-03-06T07:21:54.777702+0000","last_fullsized":"2024-03-06T07:21:54.777702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:44.794765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767222+0000","last_change":"2024-03-06T07:21:53.567165+0000","last_active":"2024-03-06T07:21:54.767222+0000","last_peered":"2024-03-06T07:21:54.767222+0000","last_clean":"2024-03-06T07:21:54.767222+0000","last_became_active":"2024-03-06T07:21:53.566826+0000","last_became_peered":"2024-03-06T07:21:53.566826+0000","last_unstale":"2024-03-06T07:21:54.767222+0000","last_undegraded":"2024-03-06T07:21:54.767222+0000","last_fullsized":"2024-03-06T07:21:54.767222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777131+0000","last_change":"2024-03-06T07:21:53.552931+0000","last_active":"2024-03-06T07:21:54.777131+0000","last_peered":"2024-03-06T07:21:54.777131+0000","last_clean":"2024-03-06T07:21:54.777131+0000","last_became_active":"2024-03-06T07:21:53.552654+0000","last_became_peered":"2024-03-06T07:21:53.552654+0000","last_unstale":"2024-03-06T07:21:54.777131+0000","last_undegraded":"2024-03-06T07:21:54.777131+0000","last_fullsized":"2024-03-06T07:21:54.777131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:04.669227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562518+0000","last_change":"2024-03-06T07:21:53.562294+0000","last_active":"2024-03-06T07:21:54.562518+0000","last_peered":"2024-03-06T07:21:54.562518+0000","last_clean":"2024-03-06T07:21:54.562518+0000","last_became_active":"2024-03-06T07:21:53.561996+0000","last_became_peered":"2024-03-06T07:21:53.561996+0000","last_unstale":"2024-03-06T07:21:54.562518+0000","last_undegraded":"2024-03-06T07:21:54.562518+0000","last_fullsized":"2024-03-06T07:21:54.562518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777264+0000","last_change":"2024-03-06T07:21:53.563516+0000","last_active":"2024-03-06T07:21:54.777264+0000","last_peered":"2024-03-06T07:21:54.777264+0000","last_clean":"2024-03-06T07:21:54.777264+0000","last_became_active":"2024-03-06T07:21:53.563237+0000","last_became_peered":"2024-03-06T07:21:53.563237+0000","last_unstale":"2024-03-06T07:21:54.777264+0000","last_undegraded":"2024-03-06T07:21:54.777264+0000","last_fullsized":"2024-03-06T07:21:54.777264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778868+0000","last_change":"2024-03-06T07:21:53.561815+0000","last_active":"2024-03-06T07:21:54.778868+0000","last_peered":"2024-03-06T07:21:54.778868+0000","last_clean":"2024-03-06T07:21:54.778868+0000","last_became_active":"2024-03-06T07:21:53.561189+0000","last_became_peered":"2024-03-06T07:21:53.561189+0000","last_unstale":"2024-03-06T07:21:54.778868+0000","last_undegraded":"2024-03-06T07:21:54.778868+0000","last_fullsized":"2024-03-06T07:21:54.778868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560237+0000","last_change":"2024-03-06T07:21:53.560903+0000","last_active":"2024-03-06T07:21:54.560237+0000","last_peered":"2024-03-06T07:21:54.560237+0000","last_clean":"2024-03-06T07:21:54.560237+0000","last_became_active":"2024-03-06T07:21:53.560624+0000","last_became_peered":"2024-03-06T07:21:53.560624+0000","last_unstale":"2024-03-06T07:21:54.560237+0000","last_undegraded":"2024-03-06T07:21:54.560237+0000","last_fullsized":"2024-03-06T07:21:54.560237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778508+0000","last_change":"2024-03-06T07:21:53.561059+0000","last_active":"2024-03-06T07:21:54.778508+0000","last_peered":"2024-03-06T07:21:54.778508+0000","last_clean":"2024-03-06T07:21:54.778508+0000","last_became_active":"2024-03-06T07:21:53.560799+0000","last_became_peered":"2024-03-06T07:21:53.560799+0000","last_unstale":"2024-03-06T07:21:54.778508+0000","last_undegraded":"2024-03-06T07:21:54.778508+0000","last_fullsized":"2024-03-06T07:21:54.778508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562750+0000","last_change":"2024-03-06T07:21:53.558851+0000","last_active":"2024-03-06T07:21:54.562750+0000","last_peered":"2024-03-06T07:21:54.562750+0000","last_clean":"2024-03-06T07:21:54.562750+0000","last_became_active":"2024-03-06T07:21:53.558543+0000","last_became_peered":"2024-03-06T07:21:53.558543+0000","last_unstale":"2024-03-06T07:21:54.562750+0000","last_undegraded":"2024-03-06T07:21:54.562750+0000","last_fullsized":"2024-03-06T07:21:54.562750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777302+0000","last_change":"2024-03-06T07:21:53.560684+0000","last_active":"2024-03-06T07:21:54.777302+0000","last_peered":"2024-03-06T07:21:54.777302+0000","last_clean":"2024-03-06T07:21:54.777302+0000","last_became_active":"2024-03-06T07:21:53.560428+0000","last_became_peered":"2024-03-06T07:21:53.560428+0000","last_unstale":"2024-03-06T07:21:54.777302+0000","last_undegraded":"2024-03-06T07:21:54.777302+0000","last_fullsized":"2024-03-06T07:21:54.777302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777813+0000","last_change":"2024-03-06T07:21:53.562732+0000","last_active":"2024-03-06T07:21:54.777813+0000","last_peered":"2024-03-06T07:21:54.777813+0000","last_clean":"2024-03-06T07:21:54.777813+0000","last_became_active":"2024-03-06T07:21:53.562351+0000","last_became_peered":"2024-03-06T07:21:53.562351+0000","last_unstale":"2024-03-06T07:21:54.777813+0000","last_undegraded":"2024-03-06T07:21:54.777813+0000","last_fullsized":"2024-03-06T07:21:54.777813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767070+0000","last_change":"2024-03-06T07:21:53.564304+0000","last_active":"2024-03-06T07:21:54.767070+0000","last_peered":"2024-03-06T07:21:54.767070+0000","last_clean":"2024-03-06T07:21:54.767070+0000","last_became_active":"2024-03-06T07:21:53.564008+0000","last_became_peered":"2024-03-06T07:21:53.564008+0000","last_unstale":"2024-03-06T07:21:54.767070+0000","last_undegraded":"2024-03-06T07:21:54.767070+0000","last_fullsized":"2024-03-06T07:21:54.767070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777401+0000","last_change":"2024-03-06T07:21:53.565197+0000","last_active":"2024-03-06T07:21:54.777401+0000","last_peered":"2024-03-06T07:21:54.777401+0000","last_clean":"2024-03-06T07:21:54.777401+0000","last_became_active":"2024-03-06T07:21:53.564878+0000","last_became_peered":"2024-03-06T07:21:53.564878+0000","last_unstale":"2024-03-06T07:21:54.777401+0000","last_undegraded":"2024-03-06T07:21:54.777401+0000","last_fullsized":"2024-03-06T07:21:54.777401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767433+0000","last_change":"2024-03-06T07:21:53.558718+0000","last_active":"2024-03-06T07:21:54.767433+0000","last_peered":"2024-03-06T07:21:54.767433+0000","last_clean":"2024-03-06T07:21:54.767433+0000","last_became_active":"2024-03-06T07:21:53.558454+0000","last_became_peered":"2024-03-06T07:21:53.558454+0000","last_unstale":"2024-03-06T07:21:54.767433+0000","last_undegraded":"2024-03-06T07:21:54.767433+0000","last_fullsized":"2024-03-06T07:21:54.767433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562735+0000","last_change":"2024-03-06T07:21:53.563894+0000","last_active":"2024-03-06T07:21:54.562735+0000","last_peered":"2024-03-06T07:21:54.562735+0000","last_clean":"2024-03-06T07:21:54.562735+0000","last_became_active":"2024-03-06T07:21:53.563493+0000","last_became_peered":"2024-03-06T07:21:53.563493+0000","last_unstale":"2024-03-06T07:21:54.562735+0000","last_undegraded":"2024-03-06T07:21:54.562735+0000","last_fullsized":"2024-03-06T07:21:54.562735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562575+0000","last_change":"2024-03-06T07:21:53.552128+0000","last_active":"2024-03-06T07:21:54.562575+0000","last_peered":"2024-03-06T07:21:54.562575+0000","last_clean":"2024-03-06T07:21:54.562575+0000","last_became_active":"2024-03-06T07:21:53.551826+0000","last_became_peered":"2024-03-06T07:21:53.551826+0000","last_unstale":"2024-03-06T07:21:54.562575+0000","last_undegraded":"2024-03-06T07:21:54.562575+0000","last_fullsized":"2024-03-06T07:21:54.562575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767379+0000","last_change":"2024-03-06T07:21:53.563089+0000","last_active":"2024-03-06T07:21:54.767379+0000","last_peered":"2024-03-06T07:21:54.767379+0000","last_clean":"2024-03-06T07:21:54.767379+0000","last_became_active":"2024-03-06T07:21:53.562833+0000","last_became_peered":"2024-03-06T07:21:53.562833+0000","last_unstale":"2024-03-06T07:21:54.767379+0000","last_undegraded":"2024-03-06T07:21:54.767379+0000","last_fullsized":"2024-03-06T07:21:54.767379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777387+0000","last_change":"2024-03-06T07:21:53.550849+0000","last_active":"2024-03-06T07:21:54.777387+0000","last_peered":"2024-03-06T07:21:54.777387+0000","last_clean":"2024-03-06T07:21:54.777387+0000","last_became_active":"2024-03-06T07:21:53.550562+0000","last_became_peered":"2024-03-06T07:21:53.550562+0000","last_unstale":"2024-03-06T07:21:54.777387+0000","last_undegraded":"2024-03-06T07:21:54.777387+0000","last_fullsized":"2024-03-06T07:21:54.777387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777942+0000","last_change":"2024-03-06T07:21:53.564528+0000","last_active":"2024-03-06T07:21:54.777942+0000","last_peered":"2024-03-06T07:21:54.777942+0000","last_clean":"2024-03-06T07:21:54.777942+0000","last_became_active":"2024-03-06T07:21:53.564250+0000","last_became_peered":"2024-03-06T07:21:53.564250+0000","last_unstale":"2024-03-06T07:21:54.777942+0000","last_undegraded":"2024-03-06T07:21:54.777942+0000","last_fullsized":"2024-03-06T07:21:54.777942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560202+0000","last_change":"2024-03-06T07:21:53.555673+0000","last_active":"2024-03-06T07:21:54.560202+0000","last_peered":"2024-03-06T07:21:54.560202+0000","last_clean":"2024-03-06T07:21:54.560202+0000","last_became_active":"2024-03-06T07:21:53.555337+0000","last_became_peered":"2024-03-06T07:21:53.555337+0000","last_unstale":"2024-03-06T07:21:54.560202+0000","last_undegraded":"2024-03-06T07:21:54.560202+0000","last_fullsized":"2024-03-06T07:21:54.560202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562445+0000","last_change":"2024-03-06T07:21:53.552447+0000","last_active":"2024-03-06T07:21:54.562445+0000","last_peered":"2024-03-06T07:21:54.562445+0000","last_clean":"2024-03-06T07:21:54.562445+0000","last_became_active":"2024-03-06T07:21:53.551907+0000","last_became_peered":"2024-03-06T07:21:53.551907+0000","last_unstale":"2024-03-06T07:21:54.562445+0000","last_undegraded":"2024-03-06T07:21:54.562445+0000","last_fullsized":"2024-03-06T07:21:54.562445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777543+0000","last_change":"2024-03-06T07:21:53.554149+0000","last_active":"2024-03-06T07:21:54.777543+0000","last_peered":"2024-03-06T07:21:54.777543+0000","last_clean":"2024-03-06T07:21:54.777543+0000","last_became_active":"2024-03-06T07:21:53.553811+0000","last_became_peered":"2024-03-06T07:21:53.553811+0000","last_unstale":"2024-03-06T07:21:54.777543+0000","last_undegraded":"2024-03-06T07:21:54.777543+0000","last_fullsized":"2024-03-06T07:21:54.777543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:11.236013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767399+0000","last_change":"2024-03-06T07:21:52.551262+0000","last_active":"2024-03-06T07:21:54.767399+0000","last_peered":"2024-03-06T07:21:54.767399+0000","last_clean":"2024-03-06T07:21:54.767399+0000","last_became_active":"2024-03-06T07:21:52.550349+0000","last_became_peered":"2024-03-06T07:21:52.550349+0000","last_unstale":"2024-03-06T07:21:54.767399+0000","last_undegraded":"2024-03-06T07:21:54.767399+0000","last_fullsized":"2024-03-06T07:21:54.767399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777450+0000","last_change":"2024-03-06T07:21:53.562215+0000","last_active":"2024-03-06T07:21:54.777450+0000","last_peered":"2024-03-06T07:21:54.777450+0000","last_clean":"2024-03-06T07:21:54.777450+0000","last_became_active":"2024-03-06T07:21:53.561933+0000","last_became_peered":"2024-03-06T07:21:53.561933+0000","last_unstale":"2024-03-06T07:21:54.777450+0000","last_undegraded":"2024-03-06T07:21:54.777450+0000","last_fullsized":"2024-03-06T07:21:54.777450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562398+0000","last_change":"2024-03-06T07:21:52.547490+0000","last_active":"2024-03-06T07:21:54.562398+0000","last_peered":"2024-03-06T07:21:54.562398+0000","last_clean":"2024-03-06T07:21:54.562398+0000","last_became_active":"2024-03-06T07:21:52.547168+0000","last_became_peered":"2024-03-06T07:21:52.547168+0000","last_unstale":"2024-03-06T07:21:54.562398+0000","last_undegraded":"2024-03-06T07:21:54.562398+0000","last_fullsized":"2024-03-06T07:21:54.562398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767143+0000","last_change":"2024-03-06T07:21:52.551838+0000","last_active":"2024-03-06T07:21:54.767143+0000","last_peered":"2024-03-06T07:21:54.767143+0000","last_clean":"2024-03-06T07:21:54.767143+0000","last_became_active":"2024-03-06T07:21:52.551599+0000","last_became_peered":"2024-03-06T07:21:52.551599+0000","last_unstale":"2024-03-06T07:21:54.767143+0000","last_undegraded":"2024-03-06T07:21:54.767143+0000","last_fullsized":"2024-03-06T07:21:54.767143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767031+0000","last_change":"2024-03-06T07:21:53.557369+0000","last_active":"2024-03-06T07:21:54.767031+0000","last_peered":"2024-03-06T07:21:54.767031+0000","last_clean":"2024-03-06T07:21:54.767031+0000","last_became_active":"2024-03-06T07:21:53.556320+0000","last_became_peered":"2024-03-06T07:21:53.556320+0000","last_unstale":"2024-03-06T07:21:54.767031+0000","last_undegraded":"2024-03-06T07:21:54.767031+0000","last_fullsized":"2024-03-06T07:21:54.767031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777454+0000","last_change":"2024-03-06T07:21:52.553279+0000","last_active":"2024-03-06T07:21:54.777454+0000","last_peered":"2024-03-06T07:21:54.777454+0000","last_clean":"2024-03-06T07:21:54.777454+0000","last_became_active":"2024-03-06T07:21:52.552991+0000","last_became_peered":"2024-03-06T07:21:52.552991+0000","last_unstale":"2024-03-06T07:21:54.777454+0000","last_undegraded":"2024-03-06T07:21:54.777454+0000","last_fullsized":"2024-03-06T07:21:54.777454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562572+0000","last_change":"2024-03-06T07:21:53.552639+0000","last_active":"2024-03-06T07:21:54.562572+0000","last_peered":"2024-03-06T07:21:54.562572+0000","last_clean":"2024-03-06T07:21:54.562572+0000","last_became_active":"2024-03-06T07:21:53.552377+0000","last_became_peered":"2024-03-06T07:21:53.552377+0000","last_unstale":"2024-03-06T07:21:54.562572+0000","last_undegraded":"2024-03-06T07:21:54.562572+0000","last_fullsized":"2024-03-06T07:21:54.562572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562577+0000","last_change":"2024-03-06T07:21:52.556094+0000","last_active":"2024-03-06T07:21:54.562577+0000","last_peered":"2024-03-06T07:21:54.562577+0000","last_clean":"2024-03-06T07:21:54.562577+0000","last_became_active":"2024-03-06T07:21:52.555825+0000","last_became_peered":"2024-03-06T07:21:52.555825+0000","last_unstale":"2024-03-06T07:21:54.562577+0000","last_undegraded":"2024-03-06T07:21:54.562577+0000","last_fullsized":"2024-03-06T07:21:54.562577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778464+0000","last_change":"2024-03-06T07:21:53.561469+0000","last_active":"2024-03-06T07:21:54.778464+0000","last_peered":"2024-03-06T07:21:54.778464+0000","last_clean":"2024-03-06T07:21:54.778464+0000","last_became_active":"2024-03-06T07:21:53.561204+0000","last_became_peered":"2024-03-06T07:21:53.561204+0000","last_unstale":"2024-03-06T07:21:54.778464+0000","last_undegraded":"2024-03-06T07:21:54.778464+0000","last_fullsized":"2024-03-06T07:21:54.778464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778543+0000","last_change":"2024-03-06T07:21:52.557218+0000","last_active":"2024-03-06T07:21:54.778543+0000","last_peered":"2024-03-06T07:21:54.778543+0000","last_clean":"2024-03-06T07:21:54.778543+0000","last_became_active":"2024-03-06T07:21:52.556955+0000","last_became_peered":"2024-03-06T07:21:52.556955+0000","last_unstale":"2024-03-06T07:21:54.778543+0000","last_undegraded":"2024-03-06T07:21:54.778543+0000","last_fullsized":"2024-03-06T07:21:54.778543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777433+0000","last_change":"2024-03-06T07:21:53.559108+0000","last_active":"2024-03-06T07:21:54.777433+0000","last_peered":"2024-03-06T07:21:54.777433+0000","last_clean":"2024-03-06T07:21:54.777433+0000","last_became_active":"2024-03-06T07:21:53.558256+0000","last_became_peered":"2024-03-06T07:21:53.558256+0000","last_unstale":"2024-03-06T07:21:54.777433+0000","last_undegraded":"2024-03-06T07:21:54.777433+0000","last_fullsized":"2024-03-06T07:21:54.777433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562655+0000","last_change":"2024-03-06T07:21:52.557898+0000","last_active":"2024-03-06T07:21:54.562655+0000","last_peered":"2024-03-06T07:21:54.562655+0000","last_clean":"2024-03-06T07:21:54.562655+0000","last_became_active":"2024-03-06T07:21:52.557619+0000","last_became_peered":"2024-03-06T07:21:52.557619+0000","last_unstale":"2024-03-06T07:21:54.562655+0000","last_undegraded":"2024-03-06T07:21:54.562655+0000","last_fullsized":"2024-03-06T07:21:54.562655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777844+0000","last_change":"2024-03-06T07:21:53.557112+0000","last_active":"2024-03-06T07:21:54.777844+0000","last_peered":"2024-03-06T07:21:54.777844+0000","last_clean":"2024-03-06T07:21:54.777844+0000","last_became_active":"2024-03-06T07:21:53.556848+0000","last_became_peered":"2024-03-06T07:21:53.556848+0000","last_unstale":"2024-03-06T07:21:54.777844+0000","last_undegraded":"2024-03-06T07:21:54.777844+0000","last_fullsized":"2024-03-06T07:21:54.777844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562841+0000","last_change":"2024-03-06T07:21:52.548388+0000","last_active":"2024-03-06T07:21:54.562841+0000","last_peered":"2024-03-06T07:21:54.562841+0000","last_clean":"2024-03-06T07:21:54.562841+0000","last_became_active":"2024-03-06T07:21:52.548095+0000","last_became_peered":"2024-03-06T07:21:52.548095+0000","last_unstale":"2024-03-06T07:21:54.562841+0000","last_undegraded":"2024-03-06T07:21:54.562841+0000","last_fullsized":"2024-03-06T07:21:54.562841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778266+0000","last_change":"2024-03-06T07:21:53.557821+0000","last_active":"2024-03-06T07:21:54.778266+0000","last_peered":"2024-03-06T07:21:54.778266+0000","last_clean":"2024-03-06T07:21:54.778266+0000","last_became_active":"2024-03-06T07:21:53.557536+0000","last_became_peered":"2024-03-06T07:21:53.557536+0000","last_unstale":"2024-03-06T07:21:54.778266+0000","last_undegraded":"2024-03-06T07:21:54.778266+0000","last_fullsized":"2024-03-06T07:21:54.778266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777481+0000","last_change":"2024-03-06T07:21:52.555948+0000","last_active":"2024-03-06T07:21:54.777481+0000","last_peered":"2024-03-06T07:21:54.777481+0000","last_clean":"2024-03-06T07:21:54.777481+0000","last_became_active":"2024-03-06T07:21:52.555657+0000","last_became_peered":"2024-03-06T07:21:52.555657+0000","last_unstale":"2024-03-06T07:21:54.777481+0000","last_undegraded":"2024-03-06T07:21:54.777481+0000","last_fullsized":"2024-03-06T07:21:54.777481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777191+0000","last_change":"2024-03-06T07:21:53.556393+0000","last_active":"2024-03-06T07:21:54.777191+0000","last_peered":"2024-03-06T07:21:54.777191+0000","last_clean":"2024-03-06T07:21:54.777191+0000","last_became_active":"2024-03-06T07:21:53.556137+0000","last_became_peered":"2024-03-06T07:21:53.556137+0000","last_unstale":"2024-03-06T07:21:54.777191+0000","last_undegraded":"2024-03-06T07:21:54.777191+0000","last_fullsized":"2024-03-06T07:21:54.777191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:33.561906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778025+0000","last_change":"2024-03-06T07:21:52.556426+0000","last_active":"2024-03-06T07:21:54.778025+0000","last_peered":"2024-03-06T07:21:54.778025+0000","last_clean":"2024-03-06T07:21:54.778025+0000","last_became_active":"2024-03-06T07:21:52.555844+0000","last_became_peered":"2024-03-06T07:21:52.555844+0000","last_unstale":"2024-03-06T07:21:54.778025+0000","last_undegraded":"2024-03-06T07:21:54.778025+0000","last_fullsized":"2024-03-06T07:21:54.778025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778299+0000","last_change":"2024-03-06T07:21:53.567432+0000","last_active":"2024-03-06T07:21:54.778299+0000","last_peered":"2024-03-06T07:21:54.778299+0000","last_clean":"2024-03-06T07:21:54.778299+0000","last_became_active":"2024-03-06T07:21:53.566788+0000","last_became_peered":"2024-03-06T07:21:53.566788+0000","last_unstale":"2024-03-06T07:21:54.778299+0000","last_undegraded":"2024-03-06T07:21:54.778299+0000","last_fullsized":"2024-03-06T07:21:54.778299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562338+0000","last_change":"2024-03-06T07:21:52.547444+0000","last_active":"2024-03-06T07:21:54.562338+0000","last_peered":"2024-03-06T07:21:54.562338+0000","last_clean":"2024-03-06T07:21:54.562338+0000","last_became_active":"2024-03-06T07:21:52.547163+0000","last_became_peered":"2024-03-06T07:21:52.547163+0000","last_unstale":"2024-03-06T07:21:54.562338+0000","last_undegraded":"2024-03-06T07:21:54.562338+0000","last_fullsized":"2024-03-06T07:21:54.562338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562392+0000","last_change":"2024-03-06T07:21:53.552210+0000","last_active":"2024-03-06T07:21:54.562392+0000","last_peered":"2024-03-06T07:21:54.562392+0000","last_clean":"2024-03-06T07:21:54.562392+0000","last_became_active":"2024-03-06T07:21:53.551529+0000","last_became_peered":"2024-03-06T07:21:53.551529+0000","last_unstale":"2024-03-06T07:21:54.562392+0000","last_undegraded":"2024-03-06T07:21:54.562392+0000","last_fullsized":"2024-03-06T07:21:54.562392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560082+0000","last_change":"2024-03-06T07:21:52.552286+0000","last_active":"2024-03-06T07:21:54.560082+0000","last_peered":"2024-03-06T07:21:54.560082+0000","last_clean":"2024-03-06T07:21:54.560082+0000","last_became_active":"2024-03-06T07:21:52.552010+0000","last_became_peered":"2024-03-06T07:21:52.552010+0000","last_unstale":"2024-03-06T07:21:54.560082+0000","last_undegraded":"2024-03-06T07:21:54.560082+0000","last_fullsized":"2024-03-06T07:21:54.560082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777580+0000","last_change":"2024-03-06T07:21:53.571841+0000","last_active":"2024-03-06T07:21:54.777580+0000","last_peered":"2024-03-06T07:21:54.777580+0000","last_clean":"2024-03-06T07:21:54.777580+0000","last_became_active":"2024-03-06T07:21:53.571350+0000","last_became_peered":"2024-03-06T07:21:53.571350+0000","last_unstale":"2024-03-06T07:21:54.777580+0000","last_undegraded":"2024-03-06T07:21:54.777580+0000","last_fullsized":"2024-03-06T07:21:54.777580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777653+0000","last_change":"2024-03-06T07:21:52.553990+0000","last_active":"2024-03-06T07:21:54.777653+0000","last_peered":"2024-03-06T07:21:54.777653+0000","last_clean":"2024-03-06T07:21:54.777653+0000","last_became_active":"2024-03-06T07:21:52.553693+0000","last_became_peered":"2024-03-06T07:21:52.553693+0000","last_unstale":"2024-03-06T07:21:54.777653+0000","last_undegraded":"2024-03-06T07:21:54.777653+0000","last_fullsized":"2024-03-06T07:21:54.777653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777984+0000","last_change":"2024-03-06T07:21:53.567545+0000","last_active":"2024-03-06T07:21:54.777984+0000","last_peered":"2024-03-06T07:21:54.777984+0000","last_clean":"2024-03-06T07:21:54.777984+0000","last_became_active":"2024-03-06T07:21:53.566157+0000","last_became_peered":"2024-03-06T07:21:53.566157+0000","last_unstale":"2024-03-06T07:21:54.777984+0000","last_undegraded":"2024-03-06T07:21:54.777984+0000","last_fullsized":"2024-03-06T07:21:54.777984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560201+0000","last_change":"2024-03-06T07:21:52.547686+0000","last_active":"2024-03-06T07:21:54.560201+0000","last_peered":"2024-03-06T07:21:54.560201+0000","last_clean":"2024-03-06T07:21:54.560201+0000","last_became_active":"2024-03-06T07:21:52.547436+0000","last_became_peered":"2024-03-06T07:21:52.547436+0000","last_unstale":"2024-03-06T07:21:54.560201+0000","last_undegraded":"2024-03-06T07:21:54.560201+0000","last_fullsized":"2024-03-06T07:21:54.560201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778797+0000","last_change":"2024-03-06T07:21:53.567657+0000","last_active":"2024-03-06T07:21:54.778797+0000","last_peered":"2024-03-06T07:21:54.778797+0000","last_clean":"2024-03-06T07:21:54.778797+0000","last_became_active":"2024-03-06T07:21:53.567134+0000","last_became_peered":"2024-03-06T07:21:53.567134+0000","last_unstale":"2024-03-06T07:21:54.778797+0000","last_undegraded":"2024-03-06T07:21:54.778797+0000","last_fullsized":"2024-03-06T07:21:54.778797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:57.048252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562696+0000","last_change":"2024-03-06T07:21:52.556768+0000","last_active":"2024-03-06T07:21:54.562696+0000","last_peered":"2024-03-06T07:21:54.562696+0000","last_clean":"2024-03-06T07:21:54.562696+0000","last_became_active":"2024-03-06T07:21:52.556507+0000","last_became_peered":"2024-03-06T07:21:52.556507+0000","last_unstale":"2024-03-06T07:21:54.562696+0000","last_undegraded":"2024-03-06T07:21:54.562696+0000","last_fullsized":"2024-03-06T07:21:54.562696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778541+0000","last_change":"2024-03-06T07:21:53.563768+0000","last_active":"2024-03-06T07:21:54.778541+0000","last_peered":"2024-03-06T07:21:54.778541+0000","last_clean":"2024-03-06T07:21:54.778541+0000","last_became_active":"2024-03-06T07:21:53.563490+0000","last_became_peered":"2024-03-06T07:21:53.563490+0000","last_unstale":"2024-03-06T07:21:54.778541+0000","last_undegraded":"2024-03-06T07:21:54.778541+0000","last_fullsized":"2024-03-06T07:21:54.778541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560265+0000","last_change":"2024-03-06T07:21:52.557535+0000","last_active":"2024-03-06T07:21:54.560265+0000","last_peered":"2024-03-06T07:21:54.560265+0000","last_clean":"2024-03-06T07:21:54.560265+0000","last_became_active":"2024-03-06T07:21:52.557275+0000","last_became_peered":"2024-03-06T07:21:52.557275+0000","last_unstale":"2024-03-06T07:21:54.560265+0000","last_undegraded":"2024-03-06T07:21:54.560265+0000","last_fullsized":"2024-03-06T07:21:54.560265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778386+0000","last_change":"2024-03-06T07:21:53.565629+0000","last_active":"2024-03-06T07:21:54.778386+0000","last_peered":"2024-03-06T07:21:54.778386+0000","last_clean":"2024-03-06T07:21:54.778386+0000","last_became_active":"2024-03-06T07:21:53.563789+0000","last_became_peered":"2024-03-06T07:21:53.563789+0000","last_unstale":"2024-03-06T07:21:54.778386+0000","last_undegraded":"2024-03-06T07:21:54.778386+0000","last_fullsized":"2024-03-06T07:21:54.778386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778063+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:54.778063+0000","last_peered":"2024-03-06T07:21:54.778063+0000","last_clean":"2024-03-06T07:21:54.778063+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:54.778063+0000","last_undegraded":"2024-03-06T07:21:54.778063+0000","last_fullsized":"2024-03-06T07:21:54.778063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777283+0000","last_change":"2024-03-06T07:21:52.557611+0000","last_active":"2024-03-06T07:21:54.777283+0000","last_peered":"2024-03-06T07:21:54.777283+0000","last_clean":"2024-03-06T07:21:54.777283+0000","last_became_active":"2024-03-06T07:21:52.556594+0000","last_became_peered":"2024-03-06T07:21:52.556594+0000","last_unstale":"2024-03-06T07:21:54.777283+0000","last_undegraded":"2024-03-06T07:21:54.777283+0000","last_fullsized":"2024-03-06T07:21:54.777283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767209+0000","last_change":"2024-03-06T07:21:53.565607+0000","last_active":"2024-03-06T07:21:54.767209+0000","last_peered":"2024-03-06T07:21:54.767209+0000","last_clean":"2024-03-06T07:21:54.767209+0000","last_became_active":"2024-03-06T07:21:53.565356+0000","last_became_peered":"2024-03-06T07:21:53.565356+0000","last_unstale":"2024-03-06T07:21:54.767209+0000","last_undegraded":"2024-03-06T07:21:54.767209+0000","last_fullsized":"2024-03-06T07:21:54.767209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778021+0000","last_change":"2024-03-06T07:21:52.555927+0000","last_active":"2024-03-06T07:21:54.778021+0000","last_peered":"2024-03-06T07:21:54.778021+0000","last_clean":"2024-03-06T07:21:54.778021+0000","last_became_active":"2024-03-06T07:21:52.555485+0000","last_became_peered":"2024-03-06T07:21:52.555485+0000","last_unstale":"2024-03-06T07:21:54.778021+0000","last_undegraded":"2024-03-06T07:21:54.778021+0000","last_fullsized":"2024-03-06T07:21:54.778021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777097+0000","last_change":"2024-03-06T07:21:53.559225+0000","last_active":"2024-03-06T07:21:54.777097+0000","last_peered":"2024-03-06T07:21:54.777097+0000","last_clean":"2024-03-06T07:21:54.777097+0000","last_became_active":"2024-03-06T07:21:53.558925+0000","last_became_peered":"2024-03-06T07:21:53.558925+0000","last_unstale":"2024-03-06T07:21:54.777097+0000","last_undegraded":"2024-03-06T07:21:54.777097+0000","last_fullsized":"2024-03-06T07:21:54.777097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:54.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560174+0000","last_change":"2024-03-06T07:21:52.546620+0000","last_active":"2024-03-06T07:21:54.560174+0000","last_peered":"2024-03-06T07:21:54.560174+0000","last_clean":"2024-03-06T07:21:54.560174+0000","last_became_active":"2024-03-06T07:21:52.546297+0000","last_became_peered":"2024-03-06T07:21:52.546297+0000","last_unstale":"2024-03-06T07:21:54.560174+0000","last_undegraded":"2024-03-06T07:21:54.560174+0000","last_fullsized":"2024-03-06T07:21:54.560174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562488+0000","last_change":"2024-03-06T07:21:53.553218+0000","last_active":"2024-03-06T07:21:54.562488+0000","last_peered":"2024-03-06T07:21:54.562488+0000","last_clean":"2024-03-06T07:21:54.562488+0000","last_became_active":"2024-03-06T07:21:53.552911+0000","last_became_peered":"2024-03-06T07:21:53.552911+0000","last_unstale":"2024-03-06T07:21:54.562488+0000","last_undegraded":"2024-03-06T07:21:54.562488+0000","last_fullsized":"2024-03-06T07:21:54.562488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778113+0000","last_change":"2024-03-06T07:21:52.556096+0000","last_active":"2024-03-06T07:21:54.778113+0000","last_peered":"2024-03-06T07:21:54.778113+0000","last_clean":"2024-03-06T07:21:54.778113+0000","last_became_active":"2024-03-06T07:21:52.555510+0000","last_became_peered":"2024-03-06T07:21:52.555510+0000","last_unstale":"2024-03-06T07:21:54.778113+0000","last_undegraded":"2024-03-06T07:21:54.778113+0000","last_fullsized":"2024-03-06T07:21:54.778113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778491+0000","last_change":"2024-03-06T07:21:53.561767+0000","last_active":"2024-03-06T07:21:54.778491+0000","last_peered":"2024-03-06T07:21:54.778491+0000","last_clean":"2024-03-06T07:21:54.778491+0000","last_became_active":"2024-03-06T07:21:53.561435+0000","last_became_peered":"2024-03-06T07:21:53.561435+0000","last_unstale":"2024-03-06T07:21:54.778491+0000","last_undegraded":"2024-03-06T07:21:54.778491+0000","last_fullsized":"2024-03-06T07:21:54.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:24.631055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562673+0000","last_change":"2024-03-06T07:21:52.553733+0000","last_active":"2024-03-06T07:21:54.562673+0000","last_peered":"2024-03-06T07:21:54.562673+0000","last_clean":"2024-03-06T07:21:54.562673+0000","last_became_active":"2024-03-06T07:21:52.553436+0000","last_became_peered":"2024-03-06T07:21:52.553436+0000","last_unstale":"2024-03-06T07:21:54.562673+0000","last_undegraded":"2024-03-06T07:21:54.562673+0000","last_fullsized":"2024-03-06T07:21:54.562673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779356+0000","last_change":"2024-03-06T07:21:53.556153+0000","last_active":"2024-03-06T07:21:54.779356+0000","last_peered":"2024-03-06T07:21:54.779356+0000","last_clean":"2024-03-06T07:21:54.779356+0000","last_became_active":"2024-03-06T07:21:53.555896+0000","last_became_peered":"2024-03-06T07:21:53.555896+0000","last_unstale":"2024-03-06T07:21:54.779356+0000","last_undegraded":"2024-03-06T07:21:54.779356+0000","last_fullsized":"2024-03-06T07:21:54.779356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778511+0000","last_change":"2024-03-06T07:21:52.557084+0000","last_active":"2024-03-06T07:21:54.778511+0000","last_peered":"2024-03-06T07:21:54.778511+0000","last_clean":"2024-03-06T07:21:54.778511+0000","last_became_active":"2024-03-06T07:21:52.556688+0000","last_became_peered":"2024-03-06T07:21:52.556688+0000","last_unstale":"2024-03-06T07:21:54.778511+0000","last_undegraded":"2024-03-06T07:21:54.778511+0000","last_fullsized":"2024-03-06T07:21:54.778511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:23.626336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562632+0000","last_change":"2024-03-06T07:21:53.552698+0000","last_active":"2024-03-06T07:21:54.562632+0000","last_peered":"2024-03-06T07:21:54.562632+0000","last_clean":"2024-03-06T07:21:54.562632+0000","last_became_active":"2024-03-06T07:21:53.552434+0000","last_became_peered":"2024-03-06T07:21:53.552434+0000","last_unstale":"2024-03-06T07:21:54.562632+0000","last_undegraded":"2024-03-06T07:21:54.562632+0000","last_fullsized":"2024-03-06T07:21:54.562632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562589+0000","last_change":"2024-03-06T07:21:52.553671+0000","last_active":"2024-03-06T07:21:54.562589+0000","last_peered":"2024-03-06T07:21:54.562589+0000","last_clean":"2024-03-06T07:21:54.562589+0000","last_became_active":"2024-03-06T07:21:52.553394+0000","last_became_peered":"2024-03-06T07:21:52.553394+0000","last_unstale":"2024-03-06T07:21:54.562589+0000","last_undegraded":"2024-03-06T07:21:54.562589+0000","last_fullsized":"2024-03-06T07:21:54.562589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767449+0000","last_change":"2024-03-06T07:21:53.555702+0000","last_active":"2024-03-06T07:21:54.767449+0000","last_peered":"2024-03-06T07:21:54.767449+0000","last_clean":"2024-03-06T07:21:54.767449+0000","last_became_active":"2024-03-06T07:21:53.555436+0000","last_became_peered":"2024-03-06T07:21:53.555436+0000","last_unstale":"2024-03-06T07:21:54.767449+0000","last_undegraded":"2024-03-06T07:21:54.767449+0000","last_fullsized":"2024-03-06T07:21:54.767449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562639+0000","last_change":"2024-03-06T07:21:52.550448+0000","last_active":"2024-03-06T07:21:54.562639+0000","last_peered":"2024-03-06T07:21:54.562639+0000","last_clean":"2024-03-06T07:21:54.562639+0000","last_became_active":"2024-03-06T07:21:52.550179+0000","last_became_peered":"2024-03-06T07:21:52.550179+0000","last_unstale":"2024-03-06T07:21:54.562639+0000","last_undegraded":"2024-03-06T07:21:54.562639+0000","last_fullsized":"2024-03-06T07:21:54.562639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777421+0000","last_change":"2024-03-06T07:21:53.553031+0000","last_active":"2024-03-06T07:21:54.777421+0000","last_peered":"2024-03-06T07:21:54.777421+0000","last_clean":"2024-03-06T07:21:54.777421+0000","last_became_active":"2024-03-06T07:21:53.552759+0000","last_became_peered":"2024-03-06T07:21:53.552759+0000","last_unstale":"2024-03-06T07:21:54.777421+0000","last_undegraded":"2024-03-06T07:21:54.777421+0000","last_fullsized":"2024-03-06T07:21:54.777421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777184+0000","last_change":"2024-03-06T07:21:52.555781+0000","last_active":"2024-03-06T07:21:54.777184+0000","last_peered":"2024-03-06T07:21:54.777184+0000","last_clean":"2024-03-06T07:21:54.777184+0000","last_became_active":"2024-03-06T07:21:52.555469+0000","last_became_peered":"2024-03-06T07:21:52.555469+0000","last_unstale":"2024-03-06T07:21:54.777184+0000","last_undegraded":"2024-03-06T07:21:54.777184+0000","last_fullsized":"2024-03-06T07:21:54.777184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777797+0000","last_change":"2024-03-06T07:21:53.558488+0000","last_active":"2024-03-06T07:21:54.777797+0000","last_peered":"2024-03-06T07:21:54.777797+0000","last_clean":"2024-03-06T07:21:54.777797+0000","last_became_active":"2024-03-06T07:21:53.558199+0000","last_became_peered":"2024-03-06T07:21:53.558199+0000","last_unstale":"2024-03-06T07:21:54.777797+0000","last_undegraded":"2024-03-06T07:21:54.777797+0000","last_fullsized":"2024-03-06T07:21:54.777797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562638+0000","last_change":"2024-03-06T07:21:52.556952+0000","last_active":"2024-03-06T07:21:54.562638+0000","last_peered":"2024-03-06T07:21:54.562638+0000","last_clean":"2024-03-06T07:21:54.562638+0000","last_became_active":"2024-03-06T07:21:52.556629+0000","last_became_peered":"2024-03-06T07:21:52.556629+0000","last_unstale":"2024-03-06T07:21:54.562638+0000","last_undegraded":"2024-03-06T07:21:54.562638+0000","last_fullsized":"2024-03-06T07:21:54.562638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767159+0000","last_change":"2024-03-06T07:21:53.554655+0000","last_active":"2024-03-06T07:21:54.767159+0000","last_peered":"2024-03-06T07:21:54.767159+0000","last_clean":"2024-03-06T07:21:54.767159+0000","last_became_active":"2024-03-06T07:21:53.554321+0000","last_became_peered":"2024-03-06T07:21:53.554321+0000","last_unstale":"2024-03-06T07:21:54.767159+0000","last_undegraded":"2024-03-06T07:21:54.767159+0000","last_fullsized":"2024-03-06T07:21:54.767159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562564+0000","last_change":"2024-03-06T07:21:52.547548+0000","last_active":"2024-03-06T07:21:54.562564+0000","last_peered":"2024-03-06T07:21:54.562564+0000","last_clean":"2024-03-06T07:21:54.562564+0000","last_became_active":"2024-03-06T07:21:52.547285+0000","last_became_peered":"2024-03-06T07:21:52.547285+0000","last_unstale":"2024-03-06T07:21:54.562564+0000","last_undegraded":"2024-03-06T07:21:54.562564+0000","last_fullsized":"2024-03-06T07:21:54.562564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767314+0000","last_change":"2024-03-06T07:21:53.557506+0000","last_active":"2024-03-06T07:21:54.767314+0000","last_peered":"2024-03-06T07:21:54.767314+0000","last_clean":"2024-03-06T07:21:54.767314+0000","last_became_active":"2024-03-06T07:21:53.556568+0000","last_became_peered":"2024-03-06T07:21:53.556568+0000","last_unstale":"2024-03-06T07:21:54.767314+0000","last_undegraded":"2024-03-06T07:21:54.767314+0000","last_fullsized":"2024-03-06T07:21:54.767314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778329+0000","last_change":"2024-03-06T07:21:52.558820+0000","last_active":"2024-03-06T07:21:54.778329+0000","last_peered":"2024-03-06T07:21:54.778329+0000","last_clean":"2024-03-06T07:21:54.778329+0000","last_became_active":"2024-03-06T07:21:52.558486+0000","last_became_peered":"2024-03-06T07:21:52.558486+0000","last_unstale":"2024-03-06T07:21:54.778329+0000","last_undegraded":"2024-03-06T07:21:54.778329+0000","last_fullsized":"2024-03-06T07:21:54.778329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562402+0000","last_change":"2024-03-06T07:21:53.558230+0000","last_active":"2024-03-06T07:21:54.562402+0000","last_peered":"2024-03-06T07:21:54.562402+0000","last_clean":"2024-03-06T07:21:54.562402+0000","last_became_active":"2024-03-06T07:21:53.557916+0000","last_became_peered":"2024-03-06T07:21:53.557916+0000","last_unstale":"2024-03-06T07:21:54.562402+0000","last_undegraded":"2024-03-06T07:21:54.562402+0000","last_fullsized":"2024-03-06T07:21:54.562402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779232+0000","last_change":"2024-03-06T07:21:53.567503+0000","last_active":"2024-03-06T07:21:54.779232+0000","last_peered":"2024-03-06T07:21:54.779232+0000","last_clean":"2024-03-06T07:21:54.779232+0000","last_became_active":"2024-03-06T07:21:53.567223+0000","last_became_peered":"2024-03-06T07:21:53.567223+0000","last_unstale":"2024-03-06T07:21:54.779232+0000","last_undegraded":"2024-03-06T07:21:54.779232+0000","last_fullsized":"2024-03-06T07:21:54.779232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777697+0000","last_change":"2024-03-06T07:21:52.559682+0000","last_active":"2024-03-06T07:21:54.777697+0000","last_peered":"2024-03-06T07:21:54.777697+0000","last_clean":"2024-03-06T07:21:54.777697+0000","last_became_active":"2024-03-06T07:21:52.559430+0000","last_became_peered":"2024-03-06T07:21:52.559430+0000","last_unstale":"2024-03-06T07:21:54.777697+0000","last_undegraded":"2024-03-06T07:21:54.777697+0000","last_fullsized":"2024-03-06T07:21:54.777697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778415+0000","last_change":"2024-03-06T07:21:53.568862+0000","last_active":"2024-03-06T07:21:54.778415+0000","last_peered":"2024-03-06T07:21:54.778415+0000","last_clean":"2024-03-06T07:21:54.778415+0000","last_became_active":"2024-03-06T07:21:53.568127+0000","last_became_peered":"2024-03-06T07:21:53.568127+0000","last_unstale":"2024-03-06T07:21:54.778415+0000","last_undegraded":"2024-03-06T07:21:54.778415+0000","last_fullsized":"2024-03-06T07:21:54.778415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778429+0000","last_change":"2024-03-06T07:21:52.558983+0000","last_active":"2024-03-06T07:21:54.778429+0000","last_peered":"2024-03-06T07:21:54.778429+0000","last_clean":"2024-03-06T07:21:54.778429+0000","last_became_active":"2024-03-06T07:21:52.558616+0000","last_became_peered":"2024-03-06T07:21:52.558616+0000","last_unstale":"2024-03-06T07:21:54.778429+0000","last_undegraded":"2024-03-06T07:21:54.778429+0000","last_fullsized":"2024-03-06T07:21:54.778429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777374+0000","last_change":"2024-03-06T07:21:53.558956+0000","last_active":"2024-03-06T07:21:54.777374+0000","last_peered":"2024-03-06T07:21:54.777374+0000","last_clean":"2024-03-06T07:21:54.777374+0000","last_became_active":"2024-03-06T07:21:53.558612+0000","last_became_peered":"2024-03-06T07:21:53.558612+0000","last_unstale":"2024-03-06T07:21:54.777374+0000","last_undegraded":"2024-03-06T07:21:54.777374+0000","last_fullsized":"2024-03-06T07:21:54.777374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767442+0000","last_change":"2024-03-06T07:21:52.553080+0000","last_active":"2024-03-06T07:21:54.767442+0000","last_peered":"2024-03-06T07:21:54.767442+0000","last_clean":"2024-03-06T07:21:54.767442+0000","last_became_active":"2024-03-06T07:21:52.552785+0000","last_became_peered":"2024-03-06T07:21:52.552785+0000","last_unstale":"2024-03-06T07:21:54.767442+0000","last_undegraded":"2024-03-06T07:21:54.767442+0000","last_fullsized":"2024-03-06T07:21:54.767442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767397+0000","last_change":"2024-03-06T07:21:53.563091+0000","last_active":"2024-03-06T07:21:54.767397+0000","last_peered":"2024-03-06T07:21:54.767397+0000","last_clean":"2024-03-06T07:21:54.767397+0000","last_became_active":"2024-03-06T07:21:53.562797+0000","last_became_peered":"2024-03-06T07:21:53.562797+0000","last_unstale":"2024-03-06T07:21:54.767397+0000","last_undegraded":"2024-03-06T07:21:54.767397+0000","last_fullsized":"2024-03-06T07:21:54.767397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777504+0000","last_change":"2024-03-06T07:21:52.557097+0000","last_active":"2024-03-06T07:21:54.777504+0000","last_peered":"2024-03-06T07:21:54.777504+0000","last_clean":"2024-03-06T07:21:54.777504+0000","last_became_active":"2024-03-06T07:21:52.556808+0000","last_became_peered":"2024-03-06T07:21:52.556808+0000","last_unstale":"2024-03-06T07:21:54.777504+0000","last_undegraded":"2024-03-06T07:21:54.777504+0000","last_fullsized":"2024-03-06T07:21:54.777504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:01.581 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:01.582 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:22:01.582 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:22:01.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:22:01.582 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:22:01.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.714+0000 7f0b8aa15640 1 Processor -- start 2024-03-06T07:22:01.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.714+0000 7f0b8aa15640 1 -- start start 2024-03-06T07:22:01.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b8aa15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 0x7f0b840b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b8aa15640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 0x7f0b840cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b8aa15640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b840d25d0 con 0x7f0b840b4170 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b8aa15640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840d4bf0 con 0x7f0b840d0a80 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b8aa15640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840ce9e0 con 0x7f0b840d4760 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b837fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 0x7f0b840b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b88fb2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 0x7f0b840cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b88fb2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 0x7f0b840cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45836/0 (socket says 172.21.15.122:45836) 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b837fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 0x7f0b840b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44438/0 (socket says 172.21.15.122:44438) 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b88fb2640 1 -- 172.21.15.122:0/1481603964 learned_addr learned my addr 172.21.15.122:0/1481603964 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.715+0000 7f0b83fff640 1 --1- 172.21.15.122:0/1481603964 >> v1:172.21.15.169:6789/0 conn(0x7f0b840b4170 0x7f0b840d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:33938/0 (socket says 172.21.15.122:33938) 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.716+0000 7f0b837fe640 1 -- 172.21.15.122:0/1481603964 >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 msgr2=0x7f0b840cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.716+0000 7f0b837fe640 1 --2- 172.21.15.122:0/1481603964 >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 0x7f0b840cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.716+0000 7f0b837fe640 1 -- 172.21.15.122:0/1481603964 >> v1:172.21.15.169:6789/0 conn(0x7f0b840b4170 legacy=0x7f0b840d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.716+0000 7f0b837fe640 1 -- 172.21.15.122:0/1481603964 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b840d2870 con 0x7f0b840d4760 2024-03-06T07:22:01.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.716+0000 7f0b88fb2640 1 --2- 172.21.15.122:0/1481603964 >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 0x7f0b840cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:01.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.716+0000 7f0b837fe640 1 --2- 172.21.15.122:0/1481603964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 0x7f0b840b6a90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0b780097a0 tx=0x7f0b7802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf2cfa4d473e034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b82ffd640 1 -- 172.21.15.122:0/1481603964 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78035020 con 0x7f0b840d4760 2024-03-06T07:22:01.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b82ffd640 1 -- 172.21.15.122:0/1481603964 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b78031830 con 0x7f0b840d4760 2024-03-06T07:22:01.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b82ffd640 1 -- 172.21.15.122:0/1481603964 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78031c00 con 0x7f0b840d4760 2024-03-06T07:22:01.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 -- 172.21.15.122:0/1481603964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 msgr2=0x7f0b840b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/1481603964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 0x7f0b840b6a90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0b780097a0 tx=0x7f0b7802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 -- 172.21.15.122:0/1481603964 shutdown_connections 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/1481603964 >> v2:172.21.15.169:3300/0 conn(0x7f0b840d0a80 0x7f0b840cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/1481603964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840d4760 0x7f0b840b6a90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 -- 172.21.15.122:0/1481603964 >> 172.21.15.122:0/1481603964 conn(0x7f0b840d3c40 msgr2=0x7f0b840d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 -- 172.21.15.122:0/1481603964 shutdown_connections 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.717+0000 7f0b8aa15640 1 -- 172.21.15.122:0/1481603964 wait complete. 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.718+0000 7f0b8aa15640 1 Processor -- start 2024-03-06T07:22:01.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 -- start start 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840b4170 0x7f0b8415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b840d0a80 0x7f0b840c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b840c85f0 0x7f0b840c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840b4dd0 con 0x7f0b840d0a80 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840b4550 con 0x7f0b840c85f0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b8aa15640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840b4f50 con 0x7f0b840b4170 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b83fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840b4170 0x7f0b8415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b837fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b840d0a80 0x7f0b840c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b83fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840b4170 0x7f0b8415a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44452/0 (socket says 172.21.15.122:44452) 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b83fff640 1 -- 172.21.15.122:0/551175974 learned_addr learned my addr 172.21.15.122:0/551175974 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.719+0000 7f0b88fb2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b840c85f0 0x7f0b840c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b837fe640 1 -- 172.21.15.122:0/551175974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840b4170 msgr2=0x7f0b8415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b837fe640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840b4170 0x7f0b8415a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b837fe640 1 -- 172.21.15.122:0/551175974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b840c85f0 msgr2=0x7f0b840c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b837fe640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b840c85f0 0x7f0b840c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b837fe640 1 -- 172.21.15.122:0/551175974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b840cc980 con 0x7f0b840d0a80 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b837fe640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b840d0a80 0x7f0b840c80b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b78033000 tx=0x7f0b780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78040070 con 0x7f0b840d0a80 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7803a930 con 0x7f0b840d0a80 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b840ccc00 con 0x7f0b840d0a80 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78035040 con 0x7f0b840d0a80 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.720+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b840cd1c0 con 0x7f0b840d0a80 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.721+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8404f330 con 0x7f0b840d0a80 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.721+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b7804b060 con 0x7f0b840d0a80 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.722+0000 7f0b80ff9640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b7006b4d0 0x7f0b7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.722+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b780ab9f0 con 0x7f0b840d0a80 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.722+0000 7f0b83fff640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b7006b4d0 0x7f0b7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.722+0000 7f0b83fff640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b7006b4d0 0x7f0b7006d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b74004de0 tx=0x7f0b74007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.726+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b840cc980 con 0x7f0b840d0a80 2024-03-06T07:22:01.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.942+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0b840733b0 con 0x7f0b840d0a80 2024-03-06T07:22:01.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.945+0000 7f0b80ff9640 1 -- 172.21.15.122:0/551175974 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171788 (secure 0 0 0) 0x7f0b78031800 con 0x7f0b840d0a80 2024-03-06T07:22:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b7006b4d0 msgr2=0x7f0b7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b7006b4d0 0x7f0b7006d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b74004de0 tx=0x7f0b74007f10 comp rx=0 tx=0).stop 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b840d0a80 msgr2=0x7f0b840c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b840d0a80 0x7f0b840c80b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b78033000 tx=0x7f0b780046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 shutdown_connections 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b7006b4d0 0x7f0b7006d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b840c85f0 0x7f0b840c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b840d0a80 0x7f0b840c80b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 --2- 172.21.15.122:0/551175974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b840b4170 0x7f0b8415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 >> 172.21.15.122:0/551175974 conn(0x7f0b840d3c40 msgr2=0x7f0b840b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 shutdown_connections 2024-03-06T07:22:01.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:01.948+0000 7f0b8aa15640 1 -- 172.21.15.122:0/551175974 wait complete. 2024-03-06T07:22:01.960 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":602544710},{"type":"v1","addr":"172.21.15.122:6833","nonce":602544710}]},"active_addr":"172.21.15.122:6833/602544710","active_change":"2024-03-06T07:21:33.319960+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":2039629980}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":2862808813}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":3607463139}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":729351803}]}]} 2024-03-06T07:22:01.964 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:22:01.964 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:22:01.964 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.061+0000 7f91a5793640 1 Processor -- start 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.061+0000 7f91a5793640 1 -- start start 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f91a5793640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f91a015dfe0 0x7f91a00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f91a5793640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f91a00d29c0 0x7f91a00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f91a5793640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00b3430 con 0x7f91a00cf830 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f91a5793640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00b31e0 con 0x7f91a015dfe0 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f91a5793640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00b3d60 con 0x7f91a00d29c0 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f91a015dfe0 0x7f91a00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f91a015dfe0 0x7f91a00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54514/0 (socket says 172.21.15.122:54514) 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 -- 172.21.15.122:0/2861790941 learned_addr learned my addr 172.21.15.122:0/2861790941 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919f7fe640 1 --2- 172.21.15.122:0/2861790941 >> v2:172.21.15.169:3300/0 conn(0x7f91a00d29c0 0x7f91a00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 -- 172.21.15.122:0/2861790941 >> v2:172.21.15.169:3300/0 conn(0x7f91a00d29c0 msgr2=0x7f91a00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 --2- 172.21.15.122:0/2861790941 >> v2:172.21.15.169:3300/0 conn(0x7f91a00d29c0 0x7f91a00b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 -- 172.21.15.122:0/2861790941 >> v1:172.21.15.169:6789/0 conn(0x7f91a00cf830 legacy=0x7f91a00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.062+0000 7f919effd640 1 -- 172.21.15.122:0/2861790941 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a00b3710 con 0x7f91a015dfe0 2024-03-06T07:22:02.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f919effd640 1 --2- 172.21.15.122:0/2861790941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a015dfe0 0x7f91a00cf2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91a00b3cd0 tx=0x7f919802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20bea19403ca15ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f919dffb640 1 -- 172.21.15.122:0/2861790941 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198034070 con 0x7f91a015dfe0 2024-03-06T07:22:02.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f919dffb640 1 -- 172.21.15.122:0/2861790941 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198030860 con 0x7f91a015dfe0 2024-03-06T07:22:02.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f919dffb640 1 -- 172.21.15.122:0/2861790941 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198030c30 con 0x7f91a015dfe0 2024-03-06T07:22:02.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 -- 172.21.15.122:0/2861790941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a015dfe0 msgr2=0x7f91a00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 --2- 172.21.15.122:0/2861790941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a015dfe0 0x7f91a00cf2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91a00b3cd0 tx=0x7f919802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 -- 172.21.15.122:0/2861790941 shutdown_connections 2024-03-06T07:22:02.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 --2- 172.21.15.122:0/2861790941 >> v2:172.21.15.169:3300/0 conn(0x7f91a00d29c0 0x7f91a00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 --2- 172.21.15.122:0/2861790941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a015dfe0 0x7f91a00cf2f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 -- 172.21.15.122:0/2861790941 >> 172.21.15.122:0/2861790941 conn(0x7f91a00d3cd0 msgr2=0x7f91a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 -- 172.21.15.122:0/2861790941 shutdown_connections 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.063+0000 7f91a5793640 1 -- 172.21.15.122:0/2861790941 wait complete. 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 Processor -- start 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 -- start start 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 0x7f91a00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91a00d29c0 0x7f91a00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a015dfe0 0x7f91a00c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00d1df0 con 0x7f91a00cf830 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00d1c70 con 0x7f91a015dfe0 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.065+0000 7f91a5793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a015e3c0 con 0x7f91a00d29c0 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 0x7f91a00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 0x7f91a00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54522/0 (socket says 172.21.15.122:54522) 2024-03-06T07:22:02.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 -- 172.21.15.122:0/3640097518 learned_addr learned my addr 172.21.15.122:0/3640097518 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919e7fc640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91a00d29c0 0x7f91a00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919f7fe640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a015dfe0 0x7f91a00c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 -- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91a00d29c0 msgr2=0x7f91a00b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91a00d29c0 0x7f91a00b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 -- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a015dfe0 msgr2=0x7f91a00c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a015dfe0 0x7f91a00c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 -- 172.21.15.122:0/3640097518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a015fef0 con 0x7f91a00cf830 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f919effd640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 0x7f91a00cada0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f919802b9e0 tx=0x7f91980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198034040 con 0x7f91a00cf830 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198039d90 con 0x7f91a00cf830 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198043610 con 0x7f91a00cf830 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a00cdcf0 con 0x7f91a00cf830 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.066+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a0160190 con 0x7f91a00cf830 2024-03-06T07:22:02.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.067+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a004f330 con 0x7f91a00cf830 2024-03-06T07:22:02.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.067+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f919804a060 con 0x7f91a00cf830 2024-03-06T07:22:02.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.068+0000 7f91877fe640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918c06b4d0 0x7f918c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.068+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91980ac110 con 0x7f91a00cf830 2024-03-06T07:22:02.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.068+0000 7f919e7fc640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918c06b4d0 0x7f918c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.068+0000 7f919e7fc640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918c06b4d0 0x7f918c06d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f91a00b5a30 tx=0x7f9194009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.072+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f919803e070 con 0x7f91a00cf830 2024-03-06T07:22:02.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.244+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91a006a1b0 con 0x7f91a00cf830 2024-03-06T07:22:02.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.245+0000 7f91877fe640 1 -- 172.21.15.122:0/3640097518 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f919807a940 con 0x7f91a00cf830 2024-03-06T07:22:02.248 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:02.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918c06b4d0 msgr2=0x7f918c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918c06b4d0 0x7f918c06d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f91a00b5a30 tx=0x7f9194009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 msgr2=0x7f91a00cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 0x7f91a00cada0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f919802b9e0 tx=0x7f91980046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 shutdown_connections 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918c06b4d0 0x7f918c06d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91a015dfe0 0x7f91a00c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91a00d29c0 0x7f91a00b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 --2- 172.21.15.122:0/3640097518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91a00cf830 0x7f91a00cada0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 >> 172.21.15.122:0/3640097518 conn(0x7f91a00d3cd0 msgr2=0x7f91a00cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 shutdown_connections 2024-03-06T07:22:02.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.247+0000 7f91a5793640 1 -- 172.21.15.122:0/3640097518 wait complete. 2024-03-06T07:22:02.260 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:22:02.261 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 Processor -- start 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 -- start start 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7d500d00c0 0x7f7d500d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7d500cf850 0x7f7d500cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d500d09e0 con 0x7f7d500ce1b0 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5006ca90 con 0x7f7d500cf850 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.359+0000 7f7d550cf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5006df30 con 0x7f7d500d00c0 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7d500cf850 0x7f7d500cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4ed9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7d500d00c0 0x7f7d500d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7d500cf850 0x7f7d500cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54524/0 (socket says 172.21.15.122:54524) 2024-03-06T07:22:02.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 -- 172.21.15.122:0/3592307648 learned_addr learned my addr 172.21.15.122:0/3592307648 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 -- 172.21.15.122:0/3592307648 >> v2:172.21.15.169:3300/0 conn(0x7f7d500d00c0 msgr2=0x7f7d500d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 --2- 172.21.15.122:0/3592307648 >> v2:172.21.15.169:3300/0 conn(0x7f7d500d00c0 0x7f7d500d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 -- 172.21.15.122:0/3592307648 >> v1:172.21.15.169:6789/0 conn(0x7f7d500ce1b0 legacy=0x7f7d500ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 -- 172.21.15.122:0/3592307648 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5015e380 con 0x7f7d500cf850 2024-03-06T07:22:02.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4ed9d640 1 --2- 172.21.15.122:0/3592307648 >> v2:172.21.15.169:3300/0 conn(0x7f7d500d00c0 0x7f7d500d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.360+0000 7f7d4e59c640 1 --2- 172.21.15.122:0/3592307648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500cf850 0x7f7d500cd3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7d500d1a40 tx=0x7f7d4402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb396ff622e8a00e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d4dd9b640 1 -- 172.21.15.122:0/3592307648 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4402cb10 con 0x7f7d500cf850 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d4dd9b640 1 -- 172.21.15.122:0/3592307648 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4402ccb0 con 0x7f7d500cf850 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d4dd9b640 1 -- 172.21.15.122:0/3592307648 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4402ce60 con 0x7f7d500cf850 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 -- 172.21.15.122:0/3592307648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500cf850 msgr2=0x7f7d500cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 --2- 172.21.15.122:0/3592307648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500cf850 0x7f7d500cd3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7d500d1a40 tx=0x7f7d4402bf70 comp rx=0 tx=0).stop 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 -- 172.21.15.122:0/3592307648 shutdown_connections 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 --2- 172.21.15.122:0/3592307648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500cf850 0x7f7d500cd3c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 --2- 172.21.15.122:0/3592307648 >> v2:172.21.15.169:3300/0 conn(0x7f7d500d00c0 0x7f7d500d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 -- 172.21.15.122:0/3592307648 >> 172.21.15.122:0/3592307648 conn(0x7f7d500b35a0 msgr2=0x7f7d500b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 -- 172.21.15.122:0/3592307648 shutdown_connections 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.361+0000 7f7d550cf640 1 -- 172.21.15.122:0/3592307648 wait complete. 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 Processor -- start 2024-03-06T07:22:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 -- start start 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 0x7f7d5014b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d500cf850 0x7f7d500c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7d500d00c0 0x7f7d500c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5006c440 con 0x7f7d500ce1b0 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5006c160 con 0x7f7d500cf850 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d550cf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5006c720 con 0x7f7d500d00c0 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d4ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 0x7f7d5014b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d4ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 0x7f7d5014b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54534/0 (socket says 172.21.15.122:54534) 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d4ed9d640 1 -- 172.21.15.122:0/600246092 learned_addr learned my addr 172.21.15.122:0/600246092 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d4e59c640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d500cf850 0x7f7d500c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.363+0000 7f7d4f59e640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7d500d00c0 0x7f7d500c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d4ed9d640 1 -- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7d500d00c0 msgr2=0x7f7d500c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d4ed9d640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7d500d00c0 0x7f7d500c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d4ed9d640 1 -- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d500cf850 msgr2=0x7f7d500c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d4ed9d640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d500cf850 0x7f7d500c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d4ed9d640 1 -- 172.21.15.122:0/600246092 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5014bd40 con 0x7f7d500ce1b0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d4ed9d640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 0x7f7d5014b4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800e960 tx=0x7f7d4800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48017070 con 0x7f7d500ce1b0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4800ca00 con 0x7f7d500ce1b0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5014c020 con 0x7f7d500ce1b0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4800cdd0 con 0x7f7d500ce1b0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.364+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5014c5e0 con 0x7f7d500ce1b0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.365+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5004f330 con 0x7f7d500ce1b0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.365+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d4801a070 con 0x7f7d500ce1b0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.366+0000 7f7d377fe640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7d3c06b500 0x7f7d3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.366+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d4808a040 con 0x7f7d500ce1b0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.366+0000 7f7d4e59c640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7d3c06b500 0x7f7d3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.366+0000 7f7d4e59c640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7d3c06b500 0x7f7d3c06d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d44035000 tx=0x7f7d44030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.370+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d48058870 con 0x7f7d500ce1b0 2024-03-06T07:22:02.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.544+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d501617f0 con 0x7f7d500ce1b0 2024-03-06T07:22:02.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.545+0000 7f7d377fe640 1 -- 172.21.15.122:0/600246092 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f7d4805c910 con 0x7f7d500ce1b0 2024-03-06T07:22:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:02.548 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7d3c06b500 msgr2=0x7f7d3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7d3c06b500 0x7f7d3c06d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d44035000 tx=0x7f7d44030000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 msgr2=0x7f7d5014b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 0x7f7d5014b4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800e960 tx=0x7f7d4800ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 shutdown_connections 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7d3c06b500 0x7f7d3c06d9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7d500d00c0 0x7f7d500c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d500cf850 0x7f7d500c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 --2- 172.21.15.122:0/600246092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d500ce1b0 0x7f7d5014b4f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 >> 172.21.15.122:0/600246092 conn(0x7f7d500b35a0 msgr2=0x7f7d500cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 shutdown_connections 2024-03-06T07:22:02.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.547+0000 7f7d550cf640 1 -- 172.21.15.122:0/600246092 wait complete. 2024-03-06T07:22:02.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:22:02.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:22:02.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:22:02.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:22:02.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:22:02.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:22:02.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:22:02.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.730+0000 7fd2e32fa640 1 Processor -- start 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.730+0000 7f06bbced640 1 Processor -- start 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.730+0000 7fd2e32fa640 1 -- start start 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.731+0000 7fef8e002640 1 Processor -- start 2024-03-06T07:22:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.731+0000 7fef8e002640 1 -- start start 2024-03-06T07:22:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.730+0000 7f06bbced640 1 -- start start 2024-03-06T07:22:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06bbced640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 0x7f06b40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06bbced640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06b40c6850 0x7f06b406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f043a8c9640 1 Processor -- start 2024-03-06T07:22:02.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e32fa640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd2d40e02d0 0x7fd2d40e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e32fa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d40e4510 con 0x7fd2d40dfef0 2024-03-06T07:22:02.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e2af9640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd2d40e4100 0x7fd2d40539f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:33962/0 (socket says 172.21.15.122:33962) 2024-03-06T07:22:02.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e1af7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd2d40e02d0 0x7fd2d40e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06b9a89640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 0x7f06b40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06b9a89640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 0x7f06b40ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45874/0 (socket says 172.21.15.122:45874) 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06b9288640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06b40c6850 0x7f06b406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06b9a89640 1 -- 172.21.15.122:0/196293470 learned_addr learned my addr 172.21.15.122:0/196293470 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b406b910 con 0x7f06b406b4f0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b40caec0 con 0x7f06b40c6850 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b40cb480 con 0x7f06b40affa0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e1af7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd2d40e02d0 0x7fd2d40e3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45878/0 (socket says 172.21.15.122:45878) 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e2af9640 1 -- 172.21.15.122:0/3590534769 learned_addr learned my addr 172.21.15.122:0/3590534769 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fef8e002640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef881630a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8e002640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fef88070cc0 0x7fef880b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.732+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d4056590 con 0x7fd2d40e4100 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d40094b0 con 0x7fd2d40e02d0 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef881630a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef881630a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44476/0 (socket says 172.21.15.122:44476) 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547376252 0 0) 0x7fd2d4056590 con 0x7fd2d40e4100 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 -- 172.21.15.122:0/3433124471 learned_addr learned my addr 172.21.15.122:0/3433124471 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef87fff640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.169:3300/0 conn(0x7fef88070cc0 0x7fef880b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c8003520 con 0x7fd2d40e4100 2024-03-06T07:22:02.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8e002640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.122:3300/0 conn(0x7fef880b43e0 0x7fef88165c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285396083 0 0) 0x7fd2cc001040 con 0x7fd2d40dfef0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f043a8c9640 1 -- start start 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f646f59e640 1 Processor -- start 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d801640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.122:3300/0 conn(0x7fef880b43e0 0x7fef88165c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef880711a0 con 0x7fef880b43e0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef88071480 con 0x7fef88070cc0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef88071760 con 0x7fef880cf640 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f646f59e640 1 -- start start 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4056590 con 0x7fd2d40dfef0 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 -- 172.21.15.122:0/3433124471 >> v2:172.21.15.169:3300/0 conn(0x7fef88070cc0 msgr2=0x7fef880b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.169:3300/0 conn(0x7fef88070cc0 0x7fef880b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 -- 172.21.15.122:0/3433124471 >> v2:172.21.15.122:3300/0 conn(0x7fef880b43e0 msgr2=0x7fef88165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.122:3300/0 conn(0x7fef880b43e0 0x7fef88165c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7fef8d000640 1 -- 172.21.15.122:0/3433124471 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef88071a40 con 0x7fef880cf640 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fef8d801640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.122:3300/0 conn(0x7fef880b43e0 0x7fef88165c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4109112277 0 0) 0x7fd2c8003520 con 0x7fd2d40e4100 2024-03-06T07:22:02.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f043a8c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 0x7f04340b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f043a8c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f043415a3f0 0x7f043415a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f043a8c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04340b37d0 con 0x7f04340b2090 2024-03-06T07:22:02.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f043a8c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04340b3340 con 0x7f043415a3f0 2024-03-06T07:22:02.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f043a8c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04340b43c0 con 0x7f04340b3a20 2024-03-06T07:22:02.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 >> v2:172.21.15.169:3300/0 conn(0x7fd2d40e02d0 msgr2=0x7fd2d40e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f646f59e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6470165580 0x7f6470162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f04337fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 0x7f04340b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f04337fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 0x7f04340b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44490/0 (socket says 172.21.15.122:44490) 2024-03-06T07:22:02.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f04337fe640 1 -- 172.21.15.122:0/1411505273 learned_addr learned my addr 172.21.15.122:0/1411505273 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fd2e12f6640 1 --2- 172.21.15.122:0/3590534769 >> v2:172.21.15.169:3300/0 conn(0x7fd2d40e02d0 0x7fd2d40e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 >> v1:172.21.15.122:6789/0 conn(0x7fd2d40dfef0 legacy=0x7fd2d40085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f646f59e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6470162c80 0x7f647006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fa27941f640 1 Processor -- start 2024-03-06T07:22:02.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7fa27941f640 1 -- start start 2024-03-06T07:22:02.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.734+0000 7f0438e66640 1 --2- 172.21.15.122:0/1411505273 >> v2:172.21.15.122:3300/0 conn(0x7f043415a3f0 0x7f043415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d4056870 con 0x7fd2d40e4100 2024-03-06T07:22:02.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646e59c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6470165580 0x7f6470162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646e59c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6470165580 0x7f6470162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45900/0 (socket says 172.21.15.122:45900) 2024-03-06T07:22:02.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646e59c640 1 -- 172.21.15.122:0/692411644 learned_addr learned my addr 172.21.15.122:0/692411644 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646f59e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64701631e0 con 0x7f647006ecd0 2024-03-06T07:22:02.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647006c960 con 0x7f6470165580 2024-03-06T07:22:02.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f0432ffd640 1 -- 172.21.15.122:0/1411505273 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560187342 0 0) 0x7f04340b37d0 con 0x7f04340b2090 2024-03-06T07:22:02.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64701634c0 con 0x7f6470162c80 2024-03-06T07:22:02.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f0432ffd640 1 -- 172.21.15.122:0/1411505273 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0424003520 con 0x7f04340b2090 2024-03-06T07:22:02.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646dd9b640 1 --2- 172.21.15.122:0/692411644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6470162c80 0x7f647006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7ff7a6204640 1 Processor -- start 2024-03-06T07:22:02.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7ff7a6204640 1 -- start start 2024-03-06T07:22:02.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fef8d000640 1 --2- 172.21.15.122:0/3433124471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef881630a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fef880b22e0 tx=0x7fef7c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8cd36f12e72e52f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f04337fe640 1 -- 172.21.15.122:0/1411505273 >> v2:172.21.15.122:3300/0 conn(0x7f043415a3f0 msgr2=0x7f043415a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f04337fe640 1 --2- 172.21.15.122:0/1411505273 >> v2:172.21.15.122:3300/0 conn(0x7f043415a3f0 0x7f043415a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f04337fe640 1 -- 172.21.15.122:0/1411505273 >> v1:172.21.15.169:6789/0 conn(0x7f04340b2090 legacy=0x7f04340cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f04337fe640 1 -- 172.21.15.122:0/1411505273 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04340b2a10 con 0x7f04340b3a20 2024-03-06T07:22:02.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/3590534769 >> v2:172.21.15.169:3300/0 conn(0x7fd2d40e02d0 0x7fd2d40e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fd2dc04f850 con 0x7fd2d40e4100 2024-03-06T07:22:02.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 >> v1:172.21.15.169:6789/0 conn(0x7fd2d40e4100 legacy=0x7fd2d40539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fd2e12f6640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2c8003b40 0x7fd2c8003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f06b9288640 1 -- 172.21.15.122:0/196293470 >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 msgr2=0x7f06b40ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e12f6640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2c8004800 0x7fd2c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f06b9288640 1 --2- 172.21.15.122:0/196293470 >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 0x7f06b40ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e12f6640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c80051c0 0x7fd2c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f06b9288640 1 -- 172.21.15.122:0/196293470 >> v1:172.21.15.169:6789/0 conn(0x7f06b406b4f0 legacy=0x7f06b4071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f06b9288640 1 -- 172.21.15.122:0/196293470 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40c6cb0 con 0x7f06b40c6850 2024-03-06T07:22:02.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2c8004800 0x7fd2c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40094b0 con 0x7fd2c8003b40 2024-03-06T07:22:02.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.733+0000 7f06b9a89640 1 --2- 172.21.15.122:0/196293470 >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 0x7f06b40ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646dd9b640 1 -- 172.21.15.122:0/692411644 >> v2:172.21.15.169:3300/0 conn(0x7f6470165580 msgr2=0x7f6470162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646dd9b640 1 --2- 172.21.15.122:0/692411644 >> v2:172.21.15.169:3300/0 conn(0x7f6470165580 0x7f6470162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c800bf30 con 0x7fd2c80051c0 2024-03-06T07:22:02.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c800c130 con 0x7fd2c8004800 2024-03-06T07:22:02.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e22f8640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2c8003b40 0x7fd2c8003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646dd9b640 1 -- 172.21.15.122:0/692411644 >> v1:172.21.15.169:6789/0 conn(0x7f647006ecd0 legacy=0x7f64700d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7f646dd9b640 1 -- 172.21.15.122:0/692411644 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647006b150 con 0x7f6470162c80 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f06b9288640 1 --2- 172.21.15.122:0/196293470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 0x7f06b406afb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f06b4067c30 tx=0x7f06b002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb5438e61f83650 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f06b8a87640 1 -- 172.21.15.122:0/196293470 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0038070 con 0x7f06b40c6850 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff7a6204640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff7980e6e20 0x7ff7980e7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff7a6204640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff798057b00 0x7ff798057f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff7a6204640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff798007340 con 0x7ff7980e33f0 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff7a6204640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7980e77c0 con 0x7ff798057b00 2024-03-06T07:22:02.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff7a6204640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff798007620 con 0x7ff7980e6e20 2024-03-06T07:22:02.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f04337fe640 1 --2- 172.21.15.122:0/1411505273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 0x7f04340b3e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0420009850 tx=0x7f042002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a9f04a7715df93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff79f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff7980e33f0 0x7ff7980068a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59632/0 (socket says 172.21.15.122:59632) 2024-03-06T07:22:02.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff79f7fe640 1 -- 172.21.15.122:0/2801253170 learned_addr learned my addr 172.21.15.122:0/2801253170 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f06b8a87640 1 -- 172.21.15.122:0/196293470 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b0031710 con 0x7f06b40c6850 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fa27941f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c04fc10 0x7fa26c051770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.735+0000 7fa27941f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa26c008170 0x7fa26c04fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa27941f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26c048520 con 0x7fa26c050530 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa27941f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26c0487c0 con 0x7fa26c008170 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa27941f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c048a80 con 0x7fa26c04fc10 2024-03-06T07:22:02.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f0432ffd640 1 -- 172.21.15.122:0/1411505273 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420004230 con 0x7f04340b3a20 2024-03-06T07:22:02.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff79effd640 1 --2- 172.21.15.122:0/2801253170 >> v2:172.21.15.169:3300/0 conn(0x7ff7980e6e20 0x7ff7980e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa278c1e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa26c050530 0x7fa26c0538a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59618/0 (socket says 172.21.15.122:59618) 2024-03-06T07:22:02.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa278c1e640 1 -- 172.21.15.122:0/2916437514 learned_addr learned my addr 172.21.15.122:0/2916437514 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:02.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa2737fe640 1 --2- 172.21.15.122:0/2916437514 >> v2:172.21.15.122:3300/0 conn(0x7fa26c008170 0x7fa26c04fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fa273fff640 1 --2- 172.21.15.122:0/2916437514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c04fc10 0x7fa26c051770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e1af7640 1 -- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c80051c0 msgr2=0x7fd2c800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:02.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c80051c0 0x7fd2c800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e1af7640 1 -- 172.21.15.122:0/3590534769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2c8003b40 msgr2=0x7fd2c8003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2c8003b40 0x7fd2c8003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7ff79ffff640 1 --2- 172.21.15.122:0/2801253170 >> v2:172.21.15.122:3300/0 conn(0x7ff798057b00 0x7ff798057f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f0432ffd640 1 -- 172.21.15.122:0/1411505273 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04200043d0 con 0x7f04340b3a20 2024-03-06T07:22:02.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f0432ffd640 1 -- 172.21.15.122:0/1411505273 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420031830 con 0x7f04340b3a20 2024-03-06T07:22:02.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fd2e1af7640 1 -- 172.21.15.122:0/3590534769 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8003520 con 0x7fd2c8004800 2024-03-06T07:22:02.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fa272ffd640 1 -- 172.21.15.122:0/2916437514 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528520876 0 0) 0x7fa26c048520 con 0x7fa26c050530 2024-03-06T07:22:02.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fd2e22f8640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2c8003b40 0x7fd2c8003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fa2737fe640 1 -- 172.21.15.122:0/2916437514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c04fc10 msgr2=0x7fa26c051770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fa2737fe640 1 --2- 172.21.15.122:0/2916437514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c04fc10 0x7fa26c051770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fa2737fe640 1 -- 172.21.15.122:0/2916437514 >> v1:172.21.15.122:6789/0 conn(0x7fa26c050530 legacy=0x7fa26c0538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fa2737fe640 1 -- 172.21.15.122:0/2916437514 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c0530a0 con 0x7fa26c008170 2024-03-06T07:22:02.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7f646dd9b640 1 --2- 172.21.15.122:0/692411644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6470162c80 0x7f647006e790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f64640097a0 tx=0x7f646402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d189dc6c1db327e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.736+0000 7f06b8a87640 1 -- 172.21.15.122:0/196293470 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0031ae0 con 0x7f06b40c6850 2024-03-06T07:22:02.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fef877fe640 1 -- 172.21.15.122:0/3433124471 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c035020 con 0x7fef880cf640 2024-03-06T07:22:02.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fef877fe640 1 -- 172.21.15.122:0/3433124471 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c031890 con 0x7fef880cf640 2024-03-06T07:22:02.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7fef877fe640 1 -- 172.21.15.122:0/3433124471 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c031ca0 con 0x7fef880cf640 2024-03-06T07:22:02.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7ff79ffff640 1 -- 172.21.15.122:0/2801253170 >> v2:172.21.15.169:3300/0 conn(0x7ff7980e6e20 msgr2=0x7ff7980e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7f043a8c9640 1 -- 172.21.15.122:0/1411505273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 msgr2=0x7f04340b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7f043a8c9640 1 --2- 172.21.15.122:0/1411505273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 0x7f04340b3e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0420009850 tx=0x7f042002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f043a8c9640 1 -- 172.21.15.122:0/1411505273 shutdown_connections 2024-03-06T07:22:02.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f043a8c9640 1 --2- 172.21.15.122:0/1411505273 >> v2:172.21.15.122:3300/0 conn(0x7f043415a3f0 0x7f043415a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f043a8c9640 1 --2- 172.21.15.122:0/1411505273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04340b3a20 0x7f04340b3e80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f043a8c9640 1 -- 172.21.15.122:0/1411505273 >> 172.21.15.122:0/1411505273 conn(0x7f04340b1570 msgr2=0x7f04340b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7ff79ffff640 1 --2- 172.21.15.122:0/2801253170 >> v2:172.21.15.169:3300/0 conn(0x7ff7980e6e20 0x7ff7980e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7ff79ffff640 1 -- 172.21.15.122:0/2801253170 >> v1:172.21.15.122:6789/0 conn(0x7ff7980e33f0 legacy=0x7ff7980068a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7ff79e7fc640 1 -- 172.21.15.122:0/2801253170 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490161258 0 0) 0x7ff798007340 con 0x7ff7980e33f0 2024-03-06T07:22:02.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.737+0000 7ff79ffff640 1 -- 172.21.15.122:0/2801253170 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798057300 con 0x7ff798057b00 2024-03-06T07:22:02.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f646d59a640 1 -- 172.21.15.122:0/692411644 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464038070 con 0x7f6470162c80 2024-03-06T07:22:02.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f043a8c9640 1 -- 172.21.15.122:0/1411505273 shutdown_connections 2024-03-06T07:22:02.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f646d59a640 1 -- 172.21.15.122:0/692411644 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6464031710 con 0x7f6470162c80 2024-03-06T07:22:02.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 msgr2=0x7f06b406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7fa2737fe640 1 --2- 172.21.15.122:0/2916437514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c008170 0x7fa26c04fff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa2680097a0 tx=0x7fa26802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd694e3c2387ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 --2- 172.21.15.122:0/196293470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 0x7f06b406afb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f06b4067c30 tx=0x7f06b002b7b0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 shutdown_connections 2024-03-06T07:22:02.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 --2- 172.21.15.122:0/196293470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 0x7f06b406afb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 --2- 172.21.15.122:0/196293470 >> v2:172.21.15.169:3300/0 conn(0x7f06b40affa0 0x7f06b40ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 >> 172.21.15.122:0/196293470 conn(0x7f06b40d0870 msgr2=0x7f06b40d0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7ff79ffff640 1 --2- 172.21.15.122:0/2801253170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff798057b00 0x7ff798057f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff7a002d580 tx=0x7ff7a00a6bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e443472ce8e214a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7fa272ffd640 1 -- 172.21.15.122:0/2916437514 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268035020 con 0x7fa26c008170 2024-03-06T07:22:02.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7fa272ffd640 1 -- 172.21.15.122:0/2916437514 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa268031830 con 0x7fa26c008170 2024-03-06T07:22:02.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7ff79e7fc640 1 -- 172.21.15.122:0/2801253170 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00b4070 con 0x7ff798057b00 2024-03-06T07:22:02.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7fa272ffd640 1 -- 172.21.15.122:0/2916437514 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268031c00 con 0x7fa26c008170 2024-03-06T07:22:02.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 shutdown_connections 2024-03-06T07:22:02.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7ff79e7fc640 1 -- 172.21.15.122:0/2801253170 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a00a7820 con 0x7ff798057b00 2024-03-06T07:22:02.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7ff79e7fc640 1 -- 172.21.15.122:0/2801253170 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00a7c10 con 0x7ff798057b00 2024-03-06T07:22:02.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f646d59a640 1 -- 172.21.15.122:0/692411644 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464031ae0 con 0x7f6470162c80 2024-03-06T07:22:02.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7fd2e2af9640 1 -- 172.21.15.122:0/3590534769 reap_dead start 2024-03-06T07:22:02.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2c8004800 0x7fd2c8004c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009c90 tx=0x7fd2c8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb4d3093042f06fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.738+0000 7f043a8c9640 1 -- 172.21.15.122:0/1411505273 wait complete. 2024-03-06T07:22:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 msgr2=0x7fef881630a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 --2- 172.21.15.122:0/3433124471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef881630a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fef880b22e0 tx=0x7fef7c02b930 comp rx=0 tx=0).stop 2024-03-06T07:22:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d803a070 con 0x7fd2c8004800 2024-03-06T07:22:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8035600 con 0x7fd2c8004800 2024-03-06T07:22:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fd2e12f6640 1 -- 172.21.15.122:0/3590534769 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8009860 con 0x7fd2c8004800 2024-03-06T07:22:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6470162c80 msgr2=0x7f647006e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 shutdown_connections 2024-03-06T07:22:02.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.122:3300/0 conn(0x7fef880b43e0 0x7fef88165c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 --2- 172.21.15.122:0/3433124471 >> v2:172.21.15.169:3300/0 conn(0x7fef88070cc0 0x7fef880b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 --2- 172.21.15.122:0/3433124471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef881630a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 >> 172.21.15.122:0/3433124471 conn(0x7fef8806b8f0 msgr2=0x7fef880b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f646f59e640 1 --2- 172.21.15.122:0/692411644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6470162c80 0x7f647006e790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f64640097a0 tx=0x7f646402b7b0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 -- 172.21.15.122:0/2916437514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c008170 msgr2=0x7fa26c04fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 shutdown_connections 2024-03-06T07:22:02.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 --2- 172.21.15.122:0/2916437514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c008170 0x7fa26c04fff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa2680097a0 tx=0x7fa26802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 -- 172.21.15.122:0/2916437514 shutdown_connections 2024-03-06T07:22:02.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 shutdown_connections 2024-03-06T07:22:02.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f646f59e640 1 --2- 172.21.15.122:0/692411644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6470162c80 0x7f647006e790 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f646f59e640 1 --2- 172.21.15.122:0/692411644 >> v2:172.21.15.169:3300/0 conn(0x7f6470165580 0x7f6470162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 -- 172.21.15.122:0/2801253170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff798057b00 msgr2=0x7ff798057f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 --2- 172.21.15.122:0/2801253170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff798057b00 0x7ff798057f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff7a002d580 tx=0x7ff7a00a6bc0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 >> 172.21.15.122:0/692411644 conn(0x7f64700b4e60 msgr2=0x7f64700b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 --2- 172.21.15.122:0/2916437514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c008170 0x7fa26c04fff0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 -- 172.21.15.122:0/2801253170 shutdown_connections 2024-03-06T07:22:02.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 --2- 172.21.15.122:0/2801253170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff798057b00 0x7ff798057f90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 --2- 172.21.15.122:0/2801253170 >> v2:172.21.15.169:3300/0 conn(0x7ff7980e6e20 0x7ff7980e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 -- 172.21.15.122:0/2801253170 >> 172.21.15.122:0/2801253170 conn(0x7ff79803aa30 msgr2=0x7ff79803ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 --2- 172.21.15.122:0/2916437514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c04fc10 0x7fa26c051770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 -- 172.21.15.122:0/2916437514 >> 172.21.15.122:0/2916437514 conn(0x7fa26c055350 msgr2=0x7fa26c055760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7f06bbced640 1 -- 172.21.15.122:0/196293470 wait complete. 2024-03-06T07:22:02.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 -- 172.21.15.122:0/2916437514 shutdown_connections 2024-03-06T07:22:02.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7ff7a6204640 1 -- 172.21.15.122:0/2801253170 shutdown_connections 2024-03-06T07:22:02.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fef8e002640 1 -- 172.21.15.122:0/3433124471 wait complete. 2024-03-06T07:22:02.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7ff7a6204640 1 -- 172.21.15.122:0/2801253170 wait complete. 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2c8004800 msgr2=0x7fd2c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2c8004800 0x7fd2c8004c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009c90 tx=0x7fd2c8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 shutdown_connections 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 --2- 172.21.15.122:0/3590534769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2c8004800 0x7fd2c8004c80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 >> 172.21.15.122:0/3590534769 conn(0x7fd2d401ef40 msgr2=0x7fd2d4055c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 shutdown_connections 2024-03-06T07:22:02.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7fd2e32fa640 1 -- 172.21.15.122:0/3590534769 wait complete. 2024-03-06T07:22:02.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.739+0000 7fa27941f640 1 -- 172.21.15.122:0/2916437514 wait complete. 2024-03-06T07:22:02.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 shutdown_connections 2024-03-06T07:22:02.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7f646f59e640 1 -- 172.21.15.122:0/692411644 wait complete. 2024-03-06T07:22:02.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7fef8e002640 1 Processor -- start 2024-03-06T07:22:02.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7fef8e002640 1 -- start start 2024-03-06T07:22:02.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7f043a8c9640 1 Processor -- start 2024-03-06T07:22:02.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.740+0000 7f043a8c9640 1 -- start start 2024-03-06T07:22:02.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f043a8c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04340b2090 0x7f0434142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7ff7a6204640 1 Processor -- start 2024-03-06T07:22:02.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f043a8c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04340b3a20 0x7f04341483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f043a8c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f043415a3f0 0x7f0434141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f043a8c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04340b2fd0 con 0x7f04340b3a20 2024-03-06T07:22:02.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f043a8c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043415b170 con 0x7f04340b2090 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f043a8c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043415ad90 con 0x7f043415a3f0 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f0438e66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f043415a3f0 0x7f0434141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f06bbced640 1 Processor -- start 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7ff7a6204640 1 -- start start 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f0438e66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f043415a3f0 0x7f0434141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44532/0 (socket says 172.21.15.122:44532) 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.741+0000 7f0438e66640 1 -- 172.21.15.122:0/539176252 learned_addr learned my addr 172.21.15.122:0/539176252 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7fa27941f640 1 Processor -- start 2024-03-06T07:22:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f04337fe640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04340b3a20 0x7f04341483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7fd2e32fa640 1 Processor -- start 2024-03-06T07:22:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7fd2e32fa640 1 -- start start 2024-03-06T07:22:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0438e66640 1 -- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04340b2090 msgr2=0x7f0434142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0433fff640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04340b2090 0x7f0434142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0438e66640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04340b2090 0x7f0434142fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0438e66640 1 -- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04340b3a20 msgr2=0x7f04341483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0438e66640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04340b3a20 0x7f04341483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0438e66640 1 -- 172.21.15.122:0/539176252 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434141b10 con 0x7f043415a3f0 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f06bbced640 1 -- start start 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f04337fe640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04340b3a20 0x7f04341483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0433fff640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04340b2090 0x7f0434142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7fd843c50640 1 Processor -- start 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0438e66640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f043415a3f0 0x7f0434141250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f042c00efc0 tx=0x7f042c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7fef8e002640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef88070cc0 0x7fef88156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fef8e002640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef880b43e0 0x7fef8815cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fef8e002640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef8815e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fef8e002640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88166150 con 0x7fef880b43e0 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fef8e002640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef880cfa20 con 0x7fef88070cc0 2024-03-06T07:22:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fef8e002640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef881662d0 con 0x7fef880cf640 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7f06bbced640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06b406b4f0 0x7f06b414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7f06bbced640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06b40affa0 0x7f06b4164620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7f06b9a89640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06b406b4f0 0x7f06b414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06b9a89640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06b406b4f0 0x7f06b414e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44572/0 (socket says 172.21.15.122:44572) 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06b9a89640 1 -- 172.21.15.122:0/237295139 learned_addr learned my addr 172.21.15.122:0/237295139 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e32fa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c8004800 0x7fd2d4051730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e32fa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2d404bf70 0x7fd2d404b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e22f8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c8004800 0x7fd2d4051730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e22f8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c8004800 0x7fd2d4051730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45928/0 (socket says 172.21.15.122:45928) 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e22f8640 1 -- 172.21.15.122:0/404146492 learned_addr learned my addr 172.21.15.122:0/404146492 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e32fa640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2d417e120 0x7fd2d4034880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2d404bf70 0x7fd2d404b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4056b50 con 0x7fd2d417e120 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4009140 con 0x7fd2c8004800 2024-03-06T07:22:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40e4910 con 0x7fd2d404bf70 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff7a6204640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff798057b00 0x7ff7980d74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff7a6204640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7980e33f0 0x7ff7980e4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff79f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff798057b00 0x7ff7980d74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff79f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff798057b00 0x7ff7980d74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45926/0 (socket says 172.21.15.122:45926) 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff79f7fe640 1 -- 172.21.15.122:0/1050621326 learned_addr learned my addr 172.21.15.122:0/1050621326 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff7a6204640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7980e6e20 0x7ff798052020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980e7b90 con 0x7ff7980e33f0 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff79ffff640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7980e6e20 0x7ff798052020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79803c0c0 con 0x7ff798057b00 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff798006fb0 con 0x7ff7980e6e20 2024-03-06T07:22:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7fd843c50640 1 -- start start 2024-03-06T07:22:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f646f59e640 1 Processor -- start 2024-03-06T07:22:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7f646f59e640 1 -- start start 2024-03-06T07:22:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06bbced640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 0x7f06b4071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06b9288640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06b40affa0 0x7f06b4164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646f59e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f647006ecd0 0x7f647014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646f59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6470162c80 0x7f6470163700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6470165580 0x7f6470163c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646f59e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b5df0 con 0x7f6470165580 2024-03-06T07:22:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646f59e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647006f160 con 0x7f6470162c80 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646f59e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b5f70 con 0x7f647006ecd0 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7fa27941f640 1 -- start start 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40cb110 con 0x7f06b40c6850 2024-03-06T07:22:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40cee20 con 0x7f06b40affa0 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f647006ecd0 0x7f647014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.742+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c018020 con 0x7f043415a3f0 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40cc920 con 0x7f06b406b4f0 2024-03-06T07:22:02.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c014480 con 0x7f043415a3f0 2024-03-06T07:22:02.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.743+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c014870 con 0x7f043415a3f0 2024-03-06T07:22:02.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f043a8c9640 1 -- 172.21.15.122:0/539176252 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0434141df0 con 0x7f043415a3f0 2024-03-06T07:22:02.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f043a8c9640 1 -- 172.21.15.122:0/539176252 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04341423b0 con 0x7f043415a3f0 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fef8d801640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef8815e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fd2e22f8640 1 -- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2d404bf70 msgr2=0x7fd2d404b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fd2e22f8640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2d404bf70 0x7fd2d404b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fd2e22f8640 1 -- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2d417e120 msgr2=0x7fd2d4034880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fd2e22f8640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2d417e120 0x7fd2d4034880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fd2e22f8640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d404bb70 con 0x7fd2c8004800 2024-03-06T07:22:02.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fef8d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef88070cc0 0x7fef88156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f647006ecd0 0x7f647014d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44586/0 (socket says 172.21.15.122:44586) 2024-03-06T07:22:02.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 -- 172.21.15.122:0/849967098 learned_addr learned my addr 172.21.15.122:0/849967098 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fa27941f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c008170 0x7fa26c04d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646dd9b640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6470162c80 0x7f6470163700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fef8d000640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef88070cc0 0x7fef88156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45914/0 (socket says 172.21.15.122:45914) 2024-03-06T07:22:02.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fef8d000640 1 -- 172.21.15.122:0/3675478768 learned_addr learned my addr 172.21.15.122:0/3675478768 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fef87fff640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef880b43e0 0x7fef8815cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff79f7fe640 1 -- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7980e6e20 msgr2=0x7ff798052020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff79f7fe640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7980e6e20 0x7ff798052020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff79f7fe640 1 -- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7980e33f0 msgr2=0x7ff7980e4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff79f7fe640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7980e33f0 0x7ff7980e4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff79f7fe640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7980500b0 con 0x7ff798057b00 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fd2e22f8640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c8004800 0x7fd2d4051730 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc001110 tx=0x7fd2cc00d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fa27941f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c04fc10 0x7fa26c04b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fa27941f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c050530 0x7fa26c044b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fa27941f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c054130 con 0x7fa26c04fc10 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fa27941f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c053fb0 con 0x7fa26c050530 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7fa27941f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0542b0 con 0x7fa26c008170 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7ff79f7fe640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff798057b00 0x7ff7980d74e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00c450 tx=0x7ff78c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 -- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6470162c80 msgr2=0x7f6470163700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6470162c80 0x7f6470163700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 -- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6470165580 msgr2=0x7f6470163c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6470165580 0x7f6470163c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.744+0000 7f646e59c640 1 -- 172.21.15.122:0/849967098 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647009b030 con 0x7f647006ecd0 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc00dc30 con 0x7fd2c8004800 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c018070 con 0x7ff798057b00 2024-03-06T07:22:02.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7f043a8c9640 1 -- 172.21.15.122:0/539176252 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0434068f10 con 0x7f043415a3f0 2024-03-06T07:22:02.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd843c50640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd834050140 0x7fd834038170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fef8d000640 1 -- 172.21.15.122:0/3675478768 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 msgr2=0x7fef8815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fef8d000640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef880cf640 0x7fef8815e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fef8d000640 1 -- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef880b43e0 msgr2=0x7fef8815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fef8d000640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef880b43e0 0x7fef8815cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fef8d000640 1 -- 172.21.15.122:0/3675478768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8815d6a0 con 0x7fef88070cc0 2024-03-06T07:22:02.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fa273fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c008170 0x7fa26c04d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fa273fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c008170 0x7fa26c04d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44598/0 (socket says 172.21.15.122:44598) 2024-03-06T07:22:02.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fa273fff640 1 -- 172.21.15.122:0/4209611859 learned_addr learned my addr 172.21.15.122:0/4209611859 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc018070 con 0x7fd2c8004800 2024-03-06T07:22:02.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0050d0 con 0x7fd2c8004800 2024-03-06T07:22:02.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff798050390 con 0x7ff798057b00 2024-03-06T07:22:02.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7980e4870 con 0x7ff798057b00 2024-03-06T07:22:02.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd843c50640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df410 0x7fd834006a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd843c50640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd834005f60 con 0x7fd8340df030 2024-03-06T07:22:02.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d404c4a0 con 0x7fd2c8004800 2024-03-06T07:22:02.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c0074e0 con 0x7ff798057b00 2024-03-06T07:22:02.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd834050140 0x7fd834038170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd834050140 0x7fd834038170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45974/0 (socket says 172.21.15.122:45974) 2024-03-06T07:22:02.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 -- 172.21.15.122:0/3482266433 learned_addr learned my addr 172.21.15.122:0/3482266433 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340e38e0 con 0x7fd834050140 2024-03-06T07:22:02.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd834006200 con 0x7fd8340df410 2024-03-06T07:22:02.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d404ae50 con 0x7fd2c8004800 2024-03-06T07:22:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7f646e59c640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f647006ecd0 0x7f647014d530 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f646000d7d0 tx=0x7f646000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fef8d000640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef88070cc0 0x7fef88156a70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0090b0 tx=0x7fef7c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460018020 con 0x7f647006ecd0 2024-03-06T07:22:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7f06ba28a640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 0x7f06b4071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64600096a0 con 0x7f647006ecd0 2024-03-06T07:22:02.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f646f59e640 1 -- 172.21.15.122:0/849967098 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647009b310 con 0x7f647006ecd0 2024-03-06T07:22:02.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7f06b9a89640 1 -- 172.21.15.122:0/237295139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06b40affa0 msgr2=0x7f06b4164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f06b9a89640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06b40affa0 0x7f06b4164620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f06b9a89640 1 -- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 msgr2=0x7f06b4071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f06b9a89640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40c6850 0x7f06b4071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f06b9a89640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b4164e70 con 0x7f06b406b4f0 2024-03-06T07:22:02.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f646f59e640 1 -- 172.21.15.122:0/849967098 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647009b8d0 con 0x7f647006ecd0 2024-03-06T07:22:02.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fa273fff640 1 -- 172.21.15.122:0/4209611859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c050530 msgr2=0x7fa26c044b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fa273fff640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c050530 0x7fa26c044b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fa273fff640 1 -- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c04fc10 msgr2=0x7fa26c04b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fa273fff640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa26c04fc10 0x7fa26c04b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fa273fff640 1 -- 172.21.15.122:0/4209611859 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c045580 con 0x7fa26c008170 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c040070 con 0x7fef88070cc0 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c030d20 con 0x7fef88070cc0 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c035040 con 0x7fef88070cc0 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 -- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df410 msgr2=0x7fd834006a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 --2- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df410 0x7fd834006a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 -- 172.21.15.122:0/3482266433 >> v1:172.21.15.122:6789/0 conn(0x7fd8340df030 legacy=0x7fd83404fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.745+0000 7fd8411eb640 1 -- 172.21.15.122:0/3482266433 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340064e0 con 0x7fd834050140 2024-03-06T07:22:02.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2d404b230 con 0x7fd2c8004800 2024-03-06T07:22:02.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.746+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460009a90 con 0x7f647006ecd0 2024-03-06T07:22:02.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c004a20 con 0x7ff798057b00 2024-03-06T07:22:02.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fa273fff640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c008170 0x7fa26c04d030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa26000d800 tx=0x7fa26000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fd8411eb640 1 --2- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd834050140 0x7fd834038170 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd83400dae0 tx=0x7fd82c02ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2379002a2c1031d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff780001030 con 0x7ff798057b00 2024-03-06T07:22:02.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8815dc60 con 0x7fef88070cc0 2024-03-06T07:22:02.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef88156fb0 con 0x7fef88070cc0 2024-03-06T07:22:02.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26000b6c0 con 0x7fa26c008170 2024-03-06T07:22:02.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa260019070 con 0x7fa26c008170 2024-03-06T07:22:02.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26000ba60 con 0x7fa26c008170 2024-03-06T07:22:02.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c04c550 con 0x7fa26c008170 2024-03-06T07:22:02.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f06b9a89640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06b406b4f0 0x7f06b414e680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c7f0 tx=0x7f06a400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff78c01b070 con 0x7ff798057b00 2024-03-06T07:22:02.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4015070 con 0x7f06b406b4f0 2024-03-06T07:22:02.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fd8409ea640 1 -- 172.21.15.122:0/3482266433 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c034070 con 0x7fd834050140 2024-03-06T07:22:02.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fd8409ea640 1 -- 172.21.15.122:0/3482266433 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c008df0 con 0x7fd834050140 2024-03-06T07:22:02.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fd8409ea640 1 -- 172.21.15.122:0/3482266433 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00a2f0 con 0x7fd834050140 2024-03-06T07:22:02.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f042c01b020 con 0x7f043415a3f0 2024-03-06T07:22:02.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f0430ff9640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f041806b5d0 0x7f041806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f042c08aba0 con 0x7f043415a3f0 2024-03-06T07:22:02.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f0430ff9640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f0418073770 0x7f0418075b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0418076240 con 0x7f0418073770 2024-03-06T07:22:02.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a4007a30 con 0x7f06b406b4f0 2024-03-06T07:22:02.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b4165150 con 0x7f06b406b4f0 2024-03-06T07:22:02.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b414c540 con 0x7f06b406b4f0 2024-03-06T07:22:02.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef88068f80 con 0x7fef88070cc0 2024-03-06T07:22:02.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.747+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c04cb10 con 0x7fa26c008170 2024-03-06T07:22:02.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7ff78bfff640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7f646f59e640 1 -- 172.21.15.122:0/849967098 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6454001030 con 0x7f647006ecd0 2024-03-06T07:22:02.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2cc005270 con 0x7fd2c8004800 2024-03-06T07:22:02.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa26c0057e0 con 0x7fa26c008170 2024-03-06T07:22:02.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2d2ffd640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b5d0 0x7fd2c006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7ff79effd640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff78406b4d0 0x7ff78406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2cc08a2a0 con 0x7fd2c8004800 2024-03-06T07:22:02.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2d2ffd640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7fd2c00737d0 0x7fd2c0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2cc00f020 con 0x7fd2c00737d0 2024-03-06T07:22:02.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7f0433fff640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f041806b5d0 0x7f041806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.748+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f042c058cf0 con 0x7f043415a3f0 2024-03-06T07:22:02.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f04337fe640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f0418073770 0x7f0418075b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2cc091050 con 0x7fd2c8004800 2024-03-06T07:22:02.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd834050140 msgr2=0x7fd834038170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd843c50640 1 --2- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd834050140 0x7fd834038170 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd83400dae0 tx=0x7fd82c02ca20 comp rx=0 tx=0).stop 2024-03-06T07:22:02.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2e2af9640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7fd2c00737d0 0x7fd2c0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f646001b020 con 0x7f647006ecd0 2024-03-06T07:22:02.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b5d0 0x7fd2c006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f645effd640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f646008a500 con 0x7f647006ecd0 2024-03-06T07:22:02.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f645effd640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f64500736f0 0x7f6450075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f646000cfb0 con 0x7f64500736f0 2024-03-06T07:22:02.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6460091050 con 0x7f647006ecd0 2024-03-06T07:22:02.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7f646ed9d640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f64500736f0 0x7f6450075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7f04337fe640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f0418073770 0x7f0418075b90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7f0433fff640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f041806b5d0 0x7f041806da90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f04280011e0 tx=0x7f0428009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 shutdown_connections 2024-03-06T07:22:02.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fd843c50640 1 --2- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df410 0x7fd834006a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fd843c50640 1 --2- 172.21.15.122:0/3482266433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd834050140 0x7fd834038170 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd83400dae0 tx=0x7fd82c02ca20 comp rx=0 tx=0).stop 2024-03-06T07:22:02.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef7c04b060 con 0x7fef88070cc0 2024-03-06T07:22:02.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fef857fa640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef6c06b5a0 0x7fef6c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef7c0abd30 con 0x7fef88070cc0 2024-03-06T07:22:02.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 >> 172.21.15.122:0/3482266433 conn(0x7fd834009fe0 msgr2=0x7fd83400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fef857fa640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7fef6c073740 0x7fef6c075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.749+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef7c031890 con 0x7fef6c073740 2024-03-06T07:22:02.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fef87fff640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef6c06b5a0 0x7fef6c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4007e00 con 0x7f06b406b4f0 2024-03-06T07:22:02.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06a4018070 con 0x7f06b406b4f0 2024-03-06T07:22:02.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff78c089480 con 0x7ff798057b00 2024-03-06T07:22:02.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7f646dd9b640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fef8d801640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7fef6c073740 0x7fef6c075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7ff78bfff640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ff7840736f0 0x7ff784075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff78c017ec0 con 0x7ff7840736f0 2024-03-06T07:22:02.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff78c090050 con 0x7ff798057b00 2024-03-06T07:22:02.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef7c076870 con 0x7fef88070cc0 2024-03-06T07:22:02.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7ff79effd640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff78406b4d0 0x7ff78406d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff794006a90 tx=0x7ff794006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 shutdown_connections 2024-03-06T07:22:02.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fd843c50640 1 -- 172.21.15.122:0/3482266433 wait complete. 2024-03-06T07:22:02.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fef87fff640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef6c06b5a0 0x7fef6c06da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fef880cca90 tx=0x7fef74009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f069c001030 con 0x7f06b406b4f0 2024-03-06T07:22:02.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fd2e1af7640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b5d0 0x7fd2c006da90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8037000 tx=0x7fd2d8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fd2e2af9640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7fd2c00737d0 0x7fd2c0075bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7ff79ffff640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ff7840736f0 0x7ff784075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06aa7fc640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f069806b5d0 0x7f069806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f646ed9d640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f64500736f0 0x7f6450075b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f646dd9b640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645006b4d0 0x7f645006d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6464035000 tx=0x7f6464008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06a408a240 con 0x7f06b406b4f0 2024-03-06T07:22:02.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.750+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa26000bc00 con 0x7fa26c008170 2024-03-06T07:22:02.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fa270ff9640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa25806b5a0 0x7fa25806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa26008a640 con 0x7fa26c008170 2024-03-06T07:22:02.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== osd.2 v2:172.21.15.122:6816/1235809965 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0418076240 con 0x7f0418073770 2024-03-06T07:22:02.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fa270ff9640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7fa2580737c0 0x7fa258075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa260001d30 con 0x7fa2580737c0 2024-03-06T07:22:02.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa260091050 con 0x7fa26c008170 2024-03-06T07:22:02.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fa2737fe640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa25806b5a0 0x7fa25806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06b9288640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f069806b5d0 0x7f069806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06aa7fc640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f0698073770 0x7f0698075b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06a4009fa0 con 0x7f0698073770 2024-03-06T07:22:02.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06a4091050 con 0x7f06b406b4f0 2024-03-06T07:22:02.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7f06ba28a640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f0698073770 0x7f0698075b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.751+0000 7fef8d801640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7fef6c073740 0x7fef6c075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== osd.4 v2:172.21.15.169:6804/1079551346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd2cc00f020 con 0x7fd2c00737d0 2024-03-06T07:22:02.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== osd.7 v2:172.21.15.169:6800/3079115781 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f646000cfb0 con 0x7f64500736f0 2024-03-06T07:22:02.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7f06b9288640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f069806b5d0 0x7f069806da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f06b0035000 tx=0x7f06b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7ff79ffff640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ff7840736f0 0x7ff784075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7f06ba28a640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f0698073770 0x7f0698075b90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7fa278c1e640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7fa2580737c0 0x7fa258075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7fa2737fe640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa25806b5a0 0x7fa25806da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa268036000 tx=0x7fa268030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7fd843c50640 1 Processor -- start 2024-03-06T07:22:02.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.752+0000 7fd843c50640 1 -- start start 2024-03-06T07:22:02.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.753+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fef7c031890 con 0x7fef6c073740 2024-03-06T07:22:02.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.753+0000 7fa278c1e640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7fa2580737c0 0x7fa258075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.753+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== osd.3 v2:172.21.15.122:6824/47863215 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f06a4009fa0 con 0x7f0698073770 2024-03-06T07:22:02.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd843c50640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8340df030 0x7fd8340ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd843c50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8340df410 0x7fd8340dd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== osd.1 v2:172.21.15.122:6808/20360309 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa260001d30 con 0x7fa2580737c0 2024-03-06T07:22:02.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd8419ec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8340df030 0x7fd8340ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd8419ec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8340df030 0x7fd8340ddc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45976/0 (socket says 172.21.15.122:45976) 2024-03-06T07:22:02.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd8419ec640 1 -- 172.21.15.122:0/2262358566 learned_addr learned my addr 172.21.15.122:0/2262358566 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:02.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd843c50640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df7f0 0x7fd8340e1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== osd.5 v2:172.21.15.169:6816/887889200 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff78c017ec0 con 0x7ff7840736f0 2024-03-06T07:22:02.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd8411eb640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8340df410 0x7fd8340dd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd8421ed640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df7f0 0x7fd8340e1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.754+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340507c0 con 0x7fd8340df410 2024-03-06T07:22:02.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834050640 con 0x7fd8340df030 2024-03-06T07:22:02.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340546c0 con 0x7fd8340df7f0 2024-03-06T07:22:02.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd8411eb640 1 -- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df7f0 msgr2=0x7fd8340e1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd8411eb640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8340df7f0 0x7fd8340e1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd8411eb640 1 -- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8340df030 msgr2=0x7fd8340ddc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd8411eb640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8340df030 0x7fd8340ddc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.755+0000 7fd8411eb640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83404a0d0 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.756+0000 7fd8411eb640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8340df410 0x7fd8340dd4d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd82c033000 tx=0x7fd82c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.756+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c034040 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.756+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c00a2d0 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.756+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834049080 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.760+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c007e60 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.760+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd82c037070 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.760+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd834049310 con 0x7fd8340df410 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.761+0000 7fd8327fc640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd82006b550 0x7fd82006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.761+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd82c077d50 con 0x7fd8340df410 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.762+0000 7fd8419ec640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd82006b550 0x7fd82006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.763+0000 7fd8419ec640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd82006b550 0x7fd82006da10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd838006fd0 tx=0x7fd838008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.763+0000 7fd81ffff640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7fd834049550 0x7fd834049930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.763+0000 7fd81ffff640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd834034550 con 0x7fd834049550 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.763+0000 7fd8421ed640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7fd834049550 0x7fd834049930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.763+0000 7fd8421ed640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7fd834049550 0x7fd834049930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.765+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== osd.6 v2:172.21.15.169:6824/1645803606 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd834034550 con 0x7fd834049550 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.781+0000 7f043a8c9640 1 -- 172.21.15.122:0/539176252 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04340b1c10 con 0x7f0418073770 2024-03-06T07:22:02.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.782+0000 7f0430ff9640 1 -- 172.21.15.122:0/539176252 <== osd.2 v2:172.21.15.122:6816/1235809965 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f04340b1c10 con 0x7f0418073770 2024-03-06T07:22:02.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.782+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f0418073770 msgr2=0x7f0418075b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.782+0000 7f041e7fc640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f0418073770 0x7f0418075b90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f041806b5d0 msgr2=0x7f041806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f041e7fc640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f041806b5d0 0x7f041806da90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f04280011e0 tx=0x7f0428009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f043415a3f0 msgr2=0x7f0434141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f041e7fc640 1 --2- 172.21.15.122:0/539176252 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f043415a3f0 0x7f0434141250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f042c00efc0 tx=0x7f042c00c490 comp rx=0 tx=0).stop 2024-03-06T07:22:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f0438e66640 1 -- 172.21.15.122:0/539176252 reap_dead start 2024-03-06T07:22:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 shutdown_connections 2024-03-06T07:22:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.783+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 >> 172.21.15.122:0/539176252 conn(0x7f04340b1570 msgr2=0x7f04341575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.784+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 shutdown_connections 2024-03-06T07:22:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.784+0000 7f646f59e640 1 -- 172.21.15.122:0/849967098 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6454002dd0 con 0x7f64500736f0 2024-03-06T07:22:02.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.784+0000 7f041e7fc640 1 -- 172.21.15.122:0/539176252 wait complete. 2024-03-06T07:22:02.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.784+0000 7f645effd640 1 -- 172.21.15.122:0/849967098 <== osd.7 v2:172.21.15.169:6800/3079115781 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6454002dd0 con 0x7f64500736f0 2024-03-06T07:22:02.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.784+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2d40057e0 con 0x7fd2c00737d0 2024-03-06T07:22:02.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef880b04b0 con 0x7fef6c073740 2024-03-06T07:22:02.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7fd2d2ffd640 1 -- 172.21.15.122:0/404146492 <== osd.4 v2:172.21.15.169:6804/1079551346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd2d40057e0 con 0x7fd2c00737d0 2024-03-06T07:22:02.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f64500736f0 msgr2=0x7f6450075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7f645cff9640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f64500736f0 0x7f6450075b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7fef857fa640 1 -- 172.21.15.122:0/3675478768 <== osd.0 v2:172.21.15.122:6800/2293549780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fef880b04b0 con 0x7fef6c073740 2024-03-06T07:22:02.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7fd2c00737d0 msgr2=0x7fd2c0075bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7fd2e32fa640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7fd2c00737d0 0x7fd2c0075bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645006b4d0 msgr2=0x7f645006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7f645cff9640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645006b4d0 0x7f645006d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6464035000 tx=0x7f6464008030 comp rx=0 tx=0).stop 2024-03-06T07:22:02.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f647006ecd0 msgr2=0x7f647014d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.785+0000 7f645cff9640 1 --2- 172.21.15.122:0/849967098 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f647006ecd0 0x7f647014d530 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f646000d7d0 tx=0x7f646000dc90 comp rx=0 tx=0).stop 2024-03-06T07:22:02.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b5d0 msgr2=0x7fd2c006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b5d0 0x7fd2c006da90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8037000 tx=0x7fd2d8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:02.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c8004800 msgr2=0x7fd2d4051730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 --2- 172.21.15.122:0/404146492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2c8004800 0x7fd2d4051730 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc001110 tx=0x7fd2cc00d520 comp rx=0 tx=0).stop 2024-03-06T07:22:02.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e2af9640 1 -- 172.21.15.122:0/404146492 reap_dead start 2024-03-06T07:22:02.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa26c0e3c40 con 0x7fa2580737c0 2024-03-06T07:22:02.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7f646ed9d640 1 -- 172.21.15.122:0/849967098 reap_dead start 2024-03-06T07:22:02.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 shutdown_connections 2024-03-06T07:22:02.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 >> 172.21.15.122:0/849967098 conn(0x7f64700b4e60 msgr2=0x7f64700b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7fef6c073740 msgr2=0x7fef6c075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7fef6c073740 0x7fef6c075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef6c06b5a0 msgr2=0x7fef6c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef6c06b5a0 0x7fef6c06da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fef880cca90 tx=0x7fef74009190 comp rx=0 tx=0).stop 2024-03-06T07:22:02.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef88070cc0 msgr2=0x7fef88156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 shutdown_connections 2024-03-06T07:22:02.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 >> 172.21.15.122:0/404146492 conn(0x7fd2d401ef40 msgr2=0x7fd2d40544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 --2- 172.21.15.122:0/3675478768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef88070cc0 0x7fef88156a70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0090b0 tx=0x7fef7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8d801640 1 -- 172.21.15.122:0/3675478768 reap_dead start 2024-03-06T07:22:02.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 shutdown_connections 2024-03-06T07:22:02.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 shutdown_connections 2024-03-06T07:22:02.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 >> 172.21.15.122:0/3675478768 conn(0x7fef8806b8f0 msgr2=0x7fef880b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fa270ff9640 1 -- 172.21.15.122:0/4209611859 <== osd.1 v2:172.21.15.122:6808/20360309 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa26c0e3c40 con 0x7fa2580737c0 2024-03-06T07:22:02.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fd2e32fa640 1 -- 172.21.15.122:0/404146492 wait complete. 2024-03-06T07:22:02.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 shutdown_connections 2024-03-06T07:22:02.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7fef8e002640 1 -- 172.21.15.122:0/3675478768 wait complete. 2024-03-06T07:22:02.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.786+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f069c002d50 con 0x7f0698073770 2024-03-06T07:22:02.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 shutdown_connections 2024-03-06T07:22:02.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff780002dd0 con 0x7ff7840736f0 2024-03-06T07:22:02.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7f645cff9640 1 -- 172.21.15.122:0/849967098 wait complete. 2024-03-06T07:22:02.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7fa2580737c0 msgr2=0x7fa258075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7fa27941f640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7fa2580737c0 0x7fa258075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7f06aa7fc640 1 -- 172.21.15.122:0/237295139 <== osd.3 v2:172.21.15.122:6824/47863215 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f069c002d50 con 0x7f0698073770 2024-03-06T07:22:02.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7ff78bfff640 1 -- 172.21.15.122:0/1050621326 <== osd.5 v2:172.21.15.169:6816/887889200 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff780002dd0 con 0x7ff7840736f0 2024-03-06T07:22:02.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa25806b5a0 msgr2=0x7fa25806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa25806b5a0 0x7fa25806da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa268036000 tx=0x7fa268030000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c008170 msgr2=0x7fa26c04d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 --2- 172.21.15.122:0/4209611859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa26c008170 0x7fa26c04d030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa26000d800 tx=0x7fa26000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa278c1e640 1 -- 172.21.15.122:0/4209611859 reap_dead start 2024-03-06T07:22:02.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.787+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ff7840736f0 msgr2=0x7ff784075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ff7840736f0 0x7ff784075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f0698073770 msgr2=0x7f0698075b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f0698073770 0x7f0698075b90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 shutdown_connections 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 >> 172.21.15.122:0/4209611859 conn(0x7fa26c055350 msgr2=0x7fa26c0e53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 shutdown_connections 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7fa27941f640 1 -- 172.21.15.122:0/4209611859 wait complete. 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f069806b5d0 msgr2=0x7f069806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f069806b5d0 0x7f069806da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f06b0035000 tx=0x7f06b0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06b406b4f0 msgr2=0x7f06b414e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 --2- 172.21.15.122:0/237295139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06b406b4f0 0x7f06b414e680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c7f0 tx=0x7f06a400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06ba28a640 1 -- 172.21.15.122:0/237295139 reap_dead start 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff78406b4d0 msgr2=0x7ff78406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff78406b4d0 0x7ff78406d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff794006a90 tx=0x7ff794006a20 comp rx=0 tx=0).stop 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff798057b00 msgr2=0x7ff7980d74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 --2- 172.21.15.122:0/1050621326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff798057b00 0x7ff7980d74e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00c450 tx=0x7ff78c00c910 comp rx=0 tx=0).stop 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff79ffff640 1 -- 172.21.15.122:0/1050621326 reap_dead start 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 shutdown_connections 2024-03-06T07:22:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 >> 172.21.15.122:0/1050621326 conn(0x7ff79803aa30 msgr2=0x7ff7980057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 shutdown_connections 2024-03-06T07:22:02.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 >> 172.21.15.122:0/237295139 conn(0x7f06b40d0870 msgr2=0x7f06b40b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 shutdown_connections 2024-03-06T07:22:02.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7f06bbced640 1 -- 172.21.15.122:0/237295139 wait complete. 2024-03-06T07:22:02.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.788+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 shutdown_connections 2024-03-06T07:22:02.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.789+0000 7ff7a6204640 1 -- 172.21.15.122:0/1050621326 wait complete. 2024-03-06T07:22:02.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.795+0000 7fd81ffff640 1 -- 172.21.15.122:0/2262358566 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd834052c80 con 0x7fd834049550 2024-03-06T07:22:02.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.796+0000 7fd8327fc640 1 -- 172.21.15.122:0/2262358566 <== osd.6 v2:172.21.15.169:6824/1645803606 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd834052c80 con 0x7fd834049550 2024-03-06T07:22:02.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.796+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7fd834049550 msgr2=0x7fd834049930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.797+0000 7fd843c50640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7fd834049550 0x7fd834049930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.797+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd82006b550 msgr2=0x7fd82006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.797+0000 7fd843c50640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd82006b550 0x7fd82006da10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd838006fd0 tx=0x7fd838008040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.797+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8340df410 msgr2=0x7fd8340dd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.797+0000 7fd843c50640 1 --2- 172.21.15.122:0/2262358566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8340df410 0x7fd8340dd4d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd82c033000 tx=0x7fd82c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.797+0000 7fd8421ed640 1 -- 172.21.15.122:0/2262358566 reap_dead start 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.798+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 shutdown_connections 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.798+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 >> 172.21.15.122:0/2262358566 conn(0x7fd834009fe0 msgr2=0x7fd8340365c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.798+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 shutdown_connections 2024-03-06T07:22:02.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:02.798+0000 7fd843c50640 1 -- 172.21.15.122:0/2262358566 wait complete. 2024-03-06T07:22:02.881 INFO:teuthology.orchestra.run.smithi122.stdout:55834574856 2024-03-06T07:22:02.882 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:02.882 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:02.883 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:02.883 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:02.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:02.885 INFO:teuthology.orchestra.run.smithi122.stdout:55834574857 2024-03-06T07:22:02.885 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:02.886 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:02.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:02.887 INFO:teuthology.orchestra.run.smithi122.stdout:55834574856 2024-03-06T07:22:02.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:02.888 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:02.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:02.889 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:02.890 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:03.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.089+0000 7faa9ef81640 1 Processor -- start 2024-03-06T07:22:03.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.089+0000 7faa9ef81640 1 -- start start 2024-03-06T07:22:03.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9ef81640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7faa9815a280 0x7faa980ae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9ef81640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa98154770 con 0x7faa9806e830 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9ef81640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9815ab90 con 0x7faa98070740 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9ef81640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9815a690 con 0x7faa9815a280 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9cd1d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faa9806e830 0x7faa9806c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59648/0 (socket says 172.21.15.122:59648) 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9cd1d640 1 -- 172.21.15.122:0/3006092700 learned_addr learned my addr 172.21.15.122:0/3006092700 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.090+0000 7faa9d51e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7faa9815a280 0x7faa980ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7faa977fe640 1 -- 172.21.15.122:0/3006092700 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581492442 0 0) 0x7faa9815ab90 con 0x7faa98070740 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7faa977fe640 1 -- 172.21.15.122:0/3006092700 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa80003520 con 0x7faa98070740 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7faa9d51e640 1 -- 172.21.15.122:0/3006092700 >> v1:172.21.15.169:6789/0 conn(0x7faa98070740 legacy=0x7faa98154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7faa9d51e640 1 -- 172.21.15.122:0/3006092700 >> v1:172.21.15.122:6789/0 conn(0x7faa9806e830 legacy=0x7faa9806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7faa9d51e640 1 -- 172.21.15.122:0/3006092700 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9815ae70 con 0x7faa9815a280 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.089+0000 7f9df566d640 1 Processor -- start 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.089+0000 7f9df566d640 1 -- start start 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9df566d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9de80e02d0 0x7f9de80e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9df566d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80e4100 0x7f9de80539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9df566d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8056590 con 0x7f9de80dfef0 2024-03-06T07:22:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9df566d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de80094b0 con 0x7f9de80e02d0 2024-03-06T07:22:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9df566d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80e4530 con 0x7f9de80e4100 2024-03-06T07:22:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.089+0000 7f992eb8a640 1 Processor -- start 2024-03-06T07:22:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.089+0000 7f992eb8a640 1 -- start start 2024-03-06T07:22:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f992eb8a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9928162550 0x7f99280b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f992eb8a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99280b39d0 0x7f992806cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f992eb8a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992806c7f0 con 0x7f992815c6d0 2024-03-06T07:22:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f992eb8a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992815cab0 con 0x7f9928162550 2024-03-06T07:22:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f992eb8a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9928071d50 con 0x7f99280b39d0 2024-03-06T07:22:03.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f7705ad0640 1 Processor -- start 2024-03-06T07:22:03.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9927fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9928162550 0x7f99280b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9de80e02d0 0x7f9de80e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9de80e02d0 0x7f9de80e3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45994/0 (socket says 172.21.15.122:45994) 2024-03-06T07:22:03.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 -- 172.21.15.122:0/2468388701 learned_addr learned my addr 172.21.15.122:0/2468388701 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9df4e6c640 1 --2- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80e4100 0x7f9de80539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 -- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80e4100 msgr2=0x7f9de80539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 --2- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80e4100 0x7f9de80539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 -- 172.21.15.122:0/2468388701 >> v1:172.21.15.122:6789/0 conn(0x7f9de80dfef0 legacy=0x7f9de80085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9def7fe640 1 -- 172.21.15.122:0/2468388701 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8056870 con 0x7f9de80e02d0 2024-03-06T07:22:03.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9927fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9928162550 0x7f99280b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54686/0 (socket says 172.21.15.122:54686) 2024-03-06T07:22:03.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f992c926640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f992815c6d0 0x7f9928071570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59666/0 (socket says 172.21.15.122:59666) 2024-03-06T07:22:03.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f9927fff640 1 -- 172.21.15.122:0/729579776 learned_addr learned my addr 172.21.15.122:0/729579776 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f9927fff640 1 -- 172.21.15.122:0/729579776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99280b39d0 msgr2=0x7f992806cca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:03.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7faa9d51e640 1 --2- 172.21.15.122:0/3006092700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9815a280 0x7faa980ae940 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa880097a0 tx=0x7faa8802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16ac76e2ceb0df87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f9927fff640 1 --2- 172.21.15.122:0/729579776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99280b39d0 0x7f992806cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f9927fff640 1 -- 172.21.15.122:0/729579776 >> v1:172.21.15.122:6789/0 conn(0x7f992815c6d0 legacy=0x7f9928071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f9927fff640 1 -- 172.21.15.122:0/729579776 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928072030 con 0x7f9928162550 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f8877154640 1 Processor -- start 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7faa977fe640 1 -- 172.21.15.122:0/3006092700 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88038070 con 0x7faa9815a280 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7faa977fe640 1 -- 172.21.15.122:0/3006092700 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa88031700 con 0x7faa9815a280 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7faa977fe640 1 -- 172.21.15.122:0/3006092700 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88031ad0 con 0x7faa9815a280 2024-03-06T07:22:03.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f992c926640 1 --1- 172.21.15.122:0/729579776 >> v1:172.21.15.122:6789/0 conn(0x7f992815c6d0 0x7f9928071570 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.091+0000 7f7705ad0640 1 -- start start 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7fa709ba2640 1 Processor -- start 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7f8877154640 1 -- start start 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7705ad0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f76f80eacf0 0x7f76f804fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7705ad0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 0x7f76f8037bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7705ad0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f8053710 con 0x7f76f8038100 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7705ad0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f8054320 con 0x7f76f804ffc0 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7705ad0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f80539f0 con 0x7f76f80eacf0 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f76fffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 0x7f76f8037bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.092+0000 7fa709ba2640 1 -- start start 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7704ace640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f76f80eacf0 0x7f76f804fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f76fffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 0x7f76f8037bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54702/0 (socket says 172.21.15.122:54702) 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f76fffff640 1 -- 172.21.15.122:0/3393502074 learned_addr learned my addr 172.21.15.122:0/3393502074 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f7704ace640 1 -- 172.21.15.122:0/3393502074 >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 msgr2=0x7f76f8037bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7faa9ef81640 1 -- 172.21.15.122:0/3006092700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9815a280 msgr2=0x7faa980ae940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f9927fff640 1 --2- 172.21.15.122:0/729579776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9928162550 0x7f99280b3490 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f992000b1a0 tx=0x7f992002c0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c9489ac25a1b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f99277fe640 1 -- 172.21.15.122:0/729579776 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992002ca50 con 0x7f9928162550 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f99277fe640 1 -- 172.21.15.122:0/729579776 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992002cbf0 con 0x7f9928162550 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f76ff7fe640 1 -- 172.21.15.122:0/3393502074 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353693476 0 0) 0x7f76f8053710 con 0x7f76f8038100 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7faa9ef81640 1 --2- 172.21.15.122:0/3006092700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9815a280 0x7faa980ae940 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa880097a0 tx=0x7faa8802b820 comp rx=0 tx=0).stop 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7f99277fe640 1 -- 172.21.15.122:0/729579776 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99200343f0 con 0x7f9928162550 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8877154640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f88680df410 0x7f886804fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f9def7fe640 1 --2- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e02d0 0x7f9de80e3bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9df0000b60 tx=0x7f9df00a6980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87e352089ab4742 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7faa9ef81640 1 -- 172.21.15.122:0/3006092700 shutdown_connections 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7faa9ef81640 1 --2- 172.21.15.122:0/3006092700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9815a280 0x7faa980ae940 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7faa9ef81640 1 -- 172.21.15.122:0/3006092700 >> 172.21.15.122:0/3006092700 conn(0x7faa980b16e0 msgr2=0x7faa980b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7fcf1f8d4640 1 Processor -- start 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.093+0000 7fcf1f8d4640 1 -- start start 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8877154640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886800aa90 0x7f886804ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8877154640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8868006ae0 con 0x7f8868053250 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8877154640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8868006890 con 0x7f88680df410 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8877154640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8868006dc0 con 0x7f886800aa90 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f9deeffd640 1 -- 172.21.15.122:0/2468388701 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df00af020 con 0x7f9de80e02d0 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f9deeffd640 1 -- 172.21.15.122:0/2468388701 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df00ab940 con 0x7f9de80e02d0 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f992eb8a640 1 -- 172.21.15.122:0/729579776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9928162550 msgr2=0x7f99280b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f992eb8a640 1 --2- 172.21.15.122:0/729579776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9928162550 0x7f99280b3490 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f992000b1a0 tx=0x7f992002c0b0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa709ba2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa7041620a0 0x7fa704163020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa709ba2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa704163560 con 0x7fa70406c4c0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa709ba2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040ae6b0 con 0x7fa7040b3e40 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa709ba2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa704162480 con 0x7fa7041620a0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa708ba0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa7041620a0 0x7fa704163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa708ba0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa7041620a0 0x7fa704163020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46010/0 (socket says 172.21.15.122:46010) 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa708ba0640 1 -- 172.21.15.122:0/3113120189 learned_addr learned my addr 172.21.15.122:0/3113120189 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa708ba0640 1 -- 172.21.15.122:0/3113120189 >> v1:172.21.15.169:6789/0 conn(0x7fa7040b3e40 legacy=0x7fa7040713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa708ba0640 1 -- 172.21.15.122:0/3113120189 >> v1:172.21.15.122:6789/0 conn(0x7fa70406c4c0 legacy=0x7fa7040b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7fa708ba0640 1 -- 172.21.15.122:0/3113120189 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040ae990 con 0x7fa7041620a0 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f9deeffd640 1 -- 172.21.15.122:0/2468388701 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df00abdf0 con 0x7f9de80e02d0 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f7704ace640 1 --2- 172.21.15.122:0/3393502074 >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 0x7f76f8037bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f7704ace640 1 -- 172.21.15.122:0/3393502074 >> v1:172.21.15.169:6789/0 conn(0x7f76f8038100 legacy=0x7f76f8038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f7704ace640 1 -- 172.21.15.122:0/3393502074 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f8053cd0 con 0x7f76f80eacf0 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f76fffff640 1 --2- 172.21.15.122:0/3393502074 >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 0x7f76f8037bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7faa9ef81640 1 -- 172.21.15.122:0/3006092700 shutdown_connections 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1f8d4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcf180ce9e0 0x7fcf180b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f88680df410 0x7f886804fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f88680df410 0x7f886804fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46020/0 (socket says 172.21.15.122:46020) 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 -- 172.21.15.122:0/928647812 learned_addr learned my addr 172.21.15.122:0/928647812 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 -- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886800aa90 msgr2=0x7f886804ffc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 --2- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886800aa90 0x7f886804ffc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 -- 172.21.15.122:0/928647812 >> v1:172.21.15.122:6789/0 conn(0x7f8868053250 legacy=0x7f8868053700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.094+0000 7f8876152640 1 -- 172.21.15.122:0/928647812 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88680070a0 con 0x7f88680df410 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1f8d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf1815e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1f8d4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1806e7b0 con 0x7fcf180cedc0 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1f8d4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf18071fd0 con 0x7fcf180ce9e0 2024-03-06T07:22:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1f8d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180722b0 con 0x7fcf1815dda0 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7faa9ef81640 1 -- 172.21.15.122:0/3006092700 wait complete. 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f992eb8a640 1 -- 172.21.15.122:0/729579776 shutdown_connections 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f992eb8a640 1 --2- 172.21.15.122:0/729579776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99280b39d0 0x7f992806cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f992eb8a640 1 --2- 172.21.15.122:0/729579776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9928162550 0x7f99280b3490 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f992eb8a640 1 -- 172.21.15.122:0/729579776 >> 172.21.15.122:0/729579776 conn(0x7f99280b1570 msgr2=0x7f99280b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf1815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf1815e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44632/0 (socket says 172.21.15.122:44632) 2024-03-06T07:22:03.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 -- 172.21.15.122:0/530071215 learned_addr learned my addr 172.21.15.122:0/530071215 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1d670640 1 --2- 172.21.15.122:0/530071215 >> v2:172.21.15.169:3300/0 conn(0x7fcf180ce9e0 0x7fcf180b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7ff788cf4640 1 Processor -- start 2024-03-06T07:22:03.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f9df566d640 1 -- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e02d0 msgr2=0x7f9de80e3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f9df566d640 1 --2- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e02d0 0x7f9de80e3bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9df0000b60 tx=0x7f9df00a6980 comp rx=0 tx=0).stop 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f9df566d640 1 -- 172.21.15.122:0/2468388701 shutdown_connections 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f9df566d640 1 --2- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80e4100 0x7f9de80539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f9df566d640 1 --2- 172.21.15.122:0/2468388701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e02d0 0x7f9de80e3bc0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9df0000b60 tx=0x7f9df00a6980 comp rx=0 tx=0).stop 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f9df566d640 1 -- 172.21.15.122:0/2468388701 >> 172.21.15.122:0/2468388701 conn(0x7f9de801ef40 msgr2=0x7f9de8055c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f992eb8a640 1 -- 172.21.15.122:0/729579776 shutdown_connections 2024-03-06T07:22:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7f992eb8a640 1 -- 172.21.15.122:0/729579776 wait complete. 2024-03-06T07:22:03.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7ff788cf4640 1 -- start start 2024-03-06T07:22:03.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7fa708ba0640 1 --2- 172.21.15.122:0/3113120189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7041620a0 0x7fa704163020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009870 tx=0x7fa6fc02b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c83174dcad1bfb13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7fa7037fe640 1 -- 172.21.15.122:0/3113120189 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc034070 con 0x7fa7041620a0 2024-03-06T07:22:03.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7fa7037fe640 1 -- 172.21.15.122:0/3113120189 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc030880 con 0x7fa7041620a0 2024-03-06T07:22:03.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f7704ace640 1 --2- 172.21.15.122:0/3393502074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f80eacf0 0x7f76f804fa80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f76f8007a50 tx=0x7f76f402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51663333d3bfc320 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f8876152640 1 --2- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680df410 0x7f886804fa80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f886c010bb0 tx=0x7f886c0343c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14dccd9f050c2403 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7fa7037fe640 1 -- 172.21.15.122:0/3113120189 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc030c90 con 0x7fa7041620a0 2024-03-06T07:22:03.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f9df566d640 1 -- 172.21.15.122:0/2468388701 shutdown_connections 2024-03-06T07:22:03.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f9df566d640 1 -- 172.21.15.122:0/2468388701 wait complete. 2024-03-06T07:22:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f8875150640 1 -- 172.21.15.122:0/928647812 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c039710 con 0x7f88680df410 2024-03-06T07:22:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f8875150640 1 -- 172.21.15.122:0/928647812 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c0398b0 con 0x7f88680df410 2024-03-06T07:22:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7f8875150640 1 -- 172.21.15.122:0/928647812 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c039c80 con 0x7f88680df410 2024-03-06T07:22:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 -- 172.21.15.122:0/530071215 >> v2:172.21.15.169:3300/0 conn(0x7fcf180ce9e0 msgr2=0x7fcf180b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 --2- 172.21.15.122:0/530071215 >> v2:172.21.15.169:3300/0 conn(0x7fcf180ce9e0 0x7fcf180b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 -- 172.21.15.122:0/530071215 >> v1:172.21.15.122:6789/0 conn(0x7fcf180cedc0 legacy=0x7fcf1806e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.095+0000 7fcf1de71640 1 -- 172.21.15.122:0/530071215 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18072590 con 0x7fcf1815dda0 2024-03-06T07:22:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7faa9ef81640 1 Processor -- start 2024-03-06T07:22:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7fcf1de71640 1 --2- 172.21.15.122:0/530071215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf1815e230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000d370 tx=0x7fcf1002e1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca7ada5d654457b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7ff788cf4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784143460 0x7ff7840b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7ff788cf4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff784066d00 0x7ff78406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.096+0000 7ff788cf4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7840cfe00 con 0x7ff7840cf9a0 2024-03-06T07:22:03.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff788cf4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7840b30c0 con 0x7ff784066d00 2024-03-06T07:22:03.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff788cf4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840b33a0 con 0x7ff784143460 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fa709ba2640 1 -- 172.21.15.122:0/3113120189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7041620a0 msgr2=0x7fa704163020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fa709ba2640 1 --2- 172.21.15.122:0/3113120189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7041620a0 0x7fa704163020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009870 tx=0x7fa6fc02b9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fa709ba2640 1 -- 172.21.15.122:0/3113120189 shutdown_connections 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fa709ba2640 1 --2- 172.21.15.122:0/3113120189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7041620a0 0x7fa704163020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fa709ba2640 1 -- 172.21.15.122:0/3113120189 >> 172.21.15.122:0/3113120189 conn(0x7fa7040b16e0 msgr2=0x7fa7040b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fcf0ffff640 1 -- 172.21.15.122:0/530071215 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10034070 con 0x7fcf1815dda0 2024-03-06T07:22:03.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fcf0ffff640 1 -- 172.21.15.122:0/530071215 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf10038400 con 0x7fcf1815dda0 2024-03-06T07:22:03.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff784066d00 0x7ff78406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff784066d00 0x7ff78406bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46042/0 (socket says 172.21.15.122:46042) 2024-03-06T07:22:03.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 -- 172.21.15.122:0/2778737471 learned_addr learned my addr 172.21.15.122:0/2778737471 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff7837fe640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784143460 0x7ff7840b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f8877154640 1 -- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680df410 msgr2=0x7f886804fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f8877154640 1 --2- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680df410 0x7f886804fa80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f886c010bb0 tx=0x7f886c0343c0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f992eb8a640 1 Processor -- start 2024-03-06T07:22:03.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f992eb8a640 1 -- start start 2024-03-06T07:22:03.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f8877154640 1 -- 172.21.15.122:0/928647812 shutdown_connections 2024-03-06T07:22:03.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f8877154640 1 --2- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886800aa90 0x7f886804ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f8877154640 1 --2- 172.21.15.122:0/928647812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680df410 0x7f886804fa80 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f886c010bb0 tx=0x7f886c0343c0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f8877154640 1 -- 172.21.15.122:0/928647812 >> 172.21.15.122:0/928647812 conn(0x7f8868009fa0 msgr2=0x7f886800a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f76ff7fe640 1 -- 172.21.15.122:0/3393502074 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f4035020 con 0x7f76f80eacf0 2024-03-06T07:22:03.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f76ff7fe640 1 -- 172.21.15.122:0/3393502074 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f4031640 con 0x7f76f80eacf0 2024-03-06T07:22:03.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f76ff7fe640 1 -- 172.21.15.122:0/3393502074 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f402d040 con 0x7f76f80eacf0 2024-03-06T07:22:03.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7f9df566d640 1 Processor -- start 2024-03-06T07:22:03.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7fa709ba2640 1 -- 172.21.15.122:0/3113120189 shutdown_connections 2024-03-06T07:22:03.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fa709ba2640 1 -- 172.21.15.122:0/3113120189 wait complete. 2024-03-06T07:22:03.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf0ffff640 1 -- 172.21.15.122:0/530071215 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1000be60 con 0x7fcf1815dda0 2024-03-06T07:22:03.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7f9df566d640 1 -- start start 2024-03-06T07:22:03.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/530071215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 msgr2=0x7fcf1815e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 --2- 172.21.15.122:0/530071215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf1815e230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000d370 tx=0x7fcf1002e1f0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/530071215 shutdown_connections 2024-03-06T07:22:03.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 --2- 172.21.15.122:0/530071215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf1815e230 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7faa9ef81640 1 -- start start 2024-03-06T07:22:03.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9ef81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 0x7faa980adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 --2- 172.21.15.122:0/530071215 >> v2:172.21.15.169:3300/0 conn(0x7fcf180ce9e0 0x7fcf180b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9ef81640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa98070740 0x7faa9806d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9cd1d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 0x7faa980adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 -- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784143460 msgr2=0x7ff7840b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784143460 0x7ff7840b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/530071215 >> 172.21.15.122:0/530071215 conn(0x7fcf180b1c80 msgr2=0x7fcf180cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 -- 172.21.15.122:0/2778737471 >> v1:172.21.15.169:6789/0 conn(0x7ff7840cf9a0 legacy=0x7ff7840cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.097+0000 7ff783fff640 1 -- 172.21.15.122:0/2778737471 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7840b3680 con 0x7ff784066d00 2024-03-06T07:22:03.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9cd1d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 0x7faa980adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46048/0 (socket says 172.21.15.122:46048) 2024-03-06T07:22:03.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7ff7837fe640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784143460 0x7ff7840b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:03.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7f7705ad0640 1 -- 172.21.15.122:0/3393502074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f80eacf0 msgr2=0x7f76f804fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7f7705ad0640 1 --2- 172.21.15.122:0/3393502074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f80eacf0 0x7f76f804fa80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f76f8007a50 tx=0x7f76f402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9ef81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa9815a280 0x7faa980af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9ef81640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa98072190 con 0x7faa98070740 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9ef81640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9815b150 con 0x7faa9806e830 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7faa9ef81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa98159900 con 0x7faa9815a280 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f7705ad0640 1 -- 172.21.15.122:0/3393502074 shutdown_connections 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f7705ad0640 1 --2- 172.21.15.122:0/3393502074 >> v2:172.21.15.122:3300/0 conn(0x7f76f804ffc0 0x7f76f8037bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f7705ad0640 1 --2- 172.21.15.122:0/3393502074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f80eacf0 0x7f76f804fa80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f7705ad0640 1 -- 172.21.15.122:0/3393502074 >> 172.21.15.122:0/3393502074 conn(0x7f76f8009fa0 msgr2=0x7f76f800a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f8877154640 1 -- 172.21.15.122:0/928647812 shutdown_connections 2024-03-06T07:22:03.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7ff783fff640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784066d00 0x7ff78406bfc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009e50 tx=0x7ff77c02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196f839f6a970659 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7ff7827fc640 1 -- 172.21.15.122:0/2778737471 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c02cb20 con 0x7ff784066d00 2024-03-06T07:22:03.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7faa9cd1d640 1 -- 172.21.15.122:0/1712741751 learned_addr learned my addr 172.21.15.122:0/1712741751 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f8877154640 1 -- 172.21.15.122:0/928647812 wait complete. 2024-03-06T07:22:03.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7f992eb8a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 0x7f992815d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7f992eb8a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f992815c6d0 0x7f99281f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992eb8a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9928162550 0x7f9928145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/530071215 shutdown_connections 2024-03-06T07:22:03.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.098+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/530071215 wait complete. 2024-03-06T07:22:03.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7ff7827fc640 1 -- 172.21.15.122:0/2778737471 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c02ccc0 con 0x7ff784066d00 2024-03-06T07:22:03.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992c926640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 0x7f992815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992eb8a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992806d1e0 con 0x7f992815c6d0 2024-03-06T07:22:03.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7ff7827fc640 1 -- 172.21.15.122:0/2778737471 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c0345f0 con 0x7ff784066d00 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992eb8a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99280725f0 con 0x7f99280b39d0 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992eb8a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992806d360 con 0x7f9928162550 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992c926640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 0x7f992815d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46060/0 (socket says 172.21.15.122:46060) 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992c926640 1 -- 172.21.15.122:0/285584542 learned_addr learned my addr 172.21.15.122:0/285584542 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9927fff640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f992815c6d0 0x7f99281f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7fa709ba2640 1 Processor -- start 2024-03-06T07:22:03.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df566d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80dfef0 0x7f9de8034ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992c926640 1 -- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9928162550 msgr2=0x7f9928145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992c926640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9928162550 0x7f9928145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f992c926640 1 -- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f992815c6d0 msgr2=0x7f99281f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7fa709ba2640 1 -- start start 2024-03-06T07:22:03.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df566d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e4100 0x7f9de8006d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f7705ad0640 1 -- 172.21.15.122:0/3393502074 shutdown_connections 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df566d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9de8051e60 0x7f9de80d9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df566d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8056b50 con 0x7f9de8051e60 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df566d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8009140 con 0x7f9de80e4100 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df4e6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9de8051e60 0x7f9de80d9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df566d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80e4910 con 0x7f9de80dfef0 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df4e6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9de8051e60 0x7f9de80d9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54730/0 (socket says 172.21.15.122:54730) 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f9df4e6c640 1 -- 172.21.15.122:0/2979858421 learned_addr learned my addr 172.21.15.122:0/2979858421 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9deffff640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80dfef0 0x7f9de8034ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9def7fe640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e4100 0x7f9de8006d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7f7705ad0640 1 -- 172.21.15.122:0/3393502074 wait complete. 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7fcf1f8d4640 1 Processor -- start 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7fcf1f8d4640 1 -- start start 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7ff788cf4640 1 -- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784066d00 msgr2=0x7ff78406bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784066d00 0x7ff78406bfc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009e50 tx=0x7ff77c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7ff788cf4640 1 -- 172.21.15.122:0/2778737471 shutdown_connections 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784066d00 0x7ff78406bfc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2778737471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784143460 0x7ff7840b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7ff788cf4640 1 -- 172.21.15.122:0/2778737471 >> 172.21.15.122:0/2778737471 conn(0x7ff78406dd50 msgr2=0x7ff78406e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9deffff640 1 -- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e4100 msgr2=0x7f9de8006d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9deffff640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e4100 0x7f9de8006d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9deffff640 1 -- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9de8051e60 msgr2=0x7f9de80d9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9deffff640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9de8051e60 0x7f9de80d9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f9deffff640 1 -- 172.21.15.122:0/2979858421 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8052370 con 0x7f9de80dfef0 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7f8877154640 1 Processor -- start 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7fa709ba2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70406c4c0 0x7fa7040ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7fa709ba2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 0x7fa70415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa709ba2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7041620a0 0x7fa704160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa709ba2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa704070ed0 con 0x7fa70406c4c0 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa709ba2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040aef50 con 0x7fa7040b3e40 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa709ba2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b4420 con 0x7fa7041620a0 2024-03-06T07:22:03.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f7705ad0640 1 Processor -- start 2024-03-06T07:22:03.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f8877154640 1 -- start start 2024-03-06T07:22:03.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7ff788cf4640 1 -- 172.21.15.122:0/2778737471 shutdown_connections 2024-03-06T07:22:03.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f992c926640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f992815c6d0 0x7f99281f7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f992c926640 1 -- 172.21.15.122:0/285584542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928146270 con 0x7f99280b39d0 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f9927fff640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f992815c6d0 0x7f99281f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f992c926640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 0x7f992815d620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f991400c430 tx=0x7f991400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f7705ad0640 1 -- start start 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 0x7fa70415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 0x7fa70415fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46082/0 (socket says 172.21.15.122:46082) 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 -- 172.21.15.122:0/2935263962 learned_addr learned my addr 172.21.15.122:0/2935263962 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa7093a1640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7041620a0 0x7fa704160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 -- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7041620a0 msgr2=0x7fa704160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7041620a0 0x7fa704160130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 -- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70406c4c0 msgr2=0x7fa7040ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70406c4c0 0x7fa7040ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 -- 172.21.15.122:0/2935263962 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7041fc860 con 0x7fa7040b3e40 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1f8d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 0x7fcf181f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1f8d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf180cedc0 0x7fcf18157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1f8d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf18157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1f8d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf18140790 con 0x7fcf180ce9e0 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1f8d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf18072b50 con 0x7fcf180cedc0 2024-03-06T07:22:03.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1f8d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1806dd60 con 0x7fcf1815dda0 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7ff788cf4640 1 -- 172.21.15.122:0/2778737471 wait complete. 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7faa9cd1d640 1 -- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa9815a280 msgr2=0x7faa980af740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7faa9cd1d640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa9815a280 0x7faa980af740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7faa9cd1d640 1 -- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa98070740 msgr2=0x7faa9806d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.099+0000 7faa9cd1d640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa98070740 0x7faa9806d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7faa9cd1d640 1 -- 172.21.15.122:0/1712741751 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa980ae480 con 0x7faa9806e830 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7faa9cd1d640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 0x7faa980adf40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c780 tx=0x7faa8c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c018070 con 0x7faa9806e830 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.100+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c0072d0 con 0x7faa9806e830 2024-03-06T07:22:03.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0047e0 con 0x7faa9806e830 2024-03-06T07:22:03.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140074e0 con 0x7f99280b39d0 2024-03-06T07:22:03.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fa703fff640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 0x7fa70415fbf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c450 tx=0x7fa6ec00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1d670640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 0x7fcf181f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1de71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf18157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1d670640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 0x7fcf181f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54734/0 (socket says 172.21.15.122:54734) 2024-03-06T07:22:03.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7fcf1d670640 1 -- 172.21.15.122:0/802124969 learned_addr learned my addr 172.21.15.122:0/802124969 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99140047e0 con 0x7f99280b39d0 2024-03-06T07:22:03.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7fcf1ce6f640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf180cedc0 0x7fcf18157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99281459d0 con 0x7f99280b39d0 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914004bd0 con 0x7f99280b39d0 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9928145f90 con 0x7f99280b39d0 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f7705ad0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f8038100 0x7f76f8049790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f7705ad0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f804ffc0 0x7f76f8036300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f7704ace640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f8038100 0x7f76f8049790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f7704ace640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f8038100 0x7f76f8049790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46102/0 (socket says 172.21.15.122:46102) 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f7704ace640 1 -- 172.21.15.122:0/4186392493 learned_addr learned my addr 172.21.15.122:0/4186392493 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76f80eacf0 0x7f76f8043670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80547b0 con 0x7f76f804ffc0 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8877154640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886800aa90 0x7f88680d1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8877154640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 0x7f886804bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8877154640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680e1be0 0x7f886804dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8875951640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 0x7f886804bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8875951640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 0x7f886804bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44664/0 (socket says 172.21.15.122:44664) 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8875951640 1 -- 172.21.15.122:0/1413775925 learned_addr learned my addr 172.21.15.122:0/1413775925 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8868052cb0 con 0x7f886800aa90 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8868052b30 con 0x7f88680e1be0 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8868053e10 con 0x7f8868053250 2024-03-06T07:22:03.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f8876953640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680e1be0 0x7f886804dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7fcf1d670640 1 -- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 msgr2=0x7fcf18157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7fcf1d670640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf18157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7fcf1d670640 1 -- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf180cedc0 msgr2=0x7fcf18157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7fcf1d670640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf180cedc0 0x7fcf18157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7fcf1d670640 1 -- 172.21.15.122:0/802124969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf180b3160 con 0x7fcf180ce9e0 2024-03-06T07:22:03.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa98159060 con 0x7faa9806e830 2024-03-06T07:22:03.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8038c80 con 0x7f76f8038100 2024-03-06T07:22:03.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8054930 con 0x7f76f80eacf0 2024-03-06T07:22:03.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa98159620 con 0x7faa9806e830 2024-03-06T07:22:03.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7ff788cf4640 1 Processor -- start 2024-03-06T07:22:03.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.101+0000 7f9deffff640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80dfef0 0x7f9de8034ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9de400c410 tx=0x7f9de400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4017070 con 0x7f9de80dfef0 2024-03-06T07:22:03.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de40035f0 con 0x7f9de80dfef0 2024-03-06T07:22:03.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f9df566d640 1 -- 172.21.15.122:0/2979858421 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de8052650 con 0x7f9de80dfef0 2024-03-06T07:22:03.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f9df566d640 1 -- 172.21.15.122:0/2979858421 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de8052c10 con 0x7f9de80dfef0 2024-03-06T07:22:03.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.102+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4017070 con 0x7f9de80dfef0 2024-03-06T07:22:03.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0052d0 con 0x7fa7040b3e40 2024-03-06T07:22:03.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec005470 con 0x7fa7040b3e40 2024-03-06T07:22:03.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec012850 con 0x7fa7040b3e40 2024-03-06T07:22:03.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70414ff10 con 0x7fa7040b3e40 2024-03-06T07:22:03.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7041fbd80 con 0x7fa7040b3e40 2024-03-06T07:22:03.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f8875951640 1 -- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680e1be0 msgr2=0x7f886804dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f8875951640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680e1be0 0x7f886804dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f8875951640 1 -- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886800aa90 msgr2=0x7f88680d1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f9df566d640 1 -- 172.21.15.122:0/2979858421 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de80057e0 con 0x7f9de80dfef0 2024-03-06T07:22:03.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f8875951640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886800aa90 0x7f88680d1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7f8875951640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886804c4b0 con 0x7f8868053250 2024-03-06T07:22:03.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9928068f10 con 0x7f99280b39d0 2024-03-06T07:22:03.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7f8875951640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 0x7f886804bb80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f886000b4d0 tx=0x7f886000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7f76fffff640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f804ffc0 0x7f76f8036300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7f77052cf640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76f80eacf0 0x7f76f8043670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7ff788cf4640 1 -- start start 2024-03-06T07:22:03.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.103+0000 7fcf1d670640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 0x7fcf181f7ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800d5f0 tx=0x7fcf0800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf080125f0 con 0x7fcf180ce9e0 2024-03-06T07:22:03.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf08019070 con 0x7fcf180ce9e0 2024-03-06T07:22:03.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf080129d0 con 0x7fcf180ce9e0 2024-03-06T07:22:03.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/802124969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf180b3440 con 0x7fcf180ce9e0 2024-03-06T07:22:03.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/802124969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf180b3a00 con 0x7fcf180ce9e0 2024-03-06T07:22:03.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9914011070 con 0x7f99280b39d0 2024-03-06T07:22:03.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f99257fa640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f990c06b5a0 0x7f990c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9914089e10 con 0x7f99280b39d0 2024-03-06T07:22:03.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa704068f80 con 0x7fa7040b3e40 2024-03-06T07:22:03.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa98068700 con 0x7faa9806e830 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa8c004980 con 0x7faa9806e830 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7ff788cf4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784066d00 0x7ff7841317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f77052cf640 1 -- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f8038100 msgr2=0x7f76f8049790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f77052cf640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f8038100 0x7f76f8049790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f77052cf640 1 -- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f804ffc0 msgr2=0x7f76f8036300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f77052cf640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f804ffc0 0x7f76f8036300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f77052cf640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f817d0f0 con 0x7f76f80eacf0 2024-03-06T07:22:03.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7f77052cf640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76f80eacf0 0x7f76f8043670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7700050b10 tx=0x7f77000691a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6ec0129f0 con 0x7fa7040b3e40 2024-03-06T07:22:03.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7f9927fff640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f990c06b5a0 0x7f990c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7ff788cf4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7840cf9a0 0x7ff7841fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7ff788cf4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 0x7ff7841fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7ff788cf4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840b3f20 con 0x7ff7840cf9a0 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7ff788cf4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840b3c40 con 0x7ff784143460 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.104+0000 7ff788cf4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff784067150 con 0x7ff784066d00 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7840cf9a0 0x7ff7841fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 0x7ff7841fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 0x7ff7841fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46118/0 (socket says 172.21.15.122:46118) 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 -- 172.21.15.122:0/2553308728 learned_addr learned my addr 172.21.15.122:0/2553308728 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:03.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 -- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784066d00 msgr2=0x7ff7841317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784066d00 0x7ff7841317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 -- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7840cf9a0 msgr2=0x7ff7841fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7840cf9a0 0x7ff7841fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff783fff640 1 -- 172.21.15.122:0/2553308728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff784133030 con 0x7ff784143460 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.105+0000 7ff782ffd640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7840cf9a0 0x7ff7841fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7fa7017fa640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6e406b5a0 0x7fa6e406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9de4007020 con 0x7f9de80dfef0 2024-03-06T07:22:03.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7faa957fa640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa84073ce0 0x7faa840761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6ec089d20 con 0x7fa7040b3e40 2024-03-06T07:22:03.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7fa708ba0640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6e406b5a0 0x7fa6e406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/802124969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1804f330 con 0x7fcf180ce9e0 2024-03-06T07:22:03.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7ff783fff640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 0x7ff7841fc170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff77c02c4c0 tx=0x7ff77c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7f9927fff640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f990c06b5a0 0x7f990c06da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9920035000 tx=0x7f9920030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7f9decff9640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9dd406b8d0 0x7f9dd406dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9de40052e0 con 0x7f9de80dfef0 2024-03-06T07:22:03.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf08012b70 con 0x7fcf180ce9e0 2024-03-06T07:22:03.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa8c089590 con 0x7faa9806e830 2024-03-06T07:22:03.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7fa708ba0640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6e406b5a0 0x7fa6e406da60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc036000 tx=0x7fa6fc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7f9def7fe640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9dd406b8d0 0x7f9dd406dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7fcf0dffb640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcefc06b4d0 0x7fcefc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7fcf1ce6f640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcefc06b4d0 0x7fcefc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf0808a570 con 0x7fcf180ce9e0 2024-03-06T07:22:03.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.106+0000 7faa97fff640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa84073ce0 0x7faa840761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7faa97fff640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa84073ce0 0x7faa840761a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faa90001460 tx=0x7faa90007cd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7f9def7fe640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9dd406b8d0 0x7f9dd406dd90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9de800aff0 tx=0x7f9df00b7040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.107+0000 7fcf1ce6f640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcefc06b4d0 0x7fcefc06d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcf181fb900 tx=0x7fcf14007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.111+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf080593f0 con 0x7fcf180ce9e0 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.113+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860014070 con 0x7f8868053250 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.113+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770008f070 con 0x7f76f80eacf0 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77000933b0 con 0x7f76f80eacf0 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9de4020070 con 0x7f9de80dfef0 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886804c790 con 0x7f8868053250 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886804cd50 con 0x7f8868053250 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8860010890 con 0x7f8868053250 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860010c80 con 0x7f8868053250 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700093760 con 0x7f76f80eacf0 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f8046dd0 con 0x7f76f80eacf0 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9914058c70 con 0x7f99280b39d0 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.114+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f8047390 con 0x7f76f80eacf0 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.115+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8858005290 con 0x7f8868053250 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.115+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f817d2c0 con 0x7f76f80eacf0 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.116+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7700092070 con 0x7f76f80eacf0 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.116+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8860017070 con 0x7f8868053250 2024-03-06T07:22:03.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.116+0000 7f8866ffd640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f885406b4d0 0x7f885406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f76fd7fa640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76e406b4d0 0x7f76e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f8876152640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f885406b4d0 0x7f885406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f7704ace640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76e406b4d0 0x7f76e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8860089e60 con 0x7f8868053250 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f8876152640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f885406b4d0 0x7f885406d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f886c03e000 tx=0x7f886c038000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7700104a30 con 0x7f76f80eacf0 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.117+0000 7f7704ace640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76e406b4d0 0x7f76e406d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009560 tx=0x7f76f403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.122+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa704163560 con 0x7fa7040b3e40 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.123+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c042030 con 0x7ff784143460 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.123+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff784133310 con 0x7ff784143460 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.123+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff784131ce0 con 0x7ff784143460 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.124+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff784068f80 con 0x7ff784143460 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.126+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8860058690 con 0x7f8868053250 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.130+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c02cb70 con 0x7ff784143460 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.130+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c004580 con 0x7ff784143460 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.132+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff77c045070 con 0x7ff784143460 2024-03-06T07:22:03.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.132+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa8c004c70 con 0x7faa9806e830 2024-03-06T07:22:03.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.133+0000 7ff76bfff640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff77006b4d0 0x7ff77006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.133+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff77c0ac9a0 con 0x7ff784143460 2024-03-06T07:22:03.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.132+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77000d31c0 con 0x7f76f80eacf0 2024-03-06T07:22:03.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.133+0000 7ff7837fe640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff77006b4d0 0x7ff77006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.134+0000 7ff7837fe640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff77006b4d0 0x7ff77006d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff774006b10 tx=0x7ff774008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.135+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff77c04c070 con 0x7ff784143460 2024-03-06T07:22:03.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.502+0000 7f9df566d640 1 -- 172.21.15.122:0/2979858421 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9de80562d0 con 0x7f9de80dfef0 2024-03-06T07:22:03.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.502+0000 7f9decff9640 1 -- 172.21.15.122:0/2979858421 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9de4057ca0 con 0x7f9de80dfef0 2024-03-06T07:22:03.504 INFO:teuthology.orchestra.run.smithi122.stdout:55834574854 2024-03-06T07:22:03.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.505+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f99280747a0 con 0x7f99280b39d0 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.505+0000 7f9dda7fc640 1 -- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9dd406b8d0 msgr2=0x7f9dd406dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.505+0000 7f9dda7fc640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9dd406b8d0 0x7f9dd406dd90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9de800aff0 tx=0x7f9df00b7040 comp rx=0 tx=0).stop 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 -- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80dfef0 msgr2=0x7f9de8034ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80dfef0 0x7f9de8034ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9de400c410 tx=0x7f9de400c810 comp rx=0 tx=0).stop 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 -- 172.21.15.122:0/2979858421 shutdown_connections 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9dd406b8d0 0x7f9dd406dd90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9de8051e60 0x7f9de80d9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9de80e4100 0x7f9de8006d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 --2- 172.21.15.122:0/2979858421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9de80dfef0 0x7f9de8034ea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 -- 172.21.15.122:0/2979858421 >> 172.21.15.122:0/2979858421 conn(0x7f9de801ef40 msgr2=0x7f9de8054100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 -- 172.21.15.122:0/2979858421 shutdown_connections 2024-03-06T07:22:03.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f99257fa640 1 -- 172.21.15.122:0/285584542 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9914058570 con 0x7f99280b39d0 2024-03-06T07:22:03.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.506+0000 7f9dda7fc640 1 -- 172.21.15.122:0/2979858421 wait complete. 2024-03-06T07:22:03.509 INFO:teuthology.orchestra.run.smithi122.stdout:55834574854 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.510+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f990c06b5a0 msgr2=0x7f990c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.510+0000 7f992eb8a640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f990c06b5a0 0x7f990c06da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9920035000 tx=0x7f9920030000 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.510+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 msgr2=0x7f992815d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.510+0000 7f992eb8a640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 0x7f992815d620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f991400c430 tx=0x7f991400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 shutdown_connections 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f990c06b5a0 0x7f990c06da60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9928162550 0x7f9928145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f992815c6d0 0x7f99281f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 --2- 172.21.15.122:0/285584542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99280b39d0 0x7f992815d620 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 >> 172.21.15.122:0/285584542 conn(0x7f99280b1570 msgr2=0x7f99280cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 shutdown_connections 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7fcf1f8d4640 1 -- 172.21.15.122:0/802124969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcf180d07b0 con 0x7fcf180ce9e0 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.511+0000 7f992eb8a640 1 -- 172.21.15.122:0/285584542 wait complete. 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.512+0000 7fcf0dffb640 1 -- 172.21.15.122:0/802124969 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf080593f0 con 0x7fcf180ce9e0 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi122.stdout:55834574854 2024-03-06T07:22:03.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.515+0000 7fcefb7fe640 1 -- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcefc06b4d0 msgr2=0x7fcefc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.516+0000 7fcefb7fe640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcefc06b4d0 0x7fcefc06d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcf181fb900 tx=0x7fcf14007e30 comp rx=0 tx=0).stop 2024-03-06T07:22:03.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.516+0000 7fcefb7fe640 1 -- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 msgr2=0x7fcf181f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.516+0000 7fcefb7fe640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 0x7fcf181f7ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800d5f0 tx=0x7fcf0800dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.516+0000 7fcefb7fe640 1 -- 172.21.15.122:0/802124969 shutdown_connections 2024-03-06T07:22:03.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.516+0000 7fcefb7fe640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcefc06b4d0 0x7fcefc06d990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.517+0000 7fcefb7fe640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf1815dda0 0x7fcf18157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.517+0000 7fcefb7fe640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf180cedc0 0x7fcf18157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.517+0000 7fcefb7fe640 1 --2- 172.21.15.122:0/802124969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf180ce9e0 0x7fcf181f7ca0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.517+0000 7fcefb7fe640 1 -- 172.21.15.122:0/802124969 >> 172.21.15.122:0/802124969 conn(0x7fcf180b1c80 msgr2=0x7fcf180b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.517+0000 7fcefb7fe640 1 -- 172.21.15.122:0/802124969 shutdown_connections 2024-03-06T07:22:03.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.517+0000 7fcefb7fe640 1 -- 172.21.15.122:0/802124969 wait complete. 2024-03-06T07:22:03.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-03-06T07:22:03.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-03-06T07:22:03.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.533+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff7840cc9d0 con 0x7ff784143460 2024-03-06T07:22:03.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.533+0000 7ff76bfff640 1 -- 172.21.15.122:0/2553308728 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff77c07b9b0 con 0x7ff784143460 2024-03-06T07:22:03.536 INFO:teuthology.orchestra.run.smithi122.stdout:55834574854 2024-03-06T07:22:03.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-03-06T07:22:03.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.535+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faa98068f80 con 0x7faa9806e830 2024-03-06T07:22:03.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.535+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa7040b05d0 con 0x7fa7040b3e40 2024-03-06T07:22:03.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.535+0000 7faa957fa640 1 -- 172.21.15.122:0/1712741751 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faa8c058370 con 0x7faa9806e830 2024-03-06T07:22:03.538 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:03.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.535+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff77006b4d0 msgr2=0x7ff77006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.535+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff77006b4d0 0x7ff77006d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff774006b10 tx=0x7ff774008040 comp rx=0 tx=0).stop 2024-03-06T07:22:03.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.535+0000 7fa7017fa640 1 -- 172.21.15.122:0/2935263962 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6ec058c10 con 0x7fa7040b3e40 2024-03-06T07:22:03.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 msgr2=0x7ff7841fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 0x7ff7841fc170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff77c02c4c0 tx=0x7ff77c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:03.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 shutdown_connections 2024-03-06T07:22:03.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff77006b4d0 0x7ff77006d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.539 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff784143460 0x7ff7841fc170 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7840cf9a0 0x7ff7841fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 --2- 172.21.15.122:0/2553308728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff784066d00 0x7ff7841317a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 >> 172.21.15.122:0/2553308728 conn(0x7ff78406dd50 msgr2=0x7ff7840cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 shutdown_connections 2024-03-06T07:22:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7ff788cf4640 1 -- 172.21.15.122:0/2553308728 wait complete. 2024-03-06T07:22:03.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8858005850 con 0x7f8868053250 2024-03-06T07:22:03.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.536+0000 7f8866ffd640 1 -- 172.21.15.122:0/1413775925 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8860010e20 con 0x7f8868053250 2024-03-06T07:22:03.541 INFO:teuthology.orchestra.run.smithi122.stdout:55834574854 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.538+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa84073ce0 msgr2=0x7faa840761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.538+0000 7faa9ef81640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa84073ce0 0x7faa840761a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faa90001460 tx=0x7faa90007cd0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.538+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 msgr2=0x7faa980adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.538+0000 7faa9ef81640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 0x7faa980adf40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c780 tx=0x7faa8c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 shutdown_connections 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa84073ce0 0x7faa840761a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa9815a280 0x7faa980af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa98070740 0x7faa9806d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 --2- 172.21.15.122:0/1712741751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa9806e830 0x7faa980adf40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 >> 172.21.15.122:0/1712741751 conn(0x7faa980b16e0 msgr2=0x7faa98154010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 shutdown_connections 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7faa9ef81640 1 -- 172.21.15.122:0/1712741751 wait complete. 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6e406b5a0 msgr2=0x7fa6e406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6e406b5a0 0x7fa6e406da60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc036000 tx=0x7fa6fc030000 comp rx=0 tx=0).stop 2024-03-06T07:22:03.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 msgr2=0x7fa70415fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 0x7fa70415fbf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c450 tx=0x7fa6ec00c910 comp rx=0 tx=0).stop 2024-03-06T07:22:03.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f885406b4d0 msgr2=0x7f885406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f885406b4d0 0x7f885406d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f886c03e000 tx=0x7f886c038000 comp rx=0 tx=0).stop 2024-03-06T07:22:03.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 msgr2=0x7f886804bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 0x7f886804bb80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f886000b4d0 tx=0x7f886000b990 comp rx=0 tx=0).stop 2024-03-06T07:22:03.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 shutdown_connections 2024-03-06T07:22:03.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6e406b5a0 0x7fa6e406da60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7041620a0 0x7fa704160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7040b3e40 0x7fa70415fbf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 --2- 172.21.15.122:0/2935263962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70406c4c0 0x7fa7040ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 shutdown_connections 2024-03-06T07:22:03.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f885406b4d0 0x7f885406d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88680e1be0 0x7f886804dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8868053250 0x7f886804bb80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 --2- 172.21.15.122:0/1413775925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886800aa90 0x7f88680d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 >> 172.21.15.122:0/2935263962 conn(0x7fa7040b16e0 msgr2=0x7fa70415c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 >> 172.21.15.122:0/1413775925 conn(0x7f8868009fa0 msgr2=0x7f88680dca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 shutdown_connections 2024-03-06T07:22:03.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.539+0000 7f8877154640 1 -- 172.21.15.122:0/1413775925 wait complete. 2024-03-06T07:22:03.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.540+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 shutdown_connections 2024-03-06T07:22:03.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.540+0000 7fa709ba2640 1 -- 172.21.15.122:0/2935263962 wait complete. 2024-03-06T07:22:03.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.546+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f76f8005380 con 0x7f76f80eacf0 2024-03-06T07:22:03.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.547+0000 7f76fd7fa640 1 -- 172.21.15.122:0/4186392493 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77000d7260 con 0x7f76f80eacf0 2024-03-06T07:22:03.548 INFO:teuthology.orchestra.run.smithi122.stdout:55834574856 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76e406b4d0 msgr2=0x7f76e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76e406b4d0 0x7f76e406d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009560 tx=0x7f76f403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76f80eacf0 msgr2=0x7f76f8043670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76f80eacf0 0x7f76f8043670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7700050b10 tx=0x7f77000691a0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 shutdown_connections 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76e406b4d0 0x7f76e406d990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76f80eacf0 0x7f76f8043670 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f804ffc0 0x7f76f8036300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 --2- 172.21.15.122:0/4186392493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76f8038100 0x7f76f8049790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 >> 172.21.15.122:0/4186392493 conn(0x7f76f8009fa0 msgr2=0x7f76f8052c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 shutdown_connections 2024-03-06T07:22:03.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:03.550+0000 7f7705ad0640 1 -- 172.21.15.122:0/4186392493 wait complete. 2024-03-06T07:22:03.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-03-06T07:22:03.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-06T07:22:03.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-03-06T07:22:03.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-03-06T07:22:03.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-03-06T07:22:04.527 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:04.531 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:04.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:04.557 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:04.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:04.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:04.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:04.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:04.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.684+0000 7f390a9a0640 1 Processor -- start 2024-03-06T07:22:04.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.684+0000 7f390a9a0640 1 -- start start 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.685+0000 7f390a9a0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3904162560 0x7f3904071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.685+0000 7f390a9a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3904071670 0x7f390406cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.685+0000 7f390a9a0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3904166410 con 0x7f39040aed00 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.685+0000 7f390a9a0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3904071ae0 con 0x7f3904071670 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.685+0000 7f390a9a0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39041666f0 con 0x7f3904162560 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3904162560 0x7f3904071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3904162560 0x7f3904071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:52352/0 (socket says 172.21.15.122:52352) 2024-03-06T07:22:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 -- 172.21.15.122:0/983935492 learned_addr learned my addr 172.21.15.122:0/983935492 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390a19f640 1 --2- 172.21.15.122:0/983935492 >> v2:172.21.15.122:3300/0 conn(0x7f3904071670 0x7f390406cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 -- 172.21.15.122:0/983935492 >> v2:172.21.15.122:3300/0 conn(0x7f3904071670 msgr2=0x7f390406cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 --2- 172.21.15.122:0/983935492 >> v2:172.21.15.122:3300/0 conn(0x7f3904071670 0x7f390406cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 -- 172.21.15.122:0/983935492 >> v1:172.21.15.122:6789/0 conn(0x7f39040aed00 legacy=0x7f3904165b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:04.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.686+0000 7f390919d640 1 -- 172.21.15.122:0/983935492 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39041669d0 con 0x7f3904162560 2024-03-06T07:22:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.687+0000 7f390919d640 1 --2- 172.21.15.122:0/983935492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f3904071130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39040b31b0 tx=0x7f390002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=719243ad2cead3bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.688+0000 7f390899c640 1 -- 172.21.15.122:0/983935492 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900035020 con 0x7f3904162560 2024-03-06T07:22:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.688+0000 7f390899c640 1 -- 172.21.15.122:0/983935492 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3900031870 con 0x7f3904162560 2024-03-06T07:22:04.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.689+0000 7f390a9a0640 1 -- 172.21.15.122:0/983935492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 msgr2=0x7f3904071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.689+0000 7f390a9a0640 1 --2- 172.21.15.122:0/983935492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f3904071130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39040b31b0 tx=0x7f390002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:04.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.689+0000 7f390a9a0640 1 -- 172.21.15.122:0/983935492 shutdown_connections 2024-03-06T07:22:04.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.689+0000 7f390a9a0640 1 --2- 172.21.15.122:0/983935492 >> v2:172.21.15.122:3300/0 conn(0x7f3904071670 0x7f390406cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.689+0000 7f390a9a0640 1 --2- 172.21.15.122:0/983935492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f3904071130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.689+0000 7f390a9a0640 1 -- 172.21.15.122:0/983935492 >> 172.21.15.122:0/983935492 conn(0x7f39040b16e0 msgr2=0x7f39040b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.690+0000 7f390a9a0640 1 -- 172.21.15.122:0/983935492 shutdown_connections 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.690+0000 7f390a9a0640 1 -- 172.21.15.122:0/983935492 wait complete. 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.692+0000 7f390a9a0640 1 Processor -- start 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.692+0000 7f390a9a0640 1 -- start start 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a9a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3904071670 0x7f390420bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a9a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f39040aed00 0x7f39040ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a9a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f39040aba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390919d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f39040aed00 0x7f39040ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390919d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f39040aed00 0x7f39040ab550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45580/0 (socket says 172.21.15.122:45580) 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390919d640 1 -- 172.21.15.122:0/4022911254 learned_addr learned my addr 172.21.15.122:0/4022911254 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a19f640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f39040aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39041636a0 con 0x7f3904071670 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390406c840 con 0x7f3904162560 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.693+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040af0e0 con 0x7f39040aed00 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.694+0000 7f390a19f640 1 -- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f39040aed00 msgr2=0x7f39040ab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.694+0000 7f390a19f640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f39040aed00 0x7f39040ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.694+0000 7f390a19f640 1 -- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3904071670 msgr2=0x7f390420bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.694+0000 7f390a19f640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3904071670 0x7f390420bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.694+0000 7f390a19f640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904165f60 con 0x7f3904162560 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.694+0000 7f390a19f640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f39040aba90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00efc0 tx=0x7f38fc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.695+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc018020 con 0x7f3904162560 2024-03-06T07:22:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.696+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904164e10 con 0x7f3904162560 2024-03-06T07:22:04.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.696+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39040a9450 con 0x7f3904162560 2024-03-06T07:22:04.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.699+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc014480 con 0x7f3904162560 2024-03-06T07:22:04.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.699+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc014870 con 0x7f3904162560 2024-03-06T07:22:04.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.699+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38fc005d10 con 0x7f3904162560 2024-03-06T07:22:04.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.699+0000 7f38fa7fc640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38e806b5d0 0x7f38e806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.700+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38fc08afd0 con 0x7f3904162560 2024-03-06T07:22:04.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.700+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3904068700 con 0x7f3904162560 2024-03-06T07:22:04.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.701+0000 7f390999e640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38e806b5d0 0x7f38e806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.702+0000 7f390999e640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38e806b5d0 0x7f38e806da90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39040b0f70 tx=0x7f38f4007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.705+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38fc022030 con 0x7f3904162560 2024-03-06T07:22:04.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.742+0000 7f3d3815e640 1 Processor -- start 2024-03-06T07:22:04.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.742+0000 7f3d3815e640 1 -- start start 2024-03-06T07:22:04.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.742+0000 7efc36fc8640 1 Processor -- start 2024-03-06T07:22:04.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7efc36fc8640 1 -- start start 2024-03-06T07:22:04.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f11e8bc7640 1 Processor -- start 2024-03-06T07:22:04.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f11e8bc7640 1 -- start start 2024-03-06T07:22:04.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d3815e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3d300b1ff0 0x7f3d30159be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d3815e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d300ccd60 0x7f3d300b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d3815e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300af200 con 0x7f3d30071530 2024-03-06T07:22:04.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d3815e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d30070b00 con 0x7f3d300b1ff0 2024-03-06T07:22:04.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d3815e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3006c9c0 con 0x7f3d300ccd60 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d35efa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3d300b1ff0 0x7f3d30159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d356f9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d300ccd60 0x7f3d300b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d35efa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3d300b1ff0 0x7f3d30159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56946/0 (socket says 172.21.15.122:56946) 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d356f9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d300ccd60 0x7f3d300b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45582/0 (socket says 172.21.15.122:45582) 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.743+0000 7f3d35efa640 1 -- 172.21.15.122:0/515153857 learned_addr learned my addr 172.21.15.122:0/515153857 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f3d35efa640 1 -- 172.21.15.122:0/515153857 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d300ccd60 msgr2=0x7f3d300b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f3d35efa640 1 --2- 172.21.15.122:0/515153857 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d300ccd60 0x7f3d300b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f3d34ef8640 1 -- 172.21.15.122:0/515153857 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285218631 0 0) 0x7f3d300af200 con 0x7f3d30071530 2024-03-06T07:22:04.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f3d35efa640 1 -- 172.21.15.122:0/515153857 >> v1:172.21.15.122:6789/0 conn(0x7f3d30071530 legacy=0x7f3d3006d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7efc36fc8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc300ccad0 0x7efc300ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7efc36fc8640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300cead0 0x7efc300cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7efc36fc8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3006ca80 con 0x7efc30145190 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7efc36fc8640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3006e230 con 0x7efc300ccad0 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7efc36fc8640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3006c830 con 0x7efc300cead0 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7efc35fc6640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efc30145190 0x7efc3006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38524/0 (socket says 172.21.15.122:38524) 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7efc35fc6640 1 -- 172.21.15.122:0/3471009846 learned_addr learned my addr 172.21.15.122:0/3471009846 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7efc357c5640 1 --2- 172.21.15.122:0/3471009846 >> v2:172.21.15.122:3300/0 conn(0x7efc300ccad0 0x7efc300ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f2dd5302640 1 Processor -- start 2024-03-06T07:22:04.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7fb1cffff640 1 Processor -- start 2024-03-06T07:22:04.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7fb1cffff640 1 -- start start 2024-03-06T07:22:04.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f3d35efa640 1 -- 172.21.15.122:0/515153857 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3006cca0 con 0x7f3d300b1ff0 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7efc34fc4640 1 -- 172.21.15.122:0/3471009846 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098929813 0 0) 0x7efc3006ca80 con 0x7efc30145190 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7efc34fc4640 1 -- 172.21.15.122:0/3471009846 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c003520 con 0x7efc30145190 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7efc367c7640 1 --2- 172.21.15.122:0/3471009846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300cead0 0x7efc300cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.744+0000 7f2dd5302640 1 -- start start 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f2dd5302640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd006d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f2dd5302640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2dd006da60 0x7f2dd0162a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dd5302640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd006dea0 con 0x7f2dd015ff70 2024-03-06T07:22:04.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dd5302640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd00717e0 con 0x7f2dd006da60 2024-03-06T07:22:04.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dd5302640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd006e170 con 0x7f2dd006d000 2024-03-06T07:22:04.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f3d35efa640 1 --2- 172.21.15.122:0/515153857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d300b1ff0 0x7f3d30159be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b87ffd78d3b98af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f65b7fff640 1 Processor -- start 2024-03-06T07:22:04.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f65b7fff640 1 -- start start 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dce7fc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dce7fc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd006d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45606/0 (socket says 172.21.15.122:45606) 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dce7fc640 1 -- 172.21.15.122:0/2489507188 learned_addr learned my addr 172.21.15.122:0/2489507188 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 0x7fb1d01632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cffff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d00b28a0 con 0x7fb1d0072520 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cffff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d00709e0 con 0x7fb1d015c710 2024-03-06T07:22:04.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cffff640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00b3770 con 0x7fb1d00cf960 2024-03-06T07:22:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dcf7fe640 1 --2- 172.21.15.122:0/2489507188 >> v2:172.21.15.169:3300/0 conn(0x7f2dd006da60 0x7f2dd0162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f3d34ef8640 1 -- 172.21.15.122:0/515153857 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20034070 con 0x7f3d300b1ff0 2024-03-06T07:22:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7efc367c7640 1 -- 172.21.15.122:0/3471009846 >> v2:172.21.15.122:3300/0 conn(0x7efc300ccad0 msgr2=0x7efc300ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7efc367c7640 1 --2- 172.21.15.122:0/3471009846 >> v2:172.21.15.122:3300/0 conn(0x7efc300ccad0 0x7efc300ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7efc367c7640 1 -- 172.21.15.122:0/3471009846 >> v1:172.21.15.122:6789/0 conn(0x7efc30145190 legacy=0x7efc3006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7efc367c7640 1 -- 172.21.15.122:0/3471009846 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3006cd60 con 0x7efc300cead0 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7efc367c7640 1 --2- 172.21.15.122:0/3471009846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300cead0 0x7efc300cef00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0097a0 tx=0x7efc2c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1447dd3c22fbfdcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7efc34fc4640 1 -- 172.21.15.122:0/3471009846 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c035020 con 0x7efc300cead0 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cf7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 0x7fb1d01632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cf7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 0x7fb1d01632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45614/0 (socket says 172.21.15.122:45614) 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7fb1cf7fe640 1 -- 172.21.15.122:0/2306747196 learned_addr learned my addr 172.21.15.122:0/2306747196 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7fb1cf7fe640 1 -- 172.21.15.122:0/2306747196 >> v1:172.21.15.169:6789/0 conn(0x7fb1d015c710 legacy=0x7fb1d00b3060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7fb1cf7fe640 1 -- 172.21.15.122:0/2306747196 >> v1:172.21.15.122:6789/0 conn(0x7fb1d0072520 legacy=0x7fb1d00cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7fb1cf7fe640 1 -- 172.21.15.122:0/2306747196 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0070cc0 con 0x7fb1d00cf960 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f11e8bc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 0x7f11dc0e3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f11e8bc7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f11dc0e40f0 0x7f11dc0539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f11e8bc7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc056580 con 0x7f11dc0dfee0 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f11e8bc7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11dc0e4540 con 0x7f11dc0e40f0 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f11e8bc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc0094a0 con 0x7f11dc0e02c0 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.745+0000 7f11e1d9b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 0x7f11dc0e3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f11e1d9b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 0x7f11dc0e3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45604/0 (socket says 172.21.15.122:45604) 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f11e1d9b640 1 -- 172.21.15.122:0/834766561 learned_addr learned my addr 172.21.15.122:0/834766561 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f11e1d9b640 1 -- 172.21.15.122:0/834766561 >> v2:172.21.15.122:3300/0 conn(0x7f11dc0e40f0 msgr2=0x7f11dc0539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f11e1d9b640 1 --2- 172.21.15.122:0/834766561 >> v2:172.21.15.122:3300/0 conn(0x7f11dc0e40f0 0x7f11dc0539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f11e1d9b640 1 -- 172.21.15.122:0/834766561 >> v1:172.21.15.122:6789/0 conn(0x7f11dc0dfee0 legacy=0x7f11dc0085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f11e1d9b640 1 -- 172.21.15.122:0/834766561 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc056860 con 0x7f11dc0e02c0 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 0x7f65b0055400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f65b00e7a20 0x7f65b00534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b7fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f65b0053a30 0x7f65b00eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b7fff640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b0053ea0 con 0x7f65b00e7a20 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b7fff640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b00e7e80 con 0x7f65b0053a30 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b7fff640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0056c80 con 0x7f65b00e7640 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b6ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 0x7f65b0055400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b6ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 0x7f65b0055400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45622/0 (socket says 172.21.15.122:45622) 2024-03-06T07:22:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b6ffd640 1 -- 172.21.15.122:0/522692220 learned_addr learned my addr 172.21.15.122:0/522692220 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b77fe640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.169:3300/0 conn(0x7f65b0053a30 0x7f65b00eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f3d34ef8640 1 -- 172.21.15.122:0/515153857 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d20030860 con 0x7f3d300b1ff0 2024-03-06T07:22:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 -- 172.21.15.122:0/515153857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d300b1ff0 msgr2=0x7f3d30159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d34ef8640 1 -- 172.21.15.122:0/515153857 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20034070 con 0x7f3d300b1ff0 2024-03-06T07:22:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 --2- 172.21.15.122:0/515153857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d300b1ff0 0x7f3d30159be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7efc34fc4640 1 -- 172.21.15.122:0/3471009846 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c031810 con 0x7efc300cead0 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 -- 172.21.15.122:0/515153857 shutdown_connections 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 --2- 172.21.15.122:0/515153857 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d300ccd60 0x7f3d300b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 --2- 172.21.15.122:0/515153857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d300b1ff0 0x7f3d30159be0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 -- 172.21.15.122:0/515153857 >> 172.21.15.122:0/515153857 conn(0x7f3d300b1500 msgr2=0x7f3d300b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 msgr2=0x7f2dd006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dcf7fe640 1 --2- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd006d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2489507188 >> v1:172.21.15.122:6789/0 conn(0x7f2dd015ff70 legacy=0x7f2dd006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.746+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2489507188 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0165ff0 con 0x7f2dd006da60 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b6ffd640 1 -- 172.21.15.122:0/522692220 >> v2:172.21.15.169:3300/0 conn(0x7f65b0053a30 msgr2=0x7f65b00eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b67fc640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.122:3300/0 conn(0x7f65b00e7a20 0x7f65b00534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b6ffd640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.169:3300/0 conn(0x7f65b0053a30 0x7f65b00eb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f65b6ffd640 1 -- 172.21.15.122:0/522692220 >> v2:172.21.15.122:3300/0 conn(0x7f65b00e7a20 msgr2=0x7f65b00534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.747+0000 7f3d3815e640 1 -- 172.21.15.122:0/515153857 shutdown_connections 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc34fc4640 1 -- 172.21.15.122:0/3471009846 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c031bc0 con 0x7efc300cead0 2024-03-06T07:22:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f11e1d9b640 1 --2- 172.21.15.122:0/834766561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 0x7f11dc0e3bb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11d8013150 tx=0x7f11d80341b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3324c1e89806151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc36fc8640 1 -- 172.21.15.122:0/3471009846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300cead0 msgr2=0x7efc300cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3471009846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300cead0 0x7efc300cef00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0097a0 tx=0x7efc2c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f65b6ffd640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.122:3300/0 conn(0x7f65b00e7a20 0x7f65b00534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f65b6ffd640 1 -- 172.21.15.122:0/522692220 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b0057f70 con 0x7f65b00e7640 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f3d3815e640 1 -- 172.21.15.122:0/515153857 wait complete. 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc36fc8640 1 -- 172.21.15.122:0/3471009846 shutdown_connections 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3471009846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300cead0 0x7efc300cef00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3471009846 >> v2:172.21.15.122:3300/0 conn(0x7efc300ccad0 0x7efc300ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7efc36fc8640 1 -- 172.21.15.122:0/3471009846 >> 172.21.15.122:0/3471009846 conn(0x7efc300b1500 msgr2=0x7efc300b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7fb1cf7fe640 1 --2- 172.21.15.122:0/2306747196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 0x7fb1d01632e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1c00098a0 tx=0x7fb1c002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f25219988ed83a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f11e159a640 1 -- 172.21.15.122:0/834766561 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d80110d0 con 0x7f11dc0e02c0 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f11e159a640 1 -- 172.21.15.122:0/834766561 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8011270 con 0x7f11dc0e02c0 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f11e159a640 1 -- 172.21.15.122:0/834766561 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8011640 con 0x7f11dc0e02c0 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f65b77fe640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.169:3300/0 conn(0x7f65b0053a30 0x7f65b00eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f65b67fc640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.122:3300/0 conn(0x7f65b00e7a20 0x7f65b00534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:04.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.748+0000 7f2dcf7fe640 1 --2- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd006da60 0x7f2dd0162a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2dc00097a0 tx=0x7f2dc002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=607227dc4b66c4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f2dcdffb640 1 -- 172.21.15.122:0/2489507188 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0039070 con 0x7f2dd006da60 2024-03-06T07:22:04.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f2dcdffb640 1 -- 172.21.15.122:0/2489507188 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc00316f0 con 0x7f2dd006da60 2024-03-06T07:22:04.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f2dcdffb640 1 -- 172.21.15.122:0/2489507188 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0031ac0 con 0x7f2dd006da60 2024-03-06T07:22:04.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f65b6ffd640 1 --2- 172.21.15.122:0/522692220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 0x7f65b0055400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65b0011cf0 tx=0x7f65a802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97d4c9b0c6ba4773 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7efc36fc8640 1 -- 172.21.15.122:0/3471009846 shutdown_connections 2024-03-06T07:22:04.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7efc36fc8640 1 -- 172.21.15.122:0/3471009846 wait complete. 2024-03-06T07:22:04.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f3d3815e640 1 Processor -- start 2024-03-06T07:22:04.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7fb1cdffb640 1 -- 172.21.15.122:0/2306747196 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0035020 con 0x7fb1d00cf960 2024-03-06T07:22:04.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f3d3815e640 1 -- start start 2024-03-06T07:22:04.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7fb1cdffb640 1 -- 172.21.15.122:0/2306747196 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c0031640 con 0x7fb1d00cf960 2024-03-06T07:22:04.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7fb1cdffb640 1 -- 172.21.15.122:0/2306747196 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c002d040 con 0x7fb1d00cf960 2024-03-06T07:22:04.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f11e8bc7640 1 -- 172.21.15.122:0/834766561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 msgr2=0x7f11dc0e3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/834766561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 0x7f11dc0e3bb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11d8013150 tx=0x7f11d80341b0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.749+0000 7fdd47565640 1 Processor -- start 2024-03-06T07:22:04.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7fdd47565640 1 -- start start 2024-03-06T07:22:04.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 -- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd006da60 msgr2=0x7f2dd0162a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd006da60 0x7f2dd0162a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2dc00097a0 tx=0x7f2dc002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 -- 172.21.15.122:0/2489507188 shutdown_connections 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd006da60 0x7f2dd0162a10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2489507188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 -- 172.21.15.122:0/2489507188 >> 172.21.15.122:0/2489507188 conn(0x7f2dd00b1c80 msgr2=0x7f2dd00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f11e8bc7640 1 -- 172.21.15.122:0/834766561 shutdown_connections 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/834766561 >> v2:172.21.15.122:3300/0 conn(0x7f11dc0e40f0 0x7f11dc0539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/834766561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e02c0 0x7f11dc0e3bb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f11e8bc7640 1 -- 172.21.15.122:0/834766561 >> 172.21.15.122:0/834766561 conn(0x7f11dc01ef30 msgr2=0x7f11dc055c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7efc36fc8640 1 Processor -- start 2024-03-06T07:22:04.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7efc36fc8640 1 -- start start 2024-03-06T07:22:04.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f65b5ffb640 1 -- 172.21.15.122:0/522692220 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8034070 con 0x7f65b00e7640 2024-03-06T07:22:04.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f65b5ffb640 1 -- 172.21.15.122:0/522692220 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8030610 con 0x7f65b00e7640 2024-03-06T07:22:04.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 -- 172.21.15.122:0/2489507188 shutdown_connections 2024-03-06T07:22:04.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.750+0000 7f2dd5302640 1 -- 172.21.15.122:0/2489507188 wait complete. 2024-03-06T07:22:04.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f3d3815e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 0x7f3d30158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b5ffb640 1 -- 172.21.15.122:0/522692220 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8034070 con 0x7f65b00e7640 2024-03-06T07:22:04.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fb1cffff640 1 -- 172.21.15.122:0/2306747196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 msgr2=0x7fb1d01632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fb1cffff640 1 --2- 172.21.15.122:0/2306747196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 0x7fb1d01632e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1c00098a0 tx=0x7fb1c002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f3d3815e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d300ccd60 0x7f3d3014b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd47565640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdd380e6d00 0x7fdd38056370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd47565640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380552d0 0x7fdd38007900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd47565640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd38006b40 con 0x7fdd380e6920 2024-03-06T07:22:04.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd47565640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd38006e20 con 0x7fdd380e6d00 2024-03-06T07:22:04.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd47565640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38007100 con 0x7fdd380552d0 2024-03-06T07:22:04.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f11e8bc7640 1 -- 172.21.15.122:0/834766561 shutdown_connections 2024-03-06T07:22:04.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd46563640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdd380e6920 0x7fdd38054bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38542/0 (socket says 172.21.15.122:38542) 2024-03-06T07:22:04.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd46563640 1 -- 172.21.15.122:0/3421226317 learned_addr learned my addr 172.21.15.122:0/3421226317 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:04.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f11e8bc7640 1 -- 172.21.15.122:0/834766561 wait complete. 2024-03-06T07:22:04.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f3d35efa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 0x7f3d30158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7fdd46d64640 1 --2- 172.21.15.122:0/3421226317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380552d0 0x7fdd38007900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f3d35efa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 0x7f3d30158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56982/0 (socket says 172.21.15.122:56982) 2024-03-06T07:22:04.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7efc36fc8640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 0x7efc30135800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 -- 172.21.15.122:0/522692220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 msgr2=0x7f65b0055400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 --2- 172.21.15.122:0/522692220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 0x7f65b0055400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65b0011cf0 tx=0x7f65a802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:04.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 -- 172.21.15.122:0/522692220 shutdown_connections 2024-03-06T07:22:04.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.122:3300/0 conn(0x7f65b00e7a20 0x7f65b00534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 --2- 172.21.15.122:0/522692220 >> v2:172.21.15.169:3300/0 conn(0x7f65b0053a30 0x7f65b00eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 --2- 172.21.15.122:0/522692220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b00e7640 0x7f65b0055400 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65b0011cf0 tx=0x7f65a802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:04.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7efc36fc8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc300cead0 0x7efc300cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f65b7fff640 1 -- 172.21.15.122:0/522692220 >> 172.21.15.122:0/522692220 conn(0x7f65b003aa20 msgr2=0x7f65b003ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f3d35efa640 1 -- 172.21.15.122:0/3408675950 learned_addr learned my addr 172.21.15.122:0/3408675950 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.751+0000 7f3d356f9640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d300ccd60 0x7f3d3014b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 0x7efc30135800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc36fc8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc30145190 0x7efc30134380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc36fc8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3006d600 con 0x7efc30145190 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc36fc8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3006d320 con 0x7efc300cead0 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc36fc8640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc300cf440 con 0x7efc300ccad0 2024-03-06T07:22:04.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 0x7efc30135800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45648/0 (socket says 172.21.15.122:45648) 2024-03-06T07:22:04.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 -- 172.21.15.122:0/3736031877 learned_addr learned my addr 172.21.15.122:0/3736031877 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d3014d2e0 0x7f3d300b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc357c5640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc300cead0 0x7efc300cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc367c7640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc30145190 0x7efc30134380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300b4ab0 con 0x7f3d30071530 2024-03-06T07:22:04.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300cd1a0 con 0x7f3d300ccd60 2024-03-06T07:22:04.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fdd46d64640 1 -- 172.21.15.122:0/3421226317 >> v2:172.21.15.169:3300/0 conn(0x7fdd380e6d00 msgr2=0x7fdd38056370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:04.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fdd46d64640 1 --2- 172.21.15.122:0/3421226317 >> v2:172.21.15.169:3300/0 conn(0x7fdd380e6d00 0x7fdd38056370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fdd46d64640 1 -- 172.21.15.122:0/3421226317 >> v1:172.21.15.122:6789/0 conn(0x7fdd380e6920 legacy=0x7fdd38054bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fdd46d64640 1 -- 172.21.15.122:0/3421226317 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380073e0 con 0x7fdd380552d0 2024-03-06T07:22:04.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f2dd5302640 1 Processor -- start 2024-03-06T07:22:04.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f3d366fb640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d3014d2e0 0x7f3d300b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30070740 con 0x7f3d3014d2e0 2024-03-06T07:22:04.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 -- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc300cead0 msgr2=0x7efc300cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc300cead0 0x7efc300cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 -- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc30145190 msgr2=0x7efc30134380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc30145190 0x7efc30134380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fb1cffff640 1 -- 172.21.15.122:0/2306747196 shutdown_connections 2024-03-06T07:22:04.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7efc35fc6640 1 -- 172.21.15.122:0/3736031877 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30135d40 con 0x7efc300ccad0 2024-03-06T07:22:04.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7efc367c7640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc30145190 0x7efc30134380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:04.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fb1cffff640 1 --2- 172.21.15.122:0/2306747196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d00cf960 0x7fb1d01632e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7fb1cffff640 1 -- 172.21.15.122:0/2306747196 >> 172.21.15.122:0/2306747196 conn(0x7fb1d006afb0 msgr2=0x7fb1d006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7efc35fc6640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 0x7efc30135800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc280021c0 tx=0x7efc280021f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f3d366fb640 1 -- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d300ccd60 msgr2=0x7f3d3014b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7fdd46d64640 1 --2- 172.21.15.122:0/3421226317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380552d0 0x7fdd38007900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd300097a0 tx=0x7fdd3002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c6e887ec7ceddb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f3d366fb640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d300ccd60 0x7f3d3014b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f65b7fff640 1 -- 172.21.15.122:0/522692220 shutdown_connections 2024-03-06T07:22:04.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc28020070 con 0x7efc300ccad0 2024-03-06T07:22:04.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc301332e0 con 0x7efc300ccad0 2024-03-06T07:22:04.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc280102d0 con 0x7efc300ccad0 2024-03-06T07:22:04.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f65b7fff640 1 -- 172.21.15.122:0/522692220 wait complete. 2024-03-06T07:22:04.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2800e640 con 0x7efc300ccad0 2024-03-06T07:22:04.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7fb1cffff640 1 -- 172.21.15.122:0/2306747196 shutdown_connections 2024-03-06T07:22:04.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7fdd45561640 1 -- 172.21.15.122:0/3421226317 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30037070 con 0x7fdd380552d0 2024-03-06T07:22:04.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7fdd45561640 1 -- 172.21.15.122:0/3421226317 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd30030710 con 0x7fdd380552d0 2024-03-06T07:22:04.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7fdd45561640 1 -- 172.21.15.122:0/3421226317 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30030ae0 con 0x7fdd380552d0 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.752+0000 7f2dd5302640 1 -- start start 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f2dd5302640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd00b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f2dd5302640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dd006da60 0x7f2dd0147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f2dd5302640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 0x7f2dd014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f2dd5302640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd0160350 con 0x7f2dd006da60 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f2dd5302640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd01665b0 con 0x7f2dd015ff70 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f2dd5302640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd0166db0 con 0x7f2dd006d000 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f2dcf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 0x7f2dd014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f3d366fb640 1 -- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 msgr2=0x7f3d30158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f3d366fb640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 0x7f3d30158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f3d366fb640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30159280 con 0x7f3d3014d2e0 2024-03-06T07:22:04.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f3d35efa640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 0x7f3d30158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:04.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f3d366fb640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d3014d2e0 0x7f3d300b0710 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00c3e0 tx=0x7f3d2c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f2dcf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 0x7f2dd014da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:52422/0 (socket says 172.21.15.122:52422) 2024-03-06T07:22:04.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c017070 con 0x7f3d3014d2e0 2024-03-06T07:22:04.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2972934526 learned_addr learned my addr 172.21.15.122:0/2972934526 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f2dceffd640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd00b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d30153660 con 0x7f3d3014d2e0 2024-03-06T07:22:04.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d30153c20 con 0x7f3d3014d2e0 2024-03-06T07:22:04.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc301338a0 con 0x7efc300ccad0 2024-03-06T07:22:04.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7fb1cffff640 1 -- 172.21.15.122:0/2306747196 wait complete. 2024-03-06T07:22:04.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7fb1cffff640 1 Processor -- start 2024-03-06T07:22:04.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f11e8bc7640 1 Processor -- start 2024-03-06T07:22:04.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.753+0000 7f11e8bc7640 1 -- start start 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f11e8bc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 0x7f11dc035310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e8bc7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11dc0e40f0 0x7f11dc0e2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e8bc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e10c0 0x7f11dc051b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e8bc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc056b40 con 0x7f11dc0dfee0 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 0x7f11dc035310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 0x7f11dc035310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57018/0 (socket says 172.21.15.122:57018) 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 -- 172.21.15.122:0/3706538818 learned_addr learned my addr 172.21.15.122:0/3706538818 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc009130 con 0x7f11dc0e40f0 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc0e4900 con 0x7f11dc0e10c0 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e1d9b640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11dc0e40f0 0x7f11dc0e2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e2d9d640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e10c0 0x7f11dc051b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 -- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e10c0 msgr2=0x7f11dc051b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c0052f0 con 0x7f3d3014d2e0 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c003850 con 0x7f3d3014d2e0 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7fdd47565640 1 -- 172.21.15.122:0/3421226317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380552d0 msgr2=0x7fdd38007900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7fdd47565640 1 --2- 172.21.15.122:0/3421226317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380552d0 0x7fdd38007900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd300097a0 tx=0x7fdd3002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7fdd47565640 1 -- 172.21.15.122:0/3421226317 shutdown_connections 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7fdd47565640 1 --2- 172.21.15.122:0/3421226317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380552d0 0x7fdd38007900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7fdd47565640 1 --2- 172.21.15.122:0/3421226317 >> v2:172.21.15.169:3300/0 conn(0x7fdd380e6d00 0x7fdd38056370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.754+0000 7fdd47565640 1 -- 172.21.15.122:0/3421226317 >> 172.21.15.122:0/3421226317 conn(0x7fdd3801ef30 msgr2=0x7fdd38055c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7fb1cffff640 1 -- start start 2024-03-06T07:22:04.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e10c0 0x7f11dc051b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 -- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11dc0e40f0 msgr2=0x7f11dc0e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11dc0e40f0 0x7f11dc0e2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e259c640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0e1680 con 0x7f11dc0dfee0 2024-03-06T07:22:04.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f65b7fff640 1 Processor -- start 2024-03-06T07:22:04.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f65b7fff640 1 -- start start 2024-03-06T07:22:04.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e1d9b640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11dc0e40f0 0x7f11dc0e2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:04.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f11e2d9d640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e10c0 0x7f11dc051b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:04.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f11e259c640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 0x7f11dc035310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c460 tx=0x7f11d400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc30068f80 con 0x7efc300ccad0 2024-03-06T07:22:04.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4016020 con 0x7f11dc0dfee0 2024-03-06T07:22:04.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d4012870 con 0x7f11dc0dfee0 2024-03-06T07:22:04.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc0e1960 con 0x7f11dc0dfee0 2024-03-06T07:22:04.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 0x7f65b00d2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65b00e7a20 0x7f65b0051430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7fdd47565640 1 -- 172.21.15.122:0/3421226317 shutdown_connections 2024-03-06T07:22:04.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65b00e8590 0x7f65b00e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b7fff640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00571b0 con 0x7f65b00e8590 2024-03-06T07:22:04.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b7fff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0058ce0 con 0x7f65b00e7a20 2024-03-06T07:22:04.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b7fff640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0057490 con 0x7f65b0053a30 2024-03-06T07:22:04.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b6ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 0x7f65b00d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b6ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 0x7f65b00d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45676/0 (socket says 172.21.15.122:45676) 2024-03-06T07:22:04.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b6ffd640 1 -- 172.21.15.122:0/1125089413 learned_addr learned my addr 172.21.15.122:0/1125089413 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b67fc640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65b00e7a20 0x7f65b0051430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7fdd47565640 1 -- 172.21.15.122:0/3421226317 wait complete. 2024-03-06T07:22:04.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b77fe640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65b00e8590 0x7f65b00e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b67fc640 1 -- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 msgr2=0x7f65b00d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.755+0000 7f2dce7fc640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dd006da60 0x7f2dd0147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 msgr2=0x7f2dd00b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f2dcf7fe640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd00b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dd006da60 msgr2=0x7f2dd0147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f2dcf7fe640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dd006da60 0x7f2dd0147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f2dcf7fe640 1 -- 172.21.15.122:0/2972934526 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd01487f0 con 0x7f2dd015ff70 2024-03-06T07:22:04.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc28023070 con 0x7efc300ccad0 2024-03-06T07:22:04.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f2dce7fc640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dd006da60 0x7f2dd0147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:04.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d0072520 0x7fb1d0151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 0x7fb1d00b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b67fc640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 0x7f65b00d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d015c710 0x7fb1d01447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b67fc640 1 -- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65b00e8590 msgr2=0x7f65b00e8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1ce7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 0x7fb1d00b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1ce7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 0x7fb1d00b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:52446/0 (socket says 172.21.15.122:52446) 2024-03-06T07:22:04.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1ce7fc640 1 -- 172.21.15.122:0/4271889355 learned_addr learned my addr 172.21.15.122:0/4271889355 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cf7fe640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d015c710 0x7fb1d01447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d0071120 con 0x7fb1d0072520 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4012c60 con 0x7f11dc0dfee0 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc0e1f20 con 0x7f11dc0dfee0 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b67fc640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65b00e8590 0x7f65b00e8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.756+0000 7f65b67fc640 1 -- 172.21.15.122:0/1125089413 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00ea420 con 0x7f65b00e7a20 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f65b6ffd640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 0x7f65b00d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f65b67fc640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65b00e7a20 0x7f65b0051430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65b8050c30 tx=0x7f65b80683c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8073500 con 0x7f65b00e7a20 2024-03-06T07:22:04.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b8093220 con 0x7f65b00e7a20 2024-03-06T07:22:04.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7efc267fc640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc1406b4d0 0x7efc1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d0070fa0 con 0x7fb1d00cf960 2024-03-06T07:22:04.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d015caf0 con 0x7fb1d015c710 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f2dcf7fe640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 0x7f2dd014da00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2dc002b7d0 tx=0x7f2dc00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0043070 con 0x7f2dd015ff70 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0034480 con 0x7f2dd015ff70 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.757+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0039040 con 0x7f2dd015ff70 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7fb1ce7fc640 1 -- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d015c710 msgr2=0x7fb1d01447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7fb1ce7fc640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d015c710 0x7fb1d01447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7fb1ce7fc640 1 -- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d0072520 msgr2=0x7fb1d0151fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:04.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7fb1ceffd640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d0072520 0x7fb1d0151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd0148ad0 con 0x7f2dd015ff70 2024-03-06T07:22:04.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7fdd47565640 1 Processor -- start 2024-03-06T07:22:04.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd0149090 con 0x7f2dd015ff70 2024-03-06T07:22:04.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7efc357c5640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc1406b4d0 0x7efc1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b80935f0 con 0x7f65b00e7a20 2024-03-06T07:22:04.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b00ea700 con 0x7f65b00e7a20 2024-03-06T07:22:04.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b00eacc0 con 0x7f65b00e7a20 2024-03-06T07:22:04.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7fdd47565640 1 -- start start 2024-03-06T07:22:04.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d18005290 con 0x7f3d3014d2e0 2024-03-06T07:22:04.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7efc357c5640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc1406b4d0 0x7efc1406d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc3015ebf0 tx=0x7efc20009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd47565640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd380552d0 0x7fdd380dd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd47565640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 0x7fdd380e3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.758+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11d4019020 con 0x7f11dc0dfee0 2024-03-06T07:22:04.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd47565640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380e6d00 0x7fdd380e3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7f11d2ffd640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f11c406b4d0 0x7f11c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11d4089fa0 con 0x7f11dc0dfee0 2024-03-06T07:22:04.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc0057d0 con 0x7f11dc0dfee0 2024-03-06T07:22:04.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fb1ce7fc640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d0072520 0x7fb1d0151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fb1ce7fc640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d00b1290 con 0x7fb1d00cf960 2024-03-06T07:22:04.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc280922b0 con 0x7efc300ccad0 2024-03-06T07:22:04.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd47565640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd380558d0 con 0x7fdd380e6920 2024-03-06T07:22:04.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fb1cf7fe640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d015c710 0x7fb1d01447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:04.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fb1ceffd640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d0072520 0x7fb1d0151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:04.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd47565640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3803b790 con 0x7fdd380552d0 2024-03-06T07:22:04.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd47565640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38055a50 con 0x7fdd380e6d00 2024-03-06T07:22:04.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fb1ce7fc640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 0x7fb1d00b0d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800c7d0 tx=0x7fb1c800cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.759+0000 7fdd46d64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380e6d00 0x7fdd380e3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd46563640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd380552d0 0x7fdd380dd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 0x7fdd380e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 0x7fdd380e3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57038/0 (socket says 172.21.15.122:57038) 2024-03-06T07:22:04.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 -- 172.21.15.122:0/2153622578 learned_addr learned my addr 172.21.15.122:0/2153622578 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:04.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b0005380 con 0x7f65b00e7a20 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 -- 172.21.15.122:0/2153622578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380e6d00 msgr2=0x7fdd380e3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380e6d00 0x7fdd380e3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 -- 172.21.15.122:0/2153622578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd380552d0 msgr2=0x7fdd380dd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd380552d0 0x7fdd380dd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 -- 172.21.15.122:0/2153622578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3804fc80 con 0x7fdd380e6920 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd004f330 con 0x7f2dd015ff70 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dc0046070 con 0x7f2dd015ff70 2024-03-06T07:22:04.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7f11e1d9b640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f11c406b4d0 0x7f11c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7fdd45d62640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 0x7fdd380e3910 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00b780 tx=0x7fdd3c00bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.760+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d2c01a070 con 0x7f3d3014d2e0 2024-03-06T07:22:04.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f2db77fe640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dbc06b5a0 0x7f2dbc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f11e1d9b640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f11c406b4d0 0x7f11c406d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f11d803d000 tx=0x7f11d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f3d267fc640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1406b520 0x7f3d1406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8017070 con 0x7fb1d00cf960 2024-03-06T07:22:04.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d0151cf0 con 0x7fb1d00cf960 2024-03-06T07:22:04.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d2c0894c0 con 0x7f3d3014d2e0 2024-03-06T07:22:04.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d00aff90 con 0x7fb1d00cf960 2024-03-06T07:22:04.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dc00abc30 con 0x7f2dd015ff70 2024-03-06T07:22:04.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f2dceffd640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dbc06b5a0 0x7f2dbc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7f3d35efa640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1406b520 0x7f3d1406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c8003d80 con 0x7fb1d00cf960 2024-03-06T07:22:04.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.761+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8005290 con 0x7fb1d00cf960 2024-03-06T07:22:04.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.762+0000 7f2dceffd640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dbc06b5a0 0x7f2dbc06da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009f20 tx=0x7f2dc80094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.762+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65b8095030 con 0x7f65b00e7a20 2024-03-06T07:22:04.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.762+0000 7f3d35efa640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1406b520 0x7f3d1406d9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d2002b9e0 tx=0x7f3d2003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.762+0000 7f659f7fe640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65a406b550 0x7f65a406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.762+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc30135d40 con 0x7efc300ccad0 2024-03-06T07:22:04.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.762+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65b81040a0 con 0x7f65b00e7a20 2024-03-06T07:22:04.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.763+0000 7f65b6ffd640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65a406b550 0x7f65a406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.763+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b0005290 con 0x7fb1d00cf960 2024-03-06T07:22:04.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.763+0000 7f65b6ffd640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65a406b550 0x7f65a406da10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f65a802bef0 tx=0x7f65a803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.765+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dc007ab20 con 0x7f2dd015ff70 2024-03-06T07:22:04.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.766+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1c801a070 con 0x7fb1d00cf960 2024-03-06T07:22:04.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.766+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11d4020030 con 0x7f11dc0dfee0 2024-03-06T07:22:04.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.767+0000 7fb1b77fe640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1bc06b520 0x7fb1bc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.766+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c019070 con 0x7fdd380e6920 2024-03-06T07:22:04.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.766+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c00c740 con 0x7fdd380e6920 2024-03-06T07:22:04.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.766+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c00cb30 con 0x7fdd380e6920 2024-03-06T07:22:04.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.767+0000 7fb1ceffd640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1bc06b520 0x7fb1bc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.767+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd380e0ae0 con 0x7fdd380e6920 2024-03-06T07:22:04.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.767+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd380e1fc0 con 0x7fdd380e6920 2024-03-06T07:22:04.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.767+0000 7fb1ceffd640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1bc06b520 0x7fb1bc06d9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400a810 tx=0x7fb1c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.768+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d2c057cd0 con 0x7f3d3014d2e0 2024-03-06T07:22:04.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.768+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd380057d0 con 0x7fdd380e6920 2024-03-06T07:22:04.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.769+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1c8056590 con 0x7fb1d00cf960 2024-03-06T07:22:04.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.769+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd3c01c070 con 0x7fdd380e6920 2024-03-06T07:22:04.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.770+0000 7fdd36ffd640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd2406b5a0 0x7fdd2406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.770+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd3c0899b0 con 0x7fdd380e6920 2024-03-06T07:22:04.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.774+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1c80584f0 con 0x7fb1d00cf960 2024-03-06T07:22:04.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.771+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65b809c020 con 0x7f65b00e7a20 2024-03-06T07:22:04.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.774+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd3c023020 con 0x7fdd380e6920 2024-03-06T07:22:04.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.775+0000 7fdd46563640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd2406b5a0 0x7fdd2406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:04.776+0000 7fdd46563640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd2406b5a0 0x7fdd2406da60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdd40062b90 tx=0x7fdd4006d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.101+0000 7f390a9a0640 1 -- 172.21.15.122:0/4022911254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f390406da20 con 0x7f3904162560 2024-03-06T07:22:05.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.102+0000 7f38fa7fc640 1 -- 172.21.15.122:0/4022911254 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38fc010040 con 0x7f3904162560 2024-03-06T07:22:05.104 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:05.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 -- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38e806b5d0 msgr2=0x7f38e806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38e806b5d0 0x7f38e806da90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39040b0f70 tx=0x7f38f4007990 comp rx=0 tx=0).stop 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 -- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 msgr2=0x7f39040aba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f39040aba90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00efc0 tx=0x7f38fc00c490 comp rx=0 tx=0).stop 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 -- 172.21.15.122:0/4022911254 shutdown_connections 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38e806b5d0 0x7f38e806da90 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39040b0f70 tx=0x7f38f4007990 comp rx=0 tx=0).stop 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.105+0000 7f38e7fff640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3904162560 0x7f39040aba90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.106+0000 7f38e7fff640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f39040aed00 0x7f39040ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.106+0000 7f38e7fff640 1 --2- 172.21.15.122:0/4022911254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3904071670 0x7f390420bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.106+0000 7f38e7fff640 1 -- 172.21.15.122:0/4022911254 >> 172.21.15.122:0/4022911254 conn(0x7f39040b16e0 msgr2=0x7f39041629b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.112+0000 7f38e7fff640 1 -- 172.21.15.122:0/4022911254 shutdown_connections 2024-03-06T07:22:05.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.112+0000 7f38e7fff640 1 -- 172.21.15.122:0/4022911254 wait complete. 2024-03-06T07:22:05.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-03-06T07:22:05.130 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.141+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efc30072650 con 0x7efc300ccad0 2024-03-06T07:22:05.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.141+0000 7efc267fc640 1 -- 172.21.15.122:0/3736031877 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc28060a40 con 0x7efc300ccad0 2024-03-06T07:22:05.143 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc1406b4d0 msgr2=0x7efc1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc1406b4d0 0x7efc1406d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc3015ebf0 tx=0x7efc20009190 comp rx=0 tx=0).stop 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 msgr2=0x7efc30135800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 0x7efc30135800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc280021c0 tx=0x7efc280021f0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 shutdown_connections 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc1406b4d0 0x7efc1406d990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc30145190 0x7efc30134380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc300cead0 0x7efc300cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 --2- 172.21.15.122:0/3736031877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc300ccad0 0x7efc30135800 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 >> 172.21.15.122:0/3736031877 conn(0x7efc300b1500 msgr2=0x7efc300cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 shutdown_connections 2024-03-06T07:22:05.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.143+0000 7efc36fc8640 1 -- 172.21.15.122:0/3736031877 wait complete. 2024-03-06T07:22:05.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-03-06T07:22:05.156 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.174+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2dd00748c0 con 0x7f2dd015ff70 2024-03-06T07:22:05.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.174+0000 7f2db77fe640 1 -- 172.21.15.122:0/2972934526 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2dc007a420 con 0x7f2dd015ff70 2024-03-06T07:22:05.176 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dbc06b5a0 msgr2=0x7f2dbc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dbc06b5a0 0x7f2dbc06da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009f20 tx=0x7f2dc80094d0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 msgr2=0x7f2dd014da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 0x7f2dd014da00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2dc002b7d0 tx=0x7f2dc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 shutdown_connections 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dbc06b5a0 0x7f2dbc06da60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd015ff70 0x7f2dd014da00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dd006da60 0x7f2dd0147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 --2- 172.21.15.122:0/2972934526 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dd006d000 0x7f2dd00b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 >> 172.21.15.122:0/2972934526 conn(0x7f2dd00b1c80 msgr2=0x7f2dd00cf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 shutdown_connections 2024-03-06T07:22:05.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.176+0000 7f2dd5302640 1 -- 172.21.15.122:0/2972934526 wait complete. 2024-03-06T07:22:05.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.179+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f11dc0063e0 con 0x7f11dc0dfee0 2024-03-06T07:22:05.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.179+0000 7f11d2ffd640 1 -- 172.21.15.122:0/3706538818 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11d4058d00 con 0x7f11dc0dfee0 2024-03-06T07:22:05.181 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f11c406b4d0 msgr2=0x7f11c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f11c406b4d0 0x7f11c406d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f11d803d000 tx=0x7f11d8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 msgr2=0x7f11dc035310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 0x7f11dc035310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c460 tx=0x7f11d400c920 comp rx=0 tx=0).stop 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 shutdown_connections 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f11c406b4d0 0x7f11c406d990 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f11d803d000 tx=0x7f11d8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11dc0e10c0 0x7f11dc051b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11dc0e40f0 0x7f11dc0e2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 --2- 172.21.15.122:0/3706538818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11dc0dfee0 0x7f11dc035310 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c460 tx=0x7f11d400c920 comp rx=0 tx=0).stop 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.181+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 >> 172.21.15.122:0/3706538818 conn(0x7f11dc01ef30 msgr2=0x7f11dc054580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.182+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 shutdown_connections 2024-03-06T07:22:05.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.182+0000 7f11e8bc7640 1 -- 172.21.15.122:0/3706538818 wait complete. 2024-03-06T07:22:05.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.184+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3d18005580 con 0x7f3d3014d2e0 2024-03-06T07:22:05.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.184+0000 7f3d267fc640 1 -- 172.21.15.122:0/3408675950 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d2c05bd70 con 0x7f3d3014d2e0 2024-03-06T07:22:05.186 INFO:teuthology.orchestra.run.smithi122.stdout:55834574856 2024-03-06T07:22:05.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.187+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1406b520 msgr2=0x7f3d1406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1406b520 0x7f3d1406d9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d2002b9e0 tx=0x7f3d2003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d3014d2e0 msgr2=0x7f3d300b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d3014d2e0 0x7f3d300b0710 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00c3e0 tx=0x7f3d2c00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 shutdown_connections 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1406b520 0x7f3d1406d9e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d3014d2e0 0x7f3d300b0710 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d300ccd60 0x7f3d3014b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 --2- 172.21.15.122:0/3408675950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d30071530 0x7f3d30158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 >> 172.21.15.122:0/3408675950 conn(0x7f3d300b1500 msgr2=0x7f3d300b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 shutdown_connections 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f3d3815e640 1 -- 172.21.15.122:0/3408675950 wait complete. 2024-03-06T07:22:05.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f65b00382a0 con 0x7f65b00e7a20 2024-03-06T07:22:05.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-03-06T07:22:05.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.188+0000 7f659f7fe640 1 -- 172.21.15.122:0/1125089413 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65b80d2f90 con 0x7f65b00e7a20 2024-03-06T07:22:05.193 INFO:teuthology.orchestra.run.smithi122.stdout:55834574856 2024-03-06T07:22:05.193 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.189+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdd380528b0 con 0x7fdd380e6920 2024-03-06T07:22:05.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.189+0000 7fdd36ffd640 1 -- 172.21.15.122:0/2153622578 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdd3c023020 con 0x7fdd380e6920 2024-03-06T07:22:05.194 INFO:teuthology.orchestra.run.smithi122.stdout:55834574857 2024-03-06T07:22:05.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.190+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb1b0005850 con 0x7fb1d00cf960 2024-03-06T07:22:05.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.190+0000 7fb1b77fe640 1 -- 172.21.15.122:0/4271889355 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb1c805c590 con 0x7fb1d00cf960 2024-03-06T07:22:05.195 INFO:teuthology.orchestra.run.smithi122.stdout:55834574855 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65a406b550 msgr2=0x7f65a406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65a406b550 0x7f65a406da10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f65a802bef0 tx=0x7f65a803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65b00e7a20 msgr2=0x7f65b0051430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65b00e7a20 0x7f65b0051430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65b8050c30 tx=0x7f65b80683c0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 shutdown_connections 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65a406b550 0x7f65a406da10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65b00e8590 0x7f65b00e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65b00e7a20 0x7f65b0051430 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 --2- 172.21.15.122:0/1125089413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65b0053a30 0x7f65b00d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 >> 172.21.15.122:0/1125089413 conn(0x7f65b003aa20 msgr2=0x7f65b00e5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 shutdown_connections 2024-03-06T07:22:05.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7f65b7fff640 1 -- 172.21.15.122:0/1125089413 wait complete. 2024-03-06T07:22:05.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd2406b5a0 msgr2=0x7fdd2406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd2406b5a0 0x7fdd2406da60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdd40062b90 tx=0x7fdd4006d040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 msgr2=0x7fdd380e3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 0x7fdd380e3910 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00b780 tx=0x7fdd3c00bc40 comp rx=0 tx=0).stop 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 shutdown_connections 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd2406b5a0 0x7fdd2406da60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.192+0000 7fdd47565640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd380e6d00 0x7fdd380e3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fdd47565640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd380e6920 0x7fdd380e3910 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fdd47565640 1 --2- 172.21.15.122:0/2153622578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd380552d0 0x7fdd380dd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 >> 172.21.15.122:0/2153622578 conn(0x7fdd3801ef30 msgr2=0x7fdd3800c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 shutdown_connections 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fdd47565640 1 -- 172.21.15.122:0/2153622578 wait complete. 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1bc06b520 msgr2=0x7fb1bc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1bc06b520 0x7fb1bc06d9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400a810 tx=0x7fb1c4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 msgr2=0x7fb1d00b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 0x7fb1d00b0d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800c7d0 tx=0x7fb1c800cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 shutdown_connections 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1bc06b520 0x7fb1bc06d9e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1d015c710 0x7fb1d01447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d00cf960 0x7fb1d00b0d50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 --2- 172.21.15.122:0/4271889355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d0072520 0x7fb1d0151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 >> 172.21.15.122:0/4271889355 conn(0x7fb1d006afb0 msgr2=0x7fb1d00cf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 shutdown_connections 2024-03-06T07:22:05.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.193+0000 7fb1cffff640 1 -- 172.21.15.122:0/4271889355 wait complete. 2024-03-06T07:22:05.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-06T07:22:05.201 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-03-06T07:22:05.209 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-03-06T07:22:05.210 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-03-06T07:22:05.210 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-03-06T07:22:05.212 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:05.212 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:22:05.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:05.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.354+0000 7f9f041f4640 1 Processor -- start 2024-03-06T07:22:05.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.354+0000 7f9f041f4640 1 -- start start 2024-03-06T07:22:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f041f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9efc166af0 0x7f9efc168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f041f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc15e380 con 0x7f9efc0d00c0 2024-03-06T07:22:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f041f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0cdc10 con 0x7f9efc0cf850 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f041f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc06ca90 con 0x7f9efc166af0 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f02791640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9efc166af0 0x7f9efc168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f01f90640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9efc0d00c0 0x7f9efc0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38552/0 (socket says 172.21.15.122:38552) 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f02791640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9efc166af0 0x7f9efc168ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57044/0 (socket says 172.21.15.122:57044) 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f01f90640 1 -- 172.21.15.122:0/1626682183 learned_addr learned my addr 172.21.15.122:0/1626682183 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f00f8e640 1 -- 172.21.15.122:0/1626682183 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312106788 0 0) 0x7f9efc15e380 con 0x7f9efc0d00c0 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f00f8e640 1 -- 172.21.15.122:0/1626682183 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee8003520 con 0x7f9efc0d00c0 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f02791640 1 -- 172.21.15.122:0/1626682183 >> v1:172.21.15.169:6789/0 conn(0x7f9efc0cf850 legacy=0x7f9efc0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f00f8e640 1 -- 172.21.15.122:0/1626682183 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293854626 0 0) 0x7f9efc0cdc10 con 0x7f9efc0cf850 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f02791640 1 -- 172.21.15.122:0/1626682183 >> v1:172.21.15.122:6789/0 conn(0x7f9efc0d00c0 legacy=0x7f9efc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.355+0000 7f9f02791640 1 -- 172.21.15.122:0/1626682183 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc06c160 con 0x7f9efc166af0 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f02791640 1 --2- 172.21.15.122:0/1626682183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc166af0 0x7f9efc168ee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009d90 tx=0x7f9ef402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87954b9096d51fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f00f8e640 1 -- 172.21.15.122:0/1626682183 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4039070 con 0x7f9efc166af0 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f00f8e640 1 -- 172.21.15.122:0/1626682183 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef402cd00 con 0x7f9efc166af0 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f00f8e640 1 -- 172.21.15.122:0/1626682183 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4034660 con 0x7f9efc166af0 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f041f4640 1 -- 172.21.15.122:0/1626682183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc166af0 msgr2=0x7f9efc168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f041f4640 1 --2- 172.21.15.122:0/1626682183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc166af0 0x7f9efc168ee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009d90 tx=0x7f9ef402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f041f4640 1 -- 172.21.15.122:0/1626682183 shutdown_connections 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f041f4640 1 --2- 172.21.15.122:0/1626682183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc166af0 0x7f9efc168ee0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.356+0000 7f9f041f4640 1 -- 172.21.15.122:0/1626682183 >> 172.21.15.122:0/1626682183 conn(0x7f9efc0b35a0 msgr2=0x7f9efc0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.357+0000 7f9f041f4640 1 -- 172.21.15.122:0/1626682183 shutdown_connections 2024-03-06T07:22:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.357+0000 7f9f041f4640 1 -- 172.21.15.122:0/1626682183 wait complete. 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 Processor -- start 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 -- start start 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 0x7f9efc0ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9efc0d00c0 0x7f9efc159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9efc166af0 0x7f9efc15a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc06c720 con 0x7f9efc0cf850 2024-03-06T07:22:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc06df30 con 0x7f9efc0d00c0 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f041f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0b1030 con 0x7f9efc166af0 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f01f90640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 0x7f9efc0ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f01f90640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 0x7f9efc0ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57054/0 (socket says 172.21.15.122:57054) 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.358+0000 7f9f01f90640 1 -- 172.21.15.122:0/417013627 learned_addr learned my addr 172.21.15.122:0/417013627 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f0178f640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9efc0d00c0 0x7f9efc159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f02791640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9efc166af0 0x7f9efc15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f01f90640 1 -- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9efc166af0 msgr2=0x7f9efc15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f01f90640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9efc166af0 0x7f9efc15a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f01f90640 1 -- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9efc0d00c0 msgr2=0x7f9efc159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f01f90640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9efc0d00c0 0x7f9efc159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f01f90640 1 -- 172.21.15.122:0/417013627 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0c95f0 con 0x7f9efc0cf850 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f01f90640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 0x7f9efc0ca370 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c430 tx=0x7f9ef800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8017070 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8005ce0 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0c9b70 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80072f0 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.359+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc0cb910 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.360+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc04f330 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.360+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ef801a070 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.361+0000 7f9ef27fc640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee006b4d0 0x7f9ee006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.361+0000 7f9f0178f640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee006b4d0 0x7f9ee006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.361+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ef8089630 con 0x7f9efc0cf850 2024-03-06T07:22:05.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.361+0000 7f9f0178f640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee006b4d0 0x7f9ee006d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0cee80 tx=0x7f9eec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.365+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ef8057e60 con 0x7f9efc0cf850 2024-03-06T07:22:05.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.533+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9efc1597c0 con 0x7f9ee006b4d0 2024-03-06T07:22:05.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.539+0000 7f9ef27fc640 1 -- 172.21.15.122:0/417013627 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f9efc1597c0 con 0x7f9ee006b4d0 2024-03-06T07:22:05.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:05.545 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:05.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee006b4d0 msgr2=0x7f9ee006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee006b4d0 0x7f9ee006d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0cee80 tx=0x7f9eec009040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 msgr2=0x7f9efc0ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 0x7f9efc0ca370 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c430 tx=0x7f9ef800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 shutdown_connections 2024-03-06T07:22:05.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee006b4d0 0x7f9ee006d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9efc166af0 0x7f9efc15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9efc0d00c0 0x7f9efc159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 --2- 172.21.15.122:0/417013627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9efc0cf850 0x7f9efc0ca370 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 >> 172.21.15.122:0/417013627 conn(0x7f9efc0b35a0 msgr2=0x7f9efc0d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 shutdown_connections 2024-03-06T07:22:05.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.542+0000 7f9f041f4640 1 -- 172.21.15.122:0/417013627 wait complete. 2024-03-06T07:22:05.555 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-06T07:22:05.333171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218996,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755724,"statfs":{"total":773094113280,"available":772869861376,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.774147"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778717+0000","last_change":"2024-03-06T07:21:53.563999+0000","last_active":"2024-03-06T07:21:54.778717+0000","last_peered":"2024-03-06T07:21:54.778717+0000","last_clean":"2024-03-06T07:21:54.778717+0000","last_became_active":"2024-03-06T07:21:53.563557+0000","last_became_peered":"2024-03-06T07:21:53.563557+0000","last_unstale":"2024-03-06T07:21:54.778717+0000","last_undegraded":"2024-03-06T07:21:54.778717+0000","last_fullsized":"2024-03-06T07:21:54.778717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562641+0000","last_change":"2024-03-06T07:21:53.560952+0000","last_active":"2024-03-06T07:21:54.562641+0000","last_peered":"2024-03-06T07:21:54.562641+0000","last_clean":"2024-03-06T07:21:54.562641+0000","last_became_active":"2024-03-06T07:21:53.560651+0000","last_became_peered":"2024-03-06T07:21:53.560651+0000","last_unstale":"2024-03-06T07:21:54.562641+0000","last_undegraded":"2024-03-06T07:21:54.562641+0000","last_fullsized":"2024-03-06T07:21:54.562641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777337+0000","last_change":"2024-03-06T07:21:53.547890+0000","last_active":"2024-03-06T07:21:54.777337+0000","last_peered":"2024-03-06T07:21:54.777337+0000","last_clean":"2024-03-06T07:21:54.777337+0000","last_became_active":"2024-03-06T07:21:53.547631+0000","last_became_peered":"2024-03-06T07:21:53.547631+0000","last_unstale":"2024-03-06T07:21:54.777337+0000","last_undegraded":"2024-03-06T07:21:54.777337+0000","last_fullsized":"2024-03-06T07:21:54.777337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778291+0000","last_change":"2024-03-06T07:21:53.565535+0000","last_active":"2024-03-06T07:21:54.778291+0000","last_peered":"2024-03-06T07:21:54.778291+0000","last_clean":"2024-03-06T07:21:54.778291+0000","last_became_active":"2024-03-06T07:21:53.565256+0000","last_became_peered":"2024-03-06T07:21:53.565256+0000","last_unstale":"2024-03-06T07:21:54.778291+0000","last_undegraded":"2024-03-06T07:21:54.778291+0000","last_fullsized":"2024-03-06T07:21:54.778291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778186+0000","last_change":"2024-03-06T07:21:53.558864+0000","last_active":"2024-03-06T07:21:54.778186+0000","last_peered":"2024-03-06T07:21:54.778186+0000","last_clean":"2024-03-06T07:21:54.778186+0000","last_became_active":"2024-03-06T07:21:53.558582+0000","last_became_peered":"2024-03-06T07:21:53.558582+0000","last_unstale":"2024-03-06T07:21:54.778186+0000","last_undegraded":"2024-03-06T07:21:54.778186+0000","last_fullsized":"2024-03-06T07:21:54.778186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777685+0000","last_change":"2024-03-06T07:21:53.567354+0000","last_active":"2024-03-06T07:21:54.777685+0000","last_peered":"2024-03-06T07:21:54.777685+0000","last_clean":"2024-03-06T07:21:54.777685+0000","last_became_active":"2024-03-06T07:21:53.567054+0000","last_became_peered":"2024-03-06T07:21:53.567054+0000","last_unstale":"2024-03-06T07:21:54.777685+0000","last_undegraded":"2024-03-06T07:21:54.777685+0000","last_fullsized":"2024-03-06T07:21:54.777685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777444+0000","last_change":"2024-03-06T07:21:53.563005+0000","last_active":"2024-03-06T07:21:54.777444+0000","last_peered":"2024-03-06T07:21:54.777444+0000","last_clean":"2024-03-06T07:21:54.777444+0000","last_became_active":"2024-03-06T07:21:53.562611+0000","last_became_peered":"2024-03-06T07:21:53.562611+0000","last_unstale":"2024-03-06T07:21:54.777444+0000","last_undegraded":"2024-03-06T07:21:54.777444+0000","last_fullsized":"2024-03-06T07:21:54.777444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.766995+0000","last_change":"2024-03-06T07:21:53.566285+0000","last_active":"2024-03-06T07:21:54.766995+0000","last_peered":"2024-03-06T07:21:54.766995+0000","last_clean":"2024-03-06T07:21:54.766995+0000","last_became_active":"2024-03-06T07:21:53.565429+0000","last_became_peered":"2024-03-06T07:21:53.565429+0000","last_unstale":"2024-03-06T07:21:54.766995+0000","last_undegraded":"2024-03-06T07:21:54.766995+0000","last_fullsized":"2024-03-06T07:21:54.766995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778770+0000","last_change":"2024-03-06T07:21:53.565916+0000","last_active":"2024-03-06T07:21:54.778770+0000","last_peered":"2024-03-06T07:21:54.778770+0000","last_clean":"2024-03-06T07:21:54.778770+0000","last_became_active":"2024-03-06T07:21:53.565555+0000","last_became_peered":"2024-03-06T07:21:53.565555+0000","last_unstale":"2024-03-06T07:21:54.778770+0000","last_undegraded":"2024-03-06T07:21:54.778770+0000","last_fullsized":"2024-03-06T07:21:54.778770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:23.946424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562690+0000","last_change":"2024-03-06T07:21:53.557656+0000","last_active":"2024-03-06T07:21:54.562690+0000","last_peered":"2024-03-06T07:21:54.562690+0000","last_clean":"2024-03-06T07:21:54.562690+0000","last_became_active":"2024-03-06T07:21:53.557224+0000","last_became_peered":"2024-03-06T07:21:53.557224+0000","last_unstale":"2024-03-06T07:21:54.562690+0000","last_undegraded":"2024-03-06T07:21:54.562690+0000","last_fullsized":"2024-03-06T07:21:54.562690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562612+0000","last_change":"2024-03-06T07:21:53.565698+0000","last_active":"2024-03-06T07:21:54.562612+0000","last_peered":"2024-03-06T07:21:54.562612+0000","last_clean":"2024-03-06T07:21:54.562612+0000","last_became_active":"2024-03-06T07:21:53.565390+0000","last_became_peered":"2024-03-06T07:21:53.565390+0000","last_unstale":"2024-03-06T07:21:54.562612+0000","last_undegraded":"2024-03-06T07:21:54.562612+0000","last_fullsized":"2024-03-06T07:21:54.562612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777702+0000","last_change":"2024-03-06T07:21:53.558368+0000","last_active":"2024-03-06T07:21:54.777702+0000","last_peered":"2024-03-06T07:21:54.777702+0000","last_clean":"2024-03-06T07:21:54.777702+0000","last_became_active":"2024-03-06T07:21:53.558065+0000","last_became_peered":"2024-03-06T07:21:53.558065+0000","last_unstale":"2024-03-06T07:21:54.777702+0000","last_undegraded":"2024-03-06T07:21:54.777702+0000","last_fullsized":"2024-03-06T07:21:54.777702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:44.794765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767222+0000","last_change":"2024-03-06T07:21:53.567165+0000","last_active":"2024-03-06T07:21:54.767222+0000","last_peered":"2024-03-06T07:21:54.767222+0000","last_clean":"2024-03-06T07:21:54.767222+0000","last_became_active":"2024-03-06T07:21:53.566826+0000","last_became_peered":"2024-03-06T07:21:53.566826+0000","last_unstale":"2024-03-06T07:21:54.767222+0000","last_undegraded":"2024-03-06T07:21:54.767222+0000","last_fullsized":"2024-03-06T07:21:54.767222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777131+0000","last_change":"2024-03-06T07:21:53.552931+0000","last_active":"2024-03-06T07:21:54.777131+0000","last_peered":"2024-03-06T07:21:54.777131+0000","last_clean":"2024-03-06T07:21:54.777131+0000","last_became_active":"2024-03-06T07:21:53.552654+0000","last_became_peered":"2024-03-06T07:21:53.552654+0000","last_unstale":"2024-03-06T07:21:54.777131+0000","last_undegraded":"2024-03-06T07:21:54.777131+0000","last_fullsized":"2024-03-06T07:21:54.777131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:04.669227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562518+0000","last_change":"2024-03-06T07:21:53.562294+0000","last_active":"2024-03-06T07:21:54.562518+0000","last_peered":"2024-03-06T07:21:54.562518+0000","last_clean":"2024-03-06T07:21:54.562518+0000","last_became_active":"2024-03-06T07:21:53.561996+0000","last_became_peered":"2024-03-06T07:21:53.561996+0000","last_unstale":"2024-03-06T07:21:54.562518+0000","last_undegraded":"2024-03-06T07:21:54.562518+0000","last_fullsized":"2024-03-06T07:21:54.562518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777264+0000","last_change":"2024-03-06T07:21:53.563516+0000","last_active":"2024-03-06T07:21:54.777264+0000","last_peered":"2024-03-06T07:21:54.777264+0000","last_clean":"2024-03-06T07:21:54.777264+0000","last_became_active":"2024-03-06T07:21:53.563237+0000","last_became_peered":"2024-03-06T07:21:53.563237+0000","last_unstale":"2024-03-06T07:21:54.777264+0000","last_undegraded":"2024-03-06T07:21:54.777264+0000","last_fullsized":"2024-03-06T07:21:54.777264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778868+0000","last_change":"2024-03-06T07:21:53.561815+0000","last_active":"2024-03-06T07:21:54.778868+0000","last_peered":"2024-03-06T07:21:54.778868+0000","last_clean":"2024-03-06T07:21:54.778868+0000","last_became_active":"2024-03-06T07:21:53.561189+0000","last_became_peered":"2024-03-06T07:21:53.561189+0000","last_unstale":"2024-03-06T07:21:54.778868+0000","last_undegraded":"2024-03-06T07:21:54.778868+0000","last_fullsized":"2024-03-06T07:21:54.778868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560237+0000","last_change":"2024-03-06T07:21:53.560903+0000","last_active":"2024-03-06T07:21:54.560237+0000","last_peered":"2024-03-06T07:21:54.560237+0000","last_clean":"2024-03-06T07:21:54.560237+0000","last_became_active":"2024-03-06T07:21:53.560624+0000","last_became_peered":"2024-03-06T07:21:53.560624+0000","last_unstale":"2024-03-06T07:21:54.560237+0000","last_undegraded":"2024-03-06T07:21:54.560237+0000","last_fullsized":"2024-03-06T07:21:54.560237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778508+0000","last_change":"2024-03-06T07:21:53.561059+0000","last_active":"2024-03-06T07:21:54.778508+0000","last_peered":"2024-03-06T07:21:54.778508+0000","last_clean":"2024-03-06T07:21:54.778508+0000","last_became_active":"2024-03-06T07:21:53.560799+0000","last_became_peered":"2024-03-06T07:21:53.560799+0000","last_unstale":"2024-03-06T07:21:54.778508+0000","last_undegraded":"2024-03-06T07:21:54.778508+0000","last_fullsized":"2024-03-06T07:21:54.778508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562750+0000","last_change":"2024-03-06T07:21:53.558851+0000","last_active":"2024-03-06T07:21:54.562750+0000","last_peered":"2024-03-06T07:21:54.562750+0000","last_clean":"2024-03-06T07:21:54.562750+0000","last_became_active":"2024-03-06T07:21:53.558543+0000","last_became_peered":"2024-03-06T07:21:53.558543+0000","last_unstale":"2024-03-06T07:21:54.562750+0000","last_undegraded":"2024-03-06T07:21:54.562750+0000","last_fullsized":"2024-03-06T07:21:54.562750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777302+0000","last_change":"2024-03-06T07:21:53.560684+0000","last_active":"2024-03-06T07:21:54.777302+0000","last_peered":"2024-03-06T07:21:54.777302+0000","last_clean":"2024-03-06T07:21:54.777302+0000","last_became_active":"2024-03-06T07:21:53.560428+0000","last_became_peered":"2024-03-06T07:21:53.560428+0000","last_unstale":"2024-03-06T07:21:54.777302+0000","last_undegraded":"2024-03-06T07:21:54.777302+0000","last_fullsized":"2024-03-06T07:21:54.777302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777813+0000","last_change":"2024-03-06T07:21:53.562732+0000","last_active":"2024-03-06T07:21:54.777813+0000","last_peered":"2024-03-06T07:21:54.777813+0000","last_clean":"2024-03-06T07:21:54.777813+0000","last_became_active":"2024-03-06T07:21:53.562351+0000","last_became_peered":"2024-03-06T07:21:53.562351+0000","last_unstale":"2024-03-06T07:21:54.777813+0000","last_undegraded":"2024-03-06T07:21:54.777813+0000","last_fullsized":"2024-03-06T07:21:54.777813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767070+0000","last_change":"2024-03-06T07:21:53.564304+0000","last_active":"2024-03-06T07:21:54.767070+0000","last_peered":"2024-03-06T07:21:54.767070+0000","last_clean":"2024-03-06T07:21:54.767070+0000","last_became_active":"2024-03-06T07:21:53.564008+0000","last_became_peered":"2024-03-06T07:21:53.564008+0000","last_unstale":"2024-03-06T07:21:54.767070+0000","last_undegraded":"2024-03-06T07:21:54.767070+0000","last_fullsized":"2024-03-06T07:21:54.767070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777401+0000","last_change":"2024-03-06T07:21:53.565197+0000","last_active":"2024-03-06T07:21:54.777401+0000","last_peered":"2024-03-06T07:21:54.777401+0000","last_clean":"2024-03-06T07:21:54.777401+0000","last_became_active":"2024-03-06T07:21:53.564878+0000","last_became_peered":"2024-03-06T07:21:53.564878+0000","last_unstale":"2024-03-06T07:21:54.777401+0000","last_undegraded":"2024-03-06T07:21:54.777401+0000","last_fullsized":"2024-03-06T07:21:54.777401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767433+0000","last_change":"2024-03-06T07:21:53.558718+0000","last_active":"2024-03-06T07:21:54.767433+0000","last_peered":"2024-03-06T07:21:54.767433+0000","last_clean":"2024-03-06T07:21:54.767433+0000","last_became_active":"2024-03-06T07:21:53.558454+0000","last_became_peered":"2024-03-06T07:21:53.558454+0000","last_unstale":"2024-03-06T07:21:54.767433+0000","last_undegraded":"2024-03-06T07:21:54.767433+0000","last_fullsized":"2024-03-06T07:21:54.767433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562735+0000","last_change":"2024-03-06T07:21:53.563894+0000","last_active":"2024-03-06T07:21:54.562735+0000","last_peered":"2024-03-06T07:21:54.562735+0000","last_clean":"2024-03-06T07:21:54.562735+0000","last_became_active":"2024-03-06T07:21:53.563493+0000","last_became_peered":"2024-03-06T07:21:53.563493+0000","last_unstale":"2024-03-06T07:21:54.562735+0000","last_undegraded":"2024-03-06T07:21:54.562735+0000","last_fullsized":"2024-03-06T07:21:54.562735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562575+0000","last_change":"2024-03-06T07:21:53.552128+0000","last_active":"2024-03-06T07:21:54.562575+0000","last_peered":"2024-03-06T07:21:54.562575+0000","last_clean":"2024-03-06T07:21:54.562575+0000","last_became_active":"2024-03-06T07:21:53.551826+0000","last_became_peered":"2024-03-06T07:21:53.551826+0000","last_unstale":"2024-03-06T07:21:54.562575+0000","last_undegraded":"2024-03-06T07:21:54.562575+0000","last_fullsized":"2024-03-06T07:21:54.562575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767379+0000","last_change":"2024-03-06T07:21:53.563089+0000","last_active":"2024-03-06T07:21:54.767379+0000","last_peered":"2024-03-06T07:21:54.767379+0000","last_clean":"2024-03-06T07:21:54.767379+0000","last_became_active":"2024-03-06T07:21:53.562833+0000","last_became_peered":"2024-03-06T07:21:53.562833+0000","last_unstale":"2024-03-06T07:21:54.767379+0000","last_undegraded":"2024-03-06T07:21:54.767379+0000","last_fullsized":"2024-03-06T07:21:54.767379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777387+0000","last_change":"2024-03-06T07:21:53.550849+0000","last_active":"2024-03-06T07:21:54.777387+0000","last_peered":"2024-03-06T07:21:54.777387+0000","last_clean":"2024-03-06T07:21:54.777387+0000","last_became_active":"2024-03-06T07:21:53.550562+0000","last_became_peered":"2024-03-06T07:21:53.550562+0000","last_unstale":"2024-03-06T07:21:54.777387+0000","last_undegraded":"2024-03-06T07:21:54.777387+0000","last_fullsized":"2024-03-06T07:21:54.777387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777942+0000","last_change":"2024-03-06T07:21:53.564528+0000","last_active":"2024-03-06T07:21:54.777942+0000","last_peered":"2024-03-06T07:21:54.777942+0000","last_clean":"2024-03-06T07:21:54.777942+0000","last_became_active":"2024-03-06T07:21:53.564250+0000","last_became_peered":"2024-03-06T07:21:53.564250+0000","last_unstale":"2024-03-06T07:21:54.777942+0000","last_undegraded":"2024-03-06T07:21:54.777942+0000","last_fullsized":"2024-03-06T07:21:54.777942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560202+0000","last_change":"2024-03-06T07:21:53.555673+0000","last_active":"2024-03-06T07:21:54.560202+0000","last_peered":"2024-03-06T07:21:54.560202+0000","last_clean":"2024-03-06T07:21:54.560202+0000","last_became_active":"2024-03-06T07:21:53.555337+0000","last_became_peered":"2024-03-06T07:21:53.555337+0000","last_unstale":"2024-03-06T07:21:54.560202+0000","last_undegraded":"2024-03-06T07:21:54.560202+0000","last_fullsized":"2024-03-06T07:21:54.560202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562445+0000","last_change":"2024-03-06T07:21:53.552447+0000","last_active":"2024-03-06T07:21:54.562445+0000","last_peered":"2024-03-06T07:21:54.562445+0000","last_clean":"2024-03-06T07:21:54.562445+0000","last_became_active":"2024-03-06T07:21:53.551907+0000","last_became_peered":"2024-03-06T07:21:53.551907+0000","last_unstale":"2024-03-06T07:21:54.562445+0000","last_undegraded":"2024-03-06T07:21:54.562445+0000","last_fullsized":"2024-03-06T07:21:54.562445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777543+0000","last_change":"2024-03-06T07:21:53.554149+0000","last_active":"2024-03-06T07:21:54.777543+0000","last_peered":"2024-03-06T07:21:54.777543+0000","last_clean":"2024-03-06T07:21:54.777543+0000","last_became_active":"2024-03-06T07:21:53.553811+0000","last_became_peered":"2024-03-06T07:21:53.553811+0000","last_unstale":"2024-03-06T07:21:54.777543+0000","last_undegraded":"2024-03-06T07:21:54.777543+0000","last_fullsized":"2024-03-06T07:21:54.777543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:11.236013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767399+0000","last_change":"2024-03-06T07:21:52.551262+0000","last_active":"2024-03-06T07:21:54.767399+0000","last_peered":"2024-03-06T07:21:54.767399+0000","last_clean":"2024-03-06T07:21:54.767399+0000","last_became_active":"2024-03-06T07:21:52.550349+0000","last_became_peered":"2024-03-06T07:21:52.550349+0000","last_unstale":"2024-03-06T07:21:54.767399+0000","last_undegraded":"2024-03-06T07:21:54.767399+0000","last_fullsized":"2024-03-06T07:21:54.767399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777450+0000","last_change":"2024-03-06T07:21:53.562215+0000","last_active":"2024-03-06T07:21:54.777450+0000","last_peered":"2024-03-06T07:21:54.777450+0000","last_clean":"2024-03-06T07:21:54.777450+0000","last_became_active":"2024-03-06T07:21:53.561933+0000","last_became_peered":"2024-03-06T07:21:53.561933+0000","last_unstale":"2024-03-06T07:21:54.777450+0000","last_undegraded":"2024-03-06T07:21:54.777450+0000","last_fullsized":"2024-03-06T07:21:54.777450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562398+0000","last_change":"2024-03-06T07:21:52.547490+0000","last_active":"2024-03-06T07:21:54.562398+0000","last_peered":"2024-03-06T07:21:54.562398+0000","last_clean":"2024-03-06T07:21:54.562398+0000","last_became_active":"2024-03-06T07:21:52.547168+0000","last_became_peered":"2024-03-06T07:21:52.547168+0000","last_unstale":"2024-03-06T07:21:54.562398+0000","last_undegraded":"2024-03-06T07:21:54.562398+0000","last_fullsized":"2024-03-06T07:21:54.562398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767143+0000","last_change":"2024-03-06T07:21:52.551838+0000","last_active":"2024-03-06T07:21:54.767143+0000","last_peered":"2024-03-06T07:21:54.767143+0000","last_clean":"2024-03-06T07:21:54.767143+0000","last_became_active":"2024-03-06T07:21:52.551599+0000","last_became_peered":"2024-03-06T07:21:52.551599+0000","last_unstale":"2024-03-06T07:21:54.767143+0000","last_undegraded":"2024-03-06T07:21:54.767143+0000","last_fullsized":"2024-03-06T07:21:54.767143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767031+0000","last_change":"2024-03-06T07:21:53.557369+0000","last_active":"2024-03-06T07:21:54.767031+0000","last_peered":"2024-03-06T07:21:54.767031+0000","last_clean":"2024-03-06T07:21:54.767031+0000","last_became_active":"2024-03-06T07:21:53.556320+0000","last_became_peered":"2024-03-06T07:21:53.556320+0000","last_unstale":"2024-03-06T07:21:54.767031+0000","last_undegraded":"2024-03-06T07:21:54.767031+0000","last_fullsized":"2024-03-06T07:21:54.767031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777454+0000","last_change":"2024-03-06T07:21:52.553279+0000","last_active":"2024-03-06T07:21:54.777454+0000","last_peered":"2024-03-06T07:21:54.777454+0000","last_clean":"2024-03-06T07:21:54.777454+0000","last_became_active":"2024-03-06T07:21:52.552991+0000","last_became_peered":"2024-03-06T07:21:52.552991+0000","last_unstale":"2024-03-06T07:21:54.777454+0000","last_undegraded":"2024-03-06T07:21:54.777454+0000","last_fullsized":"2024-03-06T07:21:54.777454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562572+0000","last_change":"2024-03-06T07:21:53.552639+0000","last_active":"2024-03-06T07:21:54.562572+0000","last_peered":"2024-03-06T07:21:54.562572+0000","last_clean":"2024-03-06T07:21:54.562572+0000","last_became_active":"2024-03-06T07:21:53.552377+0000","last_became_peered":"2024-03-06T07:21:53.552377+0000","last_unstale":"2024-03-06T07:21:54.562572+0000","last_undegraded":"2024-03-06T07:21:54.562572+0000","last_fullsized":"2024-03-06T07:21:54.562572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562577+0000","last_change":"2024-03-06T07:21:52.556094+0000","last_active":"2024-03-06T07:21:54.562577+0000","last_peered":"2024-03-06T07:21:54.562577+0000","last_clean":"2024-03-06T07:21:54.562577+0000","last_became_active":"2024-03-06T07:21:52.555825+0000","last_became_peered":"2024-03-06T07:21:52.555825+0000","last_unstale":"2024-03-06T07:21:54.562577+0000","last_undegraded":"2024-03-06T07:21:54.562577+0000","last_fullsized":"2024-03-06T07:21:54.562577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778464+0000","last_change":"2024-03-06T07:21:53.561469+0000","last_active":"2024-03-06T07:21:54.778464+0000","last_peered":"2024-03-06T07:21:54.778464+0000","last_clean":"2024-03-06T07:21:54.778464+0000","last_became_active":"2024-03-06T07:21:53.561204+0000","last_became_peered":"2024-03-06T07:21:53.561204+0000","last_unstale":"2024-03-06T07:21:54.778464+0000","last_undegraded":"2024-03-06T07:21:54.778464+0000","last_fullsized":"2024-03-06T07:21:54.778464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778543+0000","last_change":"2024-03-06T07:21:52.557218+0000","last_active":"2024-03-06T07:21:54.778543+0000","last_peered":"2024-03-06T07:21:54.778543+0000","last_clean":"2024-03-06T07:21:54.778543+0000","last_became_active":"2024-03-06T07:21:52.556955+0000","last_became_peered":"2024-03-06T07:21:52.556955+0000","last_unstale":"2024-03-06T07:21:54.778543+0000","last_undegraded":"2024-03-06T07:21:54.778543+0000","last_fullsized":"2024-03-06T07:21:54.778543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777433+0000","last_change":"2024-03-06T07:21:53.559108+0000","last_active":"2024-03-06T07:21:54.777433+0000","last_peered":"2024-03-06T07:21:54.777433+0000","last_clean":"2024-03-06T07:21:54.777433+0000","last_became_active":"2024-03-06T07:21:53.558256+0000","last_became_peered":"2024-03-06T07:21:53.558256+0000","last_unstale":"2024-03-06T07:21:54.777433+0000","last_undegraded":"2024-03-06T07:21:54.777433+0000","last_fullsized":"2024-03-06T07:21:54.777433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562655+0000","last_change":"2024-03-06T07:21:52.557898+0000","last_active":"2024-03-06T07:21:54.562655+0000","last_peered":"2024-03-06T07:21:54.562655+0000","last_clean":"2024-03-06T07:21:54.562655+0000","last_became_active":"2024-03-06T07:21:52.557619+0000","last_became_peered":"2024-03-06T07:21:52.557619+0000","last_unstale":"2024-03-06T07:21:54.562655+0000","last_undegraded":"2024-03-06T07:21:54.562655+0000","last_fullsized":"2024-03-06T07:21:54.562655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777844+0000","last_change":"2024-03-06T07:21:53.557112+0000","last_active":"2024-03-06T07:21:54.777844+0000","last_peered":"2024-03-06T07:21:54.777844+0000","last_clean":"2024-03-06T07:21:54.777844+0000","last_became_active":"2024-03-06T07:21:53.556848+0000","last_became_peered":"2024-03-06T07:21:53.556848+0000","last_unstale":"2024-03-06T07:21:54.777844+0000","last_undegraded":"2024-03-06T07:21:54.777844+0000","last_fullsized":"2024-03-06T07:21:54.777844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562841+0000","last_change":"2024-03-06T07:21:52.548388+0000","last_active":"2024-03-06T07:21:54.562841+0000","last_peered":"2024-03-06T07:21:54.562841+0000","last_clean":"2024-03-06T07:21:54.562841+0000","last_became_active":"2024-03-06T07:21:52.548095+0000","last_became_peered":"2024-03-06T07:21:52.548095+0000","last_unstale":"2024-03-06T07:21:54.562841+0000","last_undegraded":"2024-03-06T07:21:54.562841+0000","last_fullsized":"2024-03-06T07:21:54.562841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778266+0000","last_change":"2024-03-06T07:21:53.557821+0000","last_active":"2024-03-06T07:21:54.778266+0000","last_peered":"2024-03-06T07:21:54.778266+0000","last_clean":"2024-03-06T07:21:54.778266+0000","last_became_active":"2024-03-06T07:21:53.557536+0000","last_became_peered":"2024-03-06T07:21:53.557536+0000","last_unstale":"2024-03-06T07:21:54.778266+0000","last_undegraded":"2024-03-06T07:21:54.778266+0000","last_fullsized":"2024-03-06T07:21:54.778266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777481+0000","last_change":"2024-03-06T07:21:52.555948+0000","last_active":"2024-03-06T07:21:54.777481+0000","last_peered":"2024-03-06T07:21:54.777481+0000","last_clean":"2024-03-06T07:21:54.777481+0000","last_became_active":"2024-03-06T07:21:52.555657+0000","last_became_peered":"2024-03-06T07:21:52.555657+0000","last_unstale":"2024-03-06T07:21:54.777481+0000","last_undegraded":"2024-03-06T07:21:54.777481+0000","last_fullsized":"2024-03-06T07:21:54.777481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777191+0000","last_change":"2024-03-06T07:21:53.556393+0000","last_active":"2024-03-06T07:21:54.777191+0000","last_peered":"2024-03-06T07:21:54.777191+0000","last_clean":"2024-03-06T07:21:54.777191+0000","last_became_active":"2024-03-06T07:21:53.556137+0000","last_became_peered":"2024-03-06T07:21:53.556137+0000","last_unstale":"2024-03-06T07:21:54.777191+0000","last_undegraded":"2024-03-06T07:21:54.777191+0000","last_fullsized":"2024-03-06T07:21:54.777191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:33.561906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778025+0000","last_change":"2024-03-06T07:21:52.556426+0000","last_active":"2024-03-06T07:21:54.778025+0000","last_peered":"2024-03-06T07:21:54.778025+0000","last_clean":"2024-03-06T07:21:54.778025+0000","last_became_active":"2024-03-06T07:21:52.555844+0000","last_became_peered":"2024-03-06T07:21:52.555844+0000","last_unstale":"2024-03-06T07:21:54.778025+0000","last_undegraded":"2024-03-06T07:21:54.778025+0000","last_fullsized":"2024-03-06T07:21:54.778025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778299+0000","last_change":"2024-03-06T07:21:53.567432+0000","last_active":"2024-03-06T07:21:54.778299+0000","last_peered":"2024-03-06T07:21:54.778299+0000","last_clean":"2024-03-06T07:21:54.778299+0000","last_became_active":"2024-03-06T07:21:53.566788+0000","last_became_peered":"2024-03-06T07:21:53.566788+0000","last_unstale":"2024-03-06T07:21:54.778299+0000","last_undegraded":"2024-03-06T07:21:54.778299+0000","last_fullsized":"2024-03-06T07:21:54.778299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562338+0000","last_change":"2024-03-06T07:21:52.547444+0000","last_active":"2024-03-06T07:21:54.562338+0000","last_peered":"2024-03-06T07:21:54.562338+0000","last_clean":"2024-03-06T07:21:54.562338+0000","last_became_active":"2024-03-06T07:21:52.547163+0000","last_became_peered":"2024-03-06T07:21:52.547163+0000","last_unstale":"2024-03-06T07:21:54.562338+0000","last_undegraded":"2024-03-06T07:21:54.562338+0000","last_fullsized":"2024-03-06T07:21:54.562338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562392+0000","last_change":"2024-03-06T07:21:53.552210+0000","last_active":"2024-03-06T07:21:54.562392+0000","last_peered":"2024-03-06T07:21:54.562392+0000","last_clean":"2024-03-06T07:21:54.562392+0000","last_became_active":"2024-03-06T07:21:53.551529+0000","last_became_peered":"2024-03-06T07:21:53.551529+0000","last_unstale":"2024-03-06T07:21:54.562392+0000","last_undegraded":"2024-03-06T07:21:54.562392+0000","last_fullsized":"2024-03-06T07:21:54.562392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560082+0000","last_change":"2024-03-06T07:21:52.552286+0000","last_active":"2024-03-06T07:21:54.560082+0000","last_peered":"2024-03-06T07:21:54.560082+0000","last_clean":"2024-03-06T07:21:54.560082+0000","last_became_active":"2024-03-06T07:21:52.552010+0000","last_became_peered":"2024-03-06T07:21:52.552010+0000","last_unstale":"2024-03-06T07:21:54.560082+0000","last_undegraded":"2024-03-06T07:21:54.560082+0000","last_fullsized":"2024-03-06T07:21:54.560082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777580+0000","last_change":"2024-03-06T07:21:53.571841+0000","last_active":"2024-03-06T07:21:54.777580+0000","last_peered":"2024-03-06T07:21:54.777580+0000","last_clean":"2024-03-06T07:21:54.777580+0000","last_became_active":"2024-03-06T07:21:53.571350+0000","last_became_peered":"2024-03-06T07:21:53.571350+0000","last_unstale":"2024-03-06T07:21:54.777580+0000","last_undegraded":"2024-03-06T07:21:54.777580+0000","last_fullsized":"2024-03-06T07:21:54.777580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777653+0000","last_change":"2024-03-06T07:21:52.553990+0000","last_active":"2024-03-06T07:21:54.777653+0000","last_peered":"2024-03-06T07:21:54.777653+0000","last_clean":"2024-03-06T07:21:54.777653+0000","last_became_active":"2024-03-06T07:21:52.553693+0000","last_became_peered":"2024-03-06T07:21:52.553693+0000","last_unstale":"2024-03-06T07:21:54.777653+0000","last_undegraded":"2024-03-06T07:21:54.777653+0000","last_fullsized":"2024-03-06T07:21:54.777653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777984+0000","last_change":"2024-03-06T07:21:53.567545+0000","last_active":"2024-03-06T07:21:54.777984+0000","last_peered":"2024-03-06T07:21:54.777984+0000","last_clean":"2024-03-06T07:21:54.777984+0000","last_became_active":"2024-03-06T07:21:53.566157+0000","last_became_peered":"2024-03-06T07:21:53.566157+0000","last_unstale":"2024-03-06T07:21:54.777984+0000","last_undegraded":"2024-03-06T07:21:54.777984+0000","last_fullsized":"2024-03-06T07:21:54.777984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560201+0000","last_change":"2024-03-06T07:21:52.547686+0000","last_active":"2024-03-06T07:21:54.560201+0000","last_peered":"2024-03-06T07:21:54.560201+0000","last_clean":"2024-03-06T07:21:54.560201+0000","last_became_active":"2024-03-06T07:21:52.547436+0000","last_became_peered":"2024-03-06T07:21:52.547436+0000","last_unstale":"2024-03-06T07:21:54.560201+0000","last_undegraded":"2024-03-06T07:21:54.560201+0000","last_fullsized":"2024-03-06T07:21:54.560201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778797+0000","last_change":"2024-03-06T07:21:53.567657+0000","last_active":"2024-03-06T07:21:54.778797+0000","last_peered":"2024-03-06T07:21:54.778797+0000","last_clean":"2024-03-06T07:21:54.778797+0000","last_became_active":"2024-03-06T07:21:53.567134+0000","last_became_peered":"2024-03-06T07:21:53.567134+0000","last_unstale":"2024-03-06T07:21:54.778797+0000","last_undegraded":"2024-03-06T07:21:54.778797+0000","last_fullsized":"2024-03-06T07:21:54.778797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:57.048252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562696+0000","last_change":"2024-03-06T07:21:52.556768+0000","last_active":"2024-03-06T07:21:54.562696+0000","last_peered":"2024-03-06T07:21:54.562696+0000","last_clean":"2024-03-06T07:21:54.562696+0000","last_became_active":"2024-03-06T07:21:52.556507+0000","last_became_peered":"2024-03-06T07:21:52.556507+0000","last_unstale":"2024-03-06T07:21:54.562696+0000","last_undegraded":"2024-03-06T07:21:54.562696+0000","last_fullsized":"2024-03-06T07:21:54.562696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778541+0000","last_change":"2024-03-06T07:21:53.563768+0000","last_active":"2024-03-06T07:21:54.778541+0000","last_peered":"2024-03-06T07:21:54.778541+0000","last_clean":"2024-03-06T07:21:54.778541+0000","last_became_active":"2024-03-06T07:21:53.563490+0000","last_became_peered":"2024-03-06T07:21:53.563490+0000","last_unstale":"2024-03-06T07:21:54.778541+0000","last_undegraded":"2024-03-06T07:21:54.778541+0000","last_fullsized":"2024-03-06T07:21:54.778541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560265+0000","last_change":"2024-03-06T07:21:52.557535+0000","last_active":"2024-03-06T07:21:54.560265+0000","last_peered":"2024-03-06T07:21:54.560265+0000","last_clean":"2024-03-06T07:21:54.560265+0000","last_became_active":"2024-03-06T07:21:52.557275+0000","last_became_peered":"2024-03-06T07:21:52.557275+0000","last_unstale":"2024-03-06T07:21:54.560265+0000","last_undegraded":"2024-03-06T07:21:54.560265+0000","last_fullsized":"2024-03-06T07:21:54.560265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778386+0000","last_change":"2024-03-06T07:21:53.565629+0000","last_active":"2024-03-06T07:21:54.778386+0000","last_peered":"2024-03-06T07:21:54.778386+0000","last_clean":"2024-03-06T07:21:54.778386+0000","last_became_active":"2024-03-06T07:21:53.563789+0000","last_became_peered":"2024-03-06T07:21:53.563789+0000","last_unstale":"2024-03-06T07:21:54.778386+0000","last_undegraded":"2024-03-06T07:21:54.778386+0000","last_fullsized":"2024-03-06T07:21:54.778386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778063+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:54.778063+0000","last_peered":"2024-03-06T07:21:54.778063+0000","last_clean":"2024-03-06T07:21:54.778063+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:54.778063+0000","last_undegraded":"2024-03-06T07:21:54.778063+0000","last_fullsized":"2024-03-06T07:21:54.778063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777283+0000","last_change":"2024-03-06T07:21:52.557611+0000","last_active":"2024-03-06T07:21:54.777283+0000","last_peered":"2024-03-06T07:21:54.777283+0000","last_clean":"2024-03-06T07:21:54.777283+0000","last_became_active":"2024-03-06T07:21:52.556594+0000","last_became_peered":"2024-03-06T07:21:52.556594+0000","last_unstale":"2024-03-06T07:21:54.777283+0000","last_undegraded":"2024-03-06T07:21:54.777283+0000","last_fullsized":"2024-03-06T07:21:54.777283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767209+0000","last_change":"2024-03-06T07:21:53.565607+0000","last_active":"2024-03-06T07:21:54.767209+0000","last_peered":"2024-03-06T07:21:54.767209+0000","last_clean":"2024-03-06T07:21:54.767209+0000","last_became_active":"2024-03-06T07:21:53.565356+0000","last_became_peered":"2024-03-06T07:21:53.565356+0000","last_unstale":"2024-03-06T07:21:54.767209+0000","last_undegraded":"2024-03-06T07:21:54.767209+0000","last_fullsized":"2024-03-06T07:21:54.767209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778021+0000","last_change":"2024-03-06T07:21:52.555927+0000","last_active":"2024-03-06T07:21:54.778021+0000","last_peered":"2024-03-06T07:21:54.778021+0000","last_clean":"2024-03-06T07:21:54.778021+0000","last_became_active":"2024-03-06T07:21:52.555485+0000","last_became_peered":"2024-03-06T07:21:52.555485+0000","last_unstale":"2024-03-06T07:21:54.778021+0000","last_undegraded":"2024-03-06T07:21:54.778021+0000","last_fullsized":"2024-03-06T07:21:54.778021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777097+0000","last_change":"2024-03-06T07:21:53.559225+0000","last_active":"2024-03-06T07:21:54.777097+0000","last_peered":"2024-03-06T07:21:54.777097+0000","last_clean":"2024-03-06T07:21:54.777097+0000","last_became_active":"2024-03-06T07:21:53.558925+0000","last_became_peered":"2024-03-06T07:21:53.558925+0000","last_unstale":"2024-03-06T07:21:54.777097+0000","last_undegraded":"2024-03-06T07:21:54.777097+0000","last_fullsized":"2024-03-06T07:21:54.777097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:54.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560174+0000","last_change":"2024-03-06T07:21:52.546620+0000","last_active":"2024-03-06T07:21:54.560174+0000","last_peered":"2024-03-06T07:21:54.560174+0000","last_clean":"2024-03-06T07:21:54.560174+0000","last_became_active":"2024-03-06T07:21:52.546297+0000","last_became_peered":"2024-03-06T07:21:52.546297+0000","last_unstale":"2024-03-06T07:21:54.560174+0000","last_undegraded":"2024-03-06T07:21:54.560174+0000","last_fullsized":"2024-03-06T07:21:54.560174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562488+0000","last_change":"2024-03-06T07:21:53.553218+0000","last_active":"2024-03-06T07:21:54.562488+0000","last_peered":"2024-03-06T07:21:54.562488+0000","last_clean":"2024-03-06T07:21:54.562488+0000","last_became_active":"2024-03-06T07:21:53.552911+0000","last_became_peered":"2024-03-06T07:21:53.552911+0000","last_unstale":"2024-03-06T07:21:54.562488+0000","last_undegraded":"2024-03-06T07:21:54.562488+0000","last_fullsized":"2024-03-06T07:21:54.562488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778113+0000","last_change":"2024-03-06T07:21:52.556096+0000","last_active":"2024-03-06T07:21:54.778113+0000","last_peered":"2024-03-06T07:21:54.778113+0000","last_clean":"2024-03-06T07:21:54.778113+0000","last_became_active":"2024-03-06T07:21:52.555510+0000","last_became_peered":"2024-03-06T07:21:52.555510+0000","last_unstale":"2024-03-06T07:21:54.778113+0000","last_undegraded":"2024-03-06T07:21:54.778113+0000","last_fullsized":"2024-03-06T07:21:54.778113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778491+0000","last_change":"2024-03-06T07:21:53.561767+0000","last_active":"2024-03-06T07:21:54.778491+0000","last_peered":"2024-03-06T07:21:54.778491+0000","last_clean":"2024-03-06T07:21:54.778491+0000","last_became_active":"2024-03-06T07:21:53.561435+0000","last_became_peered":"2024-03-06T07:21:53.561435+0000","last_unstale":"2024-03-06T07:21:54.778491+0000","last_undegraded":"2024-03-06T07:21:54.778491+0000","last_fullsized":"2024-03-06T07:21:54.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:24.631055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562673+0000","last_change":"2024-03-06T07:21:52.553733+0000","last_active":"2024-03-06T07:21:54.562673+0000","last_peered":"2024-03-06T07:21:54.562673+0000","last_clean":"2024-03-06T07:21:54.562673+0000","last_became_active":"2024-03-06T07:21:52.553436+0000","last_became_peered":"2024-03-06T07:21:52.553436+0000","last_unstale":"2024-03-06T07:21:54.562673+0000","last_undegraded":"2024-03-06T07:21:54.562673+0000","last_fullsized":"2024-03-06T07:21:54.562673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779356+0000","last_change":"2024-03-06T07:21:53.556153+0000","last_active":"2024-03-06T07:21:54.779356+0000","last_peered":"2024-03-06T07:21:54.779356+0000","last_clean":"2024-03-06T07:21:54.779356+0000","last_became_active":"2024-03-06T07:21:53.555896+0000","last_became_peered":"2024-03-06T07:21:53.555896+0000","last_unstale":"2024-03-06T07:21:54.779356+0000","last_undegraded":"2024-03-06T07:21:54.779356+0000","last_fullsized":"2024-03-06T07:21:54.779356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778511+0000","last_change":"2024-03-06T07:21:52.557084+0000","last_active":"2024-03-06T07:21:54.778511+0000","last_peered":"2024-03-06T07:21:54.778511+0000","last_clean":"2024-03-06T07:21:54.778511+0000","last_became_active":"2024-03-06T07:21:52.556688+0000","last_became_peered":"2024-03-06T07:21:52.556688+0000","last_unstale":"2024-03-06T07:21:54.778511+0000","last_undegraded":"2024-03-06T07:21:54.778511+0000","last_fullsized":"2024-03-06T07:21:54.778511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:23.626336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562632+0000","last_change":"2024-03-06T07:21:53.552698+0000","last_active":"2024-03-06T07:21:54.562632+0000","last_peered":"2024-03-06T07:21:54.562632+0000","last_clean":"2024-03-06T07:21:54.562632+0000","last_became_active":"2024-03-06T07:21:53.552434+0000","last_became_peered":"2024-03-06T07:21:53.552434+0000","last_unstale":"2024-03-06T07:21:54.562632+0000","last_undegraded":"2024-03-06T07:21:54.562632+0000","last_fullsized":"2024-03-06T07:21:54.562632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562589+0000","last_change":"2024-03-06T07:21:52.553671+0000","last_active":"2024-03-06T07:21:54.562589+0000","last_peered":"2024-03-06T07:21:54.562589+0000","last_clean":"2024-03-06T07:21:54.562589+0000","last_became_active":"2024-03-06T07:21:52.553394+0000","last_became_peered":"2024-03-06T07:21:52.553394+0000","last_unstale":"2024-03-06T07:21:54.562589+0000","last_undegraded":"2024-03-06T07:21:54.562589+0000","last_fullsized":"2024-03-06T07:21:54.562589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767449+0000","last_change":"2024-03-06T07:21:53.555702+0000","last_active":"2024-03-06T07:21:54.767449+0000","last_peered":"2024-03-06T07:21:54.767449+0000","last_clean":"2024-03-06T07:21:54.767449+0000","last_became_active":"2024-03-06T07:21:53.555436+0000","last_became_peered":"2024-03-06T07:21:53.555436+0000","last_unstale":"2024-03-06T07:21:54.767449+0000","last_undegraded":"2024-03-06T07:21:54.767449+0000","last_fullsized":"2024-03-06T07:21:54.767449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562639+0000","last_change":"2024-03-06T07:21:52.550448+0000","last_active":"2024-03-06T07:21:54.562639+0000","last_peered":"2024-03-06T07:21:54.562639+0000","last_clean":"2024-03-06T07:21:54.562639+0000","last_became_active":"2024-03-06T07:21:52.550179+0000","last_became_peered":"2024-03-06T07:21:52.550179+0000","last_unstale":"2024-03-06T07:21:54.562639+0000","last_undegraded":"2024-03-06T07:21:54.562639+0000","last_fullsized":"2024-03-06T07:21:54.562639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777421+0000","last_change":"2024-03-06T07:21:53.553031+0000","last_active":"2024-03-06T07:21:54.777421+0000","last_peered":"2024-03-06T07:21:54.777421+0000","last_clean":"2024-03-06T07:21:54.777421+0000","last_became_active":"2024-03-06T07:21:53.552759+0000","last_became_peered":"2024-03-06T07:21:53.552759+0000","last_unstale":"2024-03-06T07:21:54.777421+0000","last_undegraded":"2024-03-06T07:21:54.777421+0000","last_fullsized":"2024-03-06T07:21:54.777421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777184+0000","last_change":"2024-03-06T07:21:52.555781+0000","last_active":"2024-03-06T07:21:54.777184+0000","last_peered":"2024-03-06T07:21:54.777184+0000","last_clean":"2024-03-06T07:21:54.777184+0000","last_became_active":"2024-03-06T07:21:52.555469+0000","last_became_peered":"2024-03-06T07:21:52.555469+0000","last_unstale":"2024-03-06T07:21:54.777184+0000","last_undegraded":"2024-03-06T07:21:54.777184+0000","last_fullsized":"2024-03-06T07:21:54.777184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777797+0000","last_change":"2024-03-06T07:21:53.558488+0000","last_active":"2024-03-06T07:21:54.777797+0000","last_peered":"2024-03-06T07:21:54.777797+0000","last_clean":"2024-03-06T07:21:54.777797+0000","last_became_active":"2024-03-06T07:21:53.558199+0000","last_became_peered":"2024-03-06T07:21:53.558199+0000","last_unstale":"2024-03-06T07:21:54.777797+0000","last_undegraded":"2024-03-06T07:21:54.777797+0000","last_fullsized":"2024-03-06T07:21:54.777797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562638+0000","last_change":"2024-03-06T07:21:52.556952+0000","last_active":"2024-03-06T07:21:54.562638+0000","last_peered":"2024-03-06T07:21:54.562638+0000","last_clean":"2024-03-06T07:21:54.562638+0000","last_became_active":"2024-03-06T07:21:52.556629+0000","last_became_peered":"2024-03-06T07:21:52.556629+0000","last_unstale":"2024-03-06T07:21:54.562638+0000","last_undegraded":"2024-03-06T07:21:54.562638+0000","last_fullsized":"2024-03-06T07:21:54.562638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767159+0000","last_change":"2024-03-06T07:21:53.554655+0000","last_active":"2024-03-06T07:21:54.767159+0000","last_peered":"2024-03-06T07:21:54.767159+0000","last_clean":"2024-03-06T07:21:54.767159+0000","last_became_active":"2024-03-06T07:21:53.554321+0000","last_became_peered":"2024-03-06T07:21:53.554321+0000","last_unstale":"2024-03-06T07:21:54.767159+0000","last_undegraded":"2024-03-06T07:21:54.767159+0000","last_fullsized":"2024-03-06T07:21:54.767159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562564+0000","last_change":"2024-03-06T07:21:52.547548+0000","last_active":"2024-03-06T07:21:54.562564+0000","last_peered":"2024-03-06T07:21:54.562564+0000","last_clean":"2024-03-06T07:21:54.562564+0000","last_became_active":"2024-03-06T07:21:52.547285+0000","last_became_peered":"2024-03-06T07:21:52.547285+0000","last_unstale":"2024-03-06T07:21:54.562564+0000","last_undegraded":"2024-03-06T07:21:54.562564+0000","last_fullsized":"2024-03-06T07:21:54.562564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767314+0000","last_change":"2024-03-06T07:21:53.557506+0000","last_active":"2024-03-06T07:21:54.767314+0000","last_peered":"2024-03-06T07:21:54.767314+0000","last_clean":"2024-03-06T07:21:54.767314+0000","last_became_active":"2024-03-06T07:21:53.556568+0000","last_became_peered":"2024-03-06T07:21:53.556568+0000","last_unstale":"2024-03-06T07:21:54.767314+0000","last_undegraded":"2024-03-06T07:21:54.767314+0000","last_fullsized":"2024-03-06T07:21:54.767314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778329+0000","last_change":"2024-03-06T07:21:52.558820+0000","last_active":"2024-03-06T07:21:54.778329+0000","last_peered":"2024-03-06T07:21:54.778329+0000","last_clean":"2024-03-06T07:21:54.778329+0000","last_became_active":"2024-03-06T07:21:52.558486+0000","last_became_peered":"2024-03-06T07:21:52.558486+0000","last_unstale":"2024-03-06T07:21:54.778329+0000","last_undegraded":"2024-03-06T07:21:54.778329+0000","last_fullsized":"2024-03-06T07:21:54.778329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562402+0000","last_change":"2024-03-06T07:21:53.558230+0000","last_active":"2024-03-06T07:21:54.562402+0000","last_peered":"2024-03-06T07:21:54.562402+0000","last_clean":"2024-03-06T07:21:54.562402+0000","last_became_active":"2024-03-06T07:21:53.557916+0000","last_became_peered":"2024-03-06T07:21:53.557916+0000","last_unstale":"2024-03-06T07:21:54.562402+0000","last_undegraded":"2024-03-06T07:21:54.562402+0000","last_fullsized":"2024-03-06T07:21:54.562402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779232+0000","last_change":"2024-03-06T07:21:53.567503+0000","last_active":"2024-03-06T07:21:54.779232+0000","last_peered":"2024-03-06T07:21:54.779232+0000","last_clean":"2024-03-06T07:21:54.779232+0000","last_became_active":"2024-03-06T07:21:53.567223+0000","last_became_peered":"2024-03-06T07:21:53.567223+0000","last_unstale":"2024-03-06T07:21:54.779232+0000","last_undegraded":"2024-03-06T07:21:54.779232+0000","last_fullsized":"2024-03-06T07:21:54.779232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777697+0000","last_change":"2024-03-06T07:21:52.559682+0000","last_active":"2024-03-06T07:21:54.777697+0000","last_peered":"2024-03-06T07:21:54.777697+0000","last_clean":"2024-03-06T07:21:54.777697+0000","last_became_active":"2024-03-06T07:21:52.559430+0000","last_became_peered":"2024-03-06T07:21:52.559430+0000","last_unstale":"2024-03-06T07:21:54.777697+0000","last_undegraded":"2024-03-06T07:21:54.777697+0000","last_fullsized":"2024-03-06T07:21:54.777697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778415+0000","last_change":"2024-03-06T07:21:53.568862+0000","last_active":"2024-03-06T07:21:54.778415+0000","last_peered":"2024-03-06T07:21:54.778415+0000","last_clean":"2024-03-06T07:21:54.778415+0000","last_became_active":"2024-03-06T07:21:53.568127+0000","last_became_peered":"2024-03-06T07:21:53.568127+0000","last_unstale":"2024-03-06T07:21:54.778415+0000","last_undegraded":"2024-03-06T07:21:54.778415+0000","last_fullsized":"2024-03-06T07:21:54.778415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778429+0000","last_change":"2024-03-06T07:21:52.558983+0000","last_active":"2024-03-06T07:21:54.778429+0000","last_peered":"2024-03-06T07:21:54.778429+0000","last_clean":"2024-03-06T07:21:54.778429+0000","last_became_active":"2024-03-06T07:21:52.558616+0000","last_became_peered":"2024-03-06T07:21:52.558616+0000","last_unstale":"2024-03-06T07:21:54.778429+0000","last_undegraded":"2024-03-06T07:21:54.778429+0000","last_fullsized":"2024-03-06T07:21:54.778429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777374+0000","last_change":"2024-03-06T07:21:53.558956+0000","last_active":"2024-03-06T07:21:54.777374+0000","last_peered":"2024-03-06T07:21:54.777374+0000","last_clean":"2024-03-06T07:21:54.777374+0000","last_became_active":"2024-03-06T07:21:53.558612+0000","last_became_peered":"2024-03-06T07:21:53.558612+0000","last_unstale":"2024-03-06T07:21:54.777374+0000","last_undegraded":"2024-03-06T07:21:54.777374+0000","last_fullsized":"2024-03-06T07:21:54.777374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767442+0000","last_change":"2024-03-06T07:21:52.553080+0000","last_active":"2024-03-06T07:21:54.767442+0000","last_peered":"2024-03-06T07:21:54.767442+0000","last_clean":"2024-03-06T07:21:54.767442+0000","last_became_active":"2024-03-06T07:21:52.552785+0000","last_became_peered":"2024-03-06T07:21:52.552785+0000","last_unstale":"2024-03-06T07:21:54.767442+0000","last_undegraded":"2024-03-06T07:21:54.767442+0000","last_fullsized":"2024-03-06T07:21:54.767442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767397+0000","last_change":"2024-03-06T07:21:53.563091+0000","last_active":"2024-03-06T07:21:54.767397+0000","last_peered":"2024-03-06T07:21:54.767397+0000","last_clean":"2024-03-06T07:21:54.767397+0000","last_became_active":"2024-03-06T07:21:53.562797+0000","last_became_peered":"2024-03-06T07:21:53.562797+0000","last_unstale":"2024-03-06T07:21:54.767397+0000","last_undegraded":"2024-03-06T07:21:54.767397+0000","last_fullsized":"2024-03-06T07:21:54.767397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777504+0000","last_change":"2024-03-06T07:21:52.557097+0000","last_active":"2024-03-06T07:21:54.777504+0000","last_peered":"2024-03-06T07:21:54.777504+0000","last_clean":"2024-03-06T07:21:54.777504+0000","last_became_active":"2024-03-06T07:21:52.556808+0000","last_became_peered":"2024-03-06T07:21:52.556808+0000","last_unstale":"2024-03-06T07:21:54.777504+0000","last_undegraded":"2024-03-06T07:21:54.777504+0000","last_fullsized":"2024-03-06T07:21:54.777504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:05.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:05.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.693+0000 7f358c615640 1 Processor -- start 2024-03-06T07:22:05.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.693+0000 7f358c615640 1 -- start start 2024-03-06T07:22:05.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.693+0000 7f358c615640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f35840d00c0 0x7f35840d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358c615640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f358406cf20 0x7f358406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358c615640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35840d09e0 con 0x7f35840cf850 2024-03-06T07:22:05.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358c615640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358406ca90 con 0x7f35840d00c0 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358c615640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358415e380 con 0x7f358406cf20 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f3589bb0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f35840cf850 0x7f35840cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38562/0 (socket says 172.21.15.122:38562) 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f3589bb0640 1 -- 172.21.15.122:0/4068106261 learned_addr learned my addr 172.21.15.122:0/4068106261 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358abb2640 1 --2- 172.21.15.122:0/4068106261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f358406cf20 0x7f358406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358a3b1640 1 --2- 172.21.15.122:0/4068106261 >> v2:172.21.15.169:3300/0 conn(0x7f35840d00c0 0x7f35840d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f35893af640 1 -- 172.21.15.122:0/4068106261 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558308762 0 0) 0x7f35840d09e0 con 0x7f35840cf850 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f35893af640 1 -- 172.21.15.122:0/4068106261 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3570003520 con 0x7f35840cf850 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358abb2640 1 -- 172.21.15.122:0/4068106261 >> v2:172.21.15.169:3300/0 conn(0x7f35840d00c0 msgr2=0x7f35840d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358abb2640 1 --2- 172.21.15.122:0/4068106261 >> v2:172.21.15.169:3300/0 conn(0x7f35840d00c0 0x7f35840d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358abb2640 1 -- 172.21.15.122:0/4068106261 >> v1:172.21.15.122:6789/0 conn(0x7f35840cf850 legacy=0x7f35840cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358abb2640 1 -- 172.21.15.122:0/4068106261 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35840cdc10 con 0x7f358406cf20 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.694+0000 7f358a3b1640 1 --2- 172.21.15.122:0/4068106261 >> v2:172.21.15.169:3300/0 conn(0x7f35840d00c0 0x7f35840d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.695+0000 7f358abb2640 1 --2- 172.21.15.122:0/4068106261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f358406cf20 0x7f358406d3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f357c009d90 tx=0x7f357c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=284147c90a9b47b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.695+0000 7f35893af640 1 -- 172.21.15.122:0/4068106261 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c039070 con 0x7f358406cf20 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.695+0000 7f35893af640 1 -- 172.21.15.122:0/4068106261 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c02cc50 con 0x7f358406cf20 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.695+0000 7f35893af640 1 -- 172.21.15.122:0/4068106261 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c034440 con 0x7f358406cf20 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.695+0000 7f358c615640 1 -- 172.21.15.122:0/4068106261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f358406cf20 msgr2=0x7f358406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.695+0000 7f358c615640 1 --2- 172.21.15.122:0/4068106261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f358406cf20 0x7f358406d3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f357c009d90 tx=0x7f357c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.696+0000 7f358c615640 1 -- 172.21.15.122:0/4068106261 shutdown_connections 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.696+0000 7f358c615640 1 --2- 172.21.15.122:0/4068106261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f358406cf20 0x7f358406d3b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.696+0000 7f358c615640 1 --2- 172.21.15.122:0/4068106261 >> v2:172.21.15.169:3300/0 conn(0x7f35840d00c0 0x7f35840d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.696+0000 7f358c615640 1 -- 172.21.15.122:0/4068106261 >> 172.21.15.122:0/4068106261 conn(0x7f35840b35a0 msgr2=0x7f35840b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.696+0000 7f358c615640 1 -- 172.21.15.122:0/4068106261 shutdown_connections 2024-03-06T07:22:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.696+0000 7f358c615640 1 -- 172.21.15.122:0/4068106261 wait complete. 2024-03-06T07:22:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 Processor -- start 2024-03-06T07:22:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 -- start start 2024-03-06T07:22:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 0x7f3584159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35840cf850 0x7f3584158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35840d00c0 0x7f35840b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358406c720 con 0x7f358406cf20 2024-03-06T07:22:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358406c440 con 0x7f35840d00c0 2024-03-06T07:22:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358c615640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35840b1030 con 0x7f35840cf850 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.697+0000 7f358a3b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 0x7f3584159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 0x7f3584159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57064/0 (socket says 172.21.15.122:57064) 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 -- 172.21.15.122:0/3720423311 learned_addr learned my addr 172.21.15.122:0/3720423311 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358abb2640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35840d00c0 0x7f35840b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f3589bb0640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35840cf850 0x7f3584158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 -- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35840cf850 msgr2=0x7f3584158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35840cf850 0x7f3584158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 -- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35840d00c0 msgr2=0x7f35840b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35840d00c0 0x7f35840b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 -- 172.21.15.122:0/3720423311 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358415b380 con 0x7f358406cf20 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358a3b1640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 0x7f3584159000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f358000ec10 tx=0x7f358000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580017070 con 0x7f358406cf20 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358000cc30 con 0x7f358406cf20 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358415e630 con 0x7f358406cf20 2024-03-06T07:22:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580013470 con 0x7f358406cf20 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.698+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358415ebb0 con 0x7f358406cf20 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.699+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358404f330 con 0x7f358406cf20 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.700+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f358001a070 con 0x7f358406cf20 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.700+0000 7f357affd640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f356806b4d0 0x7f356806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.700+0000 7f3589bb0640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f356806b4d0 0x7f356806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.700+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f358008a120 con 0x7f358406cf20 2024-03-06T07:22:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.700+0000 7f3589bb0640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f356806b4d0 0x7f356806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f358406df20 tx=0x7f3574009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.704+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3580058950 con 0x7f358406cf20 2024-03-06T07:22:05.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.874+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35840b2020 con 0x7f356806b4d0 2024-03-06T07:22:05.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.880+0000 7f357affd640 1 -- 172.21.15.122:0/3720423311 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f35840b2020 con 0x7f356806b4d0 2024-03-06T07:22:05.887 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:05.889 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:05.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f356806b4d0 msgr2=0x7f356806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f356806b4d0 0x7f356806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f358406df20 tx=0x7f3574009040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 msgr2=0x7f3584159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 0x7f3584159000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f358000ec10 tx=0x7f358000c520 comp rx=0 tx=0).stop 2024-03-06T07:22:05.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 shutdown_connections 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f356806b4d0 0x7f356806d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35840d00c0 0x7f35840b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35840cf850 0x7f3584158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 --2- 172.21.15.122:0/3720423311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f358406cf20 0x7f3584159000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 >> 172.21.15.122:0/3720423311 conn(0x7f35840b35a0 msgr2=0x7f35840b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 shutdown_connections 2024-03-06T07:22:05.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:05.883+0000 7f358c615640 1 -- 172.21.15.122:0/3720423311 wait complete. 2024-03-06T07:22:05.896 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-06T07:22:05.333171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218996,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755724,"statfs":{"total":773094113280,"available":772869861376,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.774147"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778717+0000","last_change":"2024-03-06T07:21:53.563999+0000","last_active":"2024-03-06T07:21:54.778717+0000","last_peered":"2024-03-06T07:21:54.778717+0000","last_clean":"2024-03-06T07:21:54.778717+0000","last_became_active":"2024-03-06T07:21:53.563557+0000","last_became_peered":"2024-03-06T07:21:53.563557+0000","last_unstale":"2024-03-06T07:21:54.778717+0000","last_undegraded":"2024-03-06T07:21:54.778717+0000","last_fullsized":"2024-03-06T07:21:54.778717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562641+0000","last_change":"2024-03-06T07:21:53.560952+0000","last_active":"2024-03-06T07:21:54.562641+0000","last_peered":"2024-03-06T07:21:54.562641+0000","last_clean":"2024-03-06T07:21:54.562641+0000","last_became_active":"2024-03-06T07:21:53.560651+0000","last_became_peered":"2024-03-06T07:21:53.560651+0000","last_unstale":"2024-03-06T07:21:54.562641+0000","last_undegraded":"2024-03-06T07:21:54.562641+0000","last_fullsized":"2024-03-06T07:21:54.562641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777337+0000","last_change":"2024-03-06T07:21:53.547890+0000","last_active":"2024-03-06T07:21:54.777337+0000","last_peered":"2024-03-06T07:21:54.777337+0000","last_clean":"2024-03-06T07:21:54.777337+0000","last_became_active":"2024-03-06T07:21:53.547631+0000","last_became_peered":"2024-03-06T07:21:53.547631+0000","last_unstale":"2024-03-06T07:21:54.777337+0000","last_undegraded":"2024-03-06T07:21:54.777337+0000","last_fullsized":"2024-03-06T07:21:54.777337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778291+0000","last_change":"2024-03-06T07:21:53.565535+0000","last_active":"2024-03-06T07:21:54.778291+0000","last_peered":"2024-03-06T07:21:54.778291+0000","last_clean":"2024-03-06T07:21:54.778291+0000","last_became_active":"2024-03-06T07:21:53.565256+0000","last_became_peered":"2024-03-06T07:21:53.565256+0000","last_unstale":"2024-03-06T07:21:54.778291+0000","last_undegraded":"2024-03-06T07:21:54.778291+0000","last_fullsized":"2024-03-06T07:21:54.778291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778186+0000","last_change":"2024-03-06T07:21:53.558864+0000","last_active":"2024-03-06T07:21:54.778186+0000","last_peered":"2024-03-06T07:21:54.778186+0000","last_clean":"2024-03-06T07:21:54.778186+0000","last_became_active":"2024-03-06T07:21:53.558582+0000","last_became_peered":"2024-03-06T07:21:53.558582+0000","last_unstale":"2024-03-06T07:21:54.778186+0000","last_undegraded":"2024-03-06T07:21:54.778186+0000","last_fullsized":"2024-03-06T07:21:54.778186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777685+0000","last_change":"2024-03-06T07:21:53.567354+0000","last_active":"2024-03-06T07:21:54.777685+0000","last_peered":"2024-03-06T07:21:54.777685+0000","last_clean":"2024-03-06T07:21:54.777685+0000","last_became_active":"2024-03-06T07:21:53.567054+0000","last_became_peered":"2024-03-06T07:21:53.567054+0000","last_unstale":"2024-03-06T07:21:54.777685+0000","last_undegraded":"2024-03-06T07:21:54.777685+0000","last_fullsized":"2024-03-06T07:21:54.777685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777444+0000","last_change":"2024-03-06T07:21:53.563005+0000","last_active":"2024-03-06T07:21:54.777444+0000","last_peered":"2024-03-06T07:21:54.777444+0000","last_clean":"2024-03-06T07:21:54.777444+0000","last_became_active":"2024-03-06T07:21:53.562611+0000","last_became_peered":"2024-03-06T07:21:53.562611+0000","last_unstale":"2024-03-06T07:21:54.777444+0000","last_undegraded":"2024-03-06T07:21:54.777444+0000","last_fullsized":"2024-03-06T07:21:54.777444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.766995+0000","last_change":"2024-03-06T07:21:53.566285+0000","last_active":"2024-03-06T07:21:54.766995+0000","last_peered":"2024-03-06T07:21:54.766995+0000","last_clean":"2024-03-06T07:21:54.766995+0000","last_became_active":"2024-03-06T07:21:53.565429+0000","last_became_peered":"2024-03-06T07:21:53.565429+0000","last_unstale":"2024-03-06T07:21:54.766995+0000","last_undegraded":"2024-03-06T07:21:54.766995+0000","last_fullsized":"2024-03-06T07:21:54.766995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778770+0000","last_change":"2024-03-06T07:21:53.565916+0000","last_active":"2024-03-06T07:21:54.778770+0000","last_peered":"2024-03-06T07:21:54.778770+0000","last_clean":"2024-03-06T07:21:54.778770+0000","last_became_active":"2024-03-06T07:21:53.565555+0000","last_became_peered":"2024-03-06T07:21:53.565555+0000","last_unstale":"2024-03-06T07:21:54.778770+0000","last_undegraded":"2024-03-06T07:21:54.778770+0000","last_fullsized":"2024-03-06T07:21:54.778770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:23.946424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562690+0000","last_change":"2024-03-06T07:21:53.557656+0000","last_active":"2024-03-06T07:21:54.562690+0000","last_peered":"2024-03-06T07:21:54.562690+0000","last_clean":"2024-03-06T07:21:54.562690+0000","last_became_active":"2024-03-06T07:21:53.557224+0000","last_became_peered":"2024-03-06T07:21:53.557224+0000","last_unstale":"2024-03-06T07:21:54.562690+0000","last_undegraded":"2024-03-06T07:21:54.562690+0000","last_fullsized":"2024-03-06T07:21:54.562690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562612+0000","last_change":"2024-03-06T07:21:53.565698+0000","last_active":"2024-03-06T07:21:54.562612+0000","last_peered":"2024-03-06T07:21:54.562612+0000","last_clean":"2024-03-06T07:21:54.562612+0000","last_became_active":"2024-03-06T07:21:53.565390+0000","last_became_peered":"2024-03-06T07:21:53.565390+0000","last_unstale":"2024-03-06T07:21:54.562612+0000","last_undegraded":"2024-03-06T07:21:54.562612+0000","last_fullsized":"2024-03-06T07:21:54.562612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777702+0000","last_change":"2024-03-06T07:21:53.558368+0000","last_active":"2024-03-06T07:21:54.777702+0000","last_peered":"2024-03-06T07:21:54.777702+0000","last_clean":"2024-03-06T07:21:54.777702+0000","last_became_active":"2024-03-06T07:21:53.558065+0000","last_became_peered":"2024-03-06T07:21:53.558065+0000","last_unstale":"2024-03-06T07:21:54.777702+0000","last_undegraded":"2024-03-06T07:21:54.777702+0000","last_fullsized":"2024-03-06T07:21:54.777702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:44.794765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767222+0000","last_change":"2024-03-06T07:21:53.567165+0000","last_active":"2024-03-06T07:21:54.767222+0000","last_peered":"2024-03-06T07:21:54.767222+0000","last_clean":"2024-03-06T07:21:54.767222+0000","last_became_active":"2024-03-06T07:21:53.566826+0000","last_became_peered":"2024-03-06T07:21:53.566826+0000","last_unstale":"2024-03-06T07:21:54.767222+0000","last_undegraded":"2024-03-06T07:21:54.767222+0000","last_fullsized":"2024-03-06T07:21:54.767222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777131+0000","last_change":"2024-03-06T07:21:53.552931+0000","last_active":"2024-03-06T07:21:54.777131+0000","last_peered":"2024-03-06T07:21:54.777131+0000","last_clean":"2024-03-06T07:21:54.777131+0000","last_became_active":"2024-03-06T07:21:53.552654+0000","last_became_peered":"2024-03-06T07:21:53.552654+0000","last_unstale":"2024-03-06T07:21:54.777131+0000","last_undegraded":"2024-03-06T07:21:54.777131+0000","last_fullsized":"2024-03-06T07:21:54.777131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:04.669227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562518+0000","last_change":"2024-03-06T07:21:53.562294+0000","last_active":"2024-03-06T07:21:54.562518+0000","last_peered":"2024-03-06T07:21:54.562518+0000","last_clean":"2024-03-06T07:21:54.562518+0000","last_became_active":"2024-03-06T07:21:53.561996+0000","last_became_peered":"2024-03-06T07:21:53.561996+0000","last_unstale":"2024-03-06T07:21:54.562518+0000","last_undegraded":"2024-03-06T07:21:54.562518+0000","last_fullsized":"2024-03-06T07:21:54.562518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777264+0000","last_change":"2024-03-06T07:21:53.563516+0000","last_active":"2024-03-06T07:21:54.777264+0000","last_peered":"2024-03-06T07:21:54.777264+0000","last_clean":"2024-03-06T07:21:54.777264+0000","last_became_active":"2024-03-06T07:21:53.563237+0000","last_became_peered":"2024-03-06T07:21:53.563237+0000","last_unstale":"2024-03-06T07:21:54.777264+0000","last_undegraded":"2024-03-06T07:21:54.777264+0000","last_fullsized":"2024-03-06T07:21:54.777264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778868+0000","last_change":"2024-03-06T07:21:53.561815+0000","last_active":"2024-03-06T07:21:54.778868+0000","last_peered":"2024-03-06T07:21:54.778868+0000","last_clean":"2024-03-06T07:21:54.778868+0000","last_became_active":"2024-03-06T07:21:53.561189+0000","last_became_peered":"2024-03-06T07:21:53.561189+0000","last_unstale":"2024-03-06T07:21:54.778868+0000","last_undegraded":"2024-03-06T07:21:54.778868+0000","last_fullsized":"2024-03-06T07:21:54.778868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560237+0000","last_change":"2024-03-06T07:21:53.560903+0000","last_active":"2024-03-06T07:21:54.560237+0000","last_peered":"2024-03-06T07:21:54.560237+0000","last_clean":"2024-03-06T07:21:54.560237+0000","last_became_active":"2024-03-06T07:21:53.560624+0000","last_became_peered":"2024-03-06T07:21:53.560624+0000","last_unstale":"2024-03-06T07:21:54.560237+0000","last_undegraded":"2024-03-06T07:21:54.560237+0000","last_fullsized":"2024-03-06T07:21:54.560237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778508+0000","last_change":"2024-03-06T07:21:53.561059+0000","last_active":"2024-03-06T07:21:54.778508+0000","last_peered":"2024-03-06T07:21:54.778508+0000","last_clean":"2024-03-06T07:21:54.778508+0000","last_became_active":"2024-03-06T07:21:53.560799+0000","last_became_peered":"2024-03-06T07:21:53.560799+0000","last_unstale":"2024-03-06T07:21:54.778508+0000","last_undegraded":"2024-03-06T07:21:54.778508+0000","last_fullsized":"2024-03-06T07:21:54.778508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562750+0000","last_change":"2024-03-06T07:21:53.558851+0000","last_active":"2024-03-06T07:21:54.562750+0000","last_peered":"2024-03-06T07:21:54.562750+0000","last_clean":"2024-03-06T07:21:54.562750+0000","last_became_active":"2024-03-06T07:21:53.558543+0000","last_became_peered":"2024-03-06T07:21:53.558543+0000","last_unstale":"2024-03-06T07:21:54.562750+0000","last_undegraded":"2024-03-06T07:21:54.562750+0000","last_fullsized":"2024-03-06T07:21:54.562750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777302+0000","last_change":"2024-03-06T07:21:53.560684+0000","last_active":"2024-03-06T07:21:54.777302+0000","last_peered":"2024-03-06T07:21:54.777302+0000","last_clean":"2024-03-06T07:21:54.777302+0000","last_became_active":"2024-03-06T07:21:53.560428+0000","last_became_peered":"2024-03-06T07:21:53.560428+0000","last_unstale":"2024-03-06T07:21:54.777302+0000","last_undegraded":"2024-03-06T07:21:54.777302+0000","last_fullsized":"2024-03-06T07:21:54.777302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777813+0000","last_change":"2024-03-06T07:21:53.562732+0000","last_active":"2024-03-06T07:21:54.777813+0000","last_peered":"2024-03-06T07:21:54.777813+0000","last_clean":"2024-03-06T07:21:54.777813+0000","last_became_active":"2024-03-06T07:21:53.562351+0000","last_became_peered":"2024-03-06T07:21:53.562351+0000","last_unstale":"2024-03-06T07:21:54.777813+0000","last_undegraded":"2024-03-06T07:21:54.777813+0000","last_fullsized":"2024-03-06T07:21:54.777813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767070+0000","last_change":"2024-03-06T07:21:53.564304+0000","last_active":"2024-03-06T07:21:54.767070+0000","last_peered":"2024-03-06T07:21:54.767070+0000","last_clean":"2024-03-06T07:21:54.767070+0000","last_became_active":"2024-03-06T07:21:53.564008+0000","last_became_peered":"2024-03-06T07:21:53.564008+0000","last_unstale":"2024-03-06T07:21:54.767070+0000","last_undegraded":"2024-03-06T07:21:54.767070+0000","last_fullsized":"2024-03-06T07:21:54.767070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777401+0000","last_change":"2024-03-06T07:21:53.565197+0000","last_active":"2024-03-06T07:21:54.777401+0000","last_peered":"2024-03-06T07:21:54.777401+0000","last_clean":"2024-03-06T07:21:54.777401+0000","last_became_active":"2024-03-06T07:21:53.564878+0000","last_became_peered":"2024-03-06T07:21:53.564878+0000","last_unstale":"2024-03-06T07:21:54.777401+0000","last_undegraded":"2024-03-06T07:21:54.777401+0000","last_fullsized":"2024-03-06T07:21:54.777401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767433+0000","last_change":"2024-03-06T07:21:53.558718+0000","last_active":"2024-03-06T07:21:54.767433+0000","last_peered":"2024-03-06T07:21:54.767433+0000","last_clean":"2024-03-06T07:21:54.767433+0000","last_became_active":"2024-03-06T07:21:53.558454+0000","last_became_peered":"2024-03-06T07:21:53.558454+0000","last_unstale":"2024-03-06T07:21:54.767433+0000","last_undegraded":"2024-03-06T07:21:54.767433+0000","last_fullsized":"2024-03-06T07:21:54.767433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562735+0000","last_change":"2024-03-06T07:21:53.563894+0000","last_active":"2024-03-06T07:21:54.562735+0000","last_peered":"2024-03-06T07:21:54.562735+0000","last_clean":"2024-03-06T07:21:54.562735+0000","last_became_active":"2024-03-06T07:21:53.563493+0000","last_became_peered":"2024-03-06T07:21:53.563493+0000","last_unstale":"2024-03-06T07:21:54.562735+0000","last_undegraded":"2024-03-06T07:21:54.562735+0000","last_fullsized":"2024-03-06T07:21:54.562735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562575+0000","last_change":"2024-03-06T07:21:53.552128+0000","last_active":"2024-03-06T07:21:54.562575+0000","last_peered":"2024-03-06T07:21:54.562575+0000","last_clean":"2024-03-06T07:21:54.562575+0000","last_became_active":"2024-03-06T07:21:53.551826+0000","last_became_peered":"2024-03-06T07:21:53.551826+0000","last_unstale":"2024-03-06T07:21:54.562575+0000","last_undegraded":"2024-03-06T07:21:54.562575+0000","last_fullsized":"2024-03-06T07:21:54.562575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767379+0000","last_change":"2024-03-06T07:21:53.563089+0000","last_active":"2024-03-06T07:21:54.767379+0000","last_peered":"2024-03-06T07:21:54.767379+0000","last_clean":"2024-03-06T07:21:54.767379+0000","last_became_active":"2024-03-06T07:21:53.562833+0000","last_became_peered":"2024-03-06T07:21:53.562833+0000","last_unstale":"2024-03-06T07:21:54.767379+0000","last_undegraded":"2024-03-06T07:21:54.767379+0000","last_fullsized":"2024-03-06T07:21:54.767379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:06.338805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777387+0000","last_change":"2024-03-06T07:21:53.550849+0000","last_active":"2024-03-06T07:21:54.777387+0000","last_peered":"2024-03-06T07:21:54.777387+0000","last_clean":"2024-03-06T07:21:54.777387+0000","last_became_active":"2024-03-06T07:21:53.550562+0000","last_became_peered":"2024-03-06T07:21:53.550562+0000","last_unstale":"2024-03-06T07:21:54.777387+0000","last_undegraded":"2024-03-06T07:21:54.777387+0000","last_fullsized":"2024-03-06T07:21:54.777387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777942+0000","last_change":"2024-03-06T07:21:53.564528+0000","last_active":"2024-03-06T07:21:54.777942+0000","last_peered":"2024-03-06T07:21:54.777942+0000","last_clean":"2024-03-06T07:21:54.777942+0000","last_became_active":"2024-03-06T07:21:53.564250+0000","last_became_peered":"2024-03-06T07:21:53.564250+0000","last_unstale":"2024-03-06T07:21:54.777942+0000","last_undegraded":"2024-03-06T07:21:54.777942+0000","last_fullsized":"2024-03-06T07:21:54.777942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560202+0000","last_change":"2024-03-06T07:21:53.555673+0000","last_active":"2024-03-06T07:21:54.560202+0000","last_peered":"2024-03-06T07:21:54.560202+0000","last_clean":"2024-03-06T07:21:54.560202+0000","last_became_active":"2024-03-06T07:21:53.555337+0000","last_became_peered":"2024-03-06T07:21:53.555337+0000","last_unstale":"2024-03-06T07:21:54.560202+0000","last_undegraded":"2024-03-06T07:21:54.560202+0000","last_fullsized":"2024-03-06T07:21:54.560202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562445+0000","last_change":"2024-03-06T07:21:53.552447+0000","last_active":"2024-03-06T07:21:54.562445+0000","last_peered":"2024-03-06T07:21:54.562445+0000","last_clean":"2024-03-06T07:21:54.562445+0000","last_became_active":"2024-03-06T07:21:53.551907+0000","last_became_peered":"2024-03-06T07:21:53.551907+0000","last_unstale":"2024-03-06T07:21:54.562445+0000","last_undegraded":"2024-03-06T07:21:54.562445+0000","last_fullsized":"2024-03-06T07:21:54.562445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777543+0000","last_change":"2024-03-06T07:21:53.554149+0000","last_active":"2024-03-06T07:21:54.777543+0000","last_peered":"2024-03-06T07:21:54.777543+0000","last_clean":"2024-03-06T07:21:54.777543+0000","last_became_active":"2024-03-06T07:21:53.553811+0000","last_became_peered":"2024-03-06T07:21:53.553811+0000","last_unstale":"2024-03-06T07:21:54.777543+0000","last_undegraded":"2024-03-06T07:21:54.777543+0000","last_fullsized":"2024-03-06T07:21:54.777543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:11.236013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767399+0000","last_change":"2024-03-06T07:21:52.551262+0000","last_active":"2024-03-06T07:21:54.767399+0000","last_peered":"2024-03-06T07:21:54.767399+0000","last_clean":"2024-03-06T07:21:54.767399+0000","last_became_active":"2024-03-06T07:21:52.550349+0000","last_became_peered":"2024-03-06T07:21:52.550349+0000","last_unstale":"2024-03-06T07:21:54.767399+0000","last_undegraded":"2024-03-06T07:21:54.767399+0000","last_fullsized":"2024-03-06T07:21:54.767399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777450+0000","last_change":"2024-03-06T07:21:53.562215+0000","last_active":"2024-03-06T07:21:54.777450+0000","last_peered":"2024-03-06T07:21:54.777450+0000","last_clean":"2024-03-06T07:21:54.777450+0000","last_became_active":"2024-03-06T07:21:53.561933+0000","last_became_peered":"2024-03-06T07:21:53.561933+0000","last_unstale":"2024-03-06T07:21:54.777450+0000","last_undegraded":"2024-03-06T07:21:54.777450+0000","last_fullsized":"2024-03-06T07:21:54.777450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:36.338898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562398+0000","last_change":"2024-03-06T07:21:52.547490+0000","last_active":"2024-03-06T07:21:54.562398+0000","last_peered":"2024-03-06T07:21:54.562398+0000","last_clean":"2024-03-06T07:21:54.562398+0000","last_became_active":"2024-03-06T07:21:52.547168+0000","last_became_peered":"2024-03-06T07:21:52.547168+0000","last_unstale":"2024-03-06T07:21:54.562398+0000","last_undegraded":"2024-03-06T07:21:54.562398+0000","last_fullsized":"2024-03-06T07:21:54.562398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767143+0000","last_change":"2024-03-06T07:21:52.551838+0000","last_active":"2024-03-06T07:21:54.767143+0000","last_peered":"2024-03-06T07:21:54.767143+0000","last_clean":"2024-03-06T07:21:54.767143+0000","last_became_active":"2024-03-06T07:21:52.551599+0000","last_became_peered":"2024-03-06T07:21:52.551599+0000","last_unstale":"2024-03-06T07:21:54.767143+0000","last_undegraded":"2024-03-06T07:21:54.767143+0000","last_fullsized":"2024-03-06T07:21:54.767143+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767031+0000","last_change":"2024-03-06T07:21:53.557369+0000","last_active":"2024-03-06T07:21:54.767031+0000","last_peered":"2024-03-06T07:21:54.767031+0000","last_clean":"2024-03-06T07:21:54.767031+0000","last_became_active":"2024-03-06T07:21:53.556320+0000","last_became_peered":"2024-03-06T07:21:53.556320+0000","last_unstale":"2024-03-06T07:21:54.767031+0000","last_undegraded":"2024-03-06T07:21:54.767031+0000","last_fullsized":"2024-03-06T07:21:54.767031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777454+0000","last_change":"2024-03-06T07:21:52.553279+0000","last_active":"2024-03-06T07:21:54.777454+0000","last_peered":"2024-03-06T07:21:54.777454+0000","last_clean":"2024-03-06T07:21:54.777454+0000","last_became_active":"2024-03-06T07:21:52.552991+0000","last_became_peered":"2024-03-06T07:21:52.552991+0000","last_unstale":"2024-03-06T07:21:54.777454+0000","last_undegraded":"2024-03-06T07:21:54.777454+0000","last_fullsized":"2024-03-06T07:21:54.777454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562572+0000","last_change":"2024-03-06T07:21:53.552639+0000","last_active":"2024-03-06T07:21:54.562572+0000","last_peered":"2024-03-06T07:21:54.562572+0000","last_clean":"2024-03-06T07:21:54.562572+0000","last_became_active":"2024-03-06T07:21:53.552377+0000","last_became_peered":"2024-03-06T07:21:53.552377+0000","last_unstale":"2024-03-06T07:21:54.562572+0000","last_undegraded":"2024-03-06T07:21:54.562572+0000","last_fullsized":"2024-03-06T07:21:54.562572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:25.207806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562577+0000","last_change":"2024-03-06T07:21:52.556094+0000","last_active":"2024-03-06T07:21:54.562577+0000","last_peered":"2024-03-06T07:21:54.562577+0000","last_clean":"2024-03-06T07:21:54.562577+0000","last_became_active":"2024-03-06T07:21:52.555825+0000","last_became_peered":"2024-03-06T07:21:52.555825+0000","last_unstale":"2024-03-06T07:21:54.562577+0000","last_undegraded":"2024-03-06T07:21:54.562577+0000","last_fullsized":"2024-03-06T07:21:54.562577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778464+0000","last_change":"2024-03-06T07:21:53.561469+0000","last_active":"2024-03-06T07:21:54.778464+0000","last_peered":"2024-03-06T07:21:54.778464+0000","last_clean":"2024-03-06T07:21:54.778464+0000","last_became_active":"2024-03-06T07:21:53.561204+0000","last_became_peered":"2024-03-06T07:21:53.561204+0000","last_unstale":"2024-03-06T07:21:54.778464+0000","last_undegraded":"2024-03-06T07:21:54.778464+0000","last_fullsized":"2024-03-06T07:21:54.778464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778543+0000","last_change":"2024-03-06T07:21:52.557218+0000","last_active":"2024-03-06T07:21:54.778543+0000","last_peered":"2024-03-06T07:21:54.778543+0000","last_clean":"2024-03-06T07:21:54.778543+0000","last_became_active":"2024-03-06T07:21:52.556955+0000","last_became_peered":"2024-03-06T07:21:52.556955+0000","last_unstale":"2024-03-06T07:21:54.778543+0000","last_undegraded":"2024-03-06T07:21:54.778543+0000","last_fullsized":"2024-03-06T07:21:54.778543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777433+0000","last_change":"2024-03-06T07:21:53.559108+0000","last_active":"2024-03-06T07:21:54.777433+0000","last_peered":"2024-03-06T07:21:54.777433+0000","last_clean":"2024-03-06T07:21:54.777433+0000","last_became_active":"2024-03-06T07:21:53.558256+0000","last_became_peered":"2024-03-06T07:21:53.558256+0000","last_unstale":"2024-03-06T07:21:54.777433+0000","last_undegraded":"2024-03-06T07:21:54.777433+0000","last_fullsized":"2024-03-06T07:21:54.777433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562655+0000","last_change":"2024-03-06T07:21:52.557898+0000","last_active":"2024-03-06T07:21:54.562655+0000","last_peered":"2024-03-06T07:21:54.562655+0000","last_clean":"2024-03-06T07:21:54.562655+0000","last_became_active":"2024-03-06T07:21:52.557619+0000","last_became_peered":"2024-03-06T07:21:52.557619+0000","last_unstale":"2024-03-06T07:21:54.562655+0000","last_undegraded":"2024-03-06T07:21:54.562655+0000","last_fullsized":"2024-03-06T07:21:54.562655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777844+0000","last_change":"2024-03-06T07:21:53.557112+0000","last_active":"2024-03-06T07:21:54.777844+0000","last_peered":"2024-03-06T07:21:54.777844+0000","last_clean":"2024-03-06T07:21:54.777844+0000","last_became_active":"2024-03-06T07:21:53.556848+0000","last_became_peered":"2024-03-06T07:21:53.556848+0000","last_unstale":"2024-03-06T07:21:54.777844+0000","last_undegraded":"2024-03-06T07:21:54.777844+0000","last_fullsized":"2024-03-06T07:21:54.777844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:28.940984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562841+0000","last_change":"2024-03-06T07:21:52.548388+0000","last_active":"2024-03-06T07:21:54.562841+0000","last_peered":"2024-03-06T07:21:54.562841+0000","last_clean":"2024-03-06T07:21:54.562841+0000","last_became_active":"2024-03-06T07:21:52.548095+0000","last_became_peered":"2024-03-06T07:21:52.548095+0000","last_unstale":"2024-03-06T07:21:54.562841+0000","last_undegraded":"2024-03-06T07:21:54.562841+0000","last_fullsized":"2024-03-06T07:21:54.562841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778266+0000","last_change":"2024-03-06T07:21:53.557821+0000","last_active":"2024-03-06T07:21:54.778266+0000","last_peered":"2024-03-06T07:21:54.778266+0000","last_clean":"2024-03-06T07:21:54.778266+0000","last_became_active":"2024-03-06T07:21:53.557536+0000","last_became_peered":"2024-03-06T07:21:53.557536+0000","last_unstale":"2024-03-06T07:21:54.778266+0000","last_undegraded":"2024-03-06T07:21:54.778266+0000","last_fullsized":"2024-03-06T07:21:54.778266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:52.600018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777481+0000","last_change":"2024-03-06T07:21:52.555948+0000","last_active":"2024-03-06T07:21:54.777481+0000","last_peered":"2024-03-06T07:21:54.777481+0000","last_clean":"2024-03-06T07:21:54.777481+0000","last_became_active":"2024-03-06T07:21:52.555657+0000","last_became_peered":"2024-03-06T07:21:52.555657+0000","last_unstale":"2024-03-06T07:21:54.777481+0000","last_undegraded":"2024-03-06T07:21:54.777481+0000","last_fullsized":"2024-03-06T07:21:54.777481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777191+0000","last_change":"2024-03-06T07:21:53.556393+0000","last_active":"2024-03-06T07:21:54.777191+0000","last_peered":"2024-03-06T07:21:54.777191+0000","last_clean":"2024-03-06T07:21:54.777191+0000","last_became_active":"2024-03-06T07:21:53.556137+0000","last_became_peered":"2024-03-06T07:21:53.556137+0000","last_unstale":"2024-03-06T07:21:54.777191+0000","last_undegraded":"2024-03-06T07:21:54.777191+0000","last_fullsized":"2024-03-06T07:21:54.777191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:33.561906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778025+0000","last_change":"2024-03-06T07:21:52.556426+0000","last_active":"2024-03-06T07:21:54.778025+0000","last_peered":"2024-03-06T07:21:54.778025+0000","last_clean":"2024-03-06T07:21:54.778025+0000","last_became_active":"2024-03-06T07:21:52.555844+0000","last_became_peered":"2024-03-06T07:21:52.555844+0000","last_unstale":"2024-03-06T07:21:54.778025+0000","last_undegraded":"2024-03-06T07:21:54.778025+0000","last_fullsized":"2024-03-06T07:21:54.778025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778299+0000","last_change":"2024-03-06T07:21:53.567432+0000","last_active":"2024-03-06T07:21:54.778299+0000","last_peered":"2024-03-06T07:21:54.778299+0000","last_clean":"2024-03-06T07:21:54.778299+0000","last_became_active":"2024-03-06T07:21:53.566788+0000","last_became_peered":"2024-03-06T07:21:53.566788+0000","last_unstale":"2024-03-06T07:21:54.778299+0000","last_undegraded":"2024-03-06T07:21:54.778299+0000","last_fullsized":"2024-03-06T07:21:54.778299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562338+0000","last_change":"2024-03-06T07:21:52.547444+0000","last_active":"2024-03-06T07:21:54.562338+0000","last_peered":"2024-03-06T07:21:54.562338+0000","last_clean":"2024-03-06T07:21:54.562338+0000","last_became_active":"2024-03-06T07:21:52.547163+0000","last_became_peered":"2024-03-06T07:21:52.547163+0000","last_unstale":"2024-03-06T07:21:54.562338+0000","last_undegraded":"2024-03-06T07:21:54.562338+0000","last_fullsized":"2024-03-06T07:21:54.562338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562392+0000","last_change":"2024-03-06T07:21:53.552210+0000","last_active":"2024-03-06T07:21:54.562392+0000","last_peered":"2024-03-06T07:21:54.562392+0000","last_clean":"2024-03-06T07:21:54.562392+0000","last_became_active":"2024-03-06T07:21:53.551529+0000","last_became_peered":"2024-03-06T07:21:53.551529+0000","last_unstale":"2024-03-06T07:21:54.562392+0000","last_undegraded":"2024-03-06T07:21:54.562392+0000","last_fullsized":"2024-03-06T07:21:54.562392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:52.101646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560082+0000","last_change":"2024-03-06T07:21:52.552286+0000","last_active":"2024-03-06T07:21:54.560082+0000","last_peered":"2024-03-06T07:21:54.560082+0000","last_clean":"2024-03-06T07:21:54.560082+0000","last_became_active":"2024-03-06T07:21:52.552010+0000","last_became_peered":"2024-03-06T07:21:52.552010+0000","last_unstale":"2024-03-06T07:21:54.560082+0000","last_undegraded":"2024-03-06T07:21:54.560082+0000","last_fullsized":"2024-03-06T07:21:54.560082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777580+0000","last_change":"2024-03-06T07:21:53.571841+0000","last_active":"2024-03-06T07:21:54.777580+0000","last_peered":"2024-03-06T07:21:54.777580+0000","last_clean":"2024-03-06T07:21:54.777580+0000","last_became_active":"2024-03-06T07:21:53.571350+0000","last_became_peered":"2024-03-06T07:21:53.571350+0000","last_unstale":"2024-03-06T07:21:54.777580+0000","last_undegraded":"2024-03-06T07:21:54.777580+0000","last_fullsized":"2024-03-06T07:21:54.777580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777653+0000","last_change":"2024-03-06T07:21:52.553990+0000","last_active":"2024-03-06T07:21:54.777653+0000","last_peered":"2024-03-06T07:21:54.777653+0000","last_clean":"2024-03-06T07:21:54.777653+0000","last_became_active":"2024-03-06T07:21:52.553693+0000","last_became_peered":"2024-03-06T07:21:52.553693+0000","last_unstale":"2024-03-06T07:21:54.777653+0000","last_undegraded":"2024-03-06T07:21:54.777653+0000","last_fullsized":"2024-03-06T07:21:54.777653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:58.952037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777984+0000","last_change":"2024-03-06T07:21:53.567545+0000","last_active":"2024-03-06T07:21:54.777984+0000","last_peered":"2024-03-06T07:21:54.777984+0000","last_clean":"2024-03-06T07:21:54.777984+0000","last_became_active":"2024-03-06T07:21:53.566157+0000","last_became_peered":"2024-03-06T07:21:53.566157+0000","last_unstale":"2024-03-06T07:21:54.777984+0000","last_undegraded":"2024-03-06T07:21:54.777984+0000","last_fullsized":"2024-03-06T07:21:54.777984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:51.751448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560201+0000","last_change":"2024-03-06T07:21:52.547686+0000","last_active":"2024-03-06T07:21:54.560201+0000","last_peered":"2024-03-06T07:21:54.560201+0000","last_clean":"2024-03-06T07:21:54.560201+0000","last_became_active":"2024-03-06T07:21:52.547436+0000","last_became_peered":"2024-03-06T07:21:52.547436+0000","last_unstale":"2024-03-06T07:21:54.560201+0000","last_undegraded":"2024-03-06T07:21:54.560201+0000","last_fullsized":"2024-03-06T07:21:54.560201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778797+0000","last_change":"2024-03-06T07:21:53.567657+0000","last_active":"2024-03-06T07:21:54.778797+0000","last_peered":"2024-03-06T07:21:54.778797+0000","last_clean":"2024-03-06T07:21:54.778797+0000","last_became_active":"2024-03-06T07:21:53.567134+0000","last_became_peered":"2024-03-06T07:21:53.567134+0000","last_unstale":"2024-03-06T07:21:54.778797+0000","last_undegraded":"2024-03-06T07:21:54.778797+0000","last_fullsized":"2024-03-06T07:21:54.778797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:57.048252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562696+0000","last_change":"2024-03-06T07:21:52.556768+0000","last_active":"2024-03-06T07:21:54.562696+0000","last_peered":"2024-03-06T07:21:54.562696+0000","last_clean":"2024-03-06T07:21:54.562696+0000","last_became_active":"2024-03-06T07:21:52.556507+0000","last_became_peered":"2024-03-06T07:21:52.556507+0000","last_unstale":"2024-03-06T07:21:54.562696+0000","last_undegraded":"2024-03-06T07:21:54.562696+0000","last_fullsized":"2024-03-06T07:21:54.562696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778541+0000","last_change":"2024-03-06T07:21:53.563768+0000","last_active":"2024-03-06T07:21:54.778541+0000","last_peered":"2024-03-06T07:21:54.778541+0000","last_clean":"2024-03-06T07:21:54.778541+0000","last_became_active":"2024-03-06T07:21:53.563490+0000","last_became_peered":"2024-03-06T07:21:53.563490+0000","last_unstale":"2024-03-06T07:21:54.778541+0000","last_undegraded":"2024-03-06T07:21:54.778541+0000","last_fullsized":"2024-03-06T07:21:54.778541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:06.273778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560265+0000","last_change":"2024-03-06T07:21:52.557535+0000","last_active":"2024-03-06T07:21:54.560265+0000","last_peered":"2024-03-06T07:21:54.560265+0000","last_clean":"2024-03-06T07:21:54.560265+0000","last_became_active":"2024-03-06T07:21:52.557275+0000","last_became_peered":"2024-03-06T07:21:52.557275+0000","last_unstale":"2024-03-06T07:21:54.560265+0000","last_undegraded":"2024-03-06T07:21:54.560265+0000","last_fullsized":"2024-03-06T07:21:54.560265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:05.512013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778386+0000","last_change":"2024-03-06T07:21:53.565629+0000","last_active":"2024-03-06T07:21:54.778386+0000","last_peered":"2024-03-06T07:21:54.778386+0000","last_clean":"2024-03-06T07:21:54.778386+0000","last_became_active":"2024-03-06T07:21:53.563789+0000","last_became_peered":"2024-03-06T07:21:53.563789+0000","last_unstale":"2024-03-06T07:21:54.778386+0000","last_undegraded":"2024-03-06T07:21:54.778386+0000","last_fullsized":"2024-03-06T07:21:54.778386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:31.986485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778063+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:21:54.778063+0000","last_peered":"2024-03-06T07:21:54.778063+0000","last_clean":"2024-03-06T07:21:54.778063+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:21:54.778063+0000","last_undegraded":"2024-03-06T07:21:54.778063+0000","last_fullsized":"2024-03-06T07:21:54.778063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777283+0000","last_change":"2024-03-06T07:21:52.557611+0000","last_active":"2024-03-06T07:21:54.777283+0000","last_peered":"2024-03-06T07:21:54.777283+0000","last_clean":"2024-03-06T07:21:54.777283+0000","last_became_active":"2024-03-06T07:21:52.556594+0000","last_became_peered":"2024-03-06T07:21:52.556594+0000","last_unstale":"2024-03-06T07:21:54.777283+0000","last_undegraded":"2024-03-06T07:21:54.777283+0000","last_fullsized":"2024-03-06T07:21:54.777283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767209+0000","last_change":"2024-03-06T07:21:53.565607+0000","last_active":"2024-03-06T07:21:54.767209+0000","last_peered":"2024-03-06T07:21:54.767209+0000","last_clean":"2024-03-06T07:21:54.767209+0000","last_became_active":"2024-03-06T07:21:53.565356+0000","last_became_peered":"2024-03-06T07:21:53.565356+0000","last_unstale":"2024-03-06T07:21:54.767209+0000","last_undegraded":"2024-03-06T07:21:54.767209+0000","last_fullsized":"2024-03-06T07:21:54.767209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778021+0000","last_change":"2024-03-06T07:21:52.555927+0000","last_active":"2024-03-06T07:21:54.778021+0000","last_peered":"2024-03-06T07:21:54.778021+0000","last_clean":"2024-03-06T07:21:54.778021+0000","last_became_active":"2024-03-06T07:21:52.555485+0000","last_became_peered":"2024-03-06T07:21:52.555485+0000","last_unstale":"2024-03-06T07:21:54.778021+0000","last_undegraded":"2024-03-06T07:21:54.778021+0000","last_fullsized":"2024-03-06T07:21:54.778021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777097+0000","last_change":"2024-03-06T07:21:53.559225+0000","last_active":"2024-03-06T07:21:54.777097+0000","last_peered":"2024-03-06T07:21:54.777097+0000","last_clean":"2024-03-06T07:21:54.777097+0000","last_became_active":"2024-03-06T07:21:53.558925+0000","last_became_peered":"2024-03-06T07:21:53.558925+0000","last_unstale":"2024-03-06T07:21:54.777097+0000","last_undegraded":"2024-03-06T07:21:54.777097+0000","last_fullsized":"2024-03-06T07:21:54.777097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:54.530841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.560174+0000","last_change":"2024-03-06T07:21:52.546620+0000","last_active":"2024-03-06T07:21:54.560174+0000","last_peered":"2024-03-06T07:21:54.560174+0000","last_clean":"2024-03-06T07:21:54.560174+0000","last_became_active":"2024-03-06T07:21:52.546297+0000","last_became_peered":"2024-03-06T07:21:52.546297+0000","last_unstale":"2024-03-06T07:21:54.560174+0000","last_undegraded":"2024-03-06T07:21:54.560174+0000","last_fullsized":"2024-03-06T07:21:54.560174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:07.328988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562488+0000","last_change":"2024-03-06T07:21:53.553218+0000","last_active":"2024-03-06T07:21:54.562488+0000","last_peered":"2024-03-06T07:21:54.562488+0000","last_clean":"2024-03-06T07:21:54.562488+0000","last_became_active":"2024-03-06T07:21:53.552911+0000","last_became_peered":"2024-03-06T07:21:53.552911+0000","last_unstale":"2024-03-06T07:21:54.562488+0000","last_undegraded":"2024-03-06T07:21:54.562488+0000","last_fullsized":"2024-03-06T07:21:54.562488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.386893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778113+0000","last_change":"2024-03-06T07:21:52.556096+0000","last_active":"2024-03-06T07:21:54.778113+0000","last_peered":"2024-03-06T07:21:54.778113+0000","last_clean":"2024-03-06T07:21:54.778113+0000","last_became_active":"2024-03-06T07:21:52.555510+0000","last_became_peered":"2024-03-06T07:21:52.555510+0000","last_unstale":"2024-03-06T07:21:54.778113+0000","last_undegraded":"2024-03-06T07:21:54.778113+0000","last_fullsized":"2024-03-06T07:21:54.778113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778491+0000","last_change":"2024-03-06T07:21:53.561767+0000","last_active":"2024-03-06T07:21:54.778491+0000","last_peered":"2024-03-06T07:21:54.778491+0000","last_clean":"2024-03-06T07:21:54.778491+0000","last_became_active":"2024-03-06T07:21:53.561435+0000","last_became_peered":"2024-03-06T07:21:53.561435+0000","last_unstale":"2024-03-06T07:21:54.778491+0000","last_undegraded":"2024-03-06T07:21:54.778491+0000","last_fullsized":"2024-03-06T07:21:54.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:24.631055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562673+0000","last_change":"2024-03-06T07:21:52.553733+0000","last_active":"2024-03-06T07:21:54.562673+0000","last_peered":"2024-03-06T07:21:54.562673+0000","last_clean":"2024-03-06T07:21:54.562673+0000","last_became_active":"2024-03-06T07:21:52.553436+0000","last_became_peered":"2024-03-06T07:21:52.553436+0000","last_unstale":"2024-03-06T07:21:54.562673+0000","last_undegraded":"2024-03-06T07:21:54.562673+0000","last_fullsized":"2024-03-06T07:21:54.562673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:43.790046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779356+0000","last_change":"2024-03-06T07:21:53.556153+0000","last_active":"2024-03-06T07:21:54.779356+0000","last_peered":"2024-03-06T07:21:54.779356+0000","last_clean":"2024-03-06T07:21:54.779356+0000","last_became_active":"2024-03-06T07:21:53.555896+0000","last_became_peered":"2024-03-06T07:21:53.555896+0000","last_unstale":"2024-03-06T07:21:54.779356+0000","last_undegraded":"2024-03-06T07:21:54.779356+0000","last_fullsized":"2024-03-06T07:21:54.779356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:13.698482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778511+0000","last_change":"2024-03-06T07:21:52.557084+0000","last_active":"2024-03-06T07:21:54.778511+0000","last_peered":"2024-03-06T07:21:54.778511+0000","last_clean":"2024-03-06T07:21:54.778511+0000","last_became_active":"2024-03-06T07:21:52.556688+0000","last_became_peered":"2024-03-06T07:21:52.556688+0000","last_unstale":"2024-03-06T07:21:54.778511+0000","last_undegraded":"2024-03-06T07:21:54.778511+0000","last_fullsized":"2024-03-06T07:21:54.778511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:23.626336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562632+0000","last_change":"2024-03-06T07:21:53.552698+0000","last_active":"2024-03-06T07:21:54.562632+0000","last_peered":"2024-03-06T07:21:54.562632+0000","last_clean":"2024-03-06T07:21:54.562632+0000","last_became_active":"2024-03-06T07:21:53.552434+0000","last_became_peered":"2024-03-06T07:21:53.552434+0000","last_unstale":"2024-03-06T07:21:54.562632+0000","last_undegraded":"2024-03-06T07:21:54.562632+0000","last_fullsized":"2024-03-06T07:21:54.562632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:01.848271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562589+0000","last_change":"2024-03-06T07:21:52.553671+0000","last_active":"2024-03-06T07:21:54.562589+0000","last_peered":"2024-03-06T07:21:54.562589+0000","last_clean":"2024-03-06T07:21:54.562589+0000","last_became_active":"2024-03-06T07:21:52.553394+0000","last_became_peered":"2024-03-06T07:21:52.553394+0000","last_unstale":"2024-03-06T07:21:54.562589+0000","last_undegraded":"2024-03-06T07:21:54.562589+0000","last_fullsized":"2024-03-06T07:21:54.562589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767449+0000","last_change":"2024-03-06T07:21:53.555702+0000","last_active":"2024-03-06T07:21:54.767449+0000","last_peered":"2024-03-06T07:21:54.767449+0000","last_clean":"2024-03-06T07:21:54.767449+0000","last_became_active":"2024-03-06T07:21:53.555436+0000","last_became_peered":"2024-03-06T07:21:53.555436+0000","last_unstale":"2024-03-06T07:21:54.767449+0000","last_undegraded":"2024-03-06T07:21:54.767449+0000","last_fullsized":"2024-03-06T07:21:54.767449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562639+0000","last_change":"2024-03-06T07:21:52.550448+0000","last_active":"2024-03-06T07:21:54.562639+0000","last_peered":"2024-03-06T07:21:54.562639+0000","last_clean":"2024-03-06T07:21:54.562639+0000","last_became_active":"2024-03-06T07:21:52.550179+0000","last_became_peered":"2024-03-06T07:21:52.550179+0000","last_unstale":"2024-03-06T07:21:54.562639+0000","last_undegraded":"2024-03-06T07:21:54.562639+0000","last_fullsized":"2024-03-06T07:21:54.562639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.791874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777421+0000","last_change":"2024-03-06T07:21:53.553031+0000","last_active":"2024-03-06T07:21:54.777421+0000","last_peered":"2024-03-06T07:21:54.777421+0000","last_clean":"2024-03-06T07:21:54.777421+0000","last_became_active":"2024-03-06T07:21:53.552759+0000","last_became_peered":"2024-03-06T07:21:53.552759+0000","last_unstale":"2024-03-06T07:21:54.777421+0000","last_undegraded":"2024-03-06T07:21:54.777421+0000","last_fullsized":"2024-03-06T07:21:54.777421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777184+0000","last_change":"2024-03-06T07:21:52.555781+0000","last_active":"2024-03-06T07:21:54.777184+0000","last_peered":"2024-03-06T07:21:54.777184+0000","last_clean":"2024-03-06T07:21:54.777184+0000","last_became_active":"2024-03-06T07:21:52.555469+0000","last_became_peered":"2024-03-06T07:21:52.555469+0000","last_unstale":"2024-03-06T07:21:54.777184+0000","last_undegraded":"2024-03-06T07:21:54.777184+0000","last_fullsized":"2024-03-06T07:21:54.777184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:10.231294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777797+0000","last_change":"2024-03-06T07:21:53.558488+0000","last_active":"2024-03-06T07:21:54.777797+0000","last_peered":"2024-03-06T07:21:54.777797+0000","last_clean":"2024-03-06T07:21:54.777797+0000","last_became_active":"2024-03-06T07:21:53.558199+0000","last_became_peered":"2024-03-06T07:21:53.558199+0000","last_unstale":"2024-03-06T07:21:54.777797+0000","last_undegraded":"2024-03-06T07:21:54.777797+0000","last_fullsized":"2024-03-06T07:21:54.777797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562638+0000","last_change":"2024-03-06T07:21:52.556952+0000","last_active":"2024-03-06T07:21:54.562638+0000","last_peered":"2024-03-06T07:21:54.562638+0000","last_clean":"2024-03-06T07:21:54.562638+0000","last_became_active":"2024-03-06T07:21:52.556629+0000","last_became_peered":"2024-03-06T07:21:52.556629+0000","last_unstale":"2024-03-06T07:21:54.562638+0000","last_undegraded":"2024-03-06T07:21:54.562638+0000","last_fullsized":"2024-03-06T07:21:54.562638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767159+0000","last_change":"2024-03-06T07:21:53.554655+0000","last_active":"2024-03-06T07:21:54.767159+0000","last_peered":"2024-03-06T07:21:54.767159+0000","last_clean":"2024-03-06T07:21:54.767159+0000","last_became_active":"2024-03-06T07:21:53.554321+0000","last_became_peered":"2024-03-06T07:21:53.554321+0000","last_unstale":"2024-03-06T07:21:54.767159+0000","last_undegraded":"2024-03-06T07:21:54.767159+0000","last_fullsized":"2024-03-06T07:21:54.767159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:29.210603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562564+0000","last_change":"2024-03-06T07:21:52.547548+0000","last_active":"2024-03-06T07:21:54.562564+0000","last_peered":"2024-03-06T07:21:54.562564+0000","last_clean":"2024-03-06T07:21:54.562564+0000","last_became_active":"2024-03-06T07:21:52.547285+0000","last_became_peered":"2024-03-06T07:21:52.547285+0000","last_unstale":"2024-03-06T07:21:54.562564+0000","last_undegraded":"2024-03-06T07:21:54.562564+0000","last_fullsized":"2024-03-06T07:21:54.562564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767314+0000","last_change":"2024-03-06T07:21:53.557506+0000","last_active":"2024-03-06T07:21:54.767314+0000","last_peered":"2024-03-06T07:21:54.767314+0000","last_clean":"2024-03-06T07:21:54.767314+0000","last_became_active":"2024-03-06T07:21:53.556568+0000","last_became_peered":"2024-03-06T07:21:53.556568+0000","last_unstale":"2024-03-06T07:21:54.767314+0000","last_undegraded":"2024-03-06T07:21:54.767314+0000","last_fullsized":"2024-03-06T07:21:54.767314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:50.311042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778329+0000","last_change":"2024-03-06T07:21:52.558820+0000","last_active":"2024-03-06T07:21:54.778329+0000","last_peered":"2024-03-06T07:21:54.778329+0000","last_clean":"2024-03-06T07:21:54.778329+0000","last_became_active":"2024-03-06T07:21:52.558486+0000","last_became_peered":"2024-03-06T07:21:52.558486+0000","last_unstale":"2024-03-06T07:21:54.778329+0000","last_undegraded":"2024-03-06T07:21:54.778329+0000","last_fullsized":"2024-03-06T07:21:54.778329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.316331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.562402+0000","last_change":"2024-03-06T07:21:53.558230+0000","last_active":"2024-03-06T07:21:54.562402+0000","last_peered":"2024-03-06T07:21:54.562402+0000","last_clean":"2024-03-06T07:21:54.562402+0000","last_became_active":"2024-03-06T07:21:53.557916+0000","last_became_peered":"2024-03-06T07:21:53.557916+0000","last_unstale":"2024-03-06T07:21:54.562402+0000","last_undegraded":"2024-03-06T07:21:54.562402+0000","last_fullsized":"2024-03-06T07:21:54.562402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.779232+0000","last_change":"2024-03-06T07:21:53.567503+0000","last_active":"2024-03-06T07:21:54.779232+0000","last_peered":"2024-03-06T07:21:54.779232+0000","last_clean":"2024-03-06T07:21:54.779232+0000","last_became_active":"2024-03-06T07:21:53.567223+0000","last_became_peered":"2024-03-06T07:21:53.567223+0000","last_unstale":"2024-03-06T07:21:54.779232+0000","last_undegraded":"2024-03-06T07:21:54.779232+0000","last_fullsized":"2024-03-06T07:21:54.779232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:43.063553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777697+0000","last_change":"2024-03-06T07:21:52.559682+0000","last_active":"2024-03-06T07:21:54.777697+0000","last_peered":"2024-03-06T07:21:54.777697+0000","last_clean":"2024-03-06T07:21:54.777697+0000","last_became_active":"2024-03-06T07:21:52.559430+0000","last_became_peered":"2024-03-06T07:21:52.559430+0000","last_unstale":"2024-03-06T07:21:54.777697+0000","last_undegraded":"2024-03-06T07:21:54.777697+0000","last_fullsized":"2024-03-06T07:21:54.777697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:41.946139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778415+0000","last_change":"2024-03-06T07:21:53.568862+0000","last_active":"2024-03-06T07:21:54.778415+0000","last_peered":"2024-03-06T07:21:54.778415+0000","last_clean":"2024-03-06T07:21:54.778415+0000","last_became_active":"2024-03-06T07:21:53.568127+0000","last_became_peered":"2024-03-06T07:21:53.568127+0000","last_unstale":"2024-03-06T07:21:54.778415+0000","last_undegraded":"2024-03-06T07:21:54.778415+0000","last_fullsized":"2024-03-06T07:21:54.778415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:35.313473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.778429+0000","last_change":"2024-03-06T07:21:52.558983+0000","last_active":"2024-03-06T07:21:54.778429+0000","last_peered":"2024-03-06T07:21:54.778429+0000","last_clean":"2024-03-06T07:21:54.778429+0000","last_became_active":"2024-03-06T07:21:52.558616+0000","last_became_peered":"2024-03-06T07:21:52.558616+0000","last_unstale":"2024-03-06T07:21:54.778429+0000","last_undegraded":"2024-03-06T07:21:54.778429+0000","last_fullsized":"2024-03-06T07:21:54.778429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:27.936265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777374+0000","last_change":"2024-03-06T07:21:53.558956+0000","last_active":"2024-03-06T07:21:54.777374+0000","last_peered":"2024-03-06T07:21:54.777374+0000","last_clean":"2024-03-06T07:21:54.777374+0000","last_became_active":"2024-03-06T07:21:53.558612+0000","last_became_peered":"2024-03-06T07:21:53.558612+0000","last_unstale":"2024-03-06T07:21:54.777374+0000","last_undegraded":"2024-03-06T07:21:54.777374+0000","last_fullsized":"2024-03-06T07:21:54.777374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:39.545029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767442+0000","last_change":"2024-03-06T07:21:52.553080+0000","last_active":"2024-03-06T07:21:54.767442+0000","last_peered":"2024-03-06T07:21:54.767442+0000","last_clean":"2024-03-06T07:21:54.767442+0000","last_became_active":"2024-03-06T07:21:52.552785+0000","last_became_peered":"2024-03-06T07:21:52.552785+0000","last_unstale":"2024-03-06T07:21:54.767442+0000","last_undegraded":"2024-03-06T07:21:54.767442+0000","last_fullsized":"2024-03-06T07:21:54.767442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.767397+0000","last_change":"2024-03-06T07:21:53.563091+0000","last_active":"2024-03-06T07:21:54.767397+0000","last_peered":"2024-03-06T07:21:54.767397+0000","last_clean":"2024-03-06T07:21:54.767397+0000","last_became_active":"2024-03-06T07:21:53.562797+0000","last_became_peered":"2024-03-06T07:21:53.562797+0000","last_unstale":"2024-03-06T07:21:54.767397+0000","last_undegraded":"2024-03-06T07:21:54.767397+0000","last_fullsized":"2024-03-06T07:21:54.767397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:52.532052+0000","last_clean_scrub_stamp":"2024-03-06T07:21:52.532052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:23.897275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:54.777504+0000","last_change":"2024-03-06T07:21:52.557097+0000","last_active":"2024-03-06T07:21:54.777504+0000","last_peered":"2024-03-06T07:21:54.777504+0000","last_clean":"2024-03-06T07:21:54.777504+0000","last_became_active":"2024-03-06T07:21:52.556808+0000","last_became_peered":"2024-03-06T07:21:52.556808+0000","last_unstale":"2024-03-06T07:21:54.777504+0000","last_undegraded":"2024-03-06T07:21:54.777504+0000","last_fullsized":"2024-03-06T07:21:54.777504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:51.527333+0000","last_clean_scrub_stamp":"2024-03-06T07:21:51.527333+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:43.487272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:05.900 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:22:05.900 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:22:05.900 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:22:05.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:22:06.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.033+0000 7f3c300f4640 1 Processor -- start 2024-03-06T07:22:06.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.033+0000 7f3c300f4640 1 -- start start 2024-03-06T07:22:06.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c300f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3c2806b220 0x7f3c2806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c300f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3c2806b600 0x7f3c281622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c300f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c280b04c0 con 0x7f3c280700a0 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c300f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c281627e0 con 0x7f3c2806b220 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c300f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2806d8d0 con 0x7f3c2806b600 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c2de90640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3c2806b220 0x7f3c2806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c2d68f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3c280700a0 0x7f3c2806eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38572/0 (socket says 172.21.15.122:38572) 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c2de90640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3c2806b220 0x7f3c2806e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57078/0 (socket says 172.21.15.122:57078) 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c2d68f640 1 -- 172.21.15.122:0/1764038790 learned_addr learned my addr 172.21.15.122:0/1764038790 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.034+0000 7f3c2e691640 1 --2- 172.21.15.122:0/1764038790 >> v2:172.21.15.169:3300/0 conn(0x7f3c2806b600 0x7f3c281622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2de90640 1 -- 172.21.15.122:0/1764038790 >> v2:172.21.15.169:3300/0 conn(0x7f3c2806b600 msgr2=0x7f3c281622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2de90640 1 --2- 172.21.15.122:0/1764038790 >> v2:172.21.15.169:3300/0 conn(0x7f3c2806b600 0x7f3c281622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2de90640 1 -- 172.21.15.122:0/1764038790 >> v1:172.21.15.122:6789/0 conn(0x7f3c280700a0 legacy=0x7f3c2806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2de90640 1 -- 172.21.15.122:0/1764038790 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2806db70 con 0x7f3c2806b220 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2ce8e640 1 -- 172.21.15.122:0/1764038790 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87865334 0 0) 0x7f3c280b04c0 con 0x7f3c280700a0 2024-03-06T07:22:06.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2de90640 1 --2- 172.21.15.122:0/1764038790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c2806b220 0x7f3c2806e650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c28067c90 tx=0x7f3c2402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90cc85ba83bda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2ce8e640 1 -- 172.21.15.122:0/1764038790 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24039070 con 0x7f3c2806b220 2024-03-06T07:22:06.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.035+0000 7f3c2ce8e640 1 -- 172.21.15.122:0/1764038790 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2402cbf0 con 0x7f3c2806b220 2024-03-06T07:22:06.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c2ce8e640 1 -- 172.21.15.122:0/1764038790 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24034440 con 0x7f3c2806b220 2024-03-06T07:22:06.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 -- 172.21.15.122:0/1764038790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c2806b220 msgr2=0x7f3c2806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 --2- 172.21.15.122:0/1764038790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c2806b220 0x7f3c2806e650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c28067c90 tx=0x7f3c2402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:06.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 -- 172.21.15.122:0/1764038790 shutdown_connections 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 --2- 172.21.15.122:0/1764038790 >> v2:172.21.15.169:3300/0 conn(0x7f3c2806b600 0x7f3c281622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 --2- 172.21.15.122:0/1764038790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c2806b220 0x7f3c2806e650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 -- 172.21.15.122:0/1764038790 >> 172.21.15.122:0/1764038790 conn(0x7f3c280d0a10 msgr2=0x7f3c280d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 -- 172.21.15.122:0/1764038790 shutdown_connections 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.036+0000 7f3c300f4640 1 -- 172.21.15.122:0/1764038790 wait complete. 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.037+0000 7f3c300f4640 1 Processor -- start 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.037+0000 7f3c300f4640 1 -- start start 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c300f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c2806b220 0x7f3c2815b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c300f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c2806b600 0x7f3c2815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c300f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 0x7f3c2815f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c300f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806fb10 con 0x7f3c280700a0 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c300f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806e130 con 0x7f3c2806b600 2024-03-06T07:22:06.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c300f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806fc90 con 0x7f3c2806b220 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 0x7f3c2815f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 0x7f3c2815f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57080/0 (socket says 172.21.15.122:57080) 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 -- 172.21.15.122:0/2132986649 learned_addr learned my addr 172.21.15.122:0/2132986649 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2de90640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c2806b220 0x7f3c2815b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2d68f640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c2806b600 0x7f3c2815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 -- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c2806b220 msgr2=0x7f3c2815b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c2806b220 0x7f3c2815b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 -- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c2806b600 msgr2=0x7f3c2815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c2806b600 0x7f3c2815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 -- 172.21.15.122:0/2132986649 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28158b60 con 0x7f3c280700a0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.038+0000 7f3c2e691640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 0x7f3c2815f5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000e930 tx=0x7f3c2000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.039+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20017070 con 0x7f3c280700a0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.039+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2000c980 con 0x7f3c280700a0 2024-03-06T07:22:06.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.039+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c28160620 con 0x7f3c280700a0 2024-03-06T07:22:06.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.039+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2000cd70 con 0x7f3c280700a0 2024-03-06T07:22:06.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.039+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c28158e00 con 0x7f3c280700a0 2024-03-06T07:22:06.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.040+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2804f330 con 0x7f3c280700a0 2024-03-06T07:22:06.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.040+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c2001a070 con 0x7f3c280700a0 2024-03-06T07:22:06.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.040+0000 7f3c1e7fc640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.040+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c2008a110 con 0x7f3c280700a0 2024-03-06T07:22:06.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.040+0000 7f3c2de90640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.041+0000 7f3c2de90640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c24036000 tx=0x7f3c24030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.045+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c20058a90 con 0x7f3c280700a0 2024-03-06T07:22:06.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.252+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3c280b4420 con 0x7f3c280700a0 2024-03-06T07:22:06.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.252+0000 7f3c1e7fc640 1 -- 172.21.15.122:0/2132986649 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3c2005cb30 con 0x7f3c280700a0 2024-03-06T07:22:06.255 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:06.255 INFO:teuthology.orchestra.run.smithi122.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:22:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c0c06b4d0 msgr2=0x7f3c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c24036000 tx=0x7f3c24030000 comp rx=0 tx=0).stop 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 msgr2=0x7f3c2815f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 0x7f3c2815f5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000e930 tx=0x7f3c2000edf0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 shutdown_connections 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3c0c06b4d0 0x7f3c0c06d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c280700a0 0x7f3c2815f5c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3c2806b600 0x7f3c2815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 --2- 172.21.15.122:0/2132986649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3c2806b220 0x7f3c2815b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.254+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 >> 172.21.15.122:0/2132986649 conn(0x7f3c280d0a10 msgr2=0x7f3c2806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.255+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 shutdown_connections 2024-03-06T07:22:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.255+0000 7f3c300f4640 1 -- 172.21.15.122:0/2132986649 wait complete. 2024-03-06T07:22:06.267 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:22:06.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:06.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.364+0000 7fdc6e355640 1 Processor -- start 2024-03-06T07:22:06.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.364+0000 7fdc6e355640 1 -- start start 2024-03-06T07:22:06.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc6e355640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdc681621c0 0x7fdc680cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc6e355640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc6815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc6e355640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc680ceac0 con 0x7fdc680d4920 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc6e355640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680cf690 con 0x7fdc681621c0 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc6e355640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680d2dc0 con 0x7fdc681625a0 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc677fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdc680d4920 0x7fdc680d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38574/0 (socket says 172.21.15.122:38574) 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc677fe640 1 -- 172.21.15.122:0/2724108603 learned_addr learned my addr 172.21.15.122:0/2724108603 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc67fff640 1 --2- 172.21.15.122:0/2724108603 >> v2:172.21.15.169:3300/0 conn(0x7fdc681621c0 0x7fdc680cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.365+0000 7fdc6c8f2640 1 --2- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc6815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc66ffd640 1 -- 172.21.15.122:0/2724108603 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717578138 0 0) 0x7fdc680ceac0 con 0x7fdc680d4920 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc66ffd640 1 -- 172.21.15.122:0/2724108603 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc50003520 con 0x7fdc680d4920 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc67fff640 1 -- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 msgr2=0x7fdc6815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc67fff640 1 --2- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc6815dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc67fff640 1 -- 172.21.15.122:0/2724108603 >> v1:172.21.15.122:6789/0 conn(0x7fdc680d4920 legacy=0x7fdc680d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc67fff640 1 -- 172.21.15.122:0/2724108603 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc680ced60 con 0x7fdc681621c0 2024-03-06T07:22:06.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.366+0000 7fdc67fff640 1 --2- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680cfb20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdc580097a0 tx=0x7fdc5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c218623c95dc5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc66ffd640 1 -- 172.21.15.122:0/2724108603 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58035020 con 0x7fdc681621c0 2024-03-06T07:22:06.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc66ffd640 1 -- 172.21.15.122:0/2724108603 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc58031830 con 0x7fdc681621c0 2024-03-06T07:22:06.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc66ffd640 1 -- 172.21.15.122:0/2724108603 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58031c00 con 0x7fdc681621c0 2024-03-06T07:22:06.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 -- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 msgr2=0x7fdc680cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 --2- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680cfb20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdc580097a0 tx=0x7fdc5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 -- 172.21.15.122:0/2724108603 shutdown_connections 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 --2- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc6815dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 --2- 172.21.15.122:0/2724108603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680cfb20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 -- 172.21.15.122:0/2724108603 >> 172.21.15.122:0/2724108603 conn(0x7fdc680d3d10 msgr2=0x7fdc680d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 -- 172.21.15.122:0/2724108603 shutdown_connections 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.367+0000 7fdc6e355640 1 -- 172.21.15.122:0/2724108603 wait complete. 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.368+0000 7fdc6e355640 1 Processor -- start 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.368+0000 7fdc6e355640 1 -- start start 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6e355640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 0x7fdc680c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6e355640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6e355640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc680cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6e355640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6815e280 con 0x7fdc680d4920 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6e355640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680cf320 con 0x7fdc681621c0 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6e355640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6815e400 con 0x7fdc681625a0 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 0x7fdc680c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 0x7fdc680c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57082/0 (socket says 172.21.15.122:57082) 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 -- 172.21.15.122:0/1503162708 learned_addr learned my addr 172.21.15.122:0/1503162708 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc6c8f2640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc680cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc677fe640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 -- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 msgr2=0x7fdc680cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc680cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 -- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 msgr2=0x7fdc680ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.369+0000 7fdc67fff640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc67fff640 1 -- 172.21.15.122:0/1503162708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc680c9db0 con 0x7fdc680d4920 2024-03-06T07:22:06.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc67fff640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 0x7fdc680c9870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdc5802bee0 tx=0x7fdc580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58035040 con 0x7fdc680d4920 2024-03-06T07:22:06.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5803b920 con 0x7fdc680d4920 2024-03-06T07:22:06.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc681e83f0 con 0x7fdc680d4920 2024-03-06T07:22:06.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5803bd10 con 0x7fdc680d4920 2024-03-06T07:22:06.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.370+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc68148d00 con 0x7fdc680d4920 2024-03-06T07:22:06.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.371+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6815ba80 con 0x7fdc680d4920 2024-03-06T07:22:06.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.371+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc58038020 con 0x7fdc680d4920 2024-03-06T07:22:06.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.371+0000 7fdc64ff9640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc5406b4d0 0x7fdc5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.372+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc580ac110 con 0x7fdc680d4920 2024-03-06T07:22:06.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.372+0000 7fdc677fe640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc5406b4d0 0x7fdc5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.372+0000 7fdc677fe640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc5406b4d0 0x7fdc5406d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdc680b5310 tx=0x7fdc5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.376+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc680c9db0 con 0x7fdc680d4920 2024-03-06T07:22:06.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.547+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc6804f330 con 0x7fdc680d4920 2024-03-06T07:22:06.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.548+0000 7fdc64ff9640 1 -- 172.21.15.122:0/1503162708 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7fdc5807a9d0 con 0x7fdc680d4920 2024-03-06T07:22:06.550 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:06.551 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:06.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc5406b4d0 msgr2=0x7fdc5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc5406b4d0 0x7fdc5406d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdc680b5310 tx=0x7fdc5c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 msgr2=0x7fdc680c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 0x7fdc680c9870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdc5802bee0 tx=0x7fdc580046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 shutdown_connections 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc5406b4d0 0x7fdc5406d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc681625a0 0x7fdc680cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc681621c0 0x7fdc680ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 --2- 172.21.15.122:0/1503162708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc680d4920 0x7fdc680c9870 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 >> 172.21.15.122:0/1503162708 conn(0x7fdc680d3d10 msgr2=0x7fdc6806b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 shutdown_connections 2024-03-06T07:22:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.550+0000 7fdc6e355640 1 -- 172.21.15.122:0/1503162708 wait complete. 2024-03-06T07:22:06.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:06.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:06.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.665+0000 7f6f3d0f4640 1 Processor -- start 2024-03-06T07:22:06.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.665+0000 7f6f3d0f4640 1 -- start start 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.665+0000 7f6f3d0f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6f380d0af0 0x7f6f380cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.665+0000 7f6f3d0f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380cd4b0 con 0x7f6f3815c050 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.665+0000 7f6f3d0f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380cd790 con 0x7f6f3806fd10 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.665+0000 7f6f3d0f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380cda70 con 0x7f6f380d0af0 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f36d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6f3815c050 0x7f6f3806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38578/0 (socket says 172.21.15.122:38578) 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f36d9d640 1 -- 172.21.15.122:0/3294104598 learned_addr learned my addr 172.21.15.122:0/3294104598 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f3759e640 1 --2- 172.21.15.122:0/3294104598 >> v2:172.21.15.122:3300/0 conn(0x7f6f380d0af0 0x7f6f380cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f3759e640 1 -- 172.21.15.122:0/3294104598 >> v1:172.21.15.169:6789/0 conn(0x7f6f3806fd10 legacy=0x7f6f380cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f3759e640 1 -- 172.21.15.122:0/3294104598 >> v1:172.21.15.122:6789/0 conn(0x7f6f3815c050 legacy=0x7f6f3806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f3659c640 1 -- 172.21.15.122:0/3294104598 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271355504 0 0) 0x7f6f380cd790 con 0x7f6f3806fd10 2024-03-06T07:22:06.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.666+0000 7f6f3759e640 1 -- 172.21.15.122:0/3294104598 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380cdd50 con 0x7f6f380d0af0 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3759e640 1 --2- 172.21.15.122:0/3294104598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f380d0af0 0x7f6f380cce70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009d90 tx=0x7f6f3002bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89000b6ea019a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3659c640 1 -- 172.21.15.122:0/3294104598 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30039070 con 0x7f6f380d0af0 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3659c640 1 -- 172.21.15.122:0/3294104598 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3002ca20 con 0x7f6f380d0af0 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3659c640 1 -- 172.21.15.122:0/3294104598 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3002cdf0 con 0x7f6f380d0af0 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/3294104598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f380d0af0 msgr2=0x7f6f380cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/3294104598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f380d0af0 0x7f6f380cce70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009d90 tx=0x7f6f3002bcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/3294104598 shutdown_connections 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/3294104598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f380d0af0 0x7f6f380cce70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/3294104598 >> 172.21.15.122:0/3294104598 conn(0x7f6f380b3800 msgr2=0x7f6f380b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/3294104598 shutdown_connections 2024-03-06T07:22:06.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.667+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/3294104598 wait complete. 2024-03-06T07:22:06.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 Processor -- start 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 -- start start 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f3806fd10 0x7f6f3814d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f380d0af0 0x7f6f3815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 0x7f6f38153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38166e50 con 0x7f6f3815c050 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38166cd0 con 0x7f6f3806fd10 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3d0f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38166fd0 con 0x7f6f380d0af0 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 0x7f6f38153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 0x7f6f38153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57096/0 (socket says 172.21.15.122:57096) 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f3759e640 1 -- 172.21.15.122:0/56361253 learned_addr learned my addr 172.21.15.122:0/56361253 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f2ffff640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f380d0af0 0x7f6f3815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.669+0000 7f6f36d9d640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f3806fd10 0x7f6f3814d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3759e640 1 -- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f380d0af0 msgr2=0x7f6f3815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3759e640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f380d0af0 0x7f6f3815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3759e640 1 -- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f3806fd10 msgr2=0x7f6f3814d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3759e640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f3806fd10 0x7f6f3814d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3759e640 1 -- 172.21.15.122:0/56361253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380b4990 con 0x7f6f3815c050 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3759e640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 0x7f6f38153f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6f30037040 tx=0x7f6f30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30039050 con 0x7f6f3815c050 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f30045070 con 0x7f6f3815c050 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380b4c70 con 0x7f6f3815c050 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30040690 con 0x7f6f3815c050 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.670+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380b5230 con 0x7f6f3815c050 2024-03-06T07:22:06.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.671+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3809fe60 con 0x7f6f3815c050 2024-03-06T07:22:06.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.671+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f30040830 con 0x7f6f3815c050 2024-03-06T07:22:06.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.672+0000 7f6f2f7fe640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f1806b4d0 0x7f6f1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.672+0000 7f6f36d9d640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f1806b4d0 0x7f6f1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.672+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f300ac840 con 0x7f6f3815c050 2024-03-06T07:22:06.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.672+0000 7f6f36d9d640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f1806b4d0 0x7f6f1806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6f28000f60 tx=0x7f6f28007e20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.676+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f3004f020 con 0x7f6f3815c050 2024-03-06T07:22:06.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.848+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f38068ff0 con 0x7f6f3815c050 2024-03-06T07:22:06.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.849+0000 7f6f2f7fe640 1 -- 172.21.15.122:0/56361253 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f30034e20 con 0x7f6f3815c050 2024-03-06T07:22:06.851 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f1806b4d0 msgr2=0x7f6f1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f1806b4d0 0x7f6f1806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6f28000f60 tx=0x7f6f28007e20 comp rx=0 tx=0).stop 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 msgr2=0x7f6f38153f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 0x7f6f38153f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6f30037040 tx=0x7f6f30004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 shutdown_connections 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f1806b4d0 0x7f6f1806d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f3815c050 0x7f6f38153f60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f380d0af0 0x7f6f3815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 --2- 172.21.15.122:0/56361253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f3806fd10 0x7f6f3814d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 >> 172.21.15.122:0/56361253 conn(0x7f6f380b3800 msgr2=0x7f6f380cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.850+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 shutdown_connections 2024-03-06T07:22:06.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.851+0000 7f6f3d0f4640 1 -- 172.21.15.122:0/56361253 wait complete. 2024-03-06T07:22:06.863 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.964+0000 7fe10d301640 1 Processor -- start 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.964+0000 7fe10d301640 1 -- start start 2024-03-06T07:22:06.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe10d301640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe1081614d0 0x7fe1080d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe10d301640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe1080d1910 0x7fe1080d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe10d301640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1081618b0 con 0x7fe1080d22e0 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe10d301640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080cf400 con 0x7fe1080d1910 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe10d301640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080d2710 con 0x7fe1081614d0 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe0fffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe1080d1910 0x7fe1080d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe1077fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe1080d22e0 0x7fe1080d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38588/0 (socket says 172.21.15.122:38588) 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe1077fe640 1 -- 172.21.15.122:0/343133863 learned_addr learned my addr 172.21.15.122:0/343133863 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe106ffd640 1 --2- 172.21.15.122:0/343133863 >> v2:172.21.15.169:3300/0 conn(0x7fe1081614d0 0x7fe1080d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe0fffff640 1 -- 172.21.15.122:0/343133863 >> v2:172.21.15.169:3300/0 conn(0x7fe1081614d0 msgr2=0x7fe1080d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe0fffff640 1 --2- 172.21.15.122:0/343133863 >> v2:172.21.15.169:3300/0 conn(0x7fe1081614d0 0x7fe1080d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe0fffff640 1 -- 172.21.15.122:0/343133863 >> v1:172.21.15.122:6789/0 conn(0x7fe1080d22e0 legacy=0x7fe1080d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe1067fc640 1 -- 172.21.15.122:0/343133863 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199972795 0 0) 0x7fe1081618b0 con 0x7fe1080d22e0 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.965+0000 7fe0fffff640 1 -- 172.21.15.122:0/343133863 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1080cf6a0 con 0x7fe1080d1910 2024-03-06T07:22:06.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe0fffff640 1 --2- 172.21.15.122:0/343133863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe1080d1da0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe10808c470 tx=0x7fe0f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38c6324fd54012e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe1067fc640 1 -- 172.21.15.122:0/343133863 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0035020 con 0x7fe1080d1910 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe1067fc640 1 -- 172.21.15.122:0/343133863 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f0031830 con 0x7fe1080d1910 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe1067fc640 1 -- 172.21.15.122:0/343133863 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0031c00 con 0x7fe1080d1910 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 -- 172.21.15.122:0/343133863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 msgr2=0x7fe1080d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 --2- 172.21.15.122:0/343133863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe1080d1da0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe10808c470 tx=0x7fe0f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 -- 172.21.15.122:0/343133863 shutdown_connections 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 --2- 172.21.15.122:0/343133863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe1080d1da0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 --2- 172.21.15.122:0/343133863 >> v2:172.21.15.169:3300/0 conn(0x7fe1081614d0 0x7fe1080d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 -- 172.21.15.122:0/343133863 >> 172.21.15.122:0/343133863 conn(0x7fe1080d3d40 msgr2=0x7fe1080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 -- 172.21.15.122:0/343133863 shutdown_connections 2024-03-06T07:22:06.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.966+0000 7fe10d301640 1 -- 172.21.15.122:0/343133863 wait complete. 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 Processor -- start 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 -- start start 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe10815be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1080d22e0 0x7fe1080b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1081614d0 0x7fe10815dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080b0910 con 0x7fe1080d1910 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080cfc60 con 0x7fe1080d22e0 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe10d301640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080b1e40 con 0x7fe1081614d0 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe106ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe10815be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe106ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe10815be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57108/0 (socket says 172.21.15.122:57108) 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe106ffd640 1 -- 172.21.15.122:0/2904549096 learned_addr learned my addr 172.21.15.122:0/2904549096 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe0fffff640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1080d22e0 0x7fe1080b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.968+0000 7fe1077fe640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1081614d0 0x7fe10815dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe106ffd640 1 -- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1081614d0 msgr2=0x7fe10815dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe106ffd640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1081614d0 0x7fe10815dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe106ffd640 1 -- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1080d22e0 msgr2=0x7fe1080b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe106ffd640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1080d22e0 0x7fe1080b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe106ffd640 1 -- 172.21.15.122:0/2904549096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1080cb1d0 con 0x7fe1080d1910 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe106ffd640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe10815be80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800e8a0 tx=0x7fe0f800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8017070 con 0x7fe1080d1910 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f800c940 con 0x7fe1080d1910 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1080cb4b0 con 0x7fe1080d1910 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f800cd30 con 0x7fe1080d1910 2024-03-06T07:22:06.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.969+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1080cba70 con 0x7fe1080d1910 2024-03-06T07:22:06.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.970+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1080d1da0 con 0x7fe1080d1910 2024-03-06T07:22:06.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.970+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0f801a070 con 0x7fe1080d1910 2024-03-06T07:22:06.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.971+0000 7fe0ff7fe640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0e806b4d0 0x7fe0e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.971+0000 7fe0fffff640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0e806b4d0 0x7fe0e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.971+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0f808a2f0 con 0x7fe1080d1910 2024-03-06T07:22:06.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.971+0000 7fe0fffff640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0e806b4d0 0x7fe0e806d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0036000 tx=0x7fe0f003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:06.975+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0f8058bb0 con 0x7fe1080d1910 2024-03-06T07:22:07.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.146+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe108068ff0 con 0x7fe1080d1910 2024-03-06T07:22:07.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.146+0000 7fe0ff7fe640 1 -- 172.21.15.122:0/2904549096 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0f805cc50 con 0x7fe1080d1910 2024-03-06T07:22:07.149 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:07.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0e806b4d0 msgr2=0x7fe0e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0e806b4d0 0x7fe0e806d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0036000 tx=0x7fe0f003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 msgr2=0x7fe10815be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe10815be80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800e8a0 tx=0x7fe0f800ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 shutdown_connections 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0e806b4d0 0x7fe0e806d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1081614d0 0x7fe10815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1080d22e0 0x7fe1080b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 --2- 172.21.15.122:0/2904549096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1080d1910 0x7fe10815be80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 >> 172.21.15.122:0/2904549096 conn(0x7fe1080d3d40 msgr2=0x7fe10806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 shutdown_connections 2024-03-06T07:22:07.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.148+0000 7fe10d301640 1 -- 172.21.15.122:0/2904549096 wait complete. 2024-03-06T07:22:07.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:07.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.263+0000 7fe11173e640 1 Processor -- start 2024-03-06T07:22:07.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.263+0000 7fe11173e640 1 -- start start 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe11173e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c145580 0x7fe10c0cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe11173e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0cfa90 con 0x7fe10c06b6c0 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe11173e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0cfd70 con 0x7fe10c0cd7e0 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe11173e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c06d8d0 con 0x7fe10c145580 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe10affd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c145580 0x7fe10c0cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe10b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe10c06b6c0 0x7fe10c06dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38594/0 (socket says 172.21.15.122:38594) 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe10b7fe640 1 -- 172.21.15.122:0/508483829 learned_addr learned my addr 172.21.15.122:0/508483829 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.264+0000 7fe109ffb640 1 -- 172.21.15.122:0/508483829 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759925166 0 0) 0x7fe10c0cfa90 con 0x7fe10c06b6c0 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe109ffb640 1 -- 172.21.15.122:0/508483829 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f4003520 con 0x7fe10c06b6c0 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe10affd640 1 -- 172.21.15.122:0/508483829 >> v1:172.21.15.169:6789/0 conn(0x7fe10c0cd7e0 legacy=0x7fe10c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe109ffb640 1 -- 172.21.15.122:0/508483829 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180353831 0 0) 0x7fe10c0cfd70 con 0x7fe10c0cd7e0 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe10affd640 1 -- 172.21.15.122:0/508483829 >> v1:172.21.15.122:6789/0 conn(0x7fe10c06b6c0 legacy=0x7fe10c06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe10affd640 1 -- 172.21.15.122:0/508483829 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c0d0050 con 0x7fe10c145580 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe10affd640 1 --2- 172.21.15.122:0/508483829 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c145580 0x7fe10c0cd2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009dc0 tx=0x7fe0fc02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9948b82e05473012 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe109ffb640 1 -- 172.21.15.122:0/508483829 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc039070 con 0x7fe10c145580 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe109ffb640 1 -- 172.21.15.122:0/508483829 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc02ccd0 con 0x7fe10c145580 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.265+0000 7fe109ffb640 1 -- 172.21.15.122:0/508483829 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc034640 con 0x7fe10c145580 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 -- 172.21.15.122:0/508483829 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c145580 msgr2=0x7fe10c0cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 --2- 172.21.15.122:0/508483829 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c145580 0x7fe10c0cd2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009dc0 tx=0x7fe0fc02bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 -- 172.21.15.122:0/508483829 shutdown_connections 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 --2- 172.21.15.122:0/508483829 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c145580 0x7fe10c0cd2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 -- 172.21.15.122:0/508483829 >> 172.21.15.122:0/508483829 conn(0x7fe10c0d0a30 msgr2=0x7fe10c0d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 -- 172.21.15.122:0/508483829 shutdown_connections 2024-03-06T07:22:07.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.266+0000 7fe11173e640 1 -- 172.21.15.122:0/508483829 wait complete. 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.267+0000 7fe11173e640 1 Processor -- start 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.267+0000 7fe11173e640 1 -- start start 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe11173e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe10c06b6c0 0x7fe10c1f1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe11173e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c0cd7e0 0x7fe10c0c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe11173e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 0x7fe10c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe11173e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c0d1670 con 0x7fe10c145580 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe11173e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c145960 con 0x7fe10c06b6c0 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe11173e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c0d17f0 con 0x7fe10c0cd7e0 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 0x7fe10c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 0x7fe10c0c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57120/0 (socket says 172.21.15.122:57120) 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 -- 172.21.15.122:0/249389915 learned_addr learned my addr 172.21.15.122:0/249389915 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10a7fc640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c0cd7e0 0x7fe10c0c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10affd640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe10c06b6c0 0x7fe10c1f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 -- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c0cd7e0 msgr2=0x7fe10c0c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c0cd7e0 0x7fe10c0c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 -- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe10c06b6c0 msgr2=0x7fe10c1f1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe10c06b6c0 0x7fe10c1f1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.268+0000 7fe10b7fe640 1 -- 172.21.15.122:0/249389915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c144760 con 0x7fe10c145580 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.269+0000 7fe10b7fe640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 0x7fe10c0c78d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c390 tx=0x7fe10400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.269+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104017070 con 0x7fe10c145580 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.269+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe104005ce0 con 0x7fe10c145580 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.269+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c144a00 con 0x7fe10c145580 2024-03-06T07:22:07.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.269+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104007310 con 0x7fe10c145580 2024-03-06T07:22:07.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.269+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c0c63c0 con 0x7fe10c145580 2024-03-06T07:22:07.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.270+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c0734b0 con 0x7fe10c145580 2024-03-06T07:22:07.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.270+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe10401a070 con 0x7fe10c145580 2024-03-06T07:22:07.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.271+0000 7fe0f37fe640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0f806b4d0 0x7fe0f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.271+0000 7fe10affd640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0f806b4d0 0x7fe0f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.271+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1040894d0 con 0x7fe10c145580 2024-03-06T07:22:07.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.271+0000 7fe10affd640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0f806b4d0 0x7fe0f806d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc036000 tx=0x7fe0fc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.275+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe104057d00 con 0x7fe10c145580 2024-03-06T07:22:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.449+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe10c068ff0 con 0x7fe10c145580 2024-03-06T07:22:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.449+0000 7fe0f37fe640 1 -- 172.21.15.122:0/249389915 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe104087e20 con 0x7fe10c145580 2024-03-06T07:22:07.452 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:07.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.451+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0f806b4d0 msgr2=0x7fe0f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.451+0000 7fe11173e640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0f806b4d0 0x7fe0f806d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc036000 tx=0x7fe0fc030000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.451+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 msgr2=0x7fe10c0c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.451+0000 7fe11173e640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 0x7fe10c0c78d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c390 tx=0x7fe10400c850 comp rx=0 tx=0).stop 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 shutdown_connections 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0f806b4d0 0x7fe0f806d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe10c145580 0x7fe10c0c78d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe10c0cd7e0 0x7fe10c0c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 --2- 172.21.15.122:0/249389915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe10c06b6c0 0x7fe10c1f1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 >> 172.21.15.122:0/249389915 conn(0x7fe10c0d0a30 msgr2=0x7fe10c06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 shutdown_connections 2024-03-06T07:22:07.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.452+0000 7fe11173e640 1 -- 172.21.15.122:0/249389915 wait complete. 2024-03-06T07:22:07.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:07.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.562+0000 7fcb87c2a640 1 Processor -- start 2024-03-06T07:22:07.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.562+0000 7fcb87c2a640 1 -- start start 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb87c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8015e110 0x7fcb800ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb87c2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcb800b5630 0x7fcb8006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb87c2a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb800b0160 con 0x7fcb8006b4f0 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb87c2a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb800ae030 con 0x7fcb800b5630 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb87c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800ad9e0 con 0x7fcb8015e110 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb851c5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcb800b5630 0x7fcb8006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb861c7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcb8006b4f0 0x7fcb800b38f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38610/0 (socket says 172.21.15.122:38610) 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb859c6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8015e110 0x7fcb800ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.563+0000 7fcb861c7640 1 -- 172.21.15.122:0/2614465051 learned_addr learned my addr 172.21.15.122:0/2614465051 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb851c5640 1 -- 172.21.15.122:0/2614465051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8015e110 msgr2=0x7fcb800ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb851c5640 1 --2- 172.21.15.122:0/2614465051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8015e110 0x7fcb800ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb851c5640 1 -- 172.21.15.122:0/2614465051 >> v1:172.21.15.122:6789/0 conn(0x7fcb8006b4f0 legacy=0x7fcb800b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb851c5640 1 -- 172.21.15.122:0/2614465051 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb800adcc0 con 0x7fcb800b5630 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb849c4640 1 -- 172.21.15.122:0/2614465051 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274844287 0 0) 0x7fcb800b0160 con 0x7fcb8006b4f0 2024-03-06T07:22:07.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb859c6640 1 --2- 172.21.15.122:0/2614465051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8015e110 0x7fcb800ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.564+0000 7fcb851c5640 1 --2- 172.21.15.122:0/2614465051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb8006afb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcb700097a0 tx=0x7fcb7002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda26136c8b2c324 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb849c4640 1 -- 172.21.15.122:0/2614465051 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70034070 con 0x7fcb800b5630 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb849c4640 1 -- 172.21.15.122:0/2614465051 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70030610 con 0x7fcb800b5630 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb849c4640 1 -- 172.21.15.122:0/2614465051 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb700309e0 con 0x7fcb800b5630 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2614465051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 msgr2=0x7fcb8006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2614465051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb8006afb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcb700097a0 tx=0x7fcb7002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2614465051 shutdown_connections 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2614465051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb8006afb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2614465051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8015e110 0x7fcb800ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2614465051 >> 172.21.15.122:0/2614465051 conn(0x7fcb800d0960 msgr2=0x7fcb800d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2614465051 shutdown_connections 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.565+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2614465051 wait complete. 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.566+0000 7fcb87c2a640 1 Processor -- start 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.566+0000 7fcb87c2a640 1 -- start start 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb87c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8006b4f0 0x7fcb801521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb87c2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb800cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb87c2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb8015e110 0x7fcb800cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb87c2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b5bf0 con 0x7fcb800b5630 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb87c2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b5a70 con 0x7fcb8015e110 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb87c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800d1660 con 0x7fcb8006b4f0 2024-03-06T07:22:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb800cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb800cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57146/0 (socket says 172.21.15.122:57146) 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 -- 172.21.15.122:0/2017070675 learned_addr learned my addr 172.21.15.122:0/2017070675 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb859c6640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8006b4f0 0x7fcb801521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb861c7640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb8015e110 0x7fcb800cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 -- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8006b4f0 msgr2=0x7fcb801521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8006b4f0 0x7fcb801521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 -- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb8015e110 msgr2=0x7fcb800cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb8015e110 0x7fcb800cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.567+0000 7fcb851c5640 1 -- 172.21.15.122:0/2017070675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8015e9e0 con 0x7fcb800b5630 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.568+0000 7fcb851c5640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb800cd7f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb70000c00 tx=0x7fcb700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.568+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70034040 con 0x7fcb800b5630 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.568+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70030410 con 0x7fcb800b5630 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.568+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb800ce0f0 con 0x7fcb800b5630 2024-03-06T07:22:07.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.568+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70004250 con 0x7fcb800b5630 2024-03-06T07:22:07.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.568+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb800ce6b0 con 0x7fcb800b5630 2024-03-06T07:22:07.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.569+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb800aed50 con 0x7fcb800b5630 2024-03-06T07:22:07.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.569+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb70037070 con 0x7fcb800b5630 2024-03-06T07:22:07.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.569+0000 7fcb767fc640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb6406b4d0 0x7fcb6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.570+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb700abc20 con 0x7fcb800b5630 2024-03-06T07:22:07.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.570+0000 7fcb859c6640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb6406b4d0 0x7fcb6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.570+0000 7fcb859c6640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb6406b4d0 0x7fcb6406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009bd0 tx=0x7fcb7800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.574+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb800aed50 con 0x7fcb800b5630 2024-03-06T07:22:07.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.784+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb80068fb0 con 0x7fcb800b5630 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.785+0000 7fcb767fc640 1 -- 172.21.15.122:0/2017070675 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7fcb80068fb0 con 0x7fcb800b5630 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb6406b4d0 msgr2=0x7fcb6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb6406b4d0 0x7fcb6406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009bd0 tx=0x7fcb7800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 msgr2=0x7fcb800cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb800cd7f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb70000c00 tx=0x7fcb700046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 shutdown_connections 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb6406b4d0 0x7fcb6406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb8015e110 0x7fcb800cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb800b5630 0x7fcb800cd7f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 --2- 172.21.15.122:0/2017070675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb8006b4f0 0x7fcb801521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 >> 172.21.15.122:0/2017070675 conn(0x7fcb800d0960 msgr2=0x7fcb800b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 shutdown_connections 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.787+0000 7fcb87c2a640 1 -- 172.21.15.122:0/2017070675 wait complete. 2024-03-06T07:22:07.800 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:07.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.896+0000 7ff30be37640 1 Processor -- start 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.897+0000 7ff30be37640 1 -- start start 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.897+0000 7ff30be37640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff3040b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.897+0000 7ff30be37640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 0x7ff3040d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.897+0000 7ff30be37640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3040d25d0 con 0x7ff3040b4dd0 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.897+0000 7ff30be37640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3040d4bf0 con 0x7ff3040d4760 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.897+0000 7ff30be37640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040ce9e0 con 0x7ff3040b4170 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff309bd3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff3040b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff3093d2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 0x7ff3040d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff3093d2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 0x7ff3040d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:52564/0 (socket says 172.21.15.122:52564) 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff3093d2640 1 -- 172.21.15.122:0/2218825879 learned_addr learned my addr 172.21.15.122:0/2218825879 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff309bd3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff3040b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45820/0 (socket says 172.21.15.122:45820) 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff309bd3640 1 -- 172.21.15.122:0/2218825879 >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 msgr2=0x7ff3040d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff309bd3640 1 --2- 172.21.15.122:0/2218825879 >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 0x7ff3040d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff309bd3640 1 -- 172.21.15.122:0/2218825879 >> v1:172.21.15.169:6789/0 conn(0x7ff3040b4dd0 legacy=0x7ff3040cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff308bd1640 1 -- 172.21.15.122:0/2218825879 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351716163 0 0) 0x7ff3040d25d0 con 0x7ff3040b4dd0 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff309bd3640 1 -- 172.21.15.122:0/2218825879 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040d2870 con 0x7ff3040b4170 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.898+0000 7ff3093d2640 1 --2- 172.21.15.122:0/2218825879 >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 0x7ff3040d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff309bd3640 1 --2- 172.21.15.122:0/2218825879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff3040b6a90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff300009d90 tx=0x7ff30002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fc610f209ed5fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff308bd1640 1 -- 172.21.15.122:0/2218825879 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300039070 con 0x7ff3040b4170 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff308bd1640 1 -- 172.21.15.122:0/2218825879 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30002cc50 con 0x7ff3040b4170 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff308bd1640 1 -- 172.21.15.122:0/2218825879 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300034440 con 0x7ff3040b4170 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 -- 172.21.15.122:0/2218825879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 msgr2=0x7ff3040b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 --2- 172.21.15.122:0/2218825879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff3040b6a90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff300009d90 tx=0x7ff30002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 -- 172.21.15.122:0/2218825879 shutdown_connections 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 --2- 172.21.15.122:0/2218825879 >> v2:172.21.15.169:3300/0 conn(0x7ff3040d4760 0x7ff3040d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 --2- 172.21.15.122:0/2218825879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff3040b6a90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 -- 172.21.15.122:0/2218825879 >> 172.21.15.122:0/2218825879 conn(0x7ff3040d3c40 msgr2=0x7ff3040d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 -- 172.21.15.122:0/2218825879 shutdown_connections 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.899+0000 7ff30be37640 1 -- 172.21.15.122:0/2218825879 wait complete. 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.900+0000 7ff30be37640 1 Processor -- start 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 -- start start 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff304157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 0x7ff30415e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3040cc4b0 0x7ff3041f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040b47d0 con 0x7ff3040b4dd0 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040b4550 con 0x7ff3040cc4b0 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30be37640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040b4950 con 0x7ff3040b4170 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff3093d2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 0x7ff30415e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff3093d2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 0x7ff30415e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57152/0 (socket says 172.21.15.122:57152) 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff30a3d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3040cc4b0 0x7ff3041f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff3093d2640 1 -- 172.21.15.122:0/3276599817 learned_addr learned my addr 172.21.15.122:0/3276599817 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.901+0000 7ff309bd3640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff304157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff3093d2640 1 -- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 msgr2=0x7ff304157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff3093d2640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff304157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff3093d2640 1 -- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3040cc4b0 msgr2=0x7ff3041f3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff3093d2640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3040cc4b0 0x7ff3041f3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff3093d2640 1 -- 172.21.15.122:0/3276599817 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040ccc30 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff3093d2640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 0x7ff30415e110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400e9b0 tx=0x7ff2f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4017070 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f400c9a0 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f400cd50 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30415c250 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.902+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30415c810 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.903+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3040cce00 con 0x7ff3040b4dd0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.903+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2f401a070 con 0x7ff3040b4dd0 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.904+0000 7ff2fa7fc640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff2e806b500 0x7ff2e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.904+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2f408a2f0 con 0x7ff3040b4dd0 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.904+0000 7ff309bd3640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff2e806b500 0x7ff2e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.904+0000 7ff309bd3640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff2e806b500 0x7ff2e806d9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff300036000 tx=0x7ff300030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:07.908+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2f4058b20 con 0x7ff3040b4dd0 2024-03-06T07:22:08.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.081+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff30404f330 con 0x7ff3040b4dd0 2024-03-06T07:22:08.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.082+0000 7ff2fa7fc640 1 -- 172.21.15.122:0/3276599817 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7ff2f405cbc0 con 0x7ff3040b4dd0 2024-03-06T07:22:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:08.084 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff2e806b500 msgr2=0x7ff2e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff2e806b500 0x7ff2e806d9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff300036000 tx=0x7ff300030000 comp rx=0 tx=0).stop 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 msgr2=0x7ff30415e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 0x7ff30415e110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400e9b0 tx=0x7ff2f400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 shutdown_connections 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff2e806b500 0x7ff2e806d9c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3040cc4b0 0x7ff3041f3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3040b4dd0 0x7ff30415e110 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 --2- 172.21.15.122:0/3276599817 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff3040b4170 0x7ff304157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 >> 172.21.15.122:0/3276599817 conn(0x7ff3040d3c40 msgr2=0x7ff3040d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 shutdown_connections 2024-03-06T07:22:08.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.084+0000 7ff30be37640 1 -- 172.21.15.122:0/3276599817 wait complete. 2024-03-06T07:22:08.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:08.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:08.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.199+0000 7f7dca2ec640 1 Processor -- start 2024-03-06T07:22:08.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.199+0000 7f7dca2ec640 1 -- start start 2024-03-06T07:22:08.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.199+0000 7f7dca2ec640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7dc40b1e40 0x7f7dc4168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.199+0000 7f7dca2ec640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40d2730 con 0x7f7dc40d4860 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.199+0000 7f7dca2ec640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc41618b0 con 0x7f7dc41614d0 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.199+0000 7f7dca2ec640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40d4cf0 con 0x7f7dc40b1e40 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc37fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7dc40d4860 0x7f7dc40d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38618/0 (socket says 172.21.15.122:38618) 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc37fe640 1 -- 172.21.15.122:0/706973105 learned_addr learned my addr 172.21.15.122:0/706973105 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc8889640 1 --2- 172.21.15.122:0/706973105 >> v2:172.21.15.122:3300/0 conn(0x7f7dc40b1e40 0x7f7dc4168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc2ffd640 1 -- 172.21.15.122:0/706973105 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868976927 0 0) 0x7f7dc40d2730 con 0x7f7dc40d4860 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc2ffd640 1 -- 172.21.15.122:0/706973105 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac003520 con 0x7f7dc40d4860 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc8889640 1 -- 172.21.15.122:0/706973105 >> v1:172.21.15.169:6789/0 conn(0x7f7dc41614d0 legacy=0x7f7dc40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc8889640 1 -- 172.21.15.122:0/706973105 >> v1:172.21.15.122:6789/0 conn(0x7f7dc40d4860 legacy=0x7f7dc40d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.200+0000 7f7dc8889640 1 -- 172.21.15.122:0/706973105 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40cf400 con 0x7f7dc40b1e40 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dc8889640 1 --2- 172.21.15.122:0/706973105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc40b1e40 0x7f7dc4168b60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40cf5d0 tx=0x7f7dbc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43595e8a3372bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dc2ffd640 1 -- 172.21.15.122:0/706973105 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc039070 con 0x7f7dc40b1e40 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dc2ffd640 1 -- 172.21.15.122:0/706973105 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc02cc50 con 0x7f7dc40b1e40 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dc2ffd640 1 -- 172.21.15.122:0/706973105 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc034440 con 0x7f7dc40b1e40 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 -- 172.21.15.122:0/706973105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc40b1e40 msgr2=0x7f7dc4168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/706973105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc40b1e40 0x7f7dc4168b60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40cf5d0 tx=0x7f7dbc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 -- 172.21.15.122:0/706973105 shutdown_connections 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/706973105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc40b1e40 0x7f7dc4168b60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 -- 172.21.15.122:0/706973105 >> 172.21.15.122:0/706973105 conn(0x7f7dc40d3d40 msgr2=0x7f7dc40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 -- 172.21.15.122:0/706973105 shutdown_connections 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.201+0000 7f7dca2ec640 1 -- 172.21.15.122:0/706973105 wait complete. 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.202+0000 7f7dca2ec640 1 Processor -- start 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 -- start start 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dc40b1e40 0x7f7dc4155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dc40d4860 0x7f7dc40b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 0x7f7dc40b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40cfc60 con 0x7f7dc41614d0 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40cf980 con 0x7f7dc40d4860 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dca2ec640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40b0910 con 0x7f7dc40b1e40 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dc8889640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 0x7f7dc40b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dc8889640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 0x7f7dc40b8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57172/0 (socket says 172.21.15.122:57172) 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dc8889640 1 -- 172.21.15.122:0/3167730608 learned_addr learned my addr 172.21.15.122:0/3167730608 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dc37fe640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dc40d4860 0x7f7dc40b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.203+0000 7f7dc3fff640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dc40b1e40 0x7f7dc4155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc8889640 1 -- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dc40b1e40 msgr2=0x7f7dc4155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc8889640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dc40b1e40 0x7f7dc4155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc8889640 1 -- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dc40d4860 msgr2=0x7f7dc40b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc8889640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dc40d4860 0x7f7dc40b81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc8889640 1 -- 172.21.15.122:0/3167730608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40cbad0 con 0x7f7dc41614d0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc8889640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 0x7f7dc40b8700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc037040 tx=0x7f7dbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc044020 con 0x7f7dc41614d0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc03a040 con 0x7f7dc41614d0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc40cbdb0 con 0x7f7dc41614d0 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc039050 con 0x7f7dc41614d0 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.204+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc40cc370 con 0x7f7dc41614d0 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.205+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4068ff0 con 0x7f7dc41614d0 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.205+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dbc003990 con 0x7f7dc41614d0 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.206+0000 7f7dc0ff9640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7db006b4d0 0x7f7db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.206+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dbc0ac6b0 con 0x7f7dc41614d0 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.206+0000 7f7dc3fff640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7db006b4d0 0x7f7db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.206+0000 7f7dc3fff640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7db006b4d0 0x7f7db006d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7db4001780 tx=0x7f7db4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.210+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dbc04e020 con 0x7f7dc41614d0 2024-03-06T07:22:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.381+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7dc40cecd0 con 0x7f7dc41614d0 2024-03-06T07:22:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.381+0000 7f7dc0ff9640 1 -- 172.21.15.122:0/3167730608 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7dbc07af70 con 0x7f7dc41614d0 2024-03-06T07:22:08.384 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7db006b4d0 msgr2=0x7f7db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7db006b4d0 0x7f7db006d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7db4001780 tx=0x7f7db4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 msgr2=0x7f7dc40b8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 0x7f7dc40b8700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc037040 tx=0x7f7dbc004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 shutdown_connections 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7db006b4d0 0x7f7db006d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dc41614d0 0x7f7dc40b8700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dc40d4860 0x7f7dc40b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 --2- 172.21.15.122:0/3167730608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dc40b1e40 0x7f7dc4155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.383+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 >> 172.21.15.122:0/3167730608 conn(0x7f7dc40d3d40 msgr2=0x7f7dc40d0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.384+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 shutdown_connections 2024-03-06T07:22:08.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.384+0000 7f7dca2ec640 1 -- 172.21.15.122:0/3167730608 wait complete. 2024-03-06T07:22:08.396 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:08.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 Processor -- start 2024-03-06T07:22:08.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 -- start start 2024-03-06T07:22:08.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbd581614d0 0x7fbd580d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d24b0 0x7fbd580b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd581618b0 con 0x7fbd580d1910 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd580cf400 con 0x7fbd581614d0 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.497+0000 7fbd5ef5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580cf6a0 con 0x7fbd580d24b0 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd57fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbd580d1910 0x7fbd580d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38626/0 (socket says 172.21.15.122:38626) 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd57fff640 1 -- 172.21.15.122:0/3794393369 learned_addr learned my addr 172.21.15.122:0/3794393369 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5ccfa640 1 --2- 172.21.15.122:0/3794393369 >> v2:172.21.15.169:3300/0 conn(0x7fbd581614d0 0x7fbd580d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5d4fb640 1 --2- 172.21.15.122:0/3794393369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d24b0 0x7fbd580b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd577fe640 1 -- 172.21.15.122:0/3794393369 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585159881 0 0) 0x7fbd581618b0 con 0x7fbd580d1910 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd577fe640 1 -- 172.21.15.122:0/3794393369 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd48003520 con 0x7fbd580d1910 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/3794393369 >> v2:172.21.15.169:3300/0 conn(0x7fbd581614d0 msgr2=0x7fbd580d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5d4fb640 1 --2- 172.21.15.122:0/3794393369 >> v2:172.21.15.169:3300/0 conn(0x7fbd581614d0 0x7fbd580d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/3794393369 >> v1:172.21.15.122:6789/0 conn(0x7fbd580d1910 legacy=0x7fbd580d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/3794393369 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580cf980 con 0x7fbd580d24b0 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.498+0000 7fbd5ccfa640 1 --2- 172.21.15.122:0/3794393369 >> v2:172.21.15.169:3300/0 conn(0x7fbd581614d0 0x7fbd580d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5d4fb640 1 --2- 172.21.15.122:0/3794393369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d24b0 0x7fbd580b0e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009d90 tx=0x7fbd5002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6060277d10024df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd577fe640 1 -- 172.21.15.122:0/3794393369 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50039070 con 0x7fbd580d24b0 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd577fe640 1 -- 172.21.15.122:0/3794393369 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5002cc50 con 0x7fbd580d24b0 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd577fe640 1 -- 172.21.15.122:0/3794393369 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50034440 con 0x7fbd580d24b0 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/3794393369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d24b0 msgr2=0x7fbd580b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/3794393369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d24b0 0x7fbd580b0e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009d90 tx=0x7fbd5002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/3794393369 shutdown_connections 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/3794393369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d24b0 0x7fbd580b0e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/3794393369 >> v2:172.21.15.169:3300/0 conn(0x7fbd581614d0 0x7fbd580d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/3794393369 >> 172.21.15.122:0/3794393369 conn(0x7fbd580d3d40 msgr2=0x7fbd580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/3794393369 shutdown_connections 2024-03-06T07:22:08.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.499+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/3794393369 wait complete. 2024-03-06T07:22:08.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 Processor -- start 2024-03-06T07:22:08.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 -- start start 2024-03-06T07:22:08.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d1910 0x7fbd5814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd580d24b0 0x7fbd581f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 0x7fbd581f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580b1370 con 0x7fbd581614d0 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580b0910 con 0x7fbd580d24b0 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.501+0000 7fbd5ef5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580b14f0 con 0x7fbd580d1910 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 0x7fbd581f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 0x7fbd581f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57182/0 (socket says 172.21.15.122:57182) 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/2304464761 learned_addr learned my addr 172.21.15.122:0/2304464761 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5ccfa640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d1910 0x7fbd5814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd57fff640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd580d24b0 0x7fbd581f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d1910 msgr2=0x7fbd5814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d1910 0x7fbd5814caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd580d24b0 msgr2=0x7fbd581f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd580d24b0 0x7fbd581f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 -- 172.21.15.122:0/2304464761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd581f80f0 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5d4fb640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 0x7fbd581f3b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd50037040 tx=0x7fbd50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50042030 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5003a040 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd580cce60 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50039070 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.502+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd580cd420 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.503+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5806d740 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.503+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd50045070 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.504+0000 7fbd557fa640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd3c06b4d0 0x7fbd3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.504+0000 7fbd5ccfa640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd3c06b4d0 0x7fbd3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.504+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd500abdd0 con 0x7fbd581614d0 2024-03-06T07:22:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.504+0000 7fbd5ccfa640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd3c06b4d0 0x7fbd3c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbd580d1f30 tx=0x7fbd4c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.508+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd5806d740 con 0x7fbd581614d0 2024-03-06T07:22:08.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.682+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbd580d1020 con 0x7fbd581614d0 2024-03-06T07:22:08.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.682+0000 7fbd557fa640 1 -- 172.21.15.122:0/2304464761 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbd580d1020 con 0x7fbd581614d0 2024-03-06T07:22:08.685 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd3c06b4d0 msgr2=0x7fbd3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd3c06b4d0 0x7fbd3c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbd580d1f30 tx=0x7fbd4c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 msgr2=0x7fbd581f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 0x7fbd581f3b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd50037040 tx=0x7fbd50004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 shutdown_connections 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd3c06b4d0 0x7fbd3c06d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd581614d0 0x7fbd581f3b70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd580d24b0 0x7fbd581f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 --2- 172.21.15.122:0/2304464761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd580d1910 0x7fbd5814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 >> 172.21.15.122:0/2304464761 conn(0x7fbd580d3d40 msgr2=0x7fbd5815fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 shutdown_connections 2024-03-06T07:22:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.684+0000 7fbd5ef5e640 1 -- 172.21.15.122:0/2304464761 wait complete. 2024-03-06T07:22:08.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:08.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.798+0000 7f2ce31e9640 1 Processor -- start 2024-03-06T07:22:08.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.798+0000 7f2ce31e9640 1 -- start start 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce31e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc15bbb0 0x7f2cdc06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce31e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2cdc06b4f0 0x7f2cdc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce31e9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2cdc15ec40 0x7f2cdc0cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce31e9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc161970 con 0x7f2cdc06b4f0 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce31e9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc1624f0 con 0x7f2cdc15ec40 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce31e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0cea90 con 0x7f2cdc15bbb0 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2cdc06b4f0 0x7f2cdc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2cdc06b4f0 0x7f2cdc15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57196/0 (socket says 172.21.15.122:57196) 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 -- 172.21.15.122:0/4279309119 learned_addr learned my addr 172.21.15.122:0/4279309119 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce0f85640 1 --2- 172.21.15.122:0/4279309119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc15bbb0 0x7f2cdc06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2ce1786640 1 --2- 172.21.15.122:0/4279309119 >> v2:172.21.15.169:3300/0 conn(0x7f2cdc15ec40 0x7f2cdc0cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 -- 172.21.15.122:0/4279309119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc15bbb0 msgr2=0x7f2cdc06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 --2- 172.21.15.122:0/4279309119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc15bbb0 0x7f2cdc06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 -- 172.21.15.122:0/4279309119 >> v2:172.21.15.169:3300/0 conn(0x7f2cdc15ec40 msgr2=0x7f2cdc0cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 --2- 172.21.15.122:0/4279309119 >> v2:172.21.15.169:3300/0 conn(0x7f2cdc15ec40 0x7f2cdc0cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.799+0000 7f2cdbfff640 1 -- 172.21.15.122:0/4279309119 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc161bc0 con 0x7f2cdc06b4f0 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2cdbfff640 1 --2- 172.21.15.122:0/4279309119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc06b4f0 0x7f2cdc15e700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc073210 tx=0x7f2cc802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85c05ba218b85d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2cdb7fe640 1 -- 172.21.15.122:0/4279309119 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8035020 con 0x7f2cdc06b4f0 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2cdb7fe640 1 -- 172.21.15.122:0/4279309119 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc8031870 con 0x7f2cdc06b4f0 2024-03-06T07:22:08.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2cdb7fe640 1 -- 172.21.15.122:0/4279309119 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8031c40 con 0x7f2cdc06b4f0 2024-03-06T07:22:08.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 -- 172.21.15.122:0/4279309119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc06b4f0 msgr2=0x7f2cdc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/4279309119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc06b4f0 0x7f2cdc15e700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc073210 tx=0x7f2cc802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:08.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 -- 172.21.15.122:0/4279309119 shutdown_connections 2024-03-06T07:22:08.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/4279309119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc06b4f0 0x7f2cdc15e700 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/4279309119 >> v2:172.21.15.169:3300/0 conn(0x7f2cdc15ec40 0x7f2cdc0cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/4279309119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc15bbb0 0x7f2cdc06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 -- 172.21.15.122:0/4279309119 >> 172.21.15.122:0/4279309119 conn(0x7f2cdc0b2660 msgr2=0x7f2cdc0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 -- 172.21.15.122:0/4279309119 shutdown_connections 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.800+0000 7f2ce31e9640 1 -- 172.21.15.122:0/4279309119 wait complete. 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 Processor -- start 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 -- start start 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc06b4f0 0x7f2cdc15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cdc15ec40 0x7f2cdc154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 0x7f2cdc0b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc15f0d0 con 0x7f2cdc1506d0 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc162180 con 0x7f2cdc15ec40 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce31e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0cd8e0 con 0x7f2cdc06b4f0 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.802+0000 7f2ce1786640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 0x7f2cdc0b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 0x7f2cdc0b4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57212/0 (socket says 172.21.15.122:57212) 2024-03-06T07:22:08.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 -- 172.21.15.122:0/1169205282 learned_addr learned my addr 172.21.15.122:0/1169205282 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce0f85640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc06b4f0 0x7f2cdc15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2cdbfff640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cdc15ec40 0x7f2cdc154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 -- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc06b4f0 msgr2=0x7f2cdc15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc06b4f0 0x7f2cdc15a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 -- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cdc15ec40 msgr2=0x7f2cdc154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cdc15ec40 0x7f2cdc154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 -- 172.21.15.122:0/1169205282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc155460 con 0x7f2cdc1506d0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce1786640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 0x7f2cdc0b4a40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009d30 tx=0x7f2cd400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4018020 con 0x7f2cdc1506d0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd4014480 con 0x7f2cdc1506d0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc155700 con 0x7f2cdc1506d0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4014870 con 0x7f2cdc1506d0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.803+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc155cc0 con 0x7f2cdc1506d0 2024-03-06T07:22:08.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.804+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc0733b0 con 0x7f2cdc1506d0 2024-03-06T07:22:08.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.804+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cd4005d10 con 0x7f2cdc1506d0 2024-03-06T07:22:08.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.805+0000 7f2cd97fa640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2cc006b500 0x7f2cc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.805+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cd408ab10 con 0x7f2cdc1506d0 2024-03-06T07:22:08.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.805+0000 7f2ce0f85640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2cc006b500 0x7f2cc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.805+0000 7f2ce0f85640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2cc006b500 0x7f2cc006d9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc1568c0 tx=0x7f2cd0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.809+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2cd4022070 con 0x7f2cdc1506d0 2024-03-06T07:22:08.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.981+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2cdc068ff0 con 0x7f2cdc1506d0 2024-03-06T07:22:08.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.981+0000 7f2cd97fa640 1 -- 172.21.15.122:0/1169205282 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2cd4010040 con 0x7f2cdc1506d0 2024-03-06T07:22:08.984 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.983+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2cc006b500 msgr2=0x7f2cc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.983+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2cc006b500 0x7f2cc006d9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc1568c0 tx=0x7f2cd0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.983+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 msgr2=0x7f2cdc0b4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.983+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 0x7f2cdc0b4a40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009d30 tx=0x7f2cd400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 shutdown_connections 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2cc006b500 0x7f2cc006d9c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cdc1506d0 0x7f2cdc0b4a40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cdc15ec40 0x7f2cdc154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 --2- 172.21.15.122:0/1169205282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2cdc06b4f0 0x7f2cdc15a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 >> 172.21.15.122:0/1169205282 conn(0x7f2cdc0b2660 msgr2=0x7f2cdc072650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 shutdown_connections 2024-03-06T07:22:08.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:08.984+0000 7f2ce31e9640 1 -- 172.21.15.122:0/1169205282 wait complete. 2024-03-06T07:22:08.996 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:09.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.094+0000 7f1c0255c640 1 Processor -- start 2024-03-06T07:22:09.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.094+0000 7f1c0255c640 1 -- start start 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c0255c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1bfc0d00c0 0x7f1bfc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c0255c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0cf850 0x7f1bfc0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c0255c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1bfc0ce1b0 0x7f1bfc0ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c0255c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc06ca90 con 0x7f1bfc0d00c0 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c0255c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc0d09e0 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c0255c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc15e380 con 0x7f1bfc0cf850 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1bfc0d00c0 0x7f1bfc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1bfc0d00c0 0x7f1bfc0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57218/0 (socket says 172.21.15.122:57218) 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 -- 172.21.15.122:0/1532686824 learned_addr learned my addr 172.21.15.122:0/1532686824 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfb7fe640 1 --2- 172.21.15.122:0/1532686824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0cf850 0x7f1bfc0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1c00af9640 1 --2- 172.21.15.122:0/1532686824 >> v2:172.21.15.169:3300/0 conn(0x7f1bfc0ce1b0 0x7f1bfc0ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 -- 172.21.15.122:0/1532686824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0cf850 msgr2=0x7f1bfc0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 --2- 172.21.15.122:0/1532686824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0cf850 0x7f1bfc0cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 -- 172.21.15.122:0/1532686824 >> v2:172.21.15.169:3300/0 conn(0x7f1bfc0ce1b0 msgr2=0x7f1bfc0ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 --2- 172.21.15.122:0/1532686824 >> v2:172.21.15.169:3300/0 conn(0x7f1bfc0ce1b0 0x7f1bfc0ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.095+0000 7f1bfbfff640 1 -- 172.21.15.122:0/1532686824 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0cdc10 con 0x7f1bfc0d00c0 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1bfbfff640 1 --2- 172.21.15.122:0/1532686824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0d00c0 0x7f1bfc0d04a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc071cf0 tx=0x7f1bec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82e8b7caed86550 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1bfaffd640 1 -- 172.21.15.122:0/1532686824 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec035020 con 0x7f1bfc0d00c0 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1bfaffd640 1 -- 172.21.15.122:0/1532686824 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec031870 con 0x7f1bfc0d00c0 2024-03-06T07:22:09.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1bfaffd640 1 -- 172.21.15.122:0/1532686824 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec031c40 con 0x7f1bfc0d00c0 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 -- 172.21.15.122:0/1532686824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0d00c0 msgr2=0x7f1bfc0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 --2- 172.21.15.122:0/1532686824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0d00c0 0x7f1bfc0d04a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc071cf0 tx=0x7f1bec02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 -- 172.21.15.122:0/1532686824 shutdown_connections 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 --2- 172.21.15.122:0/1532686824 >> v2:172.21.15.169:3300/0 conn(0x7f1bfc0ce1b0 0x7f1bfc0ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 --2- 172.21.15.122:0/1532686824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0cf850 0x7f1bfc0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 --2- 172.21.15.122:0/1532686824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0d00c0 0x7f1bfc0d04a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 -- 172.21.15.122:0/1532686824 >> 172.21.15.122:0/1532686824 conn(0x7f1bfc0b35a0 msgr2=0x7f1bfc0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 -- 172.21.15.122:0/1532686824 shutdown_connections 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.096+0000 7f1c0255c640 1 -- 172.21.15.122:0/1532686824 wait complete. 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.097+0000 7f1c0255c640 1 Processor -- start 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 -- start start 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 0x7f1bfc14c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1bfc0cf850 0x7f1bfc14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0d00c0 0x7f1bfc0b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc06c720 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc06c440 con 0x7f1bfc0cf850 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1c0255c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc0cd900 con 0x7f1bfc0d00c0 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1bfbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 0x7f1bfc14c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.098+0000 7f1bfbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 0x7f1bfc14c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57228/0 (socket says 172.21.15.122:57228) 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 -- 172.21.15.122:0/2016423264 learned_addr learned my addr 172.21.15.122:0/2016423264 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:09.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfb7fe640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1bfc0cf850 0x7f1bfc14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1c00af9640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0d00c0 0x7f1bfc0b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 -- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0d00c0 msgr2=0x7f1bfc0b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0d00c0 0x7f1bfc0b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 -- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1bfc0cf850 msgr2=0x7f1bfc14ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1bfc0cf850 0x7f1bfc14ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 -- 172.21.15.122:0/2016423264 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0b5e90 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bfbfff640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 0x7f1bfc14c990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000c00 tx=0x7f1bec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec035040 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec030d00 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc0b6170 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0456c0 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.099+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc0b6730 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.100+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc0b6b60 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.100+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bec038020 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.101+0000 7f1bf8ff9640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1be806b4d0 0x7f1be806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.101+0000 7f1bfb7fe640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1be806b4d0 0x7f1be806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.101+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bec0ac320 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.101+0000 7f1bfb7fe640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1be806b4d0 0x7f1be806d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc1f72b0 tx=0x7f1bf0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.106+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bec07abe0 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.315+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bfc04f330 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.316+0000 7f1bf8ff9640 1 -- 172.21.15.122:0/2016423264 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f1bec07ec80 con 0x7f1bfc0ce1b0 2024-03-06T07:22:09.318 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:09.318 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:09.318 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1be806b4d0 msgr2=0x7f1be806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1be806b4d0 0x7f1be806d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc1f72b0 tx=0x7f1bf0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 msgr2=0x7f1bfc14c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 0x7f1bfc14c990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000c00 tx=0x7f1bec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 shutdown_connections 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1be806b4d0 0x7f1be806d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bfc0d00c0 0x7f1bfc0b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.317+0000 7f1c0255c640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1bfc0cf850 0x7f1bfc14ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.318+0000 7f1c0255c640 1 --2- 172.21.15.122:0/2016423264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bfc0ce1b0 0x7f1bfc14c990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.318+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 >> 172.21.15.122:0/2016423264 conn(0x7f1bfc0b35a0 msgr2=0x7f1bfc0b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.318+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 shutdown_connections 2024-03-06T07:22:09.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.318+0000 7f1c0255c640 1 -- 172.21.15.122:0/2016423264 wait complete. 2024-03-06T07:22:09.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:09.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.428+0000 7f16b24df640 1 Processor -- start 2024-03-06T07:22:09.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16b24df640 1 -- start start 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16b24df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac0d16e0 0x7f16ac0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16b24df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0cebe0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16b24df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0ceec0 con 0x7f16ac15fa10 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16b24df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0ce980 con 0x7f16ac0d16e0 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16abfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f16ac15bdd0 0x7f16ac0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38640/0 (socket says 172.21.15.122:38640) 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.429+0000 7f16abfff640 1 -- 172.21.15.122:0/3117934383 learned_addr learned my addr 172.21.15.122:0/3117934383 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16b0a7c640 1 --2- 172.21.15.122:0/3117934383 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac0d16e0 0x7f16ac0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16aaffd640 1 -- 172.21.15.122:0/3117934383 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557368 0 0) 0x7f16ac0cebe0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16aaffd640 1 -- 172.21.15.122:0/3117934383 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1694003520 con 0x7f16ac15bdd0 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16aaffd640 1 -- 172.21.15.122:0/3117934383 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698539045 0 0) 0x7f16ac0ceec0 con 0x7f16ac15fa10 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16b0a7c640 1 -- 172.21.15.122:0/3117934383 >> v1:172.21.15.169:6789/0 conn(0x7f16ac15fa10 legacy=0x7f16ac0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16b0a7c640 1 -- 172.21.15.122:0/3117934383 >> v1:172.21.15.122:6789/0 conn(0x7f16ac15bdd0 legacy=0x7f16ac0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.430+0000 7f16b0a7c640 1 -- 172.21.15.122:0/3117934383 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0cf1a0 con 0x7f16ac0d16e0 2024-03-06T07:22:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b0a7c640 1 --2- 172.21.15.122:0/3117934383 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac0d16e0 0x7f16ac0d1b10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f16a4009dc0 tx=0x7f16a402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e98dc2c5112d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16aaffd640 1 -- 172.21.15.122:0/3117934383 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4039070 con 0x7f16ac0d16e0 2024-03-06T07:22:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16aaffd640 1 -- 172.21.15.122:0/3117934383 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a402cc40 con 0x7f16ac0d16e0 2024-03-06T07:22:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16aaffd640 1 -- 172.21.15.122:0/3117934383 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4034440 con 0x7f16ac0d16e0 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 -- 172.21.15.122:0/3117934383 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac0d16e0 msgr2=0x7f16ac0d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 --2- 172.21.15.122:0/3117934383 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac0d16e0 0x7f16ac0d1b10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f16a4009dc0 tx=0x7f16a402be90 comp rx=0 tx=0).stop 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 -- 172.21.15.122:0/3117934383 shutdown_connections 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 --2- 172.21.15.122:0/3117934383 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac0d16e0 0x7f16ac0d1b10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 -- 172.21.15.122:0/3117934383 >> 172.21.15.122:0/3117934383 conn(0x7f16ac09a9f0 msgr2=0x7f16ac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 -- 172.21.15.122:0/3117934383 shutdown_connections 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.431+0000 7f16b24df640 1 -- 172.21.15.122:0/3117934383 wait complete. 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.432+0000 7f16b24df640 1 Processor -- start 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 -- start start 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16ac0d16e0 0x7f16ac15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 0x7f16ac152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac15fa10 0x7f16ac15e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0cc5d0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0d2050 con 0x7f16ac0d16e0 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b24df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac15c1b0 con 0x7f16ac15fa10 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16ab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 0x7f16ac152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16abfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16ac0d16e0 0x7f16ac15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b0a7c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac15fa10 0x7f16ac15e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16ab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 0x7f16ac152b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57236/0 (socket says 172.21.15.122:57236) 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16b0a7c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac15fa10 0x7f16ac15e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45892/0 (socket says 172.21.15.122:45892) 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.433+0000 7f16ab7fe640 1 -- 172.21.15.122:0/1584904328 learned_addr learned my addr 172.21.15.122:0/1584904328 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16ab7fe640 1 -- 172.21.15.122:0/1584904328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac15fa10 msgr2=0x7f16ac15e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16ab7fe640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac15fa10 0x7f16ac15e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16ab7fe640 1 -- 172.21.15.122:0/1584904328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16ac0d16e0 msgr2=0x7f16ac15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16ab7fe640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16ac0d16e0 0x7f16ac15cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16ab7fe640 1 -- 172.21.15.122:0/1584904328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac15e720 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16ab7fe640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 0x7f16ac152b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f169c00c3e0 tx=0x7f169c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c017070 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c0035f0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c0039c0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac0b4f90 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.434+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac15d9e0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.435+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac0b54b0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.435+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f169c01a070 con 0x7f16ac15bdd0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.436+0000 7f16a8ff9640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f169806b4d0 0x7f169806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.436+0000 7f16abfff640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f169806b4d0 0x7f169806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.436+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f169c089520 con 0x7f16ac15bdd0 2024-03-06T07:22:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.436+0000 7f16abfff640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f169806b4d0 0x7f169806d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16a0005880 tx=0x7f16a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.440+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16ac0b54b0 con 0x7f16ac15bdd0 2024-03-06T07:22:09.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.617+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ac0c7e40 con 0x7f16ac15bdd0 2024-03-06T07:22:09.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.618+0000 7f16a8ff9640 1 -- 172.21.15.122:0/1584904328 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f16ac0c7e40 con 0x7f16ac15bdd0 2024-03-06T07:22:09.620 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:09.620 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f169806b4d0 msgr2=0x7f169806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f169806b4d0 0x7f169806d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16a0005880 tx=0x7f16a0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 msgr2=0x7f16ac152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 0x7f16ac152b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f169c00c3e0 tx=0x7f169c00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 shutdown_connections 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f169806b4d0 0x7f169806d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16ac15fa10 0x7f16ac15e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16ac15bdd0 0x7f16ac152b20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 --2- 172.21.15.122:0/1584904328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16ac0d16e0 0x7f16ac15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 >> 172.21.15.122:0/1584904328 conn(0x7f16ac09a9f0 msgr2=0x7f16ac09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 shutdown_connections 2024-03-06T07:22:09.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.620+0000 7f16b24df640 1 -- 172.21.15.122:0/1584904328 wait complete. 2024-03-06T07:22:09.633 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:09.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.730+0000 7f5aa2bd6640 1 Processor -- start 2024-03-06T07:22:09.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.730+0000 7f5aa2bd6640 1 -- start start 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa2bd6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5a9c0ce920 0x7f5a9c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa2bd6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c06b600 0x7f5a9c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa2bd6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9c15ea30 con 0x7f5a9c06b220 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa2bd6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9c161b70 con 0x7f5a9c0ce920 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa2bd6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c15ed10 con 0x7f5a9c06b600 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5a9c0ce920 0x7f5a9c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa0972640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5a9c06b220 0x7f5a9c071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38642/0 (socket says 172.21.15.122:38642) 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5a9c0ce920 0x7f5a9c15e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57248/0 (socket says 172.21.15.122:57248) 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa0972640 1 -- 172.21.15.122:0/3131032157 learned_addr learned my addr 172.21.15.122:0/3131032157 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5aa1173640 1 --2- 172.21.15.122:0/3131032157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c06b600 0x7f5a9c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9bfff640 1 -- 172.21.15.122:0/3131032157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c06b600 msgr2=0x7f5a9c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9bfff640 1 --2- 172.21.15.122:0/3131032157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c06b600 0x7f5a9c162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9bfff640 1 -- 172.21.15.122:0/3131032157 >> v1:172.21.15.122:6789/0 conn(0x7f5a9c06b220 legacy=0x7f5a9c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9b7fe640 1 -- 172.21.15.122:0/3131032157 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498753243 0 0) 0x7f5a9c15ea30 con 0x7f5a9c06b220 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.731+0000 7f5a9bfff640 1 -- 172.21.15.122:0/3131032157 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c15eff0 con 0x7f5a9c0ce920 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5a9bfff640 1 --2- 172.21.15.122:0/3131032157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c0ce920 0x7f5a9c15e450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5a880097a0 tx=0x7f5a8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc117a9a0ecdb0bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5a9b7fe640 1 -- 172.21.15.122:0/3131032157 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a880315a0 con 0x7f5a9c0ce920 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5a9b7fe640 1 -- 172.21.15.122:0/3131032157 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a88037070 con 0x7f5a9c0ce920 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5a9b7fe640 1 -- 172.21.15.122:0/3131032157 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88031960 con 0x7f5a9c0ce920 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/3131032157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c0ce920 msgr2=0x7f5a9c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/3131032157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c0ce920 0x7f5a9c15e450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5a880097a0 tx=0x7f5a8802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/3131032157 shutdown_connections 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/3131032157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c06b600 0x7f5a9c162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/3131032157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c0ce920 0x7f5a9c15e450 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/3131032157 >> 172.21.15.122:0/3131032157 conn(0x7f5a9c06c0c0 msgr2=0x7f5a9c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/3131032157 shutdown_connections 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.732+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/3131032157 wait complete. 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.733+0000 7f5aa2bd6640 1 Processor -- start 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 -- start start 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 0x7f5a9c0b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a9c06b600 0x7f5a9c1f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c0ce920 0x7f5a9c15c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c072090 con 0x7f5a9c06b220 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c0ce200 con 0x7f5a9c06b600 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa2bd6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c0b13d0 con 0x7f5a9c0ce920 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.734+0000 7f5aa0972640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 0x7f5a9c0b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 0x7f5a9c0b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57264/0 (socket says 172.21.15.122:57264) 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 -- 172.21.15.122:0/1654794150 learned_addr learned my addr 172.21.15.122:0/1654794150 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa1173640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c0ce920 0x7f5a9c15c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5a9bfff640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a9c06b600 0x7f5a9c1f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 -- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c0ce920 msgr2=0x7f5a9c15c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c0ce920 0x7f5a9c15c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 -- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a9c06b600 msgr2=0x7f5a9c1f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a9c06b600 0x7f5a9c1f5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 -- 172.21.15.122:0/1654794150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c1f6200 con 0x7f5a9c06b220 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa0972640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 0x7f5a9c0b0e90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000c400 tx=0x7f5a9000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a90017070 con 0x7f5a9c06b220 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a90005ce0 con 0x7f5a9c06b220 2024-03-06T07:22:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c1f64e0 con 0x7f5a9c06b220 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a90007310 con 0x7f5a9c06b220 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.735+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c1f6aa0 con 0x7f5a9c06b220 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.736+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c04f330 con 0x7f5a9c06b220 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.737+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a9001a070 con 0x7f5a9c06b220 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.737+0000 7f5a997fa640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5a8006b4d0 0x7f5a8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.737+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a90089500 con 0x7f5a9c06b220 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.737+0000 7f5a9bfff640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5a8006b4d0 0x7f5a8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.737+0000 7f5a9bfff640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5a8006b4d0 0x7f5a8006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a88036000 tx=0x7f5a8803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.741+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a90057dc0 con 0x7f5a9c06b220 2024-03-06T07:22:09.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.945+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a9c0cdf00 con 0x7f5a9c06b220 2024-03-06T07:22:09.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.946+0000 7f5a997fa640 1 -- 172.21.15.122:0/1654794150 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f5a9005be60 con 0x7f5a9c06b220 2024-03-06T07:22:09.948 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:09.948 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:09.948 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.947+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5a8006b4d0 msgr2=0x7f5a8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.947+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5a8006b4d0 0x7f5a8006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a88036000 tx=0x7f5a8803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.947+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 msgr2=0x7f5a9c0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.947+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 0x7f5a9c0b0e90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000c400 tx=0x7f5a9000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 shutdown_connections 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5a8006b4d0 0x7f5a8006d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5a9c0ce920 0x7f5a9c15c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a9c06b600 0x7f5a9c1f5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 --2- 172.21.15.122:0/1654794150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a9c06b220 0x7f5a9c0b0e90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 >> 172.21.15.122:0/1654794150 conn(0x7f5a9c06c0c0 msgr2=0x7f5a9c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 shutdown_connections 2024-03-06T07:22:09.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:09.948+0000 7f5aa2bd6640 1 -- 172.21.15.122:0/1654794150 wait complete. 2024-03-06T07:22:10.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:52.887640+0000', 'modified': '2024-03-06T07:21:55.266867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.169:6834/2014546388', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6832', 'nonce': 2014546388}, {'type': 'v1', 'addr': '172.21.15.169:6834', 'nonce': 2014546388}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4287, 'qdb_cluster': [4287]} 2024-03-06T07:22:10.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:10.363 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:22:10.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.464+0000 7fc5096df640 1 Processor -- start 2024-03-06T07:22:10.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.464+0000 7fc5096df640 1 -- start start 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5096df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5040b4460 0x7fc5040d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5096df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc504162970 0x7fc5040b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5096df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5040d22c0 con 0x7fc504161cf0 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5096df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc504161b00 con 0x7fc504162970 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5096df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5040cecc0 con 0x7fc5040b4460 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc504162970 0x7fc5040b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc504162970 0x7fc5040b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57266/0 (socket says 172.21.15.122:57266) 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 -- 172.21.15.122:0/3935140502 learned_addr learned my addr 172.21.15.122:0/3935140502 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5037fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc504161cf0 0x7fc5040cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38646/0 (socket says 172.21.15.122:38646) 2024-03-06T07:22:10.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc502ffd640 1 --2- 172.21.15.122:0/3935140502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5040b4460 0x7fc5040d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 -- 172.21.15.122:0/3935140502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5040b4460 msgr2=0x7fc5040d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 --2- 172.21.15.122:0/3935140502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5040b4460 0x7fc5040d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 -- 172.21.15.122:0/3935140502 >> v1:172.21.15.122:6789/0 conn(0x7fc504161cf0 legacy=0x7fc5040cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.465+0000 7fc5027fc640 1 -- 172.21.15.122:0/3935140502 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5040d25a0 con 0x7fc504162970 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.466+0000 7fc5027fc640 1 --2- 172.21.15.122:0/3935140502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504162970 0x7fc5040b3580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009870 tx=0x7fc4f802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75b09669622f465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.466+0000 7fc501ffb640 1 -- 172.21.15.122:0/3935140502 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8034070 con 0x7fc504162970 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.466+0000 7fc501ffb640 1 -- 172.21.15.122:0/3935140502 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8030840 con 0x7fc504162970 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.466+0000 7fc501ffb640 1 -- 172.21.15.122:0/3935140502 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8030c10 con 0x7fc504162970 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 -- 172.21.15.122:0/3935140502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504162970 msgr2=0x7fc5040b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 --2- 172.21.15.122:0/3935140502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504162970 0x7fc5040b3580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009870 tx=0x7fc4f802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 -- 172.21.15.122:0/3935140502 shutdown_connections 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 --2- 172.21.15.122:0/3935140502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504162970 0x7fc5040b3580 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 --2- 172.21.15.122:0/3935140502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5040b4460 0x7fc5040d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 -- 172.21.15.122:0/3935140502 >> 172.21.15.122:0/3935140502 conn(0x7fc5040d3ce0 msgr2=0x7fc5040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 -- 172.21.15.122:0/3935140502 shutdown_connections 2024-03-06T07:22:10.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.467+0000 7fc5096df640 1 -- 172.21.15.122:0/3935140502 wait complete. 2024-03-06T07:22:10.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.468+0000 7fc5096df640 1 Processor -- start 2024-03-06T07:22:10.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.468+0000 7fc5096df640 1 -- start start 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5040b4460 0x7fc5040b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 0x7fc504156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc504162970 0x7fc50406c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5040ce950 con 0x7fc504161cf0 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5040d2b60 con 0x7fc5040b4460 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5040b3ac0 con 0x7fc504162970 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 0x7fc504156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 0x7fc504156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57278/0 (socket says 172.21.15.122:57278) 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 -- 172.21.15.122:0/682814525 learned_addr learned my addr 172.21.15.122:0/682814525 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc502ffd640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5040b4460 0x7fc5040b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5037fe640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc504162970 0x7fc50406c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 -- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc504162970 msgr2=0x7fc50406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc504162970 0x7fc50406c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 -- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5040b4460 msgr2=0x7fc5040b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5040b4460 0x7fc5040b87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 -- 172.21.15.122:0/682814525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5040b6aa0 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5027fc640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 0x7fc504156130 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc4f802b9c0 tx=0x7fc4f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc4eb7fe640 1 -- 172.21.15.122:0/682814525 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8048070 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc4eb7fe640 1 -- 172.21.15.122:0/682814525 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8030640 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5040b6d80 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc4eb7fe640 1 -- 172.21.15.122:0/682814525 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8034040 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.469+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5040b7340 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.471+0000 7fc4eb7fe640 1 -- 172.21.15.122:0/682814525 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4f804b070 con 0x7fc504161cf0 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.471+0000 7fc4eb7fe640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f006b4d0 0x7fc4f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.471+0000 7fc4eb7fe640 1 -- 172.21.15.122:0/682814525 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4f80ac020 con 0x7fc504161cf0 2024-03-06T07:22:10.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.471+0000 7fc502ffd640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f006b4d0 0x7fc4f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.472+0000 7fc502ffd640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f006b4d0 0x7fc4f006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009aa0 tx=0x7fc4f4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.472+0000 7fc50c3a4740 1 Processor -- start 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.472+0000 7fc50c3a4740 1 -- start start 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062a0870 0x55fe062a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 0x55fe062a6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0620a6b0 con 0x55fe062a48c0 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe061b3a60 con 0x55fe062d0600 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fe06206750 con 0x55fe062a0870 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 0x55fe062a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 0x55fe062a6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57294/0 (socket says 172.21.15.122:57294) 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 -- 172.21.15.122:0/781928957 learned_addr learned my addr 172.21.15.122:0/781928957 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5037fe640 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc502ffd640 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062a0870 0x55fe062a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 -- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062a0870 msgr2=0x55fe062a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062a0870 0x55fe062a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 -- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 msgr2=0x55fe062d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062d0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc5027fc640 1 -- 172.21.15.122:0/781928957 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe061d1f30 con 0x55fe062a48c0 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.473+0000 7fc502ffd640 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062a0870 0x55fe062a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc5027fc640 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 0x55fe062a6360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80aa570 tx=0x7fc4f8031f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67330937f27a044a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc4e8ff9640 1 -- 172.21.15.122:0/781928957 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f807ea10 con 0x55fe062a48c0 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc4e8ff9640 1 -- 172.21.15.122:0/781928957 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8082fd0 con 0x55fe062a48c0 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc4e8ff9640 1 -- 172.21.15.122:0/781928957 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8076fe0 con 0x55fe062a48c0 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 -- 172.21.15.122:0/781928957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 msgr2=0x55fe062a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 0x55fe062a6360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80aa570 tx=0x7fc4f8031f60 comp rx=0 tx=0).stop 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 -- 172.21.15.122:0/781928957 shutdown_connections 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062d0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062a48c0 0x55fe062a6360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/781928957 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062a0870 0x55fe062a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 -- 172.21.15.122:0/781928957 >> 172.21.15.122:0/781928957 conn(0x55fe062a28f0 msgr2=0x55fe062a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 -- 172.21.15.122:0/781928957 shutdown_connections 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.474+0000 7fc50c3a4740 1 -- 172.21.15.122:0/781928957 wait complete. 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.475+0000 7fc50c3a4740 1 -- start start 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.475+0000 7fc50c3a4740 1 Processor -- start 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 0x55fe062ec170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe062cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0620a6b0 con 0x55fe062d0600 2024-03-06T07:22:10.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe061c6ea0 con 0x55fe062c43d0 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fe062a4d50 con 0x55fe062d2130 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc50c3a4740 10 client.0 _ll_register_callbacks cb 0x55fe061c6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 0x55fe062ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 0x55fe062ec170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57300/0 (socket says 172.21.15.122:57300) 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 -- 172.21.15.122:0/3897097706 learned_addr learned my addr 172.21.15.122:0/3897097706 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc5037fe640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc5027fc640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe062cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 -- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 msgr2=0x55fe062cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe062cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 -- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 msgr2=0x55fe062c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 -- 172.21.15.122:0/3897097706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe061c7270 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc502ffd640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 0x55fe062ec170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4012000 tx=0x7fc4f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb22546cc9234db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc4e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/3897097706 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4027020 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.476+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/3897097706 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4f4003da0 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4f8082fd0 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/3897097706 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4f4031050 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 10 client.4674 fetch_fsmap learned FSMap version 7 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55fe05f1f2d0 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/3897097706 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8082fd0 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/3897097706 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc4f4034020 con 0x55fe062d0600 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 10 client.4674 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 10 client.4674 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 20 client.4674 populate_metadata read hostname 'smithi122' 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe062ff850 0x55fe0630a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 4 client.4674 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 --> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe06217bf0 con 0x55fe062ff850 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.477+0000 7fc5027fc640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe062ff850 0x55fe0630a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc4e1ffb640 10 client.4674 ms_handle_connect on v2:172.21.15.169:6832/2014546388 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc5027fc640 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe062ff850 0x55fe0630a410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9b2b77b419059ef server_cookie=7fc1b7cb8f0e5043 in_seq=0 out_seq=0 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/3897097706 <== mds.0 v2:172.21.15.169:6832/2014546388 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe06217bf0 con 0x55fe062ff850 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc4e1ffb640 10 client.4674 handle_command_reply: tid=0 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 client.4674 shutdown 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 msgr2=0x55fe062ec170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 0x55fe062ec170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4012000 tx=0x7fc4f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 shutdown_connections 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe062ff850 0x55fe0630a410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe062cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc4e1ffb640 0 client.4674 ms_handle_reset on v2:172.21.15.169:6832/2014546388 2024-03-06T07:22:10.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/3897097706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d0600 0x55fe062ec170 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 >> 172.21.15.122:0/3897097706 conn(0x55fe062a0870 msgr2=0x55fe0629f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 shutdown_connections 2024-03-06T07:22:10.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.478+0000 7fc50c3a4740 1 -- 172.21.15.122:0/3897097706 wait complete. 2024-03-06T07:22:10.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.479+0000 7fc50c3a4740 20 client.4674 trim_cache size 0 max 16384 2024-03-06T07:22:10.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.495+0000 7fc50c3a4740 1 Processor -- start 2024-03-06T07:22:10.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.495+0000 7fc50c3a4740 1 -- start start 2024-03-06T07:22:10.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062fea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d0600 0x55fe062cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 0x55fe062d1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc504200300 con 0x55fe062d2130 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041ff610 con 0x55fe062c43d0 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc504157ae0 con 0x55fe062d0600 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 0x55fe062d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 0x55fe062d1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57310/0 (socket says 172.21.15.122:57310) 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 -- 172.21.15.122:0/1075543378 learned_addr learned my addr 172.21.15.122:0/1075543378 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5027fc640 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d0600 0x55fe062cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc502ffd640 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062fea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 -- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d0600 msgr2=0x55fe062cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d0600 0x55fe062cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 -- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 msgr2=0x55fe062fea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062fea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 -- 172.21.15.122:0/1075543378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe062a72a0 con 0x55fe062d2130 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc502ffd640 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062fea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc5037fe640 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 0x55fe062d1600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc001bb0 tx=0x7fc4fc001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5085d843c693e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc4e3fff640 1 -- 172.21.15.122:0/1075543378 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0079b0 con 0x55fe062d2130 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc4e3fff640 1 -- 172.21.15.122:0/1075543378 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc016420 con 0x55fe062d2130 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.496+0000 7fc4e3fff640 1 -- 172.21.15.122:0/1075543378 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0167d0 con 0x55fe062d2130 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 -- 172.21.15.122:0/1075543378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 msgr2=0x55fe062d1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 0x55fe062d1600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc001bb0 tx=0x7fc4fc001650 comp rx=0 tx=0).stop 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 -- 172.21.15.122:0/1075543378 shutdown_connections 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062d2130 0x55fe062d1600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d0600 0x55fe062cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/1075543378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062c43d0 0x55fe062fea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 -- 172.21.15.122:0/1075543378 >> 172.21.15.122:0/1075543378 conn(0x55fe062a0870 msgr2=0x55fe062d3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 -- 172.21.15.122:0/1075543378 shutdown_connections 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 -- 172.21.15.122:0/1075543378 wait complete. 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 -- start start 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.497+0000 7fc50c3a4740 1 Processor -- start 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 0x55fe0630ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062c8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe0629f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe062d2810 con 0x55fe062c43d0 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041580e0 con 0x55fe062d0600 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fe062a59a0 con 0x55fe062d2130 2024-03-06T07:22:10.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc50c3a4740 10 client.0 _ll_register_callbacks cb 0x7fc4f4011c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 0x55fe0630ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 0x55fe0630ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57316/0 (socket says 172.21.15.122:57316) 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 -- 172.21.15.122:0/193165943 learned_addr learned my addr 172.21.15.122:0/193165943 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc5037fe640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe0629f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc5027fc640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 -- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 msgr2=0x55fe0629f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe0629f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 -- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 msgr2=0x55fe062c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.498+0000 7fc502ffd640 1 -- 172.21.15.122:0/193165943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe05fe96b0 con 0x55fe062c43d0 2024-03-06T07:22:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc502ffd640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 0x55fe0630ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009520 tx=0x7fc4f4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffde02ed11b039bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc4e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/193165943 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4027050 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/193165943 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4f403fcf0 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4fc016420 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/193165943 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4f40039e0 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 10 client.4680 fetch_fsmap learned FSMap version 7 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55fe062fe7f0 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/193165943 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc016420 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/193165943 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc4f4046050 con 0x55fe062c43d0 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 10 client.4680 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 10 client.4680 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 20 client.4680 populate_metadata read hostname 'smithi122' 2024-03-06T07:22:10.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe0629fa50 0x55fe062cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 4 client.4680 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 --> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fe062d38e0 con 0x55fe0629fa50 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.499+0000 7fc5027fc640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe0629fa50 0x55fe062cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc4e1ffb640 10 client.4680 ms_handle_connect on v2:172.21.15.169:6832/2014546388 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc5027fc640 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe0629fa50 0x55fe062cc190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316af969868dcafc server_cookie=27d9a95e6696fcfe in_seq=0 out_seq=0 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc4e1ffb640 1 -- 172.21.15.122:0/193165943 <== mds.0 v2:172.21.15.169:6832/2014546388 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55fe062d38e0 con 0x55fe0629fa50 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc4e1ffb640 10 client.4680 handle_command_reply: tid=0 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 client.4680 shutdown 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 msgr2=0x55fe0630ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 0x55fe0630ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009520 tx=0x7fc4f4019000 comp rx=0 tx=0).stop 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 shutdown_connections 2024-03-06T07:22:10.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:6832/2014546388,v1:172.21.15.169:6834/2014546388] conn(0x55fe0629fa50 0x55fe062cc190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55fe062d2130 0x55fe0629f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc4e1ffb640 0 client.4680 ms_handle_reset on v2:172.21.15.169:6832/2014546388 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe062d0600 0x55fe062c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 --2- 172.21.15.122:0/193165943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fe062c43d0 0x55fe0630ede0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 >> 172.21.15.122:0/193165943 conn(0x55fe062a0870 msgr2=0x55fe062d3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 shutdown_connections 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 1 -- 172.21.15.122:0/193165943 wait complete. 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc50c3a4740 20 client.4680 trim_cache size 0 max 16384 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:22:10.510 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4287, 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 16.953203668, 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 7, 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 20, 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 20, 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 23.541710548000001 2024-03-06T07:22:10.511 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f006b4d0 msgr2=0x7fc4f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc5096df640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f006b4d0 0x7fc4f006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009aa0 tx=0x7fc4f4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.500+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 msgr2=0x7fc504156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 0x7fc504156130 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc4f802b9c0 tx=0x7fc4f8004730 comp rx=0 tx=0).stop 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 shutdown_connections 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f006b4d0 0x7fc4f006d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc504162970 0x7fc50406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc504161cf0 0x7fc504156130 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 --2- 172.21.15.122:0/682814525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5040b4460 0x7fc5040b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 >> 172.21.15.122:0/682814525 conn(0x7fc5040d3ce0 msgr2=0x7fc50415ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 shutdown_connections 2024-03-06T07:22:10.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.501+0000 7fc5096df640 1 -- 172.21.15.122:0/682814525 wait complete. 2024-03-06T07:22:10.514 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-06T07:22:10.538 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-06T07:22:10.538 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:22:10.538 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-06T07:22:10.538 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:10.538 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:10.539 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-06T07:22:10.539 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:10.539 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:10.539 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-06T07:22:10.539 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:10.539 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:10.539 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-06T07:22:10.539 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:10.539 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:10.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.539 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-06T07:22:10.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link delete ceph-brx 2024-03-06T07:22:10.624 INFO:teuthology.orchestra.run.smithi169.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:10.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.625 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:22:10.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link delete ceph-brx 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi122.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:10.710 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-06T07:22:10.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:10.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 Processor -- start 2024-03-06T07:22:10.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 -- start start 2024-03-06T07:22:10.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40cf790 0x7f9ce40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ce40723f0 0x7f9ce406d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce40ae240 con 0x7f9ce40adda0 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce406cf70 con 0x7f9ce40723f0 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.849+0000 7f9ceac95640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40ae520 con 0x7f9ce40cf790 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ce40723f0 0x7f9ce406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ce40723f0 0x7f9ce406d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57322/0 (socket says 172.21.15.122:57322) 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 -- 172.21.15.122:0/2958018687 learned_addr learned my addr 172.21.15.122:0/2958018687 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce8a31640 1 --2- 172.21.15.122:0/2958018687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40cf790 0x7f9ce40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 -- 172.21.15.122:0/2958018687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40cf790 msgr2=0x7f9ce40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 --2- 172.21.15.122:0/2958018687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40cf790 0x7f9ce40ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 -- 172.21.15.122:0/2958018687 >> v1:172.21.15.122:6789/0 conn(0x7f9ce40adda0 legacy=0x7f9ce4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce9232640 1 -- 172.21.15.122:0/2958018687 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40ae800 con 0x7f9ce40723f0 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.850+0000 7f9ce37fe640 1 -- 172.21.15.122:0/2958018687 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343227896 0 0) 0x7f9ce40ae240 con 0x7f9ce40adda0 2024-03-06T07:22:10.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ce9232640 1 --2- 172.21.15.122:0/2958018687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce406d400 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009e50 tx=0x7f9cdc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fcce2ec2986f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ce37fe640 1 -- 172.21.15.122:0/2958018687 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc02cb00 con 0x7f9ce40723f0 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ce37fe640 1 -- 172.21.15.122:0/2958018687 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc02cca0 con 0x7f9ce40723f0 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ce37fe640 1 -- 172.21.15.122:0/2958018687 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc034610 con 0x7f9ce40723f0 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 -- 172.21.15.122:0/2958018687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 msgr2=0x7f9ce406d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 --2- 172.21.15.122:0/2958018687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce406d400 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009e50 tx=0x7f9cdc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 -- 172.21.15.122:0/2958018687 shutdown_connections 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 --2- 172.21.15.122:0/2958018687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce406d400 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 --2- 172.21.15.122:0/2958018687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40cf790 0x7f9ce40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 -- 172.21.15.122:0/2958018687 >> 172.21.15.122:0/2958018687 conn(0x7f9ce40b1a90 msgr2=0x7f9ce40b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 -- 172.21.15.122:0/2958018687 shutdown_connections 2024-03-06T07:22:10.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.851+0000 7f9ceac95640 1 -- 172.21.15.122:0/2958018687 wait complete. 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.852+0000 7f9ceac95640 1 Processor -- start 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.852+0000 7f9ceac95640 1 -- start start 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ceac95640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce41519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ceac95640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40adda0 0x7f9ce4153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ceac95640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ce40cf790 0x7f9ce41542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ceac95640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce415e700 con 0x7f9ce40723f0 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ceac95640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce415e580 con 0x7f9ce40cf790 2024-03-06T07:22:10.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ceac95640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce406d940 con 0x7f9ce40adda0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ce8a31640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce41519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ce8a31640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce41519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57332/0 (socket says 172.21.15.122:57332) 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ce8a31640 1 -- 172.21.15.122:0/1680938923 learned_addr learned my addr 172.21.15.122:0/1680938923 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ce3fff640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40adda0 0x7f9ce4153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.853+0000 7f9ce9232640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ce40cf790 0x7f9ce41542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce8a31640 1 -- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40adda0 msgr2=0x7f9ce4153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce8a31640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40adda0 0x7f9ce4153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce8a31640 1 -- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ce40cf790 msgr2=0x7f9ce41542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce8a31640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ce40cf790 0x7f9ce41542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce8a31640 1 -- 172.21.15.122:0/1680938923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4154a60 con 0x7f9ce40723f0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce8a31640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce41519e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800e8a0 tx=0x7f9cd800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8017070 con 0x7f9ce40723f0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd800c940 con 0x7f9ce40723f0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4154d40 con 0x7f9ce40723f0 2024-03-06T07:22:10.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd800cd30 con 0x7f9ce40723f0 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.854+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4155d50 con 0x7f9ce40723f0 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.855+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce404f330 con 0x7f9ce40723f0 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.855+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cd801a070 con 0x7f9ce40723f0 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.856+0000 7f9ce17fa640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cc806b4d0 0x7f9cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.856+0000 7f9ce3fff640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cc806b4d0 0x7f9cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.856+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cd808a2f0 con 0x7f9ce40723f0 2024-03-06T07:22:10.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.856+0000 7f9ce3fff640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cc806b4d0 0x7f9cc806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9ce41f4970 tx=0x7f9cd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:10.860+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cd8058bb0 con 0x7f9ce40723f0 2024-03-06T07:22:11.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.068+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9ce4072dd0 con 0x7f9ce40723f0 2024-03-06T07:22:11.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.068+0000 7f9ce17fa640 1 -- 172.21.15.122:0/1680938923 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9cd805cc50 con 0x7f9ce40723f0 2024-03-06T07:22:11.070 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:11.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cc806b4d0 msgr2=0x7f9cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cc806b4d0 0x7f9cc806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9ce41f4970 tx=0x7f9cd0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 msgr2=0x7f9ce41519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce41519e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800e8a0 tx=0x7f9cd800ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 shutdown_connections 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cc806b4d0 0x7f9cc806d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ce40cf790 0x7f9ce41542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ce40adda0 0x7f9ce4153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 --2- 172.21.15.122:0/1680938923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce40723f0 0x7f9ce41519e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 >> 172.21.15.122:0/1680938923 conn(0x7f9ce40b1a90 msgr2=0x7f9ce4070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 shutdown_connections 2024-03-06T07:22:11.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:11.070+0000 7f9ceac95640 1 -- 172.21.15.122:0/1680938923 wait complete. 2024-03-06T07:22:11.082 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:11.083 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:11.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.083 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:22:11.098 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:22:11.099 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42710sec preferred_lft 42710sec 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:22:11.100 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:11.100 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:11.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set ceph-brx up 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:11.101 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:11.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.267 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:11.293 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:22:11.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.295 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:22:11.368 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.369 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:11.369 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:11.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:11.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:11.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:11.370 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:11.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.640 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:22:11.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.655 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list-id 2024-03-06T07:22:11.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.710 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:11.710 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:11.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:11.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:22:11.710 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:11.792 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:22:11.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.792 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:11.792 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:11.792 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:22:11.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:11.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:22:11.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:22:11.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:22:11.793 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:11.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.934 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:11.934 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:11.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:22:11.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:11.934 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:11.975 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:11.976 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:11.976 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:12.030 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:12.030 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:12.031 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:12.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:22:12.153 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:22:12.210 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:12.211 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:12.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.213 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:22:12.264 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:12.265 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:12.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:22:12.333 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:12.388 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:22:12.388 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:22:12.388 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:22:12.388 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:22:12.389 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:22:12.390 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:12.391 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:12.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.391 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:22:12.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:22:12.391+0000 7f3741c41480 -1 init, newargv = 0x55aa3ca42650 newargc=15 2024-03-06T07:22:12.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[47011]: starting ceph client 2024-03-06T07:22:12.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[47011]: starting fuse 2024-03-06T07:22:12.407 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:22:12.407 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:22:12.407 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:22:12.408 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:22:12.409 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:12.409 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:12.409 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:12.409 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:22:12.526 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.47011.asok 2024-03-06T07:22:12.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:12.536 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47011.asok 2024-03-06T07:22:12.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47011.asok status 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:22:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "47011", 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4695, 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 4695 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:22:12.674 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 2237631044 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 2237631044 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.4695 192.168.0.1:0/2237631044", 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/2237631044", 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 7, 2024-03-06T07:22:12.675 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 20, 2024-03-06T07:22:12.676 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:12.676 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:22:12.676 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:22:12.676 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:22:12.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:12.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.779+0000 7faa79bae640 1 Processor -- start 2024-03-06T07:22:12.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.779+0000 7faa79bae640 1 -- start start 2024-03-06T07:22:12.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa79bae640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa740b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa79bae640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7415f580 con 0x7faa7415ef60 2024-03-06T07:22:12.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa79bae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7415b450 con 0x7faa7415bf30 2024-03-06T07:22:12.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa79bae640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa740cf3f0 con 0x7faa7413dc10 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa73fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faa7415ef60 0x7faa740708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38668/0 (socket says 172.21.15.122:38668) 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa737fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa740b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa73fff640 1 -- 172.21.15.122:0/3679459825 learned_addr learned my addr 172.21.15.122:0/3679459825 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa727fc640 1 -- 172.21.15.122:0/3679459825 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695284049 0 0) 0x7faa7415f580 con 0x7faa7415ef60 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa727fc640 1 -- 172.21.15.122:0/3679459825 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c003520 con 0x7faa7415ef60 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa727fc640 1 -- 172.21.15.122:0/3679459825 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820254760 0 0) 0x7faa7415b450 con 0x7faa7415bf30 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa737fe640 1 -- 172.21.15.122:0/3679459825 >> v1:172.21.15.169:6789/0 conn(0x7faa7415bf30 legacy=0x7faa7415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa737fe640 1 -- 172.21.15.122:0/3679459825 >> v1:172.21.15.122:6789/0 conn(0x7faa7415ef60 legacy=0x7faa740708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.780+0000 7faa737fe640 1 -- 172.21.15.122:0/3679459825 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7415b730 con 0x7faa7413dc10 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.781+0000 7faa737fe640 1 --2- 172.21.15.122:0/3679459825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa740b2950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa640097d0 tx=0x7faa6402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f791f50940bef8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.781+0000 7faa727fc640 1 -- 172.21.15.122:0/3679459825 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64035020 con 0x7faa7413dc10 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.781+0000 7faa727fc640 1 -- 172.21.15.122:0/3679459825 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64031800 con 0x7faa7413dc10 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.781+0000 7faa727fc640 1 -- 172.21.15.122:0/3679459825 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64031bb0 con 0x7faa7413dc10 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 -- 172.21.15.122:0/3679459825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 msgr2=0x7faa740b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 --2- 172.21.15.122:0/3679459825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa740b2950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa640097d0 tx=0x7faa6402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 -- 172.21.15.122:0/3679459825 shutdown_connections 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 --2- 172.21.15.122:0/3679459825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa740b2950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 -- 172.21.15.122:0/3679459825 >> 172.21.15.122:0/3679459825 conn(0x7faa740cf8a0 msgr2=0x7faa740cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 -- 172.21.15.122:0/3679459825 shutdown_connections 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.782+0000 7faa79bae640 1 -- 172.21.15.122:0/3679459825 wait complete. 2024-03-06T07:22:12.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.783+0000 7faa79bae640 1 Processor -- start 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.783+0000 7faa79bae640 1 -- start start 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa79bae640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa741f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa79bae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa7415bf30 0x7faa741f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa79bae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 0x7faa740cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa79bae640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740b2e90 con 0x7faa7415ef60 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa79bae640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7415b210 con 0x7faa7415bf30 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa79bae640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa740b3010 con 0x7faa7413dc10 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 0x7faa740cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 0x7faa740cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57334/0 (socket says 172.21.15.122:57334) 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 -- 172.21.15.122:0/2154451524 learned_addr learned my addr 172.21.15.122:0/2154451524 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa737fe640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa741f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa72ffd640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa7415bf30 0x7faa741f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 -- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 msgr2=0x7faa741f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa741f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 -- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa7415bf30 msgr2=0x7faa741f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa7415bf30 0x7faa741f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 -- 172.21.15.122:0/2154451524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa741f5440 con 0x7faa7415ef60 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.784+0000 7faa73fff640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 0x7faa740cc940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00caa0 tx=0x7faa6c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.785+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c017070 con 0x7faa7415ef60 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.785+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c00ac00 con 0x7faa7415ef60 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.785+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa741f5720 con 0x7faa7415ef60 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.785+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c012400 con 0x7faa7415ef60 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.785+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa741f2a40 con 0x7faa7415ef60 2024-03-06T07:22:12.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.786+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa741f5c40 con 0x7faa7415ef60 2024-03-06T07:22:12.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.786+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa6c01a070 con 0x7faa7415ef60 2024-03-06T07:22:12.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.786+0000 7faa5bfff640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa6006b4d0 0x7faa6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.787+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa6c089ea0 con 0x7faa7415ef60 2024-03-06T07:22:12.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.787+0000 7faa737fe640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa6006b4d0 0x7faa6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.787+0000 7faa737fe640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa6006b4d0 0x7faa6006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa64036000 tx=0x7faa64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.791+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa741f5c40 con 0x7faa7415ef60 2024-03-06T07:22:13.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.998+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faa7404f330 con 0x7faa7415ef60 2024-03-06T07:22:13.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:12.998+0000 7faa5bfff640 1 -- 172.21.15.122:0/2154451524 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7faa6c058760 con 0x7faa7415ef60 2024-03-06T07:22:13.000 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:13.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa6006b4d0 msgr2=0x7faa6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa6006b4d0 0x7faa6006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa64036000 tx=0x7faa64008030 comp rx=0 tx=0).stop 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 msgr2=0x7faa740cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 0x7faa740cc940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00caa0 tx=0x7faa6c00cf60 comp rx=0 tx=0).stop 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 shutdown_connections 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa6006b4d0 0x7faa6006d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa7415ef60 0x7faa740cc940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa7415bf30 0x7faa741f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 --2- 172.21.15.122:0/2154451524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa7413dc10 0x7faa741f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.000+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 >> 172.21.15.122:0/2154451524 conn(0x7faa740cf8a0 msgr2=0x7faa740707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.001+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 shutdown_connections 2024-03-06T07:22:13.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:13.001+0000 7faa79bae640 1 -- 172.21.15.122:0/2154451524 wait complete. 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:13.013 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:13.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.014 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:22:13.029 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42708sec preferred_lft 42708sec 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:13.030 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:13.031 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link tentative 2024-03-06T07:22:13.031 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.031 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:13.094 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:22:13.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.096 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:22:13.151 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:22:13.152 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:13.152 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:13.152 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:13.152 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:13.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.152 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:13.152 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:13.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:13.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:13.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:13.153 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:13.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.248 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:22:13.263 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:13.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.263 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list-id 2024-03-06T07:22:13.318 INFO:teuthology.orchestra.run.smithi122.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:22:13.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.318 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:13.318 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:13.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:13.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:22:13.318 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:13.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.400 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.466 INFO:teuthology.orchestra.run.smithi122.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:13.467 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether da:40:ce:3c:18:53 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:13.467 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:13.467 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.467 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::d840:ceff:fe3c:1853/64 scope link 2024-03-06T07:22:13.467 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.468 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:13.534 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:13.534 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:13.534 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:13.534 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.534 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:13.534 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.535 INFO:teuthology.orchestra.run.smithi122.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:13.535 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether da:40:ce:3c:18:53 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:13.535 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:13.535 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.535 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::d840:ceff:fe3c:1853/64 scope link 2024-03-06T07:22:13.535 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:13.536 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:22:13.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.536 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:13.536 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:13.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:22:13.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:13.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:22:13.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:22:13.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:22:13.537 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:13.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.662 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:22:13.662 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:22:13.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.1 up 2024-03-06T07:22:13.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:13.663 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:22:13.704 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-06T07:22:13.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:13.704 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:13.759 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:13.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:13.760 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:13.814 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:22:13.881 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:13.936 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:22:13.936 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:22:13.936 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:22:13.937 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:13.938 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:22:13.939 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:13.939 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:13.939 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:13.939 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:13.939 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:13.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.939 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:22:13.991 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:22:13.992 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-06T07:22:13.992 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-06T07:22:14.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:22:14.060 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:14.116 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:14.119 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:14.120 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:14.120 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:14.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.120 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:22:14.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi122.stderr:2024-03-06T07:22:14.117+0000 7fb368e35480 -1 init, newargv = 0x562cf0124500 newargc=15 2024-03-06T07:22:14.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi122.stderr:ceph-fuse[47505]: starting ceph client 2024-03-06T07:22:14.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi122.stderr:ceph-fuse[47505]: starting fuse 2024-03-06T07:22:14.135 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:22:14.136 INFO:teuthology.orchestra.run.smithi122.stdout:44 2024-03-06T07:22:14.136 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-06T07:22:14.136 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:14.136 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:22:14.137 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:22:14.137 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:22:14.137 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:22:14.137 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:14.137 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:22:14.137 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-06T07:22:14.138 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:22:14.138 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-06T07:22:14.138 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:14.138 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:22:14.138 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:22:14.138 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:22:14.139 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:22:14.139 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:14.139 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:22:14.139 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:14.139 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:14.139 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:14.140 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:22:14.140 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:22:14.140 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:22:14.140 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:14.140 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:22:14.140 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.2")) 2024-03-06T07:22:14.141 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:22:14.255 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.2.47505.asok 2024-03-06T07:22:14.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:14.266 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47505.asok 2024-03-06T07:22:14.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47505.asok status 2024-03-06T07:22:14.402 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "2", 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "47505", 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4710, 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 4710 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1663298044 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:22:14.404 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1663298044 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.4710 192.168.0.2:0/1663298044", 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.2:0/1663298044", 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 7, 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 20, 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:22:14.405 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:22:14.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:14.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.509+0000 7f510fc0e640 1 Processor -- start 2024-03-06T07:22:14.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.509+0000 7f510fc0e640 1 -- start start 2024-03-06T07:22:14.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510fc0e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f510813dc10 0x7f51080b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510fc0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f51080b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510fc0e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510815f580 con 0x7f510815e850 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510fc0e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51080cf3f0 con 0x7f510813dc10 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510fc0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510815b210 con 0x7f510815bf30 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510d9aa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f510813dc10 0x7f51080b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510e1ab640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f510815e850 0x7f510815eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43974/0 (socket says 172.21.15.122:43974) 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510e1ab640 1 -- 172.21.15.122:0/3258815884 learned_addr learned my addr 172.21.15.122:0/3258815884 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510d1a9640 1 --2- 172.21.15.122:0/3258815884 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f51080b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510d9aa640 1 -- 172.21.15.122:0/3258815884 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 msgr2=0x7f51080b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510d9aa640 1 --2- 172.21.15.122:0/3258815884 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f51080b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510d9aa640 1 -- 172.21.15.122:0/3258815884 >> v1:172.21.15.122:6789/0 conn(0x7f510815e850 legacy=0x7f510815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510d9aa640 1 -- 172.21.15.122:0/3258815884 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510815b450 con 0x7f510813dc10 2024-03-06T07:22:14.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.510+0000 7f510c9a8640 1 -- 172.21.15.122:0/3258815884 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549392685 0 0) 0x7f510815f580 con 0x7f510815e850 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.511+0000 7f510d9aa640 1 --2- 172.21.15.122:0/3258815884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510813dc10 0x7f51080b2950 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f510400ba40 tx=0x7f510402dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643fb7d5157ed1a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.511+0000 7f510c9a8640 1 -- 172.21.15.122:0/3258815884 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510403b070 con 0x7f510813dc10 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.511+0000 7f510c9a8640 1 -- 172.21.15.122:0/3258815884 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510402eda0 con 0x7f510813dc10 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.511+0000 7f510c9a8640 1 -- 172.21.15.122:0/3258815884 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104036660 con 0x7f510813dc10 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 -- 172.21.15.122:0/3258815884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510813dc10 msgr2=0x7f51080b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 --2- 172.21.15.122:0/3258815884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510813dc10 0x7f51080b2950 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f510400ba40 tx=0x7f510402dff0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 -- 172.21.15.122:0/3258815884 shutdown_connections 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 --2- 172.21.15.122:0/3258815884 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f51080b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 --2- 172.21.15.122:0/3258815884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510813dc10 0x7f51080b2950 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 -- 172.21.15.122:0/3258815884 >> 172.21.15.122:0/3258815884 conn(0x7f51080cf8a0 msgr2=0x7f51080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 -- 172.21.15.122:0/3258815884 shutdown_connections 2024-03-06T07:22:14.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.512+0000 7f510fc0e640 1 -- 172.21.15.122:0/3258815884 wait complete. 2024-03-06T07:22:14.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.513+0000 7f510fc0e640 1 Processor -- start 2024-03-06T07:22:14.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.513+0000 7f510fc0e640 1 -- start start 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.513+0000 7f510fc0e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f510813dc10 0x7f510813c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f510806de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 0x7f51080b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51080cd7a0 con 0x7f510815e850 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510813dff0 con 0x7f510813dc10 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51080cd920 con 0x7f510815bf30 2024-03-06T07:22:14.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 0x7f51080b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 0x7f51080b0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49714/0 (socket says 172.21.15.122:49714) 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510d9aa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f510813dc10 0x7f510813c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 -- 172.21.15.122:0/4159349763 learned_addr learned my addr 172.21.15.122:0/4159349763 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510d1a9640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f510806de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 -- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 msgr2=0x7f510806de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f510806de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 -- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f510813dc10 msgr2=0x7f510813c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f510813dc10 0x7f510813c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 -- 172.21.15.122:0/4159349763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510814f370 con 0x7f510815e850 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510e1ab640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 0x7f51080b0740 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f510000c700 tx=0x7f510000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5100018070 con 0x7f510815e850 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510000ce10 con 0x7f510815e850 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510814f650 con 0x7f510815e850 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510000a880 con 0x7f510815e850 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.514+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510814fc10 con 0x7f510815e850 2024-03-06T07:22:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.515+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510804f330 con 0x7f510815e850 2024-03-06T07:22:14.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.516+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f510001b070 con 0x7f510815e850 2024-03-06T07:22:14.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.516+0000 7f50fe7fc640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50ec06b4d0 0x7f50ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.516+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51000895d0 con 0x7f510815e850 2024-03-06T07:22:14.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.516+0000 7f510d9aa640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50ec06b4d0 0x7f50ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.517+0000 7f510d9aa640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50ec06b4d0 0x7f50ec06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5104038000 tx=0x7f5104032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.521+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5100057e90 con 0x7f510815e850 2024-03-06T07:22:14.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.730+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5108159440 con 0x7f510815e850 2024-03-06T07:22:14.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.730+0000 7f50fe7fc640 1 -- 172.21.15.122:0/4159349763 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f510005bf30 con 0x7f510815e850 2024-03-06T07:22:14.732 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:14.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50ec06b4d0 msgr2=0x7f50ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50ec06b4d0 0x7f50ec06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5104038000 tx=0x7f5104032000 comp rx=0 tx=0).stop 2024-03-06T07:22:14.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 msgr2=0x7f51080b0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 0x7f51080b0740 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f510000c700 tx=0x7f510000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 shutdown_connections 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50ec06b4d0 0x7f50ec06d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f510815e850 0x7f51080b0740 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f510815bf30 0x7f510806de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 --2- 172.21.15.122:0/4159349763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f510813dc10 0x7f510813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 >> 172.21.15.122:0/4159349763 conn(0x7f51080cf8a0 msgr2=0x7f510815a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 shutdown_connections 2024-03-06T07:22:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:14.732+0000 7f510fc0e640 1 -- 172.21.15.122:0/4159349763 wait complete. 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi169.front.sepia.ceph.com 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:14.745 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:14.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.746 DEBUG:teuthology.orchestra.run.smithi169:> ip addr 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f0 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f1 2024-03-06T07:22:14.761 INFO:teuthology.orchestra.run.smithi169.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout: inet 172.21.15.169/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft 42672sec preferred_lft 42672sec 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.762 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:14.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.762 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set ceph-brx up 2024-03-06T07:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:14.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.936 DEBUG:teuthology.orchestra.run.smithi169:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:14.963 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-06T07:22:14.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.965 DEBUG:teuthology.orchestra.run.smithi169:> route 2024-03-06T07:22:15.038 INFO:teuthology.orchestra.run.smithi169.stdout:Kernel IP routing table 2024-03-06T07:22:15.038 INFO:teuthology.orchestra.run.smithi169.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:15.038 INFO:teuthology.orchestra.run.smithi169.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:15.038 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:15.038 INFO:teuthology.orchestra.run.smithi169.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:15.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.039 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:15.039 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:15.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:15.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:15.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:15.039 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:15.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.332 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.347 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list-id 2024-03-06T07:22:15.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.401 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:15.401 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:15.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:22:15.401 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:15.484 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:22:15.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:22:15.484 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:15.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.629 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:15.629 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:15.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set brx.0 up 2024-03-06T07:22:15.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:15.629 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:15.672 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-06T07:22:15.673 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:15.673 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.726 INFO:teuthology.orchestra.run.smithi169.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:15.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:15.727 DEBUG:teuthology.orchestra.run.smithi169:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo modprobe fuse 2024-03-06T07:22:15.849 DEBUG:teuthology.orchestra.run.smithi169:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:/proc 2024-03-06T07:22:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:/sys 2024-03-06T07:22:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:/dev 2024-03-06T07:22:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/security 2024-03-06T07:22:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/shm 2024-03-06T07:22:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/pts 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/run 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/cgroup 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/pstore 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/bpf 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/ 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/selinux 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/hugepages 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/mqueue 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/debug 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/tracing 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/config 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:15.908 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/1000 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.910 DEBUG:teuthology.orchestra.run.smithi169:> ls /sys/fs/fuse/connections 2024-03-06T07:22:15.963 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:15.963 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-06T07:22:16.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo modprobe fuse 2024-03-06T07:22:16.032 DEBUG:teuthology.orchestra.run.smithi169:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:16.087 INFO:teuthology.orchestra.run.smithi169.stdout:/proc 2024-03-06T07:22:16.087 INFO:teuthology.orchestra.run.smithi169.stdout:/sys 2024-03-06T07:22:16.087 INFO:teuthology.orchestra.run.smithi169.stdout:/dev 2024-03-06T07:22:16.087 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/security 2024-03-06T07:22:16.087 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/shm 2024-03-06T07:22:16.087 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/pts 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/run 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/cgroup 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/pstore 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/bpf 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/ 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/selinux 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/hugepages 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/mqueue 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/debug 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/tracing 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/config 2024-03-06T07:22:16.088 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/1000 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:16.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.090 DEBUG:teuthology.orchestra.run.smithi169:> ls /sys/fs/fuse/connections 2024-03-06T07:22:16.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi169.stderr:2024-03-06T07:22:16.098+0000 7fa3dbc48480 -1 init, newargv = 0x55ca736bc810 newargc=15 2024-03-06T07:22:16.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi169.stderr:ceph-fuse[44359]: starting ceph client 2024-03-06T07:22:16.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi169.stderr:ceph-fuse[44359]: starting fuse 2024-03-06T07:22:16.116 INFO:teuthology.orchestra.run.smithi169.stdout:42 2024-03-06T07:22:16.116 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> import glob 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> import re 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> import os 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> import subprocess 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> def _find_admin_socket(client_name): 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> files = glob.glob(asok_path) 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> # Given a non-glob path, it better be there 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> if "*" not in asok_path: 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> assert(len(files) == 1) 2024-03-06T07:22:16.117 DEBUG:teuthology.orchestra.run.smithi169:> return files[0] 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> for f in files: 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> contents = proc_f.read() 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> if mountpoint in contents: 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> return f 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> print(_find_admin_socket("client.1")) 2024-03-06T07:22:16.118 DEBUG:teuthology.orchestra.run.smithi169:> ' 2024-03-06T07:22:16.238 INFO:teuthology.orchestra.run.smithi169.stdout:/var/run/ceph/ceph-client.1.44359.asok 2024-03-06T07:22:16.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:22:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:16.249 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44359.asok 2024-03-06T07:22:16.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44359.asok status 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "metadata": { 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "entity_id": "1", 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "hostname": "smithi169", 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-06T07:22:16.387 INFO:teuthology.orchestra.run.smithi169.stdout: "pid": "44359", 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "root": "/" 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "dentry_count": 0, 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 4663, 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "inst": { 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "name": { 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "client", 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "num": 4663 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": { 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:16.388 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2428716332 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": { 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2428716332 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "inst_str": "client.4663 192.168.0.1:0/2428716332", 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "addr_str": "192.168.0.1:0/2428716332", 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "inode_count": 1, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_epoch": 7, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_epoch": 20, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "blocklisted": false, 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs" 2024-03-06T07:22:16.389 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-06T07:22:16.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:16.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.511+0000 7fa2d61d7640 1 Processor -- start 2024-03-06T07:22:16.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.511+0000 7fa2d61d7640 1 -- start start 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2d61d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2d61d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d015f580 con 0x7fa2d013dc10 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2d61d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d015b450 con 0x7fa2d015bf30 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2d61d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00cf3f0 con 0x7fa2d00b3060 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2cf7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa2d013dc10 0x7fa2d00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43976/0 (socket says 172.21.15.122:43976) 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2cf7fe640 1 -- 172.21.15.122:0/1214243776 learned_addr learned my addr 172.21.15.122:0/1214243776 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2ceffd640 1 --2- 172.21.15.122:0/1214243776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2ce7fc640 1 -- 172.21.15.122:0/1214243776 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803843471 0 0) 0x7fa2d015f580 con 0x7fa2d013dc10 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.512+0000 7fa2ce7fc640 1 -- 172.21.15.122:0/1214243776 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b8003520 con 0x7fa2d013dc10 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.513+0000 7fa2ce7fc640 1 -- 172.21.15.122:0/1214243776 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603007613 0 0) 0x7fa2d015b450 con 0x7fa2d015bf30 2024-03-06T07:22:16.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.513+0000 7fa2ceffd640 1 -- 172.21.15.122:0/1214243776 >> v1:172.21.15.169:6789/0 conn(0x7fa2d015bf30 legacy=0x7fa2d015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.513+0000 7fa2ceffd640 1 -- 172.21.15.122:0/1214243776 >> v1:172.21.15.122:6789/0 conn(0x7fa2d013dc10 legacy=0x7fa2d00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.513+0000 7fa2ceffd640 1 -- 172.21.15.122:0/1214243776 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d015b730 con 0x7fa2d00b3060 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.513+0000 7fa2ceffd640 1 --2- 172.21.15.122:0/1214243776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d015e850 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40097d0 tx=0x7fa2c402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e807e94520372d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2ce7fc640 1 -- 172.21.15.122:0/1214243776 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4035020 con 0x7fa2d00b3060 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2ce7fc640 1 -- 172.21.15.122:0/1214243776 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c4031800 con 0x7fa2d00b3060 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2ce7fc640 1 -- 172.21.15.122:0/1214243776 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4031bd0 con 0x7fa2d00b3060 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 -- 172.21.15.122:0/1214243776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 msgr2=0x7fa2d015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/1214243776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d015e850 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40097d0 tx=0x7fa2c402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 -- 172.21.15.122:0/1214243776 shutdown_connections 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/1214243776 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d015e850 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 -- 172.21.15.122:0/1214243776 >> 172.21.15.122:0/1214243776 conn(0x7fa2d00cf8a0 msgr2=0x7fa2d00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 -- 172.21.15.122:0/1214243776 shutdown_connections 2024-03-06T07:22:16.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.514+0000 7fa2d61d7640 1 -- 172.21.15.122:0/1214243776 wait complete. 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.515+0000 7fa2d61d7640 1 Processor -- start 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 -- start start 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d0153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2d013dc10 0x7fa2d01369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 0x7fa2d015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00718e0 con 0x7fa2d015bf30 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d015b210 con 0x7fa2d013dc10 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.516+0000 7fa2d61d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0071a60 con 0x7fa2d00b3060 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 0x7fa2d015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2ceffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2d013dc10 0x7fa2d01369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d0153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 0x7fa2d015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49716/0 (socket says 172.21.15.122:49716) 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d0153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44258/0 (socket says 172.21.15.122:44258) 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2ceffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2d013dc10 0x7fa2d01369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:37586/0 (socket says 172.21.15.122:37586) 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cffff640 1 -- 172.21.15.122:0/20012928 learned_addr learned my addr 172.21.15.122:0/20012928 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 -- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2d013dc10 msgr2=0x7fa2d01369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2d013dc10 0x7fa2d01369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 -- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 msgr2=0x7fa2d015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 0x7fa2d015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 -- 172.21.15.122:0/20012928 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d01381b0 con 0x7fa2d00b3060 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cffff640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 0x7fa2d015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2cf7fe640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d0153da0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000c400 tx=0x7fa2c000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0017070 con 0x7fa2d00b3060 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c0005ce0 con 0x7fa2d00b3060 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d0138490 con 0x7fa2d00b3060 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.517+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00072f0 con 0x7fa2d00b3060 2024-03-06T07:22:16.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.518+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d0137e60 con 0x7fa2d00b3060 2024-03-06T07:22:16.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.519+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d004f330 con 0x7fa2d00b3060 2024-03-06T07:22:16.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.519+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2c001a070 con 0x7fa2d00b3060 2024-03-06T07:22:16.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.519+0000 7fa2b7fff640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.520+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2c0089550 con 0x7fa2d00b3060 2024-03-06T07:22:16.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.520+0000 7fa2ceffd640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.520+0000 7fa2ceffd640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4036000 tx=0x7fa2c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.524+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2c0057d80 con 0x7fa2d00b3060 2024-03-06T07:22:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.737+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2d006eb00 con 0x7fa2d00b3060 2024-03-06T07:22:16.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.738+0000 7fa2b7fff640 1 -- 172.21.15.122:0/20012928 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa2c005e0d0 con 0x7fa2d00b3060 2024-03-06T07:22:16.740 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa2bc06b4d0 msgr2=0x7fa2bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4036000 tx=0x7fa2c4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 msgr2=0x7fa2d0153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d0153da0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000c400 tx=0x7fa2c000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 shutdown_connections 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa2bc06b4d0 0x7fa2bc06d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d015bf30 0x7fa2d015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2d013dc10 0x7fa2d01369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 --2- 172.21.15.122:0/20012928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa2d00b3060 0x7fa2d0153da0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 >> 172.21.15.122:0/20012928 conn(0x7fa2d00cf8a0 msgr2=0x7fa2d00cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 shutdown_connections 2024-03-06T07:22:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:16.740+0000 7fa2d61d7640 1 -- 172.21.15.122:0/20012928 wait complete. 2024-03-06T07:22:16.752 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:16.752 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi169.front.sepia.ceph.com 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:16.753 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:16.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.753 DEBUG:teuthology.orchestra.run.smithi169:> ip addr 2024-03-06T07:22:16.769 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:16.769 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f0 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f1 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.770 INFO:teuthology.orchestra.run.smithi169.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: inet 172.21.15.169/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft 42670sec preferred_lft 42670sec 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether c2:ef:d7:2f:b7:05 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::d46a:8fff:fe16:f79c/64 scope link 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether c2:ef:d7:2f:b7:05 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::c0ef:d7ff:fe2f:b705/64 scope link tentative 2024-03-06T07:22:16.771 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.772 DEBUG:teuthology.orchestra.run.smithi169:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:16.835 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-06T07:22:16.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.837 DEBUG:teuthology.orchestra.run.smithi169:> route 2024-03-06T07:22:16.893 INFO:teuthology.orchestra.run.smithi169.stdout:Kernel IP routing table 2024-03-06T07:22:16.893 INFO:teuthology.orchestra.run.smithi169.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:16.893 INFO:teuthology.orchestra.run.smithi169.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:16.893 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:16.893 INFO:teuthology.orchestra.run.smithi169.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:16.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.893 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:16.894 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:16.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:16.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:16.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:16.894 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:16.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.992 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-06T07:22:17.007 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:22:17.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.008 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list-id 2024-03-06T07:22:17.062 INFO:teuthology.orchestra.run.smithi169.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:22:17.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.062 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:17.062 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:17.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:17.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:22:17.063 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:17.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:17.175 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:17.175 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:17.175 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:17.175 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.175 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:17.175 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.176 INFO:teuthology.orchestra.run.smithi169.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:17.176 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 36:46:29:c6:a3:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:17.176 INFO:teuthology.orchestra.run.smithi169.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:17.176 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.176 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::3446:29ff:fec6:a3ce/64 scope link tentative 2024-03-06T07:22:17.176 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 36:46:29:c6:a3:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:17.245 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.245 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::3446:29ff:fec6:a3ce/64 scope link tentative 2024-03-06T07:22:17.245 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.246 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:22:17.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.246 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:17.246 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:17.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:22:17.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:17.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:22:17.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:22:17.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:22:17.247 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:17.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.374 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-06T07:22:17.374 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-06T07:22:17.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set brx.1 up 2024-03-06T07:22:17.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:17.374 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-06T07:22:17.418 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:17.418 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:17.472 INFO:teuthology.orchestra.run.smithi169.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:17.472 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:17.472 DEBUG:teuthology.orchestra.run.smithi169:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:17.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo modprobe fuse 2024-03-06T07:22:17.595 DEBUG:teuthology.orchestra.run.smithi169:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:17.651 INFO:teuthology.orchestra.run.smithi169.stdout:/proc 2024-03-06T07:22:17.651 INFO:teuthology.orchestra.run.smithi169.stdout:/sys 2024-03-06T07:22:17.651 INFO:teuthology.orchestra.run.smithi169.stdout:/dev 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/security 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/shm 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/pts 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/run 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/cgroup 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/pstore 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/bpf 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/ 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/selinux 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/hugepages 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/mqueue 2024-03-06T07:22:17.652 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/debug 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/tracing 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/config 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/1000 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:17.653 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns 2024-03-06T07:22:17.654 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:17.654 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:17.654 INFO:teuthology.orchestra.run.smithi169.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:17.654 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:17.654 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:17.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.654 DEBUG:teuthology.orchestra.run.smithi169:> ls /sys/fs/fuse/connections 2024-03-06T07:22:17.706 INFO:teuthology.orchestra.run.smithi169.stdout:42 2024-03-06T07:22:17.706 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-06T07:22:17.706 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-06T07:22:17.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo modprobe fuse 2024-03-06T07:22:17.777 DEBUG:teuthology.orchestra.run.smithi169:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/proc 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/sys 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/dev 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/security 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/shm 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/pts 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/run 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/cgroup 2024-03-06T07:22:17.833 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/pstore 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/bpf 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/ 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/selinux 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/hugepages 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/mqueue 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/debug 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/tracing 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/config 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:17.834 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/1000 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:17.835 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:17.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.836 DEBUG:teuthology.orchestra.run.smithi169:> ls /sys/fs/fuse/connections 2024-03-06T07:22:17.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi169.stderr:2024-03-06T07:22:17.846+0000 7f2bace3a480 -1 init, newargv = 0x556d2196a640 newargc=15 2024-03-06T07:22:17.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi169.stderr:ceph-fuse[44825]: starting ceph client 2024-03-06T07:22:17.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi169.stderr:ceph-fuse[44825]: starting fuse 2024-03-06T07:22:17.864 INFO:teuthology.orchestra.run.smithi169.stdout:42 2024-03-06T07:22:17.864 INFO:teuthology.orchestra.run.smithi169.stdout:44 2024-03-06T07:22:17.864 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-06T07:22:17.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> import glob 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> import re 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> import os 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> import subprocess 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> def _find_admin_socket(client_name): 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> files = glob.glob(asok_path) 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> # Given a non-glob path, it better be there 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> if "*" not in asok_path: 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> assert(len(files) == 1) 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> return files[0] 2024-03-06T07:22:17.865 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> for f in files: 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> contents = proc_f.read() 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> if mountpoint in contents: 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> return f 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> print(_find_admin_socket("client.3")) 2024-03-06T07:22:17.866 DEBUG:teuthology.orchestra.run.smithi169:> ' 2024-03-06T07:22:17.985 INFO:teuthology.orchestra.run.smithi169.stdout:/var/run/ceph/ceph-client.3.44825.asok 2024-03-06T07:22:17.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-06T07:22:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:17.995 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44825.asok 2024-03-06T07:22:17.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44825.asok status 2024-03-06T07:22:18.133 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-06T07:22:18.133 INFO:teuthology.orchestra.run.smithi169.stdout: "metadata": { 2024-03-06T07:22:18.133 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:18.133 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:18.133 INFO:teuthology.orchestra.run.smithi169.stdout: "entity_id": "3", 2024-03-06T07:22:18.133 INFO:teuthology.orchestra.run.smithi169.stdout: "hostname": "smithi169", 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "pid": "44825", 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "root": "/" 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "dentry_count": 0, 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 4589, 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "inst": { 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "name": { 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "client", 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "num": 4589 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": { 2024-03-06T07:22:18.134 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3857878328 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": { 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3857878328 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "inst_str": "client.4589 192.168.0.2:0/3857878328", 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "addr_str": "192.168.0.2:0/3857878328", 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "inode_count": 1, 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_epoch": 7, 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_epoch": 20, 2024-03-06T07:22:18.135 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:18.136 INFO:teuthology.orchestra.run.smithi169.stdout: "blocklisted": false, 2024-03-06T07:22:18.136 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs" 2024-03-06T07:22:18.136 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-06T07:22:18.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.142 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:22:18.142 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:18.161 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:22:18.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:18.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.161 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:18.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.235 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:22:18.235 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:18.293 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:22:18.293 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:18.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:18.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.367 DEBUG:teuthology.orchestra.run.smithi169:> stat --file-system '--printf=%T 2024-03-06T07:22:18.367 DEBUG:teuthology.orchestra.run.smithi169:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:18.382 INFO:teuthology.orchestra.run.smithi169.stdout:fuseblk 2024-03-06T07:22:18.383 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:18.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:18.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.449 DEBUG:teuthology.orchestra.run.smithi169:> stat --file-system '--printf=%T 2024-03-06T07:22:18.449 DEBUG:teuthology.orchestra.run.smithi169:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi169.stdout:fuseblk 2024-03-06T07:22:18.504 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:18.571 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:22:18.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:18.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.696+0000 7f122f24e640 1 Processor -- start 2024-03-06T07:22:18.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.696+0000 7f122f24e640 1 -- start start 2024-03-06T07:22:18.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122f24e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f12281625a0 0x7f12280cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122f24e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280ceac0 con 0x7f12280d4920 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122f24e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280d2dc0 con 0x7f12281621c0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122f24e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12280cf690 con 0x7f12281625a0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122d7eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f12281625a0 0x7f12280cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f1227fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f12280d4920 0x7f122815dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43986/0 (socket says 172.21.15.122:43986) 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f1227fff640 1 -- 172.21.15.122:0/4017197435 learned_addr learned my addr 172.21.15.122:0/4017197435 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122d7eb640 1 -- 172.21.15.122:0/4017197435 >> v1:172.21.15.169:6789/0 conn(0x7f12281621c0 legacy=0x7f12280d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f122d7eb640 1 -- 172.21.15.122:0/4017197435 >> v1:172.21.15.122:6789/0 conn(0x7f12280d4920 legacy=0x7f122815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.697+0000 7f12277fe640 1 -- 172.21.15.122:0/4017197435 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560060661 0 0) 0x7f12280d2dc0 con 0x7f12281621c0 2024-03-06T07:22:18.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.698+0000 7f122d7eb640 1 -- 172.21.15.122:0/4017197435 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280ced60 con 0x7f12281625a0 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.698+0000 7f122d7eb640 1 --2- 172.21.15.122:0/4017197435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280cfb20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f122002bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd2d91a9a0ca68f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.698+0000 7f12277fe640 1 -- 172.21.15.122:0/4017197435 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220039070 con 0x7f12281625a0 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.698+0000 7f12277fe640 1 -- 172.21.15.122:0/4017197435 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122002ca20 con 0x7f12281625a0 2024-03-06T07:22:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.698+0000 7f12277fe640 1 -- 172.21.15.122:0/4017197435 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122002cdf0 con 0x7f12281625a0 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 -- 172.21.15.122:0/4017197435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 msgr2=0x7f12280cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 --2- 172.21.15.122:0/4017197435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280cfb20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f122002bcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 -- 172.21.15.122:0/4017197435 shutdown_connections 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 --2- 172.21.15.122:0/4017197435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280cfb20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 -- 172.21.15.122:0/4017197435 >> 172.21.15.122:0/4017197435 conn(0x7f12280d3d10 msgr2=0x7f12280d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 -- 172.21.15.122:0/4017197435 shutdown_connections 2024-03-06T07:22:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.699+0000 7f122f24e640 1 -- 172.21.15.122:0/4017197435 wait complete. 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.700+0000 7f122f24e640 1 Processor -- start 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.700+0000 7f122f24e640 1 -- start start 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122f24e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12280d4920 0x7f12281f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122f24e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12281621c0 0x7f12280b9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122f24e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122f24e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122815e450 con 0x7f12281625a0 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122f24e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12280cf320 con 0x7f12280d4920 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122f24e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122815e5d0 con 0x7f12281621c0 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49732/0 (socket says 172.21.15.122:49732) 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 -- 172.21.15.122:0/1943470127 learned_addr learned my addr 172.21.15.122:0/1943470127 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f1227fff640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12281621c0 0x7f12280b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122cfea640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12280d4920 0x7f12281f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 -- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12281621c0 msgr2=0x7f12280b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12281621c0 0x7f12280b9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 -- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12280d4920 msgr2=0x7f12281f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12280d4920 0x7f12281f3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 -- 172.21.15.122:0/1943470127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280b6970 con 0x7f12281625a0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.701+0000 7f122d7eb640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280ccbf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f1220004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.702+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220044020 con 0x7f12281625a0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.702+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1220034c10 con 0x7f12281625a0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.702+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12280b6c50 con 0x7f12281625a0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.702+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220039040 con 0x7f12281625a0 2024-03-06T07:22:18.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.702+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12280b7210 con 0x7f12281625a0 2024-03-06T07:22:18.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.703+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122804f330 con 0x7f12281625a0 2024-03-06T07:22:18.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.703+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1220047020 con 0x7f12281625a0 2024-03-06T07:22:18.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.704+0000 7f12257fa640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f121406b4d0 0x7f121406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.704+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12200ac5c0 con 0x7f12281625a0 2024-03-06T07:22:18.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.704+0000 7f122cfea640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f121406b4d0 0x7f121406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.704+0000 7f122cfea640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f121406b4d0 0x7f121406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1218000f60 tx=0x7f1218009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.708+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f122004e070 con 0x7f12281625a0 2024-03-06T07:22:18.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.879+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1228073380 con 0x7f12281625a0 2024-03-06T07:22:18.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.880+0000 7f12257fa640 1 -- 172.21.15.122:0/1943470127 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f122007af40 con 0x7f12281625a0 2024-03-06T07:22:18.883 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:18.883 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:18.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.882+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f121406b4d0 msgr2=0x7f121406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.882+0000 7f122f24e640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f121406b4d0 0x7f121406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1218000f60 tx=0x7f1218009040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.882+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 msgr2=0x7f12280ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.882+0000 7f122f24e640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280ccbf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f1220004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 shutdown_connections 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f121406b4d0 0x7f121406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12281625a0 0x7f12280ccbf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12281621c0 0x7f12280b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 --2- 172.21.15.122:0/1943470127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12280d4920 0x7f12281f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 >> 172.21.15.122:0/1943470127 conn(0x7f12280d3d10 msgr2=0x7f122815ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 shutdown_connections 2024-03-06T07:22:18.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.883+0000 7f122f24e640 1 -- 172.21.15.122:0/1943470127 wait complete. 2024-03-06T07:22:18.896 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:18.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:19.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 Processor -- start 2024-03-06T07:22:19.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 -- start start 2024-03-06T07:22:19.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f65281614d0 0x7f65280d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d1910 0x7f65280d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65280d2740 con 0x7f65280d22b0 2024-03-06T07:22:19.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65281618b0 con 0x7f65281614d0 2024-03-06T07:22:19.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.998+0000 7f652dc4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65280cf400 con 0x7f65280d1910 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f6527fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f65280d22b0 0x7f65280d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43996/0 (socket says 172.21.15.122:43996) 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f6527fff640 1 -- 172.21.15.122:0/2029616475 learned_addr learned my addr 172.21.15.122:0/2029616475 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f65277fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f65281614d0 0x7f65280d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f6526ffd640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d1910 0x7f65280d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f65267fc640 1 -- 172.21.15.122:0/2029616475 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900904940 0 0) 0x7f6520001490 con 0x7f65280d22b0 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f65277fe640 1 -- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d1910 msgr2=0x7f65280d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f65277fe640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d1910 0x7f65280d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f65277fe640 1 -- 172.21.15.122:0/2029616475 >> v1:172.21.15.122:6789/0 conn(0x7f65280d22b0 legacy=0x7f65280d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f65277fe640 1 -- 172.21.15.122:0/2029616475 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65280cf6a0 con 0x7f65281614d0 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:18.999+0000 7f6526ffd640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d1910 0x7f65280d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f65277fe640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f65280d4860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65180097a0 tx=0x7f651802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=976ba94c5cc37c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f65267fc640 1 -- 172.21.15.122:0/2029616475 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518035020 con 0x7f65281614d0 2024-03-06T07:22:19.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f65267fc640 1 -- 172.21.15.122:0/2029616475 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6518031830 con 0x7f65281614d0 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f65267fc640 1 -- 172.21.15.122:0/2029616475 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518031c00 con 0x7f65281614d0 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 -- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 msgr2=0x7f65280d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f65280d4860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65180097a0 tx=0x7f651802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 -- 172.21.15.122:0/2029616475 shutdown_connections 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d1910 0x7f65280d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 --2- 172.21.15.122:0/2029616475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f65280d4860 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 -- 172.21.15.122:0/2029616475 >> 172.21.15.122:0/2029616475 conn(0x7f65280d3d40 msgr2=0x7f65280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 -- 172.21.15.122:0/2029616475 shutdown_connections 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.000+0000 7f652dc4e640 1 -- 172.21.15.122:0/2029616475 wait complete. 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.001+0000 7f652dc4e640 1 Processor -- start 2024-03-06T07:22:19.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 -- start start 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 0x7f65280b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d22b0 0x7f6528150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f6528150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65280b0910 con 0x7f65280d1910 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65280cfc60 con 0x7f65281614d0 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f652dc4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65280b1e40 con 0x7f65280d22b0 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.002+0000 7f65277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 0x7f65280b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 0x7f65280b6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49746/0 (socket says 172.21.15.122:49746) 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 -- 172.21.15.122:0/505700930 learned_addr learned my addr 172.21.15.122:0/505700930 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f6526ffd640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d22b0 0x7f6528150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f6527fff640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f6528150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 -- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d22b0 msgr2=0x7f6528150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d22b0 0x7f6528150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 -- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 msgr2=0x7f6528150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f6528150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 -- 172.21.15.122:0/505700930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65280b9700 con 0x7f65280d1910 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f6526ffd640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d22b0 0x7f6528150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f65277fe640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 0x7f65280b6b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6518034040 tx=0x7f65180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518040070 con 0x7f65280d1910 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65280b99e0 con 0x7f65280d1910 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651803a910 con 0x7f65280d1910 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518035040 con 0x7f65280d1910 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.003+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65280cc140 con 0x7f65280d1910 2024-03-06T07:22:19.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.004+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6528068ff0 con 0x7f65280d1910 2024-03-06T07:22:19.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.005+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6518038020 con 0x7f65280d1910 2024-03-06T07:22:19.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.005+0000 7f650ffff640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f651406b4d0 0x7f651406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.005+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65180abbb0 con 0x7f65280d1910 2024-03-06T07:22:19.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.005+0000 7f6526ffd640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f651406b4d0 0x7f651406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.005+0000 7f6526ffd640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f651406b4d0 0x7f651406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65280d0c70 tx=0x7f651c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.009+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6518031800 con 0x7f65280d1910 2024-03-06T07:22:19.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.179+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f652806ba70 con 0x7f65280d1910 2024-03-06T07:22:19.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.180+0000 7f650ffff640 1 -- 172.21.15.122:0/505700930 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f651807a360 con 0x7f65280d1910 2024-03-06T07:22:19.182 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.181+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f651406b4d0 msgr2=0x7f651406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.181+0000 7f652dc4e640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f651406b4d0 0x7f651406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65280d0c70 tx=0x7f651c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.181+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 msgr2=0x7f65280b6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.181+0000 7f652dc4e640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 0x7f65280b6b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6518034040 tx=0x7f65180046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 shutdown_connections 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f651406b4d0 0x7f651406d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65281614d0 0x7f6528150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f65280d22b0 0x7f6528150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 --2- 172.21.15.122:0/505700930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65280d1910 0x7f65280b6b00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 >> 172.21.15.122:0/505700930 conn(0x7f65280d3d40 msgr2=0x7f65280cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 shutdown_connections 2024-03-06T07:22:19.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.182+0000 7f652dc4e640 1 -- 172.21.15.122:0/505700930 wait complete. 2024-03-06T07:22:19.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.295+0000 7f84a8ba2640 1 Processor -- start 2024-03-06T07:22:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.295+0000 7f84a8ba2640 1 -- start start 2024-03-06T07:22:19.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a8ba2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84a40d19d0 0x7f84a40d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a8ba2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a40b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a8ba2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a40d0ae0 con 0x7f84a415e010 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a8ba2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a40b3570 con 0x7f84a40d19d0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a8ba2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40d0880 con 0x7f84a40d2660 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84a40d19d0 0x7f84a40d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84a40d19d0 0x7f84a40d2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49748/0 (socket says 172.21.15.122:49748) 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 -- 172.21.15.122:0/3500674014 learned_addr learned my addr 172.21.15.122:0/3500674014 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a37fe640 1 --1- 172.21.15.122:0/3500674014 >> v1:172.21.15.169:6789/0 conn(0x7f84a415e010 0x7f84a40d0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:41914/0 (socket says 172.21.15.122:41914) 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a3fff640 1 --2- 172.21.15.122:0/3500674014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a40b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 -- 172.21.15.122:0/3500674014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 msgr2=0x7f84a40b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 --2- 172.21.15.122:0/3500674014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a40b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 -- 172.21.15.122:0/3500674014 >> v1:172.21.15.169:6789/0 conn(0x7f84a415e010 legacy=0x7f84a40d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a2ffd640 1 -- 172.21.15.122:0/3500674014 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a40d0dc0 con 0x7f84a40d19d0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.296+0000 7f84a27fc640 1 -- 172.21.15.122:0/3500674014 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811375292 0 0) 0x7f84a40d0ae0 con 0x7f84a415e010 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.297+0000 7f84a2ffd640 1 --2- 172.21.15.122:0/3500674014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40d2120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8498009870 tx=0x7f849802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f68e9bae3f143e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.297+0000 7f84a27fc640 1 -- 172.21.15.122:0/3500674014 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849802f840 con 0x7f84a40d19d0 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.297+0000 7f84a27fc640 1 -- 172.21.15.122:0/3500674014 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849802f9e0 con 0x7f84a40d19d0 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.297+0000 7f84a27fc640 1 -- 172.21.15.122:0/3500674014 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849802fdb0 con 0x7f84a40d19d0 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 -- 172.21.15.122:0/3500674014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 msgr2=0x7f84a40d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/3500674014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40d2120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8498009870 tx=0x7f849802ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 -- 172.21.15.122:0/3500674014 shutdown_connections 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/3500674014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a40b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/3500674014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40d2120 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 -- 172.21.15.122:0/3500674014 >> 172.21.15.122:0/3500674014 conn(0x7f84a409ce10 msgr2=0x7f84a40d3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 -- 172.21.15.122:0/3500674014 shutdown_connections 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.298+0000 7f84a8ba2640 1 -- 172.21.15.122:0/3500674014 wait complete. 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.299+0000 7f84a8ba2640 1 Processor -- start 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.299+0000 7f84a8ba2640 1 -- start start 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a8ba2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40b78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a8ba2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a41f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a8ba2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a415e010 0x7f84a41f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a8ba2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40b3f40 con 0x7f84a40d19d0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40b78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a2ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a41f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40b78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49762/0 (socket says 172.21.15.122:49762) 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 -- 172.21.15.122:0/2584298327 learned_addr learned my addr 172.21.15.122:0/2584298327 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a3fff640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a415e010 0x7f84a41f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a8ba2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40b3200 con 0x7f84a415e010 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40b40c0 con 0x7f84a40d2660 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 -- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 msgr2=0x7f84a41f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a41f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 -- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a415e010 msgr2=0x7f84a41f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a415e010 0x7f84a41f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 -- 172.21.15.122:0/2584298327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a41f6a20 con 0x7f84a40d19d0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.300+0000 7f84a37fe640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40b78c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f849400c430 tx=0x7f849400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.301+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84940074e0 con 0x7f84a40d19d0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.301+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84940047e0 con 0x7f84a40d19d0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.301+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a41f5440 con 0x7f84a40d19d0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.301+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494004bd0 con 0x7f84a40d19d0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.301+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a41f5a00 con 0x7f84a40d19d0 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.302+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4068f10 con 0x7f84a40d19d0 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.302+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8494011070 con 0x7f84a40d19d0 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.302+0000 7f848bfff640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.302+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8494089ca0 con 0x7f84a40d19d0 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.303+0000 7f84a2ffd640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.303+0000 7f84a2ffd640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8498036000 tx=0x7f8498008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.307+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84940584d0 con 0x7f84a40d19d0 2024-03-06T07:22:19.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.478+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84a41f6100 con 0x7f84a40d19d0 2024-03-06T07:22:19.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.479+0000 7f848bfff640 1 -- 172.21.15.122:0/2584298327 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f849405c570 con 0x7f84a40d19d0 2024-03-06T07:22:19.481 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.480+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 msgr2=0x7f849006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.480+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8498036000 tx=0x7f8498008030 comp rx=0 tx=0).stop 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.480+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 msgr2=0x7f84a40b78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.480+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40b78c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f849400c430 tx=0x7f849400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.480+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 shutdown_connections 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a415e010 0x7f84a41f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a40d2660 0x7f84a41f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 --2- 172.21.15.122:0/2584298327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a40d19d0 0x7f84a40b78c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 >> 172.21.15.122:0/2584298327 conn(0x7f84a409ce10 msgr2=0x7f84a40d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 shutdown_connections 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.481+0000 7f84a8ba2640 1 -- 172.21.15.122:0/2584298327 wait complete. 2024-03-06T07:22:19.493 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:19.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.594+0000 7febab920640 1 Processor -- start 2024-03-06T07:22:19.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.594+0000 7febab920640 1 -- start start 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7febab920640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7feba415bbb0 0x7feba406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7febab920640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feba406b4f0 0x7feba415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7febab920640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba415ec40 0x7feba40cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7febab920640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba4161970 con 0x7feba406b4f0 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7febab920640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba41624f0 con 0x7feba415bbb0 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7febab920640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba40cea90 con 0x7feba415ec40 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feba406b4f0 0x7feba415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feba406b4f0 0x7feba415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49764/0 (socket says 172.21.15.122:49764) 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 -- 172.21.15.122:0/105410019 learned_addr learned my addr 172.21.15.122:0/105410019 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba96bc640 1 --2- 172.21.15.122:0/105410019 >> v2:172.21.15.169:3300/0 conn(0x7feba415bbb0 0x7feba406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba9ebd640 1 --2- 172.21.15.122:0/105410019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba415ec40 0x7feba40cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 -- 172.21.15.122:0/105410019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba415ec40 msgr2=0x7feba40cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 --2- 172.21.15.122:0/105410019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba415ec40 0x7feba40cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 -- 172.21.15.122:0/105410019 >> v2:172.21.15.169:3300/0 conn(0x7feba415bbb0 msgr2=0x7feba406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 --2- 172.21.15.122:0/105410019 >> v2:172.21.15.169:3300/0 conn(0x7feba415bbb0 0x7feba406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.595+0000 7feba8ebb640 1 -- 172.21.15.122:0/105410019 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4161bc0 con 0x7feba406b4f0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.596+0000 7feba8ebb640 1 --2- 172.21.15.122:0/105410019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba406b4f0 0x7feba415e700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7feba40ce420 tx=0x7feb9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd29ac2121240779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.596+0000 7feb9bfff640 1 -- 172.21.15.122:0/105410019 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94035020 con 0x7feba406b4f0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.596+0000 7feb9bfff640 1 -- 172.21.15.122:0/105410019 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb94031870 con 0x7feba406b4f0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.596+0000 7feb9bfff640 1 -- 172.21.15.122:0/105410019 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94031c40 con 0x7feba406b4f0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 -- 172.21.15.122:0/105410019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba406b4f0 msgr2=0x7feba415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 --2- 172.21.15.122:0/105410019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba406b4f0 0x7feba415e700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7feba40ce420 tx=0x7feb9402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 -- 172.21.15.122:0/105410019 shutdown_connections 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 --2- 172.21.15.122:0/105410019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba406b4f0 0x7feba415e700 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 --2- 172.21.15.122:0/105410019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba415ec40 0x7feba40cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 --2- 172.21.15.122:0/105410019 >> v2:172.21.15.169:3300/0 conn(0x7feba415bbb0 0x7feba406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 -- 172.21.15.122:0/105410019 >> 172.21.15.122:0/105410019 conn(0x7feba40b2660 msgr2=0x7feba40b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 -- 172.21.15.122:0/105410019 shutdown_connections 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.597+0000 7febab920640 1 -- 172.21.15.122:0/105410019 wait complete. 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.598+0000 7febab920640 1 Processor -- start 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.598+0000 7febab920640 1 -- start start 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba406b4f0 0x7feba415fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 0x7feba40cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feba4151770 0x7feba4155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba415f0d0 con 0x7feba415ec40 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4162180 con 0x7feba4151770 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba40cd8e0 con 0x7feba406b4f0 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 0x7feba40cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 0x7feba40cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49770/0 (socket says 172.21.15.122:49770) 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 -- 172.21.15.122:0/2330936810 learned_addr learned my addr 172.21.15.122:0/2330936810 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba9ebd640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feba4151770 0x7feba4155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba96bc640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba406b4f0 0x7feba415fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 -- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba406b4f0 msgr2=0x7feba415fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba406b4f0 0x7feba415fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 -- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feba4151770 msgr2=0x7feba4155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feba4151770 0x7feba4155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 -- 172.21.15.122:0/2330936810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4160540 con 0x7feba415ec40 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feba8ebb640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 0x7feba40cc800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb94000c00 tx=0x7feb940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94035040 con 0x7feba415ec40 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb94030d00 con 0x7feba415ec40 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4160820 con 0x7feba415ec40 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.599+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940456a0 con 0x7feba415ec40 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.600+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba4160de0 con 0x7feba415ec40 2024-03-06T07:22:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.600+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba40ccd40 con 0x7feba415ec40 2024-03-06T07:22:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.601+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb94038020 con 0x7feba415ec40 2024-03-06T07:22:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.601+0000 7feb99ffb640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb8806b4d0 0x7feb8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.601+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb940ac410 con 0x7feba415ec40 2024-03-06T07:22:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.601+0000 7feba96bc640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb8806b4d0 0x7feb8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.602+0000 7feba96bc640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb8806b4d0 0x7feb8806d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feba0011360 tx=0x7feba000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.606+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb9407ac40 con 0x7feba415ec40 2024-03-06T07:22:19.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.779+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feba40733b0 con 0x7feba415ec40 2024-03-06T07:22:19.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.779+0000 7feb99ffb640 1 -- 172.21.15.122:0/2330936810 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7feb9407ece0 con 0x7feba415ec40 2024-03-06T07:22:19.781 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:19.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb8806b4d0 msgr2=0x7feb8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb8806b4d0 0x7feb8806d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feba0011360 tx=0x7feba000d040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 msgr2=0x7feba40cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 0x7feba40cc800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb94000c00 tx=0x7feb940046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 shutdown_connections 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb8806b4d0 0x7feb8806d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feba4151770 0x7feba4155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba415ec40 0x7feba40cc800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 --2- 172.21.15.122:0/2330936810 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feba406b4f0 0x7feba415fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 >> 172.21.15.122:0/2330936810 conn(0x7feba40b2660 msgr2=0x7feba4157f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 shutdown_connections 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.781+0000 7febab920640 1 -- 172.21.15.122:0/2330936810 wait complete. 2024-03-06T07:22:19.794 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:19.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.892+0000 7f35aaf08640 1 Processor -- start 2024-03-06T07:22:19.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.892+0000 7f35aaf08640 1 -- start start 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35aaf08640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f35a406c6d0 0x7f35a415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35aaf08640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4066d90 0x7f35a40ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35aaf08640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40cd450 con 0x7f35a40b4f70 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35aaf08640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a415e9f0 con 0x7f35a406c6d0 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35aaf08640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40cda10 con 0x7f35a4066d90 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a8ca4640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f35a40b4f70 0x7f35a40b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43998/0 (socket says 172.21.15.122:43998) 2024-03-06T07:22:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a8ca4640 1 -- 172.21.15.122:0/1845111355 learned_addr learned my addr 172.21.15.122:0/1845111355 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a94a5640 1 --2- 172.21.15.122:0/1845111355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4066d90 0x7f35a40ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a3fff640 1 --2- 172.21.15.122:0/1845111355 >> v2:172.21.15.169:3300/0 conn(0x7f35a406c6d0 0x7f35a415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a37fe640 1 -- 172.21.15.122:0/1845111355 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393248559 0 0) 0x7f35a40cd450 con 0x7f35a40b4f70 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a37fe640 1 -- 172.21.15.122:0/1845111355 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3594003520 con 0x7f35a40b4f70 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a94a5640 1 -- 172.21.15.122:0/1845111355 >> v2:172.21.15.169:3300/0 conn(0x7f35a406c6d0 msgr2=0x7f35a415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a94a5640 1 --2- 172.21.15.122:0/1845111355 >> v2:172.21.15.169:3300/0 conn(0x7f35a406c6d0 0x7f35a415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a94a5640 1 -- 172.21.15.122:0/1845111355 >> v1:172.21.15.122:6789/0 conn(0x7f35a40b4f70 legacy=0x7f35a40b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a94a5640 1 -- 172.21.15.122:0/1845111355 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40cd730 con 0x7f35a4066d90 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.893+0000 7f35a3fff640 1 --2- 172.21.15.122:0/1845111355 >> v2:172.21.15.169:3300/0 conn(0x7f35a406c6d0 0x7f35a415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.894+0000 7f35a94a5640 1 --2- 172.21.15.122:0/1845111355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4066d90 0x7f35a40ccb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f359c009d90 tx=0x7f359c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e174834fcae35ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.894+0000 7f35a37fe640 1 -- 172.21.15.122:0/1845111355 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c039070 con 0x7f35a4066d90 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.894+0000 7f35a37fe640 1 -- 172.21.15.122:0/1845111355 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c02cc50 con 0x7f35a4066d90 2024-03-06T07:22:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.894+0000 7f35a37fe640 1 -- 172.21.15.122:0/1845111355 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c034440 con 0x7f35a4066d90 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 -- 172.21.15.122:0/1845111355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4066d90 msgr2=0x7f35a40ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 --2- 172.21.15.122:0/1845111355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4066d90 0x7f35a40ccb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f359c009d90 tx=0x7f359c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 -- 172.21.15.122:0/1845111355 shutdown_connections 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 --2- 172.21.15.122:0/1845111355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4066d90 0x7f35a40ccb50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 --2- 172.21.15.122:0/1845111355 >> v2:172.21.15.169:3300/0 conn(0x7f35a406c6d0 0x7f35a415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 -- 172.21.15.122:0/1845111355 >> 172.21.15.122:0/1845111355 conn(0x7f35a40724d0 msgr2=0x7f35a40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 -- 172.21.15.122:0/1845111355 shutdown_connections 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.895+0000 7f35aaf08640 1 -- 172.21.15.122:0/1845111355 wait complete. 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.896+0000 7f35aaf08640 1 Processor -- start 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.896+0000 7f35aaf08640 1 -- start start 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 0x7f35a40af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a40b4f70 0x7f35a40ad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4149500 0x7f35a4160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40cd090 con 0x7f35a4066d90 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a415f3c0 con 0x7f35a40b4f70 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40cd210 con 0x7f35a4149500 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 0x7f35a40af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 0x7f35a40af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49776/0 (socket says 172.21.15.122:49776) 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 -- 172.21.15.122:0/2796055553 learned_addr learned my addr 172.21.15.122:0/2796055553 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a3fff640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a40b4f70 0x7f35a40ad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a94a5640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4149500 0x7f35a4160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 -- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4149500 msgr2=0x7f35a4160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4149500 0x7f35a4160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 -- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a40b4f70 msgr2=0x7f35a40ad0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a40b4f70 0x7f35a40ad0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 -- 172.21.15.122:0/2796055553 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a415b110 con 0x7f35a4066d90 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a8ca4640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 0x7f35a40af400 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f359800c390 tx=0x7f359800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598017070 con 0x7f35a4066d90 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3598005ce0 con 0x7f35a4066d90 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.897+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a41f4430 con 0x7f35a4066d90 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.898+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35980072f0 con 0x7f35a4066d90 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.898+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a40ada60 con 0x7f35a4066d90 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.898+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a404f330 con 0x7f35a4066d90 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.899+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f359801a070 con 0x7f35a4066d90 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.899+0000 7f35a17fa640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358806b4d0 0x7f358806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.899+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3598089660 con 0x7f35a4066d90 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.899+0000 7f35a3fff640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358806b4d0 0x7f358806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.900+0000 7f35a3fff640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358806b4d0 0x7f358806d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3590009560 tx=0x7f3590009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:19.903+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3598057d80 con 0x7f35a4066d90 2024-03-06T07:22:20.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.076+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a40b1860 con 0x7f35a4066d90 2024-03-06T07:22:20.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.077+0000 7f35a17fa640 1 -- 172.21.15.122:0/2796055553 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f359805be20 con 0x7f35a4066d90 2024-03-06T07:22:20.079 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:20.079 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:20.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358806b4d0 msgr2=0x7f358806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358806b4d0 0x7f358806d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3590009560 tx=0x7f3590009190 comp rx=0 tx=0).stop 2024-03-06T07:22:20.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 msgr2=0x7f35a40af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 0x7f35a40af400 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f359800c390 tx=0x7f359800c850 comp rx=0 tx=0).stop 2024-03-06T07:22:20.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 shutdown_connections 2024-03-06T07:22:20.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358806b4d0 0x7f358806d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a4149500 0x7f35a4160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a40b4f70 0x7f35a40ad0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 --2- 172.21.15.122:0/2796055553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a4066d90 0x7f35a40af400 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 >> 172.21.15.122:0/2796055553 conn(0x7f35a40724d0 msgr2=0x7f35a40728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 shutdown_connections 2024-03-06T07:22:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.079+0000 7f35aaf08640 1 -- 172.21.15.122:0/2796055553 wait complete. 2024-03-06T07:22:20.092 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:20.092 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:20.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.194+0000 7f1421979640 1 Processor -- start 2024-03-06T07:22:20.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.194+0000 7f1421979640 1 -- start start 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.194+0000 7f1421979640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f141c0d4860 0x7f141c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.194+0000 7f1421979640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c1618b0 con 0x7f141c0b0e30 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.194+0000 7f1421979640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c0cf400 con 0x7f141c1614d0 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.194+0000 7f1421979640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141c0d4cf0 con 0x7f141c0d4860 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f141b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f141c0b0e30 0x7f141c0b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44006/0 (socket says 172.21.15.122:44006) 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f141b7fe640 1 -- 172.21.15.122:0/1455157490 learned_addr learned my addr 172.21.15.122:0/1455157490 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f14127fc640 1 --2- 172.21.15.122:0/1455157490 >> v2:172.21.15.169:3300/0 conn(0x7f141c0d4860 0x7f141c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291059282 0 0) 0x7f141c1618b0 con 0x7f141c0b0e30 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1408003520 con 0x7f141c0b0e30 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729276904 0 0) 0x7f141c0cf400 con 0x7f141c1614d0 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141c1618b0 con 0x7f141c1614d0 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f14127fc640 1 -- 172.21.15.122:0/1455157490 >> v1:172.21.15.169:6789/0 conn(0x7f141c1614d0 legacy=0x7f141c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f14127fc640 1 -- 172.21.15.122:0/1455157490 >> v1:172.21.15.122:6789/0 conn(0x7f141c0b0e30 legacy=0x7f141c0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.195+0000 7f14127fc640 1 -- 172.21.15.122:0/1455157490 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c0cf6a0 con 0x7f141c0d4860 2024-03-06T07:22:20.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f14127fc640 1 --2- 172.21.15.122:0/1455157490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c0d4860 0x7f141c0d2020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14040097a0 tx=0x7f140402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=496a26728a5d0680 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404034070 con 0x7f141c0d4860 2024-03-06T07:22:20.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1404030610 con 0x7f141c0d4860 2024-03-06T07:22:20.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f141a7fc640 1 -- 172.21.15.122:0/1455157490 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14040309e0 con 0x7f141c0d4860 2024-03-06T07:22:20.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 -- 172.21.15.122:0/1455157490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c0d4860 msgr2=0x7f141c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 --2- 172.21.15.122:0/1455157490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c0d4860 0x7f141c0d2020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14040097a0 tx=0x7f140402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 -- 172.21.15.122:0/1455157490 shutdown_connections 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 --2- 172.21.15.122:0/1455157490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c0d4860 0x7f141c0d2020 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 -- 172.21.15.122:0/1455157490 >> 172.21.15.122:0/1455157490 conn(0x7f141c0d3d40 msgr2=0x7f141c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 -- 172.21.15.122:0/1455157490 shutdown_connections 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.196+0000 7f1421979640 1 -- 172.21.15.122:0/1455157490 wait complete. 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 Processor -- start 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 -- start start 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 0x7f141c0b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f141c0d4860 0x7f141c0b9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c1614d0 0x7f141c152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c0b0910 con 0x7f141c0b0e30 2024-03-06T07:22:20.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c0cfc60 con 0x7f141c1614d0 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.198+0000 7f1421979640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141c0d2560 con 0x7f141c0d4860 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 0x7f141c0b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 0x7f141c0b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49786/0 (socket says 172.21.15.122:49786) 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 -- 172.21.15.122:0/3215838156 learned_addr learned my addr 172.21.15.122:0/3215838156 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141b7fe640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c1614d0 0x7f141c152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f14127fc640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f141c0d4860 0x7f141c0b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 -- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f141c0d4860 msgr2=0x7f141c0b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f141c0d4860 0x7f141c0b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 -- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c1614d0 msgr2=0x7f141c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c1614d0 0x7f141c152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 -- 172.21.15.122:0/3215838156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c0c8f10 con 0x7f141c0b0e30 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f141affd640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 0x7f141c0b6930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f140c00c460 tx=0x7f140c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c017070 con 0x7f141c0b0e30 2024-03-06T07:22:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c0052f0 con 0x7f141c0b0e30 2024-03-06T07:22:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c003830 con 0x7f141c0b0e30 2024-03-06T07:22:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c0c91b0 con 0x7f141c0b0e30 2024-03-06T07:22:20.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.199+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141c0c9770 con 0x7f141c0b0e30 2024-03-06T07:22:20.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.200+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141c0ce170 con 0x7f141c0b0e30 2024-03-06T07:22:20.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.201+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f140c01a070 con 0x7f141c0b0e30 2024-03-06T07:22:20.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.201+0000 7f1418ff9640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13fc06b4d0 0x7f13fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.201+0000 7f14127fc640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13fc06b4d0 0x7f13fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.201+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f140c089540 con 0x7f141c0b0e30 2024-03-06T07:22:20.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.202+0000 7f14127fc640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13fc06b4d0 0x7f13fc06d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1404036000 tx=0x7f140403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.206+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f140c057d70 con 0x7f141c0b0e30 2024-03-06T07:22:20.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.376+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f141c068ff0 con 0x7f141c0b0e30 2024-03-06T07:22:20.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.376+0000 7f1418ff9640 1 -- 172.21.15.122:0/3215838156 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f140c05be10 con 0x7f141c0b0e30 2024-03-06T07:22:20.378 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13fc06b4d0 msgr2=0x7f13fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13fc06b4d0 0x7f13fc06d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1404036000 tx=0x7f140403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 msgr2=0x7f141c0b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 0x7f141c0b6930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f140c00c460 tx=0x7f140c00c920 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 shutdown_connections 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13fc06b4d0 0x7f13fc06d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f141c1614d0 0x7f141c152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f141c0d4860 0x7f141c0b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 --2- 172.21.15.122:0/3215838156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f141c0b0e30 0x7f141c0b6930 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 >> 172.21.15.122:0/3215838156 conn(0x7f141c0d3d40 msgr2=0x7f141c06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 shutdown_connections 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.378+0000 7f1421979640 1 -- 172.21.15.122:0/3215838156 wait complete. 2024-03-06T07:22:20.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:20.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.492+0000 7efc00f2b640 1 Processor -- start 2024-03-06T07:22:20.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.492+0000 7efc00f2b640 1 -- start start 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.493+0000 7efc00f2b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15e0e0 0x7efbfc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.493+0000 7efc00f2b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc161bd0 con 0x7efbfc15dd00 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.493+0000 7efc00f2b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc070220 con 0x7efbfc06b6c0 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.493+0000 7efc00f2b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc161e90 con 0x7efbfc15e0e0 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.493+0000 7efbfa59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efbfc15dd00 0x7efbfc06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44022/0 (socket says 172.21.15.122:44022) 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.493+0000 7efbfa59c640 1 -- 172.21.15.122:0/750018765 learned_addr learned my addr 172.21.15.122:0/750018765 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf1d9b640 1 --2- 172.21.15.122:0/750018765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15e0e0 0x7efbfc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389332548 0 0) 0x7efbfc161bd0 con 0x7efbfc15dd00 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe8003520 con 0x7efbfc15dd00 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741271054 0 0) 0x7efbfc070220 con 0x7efbfc06b6c0 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbfc161bd0 con 0x7efbfc06b6c0 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf1d9b640 1 -- 172.21.15.122:0/750018765 >> v1:172.21.15.169:6789/0 conn(0x7efbfc06b6c0 legacy=0x7efbfc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf1d9b640 1 -- 172.21.15.122:0/750018765 >> v1:172.21.15.122:6789/0 conn(0x7efbfc15dd00 legacy=0x7efbfc06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.494+0000 7efbf1d9b640 1 -- 172.21.15.122:0/750018765 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc162170 con 0x7efbfc15e0e0 2024-03-06T07:22:20.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efbf1d9b640 1 --2- 172.21.15.122:0/750018765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15e0e0 0x7efbfc06e620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efbe40097d0 tx=0x7efbe402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf93b48316149c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe4035020 con 0x7efbfc15e0e0 2024-03-06T07:22:20.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe4031890 con 0x7efbfc15e0e0 2024-03-06T07:22:20.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efbf9d9b640 1 -- 172.21.15.122:0/750018765 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe4031c60 con 0x7efbfc15e0e0 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 -- 172.21.15.122:0/750018765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15e0e0 msgr2=0x7efbfc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 --2- 172.21.15.122:0/750018765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15e0e0 0x7efbfc06e620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efbe40097d0 tx=0x7efbe402b930 comp rx=0 tx=0).stop 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 -- 172.21.15.122:0/750018765 shutdown_connections 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 --2- 172.21.15.122:0/750018765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15e0e0 0x7efbfc06e620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 -- 172.21.15.122:0/750018765 >> 172.21.15.122:0/750018765 conn(0x7efbfc0b2660 msgr2=0x7efbfc0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 -- 172.21.15.122:0/750018765 shutdown_connections 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.495+0000 7efc00f2b640 1 -- 172.21.15.122:0/750018765 wait complete. 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 Processor -- start 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 -- start start 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efbfc06b6c0 0x7efbfc0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15dd00 0x7efbfc1e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 0x7efbfc15f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0b0180 con 0x7efbfc1e8900 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0b0000 con 0x7efbfc06b6c0 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.497+0000 7efc00f2b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0b0300 con 0x7efbfc15dd00 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 0x7efbfc15f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 0x7efbfc15f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49802/0 (socket says 172.21.15.122:49802) 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 -- 172.21.15.122:0/148037365 learned_addr learned my addr 172.21.15.122:0/148037365 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfa59c640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efbfc06b6c0 0x7efbfc0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbf1d9b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15dd00 0x7efbfc1e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 -- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15dd00 msgr2=0x7efbfc1e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15dd00 0x7efbfc1e83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 -- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efbfc06b6c0 msgr2=0x7efbfc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efbfc06b6c0 0x7efbfc0adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 -- 172.21.15.122:0/148037365 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc15ffa0 con 0x7efbfc1e8900 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbfad9d640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 0x7efbfc15f520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c6e0 tx=0x7efbf400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4003b70 con 0x7efbfc1e8900 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf4003d10 con 0x7efbfc1e8900 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40052d0 con 0x7efbfc1e8900 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc160280 con 0x7efbfc1e8900 2024-03-06T07:22:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.498+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc1e7ea0 con 0x7efbfc1e8900 2024-03-06T07:22:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.499+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc073390 con 0x7efbfc1e8900 2024-03-06T07:22:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.500+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efbf4005470 con 0x7efbfc1e8900 2024-03-06T07:22:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.500+0000 7efbf37fe640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efbdc06b4d0 0x7efbdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.500+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efbf4089f20 con 0x7efbfc1e8900 2024-03-06T07:22:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.500+0000 7efbfa59c640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efbdc06b4d0 0x7efbdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.500+0000 7efbfa59c640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efbdc06b4d0 0x7efbdc06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efbec0011e0 tx=0x7efbec007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.504+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efbf40587e0 con 0x7efbfc1e8900 2024-03-06T07:22:20.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.675+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efbfc068230 con 0x7efbfc1e8900 2024-03-06T07:22:20.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.676+0000 7efbf37fe640 1 -- 172.21.15.122:0/148037365 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7efbf405c880 con 0x7efbfc1e8900 2024-03-06T07:22:20.678 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.677+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efbdc06b4d0 msgr2=0x7efbdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.677+0000 7efc00f2b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efbdc06b4d0 0x7efbdc06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efbec0011e0 tx=0x7efbec007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.677+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 msgr2=0x7efbfc15f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.677+0000 7efc00f2b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 0x7efbfc15f520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c6e0 tx=0x7efbf400cba0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 shutdown_connections 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efbdc06b4d0 0x7efbdc06d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efbfc1e8900 0x7efbfc15f520 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efbfc15dd00 0x7efbfc1e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 --2- 172.21.15.122:0/148037365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efbfc06b6c0 0x7efbfc0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 >> 172.21.15.122:0/148037365 conn(0x7efbfc0b2660 msgr2=0x7efbfc0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 shutdown_connections 2024-03-06T07:22:20.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.678+0000 7efc00f2b640 1 -- 172.21.15.122:0/148037365 wait complete. 2024-03-06T07:22:20.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:20.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.791+0000 7fe8ae864640 1 Processor -- start 2024-03-06T07:22:20.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.791+0000 7fe8ae864640 1 -- start start 2024-03-06T07:22:20.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ae864640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bb40 0x7fe8a80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ae864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a80cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ae864640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a815ff90 con 0x7fe8a815f7d0 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ae864640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a806c910 con 0x7fe8a815bb40 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ae864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8160270 con 0x7fe8a80cfa90 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe8a815f7d0 0x7fe8a806d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44030/0 (socket says 172.21.15.122:44030) 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a77fe640 1 -- 172.21.15.122:0/1019562654 learned_addr learned my addr 172.21.15.122:0/1019562654 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a7fff640 1 --2- 172.21.15.122:0/1019562654 >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bb40 0x7fe8a80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ace01640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a80cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a6ffd640 1 -- 172.21.15.122:0/1019562654 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033201711 0 0) 0x7fe8a815ff90 con 0x7fe8a815f7d0 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a6ffd640 1 -- 172.21.15.122:0/1019562654 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe890003520 con 0x7fe8a815f7d0 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a6ffd640 1 -- 172.21.15.122:0/1019562654 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3992148256 0 0) 0x7fe890003520 con 0x7fe8a815f7d0 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a7fff640 1 -- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 msgr2=0x7fe8a80cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a7fff640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a80cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a7fff640 1 -- 172.21.15.122:0/1019562654 >> v1:172.21.15.122:6789/0 conn(0x7fe8a815f7d0 legacy=0x7fe8a806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8a7fff640 1 -- 172.21.15.122:0/1019562654 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a8160550 con 0x7fe8a815bb40 2024-03-06T07:22:20.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.792+0000 7fe8ace01640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a80cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.793+0000 7fe8a7fff640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a80b4970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0097a0 tx=0x7fe89c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee6480c856f79a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.793+0000 7fe8a6ffd640 1 -- 172.21.15.122:0/1019562654 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c037070 con 0x7fe8a815bb40 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.793+0000 7fe8a6ffd640 1 -- 172.21.15.122:0/1019562654 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c030510 con 0x7fe8a815bb40 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.793+0000 7fe8a6ffd640 1 -- 172.21.15.122:0/1019562654 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898001670 con 0x7fe8a815bb40 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 -- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 msgr2=0x7fe8a80b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a80b4970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0097a0 tx=0x7fe89c02b790 comp rx=0 tx=0).stop 2024-03-06T07:22:20.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 -- 172.21.15.122:0/1019562654 shutdown_connections 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a80cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 --2- 172.21.15.122:0/1019562654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a80b4970 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 -- 172.21.15.122:0/1019562654 >> 172.21.15.122:0/1019562654 conn(0x7fe8a806b190 msgr2=0x7fe8a806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 -- 172.21.15.122:0/1019562654 shutdown_connections 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.794+0000 7fe8ae864640 1 -- 172.21.15.122:0/1019562654 wait complete. 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.795+0000 7fe8ae864640 1 Processor -- start 2024-03-06T07:22:20.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.795+0000 7fe8ae864640 1 -- start start 2024-03-06T07:22:20.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a8071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a8153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 0x7fe8a8155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806d790 con 0x7fe8a815f7d0 2024-03-06T07:22:20.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a815fc30 con 0x7fe8a815bb40 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a815bf20 con 0x7fe8a80cfa90 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 0x7fe8a8155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 0x7fe8a8155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49814/0 (socket says 172.21.15.122:49814) 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 -- 172.21.15.122:0/348928607 learned_addr learned my addr 172.21.15.122:0/348928607 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8a7fff640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a8071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8a77fe640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a8153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 -- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 msgr2=0x7fe8a8071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a8071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 -- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 msgr2=0x7fe8a8153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a8153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 -- 172.21.15.122:0/348928607 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a8070e80 con 0x7fe8a815f7d0 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ace01640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 0x7fe8a8155310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000efc0 tx=0x7fe8a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0018030 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a000cdf0 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.796+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a8071160 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.797+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0003bb0 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.797+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a8071720 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.798+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a80732b0 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.798+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8a001b070 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.798+0000 7fe8a4ff9640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe89406b500 0x7fe89406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.798+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8a008aa40 con 0x7fe8a815f7d0 2024-03-06T07:22:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.798+0000 7fe8a7fff640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe89406b500 0x7fe89406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.799+0000 7fe8a7fff640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe89406b500 0x7fe89406d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe89c034000 tx=0x7fe89c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.803+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8a0010040 con 0x7fe8a815f7d0 2024-03-06T07:22:20.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.973+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8a8068ff0 con 0x7fe8a815f7d0 2024-03-06T07:22:20.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.974+0000 7fe8a4ff9640 1 -- 172.21.15.122:0/348928607 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8a008ae20 con 0x7fe8a815f7d0 2024-03-06T07:22:20.976 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe89406b500 msgr2=0x7fe89406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe89406b500 0x7fe89406d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe89c034000 tx=0x7fe89c02f000 comp rx=0 tx=0).stop 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 msgr2=0x7fe8a8155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 0x7fe8a8155310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000efc0 tx=0x7fe8a000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 shutdown_connections 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe89406b500 0x7fe89406d9c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a815f7d0 0x7fe8a8155310 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a815bb40 0x7fe8a8153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 --2- 172.21.15.122:0/348928607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a80cfa90 0x7fe8a8071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 >> 172.21.15.122:0/348928607 conn(0x7fe8a806b190 msgr2=0x7fe8a80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 shutdown_connections 2024-03-06T07:22:20.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:20.976+0000 7fe8ae864640 1 -- 172.21.15.122:0/348928607 wait complete. 2024-03-06T07:22:20.989 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:21.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.087+0000 7fd364676640 1 Processor -- start 2024-03-06T07:22:21.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.087+0000 7fd364676640 1 -- start start 2024-03-06T07:22:21.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd364676640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c15dfe0 0x7fd35c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd364676640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd35c0cf830 0x7fd35c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd364676640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd35c0d27f0 0x7fd35c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd364676640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35c0d2c50 con 0x7fd35c0cf830 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd364676640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35c0b31e0 con 0x7fd35c0d27f0 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd364676640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0b3d60 con 0x7fd35c15dfe0 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd361c11640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd35c0cf830 0x7fd35c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd361c11640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd35c0cf830 0x7fd35c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49828/0 (socket says 172.21.15.122:49828) 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd361c11640 1 -- 172.21.15.122:0/1056424770 learned_addr learned my addr 172.21.15.122:0/1056424770 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd362412640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c15dfe0 0x7fd35c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.088+0000 7fd362c13640 1 --2- 172.21.15.122:0/1056424770 >> v2:172.21.15.169:3300/0 conn(0x7fd35c0d27f0 0x7fd35c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd361c11640 1 -- 172.21.15.122:0/1056424770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c15dfe0 msgr2=0x7fd35c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd361c11640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c15dfe0 0x7fd35c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd361c11640 1 -- 172.21.15.122:0/1056424770 >> v2:172.21.15.169:3300/0 conn(0x7fd35c0d27f0 msgr2=0x7fd35c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd361c11640 1 --2- 172.21.15.122:0/1056424770 >> v2:172.21.15.169:3300/0 conn(0x7fd35c0d27f0 0x7fd35c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd361c11640 1 -- 172.21.15.122:0/1056424770 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c0b3430 con 0x7fd35c0cf830 2024-03-06T07:22:21.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd362412640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c15dfe0 0x7fd35c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:21.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.089+0000 7fd361c11640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0cf830 0x7fd35c0d22b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd35c067ad0 tx=0x7fd34c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def9d925ce5aac31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd361410640 1 -- 172.21.15.122:0/1056424770 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c035020 con 0x7fd35c0cf830 2024-03-06T07:22:21.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd361410640 1 -- 172.21.15.122:0/1056424770 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c031870 con 0x7fd35c0cf830 2024-03-06T07:22:21.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd361410640 1 -- 172.21.15.122:0/1056424770 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c031c40 con 0x7fd35c0cf830 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 -- 172.21.15.122:0/1056424770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0cf830 msgr2=0x7fd35c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0cf830 0x7fd35c0d22b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd35c067ad0 tx=0x7fd34c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 -- 172.21.15.122:0/1056424770 shutdown_connections 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0cf830 0x7fd35c0d22b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 --2- 172.21.15.122:0/1056424770 >> v2:172.21.15.169:3300/0 conn(0x7fd35c0d27f0 0x7fd35c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 --2- 172.21.15.122:0/1056424770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c15dfe0 0x7fd35c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 -- 172.21.15.122:0/1056424770 >> 172.21.15.122:0/1056424770 conn(0x7fd35c0d3cd0 msgr2=0x7fd35c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 -- 172.21.15.122:0/1056424770 shutdown_connections 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.090+0000 7fd364676640 1 -- 172.21.15.122:0/1056424770 wait complete. 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.091+0000 7fd364676640 1 Processor -- start 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.091+0000 7fd364676640 1 -- start start 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd364676640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c0cf830 0x7fd35c158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd364676640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 0x7fd35c15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd364676640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd35c15dfe0 0x7fd35c0b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd364676640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0d1c70 con 0x7fd35c0d27f0 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd364676640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0b39f0 con 0x7fd35c15dfe0 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd364676640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0d1df0 con 0x7fd35c0cf830 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 0x7fd35c15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 0x7fd35c15ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49842/0 (socket says 172.21.15.122:49842) 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 -- 172.21.15.122:0/2575806195 learned_addr learned my addr 172.21.15.122:0/2575806195 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd362412640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c0cf830 0x7fd35c158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd362c13640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd35c15dfe0 0x7fd35c0b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 -- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c0cf830 msgr2=0x7fd35c158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c0cf830 0x7fd35c158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 -- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd35c15dfe0 msgr2=0x7fd35c0b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd35c15dfe0 0x7fd35c0b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.092+0000 7fd361c11640 1 -- 172.21.15.122:0/2575806195 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c158aa0 con 0x7fd35c0d27f0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.093+0000 7fd361c11640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 0x7fd35c15ee10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd34c02b950 tx=0x7fd34c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.093+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c040070 con 0x7fd35c0d27f0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.093+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c031e10 con 0x7fd35c0d27f0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.093+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c15f350 con 0x7fd35c0d27f0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.093+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c035040 con 0x7fd35c0d27f0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.093+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c0b7040 con 0x7fd35c0d27f0 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.094+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c0b5200 con 0x7fd35c0d27f0 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.094+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd34c04b060 con 0x7fd35c0d27f0 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.095+0000 7fd352ffd640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd34006b4d0 0x7fd34006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.095+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd34c0abd20 con 0x7fd35c0d27f0 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.095+0000 7fd362412640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd34006b4d0 0x7fd34006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.095+0000 7fd362412640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd34006b4d0 0x7fd34006d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0d0800 tx=0x7fd35800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.099+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd35c0b5200 con 0x7fd35c0d27f0 2024-03-06T07:22:21.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.271+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd35c04f330 con 0x7fd35c0d27f0 2024-03-06T07:22:21.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.272+0000 7fd352ffd640 1 -- 172.21.15.122:0/2575806195 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7fd34c07a550 con 0x7fd35c0d27f0 2024-03-06T07:22:21.274 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:21.274 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd34006b4d0 msgr2=0x7fd34006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd34006b4d0 0x7fd34006d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0d0800 tx=0x7fd35800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 msgr2=0x7fd35c15ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 0x7fd35c15ee10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd34c02b950 tx=0x7fd34c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 shutdown_connections 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd34006b4d0 0x7fd34006d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd35c15dfe0 0x7fd35c0b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd35c0d27f0 0x7fd35c15ee10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 --2- 172.21.15.122:0/2575806195 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd35c0cf830 0x7fd35c158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 >> 172.21.15.122:0/2575806195 conn(0x7fd35c0d3cd0 msgr2=0x7fd35c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 shutdown_connections 2024-03-06T07:22:21.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.274+0000 7fd364676640 1 -- 172.21.15.122:0/2575806195 wait complete. 2024-03-06T07:22:21.287 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:21.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:21.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.390+0000 7fc469f35640 1 Processor -- start 2024-03-06T07:22:21.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.390+0000 7fc469f35640 1 -- start start 2024-03-06T07:22:21.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.390+0000 7fc469f35640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc46415f4f0 0x7fc464066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.390+0000 7fc469f35640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46406b050 con 0x7fc4640ce750 2024-03-06T07:22:21.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.390+0000 7fc469f35640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46406b2c0 con 0x7fc4640b0db0 2024-03-06T07:22:21.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.390+0000 7fc469f35640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4640b0890 con 0x7fc46415f4f0 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc463fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc4640ce750 0x7fc4640b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44046/0 (socket says 172.21.15.122:44046) 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc463fff640 1 -- 172.21.15.122:0/1115042140 learned_addr learned my addr 172.21.15.122:0/1115042140 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc4637fe640 1 --2- 172.21.15.122:0/1115042140 >> v2:172.21.15.169:3300/0 conn(0x7fc46415f4f0 0x7fc464066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc4627fc640 1 -- 172.21.15.122:0/1115042140 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602226580 0 0) 0x7fc46406b050 con 0x7fc4640ce750 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc4627fc640 1 -- 172.21.15.122:0/1115042140 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c003520 con 0x7fc4640ce750 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc4637fe640 1 -- 172.21.15.122:0/1115042140 >> v1:172.21.15.169:6789/0 conn(0x7fc4640b0db0 legacy=0x7fc4640ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc4637fe640 1 -- 172.21.15.122:0/1115042140 >> v1:172.21.15.122:6789/0 conn(0x7fc4640ce750 legacy=0x7fc4640b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.391+0000 7fc4637fe640 1 -- 172.21.15.122:0/1115042140 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4640672e0 con 0x7fc46415f4f0 2024-03-06T07:22:21.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc4637fe640 1 --2- 172.21.15.122:0/1115042140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46415f4f0 0x7fc464066d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc4540097a0 tx=0x7fc45402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afe88c92592a5b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc4627fc640 1 -- 172.21.15.122:0/1115042140 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454034070 con 0x7fc46415f4f0 2024-03-06T07:22:21.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc4627fc640 1 -- 172.21.15.122:0/1115042140 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc454030610 con 0x7fc46415f4f0 2024-03-06T07:22:21.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc4627fc640 1 -- 172.21.15.122:0/1115042140 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4540309e0 con 0x7fc46415f4f0 2024-03-06T07:22:21.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc469f35640 1 -- 172.21.15.122:0/1115042140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46415f4f0 msgr2=0x7fc464066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc469f35640 1 --2- 172.21.15.122:0/1115042140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46415f4f0 0x7fc464066d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc4540097a0 tx=0x7fc45402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:21.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc469f35640 1 -- 172.21.15.122:0/1115042140 shutdown_connections 2024-03-06T07:22:21.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc469f35640 1 --2- 172.21.15.122:0/1115042140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46415f4f0 0x7fc464066d00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc469f35640 1 -- 172.21.15.122:0/1115042140 >> 172.21.15.122:0/1115042140 conn(0x7fc4640b1d70 msgr2=0x7fc4640b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.392+0000 7fc469f35640 1 -- 172.21.15.122:0/1115042140 shutdown_connections 2024-03-06T07:22:21.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.393+0000 7fc469f35640 1 -- 172.21.15.122:0/1115042140 wait complete. 2024-03-06T07:22:21.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.394+0000 7fc469f35640 1 Processor -- start 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.394+0000 7fc469f35640 1 -- start start 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.394+0000 7fc469f35640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4640b0db0 0x7fc464099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.394+0000 7fc469f35640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc4640ce750 0x7fc464144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.394+0000 7fc469f35640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 0x7fc464145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc469f35640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46415f8d0 con 0x7fc46415f4f0 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc469f35640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46406b5a0 con 0x7fc4640b0db0 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc469f35640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640cfcb0 con 0x7fc4640ce750 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 0x7fc464145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 0x7fc464145280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49858/0 (socket says 172.21.15.122:49858) 2024-03-06T07:22:21.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 -- 172.21.15.122:0/1980210594 learned_addr learned my addr 172.21.15.122:0/1980210594 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc462ffd640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc4640ce750 0x7fc464144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc4637fe640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4640b0db0 0x7fc464099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 -- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc4640ce750 msgr2=0x7fc464144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc4640ce750 0x7fc464144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 -- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4640b0db0 msgr2=0x7fc464099890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4640b0db0 0x7fc464099890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 -- 172.21.15.122:0/1980210594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc464145900 con 0x7fc46415f4f0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc462ffd640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc4640ce750 0x7fc464144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc463fff640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 0x7fc464145280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00cb40 tx=0x7fc45c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c017070 con 0x7fc46415f4f0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c00aca0 con 0x7fc46415f4f0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c012620 con 0x7fc46415f4f0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc464145be0 con 0x7fc46415f4f0 2024-03-06T07:22:21.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.395+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4641461a0 con 0x7fc46415f4f0 2024-03-06T07:22:21.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.396+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46409e3d0 con 0x7fc46415f4f0 2024-03-06T07:22:21.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.397+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc45c01a070 con 0x7fc46415f4f0 2024-03-06T07:22:21.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.397+0000 7fc44bfff640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.397+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc45c089f50 con 0x7fc46415f4f0 2024-03-06T07:22:21.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.397+0000 7fc4637fe640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.397+0000 7fc4637fe640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45006b4d0 0x7fc45006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc454036000 tx=0x7fc45403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.401+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc45c058780 con 0x7fc46415f4f0 2024-03-06T07:22:21.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.571+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc46406cd40 con 0x7fc46415f4f0 2024-03-06T07:22:21.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.572+0000 7fc44bfff640 1 -- 172.21.15.122:0/1980210594 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc45c05c820 con 0x7fc46415f4f0 2024-03-06T07:22:21.574 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:21.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45006b4d0 msgr2=0x7fc45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45006b4d0 0x7fc45006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc454036000 tx=0x7fc45403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 msgr2=0x7fc464145280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 0x7fc464145280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00cb40 tx=0x7fc45c00a060 comp rx=0 tx=0).stop 2024-03-06T07:22:21.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 shutdown_connections 2024-03-06T07:22:21.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc46415f4f0 0x7fc464145280 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc4640ce750 0x7fc464144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 --2- 172.21.15.122:0/1980210594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4640b0db0 0x7fc464099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 >> 172.21.15.122:0/1980210594 conn(0x7fc4640b1d70 msgr2=0x7fc464072d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 shutdown_connections 2024-03-06T07:22:21.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.574+0000 7fc469f35640 1 -- 172.21.15.122:0/1980210594 wait complete. 2024-03-06T07:22:21.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:21.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.689+0000 7f5e739ac640 1 Processor -- start 2024-03-06T07:22:21.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.689+0000 7f5e739ac640 1 -- start start 2024-03-06T07:22:21.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e739ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e6c1455d0 0x7f5e6c072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e739ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06afb0 0x7f5e6c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e739ac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c1459b0 con 0x7f5e6c06ded0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e739ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c0cfdf0 con 0x7f5e6c1455d0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e739ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c06b930 con 0x7f5e6c06afb0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e71748640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e6c1455d0 0x7f5e6c072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e71748640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e6c1455d0 0x7f5e6c072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49860/0 (socket says 172.21.15.122:49860) 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.690+0000 7f5e71748640 1 -- 172.21.15.122:0/1755416441 learned_addr learned my addr 172.21.15.122:0/1755416441 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.691+0000 7f5e70f47640 1 --2- 172.21.15.122:0/1755416441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06afb0 0x7f5e6c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.691+0000 7f5e71748640 1 -- 172.21.15.122:0/1755416441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06afb0 msgr2=0x7f5e6c06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.691+0000 7f5e71748640 1 --2- 172.21.15.122:0/1755416441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06afb0 0x7f5e6c06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.691+0000 7f5e71748640 1 -- 172.21.15.122:0/1755416441 >> v1:172.21.15.169:6789/0 conn(0x7f5e6c06ded0 legacy=0x7f5e6c06e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.691+0000 7f5e71748640 1 -- 172.21.15.122:0/1755416441 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0cf1e0 con 0x7f5e6c1455d0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.691+0000 7f5e71748640 1 --2- 172.21.15.122:0/1755416441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c072480 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009870 tx=0x7f5e6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbaa3d0cfcb855 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e63fff640 1 -- 172.21.15.122:0/1755416441 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68034070 con 0x7f5e6c1455d0 2024-03-06T07:22:21.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e63fff640 1 -- 172.21.15.122:0/1755416441 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68030840 con 0x7f5e6c1455d0 2024-03-06T07:22:21.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e63fff640 1 -- 172.21.15.122:0/1755416441 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68030c10 con 0x7f5e6c1455d0 2024-03-06T07:22:21.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 -- 172.21.15.122:0/1755416441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c1455d0 msgr2=0x7f5e6c072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 --2- 172.21.15.122:0/1755416441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c072480 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009870 tx=0x7f5e6802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:21.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 -- 172.21.15.122:0/1755416441 shutdown_connections 2024-03-06T07:22:21.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 --2- 172.21.15.122:0/1755416441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06afb0 0x7f5e6c06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 --2- 172.21.15.122:0/1755416441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c072480 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 -- 172.21.15.122:0/1755416441 >> 172.21.15.122:0/1755416441 conn(0x7f5e6c0b2660 msgr2=0x7f5e6c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 -- 172.21.15.122:0/1755416441 shutdown_connections 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.692+0000 7f5e739ac640 1 -- 172.21.15.122:0/1755416441 wait complete. 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.693+0000 7f5e739ac640 1 Processor -- start 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.693+0000 7f5e739ac640 1 -- start start 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e739ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 0x7f5e6c13c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e739ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06ded0 0x7f5e6c1f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e739ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c13cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e739ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0cfa80 con 0x7f5e6c06afb0 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e739ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0cf7a0 con 0x7f5e6c1455d0 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e739ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0729c0 con 0x7f5e6c06ded0 2024-03-06T07:22:21.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 0x7f5e6c13c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 0x7f5e6c13c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49874/0 (socket says 172.21.15.122:49874) 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 -- 172.21.15.122:0/3576407134 learned_addr learned my addr 172.21.15.122:0/3576407134 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71f49640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c13cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e70f47640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06ded0 0x7f5e6c1f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 -- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06ded0 msgr2=0x7f5e6c1f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06ded0 0x7f5e6c1f3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 -- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e6c1455d0 msgr2=0x7f5e6c13cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c13cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.694+0000 7f5e71748640 1 -- 172.21.15.122:0/3576407134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c1f3d90 con 0x7f5e6c06afb0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.695+0000 7f5e71748640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 0x7f5e6c13c200 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.695+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68034040 con 0x7f5e6c06afb0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.695+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68039d90 con 0x7f5e6c06afb0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.695+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c154b90 con 0x7f5e6c06afb0 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.695+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68043630 con 0x7f5e6c06afb0 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.695+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c0b4d10 con 0x7f5e6c06afb0 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.696+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c073390 con 0x7f5e6c06afb0 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.696+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e6804a060 con 0x7f5e6c06afb0 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.697+0000 7f5e61ffb640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e5006b4d0 0x7f5e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.697+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e680ac070 con 0x7f5e6c06afb0 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.697+0000 7f5e70f47640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e5006b4d0 0x7f5e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.697+0000 7f5e70f47640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e5006b4d0 0x7f5e5006d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0b5750 tx=0x7f5e5c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.701+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e6803e070 con 0x7f5e6c06afb0 2024-03-06T07:22:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.872+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e6c15bdc0 con 0x7f5e6c06afb0 2024-03-06T07:22:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.872+0000 7f5e61ffb640 1 -- 172.21.15.122:0/3576407134 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e6807a8a0 con 0x7f5e6c06afb0 2024-03-06T07:22:21.875 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:21.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e5006b4d0 msgr2=0x7f5e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e5006b4d0 0x7f5e5006d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0b5750 tx=0x7f5e5c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:21.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 msgr2=0x7f5e6c13c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 0x7f5e6c13c200 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e680046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 shutdown_connections 2024-03-06T07:22:21.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e5006b4d0 0x7f5e5006d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e6c1455d0 0x7f5e6c13cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e6c06ded0 0x7f5e6c1f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 --2- 172.21.15.122:0/3576407134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e6c06afb0 0x7f5e6c13c200 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 >> 172.21.15.122:0/3576407134 conn(0x7f5e6c0b2660 msgr2=0x7f5e6c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 shutdown_connections 2024-03-06T07:22:21.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.874+0000 7f5e739ac640 1 -- 172.21.15.122:0/3576407134 wait complete. 2024-03-06T07:22:21.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.988+0000 7f77d5a33640 1 Processor -- start 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.988+0000 7f77d5a33640 1 -- start start 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77d5a33640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f77d01614d0 0x7f77d00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77d5a33640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d1910 0x7f77d00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77d5a33640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 0x7f77d00d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77d5a33640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d00d2740 con 0x7f77d00d22b0 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77d5a33640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d01618b0 con 0x7f77d01614d0 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77d5a33640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00cf400 con 0x7f77d00d1910 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77cf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 0x7f77d00d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f77d01614d0 0x7f77d00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f77d01614d0 0x7f77d00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:37732/0 (socket says 172.21.15.122:37732) 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77cf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 0x7f77d00d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49886/0 (socket says 172.21.15.122:49886) 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 -- 172.21.15.122:0/3278093481 learned_addr learned my addr 172.21.15.122:0/3278093481 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77c7fff640 1 --2- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d1910 0x7f77d00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 -- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d1910 msgr2=0x7f77d00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 --2- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d1910 0x7f77d00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 -- 172.21.15.122:0/3278093481 >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 msgr2=0x7f77d00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 --2- 172.21.15.122:0/3278093481 >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 0x7f77d00d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77ceffd640 1 -- 172.21.15.122:0/3278093481 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d00cf6a0 con 0x7f77d01614d0 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.989+0000 7f77cf7fe640 1 --2- 172.21.15.122:0/3278093481 >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 0x7f77d00d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.990+0000 7f77ceffd640 1 --2- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00d4860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f77d008c470 tx=0x7f77c002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=796cbdbebe895b02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.990+0000 7f77ce7fc640 1 -- 172.21.15.122:0/3278093481 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0035020 con 0x7f77d01614d0 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.990+0000 7f77ce7fc640 1 -- 172.21.15.122:0/3278093481 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c0031870 con 0x7f77d01614d0 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.990+0000 7f77ce7fc640 1 -- 172.21.15.122:0/3278093481 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0031c40 con 0x7f77d01614d0 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 -- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 msgr2=0x7f77d00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00d4860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f77d008c470 tx=0x7f77c002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 -- 172.21.15.122:0/3278093481 shutdown_connections 2024-03-06T07:22:21.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3278093481 >> v2:172.21.15.122:3300/0 conn(0x7f77d00d22b0 0x7f77d00d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d1910 0x7f77d00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3278093481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00d4860 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 -- 172.21.15.122:0/3278093481 >> 172.21.15.122:0/3278093481 conn(0x7f77d00d3d40 msgr2=0x7f77d00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 -- 172.21.15.122:0/3278093481 shutdown_connections 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.991+0000 7f77d5a33640 1 -- 172.21.15.122:0/3278093481 wait complete. 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.992+0000 7f77d5a33640 1 Processor -- start 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.992+0000 7f77d5a33640 1 -- start start 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 0x7f77d0155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d22b0 0x7f77d00b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00b0910 con 0x7f77d00d1910 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00cfc60 con 0x7f77d01614d0 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00d0be0 con 0x7f77d00d22b0 2024-03-06T07:22:21.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 0x7f77d0155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 0x7f77d0155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49890/0 (socket says 172.21.15.122:49890) 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 -- 172.21.15.122:0/3451448131 learned_addr learned my addr 172.21.15.122:0/3451448131 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77cf7fe640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77c7fff640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d22b0 0x7f77d00b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 -- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d22b0 msgr2=0x7f77d00b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d22b0 0x7f77d00b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 -- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 msgr2=0x7f77d00b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 -- 172.21.15.122:0/3451448131 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d01558c0 con 0x7f77d00d1910 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ceffd640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 0x7f77d0155380 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f77c002b950 tx=0x7f77c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0040070 con 0x7f77d00d1910 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c0031e10 con 0x7f77d00d1910 2024-03-06T07:22:21.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0035040 con 0x7f77d00d1910 2024-03-06T07:22:21.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d00b48e0 con 0x7f77d00d1910 2024-03-06T07:22:21.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.993+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d00b4e90 con 0x7f77d00d1910 2024-03-06T07:22:21.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.994+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d0068ff0 con 0x7f77d00d1910 2024-03-06T07:22:22.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.995+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77c0038020 con 0x7f77d00d1910 2024-03-06T07:22:22.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.995+0000 7f77ccff9640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77b006b4d0 0x7f77b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.995+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77c00abd00 con 0x7f77d00d1910 2024-03-06T07:22:22.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.995+0000 7f77c7fff640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77b006b4d0 0x7f77b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.996+0000 7f77c7fff640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77b006b4d0 0x7f77b006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f77d01f4e60 tx=0x7f77b8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:21.999+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77c007a530 con 0x7f77d00d1910 2024-03-06T07:22:22.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.169+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f77d00b2200 con 0x7f77d00d1910 2024-03-06T07:22:22.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.169+0000 7f77ccff9640 1 -- 172.21.15.122:0/3451448131 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f77c007e5d0 con 0x7f77d00d1910 2024-03-06T07:22:22.172 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:22.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77b006b4d0 msgr2=0x7f77b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77b006b4d0 0x7f77b006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f77d01f4e60 tx=0x7f77b8009190 comp rx=0 tx=0).stop 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 msgr2=0x7f77d0155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 0x7f77d0155380 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f77c002b950 tx=0x7f77c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 shutdown_connections 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f77b006b4d0 0x7f77b006d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d01614d0 0x7f77d00b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f77d00d22b0 0x7f77d00b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 --2- 172.21.15.122:0/3451448131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77d00d1910 0x7f77d0155380 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 >> 172.21.15.122:0/3451448131 conn(0x7f77d00d3d40 msgr2=0x7f77d00d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 shutdown_connections 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.171+0000 7f77d5a33640 1 -- 172.21.15.122:0/3451448131 wait complete. 2024-03-06T07:22:22.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 Processor -- start 2024-03-06T07:22:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 -- start start 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7ed00d00c0 0x7f7ed00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7ed006cf20 0x7f7ed006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed00d09e0 con 0x7f7ed00cf850 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed006ca90 con 0x7f7ed006cf20 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.284+0000 7f7ed5f01640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed015e380 con 0x7f7ed00d00c0 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ecffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7ed006cf20 0x7f7ed006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7eceffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7ed00cf850 0x7f7ed00cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44062/0 (socket says 172.21.15.122:44062) 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7eceffd640 1 -- 172.21.15.122:0/3906185502 learned_addr learned my addr 172.21.15.122:0/3906185502 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ecf7fe640 1 --2- 172.21.15.122:0/3906185502 >> v2:172.21.15.169:3300/0 conn(0x7f7ed00d00c0 0x7f7ed00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ece7fc640 1 -- 172.21.15.122:0/3906185502 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329973128 0 0) 0x7f7ed00d09e0 con 0x7f7ed00cf850 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ece7fc640 1 -- 172.21.15.122:0/3906185502 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb8003520 con 0x7f7ed00cf850 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ecffff640 1 -- 172.21.15.122:0/3906185502 >> v2:172.21.15.169:3300/0 conn(0x7f7ed00d00c0 msgr2=0x7f7ed00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ecffff640 1 --2- 172.21.15.122:0/3906185502 >> v2:172.21.15.169:3300/0 conn(0x7f7ed00d00c0 0x7f7ed00d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ecffff640 1 -- 172.21.15.122:0/3906185502 >> v1:172.21.15.122:6789/0 conn(0x7f7ed00cf850 legacy=0x7f7ed00cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.285+0000 7f7ecffff640 1 -- 172.21.15.122:0/3906185502 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed00cdc10 con 0x7f7ed006cf20 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ecffff640 1 --2- 172.21.15.122:0/3906185502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed006cf20 0x7f7ed006d3b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00d1a40 tx=0x7f7ec802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0916a06d3777e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ece7fc640 1 -- 172.21.15.122:0/3906185502 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8039070 con 0x7f7ed006cf20 2024-03-06T07:22:22.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ece7fc640 1 -- 172.21.15.122:0/3906185502 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec802cc80 con 0x7f7ed006cf20 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ece7fc640 1 -- 172.21.15.122:0/3906185502 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8034660 con 0x7f7ed006cf20 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 -- 172.21.15.122:0/3906185502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed006cf20 msgr2=0x7f7ed006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/3906185502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed006cf20 0x7f7ed006d3b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00d1a40 tx=0x7f7ec802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 -- 172.21.15.122:0/3906185502 shutdown_connections 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/3906185502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed006cf20 0x7f7ed006d3b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/3906185502 >> v2:172.21.15.169:3300/0 conn(0x7f7ed00d00c0 0x7f7ed00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 -- 172.21.15.122:0/3906185502 >> 172.21.15.122:0/3906185502 conn(0x7f7ed00b35a0 msgr2=0x7f7ed00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 -- 172.21.15.122:0/3906185502 shutdown_connections 2024-03-06T07:22:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.286+0000 7f7ed5f01640 1 -- 172.21.15.122:0/3906185502 wait complete. 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.287+0000 7f7ed5f01640 1 Processor -- start 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 -- start start 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ed006cf20 0x7f7ed0153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ed00cf850 0x7f7ed00c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 0x7f7ed00cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed006c720 con 0x7f7ed00d00c0 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed006c440 con 0x7f7ed006cf20 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ed5f01640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00b1030 con 0x7f7ed00cf850 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ecffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 0x7f7ed00cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ecf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ed006cf20 0x7f7ed0153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ecffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 0x7f7ed00cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49902/0 (socket says 172.21.15.122:49902) 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7ecffff640 1 -- 172.21.15.122:0/1605104508 learned_addr learned my addr 172.21.15.122:0/1605104508 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:22.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.288+0000 7f7eceffd640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ed00cf850 0x7f7ed00c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ecffff640 1 -- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ed00cf850 msgr2=0x7f7ed00c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ecffff640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ed00cf850 0x7f7ed00c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ecffff640 1 -- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ed006cf20 msgr2=0x7f7ed0153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ecffff640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ed006cf20 0x7f7ed0153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ecffff640 1 -- 172.21.15.122:0/1605104508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed0154450 con 0x7f7ed00d00c0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ecffff640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 0x7f7ed00cab50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8037040 tx=0x7f7ec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8044020 con 0x7f7ed00d00c0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec803f470 con 0x7f7ed00d00c0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed00c88a0 con 0x7f7ed00d00c0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8039040 con 0x7f7ed00d00c0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.289+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed00c5bd0 con 0x7f7ed00d00c0 2024-03-06T07:22:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.290+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed004f330 con 0x7f7ed00d00c0 2024-03-06T07:22:22.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.290+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ec8047020 con 0x7f7ed00d00c0 2024-03-06T07:22:22.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.291+0000 7f7eb7fff640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ebc06b4d0 0x7f7ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.291+0000 7f7ecf7fe640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ebc06b4d0 0x7f7ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.291+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ec80aca80 con 0x7f7ed00d00c0 2024-03-06T07:22:22.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.291+0000 7f7ecf7fe640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ebc06b4d0 0x7f7ebc06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009b40 tx=0x7f7ec0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.295+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ec804e030 con 0x7f7ed00d00c0 2024-03-06T07:22:22.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.468+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ed00b2160 con 0x7f7ed00d00c0 2024-03-06T07:22:22.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.469+0000 7f7eb7fff640 1 -- 172.21.15.122:0/1605104508 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f7ec807b340 con 0x7f7ed00d00c0 2024-03-06T07:22:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:22.472 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:22.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ebc06b4d0 msgr2=0x7f7ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ebc06b4d0 0x7f7ebc06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009b40 tx=0x7f7ec0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 msgr2=0x7f7ed00cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 0x7f7ed00cab50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8037040 tx=0x7f7ec8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 shutdown_connections 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ebc06b4d0 0x7f7ebc06d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ed00d00c0 0x7f7ed00cab50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ed00cf850 0x7f7ed00c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 --2- 172.21.15.122:0/1605104508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ed006cf20 0x7f7ed0153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 >> 172.21.15.122:0/1605104508 conn(0x7f7ed00b35a0 msgr2=0x7f7ed00b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 shutdown_connections 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.471+0000 7f7ed5f01640 1 -- 172.21.15.122:0/1605104508 wait complete. 2024-03-06T07:22:22.484 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:22.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.586+0000 7ffa89dbc640 1 Processor -- start 2024-03-06T07:22:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.586+0000 7ffa89dbc640 1 -- start start 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa89dbc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ffa8415a220 0x7ffa840cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa89dbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa840712d0 0x7ffa84071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa89dbc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa8415de60 0x7ffa8415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa89dbc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa840cf610 con 0x7ffa8415de60 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa89dbc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa84071c70 con 0x7ffa8415a220 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa89dbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840b3320 con 0x7ffa840712d0 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa8415de60 0x7ffa8415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa8415de60 0x7ffa8415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49906/0 (socket says 172.21.15.122:49906) 2024-03-06T07:22:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 -- 172.21.15.122:0/3605255257 learned_addr learned my addr 172.21.15.122:0/3605255257 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa82ffd640 1 --2- 172.21.15.122:0/3605255257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa840712d0 0x7ffa84071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa837fe640 1 --2- 172.21.15.122:0/3605255257 >> v2:172.21.15.169:3300/0 conn(0x7ffa8415a220 0x7ffa840cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 -- 172.21.15.122:0/3605255257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa840712d0 msgr2=0x7ffa84071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 --2- 172.21.15.122:0/3605255257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa840712d0 0x7ffa84071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 -- 172.21.15.122:0/3605255257 >> v2:172.21.15.169:3300/0 conn(0x7ffa8415a220 msgr2=0x7ffa840cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 --2- 172.21.15.122:0/3605255257 >> v2:172.21.15.169:3300/0 conn(0x7ffa8415a220 0x7ffa840cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.587+0000 7ffa83fff640 1 -- 172.21.15.122:0/3605255257 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa840728d0 con 0x7ffa8415de60 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.588+0000 7ffa83fff640 1 --2- 172.21.15.122:0/3605255257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa8415de60 0x7ffa8415e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa8406ebb0 tx=0x7ffa7c02e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4f7f4693215c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.588+0000 7ffa827fc640 1 -- 172.21.15.122:0/3605255257 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c02ea00 con 0x7ffa8415de60 2024-03-06T07:22:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.588+0000 7ffa827fc640 1 -- 172.21.15.122:0/3605255257 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c02eba0 con 0x7ffa8415de60 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.588+0000 7ffa827fc640 1 -- 172.21.15.122:0/3605255257 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c009600 con 0x7ffa8415de60 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 -- 172.21.15.122:0/3605255257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa8415de60 msgr2=0x7ffa8415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/3605255257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa8415de60 0x7ffa8415e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa8406ebb0 tx=0x7ffa7c02e060 comp rx=0 tx=0).stop 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 -- 172.21.15.122:0/3605255257 shutdown_connections 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/3605255257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa8415de60 0x7ffa8415e2f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/3605255257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa840712d0 0x7ffa84071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/3605255257 >> v2:172.21.15.169:3300/0 conn(0x7ffa8415a220 0x7ffa840cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 -- 172.21.15.122:0/3605255257 >> 172.21.15.122:0/3605255257 conn(0x7ffa8406b190 msgr2=0x7ffa8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 -- 172.21.15.122:0/3605255257 shutdown_connections 2024-03-06T07:22:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.589+0000 7ffa89dbc640 1 -- 172.21.15.122:0/3605255257 wait complete. 2024-03-06T07:22:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.590+0000 7ffa89dbc640 1 Processor -- start 2024-03-06T07:22:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.590+0000 7ffa89dbc640 1 -- start start 2024-03-06T07:22:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 0x7ffa84159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa8415a220 0x7ffa840ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa8415de60 0x7ffa841551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84072030 con 0x7ffa840712d0 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84072bb0 con 0x7ffa8415de60 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84072530 con 0x7ffa8415a220 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 0x7ffa84159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 0x7ffa84159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49914/0 (socket says 172.21.15.122:49914) 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 -- 172.21.15.122:0/1608816972 learned_addr learned my addr 172.21.15.122:0/1608816972 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa82ffd640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa8415a220 0x7ffa840ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa83fff640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa8415de60 0x7ffa841551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 -- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa8415a220 msgr2=0x7ffa840ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa8415a220 0x7ffa840ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 -- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa8415de60 msgr2=0x7ffa841551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa8415de60 0x7ffa841551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 -- 172.21.15.122:0/1608816972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa840ad870 con 0x7ffa840712d0 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa837fe640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 0x7ffa84159310 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400e9b0 tx=0x7ffa7400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74017070 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7400c9a0 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.591+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa840ad030 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.592+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7400cd90 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.592+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8415bdd0 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.592+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa840732b0 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.593+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa7401a070 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.593+0000 7ffa6bfff640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa7006b4d0 0x7ffa7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.593+0000 7ffa82ffd640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa7006b4d0 0x7ffa7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.593+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa7408a190 con 0x7ffa840712d0 2024-03-06T07:22:22.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.594+0000 7ffa82ffd640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa7006b4d0 0x7ffa7006d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa84071b00 tx=0x7ffa78009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.598+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa840ad030 con 0x7ffa840712d0 2024-03-06T07:22:22.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.772+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa84068ff0 con 0x7ffa840712d0 2024-03-06T07:22:22.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.772+0000 7ffa6bfff640 1 -- 172.21.15.122:0/1608816972 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa740589c0 con 0x7ffa840712d0 2024-03-06T07:22:22.774 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:22.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa7006b4d0 msgr2=0x7ffa7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa7006b4d0 0x7ffa7006d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa84071b00 tx=0x7ffa78009190 comp rx=0 tx=0).stop 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 msgr2=0x7ffa84159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 0x7ffa84159310 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400e9b0 tx=0x7ffa7400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 shutdown_connections 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa7006b4d0 0x7ffa7006d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa8415de60 0x7ffa841551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa8415a220 0x7ffa840ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 --2- 172.21.15.122:0/1608816972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa840712d0 0x7ffa84159310 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 >> 172.21.15.122:0/1608816972 conn(0x7ffa8406b190 msgr2=0x7ffa840ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 shutdown_connections 2024-03-06T07:22:22.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.774+0000 7ffa89dbc640 1 -- 172.21.15.122:0/1608816972 wait complete. 2024-03-06T07:22:22.787 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:22.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.887+0000 7f380e688640 1 Processor -- start 2024-03-06T07:22:22.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.887+0000 7f380e688640 1 -- start start 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380e688640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f380815def0 0x7f38080b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380e688640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 0x7f38080ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380e688640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380806d030 con 0x7f38080ce880 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380e688640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38080cecf0 con 0x7f380815def0 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380e688640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38080cf1c0 con 0x7f380806cab0 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f38077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 0x7f38080ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f3807fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f380815def0 0x7f38080b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380cc25640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f38080ce880 0x7f38080cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:41958/0 (socket says 172.21.15.122:41958) 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f38077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 0x7f38080ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44498/0 (socket says 172.21.15.122:44498) 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f380cc25640 1 -- 172.21.15.122:0/1558213126 learned_addr learned my addr 172.21.15.122:0/1558213126 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:22.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.888+0000 7f3807fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f380815def0 0x7f38080b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:37760/0 (socket says 172.21.15.122:37760) 2024-03-06T07:22:22.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.889+0000 7f3807fff640 1 -- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 msgr2=0x7f38080ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.889+0000 7f3807fff640 1 --2- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 0x7f38080ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.889+0000 7f3807fff640 1 -- 172.21.15.122:0/1558213126 >> v1:172.21.15.169:6789/0 conn(0x7f38080ce880 legacy=0x7f38080cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.889+0000 7f3807fff640 1 -- 172.21.15.122:0/1558213126 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380806d310 con 0x7f380815def0 2024-03-06T07:22:22.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.889+0000 7f38077fe640 1 --2- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 0x7f38080ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:22.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.889+0000 7f3807fff640 1 --2- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380815def0 0x7f38080b41b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f37f80097a0 tx=0x7f37f802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8618ead0035ea538 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f3806ffd640 1 -- 172.21.15.122:0/1558213126 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8034070 con 0x7f380815def0 2024-03-06T07:22:22.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f3806ffd640 1 -- 172.21.15.122:0/1558213126 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f8030610 con 0x7f380815def0 2024-03-06T07:22:22.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f3806ffd640 1 -- 172.21.15.122:0/1558213126 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80309e0 con 0x7f380815def0 2024-03-06T07:22:22.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 -- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380815def0 msgr2=0x7f38080b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 --2- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380815def0 0x7f38080b41b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f37f80097a0 tx=0x7f37f802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 -- 172.21.15.122:0/1558213126 shutdown_connections 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 --2- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380806cab0 0x7f38080ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 --2- 172.21.15.122:0/1558213126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380815def0 0x7f38080b41b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 -- 172.21.15.122:0/1558213126 >> 172.21.15.122:0/1558213126 conn(0x7f38080b3690 msgr2=0x7f38080b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 -- 172.21.15.122:0/1558213126 shutdown_connections 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.890+0000 7f380e688640 1 -- 172.21.15.122:0/1558213126 wait complete. 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.891+0000 7f380e688640 1 Processor -- start 2024-03-06T07:22:22.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.891+0000 7f380e688640 1 -- start start 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380e688640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380806cab0 0x7f3808156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380e688640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38080ce880 0x7f38080c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380e688640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 0x7f38080c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380e688640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38080cc3f0 con 0x7f380815def0 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380e688640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38080cc760 con 0x7f380806cab0 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380e688640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380815e2d0 con 0x7f38080ce880 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380cc25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 0x7f38080c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f38077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38080ce880 0x7f38080c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f3807fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380806cab0 0x7f3808156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380cc25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 0x7f38080c7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49926/0 (socket says 172.21.15.122:49926) 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.892+0000 7f380cc25640 1 -- 172.21.15.122:0/3882868120 learned_addr learned my addr 172.21.15.122:0/3882868120 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3807fff640 1 -- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38080ce880 msgr2=0x7f38080c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3807fff640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38080ce880 0x7f38080c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3807fff640 1 -- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 msgr2=0x7f38080c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3807fff640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 0x7f38080c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3807fff640 1 -- 172.21.15.122:0/3882868120 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38081571c0 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f380cc25640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 0x7f38080c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3807fff640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380806cab0 0x7f3808156c80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37f8033000 tx=0x7f37f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8034040 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f803aaa0 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80443f0 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38080c6240 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.893+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38080c6800 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.894+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380809fcf0 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.894+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37f804b060 con 0x7f380806cab0 2024-03-06T07:22:22.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.895+0000 7f3804ff9640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37f406b4d0 0x7f37f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.895+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37f80abf70 con 0x7f380806cab0 2024-03-06T07:22:22.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.895+0000 7f38077fe640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37f406b4d0 0x7f37f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.895+0000 7f38077fe640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37f406b4d0 0x7f37f406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009560 tx=0x7f37fc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:22.899+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37f803f070 con 0x7f380806cab0 2024-03-06T07:22:23.012 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:23.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:23.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.075+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3808161270 con 0x7f380806cab0 2024-03-06T07:22:23.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.076+0000 7f3804ff9640 1 -- 172.21.15.122:0/3882868120 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37f807a7a0 con 0x7f380806cab0 2024-03-06T07:22:23.077 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:23.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37f406b4d0 msgr2=0x7f37f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37f406b4d0 0x7f37f406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009560 tx=0x7f37fc009190 comp rx=0 tx=0).stop 2024-03-06T07:22:23.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380806cab0 msgr2=0x7f3808156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380806cab0 0x7f3808156c80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37f8033000 tx=0x7f37f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 shutdown_connections 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37f406b4d0 0x7f37f406d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380815def0 0x7f38080c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38080ce880 0x7f38080c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 --2- 172.21.15.122:0/3882868120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f380806cab0 0x7f3808156c80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 >> 172.21.15.122:0/3882868120 conn(0x7f38080b3690 msgr2=0x7f38080d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 shutdown_connections 2024-03-06T07:22:23.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.078+0000 7f380e688640 1 -- 172.21.15.122:0/3882868120 wait complete. 2024-03-06T07:22:23.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:23.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.234+0000 7f695efbc640 1 Processor -- start 2024-03-06T07:22:23.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.234+0000 7f695efbc640 1 -- start start 2024-03-06T07:22:23.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695efbc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69580d4860 0x7f69580d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695efbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695efbc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69581618b0 con 0x7f69581614d0 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695efbc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69580d4cf0 con 0x7f69580d4860 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695efbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69580cf400 con 0x7f69580b0e30 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f6957fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69580d4860 0x7f69580d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695cd58640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f69581614d0 0x7f69580d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44072/0 (socket says 172.21.15.122:44072) 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695cd58640 1 -- 172.21.15.122:0/3968714645 learned_addr learned my addr 172.21.15.122:0/3968714645 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f695d559640 1 --2- 172.21.15.122:0/3968714645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f6957fff640 1 -- 172.21.15.122:0/3968714645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 msgr2=0x7f69580b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f6957fff640 1 --2- 172.21.15.122:0/3968714645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f6957fff640 1 -- 172.21.15.122:0/3968714645 >> v1:172.21.15.122:6789/0 conn(0x7f69581614d0 legacy=0x7f69580d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f69577fe640 1 -- 172.21.15.122:0/3968714645 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765507320 0 0) 0x7f69581618b0 con 0x7f69581614d0 2024-03-06T07:22:23.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.235+0000 7f6957fff640 1 -- 172.21.15.122:0/3968714645 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69580cf6a0 con 0x7f69580d4860 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.236+0000 7f6957fff640 1 --2- 172.21.15.122:0/3968714645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69580d4860 0x7f69580d2020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f69440097a0 tx=0x7f694402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e198242c13325 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.236+0000 7f69577fe640 1 -- 172.21.15.122:0/3968714645 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944035020 con 0x7f69580d4860 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.236+0000 7f69577fe640 1 -- 172.21.15.122:0/3968714645 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6944031830 con 0x7f69580d4860 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.236+0000 7f69577fe640 1 -- 172.21.15.122:0/3968714645 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944031c00 con 0x7f69580d4860 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 -- 172.21.15.122:0/3968714645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69580d4860 msgr2=0x7f69580d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 --2- 172.21.15.122:0/3968714645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69580d4860 0x7f69580d2020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f69440097a0 tx=0x7f694402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 -- 172.21.15.122:0/3968714645 shutdown_connections 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 --2- 172.21.15.122:0/3968714645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 --2- 172.21.15.122:0/3968714645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69580d4860 0x7f69580d2020 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 -- 172.21.15.122:0/3968714645 >> 172.21.15.122:0/3968714645 conn(0x7f69580d3d40 msgr2=0x7f69580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 -- 172.21.15.122:0/3968714645 shutdown_connections 2024-03-06T07:22:23.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.237+0000 7f695efbc640 1 -- 172.21.15.122:0/3968714645 wait complete. 2024-03-06T07:22:23.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.238+0000 7f695efbc640 1 Processor -- start 2024-03-06T07:22:23.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.238+0000 7f695efbc640 1 -- start start 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695efbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695efbc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69580d4860 0x7f69580c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695efbc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 0x7f69580b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695efbc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69580b0910 con 0x7f69581614d0 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695efbc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69580cfc60 con 0x7f69580d4860 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695efbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69580d2560 con 0x7f69580b0e30 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 0x7f69580b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 0x7f69580b92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49934/0 (socket says 172.21.15.122:49934) 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 -- 172.21.15.122:0/1602311983 learned_addr learned my addr 172.21.15.122:0/1602311983 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:23.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695cd58640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f6957fff640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69580d4860 0x7f69580c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 -- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 msgr2=0x7f69580cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 -- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69580d4860 msgr2=0x7f69580c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69580d4860 0x7f69580c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 -- 172.21.15.122:0/1602311983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69580b7280 con 0x7f69581614d0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.239+0000 7f695d559640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 0x7f69580b92d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f695000efc0 tx=0x7f695000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.240+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950018020 con 0x7f69581614d0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.240+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6950014480 con 0x7f69581614d0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.240+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69580c87c0 con 0x7f69581614d0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.240+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950014870 con 0x7f69581614d0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.240+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69580c8d40 con 0x7f69581614d0 2024-03-06T07:22:23.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.241+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6958068ff0 con 0x7f69581614d0 2024-03-06T07:22:23.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.241+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6950005d10 con 0x7f69581614d0 2024-03-06T07:22:23.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.242+0000 7f69557fa640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f693c06b4d0 0x7f693c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.242+0000 7f695cd58640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f693c06b4d0 0x7f693c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.242+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f695008af30 con 0x7f69581614d0 2024-03-06T07:22:23.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.242+0000 7f695cd58640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f693c06b4d0 0x7f693c06d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f694c007fd0 tx=0x7f694c007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.246+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6950022030 con 0x7f69581614d0 2024-03-06T07:22:23.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.425+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69580b5550 con 0x7f69581614d0 2024-03-06T07:22:23.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.426+0000 7f69557fa640 1 -- 172.21.15.122:0/1602311983 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f69580b7280 con 0x7f69581614d0 2024-03-06T07:22:23.428 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f693c06b4d0 msgr2=0x7f693c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f693c06b4d0 0x7f693c06d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f694c007fd0 tx=0x7f694c007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 msgr2=0x7f69580b92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 0x7f69580b92d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f695000efc0 tx=0x7f695000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 shutdown_connections 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f693c06b4d0 0x7f693c06d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69581614d0 0x7f69580b92d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69580d4860 0x7f69580c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 --2- 172.21.15.122:0/1602311983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69580b0e30 0x7f69580cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 >> 172.21.15.122:0/1602311983 conn(0x7f69580d3d40 msgr2=0x7f695806c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 shutdown_connections 2024-03-06T07:22:23.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.428+0000 7f695efbc640 1 -- 172.21.15.122:0/1602311983 wait complete. 2024-03-06T07:22:23.450 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:22:23.450 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:22:23.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:23.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.553+0000 7f7c3c68a640 1 Processor -- start 2024-03-06T07:22:23.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.553+0000 7f7c3c68a640 1 -- start start 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3c68a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7c341424f0 0x7f7c340cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3c68a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7c3406da40 0x7f7c3415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3c68a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c340cc980 con 0x7f7c340cd7a0 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3c68a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c340cd220 con 0x7f7c3406da40 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3c68a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c340ccc60 con 0x7f7c341424f0 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3ac27640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7c3406da40 0x7f7c3415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c39c25640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7c340cd7a0 0x7f7c3406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44078/0 (socket says 172.21.15.122:44078) 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c39c25640 1 -- 172.21.15.122:0/2181763440 learned_addr learned my addr 172.21.15.122:0/2181763440 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3a426640 1 --2- 172.21.15.122:0/2181763440 >> v2:172.21.15.169:3300/0 conn(0x7f7c341424f0 0x7f7c340cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3ac27640 1 -- 172.21.15.122:0/2181763440 >> v2:172.21.15.169:3300/0 conn(0x7f7c341424f0 msgr2=0x7f7c340cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3ac27640 1 --2- 172.21.15.122:0/2181763440 >> v2:172.21.15.169:3300/0 conn(0x7f7c341424f0 0x7f7c340cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.554+0000 7f7c3ac27640 1 -- 172.21.15.122:0/2181763440 >> v1:172.21.15.122:6789/0 conn(0x7f7c340cd7a0 legacy=0x7f7c3406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.555+0000 7f7c3ac27640 1 -- 172.21.15.122:0/2181763440 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c340ccf40 con 0x7f7c3406da40 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.555+0000 7f7c39424640 1 -- 172.21.15.122:0/2181763440 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957161422 0 0) 0x7f7c340cc980 con 0x7f7c340cd7a0 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3ac27640 1 --2- 172.21.15.122:0/2181763440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c3406da40 0x7f7c3415cef0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c3406c6d0 tx=0x7f7c2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6707014bddb1b43f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c39424640 1 -- 172.21.15.122:0/2181763440 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c039070 con 0x7f7c3406da40 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c39424640 1 -- 172.21.15.122:0/2181763440 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c02cc80 con 0x7f7c3406da40 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c39424640 1 -- 172.21.15.122:0/2181763440 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c034660 con 0x7f7c3406da40 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3c68a640 1 -- 172.21.15.122:0/2181763440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c3406da40 msgr2=0x7f7c3415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/2181763440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c3406da40 0x7f7c3415cef0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c3406c6d0 tx=0x7f7c2c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3c68a640 1 -- 172.21.15.122:0/2181763440 shutdown_connections 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/2181763440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c3406da40 0x7f7c3415cef0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/2181763440 >> v2:172.21.15.169:3300/0 conn(0x7f7c341424f0 0x7f7c340cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.556+0000 7f7c3c68a640 1 -- 172.21.15.122:0/2181763440 >> 172.21.15.122:0/2181763440 conn(0x7f7c340cf8a0 msgr2=0x7f7c340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.557+0000 7f7c3c68a640 1 -- 172.21.15.122:0/2181763440 shutdown_connections 2024-03-06T07:22:23.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.557+0000 7f7c3c68a640 1 -- 172.21.15.122:0/2181763440 wait complete. 2024-03-06T07:22:23.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 Processor -- start 2024-03-06T07:22:23.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 -- start start 2024-03-06T07:22:23.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c3406da40 0x7f7c341f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 0x7f7c3413f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c341424f0 0x7f7c3415a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34071ce0 con 0x7f7c340cd7a0 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34071b60 con 0x7f7c3406da40 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c3c68a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34071e60 con 0x7f7c341424f0 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.558+0000 7f7c39c25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 0x7f7c3413f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 0x7f7c3413f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49954/0 (socket says 172.21.15.122:49954) 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 -- 172.21.15.122:0/3699868452 learned_addr learned my addr 172.21.15.122:0/3699868452 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c3ac27640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c341424f0 0x7f7c3415a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c3a426640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c3406da40 0x7f7c341f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 -- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c341424f0 msgr2=0x7f7c3415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c341424f0 0x7f7c3415a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 -- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c3406da40 msgr2=0x7f7c341f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c3406da40 0x7f7c341f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 -- 172.21.15.122:0/3699868452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c340b4210 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c39c25640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 0x7f7c3413f210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000c4d0 tx=0x7f7c3000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30018070 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c30007500 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c340b44f0 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30004a20 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.559+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3413f750 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.560+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3404f330 con 0x7f7c340cd7a0 2024-03-06T07:22:23.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.561+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c30004bc0 con 0x7f7c340cd7a0 2024-03-06T07:22:23.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.561+0000 7f7c26ffd640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c1806b4d0 0x7f7c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.561+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c300894e0 con 0x7f7c340cd7a0 2024-03-06T07:22:23.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.561+0000 7f7c3a426640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c1806b4d0 0x7f7c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.561+0000 7f7c3a426640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c1806b4d0 0x7f7c1806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c34141f00 tx=0x7f7c28009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.565+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c30057d20 con 0x7f7c340cd7a0 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.750+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c340d0620 con 0x7f7c340cd7a0 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.751+0000 7f7c26ffd640 1 -- 172.21.15.122:0/3699868452 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f7c3005bdc0 con 0x7f7c340cd7a0 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c1806b4d0 msgr2=0x7f7c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c1806b4d0 0x7f7c1806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c34141f00 tx=0x7f7c28009190 comp rx=0 tx=0).stop 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 msgr2=0x7f7c3413f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 0x7f7c3413f210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000c4d0 tx=0x7f7c3000c990 comp rx=0 tx=0).stop 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 shutdown_connections 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c1806b4d0 0x7f7c1806d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c341424f0 0x7f7c3415a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c340cd7a0 0x7f7c3413f210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 --2- 172.21.15.122:0/3699868452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c3406da40 0x7f7c341f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 >> 172.21.15.122:0/3699868452 conn(0x7f7c340cf8a0 msgr2=0x7f7c3415b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 shutdown_connections 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.753+0000 7f7c3c68a640 1 -- 172.21.15.122:0/3699868452 wait complete. 2024-03-06T07:22:23.766 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:23.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:23.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.869+0000 7f2c634cf640 1 Processor -- start 2024-03-06T07:22:23.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 -- start start 2024-03-06T07:22:23.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2c5c0d10c0 0x7f2c5c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2c5c15eb00 0x7f2c5c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5c15c050 con 0x7f2c5c0d10c0 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5c0d0060 con 0x7f2c5c15eb00 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c634cf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c0cfa50 con 0x7f2c5c0d14a0 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c6126b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2c5c0d10c0 0x7f2c5c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c6126b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2c5c0d10c0 0x7f2c5c0d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49960/0 (socket says 172.21.15.122:49960) 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.870+0000 7f2c6126b640 1 -- 172.21.15.122:0/2002519499 learned_addr learned my addr 172.21.15.122:0/2002519499 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c60a6a640 1 --2- 172.21.15.122:0/2002519499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c61a6c640 1 --2- 172.21.15.122:0/2002519499 >> v2:172.21.15.169:3300/0 conn(0x7f2c5c15eb00 0x7f2c5c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c6126b640 1 -- 172.21.15.122:0/2002519499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 msgr2=0x7f2c5c15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c6126b640 1 --2- 172.21.15.122:0/2002519499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c6126b640 1 -- 172.21.15.122:0/2002519499 >> v2:172.21.15.169:3300/0 conn(0x7f2c5c15eb00 msgr2=0x7f2c5c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c6126b640 1 --2- 172.21.15.122:0/2002519499 >> v2:172.21.15.169:3300/0 conn(0x7f2c5c15eb00 0x7f2c5c0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c6126b640 1 -- 172.21.15.122:0/2002519499 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c0d0a30 con 0x7f2c5c0d10c0 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.871+0000 7f2c6126b640 1 --2- 172.21.15.122:0/2002519499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0d04f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009870 tx=0x7f2c4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f4c11dab938d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c53fff640 1 -- 172.21.15.122:0/2002519499 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c035020 con 0x7f2c5c0d10c0 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c53fff640 1 -- 172.21.15.122:0/2002519499 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c031640 con 0x7f2c5c0d10c0 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c53fff640 1 -- 172.21.15.122:0/2002519499 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c02d040 con 0x7f2c5c0d10c0 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 -- 172.21.15.122:0/2002519499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c0d10c0 msgr2=0x7f2c5c0d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 --2- 172.21.15.122:0/2002519499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0d04f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009870 tx=0x7f2c4c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 -- 172.21.15.122:0/2002519499 shutdown_connections 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 --2- 172.21.15.122:0/2002519499 >> v2:172.21.15.169:3300/0 conn(0x7f2c5c15eb00 0x7f2c5c0cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 --2- 172.21.15.122:0/2002519499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 --2- 172.21.15.122:0/2002519499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0d04f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 -- 172.21.15.122:0/2002519499 >> 172.21.15.122:0/2002519499 conn(0x7f2c5c0b3800 msgr2=0x7f2c5c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 -- 172.21.15.122:0/2002519499 shutdown_connections 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.872+0000 7f2c634cf640 1 -- 172.21.15.122:0/2002519499 wait complete. 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.873+0000 7f2c634cf640 1 Processor -- start 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.873+0000 7f2c634cf640 1 -- start start 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c634cf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c634cf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c634cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 0x7f2c5c15a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c634cf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c15f440 con 0x7f2c5c15eb00 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c634cf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c15f160 con 0x7f2c5c0d10c0 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c634cf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c0cfcf0 con 0x7f2c5c0d14a0 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 0x7f2c5c15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 0x7f2c5c15a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49962/0 (socket says 172.21.15.122:49962) 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 -- 172.21.15.122:0/4197244472 learned_addr learned my addr 172.21.15.122:0/4197244472 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c60a6a640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c6126b640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 -- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 msgr2=0x7f2c5c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 -- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c5c0d10c0 msgr2=0x7f2c5c0b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.874+0000 7f2c61a6c640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0b19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c61a6c640 1 -- 172.21.15.122:0/4197244472 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c0b4a90 con 0x7f2c5c15eb00 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c61a6c640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 0x7f2c5c15a570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009dd0 tx=0x7f2c5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54018020 con 0x7f2c5c15eb00 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c54014480 con 0x7f2c5c15eb00 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5c0b4d30 con 0x7f2c5c15eb00 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54014870 con 0x7f2c5c15eb00 2024-03-06T07:22:23.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.875+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5c0b52f0 con 0x7f2c5c15eb00 2024-03-06T07:22:23.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.876+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5c09fe60 con 0x7f2c5c15eb00 2024-03-06T07:22:23.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.876+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c54005d10 con 0x7f2c5c15eb00 2024-03-06T07:22:23.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.877+0000 7f2c51ffb640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4006b4d0 0x7f2c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.877+0000 7f2c6126b640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4006b4d0 0x7f2c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.877+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c5408ab30 con 0x7f2c5c15eb00 2024-03-06T07:22:23.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.877+0000 7f2c6126b640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4006b4d0 0x7f2c4006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c036000 tx=0x7f2c4c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:23.881+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c54022070 con 0x7f2c5c15eb00 2024-03-06T07:22:24.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.072+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c5c0d0c30 con 0x7f2c5c15eb00 2024-03-06T07:22:24.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.073+0000 7f2c51ffb640 1 -- 172.21.15.122:0/4197244472 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c54010040 con 0x7f2c5c15eb00 2024-03-06T07:22:24.075 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:24.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4006b4d0 msgr2=0x7f2c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4006b4d0 0x7f2c4006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c036000 tx=0x7f2c4c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 msgr2=0x7f2c5c15a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 0x7f2c5c15a570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009dd0 tx=0x7f2c5400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:24.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 shutdown_connections 2024-03-06T07:22:24.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4006b4d0 0x7f2c4006d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c5c15eb00 0x7f2c5c15a570 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c5c0d14a0 0x7f2c5c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 --2- 172.21.15.122:0/4197244472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c5c0d10c0 0x7f2c5c0b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 >> 172.21.15.122:0/4197244472 conn(0x7f2c5c0b3800 msgr2=0x7f2c5c15b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 shutdown_connections 2024-03-06T07:22:24.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.075+0000 7f2c634cf640 1 -- 172.21.15.122:0/4197244472 wait complete. 2024-03-06T07:22:24.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:24.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.190+0000 7fba7cb01640 1 Processor -- start 2024-03-06T07:22:24.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.190+0000 7fba7cb01640 1 -- start start 2024-03-06T07:22:24.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba7cb01640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fba7815dd00 0x7fba7806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba7cb01640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7815e0e0 0x7fba7806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba7cb01640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba78161bd0 con 0x7fba780b0000 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba7cb01640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba78070220 con 0x7fba7815dd00 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba7cb01640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba78161e90 con 0x7fba7815e0e0 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba7659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fba7815dd00 0x7fba7806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba76d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fba780b0000 0x7fba780b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44082/0 (socket says 172.21.15.122:44082) 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba76d9d640 1 -- 172.21.15.122:0/2667702096 learned_addr learned my addr 172.21.15.122:0/2667702096 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.191+0000 7fba6ffff640 1 --2- 172.21.15.122:0/2667702096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7815e0e0 0x7fba7806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.192+0000 7fba7659c640 1 -- 172.21.15.122:0/2667702096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7815e0e0 msgr2=0x7fba7806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.192+0000 7fba7659c640 1 --2- 172.21.15.122:0/2667702096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7815e0e0 0x7fba7806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.192+0000 7fba7659c640 1 -- 172.21.15.122:0/2667702096 >> v1:172.21.15.122:6789/0 conn(0x7fba780b0000 legacy=0x7fba780b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.192+0000 7fba7659c640 1 -- 172.21.15.122:0/2667702096 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba78162170 con 0x7fba7815dd00 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.192+0000 7fba75d9b640 1 -- 172.21.15.122:0/2667702096 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304354707 0 0) 0x7fba78161bd0 con 0x7fba780b0000 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.192+0000 7fba7659c640 1 --2- 172.21.15.122:0/2667702096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba7806e620 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fba680097a0 tx=0x7fba6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6671e4f46516bbdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba75d9b640 1 -- 172.21.15.122:0/2667702096 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68035020 con 0x7fba7815dd00 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba75d9b640 1 -- 172.21.15.122:0/2667702096 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba68031830 con 0x7fba7815dd00 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba75d9b640 1 -- 172.21.15.122:0/2667702096 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68031c00 con 0x7fba7815dd00 2024-03-06T07:22:24.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 -- 172.21.15.122:0/2667702096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 msgr2=0x7fba7806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2667702096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba7806e620 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fba680097a0 tx=0x7fba6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 -- 172.21.15.122:0/2667702096 shutdown_connections 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2667702096 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7815e0e0 0x7fba7806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2667702096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba7806e620 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 -- 172.21.15.122:0/2667702096 >> 172.21.15.122:0/2667702096 conn(0x7fba780b2660 msgr2=0x7fba780b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 -- 172.21.15.122:0/2667702096 shutdown_connections 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.193+0000 7fba7cb01640 1 -- 172.21.15.122:0/2667702096 wait complete. 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.194+0000 7fba7cb01640 1 Processor -- start 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 -- start start 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba780b0000 0x7fba7814edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba780b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba780b1cd0 0x7fba78154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba78162810 con 0x7fba7815dd00 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba78162690 con 0x7fba780b1cd0 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7cb01640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba78162990 con 0x7fba780b0000 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba780b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba780b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49976/0 (socket says 172.21.15.122:49976) 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba6ffff640 1 -- 172.21.15.122:0/2566678436 learned_addr learned my addr 172.21.15.122:0/2566678436 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba7659c640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba780b0000 0x7fba7814edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.195+0000 7fba76d9d640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba780b1cd0 0x7fba78154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6ffff640 1 -- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba780b0000 msgr2=0x7fba7814edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6ffff640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba780b0000 0x7fba7814edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6ffff640 1 -- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba780b1cd0 msgr2=0x7fba78154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6ffff640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba780b1cd0 0x7fba78154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6ffff640 1 -- 172.21.15.122:0/2566678436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba781f72c0 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6ffff640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba780b1790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e9e0 tx=0x7fba6000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60017070 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6000c9d0 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba781f75a0 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6000cdc0 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.196+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7814f330 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.197+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba780b20e0 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.197+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba6001a070 con 0x7fba7815dd00 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.198+0000 7fba6effd640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba5806b500 0x7fba5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.198+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba6008a4b0 con 0x7fba7815dd00 2024-03-06T07:22:24.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.198+0000 7fba7659c640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba5806b500 0x7fba5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.198+0000 7fba7659c640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba5806b500 0x7fba5806d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fba68036000 tx=0x7fba68008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.202+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba780b20e0 con 0x7fba7815dd00 2024-03-06T07:22:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.379+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba78068ff0 con 0x7fba7815dd00 2024-03-06T07:22:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.379+0000 7fba6effd640 1 -- 172.21.15.122:0/2566678436 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba60058cd0 con 0x7fba7815dd00 2024-03-06T07:22:24.381 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba5806b500 msgr2=0x7fba5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba5806b500 0x7fba5806d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fba68036000 tx=0x7fba68008030 comp rx=0 tx=0).stop 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 msgr2=0x7fba780b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba780b1790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e9e0 tx=0x7fba6000eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 shutdown_connections 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba5806b500 0x7fba5806d9c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba780b1cd0 0x7fba78154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7815dd00 0x7fba780b1790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 --2- 172.21.15.122:0/2566678436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba780b0000 0x7fba7814edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 >> 172.21.15.122:0/2566678436 conn(0x7fba780b2660 msgr2=0x7fba780b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 shutdown_connections 2024-03-06T07:22:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.381+0000 7fba7cb01640 1 -- 172.21.15.122:0/2566678436 wait complete. 2024-03-06T07:22:24.394 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:24.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.495+0000 7f0d4cf8c640 1 Processor -- start 2024-03-06T07:22:24.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.495+0000 7f0d4cf8c640 1 -- start start 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4cf8c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0d480cfa90 0x7f0d480cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4cf8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d480b33a0 0x7f0d480b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4cf8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d48158ac0 con 0x7f0d481586e0 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4cf8c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4815b2a0 con 0x7f0d480cfa90 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4cf8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4815b540 con 0x7f0d480b33a0 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d3ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0d480cfa90 0x7f0d480cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d46d9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d480b33a0 0x7f0d480b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4659c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f0d481586e0 0x7f0d480aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:36458/0 (socket says 172.21.15.122:36458) 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d4659c640 1 -- 172.21.15.122:0/3462449065 learned_addr learned my addr 172.21.15.122:0/3462449065 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d3ffff640 1 --2- 172.21.15.122:0/3462449065 >> v2:172.21.15.169:3300/0 conn(0x7f0d480cfa90 0x7f0d480cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46012/0 (socket says 172.21.15.122:46012) 2024-03-06T07:22:24.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d3ffff640 1 -- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d480b33a0 msgr2=0x7f0d480b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d3ffff640 1 --2- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d480b33a0 0x7f0d480b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d3ffff640 1 -- 172.21.15.122:0/3462449065 >> v1:172.21.15.169:6789/0 conn(0x7f0d481586e0 legacy=0x7f0d480aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d3ffff640 1 -- 172.21.15.122:0/3462449065 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4815b820 con 0x7f0d480cfa90 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d45d9b640 1 -- 172.21.15.122:0/3462449065 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669537500 0 0) 0x7f0d48158ac0 con 0x7f0d481586e0 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.496+0000 7f0d46d9d640 1 --2- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d480b33a0 0x7f0d480b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.497+0000 7f0d3ffff640 1 --2- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d480cff40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d300097a0 tx=0x7f0d3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9768756bb84a9c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.497+0000 7f0d45d9b640 1 -- 172.21.15.122:0/3462449065 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30035020 con 0x7f0d480cfa90 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.497+0000 7f0d45d9b640 1 -- 172.21.15.122:0/3462449065 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d30031830 con 0x7f0d480cfa90 2024-03-06T07:22:24.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.497+0000 7f0d45d9b640 1 -- 172.21.15.122:0/3462449065 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30031c00 con 0x7f0d480cfa90 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 msgr2=0x7f0d480cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d480cff40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d300097a0 tx=0x7f0d3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/3462449065 shutdown_connections 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d480b33a0 0x7f0d480b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/3462449065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d480cff40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/3462449065 >> 172.21.15.122:0/3462449065 conn(0x7f0d4806b190 msgr2=0x7f0d4806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/3462449065 shutdown_connections 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.498+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/3462449065 wait complete. 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.499+0000 7f0d4cf8c640 1 Processor -- start 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.499+0000 7f0d4cf8c640 1 -- start start 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 0x7f0d480b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d4814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d481586e0 0x7f0d4814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480af150 con 0x7f0d480b33a0 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4815bde0 con 0x7f0d480cfa90 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d48071350 con 0x7f0d481586e0 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 0x7f0d480b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 0x7f0d480b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38584/0 (socket says 172.21.15.122:38584) 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 -- 172.21.15.122:0/2263403304 learned_addr learned my addr 172.21.15.122:0/2263403304 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d3ffff640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d4814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d46d9d640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d481586e0 0x7f0d4814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 -- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d481586e0 msgr2=0x7f0d4814e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d481586e0 0x7f0d4814e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 -- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 msgr2=0x7f0d4814df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d4814df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 -- 172.21.15.122:0/2263403304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d480aa460 con 0x7f0d480b33a0 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4659c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 0x7f0d480b3a30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c500 tx=0x7f0d3800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d38018070 con 0x7f0d480b33a0 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d38007500 con 0x7f0d480b33a0 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d38004a20 con 0x7f0d480b33a0 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d480aaeb0 con 0x7f0d480b33a0 2024-03-06T07:22:24.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.500+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d480ab470 con 0x7f0d480b33a0 2024-03-06T07:22:24.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.501+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d480732b0 con 0x7f0d480b33a0 2024-03-06T07:22:24.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.502+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d38004bc0 con 0x7f0d480b33a0 2024-03-06T07:22:24.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.502+0000 7f0d3effd640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d2806b4d0 0x7f0d2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.503+0000 7f0d3ffff640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d2806b4d0 0x7f0d2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.503+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d38089420 con 0x7f0d480b33a0 2024-03-06T07:22:24.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.503+0000 7f0d3ffff640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d2806b4d0 0x7f0d2806d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d30036000 tx=0x7f0d3003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.507+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d38057bd0 con 0x7f0d480b33a0 2024-03-06T07:22:24.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.680+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d480b4b00 con 0x7f0d480b33a0 2024-03-06T07:22:24.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.681+0000 7f0d3effd640 1 -- 172.21.15.122:0/2263403304 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0d3805bc70 con 0x7f0d480b33a0 2024-03-06T07:22:24.683 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d2806b4d0 msgr2=0x7f0d2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d2806b4d0 0x7f0d2806d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d30036000 tx=0x7f0d3003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 msgr2=0x7f0d480b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 0x7f0d480b3a30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c500 tx=0x7f0d3800c900 comp rx=0 tx=0).stop 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 shutdown_connections 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d2806b4d0 0x7f0d2806d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d481586e0 0x7f0d4814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d480cfa90 0x7f0d4814df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 --2- 172.21.15.122:0/2263403304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d480b33a0 0x7f0d480b3a30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 >> 172.21.15.122:0/2263403304 conn(0x7f0d4806b190 msgr2=0x7f0d480cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 shutdown_connections 2024-03-06T07:22:24.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.683+0000 7f0d4cf8c640 1 -- 172.21.15.122:0/2263403304 wait complete. 2024-03-06T07:22:24.695 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:22:24.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 Processor -- start 2024-03-06T07:22:24.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 -- start start 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f98b40b3df0 0x7f98b40d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f98b40d28c0 0x7f98b40d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40ce990 con 0x7f98b40d28c0 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40b4290 con 0x7f98b40b3df0 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b9839640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40b4570 con 0x7f98b415e0b0 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.794+0000 7f98b37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f98b40d28c0 0x7f98b40d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f98b40d28c0 0x7f98b40d0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38594/0 (socket says 172.21.15.122:38594) 2024-03-06T07:22:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b27fc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f98b40b3df0 0x7f98b40d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 -- 172.21.15.122:0/2714582987 learned_addr learned my addr 172.21.15.122:0/2714582987 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b2ffd640 1 --2- 172.21.15.122:0/2714582987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 -- 172.21.15.122:0/2714582987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 msgr2=0x7f98b40b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 --2- 172.21.15.122:0/2714582987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 -- 172.21.15.122:0/2714582987 >> v2:172.21.15.169:3300/0 conn(0x7f98b40b3df0 msgr2=0x7f98b40d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 --2- 172.21.15.122:0/2714582987 >> v2:172.21.15.169:3300/0 conn(0x7f98b40b3df0 0x7f98b40d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.795+0000 7f98b37fe640 1 -- 172.21.15.122:0/2714582987 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40b4850 con 0x7f98b40d28c0 2024-03-06T07:22:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b37fe640 1 --2- 172.21.15.122:0/2714582987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b40d0790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f98b40d18f0 tx=0x7f98a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b77d73a43c7843d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b1ffb640 1 -- 172.21.15.122:0/2714582987 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4035020 con 0x7f98b40d28c0 2024-03-06T07:22:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b1ffb640 1 -- 172.21.15.122:0/2714582987 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a4031870 con 0x7f98b40d28c0 2024-03-06T07:22:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b1ffb640 1 -- 172.21.15.122:0/2714582987 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4031c40 con 0x7f98b40d28c0 2024-03-06T07:22:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 -- 172.21.15.122:0/2714582987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 msgr2=0x7f98b40d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 --2- 172.21.15.122:0/2714582987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b40d0790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f98b40d18f0 tx=0x7f98a402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 -- 172.21.15.122:0/2714582987 shutdown_connections 2024-03-06T07:22:24.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 --2- 172.21.15.122:0/2714582987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b40d0790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 --2- 172.21.15.122:0/2714582987 >> v2:172.21.15.169:3300/0 conn(0x7f98b40b3df0 0x7f98b40d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 --2- 172.21.15.122:0/2714582987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 -- 172.21.15.122:0/2714582987 >> 172.21.15.122:0/2714582987 conn(0x7f98b408d6f0 msgr2=0x7f98b409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 -- 172.21.15.122:0/2714582987 shutdown_connections 2024-03-06T07:22:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.796+0000 7f98b9839640 1 -- 172.21.15.122:0/2714582987 wait complete. 2024-03-06T07:22:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 Processor -- start 2024-03-06T07:22:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 -- start start 2024-03-06T07:22:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b3df0 0x7f98b414dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40d1cd0 con 0x7f98b40d28c0 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b415e490 con 0x7f98b40b3df0 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b9839640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40d1e50 con 0x7f98b415e0b0 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.798+0000 7f98b27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b414e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38610/0 (socket says 172.21.15.122:38610) 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 -- 172.21.15.122:0/13056813 learned_addr learned my addr 172.21.15.122:0/13056813 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b2ffd640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b3df0 0x7f98b414dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b37fe640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 -- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 msgr2=0x7f98b40cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 -- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b3df0 msgr2=0x7f98b414dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b3df0 0x7f98b414dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 -- 172.21.15.122:0/13056813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40b9b60 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b27fc640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b414e280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e9b0 tx=0x7f98a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8017070 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a800c9a0 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a800cd90 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b415e930 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.799+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b415eef0 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.800+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b404f330 con 0x7f98b40d28c0 2024-03-06T07:22:24.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.801+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98a801a070 con 0x7f98b40d28c0 2024-03-06T07:22:24.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.801+0000 7f989bfff640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98a006b4d0 0x7f98a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.801+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98a808a530 con 0x7f98b40d28c0 2024-03-06T07:22:24.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.801+0000 7f98b2ffd640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98a006b4d0 0x7f98a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.801+0000 7f98b2ffd640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98a006b4d0 0x7f98a006d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f98ac011340 tx=0x7f98ac00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:24.805+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98a8058d60 con 0x7f98b40d28c0 2024-03-06T07:22:25.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.045+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f98b406d5f0 con 0x7f98b40d28c0 2024-03-06T07:22:25.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.670+0000 7f989bfff640 1 -- 172.21.15.122:0/13056813 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v55) v1 ==== 135+0+0 (secure 0 0 0) 0x7f98a805ce00 con 0x7f98b40d28c0 2024-03-06T07:22:25.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98a006b4d0 msgr2=0x7f98a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98a006b4d0 0x7f98a006d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f98ac011340 tx=0x7f98ac00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 msgr2=0x7f98b414e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b414e280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e9b0 tx=0x7f98a800ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 shutdown_connections 2024-03-06T07:22:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98a006b4d0 0x7f98a006d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b415e0b0 0x7f98b40cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40d28c0 0x7f98b414e280 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 --2- 172.21.15.122:0/13056813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b3df0 0x7f98b414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 >> 172.21.15.122:0/13056813 conn(0x7f98b408d6f0 msgr2=0x7f98b406cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.672+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 shutdown_connections 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.673+0000 7f98b9839640 1 -- 172.21.15.122:0/13056813 wait complete. 2024-03-06T07:22:25.686 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.788+0000 7f2910f25640 1 Processor -- start 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.788+0000 7f2910f25640 1 -- start start 2024-03-06T07:22:25.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.789+0000 7f2910f25640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f290c15a220 0x7f290c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.789+0000 7f2910f25640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.789+0000 7f2910f25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290c0b3320 con 0x7f290c15de60 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.789+0000 7f2910f25640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290c0cf610 con 0x7f290c15a220 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.789+0000 7f2910f25640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290c071c70 con 0x7f290c0712d0 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f290c15a220 0x7f290c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290ad9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f290c15de60 0x7f290c15e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49620/0 (socket says 172.21.15.122:49620) 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290ad9d640 1 -- 172.21.15.122:0/2698981042 learned_addr learned my addr 172.21.15.122:0/2698981042 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f2909d9b640 1 --2- 172.21.15.122:0/2698981042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290a59c640 1 -- 172.21.15.122:0/2698981042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 msgr2=0x7f290c071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290a59c640 1 --2- 172.21.15.122:0/2698981042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290a59c640 1 -- 172.21.15.122:0/2698981042 >> v1:172.21.15.122:6789/0 conn(0x7f290c15de60 legacy=0x7f290c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290a59c640 1 -- 172.21.15.122:0/2698981042 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0728d0 con 0x7f290c15a220 2024-03-06T07:22:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.790+0000 7f290959a640 1 -- 172.21.15.122:0/2698981042 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879830979 0 0) 0x7f290c0b3320 con 0x7f290c15de60 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f290a59c640 1 --2- 172.21.15.122:0/2698981042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15a220 0x7f290c0cf0d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f29000097a0 tx=0x7f290002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c0ba52270f22ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f290959a640 1 -- 172.21.15.122:0/2698981042 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29000315a0 con 0x7f290c15a220 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f290959a640 1 -- 172.21.15.122:0/2698981042 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2900037070 con 0x7f290c15a220 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f290959a640 1 -- 172.21.15.122:0/2698981042 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900031960 con 0x7f290c15a220 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 -- 172.21.15.122:0/2698981042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15a220 msgr2=0x7f290c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 --2- 172.21.15.122:0/2698981042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15a220 0x7f290c0cf0d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f29000097a0 tx=0x7f290002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 -- 172.21.15.122:0/2698981042 shutdown_connections 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 --2- 172.21.15.122:0/2698981042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 --2- 172.21.15.122:0/2698981042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15a220 0x7f290c0cf0d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 -- 172.21.15.122:0/2698981042 >> 172.21.15.122:0/2698981042 conn(0x7f290c06b190 msgr2=0x7f290c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 -- 172.21.15.122:0/2698981042 shutdown_connections 2024-03-06T07:22:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.791+0000 7f2910f25640 1 -- 172.21.15.122:0/2698981042 wait complete. 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 Processor -- start 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 -- start start 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c15a220 0x7f290c14ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 0x7f290c1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c072030 con 0x7f290c15de60 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c072bb0 con 0x7f290c15a220 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2910f25640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290c072530 con 0x7f290c0712d0 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f290ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 0x7f290c1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f290ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 0x7f290c1545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38620/0 (socket says 172.21.15.122:38620) 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f290ad9d640 1 -- 172.21.15.122:0/2706306139 learned_addr learned my addr 172.21.15.122:0/2706306139 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f290a59c640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.793+0000 7f2909d9b640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c15a220 0x7f290c14ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f290ad9d640 1 -- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 msgr2=0x7f290c1572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f290ad9d640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c1572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f290ad9d640 1 -- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c15a220 msgr2=0x7f290c14ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f290ad9d640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c15a220 0x7f290c14ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f290ad9d640 1 -- 172.21.15.122:0/2706306139 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c154ae0 con 0x7f290c15de60 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f290ad9d640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 0x7f290c1545a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f290400c700 tx=0x7f290400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904018070 con 0x7f290c15de60 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290400ce10 con 0x7f290c15de60 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290400a860 con 0x7f290c15de60 2024-03-06T07:22:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290c155640 con 0x7f290c15de60 2024-03-06T07:22:25.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.794+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290c155bc0 con 0x7f290c15de60 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.795+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290c0732b0 con 0x7f290c15de60 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.795+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f290401b070 con 0x7f290c15de60 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.796+0000 7f28faffd640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f28ec06b4d0 0x7f28ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.796+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29040896c0 con 0x7f290c15de60 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.796+0000 7f290a59c640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f28ec06b4d0 0x7f28ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.796+0000 7f290a59c640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f28ec06b4d0 0x7f28ec06d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2900036000 tx=0x7f290003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.800+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f290c154ae0 con 0x7f290c15de60 2024-03-06T07:22:25.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.971+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f290c068ff0 con 0x7f290c15de60 2024-03-06T07:22:25.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.973+0000 7f28faffd640 1 -- 172.21.15.122:0/2706306139 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2904057f80 con 0x7f290c15de60 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f28ec06b4d0 msgr2=0x7f28ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f28ec06b4d0 0x7f28ec06d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2900036000 tx=0x7f290003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 msgr2=0x7f290c1545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 0x7f290c1545a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f290400c700 tx=0x7f290400cbc0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 shutdown_connections 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f28ec06b4d0 0x7f28ec06d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15de60 0x7f290c1545a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c15a220 0x7f290c14ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 --2- 172.21.15.122:0/2706306139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f290c0712d0 0x7f290c1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 >> 172.21.15.122:0/2706306139 conn(0x7f290c06b190 msgr2=0x7f290c0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 shutdown_connections 2024-03-06T07:22:25.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:25.976+0000 7f2910f25640 1 -- 172.21.15.122:0/2706306139 wait complete. 2024-03-06T07:22:25.989 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:25.989 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:26.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:26.071 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:26.071 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:22:26.072 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:26.087 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:22:26.087 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:26.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:26.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.278+0000 7f344889a640 1 Processor -- start 2024-03-06T07:22:26.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.278+0000 7f344889a640 1 -- start start 2024-03-06T07:22:26.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344889a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f344413dc10 0x7f34440b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344889a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f344415bf30 0x7f34440b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344889a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344415f580 con 0x7f344415e850 2024-03-06T07:22:26.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344889a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34440cf3f0 con 0x7f344413dc10 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344889a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344415b210 con 0x7f344415bf30 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f344413dc10 0x7f34440b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f344413dc10 0x7f34440b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38628/0 (socket says 172.21.15.122:38628) 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f344259c640 1 -- 172.21.15.122:0/2235259985 learned_addr learned my addr 172.21.15.122:0/2235259985 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.279+0000 7f343bfff640 1 --2- 172.21.15.122:0/2235259985 >> v2:172.21.15.169:3300/0 conn(0x7f344415bf30 0x7f34440b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.280+0000 7f344259c640 1 -- 172.21.15.122:0/2235259985 >> v2:172.21.15.169:3300/0 conn(0x7f344415bf30 msgr2=0x7f34440b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.280+0000 7f344259c640 1 --2- 172.21.15.122:0/2235259985 >> v2:172.21.15.169:3300/0 conn(0x7f344415bf30 0x7f34440b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.280+0000 7f344259c640 1 -- 172.21.15.122:0/2235259985 >> v1:172.21.15.169:6789/0 conn(0x7f344415e850 legacy=0x7f344415ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.280+0000 7f344259c640 1 -- 172.21.15.122:0/2235259985 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344415b450 con 0x7f344413dc10 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.280+0000 7f344259c640 1 --2- 172.21.15.122:0/2235259985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344413dc10 0x7f34440b2950 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f344406c790 tx=0x7f343402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36dc9f6bc15b14cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f3441d9b640 1 -- 172.21.15.122:0/2235259985 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434035020 con 0x7f344413dc10 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f3441d9b640 1 -- 172.21.15.122:0/2235259985 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3434031870 con 0x7f344413dc10 2024-03-06T07:22:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f3441d9b640 1 -- 172.21.15.122:0/2235259985 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434031c40 con 0x7f344413dc10 2024-03-06T07:22:26.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 -- 172.21.15.122:0/2235259985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344413dc10 msgr2=0x7f34440b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 --2- 172.21.15.122:0/2235259985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344413dc10 0x7f34440b2950 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f344406c790 tx=0x7f343402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 -- 172.21.15.122:0/2235259985 shutdown_connections 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 --2- 172.21.15.122:0/2235259985 >> v2:172.21.15.169:3300/0 conn(0x7f344415bf30 0x7f34440b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 --2- 172.21.15.122:0/2235259985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344413dc10 0x7f34440b2950 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 -- 172.21.15.122:0/2235259985 >> 172.21.15.122:0/2235259985 conn(0x7f34440cf8a0 msgr2=0x7f34440cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 -- 172.21.15.122:0/2235259985 shutdown_connections 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.281+0000 7f344889a640 1 -- 172.21.15.122:0/2235259985 wait complete. 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.282+0000 7f344889a640 1 Processor -- start 2024-03-06T07:22:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 -- start start 2024-03-06T07:22:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f344413dc10 0x7f344413f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344415bf30 0x7f34441407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 0x7f3444072040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440cd7a0 con 0x7f344415e850 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344413dff0 con 0x7f344415bf30 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344889a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34440cd920 con 0x7f344413dc10 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f3442d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 0x7f3444072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f3442d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 0x7f3444072040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38630/0 (socket says 172.21.15.122:38630) 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f3442d9d640 1 -- 172.21.15.122:0/4101293020 learned_addr learned my addr 172.21.15.122:0/4101293020 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f343bfff640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344415bf30 0x7f34441407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.283+0000 7f344259c640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f344413dc10 0x7f344413f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f3442d9d640 1 -- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f344413dc10 msgr2=0x7f344413f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f3442d9d640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f344413dc10 0x7f344413f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f3442d9d640 1 -- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344415bf30 msgr2=0x7f34441407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f3442d9d640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344415bf30 0x7f34441407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f3442d9d640 1 -- 172.21.15.122:0/4101293020 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444140ce0 con 0x7f344415e850 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f343bfff640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344415bf30 0x7f34441407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f3442d9d640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 0x7f3444072040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c6e0 tx=0x7f343c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c003b70 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c003d30 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3444140f50 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c005310 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.284+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3444141a40 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.285+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344404f330 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.285+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f343c0054b0 con 0x7f344415e850 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.286+0000 7f343affd640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f342406b4d0 0x7f342406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.286+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f343c089f30 con 0x7f344415e850 2024-03-06T07:22:26.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.286+0000 7f344259c640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f342406b4d0 0x7f342406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.286+0000 7f344259c640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f342406b4d0 0x7f342406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3434036000 tx=0x7f343403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.290+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f343c058760 con 0x7f344415e850 2024-03-06T07:22:26.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.466+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f344406eb00 con 0x7f344415e850 2024-03-06T07:22:26.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.467+0000 7f343affd640 1 -- 172.21.15.122:0/4101293020 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f343c05c800 con 0x7f344415e850 2024-03-06T07:22:26.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:26.469 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f342406b4d0 msgr2=0x7f342406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f342406b4d0 0x7f342406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3434036000 tx=0x7f343403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 msgr2=0x7f3444072040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 0x7f3444072040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c6e0 tx=0x7f343c00cba0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 shutdown_connections 2024-03-06T07:22:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f342406b4d0 0x7f342406d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344415e850 0x7f3444072040 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344415bf30 0x7f34441407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 --2- 172.21.15.122:0/4101293020 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f344413dc10 0x7f344413f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 >> 172.21.15.122:0/4101293020 conn(0x7f34440cf8a0 msgr2=0x7f344415ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 shutdown_connections 2024-03-06T07:22:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.469+0000 7f344889a640 1 -- 172.21.15.122:0/4101293020 wait complete. 2024-03-06T07:22:26.482 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:26.483 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:26.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.584+0000 7f6d3095c640 1 Processor -- start 2024-03-06T07:22:26.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.584+0000 7f6d3095c640 1 -- start start 2024-03-06T07:22:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.584+0000 7f6d3095c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c1586e0 0x7f6d2c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.584+0000 7f6d3095c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6d2c0aef80 0x7f6d2c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.584+0000 7f6d3095c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c0d03f0 con 0x7f6d2c0cffd0 2024-03-06T07:22:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d3095c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c158ac0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d3095c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15b2a0 con 0x7f6d2c1586e0 2024-03-06T07:22:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d21d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6d2c0aef80 0x7f6d2c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d2ad9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6d2c0cffd0 0x7f6d2c06da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49628/0 (socket says 172.21.15.122:49628) 2024-03-06T07:22:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d21d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6d2c0aef80 0x7f6d2c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38636/0 (socket says 172.21.15.122:38636) 2024-03-06T07:22:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d2ad9d640 1 -- 172.21.15.122:0/2171733535 learned_addr learned my addr 172.21.15.122:0/2171733535 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d2a59c640 1 --2- 172.21.15.122:0/2171733535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c1586e0 0x7f6d2c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d21d9b640 1 -- 172.21.15.122:0/2171733535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c1586e0 msgr2=0x7f6d2c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d21d9b640 1 --2- 172.21.15.122:0/2171733535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c1586e0 0x7f6d2c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d21d9b640 1 -- 172.21.15.122:0/2171733535 >> v1:172.21.15.122:6789/0 conn(0x7f6d2c0cffd0 legacy=0x7f6d2c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d29d9b640 1 -- 172.21.15.122:0/2171733535 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131634769 0 0) 0x7f6d2c0d03f0 con 0x7f6d2c0cffd0 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.585+0000 7f6d21d9b640 1 -- 172.21.15.122:0/2171733535 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c15b540 con 0x7f6d2c0aef80 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d21d9b640 1 --2- 172.21.15.122:0/2171733535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c0cfa90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6d140097a0 tx=0x7f6d1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a50833e0e6f6909 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d29d9b640 1 -- 172.21.15.122:0/2171733535 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14035020 con 0x7f6d2c0aef80 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d29d9b640 1 -- 172.21.15.122:0/2171733535 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14031830 con 0x7f6d2c0aef80 2024-03-06T07:22:26.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d29d9b640 1 -- 172.21.15.122:0/2171733535 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14031c00 con 0x7f6d2c0aef80 2024-03-06T07:22:26.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 -- 172.21.15.122:0/2171733535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 msgr2=0x7f6d2c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 --2- 172.21.15.122:0/2171733535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c0cfa90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6d140097a0 tx=0x7f6d1402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 -- 172.21.15.122:0/2171733535 shutdown_connections 2024-03-06T07:22:26.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 --2- 172.21.15.122:0/2171733535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c0cfa90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 --2- 172.21.15.122:0/2171733535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c1586e0 0x7f6d2c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 -- 172.21.15.122:0/2171733535 >> 172.21.15.122:0/2171733535 conn(0x7f6d2c06b190 msgr2=0x7f6d2c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 -- 172.21.15.122:0/2171733535 shutdown_connections 2024-03-06T07:22:26.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.586+0000 7f6d3095c640 1 -- 172.21.15.122:0/2171733535 wait complete. 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.588+0000 7f6d3095c640 1 Processor -- start 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.588+0000 7f6d3095c640 1 -- start start 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c0cffd0 0x7f6d2c0b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d2c1586e0 0x7f6d2c0b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15bde0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15bb00 con 0x7f6d2c1586e0 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c071350 con 0x7f6d2c0cffd0 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c145470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38652/0 (socket says 172.21.15.122:38652) 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 -- 172.21.15.122:0/812979740 learned_addr learned my addr 172.21.15.122:0/812979740 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2ad9d640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d2c1586e0 0x7f6d2c0b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d21d9b640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c0cffd0 0x7f6d2c0b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 -- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c0cffd0 msgr2=0x7f6d2c0b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c0cffd0 0x7f6d2c0b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 -- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d2c1586e0 msgr2=0x7f6d2c0b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d2c1586e0 0x7f6d2c0b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 -- 172.21.15.122:0/812979740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c14baf0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d2a59c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c145470 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00e870 tx=0x7f6d1c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c017070 con 0x7f6d2c0aef80 2024-03-06T07:22:26.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c00c940 con 0x7f6d2c0aef80 2024-03-06T07:22:26.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c0b2290 con 0x7f6d2c0aef80 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c00cd30 con 0x7f6d2c0aef80 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.589+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c0b06b0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.591+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0732b0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.591+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d1c01a070 con 0x7f6d2c0aef80 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.591+0000 7f6d237fe640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d0c06b4d0 0x7f6d0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.591+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d1c08a050 con 0x7f6d2c0aef80 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.591+0000 7f6d21d9b640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d0c06b4d0 0x7f6d0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.592+0000 7f6d21d9b640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d0c06b4d0 0x7f6d0c06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d14036000 tx=0x7f6d1403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.596+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d1c058800 con 0x7f6d2c0aef80 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.766+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6d2c068ff0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.767+0000 7f6d237fe640 1 -- 172.21.15.122:0/812979740 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6d1c05c8a0 con 0x7f6d2c0aef80 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d0c06b4d0 msgr2=0x7f6d0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d0c06b4d0 0x7f6d0c06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d14036000 tx=0x7f6d1403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 msgr2=0x7f6d2c145470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c145470 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00e870 tx=0x7f6d1c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 shutdown_connections 2024-03-06T07:22:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d0c06b4d0 0x7f6d0c06d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d2c1586e0 0x7f6d2c0b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d2c0cffd0 0x7f6d2c0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 --2- 172.21.15.122:0/812979740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d2c0aef80 0x7f6d2c145470 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.768+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 >> 172.21.15.122:0/812979740 conn(0x7f6d2c06b190 msgr2=0x7f6d2c157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.769+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 shutdown_connections 2024-03-06T07:22:26.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.769+0000 7f6d3095c640 1 -- 172.21.15.122:0/812979740 wait complete. 2024-03-06T07:22:26.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:26.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.883+0000 7f0da8ba0640 1 Processor -- start 2024-03-06T07:22:26.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.883+0000 7f0da8ba0640 1 -- start start 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da8ba0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da406dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da8ba0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0da406c860 0x7f0da406cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da8ba0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da40cdf50 con 0x7f0da406d1e0 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da8ba0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da40b32b0 con 0x7f0da406c860 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da8ba0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40b3590 con 0x7f0da415bc90 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da2ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0da406c860 0x7f0da406cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da37fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da406dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da3fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f0da406d1e0 0x7f0da415f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:36486/0 (socket says 172.21.15.122:36486) 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da3fff640 1 -- 172.21.15.122:0/130419910 learned_addr learned my addr 172.21.15.122:0/130419910 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da2ffd640 1 -- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 msgr2=0x7f0da406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da2ffd640 1 --2- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da406dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da2ffd640 1 -- 172.21.15.122:0/130419910 >> v1:172.21.15.169:6789/0 conn(0x7f0da406d1e0 legacy=0x7f0da415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.884+0000 7f0da2ffd640 1 -- 172.21.15.122:0/130419910 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da40ce1f0 con 0x7f0da406c860 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.885+0000 7f0da2ffd640 1 --2- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da406cca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0d980097a0 tx=0x7f0d9802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5006d1df1f5f2025 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.885+0000 7f0da27fc640 1 -- 172.21.15.122:0/130419910 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d980315a0 con 0x7f0da406c860 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.885+0000 7f0da27fc640 1 -- 172.21.15.122:0/130419910 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d98037070 con 0x7f0da406c860 2024-03-06T07:22:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.885+0000 7f0da27fc640 1 -- 172.21.15.122:0/130419910 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98031960 con 0x7f0da406c860 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 -- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 msgr2=0x7f0da406cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da406cca0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0d980097a0 tx=0x7f0d9802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 -- 172.21.15.122:0/130419910 shutdown_connections 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da406cca0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/130419910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da406dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 -- 172.21.15.122:0/130419910 >> 172.21.15.122:0/130419910 conn(0x7f0da406afb0 msgr2=0x7f0da406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 -- 172.21.15.122:0/130419910 shutdown_connections 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.886+0000 7f0da8ba0640 1 -- 172.21.15.122:0/130419910 wait complete. 2024-03-06T07:22:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.887+0000 7f0da8ba0640 1 Processor -- start 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.887+0000 7f0da8ba0640 1 -- start start 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 0x7f0da414b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da41519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da415c070 con 0x7f0da406d1e0 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40b3870 con 0x7f0da406c860 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da415c1f0 con 0x7f0da415bc90 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 0x7f0da414b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 0x7f0da414b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38658/0 (socket says 172.21.15.122:38658) 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 -- 172.21.15.122:0/726215898 learned_addr learned my addr 172.21.15.122:0/726215898 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da3fff640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da41519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da37fe640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 -- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 msgr2=0x7f0da41519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da41519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 -- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 msgr2=0x7f0da414b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da414b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 -- 172.21.15.122:0/726215898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da414f900 con 0x7f0da406d1e0 2024-03-06T07:22:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da2ffd640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 0x7f0da414b860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0d9802b880 tx=0x7f0d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98032630 con 0x7f0da406d1e0 2024-03-06T07:22:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d980327d0 con 0x7f0da406d1e0 2024-03-06T07:22:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da414fbe0 con 0x7f0da406d1e0 2024-03-06T07:22:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.888+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98032bc0 con 0x7f0da406d1e0 2024-03-06T07:22:26.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.889+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da41501a0 con 0x7f0da406d1e0 2024-03-06T07:22:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.890+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da406d000 con 0x7f0da406d1e0 2024-03-06T07:22:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.890+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d9804d3f0 con 0x7f0da406d1e0 2024-03-06T07:22:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.890+0000 7f0d8bfff640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9006b4d0 0x7f0d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.890+0000 7f0da37fe640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9006b4d0 0x7f0d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.890+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d9802f030 con 0x7f0da406d1e0 2024-03-06T07:22:26.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.891+0000 7f0da37fe640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9006b4d0 0x7f0d9006d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0da41e6e50 tx=0x7f0d94009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:26.895+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da406d000 con 0x7f0da406d1e0 2024-03-06T07:22:27.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.069+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0da40b4460 con 0x7f0da406d1e0 2024-03-06T07:22:27.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.070+0000 7f0d8bfff640 1 -- 172.21.15.122:0/726215898 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0d98040070 con 0x7f0da406d1e0 2024-03-06T07:22:27.072 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9006b4d0 msgr2=0x7f0d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9006b4d0 0x7f0d9006d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0da41e6e50 tx=0x7f0d94009190 comp rx=0 tx=0).stop 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 msgr2=0x7f0da414b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 0x7f0da414b860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0d9802b880 tx=0x7f0d980046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 shutdown_connections 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9006b4d0 0x7f0d9006d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0da415bc90 0x7f0da41519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0da406d1e0 0x7f0da414b860 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 --2- 172.21.15.122:0/726215898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0da406c860 0x7f0da414b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 >> 172.21.15.122:0/726215898 conn(0x7f0da406afb0 msgr2=0x7f0da4156250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 shutdown_connections 2024-03-06T07:22:27.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.072+0000 7f0da8ba0640 1 -- 172.21.15.122:0/726215898 wait complete. 2024-03-06T07:22:27.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:27.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.187+0000 7f6c100a4640 1 Processor -- start 2024-03-06T07:22:27.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.187+0000 7f6c100a4640 1 -- start start 2024-03-06T07:22:27.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c100a4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6c080cf9a0 0x7f6c080b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c100a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c100a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0806b710 con 0x7f6c080b5410 2024-03-06T07:22:27.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c100a4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0806b150 con 0x7f6c080cf9a0 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c100a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0806b430 con 0x7f6c08066d90 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0e641640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6c080b5410 0x7f6c08072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49630/0 (socket says 172.21.15.122:49630) 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0e641640 1 -- 172.21.15.122:0/2305408394 learned_addr learned my addr 172.21.15.122:0/2305408394 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0d63f640 1 --2- 172.21.15.122:0/2305408394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0de40640 1 --2- 172.21.15.122:0/2305408394 >> v2:172.21.15.169:3300/0 conn(0x7f6c080cf9a0 0x7f6c080b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0ce3e640 1 -- 172.21.15.122:0/2305408394 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436235954 0 0) 0x7f6c0806b710 con 0x7f6c080b5410 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0d63f640 1 -- 172.21.15.122:0/2305408394 >> v2:172.21.15.169:3300/0 conn(0x7f6c080cf9a0 msgr2=0x7f6c080b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0d63f640 1 --2- 172.21.15.122:0/2305408394 >> v2:172.21.15.169:3300/0 conn(0x7f6c080cf9a0 0x7f6c080b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0d63f640 1 -- 172.21.15.122:0/2305408394 >> v1:172.21.15.122:6789/0 conn(0x7f6c080b5410 legacy=0x7f6c08072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.188+0000 7f6c0d63f640 1 -- 172.21.15.122:0/2305408394 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c080b0970 con 0x7f6c08066d90 2024-03-06T07:22:27.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.189+0000 7f6c0d63f640 1 --2- 172.21.15.122:0/2305408394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080b4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6bf80097a0 tx=0x7f6bf802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87a2674950029b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.189+0000 7f6c0ce3e640 1 -- 172.21.15.122:0/2305408394 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8035020 con 0x7f6c08066d90 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.189+0000 7f6c0ce3e640 1 -- 172.21.15.122:0/2305408394 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf8031830 con 0x7f6c08066d90 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.189+0000 7f6c0ce3e640 1 -- 172.21.15.122:0/2305408394 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8031c00 con 0x7f6c08066d90 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.189+0000 7f6c100a4640 1 -- 172.21.15.122:0/2305408394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 msgr2=0x7f6c080b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.189+0000 7f6c100a4640 1 --2- 172.21.15.122:0/2305408394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080b4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6bf80097a0 tx=0x7f6bf802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.190+0000 7f6c100a4640 1 -- 172.21.15.122:0/2305408394 shutdown_connections 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.190+0000 7f6c100a4640 1 --2- 172.21.15.122:0/2305408394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080b4ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.190+0000 7f6c100a4640 1 --2- 172.21.15.122:0/2305408394 >> v2:172.21.15.169:3300/0 conn(0x7f6c080cf9a0 0x7f6c080b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.190+0000 7f6c100a4640 1 -- 172.21.15.122:0/2305408394 >> 172.21.15.122:0/2305408394 conn(0x7f6c080b1d70 msgr2=0x7f6c080b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.190+0000 7f6c100a4640 1 -- 172.21.15.122:0/2305408394 shutdown_connections 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.190+0000 7f6c100a4640 1 -- 172.21.15.122:0/2305408394 wait complete. 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 Processor -- start 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 -- start start 2024-03-06T07:22:27.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c080b5410 0x7f6c08156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 0x7f6c080cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080b6170 con 0x7f6c080cf9a0 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080b5ff0 con 0x7f6c080b5410 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c100a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080cf050 con 0x7f6c08066d90 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.191+0000 7f6c0e641640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 0x7f6c080cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 0x7f6c080cfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38660/0 (socket says 172.21.15.122:38660) 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 -- 172.21.15.122:0/107947256 learned_addr learned my addr 172.21.15.122:0/107947256 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0de40640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0d63f640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c080b5410 0x7f6c08156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 -- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 msgr2=0x7f6c080cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 -- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c080b5410 msgr2=0x7f6c08156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c080b5410 0x7f6c08156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 -- 172.21.15.122:0/107947256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c08158b00 con 0x7f6c080cf9a0 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c0e641640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 0x7f6c080cfd80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000c7a0 tx=0x7f6c0000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c00018070 con 0x7f6c080cf9a0 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0000a640 con 0x7f6c080cf9a0 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c081f43a0 con 0x7f6c080cf9a0 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0000aab0 con 0x7f6c080cf9a0 2024-03-06T07:22:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.192+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c081f4960 con 0x7f6c080cf9a0 2024-03-06T07:22:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.193+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0809e3d0 con 0x7f6c080cf9a0 2024-03-06T07:22:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.194+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c0001b070 con 0x7f6c080cf9a0 2024-03-06T07:22:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.194+0000 7f6bfe7fc640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6bec06b500 0x7f6bec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.194+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c00089770 con 0x7f6c080cf9a0 2024-03-06T07:22:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.194+0000 7f6c0de40640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6bec06b500 0x7f6bec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.194+0000 7f6c0de40640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6bec06b500 0x7f6bec06d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6c080d0b50 tx=0x7f6c04009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.198+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c00057fa0 con 0x7f6c080cf9a0 2024-03-06T07:22:27.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.369+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c08072c40 con 0x7f6c080cf9a0 2024-03-06T07:22:27.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.369+0000 7f6bfe7fc640 1 -- 172.21.15.122:0/107947256 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c0005c040 con 0x7f6c080cf9a0 2024-03-06T07:22:27.371 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:27.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6bec06b500 msgr2=0x7f6bec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6bec06b500 0x7f6bec06d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6c080d0b50 tx=0x7f6c04009190 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 msgr2=0x7f6c080cfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 0x7f6c080cfd80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000c7a0 tx=0x7f6c0000cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 shutdown_connections 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6bec06b500 0x7f6bec06d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c080cf9a0 0x7f6c080cfd80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c080b5410 0x7f6c08156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 --2- 172.21.15.122:0/107947256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c08066d90 0x7f6c080cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 >> 172.21.15.122:0/107947256 conn(0x7f6c080b1d70 msgr2=0x7f6c0806e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 shutdown_connections 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.371+0000 7f6c100a4640 1 -- 172.21.15.122:0/107947256 wait complete. 2024-03-06T07:22:27.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:27.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.483+0000 7f171f4e1640 1 Processor -- start 2024-03-06T07:22:27.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.483+0000 7f171f4e1640 1 -- start start 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171f4e1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1718072520 0x7f1718066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171f4e1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171806c460 con 0x7f17180b5c30 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171f4e1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171806c740 con 0x7f171815d040 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171f4e1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1718067240 con 0x7f1718072520 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171d27d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1718072520 0x7f1718066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171ca7c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f17180b5c30 0x7f1718072900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49632/0 (socket says 172.21.15.122:49632) 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171ca7c640 1 -- 172.21.15.122:0/2963954456 learned_addr learned my addr 172.21.15.122:0/2963954456 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171d27d640 1 -- 172.21.15.122:0/2963954456 >> v1:172.21.15.169:6789/0 conn(0x7f171815d040 legacy=0x7f171815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171d27d640 1 -- 172.21.15.122:0/2963954456 >> v1:172.21.15.122:6789/0 conn(0x7f17180b5c30 legacy=0x7f1718072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f171d27d640 1 -- 172.21.15.122:0/2963954456 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171815fb50 con 0x7f1718072520 2024-03-06T07:22:27.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.484+0000 7f1717fff640 1 -- 172.21.15.122:0/2963954456 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783094605 0 0) 0x7f171806c460 con 0x7f17180b5c30 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171d27d640 1 --2- 172.21.15.122:0/2963954456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1718072520 0x7f1718066d00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1708009d90 tx=0x7f170802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0dea59baf53e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f1717fff640 1 -- 172.21.15.122:0/2963954456 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708039070 con 0x7f1718072520 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f1717fff640 1 -- 172.21.15.122:0/2963954456 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170802cc60 con 0x7f1718072520 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f1717fff640 1 -- 172.21.15.122:0/2963954456 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708034660 con 0x7f1718072520 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171f4e1640 1 -- 172.21.15.122:0/2963954456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1718072520 msgr2=0x7f1718066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171f4e1640 1 --2- 172.21.15.122:0/2963954456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1718072520 0x7f1718066d00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1708009d90 tx=0x7f170802beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171f4e1640 1 -- 172.21.15.122:0/2963954456 shutdown_connections 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171f4e1640 1 --2- 172.21.15.122:0/2963954456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1718072520 0x7f1718066d00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171f4e1640 1 -- 172.21.15.122:0/2963954456 >> 172.21.15.122:0/2963954456 conn(0x7f17180b4e60 msgr2=0x7f17180b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.485+0000 7f171f4e1640 1 -- 172.21.15.122:0/2963954456 shutdown_connections 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.486+0000 7f171f4e1640 1 -- 172.21.15.122:0/2963954456 wait complete. 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 Processor -- start 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 -- start start 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1718072520 0x7f171814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17180b5c30 0x7f171806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 0x7f171806e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171815dd80 con 0x7f171815d040 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171815dc00 con 0x7f17180b5c30 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171f4e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171815df00 con 0x7f1718072520 2024-03-06T07:22:27.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171da7e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 0x7f171806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171da7e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 0x7f171806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38680/0 (socket says 172.21.15.122:38680) 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171da7e640 1 -- 172.21.15.122:0/745392244 learned_addr learned my addr 172.21.15.122:0/745392244 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171d27d640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1718072520 0x7f171814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.487+0000 7f171ca7c640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17180b5c30 0x7f171806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171da7e640 1 -- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1718072520 msgr2=0x7f171814dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171da7e640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1718072520 0x7f171814dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171da7e640 1 -- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17180b5c30 msgr2=0x7f171806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171da7e640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17180b5c30 0x7f171806dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171da7e640 1 -- 172.21.15.122:0/745392244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17181f5620 con 0x7f171815d040 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171da7e640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 0x7f171806e270 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f171000c3c0 tx=0x7f171000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1710017070 con 0x7f171815d040 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17100052d0 con 0x7f171815d040 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17181f5900 con 0x7f171815d040 2024-03-06T07:22:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1710003830 con 0x7f171815d040 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.488+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17181f5ec0 con 0x7f171815d040 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.489+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171804f330 con 0x7f171815d040 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.489+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f171001a070 con 0x7f171815d040 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.490+0000 7f1715ffb640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.490+0000 7f171d27d640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.490+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1710089460 con 0x7f171815d040 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.490+0000 7f171d27d640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1708036000 tx=0x7f1708030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.494+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1710057d20 con 0x7f171815d040 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.679+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1718074290 con 0x7f171815d040 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.680+0000 7f1715ffb640 1 -- 172.21.15.122:0/745392244 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f171005bdc0 con 0x7f171815d040 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 msgr2=0x7f170406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1708036000 tx=0x7f1708030000 comp rx=0 tx=0).stop 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 msgr2=0x7f171806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 0x7f171806e270 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f171000c3c0 tx=0x7f171000c7c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 shutdown_connections 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f171815d040 0x7f171806e270 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17180b5c30 0x7f171806dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 --2- 172.21.15.122:0/745392244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1718072520 0x7f171814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 >> 172.21.15.122:0/745392244 conn(0x7f17180b4e60 msgr2=0x7f171806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 shutdown_connections 2024-03-06T07:22:27.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.682+0000 7f171f4e1640 1 -- 172.21.15.122:0/745392244 wait complete. 2024-03-06T07:22:27.695 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:22:27.695 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:27.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.847+0000 7fd5468b1640 1 Processor -- start 2024-03-06T07:22:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.847+0000 7fd5468b1640 1 -- start start 2024-03-06T07:22:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd5468b1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd5400cf790 0x7fd5400ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd5468b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd5400adda0 0x7fd540071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd5468b1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5400ae240 con 0x7fd540072220 2024-03-06T07:22:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd5468b1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54006cf70 con 0x7fd5400adda0 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd5468b1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5400ae520 con 0x7fd5400cf790 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd53f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd5400adda0 0x7fd540071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd53f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd5400adda0 0x7fd540071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38692/0 (socket says 172.21.15.122:38692) 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd53f7fe640 1 -- 172.21.15.122:0/719135539 learned_addr learned my addr 172.21.15.122:0/719135539 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd544e4e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd540072220 0x7fd5401619b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49646/0 (socket says 172.21.15.122:49646) 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.848+0000 7fd53ffff640 1 --2- 172.21.15.122:0/719135539 >> v2:172.21.15.169:3300/0 conn(0x7fd5400cf790 0x7fd5400ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53f7fe640 1 -- 172.21.15.122:0/719135539 >> v2:172.21.15.169:3300/0 conn(0x7fd5400cf790 msgr2=0x7fd5400ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53f7fe640 1 --2- 172.21.15.122:0/719135539 >> v2:172.21.15.169:3300/0 conn(0x7fd5400cf790 0x7fd5400ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53f7fe640 1 -- 172.21.15.122:0/719135539 >> v1:172.21.15.122:6789/0 conn(0x7fd540072220 legacy=0x7fd5401619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53f7fe640 1 -- 172.21.15.122:0/719135539 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5400ae800 con 0x7fd5400adda0 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53effd640 1 -- 172.21.15.122:0/719135539 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072502412 0 0) 0x7fd5400ae240 con 0x7fd540072220 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53f7fe640 1 --2- 172.21.15.122:0/719135539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5400adda0 0x7fd540071ce0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd5400b27b0 tx=0x7fd53402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b03384e34d4394 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53effd640 1 -- 172.21.15.122:0/719135539 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534034070 con 0x7fd5400adda0 2024-03-06T07:22:27.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53effd640 1 -- 172.21.15.122:0/719135539 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534030860 con 0x7fd5400adda0 2024-03-06T07:22:27.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.849+0000 7fd53effd640 1 -- 172.21.15.122:0/719135539 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534030c30 con 0x7fd5400adda0 2024-03-06T07:22:27.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 -- 172.21.15.122:0/719135539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5400adda0 msgr2=0x7fd540071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 --2- 172.21.15.122:0/719135539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5400adda0 0x7fd540071ce0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd5400b27b0 tx=0x7fd53402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 -- 172.21.15.122:0/719135539 shutdown_connections 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 --2- 172.21.15.122:0/719135539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5400adda0 0x7fd540071ce0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 --2- 172.21.15.122:0/719135539 >> v2:172.21.15.169:3300/0 conn(0x7fd5400cf790 0x7fd5400ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 -- 172.21.15.122:0/719135539 >> 172.21.15.122:0/719135539 conn(0x7fd5400b1a90 msgr2=0x7fd5400b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 -- 172.21.15.122:0/719135539 shutdown_connections 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.850+0000 7fd5468b1640 1 -- 172.21.15.122:0/719135539 wait complete. 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.851+0000 7fd5468b1640 1 Processor -- start 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.851+0000 7fd5468b1640 1 -- start start 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd5468b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 0x7fd54014b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd5468b1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5400adda0 0x7fd5400c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd5468b1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5400cf790 0x7fd5400c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd5468b1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54015e700 con 0x7fd540072220 2024-03-06T07:22:27.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd5468b1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54015e580 con 0x7fd5400adda0 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd5468b1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54006e410 con 0x7fd5400cf790 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 0x7fd54014b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 0x7fd54014b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38706/0 (socket says 172.21.15.122:38706) 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 -- 172.21.15.122:0/818573355 learned_addr learned my addr 172.21.15.122:0/818573355 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53f7fe640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5400adda0 0x7fd5400c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd544e4e640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5400cf790 0x7fd5400c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 -- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5400cf790 msgr2=0x7fd5400c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5400cf790 0x7fd5400c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 -- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5400adda0 msgr2=0x7fd5400c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5400adda0 0x7fd5400c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.852+0000 7fd53ffff640 1 -- 172.21.15.122:0/818573355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5400b3850 con 0x7fd540072220 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.853+0000 7fd53ffff640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 0x7fd54014b780 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd53000e8a0 tx=0x7fd53000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.853+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530017070 con 0x7fd540072220 2024-03-06T07:22:27.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.853+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53000c940 con 0x7fd540072220 2024-03-06T07:22:27.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.853+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53000cd30 con 0x7fd540072220 2024-03-06T07:22:27.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.853+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5400b3b30 con 0x7fd540072220 2024-03-06T07:22:27.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.853+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5400b40f0 con 0x7fd540072220 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.854+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54004f330 con 0x7fd540072220 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.854+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd53001a070 con 0x7fd540072220 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.855+0000 7fd53cff9640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd52c06b4d0 0x7fd52c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.855+0000 7fd53f7fe640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd52c06b4d0 0x7fd52c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.855+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd53008a2f0 con 0x7fd540072220 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.855+0000 7fd53f7fe640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd52c06b4d0 0x7fd52c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd534036000 tx=0x7fd534008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:27.859+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd530058bb0 con 0x7fd540072220 2024-03-06T07:22:28.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.030+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd54006e6b0 con 0x7fd540072220 2024-03-06T07:22:28.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.032+0000 7fd53cff9640 1 -- 172.21.15.122:0/818573355 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7fd53005cc50 con 0x7fd540072220 2024-03-06T07:22:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:28.034 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:28.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd52c06b4d0 msgr2=0x7fd52c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd52c06b4d0 0x7fd52c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd534036000 tx=0x7fd534008030 comp rx=0 tx=0).stop 2024-03-06T07:22:28.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 msgr2=0x7fd54014b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 0x7fd54014b780 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd53000e8a0 tx=0x7fd53000ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 shutdown_connections 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd52c06b4d0 0x7fd52c06d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5400cf790 0x7fd5400c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5400adda0 0x7fd5400c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 --2- 172.21.15.122:0/818573355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd540072220 0x7fd54014b780 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 >> 172.21.15.122:0/818573355 conn(0x7fd5400b1a90 msgr2=0x7fd54015e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 shutdown_connections 2024-03-06T07:22:28.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.034+0000 7fd5468b1640 1 -- 172.21.15.122:0/818573355 wait complete. 2024-03-06T07:22:28.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:28.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:28.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.148+0000 7f5cd475c640 1 Processor -- start 2024-03-06T07:22:28.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.148+0000 7f5cd475c640 1 -- start start 2024-03-06T07:22:28.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd475c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5ccc15bb40 0x7f5ccc0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd475c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5ccc15f7d0 0x7f5ccc06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd475c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc15ff90 con 0x7f5ccc06d5c0 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd475c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc06c910 con 0x7f5ccc15bb40 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd475c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc160270 con 0x7f5ccc15f7d0 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd24f8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5ccc15bb40 0x7f5ccc0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd2cf9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5ccc06d5c0 0x7f5ccc0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49662/0 (socket says 172.21.15.122:49662) 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd2cf9640 1 -- 172.21.15.122:0/2157218364 learned_addr learned my addr 172.21.15.122:0/2157218364 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.149+0000 7f5cd1cf7640 1 --2- 172.21.15.122:0/2157218364 >> v2:172.21.15.169:3300/0 conn(0x7f5ccc15f7d0 0x7f5ccc06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd24f8640 1 -- 172.21.15.122:0/2157218364 >> v2:172.21.15.169:3300/0 conn(0x7f5ccc15f7d0 msgr2=0x7f5ccc06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd24f8640 1 --2- 172.21.15.122:0/2157218364 >> v2:172.21.15.169:3300/0 conn(0x7f5ccc15f7d0 0x7f5ccc06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd24f8640 1 -- 172.21.15.122:0/2157218364 >> v1:172.21.15.122:6789/0 conn(0x7f5ccc06d5c0 legacy=0x7f5ccc0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd14f6640 1 -- 172.21.15.122:0/2157218364 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017432779 0 0) 0x7f5ccc15ff90 con 0x7f5ccc06d5c0 2024-03-06T07:22:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd24f8640 1 -- 172.21.15.122:0/2157218364 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc160550 con 0x7f5ccc15bb40 2024-03-06T07:22:28.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd1cf7640 1 --2- 172.21.15.122:0/2157218364 >> v2:172.21.15.169:3300/0 conn(0x7f5ccc15f7d0 0x7f5ccc06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd24f8640 1 --2- 172.21.15.122:0/2157218364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc0b4970 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc06bfe0 tx=0x7f5cc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d77993fd7febe70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd14f6640 1 -- 172.21.15.122:0/2157218364 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8039070 con 0x7f5ccc15bb40 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd14f6640 1 -- 172.21.15.122:0/2157218364 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc802cc80 con 0x7f5ccc15bb40 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.150+0000 7f5cd14f6640 1 -- 172.21.15.122:0/2157218364 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8034660 con 0x7f5ccc15bb40 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 -- 172.21.15.122:0/2157218364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 msgr2=0x7f5ccc0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2157218364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc0b4970 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc06bfe0 tx=0x7f5cc802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 -- 172.21.15.122:0/2157218364 shutdown_connections 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2157218364 >> v2:172.21.15.169:3300/0 conn(0x7f5ccc15f7d0 0x7f5ccc06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2157218364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc0b4970 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 -- 172.21.15.122:0/2157218364 >> 172.21.15.122:0/2157218364 conn(0x7f5ccc06b190 msgr2=0x7f5ccc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 -- 172.21.15.122:0/2157218364 shutdown_connections 2024-03-06T07:22:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.151+0000 7f5cd475c640 1 -- 172.21.15.122:0/2157218364 wait complete. 2024-03-06T07:22:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.152+0000 7f5cd475c640 1 Processor -- start 2024-03-06T07:22:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.152+0000 7f5cd475c640 1 -- start start 2024-03-06T07:22:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd475c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ccc06d5c0 0x7f5ccc070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd475c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc14b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd475c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ccc15f7d0 0x7f5ccc0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd475c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc15bf20 con 0x7f5ccc15bb40 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd475c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc15fc30 con 0x7f5ccc15f7d0 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd475c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc15c0a0 con 0x7f5ccc06d5c0 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc14b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc14b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38728/0 (socket says 172.21.15.122:38728) 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd24f8640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ccc06d5c0 0x7f5ccc070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 -- 172.21.15.122:0/2333189143 learned_addr learned my addr 172.21.15.122:0/2333189143 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd2cf9640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ccc15f7d0 0x7f5ccc0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 -- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ccc06d5c0 msgr2=0x7f5ccc070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ccc06d5c0 0x7f5ccc070980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 -- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ccc15f7d0 msgr2=0x7f5ccc0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ccc15f7d0 0x7f5ccc0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.153+0000 7f5cd1cf7640 1 -- 172.21.15.122:0/2333189143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc15e5e0 con 0x7f5ccc15bb40 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.154+0000 7f5cd1cf7640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc14b1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00e960 tx=0x7f5cbc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.154+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc017070 con 0x7f5ccc15bb40 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.154+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00ca00 con 0x7f5ccc15bb40 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.154+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc15e8a0 con 0x7f5ccc15bb40 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.154+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00cdd0 con 0x7f5ccc15bb40 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.154+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc15f140 con 0x7f5ccc15bb40 2024-03-06T07:22:28.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.155+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc15ec90 con 0x7f5ccc15bb40 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.155+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cbc01a070 con 0x7f5ccc15bb40 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.155+0000 7f5cc2ffd640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb006b4d0 0x7f5cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.156+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cbc08a040 con 0x7f5ccc15bb40 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.156+0000 7f5cd24f8640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb006b4d0 0x7f5cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.156+0000 7f5cd24f8640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb006b4d0 0x7f5cb006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8036000 tx=0x7f5cc8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.160+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cbc058870 con 0x7f5ccc15bb40 2024-03-06T07:22:28.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.331+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ccc15ea70 con 0x7f5ccc15bb40 2024-03-06T07:22:28.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.332+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/2333189143 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5cbc05c910 con 0x7f5ccc15bb40 2024-03-06T07:22:28.334 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb006b4d0 msgr2=0x7f5cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb006b4d0 0x7f5cb006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8036000 tx=0x7f5cc8030000 comp rx=0 tx=0).stop 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 msgr2=0x7f5ccc14b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc14b1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00e960 tx=0x7f5cbc00ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 shutdown_connections 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb006b4d0 0x7f5cb006d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ccc15f7d0 0x7f5ccc0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ccc15bb40 0x7f5ccc14b1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 --2- 172.21.15.122:0/2333189143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ccc06d5c0 0x7f5ccc070980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 >> 172.21.15.122:0/2333189143 conn(0x7f5ccc06b190 msgr2=0x7f5ccc0af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 shutdown_connections 2024-03-06T07:22:28.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.334+0000 7f5cd475c640 1 -- 172.21.15.122:0/2333189143 wait complete. 2024-03-06T07:22:28.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:28.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.448+0000 7f8a3348a640 1 Processor -- start 2024-03-06T07:22:28.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.449+0000 7f8a3348a640 1 -- start start 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.449+0000 7f8a3348a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c15fc30 0x7f8a2c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.449+0000 7f8a3348a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c0d47f0 con 0x7f8a2c0d06e0 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.449+0000 7f8a3348a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c160070 con 0x7f8a2c0b4d10 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.449+0000 7f8a3348a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0ced70 con 0x7f8a2c15fc30 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a31226640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8a2c0d06e0 0x7f8a2c0d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49664/0 (socket says 172.21.15.122:49664) 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a31226640 1 -- 172.21.15.122:0/1597007398 learned_addr learned my addr 172.21.15.122:0/1597007398 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:28.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a30a25640 1 --2- 172.21.15.122:0/1597007398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c15fc30 0x7f8a2c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723016028 0 0) 0x7f8a2c0d47f0 con 0x7f8a2c0d06e0 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18003520 con 0x7f8a2c0d06e0 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381463833 0 0) 0x7f8a2c160070 con 0x7f8a2c0b4d10 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c0d47f0 con 0x7f8a2c0b4d10 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a30a25640 1 -- 172.21.15.122:0/1597007398 >> v1:172.21.15.169:6789/0 conn(0x7f8a2c0b4d10 legacy=0x7f8a2c0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a30a25640 1 -- 172.21.15.122:0/1597007398 >> v1:172.21.15.122:6789/0 conn(0x7f8a2c0d06e0 legacy=0x7f8a2c0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.450+0000 7f8a30a25640 1 -- 172.21.15.122:0/1597007398 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c0d1ab0 con 0x7f8a2c15fc30 2024-03-06T07:22:28.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.451+0000 7f8a30a25640 1 --2- 172.21.15.122:0/1597007398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c15fc30 0x7f8a2c0b47d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097d0 tx=0x7f8a2802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b30c7d6225c8292d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.451+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28035020 con 0x7f8a2c15fc30 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.451+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a28031890 con 0x7f8a2c15fc30 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.451+0000 7f8a23fff640 1 -- 172.21.15.122:0/1597007398 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28031c60 con 0x7f8a2c15fc30 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.451+0000 7f8a3348a640 1 -- 172.21.15.122:0/1597007398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c15fc30 msgr2=0x7f8a2c0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.451+0000 7f8a3348a640 1 --2- 172.21.15.122:0/1597007398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c15fc30 0x7f8a2c0b47d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097d0 tx=0x7f8a2802b930 comp rx=0 tx=0).stop 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.452+0000 7f8a3348a640 1 -- 172.21.15.122:0/1597007398 shutdown_connections 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.452+0000 7f8a3348a640 1 --2- 172.21.15.122:0/1597007398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c15fc30 0x7f8a2c0b47d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.452+0000 7f8a3348a640 1 -- 172.21.15.122:0/1597007398 >> 172.21.15.122:0/1597007398 conn(0x7f8a2c0d3cd0 msgr2=0x7f8a2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.452+0000 7f8a3348a640 1 -- 172.21.15.122:0/1597007398 shutdown_connections 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.452+0000 7f8a3348a640 1 -- 172.21.15.122:0/1597007398 wait complete. 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 Processor -- start 2024-03-06T07:22:28.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 -- start start 2024-03-06T07:22:28.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a2c0b4d10 0x7f8a2c14e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c0d06e0 0x7f8a2c0ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 0x7f8a2c0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0ce990 con 0x7f8a2c15fc30 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0d4ab0 con 0x7f8a2c0b4d10 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.453+0000 7f8a3348a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0b44d0 con 0x7f8a2c0d06e0 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 0x7f8a2c0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 0x7f8a2c0c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38730/0 (socket says 172.21.15.122:38730) 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 -- 172.21.15.122:0/2056405524 learned_addr learned my addr 172.21.15.122:0/2056405524 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31226640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a2c0b4d10 0x7f8a2c14e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a30a25640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c0d06e0 0x7f8a2c0ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 -- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c0d06e0 msgr2=0x7f8a2c0ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c0d06e0 0x7f8a2c0ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 -- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a2c0b4d10 msgr2=0x7f8a2c14e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a2c0b4d10 0x7f8a2c14e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 -- 172.21.15.122:0/2056405524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c14e650 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a31a27640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 0x7f8a2c0c94b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400c710 tx=0x7f8a2400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a24003b70 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a24003d10 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c0c8410 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2400aa50 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.454+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c0c89d0 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.455+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c0b47d0 con 0x7f8a2c15fc30 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.455+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a2400abf0 con 0x7f8a2c15fc30 2024-03-06T07:22:28.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.456+0000 7f8a21ffb640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8a1006b4d0 0x7f8a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.456+0000 7f8a31226640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8a1006b4d0 0x7f8a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.456+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a24089f90 con 0x7f8a2c15fc30 2024-03-06T07:22:28.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.456+0000 7f8a31226640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8a1006b4d0 0x7f8a1006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0011e0 tx=0x7f8a1c007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.460+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a2c0b47d0 con 0x7f8a2c15fc30 2024-03-06T07:22:28.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.631+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a2c0a1c80 con 0x7f8a2c15fc30 2024-03-06T07:22:28.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.632+0000 7f8a21ffb640 1 -- 172.21.15.122:0/2056405524 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a2c14e650 con 0x7f8a2c15fc30 2024-03-06T07:22:28.634 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:28.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8a1006b4d0 msgr2=0x7f8a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8a1006b4d0 0x7f8a1006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0011e0 tx=0x7f8a1c007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:28.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 msgr2=0x7f8a2c0c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 0x7f8a2c0c94b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400c710 tx=0x7f8a2400cbd0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 shutdown_connections 2024-03-06T07:22:28.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8a1006b4d0 0x7f8a1006d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a2c15fc30 0x7f8a2c0c94b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8a2c0d06e0 0x7f8a2c0ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 --2- 172.21.15.122:0/2056405524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a2c0b4d10 0x7f8a2c14e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 >> 172.21.15.122:0/2056405524 conn(0x7f8a2c0d3cd0 msgr2=0x7f8a2c15e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 shutdown_connections 2024-03-06T07:22:28.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.634+0000 7f8a3348a640 1 -- 172.21.15.122:0/2056405524 wait complete. 2024-03-06T07:22:28.647 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:28.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.747+0000 7fb314a50640 1 Processor -- start 2024-03-06T07:22:28.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.747+0000 7fb314a50640 1 -- start start 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb314a50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb3100cfa90 0x7fb3100cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb314a50640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100b33a0 0x7fb3100b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb314a50640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb310158ac0 con 0x7fb3101586e0 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb314a50640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31015b2a0 con 0x7fb3100cfa90 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb314a50640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31015b540 con 0x7fb3100b33a0 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb3100cfa90 0x7fb3100cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb3100cfa90 0x7fb3100cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38736/0 (socket says 172.21.15.122:38736) 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 -- 172.21.15.122:0/3485159065 learned_addr learned my addr 172.21.15.122:0/3485159065 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb30ed9d640 1 --2- 172.21.15.122:0/3485159065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100b33a0 0x7fb3100b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 -- 172.21.15.122:0/3485159065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100b33a0 msgr2=0x7fb3100b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 --2- 172.21.15.122:0/3485159065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100b33a0 0x7fb3100b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 -- 172.21.15.122:0/3485159065 >> v1:172.21.15.169:6789/0 conn(0x7fb3101586e0 legacy=0x7fb3100aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.748+0000 7fb307fff640 1 -- 172.21.15.122:0/3485159065 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31015b820 con 0x7fb3100cfa90 2024-03-06T07:22:28.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb307fff640 1 --2- 172.21.15.122:0/3485159065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3100cfa90 0x7fb3100cff40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009870 tx=0x7fb2f802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e4681ddf4f03b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb30dd9b640 1 -- 172.21.15.122:0/3485159065 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8034070 con 0x7fb3100cfa90 2024-03-06T07:22:28.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb30dd9b640 1 -- 172.21.15.122:0/3485159065 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8030840 con 0x7fb3100cfa90 2024-03-06T07:22:28.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb30dd9b640 1 -- 172.21.15.122:0/3485159065 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8030c10 con 0x7fb3100cfa90 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 -- 172.21.15.122:0/3485159065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3100cfa90 msgr2=0x7fb3100cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 --2- 172.21.15.122:0/3485159065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3100cfa90 0x7fb3100cff40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009870 tx=0x7fb2f802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 -- 172.21.15.122:0/3485159065 shutdown_connections 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 --2- 172.21.15.122:0/3485159065 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100b33a0 0x7fb3100b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 --2- 172.21.15.122:0/3485159065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3100cfa90 0x7fb3100cff40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 -- 172.21.15.122:0/3485159065 >> 172.21.15.122:0/3485159065 conn(0x7fb31006b190 msgr2=0x7fb31006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 -- 172.21.15.122:0/3485159065 shutdown_connections 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.749+0000 7fb314a50640 1 -- 172.21.15.122:0/3485159065 wait complete. 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 Processor -- start 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 -- start start 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3100b33a0 0x7fb3101485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100cfa90 0x7fb3100b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 0x7fb3100b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100af150 con 0x7fb3101586e0 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31015bde0 con 0x7fb3100b33a0 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.751+0000 7fb314a50640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb310071350 con 0x7fb3100cfa90 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 0x7fb3100b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 0x7fb3100b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38738/0 (socket says 172.21.15.122:38738) 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 -- 172.21.15.122:0/1977893633 learned_addr learned my addr 172.21.15.122:0/1977893633 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30e59c640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3100b33a0 0x7fb3101485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb307fff640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100cfa90 0x7fb3100b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 -- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100cfa90 msgr2=0x7fb3100b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100cfa90 0x7fb3100b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 -- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3100b33a0 msgr2=0x7fb3101485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3100b33a0 0x7fb3101485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 -- 172.21.15.122:0/1977893633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb310148b00 con 0x7fb3101586e0 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb30ed9d640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 0x7fb3100b0af0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb30800efc0 tx=0x7fb30800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308018020 con 0x7fb3101586e0 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308014480 con 0x7fb3101586e0 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308014870 con 0x7fb3101586e0 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31014ee00 con 0x7fb3101586e0 2024-03-06T07:22:28.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.752+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31014f3c0 con 0x7fb3101586e0 2024-03-06T07:22:28.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.753+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3100732b0 con 0x7fb3101586e0 2024-03-06T07:22:28.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.754+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb308005d10 con 0x7fb3101586e0 2024-03-06T07:22:28.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.754+0000 7fb306ffd640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2f006b4d0 0x7fb2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.754+0000 7fb30e59c640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2f006b4d0 0x7fb2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.754+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb30808aa50 con 0x7fb3101586e0 2024-03-06T07:22:28.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.755+0000 7fb30e59c640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2f006b4d0 0x7fb2f006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb310071d60 tx=0x7fb300009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.759+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb308022070 con 0x7fb3101586e0 2024-03-06T07:22:28.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.931+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb31015af00 con 0x7fb3101586e0 2024-03-06T07:22:28.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.932+0000 7fb306ffd640 1 -- 172.21.15.122:0/1977893633 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb310148b00 con 0x7fb3101586e0 2024-03-06T07:22:28.934 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:28.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2f006b4d0 msgr2=0x7fb2f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2f006b4d0 0x7fb2f006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb310071d60 tx=0x7fb300009040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 msgr2=0x7fb3100b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 0x7fb3100b0af0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb30800efc0 tx=0x7fb30800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 shutdown_connections 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2f006b4d0 0x7fb2f006d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3101586e0 0x7fb3100b0af0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3100cfa90 0x7fb3100b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 --2- 172.21.15.122:0/1977893633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3100b33a0 0x7fb3101485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 >> 172.21.15.122:0/1977893633 conn(0x7fb31006b190 msgr2=0x7fb3101578f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 shutdown_connections 2024-03-06T07:22:28.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:28.934+0000 7fb314a50640 1 -- 172.21.15.122:0/1977893633 wait complete. 2024-03-06T07:22:28.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:29.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.048+0000 7f7df8288640 1 Processor -- start 2024-03-06T07:22:29.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.048+0000 7f7df8288640 1 -- start start 2024-03-06T07:22:29.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df8288640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7df006e000 0x7f7df015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df8288640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7df00b4a50 0x7f7df00cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df8288640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df006e770 con 0x7f7df00cfff0 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df8288640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df015e660 con 0x7f7df006e000 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df8288640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df00cce60 con 0x7f7df00b4a50 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df6024640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7df006e000 0x7f7df015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df6825640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7df00cfff0 0x7f7df0072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49674/0 (socket says 172.21.15.122:49674) 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df6825640 1 -- 172.21.15.122:0/2506754185 learned_addr learned my addr 172.21.15.122:0/2506754185 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.049+0000 7f7df5823640 1 --2- 172.21.15.122:0/2506754185 >> v2:172.21.15.169:3300/0 conn(0x7f7df00b4a50 0x7f7df00cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df6024640 1 -- 172.21.15.122:0/2506754185 >> v2:172.21.15.169:3300/0 conn(0x7f7df00b4a50 msgr2=0x7f7df00cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df6024640 1 --2- 172.21.15.122:0/2506754185 >> v2:172.21.15.169:3300/0 conn(0x7f7df00b4a50 0x7f7df00cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df6024640 1 -- 172.21.15.122:0/2506754185 >> v1:172.21.15.122:6789/0 conn(0x7f7df00cfff0 legacy=0x7f7df0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df6024640 1 -- 172.21.15.122:0/2506754185 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df006e490 con 0x7f7df006e000 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df5022640 1 -- 172.21.15.122:0/2506754185 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167398725 0 0) 0x7f7df006e770 con 0x7f7df00cfff0 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df6024640 1 --2- 172.21.15.122:0/2506754185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df006e000 0x7f7df015eaf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7df00b1600 tx=0x7f7dec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77460a9e51afefe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df5022640 1 -- 172.21.15.122:0/2506754185 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec035020 con 0x7f7df006e000 2024-03-06T07:22:29.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df5022640 1 -- 172.21.15.122:0/2506754185 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec031830 con 0x7f7df006e000 2024-03-06T07:22:29.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.050+0000 7f7df5022640 1 -- 172.21.15.122:0/2506754185 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec031c00 con 0x7f7df006e000 2024-03-06T07:22:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 -- 172.21.15.122:0/2506754185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df006e000 msgr2=0x7f7df015eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 --2- 172.21.15.122:0/2506754185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df006e000 0x7f7df015eaf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7df00b1600 tx=0x7f7dec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 -- 172.21.15.122:0/2506754185 shutdown_connections 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 --2- 172.21.15.122:0/2506754185 >> v2:172.21.15.169:3300/0 conn(0x7f7df00b4a50 0x7f7df00cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 --2- 172.21.15.122:0/2506754185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df006e000 0x7f7df015eaf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 -- 172.21.15.122:0/2506754185 >> 172.21.15.122:0/2506754185 conn(0x7f7df00b1c80 msgr2=0x7f7df00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 -- 172.21.15.122:0/2506754185 shutdown_connections 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.051+0000 7f7df8288640 1 -- 172.21.15.122:0/2506754185 wait complete. 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.052+0000 7f7df8288640 1 Processor -- start 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.052+0000 7f7df8288640 1 -- start start 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df8288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7df00b4a50 0x7f7df0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df8288640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 0x7f7df01598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df8288640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df0153a70 0x7f7df00722a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df8288640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0070940 con 0x7f7df00cfff0 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df8288640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df015e2f0 con 0x7f7df0153a70 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df8288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0070ac0 con 0x7f7df00b4a50 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 0x7f7df01598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 0x7f7df01598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38750/0 (socket says 172.21.15.122:38750) 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 -- 172.21.15.122:0/2334402768 learned_addr learned my addr 172.21.15.122:0/2334402768 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df6024640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7df00b4a50 0x7f7df0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df6825640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df0153a70 0x7f7df00722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 -- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7df00b4a50 msgr2=0x7f7df0156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7df00b4a50 0x7f7df0156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 -- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df0153a70 msgr2=0x7f7df00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df0153a70 0x7f7df00722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.053+0000 7f7df5823640 1 -- 172.21.15.122:0/2334402768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df0157440 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.054+0000 7f7df5823640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 0x7f7df01598e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7de000e870 tx=0x7f7de000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.054+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0017070 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.054+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de000c940 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.054+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df0159100 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.054+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de000cd30 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.054+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df0157130 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.055+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df004f330 con 0x7f7df00cfff0 2024-03-06T07:22:29.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.055+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7de001a070 con 0x7f7df00cfff0 2024-03-06T07:22:29.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.055+0000 7f7de6ffd640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dd406b4d0 0x7f7dd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.055+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7de008a3d0 con 0x7f7df00cfff0 2024-03-06T07:22:29.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.056+0000 7f7df6024640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dd406b4d0 0x7f7dd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.056+0000 7f7df6024640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dd406b4d0 0x7f7dd406d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7dec036000 tx=0x7f7dec03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.060+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7de0058c00 con 0x7f7df00cfff0 2024-03-06T07:22:29.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.244+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7df0157720 con 0x7f7df00cfff0 2024-03-06T07:22:29.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.244+0000 7f7de6ffd640 1 -- 172.21.15.122:0/2334402768 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7de005cca0 con 0x7f7df00cfff0 2024-03-06T07:22:29.246 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dd406b4d0 msgr2=0x7f7dd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dd406b4d0 0x7f7dd406d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7dec036000 tx=0x7f7dec03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 msgr2=0x7f7df01598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 0x7f7df01598e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7de000e870 tx=0x7f7de000ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 shutdown_connections 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dd406b4d0 0x7f7dd406d990 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7dec036000 tx=0x7f7dec03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df0153a70 0x7f7df00722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df00cfff0 0x7f7df01598e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 --2- 172.21.15.122:0/2334402768 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7df00b4a50 0x7f7df0156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 >> 172.21.15.122:0/2334402768 conn(0x7f7df00b1c80 msgr2=0x7f7df00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 shutdown_connections 2024-03-06T07:22:29.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.246+0000 7f7df8288640 1 -- 172.21.15.122:0/2334402768 wait complete. 2024-03-06T07:22:29.259 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:22:29.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:29.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:29.287 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:22:29.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:29.287 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:22:29.349 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:22:29.350 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:29.367 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:22:29.367 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:29.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:29.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.524+0000 7f61829d7640 1 Processor -- start 2024-03-06T07:22:29.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.524+0000 7f61829d7640 1 -- start start 2024-03-06T07:22:29.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f61829d7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f617c072140 0x7f617c06db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f61829d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0adf70 con 0x7f617c0cf790 2024-03-06T07:22:29.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f61829d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0ae240 con 0x7f617c071ce0 2024-03-06T07:22:29.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f61829d7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c06cf70 con 0x7f617c072140 2024-03-06T07:22:29.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f6180f74640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f617c072140 0x7f617c06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f617bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f617c0cf790 0x7f617c0ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49686/0 (socket says 172.21.15.122:49686) 2024-03-06T07:22:29.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f617bfff640 1 -- 172.21.15.122:0/1103838848 learned_addr learned my addr 172.21.15.122:0/1103838848 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:29.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f6180f74640 1 -- 172.21.15.122:0/1103838848 >> v1:172.21.15.169:6789/0 conn(0x7f617c071ce0 legacy=0x7f617c06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f6180f74640 1 -- 172.21.15.122:0/1103838848 >> v1:172.21.15.122:6789/0 conn(0x7f617c0cf790 legacy=0x7f617c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.525+0000 7f6180f74640 1 -- 172.21.15.122:0/1103838848 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0ae520 con 0x7f617c072140 2024-03-06T07:22:29.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f6180f74640 1 --2- 172.21.15.122:0/1103838848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c072140 0x7f617c06db10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6174009d90 tx=0x7f617402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7406b7f27a1402db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f617affd640 1 -- 172.21.15.122:0/1103838848 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174039070 con 0x7f617c072140 2024-03-06T07:22:29.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f617affd640 1 -- 172.21.15.122:0/1103838848 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617402cd00 con 0x7f617c072140 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f617affd640 1 -- 172.21.15.122:0/1103838848 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174034660 con 0x7f617c072140 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 -- 172.21.15.122:0/1103838848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c072140 msgr2=0x7f617c06db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 --2- 172.21.15.122:0/1103838848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c072140 0x7f617c06db10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6174009d90 tx=0x7f617402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 -- 172.21.15.122:0/1103838848 shutdown_connections 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 --2- 172.21.15.122:0/1103838848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c072140 0x7f617c06db10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 -- 172.21.15.122:0/1103838848 >> 172.21.15.122:0/1103838848 conn(0x7f617c0b1a90 msgr2=0x7f617c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 -- 172.21.15.122:0/1103838848 shutdown_connections 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.526+0000 7f61829d7640 1 -- 172.21.15.122:0/1103838848 wait complete. 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 Processor -- start 2024-03-06T07:22:29.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 -- start start 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f617c071ce0 0x7f617c153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c072140 0x7f617c155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 0x7f617c1428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0725d0 con 0x7f617c0cf790 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0aeae0 con 0x7f617c072140 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.528+0000 7f61829d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c15e580 con 0x7f617c071ce0 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 0x7f617c1428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 0x7f617c1428c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38764/0 (socket says 172.21.15.122:38764) 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 -- 172.21.15.122:0/2277093903 learned_addr learned my addr 172.21.15.122:0/2277093903 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f617b7fe640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c072140 0x7f617c155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f617bfff640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f617c071ce0 0x7f617c153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 -- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f617c071ce0 msgr2=0x7f617c153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f617c071ce0 0x7f617c153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 -- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c072140 msgr2=0x7f617c155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c072140 0x7f617c155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 -- 172.21.15.122:0/2277093903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c153710 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f617bfff640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f617c071ce0 0x7f617c153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6180f74640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 0x7f617c1428c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6174009d90 tx=0x7f6174004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174039050 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174045070 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c1539f0 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617403e830 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.529+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c1423a0 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.530+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c153bf0 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.531+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f617403e9d0 con 0x7f617c0cf790 2024-03-06T07:22:29.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.531+0000 7f6178ff9640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616806b4d0 0x7f616806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.531+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61740acf10 con 0x7f617c0cf790 2024-03-06T07:22:29.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.531+0000 7f617bfff640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616806b4d0 0x7f616806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.531+0000 7f617bfff640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616806b4d0 0x7f616806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f616c005880 tx=0x7f616c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.535+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f617407beb0 con 0x7f617c0cf790 2024-03-06T07:22:29.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.709+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f617c04f330 con 0x7f617c0cf790 2024-03-06T07:22:29.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.710+0000 7f6178ff9640 1 -- 172.21.15.122:0/2277093903 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f617407b7b0 con 0x7f617c0cf790 2024-03-06T07:22:29.712 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:29.712 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616806b4d0 msgr2=0x7f616806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616806b4d0 0x7f616806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f616c005880 tx=0x7f616c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 msgr2=0x7f617c1428c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 0x7f617c1428c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6174009d90 tx=0x7f6174004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 shutdown_connections 2024-03-06T07:22:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616806b4d0 0x7f616806d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0cf790 0x7f617c1428c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c072140 0x7f617c155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 --2- 172.21.15.122:0/2277093903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f617c071ce0 0x7f617c153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 >> 172.21.15.122:0/2277093903 conn(0x7f617c0b1a90 msgr2=0x7f617c15e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 shutdown_connections 2024-03-06T07:22:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.712+0000 7f61829d7640 1 -- 172.21.15.122:0/2277093903 wait complete. 2024-03-06T07:22:29.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:29.725 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:29.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.826+0000 7f9951e6d640 1 Processor -- start 2024-03-06T07:22:29.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.826+0000 7f9951e6d640 1 -- start start 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f9951e6d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15bb40 0x7f994c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f9951e6d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f994c15f7d0 0x7f994c06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f9951e6d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f994c06d5c0 0x7f994c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f9951e6d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c06c910 con 0x7f994c15f7d0 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f9951e6d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c15ff90 con 0x7f994c06d5c0 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f9951e6d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c160270 con 0x7f994c15bb40 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f994c15f7d0 0x7f994c06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f994c15f7d0 0x7f994c06d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38776/0 (socket says 172.21.15.122:38776) 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 -- 172.21.15.122:0/3431982480 learned_addr learned my addr 172.21.15.122:0/3431982480 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994bfff640 1 --2- 172.21.15.122:0/3431982480 >> v2:172.21.15.169:3300/0 conn(0x7f994c06d5c0 0x7f994c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994b7fe640 1 --2- 172.21.15.122:0/3431982480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15bb40 0x7f994c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 -- 172.21.15.122:0/3431982480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15bb40 msgr2=0x7f994c0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 --2- 172.21.15.122:0/3431982480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15bb40 0x7f994c0b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 -- 172.21.15.122:0/3431982480 >> v2:172.21.15.169:3300/0 conn(0x7f994c06d5c0 msgr2=0x7f994c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 --2- 172.21.15.122:0/3431982480 >> v2:172.21.15.169:3300/0 conn(0x7f994c06d5c0 0x7f994c0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.827+0000 7f994affd640 1 -- 172.21.15.122:0/3431982480 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c160550 con 0x7f994c15f7d0 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.828+0000 7f994affd640 1 --2- 172.21.15.122:0/3431982480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15f7d0 0x7f994c06d080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f994c0b4180 tx=0x7f993c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8169a96a7a66c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.828+0000 7f994a7fc640 1 -- 172.21.15.122:0/3431982480 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c035020 con 0x7f994c15f7d0 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.828+0000 7f994a7fc640 1 -- 172.21.15.122:0/3431982480 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c031870 con 0x7f994c15f7d0 2024-03-06T07:22:29.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.828+0000 7f994a7fc640 1 -- 172.21.15.122:0/3431982480 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c031c40 con 0x7f994c15f7d0 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.828+0000 7f9951e6d640 1 -- 172.21.15.122:0/3431982480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15f7d0 msgr2=0x7f994c06d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.828+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3431982480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15f7d0 0x7f994c06d080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f994c0b4180 tx=0x7f993c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 -- 172.21.15.122:0/3431982480 shutdown_connections 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3431982480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15f7d0 0x7f994c06d080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3431982480 >> v2:172.21.15.169:3300/0 conn(0x7f994c06d5c0 0x7f994c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3431982480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15bb40 0x7f994c0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 -- 172.21.15.122:0/3431982480 >> 172.21.15.122:0/3431982480 conn(0x7f994c06b190 msgr2=0x7f994c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 -- 172.21.15.122:0/3431982480 shutdown_connections 2024-03-06T07:22:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.829+0000 7f9951e6d640 1 -- 172.21.15.122:0/3431982480 wait complete. 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.830+0000 7f9951e6d640 1 Processor -- start 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.830+0000 7f9951e6d640 1 -- start start 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994c06d5c0 0x7f994c1f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 0x7f994c1f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15f7d0 0x7f994c153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15bf20 con 0x7f994c15bb40 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15fc30 con 0x7f994c06d5c0 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15c0a0 con 0x7f994c15f7d0 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 0x7f994c1f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 0x7f994c1f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38782/0 (socket says 172.21.15.122:38782) 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 -- 172.21.15.122:0/3174891425 learned_addr learned my addr 172.21.15.122:0/3174891425 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994bfff640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15f7d0 0x7f994c153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994b7fe640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994c06d5c0 0x7f994c1f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 -- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15f7d0 msgr2=0x7f994c153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15f7d0 0x7f994c153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 -- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994c06d5c0 msgr2=0x7f994c1f49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994c06d5c0 0x7f994c1f49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 -- 172.21.15.122:0/3174891425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c1f4f10 con 0x7f994c15bb40 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f994affd640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 0x7f994c1f2590 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f993c02b950 tx=0x7f993c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c040070 con 0x7f994c15bb40 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c031e10 con 0x7f994c15bb40 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c1f2ad0 con 0x7f994c15bb40 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c035040 con 0x7f994c15bb40 2024-03-06T07:22:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.831+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c15ca80 con 0x7f994c15bb40 2024-03-06T07:22:29.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.832+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c0732b0 con 0x7f994c15bb40 2024-03-06T07:22:29.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.833+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993c04b060 con 0x7f994c15bb40 2024-03-06T07:22:29.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.833+0000 7f9933fff640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f993806b4d0 0x7f993806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.833+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f993c0abce0 con 0x7f994c15bb40 2024-03-06T07:22:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.833+0000 7f994b7fe640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f993806b4d0 0x7f993806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.834+0000 7f994b7fe640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f993806b4d0 0x7f993806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f994c15b4e0 tx=0x7f9940009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:29.838+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f993c07a510 con 0x7f994c15bb40 2024-03-06T07:22:30.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.007+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f994c068ff0 con 0x7f994c15bb40 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.008+0000 7f9933fff640 1 -- 172.21.15.122:0/3174891425 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f993c07e5b0 con 0x7f994c15bb40 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f993806b4d0 msgr2=0x7f993806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f993806b4d0 0x7f993806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f994c15b4e0 tx=0x7f9940009190 comp rx=0 tx=0).stop 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 msgr2=0x7f994c1f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 0x7f994c1f2590 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f993c02b950 tx=0x7f993c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 shutdown_connections 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f993806b4d0 0x7f993806d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994c15f7d0 0x7f994c153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994c15bb40 0x7f994c1f2590 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 --2- 172.21.15.122:0/3174891425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994c06d5c0 0x7f994c1f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.009+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 >> 172.21.15.122:0/3174891425 conn(0x7f994c06b190 msgr2=0x7f994c0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.010+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 shutdown_connections 2024-03-06T07:22:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.010+0000 7f9951e6d640 1 -- 172.21.15.122:0/3174891425 wait complete. 2024-03-06T07:22:30.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.123+0000 7fa524356640 1 Processor -- start 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.123+0000 7fa524356640 1 -- start start 2024-03-06T07:22:30.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa524356640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa51c066d00 0x7fa51c166c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa524356640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c06b050 con 0x7fa51c0b0db0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa524356640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c06b2c0 con 0x7fa51c15f4f0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa524356640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51c0b0890 con 0x7fa51c066d00 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa5218f1640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa51c0b0db0 0x7fa51c0ce750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49694/0 (socket says 172.21.15.122:49694) 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa5218f1640 1 -- 172.21.15.122:0/3464796218 learned_addr learned my addr 172.21.15.122:0/3464796218 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.124+0000 7fa5228f3640 1 --2- 172.21.15.122:0/3464796218 >> v2:172.21.15.169:3300/0 conn(0x7fa51c066d00 0x7fa51c166c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173352174 0 0) 0x7fa51c06b050 con 0x7fa51c0b0db0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa508003520 con 0x7fa51c0b0db0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667491216 0 0) 0x7fa51c06b2c0 con 0x7fa51c15f4f0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c06b050 con 0x7fa51c15f4f0 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5228f3640 1 -- 172.21.15.122:0/3464796218 >> v1:172.21.15.169:6789/0 conn(0x7fa51c15f4f0 legacy=0x7fa51c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5228f3640 1 -- 172.21.15.122:0/3464796218 >> v1:172.21.15.122:6789/0 conn(0x7fa51c0b0db0 legacy=0x7fa51c0ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5228f3640 1 -- 172.21.15.122:0/3464796218 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0672e0 con 0x7fa51c066d00 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.125+0000 7fa5228f3640 1 --2- 172.21.15.122:0/3464796218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c066d00 0x7fa51c166c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa514009d90 tx=0x7fa51402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ff681a6619fb46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514039070 con 0x7fa51c066d00 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51402cd00 con 0x7fa51c066d00 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa5210f0640 1 -- 172.21.15.122:0/3464796218 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514034640 con 0x7fa51c066d00 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 -- 172.21.15.122:0/3464796218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c066d00 msgr2=0x7fa51c166c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 --2- 172.21.15.122:0/3464796218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c066d00 0x7fa51c166c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa514009d90 tx=0x7fa51402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 -- 172.21.15.122:0/3464796218 shutdown_connections 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 --2- 172.21.15.122:0/3464796218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c066d00 0x7fa51c166c90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 -- 172.21.15.122:0/3464796218 >> 172.21.15.122:0/3464796218 conn(0x7fa51c0b1d70 msgr2=0x7fa51c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 -- 172.21.15.122:0/3464796218 shutdown_connections 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.126+0000 7fa524356640 1 -- 172.21.15.122:0/3464796218 wait complete. 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.127+0000 7fa524356640 1 Processor -- start 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.127+0000 7fa524356640 1 -- start start 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa524356640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa51c066d00 0x7fa51c1f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa524356640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 0x7fa51c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa524356640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c15f4f0 0x7fa51c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa524356640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c15f8d0 con 0x7fa51c0b0db0 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa524356640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c06b5a0 con 0x7fa51c15f4f0 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa524356640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c1f6050 con 0x7fa51c066d00 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 0x7fa51c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 0x7fa51c0ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38792/0 (socket says 172.21.15.122:38792) 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 -- 172.21.15.122:0/232767159 learned_addr learned my addr 172.21.15.122:0/232767159 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5228f3640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c15f4f0 0x7fa51c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5220f2640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa51c066d00 0x7fa51c1f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 -- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa51c066d00 msgr2=0x7fa51c1f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa51c066d00 0x7fa51c1f1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 -- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c15f4f0 msgr2=0x7fa51c06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c15f4f0 0x7fa51c06d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.128+0000 7fa5218f1640 1 -- 172.21.15.122:0/232767159 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0cd800 con 0x7fa51c0b0db0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.129+0000 7fa5218f1640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 0x7fa51c0ccd90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00c500 tx=0x7fa50c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.129+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c017070 con 0x7fa51c0b0db0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.129+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c005ce0 con 0x7fa51c0b0db0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.129+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c1f5a00 con 0x7fa51c0b0db0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.129+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c0072f0 con 0x7fa51c0b0db0 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.129+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c1f1fb0 con 0x7fa51c0b0db0 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.130+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c09e3d0 con 0x7fa51c0b0db0 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.130+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa50c01a070 con 0x7fa51c0b0db0 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.131+0000 7fa512ffd640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa50006b4d0 0x7fa50006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.131+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa50c089550 con 0x7fa51c0b0db0 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.131+0000 7fa5220f2640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa50006b4d0 0x7fa50006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.131+0000 7fa5220f2640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa50006b4d0 0x7fa50006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa518007fd0 tx=0x7fa518009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.135+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa50c057d90 con 0x7fa51c0b0db0 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.306+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa51c1f61d0 con 0x7fa51c0b0db0 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.307+0000 7fa512ffd640 1 -- 172.21.15.122:0/232767159 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa50c05be30 con 0x7fa51c0b0db0 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa50006b4d0 msgr2=0x7fa50006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa50006b4d0 0x7fa50006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa518007fd0 tx=0x7fa518009040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 msgr2=0x7fa51c0ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 0x7fa51c0ccd90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00c500 tx=0x7fa50c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 shutdown_connections 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa50006b4d0 0x7fa50006d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa51c15f4f0 0x7fa51c06d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa51c0b0db0 0x7fa51c0ccd90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 --2- 172.21.15.122:0/232767159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa51c066d00 0x7fa51c1f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.308+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 >> 172.21.15.122:0/232767159 conn(0x7fa51c0b1d70 msgr2=0x7fa51c072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.309+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 shutdown_connections 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.309+0000 7fa524356640 1 -- 172.21.15.122:0/232767159 wait complete. 2024-03-06T07:22:30.321 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:30.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.423+0000 7fb272639640 1 Processor -- start 2024-03-06T07:22:30.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.423+0000 7fb272639640 1 -- start start 2024-03-06T07:22:30.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb272639640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb26c15a220 0x7fb26c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb272639640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb26c06da00 0x7fb26c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb272639640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c0b3320 con 0x7fb26c0712d0 2024-03-06T07:22:30.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb272639640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0cf610 con 0x7fb26c15a220 2024-03-06T07:22:30.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb272639640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c071c70 con 0x7fb26c06da00 2024-03-06T07:22:30.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb26c15a220 0x7fb26c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb26c0712d0 0x7fb26c15de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49706/0 (socket says 172.21.15.122:49706) 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26b7fe640 1 -- 172.21.15.122:0/3350983193 learned_addr learned my addr 172.21.15.122:0/3350983193 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb270bd6640 1 --2- 172.21.15.122:0/3350983193 >> v2:172.21.15.169:3300/0 conn(0x7fb26c06da00 0x7fb26c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26bfff640 1 -- 172.21.15.122:0/3350983193 >> v2:172.21.15.169:3300/0 conn(0x7fb26c06da00 msgr2=0x7fb26c071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26bfff640 1 --2- 172.21.15.122:0/3350983193 >> v2:172.21.15.169:3300/0 conn(0x7fb26c06da00 0x7fb26c071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26bfff640 1 -- 172.21.15.122:0/3350983193 >> v1:172.21.15.122:6789/0 conn(0x7fb26c0712d0 legacy=0x7fb26c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26bfff640 1 -- 172.21.15.122:0/3350983193 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0728d0 con 0x7fb26c15a220 2024-03-06T07:22:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.424+0000 7fb26affd640 1 -- 172.21.15.122:0/3350983193 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711543731 0 0) 0x7fb26c0b3320 con 0x7fb26c0712d0 2024-03-06T07:22:30.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.425+0000 7fb26bfff640 1 --2- 172.21.15.122:0/3350983193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c15a220 0x7fb26c0cf0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb26c15ef00 tx=0x7fb26002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef944576a06b7fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.425+0000 7fb26affd640 1 -- 172.21.15.122:0/3350983193 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2600315a0 con 0x7fb26c15a220 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.425+0000 7fb26affd640 1 -- 172.21.15.122:0/3350983193 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb260037070 con 0x7fb26c15a220 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.425+0000 7fb26affd640 1 -- 172.21.15.122:0/3350983193 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260031960 con 0x7fb26c15a220 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.425+0000 7fb272639640 1 -- 172.21.15.122:0/3350983193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c15a220 msgr2=0x7fb26c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.425+0000 7fb272639640 1 --2- 172.21.15.122:0/3350983193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c15a220 0x7fb26c0cf0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb26c15ef00 tx=0x7fb26002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.426+0000 7fb272639640 1 -- 172.21.15.122:0/3350983193 shutdown_connections 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.426+0000 7fb272639640 1 --2- 172.21.15.122:0/3350983193 >> v2:172.21.15.169:3300/0 conn(0x7fb26c06da00 0x7fb26c071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.426+0000 7fb272639640 1 --2- 172.21.15.122:0/3350983193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c15a220 0x7fb26c0cf0d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.426+0000 7fb272639640 1 -- 172.21.15.122:0/3350983193 >> 172.21.15.122:0/3350983193 conn(0x7fb26c06b190 msgr2=0x7fb26c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.426+0000 7fb272639640 1 -- 172.21.15.122:0/3350983193 shutdown_connections 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.426+0000 7fb272639640 1 -- 172.21.15.122:0/3350983193 wait complete. 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.427+0000 7fb272639640 1 Processor -- start 2024-03-06T07:22:30.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.427+0000 7fb272639640 1 -- start start 2024-03-06T07:22:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 0x7fb26c158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb26c0712d0 0x7fb26c1f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb26c15a220 0x7fb26c1f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c072030 con 0x7fb26c06da00 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c072bb0 con 0x7fb26c0712d0 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c072530 con 0x7fb26c15a220 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 0x7fb26c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 0x7fb26c158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38800/0 (socket says 172.21.15.122:38800) 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 -- 172.21.15.122:0/1944004567 learned_addr learned my addr 172.21.15.122:0/1944004567 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26b7fe640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb26c0712d0 0x7fb26c1f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb270bd6640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb26c15a220 0x7fb26c1f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 -- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb26c15a220 msgr2=0x7fb26c1f5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb26c15a220 0x7fb26c1f5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 -- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb26c0712d0 msgr2=0x7fb26c1f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb26c0712d0 0x7fb26c1f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 -- 172.21.15.122:0/1944004567 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c154ce0 con 0x7fb26c06da00 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb270bd6640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb26c15a220 0x7fb26c1f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb26bfff640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 0x7fb26c158420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb260034040 tx=0x7fb2600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260004230 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2600043d0 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c154fc0 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260003850 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.428+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c0a94f0 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.429+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c0732b0 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.430+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2600039f0 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.430+0000 7fb268ff9640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb25806b4d0 0x7fb25806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.430+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb26002f030 con 0x7fb26c06da00 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.430+0000 7fb26b7fe640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb25806b4d0 0x7fb25806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.431+0000 7fb26b7fe640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb25806b4d0 0x7fb25806d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0cb140 tx=0x7fb25c007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.434+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb260041070 con 0x7fb26c06da00 2024-03-06T07:22:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.603+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb26c070ba0 con 0x7fb26c06da00 2024-03-06T07:22:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.604+0000 7fb268ff9640 1 -- 172.21.15.122:0/1944004567 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb26007b1b0 con 0x7fb26c06da00 2024-03-06T07:22:30.606 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:30.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb25806b4d0 msgr2=0x7fb25806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb25806b4d0 0x7fb25806d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0cb140 tx=0x7fb25c007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 msgr2=0x7fb26c158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 0x7fb26c158420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb260034040 tx=0x7fb2600046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 shutdown_connections 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb25806b4d0 0x7fb25806d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb26c15a220 0x7fb26c1f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb26c0712d0 0x7fb26c1f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 --2- 172.21.15.122:0/1944004567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb26c06da00 0x7fb26c158420 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 >> 172.21.15.122:0/1944004567 conn(0x7fb26c06b190 msgr2=0x7fb26c0cd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 shutdown_connections 2024-03-06T07:22:30.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.606+0000 7fb272639640 1 -- 172.21.15.122:0/1944004567 wait complete. 2024-03-06T07:22:30.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:30.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.718+0000 7fafa6501640 1 Processor -- start 2024-03-06T07:22:30.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.718+0000 7fafa6501640 1 -- start start 2024-03-06T07:22:30.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa6501640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fafa0162400 0x7fafa00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa6501640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00d22c0 con 0x7fafa00b4460 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa6501640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00d25a0 con 0x7fafa0162970 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa6501640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa0161b00 con 0x7fafa0162400 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7faf9ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fafa00b4460 0x7fafa00b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49708/0 (socket says 172.21.15.122:49708) 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7faf9ffff640 1 -- 172.21.15.122:0/1518825707 learned_addr learned my addr 172.21.15.122:0/1518825707 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa4a9e640 1 --2- 172.21.15.122:0/1518825707 >> v2:172.21.15.169:3300/0 conn(0x7fafa0162400 0x7fafa00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7faf9effd640 1 -- 172.21.15.122:0/1518825707 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617543113 0 0) 0x7fafa00d22c0 con 0x7fafa00b4460 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7faf9effd640 1 -- 172.21.15.122:0/1518825707 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf88003520 con 0x7fafa00b4460 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa4a9e640 1 -- 172.21.15.122:0/1518825707 >> v1:172.21.15.169:6789/0 conn(0x7fafa0162970 legacy=0x7fafa0161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa4a9e640 1 -- 172.21.15.122:0/1518825707 >> v1:172.21.15.122:6789/0 conn(0x7fafa00b4460 legacy=0x7fafa00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7faf9effd640 1 -- 172.21.15.122:0/1518825707 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540302224 0 0) 0x7fafa00d25a0 con 0x7fafa0162970 2024-03-06T07:22:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.719+0000 7fafa4a9e640 1 -- 172.21.15.122:0/1518825707 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00d2880 con 0x7fafa0162400 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.720+0000 7fafa4a9e640 1 --2- 172.21.15.122:0/1518825707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00d4800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faf98009d90 tx=0x7faf9802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a2887a80d22ed84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.720+0000 7faf9effd640 1 -- 172.21.15.122:0/1518825707 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98039070 con 0x7fafa0162400 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.720+0000 7faf9effd640 1 -- 172.21.15.122:0/1518825707 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9802cd00 con 0x7fafa0162400 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.720+0000 7faf9effd640 1 -- 172.21.15.122:0/1518825707 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98034640 con 0x7fafa0162400 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 -- 172.21.15.122:0/1518825707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 msgr2=0x7fafa00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 --2- 172.21.15.122:0/1518825707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00d4800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faf98009d90 tx=0x7faf9802bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 -- 172.21.15.122:0/1518825707 shutdown_connections 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 --2- 172.21.15.122:0/1518825707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00d4800 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 -- 172.21.15.122:0/1518825707 >> 172.21.15.122:0/1518825707 conn(0x7fafa00d3ce0 msgr2=0x7fafa00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 -- 172.21.15.122:0/1518825707 shutdown_connections 2024-03-06T07:22:30.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.721+0000 7fafa6501640 1 -- 172.21.15.122:0/1518825707 wait complete. 2024-03-06T07:22:30.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.722+0000 7fafa6501640 1 Processor -- start 2024-03-06T07:22:30.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.722+0000 7fafa6501640 1 -- start start 2024-03-06T07:22:30.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa6501640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafa00b4460 0x7fafa00b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa6501640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa6501640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 0x7fafa01588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa6501640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00ce950 con 0x7fafa0162970 2024-03-06T07:22:30.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa6501640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00cecc0 con 0x7fafa0162400 2024-03-06T07:22:30.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa6501640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00b4840 con 0x7fafa00b4460 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 0x7fafa01588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 0x7fafa01588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38810/0 (socket says 172.21.15.122:38810) 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 -- 172.21.15.122:0/2399511524 learned_addr learned my addr 172.21.15.122:0/2399511524 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7faf9ffff640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafa00b4460 0x7fafa00b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7faf9f7fe640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 -- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafa00b4460 msgr2=0x7fafa00b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafa00b4460 0x7fafa00b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 -- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 msgr2=0x7fafa00c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 -- 172.21.15.122:0/2399511524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00c9390 con 0x7fafa0162970 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.723+0000 7fafa4a9e640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 0x7fafa01588c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf98037040 tx=0x7faf98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.724+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98039050 con 0x7fafa0162970 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.724+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf98045070 con 0x7fafa0162970 2024-03-06T07:22:30.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.724+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa00c9670 con 0x7fafa0162970 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.724+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98009450 con 0x7fafa0162970 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.724+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa00c9c30 con 0x7fafa0162970 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.725+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa004f330 con 0x7fafa0162970 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.725+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf9803e830 con 0x7fafa0162970 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.725+0000 7faf9cff9640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf8c06b4d0 0x7faf8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.725+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf980ad100 con 0x7fafa0162970 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.725+0000 7faf9ffff640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf8c06b4d0 0x7faf8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.726+0000 7faf9ffff640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf8c06b4d0 0x7faf8c06d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf90005880 tx=0x7faf90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.730+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf9807ba50 con 0x7fafa0162970 2024-03-06T07:22:30.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.914+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fafa00ce010 con 0x7fafa0162970 2024-03-06T07:22:30.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.914+0000 7faf9cff9640 1 -- 172.21.15.122:0/2399511524 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7faf9807faf0 con 0x7fafa0162970 2024-03-06T07:22:30.917 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf8c06b4d0 msgr2=0x7faf8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf8c06b4d0 0x7faf8c06d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf90005880 tx=0x7faf90009040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 msgr2=0x7fafa01588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 0x7fafa01588c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf98037040 tx=0x7faf98004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 shutdown_connections 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf8c06b4d0 0x7faf8c06d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafa0162970 0x7fafa01588c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafa0162400 0x7fafa00c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 --2- 172.21.15.122:0/2399511524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafa00b4460 0x7fafa00b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 >> 172.21.15.122:0/2399511524 conn(0x7fafa00d3ce0 msgr2=0x7fafa00d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 shutdown_connections 2024-03-06T07:22:30.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:30.916+0000 7fafa6501640 1 -- 172.21.15.122:0/2399511524 wait complete. 2024-03-06T07:22:30.929 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:22:30.929 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:30.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:31.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.043+0000 7f3d3440e640 1 Processor -- start 2024-03-06T07:22:31.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.044+0000 7f3d3440e640 1 -- start start 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.044+0000 7f3d3440e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 0x7f3d2c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.044+0000 7f3d3440e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3d2c0b41f0 0x7f3d2c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.044+0000 7f3d3440e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c0b3430 con 0x7f3d2c15dfe0 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.044+0000 7f3d3440e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c0d2c50 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.044+0000 7f3d3440e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0b31e0 con 0x7f3d2c0d22b0 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 0x7f3d2c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d329ab640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3d2c0b41f0 0x7f3d2c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 0x7f3d2c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:43540/0 (socket says 172.21.15.122:43540) 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 -- 172.21.15.122:0/2516181955 learned_addr learned my addr 172.21.15.122:0/2516181955 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d321aa640 1 --1- 172.21.15.122:0/2516181955 >> v1:172.21.15.169:6789/0 conn(0x7f3d2c15dfe0 0x7f3d2c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:36542/0 (socket says 172.21.15.122:36542) 2024-03-06T07:22:31.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 -- 172.21.15.122:0/2516181955 >> v2:172.21.15.169:3300/0 conn(0x7f3d2c0b41f0 msgr2=0x7f3d2c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 --2- 172.21.15.122:0/2516181955 >> v2:172.21.15.169:3300/0 conn(0x7f3d2c0b41f0 0x7f3d2c0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 -- 172.21.15.122:0/2516181955 >> v1:172.21.15.169:6789/0 conn(0x7f3d2c15dfe0 legacy=0x7f3d2c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d319a9640 1 -- 172.21.15.122:0/2516181955 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c0b3710 con 0x7f3d2c0d22b0 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.045+0000 7f3d329ab640 1 --2- 172.21.15.122:0/2516181955 >> v2:172.21.15.169:3300/0 conn(0x7f3d2c0b41f0 0x7f3d2c0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.046+0000 7f3d319a9640 1 --2- 172.21.15.122:0/2516181955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 0x7f3d2c0d2710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61c231042138e3ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.046+0000 7f3d311a8640 1 -- 172.21.15.122:0/2516181955 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20034070 con 0x7f3d2c0d22b0 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.046+0000 7f3d311a8640 1 -- 172.21.15.122:0/2516181955 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d20030840 con 0x7f3d2c0d22b0 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.046+0000 7f3d311a8640 1 -- 172.21.15.122:0/2516181955 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20030c10 con 0x7f3d2c0d22b0 2024-03-06T07:22:31.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.046+0000 7f3d3440e640 1 -- 172.21.15.122:0/2516181955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 msgr2=0x7f3d2c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.046+0000 7f3d3440e640 1 --2- 172.21.15.122:0/2516181955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 0x7f3d2c0d2710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.047+0000 7f3d3440e640 1 -- 172.21.15.122:0/2516181955 shutdown_connections 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.047+0000 7f3d3440e640 1 --2- 172.21.15.122:0/2516181955 >> v2:172.21.15.169:3300/0 conn(0x7f3d2c0b41f0 0x7f3d2c0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.047+0000 7f3d3440e640 1 --2- 172.21.15.122:0/2516181955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c0d22b0 0x7f3d2c0d2710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.047+0000 7f3d3440e640 1 -- 172.21.15.122:0/2516181955 >> 172.21.15.122:0/2516181955 conn(0x7f3d2c0d3cd0 msgr2=0x7f3d2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.047+0000 7f3d3440e640 1 -- 172.21.15.122:0/2516181955 shutdown_connections 2024-03-06T07:22:31.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.047+0000 7f3d3440e640 1 -- 172.21.15.122:0/2516181955 wait complete. 2024-03-06T07:22:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 Processor -- start 2024-03-06T07:22:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 -- start start 2024-03-06T07:22:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 0x7f3d2c1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d2c0d22b0 0x7f3d2c0cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c15dfe0 0x7f3d2c0cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0cfa00 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0b3d60 con 0x7f3d2c0d22b0 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d3440e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0d1c70 con 0x7f3d2c15dfe0 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d321aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 0x7f3d2c1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d321aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 0x7f3d2c1f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38818/0 (socket says 172.21.15.122:38818) 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.048+0000 7f3d321aa640 1 -- 172.21.15.122:0/1164553820 learned_addr learned my addr 172.21.15.122:0/1164553820 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d319a9640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d2c0d22b0 0x7f3d2c0cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d329ab640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c15dfe0 0x7f3d2c0cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d321aa640 1 -- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c15dfe0 msgr2=0x7f3d2c0cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d321aa640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c15dfe0 0x7f3d2c0cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d321aa640 1 -- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d2c0d22b0 msgr2=0x7f3d2c0cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d321aa640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d2c0d22b0 0x7f3d2c0cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d321aa640 1 -- 172.21.15.122:0/1164553820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c0cdbd0 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d321aa640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 0x7f3d2c1f6870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800c3c0 tx=0x7f3d2800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d280074e0 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d28007680 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c0cdeb0 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28004a00 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.049+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c0cd8e0 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.050+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c04f330 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.050+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d2801a020 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.051+0000 7f3d1effd640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1006b500 0x7f3d1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.051+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d28089740 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.051+0000 7f3d319a9640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1006b500 0x7f3d1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.051+0000 7f3d319a9640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1006b500 0x7f3d1006d9c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d20036000 tx=0x7f3d20008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.055+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d28058650 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.228+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d2c0b09c0 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.229+0000 7f3d1effd640 1 -- 172.21.15.122:0/1164553820 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f3d28057f50 con 0x7f3d2c0b41f0 2024-03-06T07:22:31.231 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:31.232 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:31.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1006b500 msgr2=0x7f3d1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1006b500 0x7f3d1006d9c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d20036000 tx=0x7f3d20008030 comp rx=0 tx=0).stop 2024-03-06T07:22:31.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 msgr2=0x7f3d2c1f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 0x7f3d2c1f6870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800c3c0 tx=0x7f3d2800c7c0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 shutdown_connections 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d1006b500 0x7f3d1006d9c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d2c15dfe0 0x7f3d2c0cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d2c0d22b0 0x7f3d2c0cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 --2- 172.21.15.122:0/1164553820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2c0b41f0 0x7f3d2c1f6870 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 >> 172.21.15.122:0/1164553820 conn(0x7f3d2c0d3cd0 msgr2=0x7f3d2c06cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 shutdown_connections 2024-03-06T07:22:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.231+0000 7f3d3440e640 1 -- 172.21.15.122:0/1164553820 wait complete. 2024-03-06T07:22:31.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:31.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:31.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.345+0000 7f9aab2fb640 1 Processor -- start 2024-03-06T07:22:31.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.345+0000 7f9aab2fb640 1 -- start start 2024-03-06T07:22:31.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aab2fb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9aa40b41b0 0x7f9aa40cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aab2fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa40d0240 con 0x7f9aa415bdd0 2024-03-06T07:22:31.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aab2fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa40cef50 con 0x7f9aa40b15c0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aab2fb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa40cf4f0 con 0x7f9aa40b41b0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aa9097640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9aa415bdd0 0x7f9aa40ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49714/0 (socket says 172.21.15.122:49714) 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aa9097640 1 -- 172.21.15.122:0/1078215336 learned_addr learned my addr 172.21.15.122:0/1078215336 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.346+0000 7f9aa8896640 1 --2- 172.21.15.122:0/1078215336 >> v2:172.21.15.169:3300/0 conn(0x7f9aa40b41b0 0x7f9aa40cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672314129 0 0) 0x7f9aa40d0240 con 0x7f9aa415bdd0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a90003520 con 0x7f9aa415bdd0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446741534 0 0) 0x7f9aa40cef50 con 0x7f9aa40b15c0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa40d0240 con 0x7f9aa40b15c0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9aa8896640 1 -- 172.21.15.122:0/1078215336 >> v1:172.21.15.169:6789/0 conn(0x7f9aa40b15c0 legacy=0x7f9aa40b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9aa8896640 1 -- 172.21.15.122:0/1078215336 >> v1:172.21.15.122:6789/0 conn(0x7f9aa415bdd0 legacy=0x7f9aa40ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.347+0000 7f9aa8896640 1 -- 172.21.15.122:0/1078215336 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40cf210 con 0x7f9aa40b41b0 2024-03-06T07:22:31.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aa8896640 1 --2- 172.21.15.122:0/1078215336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cea10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a940097a0 tx=0x7f9a9402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e5fa8557ae7e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94034070 con 0x7f9aa40b41b0 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a94030610 con 0x7f9aa40b41b0 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9a9bfff640 1 -- 172.21.15.122:0/1078215336 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940309e0 con 0x7f9aa40b41b0 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 -- 172.21.15.122:0/1078215336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 msgr2=0x7f9aa40cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/1078215336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cea10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a940097a0 tx=0x7f9a9402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 -- 172.21.15.122:0/1078215336 shutdown_connections 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/1078215336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cea10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 -- 172.21.15.122:0/1078215336 >> 172.21.15.122:0/1078215336 conn(0x7f9aa40b3690 msgr2=0x7f9aa40b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 -- 172.21.15.122:0/1078215336 shutdown_connections 2024-03-06T07:22:31.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.348+0000 7f9aab2fb640 1 -- 172.21.15.122:0/1078215336 wait complete. 2024-03-06T07:22:31.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.349+0000 7f9aab2fb640 1 Processor -- start 2024-03-06T07:22:31.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.349+0000 7f9aab2fb640 1 -- start start 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aab2fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 0x7f9aa4151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aab2fb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aab2fb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa415bdd0 0x7f9aa40afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aab2fb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa415c1b0 con 0x7f9aa40b15c0 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aab2fb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa40d0850 con 0x7f9aa40b41b0 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aab2fb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa40b10c0 con 0x7f9aa415bdd0 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aa9097640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 0x7f9aa4151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aa9097640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 0x7f9aa4151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38830/0 (socket says 172.21.15.122:38830) 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aa9097640 1 -- 172.21.15.122:0/939336167 learned_addr learned my addr 172.21.15.122:0/939336167 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aa8896640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.350+0000 7f9aa9898640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa415bdd0 0x7f9aa40afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aa9097640 1 -- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa415bdd0 msgr2=0x7f9aa40afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aa9097640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa415bdd0 0x7f9aa40afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aa9097640 1 -- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 msgr2=0x7f9aa40cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aa9097640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aa9097640 1 -- 172.21.15.122:0/939336167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40c9e80 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aa9097640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 0x7f9aa4151300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000c500 tx=0x7f9aa000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0017070 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa0005ce0 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa40c8de0 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00072f0 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.351+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa40c93a0 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.352+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa40c8b70 con 0x7f9aa40b15c0 2024-03-06T07:22:31.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.353+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9aa001a070 con 0x7f9aa40b15c0 2024-03-06T07:22:31.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.353+0000 7f9a99ffb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8806b4d0 0x7f9a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.353+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9aa0089530 con 0x7f9aa40b15c0 2024-03-06T07:22:31.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.353+0000 7f9aa8896640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8806b4d0 0x7f9a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.354+0000 7f9aa8896640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8806b4d0 0x7f9a8806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a94036000 tx=0x7f9a9403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.357+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9aa0057d60 con 0x7f9aa40b15c0 2024-03-06T07:22:31.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.529+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9aa409fcf0 con 0x7f9aa40b15c0 2024-03-06T07:22:31.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.529+0000 7f9a99ffb640 1 -- 172.21.15.122:0/939336167 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9aa005be00 con 0x7f9aa40b15c0 2024-03-06T07:22:31.531 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:31.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8806b4d0 msgr2=0x7f9a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8806b4d0 0x7f9a8806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a94036000 tx=0x7f9a9403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 msgr2=0x7f9aa4151300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 0x7f9aa4151300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000c500 tx=0x7f9aa000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 shutdown_connections 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8806b4d0 0x7f9a8806d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa415bdd0 0x7f9aa40afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa40b41b0 0x7f9aa40cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 --2- 172.21.15.122:0/939336167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa40b15c0 0x7f9aa4151300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 >> 172.21.15.122:0/939336167 conn(0x7f9aa40b3690 msgr2=0x7f9aa40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 shutdown_connections 2024-03-06T07:22:31.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.531+0000 7f9aab2fb640 1 -- 172.21.15.122:0/939336167 wait complete. 2024-03-06T07:22:31.544 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:31.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.645+0000 7f5142ed4640 1 Processor -- start 2024-03-06T07:22:31.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.645+0000 7f5142ed4640 1 -- start start 2024-03-06T07:22:31.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5142ed4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f513c1586e0 0x7f513c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5142ed4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c158ac0 con 0x7f513c0d01a0 2024-03-06T07:22:31.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5142ed4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c15b2a0 con 0x7f513c0aef80 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5142ed4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c15b540 con 0x7f513c1586e0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5141471640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f513c0d01a0 0x7f513c0b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49722/0 (socket says 172.21.15.122:49722) 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5141471640 1 -- 172.21.15.122:0/2618828433 learned_addr learned my addr 172.21.15.122:0/2618828433 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f5140c70640 1 --2- 172.21.15.122:0/2618828433 >> v2:172.21.15.122:3300/0 conn(0x7f513c1586e0 0x7f513c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222230782 0 0) 0x7f513c158ac0 con 0x7f513c0d01a0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5124003520 con 0x7f513c0d01a0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649550318 0 0) 0x7f513c15b2a0 con 0x7f513c0aef80 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.646+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f513c158ac0 con 0x7f513c0aef80 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f5140c70640 1 -- 172.21.15.122:0/2618828433 >> v1:172.21.15.169:6789/0 conn(0x7f513c0aef80 legacy=0x7f513c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f5140c70640 1 -- 172.21.15.122:0/2618828433 >> v1:172.21.15.122:6789/0 conn(0x7f513c0d01a0 legacy=0x7f513c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875178122 0 0) 0x7f5124003520 con 0x7f513c0d01a0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f5140c70640 1 -- 172.21.15.122:0/2618828433 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c15b820 con 0x7f513c1586e0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f5140c70640 1 --2- 172.21.15.122:0/2618828433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c1586e0 0x7f513c0aea40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f512c0097a0 tx=0x7f512c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad489036b8c85230 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c037070 con 0x7f513c1586e0 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c030510 con 0x7f513c1586e0 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.647+0000 7f513b7fe640 1 -- 172.21.15.122:0/2618828433 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51340019f0 con 0x7f513c1586e0 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 -- 172.21.15.122:0/2618828433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c1586e0 msgr2=0x7f513c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 --2- 172.21.15.122:0/2618828433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c1586e0 0x7f513c0aea40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f512c0097a0 tx=0x7f512c02b790 comp rx=0 tx=0).stop 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 -- 172.21.15.122:0/2618828433 shutdown_connections 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 --2- 172.21.15.122:0/2618828433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c1586e0 0x7f513c0aea40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 -- 172.21.15.122:0/2618828433 >> 172.21.15.122:0/2618828433 conn(0x7f513c06b190 msgr2=0x7f513c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 -- 172.21.15.122:0/2618828433 shutdown_connections 2024-03-06T07:22:31.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.648+0000 7f5142ed4640 1 -- 172.21.15.122:0/2618828433 wait complete. 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.649+0000 7f5142ed4640 1 Processor -- start 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.649+0000 7f5142ed4640 1 -- start start 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5142ed4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 0x7f513c1f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5142ed4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f513c0d01a0 0x7f513c1e3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5142ed4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c1586e0 0x7f513c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5142ed4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c071350 con 0x7f513c0aef80 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5142ed4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c15bde0 con 0x7f513c1586e0 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5142ed4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0714d0 con 0x7f513c0d01a0 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 0x7f513c1f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 0x7f513c1f0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38848/0 (socket says 172.21.15.122:38848) 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 -- 172.21.15.122:0/382633745 learned_addr learned my addr 172.21.15.122:0/382633745 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f513bfff640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f513c0d01a0 0x7f513c1e3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5141471640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c1586e0 0x7f513c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 -- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f513c0d01a0 msgr2=0x7f513c1e3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f513c0d01a0 0x7f513c1e3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 -- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c1586e0 msgr2=0x7f513c156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c1586e0 0x7f513c156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.650+0000 7f5140c70640 1 -- 172.21.15.122:0/382633745 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c1e3740 con 0x7f513c0aef80 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.651+0000 7f5140c70640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 0x7f513c1f0d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f512c0098c0 tx=0x7f512c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.651+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c042070 con 0x7f513c0aef80 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.651+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c03e870 con 0x7f513c0aef80 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.651+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c0cd180 con 0x7f513c0aef80 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.651+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c037040 con 0x7f513c0aef80 2024-03-06T07:22:31.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.651+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c0cd740 con 0x7f513c0aef80 2024-03-06T07:22:31.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.652+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c0b4540 con 0x7f513c0aef80 2024-03-06T07:22:31.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.652+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f512c045070 con 0x7f513c0aef80 2024-03-06T07:22:31.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.652+0000 7f51397fa640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f512806b4d0 0x7f512806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.653+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f512c0abe00 con 0x7f513c0aef80 2024-03-06T07:22:31.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.653+0000 7f513bfff640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f512806b4d0 0x7f512806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.653+0000 7f513bfff640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f512806b4d0 0x7f512806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5130007fd0 tx=0x7f5130009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.657+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f513c0b4540 con 0x7f513c0aef80 2024-03-06T07:22:31.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.827+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f513c155650 con 0x7f513c0aef80 2024-03-06T07:22:31.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.827+0000 7f51397fa640 1 -- 172.21.15.122:0/382633745 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f513c155650 con 0x7f513c0aef80 2024-03-06T07:22:31.829 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f512806b4d0 msgr2=0x7f512806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f512806b4d0 0x7f512806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5130007fd0 tx=0x7f5130009040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 msgr2=0x7f513c1f0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 0x7f513c1f0d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f512c0098c0 tx=0x7f512c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 shutdown_connections 2024-03-06T07:22:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f512806b4d0 0x7f512806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c1586e0 0x7f513c156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f513c0d01a0 0x7f513c1e3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 --2- 172.21.15.122:0/382633745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f513c0aef80 0x7f513c1f0d80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 >> 172.21.15.122:0/382633745 conn(0x7f513c06b190 msgr2=0x7f513c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 shutdown_connections 2024-03-06T07:22:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.829+0000 7f5142ed4640 1 -- 172.21.15.122:0/382633745 wait complete. 2024-03-06T07:22:31.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:31.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.942+0000 7f330bb35640 1 Processor -- start 2024-03-06T07:22:31.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.942+0000 7f330bb35640 1 -- start start 2024-03-06T07:22:31.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330bb35640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33041614d0 0x7f33040d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330bb35640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f33040d1910 0x7f33040d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330bb35640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f33040d22b0 0x7f33040d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330bb35640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33040d2740 con 0x7f33040d1910 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330bb35640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33041618b0 con 0x7f33040d22b0 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330bb35640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33040cf400 con 0x7f33041614d0 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f33090d0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f33040d1910 0x7f33040d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f33090d0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f33040d1910 0x7f33040d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38852/0 (socket says 172.21.15.122:38852) 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f33090d0640 1 -- 172.21.15.122:0/1259449596 learned_addr learned my addr 172.21.15.122:0/1259449596 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f330a0d2640 1 --2- 172.21.15.122:0/1259449596 >> v2:172.21.15.169:3300/0 conn(0x7f33040d22b0 0x7f33040d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.943+0000 7f33098d1640 1 --2- 172.21.15.122:0/1259449596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33041614d0 0x7f33040d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.944+0000 7f33090d0640 1 -- 172.21.15.122:0/1259449596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33041614d0 msgr2=0x7f33040d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.944+0000 7f33090d0640 1 --2- 172.21.15.122:0/1259449596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33041614d0 0x7f33040d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.944+0000 7f33090d0640 1 -- 172.21.15.122:0/1259449596 >> v2:172.21.15.169:3300/0 conn(0x7f33040d22b0 msgr2=0x7f33040d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.944+0000 7f33090d0640 1 --2- 172.21.15.122:0/1259449596 >> v2:172.21.15.169:3300/0 conn(0x7f33040d22b0 0x7f33040d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.944+0000 7f33090d0640 1 -- 172.21.15.122:0/1259449596 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33040cf6a0 con 0x7f33040d1910 2024-03-06T07:22:31.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.944+0000 7f33090d0640 1 --2- 172.21.15.122:0/1259449596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d1910 0x7f33040d1d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f33040b57d0 tx=0x7f32f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8568ce6131e99f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f33088cf640 1 -- 172.21.15.122:0/1259449596 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4035020 con 0x7f33040d1910 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f33088cf640 1 -- 172.21.15.122:0/1259449596 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f4031870 con 0x7f33040d1910 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f33088cf640 1 -- 172.21.15.122:0/1259449596 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4031c40 con 0x7f33040d1910 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 -- 172.21.15.122:0/1259449596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d1910 msgr2=0x7f33040d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 --2- 172.21.15.122:0/1259449596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d1910 0x7f33040d1d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f33040b57d0 tx=0x7f32f402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 -- 172.21.15.122:0/1259449596 shutdown_connections 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 --2- 172.21.15.122:0/1259449596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d1910 0x7f33040d1d70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 --2- 172.21.15.122:0/1259449596 >> v2:172.21.15.169:3300/0 conn(0x7f33040d22b0 0x7f33040d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 --2- 172.21.15.122:0/1259449596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33041614d0 0x7f33040d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 -- 172.21.15.122:0/1259449596 >> 172.21.15.122:0/1259449596 conn(0x7f33040d3d40 msgr2=0x7f33040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 -- 172.21.15.122:0/1259449596 shutdown_connections 2024-03-06T07:22:31.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.945+0000 7f330bb35640 1 -- 172.21.15.122:0/1259449596 wait complete. 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.946+0000 7f330bb35640 1 Processor -- start 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.946+0000 7f330bb35640 1 -- start start 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330bb35640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33040d1910 0x7f33040b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330bb35640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 0x7f33040b9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330bb35640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33041614d0 0x7f33041f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330bb35640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33040b0910 con 0x7f33040d22b0 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330bb35640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33040cfc60 con 0x7f33041614d0 2024-03-06T07:22:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330bb35640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33040d0be0 con 0x7f33040d1910 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f33090d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 0x7f33040b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f33090d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 0x7f33040b9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38862/0 (socket says 172.21.15.122:38862) 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f33090d0640 1 -- 172.21.15.122:0/3236619635 learned_addr learned my addr 172.21.15.122:0/3236619635 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f33098d1640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33040d1910 0x7f33040b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.947+0000 7f330a0d2640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33041614d0 0x7f33041f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f33090d0640 1 -- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33040d1910 msgr2=0x7f33040b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f33090d0640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33040d1910 0x7f33040b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f33090d0640 1 -- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33041614d0 msgr2=0x7f33041f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f33090d0640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33041614d0 0x7f33041f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f33090d0640 1 -- 172.21.15.122:0/3236619635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330415d120 con 0x7f33040d22b0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f33090d0640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 0x7f33040b9590 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f32f402b950 tx=0x7f32f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4040070 con 0x7f33040d22b0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f4031e10 con 0x7f33040d22b0 2024-03-06T07:22:31.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4035040 con 0x7f33040d22b0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330415d400 con 0x7f33040d22b0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.948+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330415d9c0 con 0x7f33040d22b0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.949+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3304068ff0 con 0x7f33040d22b0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.949+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32f4038020 con 0x7f33040d22b0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.950+0000 7f32fa7fc640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32e806b4d0 0x7f32e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.950+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32f40abdb0 con 0x7f33040d22b0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.950+0000 7f33098d1640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32e806b4d0 0x7f32e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.950+0000 7f33098d1640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32e806b4d0 0x7f32e806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f33040cdba0 tx=0x7f3300009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:31.954+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32f407a670 con 0x7f33040d22b0 2024-03-06T07:22:32.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.123+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f33040a1c10 con 0x7f33040d22b0 2024-03-06T07:22:32.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.124+0000 7f32fa7fc640 1 -- 172.21.15.122:0/3236619635 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f32f407e710 con 0x7f33040d22b0 2024-03-06T07:22:32.126 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32e806b4d0 msgr2=0x7f32e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32e806b4d0 0x7f32e806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f33040cdba0 tx=0x7f3300009190 comp rx=0 tx=0).stop 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 msgr2=0x7f33040b9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 0x7f33040b9590 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f32f402b950 tx=0x7f32f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 shutdown_connections 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32e806b4d0 0x7f32e806d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33041614d0 0x7f33041f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33040d22b0 0x7f33040b9590 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 --2- 172.21.15.122:0/3236619635 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33040d1910 0x7f33040b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 >> 172.21.15.122:0/3236619635 conn(0x7f33040d3d40 msgr2=0x7f33040b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 shutdown_connections 2024-03-06T07:22:32.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.126+0000 7f330bb35640 1 -- 172.21.15.122:0/3236619635 wait complete. 2024-03-06T07:22:32.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:32.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.237+0000 7fc9febf6640 1 Processor -- start 2024-03-06T07:22:32.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.237+0000 7fc9febf6640 1 -- start start 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9febf6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc9f80b4fe0 0x7fc9f80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9febf6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f815d510 con 0x7fc9f80b4c00 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9febf6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f815d780 con 0x7fc9f815ba50 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9febf6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f815e0b0 con 0x7fc9f80b4fe0 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9f7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc9f80b4fe0 0x7fc9f80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9fc992640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc9f80b4c00 0x7fc9f80b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49738/0 (socket says 172.21.15.122:49738) 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.238+0000 7fc9fc992640 1 -- 172.21.15.122:0/2091542210 learned_addr learned my addr 172.21.15.122:0/2091542210 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.239+0000 7fc9f7fff640 1 -- 172.21.15.122:0/2091542210 >> v1:172.21.15.169:6789/0 conn(0x7fc9f815ba50 legacy=0x7fc9f815bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.239+0000 7fc9f7fff640 1 -- 172.21.15.122:0/2091542210 >> v1:172.21.15.122:6789/0 conn(0x7fc9f80b4c00 legacy=0x7fc9f80b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.239+0000 7fc9f77fe640 1 -- 172.21.15.122:0/2091542210 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868007317 0 0) 0x7fc9f815d510 con 0x7fc9f80b4c00 2024-03-06T07:22:32.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.239+0000 7fc9f7fff640 1 -- 172.21.15.122:0/2091542210 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f815da60 con 0x7fc9f80b4fe0 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.239+0000 7fc9f7fff640 1 --2- 172.21.15.122:0/2091542210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f80b2fb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0097a0 tx=0x7fc9ec02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed13f67bbef4fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9f77fe640 1 -- 172.21.15.122:0/2091542210 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec02f810 con 0x7fc9f80b4fe0 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9f77fe640 1 -- 172.21.15.122:0/2091542210 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec02f9b0 con 0x7fc9f80b4fe0 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9f77fe640 1 -- 172.21.15.122:0/2091542210 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec02fd80 con 0x7fc9f80b4fe0 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 -- 172.21.15.122:0/2091542210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f80b4fe0 msgr2=0x7fc9f80b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 --2- 172.21.15.122:0/2091542210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f80b2fb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0097a0 tx=0x7fc9ec02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 -- 172.21.15.122:0/2091542210 shutdown_connections 2024-03-06T07:22:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 --2- 172.21.15.122:0/2091542210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f80b2fb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 -- 172.21.15.122:0/2091542210 >> 172.21.15.122:0/2091542210 conn(0x7fc9f809a9f0 msgr2=0x7fc9f809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 -- 172.21.15.122:0/2091542210 shutdown_connections 2024-03-06T07:22:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.240+0000 7fc9febf6640 1 -- 172.21.15.122:0/2091542210 wait complete. 2024-03-06T07:22:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.241+0000 7fc9febf6640 1 Processor -- start 2024-03-06T07:22:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.241+0000 7fc9febf6640 1 -- start start 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9febf6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc9f80b4c00 0x7fc9f80afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9febf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f814f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9febf6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 0x7fc9f814e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9febf6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f815f6d0 con 0x7fc9f815ba50 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9febf6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f815f550 con 0x7fc9f80b4fe0 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9febf6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f815c5d0 con 0x7fc9f80b4c00 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9fd193640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 0x7fc9f814e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9fd193640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 0x7fc9f814e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38884/0 (socket says 172.21.15.122:38884) 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9fd193640 1 -- 172.21.15.122:0/1771611129 learned_addr learned my addr 172.21.15.122:0/1771611129 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9f7fff640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f814f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.242+0000 7fc9fc992640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc9f80b4c00 0x7fc9f80afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9fd193640 1 -- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc9f80b4c00 msgr2=0x7fc9f80afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9fd193640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc9f80b4c00 0x7fc9f80afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9fd193640 1 -- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9f80b4fe0 msgr2=0x7fc9f814f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9fd193640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f814f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9fd193640 1 -- 172.21.15.122:0/1771611129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f806c670 con 0x7fc9f815ba50 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9fd193640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 0x7fc9f814e750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000c6e0 tx=0x7fc9f000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0003b70 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f0003d30 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f806c950 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0005310 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.243+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f806cf10 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.244+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f804f330 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.244+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9f00054b0 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.245+0000 7fc9f57fa640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc9e406b500 0x7fc9e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.245+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9f0089e10 con 0x7fc9f815ba50 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.245+0000 7fc9fc992640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc9e406b500 0x7fc9e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.245+0000 7fc9fc992640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc9e406b500 0x7fc9e406d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007f70 tx=0x7fc9e8007f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.249+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9f00586d0 con 0x7fc9f815ba50 2024-03-06T07:22:32.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.431+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc9f806f490 con 0x7fc9f815ba50 2024-03-06T07:22:32.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.431+0000 7fc9f57fa640 1 -- 172.21.15.122:0/1771611129 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc9f005c770 con 0x7fc9f815ba50 2024-03-06T07:22:32.433 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:32.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc9e406b500 msgr2=0x7fc9e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc9e406b500 0x7fc9e406d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007f70 tx=0x7fc9e8007f00 comp rx=0 tx=0).stop 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 msgr2=0x7fc9f814e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 0x7fc9f814e750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000c6e0 tx=0x7fc9f000cba0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 shutdown_connections 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc9e406b500 0x7fc9e406d9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9f815ba50 0x7fc9f814e750 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9f80b4fe0 0x7fc9f814f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 --2- 172.21.15.122:0/1771611129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc9f80b4c00 0x7fc9f80afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 >> 172.21.15.122:0/1771611129 conn(0x7fc9f809a9f0 msgr2=0x7fc9f80b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 shutdown_connections 2024-03-06T07:22:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.433+0000 7fc9febf6640 1 -- 172.21.15.122:0/1771611129 wait complete. 2024-03-06T07:22:32.445 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi169.front.sepia.ceph.com... 2024-03-06T07:22:32.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:32.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:32.473 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:32.473 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:22:32.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:32.474 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:22:32.626 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:22:32.626 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:32.642 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:22:32.642 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:32.700 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:32.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.837+0000 7f73c6f3e640 1 Processor -- start 2024-03-06T07:22:32.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.837+0000 7f73c6f3e640 1 -- start start 2024-03-06T07:22:32.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73c6f3e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f73c006da40 0x7f73c00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73c6f3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73c6f3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c00cc980 con 0x7f73c01424f0 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73c6f3e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c00cd220 con 0x7f73c006da40 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73c6f3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00ccc60 con 0x7f73c00cd7a0 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f73c006da40 0x7f73c00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f73c006da40 0x7f73c00cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38892/0 (socket says 172.21.15.122:38892) 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 -- 172.21.15.122:0/1411065993 learned_addr learned my addr 172.21.15.122:0/1411065993 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73c54db640 1 --2- 172.21.15.122:0/1411065993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 -- 172.21.15.122:0/1411065993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 msgr2=0x7f73c015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 --2- 172.21.15.122:0/1411065993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 -- 172.21.15.122:0/1411065993 >> v1:172.21.15.169:6789/0 conn(0x7f73c01424f0 legacy=0x7f73c006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bf7fe640 1 -- 172.21.15.122:0/1411065993 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539295329 0 0) 0x7f73c00cc980 con 0x7f73c01424f0 2024-03-06T07:22:32.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.838+0000 7f73bffff640 1 -- 172.21.15.122:0/1411065993 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c00ccf40 con 0x7f73c006da40 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.839+0000 7f73bffff640 1 --2- 172.21.15.122:0/1411065993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c00cc2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009870 tx=0x7f73b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33a6812f22737a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.839+0000 7f73bf7fe640 1 -- 172.21.15.122:0/1411065993 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b4034070 con 0x7f73c006da40 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.839+0000 7f73bf7fe640 1 -- 172.21.15.122:0/1411065993 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b4030840 con 0x7f73c006da40 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.839+0000 7f73bf7fe640 1 -- 172.21.15.122:0/1411065993 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b4030c10 con 0x7f73c006da40 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 -- 172.21.15.122:0/1411065993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 msgr2=0x7f73c00cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/1411065993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c00cc2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009870 tx=0x7f73b402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 -- 172.21.15.122:0/1411065993 shutdown_connections 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/1411065993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/1411065993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c00cc2f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 -- 172.21.15.122:0/1411065993 >> 172.21.15.122:0/1411065993 conn(0x7f73c00cf8a0 msgr2=0x7f73c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 -- 172.21.15.122:0/1411065993 shutdown_connections 2024-03-06T07:22:32.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.840+0000 7f73c6f3e640 1 -- 172.21.15.122:0/1411065993 wait complete. 2024-03-06T07:22:32.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.841+0000 7f73c6f3e640 1 Processor -- start 2024-03-06T07:22:32.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.841+0000 7f73c6f3e640 1 -- start start 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c6f3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c0154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c6f3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c00b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c6f3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c01424f0 0x7f73c00b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c6f3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c0071ce0 con 0x7f73c006da40 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c6f3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c0071b60 con 0x7f73c01424f0 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c6f3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c0071e60 con 0x7f73c00cd7a0 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c0154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c0154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38904/0 (socket says 172.21.15.122:38904) 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 -- 172.21.15.122:0/3748620767 learned_addr learned my addr 172.21.15.122:0/3748620767 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73bffff640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c00b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c54db640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c01424f0 0x7f73c00b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 -- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 msgr2=0x7f73c00b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c00b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 -- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c01424f0 msgr2=0x7f73c00b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c01424f0 0x7f73c00b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 -- 172.21.15.122:0/3748620767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0135d40 con 0x7f73c006da40 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.842+0000 7f73c4cda640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c0154240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f73b000c410 tx=0x7f73b000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.843+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0017070 con 0x7f73c006da40 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.843+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b00035f0 con 0x7f73c006da40 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.843+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c0136020 con 0x7f73c006da40 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.843+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b00039e0 con 0x7f73c006da40 2024-03-06T07:22:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.843+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c01365e0 con 0x7f73c006da40 2024-03-06T07:22:32.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.844+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c004f330 con 0x7f73c006da40 2024-03-06T07:22:32.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.844+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73b001a070 con 0x7f73c006da40 2024-03-06T07:22:32.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.845+0000 7f73bd7fa640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f73ac06b4d0 0x7f73ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.845+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73b00895c0 con 0x7f73c006da40 2024-03-06T07:22:32.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.845+0000 7f73bffff640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f73ac06b4d0 0x7f73ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.845+0000 7f73bffff640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f73ac06b4d0 0x7f73ac06d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73b4036000 tx=0x7f73b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:32.849+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b0057df0 con 0x7f73c006da40 2024-03-06T07:22:33.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.020+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c00d0620 con 0x7f73c006da40 2024-03-06T07:22:33.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.021+0000 7f73bd7fa640 1 -- 172.21.15.122:0/3748620767 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f73b005be90 con 0x7f73c006da40 2024-03-06T07:22:33.023 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:33.023 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":20,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:21:54.547421+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f73ac06b4d0 msgr2=0x7f73ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f73ac06b4d0 0x7f73ac06d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73b4036000 tx=0x7f73b4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 msgr2=0x7f73c0154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c0154240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f73b000c410 tx=0x7f73b000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 shutdown_connections 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f73ac06b4d0 0x7f73ac06d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c01424f0 0x7f73c00b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73c00cd7a0 0x7f73c00b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 --2- 172.21.15.122:0/3748620767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73c006da40 0x7f73c0154240 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 >> 172.21.15.122:0/3748620767 conn(0x7f73c00cf8a0 msgr2=0x7f73c015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 shutdown_connections 2024-03-06T07:22:33.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.023+0000 7f73c6f3e640 1 -- 172.21.15.122:0/3748620767 wait complete. 2024-03-06T07:22:33.036 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:33.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:33.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.137+0000 7f58c5e88640 1 Processor -- start 2024-03-06T07:22:33.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.137+0000 7f58c5e88640 1 -- start start 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58c5e88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f58c01586e0 0x7f58c00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58c5e88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c00aef80 0x7f58c00cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58c5e88640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c00d03f0 con 0x7f58c00cffd0 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58c5e88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c0158ac0 con 0x7f58c01586e0 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58c5e88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c015b2a0 con 0x7f58c00aef80 2024-03-06T07:22:33.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f58c01586e0 0x7f58c00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f58c00cffd0 0x7f58c006da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49746/0 (socket says 172.21.15.122:49746) 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bffff640 1 -- 172.21.15.122:0/4208043430 learned_addr learned my addr 172.21.15.122:0/4208043430 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bf7fe640 1 --2- 172.21.15.122:0/4208043430 >> v2:172.21.15.122:3300/0 conn(0x7f58c01586e0 0x7f58c00aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38916/0 (socket says 172.21.15.122:38916) 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58beffd640 1 --2- 172.21.15.122:0/4208043430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c00aef80 0x7f58c00cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bf7fe640 1 -- 172.21.15.122:0/4208043430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c00aef80 msgr2=0x7f58c00cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bf7fe640 1 --2- 172.21.15.122:0/4208043430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c00aef80 0x7f58c00cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bf7fe640 1 -- 172.21.15.122:0/4208043430 >> v1:172.21.15.122:6789/0 conn(0x7f58c00cffd0 legacy=0x7f58c006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58bf7fe640 1 -- 172.21.15.122:0/4208043430 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c015b540 con 0x7f58c01586e0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.138+0000 7f58be7fc640 1 -- 172.21.15.122:0/4208043430 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766734230 0 0) 0x7f58c00d03f0 con 0x7f58c00cffd0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.139+0000 7f58bf7fe640 1 --2- 172.21.15.122:0/4208043430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c01586e0 0x7f58c00aea40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f58b00097a0 tx=0x7f58b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b36605ab120af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.139+0000 7f58be7fc640 1 -- 172.21.15.122:0/4208043430 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0035020 con 0x7f58c01586e0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.139+0000 7f58be7fc640 1 -- 172.21.15.122:0/4208043430 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b0031830 con 0x7f58c01586e0 2024-03-06T07:22:33.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.139+0000 7f58be7fc640 1 -- 172.21.15.122:0/4208043430 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0031c00 con 0x7f58c01586e0 2024-03-06T07:22:33.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 -- 172.21.15.122:0/4208043430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c01586e0 msgr2=0x7f58c00aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 --2- 172.21.15.122:0/4208043430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c01586e0 0x7f58c00aea40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f58b00097a0 tx=0x7f58b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 -- 172.21.15.122:0/4208043430 shutdown_connections 2024-03-06T07:22:33.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 --2- 172.21.15.122:0/4208043430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c00aef80 0x7f58c00cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 --2- 172.21.15.122:0/4208043430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c01586e0 0x7f58c00aea40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 -- 172.21.15.122:0/4208043430 >> 172.21.15.122:0/4208043430 conn(0x7f58c006b190 msgr2=0x7f58c006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 -- 172.21.15.122:0/4208043430 shutdown_connections 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.140+0000 7f58c5e88640 1 -- 172.21.15.122:0/4208043430 wait complete. 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.141+0000 7f58c5e88640 1 Processor -- start 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.141+0000 7f58c5e88640 1 -- start start 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58c5e88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 0x7f58c0141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58c5e88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c00cffd0 0x7f58c00cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58c5e88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c01586e0 0x7f58c014d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58c5e88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c015bde0 con 0x7f58c00aef80 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58c5e88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c015bb00 con 0x7f58c00cffd0 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58c5e88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c0071350 con 0x7f58c01586e0 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 0x7f58c0141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 0x7f58c0141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38920/0 (socket says 172.21.15.122:38920) 2024-03-06T07:22:33.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 -- 172.21.15.122:0/1435762226 learned_addr learned my addr 172.21.15.122:0/1435762226 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bffff640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c01586e0 0x7f58c014d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58beffd640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c00cffd0 0x7f58c00cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 -- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c01586e0 msgr2=0x7f58c014d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c01586e0 0x7f58c014d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 -- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c00cffd0 msgr2=0x7f58c00cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c00cffd0 0x7f58c00cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 -- 172.21.15.122:0/1435762226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00cdf70 con 0x7f58c00aef80 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bffff640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c01586e0 0x7f58c014d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58beffd640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c00cffd0 0x7f58c00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58bf7fe640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 0x7f58c0141330 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58b0034040 tx=0x7f58b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.142+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0040070 con 0x7f58c00aef80 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.143+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b003a910 con 0x7f58c00aef80 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.143+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c00ce250 con 0x7f58c00aef80 2024-03-06T07:22:33.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.143+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0035040 con 0x7f58c00aef80 2024-03-06T07:22:33.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.143+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c00ce810 con 0x7f58c00aef80 2024-03-06T07:22:33.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.143+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c00732b0 con 0x7f58c00aef80 2024-03-06T07:22:33.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.144+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58b0038020 con 0x7f58c00aef80 2024-03-06T07:22:33.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.144+0000 7f58a7fff640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f58ac06b4d0 0x7f58ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.144+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58b00aba00 con 0x7f58c00aef80 2024-03-06T07:22:33.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.144+0000 7f58beffd640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f58ac06b4d0 0x7f58ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.145+0000 7f58beffd640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f58ac06b4d0 0x7f58ac06d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f58c0158d20 tx=0x7f58b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.149+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58b0031800 con 0x7f58c00aef80 2024-03-06T07:22:33.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.321+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58c0068ff0 con 0x7f58c00aef80 2024-03-06T07:22:33.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.322+0000 7f58a7fff640 1 -- 172.21.15.122:0/1435762226 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58b007a910 con 0x7f58c00aef80 2024-03-06T07:22:33.324 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f58ac06b4d0 msgr2=0x7f58ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f58ac06b4d0 0x7f58ac06d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f58c0158d20 tx=0x7f58b4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 msgr2=0x7f58c0141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 0x7f58c0141330 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58b0034040 tx=0x7f58b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 shutdown_connections 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f58ac06b4d0 0x7f58ac06d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58c01586e0 0x7f58c014d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c00cffd0 0x7f58c00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 --2- 172.21.15.122:0/1435762226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58c00aef80 0x7f58c0141330 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 >> 172.21.15.122:0/1435762226 conn(0x7f58c006b190 msgr2=0x7f58c0072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 shutdown_connections 2024-03-06T07:22:33.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.324+0000 7f58c5e88640 1 -- 172.21.15.122:0/1435762226 wait complete. 2024-03-06T07:22:33.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:33.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.437+0000 7f3eb050c640 1 Processor -- start 2024-03-06T07:22:33.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.437+0000 7f3eb050c640 1 -- start start 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.438+0000 7f3eb050c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3ea80b4970 0x7f3ea806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.438+0000 7f3eb050c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea815ff90 con 0x7f3ea806d5c0 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.438+0000 7f3eb050c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea8160270 con 0x7f3ea815bb40 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.438+0000 7f3eb050c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea80b4dd0 con 0x7f3ea80b4970 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eadaa7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3ea80b4970 0x7f3ea806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eadaa7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3ea80b4970 0x7f3ea806d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38930/0 (socket says 172.21.15.122:38930) 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eaeaa9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3ea806d5c0 0x7f3ea80cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49750/0 (socket says 172.21.15.122:49750) 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eadaa7640 1 -- 172.21.15.122:0/3019877934 learned_addr learned my addr 172.21.15.122:0/3019877934 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eadaa7640 1 -- 172.21.15.122:0/3019877934 >> v1:172.21.15.169:6789/0 conn(0x7f3ea815bb40 legacy=0x7f3ea815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eadaa7640 1 -- 172.21.15.122:0/3019877934 >> v1:172.21.15.122:6789/0 conn(0x7f3ea806d5c0 legacy=0x7f3ea80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3eadaa7640 1 -- 172.21.15.122:0/3019877934 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8160550 con 0x7f3ea80b4970 2024-03-06T07:22:33.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.439+0000 7f3ead2a6640 1 -- 172.21.15.122:0/3019877934 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159630988 0 0) 0x7f3ea815ff90 con 0x7f3ea806d5c0 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eadaa7640 1 --2- 172.21.15.122:0/3019877934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea80b4970 0x7f3ea806d080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009870 tx=0x7f3ea402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff830b836ac976bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3ead2a6640 1 -- 172.21.15.122:0/3019877934 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea402f840 con 0x7f3ea80b4970 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3ead2a6640 1 -- 172.21.15.122:0/3019877934 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea402f9e0 con 0x7f3ea80b4970 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3ead2a6640 1 -- 172.21.15.122:0/3019877934 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea402fdb0 con 0x7f3ea80b4970 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 -- 172.21.15.122:0/3019877934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea80b4970 msgr2=0x7f3ea806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 --2- 172.21.15.122:0/3019877934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea80b4970 0x7f3ea806d080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009870 tx=0x7f3ea402ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 -- 172.21.15.122:0/3019877934 shutdown_connections 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 --2- 172.21.15.122:0/3019877934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea80b4970 0x7f3ea806d080 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 -- 172.21.15.122:0/3019877934 >> 172.21.15.122:0/3019877934 conn(0x7f3ea806b190 msgr2=0x7f3ea806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 -- 172.21.15.122:0/3019877934 shutdown_connections 2024-03-06T07:22:33.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.440+0000 7f3eb050c640 1 -- 172.21.15.122:0/3019877934 wait complete. 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.441+0000 7f3eb050c640 1 Processor -- start 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.441+0000 7f3eb050c640 1 -- start start 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eb050c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ea806d5c0 0x7f3ea80b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eb050c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3ea80b4970 0x7f3ea80af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eb050c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 0x7f3ea80afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eb050c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea815bf20 con 0x7f3ea815bb40 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eb050c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea806c910 con 0x7f3ea806d5c0 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eb050c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea815c0a0 con 0x7f3ea80b4970 2024-03-06T07:22:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eaeaa9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 0x7f3ea80afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eaeaa9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 0x7f3ea80afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38938/0 (socket says 172.21.15.122:38938) 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eaeaa9640 1 -- 172.21.15.122:0/472566958 learned_addr learned my addr 172.21.15.122:0/472566958 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eae2a8640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ea806d5c0 0x7f3ea80b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.442+0000 7f3eadaa7640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3ea80b4970 0x7f3ea80af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eaeaa9640 1 -- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3ea80b4970 msgr2=0x7f3ea80af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eaeaa9640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3ea80b4970 0x7f3ea80af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eaeaa9640 1 -- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ea806d5c0 msgr2=0x7f3ea80b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eaeaa9640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ea806d5c0 0x7f3ea80b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eaeaa9640 1 -- 172.21.15.122:0/472566958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea815cb70 con 0x7f3ea815bb40 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eaeaa9640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 0x7f3ea80afc50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000c7d0 tx=0x7f3ea000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0018070 con 0x7f3ea815bb40 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea000a640 con 0x7f3ea815bb40 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea80b0190 con 0x7f3ea815bb40 2024-03-06T07:22:33.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000aa70 con 0x7f3ea815bb40 2024-03-06T07:22:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.443+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea80b0740 con 0x7f3ea815bb40 2024-03-06T07:22:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.444+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea806b7a0 con 0x7f3ea815bb40 2024-03-06T07:22:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.444+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ea001b070 con 0x7f3ea815bb40 2024-03-06T07:22:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.445+0000 7f3e9affd640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.445+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ea00896e0 con 0x7f3ea815bb40 2024-03-06T07:22:33.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.445+0000 7f3eae2a8640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.445+0000 7f3eae2a8640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c000f60 tx=0x7f3e9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.449+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ea806b7a0 con 0x7f3ea815bb40 2024-03-06T07:22:33.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.623+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ea80732b0 con 0x7f3ea815bb40 2024-03-06T07:22:33.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.623+0000 7f3e9affd640 1 -- 172.21.15.122:0/472566958 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ea815cb70 con 0x7f3ea815bb40 2024-03-06T07:22:33.626 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:33.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e8c06b4d0 msgr2=0x7f3e8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c000f60 tx=0x7f3e9c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 msgr2=0x7f3ea80afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 0x7f3ea80afc50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000c7d0 tx=0x7f3ea000cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 shutdown_connections 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e8c06b4d0 0x7f3e8c06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ea815bb40 0x7f3ea80afc50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3ea80b4970 0x7f3ea80af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 --2- 172.21.15.122:0/472566958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ea806d5c0 0x7f3ea80b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 >> 172.21.15.122:0/472566958 conn(0x7f3ea806b190 msgr2=0x7f3ea8156ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 shutdown_connections 2024-03-06T07:22:33.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.625+0000 7f3eb050c640 1 -- 172.21.15.122:0/472566958 wait complete. 2024-03-06T07:22:33.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:33.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.753+0000 7f84d02c9640 1 Processor -- start 2024-03-06T07:22:33.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.753+0000 7f84d02c9640 1 -- start start 2024-03-06T07:22:33.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84d02c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815e0e0 0x7f84c806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84d02c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84c806b6c0 0x7f84c80b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84d02c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c8161bd0 con 0x7f84c815dd00 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84d02c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c8070220 con 0x7f84c806b6c0 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84d02c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c8161e90 con 0x7f84c815e0e0 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce866640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84c806b6c0 0x7f84c80b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce065640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f84c815dd00 0x7f84c806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49766/0 (socket says 172.21.15.122:49766) 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce065640 1 -- 172.21.15.122:0/3936246910 learned_addr learned my addr 172.21.15.122:0/3936246910 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84cd864640 1 --2- 172.21.15.122:0/3936246910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815e0e0 0x7f84c806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce866640 1 -- 172.21.15.122:0/3936246910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815e0e0 msgr2=0x7f84c806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce866640 1 --2- 172.21.15.122:0/3936246910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815e0e0 0x7f84c806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce866640 1 -- 172.21.15.122:0/3936246910 >> v1:172.21.15.122:6789/0 conn(0x7f84c815dd00 legacy=0x7f84c806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84ce866640 1 -- 172.21.15.122:0/3936246910 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c8162170 con 0x7f84c806b6c0 2024-03-06T07:22:33.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.754+0000 7f84cd063640 1 -- 172.21.15.122:0/3936246910 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479683417 0 0) 0x7f84c8161bd0 con 0x7f84c815dd00 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.755+0000 7f84ce866640 1 --2- 172.21.15.122:0/3936246910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c80b42d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009d90 tx=0x7f84c002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90716c73cb7c937 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.755+0000 7f84cd063640 1 -- 172.21.15.122:0/3936246910 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0039070 con 0x7f84c806b6c0 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.755+0000 7f84cd063640 1 -- 172.21.15.122:0/3936246910 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c002cc50 con 0x7f84c806b6c0 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.755+0000 7f84cd063640 1 -- 172.21.15.122:0/3936246910 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0034440 con 0x7f84c806b6c0 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 -- 172.21.15.122:0/3936246910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 msgr2=0x7f84c80b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 --2- 172.21.15.122:0/3936246910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c80b42d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009d90 tx=0x7f84c002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 -- 172.21.15.122:0/3936246910 shutdown_connections 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 --2- 172.21.15.122:0/3936246910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c80b42d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 --2- 172.21.15.122:0/3936246910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815e0e0 0x7f84c806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 -- 172.21.15.122:0/3936246910 >> 172.21.15.122:0/3936246910 conn(0x7f84c80b2660 msgr2=0x7f84c80b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 -- 172.21.15.122:0/3936246910 shutdown_connections 2024-03-06T07:22:33.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.756+0000 7f84d02c9640 1 -- 172.21.15.122:0/3936246910 wait complete. 2024-03-06T07:22:33.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.757+0000 7f84d02c9640 1 Processor -- start 2024-03-06T07:22:33.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.757+0000 7f84d02c9640 1 -- start start 2024-03-06T07:22:33.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c8152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815dd00 0x7f84c813ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84c81e7900 0x7f84c813d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c80b0180 con 0x7f84c806b6c0 2024-03-06T07:22:33.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c80b0000 con 0x7f84c81e7900 2024-03-06T07:22:33.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c80b0300 con 0x7f84c815dd00 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c8152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c8152530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38958/0 (socket says 172.21.15.122:38958) 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 -- 172.21.15.122:0/865302752 learned_addr learned my addr 172.21.15.122:0/865302752 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce866640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84c81e7900 0x7f84c813d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84cd864640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815dd00 0x7f84c813ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 -- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815dd00 msgr2=0x7f84c813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815dd00 0x7f84c813ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 -- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84c81e7900 msgr2=0x7f84c813d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84c81e7900 0x7f84c813d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 -- 172.21.15.122:0/865302752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c813f1c0 con 0x7f84c806b6c0 2024-03-06T07:22:33.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84ce065640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c8152530 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c500 tx=0x7f84c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4018070 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4007580 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c813dbe0 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.758+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4004a20 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.759+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c813e1a0 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.759+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c8073390 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.760+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84c401b070 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.760+0000 7f84beffd640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ac06b4d0 0x7f84ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.760+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f84c4089450 con 0x7f84c806b6c0 2024-03-06T07:22:33.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.760+0000 7f84cd864640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ac06b4d0 0x7f84ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.761+0000 7f84cd864640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ac06b4d0 0x7f84ac06d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f84c80b2630 tx=0x7f84b8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.765+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84c4057c90 con 0x7f84c806b6c0 2024-03-06T07:22:33.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.938+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84c8068ff0 con 0x7f84c806b6c0 2024-03-06T07:22:33.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.938+0000 7f84beffd640 1 -- 172.21.15.122:0/865302752 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84c405bd30 con 0x7f84c806b6c0 2024-03-06T07:22:33.940 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:33.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ac06b4d0 msgr2=0x7f84ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ac06b4d0 0x7f84ac06d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f84c80b2630 tx=0x7f84b8009190 comp rx=0 tx=0).stop 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 msgr2=0x7f84c8152530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c8152530 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c500 tx=0x7f84c400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 shutdown_connections 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ac06b4d0 0x7f84ac06d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84c81e7900 0x7f84c813d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84c815dd00 0x7f84c813ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 --2- 172.21.15.122:0/865302752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c806b6c0 0x7f84c8152530 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 >> 172.21.15.122:0/865302752 conn(0x7f84c80b2660 msgr2=0x7f84c80b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 shutdown_connections 2024-03-06T07:22:33.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:33.940+0000 7f84d02c9640 1 -- 172.21.15.122:0/865302752 wait complete. 2024-03-06T07:22:33.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:34.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.052+0000 7ff851078640 1 Processor -- start 2024-03-06T07:22:34.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.052+0000 7ff851078640 1 -- start start 2024-03-06T07:22:34.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff851078640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff84c15c2c0 0x7ff84c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff851078640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff851078640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0b3680 con 0x7ff84c1456f0 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff851078640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c15c760 con 0x7ff84c15c2c0 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff851078640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c15ca40 con 0x7ff84c0cc830 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff84c15c2c0 0x7ff84c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff84c15c2c0 0x7ff84c0cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38964/0 (socket says 172.21.15.122:38964) 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 -- 172.21.15.122:0/2683364525 learned_addr learned my addr 172.21.15.122:0/2683364525 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff84b59e640 1 --2- 172.21.15.122:0/2683364525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 -- 172.21.15.122:0/2683364525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 msgr2=0x7ff84c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 --2- 172.21.15.122:0/2683364525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 -- 172.21.15.122:0/2683364525 >> v1:172.21.15.169:6789/0 conn(0x7ff84c1456f0 legacy=0x7ff84c070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff843fff640 1 -- 172.21.15.122:0/2683364525 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c15cd20 con 0x7ff84c15c2c0 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.053+0000 7ff84a59c640 1 -- 172.21.15.122:0/2683364525 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112631041 0 0) 0x7ff83c001110 con 0x7ff84c1456f0 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.054+0000 7ff843fff640 1 --2- 172.21.15.122:0/2683364525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0cc2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff834009870 tx=0x7ff83402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f8bd5576ca37ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.054+0000 7ff84a59c640 1 -- 172.21.15.122:0/2683364525 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834034070 con 0x7ff84c15c2c0 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.054+0000 7ff84a59c640 1 -- 172.21.15.122:0/2683364525 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834030840 con 0x7ff84c15c2c0 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.054+0000 7ff84a59c640 1 -- 172.21.15.122:0/2683364525 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834030c10 con 0x7ff84c15c2c0 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.054+0000 7ff851078640 1 -- 172.21.15.122:0/2683364525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 msgr2=0x7ff84c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.054+0000 7ff851078640 1 --2- 172.21.15.122:0/2683364525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0cc2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff834009870 tx=0x7ff83402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.055+0000 7ff851078640 1 -- 172.21.15.122:0/2683364525 shutdown_connections 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.055+0000 7ff851078640 1 --2- 172.21.15.122:0/2683364525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.055+0000 7ff851078640 1 --2- 172.21.15.122:0/2683364525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0cc2f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.055+0000 7ff851078640 1 -- 172.21.15.122:0/2683364525 >> 172.21.15.122:0/2683364525 conn(0x7ff84c0b1c80 msgr2=0x7ff84c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.055+0000 7ff851078640 1 -- 172.21.15.122:0/2683364525 shutdown_connections 2024-03-06T07:22:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.055+0000 7ff851078640 1 -- 172.21.15.122:0/2683364525 wait complete. 2024-03-06T07:22:34.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.056+0000 7ff851078640 1 Processor -- start 2024-03-06T07:22:34.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.056+0000 7ff851078640 1 -- start start 2024-03-06T07:22:34.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff84c1456f0 0x7ff84c0aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c15d800 con 0x7ff84c15c2c0 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c071050 con 0x7ff84c1456f0 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c15d980 con 0x7ff84c0cc830 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38966/0 (socket says 172.21.15.122:38966) 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 -- 172.21.15.122:0/2459443157 learned_addr learned my addr 172.21.15.122:0/2459443157 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84ad9d640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff843fff640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff84c1456f0 0x7ff84c0aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 -- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 msgr2=0x7ff84c15b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c15b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 -- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff84c1456f0 msgr2=0x7ff84c0aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff84c1456f0 0x7ff84c0aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 -- 172.21.15.122:0/2459443157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c15ba30 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff84b59e640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0a9450 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff84400efc0 tx=0x7ff84400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844018020 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844014480 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c0a9d50 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844014870 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.057+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c0aa310 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.058+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c04f330 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.059+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff844005d10 con 0x7ff84c15c2c0 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.059+0000 7ff8437fe640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff82c06b500 0x7ff82c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.059+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff84408ab70 con 0x7ff84c15c2c0 2024-03-06T07:22:34.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.059+0000 7ff84ad9d640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff82c06b500 0x7ff82c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.060+0000 7ff84ad9d640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff82c06b500 0x7ff82c06d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff83c000f60 tx=0x7ff83c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.064+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff844022070 con 0x7ff84c15c2c0 2024-03-06T07:22:34.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.245+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff84c144b50 con 0x7ff84c15c2c0 2024-03-06T07:22:34.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.246+0000 7ff8437fe640 1 -- 172.21.15.122:0/2459443157 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff844010040 con 0x7ff84c15c2c0 2024-03-06T07:22:34.248 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff82c06b500 msgr2=0x7ff82c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff82c06b500 0x7ff82c06d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff83c000f60 tx=0x7ff83c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 msgr2=0x7ff84c0a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0a9450 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff84400efc0 tx=0x7ff84400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 shutdown_connections 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff82c06b500 0x7ff82c06d9c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff84c15c2c0 0x7ff84c0a9450 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff84c1456f0 0x7ff84c0aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 --2- 172.21.15.122:0/2459443157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff84c0cc830 0x7ff84c15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.247+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 >> 172.21.15.122:0/2459443157 conn(0x7ff84c0b1c80 msgr2=0x7ff84c0cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.248+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 shutdown_connections 2024-03-06T07:22:34.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.248+0000 7ff851078640 1 -- 172.21.15.122:0/2459443157 wait complete. 2024-03-06T07:22:34.260 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:22:34.260 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:34.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:34.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.416+0000 7f500ba15640 1 Processor -- start 2024-03-06T07:22:34.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.416+0000 7f500ba15640 1 -- start start 2024-03-06T07:22:34.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f500ba15640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 0x7f50040d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f500ba15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f50040cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f500ba15640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50040b1070 con 0x7f5004162020 2024-03-06T07:22:34.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f500ba15640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040ce9d0 con 0x7f5004162a00 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f500ba15640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040d2da0 con 0x7f5004162400 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f50040cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5008fb0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 0x7f50040d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f50040cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:43678/0 (socket says 172.21.15.122:43678) 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5008fb0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 0x7f50040d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:46408/0 (socket says 172.21.15.122:46408) 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 -- 172.21.15.122:0/2701299636 learned_addr learned my addr 172.21.15.122:0/2701299636 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f50097b1640 1 --1- 172.21.15.122:0/2701299636 >> v1:172.21.15.169:6789/0 conn(0x7f5004162020 0x7f50040b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:36588/0 (socket says 172.21.15.122:36588) 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 -- 172.21.15.122:0/2701299636 >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 msgr2=0x7f50040d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 --2- 172.21.15.122:0/2701299636 >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 0x7f50040d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 -- 172.21.15.122:0/2701299636 >> v1:172.21.15.169:6789/0 conn(0x7f5004162020 legacy=0x7f50040b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5009fb2640 1 -- 172.21.15.122:0/2701299636 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040d2500 con 0x7f5004162400 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.417+0000 7f5008fb0640 1 --2- 172.21.15.122:0/2701299636 >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 0x7f50040d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.418+0000 7f5009fb2640 1 --2- 172.21.15.122:0/2701299636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f50040cf400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50040d2f20 tx=0x7f4ffc02bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=711e70f3469c03d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.418+0000 7f4ffbfff640 1 -- 172.21.15.122:0/2701299636 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc02cb70 con 0x7f5004162400 2024-03-06T07:22:34.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.418+0000 7f4ffbfff640 1 -- 172.21.15.122:0/2701299636 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc02cd10 con 0x7f5004162400 2024-03-06T07:22:34.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.418+0000 7f4ffbfff640 1 -- 172.21.15.122:0/2701299636 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc034610 con 0x7f5004162400 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 -- 172.21.15.122:0/2701299636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 msgr2=0x7f50040cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 --2- 172.21.15.122:0/2701299636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f50040cf400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50040d2f20 tx=0x7f4ffc02bfd0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 -- 172.21.15.122:0/2701299636 shutdown_connections 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 --2- 172.21.15.122:0/2701299636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f50040cf400 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 --2- 172.21.15.122:0/2701299636 >> v2:172.21.15.169:3300/0 conn(0x7f5004162a00 0x7f50040d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 -- 172.21.15.122:0/2701299636 >> 172.21.15.122:0/2701299636 conn(0x7f50040d3d10 msgr2=0x7f50040d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 -- 172.21.15.122:0/2701299636 shutdown_connections 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.419+0000 7f500ba15640 1 -- 172.21.15.122:0/2701299636 wait complete. 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.420+0000 7f500ba15640 1 Processor -- start 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.420+0000 7f500ba15640 1 -- start start 2024-03-06T07:22:34.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f500ba15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 0x7f500415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f500ba15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f500415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f500ba15640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5004162a00 0x7f50040b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f500ba15640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cec00 con 0x7f5004162020 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f500ba15640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040d2ac0 con 0x7f5004162a00 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f500ba15640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cf940 con 0x7f5004162400 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 0x7f500415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 0x7f500415b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38974/0 (socket says 172.21.15.122:38974) 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 -- 172.21.15.122:0/3207153948 learned_addr learned my addr 172.21.15.122:0/3207153948 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f5008fb0640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f500415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f5009fb2640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5004162a00 0x7f50040b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 -- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 msgr2=0x7f500415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f500415d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 -- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5004162a00 msgr2=0x7f50040b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5004162a00 0x7f50040b8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 -- 172.21.15.122:0/3207153948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040ca550 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f50097b1640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 0x7f500415b200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f500000c400 tx=0x7f500000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.421+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50000074e0 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.422+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000007680 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.422+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000004a00 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.422+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50040c7840 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.422+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50040c7e00 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.423+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500404f330 con 0x7f5004162020 2024-03-06T07:22:34.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.423+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f500001a020 con 0x7f5004162020 2024-03-06T07:22:34.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.423+0000 7f4ff9ffb640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.423+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f50000898a0 con 0x7f5004162020 2024-03-06T07:22:34.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.423+0000 7f5008fb0640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.424+0000 7f5008fb0640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fe806b4d0 0x7f4fe806d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009560 tx=0x7f4ff4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.428+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50000580e0 con 0x7f5004162020 2024-03-06T07:22:34.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.601+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5004073380 con 0x7f5004162020 2024-03-06T07:22:34.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.602+0000 7f4ff9ffb640 1 -- 172.21.15.122:0/3207153948 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16854 (secure 0 0 0) 0x7f500005c180 con 0x7f5004162020 2024-03-06T07:22:34.604 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:34.604 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:33.694334+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:34.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fe806b4d0 msgr2=0x7f4fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fe806b4d0 0x7f4fe806d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009560 tx=0x7f4ff4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 msgr2=0x7f500415b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 0x7f500415b200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f500000c400 tx=0x7f500000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 shutdown_connections 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5004162a00 0x7f50040b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5004162400 0x7f500415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 --2- 172.21.15.122:0/3207153948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5004162020 0x7f500415b200 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 >> 172.21.15.122:0/3207153948 conn(0x7f50040d3d10 msgr2=0x7f50040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 shutdown_connections 2024-03-06T07:22:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.604+0000 7f500ba15640 1 -- 172.21.15.122:0/3207153948 wait complete. 2024-03-06T07:22:34.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:34.618 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:34.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.723+0000 7f633af2b640 1 Processor -- start 2024-03-06T07:22:34.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.723+0000 7f633af2b640 1 -- start start 2024-03-06T07:22:34.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f633af2b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63340d0af0 0x7f63340cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f633af2b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63340cd4b0 con 0x7f633415c050 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f633af2b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63340cd790 con 0x7f633406fd10 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f633af2b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63340cda70 con 0x7f63340d0af0 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f63394c8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63340d0af0 0x7f63340cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f6338cc7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f633415c050 0x7f633406f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49786/0 (socket says 172.21.15.122:49786) 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f6338cc7640 1 -- 172.21.15.122:0/2772119720 learned_addr learned my addr 172.21.15.122:0/2772119720 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f63394c8640 1 -- 172.21.15.122:0/2772119720 >> v1:172.21.15.169:6789/0 conn(0x7f633406fd10 legacy=0x7f63340cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f63394c8640 1 -- 172.21.15.122:0/2772119720 >> v1:172.21.15.122:6789/0 conn(0x7f633415c050 legacy=0x7f633406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.724+0000 7f63337fe640 1 -- 172.21.15.122:0/2772119720 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277588739 0 0) 0x7f63340cd4b0 con 0x7f633415c050 2024-03-06T07:22:34.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.725+0000 7f63394c8640 1 -- 172.21.15.122:0/2772119720 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63340cdd50 con 0x7f63340d0af0 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.725+0000 7f63394c8640 1 --2- 172.21.15.122:0/2772119720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63340d0af0 0x7f63340cce70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f632c009d90 tx=0x7f632c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4b89edbf6b18e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.725+0000 7f63337fe640 1 -- 172.21.15.122:0/2772119720 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c039070 con 0x7f63340d0af0 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.725+0000 7f63337fe640 1 -- 172.21.15.122:0/2772119720 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c02cc60 con 0x7f63340d0af0 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.725+0000 7f63337fe640 1 -- 172.21.15.122:0/2772119720 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c034660 con 0x7f63340d0af0 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 -- 172.21.15.122:0/2772119720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63340d0af0 msgr2=0x7f63340cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 --2- 172.21.15.122:0/2772119720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63340d0af0 0x7f63340cce70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f632c009d90 tx=0x7f632c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 -- 172.21.15.122:0/2772119720 shutdown_connections 2024-03-06T07:22:34.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 --2- 172.21.15.122:0/2772119720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63340d0af0 0x7f63340cce70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 -- 172.21.15.122:0/2772119720 >> 172.21.15.122:0/2772119720 conn(0x7f63340b3800 msgr2=0x7f63340b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 -- 172.21.15.122:0/2772119720 shutdown_connections 2024-03-06T07:22:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.726+0000 7f633af2b640 1 -- 172.21.15.122:0/2772119720 wait complete. 2024-03-06T07:22:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.727+0000 7f633af2b640 1 Processor -- start 2024-03-06T07:22:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.727+0000 7f633af2b640 1 -- start start 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f633af2b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f633406fd10 0x7f63341519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f633af2b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63340d0af0 0x7f6334150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f633af2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 0x7f63341563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f633af2b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334166e50 con 0x7f633415c050 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f633af2b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334166cd0 con 0x7f63340d0af0 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f633af2b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6334166fd0 con 0x7f633406fd10 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 0x7f63341563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 0x7f63341563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56194/0 (socket says 172.21.15.122:56194) 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 -- 172.21.15.122:0/4090028734 learned_addr learned my addr 172.21.15.122:0/4090028734 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f6333fff640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63340d0af0 0x7f6334150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f6338cc7640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f633406fd10 0x7f63341519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 -- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f633406fd10 msgr2=0x7f63341519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f633406fd10 0x7f63341519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 -- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63340d0af0 msgr2=0x7f6334150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63340d0af0 0x7f6334150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.728+0000 7f63394c8640 1 -- 172.21.15.122:0/4090028734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334150690 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.729+0000 7f63394c8640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 0x7f63341563d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f632c009eb0 tx=0x7f632c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.729+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c039050 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.729+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c045070 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.729+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6334150930 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.729+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c03e830 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.729+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6334150ef0 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.730+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633409fe60 con 0x7f633415c050 2024-03-06T07:22:34.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.730+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f632c03e9d0 con 0x7f633415c050 2024-03-06T07:22:34.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.730+0000 7f63317fa640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f632006b4d0 0x7f632006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.731+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f632c0acfd0 con 0x7f633415c050 2024-03-06T07:22:34.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.731+0000 7f6338cc7640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f632006b4d0 0x7f632006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.731+0000 7f6338cc7640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f632006b4d0 0x7f632006d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6324007f70 tx=0x7f6324007f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.735+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f632c07b8a0 con 0x7f633415c050 2024-03-06T07:22:34.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.906+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6334068ff0 con 0x7f633415c050 2024-03-06T07:22:34.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.906+0000 7f63317fa640 1 -- 172.21.15.122:0/4090028734 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f632c07f940 con 0x7f633415c050 2024-03-06T07:22:34.908 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:34.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f632006b4d0 msgr2=0x7f632006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f632006b4d0 0x7f632006d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6324007f70 tx=0x7f6324007f00 comp rx=0 tx=0).stop 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 msgr2=0x7f63341563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 0x7f63341563d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f632c009eb0 tx=0x7f632c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 shutdown_connections 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f632006b4d0 0x7f632006d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f633415c050 0x7f63341563d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63340d0af0 0x7f6334150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 --2- 172.21.15.122:0/4090028734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f633406fd10 0x7f63341519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 >> 172.21.15.122:0/4090028734 conn(0x7f63340b3800 msgr2=0x7f63340cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 shutdown_connections 2024-03-06T07:22:34.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:34.908+0000 7f633af2b640 1 -- 172.21.15.122:0/4090028734 wait complete. 2024-03-06T07:22:34.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:35.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.022+0000 7f6c300d4640 1 Processor -- start 2024-03-06T07:22:35.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.022+0000 7f6c300d4640 1 -- start start 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c300d4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6c28146f60 0x7f6c28147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c300d4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c280712b0 con 0x7f6c28146b80 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c300d4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c280b5080 con 0x7f6c28070e20 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c300d4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c280ad900 con 0x7f6c28146f60 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2de70640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6c28146b80 0x7f6c280b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49798/0 (socket says 172.21.15.122:49798) 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2de70640 1 -- 172.21.15.122:0/2244105162 learned_addr learned my addr 172.21.15.122:0/2244105162 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:35.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2d66f640 1 --2- 172.21.15.122:0/2244105162 >> v2:172.21.15.169:3300/0 conn(0x7f6c28146f60 0x7f6c28147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405698921 0 0) 0x7f6c280712b0 con 0x7f6c28146b80 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c14003520 con 0x7f6c28146b80 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540676490 0 0) 0x7f6c280b5080 con 0x7f6c28070e20 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c280712b0 con 0x7f6c28070e20 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2d66f640 1 -- 172.21.15.122:0/2244105162 >> v1:172.21.15.169:6789/0 conn(0x7f6c28070e20 legacy=0x7f6c2806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2d66f640 1 -- 172.21.15.122:0/2244105162 >> v1:172.21.15.122:6789/0 conn(0x7f6c28146b80 legacy=0x7f6c280b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.023+0000 7f6c2d66f640 1 -- 172.21.15.122:0/2244105162 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280adbe0 con 0x7f6c28146f60 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.024+0000 7f6c2d66f640 1 --2- 172.21.15.122:0/2244105162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28146f60 0x7f6c28147890 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6c180097a0 tx=0x7f6c1802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6821bc095db6a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.024+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18034070 con 0x7f6c28146f60 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.024+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c18030610 con 0x7f6c28146f60 2024-03-06T07:22:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.024+0000 7f6c2ce6e640 1 -- 172.21.15.122:0/2244105162 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c180309c0 con 0x7f6c28146f60 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 -- 172.21.15.122:0/2244105162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28146f60 msgr2=0x7f6c28147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 --2- 172.21.15.122:0/2244105162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28146f60 0x7f6c28147890 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6c180097a0 tx=0x7f6c1802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 -- 172.21.15.122:0/2244105162 shutdown_connections 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 --2- 172.21.15.122:0/2244105162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28146f60 0x7f6c28147890 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 -- 172.21.15.122:0/2244105162 >> 172.21.15.122:0/2244105162 conn(0x7f6c2806b830 msgr2=0x7f6c280b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 -- 172.21.15.122:0/2244105162 shutdown_connections 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.025+0000 7f6c300d4640 1 -- 172.21.15.122:0/2244105162 wait complete. 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.026+0000 7f6c300d4640 1 Processor -- start 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.026+0000 7f6c300d4640 1 -- start start 2024-03-06T07:22:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c300d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28070e20 0x7f6c2813a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c300d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c28146b80 0x7f6c2815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c300d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 0x7f6c281ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c300d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2806e9d0 con 0x7f6c2815eb50 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c300d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280b5360 con 0x7f6c28070e20 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c300d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2806eb50 con 0x7f6c28146b80 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 0x7f6c281ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 0x7f6c281ea900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56196/0 (socket says 172.21.15.122:56196) 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 -- 172.21.15.122:0/3090047093 learned_addr learned my addr 172.21.15.122:0/3090047093 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2d66f640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c28146b80 0x7f6c2815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2de70640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28070e20 0x7f6c2813a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 -- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c28146b80 msgr2=0x7f6c2815e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c28146b80 0x7f6c2815e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 -- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28070e20 msgr2=0x7f6c2813a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28070e20 0x7f6c2813a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 -- 172.21.15.122:0/3090047093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2815eff0 con 0x7f6c2815eb50 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c2e671640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 0x7f6c281ea900 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c7a0 tx=0x7f6c2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.027+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20003b70 con 0x7f6c2815eb50 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.028+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c20003d30 con 0x7f6c2815eb50 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.028+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2815f2d0 con 0x7f6c2815eb50 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.028+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2000aa90 con 0x7f6c2815eb50 2024-03-06T07:22:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.028+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2815f890 con 0x7f6c2815eb50 2024-03-06T07:22:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.028+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2806e0b0 con 0x7f6c2815eb50 2024-03-06T07:22:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.029+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c2000ac30 con 0x7f6c2815eb50 2024-03-06T07:22:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.029+0000 7f6c1e7fc640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c0c06b500 0x7f6c0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.029+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6c2008a060 con 0x7f6c2815eb50 2024-03-06T07:22:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.029+0000 7f6c2de70640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c0c06b500 0x7f6c0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.030+0000 7f6c2de70640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c0c06b500 0x7f6c0c06d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c240011e0 tx=0x7f6c24007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.034+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c20058810 con 0x7f6c2815eb50 2024-03-06T07:22:35.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.206+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6c280cf740 con 0x7f6c2815eb50 2024-03-06T07:22:35.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.206+0000 7f6c1e7fc640 1 -- 172.21.15.122:0/3090047093 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6c2005c8b0 con 0x7f6c2815eb50 2024-03-06T07:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c0c06b500 msgr2=0x7f6c0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c0c06b500 0x7f6c0c06d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c240011e0 tx=0x7f6c24007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 msgr2=0x7f6c281ea900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 0x7f6c281ea900 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c7a0 tx=0x7f6c2000cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 shutdown_connections 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c0c06b500 0x7f6c0c06d9c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c2815eb50 0x7f6c281ea900 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c28146b80 0x7f6c2815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 --2- 172.21.15.122:0/3090047093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c28070e20 0x7f6c2813a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 >> 172.21.15.122:0/3090047093 conn(0x7f6c2806b830 msgr2=0x7f6c28072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.208+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 shutdown_connections 2024-03-06T07:22:35.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.209+0000 7f6c300d4640 1 -- 172.21.15.122:0/3090047093 wait complete. 2024-03-06T07:22:35.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:35.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.322+0000 7f93e7f56640 1 Processor -- start 2024-03-06T07:22:35.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.322+0000 7f93e7f56640 1 -- start start 2024-03-06T07:22:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e7f56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e7f56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e006b6c0 con 0x7f93e006efd0 2024-03-06T07:22:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e7f56640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e006b970 con 0x7f93e0145590 2024-03-06T07:22:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e7f56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0145970 con 0x7f93e006f430 2024-03-06T07:22:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e54f1640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f93e006efd0 0x7f93e00cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49806/0 (socket says 172.21.15.122:49806) 2024-03-06T07:22:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e54f1640 1 -- 172.21.15.122:0/1793544287 learned_addr learned my addr 172.21.15.122:0/1793544287 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e64f3640 1 --2- 172.21.15.122:0/1793544287 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991396272 0 0) 0x7f93e006b6c0 con 0x7f93e006efd0 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93cc003520 con 0x7f93e006efd0 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383559782 0 0) 0x7f93e006b970 con 0x7f93e0145590 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e006b6c0 con 0x7f93e0145590 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e64f3640 1 -- 172.21.15.122:0/1793544287 >> v1:172.21.15.169:6789/0 conn(0x7f93e0145590 legacy=0x7f93e006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e64f3640 1 -- 172.21.15.122:0/1793544287 >> v1:172.21.15.122:6789/0 conn(0x7f93e006efd0 legacy=0x7f93e00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.323+0000 7f93e64f3640 1 -- 172.21.15.122:0/1793544287 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e015f7e0 con 0x7f93e006f430 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.324+0000 7f93e64f3640 1 --2- 172.21.15.122:0/1793544287 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e00cccd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009dc0 tx=0x7f93d802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af370c22d1222c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.324+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8039070 con 0x7f93e006f430 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.324+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d802ccd0 con 0x7f93e006f430 2024-03-06T07:22:35.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.324+0000 7f93e4cf0640 1 -- 172.21.15.122:0/1793544287 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8034640 con 0x7f93e006f430 2024-03-06T07:22:35.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 -- 172.21.15.122:0/1793544287 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 msgr2=0x7f93e00cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 --2- 172.21.15.122:0/1793544287 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e00cccd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009dc0 tx=0x7f93d802bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 -- 172.21.15.122:0/1793544287 shutdown_connections 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 --2- 172.21.15.122:0/1793544287 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e00cccd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 -- 172.21.15.122:0/1793544287 >> 172.21.15.122:0/1793544287 conn(0x7f93e00b2660 msgr2=0x7f93e00b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 -- 172.21.15.122:0/1793544287 shutdown_connections 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.325+0000 7f93e7f56640 1 -- 172.21.15.122:0/1793544287 wait complete. 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.326+0000 7f93e7f56640 1 Processor -- start 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.326+0000 7f93e7f56640 1 -- start start 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93e006efd0 0x7f93e015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e013c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 0x7f93e01427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e015fda0 con 0x7f93e0145590 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0160110 con 0x7f93e006efd0 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e006f8c0 con 0x7f93e006f430 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 0x7f93e01427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 0x7f93e01427e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56210/0 (socket says 172.21.15.122:56210) 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 -- 172.21.15.122:0/2874375730 learned_addr learned my addr 172.21.15.122:0/2874375730 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e5cf2640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93e006efd0 0x7f93e015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e54f1640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e013c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 -- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 msgr2=0x7f93e013c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e013c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 -- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93e006efd0 msgr2=0x7f93e015b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93e006efd0 0x7f93e015b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 -- 172.21.15.122:0/2874375730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e013b330 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e64f3640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 0x7f93e01427e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93d802c530 tx=0x7f93d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8039050 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d8045070 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e013b610 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8009450 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.327+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e013bbd0 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.328+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e013c0c0 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.329+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93d803e850 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.329+0000 7f93d67fc640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93c406b4d0 0x7f93c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.329+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f93d80ad030 con 0x7f93e0145590 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.329+0000 7f93e5cf2640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93c406b4d0 0x7f93c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.330+0000 7f93e5cf2640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93c406b4d0 0x7f93c406d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f93dc007fd0 tx=0x7f93dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.333+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93d807ba50 con 0x7f93e0145590 2024-03-06T07:22:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.504+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93e00d12a0 con 0x7f93e0145590 2024-03-06T07:22:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.504+0000 7f93d67fc640 1 -- 172.21.15.122:0/2874375730 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93e00d12a0 con 0x7f93e0145590 2024-03-06T07:22:35.507 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:35.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93c406b4d0 msgr2=0x7f93c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93c406b4d0 0x7f93c406d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f93dc007fd0 tx=0x7f93dc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 msgr2=0x7f93e01427e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 0x7f93e01427e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93d802c530 tx=0x7f93d8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 shutdown_connections 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93c406b4d0 0x7f93c406d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0145590 0x7f93e01427e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93e006f430 0x7f93e013c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 --2- 172.21.15.122:0/2874375730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93e006efd0 0x7f93e015b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 >> 172.21.15.122:0/2874375730 conn(0x7f93e00b2660 msgr2=0x7f93e00d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 shutdown_connections 2024-03-06T07:22:35.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.506+0000 7f93e7f56640 1 -- 172.21.15.122:0/2874375730 wait complete. 2024-03-06T07:22:35.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:35.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.618+0000 7f909471e640 1 Processor -- start 2024-03-06T07:22:35.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.618+0000 7f909471e640 1 -- start start 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f909471e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f908c15bbf0 0x7f908c071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f909471e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 0x7f908c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f909471e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c072490 con 0x7f908c15c510 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f909471e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c0d0090 con 0x7f908c15bbf0 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f909471e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d0370 con 0x7f908c0cebc0 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f9091cb9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 0x7f908c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f90924ba640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f908c15bbf0 0x7f908c071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f9091cb9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 0x7f908c15bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:56508/0 (socket says 172.21.15.122:56508) 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.619+0000 7f9091cb9640 1 -- 172.21.15.122:0/2723541651 learned_addr learned my addr 172.21.15.122:0/2723541651 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f90924ba640 1 -- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 msgr2=0x7f908c15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f90924ba640 1 --2- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 0x7f908c15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f90924ba640 1 -- 172.21.15.122:0/2723541651 >> v1:172.21.15.169:6789/0 conn(0x7f908c15c510 legacy=0x7f908c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f90924ba640 1 -- 172.21.15.122:0/2723541651 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0d0650 con 0x7f908c15bbf0 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f90914b8640 1 -- 172.21.15.122:0/2723541651 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75243 0 0) 0x7f908c072490 con 0x7f908c15c510 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f9091cb9640 1 --2- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 0x7f908c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.620+0000 7f90924ba640 1 --2- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c071e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f90880097a0 tx=0x7f908802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e5b7e0717f1ac0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f90914b8640 1 -- 172.21.15.122:0/2723541651 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088035020 con 0x7f908c15bbf0 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f90914b8640 1 -- 172.21.15.122:0/2723541651 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9088031830 con 0x7f908c15bbf0 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f90914b8640 1 -- 172.21.15.122:0/2723541651 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088031c00 con 0x7f908c15bbf0 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 -- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 msgr2=0x7f908c071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 --2- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c071e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f90880097a0 tx=0x7f908802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 -- 172.21.15.122:0/2723541651 shutdown_connections 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 --2- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c0cebc0 0x7f908c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 --2- 172.21.15.122:0/2723541651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c071e00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 -- 172.21.15.122:0/2723541651 >> 172.21.15.122:0/2723541651 conn(0x7f908c0b1c80 msgr2=0x7f908c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 -- 172.21.15.122:0/2723541651 shutdown_connections 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.621+0000 7f909471e640 1 -- 172.21.15.122:0/2723541651 wait complete. 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.622+0000 7f909471e640 1 Processor -- start 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.622+0000 7f909471e640 1 -- start start 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f909471e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 0x7f908c15e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f909471e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f909471e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c15c510 0x7f908c13f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f909471e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0b3800 con 0x7f908c0cebc0 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f909471e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0b3680 con 0x7f908c15bbf0 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f909471e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0cded0 con 0x7f908c15c510 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f90924ba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 0x7f908c15e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f90924ba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 0x7f908c15e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56222/0 (socket says 172.21.15.122:56222) 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f90924ba640 1 -- 172.21.15.122:0/253640220 learned_addr learned my addr 172.21.15.122:0/253640220 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f9091cb9640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.623+0000 7f9092cbb640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c15c510 0x7f908c13f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f90924ba640 1 -- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c15c510 msgr2=0x7f908c13f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f90924ba640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c15c510 0x7f908c13f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f90924ba640 1 -- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 msgr2=0x7f908c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f90924ba640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f90924ba640 1 -- 172.21.15.122:0/253640220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c15ce80 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f90924ba640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 0x7f908c15e1c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9088000c00 tx=0x7f90880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088035070 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9088004230 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c13f6f0 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908803a630 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.624+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c15d400 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.625+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c04f330 con 0x7f908c0cebc0 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.625+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9088038020 con 0x7f908c0cebc0 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.626+0000 7f9082ffd640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f907006b4d0 0x7f907006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.626+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f90880abcb0 con 0x7f908c0cebc0 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.626+0000 7f9091cb9640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f907006b4d0 0x7f907006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.626+0000 7f9091cb9640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f907006b4d0 0x7f907006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f908c1493d0 tx=0x7f907c009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.630+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9088040070 con 0x7f908c0cebc0 2024-03-06T07:22:35.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.813+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f908c071fe0 con 0x7f908c0cebc0 2024-03-06T07:22:35.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.814+0000 7f9082ffd640 1 -- 172.21.15.122:0/253640220 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f908807a4f0 con 0x7f908c0cebc0 2024-03-06T07:22:35.816 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f907006b4d0 msgr2=0x7f907006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f907006b4d0 0x7f907006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f908c1493d0 tx=0x7f907c009240 comp rx=0 tx=0).stop 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 msgr2=0x7f908c15e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 0x7f908c15e1c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9088000c00 tx=0x7f90880046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 shutdown_connections 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f907006b4d0 0x7f907006d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f908c15c510 0x7f908c13f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f908c15bbf0 0x7f908c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 --2- 172.21.15.122:0/253640220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f908c0cebc0 0x7f908c15e1c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 >> 172.21.15.122:0/253640220 conn(0x7f908c0b1c80 msgr2=0x7f908c1595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 shutdown_connections 2024-03-06T07:22:35.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:35.816+0000 7f909471e640 1 -- 172.21.15.122:0/253640220 wait complete. 2024-03-06T07:22:35.828 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:22:35.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:35.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:35.855 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:35.856 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:22:35.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:35.856 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:22:35.911 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:22:35.911 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:35.926 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:22:35.926 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:35.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:36.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.079+0000 7f9da09e2640 1 Processor -- start 2024-03-06T07:22:36.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.079+0000 7f9da09e2640 1 -- start start 2024-03-06T07:22:36.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9da09e2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9d9c15dfe0 0x7f9d9c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9da09e2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d9c0d29c0 0x7f9d9c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9da09e2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c0b3430 con 0x7f9d9c0cf830 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9da09e2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b31e0 con 0x7f9d9c0d29c0 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9da09e2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b3d60 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9d9ad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d9c0d29c0 0x7f9d9c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9d99d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9d9c0cf830 0x7f9d9c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49808/0 (socket says 172.21.15.122:49808) 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9d9ad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d9c0d29c0 0x7f9d9c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56234/0 (socket says 172.21.15.122:56234) 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9d99d9b640 1 -- 172.21.15.122:0/811716633 learned_addr learned my addr 172.21.15.122:0/811716633 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.080+0000 7f9d9a59c640 1 --2- 172.21.15.122:0/811716633 >> v2:172.21.15.169:3300/0 conn(0x7f9d9c15dfe0 0x7f9d9c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/811716633 >> v2:172.21.15.169:3300/0 conn(0x7f9d9c15dfe0 msgr2=0x7f9d9c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9ad9d640 1 --2- 172.21.15.122:0/811716633 >> v2:172.21.15.169:3300/0 conn(0x7f9d9c15dfe0 0x7f9d9c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/811716633 >> v1:172.21.15.122:6789/0 conn(0x7f9d9c0cf830 legacy=0x7f9d9c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/811716633 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0b3710 con 0x7f9d9c0d29c0 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9959a640 1 -- 172.21.15.122:0/811716633 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179781472 0 0) 0x7f9d9c0b3430 con 0x7f9d9c0cf830 2024-03-06T07:22:36.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9ad9d640 1 --2- 172.21.15.122:0/811716633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0b41f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c067990 tx=0x7f9d9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c9de784d336964 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.081+0000 7f9d9959a640 1 -- 172.21.15.122:0/811716633 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94035020 con 0x7f9d9c0d29c0 2024-03-06T07:22:36.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9d9959a640 1 -- 172.21.15.122:0/811716633 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d94031830 con 0x7f9d9c0d29c0 2024-03-06T07:22:36.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9d9959a640 1 -- 172.21.15.122:0/811716633 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94031c00 con 0x7f9d9c0d29c0 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 -- 172.21.15.122:0/811716633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c0d29c0 msgr2=0x7f9d9c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 --2- 172.21.15.122:0/811716633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0b41f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c067990 tx=0x7f9d9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 -- 172.21.15.122:0/811716633 shutdown_connections 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 --2- 172.21.15.122:0/811716633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0b41f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 --2- 172.21.15.122:0/811716633 >> v2:172.21.15.169:3300/0 conn(0x7f9d9c15dfe0 0x7f9d9c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 -- 172.21.15.122:0/811716633 >> 172.21.15.122:0/811716633 conn(0x7f9d9c0d3cd0 msgr2=0x7f9d9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 -- 172.21.15.122:0/811716633 shutdown_connections 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.082+0000 7f9da09e2640 1 -- 172.21.15.122:0/811716633 wait complete. 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.083+0000 7f9da09e2640 1 Processor -- start 2024-03-06T07:22:36.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.083+0000 7f9da09e2640 1 -- start start 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9da09e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d9c0cf830 0x7f9d9c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9da09e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9da09e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 0x7f9d9c1e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9da09e2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0d1df0 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9da09e2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0d1c70 con 0x7f9d9c0d29c0 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9da09e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c15e3c0 con 0x7f9d9c0cf830 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9d9ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 0x7f9d9c1e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9d9ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 0x7f9d9c1e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56240/0 (socket says 172.21.15.122:56240) 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/924616338 learned_addr learned my addr 172.21.15.122:0/924616338 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9d99d9b640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.084+0000 7f9d9a59c640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d9c0cf830 0x7f9d9c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d9c0cf830 msgr2=0x7f9d9c160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d9ad9d640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d9c0cf830 0x7f9d9c160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9c0d29c0 msgr2=0x7f9d9c0ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d9ad9d640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d9ad9d640 1 -- 172.21.15.122:0/924616338 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c160830 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d9ad9d640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 0x7f9d9c1e8270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009890 tx=0x7f9d940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94035070 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c161300 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d94004230 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9403a630 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.085+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c1f76c0 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.086+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c04f330 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.086+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d94038020 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.087+0000 7f9d8affd640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.087+0000 7f9d9a59c640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.087+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9d940abe30 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.087+0000 7f9d9a59c640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c011340 tx=0x7f9d8c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.091+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d94040070 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.264+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d9c0d0eb0 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.265+0000 7f9d8affd640 1 -- 172.21.15.122:0/924616338 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16854 (secure 0 0 0) 0x7f9d9407a5f0 con 0x7f9d9c15dfe0 2024-03-06T07:22:36.268 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:36.268 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":22,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:34.698183+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7c06b4d0 msgr2=0x7f9d7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c011340 tx=0x7f9d8c00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 msgr2=0x7f9d9c1e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 0x7f9d9c1e8270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009890 tx=0x7f9d940046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 shutdown_connections 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d9c15dfe0 0x7f9d9c1e8270 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d9c0d29c0 0x7f9d9c0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 --2- 172.21.15.122:0/924616338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d9c0cf830 0x7f9d9c160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.267+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 >> 172.21.15.122:0/924616338 conn(0x7f9d9c0d3cd0 msgr2=0x7f9d9c0d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.268+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 shutdown_connections 2024-03-06T07:22:36.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.268+0000 7f9da09e2640 1 -- 172.21.15.122:0/924616338 wait complete. 2024-03-06T07:22:36.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:36.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:36.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.382+0000 7f69f9d5e640 1 Processor -- start 2024-03-06T07:22:36.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.382+0000 7f69f9d5e640 1 -- start start 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.382+0000 7f69f9d5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415a220 0x7f69f40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.382+0000 7f69f9d5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69f40712d0 0x7f69f4071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.382+0000 7f69f9d5e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40b3320 con 0x7f69f415de60 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.382+0000 7f69f9d5e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40cf610 con 0x7f69f40712d0 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f9d5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4071c70 con 0x7f69f415a220 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69f40712d0 0x7f69f4071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69f40712d0 0x7f69f4071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56256/0 (socket says 172.21.15.122:56256) 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 -- 172.21.15.122:0/4134992848 learned_addr learned my addr 172.21.15.122:0/4134992848 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f37fe640 1 --2- 172.21.15.122:0/4134992848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415a220 0x7f69f40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 -- 172.21.15.122:0/4134992848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415a220 msgr2=0x7f69f40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 --2- 172.21.15.122:0/4134992848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415a220 0x7f69f40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 -- 172.21.15.122:0/4134992848 >> v1:172.21.15.169:6789/0 conn(0x7f69f415de60 legacy=0x7f69f415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.383+0000 7f69f2ffd640 1 -- 172.21.15.122:0/4134992848 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f40728d0 con 0x7f69f40712d0 2024-03-06T07:22:36.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f2ffd640 1 --2- 172.21.15.122:0/4134992848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40712d0 0x7f69f4071730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009870 tx=0x7f69e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ac1b1a0d9f6d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f27fc640 1 -- 172.21.15.122:0/4134992848 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8035020 con 0x7f69f40712d0 2024-03-06T07:22:36.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f27fc640 1 -- 172.21.15.122:0/4134992848 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8031640 con 0x7f69f40712d0 2024-03-06T07:22:36.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f27fc640 1 -- 172.21.15.122:0/4134992848 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e802d040 con 0x7f69f40712d0 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 -- 172.21.15.122:0/4134992848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40712d0 msgr2=0x7f69f4071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/4134992848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40712d0 0x7f69f4071730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009870 tx=0x7f69e802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 -- 172.21.15.122:0/4134992848 shutdown_connections 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/4134992848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40712d0 0x7f69f4071730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/4134992848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415a220 0x7f69f40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 -- 172.21.15.122:0/4134992848 >> 172.21.15.122:0/4134992848 conn(0x7f69f406b190 msgr2=0x7f69f406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 -- 172.21.15.122:0/4134992848 shutdown_connections 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.384+0000 7f69f9d5e640 1 -- 172.21.15.122:0/4134992848 wait complete. 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 Processor -- start 2024-03-06T07:22:36.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 -- start start 2024-03-06T07:22:36.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69f40712d0 0x7f69f4153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 0x7f69f40cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415de60 0x7f69f4150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4072030 con 0x7f69f415a220 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4072bb0 con 0x7f69f40712d0 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.386+0000 7f69f9d5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4072530 con 0x7f69f415de60 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 0x7f69f40cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 0x7f69f40cbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56260/0 (socket says 172.21.15.122:56260) 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f3fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415de60 0x7f69f4150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 -- 172.21.15.122:0/1928356140 learned_addr learned my addr 172.21.15.122:0/1928356140 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f37fe640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69f40712d0 0x7f69f4153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 -- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415de60 msgr2=0x7f69f4150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415de60 0x7f69f4150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 -- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69f40712d0 msgr2=0x7f69f4153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69f40712d0 0x7f69f4153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 -- 172.21.15.122:0/1928356140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f41686f0 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f2ffd640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 0x7f69f40cbbf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f69e8000c00 tx=0x7f69e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8040070 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e80399f0 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8035040 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f41689d0 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.387+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f4168f90 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.388+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f40cc130 con 0x7f69f415a220 2024-03-06T07:22:36.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.389+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69e804b060 con 0x7f69f415a220 2024-03-06T07:22:36.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.389+0000 7f69dbfff640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69e006b4d0 0x7f69e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.389+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f69e80ab9a0 con 0x7f69f415a220 2024-03-06T07:22:36.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.389+0000 7f69f37fe640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69e006b4d0 0x7f69e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.390+0000 7f69f37fe640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69e006b4d0 0x7f69e006d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009b40 tx=0x7f69e4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.393+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69e807a1e0 con 0x7f69f415a220 2024-03-06T07:22:36.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.563+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69f40732b0 con 0x7f69f415a220 2024-03-06T07:22:36.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.563+0000 7f69dbfff640 1 -- 172.21.15.122:0/1928356140 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69e807e280 con 0x7f69f415a220 2024-03-06T07:22:36.565 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:36.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69e006b4d0 msgr2=0x7f69e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69e006b4d0 0x7f69e006d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009b40 tx=0x7f69e4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 msgr2=0x7f69f40cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 0x7f69f40cbbf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f69e8000c00 tx=0x7f69e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 shutdown_connections 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69e006b4d0 0x7f69e006d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69f415de60 0x7f69f4150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f415a220 0x7f69f40cbbf0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 --2- 172.21.15.122:0/1928356140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69f40712d0 0x7f69f4153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 >> 172.21.15.122:0/1928356140 conn(0x7f69f406b190 msgr2=0x7f69f40cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 shutdown_connections 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.565+0000 7f69f9d5e640 1 -- 172.21.15.122:0/1928356140 wait complete. 2024-03-06T07:22:36.580 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:36.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.681+0000 7f517f36c640 1 Processor -- start 2024-03-06T07:22:36.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.681+0000 7f517f36c640 1 -- start start 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517f36c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51780cf0d0 0x7f517806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517f36c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780b3320 con 0x7f51780719e0 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517f36c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780cf560 con 0x7f517815a220 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517f36c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51780728d0 con 0x7f51780cf0d0 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517c907640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f51780719e0 0x7f517806c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49820/0 (socket says 172.21.15.122:49820) 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517c907640 1 -- 172.21.15.122:0/2089587264 learned_addr learned my addr 172.21.15.122:0/2089587264 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.682+0000 7f517d909640 1 --2- 172.21.15.122:0/2089587264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51780cf0d0 0x7f517806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f516ffff640 1 -- 172.21.15.122:0/2089587264 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188979682 0 0) 0x7f51780cf560 con 0x7f517815a220 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f516ffff640 1 -- 172.21.15.122:0/2089587264 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5164003520 con 0x7f517815a220 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f516ffff640 1 -- 172.21.15.122:0/2089587264 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590031505 0 0) 0x7f51780b3320 con 0x7f51780719e0 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f517d909640 1 -- 172.21.15.122:0/2089587264 >> v1:172.21.15.169:6789/0 conn(0x7f517815a220 legacy=0x7f51780712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f517d909640 1 -- 172.21.15.122:0/2089587264 >> v1:172.21.15.122:6789/0 conn(0x7f51780719e0 legacy=0x7f517806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f517d909640 1 -- 172.21.15.122:0/2089587264 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780b3590 con 0x7f51780cf0d0 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.683+0000 7f517d909640 1 --2- 172.21.15.122:0/2089587264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51780cf0d0 0x7f517806d100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f51700097d0 tx=0x7f517002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4515b7781560151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f516ffff640 1 -- 172.21.15.122:0/2089587264 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5170035020 con 0x7f51780cf0d0 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f516ffff640 1 -- 172.21.15.122:0/2089587264 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5170031800 con 0x7f51780cf0d0 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f516ffff640 1 -- 172.21.15.122:0/2089587264 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5170031bb0 con 0x7f51780cf0d0 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 -- 172.21.15.122:0/2089587264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51780cf0d0 msgr2=0x7f517806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 --2- 172.21.15.122:0/2089587264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51780cf0d0 0x7f517806d100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f51700097d0 tx=0x7f517002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 -- 172.21.15.122:0/2089587264 shutdown_connections 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 --2- 172.21.15.122:0/2089587264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51780cf0d0 0x7f517806d100 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 -- 172.21.15.122:0/2089587264 >> 172.21.15.122:0/2089587264 conn(0x7f517806b190 msgr2=0x7f517806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 -- 172.21.15.122:0/2089587264 shutdown_connections 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.684+0000 7f517f36c640 1 -- 172.21.15.122:0/2089587264 wait complete. 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.685+0000 7f517f36c640 1 Processor -- start 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.685+0000 7f517f36c640 1 -- start start 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517f36c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 0x7f5178159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517f36c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51780cf0d0 0x7f51780ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517f36c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f517815a220 0x7f51780cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517f36c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178072530 con 0x7f51780719e0 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517f36c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178072030 con 0x7f51780cf0d0 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517f36c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51780726b0 con 0x7f517815a220 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 0x7f5178159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 0x7f5178159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56266/0 (socket says 172.21.15.122:56266) 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 -- 172.21.15.122:0/769115499 learned_addr learned my addr 172.21.15.122:0/769115499 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d909640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f517815a220 0x7f51780cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517c907640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51780cf0d0 0x7f51780ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 -- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f517815a220 msgr2=0x7f51780cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f517815a220 0x7f51780cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 -- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51780cf0d0 msgr2=0x7f51780ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51780cf0d0 0x7f51780ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.686+0000 7f517d108640 1 -- 172.21.15.122:0/769115499 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517814f060 con 0x7f51780719e0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.687+0000 7f517d108640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 0x7f5178159420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f517400c7d0 tx=0x7f517400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.687+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174003b70 con 0x7f51780719e0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.687+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174003d10 con 0x7f51780719e0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.687+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517814f340 con 0x7f51780719e0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.687+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517400aa90 con 0x7f51780719e0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.687+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517814f900 con 0x7f51780719e0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.688+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51780732b0 con 0x7f51780719e0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.688+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f517400ac30 con 0x7f51780719e0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.689+0000 7f516dffb640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f515c06b4d0 0x7f515c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.689+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f517408a0d0 con 0x7f51780719e0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.689+0000 7f517c907640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f515c06b4d0 0x7f515c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.689+0000 7f517c907640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f515c06b4d0 0x7f515c06d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5178151970 tx=0x7f5168007240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.693+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5174058880 con 0x7f51780719e0 2024-03-06T07:22:36.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.865+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5178068ff0 con 0x7f51780719e0 2024-03-06T07:22:36.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.865+0000 7f516dffb640 1 -- 172.21.15.122:0/769115499 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f517405c920 con 0x7f51780719e0 2024-03-06T07:22:36.867 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f515c06b4d0 msgr2=0x7f515c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f515c06b4d0 0x7f515c06d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5178151970 tx=0x7f5168007240 comp rx=0 tx=0).stop 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 msgr2=0x7f5178159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 0x7f5178159420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f517400c7d0 tx=0x7f517400cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 shutdown_connections 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f515c06b4d0 0x7f515c06d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f517815a220 0x7f51780cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51780cf0d0 0x7f51780ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 --2- 172.21.15.122:0/769115499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51780719e0 0x7f5178159420 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 >> 172.21.15.122:0/769115499 conn(0x7f517806b190 msgr2=0x7f51780cdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 shutdown_connections 2024-03-06T07:22:36.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.867+0000 7f517f36c640 1 -- 172.21.15.122:0/769115499 wait complete. 2024-03-06T07:22:36.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:36.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.980+0000 7ff6ab8d4640 1 Processor -- start 2024-03-06T07:22:36.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.980+0000 7ff6ab8d4640 1 -- start start 2024-03-06T07:22:36.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6ab8d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff6a40d4860 0x7ff6a40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6ab8d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6ab8d4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a41618b0 con 0x7ff6a41614d0 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6ab8d4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a40d4cf0 con 0x7ff6a40d4860 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6ab8d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a40cf400 con 0x7ff6a40b0e30 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6a8e6f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff6a40d4860 0x7ff6a40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6a9670640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff6a41614d0 0x7ff6a40d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49826/0 (socket says 172.21.15.122:49826) 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6a9670640 1 -- 172.21.15.122:0/668212276 learned_addr learned my addr 172.21.15.122:0/668212276 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6a8e6f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff6a40d4860 0x7ff6a40d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56274/0 (socket says 172.21.15.122:56274) 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.981+0000 7ff6a9e71640 1 --2- 172.21.15.122:0/668212276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/668212276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 msgr2=0x7ff6a40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff6a8e6f640 1 --2- 172.21.15.122:0/668212276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/668212276 >> v1:172.21.15.122:6789/0 conn(0x7ff6a41614d0 legacy=0x7ff6a40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/668212276 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40cf6a0 con 0x7ff6a40d4860 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff69bfff640 1 -- 172.21.15.122:0/668212276 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086468640 0 0) 0x7ff6a41618b0 con 0x7ff6a41614d0 2024-03-06T07:22:36.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff6a8e6f640 1 --2- 172.21.15.122:0/668212276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40d2020 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097a0 tx=0x7ff69402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3533da16e9978985 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.982+0000 7ff69bfff640 1 -- 172.21.15.122:0/668212276 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694035020 con 0x7ff6a40d4860 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff69bfff640 1 -- 172.21.15.122:0/668212276 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff694031830 con 0x7ff6a40d4860 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff69bfff640 1 -- 172.21.15.122:0/668212276 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694031c00 con 0x7ff6a40d4860 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/668212276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 msgr2=0x7ff6a40d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/668212276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40d2020 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097a0 tx=0x7ff69402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/668212276 shutdown_connections 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/668212276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/668212276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40d2020 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/668212276 >> 172.21.15.122:0/668212276 conn(0x7ff6a40d3d40 msgr2=0x7ff6a40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/668212276 shutdown_connections 2024-03-06T07:22:36.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.983+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/668212276 wait complete. 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.984+0000 7ff6ab8d4640 1 Processor -- start 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.984+0000 7ff6ab8d4640 1 -- start start 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6ab8d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6ab8d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6ab8d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6a41614d0 0x7ff6a40b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6ab8d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a40b0910 con 0x7ff6a40d4860 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6ab8d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a40cfc60 con 0x7ff6a41614d0 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6ab8d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a40d2560 con 0x7ff6a40b0e30 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56278/0 (socket says 172.21.15.122:56278) 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/2746174787 learned_addr learned my addr 172.21.15.122:0/2746174787 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a9670640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a9e71640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6a41614d0 0x7ff6a40b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 msgr2=0x7ff6a40cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6a41614d0 msgr2=0x7ff6a40b92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6a41614d0 0x7ff6a40b92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.985+0000 7ff6a8e6f640 1 -- 172.21.15.122:0/2746174787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40b7280 con 0x7ff6a40d4860 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.986+0000 7ff6a8e6f640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40c8280 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff69402bee0 tx=0x7ff6940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.986+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694035040 con 0x7ff6a40d4860 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.986+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69403b920 con 0x7ff6a40d4860 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.986+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a40c87c0 con 0x7ff6a40d4860 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.986+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69403bd10 con 0x7ff6a40d4860 2024-03-06T07:22:36.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.986+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a40c8d40 con 0x7ff6a40d4860 2024-03-06T07:22:36.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.987+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a4068ff0 con 0x7ff6a40d4860 2024-03-06T07:22:36.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.987+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff694038020 con 0x7ff6a40d4860 2024-03-06T07:22:36.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.988+0000 7ff699ffb640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff68806b4d0 0x7ff68806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.988+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff6940ac260 con 0x7ff6a40d4860 2024-03-06T07:22:36.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.988+0000 7ff6a9670640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff68806b4d0 0x7ff68806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.988+0000 7ff6a9670640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff68806b4d0 0x7ff68806d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00012e0 tx=0x7ff6a0001630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:36.992+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6a40b7280 con 0x7ff6a40d4860 2024-03-06T07:22:37.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.162+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff6a40b5550 con 0x7ff6a40d4860 2024-03-06T07:22:37.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.162+0000 7ff699ffb640 1 -- 172.21.15.122:0/2746174787 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff694031800 con 0x7ff6a40d4860 2024-03-06T07:22:37.164 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:37.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff68806b4d0 msgr2=0x7ff68806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff68806b4d0 0x7ff68806d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00012e0 tx=0x7ff6a0001630 comp rx=0 tx=0).stop 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 msgr2=0x7ff6a40c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40c8280 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff69402bee0 tx=0x7ff6940046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 shutdown_connections 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff68806b4d0 0x7ff68806d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6a41614d0 0x7ff6a40b92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a40d4860 0x7ff6a40c8280 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 --2- 172.21.15.122:0/2746174787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6a40b0e30 0x7ff6a40cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 >> 172.21.15.122:0/2746174787 conn(0x7ff6a40d3d40 msgr2=0x7ff6a406c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 shutdown_connections 2024-03-06T07:22:37.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.164+0000 7ff6ab8d4640 1 -- 172.21.15.122:0/2746174787 wait complete. 2024-03-06T07:22:37.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:37.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.277+0000 7f1a32f3e640 1 Processor -- start 2024-03-06T07:22:37.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.277+0000 7f1a32f3e640 1 -- start start 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a32f3e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a2c15b380 0x7f1a2c0708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a32f3e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c06cd60 con 0x7f1a2c070e10 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a32f3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c06d030 con 0x7f1a2c158880 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a32f3e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c071280 con 0x7f1a2c15b380 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a2bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a2c15b380 0x7f1a2c0708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a314db640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1a2c070e10 0x7f1a2c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49842/0 (socket says 172.21.15.122:49842) 2024-03-06T07:22:37.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a314db640 1 -- 172.21.15.122:0/139419757 learned_addr learned my addr 172.21.15.122:0/139419757 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a2bfff640 1 -- 172.21.15.122:0/139419757 >> v1:172.21.15.169:6789/0 conn(0x7f1a2c158880 legacy=0x7f1a2c06c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a2bfff640 1 -- 172.21.15.122:0/139419757 >> v1:172.21.15.122:6789/0 conn(0x7f1a2c070e10 legacy=0x7f1a2c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a2b7fe640 1 -- 172.21.15.122:0/139419757 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466990590 0 0) 0x7f1a2c06cd60 con 0x7f1a2c070e10 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.278+0000 7f1a2bfff640 1 -- 172.21.15.122:0/139419757 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c06d310 con 0x7f1a2c15b380 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.279+0000 7f1a2bfff640 1 --2- 172.21.15.122:0/139419757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c15b380 0x7f1a2c0708d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a200097a0 tx=0x7f1a2002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acd1d0eb529b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.279+0000 7f1a2b7fe640 1 -- 172.21.15.122:0/139419757 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20034070 con 0x7f1a2c15b380 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.279+0000 7f1a2b7fe640 1 -- 172.21.15.122:0/139419757 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20030610 con 0x7f1a2c15b380 2024-03-06T07:22:37.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.279+0000 7f1a2b7fe640 1 -- 172.21.15.122:0/139419757 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a200309e0 con 0x7f1a2c15b380 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 -- 172.21.15.122:0/139419757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c15b380 msgr2=0x7f1a2c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/139419757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c15b380 0x7f1a2c0708d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a200097a0 tx=0x7f1a2002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 -- 172.21.15.122:0/139419757 shutdown_connections 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/139419757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c15b380 0x7f1a2c0708d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 -- 172.21.15.122:0/139419757 >> 172.21.15.122:0/139419757 conn(0x7f1a2c0b1c80 msgr2=0x7f1a2c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 -- 172.21.15.122:0/139419757 shutdown_connections 2024-03-06T07:22:37.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.280+0000 7f1a32f3e640 1 -- 172.21.15.122:0/139419757 wait complete. 2024-03-06T07:22:37.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.281+0000 7f1a32f3e640 1 Processor -- start 2024-03-06T07:22:37.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.281+0000 7f1a32f3e640 1 -- start start 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a32f3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c070e10 0x7f1a2c1f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a32f3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 0x7f1a2c1f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a32f3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c15b380 0x7f1a2c15a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a32f3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c06dbb0 con 0x7f1a2c158880 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a32f3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c06d8d0 con 0x7f1a2c070e10 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a32f3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0cded0 con 0x7f1a2c15b380 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 0x7f1a2c1f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 0x7f1a2c1f1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56304/0 (socket says 172.21.15.122:56304) 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 -- 172.21.15.122:0/1305434577 learned_addr learned my addr 172.21.15.122:0/1305434577 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a314db640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c15b380 0x7f1a2c15a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a30cda640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c070e10 0x7f1a2c1f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 -- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c15b380 msgr2=0x7f1a2c15a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c15b380 0x7f1a2c15a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 -- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c070e10 msgr2=0x7f1a2c1f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c070e10 0x7f1a2c1f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 -- 172.21.15.122:0/1305434577 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c1f2240 con 0x7f1a2c158880 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.282+0000 7f1a2bfff640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 0x7f1a2c1f1980 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1a200098c0 tx=0x7f1a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.283+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20047070 con 0x7f1a2c158880 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.283+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20039a10 con 0x7f1a2c158880 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.283+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20034040 con 0x7f1a2c158880 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.283+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c1f2520 con 0x7f1a2c158880 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.283+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c1f52d0 con 0x7f1a2c158880 2024-03-06T07:22:37.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.284+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c04f330 con 0x7f1a2c158880 2024-03-06T07:22:37.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.284+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a2004a070 con 0x7f1a2c158880 2024-03-06T07:22:37.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.284+0000 7f1a297fa640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.284+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1a200abfc0 con 0x7f1a2c158880 2024-03-06T07:22:37.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.285+0000 7f1a30cda640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.285+0000 7f1a30cda640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c000f60 tx=0x7f1a1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.289+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a2007a890 con 0x7f1a2c158880 2024-03-06T07:22:37.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.472+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1a2c0b4670 con 0x7f1a2c158880 2024-03-06T07:22:37.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.472+0000 7f1a297fa640 1 -- 172.21.15.122:0/1305434577 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1a2007e930 con 0x7f1a2c158880 2024-03-06T07:22:37.475 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:37.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 msgr2=0x7f1a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c000f60 tx=0x7f1a1c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 msgr2=0x7f1a2c1f1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 0x7f1a2c1f1980 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1a200098c0 tx=0x7f1a200046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 shutdown_connections 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c15b380 0x7f1a2c15a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c158880 0x7f1a2c1f1980 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 --2- 172.21.15.122:0/1305434577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c070e10 0x7f1a2c1f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 >> 172.21.15.122:0/1305434577 conn(0x7f1a2c0b1c80 msgr2=0x7f1a2c15c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 shutdown_connections 2024-03-06T07:22:37.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.474+0000 7f1a32f3e640 1 -- 172.21.15.122:0/1305434577 wait complete. 2024-03-06T07:22:37.487 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:22:37.487 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:37.503 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:37.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.601+0000 7f1e36895640 1 Processor -- start 2024-03-06T07:22:37.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.601+0000 7f1e36895640 1 -- start start 2024-03-06T07:22:37.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.601+0000 7f1e36895640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1e3015bf30 0x7f1e3015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.601+0000 7f1e36895640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3015f580 con 0x7f1e300b3060 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.601+0000 7f1e36895640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3015b450 con 0x7f1e3013dc10 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.601+0000 7f1e36895640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e300cf3f0 con 0x7f1e3015bf30 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e34e32640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1e3015bf30 0x7f1e3015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e2f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1e300b3060 0x7f1e3015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49848/0 (socket says 172.21.15.122:49848) 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e2f7fe640 1 -- 172.21.15.122:0/1766566314 learned_addr learned my addr 172.21.15.122:0/1766566314 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e34e32640 1 -- 172.21.15.122:0/1766566314 >> v1:172.21.15.169:6789/0 conn(0x7f1e3013dc10 legacy=0x7f1e300b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e34e32640 1 -- 172.21.15.122:0/1766566314 >> v1:172.21.15.122:6789/0 conn(0x7f1e300b3060 legacy=0x7f1e3015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e2effd640 1 -- 172.21.15.122:0/1766566314 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120231208 0 0) 0x7f1e3015b450 con 0x7f1e3013dc10 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.602+0000 7f1e34e32640 1 -- 172.21.15.122:0/1766566314 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3015b730 con 0x7f1e3015bf30 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e34e32640 1 --2- 172.21.15.122:0/1766566314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3015bf30 0x7f1e3015ef60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009d90 tx=0x7f1e2802bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2864272a6cfebe84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e2effd640 1 -- 172.21.15.122:0/1766566314 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28039070 con 0x7f1e3015bf30 2024-03-06T07:22:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e2effd640 1 -- 172.21.15.122:0/1766566314 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2802ca20 con 0x7f1e3015bf30 2024-03-06T07:22:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e2effd640 1 -- 172.21.15.122:0/1766566314 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2802cdf0 con 0x7f1e3015bf30 2024-03-06T07:22:37.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 -- 172.21.15.122:0/1766566314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3015bf30 msgr2=0x7f1e3015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 --2- 172.21.15.122:0/1766566314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3015bf30 0x7f1e3015ef60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009d90 tx=0x7f1e2802bcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 -- 172.21.15.122:0/1766566314 shutdown_connections 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 --2- 172.21.15.122:0/1766566314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3015bf30 0x7f1e3015ef60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 -- 172.21.15.122:0/1766566314 >> 172.21.15.122:0/1766566314 conn(0x7f1e300cf8a0 msgr2=0x7f1e300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 -- 172.21.15.122:0/1766566314 shutdown_connections 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.603+0000 7f1e36895640 1 -- 172.21.15.122:0/1766566314 wait complete. 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.604+0000 7f1e36895640 1 Processor -- start 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 -- start start 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e300b3060 0x7f1e30151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 0x7f1e30140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e3015bf30 0x7f1e30140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e300718e0 con 0x7f1e3013dc10 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3015b210 con 0x7f1e300b3060 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e36895640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30071a60 con 0x7f1e3015bf30 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e2f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 0x7f1e30140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e2f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 0x7f1e30140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56324/0 (socket says 172.21.15.122:56324) 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e2f7fe640 1 -- 172.21.15.122:0/1244735176 learned_addr learned my addr 172.21.15.122:0/1244735176 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e2ffff640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e300b3060 0x7f1e30151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.605+0000 7f1e34e32640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e3015bf30 0x7f1e30140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2f7fe640 1 -- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e3015bf30 msgr2=0x7f1e30140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2f7fe640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e3015bf30 0x7f1e30140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2f7fe640 1 -- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e300b3060 msgr2=0x7f1e30151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2f7fe640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e300b3060 0x7f1e30151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2f7fe640 1 -- 172.21.15.122:0/1244735176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30143fd0 con 0x7f1e3013dc10 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2f7fe640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 0x7f1e30140600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c800 tx=0x7f1e2400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24018070 con 0x7f1e3013dc10 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e240072d0 con 0x7f1e3013dc10 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e301442b0 con 0x7f1e3013dc10 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e240047e0 con 0x7f1e3013dc10 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.606+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e301400e0 con 0x7f1e3013dc10 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.607+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3004f330 con 0x7f1e3013dc10 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.607+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e2401b070 con 0x7f1e3013dc10 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.608+0000 7f1e2cff9640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e1c06b4d0 0x7f1e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.608+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1e24089690 con 0x7f1e3013dc10 2024-03-06T07:22:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.608+0000 7f1e2ffff640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e1c06b4d0 0x7f1e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.608+0000 7f1e2ffff640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e1c06b4d0 0x7f1e1c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1e20001120 tx=0x7f1e20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.612+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e24057e40 con 0x7f1e3013dc10 2024-03-06T07:22:37.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.786+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e30144480 con 0x7f1e3013dc10 2024-03-06T07:22:37.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.787+0000 7f1e2cff9640 1 -- 172.21.15.122:0/1244735176 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16854 (secure 0 0 0) 0x7f1e2405bee0 con 0x7f1e3013dc10 2024-03-06T07:22:37.789 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:37.789 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":22,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:34.698183+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:37.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e1c06b4d0 msgr2=0x7f1e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e1c06b4d0 0x7f1e1c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1e20001120 tx=0x7f1e20009040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 msgr2=0x7f1e30140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 0x7f1e30140600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c800 tx=0x7f1e2400ccc0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 shutdown_connections 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e1c06b4d0 0x7f1e1c06d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e3015bf30 0x7f1e30140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e3013dc10 0x7f1e30140600 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 --2- 172.21.15.122:0/1244735176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e300b3060 0x7f1e30151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 >> 172.21.15.122:0/1244735176 conn(0x7f1e300cf8a0 msgr2=0x7f1e300af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 shutdown_connections 2024-03-06T07:22:37.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.789+0000 7f1e36895640 1 -- 172.21.15.122:0/1244735176 wait complete. 2024-03-06T07:22:37.802 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:37.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.904+0000 7f8016dd7640 1 Processor -- start 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.904+0000 7f8016dd7640 1 -- start start 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8016dd7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801015bb40 0x7f80100b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8016dd7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80100cfa90 0x7f80100cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8016dd7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801015ff90 con 0x7f801015f7d0 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8016dd7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801006c910 con 0x7f80100cfa90 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8016dd7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8010160270 con 0x7f801015bb40 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80100cfa90 0x7f80100cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80100cfa90 0x7f80100cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56326/0 (socket says 172.21.15.122:56326) 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 -- 172.21.15.122:0/3273119369 learned_addr learned my addr 172.21.15.122:0/3273119369 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8014b73640 1 --2- 172.21.15.122:0/3273119369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801015bb40 0x7f80100b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 -- 172.21.15.122:0/3273119369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801015bb40 msgr2=0x7f80100b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 --2- 172.21.15.122:0/3273119369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801015bb40 0x7f80100b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 -- 172.21.15.122:0/3273119369 >> v1:172.21.15.169:6789/0 conn(0x7f801015f7d0 legacy=0x7f801006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.905+0000 7f8015374640 1 -- 172.21.15.122:0/3273119369 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010160550 con 0x7f80100cfa90 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8015374640 1 --2- 172.21.15.122:0/3273119369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80100cfa90 0x7f80100cff20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8008009e50 tx=0x7f800802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4357f1ad7850961 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f800f7fe640 1 -- 172.21.15.122:0/3273119369 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800802cb00 con 0x7f80100cfa90 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f800f7fe640 1 -- 172.21.15.122:0/3273119369 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800802cca0 con 0x7f80100cfa90 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f800f7fe640 1 -- 172.21.15.122:0/3273119369 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008034610 con 0x7f80100cfa90 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 -- 172.21.15.122:0/3273119369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80100cfa90 msgr2=0x7f80100cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 --2- 172.21.15.122:0/3273119369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80100cfa90 0x7f80100cff20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8008009e50 tx=0x7f800802bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 -- 172.21.15.122:0/3273119369 shutdown_connections 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 --2- 172.21.15.122:0/3273119369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80100cfa90 0x7f80100cff20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 --2- 172.21.15.122:0/3273119369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801015bb40 0x7f80100b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 -- 172.21.15.122:0/3273119369 >> 172.21.15.122:0/3273119369 conn(0x7f801006b190 msgr2=0x7f801006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 -- 172.21.15.122:0/3273119369 shutdown_connections 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.906+0000 7f8016dd7640 1 -- 172.21.15.122:0/3273119369 wait complete. 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.908+0000 7f8016dd7640 1 Processor -- start 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.908+0000 7f8016dd7640 1 -- start start 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.908+0000 7f8016dd7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80100cfa90 0x7f80100adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.908+0000 7f8016dd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801015bb40 0x7f80101464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8016dd7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 0x7f8010071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8016dd7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801006d790 con 0x7f801015f7d0 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8016dd7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801015fc30 con 0x7f801015bb40 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8016dd7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801015bf20 con 0x7f80100cfa90 2024-03-06T07:22:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 0x7f8010071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 0x7f8010071380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56334/0 (socket says 172.21.15.122:56334) 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 -- 172.21.15.122:0/657911029 learned_addr learned my addr 172.21.15.122:0/657911029 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f800ffff640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801015bb40 0x7f80101464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8014b73640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80100cfa90 0x7f80100adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 -- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80100cfa90 msgr2=0x7f80100adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80100cfa90 0x7f80100adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 -- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801015bb40 msgr2=0x7f80101464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801015bb40 0x7f80101464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 -- 172.21.15.122:0/657911029 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100727a0 con 0x7f801015f7d0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8015374640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 0x7f8010071380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8008036040 tx=0x7f8008004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008043020 con 0x7f801015f7d0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800802ce50 con 0x7f801015f7d0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80100718c0 con 0x7f801015f7d0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008009510 con 0x7f801015f7d0 2024-03-06T07:22:37.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.909+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8010071e70 con 0x7f801015f7d0 2024-03-06T07:22:37.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.910+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80100732b0 con 0x7f801015f7d0 2024-03-06T07:22:37.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.911+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8008046020 con 0x7f801015f7d0 2024-03-06T07:22:37.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.911+0000 7f800d7fa640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.911+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f80080ac750 con 0x7f801015f7d0 2024-03-06T07:22:37.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.911+0000 7f8014b73640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.911+0000 7f8014b73640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8004009b40 tx=0x7f8004009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:37.915+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f800804d030 con 0x7f801015f7d0 2024-03-06T07:22:38.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.086+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8010068ff0 con 0x7f801015f7d0 2024-03-06T07:22:38.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.086+0000 7f800d7fa640 1 -- 172.21.15.122:0/657911029 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f800807af00 con 0x7f801015f7d0 2024-03-06T07:22:38.089 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:38.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ff406b4d0 msgr2=0x7f7ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8004009b40 tx=0x7f8004009190 comp rx=0 tx=0).stop 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 msgr2=0x7f8010071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 0x7f8010071380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8008036040 tx=0x7f8008004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 shutdown_connections 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801015f7d0 0x7f8010071380 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801015bb40 0x7f80101464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 --2- 172.21.15.122:0/657911029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80100cfa90 0x7f80100adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 >> 172.21.15.122:0/657911029 conn(0x7f801006b190 msgr2=0x7f80100b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 shutdown_connections 2024-03-06T07:22:38.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.088+0000 7f8016dd7640 1 -- 172.21.15.122:0/657911029 wait complete. 2024-03-06T07:22:38.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:38.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.202+0000 7fc748a40640 1 Processor -- start 2024-03-06T07:22:38.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.202+0000 7fc748a40640 1 -- start start 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc748a40640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc7441586e0 0x7fc7440aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc748a40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7440aef80 0x7fc7440cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc748a40640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc744158ac0 con 0x7fc7440cffd0 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc748a40640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74415b2a0 con 0x7fc7440aef80 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc748a40640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74415b540 con 0x7fc7441586e0 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7440aef80 0x7fc7440cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7440aef80 0x7fc7440cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56348/0 (socket says 172.21.15.122:56348) 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 -- 172.21.15.122:0/2900567699 learned_addr learned my addr 172.21.15.122:0/2900567699 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc74259c640 1 --2- 172.21.15.122:0/2900567699 >> v2:172.21.15.169:3300/0 conn(0x7fc7441586e0 0x7fc7440aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 -- 172.21.15.122:0/2900567699 >> v2:172.21.15.169:3300/0 conn(0x7fc7441586e0 msgr2=0x7fc7440aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 --2- 172.21.15.122:0/2900567699 >> v2:172.21.15.169:3300/0 conn(0x7fc7441586e0 0x7fc7440aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 -- 172.21.15.122:0/2900567699 >> v1:172.21.15.169:6789/0 conn(0x7fc7440cffd0 legacy=0x7fc74406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.203+0000 7fc741d9b640 1 -- 172.21.15.122:0/2900567699 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74415b820 con 0x7fc7440aef80 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.204+0000 7fc741d9b640 1 --2- 172.21.15.122:0/2900567699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7440aef80 0x7fc7440cfa90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc744070d10 tx=0x7fc72c02e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c534aafcf4c0595 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.204+0000 7fc74159a640 1 -- 172.21.15.122:0/2900567699 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c03b070 con 0x7fc7440aef80 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.204+0000 7fc74159a640 1 -- 172.21.15.122:0/2900567699 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c036440 con 0x7fc7440aef80 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.204+0000 7fc74159a640 1 -- 172.21.15.122:0/2900567699 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c036810 con 0x7fc7440aef80 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 -- 172.21.15.122:0/2900567699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7440aef80 msgr2=0x7fc7440cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 --2- 172.21.15.122:0/2900567699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7440aef80 0x7fc7440cfa90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc744070d10 tx=0x7fc72c02e170 comp rx=0 tx=0).stop 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 -- 172.21.15.122:0/2900567699 shutdown_connections 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 --2- 172.21.15.122:0/2900567699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7440aef80 0x7fc7440cfa90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 --2- 172.21.15.122:0/2900567699 >> v2:172.21.15.169:3300/0 conn(0x7fc7441586e0 0x7fc7440aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 -- 172.21.15.122:0/2900567699 >> 172.21.15.122:0/2900567699 conn(0x7fc74406b190 msgr2=0x7fc74406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 -- 172.21.15.122:0/2900567699 shutdown_connections 2024-03-06T07:22:38.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.205+0000 7fc748a40640 1 -- 172.21.15.122:0/2900567699 wait complete. 2024-03-06T07:22:38.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.206+0000 7fc748a40640 1 Processor -- start 2024-03-06T07:22:38.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.206+0000 7fc748a40640 1 -- start start 2024-03-06T07:22:38.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc748a40640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7440aef80 0x7fc7440adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc748a40640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7440cffd0 0x7fc744147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc748a40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 0x7fc744148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc748a40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc744071350 con 0x7fc7441586e0 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc748a40640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74415bde0 con 0x7fc7440cffd0 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc748a40640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440714d0 con 0x7fc7440aef80 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 0x7fc744148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 0x7fc744148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56360/0 (socket says 172.21.15.122:56360) 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 -- 172.21.15.122:0/515805200 learned_addr learned my addr 172.21.15.122:0/515805200 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc74259c640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7440aef80 0x7fc7440adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc741d9b640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7440cffd0 0x7fc744147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 -- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7440aef80 msgr2=0x7fc7440adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7440aef80 0x7fc7440adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 -- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7440cffd0 msgr2=0x7fc744147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7440cffd0 0x7fc744147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 -- 172.21.15.122:0/515805200 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc744148b50 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc741d9b640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7440cffd0 0x7fc744147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.207+0000 7fc742d9d640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 0x7fc744148340 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c7d0 tx=0x7fc73400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.208+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734003b70 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.208+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc734003d30 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.208+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc744148880 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.208+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73400aa90 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.208+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7441f4050 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.209+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7441f3ba0 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.209+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc73400ac30 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.209+0000 7fc73affd640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72406b4d0 0x7fc72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.210+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc73408a050 con 0x7fc7441586e0 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.210+0000 7fc74259c640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72406b4d0 0x7fc72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.210+0000 7fc74259c640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72406b4d0 0x7fc72406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009aa0 tx=0x7fc73c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.214+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc734058800 con 0x7fc7441586e0 2024-03-06T07:22:38.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.386+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc744068ff0 con 0x7fc7441586e0 2024-03-06T07:22:38.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.386+0000 7fc73affd640 1 -- 172.21.15.122:0/515805200 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc73405c8a0 con 0x7fc7441586e0 2024-03-06T07:22:38.389 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72406b4d0 msgr2=0x7fc72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72406b4d0 0x7fc72406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009aa0 tx=0x7fc73c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 msgr2=0x7fc744148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 0x7fc744148340 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c7d0 tx=0x7fc73400cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 shutdown_connections 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72406b4d0 0x7fc72406d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7441586e0 0x7fc744148340 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7440cffd0 0x7fc744147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 --2- 172.21.15.122:0/515805200 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7440aef80 0x7fc7440adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 >> 172.21.15.122:0/515805200 conn(0x7fc74406b190 msgr2=0x7fc74406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 shutdown_connections 2024-03-06T07:22:38.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.389+0000 7fc748a40640 1 -- 172.21.15.122:0/515805200 wait complete. 2024-03-06T07:22:38.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:38.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 Processor -- start 2024-03-06T07:22:38.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 -- start start 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbd981614d0 0x7fbd980d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd980d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd980d2740 con 0x7fbd980d22b0 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd981618b0 con 0x7fbd981614d0 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.502+0000 7fbd9dc71640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980cf400 con 0x7fbd980d1910 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd97fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fbd980d22b0 0x7fbd980d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:55744/0 (socket says 172.21.15.122:55744) 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd97fff640 1 -- 172.21.15.122:0/68306052 learned_addr learned my addr 172.21.15.122:0/68306052 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd977fe640 1 --2- 172.21.15.122:0/68306052 >> v2:172.21.15.169:3300/0 conn(0x7fbd981614d0 0x7fbd980d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd96ffd640 1 --2- 172.21.15.122:0/68306052 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd980d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd967fc640 1 -- 172.21.15.122:0/68306052 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608451092 0 0) 0x7fbd90001490 con 0x7fbd980d22b0 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd967fc640 1 -- 172.21.15.122:0/68306052 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd80003520 con 0x7fbd980d22b0 2024-03-06T07:22:38.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd96ffd640 1 -- 172.21.15.122:0/68306052 >> v2:172.21.15.169:3300/0 conn(0x7fbd981614d0 msgr2=0x7fbd980d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd96ffd640 1 --2- 172.21.15.122:0/68306052 >> v2:172.21.15.169:3300/0 conn(0x7fbd981614d0 0x7fbd980d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd96ffd640 1 -- 172.21.15.122:0/68306052 >> v1:172.21.15.169:6789/0 conn(0x7fbd980d22b0 legacy=0x7fbd980d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd96ffd640 1 -- 172.21.15.122:0/68306052 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd980cf6a0 con 0x7fbd980d1910 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.503+0000 7fbd977fe640 1 --2- 172.21.15.122:0/68306052 >> v2:172.21.15.169:3300/0 conn(0x7fbd981614d0 0x7fbd980d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd96ffd640 1 --2- 172.21.15.122:0/68306052 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd980d1d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd880097a0 tx=0x7fbd8802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=151b8f3ab442ef5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd967fc640 1 -- 172.21.15.122:0/68306052 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88035020 con 0x7fbd980d1910 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd967fc640 1 -- 172.21.15.122:0/68306052 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd88031830 con 0x7fbd980d1910 2024-03-06T07:22:38.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd967fc640 1 -- 172.21.15.122:0/68306052 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88031be0 con 0x7fbd980d1910 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd9dc71640 1 -- 172.21.15.122:0/68306052 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 msgr2=0x7fbd980d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/68306052 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd980d1d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd880097a0 tx=0x7fbd8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd9dc71640 1 -- 172.21.15.122:0/68306052 shutdown_connections 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/68306052 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd980d1d70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/68306052 >> v2:172.21.15.169:3300/0 conn(0x7fbd981614d0 0x7fbd980d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.504+0000 7fbd9dc71640 1 -- 172.21.15.122:0/68306052 >> 172.21.15.122:0/68306052 conn(0x7fbd980d3d40 msgr2=0x7fbd980d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.505+0000 7fbd9dc71640 1 -- 172.21.15.122:0/68306052 shutdown_connections 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.505+0000 7fbd9dc71640 1 -- 172.21.15.122:0/68306052 wait complete. 2024-03-06T07:22:38.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 Processor -- start 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 -- start start 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd98153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd980d22b0 0x7fbd980d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 0x7fbd980d0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980b0910 con 0x7fbd981614d0 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980cfc60 con 0x7fbd980d22b0 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.506+0000 7fbd9dc71640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980b1e40 con 0x7fbd980d1910 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd97fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 0x7fbd980d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd98153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd96ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd980d22b0 0x7fbd980d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd98153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:56656/0 (socket says 172.21.15.122:56656) 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd96ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd980d22b0 0x7fbd980d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:50226/0 (socket says 172.21.15.122:50226) 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 -- 172.21.15.122:0/1350515530 learned_addr learned my addr 172.21.15.122:0/1350515530 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd97fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 0x7fbd980d0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56374/0 (socket says 172.21.15.122:56374) 2024-03-06T07:22:38.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 -- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd980d22b0 msgr2=0x7fbd980d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd980d22b0 0x7fbd980d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 -- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 msgr2=0x7fbd980d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 0x7fbd980d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 -- 172.21.15.122:0/1350515530 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd980c7910 con 0x7fbd980d1910 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd97fff640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 0x7fbd980d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd977fe640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd98153140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00e960 tx=0x7fbd8c00ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c017070 con 0x7fbd980d1910 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c00ca00 con 0x7fbd980d1910 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c00cdd0 con 0x7fbd980d1910 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd980c7bf0 con 0x7fbd980d1910 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.507+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd980c81b0 con 0x7fbd980d1910 2024-03-06T07:22:38.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.508+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd98068ff0 con 0x7fbd980d1910 2024-03-06T07:22:38.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.509+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd8c01a070 con 0x7fbd980d1910 2024-03-06T07:22:38.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.509+0000 7fbd7ffff640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.509+0000 7fbd96ffd640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.509+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbd8c08a120 con 0x7fbd980d1910 2024-03-06T07:22:38.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.509+0000 7fbd96ffd640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd88036000 tx=0x7fbd88008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.514+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd8c058830 con 0x7fbd980d1910 2024-03-06T07:22:38.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.686+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd980b5240 con 0x7fbd980d1910 2024-03-06T07:22:38.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.686+0000 7fbd7ffff640 1 -- 172.21.15.122:0/1350515530 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd8c05c8d0 con 0x7fbd980d1910 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 msgr2=0x7fbd8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd88036000 tx=0x7fbd88008030 comp rx=0 tx=0).stop 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 msgr2=0x7fbd98153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd98153140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00e960 tx=0x7fbd8c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 shutdown_connections 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd981614d0 0x7fbd980d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd980d22b0 0x7fbd980d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 --2- 172.21.15.122:0/1350515530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd980d1910 0x7fbd98153140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 >> 172.21.15.122:0/1350515530 conn(0x7fbd980d3d40 msgr2=0x7fbd9806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 shutdown_connections 2024-03-06T07:22:38.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.689+0000 7fbd9dc71640 1 -- 172.21.15.122:0/1350515530 wait complete. 2024-03-06T07:22:38.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:38.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.801+0000 7ffb14fb5640 1 Processor -- start 2024-03-06T07:22:38.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.801+0000 7ffb14fb5640 1 -- start start 2024-03-06T07:22:38.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb14fb5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb10071e00 0x7ffb100cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb14fb5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb14fb5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100ce120 con 0x7ffb1015bbf0 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb14fb5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb10072250 con 0x7ffb10071e00 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb14fb5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb100cf100 con 0x7ffb100cd7c0 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb07fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb10071e00 0x7ffb100cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb0e59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffb1015bbf0 0x7ffb1015bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49864/0 (socket says 172.21.15.122:49864) 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb07fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb10071e00 0x7ffb100cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56384/0 (socket says 172.21.15.122:56384) 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb0e59c640 1 -- 172.21.15.122:0/2429014716 learned_addr learned my addr 172.21.15.122:0/2429014716 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb0ed9d640 1 --2- 172.21.15.122:0/2429014716 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb07fff640 1 -- 172.21.15.122:0/2429014716 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cd7c0 msgr2=0x7ffb100cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb07fff640 1 --2- 172.21.15.122:0/2429014716 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb07fff640 1 -- 172.21.15.122:0/2429014716 >> v1:172.21.15.122:6789/0 conn(0x7ffb1015bbf0 legacy=0x7ffb1015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb07fff640 1 -- 172.21.15.122:0/2429014716 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10072490 con 0x7ffb10071e00 2024-03-06T07:22:38.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.802+0000 7ffb0dd9b640 1 -- 172.21.15.122:0/2429014716 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039445002 0 0) 0x7ffb100ce120 con 0x7ffb1015bbf0 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb07fff640 1 --2- 172.21.15.122:0/2429014716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb100cebc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffb1006ec30 tx=0x7ffaf802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b6928bbfc77a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb0dd9b640 1 -- 172.21.15.122:0/2429014716 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8035020 con 0x7ffb10071e00 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb0dd9b640 1 -- 172.21.15.122:0/2429014716 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf8031830 con 0x7ffb10071e00 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb0dd9b640 1 -- 172.21.15.122:0/2429014716 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8031c00 con 0x7ffb10071e00 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb14fb5640 1 -- 172.21.15.122:0/2429014716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 msgr2=0x7ffb100cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/2429014716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb100cebc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffb1006ec30 tx=0x7ffaf802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.803+0000 7ffb14fb5640 1 -- 172.21.15.122:0/2429014716 shutdown_connections 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.804+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/2429014716 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.804+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/2429014716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb100cebc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.804+0000 7ffb14fb5640 1 -- 172.21.15.122:0/2429014716 >> 172.21.15.122:0/2429014716 conn(0x7ffb100b1c80 msgr2=0x7ffb100cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.804+0000 7ffb14fb5640 1 -- 172.21.15.122:0/2429014716 shutdown_connections 2024-03-06T07:22:38.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.804+0000 7ffb14fb5640 1 -- 172.21.15.122:0/2429014716 wait complete. 2024-03-06T07:22:38.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.805+0000 7ffb14fb5640 1 Processor -- start 2024-03-06T07:22:38.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.805+0000 7ffb14fb5640 1 -- start start 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.805+0000 7ffb14fb5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb101f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.805+0000 7ffb14fb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb14fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015bbf0 0x7ffb100ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb14fb5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1015c6e0 con 0x7ffb10071e00 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb14fb5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100d0650 con 0x7ffb1015bbf0 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb14fb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100b3680 con 0x7ffb100cd7c0 2024-03-06T07:22:38.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb101f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb101f1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56394/0 (socket says 172.21.15.122:56394) 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 -- 172.21.15.122:0/1119231161 learned_addr learned my addr 172.21.15.122:0/1119231161 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb07fff640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0ed9d640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015bbf0 0x7ffb100ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 -- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cd7c0 msgr2=0x7ffb100cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 -- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015bbf0 msgr2=0x7ffb100ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015bbf0 0x7ffb100ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 -- 172.21.15.122:0/1119231161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10138330 con 0x7ffb10071e00 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb0e59c640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb101f1130 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c460 tx=0x7ffb0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00018070 con 0x7ffb10071e00 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00007500 con 0x7ffb10071e00 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00004a20 con 0x7ffb10071e00 2024-03-06T07:22:38.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb10138610 con 0x7ffb10071e00 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.806+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb101376b0 con 0x7ffb10071e00 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.807+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1004f330 con 0x7ffb10071e00 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.808+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb0001b070 con 0x7ffb10071e00 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.808+0000 7ffb06ffd640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaf006b4d0 0x7ffaf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.808+0000 7ffb07fff640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaf006b4d0 0x7ffaf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.808+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffb00089410 con 0x7ffb10071e00 2024-03-06T07:22:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.809+0000 7ffb07fff640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaf006b4d0 0x7ffaf006d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8036000 tx=0x7ffaf803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.813+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb00057c50 con 0x7ffb10071e00 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.996+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffb1015acd0 con 0x7ffb10071e00 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.996+0000 7ffb06ffd640 1 -- 172.21.15.122:0/1119231161 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffb0005bcf0 con 0x7ffb10071e00 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaf006b4d0 msgr2=0x7ffaf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaf006b4d0 0x7ffaf006d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8036000 tx=0x7ffaf803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 msgr2=0x7ffb101f1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb101f1130 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c460 tx=0x7ffb0000c920 comp rx=0 tx=0).stop 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 shutdown_connections 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaf006b4d0 0x7ffaf006d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015bbf0 0x7ffb100ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cd7c0 0x7ffb100cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 --2- 172.21.15.122:0/1119231161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb10071e00 0x7ffb101f1130 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.998+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 >> 172.21.15.122:0/1119231161 conn(0x7ffb100b1c80 msgr2=0x7ffb10159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.999+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 shutdown_connections 2024-03-06T07:22:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:38.999+0000 7ffb14fb5640 1 -- 172.21.15.122:0/1119231161 wait complete. 2024-03-06T07:22:39.011 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi169.front.sepia.ceph.com... 2024-03-06T07:22:39.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:39.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:39.040 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:39.040 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:22:39.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:39.040 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:22:39.095 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:22:39.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:39.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.194+0000 7f105db9a640 1 Processor -- start 2024-03-06T07:22:39.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.194+0000 7f105db9a640 1 -- start start 2024-03-06T07:22:39.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f105db9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10581515e0 0x7f10580cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f105db9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1058071f40 con 0x7f10580cd7a0 2024-03-06T07:22:39.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f105db9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105815b8b0 con 0x7f105815b430 2024-03-06T07:22:39.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f105db9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1058070a70 con 0x7f10581515e0 2024-03-06T07:22:39.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f1057fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f10580cd7a0 0x7f10580cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49878/0 (socket says 172.21.15.122:49878) 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f1057fff640 1 -- 172.21.15.122:0/799281268 learned_addr learned my addr 172.21.15.122:0/799281268 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10577fe640 1 --2- 172.21.15.122:0/799281268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10581515e0 0x7f10580cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10567fc640 1 -- 172.21.15.122:0/799281268 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438182492 0 0) 0x7f1058071f40 con 0x7f10580cd7a0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10567fc640 1 -- 172.21.15.122:0/799281268 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040003520 con 0x7f10580cd7a0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10567fc640 1 -- 172.21.15.122:0/799281268 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590482451 0 0) 0x7f105815b8b0 con 0x7f105815b430 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10577fe640 1 -- 172.21.15.122:0/799281268 >> v1:172.21.15.169:6789/0 conn(0x7f105815b430 legacy=0x7f1058071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10577fe640 1 -- 172.21.15.122:0/799281268 >> v1:172.21.15.122:6789/0 conn(0x7f10580cd7a0 legacy=0x7f10580cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.195+0000 7f10577fe640 1 -- 172.21.15.122:0/799281268 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058070d50 con 0x7f10581515e0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.196+0000 7f10577fe640 1 --2- 172.21.15.122:0/799281268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10581515e0 0x7f10580cce40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10480097d0 tx=0x7f104802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac8ef8c90aa5834b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.196+0000 7f10567fc640 1 -- 172.21.15.122:0/799281268 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048035020 con 0x7f10581515e0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.196+0000 7f10567fc640 1 -- 172.21.15.122:0/799281268 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1048031890 con 0x7f10581515e0 2024-03-06T07:22:39.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.196+0000 7f10567fc640 1 -- 172.21.15.122:0/799281268 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048031c60 con 0x7f10581515e0 2024-03-06T07:22:39.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 -- 172.21.15.122:0/799281268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10581515e0 msgr2=0x7f10580cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 --2- 172.21.15.122:0/799281268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10581515e0 0x7f10580cce40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10480097d0 tx=0x7f104802b930 comp rx=0 tx=0).stop 2024-03-06T07:22:39.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 -- 172.21.15.122:0/799281268 shutdown_connections 2024-03-06T07:22:39.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 --2- 172.21.15.122:0/799281268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10581515e0 0x7f10580cce40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 -- 172.21.15.122:0/799281268 >> 172.21.15.122:0/799281268 conn(0x7f10580cf8a0 msgr2=0x7f10580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 -- 172.21.15.122:0/799281268 shutdown_connections 2024-03-06T07:22:39.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.197+0000 7f105db9a640 1 -- 172.21.15.122:0/799281268 wait complete. 2024-03-06T07:22:39.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.198+0000 7f105db9a640 1 Processor -- start 2024-03-06T07:22:39.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.198+0000 7f105db9a640 1 -- start start 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f105db9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 0x7f105814c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f105db9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10581515e0 0x7f10580c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f105db9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f105815b430 0x7f10580c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f105db9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580b3f90 con 0x7f10580cd7a0 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f105db9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058071310 con 0x7f10581515e0 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f105db9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580b4110 con 0x7f105815b430 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f10577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 0x7f105814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f10577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 0x7f105814c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56410/0 (socket says 172.21.15.122:56410) 2024-03-06T07:22:39.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f10577fe640 1 -- 172.21.15.122:0/2643867625 learned_addr learned my addr 172.21.15.122:0/2643867625 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f1057fff640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f105815b430 0x7f10580c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f1056ffd640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10581515e0 0x7f10580c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f10577fe640 1 -- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f105815b430 msgr2=0x7f10580c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f10577fe640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f105815b430 0x7f10580c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.199+0000 7f10577fe640 1 -- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10581515e0 msgr2=0x7f10580c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f10577fe640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10581515e0 0x7f10580c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f10577fe640 1 -- 172.21.15.122:0/2643867625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580c62c0 con 0x7f10580cd7a0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f10577fe640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 0x7f105814c500 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f10480097d0 tx=0x7f10480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048035040 con 0x7f10580cd7a0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1048030ca0 con 0x7f10580cd7a0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105814ca40 con 0x7f10580cd7a0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048004270 con 0x7f10580cd7a0 2024-03-06T07:22:39.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.200+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580c5a20 con 0x7f10580cd7a0 2024-03-06T07:22:39.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.201+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058068840 con 0x7f10580cd7a0 2024-03-06T07:22:39.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.201+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1048038020 con 0x7f10580cd7a0 2024-03-06T07:22:39.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.202+0000 7f103ffff640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f104406b4d0 0x7f104406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.202+0000 7f1056ffd640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f104406b4d0 0x7f104406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.202+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f10480ac450 con 0x7f10580cd7a0 2024-03-06T07:22:39.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.202+0000 7f1056ffd640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f104406b4d0 0x7f104406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10581e59e0 tx=0x7f104c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.206+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f104802d040 con 0x7f10580cd7a0 2024-03-06T07:22:39.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.415+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10580b4c30 con 0x7f10580cd7a0 2024-03-06T07:22:39.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.415+0000 7f103ffff640 1 -- 172.21.15.122:0/2643867625 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f104807ac10 con 0x7f10580cd7a0 2024-03-06T07:22:39.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:39.417 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:39.417 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:39.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f104406b4d0 msgr2=0x7f104406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f104406b4d0 0x7f104406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10581e59e0 tx=0x7f104c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 msgr2=0x7f105814c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 0x7f105814c500 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f10480097d0 tx=0x7f10480046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 shutdown_connections 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f104406b4d0 0x7f104406d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f105815b430 0x7f10580c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10581515e0 0x7f10580c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 --2- 172.21.15.122:0/2643867625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10580cd7a0 0x7f105814c500 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 >> 172.21.15.122:0/2643867625 conn(0x7f10580cf8a0 msgr2=0x7f105814e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 shutdown_connections 2024-03-06T07:22:39.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.417+0000 7f105db9a640 1 -- 172.21.15.122:0/2643867625 wait complete. 2024-03-06T07:22:39.429 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:39.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.529+0000 7fcee8240640 1 Processor -- start 2024-03-06T07:22:39.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.529+0000 7fcee8240640 1 -- start start 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee8240640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcee01424f0 0x7fcee00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee8240640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee006da40 0x7fcee015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee8240640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee00cc980 con 0x7fcee00cd7a0 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee8240640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee00cd220 con 0x7fcee01424f0 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee8240640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00ccc60 con 0x7fcee006da40 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcee01424f0 0x7fcee00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcee01424f0 0x7fcee00cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56426/0 (socket says 172.21.15.122:56426) 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 -- 172.21.15.122:0/1537796 learned_addr learned my addr 172.21.15.122:0/1537796 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee67dd640 1 --2- 172.21.15.122:0/1537796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee006da40 0x7fcee015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 -- 172.21.15.122:0/1537796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee006da40 msgr2=0x7fcee015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 --2- 172.21.15.122:0/1537796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee006da40 0x7fcee015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 -- 172.21.15.122:0/1537796 >> v1:172.21.15.169:6789/0 conn(0x7fcee00cd7a0 legacy=0x7fcee006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee4fda640 1 -- 172.21.15.122:0/1537796 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706510799 0 0) 0x7fcee00cc980 con 0x7fcee00cd7a0 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.530+0000 7fcee5fdc640 1 -- 172.21.15.122:0/1537796 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00ccf40 con 0x7fcee01424f0 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.531+0000 7fcee5fdc640 1 --2- 172.21.15.122:0/1537796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee01424f0 0x7fcee00cc2f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009870 tx=0x7fcedc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391e35c57840597a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.531+0000 7fcee4fda640 1 -- 172.21.15.122:0/1537796 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc034070 con 0x7fcee01424f0 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.531+0000 7fcee4fda640 1 -- 172.21.15.122:0/1537796 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc030840 con 0x7fcee01424f0 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.531+0000 7fcee4fda640 1 -- 172.21.15.122:0/1537796 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc030c10 con 0x7fcee01424f0 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 -- 172.21.15.122:0/1537796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee01424f0 msgr2=0x7fcee00cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 --2- 172.21.15.122:0/1537796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee01424f0 0x7fcee00cc2f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009870 tx=0x7fcedc02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 -- 172.21.15.122:0/1537796 shutdown_connections 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 --2- 172.21.15.122:0/1537796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee006da40 0x7fcee015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 --2- 172.21.15.122:0/1537796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee01424f0 0x7fcee00cc2f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 -- 172.21.15.122:0/1537796 >> 172.21.15.122:0/1537796 conn(0x7fcee00cf8a0 msgr2=0x7fcee00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 -- 172.21.15.122:0/1537796 shutdown_connections 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.532+0000 7fcee8240640 1 -- 172.21.15.122:0/1537796 wait complete. 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.533+0000 7fcee8240640 1 Processor -- start 2024-03-06T07:22:39.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.533+0000 7fcee8240640 1 -- start start 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee8240640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcee006da40 0x7fcee00aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee8240640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 0x7fcee015dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee8240640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee01424f0 0x7fcee015e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee8240640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0071ce0 con 0x7fcee00cd7a0 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee8240640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0071b60 con 0x7fcee006da40 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee8240640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0071e60 con 0x7fcee01424f0 2024-03-06T07:22:39.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 0x7fcee015dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 0x7fcee015dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56438/0 (socket says 172.21.15.122:56438) 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 -- 172.21.15.122:0/2761657559 learned_addr learned my addr 172.21.15.122:0/2761657559 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee5fdc640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcee006da40 0x7fcee00aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee67dd640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee01424f0 0x7fcee015e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 -- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee01424f0 msgr2=0x7fcee015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee01424f0 0x7fcee015e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 -- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcee006da40 msgr2=0x7fcee00aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcee006da40 0x7fcee00aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 -- 172.21.15.122:0/2761657559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee01f1af0 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.534+0000 7fcee57db640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 0x7fcee015dcd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fced000c4b0 tx=0x7fced000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.535+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0017070 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.535+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0005480 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.535+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee00a9950 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.535+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0003830 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.535+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee00a9f10 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.536+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee004f330 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.536+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fced001a070 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.537+0000 7fced67fc640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcec406b4d0 0x7fcec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.537+0000 7fcee5fdc640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcec406b4d0 0x7fcec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.537+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fced0089580 con 0x7fcee00cd7a0 2024-03-06T07:22:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.537+0000 7fcee5fdc640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcec406b4d0 0x7fcec406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcedc036000 tx=0x7fcedc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.541+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fced0057d30 con 0x7fcee00cd7a0 2024-03-06T07:22:39.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.712+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcee00cef80 con 0x7fcee00cd7a0 2024-03-06T07:22:39.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.714+0000 7fced67fc640 1 -- 172.21.15.122:0/2761657559 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16854 (secure 0 0 0) 0x7fced005bdd0 con 0x7fcee00cd7a0 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":22,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:34.698183+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcec406b4d0 msgr2=0x7fcec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcec406b4d0 0x7fcec406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcedc036000 tx=0x7fcedc03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 msgr2=0x7fcee015dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 0x7fcee015dcd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fced000c4b0 tx=0x7fced000c970 comp rx=0 tx=0).stop 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 shutdown_connections 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcec406b4d0 0x7fcec406d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcee01424f0 0x7fcee015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcee00cd7a0 0x7fcee015dcd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 --2- 172.21.15.122:0/2761657559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcee006da40 0x7fcee00aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 >> 172.21.15.122:0/2761657559 conn(0x7fcee00cf8a0 msgr2=0x7fcee015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 shutdown_connections 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.716+0000 7fcee8240640 1 -- 172.21.15.122:0/2761657559 wait complete. 2024-03-06T07:22:39.728 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:51.355805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:51.854804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:39.728 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:39.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 Processor -- start 2024-03-06T07:22:39.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 -- start start 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a181614d0 0x7f1a180d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a180d1910 0x7f1a180d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a180d2740 con 0x7f1a180d22b0 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a181618b0 con 0x7f1a180d1910 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.830+0000 7f1a1d981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180cf400 con 0x7f1a181614d0 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a167fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a180d1910 0x7f1a180d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a177fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1a180d22b0 0x7f1a180d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49884/0 (socket says 172.21.15.122:49884) 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a177fe640 1 -- 172.21.15.122:0/3496902632 learned_addr learned my addr 172.21.15.122:0/3496902632 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:39.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a16ffd640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a181614d0 0x7f1a180d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a167fc640 1 -- 172.21.15.122:0/3496902632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a181614d0 msgr2=0x7f1a180d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a167fc640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a181614d0 0x7f1a180d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a167fc640 1 -- 172.21.15.122:0/3496902632 >> v1:172.21.15.122:6789/0 conn(0x7f1a180d22b0 legacy=0x7f1a180d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a15ffb640 1 -- 172.21.15.122:0/3496902632 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893968652 0 0) 0x7f1a10001490 con 0x7f1a180d22b0 2024-03-06T07:22:39.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a167fc640 1 -- 172.21.15.122:0/3496902632 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180cf6a0 con 0x7f1a180d1910 2024-03-06T07:22:39.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.831+0000 7f1a16ffd640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a181614d0 0x7f1a180d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:39.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a167fc640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a180d1910 0x7f1a180d1d70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0097a0 tx=0x7f1a0c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163d83e56846e94e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a15ffb640 1 -- 172.21.15.122:0/3496902632 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c035020 con 0x7f1a180d1910 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a15ffb640 1 -- 172.21.15.122:0/3496902632 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c031810 con 0x7f1a180d1910 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a15ffb640 1 -- 172.21.15.122:0/3496902632 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c031be0 con 0x7f1a180d1910 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 -- 172.21.15.122:0/3496902632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a180d1910 msgr2=0x7f1a180d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a180d1910 0x7f1a180d1d70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0097a0 tx=0x7f1a0c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 -- 172.21.15.122:0/3496902632 shutdown_connections 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a180d1910 0x7f1a180d1d70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 --2- 172.21.15.122:0/3496902632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a181614d0 0x7f1a180d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 -- 172.21.15.122:0/3496902632 >> 172.21.15.122:0/3496902632 conn(0x7f1a180d3d40 msgr2=0x7f1a180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 -- 172.21.15.122:0/3496902632 shutdown_connections 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.832+0000 7f1a1d981640 1 -- 172.21.15.122:0/3496902632 wait complete. 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.833+0000 7f1a1d981640 1 Processor -- start 2024-03-06T07:22:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 -- start start 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a180d1910 0x7f1a180c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a180d22b0 0x7f1a180c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 0x7f1a1806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180b0910 con 0x7f1a181614d0 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180cfc60 con 0x7f1a180d1910 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a1d981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180b1e40 con 0x7f1a180d22b0 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.834+0000 7f1a177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 0x7f1a1806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 0x7f1a1806cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56450/0 (socket says 172.21.15.122:56450) 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 -- 172.21.15.122:0/769969961 learned_addr learned my addr 172.21.15.122:0/769969961 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a16ffd640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a180d1910 0x7f1a180c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a167fc640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a180d22b0 0x7f1a180c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 -- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a180d22b0 msgr2=0x7f1a180c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a180d22b0 0x7f1a180c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 -- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a180d1910 msgr2=0x7f1a180c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a180d1910 0x7f1a180c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 -- 172.21.15.122:0/769969961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180b7350 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a177fe640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 0x7f1a1806cba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000c7f0 tx=0x7f1a1000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a10018070 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1000a5e0 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000aa10 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a180c87c0 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.835+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a180cbf90 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.836+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a18068ff0 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.837+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a1001b070 con 0x7f1a181614d0 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.837+0000 7f19fffff640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a0406b4d0 0x7f1a0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.837+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1a10089730 con 0x7f1a181614d0 2024-03-06T07:22:39.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.837+0000 7f1a16ffd640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a0406b4d0 0x7f1a0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.837+0000 7f1a16ffd640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a0406b4d0 0x7f1a0406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009560 tx=0x7f1a08009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:39.841+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a180b7350 con 0x7f1a181614d0 2024-03-06T07:22:40.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.012+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a18069e80 con 0x7f1a181614d0 2024-03-06T07:22:40.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.012+0000 7f19fffff640 1 -- 172.21.15.122:0/769969961 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a180d2740 con 0x7f1a181614d0 2024-03-06T07:22:40.015 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:40.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a0406b4d0 msgr2=0x7f1a0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a0406b4d0 0x7f1a0406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009560 tx=0x7f1a08009190 comp rx=0 tx=0).stop 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 msgr2=0x7f1a1806cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 0x7f1a1806cba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000c7f0 tx=0x7f1a1000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 shutdown_connections 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a0406b4d0 0x7f1a0406d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a181614d0 0x7f1a1806cba0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a180d22b0 0x7f1a180c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 --2- 172.21.15.122:0/769969961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a180d1910 0x7f1a180c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 >> 172.21.15.122:0/769969961 conn(0x7f1a180d3d40 msgr2=0x7f1a1806bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 shutdown_connections 2024-03-06T07:22:40.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.014+0000 7f1a1d981640 1 -- 172.21.15.122:0/769969961 wait complete. 2024-03-06T07:22:40.026 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:40.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.127+0000 7fa986155640 1 Processor -- start 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.127+0000 7fa986155640 1 -- start start 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa986155640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa98015a220 0x7fa9800cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa986155640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa98006da00 0x7fa980071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa986155640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9800b3320 con 0x7fa9800712d0 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa986155640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9800cf610 con 0x7fa98015a220 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa986155640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa980071c70 con 0x7fa98006da00 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa97f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa98015a220 0x7fa9800cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa97effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa9800712d0 0x7fa98015de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49896/0 (socket says 172.21.15.122:49896) 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa97effd640 1 -- 172.21.15.122:0/3084482714 learned_addr learned my addr 172.21.15.122:0/3084482714 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.128+0000 7fa97ffff640 1 --2- 172.21.15.122:0/3084482714 >> v2:172.21.15.169:3300/0 conn(0x7fa98006da00 0x7fa980071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97f7fe640 1 -- 172.21.15.122:0/3084482714 >> v2:172.21.15.169:3300/0 conn(0x7fa98006da00 msgr2=0x7fa980071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97f7fe640 1 --2- 172.21.15.122:0/3084482714 >> v2:172.21.15.169:3300/0 conn(0x7fa98006da00 0x7fa980071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97f7fe640 1 -- 172.21.15.122:0/3084482714 >> v1:172.21.15.122:6789/0 conn(0x7fa9800712d0 legacy=0x7fa98015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97f7fe640 1 -- 172.21.15.122:0/3084482714 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800728d0 con 0x7fa98015a220 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97e7fc640 1 -- 172.21.15.122:0/3084482714 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718401653 0 0) 0x7fa9800b3320 con 0x7fa9800712d0 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97f7fe640 1 --2- 172.21.15.122:0/3084482714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98015a220 0x7fa9800cf0d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa98015ef00 tx=0x7fa97002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be294c3e99f71776 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.129+0000 7fa97e7fc640 1 -- 172.21.15.122:0/3084482714 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9700315a0 con 0x7fa98015a220 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa97e7fc640 1 -- 172.21.15.122:0/3084482714 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa970037070 con 0x7fa98015a220 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa97e7fc640 1 -- 172.21.15.122:0/3084482714 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970031960 con 0x7fa98015a220 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 -- 172.21.15.122:0/3084482714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98015a220 msgr2=0x7fa9800cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 --2- 172.21.15.122:0/3084482714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98015a220 0x7fa9800cf0d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa98015ef00 tx=0x7fa97002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:40.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 -- 172.21.15.122:0/3084482714 shutdown_connections 2024-03-06T07:22:40.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 --2- 172.21.15.122:0/3084482714 >> v2:172.21.15.169:3300/0 conn(0x7fa98006da00 0x7fa980071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 --2- 172.21.15.122:0/3084482714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98015a220 0x7fa9800cf0d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 -- 172.21.15.122:0/3084482714 >> 172.21.15.122:0/3084482714 conn(0x7fa98006b190 msgr2=0x7fa98006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 -- 172.21.15.122:0/3084482714 shutdown_connections 2024-03-06T07:22:40.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.130+0000 7fa986155640 1 -- 172.21.15.122:0/3084482714 wait complete. 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.131+0000 7fa986155640 1 Processor -- start 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.131+0000 7fa986155640 1 -- start start 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa986155640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa98006da00 0x7fa980152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa986155640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 0x7fa9800addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa986155640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa98015a220 0x7fa9800b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa986155640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa980072030 con 0x7fa9800712d0 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa986155640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa980072bb0 con 0x7fa98006da00 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa986155640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa980072530 con 0x7fa98015a220 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 0x7fa9800addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 0x7fa9800addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56462/0 (socket says 172.21.15.122:56462) 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 -- 172.21.15.122:0/823234848 learned_addr learned my addr 172.21.15.122:0/823234848 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:40.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97ffff640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa98015a220 0x7fa9800b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97f7fe640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa98006da00 0x7fa980152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 -- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa98015a220 msgr2=0x7fa9800b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa98015a220 0x7fa9800b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 -- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa98006da00 msgr2=0x7fa980152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa98006da00 0x7fa980152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.132+0000 7fa97effd640 1 -- 172.21.15.122:0/823234848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98013e1d0 con 0x7fa9800712d0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.133+0000 7fa97effd640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 0x7fa9800addf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c500 tx=0x7fa97400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.133+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974018070 con 0x7fa9800712d0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.133+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa974007580 con 0x7fa9800712d0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.133+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98013e4b0 con 0x7fa9800712d0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.133+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974004a20 con 0x7fa9800712d0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.133+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98013ea70 con 0x7fa9800712d0 2024-03-06T07:22:40.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.134+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9800732b0 con 0x7fa9800712d0 2024-03-06T07:22:40.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.134+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa97401b070 con 0x7fa9800712d0 2024-03-06T07:22:40.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.135+0000 7fa967fff640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96c06b4d0 0x7fa96c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.135+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa974089490 con 0x7fa9800712d0 2024-03-06T07:22:40.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.135+0000 7fa97f7fe640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96c06b4d0 0x7fa96c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.135+0000 7fa97f7fe640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96c06b4d0 0x7fa96c06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa970036000 tx=0x7fa97003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.139+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa974057cd0 con 0x7fa9800712d0 2024-03-06T07:22:40.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.309+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa980070ba0 con 0x7fa9800712d0 2024-03-06T07:22:40.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.310+0000 7fa967fff640 1 -- 172.21.15.122:0/823234848 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa97405bd70 con 0x7fa9800712d0 2024-03-06T07:22:40.311 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96c06b4d0 msgr2=0x7fa96c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96c06b4d0 0x7fa96c06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa970036000 tx=0x7fa97003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 msgr2=0x7fa9800addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 0x7fa9800addf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c500 tx=0x7fa97400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 shutdown_connections 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96c06b4d0 0x7fa96c06d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa98015a220 0x7fa9800b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9800712d0 0x7fa9800addf0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 --2- 172.21.15.122:0/823234848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa98006da00 0x7fa980152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 >> 172.21.15.122:0/823234848 conn(0x7fa98006b190 msgr2=0x7fa9800cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 shutdown_connections 2024-03-06T07:22:40.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.312+0000 7fa986155640 1 -- 172.21.15.122:0/823234848 wait complete. 2024-03-06T07:22:40.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:40.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.425+0000 7f428a25c640 1 Processor -- start 2024-03-06T07:22:40.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.425+0000 7f428a25c640 1 -- start start 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f428a25c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f428a25c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f428a25c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f428a25c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42840d03f0 con 0x7f42840cffd0 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f428a25c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4284158ac0 con 0x7f42841586e0 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f428a25c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428415b2a0 con 0x7f42840aef80 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56464/0 (socket says 172.21.15.122:56464) 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 -- 172.21.15.122:0/3378271624 learned_addr learned my addr 172.21.15.122:0/3378271624 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3378271624 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f42837fe640 1 --2- 172.21.15.122:0/3378271624 >> v2:172.21.15.169:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 -- 172.21.15.122:0/3378271624 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 msgr2=0x7f42840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 --2- 172.21.15.122:0/3378271624 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 -- 172.21.15.122:0/3378271624 >> v2:172.21.15.169:3300/0 conn(0x7f42841586e0 msgr2=0x7f42840aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 --2- 172.21.15.122:0/3378271624 >> v2:172.21.15.169:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.426+0000 7f4283fff640 1 -- 172.21.15.122:0/3378271624 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428415b540 con 0x7f42840cffd0 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.427+0000 7f4283fff640 1 --2- 172.21.15.122:0/3378271624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428406da20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4284070d10 tx=0x7f427c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fd87939fac0439 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.427+0000 7f42827fc640 1 -- 172.21.15.122:0/3378271624 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c02ea80 con 0x7f42840cffd0 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.427+0000 7f42827fc640 1 -- 172.21.15.122:0/3378271624 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c02ec20 con 0x7f42840cffd0 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.427+0000 7f42827fc640 1 -- 172.21.15.122:0/3378271624 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c009600 con 0x7f42840cffd0 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 -- 172.21.15.122:0/3378271624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 msgr2=0x7f428406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 --2- 172.21.15.122:0/3378271624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428406da20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4284070d10 tx=0x7f427c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 -- 172.21.15.122:0/3378271624 shutdown_connections 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 --2- 172.21.15.122:0/3378271624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 --2- 172.21.15.122:0/3378271624 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 --2- 172.21.15.122:0/3378271624 >> v2:172.21.15.169:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 -- 172.21.15.122:0/3378271624 >> 172.21.15.122:0/3378271624 conn(0x7f428406b190 msgr2=0x7f428406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 -- 172.21.15.122:0/3378271624 shutdown_connections 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.428+0000 7f428a25c640 1 -- 172.21.15.122:0/3378271624 wait complete. 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.429+0000 7f428a25c640 1 Processor -- start 2024-03-06T07:22:40.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.429+0000 7f428a25c640 1 -- start start 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f428a25c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42841453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f428a25c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f42840ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f428a25c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42841586e0 0x7f428406d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f428a25c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428415bde0 con 0x7f42840cffd0 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f428a25c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428415bb00 con 0x7f42841586e0 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f428a25c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4284071350 con 0x7f42840aef80 2024-03-06T07:22:40.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f42840ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f42840ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56466/0 (socket says 172.21.15.122:56466) 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 -- 172.21.15.122:0/3014310061 learned_addr learned my addr 172.21.15.122:0/3014310061 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f42837fe640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42841453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4283fff640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42841586e0 0x7f428406d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 -- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 msgr2=0x7f42841453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42841453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 -- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42841586e0 msgr2=0x7f428406d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42841586e0 0x7f428406d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 -- 172.21.15.122:0/3014310061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840ad0b0 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.430+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f42840ab330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f427800e890 tx=0x7f427800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.431+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278017070 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.431+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427800c940 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.431+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42840ad390 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.431+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427800cd10 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.431+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42840ad950 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.432+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42840732b0 con 0x7f42840cffd0 2024-03-06T07:22:40.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.432+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f427801a070 con 0x7f42840cffd0 2024-03-06T07:22:40.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.433+0000 7f426bfff640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427006b4d0 0x7f427006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.433+0000 7f42837fe640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427006b4d0 0x7f427006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.433+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f427808a450 con 0x7f42840cffd0 2024-03-06T07:22:40.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.433+0000 7f42837fe640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427006b4d0 0x7f427006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4274009aa0 tx=0x7f4274009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.437+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4278058c90 con 0x7f42840cffd0 2024-03-06T07:22:40.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.609+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4284070a80 con 0x7f42840cffd0 2024-03-06T07:22:40.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.610+0000 7f426bfff640 1 -- 172.21.15.122:0/3014310061 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f427805cd30 con 0x7f42840cffd0 2024-03-06T07:22:40.611 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427006b4d0 msgr2=0x7f427006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427006b4d0 0x7f427006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4274009aa0 tx=0x7f4274009190 comp rx=0 tx=0).stop 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 msgr2=0x7f42840ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f42840ab330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f427800e890 tx=0x7f427800ed50 comp rx=0 tx=0).stop 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 shutdown_connections 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427006b4d0 0x7f427006d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42841586e0 0x7f428406d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f42840ab330 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 --2- 172.21.15.122:0/3014310061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840aef80 0x7f42841453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 >> 172.21.15.122:0/3014310061 conn(0x7f428406b190 msgr2=0x7f42840b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 shutdown_connections 2024-03-06T07:22:40.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.612+0000 7f428a25c640 1 -- 172.21.15.122:0/3014310061 wait complete. 2024-03-06T07:22:40.624 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:40.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.723+0000 7f78f297a640 1 Processor -- start 2024-03-06T07:22:40.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.723+0000 7f78f297a640 1 -- start start 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f297a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78ec166af0 0x7f78ec168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f297a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ec15e380 con 0x7f78ec0cf850 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f297a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ec0cdc10 con 0x7f78ec0d00c0 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f297a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ec06ca90 con 0x7f78ec166af0 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f0f17640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78ec166af0 0x7f78ec168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78eb7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f78ec0cf850 0x7f78ec0cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49904/0 (socket says 172.21.15.122:49904) 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78eb7fe640 1 -- 172.21.15.122:0/900817345 learned_addr learned my addr 172.21.15.122:0/900817345 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:40.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f0f17640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78ec166af0 0x7f78ec168ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56474/0 (socket says 172.21.15.122:56474) 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f0f17640 1 -- 172.21.15.122:0/900817345 >> v1:172.21.15.169:6789/0 conn(0x7f78ec0d00c0 legacy=0x7f78ec0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f0f17640 1 -- 172.21.15.122:0/900817345 >> v1:172.21.15.122:6789/0 conn(0x7f78ec0cf850 legacy=0x7f78ec0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.724+0000 7f78f0f17640 1 -- 172.21.15.122:0/900817345 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec06c160 con 0x7f78ec166af0 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.725+0000 7f78f0f17640 1 --2- 172.21.15.122:0/900817345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec168ee0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009d90 tx=0x7f78e402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7f6540d7a5405a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.725+0000 7f78eaffd640 1 -- 172.21.15.122:0/900817345 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4039070 con 0x7f78ec166af0 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.725+0000 7f78eaffd640 1 -- 172.21.15.122:0/900817345 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e402cd00 con 0x7f78ec166af0 2024-03-06T07:22:40.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.725+0000 7f78eaffd640 1 -- 172.21.15.122:0/900817345 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4034660 con 0x7f78ec166af0 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 -- 172.21.15.122:0/900817345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 msgr2=0x7f78ec168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 --2- 172.21.15.122:0/900817345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec168ee0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009d90 tx=0x7f78e402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 -- 172.21.15.122:0/900817345 shutdown_connections 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 --2- 172.21.15.122:0/900817345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec168ee0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 -- 172.21.15.122:0/900817345 >> 172.21.15.122:0/900817345 conn(0x7f78ec0b35a0 msgr2=0x7f78ec0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 -- 172.21.15.122:0/900817345 shutdown_connections 2024-03-06T07:22:40.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.726+0000 7f78f297a640 1 -- 172.21.15.122:0/900817345 wait complete. 2024-03-06T07:22:40.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.727+0000 7f78f297a640 1 Processor -- start 2024-03-06T07:22:40.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.727+0000 7f78f297a640 1 -- start start 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f297a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78ec0cf850 0x7f78ec0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f297a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78ec0d00c0 0x7f78ec1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f297a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec1574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f297a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec06c720 con 0x7f78ec166af0 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f297a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec06df30 con 0x7f78ec0cf850 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f297a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec0b1030 con 0x7f78ec0d00c0 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec1574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec1574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56476/0 (socket says 172.21.15.122:56476) 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 -- 172.21.15.122:0/1384651848 learned_addr learned my addr 172.21.15.122:0/1384651848 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78eb7fe640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78ec0d00c0 0x7f78ec1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78ebfff640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78ec0cf850 0x7f78ec0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 -- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78ec0d00c0 msgr2=0x7f78ec1556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78ec0d00c0 0x7f78ec1556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 -- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78ec0cf850 msgr2=0x7f78ec0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78ec0cf850 0x7f78ec0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 -- 172.21.15.122:0/1384651848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec158700 con 0x7f78ec166af0 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.728+0000 7f78f0f17640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec1574d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009d90 tx=0x7f78e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.729+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4039050 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.729+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4045070 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.729+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e403e830 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.729+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ec1589a0 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.729+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ec0c7130 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.730+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ec04f330 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.730+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78e403e9d0 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.730+0000 7f78e8ff9640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b500 0x7f78d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.731+0000 7f78ebfff640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b500 0x7f78d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.731+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f78e40acf50 con 0x7f78ec166af0 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.731+0000 7f78ebfff640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b500 0x7f78d806d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f78dc001780 tx=0x7f78dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.735+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78e407b7a0 con 0x7f78ec166af0 2024-03-06T07:22:40.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.940+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78ec0b1790 con 0x7f78ec166af0 2024-03-06T07:22:40.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.940+0000 7f78e8ff9640 1 -- 172.21.15.122:0/1384651848 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f78e407f840 con 0x7f78ec166af0 2024-03-06T07:22:40.942 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:40.942 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:40.942 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b500 msgr2=0x7f78d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b500 0x7f78d806d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f78dc001780 tx=0x7f78dc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 msgr2=0x7f78ec1574d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec1574d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009d90 tx=0x7f78e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 shutdown_connections 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b500 0x7f78d806d9c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78ec166af0 0x7f78ec1574d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78ec0d00c0 0x7f78ec1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 --2- 172.21.15.122:0/1384651848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78ec0cf850 0x7f78ec0c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 >> 172.21.15.122:0/1384651848 conn(0x7f78ec0b35a0 msgr2=0x7f78ec0ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 shutdown_connections 2024-03-06T07:22:40.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:40.943+0000 7f78f297a640 1 -- 172.21.15.122:0/1384651848 wait complete. 2024-03-06T07:22:40.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:41.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.052+0000 7f701f7bc640 1 Processor -- start 2024-03-06T07:22:41.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.053+0000 7f701f7bc640 1 -- start start 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.053+0000 7f701f7bc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f70180cf100 0x7f70180cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.053+0000 7f701f7bc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180ccc70 con 0x7f701806e270 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.053+0000 7f701f7bc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701815e440 con 0x7f70180cc810 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.053+0000 7f701f7bc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701806e680 con 0x7f70180cf100 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f701dd59640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f701806e270 0x7f70180ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49906/0 (socket says 172.21.15.122:49906) 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f701dd59640 1 -- 172.21.15.122:0/3653637366 learned_addr learned my addr 172.21.15.122:0/3653637366 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f701d558640 1 --2- 172.21.15.122:0/3653637366 >> v2:172.21.15.169:3300/0 conn(0x7f70180cf100 0x7f70180cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f700ffff640 1 -- 172.21.15.122:0/3653637366 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248397943 0 0) 0x7f70180ccc70 con 0x7f701806e270 2024-03-06T07:22:41.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f700ffff640 1 -- 172.21.15.122:0/3653637366 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004003520 con 0x7f701806e270 2024-03-06T07:22:41.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f701d558640 1 -- 172.21.15.122:0/3653637366 >> v1:172.21.15.169:6789/0 conn(0x7f70180cc810 legacy=0x7f701806db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f701d558640 1 -- 172.21.15.122:0/3653637366 >> v1:172.21.15.122:6789/0 conn(0x7f701806e270 legacy=0x7f70180ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.054+0000 7f701d558640 1 -- 172.21.15.122:0/3653637366 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701815e700 con 0x7f70180cf100 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f701d558640 1 --2- 172.21.15.122:0/3653637366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70180cf100 0x7f70180cc2d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70080097a0 tx=0x7f700802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f47147dfa089f1fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f700ffff640 1 -- 172.21.15.122:0/3653637366 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008035020 con 0x7f70180cf100 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f700ffff640 1 -- 172.21.15.122:0/3653637366 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008031830 con 0x7f70180cf100 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f700ffff640 1 -- 172.21.15.122:0/3653637366 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008031c00 con 0x7f70180cf100 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f701f7bc640 1 -- 172.21.15.122:0/3653637366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70180cf100 msgr2=0x7f70180cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f701f7bc640 1 --2- 172.21.15.122:0/3653637366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70180cf100 0x7f70180cc2d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70080097a0 tx=0x7f700802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f701f7bc640 1 -- 172.21.15.122:0/3653637366 shutdown_connections 2024-03-06T07:22:41.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f701f7bc640 1 --2- 172.21.15.122:0/3653637366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70180cf100 0x7f70180cc2d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.055+0000 7f701f7bc640 1 -- 172.21.15.122:0/3653637366 >> 172.21.15.122:0/3653637366 conn(0x7f70180cf8a0 msgr2=0x7f70180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.056+0000 7f701f7bc640 1 -- 172.21.15.122:0/3653637366 shutdown_connections 2024-03-06T07:22:41.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.056+0000 7f701f7bc640 1 -- 172.21.15.122:0/3653637366 wait complete. 2024-03-06T07:22:41.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 Processor -- start 2024-03-06T07:22:41.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 -- start start 2024-03-06T07:22:41.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f701806e270 0x7f70181601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70180cc810 0x7f7018159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 0x7f7018071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701815efa0 con 0x7f70180cf100 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701815ecc0 con 0x7f701806e270 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701f7bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70180cf4e0 con 0x7f70180cc810 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.057+0000 7f701dd59640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 0x7f7018071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 0x7f7018071a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56480/0 (socket says 172.21.15.122:56480) 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 -- 172.21.15.122:0/2357553188 learned_addr learned my addr 172.21.15.122:0/2357553188 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701d558640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f701806e270 0x7f70181601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701cd57640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70180cc810 0x7f7018159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 -- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70180cc810 msgr2=0x7f7018159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70180cc810 0x7f7018159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 -- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f701806e270 msgr2=0x7f70181601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f701806e270 0x7f70181601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 -- 172.21.15.122:0/2357553188 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70181f4640 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701cd57640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70180cc810 0x7f7018159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701dd59640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 0x7f7018071a30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f701000cb70 tx=0x7f701000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010017070 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701000ad40 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010012620 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701815f710 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.058+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701815fcd0 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.059+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701804f330 con 0x7f70180cf100 2024-03-06T07:22:41.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.059+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f701001a070 con 0x7f70180cf100 2024-03-06T07:22:41.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.060+0000 7f700dffb640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ffc06b4d0 0x7f6ffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.060+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7010089fc0 con 0x7f70180cf100 2024-03-06T07:22:41.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.060+0000 7f701d558640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ffc06b4d0 0x7f6ffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.060+0000 7f701d558640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ffc06b4d0 0x7f6ffc06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7008036000 tx=0x7f700803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.064+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7010058800 con 0x7f70180cf100 2024-03-06T07:22:41.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.237+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70180d0770 con 0x7f70180cf100 2024-03-06T07:22:41.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.238+0000 7f700dffb640 1 -- 172.21.15.122:0/2357553188 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16854 (secure 0 0 0) 0x7f701005c8a0 con 0x7f70180cf100 2024-03-06T07:22:41.239 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:41.239 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":22,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:34.698183+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:41.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ffc06b4d0 msgr2=0x7f6ffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ffc06b4d0 0x7f6ffc06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7008036000 tx=0x7f700803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 msgr2=0x7f7018071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 0x7f7018071a30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f701000cb70 tx=0x7f701000a430 comp rx=0 tx=0).stop 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 shutdown_connections 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ffc06b4d0 0x7f6ffc06d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70180cf100 0x7f7018071a30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70180cc810 0x7f7018159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 --2- 172.21.15.122:0/2357553188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f701806e270 0x7f70181601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 >> 172.21.15.122:0/2357553188 conn(0x7f70180cf8a0 msgr2=0x7f70180b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 shutdown_connections 2024-03-06T07:22:41.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.240+0000 7f701f7bc640 1 -- 172.21.15.122:0/2357553188 wait complete. 2024-03-06T07:22:41.252 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:41.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:41.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.351+0000 7f02a12e0640 1 Processor -- start 2024-03-06T07:22:41.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.351+0000 7f02a12e0640 1 -- start start 2024-03-06T07:22:41.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f02a12e0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f029c0cfff0 0x7f029c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f02a12e0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c161980 con 0x7f029c0cc8b0 2024-03-06T07:22:41.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f02a12e0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c161c60 con 0x7f029c15bb20 2024-03-06T07:22:41.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f02a12e0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c0d2500 con 0x7f029c0cfff0 2024-03-06T07:22:41.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f029a7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f029c0cfff0 0x7f029c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f029a7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f029c0cfff0 0x7f029c0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56484/0 (socket says 172.21.15.122:56484) 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f029a7fc640 1 -- 172.21.15.122:0/3860782557 learned_addr learned my addr 172.21.15.122:0/3860782557 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.352+0000 7f029b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f029c0cc8b0 0x7f029c0ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49918/0 (socket says 172.21.15.122:49918) 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f029a7fc640 1 -- 172.21.15.122:0/3860782557 >> v1:172.21.15.169:6789/0 conn(0x7f029c15bb20 legacy=0x7f029c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f029a7fc640 1 -- 172.21.15.122:0/3860782557 >> v1:172.21.15.122:6789/0 conn(0x7f029c0cc8b0 legacy=0x7f029c0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f029a7fc640 1 -- 172.21.15.122:0/3860782557 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c161f40 con 0x7f029c0cfff0 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f0299ffb640 1 -- 172.21.15.122:0/3860782557 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197080229 0 0) 0x7f029c161980 con 0x7f029c0cc8b0 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f029a7fc640 1 --2- 172.21.15.122:0/3860782557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cfff0 0x7f029c0d04a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0290009870 tx=0x7f029002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e83135f44ce8d97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f0299ffb640 1 -- 172.21.15.122:0/3860782557 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029002f840 con 0x7f029c0cfff0 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.353+0000 7f0299ffb640 1 -- 172.21.15.122:0/3860782557 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029002f9e0 con 0x7f029c0cfff0 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f0299ffb640 1 -- 172.21.15.122:0/3860782557 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029002fdb0 con 0x7f029c0cfff0 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 -- 172.21.15.122:0/3860782557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cfff0 msgr2=0x7f029c0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 --2- 172.21.15.122:0/3860782557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cfff0 0x7f029c0d04a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0290009870 tx=0x7f029002ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 -- 172.21.15.122:0/3860782557 shutdown_connections 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 --2- 172.21.15.122:0/3860782557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cfff0 0x7f029c0d04a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 -- 172.21.15.122:0/3860782557 >> 172.21.15.122:0/3860782557 conn(0x7f029c09a9f0 msgr2=0x7f029c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 -- 172.21.15.122:0/3860782557 shutdown_connections 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.354+0000 7f02a12e0640 1 -- 172.21.15.122:0/3860782557 wait complete. 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.355+0000 7f02a12e0640 1 Processor -- start 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.355+0000 7f02a12e0640 1 -- start start 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f02a12e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 0x7f029c0b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f02a12e0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f029c0cfff0 0x7f029c1e7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f02a12e0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f029c15bb20 0x7f029c0c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f02a12e0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0d1980 con 0x7f029c0cc8b0 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f02a12e0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c162500 con 0x7f029c15bb20 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f02a12e0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0d1b00 con 0x7f029c0cfff0 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 0x7f029c0b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 0x7f029c0b55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56494/0 (socket says 172.21.15.122:56494) 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 -- 172.21.15.122:0/1803210609 learned_addr learned my addr 172.21.15.122:0/1803210609 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029a7fc640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f029c0cfff0 0x7f029c1e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029b7fe640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f029c15bb20 0x7f029c0c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 -- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f029c0cfff0 msgr2=0x7f029c1e7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f029c0cfff0 0x7f029c1e7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 -- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f029c15bb20 msgr2=0x7f029c0c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f029c15bb20 0x7f029c0c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.356+0000 7f029affd640 1 -- 172.21.15.122:0/1803210609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0b5ba0 con 0x7f029c0cc8b0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.357+0000 7f029affd640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 0x7f029c0b55b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c870 tx=0x7f028c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.357+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c003b70 con 0x7f029c0cc8b0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.357+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c003d30 con 0x7f029c0cc8b0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.357+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c0b5e80 con 0x7f029c0cc8b0 2024-03-06T07:22:41.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.357+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00aa90 con 0x7f029c0cc8b0 2024-03-06T07:22:41.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.357+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c0b6440 con 0x7f029c0cc8b0 2024-03-06T07:22:41.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.358+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c04f330 con 0x7f029c0cc8b0 2024-03-06T07:22:41.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.358+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f028c00ac30 con 0x7f029c0cc8b0 2024-03-06T07:22:41.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.358+0000 7f02837fe640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.359+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f028c08a0d0 con 0x7f029c0cc8b0 2024-03-06T07:22:41.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.359+0000 7f029a7fc640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.359+0000 7f029a7fc640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f028806b4d0 0x7f028806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0290036000 tx=0x7f0290008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.363+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f028c058880 con 0x7f029c0cc8b0 2024-03-06T07:22:41.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.573+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f029c0cdff0 con 0x7f029c0cc8b0 2024-03-06T07:22:41.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.573+0000 7f02837fe640 1 -- 172.21.15.122:0/1803210609 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f028c05c920 con 0x7f029c0cc8b0 2024-03-06T07:22:41.574 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:41.575 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:41.575 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f028806b4d0 msgr2=0x7f028806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f028806b4d0 0x7f028806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0290036000 tx=0x7f0290008030 comp rx=0 tx=0).stop 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 msgr2=0x7f029c0b55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 0x7f029c0b55b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c870 tx=0x7f028c00cd30 comp rx=0 tx=0).stop 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 shutdown_connections 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f029c15bb20 0x7f029c0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f029c0cfff0 0x7f029c1e7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 --2- 172.21.15.122:0/1803210609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f029c0cc8b0 0x7f029c0b55b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 >> 172.21.15.122:0/1803210609 conn(0x7f029c09a9f0 msgr2=0x7f029c0ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 shutdown_connections 2024-03-06T07:22:41.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.575+0000 7f02a12e0640 1 -- 172.21.15.122:0/1803210609 wait complete. 2024-03-06T07:22:41.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:41.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.685+0000 7ff7d4c9d640 1 Processor -- start 2024-03-06T07:22:41.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.685+0000 7ff7d4c9d640 1 -- start start 2024-03-06T07:22:41.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.685+0000 7ff7d4c9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff7d00d2710 0x7ff7d00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.685+0000 7ff7d4c9d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00b3430 con 0x7ff7d00d22b0 2024-03-06T07:22:41.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.685+0000 7ff7d4c9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00b3710 con 0x7ff7d015dfe0 2024-03-06T07:22:41.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.685+0000 7ff7d4c9d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d00b31e0 con 0x7ff7d00d2710 2024-03-06T07:22:41.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.686+0000 7ff7c5d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff7d00d22b0 0x7ff7d00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49924/0 (socket says 172.21.15.122:49924) 2024-03-06T07:22:41.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.686+0000 7ff7c5d9b640 1 -- 172.21.15.122:0/2547242701 learned_addr learned my addr 172.21.15.122:0/2547242701 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:41.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.686+0000 7ff7ced9d640 1 --2- 172.21.15.122:0/2547242701 >> v2:172.21.15.122:3300/0 conn(0x7ff7d00d2710 0x7ff7d00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.686+0000 7ff7ced9d640 1 -- 172.21.15.122:0/2547242701 >> v1:172.21.15.169:6789/0 conn(0x7ff7d015dfe0 legacy=0x7ff7d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.686+0000 7ff7ced9d640 1 -- 172.21.15.122:0/2547242701 >> v1:172.21.15.122:6789/0 conn(0x7ff7d00d22b0 legacy=0x7ff7d00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.686+0000 7ff7ced9d640 1 -- 172.21.15.122:0/2547242701 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00b39f0 con 0x7ff7d00d2710 2024-03-06T07:22:41.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7ced9d640 1 --2- 172.21.15.122:0/2547242701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b4900 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009dc0 tx=0x7ff7c802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40d281e4757fa32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7cdd9b640 1 -- 172.21.15.122:0/2547242701 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8039070 con 0x7ff7d00d2710 2024-03-06T07:22:41.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7cdd9b640 1 -- 172.21.15.122:0/2547242701 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c802ccd0 con 0x7ff7d00d2710 2024-03-06T07:22:41.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7cdd9b640 1 -- 172.21.15.122:0/2547242701 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8034660 con 0x7ff7d00d2710 2024-03-06T07:22:41.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2547242701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 msgr2=0x7ff7d00b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2547242701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b4900 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009dc0 tx=0x7ff7c802bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:41.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2547242701 shutdown_connections 2024-03-06T07:22:41.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2547242701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b4900 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2547242701 >> 172.21.15.122:0/2547242701 conn(0x7ff7d00d3cd0 msgr2=0x7ff7d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2547242701 shutdown_connections 2024-03-06T07:22:41.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.687+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2547242701 wait complete. 2024-03-06T07:22:41.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.689+0000 7ff7d4c9d640 1 Processor -- start 2024-03-06T07:22:41.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.689+0000 7ff7d4c9d640 1 -- start start 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.689+0000 7ff7d4c9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7d00d22b0 0x7ff7d00b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7d015dfe0 0x7ff7d00b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00cfa00 con 0x7ff7d00d2710 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00b3d60 con 0x7ff7d015dfe0 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00d1c70 con 0x7ff7d00d22b0 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56524/0 (socket says 172.21.15.122:56524) 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 -- 172.21.15.122:0/2715320753 learned_addr learned my addr 172.21.15.122:0/2715320753 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7ced9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7d015dfe0 0x7ff7d00b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7ce59c640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7d00d22b0 0x7ff7d00b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 -- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7d00d22b0 msgr2=0x7ff7d00b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7d00d22b0 0x7ff7d00b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 -- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7d015dfe0 msgr2=0x7ff7d00b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7d015dfe0 0x7ff7d00b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 -- 172.21.15.122:0/2715320753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00b17a0 con 0x7ff7d00d2710 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c5d9b640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b68d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800c4d0 tx=0x7ff7b800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8017070 con 0x7ff7d00d2710 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b8005ce0 con 0x7ff7d00d2710 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d00b1a80 con 0x7ff7d00d2710 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8007270 con 0x7ff7d00d2710 2024-03-06T07:22:41.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.690+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d00b2040 con 0x7ff7d00d2710 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.691+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d004f330 con 0x7ff7d00d2710 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.692+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7b801a070 con 0x7ff7d00d2710 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.692+0000 7ff7c77fe640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7b006b4d0 0x7ff7b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.692+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff7b80894c0 con 0x7ff7d00d2710 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.692+0000 7ff7ce59c640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7b006b4d0 0x7ff7b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.693+0000 7ff7ce59c640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7b006b4d0 0x7ff7b006d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000f60 tx=0x7ff7c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.696+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7b8057c70 con 0x7ff7d00d2710 2024-03-06T07:22:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.903+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7d015d340 con 0x7ff7d00d2710 2024-03-06T07:22:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.904+0000 7ff7c77fe640 1 -- 172.21.15.122:0/2715320753 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4983 (secure 0 0 0) 0x7ff7b805bd10 con 0x7ff7d00d2710 2024-03-06T07:22:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:41.906 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1151705641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1151705641},{"type":"v1","addr":"172.21.15.169:6835","nonce":1151705641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1541863767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1541863767},{"type":"v1","addr":"172.21.15.122:6835","nonce":1541863767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2081413677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2081413677},{"type":"v1","addr":"172.21.15.122:6837","nonce":2081413677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:52.887640+0000","modified":"2024-03-06T07:21:55.266867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2014546388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2014546388},{"type":"v1","addr":"172.21.15.169:6834","nonce":2014546388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:41.906 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 7 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.905+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7b006b4d0 msgr2=0x7ff7b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.905+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7b006b4d0 0x7ff7b006d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000f60 tx=0x7ff7c0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.905+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 msgr2=0x7ff7d00b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.905+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b68d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800c4d0 tx=0x7ff7b800c990 comp rx=0 tx=0).stop 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 shutdown_connections 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7b006b4d0 0x7ff7b006d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7d015dfe0 0x7ff7d00b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7d00d2710 0x7ff7d00b68d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 --2- 172.21.15.122:0/2715320753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7d00d22b0 0x7ff7d00b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 >> 172.21.15.122:0/2715320753 conn(0x7ff7d00d3cd0 msgr2=0x7ff7d00d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 shutdown_connections 2024-03-06T07:22:41.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:41.906+0000 7ff7d4c9d640 1 -- 172.21.15.122:0/2715320753 wait complete. 2024-03-06T07:22:41.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.020+0000 7efe4c7f7640 1 Processor -- start 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.020+0000 7efe4c7f7640 1 -- start start 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4c7f7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efe440b1080 0x7efe441674e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4c7f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4415c270 con 0x7efe440d0320 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4c7f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe440d0e90 con 0x7efe440d0700 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4c7f7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe440b04e0 con 0x7efe440b1080 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4a593640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efe440d0320 0x7efe440b4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49938/0 (socket says 172.21.15.122:49938) 2024-03-06T07:22:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4a593640 1 -- 172.21.15.122:0/1626358882 learned_addr learned my addr 172.21.15.122:0/1626358882 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4ad94640 1 --2- 172.21.15.122:0/1626358882 >> v2:172.21.15.169:3300/0 conn(0x7efe440b1080 0x7efe441674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe49591640 1 -- 172.21.15.122:0/1626358882 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713499453 0 0) 0x7efe4415c270 con 0x7efe440d0320 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe49591640 1 -- 172.21.15.122:0/1626358882 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30003520 con 0x7efe440d0320 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4ad94640 1 -- 172.21.15.122:0/1626358882 >> v1:172.21.15.169:6789/0 conn(0x7efe440d0700 legacy=0x7efe440b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4ad94640 1 -- 172.21.15.122:0/1626358882 >> v1:172.21.15.122:6789/0 conn(0x7efe440d0320 legacy=0x7efe440b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.021+0000 7efe4ad94640 1 -- 172.21.15.122:0/1626358882 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe440b5ae0 con 0x7efe440b1080 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.022+0000 7efe4ad94640 1 --2- 172.21.15.122:0/1626358882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440b1080 0x7efe441674e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009d90 tx=0x7efe3c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcf247d698ce9827 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.022+0000 7efe49591640 1 -- 172.21.15.122:0/1626358882 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c039070 con 0x7efe440b1080 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.022+0000 7efe49591640 1 -- 172.21.15.122:0/1626358882 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c02cd80 con 0x7efe440b1080 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.022+0000 7efe49591640 1 -- 172.21.15.122:0/1626358882 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c034640 con 0x7efe440b1080 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1626358882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440b1080 msgr2=0x7efe441674e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1626358882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440b1080 0x7efe441674e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009d90 tx=0x7efe3c02bfd0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1626358882 shutdown_connections 2024-03-06T07:22:42.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1626358882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440b1080 0x7efe441674e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1626358882 >> 172.21.15.122:0/1626358882 conn(0x7efe440724d0 msgr2=0x7efe440728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1626358882 shutdown_connections 2024-03-06T07:22:42.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.023+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1626358882 wait complete. 2024-03-06T07:22:42.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.024+0000 7efe4c7f7640 1 Processor -- start 2024-03-06T07:22:42.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.024+0000 7efe4c7f7640 1 -- start start 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 0x7efe44151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440d0320 0x7efe440c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe440d0700 0x7efe440c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe440b60a0 con 0x7efe440b1080 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe440b0170 con 0x7efe440d0320 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe440ca790 con 0x7efe440d0700 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 0x7efe44151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 0x7efe44151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56534/0 (socket says 172.21.15.122:56534) 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 -- 172.21.15.122:0/1083903921 learned_addr learned my addr 172.21.15.122:0/1083903921 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:42.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe49d92640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440d0320 0x7efe440c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4ad94640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe440d0700 0x7efe440c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 -- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe440d0700 msgr2=0x7efe440c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe440d0700 0x7efe440c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 -- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440d0320 msgr2=0x7efe440c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440d0320 0x7efe440c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 -- 172.21.15.122:0/1083903921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe440cee50 con 0x7efe440b1080 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4a593640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 0x7efe44151b40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efe4000c500 tx=0x7efe4000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe40017070 con 0x7efe440b1080 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe40005ce0 con 0x7efe440b1080 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe440cf130 con 0x7efe440b1080 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe40007330 con 0x7efe440b1080 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.025+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe440cf6f0 con 0x7efe440b1080 2024-03-06T07:22:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.026+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4404f330 con 0x7efe440b1080 2024-03-06T07:22:42.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.027+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe4001a070 con 0x7efe440b1080 2024-03-06T07:22:42.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.027+0000 7efe3affd640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2806b4d0 0x7efe2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.027+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efe40089600 con 0x7efe440b1080 2024-03-06T07:22:42.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.027+0000 7efe49d92640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2806b4d0 0x7efe2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.028+0000 7efe49d92640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2806b4d0 0x7efe2806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efe44155ea0 tx=0x7efe34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.032+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe40057f00 con 0x7efe440b1080 2024-03-06T07:22:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.206+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe4406d970 con 0x7efe440b1080 2024-03-06T07:22:42.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.207+0000 7efe3affd640 1 -- 172.21.15.122:0/1083903921 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16854 (secure 0 0 0) 0x7efe4005bfa0 con 0x7efe440b1080 2024-03-06T07:22:42.209 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:42.209 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":22,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:34.698183+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:51.355805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:51.854804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:42.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2806b4d0 msgr2=0x7efe2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2806b4d0 0x7efe2806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efe44155ea0 tx=0x7efe34009040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 msgr2=0x7efe44151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 0x7efe44151b40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efe4000c500 tx=0x7efe4000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 shutdown_connections 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2806b4d0 0x7efe2806d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe440d0700 0x7efe440c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe440d0320 0x7efe440c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 --2- 172.21.15.122:0/1083903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe440b1080 0x7efe44151b40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 >> 172.21.15.122:0/1083903921 conn(0x7efe440724d0 msgr2=0x7efe440b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 shutdown_connections 2024-03-06T07:22:42.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.209+0000 7efe4c7f7640 1 -- 172.21.15.122:0/1083903921 wait complete. 2024-03-06T07:22:42.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:42.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.320+0000 7f0986a9a640 1 Processor -- start 2024-03-06T07:22:42.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.320+0000 7f0986a9a640 1 -- start start 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f0986a9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b3240 0x7f09800b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f0986a9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098015e3f0 con 0x7f09800b2910 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f0986a9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09800b1c10 con 0x7f098015e010 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f0986a9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09800b1f40 con 0x7f09800b3240 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f0985037640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f09800b2910 0x7f09800d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49946/0 (socket says 172.21.15.122:49946) 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f0985037640 1 -- 172.21.15.122:0/1040190203 learned_addr learned my addr 172.21.15.122:0/1040190203 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f097ffff640 1 --2- 172.21.15.122:0/1040190203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b3240 0x7f09800b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.321+0000 7f097f7fe640 1 -- 172.21.15.122:0/1040190203 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963053711 0 0) 0x7f098015e3f0 con 0x7f09800b2910 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097f7fe640 1 -- 172.21.15.122:0/1040190203 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968003520 con 0x7f09800b2910 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097ffff640 1 -- 172.21.15.122:0/1040190203 >> v1:172.21.15.169:6789/0 conn(0x7f098015e010 legacy=0x7f09800cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097ffff640 1 -- 172.21.15.122:0/1040190203 >> v1:172.21.15.122:6789/0 conn(0x7f09800b2910 legacy=0x7f09800d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097f7fe640 1 -- 172.21.15.122:0/1040190203 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252858319 0 0) 0x7f09800b1c10 con 0x7f098015e010 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097ffff640 1 -- 172.21.15.122:0/1040190203 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098006d1b0 con 0x7f09800b3240 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097ffff640 1 --2- 172.21.15.122:0/1040190203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b3240 0x7f09800b23d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0974009800 tx=0x7f097402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=597ca2c02550ef89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.322+0000 7f097f7fe640 1 -- 172.21.15.122:0/1040190203 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974035020 con 0x7f09800b3240 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f097f7fe640 1 -- 172.21.15.122:0/1040190203 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0974031800 con 0x7f09800b3240 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f097f7fe640 1 -- 172.21.15.122:0/1040190203 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974031bb0 con 0x7f09800b3240 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 -- 172.21.15.122:0/1040190203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b3240 msgr2=0x7f09800b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 --2- 172.21.15.122:0/1040190203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b3240 0x7f09800b23d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0974009800 tx=0x7f097402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 -- 172.21.15.122:0/1040190203 shutdown_connections 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 --2- 172.21.15.122:0/1040190203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b3240 0x7f09800b23d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 -- 172.21.15.122:0/1040190203 >> 172.21.15.122:0/1040190203 conn(0x7f098009a9f0 msgr2=0x7f098009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 -- 172.21.15.122:0/1040190203 shutdown_connections 2024-03-06T07:22:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.323+0000 7f0986a9a640 1 -- 172.21.15.122:0/1040190203 wait complete. 2024-03-06T07:22:42.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.324+0000 7f0986a9a640 1 Processor -- start 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.324+0000 7f0986a9a640 1 -- start start 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0986a9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 0x7f098014c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0986a9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b5d40 0x7f09800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0986a9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 0x7f0980158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0986a9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09800d21e0 con 0x7f0980157db0 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0986a9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098006ced0 con 0x7f098015e010 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0986a9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09800d2360 con 0x7f09800b5d40 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0985037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 0x7f0980158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0984836640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 0x7f098014c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0984836640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 0x7f098014c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:50386/0 (socket says 172.21.15.122:50386) 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0985037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 0x7f0980158230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56546/0 (socket says 172.21.15.122:56546) 2024-03-06T07:22:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f0984836640 1 -- 172.21.15.122:0/2561292505 learned_addr learned my addr 172.21.15.122:0/2561292505 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.325+0000 7f097ffff640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b5d40 0x7f09800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0984836640 1 -- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b5d40 msgr2=0x7f09800b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0984836640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b5d40 0x7f09800b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0984836640 1 -- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 msgr2=0x7f0980158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0984836640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 0x7f0980158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0984836640 1 -- 172.21.15.122:0/2561292505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0980158ae0 con 0x7f098015e010 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0985037640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 0x7f0980158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0984836640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 0x7f098014c040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f097000c410 tx=0x7f097000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970017070 con 0x7f098015e010 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09700035f0 con 0x7f098015e010 2024-03-06T07:22:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0980158dc0 con 0x7f098015e010 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700039e0 con 0x7f098015e010 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.326+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09800c58c0 con 0x7f098015e010 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.327+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098004f330 con 0x7f098015e010 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.327+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f097001a070 con 0x7f098015e010 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.328+0000 7f097d7fa640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.328+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0970089650 con 0x7f098015e010 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.328+0000 7f097ffff640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.328+0000 7f097ffff640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f096c06b500 0x7f096c06d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0974036000 tx=0x7f097403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.332+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0970057d80 con 0x7f098015e010 2024-03-06T07:22:42.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.538+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f09800709f0 con 0x7f098015e010 2024-03-06T07:22:42.541 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:22:42.539+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:42.547 INFO:tasks.ceph.mds.d.smithi169.stderr: -3004> 2024-03-06T07:21:57.959+0000 7f571cae9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:42.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.546+0000 7f097d7fa640 1 -- 172.21.15.122:0/2561292505 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f097005be20 con 0x7f098015e010 2024-03-06T07:22:42.548 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f096c06b500 msgr2=0x7f096c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f096c06b500 0x7f096c06d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0974036000 tx=0x7f097403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 msgr2=0x7f098014c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 0x7f098014c040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f097000c410 tx=0x7f097000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 shutdown_connections 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0980157db0 0x7f0980158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09800b5d40 0x7f09800b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 --2- 172.21.15.122:0/2561292505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f098015e010 0x7f098014c040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 >> 172.21.15.122:0/2561292505 conn(0x7f098009a9f0 msgr2=0x7f098006ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 shutdown_connections 2024-03-06T07:22:42.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.548+0000 7f0986a9a640 1 -- 172.21.15.122:0/2561292505 wait complete. 2024-03-06T07:22:42.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:42.574 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:22:42.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 Processor -- start 2024-03-06T07:22:42.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 -- start start 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c01586e0 0x7f25c00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f25c00aef80 0x7f25c00cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00d03f0 con 0x7f25c00cffd0 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c0158ac0 con 0x7f25c00aef80 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.661+0000 7f25c6163640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c015b2a0 con 0x7f25c01586e0 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25bffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f25c00cffd0 0x7f25c006da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49952/0 (socket says 172.21.15.122:49952) 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25bffff640 1 -- 172.21.15.122:0/3362512877 learned_addr learned my addr 172.21.15.122:0/3362512877 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25beffd640 1 --2- 172.21.15.122:0/3362512877 >> v2:172.21.15.169:3300/0 conn(0x7f25c00aef80 0x7f25c00cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25bf7fe640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c01586e0 0x7f25c00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25be7fc640 1 -- 172.21.15.122:0/3362512877 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262515633 0 0) 0x7f25c00d03f0 con 0x7f25c00cffd0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25be7fc640 1 -- 172.21.15.122:0/3362512877 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8003520 con 0x7f25c00cffd0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25beffd640 1 -- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c01586e0 msgr2=0x7f25c00aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25be7fc640 1 -- 172.21.15.122:0/3362512877 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 304330790 0 0) 0x7f25a8003520 con 0x7f25c00cffd0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25beffd640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c01586e0 0x7f25c00aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25beffd640 1 -- 172.21.15.122:0/3362512877 >> v1:172.21.15.122:6789/0 conn(0x7f25c00cffd0 legacy=0x7f25c006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25beffd640 1 -- 172.21.15.122:0/3362512877 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c015b540 con 0x7f25c00aef80 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.662+0000 7f25bf7fe640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c01586e0 0x7f25c00aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25beffd640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00aef80 0x7f25c00cfa90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f25b40097a0 tx=0x7f25b402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bd74101d27e8771 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25be7fc640 1 -- 172.21.15.122:0/3362512877 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4037070 con 0x7f25c00aef80 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25be7fc640 1 -- 172.21.15.122:0/3362512877 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b4030510 con 0x7f25c00aef80 2024-03-06T07:22:42.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25be7fc640 1 -- 172.21.15.122:0/3362512877 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80019f0 con 0x7f25c00aef80 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25c6163640 1 -- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00aef80 msgr2=0x7f25c00cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25c6163640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00aef80 0x7f25c00cfa90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f25b40097a0 tx=0x7f25b402b790 comp rx=0 tx=0).stop 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25c6163640 1 -- 172.21.15.122:0/3362512877 shutdown_connections 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25c6163640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00aef80 0x7f25c00cfa90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25c6163640 1 --2- 172.21.15.122:0/3362512877 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c01586e0 0x7f25c00aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.663+0000 7f25c6163640 1 -- 172.21.15.122:0/3362512877 >> 172.21.15.122:0/3362512877 conn(0x7f25c006b190 msgr2=0x7f25c006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.664+0000 7f25c6163640 1 -- 172.21.15.122:0/3362512877 shutdown_connections 2024-03-06T07:22:42.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.664+0000 7f25c6163640 1 -- 172.21.15.122:0/3362512877 wait complete. 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 Processor -- start 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 -- start start 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c00aef80 0x7f25c014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00cffd0 0x7f25c00ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 0x7f25c006d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c015bde0 con 0x7f25c01586e0 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c015bb00 con 0x7f25c00cffd0 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.665+0000 7f25c6163640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c0071350 con 0x7f25c00aef80 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 0x7f25c006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 0x7f25c006d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56554/0 (socket says 172.21.15.122:56554) 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 -- 172.21.15.122:0/3189915773 learned_addr learned my addr 172.21.15.122:0/3189915773 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bf7fe640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c00aef80 0x7f25c014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25beffd640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00cffd0 0x7f25c00ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 -- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c00aef80 msgr2=0x7f25c014d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c00aef80 0x7f25c014d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 -- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00cffd0 msgr2=0x7f25c00ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00cffd0 0x7f25c00ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 -- 172.21.15.122:0/3189915773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c014d8c0 con 0x7f25c01586e0 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25bffff640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 0x7f25c006d960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f25b8001890 tx=0x7f25b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8012400 con 0x7f25c01586e0 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b80125c0 con 0x7f25c01586e0 2024-03-06T07:22:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00cd590 con 0x7f25c01586e0 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80129b0 con 0x7f25c01586e0 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.666+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c00c49f0 con 0x7f25c01586e0 2024-03-06T07:22:42.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.667+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c00ae920 con 0x7f25c01586e0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.668+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25b8012b50 con 0x7f25c01586e0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.668+0000 7f25a7fff640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25ac06b4d0 0x7f25ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.668+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f25b808aaf0 con 0x7f25c01586e0 2024-03-06T07:22:42.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.668+0000 7f25bf7fe640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25ac06b4d0 0x7f25ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.668+0000 7f25bf7fe640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25ac06b4d0 0x7f25ac06d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009a30 tx=0x7f25b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.672+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25c00ae920 con 0x7f25c01586e0 2024-03-06T07:22:42.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:42.884+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f25c00c47e0 con 0x7f25c01586e0 2024-03-06T07:22:43.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.541+0000 7f25a7fff640 1 -- 172.21.15.122:0/3189915773 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f25c014d8c0 con 0x7f25c01586e0 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.544+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25ac06b4d0 msgr2=0x7f25ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.544+0000 7f25c6163640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25ac06b4d0 0x7f25ac06d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009a30 tx=0x7f25b0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 msgr2=0x7f25c006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 0x7f25c006d960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f25b8001890 tx=0x7f25b800a430 comp rx=0 tx=0).stop 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 shutdown_connections 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25ac06b4d0 0x7f25ac06d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25c01586e0 0x7f25c006d960 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25c00cffd0 0x7f25c00ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 --2- 172.21.15.122:0/3189915773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25c00aef80 0x7f25c014d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 >> 172.21.15.122:0/3189915773 conn(0x7f25c006b190 msgr2=0x7f25c0156e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 shutdown_connections 2024-03-06T07:22:43.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.545+0000 7f25c6163640 1 -- 172.21.15.122:0/3189915773 wait complete. 2024-03-06T07:22:43.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:43.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.661+0000 7fa6c06c2640 1 Processor -- start 2024-03-06T07:22:43.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.661+0000 7fa6c06c2640 1 -- start start 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.662+0000 7fa6c06c2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa6b815dfa0 0x7fa6b80d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.662+0000 7fa6c06c2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d4630 0x7fa6b80d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.662+0000 7fa6c06c2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6b80d2820 0x7fa6b80d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.662+0000 7fa6c06c2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b80d2100 con 0x7fa6b80d2820 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.662+0000 7fa6c06c2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b815e380 con 0x7fa6b815dfa0 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.662+0000 7fa6c06c2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80d1e30 con 0x7fa6b80d4630 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6b80d2820 0x7fa6b80d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6b80d2820 0x7fa6b80d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56560/0 (socket says 172.21.15.122:56560) 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 -- 172.21.15.122:0/476287570 learned_addr learned my addr 172.21.15.122:0/476287570 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6be45e640 1 --2- 172.21.15.122:0/476287570 >> v2:172.21.15.169:3300/0 conn(0x7fa6b815dfa0 0x7fa6b80d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bdc5d640 1 --2- 172.21.15.122:0/476287570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d4630 0x7fa6b80d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 -- 172.21.15.122:0/476287570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d4630 msgr2=0x7fa6b80d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 --2- 172.21.15.122:0/476287570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d4630 0x7fa6b80d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 -- 172.21.15.122:0/476287570 >> v2:172.21.15.169:3300/0 conn(0x7fa6b815dfa0 msgr2=0x7fa6b80d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 --2- 172.21.15.122:0/476287570 >> v2:172.21.15.169:3300/0 conn(0x7fa6b815dfa0 0x7fa6b80d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.663+0000 7fa6bec5f640 1 -- 172.21.15.122:0/476287570 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80cf470 con 0x7fa6b80d2820 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6bec5f640 1 --2- 172.21.15.122:0/476287570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b80d2820 0x7fa6b80d05f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6b808d560 tx=0x7fa6b002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179547c034e2302 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6bd45c640 1 -- 172.21.15.122:0/476287570 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b002c9e0 con 0x7fa6b80d2820 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6bd45c640 1 -- 172.21.15.122:0/476287570 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b002cb80 con 0x7fa6b80d2820 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6bd45c640 1 -- 172.21.15.122:0/476287570 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b00343f0 con 0x7fa6b80d2820 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 -- 172.21.15.122:0/476287570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b80d2820 msgr2=0x7fa6b80d05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/476287570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b80d2820 0x7fa6b80d05f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6b808d560 tx=0x7fa6b002be40 comp rx=0 tx=0).stop 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 -- 172.21.15.122:0/476287570 shutdown_connections 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/476287570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b80d2820 0x7fa6b80d05f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/476287570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d4630 0x7fa6b80d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/476287570 >> v2:172.21.15.169:3300/0 conn(0x7fa6b815dfa0 0x7fa6b80d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 -- 172.21.15.122:0/476287570 >> 172.21.15.122:0/476287570 conn(0x7fa6b809d000 msgr2=0x7fa6b80d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 -- 172.21.15.122:0/476287570 shutdown_connections 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.664+0000 7fa6c06c2640 1 -- 172.21.15.122:0/476287570 wait complete. 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.665+0000 7fa6c06c2640 1 Processor -- start 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 -- start start 2024-03-06T07:22:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d2820 0x7fa6b80cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b80d4630 0x7fa6b81f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 0x7fa6b81f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80d0b30 con 0x7fa6b815dfa0 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80cfa30 con 0x7fa6b80d4630 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6c06c2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80d0cb0 con 0x7fa6b80d2820 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6bec5f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 0x7fa6b81f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6bec5f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 0x7fa6b81f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56574/0 (socket says 172.21.15.122:56574) 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6bec5f640 1 -- 172.21.15.122:0/3902236982 learned_addr learned my addr 172.21.15.122:0/3902236982 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6be45e640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d2820 0x7fa6b80cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.666+0000 7fa6bdc5d640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b80d4630 0x7fa6b81f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bec5f640 1 -- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d2820 msgr2=0x7fa6b80cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bec5f640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d2820 0x7fa6b80cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bec5f640 1 -- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b80d4630 msgr2=0x7fa6b81f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bec5f640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b80d4630 0x7fa6b81f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bec5f640 1 -- 172.21.15.122:0/3902236982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80b7d80 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bdc5d640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b80d4630 0x7fa6b81f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6bec5f640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 0x7fa6b81f74b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009f40 tx=0x7fa6b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0043020 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b003e450 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b80b8060 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0009510 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.667+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b81f4f50 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.668+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b80cabf0 con 0x7fa6b815dfa0 2024-03-06T07:22:43.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.668+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6b0046020 con 0x7fa6b815dfa0 2024-03-06T07:22:43.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.669+0000 7fa6aeffd640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa69c06b4d0 0x7fa69c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.669+0000 7fa6be45e640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa69c06b4d0 0x7fa69c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.669+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fa6b00acdf0 con 0x7fa6b815dfa0 2024-03-06T07:22:43.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.669+0000 7fa6be45e640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa69c06b4d0 0x7fa69c06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40079a0 tx=0x7fa6b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.673+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6b80cabf0 con 0x7fa6b815dfa0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:43.849+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa6b81f5150 con 0x7fa6b815dfa0 2024-03-06T07:22:44.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.740+0000 7fa6aeffd640 1 -- 172.21.15.122:0/3902236982 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa6b81f5150 con 0x7fa6b815dfa0 2024-03-06T07:22:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa69c06b4d0 msgr2=0x7fa69c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa69c06b4d0 0x7fa69c06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40079a0 tx=0x7fa6b4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 msgr2=0x7fa6b81f74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 0x7fa6b81f74b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009f40 tx=0x7fa6b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 shutdown_connections 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa69c06b4d0 0x7fa69c06d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b815dfa0 0x7fa6b81f74b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b80d4630 0x7fa6b81f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 --2- 172.21.15.122:0/3902236982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b80d2820 0x7fa6b80cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 >> 172.21.15.122:0/3902236982 conn(0x7fa6b809d000 msgr2=0x7fa6b80cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 shutdown_connections 2024-03-06T07:22:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.744+0000 7fa6c06c2640 1 -- 172.21.15.122:0/3902236982 wait complete. 2024-03-06T07:22:44.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:44.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 Processor -- start 2024-03-06T07:22:44.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 -- start start 2024-03-06T07:22:44.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 0x7fc84c06f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c06c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c0cf3c0 con 0x7fc84c162850 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84c06f270 con 0x7fc84c162470 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.867+0000 7fc852d93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c0cf0e0 con 0x7fc84c0d2160 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc850b2f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 0x7fc84c06f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc851330640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c06c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc850b2f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 0x7fc84c06f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:60382/0 (socket says 172.21.15.122:60382) 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc851330640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c06c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:37868/0 (socket says 172.21.15.122:37868) 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc84bfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc84c162850 0x7fc84c0d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:59850/0 (socket says 172.21.15.122:59850) 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc850b2f640 1 -- 172.21.15.122:0/2672936533 learned_addr learned my addr 172.21.15.122:0/2672936533 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc851330640 1 -- 172.21.15.122:0/2672936533 >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 msgr2=0x7fc84c06f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc851330640 1 --2- 172.21.15.122:0/2672936533 >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 0x7fc84c06f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc851330640 1 -- 172.21.15.122:0/2672936533 >> v1:172.21.15.169:6789/0 conn(0x7fc84c162850 legacy=0x7fc84c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc851330640 1 -- 172.21.15.122:0/2672936533 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c06ec20 con 0x7fc84c0d2160 2024-03-06T07:22:44.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.868+0000 7fc850b2f640 1 --2- 172.21.15.122:0/2672936533 >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 0x7fc84c06f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.869+0000 7fc851330640 1 --2- 172.21.15.122:0/2672936533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c06c3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc844009d90 tx=0x7fc84402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b559f15175b107 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.869+0000 7fc84b7fe640 1 -- 172.21.15.122:0/2672936533 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844039070 con 0x7fc84c0d2160 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.869+0000 7fc84b7fe640 1 -- 172.21.15.122:0/2672936533 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84402cc50 con 0x7fc84c0d2160 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.869+0000 7fc84b7fe640 1 -- 172.21.15.122:0/2672936533 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844034440 con 0x7fc84c0d2160 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 -- 172.21.15.122:0/2672936533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 msgr2=0x7fc84c06c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 --2- 172.21.15.122:0/2672936533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c06c3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc844009d90 tx=0x7fc84402bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 -- 172.21.15.122:0/2672936533 shutdown_connections 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 --2- 172.21.15.122:0/2672936533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c06c3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 --2- 172.21.15.122:0/2672936533 >> v2:172.21.15.169:3300/0 conn(0x7fc84c162470 0x7fc84c06f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 -- 172.21.15.122:0/2672936533 >> 172.21.15.122:0/2672936533 conn(0x7fc84c09a9f0 msgr2=0x7fc84c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 -- 172.21.15.122:0/2672936533 shutdown_connections 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.870+0000 7fc852d93640 1 -- 172.21.15.122:0/2672936533 wait complete. 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.871+0000 7fc852d93640 1 Processor -- start 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.871+0000 7fc852d93640 1 -- start start 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc852d93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c0afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc852d93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 0x7fc84c15ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc852d93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc84c162850 0x7fc84c15f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc852d93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c06fdc0 con 0x7fc84c162470 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc852d93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c06fc40 con 0x7fc84c162850 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc852d93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c06ff40 con 0x7fc84c0d2160 2024-03-06T07:22:44.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 0x7fc84c15ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 0x7fc84c15ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40992/0 (socket says 172.21.15.122:40992) 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 -- 172.21.15.122:0/4059302951 learned_addr learned my addr 172.21.15.122:0/4059302951 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc850b2f640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c0afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc851330640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc84c162850 0x7fc84c15f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 -- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 msgr2=0x7fc84c0afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c0afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 -- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc84c162850 msgr2=0x7fc84c15f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc84c162850 0x7fc84c15f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 -- 172.21.15.122:0/4059302951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c1f3af0 con 0x7fc84c162470 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.872+0000 7fc84bfff640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 0x7fc84c15ed20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c3c0 tx=0x7fc84000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.873+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8400074e0 con 0x7fc84c162470 2024-03-06T07:22:44.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.873+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc840007680 con 0x7fc84c162470 2024-03-06T07:22:44.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.873+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc840004a00 con 0x7fc84c162470 2024-03-06T07:22:44.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.873+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c1f3dd0 con 0x7fc84c162470 2024-03-06T07:22:44.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.873+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c1f4390 con 0x7fc84c162470 2024-03-06T07:22:44.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.874+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c04f330 con 0x7fc84c162470 2024-03-06T07:22:44.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.874+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc84001a020 con 0x7fc84c162470 2024-03-06T07:22:44.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.874+0000 7fc8497fa640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc83806b4d0 0x7fc83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.875+0000 7fc850b2f640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc83806b4d0 0x7fc83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.875+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fc840089820 con 0x7fc84c162470 2024-03-06T07:22:44.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.875+0000 7fc850b2f640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc83806b4d0 0x7fc83806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc83c006ca0 tx=0x7fc83c009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:44.879+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc840058020 con 0x7fc84c162470 2024-03-06T07:22:45.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.057+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc84c06c190 con 0x7fc84c162470 2024-03-06T07:22:45.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.747+0000 7fc8497fa640 1 -- 172.21.15.122:0/4059302951 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc84005c0c0 con 0x7fc84c162470 2024-03-06T07:22:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc83806b4d0 msgr2=0x7fc83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc83806b4d0 0x7fc83806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc83c006ca0 tx=0x7fc83c009240 comp rx=0 tx=0).stop 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 msgr2=0x7fc84c15ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 0x7fc84c15ed20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c3c0 tx=0x7fc84000c880 comp rx=0 tx=0).stop 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 shutdown_connections 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc83806b4d0 0x7fc83806d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc84c162850 0x7fc84c15f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc84c162470 0x7fc84c15ed20 secure :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c3c0 tx=0x7fc84000c880 comp rx=0 tx=0).stop 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 --2- 172.21.15.122:0/4059302951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc84c0d2160 0x7fc84c0afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 >> 172.21.15.122:0/4059302951 conn(0x7fc84c09a9f0 msgr2=0x7fc84c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 shutdown_connections 2024-03-06T07:22:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.751+0000 7fc852d93640 1 -- 172.21.15.122:0/4059302951 wait complete. 2024-03-06T07:22:45.767 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:45.767 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:45.768 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:45.768 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:45.768 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:45.768 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:45.768 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:45.769 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:45.769 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:45.769 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:45.769 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:45.769 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:45.778 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:45.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:45.779 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:45.779 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:45.780 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:45.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:45.781 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:45.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:45.820 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:45.820 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:45.821 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:45.821 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:45.823 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:45.823 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:45.824 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:45.824 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:45.824 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:45.898 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:22:45.899 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:22:45.908 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:22:45.908 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:22:45.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.932+0000 7feaaf4ed640 1 Processor -- start 2024-03-06T07:22:45.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.932+0000 7feaaf4ed640 1 -- start start 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaaf4ed640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7feaa8162970 0x7feaa80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaaf4ed640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa80d22c0 con 0x7feaa80b4460 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaaf4ed640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa80d25a0 con 0x7feaa8161cf0 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaaf4ed640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa8161b00 con 0x7feaa8162970 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaad289640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7feaa80b4460 0x7feaa80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59526/0 (socket says 172.21.15.122:59526) 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaad289640 1 -- 172.21.15.122:0/3079117603 learned_addr learned my addr 172.21.15.122:0/3079117603 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302833557 0 0) 0x7feaa80d22c0 con 0x7feaa80b4460 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94003520 con 0x7feaa80b4460 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7feaaca88640 1 --2- 172.21.15.122:0/3079117603 >> v2:172.21.15.169:3300/0 conn(0x7feaa8162970 0x7feaa80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885536310 0 0) 0x7fea94003520 con 0x7feaa80b4460 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 >> v2:172.21.15.169:3300/0 conn(0x7feaa8162970 msgr2=0x7feaa80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 --2- 172.21.15.122:0/3079117603 >> v2:172.21.15.169:3300/0 conn(0x7feaa8162970 0x7feaa80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 >> v1:172.21.15.169:6789/0 conn(0x7feaa8161cf0 legacy=0x7feaa80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80d2880 con 0x7feaa80b4460 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7feaa4001670 con 0x7feaa80b4460 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 >> v1:172.21.15.122:6789/0 conn(0x7feaa80b4460 legacy=0x7feaa80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea940039b0 0x7fea94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7fea94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.933+0000 7fea9ffff640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea940051c0 0x7fea9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa8161b00 con 0x7fea940039b0 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9400c060 con 0x7fea940051c0 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9400c260 con 0x7fea94004800 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaaca88640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7fea94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaad289640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea940039b0 0x7fea94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaada8a640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea940051c0 0x7fea9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaaca88640 1 -- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea940051c0 msgr2=0x7fea9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaaca88640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea940051c0 0x7fea9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaaca88640 1 -- 172.21.15.122:0/3079117603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea940039b0 msgr2=0x7fea94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaaca88640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea940039b0 0x7fea94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaaca88640 1 -- 172.21.15.122:0/3079117603 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea94003520 con 0x7fea94004800 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaada8a640 1 -- 172.21.15.122:0/3079117603 reap_dead start 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.934+0000 7feaad289640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea940039b0 0x7fea94003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaca88640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7fea94004c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea98009a90 tx=0x7fea94005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef4a0a500ed3ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98039070 con 0x7fea94004800 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9802c6b0 con 0x7fea94004800 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7fea9ffff640 1 -- 172.21.15.122:0/3079117603 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98009690 con 0x7fea94004800 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 -- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 msgr2=0x7fea94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7fea94004c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea98009a90 tx=0x7fea94005830 comp rx=0 tx=0).stop 2024-03-06T07:22:45.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 -- 172.21.15.122:0/3079117603 shutdown_connections 2024-03-06T07:22:45.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/3079117603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7fea94004c80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 -- 172.21.15.122:0/3079117603 >> 172.21.15.122:0/3079117603 conn(0x7feaa80d3ce0 msgr2=0x7feaa80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 -- 172.21.15.122:0/3079117603 shutdown_connections 2024-03-06T07:22:45.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.935+0000 7feaaf4ed640 1 -- 172.21.15.122:0/3079117603 wait complete. 2024-03-06T07:22:45.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 Processor -- start 2024-03-06T07:22:45.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 -- start start 2024-03-06T07:22:45.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7feaa81f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 0x7feaa80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feaa80cf690 0x7feaa80b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa80ce950 con 0x7feaa81557e0 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa80cecc0 con 0x7feaa80cf690 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaf4ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa80b4840 con 0x7fea94004800 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaca88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 0x7feaa80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaca88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 0x7feaa80cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41034/0 (socket says 172.21.15.122:41034) 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.937+0000 7feaaca88640 1 -- 172.21.15.122:0/2581224136 learned_addr learned my addr 172.21.15.122:0/2581224136 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaada8a640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feaa80cf690 0x7feaa80b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaad289640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7feaa81f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaca88640 1 -- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 msgr2=0x7feaa81f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaca88640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7feaa81f81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaca88640 1 -- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feaa80cf690 msgr2=0x7feaa80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaca88640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feaa80cf690 0x7feaa80b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaca88640 1 -- 172.21.15.122:0/2581224136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa81f8710 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaca88640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 0x7feaa80cf150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fea98009a90 tx=0x7fea98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98039050 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa815c070 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea98045070 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa80c7940 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.938+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea980044e0 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.939+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa804f330 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.940+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea98005b90 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.940+0000 7fea9dffb640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea8c06b4d0 0x7fea8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.940+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fea980aba00 con 0x7feaa81557e0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.940+0000 7feaad289640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea8c06b4d0 0x7fea8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.941+0000 7feaad289640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea8c06b4d0 0x7fea8c06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feaa4007ce0 tx=0x7feaa4007c70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:45.944+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea9804e020 con 0x7feaa81557e0 2024-03-06T07:22:46.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.129+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7feaa81566a0 con 0x7feaa81557e0 2024-03-06T07:22:46.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.761+0000 7fea9dffb640 1 -- 172.21.15.122:0/2581224136 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7feaa81f8710 con 0x7feaa81557e0 2024-03-06T07:22:46.763 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:22:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea8c06b4d0 msgr2=0x7fea8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea8c06b4d0 0x7fea8c06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feaa4007ce0 tx=0x7feaa4007c70 comp rx=0 tx=0).stop 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 msgr2=0x7feaa80cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 0x7feaa80cf150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fea98009a90 tx=0x7fea98004730 comp rx=0 tx=0).stop 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 shutdown_connections 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea8c06b4d0 0x7fea8c06d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.763+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feaa80cf690 0x7feaa80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.764+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaa81557e0 0x7feaa80cf150 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.764+0000 7feaaf4ed640 1 --2- 172.21.15.122:0/2581224136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea94004800 0x7feaa81f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.764+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 >> 172.21.15.122:0/2581224136 conn(0x7feaa80d3ce0 msgr2=0x7feaa806cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.764+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 shutdown_connections 2024-03-06T07:22:46.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.764+0000 7feaaf4ed640 1 -- 172.21.15.122:0/2581224136 wait complete. 2024-03-06T07:22:46.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:46.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.877+0000 7f5c774c3640 1 Processor -- start 2024-03-06T07:22:46.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.878+0000 7f5c774c3640 1 -- start start 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.878+0000 7f5c774c3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c700b3060 0x7f5c7015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.878+0000 7f5c774c3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7015bf30 0x7f5c7015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.878+0000 7f5c774c3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7015f2d0 con 0x7f5c7013dc10 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.878+0000 7f5c774c3640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c700cf3f0 con 0x7f5c700b3060 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.878+0000 7f5c774c3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7015b210 con 0x7f5c7015bf30 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c700b3060 0x7f5c7015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c700b3060 0x7f5c7015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41044/0 (socket says 172.21.15.122:41044) 2024-03-06T07:22:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 -- 172.21.15.122:0/993307830 learned_addr learned my addr 172.21.15.122:0/993307830 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:46.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c75a60640 1 --2- 172.21.15.122:0/993307830 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7015bf30 0x7f5c7015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 -- 172.21.15.122:0/993307830 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7015bf30 msgr2=0x7f5c7015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 --2- 172.21.15.122:0/993307830 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7015bf30 0x7f5c7015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 -- 172.21.15.122:0/993307830 >> v1:172.21.15.169:6789/0 conn(0x7f5c7013dc10 legacy=0x7f5c700b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.879+0000 7f5c74a5e640 1 -- 172.21.15.122:0/993307830 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7015f580 con 0x7f5c700b3060 2024-03-06T07:22:46.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c74a5e640 1 --2- 172.21.15.122:0/993307830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c700b3060 0x7f5c7015e850 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009870 tx=0x7f5c6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b937c259161277 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c67fff640 1 -- 172.21.15.122:0/993307830 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c035020 con 0x7f5c700b3060 2024-03-06T07:22:46.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c67fff640 1 -- 172.21.15.122:0/993307830 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c031640 con 0x7f5c700b3060 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c67fff640 1 -- 172.21.15.122:0/993307830 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c02d040 con 0x7f5c700b3060 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 -- 172.21.15.122:0/993307830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c700b3060 msgr2=0x7f5c7015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 --2- 172.21.15.122:0/993307830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c700b3060 0x7f5c7015e850 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009870 tx=0x7f5c6c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 -- 172.21.15.122:0/993307830 shutdown_connections 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 --2- 172.21.15.122:0/993307830 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7015bf30 0x7f5c7015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 --2- 172.21.15.122:0/993307830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c700b3060 0x7f5c7015e850 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 -- 172.21.15.122:0/993307830 >> 172.21.15.122:0/993307830 conn(0x7f5c700cf8a0 msgr2=0x7f5c700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.880+0000 7f5c774c3640 1 -- 172.21.15.122:0/993307830 shutdown_connections 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.881+0000 7f5c774c3640 1 -- 172.21.15.122:0/993307830 wait complete. 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 Processor -- start 2024-03-06T07:22:46.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 -- start start 2024-03-06T07:22:46.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c700b3060 0x7f5c701543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7013dc10 0x7f5c7012bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 0x7f5c70140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7013dff0 con 0x7f5c7015bf30 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7015b730 con 0x7f5c7013dc10 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c774c3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700708d0 con 0x7f5c700b3060 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c75a60640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 0x7f5c70140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c75a60640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 0x7f5c70140380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41046/0 (socket says 172.21.15.122:41046) 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c75a60640 1 -- 172.21.15.122:0/346398244 learned_addr learned my addr 172.21.15.122:0/346398244 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c7525f640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c700b3060 0x7f5c701543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.882+0000 7f5c74a5e640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7013dc10 0x7f5c7012bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c75a60640 1 -- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c700b3060 msgr2=0x7f5c701543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c75a60640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c700b3060 0x7f5c701543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c75a60640 1 -- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7013dc10 msgr2=0x7f5c7012bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c75a60640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7013dc10 0x7f5c7012bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c75a60640 1 -- 172.21.15.122:0/346398244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70154930 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c75a60640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 0x7f5c70140380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5c68007e60 tx=0x7f5c6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68014420 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68019070 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68014820 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7012c600 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.883+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7012cbc0 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.884+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7004f330 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.884+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c680149c0 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.885+0000 7f5c65ffb640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c5406b4d0 0x7f5c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.885+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5c6808a5d0 con 0x7f5c7015bf30 2024-03-06T07:22:46.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.885+0000 7f5c7525f640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c5406b4d0 0x7f5c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.885+0000 7f5c7525f640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c5406b4d0 0x7f5c5406d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c60001780 tx=0x7f5c60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:46.889+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c68059150 con 0x7f5c7015bf30 2024-03-06T07:22:47.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.123+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5c7015e390 con 0x7f5c7015bf30 2024-03-06T07:22:47.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.123+0000 7f5c65ffb640 1 -- 172.21.15.122:0/346398244 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f5c6805d1f0 con 0x7f5c7015bf30 2024-03-06T07:22:47.125 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:47.125 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:22:47.125 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:22:47.126 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.127 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:22:47.129 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.131 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:22:47.133 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:22:47.134 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:22:47.135 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:22:47.136 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.137 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQB5GehlTzpSChAAS7Z6d2zNev+PjrYwwTUJYg==", 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQB5GehlLkRSChAA3cDXraXLXHQYPTEXFahJZQ==", 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQB5GehlCktSChAA9phPKxdBT9DAG5E4waeYww==", 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQB5GehlZ1FSChAAqmpQxLdPWxFpXvjsieJTYw==", 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQB5GehlElhSChAAjCgk9osKdGN5Odo3yPqN8A==", 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQB5GehlG19SChAAQ4XOcj2Oc2Qo5RmKizjR5Q==", 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c5406b4d0 msgr2=0x7f5c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c5406b4d0 0x7f5c5406d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c60001780 tx=0x7f5c60009040 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 msgr2=0x7f5c70140380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 0x7f5c70140380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5c68007e60 tx=0x7f5c6800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 shutdown_connections 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c5406b4d0 0x7f5c5406d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7015bf30 0x7f5c70140380 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7013dc10 0x7f5c7012bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 --2- 172.21.15.122:0/346398244 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c700b3060 0x7f5c701543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 >> 172.21.15.122:0/346398244 conn(0x7f5c700cf8a0 msgr2=0x7f5c700b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 shutdown_connections 2024-03-06T07:22:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.125+0000 7f5c774c3640 1 -- 172.21.15.122:0/346398244 wait complete. 2024-03-06T07:22:47.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:22:47.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.243+0000 7fc2eb6f4640 1 Processor -- start 2024-03-06T07:22:47.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.243+0000 7fc2eb6f4640 1 -- start start 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2eb6f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e41456f0 0x7fc2e4070940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2eb6f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc2e415c2c0 0x7fc2e40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2eb6f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40b3680 con 0x7fc2e4070e80 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2eb6f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e415c760 con 0x7fc2e415c2c0 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2eb6f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e415ca40 con 0x7fc2e41456f0 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2e9c91640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc2e4070e80 0x7fc2e415f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59532/0 (socket says 172.21.15.122:59532) 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2e9c91640 1 -- 172.21.15.122:0/917023351 learned_addr learned my addr 172.21.15.122:0/917023351 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2e8c8f640 1 --2- 172.21.15.122:0/917023351 >> v2:172.21.15.169:3300/0 conn(0x7fc2e415c2c0 0x7fc2e40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.244+0000 7fc2e9490640 1 --2- 172.21.15.122:0/917023351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e41456f0 0x7fc2e4070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2dbfff640 1 -- 172.21.15.122:0/917023351 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048499109 0 0) 0x7fc2dc001490 con 0x7fc2e4070e80 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2dbfff640 1 -- 172.21.15.122:0/917023351 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d0003520 con 0x7fc2e4070e80 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2e9490640 1 -- 172.21.15.122:0/917023351 >> v2:172.21.15.169:3300/0 conn(0x7fc2e415c2c0 msgr2=0x7fc2e40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2e9490640 1 --2- 172.21.15.122:0/917023351 >> v2:172.21.15.169:3300/0 conn(0x7fc2e415c2c0 0x7fc2e40cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2dbfff640 1 -- 172.21.15.122:0/917023351 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021845908 0 0) 0x7fc2d0003520 con 0x7fc2e4070e80 2024-03-06T07:22:47.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2e9490640 1 -- 172.21.15.122:0/917023351 >> v1:172.21.15.122:6789/0 conn(0x7fc2e4070e80 legacy=0x7fc2e415f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2e9490640 1 -- 172.21.15.122:0/917023351 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e415cd20 con 0x7fc2e41456f0 2024-03-06T07:22:47.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.245+0000 7fc2e8c8f640 1 --2- 172.21.15.122:0/917023351 >> v2:172.21.15.169:3300/0 conn(0x7fc2e415c2c0 0x7fc2e40cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:47.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2e9490640 1 --2- 172.21.15.122:0/917023351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e41456f0 0x7fc2e4070940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00097a0 tx=0x7fc2e002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348fdb0c4ead7054 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2dbfff640 1 -- 172.21.15.122:0/917023351 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0037070 con 0x7fc2e41456f0 2024-03-06T07:22:47.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2dbfff640 1 -- 172.21.15.122:0/917023351 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e0030510 con 0x7fc2e41456f0 2024-03-06T07:22:47.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2dbfff640 1 -- 172.21.15.122:0/917023351 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc001bc0 con 0x7fc2e41456f0 2024-03-06T07:22:47.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/917023351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e41456f0 msgr2=0x7fc2e4070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/917023351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e41456f0 0x7fc2e4070940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00097a0 tx=0x7fc2e002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/917023351 shutdown_connections 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/917023351 >> v2:172.21.15.169:3300/0 conn(0x7fc2e415c2c0 0x7fc2e40cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/917023351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e41456f0 0x7fc2e4070940 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/917023351 >> 172.21.15.122:0/917023351 conn(0x7fc2e40b1c80 msgr2=0x7fc2e40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/917023351 shutdown_connections 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.246+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/917023351 wait complete. 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.247+0000 7fc2eb6f4640 1 Processor -- start 2024-03-06T07:22:47.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.247+0000 7fc2eb6f4640 1 -- start start 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2eb6f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 0x7fc2e41f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2eb6f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e41456f0 0x7fc2e4138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2eb6f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e415c2c0 0x7fc2e415f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2eb6f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e415d980 con 0x7fc2e4070e80 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2eb6f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e415d800 con 0x7fc2e41456f0 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2eb6f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40cc830 con 0x7fc2e415c2c0 2024-03-06T07:22:47.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2e9490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 0x7fc2e41f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2e9490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 0x7fc2e41f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41060/0 (socket says 172.21.15.122:41060) 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2e9490640 1 -- 172.21.15.122:0/1749549429 learned_addr learned my addr 172.21.15.122:0/1749549429 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2e8c8f640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e41456f0 0x7fc2e4138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.248+0000 7fc2e9c91640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e415c2c0 0x7fc2e415f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2e9490640 1 -- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e415c2c0 msgr2=0x7fc2e415f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2e9490640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e415c2c0 0x7fc2e415f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2e9490640 1 -- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e41456f0 msgr2=0x7fc2e4138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2e9490640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e41456f0 0x7fc2e4138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2e9490640 1 -- 172.21.15.122:0/1749549429 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e413b3c0 con 0x7fc2e4070e80 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2e9490640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 0x7fc2e41f3170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0000c00 tx=0x7fc2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0040020 con 0x7fc2e4070e80 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e0038040 con 0x7fc2e4070e80 2024-03-06T07:22:47.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0037070 con 0x7fc2e4070e80 2024-03-06T07:22:47.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e413a840 con 0x7fc2e4070e80 2024-03-06T07:22:47.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.249+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e413ae00 con 0x7fc2e4070e80 2024-03-06T07:22:47.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.250+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4140eb0 con 0x7fc2e4070e80 2024-03-06T07:22:47.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.250+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2e0043020 con 0x7fc2e4070e80 2024-03-06T07:22:47.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.251+0000 7fc2d9ffb640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2c806b500 0x7fc2c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.251+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc2e00ab140 con 0x7fc2e4070e80 2024-03-06T07:22:47.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.251+0000 7fc2e8c8f640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2c806b500 0x7fc2c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.251+0000 7fc2e8c8f640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2c806b500 0x7fc2c806d9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2e415fe80 tx=0x7fc2d4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.255+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2e4140eb0 con 0x7fc2e4070e80 2024-03-06T07:22:47.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.499+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fc2e404f330 con 0x7fc2e4070e80 2024-03-06T07:22:47.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.501+0000 7fc2d9ffb640 1 -- 172.21.15.122:0/1749549429 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc2e004a030 con 0x7fc2e4070e80 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2c806b500 msgr2=0x7fc2c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2c806b500 0x7fc2c806d9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2e415fe80 tx=0x7fc2d4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 msgr2=0x7fc2e41f3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 0x7fc2e41f3170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0000c00 tx=0x7fc2e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 shutdown_connections 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2c806b500 0x7fc2c806d9c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2e415c2c0 0x7fc2e415f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e41456f0 0x7fc2e4138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 --2- 172.21.15.122:0/1749549429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2e4070e80 0x7fc2e41f3170 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 >> 172.21.15.122:0/1749549429 conn(0x7fc2e40b1c80 msgr2=0x7fc2e40cf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 shutdown_connections 2024-03-06T07:22:47.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.504+0000 7fc2eb6f4640 1 -- 172.21.15.122:0/1749549429 wait complete. 2024-03-06T07:22:47.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:22:47.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.614+0000 7f18289bc640 1 Processor -- start 2024-03-06T07:22:47.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.614+0000 7f18289bc640 1 -- start start 2024-03-06T07:22:47.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.615+0000 7f18289bc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 0x7f182415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.615+0000 7f18289bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 0x7f1824072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.615+0000 7f18289bc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182406e770 con 0x7f18240b4a50 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.615+0000 7f18289bc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240d0700 con 0x7f18240cc920 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.615+0000 7f18289bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182415e660 con 0x7f182406e000 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 0x7f1824072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f182259c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 0x7f182415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f182259c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 0x7f182415eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:60478/0 (socket says 172.21.15.122:60478) 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1822d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f18240b4a50 0x7f18240cfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:59894/0 (socket says 172.21.15.122:59894) 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 0x7f1824072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:37952/0 (socket says 172.21.15.122:37952) 2024-03-06T07:22:47.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f182259c640 1 -- 172.21.15.122:0/1784384414 learned_addr learned my addr 172.21.15.122:0/1784384414 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1821d9b640 1 -- 172.21.15.122:0/1784384414 >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 msgr2=0x7f182415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1821d9b640 1 --2- 172.21.15.122:0/1784384414 >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 0x7f182415eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1821d9b640 1 -- 172.21.15.122:0/1784384414 >> v1:172.21.15.169:6789/0 conn(0x7f18240b4a50 legacy=0x7f18240cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f182159a640 1 -- 172.21.15.122:0/1784384414 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124434108 0 0) 0x7f182406e770 con 0x7f18240b4a50 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f1821d9b640 1 -- 172.21.15.122:0/1784384414 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182406e490 con 0x7f182406e000 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.616+0000 7f182259c640 1 --2- 172.21.15.122:0/1784384414 >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 0x7f182415eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.617+0000 7f1821d9b640 1 --2- 172.21.15.122:0/1784384414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 0x7f1824072070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f182415e7e0 tx=0x7f181802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8baae466d2491595 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.617+0000 7f182159a640 1 -- 172.21.15.122:0/1784384414 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818039070 con 0x7f182406e000 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.617+0000 7f182159a640 1 -- 172.21.15.122:0/1784384414 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181802cc40 con 0x7f182406e000 2024-03-06T07:22:47.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.617+0000 7f182159a640 1 -- 172.21.15.122:0/1784384414 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818034440 con 0x7f182406e000 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.617+0000 7f18289bc640 1 -- 172.21.15.122:0/1784384414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 msgr2=0x7f1824072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.617+0000 7f18289bc640 1 --2- 172.21.15.122:0/1784384414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 0x7f1824072070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f182415e7e0 tx=0x7f181802be90 comp rx=0 tx=0).stop 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.618+0000 7f18289bc640 1 -- 172.21.15.122:0/1784384414 shutdown_connections 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.618+0000 7f18289bc640 1 --2- 172.21.15.122:0/1784384414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f182406e000 0x7f1824072070 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.618+0000 7f18289bc640 1 --2- 172.21.15.122:0/1784384414 >> v2:172.21.15.169:3300/0 conn(0x7f18240cc920 0x7f182415eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.618+0000 7f18289bc640 1 -- 172.21.15.122:0/1784384414 >> 172.21.15.122:0/1784384414 conn(0x7f18240b1c80 msgr2=0x7f18240cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.618+0000 7f18289bc640 1 -- 172.21.15.122:0/1784384414 shutdown_connections 2024-03-06T07:22:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.618+0000 7f18289bc640 1 -- 172.21.15.122:0/1784384414 wait complete. 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.619+0000 7f18289bc640 1 Processor -- start 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.619+0000 7f18289bc640 1 -- start start 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 0x7f1824154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18240cc920 0x7f182414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1824153790 0x7f18241434b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240ccd00 con 0x7f18240b4a50 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182415e2f0 con 0x7f18240cc920 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cce80 con 0x7f1824153790 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 0x7f1824154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 0x7f1824154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41064/0 (socket says 172.21.15.122:41064) 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 -- 172.21.15.122:0/1034638411 learned_addr learned my addr 172.21.15.122:0/1034638411 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f1822d9d640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1824153790 0x7f18241434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f1821d9b640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18240cc920 0x7f182414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 -- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1824153790 msgr2=0x7f18241434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1824153790 0x7f18241434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 -- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18240cc920 msgr2=0x7f182414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18240cc920 0x7f182414b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 -- 172.21.15.122:0/1034638411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824153c40 con 0x7f18240b4a50 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f182259c640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 0x7f1824154ad0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f181c00e9b0 tx=0x7f181c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c017070 con 0x7f18240b4a50 2024-03-06T07:22:47.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c00c9a0 con 0x7f18240b4a50 2024-03-06T07:22:47.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00cd50 con 0x7f18240b4a50 2024-03-06T07:22:47.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1824153f00 con 0x7f18240b4a50 2024-03-06T07:22:47.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.620+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18240affc0 con 0x7f18240b4a50 2024-03-06T07:22:47.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.621+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182404f330 con 0x7f18240b4a50 2024-03-06T07:22:47.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.622+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f181c01a070 con 0x7f18240b4a50 2024-03-06T07:22:47.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.622+0000 7f1812ffd640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f180406b500 0x7f180406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.622+0000 7f1821d9b640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f180406b500 0x7f180406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.622+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f181c08a070 con 0x7f18240b4a50 2024-03-06T07:22:47.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.623+0000 7f1821d9b640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f180406b500 0x7f180406d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1818036000 tx=0x7f1818030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.627+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f181c058b70 con 0x7f18240b4a50 2024-03-06T07:22:47.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.869+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f18240ceae0 con 0x7f18240b4a50 2024-03-06T07:22:47.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.873+0000 7f1812ffd640 1 -- 172.21.15.122:0/1034638411 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f181c05cc10 con 0x7f18240b4a50 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f180406b500 msgr2=0x7f180406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f180406b500 0x7f180406d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1818036000 tx=0x7f1818030000 comp rx=0 tx=0).stop 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 msgr2=0x7f1824154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 0x7f1824154ad0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f181c00e9b0 tx=0x7f181c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 shutdown_connections 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f180406b500 0x7f180406d9c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1824153790 0x7f18241434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18240cc920 0x7f182414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 --2- 172.21.15.122:0/1034638411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18240b4a50 0x7f1824154ad0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 >> 172.21.15.122:0/1034638411 conn(0x7f18240b1c80 msgr2=0x7f182414e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 shutdown_connections 2024-03-06T07:22:47.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.876+0000 7f18289bc640 1 -- 172.21.15.122:0/1034638411 wait complete. 2024-03-06T07:22:47.889 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.989+0000 7fe8ace50640 1 Processor -- start 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.989+0000 7fe8ace50640 1 -- start start 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe8ace50640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bed0 0x7fe8a80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe8ace50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe8a80d04e0 0x7fe8a80d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe8ace50640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a80d2430 con 0x7fe8a80d1fe0 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe8ace50640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a80b2d80 con 0x7fe8a80d04e0 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe8ace50640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a80ae9c0 con 0x7fe8a815bed0 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe8a80d04e0 0x7fe8a80d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe8a80d04e0 0x7fe8a80d1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41080/0 (socket says 172.21.15.122:41080) 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 -- 172.21.15.122:0/3352097345 learned_addr learned my addr 172.21.15.122:0/3352097345 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe8a659c640 1 --2- 172.21.15.122:0/3352097345 >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bed0 0x7fe8a80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 -- 172.21.15.122:0/3352097345 >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bed0 msgr2=0x7fe8a80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 --2- 172.21.15.122:0/3352097345 >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bed0 0x7fe8a80b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 -- 172.21.15.122:0/3352097345 >> v1:172.21.15.169:6789/0 conn(0x7fe8a80d1fe0 legacy=0x7fe8a815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.990+0000 7fe89ffff640 1 -- 172.21.15.122:0/3352097345 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a80aeca0 con 0x7fe8a80d04e0 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.991+0000 7fe89ffff640 1 --2- 172.21.15.122:0/3352097345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80d1aa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe8a809fbe0 tx=0x7fe89002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd91623fb150fcbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.991+0000 7fe8a5d9b640 1 -- 172.21.15.122:0/3352097345 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89002e070 con 0x7fe8a80d04e0 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.991+0000 7fe8a5d9b640 1 -- 172.21.15.122:0/3352097345 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe890037070 con 0x7fe8a80d04e0 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.991+0000 7fe8a5d9b640 1 -- 172.21.15.122:0/3352097345 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890031a00 con 0x7fe8a80d04e0 2024-03-06T07:22:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 -- 172.21.15.122:0/3352097345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 msgr2=0x7fe8a80d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 --2- 172.21.15.122:0/3352097345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80d1aa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe8a809fbe0 tx=0x7fe89002b880 comp rx=0 tx=0).stop 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 -- 172.21.15.122:0/3352097345 shutdown_connections 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 --2- 172.21.15.122:0/3352097345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80d1aa0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 --2- 172.21.15.122:0/3352097345 >> v2:172.21.15.169:3300/0 conn(0x7fe8a815bed0 0x7fe8a80b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 -- 172.21.15.122:0/3352097345 >> 172.21.15.122:0/3352097345 conn(0x7fe8a809a9f0 msgr2=0x7fe8a809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 -- 172.21.15.122:0/3352097345 shutdown_connections 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.992+0000 7fe8ace50640 1 -- 172.21.15.122:0/3352097345 wait complete. 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.993+0000 7fe8ace50640 1 Processor -- start 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.993+0000 7fe8ace50640 1 -- start start 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a80d1fe0 0x7fe8a80b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a815bed0 0x7fe8a80c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a815c2b0 con 0x7fe8a80d04e0 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80af260 con 0x7fe8a80d1fe0 2024-03-06T07:22:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80cc330 con 0x7fe8a815bed0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41092/0 (socket says 172.21.15.122:41092) 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 -- 172.21.15.122:0/806206517 learned_addr learned my addr 172.21.15.122:0/806206517 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe89ffff640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a80d1fe0 0x7fe8a80b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a6d9d640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a815bed0 0x7fe8a80c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 -- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a815bed0 msgr2=0x7fe8a80c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a815bed0 0x7fe8a80c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 -- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a80d1fe0 msgr2=0x7fe8a80b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a80d1fe0 0x7fe8a80b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 -- 172.21.15.122:0/806206517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a80b6010 con 0x7fe8a80d04e0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8a659c640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80b5050 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e9e0 tx=0x7fe89800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898017070 con 0x7fe8a80d04e0 2024-03-06T07:22:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89800c9d0 con 0x7fe8a80d04e0 2024-03-06T07:22:47.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a815eac0 con 0x7fe8a80d04e0 2024-03-06T07:22:47.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89800cdc0 con 0x7fe8a80d04e0 2024-03-06T07:22:47.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.994+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a81575f0 con 0x7fe8a80d04e0 2024-03-06T07:22:47.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.995+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a804f330 con 0x7fe8a80d04e0 2024-03-06T07:22:48.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.996+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe89801a070 con 0x7fe8a80d04e0 2024-03-06T07:22:48.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.996+0000 7fe89effd640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe88806b4d0 0x7fe88806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.996+0000 7fe89ffff640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe88806b4d0 0x7fe88806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.996+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe898089db0 con 0x7fe8a80d04e0 2024-03-06T07:22:48.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:47.997+0000 7fe89ffff640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe88806b4d0 0x7fe88806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe890036000 tx=0x7fe89003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.001+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8a80b6010 con 0x7fe8a80d04e0 2024-03-06T07:22:48.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.235+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fe8a80c7390 con 0x7fe8a80d04e0 2024-03-06T07:22:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.237+0000 7fe89effd640 1 -- 172.21.15.122:0/806206517 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe8980589c0 con 0x7fe8a80d04e0 2024-03-06T07:22:48.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe88806b4d0 msgr2=0x7fe88806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe88806b4d0 0x7fe88806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe890036000 tx=0x7fe89003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:48.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 msgr2=0x7fe8a80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80b5050 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e9e0 tx=0x7fe89800eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 shutdown_connections 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe88806b4d0 0x7fe88806d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe8a815bed0 0x7fe8a80c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8a80d1fe0 0x7fe8a80b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 --2- 172.21.15.122:0/806206517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a80d04e0 0x7fe8a80b5050 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 >> 172.21.15.122:0/806206517 conn(0x7fe8a809a9f0 msgr2=0x7fe8a8159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 shutdown_connections 2024-03-06T07:22:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.240+0000 7fe8ace50640 1 -- 172.21.15.122:0/806206517 wait complete. 2024-03-06T07:22:48.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:22:48.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.352+0000 7fe4fcf25640 1 Processor -- start 2024-03-06T07:22:48.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.352+0000 7fe4fcf25640 1 -- start start 2024-03-06T07:22:48.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4fcf25640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe4f815bbf0 0x7fe4f8071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4fcf25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f8072490 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4fcf25640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f80d0090 con 0x7fe4f80cebc0 2024-03-06T07:22:48.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4fcf25640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f80d0370 con 0x7fe4f815bbf0 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f6d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe4f80cd7c0 0x7fe4f80cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59542/0 (socket says 172.21.15.122:59542) 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f6d9d640 1 -- 172.21.15.122:0/1417638446 learned_addr learned my addr 172.21.15.122:0/1417638446 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f659c640 1 --2- 172.21.15.122:0/1417638446 >> v2:172.21.15.169:3300/0 conn(0x7fe4f815bbf0 0x7fe4f8071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f5d9b640 1 -- 172.21.15.122:0/1417638446 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282685627 0 0) 0x7fe4f8072490 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f5d9b640 1 -- 172.21.15.122:0/1417638446 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4003520 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f5d9b640 1 -- 172.21.15.122:0/1417638446 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3313108396 0 0) 0x7fe4e4003520 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f659c640 1 -- 172.21.15.122:0/1417638446 >> v1:172.21.15.169:6789/0 conn(0x7fe4f80cebc0 legacy=0x7fe4f815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f659c640 1 -- 172.21.15.122:0/1417638446 >> v1:172.21.15.122:6789/0 conn(0x7fe4f80cd7c0 legacy=0x7fe4f80cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.353+0000 7fe4f659c640 1 -- 172.21.15.122:0/1417638446 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f80d0650 con 0x7fe4f815bbf0 2024-03-06T07:22:48.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.354+0000 7fe4f659c640 1 --2- 172.21.15.122:0/1417638446 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f815bbf0 0x7fe4f8071e00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80097a0 tx=0x7fe4e802b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0de633e6d146e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.354+0000 7fe4f5d9b640 1 -- 172.21.15.122:0/1417638446 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8037070 con 0x7fe4f815bbf0 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.354+0000 7fe4f5d9b640 1 -- 172.21.15.122:0/1417638446 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8030510 con 0x7fe4f815bbf0 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.354+0000 7fe4f5d9b640 1 -- 172.21.15.122:0/1417638446 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00019f0 con 0x7fe4f815bbf0 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 -- 172.21.15.122:0/1417638446 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f815bbf0 msgr2=0x7fe4f8071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/1417638446 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f815bbf0 0x7fe4f8071e00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80097a0 tx=0x7fe4e802b680 comp rx=0 tx=0).stop 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 -- 172.21.15.122:0/1417638446 shutdown_connections 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/1417638446 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f815bbf0 0x7fe4f8071e00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 -- 172.21.15.122:0/1417638446 >> 172.21.15.122:0/1417638446 conn(0x7fe4f80b1c80 msgr2=0x7fe4f80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 -- 172.21.15.122:0/1417638446 shutdown_connections 2024-03-06T07:22:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.355+0000 7fe4fcf25640 1 -- 172.21.15.122:0/1417638446 wait complete. 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.356+0000 7fe4fcf25640 1 Processor -- start 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.356+0000 7fe4fcf25640 1 -- start start 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4fcf25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 0x7fe4f8151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4fcf25640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f80cebc0 0x7fe4f813b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4fcf25640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4f815bbf0 0x7fe4f813c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4fcf25640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80b3680 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4fcf25640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80cf010 con 0x7fe4f80cebc0 2024-03-06T07:22:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4fcf25640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80b3800 con 0x7fe4f815bbf0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 0x7fe4f8151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 0x7fe4f8151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41096/0 (socket says 172.21.15.122:41096) 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 -- 172.21.15.122:0/986051831 learned_addr learned my addr 172.21.15.122:0/986051831 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4edd9b640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f80cebc0 0x7fe4f813b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f6d9d640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4f815bbf0 0x7fe4f813c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 -- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4f815bbf0 msgr2=0x7fe4f813c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4f815bbf0 0x7fe4f813c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 -- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f80cebc0 msgr2=0x7fe4f813b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f80cebc0 0x7fe4f813b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.357+0000 7fe4f659c640 1 -- 172.21.15.122:0/986051831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8152350 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.358+0000 7fe4f659c640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 0x7fe4f8151e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe4e802b6c0 tx=0x7fe4e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.358+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8042070 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.358+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e803e870 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.358+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f8152620 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.358+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8037040 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.358+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f8152ec0 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.359+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f8152a10 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.359+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4e8045070 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.359+0000 7fe4ef7fe640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe4d806b4d0 0x7fe4d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.360+0000 7fe4edd9b640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe4d806b4d0 0x7fe4d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.360+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe4e80abd50 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.360+0000 7fe4edd9b640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe4d806b4d0 0x7fe4d806d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe4f80b15b0 tx=0x7fe4e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.364+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4e80af030 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.601+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe4f804f330 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.605+0000 7fe4ef7fe640 1 -- 172.21.15.122:0/986051831 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe4e807aa00 con 0x7fe4f80cd7c0 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe4d806b4d0 msgr2=0x7fe4d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe4d806b4d0 0x7fe4d806d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe4f80b15b0 tx=0x7fe4e0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 msgr2=0x7fe4f8151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 0x7fe4f8151e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe4e802b6c0 tx=0x7fe4e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 shutdown_connections 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe4d806b4d0 0x7fe4d806d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4f815bbf0 0x7fe4f813c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4f80cebc0 0x7fe4f813b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 --2- 172.21.15.122:0/986051831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4f80cd7c0 0x7fe4f8151e10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 >> 172.21.15.122:0/986051831 conn(0x7fe4f80b1c80 msgr2=0x7fe4f815bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 shutdown_connections 2024-03-06T07:22:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.607+0000 7fe4fcf25640 1 -- 172.21.15.122:0/986051831 wait complete. 2024-03-06T07:22:48.620 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:22:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.719+0000 7efe46719640 1 Processor -- start 2024-03-06T07:22:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.719+0000 7efe46719640 1 -- start start 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.719+0000 7efe46719640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efe40162400 0x7efe400d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.719+0000 7efe46719640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe400d22c0 con 0x7efe400b4460 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.719+0000 7efe46719640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe400d25a0 con 0x7efe40162970 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.719+0000 7efe46719640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe40161b00 con 0x7efe40162400 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe3ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efe400b4460 0x7efe400b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59558/0 (socket says 172.21.15.122:59558) 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe3ffff640 1 -- 172.21.15.122:0/455206884 learned_addr learned my addr 172.21.15.122:0/455206884 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe44cb6640 1 --2- 172.21.15.122:0/455206884 >> v2:172.21.15.169:3300/0 conn(0x7efe40162400 0x7efe400d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe3effd640 1 -- 172.21.15.122:0/455206884 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450793450 0 0) 0x7efe400d22c0 con 0x7efe400b4460 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe3effd640 1 -- 172.21.15.122:0/455206884 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe28003520 con 0x7efe400b4460 2024-03-06T07:22:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe44cb6640 1 -- 172.21.15.122:0/455206884 >> v1:172.21.15.169:6789/0 conn(0x7efe40162970 legacy=0x7efe40161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe3effd640 1 -- 172.21.15.122:0/455206884 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527568558 0 0) 0x7efe28003520 con 0x7efe400b4460 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe44cb6640 1 -- 172.21.15.122:0/455206884 >> v1:172.21.15.122:6789/0 conn(0x7efe400b4460 legacy=0x7efe400b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.720+0000 7efe44cb6640 1 -- 172.21.15.122:0/455206884 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe400d2880 con 0x7efe40162400 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.721+0000 7efe44cb6640 1 --2- 172.21.15.122:0/455206884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe40162400 0x7efe400d4800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efe38009d90 tx=0x7efe3802bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ab649c50077e84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.721+0000 7efe3effd640 1 -- 172.21.15.122:0/455206884 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38039070 con 0x7efe40162400 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.721+0000 7efe3effd640 1 -- 172.21.15.122:0/455206884 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3802c720 con 0x7efe40162400 2024-03-06T07:22:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.721+0000 7efe3effd640 1 -- 172.21.15.122:0/455206884 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34001670 con 0x7efe40162400 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.721+0000 7efe46719640 1 -- 172.21.15.122:0/455206884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe40162400 msgr2=0x7efe400d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.721+0000 7efe46719640 1 --2- 172.21.15.122:0/455206884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe40162400 0x7efe400d4800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efe38009d90 tx=0x7efe3802bc70 comp rx=0 tx=0).stop 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.722+0000 7efe46719640 1 -- 172.21.15.122:0/455206884 shutdown_connections 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.722+0000 7efe46719640 1 --2- 172.21.15.122:0/455206884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe40162400 0x7efe400d4800 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.722+0000 7efe46719640 1 -- 172.21.15.122:0/455206884 >> 172.21.15.122:0/455206884 conn(0x7efe400d3ce0 msgr2=0x7efe400d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.722+0000 7efe46719640 1 -- 172.21.15.122:0/455206884 shutdown_connections 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.722+0000 7efe46719640 1 -- 172.21.15.122:0/455206884 wait complete. 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.723+0000 7efe46719640 1 Processor -- start 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.723+0000 7efe46719640 1 -- start start 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe400b4460 0x7efe40155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe40162400 0x7efe400b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 0x7efe400b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe400ce950 con 0x7efe40162970 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe400cecc0 con 0x7efe400b4460 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe400b4840 con 0x7efe40162400 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 0x7efe400b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 0x7efe400b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41098/0 (socket says 172.21.15.122:41098) 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 -- 172.21.15.122:0/1062170748 learned_addr learned my addr 172.21.15.122:0/1062170748 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe3ffff640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe400b4460 0x7efe40155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe3f7fe640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe40162400 0x7efe400b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 -- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe40162400 msgr2=0x7efe400b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe40162400 0x7efe400b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 -- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe400b4460 msgr2=0x7efe40155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe400b4460 0x7efe40155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 -- 172.21.15.122:0/1062170748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe400b7780 con 0x7efe40162970 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe44cb6640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 0x7efe400b1ea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7efe38037040 tx=0x7efe38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38039040 con 0x7efe40162970 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe38003bb0 con 0x7efe40162970 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe400b7a60 con 0x7efe40162970 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38034770 con 0x7efe40162970 2024-03-06T07:22:48.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.724+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe400b8020 con 0x7efe40162970 2024-03-06T07:22:48.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.725+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4004f330 con 0x7efe40162970 2024-03-06T07:22:48.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.726+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe38034910 con 0x7efe40162970 2024-03-06T07:22:48.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.726+0000 7efe3cff9640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2c06b4d0 0x7efe2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.726+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efe380ac520 con 0x7efe40162970 2024-03-06T07:22:48.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.726+0000 7efe3ffff640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2c06b4d0 0x7efe2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.727+0000 7efe3ffff640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2c06b4d0 0x7efe2c06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efe34001eb0 tx=0x7efe34001e00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.731+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe3804e070 con 0x7efe40162970 2024-03-06T07:22:48.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.975+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7efe4006b180 con 0x7efe40162970 2024-03-06T07:22:48.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.977+0000 7efe3cff9640 1 -- 172.21.15.122:0/1062170748 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7efe3807b0a0 con 0x7efe40162970 2024-03-06T07:22:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2c06b4d0 msgr2=0x7efe2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2c06b4d0 0x7efe2c06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efe34001eb0 tx=0x7efe34001e00 comp rx=0 tx=0).stop 2024-03-06T07:22:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 msgr2=0x7efe400b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 0x7efe400b1ea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7efe38037040 tx=0x7efe38004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 shutdown_connections 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efe2c06b4d0 0x7efe2c06d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe40162970 0x7efe400b1ea0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efe40162400 0x7efe400b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 --2- 172.21.15.122:0/1062170748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efe400b4460 0x7efe40155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 >> 172.21.15.122:0/1062170748 conn(0x7efe400d3ce0 msgr2=0x7efe40160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 shutdown_connections 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:48.980+0000 7efe46719640 1 -- 172.21.15.122:0/1062170748 wait complete. 2024-03-06T07:22:48.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:22:49.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.092+0000 7fea098f4640 1 Processor -- start 2024-03-06T07:22:49.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.092+0000 7fea098f4640 1 -- start start 2024-03-06T07:22:49.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea098f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fea040b4460 0x7fea040d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea098f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea04162970 0x7fea040b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea098f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04161cf0 0x7fea04162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea098f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea04161b00 con 0x7fea04162970 2024-03-06T07:22:49.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea098f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea040cecc0 con 0x7fea040b4460 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea098f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea040d22c0 con 0x7fea04161cf0 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea04162970 0x7fea040b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea04162970 0x7fea040b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41108/0 (socket says 172.21.15.122:41108) 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 -- 172.21.15.122:0/394914968 learned_addr learned my addr 172.21.15.122:0/394914968 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea02ffd640 1 --2- 172.21.15.122:0/394914968 >> v2:172.21.15.169:3300/0 conn(0x7fea040b4460 0x7fea040d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea037fe640 1 --2- 172.21.15.122:0/394914968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04161cf0 0x7fea04162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 -- 172.21.15.122:0/394914968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04161cf0 msgr2=0x7fea04162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 --2- 172.21.15.122:0/394914968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04161cf0 0x7fea04162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 -- 172.21.15.122:0/394914968 >> v2:172.21.15.169:3300/0 conn(0x7fea040b4460 msgr2=0x7fea040d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 --2- 172.21.15.122:0/394914968 >> v2:172.21.15.169:3300/0 conn(0x7fea040b4460 0x7fea040d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea027fc640 1 -- 172.21.15.122:0/394914968 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea040d25a0 con 0x7fea04162970 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.093+0000 7fea02ffd640 1 --2- 172.21.15.122:0/394914968 >> v2:172.21.15.169:3300/0 conn(0x7fea040b4460 0x7fea040d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.094+0000 7fea027fc640 1 --2- 172.21.15.122:0/394914968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04162970 0x7fea040b3580 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fea0408ddd0 tx=0x7fe9f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3028b38ac05f6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.094+0000 7fea01ffb640 1 -- 172.21.15.122:0/394914968 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8035020 con 0x7fea04162970 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.094+0000 7fea01ffb640 1 -- 172.21.15.122:0/394914968 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f8031870 con 0x7fea04162970 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.094+0000 7fea01ffb640 1 -- 172.21.15.122:0/394914968 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8031c40 con 0x7fea04162970 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 -- 172.21.15.122:0/394914968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04162970 msgr2=0x7fea040b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 --2- 172.21.15.122:0/394914968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04162970 0x7fea040b3580 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fea0408ddd0 tx=0x7fe9f802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 -- 172.21.15.122:0/394914968 shutdown_connections 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 --2- 172.21.15.122:0/394914968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04162970 0x7fea040b3580 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 --2- 172.21.15.122:0/394914968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04161cf0 0x7fea04162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 --2- 172.21.15.122:0/394914968 >> v2:172.21.15.169:3300/0 conn(0x7fea040b4460 0x7fea040d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 -- 172.21.15.122:0/394914968 >> 172.21.15.122:0/394914968 conn(0x7fea040d3ce0 msgr2=0x7fea040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 -- 172.21.15.122:0/394914968 shutdown_connections 2024-03-06T07:22:49.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.095+0000 7fea098f4640 1 -- 172.21.15.122:0/394914968 wait complete. 2024-03-06T07:22:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.096+0000 7fea098f4640 1 Processor -- start 2024-03-06T07:22:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.096+0000 7fea098f4640 1 -- start start 2024-03-06T07:22:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea098f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea040b4460 0x7fea04155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea098f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 0x7fea040b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea098f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04162970 0x7fea040b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea098f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea040ce950 con 0x7fea04161cf0 2024-03-06T07:22:49.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea098f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea040d2b60 con 0x7fea040b4460 2024-03-06T07:22:49.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea098f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea040b3ac0 con 0x7fea04162970 2024-03-06T07:22:49.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 0x7fea040b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 0x7fea040b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41120/0 (socket says 172.21.15.122:41120) 2024-03-06T07:22:49.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 -- 172.21.15.122:0/4114092262 learned_addr learned my addr 172.21.15.122:0/4114092262 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:49.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea02ffd640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea040b4460 0x7fea04155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea037fe640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04162970 0x7fea040b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 -- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04162970 msgr2=0x7fea040b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04162970 0x7fea040b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 -- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea040b4460 msgr2=0x7fea04155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea040b4460 0x7fea04155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 -- 172.21.15.122:0/4114092262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea040b7150 con 0x7fea04161cf0 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea037fe640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04162970 0x7fea040b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fea027fc640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 0x7fea040b6900 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe9f802b950 tx=0x7fe9f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.097+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8041070 con 0x7fea04161cf0 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.098+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f803ac80 con 0x7fea04161cf0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.098+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea040b7430 con 0x7fea04161cf0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.098+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8035040 con 0x7fea04161cf0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.098+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea040b79f0 con 0x7fea04161cf0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.099+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0404f330 con 0x7fea04161cf0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.099+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9f8038020 con 0x7fea04161cf0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.099+0000 7fe9ebfff640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9f006b4d0 0x7fe9f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.099+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe9f80abc30 con 0x7fea04161cf0 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.099+0000 7fea02ffd640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9f006b4d0 0x7fe9f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.100+0000 7fea02ffd640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9f006b4d0 0x7fe9f006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fea040cdea0 tx=0x7fe9f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.104+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9f807a840 con 0x7fea04161cf0 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.340+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fea040d3400 con 0x7fea04161cf0 2024-03-06T07:22:49.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.342+0000 7fe9ebfff640 1 -- 172.21.15.122:0/4114092262 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe9f807e8e0 con 0x7fea04161cf0 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9f006b4d0 msgr2=0x7fe9f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9f006b4d0 0x7fe9f006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fea040cdea0 tx=0x7fe9f4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 msgr2=0x7fea040b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 0x7fea040b6900 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe9f802b950 tx=0x7fe9f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 shutdown_connections 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9f006b4d0 0x7fe9f006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea04162970 0x7fea040b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea04161cf0 0x7fea040b6900 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 --2- 172.21.15.122:0/4114092262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea040b4460 0x7fea04155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 >> 172.21.15.122:0/4114092262 conn(0x7fea040d3ce0 msgr2=0x7fea040cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 shutdown_connections 2024-03-06T07:22:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.344+0000 7fea098f4640 1 -- 172.21.15.122:0/4114092262 wait complete. 2024-03-06T07:22:49.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:49.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.455+0000 7fd55fa5c640 1 Processor -- start 2024-03-06T07:22:49.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.455+0000 7fd55fa5c640 1 -- start start 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55fa5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd55806da40 0x7fd5580cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55fa5c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd5580cd7a0 0x7fd55815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55fa5c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5580cc980 con 0x7fd5581424f0 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55fa5c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5580cd220 con 0x7fd5580cd7a0 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55fa5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5580ccc60 con 0x7fd55806da40 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55dff9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd5580cd7a0 0x7fd55815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55d7f8640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd5581424f0 0x7fd55806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59560/0 (socket says 172.21.15.122:59560) 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55d7f8640 1 -- 172.21.15.122:0/2995931270 learned_addr learned my addr 172.21.15.122:0/2995931270 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55dff9640 1 -- 172.21.15.122:0/2995931270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd55806da40 msgr2=0x7fd5580cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55cff7640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd55806da40 0x7fd5580cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd54ffff640 1 -- 172.21.15.122:0/2995931270 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816393952 0 0) 0x7fd5580cc980 con 0x7fd5581424f0 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55dff9640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd55806da40 0x7fd5580cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55dff9640 1 -- 172.21.15.122:0/2995931270 >> v1:172.21.15.122:6789/0 conn(0x7fd5581424f0 legacy=0x7fd55806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55dff9640 1 -- 172.21.15.122:0/2995931270 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580ccf40 con 0x7fd5580cd7a0 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.456+0000 7fd55cff7640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd55806da40 0x7fd5580cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.457+0000 7fd55dff9640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5580cd7a0 0x7fd55815cef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd550009d90 tx=0x7fd55002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4757f784a3c3bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.457+0000 7fd54ffff640 1 -- 172.21.15.122:0/2995931270 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550039070 con 0x7fd5580cd7a0 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.457+0000 7fd54ffff640 1 -- 172.21.15.122:0/2995931270 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55002cc50 con 0x7fd5580cd7a0 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.457+0000 7fd54ffff640 1 -- 172.21.15.122:0/2995931270 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550034440 con 0x7fd5580cd7a0 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2995931270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5580cd7a0 msgr2=0x7fd55815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5580cd7a0 0x7fd55815cef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd550009d90 tx=0x7fd55002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2995931270 shutdown_connections 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5580cd7a0 0x7fd55815cef0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2995931270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd55806da40 0x7fd5580cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2995931270 >> 172.21.15.122:0/2995931270 conn(0x7fd5580cf8a0 msgr2=0x7fd5580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2995931270 shutdown_connections 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.458+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2995931270 wait complete. 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.459+0000 7fd55fa5c640 1 Processor -- start 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.459+0000 7fd55fa5c640 1 -- start start 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 0x7fd5580aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5580cd7a0 0x7fd55813d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5581424f0 0x7fd5580b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558071ce0 con 0x7fd55806da40 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558071b60 con 0x7fd5580cd7a0 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd558071e60 con 0x7fd5581424f0 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 0x7fd5580aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 0x7fd5580aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41136/0 (socket says 172.21.15.122:41136) 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 -- 172.21.15.122:0/2166581545 learned_addr learned my addr 172.21.15.122:0/2166581545 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55dff9640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5581424f0 0x7fd5580b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55cff7640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5580cd7a0 0x7fd55813d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 -- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5581424f0 msgr2=0x7fd5580b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5581424f0 0x7fd5580b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 -- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5580cd7a0 msgr2=0x7fd55813d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5580cd7a0 0x7fd55813d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 -- 172.21.15.122:0/2166581545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580ad860 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55d7f8640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 0x7fd5580aeff0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd55400c460 tx=0x7fd55400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554018070 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd554007500 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5580adb40 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.460+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554004a20 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.461+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5580ae100 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.461+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55804f330 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.462+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd55401b070 con 0x7fd55806da40 2024-03-06T07:22:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.462+0000 7fd54dffb640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd53c06b500 0x7fd53c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.462+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd554088fc0 con 0x7fd55806da40 2024-03-06T07:22:49.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.462+0000 7fd55cff7640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd53c06b500 0x7fd53c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.463+0000 7fd55cff7640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd53c06b500 0x7fd53c06d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd5580cc450 tx=0x7fd548008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.467+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd554057bd0 con 0x7fd55806da40 2024-03-06T07:22:49.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.641+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5580b4a40 con 0x7fd55806da40 2024-03-06T07:22:49.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.642+0000 7fd54dffb640 1 -- 172.21.15.122:0/2166581545 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12891 (secure 0 0 0) 0x7fd55405bc70 con 0x7fd55806da40 2024-03-06T07:22:49.644 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:49.644 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":29,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:46.772495+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd53c06b500 msgr2=0x7fd53c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd53c06b500 0x7fd53c06d9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd5580cc450 tx=0x7fd548008040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 msgr2=0x7fd5580aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 0x7fd5580aeff0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd55400c460 tx=0x7fd55400c920 comp rx=0 tx=0).stop 2024-03-06T07:22:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 shutdown_connections 2024-03-06T07:22:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd53c06b500 0x7fd53c06d9c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5581424f0 0x7fd5580b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5580cd7a0 0x7fd55813d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 --2- 172.21.15.122:0/2166581545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd55806da40 0x7fd5580aeff0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 >> 172.21.15.122:0/2166581545 conn(0x7fd5580cf8a0 msgr2=0x7fd55813f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 shutdown_connections 2024-03-06T07:22:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.644+0000 7fd55fa5c640 1 -- 172.21.15.122:0/2166581545 wait complete. 2024-03-06T07:22:49.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:49.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:49.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.757+0000 7f2f9315e640 1 Processor -- start 2024-03-06T07:22:49.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.757+0000 7f2f9315e640 1 -- start start 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f9315e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2f8c15c050 0x7f2f8c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f9315e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2f8c0d0af0 0x7f2f8c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f9315e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0cd4b0 con 0x7f2f8c06fb40 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f9315e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c06ffb0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f9315e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c0cd790 con 0x7f2f8c15c050 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f916fb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2f8c0d0af0 0x7f2f8c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f83fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2f8c06fb40 0x7f2f8c0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59566/0 (socket says 172.21.15.122:59566) 2024-03-06T07:22:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f83fff640 1 -- 172.21.15.122:0/725998269 learned_addr learned my addr 172.21.15.122:0/725998269 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f916fb640 1 -- 172.21.15.122:0/725998269 >> v2:172.21.15.169:3300/0 conn(0x7f2f8c15c050 msgr2=0x7f2f8c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f916fb640 1 --2- 172.21.15.122:0/725998269 >> v2:172.21.15.169:3300/0 conn(0x7f2f8c15c050 0x7f2f8c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f916fb640 1 -- 172.21.15.122:0/725998269 >> v1:172.21.15.122:6789/0 conn(0x7f2f8c06fb40 legacy=0x7f2f8c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f916fb640 1 -- 172.21.15.122:0/725998269 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0cda70 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.758+0000 7f2f837fe640 1 -- 172.21.15.122:0/725998269 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410300112 0 0) 0x7f2f8c0cd4b0 con 0x7f2f8c06fb40 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.759+0000 7f2f916fb640 1 --2- 172.21.15.122:0/725998269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c0cce70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c070ef0 tx=0x7f2f8402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0288e9a2cd6e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.759+0000 7f2f837fe640 1 -- 172.21.15.122:0/725998269 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84039070 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.759+0000 7f2f837fe640 1 -- 172.21.15.122:0/725998269 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8402cbf0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.759+0000 7f2f837fe640 1 -- 172.21.15.122:0/725998269 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84034440 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 -- 172.21.15.122:0/725998269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 msgr2=0x7f2f8c0cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 --2- 172.21.15.122:0/725998269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c0cce70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c070ef0 tx=0x7f2f8402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 -- 172.21.15.122:0/725998269 shutdown_connections 2024-03-06T07:22:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 --2- 172.21.15.122:0/725998269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c0cce70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 --2- 172.21.15.122:0/725998269 >> v2:172.21.15.169:3300/0 conn(0x7f2f8c15c050 0x7f2f8c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 -- 172.21.15.122:0/725998269 >> 172.21.15.122:0/725998269 conn(0x7f2f8c0b3800 msgr2=0x7f2f8c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 -- 172.21.15.122:0/725998269 shutdown_connections 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.760+0000 7f2f9315e640 1 -- 172.21.15.122:0/725998269 wait complete. 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.761+0000 7f2f9315e640 1 Processor -- start 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.761+0000 7f2f9315e640 1 -- start start 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f9315e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f8c06fb40 0x7f2f8c13ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f9315e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f9315e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f8c15c050 0x7f2f8c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f9315e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c165c30 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f9315e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c165ab0 con 0x7f2f8c15c050 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f9315e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c165db0 con 0x7f2f8c06fb40 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41148/0 (socket says 172.21.15.122:41148) 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 -- 172.21.15.122:0/917296513 learned_addr learned my addr 172.21.15.122:0/917296513 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f916fb640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f8c15c050 0x7f2f8c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f90efa640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f8c06fb40 0x7f2f8c13ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 -- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f8c06fb40 msgr2=0x7f2f8c13ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f8c06fb40 0x7f2f8c13ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 -- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f8c15c050 msgr2=0x7f2f8c1554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f8c15c050 0x7f2f8c1554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 -- 172.21.15.122:0/917296513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c153f00 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f916fb640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f8c15c050 0x7f2f8c1554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f90efa640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f8c06fb40 0x7f2f8c13ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f83fff640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c153600 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00c430 tx=0x7f2f7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.762+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0074e0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.763+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c0047e0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.763+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c1541e0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.763+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c004bd0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.763+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c1547a0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.764+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c06b460 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.764+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f7c011070 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.764+0000 7f2f817fa640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f7806b4d0 0x7f2f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.765+0000 7f2f90efa640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f7806b4d0 0x7f2f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.765+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2f7c089ae0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.765+0000 7f2f90efa640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f7806b4d0 0x7f2f7806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800a870 tx=0x7f2f88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.771+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f7c058d40 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.941+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f8c158bb0 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.942+0000 7f2f817fa640 1 -- 172.21.15.122:0/917296513 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f7c058640 con 0x7f2f8c0d0af0 2024-03-06T07:22:49.944 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.943+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f7806b4d0 msgr2=0x7f2f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.943+0000 7f2f9315e640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f7806b4d0 0x7f2f7806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800a870 tx=0x7f2f88008040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.943+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 msgr2=0x7f2f8c153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.943+0000 7f2f9315e640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c153600 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00c430 tx=0x7f2f7c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 shutdown_connections 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f7806b4d0 0x7f2f7806d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f8c15c050 0x7f2f8c1554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f8c0d0af0 0x7f2f8c153600 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 --2- 172.21.15.122:0/917296513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f8c06fb40 0x7f2f8c13ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 >> 172.21.15.122:0/917296513 conn(0x7f2f8c0b3800 msgr2=0x7f2f8c0aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 shutdown_connections 2024-03-06T07:22:49.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:49.944+0000 7f2f9315e640 1 -- 172.21.15.122:0/917296513 wait complete. 2024-03-06T07:22:49.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:50.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.055+0000 7f56c0a1c640 1 Processor -- start 2024-03-06T07:22:50.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.055+0000 7f56c0a1c640 1 -- start start 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.055+0000 7f56c0a1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.055+0000 7f56c0a1c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc0b3430 con 0x7f56bc0cf830 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.055+0000 7f56c0a1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc0b3710 con 0x7f56bc0d29c0 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.055+0000 7f56c0a1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b31e0 con 0x7f56bc15dfe0 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f56bc0cf830 0x7f56bc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59568/0 (socket says 172.21.15.122:59568) 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b3fff640 1 -- 172.21.15.122:0/223387331 learned_addr learned my addr 172.21.15.122:0/223387331 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56ba59c640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735586820 0 0) 0x7f56bc0b3430 con 0x7f56bc0cf830 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a8003520 con 0x7f56bc0cf830 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287812016 0 0) 0x7f56bc0b3710 con 0x7f56bc0d29c0 2024-03-06T07:22:50.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56bc0b3430 con 0x7f56bc0d29c0 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157171276 0 0) 0x7f56a8003520 con 0x7f56bc0cf830 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 msgr2=0x7f56bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 >> v1:172.21.15.169:6789/0 conn(0x7f56bc0d29c0 legacy=0x7f56bc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc0b39f0 con 0x7f56bc0cf830 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f56a4001670 con 0x7f56bc0cf830 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56ba59c640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 >> v1:172.21.15.122:6789/0 conn(0x7f56bc0cf830 legacy=0x7f56bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8003b70 0x7f56a8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56a8004800 0x7f56a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b31e0 con 0x7f56a8004800 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a800ba70 con 0x7f56a8003b70 2024-03-06T07:22:50.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a800bc40 con 0x7f56bc15dfe0 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56b3fff640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56a8004800 0x7f56a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56ba59c640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8003b70 0x7f56a8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.056+0000 7f56bad9d640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b3fff640 1 -- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 msgr2=0x7f56a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b3fff640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc15dfe0 0x7f56a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b3fff640 1 -- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8003b70 msgr2=0x7f56a8003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b3fff640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8003b70 0x7f56a8003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b3fff640 1 -- 172.21.15.122:0/223387331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8003520 con 0x7f56a8004800 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b3fff640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56a8004800 0x7f56a8004c80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0cfc70 tx=0x7f56a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0bf3e02d405e64c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4007c00 con 0x7f56a8004800 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a4030070 con 0x7f56a8004800 2024-03-06T07:22:50.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.057+0000 7f56b9d9b640 1 -- 172.21.15.122:0/223387331 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4005d10 con 0x7f56a8004800 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56c0a1c640 1 -- 172.21.15.122:0/223387331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56a8004800 msgr2=0x7f56a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/223387331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56a8004800 0x7f56a8004c80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0cfc70 tx=0x7f56a8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56bad9d640 1 -- 172.21.15.122:0/223387331 reap_dead start 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56c0a1c640 1 -- 172.21.15.122:0/223387331 shutdown_connections 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56c0a1c640 1 -- 172.21.15.122:0/223387331 >> 172.21.15.122:0/223387331 conn(0x7f56bc0d3cd0 msgr2=0x7f56bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56c0a1c640 1 -- 172.21.15.122:0/223387331 shutdown_connections 2024-03-06T07:22:50.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.058+0000 7f56c0a1c640 1 -- 172.21.15.122:0/223387331 wait complete. 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.059+0000 7f56c0a1c640 1 Processor -- start 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.059+0000 7f56c0a1c640 1 -- start start 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56c0a1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8004800 0x7f56bc06c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56c0a1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc06cd90 0x7f56bc1f7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56c0a1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 0x7f56bc15e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56c0a1c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc15e3c0 con 0x7f56bc1f77c0 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56c0a1c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b3d60 con 0x7f56a8004800 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56c0a1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc15e540 con 0x7f56bc06cd90 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56bad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 0x7f56bc15e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56bad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 0x7f56bc15e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41156/0 (socket says 172.21.15.122:41156) 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56bad9d640 1 -- 172.21.15.122:0/889263609 learned_addr learned my addr 172.21.15.122:0/889263609 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56b3fff640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc06cd90 0x7f56bc1f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56ba59c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8004800 0x7f56bc06c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56bad9d640 1 -- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc06cd90 msgr2=0x7f56bc1f7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56bad9d640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc06cd90 0x7f56bc1f7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.060+0000 7f56bad9d640 1 -- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8004800 msgr2=0x7f56bc06c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56bad9d640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8004800 0x7f56bc06c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56bad9d640 1 -- 172.21.15.122:0/889263609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc1f7f70 con 0x7f56bc1f77c0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56bad9d640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 0x7f56bc15e770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0b5770 tx=0x7f56b400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4019070 con 0x7f56bc1f77c0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b4006e20 con 0x7f56bc1f77c0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc1f3c30 con 0x7f56bc1f77c0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4003b90 con 0x7f56bc1f77c0 2024-03-06T07:22:50.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.061+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc1f41f0 con 0x7f56bc1f77c0 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.062+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc04f330 con 0x7f56bc1f77c0 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.062+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56b401c070 con 0x7f56bc1f77c0 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.062+0000 7f56b2ffd640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569c06b4d0 0x7f569c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.063+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f56b4089100 con 0x7f56bc1f77c0 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.063+0000 7f56ba59c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569c06b4d0 0x7f569c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.063+0000 7f56ba59c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569c06b4d0 0x7f569c06d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0a1ce0 tx=0x7f56ac009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.067+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56b4057d10 con 0x7f56bc1f77c0 2024-03-06T07:22:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.240+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f56bc0b51c0 con 0x7f56bc1f77c0 2024-03-06T07:22:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.240+0000 7f56b2ffd640 1 -- 172.21.15.122:0/889263609 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f56b405bdb0 con 0x7f56bc1f77c0 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:22:50.243 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:22:50.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569c06b4d0 msgr2=0x7f569c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569c06b4d0 0x7f569c06d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0a1ce0 tx=0x7f56ac009190 comp rx=0 tx=0).stop 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 msgr2=0x7f56bc15e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 0x7f56bc15e770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0b5770 tx=0x7f56b400ff70 comp rx=0 tx=0).stop 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 shutdown_connections 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569c06b4d0 0x7f569c06d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56bc1f77c0 0x7f56bc15e770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56bc06cd90 0x7f56bc1f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 --2- 172.21.15.122:0/889263609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56a8004800 0x7f56bc06c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 >> 172.21.15.122:0/889263609 conn(0x7f56bc0d3cd0 msgr2=0x7f56bc0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 shutdown_connections 2024-03-06T07:22:50.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.242+0000 7f56c0a1c640 1 -- 172.21.15.122:0/889263609 wait complete. 2024-03-06T07:22:50.255 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:50.255 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:50.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.358+0000 7fc75f1f6640 1 Processor -- start 2024-03-06T07:22:50.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.358+0000 7fc75f1f6640 1 -- start start 2024-03-06T07:22:50.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.358+0000 7fc75f1f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7580d2160 0x7fc75806c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.358+0000 7fc75f1f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7580cf3c0 con 0x7fc758162850 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.358+0000 7fc75f1f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7580cf0e0 con 0x7fc758162470 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.358+0000 7fc75f1f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75806f270 con 0x7fc7580d2160 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc75d793640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7580d2160 0x7fc75806c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc757fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc758162850 0x7fc7580d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59582/0 (socket says 172.21.15.122:59582) 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc757fff640 1 -- 172.21.15.122:0/354758068 learned_addr learned my addr 172.21.15.122:0/354758068 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc75d793640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7580d2160 0x7fc75806c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41172/0 (socket says 172.21.15.122:41172) 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc75d793640 1 -- 172.21.15.122:0/354758068 >> v1:172.21.15.169:6789/0 conn(0x7fc758162470 legacy=0x7fc75806f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc75d793640 1 -- 172.21.15.122:0/354758068 >> v1:172.21.15.122:6789/0 conn(0x7fc758162850 legacy=0x7fc7580d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc75d793640 1 -- 172.21.15.122:0/354758068 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75806ec20 con 0x7fc7580d2160 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.359+0000 7fc7577fe640 1 -- 172.21.15.122:0/354758068 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382892977 0 0) 0x7fc7580cf3c0 con 0x7fc758162850 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75d793640 1 --2- 172.21.15.122:0/354758068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7580d2160 0x7fc75806c3b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc750009d90 tx=0x7fc75002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6f1b90493a4da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc7577fe640 1 -- 172.21.15.122:0/354758068 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750039070 con 0x7fc7580d2160 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc7577fe640 1 -- 172.21.15.122:0/354758068 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75002cc60 con 0x7fc7580d2160 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc7577fe640 1 -- 172.21.15.122:0/354758068 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750034660 con 0x7fc7580d2160 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 -- 172.21.15.122:0/354758068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7580d2160 msgr2=0x7fc75806c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/354758068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7580d2160 0x7fc75806c3b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc750009d90 tx=0x7fc75002beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 -- 172.21.15.122:0/354758068 shutdown_connections 2024-03-06T07:22:50.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/354758068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7580d2160 0x7fc75806c3b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 -- 172.21.15.122:0/354758068 >> 172.21.15.122:0/354758068 conn(0x7fc75809a9f0 msgr2=0x7fc75809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 -- 172.21.15.122:0/354758068 shutdown_connections 2024-03-06T07:22:50.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.360+0000 7fc75f1f6640 1 -- 172.21.15.122:0/354758068 wait complete. 2024-03-06T07:22:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.361+0000 7fc75f1f6640 1 Processor -- start 2024-03-06T07:22:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.361+0000 7fc75f1f6640 1 -- start start 2024-03-06T07:22:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75f1f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7580d2160 0x7fc7580b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75f1f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 0x7fc7581f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75f1f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc758162850 0x7fc7581583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75f1f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806ff90 con 0x7fc758162470 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75f1f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806fe10 con 0x7fc7580d2160 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75f1f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806c8f0 con 0x7fc758162850 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc757fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 0x7fc7581f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc757fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 0x7fc7581f7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41182/0 (socket says 172.21.15.122:41182) 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc757fff640 1 -- 172.21.15.122:0/3112902791 learned_addr learned my addr 172.21.15.122:0/3112902791 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75cf92640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7580d2160 0x7fc7580b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.362+0000 7fc75d793640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc758162850 0x7fc7581583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc757fff640 1 -- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc758162850 msgr2=0x7fc7581583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc757fff640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc758162850 0x7fc7581583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc757fff640 1 -- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7580d2160 msgr2=0x7fc7580b62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc757fff640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7580d2160 0x7fc7580b62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc757fff640 1 -- 172.21.15.122:0/3112902791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758158920 con 0x7fc758162470 2024-03-06T07:22:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc757fff640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 0x7fc7581f7220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc74400c430 tx=0x7fc74400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7440074e0 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7440047e0 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7581f7950 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744004bd0 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.363+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7581f7f10 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.364+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75804f330 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.364+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc744011070 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.365+0000 7fc7557fa640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc73c06b4d0 0x7fc73c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.365+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc7440898c0 con 0x7fc758162470 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.365+0000 7fc75cf92640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc73c06b4d0 0x7fc73c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.365+0000 7fc75cf92640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc73c06b4d0 0x7fc73c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004f60 tx=0x7fc74c007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.369+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc758158920 con 0x7fc758162470 2024-03-06T07:22:50.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.542+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc7580ccd10 con 0x7fc758162470 2024-03-06T07:22:50.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.980+0000 7fc7557fa640 1 -- 172.21.15.122:0/3112902791 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc7440584d0 con 0x7fc758162470 2024-03-06T07:22:50.983 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:50.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc73c06b4d0 msgr2=0x7fc73c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc73c06b4d0 0x7fc73c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004f60 tx=0x7fc74c007e30 comp rx=0 tx=0).stop 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 msgr2=0x7fc7581f7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 0x7fc7581f7220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc74400c430 tx=0x7fc74400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 shutdown_connections 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc73c06b4d0 0x7fc73c06d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc758162850 0x7fc7581583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc758162470 0x7fc7581f7220 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 --2- 172.21.15.122:0/3112902791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7580d2160 0x7fc7580b62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 >> 172.21.15.122:0/3112902791 conn(0x7fc75809a9f0 msgr2=0x7fc7580d19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 shutdown_connections 2024-03-06T07:22:50.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:50.983+0000 7fc75f1f6640 1 -- 172.21.15.122:0/3112902791 wait complete. 2024-03-06T07:22:51.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:51.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.105+0000 7fb840f6c640 1 Processor -- start 2024-03-06T07:22:51.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.105+0000 7fb840f6c640 1 -- start start 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb840f6c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb83c0b4970 0x7fb83c06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb840f6c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c15ff90 con 0x7fb83c06d5c0 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb840f6c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c160270 con 0x7fb83c15bb40 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb840f6c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c0b4dd0 con 0x7fb83c0b4970 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb833fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb83c0b4970 0x7fb83c06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb83ad9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb83c06d5c0 0x7fb83c0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59596/0 (socket says 172.21.15.122:59596) 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb83ad9d640 1 -- 172.21.15.122:0/2018912312 learned_addr learned my addr 172.21.15.122:0/2018912312 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb833fff640 1 -- 172.21.15.122:0/2018912312 >> v1:172.21.15.169:6789/0 conn(0x7fb83c15bb40 legacy=0x7fb83c15f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb833fff640 1 -- 172.21.15.122:0/2018912312 >> v1:172.21.15.122:6789/0 conn(0x7fb83c06d5c0 legacy=0x7fb83c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb833fff640 1 -- 172.21.15.122:0/2018912312 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c160550 con 0x7fb83c0b4970 2024-03-06T07:22:51.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.106+0000 7fb839d9b640 1 -- 172.21.15.122:0/2018912312 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410712469 0 0) 0x7fb83c15ff90 con 0x7fb83c06d5c0 2024-03-06T07:22:51.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.107+0000 7fb833fff640 1 --2- 172.21.15.122:0/2018912312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c0b4970 0x7fb83c06d080 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb8240097a0 tx=0x7fb82402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284c34e4a518e42f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.107+0000 7fb839d9b640 1 -- 172.21.15.122:0/2018912312 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824035020 con 0x7fb83c0b4970 2024-03-06T07:22:51.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.107+0000 7fb839d9b640 1 -- 172.21.15.122:0/2018912312 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824031830 con 0x7fb83c0b4970 2024-03-06T07:22:51.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.107+0000 7fb839d9b640 1 -- 172.21.15.122:0/2018912312 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824031c00 con 0x7fb83c0b4970 2024-03-06T07:22:51.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 -- 172.21.15.122:0/2018912312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c0b4970 msgr2=0x7fb83c06d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 --2- 172.21.15.122:0/2018912312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c0b4970 0x7fb83c06d080 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb8240097a0 tx=0x7fb82402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 -- 172.21.15.122:0/2018912312 shutdown_connections 2024-03-06T07:22:51.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 --2- 172.21.15.122:0/2018912312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c0b4970 0x7fb83c06d080 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 -- 172.21.15.122:0/2018912312 >> 172.21.15.122:0/2018912312 conn(0x7fb83c06b190 msgr2=0x7fb83c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 -- 172.21.15.122:0/2018912312 shutdown_connections 2024-03-06T07:22:51.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.108+0000 7fb840f6c640 1 -- 172.21.15.122:0/2018912312 wait complete. 2024-03-06T07:22:51.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.109+0000 7fb840f6c640 1 Processor -- start 2024-03-06T07:22:51.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.109+0000 7fb840f6c640 1 -- start start 2024-03-06T07:22:51.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb83c06d5c0 0x7fb83c0b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb83c0b4970 0x7fb83c0af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 0x7fb83c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c15bf20 con 0x7fb83c15bb40 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c06c910 con 0x7fb83c0b4970 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c15c0a0 con 0x7fb83c06d5c0 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 0x7fb83c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 0x7fb83c0b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41190/0 (socket says 172.21.15.122:41190) 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 -- 172.21.15.122:0/984938044 learned_addr learned my addr 172.21.15.122:0/984938044 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83a59c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb83c06d5c0 0x7fb83c0b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb833fff640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb83c0b4970 0x7fb83c0af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 -- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb83c06d5c0 msgr2=0x7fb83c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb83c06d5c0 0x7fb83c0b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 -- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb83c0b4970 msgr2=0x7fb83c0af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb83c0b4970 0x7fb83c0af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 -- 172.21.15.122:0/984938044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0afde0 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb83ad9d640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 0x7fb83c0b2f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c7a0 tx=0x7fb83400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834018070 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83400a640 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c0b00c0 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83400aab0 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.110+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c0b0680 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.111+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c0d0230 con 0x7fb83c15bb40 2024-03-06T07:22:51.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.112+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb83401b070 con 0x7fb83c15bb40 2024-03-06T07:22:51.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.112+0000 7fb832ffd640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb81c06b4d0 0x7fb81c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.112+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb8340895d0 con 0x7fb83c15bb40 2024-03-06T07:22:51.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.112+0000 7fb83a59c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb81c06b4d0 0x7fb81c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.113+0000 7fb83a59c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb81c06b4d0 0x7fb81c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb82c006fd0 tx=0x7fb82c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.117+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb83c0d0230 con 0x7fb83c15bb40 2024-03-06T07:22:51.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.288+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb83c0732b0 con 0x7fb83c15bb40 2024-03-06T07:22:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.985+0000 7fb832ffd640 1 -- 172.21.15.122:0/984938044 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb8340586c0 con 0x7fb83c15bb40 2024-03-06T07:22:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:22:51.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.989+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb81c06b4d0 msgr2=0x7fb81c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.989+0000 7fb840f6c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb81c06b4d0 0x7fb81c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb82c006fd0 tx=0x7fb82c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.989+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 msgr2=0x7fb83c0b2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.990+0000 7fb840f6c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 0x7fb83c0b2f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c7a0 tx=0x7fb83400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:51.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.991+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 shutdown_connections 2024-03-06T07:22:51.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.991+0000 7fb840f6c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb81c06b4d0 0x7fb81c06d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.991+0000 7fb840f6c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb83c15bb40 0x7fb83c0b2f80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.991+0000 7fb840f6c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb83c0b4970 0x7fb83c0af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.991+0000 7fb840f6c640 1 --2- 172.21.15.122:0/984938044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb83c06d5c0 0x7fb83c0b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.991+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 >> 172.21.15.122:0/984938044 conn(0x7fb83c06b190 msgr2=0x7fb83c1593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.994+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 shutdown_connections 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:51.994+0000 7fb840f6c640 1 -- 172.21.15.122:0/984938044 wait complete. 2024-03-06T07:22:52.018 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:52.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.121+0000 7f79acdb6640 1 Processor -- start 2024-03-06T07:22:52.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.121+0000 7f79acdb6640 1 -- start start 2024-03-06T07:22:52.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79acdb6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f79a815bc50 0x7f79a80cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79acdb6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80aee20 0x7f79a80cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79acdb6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a806d5b0 con 0x7f79a80cc280 2024-03-06T07:22:52.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79acdb6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a80b4850 con 0x7f79a815bc50 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79acdb6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a80cc7b0 con 0x7f79a80aee20 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79a659c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f79a815bc50 0x7f79a80cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79a6d9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80aee20 0x7f79a80cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79a659c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f79a815bc50 0x7f79a80cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:60646/0 (socket says 172.21.15.122:60646) 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79a659c640 1 -- 172.21.15.122:0/2847298517 learned_addr learned my addr 172.21.15.122:0/2847298517 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.122+0000 7f79a5d9b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f79a80cc280 0x7f79a815f8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:59942/0 (socket says 172.21.15.122:59942) 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.123+0000 7f79a659c640 1 -- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80aee20 msgr2=0x7f79a80cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.123+0000 7f79a659c640 1 --2- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80aee20 0x7f79a80cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.123+0000 7f79a659c640 1 -- 172.21.15.122:0/2847298517 >> v1:172.21.15.169:6789/0 conn(0x7f79a80cc280 legacy=0x7f79a815f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.123+0000 7f79a659c640 1 -- 172.21.15.122:0/2847298517 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a80b4ce0 con 0x7f79a815bc50 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.123+0000 7f79a659c640 1 --2- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a815bc50 0x7f79a80cf430 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7998009e50 tx=0x7f799802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14257b3300e10c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79a559a640 1 -- 172.21.15.122:0/2847298517 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799802ca70 con 0x7f79a815bc50 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79a559a640 1 -- 172.21.15.122:0/2847298517 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799802cc10 con 0x7f79a815bc50 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79a559a640 1 -- 172.21.15.122:0/2847298517 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79980343f0 con 0x7f79a815bc50 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 -- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a815bc50 msgr2=0x7f79a80cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 --2- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a815bc50 0x7f79a80cf430 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7998009e50 tx=0x7f799802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 -- 172.21.15.122:0/2847298517 shutdown_connections 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 --2- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80aee20 0x7f79a80cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 --2- 172.21.15.122:0/2847298517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a815bc50 0x7f79a80cf430 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 -- 172.21.15.122:0/2847298517 >> 172.21.15.122:0/2847298517 conn(0x7f79a806afb0 msgr2=0x7f79a806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 -- 172.21.15.122:0/2847298517 shutdown_connections 2024-03-06T07:22:52.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.124+0000 7f79acdb6640 1 -- 172.21.15.122:0/2847298517 wait complete. 2024-03-06T07:22:52.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.125+0000 7f79acdb6640 1 Processor -- start 2024-03-06T07:22:52.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 -- start start 2024-03-06T07:22:52.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a80aee20 0x7f79a80add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80cc280 0x7f79a81f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 0x7f79a815f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a815c030 con 0x7f79a815bc50 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a80ae900 con 0x7f79a80aee20 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79acdb6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a815c1b0 con 0x7f79a80cc280 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79a6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 0x7f79a815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79a6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 0x7f79a815f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41196/0 (socket says 172.21.15.122:41196) 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79a6d9d640 1 -- 172.21.15.122:0/24839413 learned_addr learned my addr 172.21.15.122:0/24839413 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79a659c640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a80aee20 0x7f79a80add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.126+0000 7f79a5d9b640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80cc280 0x7f79a81f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a6d9d640 1 -- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80cc280 msgr2=0x7f79a81f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a6d9d640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80cc280 0x7f79a81f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a6d9d640 1 -- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a80aee20 msgr2=0x7f79a80add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a6d9d640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a80aee20 0x7f79a80add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a6d9d640 1 -- 172.21.15.122:0/24839413 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a81f5650 con 0x7f79a815bc50 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a659c640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a80aee20 0x7f79a80add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79a6d9d640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 0x7f79a815f860 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f79a000e7f0 tx=0x7f79a000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0017070 con 0x7f79a815bc50 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a000ee40 con 0x7f79a815bc50 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a81f5930 con 0x7f79a815bc50 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a000cb80 con 0x7f79a815bc50 2024-03-06T07:22:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.127+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a8144230 con 0x7f79a815bc50 2024-03-06T07:22:52.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.128+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a804f330 con 0x7f79a815bc50 2024-03-06T07:22:52.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.129+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79a001a070 con 0x7f79a815bc50 2024-03-06T07:22:52.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.129+0000 7f7996ffd640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798806b500 0x7f798806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.129+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f79a008a2f0 con 0x7f79a815bc50 2024-03-06T07:22:52.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.129+0000 7f79a659c640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798806b500 0x7f798806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.130+0000 7f79a659c640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798806b500 0x7f798806d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7998035000 tx=0x7f7998030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.134+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79a0058b80 con 0x7f79a815bc50 2024-03-06T07:22:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.340+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f79a8074930 con 0x7f79a815bc50 2024-03-06T07:22:52.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.987+0000 7f7996ffd640 1 -- 172.21.15.122:0/24839413 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f79a005cc20 con 0x7f79a815bc50 2024-03-06T07:22:52.989 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:22:52.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.994+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798806b500 msgr2=0x7f798806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.994+0000 7f79acdb6640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798806b500 0x7f798806d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7998035000 tx=0x7f7998030000 comp rx=0 tx=0).stop 2024-03-06T07:22:52.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.994+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 msgr2=0x7f79a815f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.994+0000 7f79acdb6640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 0x7f79a815f860 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f79a000e7f0 tx=0x7f79a000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 shutdown_connections 2024-03-06T07:22:52.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798806b500 0x7f798806d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a815bc50 0x7f79a815f860 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a80cc280 0x7f79a81f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 --2- 172.21.15.122:0/24839413 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a80aee20 0x7f79a80add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 >> 172.21.15.122:0/24839413 conn(0x7f79a806afb0 msgr2=0x7f79a80ce580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.995+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 shutdown_connections 2024-03-06T07:22:52.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:52.996+0000 7f79acdb6640 1 -- 172.21.15.122:0/24839413 wait complete. 2024-03-06T07:22:53.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:53.072 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:53.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:53.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.132+0000 7fc59d6fb640 1 Processor -- start 2024-03-06T07:22:53.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.133+0000 7fc59d6fb640 1 -- start start 2024-03-06T07:22:53.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.133+0000 7fc59d6fb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc5980d00c0 0x7fc5980d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.133+0000 7fc59d6fb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc59806cf20 0x7fc59806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.133+0000 7fc59d6fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5980d09e0 con 0x7fc5980cf850 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.133+0000 7fc59d6fb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59806ca90 con 0x7fc5980d00c0 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.133+0000 7fc59d6fb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59815e380 con 0x7fc59806cf20 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5967fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc5980cf850 0x7fc5980cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59612/0 (socket says 172.21.15.122:59612) 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5967fc640 1 -- 172.21.15.122:0/1726446078 learned_addr learned my addr 172.21.15.122:0/1726446078 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc596ffd640 1 --2- 172.21.15.122:0/1726446078 >> v2:172.21.15.169:3300/0 conn(0x7fc5980d00c0 0x7fc5980d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5977fe640 1 --2- 172.21.15.122:0/1726446078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc59806cf20 0x7fc59806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc595ffb640 1 -- 172.21.15.122:0/1726446078 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175625266 0 0) 0x7fc5980d09e0 con 0x7fc5980cf850 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc595ffb640 1 -- 172.21.15.122:0/1726446078 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc584003520 con 0x7fc5980cf850 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5977fe640 1 -- 172.21.15.122:0/1726446078 >> v2:172.21.15.169:3300/0 conn(0x7fc5980d00c0 msgr2=0x7fc5980d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5977fe640 1 --2- 172.21.15.122:0/1726446078 >> v2:172.21.15.169:3300/0 conn(0x7fc5980d00c0 0x7fc5980d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5977fe640 1 -- 172.21.15.122:0/1726446078 >> v1:172.21.15.122:6789/0 conn(0x7fc5980cf850 legacy=0x7fc5980cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc5977fe640 1 -- 172.21.15.122:0/1726446078 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980cdc10 con 0x7fc59806cf20 2024-03-06T07:22:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.134+0000 7fc596ffd640 1 --2- 172.21.15.122:0/1726446078 >> v2:172.21.15.169:3300/0 conn(0x7fc5980d00c0 0x7fc5980d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc5977fe640 1 --2- 172.21.15.122:0/1726446078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc59806cf20 0x7fc59806d3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc590009d90 tx=0x7fc59002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f7d6c2be89c7cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc595ffb640 1 -- 172.21.15.122:0/1726446078 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590039070 con 0x7fc59806cf20 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc595ffb640 1 -- 172.21.15.122:0/1726446078 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59002cc50 con 0x7fc59806cf20 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc595ffb640 1 -- 172.21.15.122:0/1726446078 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590034440 con 0x7fc59806cf20 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1726446078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc59806cf20 msgr2=0x7fc59806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1726446078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc59806cf20 0x7fc59806d3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc590009d90 tx=0x7fc59002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1726446078 shutdown_connections 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1726446078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc59806cf20 0x7fc59806d3b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1726446078 >> v2:172.21.15.169:3300/0 conn(0x7fc5980d00c0 0x7fc5980d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1726446078 >> 172.21.15.122:0/1726446078 conn(0x7fc5980b35a0 msgr2=0x7fc5980b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1726446078 shutdown_connections 2024-03-06T07:22:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.135+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1726446078 wait complete. 2024-03-06T07:22:53.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 Processor -- start 2024-03-06T07:22:53.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 -- start start 2024-03-06T07:22:53.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc59806cf20 0x7fc5980afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 0x7fc5980b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5980d00c0 0x7fc5981f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806c720 con 0x7fc5980cf850 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806c440 con 0x7fc59806cf20 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.137+0000 7fc59d6fb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5980b1030 con 0x7fc5980d00c0 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 0x7fc5980b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 0x7fc5980b0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41200/0 (socket says 172.21.15.122:41200) 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 -- 172.21.15.122:0/1043482786 learned_addr learned my addr 172.21.15.122:0/1043482786 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc596ffd640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc59806cf20 0x7fc5980afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5977fe640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5980d00c0 0x7fc5981f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 -- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5980d00c0 msgr2=0x7fc5981f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5980d00c0 0x7fc5981f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 -- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc59806cf20 msgr2=0x7fc5980afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc59806cf20 0x7fc5980afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 -- 172.21.15.122:0/1043482786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5981f49a0 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc5967fc640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 0x7fc5980b0340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc58800c460 tx=0x7fc58800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588017070 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc588005ce0 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5980c8be0 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588007290 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.138+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5980c91a0 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.139+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc598068150 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.140+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc58801a070 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.140+0000 7fc58f7fe640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc57806b4d0 0x7fc57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.140+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5880894e0 con 0x7fc5980cf850 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.140+0000 7fc596ffd640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc57806b4d0 0x7fc57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.140+0000 7fc596ffd640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc57806b4d0 0x7fc57806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc580009560 tx=0x7fc580009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.144+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc598068150 con 0x7fc5980cf850 2024-03-06T07:22:53.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.318+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc59804f330 con 0x7fc5980cf850 2024-03-06T07:22:53.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.319+0000 7fc58f7fe640 1 -- 172.21.15.122:0/1043482786 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16682 (secure 0 0 0) 0x7fc588057c90 con 0x7fc5980cf850 2024-03-06T07:22:53.321 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:53.321 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":32,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:52.341872+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.320+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc57806b4d0 msgr2=0x7fc57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.320+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc57806b4d0 0x7fc57806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc580009560 tx=0x7fc580009190 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.320+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 msgr2=0x7fc5980b0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.320+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 0x7fc5980b0340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc58800c460 tx=0x7fc58800c920 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 shutdown_connections 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc57806b4d0 0x7fc57806d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5980d00c0 0x7fc5981f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5980cf850 0x7fc5980b0340 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 --2- 172.21.15.122:0/1043482786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc59806cf20 0x7fc5980afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 >> 172.21.15.122:0/1043482786 conn(0x7fc5980b35a0 msgr2=0x7fc5980cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 shutdown_connections 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.321+0000 7fc59d6fb640 1 -- 172.21.15.122:0/1043482786 wait complete. 2024-03-06T07:22:53.334 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:53.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.431+0000 7f2014313640 1 Processor -- start 2024-03-06T07:22:53.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.431+0000 7f2014313640 1 -- start start 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f2014313640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f200c1424f0 0x7f200c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f2014313640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c0cc980 con 0x7f200c0cd7a0 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f2014313640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c0ccc60 con 0x7f200c06da40 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f2014313640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c0cd220 con 0x7f200c1424f0 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20120af640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f200c1424f0 0x7f200c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20118ae640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f200c0cd7a0 0x7f200c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59620/0 (socket says 172.21.15.122:59620) 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20118ae640 1 -- 172.21.15.122:0/83412568 learned_addr learned my addr 172.21.15.122:0/83412568 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20120af640 1 -- 172.21.15.122:0/83412568 >> v1:172.21.15.169:6789/0 conn(0x7f200c06da40 legacy=0x7f200c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20120af640 1 -- 172.21.15.122:0/83412568 >> v1:172.21.15.122:6789/0 conn(0x7f200c0cd7a0 legacy=0x7f200c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20120af640 1 -- 172.21.15.122:0/83412568 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c0ccf40 con 0x7f200c1424f0 2024-03-06T07:22:53.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.432+0000 7f20110ad640 1 -- 172.21.15.122:0/83412568 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234624132 0 0) 0x7f200c0cc980 con 0x7f200c0cd7a0 2024-03-06T07:22:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.433+0000 7f20120af640 1 --2- 172.21.15.122:0/83412568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c0cc2f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f20080097a0 tx=0x7f200802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d05d4ce3fa18bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.433+0000 7f20110ad640 1 -- 172.21.15.122:0/83412568 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008035020 con 0x7f200c1424f0 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.433+0000 7f20110ad640 1 -- 172.21.15.122:0/83412568 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2008031830 con 0x7f200c1424f0 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.433+0000 7f20110ad640 1 -- 172.21.15.122:0/83412568 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008031c00 con 0x7f200c1424f0 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.433+0000 7f2014313640 1 -- 172.21.15.122:0/83412568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 msgr2=0x7f200c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.433+0000 7f2014313640 1 --2- 172.21.15.122:0/83412568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c0cc2f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f20080097a0 tx=0x7f200802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.434+0000 7f2014313640 1 -- 172.21.15.122:0/83412568 shutdown_connections 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.434+0000 7f2014313640 1 --2- 172.21.15.122:0/83412568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c0cc2f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.434+0000 7f2014313640 1 -- 172.21.15.122:0/83412568 >> 172.21.15.122:0/83412568 conn(0x7f200c0cf8a0 msgr2=0x7f200c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.434+0000 7f2014313640 1 -- 172.21.15.122:0/83412568 shutdown_connections 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.434+0000 7f2014313640 1 -- 172.21.15.122:0/83412568 wait complete. 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.435+0000 7f2014313640 1 Processor -- start 2024-03-06T07:22:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.435+0000 7f2014313640 1 -- start start 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.435+0000 7f2014313640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f200c06da40 0x7f200c0b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.435+0000 7f2014313640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f200c0cd7a0 0x7f200c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.435+0000 7f2014313640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c1f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.435+0000 7f2014313640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c15d070 con 0x7f200c1424f0 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2014313640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c15cef0 con 0x7f200c0cd7a0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2014313640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c15d1f0 con 0x7f200c06da40 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c1f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c1f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41216/0 (socket says 172.21.15.122:41216) 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 -- 172.21.15.122:0/1209354988 learned_addr learned my addr 172.21.15.122:0/1209354988 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20118ae640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f200c0cd7a0 0x7f200c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20120af640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f200c06da40 0x7f200c0b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 -- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f200c06da40 msgr2=0x7f200c0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f200c06da40 0x7f200c0b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 -- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f200c0cd7a0 msgr2=0x7f200c0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f200c0cd7a0 0x7f200c0b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 -- 172.21.15.122:0/1209354988 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c15a150 con 0x7f200c1424f0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f20128b0640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c1f3d30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f200400b930 tx=0x7f200400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004019070 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200400cb60 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200c0b3e70 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004014440 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.436+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200c0b4430 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.437+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c04f330 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.438+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f200401c070 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.438+0000 7f2002ffd640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ff006b4d0 0x7f1ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.438+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2004089cc0 con 0x7f200c1424f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.438+0000 7f20120af640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ff006b4d0 0x7f1ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.438+0000 7f20120af640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ff006b4d0 0x7f1ff006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2008036000 tx=0x7f2008008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.442+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2004058580 con 0x7f200c1424f0 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.615+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f200c0d0620 con 0x7f200c1424f0 2024-03-06T07:22:53.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.616+0000 7f2002ffd640 1 -- 172.21.15.122:0/1209354988 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16682 (secure 0 0 0) 0x7f200405c620 con 0x7f200c1424f0 2024-03-06T07:22:53.619 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:53.619 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":32,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:52.341872+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:53.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ff006b4d0 msgr2=0x7f1ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ff006b4d0 0x7f1ff006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2008036000 tx=0x7f2008008030 comp rx=0 tx=0).stop 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 msgr2=0x7f200c1f3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c1f3d30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f200400b930 tx=0x7f200400bdf0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 shutdown_connections 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ff006b4d0 0x7f1ff006d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f200c1424f0 0x7f200c1f3d30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f200c0cd7a0 0x7f200c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 --2- 172.21.15.122:0/1209354988 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f200c06da40 0x7f200c0b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 >> 172.21.15.122:0/1209354988 conn(0x7f200c0cf8a0 msgr2=0x7f200c15abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 shutdown_connections 2024-03-06T07:22:53.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.618+0000 7f2014313640 1 -- 172.21.15.122:0/1209354988 wait complete. 2024-03-06T07:22:53.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:53.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.733+0000 7f2f3e04e640 1 Processor -- start 2024-03-06T07:22:53.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.733+0000 7f2f3e04e640 1 -- start start 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f3e04e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f381614d0 0x7f2f380d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f3e04e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2f380d2480 0x7f2f380b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f3e04e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f381618b0 con 0x7f2f380d1910 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f3e04e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f380cf400 con 0x7f2f380d2480 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f3e04e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380cf6a0 con 0x7f2f381614d0 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2f380d2480 0x7f2f380b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2f380d2480 0x7f2f380b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41222/0 (socket says 172.21.15.122:41222) 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 -- 172.21.15.122:0/4008919271 learned_addr learned my addr 172.21.15.122:0/4008919271 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 -- 172.21.15.122:0/4008919271 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f381614d0 msgr2=0x7f2f380d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 --2- 172.21.15.122:0/4008919271 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f381614d0 0x7f2f380d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 -- 172.21.15.122:0/4008919271 >> v1:172.21.15.169:6789/0 conn(0x7f2f380d1910 legacy=0x7f2f380d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.734+0000 7f2f37fff640 1 -- 172.21.15.122:0/4008919271 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f380cf980 con 0x7f2f380d2480 2024-03-06T07:22:53.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.735+0000 7f2f37fff640 1 --2- 172.21.15.122:0/4008919271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f380d2480 0x7f2f380b0e30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009e50 tx=0x7f2f3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6223522c7143f1a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.735+0000 7f2f36ffd640 1 -- 172.21.15.122:0/4008919271 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3002ca70 con 0x7f2f380d2480 2024-03-06T07:22:53.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.735+0000 7f2f36ffd640 1 -- 172.21.15.122:0/4008919271 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3002cc10 con 0x7f2f380d2480 2024-03-06T07:22:53.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.735+0000 7f2f36ffd640 1 -- 172.21.15.122:0/4008919271 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f300343f0 con 0x7f2f380d2480 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.735+0000 7f2f3e04e640 1 -- 172.21.15.122:0/4008919271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f380d2480 msgr2=0x7f2f380b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.735+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/4008919271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f380d2480 0x7f2f380b0e30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009e50 tx=0x7f2f3002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.736+0000 7f2f3e04e640 1 -- 172.21.15.122:0/4008919271 shutdown_connections 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.736+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/4008919271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f380d2480 0x7f2f380b0e30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.736+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/4008919271 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f381614d0 0x7f2f380d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.736+0000 7f2f3e04e640 1 -- 172.21.15.122:0/4008919271 >> 172.21.15.122:0/4008919271 conn(0x7f2f380d3d40 msgr2=0x7f2f380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.736+0000 7f2f3e04e640 1 -- 172.21.15.122:0/4008919271 shutdown_connections 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.736+0000 7f2f3e04e640 1 -- 172.21.15.122:0/4008919271 wait complete. 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 Processor -- start 2024-03-06T07:22:53.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 -- start start 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f380d1910 0x7f2f38158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f380d2480 0x7f2f380cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 0x7f2f380ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380b1370 con 0x7f2f381614d0 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380b0910 con 0x7f2f380d1910 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.737+0000 7f2f3e04e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380b14f0 con 0x7f2f380d2480 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 0x7f2f380ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 0x7f2f380ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41232/0 (socket says 172.21.15.122:41232) 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 -- 172.21.15.122:0/800990255 learned_addr learned my addr 172.21.15.122:0/800990255 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:53.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f377fe640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f380d1910 0x7f2f38158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f2ffff640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f380d2480 0x7f2f380cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 -- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f380d2480 msgr2=0x7f2f380cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f380d2480 0x7f2f380cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 -- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f380d1910 msgr2=0x7f2f38158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f380d1910 0x7f2f38158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 -- 172.21.15.122:0/800990255 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f380b6a70 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f37fff640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 0x7f2f380ccdb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009f70 tx=0x7f2f30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30043020 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3003e450 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30009510 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f380b6d50 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.738+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f380b7310 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.739+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f38068ff0 con 0x7f2f381614d0 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.740+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f30046020 con 0x7f2f381614d0 2024-03-06T07:22:53.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.740+0000 7f2f34ff9640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f1806b4d0 0x7f2f1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.740+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f300acb40 con 0x7f2f381614d0 2024-03-06T07:22:53.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.740+0000 7f2f377fe640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f1806b4d0 0x7f2f1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.740+0000 7f2f377fe640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f1806b4d0 0x7f2f1806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2f380c8c40 tx=0x7f2f28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.745+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f3004d030 con 0x7f2f381614d0 2024-03-06T07:22:53.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:53.955+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2f380d4b50 con 0x7f2f381614d0 2024-03-06T07:22:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.935+0000 7f2f34ff9640 1 -- 172.21.15.122:0/800990255 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2f3007b9c0 con 0x7f2f381614d0 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f1806b4d0 msgr2=0x7f2f1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f1806b4d0 0x7f2f1806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2f380c8c40 tx=0x7f2f28008040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 msgr2=0x7f2f380ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 0x7f2f380ccdb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009f70 tx=0x7f2f30004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 shutdown_connections 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2f1806b4d0 0x7f2f1806d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f381614d0 0x7f2f380ccdb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2f380d2480 0x7f2f380cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 --2- 172.21.15.122:0/800990255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f380d1910 0x7f2f38158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 >> 172.21.15.122:0/800990255 conn(0x7f2f380d3d40 msgr2=0x7f2f3815ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 shutdown_connections 2024-03-06T07:22:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:54.938+0000 7f2f3e04e640 1 -- 172.21.15.122:0/800990255 wait complete. 2024-03-06T07:22:54.950 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:54.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:55.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 Processor -- start 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 -- start start 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa1141614d0 0x7fa1140d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa1140d1910 0x7fa1140d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1140d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1140d2750 con 0x7fa1141614d0 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1141618b0 con 0x7fa1140d1910 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.055+0000 7fa11b415640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140cf400 con 0x7fa1140d22e0 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa1141614d0 0x7fa1140d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa1141614d0 0x7fa1140d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38398/0 (socket says 172.21.15.122:38398) 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 -- 172.21.15.122:0/3342676684 learned_addr learned my addr 172.21.15.122:0/3342676684 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:55.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1189b0640 1 --2- 172.21.15.122:0/3342676684 >> v2:172.21.15.169:3300/0 conn(0x7fa1140d1910 0x7fa1140d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1199b2640 1 --2- 172.21.15.122:0/3342676684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1140d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 -- 172.21.15.122:0/3342676684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 msgr2=0x7fa1140d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 --2- 172.21.15.122:0/3342676684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1140d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 -- 172.21.15.122:0/3342676684 >> v2:172.21.15.169:3300/0 conn(0x7fa1140d1910 msgr2=0x7fa1140d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 --2- 172.21.15.122:0/3342676684 >> v2:172.21.15.169:3300/0 conn(0x7fa1140d1910 0x7fa1140d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.056+0000 7fa1191b1640 1 -- 172.21.15.122:0/3342676684 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140cf6a0 con 0x7fa1141614d0 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa1191b1640 1 --2- 172.21.15.122:0/3342676684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1141614d0 0x7fa1140d4860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ce550 tx=0x7fa11002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635c65b980c50249 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa10bfff640 1 -- 172.21.15.122:0/3342676684 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110035020 con 0x7fa1141614d0 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa10bfff640 1 -- 172.21.15.122:0/3342676684 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110031870 con 0x7fa1141614d0 2024-03-06T07:22:55.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa10bfff640 1 -- 172.21.15.122:0/3342676684 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110031c40 con 0x7fa1141614d0 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 -- 172.21.15.122:0/3342676684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1141614d0 msgr2=0x7fa1140d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 --2- 172.21.15.122:0/3342676684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1141614d0 0x7fa1140d4860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ce550 tx=0x7fa11002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 -- 172.21.15.122:0/3342676684 shutdown_connections 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 --2- 172.21.15.122:0/3342676684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1140d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 --2- 172.21.15.122:0/3342676684 >> v2:172.21.15.169:3300/0 conn(0x7fa1140d1910 0x7fa1140d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 --2- 172.21.15.122:0/3342676684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1141614d0 0x7fa1140d4860 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 -- 172.21.15.122:0/3342676684 >> 172.21.15.122:0/3342676684 conn(0x7fa1140d3d40 msgr2=0x7fa1140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 -- 172.21.15.122:0/3342676684 shutdown_connections 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.057+0000 7fa11b415640 1 -- 172.21.15.122:0/3342676684 wait complete. 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.059+0000 7fa11b415640 1 Processor -- start 2024-03-06T07:22:55.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.059+0000 7fa11b415640 1 -- start start 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.060+0000 7fa11b415640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 0x7fa1140b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.060+0000 7fa11b415640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1141f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.060+0000 7fa11b415640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1141614d0 0x7fa114152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.060+0000 7fa11b415640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140b0910 con 0x7fa1140d1910 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.060+0000 7fa11b415640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140cfc60 con 0x7fa1141614d0 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.060+0000 7fa11b415640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140d0be0 con 0x7fa1140d22e0 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 0x7fa1140b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 0x7fa1140b4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38406/0 (socket says 172.21.15.122:38406) 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 -- 172.21.15.122:0/1638106321 learned_addr learned my addr 172.21.15.122:0/1638106321 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1199b2640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1141614d0 0x7fa114152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1189b0640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1141f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 -- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 msgr2=0x7fa1141f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1141f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 -- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1141614d0 msgr2=0x7fa114152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1141614d0 0x7fa114152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 -- 172.21.15.122:0/1638106321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1141f49c0 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa1191b1640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 0x7fa1140b4fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa110000c00 tx=0x7fa1100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110035040 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110045480 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140b3f10 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110045870 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.061+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1140b44d0 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.063+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa114068ff0 con 0x7fa1140d1910 2024-03-06T07:22:55.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.063+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa110038020 con 0x7fa1140d1910 2024-03-06T07:22:55.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.063+0000 7fa109ffb640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0f806b500 0x7fa0f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.063+0000 7fa1189b0640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0f806b500 0x7fa0f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.063+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1100ac610 con 0x7fa1140d1910 2024-03-06T07:22:55.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.064+0000 7fa1189b0640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0f806b500 0x7fa0f806d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1140d49a0 tx=0x7fa104008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.068+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa11007b520 con 0x7fa1140d1910 2024-03-06T07:22:55.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:55.378+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa114152690 con 0x7fa1140d1910 2024-03-06T07:22:56.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.007+0000 7fa109ffb640 1 -- 172.21.15.122:0/1638106321 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa11007ae20 con 0x7fa1140d1910 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.009+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0f806b500 msgr2=0x7fa0f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0f806b500 0x7fa0f806d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1140d49a0 tx=0x7fa104008040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 msgr2=0x7fa1140b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 0x7fa1140b4fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa110000c00 tx=0x7fa1100046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 shutdown_connections 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0f806b500 0x7fa0f806d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1141614d0 0x7fa114152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa1140d22e0 0x7fa1141f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 --2- 172.21.15.122:0/1638106321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1140d1910 0x7fa1140b4fb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 >> 172.21.15.122:0/1638106321 conn(0x7fa1140d3d40 msgr2=0x7fa1140cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 shutdown_connections 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.010+0000 7fa11b415640 1 -- 172.21.15.122:0/1638106321 wait complete. 2024-03-06T07:22:56.022 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:56.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:56.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.122+0000 7f8f7c803640 1 Processor -- start 2024-03-06T07:22:56.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.122+0000 7f8f7c803640 1 -- start start 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7c803640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f78152bc0 0x7f8f780b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7c803640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8f7815cb60 0x7f8f780b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7c803640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780d2780 con 0x7f8f780b7370 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7c803640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f78152fa0 con 0x7f8f78152bc0 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7c803640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f780d16e0 con 0x7f8f7815cb60 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f78152bc0 0x7f8f780b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f78152bc0 0x7f8f780b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38418/0 (socket says 172.21.15.122:38418) 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 -- 172.21.15.122:0/1227012873 learned_addr learned my addr 172.21.15.122:0/1227012873 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7759e640 1 --2- 172.21.15.122:0/1227012873 >> v2:172.21.15.169:3300/0 conn(0x7f8f7815cb60 0x7f8f780b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 -- 172.21.15.122:0/1227012873 >> v2:172.21.15.169:3300/0 conn(0x7f8f7815cb60 msgr2=0x7f8f780b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 --2- 172.21.15.122:0/1227012873 >> v2:172.21.15.169:3300/0 conn(0x7f8f7815cb60 0x7f8f780b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 -- 172.21.15.122:0/1227012873 >> v1:172.21.15.169:6789/0 conn(0x7f8f780b7370 legacy=0x7f8f7815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f76d9d640 1 -- 172.21.15.122:0/1227012873 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780d19c0 con 0x7f8f78152bc0 2024-03-06T07:22:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.123+0000 7f8f7759e640 1 --2- 172.21.15.122:0/1227012873 >> v2:172.21.15.169:3300/0 conn(0x7f8f7815cb60 0x7f8f780b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.124+0000 7f8f76d9d640 1 --2- 172.21.15.122:0/1227012873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f78152bc0 0x7f8f780b6e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009870 tx=0x7f8f6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0ff09610bdfc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f75d9b640 1 -- 172.21.15.122:0/1227012873 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c035020 con 0x7f8f78152bc0 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f75d9b640 1 -- 172.21.15.122:0/1227012873 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c031870 con 0x7f8f78152bc0 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f75d9b640 1 -- 172.21.15.122:0/1227012873 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c031c40 con 0x7f8f78152bc0 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 -- 172.21.15.122:0/1227012873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f78152bc0 msgr2=0x7f8f780b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 --2- 172.21.15.122:0/1227012873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f78152bc0 0x7f8f780b6e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009870 tx=0x7f8f6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:56.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 -- 172.21.15.122:0/1227012873 shutdown_connections 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 --2- 172.21.15.122:0/1227012873 >> v2:172.21.15.169:3300/0 conn(0x7f8f7815cb60 0x7f8f780b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 --2- 172.21.15.122:0/1227012873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f78152bc0 0x7f8f780b6e30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 -- 172.21.15.122:0/1227012873 >> 172.21.15.122:0/1227012873 conn(0x7f8f7809a9f0 msgr2=0x7f8f7809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 -- 172.21.15.122:0/1227012873 shutdown_connections 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.125+0000 7f8f7c803640 1 -- 172.21.15.122:0/1227012873 wait complete. 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 Processor -- start 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 -- start start 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f780b7370 0x7f8f781e7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f78152bc0 0x7f8f780ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 0x7f8f780ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780d2260 con 0x7f8f7815cb60 2024-03-06T07:22:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780d1f80 con 0x7f8f78152bc0 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.127+0000 7f8f7c803640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7815cfb0 con 0x7f8f780b7370 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 0x7f8f780ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 0x7f8f780ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38422/0 (socket says 172.21.15.122:38422) 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 -- 172.21.15.122:0/3617576146 learned_addr learned my addr 172.21.15.122:0/3617576146 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7659c640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f78152bc0 0x7f8f780ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f76d9d640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f780b7370 0x7f8f781e7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 -- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f780b7370 msgr2=0x7f8f781e7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f780b7370 0x7f8f781e7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 -- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f78152bc0 msgr2=0x7f8f780ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f78152bc0 0x7f8f780ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 -- 172.21.15.122:0/3617576146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7815d3d0 con 0x7f8f7815cb60 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7659c640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f78152bc0 0x7f8f780ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7759e640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 0x7f8f780ce8d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000cef0 tx=0x7f8f7000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7001a070 con 0x7f8f7815cb60 2024-03-06T07:22:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7000d860 con 0x7f8f7815cb60 2024-03-06T07:22:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7815d6b0 con 0x7f8f7815cb60 2024-03-06T07:22:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7000dc50 con 0x7f8f7815cb60 2024-03-06T07:22:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.128+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f781e81c0 con 0x7f8f7815cb60 2024-03-06T07:22:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.129+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78068f80 con 0x7f8f7815cb60 2024-03-06T07:22:56.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.130+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f7001d070 con 0x7f8f7815cb60 2024-03-06T07:22:56.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.130+0000 7f8f5f7fe640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f6406b4d0 0x7f8f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.130+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f7008ab40 con 0x7f8f7815cb60 2024-03-06T07:22:56.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.130+0000 7f8f76d9d640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f6406b4d0 0x7f8f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.130+0000 7f8f76d9d640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f6406b4d0 0x7f8f6406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c036000 tx=0x7f8f6c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.134+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f70059a00 con 0x7f8f7815cb60 2024-03-06T07:22:56.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.346+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f7815bb00 con 0x7f8f7815cb60 2024-03-06T07:22:56.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.347+0000 7f8f5f7fe640 1 -- 172.21.15.122:0/3617576146 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f8f70059300 con 0x7f8f7815cb60 2024-03-06T07:22:56.349 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:56.349 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:22:56.349 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:22:56.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f6406b4d0 msgr2=0x7f8f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f6406b4d0 0x7f8f6406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c036000 tx=0x7f8f6c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 msgr2=0x7f8f780ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 0x7f8f780ce8d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000cef0 tx=0x7f8f7000a710 comp rx=0 tx=0).stop 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 shutdown_connections 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f6406b4d0 0x7f8f6406d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f7815cb60 0x7f8f780ce8d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f78152bc0 0x7f8f780ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 --2- 172.21.15.122:0/3617576146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f780b7370 0x7f8f781e7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 >> 172.21.15.122:0/3617576146 conn(0x7f8f7809a9f0 msgr2=0x7f8f7809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.349+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 shutdown_connections 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.350+0000 7f8f7c803640 1 -- 172.21.15.122:0/3617576146 wait complete. 2024-03-06T07:22:56.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:56.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.474+0000 7f864ce2a640 1 Processor -- start 2024-03-06T07:22:56.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.475+0000 7f864ce2a640 1 -- start start 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.475+0000 7f864ce2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.475+0000 7f864ce2a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86480b3430 con 0x7f86480d29c0 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.475+0000 7f864ce2a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86480b3710 con 0x7f86480cf830 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.475+0000 7f864ce2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86480b31e0 con 0x7f864815dfe0 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f8646d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f86480d29c0 0x7f86480b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56600/0 (socket says 172.21.15.122:56600) 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f8646d9d640 1 -- 172.21.15.122:0/2715388622 learned_addr learned my addr 172.21.15.122:0/2715388622 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:56.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864659c640 1 --2- 172.21.15.122:0/2715388622 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864559a640 1 -- 172.21.15.122:0/2715388622 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332228466 0 0) 0x7f86480b3430 con 0x7f86480d29c0 2024-03-06T07:22:56.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864559a640 1 -- 172.21.15.122:0/2715388622 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630003520 con 0x7f86480d29c0 2024-03-06T07:22:56.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864559a640 1 -- 172.21.15.122:0/2715388622 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432148089 0 0) 0x7f86480b3710 con 0x7f86480cf830 2024-03-06T07:22:56.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864659c640 1 -- 172.21.15.122:0/2715388622 >> v1:172.21.15.169:6789/0 conn(0x7f86480cf830 legacy=0x7f86480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864659c640 1 -- 172.21.15.122:0/2715388622 >> v1:172.21.15.122:6789/0 conn(0x7f86480d29c0 legacy=0x7f86480b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.476+0000 7f864659c640 1 -- 172.21.15.122:0/2715388622 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86480b39f0 con 0x7f864815dfe0 2024-03-06T07:22:56.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.477+0000 7f864659c640 1 --2- 172.21.15.122:0/2715388622 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86480cf2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86400097d0 tx=0x7f864002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae298bd99aa72b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.477+0000 7f864559a640 1 -- 172.21.15.122:0/2715388622 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640035020 con 0x7f864815dfe0 2024-03-06T07:22:56.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.477+0000 7f864559a640 1 -- 172.21.15.122:0/2715388622 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640031890 con 0x7f864815dfe0 2024-03-06T07:22:56.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.477+0000 7f864559a640 1 -- 172.21.15.122:0/2715388622 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640031c40 con 0x7f864815dfe0 2024-03-06T07:22:56.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 -- 172.21.15.122:0/2715388622 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 msgr2=0x7f86480cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2715388622 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86480cf2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86400097d0 tx=0x7f864002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 -- 172.21.15.122:0/2715388622 shutdown_connections 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2715388622 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86480cf2f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 -- 172.21.15.122:0/2715388622 >> 172.21.15.122:0/2715388622 conn(0x7f86480d3cd0 msgr2=0x7f86480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 -- 172.21.15.122:0/2715388622 shutdown_connections 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.478+0000 7f864ce2a640 1 -- 172.21.15.122:0/2715388622 wait complete. 2024-03-06T07:22:56.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.480+0000 7f864ce2a640 1 Processor -- start 2024-03-06T07:22:56.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.480+0000 7f864ce2a640 1 -- start start 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864ce2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86480cf830 0x7f86481520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864ce2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 0x7f86480b9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864ce2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86481f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864ce2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864815e3c0 con 0x7f86480d29c0 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864ce2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86480b3d60 con 0x7f86480cf830 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864ce2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864815e540 con 0x7f864815dfe0 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f8645d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 0x7f86480b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f8645d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 0x7f86480b9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38426/0 (socket says 172.21.15.122:38426) 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f8645d9b640 1 -- 172.21.15.122:0/2838543965 learned_addr learned my addr 172.21.15.122:0/2838543965 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f864659c640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86480cf830 0x7f86481520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.481+0000 7f8646d9d640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86481f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8645d9b640 1 -- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 msgr2=0x7f86481f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8645d9b640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86481f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8645d9b640 1 -- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86480cf830 msgr2=0x7f86481520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8645d9b640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86480cf830 0x7f86481520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8645d9b640 1 -- 172.21.15.122:0/2838543965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86480b6b70 con 0x7f86480d29c0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8646d9d640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86481f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8645d9b640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 0x7f86480b9570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f863c00c550 tx=0x7f863c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c017070 con 0x7f86480d29c0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c0035f0 con 0x7f86480d29c0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864815fcf0 con 0x7f86480d29c0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c0039e0 con 0x7f86480d29c0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.482+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86481f4130 con 0x7f86480d29c0 2024-03-06T07:22:56.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.483+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86480ceaa0 con 0x7f86480d29c0 2024-03-06T07:22:56.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.484+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f863c01a070 con 0x7f86480d29c0 2024-03-06T07:22:56.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.484+0000 7f8636ffd640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862806b4d0 0x7f862806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.484+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f863c089500 con 0x7f86480d29c0 2024-03-06T07:22:56.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.484+0000 7f864659c640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862806b4d0 0x7f862806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.485+0000 7f864659c640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862806b4d0 0x7f862806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8640036000 tx=0x7f864003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.492+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86480ceaa0 con 0x7f86480d29c0 2024-03-06T07:22:56.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.670+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f864804f330 con 0x7f86480d29c0 2024-03-06T07:22:56.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.671+0000 7f8636ffd640 1 -- 172.21.15.122:0/2838543965 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f863c057dc0 con 0x7f86480d29c0 2024-03-06T07:22:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:56.673 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:53.997648+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:56.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862806b4d0 msgr2=0x7f862806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862806b4d0 0x7f862806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8640036000 tx=0x7f864003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 msgr2=0x7f86480b9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 0x7f86480b9570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f863c00c550 tx=0x7f863c00ca10 comp rx=0 tx=0).stop 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 shutdown_connections 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862806b4d0 0x7f862806d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f864815dfe0 0x7f86481f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86480d29c0 0x7f86480b9570 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 --2- 172.21.15.122:0/2838543965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86480cf830 0x7f86481520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 >> 172.21.15.122:0/2838543965 conn(0x7f86480d3cd0 msgr2=0x7f86480b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 shutdown_connections 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.673+0000 7f864ce2a640 1 -- 172.21.15.122:0/2838543965 wait complete. 2024-03-06T07:22:56.686 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:56.687 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:56.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.785+0000 7f6b7748d640 1 Processor -- start 2024-03-06T07:22:56.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.785+0000 7f6b7748d640 1 -- start start 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b7748d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6b700d22b0 0x7f6b700d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b7748d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6b700b41f0 0x7f6b700b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b7748d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b700b3430 con 0x7f6b7015dfe0 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b7748d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b700d2c50 con 0x7f6b700b41f0 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b7748d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b700b31e0 con 0x7f6b700d22b0 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75a2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6b700b41f0 0x7f6b700b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75229640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6b7015dfe0 0x7f6b700cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56616/0 (socket says 172.21.15.122:56616) 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75229640 1 -- 172.21.15.122:0/820074106 learned_addr learned my addr 172.21.15.122:0/820074106 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75a2a640 1 -- 172.21.15.122:0/820074106 >> v2:172.21.15.169:3300/0 conn(0x7f6b700d22b0 msgr2=0x7f6b700d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b74a28640 1 --2- 172.21.15.122:0/820074106 >> v2:172.21.15.169:3300/0 conn(0x7f6b700d22b0 0x7f6b700d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b67fff640 1 -- 172.21.15.122:0/820074106 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006146901 0 0) 0x7f6b700b3430 con 0x7f6b7015dfe0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75a2a640 1 --2- 172.21.15.122:0/820074106 >> v2:172.21.15.169:3300/0 conn(0x7f6b700d22b0 0x7f6b700d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75a2a640 1 -- 172.21.15.122:0/820074106 >> v1:172.21.15.122:6789/0 conn(0x7f6b7015dfe0 legacy=0x7f6b700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.786+0000 7f6b75a2a640 1 -- 172.21.15.122:0/820074106 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b700b3710 con 0x7f6b700b41f0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.787+0000 7f6b75a2a640 1 --2- 172.21.15.122:0/820074106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700b41f0 0x7f6b700b4680 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009d90 tx=0x7f6b6802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94044499f85905e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.787+0000 7f6b67fff640 1 -- 172.21.15.122:0/820074106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68039070 con 0x7f6b700b41f0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.787+0000 7f6b67fff640 1 -- 172.21.15.122:0/820074106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6802cc60 con 0x7f6b700b41f0 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.787+0000 7f6b67fff640 1 -- 172.21.15.122:0/820074106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68034660 con 0x7f6b700b41f0 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 -- 172.21.15.122:0/820074106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700b41f0 msgr2=0x7f6b700b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 --2- 172.21.15.122:0/820074106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700b41f0 0x7f6b700b4680 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009d90 tx=0x7f6b6802beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 -- 172.21.15.122:0/820074106 shutdown_connections 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 --2- 172.21.15.122:0/820074106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700b41f0 0x7f6b700b4680 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 --2- 172.21.15.122:0/820074106 >> v2:172.21.15.169:3300/0 conn(0x7f6b700d22b0 0x7f6b700d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 -- 172.21.15.122:0/820074106 >> 172.21.15.122:0/820074106 conn(0x7f6b700d3cd0 msgr2=0x7f6b700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 -- 172.21.15.122:0/820074106 shutdown_connections 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.788+0000 7f6b7748d640 1 -- 172.21.15.122:0/820074106 wait complete. 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.789+0000 7f6b7748d640 1 Processor -- start 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.789+0000 7f6b7748d640 1 -- start start 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b7748d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700b41f0 0x7f6b7015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b7748d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d22b0 0x7f6b700ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b7748d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 0x7f6b7006c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b7748d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b700cfa00 con 0x7f6b7015dfe0 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b7748d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b700b3d60 con 0x7f6b700d22b0 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b7748d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b700d1c70 con 0x7f6b700b41f0 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 0x7f6b7006c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 0x7f6b7006c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38446/0 (socket says 172.21.15.122:38446) 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 -- 172.21.15.122:0/3337837925 learned_addr learned my addr 172.21.15.122:0/3337837925 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b74a28640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d22b0 0x7f6b700ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75229640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700b41f0 0x7f6b7015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 -- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700b41f0 msgr2=0x7f6b7015b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700b41f0 0x7f6b7015b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 -- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d22b0 msgr2=0x7f6b700ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d22b0 0x7f6b700ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 -- 172.21.15.122:0/3337837925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b700cd000 con 0x7f6b7015dfe0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b74a28640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d22b0 0x7f6b700ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.790+0000 7f6b75a2a640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 0x7f6b7006c630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009d90 tx=0x7f6b68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.791+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68044020 con 0x7f6b7015dfe0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.791+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6803a040 con 0x7f6b7015dfe0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.791+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68039050 con 0x7f6b7015dfe0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.791+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b700cba20 con 0x7f6b7015dfe0 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.791+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b700cbfe0 con 0x7f6b7015dfe0 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.792+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7004f330 con 0x7f6b7015dfe0 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.792+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b68003bd0 con 0x7f6b7015dfe0 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.793+0000 7f6b65ffb640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5406b4d0 0x7f6b5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.793+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b680ac880 con 0x7f6b7015dfe0 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.793+0000 7f6b75229640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5406b4d0 0x7f6b5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.793+0000 7f6b75229640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5406b4d0 0x7f6b5406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c000f60 tx=0x7f6b6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.801+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b6804e060 con 0x7f6b7015dfe0 2024-03-06T07:22:56.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.975+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b700d0e40 con 0x7f6b5406b4d0 2024-03-06T07:22:56.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.986+0000 7f6b65ffb640 1 -- 172.21.15.122:0/3337837925 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237187 (secure 0 0 0) 0x7f6b700d0e40 con 0x7f6b5406b4d0 2024-03-06T07:22:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:56.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5406b4d0 msgr2=0x7f6b5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5406b4d0 0x7f6b5406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c000f60 tx=0x7f6b6c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 msgr2=0x7f6b7006c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 0x7f6b7006c630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009d90 tx=0x7f6b68004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 shutdown_connections 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5406b4d0 0x7f6b5406d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b7015dfe0 0x7f6b7006c630 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d22b0 0x7f6b700ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 --2- 172.21.15.122:0/3337837925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700b41f0 0x7f6b7015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 >> 172.21.15.122:0/3337837925 conn(0x7f6b700d3cd0 msgr2=0x7f6b700cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 shutdown_connections 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:56.989+0000 7f6b7748d640 1 -- 172.21.15.122:0/3337837925 wait complete. 2024-03-06T07:22:57.002 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-06T07:22:55.343193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219928,"kb_used_data":3688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754792,"statfs":{"total":773094113280,"available":772868907008,"internally_reserved":0,"allocated":3776512,"data_stored":2152328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.361037"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.780334+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:22:46.780334+0000","last_peered":"2024-03-06T07:22:46.780334+0000","last_clean":"2024-03-06T07:22:46.780334+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:22:46.780334+0000","last_undegraded":"2024-03-06T07:22:46.780334+0000","last_fullsized":"2024-03-06T07:22:46.780334+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":917504,"data_stored":711817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":711817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:57.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.139+0000 7f0e3317a640 1 Processor -- start 2024-03-06T07:22:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.139+0000 7f0e3317a640 1 -- start start 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e3317a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e2c15dfe0 0x7f0e2c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e3317a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0e2c0cf830 0x7f0e2c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e3317a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c0d27f0 0x7f0e2c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e3317a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2c0b31e0 con 0x7f0e2c15dfe0 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e3317a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2c0b3d60 con 0x7f0e2c0cf830 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e3317a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c0b3430 con 0x7f0e2c0d27f0 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e2c15dfe0 0x7f0e2c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e2c15dfe0 0x7f0e2c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38458/0 (socket says 172.21.15.122:38458) 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 -- 172.21.15.122:0/2010035144 learned_addr learned my addr 172.21.15.122:0/2010035144 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e31717640 1 --2- 172.21.15.122:0/2010035144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c0d27f0 0x7f0e2c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e2bfff640 1 --2- 172.21.15.122:0/2010035144 >> v2:172.21.15.169:3300/0 conn(0x7f0e2c0cf830 0x7f0e2c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 -- 172.21.15.122:0/2010035144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c0d27f0 msgr2=0x7f0e2c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 --2- 172.21.15.122:0/2010035144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c0d27f0 0x7f0e2c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 -- 172.21.15.122:0/2010035144 >> v2:172.21.15.169:3300/0 conn(0x7f0e2c0cf830 msgr2=0x7f0e2c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 --2- 172.21.15.122:0/2010035144 >> v2:172.21.15.169:3300/0 conn(0x7f0e2c0cf830 0x7f0e2c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.140+0000 7f0e30f16640 1 -- 172.21.15.122:0/2010035144 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c0b3710 con 0x7f0e2c15dfe0 2024-03-06T07:22:57.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.141+0000 7f0e30f16640 1 --2- 172.21.15.122:0/2010035144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c15dfe0 0x7f0e2c0cf2f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0d16b0 tx=0x7f0e2402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d475a5449d973db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e2b7fe640 1 -- 172.21.15.122:0/2010035144 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2402ca70 con 0x7f0e2c15dfe0 2024-03-06T07:22:57.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e2b7fe640 1 -- 172.21.15.122:0/2010035144 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2402cc10 con 0x7f0e2c15dfe0 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e2b7fe640 1 -- 172.21.15.122:0/2010035144 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e240343f0 con 0x7f0e2c15dfe0 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 -- 172.21.15.122:0/2010035144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c15dfe0 msgr2=0x7f0e2c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 --2- 172.21.15.122:0/2010035144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c15dfe0 0x7f0e2c0cf2f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0d16b0 tx=0x7f0e2402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 -- 172.21.15.122:0/2010035144 shutdown_connections 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 --2- 172.21.15.122:0/2010035144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c0d27f0 0x7f0e2c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 --2- 172.21.15.122:0/2010035144 >> v2:172.21.15.169:3300/0 conn(0x7f0e2c0cf830 0x7f0e2c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 --2- 172.21.15.122:0/2010035144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c15dfe0 0x7f0e2c0cf2f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 -- 172.21.15.122:0/2010035144 >> 172.21.15.122:0/2010035144 conn(0x7f0e2c0d3cd0 msgr2=0x7f0e2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 -- 172.21.15.122:0/2010035144 shutdown_connections 2024-03-06T07:22:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.142+0000 7f0e3317a640 1 -- 172.21.15.122:0/2010035144 wait complete. 2024-03-06T07:22:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.143+0000 7f0e3317a640 1 Processor -- start 2024-03-06T07:22:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.143+0000 7f0e3317a640 1 -- start start 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e3317a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 0x7f0e2c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e3317a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e2c0d27f0 0x7f0e2c15a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e3317a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c15dfe0 0x7f0e2c0b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e3317a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c0d1df0 con 0x7f0e2c0cf830 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e3317a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c0d1c70 con 0x7f0e2c0d27f0 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e3317a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c15e3c0 con 0x7f0e2c15dfe0 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e30f16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 0x7f0e2c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e30f16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 0x7f0e2c0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38468/0 (socket says 172.21.15.122:38468) 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.144+0000 7f0e30f16640 1 -- 172.21.15.122:0/3819821906 learned_addr learned my addr 172.21.15.122:0/3819821906 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e31717640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c15dfe0 0x7f0e2c0b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e2bfff640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e2c0d27f0 0x7f0e2c15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e30f16640 1 -- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c15dfe0 msgr2=0x7f0e2c0b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e30f16640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c15dfe0 0x7f0e2c0b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e30f16640 1 -- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e2c0d27f0 msgr2=0x7f0e2c15a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e30f16640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e2c0d27f0 0x7f0e2c15a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e30f16640 1 -- 172.21.15.122:0/3819821906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c0caf40 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e30f16640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 0x7f0e2c0cd7a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0e24036040 tx=0x7f0e24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24043020 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2403e450 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c0cb1e0 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24009510 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.145+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c0cb7a0 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.146+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c04f330 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.147+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e24046020 con 0x7f0e2c0cf830 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.147+0000 7f0e297fa640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0c06b4d0 0x7f0e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.147+0000 7f0e2bfff640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0c06b4d0 0x7f0e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.147+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e240ac960 con 0x7f0e2c0cf830 2024-03-06T07:22:57.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.148+0000 7f0e2bfff640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0c06b4d0 0x7f0e0c06d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0d4960 tx=0x7f0e14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.152+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e2404d030 con 0x7f0e2c0cf830 2024-03-06T07:22:57.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.325+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e2c0cb970 con 0x7f0e0c06b4d0 2024-03-06T07:22:57.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.330+0000 7f0e297fa640 1 -- 172.21.15.122:0/3819821906 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237187 (secure 0 0 0) 0x7f0e2c0cb970 con 0x7f0e0c06b4d0 2024-03-06T07:22:57.392 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:57.393 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:57.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0c06b4d0 msgr2=0x7f0e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0c06b4d0 0x7f0e0c06d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0d4960 tx=0x7f0e14008040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 msgr2=0x7f0e2c0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 0x7f0e2c0cd7a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0e24036040 tx=0x7f0e24004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 shutdown_connections 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0c06b4d0 0x7f0e0c06d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e2c15dfe0 0x7f0e2c0b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e2c0d27f0 0x7f0e2c15a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 --2- 172.21.15.122:0/3819821906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e2c0cf830 0x7f0e2c0cd7a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 >> 172.21.15.122:0/3819821906 conn(0x7f0e2c0d3cd0 msgr2=0x7f0e2c06c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 shutdown_connections 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.333+0000 7f0e3317a640 1 -- 172.21.15.122:0/3819821906 wait complete. 2024-03-06T07:22:57.396 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-06T07:22:55.343193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219928,"kb_used_data":3688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754792,"statfs":{"total":773094113280,"available":772868907008,"internally_reserved":0,"allocated":3776512,"data_stored":2152328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.361037"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:51.982203+0000","last_change":"2024-03-06T07:22:51.982203+0000","last_active":"2024-03-06T07:22:51.982203+0000","last_peered":"2024-03-06T07:22:51.982203+0000","last_clean":"2024-03-06T07:22:51.982203+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:51.982203+0000","last_undegraded":"2024-03-06T07:22:51.982203+0000","last_fullsized":"2024-03-06T07:22:51.982203+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:22:50.980101+0000","last_change":"2024-03-06T07:22:50.980101+0000","last_active":"2024-03-06T07:22:50.980101+0000","last_peered":"2024-03-06T07:22:50.980101+0000","last_clean":"2024-03-06T07:22:50.980101+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:50.980101+0000","last_undegraded":"2024-03-06T07:22:50.980101+0000","last_fullsized":"2024-03-06T07:22:50.980101+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.780334+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:22:46.780334+0000","last_peered":"2024-03-06T07:22:46.780334+0000","last_clean":"2024-03-06T07:22:46.780334+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:22:46.780334+0000","last_undegraded":"2024-03-06T07:22:46.780334+0000","last_fullsized":"2024-03-06T07:22:46.780334+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":917504,"data_stored":711817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":121449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":711817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:57.400 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:57.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.498+0000 7f7ab2b56640 1 Processor -- start 2024-03-06T07:22:57.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.498+0000 7f7ab2b56640 1 -- start start 2024-03-06T07:22:57.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.499+0000 7f7ab2b56640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7aac15dfe0 0x7f7aac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.499+0000 7f7ab2b56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0b3430 con 0x7f7aac0cf830 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.499+0000 7f7ab2b56640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0b3710 con 0x7f7aac0d29c0 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.499+0000 7f7ab2b56640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0b31e0 con 0x7f7aac15dfe0 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.499+0000 7f7aabfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7aac0cf830 0x7f7aac0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56620/0 (socket says 172.21.15.122:56620) 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.499+0000 7f7aabfff640 1 -- 172.21.15.122:0/3923917054 learned_addr learned my addr 172.21.15.122:0/3923917054 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/3923917054 >> v2:172.21.15.169:3300/0 conn(0x7f7aac15dfe0 0x7f7aac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279386865 0 0) 0x7f7aac0b3430 con 0x7f7aac0cf830 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a9c003520 con 0x7f7aac0cf830 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879764804 0 0) 0x7f7a9c003520 con 0x7f7aac0cf830 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 >> v2:172.21.15.169:3300/0 conn(0x7f7aac15dfe0 msgr2=0x7f7aac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 --2- 172.21.15.122:0/3923917054 >> v2:172.21.15.169:3300/0 conn(0x7f7aac15dfe0 0x7f7aac0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 >> v1:172.21.15.169:6789/0 conn(0x7f7aac0d29c0 legacy=0x7f7aac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac0b39f0 con 0x7f7aac0cf830 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7a98001670 con 0x7f7aac0cf830 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 >> v1:172.21.15.122:6789/0 conn(0x7f7aac0cf830 legacy=0x7f7aac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a9c0039b0 0x7f7a9c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a9c004800 0x7f7a9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0051c0 0x7f7a9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0b31e0 con 0x7f7a9c0039b0 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c00c090 con 0x7f7a9c004800 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c00c290 con 0x7f7a9c0051c0 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a9c0039b0 0x7f7a9c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab10f3640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0051c0 0x7f7a9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7aabfff640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a9c004800 0x7f7a9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 -- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0051c0 msgr2=0x7f7a9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0051c0 0x7f7a9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 -- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a9c004800 msgr2=0x7f7a9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a9c004800 0x7f7a9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab08f2640 1 -- 172.21.15.122:0/3923917054 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c003520 con 0x7f7a9c0039b0 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.500+0000 7f7ab10f3640 1 -- 172.21.15.122:0/3923917054 reap_dead start 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.501+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a9c0039b0 0x7f7a9c003df0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ac0 tx=0x7f7a9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43c1ffe5dc34122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.501+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0039070 con 0x7f7a9c0039b0 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.501+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa002c6d0 con 0x7f7a9c0039b0 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.501+0000 7f7aab7fe640 1 -- 172.21.15.122:0/3923917054 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0009690 con 0x7f7a9c0039b0 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 -- 172.21.15.122:0/3923917054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a9c0039b0 msgr2=0x7f7a9c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a9c0039b0 0x7f7a9c003df0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ac0 tx=0x7f7a9c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 -- 172.21.15.122:0/3923917054 shutdown_connections 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/3923917054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a9c0039b0 0x7f7a9c003df0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 -- 172.21.15.122:0/3923917054 >> 172.21.15.122:0/3923917054 conn(0x7f7aac0d3cd0 msgr2=0x7f7aac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 -- 172.21.15.122:0/3923917054 shutdown_connections 2024-03-06T07:22:57.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.502+0000 7f7ab2b56640 1 -- 172.21.15.122:0/3923917054 wait complete. 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.504+0000 7f7ab2b56640 1 Processor -- start 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.504+0000 7f7ab2b56640 1 -- start start 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab2b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0039b0 0x7f7aac1f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab2b56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 0x7f7aac1f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab2b56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aac1f7510 0x7f7aac0b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab2b56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac15e3c0 con 0x7f7aac0ccd10 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab2b56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0b3d60 con 0x7f7aac1f7510 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab2b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac15e540 con 0x7f7a9c0039b0 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7aabfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 0x7f7aac1f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7aabfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 0x7f7aac1f6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38496/0 (socket says 172.21.15.122:38496) 2024-03-06T07:22:57.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7aabfff640 1 -- 172.21.15.122:0/1165218302 learned_addr learned my addr 172.21.15.122:0/1165218302 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0039b0 0x7f7aac1f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7ab10f3640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aac1f7510 0x7f7aac0b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7aabfff640 1 -- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0039b0 msgr2=0x7f7aac1f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.505+0000 7f7aabfff640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0039b0 0x7f7aac1f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aabfff640 1 -- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aac1f7510 msgr2=0x7f7aac0b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aabfff640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aac1f7510 0x7f7aac0b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aabfff640 1 -- 172.21.15.122:0/1165218302 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac1f7e20 con 0x7f7aac0ccd10 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0039b0 0x7f7aac1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7ab10f3640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aac1f7510 0x7f7aac0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aabfff640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 0x7f7aac1f6fd0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800e7f0 tx=0x7f7a9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9800c8d0 con 0x7f7aac0ccd10 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a98018070 con 0x7f7aac0ccd10 2024-03-06T07:22:57.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac160970 con 0x7f7aac0ccd10 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9800ccb0 con 0x7f7aac0ccd10 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.506+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac1f3af0 con 0x7f7aac0ccd10 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.507+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac04f330 con 0x7f7aac0ccd10 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.507+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a98013420 con 0x7f7aac0ccd10 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.508+0000 7f7aa97fa640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a9006b500 0x7f7a9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.508+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a98089cf0 con 0x7f7aac0ccd10 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.508+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a9006b500 0x7f7a9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.508+0000 7f7ab08f2640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a9006b500 0x7f7a9006d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ac0 tx=0x7f7aa0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.512+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a9808e070 con 0x7f7aac0ccd10 2024-03-06T07:22:57.766 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:22:57.765+0000 7f9bd7148640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:57.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.799+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7aac1f78f0 con 0x7f7a9006b500 2024-03-06T07:22:57.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.805+0000 7f7aa97fa640 1 -- 172.21.15.122:0/1165218302 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251948 (secure 0 0 0) 0x7f7aa0082840 con 0x7f7a9006b500 2024-03-06T07:22:57.810 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:57.811 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:57.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.808+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a9006b500 msgr2=0x7f7a9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.808+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a9006b500 0x7f7a9006d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ac0 tx=0x7f7aa0030000 comp rx=0 tx=0).stop 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.808+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 msgr2=0x7f7aac1f6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.808+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 0x7f7aac1f6fd0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800e7f0 tx=0x7f7a9800ecb0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 shutdown_connections 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a9006b500 0x7f7a9006d9c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aac1f7510 0x7f7aac0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aac0ccd10 0x7f7aac1f6fd0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 --2- 172.21.15.122:0/1165218302 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a9c0039b0 0x7f7aac1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 >> 172.21.15.122:0/1165218302 conn(0x7f7aac0d3cd0 msgr2=0x7f7aac06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 shutdown_connections 2024-03-06T07:22:57.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.809+0000 7f7ab2b56640 1 -- 172.21.15.122:0/1165218302 wait complete. 2024-03-06T07:22:57.821 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-06T07:22:57.343477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220304,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754416,"statfs":{"total":773094113280,"available":772868521984,"internally_reserved":0,"allocated":4235264,"data_stored":2358420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000931"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007270+0000","last_change":"2024-03-06T07:22:53.015900+0000","last_active":"2024-03-06T07:22:54.007270+0000","last_peered":"2024-03-06T07:22:54.007270+0000","last_clean":"2024-03-06T07:22:54.007270+0000","last_became_active":"2024-03-06T07:22:53.015567+0000","last_became_peered":"2024-03-06T07:22:53.015567+0000","last_unstale":"2024-03-06T07:22:54.007270+0000","last_undegraded":"2024-03-06T07:22:54.007270+0000","last_fullsized":"2024-03-06T07:22:54.007270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:08.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282163+0000","last_change":"2024-03-06T07:22:53.005999+0000","last_active":"2024-03-06T07:22:54.282163+0000","last_peered":"2024-03-06T07:22:54.282163+0000","last_clean":"2024-03-06T07:22:54.282163+0000","last_became_active":"2024-03-06T07:22:53.005727+0000","last_became_peered":"2024-03-06T07:22:53.005727+0000","last_unstale":"2024-03-06T07:22:54.282163+0000","last_undegraded":"2024-03-06T07:22:54.282163+0000","last_fullsized":"2024-03-06T07:22:54.282163+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:04.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283677+0000","last_change":"2024-03-06T07:22:53.009774+0000","last_active":"2024-03-06T07:22:54.283677+0000","last_peered":"2024-03-06T07:22:54.283677+0000","last_clean":"2024-03-06T07:22:54.283677+0000","last_became_active":"2024-03-06T07:22:53.009488+0000","last_became_peered":"2024-03-06T07:22:53.009488+0000","last_unstale":"2024-03-06T07:22:54.283677+0000","last_undegraded":"2024-03-06T07:22:54.283677+0000","last_fullsized":"2024-03-06T07:22:54.283677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006758+0000","last_change":"2024-03-06T07:22:53.019399+0000","last_active":"2024-03-06T07:22:54.006758+0000","last_peered":"2024-03-06T07:22:54.006758+0000","last_clean":"2024-03-06T07:22:54.006758+0000","last_became_active":"2024-03-06T07:22:53.019122+0000","last_became_peered":"2024-03-06T07:22:53.019122+0000","last_unstale":"2024-03-06T07:22:54.006758+0000","last_undegraded":"2024-03-06T07:22:54.006758+0000","last_fullsized":"2024-03-06T07:22:54.006758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281419+0000","last_change":"2024-03-06T07:22:53.020095+0000","last_active":"2024-03-06T07:22:54.281419+0000","last_peered":"2024-03-06T07:22:54.281419+0000","last_clean":"2024-03-06T07:22:54.281419+0000","last_became_active":"2024-03-06T07:22:53.019764+0000","last_became_peered":"2024-03-06T07:22:53.019764+0000","last_unstale":"2024-03-06T07:22:54.281419+0000","last_undegraded":"2024-03-06T07:22:54.281419+0000","last_fullsized":"2024-03-06T07:22:54.281419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:16.936064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281986+0000","last_change":"2024-03-06T07:22:53.011642+0000","last_active":"2024-03-06T07:22:54.281986+0000","last_peered":"2024-03-06T07:22:54.281986+0000","last_clean":"2024-03-06T07:22:54.281986+0000","last_became_active":"2024-03-06T07:22:53.011303+0000","last_became_peered":"2024-03-06T07:22:53.011303+0000","last_unstale":"2024-03-06T07:22:54.281986+0000","last_undegraded":"2024-03-06T07:22:54.281986+0000","last_fullsized":"2024-03-06T07:22:54.281986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282515+0000","last_change":"2024-03-06T07:22:53.009740+0000","last_active":"2024-03-06T07:22:54.282515+0000","last_peered":"2024-03-06T07:22:54.282515+0000","last_clean":"2024-03-06T07:22:54.282515+0000","last_became_active":"2024-03-06T07:22:53.009431+0000","last_became_peered":"2024-03-06T07:22:53.009431+0000","last_unstale":"2024-03-06T07:22:54.282515+0000","last_undegraded":"2024-03-06T07:22:54.282515+0000","last_fullsized":"2024-03-06T07:22:54.282515+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282887+0000","last_change":"2024-03-06T07:22:53.011772+0000","last_active":"2024-03-06T07:22:54.282887+0000","last_peered":"2024-03-06T07:22:54.282887+0000","last_clean":"2024-03-06T07:22:54.282887+0000","last_became_active":"2024-03-06T07:22:53.011391+0000","last_became_peered":"2024-03-06T07:22:53.011391+0000","last_unstale":"2024-03-06T07:22:54.282887+0000","last_undegraded":"2024-03-06T07:22:54.282887+0000","last_fullsized":"2024-03-06T07:22:54.282887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282966+0000","last_change":"2024-03-06T07:22:53.008973+0000","last_active":"2024-03-06T07:22:54.282966+0000","last_peered":"2024-03-06T07:22:54.282966+0000","last_clean":"2024-03-06T07:22:54.282966+0000","last_became_active":"2024-03-06T07:22:53.008608+0000","last_became_peered":"2024-03-06T07:22:53.008608+0000","last_unstale":"2024-03-06T07:22:54.282966+0000","last_undegraded":"2024-03-06T07:22:54.282966+0000","last_fullsized":"2024-03-06T07:22:54.282966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.337698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006851+0000","last_change":"2024-03-06T07:22:53.014766+0000","last_active":"2024-03-06T07:22:54.006851+0000","last_peered":"2024-03-06T07:22:54.006851+0000","last_clean":"2024-03-06T07:22:54.006851+0000","last_became_active":"2024-03-06T07:22:53.014521+0000","last_became_peered":"2024-03-06T07:22:53.014521+0000","last_unstale":"2024-03-06T07:22:54.006851+0000","last_undegraded":"2024-03-06T07:22:54.006851+0000","last_fullsized":"2024-03-06T07:22:54.006851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:01.752069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282270+0000","last_change":"2024-03-06T07:22:53.008566+0000","last_active":"2024-03-06T07:22:54.282270+0000","last_peered":"2024-03-06T07:22:54.282270+0000","last_clean":"2024-03-06T07:22:54.282270+0000","last_became_active":"2024-03-06T07:22:53.008322+0000","last_became_peered":"2024-03-06T07:22:53.008322+0000","last_unstale":"2024-03-06T07:22:54.282270+0000","last_undegraded":"2024-03-06T07:22:54.282270+0000","last_fullsized":"2024-03-06T07:22:54.282270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283396+0000","last_change":"2024-03-06T07:22:53.018934+0000","last_active":"2024-03-06T07:22:54.283396+0000","last_peered":"2024-03-06T07:22:54.283396+0000","last_clean":"2024-03-06T07:22:54.283396+0000","last_became_active":"2024-03-06T07:22:53.018657+0000","last_became_peered":"2024-03-06T07:22:53.018657+0000","last_unstale":"2024-03-06T07:22:54.283396+0000","last_undegraded":"2024-03-06T07:22:54.283396+0000","last_fullsized":"2024-03-06T07:22:54.283396+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281455+0000","last_change":"2024-03-06T07:22:53.017169+0000","last_active":"2024-03-06T07:22:54.281455+0000","last_peered":"2024-03-06T07:22:54.281455+0000","last_clean":"2024-03-06T07:22:54.281455+0000","last_became_active":"2024-03-06T07:22:53.016669+0000","last_became_peered":"2024-03-06T07:22:53.016669+0000","last_unstale":"2024-03-06T07:22:54.281455+0000","last_undegraded":"2024-03-06T07:22:54.281455+0000","last_fullsized":"2024-03-06T07:22:54.281455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:26.061969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282078+0000","last_change":"2024-03-06T07:22:53.011930+0000","last_active":"2024-03-06T07:22:54.282078+0000","last_peered":"2024-03-06T07:22:54.282078+0000","last_clean":"2024-03-06T07:22:54.282078+0000","last_became_active":"2024-03-06T07:22:53.011616+0000","last_became_peered":"2024-03-06T07:22:53.011616+0000","last_unstale":"2024-03-06T07:22:54.282078+0000","last_undegraded":"2024-03-06T07:22:54.282078+0000","last_fullsized":"2024-03-06T07:22:54.282078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:19.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283416+0000","last_change":"2024-03-06T07:22:53.017836+0000","last_active":"2024-03-06T07:22:54.283416+0000","last_peered":"2024-03-06T07:22:54.283416+0000","last_clean":"2024-03-06T07:22:54.283416+0000","last_became_active":"2024-03-06T07:22:53.017495+0000","last_became_peered":"2024-03-06T07:22:53.017495+0000","last_unstale":"2024-03-06T07:22:54.283416+0000","last_undegraded":"2024-03-06T07:22:54.283416+0000","last_fullsized":"2024-03-06T07:22:54.283416+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282521+0000","last_change":"2024-03-06T07:22:53.014100+0000","last_active":"2024-03-06T07:22:54.282521+0000","last_peered":"2024-03-06T07:22:54.282521+0000","last_clean":"2024-03-06T07:22:54.282521+0000","last_became_active":"2024-03-06T07:22:53.013806+0000","last_became_peered":"2024-03-06T07:22:53.013806+0000","last_unstale":"2024-03-06T07:22:54.282521+0000","last_undegraded":"2024-03-06T07:22:54.282521+0000","last_fullsized":"2024-03-06T07:22:54.282521+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282326+0000","last_change":"2024-03-06T07:22:53.009816+0000","last_active":"2024-03-06T07:22:54.282326+0000","last_peered":"2024-03-06T07:22:54.282326+0000","last_clean":"2024-03-06T07:22:54.282326+0000","last_became_active":"2024-03-06T07:22:53.009497+0000","last_became_peered":"2024-03-06T07:22:53.009497+0000","last_unstale":"2024-03-06T07:22:54.282326+0000","last_undegraded":"2024-03-06T07:22:54.282326+0000","last_fullsized":"2024-03-06T07:22:54.282326+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282288+0000","last_change":"2024-03-06T07:22:53.008196+0000","last_active":"2024-03-06T07:22:54.282288+0000","last_peered":"2024-03-06T07:22:54.282288+0000","last_clean":"2024-03-06T07:22:54.282288+0000","last_became_active":"2024-03-06T07:22:53.007916+0000","last_became_peered":"2024-03-06T07:22:53.007916+0000","last_unstale":"2024-03-06T07:22:54.282288+0000","last_undegraded":"2024-03-06T07:22:54.282288+0000","last_fullsized":"2024-03-06T07:22:54.282288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:33.590994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281796+0000","last_change":"2024-03-06T07:22:53.009095+0000","last_active":"2024-03-06T07:22:54.281796+0000","last_peered":"2024-03-06T07:22:54.281796+0000","last_clean":"2024-03-06T07:22:54.281796+0000","last_became_active":"2024-03-06T07:22:53.008716+0000","last_became_peered":"2024-03-06T07:22:53.008716+0000","last_unstale":"2024-03-06T07:22:54.281796+0000","last_undegraded":"2024-03-06T07:22:54.281796+0000","last_fullsized":"2024-03-06T07:22:54.281796+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281677+0000","last_change":"2024-03-06T07:22:53.018889+0000","last_active":"2024-03-06T07:22:54.281677+0000","last_peered":"2024-03-06T07:22:54.281677+0000","last_clean":"2024-03-06T07:22:54.281677+0000","last_became_active":"2024-03-06T07:22:53.018596+0000","last_became_peered":"2024-03-06T07:22:53.018596+0000","last_unstale":"2024-03-06T07:22:54.281677+0000","last_undegraded":"2024-03-06T07:22:54.281677+0000","last_fullsized":"2024-03-06T07:22:54.281677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.014845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282871+0000","last_change":"2024-03-06T07:22:53.002357+0000","last_active":"2024-03-06T07:22:54.282871+0000","last_peered":"2024-03-06T07:22:54.282871+0000","last_clean":"2024-03-06T07:22:54.282871+0000","last_became_active":"2024-03-06T07:22:53.002122+0000","last_became_peered":"2024-03-06T07:22:53.002122+0000","last_unstale":"2024-03-06T07:22:54.282871+0000","last_undegraded":"2024-03-06T07:22:54.282871+0000","last_fullsized":"2024-03-06T07:22:54.282871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:13.547310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282221+0000","last_change":"2024-03-06T07:22:53.011490+0000","last_active":"2024-03-06T07:22:54.282221+0000","last_peered":"2024-03-06T07:22:54.282221+0000","last_clean":"2024-03-06T07:22:54.282221+0000","last_became_active":"2024-03-06T07:22:53.011196+0000","last_became_peered":"2024-03-06T07:22:53.011196+0000","last_unstale":"2024-03-06T07:22:54.282221+0000","last_undegraded":"2024-03-06T07:22:54.282221+0000","last_fullsized":"2024-03-06T07:22:54.282221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004180+0000","last_change":"2024-03-06T07:22:53.005749+0000","last_active":"2024-03-06T07:22:54.004180+0000","last_peered":"2024-03-06T07:22:54.004180+0000","last_clean":"2024-03-06T07:22:54.004180+0000","last_became_active":"2024-03-06T07:22:53.005355+0000","last_became_peered":"2024-03-06T07:22:53.005355+0000","last_unstale":"2024-03-06T07:22:54.004180+0000","last_undegraded":"2024-03-06T07:22:54.004180+0000","last_fullsized":"2024-03-06T07:22:54.004180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281870+0000","last_change":"2024-03-06T07:22:53.015800+0000","last_active":"2024-03-06T07:22:54.281870+0000","last_peered":"2024-03-06T07:22:54.281870+0000","last_clean":"2024-03-06T07:22:54.281870+0000","last_became_active":"2024-03-06T07:22:53.015308+0000","last_became_peered":"2024-03-06T07:22:53.015308+0000","last_unstale":"2024-03-06T07:22:54.281870+0000","last_undegraded":"2024-03-06T07:22:54.281870+0000","last_fullsized":"2024-03-06T07:22:54.281870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006942+0000","last_change":"2024-03-06T07:22:53.006841+0000","last_active":"2024-03-06T07:22:54.006942+0000","last_peered":"2024-03-06T07:22:54.006942+0000","last_clean":"2024-03-06T07:22:54.006942+0000","last_became_active":"2024-03-06T07:22:53.006562+0000","last_became_peered":"2024-03-06T07:22:53.006562+0000","last_unstale":"2024-03-06T07:22:54.006942+0000","last_undegraded":"2024-03-06T07:22:54.006942+0000","last_fullsized":"2024-03-06T07:22:54.006942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:08.594990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282409+0000","last_change":"2024-03-06T07:22:53.003197+0000","last_active":"2024-03-06T07:22:54.282409+0000","last_peered":"2024-03-06T07:22:54.282409+0000","last_clean":"2024-03-06T07:22:54.282409+0000","last_became_active":"2024-03-06T07:22:53.002347+0000","last_became_peered":"2024-03-06T07:22:53.002347+0000","last_unstale":"2024-03-06T07:22:54.282409+0000","last_undegraded":"2024-03-06T07:22:54.282409+0000","last_fullsized":"2024-03-06T07:22:54.282409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:25.191495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006974+0000","last_change":"2024-03-06T07:22:53.007183+0000","last_active":"2024-03-06T07:22:54.006974+0000","last_peered":"2024-03-06T07:22:54.006974+0000","last_clean":"2024-03-06T07:22:54.006974+0000","last_became_active":"2024-03-06T07:22:53.006620+0000","last_became_peered":"2024-03-06T07:22:53.006620+0000","last_unstale":"2024-03-06T07:22:54.006974+0000","last_undegraded":"2024-03-06T07:22:54.006974+0000","last_fullsized":"2024-03-06T07:22:54.006974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283597+0000","last_change":"2024-03-06T07:22:53.014691+0000","last_active":"2024-03-06T07:22:54.283597+0000","last_peered":"2024-03-06T07:22:54.283597+0000","last_clean":"2024-03-06T07:22:54.283597+0000","last_became_active":"2024-03-06T07:22:53.014414+0000","last_became_peered":"2024-03-06T07:22:53.014414+0000","last_unstale":"2024-03-06T07:22:54.283597+0000","last_undegraded":"2024-03-06T07:22:54.283597+0000","last_fullsized":"2024-03-06T07:22:54.283597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282606+0000","last_change":"2024-03-06T07:22:53.011594+0000","last_active":"2024-03-06T07:22:54.282606+0000","last_peered":"2024-03-06T07:22:54.282606+0000","last_clean":"2024-03-06T07:22:54.282606+0000","last_became_active":"2024-03-06T07:22:53.011237+0000","last_became_peered":"2024-03-06T07:22:53.011237+0000","last_unstale":"2024-03-06T07:22:54.282606+0000","last_undegraded":"2024-03-06T07:22:54.282606+0000","last_fullsized":"2024-03-06T07:22:54.282606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:50.691781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282618+0000","last_change":"2024-03-06T07:22:53.006582+0000","last_active":"2024-03-06T07:22:54.282618+0000","last_peered":"2024-03-06T07:22:54.282618+0000","last_clean":"2024-03-06T07:22:54.282618+0000","last_became_active":"2024-03-06T07:22:53.006235+0000","last_became_peered":"2024-03-06T07:22:53.006235+0000","last_unstale":"2024-03-06T07:22:54.282618+0000","last_undegraded":"2024-03-06T07:22:54.282618+0000","last_fullsized":"2024-03-06T07:22:54.282618+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006986+0000","last_change":"2024-03-06T07:22:53.008162+0000","last_active":"2024-03-06T07:22:54.006986+0000","last_peered":"2024-03-06T07:22:54.006986+0000","last_clean":"2024-03-06T07:22:54.006986+0000","last_became_active":"2024-03-06T07:22:53.007160+0000","last_became_peered":"2024-03-06T07:22:53.007160+0000","last_unstale":"2024-03-06T07:22:54.006986+0000","last_undegraded":"2024-03-06T07:22:54.006986+0000","last_fullsized":"2024-03-06T07:22:54.006986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:54.295092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281950+0000","last_change":"2024-03-06T07:22:53.005295+0000","last_active":"2024-03-06T07:22:54.281950+0000","last_peered":"2024-03-06T07:22:54.281950+0000","last_clean":"2024-03-06T07:22:54.281950+0000","last_became_active":"2024-03-06T07:22:53.004573+0000","last_became_peered":"2024-03-06T07:22:53.004573+0000","last_unstale":"2024-03-06T07:22:54.281950+0000","last_undegraded":"2024-03-06T07:22:54.281950+0000","last_fullsized":"2024-03-06T07:22:54.281950+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:01.752069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004410+0000","last_change":"2024-03-06T07:22:53.004824+0000","last_active":"2024-03-06T07:22:54.004410+0000","last_peered":"2024-03-06T07:22:54.004410+0000","last_clean":"2024-03-06T07:22:54.004410+0000","last_became_active":"2024-03-06T07:22:53.004472+0000","last_became_peered":"2024-03-06T07:22:53.004472+0000","last_unstale":"2024-03-06T07:22:54.004410+0000","last_undegraded":"2024-03-06T07:22:54.004410+0000","last_fullsized":"2024-03-06T07:22:54.004410+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:13.547310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007338+0000","last_change":"2024-03-06T07:22:51.998349+0000","last_active":"2024-03-06T07:22:54.007338+0000","last_peered":"2024-03-06T07:22:54.007338+0000","last_clean":"2024-03-06T07:22:54.007338+0000","last_became_active":"2024-03-06T07:22:51.998094+0000","last_became_peered":"2024-03-06T07:22:51.998094+0000","last_unstale":"2024-03-06T07:22:54.007338+0000","last_undegraded":"2024-03-06T07:22:54.007338+0000","last_fullsized":"2024-03-06T07:22:54.007338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:22.157883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007280+0000","last_change":"2024-03-06T07:22:53.005215+0000","last_active":"2024-03-06T07:22:54.007280+0000","last_peered":"2024-03-06T07:22:54.007280+0000","last_clean":"2024-03-06T07:22:54.007280+0000","last_became_active":"2024-03-06T07:22:53.004955+0000","last_became_peered":"2024-03-06T07:22:53.004955+0000","last_unstale":"2024-03-06T07:22:54.007280+0000","last_undegraded":"2024-03-06T07:22:54.007280+0000","last_fullsized":"2024-03-06T07:22:54.007280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004321+0000","last_change":"2024-03-06T07:22:52.006247+0000","last_active":"2024-03-06T07:22:54.004321+0000","last_peered":"2024-03-06T07:22:54.004321+0000","last_clean":"2024-03-06T07:22:54.004321+0000","last_became_active":"2024-03-06T07:22:52.005917+0000","last_became_peered":"2024-03-06T07:22:52.005917+0000","last_unstale":"2024-03-06T07:22:54.004321+0000","last_undegraded":"2024-03-06T07:22:54.004321+0000","last_fullsized":"2024-03-06T07:22:54.004321+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:16.012744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282625+0000","last_change":"2024-03-06T07:22:53.011697+0000","last_active":"2024-03-06T07:22:54.282625+0000","last_peered":"2024-03-06T07:22:54.282625+0000","last_clean":"2024-03-06T07:22:54.282625+0000","last_became_active":"2024-03-06T07:22:53.011304+0000","last_became_peered":"2024-03-06T07:22:53.011304+0000","last_unstale":"2024-03-06T07:22:54.282625+0000","last_undegraded":"2024-03-06T07:22:54.282625+0000","last_fullsized":"2024-03-06T07:22:54.282625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281799+0000","last_change":"2024-03-06T07:22:52.001909+0000","last_active":"2024-03-06T07:22:54.281799+0000","last_peered":"2024-03-06T07:22:54.281799+0000","last_clean":"2024-03-06T07:22:54.281799+0000","last_became_active":"2024-03-06T07:22:52.001539+0000","last_became_peered":"2024-03-06T07:22:52.001539+0000","last_unstale":"2024-03-06T07:22:54.281799+0000","last_undegraded":"2024-03-06T07:22:54.281799+0000","last_fullsized":"2024-03-06T07:22:54.281799+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:07.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006973+0000","last_change":"2024-03-06T07:22:53.006926+0000","last_active":"2024-03-06T07:22:54.006973+0000","last_peered":"2024-03-06T07:22:54.006973+0000","last_clean":"2024-03-06T07:22:54.006973+0000","last_became_active":"2024-03-06T07:22:53.006338+0000","last_became_peered":"2024-03-06T07:22:53.006338+0000","last_unstale":"2024-03-06T07:22:54.006973+0000","last_undegraded":"2024-03-06T07:22:54.006973+0000","last_fullsized":"2024-03-06T07:22:54.006973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281690+0000","last_change":"2024-03-06T07:22:52.006047+0000","last_active":"2024-03-06T07:22:54.281690+0000","last_peered":"2024-03-06T07:22:54.281690+0000","last_clean":"2024-03-06T07:22:54.281690+0000","last_became_active":"2024-03-06T07:22:52.005713+0000","last_became_peered":"2024-03-06T07:22:52.005713+0000","last_unstale":"2024-03-06T07:22:54.281690+0000","last_undegraded":"2024-03-06T07:22:54.281690+0000","last_fullsized":"2024-03-06T07:22:54.281690+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:12.146531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282779+0000","last_change":"2024-03-06T07:22:53.005043+0000","last_active":"2024-03-06T07:22:54.282779+0000","last_peered":"2024-03-06T07:22:54.282779+0000","last_clean":"2024-03-06T07:22:54.282779+0000","last_became_active":"2024-03-06T07:22:53.004749+0000","last_became_peered":"2024-03-06T07:22:53.004749+0000","last_unstale":"2024-03-06T07:22:54.282779+0000","last_undegraded":"2024-03-06T07:22:54.282779+0000","last_fullsized":"2024-03-06T07:22:54.282779+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282996+0000","last_change":"2024-03-06T07:22:52.003516+0000","last_active":"2024-03-06T07:22:54.282996+0000","last_peered":"2024-03-06T07:22:54.282996+0000","last_clean":"2024-03-06T07:22:54.282996+0000","last_became_active":"2024-03-06T07:22:52.003211+0000","last_became_peered":"2024-03-06T07:22:52.003211+0000","last_unstale":"2024-03-06T07:22:54.282996+0000","last_undegraded":"2024-03-06T07:22:54.282996+0000","last_fullsized":"2024-03-06T07:22:54.282996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:23.061079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283131+0000","last_change":"2024-03-06T07:22:53.014133+0000","last_active":"2024-03-06T07:22:54.283131+0000","last_peered":"2024-03-06T07:22:54.283131+0000","last_clean":"2024-03-06T07:22:54.283131+0000","last_became_active":"2024-03-06T07:22:53.013771+0000","last_became_peered":"2024-03-06T07:22:53.013771+0000","last_unstale":"2024-03-06T07:22:54.283131+0000","last_undegraded":"2024-03-06T07:22:54.283131+0000","last_fullsized":"2024-03-06T07:22:54.283131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282370+0000","last_change":"2024-03-06T07:22:52.003559+0000","last_active":"2024-03-06T07:22:54.282370+0000","last_peered":"2024-03-06T07:22:54.282370+0000","last_clean":"2024-03-06T07:22:54.282370+0000","last_became_active":"2024-03-06T07:22:52.003252+0000","last_became_peered":"2024-03-06T07:22:52.003252+0000","last_unstale":"2024-03-06T07:22:54.282370+0000","last_undegraded":"2024-03-06T07:22:54.282370+0000","last_fullsized":"2024-03-06T07:22:54.282370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:15.933962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282101+0000","last_change":"2024-03-06T07:22:52.996122+0000","last_active":"2024-03-06T07:22:54.282101+0000","last_peered":"2024-03-06T07:22:54.282101+0000","last_clean":"2024-03-06T07:22:54.282101+0000","last_became_active":"2024-03-06T07:22:52.995681+0000","last_became_peered":"2024-03-06T07:22:52.995681+0000","last_unstale":"2024-03-06T07:22:54.282101+0000","last_undegraded":"2024-03-06T07:22:54.282101+0000","last_fullsized":"2024-03-06T07:22:54.282101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:08.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281565+0000","last_change":"2024-03-06T07:22:52.004065+0000","last_active":"2024-03-06T07:22:54.281565+0000","last_peered":"2024-03-06T07:22:54.281565+0000","last_clean":"2024-03-06T07:22:54.281565+0000","last_became_active":"2024-03-06T07:22:52.003744+0000","last_became_peered":"2024-03-06T07:22:52.003744+0000","last_unstale":"2024-03-06T07:22:54.281565+0000","last_undegraded":"2024-03-06T07:22:54.281565+0000","last_fullsized":"2024-03-06T07:22:54.281565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:53.292991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006493+0000","last_change":"2024-03-06T07:22:53.008401+0000","last_active":"2024-03-06T07:22:54.006493+0000","last_peered":"2024-03-06T07:22:54.006493+0000","last_clean":"2024-03-06T07:22:54.006493+0000","last_became_active":"2024-03-06T07:22:53.008139+0000","last_became_peered":"2024-03-06T07:22:53.008139+0000","last_unstale":"2024-03-06T07:22:54.006493+0000","last_undegraded":"2024-03-06T07:22:54.006493+0000","last_fullsized":"2024-03-06T07:22:54.006493+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:04.433486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282222+0000","last_change":"2024-03-06T07:22:52.002965+0000","last_active":"2024-03-06T07:22:54.282222+0000","last_peered":"2024-03-06T07:22:54.282222+0000","last_clean":"2024-03-06T07:22:54.282222+0000","last_became_active":"2024-03-06T07:22:52.002690+0000","last_became_peered":"2024-03-06T07:22:52.002690+0000","last_unstale":"2024-03-06T07:22:54.282222+0000","last_undegraded":"2024-03-06T07:22:54.282222+0000","last_fullsized":"2024-03-06T07:22:54.282222+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:46.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283607+0000","last_change":"2024-03-06T07:22:53.013424+0000","last_active":"2024-03-06T07:22:54.283607+0000","last_peered":"2024-03-06T07:22:54.283607+0000","last_clean":"2024-03-06T07:22:54.283607+0000","last_became_active":"2024-03-06T07:22:53.013124+0000","last_became_peered":"2024-03-06T07:22:53.013124+0000","last_unstale":"2024-03-06T07:22:54.283607+0000","last_undegraded":"2024-03-06T07:22:54.283607+0000","last_fullsized":"2024-03-06T07:22:54.283607+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281752+0000","last_change":"2024-03-06T07:22:52.006835+0000","last_active":"2024-03-06T07:22:54.281752+0000","last_peered":"2024-03-06T07:22:54.281752+0000","last_clean":"2024-03-06T07:22:54.281752+0000","last_became_active":"2024-03-06T07:22:52.006449+0000","last_became_peered":"2024-03-06T07:22:52.006449+0000","last_unstale":"2024-03-06T07:22:54.281752+0000","last_undegraded":"2024-03-06T07:22:54.281752+0000","last_fullsized":"2024-03-06T07:22:54.281752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:49.689679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281589+0000","last_change":"2024-03-06T07:22:53.010786+0000","last_active":"2024-03-06T07:22:54.281589+0000","last_peered":"2024-03-06T07:22:54.281589+0000","last_clean":"2024-03-06T07:22:54.281589+0000","last_became_active":"2024-03-06T07:22:53.010470+0000","last_became_peered":"2024-03-06T07:22:53.010470+0000","last_unstale":"2024-03-06T07:22:54.281589+0000","last_undegraded":"2024-03-06T07:22:54.281589+0000","last_fullsized":"2024-03-06T07:22:54.281589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004403+0000","last_change":"2024-03-06T07:22:51.998392+0000","last_active":"2024-03-06T07:22:54.004403+0000","last_peered":"2024-03-06T07:22:54.004403+0000","last_clean":"2024-03-06T07:22:54.004403+0000","last_became_active":"2024-03-06T07:22:51.998047+0000","last_became_peered":"2024-03-06T07:22:51.998047+0000","last_unstale":"2024-03-06T07:22:54.004403+0000","last_undegraded":"2024-03-06T07:22:54.004403+0000","last_fullsized":"2024-03-06T07:22:54.004403+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:43.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282016+0000","last_change":"2024-03-06T07:22:53.002480+0000","last_active":"2024-03-06T07:22:54.282016+0000","last_peered":"2024-03-06T07:22:54.282016+0000","last_clean":"2024-03-06T07:22:54.282016+0000","last_became_active":"2024-03-06T07:22:53.002193+0000","last_became_peered":"2024-03-06T07:22:53.002193+0000","last_unstale":"2024-03-06T07:22:54.282016+0000","last_undegraded":"2024-03-06T07:22:54.282016+0000","last_fullsized":"2024-03-06T07:22:54.282016+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283382+0000","last_change":"2024-03-06T07:22:52.006845+0000","last_active":"2024-03-06T07:22:54.283382+0000","last_peered":"2024-03-06T07:22:54.283382+0000","last_clean":"2024-03-06T07:22:54.283382+0000","last_became_active":"2024-03-06T07:22:52.006519+0000","last_became_peered":"2024-03-06T07:22:52.006519+0000","last_unstale":"2024-03-06T07:22:54.283382+0000","last_undegraded":"2024-03-06T07:22:54.283382+0000","last_fullsized":"2024-03-06T07:22:54.283382+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:50.470263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283254+0000","last_change":"2024-03-06T07:22:53.009814+0000","last_active":"2024-03-06T07:22:54.283254+0000","last_peered":"2024-03-06T07:22:54.283254+0000","last_clean":"2024-03-06T07:22:54.283254+0000","last_became_active":"2024-03-06T07:22:53.009525+0000","last_became_peered":"2024-03-06T07:22:53.009525+0000","last_unstale":"2024-03-06T07:22:54.283254+0000","last_undegraded":"2024-03-06T07:22:54.283254+0000","last_fullsized":"2024-03-06T07:22:54.283254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282039+0000","last_change":"2024-03-06T07:22:51.999176+0000","last_active":"2024-03-06T07:22:54.282039+0000","last_peered":"2024-03-06T07:22:54.282039+0000","last_clean":"2024-03-06T07:22:54.282039+0000","last_became_active":"2024-03-06T07:22:51.998910+0000","last_became_peered":"2024-03-06T07:22:51.998910+0000","last_unstale":"2024-03-06T07:22:54.282039+0000","last_undegraded":"2024-03-06T07:22:54.282039+0000","last_fullsized":"2024-03-06T07:22:54.282039+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:12.477243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006543+0000","last_change":"2024-03-06T07:22:53.010178+0000","last_active":"2024-03-06T07:22:54.006543+0000","last_peered":"2024-03-06T07:22:54.006543+0000","last_clean":"2024-03-06T07:22:54.006543+0000","last_became_active":"2024-03-06T07:22:53.009892+0000","last_became_peered":"2024-03-06T07:22:53.009892+0000","last_unstale":"2024-03-06T07:22:54.006543+0000","last_undegraded":"2024-03-06T07:22:54.006543+0000","last_fullsized":"2024-03-06T07:22:54.006543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.337698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282986+0000","last_change":"2024-03-06T07:22:52.003595+0000","last_active":"2024-03-06T07:22:54.282986+0000","last_peered":"2024-03-06T07:22:54.282986+0000","last_clean":"2024-03-06T07:22:54.282986+0000","last_became_active":"2024-03-06T07:22:52.003239+0000","last_became_peered":"2024-03-06T07:22:52.003239+0000","last_unstale":"2024-03-06T07:22:54.282986+0000","last_undegraded":"2024-03-06T07:22:54.282986+0000","last_fullsized":"2024-03-06T07:22:54.282986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:10.020728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281828+0000","last_change":"2024-03-06T07:22:53.004565+0000","last_active":"2024-03-06T07:22:54.281828+0000","last_peered":"2024-03-06T07:22:54.281828+0000","last_clean":"2024-03-06T07:22:54.281828+0000","last_became_active":"2024-03-06T07:22:53.004149+0000","last_became_peered":"2024-03-06T07:22:53.004149+0000","last_unstale":"2024-03-06T07:22:54.281828+0000","last_undegraded":"2024-03-06T07:22:54.281828+0000","last_fullsized":"2024-03-06T07:22:54.281828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006539+0000","last_change":"2024-03-06T07:22:51.997940+0000","last_active":"2024-03-06T07:22:54.006539+0000","last_peered":"2024-03-06T07:22:54.006539+0000","last_clean":"2024-03-06T07:22:54.006539+0000","last_became_active":"2024-03-06T07:22:51.997652+0000","last_became_peered":"2024-03-06T07:22:51.997652+0000","last_unstale":"2024-03-06T07:22:54.006539+0000","last_undegraded":"2024-03-06T07:22:54.006539+0000","last_fullsized":"2024-03-06T07:22:54.006539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:24.629441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282148+0000","last_change":"2024-03-06T07:22:53.005336+0000","last_active":"2024-03-06T07:22:54.282148+0000","last_peered":"2024-03-06T07:22:54.282148+0000","last_clean":"2024-03-06T07:22:54.282148+0000","last_became_active":"2024-03-06T07:22:53.004960+0000","last_became_peered":"2024-03-06T07:22:53.004960+0000","last_unstale":"2024-03-06T07:22:54.282148+0000","last_undegraded":"2024-03-06T07:22:54.282148+0000","last_fullsized":"2024-03-06T07:22:54.282148+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:08.594990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281482+0000","last_change":"2024-03-06T07:22:52.006431+0000","last_active":"2024-03-06T07:22:54.281482+0000","last_peered":"2024-03-06T07:22:54.281482+0000","last_clean":"2024-03-06T07:22:54.281482+0000","last_became_active":"2024-03-06T07:22:52.006190+0000","last_became_peered":"2024-03-06T07:22:52.006190+0000","last_unstale":"2024-03-06T07:22:54.281482+0000","last_undegraded":"2024-03-06T07:22:54.281482+0000","last_fullsized":"2024-03-06T07:22:54.281482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006998+0000","last_change":"2024-03-06T07:22:53.004526+0000","last_active":"2024-03-06T07:22:54.006998+0000","last_peered":"2024-03-06T07:22:54.006998+0000","last_clean":"2024-03-06T07:22:54.006998+0000","last_became_active":"2024-03-06T07:22:53.003998+0000","last_became_peered":"2024-03-06T07:22:53.003998+0000","last_unstale":"2024-03-06T07:22:54.006998+0000","last_undegraded":"2024-03-06T07:22:54.006998+0000","last_fullsized":"2024-03-06T07:22:54.006998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:42.114122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281911+0000","last_change":"2024-03-06T07:22:52.002503+0000","last_active":"2024-03-06T07:22:54.281911+0000","last_peered":"2024-03-06T07:22:54.281911+0000","last_clean":"2024-03-06T07:22:54.281911+0000","last_became_active":"2024-03-06T07:22:52.001554+0000","last_became_peered":"2024-03-06T07:22:52.001554+0000","last_unstale":"2024-03-06T07:22:54.281911+0000","last_undegraded":"2024-03-06T07:22:54.281911+0000","last_fullsized":"2024-03-06T07:22:54.281911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:41.112020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282874+0000","last_change":"2024-03-06T07:22:53.007256+0000","last_active":"2024-03-06T07:22:54.282874+0000","last_peered":"2024-03-06T07:22:54.282874+0000","last_clean":"2024-03-06T07:22:54.282874+0000","last_became_active":"2024-03-06T07:22:53.006985+0000","last_became_peered":"2024-03-06T07:22:53.006985+0000","last_unstale":"2024-03-06T07:22:54.282874+0000","last_undegraded":"2024-03-06T07:22:54.282874+0000","last_fullsized":"2024-03-06T07:22:54.282874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:04.433486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007006+0000","last_change":"2024-03-06T07:22:51.996405+0000","last_active":"2024-03-06T07:22:54.007006+0000","last_peered":"2024-03-06T07:22:54.007006+0000","last_clean":"2024-03-06T07:22:54.007006+0000","last_became_active":"2024-03-06T07:22:51.996082+0000","last_became_peered":"2024-03-06T07:22:51.996082+0000","last_unstale":"2024-03-06T07:22:54.007006+0000","last_undegraded":"2024-03-06T07:22:54.007006+0000","last_fullsized":"2024-03-06T07:22:54.007006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:57.305195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283827+0000","last_change":"2024-03-06T07:22:53.018922+0000","last_active":"2024-03-06T07:22:54.283827+0000","last_peered":"2024-03-06T07:22:54.283827+0000","last_clean":"2024-03-06T07:22:54.283827+0000","last_became_active":"2024-03-06T07:22:53.018664+0000","last_became_peered":"2024-03-06T07:22:53.018664+0000","last_unstale":"2024-03-06T07:22:54.283827+0000","last_undegraded":"2024-03-06T07:22:54.283827+0000","last_fullsized":"2024-03-06T07:22:54.283827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:04.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282921+0000","last_change":"2024-03-06T07:22:52.002762+0000","last_active":"2024-03-06T07:22:54.282921+0000","last_peered":"2024-03-06T07:22:54.282921+0000","last_clean":"2024-03-06T07:22:54.282921+0000","last_became_active":"2024-03-06T07:22:52.002461+0000","last_became_peered":"2024-03-06T07:22:52.002461+0000","last_unstale":"2024-03-06T07:22:54.282921+0000","last_undegraded":"2024-03-06T07:22:54.282921+0000","last_fullsized":"2024-03-06T07:22:54.282921+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.704065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006733+0000","last_change":"2024-03-06T07:22:53.006268+0000","last_active":"2024-03-06T07:22:54.006733+0000","last_peered":"2024-03-06T07:22:54.006733+0000","last_clean":"2024-03-06T07:22:54.006733+0000","last_became_active":"2024-03-06T07:22:53.005993+0000","last_became_peered":"2024-03-06T07:22:53.005993+0000","last_unstale":"2024-03-06T07:22:54.006733+0000","last_undegraded":"2024-03-06T07:22:54.006733+0000","last_fullsized":"2024-03-06T07:22:54.006733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:04.078810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282767+0000","last_change":"2024-03-06T07:22:52.004453+0000","last_active":"2024-03-06T07:22:54.282767+0000","last_peered":"2024-03-06T07:22:54.282767+0000","last_clean":"2024-03-06T07:22:54.282767+0000","last_became_active":"2024-03-06T07:22:52.003997+0000","last_became_peered":"2024-03-06T07:22:52.003997+0000","last_unstale":"2024-03-06T07:22:54.282767+0000","last_undegraded":"2024-03-06T07:22:54.282767+0000","last_fullsized":"2024-03-06T07:22:54.282767+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:37.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281621+0000","last_change":"2024-03-06T07:22:53.014780+0000","last_active":"2024-03-06T07:22:54.281621+0000","last_peered":"2024-03-06T07:22:54.281621+0000","last_clean":"2024-03-06T07:22:54.281621+0000","last_became_active":"2024-03-06T07:22:53.014501+0000","last_became_peered":"2024-03-06T07:22:53.014501+0000","last_unstale":"2024-03-06T07:22:54.281621+0000","last_undegraded":"2024-03-06T07:22:54.281621+0000","last_fullsized":"2024-03-06T07:22:54.281621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.631542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282752+0000","last_change":"2024-03-06T07:22:52.002095+0000","last_active":"2024-03-06T07:22:54.282752+0000","last_peered":"2024-03-06T07:22:54.282752+0000","last_clean":"2024-03-06T07:22:54.282752+0000","last_became_active":"2024-03-06T07:22:52.001749+0000","last_became_peered":"2024-03-06T07:22:52.001749+0000","last_unstale":"2024-03-06T07:22:54.282752+0000","last_undegraded":"2024-03-06T07:22:54.282752+0000","last_fullsized":"2024-03-06T07:22:54.282752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.963836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281407+0000","last_change":"2024-03-06T07:22:53.020156+0000","last_active":"2024-03-06T07:22:54.281407+0000","last_peered":"2024-03-06T07:22:54.281407+0000","last_clean":"2024-03-06T07:22:54.281407+0000","last_became_active":"2024-03-06T07:22:53.019845+0000","last_became_peered":"2024-03-06T07:22:53.019845+0000","last_unstale":"2024-03-06T07:22:54.281407+0000","last_undegraded":"2024-03-06T07:22:54.281407+0000","last_fullsized":"2024-03-06T07:22:54.281407+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:11.022830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004216+0000","last_change":"2024-03-06T07:22:51.998103+0000","last_active":"2024-03-06T07:22:54.004216+0000","last_peered":"2024-03-06T07:22:54.004216+0000","last_clean":"2024-03-06T07:22:54.004216+0000","last_became_active":"2024-03-06T07:22:51.997234+0000","last_became_peered":"2024-03-06T07:22:51.997234+0000","last_unstale":"2024-03-06T07:22:54.004216+0000","last_undegraded":"2024-03-06T07:22:54.004216+0000","last_fullsized":"2024-03-06T07:22:54.004216+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:24.629441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281560+0000","last_change":"2024-03-06T07:22:53.018761+0000","last_active":"2024-03-06T07:22:54.281560+0000","last_peered":"2024-03-06T07:22:54.281560+0000","last_clean":"2024-03-06T07:22:54.281560+0000","last_became_active":"2024-03-06T07:22:53.018435+0000","last_became_peered":"2024-03-06T07:22:53.018435+0000","last_unstale":"2024-03-06T07:22:54.281560+0000","last_undegraded":"2024-03-06T07:22:54.281560+0000","last_fullsized":"2024-03-06T07:22:54.281560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:57.965937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007049+0000","last_change":"2024-03-06T07:22:51.997725+0000","last_active":"2024-03-06T07:22:54.007049+0000","last_peered":"2024-03-06T07:22:54.007049+0000","last_clean":"2024-03-06T07:22:54.007049+0000","last_became_active":"2024-03-06T07:22:51.997442+0000","last_became_peered":"2024-03-06T07:22:51.997442+0000","last_unstale":"2024-03-06T07:22:54.007049+0000","last_undegraded":"2024-03-06T07:22:54.007049+0000","last_fullsized":"2024-03-06T07:22:54.007049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.963836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281313+0000","last_change":"2024-03-06T07:22:53.018347+0000","last_active":"2024-03-06T07:22:54.281313+0000","last_peered":"2024-03-06T07:22:54.281313+0000","last_clean":"2024-03-06T07:22:54.281313+0000","last_became_active":"2024-03-06T07:22:53.018073+0000","last_became_peered":"2024-03-06T07:22:53.018073+0000","last_unstale":"2024-03-06T07:22:54.281313+0000","last_undegraded":"2024-03-06T07:22:54.281313+0000","last_fullsized":"2024-03-06T07:22:54.281313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:24.063180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283184+0000","last_change":"2024-03-06T07:22:52.002891+0000","last_active":"2024-03-06T07:22:54.283184+0000","last_peered":"2024-03-06T07:22:54.283184+0000","last_clean":"2024-03-06T07:22:54.283184+0000","last_became_active":"2024-03-06T07:22:52.002607+0000","last_became_peered":"2024-03-06T07:22:52.002607+0000","last_unstale":"2024-03-06T07:22:54.283184+0000","last_undegraded":"2024-03-06T07:22:54.283184+0000","last_fullsized":"2024-03-06T07:22:54.283184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:25.059867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281923+0000","last_change":"2024-03-06T07:22:53.006187+0000","last_active":"2024-03-06T07:22:54.281923+0000","last_peered":"2024-03-06T07:22:54.281923+0000","last_clean":"2024-03-06T07:22:54.281923+0000","last_became_active":"2024-03-06T07:22:53.005793+0000","last_became_peered":"2024-03-06T07:22:53.005793+0000","last_unstale":"2024-03-06T07:22:54.281923+0000","last_undegraded":"2024-03-06T07:22:54.281923+0000","last_fullsized":"2024-03-06T07:22:54.281923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283061+0000","last_change":"2024-03-06T07:22:52.006361+0000","last_active":"2024-03-06T07:22:54.283061+0000","last_peered":"2024-03-06T07:22:54.283061+0000","last_clean":"2024-03-06T07:22:54.283061+0000","last_became_active":"2024-03-06T07:22:52.006055+0000","last_became_peered":"2024-03-06T07:22:52.006055+0000","last_unstale":"2024-03-06T07:22:54.283061+0000","last_undegraded":"2024-03-06T07:22:54.283061+0000","last_fullsized":"2024-03-06T07:22:54.283061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:41.112020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281531+0000","last_change":"2024-03-06T07:22:53.020189+0000","last_active":"2024-03-06T07:22:54.281531+0000","last_peered":"2024-03-06T07:22:54.281531+0000","last_clean":"2024-03-06T07:22:54.281531+0000","last_became_active":"2024-03-06T07:22:53.019828+0000","last_became_peered":"2024-03-06T07:22:53.019828+0000","last_unstale":"2024-03-06T07:22:54.281531+0000","last_undegraded":"2024-03-06T07:22:54.281531+0000","last_fullsized":"2024-03-06T07:22:54.281531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:23.159984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007053+0000","last_change":"2024-03-06T07:22:52.000983+0000","last_active":"2024-03-06T07:22:54.007053+0000","last_peered":"2024-03-06T07:22:54.007053+0000","last_clean":"2024-03-06T07:22:54.007053+0000","last_became_active":"2024-03-06T07:22:52.000680+0000","last_became_peered":"2024-03-06T07:22:52.000680+0000","last_unstale":"2024-03-06T07:22:54.007053+0000","last_undegraded":"2024-03-06T07:22:54.007053+0000","last_fullsized":"2024-03-06T07:22:54.007053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283816+0000","last_change":"2024-03-06T07:22:53.014876+0000","last_active":"2024-03-06T07:22:54.283816+0000","last_peered":"2024-03-06T07:22:54.283816+0000","last_clean":"2024-03-06T07:22:54.283816+0000","last_became_active":"2024-03-06T07:22:53.014322+0000","last_became_peered":"2024-03-06T07:22:53.014322+0000","last_unstale":"2024-03-06T07:22:54.283816+0000","last_undegraded":"2024-03-06T07:22:54.283816+0000","last_fullsized":"2024-03-06T07:22:54.283816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:19.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004490+0000","last_change":"2024-03-06T07:22:51.998753+0000","last_active":"2024-03-06T07:22:54.004490+0000","last_peered":"2024-03-06T07:22:54.004490+0000","last_clean":"2024-03-06T07:22:54.004490+0000","last_became_active":"2024-03-06T07:22:51.998428+0000","last_became_peered":"2024-03-06T07:22:51.998428+0000","last_unstale":"2024-03-06T07:22:54.004490+0000","last_undegraded":"2024-03-06T07:22:54.004490+0000","last_fullsized":"2024-03-06T07:22:54.004490+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004285+0000","last_change":"2024-03-06T07:22:53.012683+0000","last_active":"2024-03-06T07:22:54.004285+0000","last_peered":"2024-03-06T07:22:54.004285+0000","last_clean":"2024-03-06T07:22:54.004285+0000","last_became_active":"2024-03-06T07:22:53.012154+0000","last_became_peered":"2024-03-06T07:22:53.012154+0000","last_unstale":"2024-03-06T07:22:54.004285+0000","last_undegraded":"2024-03-06T07:22:54.004285+0000","last_fullsized":"2024-03-06T07:22:54.004285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006819+0000","last_change":"2024-03-06T07:22:52.004074+0000","last_active":"2024-03-06T07:22:54.006819+0000","last_peered":"2024-03-06T07:22:54.006819+0000","last_clean":"2024-03-06T07:22:54.006819+0000","last_became_active":"2024-03-06T07:22:52.003773+0000","last_became_peered":"2024-03-06T07:22:52.003773+0000","last_unstale":"2024-03-06T07:22:54.006819+0000","last_undegraded":"2024-03-06T07:22:54.006819+0000","last_fullsized":"2024-03-06T07:22:54.006819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:43.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007154+0000","last_change":"2024-03-06T07:22:53.001845+0000","last_active":"2024-03-06T07:22:54.007154+0000","last_peered":"2024-03-06T07:22:54.007154+0000","last_clean":"2024-03-06T07:22:54.007154+0000","last_became_active":"2024-03-06T07:22:53.001480+0000","last_became_peered":"2024-03-06T07:22:53.001480+0000","last_unstale":"2024-03-06T07:22:54.007154+0000","last_undegraded":"2024-03-06T07:22:54.007154+0000","last_fullsized":"2024-03-06T07:22:54.007154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:51.472365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004349+0000","last_change":"2024-03-06T07:22:52.001878+0000","last_active":"2024-03-06T07:22:54.004349+0000","last_peered":"2024-03-06T07:22:54.004349+0000","last_clean":"2024-03-06T07:22:54.004349+0000","last_became_active":"2024-03-06T07:22:52.001618+0000","last_became_peered":"2024-03-06T07:22:52.001618+0000","last_unstale":"2024-03-06T07:22:54.004349+0000","last_undegraded":"2024-03-06T07:22:54.004349+0000","last_fullsized":"2024-03-06T07:22:54.004349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:57.305195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004250+0000","last_change":"2024-03-06T07:22:53.010139+0000","last_active":"2024-03-06T07:22:54.004250+0000","last_peered":"2024-03-06T07:22:54.004250+0000","last_clean":"2024-03-06T07:22:54.004250+0000","last_became_active":"2024-03-06T07:22:53.009822+0000","last_became_peered":"2024-03-06T07:22:53.009822+0000","last_unstale":"2024-03-06T07:22:54.004250+0000","last_undegraded":"2024-03-06T07:22:54.004250+0000","last_fullsized":"2024-03-06T07:22:54.004250+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:50.691781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006648+0000","last_change":"2024-03-06T07:22:51.999813+0000","last_active":"2024-03-06T07:22:54.006648+0000","last_peered":"2024-03-06T07:22:54.006648+0000","last_clean":"2024-03-06T07:22:54.006648+0000","last_became_active":"2024-03-06T07:22:51.999483+0000","last_became_peered":"2024-03-06T07:22:51.999483+0000","last_unstale":"2024-03-06T07:22:54.006648+0000","last_undegraded":"2024-03-06T07:22:54.006648+0000","last_fullsized":"2024-03-06T07:22:54.006648+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:16.012744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281797+0000","last_change":"2024-03-06T07:22:53.011139+0000","last_active":"2024-03-06T07:22:54.281797+0000","last_peered":"2024-03-06T07:22:54.281797+0000","last_clean":"2024-03-06T07:22:54.281797+0000","last_became_active":"2024-03-06T07:22:53.010277+0000","last_became_peered":"2024-03-06T07:22:53.010277+0000","last_unstale":"2024-03-06T07:22:54.281797+0000","last_undegraded":"2024-03-06T07:22:54.281797+0000","last_fullsized":"2024-03-06T07:22:54.281797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283282+0000","last_change":"2024-03-06T07:22:52.003098+0000","last_active":"2024-03-06T07:22:54.283282+0000","last_peered":"2024-03-06T07:22:54.283282+0000","last_clean":"2024-03-06T07:22:54.283282+0000","last_became_active":"2024-03-06T07:22:52.002798+0000","last_became_peered":"2024-03-06T07:22:52.002798+0000","last_unstale":"2024-03-06T07:22:54.283282+0000","last_undegraded":"2024-03-06T07:22:54.283282+0000","last_fullsized":"2024-03-06T07:22:54.283282+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:22.157883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283083+0000","last_change":"2024-03-06T07:22:53.003411+0000","last_active":"2024-03-06T07:22:54.283083+0000","last_peered":"2024-03-06T07:22:54.283083+0000","last_clean":"2024-03-06T07:22:54.283083+0000","last_became_active":"2024-03-06T07:22:53.003120+0000","last_became_peered":"2024-03-06T07:22:53.003120+0000","last_unstale":"2024-03-06T07:22:54.283083+0000","last_undegraded":"2024-03-06T07:22:54.283083+0000","last_fullsized":"2024-03-06T07:22:54.283083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:04.078810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281937+0000","last_change":"2024-03-06T07:22:52.007081+0000","last_active":"2024-03-06T07:22:54.281937+0000","last_peered":"2024-03-06T07:22:54.281937+0000","last_clean":"2024-03-06T07:22:54.281937+0000","last_became_active":"2024-03-06T07:22:52.006599+0000","last_became_peered":"2024-03-06T07:22:52.006599+0000","last_unstale":"2024-03-06T07:22:54.281937+0000","last_undegraded":"2024-03-06T07:22:54.281937+0000","last_fullsized":"2024-03-06T07:22:54.281937+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:50.470263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282354+0000","last_change":"2024-03-06T07:22:53.007286+0000","last_active":"2024-03-06T07:22:54.282354+0000","last_peered":"2024-03-06T07:22:54.282354+0000","last_clean":"2024-03-06T07:22:54.282354+0000","last_became_active":"2024-03-06T07:22:53.007032+0000","last_became_peered":"2024-03-06T07:22:53.007032+0000","last_unstale":"2024-03-06T07:22:54.282354+0000","last_undegraded":"2024-03-06T07:22:54.282354+0000","last_fullsized":"2024-03-06T07:22:54.282354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007110+0000","last_change":"2024-03-06T07:22:52.003894+0000","last_active":"2024-03-06T07:22:54.007110+0000","last_peered":"2024-03-06T07:22:54.007110+0000","last_clean":"2024-03-06T07:22:54.007110+0000","last_became_active":"2024-03-06T07:22:52.003608+0000","last_became_peered":"2024-03-06T07:22:52.003608+0000","last_unstale":"2024-03-06T07:22:54.007110+0000","last_undegraded":"2024-03-06T07:22:54.007110+0000","last_fullsized":"2024-03-06T07:22:54.007110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:25.059867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282579+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:22:54.282579+0000","last_peered":"2024-03-06T07:22:54.282579+0000","last_clean":"2024-03-06T07:22:54.282579+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:22:54.282579+0000","last_undegraded":"2024-03-06T07:22:54.282579+0000","last_fullsized":"2024-03-06T07:22:54.282579+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":736976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:57.826 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.981+0000 7f41569f7640 1 Processor -- start 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.981+0000 7f41569f7640 1 -- start start 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f41569f7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f415015e010 0x7f41500cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f41569f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415015e3f0 con 0x7f41500b3240 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f41569f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41500b1c10 con 0x7f41500d1ad0 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f41569f7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41500b1f40 con 0x7f415015e010 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f41500b3240 0x7f41500b23d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56634/0 (socket says 172.21.15.122:56634) 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414f7fe640 1 -- 172.21.15.122:0/1886686942 learned_addr learned my addr 172.21.15.122:0/1886686942 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414ffff640 1 --2- 172.21.15.122:0/1886686942 >> v2:172.21.15.169:3300/0 conn(0x7f415015e010 0x7f41500cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235179799 0 0) 0x7f415015e3f0 con 0x7f41500b3240 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4138003520 con 0x7f41500b3240 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327987608 0 0) 0x7f4138003520 con 0x7f41500b3240 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 >> v2:172.21.15.169:3300/0 conn(0x7f415015e010 msgr2=0x7f41500cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 --2- 172.21.15.122:0/1886686942 >> v2:172.21.15.169:3300/0 conn(0x7f415015e010 0x7f41500cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 >> v1:172.21.15.169:6789/0 conn(0x7f41500d1ad0 legacy=0x7f41500cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415006d1b0 con 0x7f41500b3240 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4140001670 con 0x7f41500b3240 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 >> v1:172.21.15.122:6789/0 conn(0x7f41500b3240 legacy=0x7f41500b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41380039b0 0x7f4138003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.982+0000 7f414effd640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f4138004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414effd640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41380051c0 0x7f413800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41500b1f40 con 0x7f4138004800 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413800c160 con 0x7f41380039b0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413800c360 con 0x7f41380051c0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414f7fe640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f4138004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414ffff640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41380039b0 0x7f4138003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f4154f94640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41380051c0 0x7f413800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414f7fe640 1 -- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41380051c0 msgr2=0x7f413800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414f7fe640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41380051c0 0x7f413800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414f7fe640 1 -- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41380039b0 msgr2=0x7f4138003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414f7fe640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41380039b0 0x7f4138003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f414f7fe640 1 -- 172.21.15.122:0/1886686942 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4138003520 con 0x7f4138004800 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f4154f94640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41380051c0 0x7f413800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.983+0000 7f4154f94640 1 -- 172.21.15.122:0/1886686942 reap_dead start 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f414f7fe640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f4138004c80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f414000c240 tx=0x7f4138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cdfb9c414e444e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140007d10 con 0x7f4138004800 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4140030070 con 0x7f4138004800 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f414effd640 1 -- 172.21.15.122:0/1886686942 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414002cda0 con 0x7f4138004800 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 -- 172.21.15.122:0/1886686942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 msgr2=0x7f4138004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f4138004c80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f414000c240 tx=0x7f4138005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 -- 172.21.15.122:0/1886686942 shutdown_connections 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 --2- 172.21.15.122:0/1886686942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f4138004c80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 -- 172.21.15.122:0/1886686942 >> 172.21.15.122:0/1886686942 conn(0x7f415009a9f0 msgr2=0x7f415009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 -- 172.21.15.122:0/1886686942 shutdown_connections 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.984+0000 7f41569f7640 1 -- 172.21.15.122:0/1886686942 wait complete. 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.985+0000 7f41569f7640 1 Processor -- start 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 -- start start 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f41500b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41500b5e70 0x7f41501f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41500c6e70 0x7f41500b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41500d1f40 con 0x7f4138004800 2024-03-06T07:22:57.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415006ced0 con 0x7f41500c6e70 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f41569f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41500d20c0 con 0x7f41500b5e70 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.986+0000 7f414ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f41500b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f41500b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38516/0 (socket says 172.21.15.122:38516) 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 -- 172.21.15.122:0/386249250 learned_addr learned my addr 172.21.15.122:0/386249250 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f4154f94640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41500c6e70 0x7f41500b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414f7fe640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41500b5e70 0x7f41501f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 -- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41500b5e70 msgr2=0x7f41501f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41500b5e70 0x7f41501f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 -- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41500c6e70 msgr2=0x7f41500b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41500c6e70 0x7f41500b66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 -- 172.21.15.122:0/386249250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41500b4dd0 con 0x7f4138004800 2024-03-06T07:22:57.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414ffff640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f41500b4740 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4144011aa0 tx=0x7f4144011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400fb20 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414400fcc0 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41500b50b0 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144014660 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.987+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41500b5670 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.988+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415004f330 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.989+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4144012040 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.989+0000 7f414cff9640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f413c06b500 0x7f413c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.989+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f414408a8d0 con 0x7f4138004800 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.989+0000 7f414f7fe640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f413c06b500 0x7f413c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.990+0000 7f414f7fe640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f413c06b500 0x7f413c06d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4140034000 tx=0x7f4140009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:57.993+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4144019030 con 0x7f4138004800 2024-03-06T07:22:58.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.273+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f41500d2980 con 0x7f4138004800 2024-03-06T07:22:58.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.274+0000 7f414cff9640 1 -- 172.21.15.122:0/386249250 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1431 (secure 0 0 0) 0x7f4144059750 con 0x7f4138004800 2024-03-06T07:22:58.276 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:58.276 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":89,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":225591296,"bytes_avail":772868521984,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5262}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:22:47.341990+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f413c06b500 msgr2=0x7f413c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f413c06b500 0x7f413c06d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4140034000 tx=0x7f4140009030 comp rx=0 tx=0).stop 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 msgr2=0x7f41500b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f41500b4740 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4144011aa0 tx=0x7f4144011f60 comp rx=0 tx=0).stop 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 shutdown_connections 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f413c06b500 0x7f413c06d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41500c6e70 0x7f41500b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41500b5e70 0x7f41501f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 --2- 172.21.15.122:0/386249250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138004800 0x7f41500b4740 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.276+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 >> 172.21.15.122:0/386249250 conn(0x7f415009a9f0 msgr2=0x7f41500cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.277+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 shutdown_connections 2024-03-06T07:22:58.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.277+0000 7f41569f7640 1 -- 172.21.15.122:0/386249250 wait complete. 2024-03-06T07:22:58.291 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:22:58.291 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:58.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.431+0000 7fec51b3a640 1 Processor -- start 2024-03-06T07:22:58.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.431+0000 7fec51b3a640 1 -- start start 2024-03-06T07:22:58.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.431+0000 7fec51b3a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fec4c0d2710 0x7fec4c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.431+0000 7fec51b3a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0b3430 con 0x7fec4c0d22b0 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.431+0000 7fec51b3a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0b3710 con 0x7fec4c15dfe0 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.431+0000 7fec51b3a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4c0b31e0 con 0x7fec4c0d2710 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec4bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fec4c0d2710 0x7fec4c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec43fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fec4c0d22b0 0x7fec4c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56644/0 (socket says 172.21.15.122:56644) 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec43fff640 1 -- 172.21.15.122:0/2429099338 learned_addr learned my addr 172.21.15.122:0/2429099338 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec4bfff640 1 -- 172.21.15.122:0/2429099338 >> v1:172.21.15.169:6789/0 conn(0x7fec4c15dfe0 legacy=0x7fec4c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec4bfff640 1 -- 172.21.15.122:0/2429099338 >> v1:172.21.15.122:6789/0 conn(0x7fec4c0d22b0 legacy=0x7fec4c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec4affd640 1 -- 172.21.15.122:0/2429099338 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960856207 0 0) 0x7fec4c0b3430 con 0x7fec4c0d22b0 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.432+0000 7fec4bfff640 1 -- 172.21.15.122:0/2429099338 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c0b39f0 con 0x7fec4c0d2710 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.433+0000 7fec4bfff640 1 --2- 172.21.15.122:0/2429099338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d2710 0x7fec4c0b4900 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fec44009dc0 tx=0x7fec4402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e051fefe4d3367a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.433+0000 7fec4affd640 1 -- 172.21.15.122:0/2429099338 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec44039070 con 0x7fec4c0d2710 2024-03-06T07:22:58.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.433+0000 7fec4affd640 1 -- 172.21.15.122:0/2429099338 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4402cc40 con 0x7fec4c0d2710 2024-03-06T07:22:58.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.433+0000 7fec4affd640 1 -- 172.21.15.122:0/2429099338 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec44034440 con 0x7fec4c0d2710 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 -- 172.21.15.122:0/2429099338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d2710 msgr2=0x7fec4c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 --2- 172.21.15.122:0/2429099338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d2710 0x7fec4c0b4900 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fec44009dc0 tx=0x7fec4402be90 comp rx=0 tx=0).stop 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 -- 172.21.15.122:0/2429099338 shutdown_connections 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 --2- 172.21.15.122:0/2429099338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d2710 0x7fec4c0b4900 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 -- 172.21.15.122:0/2429099338 >> 172.21.15.122:0/2429099338 conn(0x7fec4c0d3cd0 msgr2=0x7fec4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 -- 172.21.15.122:0/2429099338 shutdown_connections 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.434+0000 7fec51b3a640 1 -- 172.21.15.122:0/2429099338 wait complete. 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.435+0000 7fec51b3a640 1 Processor -- start 2024-03-06T07:22:58.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.435+0000 7fec51b3a640 1 -- start start 2024-03-06T07:22:58.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.435+0000 7fec51b3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 0x7fec4c15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.435+0000 7fec51b3a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec4c0d2710 0x7fec4c15b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec51b3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec4c15dfe0 0x7fec4c06ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec51b3a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0cfa00 con 0x7fec4c0d22b0 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec51b3a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0b3d60 con 0x7fec4c0d2710 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec51b3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0d1c70 con 0x7fec4c15dfe0 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 0x7fec4c15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 0x7fec4c15edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38528/0 (socket says 172.21.15.122:38528) 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 -- 172.21.15.122:0/3434152961 learned_addr learned my addr 172.21.15.122:0/3434152961 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec43fff640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec4c0d2710 0x7fec4c15b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4bfff640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec4c15dfe0 0x7fec4c06ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 -- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec4c15dfe0 msgr2=0x7fec4c06ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec4c15dfe0 0x7fec4c06ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 -- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec4c0d2710 msgr2=0x7fec4c15b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec4c0d2710 0x7fec4c15b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 -- 172.21.15.122:0/3434152961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c15f600 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec43fff640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec4c0d2710 0x7fec4c15b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4bfff640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec4c15dfe0 0x7fec4c06ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec4b7fe640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 0x7fec4c15edd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b3f0 tx=0x7fec3c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c015020 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c011830 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c011c20 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4c15f8e0 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.436+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c15fea0 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.438+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4c04f330 con 0x7fec4c0d22b0 2024-03-06T07:22:58.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.438+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec3c018020 con 0x7fec4c0d22b0 2024-03-06T07:22:58.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.438+0000 7fec48ff9640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2c06b4d0 0x7fec2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.438+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec3c089e50 con 0x7fec4c0d22b0 2024-03-06T07:22:58.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.438+0000 7fec43fff640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2c06b4d0 0x7fec2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.439+0000 7fec43fff640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2c06b4d0 0x7fec2c06d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0cc430 tx=0x7fec34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.442+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec3c01f030 con 0x7fec4c0d22b0 2024-03-06T07:22:58.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.617+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec4c0b0af0 con 0x7fec2c06b4d0 2024-03-06T07:22:58.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.623+0000 7fec48ff9640 1 -- 172.21.15.122:0/3434152961 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251948 (secure 0 0 0) 0x7fec4c0b0af0 con 0x7fec2c06b4d0 2024-03-06T07:22:58.628 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:22:58.629 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.626+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2c06b4d0 msgr2=0x7fec2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.626+0000 7fec51b3a640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2c06b4d0 0x7fec2c06d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0cc430 tx=0x7fec34009040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.626+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 msgr2=0x7fec4c15edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.626+0000 7fec51b3a640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 0x7fec4c15edd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b3f0 tx=0x7fec3c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.626+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 shutdown_connections 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2c06b4d0 0x7fec2c06d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec4c15dfe0 0x7fec4c06ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec4c0d2710 0x7fec4c15b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 --2- 172.21.15.122:0/3434152961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec4c0d22b0 0x7fec4c15edd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 >> 172.21.15.122:0/3434152961 conn(0x7fec4c0d3cd0 msgr2=0x7fec4c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 shutdown_connections 2024-03-06T07:22:58.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:22:58.627+0000 7fec51b3a640 1 -- 172.21.15.122:0/3434152961 wait complete. 2024-03-06T07:22:58.639 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-06T07:22:57.343477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220304,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754416,"statfs":{"total":773094113280,"available":772868521984,"internally_reserved":0,"allocated":4235264,"data_stored":2358420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000931"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007270+0000","last_change":"2024-03-06T07:22:53.015900+0000","last_active":"2024-03-06T07:22:54.007270+0000","last_peered":"2024-03-06T07:22:54.007270+0000","last_clean":"2024-03-06T07:22:54.007270+0000","last_became_active":"2024-03-06T07:22:53.015567+0000","last_became_peered":"2024-03-06T07:22:53.015567+0000","last_unstale":"2024-03-06T07:22:54.007270+0000","last_undegraded":"2024-03-06T07:22:54.007270+0000","last_fullsized":"2024-03-06T07:22:54.007270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:08.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282163+0000","last_change":"2024-03-06T07:22:53.005999+0000","last_active":"2024-03-06T07:22:54.282163+0000","last_peered":"2024-03-06T07:22:54.282163+0000","last_clean":"2024-03-06T07:22:54.282163+0000","last_became_active":"2024-03-06T07:22:53.005727+0000","last_became_peered":"2024-03-06T07:22:53.005727+0000","last_unstale":"2024-03-06T07:22:54.282163+0000","last_undegraded":"2024-03-06T07:22:54.282163+0000","last_fullsized":"2024-03-06T07:22:54.282163+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:04.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283677+0000","last_change":"2024-03-06T07:22:53.009774+0000","last_active":"2024-03-06T07:22:54.283677+0000","last_peered":"2024-03-06T07:22:54.283677+0000","last_clean":"2024-03-06T07:22:54.283677+0000","last_became_active":"2024-03-06T07:22:53.009488+0000","last_became_peered":"2024-03-06T07:22:53.009488+0000","last_unstale":"2024-03-06T07:22:54.283677+0000","last_undegraded":"2024-03-06T07:22:54.283677+0000","last_fullsized":"2024-03-06T07:22:54.283677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006758+0000","last_change":"2024-03-06T07:22:53.019399+0000","last_active":"2024-03-06T07:22:54.006758+0000","last_peered":"2024-03-06T07:22:54.006758+0000","last_clean":"2024-03-06T07:22:54.006758+0000","last_became_active":"2024-03-06T07:22:53.019122+0000","last_became_peered":"2024-03-06T07:22:53.019122+0000","last_unstale":"2024-03-06T07:22:54.006758+0000","last_undegraded":"2024-03-06T07:22:54.006758+0000","last_fullsized":"2024-03-06T07:22:54.006758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281419+0000","last_change":"2024-03-06T07:22:53.020095+0000","last_active":"2024-03-06T07:22:54.281419+0000","last_peered":"2024-03-06T07:22:54.281419+0000","last_clean":"2024-03-06T07:22:54.281419+0000","last_became_active":"2024-03-06T07:22:53.019764+0000","last_became_peered":"2024-03-06T07:22:53.019764+0000","last_unstale":"2024-03-06T07:22:54.281419+0000","last_undegraded":"2024-03-06T07:22:54.281419+0000","last_fullsized":"2024-03-06T07:22:54.281419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:16.936064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281986+0000","last_change":"2024-03-06T07:22:53.011642+0000","last_active":"2024-03-06T07:22:54.281986+0000","last_peered":"2024-03-06T07:22:54.281986+0000","last_clean":"2024-03-06T07:22:54.281986+0000","last_became_active":"2024-03-06T07:22:53.011303+0000","last_became_peered":"2024-03-06T07:22:53.011303+0000","last_unstale":"2024-03-06T07:22:54.281986+0000","last_undegraded":"2024-03-06T07:22:54.281986+0000","last_fullsized":"2024-03-06T07:22:54.281986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282515+0000","last_change":"2024-03-06T07:22:53.009740+0000","last_active":"2024-03-06T07:22:54.282515+0000","last_peered":"2024-03-06T07:22:54.282515+0000","last_clean":"2024-03-06T07:22:54.282515+0000","last_became_active":"2024-03-06T07:22:53.009431+0000","last_became_peered":"2024-03-06T07:22:53.009431+0000","last_unstale":"2024-03-06T07:22:54.282515+0000","last_undegraded":"2024-03-06T07:22:54.282515+0000","last_fullsized":"2024-03-06T07:22:54.282515+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282887+0000","last_change":"2024-03-06T07:22:53.011772+0000","last_active":"2024-03-06T07:22:54.282887+0000","last_peered":"2024-03-06T07:22:54.282887+0000","last_clean":"2024-03-06T07:22:54.282887+0000","last_became_active":"2024-03-06T07:22:53.011391+0000","last_became_peered":"2024-03-06T07:22:53.011391+0000","last_unstale":"2024-03-06T07:22:54.282887+0000","last_undegraded":"2024-03-06T07:22:54.282887+0000","last_fullsized":"2024-03-06T07:22:54.282887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282966+0000","last_change":"2024-03-06T07:22:53.008973+0000","last_active":"2024-03-06T07:22:54.282966+0000","last_peered":"2024-03-06T07:22:54.282966+0000","last_clean":"2024-03-06T07:22:54.282966+0000","last_became_active":"2024-03-06T07:22:53.008608+0000","last_became_peered":"2024-03-06T07:22:53.008608+0000","last_unstale":"2024-03-06T07:22:54.282966+0000","last_undegraded":"2024-03-06T07:22:54.282966+0000","last_fullsized":"2024-03-06T07:22:54.282966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.337698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006851+0000","last_change":"2024-03-06T07:22:53.014766+0000","last_active":"2024-03-06T07:22:54.006851+0000","last_peered":"2024-03-06T07:22:54.006851+0000","last_clean":"2024-03-06T07:22:54.006851+0000","last_became_active":"2024-03-06T07:22:53.014521+0000","last_became_peered":"2024-03-06T07:22:53.014521+0000","last_unstale":"2024-03-06T07:22:54.006851+0000","last_undegraded":"2024-03-06T07:22:54.006851+0000","last_fullsized":"2024-03-06T07:22:54.006851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:01.752069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282270+0000","last_change":"2024-03-06T07:22:53.008566+0000","last_active":"2024-03-06T07:22:54.282270+0000","last_peered":"2024-03-06T07:22:54.282270+0000","last_clean":"2024-03-06T07:22:54.282270+0000","last_became_active":"2024-03-06T07:22:53.008322+0000","last_became_peered":"2024-03-06T07:22:53.008322+0000","last_unstale":"2024-03-06T07:22:54.282270+0000","last_undegraded":"2024-03-06T07:22:54.282270+0000","last_fullsized":"2024-03-06T07:22:54.282270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283396+0000","last_change":"2024-03-06T07:22:53.018934+0000","last_active":"2024-03-06T07:22:54.283396+0000","last_peered":"2024-03-06T07:22:54.283396+0000","last_clean":"2024-03-06T07:22:54.283396+0000","last_became_active":"2024-03-06T07:22:53.018657+0000","last_became_peered":"2024-03-06T07:22:53.018657+0000","last_unstale":"2024-03-06T07:22:54.283396+0000","last_undegraded":"2024-03-06T07:22:54.283396+0000","last_fullsized":"2024-03-06T07:22:54.283396+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281455+0000","last_change":"2024-03-06T07:22:53.017169+0000","last_active":"2024-03-06T07:22:54.281455+0000","last_peered":"2024-03-06T07:22:54.281455+0000","last_clean":"2024-03-06T07:22:54.281455+0000","last_became_active":"2024-03-06T07:22:53.016669+0000","last_became_peered":"2024-03-06T07:22:53.016669+0000","last_unstale":"2024-03-06T07:22:54.281455+0000","last_undegraded":"2024-03-06T07:22:54.281455+0000","last_fullsized":"2024-03-06T07:22:54.281455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:26.061969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282078+0000","last_change":"2024-03-06T07:22:53.011930+0000","last_active":"2024-03-06T07:22:54.282078+0000","last_peered":"2024-03-06T07:22:54.282078+0000","last_clean":"2024-03-06T07:22:54.282078+0000","last_became_active":"2024-03-06T07:22:53.011616+0000","last_became_peered":"2024-03-06T07:22:53.011616+0000","last_unstale":"2024-03-06T07:22:54.282078+0000","last_undegraded":"2024-03-06T07:22:54.282078+0000","last_fullsized":"2024-03-06T07:22:54.282078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:19.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283416+0000","last_change":"2024-03-06T07:22:53.017836+0000","last_active":"2024-03-06T07:22:54.283416+0000","last_peered":"2024-03-06T07:22:54.283416+0000","last_clean":"2024-03-06T07:22:54.283416+0000","last_became_active":"2024-03-06T07:22:53.017495+0000","last_became_peered":"2024-03-06T07:22:53.017495+0000","last_unstale":"2024-03-06T07:22:54.283416+0000","last_undegraded":"2024-03-06T07:22:54.283416+0000","last_fullsized":"2024-03-06T07:22:54.283416+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282521+0000","last_change":"2024-03-06T07:22:53.014100+0000","last_active":"2024-03-06T07:22:54.282521+0000","last_peered":"2024-03-06T07:22:54.282521+0000","last_clean":"2024-03-06T07:22:54.282521+0000","last_became_active":"2024-03-06T07:22:53.013806+0000","last_became_peered":"2024-03-06T07:22:53.013806+0000","last_unstale":"2024-03-06T07:22:54.282521+0000","last_undegraded":"2024-03-06T07:22:54.282521+0000","last_fullsized":"2024-03-06T07:22:54.282521+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282326+0000","last_change":"2024-03-06T07:22:53.009816+0000","last_active":"2024-03-06T07:22:54.282326+0000","last_peered":"2024-03-06T07:22:54.282326+0000","last_clean":"2024-03-06T07:22:54.282326+0000","last_became_active":"2024-03-06T07:22:53.009497+0000","last_became_peered":"2024-03-06T07:22:53.009497+0000","last_unstale":"2024-03-06T07:22:54.282326+0000","last_undegraded":"2024-03-06T07:22:54.282326+0000","last_fullsized":"2024-03-06T07:22:54.282326+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282288+0000","last_change":"2024-03-06T07:22:53.008196+0000","last_active":"2024-03-06T07:22:54.282288+0000","last_peered":"2024-03-06T07:22:54.282288+0000","last_clean":"2024-03-06T07:22:54.282288+0000","last_became_active":"2024-03-06T07:22:53.007916+0000","last_became_peered":"2024-03-06T07:22:53.007916+0000","last_unstale":"2024-03-06T07:22:54.282288+0000","last_undegraded":"2024-03-06T07:22:54.282288+0000","last_fullsized":"2024-03-06T07:22:54.282288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:33.590994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281796+0000","last_change":"2024-03-06T07:22:53.009095+0000","last_active":"2024-03-06T07:22:54.281796+0000","last_peered":"2024-03-06T07:22:54.281796+0000","last_clean":"2024-03-06T07:22:54.281796+0000","last_became_active":"2024-03-06T07:22:53.008716+0000","last_became_peered":"2024-03-06T07:22:53.008716+0000","last_unstale":"2024-03-06T07:22:54.281796+0000","last_undegraded":"2024-03-06T07:22:54.281796+0000","last_fullsized":"2024-03-06T07:22:54.281796+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281677+0000","last_change":"2024-03-06T07:22:53.018889+0000","last_active":"2024-03-06T07:22:54.281677+0000","last_peered":"2024-03-06T07:22:54.281677+0000","last_clean":"2024-03-06T07:22:54.281677+0000","last_became_active":"2024-03-06T07:22:53.018596+0000","last_became_peered":"2024-03-06T07:22:53.018596+0000","last_unstale":"2024-03-06T07:22:54.281677+0000","last_undegraded":"2024-03-06T07:22:54.281677+0000","last_fullsized":"2024-03-06T07:22:54.281677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.014845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282871+0000","last_change":"2024-03-06T07:22:53.002357+0000","last_active":"2024-03-06T07:22:54.282871+0000","last_peered":"2024-03-06T07:22:54.282871+0000","last_clean":"2024-03-06T07:22:54.282871+0000","last_became_active":"2024-03-06T07:22:53.002122+0000","last_became_peered":"2024-03-06T07:22:53.002122+0000","last_unstale":"2024-03-06T07:22:54.282871+0000","last_undegraded":"2024-03-06T07:22:54.282871+0000","last_fullsized":"2024-03-06T07:22:54.282871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:13.547310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282221+0000","last_change":"2024-03-06T07:22:53.011490+0000","last_active":"2024-03-06T07:22:54.282221+0000","last_peered":"2024-03-06T07:22:54.282221+0000","last_clean":"2024-03-06T07:22:54.282221+0000","last_became_active":"2024-03-06T07:22:53.011196+0000","last_became_peered":"2024-03-06T07:22:53.011196+0000","last_unstale":"2024-03-06T07:22:54.282221+0000","last_undegraded":"2024-03-06T07:22:54.282221+0000","last_fullsized":"2024-03-06T07:22:54.282221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004180+0000","last_change":"2024-03-06T07:22:53.005749+0000","last_active":"2024-03-06T07:22:54.004180+0000","last_peered":"2024-03-06T07:22:54.004180+0000","last_clean":"2024-03-06T07:22:54.004180+0000","last_became_active":"2024-03-06T07:22:53.005355+0000","last_became_peered":"2024-03-06T07:22:53.005355+0000","last_unstale":"2024-03-06T07:22:54.004180+0000","last_undegraded":"2024-03-06T07:22:54.004180+0000","last_fullsized":"2024-03-06T07:22:54.004180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281870+0000","last_change":"2024-03-06T07:22:53.015800+0000","last_active":"2024-03-06T07:22:54.281870+0000","last_peered":"2024-03-06T07:22:54.281870+0000","last_clean":"2024-03-06T07:22:54.281870+0000","last_became_active":"2024-03-06T07:22:53.015308+0000","last_became_peered":"2024-03-06T07:22:53.015308+0000","last_unstale":"2024-03-06T07:22:54.281870+0000","last_undegraded":"2024-03-06T07:22:54.281870+0000","last_fullsized":"2024-03-06T07:22:54.281870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006942+0000","last_change":"2024-03-06T07:22:53.006841+0000","last_active":"2024-03-06T07:22:54.006942+0000","last_peered":"2024-03-06T07:22:54.006942+0000","last_clean":"2024-03-06T07:22:54.006942+0000","last_became_active":"2024-03-06T07:22:53.006562+0000","last_became_peered":"2024-03-06T07:22:53.006562+0000","last_unstale":"2024-03-06T07:22:54.006942+0000","last_undegraded":"2024-03-06T07:22:54.006942+0000","last_fullsized":"2024-03-06T07:22:54.006942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:08.594990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282409+0000","last_change":"2024-03-06T07:22:53.003197+0000","last_active":"2024-03-06T07:22:54.282409+0000","last_peered":"2024-03-06T07:22:54.282409+0000","last_clean":"2024-03-06T07:22:54.282409+0000","last_became_active":"2024-03-06T07:22:53.002347+0000","last_became_peered":"2024-03-06T07:22:53.002347+0000","last_unstale":"2024-03-06T07:22:54.282409+0000","last_undegraded":"2024-03-06T07:22:54.282409+0000","last_fullsized":"2024-03-06T07:22:54.282409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:25.191495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006974+0000","last_change":"2024-03-06T07:22:53.007183+0000","last_active":"2024-03-06T07:22:54.006974+0000","last_peered":"2024-03-06T07:22:54.006974+0000","last_clean":"2024-03-06T07:22:54.006974+0000","last_became_active":"2024-03-06T07:22:53.006620+0000","last_became_peered":"2024-03-06T07:22:53.006620+0000","last_unstale":"2024-03-06T07:22:54.006974+0000","last_undegraded":"2024-03-06T07:22:54.006974+0000","last_fullsized":"2024-03-06T07:22:54.006974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283597+0000","last_change":"2024-03-06T07:22:53.014691+0000","last_active":"2024-03-06T07:22:54.283597+0000","last_peered":"2024-03-06T07:22:54.283597+0000","last_clean":"2024-03-06T07:22:54.283597+0000","last_became_active":"2024-03-06T07:22:53.014414+0000","last_became_peered":"2024-03-06T07:22:53.014414+0000","last_unstale":"2024-03-06T07:22:54.283597+0000","last_undegraded":"2024-03-06T07:22:54.283597+0000","last_fullsized":"2024-03-06T07:22:54.283597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282606+0000","last_change":"2024-03-06T07:22:53.011594+0000","last_active":"2024-03-06T07:22:54.282606+0000","last_peered":"2024-03-06T07:22:54.282606+0000","last_clean":"2024-03-06T07:22:54.282606+0000","last_became_active":"2024-03-06T07:22:53.011237+0000","last_became_peered":"2024-03-06T07:22:53.011237+0000","last_unstale":"2024-03-06T07:22:54.282606+0000","last_undegraded":"2024-03-06T07:22:54.282606+0000","last_fullsized":"2024-03-06T07:22:54.282606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:50.691781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282618+0000","last_change":"2024-03-06T07:22:53.006582+0000","last_active":"2024-03-06T07:22:54.282618+0000","last_peered":"2024-03-06T07:22:54.282618+0000","last_clean":"2024-03-06T07:22:54.282618+0000","last_became_active":"2024-03-06T07:22:53.006235+0000","last_became_peered":"2024-03-06T07:22:53.006235+0000","last_unstale":"2024-03-06T07:22:54.282618+0000","last_undegraded":"2024-03-06T07:22:54.282618+0000","last_fullsized":"2024-03-06T07:22:54.282618+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006986+0000","last_change":"2024-03-06T07:22:53.008162+0000","last_active":"2024-03-06T07:22:54.006986+0000","last_peered":"2024-03-06T07:22:54.006986+0000","last_clean":"2024-03-06T07:22:54.006986+0000","last_became_active":"2024-03-06T07:22:53.007160+0000","last_became_peered":"2024-03-06T07:22:53.007160+0000","last_unstale":"2024-03-06T07:22:54.006986+0000","last_undegraded":"2024-03-06T07:22:54.006986+0000","last_fullsized":"2024-03-06T07:22:54.006986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:54.295092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281950+0000","last_change":"2024-03-06T07:22:53.005295+0000","last_active":"2024-03-06T07:22:54.281950+0000","last_peered":"2024-03-06T07:22:54.281950+0000","last_clean":"2024-03-06T07:22:54.281950+0000","last_became_active":"2024-03-06T07:22:53.004573+0000","last_became_peered":"2024-03-06T07:22:53.004573+0000","last_unstale":"2024-03-06T07:22:54.281950+0000","last_undegraded":"2024-03-06T07:22:54.281950+0000","last_fullsized":"2024-03-06T07:22:54.281950+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:01.752069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004410+0000","last_change":"2024-03-06T07:22:53.004824+0000","last_active":"2024-03-06T07:22:54.004410+0000","last_peered":"2024-03-06T07:22:54.004410+0000","last_clean":"2024-03-06T07:22:54.004410+0000","last_became_active":"2024-03-06T07:22:53.004472+0000","last_became_peered":"2024-03-06T07:22:53.004472+0000","last_unstale":"2024-03-06T07:22:54.004410+0000","last_undegraded":"2024-03-06T07:22:54.004410+0000","last_fullsized":"2024-03-06T07:22:54.004410+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:13.547310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007338+0000","last_change":"2024-03-06T07:22:51.998349+0000","last_active":"2024-03-06T07:22:54.007338+0000","last_peered":"2024-03-06T07:22:54.007338+0000","last_clean":"2024-03-06T07:22:54.007338+0000","last_became_active":"2024-03-06T07:22:51.998094+0000","last_became_peered":"2024-03-06T07:22:51.998094+0000","last_unstale":"2024-03-06T07:22:54.007338+0000","last_undegraded":"2024-03-06T07:22:54.007338+0000","last_fullsized":"2024-03-06T07:22:54.007338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:22.157883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007280+0000","last_change":"2024-03-06T07:22:53.005215+0000","last_active":"2024-03-06T07:22:54.007280+0000","last_peered":"2024-03-06T07:22:54.007280+0000","last_clean":"2024-03-06T07:22:54.007280+0000","last_became_active":"2024-03-06T07:22:53.004955+0000","last_became_peered":"2024-03-06T07:22:53.004955+0000","last_unstale":"2024-03-06T07:22:54.007280+0000","last_undegraded":"2024-03-06T07:22:54.007280+0000","last_fullsized":"2024-03-06T07:22:54.007280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004321+0000","last_change":"2024-03-06T07:22:52.006247+0000","last_active":"2024-03-06T07:22:54.004321+0000","last_peered":"2024-03-06T07:22:54.004321+0000","last_clean":"2024-03-06T07:22:54.004321+0000","last_became_active":"2024-03-06T07:22:52.005917+0000","last_became_peered":"2024-03-06T07:22:52.005917+0000","last_unstale":"2024-03-06T07:22:54.004321+0000","last_undegraded":"2024-03-06T07:22:54.004321+0000","last_fullsized":"2024-03-06T07:22:54.004321+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:16.012744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282625+0000","last_change":"2024-03-06T07:22:53.011697+0000","last_active":"2024-03-06T07:22:54.282625+0000","last_peered":"2024-03-06T07:22:54.282625+0000","last_clean":"2024-03-06T07:22:54.282625+0000","last_became_active":"2024-03-06T07:22:53.011304+0000","last_became_peered":"2024-03-06T07:22:53.011304+0000","last_unstale":"2024-03-06T07:22:54.282625+0000","last_undegraded":"2024-03-06T07:22:54.282625+0000","last_fullsized":"2024-03-06T07:22:54.282625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281799+0000","last_change":"2024-03-06T07:22:52.001909+0000","last_active":"2024-03-06T07:22:54.281799+0000","last_peered":"2024-03-06T07:22:54.281799+0000","last_clean":"2024-03-06T07:22:54.281799+0000","last_became_active":"2024-03-06T07:22:52.001539+0000","last_became_peered":"2024-03-06T07:22:52.001539+0000","last_unstale":"2024-03-06T07:22:54.281799+0000","last_undegraded":"2024-03-06T07:22:54.281799+0000","last_fullsized":"2024-03-06T07:22:54.281799+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:07.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006973+0000","last_change":"2024-03-06T07:22:53.006926+0000","last_active":"2024-03-06T07:22:54.006973+0000","last_peered":"2024-03-06T07:22:54.006973+0000","last_clean":"2024-03-06T07:22:54.006973+0000","last_became_active":"2024-03-06T07:22:53.006338+0000","last_became_peered":"2024-03-06T07:22:53.006338+0000","last_unstale":"2024-03-06T07:22:54.006973+0000","last_undegraded":"2024-03-06T07:22:54.006973+0000","last_fullsized":"2024-03-06T07:22:54.006973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281690+0000","last_change":"2024-03-06T07:22:52.006047+0000","last_active":"2024-03-06T07:22:54.281690+0000","last_peered":"2024-03-06T07:22:54.281690+0000","last_clean":"2024-03-06T07:22:54.281690+0000","last_became_active":"2024-03-06T07:22:52.005713+0000","last_became_peered":"2024-03-06T07:22:52.005713+0000","last_unstale":"2024-03-06T07:22:54.281690+0000","last_undegraded":"2024-03-06T07:22:54.281690+0000","last_fullsized":"2024-03-06T07:22:54.281690+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:12.146531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282779+0000","last_change":"2024-03-06T07:22:53.005043+0000","last_active":"2024-03-06T07:22:54.282779+0000","last_peered":"2024-03-06T07:22:54.282779+0000","last_clean":"2024-03-06T07:22:54.282779+0000","last_became_active":"2024-03-06T07:22:53.004749+0000","last_became_peered":"2024-03-06T07:22:53.004749+0000","last_unstale":"2024-03-06T07:22:54.282779+0000","last_undegraded":"2024-03-06T07:22:54.282779+0000","last_fullsized":"2024-03-06T07:22:54.282779+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282996+0000","last_change":"2024-03-06T07:22:52.003516+0000","last_active":"2024-03-06T07:22:54.282996+0000","last_peered":"2024-03-06T07:22:54.282996+0000","last_clean":"2024-03-06T07:22:54.282996+0000","last_became_active":"2024-03-06T07:22:52.003211+0000","last_became_peered":"2024-03-06T07:22:52.003211+0000","last_unstale":"2024-03-06T07:22:54.282996+0000","last_undegraded":"2024-03-06T07:22:54.282996+0000","last_fullsized":"2024-03-06T07:22:54.282996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:23.061079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283131+0000","last_change":"2024-03-06T07:22:53.014133+0000","last_active":"2024-03-06T07:22:54.283131+0000","last_peered":"2024-03-06T07:22:54.283131+0000","last_clean":"2024-03-06T07:22:54.283131+0000","last_became_active":"2024-03-06T07:22:53.013771+0000","last_became_peered":"2024-03-06T07:22:53.013771+0000","last_unstale":"2024-03-06T07:22:54.283131+0000","last_undegraded":"2024-03-06T07:22:54.283131+0000","last_fullsized":"2024-03-06T07:22:54.283131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282370+0000","last_change":"2024-03-06T07:22:52.003559+0000","last_active":"2024-03-06T07:22:54.282370+0000","last_peered":"2024-03-06T07:22:54.282370+0000","last_clean":"2024-03-06T07:22:54.282370+0000","last_became_active":"2024-03-06T07:22:52.003252+0000","last_became_peered":"2024-03-06T07:22:52.003252+0000","last_unstale":"2024-03-06T07:22:54.282370+0000","last_undegraded":"2024-03-06T07:22:54.282370+0000","last_fullsized":"2024-03-06T07:22:54.282370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:15.933962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282101+0000","last_change":"2024-03-06T07:22:52.996122+0000","last_active":"2024-03-06T07:22:54.282101+0000","last_peered":"2024-03-06T07:22:54.282101+0000","last_clean":"2024-03-06T07:22:54.282101+0000","last_became_active":"2024-03-06T07:22:52.995681+0000","last_became_peered":"2024-03-06T07:22:52.995681+0000","last_unstale":"2024-03-06T07:22:54.282101+0000","last_undegraded":"2024-03-06T07:22:54.282101+0000","last_fullsized":"2024-03-06T07:22:54.282101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:08.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281565+0000","last_change":"2024-03-06T07:22:52.004065+0000","last_active":"2024-03-06T07:22:54.281565+0000","last_peered":"2024-03-06T07:22:54.281565+0000","last_clean":"2024-03-06T07:22:54.281565+0000","last_became_active":"2024-03-06T07:22:52.003744+0000","last_became_peered":"2024-03-06T07:22:52.003744+0000","last_unstale":"2024-03-06T07:22:54.281565+0000","last_undegraded":"2024-03-06T07:22:54.281565+0000","last_fullsized":"2024-03-06T07:22:54.281565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:53.292991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006493+0000","last_change":"2024-03-06T07:22:53.008401+0000","last_active":"2024-03-06T07:22:54.006493+0000","last_peered":"2024-03-06T07:22:54.006493+0000","last_clean":"2024-03-06T07:22:54.006493+0000","last_became_active":"2024-03-06T07:22:53.008139+0000","last_became_peered":"2024-03-06T07:22:53.008139+0000","last_unstale":"2024-03-06T07:22:54.006493+0000","last_undegraded":"2024-03-06T07:22:54.006493+0000","last_fullsized":"2024-03-06T07:22:54.006493+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:04.433486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282222+0000","last_change":"2024-03-06T07:22:52.002965+0000","last_active":"2024-03-06T07:22:54.282222+0000","last_peered":"2024-03-06T07:22:54.282222+0000","last_clean":"2024-03-06T07:22:54.282222+0000","last_became_active":"2024-03-06T07:22:52.002690+0000","last_became_peered":"2024-03-06T07:22:52.002690+0000","last_unstale":"2024-03-06T07:22:54.282222+0000","last_undegraded":"2024-03-06T07:22:54.282222+0000","last_fullsized":"2024-03-06T07:22:54.282222+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:46.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283607+0000","last_change":"2024-03-06T07:22:53.013424+0000","last_active":"2024-03-06T07:22:54.283607+0000","last_peered":"2024-03-06T07:22:54.283607+0000","last_clean":"2024-03-06T07:22:54.283607+0000","last_became_active":"2024-03-06T07:22:53.013124+0000","last_became_peered":"2024-03-06T07:22:53.013124+0000","last_unstale":"2024-03-06T07:22:54.283607+0000","last_undegraded":"2024-03-06T07:22:54.283607+0000","last_fullsized":"2024-03-06T07:22:54.283607+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281752+0000","last_change":"2024-03-06T07:22:52.006835+0000","last_active":"2024-03-06T07:22:54.281752+0000","last_peered":"2024-03-06T07:22:54.281752+0000","last_clean":"2024-03-06T07:22:54.281752+0000","last_became_active":"2024-03-06T07:22:52.006449+0000","last_became_peered":"2024-03-06T07:22:52.006449+0000","last_unstale":"2024-03-06T07:22:54.281752+0000","last_undegraded":"2024-03-06T07:22:54.281752+0000","last_fullsized":"2024-03-06T07:22:54.281752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:49.689679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281589+0000","last_change":"2024-03-06T07:22:53.010786+0000","last_active":"2024-03-06T07:22:54.281589+0000","last_peered":"2024-03-06T07:22:54.281589+0000","last_clean":"2024-03-06T07:22:54.281589+0000","last_became_active":"2024-03-06T07:22:53.010470+0000","last_became_peered":"2024-03-06T07:22:53.010470+0000","last_unstale":"2024-03-06T07:22:54.281589+0000","last_undegraded":"2024-03-06T07:22:54.281589+0000","last_fullsized":"2024-03-06T07:22:54.281589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004403+0000","last_change":"2024-03-06T07:22:51.998392+0000","last_active":"2024-03-06T07:22:54.004403+0000","last_peered":"2024-03-06T07:22:54.004403+0000","last_clean":"2024-03-06T07:22:54.004403+0000","last_became_active":"2024-03-06T07:22:51.998047+0000","last_became_peered":"2024-03-06T07:22:51.998047+0000","last_unstale":"2024-03-06T07:22:54.004403+0000","last_undegraded":"2024-03-06T07:22:54.004403+0000","last_fullsized":"2024-03-06T07:22:54.004403+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:43.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282016+0000","last_change":"2024-03-06T07:22:53.002480+0000","last_active":"2024-03-06T07:22:54.282016+0000","last_peered":"2024-03-06T07:22:54.282016+0000","last_clean":"2024-03-06T07:22:54.282016+0000","last_became_active":"2024-03-06T07:22:53.002193+0000","last_became_peered":"2024-03-06T07:22:53.002193+0000","last_unstale":"2024-03-06T07:22:54.282016+0000","last_undegraded":"2024-03-06T07:22:54.282016+0000","last_fullsized":"2024-03-06T07:22:54.282016+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283382+0000","last_change":"2024-03-06T07:22:52.006845+0000","last_active":"2024-03-06T07:22:54.283382+0000","last_peered":"2024-03-06T07:22:54.283382+0000","last_clean":"2024-03-06T07:22:54.283382+0000","last_became_active":"2024-03-06T07:22:52.006519+0000","last_became_peered":"2024-03-06T07:22:52.006519+0000","last_unstale":"2024-03-06T07:22:54.283382+0000","last_undegraded":"2024-03-06T07:22:54.283382+0000","last_fullsized":"2024-03-06T07:22:54.283382+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:50.470263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283254+0000","last_change":"2024-03-06T07:22:53.009814+0000","last_active":"2024-03-06T07:22:54.283254+0000","last_peered":"2024-03-06T07:22:54.283254+0000","last_clean":"2024-03-06T07:22:54.283254+0000","last_became_active":"2024-03-06T07:22:53.009525+0000","last_became_peered":"2024-03-06T07:22:53.009525+0000","last_unstale":"2024-03-06T07:22:54.283254+0000","last_undegraded":"2024-03-06T07:22:54.283254+0000","last_fullsized":"2024-03-06T07:22:54.283254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282039+0000","last_change":"2024-03-06T07:22:51.999176+0000","last_active":"2024-03-06T07:22:54.282039+0000","last_peered":"2024-03-06T07:22:54.282039+0000","last_clean":"2024-03-06T07:22:54.282039+0000","last_became_active":"2024-03-06T07:22:51.998910+0000","last_became_peered":"2024-03-06T07:22:51.998910+0000","last_unstale":"2024-03-06T07:22:54.282039+0000","last_undegraded":"2024-03-06T07:22:54.282039+0000","last_fullsized":"2024-03-06T07:22:54.282039+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:12.477243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006543+0000","last_change":"2024-03-06T07:22:53.010178+0000","last_active":"2024-03-06T07:22:54.006543+0000","last_peered":"2024-03-06T07:22:54.006543+0000","last_clean":"2024-03-06T07:22:54.006543+0000","last_became_active":"2024-03-06T07:22:53.009892+0000","last_became_peered":"2024-03-06T07:22:53.009892+0000","last_unstale":"2024-03-06T07:22:54.006543+0000","last_undegraded":"2024-03-06T07:22:54.006543+0000","last_fullsized":"2024-03-06T07:22:54.006543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.337698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282986+0000","last_change":"2024-03-06T07:22:52.003595+0000","last_active":"2024-03-06T07:22:54.282986+0000","last_peered":"2024-03-06T07:22:54.282986+0000","last_clean":"2024-03-06T07:22:54.282986+0000","last_became_active":"2024-03-06T07:22:52.003239+0000","last_became_peered":"2024-03-06T07:22:52.003239+0000","last_unstale":"2024-03-06T07:22:54.282986+0000","last_undegraded":"2024-03-06T07:22:54.282986+0000","last_fullsized":"2024-03-06T07:22:54.282986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:10.020728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281828+0000","last_change":"2024-03-06T07:22:53.004565+0000","last_active":"2024-03-06T07:22:54.281828+0000","last_peered":"2024-03-06T07:22:54.281828+0000","last_clean":"2024-03-06T07:22:54.281828+0000","last_became_active":"2024-03-06T07:22:53.004149+0000","last_became_peered":"2024-03-06T07:22:53.004149+0000","last_unstale":"2024-03-06T07:22:54.281828+0000","last_undegraded":"2024-03-06T07:22:54.281828+0000","last_fullsized":"2024-03-06T07:22:54.281828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006539+0000","last_change":"2024-03-06T07:22:51.997940+0000","last_active":"2024-03-06T07:22:54.006539+0000","last_peered":"2024-03-06T07:22:54.006539+0000","last_clean":"2024-03-06T07:22:54.006539+0000","last_became_active":"2024-03-06T07:22:51.997652+0000","last_became_peered":"2024-03-06T07:22:51.997652+0000","last_unstale":"2024-03-06T07:22:54.006539+0000","last_undegraded":"2024-03-06T07:22:54.006539+0000","last_fullsized":"2024-03-06T07:22:54.006539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:24.629441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282148+0000","last_change":"2024-03-06T07:22:53.005336+0000","last_active":"2024-03-06T07:22:54.282148+0000","last_peered":"2024-03-06T07:22:54.282148+0000","last_clean":"2024-03-06T07:22:54.282148+0000","last_became_active":"2024-03-06T07:22:53.004960+0000","last_became_peered":"2024-03-06T07:22:53.004960+0000","last_unstale":"2024-03-06T07:22:54.282148+0000","last_undegraded":"2024-03-06T07:22:54.282148+0000","last_fullsized":"2024-03-06T07:22:54.282148+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:08.594990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281482+0000","last_change":"2024-03-06T07:22:52.006431+0000","last_active":"2024-03-06T07:22:54.281482+0000","last_peered":"2024-03-06T07:22:54.281482+0000","last_clean":"2024-03-06T07:22:54.281482+0000","last_became_active":"2024-03-06T07:22:52.006190+0000","last_became_peered":"2024-03-06T07:22:52.006190+0000","last_unstale":"2024-03-06T07:22:54.281482+0000","last_undegraded":"2024-03-06T07:22:54.281482+0000","last_fullsized":"2024-03-06T07:22:54.281482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006998+0000","last_change":"2024-03-06T07:22:53.004526+0000","last_active":"2024-03-06T07:22:54.006998+0000","last_peered":"2024-03-06T07:22:54.006998+0000","last_clean":"2024-03-06T07:22:54.006998+0000","last_became_active":"2024-03-06T07:22:53.003998+0000","last_became_peered":"2024-03-06T07:22:53.003998+0000","last_unstale":"2024-03-06T07:22:54.006998+0000","last_undegraded":"2024-03-06T07:22:54.006998+0000","last_fullsized":"2024-03-06T07:22:54.006998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:42.114122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281911+0000","last_change":"2024-03-06T07:22:52.002503+0000","last_active":"2024-03-06T07:22:54.281911+0000","last_peered":"2024-03-06T07:22:54.281911+0000","last_clean":"2024-03-06T07:22:54.281911+0000","last_became_active":"2024-03-06T07:22:52.001554+0000","last_became_peered":"2024-03-06T07:22:52.001554+0000","last_unstale":"2024-03-06T07:22:54.281911+0000","last_undegraded":"2024-03-06T07:22:54.281911+0000","last_fullsized":"2024-03-06T07:22:54.281911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:41.112020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282874+0000","last_change":"2024-03-06T07:22:53.007256+0000","last_active":"2024-03-06T07:22:54.282874+0000","last_peered":"2024-03-06T07:22:54.282874+0000","last_clean":"2024-03-06T07:22:54.282874+0000","last_became_active":"2024-03-06T07:22:53.006985+0000","last_became_peered":"2024-03-06T07:22:53.006985+0000","last_unstale":"2024-03-06T07:22:54.282874+0000","last_undegraded":"2024-03-06T07:22:54.282874+0000","last_fullsized":"2024-03-06T07:22:54.282874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:04.433486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007006+0000","last_change":"2024-03-06T07:22:51.996405+0000","last_active":"2024-03-06T07:22:54.007006+0000","last_peered":"2024-03-06T07:22:54.007006+0000","last_clean":"2024-03-06T07:22:54.007006+0000","last_became_active":"2024-03-06T07:22:51.996082+0000","last_became_peered":"2024-03-06T07:22:51.996082+0000","last_unstale":"2024-03-06T07:22:54.007006+0000","last_undegraded":"2024-03-06T07:22:54.007006+0000","last_fullsized":"2024-03-06T07:22:54.007006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:57.305195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283827+0000","last_change":"2024-03-06T07:22:53.018922+0000","last_active":"2024-03-06T07:22:54.283827+0000","last_peered":"2024-03-06T07:22:54.283827+0000","last_clean":"2024-03-06T07:22:54.283827+0000","last_became_active":"2024-03-06T07:22:53.018664+0000","last_became_peered":"2024-03-06T07:22:53.018664+0000","last_unstale":"2024-03-06T07:22:54.283827+0000","last_undegraded":"2024-03-06T07:22:54.283827+0000","last_fullsized":"2024-03-06T07:22:54.283827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:04.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282921+0000","last_change":"2024-03-06T07:22:52.002762+0000","last_active":"2024-03-06T07:22:54.282921+0000","last_peered":"2024-03-06T07:22:54.282921+0000","last_clean":"2024-03-06T07:22:54.282921+0000","last_became_active":"2024-03-06T07:22:52.002461+0000","last_became_peered":"2024-03-06T07:22:52.002461+0000","last_unstale":"2024-03-06T07:22:54.282921+0000","last_undegraded":"2024-03-06T07:22:54.282921+0000","last_fullsized":"2024-03-06T07:22:54.282921+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.704065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006733+0000","last_change":"2024-03-06T07:22:53.006268+0000","last_active":"2024-03-06T07:22:54.006733+0000","last_peered":"2024-03-06T07:22:54.006733+0000","last_clean":"2024-03-06T07:22:54.006733+0000","last_became_active":"2024-03-06T07:22:53.005993+0000","last_became_peered":"2024-03-06T07:22:53.005993+0000","last_unstale":"2024-03-06T07:22:54.006733+0000","last_undegraded":"2024-03-06T07:22:54.006733+0000","last_fullsized":"2024-03-06T07:22:54.006733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:04.078810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282767+0000","last_change":"2024-03-06T07:22:52.004453+0000","last_active":"2024-03-06T07:22:54.282767+0000","last_peered":"2024-03-06T07:22:54.282767+0000","last_clean":"2024-03-06T07:22:54.282767+0000","last_became_active":"2024-03-06T07:22:52.003997+0000","last_became_peered":"2024-03-06T07:22:52.003997+0000","last_unstale":"2024-03-06T07:22:54.282767+0000","last_undegraded":"2024-03-06T07:22:54.282767+0000","last_fullsized":"2024-03-06T07:22:54.282767+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:37.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281621+0000","last_change":"2024-03-06T07:22:53.014780+0000","last_active":"2024-03-06T07:22:54.281621+0000","last_peered":"2024-03-06T07:22:54.281621+0000","last_clean":"2024-03-06T07:22:54.281621+0000","last_became_active":"2024-03-06T07:22:53.014501+0000","last_became_peered":"2024-03-06T07:22:53.014501+0000","last_unstale":"2024-03-06T07:22:54.281621+0000","last_undegraded":"2024-03-06T07:22:54.281621+0000","last_fullsized":"2024-03-06T07:22:54.281621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.631542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282752+0000","last_change":"2024-03-06T07:22:52.002095+0000","last_active":"2024-03-06T07:22:54.282752+0000","last_peered":"2024-03-06T07:22:54.282752+0000","last_clean":"2024-03-06T07:22:54.282752+0000","last_became_active":"2024-03-06T07:22:52.001749+0000","last_became_peered":"2024-03-06T07:22:52.001749+0000","last_unstale":"2024-03-06T07:22:54.282752+0000","last_undegraded":"2024-03-06T07:22:54.282752+0000","last_fullsized":"2024-03-06T07:22:54.282752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.963836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281407+0000","last_change":"2024-03-06T07:22:53.020156+0000","last_active":"2024-03-06T07:22:54.281407+0000","last_peered":"2024-03-06T07:22:54.281407+0000","last_clean":"2024-03-06T07:22:54.281407+0000","last_became_active":"2024-03-06T07:22:53.019845+0000","last_became_peered":"2024-03-06T07:22:53.019845+0000","last_unstale":"2024-03-06T07:22:54.281407+0000","last_undegraded":"2024-03-06T07:22:54.281407+0000","last_fullsized":"2024-03-06T07:22:54.281407+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:11.022830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004216+0000","last_change":"2024-03-06T07:22:51.998103+0000","last_active":"2024-03-06T07:22:54.004216+0000","last_peered":"2024-03-06T07:22:54.004216+0000","last_clean":"2024-03-06T07:22:54.004216+0000","last_became_active":"2024-03-06T07:22:51.997234+0000","last_became_peered":"2024-03-06T07:22:51.997234+0000","last_unstale":"2024-03-06T07:22:54.004216+0000","last_undegraded":"2024-03-06T07:22:54.004216+0000","last_fullsized":"2024-03-06T07:22:54.004216+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:24.629441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281560+0000","last_change":"2024-03-06T07:22:53.018761+0000","last_active":"2024-03-06T07:22:54.281560+0000","last_peered":"2024-03-06T07:22:54.281560+0000","last_clean":"2024-03-06T07:22:54.281560+0000","last_became_active":"2024-03-06T07:22:53.018435+0000","last_became_peered":"2024-03-06T07:22:53.018435+0000","last_unstale":"2024-03-06T07:22:54.281560+0000","last_undegraded":"2024-03-06T07:22:54.281560+0000","last_fullsized":"2024-03-06T07:22:54.281560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:57.965937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007049+0000","last_change":"2024-03-06T07:22:51.997725+0000","last_active":"2024-03-06T07:22:54.007049+0000","last_peered":"2024-03-06T07:22:54.007049+0000","last_clean":"2024-03-06T07:22:54.007049+0000","last_became_active":"2024-03-06T07:22:51.997442+0000","last_became_peered":"2024-03-06T07:22:51.997442+0000","last_unstale":"2024-03-06T07:22:54.007049+0000","last_undegraded":"2024-03-06T07:22:54.007049+0000","last_fullsized":"2024-03-06T07:22:54.007049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.963836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281313+0000","last_change":"2024-03-06T07:22:53.018347+0000","last_active":"2024-03-06T07:22:54.281313+0000","last_peered":"2024-03-06T07:22:54.281313+0000","last_clean":"2024-03-06T07:22:54.281313+0000","last_became_active":"2024-03-06T07:22:53.018073+0000","last_became_peered":"2024-03-06T07:22:53.018073+0000","last_unstale":"2024-03-06T07:22:54.281313+0000","last_undegraded":"2024-03-06T07:22:54.281313+0000","last_fullsized":"2024-03-06T07:22:54.281313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:24.063180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283184+0000","last_change":"2024-03-06T07:22:52.002891+0000","last_active":"2024-03-06T07:22:54.283184+0000","last_peered":"2024-03-06T07:22:54.283184+0000","last_clean":"2024-03-06T07:22:54.283184+0000","last_became_active":"2024-03-06T07:22:52.002607+0000","last_became_peered":"2024-03-06T07:22:52.002607+0000","last_unstale":"2024-03-06T07:22:54.283184+0000","last_undegraded":"2024-03-06T07:22:54.283184+0000","last_fullsized":"2024-03-06T07:22:54.283184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:25.059867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281923+0000","last_change":"2024-03-06T07:22:53.006187+0000","last_active":"2024-03-06T07:22:54.281923+0000","last_peered":"2024-03-06T07:22:54.281923+0000","last_clean":"2024-03-06T07:22:54.281923+0000","last_became_active":"2024-03-06T07:22:53.005793+0000","last_became_peered":"2024-03-06T07:22:53.005793+0000","last_unstale":"2024-03-06T07:22:54.281923+0000","last_undegraded":"2024-03-06T07:22:54.281923+0000","last_fullsized":"2024-03-06T07:22:54.281923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283061+0000","last_change":"2024-03-06T07:22:52.006361+0000","last_active":"2024-03-06T07:22:54.283061+0000","last_peered":"2024-03-06T07:22:54.283061+0000","last_clean":"2024-03-06T07:22:54.283061+0000","last_became_active":"2024-03-06T07:22:52.006055+0000","last_became_peered":"2024-03-06T07:22:52.006055+0000","last_unstale":"2024-03-06T07:22:54.283061+0000","last_undegraded":"2024-03-06T07:22:54.283061+0000","last_fullsized":"2024-03-06T07:22:54.283061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:41.112020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281531+0000","last_change":"2024-03-06T07:22:53.020189+0000","last_active":"2024-03-06T07:22:54.281531+0000","last_peered":"2024-03-06T07:22:54.281531+0000","last_clean":"2024-03-06T07:22:54.281531+0000","last_became_active":"2024-03-06T07:22:53.019828+0000","last_became_peered":"2024-03-06T07:22:53.019828+0000","last_unstale":"2024-03-06T07:22:54.281531+0000","last_undegraded":"2024-03-06T07:22:54.281531+0000","last_fullsized":"2024-03-06T07:22:54.281531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:23.159984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007053+0000","last_change":"2024-03-06T07:22:52.000983+0000","last_active":"2024-03-06T07:22:54.007053+0000","last_peered":"2024-03-06T07:22:54.007053+0000","last_clean":"2024-03-06T07:22:54.007053+0000","last_became_active":"2024-03-06T07:22:52.000680+0000","last_became_peered":"2024-03-06T07:22:52.000680+0000","last_unstale":"2024-03-06T07:22:54.007053+0000","last_undegraded":"2024-03-06T07:22:54.007053+0000","last_fullsized":"2024-03-06T07:22:54.007053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283816+0000","last_change":"2024-03-06T07:22:53.014876+0000","last_active":"2024-03-06T07:22:54.283816+0000","last_peered":"2024-03-06T07:22:54.283816+0000","last_clean":"2024-03-06T07:22:54.283816+0000","last_became_active":"2024-03-06T07:22:53.014322+0000","last_became_peered":"2024-03-06T07:22:53.014322+0000","last_unstale":"2024-03-06T07:22:54.283816+0000","last_undegraded":"2024-03-06T07:22:54.283816+0000","last_fullsized":"2024-03-06T07:22:54.283816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:19.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004490+0000","last_change":"2024-03-06T07:22:51.998753+0000","last_active":"2024-03-06T07:22:54.004490+0000","last_peered":"2024-03-06T07:22:54.004490+0000","last_clean":"2024-03-06T07:22:54.004490+0000","last_became_active":"2024-03-06T07:22:51.998428+0000","last_became_peered":"2024-03-06T07:22:51.998428+0000","last_unstale":"2024-03-06T07:22:54.004490+0000","last_undegraded":"2024-03-06T07:22:54.004490+0000","last_fullsized":"2024-03-06T07:22:54.004490+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004285+0000","last_change":"2024-03-06T07:22:53.012683+0000","last_active":"2024-03-06T07:22:54.004285+0000","last_peered":"2024-03-06T07:22:54.004285+0000","last_clean":"2024-03-06T07:22:54.004285+0000","last_became_active":"2024-03-06T07:22:53.012154+0000","last_became_peered":"2024-03-06T07:22:53.012154+0000","last_unstale":"2024-03-06T07:22:54.004285+0000","last_undegraded":"2024-03-06T07:22:54.004285+0000","last_fullsized":"2024-03-06T07:22:54.004285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006819+0000","last_change":"2024-03-06T07:22:52.004074+0000","last_active":"2024-03-06T07:22:54.006819+0000","last_peered":"2024-03-06T07:22:54.006819+0000","last_clean":"2024-03-06T07:22:54.006819+0000","last_became_active":"2024-03-06T07:22:52.003773+0000","last_became_peered":"2024-03-06T07:22:52.003773+0000","last_unstale":"2024-03-06T07:22:54.006819+0000","last_undegraded":"2024-03-06T07:22:54.006819+0000","last_fullsized":"2024-03-06T07:22:54.006819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:43.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007154+0000","last_change":"2024-03-06T07:22:53.001845+0000","last_active":"2024-03-06T07:22:54.007154+0000","last_peered":"2024-03-06T07:22:54.007154+0000","last_clean":"2024-03-06T07:22:54.007154+0000","last_became_active":"2024-03-06T07:22:53.001480+0000","last_became_peered":"2024-03-06T07:22:53.001480+0000","last_unstale":"2024-03-06T07:22:54.007154+0000","last_undegraded":"2024-03-06T07:22:54.007154+0000","last_fullsized":"2024-03-06T07:22:54.007154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:51.472365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004349+0000","last_change":"2024-03-06T07:22:52.001878+0000","last_active":"2024-03-06T07:22:54.004349+0000","last_peered":"2024-03-06T07:22:54.004349+0000","last_clean":"2024-03-06T07:22:54.004349+0000","last_became_active":"2024-03-06T07:22:52.001618+0000","last_became_peered":"2024-03-06T07:22:52.001618+0000","last_unstale":"2024-03-06T07:22:54.004349+0000","last_undegraded":"2024-03-06T07:22:54.004349+0000","last_fullsized":"2024-03-06T07:22:54.004349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:57.305195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004250+0000","last_change":"2024-03-06T07:22:53.010139+0000","last_active":"2024-03-06T07:22:54.004250+0000","last_peered":"2024-03-06T07:22:54.004250+0000","last_clean":"2024-03-06T07:22:54.004250+0000","last_became_active":"2024-03-06T07:22:53.009822+0000","last_became_peered":"2024-03-06T07:22:53.009822+0000","last_unstale":"2024-03-06T07:22:54.004250+0000","last_undegraded":"2024-03-06T07:22:54.004250+0000","last_fullsized":"2024-03-06T07:22:54.004250+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:50.691781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006648+0000","last_change":"2024-03-06T07:22:51.999813+0000","last_active":"2024-03-06T07:22:54.006648+0000","last_peered":"2024-03-06T07:22:54.006648+0000","last_clean":"2024-03-06T07:22:54.006648+0000","last_became_active":"2024-03-06T07:22:51.999483+0000","last_became_peered":"2024-03-06T07:22:51.999483+0000","last_unstale":"2024-03-06T07:22:54.006648+0000","last_undegraded":"2024-03-06T07:22:54.006648+0000","last_fullsized":"2024-03-06T07:22:54.006648+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:16.012744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281797+0000","last_change":"2024-03-06T07:22:53.011139+0000","last_active":"2024-03-06T07:22:54.281797+0000","last_peered":"2024-03-06T07:22:54.281797+0000","last_clean":"2024-03-06T07:22:54.281797+0000","last_became_active":"2024-03-06T07:22:53.010277+0000","last_became_peered":"2024-03-06T07:22:53.010277+0000","last_unstale":"2024-03-06T07:22:54.281797+0000","last_undegraded":"2024-03-06T07:22:54.281797+0000","last_fullsized":"2024-03-06T07:22:54.281797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283282+0000","last_change":"2024-03-06T07:22:52.003098+0000","last_active":"2024-03-06T07:22:54.283282+0000","last_peered":"2024-03-06T07:22:54.283282+0000","last_clean":"2024-03-06T07:22:54.283282+0000","last_became_active":"2024-03-06T07:22:52.002798+0000","last_became_peered":"2024-03-06T07:22:52.002798+0000","last_unstale":"2024-03-06T07:22:54.283282+0000","last_undegraded":"2024-03-06T07:22:54.283282+0000","last_fullsized":"2024-03-06T07:22:54.283282+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:22.157883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283083+0000","last_change":"2024-03-06T07:22:53.003411+0000","last_active":"2024-03-06T07:22:54.283083+0000","last_peered":"2024-03-06T07:22:54.283083+0000","last_clean":"2024-03-06T07:22:54.283083+0000","last_became_active":"2024-03-06T07:22:53.003120+0000","last_became_peered":"2024-03-06T07:22:53.003120+0000","last_unstale":"2024-03-06T07:22:54.283083+0000","last_undegraded":"2024-03-06T07:22:54.283083+0000","last_fullsized":"2024-03-06T07:22:54.283083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:04.078810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281937+0000","last_change":"2024-03-06T07:22:52.007081+0000","last_active":"2024-03-06T07:22:54.281937+0000","last_peered":"2024-03-06T07:22:54.281937+0000","last_clean":"2024-03-06T07:22:54.281937+0000","last_became_active":"2024-03-06T07:22:52.006599+0000","last_became_peered":"2024-03-06T07:22:52.006599+0000","last_unstale":"2024-03-06T07:22:54.281937+0000","last_undegraded":"2024-03-06T07:22:54.281937+0000","last_fullsized":"2024-03-06T07:22:54.281937+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:50.470263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282354+0000","last_change":"2024-03-06T07:22:53.007286+0000","last_active":"2024-03-06T07:22:54.282354+0000","last_peered":"2024-03-06T07:22:54.282354+0000","last_clean":"2024-03-06T07:22:54.282354+0000","last_became_active":"2024-03-06T07:22:53.007032+0000","last_became_peered":"2024-03-06T07:22:53.007032+0000","last_unstale":"2024-03-06T07:22:54.282354+0000","last_undegraded":"2024-03-06T07:22:54.282354+0000","last_fullsized":"2024-03-06T07:22:54.282354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007110+0000","last_change":"2024-03-06T07:22:52.003894+0000","last_active":"2024-03-06T07:22:54.007110+0000","last_peered":"2024-03-06T07:22:54.007110+0000","last_clean":"2024-03-06T07:22:54.007110+0000","last_became_active":"2024-03-06T07:22:52.003608+0000","last_became_peered":"2024-03-06T07:22:52.003608+0000","last_unstale":"2024-03-06T07:22:54.007110+0000","last_undegraded":"2024-03-06T07:22:54.007110+0000","last_fullsized":"2024-03-06T07:22:54.007110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:25.059867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282579+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:22:54.282579+0000","last_peered":"2024-03-06T07:22:54.282579+0000","last_clean":"2024-03-06T07:22:54.282579+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:22:54.282579+0000","last_undegraded":"2024-03-06T07:22:54.282579+0000","last_fullsized":"2024-03-06T07:22:54.282579+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":736976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:01.644 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.746+0000 7f95a52df640 1 Processor -- start 2024-03-06T07:23:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.746+0000 7f95a52df640 1 -- start start 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f95a52df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a00b4170 0x7f95a00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f95a52df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95a00d4760 0x7f95a00d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f95a52df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00d25d0 con 0x7f95a00b4dd0 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f95a52df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00d4bf0 con 0x7f95a00d4760 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f95a52df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00ce9e0 con 0x7f95a00b4170 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95a00d4760 0x7f95a00d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95a00d4760 0x7f95a00d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38542/0 (socket says 172.21.15.122:38542) 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 -- 172.21.15.122:0/1904520019 learned_addr learned my addr 172.21.15.122:0/1904520019 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959effd640 1 --2- 172.21.15.122:0/1904520019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a00b4170 0x7f95a00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 -- 172.21.15.122:0/1904520019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a00b4170 msgr2=0x7f95a00b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 --2- 172.21.15.122:0/1904520019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a00b4170 0x7f95a00b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 -- 172.21.15.122:0/1904520019 >> v1:172.21.15.169:6789/0 conn(0x7f95a00b4dd0 legacy=0x7f95a00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.747+0000 7f959e7fc640 1 -- 172.21.15.122:0/1904520019 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00d2870 con 0x7f95a00d4760 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.748+0000 7f959e7fc640 1 --2- 172.21.15.122:0/1904520019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00d4760 0x7f95a00d0370 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9590009870 tx=0x7f959002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152fa19d7f78a769 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.748+0000 7f959dffb640 1 -- 172.21.15.122:0/1904520019 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590035020 con 0x7f95a00d4760 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.748+0000 7f959dffb640 1 -- 172.21.15.122:0/1904520019 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9590031640 con 0x7f95a00d4760 2024-03-06T07:23:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.748+0000 7f959dffb640 1 -- 172.21.15.122:0/1904520019 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959002d040 con 0x7f95a00d4760 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 -- 172.21.15.122:0/1904520019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00d4760 msgr2=0x7f95a00d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 --2- 172.21.15.122:0/1904520019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00d4760 0x7f95a00d0370 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9590009870 tx=0x7f959002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 -- 172.21.15.122:0/1904520019 shutdown_connections 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 --2- 172.21.15.122:0/1904520019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00d4760 0x7f95a00d0370 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 --2- 172.21.15.122:0/1904520019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a00b4170 0x7f95a00b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 -- 172.21.15.122:0/1904520019 >> 172.21.15.122:0/1904520019 conn(0x7f95a00d3c40 msgr2=0x7f95a00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 -- 172.21.15.122:0/1904520019 shutdown_connections 2024-03-06T07:23:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.749+0000 7f95a52df640 1 -- 172.21.15.122:0/1904520019 wait complete. 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.750+0000 7f95a52df640 1 Processor -- start 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.750+0000 7f95a52df640 1 -- start start 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95a52df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a00b4170 0x7f95a01f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95a52df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 0x7f95a015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95a52df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a015b8b0 0x7f95a00b8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95a52df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00b47d0 con 0x7f95a00b4dd0 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95a52df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00b4550 con 0x7f95a00b4170 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95a52df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00b4950 con 0x7f95a015b8b0 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 0x7f95a015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 0x7f95a015b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38548/0 (socket says 172.21.15.122:38548) 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 -- 172.21.15.122:0/2254329281 learned_addr learned my addr 172.21.15.122:0/2254329281 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959f7fe640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a015b8b0 0x7f95a00b8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959effd640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a00b4170 0x7f95a01f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 -- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a015b8b0 msgr2=0x7f95a00b8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a015b8b0 0x7f95a00b8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 -- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a00b4170 msgr2=0x7f95a01f4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a00b4170 0x7f95a01f4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 -- 172.21.15.122:0/2254329281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a015bd30 con 0x7f95a00b4dd0 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959effd640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a00b4170 0x7f95a01f4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f959e7fc640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 0x7f95a015b370 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9590000c00 tx=0x7f95900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.751+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590040070 con 0x7f95a00b4dd0 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.752+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95900399f0 con 0x7f95a00b4dd0 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.752+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a01f4020 con 0x7f95a00b4dd0 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.752+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590035040 con 0x7f95a00b4dd0 2024-03-06T07:23:01.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.752+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a01f3d10 con 0x7f95a00b4dd0 2024-03-06T07:23:01.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.753+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a004f330 con 0x7f95a00b4dd0 2024-03-06T07:23:01.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.753+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9590038020 con 0x7f95a00b4dd0 2024-03-06T07:23:01.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.753+0000 7f95877fe640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f958c06b500 0x7f958c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.754+0000 7f959effd640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f958c06b500 0x7f958c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.754+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95900abb50 con 0x7f95a00b4dd0 2024-03-06T07:23:01.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.754+0000 7f959effd640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f958c06b500 0x7f958c06d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9594006fd0 tx=0x7f9594008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.758+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f959007a9d0 con 0x7f95a00b4dd0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.933+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95a00733b0 con 0x7f958c06b500 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.939+0000 7f95877fe640 1 -- 172.21.15.122:0/2254329281 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251839 (secure 0 0 0) 0x7f95a00733b0 con 0x7f958c06b500 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f958c06b500 msgr2=0x7f958c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f958c06b500 0x7f958c06d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9594006fd0 tx=0x7f9594008040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 msgr2=0x7f95a015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 0x7f95a015b370 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9590000c00 tx=0x7f95900046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 shutdown_connections 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f958c06b500 0x7f958c06d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95a015b8b0 0x7f95a00b8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95a00b4dd0 0x7f95a015b370 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 --2- 172.21.15.122:0/2254329281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95a00b4170 0x7f95a01f4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 >> 172.21.15.122:0/2254329281 conn(0x7f95a00d3c40 msgr2=0x7f95a006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 shutdown_connections 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:01.942+0000 7f95a52df640 1 -- 172.21.15.122:0/2254329281 wait complete. 2024-03-06T07:23:01.959 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-03-06T07:23:01.344176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220304,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754416,"statfs":{"total":773094113280,"available":772868521984,"internally_reserved":0,"allocated":4235264,"data_stored":2358420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.355091"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007270+0000","last_change":"2024-03-06T07:22:53.015900+0000","last_active":"2024-03-06T07:22:54.007270+0000","last_peered":"2024-03-06T07:22:54.007270+0000","last_clean":"2024-03-06T07:22:54.007270+0000","last_became_active":"2024-03-06T07:22:53.015567+0000","last_became_peered":"2024-03-06T07:22:53.015567+0000","last_unstale":"2024-03-06T07:22:54.007270+0000","last_undegraded":"2024-03-06T07:22:54.007270+0000","last_fullsized":"2024-03-06T07:22:54.007270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:08.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282163+0000","last_change":"2024-03-06T07:22:53.005999+0000","last_active":"2024-03-06T07:22:54.282163+0000","last_peered":"2024-03-06T07:22:54.282163+0000","last_clean":"2024-03-06T07:22:54.282163+0000","last_became_active":"2024-03-06T07:22:53.005727+0000","last_became_peered":"2024-03-06T07:22:53.005727+0000","last_unstale":"2024-03-06T07:22:54.282163+0000","last_undegraded":"2024-03-06T07:22:54.282163+0000","last_fullsized":"2024-03-06T07:22:54.282163+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:04.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283677+0000","last_change":"2024-03-06T07:22:53.009774+0000","last_active":"2024-03-06T07:22:54.283677+0000","last_peered":"2024-03-06T07:22:54.283677+0000","last_clean":"2024-03-06T07:22:54.283677+0000","last_became_active":"2024-03-06T07:22:53.009488+0000","last_became_peered":"2024-03-06T07:22:53.009488+0000","last_unstale":"2024-03-06T07:22:54.283677+0000","last_undegraded":"2024-03-06T07:22:54.283677+0000","last_fullsized":"2024-03-06T07:22:54.283677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006758+0000","last_change":"2024-03-06T07:22:53.019399+0000","last_active":"2024-03-06T07:22:54.006758+0000","last_peered":"2024-03-06T07:22:54.006758+0000","last_clean":"2024-03-06T07:22:54.006758+0000","last_became_active":"2024-03-06T07:22:53.019122+0000","last_became_peered":"2024-03-06T07:22:53.019122+0000","last_unstale":"2024-03-06T07:22:54.006758+0000","last_undegraded":"2024-03-06T07:22:54.006758+0000","last_fullsized":"2024-03-06T07:22:54.006758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281419+0000","last_change":"2024-03-06T07:22:53.020095+0000","last_active":"2024-03-06T07:22:54.281419+0000","last_peered":"2024-03-06T07:22:54.281419+0000","last_clean":"2024-03-06T07:22:54.281419+0000","last_became_active":"2024-03-06T07:22:53.019764+0000","last_became_peered":"2024-03-06T07:22:53.019764+0000","last_unstale":"2024-03-06T07:22:54.281419+0000","last_undegraded":"2024-03-06T07:22:54.281419+0000","last_fullsized":"2024-03-06T07:22:54.281419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:16.936064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281986+0000","last_change":"2024-03-06T07:22:53.011642+0000","last_active":"2024-03-06T07:22:54.281986+0000","last_peered":"2024-03-06T07:22:54.281986+0000","last_clean":"2024-03-06T07:22:54.281986+0000","last_became_active":"2024-03-06T07:22:53.011303+0000","last_became_peered":"2024-03-06T07:22:53.011303+0000","last_unstale":"2024-03-06T07:22:54.281986+0000","last_undegraded":"2024-03-06T07:22:54.281986+0000","last_fullsized":"2024-03-06T07:22:54.281986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282515+0000","last_change":"2024-03-06T07:22:53.009740+0000","last_active":"2024-03-06T07:22:54.282515+0000","last_peered":"2024-03-06T07:22:54.282515+0000","last_clean":"2024-03-06T07:22:54.282515+0000","last_became_active":"2024-03-06T07:22:53.009431+0000","last_became_peered":"2024-03-06T07:22:53.009431+0000","last_unstale":"2024-03-06T07:22:54.282515+0000","last_undegraded":"2024-03-06T07:22:54.282515+0000","last_fullsized":"2024-03-06T07:22:54.282515+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282887+0000","last_change":"2024-03-06T07:22:53.011772+0000","last_active":"2024-03-06T07:22:54.282887+0000","last_peered":"2024-03-06T07:22:54.282887+0000","last_clean":"2024-03-06T07:22:54.282887+0000","last_became_active":"2024-03-06T07:22:53.011391+0000","last_became_peered":"2024-03-06T07:22:53.011391+0000","last_unstale":"2024-03-06T07:22:54.282887+0000","last_undegraded":"2024-03-06T07:22:54.282887+0000","last_fullsized":"2024-03-06T07:22:54.282887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282966+0000","last_change":"2024-03-06T07:22:53.008973+0000","last_active":"2024-03-06T07:22:54.282966+0000","last_peered":"2024-03-06T07:22:54.282966+0000","last_clean":"2024-03-06T07:22:54.282966+0000","last_became_active":"2024-03-06T07:22:53.008608+0000","last_became_peered":"2024-03-06T07:22:53.008608+0000","last_unstale":"2024-03-06T07:22:54.282966+0000","last_undegraded":"2024-03-06T07:22:54.282966+0000","last_fullsized":"2024-03-06T07:22:54.282966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.337698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006851+0000","last_change":"2024-03-06T07:22:53.014766+0000","last_active":"2024-03-06T07:22:54.006851+0000","last_peered":"2024-03-06T07:22:54.006851+0000","last_clean":"2024-03-06T07:22:54.006851+0000","last_became_active":"2024-03-06T07:22:53.014521+0000","last_became_peered":"2024-03-06T07:22:53.014521+0000","last_unstale":"2024-03-06T07:22:54.006851+0000","last_undegraded":"2024-03-06T07:22:54.006851+0000","last_fullsized":"2024-03-06T07:22:54.006851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:01.752069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282270+0000","last_change":"2024-03-06T07:22:53.008566+0000","last_active":"2024-03-06T07:22:54.282270+0000","last_peered":"2024-03-06T07:22:54.282270+0000","last_clean":"2024-03-06T07:22:54.282270+0000","last_became_active":"2024-03-06T07:22:53.008322+0000","last_became_peered":"2024-03-06T07:22:53.008322+0000","last_unstale":"2024-03-06T07:22:54.282270+0000","last_undegraded":"2024-03-06T07:22:54.282270+0000","last_fullsized":"2024-03-06T07:22:54.282270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283396+0000","last_change":"2024-03-06T07:22:53.018934+0000","last_active":"2024-03-06T07:22:54.283396+0000","last_peered":"2024-03-06T07:22:54.283396+0000","last_clean":"2024-03-06T07:22:54.283396+0000","last_became_active":"2024-03-06T07:22:53.018657+0000","last_became_peered":"2024-03-06T07:22:53.018657+0000","last_unstale":"2024-03-06T07:22:54.283396+0000","last_undegraded":"2024-03-06T07:22:54.283396+0000","last_fullsized":"2024-03-06T07:22:54.283396+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281455+0000","last_change":"2024-03-06T07:22:53.017169+0000","last_active":"2024-03-06T07:22:54.281455+0000","last_peered":"2024-03-06T07:22:54.281455+0000","last_clean":"2024-03-06T07:22:54.281455+0000","last_became_active":"2024-03-06T07:22:53.016669+0000","last_became_peered":"2024-03-06T07:22:53.016669+0000","last_unstale":"2024-03-06T07:22:54.281455+0000","last_undegraded":"2024-03-06T07:22:54.281455+0000","last_fullsized":"2024-03-06T07:22:54.281455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:26.061969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282078+0000","last_change":"2024-03-06T07:22:53.011930+0000","last_active":"2024-03-06T07:22:54.282078+0000","last_peered":"2024-03-06T07:22:54.282078+0000","last_clean":"2024-03-06T07:22:54.282078+0000","last_became_active":"2024-03-06T07:22:53.011616+0000","last_became_peered":"2024-03-06T07:22:53.011616+0000","last_unstale":"2024-03-06T07:22:54.282078+0000","last_undegraded":"2024-03-06T07:22:54.282078+0000","last_fullsized":"2024-03-06T07:22:54.282078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:19.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283416+0000","last_change":"2024-03-06T07:22:53.017836+0000","last_active":"2024-03-06T07:22:54.283416+0000","last_peered":"2024-03-06T07:22:54.283416+0000","last_clean":"2024-03-06T07:22:54.283416+0000","last_became_active":"2024-03-06T07:22:53.017495+0000","last_became_peered":"2024-03-06T07:22:53.017495+0000","last_unstale":"2024-03-06T07:22:54.283416+0000","last_undegraded":"2024-03-06T07:22:54.283416+0000","last_fullsized":"2024-03-06T07:22:54.283416+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282521+0000","last_change":"2024-03-06T07:22:53.014100+0000","last_active":"2024-03-06T07:22:54.282521+0000","last_peered":"2024-03-06T07:22:54.282521+0000","last_clean":"2024-03-06T07:22:54.282521+0000","last_became_active":"2024-03-06T07:22:53.013806+0000","last_became_peered":"2024-03-06T07:22:53.013806+0000","last_unstale":"2024-03-06T07:22:54.282521+0000","last_undegraded":"2024-03-06T07:22:54.282521+0000","last_fullsized":"2024-03-06T07:22:54.282521+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282326+0000","last_change":"2024-03-06T07:22:53.009816+0000","last_active":"2024-03-06T07:22:54.282326+0000","last_peered":"2024-03-06T07:22:54.282326+0000","last_clean":"2024-03-06T07:22:54.282326+0000","last_became_active":"2024-03-06T07:22:53.009497+0000","last_became_peered":"2024-03-06T07:22:53.009497+0000","last_unstale":"2024-03-06T07:22:54.282326+0000","last_undegraded":"2024-03-06T07:22:54.282326+0000","last_fullsized":"2024-03-06T07:22:54.282326+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282288+0000","last_change":"2024-03-06T07:22:53.008196+0000","last_active":"2024-03-06T07:22:54.282288+0000","last_peered":"2024-03-06T07:22:54.282288+0000","last_clean":"2024-03-06T07:22:54.282288+0000","last_became_active":"2024-03-06T07:22:53.007916+0000","last_became_peered":"2024-03-06T07:22:53.007916+0000","last_unstale":"2024-03-06T07:22:54.282288+0000","last_undegraded":"2024-03-06T07:22:54.282288+0000","last_fullsized":"2024-03-06T07:22:54.282288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:33.590994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281796+0000","last_change":"2024-03-06T07:22:53.009095+0000","last_active":"2024-03-06T07:22:54.281796+0000","last_peered":"2024-03-06T07:22:54.281796+0000","last_clean":"2024-03-06T07:22:54.281796+0000","last_became_active":"2024-03-06T07:22:53.008716+0000","last_became_peered":"2024-03-06T07:22:53.008716+0000","last_unstale":"2024-03-06T07:22:54.281796+0000","last_undegraded":"2024-03-06T07:22:54.281796+0000","last_fullsized":"2024-03-06T07:22:54.281796+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:58.240026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281677+0000","last_change":"2024-03-06T07:22:53.018889+0000","last_active":"2024-03-06T07:22:54.281677+0000","last_peered":"2024-03-06T07:22:54.281677+0000","last_clean":"2024-03-06T07:22:54.281677+0000","last_became_active":"2024-03-06T07:22:53.018596+0000","last_became_peered":"2024-03-06T07:22:53.018596+0000","last_unstale":"2024-03-06T07:22:54.281677+0000","last_undegraded":"2024-03-06T07:22:54.281677+0000","last_fullsized":"2024-03-06T07:22:54.281677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:17.014845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282871+0000","last_change":"2024-03-06T07:22:53.002357+0000","last_active":"2024-03-06T07:22:54.282871+0000","last_peered":"2024-03-06T07:22:54.282871+0000","last_clean":"2024-03-06T07:22:54.282871+0000","last_became_active":"2024-03-06T07:22:53.002122+0000","last_became_peered":"2024-03-06T07:22:53.002122+0000","last_unstale":"2024-03-06T07:22:54.282871+0000","last_undegraded":"2024-03-06T07:22:54.282871+0000","last_fullsized":"2024-03-06T07:22:54.282871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:13.547310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282221+0000","last_change":"2024-03-06T07:22:53.011490+0000","last_active":"2024-03-06T07:22:54.282221+0000","last_peered":"2024-03-06T07:22:54.282221+0000","last_clean":"2024-03-06T07:22:54.282221+0000","last_became_active":"2024-03-06T07:22:53.011196+0000","last_became_peered":"2024-03-06T07:22:53.011196+0000","last_unstale":"2024-03-06T07:22:54.282221+0000","last_undegraded":"2024-03-06T07:22:54.282221+0000","last_fullsized":"2024-03-06T07:22:54.282221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004180+0000","last_change":"2024-03-06T07:22:53.005749+0000","last_active":"2024-03-06T07:22:54.004180+0000","last_peered":"2024-03-06T07:22:54.004180+0000","last_clean":"2024-03-06T07:22:54.004180+0000","last_became_active":"2024-03-06T07:22:53.005355+0000","last_became_peered":"2024-03-06T07:22:53.005355+0000","last_unstale":"2024-03-06T07:22:54.004180+0000","last_undegraded":"2024-03-06T07:22:54.004180+0000","last_fullsized":"2024-03-06T07:22:54.004180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281870+0000","last_change":"2024-03-06T07:22:53.015800+0000","last_active":"2024-03-06T07:22:54.281870+0000","last_peered":"2024-03-06T07:22:54.281870+0000","last_clean":"2024-03-06T07:22:54.281870+0000","last_became_active":"2024-03-06T07:22:53.015308+0000","last_became_peered":"2024-03-06T07:22:53.015308+0000","last_unstale":"2024-03-06T07:22:54.281870+0000","last_undegraded":"2024-03-06T07:22:54.281870+0000","last_fullsized":"2024-03-06T07:22:54.281870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006942+0000","last_change":"2024-03-06T07:22:53.006841+0000","last_active":"2024-03-06T07:22:54.006942+0000","last_peered":"2024-03-06T07:22:54.006942+0000","last_clean":"2024-03-06T07:22:54.006942+0000","last_became_active":"2024-03-06T07:22:53.006562+0000","last_became_peered":"2024-03-06T07:22:53.006562+0000","last_unstale":"2024-03-06T07:22:54.006942+0000","last_undegraded":"2024-03-06T07:22:54.006942+0000","last_fullsized":"2024-03-06T07:22:54.006942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:08.594990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282409+0000","last_change":"2024-03-06T07:22:53.003197+0000","last_active":"2024-03-06T07:22:54.282409+0000","last_peered":"2024-03-06T07:22:54.282409+0000","last_clean":"2024-03-06T07:22:54.282409+0000","last_became_active":"2024-03-06T07:22:53.002347+0000","last_became_peered":"2024-03-06T07:22:53.002347+0000","last_unstale":"2024-03-06T07:22:54.282409+0000","last_undegraded":"2024-03-06T07:22:54.282409+0000","last_fullsized":"2024-03-06T07:22:54.282409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:25.191495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006974+0000","last_change":"2024-03-06T07:22:53.007183+0000","last_active":"2024-03-06T07:22:54.006974+0000","last_peered":"2024-03-06T07:22:54.006974+0000","last_clean":"2024-03-06T07:22:54.006974+0000","last_became_active":"2024-03-06T07:22:53.006620+0000","last_became_peered":"2024-03-06T07:22:53.006620+0000","last_unstale":"2024-03-06T07:22:54.006974+0000","last_undegraded":"2024-03-06T07:22:54.006974+0000","last_fullsized":"2024-03-06T07:22:54.006974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283597+0000","last_change":"2024-03-06T07:22:53.014691+0000","last_active":"2024-03-06T07:22:54.283597+0000","last_peered":"2024-03-06T07:22:54.283597+0000","last_clean":"2024-03-06T07:22:54.283597+0000","last_became_active":"2024-03-06T07:22:53.014414+0000","last_became_peered":"2024-03-06T07:22:53.014414+0000","last_unstale":"2024-03-06T07:22:54.283597+0000","last_undegraded":"2024-03-06T07:22:54.283597+0000","last_fullsized":"2024-03-06T07:22:54.283597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282606+0000","last_change":"2024-03-06T07:22:53.011594+0000","last_active":"2024-03-06T07:22:54.282606+0000","last_peered":"2024-03-06T07:22:54.282606+0000","last_clean":"2024-03-06T07:22:54.282606+0000","last_became_active":"2024-03-06T07:22:53.011237+0000","last_became_peered":"2024-03-06T07:22:53.011237+0000","last_unstale":"2024-03-06T07:22:54.282606+0000","last_undegraded":"2024-03-06T07:22:54.282606+0000","last_fullsized":"2024-03-06T07:22:54.282606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:50.691781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282618+0000","last_change":"2024-03-06T07:22:53.006582+0000","last_active":"2024-03-06T07:22:54.282618+0000","last_peered":"2024-03-06T07:22:54.282618+0000","last_clean":"2024-03-06T07:22:54.282618+0000","last_became_active":"2024-03-06T07:22:53.006235+0000","last_became_peered":"2024-03-06T07:22:53.006235+0000","last_unstale":"2024-03-06T07:22:54.282618+0000","last_undegraded":"2024-03-06T07:22:54.282618+0000","last_fullsized":"2024-03-06T07:22:54.282618+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006986+0000","last_change":"2024-03-06T07:22:53.008162+0000","last_active":"2024-03-06T07:22:54.006986+0000","last_peered":"2024-03-06T07:22:54.006986+0000","last_clean":"2024-03-06T07:22:54.006986+0000","last_became_active":"2024-03-06T07:22:53.007160+0000","last_became_peered":"2024-03-06T07:22:53.007160+0000","last_unstale":"2024-03-06T07:22:54.006986+0000","last_undegraded":"2024-03-06T07:22:54.006986+0000","last_fullsized":"2024-03-06T07:22:54.006986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:54.295092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281950+0000","last_change":"2024-03-06T07:22:53.005295+0000","last_active":"2024-03-06T07:22:54.281950+0000","last_peered":"2024-03-06T07:22:54.281950+0000","last_clean":"2024-03-06T07:22:54.281950+0000","last_became_active":"2024-03-06T07:22:53.004573+0000","last_became_peered":"2024-03-06T07:22:53.004573+0000","last_unstale":"2024-03-06T07:22:54.281950+0000","last_undegraded":"2024-03-06T07:22:54.281950+0000","last_fullsized":"2024-03-06T07:22:54.281950+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:01.752069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004410+0000","last_change":"2024-03-06T07:22:53.004824+0000","last_active":"2024-03-06T07:22:54.004410+0000","last_peered":"2024-03-06T07:22:54.004410+0000","last_clean":"2024-03-06T07:22:54.004410+0000","last_became_active":"2024-03-06T07:22:53.004472+0000","last_became_peered":"2024-03-06T07:22:53.004472+0000","last_unstale":"2024-03-06T07:22:54.004410+0000","last_undegraded":"2024-03-06T07:22:54.004410+0000","last_fullsized":"2024-03-06T07:22:54.004410+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:13.547310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007338+0000","last_change":"2024-03-06T07:22:51.998349+0000","last_active":"2024-03-06T07:22:54.007338+0000","last_peered":"2024-03-06T07:22:54.007338+0000","last_clean":"2024-03-06T07:22:54.007338+0000","last_became_active":"2024-03-06T07:22:51.998094+0000","last_became_peered":"2024-03-06T07:22:51.998094+0000","last_unstale":"2024-03-06T07:22:54.007338+0000","last_undegraded":"2024-03-06T07:22:54.007338+0000","last_fullsized":"2024-03-06T07:22:54.007338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:22.157883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007280+0000","last_change":"2024-03-06T07:22:53.005215+0000","last_active":"2024-03-06T07:22:54.007280+0000","last_peered":"2024-03-06T07:22:54.007280+0000","last_clean":"2024-03-06T07:22:54.007280+0000","last_became_active":"2024-03-06T07:22:53.004955+0000","last_became_peered":"2024-03-06T07:22:53.004955+0000","last_unstale":"2024-03-06T07:22:54.007280+0000","last_undegraded":"2024-03-06T07:22:54.007280+0000","last_fullsized":"2024-03-06T07:22:54.007280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:13.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004321+0000","last_change":"2024-03-06T07:22:52.006247+0000","last_active":"2024-03-06T07:22:54.004321+0000","last_peered":"2024-03-06T07:22:54.004321+0000","last_clean":"2024-03-06T07:22:54.004321+0000","last_became_active":"2024-03-06T07:22:52.005917+0000","last_became_peered":"2024-03-06T07:22:52.005917+0000","last_unstale":"2024-03-06T07:22:54.004321+0000","last_undegraded":"2024-03-06T07:22:54.004321+0000","last_fullsized":"2024-03-06T07:22:54.004321+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:16.012744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282625+0000","last_change":"2024-03-06T07:22:53.011697+0000","last_active":"2024-03-06T07:22:54.282625+0000","last_peered":"2024-03-06T07:22:54.282625+0000","last_clean":"2024-03-06T07:22:54.282625+0000","last_became_active":"2024-03-06T07:22:53.011304+0000","last_became_peered":"2024-03-06T07:22:53.011304+0000","last_unstale":"2024-03-06T07:22:54.282625+0000","last_undegraded":"2024-03-06T07:22:54.282625+0000","last_fullsized":"2024-03-06T07:22:54.282625+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281799+0000","last_change":"2024-03-06T07:22:52.001909+0000","last_active":"2024-03-06T07:22:54.281799+0000","last_peered":"2024-03-06T07:22:54.281799+0000","last_clean":"2024-03-06T07:22:54.281799+0000","last_became_active":"2024-03-06T07:22:52.001539+0000","last_became_peered":"2024-03-06T07:22:52.001539+0000","last_unstale":"2024-03-06T07:22:54.281799+0000","last_undegraded":"2024-03-06T07:22:54.281799+0000","last_fullsized":"2024-03-06T07:22:54.281799+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:07.099743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006973+0000","last_change":"2024-03-06T07:22:53.006926+0000","last_active":"2024-03-06T07:22:54.006973+0000","last_peered":"2024-03-06T07:22:54.006973+0000","last_clean":"2024-03-06T07:22:54.006973+0000","last_became_active":"2024-03-06T07:22:53.006338+0000","last_became_peered":"2024-03-06T07:22:53.006338+0000","last_unstale":"2024-03-06T07:22:54.006973+0000","last_undegraded":"2024-03-06T07:22:54.006973+0000","last_fullsized":"2024-03-06T07:22:54.006973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:02.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281690+0000","last_change":"2024-03-06T07:22:52.006047+0000","last_active":"2024-03-06T07:22:54.281690+0000","last_peered":"2024-03-06T07:22:54.281690+0000","last_clean":"2024-03-06T07:22:54.281690+0000","last_became_active":"2024-03-06T07:22:52.005713+0000","last_became_peered":"2024-03-06T07:22:52.005713+0000","last_unstale":"2024-03-06T07:22:54.281690+0000","last_undegraded":"2024-03-06T07:22:54.281690+0000","last_fullsized":"2024-03-06T07:22:54.281690+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:12.146531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282779+0000","last_change":"2024-03-06T07:22:53.005043+0000","last_active":"2024-03-06T07:22:54.282779+0000","last_peered":"2024-03-06T07:22:54.282779+0000","last_clean":"2024-03-06T07:22:54.282779+0000","last_became_active":"2024-03-06T07:22:53.004749+0000","last_became_peered":"2024-03-06T07:22:53.004749+0000","last_unstale":"2024-03-06T07:22:54.282779+0000","last_undegraded":"2024-03-06T07:22:54.282779+0000","last_fullsized":"2024-03-06T07:22:54.282779+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282996+0000","last_change":"2024-03-06T07:22:52.003516+0000","last_active":"2024-03-06T07:22:54.282996+0000","last_peered":"2024-03-06T07:22:54.282996+0000","last_clean":"2024-03-06T07:22:54.282996+0000","last_became_active":"2024-03-06T07:22:52.003211+0000","last_became_peered":"2024-03-06T07:22:52.003211+0000","last_unstale":"2024-03-06T07:22:54.282996+0000","last_undegraded":"2024-03-06T07:22:54.282996+0000","last_fullsized":"2024-03-06T07:22:54.282996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:23.061079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283131+0000","last_change":"2024-03-06T07:22:53.014133+0000","last_active":"2024-03-06T07:22:54.283131+0000","last_peered":"2024-03-06T07:22:54.283131+0000","last_clean":"2024-03-06T07:22:54.283131+0000","last_became_active":"2024-03-06T07:22:53.013771+0000","last_became_peered":"2024-03-06T07:22:53.013771+0000","last_unstale":"2024-03-06T07:22:54.283131+0000","last_undegraded":"2024-03-06T07:22:54.283131+0000","last_fullsized":"2024-03-06T07:22:54.283131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:56.430981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282370+0000","last_change":"2024-03-06T07:22:52.003559+0000","last_active":"2024-03-06T07:22:54.282370+0000","last_peered":"2024-03-06T07:22:54.282370+0000","last_clean":"2024-03-06T07:22:54.282370+0000","last_became_active":"2024-03-06T07:22:52.003252+0000","last_became_peered":"2024-03-06T07:22:52.003252+0000","last_unstale":"2024-03-06T07:22:54.282370+0000","last_undegraded":"2024-03-06T07:22:54.282370+0000","last_fullsized":"2024-03-06T07:22:54.282370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:15.933962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282101+0000","last_change":"2024-03-06T07:22:52.996122+0000","last_active":"2024-03-06T07:22:54.282101+0000","last_peered":"2024-03-06T07:22:54.282101+0000","last_clean":"2024-03-06T07:22:54.282101+0000","last_became_active":"2024-03-06T07:22:52.995681+0000","last_became_peered":"2024-03-06T07:22:52.995681+0000","last_unstale":"2024-03-06T07:22:54.282101+0000","last_undegraded":"2024-03-06T07:22:54.282101+0000","last_fullsized":"2024-03-06T07:22:54.282101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:08.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281565+0000","last_change":"2024-03-06T07:22:52.004065+0000","last_active":"2024-03-06T07:22:54.281565+0000","last_peered":"2024-03-06T07:22:54.281565+0000","last_clean":"2024-03-06T07:22:54.281565+0000","last_became_active":"2024-03-06T07:22:52.003744+0000","last_became_peered":"2024-03-06T07:22:52.003744+0000","last_unstale":"2024-03-06T07:22:54.281565+0000","last_undegraded":"2024-03-06T07:22:54.281565+0000","last_fullsized":"2024-03-06T07:22:54.281565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:53.292991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006493+0000","last_change":"2024-03-06T07:22:53.008401+0000","last_active":"2024-03-06T07:22:54.006493+0000","last_peered":"2024-03-06T07:22:54.006493+0000","last_clean":"2024-03-06T07:22:54.006493+0000","last_became_active":"2024-03-06T07:22:53.008139+0000","last_became_peered":"2024-03-06T07:22:53.008139+0000","last_unstale":"2024-03-06T07:22:54.006493+0000","last_undegraded":"2024-03-06T07:22:54.006493+0000","last_fullsized":"2024-03-06T07:22:54.006493+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:04.433486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282222+0000","last_change":"2024-03-06T07:22:52.002965+0000","last_active":"2024-03-06T07:22:54.282222+0000","last_peered":"2024-03-06T07:22:54.282222+0000","last_clean":"2024-03-06T07:22:54.282222+0000","last_became_active":"2024-03-06T07:22:52.002690+0000","last_became_peered":"2024-03-06T07:22:52.002690+0000","last_unstale":"2024-03-06T07:22:54.282222+0000","last_undegraded":"2024-03-06T07:22:54.282222+0000","last_fullsized":"2024-03-06T07:22:54.282222+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:46.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283607+0000","last_change":"2024-03-06T07:22:53.013424+0000","last_active":"2024-03-06T07:22:54.283607+0000","last_peered":"2024-03-06T07:22:54.283607+0000","last_clean":"2024-03-06T07:22:54.283607+0000","last_became_active":"2024-03-06T07:22:53.013124+0000","last_became_peered":"2024-03-06T07:22:53.013124+0000","last_unstale":"2024-03-06T07:22:54.283607+0000","last_undegraded":"2024-03-06T07:22:54.283607+0000","last_fullsized":"2024-03-06T07:22:54.283607+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:30.321003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281752+0000","last_change":"2024-03-06T07:22:52.006835+0000","last_active":"2024-03-06T07:22:54.281752+0000","last_peered":"2024-03-06T07:22:54.281752+0000","last_clean":"2024-03-06T07:22:54.281752+0000","last_became_active":"2024-03-06T07:22:52.006449+0000","last_became_peered":"2024-03-06T07:22:52.006449+0000","last_unstale":"2024-03-06T07:22:54.281752+0000","last_undegraded":"2024-03-06T07:22:54.281752+0000","last_fullsized":"2024-03-06T07:22:54.281752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:49.689679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281589+0000","last_change":"2024-03-06T07:22:53.010786+0000","last_active":"2024-03-06T07:22:54.281589+0000","last_peered":"2024-03-06T07:22:54.281589+0000","last_clean":"2024-03-06T07:22:54.281589+0000","last_became_active":"2024-03-06T07:22:53.010470+0000","last_became_peered":"2024-03-06T07:22:53.010470+0000","last_unstale":"2024-03-06T07:22:54.281589+0000","last_undegraded":"2024-03-06T07:22:54.281589+0000","last_fullsized":"2024-03-06T07:22:54.281589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004403+0000","last_change":"2024-03-06T07:22:51.998392+0000","last_active":"2024-03-06T07:22:54.004403+0000","last_peered":"2024-03-06T07:22:54.004403+0000","last_clean":"2024-03-06T07:22:54.004403+0000","last_became_active":"2024-03-06T07:22:51.998047+0000","last_became_peered":"2024-03-06T07:22:51.998047+0000","last_unstale":"2024-03-06T07:22:54.004403+0000","last_undegraded":"2024-03-06T07:22:54.004403+0000","last_fullsized":"2024-03-06T07:22:54.004403+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:43.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282016+0000","last_change":"2024-03-06T07:22:53.002480+0000","last_active":"2024-03-06T07:22:54.282016+0000","last_peered":"2024-03-06T07:22:54.282016+0000","last_clean":"2024-03-06T07:22:54.282016+0000","last_became_active":"2024-03-06T07:22:53.002193+0000","last_became_peered":"2024-03-06T07:22:53.002193+0000","last_unstale":"2024-03-06T07:22:54.282016+0000","last_undegraded":"2024-03-06T07:22:54.282016+0000","last_fullsized":"2024-03-06T07:22:54.282016+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:02.413213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283382+0000","last_change":"2024-03-06T07:22:52.006845+0000","last_active":"2024-03-06T07:22:54.283382+0000","last_peered":"2024-03-06T07:22:54.283382+0000","last_clean":"2024-03-06T07:22:54.283382+0000","last_became_active":"2024-03-06T07:22:52.006519+0000","last_became_peered":"2024-03-06T07:22:52.006519+0000","last_unstale":"2024-03-06T07:22:54.283382+0000","last_undegraded":"2024-03-06T07:22:54.283382+0000","last_fullsized":"2024-03-06T07:22:54.283382+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:50.470263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283254+0000","last_change":"2024-03-06T07:22:53.009814+0000","last_active":"2024-03-06T07:22:54.283254+0000","last_peered":"2024-03-06T07:22:54.283254+0000","last_clean":"2024-03-06T07:22:54.283254+0000","last_became_active":"2024-03-06T07:22:53.009525+0000","last_became_peered":"2024-03-06T07:22:53.009525+0000","last_unstale":"2024-03-06T07:22:54.283254+0000","last_undegraded":"2024-03-06T07:22:54.283254+0000","last_fullsized":"2024-03-06T07:22:54.283254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282039+0000","last_change":"2024-03-06T07:22:51.999176+0000","last_active":"2024-03-06T07:22:54.282039+0000","last_peered":"2024-03-06T07:22:54.282039+0000","last_clean":"2024-03-06T07:22:54.282039+0000","last_became_active":"2024-03-06T07:22:51.998910+0000","last_became_peered":"2024-03-06T07:22:51.998910+0000","last_unstale":"2024-03-06T07:22:54.282039+0000","last_undegraded":"2024-03-06T07:22:54.282039+0000","last_fullsized":"2024-03-06T07:22:54.282039+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:12.477243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006543+0000","last_change":"2024-03-06T07:22:53.010178+0000","last_active":"2024-03-06T07:22:54.006543+0000","last_peered":"2024-03-06T07:22:54.006543+0000","last_clean":"2024-03-06T07:22:54.006543+0000","last_became_active":"2024-03-06T07:22:53.009892+0000","last_became_peered":"2024-03-06T07:22:53.009892+0000","last_unstale":"2024-03-06T07:22:54.006543+0000","last_undegraded":"2024-03-06T07:22:54.006543+0000","last_fullsized":"2024-03-06T07:22:54.006543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.337698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282986+0000","last_change":"2024-03-06T07:22:52.003595+0000","last_active":"2024-03-06T07:22:54.282986+0000","last_peered":"2024-03-06T07:22:54.282986+0000","last_clean":"2024-03-06T07:22:54.282986+0000","last_became_active":"2024-03-06T07:22:52.003239+0000","last_became_peered":"2024-03-06T07:22:52.003239+0000","last_unstale":"2024-03-06T07:22:54.282986+0000","last_undegraded":"2024-03-06T07:22:54.282986+0000","last_fullsized":"2024-03-06T07:22:54.282986+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:10.020728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281828+0000","last_change":"2024-03-06T07:22:53.004565+0000","last_active":"2024-03-06T07:22:54.281828+0000","last_peered":"2024-03-06T07:22:54.281828+0000","last_clean":"2024-03-06T07:22:54.281828+0000","last_became_active":"2024-03-06T07:22:53.004149+0000","last_became_peered":"2024-03-06T07:22:53.004149+0000","last_unstale":"2024-03-06T07:22:54.281828+0000","last_undegraded":"2024-03-06T07:22:54.281828+0000","last_fullsized":"2024-03-06T07:22:54.281828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:54.253939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006539+0000","last_change":"2024-03-06T07:22:51.997940+0000","last_active":"2024-03-06T07:22:54.006539+0000","last_peered":"2024-03-06T07:22:54.006539+0000","last_clean":"2024-03-06T07:22:54.006539+0000","last_became_active":"2024-03-06T07:22:51.997652+0000","last_became_peered":"2024-03-06T07:22:51.997652+0000","last_unstale":"2024-03-06T07:22:54.006539+0000","last_undegraded":"2024-03-06T07:22:54.006539+0000","last_fullsized":"2024-03-06T07:22:54.006539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:24.629441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282148+0000","last_change":"2024-03-06T07:22:53.005336+0000","last_active":"2024-03-06T07:22:54.282148+0000","last_peered":"2024-03-06T07:22:54.282148+0000","last_clean":"2024-03-06T07:22:54.282148+0000","last_became_active":"2024-03-06T07:22:53.004960+0000","last_became_peered":"2024-03-06T07:22:53.004960+0000","last_unstale":"2024-03-06T07:22:54.282148+0000","last_undegraded":"2024-03-06T07:22:54.282148+0000","last_fullsized":"2024-03-06T07:22:54.282148+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:08.594990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281482+0000","last_change":"2024-03-06T07:22:52.006431+0000","last_active":"2024-03-06T07:22:54.281482+0000","last_peered":"2024-03-06T07:22:54.281482+0000","last_clean":"2024-03-06T07:22:54.281482+0000","last_became_active":"2024-03-06T07:22:52.006190+0000","last_became_peered":"2024-03-06T07:22:52.006190+0000","last_unstale":"2024-03-06T07:22:54.281482+0000","last_undegraded":"2024-03-06T07:22:54.281482+0000","last_fullsized":"2024-03-06T07:22:54.281482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006998+0000","last_change":"2024-03-06T07:22:53.004526+0000","last_active":"2024-03-06T07:22:54.006998+0000","last_peered":"2024-03-06T07:22:54.006998+0000","last_clean":"2024-03-06T07:22:54.006998+0000","last_became_active":"2024-03-06T07:22:53.003998+0000","last_became_peered":"2024-03-06T07:22:53.003998+0000","last_unstale":"2024-03-06T07:22:54.006998+0000","last_undegraded":"2024-03-06T07:22:54.006998+0000","last_fullsized":"2024-03-06T07:22:54.006998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:42.114122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281911+0000","last_change":"2024-03-06T07:22:52.002503+0000","last_active":"2024-03-06T07:22:54.281911+0000","last_peered":"2024-03-06T07:22:54.281911+0000","last_clean":"2024-03-06T07:22:54.281911+0000","last_became_active":"2024-03-06T07:22:52.001554+0000","last_became_peered":"2024-03-06T07:22:52.001554+0000","last_unstale":"2024-03-06T07:22:54.281911+0000","last_undegraded":"2024-03-06T07:22:54.281911+0000","last_fullsized":"2024-03-06T07:22:54.281911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:41.112020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282874+0000","last_change":"2024-03-06T07:22:53.007256+0000","last_active":"2024-03-06T07:22:54.282874+0000","last_peered":"2024-03-06T07:22:54.282874+0000","last_clean":"2024-03-06T07:22:54.282874+0000","last_became_active":"2024-03-06T07:22:53.006985+0000","last_became_peered":"2024-03-06T07:22:53.006985+0000","last_unstale":"2024-03-06T07:22:54.282874+0000","last_undegraded":"2024-03-06T07:22:54.282874+0000","last_fullsized":"2024-03-06T07:22:54.282874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:04.433486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007006+0000","last_change":"2024-03-06T07:22:51.996405+0000","last_active":"2024-03-06T07:22:54.007006+0000","last_peered":"2024-03-06T07:22:54.007006+0000","last_clean":"2024-03-06T07:22:54.007006+0000","last_became_active":"2024-03-06T07:22:51.996082+0000","last_became_peered":"2024-03-06T07:22:51.996082+0000","last_unstale":"2024-03-06T07:22:54.007006+0000","last_undegraded":"2024-03-06T07:22:54.007006+0000","last_fullsized":"2024-03-06T07:22:54.007006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:57.305195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283827+0000","last_change":"2024-03-06T07:22:53.018922+0000","last_active":"2024-03-06T07:22:54.283827+0000","last_peered":"2024-03-06T07:22:54.283827+0000","last_clean":"2024-03-06T07:22:54.283827+0000","last_became_active":"2024-03-06T07:22:53.018664+0000","last_became_peered":"2024-03-06T07:22:53.018664+0000","last_unstale":"2024-03-06T07:22:54.283827+0000","last_undegraded":"2024-03-06T07:22:54.283827+0000","last_fullsized":"2024-03-06T07:22:54.283827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:04.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282921+0000","last_change":"2024-03-06T07:22:52.002762+0000","last_active":"2024-03-06T07:22:54.282921+0000","last_peered":"2024-03-06T07:22:54.282921+0000","last_clean":"2024-03-06T07:22:54.282921+0000","last_became_active":"2024-03-06T07:22:52.002461+0000","last_became_peered":"2024-03-06T07:22:52.002461+0000","last_unstale":"2024-03-06T07:22:54.282921+0000","last_undegraded":"2024-03-06T07:22:54.282921+0000","last_fullsized":"2024-03-06T07:22:54.282921+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.704065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006733+0000","last_change":"2024-03-06T07:22:53.006268+0000","last_active":"2024-03-06T07:22:54.006733+0000","last_peered":"2024-03-06T07:22:54.006733+0000","last_clean":"2024-03-06T07:22:54.006733+0000","last_became_active":"2024-03-06T07:22:53.005993+0000","last_became_peered":"2024-03-06T07:22:53.005993+0000","last_unstale":"2024-03-06T07:22:54.006733+0000","last_undegraded":"2024-03-06T07:22:54.006733+0000","last_fullsized":"2024-03-06T07:22:54.006733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:04.078810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282767+0000","last_change":"2024-03-06T07:22:52.004453+0000","last_active":"2024-03-06T07:22:54.282767+0000","last_peered":"2024-03-06T07:22:54.282767+0000","last_clean":"2024-03-06T07:22:54.282767+0000","last_became_active":"2024-03-06T07:22:52.003997+0000","last_became_peered":"2024-03-06T07:22:52.003997+0000","last_unstale":"2024-03-06T07:22:54.282767+0000","last_undegraded":"2024-03-06T07:22:54.282767+0000","last_fullsized":"2024-03-06T07:22:54.282767+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:37.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281621+0000","last_change":"2024-03-06T07:22:53.014780+0000","last_active":"2024-03-06T07:22:54.281621+0000","last_peered":"2024-03-06T07:22:54.281621+0000","last_clean":"2024-03-06T07:22:54.281621+0000","last_became_active":"2024-03-06T07:22:53.014501+0000","last_became_peered":"2024-03-06T07:22:53.014501+0000","last_unstale":"2024-03-06T07:22:54.281621+0000","last_undegraded":"2024-03-06T07:22:54.281621+0000","last_fullsized":"2024-03-06T07:22:54.281621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.631542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282752+0000","last_change":"2024-03-06T07:22:52.002095+0000","last_active":"2024-03-06T07:22:54.282752+0000","last_peered":"2024-03-06T07:22:54.282752+0000","last_clean":"2024-03-06T07:22:54.282752+0000","last_became_active":"2024-03-06T07:22:52.001749+0000","last_became_peered":"2024-03-06T07:22:52.001749+0000","last_unstale":"2024-03-06T07:22:54.282752+0000","last_undegraded":"2024-03-06T07:22:54.282752+0000","last_fullsized":"2024-03-06T07:22:54.282752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.963836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281407+0000","last_change":"2024-03-06T07:22:53.020156+0000","last_active":"2024-03-06T07:22:54.281407+0000","last_peered":"2024-03-06T07:22:54.281407+0000","last_clean":"2024-03-06T07:22:54.281407+0000","last_became_active":"2024-03-06T07:22:53.019845+0000","last_became_peered":"2024-03-06T07:22:53.019845+0000","last_unstale":"2024-03-06T07:22:54.281407+0000","last_undegraded":"2024-03-06T07:22:54.281407+0000","last_fullsized":"2024-03-06T07:22:54.281407+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:11.022830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004216+0000","last_change":"2024-03-06T07:22:51.998103+0000","last_active":"2024-03-06T07:22:54.004216+0000","last_peered":"2024-03-06T07:22:54.004216+0000","last_clean":"2024-03-06T07:22:54.004216+0000","last_became_active":"2024-03-06T07:22:51.997234+0000","last_became_peered":"2024-03-06T07:22:51.997234+0000","last_unstale":"2024-03-06T07:22:54.004216+0000","last_undegraded":"2024-03-06T07:22:54.004216+0000","last_fullsized":"2024-03-06T07:22:54.004216+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:24.629441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281560+0000","last_change":"2024-03-06T07:22:53.018761+0000","last_active":"2024-03-06T07:22:54.281560+0000","last_peered":"2024-03-06T07:22:54.281560+0000","last_clean":"2024-03-06T07:22:54.281560+0000","last_became_active":"2024-03-06T07:22:53.018435+0000","last_became_peered":"2024-03-06T07:22:53.018435+0000","last_unstale":"2024-03-06T07:22:54.281560+0000","last_undegraded":"2024-03-06T07:22:54.281560+0000","last_fullsized":"2024-03-06T07:22:54.281560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:57.965937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007049+0000","last_change":"2024-03-06T07:22:51.997725+0000","last_active":"2024-03-06T07:22:54.007049+0000","last_peered":"2024-03-06T07:22:54.007049+0000","last_clean":"2024-03-06T07:22:54.007049+0000","last_became_active":"2024-03-06T07:22:51.997442+0000","last_became_peered":"2024-03-06T07:22:51.997442+0000","last_unstale":"2024-03-06T07:22:54.007049+0000","last_undegraded":"2024-03-06T07:22:54.007049+0000","last_fullsized":"2024-03-06T07:22:54.007049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.963836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281313+0000","last_change":"2024-03-06T07:22:53.018347+0000","last_active":"2024-03-06T07:22:54.281313+0000","last_peered":"2024-03-06T07:22:54.281313+0000","last_clean":"2024-03-06T07:22:54.281313+0000","last_became_active":"2024-03-06T07:22:53.018073+0000","last_became_peered":"2024-03-06T07:22:53.018073+0000","last_unstale":"2024-03-06T07:22:54.281313+0000","last_undegraded":"2024-03-06T07:22:54.281313+0000","last_fullsized":"2024-03-06T07:22:54.281313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:24.063180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283184+0000","last_change":"2024-03-06T07:22:52.002891+0000","last_active":"2024-03-06T07:22:54.283184+0000","last_peered":"2024-03-06T07:22:54.283184+0000","last_clean":"2024-03-06T07:22:54.283184+0000","last_became_active":"2024-03-06T07:22:52.002607+0000","last_became_peered":"2024-03-06T07:22:52.002607+0000","last_unstale":"2024-03-06T07:22:54.283184+0000","last_undegraded":"2024-03-06T07:22:54.283184+0000","last_fullsized":"2024-03-06T07:22:54.283184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:25.059867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281923+0000","last_change":"2024-03-06T07:22:53.006187+0000","last_active":"2024-03-06T07:22:54.281923+0000","last_peered":"2024-03-06T07:22:54.281923+0000","last_clean":"2024-03-06T07:22:54.281923+0000","last_became_active":"2024-03-06T07:22:53.005793+0000","last_became_peered":"2024-03-06T07:22:53.005793+0000","last_unstale":"2024-03-06T07:22:54.281923+0000","last_undegraded":"2024-03-06T07:22:54.281923+0000","last_fullsized":"2024-03-06T07:22:54.281923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.599916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283061+0000","last_change":"2024-03-06T07:22:52.006361+0000","last_active":"2024-03-06T07:22:54.283061+0000","last_peered":"2024-03-06T07:22:54.283061+0000","last_clean":"2024-03-06T07:22:54.283061+0000","last_became_active":"2024-03-06T07:22:52.006055+0000","last_became_peered":"2024-03-06T07:22:52.006055+0000","last_unstale":"2024-03-06T07:22:54.283061+0000","last_undegraded":"2024-03-06T07:22:54.283061+0000","last_fullsized":"2024-03-06T07:22:54.283061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:41.112020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281531+0000","last_change":"2024-03-06T07:22:53.020189+0000","last_active":"2024-03-06T07:22:54.281531+0000","last_peered":"2024-03-06T07:22:54.281531+0000","last_clean":"2024-03-06T07:22:54.281531+0000","last_became_active":"2024-03-06T07:22:53.019828+0000","last_became_peered":"2024-03-06T07:22:53.019828+0000","last_unstale":"2024-03-06T07:22:54.281531+0000","last_undegraded":"2024-03-06T07:22:54.281531+0000","last_fullsized":"2024-03-06T07:22:54.281531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:23.159984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007053+0000","last_change":"2024-03-06T07:22:52.000983+0000","last_active":"2024-03-06T07:22:54.007053+0000","last_peered":"2024-03-06T07:22:54.007053+0000","last_clean":"2024-03-06T07:22:54.007053+0000","last_became_active":"2024-03-06T07:22:52.000680+0000","last_became_peered":"2024-03-06T07:22:52.000680+0000","last_unstale":"2024-03-06T07:22:54.007053+0000","last_undegraded":"2024-03-06T07:22:54.007053+0000","last_fullsized":"2024-03-06T07:22:54.007053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283816+0000","last_change":"2024-03-06T07:22:53.014876+0000","last_active":"2024-03-06T07:22:54.283816+0000","last_peered":"2024-03-06T07:22:54.283816+0000","last_clean":"2024-03-06T07:22:54.283816+0000","last_became_active":"2024-03-06T07:22:53.014322+0000","last_became_peered":"2024-03-06T07:22:53.014322+0000","last_unstale":"2024-03-06T07:22:54.283816+0000","last_undegraded":"2024-03-06T07:22:54.283816+0000","last_fullsized":"2024-03-06T07:22:54.283816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:19.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004490+0000","last_change":"2024-03-06T07:22:51.998753+0000","last_active":"2024-03-06T07:22:54.004490+0000","last_peered":"2024-03-06T07:22:54.004490+0000","last_clean":"2024-03-06T07:22:54.004490+0000","last_became_active":"2024-03-06T07:22:51.998428+0000","last_became_peered":"2024-03-06T07:22:51.998428+0000","last_unstale":"2024-03-06T07:22:54.004490+0000","last_undegraded":"2024-03-06T07:22:54.004490+0000","last_fullsized":"2024-03-06T07:22:54.004490+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:52.691274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004285+0000","last_change":"2024-03-06T07:22:53.012683+0000","last_active":"2024-03-06T07:22:54.004285+0000","last_peered":"2024-03-06T07:22:54.004285+0000","last_clean":"2024-03-06T07:22:54.004285+0000","last_became_active":"2024-03-06T07:22:53.012154+0000","last_became_peered":"2024-03-06T07:22:53.012154+0000","last_unstale":"2024-03-06T07:22:54.004285+0000","last_undegraded":"2024-03-06T07:22:54.004285+0000","last_fullsized":"2024-03-06T07:22:54.004285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006819+0000","last_change":"2024-03-06T07:22:52.004074+0000","last_active":"2024-03-06T07:22:54.006819+0000","last_peered":"2024-03-06T07:22:54.006819+0000","last_clean":"2024-03-06T07:22:54.006819+0000","last_became_active":"2024-03-06T07:22:52.003773+0000","last_became_peered":"2024-03-06T07:22:52.003773+0000","last_unstale":"2024-03-06T07:22:54.006819+0000","last_undegraded":"2024-03-06T07:22:54.006819+0000","last_fullsized":"2024-03-06T07:22:54.006819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:43.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007154+0000","last_change":"2024-03-06T07:22:53.001845+0000","last_active":"2024-03-06T07:22:54.007154+0000","last_peered":"2024-03-06T07:22:54.007154+0000","last_clean":"2024-03-06T07:22:54.007154+0000","last_became_active":"2024-03-06T07:22:53.001480+0000","last_became_peered":"2024-03-06T07:22:53.001480+0000","last_unstale":"2024-03-06T07:22:54.007154+0000","last_undegraded":"2024-03-06T07:22:54.007154+0000","last_fullsized":"2024-03-06T07:22:54.007154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:51.472365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004349+0000","last_change":"2024-03-06T07:22:52.001878+0000","last_active":"2024-03-06T07:22:54.004349+0000","last_peered":"2024-03-06T07:22:54.004349+0000","last_clean":"2024-03-06T07:22:54.004349+0000","last_became_active":"2024-03-06T07:22:52.001618+0000","last_became_peered":"2024-03-06T07:22:52.001618+0000","last_unstale":"2024-03-06T07:22:54.004349+0000","last_undegraded":"2024-03-06T07:22:54.004349+0000","last_fullsized":"2024-03-06T07:22:54.004349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:57.305195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.004250+0000","last_change":"2024-03-06T07:22:53.010139+0000","last_active":"2024-03-06T07:22:54.004250+0000","last_peered":"2024-03-06T07:22:54.004250+0000","last_clean":"2024-03-06T07:22:54.004250+0000","last_became_active":"2024-03-06T07:22:53.009822+0000","last_became_peered":"2024-03-06T07:22:53.009822+0000","last_unstale":"2024-03-06T07:22:54.004250+0000","last_undegraded":"2024-03-06T07:22:54.004250+0000","last_fullsized":"2024-03-06T07:22:54.004250+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:50.691781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.006648+0000","last_change":"2024-03-06T07:22:51.999813+0000","last_active":"2024-03-06T07:22:54.006648+0000","last_peered":"2024-03-06T07:22:54.006648+0000","last_clean":"2024-03-06T07:22:54.006648+0000","last_became_active":"2024-03-06T07:22:51.999483+0000","last_became_peered":"2024-03-06T07:22:51.999483+0000","last_unstale":"2024-03-06T07:22:54.006648+0000","last_undegraded":"2024-03-06T07:22:54.006648+0000","last_fullsized":"2024-03-06T07:22:54.006648+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:16.012744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281797+0000","last_change":"2024-03-06T07:22:53.011139+0000","last_active":"2024-03-06T07:22:54.281797+0000","last_peered":"2024-03-06T07:22:54.281797+0000","last_clean":"2024-03-06T07:22:54.281797+0000","last_became_active":"2024-03-06T07:22:53.010277+0000","last_became_peered":"2024-03-06T07:22:53.010277+0000","last_unstale":"2024-03-06T07:22:54.281797+0000","last_undegraded":"2024-03-06T07:22:54.281797+0000","last_fullsized":"2024-03-06T07:22:54.281797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:43.372586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283282+0000","last_change":"2024-03-06T07:22:52.003098+0000","last_active":"2024-03-06T07:22:54.283282+0000","last_peered":"2024-03-06T07:22:54.283282+0000","last_clean":"2024-03-06T07:22:54.283282+0000","last_became_active":"2024-03-06T07:22:52.002798+0000","last_became_peered":"2024-03-06T07:22:52.002798+0000","last_unstale":"2024-03-06T07:22:54.283282+0000","last_undegraded":"2024-03-06T07:22:54.283282+0000","last_fullsized":"2024-03-06T07:22:54.283282+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:22.157883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.283083+0000","last_change":"2024-03-06T07:22:53.003411+0000","last_active":"2024-03-06T07:22:54.283083+0000","last_peered":"2024-03-06T07:22:54.283083+0000","last_clean":"2024-03-06T07:22:54.283083+0000","last_became_active":"2024-03-06T07:22:53.003120+0000","last_became_peered":"2024-03-06T07:22:53.003120+0000","last_unstale":"2024-03-06T07:22:54.283083+0000","last_undegraded":"2024-03-06T07:22:54.283083+0000","last_fullsized":"2024-03-06T07:22:54.283083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:04.078810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.281937+0000","last_change":"2024-03-06T07:22:52.007081+0000","last_active":"2024-03-06T07:22:54.281937+0000","last_peered":"2024-03-06T07:22:54.281937+0000","last_clean":"2024-03-06T07:22:54.281937+0000","last_became_active":"2024-03-06T07:22:52.006599+0000","last_became_peered":"2024-03-06T07:22:52.006599+0000","last_unstale":"2024-03-06T07:22:54.281937+0000","last_undegraded":"2024-03-06T07:22:54.281937+0000","last_fullsized":"2024-03-06T07:22:54.281937+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:50.470263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282354+0000","last_change":"2024-03-06T07:22:53.007286+0000","last_active":"2024-03-06T07:22:54.282354+0000","last_peered":"2024-03-06T07:22:54.282354+0000","last_clean":"2024-03-06T07:22:54.282354+0000","last_became_active":"2024-03-06T07:22:53.007032+0000","last_became_peered":"2024-03-06T07:22:53.007032+0000","last_unstale":"2024-03-06T07:22:54.282354+0000","last_undegraded":"2024-03-06T07:22:54.282354+0000","last_fullsized":"2024-03-06T07:22:54.282354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:51.982203+0000","last_clean_scrub_stamp":"2024-03-06T07:22:51.982203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:03.529976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.007110+0000","last_change":"2024-03-06T07:22:52.003894+0000","last_active":"2024-03-06T07:22:54.007110+0000","last_peered":"2024-03-06T07:22:54.007110+0000","last_clean":"2024-03-06T07:22:54.007110+0000","last_became_active":"2024-03-06T07:22:52.003608+0000","last_became_peered":"2024-03-06T07:22:52.003608+0000","last_unstale":"2024-03-06T07:22:54.007110+0000","last_undegraded":"2024-03-06T07:22:54.007110+0000","last_fullsized":"2024-03-06T07:22:54.007110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.980101+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.980101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:25.059867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:54.282579+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:22:54.282579+0000","last_peered":"2024-03-06T07:22:54.282579+0000","last_clean":"2024-03-06T07:22:54.282579+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:22:54.282579+0000","last_undegraded":"2024-03-06T07:22:54.282579+0000","last_fullsized":"2024-03-06T07:22:54.282579+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":736976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:01.963 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:01.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.108+0000 7f466bbae640 1 Processor -- start 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.109+0000 7f466bbae640 1 -- start start 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.109+0000 7f466bbae640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f46640d4860 0x7f46640d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.109+0000 7f466bbae640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46641618b0 con 0x7f46640b0e30 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.109+0000 7f466bbae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46640cf400 con 0x7f46641614d0 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.109+0000 7f466bbae640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46640d4cf0 con 0x7f46640d4860 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f466a14b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f46640b0e30 0x7f46640b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56654/0 (socket says 172.21.15.122:56654) 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f466a14b640 1 -- 172.21.15.122:0/2432524548 learned_addr learned my addr 172.21.15.122:0/2432524548 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 --2- 172.21.15.122:0/2432524548 >> v2:172.21.15.169:3300/0 conn(0x7f46640d4860 0x7f46640d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128932992 0 0) 0x7f46641618b0 con 0x7f46640b0e30 2024-03-06T07:23:02.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4650003520 con 0x7f46640b0e30 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 443611574 0 0) 0x7f4650003520 con 0x7f46640b0e30 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 >> v2:172.21.15.169:3300/0 conn(0x7f46640d4860 msgr2=0x7f46640d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 --2- 172.21.15.122:0/2432524548 >> v2:172.21.15.169:3300/0 conn(0x7f46640d4860 0x7f46640d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 >> v1:172.21.15.169:6789/0 conn(0x7f46641614d0 legacy=0x7f46640d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640cf6a0 con 0x7f46640b0e30 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f465c001a10 con 0x7f46640b0e30 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 >> v1:172.21.15.122:6789/0 conn(0x7f46640b0e30 legacy=0x7f46640b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46500039b0 0x7f4650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4650004800 0x7f4650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46500051c0 0x7f465000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640d4cf0 con 0x7f4650004800 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465000c060 con 0x7f46500039b0 2024-03-06T07:23:02.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465000c260 con 0x7f46500051c0 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4650004800 0x7f4650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f466994a640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46500039b0 0x7f4650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f466a14b640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46500051c0 0x7f465000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 -- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46500051c0 msgr2=0x7f465000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46500051c0 0x7f465000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 -- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46500039b0 msgr2=0x7f4650003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46500039b0 0x7f4650003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.110+0000 7f4669149640 1 -- 172.21.15.122:0/2432524548 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650003520 con 0x7f4650004800 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.111+0000 7f466a14b640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46500051c0 0x7f465000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.111+0000 7f466a14b640 1 -- 172.21.15.122:0/2432524548 reap_dead start 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.111+0000 7f4669149640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4650004800 0x7f4650004c80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4660009ac0 tx=0x7f4650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65ab3c3c0385cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.111+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660039070 con 0x7f4650004800 2024-03-06T07:23:02.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.111+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466002c6b0 con 0x7f4650004800 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.111+0000 7f4668948640 1 -- 172.21.15.122:0/2432524548 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660009690 con 0x7f4650004800 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 -- 172.21.15.122:0/2432524548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4650004800 msgr2=0x7f4650004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4650004800 0x7f4650004c80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4660009ac0 tx=0x7f4650005830 comp rx=0 tx=0).stop 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 -- 172.21.15.122:0/2432524548 shutdown_connections 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 --2- 172.21.15.122:0/2432524548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4650004800 0x7f4650004c80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 -- 172.21.15.122:0/2432524548 >> 172.21.15.122:0/2432524548 conn(0x7f46640d3d40 msgr2=0x7f46640d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 -- 172.21.15.122:0/2432524548 shutdown_connections 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.112+0000 7f466bbae640 1 -- 172.21.15.122:0/2432524548 wait complete. 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.113+0000 7f466bbae640 1 Processor -- start 2024-03-06T07:23:02.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.113+0000 7f466bbae640 1 -- start start 2024-03-06T07:23:02.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466bbae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4650004800 0x7f46640a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466bbae640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46640c7870 0x7f4664151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466bbae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 0x7f46640b8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466bbae640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640b0910 con 0x7f46640b8b00 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466bbae640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640cfc60 con 0x7f4650004800 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466bbae640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640d1910 con 0x7f46640c7870 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 0x7f46640b8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 0x7f46640b8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38558/0 (socket says 172.21.15.122:38558) 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 -- 172.21.15.122:0/3279656408 learned_addr learned my addr 172.21.15.122:0/3279656408 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f4669149640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46640c7870 0x7f4664151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466994a640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4650004800 0x7f46640a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 -- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46640c7870 msgr2=0x7f4664151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46640c7870 0x7f4664151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 -- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4650004800 msgr2=0x7f46640a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4650004800 0x7f46640a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 -- 172.21.15.122:0/3279656408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664152140 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466994a640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4650004800 0x7f46640a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.114+0000 7f466a14b640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 0x7f46640b8f80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4664161a80 tx=0x7f465c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.115+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c00ebc0 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.115+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c00ed60 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.115+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c014670 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.115+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46641523e0 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.115+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46641529a0 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.116+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46640cdf40 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.116+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f465c014810 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.116+0000 7f46567fc640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f464806b500 0x7f464806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.117+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f465c089ed0 con 0x7f46640b8b00 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.117+0000 7f466994a640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f464806b500 0x7f464806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.117+0000 7f466994a640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f464806b500 0x7f464806d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4658000c00 tx=0x7f4658009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.121+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46640cdf40 con 0x7f46640b8b00 2024-03-06T07:23:02.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.366+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4664068ff0 con 0x7f46640b8b00 2024-03-06T07:23:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.368+0000 7f46567fc640 1 -- 172.21.15.122:0/3279656408 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f465c058700 con 0x7f46640b8b00 2024-03-06T07:23:02.369 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:23:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:23:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:23:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:23:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f464806b500 msgr2=0x7f464806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f464806b500 0x7f464806d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4658000c00 tx=0x7f4658009040 comp rx=0 tx=0).stop 2024-03-06T07:23:02.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 msgr2=0x7f46640b8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 0x7f46640b8f80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4664161a80 tx=0x7f465c00e530 comp rx=0 tx=0).stop 2024-03-06T07:23:02.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 shutdown_connections 2024-03-06T07:23:02.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f464806b500 0x7f464806d9c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46640b8b00 0x7f46640b8f80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f46640c7870 0x7f4664151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 --2- 172.21.15.122:0/3279656408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4650004800 0x7f46640a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 >> 172.21.15.122:0/3279656408 conn(0x7f46640d3d40 msgr2=0x7f46640b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 shutdown_connections 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.370+0000 7f466bbae640 1 -- 172.21.15.122:0/3279656408 wait complete. 2024-03-06T07:23:02.383 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:02.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.482+0000 7f2726755640 1 Processor -- start 2024-03-06T07:23:02.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.482+0000 7f2726755640 1 -- start start 2024-03-06T07:23:02.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f2726755640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f272006da40 0x7f27200cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f2726755640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27200cc980 con 0x7f27201424f0 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f2726755640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27200ccc60 con 0x7f27200cd7a0 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f2726755640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27200cd220 con 0x7f272006da40 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f271f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f272006da40 0x7f27200cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f271ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f27201424f0 0x7f272006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56664/0 (socket says 172.21.15.122:56664) 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f271ffff640 1 -- 172.21.15.122:0/2449103955 learned_addr learned my addr 172.21.15.122:0/2449103955 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f271f7fe640 1 -- 172.21.15.122:0/2449103955 >> v1:172.21.15.169:6789/0 conn(0x7f27200cd7a0 legacy=0x7f2720071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.483+0000 7f271effd640 1 -- 172.21.15.122:0/2449103955 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942091874 0 0) 0x7f27200cc980 con 0x7f27201424f0 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.484+0000 7f271f7fe640 1 -- 172.21.15.122:0/2449103955 >> v1:172.21.15.122:6789/0 conn(0x7f27201424f0 legacy=0x7f272006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.484+0000 7f271f7fe640 1 -- 172.21.15.122:0/2449103955 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27200ccf40 con 0x7f272006da40 2024-03-06T07:23:02.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.484+0000 7f271f7fe640 1 --2- 172.21.15.122:0/2449103955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272006da40 0x7f27200cc2f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f27140097a0 tx=0x7f271402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7864ad2bd1b95675 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.484+0000 7f271effd640 1 -- 172.21.15.122:0/2449103955 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714034070 con 0x7f272006da40 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f271effd640 1 -- 172.21.15.122:0/2449103955 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714030610 con 0x7f272006da40 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f271effd640 1 -- 172.21.15.122:0/2449103955 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140309e0 con 0x7f272006da40 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 -- 172.21.15.122:0/2449103955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272006da40 msgr2=0x7f27200cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 --2- 172.21.15.122:0/2449103955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272006da40 0x7f27200cc2f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f27140097a0 tx=0x7f271402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 -- 172.21.15.122:0/2449103955 shutdown_connections 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 --2- 172.21.15.122:0/2449103955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f272006da40 0x7f27200cc2f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 -- 172.21.15.122:0/2449103955 >> 172.21.15.122:0/2449103955 conn(0x7f27200cf8a0 msgr2=0x7f27200cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 -- 172.21.15.122:0/2449103955 shutdown_connections 2024-03-06T07:23:02.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.485+0000 7f2726755640 1 -- 172.21.15.122:0/2449103955 wait complete. 2024-03-06T07:23:02.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.487+0000 7f2726755640 1 Processor -- start 2024-03-06T07:23:02.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.487+0000 7f2726755640 1 -- start start 2024-03-06T07:23:02.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2726755640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f272006da40 0x7f272013f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2726755640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27200cd7a0 0x7f2720156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2726755640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 0x7f2720140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2726755640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272015d070 con 0x7f27201424f0 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2726755640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272015cef0 con 0x7f272006da40 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2726755640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272015d1f0 con 0x7f27200cd7a0 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2724cf2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 0x7f2720140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2724cf2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 0x7f2720140b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38580/0 (socket says 172.21.15.122:38580) 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f2724cf2640 1 -- 172.21.15.122:0/1444881397 learned_addr learned my addr 172.21.15.122:0/1444881397 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f271ffff640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f272006da40 0x7f272013f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.488+0000 7f271f7fe640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27200cd7a0 0x7f2720156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2724cf2640 1 -- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27200cd7a0 msgr2=0x7f2720156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2724cf2640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27200cd7a0 0x7f2720156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2724cf2640 1 -- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f272006da40 msgr2=0x7f272013f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2724cf2640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f272006da40 0x7f272013f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2724cf2640 1 -- 172.21.15.122:0/1444881397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272013f7c0 con 0x7f27201424f0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2724cf2640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 0x7f2720140b30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f271800c7a0 tx=0x7f271800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718003b70 con 0x7f27201424f0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2718003d30 con 0x7f27201424f0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27201593a0 con 0x7f27201424f0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271800aa90 con 0x7f27201424f0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.489+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2720159960 con 0x7f27201424f0 2024-03-06T07:23:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.490+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272004f330 con 0x7f27201424f0 2024-03-06T07:23:02.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.490+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f271800ac30 con 0x7f27201424f0 2024-03-06T07:23:02.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.491+0000 7f271cff9640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f270c06b4d0 0x7f270c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.491+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f271808a150 con 0x7f27201424f0 2024-03-06T07:23:02.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.491+0000 7f271ffff640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f270c06b4d0 0x7f270c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.491+0000 7f271ffff640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f270c06b4d0 0x7f270c06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2710000f60 tx=0x7f2710007d00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.499+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f272013f7c0 con 0x7f27201424f0 2024-03-06T07:23:02.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.714+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27200d0620 con 0x7f27201424f0 2024-03-06T07:23:02.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.714+0000 7f271cff9640 1 -- 172.21.15.122:0/1444881397 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f2718058fd0 con 0x7f27201424f0 2024-03-06T07:23:02.716 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:02.717 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:02.717 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:02.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f270c06b4d0 msgr2=0x7f270c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f270c06b4d0 0x7f270c06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2710000f60 tx=0x7f2710007d00 comp rx=0 tx=0).stop 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 msgr2=0x7f2720140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 0x7f2720140b30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f271800c7a0 tx=0x7f271800cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 shutdown_connections 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f270c06b4d0 0x7f270c06d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27201424f0 0x7f2720140b30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27200cd7a0 0x7f2720156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 --2- 172.21.15.122:0/1444881397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f272006da40 0x7f272013f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 >> 172.21.15.122:0/1444881397 conn(0x7f27200cf8a0 msgr2=0x7f27200ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 shutdown_connections 2024-03-06T07:23:02.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:02.716+0000 7f2726755640 1 -- 172.21.15.122:0/1444881397 wait complete. 2024-03-06T07:23:03.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:52.342190+0000', 'modified': '2024-03-06T07:22:55.380259+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5262}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5262': {'gid': 5262, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6835/2892378792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 2892378792}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 2892378792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5262, 'qdb_cluster': [5262]} 2024-03-06T07:23:03.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:03.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.232+0000 7fd42498c640 1 Processor -- start 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.232+0000 7fd42498c640 1 -- start start 2024-03-06T07:23:03.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd42498c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd4200d0d90 0x7fd4200ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd42498c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4200ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd42498c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4200d25e0 con 0x7fd42015e000 2024-03-06T07:23:03.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd42498c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4200c7c50 con 0x7fd4200d0d90 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd42498c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d2880 con 0x7fd4200c90f0 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd4200d0d90 0x7fd4200ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd4200d0d90 0x7fd4200ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38586/0 (socket says 172.21.15.122:38586) 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1772581377 learned_addr learned my addr 172.21.15.122:0/1772581377 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/1772581377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4200ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1772581377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 msgr2=0x7fd4200ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/1772581377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4200ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1772581377 >> v1:172.21.15.169:6789/0 conn(0x7fd42015e000 legacy=0x7fd4200d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.233+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1772581377 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4200d2b40 con 0x7fd4200d0d90 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.234+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/1772581377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd4200ce7c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd414009870 tx=0x7fd41402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1066301f0c334a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.234+0000 7fd41d59a640 1 -- 172.21.15.122:0/1772581377 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414034070 con 0x7fd4200d0d90 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.234+0000 7fd41d59a640 1 -- 172.21.15.122:0/1772581377 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414030840 con 0x7fd4200d0d90 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.234+0000 7fd41d59a640 1 -- 172.21.15.122:0/1772581377 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414030c10 con 0x7fd4200d0d90 2024-03-06T07:23:03.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 -- 172.21.15.122:0/1772581377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 msgr2=0x7fd4200ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 --2- 172.21.15.122:0/1772581377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd4200ce7c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd414009870 tx=0x7fd41402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:03.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 -- 172.21.15.122:0/1772581377 shutdown_connections 2024-03-06T07:23:03.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 --2- 172.21.15.122:0/1772581377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4200ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 --2- 172.21.15.122:0/1772581377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd4200ce7c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 -- 172.21.15.122:0/1772581377 >> 172.21.15.122:0/1772581377 conn(0x7fd4200d3cf0 msgr2=0x7fd4200d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 -- 172.21.15.122:0/1772581377 shutdown_connections 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.235+0000 7fd42498c640 1 -- 172.21.15.122:0/1772581377 wait complete. 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.236+0000 7fd42498c640 1 Processor -- start 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.236+0000 7fd42498c640 1 -- start start 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd42498c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4201e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd42498c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd420150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd42498c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd42015e000 0x7fd4201e3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd42498c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42015e3e0 con 0x7fd4200d0d90 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd42498c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d22d0 con 0x7fd42015e000 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd42498c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200d4810 con 0x7fd4200c90f0 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd420150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd420150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38602/0 (socket says 172.21.15.122:38602) 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2396474523 learned_addr learned my addr 172.21.15.122:0/2396474523 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd42015e000 0x7fd4201e3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4201e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 msgr2=0x7fd4201e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4201e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd42015e000 msgr2=0x7fd4201e3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.237+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd42015e000 0x7fd4201e3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2396474523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420150690 con 0x7fd4200d0d90 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd420150150 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd414000c00 tx=0x7fd414004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd40effd640 1 -- 172.21.15.122:0/2396474523 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414048070 con 0x7fd4200d0d90 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd40effd640 1 -- 172.21.15.122:0/2396474523 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414030640 con 0x7fd4200d0d90 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd40effd640 1 -- 172.21.15.122:0/2396474523 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414034040 con 0x7fd4200d0d90 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4201ea0b0 con 0x7fd4200d0d90 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.238+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4201e9dd0 con 0x7fd4200d0d90 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.239+0000 7fd40effd640 1 -- 172.21.15.122:0/2396474523 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd414037070 con 0x7fd4200d0d90 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.240+0000 7fd40effd640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd40006b500 0x7fd40006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.240+0000 7fd40effd640 1 -- 172.21.15.122:0/2396474523 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4140ac090 con 0x7fd4200d0d90 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.240+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd40006b500 0x7fd40006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.240+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd40006b500 0x7fd40006d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd410006fd0 tx=0x7fd410008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.241+0000 7fd42757b740 1 Processor -- start 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.241+0000 7fd42757b740 1 -- start start 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a5e870 0x560355a5ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd42757b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 0x560355a64360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a8e600 0x560355a8ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd42757b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603559c86b0 con 0x560355a628c0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560355971a60 con 0x560355a5e870 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603559c4750 con 0x560355a8e600 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 0x560355a64360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 0x560355a64360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38606/0 (socket says 172.21.15.122:38606) 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2727343916 learned_addr learned my addr 172.21.15.122:0/2727343916 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a5e870 0x560355a5ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a8e600 0x560355a8ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a8e600 msgr2=0x560355a8ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a8e600 0x560355a8ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a5e870 msgr2=0x560355a5ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a5e870 0x560355a5ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 -- 172.21.15.122:0/2727343916 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56035598ff30 con 0x560355a628c0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a8e600 0x560355a8ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 0x560355a64360 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd4140aa670 tx=0x7fd41407b140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a94461a8ebe3d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd3fffff640 1 -- 172.21.15.122:0/2727343916 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140830d0 con 0x560355a628c0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd3fffff640 1 -- 172.21.15.122:0/2727343916 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414078a60 con 0x560355a628c0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.242+0000 7fd3fffff640 1 -- 172.21.15.122:0/2727343916 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414077030 con 0x560355a628c0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 -- 172.21.15.122:0/2727343916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 msgr2=0x560355a64360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 0x560355a64360 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd4140aa670 tx=0x7fd41407b140 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 -- 172.21.15.122:0/2727343916 shutdown_connections 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a8e600 0x560355a8ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a628c0 0x560355a64360 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 --2- 172.21.15.122:0/2727343916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a5e870 0x560355a5ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 -- 172.21.15.122:0/2727343916 >> 172.21.15.122:0/2727343916 conn(0x560355a608f0 msgr2=0x560355a621b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 -- 172.21.15.122:0/2727343916 shutdown_connections 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.243+0000 7fd42757b740 1 -- 172.21.15.122:0/2727343916 wait complete. 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.244+0000 7fd42757b740 1 -- start start 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.244+0000 7fd42757b740 1 Processor -- start 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a8e600 0x560355aa9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 0x560355a8f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88230 0x560355a63cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603559c86b0 con 0x560355a88730 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560355984ea0 con 0x560355a8e600 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x560355a62d50 con 0x560355a88230 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 10 client.0 _ll_register_callbacks cb 0x560355984870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 0x560355a8f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 0x560355a8f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38612/0 (socket says 172.21.15.122:38612) 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1878260056 learned_addr learned my addr 172.21.15.122:0/1878260056 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88230 0x560355a63cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41e59c640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a8e600 0x560355aa9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88230 msgr2=0x560355a63cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88230 0x560355a63cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a8e600 msgr2=0x560355aa9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a8e600 0x560355aa9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 -- 172.21.15.122:0/1878260056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560355985270 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 0x560355a8f870 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd41407b270 tx=0x7fd4140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ad83744e6f232 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd3fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/1878260056 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140307f0 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/1878260056 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4140aa220 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.245+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd414078a60 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/1878260056 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd414079e10 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 10 client.5475 fetch_fsmap learned FSMap version 15 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x5603556dd2d0 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/1878260056 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414078a60 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/1878260056 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 15) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd414079fb0 con 0x560355a88730 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 10 client.5475 fetch_fsmap finished waiting for FSMap version 15 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 10 client.5475 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 20 client.5475 populate_metadata read hostname 'smithi122' 2024-03-06T07:23:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355aa5060 0x560355aa5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 4 client.5475 mds_command: new command op to 5262 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 --> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603559d5bf0 con 0x560355aa5060 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd41e59c640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355aa5060 0x560355aa5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd3fd7fa640 10 client.5475 ms_handle_connect on v2:172.21.15.122:6834/2892378792 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.246+0000 7fd41e59c640 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355aa5060 0x560355aa5440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f13f708232ecd52c server_cookie=813232c5087fcc03 in_seq=0 out_seq=0 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/1878260056 <== mds.0 v2:172.21.15.122:6834/2892378792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5603559d5bf0 con 0x560355aa5060 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd3fd7fa640 10 client.5475 handle_command_reply: tid=0 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 client.5475 shutdown 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 msgr2=0x560355a8f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 0x560355a8f870 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd41407b270 tx=0x7fd4140046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 shutdown_connections 2024-03-06T07:23:03.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355aa5060 0x560355aa5440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88230 0x560355a63cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a88730 0x560355a8f870 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 --2- 172.21.15.122:0/1878260056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a8e600 0x560355aa9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd3fd7fa640 0 client.5475 ms_handle_reset on v2:172.21.15.122:6834/2892378792 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 >> 172.21.15.122:0/1878260056 conn(0x560355a5e870 msgr2=0x560355a5d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 shutdown_connections 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.247+0000 7fd42757b740 1 -- 172.21.15.122:0/1878260056 wait complete. 2024-03-06T07:23:03.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.248+0000 7fd42757b740 20 client.5475 trim_cache size 0 max 16384 2024-03-06T07:23:03.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.264+0000 7fd42757b740 1 Processor -- start 2024-03-06T07:23:03.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.264+0000 7fd42757b740 1 -- start start 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355aa5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd42757b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355aa5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd42757b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4201ff560 con 0x560355a8e600 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4201e87f0 con 0x560355a88230 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4201e8df0 con 0x560355a88730 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355aa5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355aa5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38622/0 (socket says 172.21.15.122:38622) 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2676428678 learned_addr learned my addr 172.21.15.122:0/2676428678 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355aa5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 msgr2=0x560355aa5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355aa5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 msgr2=0x560355aa31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2676428678 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603557a76b0 con 0x560355a8e600 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.265+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355aa5980 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd418007f80 tx=0x7fd4180114e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7b4d3380277144 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd3ff7fe640 1 -- 172.21.15.122:0/2676428678 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41800f890 con 0x560355a8e600 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd3ff7fe640 1 -- 172.21.15.122:0/2676428678 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41800fa30 con 0x560355a8e600 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd3ff7fe640 1 -- 172.21.15.122:0/2676428678 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41800fe20 con 0x560355a8e600 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 -- 172.21.15.122:0/2676428678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 msgr2=0x560355aa5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355aa5980 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd418007f80 tx=0x7fd4180114e0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 -- 172.21.15.122:0/2676428678 shutdown_connections 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355aa5980 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355aa5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 --2- 172.21.15.122:0/2676428678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 -- 172.21.15.122:0/2676428678 >> 172.21.15.122:0/2676428678 conn(0x560355a5e870 msgr2=0x560355aa9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 -- 172.21.15.122:0/2676428678 shutdown_connections 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.266+0000 7fd42757b740 1 -- 172.21.15.122:0/2676428678 wait complete. 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.267+0000 7fd42757b740 1 -- start start 2024-03-06T07:23:03.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.267+0000 7fd42757b740 1 Processor -- start 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355b18810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355a8e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603559c86b0 con 0x560355a8e600 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560355984ea0 con 0x560355a88230 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x560355a89e50 con 0x560355a88730 2024-03-06T07:23:03.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 10 client.0 _ll_register_callbacks cb 0x7fd4140830d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355a8e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355a8e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38632/0 (socket says 172.21.15.122:38632) 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2707028335 learned_addr learned my addr 172.21.15.122:0/2707028335 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41dd9b640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355b18810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 msgr2=0x560355b18810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355b18810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 msgr2=0x560355aa4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 -- 172.21.15.122:0/2707028335 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560355a624b0 con 0x560355a8e600 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd41ed9d640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355a8e9e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd4180119e0 tx=0x7fd418004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2481839bd3c94631 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd3fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:23:03.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/2707028335 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41800fbe0 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/2707028335 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd41800fd80 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.268+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd41800fa30 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/2707028335 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd418030050 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 10 client.5481 fetch_fsmap learned FSMap version 15 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x560355aa9440 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/2707028335 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41800fa30 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/2707028335 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 15) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd418033020 con 0x560355a8e600 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 10 client.5481 fetch_fsmap finished waiting for FSMap version 15 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 10 client.5481 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 20 client.5481 populate_metadata read hostname 'smithi122' 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355a8ef70 0x560355aa5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 4 client.5481 mds_command: new command op to 5262 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 --> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5603559cff40 con 0x560355a8ef70 2024-03-06T07:23:03.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355a8ef70 0x560355aa5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd3fd7fa640 10 client.5481 ms_handle_connect on v2:172.21.15.122:6834/2892378792 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd41e59c640 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355a8ef70 0x560355aa5960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96318c72c55b976 server_cookie=1c9e4c45c6ca3d3e in_seq=0 out_seq=0 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd3fd7fa640 1 -- 172.21.15.122:0/2707028335 <== mds.0 v2:172.21.15.122:6834/2892378792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5603559cff40 con 0x560355a8ef70 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd3fd7fa640 10 client.5481 handle_command_reply: tid=0 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.269+0000 7fd42757b740 1 client.5481 shutdown 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 msgr2=0x560355a8e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355a8e9e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd4180119e0 tx=0x7fd418004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 shutdown_connections 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:6834/2892378792,v1:172.21.15.122:6835/2892378792] conn(0x560355a8ef70 0x560355aa5960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560355a8e600 0x560355a8e9e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560355a88730 0x560355b18810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 --2- 172.21.15.122:0/2707028335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560355a88230 0x560355aa4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd3fd7fa640 0 client.5481 ms_handle_reset on v2:172.21.15.122:6834/2892378792 2024-03-06T07:23:03.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 >> 172.21.15.122:0/2707028335 conn(0x560355a5e870 msgr2=0x560355aa9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 shutdown_connections 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 1 -- 172.21.15.122:0/2707028335 wait complete. 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42757b740 20 client.5481 trim_cache size 0 max 16384 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 5262, 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:23:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:23:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 10.282037047999999, 2024-03-06T07:23:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 15, 2024-03-06T07:23:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 33, 2024-03-06T07:23:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 33, 2024-03-06T07:23:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 17.37199279 2024-03-06T07:23:03.280 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd40006b500 msgr2=0x7fd40006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd40006b500 0x7fd40006d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd410006fd0 tx=0x7fd410008040 comp rx=0 tx=0).stop 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 msgr2=0x7fd420150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd420150150 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd414000c00 tx=0x7fd414004730 comp rx=0 tx=0).stop 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 shutdown_connections 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd40006b500 0x7fd40006d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd42015e000 0x7fd4201e3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4200d0d90 0x7fd420150150 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 --2- 172.21.15.122:0/2396474523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4200c90f0 0x7fd4201e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 >> 172.21.15.122:0/2396474523 conn(0x7fd4200d3cf0 msgr2=0x7fd4200b8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.270+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 shutdown_connections 2024-03-06T07:23:03.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.271+0000 7fd42498c640 1 -- 172.21.15.122:0/2396474523 wait complete. 2024-03-06T07:23:03.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:03.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.381+0000 7f6610dbc640 1 Processor -- start 2024-03-06T07:23:03.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.381+0000 7f6610dbc640 1 -- start start 2024-03-06T07:23:03.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6610dbc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f660c15bdd0 0x7f660c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6610dbc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f660c0d16e0 0x7f660c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6610dbc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f660c0cebe0 con 0x7f660c15fa10 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6610dbc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f660c0ce980 con 0x7f660c15bdd0 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6610dbc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f660c0ceec0 con 0x7f660c0d16e0 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f660c15bdd0 0x7f660c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6609d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f660c15fa10 0x7f660c0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56672/0 (socket says 172.21.15.122:56672) 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f6609d9b640 1 -- 172.21.15.122:0/4043440346 learned_addr learned my addr 172.21.15.122:0/4043440346 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f660c15bdd0 0x7f660c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38634/0 (socket says 172.21.15.122:38634) 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660ad9d640 1 --2- 172.21.15.122:0/4043440346 >> v2:172.21.15.169:3300/0 conn(0x7f660c0d16e0 0x7f660c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660a59c640 1 -- 172.21.15.122:0/4043440346 >> v2:172.21.15.169:3300/0 conn(0x7f660c0d16e0 msgr2=0x7f660c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660a59c640 1 --2- 172.21.15.122:0/4043440346 >> v2:172.21.15.169:3300/0 conn(0x7f660c0d16e0 0x7f660c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660a59c640 1 -- 172.21.15.122:0/4043440346 >> v1:172.21.15.122:6789/0 conn(0x7f660c15fa10 legacy=0x7f660c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660959a640 1 -- 172.21.15.122:0/4043440346 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023099112 0 0) 0x7f660c0cebe0 con 0x7f660c15fa10 2024-03-06T07:23:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.382+0000 7f660a59c640 1 -- 172.21.15.122:0/4043440346 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c0cf1a0 con 0x7f660c15bdd0 2024-03-06T07:23:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.383+0000 7f660a59c640 1 --2- 172.21.15.122:0/4043440346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c0c5610 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f660c0c5520 tx=0x7f660002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a966bc068d540c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f660959a640 1 -- 172.21.15.122:0/4043440346 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000315a0 con 0x7f660c15bdd0 2024-03-06T07:23:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f660959a640 1 -- 172.21.15.122:0/4043440346 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6600037070 con 0x7f660c15bdd0 2024-03-06T07:23:03.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f660959a640 1 -- 172.21.15.122:0/4043440346 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600031960 con 0x7f660c15bdd0 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 -- 172.21.15.122:0/4043440346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 msgr2=0x7f660c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 --2- 172.21.15.122:0/4043440346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c0c5610 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f660c0c5520 tx=0x7f660002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 -- 172.21.15.122:0/4043440346 shutdown_connections 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 --2- 172.21.15.122:0/4043440346 >> v2:172.21.15.169:3300/0 conn(0x7f660c0d16e0 0x7f660c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 --2- 172.21.15.122:0/4043440346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c0c5610 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 -- 172.21.15.122:0/4043440346 >> 172.21.15.122:0/4043440346 conn(0x7f660c09a9f0 msgr2=0x7f660c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 -- 172.21.15.122:0/4043440346 shutdown_connections 2024-03-06T07:23:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.384+0000 7f6610dbc640 1 -- 172.21.15.122:0/4043440346 wait complete. 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 Processor -- start 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 -- start start 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f660c0d16e0 0x7f660c0ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f660c15fa10 0x7f660c0b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c0cc5d0 con 0x7f660c15bdd0 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c0c5b50 con 0x7f660c0d16e0 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6610dbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660c15c1b0 con 0x7f660c15fa10 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c15a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38646/0 (socket says 172.21.15.122:38646) 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.386+0000 7f6609d9b640 1 -- 172.21.15.122:0/2370968170 learned_addr learned my addr 172.21.15.122:0/2370968170 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f660a59c640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f660c0d16e0 0x7f660c0ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f660ad9d640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f660c15fa10 0x7f660c0b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6609d9b640 1 -- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f660c15fa10 msgr2=0x7f660c0b66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6609d9b640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f660c15fa10 0x7f660c0b66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6609d9b640 1 -- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f660c0d16e0 msgr2=0x7f660c0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6609d9b640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f660c0d16e0 0x7f660c0ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6609d9b640 1 -- 172.21.15.122:0/2370968170 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c0c9060 con 0x7f660c15bdd0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6609d9b640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c15a950 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00c470 tx=0x7f65fc00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc017070 con 0x7f660c15bdd0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65fc005ce0 con 0x7f660c15bdd0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc0072b0 con 0x7f660c15bdd0 2024-03-06T07:23:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660c0c9340 con 0x7f660c15bdd0 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.387+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660c0c9900 con 0x7f660c15bdd0 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.388+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660c04f330 con 0x7f660c15bdd0 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.389+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65fc01a070 con 0x7f660c15bdd0 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.389+0000 7f65faffd640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65ec06b4d0 0x7f65ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.390+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65fc089410 con 0x7f660c15bdd0 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.390+0000 7f660a59c640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65ec06b4d0 0x7f65ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.390+0000 7f660a59c640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65ec06b4d0 0x7f65ec06d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6600036000 tx=0x7f6600008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.394+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65fc057cd0 con 0x7f660c15bdd0 2024-03-06T07:23:03.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.609+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f660c0c8e40 con 0x7f660c15bdd0 2024-03-06T07:23:03.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.609+0000 7f65faffd640 1 -- 172.21.15.122:0/2370968170 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f65fc05bd70 con 0x7f660c15bdd0 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65ec06b4d0 msgr2=0x7f65ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65ec06b4d0 0x7f65ec06d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6600036000 tx=0x7f6600008030 comp rx=0 tx=0).stop 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 msgr2=0x7f660c15a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c15a950 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00c470 tx=0x7f65fc00c930 comp rx=0 tx=0).stop 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 shutdown_connections 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f65ec06b4d0 0x7f65ec06d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f660c15fa10 0x7f660c0b66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c15bdd0 0x7f660c15a950 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 --2- 172.21.15.122:0/2370968170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f660c0d16e0 0x7f660c0ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 >> 172.21.15.122:0/2370968170 conn(0x7f660c09a9f0 msgr2=0x7f660c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 shutdown_connections 2024-03-06T07:23:03.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:03.612+0000 7f6610dbc640 1 -- 172.21.15.122:0/2370968170 wait complete. 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:03.625 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:03.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.626 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:23:03.641 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:23:03.642 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42657sec preferred_lft 42657sec 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.645 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:03.707 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:23:03.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.708 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:23:03.764 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:23:03.764 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:03.764 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:03.765 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:03.765 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:03.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.765 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:23:03.765 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:23:03.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:03.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:03.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:03.765 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:23:03.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.860 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:23:03.875 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:03.875 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:03.875 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:03.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:23:03.944 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:03.944 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:03.944 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.999 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:03.999 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:04.000 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:04.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:23:04.122 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:04.178 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:23:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:23:04.180 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:04.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.181 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:23:04.235 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:04.235 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:04.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:23:04.308 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:04.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:23:04.361+0000 7f522c43d480 -1 init, newargv = 0x55a068d501b0 newargc=15 2024-03-06T07:23:04.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[52804]: starting ceph client 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:23:04.365 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:04.366 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:23:04.367 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:04.368 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:04.368 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:04.368 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:04.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.368 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:23:04.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[52804]: starting fuse 2024-03-06T07:23:04.384 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:23:04.385 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:04.385 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:04.386 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:04.387 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:23:04.505 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.52804.asok 2024-03-06T07:23:04.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:04.515 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52804.asok 2024-03-06T07:23:04.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52804.asok status 2024-03-06T07:23:04.654 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "52804", 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:04.655 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 5496, 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 5496 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 35856473 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:23:04.656 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 35856473 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.5496 192.168.0.1:0/35856473", 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/35856473", 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 15, 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 33, 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:23:04.657 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:04.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.663 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:23:04.663 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:04.678 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:23:04.678 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:04.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:04.746 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:23:04.746 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:23:04.746 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> subdir = "delete_me" 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> size = 1024 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> file_count = 1000 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(0, file_count): 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> f.write(size * '"'"'x'"'"') 2024-03-06T07:23:04.747 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:23:06.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:06.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:06.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.758+0000 7f53caced640 1 Processor -- start 2024-03-06T07:23:06.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.758+0000 7f53caced640 1 -- start start 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53caced640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53c415dfe0 0x7f53c40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53caced640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40cf830 0x7f53c40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53caced640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f53c40d27f0 0x7f53c40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53caced640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c40d2c50 con 0x7f53c415dfe0 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53caced640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c40b31e0 con 0x7f53c40d27f0 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53caced640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c40b3d60 con 0x7f53c40cf830 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.759+0000 7f53c8a89640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53c415dfe0 0x7f53c40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53c415dfe0 0x7f53c40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43038/0 (socket says 172.21.15.122:43038) 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 -- 172.21.15.122:0/2010981442 learned_addr learned my addr 172.21.15.122:0/2010981442 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c3fff640 1 --2- 172.21.15.122:0/2010981442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40cf830 0x7f53c40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c928a640 1 --2- 172.21.15.122:0/2010981442 >> v2:172.21.15.169:3300/0 conn(0x7f53c40d27f0 0x7f53c40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 -- 172.21.15.122:0/2010981442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40cf830 msgr2=0x7f53c40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 --2- 172.21.15.122:0/2010981442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40cf830 0x7f53c40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 -- 172.21.15.122:0/2010981442 >> v2:172.21.15.169:3300/0 conn(0x7f53c40d27f0 msgr2=0x7f53c40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 --2- 172.21.15.122:0/2010981442 >> v2:172.21.15.169:3300/0 conn(0x7f53c40d27f0 0x7f53c40d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.760+0000 7f53c8a89640 1 -- 172.21.15.122:0/2010981442 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c40b3430 con 0x7f53c415dfe0 2024-03-06T07:23:06.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53c8a89640 1 --2- 172.21.15.122:0/2010981442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c415dfe0 0x7f53c40cf2f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f53c4067ad0 tx=0x7f53b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3abe184441e98d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53c37fe640 1 -- 172.21.15.122:0/2010981442 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8035020 con 0x7f53c415dfe0 2024-03-06T07:23:06.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53c37fe640 1 -- 172.21.15.122:0/2010981442 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b8031870 con 0x7f53c415dfe0 2024-03-06T07:23:06.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53c37fe640 1 -- 172.21.15.122:0/2010981442 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8031c40 con 0x7f53c415dfe0 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 -- 172.21.15.122:0/2010981442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c415dfe0 msgr2=0x7f53c40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 --2- 172.21.15.122:0/2010981442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c415dfe0 0x7f53c40cf2f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f53c4067ad0 tx=0x7f53b802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 -- 172.21.15.122:0/2010981442 shutdown_connections 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 --2- 172.21.15.122:0/2010981442 >> v2:172.21.15.169:3300/0 conn(0x7f53c40d27f0 0x7f53c40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 --2- 172.21.15.122:0/2010981442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40cf830 0x7f53c40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 --2- 172.21.15.122:0/2010981442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c415dfe0 0x7f53c40cf2f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 -- 172.21.15.122:0/2010981442 >> 172.21.15.122:0/2010981442 conn(0x7f53c40d3cd0 msgr2=0x7f53c40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 -- 172.21.15.122:0/2010981442 shutdown_connections 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.761+0000 7f53caced640 1 -- 172.21.15.122:0/2010981442 wait complete. 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.762+0000 7f53caced640 1 Processor -- start 2024-03-06T07:23:06.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.762+0000 7f53caced640 1 -- start start 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53caced640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 0x7f53c41f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53caced640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40d27f0 0x7f53c4160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53caced640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53c415dfe0 0x7f53c415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53caced640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c40d1c70 con 0x7f53c40cf830 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53caced640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c40b39f0 con 0x7f53c415dfe0 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53caced640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c40d1df0 con 0x7f53c40d27f0 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53c8a89640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 0x7f53c41f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53c8a89640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 0x7f53c41f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43042/0 (socket says 172.21.15.122:43042) 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.763+0000 7f53c8a89640 1 -- 172.21.15.122:0/4078998905 learned_addr learned my addr 172.21.15.122:0/4078998905 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c3fff640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40d27f0 0x7f53c4160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c928a640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53c415dfe0 0x7f53c415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c8a89640 1 -- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40d27f0 msgr2=0x7f53c4160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c8a89640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40d27f0 0x7f53c4160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c8a89640 1 -- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53c415dfe0 msgr2=0x7f53c415fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c8a89640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53c415dfe0 0x7f53c415fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c8a89640 1 -- 172.21.15.122:0/4078998905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c40b9740 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c928a640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53c415dfe0 0x7f53c415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c8a89640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 0x7f53c41f4ce0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f53b802b950 tx=0x7f53b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8040070 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b8031e10 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c40b9a20 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8035040 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.764+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c40cceb0 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.765+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c40b5200 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.766+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53b8038020 con 0x7f53c40cf830 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.766+0000 7f53c17fa640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53a806b4d0 0x7f53a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.766+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53b80abf00 con 0x7f53c40cf830 2024-03-06T07:23:06.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.766+0000 7f53c3fff640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53a806b4d0 0x7f53a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.766+0000 7f53c3fff640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53a806b4d0 0x7f53a806d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f53c415a9c0 tx=0x7f53b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.770+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53c40b5200 con 0x7f53c40cf830 2024-03-06T07:23:06.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.978+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53c404f330 con 0x7f53c40cf830 2024-03-06T07:23:06.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.978+0000 7f53c17fa640 1 -- 172.21.15.122:0/4078998905 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f53b807ad80 con 0x7f53c40cf830 2024-03-06T07:23:06.980 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:06.981 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:06.981 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:06.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53a806b4d0 msgr2=0x7f53a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53a806b4d0 0x7f53a806d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f53c415a9c0 tx=0x7f53b0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 msgr2=0x7f53c41f4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 0x7f53c41f4ce0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f53b802b950 tx=0x7f53b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 shutdown_connections 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53a806b4d0 0x7f53a806d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53c415dfe0 0x7f53c415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53c40d27f0 0x7f53c4160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 --2- 172.21.15.122:0/4078998905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40cf830 0x7f53c41f4ce0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 >> 172.21.15.122:0/4078998905 conn(0x7f53c40d3cd0 msgr2=0x7f53c406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 shutdown_connections 2024-03-06T07:23:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:06.980+0000 7f53caced640 1 -- 172.21.15.122:0/4078998905 wait complete. 2024-03-06T07:23:06.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:23:07.363 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:23:07.364 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:07.364 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:23:07.365 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:23:07.365 DEBUG:teuthology.orchestra.run.smithi122:> import stat 2024-03-06T07:23:07.365 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:07.365 DEBUG:teuthology.orchestra.run.smithi122:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:23:07.365 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:23:07.450 INFO:teuthology.orchestra.run.smithi122.stdout:1099511627776 2024-03-06T07:23:07.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:07.458 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.538+0000 7f0f1fe3ff40 1 Processor -- start 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.538+0000 7f0f1fe3ff40 1 -- start start 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1fe3ff40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f868bec0 0x5626f867f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1fe3ff40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f85e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1fe3ff40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 0x5626f85e8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1fe3ff40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f868c2d0 con 0x5626f8692010 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1fe3ff40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f855fb70 con 0x5626f868dcc0 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1fe3ff40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f868fa70 con 0x5626f868bec0 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 0x5626f85e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 0x5626f85e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43052/0 (socket says 172.21.15.122:43052) 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 -- 172.21.15.122:0/3494982258 learned_addr learned my addr 172.21.15.122:0/3494982258 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1c8c5640 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f85e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d0c6640 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f868bec0 0x5626f867f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 -- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f868bec0 msgr2=0x5626f867f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f868bec0 0x5626f867f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 -- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 msgr2=0x5626f85e7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f85e7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1d8c7640 1 -- 172.21.15.122:0/3494982258 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f85b8eb0 con 0x5626f8692010 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.539+0000 7f0f1c8c5640 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f85e7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1d8c7640 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 0x5626f85e8260 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0f1801e140 tx=0x7f0f18051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f59bc255b75706 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f0f7fe640 1 -- 172.21.15.122:0/3494982258 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1805b040 con 0x5626f8692010 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f0f7fe640 1 -- 172.21.15.122:0/3494982258 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f18066070 con 0x5626f8692010 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f0f7fe640 1 -- 172.21.15.122:0/3494982258 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1804ee40 con 0x5626f8692010 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/3494982258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 msgr2=0x5626f85e8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 0x5626f85e8260 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0f1801e140 tx=0x7f0f18051000 comp rx=0 tx=0).stop 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/3494982258 shutdown_connections 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f8692010 0x5626f85e8260 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f85e7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/3494982258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f868bec0 0x5626f867f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/3494982258 >> 172.21.15.122:0/3494982258 conn(0x5626f85e4bd0 msgr2=0x5626f85a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/3494982258 shutdown_connections 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.540+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/3494982258 wait complete. 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 Processor -- start 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 -- start start 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f868bec0 0x5626f871f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f8688850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f8692010 0x5626f8718f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f85e6f80 con 0x5626f868bec0 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f85b9150 con 0x5626f868dcc0 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.541+0000 7f0f1fe3ff40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f85e7220 con 0x5626f8692010 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f868bec0 0x5626f871f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f868bec0 0x5626f871f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43068/0 (socket says 172.21.15.122:43068) 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 -- 172.21.15.122:0/165725968 learned_addr learned my addr 172.21.15.122:0/165725968 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1c8c5640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f8688850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d8c7640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f8692010 0x5626f8718f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 -- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f8692010 msgr2=0x5626f8718f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5626f8692010 0x5626f8718f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 -- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 msgr2=0x5626f8688850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f8688850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 -- 172.21.15.122:0/165725968 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f871e9c0 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1c8c5640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5626f868dcc0 0x5626f8688850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1d0c6640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f868bec0 0x5626f871f230 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0f10001eb0 tx=0x7f0f1000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f0e7fc640 1 -- 172.21.15.122:0/165725968 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000f620 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f0e7fc640 1 -- 172.21.15.122:0/165725968 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f10006070 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5626f867c470 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f0e7fc640 1 -- 172.21.15.122:0/165725968 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000f9e0 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.542+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626f871f770 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.544+0000 7f0f0e7fc640 1 -- 172.21.15.122:0/165725968 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f1000fb80 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.544+0000 7f0f0e7fc640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f00069eb0 0x7f0f0006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.544+0000 7f0f0e7fc640 1 -- 172.21.15.122:0/165725968 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f100890b0 con 0x5626f868bec0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.544+0000 7f0f1c8c5640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f00069eb0 0x7f0f0006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.544+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x5626f85b7be0 0x5626f871cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.544+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x5626f8688d90 con 0x5626f85b7be0 2024-03-06T07:23:07.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1d8c7640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x5626f85b7be0 0x5626f871cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1c8c5640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f00069eb0 0x7f0f0006c370 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5626f8691e00 tx=0x7f0f14005ac0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1d8c7640 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x5626f85b7be0 0x5626f871cf80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1d8c7640 1 -- 172.21.15.122:0/165725968 <== osd.3 v2:172.21.15.122:6824/47863215 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f0f1805e040 con 0x5626f85b7be0 2024-03-06T07:23:07.552 INFO:teuthology.orchestra.run.smithi122.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:23:07.000000+0000, size 0 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x5626f85b7be0 msgr2=0x5626f871cf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x5626f85b7be0 0x5626f871cf80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f00069eb0 msgr2=0x7f0f0006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f00069eb0 0x7f0f0006c370 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5626f8691e00 tx=0x7f0f14005ac0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f868bec0 msgr2=0x5626f871f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 --2- 172.21.15.122:0/165725968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5626f868bec0 0x5626f871f230 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0f10001eb0 tx=0x7f0f1000a5f0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1d8c7640 1 -- 172.21.15.122:0/165725968 reap_dead start 2024-03-06T07:23:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 shutdown_connections 2024-03-06T07:23:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 >> 172.21.15.122:0/165725968 conn(0x5626f85e4bd0 msgr2=0x5626f868d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 shutdown_connections 2024-03-06T07:23:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:07.545+0000 7f0f1fe3ff40 1 -- 172.21.15.122:0/165725968 wait complete. 2024-03-06T07:23:07.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.555 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:23:09.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:09.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.988+0000 7f7c907f7640 1 Processor -- start 2024-03-06T07:23:09.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.988+0000 7f7c907f7640 1 -- start start 2024-03-06T07:23:10.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c907f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 0x7f7c880b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c907f7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7c8806c6d0 0x7f7c8815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c907f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880b5890 con 0x7f7c88066d90 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c907f7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8815e9f0 con 0x7f7c8806c6d0 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c907f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880cda10 con 0x7f7c880b4f70 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c8ed94640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7c88066d90 0x7f7c880ccb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50486/0 (socket says 172.21.15.122:50486) 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c8ed94640 1 -- 172.21.15.122:0/2360608764 learned_addr learned my addr 172.21.15.122:0/2360608764 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c8e593640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 0x7f7c880b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018959215 0 0) 0x7f7c880b5890 con 0x7f7c88066d90 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c8dd92640 1 --2- 172.21.15.122:0/2360608764 >> v2:172.21.15.169:3300/0 conn(0x7f7c8806c6d0 0x7f7c8815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.989+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c74003520 con 0x7f7c88066d90 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3543834461 0 0) 0x7f7c74003520 con 0x7f7c88066d90 2024-03-06T07:23:10.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 msgr2=0x7f7c880b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 0x7f7c880b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 >> v2:172.21.15.169:3300/0 conn(0x7f7c8806c6d0 msgr2=0x7f7c8815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 --2- 172.21.15.122:0/2360608764 >> v2:172.21.15.169:3300/0 conn(0x7f7c8806c6d0 0x7f7c8815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880cd450 con 0x7f7c88066d90 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7c800019f0 con 0x7f7c88066d90 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 >> v1:172.21.15.122:6789/0 conn(0x7f7c88066d90 legacy=0x7f7c880ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c740039b0 0x7f7c74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 0x7f7c74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c74004d40 0x7f7c740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8815e9f0 con 0x7f7c74004d40 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880cda10 con 0x7f7c740039b0 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7400b9c0 con 0x7f7c880b4f70 2024-03-06T07:23:10.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c74004d40 0x7f7c740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8dd92640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 0x7f7c74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8e593640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c740039b0 0x7f7c74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8ed94640 1 -- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 msgr2=0x7f7c74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c880b4f70 0x7f7c74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8ed94640 1 -- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c740039b0 msgr2=0x7f7c74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c740039b0 0x7f7c74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.990+0000 7f7c8ed94640 1 -- 172.21.15.122:0/2360608764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c74003520 con 0x7f7c74004d40 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.991+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c74004d40 0x7f7c740051c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7c800013d0 tx=0x7f7c74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdeb60d2d89eabf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.991+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8000a4e0 con 0x7f7c74004d40 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.991+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80034650 con 0x7f7c74004d40 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.991+0000 7f7c8d591640 1 -- 172.21.15.122:0/2360608764 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80034a40 con 0x7f7c74004d40 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c907f7640 1 -- 172.21.15.122:0/2360608764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c74004d40 msgr2=0x7f7c740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c907f7640 1 --2- 172.21.15.122:0/2360608764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c74004d40 0x7f7c740051c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7c800013d0 tx=0x7f7c74005830 comp rx=0 tx=0).stop 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c8ed94640 1 -- 172.21.15.122:0/2360608764 reap_dead start 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c907f7640 1 -- 172.21.15.122:0/2360608764 shutdown_connections 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c907f7640 1 -- 172.21.15.122:0/2360608764 >> 172.21.15.122:0/2360608764 conn(0x7f7c880724d0 msgr2=0x7f7c880728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c907f7640 1 -- 172.21.15.122:0/2360608764 shutdown_connections 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.992+0000 7f7c907f7640 1 -- 172.21.15.122:0/2360608764 wait complete. 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.993+0000 7f7c907f7640 1 Processor -- start 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.993+0000 7f7c907f7640 1 -- start start 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c74004d40 0x7f7c881466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c88155220 0x7f7c881f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 0x7f7c881592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8815f3c0 con 0x7f7c88158e50 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8815e680 con 0x7f7c74004d40 2024-03-06T07:23:10.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880b5350 con 0x7f7c88155220 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 0x7f7c881592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 0x7f7c881592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43090/0 (socket says 172.21.15.122:43090) 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 -- 172.21.15.122:0/831034531 learned_addr learned my addr 172.21.15.122:0/831034531 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8dd92640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c88155220 0x7f7c881f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8e593640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c74004d40 0x7f7c881466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 -- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c88155220 msgr2=0x7f7c881f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c88155220 0x7f7c881f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 -- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c74004d40 msgr2=0x7f7c881466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c74004d40 0x7f7c881466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 -- 172.21.15.122:0/831034531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88155720 con 0x7f7c88158e50 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8e593640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c74004d40 0x7f7c881466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c8ed94640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 0x7f7c881592e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000a0b0 tx=0x7f7c80001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80039040 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80043070 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80004d80 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c88155a00 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.994+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88155fc0 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.995+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8804f330 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.996+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c80034650 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.996+0000 7f7c7effd640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c6c06b550 0x7f7c6c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.996+0000 7f7c8e593640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c6c06b550 0x7f7c6c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.996+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c800ac150 con 0x7f7c88158e50 2024-03-06T07:23:10.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:09.997+0000 7f7c8e593640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c6c06b550 0x7f7c6c06da10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009a40 tx=0x7f7c78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.001+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c8007b1d0 con 0x7f7c88158e50 2024-03-06T07:23:10.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.216+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c880b4760 con 0x7f7c88158e50 2024-03-06T07:23:10.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.217+0000 7f7c7effd640 1 -- 172.21.15.122:0/831034531 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f7c8007aad0 con 0x7f7c88158e50 2024-03-06T07:23:10.220 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:10.220 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:10.221 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:10.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c6c06b550 msgr2=0x7f7c6c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c6c06b550 0x7f7c6c06da10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009a40 tx=0x7f7c78008040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 msgr2=0x7f7c881592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 0x7f7c881592e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000a0b0 tx=0x7f7c80001360 comp rx=0 tx=0).stop 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 shutdown_connections 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7c6c06b550 0x7f7c6c06da10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88158e50 0x7f7c881592e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c88155220 0x7f7c881f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 --2- 172.21.15.122:0/831034531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c74004d40 0x7f7c881466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 >> 172.21.15.122:0/831034531 conn(0x7f7c880724d0 msgr2=0x7f7c8806bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 shutdown_connections 2024-03-06T07:23:10.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.219+0000 7f7c907f7640 1 -- 172.21.15.122:0/831034531 wait complete. 2024-03-06T07:23:10.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:23:10.502 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:23:10.502 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:10.502 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:10.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.601+0000 7f79db6af640 1 Processor -- start 2024-03-06T07:23:10.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79db6af640 1 -- start start 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79db6af640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f79d415dfe0 0x7f79d40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79db6af640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79d40cf830 0x7f79d40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79db6af640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d40b3430 con 0x7f79d40d27f0 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79db6af640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d40b31e0 con 0x7f79d40cf830 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79db6af640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d40b3d60 con 0x7f79d415dfe0 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.602+0000 7f79d8c4a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79d40cf830 0x7f79d40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79d40cf830 0x7f79d40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43104/0 (socket says 172.21.15.122:43104) 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 -- 172.21.15.122:0/1053779574 learned_addr learned my addr 172.21.15.122:0/1053779574 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:10.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d944b640 1 --2- 172.21.15.122:0/1053779574 >> v2:172.21.15.169:3300/0 conn(0x7f79d415dfe0 0x7f79d40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 -- 172.21.15.122:0/1053779574 >> v2:172.21.15.169:3300/0 conn(0x7f79d415dfe0 msgr2=0x7f79d40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 --2- 172.21.15.122:0/1053779574 >> v2:172.21.15.169:3300/0 conn(0x7f79d415dfe0 0x7f79d40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 -- 172.21.15.122:0/1053779574 >> v1:172.21.15.169:6789/0 conn(0x7f79d40d27f0 legacy=0x7f79d40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 -- 172.21.15.122:0/1053779574 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40b3710 con 0x7f79d40cf830 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d944b640 1 --2- 172.21.15.122:0/1053779574 >> v2:172.21.15.169:3300/0 conn(0x7f79d415dfe0 0x7f79d40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.603+0000 7f79d8c4a640 1 --2- 172.21.15.122:0/1053779574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d40cf830 0x7f79d40d22b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009870 tx=0x7f79c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc6f71debf92f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79d3fff640 1 -- 172.21.15.122:0/1053779574 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8034070 con 0x7f79d40cf830 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79d3fff640 1 -- 172.21.15.122:0/1053779574 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c8030860 con 0x7f79d40cf830 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79d3fff640 1 -- 172.21.15.122:0/1053779574 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8030c30 con 0x7f79d40cf830 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 -- 172.21.15.122:0/1053779574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d40cf830 msgr2=0x7f79d40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 --2- 172.21.15.122:0/1053779574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d40cf830 0x7f79d40d22b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009870 tx=0x7f79c802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 -- 172.21.15.122:0/1053779574 shutdown_connections 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 --2- 172.21.15.122:0/1053779574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d40cf830 0x7f79d40d22b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 --2- 172.21.15.122:0/1053779574 >> v2:172.21.15.169:3300/0 conn(0x7f79d415dfe0 0x7f79d40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 -- 172.21.15.122:0/1053779574 >> 172.21.15.122:0/1053779574 conn(0x7f79d40d3cd0 msgr2=0x7f79d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 -- 172.21.15.122:0/1053779574 shutdown_connections 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.604+0000 7f79db6af640 1 -- 172.21.15.122:0/1053779574 wait complete. 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.605+0000 7f79db6af640 1 Processor -- start 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 -- start start 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d40cf830 0x7f79d40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79d40d27f0 0x7f79d40b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 0x7f79d40b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40d1df0 con 0x7f79d415dfe0 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40d1c70 con 0x7f79d40cf830 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79db6af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d415e3c0 con 0x7f79d40d27f0 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79d9c4c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 0x7f79d40b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79d9c4c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 0x7f79d40b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43106/0 (socket says 172.21.15.122:43106) 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79d9c4c640 1 -- 172.21.15.122:0/527373893 learned_addr learned my addr 172.21.15.122:0/527373893 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79d8c4a640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79d40d27f0 0x7f79d40b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.606+0000 7f79d944b640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d40cf830 0x7f79d40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d9c4c640 1 -- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79d40d27f0 msgr2=0x7f79d40b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d9c4c640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79d40d27f0 0x7f79d40b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d9c4c640 1 -- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d40cf830 msgr2=0x7f79d40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d9c4c640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d40cf830 0x7f79d40cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d9c4c640 1 -- 172.21.15.122:0/527373893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40cc330 con 0x7f79d415dfe0 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d9c4c640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 0x7f79d40b6de0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c400 tx=0x7f79c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4017070 con 0x7f79d415dfe0 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c40052d0 con 0x7f79d415dfe0 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4003810 con 0x7f79d415dfe0 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d40cc5b0 con 0x7f79d415dfe0 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.607+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d40ccb70 con 0x7f79d415dfe0 2024-03-06T07:23:10.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.608+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d404f330 con 0x7f79d415dfe0 2024-03-06T07:23:10.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.609+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79c401a070 con 0x7f79d415dfe0 2024-03-06T07:23:10.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.609+0000 7f79d1ffb640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f79c006b4d0 0x7f79c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.609+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79c40894b0 con 0x7f79d415dfe0 2024-03-06T07:23:10.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.609+0000 7f79d944b640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f79c006b4d0 0x7f79c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.610+0000 7f79d944b640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f79c006b4d0 0x7f79c006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f79cc003b80 tx=0x7f79cc009890 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.614+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79d40cc330 con 0x7f79d415dfe0 2024-03-06T07:23:10.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.978+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79d40b09c0 con 0x7f79d415dfe0 2024-03-06T07:23:10.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.978+0000 7f79d1ffb640 1 -- 172.21.15.122:0/527373893 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f79c4057ce0 con 0x7f79d415dfe0 2024-03-06T07:23:10.981 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:10.981 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:10.981 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:10.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f79c006b4d0 msgr2=0x7f79c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f79c006b4d0 0x7f79c006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f79cc003b80 tx=0x7f79cc009890 comp rx=0 tx=0).stop 2024-03-06T07:23:10.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 msgr2=0x7f79d40b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 0x7f79d40b6de0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c400 tx=0x7f79c400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 shutdown_connections 2024-03-06T07:23:10.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f79c006b4d0 0x7f79c006d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79d415dfe0 0x7f79d40b6de0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79d40d27f0 0x7f79d40b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 --2- 172.21.15.122:0/527373893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d40cf830 0x7f79d40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 >> 172.21.15.122:0/527373893 conn(0x7f79d40d3cd0 msgr2=0x7f79d40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 shutdown_connections 2024-03-06T07:23:10.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:10.981+0000 7f79db6af640 1 -- 172.21.15.122:0/527373893 wait complete. 2024-03-06T07:23:10.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:23:11.149 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:23:11.149 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:23:11.150 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.269+0000 7fa5931a0640 1 Processor -- start 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.269+0000 7fa5931a0640 1 -- start start 2024-03-06T07:23:11.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.270+0000 7fa5931a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c06f060 0x7fa58c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.270+0000 7fa5931a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa58c06b4f0 0x7fa58c0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.270+0000 7fa5931a0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c15e180 con 0x7fa58c0adfb0 2024-03-06T07:23:11.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.270+0000 7fa5931a0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58c0ae420 con 0x7fa58c06b4f0 2024-03-06T07:23:11.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.270+0000 7fa5931a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0ae680 con 0x7fa58c06f060 2024-03-06T07:23:11.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa58c06b4f0 0x7fa58c0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa58c06b4f0 0x7fa58c0ada70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43110/0 (socket says 172.21.15.122:43110) 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 -- 172.21.15.122:0/2173784168 learned_addr learned my addr 172.21.15.122:0/2173784168 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59219e640 1 --2- 172.21.15.122:0/2173784168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c06f060 0x7fa58c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 -- 172.21.15.122:0/2173784168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c06f060 msgr2=0x7fa58c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 --2- 172.21.15.122:0/2173784168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c06f060 0x7fa58c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 -- 172.21.15.122:0/2173784168 >> v1:172.21.15.169:6789/0 conn(0x7fa58c0adfb0 legacy=0x7fa58c071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.271+0000 7fa59199d640 1 -- 172.21.15.122:0/2173784168 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c15e440 con 0x7fa58c06b4f0 2024-03-06T07:23:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa59199d640 1 --2- 172.21.15.122:0/2173784168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06b4f0 0x7fa58c0ada70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009870 tx=0x7fa57c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c40666cff0e5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa59119c640 1 -- 172.21.15.122:0/2173784168 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c035020 con 0x7fa58c06b4f0 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa59119c640 1 -- 172.21.15.122:0/2173784168 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c031640 con 0x7fa58c06b4f0 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa59119c640 1 -- 172.21.15.122:0/2173784168 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c02d040 con 0x7fa58c06b4f0 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 -- 172.21.15.122:0/2173784168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06b4f0 msgr2=0x7fa58c0ada70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 --2- 172.21.15.122:0/2173784168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06b4f0 0x7fa58c0ada70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009870 tx=0x7fa57c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 -- 172.21.15.122:0/2173784168 shutdown_connections 2024-03-06T07:23:11.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 --2- 172.21.15.122:0/2173784168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06b4f0 0x7fa58c0ada70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 --2- 172.21.15.122:0/2173784168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c06f060 0x7fa58c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 -- 172.21.15.122:0/2173784168 >> 172.21.15.122:0/2173784168 conn(0x7fa58c0d0860 msgr2=0x7fa58c0d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 -- 172.21.15.122:0/2173784168 shutdown_connections 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.272+0000 7fa5931a0640 1 -- 172.21.15.122:0/2173784168 wait complete. 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.273+0000 7fa5931a0640 1 Processor -- start 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.273+0000 7fa5931a0640 1 -- start start 2024-03-06T07:23:11.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa5931a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa58c06b4f0 0x7fa58c14c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa5931a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 0x7fa58c0cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa5931a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c0adfb0 0x7fa58c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa5931a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c15ef20 con 0x7fa58c06f060 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa5931a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c15ea00 con 0x7fa58c06b4f0 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa5931a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c15f0a0 con 0x7fa58c0adfb0 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa59199d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 0x7fa58c0cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa59199d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 0x7fa58c0cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43116/0 (socket says 172.21.15.122:43116) 2024-03-06T07:23:11.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa59199d640 1 -- 172.21.15.122:0/285382524 learned_addr learned my addr 172.21.15.122:0/285382524 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:11.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa59219e640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa58c06b4f0 0x7fa58c14c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.274+0000 7fa59299f640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c0adfb0 0x7fa58c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa59199d640 1 -- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c0adfb0 msgr2=0x7fa58c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa59199d640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c0adfb0 0x7fa58c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa59199d640 1 -- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa58c06b4f0 msgr2=0x7fa58c14c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa59199d640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa58c06b4f0 0x7fa58c14c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa59199d640 1 -- 172.21.15.122:0/285382524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0cf370 con 0x7fa58c06f060 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa59199d640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 0x7fa58c0cfb50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa57c000c00 tx=0x7fa57c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c040070 con 0x7fa58c06f060 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c0399f0 con 0x7fa58c06f060 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c0ce2d0 con 0x7fa58c06f060 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c035040 con 0x7fa58c06f060 2024-03-06T07:23:11.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.275+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c0ce890 con 0x7fa58c06f060 2024-03-06T07:23:11.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.276+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c068f80 con 0x7fa58c06f060 2024-03-06T07:23:11.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.277+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa57c04b060 con 0x7fa58c06f060 2024-03-06T07:23:11.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.277+0000 7fa582ffd640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa57006b4d0 0x7fa57006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.277+0000 7fa59219e640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa57006b4d0 0x7fa57006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.277+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa57c0abcb0 con 0x7fa58c06f060 2024-03-06T07:23:11.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.278+0000 7fa59219e640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa57006b4d0 0x7fa57006d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa588006fd0 tx=0x7fa588008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.281+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa57c07a650 con 0x7fa58c06f060 2024-03-06T07:23:11.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.496+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa58c06b0c0 con 0x7fa58c06f060 2024-03-06T07:23:11.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.496+0000 7fa582ffd640 1 -- 172.21.15.122:0/285382524 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fa57c07e6f0 con 0x7fa58c06f060 2024-03-06T07:23:11.498 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:11.499 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:11.499 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa57006b4d0 msgr2=0x7fa57006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa57006b4d0 0x7fa57006d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa588006fd0 tx=0x7fa588008040 comp rx=0 tx=0).stop 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 msgr2=0x7fa58c0cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 0x7fa58c0cfb50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa57c000c00 tx=0x7fa57c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 shutdown_connections 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa57006b4d0 0x7fa57006d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa58c0adfb0 0x7fa58c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa58c06f060 0x7fa58c0cfb50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 --2- 172.21.15.122:0/285382524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa58c06b4f0 0x7fa58c14c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.498+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 >> 172.21.15.122:0/285382524 conn(0x7fa58c0d0860 msgr2=0x7fa58c0d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.499+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 shutdown_connections 2024-03-06T07:23:11.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:11.499+0000 7fa5931a0640 1 -- 172.21.15.122:0/285382524 wait complete. 2024-03-06T07:23:11.511 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:11.628 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":427}} 2024-03-06T07:23:11.629 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 427 } } 2024-03-06T07:23:11.629 DEBUG:tasks.ceph_test_case:wait_until_equal: 427 != 1001, waiting (timeout=600)... 2024-03-06T07:23:16.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:16.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.729+0000 7ff584f93640 1 Processor -- start 2024-03-06T07:23:16.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.729+0000 7ff584f93640 1 -- start start 2024-03-06T07:23:16.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff584f93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff58015dfe0 0x7ff5800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff584f93640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff5800cf830 0x7ff5800d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff584f93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d27f0 0x7ff5800d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff584f93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5800b31e0 con 0x7ff58015dfe0 2024-03-06T07:23:16.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff584f93640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5800b3d60 con 0x7ff5800cf830 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff584f93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800b3430 con 0x7ff5800d27f0 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff58015dfe0 0x7ff5800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff58015dfe0 0x7ff5800cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55442/0 (socket says 172.21.15.122:55442) 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 -- 172.21.15.122:0/1221404360 learned_addr learned my addr 172.21.15.122:0/1221404360 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff575d9b640 1 --2- 172.21.15.122:0/1221404360 >> v2:172.21.15.169:3300/0 conn(0x7ff5800cf830 0x7ff5800d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d27f0 0x7ff5800d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 -- 172.21.15.122:0/1221404360 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d27f0 msgr2=0x7ff5800d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d27f0 0x7ff5800d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 -- 172.21.15.122:0/1221404360 >> v2:172.21.15.169:3300/0 conn(0x7ff5800cf830 msgr2=0x7ff5800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 --2- 172.21.15.122:0/1221404360 >> v2:172.21.15.169:3300/0 conn(0x7ff5800cf830 0x7ff5800d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57e59c640 1 -- 172.21.15.122:0/1221404360 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800b3710 con 0x7ff58015dfe0 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff575d9b640 1 --2- 172.21.15.122:0/1221404360 >> v2:172.21.15.169:3300/0 conn(0x7ff5800cf830 0x7ff5800d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.730+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d27f0 0x7ff5800d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.731+0000 7ff57e59c640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cf2f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff5800d16b0 tx=0x7ff57002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd20381028047c0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff57dd9b640 1 -- 172.21.15.122:0/1221404360 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570035020 con 0x7ff58015dfe0 2024-03-06T07:23:16.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff57dd9b640 1 -- 172.21.15.122:0/1221404360 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff570031870 con 0x7ff58015dfe0 2024-03-06T07:23:16.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff57dd9b640 1 -- 172.21.15.122:0/1221404360 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570031c40 con 0x7ff58015dfe0 2024-03-06T07:23:16.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 -- 172.21.15.122:0/1221404360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 msgr2=0x7ff5800cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cf2f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff5800d16b0 tx=0x7ff57002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:16.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 -- 172.21.15.122:0/1221404360 shutdown_connections 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d27f0 0x7ff5800d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 --2- 172.21.15.122:0/1221404360 >> v2:172.21.15.169:3300/0 conn(0x7ff5800cf830 0x7ff5800d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 --2- 172.21.15.122:0/1221404360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cf2f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 -- 172.21.15.122:0/1221404360 >> 172.21.15.122:0/1221404360 conn(0x7ff5800d3cd0 msgr2=0x7ff5800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 -- 172.21.15.122:0/1221404360 shutdown_connections 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.732+0000 7ff584f93640 1 -- 172.21.15.122:0/1221404360 wait complete. 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 Processor -- start 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 -- start start 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800cf830 0x7ff58015bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d27f0 0x7ff5800cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800d1df0 con 0x7ff58015dfe0 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800d1c70 con 0x7ff5800d27f0 2024-03-06T07:23:16.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff584f93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015e3c0 con 0x7ff5800cf830 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff57ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff57ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55458/0 (socket says 172.21.15.122:55458) 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff57ed9d640 1 -- 172.21.15.122:0/4244592523 learned_addr learned my addr 172.21.15.122:0/4244592523 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff57e59c640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800cf830 0x7ff58015bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.734+0000 7ff575d9b640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d27f0 0x7ff5800cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff57ed9d640 1 -- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800cf830 msgr2=0x7ff58015bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800cf830 0x7ff58015bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff57ed9d640 1 -- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d27f0 msgr2=0x7ff5800cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d27f0 0x7ff5800cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff57ed9d640 1 -- 172.21.15.122:0/4244592523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800cbb80 con 0x7ff58015dfe0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cd8f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff57800cf20 tx=0x7ff57800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57801a070 con 0x7ff58015dfe0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57800d840 con 0x7ff58015dfe0 2024-03-06T07:23:16.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57800dc30 con 0x7ff58015dfe0 2024-03-06T07:23:16.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5800cbe60 con 0x7ff58015dfe0 2024-03-06T07:23:16.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.735+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5800cc420 con 0x7ff58015dfe0 2024-03-06T07:23:16.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.736+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58004f330 con 0x7ff58015dfe0 2024-03-06T07:23:16.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.736+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff57801d070 con 0x7ff58015dfe0 2024-03-06T07:23:16.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.737+0000 7ff5777fe640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.737+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff57808a980 con 0x7ff58015dfe0 2024-03-06T07:23:16.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.737+0000 7ff57e59c640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.737+0000 7ff57e59c640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff570036000 tx=0x7ff57003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.741+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff578059240 con 0x7ff58015dfe0 2024-03-06T07:23:16.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.951+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5800cc790 con 0x7ff58015dfe0 2024-03-06T07:23:16.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.952+0000 7ff5777fe640 1 -- 172.21.15.122:0/4244592523 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7ff57805d2e0 con 0x7ff58015dfe0 2024-03-06T07:23:16.954 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:16.954 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:16.954 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 msgr2=0x7ff56006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff570036000 tx=0x7ff57003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 msgr2=0x7ff5800cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cd8f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff57800cf20 tx=0x7ff57800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 shutdown_connections 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015dfe0 0x7ff5800cd8f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d27f0 0x7ff5800cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 --2- 172.21.15.122:0/4244592523 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800cf830 0x7ff58015bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 >> 172.21.15.122:0/4244592523 conn(0x7ff5800d3cd0 msgr2=0x7ff58006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 shutdown_connections 2024-03-06T07:23:16.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:16.954+0000 7ff584f93640 1 -- 172.21.15.122:0/4244592523 wait complete. 2024-03-06T07:23:16.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:17.105 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:23:17.105 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:23:17.105 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:17.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.220+0000 7fd7035e9640 1 Processor -- start 2024-03-06T07:23:17.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.220+0000 7fd7035e9640 1 -- start start 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd7035e9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd6fc15fa10 0x7fd6fc0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd7035e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6fc0d16e0 0x7fd6fc0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd7035e9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0cebe0 con 0x7fd6fc15bdd0 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd7035e9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0ceec0 con 0x7fd6fc0d16e0 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd7035e9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0cf1a0 con 0x7fd6fc15fa10 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6fc0d16e0 0x7fd6fc0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6fc0d16e0 0x7fd6fc0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55462/0 (socket says 172.21.15.122:55462) 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 -- 172.21.15.122:0/1341781907 learned_addr learned my addr 172.21.15.122:0/1341781907 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 -- 172.21.15.122:0/1341781907 >> v2:172.21.15.169:3300/0 conn(0x7fd6fc15fa10 msgr2=0x7fd6fc0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 --2- 172.21.15.122:0/1341781907 >> v2:172.21.15.169:3300/0 conn(0x7fd6fc15fa10 0x7fd6fc0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 -- 172.21.15.122:0/1341781907 >> v1:172.21.15.169:6789/0 conn(0x7fd6fc15bdd0 legacy=0x7fd6fc0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.221+0000 7fd701b86640 1 -- 172.21.15.122:0/1341781907 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0cc8e0 con 0x7fd6fc0d16e0 2024-03-06T07:23:17.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd701b86640 1 --2- 172.21.15.122:0/1341781907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0d1af0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0c5520 tx=0x7fd6f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8520100363787df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd6f3fff640 1 -- 172.21.15.122:0/1341781907 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f402c9e0 con 0x7fd6fc0d16e0 2024-03-06T07:23:17.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd6f3fff640 1 -- 172.21.15.122:0/1341781907 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f402cb80 con 0x7fd6fc0d16e0 2024-03-06T07:23:17.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd6f3fff640 1 -- 172.21.15.122:0/1341781907 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40343f0 con 0x7fd6fc0d16e0 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd7035e9640 1 -- 172.21.15.122:0/1341781907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc0d16e0 msgr2=0x7fd6fc0d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1341781907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0d1af0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0c5520 tx=0x7fd6f402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd7035e9640 1 -- 172.21.15.122:0/1341781907 shutdown_connections 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1341781907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0d1af0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1341781907 >> v2:172.21.15.169:3300/0 conn(0x7fd6fc15fa10 0x7fd6fc0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.222+0000 7fd7035e9640 1 -- 172.21.15.122:0/1341781907 >> 172.21.15.122:0/1341781907 conn(0x7fd6fc09a9f0 msgr2=0x7fd6fc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.223+0000 7fd7035e9640 1 -- 172.21.15.122:0/1341781907 shutdown_connections 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.223+0000 7fd7035e9640 1 -- 172.21.15.122:0/1341781907 wait complete. 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 Processor -- start 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 -- start start 2024-03-06T07:23:17.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6fc15bdd0 0x7fd6fc0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 0x7fd6fc146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0d2030 con 0x7fd6fc15fa10 2024-03-06T07:23:17.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc15c1b0 con 0x7fd6fc0d16e0 2024-03-06T07:23:17.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.224+0000 7fd7035e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0ce7b0 con 0x7fd6fc15bdd0 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 0x7fd6fc146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 0x7fd6fc146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55470/0 (socket says 172.21.15.122:55470) 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 -- 172.21.15.122:0/1236363466 learned_addr learned my addr 172.21.15.122:0/1236363466 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701385640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd700b84640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6fc15bdd0 0x7fd6fc0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 -- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6fc15bdd0 msgr2=0x7fd6fc0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6fc15bdd0 0x7fd6fc0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 -- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6fc0d16e0 msgr2=0x7fd6fc0cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 -- 172.21.15.122:0/1236363466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc15b050 con 0x7fd6fc15fa10 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701385640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd701b86640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 0x7fd6fc146a10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4036040 tx=0x7fd6f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4043020 con 0x7fd6fc15fa10 2024-03-06T07:23:17.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f403e450 con 0x7fd6fc15fa10 2024-03-06T07:23:17.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc0ca930 con 0x7fd6fc15fa10 2024-03-06T07:23:17.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f403e820 con 0x7fd6fc15fa10 2024-03-06T07:23:17.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.225+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc0caef0 con 0x7fd6fc15fa10 2024-03-06T07:23:17.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.226+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc04f330 con 0x7fd6fc15fa10 2024-03-06T07:23:17.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.227+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6f4046020 con 0x7fd6fc15fa10 2024-03-06T07:23:17.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.227+0000 7fd6f1ffb640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.227+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6f40acc30 con 0x7fd6fc15fa10 2024-03-06T07:23:17.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.227+0000 7fd701385640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.228+0000 7fd701385640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e006b4d0 0x7fd6e006d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8006fd0 tx=0x7fd6f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.236+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6f404d030 con 0x7fd6fc15fa10 2024-03-06T07:23:17.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.444+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6fc0b7a40 con 0x7fd6fc15fa10 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.444+0000 7fd6f1ffb640 1 -- 172.21.15.122:0/1236363466 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fd6fc15b050 con 0x7fd6fc15fa10 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e006b4d0 msgr2=0x7fd6e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e006b4d0 0x7fd6e006d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8006fd0 tx=0x7fd6f8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 msgr2=0x7fd6fc146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 0x7fd6fc146a10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4036040 tx=0x7fd6f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 shutdown_connections 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6fc15fa10 0x7fd6fc146a10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6fc15bdd0 0x7fd6fc0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 --2- 172.21.15.122:0/1236363466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6fc0d16e0 0x7fd6fc0cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 >> 172.21.15.122:0/1236363466 conn(0x7fd6fc09a9f0 msgr2=0x7fd6fc09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 shutdown_connections 2024-03-06T07:23:17.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.447+0000 7fd7035e9640 1 -- 172.21.15.122:0/1236363466 wait complete. 2024-03-06T07:23:17.459 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:23:17.575 INFO:teuthology.orchestra.run.smithi122.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:23:17.575 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:23:17.576 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.614+0000 7efda1829f40 1 Processor -- start 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.614+0000 7efda1829f40 1 -- start start 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efda1829f40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a854d2bc0 0x555a855713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efda1829f40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a8557b880 0x555a854d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efda1829f40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 0x555a854d1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efda1829f40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8557bc60 con 0x555a8557d520 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efda1829f40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85449b70 con 0x555a8557b880 2024-03-06T07:23:17.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efda1829f40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a8555d9f0 con 0x555a854d2bc0 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 0x555a854d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 0x555a854d1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55496/0 (socket says 172.21.15.122:55496) 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 -- 172.21.15.122:0/1749703803 learned_addr learned my addr 172.21.15.122:0/1749703803 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9dc1e640 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a8557b880 0x555a854d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9e41f640 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a854d2bc0 0x555a855713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 -- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a854d2bc0 msgr2=0x555a855713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a854d2bc0 0x555a855713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 -- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a8557b880 msgr2=0x555a854d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a8557b880 0x555a854d2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 -- 172.21.15.122:0/1749703803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a855740e0 con 0x555a8557d520 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9e41f640 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a854d2bc0 0x555a855713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.615+0000 7efd9ec20640 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 0x555a854d1240 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efd98057040 tx=0x7efd9804f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97642d10062fe6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efd9cc1c640 1 -- 172.21.15.122:0/1749703803 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98066070 con 0x555a8557d520 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efd9cc1c640 1 -- 172.21.15.122:0/1749703803 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9804dc90 con 0x555a8557d520 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efd9cc1c640 1 -- 172.21.15.122:0/1749703803 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9804d740 con 0x555a8557d520 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 -- 172.21.15.122:0/1749703803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 msgr2=0x555a854d1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 0x555a854d1240 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efd98057040 tx=0x7efd9804f000 comp rx=0 tx=0).stop 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 -- 172.21.15.122:0/1749703803 shutdown_connections 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557d520 0x555a854d1240 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a8557b880 0x555a854d2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 --2- 172.21.15.122:0/1749703803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a854d2bc0 0x555a855713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 -- 172.21.15.122:0/1749703803 >> 172.21.15.122:0/1749703803 conn(0x555a854a18b0 msgr2=0x555a854a1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 -- 172.21.15.122:0/1749703803 shutdown_connections 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.616+0000 7efda1829f40 1 -- 172.21.15.122:0/1749703803 wait complete. 2024-03-06T07:23:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 Processor -- start 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 -- start start 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a854d2bc0 0x555a854d1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557b880 0x555a854d1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a8557d520 0x555a854eab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a854a6590 con 0x555a8557b880 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85574380 con 0x555a854d2bc0 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efda1829f40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a854cfe80 con 0x555a8557d520 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.617+0000 7efd9dc1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557b880 0x555a854d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557b880 0x555a854d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55504/0 (socket says 172.21.15.122:55504) 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 -- 172.21.15.122:0/1025659908 learned_addr learned my addr 172.21.15.122:0/1025659908 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:17.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9e41f640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a854d2bc0 0x555a854d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9ec20640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a8557d520 0x555a854eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 -- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a8557d520 msgr2=0x555a854eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x555a8557d520 0x555a854eab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 -- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a854d2bc0 msgr2=0x555a854d1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a854d2bc0 0x555a854d1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 -- 172.21.15.122:0/1025659908 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a8555b450 con 0x555a8557b880 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9e41f640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555a854d2bc0 0x555a854d1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd9dc1e640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557b880 0x555a854d1780 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efd94007ed0 tx=0x7efd94006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd8f7fe640 1 -- 172.21.15.122:0/1025659908 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94011700 con 0x555a8557b880 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd8f7fe640 1 -- 172.21.15.122:0/1025659908 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd940118a0 con 0x555a8557b880 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efd8f7fe640 1 -- 172.21.15.122:0/1025659908 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94011c70 con 0x555a8557b880 2024-03-06T07:23:17.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555a8555c300 con 0x555a8557b880 2024-03-06T07:23:17.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.618+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a85572470 con 0x555a8557b880 2024-03-06T07:23:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:17.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efd8f7fe640 1 -- 172.21.15.122:0/1025659908 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd940133f0 con 0x555a8557b880 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efd8f7fe640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd80069eb0 0x7efd8006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efd8f7fe640 1 -- 172.21.15.122:0/1025659908 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd940885c0 con 0x555a8557b880 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efd9e41f640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd80069eb0 0x7efd8006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efda1829f40 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x555a85570a90 0x555a854eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x555a854ed5a0 con 0x555a85570a90 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efd9ec20640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x555a85570a90 0x555a854eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.620+0000 7efd9e41f640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd80069eb0 0x7efd8006c370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efd90000c00 tx=0x7efd90001500 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efd9ec20640 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x555a85570a90 0x555a854eb5e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efd9ec20640 1 -- 172.21.15.122:0/1025659908 <== osd.3 v2:172.21.15.122:6824/47863215 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7efd9806a040 con 0x555a85570a90 2024-03-06T07:23:17.629 INFO:teuthology.orchestra.run.smithi122.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x555a85570a90 msgr2=0x555a854eb5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x555a85570a90 0x555a854eb5e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd80069eb0 msgr2=0x7efd8006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd80069eb0 0x7efd8006c370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efd90000c00 tx=0x7efd90001500 comp rx=0 tx=0).stop 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557b880 msgr2=0x555a854d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 --2- 172.21.15.122:0/1025659908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555a8557b880 0x555a854d1780 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efd94007ed0 tx=0x7efd94006750 comp rx=0 tx=0).stop 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efd9ec20640 1 -- 172.21.15.122:0/1025659908 reap_dead start 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 shutdown_connections 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 >> 172.21.15.122:0/1025659908 conn(0x555a854a18b0 msgr2=0x555a854edc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 shutdown_connections 2024-03-06T07:23:17.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.621+0000 7efda1829f40 1 -- 172.21.15.122:0/1025659908 wait complete. 2024-03-06T07:23:17.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:23:17.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.767+0000 7f1e722d6640 1 Processor -- start 2024-03-06T07:23:17.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.767+0000 7f1e722d6640 1 -- start start 2024-03-06T07:23:17.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.767+0000 7f1e722d6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e6c162400 0x7f1e6c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.767+0000 7f1e722d6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c0d22c0 con 0x7f1e6c162970 2024-03-06T07:23:17.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.767+0000 7f1e722d6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c0d25a0 con 0x7f1e6c0b4460 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.767+0000 7f1e722d6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c161b00 con 0x7f1e6c162400 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1e6c162970 0x7f1e6c161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46380/0 (socket says 172.21.15.122:46380) 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/757107506 learned_addr learned my addr 172.21.15.122:0/757107506 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e70873640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e6c162400 0x7f1e6c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384843535 0 0) 0x7f1e6c0d22c0 con 0x7f1e6c162970 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e54003520 con 0x7f1e6c162970 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648856880 0 0) 0x7f1e6c0d25a0 con 0x7f1e6c0b4460 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e6c0d22c0 con 0x7f1e6c0b4460 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235919653 0 0) 0x7f1e54003520 con 0x7f1e6c162970 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e6c162400 msgr2=0x7f1e6c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e6c162400 0x7f1e6c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 >> v1:172.21.15.169:6789/0 conn(0x7f1e6c0b4460 legacy=0x7f1e6c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0d2880 con 0x7f1e6c162970 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f1e60001670 con 0x7f1e6c162970 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 >> v1:172.21.15.122:6789/0 conn(0x7f1e6c162970 legacy=0x7f1e6c161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54003b70 0x7f1e54003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e54004800 0x7f1e54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e540051c0 0x7f1e5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c161b00 con 0x7f1e54004800 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5400bfb0 con 0x7f1e540051c0 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5400c1b0 con 0x7f1e54003b70 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e70873640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e6c162400 0x7f1e6c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e54004800 0x7f1e54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e6bfff640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54003b70 0x7f1e54003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.768+0000 7f1e70873640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e540051c0 0x7f1e5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54003b70 msgr2=0x7f1e54003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54003b70 0x7f1e54003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e540051c0 msgr2=0x7f1e5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e540051c0 0x7f1e5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/757107506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e54003520 con 0x7f1e54004800 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e54004800 0x7f1e54004c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1e60007fd0 tx=0x7f1e54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704a5587ff430cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60007ce0 con 0x7f1e54004800 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60030070 con 0x7f1e54004800 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.769+0000 7f1e6affd640 1 -- 172.21.15.122:0/757107506 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60005d90 con 0x7f1e54004800 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e722d6640 1 -- 172.21.15.122:0/757107506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e54004800 msgr2=0x7f1e54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e722d6640 1 --2- 172.21.15.122:0/757107506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e54004800 0x7f1e54004c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1e60007fd0 tx=0x7f1e54005830 comp rx=0 tx=0).stop 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e70873640 1 -- 172.21.15.122:0/757107506 reap_dead start 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e722d6640 1 -- 172.21.15.122:0/757107506 shutdown_connections 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e722d6640 1 -- 172.21.15.122:0/757107506 >> 172.21.15.122:0/757107506 conn(0x7f1e6c0d3ce0 msgr2=0x7f1e6c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e722d6640 1 -- 172.21.15.122:0/757107506 shutdown_connections 2024-03-06T07:23:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.770+0000 7f1e722d6640 1 -- 172.21.15.122:0/757107506 wait complete. 2024-03-06T07:23:17.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.771+0000 7f1e722d6640 1 Processor -- start 2024-03-06T07:23:17.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.771+0000 7f1e722d6640 1 -- start start 2024-03-06T07:23:17.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e722d6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54004800 0x7f1e6c0ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e722d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 0x7f1e6c0b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e722d6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e6c15c2a0 0x7f1e6c151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e722d6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c0ce950 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e722d6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c0cecc0 con 0x7f1e6c15c2a0 2024-03-06T07:23:17.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e722d6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c0b4840 con 0x7f1e54004800 2024-03-06T07:23:17.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 0x7f1e6c0b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 0x7f1e6c0b1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55522/0 (socket says 172.21.15.122:55522) 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/3116760767 learned_addr learned my addr 172.21.15.122:0/3116760767 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6bfff640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54004800 0x7f1e6c0ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e70873640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e6c15c2a0 0x7f1e6c151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54004800 msgr2=0x7f1e6c0ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54004800 0x7f1e6c0ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e6c15c2a0 msgr2=0x7f1e6c151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e6c15c2a0 0x7f1e6c151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6b7fe640 1 -- 172.21.15.122:0/3116760767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0b7b70 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e6bfff640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54004800 0x7f1e6c0ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.772+0000 7f1e70873640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e6c15c2a0 0x7f1e6c151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.773+0000 7f1e6b7fe640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 0x7f1e6c0b1bd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1e6002c4c0 tx=0x7f1e60001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.773+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60001c10 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.773+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60004bf0 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.773+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6c15c8d0 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.773+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60007070 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.773+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6c0b80c0 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.774+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6c04f330 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.774+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e60048020 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.774+0000 7f1e68ff9640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e5806b4d0 0x7f1e5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.775+0000 7f1e6bfff640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e5806b4d0 0x7f1e5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.775+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e600ab560 con 0x7f1e6c0cadb0 2024-03-06T07:23:17.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.775+0000 7f1e6bfff640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e5806b4d0 0x7f1e5806d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0054d0 tx=0x7f1e5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:17.783+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e6007a3e0 con 0x7f1e6c0cadb0 2024-03-06T07:23:18.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.029+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f1e6c1617c0 con 0x7f1e6c0cadb0 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.029+0000 7f1e68ff9640 1 -- 172.21.15.122:0/3116760767 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1697 (secure 0 0 0) 0x7f1e60079ce0 con 0x7f1e6c0cadb0 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stdout: "total_avail_bytes": 772829081600, 2024-03-06T07:23:18.031 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_bytes": 265031680, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_bytes": 265031680, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_ratio": 0.00034281943226233125, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "num_osds": 8, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "num_per_pool_osds": 8, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "stats_by_class": { 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "ssd": { 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_avail_bytes": 772829081600, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_bytes": 265031680, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_bytes": 265031680, 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_ratio": 0.00034281943226233125 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "pools": [ 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "name": ".mgr", 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 1, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 590368, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 2, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 1160, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 1187840, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 1.6180566717594047e-06, 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367057010688 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:18.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "name": "cephfs_metadata", 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 7187699, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 25, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 14096, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 14434304, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 1.9661823898786679e-05, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367057010688 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "name": "cephfs_data", 2024-03-06T07:23:18.034 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 5, 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 995328, 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 972, 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 7776, 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 7962624, 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 1.0846459190361202e-05, 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367057010688 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:23:18.035 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e5806b4d0 msgr2=0x7f1e5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e5806b4d0 0x7f1e5806d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0054d0 tx=0x7f1e5c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 msgr2=0x7f1e6c0b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 0x7f1e6c0b1bd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1e6002c4c0 tx=0x7f1e60001a30 comp rx=0 tx=0).stop 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 shutdown_connections 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e5806b4d0 0x7f1e5806d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e6c15c2a0 0x7f1e6c151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e6c0cadb0 0x7f1e6c0b1bd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 --2- 172.21.15.122:0/3116760767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e54004800 0x7f1e6c0ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 >> 172.21.15.122:0/3116760767 conn(0x7f1e6c0d3ce0 msgr2=0x7f1e6c0d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 shutdown_connections 2024-03-06T07:23:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:18.031+0000 7f1e722d6640 1 -- 172.21.15.122:0/3116760767 wait complete. 2024-03-06T07:23:18.044 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:23:23.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:23:23.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.144+0000 7fbb0bd7a640 1 Processor -- start 2024-03-06T07:23:23.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.144+0000 7fbb0bd7a640 1 -- start start 2024-03-06T07:23:23.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0bd7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbb040b4460 0x7fbb040d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0bd7a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb04161cf0 0x7fbb04162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0bd7a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb040d22c0 con 0x7fbb04162970 2024-03-06T07:23:23.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0bd7a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb04161b00 con 0x7fbb04161cf0 2024-03-06T07:23:23.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0bd7a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb040cecc0 con 0x7fbb040b4460 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0a317640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb04161cf0 0x7fbb04162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb09315640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbb04162970 0x7fbb040b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46386/0 (socket says 172.21.15.122:46386) 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb09315640 1 -- 172.21.15.122:0/3260329542 learned_addr learned my addr 172.21.15.122:0/3260329542 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0a317640 1 -- 172.21.15.122:0/3260329542 >> v2:172.21.15.169:3300/0 conn(0x7fbb040b4460 msgr2=0x7fbb040d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0a317640 1 --2- 172.21.15.122:0/3260329542 >> v2:172.21.15.169:3300/0 conn(0x7fbb040b4460 0x7fbb040d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0a317640 1 -- 172.21.15.122:0/3260329542 >> v1:172.21.15.122:6789/0 conn(0x7fbb04162970 legacy=0x7fbb040b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb0a317640 1 -- 172.21.15.122:0/3260329542 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb040d25a0 con 0x7fbb04161cf0 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.145+0000 7fbb08b14640 1 -- 172.21.15.122:0/3260329542 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065026086 0 0) 0x7fbb040d22c0 con 0x7fbb04162970 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.146+0000 7fbb0a317640 1 --2- 172.21.15.122:0/3260329542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04161cf0 0x7fbb04162180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbb0408ddd0 tx=0x7fbafc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd96ee4ea9967190 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.146+0000 7fbb08b14640 1 -- 172.21.15.122:0/3260329542 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc039070 con 0x7fbb04161cf0 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.146+0000 7fbb08b14640 1 -- 172.21.15.122:0/3260329542 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc02cc80 con 0x7fbb04161cf0 2024-03-06T07:23:23.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.146+0000 7fbb08b14640 1 -- 172.21.15.122:0/3260329542 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc034660 con 0x7fbb04161cf0 2024-03-06T07:23:23.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/3260329542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04161cf0 msgr2=0x7fbb04162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/3260329542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04161cf0 0x7fbb04162180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbb0408ddd0 tx=0x7fbafc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/3260329542 shutdown_connections 2024-03-06T07:23:23.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/3260329542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04161cf0 0x7fbb04162180 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/3260329542 >> v2:172.21.15.169:3300/0 conn(0x7fbb040b4460 0x7fbb040d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/3260329542 >> 172.21.15.122:0/3260329542 conn(0x7fbb040d3ce0 msgr2=0x7fbb040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/3260329542 shutdown_connections 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.147+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/3260329542 wait complete. 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.148+0000 7fbb0bd7a640 1 Processor -- start 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.148+0000 7fbb0bd7a640 1 -- start start 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0bd7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb040b4460 0x7fbb04155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0bd7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb04161cf0 0x7fbb040b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0bd7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 0x7fbb040c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0bd7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040ce950 con 0x7fbb04162970 2024-03-06T07:23:23.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0bd7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040d2b60 con 0x7fbb04161cf0 2024-03-06T07:23:23.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0bd7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb040b4840 con 0x7fbb040b4460 2024-03-06T07:23:23.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 0x7fbb040c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 0x7fbb040c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55534/0 (socket says 172.21.15.122:55534) 2024-03-06T07:23:23.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 -- 172.21.15.122:0/1895122208 learned_addr learned my addr 172.21.15.122:0/1895122208 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:23.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb09315640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb04161cf0 0x7fbb040b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb09b16640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb040b4460 0x7fbb04155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 -- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb040b4460 msgr2=0x7fbb04155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb040b4460 0x7fbb04155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 -- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb04161cf0 msgr2=0x7fbb040b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb04161cf0 0x7fbb040b7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 -- 172.21.15.122:0/1895122208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb040b8830 con 0x7fbb04162970 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb09315640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb04161cf0 0x7fbb040b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb09b16640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb040b4460 0x7fbb04155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.149+0000 7fbb0a317640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 0x7fbb040c99e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009e80 tx=0x7fbafc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.150+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc044030 con 0x7fbb04162970 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.150+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc02ca80 con 0x7fbb04162970 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.150+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb040b8b10 con 0x7fbb04162970 2024-03-06T07:23:23.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.150+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc039040 con 0x7fbb04162970 2024-03-06T07:23:23.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.150+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb040b90d0 con 0x7fbb04162970 2024-03-06T07:23:23.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.150+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0404f330 con 0x7fbb04162970 2024-03-06T07:23:23.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:23.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.151+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbafc03f470 con 0x7fbb04162970 2024-03-06T07:23:23.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.151+0000 7fbafa7fc640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbae806b4d0 0x7fbae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.152+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbafc0acc00 con 0x7fbb04162970 2024-03-06T07:23:23.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.152+0000 7fbb09b16640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbae806b4d0 0x7fbae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.152+0000 7fbb09b16640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbae806b4d0 0x7fbae806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009890 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.156+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbafc0b3080 con 0x7fbb04162970 2024-03-06T07:23:23.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:23.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.403+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fbb0406b180 con 0x7fbb04162970 2024-03-06T07:23:23.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.403+0000 7fbafa7fc640 1 -- 172.21.15.122:0/1895122208 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1656 (secure 0 0 0) 0x7fbafc07bb10 con 0x7fbb04162970 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "total_avail_bytes": 772831395840, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_bytes": 262717440, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_bytes": 262717440, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_ratio": 0.0003398259577807039, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "num_osds": 8, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "num_per_pool_osds": 8, 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:23:23.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "stats_by_class": { 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "ssd": { 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "total_avail_bytes": 772831395840, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_bytes": 262717440, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_bytes": 262717440, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_ratio": 0.0003398259577807039 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "pools": [ 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "name": ".mgr", 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 1, 2024-03-06T07:23:23.406 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 590368, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 2, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 1160, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 1187840, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 1.6179740214283811e-06, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367075753984 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "name": "cephfs_metadata", 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 3677172, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 23, 2024-03-06T07:23:23.407 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 7240, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 7413760, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 1.0098304301209282e-05, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367075753984 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "name": "cephfs_data", 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 5, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 0, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 0, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 0, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 0, 2024-03-06T07:23:23.408 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 0, 2024-03-06T07:23:23.409 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367075753984 2024-03-06T07:23:23.409 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:23.409 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:23.409 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:23:23.409 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbae806b4d0 msgr2=0x7fbae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbae806b4d0 0x7fbae806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009890 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 msgr2=0x7fbb040c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 0x7fbb040c99e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009e80 tx=0x7fbafc004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 shutdown_connections 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbae806b4d0 0x7fbae806d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb04162970 0x7fbb040c99e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb04161cf0 0x7fbb040b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 --2- 172.21.15.122:0/1895122208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb040b4460 0x7fbb04155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 >> 172.21.15.122:0/1895122208 conn(0x7fbb040d3ce0 msgr2=0x7fbb040b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 shutdown_connections 2024-03-06T07:23:23.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.405+0000 7fbb0bd7a640 1 -- 172.21.15.122:0/1895122208 wait complete. 2024-03-06T07:23:23.420 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:23:23.421 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:23.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:23.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:23.534 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:23.535 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:23:23.535 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:23.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.689+0000 7f0e57e47640 1 Processor -- start 2024-03-06T07:23:23.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.689+0000 7f0e57e47640 1 -- start start 2024-03-06T07:23:23.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e57e47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 0x7f0e5015c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e57e47640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0e500afdd0 0x7f0e5006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e57e47640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5015e740 con 0x7f0e500d14f0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e57e47640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5006fae0 con 0x7f0e500afdd0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e57e47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015f5f0 con 0x7f0e50158920 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e553e2640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0e500d14f0 0x7f0e500af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46396/0 (socket says 172.21.15.122:46396) 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e553e2640 1 -- 172.21.15.122:0/1446565334 learned_addr learned my addr 172.21.15.122:0/1446565334 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e55be3640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 0x7f0e5015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941667987 0 0) 0x7f0e5015e740 con 0x7f0e500d14f0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e563e4640 1 --2- 172.21.15.122:0/1446565334 >> v2:172.21.15.169:3300/0 conn(0x7f0e500afdd0 0x7f0e5006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e3c003520 con 0x7f0e500d14f0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736365862 0 0) 0x7f0e3c003520 con 0x7f0e500d14f0 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 msgr2=0x7f0e5015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 0x7f0e5015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 >> v2:172.21.15.169:3300/0 conn(0x7f0e500afdd0 msgr2=0x7f0e5006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.690+0000 7f0e54be1640 1 --2- 172.21.15.122:0/1446565334 >> v2:172.21.15.169:3300/0 conn(0x7f0e500afdd0 0x7f0e5006f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5015e9e0 con 0x7f0e500d14f0 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f0e40001670 con 0x7f0e500d14f0 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 >> v1:172.21.15.122:6789/0 conn(0x7f0e500d14f0 legacy=0x7f0e500af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c0039b0 0x7f0e3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3c004800 0x7f0e3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 0x7f0e3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5006fae0 con 0x7f0e3c004800 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015f5f0 con 0x7f0e3c0039b0 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c00b9c0 con 0x7f0e50158920 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e553e2640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3c004800 0x7f0e3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e55be3640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c0039b0 0x7f0e3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e563e4640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 0x7f0e3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e553e2640 1 -- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 msgr2=0x7f0e3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e553e2640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e50158920 0x7f0e3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e553e2640 1 -- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c0039b0 msgr2=0x7f0e3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e553e2640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c0039b0 0x7f0e3c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.691+0000 7f0e553e2640 1 -- 172.21.15.122:0/1446565334 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c003520 con 0x7f0e3c004800 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e553e2640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3c004800 0x7f0e3c004c80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0e40001050 tx=0x7f0e3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84407a4ab939cc3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40007d70 con 0x7f0e3c004800 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e40030070 con 0x7f0e3c004800 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e54be1640 1 -- 172.21.15.122:0/1446565334 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4002cd40 con 0x7f0e3c004800 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e57e47640 1 -- 172.21.15.122:0/1446565334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3c004800 msgr2=0x7f0e3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e57e47640 1 --2- 172.21.15.122:0/1446565334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3c004800 0x7f0e3c004c80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0e40001050 tx=0x7f0e3c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e563e4640 1 -- 172.21.15.122:0/1446565334 reap_dead start 2024-03-06T07:23:23.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e57e47640 1 -- 172.21.15.122:0/1446565334 shutdown_connections 2024-03-06T07:23:23.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.692+0000 7f0e57e47640 1 -- 172.21.15.122:0/1446565334 >> 172.21.15.122:0/1446565334 conn(0x7f0e500d0960 msgr2=0x7f0e500d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.693+0000 7f0e57e47640 1 -- 172.21.15.122:0/1446565334 shutdown_connections 2024-03-06T07:23:23.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.693+0000 7f0e57e47640 1 -- 172.21.15.122:0/1446565334 wait complete. 2024-03-06T07:23:23.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 Processor -- start 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 -- start start 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c004800 0x7f0e500aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 0x7f0e50154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e5006afb0 0x7f0e5006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015f280 con 0x7f0e50145cb0 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015efa0 con 0x7f0e3c004800 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.694+0000 7f0e57e47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015c310 con 0x7f0e5006afb0 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 0x7f0e50154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 0x7f0e50154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55554/0 (socket says 172.21.15.122:55554) 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 -- 172.21.15.122:0/3639472280 learned_addr learned my addr 172.21.15.122:0/3639472280 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e55be3640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c004800 0x7f0e500aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e563e4640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e5006afb0 0x7f0e5006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 -- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e5006afb0 msgr2=0x7f0e5006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e5006afb0 0x7f0e5006b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 -- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c004800 msgr2=0x7f0e500aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c004800 0x7f0e500aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 -- 172.21.15.122:0/3639472280 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e500af2d0 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e563e4640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e5006afb0 0x7f0e5006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e553e2640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 0x7f0e50154900 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000c2d0 tx=0x7f0e40000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40045070 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e40004bf0 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e501435d0 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40007d50 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.695+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e50143b90 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.696+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5004f330 con 0x7f0e50145cb0 2024-03-06T07:23:23.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.697+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e40007070 con 0x7f0e50145cb0 2024-03-06T07:23:23.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.697+0000 7f0e467fc640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e3406b4d0 0x7f0e3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.697+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e400ab0b0 con 0x7f0e50145cb0 2024-03-06T07:23:23.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.697+0000 7f0e55be3640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e3406b4d0 0x7f0e3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.698+0000 7f0e55be3640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e3406b4d0 0x7f0e3406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009a60 tx=0x7f0e4c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.701+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e500af2d0 con 0x7f0e50145cb0 2024-03-06T07:23:23.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.894+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e50072b80 con 0x7f0e50145cb0 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.895+0000 7f0e467fc640 1 -- 172.21.15.122:0/3639472280 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f0e40079a80 con 0x7f0e50145cb0 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:53.997648+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e3406b4d0 msgr2=0x7f0e3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e3406b4d0 0x7f0e3406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009a60 tx=0x7f0e4c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 msgr2=0x7f0e50154900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 0x7f0e50154900 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000c2d0 tx=0x7f0e40000fe0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 shutdown_connections 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e3406b4d0 0x7f0e3406d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e5006afb0 0x7f0e5006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e50145cb0 0x7f0e50154900 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 --2- 172.21.15.122:0/3639472280 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e3c004800 0x7f0e500aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 >> 172.21.15.122:0/3639472280 conn(0x7f0e500d0960 msgr2=0x7f0e5015b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 shutdown_connections 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:23.897+0000 7f0e57e47640 1 -- 172.21.15.122:0/3639472280 wait complete. 2024-03-06T07:23:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:50.561315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:51.306826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:23.910 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:24.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.017+0000 7f4ad9aa4640 1 Processor -- start 2024-03-06T07:23:24.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.017+0000 7f4ad9aa4640 1 -- start start 2024-03-06T07:23:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad9aa4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad415c050 0x7f4ad406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad9aa4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4ad406fb40 0x7f4ad40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad9aa4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40cd4b0 con 0x7f4ad40ccca0 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad9aa4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad406ff80 con 0x7f4ad406fb40 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad9aa4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40cd790 con 0x7f4ad415c050 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4ad406fb40 0x7f4ad40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4ad40ccca0 0x7f4ad40cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46404/0 (socket says 172.21.15.122:46404) 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad3fff640 1 -- 172.21.15.122:0/1206303918 learned_addr learned my addr 172.21.15.122:0/1206303918 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.018+0000 7f4ad37fe640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad415c050 0x7f4ad406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/1206303918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad415c050 msgr2=0x7f4ad406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad2ffd640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad415c050 0x7f4ad406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/1206303918 >> v1:172.21.15.122:6789/0 conn(0x7f4ad40ccca0 legacy=0x7f4ad40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad27fc640 1 -- 172.21.15.122:0/1206303918 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905897838 0 0) 0x7f4ad40cd4b0 con 0x7f4ad40ccca0 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/1206303918 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40cda70 con 0x7f4ad406fb40 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad37fe640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad415c050 0x7f4ad406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad2ffd640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad406fb40 0x7f4ad40cc760 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80097a0 tx=0x7f4ac802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e6b6858419ca74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.019+0000 7f4ad27fc640 1 -- 172.21.15.122:0/1206303918 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80315a0 con 0x7f4ad406fb40 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad27fc640 1 -- 172.21.15.122:0/1206303918 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac8037070 con 0x7f4ad406fb40 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad27fc640 1 -- 172.21.15.122:0/1206303918 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8031960 con 0x7f4ad406fb40 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/1206303918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad406fb40 msgr2=0x7f4ad40cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad406fb40 0x7f4ad40cc760 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80097a0 tx=0x7f4ac802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:24.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/1206303918 shutdown_connections 2024-03-06T07:23:24.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad406fb40 0x7f4ad40cc760 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/1206303918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad415c050 0x7f4ad406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/1206303918 >> 172.21.15.122:0/1206303918 conn(0x7f4ad40b3800 msgr2=0x7f4ad40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/1206303918 shutdown_connections 2024-03-06T07:23:24.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.020+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/1206303918 wait complete. 2024-03-06T07:23:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.021+0000 7f4ad9aa4640 1 Processor -- start 2024-03-06T07:23:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.021+0000 7f4ad9aa4640 1 -- start start 2024-03-06T07:23:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad9aa4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad406fb40 0x7f4ad409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad9aa4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 0x7f4ad41582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad9aa4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ad415c050 0x7f4ad4156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad9aa4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40d01f0 con 0x7f4ad40ccca0 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad9aa4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40cd130 con 0x7f4ad415c050 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad9aa4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40d0370 con 0x7f4ad406fb40 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 0x7f4ad41582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 0x7f4ad41582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55564/0 (socket says 172.21.15.122:55564) 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/197330146 learned_addr learned my addr 172.21.15.122:0/197330146 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad37fe640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad406fb40 0x7f4ad409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad3fff640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ad415c050 0x7f4ad4156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad406fb40 msgr2=0x7f4ad409d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad406fb40 0x7f4ad409d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ad415c050 msgr2=0x7f4ad4156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ad415c050 0x7f4ad4156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.022+0000 7f4ad2ffd640 1 -- 172.21.15.122:0/197330146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4154980 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.023+0000 7f4ad2ffd640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 0x7f4ad41582f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80097a0 tx=0x7f4ac80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.023+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8004230 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.023+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac80043f0 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.023+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8032880 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.023+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad409da90 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.023+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad4155b40 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.024+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad409fe60 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.024+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ac8032a20 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.025+0000 7f4abbfff640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ac006b4d0 0x7f4ac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.025+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ac802f030 con 0x7f4ad40ccca0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.025+0000 7f4ad37fe640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ac006b4d0 0x7f4ac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.025+0000 7f4ad37fe640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ac006b4d0 0x7f4ac006d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400a990 tx=0x7f4ac4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.033+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ac8041070 con 0x7f4ad40ccca0 2024-03-06T07:23:24.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.327+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ad4155e20 con 0x7f4ad40ccca0 2024-03-06T07:23:24.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.328+0000 7f4abbfff640 1 -- 172.21.15.122:0/197330146 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ac8044340 con 0x7f4ad40ccca0 2024-03-06T07:23:24.329 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:23:24.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ac006b4d0 msgr2=0x7f4ac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ac006b4d0 0x7f4ac006d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400a990 tx=0x7f4ac4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:24.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 msgr2=0x7f4ad41582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 0x7f4ad41582f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80097a0 tx=0x7f4ac80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 shutdown_connections 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ac006b4d0 0x7f4ac006d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ad415c050 0x7f4ad4156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad40ccca0 0x7f4ad41582f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 --2- 172.21.15.122:0/197330146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ad406fb40 0x7f4ad409d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 >> 172.21.15.122:0/197330146 conn(0x7f4ad40b3800 msgr2=0x7f4ad406ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.330+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 shutdown_connections 2024-03-06T07:23:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.331+0000 7f4ad9aa4640 1 -- 172.21.15.122:0/197330146 wait complete. 2024-03-06T07:23:24.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:24.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.445+0000 7f25549bf640 1 Processor -- start 2024-03-06T07:23:24.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.445+0000 7f25549bf640 1 -- start start 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f25549bf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f25501613f0 0x7f25500b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f25549bf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 0x7f2550161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f25549bf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255006bf90 con 0x7f25500ad260 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f25549bf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255006c270 con 0x7f25501613f0 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f25549bf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255006c550 con 0x7f2550161b20 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254ed9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f25500ad260 0x7f25500ad6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46406/0 (socket says 172.21.15.122:46406) 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254ed9d640 1 -- 172.21.15.122:0/713165706 learned_addr learned my addr 172.21.15.122:0/713165706 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254e59c640 1 --2- 172.21.15.122:0/713165706 >> v2:172.21.15.169:3300/0 conn(0x7f25501613f0 0x7f25500b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f2547fff640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 0x7f2550161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984681706 0 0) 0x7f255006bf90 con 0x7f25500ad260 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f253c003520 con 0x7f25500ad260 2024-03-06T07:23:24.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360918635 0 0) 0x7f253c003520 con 0x7f25500ad260 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 msgr2=0x7f2550161fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 0x7f2550161fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 >> v2:172.21.15.169:3300/0 conn(0x7f25501613f0 msgr2=0x7f25500b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 --2- 172.21.15.122:0/713165706 >> v2:172.21.15.169:3300/0 conn(0x7f25501613f0 0x7f25500b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255006c830 con 0x7f25500ad260 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f25480019f0 con 0x7f25500ad260 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 >> v1:172.21.15.122:6789/0 conn(0x7f25500ad260 legacy=0x7f25500ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 0x7f253c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c003f30 0x7f253c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f253c004d40 0x7f253c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255006c270 con 0x7f253c003f30 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255006c550 con 0x7f253c004d40 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.446+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c00ba10 con 0x7f2550161b20 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f2547fff640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c003f30 0x7f253c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f254e59c640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 0x7f253c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f254ed9d640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f253c004d40 0x7f253c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f2547fff640 1 -- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 msgr2=0x7f253c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f2547fff640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2550161b20 0x7f253c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f2547fff640 1 -- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f253c004d40 msgr2=0x7f253c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f2547fff640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f253c004d40 0x7f253c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f2547fff640 1 -- 172.21.15.122:0/713165706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c003520 con 0x7f253c003f30 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.447+0000 7f254ed9d640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f253c004d40 0x7f253c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f2547fff640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c003f30 0x7f253c004800 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2538009af0 tx=0x7f253c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4b97ec7099572f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538039070 con 0x7f253c003f30 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253802c6b0 con 0x7f253c003f30 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f254dd9b640 1 -- 172.21.15.122:0/713165706 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538009690 con 0x7f253c003f30 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f25549bf640 1 -- 172.21.15.122:0/713165706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c003f30 msgr2=0x7f253c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f25549bf640 1 --2- 172.21.15.122:0/713165706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c003f30 0x7f253c004800 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2538009af0 tx=0x7f253c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f254ed9d640 1 -- 172.21.15.122:0/713165706 reap_dead start 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f25549bf640 1 -- 172.21.15.122:0/713165706 shutdown_connections 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f25549bf640 1 -- 172.21.15.122:0/713165706 >> 172.21.15.122:0/713165706 conn(0x7f25500b4fe0 msgr2=0x7f25500b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f25549bf640 1 -- 172.21.15.122:0/713165706 shutdown_connections 2024-03-06T07:23:24.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.448+0000 7f25549bf640 1 -- 172.21.15.122:0/713165706 wait complete. 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 Processor -- start 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 -- start start 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f253c003f30 0x7f2550099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 0x7f25501573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25501e8aa0 0x7f25501e8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2550162120 con 0x7f2550156f70 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2550161fa0 con 0x7f25501e8aa0 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f25549bf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25501622a0 con 0x7f253c003f30 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f2547fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 0x7f25501573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f2547fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 0x7f25501573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55586/0 (socket says 172.21.15.122:55586) 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f2547fff640 1 -- 172.21.15.122:0/1035310105 learned_addr learned my addr 172.21.15.122:0/1035310105 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:24.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f254e59c640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f253c003f30 0x7f2550099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.450+0000 7f254ed9d640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25501e8aa0 0x7f25501e8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2547fff640 1 -- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f253c003f30 msgr2=0x7f2550099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2547fff640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f253c003f30 0x7f2550099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2547fff640 1 -- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25501e8aa0 msgr2=0x7f25501e8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2547fff640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25501e8aa0 0x7f25501e8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2547fff640 1 -- 172.21.15.122:0/1035310105 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2550153350 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2547fff640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 0x7f25501573f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2538009af0 tx=0x7f2538004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538042020 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253803a040 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2550153630 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538039070 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.451+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2550153bf0 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.452+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2550153fd0 con 0x7f2550156f70 2024-03-06T07:23:24.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.452+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2538045020 con 0x7f2550156f70 2024-03-06T07:23:24.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.453+0000 7f2546ffd640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f253006b4d0 0x7f253006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.453+0000 7f254e59c640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f253006b4d0 0x7f253006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.453+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25380ab760 con 0x7f2550156f70 2024-03-06T07:23:24.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.453+0000 7f254e59c640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f253006b4d0 0x7f253006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2540009c40 tx=0x7f2540009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.457+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2550153fd0 con 0x7f2550156f70 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.635+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f255009e4e0 con 0x7f2550156f70 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.635+0000 7f2546ffd640 1 -- 172.21.15.122:0/1035310105 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f253804c030 con 0x7f2550156f70 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f253006b4d0 msgr2=0x7f253006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f253006b4d0 0x7f253006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2540009c40 tx=0x7f2540009190 comp rx=0 tx=0).stop 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 msgr2=0x7f25501573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 0x7f25501573f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2538009af0 tx=0x7f2538004730 comp rx=0 tx=0).stop 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 shutdown_connections 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f253006b4d0 0x7f253006d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25501e8aa0 0x7f25501e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2550156f70 0x7f25501573f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 --2- 172.21.15.122:0/1035310105 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f253c003f30 0x7f2550099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 >> 172.21.15.122:0/1035310105 conn(0x7f25500b4fe0 msgr2=0x7f25500ad6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 shutdown_connections 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.638+0000 7f25549bf640 1 -- 172.21.15.122:0/1035310105 wait complete. 2024-03-06T07:23:24.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:24.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.753+0000 7f0b18da1640 1 Processor -- start 2024-03-06T07:23:24.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.753+0000 7f0b18da1640 1 -- start start 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b18da1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b18da1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0b140b0e30 0x7f0b140b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b18da1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b141618b0 con 0x7f0b141614d0 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b18da1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b140d4cc0 con 0x7f0b140b0e30 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b18da1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140cf400 con 0x7f0b140d4860 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b12d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0b140b0e30 0x7f0b140b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.754+0000 7f0b12d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0b140b0e30 0x7f0b140b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57230/0 (socket says 172.21.15.122:57230) 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b12d9d640 1 -- 172.21.15.122:0/292495741 learned_addr learned my addr 172.21.15.122:0/292495741 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b11d9b640 1 --2- 172.21.15.122:0/292495741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b12d9d640 1 -- 172.21.15.122:0/292495741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 msgr2=0x7f0b140d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b12d9d640 1 --2- 172.21.15.122:0/292495741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b12d9d640 1 -- 172.21.15.122:0/292495741 >> v1:172.21.15.169:6789/0 conn(0x7f0b141614d0 legacy=0x7f0b140d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b12d9d640 1 -- 172.21.15.122:0/292495741 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b140cf6a0 con 0x7f0b140b0e30 2024-03-06T07:23:24.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.755+0000 7f0b12d9d640 1 --2- 172.21.15.122:0/292495741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b140b12c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009e50 tx=0x7f0b0c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b2190a1e4cb85d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b1159a640 1 -- 172.21.15.122:0/292495741 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c02cb00 con 0x7f0b140b0e30 2024-03-06T07:23:24.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b1159a640 1 -- 172.21.15.122:0/292495741 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c02cca0 con 0x7f0b140b0e30 2024-03-06T07:23:24.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b1159a640 1 -- 172.21.15.122:0/292495741 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c034610 con 0x7f0b140b0e30 2024-03-06T07:23:24.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 -- 172.21.15.122:0/292495741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 msgr2=0x7f0b140b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 --2- 172.21.15.122:0/292495741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b140b12c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009e50 tx=0x7f0b0c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:24.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 -- 172.21.15.122:0/292495741 shutdown_connections 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 --2- 172.21.15.122:0/292495741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b140b12c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 --2- 172.21.15.122:0/292495741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 -- 172.21.15.122:0/292495741 >> 172.21.15.122:0/292495741 conn(0x7f0b140d3d40 msgr2=0x7f0b140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 -- 172.21.15.122:0/292495741 shutdown_connections 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.756+0000 7f0b18da1640 1 -- 172.21.15.122:0/292495741 wait complete. 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 Processor -- start 2024-03-06T07:23:24.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 -- start start 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b1415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b141614d0 0x7f0b140cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140b0910 con 0x7f0b140b0e30 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140cfc60 con 0x7f0b141614d0 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b18da1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140d2560 con 0x7f0b140d4860 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b1259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b1415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b1259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b1415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57238/0 (socket says 172.21.15.122:57238) 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b1259c640 1 -- 172.21.15.122:0/135134208 learned_addr learned my addr 172.21.15.122:0/135134208 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b11d9b640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.758+0000 7f0b12d9d640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b141614d0 0x7f0b140cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b1259c640 1 -- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 msgr2=0x7f0b140c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b1259c640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b1259c640 1 -- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b141614d0 msgr2=0x7f0b140cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b1259c640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b141614d0 0x7f0b140cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b1259c640 1 -- 172.21.15.122:0/135134208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1415f380 con 0x7f0b140b0e30 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b1259c640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b1415ee40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c3c0 tx=0x7f0b0400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b040052d0 con 0x7f0b140b0e30 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b04005490 con 0x7f0b140b0e30 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b140c7910 con 0x7f0b140b0e30 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04007260 con 0x7f0b140b0e30 2024-03-06T07:23:24.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.759+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b140c7ed0 con 0x7f0b140b0e30 2024-03-06T07:23:24.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.760+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b140b4d50 con 0x7f0b140b0e30 2024-03-06T07:23:24.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.760+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b04007400 con 0x7f0b140b0e30 2024-03-06T07:23:24.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.761+0000 7f0b02ffd640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0af406b500 0x7f0af406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.761+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b040895a0 con 0x7f0b140b0e30 2024-03-06T07:23:24.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.761+0000 7f0b11d9b640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0af406b500 0x7f0af406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.761+0000 7f0b11d9b640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0af406b500 0x7f0af406d9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0b140cde10 tx=0x7f0b08009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.765+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b04057dd0 con 0x7f0b140b0e30 2024-03-06T07:23:24.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.941+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b140d0040 con 0x7f0b140b0e30 2024-03-06T07:23:24.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.942+0000 7f0b02ffd640 1 -- 172.21.15.122:0/135134208 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b0405be70 con 0x7f0b140b0e30 2024-03-06T07:23:24.943 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.943+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0af406b500 msgr2=0x7f0af406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.943+0000 7f0b18da1640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0af406b500 0x7f0af406d9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0b140cde10 tx=0x7f0b08009190 comp rx=0 tx=0).stop 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.943+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 msgr2=0x7f0b1415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.943+0000 7f0b18da1640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b1415ee40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c3c0 tx=0x7f0b0400c880 comp rx=0 tx=0).stop 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 shutdown_connections 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0af406b500 0x7f0af406d9c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b141614d0 0x7f0b140cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b140d4860 0x7f0b140c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 --2- 172.21.15.122:0/135134208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b140b0e30 0x7f0b1415ee40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 >> 172.21.15.122:0/135134208 conn(0x7f0b140d3d40 msgr2=0x7f0b140d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 shutdown_connections 2024-03-06T07:23:24.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:24.944+0000 7f0b18da1640 1 -- 172.21.15.122:0/135134208 wait complete. 2024-03-06T07:23:24.959 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:25.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.066+0000 7fdf785f6640 1 Processor -- start 2024-03-06T07:23:25.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.066+0000 7fdf785f6640 1 -- start start 2024-03-06T07:23:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf785f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 0x7fdf7006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf785f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdf700700a0 0x7fdf7006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf785f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf700b04c0 con 0x7fdf7006b600 2024-03-06T07:23:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf785f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7006d8d0 con 0x7fdf700700a0 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf785f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7006db70 con 0x7fdf7006b220 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf76b93640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdf7006b600 0x7fdf70161290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49132/0 (socket says 172.21.15.122:49132) 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf76b93640 1 -- 172.21.15.122:0/3385708825 learned_addr learned my addr 172.21.15.122:0/3385708825 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf75b91640 1 --2- 172.21.15.122:0/3385708825 >> v2:172.21.15.169:3300/0 conn(0x7fdf700700a0 0x7fdf7006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.067+0000 7fdf76392640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 0x7fdf7006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870875451 0 0) 0x7fdf700b04c0 con 0x7fdf7006b600 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c003520 con 0x7fdf7006b600 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384378439 0 0) 0x7fdf5c003520 con 0x7fdf7006b600 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 msgr2=0x7fdf7006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 0x7fdf7006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 >> v2:172.21.15.169:3300/0 conn(0x7fdf700700a0 msgr2=0x7fdf7006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 --2- 172.21.15.122:0/3385708825 >> v2:172.21.15.169:3300/0 conn(0x7fdf700700a0 0x7fdf7006eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7006de50 con 0x7fdf7006b600 2024-03-06T07:23:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fdf680019f0 con 0x7fdf7006b600 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 --2- 172.21.15.122:0/3385708825 >> v2:172.21.15.169:3300/0 conn(0x7fdf700700a0 0x7fdf7006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 >> v1:172.21.15.122:6789/0 conn(0x7fdf7006b600 legacy=0x7fdf70161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 0x7fdf5c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf5c004d40 0x7fdf5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7006d8d0 con 0x7fdf5c003f30 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7006db70 con 0x7fdf5c004d40 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c00ba10 con 0x7fdf7006b220 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf76392640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 0x7fdf5c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf76b93640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf5c004d40 0x7fdf5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 -- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 msgr2=0x7fdf5c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf7006b220 0x7fdf5c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 -- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf5c004d40 msgr2=0x7fdf5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf5c004d40 0x7fdf5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf75b91640 1 -- 172.21.15.122:0/3385708825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c003520 con 0x7fdf5c003f30 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.068+0000 7fdf76b93640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf5c004d40 0x7fdf5c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.069+0000 7fdf75b91640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf5c004800 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009cc0 tx=0x7fdf5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0e1a8a3fabfbb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60039070 con 0x7fdf5c003f30 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6002c960 con 0x7fdf5c003f30 2024-03-06T07:23:25.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf75390640 1 -- 172.21.15.122:0/3385708825 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60009860 con 0x7fdf5c003f30 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf785f6640 1 -- 172.21.15.122:0/3385708825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 msgr2=0x7fdf5c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf785f6640 1 --2- 172.21.15.122:0/3385708825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf5c004800 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009cc0 tx=0x7fdf5c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf76b93640 1 -- 172.21.15.122:0/3385708825 reap_dead start 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf785f6640 1 -- 172.21.15.122:0/3385708825 shutdown_connections 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf785f6640 1 -- 172.21.15.122:0/3385708825 >> 172.21.15.122:0/3385708825 conn(0x7fdf700d0a10 msgr2=0x7fdf700d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf785f6640 1 -- 172.21.15.122:0/3385708825 shutdown_connections 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.070+0000 7fdf785f6640 1 -- 172.21.15.122:0/3385708825 wait complete. 2024-03-06T07:23:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.072+0000 7fdf785f6640 1 Processor -- start 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.072+0000 7fdf785f6640 1 -- start start 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf785f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf70146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf785f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf70155800 0x7fdf701447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf785f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf701450e0 0x7fdf7006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf785f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7006fc90 con 0x7fdf5c003f30 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf785f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7006fb10 con 0x7fdf70155800 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf785f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf700b3a40 con 0x7fdf701450e0 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf76392640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf70146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf76392640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf70146af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57248/0 (socket says 172.21.15.122:57248) 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf76392640 1 -- 172.21.15.122:0/608715227 learned_addr learned my addr 172.21.15.122:0/608715227 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf75b91640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf70155800 0x7fdf701447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.073+0000 7fdf76b93640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf701450e0 0x7fdf7006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf76392640 1 -- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf701450e0 msgr2=0x7fdf7006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf76392640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf701450e0 0x7fdf7006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf76392640 1 -- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf70155800 msgr2=0x7fdf701447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf76392640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf70155800 0x7fdf701447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf76392640 1 -- 172.21.15.122:0/608715227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70145a50 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf76392640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf70146af0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0119a0 tx=0x7fdf6c011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c00fa30 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c00fbd0 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf70145d30 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c014440 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.074+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf701462f0 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.075+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7004f330 con 0x7fdf5c003f30 2024-03-06T07:23:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.075+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf6c012040 con 0x7fdf5c003f30 2024-03-06T07:23:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.076+0000 7fdf66ffd640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf5406b4d0 0x7fdf5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.076+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf6c08a770 con 0x7fdf5c003f30 2024-03-06T07:23:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.076+0000 7fdf75b91640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf5406b4d0 0x7fdf5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.076+0000 7fdf75b91640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf5406b4d0 0x7fdf5406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf60036000 tx=0x7fdf60030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.080+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf6c019030 con 0x7fdf5c003f30 2024-03-06T07:23:25.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.277+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdf7006f6b0 con 0x7fdf5c003f30 2024-03-06T07:23:25.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.278+0000 7fdf66ffd640 1 -- 172.21.15.122:0/608715227 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdf6c059680 con 0x7fdf5c003f30 2024-03-06T07:23:25.279 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf5406b4d0 msgr2=0x7fdf5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf5406b4d0 0x7fdf5406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf60036000 tx=0x7fdf60030000 comp rx=0 tx=0).stop 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 msgr2=0x7fdf70146af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf70146af0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0119a0 tx=0x7fdf6c011da0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 shutdown_connections 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf5406b4d0 0x7fdf5406d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf701450e0 0x7fdf7006e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf70155800 0x7fdf701447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 --2- 172.21.15.122:0/608715227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf5c003f30 0x7fdf70146af0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 >> 172.21.15.122:0/608715227 conn(0x7fdf700d0a10 msgr2=0x7fdf700b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 shutdown_connections 2024-03-06T07:23:25.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.280+0000 7fdf785f6640 1 -- 172.21.15.122:0/608715227 wait complete. 2024-03-06T07:23:25.291 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:23:25.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:25.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:25.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[52804]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:25.319 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:25.319 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:25.376 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:23:25.376 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:25.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:25.394 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:25.394 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:25.450 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:25.450 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:23:25.450 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:25.465 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:25.465 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:25.465 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:25.521 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:25.522 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:23:25.522 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:25.539 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:25.539 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:23:25.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:25.539 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:25.596 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:25.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.699+0000 7f503eb56640 1 Processor -- start 2024-03-06T07:23:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.699+0000 7f503eb56640 1 -- start start 2024-03-06T07:23:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503eb56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f50380b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503eb56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50380b3430 con 0x7f503815dfe0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503eb56640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50380b3710 con 0x7f50380d22b0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503eb56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380b31e0 con 0x7f50380d2710 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503c8f2640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f503815dfe0 0x7f50380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49138/0 (socket says 172.21.15.122:49138) 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503c8f2640 1 -- 172.21.15.122:0/1404942463 learned_addr learned my addr 172.21.15.122:0/1404942463 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.700+0000 7f503d0f3640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f50380b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829228833 0 0) 0x7f50380b3430 con 0x7f503815dfe0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5028003520 con 0x7f503815dfe0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971315062 0 0) 0x7f50380b3710 con 0x7f50380d22b0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50380b3430 con 0x7f50380d22b0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347945463 0 0) 0x7f5028003520 con 0x7f503815dfe0 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 msgr2=0x7f50380b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f50380b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 >> v1:172.21.15.169:6789/0 conn(0x7f50380d22b0 legacy=0x7f50380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50380b39f0 con 0x7f503815dfe0 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f502c001670 con 0x7f503815dfe0 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503d0f3640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f50380b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 >> v1:172.21.15.122:6789/0 conn(0x7f503815dfe0 legacy=0x7f50380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f5028003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f5028004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5028004d40 0x7f50280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50380b31e0 con 0x7f5028003b70 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502800ba70 con 0x7f5028004d40 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502800bc40 con 0x7f50380d2710 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503c8f2640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f5028003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f5037fff640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f5028004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503d0f3640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5028004d40 0x7f50280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503c8f2640 1 -- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 msgr2=0x7f5028004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503c8f2640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50380d2710 0x7f5028004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503c8f2640 1 -- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5028004d40 msgr2=0x7f50280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503c8f2640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5028004d40 0x7f50280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503c8f2640 1 -- 172.21.15.122:0/1404942463 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5028003520 con 0x7f5028003b70 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.701+0000 7f503d0f3640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5028004d40 0x7f50280051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.702+0000 7f503c8f2640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f5028003f90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c1c0 tx=0x7f5028005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3217809f819d8349 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c007cc0 con 0x7f5028003b70 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c030070 con 0x7f5028003b70 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f50377fe640 1 -- 172.21.15.122:0/1404942463 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c005d30 con 0x7f5028003b70 2024-03-06T07:23:25.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503eb56640 1 -- 172.21.15.122:0/1404942463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 msgr2=0x7f5028003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503eb56640 1 --2- 172.21.15.122:0/1404942463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f5028003f90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c1c0 tx=0x7f5028005830 comp rx=0 tx=0).stop 2024-03-06T07:23:25.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503d0f3640 1 -- 172.21.15.122:0/1404942463 reap_dead start 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503eb56640 1 -- 172.21.15.122:0/1404942463 shutdown_connections 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503eb56640 1 -- 172.21.15.122:0/1404942463 >> 172.21.15.122:0/1404942463 conn(0x7f50380d3cd0 msgr2=0x7f50380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503eb56640 1 -- 172.21.15.122:0/1404942463 shutdown_connections 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.703+0000 7f503eb56640 1 -- 172.21.15.122:0/1404942463 wait complete. 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.705+0000 7f503eb56640 1 Processor -- start 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.705+0000 7f503eb56640 1 -- start start 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503eb56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f503814d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503eb56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5038150e70 0x7f50381f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503eb56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f503815f7e0 0x7f503815fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503eb56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50380cfa00 con 0x7f5028003b70 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503eb56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50380b3d60 con 0x7f503815f7e0 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503eb56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380d1c70 con 0x7f5038150e70 2024-03-06T07:23:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503c8f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f503814d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503c8f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f503814d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57268/0 (socket says 172.21.15.122:57268) 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503c8f2640 1 -- 172.21.15.122:0/3479247031 learned_addr learned my addr 172.21.15.122:0/3479247031 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f5037fff640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5038150e70 0x7f50381f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.706+0000 7f503d0f3640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f503815f7e0 0x7f503815fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503c8f2640 1 -- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5038150e70 msgr2=0x7f50381f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503c8f2640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5038150e70 0x7f50381f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503c8f2640 1 -- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f503815f7e0 msgr2=0x7f503815fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503c8f2640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f503815f7e0 0x7f503815fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503c8f2640 1 -- 172.21.15.122:0/3479247031 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50381512f0 con 0x7f5028003b70 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503c8f2640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f503814d870 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c1c0 tx=0x7f50380cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c045070 con 0x7f5028003b70 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c02cbd0 con 0x7f5028003b70 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c001c00 con 0x7f5028003b70 2024-03-06T07:23:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503806bba0 con 0x7f5028003b70 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.707+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503806c160 con 0x7f5028003b70 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.708+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503804f330 con 0x7f5028003b70 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.709+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f502c007070 con 0x7f5028003b70 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.709+0000 7f50357fa640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f501c06b500 0x7f501c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.709+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f502c0ab260 con 0x7f5028003b70 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.709+0000 7f5037fff640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f501c06b500 0x7f501c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.709+0000 7f5037fff640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f501c06b500 0x7f501c06d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f503815c4e0 tx=0x7f5024009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.717+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f502c079a90 con 0x7f5028003b70 2024-03-06T07:23:25.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.949+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50380ce9b0 con 0x7f5028003b70 2024-03-06T07:23:25.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.949+0000 7f50357fa640 1 -- 172.21.15.122:0/3479247031 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f502c07db30 con 0x7f5028003b70 2024-03-06T07:23:25.952 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:25.952 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:25.952 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f501c06b500 msgr2=0x7f501c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f501c06b500 0x7f501c06d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f503815c4e0 tx=0x7f5024009040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 msgr2=0x7f503814d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f503814d870 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c1c0 tx=0x7f50380cfd40 comp rx=0 tx=0).stop 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 shutdown_connections 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f501c06b500 0x7f501c06d9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f503815f7e0 0x7f503815fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5038150e70 0x7f50381f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 --2- 172.21.15.122:0/3479247031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5028003b70 0x7f503814d870 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.951+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 >> 172.21.15.122:0/3479247031 conn(0x7f50380d3cd0 msgr2=0x7f50380d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.952+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 shutdown_connections 2024-03-06T07:23:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:25.952+0000 7f503eb56640 1 -- 172.21.15.122:0/3479247031 wait complete. 2024-03-06T07:23:25.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:26.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.065+0000 7fd9bc725640 1 Processor -- start 2024-03-06T07:23:26.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.065+0000 7fd9bc725640 1 -- start start 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bc725640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd9b415baf0 0x7fd9b4072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bc725640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd9b40cd390 0x7fd9b406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bc725640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b40d0610 con 0x7fd9b406b4f0 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bc725640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b4072c40 con 0x7fd9b415baf0 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bc725640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b406b920 con 0x7fd9b40cd390 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9ba4c1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd9b415baf0 0x7fd9b4072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bacc2640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd9b406b4f0 0x7fd9b415f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49154/0 (socket says 172.21.15.122:49154) 2024-03-06T07:23:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9bacc2640 1 -- 172.21.15.122:0/3083103356 learned_addr learned my addr 172.21.15.122:0/3083103356 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9b9cc0640 1 --2- 172.21.15.122:0/3083103356 >> v2:172.21.15.169:3300/0 conn(0x7fd9b40cd390 0x7fd9b406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/3083103356 >> v2:172.21.15.169:3300/0 conn(0x7fd9b40cd390 msgr2=0x7fd9b406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9ba4c1640 1 --2- 172.21.15.122:0/3083103356 >> v2:172.21.15.169:3300/0 conn(0x7fd9b40cd390 0x7fd9b406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/3083103356 >> v1:172.21.15.122:6789/0 conn(0x7fd9b406b4f0 legacy=0x7fd9b415f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/3083103356 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b40d08f0 con 0x7fd9b415baf0 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9b94bf640 1 -- 172.21.15.122:0/3083103356 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563409103 0 0) 0x7fd9b40d0610 con 0x7fd9b406b4f0 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.066+0000 7fd9b9cc0640 1 --2- 172.21.15.122:0/3083103356 >> v2:172.21.15.169:3300/0 conn(0x7fd9b40cd390 0x7fd9b406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.067+0000 7fd9ba4c1640 1 --2- 172.21.15.122:0/3083103356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b415baf0 0x7fd9b4072700 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40732b0 tx=0x7fd9b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10ceeadabe2790a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.067+0000 7fd9b94bf640 1 -- 172.21.15.122:0/3083103356 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0035020 con 0x7fd9b415baf0 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.067+0000 7fd9b94bf640 1 -- 172.21.15.122:0/3083103356 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b0031830 con 0x7fd9b415baf0 2024-03-06T07:23:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.067+0000 7fd9b94bf640 1 -- 172.21.15.122:0/3083103356 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0031c00 con 0x7fd9b415baf0 2024-03-06T07:23:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 -- 172.21.15.122:0/3083103356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b415baf0 msgr2=0x7fd9b4072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 --2- 172.21.15.122:0/3083103356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b415baf0 0x7fd9b4072700 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40732b0 tx=0x7fd9b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 -- 172.21.15.122:0/3083103356 shutdown_connections 2024-03-06T07:23:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 --2- 172.21.15.122:0/3083103356 >> v2:172.21.15.169:3300/0 conn(0x7fd9b40cd390 0x7fd9b406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 --2- 172.21.15.122:0/3083103356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b415baf0 0x7fd9b4072700 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 -- 172.21.15.122:0/3083103356 >> 172.21.15.122:0/3083103356 conn(0x7fd9b406c0c0 msgr2=0x7fd9b406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 -- 172.21.15.122:0/3083103356 shutdown_connections 2024-03-06T07:23:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.068+0000 7fd9bc725640 1 -- 172.21.15.122:0/3083103356 wait complete. 2024-03-06T07:23:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.069+0000 7fd9bc725640 1 Processor -- start 2024-03-06T07:23:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.069+0000 7fd9bc725640 1 -- start start 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bc725640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 0x7fd9b4150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bc725640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9b40cd390 0x7fd9b415acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bc725640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9b415baf0 0x7fd9b406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bc725640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40d1190 con 0x7fd9b406b4f0 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bc725640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40d0eb0 con 0x7fd9b415baf0 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bc725640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40cfda0 con 0x7fd9b40cd390 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9ba4c1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 0x7fd9b4150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9ba4c1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 0x7fd9b4150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57292/0 (socket says 172.21.15.122:57292) 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/73027268 learned_addr learned my addr 172.21.15.122:0/73027268 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9b9cc0640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9b40cd390 0x7fd9b415acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.070+0000 7fd9bacc2640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9b415baf0 0x7fd9b406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9b40cd390 msgr2=0x7fd9b415acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9ba4c1640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9b40cd390 0x7fd9b415acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9b415baf0 msgr2=0x7fd9b406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9ba4c1640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9b415baf0 0x7fd9b406ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9ba4c1640 1 -- 172.21.15.122:0/73027268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b40ac5e0 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9bacc2640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9b415baf0 0x7fd9b406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9b9cc0640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9b40cd390 0x7fd9b415acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9ba4c1640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 0x7fd9b4150e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd9b002bee0 tx=0x7fd9b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0041070 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b003b920 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b40ac8c0 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0035040 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.071+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b40ace80 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.072+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b404f330 con 0x7fd9b406b4f0 2024-03-06T07:23:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.072+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9b0038020 con 0x7fd9b406b4f0 2024-03-06T07:23:26.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.073+0000 7fd9aaffd640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd99806b4d0 0x7fd99806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.073+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9b00abbb0 con 0x7fd9b406b4f0 2024-03-06T07:23:26.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.073+0000 7fd9b9cc0640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd99806b4d0 0x7fd99806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.073+0000 7fd9b9cc0640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd99806b4d0 0x7fd99806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4006fd0 tx=0x7fd9a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.077+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9b007aa40 con 0x7fd9b406b4f0 2024-03-06T07:23:26.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.260+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9b40747a0 con 0x7fd9b406b4f0 2024-03-06T07:23:26.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.261+0000 7fd9aaffd640 1 -- 172.21.15.122:0/73027268 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16669 (secure 0 0 0) 0x7fd9b007a340 con 0x7fd9b406b4f0 2024-03-06T07:23:26.263 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:26.263 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:53.997648+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:26.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd99806b4d0 msgr2=0x7fd99806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd99806b4d0 0x7fd99806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4006fd0 tx=0x7fd9a4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 msgr2=0x7fd9b4150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 0x7fd9b4150e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd9b002bee0 tx=0x7fd9b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 shutdown_connections 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd99806b4d0 0x7fd99806d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9b415baf0 0x7fd9b406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9b40cd390 0x7fd9b415acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 --2- 172.21.15.122:0/73027268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9b406b4f0 0x7fd9b4150e30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 >> 172.21.15.122:0/73027268 conn(0x7fd9b406c0c0 msgr2=0x7fd9b415c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 shutdown_connections 2024-03-06T07:23:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.263+0000 7fd9bc725640 1 -- 172.21.15.122:0/73027268 wait complete. 2024-03-06T07:23:26.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:50.561315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:51.306826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:26.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.378+0000 7f964bdac640 1 Processor -- start 2024-03-06T07:23:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.378+0000 7f964bdac640 1 -- start start 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.378+0000 7f964bdac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96441614d0 0x7f96440d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.378+0000 7f964bdac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f96440d1910 0x7f96440d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f964bdac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96440d2740 con 0x7f96440d22b0 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f964bdac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96441618b0 con 0x7f96440d1910 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f964bdac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96440cf400 con 0x7f96441614d0 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f9649347640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f96440d1910 0x7f96440d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f964a349640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f96440d22b0 0x7f96440d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49164/0 (socket says 172.21.15.122:49164) 2024-03-06T07:23:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f964a349640 1 -- 172.21.15.122:0/443946069 learned_addr learned my addr 172.21.15.122:0/443946069 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f9649347640 1 -- 172.21.15.122:0/443946069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96441614d0 msgr2=0x7f96440d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f9649347640 1 --2- 172.21.15.122:0/443946069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96441614d0 0x7f96440d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f9649347640 1 -- 172.21.15.122:0/443946069 >> v1:172.21.15.122:6789/0 conn(0x7f96440d22b0 legacy=0x7f96440d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f9649347640 1 -- 172.21.15.122:0/443946069 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96440cf6a0 con 0x7f96440d1910 2024-03-06T07:23:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.379+0000 7f9648b46640 1 -- 172.21.15.122:0/443946069 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321932179 0 0) 0x7f963c001490 con 0x7f96440d22b0 2024-03-06T07:23:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.380+0000 7f9649347640 1 --2- 172.21.15.122:0/443946069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f96440d1d70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96340097a0 tx=0x7f963402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f21cf63ed19990 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.380+0000 7f9648b46640 1 -- 172.21.15.122:0/443946069 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634035020 con 0x7f96440d1910 2024-03-06T07:23:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.380+0000 7f9648b46640 1 -- 172.21.15.122:0/443946069 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9634031830 con 0x7f96440d1910 2024-03-06T07:23:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.380+0000 7f9648b46640 1 -- 172.21.15.122:0/443946069 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634031c00 con 0x7f96440d1910 2024-03-06T07:23:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 -- 172.21.15.122:0/443946069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 msgr2=0x7f96440d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 --2- 172.21.15.122:0/443946069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f96440d1d70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96340097a0 tx=0x7f963402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 -- 172.21.15.122:0/443946069 shutdown_connections 2024-03-06T07:23:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 --2- 172.21.15.122:0/443946069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f96440d1d70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 --2- 172.21.15.122:0/443946069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96441614d0 0x7f96440d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 -- 172.21.15.122:0/443946069 >> 172.21.15.122:0/443946069 conn(0x7f96440d3d40 msgr2=0x7f96440d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 -- 172.21.15.122:0/443946069 shutdown_connections 2024-03-06T07:23:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.381+0000 7f964bdac640 1 -- 172.21.15.122:0/443946069 wait complete. 2024-03-06T07:23:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.382+0000 7f964bdac640 1 Processor -- start 2024-03-06T07:23:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.382+0000 7f964bdac640 1 -- start start 2024-03-06T07:23:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f964414f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96440d22b0 0x7f96441e6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96441614d0 0x7f96440cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96440b0910 con 0x7f96440d1910 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96440cfc60 con 0x7f96441614d0 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96440b1e40 con 0x7f96440d22b0 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f964414f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f964414f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57302/0 (socket says 172.21.15.122:57302) 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 -- 172.21.15.122:0/412971661 learned_addr learned my addr 172.21.15.122:0/412971661 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964a349640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96441614d0 0x7f96440cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649347640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96440d22b0 0x7f96441e6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 -- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96440d22b0 msgr2=0x7f96441e6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96440d22b0 0x7f96441e6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 -- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96441614d0 msgr2=0x7f96440cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96441614d0 0x7f96440cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 -- 172.21.15.122:0/412971661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96441e7470 con 0x7f96440d1910 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964a349640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96441614d0 0x7f96440cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f9649b48640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f964414f100 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f964000b410 tx=0x7f964000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640015020 con 0x7f96440d1910 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640011850 con 0x7f96440d1910 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96441e7750 con 0x7f96440d1910 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640011c40 con 0x7f96440d1910 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.383+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96441e7d10 con 0x7f96440d1910 2024-03-06T07:23:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.384+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9644068ff0 con 0x7f96440d1910 2024-03-06T07:23:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.385+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9640018020 con 0x7f96440d1910 2024-03-06T07:23:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.385+0000 7f963a7fc640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f962806b4d0 0x7f962806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.386+0000 7f9649347640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f962806b4d0 0x7f962806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.386+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f964008a160 con 0x7f96440d1910 2024-03-06T07:23:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.386+0000 7f9649347640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f962806b4d0 0x7f962806d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9634036000 tx=0x7f9634008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.390+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f964001f020 con 0x7f96440d1910 2024-03-06T07:23:26.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.562+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96440b51f0 con 0x7f96440d1910 2024-03-06T07:23:26.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.563+0000 7f963a7fc640 1 -- 172.21.15.122:0/412971661 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96400172c0 con 0x7f96440d1910 2024-03-06T07:23:26.565 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f962806b4d0 msgr2=0x7f962806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f962806b4d0 0x7f962806d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9634036000 tx=0x7f9634008030 comp rx=0 tx=0).stop 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 msgr2=0x7f964414f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f964414f100 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f964000b410 tx=0x7f964000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 shutdown_connections 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f962806b4d0 0x7f962806d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96441614d0 0x7f96440cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96440d22b0 0x7f96441e6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 --2- 172.21.15.122:0/412971661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96440d1910 0x7f964414f100 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.564+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 >> 172.21.15.122:0/412971661 conn(0x7f96440d3d40 msgr2=0x7f964406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.565+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 shutdown_connections 2024-03-06T07:23:26.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.565+0000 7f964bdac640 1 -- 172.21.15.122:0/412971661 wait complete. 2024-03-06T07:23:26.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:26.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.680+0000 7fd64ccec640 1 Processor -- start 2024-03-06T07:23:26.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.680+0000 7fd64ccec640 1 -- start start 2024-03-06T07:23:26.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64ccec640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd64806cab0 0x7fd6480b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64ccec640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64806d030 con 0x7fd64815def0 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64ccec640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64806d310 con 0x7fd6480cea50 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64ccec640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480cf1c0 con 0x7fd64806cab0 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd646d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd64806cab0 0x7fd6480b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd64815def0 0x7fd6480ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49168/0 (socket says 172.21.15.122:49168) 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64659c640 1 -- 172.21.15.122:0/2258047879 learned_addr learned my addr 172.21.15.122:0/2258047879 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd646d9d640 1 -- 172.21.15.122:0/2258047879 >> v1:172.21.15.169:6789/0 conn(0x7fd6480cea50 legacy=0x7fd6480ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd646d9d640 1 -- 172.21.15.122:0/2258047879 >> v1:172.21.15.122:6789/0 conn(0x7fd64815def0 legacy=0x7fd6480ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd646d9d640 1 -- 172.21.15.122:0/2258047879 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480cc110 con 0x7fd64806cab0 2024-03-06T07:23:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.681+0000 7fd64559a640 1 -- 172.21.15.122:0/2258047879 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397916091 0 0) 0x7fd64806d030 con 0x7fd64815def0 2024-03-06T07:23:26.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.682+0000 7fd646d9d640 1 --2- 172.21.15.122:0/2258047879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd64806cab0 0x7fd6480b41b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd640009d90 tx=0x7fd64002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2883614af13b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.682+0000 7fd64559a640 1 -- 172.21.15.122:0/2258047879 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640039070 con 0x7fd64806cab0 2024-03-06T07:23:26.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.682+0000 7fd64559a640 1 -- 172.21.15.122:0/2258047879 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64002cc60 con 0x7fd64806cab0 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.682+0000 7fd64559a640 1 -- 172.21.15.122:0/2258047879 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640034660 con 0x7fd64806cab0 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 -- 172.21.15.122:0/2258047879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd64806cab0 msgr2=0x7fd6480b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 --2- 172.21.15.122:0/2258047879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd64806cab0 0x7fd6480b41b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd640009d90 tx=0x7fd64002beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 -- 172.21.15.122:0/2258047879 shutdown_connections 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 --2- 172.21.15.122:0/2258047879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd64806cab0 0x7fd6480b41b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 -- 172.21.15.122:0/2258047879 >> 172.21.15.122:0/2258047879 conn(0x7fd6480b3690 msgr2=0x7fd6480b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 -- 172.21.15.122:0/2258047879 shutdown_connections 2024-03-06T07:23:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.683+0000 7fd64ccec640 1 -- 172.21.15.122:0/2258047879 wait complete. 2024-03-06T07:23:26.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.684+0000 7fd64ccec640 1 Processor -- start 2024-03-06T07:23:26.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.684+0000 7fd64ccec640 1 -- start start 2024-03-06T07:23:26.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd64ccec640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64806cab0 0x7fd6480c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd64ccec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 0x7fd648154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd64ccec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd64815def0 0x7fd6481f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd64ccec640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64815e2d0 con 0x7fd6480cea50 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd64ccec640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6480cc760 con 0x7fd64815def0 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd64ccec640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64815e450 con 0x7fd64806cab0 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 0x7fd648154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 0x7fd648154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57322/0 (socket says 172.21.15.122:57322) 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 -- 172.21.15.122:0/3457393861 learned_addr learned my addr 172.21.15.122:0/3457393861 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd646d9d640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd64815def0 0x7fd6481f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 -- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64806cab0 msgr2=0x7fd6480c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64806cab0 0x7fd6480c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 -- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd64815def0 msgr2=0x7fd6481f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd64815def0 0x7fd6481f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 -- 172.21.15.122:0/3457393861 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6481f4380 con 0x7fd6480cea50 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd646d9d640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd64815def0 0x7fd6481f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.685+0000 7fd645d9b640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 0x7fd648154bd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c460 tx=0x7fd63800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.686+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638016020 con 0x7fd6480cea50 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.686+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638012870 con 0x7fd6480cea50 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.686+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6480c5f10 con 0x7fd6480cea50 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.686+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638012c60 con 0x7fd6480cea50 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.686+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6480ca3d0 con 0x7fd6480cea50 2024-03-06T07:23:26.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.687+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64809fcf0 con 0x7fd6480cea50 2024-03-06T07:23:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.687+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd638019020 con 0x7fd6480cea50 2024-03-06T07:23:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.687+0000 7fd636ffd640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62806b500 0x7fd62806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.688+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd63808a1b0 con 0x7fd6480cea50 2024-03-06T07:23:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.688+0000 7fd64659c640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62806b500 0x7fd62806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.688+0000 7fd64659c640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62806b500 0x7fd62806d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd63c007fd0 tx=0x7fd63c007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.692+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd638020020 con 0x7fd6480cea50 2024-03-06T07:23:26.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.868+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6480cdc90 con 0x7fd6480cea50 2024-03-06T07:23:26.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.868+0000 7fd636ffd640 1 -- 172.21.15.122:0/3457393861 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6380182c0 con 0x7fd6480cea50 2024-03-06T07:23:26.870 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:23:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62806b500 msgr2=0x7fd62806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62806b500 0x7fd62806d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd63c007fd0 tx=0x7fd63c007960 comp rx=0 tx=0).stop 2024-03-06T07:23:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 msgr2=0x7fd648154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 0x7fd648154bd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c460 tx=0x7fd63800c920 comp rx=0 tx=0).stop 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 shutdown_connections 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62806b500 0x7fd62806d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd64815def0 0x7fd6481f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6480cea50 0x7fd648154bd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 --2- 172.21.15.122:0/3457393861 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64806cab0 0x7fd6480c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 >> 172.21.15.122:0/3457393861 conn(0x7fd6480b3690 msgr2=0x7fd648160ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 shutdown_connections 2024-03-06T07:23:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.870+0000 7fd64ccec640 1 -- 172.21.15.122:0/3457393861 wait complete. 2024-03-06T07:23:26.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:26.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.985+0000 7f49c7fc7640 1 Processor -- start 2024-03-06T07:23:26.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.986+0000 7f49c7fc7640 1 -- start start 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.986+0000 7f49c7fc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.986+0000 7f49c7fc7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f49c00d0af0 0x7f49c00ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.986+0000 7f49c7fc7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00cd1e0 con 0x7f49c015c050 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.986+0000 7f49c7fc7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c00cd4b0 con 0x7f49c00d0af0 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.986+0000 7f49c7fc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00cd790 con 0x7f49c006fd10 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c6564640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f49c00d0af0 0x7f49c00ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c5d63640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f49c015c050 0x7f49c006f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49174/0 (socket says 172.21.15.122:49174) 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c5d63640 1 -- 172.21.15.122:0/3073967366 learned_addr learned my addr 172.21.15.122:0/3073967366 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c5562640 1 --2- 172.21.15.122:0/3073967366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c6564640 1 -- 172.21.15.122:0/3073967366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 msgr2=0x7f49c00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c6564640 1 --2- 172.21.15.122:0/3073967366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c00cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c6564640 1 -- 172.21.15.122:0/3073967366 >> v1:172.21.15.122:6789/0 conn(0x7f49c015c050 legacy=0x7f49c006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c6564640 1 -- 172.21.15.122:0/3073967366 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00cda70 con 0x7f49c00d0af0 2024-03-06T07:23:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.987+0000 7f49c4d61640 1 -- 172.21.15.122:0/3073967366 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441397827 0 0) 0x7f49c00cd1e0 con 0x7f49c015c050 2024-03-06T07:23:26.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c6564640 1 --2- 172.21.15.122:0/3073967366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c00d0af0 0x7f49c00ccca0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009d90 tx=0x7f49b802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f08223ca1df7e1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c4d61640 1 -- 172.21.15.122:0/3073967366 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8039070 con 0x7f49c00d0af0 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c4d61640 1 -- 172.21.15.122:0/3073967366 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b802cbc0 con 0x7f49c00d0af0 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c4d61640 1 -- 172.21.15.122:0/3073967366 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8034440 con 0x7f49c00d0af0 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3073967366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c00d0af0 msgr2=0x7f49c00ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3073967366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c00d0af0 0x7f49c00ccca0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009d90 tx=0x7f49b802be10 comp rx=0 tx=0).stop 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3073967366 shutdown_connections 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3073967366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c00d0af0 0x7f49c00ccca0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3073967366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3073967366 >> 172.21.15.122:0/3073967366 conn(0x7f49c00b3800 msgr2=0x7f49c00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3073967366 shutdown_connections 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.988+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3073967366 wait complete. 2024-03-06T07:23:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.989+0000 7f49c7fc7640 1 Processor -- start 2024-03-06T07:23:26.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 -- start start 2024-03-06T07:23:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c01541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49c00d0af0 0x7f49c015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 0x7f49c00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0165c30 con 0x7f49c015c050 2024-03-06T07:23:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0165ab0 con 0x7f49c00d0af0 2024-03-06T07:23:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c7fc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0165db0 con 0x7f49c006fd10 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c6564640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 0x7f49c00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c6564640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 0x7f49c00cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57340/0 (socket says 172.21.15.122:57340) 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c6564640 1 -- 172.21.15.122:0/3633089403 learned_addr learned my addr 172.21.15.122:0/3633089403 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c5562640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49c00d0af0 0x7f49c015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.990+0000 7f49c5d63640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c01541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c6564640 1 -- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 msgr2=0x7f49c01541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c6564640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c01541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c6564640 1 -- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49c00d0af0 msgr2=0x7f49c015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c6564640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49c00d0af0 0x7f49c015b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c6564640 1 -- 172.21.15.122:0/3633089403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c006e0c0 con 0x7f49c015c050 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c5562640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49c00d0af0 0x7f49c015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c6564640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 0x7f49c00cae80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f49b8037040 tx=0x7f49b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8044030 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b802c9c0 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c006db00 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8039040 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.991+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c01f4080 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.992+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c009fe60 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.993+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49b8047070 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.993+0000 7f49b67fc640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49a406b4d0 0x7f49a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.993+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49b80ac9e0 con 0x7f49c015c050 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.993+0000 7f49c5d63640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49a406b4d0 0x7f49a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.993+0000 7f49c5d63640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49a406b4d0 0x7f49a406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49bc000f60 tx=0x7f49bc007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:26.997+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49b804e070 con 0x7f49c015c050 2024-03-06T07:23:27.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.186+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49c0068ff0 con 0x7f49c015c050 2024-03-06T07:23:27.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.186+0000 7f49b67fc640 1 -- 172.21.15.122:0/3633089403 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49b802cb90 con 0x7f49c015c050 2024-03-06T07:23:27.190 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:23:27.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49a406b4d0 msgr2=0x7f49a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49a406b4d0 0x7f49a406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49bc000f60 tx=0x7f49bc007c00 comp rx=0 tx=0).stop 2024-03-06T07:23:27.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 msgr2=0x7f49c00cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 0x7f49c00cae80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f49b8037040 tx=0x7f49b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 shutdown_connections 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49a406b4d0 0x7f49a406d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c015c050 0x7f49c00cae80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49c00d0af0 0x7f49c015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 --2- 172.21.15.122:0/3633089403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49c006fd10 0x7f49c01541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 >> 172.21.15.122:0/3633089403 conn(0x7f49c00b3800 msgr2=0x7f49c01589f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 shutdown_connections 2024-03-06T07:23:27.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.188+0000 7f49c7fc7640 1 -- 172.21.15.122:0/3633089403 wait complete. 2024-03-06T07:23:27.201 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:27.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.300+0000 7fae1ab20640 1 Processor -- start 2024-03-06T07:23:27.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.300+0000 7fae1ab20640 1 -- start start 2024-03-06T07:23:27.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae1ab20640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 0x7fae140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae1ab20640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fae140b41f0 0x7fae140b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae1ab20640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140b3430 con 0x7fae1415dfe0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae1ab20640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae140d2c50 con 0x7fae140b41f0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae1ab20640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae140b31e0 con 0x7fae140d22b0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae188bc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fae1415dfe0 0x7fae140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49186/0 (socket says 172.21.15.122:49186) 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae188bc640 1 -- 172.21.15.122:0/305166697 learned_addr learned my addr 172.21.15.122:0/305166697 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0bfff640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 0x7fae140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae190bd640 1 --2- 172.21.15.122:0/305166697 >> v2:172.21.15.169:3300/0 conn(0x7fae140b41f0 0x7fae140b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839274677 0 0) 0x7fae140b3430 con 0x7fae1415dfe0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc003520 con 0x7fae1415dfe0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616843628 0 0) 0x7fadfc003520 con 0x7fae1415dfe0 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 msgr2=0x7fae140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 0x7fae140d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 >> v2:172.21.15.169:3300/0 conn(0x7fae140b41f0 msgr2=0x7fae140b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 --2- 172.21.15.122:0/305166697 >> v2:172.21.15.169:3300/0 conn(0x7fae140b41f0 0x7fae140b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae140b3710 con 0x7fae1415dfe0 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fae10001670 con 0x7fae1415dfe0 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 >> v1:172.21.15.122:6789/0 conn(0x7fae1415dfe0 legacy=0x7fae140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0039b0 0x7fadfc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 0x7fadfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadfc004d40 0x7fadfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140b31e0 con 0x7fadfc004d40 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc00b9c0 con 0x7fadfc0039b0 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc00bb40 con 0x7fae140d22b0 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.301+0000 7fae190bd640 1 --2- 172.21.15.122:0/305166697 >> v2:172.21.15.169:3300/0 conn(0x7fae140b41f0 0x7fae140b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae190bd640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadfc004d40 0x7fadfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae188bc640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0039b0 0x7fadfc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae0bfff640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 0x7fadfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae190bd640 1 -- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 msgr2=0x7fadfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae190bd640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140d22b0 0x7fadfc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae190bd640 1 -- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0039b0 msgr2=0x7fadfc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae190bd640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0039b0 0x7fadfc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.302+0000 7fae190bd640 1 -- 172.21.15.122:0/305166697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc003520 con 0x7fadfc004d40 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae190bd640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadfc004d40 0x7fadfc0051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00e350 tx=0x7fadfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcecea9f588e3f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c009dc0 con 0x7fadfc004d40 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c02cde0 con 0x7fadfc004d40 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae0b7fe640 1 -- 172.21.15.122:0/305166697 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c003b90 con 0x7fadfc004d40 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae1ab20640 1 -- 172.21.15.122:0/305166697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadfc004d40 msgr2=0x7fadfc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae1ab20640 1 --2- 172.21.15.122:0/305166697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadfc004d40 0x7fadfc0051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00e350 tx=0x7fadfc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae190bd640 1 -- 172.21.15.122:0/305166697 reap_dead start 2024-03-06T07:23:27.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae1ab20640 1 -- 172.21.15.122:0/305166697 shutdown_connections 2024-03-06T07:23:27.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae1ab20640 1 -- 172.21.15.122:0/305166697 >> 172.21.15.122:0/305166697 conn(0x7fae140d3cd0 msgr2=0x7fae140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae1ab20640 1 -- 172.21.15.122:0/305166697 shutdown_connections 2024-03-06T07:23:27.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.303+0000 7fae1ab20640 1 -- 172.21.15.122:0/305166697 wait complete. 2024-03-06T07:23:27.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.304+0000 7fae1ab20640 1 Processor -- start 2024-03-06T07:23:27.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.304+0000 7fae1ab20640 1 -- start start 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae1ab20640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae141f31b0 0x7fae14153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae1ab20640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 0x7fae140cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae1ab20640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140ccd90 0x7fae141f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae1ab20640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140cfa00 con 0x7fae140b8f50 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae1ab20640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140b3d60 con 0x7fae141f31b0 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae1ab20640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae140d1c70 con 0x7fae140ccd90 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae0bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 0x7fae140cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae0bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 0x7fae140cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57354/0 (socket says 172.21.15.122:57354) 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.305+0000 7fae0bfff640 1 -- 172.21.15.122:0/1400471123 learned_addr learned my addr 172.21.15.122:0/1400471123 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae188bc640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae141f31b0 0x7fae14153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae190bd640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140ccd90 0x7fae141f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae0bfff640 1 -- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140ccd90 msgr2=0x7fae141f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae0bfff640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140ccd90 0x7fae141f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae0bfff640 1 -- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae141f31b0 msgr2=0x7fae14153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae0bfff640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae141f31b0 0x7fae14153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae0bfff640 1 -- 172.21.15.122:0/1400471123 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae140b9450 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae188bc640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae141f31b0 0x7fae14153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae0bfff640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 0x7fae140cc850 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fae04011a00 tx=0x7fae04011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0400fb50 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0400fcf0 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0400fea0 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae140cba00 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.306+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae140cbfc0 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.307+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1404f330 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.307+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae04012040 con 0x7fae140b8f50 2024-03-06T07:23:27.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.308+0000 7fae097fa640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0006b500 0x7fae0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.308+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae0408a910 con 0x7fae140b8f50 2024-03-06T07:23:27.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.308+0000 7fae188bc640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0006b500 0x7fae0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.308+0000 7fae188bc640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0006b500 0x7fae0006d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae10004840 tx=0x7fae10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.312+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae04019030 con 0x7fae140b8f50 2024-03-06T07:23:27.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.719+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae140cd170 con 0x7fae140b8f50 2024-03-06T07:23:27.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.719+0000 7fae097fa640 1 -- 172.21.15.122:0/1400471123 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fae04059790 con 0x7fae140b8f50 2024-03-06T07:23:27.721 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:27.721 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:27.722 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:27.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0006b500 msgr2=0x7fae0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0006b500 0x7fae0006d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae10004840 tx=0x7fae10009040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 msgr2=0x7fae140cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 0x7fae140cc850 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fae04011a00 tx=0x7fae04011ec0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 shutdown_connections 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0006b500 0x7fae0006d9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae140ccd90 0x7fae141f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae140b8f50 0x7fae140cc850 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 --2- 172.21.15.122:0/1400471123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae141f31b0 0x7fae14153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 >> 172.21.15.122:0/1400471123 conn(0x7fae140d3cd0 msgr2=0x7fae140cee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 shutdown_connections 2024-03-06T07:23:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.722+0000 7fae1ab20640 1 -- 172.21.15.122:0/1400471123 wait complete. 2024-03-06T07:23:27.735 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:27.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.834+0000 7f1db0037640 1 Processor -- start 2024-03-06T07:23:27.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.834+0000 7f1db0037640 1 -- start start 2024-03-06T07:23:27.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1db0037640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da815bd20 0x7f1da80724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1db0037640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1da815f8f0 0x7f1da80ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1db0037640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da8072ad0 con 0x7f1da806afb0 2024-03-06T07:23:27.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1db0037640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da8072db0 con 0x7f1da815f8f0 2024-03-06T07:23:27.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1db0037640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da8073090 con 0x7f1da815bd20 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dad5d2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1da815f8f0 0x7f1da80ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dae5d4640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1da806afb0 0x7f1da80ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49190/0 (socket says 172.21.15.122:49190) 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dae5d4640 1 -- 172.21.15.122:0/3095962257 learned_addr learned my addr 172.21.15.122:0/3095962257 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1daddd3640 1 --2- 172.21.15.122:0/3095962257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da815bd20 0x7f1da80724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dad5d2640 1 -- 172.21.15.122:0/3095962257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da815bd20 msgr2=0x7f1da80724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dad5d2640 1 --2- 172.21.15.122:0/3095962257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da815bd20 0x7f1da80724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dad5d2640 1 -- 172.21.15.122:0/3095962257 >> v1:172.21.15.122:6789/0 conn(0x7f1da806afb0 legacy=0x7f1da80ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.836+0000 7f1dad5d2640 1 -- 172.21.15.122:0/3095962257 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da8073370 con 0x7f1da815f8f0 2024-03-06T07:23:27.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.837+0000 7f1dad5d2640 1 --2- 172.21.15.122:0/3095962257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da80ce290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0097a0 tx=0x7f1d9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c2e517fd953aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.837+0000 7f1dacdd1640 1 -- 172.21.15.122:0/3095962257 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c035020 con 0x7f1da815f8f0 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.837+0000 7f1dacdd1640 1 -- 172.21.15.122:0/3095962257 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9c031830 con 0x7f1da815f8f0 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.837+0000 7f1dacdd1640 1 -- 172.21.15.122:0/3095962257 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c031c00 con 0x7f1da815f8f0 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 -- 172.21.15.122:0/3095962257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 msgr2=0x7f1da80ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 --2- 172.21.15.122:0/3095962257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da80ce290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0097a0 tx=0x7f1d9c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 -- 172.21.15.122:0/3095962257 shutdown_connections 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 --2- 172.21.15.122:0/3095962257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da80ce290 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 --2- 172.21.15.122:0/3095962257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da815bd20 0x7f1da80724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 -- 172.21.15.122:0/3095962257 >> 172.21.15.122:0/3095962257 conn(0x7f1da80b1d00 msgr2=0x7f1da80b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 -- 172.21.15.122:0/3095962257 shutdown_connections 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.838+0000 7f1db0037640 1 -- 172.21.15.122:0/3095962257 wait complete. 2024-03-06T07:23:27.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.839+0000 7f1db0037640 1 Processor -- start 2024-03-06T07:23:27.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.839+0000 7f1db0037640 1 -- start start 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da806afb0 0x7f1da815ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1da815bd20 0x7f1da8158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da815c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da815c100 con 0x7f1da815f8f0 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80672f0 con 0x7f1da815bd20 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80ce7d0 con 0x7f1da806afb0 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57372/0 (socket says 172.21.15.122:57372) 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 -- 172.21.15.122:0/794713770 learned_addr learned my addr 172.21.15.122:0/794713770 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1daddd3640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da806afb0 0x7f1da815ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dad5d2640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1da815bd20 0x7f1da8158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 -- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da806afb0 msgr2=0x7f1da815ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da806afb0 0x7f1da815ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 -- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1da815bd20 msgr2=0x7f1da8158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1da815bd20 0x7f1da8158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 -- 172.21.15.122:0/794713770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da80c9e40 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dad5d2640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1da815bd20 0x7f1da8158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1dae5d4640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da815c280 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1da000cbf0 tx=0x7f1da000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0017070 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da000ad40 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0012600 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da80c7bd0 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.840+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da80c8190 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.841+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da80aec10 con 0x7f1da815f8f0 2024-03-06T07:23:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.842+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1da001a070 con 0x7f1da815f8f0 2024-03-06T07:23:27.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.842+0000 7f1da67fc640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d9406b4d0 0x7f1d9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.842+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1da0089dd0 con 0x7f1da815f8f0 2024-03-06T07:23:27.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.842+0000 7f1daddd3640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d9406b4d0 0x7f1d9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.843+0000 7f1daddd3640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d9406b4d0 0x7f1d9406d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1d98006fd0 tx=0x7f1d98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:27.847+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1da80aec10 con 0x7f1da815f8f0 2024-03-06T07:23:28.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.027+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1da804f330 con 0x7f1da815f8f0 2024-03-06T07:23:28.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.028+0000 7f1da67fc640 1 -- 172.21.15.122:0/794713770 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f1da008b090 con 0x7f1da815f8f0 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:28.052 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:53.997648+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:28.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d9406b4d0 msgr2=0x7f1d9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d9406b4d0 0x7f1d9406d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1d98006fd0 tx=0x7f1d98008040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 msgr2=0x7f1da815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da815c280 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1da000cbf0 tx=0x7f1da000a430 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 shutdown_connections 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d9406b4d0 0x7f1d9406d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1da815f8f0 0x7f1da815c280 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1da815bd20 0x7f1da8158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 --2- 172.21.15.122:0/794713770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1da806afb0 0x7f1da815ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 >> 172.21.15.122:0/794713770 conn(0x7f1da80b1d00 msgr2=0x7f1da80ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 shutdown_connections 2024-03-06T07:23:28.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.030+0000 7f1db0037640 1 -- 172.21.15.122:0/794713770 wait complete. 2024-03-06T07:23:28.055 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:28.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.155+0000 7f76b8f3e640 1 Processor -- start 2024-03-06T07:23:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.155+0000 7f76b8f3e640 1 -- start start 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b8f3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 0x7f76b415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b8f3e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b415ea30 con 0x7f76b406b220 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b8f3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b415ed10 con 0x7f76b406b600 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b8f3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4161b70 con 0x7f76b40ce920 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b259c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f76b406b220 0x7f76b4071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49194/0 (socket says 172.21.15.122:49194) 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b259c640 1 -- 172.21.15.122:0/259495665 learned_addr learned my addr 172.21.15.122:0/259495665 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76abfff640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 0x7f76b415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753847863 0 0) 0x7f76b415ea30 con 0x7f76b406b220 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.156+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a0003520 con 0x7f76b406b220 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106977845 0 0) 0x7f76a0003520 con 0x7f76b406b220 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 msgr2=0x7f76b415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 0x7f76b415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 >> v1:172.21.15.169:6789/0 conn(0x7f76b406b600 legacy=0x7f76b4162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b415eff0 con 0x7f76b406b220 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f76a4001670 con 0x7f76b406b220 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 >> v1:172.21.15.122:6789/0 conn(0x7f76b406b220 legacy=0x7f76b4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76a0004800 0x7f76a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 0x7f76a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4161b70 con 0x7f76a00039e0 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a000ba70 con 0x7f76a0004800 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a000bc70 con 0x7f76b40ce920 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b259c640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b2d9d640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 0x7f76a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76abfff640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76a0004800 0x7f76a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b259c640 1 -- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 msgr2=0x7f76a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b259c640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b40ce920 0x7f76a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b259c640 1 -- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76a0004800 msgr2=0x7f76a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b259c640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76a0004800 0x7f76a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76b259c640 1 -- 172.21.15.122:0/259495665 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a0003520 con 0x7f76a00039e0 2024-03-06T07:23:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.157+0000 7f76abfff640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76a0004800 0x7f76a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.158+0000 7f76b259c640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76a0003e00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f76a400c210 tx=0x7f76a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba475e885e37015 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.158+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4007d10 con 0x7f76a00039e0 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.158+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4030070 con 0x7f76a00039e0 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.158+0000 7f76b1d9b640 1 -- 172.21.15.122:0/259495665 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4005d50 con 0x7f76a00039e0 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b8f3e640 1 -- 172.21.15.122:0/259495665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 msgr2=0x7f76a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/259495665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76a0003e00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f76a400c210 tx=0x7f76a0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b2d9d640 1 -- 172.21.15.122:0/259495665 reap_dead start 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b8f3e640 1 -- 172.21.15.122:0/259495665 shutdown_connections 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b8f3e640 1 -- 172.21.15.122:0/259495665 >> 172.21.15.122:0/259495665 conn(0x7f76b406c0c0 msgr2=0x7f76b406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b8f3e640 1 -- 172.21.15.122:0/259495665 shutdown_connections 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.159+0000 7f76b8f3e640 1 -- 172.21.15.122:0/259495665 wait complete. 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.160+0000 7f76b8f3e640 1 Processor -- start 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.160+0000 7f76b8f3e640 1 -- start start 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b8f3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76b4142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b8f3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76b4152ae0 0x7f76b41530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b8f3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b41535f0 0x7f76b415a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b8f3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4072090 con 0x7f76a00039e0 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b8f3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40ce200 con 0x7f76b4152ae0 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b8f3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b41566d0 con 0x7f76b41535f0 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76b4142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76b4142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57390/0 (socket says 172.21.15.122:57390) 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 -- 172.21.15.122:0/57783366 learned_addr learned my addr 172.21.15.122:0/57783366 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76abfff640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76b4152ae0 0x7f76b41530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b2d9d640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b41535f0 0x7f76b415a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 -- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b41535f0 msgr2=0x7f76b415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b41535f0 0x7f76b415a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 -- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76b4152ae0 msgr2=0x7f76b41530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76b4152ae0 0x7f76b41530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.161+0000 7f76b259c640 1 -- 172.21.15.122:0/57783366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b41569b0 con 0x7f76a00039e0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.162+0000 7f76b259c640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76b4142560 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f76a4001970 tx=0x7f76b4161ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.162+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4045070 con 0x7f76a00039e0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.162+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4004bf0 con 0x7f76a00039e0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.162+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4007d10 con 0x7f76a00039e0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.162+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b4156c90 con 0x7f76a00039e0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.162+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b415aeb0 con 0x7f76a00039e0 2024-03-06T07:23:28.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.163+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b404f330 con 0x7f76a00039e0 2024-03-06T07:23:28.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.164+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76a4007070 con 0x7f76a00039e0 2024-03-06T07:23:28.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.164+0000 7f76aaffd640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f769406b4d0 0x7f769406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.164+0000 7f76abfff640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f769406b4d0 0x7f769406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.164+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76a40ab100 con 0x7f76a00039e0 2024-03-06T07:23:28.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.165+0000 7f76abfff640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f769406b4d0 0x7f769406d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f76b41f7640 tx=0x7f769c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.169+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76a4079a10 con 0x7f76a00039e0 2024-03-06T07:23:28.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.380+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76b40cdf00 con 0x7f76a00039e0 2024-03-06T07:23:28.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.381+0000 7f76aaffd640 1 -- 172.21.15.122:0/57783366 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f76a407dab0 con 0x7f76a00039e0 2024-03-06T07:23:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:28.383 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:28.383 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:28.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f769406b4d0 msgr2=0x7f769406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f769406b4d0 0x7f769406d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f76b41f7640 tx=0x7f769c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 msgr2=0x7f76b4142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76b4142560 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f76a4001970 tx=0x7f76b4161ae0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 shutdown_connections 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f769406b4d0 0x7f769406d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76b41535f0 0x7f76b415a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76b4152ae0 0x7f76b41530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 --2- 172.21.15.122:0/57783366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a00039e0 0x7f76b4142560 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 >> 172.21.15.122:0/57783366 conn(0x7f76b406c0c0 msgr2=0x7f76b41e7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 shutdown_connections 2024-03-06T07:23:28.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.383+0000 7f76b8f3e640 1 -- 172.21.15.122:0/57783366 wait complete. 2024-03-06T07:23:28.396 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:28.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.516+0000 7fb7469c7640 1 Processor -- start 2024-03-06T07:23:28.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb7469c7640 1 -- start start 2024-03-06T07:23:28.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb7469c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb7400d2710 0x7fb7400b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb7469c7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400b3430 con 0x7fb7400d22b0 2024-03-06T07:23:28.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb7469c7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400b3710 con 0x7fb74015dfe0 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb7469c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7400b31e0 con 0x7fb7400d2710 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb73f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb7400d22b0 0x7fb7400b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49196/0 (socket says 172.21.15.122:49196) 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.517+0000 7fb73f7fe640 1 -- 172.21.15.122:0/331365268 learned_addr learned my addr 172.21.15.122:0/331365268 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb744f64640 1 --2- 172.21.15.122:0/331365268 >> v2:172.21.15.169:3300/0 conn(0x7fb7400d2710 0x7fb7400b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb73effd640 1 -- 172.21.15.122:0/331365268 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613428735 0 0) 0x7fb7400b3430 con 0x7fb7400d22b0 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb73effd640 1 -- 172.21.15.122:0/331365268 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb728003520 con 0x7fb7400d22b0 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb73effd640 1 -- 172.21.15.122:0/331365268 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879065592 0 0) 0x7fb728003520 con 0x7fb7400d22b0 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb744f64640 1 -- 172.21.15.122:0/331365268 >> v1:172.21.15.169:6789/0 conn(0x7fb74015dfe0 legacy=0x7fb7400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb744f64640 1 -- 172.21.15.122:0/331365268 >> v1:172.21.15.122:6789/0 conn(0x7fb7400d22b0 legacy=0x7fb7400b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.518+0000 7fb744f64640 1 -- 172.21.15.122:0/331365268 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400b39f0 con 0x7fb7400d2710 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb744f64640 1 --2- 172.21.15.122:0/331365268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400b4900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb738009dc0 tx=0x7fb73802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e887ff94cf3643f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb73effd640 1 -- 172.21.15.122:0/331365268 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738039070 con 0x7fb7400d2710 2024-03-06T07:23:28.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb73effd640 1 -- 172.21.15.122:0/331365268 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73802c930 con 0x7fb7400d2710 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb73effd640 1 -- 172.21.15.122:0/331365268 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734001670 con 0x7fb7400d2710 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb7469c7640 1 -- 172.21.15.122:0/331365268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 msgr2=0x7fb7400b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb7469c7640 1 --2- 172.21.15.122:0/331365268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400b4900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb738009dc0 tx=0x7fb73802be00 comp rx=0 tx=0).stop 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb7469c7640 1 -- 172.21.15.122:0/331365268 shutdown_connections 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb7469c7640 1 --2- 172.21.15.122:0/331365268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400b4900 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb7469c7640 1 -- 172.21.15.122:0/331365268 >> 172.21.15.122:0/331365268 conn(0x7fb7400d3cd0 msgr2=0x7fb7400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.519+0000 7fb7469c7640 1 -- 172.21.15.122:0/331365268 shutdown_connections 2024-03-06T07:23:28.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.520+0000 7fb7469c7640 1 -- 172.21.15.122:0/331365268 wait complete. 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 Processor -- start 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 -- start start 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 0x7fb7400b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb74015dfe0 0x7fb7400ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400cfa00 con 0x7fb7400d22b0 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400b3d60 con 0x7fb7400d2710 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.521+0000 7fb7469c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400d1c70 con 0x7fb74015dfe0 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 0x7fb7400b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 0x7fb7400b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57400/0 (socket says 172.21.15.122:57400) 2024-03-06T07:23:28.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 -- 172.21.15.122:0/1245911801 learned_addr learned my addr 172.21.15.122:0/1245911801 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73f7fe640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb744f64640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb74015dfe0 0x7fb7400ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 -- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb74015dfe0 msgr2=0x7fb7400ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb74015dfe0 0x7fb7400ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 -- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 msgr2=0x7fb7400cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 -- 172.21.15.122:0/1245911801 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400b71b0 con 0x7fb7400d22b0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb744f64640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb74015dfe0 0x7fb7400ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73ffff640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 0x7fb7400b68d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb73000c400 tx=0x7fb73000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730017070 con 0x7fb7400d22b0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7300052d0 con 0x7fb7400d22b0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7400b7490 con 0x7fb7400d22b0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730003830 con 0x7fb7400d22b0 2024-03-06T07:23:28.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.522+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7400b7a50 con 0x7fb7400d22b0 2024-03-06T07:23:28.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.523+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74004f330 con 0x7fb7400d22b0 2024-03-06T07:23:28.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.524+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb73001a070 con 0x7fb7400d22b0 2024-03-06T07:23:28.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.524+0000 7fb73cff9640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb72c06b4d0 0x7fb72c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.525+0000 7fb73f7fe640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb72c06b4d0 0x7fb72c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.525+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb730089620 con 0x7fb7400d22b0 2024-03-06T07:23:28.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.525+0000 7fb73f7fe640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb72c06b4d0 0x7fb72c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb7401f6fb0 tx=0x7fb734009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.529+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb730058400 con 0x7fb7400d22b0 2024-03-06T07:23:28.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.880+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb74015d250 con 0x7fb7400d22b0 2024-03-06T07:23:28.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.881+0000 7fb73cff9640 1 -- 172.21.15.122:0/1245911801 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fb730057d00 con 0x7fb7400d22b0 2024-03-06T07:23:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:28.883 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:52.342190+0000","modified":"2024-03-06T07:22:55.380259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/2892378792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2892378792},{"type":"v1","addr":"172.21.15.122:6835","nonce":2892378792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5262,"qdb_cluster":[5262]},"id":2}]} 2024-03-06T07:23:28.884 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 15 2024-03-06T07:23:28.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb72c06b4d0 msgr2=0x7fb72c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb72c06b4d0 0x7fb72c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb7401f6fb0 tx=0x7fb734009040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 msgr2=0x7fb7400b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 0x7fb7400b68d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb73000c400 tx=0x7fb73000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 shutdown_connections 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb72c06b4d0 0x7fb72c06d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb74015dfe0 0x7fb7400ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7400d2710 0x7fb7400cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 --2- 172.21.15.122:0/1245911801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7400d22b0 0x7fb7400b68d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 >> 172.21.15.122:0/1245911801 conn(0x7fb7400d3cd0 msgr2=0x7fb7400b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 shutdown_connections 2024-03-06T07:23:28.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.883+0000 7fb7469c7640 1 -- 172.21.15.122:0/1245911801 wait complete. 2024-03-06T07:23:28.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:28.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.995+0000 7f379b7c7640 1 Processor -- start 2024-03-06T07:23:28.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.995+0000 7f379b7c7640 1 -- start start 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f379b7c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f379406b600 0x7f379415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f379b7c7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379415ea30 con 0x7f379406b220 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f379b7c7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379415ed10 con 0x7f37940ce920 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f379b7c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3794161b70 con 0x7f379406b600 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f3799563640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f379406b220 0x7f3794071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49204/0 (socket says 172.21.15.122:49204) 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f3799563640 1 -- 172.21.15.122:0/3488684999 learned_addr learned my addr 172.21.15.122:0/3488684999 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622962169 0 0) 0x7f379415ea30 con 0x7f379406b220 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003520 con 0x7f379406b220 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f3799d64640 1 --2- 172.21.15.122:0/3488684999 >> v2:172.21.15.169:3300/0 conn(0x7f379406b600 0x7f379415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344078664 0 0) 0x7f3780003520 con 0x7f379406b220 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 >> v2:172.21.15.169:3300/0 conn(0x7f379406b600 msgr2=0x7f379415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 --2- 172.21.15.122:0/3488684999 >> v2:172.21.15.169:3300/0 conn(0x7f379406b600 0x7f379415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 >> v1:172.21.15.169:6789/0 conn(0x7f37940ce920 legacy=0x7f3794162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379415eff0 con 0x7f379406b220 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f3790001670 con 0x7f379406b220 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 >> v1:172.21.15.122:6789/0 conn(0x7f379406b220 legacy=0x7f3794071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f3780003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3780004800 0x7f3780004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37800051c0 0x7f378000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3794161b70 con 0x7f37800039b0 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378000c090 con 0x7f3780004800 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.996+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378000c290 con 0x7f37800051c0 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799563640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f3780003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3798d62640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3780004800 0x7f3780004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799d64640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37800051c0 0x7f378000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799563640 1 -- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37800051c0 msgr2=0x7f378000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799563640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37800051c0 0x7f378000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799563640 1 -- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3780004800 msgr2=0x7f3780004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799563640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3780004800 0x7f3780004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799563640 1 -- 172.21.15.122:0/3488684999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780003520 con 0x7f37800039b0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799d64640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37800051c0 0x7f378000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.997+0000 7f3799d64640 1 -- 172.21.15.122:0/3488684999 reap_dead start 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f3799563640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f3780003df0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f379000c240 tx=0x7f3780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf2841f61b90f39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790007d10 con 0x7f37800039b0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790030070 con 0x7f37800039b0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f378bfff640 1 -- 172.21.15.122:0/3488684999 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790005d50 con 0x7f37800039b0 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f379b7c7640 1 -- 172.21.15.122:0/3488684999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 msgr2=0x7f3780003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f3780003df0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f379000c240 tx=0x7f3780005830 comp rx=0 tx=0).stop 2024-03-06T07:23:29.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f379b7c7640 1 -- 172.21.15.122:0/3488684999 shutdown_connections 2024-03-06T07:23:29.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3488684999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f3780003df0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.998+0000 7f379b7c7640 1 -- 172.21.15.122:0/3488684999 >> 172.21.15.122:0/3488684999 conn(0x7f379406c0c0 msgr2=0x7f379406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.999+0000 7f379b7c7640 1 -- 172.21.15.122:0/3488684999 shutdown_connections 2024-03-06T07:23:29.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:28.999+0000 7f379b7c7640 1 -- 172.21.15.122:0/3488684999 wait complete. 2024-03-06T07:23:29.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 Processor -- start 2024-03-06T07:23:29.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 -- start start 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f37941eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f379414a730 0x7f3794071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37941e9a30 0x7f37941e9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940ce380 con 0x7f37800039b0 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940ce200 con 0x7f37941e9a30 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.000+0000 7f379b7c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379415e450 con 0x7f379414a730 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f37941eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f37941eb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57416/0 (socket says 172.21.15.122:57416) 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 -- 172.21.15.122:0/3519962589 learned_addr learned my addr 172.21.15.122:0/3519962589 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3798d62640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f379414a730 0x7f3794071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799d64640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37941e9a30 0x7f37941e9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 -- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f379414a730 msgr2=0x7f3794071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f379414a730 0x7f3794071bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 -- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37941e9a30 msgr2=0x7f37941e9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37941e9a30 0x7f37941e9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 -- 172.21.15.122:0/3519962589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794147d70 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3798d62640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f379414a730 0x7f3794071bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3799563640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f37941eb1f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f379002c4e0 tx=0x7f379415eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790045070 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790000ed0 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3794148010 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790007d10 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.001+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379414ac30 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.002+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379404f330 con 0x7f37800039b0 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.003+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3790004bf0 con 0x7f37800039b0 2024-03-06T07:23:29.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.004+0000 7f3789ffb640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f377806b500 0x7f377806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.004+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37900aad50 con 0x7f37800039b0 2024-03-06T07:23:29.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.004+0000 7f3798d62640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f377806b500 0x7f377806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.004+0000 7f3798d62640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f377806b500 0x7f377806d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f37941f7f30 tx=0x7f3784009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.008+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37900ae030 con 0x7f37800039b0 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.189+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37941618c0 con 0x7f37800039b0 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.190+0000 7f3789ffb640 1 -- 172.21.15.122:0/3519962589 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f3790079580 con 0x7f37800039b0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:22:53.997648+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:50.561315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:51.306826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f377806b500 msgr2=0x7f377806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f377806b500 0x7f377806d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f37941f7f30 tx=0x7f3784009040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 msgr2=0x7f37941eb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f37941eb1f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f379002c4e0 tx=0x7f379415eee0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 shutdown_connections 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f377806b500 0x7f377806d9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37941e9a30 0x7f37941e9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f379414a730 0x7f3794071bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 --2- 172.21.15.122:0/3519962589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37800039b0 0x7f37941eb1f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 >> 172.21.15.122:0/3519962589 conn(0x7f379406c0c0 msgr2=0x7f379406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 shutdown_connections 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.192+0000 7f379b7c7640 1 -- 172.21.15.122:0/3519962589 wait complete. 2024-03-06T07:23:29.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:29.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.306+0000 7fc992981640 1 Processor -- start 2024-03-06T07:23:29.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.306+0000 7fc992981640 1 -- start start 2024-03-06T07:23:29.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc992981640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc98c15e000 0x7fc98c0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc992981640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc98c0cf7d0 0x7fc98c0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc992981640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c0d25e0 con 0x7fc98c0d4810 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc992981640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0c7c50 con 0x7fc98c0cf7d0 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc992981640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0d2880 con 0x7fc98c15e000 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc990f1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc98c0cf7d0 0x7fc98c0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc990f1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc98c0cf7d0 0x7fc98c0d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57418/0 (socket says 172.21.15.122:57418) 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.307+0000 7fc990f1e640 1 -- 172.21.15.122:0/3737428537 learned_addr learned my addr 172.21.15.122:0/3737428537 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.308+0000 7fc98bfff640 1 --2- 172.21.15.122:0/3737428537 >> v2:172.21.15.169:3300/0 conn(0x7fc98c15e000 0x7fc98c0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.308+0000 7fc990f1e640 1 -- 172.21.15.122:0/3737428537 >> v2:172.21.15.169:3300/0 conn(0x7fc98c15e000 msgr2=0x7fc98c0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.308+0000 7fc990f1e640 1 --2- 172.21.15.122:0/3737428537 >> v2:172.21.15.169:3300/0 conn(0x7fc98c15e000 0x7fc98c0d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.308+0000 7fc990f1e640 1 -- 172.21.15.122:0/3737428537 >> v1:172.21.15.169:6789/0 conn(0x7fc98c0d4810 legacy=0x7fc98c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.308+0000 7fc990f1e640 1 -- 172.21.15.122:0/3737428537 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0d2b40 con 0x7fc98c0cf7d0 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc990f1e640 1 --2- 172.21.15.122:0/3737428537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0cf7d0 0x7fc98c0d0bc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc98c08c360 tx=0x7fc98402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851c1d005cf9b614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc98affd640 1 -- 172.21.15.122:0/3737428537 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98402cb20 con 0x7fc98c0cf7d0 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc98affd640 1 -- 172.21.15.122:0/3737428537 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98402ccc0 con 0x7fc98c0cf7d0 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc98affd640 1 -- 172.21.15.122:0/3737428537 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984034610 con 0x7fc98c0cf7d0 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc992981640 1 -- 172.21.15.122:0/3737428537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0cf7d0 msgr2=0x7fc98c0d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc992981640 1 --2- 172.21.15.122:0/3737428537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0cf7d0 0x7fc98c0d0bc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc98c08c360 tx=0x7fc98402bf80 comp rx=0 tx=0).stop 2024-03-06T07:23:29.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc992981640 1 -- 172.21.15.122:0/3737428537 shutdown_connections 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc992981640 1 --2- 172.21.15.122:0/3737428537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0cf7d0 0x7fc98c0d0bc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc992981640 1 --2- 172.21.15.122:0/3737428537 >> v2:172.21.15.169:3300/0 conn(0x7fc98c15e000 0x7fc98c0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.309+0000 7fc992981640 1 -- 172.21.15.122:0/3737428537 >> 172.21.15.122:0/3737428537 conn(0x7fc98c0d3cf0 msgr2=0x7fc98c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.310+0000 7fc992981640 1 -- 172.21.15.122:0/3737428537 shutdown_connections 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.310+0000 7fc992981640 1 -- 172.21.15.122:0/3737428537 wait complete. 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 Processor -- start 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 -- start start 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc98c0cf7d0 0x7fc98c0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 0x7fc98c152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc98c15e000 0x7fc98c1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c15e3e0 con 0x7fc98c0d4810 2024-03-06T07:23:29.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0d22d0 con 0x7fc98c15e000 2024-03-06T07:23:29.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.311+0000 7fc992981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0c87f0 con 0x7fc98c0cf7d0 2024-03-06T07:23:29.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 0x7fc98c152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 0x7fc98c152f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57432/0 (socket says 172.21.15.122:57432) 2024-03-06T07:23:29.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 -- 172.21.15.122:0/1270364169 learned_addr learned my addr 172.21.15.122:0/1270364169 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc990f1e640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc98c15e000 0x7fc98c1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98bfff640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc98c0cf7d0 0x7fc98c0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 -- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc98c0cf7d0 msgr2=0x7fc98c0b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc98c0cf7d0 0x7fc98c0b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 -- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc98c15e000 msgr2=0x7fc98c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc98c15e000 0x7fc98c1534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 -- 172.21.15.122:0/1270364169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c150f80 con 0x7fc98c0d4810 2024-03-06T07:23:29.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc98b7fe640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 0x7fc98c152f70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc98000c430 tx=0x7fc98000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011790 con 0x7fc98c0d4810 2024-03-06T07:23:29.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980017070 con 0x7fc98c0d4810 2024-03-06T07:23:29.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011b70 con 0x7fc98c0d4810 2024-03-06T07:23:29.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c151260 con 0x7fc98c0d4810 2024-03-06T07:23:29.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.312+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c151820 con 0x7fc98c0d4810 2024-03-06T07:23:29.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.313+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c06c530 con 0x7fc98c0d4810 2024-03-06T07:23:29.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.314+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc980011d10 con 0x7fc98c0d4810 2024-03-06T07:23:29.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.314+0000 7fc988ff9640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc97806b4d0 0x7fc97806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.314+0000 7fc98bfff640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc97806b4d0 0x7fc97806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.314+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc980089e80 con 0x7fc98c0d4810 2024-03-06T07:23:29.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.315+0000 7fc98bfff640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc97806b4d0 0x7fc97806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00aa20 tx=0x7fc97c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.319+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc98c06c530 con 0x7fc98c0d4810 2024-03-06T07:23:29.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.536+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc98c04f330 con 0x7fc98c0d4810 2024-03-06T07:23:29.539 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:23:29.537+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:29.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.540+0000 7fc988ff9640 1 -- 172.21.15.122:0/1270364169 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc980058740 con 0x7fc98c0d4810 2024-03-06T07:23:29.542 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:29.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc97806b4d0 msgr2=0x7fc97806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc97806b4d0 0x7fc97806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00aa20 tx=0x7fc97c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 msgr2=0x7fc98c152f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 0x7fc98c152f70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc98000c430 tx=0x7fc98000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 shutdown_connections 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc97806b4d0 0x7fc97806d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc98c15e000 0x7fc98c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc98c0d4810 0x7fc98c152f70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 --2- 172.21.15.122:0/1270364169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc98c0cf7d0 0x7fc98c0b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 >> 172.21.15.122:0/1270364169 conn(0x7fc98c0d3cf0 msgr2=0x7fc98c06bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.543+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 shutdown_connections 2024-03-06T07:23:29.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.544+0000 7fc992981640 1 -- 172.21.15.122:0/1270364169 wait complete. 2024-03-06T07:23:29.557 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:29.591 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:23:29.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 Processor -- start 2024-03-06T07:23:29.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 -- start start 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f41301586e0 0x7f41300aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f41300cffd0 0x7f413006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41300d03f0 con 0x7f41300cffd0 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4130158ac0 con 0x7f41301586e0 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.660+0000 7f4136d7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413015b2a0 con 0x7f41300aef80 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f41300cffd0 0x7f413006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f41300cffd0 0x7f413006da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57456/0 (socket says 172.21.15.122:57456) 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 -- 172.21.15.122:0/1406731803 learned_addr learned my addr 172.21.15.122:0/1406731803 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f412ffff640 1 --2- 172.21.15.122:0/1406731803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4134b16640 1 --2- 172.21.15.122:0/1406731803 >> v2:172.21.15.169:3300/0 conn(0x7f41301586e0 0x7f41300aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 -- 172.21.15.122:0/1406731803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 msgr2=0x7f41300cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 --2- 172.21.15.122:0/1406731803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 -- 172.21.15.122:0/1406731803 >> v2:172.21.15.169:3300/0 conn(0x7f41301586e0 msgr2=0x7f41300aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 --2- 172.21.15.122:0/1406731803 >> v2:172.21.15.169:3300/0 conn(0x7f41301586e0 0x7f41300aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.661+0000 7f4135317640 1 -- 172.21.15.122:0/1406731803 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413015b540 con 0x7f41300cffd0 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4135317640 1 --2- 172.21.15.122:0/1406731803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41300cffd0 0x7f413006da20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4130070d10 tx=0x7f412002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e71b2e65ba702f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f412f7fe640 1 -- 172.21.15.122:0/1406731803 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120035020 con 0x7f41300cffd0 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f412f7fe640 1 -- 172.21.15.122:0/1406731803 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4120031870 con 0x7f41300cffd0 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f412f7fe640 1 -- 172.21.15.122:0/1406731803 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120031c40 con 0x7f41300cffd0 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 -- 172.21.15.122:0/1406731803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41300cffd0 msgr2=0x7f413006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 --2- 172.21.15.122:0/1406731803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41300cffd0 0x7f413006da20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4130070d10 tx=0x7f412002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 -- 172.21.15.122:0/1406731803 shutdown_connections 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 --2- 172.21.15.122:0/1406731803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41300cffd0 0x7f413006da20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 --2- 172.21.15.122:0/1406731803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 --2- 172.21.15.122:0/1406731803 >> v2:172.21.15.169:3300/0 conn(0x7f41301586e0 0x7f41300aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.662+0000 7f4136d7a640 1 -- 172.21.15.122:0/1406731803 >> 172.21.15.122:0/1406731803 conn(0x7f413006b190 msgr2=0x7f413006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.663+0000 7f4136d7a640 1 -- 172.21.15.122:0/1406731803 shutdown_connections 2024-03-06T07:23:29.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.663+0000 7f4136d7a640 1 -- 172.21.15.122:0/1406731803 wait complete. 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.664+0000 7f4136d7a640 1 Processor -- start 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.664+0000 7f4136d7a640 1 -- start start 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.664+0000 7f4136d7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.664+0000 7f4136d7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41300cffd0 0x7f413006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.664+0000 7f4136d7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 0x7f413014b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4136d7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413015bde0 con 0x7f41301586e0 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4136d7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413015bb00 con 0x7f41300cffd0 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4136d7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4130071350 con 0x7f41300aef80 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 0x7f413014b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 0x7f413014b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57470/0 (socket says 172.21.15.122:57470) 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 -- 172.21.15.122:0/2106679405 learned_addr learned my addr 172.21.15.122:0/2106679405 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4134b16640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 -- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 msgr2=0x7f41300ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f412ffff640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41300cffd0 0x7f413006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 -- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41300cffd0 msgr2=0x7f413006e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41300cffd0 0x7f413006e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 -- 172.21.15.122:0/2106679405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41300ce790 con 0x7f41301586e0 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4134b16640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f412ffff640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41300cffd0 0x7f413006e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4135317640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 0x7f413014b460 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f412002b950 tx=0x7f41200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120040070 con 0x7f41301586e0 2024-03-06T07:23:29.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4120030c50 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41300cd110 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120035040 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.665+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41300cd6d0 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.666+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41300732b0 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.667+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4120038020 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.667+0000 7f412d7fa640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f411c06b4d0 0x7f411c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.667+0000 7f4134b16640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f411c06b4d0 0x7f411c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.667+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f41200ac110 con 0x7f41301586e0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.668+0000 7f4134b16640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f411c06b4d0 0x7f411c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f413015a5b0 tx=0x7f412800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.672+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f412007af30 con 0x7f41301586e0 2024-03-06T07:23:29.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:29.880+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4130068ff0 con 0x7f41301586e0 2024-03-06T07:23:30.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.542+0000 7f412d7fa640 1 -- 172.21.15.122:0/2106679405 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f412007a830 con 0x7f41301586e0 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f411c06b4d0 msgr2=0x7f411c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f411c06b4d0 0x7f411c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f413015a5b0 tx=0x7f412800b040 comp rx=0 tx=0).stop 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 msgr2=0x7f413014b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 0x7f413014b460 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f412002b950 tx=0x7f41200046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 shutdown_connections 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f411c06b4d0 0x7f411c06d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41301586e0 0x7f413014b460 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41300cffd0 0x7f413006e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 --2- 172.21.15.122:0/2106679405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41300aef80 0x7f41300ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 >> 172.21.15.122:0/2106679405 conn(0x7f413006b190 msgr2=0x7f4130155780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 shutdown_connections 2024-03-06T07:23:30.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.545+0000 7f4136d7a640 1 -- 172.21.15.122:0/2106679405 wait complete. 2024-03-06T07:23:30.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:30.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.710+0000 7f9e81b7a640 1 Processor -- start 2024-03-06T07:23:30.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.710+0000 7f9e81b7a640 1 -- start start 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e81b7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 0x7f9e7c070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e81b7a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c15afe0 con 0x7f9e7c0ce0a0 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e81b7a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0cd000 con 0x7f9e7c15ac00 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e81b7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c071250 con 0x7f9e7c0708b0 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e7bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9e7c0ce0a0 0x7f9e7c0ce520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49212/0 (socket says 172.21.15.122:49212) 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e7bfff640 1 -- 172.21.15.122:0/827997678 learned_addr learned my addr 172.21.15.122:0/827997678 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706038096 0 0) 0x7f9e7c15afe0 con 0x7f9e7c0ce0a0 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e64003520 con 0x7f9e7c0ce0a0 2024-03-06T07:23:30.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.711+0000 7f9e7affd640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 0x7f9e7c070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059479063 0 0) 0x7f9e64003520 con 0x7f9e7c0ce0a0 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 msgr2=0x7f9e7c070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 0x7f9e7c070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 >> v1:172.21.15.169:6789/0 conn(0x7f9e7c15ac00 legacy=0x7f9e7c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0cd2e0 con 0x7f9e7c0ce0a0 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f9e740019f0 con 0x7f9e7c0ce0a0 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 >> v1:172.21.15.122:6789/0 conn(0x7f9e7c0ce0a0 legacy=0x7f9e7c0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 0x7f9e640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e64003f20 0x7f9e64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e64004d40 0x7f9e640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400baa0 con 0x7f9e64004d40 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400bc50 con 0x7f9e64003f20 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400be50 con 0x7f9e7c0708b0 2024-03-06T07:23:30.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e64004d40 0x7f9e640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7affd640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e64003f20 0x7f9e64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7b7fe640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 0x7f9e640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7bfff640 1 -- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 msgr2=0x7f9e640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e7c0708b0 0x7f9e640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7bfff640 1 -- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e64003f20 msgr2=0x7f9e64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e64003f20 0x7f9e64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.712+0000 7f9e7bfff640 1 -- 172.21.15.122:0/827997678 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64003520 con 0x7f9e64004d40 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.713+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e64004d40 0x7f9e640051c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400a830 tx=0x7f9e64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678929091da978f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.713+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400a480 con 0x7f9e64004d40 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.713+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e74034630 con 0x7f9e64004d40 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.713+0000 7f9e7a7fc640 1 -- 172.21.15.122:0/827997678 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74034a80 con 0x7f9e64004d40 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.713+0000 7f9e81b7a640 1 -- 172.21.15.122:0/827997678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e64004d40 msgr2=0x7f9e640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.713+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/827997678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e64004d40 0x7f9e640051c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400a830 tx=0x7f9e64005830 comp rx=0 tx=0).stop 2024-03-06T07:23:30.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.714+0000 7f9e7bfff640 1 -- 172.21.15.122:0/827997678 reap_dead start 2024-03-06T07:23:30.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.714+0000 7f9e81b7a640 1 -- 172.21.15.122:0/827997678 shutdown_connections 2024-03-06T07:23:30.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.714+0000 7f9e81b7a640 1 -- 172.21.15.122:0/827997678 >> 172.21.15.122:0/827997678 conn(0x7f9e7c0b1650 msgr2=0x7f9e7c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.714+0000 7f9e81b7a640 1 -- 172.21.15.122:0/827997678 shutdown_connections 2024-03-06T07:23:30.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.714+0000 7f9e81b7a640 1 -- 172.21.15.122:0/827997678 wait complete. 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.715+0000 7f9e81b7a640 1 Processor -- start 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.715+0000 7f9e81b7a640 1 -- start start 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e64004d40 0x7f9e7c1579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e7c153a40 0x7f9e7c153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 0x7f9e7c156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0cd8a0 con 0x7f9e7c156490 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0cdc10 con 0x7f9e7c153a40 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0cf5d0 con 0x7f9e64004d40 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 0x7f9e7c156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 0x7f9e7c156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57496/0 (socket says 172.21.15.122:57496) 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 -- 172.21.15.122:0/3090358233 learned_addr learned my addr 172.21.15.122:0/3090358233 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7b7fe640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e64004d40 0x7f9e7c1579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7affd640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e7c153a40 0x7f9e7c153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 -- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e64004d40 msgr2=0x7f9e7c1579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e64004d40 0x7f9e7c1579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 -- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e7c153a40 msgr2=0x7f9e7c153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e7c153a40 0x7f9e7c153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 -- 172.21.15.122:0/3090358233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c159620 con 0x7f9e7c156490 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7affd640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e7c153a40 0x7f9e7c153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7b7fe640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e64004d40 0x7f9e7c1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e7bfff640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 0x7f9e7c156910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9e74000f80 tx=0x7f9e7c15b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400a480 con 0x7f9e7c156490 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7403a040 con 0x7f9e7c156490 2024-03-06T07:23:30.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c159900 con 0x7f9e7c156490 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74001fd0 con 0x7f9e7c156490 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.716+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c159ec0 con 0x7f9e7c156490 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.717+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c0687d0 con 0x7f9e7c156490 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.718+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e74047020 con 0x7f9e7c156490 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.718+0000 7f9e63fff640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e6806b500 0x7f9e6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.718+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9e740abd90 con 0x7f9e7c156490 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.718+0000 7f9e7b7fe640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e6806b500 0x7f9e6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.719+0000 7f9e7b7fe640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e6806b500 0x7f9e6806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c003700 tx=0x7f9e6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.723+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e7407acd0 con 0x7f9e7c156490 2024-03-06T07:23:30.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:30.894+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9e7c15a0c0 con 0x7f9e7c156490 2024-03-06T07:23:31.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.543+0000 7f9e63fff640 1 -- 172.21.15.122:0/3090358233 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9e7c15a0c0 con 0x7f9e7c156490 2024-03-06T07:23:31.546 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:31.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e6806b500 msgr2=0x7f9e6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e6806b500 0x7f9e6806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c003700 tx=0x7f9e6c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 msgr2=0x7f9e7c156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 0x7f9e7c156910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9e74000f80 tx=0x7f9e7c15b1b0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 shutdown_connections 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e6806b500 0x7f9e6806d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e7c156490 0x7f9e7c156910 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e7c153a40 0x7f9e7c153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 --2- 172.21.15.122:0/3090358233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e64004d40 0x7f9e7c1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.546+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 >> 172.21.15.122:0/3090358233 conn(0x7f9e7c0b1650 msgr2=0x7f9e7c0ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.547+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 shutdown_connections 2024-03-06T07:23:31.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.547+0000 7f9e81b7a640 1 -- 172.21.15.122:0/3090358233 wait complete. 2024-03-06T07:23:31.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:31.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.665+0000 7f79a5fb5640 1 Processor -- start 2024-03-06T07:23:31.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.665+0000 7f79a5fb5640 1 -- start start 2024-03-06T07:23:31.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.666+0000 7f79a5fb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 0x7f79a00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.666+0000 7f79a5fb5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f79a00cdb30 0x7f79a00ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.666+0000 7f79a5fb5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a006d760 con 0x7f79a00b3a40 2024-03-06T07:23:31.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.666+0000 7f79a5fb5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a00d0830 con 0x7f79a00cdb30 2024-03-06T07:23:31.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.666+0000 7f79a5fb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a0144a70 con 0x7f79a0144690 2024-03-06T07:23:31.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f79a00cdb30 0x7f79a00ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 0x7f79a00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 0x7f79a00d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:52642/0 (socket says 172.21.15.122:52642) 2024-03-06T07:23:31.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 -- 172.21.15.122:0/418335377 learned_addr learned my addr 172.21.15.122:0/418335377 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:31.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 -- 172.21.15.122:0/418335377 >> v2:172.21.15.169:3300/0 conn(0x7f79a00cdb30 msgr2=0x7f79a00ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 --2- 172.21.15.122:0/418335377 >> v2:172.21.15.169:3300/0 conn(0x7f79a00cdb30 0x7f79a00ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 -- 172.21.15.122:0/418335377 >> v1:172.21.15.169:6789/0 conn(0x7f79a00b3a40 legacy=0x7f79a00cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799f7fe640 1 -- 172.21.15.122:0/418335377 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a015f660 con 0x7f79a0144690 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.667+0000 7f799ffff640 1 --2- 172.21.15.122:0/418335377 >> v2:172.21.15.169:3300/0 conn(0x7f79a00cdb30 0x7f79a00ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.668+0000 7f799f7fe640 1 --2- 172.21.15.122:0/418335377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 0x7f79a00d02f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7990009870 tx=0x7f799002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d00c007847d2122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.668+0000 7f799effd640 1 -- 172.21.15.122:0/418335377 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990034070 con 0x7f79a0144690 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.668+0000 7f799effd640 1 -- 172.21.15.122:0/418335377 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7990030840 con 0x7f79a0144690 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.668+0000 7f799effd640 1 -- 172.21.15.122:0/418335377 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990030c10 con 0x7f79a0144690 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 -- 172.21.15.122:0/418335377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 msgr2=0x7f79a00d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/418335377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 0x7f79a00d02f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7990009870 tx=0x7f799002b980 comp rx=0 tx=0).stop 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 -- 172.21.15.122:0/418335377 shutdown_connections 2024-03-06T07:23:31.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/418335377 >> v2:172.21.15.169:3300/0 conn(0x7f79a00cdb30 0x7f79a00ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/418335377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a0144690 0x7f79a00d02f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 -- 172.21.15.122:0/418335377 >> 172.21.15.122:0/418335377 conn(0x7f79a00d09b0 msgr2=0x7f79a00d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 -- 172.21.15.122:0/418335377 shutdown_connections 2024-03-06T07:23:31.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.669+0000 7f79a5fb5640 1 -- 172.21.15.122:0/418335377 wait complete. 2024-03-06T07:23:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.670+0000 7f79a5fb5640 1 Processor -- start 2024-03-06T07:23:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.670+0000 7f79a5fb5640 1 -- start start 2024-03-06T07:23:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 0x7f79a013cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a00cdb30 0x7f79a01384e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a0144690 0x7f79a013e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a015f280 con 0x7f79a00b3a40 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a006dd00 con 0x7f79a0144690 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a015f400 con 0x7f79a00cdb30 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 0x7f79a013cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 0x7f79a013cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57504/0 (socket says 172.21.15.122:57504) 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 -- 172.21.15.122:0/1696584594 learned_addr learned my addr 172.21.15.122:0/1696584594 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799ffff640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a0144690 0x7f79a013e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f7996ffd640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a00cdb30 0x7f79a01384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 -- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a00cdb30 msgr2=0x7f79a01384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a00cdb30 0x7f79a01384e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 -- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a0144690 msgr2=0x7f79a013e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a0144690 0x7f79a013e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 -- 172.21.15.122:0/1696584594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a013d060 con 0x7f79a00b3a40 2024-03-06T07:23:31.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799ffff640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a0144690 0x7f79a013e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799f7fe640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 0x7f79a013cb20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f799002b9c0 tx=0x7f79900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990034040 con 0x7f79a00b3a40 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799002f5a0 con 0x7f79a00b3a40 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a013f3e0 con 0x7f79a00b3a40 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799002f970 con 0x7f79a00b3a40 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.671+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a013f9a0 con 0x7f79a00b3a40 2024-03-06T07:23:31.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.672+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a0069f80 con 0x7f79a00b3a40 2024-03-06T07:23:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.673+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f799004d060 con 0x7f79a00b3a40 2024-03-06T07:23:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.673+0000 7f799cff9640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798006b4d0 0x7f798006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.673+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f79900ae4b0 con 0x7f79a00b3a40 2024-03-06T07:23:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.673+0000 7f7996ffd640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798006b4d0 0x7f798006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.674+0000 7f7996ffd640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798006b4d0 0x7f798006d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f79a0072bc0 tx=0x7f7988009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.677+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79a0069f80 con 0x7f79a00b3a40 2024-03-06T07:23:31.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:31.849+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f79a004f330 con 0x7f79a00b3a40 2024-03-06T07:23:32.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.548+0000 7f799cff9640 1 -- 172.21.15.122:0/1696584594 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7990041070 con 0x7f79a00b3a40 2024-03-06T07:23:32.551 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.551+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798006b4d0 msgr2=0x7f798006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.551+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798006b4d0 0x7f798006d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f79a0072bc0 tx=0x7f7988009040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.551+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 msgr2=0x7f79a013cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 0x7f79a013cb20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f799002b9c0 tx=0x7f79900046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 shutdown_connections 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f798006b4d0 0x7f798006d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79a0144690 0x7f79a013e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79a00cdb30 0x7f79a01384e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 --2- 172.21.15.122:0/1696584594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79a00b3a40 0x7f79a013cb20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 >> 172.21.15.122:0/1696584594 conn(0x7f79a00d09b0 msgr2=0x7f79a00b52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 shutdown_connections 2024-03-06T07:23:32.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.552+0000 7f79a5fb5640 1 -- 172.21.15.122:0/1696584594 wait complete. 2024-03-06T07:23:32.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:32.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.668+0000 7f61d691e640 1 Processor -- start 2024-03-06T07:23:32.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.668+0000 7f61d691e640 1 -- start start 2024-03-06T07:23:32.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61d691e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d006afb0 0x7f61d0070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61d691e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61d00b39d0 0x7f61d006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61d691e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f61d015ee20 0x7f61d00cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61d691e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d0072ca0 con 0x7f61d00b39d0 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61d691e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d006b8d0 con 0x7f61d015ee20 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61d691e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00726e0 con 0x7f61d006afb0 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61cf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61d00b39d0 0x7f61d006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61cf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61d00b39d0 0x7f61d006b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57514/0 (socket says 172.21.15.122:57514) 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.669+0000 7f61cf7fe640 1 -- 172.21.15.122:0/1569952949 learned_addr learned my addr 172.21.15.122:0/1569952949 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61d4ebb640 1 --2- 172.21.15.122:0/1569952949 >> v2:172.21.15.169:3300/0 conn(0x7f61d015ee20 0x7f61d00cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cffff640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d006afb0 0x7f61d0070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cf7fe640 1 -- 172.21.15.122:0/1569952949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d006afb0 msgr2=0x7f61d0070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cf7fe640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d006afb0 0x7f61d0070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cf7fe640 1 -- 172.21.15.122:0/1569952949 >> v2:172.21.15.169:3300/0 conn(0x7f61d015ee20 msgr2=0x7f61d00cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cf7fe640 1 --2- 172.21.15.122:0/1569952949 >> v2:172.21.15.169:3300/0 conn(0x7f61d015ee20 0x7f61d00cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cf7fe640 1 -- 172.21.15.122:0/1569952949 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00729c0 con 0x7f61d00b39d0 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61d4ebb640 1 --2- 172.21.15.122:0/1569952949 >> v2:172.21.15.169:3300/0 conn(0x7f61d015ee20 0x7f61d00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cffff640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d006afb0 0x7f61d0070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.670+0000 7f61cf7fe640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d00b39d0 0x7f61d006b390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f61d006be20 tx=0x7f61c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc712e1d63533b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61ceffd640 1 -- 172.21.15.122:0/1569952949 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4035020 con 0x7f61d00b39d0 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61ceffd640 1 -- 172.21.15.122:0/1569952949 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c4031870 con 0x7f61d00b39d0 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61ceffd640 1 -- 172.21.15.122:0/1569952949 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4031c40 con 0x7f61d00b39d0 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 -- 172.21.15.122:0/1569952949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d00b39d0 msgr2=0x7f61d006b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d00b39d0 0x7f61d006b390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f61d006be20 tx=0x7f61c402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 -- 172.21.15.122:0/1569952949 shutdown_connections 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d00b39d0 0x7f61d006b390 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 --2- 172.21.15.122:0/1569952949 >> v2:172.21.15.169:3300/0 conn(0x7f61d015ee20 0x7f61d00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 --2- 172.21.15.122:0/1569952949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d006afb0 0x7f61d0070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 -- 172.21.15.122:0/1569952949 >> 172.21.15.122:0/1569952949 conn(0x7f61d00b26d0 msgr2=0x7f61d00b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 -- 172.21.15.122:0/1569952949 shutdown_connections 2024-03-06T07:23:32.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.671+0000 7f61d691e640 1 -- 172.21.15.122:0/1569952949 wait complete. 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.672+0000 7f61d691e640 1 Processor -- start 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 -- start start 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d006afb0 0x7f61d00aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d00b39d0 0x7f61d015d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 0x7f61d0154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d015f2b0 con 0x7f61d015ee20 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d015e900 con 0x7f61d006afb0 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d691e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d015f430 con 0x7f61d00b39d0 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d4ebb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 0x7f61d0154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d4ebb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 0x7f61d0154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57516/0 (socket says 172.21.15.122:57516) 2024-03-06T07:23:32.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.673+0000 7f61d4ebb640 1 -- 172.21.15.122:0/382565242 learned_addr learned my addr 172.21.15.122:0/382565242 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61cf7fe640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d00b39d0 0x7f61d015d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61cffff640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d006afb0 0x7f61d00aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d4ebb640 1 -- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d00b39d0 msgr2=0x7f61d015d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d4ebb640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d00b39d0 0x7f61d015d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d4ebb640 1 -- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d006afb0 msgr2=0x7f61d00aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d4ebb640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d006afb0 0x7f61d00aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d4ebb640 1 -- 172.21.15.122:0/382565242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00af5a0 con 0x7f61d015ee20 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d4ebb640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 0x7f61d0154c60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f61c800cf20 tx=0x7f61c800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c801a070 con 0x7f61d015ee20 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c800d840 con 0x7f61d015ee20 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d0160630 con 0x7f61d015ee20 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800dc30 con 0x7f61d015ee20 2024-03-06T07:23:32.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.674+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d0154300 con 0x7f61d015ee20 2024-03-06T07:23:32.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.675+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d0073420 con 0x7f61d015ee20 2024-03-06T07:23:32.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.676+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61c801d070 con 0x7f61d015ee20 2024-03-06T07:23:32.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.676+0000 7f61ccff9640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.676+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f61c808a680 con 0x7f61d015ee20 2024-03-06T07:23:32.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.676+0000 7f61cffff640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.677+0000 7f61cffff640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61d00cf4d0 tx=0x7f61c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.680+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61c8059240 con 0x7f61d015ee20 2024-03-06T07:23:32.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.851+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f61d015df50 con 0x7f61d015ee20 2024-03-06T07:23:32.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.854+0000 7f61ccff9640 1 -- 172.21.15.122:0/382565242 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f61c805d2e0 con 0x7f61d015ee20 2024-03-06T07:23:32.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 msgr2=0x7f61bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61d00cf4d0 tx=0x7f61c0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 msgr2=0x7f61d0154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 0x7f61d0154c60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f61c800cf20 tx=0x7f61c800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 shutdown_connections 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d015ee20 0x7f61d0154c60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61d00b39d0 0x7f61d015d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 --2- 172.21.15.122:0/382565242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61d006afb0 0x7f61d00aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 >> 172.21.15.122:0/382565242 conn(0x7f61d00b26d0 msgr2=0x7f61d006f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 shutdown_connections 2024-03-06T07:23:32.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.856+0000 7f61d691e640 1 -- 172.21.15.122:0/382565242 wait complete. 2024-03-06T07:23:32.869 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:23:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.967+0000 7f658f4c9640 1 Processor -- start 2024-03-06T07:23:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.967+0000 7f658f4c9640 1 -- start start 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658f4c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f658813dc10 0x7f65880b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658f4c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f65880b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658f4c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658815f580 con 0x7f658815e850 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658f4c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65880cf3f0 con 0x7f658813dc10 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658f4c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658815b210 con 0x7f658815bf30 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658d265640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f658813dc10 0x7f65880b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658ca64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f65880b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658d265640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f658813dc10 0x7f65880b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:42300/0 (socket says 172.21.15.122:42300) 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658d265640 1 -- 172.21.15.122:0/3069101851 learned_addr learned my addr 172.21.15.122:0/3069101851 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.968+0000 7f658da66640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f658815e850 0x7f658815eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:46344/0 (socket says 172.21.15.122:46344) 2024-03-06T07:23:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.969+0000 7f658d265640 1 -- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 msgr2=0x7f65880b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.969+0000 7f658d265640 1 --2- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f65880b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.969+0000 7f658d265640 1 -- 172.21.15.122:0/3069101851 >> v1:172.21.15.169:6789/0 conn(0x7f658815e850 legacy=0x7f658815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.969+0000 7f658d265640 1 -- 172.21.15.122:0/3069101851 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658815b450 con 0x7f658813dc10 2024-03-06T07:23:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.969+0000 7f658d265640 1 --2- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658813dc10 0x7f65880b2950 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6584009e50 tx=0x7f658402e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b3758f930a1671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f657ffff640 1 -- 172.21.15.122:0/3069101851 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658402ec80 con 0x7f658813dc10 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f657ffff640 1 -- 172.21.15.122:0/3069101851 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658402ee20 con 0x7f658813dc10 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f657ffff640 1 -- 172.21.15.122:0/3069101851 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584036610 con 0x7f658813dc10 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 -- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658813dc10 msgr2=0x7f65880b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 --2- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658813dc10 0x7f65880b2950 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6584009e50 tx=0x7f658402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 -- 172.21.15.122:0/3069101851 shutdown_connections 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 --2- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f65880b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 --2- 172.21.15.122:0/3069101851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658813dc10 0x7f65880b2950 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 -- 172.21.15.122:0/3069101851 >> 172.21.15.122:0/3069101851 conn(0x7f65880cf8a0 msgr2=0x7f65880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 -- 172.21.15.122:0/3069101851 shutdown_connections 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.970+0000 7f658f4c9640 1 -- 172.21.15.122:0/3069101851 wait complete. 2024-03-06T07:23:32.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.971+0000 7f658f4c9640 1 Processor -- start 2024-03-06T07:23:32.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.971+0000 7f658f4c9640 1 -- start start 2024-03-06T07:23:32.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658f4c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 0x7f658806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658f4c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f658806ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658f4c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658815e850 0x7f6588139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658f4c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880cd7a0 con 0x7f658813dc10 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658f4c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658813dff0 con 0x7f658815e850 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658f4c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65880cd920 con 0x7f658815bf30 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658d265640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 0x7f658806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658d265640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 0x7f658806e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57530/0 (socket says 172.21.15.122:57530) 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658d265640 1 -- 172.21.15.122:0/2043989051 learned_addr learned my addr 172.21.15.122:0/2043989051 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658ca64640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f658806ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.972+0000 7f658da66640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658815e850 0x7f6588139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658d265640 1 -- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 msgr2=0x7f658806ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658d265640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f658806ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658d265640 1 -- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658815e850 msgr2=0x7f6588139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658d265640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658815e850 0x7f6588139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658d265640 1 -- 172.21.15.122:0/2043989051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658806d2b0 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658d265640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 0x7f658806e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6584000f80 tx=0x7f6584004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584045070 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6584041640 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658806d590 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584009510 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.973+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658806db50 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.974+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658804f330 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.974+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6584041a30 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.975+0000 7f657dffb640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f656c06b500 0x7f656c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.975+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f65840ae370 con 0x7f658813dc10 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.975+0000 7f658ca64640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f656c06b500 0x7f656c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.975+0000 7f658ca64640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f656c06b500 0x7f656c06d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6588073090 tx=0x7f6578009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:32.979+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f658407ceb0 con 0x7f658813dc10 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.223+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f6588159520 con 0x7f658813dc10 2024-03-06T07:23:33.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.550+0000 7f657dffb640 1 -- 172.21.15.122:0/2043989051 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v114) v1 ==== 132+0+0 (secure 0 0 0) 0x7f6584080f50 con 0x7f658813dc10 2024-03-06T07:23:33.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f656c06b500 msgr2=0x7f656c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f656c06b500 0x7f656c06d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6588073090 tx=0x7f6578009190 comp rx=0 tx=0).stop 2024-03-06T07:23:33.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 msgr2=0x7f658806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 0x7f658806e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6584000f80 tx=0x7f6584004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 shutdown_connections 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f656c06b500 0x7f656c06d9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f658815e850 0x7f6588139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f658815bf30 0x7f658806ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 --2- 172.21.15.122:0/2043989051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f658813dc10 0x7f658806e350 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 >> 172.21.15.122:0/2043989051 conn(0x7f65880cf8a0 msgr2=0x7f65880b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 shutdown_connections 2024-03-06T07:23:33.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.552+0000 7f658f4c9640 1 -- 172.21.15.122:0/2043989051 wait complete. 2024-03-06T07:23:33.565 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:23:33.566 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:23:33.566 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:33.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:33.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.665+0000 7f4882c0e640 1 Processor -- start 2024-03-06T07:23:33.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.665+0000 7f4882c0e640 1 -- start start 2024-03-06T07:23:33.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.665+0000 7f4882c0e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f487c13dc10 0x7f487c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.665+0000 7f4882c0e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487c15f580 con 0x7f487c15bf30 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.665+0000 7f4882c0e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487c15b450 con 0x7f487c15ef60 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.665+0000 7f4882c0e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487c0cf3f0 con 0x7f487c13dc10 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f487c15bf30 0x7f487c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49222/0 (socket says 172.21.15.122:49222) 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487bfff640 1 -- 172.21.15.122:0/2923802634 learned_addr learned my addr 172.21.15.122:0/2923802634 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328198116 0 0) 0x7f487c15f580 con 0x7f487c15bf30 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4868003520 con 0x7f487c15bf30 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f48809aa640 1 --2- 172.21.15.122:0/2923802634 >> v2:172.21.15.169:3300/0 conn(0x7f487c13dc10 0x7f487c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1675795628 0 0) 0x7f4868003520 con 0x7f487c15bf30 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 >> v2:172.21.15.169:3300/0 conn(0x7f487c13dc10 msgr2=0x7f487c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 --2- 172.21.15.122:0/2923802634 >> v2:172.21.15.169:3300/0 conn(0x7f487c13dc10 0x7f487c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 >> v1:172.21.15.169:6789/0 conn(0x7f487c15ef60 legacy=0x7f487c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c15b730 con 0x7f487c15bf30 2024-03-06T07:23:33.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4864001670 con 0x7f487c15bf30 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 >> v1:172.21.15.122:6789/0 conn(0x7f487c15bf30 legacy=0x7f487c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48680039b0 0x7f4868003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4868004800 0x7f4868004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f486800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cf3f0 con 0x7f48680052e0 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486800c1b0 con 0x7f48680039b0 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486800c3b0 con 0x7f4868004800 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f48811ab640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f486800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f48809aa640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48680039b0 0x7f4868003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.666+0000 7f487bfff640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4868004800 0x7f4868004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 -- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4868004800 msgr2=0x7f4868004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4868004800 0x7f4868004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 -- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48680039b0 msgr2=0x7f4868003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48680039b0 0x7f4868003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 -- 172.21.15.122:0/2923802634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868003520 con 0x7f48680052e0 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f487bfff640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4868004800 0x7f4868004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 -- 172.21.15.122:0/2923802634 reap_dead start 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f48811ab640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f486800b9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f486c001720 tx=0x7f4868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69a2cb90d77da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c001370 con 0x7f48680052e0 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c0052d0 con 0x7f48680052e0 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.667+0000 7f487b7fe640 1 -- 172.21.15.122:0/2923802634 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c003810 con 0x7f48680052e0 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 -- 172.21.15.122:0/2923802634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 msgr2=0x7f486800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f486800b9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f486c001720 tx=0x7f4868005830 comp rx=0 tx=0).stop 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 -- 172.21.15.122:0/2923802634 shutdown_connections 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 --2- 172.21.15.122:0/2923802634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f486800b9c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 -- 172.21.15.122:0/2923802634 >> 172.21.15.122:0/2923802634 conn(0x7f487c0cf8a0 msgr2=0x7f487c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 -- 172.21.15.122:0/2923802634 shutdown_connections 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.668+0000 7f4882c0e640 1 -- 172.21.15.122:0/2923802634 wait complete. 2024-03-06T07:23:33.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.669+0000 7f4882c0e640 1 Processor -- start 2024-03-06T07:23:33.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.669+0000 7f4882c0e640 1 -- start start 2024-03-06T07:23:33.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f4882c0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f487c153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f4882c0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f487c1f1bb0 0x7f487c143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f4882c0e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f487c143c60 0x7f487c13e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f4882c0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c15e850 con 0x7f48680052e0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f4882c0e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c15b210 con 0x7f487c143c60 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f4882c0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c15e9d0 con 0x7f487c1f1bb0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f487c153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f487c153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57538/0 (socket says 172.21.15.122:57538) 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 -- 172.21.15.122:0/3837652423 learned_addr learned my addr 172.21.15.122:0/3837652423 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 -- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f487c1f1bb0 msgr2=0x7f487c143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f487bfff640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f487c1f1bb0 0x7f487c143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f487c1f1bb0 0x7f487c143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 -- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f487c143c60 msgr2=0x7f487c13e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f487c143c60 0x7f487c13e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 -- 172.21.15.122:0/3837652423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c13f190 con 0x7f48680052e0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.670+0000 7f48809aa640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f487c153e30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4874004630 tx=0x7f4874007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.671+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874015420 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.671+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487401a070 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.671+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c13f470 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.671+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874015800 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.671+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c13fa30 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.671+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c04f330 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.672+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48740159a0 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.672+0000 7f48797fa640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f485c06b500 0x7f485c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.672+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f487408a930 con 0x7f48680052e0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.673+0000 7f487bfff640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f485c06b500 0x7f485c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.673+0000 7f487bfff640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f485c06b500 0x7f485c06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f487c13d0e0 tx=0x7f4864000ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.677+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4874024020 con 0x7f48680052e0 2024-03-06T07:23:33.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.847+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f487c144080 con 0x7f48680052e0 2024-03-06T07:23:33.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.848+0000 7f48797fa640 1 -- 172.21.15.122:0/3837652423 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13022 (secure 0 0 0) 0x7f4874059ab0 con 0x7f48680052e0 2024-03-06T07:23:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:33.850 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":36,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:32.545027+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6835/2892378792":"2024-03-07T07:23:29.538236+0000","172.21.15.122:6834/2892378792":"2024-03-07T07:23:29.538236+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:33.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f485c06b500 msgr2=0x7f485c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f485c06b500 0x7f485c06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f487c13d0e0 tx=0x7f4864000ef0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 msgr2=0x7f487c153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f487c153e30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4874004630 tx=0x7f4874007700 comp rx=0 tx=0).stop 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 shutdown_connections 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f485c06b500 0x7f485c06d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f487c143c60 0x7f487c13e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f487c1f1bb0 0x7f487c143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 --2- 172.21.15.122:0/3837652423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48680052e0 0x7f487c153e30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 >> 172.21.15.122:0/3837652423 conn(0x7f487c0cf8a0 msgr2=0x7f487c070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 shutdown_connections 2024-03-06T07:23:33.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.850+0000 7f4882c0e640 1 -- 172.21.15.122:0/3837652423 wait complete. 2024-03-06T07:23:33.864 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:33.864 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.964+0000 7f864768f640 1 Processor -- start 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.964+0000 7f864768f640 1 -- start start 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f864768f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 0x7f86400b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f864768f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864006b710 con 0x7f86400b55e0 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f864768f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864006b150 con 0x7f8640066d90 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f864768f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864006b430 con 0x7f86400cf9a0 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f8645c2c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f86400b55e0 0x7f86400ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49230/0 (socket says 172.21.15.122:49230) 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f8645c2c640 1 -- 172.21.15.122:0/3209442819 learned_addr learned my addr 172.21.15.122:0/3209442819 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:33.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.965+0000 7f864542b640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 0x7f86400b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155269679 0 0) 0x7f864006b710 con 0x7f86400b55e0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c003520 con 0x7f86400b55e0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2434751235 0 0) 0x7f862c003520 con 0x7f86400b55e0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 msgr2=0x7f86400b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 0x7f86400b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 >> v1:172.21.15.169:6789/0 conn(0x7f8640066d90 legacy=0x7f86400b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400b0970 con 0x7f86400b55e0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f8638001670 con 0x7f86400b55e0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 >> v1:172.21.15.122:6789/0 conn(0x7f86400b55e0 legacy=0x7f86400ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 0x7f862c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f862c003f20 0x7f862c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f862c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864006b430 con 0x7f862c004d40 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00ba20 con 0x7f862c003f20 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00bc20 con 0x7f86400cf9a0 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8645c2c640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f862c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8644c2a640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f862c003f20 0x7f862c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f864542b640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 0x7f862c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8645c2c640 1 -- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 msgr2=0x7f862c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8645c2c640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400cf9a0 0x7f862c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8645c2c640 1 -- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f862c003f20 msgr2=0x7f862c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8645c2c640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f862c003f20 0x7f862c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8645c2c640 1 -- 172.21.15.122:0/3209442819 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c003520 con 0x7f862c004d40 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.966+0000 7f8644c2a640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f862c003f20 0x7f862c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.967+0000 7f8645c2c640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f862c0051c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f86400b3140 tx=0x7f862c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221caae787be83b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.967+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638007d50 con 0x7f862c004d40 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.967+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8638030070 con 0x7f862c004d40 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.967+0000 7f8637fff640 1 -- 172.21.15.122:0/3209442819 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638005d70 con 0x7f862c004d40 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.967+0000 7f864768f640 1 -- 172.21.15.122:0/3209442819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 msgr2=0x7f862c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.967+0000 7f864768f640 1 --2- 172.21.15.122:0/3209442819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f862c0051c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f86400b3140 tx=0x7f862c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.968+0000 7f8645c2c640 1 -- 172.21.15.122:0/3209442819 reap_dead start 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.968+0000 7f864768f640 1 -- 172.21.15.122:0/3209442819 shutdown_connections 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.968+0000 7f864768f640 1 -- 172.21.15.122:0/3209442819 >> 172.21.15.122:0/3209442819 conn(0x7f86400b1d70 msgr2=0x7f86400b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.968+0000 7f864768f640 1 -- 172.21.15.122:0/3209442819 shutdown_connections 2024-03-06T07:23:33.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.968+0000 7f864768f640 1 -- 172.21.15.122:0/3209442819 wait complete. 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 Processor -- start 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 -- start start 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f86400cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400caac0 0x7f86400caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86400ad300 0x7f86400ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86400cff00 con 0x7f862c004d40 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86400cfd80 con 0x7f86400ad300 2024-03-06T07:23:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.969+0000 7f864768f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400b5ab0 con 0x7f86400caac0 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f86400cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f86400cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57566/0 (socket says 172.21.15.122:57566) 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 -- 172.21.15.122:0/2110870941 learned_addr learned my addr 172.21.15.122:0/2110870941 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f8644c2a640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400caac0 0x7f86400caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f8645c2c640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86400ad300 0x7f86400ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 -- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400caac0 msgr2=0x7f86400caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400caac0 0x7f86400caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 -- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86400ad300 msgr2=0x7f86400ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86400ad300 0x7f86400ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 -- 172.21.15.122:0/2110870941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8640148040 con 0x7f862c004d40 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864542b640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f86400cd5a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f863c011e90 tx=0x7f863c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c015630 con 0x7f862c004d40 2024-03-06T07:23:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c0157d0 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86400ae090 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c015ba0 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.970+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86400ae650 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.971+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86400681e0 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.972+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f863c01d400 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.972+0000 7f8635ffb640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862406b500 0x7f862406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.972+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f863c08af60 con 0x7f862c004d40 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.972+0000 7f8644c2a640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862406b500 0x7f862406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.973+0000 7f8644c2a640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862406b500 0x7f862406d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8640147e70 tx=0x7f8630009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:33.977+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86400681e0 con 0x7f862c004d40 2024-03-06T07:23:34.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.148+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f864009e3d0 con 0x7f862c004d40 2024-03-06T07:23:34.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.149+0000 7f8635ffb640 1 -- 172.21.15.122:0/2110870941 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f863c019030 con 0x7f862c004d40 2024-03-06T07:23:34.151 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862406b500 msgr2=0x7f862406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862406b500 0x7f862406d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8640147e70 tx=0x7f8630009040 comp rx=0 tx=0).stop 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 msgr2=0x7f86400cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f86400cd5a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f863c011e90 tx=0x7f863c00f580 comp rx=0 tx=0).stop 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 shutdown_connections 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f862406b500 0x7f862406d9c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86400ad300 0x7f86400ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86400caac0 0x7f86400caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 --2- 172.21.15.122:0/2110870941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862c004d40 0x7f86400cd5a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.150+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 >> 172.21.15.122:0/2110870941 conn(0x7f86400b1d70 msgr2=0x7f864006dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.151+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 shutdown_connections 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.151+0000 7f864768f640 1 -- 172.21.15.122:0/2110870941 wait complete. 2024-03-06T07:23:34.164 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:34.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.262+0000 7f8e6737a640 1 Processor -- start 2024-03-06T07:23:34.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.262+0000 7f8e6737a640 1 -- start start 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e6737a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8e60144340 0x7f8e60162390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e6737a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8e60071f30 0x7f8e600723c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e6737a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e601628d0 con 0x7f8e6015e3a0 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e6737a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e600b4110 con 0x7f8e60071f30 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e6737a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e600b43f0 con 0x7f8e60144340 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e64915640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8e60071f30 0x7f8e600723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e64915640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8e60071f30 0x7f8e600723c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57580/0 (socket says 172.21.15.122:57580) 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e64915640 1 -- 172.21.15.122:0/219209377 learned_addr learned my addr 172.21.15.122:0/219209377 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.263+0000 7f8e65116640 1 --2- 172.21.15.122:0/219209377 >> v2:172.21.15.169:3300/0 conn(0x7f8e60144340 0x7f8e60162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e64915640 1 -- 172.21.15.122:0/219209377 >> v2:172.21.15.169:3300/0 conn(0x7f8e60144340 msgr2=0x7f8e60162390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e64915640 1 --2- 172.21.15.122:0/219209377 >> v2:172.21.15.169:3300/0 conn(0x7f8e60144340 0x7f8e60162390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e64915640 1 -- 172.21.15.122:0/219209377 >> v1:172.21.15.169:6789/0 conn(0x7f8e6015e3a0 legacy=0x7f8e6015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e64915640 1 -- 172.21.15.122:0/219209377 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e600b46d0 con 0x7f8e60071f30 2024-03-06T07:23:34.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e64915640 1 --2- 172.21.15.122:0/219209377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e600723c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009870 tx=0x7f8e5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6485b8effc4bf4bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e57fff640 1 -- 172.21.15.122:0/219209377 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50035020 con 0x7f8e60071f30 2024-03-06T07:23:34.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e57fff640 1 -- 172.21.15.122:0/219209377 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e50031870 con 0x7f8e60071f30 2024-03-06T07:23:34.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.264+0000 7f8e57fff640 1 -- 172.21.15.122:0/219209377 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50031c40 con 0x7f8e60071f30 2024-03-06T07:23:34.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 -- 172.21.15.122:0/219209377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 msgr2=0x7f8e600723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 --2- 172.21.15.122:0/219209377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e600723c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009870 tx=0x7f8e5002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:34.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 -- 172.21.15.122:0/219209377 shutdown_connections 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 --2- 172.21.15.122:0/219209377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e600723c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 --2- 172.21.15.122:0/219209377 >> v2:172.21.15.169:3300/0 conn(0x7f8e60144340 0x7f8e60162390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 -- 172.21.15.122:0/219209377 >> 172.21.15.122:0/219209377 conn(0x7f8e6006b3d0 msgr2=0x7f8e6006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 -- 172.21.15.122:0/219209377 shutdown_connections 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.265+0000 7f8e6737a640 1 -- 172.21.15.122:0/219209377 wait complete. 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.266+0000 7f8e6737a640 1 Processor -- start 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.266+0000 7f8e6737a640 1 -- start start 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e6737a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e6015d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e6737a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e60144340 0x7f8e60147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e6737a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e6015e3a0 0x7f8e600b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e6737a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6006dbc0 con 0x7f8e60071f30 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e6737a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e600b4c90 con 0x7f8e6015e3a0 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e6737a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6006dd40 con 0x7f8e60144340 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e6015d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e6015d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57588/0 (socket says 172.21.15.122:57588) 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 -- 172.21.15.122:0/2390983216 learned_addr learned my addr 172.21.15.122:0/2390983216 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e64915640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e60144340 0x7f8e60147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65917640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e6015e3a0 0x7f8e600b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 -- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e60144340 msgr2=0x7f8e60147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e60144340 0x7f8e60147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 -- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e6015e3a0 msgr2=0x7f8e600b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e6015e3a0 0x7f8e600b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 -- 172.21.15.122:0/2390983216 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e601f4990 con 0x7f8e60071f30 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.267+0000 7f8e65116640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e6015d6b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c007ef0 tx=0x7f8e5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.268+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c014620 con 0x7f8e60071f30 2024-03-06T07:23:34.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.268+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c019070 con 0x7f8e60071f30 2024-03-06T07:23:34.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.268+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e601f3d80 con 0x7f8e60071f30 2024-03-06T07:23:34.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.268+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c014a00 con 0x7f8e60071f30 2024-03-06T07:23:34.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.268+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e601f4340 con 0x7f8e60071f30 2024-03-06T07:23:34.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.269+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6004f330 con 0x7f8e60071f30 2024-03-06T07:23:34.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.269+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e5c014ba0 con 0x7f8e60071f30 2024-03-06T07:23:34.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.269+0000 7f8e55ffb640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e4406b4d0 0x7f8e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.269+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e5c08a840 con 0x7f8e60071f30 2024-03-06T07:23:34.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.269+0000 7f8e64915640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e4406b4d0 0x7f8e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.270+0000 7f8e64915640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e4406b4d0 0x7f8e4406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8e50036000 tx=0x7f8e5003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.274+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e5c0599c0 con 0x7f8e60071f30 2024-03-06T07:23:34.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.512+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f8e600aee60 con 0x7f8e60071f30 2024-03-06T07:23:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.551+0000 7f8e55ffb640 1 -- 172.21.15.122:0/2390983216 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v115) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8e5c0592c0 con 0x7f8e60071f30 2024-03-06T07:23:34.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e4406b4d0 msgr2=0x7f8e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e4406b4d0 0x7f8e4406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8e50036000 tx=0x7f8e5003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:34.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 msgr2=0x7f8e6015d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e6015d6b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c007ef0 tx=0x7f8e5c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 shutdown_connections 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e4406b4d0 0x7f8e4406d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e6015e3a0 0x7f8e600b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e60144340 0x7f8e60147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 --2- 172.21.15.122:0/2390983216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e60071f30 0x7f8e6015d6b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 >> 172.21.15.122:0/2390983216 conn(0x7f8e6006b3d0 msgr2=0x7f8e6006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 shutdown_connections 2024-03-06T07:23:34.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.554+0000 7f8e6737a640 1 -- 172.21.15.122:0/2390983216 wait complete. 2024-03-06T07:23:34.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:34.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.666+0000 7fa5ba642640 1 Processor -- start 2024-03-06T07:23:34.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.666+0000 7fa5ba642640 1 -- start start 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.667+0000 7fa5ba642640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa5b4071a70 0x7fa5b4071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.667+0000 7fa5ba642640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa5b40b42e0 0x7fa5b415c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.667+0000 7fa5ba642640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b4072390 con 0x7fa5b415ad90 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.667+0000 7fa5ba642640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b415c020 con 0x7fa5b4071a70 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.667+0000 7fa5ba642640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b40b4720 con 0x7fa5b40b42e0 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.667+0000 7fa5b3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa5b4071a70 0x7fa5b4071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b37fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa5b415ad90 0x7fa5b40b3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51378/0 (socket says 172.21.15.122:51378) 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b37fe640 1 -- 172.21.15.122:0/3559585156 learned_addr learned my addr 172.21.15.122:0/3559585156 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b8bdf640 1 --2- 172.21.15.122:0/3559585156 >> v2:172.21.15.169:3300/0 conn(0x7fa5b40b42e0 0x7fa5b415c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3559585156 >> v2:172.21.15.169:3300/0 conn(0x7fa5b40b42e0 msgr2=0x7fa5b415c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/3559585156 >> v2:172.21.15.169:3300/0 conn(0x7fa5b40b42e0 0x7fa5b415c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3559585156 >> v1:172.21.15.122:6789/0 conn(0x7fa5b415ad90 legacy=0x7fa5b40b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b2ffd640 1 -- 172.21.15.122:0/3559585156 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353906773 0 0) 0x7fa5b4072390 con 0x7fa5b415ad90 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3559585156 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b415b260 con 0x7fa5b4071a70 2024-03-06T07:23:34.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.668+0000 7fa5b8bdf640 1 --2- 172.21.15.122:0/3559585156 >> v2:172.21.15.169:3300/0 conn(0x7fa5b40b42e0 0x7fa5b415c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:34.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/3559585156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b4071e50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa5b406f050 tx=0x7fa5a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33e04f16c77754c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5b2ffd640 1 -- 172.21.15.122:0/3559585156 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40315a0 con 0x7fa5b4071a70 2024-03-06T07:23:34.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5b2ffd640 1 -- 172.21.15.122:0/3559585156 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4037070 con 0x7fa5b4071a70 2024-03-06T07:23:34.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5b2ffd640 1 -- 172.21.15.122:0/3559585156 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4031960 con 0x7fa5b4071a70 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 -- 172.21.15.122:0/3559585156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 msgr2=0x7fa5b4071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3559585156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b4071e50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa5b406f050 tx=0x7fa5a402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 -- 172.21.15.122:0/3559585156 shutdown_connections 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3559585156 >> v2:172.21.15.169:3300/0 conn(0x7fa5b40b42e0 0x7fa5b415c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3559585156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b4071e50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 -- 172.21.15.122:0/3559585156 >> 172.21.15.122:0/3559585156 conn(0x7fa5b40b2660 msgr2=0x7fa5b40b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 -- 172.21.15.122:0/3559585156 shutdown_connections 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.669+0000 7fa5ba642640 1 -- 172.21.15.122:0/3559585156 wait complete. 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.670+0000 7fa5ba642640 1 Processor -- start 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 -- start start 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b40cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b40b42e0 0x7fa5b41483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b415ad90 0x7fa5b41f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415bb00 con 0x7fa5b4071a70 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415b820 con 0x7fa5b40b42e0 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5ba642640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415c750 con 0x7fa5b415ad90 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b40cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b40cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35620/0 (socket says 172.21.15.122:35620) 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.671+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3034870378 learned_addr learned my addr 172.21.15.122:0/3034870378 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b40b42e0 0x7fa5b41483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b8bdf640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b415ad90 0x7fa5b41f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b415ad90 msgr2=0x7fa5b41f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b415ad90 0x7fa5b41f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b40b42e0 msgr2=0x7fa5b41483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b40b42e0 0x7fa5b41483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b3fff640 1 -- 172.21.15.122:0/3034870378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b415a390 con 0x7fa5b4071a70 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b8bdf640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b415ad90 0x7fa5b41f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b40cf5b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4000c00 tx=0x7fa5a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4032840 con 0x7fa5b4071a70 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4032a00 con 0x7fa5b4071a70 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b41f5e40 con 0x7fa5b4071a70 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4032df0 con 0x7fa5b4071a70 2024-03-06T07:23:34.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.672+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b41478f0 con 0x7fa5b4071a70 2024-03-06T07:23:34.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.673+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b40b1620 con 0x7fa5b4071a70 2024-03-06T07:23:34.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.674+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5a403b3f0 con 0x7fa5b4071a70 2024-03-06T07:23:34.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.674+0000 7fa5b0ff9640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5a006b4d0 0x7fa5a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.674+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa5a402f030 con 0x7fa5b4071a70 2024-03-06T07:23:34.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.674+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5a006b4d0 0x7fa5a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.674+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5a006b4d0 0x7fa5a006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa5b41f4f50 tx=0x7fa5a8007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.678+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5b40b1620 con 0x7fa5b4071a70 2024-03-06T07:23:34.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.851+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa5b40d07d0 con 0x7fa5b4071a70 2024-03-06T07:23:34.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.854+0000 7fa5b0ff9640 1 -- 172.21.15.122:0/3034870378 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa5a40b1020 con 0x7fa5b4071a70 2024-03-06T07:23:34.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5a006b4d0 msgr2=0x7fa5a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5a006b4d0 0x7fa5a006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa5b41f4f50 tx=0x7fa5a8007c00 comp rx=0 tx=0).stop 2024-03-06T07:23:34.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 msgr2=0x7fa5b40cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b40cf5b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4000c00 tx=0x7fa5a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 shutdown_connections 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5a006b4d0 0x7fa5a006d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b415ad90 0x7fa5b41f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b40b42e0 0x7fa5b41483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 --2- 172.21.15.122:0/3034870378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5b4071a70 0x7fa5b40cf5b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 >> 172.21.15.122:0/3034870378 conn(0x7fa5b40b2660 msgr2=0x7fa5b406da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 shutdown_connections 2024-03-06T07:23:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:34.856+0000 7fa5ba642640 1 -- 172.21.15.122:0/3034870378 wait complete. 2024-03-06T07:23:34.869 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:34.869 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:34.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:34.971 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:34.971 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:23:34.971 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:34.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:34.987 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:23:34.987 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:35.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:35.004 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:23:35.004 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:35.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:35.042 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:23:35.042 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:35.060 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:35.060 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:35.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:35.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.180+0000 7f64c0447640 1 Processor -- start 2024-03-06T07:23:35.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.180+0000 7f64c0447640 1 -- start start 2024-03-06T07:23:35.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64c0447640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f64b80b3060 0x7f64b815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64c0447640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64b815bf30 0x7f64b815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64c0447640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b815f2d0 con 0x7f64b813dc10 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64c0447640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b80cf3f0 con 0x7f64b815bf30 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64c0447640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b815b210 con 0x7f64b80b3060 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be9e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64b815bf30 0x7f64b815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be1e3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f64b813dc10 0x7f64b80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51390/0 (socket says 172.21.15.122:51390) 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be1e3640 1 -- 172.21.15.122:0/1335515534 learned_addr learned my addr 172.21.15.122:0/1335515534 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64bd9e2640 1 --2- 172.21.15.122:0/1335515534 >> v2:172.21.15.169:3300/0 conn(0x7f64b80b3060 0x7f64b815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be9e4640 1 -- 172.21.15.122:0/1335515534 >> v2:172.21.15.169:3300/0 conn(0x7f64b80b3060 msgr2=0x7f64b815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be9e4640 1 --2- 172.21.15.122:0/1335515534 >> v2:172.21.15.169:3300/0 conn(0x7f64b80b3060 0x7f64b815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be9e4640 1 -- 172.21.15.122:0/1335515534 >> v1:172.21.15.122:6789/0 conn(0x7f64b813dc10 legacy=0x7f64b80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64be9e4640 1 -- 172.21.15.122:0/1335515534 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b815f580 con 0x7f64b815bf30 2024-03-06T07:23:35.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.181+0000 7f64bd1e1640 1 -- 172.21.15.122:0/1335515534 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186095726 0 0) 0x7f64b815f2d0 con 0x7f64b813dc10 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.182+0000 7f64be9e4640 1 --2- 172.21.15.122:0/1335515534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b815bf30 0x7f64b815ed90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f64b8068700 tx=0x7f64b402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26220b46f19d41f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.182+0000 7f64bd1e1640 1 -- 172.21.15.122:0/1335515534 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b402e9d0 con 0x7f64b815bf30 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.182+0000 7f64bd1e1640 1 -- 172.21.15.122:0/1335515534 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b402eb70 con 0x7f64b815bf30 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.182+0000 7f64bd1e1640 1 -- 172.21.15.122:0/1335515534 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4009540 con 0x7f64b815bf30 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 -- 172.21.15.122:0/1335515534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b815bf30 msgr2=0x7f64b815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 --2- 172.21.15.122:0/1335515534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b815bf30 0x7f64b815ed90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f64b8068700 tx=0x7f64b402e030 comp rx=0 tx=0).stop 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 -- 172.21.15.122:0/1335515534 shutdown_connections 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 --2- 172.21.15.122:0/1335515534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b815bf30 0x7f64b815ed90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 --2- 172.21.15.122:0/1335515534 >> v2:172.21.15.169:3300/0 conn(0x7f64b80b3060 0x7f64b815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 -- 172.21.15.122:0/1335515534 >> 172.21.15.122:0/1335515534 conn(0x7f64b80cf8a0 msgr2=0x7f64b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 -- 172.21.15.122:0/1335515534 shutdown_connections 2024-03-06T07:23:35.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.183+0000 7f64c0447640 1 -- 172.21.15.122:0/1335515534 wait complete. 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.184+0000 7f64c0447640 1 Processor -- start 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.184+0000 7f64c0447640 1 -- start start 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64c0447640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64b80b3060 0x7f64b8142aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64c0447640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 0x7f64b80af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64c0447640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b815bf30 0x7f64b813ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64c0447640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b813dff0 con 0x7f64b813dc10 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64c0447640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b815b730 con 0x7f64b815bf30 2024-03-06T07:23:35.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64c0447640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b80cd7a0 con 0x7f64b80b3060 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64bd9e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 0x7f64b80af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64bd9e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 0x7f64b80af860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35640/0 (socket says 172.21.15.122:35640) 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64bd9e2640 1 -- 172.21.15.122:0/1688066775 learned_addr learned my addr 172.21.15.122:0/1688066775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64be1e3640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64b80b3060 0x7f64b8142aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.185+0000 7f64be9e4640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b815bf30 0x7f64b813ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64bd9e2640 1 -- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64b80b3060 msgr2=0x7f64b8142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64bd9e2640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64b80b3060 0x7f64b8142aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64bd9e2640 1 -- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b815bf30 msgr2=0x7f64b813ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64bd9e2640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b815bf30 0x7f64b813ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64bd9e2640 1 -- 172.21.15.122:0/1688066775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8143080 con 0x7f64b813dc10 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64bd9e2640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 0x7f64b80af860 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f64a800e9e0 tx=0x7f64a800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8017070 con 0x7f64b813dc10 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a800c9d0 con 0x7f64b813dc10 2024-03-06T07:23:35.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a800cdc0 con 0x7f64b813dc10 2024-03-06T07:23:35.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b8143360 con 0x7f64b813dc10 2024-03-06T07:23:35.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.186+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b8143920 con 0x7f64b813dc10 2024-03-06T07:23:35.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.187+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b804f330 con 0x7f64b813dc10 2024-03-06T07:23:35.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.187+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64a801a070 con 0x7f64b813dc10 2024-03-06T07:23:35.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.188+0000 7f64aeffd640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f649c06b4d0 0x7f649c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.188+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f64a808a200 con 0x7f64b813dc10 2024-03-06T07:23:35.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.188+0000 7f64be1e3640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f649c06b4d0 0x7f649c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.188+0000 7f64be1e3640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f649c06b4d0 0x7f649c06d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f64b0000f60 tx=0x7f64b0007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.192+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64a8058dc0 con 0x7f64b813dc10 2024-03-06T07:23:35.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.400+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64b80b1110 con 0x7f64b813dc10 2024-03-06T07:23:35.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.401+0000 7f64aeffd640 1 -- 172.21.15.122:0/1688066775 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f64a805ce60 con 0x7f64b813dc10 2024-03-06T07:23:35.403 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:35.403 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4961,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3523054214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3523054214},{"type":"v1","addr":"172.21.15.122:6837","nonce":3523054214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4967,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3996948895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3996948895},{"type":"v1","addr":"172.21.15.169:6834","nonce":3996948895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4970,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/125562579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":125562579},{"type":"v1","addr":"172.21.15.169:6835","nonce":125562579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/2566364589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2566364589},{"type":"v1","addr":"172.21.15.122:6835","nonce":2566364589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2024-03-06T07:23:35.404 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 17 2024-03-06T07:23:35.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f649c06b4d0 msgr2=0x7f649c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f649c06b4d0 0x7f649c06d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f64b0000f60 tx=0x7f64b0007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:35.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 msgr2=0x7f64b80af860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 0x7f64b80af860 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f64a800e9e0 tx=0x7f64a800eea0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 shutdown_connections 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f649c06b4d0 0x7f649c06d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b815bf30 0x7f64b813ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b813dc10 0x7f64b80af860 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 --2- 172.21.15.122:0/1688066775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64b80b3060 0x7f64b8142aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 >> 172.21.15.122:0/1688066775 conn(0x7f64b80cf8a0 msgr2=0x7f64b813c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 shutdown_connections 2024-03-06T07:23:35.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.403+0000 7f64c0447640 1 -- 172.21.15.122:0/1688066775 wait complete. 2024-03-06T07:23:35.416 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:35.417 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:35.417 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:35.417 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:35.417 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:35.417 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:35.417 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:35.417 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:35.418 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:35.418 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:35.418 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:35.418 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:35.418 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:35.418 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:35.418 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:35.418 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:35.427 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:35.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:35.427 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:35.428 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:35.430 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:35.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:35.430 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:35.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:35.469 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:35.470 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:35.470 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:35.470 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:35.473 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:35.473 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:35.474 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:35.474 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:35.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:35.545 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:23:35.545 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:23:35.558 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:23:35.559 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:23:35.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.576+0000 7f641f8c9640 1 Processor -- start 2024-03-06T07:23:35.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.576+0000 7f641f8c9640 1 -- start start 2024-03-06T07:23:35.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641f8c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 0x7f64180d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641f8c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64180d22c0 con 0x7f6418162970 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641f8c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64180d25a0 con 0x7f64180b4460 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641f8c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6418161b00 con 0x7f6418162400 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641ce64640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6418162970 0x7f6418161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51396/0 (socket says 172.21.15.122:51396) 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641ce64640 1 -- 172.21.15.122:0/2805887417 learned_addr learned my addr 172.21.15.122:0/2805887417 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f641de66640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 0x7f64180d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120091977 0 0) 0x7f64180d22c0 con 0x7f6418162970 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.577+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6404003520 con 0x7f6418162970 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463310299 0 0) 0x7f6404003520 con 0x7f6418162970 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 msgr2=0x7f64180d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 0x7f64180d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 >> v1:172.21.15.169:6789/0 conn(0x7f64180b4460 legacy=0x7f64180b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64180d2880 con 0x7f6418162970 2024-03-06T07:23:35.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f6408001670 con 0x7f6418162970 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 >> v1:172.21.15.122:6789/0 conn(0x7f6418162970 legacy=0x7f6418161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64040039e0 0x7f6404003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6404004800 0x7f6404004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 0x7f64040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6418161b00 con 0x7f64040039e0 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040142b0 con 0x7f6404004800 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64040144b0 con 0x7f6418162400 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641d665640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64040039e0 0x7f6404003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641de66640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 0x7f64040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641ce64640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6404004800 0x7f6404004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641d665640 1 -- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 msgr2=0x7f64040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641d665640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6418162400 0x7f64040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641d665640 1 -- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6404004800 msgr2=0x7f6404004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641d665640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6404004800 0x7f6404004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.578+0000 7f641d665640 1 -- 172.21.15.122:0/2805887417 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6404003520 con 0x7f64040039e0 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f641d665640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64040039e0 0x7f6404003e00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6414001720 tx=0x7f6404005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3649c87cf2eca61e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414001370 con 0x7f64040039e0 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64140052d0 con 0x7f64040039e0 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f640ffff640 1 -- 172.21.15.122:0/2805887417 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414003830 con 0x7f64040039e0 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f641f8c9640 1 -- 172.21.15.122:0/2805887417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64040039e0 msgr2=0x7f6404003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2805887417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64040039e0 0x7f6404003e00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6414001720 tx=0x7f6404005830 comp rx=0 tx=0).stop 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.579+0000 7f641de66640 1 -- 172.21.15.122:0/2805887417 reap_dead start 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.580+0000 7f641f8c9640 1 -- 172.21.15.122:0/2805887417 shutdown_connections 2024-03-06T07:23:35.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.580+0000 7f641f8c9640 1 -- 172.21.15.122:0/2805887417 >> 172.21.15.122:0/2805887417 conn(0x7f64180d3ce0 msgr2=0x7f64180d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.580+0000 7f641f8c9640 1 -- 172.21.15.122:0/2805887417 shutdown_connections 2024-03-06T07:23:35.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.580+0000 7f641f8c9640 1 -- 172.21.15.122:0/2805887417 wait complete. 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 Processor -- start 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 -- start start 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64040039e0 0x7f64181544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f641815a900 0x7f6418154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 0x7f64181550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64180ce950 con 0x7f641815ad80 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64180cecc0 con 0x7f64040039e0 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.581+0000 7f641f8c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64180b4840 con 0x7f641815a900 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 0x7f64181550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 0x7f64181550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35718/0 (socket says 172.21.15.122:35718) 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 -- 172.21.15.122:0/2644932397 learned_addr learned my addr 172.21.15.122:0/2644932397 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:35.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641ce64640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f641815a900 0x7f6418154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641d665640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64040039e0 0x7f64181544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 -- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f641815a900 msgr2=0x7f6418154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f641815a900 0x7f6418154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 -- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64040039e0 msgr2=0x7f64181544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64040039e0 0x7f64181544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 -- 172.21.15.122:0/2644932397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641806c390 con 0x7f641815ad80 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641ce64640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f641815a900 0x7f6418154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641d665640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64040039e0 0x7f64181544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641de66640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 0x7f64181550e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f64180c7610 tx=0x7f641000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410015430 con 0x7f641815ad80 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64100155d0 con 0x7f641815ad80 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641806c670 con 0x7f641815ad80 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100159c0 con 0x7f641815ad80 2024-03-06T07:23:35.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.582+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641815a3e0 con 0x7f641815ad80 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.583+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641815a5e0 con 0x7f641815ad80 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.584+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6410015b60 con 0x7f641815ad80 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.584+0000 7f640dffb640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63fc06b4d0 0x7f63fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.584+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f641008ad00 con 0x7f641815ad80 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.584+0000 7f641d665640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63fc06b4d0 0x7f63fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.585+0000 7f641d665640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63fc06b4d0 0x7f63fc06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6414033000 tx=0x7f6414009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.589+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f641815a5e0 con 0x7f641815ad80 2024-03-06T07:23:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:35.772+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f641804f330 con 0x7f641815ad80 2024-03-06T07:23:36.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.876+0000 7f640dffb640 1 -- 172.21.15.122:0/2644932397 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6410019030 con 0x7f641815ad80 2024-03-06T07:23:36.878 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.878+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63fc06b4d0 msgr2=0x7f63fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.878+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63fc06b4d0 0x7f63fc06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6414033000 tx=0x7f6414009030 comp rx=0 tx=0).stop 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.878+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 msgr2=0x7f64181550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.878+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 0x7f64181550e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f64180c7610 tx=0x7f641000f580 comp rx=0 tx=0).stop 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.878+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 shutdown_connections 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.878+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63fc06b4d0 0x7f63fc06d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.879+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f641815ad80 0x7f64181550e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.879+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f641815a900 0x7f6418154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.879+0000 7f641f8c9640 1 --2- 172.21.15.122:0/2644932397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64040039e0 0x7f64181544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.879+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 >> 172.21.15.122:0/2644932397 conn(0x7f64180d3ce0 msgr2=0x7f641815bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.879+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 shutdown_connections 2024-03-06T07:23:36.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.879+0000 7f641f8c9640 1 -- 172.21.15.122:0/2644932397 wait complete. 2024-03-06T07:23:36.894 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:37.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.998+0000 7f6aba6d4640 1 Processor -- start 2024-03-06T07:23:37.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.998+0000 7f6aba6d4640 1 -- start start 2024-03-06T07:23:37.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6aba6d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40d0700 0x7f6ab40b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6aba6d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ab40b0eb0 0x7f6ab40b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6aba6d4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab415c270 con 0x7f6ab40d0320 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6aba6d4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab40b1870 con 0x7f6ab40b0eb0 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6aba6d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab40b04e0 con 0x7f6ab40d0700 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ab40b0eb0 0x7f6ab40b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ab40b0eb0 0x7f6ab40b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35730/0 (socket says 172.21.15.122:35730) 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 -- 172.21.15.122:0/528161630 learned_addr learned my addr 172.21.15.122:0/528161630 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab37fe640 1 --2- 172.21.15.122:0/528161630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40d0700 0x7f6ab40b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 -- 172.21.15.122:0/528161630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40d0700 msgr2=0x7f6ab40b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 --2- 172.21.15.122:0/528161630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40d0700 0x7f6ab40b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 -- 172.21.15.122:0/528161630 >> v1:172.21.15.169:6789/0 conn(0x7f6ab40d0320 legacy=0x7f6ab40b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:36.999+0000 7f6ab8c71640 1 -- 172.21.15.122:0/528161630 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab40d0e90 con 0x7f6ab40b0eb0 2024-03-06T07:23:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.000+0000 7f6ab8c71640 1 --2- 172.21.15.122:0/528161630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40b0eb0 0x7f6ab40b1330 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009e50 tx=0x7f6aac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d484d667687d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.000+0000 7f6ab2ffd640 1 -- 172.21.15.122:0/528161630 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac02cb00 con 0x7f6ab40b0eb0 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.000+0000 7f6ab2ffd640 1 -- 172.21.15.122:0/528161630 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac02cca0 con 0x7f6ab40b0eb0 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.000+0000 7f6ab2ffd640 1 -- 172.21.15.122:0/528161630 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac034610 con 0x7f6ab40b0eb0 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 -- 172.21.15.122:0/528161630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40b0eb0 msgr2=0x7f6ab40b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/528161630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40b0eb0 0x7f6ab40b1330 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009e50 tx=0x7f6aac02bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 -- 172.21.15.122:0/528161630 shutdown_connections 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/528161630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40b0eb0 0x7f6ab40b1330 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/528161630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40d0700 0x7f6ab40b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 -- 172.21.15.122:0/528161630 >> 172.21.15.122:0/528161630 conn(0x7f6ab40724d0 msgr2=0x7f6ab40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 -- 172.21.15.122:0/528161630 shutdown_connections 2024-03-06T07:23:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.001+0000 7f6aba6d4640 1 -- 172.21.15.122:0/528161630 wait complete. 2024-03-06T07:23:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.002+0000 7f6aba6d4640 1 Processor -- start 2024-03-06T07:23:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.002+0000 7f6aba6d4640 1 -- start start 2024-03-06T07:23:37.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40b0eb0 0x7f6ab414a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab40d0320 0x7f6ab414aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 0x7f6ab414b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab40b0170 con 0x7f6ab40d0700 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab40b5dc0 con 0x7f6ab40d0320 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab40b60a0 con 0x7f6ab40b0eb0 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 0x7f6ab414b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 0x7f6ab414b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35734/0 (socket says 172.21.15.122:35734) 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 -- 172.21.15.122:0/206144014 learned_addr learned my addr 172.21.15.122:0/206144014 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 -- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40b0eb0 msgr2=0x7f6ab414a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab3fff640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40b0eb0 0x7f6ab414a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab37fe640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab40d0320 0x7f6ab414aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40b0eb0 0x7f6ab414a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 -- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab40d0320 msgr2=0x7f6ab414aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab40d0320 0x7f6ab414aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 -- 172.21.15.122:0/206144014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab4156ec0 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab8c71640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 0x7f6ab414b0e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6aac036040 tx=0x7f6aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac043020 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac02ce50 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac009510 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.003+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab414b710 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.004+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab414bcd0 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.004+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab404f330 con 0x7f6ab40d0700 2024-03-06T07:23:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.005+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6aac005d10 con 0x7f6ab40d0700 2024-03-06T07:23:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.005+0000 7f6ab0ff9640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.005+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6aac0ac050 con 0x7f6ab40d0700 2024-03-06T07:23:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.005+0000 7f6ab3fff640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.006+0000 7f6ab3fff640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4003d90 tx=0x7f6aa4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.010+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6aac04d020 con 0x7f6ab40d0700 2024-03-06T07:23:37.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.245+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6ab414a850 con 0x7f6ab40d0700 2024-03-06T07:23:37.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.245+0000 7f6ab0ff9640 1 -- 172.21.15.122:0/206144014 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6aac07b2b0 con 0x7f6ab40d0700 2024-03-06T07:23:37.247 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:37.247 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:37.247 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:23:37.248 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:23:37.249 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.250 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.251 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:23:37.252 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.253 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:23:37.254 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.256 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.258 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.259 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:23:37.260 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 msgr2=0x7f6aa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4003d90 tx=0x7f6aa4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 msgr2=0x7f6ab414b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 0x7f6ab414b0e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6aac036040 tx=0x7f6aac004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 shutdown_connections 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ab40d0700 0x7f6ab414b0e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab40d0320 0x7f6ab414aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 --2- 172.21.15.122:0/206144014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ab40b0eb0 0x7f6ab414a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 >> 172.21.15.122:0/206144014 conn(0x7f6ab40724d0 msgr2=0x7f6ab4072b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 shutdown_connections 2024-03-06T07:23:37.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.247+0000 7f6aba6d4640 1 -- 172.21.15.122:0/206144014 wait complete. 2024-03-06T07:23:37.263 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:37.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.361+0000 7fdbc313e640 1 Processor -- start 2024-03-06T07:23:37.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.361+0000 7fdbc313e640 1 -- start start 2024-03-06T07:23:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc313e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdbbc15dfe0 0x7fdbbc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc313e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdbbc0d29c0 0x7fdbbc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc313e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc0b3430 con 0x7fdbbc0cf830 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc313e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0b31e0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc313e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0b3d60 con 0x7fdbbc15dfe0 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc16db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdbbc0d29c0 0x7fdbbc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc16db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdbbc0d29c0 0x7fdbbc0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35742/0 (socket says 172.21.15.122:35742) 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbbbfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdbbc0cf830 0x7fdbbc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51412/0 (socket says 172.21.15.122:51412) 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.362+0000 7fdbc16db640 1 -- 172.21.15.122:0/3373292395 learned_addr learned my addr 172.21.15.122:0/3373292395 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc0eda640 1 --2- 172.21.15.122:0/3373292395 >> v2:172.21.15.169:3300/0 conn(0x7fdbbc15dfe0 0x7fdbbc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc16db640 1 -- 172.21.15.122:0/3373292395 >> v2:172.21.15.169:3300/0 conn(0x7fdbbc15dfe0 msgr2=0x7fdbbc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc16db640 1 --2- 172.21.15.122:0/3373292395 >> v2:172.21.15.169:3300/0 conn(0x7fdbbc15dfe0 0x7fdbbc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc16db640 1 -- 172.21.15.122:0/3373292395 >> v1:172.21.15.122:6789/0 conn(0x7fdbbc0cf830 legacy=0x7fdbbc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbbb7fe640 1 -- 172.21.15.122:0/3373292395 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527043567 0 0) 0x7fdbbc0b3430 con 0x7fdbbc0cf830 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc16db640 1 -- 172.21.15.122:0/3373292395 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0b3710 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc0eda640 1 --2- 172.21.15.122:0/3373292395 >> v2:172.21.15.169:3300/0 conn(0x7fdbbc15dfe0 0x7fdbbc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.363+0000 7fdbc16db640 1 --2- 172.21.15.122:0/3373292395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0b41f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc067990 tx=0x7fdbb402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db415f55d4cb23d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbbb7fe640 1 -- 172.21.15.122:0/3373292395 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb402cb20 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbbb7fe640 1 -- 172.21.15.122:0/3373292395 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb402ccc0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbbb7fe640 1 -- 172.21.15.122:0/3373292395 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4034610 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 -- 172.21.15.122:0/3373292395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 msgr2=0x7fdbbc0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 --2- 172.21.15.122:0/3373292395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0b41f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc067990 tx=0x7fdbb402bf80 comp rx=0 tx=0).stop 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 -- 172.21.15.122:0/3373292395 shutdown_connections 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 --2- 172.21.15.122:0/3373292395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0b41f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 --2- 172.21.15.122:0/3373292395 >> v2:172.21.15.169:3300/0 conn(0x7fdbbc15dfe0 0x7fdbbc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 -- 172.21.15.122:0/3373292395 >> 172.21.15.122:0/3373292395 conn(0x7fdbbc0d3cd0 msgr2=0x7fdbbc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 -- 172.21.15.122:0/3373292395 shutdown_connections 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.364+0000 7fdbc313e640 1 -- 172.21.15.122:0/3373292395 wait complete. 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.365+0000 7fdbc313e640 1 Processor -- start 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.365+0000 7fdbc313e640 1 -- start start 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbc313e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdbbc0cf830 0x7fdbbc1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbc313e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbc313e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbbc15dfe0 0x7fdbbc15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbc313e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0d1df0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbc313e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0d1c70 con 0x7fdbbc15dfe0 2024-03-06T07:23:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbc313e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc15e3c0 con 0x7fdbbc0cf830 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbbbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35746/0 (socket says 172.21.15.122:35746) 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.366+0000 7fdbbbfff640 1 -- 172.21.15.122:0/1535911074 learned_addr learned my addr 172.21.15.122:0/1535911074 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbc0eda640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdbbc0cf830 0x7fdbbc1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbc16db640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbbc15dfe0 0x7fdbbc15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbbbfff640 1 -- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdbbc0cf830 msgr2=0x7fdbbc1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbbbfff640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdbbc0cf830 0x7fdbbc1610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbbbfff640 1 -- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbbc15dfe0 msgr2=0x7fdbbc15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbbbfff640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbbc15dfe0 0x7fdbbc15b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbbbfff640 1 -- 172.21.15.122:0/1535911074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0cbf00 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbbbfff640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0cb7b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000c4b0 tx=0x7fdbb000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0018070 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb0007500 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc0cc1e0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0004a20 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.367+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc0cc7a0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.368+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc04f330 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.368+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbb001b070 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.369+0000 7fdbb97fa640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.369+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdbb0089210 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.369+0000 7fdbc0eda640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.369+0000 7fdbc0eda640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdba806b4d0 0x7fdba806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b840 tx=0x7fdbac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.373+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbb00584a0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.544+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbbc0ce070 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.545+0000 7fdbb97fa640 1 -- 172.21.15.122:0/1535911074 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12891 (secure 0 0 0) 0x7fdbb0057da0 con 0x7fdbbc0d29c0 2024-03-06T07:23:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:37.548 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":41,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:36.875459+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:37.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdba806b4d0 msgr2=0x7fdba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdba806b4d0 0x7fdba806d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b840 tx=0x7fdbac008040 comp rx=0 tx=0).stop 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 msgr2=0x7fdbbc0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0cb7b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000c4b0 tx=0x7fdbb000c970 comp rx=0 tx=0).stop 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 shutdown_connections 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbbc15dfe0 0x7fdbbc15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbbc0d29c0 0x7fdbbc0cb7b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 --2- 172.21.15.122:0/1535911074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdbbc0cf830 0x7fdbbc1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 >> 172.21.15.122:0/1535911074 conn(0x7fdbbc0d3cd0 msgr2=0x7fdbbc0cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 shutdown_connections 2024-03-06T07:23:37.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.547+0000 7fdbc313e640 1 -- 172.21.15.122:0/1535911074 wait complete. 2024-03-06T07:23:37.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:37.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:37.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.663+0000 7f3cc3eaf640 1 Processor -- start 2024-03-06T07:23:37.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.663+0000 7f3cc3eaf640 1 -- start start 2024-03-06T07:23:37.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc3eaf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3cbc15f4f0 0x7f3cbc066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc3eaf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3cbc0ce750 0x7f3cbc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc3eaf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc06b050 con 0x7f3cbc0b0db0 2024-03-06T07:23:37.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc3eaf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cbc06b2c0 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc3eaf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cbc0b0890 con 0x7f3cbc0ce750 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc1c4b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3cbc15f4f0 0x7f3cbc066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc144a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3cbc0b0db0 0x7f3cbc0ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51424/0 (socket says 172.21.15.122:51424) 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc144a640 1 -- 172.21.15.122:0/2699194678 learned_addr learned my addr 172.21.15.122:0/2699194678 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc1c4b640 1 -- 172.21.15.122:0/2699194678 >> v2:172.21.15.169:3300/0 conn(0x7f3cbc0ce750 msgr2=0x7f3cbc0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc1c4b640 1 --2- 172.21.15.122:0/2699194678 >> v2:172.21.15.169:3300/0 conn(0x7f3cbc0ce750 0x7f3cbc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc1c4b640 1 -- 172.21.15.122:0/2699194678 >> v1:172.21.15.122:6789/0 conn(0x7f3cbc0b0db0 legacy=0x7f3cbc0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc0c49640 1 -- 172.21.15.122:0/2699194678 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851521074 0 0) 0x7f3cbc06b050 con 0x7f3cbc0b0db0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.664+0000 7f3cc1c4b640 1 -- 172.21.15.122:0/2699194678 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0672e0 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc1c4b640 1 --2- 172.21.15.122:0/2699194678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc066d00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc070870 tx=0x7f3cb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cddb56a6bec1a346 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc0c49640 1 -- 172.21.15.122:0/2699194678 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8035020 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc0c49640 1 -- 172.21.15.122:0/2699194678 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb8031830 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc0c49640 1 -- 172.21.15.122:0/2699194678 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8031c00 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/2699194678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 msgr2=0x7f3cbc066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/2699194678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc066d00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc070870 tx=0x7f3cb802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/2699194678 shutdown_connections 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/2699194678 >> v2:172.21.15.169:3300/0 conn(0x7f3cbc0ce750 0x7f3cbc0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/2699194678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc066d00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.665+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/2699194678 >> 172.21.15.122:0/2699194678 conn(0x7f3cbc0b1d70 msgr2=0x7f3cbc0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.666+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/2699194678 shutdown_connections 2024-03-06T07:23:37.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.666+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/2699194678 wait complete. 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 Processor -- start 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 -- start start 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cbc0b0db0 0x7f3cbc1536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cbc0ce750 0x7f3cbc1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc0c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc15f8d0 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc06b5a0 con 0x7f3cbc0b0db0 2024-03-06T07:23:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc3eaf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc153be0 con 0x7f3cbc0ce750 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc244c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc0c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc244c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc0c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35762/0 (socket says 172.21.15.122:35762) 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.667+0000 7f3cc244c640 1 -- 172.21.15.122:0/6934513 learned_addr learned my addr 172.21.15.122:0/6934513 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc244c640 1 -- 172.21.15.122:0/6934513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cbc0ce750 msgr2=0x7f3cbc1f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc244c640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cbc0ce750 0x7f3cbc1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc244c640 1 -- 172.21.15.122:0/6934513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cbc0b0db0 msgr2=0x7f3cbc1536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc244c640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cbc0b0db0 0x7f3cbc1536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc244c640 1 -- 172.21.15.122:0/6934513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc1f2a70 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc244c640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc0c9180 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400b9d0 tx=0x7f3cb400be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4019070 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb400cbf0 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4014440 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc0c7710 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.668+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc0c7cd0 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.669+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc09e3d0 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.669+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cb401c070 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.670+0000 7f3cb27fc640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ca006b4d0 0x7f3ca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.670+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3cb4089b30 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.670+0000 7f3cc1c4b640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ca006b4d0 0x7f3ca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.670+0000 7f3cc1c4b640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ca006b4d0 0x7f3ca006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8036000 tx=0x7f3cb8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.674+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cbc1f2a70 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.843+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3cbc06cd40 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.844+0000 7f3cb27fc640 1 -- 172.21.15.122:0/6934513 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3cb4058d90 con 0x7f3cbc15f4f0 2024-03-06T07:23:37.846 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:23:37.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ca006b4d0 msgr2=0x7f3ca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ca006b4d0 0x7f3ca006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8036000 tx=0x7f3cb8030000 comp rx=0 tx=0).stop 2024-03-06T07:23:37.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 msgr2=0x7f3cbc0c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc0c9180 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400b9d0 tx=0x7f3cb400be90 comp rx=0 tx=0).stop 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 shutdown_connections 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ca006b4d0 0x7f3ca006d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cbc15f4f0 0x7f3cbc0c9180 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cbc0ce750 0x7f3cbc1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 --2- 172.21.15.122:0/6934513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cbc0b0db0 0x7f3cbc1536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 >> 172.21.15.122:0/6934513 conn(0x7f3cbc0b1d70 msgr2=0x7f3cbc06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 shutdown_connections 2024-03-06T07:23:37.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.846+0000 7f3cc3eaf640 1 -- 172.21.15.122:0/6934513 wait complete. 2024-03-06T07:23:37.859 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:37.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.956+0000 7fe03826b640 1 Processor -- start 2024-03-06T07:23:37.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.956+0000 7fe03826b640 1 -- start start 2024-03-06T07:23:37.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe03826b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe03015bb20 0x7fe0300d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe03826b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0300cfff0 0x7fe0300d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe03826b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe0300ce4b0 0x7fe0300ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe03826b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe030161980 con 0x7fe03015bb20 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe03826b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe030161c60 con 0x7fe0300ce4b0 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe03826b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe030161f40 con 0x7fe0300cfff0 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe036007640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe03015bb20 0x7fe0300d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe036007640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe03015bb20 0x7fe0300d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35770/0 (socket says 172.21.15.122:35770) 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.957+0000 7fe036007640 1 -- 172.21.15.122:0/1750830855 learned_addr learned my addr 172.21.15.122:0/1750830855 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe035806640 1 --2- 172.21.15.122:0/1750830855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0300cfff0 0x7fe0300d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036808640 1 --2- 172.21.15.122:0/1750830855 >> v2:172.21.15.169:3300/0 conn(0x7fe0300ce4b0 0x7fe0300ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036007640 1 -- 172.21.15.122:0/1750830855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0300cfff0 msgr2=0x7fe0300d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036007640 1 --2- 172.21.15.122:0/1750830855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0300cfff0 0x7fe0300d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036007640 1 -- 172.21.15.122:0/1750830855 >> v2:172.21.15.169:3300/0 conn(0x7fe0300ce4b0 msgr2=0x7fe0300ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036007640 1 --2- 172.21.15.122:0/1750830855 >> v2:172.21.15.169:3300/0 conn(0x7fe0300ce4b0 0x7fe0300ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036007640 1 -- 172.21.15.122:0/1750830855 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe030162220 con 0x7fe03015bb20 2024-03-06T07:23:37.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.958+0000 7fe036007640 1 --2- 172.21.15.122:0/1750830855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe03015bb20 0x7fe0300d1df0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe0300b13b0 tx=0x7fe02c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14605da533c718e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe035005640 1 -- 172.21.15.122:0/1750830855 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c02ca70 con 0x7fe03015bb20 2024-03-06T07:23:37.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe035005640 1 -- 172.21.15.122:0/1750830855 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c02cc10 con 0x7fe03015bb20 2024-03-06T07:23:37.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe035005640 1 -- 172.21.15.122:0/1750830855 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c0343f0 con 0x7fe03015bb20 2024-03-06T07:23:37.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 -- 172.21.15.122:0/1750830855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe03015bb20 msgr2=0x7fe0300d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 --2- 172.21.15.122:0/1750830855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe03015bb20 0x7fe0300d1df0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe0300b13b0 tx=0x7fe02c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 -- 172.21.15.122:0/1750830855 shutdown_connections 2024-03-06T07:23:37.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 --2- 172.21.15.122:0/1750830855 >> v2:172.21.15.169:3300/0 conn(0x7fe0300ce4b0 0x7fe0300ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 --2- 172.21.15.122:0/1750830855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0300cfff0 0x7fe0300d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 --2- 172.21.15.122:0/1750830855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe03015bb20 0x7fe0300d1df0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 -- 172.21.15.122:0/1750830855 >> 172.21.15.122:0/1750830855 conn(0x7fe03009a9f0 msgr2=0x7fe03009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 -- 172.21.15.122:0/1750830855 shutdown_connections 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.959+0000 7fe03826b640 1 -- 172.21.15.122:0/1750830855 wait complete. 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.960+0000 7fe03826b640 1 Processor -- start 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.960+0000 7fe03826b640 1 -- start start 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe03826b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 0x7fe0301f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe03826b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0300cfff0 0x7fe0300af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe03826b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe03015bb20 0x7fe0300afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe03826b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0300d24b0 con 0x7fe0300ce4b0 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe03826b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0300d2330 con 0x7fe0300cfff0 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe03826b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0300d1980 con 0x7fe03015bb20 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 0x7fe0301f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 0x7fe0301f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35780/0 (socket says 172.21.15.122:35780) 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 -- 172.21.15.122:0/3795996406 learned_addr learned my addr 172.21.15.122:0/3795996406 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036808640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe03015bb20 0x7fe0300afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe035806640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0300cfff0 0x7fe0300af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 -- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe03015bb20 msgr2=0x7fe0300afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe03015bb20 0x7fe0300afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 -- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0300cfff0 msgr2=0x7fe0300af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0300cfff0 0x7fe0300af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.961+0000 7fe036007640 1 -- 172.21.15.122:0/3795996406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0300c5960 con 0x7fe0300ce4b0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.962+0000 7fe036007640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 0x7fe0301f3180 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe02c000f80 tx=0x7fe02c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.962+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c043020 con 0x7fe0300ce4b0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.962+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c03e450 con 0x7fe0300ce4b0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.962+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c009510 con 0x7fe0300ce4b0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.962+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0300c5c40 con 0x7fe0300ce4b0 2024-03-06T07:23:37.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.962+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0300c6200 con 0x7fe0300ce4b0 2024-03-06T07:23:37.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.963+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03004f330 con 0x7fe0300ce4b0 2024-03-06T07:23:37.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.963+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe02c046020 con 0x7fe0300ce4b0 2024-03-06T07:23:37.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.963+0000 7fe026ffd640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe01406b4d0 0x7fe01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.964+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe02c0ac780 con 0x7fe0300ce4b0 2024-03-06T07:23:37.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.964+0000 7fe035806640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe01406b4d0 0x7fe01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.964+0000 7fe035806640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe01406b4d0 0x7fe01406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe0300b79e0 tx=0x7fe020008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:37.968+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe02c04d030 con 0x7fe0300ce4b0 2024-03-06T07:23:38.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.142+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe0300747a0 con 0x7fe0300ce4b0 2024-03-06T07:23:38.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.142+0000 7fe026ffd640 1 -- 172.21.15.122:0/3795996406 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe02c07b950 con 0x7fe0300ce4b0 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:38.145 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:23:38.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe01406b4d0 msgr2=0x7fe01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe01406b4d0 0x7fe01406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe0300b79e0 tx=0x7fe020008040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 msgr2=0x7fe0301f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 0x7fe0301f3180 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe02c000f80 tx=0x7fe02c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 shutdown_connections 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe01406b4d0 0x7fe01406d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe03015bb20 0x7fe0300afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0300cfff0 0x7fe0300af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 --2- 172.21.15.122:0/3795996406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0300ce4b0 0x7fe0301f3180 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 >> 172.21.15.122:0/3795996406 conn(0x7fe03009a9f0 msgr2=0x7fe03006fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 shutdown_connections 2024-03-06T07:23:38.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.144+0000 7fe03826b640 1 -- 172.21.15.122:0/3795996406 wait complete. 2024-03-06T07:23:38.157 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:38.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:38.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.260+0000 7efcbbb8c640 1 Processor -- start 2024-03-06T07:23:38.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.260+0000 7efcbbb8c640 1 -- start start 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcbbb8c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efcb415ac00 0x7efcb406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcbbb8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb415f5c0 con 0x7efcb40708b0 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcbbb8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40711c0 con 0x7efcb4070d10 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcbbb8c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb40cdc10 con 0x7efcb415ac00 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb9127640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efcb40708b0 0x7efcb415e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51436/0 (socket says 172.21.15.122:51436) 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb9127640 1 -- 172.21.15.122:0/920913330 learned_addr learned my addr 172.21.15.122:0/920913330 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb9928640 1 --2- 172.21.15.122:0/920913330 >> v2:172.21.15.169:3300/0 conn(0x7efcb415ac00 0x7efcb406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598142806 0 0) 0x7efcb415f5c0 con 0x7efcb40708b0 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0005730 con 0x7efcb40708b0 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3699757546 0 0) 0x7efca0005730 con 0x7efcb40708b0 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 >> v2:172.21.15.169:3300/0 conn(0x7efcb415ac00 msgr2=0x7efcb406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 --2- 172.21.15.122:0/920913330 >> v2:172.21.15.169:3300/0 conn(0x7efcb415ac00 0x7efcb406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 >> v1:172.21.15.169:6789/0 conn(0x7efcb4070d10 legacy=0x7efcb415eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb415afe0 con 0x7efcb40708b0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7efca4001670 con 0x7efcb40708b0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 >> v1:172.21.15.122:6789/0 conn(0x7efcb40708b0 legacy=0x7efcb415e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca0005bc0 0x7efca0005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0006a10 0x7efca0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca00073d0 0x7efca000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40cdc10 con 0x7efca0005bc0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca000e370 con 0x7efca00073d0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca000e570 con 0x7efca0006a10 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.261+0000 7efcb9928640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca0005bc0 0x7efca0005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcba129640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca00073d0 0x7efca000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9127640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0006a10 0x7efca0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9928640 1 -- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0006a10 msgr2=0x7efca0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9928640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0006a10 0x7efca0006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9928640 1 -- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca00073d0 msgr2=0x7efca000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9928640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca00073d0 0x7efca000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9928640 1 -- 172.21.15.122:0/920913330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0005730 con 0x7efca0005bc0 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcba129640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca00073d0 0x7efca000dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcba129640 1 -- 172.21.15.122:0/920913330 reap_dead start 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.262+0000 7efcb9928640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca0005bc0 0x7efca0005fe0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009ac0 tx=0x7efca0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320c574e483f1ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0039070 con 0x7efca0005bc0 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb002c6b0 con 0x7efca0005bc0 2024-03-06T07:23:38.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcb8926640 1 -- 172.21.15.122:0/920913330 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0009690 con 0x7efca0005bc0 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 -- 172.21.15.122:0/920913330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca0005bc0 msgr2=0x7efca0005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca0005bc0 0x7efca0005fe0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009ac0 tx=0x7efca0007a40 comp rx=0 tx=0).stop 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 -- 172.21.15.122:0/920913330 shutdown_connections 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/920913330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca0005bc0 0x7efca0005fe0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 -- 172.21.15.122:0/920913330 >> 172.21.15.122:0/920913330 conn(0x7efcb40b1650 msgr2=0x7efcb40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 -- 172.21.15.122:0/920913330 shutdown_connections 2024-03-06T07:23:38.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.263+0000 7efcbbb8c640 1 -- 172.21.15.122:0/920913330 wait complete. 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.264+0000 7efcbbb8c640 1 Processor -- start 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.264+0000 7efcbbb8c640 1 -- start start 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcbbb8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0005bc0 0x7efcb414a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcbbb8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 0x7efcb40ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcbbb8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcb414d340 0x7efcb414d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcbbb8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40cd5c0 con 0x7efcb40ad860 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcbbb8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40cd2e0 con 0x7efcb414d340 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcbbb8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40cd8a0 con 0x7efca0005bc0 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcb9127640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 0x7efcb40ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcb9127640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 0x7efcb40ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35798/0 (socket says 172.21.15.122:35798) 2024-03-06T07:23:38.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcb9127640 1 -- 172.21.15.122:0/1873245135 learned_addr learned my addr 172.21.15.122:0/1873245135 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcb9928640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0005bc0 0x7efcb414a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.265+0000 7efcba129640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcb414d340 0x7efcb414d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcb9127640 1 -- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0005bc0 msgr2=0x7efcb414a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcb9127640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0005bc0 0x7efcb414a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcb9127640 1 -- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcb414d340 msgr2=0x7efcb414d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcb9127640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcb414d340 0x7efcb414d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcb9127640 1 -- 172.21.15.122:0/1873245135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb415e070 con 0x7efcb40ad860 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcb9127640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 0x7efcb40ae780 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7efca400e780 tx=0x7efca400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400c870 con 0x7efcb40ad860 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca4018070 con 0x7efcb40ad860 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb40aee80 con 0x7efcb40ad860 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400cc70 con 0x7efcb40ad860 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.266+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb40b2fe0 con 0x7efcb40ad860 2024-03-06T07:23:38.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.267+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb40687d0 con 0x7efcb40ad860 2024-03-06T07:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.267+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efca401d460 con 0x7efcb40ad860 2024-03-06T07:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.268+0000 7efcaa7fc640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.268+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efca4089d50 con 0x7efcb40ad860 2024-03-06T07:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.268+0000 7efcb9928640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.268+0000 7efcb9928640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc9806b4d0 0x7efc9806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efcb0036000 tx=0x7efcb0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.272+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efca408d030 con 0x7efcb40ad860 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.444+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efcb404f330 con 0x7efcb40ad860 2024-03-06T07:23:38.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.878+0000 7efcaa7fc640 1 -- 172.21.15.122:0/1873245135 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7efcb415e070 con 0x7efcb40ad860 2024-03-06T07:23:38.881 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:38.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc9806b4d0 msgr2=0x7efc9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc9806b4d0 0x7efc9806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efcb0036000 tx=0x7efcb0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 msgr2=0x7efcb40ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 0x7efcb40ae780 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7efca400e780 tx=0x7efca400ec40 comp rx=0 tx=0).stop 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 shutdown_connections 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcb414d340 0x7efcb414d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb40ad860 0x7efcb40ae780 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 --2- 172.21.15.122:0/1873245135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efca0005bc0 0x7efcb414a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 >> 172.21.15.122:0/1873245135 conn(0x7efcb40b1650 msgr2=0x7efcb40b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 shutdown_connections 2024-03-06T07:23:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:38.881+0000 7efcbbb8c640 1 -- 172.21.15.122:0/1873245135 wait complete. 2024-03-06T07:23:38.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:39.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.002+0000 7fad861b6640 1 Processor -- start 2024-03-06T07:23:39.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.002+0000 7fad861b6640 1 -- start start 2024-03-06T07:23:39.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad861b6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad800b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad861b6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad80158ac0 con 0x7fad800cfa90 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad861b6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8015b2a0 con 0x7fad801586e0 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad861b6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8015b540 con 0x7fad800cff40 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fad800cfa90 0x7fad800b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51440/0 (socket says 172.21.15.122:51440) 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7effd640 1 -- 172.21.15.122:0/1546948694 learned_addr learned my addr 172.21.15.122:0/1546948694 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577856108 0 0) 0x7fad80158ac0 con 0x7fad800cfa90 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad68003520 con 0x7fad800cfa90 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7ffff640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad800b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118761840 0 0) 0x7fad68003520 con 0x7fad800cfa90 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 msgr2=0x7fad800b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad800b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 >> v1:172.21.15.169:6789/0 conn(0x7fad801586e0 legacy=0x7fad800aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8015b820 con 0x7fad800cfa90 2024-03-06T07:23:39.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fad74001670 con 0x7fad800cfa90 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 >> v1:172.21.15.122:6789/0 conn(0x7fad800cfa90 legacy=0x7fad800b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad680039e0 0x7fad68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad68004800 0x7fad68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8015b540 con 0x7fad68004800 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6800ba70 con 0x7fad680039e0 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6800bc70 con 0x7fad800cff40 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7effd640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad68004800 0x7fad68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.003+0000 7fad7f7fe640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad680039e0 0x7fad68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7ffff640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7effd640 1 -- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 msgr2=0x7fad680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7effd640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7effd640 1 -- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad680039e0 msgr2=0x7fad68003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7effd640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad680039e0 0x7fad68003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7effd640 1 -- 172.21.15.122:0/1546948694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad68003520 con 0x7fad68004800 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7ffff640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad800cff40 0x7fad680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7effd640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad68004800 0x7fad68004c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fad7400c210 tx=0x7fad68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f914eb4fb6f3f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.004+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74007cf0 con 0x7fad68004800 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad74030070 con 0x7fad68004800 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad7e7fc640 1 -- 172.21.15.122:0/1546948694 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74005d30 con 0x7fad68004800 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad861b6640 1 -- 172.21.15.122:0/1546948694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad68004800 msgr2=0x7fad68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad861b6640 1 --2- 172.21.15.122:0/1546948694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad68004800 0x7fad68004c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fad7400c210 tx=0x7fad68005830 comp rx=0 tx=0).stop 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad7ffff640 1 -- 172.21.15.122:0/1546948694 reap_dead start 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad861b6640 1 -- 172.21.15.122:0/1546948694 shutdown_connections 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad861b6640 1 -- 172.21.15.122:0/1546948694 >> 172.21.15.122:0/1546948694 conn(0x7fad8006b190 msgr2=0x7fad8006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad861b6640 1 -- 172.21.15.122:0/1546948694 shutdown_connections 2024-03-06T07:23:39.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.005+0000 7fad861b6640 1 -- 172.21.15.122:0/1546948694 wait complete. 2024-03-06T07:23:39.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.006+0000 7fad861b6640 1 Processor -- start 2024-03-06T07:23:39.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.006+0000 7fad861b6640 1 -- start start 2024-03-06T07:23:39.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad861b6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad68004800 0x7fad8014ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad861b6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad801f70e0 0x7fad801f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad861b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 0x7fad8006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad861b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad800d0360 con 0x7fad8006cbb0 2024-03-06T07:23:39.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad861b6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8015bde0 con 0x7fad801f70e0 2024-03-06T07:23:39.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad861b6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad800af150 con 0x7fad68004800 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad7ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 0x7fad8006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad7ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 0x7fad8006d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35816/0 (socket says 172.21.15.122:35816) 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad7ffff640 1 -- 172.21.15.122:0/1799258915 learned_addr learned my addr 172.21.15.122:0/1799258915 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad7effd640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad801f70e0 0x7fad801f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad7f7fe640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad68004800 0x7fad8014ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.007+0000 7fad7ffff640 1 -- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad68004800 msgr2=0x7fad8014ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad7ffff640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad68004800 0x7fad8014ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad7ffff640 1 -- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad801f70e0 msgr2=0x7fad801f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad7ffff640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad801f70e0 0x7fad801f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad7ffff640 1 -- 172.21.15.122:0/1799258915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8006d8e0 con 0x7fad8006cbb0 2024-03-06T07:23:39.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad7effd640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad801f70e0 0x7fad801f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad7ffff640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 0x7fad8006d030 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fad800d0910 tx=0x7fad78009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7801a030 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad78010e80 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8006dbc0 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad78003bd0 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.008+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8006e180 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.009+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad800732b0 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.009+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad7801d070 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.010+0000 7fad67fff640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad6c06b4d0 0x7fad6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.010+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fad7808a750 con 0x7fad8006cbb0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.010+0000 7fad7f7fe640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad6c06b4d0 0x7fad6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.010+0000 7fad7f7fe640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad6c06b4d0 0x7fad6c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad70005590 tx=0x7fad70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.014+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad78059820 con 0x7fad8006cbb0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.186+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fad80068ff0 con 0x7fad8006cbb0 2024-03-06T07:23:39.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.885+0000 7fad67fff640 1 -- 172.21.15.122:0/1799258915 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7fad78059120 con 0x7fad8006cbb0 2024-03-06T07:23:39.887 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad6c06b4d0 msgr2=0x7fad6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad6c06b4d0 0x7fad6c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad70005590 tx=0x7fad70009040 comp rx=0 tx=0).stop 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 msgr2=0x7fad8006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 0x7fad8006d030 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fad800d0910 tx=0x7fad78009f90 comp rx=0 tx=0).stop 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 shutdown_connections 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad6c06b4d0 0x7fad6c06d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad8006cbb0 0x7fad8006d030 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad801f70e0 0x7fad801f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 --2- 172.21.15.122:0/1799258915 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad68004800 0x7fad8014ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.888+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 >> 172.21.15.122:0/1799258915 conn(0x7fad8006b190 msgr2=0x7fad801577f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.889+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 shutdown_connections 2024-03-06T07:23:39.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:39.889+0000 7fad861b6640 1 -- 172.21.15.122:0/1799258915 wait complete. 2024-03-06T07:23:39.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:40.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.020+0000 7f06110c3640 1 Processor -- start 2024-03-06T07:23:40.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.020+0000 7f06110c3640 1 -- start start 2024-03-06T07:23:40.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f06110c3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c158420 0x7f060c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f06110c3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f060c15ba80 0x7f060c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f06110c3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c15e8a0 con 0x7f060c158800 2024-03-06T07:23:40.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f06110c3640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060c15e650 con 0x7f060c15ba80 2024-03-06T07:23:40.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f06110c3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c15eb80 con 0x7f060c158420 2024-03-06T07:23:40.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060b59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f060c15ba80 0x7f060c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060a59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f060c158800 0x7f060c15b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51442/0 (socket says 172.21.15.122:51442) 2024-03-06T07:23:40.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060a59c640 1 -- 172.21.15.122:0/387196313 learned_addr learned my addr 172.21.15.122:0/387196313 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:40.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060b59e640 1 --2- 172.21.15.122:0/387196313 >> v2:172.21.15.122:3300/0 conn(0x7f060c15ba80 0x7f060c0b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35830/0 (socket says 172.21.15.122:35830) 2024-03-06T07:23:40.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060b59e640 1 -- 172.21.15.122:0/387196313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c158420 msgr2=0x7f060c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060ad9d640 1 --2- 172.21.15.122:0/387196313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c158420 0x7f060c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f0609d9b640 1 -- 172.21.15.122:0/387196313 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953127882 0 0) 0x7f060c15e8a0 con 0x7f060c158800 2024-03-06T07:23:40.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060b59e640 1 --2- 172.21.15.122:0/387196313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c158420 0x7f060c0b35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060b59e640 1 -- 172.21.15.122:0/387196313 >> v1:172.21.15.122:6789/0 conn(0x7f060c158800 legacy=0x7f060c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.021+0000 7f060b59e640 1 -- 172.21.15.122:0/387196313 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c15ee60 con 0x7f060c15ba80 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.022+0000 7f060b59e640 1 --2- 172.21.15.122:0/387196313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c15ba80 0x7f060c0b3b00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f060400ba40 tx=0x7f060402dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd9e91c8f4a69bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.022+0000 7f0609d9b640 1 -- 172.21.15.122:0/387196313 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060403b070 con 0x7f060c15ba80 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.022+0000 7f0609d9b640 1 -- 172.21.15.122:0/387196313 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060402eda0 con 0x7f060c15ba80 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.022+0000 7f0609d9b640 1 -- 172.21.15.122:0/387196313 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604036660 con 0x7f060c15ba80 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 -- 172.21.15.122:0/387196313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c15ba80 msgr2=0x7f060c0b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 --2- 172.21.15.122:0/387196313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c15ba80 0x7f060c0b3b00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f060400ba40 tx=0x7f060402dff0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 -- 172.21.15.122:0/387196313 shutdown_connections 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 --2- 172.21.15.122:0/387196313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c15ba80 0x7f060c0b3b00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 --2- 172.21.15.122:0/387196313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c158420 0x7f060c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 -- 172.21.15.122:0/387196313 >> 172.21.15.122:0/387196313 conn(0x7f060c06afb0 msgr2=0x7f060c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 -- 172.21.15.122:0/387196313 shutdown_connections 2024-03-06T07:23:40.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.023+0000 7f06110c3640 1 -- 172.21.15.122:0/387196313 wait complete. 2024-03-06T07:23:40.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.024+0000 7f06110c3640 1 Processor -- start 2024-03-06T07:23:40.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.024+0000 7f06110c3640 1 -- start start 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f06110c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 0x7f060c1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f06110c3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f060c15ba80 0x7f060c0af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f06110c3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c0b0f60 0x7f060c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f06110c3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06cb80 con 0x7f060c158420 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f06110c3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06ca00 con 0x7f060c15ba80 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f06110c3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06cd00 con 0x7f060c0b0f60 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 0x7f060c1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 0x7f060c1527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35844/0 (socket says 172.21.15.122:35844) 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 -- 172.21.15.122:0/2799163339 learned_addr learned my addr 172.21.15.122:0/2799163339 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060b59e640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c0b0f60 0x7f060c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060a59c640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f060c15ba80 0x7f060c0af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 -- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c0b0f60 msgr2=0x7f060c153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c0b0f60 0x7f060c153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 -- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f060c15ba80 msgr2=0x7f060c0af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f060c15ba80 0x7f060c0af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.025+0000 7f060ad9d640 1 -- 172.21.15.122:0/2799163339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c0afec0 con 0x7f060c158420 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.026+0000 7f060ad9d640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 0x7f060c1527e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00ce10 tx=0x7f05fc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.026+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc003b90 con 0x7f060c158420 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.026+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05fc003d30 con 0x7f060c158420 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.026+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c0b01a0 con 0x7f060c158420 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.026+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc004240 con 0x7f060c158420 2024-03-06T07:23:40.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.026+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c0b0760 con 0x7f060c158420 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.027+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c04f330 con 0x7f060c158420 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.027+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05fc0043e0 con 0x7f060c158420 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.028+0000 7f05f37fe640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05f806b4d0 0x7f05f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.028+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f05fc08ad00 con 0x7f060c158420 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.028+0000 7f060a59c640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05f806b4d0 0x7f05f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.028+0000 7f060a59c640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05f806b4d0 0x7f05f806d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f060c15ad00 tx=0x7f0600007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.032+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05fc01d340 con 0x7f060c158420 2024-03-06T07:23:40.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.241+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f060c0b2a30 con 0x7f060c158420 2024-03-06T07:23:40.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.887+0000 7f05f37fe640 1 -- 172.21.15.122:0/2799163339 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v19) v1 ==== 172+0+0 (secure 0 0 0) 0x7f05fc059be0 con 0x7f060c158420 2024-03-06T07:23:40.889 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:23:40.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05f806b4d0 msgr2=0x7f05f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05f806b4d0 0x7f05f806d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f060c15ad00 tx=0x7f0600007b90 comp rx=0 tx=0).stop 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 msgr2=0x7f060c1527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 0x7f060c1527e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00ce10 tx=0x7f05fc00a660 comp rx=0 tx=0).stop 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 shutdown_connections 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05f806b4d0 0x7f05f806d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f060c0b0f60 0x7f060c153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f060c15ba80 0x7f060c0af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 --2- 172.21.15.122:0/2799163339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f060c158420 0x7f060c1527e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.890+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 >> 172.21.15.122:0/2799163339 conn(0x7f060c06afb0 msgr2=0x7f060c155c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.891+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 shutdown_connections 2024-03-06T07:23:40.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:40.891+0000 7f06110c3640 1 -- 172.21.15.122:0/2799163339 wait complete. 2024-03-06T07:23:40.925 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:41.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.025+0000 7f6365181640 1 Processor -- start 2024-03-06T07:23:41.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.025+0000 7f6365181640 1 -- start start 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f6365181640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f63601424f0 0x7f63600cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f6365181640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63600cd7a0 0x7f636015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f6365181640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600cc980 con 0x7f636006d330 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f6365181640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600cd220 con 0x7f63600cd7a0 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f6365181640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600ccc60 con 0x7f63601424f0 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63600cd7a0 0x7f636015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63600cd7a0 0x7f636015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35848/0 (socket says 172.21.15.122:35848) 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 -- 172.21.15.122:0/3269305017 learned_addr learned my addr 172.21.15.122:0/3269305017 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 -- 172.21.15.122:0/3269305017 >> v2:172.21.15.169:3300/0 conn(0x7f63601424f0 msgr2=0x7f63600cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 --2- 172.21.15.122:0/3269305017 >> v2:172.21.15.169:3300/0 conn(0x7f63601424f0 0x7f63600cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 -- 172.21.15.122:0/3269305017 >> v1:172.21.15.169:6789/0 conn(0x7f636006d330 legacy=0x7f6360071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.026+0000 7f635e59c640 1 -- 172.21.15.122:0/3269305017 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600ccf40 con 0x7f63600cd7a0 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.027+0000 7f635e59c640 1 --2- 172.21.15.122:0/3269305017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63600cd7a0 0x7f636015cef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f636006c6d0 tx=0x7f635402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76df03eafb7a251b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.027+0000 7f635dd9b640 1 -- 172.21.15.122:0/3269305017 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354035020 con 0x7f63600cd7a0 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.027+0000 7f635dd9b640 1 -- 172.21.15.122:0/3269305017 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6354031870 con 0x7f63600cd7a0 2024-03-06T07:23:41.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.027+0000 7f635dd9b640 1 -- 172.21.15.122:0/3269305017 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354031c40 con 0x7f63600cd7a0 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 -- 172.21.15.122:0/3269305017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63600cd7a0 msgr2=0x7f636015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 --2- 172.21.15.122:0/3269305017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63600cd7a0 0x7f636015cef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f636006c6d0 tx=0x7f635402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 -- 172.21.15.122:0/3269305017 shutdown_connections 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 --2- 172.21.15.122:0/3269305017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63600cd7a0 0x7f636015cef0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 --2- 172.21.15.122:0/3269305017 >> v2:172.21.15.169:3300/0 conn(0x7f63601424f0 0x7f63600cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 -- 172.21.15.122:0/3269305017 >> 172.21.15.122:0/3269305017 conn(0x7f63600cf8a0 msgr2=0x7f63600cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 -- 172.21.15.122:0/3269305017 shutdown_connections 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.028+0000 7f6365181640 1 -- 172.21.15.122:0/3269305017 wait complete. 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.029+0000 7f6365181640 1 Processor -- start 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.029+0000 7f6365181640 1 -- start start 2024-03-06T07:23:41.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f6365181640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 0x7f63600aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f6365181640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63600cd7a0 0x7f636013f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f6365181640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63601424f0 0x7f63600b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f6365181640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636006d920 con 0x7f636006d330 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f6365181640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636006d7a0 con 0x7f63600cd7a0 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f6365181640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636006daa0 con 0x7f63601424f0 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 0x7f63600aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 0x7f63600aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35852/0 (socket says 172.21.15.122:35852) 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 -- 172.21.15.122:0/372215544 learned_addr learned my addr 172.21.15.122:0/372215544 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635e59c640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63600cd7a0 0x7f636013f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635f59e640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63601424f0 0x7f63600b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 -- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63601424f0 msgr2=0x7f63600b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63601424f0 0x7f63600b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 -- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63600cd7a0 msgr2=0x7f636013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63600cd7a0 0x7f636013f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 -- 172.21.15.122:0/372215544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600b2af0 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635f59e640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63601424f0 0x7f63600b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.030+0000 7f635ed9d640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 0x7f63600aefd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f635000ba80 tx=0x7f635000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.031+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350019070 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.031+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635000cc20 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.031+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6360154560 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.031+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350014440 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.031+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6360154840 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.032+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636004f330 con 0x7f636006d330 2024-03-06T07:23:41.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.032+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f635001c070 con 0x7f636006d330 2024-03-06T07:23:41.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.032+0000 7f63477fe640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f634c06b4d0 0x7f634c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.032+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6350089e40 con 0x7f636006d330 2024-03-06T07:23:41.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.033+0000 7f635e59c640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f634c06b4d0 0x7f634c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.033+0000 7f635e59c640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f634c06b4d0 0x7f634c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6354036000 tx=0x7f635403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.037+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6350058670 con 0x7f636006d330 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.210+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63600d0620 con 0x7f636006d330 2024-03-06T07:23:41.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.211+0000 7f63477fe640 1 -- 172.21.15.122:0/372215544 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f635005c710 con 0x7f636006d330 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":44,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:40.242874+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f634c06b4d0 msgr2=0x7f634c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f634c06b4d0 0x7f634c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6354036000 tx=0x7f635403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 msgr2=0x7f63600aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 0x7f63600aefd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f635000ba80 tx=0x7f635000bf40 comp rx=0 tx=0).stop 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 shutdown_connections 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f634c06b4d0 0x7f634c06d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63601424f0 0x7f63600b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63600cd7a0 0x7f636013f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 --2- 172.21.15.122:0/372215544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f636006d330 0x7f63600aefd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.213+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 >> 172.21.15.122:0/372215544 conn(0x7f63600cf8a0 msgr2=0x7f636015d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.214+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 shutdown_connections 2024-03-06T07:23:41.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.214+0000 7f6365181640 1 -- 172.21.15.122:0/372215544 wait complete. 2024-03-06T07:23:41.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:41.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.324+0000 7f6659356640 1 Processor -- start 2024-03-06T07:23:41.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.324+0000 7f6659356640 1 -- start start 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6659356640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f66541424f0 0x7f66540cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6659356640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f66540cd7a0 0x7f665415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6659356640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540cc980 con 0x7f665406d330 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6659356640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540cd220 con 0x7f66540cd7a0 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6659356640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540ccc60 con 0x7f66541424f0 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f66540cd7a0 0x7f665415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f66540cd7a0 0x7f665415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35860/0 (socket says 172.21.15.122:35860) 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66537fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f665406d330 0x7f6654071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51446/0 (socket says 172.21.15.122:51446) 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 -- 172.21.15.122:0/2824154230 learned_addr learned my addr 172.21.15.122:0/2824154230 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6652ffd640 1 --2- 172.21.15.122:0/2824154230 >> v2:172.21.15.169:3300/0 conn(0x7f66541424f0 0x7f66540cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 -- 172.21.15.122:0/2824154230 >> v2:172.21.15.169:3300/0 conn(0x7f66541424f0 msgr2=0x7f66540cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 --2- 172.21.15.122:0/2824154230 >> v2:172.21.15.169:3300/0 conn(0x7f66541424f0 0x7f66540cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 -- 172.21.15.122:0/2824154230 >> v1:172.21.15.122:6789/0 conn(0x7f665406d330 legacy=0x7f6654071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f66527fc640 1 -- 172.21.15.122:0/2824154230 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66540ccf40 con 0x7f66540cd7a0 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.325+0000 7f6651ffb640 1 -- 172.21.15.122:0/2824154230 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487282108 0 0) 0x7f66540cc980 con 0x7f665406d330 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.326+0000 7f66527fc640 1 --2- 172.21.15.122:0/2824154230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f665415cef0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f665406c6d0 tx=0x7f664802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1b5001c5d8f2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.326+0000 7f6651ffb640 1 -- 172.21.15.122:0/2824154230 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648034070 con 0x7f66540cd7a0 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.326+0000 7f6651ffb640 1 -- 172.21.15.122:0/2824154230 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648030860 con 0x7f66540cd7a0 2024-03-06T07:23:41.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.326+0000 7f6651ffb640 1 -- 172.21.15.122:0/2824154230 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648030c30 con 0x7f66540cd7a0 2024-03-06T07:23:41.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 -- 172.21.15.122:0/2824154230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 msgr2=0x7f665415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 --2- 172.21.15.122:0/2824154230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f665415cef0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f665406c6d0 tx=0x7f664802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 -- 172.21.15.122:0/2824154230 shutdown_connections 2024-03-06T07:23:41.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 --2- 172.21.15.122:0/2824154230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f665415cef0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 --2- 172.21.15.122:0/2824154230 >> v2:172.21.15.169:3300/0 conn(0x7f66541424f0 0x7f66540cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 -- 172.21.15.122:0/2824154230 >> 172.21.15.122:0/2824154230 conn(0x7f66540cf8a0 msgr2=0x7f66540cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 -- 172.21.15.122:0/2824154230 shutdown_connections 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.327+0000 7f6659356640 1 -- 172.21.15.122:0/2824154230 wait complete. 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.328+0000 7f6659356640 1 Processor -- start 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.328+0000 7f6659356640 1 -- start start 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6659356640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f665406d330 0x7f66540ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6659356640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f66540b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6659356640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66541424f0 0x7f66540acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6659356640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665406d920 con 0x7f66540cd7a0 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6659356640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665406d7a0 con 0x7f66541424f0 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6659356640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665406daa0 con 0x7f665406d330 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f66540b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f66540b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35874/0 (socket says 172.21.15.122:35874) 2024-03-06T07:23:41.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 -- 172.21.15.122:0/2811359893 learned_addr learned my addr 172.21.15.122:0/2811359893 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66537fe640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66541424f0 0x7f66540acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f6652ffd640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f665406d330 0x7f66540ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 -- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f665406d330 msgr2=0x7f66540ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f665406d330 0x7f66540ab400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 -- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66541424f0 msgr2=0x7f66540acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66541424f0 0x7f66540acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 -- 172.21.15.122:0/2811359893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6654157f10 con 0x7f66540cd7a0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66537fe640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66541424f0 0x7f66540acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.329+0000 7f66527fc640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f66540b0880 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f664802b9e0 tx=0x7f6648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.330+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648048070 con 0x7f66540cd7a0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.330+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648030660 con 0x7f66540cd7a0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.330+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6654159730 con 0x7f66540cd7a0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.330+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648034040 con 0x7f66540cd7a0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.330+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6654159cf0 con 0x7f66540cd7a0 2024-03-06T07:23:41.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.331+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66541580e0 con 0x7f66540cd7a0 2024-03-06T07:23:41.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.331+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6648037070 con 0x7f66540cd7a0 2024-03-06T07:23:41.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.331+0000 7f663b7fe640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.332+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66480ac010 con 0x7f66540cd7a0 2024-03-06T07:23:41.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.332+0000 7f6652ffd640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.332+0000 7f6652ffd640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f664006b4d0 0x7f664006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f66440099c0 tx=0x7f6644003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.336+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f664807b040 con 0x7f66540cd7a0 2024-03-06T07:23:41.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.508+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66540709f0 con 0x7f66540cd7a0 2024-03-06T07:23:41.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.509+0000 7f663b7fe640 1 -- 172.21.15.122:0/2811359893 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f664807a940 con 0x7f66540cd7a0 2024-03-06T07:23:41.511 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:41.511 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":44,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:40.242874+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f664006b4d0 msgr2=0x7f664006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f664006b4d0 0x7f664006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f66440099c0 tx=0x7f6644003990 comp rx=0 tx=0).stop 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 msgr2=0x7f66540b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f66540b0880 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f664802b9e0 tx=0x7f6648004730 comp rx=0 tx=0).stop 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 shutdown_connections 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66541424f0 0x7f66540acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66540cd7a0 0x7f66540b0880 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 --2- 172.21.15.122:0/2811359893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f665406d330 0x7f66540ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 >> 172.21.15.122:0/2811359893 conn(0x7f66540cf8a0 msgr2=0x7f665415cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 shutdown_connections 2024-03-06T07:23:41.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.511+0000 7f6659356640 1 -- 172.21.15.122:0/2811359893 wait complete. 2024-03-06T07:23:41.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:41.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.626+0000 7fc29bd93640 1 Processor -- start 2024-03-06T07:23:41.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.626+0000 7fc29bd93640 1 -- start start 2024-03-06T07:23:41.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29bd93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2940b4970 0x7fc29406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29bd93640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc29406d5c0 0x7fc2940cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29bd93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29415ff90 con 0x7fc29415bb40 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29bd93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2940b4dd0 con 0x7fc2940b4970 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29bd93640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29406c910 con 0x7fc29406d5c0 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2940b4970 0x7fc29406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2940b4970 0x7fc29406d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35884/0 (socket says 172.21.15.122:35884) 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 -- 172.21.15.122:0/2037711172 learned_addr learned my addr 172.21.15.122:0/2037711172 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29a330640 1 --2- 172.21.15.122:0/2037711172 >> v2:172.21.15.169:3300/0 conn(0x7fc29406d5c0 0x7fc2940cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 -- 172.21.15.122:0/2037711172 >> v2:172.21.15.169:3300/0 conn(0x7fc29406d5c0 msgr2=0x7fc2940cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 --2- 172.21.15.122:0/2037711172 >> v2:172.21.15.169:3300/0 conn(0x7fc29406d5c0 0x7fc2940cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 -- 172.21.15.122:0/2037711172 >> v1:172.21.15.169:6789/0 conn(0x7fc29415bb40 legacy=0x7fc29415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29932e640 1 -- 172.21.15.122:0/2037711172 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294160270 con 0x7fc2940b4970 2024-03-06T07:23:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.627+0000 7fc29a330640 1 --2- 172.21.15.122:0/2037711172 >> v2:172.21.15.169:3300/0 conn(0x7fc29406d5c0 0x7fc2940cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.628+0000 7fc29932e640 1 --2- 172.21.15.122:0/2037711172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2940b4970 0x7fc29406d080 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc294072c60 tx=0x7fc28402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6267adef802fd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.628+0000 7fc298b2d640 1 -- 172.21.15.122:0/2037711172 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284034070 con 0x7fc2940b4970 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.628+0000 7fc298b2d640 1 -- 172.21.15.122:0/2037711172 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284030860 con 0x7fc2940b4970 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.628+0000 7fc298b2d640 1 -- 172.21.15.122:0/2037711172 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284030c30 con 0x7fc2940b4970 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 -- 172.21.15.122:0/2037711172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2940b4970 msgr2=0x7fc29406d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 --2- 172.21.15.122:0/2037711172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2940b4970 0x7fc29406d080 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc294072c60 tx=0x7fc28402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 -- 172.21.15.122:0/2037711172 shutdown_connections 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 --2- 172.21.15.122:0/2037711172 >> v2:172.21.15.169:3300/0 conn(0x7fc29406d5c0 0x7fc2940cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 --2- 172.21.15.122:0/2037711172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2940b4970 0x7fc29406d080 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 -- 172.21.15.122:0/2037711172 >> 172.21.15.122:0/2037711172 conn(0x7fc29406b190 msgr2=0x7fc29406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 -- 172.21.15.122:0/2037711172 shutdown_connections 2024-03-06T07:23:41.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.629+0000 7fc29bd93640 1 -- 172.21.15.122:0/2037711172 wait complete. 2024-03-06T07:23:41.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.630+0000 7fc29bd93640 1 Processor -- start 2024-03-06T07:23:41.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.630+0000 7fc29bd93640 1 -- start start 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc29406d5c0 0x7fc2941f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2940b4970 0x7fc294150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 0x7fc2941e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29415bf20 con 0x7fc29415bb40 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29406cb60 con 0x7fc2940b4970 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29415c0a0 con 0x7fc29406d5c0 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 0x7fc2941e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 0x7fc2941e6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35898/0 (socket says 172.21.15.122:35898) 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 -- 172.21.15.122:0/45060496 learned_addr learned my addr 172.21.15.122:0/45060496 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 -- 172.21.15.122:0/45060496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc29406d5c0 msgr2=0x7fc2941f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc29406d5c0 0x7fc2941f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 -- 172.21.15.122:0/45060496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2940b4970 msgr2=0x7fc294150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2940b4970 0x7fc294150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 -- 172.21.15.122:0/45060496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2941f60e0 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29a330640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 0x7fc2941e6820 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009fa0 tx=0x7fc28c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c01a070 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c003bb0 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2941e5700 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c004030 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.631+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2941e6020 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.632+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2940732b0 con 0x7fc29415bb40 2024-03-06T07:23:41.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.633+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc28c01d070 con 0x7fc29415bb40 2024-03-06T07:23:41.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.633+0000 7fc28a7fc640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27806b4d0 0x7fc27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.634+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc28c08a480 con 0x7fc29415bb40 2024-03-06T07:23:41.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.634+0000 7fc299b2f640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27806b4d0 0x7fc27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.634+0000 7fc299b2f640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27806b4d0 0x7fc27806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc290000f60 tx=0x7fc2900079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.638+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2941f60e0 con 0x7fc29415bb40 2024-03-06T07:23:41.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.846+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc29406e130 con 0x7fc29415bb40 2024-03-06T07:23:41.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.899+0000 7fc28a7fc640 1 -- 172.21.15.122:0/45060496 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v20) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc28c010070 con 0x7fc29415bb40 2024-03-06T07:23:41.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27806b4d0 msgr2=0x7fc27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27806b4d0 0x7fc27806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc290000f60 tx=0x7fc2900079d0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 msgr2=0x7fc2941e6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 0x7fc2941e6820 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009fa0 tx=0x7fc28c00d560 comp rx=0 tx=0).stop 2024-03-06T07:23:41.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 shutdown_connections 2024-03-06T07:23:41.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27806b4d0 0x7fc27806d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29415bb40 0x7fc2941e6820 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2940b4970 0x7fc294150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 --2- 172.21.15.122:0/45060496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc29406d5c0 0x7fc2941f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 >> 172.21.15.122:0/45060496 conn(0x7fc29406b190 msgr2=0x7fc2940b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 shutdown_connections 2024-03-06T07:23:41.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:41.902+0000 7fc29bd93640 1 -- 172.21.15.122:0/45060496 wait complete. 2024-03-06T07:23:41.915 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:41.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.015+0000 7f4a0cd56640 1 Processor -- start 2024-03-06T07:23:42.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.015+0000 7f4a0cd56640 1 -- start start 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0cd56640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a081614d0 0x7f4a080d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0cd56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a080d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0cd56640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4a080d22b0 0x7f4a080d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0cd56640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a080d2740 con 0x7f4a081614d0 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0cd56640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a081618b0 con 0x7f4a080d22b0 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0cd56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080cf400 con 0x7f4a080d1910 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a081614d0 0x7f4a080d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a081614d0 0x7f4a080d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35914/0 (socket says 172.21.15.122:35914) 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.016+0000 7f4a0659c640 1 -- 172.21.15.122:0/3486746575 learned_addr learned my addr 172.21.15.122:0/3486746575 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:42.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a06d9d640 1 --2- 172.21.15.122:0/3486746575 >> v2:172.21.15.169:3300/0 conn(0x7f4a080d22b0 0x7f4a080d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a05d9b640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a080d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a0659c640 1 -- 172.21.15.122:0/3486746575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 msgr2=0x7f4a080d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a0659c640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a080d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a0659c640 1 -- 172.21.15.122:0/3486746575 >> v2:172.21.15.169:3300/0 conn(0x7f4a080d22b0 msgr2=0x7f4a080d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a0659c640 1 --2- 172.21.15.122:0/3486746575 >> v2:172.21.15.169:3300/0 conn(0x7f4a080d22b0 0x7f4a080d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a0659c640 1 -- 172.21.15.122:0/3486746575 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a080cf6a0 con 0x7f4a081614d0 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a05d9b640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a080d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:42.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a06d9d640 1 --2- 172.21.15.122:0/3486746575 >> v2:172.21.15.169:3300/0 conn(0x7f4a080d22b0 0x7f4a080d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.017+0000 7f4a0659c640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a081614d0 0x7f4a080d4860 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4a080b57d0 tx=0x7f49f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655dd5fe51a285a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0559a640 1 -- 172.21.15.122:0/3486746575 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f802ca70 con 0x7f4a081614d0 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0559a640 1 -- 172.21.15.122:0/3486746575 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f802cc10 con 0x7f4a081614d0 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0559a640 1 -- 172.21.15.122:0/3486746575 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f80343f0 con 0x7f4a081614d0 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 -- 172.21.15.122:0/3486746575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a081614d0 msgr2=0x7f4a080d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a081614d0 0x7f4a080d4860 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4a080b57d0 tx=0x7f49f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 -- 172.21.15.122:0/3486746575 shutdown_connections 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/3486746575 >> v2:172.21.15.169:3300/0 conn(0x7f4a080d22b0 0x7f4a080d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a080d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/3486746575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a081614d0 0x7f4a080d4860 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 -- 172.21.15.122:0/3486746575 >> 172.21.15.122:0/3486746575 conn(0x7f4a080d3d40 msgr2=0x7f4a080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 -- 172.21.15.122:0/3486746575 shutdown_connections 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.018+0000 7f4a0cd56640 1 -- 172.21.15.122:0/3486746575 wait complete. 2024-03-06T07:23:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.019+0000 7f4a0cd56640 1 Processor -- start 2024-03-06T07:23:42.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 -- start start 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a081f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 0x7f4a081f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a081614d0 0x7f4a081584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080b0910 con 0x7f4a080d22b0 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080cfc60 con 0x7f4a081614d0 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0cd56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080d0be0 con 0x7f4a080d1910 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a05d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 0x7f4a081f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a05d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 0x7f4a081f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35922/0 (socket says 172.21.15.122:35922) 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a06d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a081614d0 0x7f4a081584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a05d9b640 1 -- 172.21.15.122:0/4088046606 learned_addr learned my addr 172.21.15.122:0/4088046606 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.020+0000 7f4a0659c640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a081f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a05d9b640 1 -- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 msgr2=0x7f4a081f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a05d9b640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a081f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a05d9b640 1 -- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a081614d0 msgr2=0x7f4a081584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a05d9b640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a081614d0 0x7f4a081584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a05d9b640 1 -- 172.21.15.122:0/4088046606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08158a10 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a05d9b640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 0x7f4a081f7aa0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00e810 tx=0x7f49fc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00f040 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc004030 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a080cb2a0 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004400 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.021+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a080cb860 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.022+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a080a1c90 con 0x7f4a080d22b0 2024-03-06T07:23:42.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.022+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49fc00c6d0 con 0x7f4a080d22b0 2024-03-06T07:23:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.023+0000 7f49f6ffd640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e806b500 0x7f49e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.023+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49fc089ea0 con 0x7f4a080d22b0 2024-03-06T07:23:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.023+0000 7f4a0659c640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e806b500 0x7f49e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.023+0000 7f4a0659c640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e806b500 0x7f49e806d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f49f8035000 tx=0x7f49f8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.027+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49fc017020 con 0x7f4a080d22b0 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.236+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4a08068ff0 con 0x7f4a080d22b0 2024-03-06T07:23:42.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.904+0000 7f49f6ffd640 1 -- 172.21.15.122:0/4088046606 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v21) v1 ==== 122+0+0 (secure 0 0 0) 0x7f49fc0586d0 con 0x7f4a080d22b0 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e806b500 msgr2=0x7f49e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e806b500 0x7f49e806d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f49f8035000 tx=0x7f49f8030000 comp rx=0 tx=0).stop 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 msgr2=0x7f4a081f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 0x7f4a081f7aa0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00e810 tx=0x7f49fc00ecd0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 shutdown_connections 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e806b500 0x7f49e806d9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a081614d0 0x7f4a081584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a080d22b0 0x7f4a081f7aa0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 --2- 172.21.15.122:0/4088046606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a080d1910 0x7f4a081f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 >> 172.21.15.122:0/4088046606 conn(0x7f4a080d3d40 msgr2=0x7f4a0806d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 shutdown_connections 2024-03-06T07:23:42.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:42.907+0000 7f4a0cd56640 1 -- 172.21.15.122:0/4088046606 wait complete. 2024-03-06T07:23:42.922 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:42.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:43.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.024+0000 7f35a5f5e640 1 Processor -- start 2024-03-06T07:23:43.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.024+0000 7f35a5f5e640 1 -- start start 2024-03-06T07:23:43.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f35a5f5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a01515e0 0x7f35a00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f35a5f5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f35a015b430 0x7f35a0071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f35a5f5e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a015b8b0 con 0x7f35a00cd7a0 2024-03-06T07:23:43.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f35a5f5e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a0070a70 con 0x7f35a015b430 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f35a5f5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0070d50 con 0x7f35a01515e0 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f359effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f35a015b430 0x7f35a0071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f359effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f35a015b430 0x7f35a0071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35926/0 (socket says 172.21.15.122:35926) 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f359effd640 1 -- 172.21.15.122:0/3950293227 learned_addr learned my addr 172.21.15.122:0/3950293227 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.025+0000 7f359f7fe640 1 --2- 172.21.15.122:0/3950293227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a01515e0 0x7f35a00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.026+0000 7f359effd640 1 -- 172.21.15.122:0/3950293227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a01515e0 msgr2=0x7f35a00cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.026+0000 7f359effd640 1 --2- 172.21.15.122:0/3950293227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a01515e0 0x7f35a00cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.026+0000 7f359effd640 1 -- 172.21.15.122:0/3950293227 >> v1:172.21.15.169:6789/0 conn(0x7f35a00cd7a0 legacy=0x7f35a00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.026+0000 7f359effd640 1 -- 172.21.15.122:0/3950293227 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0071030 con 0x7f35a015b430 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.026+0000 7f359effd640 1 --2- 172.21.15.122:0/3950293227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a015b430 0x7f35a0071830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3594009870 tx=0x7f359402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4f36d5d7f60c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.026+0000 7f359e7fc640 1 -- 172.21.15.122:0/3950293227 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594034070 con 0x7f35a015b430 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f359e7fc640 1 -- 172.21.15.122:0/3950293227 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594030840 con 0x7f35a015b430 2024-03-06T07:23:43.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f359e7fc640 1 -- 172.21.15.122:0/3950293227 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594030c10 con 0x7f35a015b430 2024-03-06T07:23:43.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 -- 172.21.15.122:0/3950293227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a015b430 msgr2=0x7f35a0071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/3950293227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a015b430 0x7f35a0071830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3594009870 tx=0x7f359402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:43.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 -- 172.21.15.122:0/3950293227 shutdown_connections 2024-03-06T07:23:43.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/3950293227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a015b430 0x7f35a0071830 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/3950293227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a01515e0 0x7f35a00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 -- 172.21.15.122:0/3950293227 >> 172.21.15.122:0/3950293227 conn(0x7f35a00cf8a0 msgr2=0x7f35a00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 -- 172.21.15.122:0/3950293227 shutdown_connections 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.027+0000 7f35a5f5e640 1 -- 172.21.15.122:0/3950293227 wait complete. 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.028+0000 7f35a5f5e640 1 Processor -- start 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.028+0000 7f35a5f5e640 1 -- start start 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f35a5f5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a00cd7a0 0x7f35a00c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f35a5f5e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 0x7f35a0167220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f35a5f5e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a015b430 0x7f35a01485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f35a5f5e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0071ef0 con 0x7f35a01515e0 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f35a5f5e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0071d70 con 0x7f35a015b430 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f35a5f5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00cc3a0 con 0x7f35a00cd7a0 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f359effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 0x7f35a0167220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f359effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 0x7f35a0167220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35942/0 (socket says 172.21.15.122:35942) 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f359effd640 1 -- 172.21.15.122:0/612841518 learned_addr learned my addr 172.21.15.122:0/612841518 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f359ffff640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a015b430 0x7f35a01485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.029+0000 7f359f7fe640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a00cd7a0 0x7f35a00c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f359effd640 1 -- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a00cd7a0 msgr2=0x7f35a00c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f359effd640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a00cd7a0 0x7f35a00c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f359effd640 1 -- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a015b430 msgr2=0x7f35a01485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f359effd640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a015b430 0x7f35a01485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f359effd640 1 -- 172.21.15.122:0/612841518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0167920 con 0x7f35a01515e0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f359effd640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 0x7f35a0167220 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3594000c00 tx=0x7f35940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594034040 con 0x7f35a01515e0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594039d90 con 0x7f35a01515e0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a0167c00 con 0x7f35a01515e0 2024-03-06T07:23:43.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594043630 con 0x7f35a01515e0 2024-03-06T07:23:43.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.030+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a0148b00 con 0x7f35a01515e0 2024-03-06T07:23:43.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.031+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a0068840 con 0x7f35a01515e0 2024-03-06T07:23:43.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.031+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f359404a060 con 0x7f35a01515e0 2024-03-06T07:23:43.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.032+0000 7f3587fff640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358c06b500 0x7f358c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.032+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35940abf70 con 0x7f35a01515e0 2024-03-06T07:23:43.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.032+0000 7f359f7fe640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358c06b500 0x7f358c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.032+0000 7f359f7fe640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358c06b500 0x7f358c06d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f35a0168670 tx=0x7f3590009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.036+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f359403e070 con 0x7f35a01515e0 2024-03-06T07:23:43.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.245+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a0168120 con 0x7f35a01515e0 2024-03-06T07:23:43.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.245+0000 7f3587fff640 1 -- 172.21.15.122:0/612841518 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f359407a830 con 0x7f35a01515e0 2024-03-06T07:23:43.247 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:43.248 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:40.243216+0000","modified":"2024-03-06T07:23:42.237773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/3905558085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3905558085},{"type":"v1","addr":"172.21.15.169:6835","nonce":3905558085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":3}]} 2024-03-06T07:23:43.248 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 21 2024-03-06T07:23:43.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358c06b500 msgr2=0x7f358c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358c06b500 0x7f358c06d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f35a0168670 tx=0x7f3590009190 comp rx=0 tx=0).stop 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 msgr2=0x7f35a0167220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 0x7f35a0167220 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3594000c00 tx=0x7f35940046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 shutdown_connections 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f358c06b500 0x7f358c06d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35a015b430 0x7f35a01485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a01515e0 0x7f35a0167220 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 --2- 172.21.15.122:0/612841518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35a00cd7a0 0x7f35a00c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 >> 172.21.15.122:0/612841518 conn(0x7f35a00cf8a0 msgr2=0x7f35a014ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 shutdown_connections 2024-03-06T07:23:43.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.247+0000 7f35a5f5e640 1 -- 172.21.15.122:0/612841518 wait complete. 2024-03-06T07:23:43.261 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:43.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.359+0000 7fd58155e640 1 Processor -- start 2024-03-06T07:23:43.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.359+0000 7fd58155e640 1 -- start start 2024-03-06T07:23:43.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd58155e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd57c162020 0x7fd57c0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd58155e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0b1070 con 0x7fd57c162400 2024-03-06T07:23:43.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd58155e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0d2500 con 0x7fd57c162a00 2024-03-06T07:23:43.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd58155e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0ce9d0 con 0x7fd57c162020 2024-03-06T07:23:43.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd57c162400 0x7fd57c0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51452/0 (socket says 172.21.15.122:51452) 2024-03-06T07:23:43.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57b7fe640 1 -- 172.21.15.122:0/2862345860 learned_addr learned my addr 172.21.15.122:0/2862345860 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57affd640 1 --2- 172.21.15.122:0/2862345860 >> v2:172.21.15.169:3300/0 conn(0x7fd57c162020 0x7fd57c0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57a7fc640 1 -- 172.21.15.122:0/2862345860 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027685054 0 0) 0x7fd57c0b1070 con 0x7fd57c162400 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57a7fc640 1 -- 172.21.15.122:0/2862345860 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd568003520 con 0x7fd57c162400 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57affd640 1 -- 172.21.15.122:0/2862345860 >> v1:172.21.15.169:6789/0 conn(0x7fd57c162a00 legacy=0x7fd57c0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57affd640 1 -- 172.21.15.122:0/2862345860 >> v1:172.21.15.122:6789/0 conn(0x7fd57c162400 legacy=0x7fd57c0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.360+0000 7fd57affd640 1 -- 172.21.15.122:0/2862345860 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0d27e0 con 0x7fd57c162020 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.361+0000 7fd57affd640 1 --2- 172.21.15.122:0/2862345860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162020 0x7fd57c0d4830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0097a0 tx=0x7fd56c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9504b95e7e70ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.361+0000 7fd57a7fc640 1 -- 172.21.15.122:0/2862345860 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c02f810 con 0x7fd57c162020 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.361+0000 7fd57a7fc640 1 -- 172.21.15.122:0/2862345860 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c02f9b0 con 0x7fd57c162020 2024-03-06T07:23:43.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.361+0000 7fd57a7fc640 1 -- 172.21.15.122:0/2862345860 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c02fd80 con 0x7fd57c162020 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 -- 172.21.15.122:0/2862345860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162020 msgr2=0x7fd57c0d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 --2- 172.21.15.122:0/2862345860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162020 0x7fd57c0d4830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0097a0 tx=0x7fd56c02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 -- 172.21.15.122:0/2862345860 shutdown_connections 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 --2- 172.21.15.122:0/2862345860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162020 0x7fd57c0d4830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 -- 172.21.15.122:0/2862345860 >> 172.21.15.122:0/2862345860 conn(0x7fd57c0d3d10 msgr2=0x7fd57c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 -- 172.21.15.122:0/2862345860 shutdown_connections 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.362+0000 7fd58155e640 1 -- 172.21.15.122:0/2862345860 wait complete. 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.363+0000 7fd58155e640 1 Processor -- start 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.363+0000 7fd58155e640 1 -- start start 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd58155e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 0x7fd57c14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd58155e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162400 0x7fd57c14d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd58155e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd57c162a00 0x7fd57c0cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd58155e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0cec00 con 0x7fd57c162020 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd58155e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0d2da0 con 0x7fd57c162400 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd58155e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0cfb10 con 0x7fd57c162a00 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 0x7fd57c14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 0x7fd57c14d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35952/0 (socket says 172.21.15.122:35952) 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 -- 172.21.15.122:0/3199068199 learned_addr learned my addr 172.21.15.122:0/3199068199 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57b7fe640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd57c162a00 0x7fd57c0cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd573fff640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162400 0x7fd57c14d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 -- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd57c162a00 msgr2=0x7fd57c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd57c162a00 0x7fd57c0cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 -- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162400 msgr2=0x7fd57c14d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162400 0x7fd57c14d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 -- 172.21.15.122:0/3199068199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0b88c0 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd57affd640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 0x7fd57c14d0d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0098c0 tx=0x7fd56c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.364+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c047070 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.365+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c0378d0 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.365+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c0b8ba0 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.365+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c037cc0 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.365+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c0b9160 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.366+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c04f330 con 0x7fd57c162020 2024-03-06T07:23:43.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.366+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd56c04a070 con 0x7fd57c162020 2024-03-06T07:23:43.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.366+0000 7fd5737fe640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd55c06b500 0x7fd55c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.366+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd56c0abdf0 con 0x7fd57c162020 2024-03-06T07:23:43.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.366+0000 7fd573fff640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd55c06b500 0x7fd55c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.367+0000 7fd573fff640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd55c06b500 0x7fd55c06d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd564007fd0 tx=0x7fd564009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.371+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd56c033030 con 0x7fd57c162020 2024-03-06T07:23:43.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.544+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd57c0b9920 con 0x7fd57c162020 2024-03-06T07:23:43.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.545+0000 7fd5737fe640 1 -- 172.21.15.122:0/3199068199 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16613 (secure 0 0 0) 0x7fd56c07ac70 con 0x7fd57c162020 2024-03-06T07:23:43.547 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":45,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:41.897227+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:43.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd55c06b500 msgr2=0x7fd55c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd55c06b500 0x7fd55c06d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd564007fd0 tx=0x7fd564009040 comp rx=0 tx=0).stop 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 msgr2=0x7fd57c14d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 0x7fd57c14d0d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0098c0 tx=0x7fd56c004730 comp rx=0 tx=0).stop 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 shutdown_connections 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd55c06b500 0x7fd55c06d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd57c162a00 0x7fd57c0cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd57c162400 0x7fd57c14d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 --2- 172.21.15.122:0/3199068199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd57c162020 0x7fd57c14d0d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 >> 172.21.15.122:0/3199068199 conn(0x7fd57c0d3d10 msgr2=0x7fd57c0b4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 shutdown_connections 2024-03-06T07:23:43.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.547+0000 7fd58155e640 1 -- 172.21.15.122:0/3199068199 wait complete. 2024-03-06T07:23:43.561 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:43.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:43.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.658+0000 7f923c60e640 1 Processor -- start 2024-03-06T07:23:43.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.658+0000 7f923c60e640 1 -- start start 2024-03-06T07:23:43.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923c60e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f923415dfe0 0x7f92340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923c60e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f92340d29c0 0x7f92340b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923c60e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92340b3430 con 0x7f92340cf830 2024-03-06T07:23:43.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923c60e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340b31e0 con 0x7f92340d29c0 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923c60e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340b3d60 con 0x7f923415dfe0 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923abab640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f92340d29c0 0x7f92340b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f9239ba9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f92340cf830 0x7f92340d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51468/0 (socket says 172.21.15.122:51468) 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923abab640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f92340d29c0 0x7f92340b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35964/0 (socket says 172.21.15.122:35964) 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f9239ba9640 1 -- 172.21.15.122:0/2157950950 learned_addr learned my addr 172.21.15.122:0/2157950950 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.659+0000 7f923a3aa640 1 --2- 172.21.15.122:0/2157950950 >> v2:172.21.15.169:3300/0 conn(0x7f923415dfe0 0x7f92340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.660+0000 7f923abab640 1 -- 172.21.15.122:0/2157950950 >> v2:172.21.15.169:3300/0 conn(0x7f923415dfe0 msgr2=0x7f92340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.660+0000 7f923abab640 1 --2- 172.21.15.122:0/2157950950 >> v2:172.21.15.169:3300/0 conn(0x7f923415dfe0 0x7f92340cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.660+0000 7f923abab640 1 -- 172.21.15.122:0/2157950950 >> v1:172.21.15.122:6789/0 conn(0x7f92340cf830 legacy=0x7f92340d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.660+0000 7f92393a8640 1 -- 172.21.15.122:0/2157950950 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314866592 0 0) 0x7f92340b3430 con 0x7f92340cf830 2024-03-06T07:23:43.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.660+0000 7f923abab640 1 -- 172.21.15.122:0/2157950950 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92340b3710 con 0x7f92340d29c0 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.660+0000 7f923abab640 1 --2- 172.21.15.122:0/2157950950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340b41f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9234067990 tx=0x7f922c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc84790e48430a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f92393a8640 1 -- 172.21.15.122:0/2157950950 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c035020 con 0x7f92340d29c0 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f92393a8640 1 -- 172.21.15.122:0/2157950950 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c031830 con 0x7f92340d29c0 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f92393a8640 1 -- 172.21.15.122:0/2157950950 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c031c00 con 0x7f92340d29c0 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 -- 172.21.15.122:0/2157950950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 msgr2=0x7f92340b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 --2- 172.21.15.122:0/2157950950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340b41f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9234067990 tx=0x7f922c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 -- 172.21.15.122:0/2157950950 shutdown_connections 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 --2- 172.21.15.122:0/2157950950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340b41f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 --2- 172.21.15.122:0/2157950950 >> v2:172.21.15.169:3300/0 conn(0x7f923415dfe0 0x7f92340cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 -- 172.21.15.122:0/2157950950 >> 172.21.15.122:0/2157950950 conn(0x7f92340d3cd0 msgr2=0x7f92340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 -- 172.21.15.122:0/2157950950 shutdown_connections 2024-03-06T07:23:43.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.661+0000 7f923c60e640 1 -- 172.21.15.122:0/2157950950 wait complete. 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.662+0000 7f923c60e640 1 Processor -- start 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.662+0000 7f923c60e640 1 -- start start 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923c60e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92340cf830 0x7f9234160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923c60e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923c60e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f923415dfe0 0x7f92341e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923c60e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340d1df0 con 0x7f92340d29c0 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923c60e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340d1c70 con 0x7f923415dfe0 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923c60e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923415e3c0 con 0x7f92340cf830 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f9239ba9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f9239ba9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35976/0 (socket says 172.21.15.122:35976) 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f9239ba9640 1 -- 172.21.15.122:0/519747964 learned_addr learned my addr 172.21.15.122:0/519747964 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923abab640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f923415dfe0 0x7f92341e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f923a3aa640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92340cf830 0x7f9234160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f9239ba9640 1 -- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92340cf830 msgr2=0x7f9234160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f9239ba9640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92340cf830 0x7f9234160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.663+0000 7f9239ba9640 1 -- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f923415dfe0 msgr2=0x7f92341e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f9239ba9640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f923415dfe0 0x7f92341e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f9239ba9640 1 -- 172.21.15.122:0/519747964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234160830 con 0x7f92340d29c0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f9239ba9640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340ccc10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f922400c4b0 tx=0x7f922400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224018070 con 0x7f92340d29c0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9224007500 con 0x7f92340d29c0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9234161300 con 0x7f92340d29c0 2024-03-06T07:23:43.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224004a20 con 0x7f92340d29c0 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.664+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92341f76c0 con 0x7f92340d29c0 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.665+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923404f330 con 0x7f92340d29c0 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.665+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f922401b070 con 0x7f92340d29c0 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.666+0000 7f922affd640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f921806b4d0 0x7f921806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.666+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9224089480 con 0x7f92340d29c0 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.666+0000 7f923a3aa640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f921806b4d0 0x7f921806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.666+0000 7f923a3aa640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f921806b4d0 0x7f921806d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9230011340 tx=0x7f923000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.670+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9224057d70 con 0x7f92340d29c0 2024-03-06T07:23:43.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.838+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92340d0eb0 con 0x7f921806b4d0 2024-03-06T07:23:43.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.844+0000 7f922affd640 1 -- 172.21.15.122:0/519747964 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7f92340d0eb0 con 0x7f921806b4d0 2024-03-06T07:23:43.848 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:43.850 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:23:43.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.846+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f921806b4d0 msgr2=0x7f921806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.846+0000 7f923c60e640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f921806b4d0 0x7f921806d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9230011340 tx=0x7f923000b040 comp rx=0 tx=0).stop 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.846+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 msgr2=0x7f92340ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340ccc10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f922400c4b0 tx=0x7f922400c970 comp rx=0 tx=0).stop 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 shutdown_connections 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f921806b4d0 0x7f921806d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f923415dfe0 0x7f92341e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92340d29c0 0x7f92340ccc10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 --2- 172.21.15.122:0/519747964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92340cf830 0x7f9234160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 >> 172.21.15.122:0/519747964 conn(0x7f92340d3cd0 msgr2=0x7f92340d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 shutdown_connections 2024-03-06T07:23:43.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.847+0000 7f923c60e640 1 -- 172.21.15.122:0/519747964 wait complete. 2024-03-06T07:23:43.859 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-03-06T07:23:43.352031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254492,"kb_used_data":5036,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720228,"statfs":{"total":773094113280,"available":772833513472,"internally_reserved":0,"allocated":5156864,"data_stored":2765721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471774"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896350+0000","last_change":"2024-03-06T07:23:39.896350+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896350+0000","last_undegraded":"2024-03-06T07:23:39.896350+0000","last_fullsized":"2024-03-06T07:23:39.896350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902983+0000","last_change":"2024-03-06T07:23:39.902983+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902983+0000","last_undegraded":"2024-03-06T07:23:39.902983+0000","last_fullsized":"2024-03-06T07:23:39.902983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895197+0000","last_change":"2024-03-06T07:23:39.895197+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895197+0000","last_undegraded":"2024-03-06T07:23:39.895197+0000","last_fullsized":"2024-03-06T07:23:39.895197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898515+0000","last_change":"2024-03-06T07:23:39.898515+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898515+0000","last_undegraded":"2024-03-06T07:23:39.898515+0000","last_fullsized":"2024-03-06T07:23:39.898515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897220+0000","last_change":"2024-03-06T07:23:39.897220+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897220+0000","last_undegraded":"2024-03-06T07:23:39.897220+0000","last_fullsized":"2024-03-06T07:23:39.897220+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899153+0000","last_change":"2024-03-06T07:23:39.899153+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899153+0000","last_undegraded":"2024-03-06T07:23:39.899153+0000","last_fullsized":"2024-03-06T07:23:39.899153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896635+0000","last_change":"2024-03-06T07:23:39.896635+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896635+0000","last_undegraded":"2024-03-06T07:23:39.896635+0000","last_fullsized":"2024-03-06T07:23:39.896635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895129+0000","last_change":"2024-03-06T07:23:39.895129+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895129+0000","last_undegraded":"2024-03-06T07:23:39.895129+0000","last_fullsized":"2024-03-06T07:23:39.895129+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900467+0000","last_change":"2024-03-06T07:23:39.900467+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900467+0000","last_undegraded":"2024-03-06T07:23:39.900467+0000","last_fullsized":"2024-03-06T07:23:39.900467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895165+0000","last_change":"2024-03-06T07:23:39.895165+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895165+0000","last_undegraded":"2024-03-06T07:23:39.895165+0000","last_fullsized":"2024-03-06T07:23:39.895165+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894678+0000","last_change":"2024-03-06T07:23:39.894678+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894678+0000","last_undegraded":"2024-03-06T07:23:39.894678+0000","last_fullsized":"2024-03-06T07:23:39.894678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902759+0000","last_change":"2024-03-06T07:23:39.902759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902759+0000","last_undegraded":"2024-03-06T07:23:39.902759+0000","last_fullsized":"2024-03-06T07:23:39.902759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900672+0000","last_change":"2024-03-06T07:23:39.900672+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900672+0000","last_undegraded":"2024-03-06T07:23:39.900672+0000","last_fullsized":"2024-03-06T07:23:39.900672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899316+0000","last_change":"2024-03-06T07:23:39.899316+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899316+0000","last_undegraded":"2024-03-06T07:23:39.899316+0000","last_fullsized":"2024-03-06T07:23:39.899316+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897463+0000","last_change":"2024-03-06T07:23:39.897463+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897463+0000","last_undegraded":"2024-03-06T07:23:39.897463+0000","last_fullsized":"2024-03-06T07:23:39.897463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893246+0000","last_change":"2024-03-06T07:23:39.893246+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893246+0000","last_undegraded":"2024-03-06T07:23:39.893246+0000","last_fullsized":"2024-03-06T07:23:39.893246+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894229+0000","last_change":"2024-03-06T07:23:39.894229+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894229+0000","last_undegraded":"2024-03-06T07:23:39.894229+0000","last_fullsized":"2024-03-06T07:23:39.894229+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896126+0000","last_change":"2024-03-06T07:23:39.896126+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896126+0000","last_undegraded":"2024-03-06T07:23:39.896126+0000","last_fullsized":"2024-03-06T07:23:39.896126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894221+0000","last_change":"2024-03-06T07:23:39.894221+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894221+0000","last_undegraded":"2024-03-06T07:23:39.894221+0000","last_fullsized":"2024-03-06T07:23:39.894221+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900319+0000","last_change":"2024-03-06T07:23:39.900319+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900319+0000","last_undegraded":"2024-03-06T07:23:39.900319+0000","last_fullsized":"2024-03-06T07:23:39.900319+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901518+0000","last_change":"2024-03-06T07:23:39.901518+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901518+0000","last_undegraded":"2024-03-06T07:23:39.901518+0000","last_fullsized":"2024-03-06T07:23:39.901518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896014+0000","last_change":"2024-03-06T07:23:39.896014+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896014+0000","last_undegraded":"2024-03-06T07:23:39.896014+0000","last_fullsized":"2024-03-06T07:23:39.896014+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895095+0000","last_change":"2024-03-06T07:23:39.895095+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895095+0000","last_undegraded":"2024-03-06T07:23:39.895095+0000","last_fullsized":"2024-03-06T07:23:39.895095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892690+0000","last_change":"2024-03-06T07:23:39.892690+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892690+0000","last_undegraded":"2024-03-06T07:23:39.892690+0000","last_fullsized":"2024-03-06T07:23:39.892690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893762+0000","last_change":"2024-03-06T07:23:39.893762+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893762+0000","last_undegraded":"2024-03-06T07:23:39.893762+0000","last_fullsized":"2024-03-06T07:23:39.893762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897975+0000","last_change":"2024-03-06T07:23:39.897975+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897975+0000","last_undegraded":"2024-03-06T07:23:39.897975+0000","last_fullsized":"2024-03-06T07:23:39.897975+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893063+0000","last_change":"2024-03-06T07:23:39.893063+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893063+0000","last_undegraded":"2024-03-06T07:23:39.893063+0000","last_fullsized":"2024-03-06T07:23:39.893063+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898452+0000","last_change":"2024-03-06T07:23:39.898452+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898452+0000","last_undegraded":"2024-03-06T07:23:39.898452+0000","last_fullsized":"2024-03-06T07:23:39.898452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893292+0000","last_change":"2024-03-06T07:23:39.893292+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893292+0000","last_undegraded":"2024-03-06T07:23:39.893292+0000","last_fullsized":"2024-03-06T07:23:39.893292+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893704+0000","last_change":"2024-03-06T07:23:39.893704+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893704+0000","last_undegraded":"2024-03-06T07:23:39.893704+0000","last_fullsized":"2024-03-06T07:23:39.893704+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896976+0000","last_change":"2024-03-06T07:23:39.896976+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896976+0000","last_undegraded":"2024-03-06T07:23:39.896976+0000","last_fullsized":"2024-03-06T07:23:39.896976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894325+0000","last_change":"2024-03-06T07:23:39.894325+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894325+0000","last_undegraded":"2024-03-06T07:23:39.894325+0000","last_fullsized":"2024-03-06T07:23:39.894325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898773+0000","last_change":"2024-03-06T07:23:39.898773+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898773+0000","last_undegraded":"2024-03-06T07:23:39.898773+0000","last_fullsized":"2024-03-06T07:23:39.898773+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.902762+0000","last_change":"2024-03-06T07:23:39.900097+0000","last_active":"2024-03-06T07:23:39.902762+0000","last_peered":"2024-03-06T07:23:39.902762+0000","last_clean":"2024-03-06T07:23:39.902762+0000","last_became_active":"2024-03-06T07:23:39.899808+0000","last_became_peered":"2024-03-06T07:23:39.899808+0000","last_unstale":"2024-03-06T07:23:39.902762+0000","last_undegraded":"2024-03-06T07:23:39.902762+0000","last_fullsized":"2024-03-06T07:23:39.902762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893338+0000","last_change":"2024-03-06T07:23:39.893338+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893338+0000","last_undegraded":"2024-03-06T07:23:39.893338+0000","last_fullsized":"2024-03-06T07:23:39.893338+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907516+0000","last_change":"2024-03-06T07:23:39.899915+0000","last_active":"2024-03-06T07:23:39.907516+0000","last_peered":"2024-03-06T07:23:39.907516+0000","last_clean":"2024-03-06T07:23:39.907516+0000","last_became_active":"2024-03-06T07:23:39.899555+0000","last_became_peered":"2024-03-06T07:23:39.899555+0000","last_unstale":"2024-03-06T07:23:39.907516+0000","last_undegraded":"2024-03-06T07:23:39.907516+0000","last_fullsized":"2024-03-06T07:23:39.907516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:45.519711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898062+0000","last_change":"2024-03-06T07:23:39.898062+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898062+0000","last_undegraded":"2024-03-06T07:23:39.898062+0000","last_fullsized":"2024-03-06T07:23:39.898062+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915812+0000","last_change":"2024-03-06T07:23:39.911329+0000","last_active":"2024-03-06T07:23:39.915812+0000","last_peered":"2024-03-06T07:23:39.915812+0000","last_clean":"2024-03-06T07:23:39.915812+0000","last_became_active":"2024-03-06T07:23:39.911011+0000","last_became_peered":"2024-03-06T07:23:39.911011+0000","last_unstale":"2024-03-06T07:23:39.915812+0000","last_undegraded":"2024-03-06T07:23:39.915812+0000","last_fullsized":"2024-03-06T07:23:39.915812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:45.430603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.904164+0000","last_change":"2024-03-06T07:23:39.904164+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.904164+0000","last_undegraded":"2024-03-06T07:23:39.904164+0000","last_fullsized":"2024-03-06T07:23:39.904164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910560+0000","last_change":"2024-03-06T07:23:39.908704+0000","last_active":"2024-03-06T07:23:39.910560+0000","last_peered":"2024-03-06T07:23:39.910560+0000","last_clean":"2024-03-06T07:23:39.910560+0000","last_became_active":"2024-03-06T07:23:39.908289+0000","last_became_peered":"2024-03-06T07:23:39.908289+0000","last_unstale":"2024-03-06T07:23:39.910560+0000","last_undegraded":"2024-03-06T07:23:39.910560+0000","last_fullsized":"2024-03-06T07:23:39.910560+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893272+0000","last_change":"2024-03-06T07:23:39.893272+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893272+0000","last_undegraded":"2024-03-06T07:23:39.893272+0000","last_fullsized":"2024-03-06T07:23:39.893272+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.909418+0000","last_change":"2024-03-06T07:23:39.908607+0000","last_active":"2024-03-06T07:23:39.909418+0000","last_peered":"2024-03-06T07:23:39.909418+0000","last_clean":"2024-03-06T07:23:39.909418+0000","last_became_active":"2024-03-06T07:23:39.908334+0000","last_became_peered":"2024-03-06T07:23:39.908334+0000","last_unstale":"2024-03-06T07:23:39.909418+0000","last_undegraded":"2024-03-06T07:23:39.909418+0000","last_fullsized":"2024-03-06T07:23:39.909418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:06.382376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899907+0000","last_change":"2024-03-06T07:23:39.899907+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899907+0000","last_undegraded":"2024-03-06T07:23:39.899907+0000","last_fullsized":"2024-03-06T07:23:39.899907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913163+0000","last_change":"2024-03-06T07:23:39.908741+0000","last_active":"2024-03-06T07:23:39.913163+0000","last_peered":"2024-03-06T07:23:39.913163+0000","last_clean":"2024-03-06T07:23:39.913163+0000","last_became_active":"2024-03-06T07:23:39.908470+0000","last_became_peered":"2024-03-06T07:23:39.908470+0000","last_unstale":"2024-03-06T07:23:39.913163+0000","last_undegraded":"2024-03-06T07:23:39.913163+0000","last_fullsized":"2024-03-06T07:23:39.913163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:07.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894696+0000","last_change":"2024-03-06T07:23:39.894696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894696+0000","last_undegraded":"2024-03-06T07:23:39.894696+0000","last_fullsized":"2024-03-06T07:23:39.894696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913247+0000","last_change":"2024-03-06T07:23:39.909770+0000","last_active":"2024-03-06T07:23:39.913247+0000","last_peered":"2024-03-06T07:23:39.913247+0000","last_clean":"2024-03-06T07:23:39.913247+0000","last_became_active":"2024-03-06T07:23:39.909494+0000","last_became_peered":"2024-03-06T07:23:39.909494+0000","last_unstale":"2024-03-06T07:23:39.913247+0000","last_undegraded":"2024-03-06T07:23:39.913247+0000","last_fullsized":"2024-03-06T07:23:39.913247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895537+0000","last_change":"2024-03-06T07:23:39.895537+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895537+0000","last_undegraded":"2024-03-06T07:23:39.895537+0000","last_fullsized":"2024-03-06T07:23:39.895537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910678+0000","last_change":"2024-03-06T07:23:39.906386+0000","last_active":"2024-03-06T07:23:39.910678+0000","last_peered":"2024-03-06T07:23:39.910678+0000","last_clean":"2024-03-06T07:23:39.910678+0000","last_became_active":"2024-03-06T07:23:39.905390+0000","last_became_peered":"2024-03-06T07:23:39.905390+0000","last_unstale":"2024-03-06T07:23:39.910678+0000","last_undegraded":"2024-03-06T07:23:39.910678+0000","last_fullsized":"2024-03-06T07:23:39.910678+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895696+0000","last_change":"2024-03-06T07:23:39.895696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895696+0000","last_undegraded":"2024-03-06T07:23:39.895696+0000","last_fullsized":"2024-03-06T07:23:39.895696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907953+0000","last_change":"2024-03-06T07:23:39.901277+0000","last_active":"2024-03-06T07:23:39.907953+0000","last_peered":"2024-03-06T07:23:39.907953+0000","last_clean":"2024-03-06T07:23:39.907953+0000","last_became_active":"2024-03-06T07:23:39.900996+0000","last_became_peered":"2024-03-06T07:23:39.900996+0000","last_unstale":"2024-03-06T07:23:39.907953+0000","last_undegraded":"2024-03-06T07:23:39.907953+0000","last_fullsized":"2024-03-06T07:23:39.907953+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:05.084281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895950+0000","last_change":"2024-03-06T07:23:39.895950+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895950+0000","last_undegraded":"2024-03-06T07:23:39.895950+0000","last_fullsized":"2024-03-06T07:23:39.895950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910643+0000","last_change":"2024-03-06T07:23:39.904382+0000","last_active":"2024-03-06T07:23:39.910643+0000","last_peered":"2024-03-06T07:23:39.910643+0000","last_clean":"2024-03-06T07:23:39.910643+0000","last_became_active":"2024-03-06T07:23:39.904076+0000","last_became_peered":"2024-03-06T07:23:39.904076+0000","last_unstale":"2024-03-06T07:23:39.910643+0000","last_undegraded":"2024-03-06T07:23:39.910643+0000","last_fullsized":"2024-03-06T07:23:39.910643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:51.296844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898603+0000","last_change":"2024-03-06T07:23:39.898603+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898603+0000","last_undegraded":"2024-03-06T07:23:39.898603+0000","last_fullsized":"2024-03-06T07:23:39.898603+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913122+0000","last_change":"2024-03-06T07:23:39.909133+0000","last_active":"2024-03-06T07:23:39.913122+0000","last_peered":"2024-03-06T07:23:39.913122+0000","last_clean":"2024-03-06T07:23:39.913122+0000","last_became_active":"2024-03-06T07:23:39.908563+0000","last_became_peered":"2024-03-06T07:23:39.908563+0000","last_unstale":"2024-03-06T07:23:39.913122+0000","last_undegraded":"2024-03-06T07:23:39.913122+0000","last_fullsized":"2024-03-06T07:23:39.913122+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896306+0000","last_change":"2024-03-06T07:23:39.896306+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896306+0000","last_undegraded":"2024-03-06T07:23:39.896306+0000","last_fullsized":"2024-03-06T07:23:39.896306+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.900855+0000","last_change":"2024-03-06T07:23:39.896528+0000","last_active":"2024-03-06T07:23:39.900855+0000","last_peered":"2024-03-06T07:23:39.900855+0000","last_clean":"2024-03-06T07:23:39.900855+0000","last_became_active":"2024-03-06T07:23:39.896142+0000","last_became_peered":"2024-03-06T07:23:39.896142+0000","last_unstale":"2024-03-06T07:23:39.900855+0000","last_undegraded":"2024-03-06T07:23:39.900855+0000","last_fullsized":"2024-03-06T07:23:39.900855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:17.233793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892802+0000","last_change":"2024-03-06T07:23:39.892802+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892802+0000","last_undegraded":"2024-03-06T07:23:39.892802+0000","last_fullsized":"2024-03-06T07:23:39.892802+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906489+0000","last_change":"2024-03-06T07:23:39.905467+0000","last_active":"2024-03-06T07:23:39.906489+0000","last_peered":"2024-03-06T07:23:39.906489+0000","last_clean":"2024-03-06T07:23:39.906489+0000","last_became_active":"2024-03-06T07:23:39.905185+0000","last_became_peered":"2024-03-06T07:23:39.905185+0000","last_unstale":"2024-03-06T07:23:39.906489+0000","last_undegraded":"2024-03-06T07:23:39.906489+0000","last_fullsized":"2024-03-06T07:23:39.906489+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:48.343179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896114+0000","last_change":"2024-03-06T07:23:39.896114+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896114+0000","last_undegraded":"2024-03-06T07:23:39.896114+0000","last_fullsized":"2024-03-06T07:23:39.896114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908901+0000","last_change":"2024-03-06T07:23:39.905513+0000","last_active":"2024-03-06T07:23:39.908901+0000","last_peered":"2024-03-06T07:23:39.908901+0000","last_clean":"2024-03-06T07:23:39.908901+0000","last_became_active":"2024-03-06T07:23:39.905121+0000","last_became_peered":"2024-03-06T07:23:39.905121+0000","last_unstale":"2024-03-06T07:23:39.908901+0000","last_undegraded":"2024-03-06T07:23:39.908901+0000","last_fullsized":"2024-03-06T07:23:39.908901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:57.881886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893960+0000","last_change":"2024-03-06T07:23:39.893960+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893960+0000","last_undegraded":"2024-03-06T07:23:39.893960+0000","last_fullsized":"2024-03-06T07:23:39.893960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906161+0000","last_change":"2024-03-06T07:23:39.902543+0000","last_active":"2024-03-06T07:23:39.906161+0000","last_peered":"2024-03-06T07:23:39.906161+0000","last_clean":"2024-03-06T07:23:39.906161+0000","last_became_active":"2024-03-06T07:23:39.902271+0000","last_became_peered":"2024-03-06T07:23:39.902271+0000","last_unstale":"2024-03-06T07:23:39.906161+0000","last_undegraded":"2024-03-06T07:23:39.906161+0000","last_fullsized":"2024-03-06T07:23:39.906161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:42.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896222+0000","last_change":"2024-03-06T07:23:39.896222+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896222+0000","last_undegraded":"2024-03-06T07:23:39.896222+0000","last_fullsized":"2024-03-06T07:23:39.896222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904631+0000","last_change":"2024-03-06T07:23:39.903255+0000","last_active":"2024-03-06T07:23:39.904631+0000","last_peered":"2024-03-06T07:23:39.904631+0000","last_clean":"2024-03-06T07:23:39.904631+0000","last_became_active":"2024-03-06T07:23:39.902984+0000","last_became_peered":"2024-03-06T07:23:39.902984+0000","last_unstale":"2024-03-06T07:23:39.904631+0000","last_undegraded":"2024-03-06T07:23:39.904631+0000","last_fullsized":"2024-03-06T07:23:39.904631+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:06.103875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894330+0000","last_change":"2024-03-06T07:23:39.894330+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894330+0000","last_undegraded":"2024-03-06T07:23:39.894330+0000","last_fullsized":"2024-03-06T07:23:39.894330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913144+0000","last_change":"2024-03-06T07:23:39.906113+0000","last_active":"2024-03-06T07:23:39.913144+0000","last_peered":"2024-03-06T07:23:39.913144+0000","last_clean":"2024-03-06T07:23:39.913144+0000","last_became_active":"2024-03-06T07:23:39.905836+0000","last_became_peered":"2024-03-06T07:23:39.905836+0000","last_unstale":"2024-03-06T07:23:39.913144+0000","last_undegraded":"2024-03-06T07:23:39.913144+0000","last_fullsized":"2024-03-06T07:23:39.913144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:50.974905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896241+0000","last_change":"2024-03-06T07:23:39.896241+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896241+0000","last_undegraded":"2024-03-06T07:23:39.896241+0000","last_fullsized":"2024-03-06T07:23:39.896241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908710+0000","last_change":"2024-03-06T07:23:39.904607+0000","last_active":"2024-03-06T07:23:39.908710+0000","last_peered":"2024-03-06T07:23:39.908710+0000","last_clean":"2024-03-06T07:23:39.908710+0000","last_became_active":"2024-03-06T07:23:39.904289+0000","last_became_peered":"2024-03-06T07:23:39.904289+0000","last_unstale":"2024-03-06T07:23:39.908710+0000","last_undegraded":"2024-03-06T07:23:39.908710+0000","last_fullsized":"2024-03-06T07:23:39.908710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:47.230884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893759+0000","last_change":"2024-03-06T07:23:39.893759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893759+0000","last_undegraded":"2024-03-06T07:23:39.893759+0000","last_fullsized":"2024-03-06T07:23:39.893759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913774+0000","last_change":"2024-03-06T07:23:39.911859+0000","last_active":"2024-03-06T07:23:39.913774+0000","last_peered":"2024-03-06T07:23:39.913774+0000","last_clean":"2024-03-06T07:23:39.913774+0000","last_became_active":"2024-03-06T07:23:39.911529+0000","last_became_peered":"2024-03-06T07:23:39.911529+0000","last_unstale":"2024-03-06T07:23:39.913774+0000","last_undegraded":"2024-03-06T07:23:39.913774+0000","last_fullsized":"2024-03-06T07:23:39.913774+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894652+0000","last_change":"2024-03-06T07:23:39.894652+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894652+0000","last_undegraded":"2024-03-06T07:23:39.894652+0000","last_fullsized":"2024-03-06T07:23:39.894652+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.911862+0000","last_change":"2024-03-06T07:23:39.911223+0000","last_active":"2024-03-06T07:23:39.911862+0000","last_peered":"2024-03-06T07:23:39.911862+0000","last_clean":"2024-03-06T07:23:39.911862+0000","last_became_active":"2024-03-06T07:23:39.910935+0000","last_became_peered":"2024-03-06T07:23:39.910935+0000","last_unstale":"2024-03-06T07:23:39.911862+0000","last_undegraded":"2024-03-06T07:23:39.911862+0000","last_fullsized":"2024-03-06T07:23:39.911862+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:46.994267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897546+0000","last_change":"2024-03-06T07:23:39.897546+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897546+0000","last_undegraded":"2024-03-06T07:23:39.897546+0000","last_fullsized":"2024-03-06T07:23:39.897546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.905864+0000","last_change":"2024-03-06T07:23:39.903773+0000","last_active":"2024-03-06T07:23:39.905864+0000","last_peered":"2024-03-06T07:23:39.905864+0000","last_clean":"2024-03-06T07:23:39.905864+0000","last_became_active":"2024-03-06T07:23:39.903494+0000","last_became_peered":"2024-03-06T07:23:39.903494+0000","last_unstale":"2024-03-06T07:23:39.905864+0000","last_undegraded":"2024-03-06T07:23:39.905864+0000","last_fullsized":"2024-03-06T07:23:39.905864+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:14.991973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898830+0000","last_change":"2024-03-06T07:23:39.898830+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898830+0000","last_undegraded":"2024-03-06T07:23:39.898830+0000","last_fullsized":"2024-03-06T07:23:39.898830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913738+0000","last_change":"2024-03-06T07:23:39.911261+0000","last_active":"2024-03-06T07:23:39.913738+0000","last_peered":"2024-03-06T07:23:39.913738+0000","last_clean":"2024-03-06T07:23:39.913738+0000","last_became_active":"2024-03-06T07:23:39.910917+0000","last_became_peered":"2024-03-06T07:23:39.910917+0000","last_unstale":"2024-03-06T07:23:39.913738+0000","last_undegraded":"2024-03-06T07:23:39.913738+0000","last_fullsized":"2024-03-06T07:23:39.913738+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893681+0000","last_change":"2024-03-06T07:23:39.893681+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893681+0000","last_undegraded":"2024-03-06T07:23:39.893681+0000","last_fullsized":"2024-03-06T07:23:39.893681+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913098+0000","last_change":"2024-03-06T07:23:39.909408+0000","last_active":"2024-03-06T07:23:39.913098+0000","last_peered":"2024-03-06T07:23:39.913098+0000","last_clean":"2024-03-06T07:23:39.913098+0000","last_became_active":"2024-03-06T07:23:39.909128+0000","last_became_peered":"2024-03-06T07:23:39.909128+0000","last_unstale":"2024-03-06T07:23:39.913098+0000","last_undegraded":"2024-03-06T07:23:39.913098+0000","last_fullsized":"2024-03-06T07:23:39.913098+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.087590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895570+0000","last_change":"2024-03-06T07:23:39.895570+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895570+0000","last_undegraded":"2024-03-06T07:23:39.895570+0000","last_fullsized":"2024-03-06T07:23:39.895570+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908963+0000","last_change":"2024-03-06T07:23:39.905798+0000","last_active":"2024-03-06T07:23:39.908963+0000","last_peered":"2024-03-06T07:23:39.908963+0000","last_clean":"2024-03-06T07:23:39.908963+0000","last_became_active":"2024-03-06T07:23:39.905513+0000","last_became_peered":"2024-03-06T07:23:39.905513+0000","last_unstale":"2024-03-06T07:23:39.908963+0000","last_undegraded":"2024-03-06T07:23:39.908963+0000","last_fullsized":"2024-03-06T07:23:39.908963+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892814+0000","last_change":"2024-03-06T07:23:39.892814+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892814+0000","last_undegraded":"2024-03-06T07:23:39.892814+0000","last_fullsized":"2024-03-06T07:23:39.892814+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906475+0000","last_change":"2024-03-06T07:23:39.905335+0000","last_active":"2024-03-06T07:23:39.906475+0000","last_peered":"2024-03-06T07:23:39.906475+0000","last_clean":"2024-03-06T07:23:39.906475+0000","last_became_active":"2024-03-06T07:23:39.903169+0000","last_became_peered":"2024-03-06T07:23:39.903169+0000","last_unstale":"2024-03-06T07:23:39.906475+0000","last_undegraded":"2024-03-06T07:23:39.906475+0000","last_fullsized":"2024-03-06T07:23:39.906475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:44.722551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894171+0000","last_change":"2024-03-06T07:23:39.894171+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894171+0000","last_undegraded":"2024-03-06T07:23:39.894171+0000","last_fullsized":"2024-03-06T07:23:39.894171+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910527+0000","last_change":"2024-03-06T07:23:39.908894+0000","last_active":"2024-03-06T07:23:39.910527+0000","last_peered":"2024-03-06T07:23:39.910527+0000","last_clean":"2024-03-06T07:23:39.910527+0000","last_became_active":"2024-03-06T07:23:39.908206+0000","last_became_peered":"2024-03-06T07:23:39.908206+0000","last_unstale":"2024-03-06T07:23:39.910527+0000","last_undegraded":"2024-03-06T07:23:39.910527+0000","last_fullsized":"2024-03-06T07:23:39.910527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895809+0000","last_change":"2024-03-06T07:23:39.895809+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895809+0000","last_undegraded":"2024-03-06T07:23:39.895809+0000","last_fullsized":"2024-03-06T07:23:39.895809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915464+0000","last_change":"2024-03-06T07:23:39.909279+0000","last_active":"2024-03-06T07:23:39.915464+0000","last_peered":"2024-03-06T07:23:39.915464+0000","last_clean":"2024-03-06T07:23:39.915464+0000","last_became_active":"2024-03-06T07:23:39.908955+0000","last_became_peered":"2024-03-06T07:23:39.908955+0000","last_unstale":"2024-03-06T07:23:39.915464+0000","last_undegraded":"2024-03-06T07:23:39.915464+0000","last_fullsized":"2024-03-06T07:23:39.915464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:42.900316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897107+0000","last_change":"2024-03-06T07:23:39.897107+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897107+0000","last_undegraded":"2024-03-06T07:23:39.897107+0000","last_fullsized":"2024-03-06T07:23:39.897107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910530+0000","last_change":"2024-03-06T07:23:39.908492+0000","last_active":"2024-03-06T07:23:39.910530+0000","last_peered":"2024-03-06T07:23:39.910530+0000","last_clean":"2024-03-06T07:23:39.910530+0000","last_became_active":"2024-03-06T07:23:39.908214+0000","last_became_peered":"2024-03-06T07:23:39.908214+0000","last_unstale":"2024-03-06T07:23:39.910530+0000","last_undegraded":"2024-03-06T07:23:39.910530+0000","last_fullsized":"2024-03-06T07:23:39.910530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901075+0000","last_change":"2024-03-06T07:23:39.901075+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901075+0000","last_undegraded":"2024-03-06T07:23:39.901075+0000","last_fullsized":"2024-03-06T07:23:39.901075+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907658+0000","last_change":"2024-03-06T07:23:39.901912+0000","last_active":"2024-03-06T07:23:39.907658+0000","last_peered":"2024-03-06T07:23:39.907658+0000","last_clean":"2024-03-06T07:23:39.907658+0000","last_became_active":"2024-03-06T07:23:39.901713+0000","last_became_peered":"2024-03-06T07:23:39.901713+0000","last_unstale":"2024-03-06T07:23:39.907658+0000","last_undegraded":"2024-03-06T07:23:39.907658+0000","last_fullsized":"2024-03-06T07:23:39.907658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:31.271276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897613+0000","last_change":"2024-03-06T07:23:39.897613+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897613+0000","last_undegraded":"2024-03-06T07:23:39.897613+0000","last_fullsized":"2024-03-06T07:23:39.897613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915433+0000","last_change":"2024-03-06T07:23:39.908817+0000","last_active":"2024-03-06T07:23:39.915433+0000","last_peered":"2024-03-06T07:23:39.915433+0000","last_clean":"2024-03-06T07:23:39.915433+0000","last_became_active":"2024-03-06T07:23:39.908523+0000","last_became_peered":"2024-03-06T07:23:39.908523+0000","last_unstale":"2024-03-06T07:23:39.915433+0000","last_undegraded":"2024-03-06T07:23:39.915433+0000","last_fullsized":"2024-03-06T07:23:39.915433+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:23.516812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892303+0000","last_change":"2024-03-06T07:23:39.892303+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892303+0000","last_undegraded":"2024-03-06T07:23:39.892303+0000","last_fullsized":"2024-03-06T07:23:39.892303+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904611+0000","last_change":"2024-03-06T07:23:39.902253+0000","last_active":"2024-03-06T07:23:39.904611+0000","last_peered":"2024-03-06T07:23:39.904611+0000","last_clean":"2024-03-06T07:23:39.904611+0000","last_became_active":"2024-03-06T07:23:39.901938+0000","last_became_peered":"2024-03-06T07:23:39.901938+0000","last_unstale":"2024-03-06T07:23:39.904611+0000","last_undegraded":"2024-03-06T07:23:39.904611+0000","last_fullsized":"2024-03-06T07:23:39.904611+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:49.309308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894516+0000","last_change":"2024-03-06T07:23:39.894516+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894516+0000","last_undegraded":"2024-03-06T07:23:39.894516+0000","last_fullsized":"2024-03-06T07:23:39.894516+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913255+0000","last_change":"2024-03-06T07:23:39.908459+0000","last_active":"2024-03-06T07:23:39.913255+0000","last_peered":"2024-03-06T07:23:39.913255+0000","last_clean":"2024-03-06T07:23:39.913255+0000","last_became_active":"2024-03-06T07:23:39.908193+0000","last_became_peered":"2024-03-06T07:23:39.908193+0000","last_unstale":"2024-03-06T07:23:39.913255+0000","last_undegraded":"2024-03-06T07:23:39.913255+0000","last_fullsized":"2024-03-06T07:23:39.913255+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.898698+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:23:39.898698+0000","last_peered":"2024-03-06T07:23:39.898698+0000","last_clean":"2024-03-06T07:23:39.898698+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:23:39.898698+0000","last_undegraded":"2024-03-06T07:23:39.898698+0000","last_fullsized":"2024-03-06T07:23:39.898698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:43.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:44.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.998+0000 7f842146f640 1 Processor -- start 2024-03-06T07:23:44.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.998+0000 7f842146f640 1 -- start start 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f842146f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f841c06b600 0x7f841c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f842146f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841c15ea30 con 0x7f841c06b220 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f842146f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841c15ed10 con 0x7f841c0ce920 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f842146f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841c161b70 con 0x7f841c06b600 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f841affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f841c06b220 0x7f841c071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51476/0 (socket says 172.21.15.122:51476) 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f841affd640 1 -- 172.21.15.122:0/3064000676 learned_addr learned my addr 172.21.15.122:0/3064000676 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:44.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f841b7fe640 1 --2- 172.21.15.122:0/3064000676 >> v2:172.21.15.169:3300/0 conn(0x7f841c06b600 0x7f841c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877064536 0 0) 0x7f841c15ea30 con 0x7f841c06b220 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8404003520 con 0x7f841c06b220 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993932936 0 0) 0x7f841c15ed10 con 0x7f841c0ce920 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f841c15ea30 con 0x7f841c0ce920 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f841b7fe640 1 -- 172.21.15.122:0/3064000676 >> v1:172.21.15.169:6789/0 conn(0x7f841c0ce920 legacy=0x7f841c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f841b7fe640 1 -- 172.21.15.122:0/3064000676 >> v1:172.21.15.122:6789/0 conn(0x7f841c06b220 legacy=0x7f841c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:43.999+0000 7f841b7fe640 1 -- 172.21.15.122:0/3064000676 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c15eff0 con 0x7f841c06b600 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.000+0000 7f841b7fe640 1 --2- 172.21.15.122:0/3064000676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b600 0x7f841c15e450 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f841c161cf0 tx=0x7f841402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de37c37cc545990 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.000+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414034070 con 0x7f841c06b600 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.000+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8414030610 con 0x7f841c06b600 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.000+0000 7f8419ffb640 1 -- 172.21.15.122:0/3064000676 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84140309e0 con 0x7f841c06b600 2024-03-06T07:23:44.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 -- 172.21.15.122:0/3064000676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b600 msgr2=0x7f841c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 --2- 172.21.15.122:0/3064000676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b600 0x7f841c15e450 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f841c161cf0 tx=0x7f841402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:44.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 -- 172.21.15.122:0/3064000676 shutdown_connections 2024-03-06T07:23:44.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 --2- 172.21.15.122:0/3064000676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b600 0x7f841c15e450 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 -- 172.21.15.122:0/3064000676 >> 172.21.15.122:0/3064000676 conn(0x7f841c06c0c0 msgr2=0x7f841c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 -- 172.21.15.122:0/3064000676 shutdown_connections 2024-03-06T07:23:44.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.001+0000 7f842146f640 1 -- 172.21.15.122:0/3064000676 wait complete. 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.002+0000 7f842146f640 1 Processor -- start 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.002+0000 7f842146f640 1 -- start start 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f842146f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b220 0x7f841c14ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f842146f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 0x7f841c1f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f842146f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f841c0ce920 0x7f841c1f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f842146f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c072090 con 0x7f841c06b600 2024-03-06T07:23:44.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f842146f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0ce200 con 0x7f841c06b220 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f842146f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c1f4540 con 0x7f841c0ce920 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 0x7f841c1f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 0x7f841c1f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35986/0 (socket says 172.21.15.122:35986) 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 -- 172.21.15.122:0/3662624332 learned_addr learned my addr 172.21.15.122:0/3662624332 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841b7fe640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f841c0ce920 0x7f841c1f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841affd640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b220 0x7f841c14ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 -- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f841c0ce920 msgr2=0x7f841c1f83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f841c0ce920 0x7f841c1f83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 -- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b220 msgr2=0x7f841c14ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b220 0x7f841c14ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 -- 172.21.15.122:0/3662624332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c14b250 con 0x7f841c06b600 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.003+0000 7f841a7fc640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 0x7f841c1f73a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f841000c500 tx=0x7f841000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.004+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84100052d0 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.004+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410005490 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.004+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410012870 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.004+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c14b530 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.004+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c1f8900 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.005+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c162910 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.005+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8410012a10 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.005+0000 7f84037fe640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f840806b500 0x7f840806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.006+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8410089d60 con 0x7f841c06b600 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.006+0000 7f841affd640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f840806b500 0x7f840806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.006+0000 7f841affd640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f840806b500 0x7f840806d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f840c001560 tx=0x7f840c0019f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.010+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84100585a0 con 0x7f841c06b600 2024-03-06T07:23:44.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.178+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f841c1f46f0 con 0x7f840806b500 2024-03-06T07:23:44.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.184+0000 7f84037fe640 1 -- 172.21.15.122:0/3662624332 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7f840c001ee0 con 0x7f840806b500 2024-03-06T07:23:44.189 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:44.190 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:23:44.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f840806b500 msgr2=0x7f840806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f840806b500 0x7f840806d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f840c001560 tx=0x7f840c0019f0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 msgr2=0x7f841c1f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 0x7f841c1f73a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f841000c500 tx=0x7f841000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 shutdown_connections 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f840806b500 0x7f840806d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f841c0ce920 0x7f841c1f83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c06b600 0x7f841c1f73a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 --2- 172.21.15.122:0/3662624332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f841c06b220 0x7f841c14ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.186+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 >> 172.21.15.122:0/3662624332 conn(0x7f841c06c0c0 msgr2=0x7f841c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.187+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 shutdown_connections 2024-03-06T07:23:44.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.187+0000 7f842146f640 1 -- 172.21.15.122:0/3662624332 wait complete. 2024-03-06T07:23:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-03-06T07:23:43.352031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254492,"kb_used_data":5036,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720228,"statfs":{"total":773094113280,"available":772833513472,"internally_reserved":0,"allocated":5156864,"data_stored":2765721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471774"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896350+0000","last_change":"2024-03-06T07:23:39.896350+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896350+0000","last_undegraded":"2024-03-06T07:23:39.896350+0000","last_fullsized":"2024-03-06T07:23:39.896350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902983+0000","last_change":"2024-03-06T07:23:39.902983+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902983+0000","last_undegraded":"2024-03-06T07:23:39.902983+0000","last_fullsized":"2024-03-06T07:23:39.902983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895197+0000","last_change":"2024-03-06T07:23:39.895197+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895197+0000","last_undegraded":"2024-03-06T07:23:39.895197+0000","last_fullsized":"2024-03-06T07:23:39.895197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898515+0000","last_change":"2024-03-06T07:23:39.898515+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898515+0000","last_undegraded":"2024-03-06T07:23:39.898515+0000","last_fullsized":"2024-03-06T07:23:39.898515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897220+0000","last_change":"2024-03-06T07:23:39.897220+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897220+0000","last_undegraded":"2024-03-06T07:23:39.897220+0000","last_fullsized":"2024-03-06T07:23:39.897220+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899153+0000","last_change":"2024-03-06T07:23:39.899153+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899153+0000","last_undegraded":"2024-03-06T07:23:39.899153+0000","last_fullsized":"2024-03-06T07:23:39.899153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896635+0000","last_change":"2024-03-06T07:23:39.896635+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896635+0000","last_undegraded":"2024-03-06T07:23:39.896635+0000","last_fullsized":"2024-03-06T07:23:39.896635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895129+0000","last_change":"2024-03-06T07:23:39.895129+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895129+0000","last_undegraded":"2024-03-06T07:23:39.895129+0000","last_fullsized":"2024-03-06T07:23:39.895129+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900467+0000","last_change":"2024-03-06T07:23:39.900467+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900467+0000","last_undegraded":"2024-03-06T07:23:39.900467+0000","last_fullsized":"2024-03-06T07:23:39.900467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895165+0000","last_change":"2024-03-06T07:23:39.895165+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895165+0000","last_undegraded":"2024-03-06T07:23:39.895165+0000","last_fullsized":"2024-03-06T07:23:39.895165+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894678+0000","last_change":"2024-03-06T07:23:39.894678+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894678+0000","last_undegraded":"2024-03-06T07:23:39.894678+0000","last_fullsized":"2024-03-06T07:23:39.894678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902759+0000","last_change":"2024-03-06T07:23:39.902759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902759+0000","last_undegraded":"2024-03-06T07:23:39.902759+0000","last_fullsized":"2024-03-06T07:23:39.902759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900672+0000","last_change":"2024-03-06T07:23:39.900672+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900672+0000","last_undegraded":"2024-03-06T07:23:39.900672+0000","last_fullsized":"2024-03-06T07:23:39.900672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899316+0000","last_change":"2024-03-06T07:23:39.899316+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899316+0000","last_undegraded":"2024-03-06T07:23:39.899316+0000","last_fullsized":"2024-03-06T07:23:39.899316+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897463+0000","last_change":"2024-03-06T07:23:39.897463+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897463+0000","last_undegraded":"2024-03-06T07:23:39.897463+0000","last_fullsized":"2024-03-06T07:23:39.897463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893246+0000","last_change":"2024-03-06T07:23:39.893246+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893246+0000","last_undegraded":"2024-03-06T07:23:39.893246+0000","last_fullsized":"2024-03-06T07:23:39.893246+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894229+0000","last_change":"2024-03-06T07:23:39.894229+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894229+0000","last_undegraded":"2024-03-06T07:23:39.894229+0000","last_fullsized":"2024-03-06T07:23:39.894229+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896126+0000","last_change":"2024-03-06T07:23:39.896126+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896126+0000","last_undegraded":"2024-03-06T07:23:39.896126+0000","last_fullsized":"2024-03-06T07:23:39.896126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894221+0000","last_change":"2024-03-06T07:23:39.894221+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894221+0000","last_undegraded":"2024-03-06T07:23:39.894221+0000","last_fullsized":"2024-03-06T07:23:39.894221+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900319+0000","last_change":"2024-03-06T07:23:39.900319+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900319+0000","last_undegraded":"2024-03-06T07:23:39.900319+0000","last_fullsized":"2024-03-06T07:23:39.900319+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901518+0000","last_change":"2024-03-06T07:23:39.901518+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901518+0000","last_undegraded":"2024-03-06T07:23:39.901518+0000","last_fullsized":"2024-03-06T07:23:39.901518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896014+0000","last_change":"2024-03-06T07:23:39.896014+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896014+0000","last_undegraded":"2024-03-06T07:23:39.896014+0000","last_fullsized":"2024-03-06T07:23:39.896014+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895095+0000","last_change":"2024-03-06T07:23:39.895095+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895095+0000","last_undegraded":"2024-03-06T07:23:39.895095+0000","last_fullsized":"2024-03-06T07:23:39.895095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892690+0000","last_change":"2024-03-06T07:23:39.892690+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892690+0000","last_undegraded":"2024-03-06T07:23:39.892690+0000","last_fullsized":"2024-03-06T07:23:39.892690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893762+0000","last_change":"2024-03-06T07:23:39.893762+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893762+0000","last_undegraded":"2024-03-06T07:23:39.893762+0000","last_fullsized":"2024-03-06T07:23:39.893762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897975+0000","last_change":"2024-03-06T07:23:39.897975+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897975+0000","last_undegraded":"2024-03-06T07:23:39.897975+0000","last_fullsized":"2024-03-06T07:23:39.897975+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893063+0000","last_change":"2024-03-06T07:23:39.893063+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893063+0000","last_undegraded":"2024-03-06T07:23:39.893063+0000","last_fullsized":"2024-03-06T07:23:39.893063+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898452+0000","last_change":"2024-03-06T07:23:39.898452+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898452+0000","last_undegraded":"2024-03-06T07:23:39.898452+0000","last_fullsized":"2024-03-06T07:23:39.898452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893292+0000","last_change":"2024-03-06T07:23:39.893292+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893292+0000","last_undegraded":"2024-03-06T07:23:39.893292+0000","last_fullsized":"2024-03-06T07:23:39.893292+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893704+0000","last_change":"2024-03-06T07:23:39.893704+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893704+0000","last_undegraded":"2024-03-06T07:23:39.893704+0000","last_fullsized":"2024-03-06T07:23:39.893704+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896976+0000","last_change":"2024-03-06T07:23:39.896976+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896976+0000","last_undegraded":"2024-03-06T07:23:39.896976+0000","last_fullsized":"2024-03-06T07:23:39.896976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894325+0000","last_change":"2024-03-06T07:23:39.894325+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894325+0000","last_undegraded":"2024-03-06T07:23:39.894325+0000","last_fullsized":"2024-03-06T07:23:39.894325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898773+0000","last_change":"2024-03-06T07:23:39.898773+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898773+0000","last_undegraded":"2024-03-06T07:23:39.898773+0000","last_fullsized":"2024-03-06T07:23:39.898773+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.902762+0000","last_change":"2024-03-06T07:23:39.900097+0000","last_active":"2024-03-06T07:23:39.902762+0000","last_peered":"2024-03-06T07:23:39.902762+0000","last_clean":"2024-03-06T07:23:39.902762+0000","last_became_active":"2024-03-06T07:23:39.899808+0000","last_became_peered":"2024-03-06T07:23:39.899808+0000","last_unstale":"2024-03-06T07:23:39.902762+0000","last_undegraded":"2024-03-06T07:23:39.902762+0000","last_fullsized":"2024-03-06T07:23:39.902762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893338+0000","last_change":"2024-03-06T07:23:39.893338+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893338+0000","last_undegraded":"2024-03-06T07:23:39.893338+0000","last_fullsized":"2024-03-06T07:23:39.893338+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907516+0000","last_change":"2024-03-06T07:23:39.899915+0000","last_active":"2024-03-06T07:23:39.907516+0000","last_peered":"2024-03-06T07:23:39.907516+0000","last_clean":"2024-03-06T07:23:39.907516+0000","last_became_active":"2024-03-06T07:23:39.899555+0000","last_became_peered":"2024-03-06T07:23:39.899555+0000","last_unstale":"2024-03-06T07:23:39.907516+0000","last_undegraded":"2024-03-06T07:23:39.907516+0000","last_fullsized":"2024-03-06T07:23:39.907516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:45.519711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898062+0000","last_change":"2024-03-06T07:23:39.898062+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898062+0000","last_undegraded":"2024-03-06T07:23:39.898062+0000","last_fullsized":"2024-03-06T07:23:39.898062+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915812+0000","last_change":"2024-03-06T07:23:39.911329+0000","last_active":"2024-03-06T07:23:39.915812+0000","last_peered":"2024-03-06T07:23:39.915812+0000","last_clean":"2024-03-06T07:23:39.915812+0000","last_became_active":"2024-03-06T07:23:39.911011+0000","last_became_peered":"2024-03-06T07:23:39.911011+0000","last_unstale":"2024-03-06T07:23:39.915812+0000","last_undegraded":"2024-03-06T07:23:39.915812+0000","last_fullsized":"2024-03-06T07:23:39.915812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:45.430603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.904164+0000","last_change":"2024-03-06T07:23:39.904164+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.904164+0000","last_undegraded":"2024-03-06T07:23:39.904164+0000","last_fullsized":"2024-03-06T07:23:39.904164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910560+0000","last_change":"2024-03-06T07:23:39.908704+0000","last_active":"2024-03-06T07:23:39.910560+0000","last_peered":"2024-03-06T07:23:39.910560+0000","last_clean":"2024-03-06T07:23:39.910560+0000","last_became_active":"2024-03-06T07:23:39.908289+0000","last_became_peered":"2024-03-06T07:23:39.908289+0000","last_unstale":"2024-03-06T07:23:39.910560+0000","last_undegraded":"2024-03-06T07:23:39.910560+0000","last_fullsized":"2024-03-06T07:23:39.910560+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893272+0000","last_change":"2024-03-06T07:23:39.893272+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893272+0000","last_undegraded":"2024-03-06T07:23:39.893272+0000","last_fullsized":"2024-03-06T07:23:39.893272+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.909418+0000","last_change":"2024-03-06T07:23:39.908607+0000","last_active":"2024-03-06T07:23:39.909418+0000","last_peered":"2024-03-06T07:23:39.909418+0000","last_clean":"2024-03-06T07:23:39.909418+0000","last_became_active":"2024-03-06T07:23:39.908334+0000","last_became_peered":"2024-03-06T07:23:39.908334+0000","last_unstale":"2024-03-06T07:23:39.909418+0000","last_undegraded":"2024-03-06T07:23:39.909418+0000","last_fullsized":"2024-03-06T07:23:39.909418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:06.382376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899907+0000","last_change":"2024-03-06T07:23:39.899907+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899907+0000","last_undegraded":"2024-03-06T07:23:39.899907+0000","last_fullsized":"2024-03-06T07:23:39.899907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913163+0000","last_change":"2024-03-06T07:23:39.908741+0000","last_active":"2024-03-06T07:23:39.913163+0000","last_peered":"2024-03-06T07:23:39.913163+0000","last_clean":"2024-03-06T07:23:39.913163+0000","last_became_active":"2024-03-06T07:23:39.908470+0000","last_became_peered":"2024-03-06T07:23:39.908470+0000","last_unstale":"2024-03-06T07:23:39.913163+0000","last_undegraded":"2024-03-06T07:23:39.913163+0000","last_fullsized":"2024-03-06T07:23:39.913163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:07.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894696+0000","last_change":"2024-03-06T07:23:39.894696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894696+0000","last_undegraded":"2024-03-06T07:23:39.894696+0000","last_fullsized":"2024-03-06T07:23:39.894696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913247+0000","last_change":"2024-03-06T07:23:39.909770+0000","last_active":"2024-03-06T07:23:39.913247+0000","last_peered":"2024-03-06T07:23:39.913247+0000","last_clean":"2024-03-06T07:23:39.913247+0000","last_became_active":"2024-03-06T07:23:39.909494+0000","last_became_peered":"2024-03-06T07:23:39.909494+0000","last_unstale":"2024-03-06T07:23:39.913247+0000","last_undegraded":"2024-03-06T07:23:39.913247+0000","last_fullsized":"2024-03-06T07:23:39.913247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895537+0000","last_change":"2024-03-06T07:23:39.895537+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895537+0000","last_undegraded":"2024-03-06T07:23:39.895537+0000","last_fullsized":"2024-03-06T07:23:39.895537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910678+0000","last_change":"2024-03-06T07:23:39.906386+0000","last_active":"2024-03-06T07:23:39.910678+0000","last_peered":"2024-03-06T07:23:39.910678+0000","last_clean":"2024-03-06T07:23:39.910678+0000","last_became_active":"2024-03-06T07:23:39.905390+0000","last_became_peered":"2024-03-06T07:23:39.905390+0000","last_unstale":"2024-03-06T07:23:39.910678+0000","last_undegraded":"2024-03-06T07:23:39.910678+0000","last_fullsized":"2024-03-06T07:23:39.910678+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895696+0000","last_change":"2024-03-06T07:23:39.895696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895696+0000","last_undegraded":"2024-03-06T07:23:39.895696+0000","last_fullsized":"2024-03-06T07:23:39.895696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907953+0000","last_change":"2024-03-06T07:23:39.901277+0000","last_active":"2024-03-06T07:23:39.907953+0000","last_peered":"2024-03-06T07:23:39.907953+0000","last_clean":"2024-03-06T07:23:39.907953+0000","last_became_active":"2024-03-06T07:23:39.900996+0000","last_became_peered":"2024-03-06T07:23:39.900996+0000","last_unstale":"2024-03-06T07:23:39.907953+0000","last_undegraded":"2024-03-06T07:23:39.907953+0000","last_fullsized":"2024-03-06T07:23:39.907953+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:05.084281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895950+0000","last_change":"2024-03-06T07:23:39.895950+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895950+0000","last_undegraded":"2024-03-06T07:23:39.895950+0000","last_fullsized":"2024-03-06T07:23:39.895950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910643+0000","last_change":"2024-03-06T07:23:39.904382+0000","last_active":"2024-03-06T07:23:39.910643+0000","last_peered":"2024-03-06T07:23:39.910643+0000","last_clean":"2024-03-06T07:23:39.910643+0000","last_became_active":"2024-03-06T07:23:39.904076+0000","last_became_peered":"2024-03-06T07:23:39.904076+0000","last_unstale":"2024-03-06T07:23:39.910643+0000","last_undegraded":"2024-03-06T07:23:39.910643+0000","last_fullsized":"2024-03-06T07:23:39.910643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:51.296844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898603+0000","last_change":"2024-03-06T07:23:39.898603+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898603+0000","last_undegraded":"2024-03-06T07:23:39.898603+0000","last_fullsized":"2024-03-06T07:23:39.898603+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913122+0000","last_change":"2024-03-06T07:23:39.909133+0000","last_active":"2024-03-06T07:23:39.913122+0000","last_peered":"2024-03-06T07:23:39.913122+0000","last_clean":"2024-03-06T07:23:39.913122+0000","last_became_active":"2024-03-06T07:23:39.908563+0000","last_became_peered":"2024-03-06T07:23:39.908563+0000","last_unstale":"2024-03-06T07:23:39.913122+0000","last_undegraded":"2024-03-06T07:23:39.913122+0000","last_fullsized":"2024-03-06T07:23:39.913122+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896306+0000","last_change":"2024-03-06T07:23:39.896306+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896306+0000","last_undegraded":"2024-03-06T07:23:39.896306+0000","last_fullsized":"2024-03-06T07:23:39.896306+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.900855+0000","last_change":"2024-03-06T07:23:39.896528+0000","last_active":"2024-03-06T07:23:39.900855+0000","last_peered":"2024-03-06T07:23:39.900855+0000","last_clean":"2024-03-06T07:23:39.900855+0000","last_became_active":"2024-03-06T07:23:39.896142+0000","last_became_peered":"2024-03-06T07:23:39.896142+0000","last_unstale":"2024-03-06T07:23:39.900855+0000","last_undegraded":"2024-03-06T07:23:39.900855+0000","last_fullsized":"2024-03-06T07:23:39.900855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:17.233793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892802+0000","last_change":"2024-03-06T07:23:39.892802+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892802+0000","last_undegraded":"2024-03-06T07:23:39.892802+0000","last_fullsized":"2024-03-06T07:23:39.892802+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906489+0000","last_change":"2024-03-06T07:23:39.905467+0000","last_active":"2024-03-06T07:23:39.906489+0000","last_peered":"2024-03-06T07:23:39.906489+0000","last_clean":"2024-03-06T07:23:39.906489+0000","last_became_active":"2024-03-06T07:23:39.905185+0000","last_became_peered":"2024-03-06T07:23:39.905185+0000","last_unstale":"2024-03-06T07:23:39.906489+0000","last_undegraded":"2024-03-06T07:23:39.906489+0000","last_fullsized":"2024-03-06T07:23:39.906489+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:48.343179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896114+0000","last_change":"2024-03-06T07:23:39.896114+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896114+0000","last_undegraded":"2024-03-06T07:23:39.896114+0000","last_fullsized":"2024-03-06T07:23:39.896114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908901+0000","last_change":"2024-03-06T07:23:39.905513+0000","last_active":"2024-03-06T07:23:39.908901+0000","last_peered":"2024-03-06T07:23:39.908901+0000","last_clean":"2024-03-06T07:23:39.908901+0000","last_became_active":"2024-03-06T07:23:39.905121+0000","last_became_peered":"2024-03-06T07:23:39.905121+0000","last_unstale":"2024-03-06T07:23:39.908901+0000","last_undegraded":"2024-03-06T07:23:39.908901+0000","last_fullsized":"2024-03-06T07:23:39.908901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:57.881886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893960+0000","last_change":"2024-03-06T07:23:39.893960+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893960+0000","last_undegraded":"2024-03-06T07:23:39.893960+0000","last_fullsized":"2024-03-06T07:23:39.893960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906161+0000","last_change":"2024-03-06T07:23:39.902543+0000","last_active":"2024-03-06T07:23:39.906161+0000","last_peered":"2024-03-06T07:23:39.906161+0000","last_clean":"2024-03-06T07:23:39.906161+0000","last_became_active":"2024-03-06T07:23:39.902271+0000","last_became_peered":"2024-03-06T07:23:39.902271+0000","last_unstale":"2024-03-06T07:23:39.906161+0000","last_undegraded":"2024-03-06T07:23:39.906161+0000","last_fullsized":"2024-03-06T07:23:39.906161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:42.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896222+0000","last_change":"2024-03-06T07:23:39.896222+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896222+0000","last_undegraded":"2024-03-06T07:23:39.896222+0000","last_fullsized":"2024-03-06T07:23:39.896222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904631+0000","last_change":"2024-03-06T07:23:39.903255+0000","last_active":"2024-03-06T07:23:39.904631+0000","last_peered":"2024-03-06T07:23:39.904631+0000","last_clean":"2024-03-06T07:23:39.904631+0000","last_became_active":"2024-03-06T07:23:39.902984+0000","last_became_peered":"2024-03-06T07:23:39.902984+0000","last_unstale":"2024-03-06T07:23:39.904631+0000","last_undegraded":"2024-03-06T07:23:39.904631+0000","last_fullsized":"2024-03-06T07:23:39.904631+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:06.103875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894330+0000","last_change":"2024-03-06T07:23:39.894330+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894330+0000","last_undegraded":"2024-03-06T07:23:39.894330+0000","last_fullsized":"2024-03-06T07:23:39.894330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913144+0000","last_change":"2024-03-06T07:23:39.906113+0000","last_active":"2024-03-06T07:23:39.913144+0000","last_peered":"2024-03-06T07:23:39.913144+0000","last_clean":"2024-03-06T07:23:39.913144+0000","last_became_active":"2024-03-06T07:23:39.905836+0000","last_became_peered":"2024-03-06T07:23:39.905836+0000","last_unstale":"2024-03-06T07:23:39.913144+0000","last_undegraded":"2024-03-06T07:23:39.913144+0000","last_fullsized":"2024-03-06T07:23:39.913144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:50.974905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896241+0000","last_change":"2024-03-06T07:23:39.896241+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896241+0000","last_undegraded":"2024-03-06T07:23:39.896241+0000","last_fullsized":"2024-03-06T07:23:39.896241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908710+0000","last_change":"2024-03-06T07:23:39.904607+0000","last_active":"2024-03-06T07:23:39.908710+0000","last_peered":"2024-03-06T07:23:39.908710+0000","last_clean":"2024-03-06T07:23:39.908710+0000","last_became_active":"2024-03-06T07:23:39.904289+0000","last_became_peered":"2024-03-06T07:23:39.904289+0000","last_unstale":"2024-03-06T07:23:39.908710+0000","last_undegraded":"2024-03-06T07:23:39.908710+0000","last_fullsized":"2024-03-06T07:23:39.908710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:47.230884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893759+0000","last_change":"2024-03-06T07:23:39.893759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893759+0000","last_undegraded":"2024-03-06T07:23:39.893759+0000","last_fullsized":"2024-03-06T07:23:39.893759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913774+0000","last_change":"2024-03-06T07:23:39.911859+0000","last_active":"2024-03-06T07:23:39.913774+0000","last_peered":"2024-03-06T07:23:39.913774+0000","last_clean":"2024-03-06T07:23:39.913774+0000","last_became_active":"2024-03-06T07:23:39.911529+0000","last_became_peered":"2024-03-06T07:23:39.911529+0000","last_unstale":"2024-03-06T07:23:39.913774+0000","last_undegraded":"2024-03-06T07:23:39.913774+0000","last_fullsized":"2024-03-06T07:23:39.913774+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894652+0000","last_change":"2024-03-06T07:23:39.894652+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894652+0000","last_undegraded":"2024-03-06T07:23:39.894652+0000","last_fullsized":"2024-03-06T07:23:39.894652+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.911862+0000","last_change":"2024-03-06T07:23:39.911223+0000","last_active":"2024-03-06T07:23:39.911862+0000","last_peered":"2024-03-06T07:23:39.911862+0000","last_clean":"2024-03-06T07:23:39.911862+0000","last_became_active":"2024-03-06T07:23:39.910935+0000","last_became_peered":"2024-03-06T07:23:39.910935+0000","last_unstale":"2024-03-06T07:23:39.911862+0000","last_undegraded":"2024-03-06T07:23:39.911862+0000","last_fullsized":"2024-03-06T07:23:39.911862+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:46.994267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897546+0000","last_change":"2024-03-06T07:23:39.897546+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897546+0000","last_undegraded":"2024-03-06T07:23:39.897546+0000","last_fullsized":"2024-03-06T07:23:39.897546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.905864+0000","last_change":"2024-03-06T07:23:39.903773+0000","last_active":"2024-03-06T07:23:39.905864+0000","last_peered":"2024-03-06T07:23:39.905864+0000","last_clean":"2024-03-06T07:23:39.905864+0000","last_became_active":"2024-03-06T07:23:39.903494+0000","last_became_peered":"2024-03-06T07:23:39.903494+0000","last_unstale":"2024-03-06T07:23:39.905864+0000","last_undegraded":"2024-03-06T07:23:39.905864+0000","last_fullsized":"2024-03-06T07:23:39.905864+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:14.991973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898830+0000","last_change":"2024-03-06T07:23:39.898830+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898830+0000","last_undegraded":"2024-03-06T07:23:39.898830+0000","last_fullsized":"2024-03-06T07:23:39.898830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913738+0000","last_change":"2024-03-06T07:23:39.911261+0000","last_active":"2024-03-06T07:23:39.913738+0000","last_peered":"2024-03-06T07:23:39.913738+0000","last_clean":"2024-03-06T07:23:39.913738+0000","last_became_active":"2024-03-06T07:23:39.910917+0000","last_became_peered":"2024-03-06T07:23:39.910917+0000","last_unstale":"2024-03-06T07:23:39.913738+0000","last_undegraded":"2024-03-06T07:23:39.913738+0000","last_fullsized":"2024-03-06T07:23:39.913738+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893681+0000","last_change":"2024-03-06T07:23:39.893681+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893681+0000","last_undegraded":"2024-03-06T07:23:39.893681+0000","last_fullsized":"2024-03-06T07:23:39.893681+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913098+0000","last_change":"2024-03-06T07:23:39.909408+0000","last_active":"2024-03-06T07:23:39.913098+0000","last_peered":"2024-03-06T07:23:39.913098+0000","last_clean":"2024-03-06T07:23:39.913098+0000","last_became_active":"2024-03-06T07:23:39.909128+0000","last_became_peered":"2024-03-06T07:23:39.909128+0000","last_unstale":"2024-03-06T07:23:39.913098+0000","last_undegraded":"2024-03-06T07:23:39.913098+0000","last_fullsized":"2024-03-06T07:23:39.913098+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.087590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895570+0000","last_change":"2024-03-06T07:23:39.895570+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895570+0000","last_undegraded":"2024-03-06T07:23:39.895570+0000","last_fullsized":"2024-03-06T07:23:39.895570+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908963+0000","last_change":"2024-03-06T07:23:39.905798+0000","last_active":"2024-03-06T07:23:39.908963+0000","last_peered":"2024-03-06T07:23:39.908963+0000","last_clean":"2024-03-06T07:23:39.908963+0000","last_became_active":"2024-03-06T07:23:39.905513+0000","last_became_peered":"2024-03-06T07:23:39.905513+0000","last_unstale":"2024-03-06T07:23:39.908963+0000","last_undegraded":"2024-03-06T07:23:39.908963+0000","last_fullsized":"2024-03-06T07:23:39.908963+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892814+0000","last_change":"2024-03-06T07:23:39.892814+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892814+0000","last_undegraded":"2024-03-06T07:23:39.892814+0000","last_fullsized":"2024-03-06T07:23:39.892814+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906475+0000","last_change":"2024-03-06T07:23:39.905335+0000","last_active":"2024-03-06T07:23:39.906475+0000","last_peered":"2024-03-06T07:23:39.906475+0000","last_clean":"2024-03-06T07:23:39.906475+0000","last_became_active":"2024-03-06T07:23:39.903169+0000","last_became_peered":"2024-03-06T07:23:39.903169+0000","last_unstale":"2024-03-06T07:23:39.906475+0000","last_undegraded":"2024-03-06T07:23:39.906475+0000","last_fullsized":"2024-03-06T07:23:39.906475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:44.722551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894171+0000","last_change":"2024-03-06T07:23:39.894171+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894171+0000","last_undegraded":"2024-03-06T07:23:39.894171+0000","last_fullsized":"2024-03-06T07:23:39.894171+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910527+0000","last_change":"2024-03-06T07:23:39.908894+0000","last_active":"2024-03-06T07:23:39.910527+0000","last_peered":"2024-03-06T07:23:39.910527+0000","last_clean":"2024-03-06T07:23:39.910527+0000","last_became_active":"2024-03-06T07:23:39.908206+0000","last_became_peered":"2024-03-06T07:23:39.908206+0000","last_unstale":"2024-03-06T07:23:39.910527+0000","last_undegraded":"2024-03-06T07:23:39.910527+0000","last_fullsized":"2024-03-06T07:23:39.910527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895809+0000","last_change":"2024-03-06T07:23:39.895809+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895809+0000","last_undegraded":"2024-03-06T07:23:39.895809+0000","last_fullsized":"2024-03-06T07:23:39.895809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915464+0000","last_change":"2024-03-06T07:23:39.909279+0000","last_active":"2024-03-06T07:23:39.915464+0000","last_peered":"2024-03-06T07:23:39.915464+0000","last_clean":"2024-03-06T07:23:39.915464+0000","last_became_active":"2024-03-06T07:23:39.908955+0000","last_became_peered":"2024-03-06T07:23:39.908955+0000","last_unstale":"2024-03-06T07:23:39.915464+0000","last_undegraded":"2024-03-06T07:23:39.915464+0000","last_fullsized":"2024-03-06T07:23:39.915464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:42.900316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897107+0000","last_change":"2024-03-06T07:23:39.897107+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897107+0000","last_undegraded":"2024-03-06T07:23:39.897107+0000","last_fullsized":"2024-03-06T07:23:39.897107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910530+0000","last_change":"2024-03-06T07:23:39.908492+0000","last_active":"2024-03-06T07:23:39.910530+0000","last_peered":"2024-03-06T07:23:39.910530+0000","last_clean":"2024-03-06T07:23:39.910530+0000","last_became_active":"2024-03-06T07:23:39.908214+0000","last_became_peered":"2024-03-06T07:23:39.908214+0000","last_unstale":"2024-03-06T07:23:39.910530+0000","last_undegraded":"2024-03-06T07:23:39.910530+0000","last_fullsized":"2024-03-06T07:23:39.910530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901075+0000","last_change":"2024-03-06T07:23:39.901075+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901075+0000","last_undegraded":"2024-03-06T07:23:39.901075+0000","last_fullsized":"2024-03-06T07:23:39.901075+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907658+0000","last_change":"2024-03-06T07:23:39.901912+0000","last_active":"2024-03-06T07:23:39.907658+0000","last_peered":"2024-03-06T07:23:39.907658+0000","last_clean":"2024-03-06T07:23:39.907658+0000","last_became_active":"2024-03-06T07:23:39.901713+0000","last_became_peered":"2024-03-06T07:23:39.901713+0000","last_unstale":"2024-03-06T07:23:39.907658+0000","last_undegraded":"2024-03-06T07:23:39.907658+0000","last_fullsized":"2024-03-06T07:23:39.907658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:31.271276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897613+0000","last_change":"2024-03-06T07:23:39.897613+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897613+0000","last_undegraded":"2024-03-06T07:23:39.897613+0000","last_fullsized":"2024-03-06T07:23:39.897613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915433+0000","last_change":"2024-03-06T07:23:39.908817+0000","last_active":"2024-03-06T07:23:39.915433+0000","last_peered":"2024-03-06T07:23:39.915433+0000","last_clean":"2024-03-06T07:23:39.915433+0000","last_became_active":"2024-03-06T07:23:39.908523+0000","last_became_peered":"2024-03-06T07:23:39.908523+0000","last_unstale":"2024-03-06T07:23:39.915433+0000","last_undegraded":"2024-03-06T07:23:39.915433+0000","last_fullsized":"2024-03-06T07:23:39.915433+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:23.516812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892303+0000","last_change":"2024-03-06T07:23:39.892303+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892303+0000","last_undegraded":"2024-03-06T07:23:39.892303+0000","last_fullsized":"2024-03-06T07:23:39.892303+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904611+0000","last_change":"2024-03-06T07:23:39.902253+0000","last_active":"2024-03-06T07:23:39.904611+0000","last_peered":"2024-03-06T07:23:39.904611+0000","last_clean":"2024-03-06T07:23:39.904611+0000","last_became_active":"2024-03-06T07:23:39.901938+0000","last_became_peered":"2024-03-06T07:23:39.901938+0000","last_unstale":"2024-03-06T07:23:39.904611+0000","last_undegraded":"2024-03-06T07:23:39.904611+0000","last_fullsized":"2024-03-06T07:23:39.904611+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:49.309308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894516+0000","last_change":"2024-03-06T07:23:39.894516+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894516+0000","last_undegraded":"2024-03-06T07:23:39.894516+0000","last_fullsized":"2024-03-06T07:23:39.894516+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913255+0000","last_change":"2024-03-06T07:23:39.908459+0000","last_active":"2024-03-06T07:23:39.913255+0000","last_peered":"2024-03-06T07:23:39.913255+0000","last_clean":"2024-03-06T07:23:39.913255+0000","last_became_active":"2024-03-06T07:23:39.908193+0000","last_became_peered":"2024-03-06T07:23:39.908193+0000","last_unstale":"2024-03-06T07:23:39.913255+0000","last_undegraded":"2024-03-06T07:23:39.913255+0000","last_fullsized":"2024-03-06T07:23:39.913255+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.898698+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:23:39.898698+0000","last_peered":"2024-03-06T07:23:39.898698+0000","last_clean":"2024-03-06T07:23:39.898698+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:23:39.898698+0000","last_undegraded":"2024-03-06T07:23:39.898698+0000","last_fullsized":"2024-03-06T07:23:39.898698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:44.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:44.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.338+0000 7fb8ae5a0640 1 Processor -- start 2024-03-06T07:23:44.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.338+0000 7fb8ae5a0640 1 -- start start 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ae5a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb8a8152bc0 0x7fb8a80b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ae5a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a815c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ae5a0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80b2960 con 0x7fb8a815c990 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ae5a0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a80b77d0 con 0x7fb8a8152bc0 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ae5a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8152fa0 con 0x7fb8a80b7370 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ad59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb8a8152bc0 0x7fb8a80b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8add9f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb8a815c990 0x7fb8a815f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51490/0 (socket says 172.21.15.122:51490) 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8add9f640 1 -- 172.21.15.122:0/1642436153 learned_addr learned my addr 172.21.15.122:0/1642436153 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8acd9d640 1 --2- 172.21.15.122:0/1642436153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ad59e640 1 -- 172.21.15.122:0/1642436153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 msgr2=0x7fb8a815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ad59e640 1 --2- 172.21.15.122:0/1642436153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.339+0000 7fb8ad59e640 1 -- 172.21.15.122:0/1642436153 >> v1:172.21.15.122:6789/0 conn(0x7fb8a815c990 legacy=0x7fb8a815f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.340+0000 7fb89ffff640 1 -- 172.21.15.122:0/1642436153 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734791643 0 0) 0x7fb8a0001490 con 0x7fb8a815c990 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.340+0000 7fb8ad59e640 1 -- 172.21.15.122:0/1642436153 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a80d2780 con 0x7fb8a8152bc0 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.340+0000 7fb8ad59e640 1 --2- 172.21.15.122:0/1642436153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a80b6e30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40097a0 tx=0x7fb8a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5421a20987dec31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.340+0000 7fb89ffff640 1 -- 172.21.15.122:0/1642436153 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4035020 con 0x7fb8a8152bc0 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.340+0000 7fb89ffff640 1 -- 172.21.15.122:0/1642436153 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4031830 con 0x7fb8a8152bc0 2024-03-06T07:23:44.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.340+0000 7fb89ffff640 1 -- 172.21.15.122:0/1642436153 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4031c00 con 0x7fb8a8152bc0 2024-03-06T07:23:44.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/1642436153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 msgr2=0x7fb8a80b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/1642436153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a80b6e30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40097a0 tx=0x7fb8a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/1642436153 shutdown_connections 2024-03-06T07:23:44.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/1642436153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/1642436153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a80b6e30 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/1642436153 >> 172.21.15.122:0/1642436153 conn(0x7fb8a809a9f0 msgr2=0x7fb8a809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/1642436153 shutdown_connections 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.341+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/1642436153 wait complete. 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.342+0000 7fb8ae5a0640 1 Processor -- start 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.342+0000 7fb8ae5a0640 1 -- start start 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ae5a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a81533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ae5a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a81e6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ae5a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8a815c990 0x7fb8a81e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ae5a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80d1ca0 con 0x7fb8a8152bc0 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ae5a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80d19c0 con 0x7fb8a815c990 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ae5a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80d1f80 con 0x7fb8a80b7370 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a81e6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a81e6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36004/0 (socket says 172.21.15.122:36004) 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 -- 172.21.15.122:0/2134301060 learned_addr learned my addr 172.21.15.122:0/2134301060 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8ad59e640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a81533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8add9f640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8a815c990 0x7fb8a81e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 -- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 msgr2=0x7fb8a81533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a81533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 -- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8a815c990 msgr2=0x7fb8a81e6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8a815c990 0x7fb8a81e6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 -- 172.21.15.122:0/2134301060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a8154de0 con 0x7fb8a8152bc0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.343+0000 7fb8acd9d640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a81e6150 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb89800e8a0 tx=0x7fb89800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.344+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898017070 con 0x7fb8a8152bc0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.344+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89800c940 con 0x7fb8a8152bc0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.344+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a813d840 con 0x7fb8a8152bc0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.344+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89800cd30 con 0x7fb8a8152bc0 2024-03-06T07:23:44.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.344+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a813de00 con 0x7fb8a8152bc0 2024-03-06T07:23:44.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.345+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a813e040 con 0x7fb8a8152bc0 2024-03-06T07:23:44.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.345+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb89801a070 con 0x7fb8a8152bc0 2024-03-06T07:23:44.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.345+0000 7fb89dffb640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb88c06b4d0 0x7fb88c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.346+0000 7fb8ad59e640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb88c06b4d0 0x7fb88c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.346+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb898089f50 con 0x7fb8a8152bc0 2024-03-06T07:23:44.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.346+0000 7fb8ad59e640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb88c06b4d0 0x7fb88c06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4036000 tx=0x7fb8a403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.350+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8a813e040 con 0x7fb8a8152bc0 2024-03-06T07:23:44.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.517+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8a80b6a30 con 0x7fb88c06b4d0 2024-03-06T07:23:44.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.522+0000 7fb89dffb640 1 -- 172.21.15.122:0/2134301060 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7fb8a4087380 con 0x7fb88c06b4d0 2024-03-06T07:23:44.527 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:44.528 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:23:44.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb88c06b4d0 msgr2=0x7fb88c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb88c06b4d0 0x7fb88c06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4036000 tx=0x7fb8a403d040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 msgr2=0x7fb8a81e6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a81e6150 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb89800e8a0 tx=0x7fb89800ed60 comp rx=0 tx=0).stop 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 shutdown_connections 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb88c06b4d0 0x7fb88c06d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8a815c990 0x7fb8a81e6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8a8152bc0 0x7fb8a81e6150 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 --2- 172.21.15.122:0/2134301060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8a80b7370 0x7fb8a81533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.525+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 >> 172.21.15.122:0/2134301060 conn(0x7fb8a809a9f0 msgr2=0x7fb8a814e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.526+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 shutdown_connections 2024-03-06T07:23:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.526+0000 7fb8ae5a0640 1 -- 172.21.15.122:0/2134301060 wait complete. 2024-03-06T07:23:44.538 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-03-06T07:23:43.352031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254492,"kb_used_data":5036,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720228,"statfs":{"total":773094113280,"available":772833513472,"internally_reserved":0,"allocated":5156864,"data_stored":2765721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471774"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896350+0000","last_change":"2024-03-06T07:23:39.896350+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896350+0000","last_undegraded":"2024-03-06T07:23:39.896350+0000","last_fullsized":"2024-03-06T07:23:39.896350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902983+0000","last_change":"2024-03-06T07:23:39.902983+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902983+0000","last_undegraded":"2024-03-06T07:23:39.902983+0000","last_fullsized":"2024-03-06T07:23:39.902983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895197+0000","last_change":"2024-03-06T07:23:39.895197+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895197+0000","last_undegraded":"2024-03-06T07:23:39.895197+0000","last_fullsized":"2024-03-06T07:23:39.895197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898515+0000","last_change":"2024-03-06T07:23:39.898515+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898515+0000","last_undegraded":"2024-03-06T07:23:39.898515+0000","last_fullsized":"2024-03-06T07:23:39.898515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897220+0000","last_change":"2024-03-06T07:23:39.897220+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897220+0000","last_undegraded":"2024-03-06T07:23:39.897220+0000","last_fullsized":"2024-03-06T07:23:39.897220+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899153+0000","last_change":"2024-03-06T07:23:39.899153+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899153+0000","last_undegraded":"2024-03-06T07:23:39.899153+0000","last_fullsized":"2024-03-06T07:23:39.899153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896635+0000","last_change":"2024-03-06T07:23:39.896635+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896635+0000","last_undegraded":"2024-03-06T07:23:39.896635+0000","last_fullsized":"2024-03-06T07:23:39.896635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895129+0000","last_change":"2024-03-06T07:23:39.895129+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895129+0000","last_undegraded":"2024-03-06T07:23:39.895129+0000","last_fullsized":"2024-03-06T07:23:39.895129+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900467+0000","last_change":"2024-03-06T07:23:39.900467+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900467+0000","last_undegraded":"2024-03-06T07:23:39.900467+0000","last_fullsized":"2024-03-06T07:23:39.900467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895165+0000","last_change":"2024-03-06T07:23:39.895165+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895165+0000","last_undegraded":"2024-03-06T07:23:39.895165+0000","last_fullsized":"2024-03-06T07:23:39.895165+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894678+0000","last_change":"2024-03-06T07:23:39.894678+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894678+0000","last_undegraded":"2024-03-06T07:23:39.894678+0000","last_fullsized":"2024-03-06T07:23:39.894678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902759+0000","last_change":"2024-03-06T07:23:39.902759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902759+0000","last_undegraded":"2024-03-06T07:23:39.902759+0000","last_fullsized":"2024-03-06T07:23:39.902759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900672+0000","last_change":"2024-03-06T07:23:39.900672+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900672+0000","last_undegraded":"2024-03-06T07:23:39.900672+0000","last_fullsized":"2024-03-06T07:23:39.900672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899316+0000","last_change":"2024-03-06T07:23:39.899316+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899316+0000","last_undegraded":"2024-03-06T07:23:39.899316+0000","last_fullsized":"2024-03-06T07:23:39.899316+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897463+0000","last_change":"2024-03-06T07:23:39.897463+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897463+0000","last_undegraded":"2024-03-06T07:23:39.897463+0000","last_fullsized":"2024-03-06T07:23:39.897463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893246+0000","last_change":"2024-03-06T07:23:39.893246+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893246+0000","last_undegraded":"2024-03-06T07:23:39.893246+0000","last_fullsized":"2024-03-06T07:23:39.893246+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894229+0000","last_change":"2024-03-06T07:23:39.894229+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894229+0000","last_undegraded":"2024-03-06T07:23:39.894229+0000","last_fullsized":"2024-03-06T07:23:39.894229+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896126+0000","last_change":"2024-03-06T07:23:39.896126+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896126+0000","last_undegraded":"2024-03-06T07:23:39.896126+0000","last_fullsized":"2024-03-06T07:23:39.896126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894221+0000","last_change":"2024-03-06T07:23:39.894221+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894221+0000","last_undegraded":"2024-03-06T07:23:39.894221+0000","last_fullsized":"2024-03-06T07:23:39.894221+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900319+0000","last_change":"2024-03-06T07:23:39.900319+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900319+0000","last_undegraded":"2024-03-06T07:23:39.900319+0000","last_fullsized":"2024-03-06T07:23:39.900319+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901518+0000","last_change":"2024-03-06T07:23:39.901518+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901518+0000","last_undegraded":"2024-03-06T07:23:39.901518+0000","last_fullsized":"2024-03-06T07:23:39.901518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896014+0000","last_change":"2024-03-06T07:23:39.896014+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896014+0000","last_undegraded":"2024-03-06T07:23:39.896014+0000","last_fullsized":"2024-03-06T07:23:39.896014+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895095+0000","last_change":"2024-03-06T07:23:39.895095+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895095+0000","last_undegraded":"2024-03-06T07:23:39.895095+0000","last_fullsized":"2024-03-06T07:23:39.895095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892690+0000","last_change":"2024-03-06T07:23:39.892690+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892690+0000","last_undegraded":"2024-03-06T07:23:39.892690+0000","last_fullsized":"2024-03-06T07:23:39.892690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893762+0000","last_change":"2024-03-06T07:23:39.893762+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893762+0000","last_undegraded":"2024-03-06T07:23:39.893762+0000","last_fullsized":"2024-03-06T07:23:39.893762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897975+0000","last_change":"2024-03-06T07:23:39.897975+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897975+0000","last_undegraded":"2024-03-06T07:23:39.897975+0000","last_fullsized":"2024-03-06T07:23:39.897975+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893063+0000","last_change":"2024-03-06T07:23:39.893063+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893063+0000","last_undegraded":"2024-03-06T07:23:39.893063+0000","last_fullsized":"2024-03-06T07:23:39.893063+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898452+0000","last_change":"2024-03-06T07:23:39.898452+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898452+0000","last_undegraded":"2024-03-06T07:23:39.898452+0000","last_fullsized":"2024-03-06T07:23:39.898452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893292+0000","last_change":"2024-03-06T07:23:39.893292+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893292+0000","last_undegraded":"2024-03-06T07:23:39.893292+0000","last_fullsized":"2024-03-06T07:23:39.893292+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893704+0000","last_change":"2024-03-06T07:23:39.893704+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893704+0000","last_undegraded":"2024-03-06T07:23:39.893704+0000","last_fullsized":"2024-03-06T07:23:39.893704+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896976+0000","last_change":"2024-03-06T07:23:39.896976+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896976+0000","last_undegraded":"2024-03-06T07:23:39.896976+0000","last_fullsized":"2024-03-06T07:23:39.896976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894325+0000","last_change":"2024-03-06T07:23:39.894325+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894325+0000","last_undegraded":"2024-03-06T07:23:39.894325+0000","last_fullsized":"2024-03-06T07:23:39.894325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898773+0000","last_change":"2024-03-06T07:23:39.898773+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898773+0000","last_undegraded":"2024-03-06T07:23:39.898773+0000","last_fullsized":"2024-03-06T07:23:39.898773+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.902762+0000","last_change":"2024-03-06T07:23:39.900097+0000","last_active":"2024-03-06T07:23:39.902762+0000","last_peered":"2024-03-06T07:23:39.902762+0000","last_clean":"2024-03-06T07:23:39.902762+0000","last_became_active":"2024-03-06T07:23:39.899808+0000","last_became_peered":"2024-03-06T07:23:39.899808+0000","last_unstale":"2024-03-06T07:23:39.902762+0000","last_undegraded":"2024-03-06T07:23:39.902762+0000","last_fullsized":"2024-03-06T07:23:39.902762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893338+0000","last_change":"2024-03-06T07:23:39.893338+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893338+0000","last_undegraded":"2024-03-06T07:23:39.893338+0000","last_fullsized":"2024-03-06T07:23:39.893338+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907516+0000","last_change":"2024-03-06T07:23:39.899915+0000","last_active":"2024-03-06T07:23:39.907516+0000","last_peered":"2024-03-06T07:23:39.907516+0000","last_clean":"2024-03-06T07:23:39.907516+0000","last_became_active":"2024-03-06T07:23:39.899555+0000","last_became_peered":"2024-03-06T07:23:39.899555+0000","last_unstale":"2024-03-06T07:23:39.907516+0000","last_undegraded":"2024-03-06T07:23:39.907516+0000","last_fullsized":"2024-03-06T07:23:39.907516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:45.519711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898062+0000","last_change":"2024-03-06T07:23:39.898062+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898062+0000","last_undegraded":"2024-03-06T07:23:39.898062+0000","last_fullsized":"2024-03-06T07:23:39.898062+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915812+0000","last_change":"2024-03-06T07:23:39.911329+0000","last_active":"2024-03-06T07:23:39.915812+0000","last_peered":"2024-03-06T07:23:39.915812+0000","last_clean":"2024-03-06T07:23:39.915812+0000","last_became_active":"2024-03-06T07:23:39.911011+0000","last_became_peered":"2024-03-06T07:23:39.911011+0000","last_unstale":"2024-03-06T07:23:39.915812+0000","last_undegraded":"2024-03-06T07:23:39.915812+0000","last_fullsized":"2024-03-06T07:23:39.915812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:45.430603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.904164+0000","last_change":"2024-03-06T07:23:39.904164+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.904164+0000","last_undegraded":"2024-03-06T07:23:39.904164+0000","last_fullsized":"2024-03-06T07:23:39.904164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910560+0000","last_change":"2024-03-06T07:23:39.908704+0000","last_active":"2024-03-06T07:23:39.910560+0000","last_peered":"2024-03-06T07:23:39.910560+0000","last_clean":"2024-03-06T07:23:39.910560+0000","last_became_active":"2024-03-06T07:23:39.908289+0000","last_became_peered":"2024-03-06T07:23:39.908289+0000","last_unstale":"2024-03-06T07:23:39.910560+0000","last_undegraded":"2024-03-06T07:23:39.910560+0000","last_fullsized":"2024-03-06T07:23:39.910560+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893272+0000","last_change":"2024-03-06T07:23:39.893272+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893272+0000","last_undegraded":"2024-03-06T07:23:39.893272+0000","last_fullsized":"2024-03-06T07:23:39.893272+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.909418+0000","last_change":"2024-03-06T07:23:39.908607+0000","last_active":"2024-03-06T07:23:39.909418+0000","last_peered":"2024-03-06T07:23:39.909418+0000","last_clean":"2024-03-06T07:23:39.909418+0000","last_became_active":"2024-03-06T07:23:39.908334+0000","last_became_peered":"2024-03-06T07:23:39.908334+0000","last_unstale":"2024-03-06T07:23:39.909418+0000","last_undegraded":"2024-03-06T07:23:39.909418+0000","last_fullsized":"2024-03-06T07:23:39.909418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:06.382376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899907+0000","last_change":"2024-03-06T07:23:39.899907+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899907+0000","last_undegraded":"2024-03-06T07:23:39.899907+0000","last_fullsized":"2024-03-06T07:23:39.899907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913163+0000","last_change":"2024-03-06T07:23:39.908741+0000","last_active":"2024-03-06T07:23:39.913163+0000","last_peered":"2024-03-06T07:23:39.913163+0000","last_clean":"2024-03-06T07:23:39.913163+0000","last_became_active":"2024-03-06T07:23:39.908470+0000","last_became_peered":"2024-03-06T07:23:39.908470+0000","last_unstale":"2024-03-06T07:23:39.913163+0000","last_undegraded":"2024-03-06T07:23:39.913163+0000","last_fullsized":"2024-03-06T07:23:39.913163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:07.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894696+0000","last_change":"2024-03-06T07:23:39.894696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894696+0000","last_undegraded":"2024-03-06T07:23:39.894696+0000","last_fullsized":"2024-03-06T07:23:39.894696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913247+0000","last_change":"2024-03-06T07:23:39.909770+0000","last_active":"2024-03-06T07:23:39.913247+0000","last_peered":"2024-03-06T07:23:39.913247+0000","last_clean":"2024-03-06T07:23:39.913247+0000","last_became_active":"2024-03-06T07:23:39.909494+0000","last_became_peered":"2024-03-06T07:23:39.909494+0000","last_unstale":"2024-03-06T07:23:39.913247+0000","last_undegraded":"2024-03-06T07:23:39.913247+0000","last_fullsized":"2024-03-06T07:23:39.913247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895537+0000","last_change":"2024-03-06T07:23:39.895537+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895537+0000","last_undegraded":"2024-03-06T07:23:39.895537+0000","last_fullsized":"2024-03-06T07:23:39.895537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910678+0000","last_change":"2024-03-06T07:23:39.906386+0000","last_active":"2024-03-06T07:23:39.910678+0000","last_peered":"2024-03-06T07:23:39.910678+0000","last_clean":"2024-03-06T07:23:39.910678+0000","last_became_active":"2024-03-06T07:23:39.905390+0000","last_became_peered":"2024-03-06T07:23:39.905390+0000","last_unstale":"2024-03-06T07:23:39.910678+0000","last_undegraded":"2024-03-06T07:23:39.910678+0000","last_fullsized":"2024-03-06T07:23:39.910678+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895696+0000","last_change":"2024-03-06T07:23:39.895696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895696+0000","last_undegraded":"2024-03-06T07:23:39.895696+0000","last_fullsized":"2024-03-06T07:23:39.895696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907953+0000","last_change":"2024-03-06T07:23:39.901277+0000","last_active":"2024-03-06T07:23:39.907953+0000","last_peered":"2024-03-06T07:23:39.907953+0000","last_clean":"2024-03-06T07:23:39.907953+0000","last_became_active":"2024-03-06T07:23:39.900996+0000","last_became_peered":"2024-03-06T07:23:39.900996+0000","last_unstale":"2024-03-06T07:23:39.907953+0000","last_undegraded":"2024-03-06T07:23:39.907953+0000","last_fullsized":"2024-03-06T07:23:39.907953+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:05.084281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895950+0000","last_change":"2024-03-06T07:23:39.895950+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895950+0000","last_undegraded":"2024-03-06T07:23:39.895950+0000","last_fullsized":"2024-03-06T07:23:39.895950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910643+0000","last_change":"2024-03-06T07:23:39.904382+0000","last_active":"2024-03-06T07:23:39.910643+0000","last_peered":"2024-03-06T07:23:39.910643+0000","last_clean":"2024-03-06T07:23:39.910643+0000","last_became_active":"2024-03-06T07:23:39.904076+0000","last_became_peered":"2024-03-06T07:23:39.904076+0000","last_unstale":"2024-03-06T07:23:39.910643+0000","last_undegraded":"2024-03-06T07:23:39.910643+0000","last_fullsized":"2024-03-06T07:23:39.910643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:51.296844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898603+0000","last_change":"2024-03-06T07:23:39.898603+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898603+0000","last_undegraded":"2024-03-06T07:23:39.898603+0000","last_fullsized":"2024-03-06T07:23:39.898603+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913122+0000","last_change":"2024-03-06T07:23:39.909133+0000","last_active":"2024-03-06T07:23:39.913122+0000","last_peered":"2024-03-06T07:23:39.913122+0000","last_clean":"2024-03-06T07:23:39.913122+0000","last_became_active":"2024-03-06T07:23:39.908563+0000","last_became_peered":"2024-03-06T07:23:39.908563+0000","last_unstale":"2024-03-06T07:23:39.913122+0000","last_undegraded":"2024-03-06T07:23:39.913122+0000","last_fullsized":"2024-03-06T07:23:39.913122+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896306+0000","last_change":"2024-03-06T07:23:39.896306+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896306+0000","last_undegraded":"2024-03-06T07:23:39.896306+0000","last_fullsized":"2024-03-06T07:23:39.896306+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.900855+0000","last_change":"2024-03-06T07:23:39.896528+0000","last_active":"2024-03-06T07:23:39.900855+0000","last_peered":"2024-03-06T07:23:39.900855+0000","last_clean":"2024-03-06T07:23:39.900855+0000","last_became_active":"2024-03-06T07:23:39.896142+0000","last_became_peered":"2024-03-06T07:23:39.896142+0000","last_unstale":"2024-03-06T07:23:39.900855+0000","last_undegraded":"2024-03-06T07:23:39.900855+0000","last_fullsized":"2024-03-06T07:23:39.900855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:17.233793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892802+0000","last_change":"2024-03-06T07:23:39.892802+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892802+0000","last_undegraded":"2024-03-06T07:23:39.892802+0000","last_fullsized":"2024-03-06T07:23:39.892802+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906489+0000","last_change":"2024-03-06T07:23:39.905467+0000","last_active":"2024-03-06T07:23:39.906489+0000","last_peered":"2024-03-06T07:23:39.906489+0000","last_clean":"2024-03-06T07:23:39.906489+0000","last_became_active":"2024-03-06T07:23:39.905185+0000","last_became_peered":"2024-03-06T07:23:39.905185+0000","last_unstale":"2024-03-06T07:23:39.906489+0000","last_undegraded":"2024-03-06T07:23:39.906489+0000","last_fullsized":"2024-03-06T07:23:39.906489+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:48.343179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896114+0000","last_change":"2024-03-06T07:23:39.896114+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896114+0000","last_undegraded":"2024-03-06T07:23:39.896114+0000","last_fullsized":"2024-03-06T07:23:39.896114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908901+0000","last_change":"2024-03-06T07:23:39.905513+0000","last_active":"2024-03-06T07:23:39.908901+0000","last_peered":"2024-03-06T07:23:39.908901+0000","last_clean":"2024-03-06T07:23:39.908901+0000","last_became_active":"2024-03-06T07:23:39.905121+0000","last_became_peered":"2024-03-06T07:23:39.905121+0000","last_unstale":"2024-03-06T07:23:39.908901+0000","last_undegraded":"2024-03-06T07:23:39.908901+0000","last_fullsized":"2024-03-06T07:23:39.908901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:57.881886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893960+0000","last_change":"2024-03-06T07:23:39.893960+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893960+0000","last_undegraded":"2024-03-06T07:23:39.893960+0000","last_fullsized":"2024-03-06T07:23:39.893960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906161+0000","last_change":"2024-03-06T07:23:39.902543+0000","last_active":"2024-03-06T07:23:39.906161+0000","last_peered":"2024-03-06T07:23:39.906161+0000","last_clean":"2024-03-06T07:23:39.906161+0000","last_became_active":"2024-03-06T07:23:39.902271+0000","last_became_peered":"2024-03-06T07:23:39.902271+0000","last_unstale":"2024-03-06T07:23:39.906161+0000","last_undegraded":"2024-03-06T07:23:39.906161+0000","last_fullsized":"2024-03-06T07:23:39.906161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:42.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896222+0000","last_change":"2024-03-06T07:23:39.896222+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896222+0000","last_undegraded":"2024-03-06T07:23:39.896222+0000","last_fullsized":"2024-03-06T07:23:39.896222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904631+0000","last_change":"2024-03-06T07:23:39.903255+0000","last_active":"2024-03-06T07:23:39.904631+0000","last_peered":"2024-03-06T07:23:39.904631+0000","last_clean":"2024-03-06T07:23:39.904631+0000","last_became_active":"2024-03-06T07:23:39.902984+0000","last_became_peered":"2024-03-06T07:23:39.902984+0000","last_unstale":"2024-03-06T07:23:39.904631+0000","last_undegraded":"2024-03-06T07:23:39.904631+0000","last_fullsized":"2024-03-06T07:23:39.904631+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:06.103875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894330+0000","last_change":"2024-03-06T07:23:39.894330+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894330+0000","last_undegraded":"2024-03-06T07:23:39.894330+0000","last_fullsized":"2024-03-06T07:23:39.894330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913144+0000","last_change":"2024-03-06T07:23:39.906113+0000","last_active":"2024-03-06T07:23:39.913144+0000","last_peered":"2024-03-06T07:23:39.913144+0000","last_clean":"2024-03-06T07:23:39.913144+0000","last_became_active":"2024-03-06T07:23:39.905836+0000","last_became_peered":"2024-03-06T07:23:39.905836+0000","last_unstale":"2024-03-06T07:23:39.913144+0000","last_undegraded":"2024-03-06T07:23:39.913144+0000","last_fullsized":"2024-03-06T07:23:39.913144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:50.974905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896241+0000","last_change":"2024-03-06T07:23:39.896241+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896241+0000","last_undegraded":"2024-03-06T07:23:39.896241+0000","last_fullsized":"2024-03-06T07:23:39.896241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908710+0000","last_change":"2024-03-06T07:23:39.904607+0000","last_active":"2024-03-06T07:23:39.908710+0000","last_peered":"2024-03-06T07:23:39.908710+0000","last_clean":"2024-03-06T07:23:39.908710+0000","last_became_active":"2024-03-06T07:23:39.904289+0000","last_became_peered":"2024-03-06T07:23:39.904289+0000","last_unstale":"2024-03-06T07:23:39.908710+0000","last_undegraded":"2024-03-06T07:23:39.908710+0000","last_fullsized":"2024-03-06T07:23:39.908710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:47.230884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893759+0000","last_change":"2024-03-06T07:23:39.893759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893759+0000","last_undegraded":"2024-03-06T07:23:39.893759+0000","last_fullsized":"2024-03-06T07:23:39.893759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913774+0000","last_change":"2024-03-06T07:23:39.911859+0000","last_active":"2024-03-06T07:23:39.913774+0000","last_peered":"2024-03-06T07:23:39.913774+0000","last_clean":"2024-03-06T07:23:39.913774+0000","last_became_active":"2024-03-06T07:23:39.911529+0000","last_became_peered":"2024-03-06T07:23:39.911529+0000","last_unstale":"2024-03-06T07:23:39.913774+0000","last_undegraded":"2024-03-06T07:23:39.913774+0000","last_fullsized":"2024-03-06T07:23:39.913774+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894652+0000","last_change":"2024-03-06T07:23:39.894652+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894652+0000","last_undegraded":"2024-03-06T07:23:39.894652+0000","last_fullsized":"2024-03-06T07:23:39.894652+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.911862+0000","last_change":"2024-03-06T07:23:39.911223+0000","last_active":"2024-03-06T07:23:39.911862+0000","last_peered":"2024-03-06T07:23:39.911862+0000","last_clean":"2024-03-06T07:23:39.911862+0000","last_became_active":"2024-03-06T07:23:39.910935+0000","last_became_peered":"2024-03-06T07:23:39.910935+0000","last_unstale":"2024-03-06T07:23:39.911862+0000","last_undegraded":"2024-03-06T07:23:39.911862+0000","last_fullsized":"2024-03-06T07:23:39.911862+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:46.994267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897546+0000","last_change":"2024-03-06T07:23:39.897546+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897546+0000","last_undegraded":"2024-03-06T07:23:39.897546+0000","last_fullsized":"2024-03-06T07:23:39.897546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.905864+0000","last_change":"2024-03-06T07:23:39.903773+0000","last_active":"2024-03-06T07:23:39.905864+0000","last_peered":"2024-03-06T07:23:39.905864+0000","last_clean":"2024-03-06T07:23:39.905864+0000","last_became_active":"2024-03-06T07:23:39.903494+0000","last_became_peered":"2024-03-06T07:23:39.903494+0000","last_unstale":"2024-03-06T07:23:39.905864+0000","last_undegraded":"2024-03-06T07:23:39.905864+0000","last_fullsized":"2024-03-06T07:23:39.905864+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:14.991973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898830+0000","last_change":"2024-03-06T07:23:39.898830+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898830+0000","last_undegraded":"2024-03-06T07:23:39.898830+0000","last_fullsized":"2024-03-06T07:23:39.898830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913738+0000","last_change":"2024-03-06T07:23:39.911261+0000","last_active":"2024-03-06T07:23:39.913738+0000","last_peered":"2024-03-06T07:23:39.913738+0000","last_clean":"2024-03-06T07:23:39.913738+0000","last_became_active":"2024-03-06T07:23:39.910917+0000","last_became_peered":"2024-03-06T07:23:39.910917+0000","last_unstale":"2024-03-06T07:23:39.913738+0000","last_undegraded":"2024-03-06T07:23:39.913738+0000","last_fullsized":"2024-03-06T07:23:39.913738+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893681+0000","last_change":"2024-03-06T07:23:39.893681+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893681+0000","last_undegraded":"2024-03-06T07:23:39.893681+0000","last_fullsized":"2024-03-06T07:23:39.893681+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913098+0000","last_change":"2024-03-06T07:23:39.909408+0000","last_active":"2024-03-06T07:23:39.913098+0000","last_peered":"2024-03-06T07:23:39.913098+0000","last_clean":"2024-03-06T07:23:39.913098+0000","last_became_active":"2024-03-06T07:23:39.909128+0000","last_became_peered":"2024-03-06T07:23:39.909128+0000","last_unstale":"2024-03-06T07:23:39.913098+0000","last_undegraded":"2024-03-06T07:23:39.913098+0000","last_fullsized":"2024-03-06T07:23:39.913098+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.087590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895570+0000","last_change":"2024-03-06T07:23:39.895570+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895570+0000","last_undegraded":"2024-03-06T07:23:39.895570+0000","last_fullsized":"2024-03-06T07:23:39.895570+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908963+0000","last_change":"2024-03-06T07:23:39.905798+0000","last_active":"2024-03-06T07:23:39.908963+0000","last_peered":"2024-03-06T07:23:39.908963+0000","last_clean":"2024-03-06T07:23:39.908963+0000","last_became_active":"2024-03-06T07:23:39.905513+0000","last_became_peered":"2024-03-06T07:23:39.905513+0000","last_unstale":"2024-03-06T07:23:39.908963+0000","last_undegraded":"2024-03-06T07:23:39.908963+0000","last_fullsized":"2024-03-06T07:23:39.908963+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892814+0000","last_change":"2024-03-06T07:23:39.892814+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892814+0000","last_undegraded":"2024-03-06T07:23:39.892814+0000","last_fullsized":"2024-03-06T07:23:39.892814+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906475+0000","last_change":"2024-03-06T07:23:39.905335+0000","last_active":"2024-03-06T07:23:39.906475+0000","last_peered":"2024-03-06T07:23:39.906475+0000","last_clean":"2024-03-06T07:23:39.906475+0000","last_became_active":"2024-03-06T07:23:39.903169+0000","last_became_peered":"2024-03-06T07:23:39.903169+0000","last_unstale":"2024-03-06T07:23:39.906475+0000","last_undegraded":"2024-03-06T07:23:39.906475+0000","last_fullsized":"2024-03-06T07:23:39.906475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:44.722551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894171+0000","last_change":"2024-03-06T07:23:39.894171+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894171+0000","last_undegraded":"2024-03-06T07:23:39.894171+0000","last_fullsized":"2024-03-06T07:23:39.894171+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910527+0000","last_change":"2024-03-06T07:23:39.908894+0000","last_active":"2024-03-06T07:23:39.910527+0000","last_peered":"2024-03-06T07:23:39.910527+0000","last_clean":"2024-03-06T07:23:39.910527+0000","last_became_active":"2024-03-06T07:23:39.908206+0000","last_became_peered":"2024-03-06T07:23:39.908206+0000","last_unstale":"2024-03-06T07:23:39.910527+0000","last_undegraded":"2024-03-06T07:23:39.910527+0000","last_fullsized":"2024-03-06T07:23:39.910527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895809+0000","last_change":"2024-03-06T07:23:39.895809+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895809+0000","last_undegraded":"2024-03-06T07:23:39.895809+0000","last_fullsized":"2024-03-06T07:23:39.895809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915464+0000","last_change":"2024-03-06T07:23:39.909279+0000","last_active":"2024-03-06T07:23:39.915464+0000","last_peered":"2024-03-06T07:23:39.915464+0000","last_clean":"2024-03-06T07:23:39.915464+0000","last_became_active":"2024-03-06T07:23:39.908955+0000","last_became_peered":"2024-03-06T07:23:39.908955+0000","last_unstale":"2024-03-06T07:23:39.915464+0000","last_undegraded":"2024-03-06T07:23:39.915464+0000","last_fullsized":"2024-03-06T07:23:39.915464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:42.900316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897107+0000","last_change":"2024-03-06T07:23:39.897107+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897107+0000","last_undegraded":"2024-03-06T07:23:39.897107+0000","last_fullsized":"2024-03-06T07:23:39.897107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910530+0000","last_change":"2024-03-06T07:23:39.908492+0000","last_active":"2024-03-06T07:23:39.910530+0000","last_peered":"2024-03-06T07:23:39.910530+0000","last_clean":"2024-03-06T07:23:39.910530+0000","last_became_active":"2024-03-06T07:23:39.908214+0000","last_became_peered":"2024-03-06T07:23:39.908214+0000","last_unstale":"2024-03-06T07:23:39.910530+0000","last_undegraded":"2024-03-06T07:23:39.910530+0000","last_fullsized":"2024-03-06T07:23:39.910530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901075+0000","last_change":"2024-03-06T07:23:39.901075+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901075+0000","last_undegraded":"2024-03-06T07:23:39.901075+0000","last_fullsized":"2024-03-06T07:23:39.901075+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907658+0000","last_change":"2024-03-06T07:23:39.901912+0000","last_active":"2024-03-06T07:23:39.907658+0000","last_peered":"2024-03-06T07:23:39.907658+0000","last_clean":"2024-03-06T07:23:39.907658+0000","last_became_active":"2024-03-06T07:23:39.901713+0000","last_became_peered":"2024-03-06T07:23:39.901713+0000","last_unstale":"2024-03-06T07:23:39.907658+0000","last_undegraded":"2024-03-06T07:23:39.907658+0000","last_fullsized":"2024-03-06T07:23:39.907658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:31.271276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897613+0000","last_change":"2024-03-06T07:23:39.897613+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897613+0000","last_undegraded":"2024-03-06T07:23:39.897613+0000","last_fullsized":"2024-03-06T07:23:39.897613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915433+0000","last_change":"2024-03-06T07:23:39.908817+0000","last_active":"2024-03-06T07:23:39.915433+0000","last_peered":"2024-03-06T07:23:39.915433+0000","last_clean":"2024-03-06T07:23:39.915433+0000","last_became_active":"2024-03-06T07:23:39.908523+0000","last_became_peered":"2024-03-06T07:23:39.908523+0000","last_unstale":"2024-03-06T07:23:39.915433+0000","last_undegraded":"2024-03-06T07:23:39.915433+0000","last_fullsized":"2024-03-06T07:23:39.915433+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:23.516812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892303+0000","last_change":"2024-03-06T07:23:39.892303+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892303+0000","last_undegraded":"2024-03-06T07:23:39.892303+0000","last_fullsized":"2024-03-06T07:23:39.892303+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904611+0000","last_change":"2024-03-06T07:23:39.902253+0000","last_active":"2024-03-06T07:23:39.904611+0000","last_peered":"2024-03-06T07:23:39.904611+0000","last_clean":"2024-03-06T07:23:39.904611+0000","last_became_active":"2024-03-06T07:23:39.901938+0000","last_became_peered":"2024-03-06T07:23:39.901938+0000","last_unstale":"2024-03-06T07:23:39.904611+0000","last_undegraded":"2024-03-06T07:23:39.904611+0000","last_fullsized":"2024-03-06T07:23:39.904611+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:49.309308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894516+0000","last_change":"2024-03-06T07:23:39.894516+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894516+0000","last_undegraded":"2024-03-06T07:23:39.894516+0000","last_fullsized":"2024-03-06T07:23:39.894516+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913255+0000","last_change":"2024-03-06T07:23:39.908459+0000","last_active":"2024-03-06T07:23:39.913255+0000","last_peered":"2024-03-06T07:23:39.913255+0000","last_clean":"2024-03-06T07:23:39.913255+0000","last_became_active":"2024-03-06T07:23:39.908193+0000","last_became_peered":"2024-03-06T07:23:39.908193+0000","last_unstale":"2024-03-06T07:23:39.913255+0000","last_undegraded":"2024-03-06T07:23:39.913255+0000","last_fullsized":"2024-03-06T07:23:39.913255+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.898698+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:23:39.898698+0000","last_peered":"2024-03-06T07:23:39.898698+0000","last_clean":"2024-03-06T07:23:39.898698+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:23:39.898698+0000","last_undegraded":"2024-03-06T07:23:39.898698+0000","last_fullsized":"2024-03-06T07:23:39.898698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:44.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-03-06T07:23:44.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896350+0000', 'last_change': '2024-03-06T07:23:39.896350+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896350+0000', 'last_undegraded': '2024-03-06T07:23:39.896350+0000', 'last_fullsized': '2024-03-06T07:23:39.896350+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.902983+0000', 'last_change': '2024-03-06T07:23:39.902983+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.902983+0000', 'last_undegraded': '2024-03-06T07:23:39.902983+0000', 'last_fullsized': '2024-03-06T07:23:39.902983+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895197+0000', 'last_change': '2024-03-06T07:23:39.895197+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895197+0000', 'last_undegraded': '2024-03-06T07:23:39.895197+0000', 'last_fullsized': '2024-03-06T07:23:39.895197+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.898515+0000', 'last_change': '2024-03-06T07:23:39.898515+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.898515+0000', 'last_undegraded': '2024-03-06T07:23:39.898515+0000', 'last_fullsized': '2024-03-06T07:23:39.898515+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.897220+0000', 'last_change': '2024-03-06T07:23:39.897220+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.897220+0000', 'last_undegraded': '2024-03-06T07:23:39.897220+0000', 'last_fullsized': '2024-03-06T07:23:39.897220+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.899153+0000', 'last_change': '2024-03-06T07:23:39.899153+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.899153+0000', 'last_undegraded': '2024-03-06T07:23:39.899153+0000', 'last_fullsized': '2024-03-06T07:23:39.899153+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896635+0000', 'last_change': '2024-03-06T07:23:39.896635+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896635+0000', 'last_undegraded': '2024-03-06T07:23:39.896635+0000', 'last_fullsized': '2024-03-06T07:23:39.896635+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895129+0000', 'last_change': '2024-03-06T07:23:39.895129+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895129+0000', 'last_undegraded': '2024-03-06T07:23:39.895129+0000', 'last_fullsized': '2024-03-06T07:23:39.895129+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.900467+0000', 'last_change': '2024-03-06T07:23:39.900467+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.900467+0000', 'last_undegraded': '2024-03-06T07:23:39.900467+0000', 'last_fullsized': '2024-03-06T07:23:39.900467+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895165+0000', 'last_change': '2024-03-06T07:23:39.895165+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895165+0000', 'last_undegraded': '2024-03-06T07:23:39.895165+0000', 'last_fullsized': '2024-03-06T07:23:39.895165+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894678+0000', 'last_change': '2024-03-06T07:23:39.894678+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894678+0000', 'last_undegraded': '2024-03-06T07:23:39.894678+0000', 'last_fullsized': '2024-03-06T07:23:39.894678+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-03-06T07:23:44.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.902759+0000', 'last_change': '2024-03-06T07:23:39.902759+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.902759+0000', 'last_undegraded': '2024-03-06T07:23:39.902759+0000', 'last_fullsized': '2024-03-06T07:23:39.902759+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.900672+0000', 'last_change': '2024-03-06T07:23:39.900672+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.900672+0000', 'last_undegraded': '2024-03-06T07:23:39.900672+0000', 'last_fullsized': '2024-03-06T07:23:39.900672+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.899316+0000', 'last_change': '2024-03-06T07:23:39.899316+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.899316+0000', 'last_undegraded': '2024-03-06T07:23:39.899316+0000', 'last_fullsized': '2024-03-06T07:23:39.899316+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.897463+0000', 'last_change': '2024-03-06T07:23:39.897463+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.897463+0000', 'last_undegraded': '2024-03-06T07:23:39.897463+0000', 'last_fullsized': '2024-03-06T07:23:39.897463+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893246+0000', 'last_change': '2024-03-06T07:23:39.893246+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893246+0000', 'last_undegraded': '2024-03-06T07:23:39.893246+0000', 'last_fullsized': '2024-03-06T07:23:39.893246+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894229+0000', 'last_change': '2024-03-06T07:23:39.894229+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894229+0000', 'last_undegraded': '2024-03-06T07:23:39.894229+0000', 'last_fullsized': '2024-03-06T07:23:39.894229+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2024-03-06T07:23:44.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896126+0000', 'last_change': '2024-03-06T07:23:39.896126+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896126+0000', 'last_undegraded': '2024-03-06T07:23:39.896126+0000', 'last_fullsized': '2024-03-06T07:23:39.896126+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894221+0000', 'last_change': '2024-03-06T07:23:39.894221+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894221+0000', 'last_undegraded': '2024-03-06T07:23:39.894221+0000', 'last_fullsized': '2024-03-06T07:23:39.894221+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.900319+0000', 'last_change': '2024-03-06T07:23:39.900319+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.900319+0000', 'last_undegraded': '2024-03-06T07:23:39.900319+0000', 'last_fullsized': '2024-03-06T07:23:39.900319+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.901518+0000', 'last_change': '2024-03-06T07:23:39.901518+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.901518+0000', 'last_undegraded': '2024-03-06T07:23:39.901518+0000', 'last_fullsized': '2024-03-06T07:23:39.901518+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896014+0000', 'last_change': '2024-03-06T07:23:39.896014+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896014+0000', 'last_undegraded': '2024-03-06T07:23:39.896014+0000', 'last_fullsized': '2024-03-06T07:23:39.896014+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895095+0000', 'last_change': '2024-03-06T07:23:39.895095+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895095+0000', 'last_undegraded': '2024-03-06T07:23:39.895095+0000', 'last_fullsized': '2024-03-06T07:23:39.895095+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.892690+0000', 'last_change': '2024-03-06T07:23:39.892690+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.892690+0000', 'last_undegraded': '2024-03-06T07:23:39.892690+0000', 'last_fullsized': '2024-03-06T07:23:39.892690+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893762+0000', 'last_change': '2024-03-06T07:23:39.893762+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893762+0000', 'last_undegraded': '2024-03-06T07:23:39.893762+0000', 'last_fullsized': '2024-03-06T07:23:39.893762+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.897975+0000', 'last_change': '2024-03-06T07:23:39.897975+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.897975+0000', 'last_undegraded': '2024-03-06T07:23:39.897975+0000', 'last_fullsized': '2024-03-06T07:23:39.897975+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893063+0000', 'last_change': '2024-03-06T07:23:39.893063+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893063+0000', 'last_undegraded': '2024-03-06T07:23:39.893063+0000', 'last_fullsized': '2024-03-06T07:23:39.893063+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.898452+0000', 'last_change': '2024-03-06T07:23:39.898452+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.898452+0000', 'last_undegraded': '2024-03-06T07:23:39.898452+0000', 'last_fullsized': '2024-03-06T07:23:39.898452+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:44.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893292+0000', 'last_change': '2024-03-06T07:23:39.893292+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893292+0000', 'last_undegraded': '2024-03-06T07:23:39.893292+0000', 'last_fullsized': '2024-03-06T07:23:39.893292+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893704+0000', 'last_change': '2024-03-06T07:23:39.893704+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893704+0000', 'last_undegraded': '2024-03-06T07:23:39.893704+0000', 'last_fullsized': '2024-03-06T07:23:39.893704+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896976+0000', 'last_change': '2024-03-06T07:23:39.896976+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896976+0000', 'last_undegraded': '2024-03-06T07:23:39.896976+0000', 'last_fullsized': '2024-03-06T07:23:39.896976+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894325+0000', 'last_change': '2024-03-06T07:23:39.894325+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894325+0000', 'last_undegraded': '2024-03-06T07:23:39.894325+0000', 'last_fullsized': '2024-03-06T07:23:39.894325+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.898773+0000', 'last_change': '2024-03-06T07:23:39.898773+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.898773+0000', 'last_undegraded': '2024-03-06T07:23:39.898773+0000', 'last_fullsized': '2024-03-06T07:23:39.898773+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:23:44.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893338+0000', 'last_change': '2024-03-06T07:23:39.893338+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893338+0000', 'last_undegraded': '2024-03-06T07:23:39.893338+0000', 'last_fullsized': '2024-03-06T07:23:39.893338+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.898062+0000', 'last_change': '2024-03-06T07:23:39.898062+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.898062+0000', 'last_undegraded': '2024-03-06T07:23:39.898062+0000', 'last_fullsized': '2024-03-06T07:23:39.898062+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.904164+0000', 'last_change': '2024-03-06T07:23:39.904164+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.904164+0000', 'last_undegraded': '2024-03-06T07:23:39.904164+0000', 'last_fullsized': '2024-03-06T07:23:39.904164+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893272+0000', 'last_change': '2024-03-06T07:23:39.893272+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893272+0000', 'last_undegraded': '2024-03-06T07:23:39.893272+0000', 'last_fullsized': '2024-03-06T07:23:39.893272+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.899907+0000', 'last_change': '2024-03-06T07:23:39.899907+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.899907+0000', 'last_undegraded': '2024-03-06T07:23:39.899907+0000', 'last_fullsized': '2024-03-06T07:23:39.899907+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894696+0000', 'last_change': '2024-03-06T07:23:39.894696+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894696+0000', 'last_undegraded': '2024-03-06T07:23:39.894696+0000', 'last_fullsized': '2024-03-06T07:23:39.894696+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895537+0000', 'last_change': '2024-03-06T07:23:39.895537+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895537+0000', 'last_undegraded': '2024-03-06T07:23:39.895537+0000', 'last_fullsized': '2024-03-06T07:23:39.895537+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895696+0000', 'last_change': '2024-03-06T07:23:39.895696+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895696+0000', 'last_undegraded': '2024-03-06T07:23:39.895696+0000', 'last_fullsized': '2024-03-06T07:23:39.895696+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895950+0000', 'last_change': '2024-03-06T07:23:39.895950+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895950+0000', 'last_undegraded': '2024-03-06T07:23:39.895950+0000', 'last_fullsized': '2024-03-06T07:23:39.895950+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.898603+0000', 'last_change': '2024-03-06T07:23:39.898603+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.898603+0000', 'last_undegraded': '2024-03-06T07:23:39.898603+0000', 'last_fullsized': '2024-03-06T07:23:39.898603+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896306+0000', 'last_change': '2024-03-06T07:23:39.896306+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896306+0000', 'last_undegraded': '2024-03-06T07:23:39.896306+0000', 'last_fullsized': '2024-03-06T07:23:39.896306+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:23:44.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.892802+0000', 'last_change': '2024-03-06T07:23:39.892802+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.892802+0000', 'last_undegraded': '2024-03-06T07:23:39.892802+0000', 'last_fullsized': '2024-03-06T07:23:39.892802+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896114+0000', 'last_change': '2024-03-06T07:23:39.896114+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896114+0000', 'last_undegraded': '2024-03-06T07:23:39.896114+0000', 'last_fullsized': '2024-03-06T07:23:39.896114+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893960+0000', 'last_change': '2024-03-06T07:23:39.893960+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893960+0000', 'last_undegraded': '2024-03-06T07:23:39.893960+0000', 'last_fullsized': '2024-03-06T07:23:39.893960+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896222+0000', 'last_change': '2024-03-06T07:23:39.896222+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896222+0000', 'last_undegraded': '2024-03-06T07:23:39.896222+0000', 'last_fullsized': '2024-03-06T07:23:39.896222+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894330+0000', 'last_change': '2024-03-06T07:23:39.894330+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894330+0000', 'last_undegraded': '2024-03-06T07:23:39.894330+0000', 'last_fullsized': '2024-03-06T07:23:39.894330+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:23:44.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.896241+0000', 'last_change': '2024-03-06T07:23:39.896241+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.896241+0000', 'last_undegraded': '2024-03-06T07:23:39.896241+0000', 'last_fullsized': '2024-03-06T07:23:39.896241+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893759+0000', 'last_change': '2024-03-06T07:23:39.893759+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893759+0000', 'last_undegraded': '2024-03-06T07:23:39.893759+0000', 'last_fullsized': '2024-03-06T07:23:39.893759+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894652+0000', 'last_change': '2024-03-06T07:23:39.894652+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894652+0000', 'last_undegraded': '2024-03-06T07:23:39.894652+0000', 'last_fullsized': '2024-03-06T07:23:39.894652+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.897546+0000', 'last_change': '2024-03-06T07:23:39.897546+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.897546+0000', 'last_undegraded': '2024-03-06T07:23:39.897546+0000', 'last_fullsized': '2024-03-06T07:23:39.897546+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.898830+0000', 'last_change': '2024-03-06T07:23:39.898830+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.898830+0000', 'last_undegraded': '2024-03-06T07:23:39.898830+0000', 'last_fullsized': '2024-03-06T07:23:39.898830+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.893681+0000', 'last_change': '2024-03-06T07:23:39.893681+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.893681+0000', 'last_undegraded': '2024-03-06T07:23:39.893681+0000', 'last_fullsized': '2024-03-06T07:23:39.893681+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:44.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895570+0000', 'last_change': '2024-03-06T07:23:39.895570+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895570+0000', 'last_undegraded': '2024-03-06T07:23:39.895570+0000', 'last_fullsized': '2024-03-06T07:23:39.895570+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.892814+0000', 'last_change': '2024-03-06T07:23:39.892814+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.892814+0000', 'last_undegraded': '2024-03-06T07:23:39.892814+0000', 'last_fullsized': '2024-03-06T07:23:39.892814+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894171+0000', 'last_change': '2024-03-06T07:23:39.894171+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894171+0000', 'last_undegraded': '2024-03-06T07:23:39.894171+0000', 'last_fullsized': '2024-03-06T07:23:39.894171+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.895809+0000', 'last_change': '2024-03-06T07:23:39.895809+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.895809+0000', 'last_undegraded': '2024-03-06T07:23:39.895809+0000', 'last_fullsized': '2024-03-06T07:23:39.895809+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.897107+0000', 'last_change': '2024-03-06T07:23:39.897107+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.897107+0000', 'last_undegraded': '2024-03-06T07:23:39.897107+0000', 'last_fullsized': '2024-03-06T07:23:39.897107+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:23:44.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.901075+0000', 'last_change': '2024-03-06T07:23:39.901075+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.901075+0000', 'last_undegraded': '2024-03-06T07:23:39.901075+0000', 'last_fullsized': '2024-03-06T07:23:39.901075+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:44.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:23:44.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.897613+0000', 'last_change': '2024-03-06T07:23:39.897613+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.897613+0000', 'last_undegraded': '2024-03-06T07:23:39.897613+0000', 'last_fullsized': '2024-03-06T07:23:39.897613+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:44.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:23:44.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.892303+0000', 'last_change': '2024-03-06T07:23:39.892303+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.892303+0000', 'last_undegraded': '2024-03-06T07:23:39.892303+0000', 'last_fullsized': '2024-03-06T07:23:39.892303+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:44.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:23:44.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:39.894516+0000', 'last_change': '2024-03-06T07:23:39.894516+0000', 'last_active': '2024-03-06T07:23:39.880214+0000', 'last_peered': '2024-03-06T07:23:39.880214+0000', 'last_clean': '2024-03-06T07:23:39.880214+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:39.894516+0000', 'last_undegraded': '2024-03-06T07:23:39.894516+0000', 'last_fullsized': '2024-03-06T07:23:39.894516+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:39.880214+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:44.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:23:44.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.676+0000 7f6fb6eaf640 1 Processor -- start 2024-03-06T07:23:44.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.676+0000 7f6fb6eaf640 1 -- start start 2024-03-06T07:23:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.676+0000 7f6fb6eaf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6fb0071e00 0x7f6fb00cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.676+0000 7f6fb6eaf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb0072490 con 0x7f6fb015bbf0 2024-03-06T07:23:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6fb6eaf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb00d0090 con 0x7f6fb00cd7c0 2024-03-06T07:23:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6fb6eaf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb0072250 con 0x7f6fb0071e00 2024-03-06T07:23:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6fb4c4b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6fb015bbf0 0x7f6fb015bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41266/0 (socket says 172.21.15.122:41266) 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6fb4c4b640 1 -- 172.21.15.122:0/2076692819 learned_addr learned my addr 172.21.15.122:0/2076692819 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faffff640 1 --2- 172.21.15.122:0/2076692819 >> v2:172.21.15.169:3300/0 conn(0x7f6fb0071e00 0x7f6fb00cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faf7fe640 1 -- 172.21.15.122:0/2076692819 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794288830 0 0) 0x7f6fb0072490 con 0x7f6fb015bbf0 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faf7fe640 1 -- 172.21.15.122:0/2076692819 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa0003520 con 0x7f6fb015bbf0 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faffff640 1 -- 172.21.15.122:0/2076692819 >> v1:172.21.15.169:6789/0 conn(0x7f6fb00cd7c0 legacy=0x7f6fb00cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faf7fe640 1 -- 172.21.15.122:0/2076692819 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419974182 0 0) 0x7f6fb00d0090 con 0x7f6fb00cd7c0 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faffff640 1 -- 172.21.15.122:0/2076692819 >> v1:172.21.15.122:6789/0 conn(0x7f6fb015bbf0 legacy=0x7f6fb015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.677+0000 7f6faffff640 1 -- 172.21.15.122:0/2076692819 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00d0370 con 0x7f6fb0071e00 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.678+0000 7f6faffff640 1 --2- 172.21.15.122:0/2076692819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb00cebc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0097a0 tx=0x7f6f9c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cebc46739a7cdc6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.678+0000 7f6faf7fe640 1 -- 172.21.15.122:0/2076692819 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c034070 con 0x7f6fb0071e00 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.678+0000 7f6faf7fe640 1 -- 172.21.15.122:0/2076692819 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c030610 con 0x7f6fb0071e00 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.678+0000 7f6faf7fe640 1 -- 172.21.15.122:0/2076692819 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c0309e0 con 0x7f6fb0071e00 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.678+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/2076692819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 msgr2=0x7f6fb00cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.678+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/2076692819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb00cebc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0097a0 tx=0x7f6f9c02b960 comp rx=0 tx=0).stop 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.679+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/2076692819 shutdown_connections 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.679+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/2076692819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb00cebc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.679+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/2076692819 >> 172.21.15.122:0/2076692819 conn(0x7f6fb00b1c80 msgr2=0x7f6fb00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.679+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/2076692819 shutdown_connections 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.679+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/2076692819 wait complete. 2024-03-06T07:23:44.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.680+0000 7f6fb6eaf640 1 Processor -- start 2024-03-06T07:23:44.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.680+0000 7f6fb6eaf640 1 -- start start 2024-03-06T07:23:44.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb01497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6fb00cd7c0 0x7f6fb006e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 0x7f6fb0156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb015c6e0 con 0x7f6fb015bbf0 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00cf100 con 0x7f6fb0071e00 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00b3680 con 0x7f6fb00cd7c0 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 0x7f6fb0156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 0x7f6fb0156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47114/0 (socket says 172.21.15.122:47114) 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 -- 172.21.15.122:0/47528787 learned_addr learned my addr 172.21.15.122:0/47528787 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6faffff640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6fb00cd7c0 0x7f6fb006e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb4c4b640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb01497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 -- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6fb00cd7c0 msgr2=0x7f6fb006e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6fb00cd7c0 0x7f6fb006e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 -- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 msgr2=0x7f6fb01497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb01497a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 -- 172.21.15.122:0/47528787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb006cc90 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb544c640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 0x7f6fb0156c30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800c760 tx=0x7f6fa800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003da0 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8005070 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8005460 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb006cf70 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.681+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb006d530 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.682+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb004f330 con 0x7f6fb015bbf0 2024-03-06T07:23:44.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.683+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fa8005600 con 0x7f6fb015bbf0 2024-03-06T07:23:44.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.683+0000 7f6fad7fa640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f9406b4d0 0x7f6f9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.683+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fa808a050 con 0x7f6fb015bbf0 2024-03-06T07:23:44.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.683+0000 7f6fb4c4b640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f9406b4d0 0x7f6f9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.684+0000 7f6fb4c4b640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f9406b4d0 0x7f6f9406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4005880 tx=0x7f6fa4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.688+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fa8058880 con 0x7f6fb015bbf0 2024-03-06T07:23:44.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.926+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6fb00709d0 con 0x7f6fb015bbf0 2024-03-06T07:23:44.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.927+0000 7f6fad7fa640 1 -- 172.21.15.122:0/47528787 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7f6fa805c920 con 0x7f6fb015bbf0 2024-03-06T07:23:44.929 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:44.929 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":135,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":260599808,"bytes_avail":772833513472,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":21,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":5769}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:22:47.341990+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f9406b4d0 msgr2=0x7f6f9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f9406b4d0 0x7f6f9406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4005880 tx=0x7f6fa4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 msgr2=0x7f6fb0156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 0x7f6fb0156c30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800c760 tx=0x7f6fa800cc20 comp rx=0 tx=0).stop 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 shutdown_connections 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f9406b4d0 0x7f6f9406d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb015bbf0 0x7f6fb0156c30 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6fb00cd7c0 0x7f6fb006e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 --2- 172.21.15.122:0/47528787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fb0071e00 0x7f6fb01497a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 >> 172.21.15.122:0/47528787 conn(0x7f6fb00b1c80 msgr2=0x7f6fb0159f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 shutdown_connections 2024-03-06T07:23:44.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:44.929+0000 7f6fb6eaf640 1 -- 172.21.15.122:0/47528787 wait complete. 2024-03-06T07:23:44.942 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:23:44.942 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:45.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.039+0000 7fe13ab3e640 1 Processor -- start 2024-03-06T07:23:45.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.039+0000 7fe13ab3e640 1 -- start start 2024-03-06T07:23:45.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe13ab3e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe13415bf30 0x7fe13415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe13ab3e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13415f580 con 0x7fe13413dc10 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe13ab3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13415b450 con 0x7fe1340b3060 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe13ab3e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1340cf3f0 con 0x7fe13415bf30 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe1388da640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe13413dc10 0x7fe1340b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41280/0 (socket says 172.21.15.122:41280) 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe1388da640 1 -- 172.21.15.122:0/196007252 learned_addr learned my addr 172.21.15.122:0/196007252 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe1390db640 1 --2- 172.21.15.122:0/196007252 >> v2:172.21.15.169:3300/0 conn(0x7fe13415bf30 0x7fe13415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe1337fe640 1 -- 172.21.15.122:0/196007252 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210687957 0 0) 0x7fe13415f580 con 0x7fe13413dc10 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.040+0000 7fe1337fe640 1 -- 172.21.15.122:0/196007252 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe11c003520 con 0x7fe13413dc10 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.041+0000 7fe1390db640 1 -- 172.21.15.122:0/196007252 >> v1:172.21.15.169:6789/0 conn(0x7fe1340b3060 legacy=0x7fe13415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.041+0000 7fe1390db640 1 -- 172.21.15.122:0/196007252 >> v1:172.21.15.122:6789/0 conn(0x7fe13413dc10 legacy=0x7fe1340b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.041+0000 7fe1337fe640 1 -- 172.21.15.122:0/196007252 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275666520 0 0) 0x7fe13415b450 con 0x7fe1340b3060 2024-03-06T07:23:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.041+0000 7fe1390db640 1 -- 172.21.15.122:0/196007252 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13415b730 con 0x7fe13415bf30 2024-03-06T07:23:45.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.041+0000 7fe1390db640 1 --2- 172.21.15.122:0/196007252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13415bf30 0x7fe13415ef60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009d90 tx=0x7fe12c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fef23439a2e6d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.041+0000 7fe1337fe640 1 -- 172.21.15.122:0/196007252 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c039070 con 0x7fe13415bf30 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe1337fe640 1 -- 172.21.15.122:0/196007252 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12c02cc60 con 0x7fe13415bf30 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe1337fe640 1 -- 172.21.15.122:0/196007252 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c034660 con 0x7fe13415bf30 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 -- 172.21.15.122:0/196007252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13415bf30 msgr2=0x7fe13415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/196007252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13415bf30 0x7fe13415ef60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009d90 tx=0x7fe12c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 -- 172.21.15.122:0/196007252 shutdown_connections 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/196007252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13415bf30 0x7fe13415ef60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 -- 172.21.15.122:0/196007252 >> 172.21.15.122:0/196007252 conn(0x7fe1340cf8a0 msgr2=0x7fe1340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 -- 172.21.15.122:0/196007252 shutdown_connections 2024-03-06T07:23:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.042+0000 7fe13ab3e640 1 -- 172.21.15.122:0/196007252 wait complete. 2024-03-06T07:23:45.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.043+0000 7fe13ab3e640 1 Processor -- start 2024-03-06T07:23:45.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.043+0000 7fe13ab3e640 1 -- start start 2024-03-06T07:23:45.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe13ab3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1340b3060 0x7fe134137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe13ab3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13413dc10 0x7fe134154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe13ab3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 0x7fe1341546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe13ab3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340718e0 con 0x7fe13415bf30 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe13ab3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13415b210 con 0x7fe13413dc10 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe13ab3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe134071a60 con 0x7fe1340b3060 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 0x7fe1341546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 0x7fe1341546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47122/0 (socket says 172.21.15.122:47122) 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 -- 172.21.15.122:0/620219969 learned_addr learned my addr 172.21.15.122:0/620219969 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1388da640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1340b3060 0x7fe134137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe133fff640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13413dc10 0x7fe134154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 -- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1340b3060 msgr2=0x7fe134137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1340b3060 0x7fe134137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 -- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13413dc10 msgr2=0x7fe134154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.044+0000 7fe1390db640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13413dc10 0x7fe134154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe1390db640 1 -- 172.21.15.122:0/620219969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe134138320 con 0x7fe13415bf30 2024-03-06T07:23:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe1390db640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 0x7fe1341546c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009d90 tx=0x7fe12c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c039050 con 0x7fe13415bf30 2024-03-06T07:23:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12c03a040 con 0x7fe13415bf30 2024-03-06T07:23:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c009450 con 0x7fe13415bf30 2024-03-06T07:23:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe134155140 con 0x7fe13415bf30 2024-03-06T07:23:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.045+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe134153210 con 0x7fe13415bf30 2024-03-06T07:23:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.046+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe134140be0 con 0x7fe13415bf30 2024-03-06T07:23:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.046+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe12c047020 con 0x7fe13415bf30 2024-03-06T07:23:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.047+0000 7fe1317fa640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe12006b4d0 0x7fe12006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.047+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe12c0ac320 con 0x7fe13415bf30 2024-03-06T07:23:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.047+0000 7fe1388da640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe12006b4d0 0x7fe12006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.047+0000 7fe1388da640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe12006b4d0 0x7fe12006d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe124005880 tx=0x7fe124009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.051+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe12c04e020 con 0x7fe13415bf30 2024-03-06T07:23:45.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.220+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe13406eb00 con 0x7fe12006b4d0 2024-03-06T07:23:45.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.225+0000 7fe1317fa640 1 -- 172.21.15.122:0/620219969 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7fe13406eb00 con 0x7fe12006b4d0 2024-03-06T07:23:45.230 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:45.231 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe12006b4d0 msgr2=0x7fe12006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe12006b4d0 0x7fe12006d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe124005880 tx=0x7fe124009040 comp rx=0 tx=0).stop 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 msgr2=0x7fe1341546c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 0x7fe1341546c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009d90 tx=0x7fe12c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:45.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 shutdown_connections 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe12006b4d0 0x7fe12006d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe13415bf30 0x7fe1341546c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe13413dc10 0x7fe134154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 --2- 172.21.15.122:0/620219969 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1340b3060 0x7fe134137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 >> 172.21.15.122:0/620219969 conn(0x7fe1340cf8a0 msgr2=0x7fe1340b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 shutdown_connections 2024-03-06T07:23:45.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:45.228+0000 7fe13ab3e640 1 -- 172.21.15.122:0/620219969 wait complete. 2024-03-06T07:23:45.241 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-03-06T07:23:43.352031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254492,"kb_used_data":5036,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720228,"statfs":{"total":773094113280,"available":772833513472,"internally_reserved":0,"allocated":5156864,"data_stored":2765721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471774"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896350+0000","last_change":"2024-03-06T07:23:39.896350+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896350+0000","last_undegraded":"2024-03-06T07:23:39.896350+0000","last_fullsized":"2024-03-06T07:23:39.896350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902983+0000","last_change":"2024-03-06T07:23:39.902983+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902983+0000","last_undegraded":"2024-03-06T07:23:39.902983+0000","last_fullsized":"2024-03-06T07:23:39.902983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895197+0000","last_change":"2024-03-06T07:23:39.895197+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895197+0000","last_undegraded":"2024-03-06T07:23:39.895197+0000","last_fullsized":"2024-03-06T07:23:39.895197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898515+0000","last_change":"2024-03-06T07:23:39.898515+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898515+0000","last_undegraded":"2024-03-06T07:23:39.898515+0000","last_fullsized":"2024-03-06T07:23:39.898515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897220+0000","last_change":"2024-03-06T07:23:39.897220+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897220+0000","last_undegraded":"2024-03-06T07:23:39.897220+0000","last_fullsized":"2024-03-06T07:23:39.897220+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899153+0000","last_change":"2024-03-06T07:23:39.899153+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899153+0000","last_undegraded":"2024-03-06T07:23:39.899153+0000","last_fullsized":"2024-03-06T07:23:39.899153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896635+0000","last_change":"2024-03-06T07:23:39.896635+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896635+0000","last_undegraded":"2024-03-06T07:23:39.896635+0000","last_fullsized":"2024-03-06T07:23:39.896635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895129+0000","last_change":"2024-03-06T07:23:39.895129+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895129+0000","last_undegraded":"2024-03-06T07:23:39.895129+0000","last_fullsized":"2024-03-06T07:23:39.895129+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900467+0000","last_change":"2024-03-06T07:23:39.900467+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900467+0000","last_undegraded":"2024-03-06T07:23:39.900467+0000","last_fullsized":"2024-03-06T07:23:39.900467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895165+0000","last_change":"2024-03-06T07:23:39.895165+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895165+0000","last_undegraded":"2024-03-06T07:23:39.895165+0000","last_fullsized":"2024-03-06T07:23:39.895165+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894678+0000","last_change":"2024-03-06T07:23:39.894678+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894678+0000","last_undegraded":"2024-03-06T07:23:39.894678+0000","last_fullsized":"2024-03-06T07:23:39.894678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.902759+0000","last_change":"2024-03-06T07:23:39.902759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.902759+0000","last_undegraded":"2024-03-06T07:23:39.902759+0000","last_fullsized":"2024-03-06T07:23:39.902759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900672+0000","last_change":"2024-03-06T07:23:39.900672+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900672+0000","last_undegraded":"2024-03-06T07:23:39.900672+0000","last_fullsized":"2024-03-06T07:23:39.900672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899316+0000","last_change":"2024-03-06T07:23:39.899316+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899316+0000","last_undegraded":"2024-03-06T07:23:39.899316+0000","last_fullsized":"2024-03-06T07:23:39.899316+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897463+0000","last_change":"2024-03-06T07:23:39.897463+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897463+0000","last_undegraded":"2024-03-06T07:23:39.897463+0000","last_fullsized":"2024-03-06T07:23:39.897463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893246+0000","last_change":"2024-03-06T07:23:39.893246+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893246+0000","last_undegraded":"2024-03-06T07:23:39.893246+0000","last_fullsized":"2024-03-06T07:23:39.893246+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894229+0000","last_change":"2024-03-06T07:23:39.894229+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894229+0000","last_undegraded":"2024-03-06T07:23:39.894229+0000","last_fullsized":"2024-03-06T07:23:39.894229+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896126+0000","last_change":"2024-03-06T07:23:39.896126+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896126+0000","last_undegraded":"2024-03-06T07:23:39.896126+0000","last_fullsized":"2024-03-06T07:23:39.896126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894221+0000","last_change":"2024-03-06T07:23:39.894221+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894221+0000","last_undegraded":"2024-03-06T07:23:39.894221+0000","last_fullsized":"2024-03-06T07:23:39.894221+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.900319+0000","last_change":"2024-03-06T07:23:39.900319+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.900319+0000","last_undegraded":"2024-03-06T07:23:39.900319+0000","last_fullsized":"2024-03-06T07:23:39.900319+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901518+0000","last_change":"2024-03-06T07:23:39.901518+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901518+0000","last_undegraded":"2024-03-06T07:23:39.901518+0000","last_fullsized":"2024-03-06T07:23:39.901518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896014+0000","last_change":"2024-03-06T07:23:39.896014+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896014+0000","last_undegraded":"2024-03-06T07:23:39.896014+0000","last_fullsized":"2024-03-06T07:23:39.896014+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895095+0000","last_change":"2024-03-06T07:23:39.895095+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895095+0000","last_undegraded":"2024-03-06T07:23:39.895095+0000","last_fullsized":"2024-03-06T07:23:39.895095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892690+0000","last_change":"2024-03-06T07:23:39.892690+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892690+0000","last_undegraded":"2024-03-06T07:23:39.892690+0000","last_fullsized":"2024-03-06T07:23:39.892690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893762+0000","last_change":"2024-03-06T07:23:39.893762+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893762+0000","last_undegraded":"2024-03-06T07:23:39.893762+0000","last_fullsized":"2024-03-06T07:23:39.893762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897975+0000","last_change":"2024-03-06T07:23:39.897975+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897975+0000","last_undegraded":"2024-03-06T07:23:39.897975+0000","last_fullsized":"2024-03-06T07:23:39.897975+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893063+0000","last_change":"2024-03-06T07:23:39.893063+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893063+0000","last_undegraded":"2024-03-06T07:23:39.893063+0000","last_fullsized":"2024-03-06T07:23:39.893063+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898452+0000","last_change":"2024-03-06T07:23:39.898452+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898452+0000","last_undegraded":"2024-03-06T07:23:39.898452+0000","last_fullsized":"2024-03-06T07:23:39.898452+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893292+0000","last_change":"2024-03-06T07:23:39.893292+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893292+0000","last_undegraded":"2024-03-06T07:23:39.893292+0000","last_fullsized":"2024-03-06T07:23:39.893292+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893704+0000","last_change":"2024-03-06T07:23:39.893704+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893704+0000","last_undegraded":"2024-03-06T07:23:39.893704+0000","last_fullsized":"2024-03-06T07:23:39.893704+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896976+0000","last_change":"2024-03-06T07:23:39.896976+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896976+0000","last_undegraded":"2024-03-06T07:23:39.896976+0000","last_fullsized":"2024-03-06T07:23:39.896976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894325+0000","last_change":"2024-03-06T07:23:39.894325+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894325+0000","last_undegraded":"2024-03-06T07:23:39.894325+0000","last_fullsized":"2024-03-06T07:23:39.894325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898773+0000","last_change":"2024-03-06T07:23:39.898773+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898773+0000","last_undegraded":"2024-03-06T07:23:39.898773+0000","last_fullsized":"2024-03-06T07:23:39.898773+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.902762+0000","last_change":"2024-03-06T07:23:39.900097+0000","last_active":"2024-03-06T07:23:39.902762+0000","last_peered":"2024-03-06T07:23:39.902762+0000","last_clean":"2024-03-06T07:23:39.902762+0000","last_became_active":"2024-03-06T07:23:39.899808+0000","last_became_peered":"2024-03-06T07:23:39.899808+0000","last_unstale":"2024-03-06T07:23:39.902762+0000","last_undegraded":"2024-03-06T07:23:39.902762+0000","last_fullsized":"2024-03-06T07:23:39.902762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893338+0000","last_change":"2024-03-06T07:23:39.893338+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893338+0000","last_undegraded":"2024-03-06T07:23:39.893338+0000","last_fullsized":"2024-03-06T07:23:39.893338+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907516+0000","last_change":"2024-03-06T07:23:39.899915+0000","last_active":"2024-03-06T07:23:39.907516+0000","last_peered":"2024-03-06T07:23:39.907516+0000","last_clean":"2024-03-06T07:23:39.907516+0000","last_became_active":"2024-03-06T07:23:39.899555+0000","last_became_peered":"2024-03-06T07:23:39.899555+0000","last_unstale":"2024-03-06T07:23:39.907516+0000","last_undegraded":"2024-03-06T07:23:39.907516+0000","last_fullsized":"2024-03-06T07:23:39.907516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:45.519711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898062+0000","last_change":"2024-03-06T07:23:39.898062+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898062+0000","last_undegraded":"2024-03-06T07:23:39.898062+0000","last_fullsized":"2024-03-06T07:23:39.898062+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915812+0000","last_change":"2024-03-06T07:23:39.911329+0000","last_active":"2024-03-06T07:23:39.915812+0000","last_peered":"2024-03-06T07:23:39.915812+0000","last_clean":"2024-03-06T07:23:39.915812+0000","last_became_active":"2024-03-06T07:23:39.911011+0000","last_became_peered":"2024-03-06T07:23:39.911011+0000","last_unstale":"2024-03-06T07:23:39.915812+0000","last_undegraded":"2024-03-06T07:23:39.915812+0000","last_fullsized":"2024-03-06T07:23:39.915812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:45.430603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.904164+0000","last_change":"2024-03-06T07:23:39.904164+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.904164+0000","last_undegraded":"2024-03-06T07:23:39.904164+0000","last_fullsized":"2024-03-06T07:23:39.904164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910560+0000","last_change":"2024-03-06T07:23:39.908704+0000","last_active":"2024-03-06T07:23:39.910560+0000","last_peered":"2024-03-06T07:23:39.910560+0000","last_clean":"2024-03-06T07:23:39.910560+0000","last_became_active":"2024-03-06T07:23:39.908289+0000","last_became_peered":"2024-03-06T07:23:39.908289+0000","last_unstale":"2024-03-06T07:23:39.910560+0000","last_undegraded":"2024-03-06T07:23:39.910560+0000","last_fullsized":"2024-03-06T07:23:39.910560+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893272+0000","last_change":"2024-03-06T07:23:39.893272+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893272+0000","last_undegraded":"2024-03-06T07:23:39.893272+0000","last_fullsized":"2024-03-06T07:23:39.893272+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.909418+0000","last_change":"2024-03-06T07:23:39.908607+0000","last_active":"2024-03-06T07:23:39.909418+0000","last_peered":"2024-03-06T07:23:39.909418+0000","last_clean":"2024-03-06T07:23:39.909418+0000","last_became_active":"2024-03-06T07:23:39.908334+0000","last_became_peered":"2024-03-06T07:23:39.908334+0000","last_unstale":"2024-03-06T07:23:39.909418+0000","last_undegraded":"2024-03-06T07:23:39.909418+0000","last_fullsized":"2024-03-06T07:23:39.909418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:06.382376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.899907+0000","last_change":"2024-03-06T07:23:39.899907+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.899907+0000","last_undegraded":"2024-03-06T07:23:39.899907+0000","last_fullsized":"2024-03-06T07:23:39.899907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913163+0000","last_change":"2024-03-06T07:23:39.908741+0000","last_active":"2024-03-06T07:23:39.913163+0000","last_peered":"2024-03-06T07:23:39.913163+0000","last_clean":"2024-03-06T07:23:39.913163+0000","last_became_active":"2024-03-06T07:23:39.908470+0000","last_became_peered":"2024-03-06T07:23:39.908470+0000","last_unstale":"2024-03-06T07:23:39.913163+0000","last_undegraded":"2024-03-06T07:23:39.913163+0000","last_fullsized":"2024-03-06T07:23:39.913163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:07.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894696+0000","last_change":"2024-03-06T07:23:39.894696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894696+0000","last_undegraded":"2024-03-06T07:23:39.894696+0000","last_fullsized":"2024-03-06T07:23:39.894696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913247+0000","last_change":"2024-03-06T07:23:39.909770+0000","last_active":"2024-03-06T07:23:39.913247+0000","last_peered":"2024-03-06T07:23:39.913247+0000","last_clean":"2024-03-06T07:23:39.913247+0000","last_became_active":"2024-03-06T07:23:39.909494+0000","last_became_peered":"2024-03-06T07:23:39.909494+0000","last_unstale":"2024-03-06T07:23:39.913247+0000","last_undegraded":"2024-03-06T07:23:39.913247+0000","last_fullsized":"2024-03-06T07:23:39.913247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895537+0000","last_change":"2024-03-06T07:23:39.895537+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895537+0000","last_undegraded":"2024-03-06T07:23:39.895537+0000","last_fullsized":"2024-03-06T07:23:39.895537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910678+0000","last_change":"2024-03-06T07:23:39.906386+0000","last_active":"2024-03-06T07:23:39.910678+0000","last_peered":"2024-03-06T07:23:39.910678+0000","last_clean":"2024-03-06T07:23:39.910678+0000","last_became_active":"2024-03-06T07:23:39.905390+0000","last_became_peered":"2024-03-06T07:23:39.905390+0000","last_unstale":"2024-03-06T07:23:39.910678+0000","last_undegraded":"2024-03-06T07:23:39.910678+0000","last_fullsized":"2024-03-06T07:23:39.910678+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895696+0000","last_change":"2024-03-06T07:23:39.895696+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895696+0000","last_undegraded":"2024-03-06T07:23:39.895696+0000","last_fullsized":"2024-03-06T07:23:39.895696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907953+0000","last_change":"2024-03-06T07:23:39.901277+0000","last_active":"2024-03-06T07:23:39.907953+0000","last_peered":"2024-03-06T07:23:39.907953+0000","last_clean":"2024-03-06T07:23:39.907953+0000","last_became_active":"2024-03-06T07:23:39.900996+0000","last_became_peered":"2024-03-06T07:23:39.900996+0000","last_unstale":"2024-03-06T07:23:39.907953+0000","last_undegraded":"2024-03-06T07:23:39.907953+0000","last_fullsized":"2024-03-06T07:23:39.907953+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:05.084281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895950+0000","last_change":"2024-03-06T07:23:39.895950+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895950+0000","last_undegraded":"2024-03-06T07:23:39.895950+0000","last_fullsized":"2024-03-06T07:23:39.895950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910643+0000","last_change":"2024-03-06T07:23:39.904382+0000","last_active":"2024-03-06T07:23:39.910643+0000","last_peered":"2024-03-06T07:23:39.910643+0000","last_clean":"2024-03-06T07:23:39.910643+0000","last_became_active":"2024-03-06T07:23:39.904076+0000","last_became_peered":"2024-03-06T07:23:39.904076+0000","last_unstale":"2024-03-06T07:23:39.910643+0000","last_undegraded":"2024-03-06T07:23:39.910643+0000","last_fullsized":"2024-03-06T07:23:39.910643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:51.296844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898603+0000","last_change":"2024-03-06T07:23:39.898603+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898603+0000","last_undegraded":"2024-03-06T07:23:39.898603+0000","last_fullsized":"2024-03-06T07:23:39.898603+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913122+0000","last_change":"2024-03-06T07:23:39.909133+0000","last_active":"2024-03-06T07:23:39.913122+0000","last_peered":"2024-03-06T07:23:39.913122+0000","last_clean":"2024-03-06T07:23:39.913122+0000","last_became_active":"2024-03-06T07:23:39.908563+0000","last_became_peered":"2024-03-06T07:23:39.908563+0000","last_unstale":"2024-03-06T07:23:39.913122+0000","last_undegraded":"2024-03-06T07:23:39.913122+0000","last_fullsized":"2024-03-06T07:23:39.913122+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:33.174798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896306+0000","last_change":"2024-03-06T07:23:39.896306+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896306+0000","last_undegraded":"2024-03-06T07:23:39.896306+0000","last_fullsized":"2024-03-06T07:23:39.896306+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.900855+0000","last_change":"2024-03-06T07:23:39.896528+0000","last_active":"2024-03-06T07:23:39.900855+0000","last_peered":"2024-03-06T07:23:39.900855+0000","last_clean":"2024-03-06T07:23:39.900855+0000","last_became_active":"2024-03-06T07:23:39.896142+0000","last_became_peered":"2024-03-06T07:23:39.896142+0000","last_unstale":"2024-03-06T07:23:39.900855+0000","last_undegraded":"2024-03-06T07:23:39.900855+0000","last_fullsized":"2024-03-06T07:23:39.900855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:17.233793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892802+0000","last_change":"2024-03-06T07:23:39.892802+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892802+0000","last_undegraded":"2024-03-06T07:23:39.892802+0000","last_fullsized":"2024-03-06T07:23:39.892802+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906489+0000","last_change":"2024-03-06T07:23:39.905467+0000","last_active":"2024-03-06T07:23:39.906489+0000","last_peered":"2024-03-06T07:23:39.906489+0000","last_clean":"2024-03-06T07:23:39.906489+0000","last_became_active":"2024-03-06T07:23:39.905185+0000","last_became_peered":"2024-03-06T07:23:39.905185+0000","last_unstale":"2024-03-06T07:23:39.906489+0000","last_undegraded":"2024-03-06T07:23:39.906489+0000","last_fullsized":"2024-03-06T07:23:39.906489+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:48.343179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896114+0000","last_change":"2024-03-06T07:23:39.896114+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896114+0000","last_undegraded":"2024-03-06T07:23:39.896114+0000","last_fullsized":"2024-03-06T07:23:39.896114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908901+0000","last_change":"2024-03-06T07:23:39.905513+0000","last_active":"2024-03-06T07:23:39.908901+0000","last_peered":"2024-03-06T07:23:39.908901+0000","last_clean":"2024-03-06T07:23:39.908901+0000","last_became_active":"2024-03-06T07:23:39.905121+0000","last_became_peered":"2024-03-06T07:23:39.905121+0000","last_unstale":"2024-03-06T07:23:39.908901+0000","last_undegraded":"2024-03-06T07:23:39.908901+0000","last_fullsized":"2024-03-06T07:23:39.908901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:57.881886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893960+0000","last_change":"2024-03-06T07:23:39.893960+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893960+0000","last_undegraded":"2024-03-06T07:23:39.893960+0000","last_fullsized":"2024-03-06T07:23:39.893960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906161+0000","last_change":"2024-03-06T07:23:39.902543+0000","last_active":"2024-03-06T07:23:39.906161+0000","last_peered":"2024-03-06T07:23:39.906161+0000","last_clean":"2024-03-06T07:23:39.906161+0000","last_became_active":"2024-03-06T07:23:39.902271+0000","last_became_peered":"2024-03-06T07:23:39.902271+0000","last_unstale":"2024-03-06T07:23:39.906161+0000","last_undegraded":"2024-03-06T07:23:39.906161+0000","last_fullsized":"2024-03-06T07:23:39.906161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:42.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896222+0000","last_change":"2024-03-06T07:23:39.896222+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896222+0000","last_undegraded":"2024-03-06T07:23:39.896222+0000","last_fullsized":"2024-03-06T07:23:39.896222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904631+0000","last_change":"2024-03-06T07:23:39.903255+0000","last_active":"2024-03-06T07:23:39.904631+0000","last_peered":"2024-03-06T07:23:39.904631+0000","last_clean":"2024-03-06T07:23:39.904631+0000","last_became_active":"2024-03-06T07:23:39.902984+0000","last_became_peered":"2024-03-06T07:23:39.902984+0000","last_unstale":"2024-03-06T07:23:39.904631+0000","last_undegraded":"2024-03-06T07:23:39.904631+0000","last_fullsized":"2024-03-06T07:23:39.904631+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:06.103875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894330+0000","last_change":"2024-03-06T07:23:39.894330+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894330+0000","last_undegraded":"2024-03-06T07:23:39.894330+0000","last_fullsized":"2024-03-06T07:23:39.894330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913144+0000","last_change":"2024-03-06T07:23:39.906113+0000","last_active":"2024-03-06T07:23:39.913144+0000","last_peered":"2024-03-06T07:23:39.913144+0000","last_clean":"2024-03-06T07:23:39.913144+0000","last_became_active":"2024-03-06T07:23:39.905836+0000","last_became_peered":"2024-03-06T07:23:39.905836+0000","last_unstale":"2024-03-06T07:23:39.913144+0000","last_undegraded":"2024-03-06T07:23:39.913144+0000","last_fullsized":"2024-03-06T07:23:39.913144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:50.974905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.896241+0000","last_change":"2024-03-06T07:23:39.896241+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.896241+0000","last_undegraded":"2024-03-06T07:23:39.896241+0000","last_fullsized":"2024-03-06T07:23:39.896241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908710+0000","last_change":"2024-03-06T07:23:39.904607+0000","last_active":"2024-03-06T07:23:39.908710+0000","last_peered":"2024-03-06T07:23:39.908710+0000","last_clean":"2024-03-06T07:23:39.908710+0000","last_became_active":"2024-03-06T07:23:39.904289+0000","last_became_peered":"2024-03-06T07:23:39.904289+0000","last_unstale":"2024-03-06T07:23:39.908710+0000","last_undegraded":"2024-03-06T07:23:39.908710+0000","last_fullsized":"2024-03-06T07:23:39.908710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:47.230884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893759+0000","last_change":"2024-03-06T07:23:39.893759+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893759+0000","last_undegraded":"2024-03-06T07:23:39.893759+0000","last_fullsized":"2024-03-06T07:23:39.893759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913774+0000","last_change":"2024-03-06T07:23:39.911859+0000","last_active":"2024-03-06T07:23:39.913774+0000","last_peered":"2024-03-06T07:23:39.913774+0000","last_clean":"2024-03-06T07:23:39.913774+0000","last_became_active":"2024-03-06T07:23:39.911529+0000","last_became_peered":"2024-03-06T07:23:39.911529+0000","last_unstale":"2024-03-06T07:23:39.913774+0000","last_undegraded":"2024-03-06T07:23:39.913774+0000","last_fullsized":"2024-03-06T07:23:39.913774+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894652+0000","last_change":"2024-03-06T07:23:39.894652+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894652+0000","last_undegraded":"2024-03-06T07:23:39.894652+0000","last_fullsized":"2024-03-06T07:23:39.894652+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.911862+0000","last_change":"2024-03-06T07:23:39.911223+0000","last_active":"2024-03-06T07:23:39.911862+0000","last_peered":"2024-03-06T07:23:39.911862+0000","last_clean":"2024-03-06T07:23:39.911862+0000","last_became_active":"2024-03-06T07:23:39.910935+0000","last_became_peered":"2024-03-06T07:23:39.910935+0000","last_unstale":"2024-03-06T07:23:39.911862+0000","last_undegraded":"2024-03-06T07:23:39.911862+0000","last_fullsized":"2024-03-06T07:23:39.911862+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:46.994267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897546+0000","last_change":"2024-03-06T07:23:39.897546+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897546+0000","last_undegraded":"2024-03-06T07:23:39.897546+0000","last_fullsized":"2024-03-06T07:23:39.897546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.905864+0000","last_change":"2024-03-06T07:23:39.903773+0000","last_active":"2024-03-06T07:23:39.905864+0000","last_peered":"2024-03-06T07:23:39.905864+0000","last_clean":"2024-03-06T07:23:39.905864+0000","last_became_active":"2024-03-06T07:23:39.903494+0000","last_became_peered":"2024-03-06T07:23:39.903494+0000","last_unstale":"2024-03-06T07:23:39.905864+0000","last_undegraded":"2024-03-06T07:23:39.905864+0000","last_fullsized":"2024-03-06T07:23:39.905864+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:14.991973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.898830+0000","last_change":"2024-03-06T07:23:39.898830+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.898830+0000","last_undegraded":"2024-03-06T07:23:39.898830+0000","last_fullsized":"2024-03-06T07:23:39.898830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913738+0000","last_change":"2024-03-06T07:23:39.911261+0000","last_active":"2024-03-06T07:23:39.913738+0000","last_peered":"2024-03-06T07:23:39.913738+0000","last_clean":"2024-03-06T07:23:39.913738+0000","last_became_active":"2024-03-06T07:23:39.910917+0000","last_became_peered":"2024-03-06T07:23:39.910917+0000","last_unstale":"2024-03-06T07:23:39.913738+0000","last_undegraded":"2024-03-06T07:23:39.913738+0000","last_fullsized":"2024-03-06T07:23:39.913738+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:31.499579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.893681+0000","last_change":"2024-03-06T07:23:39.893681+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.893681+0000","last_undegraded":"2024-03-06T07:23:39.893681+0000","last_fullsized":"2024-03-06T07:23:39.893681+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913098+0000","last_change":"2024-03-06T07:23:39.909408+0000","last_active":"2024-03-06T07:23:39.913098+0000","last_peered":"2024-03-06T07:23:39.913098+0000","last_clean":"2024-03-06T07:23:39.913098+0000","last_became_active":"2024-03-06T07:23:39.909128+0000","last_became_peered":"2024-03-06T07:23:39.909128+0000","last_unstale":"2024-03-06T07:23:39.913098+0000","last_undegraded":"2024-03-06T07:23:39.913098+0000","last_fullsized":"2024-03-06T07:23:39.913098+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:12.087590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895570+0000","last_change":"2024-03-06T07:23:39.895570+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895570+0000","last_undegraded":"2024-03-06T07:23:39.895570+0000","last_fullsized":"2024-03-06T07:23:39.895570+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.908963+0000","last_change":"2024-03-06T07:23:39.905798+0000","last_active":"2024-03-06T07:23:39.908963+0000","last_peered":"2024-03-06T07:23:39.908963+0000","last_clean":"2024-03-06T07:23:39.908963+0000","last_became_active":"2024-03-06T07:23:39.905513+0000","last_became_peered":"2024-03-06T07:23:39.905513+0000","last_unstale":"2024-03-06T07:23:39.908963+0000","last_undegraded":"2024-03-06T07:23:39.908963+0000","last_fullsized":"2024-03-06T07:23:39.908963+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892814+0000","last_change":"2024-03-06T07:23:39.892814+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892814+0000","last_undegraded":"2024-03-06T07:23:39.892814+0000","last_fullsized":"2024-03-06T07:23:39.892814+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.906475+0000","last_change":"2024-03-06T07:23:39.905335+0000","last_active":"2024-03-06T07:23:39.906475+0000","last_peered":"2024-03-06T07:23:39.906475+0000","last_clean":"2024-03-06T07:23:39.906475+0000","last_became_active":"2024-03-06T07:23:39.903169+0000","last_became_peered":"2024-03-06T07:23:39.903169+0000","last_unstale":"2024-03-06T07:23:39.906475+0000","last_undegraded":"2024-03-06T07:23:39.906475+0000","last_fullsized":"2024-03-06T07:23:39.906475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:44.722551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894171+0000","last_change":"2024-03-06T07:23:39.894171+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894171+0000","last_undegraded":"2024-03-06T07:23:39.894171+0000","last_fullsized":"2024-03-06T07:23:39.894171+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910527+0000","last_change":"2024-03-06T07:23:39.908894+0000","last_active":"2024-03-06T07:23:39.910527+0000","last_peered":"2024-03-06T07:23:39.910527+0000","last_clean":"2024-03-06T07:23:39.910527+0000","last_became_active":"2024-03-06T07:23:39.908206+0000","last_became_peered":"2024-03-06T07:23:39.908206+0000","last_unstale":"2024-03-06T07:23:39.910527+0000","last_undegraded":"2024-03-06T07:23:39.910527+0000","last_fullsized":"2024-03-06T07:23:39.910527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:10.543215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.895809+0000","last_change":"2024-03-06T07:23:39.895809+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.895809+0000","last_undegraded":"2024-03-06T07:23:39.895809+0000","last_fullsized":"2024-03-06T07:23:39.895809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915464+0000","last_change":"2024-03-06T07:23:39.909279+0000","last_active":"2024-03-06T07:23:39.915464+0000","last_peered":"2024-03-06T07:23:39.915464+0000","last_clean":"2024-03-06T07:23:39.915464+0000","last_became_active":"2024-03-06T07:23:39.908955+0000","last_became_peered":"2024-03-06T07:23:39.908955+0000","last_unstale":"2024-03-06T07:23:39.915464+0000","last_undegraded":"2024-03-06T07:23:39.915464+0000","last_fullsized":"2024-03-06T07:23:39.915464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:42.900316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897107+0000","last_change":"2024-03-06T07:23:39.897107+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897107+0000","last_undegraded":"2024-03-06T07:23:39.897107+0000","last_fullsized":"2024-03-06T07:23:39.897107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.910530+0000","last_change":"2024-03-06T07:23:39.908492+0000","last_active":"2024-03-06T07:23:39.910530+0000","last_peered":"2024-03-06T07:23:39.910530+0000","last_clean":"2024-03-06T07:23:39.910530+0000","last_became_active":"2024-03-06T07:23:39.908214+0000","last_became_peered":"2024-03-06T07:23:39.908214+0000","last_unstale":"2024-03-06T07:23:39.910530+0000","last_undegraded":"2024-03-06T07:23:39.910530+0000","last_fullsized":"2024-03-06T07:23:39.910530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.562010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.901075+0000","last_change":"2024-03-06T07:23:39.901075+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.901075+0000","last_undegraded":"2024-03-06T07:23:39.901075+0000","last_fullsized":"2024-03-06T07:23:39.901075+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.907658+0000","last_change":"2024-03-06T07:23:39.901912+0000","last_active":"2024-03-06T07:23:39.907658+0000","last_peered":"2024-03-06T07:23:39.907658+0000","last_clean":"2024-03-06T07:23:39.907658+0000","last_became_active":"2024-03-06T07:23:39.901713+0000","last_became_peered":"2024-03-06T07:23:39.901713+0000","last_unstale":"2024-03-06T07:23:39.907658+0000","last_undegraded":"2024-03-06T07:23:39.907658+0000","last_fullsized":"2024-03-06T07:23:39.907658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:31.271276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.897613+0000","last_change":"2024-03-06T07:23:39.897613+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.897613+0000","last_undegraded":"2024-03-06T07:23:39.897613+0000","last_fullsized":"2024-03-06T07:23:39.897613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.915433+0000","last_change":"2024-03-06T07:23:39.908817+0000","last_active":"2024-03-06T07:23:39.915433+0000","last_peered":"2024-03-06T07:23:39.915433+0000","last_clean":"2024-03-06T07:23:39.915433+0000","last_became_active":"2024-03-06T07:23:39.908523+0000","last_became_peered":"2024-03-06T07:23:39.908523+0000","last_unstale":"2024-03-06T07:23:39.915433+0000","last_undegraded":"2024-03-06T07:23:39.915433+0000","last_fullsized":"2024-03-06T07:23:39.915433+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:23.516812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.892303+0000","last_change":"2024-03-06T07:23:39.892303+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.892303+0000","last_undegraded":"2024-03-06T07:23:39.892303+0000","last_fullsized":"2024-03-06T07:23:39.892303+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.904611+0000","last_change":"2024-03-06T07:23:39.902253+0000","last_active":"2024-03-06T07:23:39.904611+0000","last_peered":"2024-03-06T07:23:39.904611+0000","last_clean":"2024-03-06T07:23:39.904611+0000","last_became_active":"2024-03-06T07:23:39.901938+0000","last_became_peered":"2024-03-06T07:23:39.901938+0000","last_unstale":"2024-03-06T07:23:39.904611+0000","last_undegraded":"2024-03-06T07:23:39.904611+0000","last_fullsized":"2024-03-06T07:23:39.904611+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:49.309308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:39.894516+0000","last_change":"2024-03-06T07:23:39.894516+0000","last_active":"2024-03-06T07:23:39.880214+0000","last_peered":"2024-03-06T07:23:39.880214+0000","last_clean":"2024-03-06T07:23:39.880214+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:39.894516+0000","last_undegraded":"2024-03-06T07:23:39.894516+0000","last_fullsized":"2024-03-06T07:23:39.894516+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.913255+0000","last_change":"2024-03-06T07:23:39.908459+0000","last_active":"2024-03-06T07:23:39.913255+0000","last_peered":"2024-03-06T07:23:39.913255+0000","last_clean":"2024-03-06T07:23:39.913255+0000","last_became_active":"2024-03-06T07:23:39.908193+0000","last_became_peered":"2024-03-06T07:23:39.908193+0000","last_unstale":"2024-03-06T07:23:39.913255+0000","last_undegraded":"2024-03-06T07:23:39.913255+0000","last_fullsized":"2024-03-06T07:23:39.913255+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:00.773349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:39.898698+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:23:39.898698+0000","last_peered":"2024-03-06T07:23:39.898698+0000","last_clean":"2024-03-06T07:23:39.898698+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:23:39.898698+0000","last_undegraded":"2024-03-06T07:23:39.898698+0000","last_fullsized":"2024-03-06T07:23:39.898698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":202040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":195773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":792408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:46.571 INFO:tasks.ceph.mds.b.smithi169.stderr:2024-03-06T07:23:46.570+0000 7f1ae3760640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:48.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:48.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.355+0000 7f4b82864640 1 Processor -- start 2024-03-06T07:23:48.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.355+0000 7f4b82864640 1 -- start start 2024-03-06T07:23:48.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b82864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c0d4760 0x7f4b7c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b82864640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4b7c0d0a80 0x7f4b7c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b82864640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0d25d0 con 0x7f4b7c0b4170 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b82864640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7c0d4bf0 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b82864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0ce9e0 con 0x7f4b7c0d4760 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4b7c0b4170 0x7f4b7c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41296/0 (socket says 172.21.15.122:41296) 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7bfff640 1 -- 172.21.15.122:0/3179471405 learned_addr learned my addr 172.21.15.122:0/3179471405 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7b7fe640 1 --2- 172.21.15.122:0/3179471405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c0d4760 0x7f4b7c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b80e01640 1 --2- 172.21.15.122:0/3179471405 >> v2:172.21.15.122:3300/0 conn(0x7f4b7c0d0a80 0x7f4b7c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7affd640 1 -- 172.21.15.122:0/3179471405 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35620026 0 0) 0x7f4b7c0d25d0 con 0x7f4b7c0b4170 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7affd640 1 -- 172.21.15.122:0/3179471405 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b64003520 con 0x7f4b7c0b4170 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/3179471405 >> v2:172.21.15.122:3300/0 conn(0x7f4b7c0d0a80 msgr2=0x7f4b7c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7b7fe640 1 --2- 172.21.15.122:0/3179471405 >> v2:172.21.15.122:3300/0 conn(0x7f4b7c0d0a80 0x7f4b7c0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/3179471405 >> v1:172.21.15.122:6789/0 conn(0x7f4b7c0b4170 legacy=0x7f4b7c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.356+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/3179471405 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0d2870 con 0x7f4b7c0d4760 2024-03-06T07:23:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.357+0000 7f4b7b7fe640 1 --2- 172.21.15.122:0/3179471405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c0d4760 0x7f4b7c0b6a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b700097a0 tx=0x7f4b7002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7abb4c67523ed28e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.357+0000 7f4b7affd640 1 -- 172.21.15.122:0/3179471405 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70035020 con 0x7f4b7c0d4760 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.357+0000 7f4b7affd640 1 -- 172.21.15.122:0/3179471405 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b70031830 con 0x7f4b7c0d4760 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.357+0000 7f4b7affd640 1 -- 172.21.15.122:0/3179471405 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70031be0 con 0x7f4b7c0d4760 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 -- 172.21.15.122:0/3179471405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c0d4760 msgr2=0x7f4b7c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 --2- 172.21.15.122:0/3179471405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c0d4760 0x7f4b7c0b6a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b700097a0 tx=0x7f4b7002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 -- 172.21.15.122:0/3179471405 shutdown_connections 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 --2- 172.21.15.122:0/3179471405 >> v2:172.21.15.122:3300/0 conn(0x7f4b7c0d0a80 0x7f4b7c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 --2- 172.21.15.122:0/3179471405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c0d4760 0x7f4b7c0b6a90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 -- 172.21.15.122:0/3179471405 >> 172.21.15.122:0/3179471405 conn(0x7f4b7c0d3c40 msgr2=0x7f4b7c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 -- 172.21.15.122:0/3179471405 shutdown_connections 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.358+0000 7f4b82864640 1 -- 172.21.15.122:0/3179471405 wait complete. 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.359+0000 7f4b82864640 1 Processor -- start 2024-03-06T07:23:48.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.359+0000 7f4b82864640 1 -- start start 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b82864640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b7c0b4170 0x7f4b7c1f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b82864640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 0x7f4b7c06c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b82864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c1522e0 0x7f4b7c06c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b82864640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0b4dd0 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b82864640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0b4550 con 0x7f4b7c0b4170 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b82864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0b4f50 con 0x7f4b7c1522e0 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 0x7f4b7c06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 0x7f4b7c06c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47144/0 (socket says 172.21.15.122:47144) 2024-03-06T07:23:48.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/2304811394 learned_addr learned my addr 172.21.15.122:0/2304811394 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b80e01640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c1522e0 0x7f4b7c06c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7bfff640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b7c0b4170 0x7f4b7c1f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c1522e0 msgr2=0x7f4b7c06c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c1522e0 0x7f4b7c06c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b7c0b4170 msgr2=0x7f4b7c1f49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b7c0b4170 0x7f4b7c1f49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7b7fe640 1 -- 172.21.15.122:0/2304811394 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0b6a60 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.360+0000 7f4b7bfff640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b7c0b4170 0x7f4b7c1f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.361+0000 7f4b7b7fe640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 0x7f4b7c06c410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b70000c00 tx=0x7f4b700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.361+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70040070 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.361+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7003a830 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.361+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70035040 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.361+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c0b6d40 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.361+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c0cc6d0 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.362+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7c04f330 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.362+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b7004b060 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.363+0000 7f4b78ff9640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b6806b4d0 0x7f4b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.363+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b700ab910 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.363+0000 7f4b7bfff640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b6806b4d0 0x7f4b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.363+0000 7f4b7bfff640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b6806b4d0 0x7f4b6806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c007fd0 tx=0x7f4b6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.367+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b7007a7a0 con 0x7f4b7c0d0a80 2024-03-06T07:23:48.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.536+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b7c0b6810 con 0x7f4b6806b4d0 2024-03-06T07:23:48.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.542+0000 7f4b78ff9640 1 -- 172.21.15.122:0/2304811394 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251960 (secure 0 0 0) 0x7f4b7c0b6810 con 0x7f4b6806b4d0 2024-03-06T07:23:48.546 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b6806b4d0 msgr2=0x7f4b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b6806b4d0 0x7f4b6806d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c007fd0 tx=0x7f4b6c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 msgr2=0x7f4b7c06c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 0x7f4b7c06c410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b70000c00 tx=0x7f4b700046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 shutdown_connections 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b6806b4d0 0x7f4b6806d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b7c1522e0 0x7f4b7c06c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b7c0d0a80 0x7f4b7c06c410 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 --2- 172.21.15.122:0/2304811394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b7c0b4170 0x7f4b7c1f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 >> 172.21.15.122:0/2304811394 conn(0x7f4b7c0d3c40 msgr2=0x7f4b7c06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 shutdown_connections 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.545+0000 7f4b82864640 1 -- 172.21.15.122:0/2304811394 wait complete. 2024-03-06T07:23:48.560 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-06T07:23:47.352719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254800,"kb_used_data":5352,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719920,"statfs":{"total":773094113280,"available":772833198080,"internally_reserved":0,"allocated":5480448,"data_stored":2906476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.463589"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.541011+0000","last_change":"2024-03-06T07:23:40.912792+0000","last_active":"2024-03-06T07:23:42.541011+0000","last_peered":"2024-03-06T07:23:42.541011+0000","last_clean":"2024-03-06T07:23:42.541011+0000","last_became_active":"2024-03-06T07:23:40.912244+0000","last_became_peered":"2024-03-06T07:23:40.912244+0000","last_unstale":"2024-03-06T07:23:42.541011+0000","last_undegraded":"2024-03-06T07:23:42.541011+0000","last_fullsized":"2024-03-06T07:23:42.541011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:18.563284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389748+0000","last_change":"2024-03-06T07:23:40.909454+0000","last_active":"2024-03-06T07:23:42.389748+0000","last_peered":"2024-03-06T07:23:42.389748+0000","last_clean":"2024-03-06T07:23:42.389748+0000","last_became_active":"2024-03-06T07:23:40.909116+0000","last_became_peered":"2024-03-06T07:23:40.909116+0000","last_unstale":"2024-03-06T07:23:42.389748+0000","last_undegraded":"2024-03-06T07:23:42.389748+0000","last_fullsized":"2024-03-06T07:23:42.389748+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:18.563284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905327+0000","last_change":"2024-03-06T07:23:40.906763+0000","last_active":"2024-03-06T07:23:41.905327+0000","last_peered":"2024-03-06T07:23:41.905327+0000","last_clean":"2024-03-06T07:23:41.905327+0000","last_became_active":"2024-03-06T07:23:40.906485+0000","last_became_peered":"2024-03-06T07:23:40.906485+0000","last_unstale":"2024-03-06T07:23:41.905327+0000","last_undegraded":"2024-03-06T07:23:41.905327+0000","last_fullsized":"2024-03-06T07:23:41.905327+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:49.345095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888646+0000","last_change":"2024-03-06T07:23:40.910438+0000","last_active":"2024-03-06T07:23:42.888646+0000","last_peered":"2024-03-06T07:23:42.888646+0000","last_clean":"2024-03-06T07:23:42.888646+0000","last_became_active":"2024-03-06T07:23:40.910161+0000","last_became_peered":"2024-03-06T07:23:40.910161+0000","last_unstale":"2024-03-06T07:23:42.888646+0000","last_undegraded":"2024-03-06T07:23:42.888646+0000","last_fullsized":"2024-03-06T07:23:42.888646+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:26.170326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389595+0000","last_change":"2024-03-06T07:23:40.909415+0000","last_active":"2024-03-06T07:23:42.389595+0000","last_peered":"2024-03-06T07:23:42.389595+0000","last_clean":"2024-03-06T07:23:42.389595+0000","last_became_active":"2024-03-06T07:23:40.909131+0000","last_became_peered":"2024-03-06T07:23:40.909131+0000","last_unstale":"2024-03-06T07:23:42.389595+0000","last_undegraded":"2024-03-06T07:23:42.389595+0000","last_fullsized":"2024-03-06T07:23:42.389595+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:26.170326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887496+0000","last_change":"2024-03-06T07:23:40.908094+0000","last_active":"2024-03-06T07:23:42.887496+0000","last_peered":"2024-03-06T07:23:42.887496+0000","last_clean":"2024-03-06T07:23:42.887496+0000","last_became_active":"2024-03-06T07:23:40.907846+0000","last_became_peered":"2024-03-06T07:23:40.907846+0000","last_unstale":"2024-03-06T07:23:42.887496+0000","last_undegraded":"2024-03-06T07:23:42.887496+0000","last_fullsized":"2024-03-06T07:23:42.887496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:30.697324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540870+0000","last_change":"2024-03-06T07:23:40.917077+0000","last_active":"2024-03-06T07:23:42.540870+0000","last_peered":"2024-03-06T07:23:42.540870+0000","last_clean":"2024-03-06T07:23:42.540870+0000","last_became_active":"2024-03-06T07:23:40.916775+0000","last_became_peered":"2024-03-06T07:23:40.916775+0000","last_unstale":"2024-03-06T07:23:42.540870+0000","last_undegraded":"2024-03-06T07:23:42.540870+0000","last_fullsized":"2024-03-06T07:23:42.540870+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:06.065506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888399+0000","last_change":"2024-03-06T07:23:40.912930+0000","last_active":"2024-03-06T07:23:42.888399+0000","last_peered":"2024-03-06T07:23:42.888399+0000","last_clean":"2024-03-06T07:23:42.888399+0000","last_became_active":"2024-03-06T07:23:40.912666+0000","last_became_peered":"2024-03-06T07:23:40.912666+0000","last_unstale":"2024-03-06T07:23:42.888399+0000","last_undegraded":"2024-03-06T07:23:42.888399+0000","last_fullsized":"2024-03-06T07:23:42.888399+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:59.424073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888049+0000","last_change":"2024-03-06T07:23:40.918493+0000","last_active":"2024-03-06T07:23:42.888049+0000","last_peered":"2024-03-06T07:23:42.888049+0000","last_clean":"2024-03-06T07:23:42.888049+0000","last_became_active":"2024-03-06T07:23:40.918213+0000","last_became_peered":"2024-03-06T07:23:40.918213+0000","last_unstale":"2024-03-06T07:23:42.888049+0000","last_undegraded":"2024-03-06T07:23:42.888049+0000","last_fullsized":"2024-03-06T07:23:42.888049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:49.345095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888208+0000","last_change":"2024-03-06T07:23:40.911637+0000","last_active":"2024-03-06T07:23:42.888208+0000","last_peered":"2024-03-06T07:23:42.888208+0000","last_clean":"2024-03-06T07:23:42.888208+0000","last_became_active":"2024-03-06T07:23:40.911384+0000","last_became_peered":"2024-03-06T07:23:40.911384+0000","last_unstale":"2024-03-06T07:23:42.888208+0000","last_undegraded":"2024-03-06T07:23:42.888208+0000","last_fullsized":"2024-03-06T07:23:42.888208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:17.815302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888307+0000","last_change":"2024-03-06T07:23:40.918532+0000","last_active":"2024-03-06T07:23:42.888307+0000","last_peered":"2024-03-06T07:23:42.888307+0000","last_clean":"2024-03-06T07:23:42.888307+0000","last_became_active":"2024-03-06T07:23:40.918079+0000","last_became_peered":"2024-03-06T07:23:40.918079+0000","last_unstale":"2024-03-06T07:23:42.888307+0000","last_undegraded":"2024-03-06T07:23:42.888307+0000","last_fullsized":"2024-03-06T07:23:42.888307+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:40.102441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389709+0000","last_change":"2024-03-06T07:23:40.914175+0000","last_active":"2024-03-06T07:23:42.389709+0000","last_peered":"2024-03-06T07:23:42.389709+0000","last_clean":"2024-03-06T07:23:42.389709+0000","last_became_active":"2024-03-06T07:23:40.913493+0000","last_became_peered":"2024-03-06T07:23:40.913493+0000","last_unstale":"2024-03-06T07:23:42.389709+0000","last_undegraded":"2024-03-06T07:23:42.389709+0000","last_fullsized":"2024-03-06T07:23:42.389709+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:14.637055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903092+0000","last_change":"2024-03-06T07:23:40.900067+0000","last_active":"2024-03-06T07:23:41.903092+0000","last_peered":"2024-03-06T07:23:41.903092+0000","last_clean":"2024-03-06T07:23:41.903092+0000","last_became_active":"2024-03-06T07:23:40.899753+0000","last_became_peered":"2024-03-06T07:23:40.899753+0000","last_unstale":"2024-03-06T07:23:41.903092+0000","last_undegraded":"2024-03-06T07:23:41.903092+0000","last_fullsized":"2024-03-06T07:23:41.903092+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:50.311224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887877+0000","last_change":"2024-03-06T07:23:40.918249+0000","last_active":"2024-03-06T07:23:42.887877+0000","last_peered":"2024-03-06T07:23:42.887877+0000","last_clean":"2024-03-06T07:23:42.887877+0000","last_became_active":"2024-03-06T07:23:40.917529+0000","last_became_peered":"2024-03-06T07:23:40.917529+0000","last_unstale":"2024-03-06T07:23:42.887877+0000","last_undegraded":"2024-03-06T07:23:42.887877+0000","last_fullsized":"2024-03-06T07:23:42.887877+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:15.993890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887491+0000","last_change":"2024-03-06T07:23:40.914851+0000","last_active":"2024-03-06T07:23:42.887491+0000","last_peered":"2024-03-06T07:23:42.887491+0000","last_clean":"2024-03-06T07:23:42.887491+0000","last_became_active":"2024-03-06T07:23:40.914624+0000","last_became_peered":"2024-03-06T07:23:40.914624+0000","last_unstale":"2024-03-06T07:23:42.887491+0000","last_undegraded":"2024-03-06T07:23:42.887491+0000","last_fullsized":"2024-03-06T07:23:42.887491+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:11.333997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904342+0000","last_change":"2024-03-06T07:23:40.906640+0000","last_active":"2024-03-06T07:23:41.904342+0000","last_peered":"2024-03-06T07:23:41.904342+0000","last_clean":"2024-03-06T07:23:41.904342+0000","last_became_active":"2024-03-06T07:23:40.906351+0000","last_became_peered":"2024-03-06T07:23:40.906351+0000","last_unstale":"2024-03-06T07:23:41.904342+0000","last_undegraded":"2024-03-06T07:23:41.904342+0000","last_fullsized":"2024-03-06T07:23:41.904342+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:47.722962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905803+0000","last_change":"2024-03-06T07:23:40.912247+0000","last_active":"2024-03-06T07:23:41.905803+0000","last_peered":"2024-03-06T07:23:41.905803+0000","last_clean":"2024-03-06T07:23:41.905803+0000","last_became_active":"2024-03-06T07:23:40.911962+0000","last_became_peered":"2024-03-06T07:23:40.911962+0000","last_unstale":"2024-03-06T07:23:41.905803+0000","last_undegraded":"2024-03-06T07:23:41.905803+0000","last_fullsized":"2024-03-06T07:23:41.905803+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:15.221781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905611+0000","last_change":"2024-03-06T07:23:40.907904+0000","last_active":"2024-03-06T07:23:41.905611+0000","last_peered":"2024-03-06T07:23:41.905611+0000","last_clean":"2024-03-06T07:23:41.905611+0000","last_became_active":"2024-03-06T07:23:40.907329+0000","last_became_peered":"2024-03-06T07:23:40.907329+0000","last_unstale":"2024-03-06T07:23:41.905611+0000","last_undegraded":"2024-03-06T07:23:41.905611+0000","last_fullsized":"2024-03-06T07:23:41.905611+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:27.683895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904437+0000","last_change":"2024-03-06T07:23:40.904985+0000","last_active":"2024-03-06T07:23:41.904437+0000","last_peered":"2024-03-06T07:23:41.904437+0000","last_clean":"2024-03-06T07:23:41.904437+0000","last_became_active":"2024-03-06T07:23:40.904710+0000","last_became_peered":"2024-03-06T07:23:40.904710+0000","last_unstale":"2024-03-06T07:23:41.904437+0000","last_undegraded":"2024-03-06T07:23:41.904437+0000","last_fullsized":"2024-03-06T07:23:41.904437+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:24.518728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903313+0000","last_change":"2024-03-06T07:23:40.905262+0000","last_active":"2024-03-06T07:23:41.903313+0000","last_peered":"2024-03-06T07:23:41.903313+0000","last_clean":"2024-03-06T07:23:41.903313+0000","last_became_active":"2024-03-06T07:23:40.904911+0000","last_became_peered":"2024-03-06T07:23:40.904911+0000","last_unstale":"2024-03-06T07:23:41.903313+0000","last_undegraded":"2024-03-06T07:23:41.903313+0000","last_fullsized":"2024-03-06T07:23:41.903313+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:13.089506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888397+0000","last_change":"2024-03-06T07:23:40.910322+0000","last_active":"2024-03-06T07:23:42.888397+0000","last_peered":"2024-03-06T07:23:42.888397+0000","last_clean":"2024-03-06T07:23:42.888397+0000","last_became_active":"2024-03-06T07:23:40.910031+0000","last_became_peered":"2024-03-06T07:23:40.910031+0000","last_unstale":"2024-03-06T07:23:42.888397+0000","last_undegraded":"2024-03-06T07:23:42.888397+0000","last_fullsized":"2024-03-06T07:23:42.888397+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:04.893985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888415+0000","last_change":"2024-03-06T07:23:40.910419+0000","last_active":"2024-03-06T07:23:42.888415+0000","last_peered":"2024-03-06T07:23:42.888415+0000","last_clean":"2024-03-06T07:23:42.888415+0000","last_became_active":"2024-03-06T07:23:40.910133+0000","last_became_peered":"2024-03-06T07:23:40.910133+0000","last_unstale":"2024-03-06T07:23:42.888415+0000","last_undegraded":"2024-03-06T07:23:42.888415+0000","last_fullsized":"2024-03-06T07:23:42.888415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:48.431072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904854+0000","last_change":"2024-03-06T07:23:40.907695+0000","last_active":"2024-03-06T07:23:41.904854+0000","last_peered":"2024-03-06T07:23:41.904854+0000","last_clean":"2024-03-06T07:23:41.904854+0000","last_became_active":"2024-03-06T07:23:40.907319+0000","last_became_peered":"2024-03-06T07:23:40.907319+0000","last_unstale":"2024-03-06T07:23:41.904854+0000","last_undegraded":"2024-03-06T07:23:41.904854+0000","last_fullsized":"2024-03-06T07:23:41.904854+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:01.775265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903977+0000","last_change":"2024-03-06T07:23:40.900730+0000","last_active":"2024-03-06T07:23:41.903977+0000","last_peered":"2024-03-06T07:23:41.903977+0000","last_clean":"2024-03-06T07:23:41.903977+0000","last_became_active":"2024-03-06T07:23:40.900157+0000","last_became_peered":"2024-03-06T07:23:40.900157+0000","last_unstale":"2024-03-06T07:23:41.903977+0000","last_undegraded":"2024-03-06T07:23:41.903977+0000","last_fullsized":"2024-03-06T07:23:41.903977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:32.273192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904675+0000","last_change":"2024-03-06T07:23:40.916916+0000","last_active":"2024-03-06T07:23:41.904675+0000","last_peered":"2024-03-06T07:23:41.904675+0000","last_clean":"2024-03-06T07:23:41.904675+0000","last_became_active":"2024-03-06T07:23:40.916528+0000","last_became_peered":"2024-03-06T07:23:40.916528+0000","last_unstale":"2024-03-06T07:23:41.904675+0000","last_undegraded":"2024-03-06T07:23:41.904675+0000","last_fullsized":"2024-03-06T07:23:41.904675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:05.467319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903051+0000","last_change":"2024-03-06T07:23:40.901029+0000","last_active":"2024-03-06T07:23:41.903051+0000","last_peered":"2024-03-06T07:23:41.903051+0000","last_clean":"2024-03-06T07:23:41.903051+0000","last_became_active":"2024-03-06T07:23:40.900760+0000","last_became_peered":"2024-03-06T07:23:40.900760+0000","last_unstale":"2024-03-06T07:23:41.903051+0000","last_undegraded":"2024-03-06T07:23:41.903051+0000","last_fullsized":"2024-03-06T07:23:41.903051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:18.219014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905461+0000","last_change":"2024-03-06T07:23:40.911879+0000","last_active":"2024-03-06T07:23:41.905461+0000","last_peered":"2024-03-06T07:23:41.905461+0000","last_clean":"2024-03-06T07:23:41.905461+0000","last_became_active":"2024-03-06T07:23:40.911458+0000","last_became_peered":"2024-03-06T07:23:40.911458+0000","last_unstale":"2024-03-06T07:23:41.905461+0000","last_undegraded":"2024-03-06T07:23:41.905461+0000","last_fullsized":"2024-03-06T07:23:41.905461+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:47.996183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389700+0000","last_change":"2024-03-06T07:23:40.913181+0000","last_active":"2024-03-06T07:23:42.389700+0000","last_peered":"2024-03-06T07:23:42.389700+0000","last_clean":"2024-03-06T07:23:42.389700+0000","last_became_active":"2024-03-06T07:23:40.912626+0000","last_became_peered":"2024-03-06T07:23:40.912626+0000","last_unstale":"2024-03-06T07:23:42.389700+0000","last_undegraded":"2024-03-06T07:23:42.389700+0000","last_fullsized":"2024-03-06T07:23:42.389700+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:06.065506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904555+0000","last_change":"2024-03-06T07:23:40.917768+0000","last_active":"2024-03-06T07:23:41.904555+0000","last_peered":"2024-03-06T07:23:41.904555+0000","last_clean":"2024-03-06T07:23:41.904555+0000","last_became_active":"2024-03-06T07:23:40.917435+0000","last_became_peered":"2024-03-06T07:23:40.917435+0000","last_unstale":"2024-03-06T07:23:41.904555+0000","last_undegraded":"2024-03-06T07:23:41.904555+0000","last_fullsized":"2024-03-06T07:23:41.904555+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:43.902232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905302+0000","last_change":"2024-03-06T07:23:40.908456+0000","last_active":"2024-03-06T07:23:41.905302+0000","last_peered":"2024-03-06T07:23:41.905302+0000","last_clean":"2024-03-06T07:23:41.905302+0000","last_became_active":"2024-03-06T07:23:40.908170+0000","last_became_peered":"2024-03-06T07:23:40.908170+0000","last_unstale":"2024-03-06T07:23:41.905302+0000","last_undegraded":"2024-03-06T07:23:41.905302+0000","last_fullsized":"2024-03-06T07:23:41.905302+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:45.724467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887582+0000","last_change":"2024-03-06T07:23:40.913414+0000","last_active":"2024-03-06T07:23:42.887582+0000","last_peered":"2024-03-06T07:23:42.887582+0000","last_clean":"2024-03-06T07:23:42.887582+0000","last_became_active":"2024-03-06T07:23:40.913149+0000","last_became_peered":"2024-03-06T07:23:40.913149+0000","last_unstale":"2024-03-06T07:23:42.887582+0000","last_undegraded":"2024-03-06T07:23:42.887582+0000","last_fullsized":"2024-03-06T07:23:42.887582+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:33.229068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903310+0000","last_change":"2024-03-06T07:23:40.916759+0000","last_active":"2024-03-06T07:23:41.903310+0000","last_peered":"2024-03-06T07:23:41.903310+0000","last_clean":"2024-03-06T07:23:41.903310+0000","last_became_active":"2024-03-06T07:23:40.916509+0000","last_became_peered":"2024-03-06T07:23:40.916509+0000","last_unstale":"2024-03-06T07:23:41.903310+0000","last_undegraded":"2024-03-06T07:23:41.903310+0000","last_fullsized":"2024-03-06T07:23:41.903310+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:21.489005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.252726+0000","last_change":"2024-03-06T07:23:40.916664+0000","last_active":"2024-03-06T07:23:42.252726+0000","last_peered":"2024-03-06T07:23:42.252726+0000","last_clean":"2024-03-06T07:23:42.252726+0000","last_became_active":"2024-03-06T07:23:40.916382+0000","last_became_peered":"2024-03-06T07:23:40.916382+0000","last_unstale":"2024-03-06T07:23:42.252726+0000","last_undegraded":"2024-03-06T07:23:42.252726+0000","last_fullsized":"2024-03-06T07:23:42.252726+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:43.087272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888238+0000","last_change":"2024-03-06T07:23:39.900097+0000","last_active":"2024-03-06T07:23:42.888238+0000","last_peered":"2024-03-06T07:23:42.888238+0000","last_clean":"2024-03-06T07:23:42.888238+0000","last_became_active":"2024-03-06T07:23:39.899808+0000","last_became_peered":"2024-03-06T07:23:39.899808+0000","last_unstale":"2024-03-06T07:23:42.888238+0000","last_undegraded":"2024-03-06T07:23:42.888238+0000","last_fullsized":"2024-03-06T07:23:42.888238+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:16.813386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904685+0000","last_change":"2024-03-06T07:23:40.912578+0000","last_active":"2024-03-06T07:23:41.904685+0000","last_peered":"2024-03-06T07:23:41.904685+0000","last_clean":"2024-03-06T07:23:41.904685+0000","last_became_active":"2024-03-06T07:23:40.912319+0000","last_became_peered":"2024-03-06T07:23:40.912319+0000","last_unstale":"2024-03-06T07:23:41.904685+0000","last_undegraded":"2024-03-06T07:23:41.904685+0000","last_fullsized":"2024-03-06T07:23:41.904685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:23.419790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905778+0000","last_change":"2024-03-06T07:23:39.899915+0000","last_active":"2024-03-06T07:23:41.905778+0000","last_peered":"2024-03-06T07:23:41.905778+0000","last_clean":"2024-03-06T07:23:41.905778+0000","last_became_active":"2024-03-06T07:23:39.899555+0000","last_became_peered":"2024-03-06T07:23:39.899555+0000","last_unstale":"2024-03-06T07:23:41.905778+0000","last_undegraded":"2024-03-06T07:23:41.905778+0000","last_fullsized":"2024-03-06T07:23:41.905778+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:42.900316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389827+0000","last_change":"2024-03-06T07:23:40.915080+0000","last_active":"2024-03-06T07:23:42.389827+0000","last_peered":"2024-03-06T07:23:42.389827+0000","last_clean":"2024-03-06T07:23:42.389827+0000","last_became_active":"2024-03-06T07:23:40.914820+0000","last_became_peered":"2024-03-06T07:23:40.914820+0000","last_unstale":"2024-03-06T07:23:42.389827+0000","last_undegraded":"2024-03-06T07:23:42.389827+0000","last_fullsized":"2024-03-06T07:23:42.389827+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:14.418071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.252799+0000","last_change":"2024-03-06T07:23:39.911329+0000","last_active":"2024-03-06T07:23:42.252799+0000","last_peered":"2024-03-06T07:23:42.252799+0000","last_clean":"2024-03-06T07:23:42.252799+0000","last_became_active":"2024-03-06T07:23:39.911011+0000","last_became_peered":"2024-03-06T07:23:39.911011+0000","last_unstale":"2024-03-06T07:23:42.252799+0000","last_undegraded":"2024-03-06T07:23:42.252799+0000","last_fullsized":"2024-03-06T07:23:42.252799+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:57.881886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.252759+0000","last_change":"2024-03-06T07:23:40.917227+0000","last_active":"2024-03-06T07:23:42.252759+0000","last_peered":"2024-03-06T07:23:42.252759+0000","last_clean":"2024-03-06T07:23:42.252759+0000","last_became_active":"2024-03-06T07:23:40.916808+0000","last_became_peered":"2024-03-06T07:23:40.916808+0000","last_unstale":"2024-03-06T07:23:42.252759+0000","last_undegraded":"2024-03-06T07:23:42.252759+0000","last_fullsized":"2024-03-06T07:23:42.252759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:15.993890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540853+0000","last_change":"2024-03-06T07:23:39.908704+0000","last_active":"2024-03-06T07:23:42.540853+0000","last_peered":"2024-03-06T07:23:42.540853+0000","last_clean":"2024-03-06T07:23:42.540853+0000","last_became_active":"2024-03-06T07:23:39.908289+0000","last_became_peered":"2024-03-06T07:23:39.908289+0000","last_unstale":"2024-03-06T07:23:42.540853+0000","last_undegraded":"2024-03-06T07:23:42.540853+0000","last_fullsized":"2024-03-06T07:23:42.540853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:14.991973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905390+0000","last_change":"2024-03-06T07:23:40.909707+0000","last_active":"2024-03-06T07:23:41.905390+0000","last_peered":"2024-03-06T07:23:41.905390+0000","last_clean":"2024-03-06T07:23:41.905390+0000","last_became_active":"2024-03-06T07:23:40.909438+0000","last_became_peered":"2024-03-06T07:23:40.909438+0000","last_unstale":"2024-03-06T07:23:41.905390+0000","last_undegraded":"2024-03-06T07:23:41.905390+0000","last_fullsized":"2024-03-06T07:23:41.905390+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:07.384292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888523+0000","last_change":"2024-03-06T07:23:39.908607+0000","last_active":"2024-03-06T07:23:42.888523+0000","last_peered":"2024-03-06T07:23:42.888523+0000","last_clean":"2024-03-06T07:23:42.888523+0000","last_became_active":"2024-03-06T07:23:39.908334+0000","last_became_peered":"2024-03-06T07:23:39.908334+0000","last_unstale":"2024-03-06T07:23:42.888523+0000","last_undegraded":"2024-03-06T07:23:42.888523+0000","last_fullsized":"2024-03-06T07:23:42.888523+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.100525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888106+0000","last_change":"2024-03-06T07:23:40.909260+0000","last_active":"2024-03-06T07:23:42.888106+0000","last_peered":"2024-03-06T07:23:42.888106+0000","last_clean":"2024-03-06T07:23:42.888106+0000","last_became_active":"2024-03-06T07:23:40.908964+0000","last_became_peered":"2024-03-06T07:23:40.908964+0000","last_unstale":"2024-03-06T07:23:42.888106+0000","last_undegraded":"2024-03-06T07:23:42.888106+0000","last_fullsized":"2024-03-06T07:23:42.888106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:07.384292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905605+0000","last_change":"2024-03-06T07:23:39.908741+0000","last_active":"2024-03-06T07:23:41.905605+0000","last_peered":"2024-03-06T07:23:41.905605+0000","last_clean":"2024-03-06T07:23:41.905605+0000","last_became_active":"2024-03-06T07:23:39.908470+0000","last_became_peered":"2024-03-06T07:23:39.908470+0000","last_unstale":"2024-03-06T07:23:41.905605+0000","last_undegraded":"2024-03-06T07:23:41.905605+0000","last_fullsized":"2024-03-06T07:23:41.905605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:21.403195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888205+0000","last_change":"2024-03-06T07:23:40.912356+0000","last_active":"2024-03-06T07:23:42.888205+0000","last_peered":"2024-03-06T07:23:42.888205+0000","last_clean":"2024-03-06T07:23:42.888205+0000","last_became_active":"2024-03-06T07:23:40.912080+0000","last_became_peered":"2024-03-06T07:23:40.912080+0000","last_unstale":"2024-03-06T07:23:42.888205+0000","last_undegraded":"2024-03-06T07:23:42.888205+0000","last_fullsized":"2024-03-06T07:23:42.888205+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:27.683895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389420+0000","last_change":"2024-03-06T07:23:39.909770+0000","last_active":"2024-03-06T07:23:42.389420+0000","last_peered":"2024-03-06T07:23:42.389420+0000","last_clean":"2024-03-06T07:23:42.389420+0000","last_became_active":"2024-03-06T07:23:39.909494+0000","last_became_peered":"2024-03-06T07:23:39.909494+0000","last_unstale":"2024-03-06T07:23:42.389420+0000","last_undegraded":"2024-03-06T07:23:42.389420+0000","last_fullsized":"2024-03-06T07:23:42.389420+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:44.722551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904281+0000","last_change":"2024-03-06T07:23:40.909433+0000","last_active":"2024-03-06T07:23:41.904281+0000","last_peered":"2024-03-06T07:23:41.904281+0000","last_clean":"2024-03-06T07:23:41.904281+0000","last_became_active":"2024-03-06T07:23:40.908682+0000","last_became_peered":"2024-03-06T07:23:40.908682+0000","last_unstale":"2024-03-06T07:23:41.904281+0000","last_undegraded":"2024-03-06T07:23:41.904281+0000","last_fullsized":"2024-03-06T07:23:41.904281+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:22.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888523+0000","last_change":"2024-03-06T07:23:39.906386+0000","last_active":"2024-03-06T07:23:42.888523+0000","last_peered":"2024-03-06T07:23:42.888523+0000","last_clean":"2024-03-06T07:23:42.888523+0000","last_became_active":"2024-03-06T07:23:39.905390+0000","last_became_peered":"2024-03-06T07:23:39.905390+0000","last_unstale":"2024-03-06T07:23:42.888523+0000","last_undegraded":"2024-03-06T07:23:42.888523+0000","last_fullsized":"2024-03-06T07:23:42.888523+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:31.271276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905737+0000","last_change":"2024-03-06T07:23:40.906536+0000","last_active":"2024-03-06T07:23:41.905737+0000","last_peered":"2024-03-06T07:23:41.905737+0000","last_clean":"2024-03-06T07:23:41.905737+0000","last_became_active":"2024-03-06T07:23:40.906287+0000","last_became_peered":"2024-03-06T07:23:40.906287+0000","last_unstale":"2024-03-06T07:23:41.905737+0000","last_undegraded":"2024-03-06T07:23:41.905737+0000","last_fullsized":"2024-03-06T07:23:41.905737+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:17.815302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905850+0000","last_change":"2024-03-06T07:23:39.901277+0000","last_active":"2024-03-06T07:23:41.905850+0000","last_peered":"2024-03-06T07:23:41.905850+0000","last_clean":"2024-03-06T07:23:41.905850+0000","last_became_active":"2024-03-06T07:23:39.900996+0000","last_became_peered":"2024-03-06T07:23:39.900996+0000","last_unstale":"2024-03-06T07:23:41.905850+0000","last_undegraded":"2024-03-06T07:23:41.905850+0000","last_fullsized":"2024-03-06T07:23:41.905850+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:45.430603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.541230+0000","last_change":"2024-03-06T07:23:40.906428+0000","last_active":"2024-03-06T07:23:42.541230+0000","last_peered":"2024-03-06T07:23:42.541230+0000","last_clean":"2024-03-06T07:23:42.541230+0000","last_became_active":"2024-03-06T07:23:40.906063+0000","last_became_peered":"2024-03-06T07:23:40.906063+0000","last_unstale":"2024-03-06T07:23:42.541230+0000","last_undegraded":"2024-03-06T07:23:42.541230+0000","last_fullsized":"2024-03-06T07:23:42.541230+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:01.088181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888324+0000","last_change":"2024-03-06T07:23:39.904382+0000","last_active":"2024-03-06T07:23:42.888324+0000","last_peered":"2024-03-06T07:23:42.888324+0000","last_clean":"2024-03-06T07:23:42.888324+0000","last_became_active":"2024-03-06T07:23:39.904076+0000","last_became_peered":"2024-03-06T07:23:39.904076+0000","last_unstale":"2024-03-06T07:23:42.888324+0000","last_undegraded":"2024-03-06T07:23:42.888324+0000","last_fullsized":"2024-03-06T07:23:42.888324+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:45.519711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888047+0000","last_change":"2024-03-06T07:23:40.906052+0000","last_active":"2024-03-06T07:23:42.888047+0000","last_peered":"2024-03-06T07:23:42.888047+0000","last_clean":"2024-03-06T07:23:42.888047+0000","last_became_active":"2024-03-06T07:23:40.905786+0000","last_became_peered":"2024-03-06T07:23:40.905786+0000","last_unstale":"2024-03-06T07:23:42.888047+0000","last_undegraded":"2024-03-06T07:23:42.888047+0000","last_fullsized":"2024-03-06T07:23:42.888047+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:40.437973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.541091+0000","last_change":"2024-03-06T07:23:39.909133+0000","last_active":"2024-03-06T07:23:42.541091+0000","last_peered":"2024-03-06T07:23:42.541091+0000","last_clean":"2024-03-06T07:23:42.541091+0000","last_became_active":"2024-03-06T07:23:39.908563+0000","last_became_peered":"2024-03-06T07:23:39.908563+0000","last_unstale":"2024-03-06T07:23:42.541091+0000","last_undegraded":"2024-03-06T07:23:42.541091+0000","last_fullsized":"2024-03-06T07:23:42.541091+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:06.382376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.541045+0000","last_change":"2024-03-06T07:23:40.917685+0000","last_active":"2024-03-06T07:23:42.541045+0000","last_peered":"2024-03-06T07:23:42.541045+0000","last_clean":"2024-03-06T07:23:42.541045+0000","last_became_active":"2024-03-06T07:23:40.917423+0000","last_became_peered":"2024-03-06T07:23:40.917423+0000","last_unstale":"2024-03-06T07:23:42.541045+0000","last_undegraded":"2024-03-06T07:23:42.541045+0000","last_fullsized":"2024-03-06T07:23:42.541045+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:43.155927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903274+0000","last_change":"2024-03-06T07:23:39.896528+0000","last_active":"2024-03-06T07:23:41.903274+0000","last_peered":"2024-03-06T07:23:41.903274+0000","last_clean":"2024-03-06T07:23:41.903274+0000","last_became_active":"2024-03-06T07:23:39.896142+0000","last_became_peered":"2024-03-06T07:23:39.896142+0000","last_unstale":"2024-03-06T07:23:41.903274+0000","last_undegraded":"2024-03-06T07:23:41.903274+0000","last_fullsized":"2024-03-06T07:23:41.903274+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:48.648164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904100+0000","last_change":"2024-03-06T07:23:40.905794+0000","last_active":"2024-03-06T07:23:41.904100+0000","last_peered":"2024-03-06T07:23:41.904100+0000","last_clean":"2024-03-06T07:23:41.904100+0000","last_became_active":"2024-03-06T07:23:40.905532+0000","last_became_peered":"2024-03-06T07:23:40.905532+0000","last_unstale":"2024-03-06T07:23:41.904100+0000","last_undegraded":"2024-03-06T07:23:41.904100+0000","last_fullsized":"2024-03-06T07:23:41.904100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:48.232800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888629+0000","last_change":"2024-03-06T07:23:39.905467+0000","last_active":"2024-03-06T07:23:42.888629+0000","last_peered":"2024-03-06T07:23:42.888629+0000","last_clean":"2024-03-06T07:23:42.888629+0000","last_became_active":"2024-03-06T07:23:39.905185+0000","last_became_peered":"2024-03-06T07:23:39.905185+0000","last_unstale":"2024-03-06T07:23:42.888629+0000","last_undegraded":"2024-03-06T07:23:42.888629+0000","last_fullsized":"2024-03-06T07:23:42.888629+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:58.422157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888692+0000","last_change":"2024-03-06T07:23:40.906773+0000","last_active":"2024-03-06T07:23:42.888692+0000","last_peered":"2024-03-06T07:23:42.888692+0000","last_clean":"2024-03-06T07:23:42.888692+0000","last_became_active":"2024-03-06T07:23:40.906535+0000","last_became_peered":"2024-03-06T07:23:40.906535+0000","last_unstale":"2024-03-06T07:23:42.888692+0000","last_undegraded":"2024-03-06T07:23:42.888692+0000","last_fullsized":"2024-03-06T07:23:42.888692+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:06.065506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540846+0000","last_change":"2024-03-06T07:23:39.905513+0000","last_active":"2024-03-06T07:23:42.540846+0000","last_peered":"2024-03-06T07:23:42.540846+0000","last_clean":"2024-03-06T07:23:42.540846+0000","last_became_active":"2024-03-06T07:23:39.905121+0000","last_became_peered":"2024-03-06T07:23:39.905121+0000","last_unstale":"2024-03-06T07:23:42.540846+0000","last_undegraded":"2024-03-06T07:23:42.540846+0000","last_fullsized":"2024-03-06T07:23:42.540846+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:44.722551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887460+0000","last_change":"2024-03-06T07:23:40.906996+0000","last_active":"2024-03-06T07:23:42.887460+0000","last_peered":"2024-03-06T07:23:42.887460+0000","last_clean":"2024-03-06T07:23:42.887460+0000","last_became_active":"2024-03-06T07:23:40.906733+0000","last_became_peered":"2024-03-06T07:23:40.906733+0000","last_unstale":"2024-03-06T07:23:42.887460+0000","last_undegraded":"2024-03-06T07:23:42.887460+0000","last_fullsized":"2024-03-06T07:23:42.887460+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:28.813103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887592+0000","last_change":"2024-03-06T07:23:39.902543+0000","last_active":"2024-03-06T07:23:42.887592+0000","last_peered":"2024-03-06T07:23:42.887592+0000","last_clean":"2024-03-06T07:23:42.887592+0000","last_became_active":"2024-03-06T07:23:39.902271+0000","last_became_peered":"2024-03-06T07:23:39.902271+0000","last_unstale":"2024-03-06T07:23:42.887592+0000","last_undegraded":"2024-03-06T07:23:42.887592+0000","last_fullsized":"2024-03-06T07:23:42.887592+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:14.219864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.541024+0000","last_change":"2024-03-06T07:23:40.904374+0000","last_active":"2024-03-06T07:23:42.541024+0000","last_peered":"2024-03-06T07:23:42.541024+0000","last_clean":"2024-03-06T07:23:42.541024+0000","last_became_active":"2024-03-06T07:23:40.903894+0000","last_became_peered":"2024-03-06T07:23:40.903894+0000","last_unstale":"2024-03-06T07:23:42.541024+0000","last_undegraded":"2024-03-06T07:23:42.541024+0000","last_fullsized":"2024-03-06T07:23:42.541024+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:14.418071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888487+0000","last_change":"2024-03-06T07:23:39.903255+0000","last_active":"2024-03-06T07:23:42.888487+0000","last_peered":"2024-03-06T07:23:42.888487+0000","last_clean":"2024-03-06T07:23:42.888487+0000","last_became_active":"2024-03-06T07:23:39.902984+0000","last_became_peered":"2024-03-06T07:23:39.902984+0000","last_unstale":"2024-03-06T07:23:42.888487+0000","last_undegraded":"2024-03-06T07:23:42.888487+0000","last_fullsized":"2024-03-06T07:23:42.888487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:47.230884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903230+0000","last_change":"2024-03-06T07:23:40.911924+0000","last_active":"2024-03-06T07:23:41.903230+0000","last_peered":"2024-03-06T07:23:41.903230+0000","last_clean":"2024-03-06T07:23:41.903230+0000","last_became_active":"2024-03-06T07:23:40.911652+0000","last_became_peered":"2024-03-06T07:23:40.911652+0000","last_unstale":"2024-03-06T07:23:41.903230+0000","last_undegraded":"2024-03-06T07:23:41.903230+0000","last_fullsized":"2024-03-06T07:23:41.903230+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:52.298760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903103+0000","last_change":"2024-03-06T07:23:39.906113+0000","last_active":"2024-03-06T07:23:41.903103+0000","last_peered":"2024-03-06T07:23:41.903103+0000","last_clean":"2024-03-06T07:23:41.903103+0000","last_became_active":"2024-03-06T07:23:39.905836+0000","last_became_peered":"2024-03-06T07:23:39.905836+0000","last_unstale":"2024-03-06T07:23:41.903103+0000","last_undegraded":"2024-03-06T07:23:41.903103+0000","last_fullsized":"2024-03-06T07:23:41.903103+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:12.496011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888269+0000","last_change":"2024-03-06T07:23:40.901215+0000","last_active":"2024-03-06T07:23:42.888269+0000","last_peered":"2024-03-06T07:23:42.888269+0000","last_clean":"2024-03-06T07:23:42.888269+0000","last_became_active":"2024-03-06T07:23:40.900949+0000","last_became_peered":"2024-03-06T07:23:40.900949+0000","last_unstale":"2024-03-06T07:23:42.888269+0000","last_undegraded":"2024-03-06T07:23:42.888269+0000","last_fullsized":"2024-03-06T07:23:42.888269+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:15.221781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905849+0000","last_change":"2024-03-06T07:23:39.904607+0000","last_active":"2024-03-06T07:23:41.905849+0000","last_peered":"2024-03-06T07:23:41.905849+0000","last_clean":"2024-03-06T07:23:41.905849+0000","last_became_active":"2024-03-06T07:23:39.904289+0000","last_became_peered":"2024-03-06T07:23:39.904289+0000","last_unstale":"2024-03-06T07:23:41.905849+0000","last_undegraded":"2024-03-06T07:23:41.905849+0000","last_fullsized":"2024-03-06T07:23:41.905849+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:23.516812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904227+0000","last_change":"2024-03-06T07:23:40.901116+0000","last_active":"2024-03-06T07:23:41.904227+0000","last_peered":"2024-03-06T07:23:41.904227+0000","last_clean":"2024-03-06T07:23:41.904227+0000","last_became_active":"2024-03-06T07:23:40.900577+0000","last_became_peered":"2024-03-06T07:23:40.900577+0000","last_unstale":"2024-03-06T07:23:41.904227+0000","last_undegraded":"2024-03-06T07:23:41.904227+0000","last_fullsized":"2024-03-06T07:23:41.904227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:08.399106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389730+0000","last_change":"2024-03-06T07:23:39.911859+0000","last_active":"2024-03-06T07:23:42.389730+0000","last_peered":"2024-03-06T07:23:42.389730+0000","last_clean":"2024-03-06T07:23:42.389730+0000","last_became_active":"2024-03-06T07:23:39.911529+0000","last_became_peered":"2024-03-06T07:23:39.911529+0000","last_unstale":"2024-03-06T07:23:42.389730+0000","last_undegraded":"2024-03-06T07:23:42.389730+0000","last_fullsized":"2024-03-06T07:23:42.389730+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:14.991973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904640+0000","last_change":"2024-03-06T07:23:40.917746+0000","last_active":"2024-03-06T07:23:41.904640+0000","last_peered":"2024-03-06T07:23:41.904640+0000","last_clean":"2024-03-06T07:23:41.904640+0000","last_became_active":"2024-03-06T07:23:40.917399+0000","last_became_peered":"2024-03-06T07:23:40.917399+0000","last_unstale":"2024-03-06T07:23:41.904640+0000","last_undegraded":"2024-03-06T07:23:41.904640+0000","last_fullsized":"2024-03-06T07:23:41.904640+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:46.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888154+0000","last_change":"2024-03-06T07:23:39.911223+0000","last_active":"2024-03-06T07:23:42.888154+0000","last_peered":"2024-03-06T07:23:42.888154+0000","last_clean":"2024-03-06T07:23:42.888154+0000","last_became_active":"2024-03-06T07:23:39.910935+0000","last_became_peered":"2024-03-06T07:23:39.910935+0000","last_unstale":"2024-03-06T07:23:42.888154+0000","last_undegraded":"2024-03-06T07:23:42.888154+0000","last_fullsized":"2024-03-06T07:23:42.888154+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:26.681979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887540+0000","last_change":"2024-03-06T07:23:40.905993+0000","last_active":"2024-03-06T07:23:42.887540+0000","last_peered":"2024-03-06T07:23:42.887540+0000","last_clean":"2024-03-06T07:23:42.887540+0000","last_became_active":"2024-03-06T07:23:40.905662+0000","last_became_peered":"2024-03-06T07:23:40.905662+0000","last_unstale":"2024-03-06T07:23:42.887540+0000","last_undegraded":"2024-03-06T07:23:42.887540+0000","last_fullsized":"2024-03-06T07:23:42.887540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:47.816692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888745+0000","last_change":"2024-03-06T07:23:39.903773+0000","last_active":"2024-03-06T07:23:42.888745+0000","last_peered":"2024-03-06T07:23:42.888745+0000","last_clean":"2024-03-06T07:23:42.888745+0000","last_became_active":"2024-03-06T07:23:39.903494+0000","last_became_peered":"2024-03-06T07:23:39.903494+0000","last_unstale":"2024-03-06T07:23:42.888745+0000","last_undegraded":"2024-03-06T07:23:42.888745+0000","last_fullsized":"2024-03-06T07:23:42.888745+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:47.429156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.252654+0000","last_change":"2024-03-06T07:23:40.917547+0000","last_active":"2024-03-06T07:23:42.252654+0000","last_peered":"2024-03-06T07:23:42.252654+0000","last_clean":"2024-03-06T07:23:42.252654+0000","last_became_active":"2024-03-06T07:23:40.916934+0000","last_became_peered":"2024-03-06T07:23:40.916934+0000","last_unstale":"2024-03-06T07:23:42.252654+0000","last_undegraded":"2024-03-06T07:23:42.252654+0000","last_fullsized":"2024-03-06T07:23:42.252654+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:34.176714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389474+0000","last_change":"2024-03-06T07:23:39.911261+0000","last_active":"2024-03-06T07:23:42.389474+0000","last_peered":"2024-03-06T07:23:42.389474+0000","last_clean":"2024-03-06T07:23:42.389474+0000","last_became_active":"2024-03-06T07:23:39.910917+0000","last_became_peered":"2024-03-06T07:23:39.910917+0000","last_unstale":"2024-03-06T07:23:42.389474+0000","last_undegraded":"2024-03-06T07:23:42.389474+0000","last_fullsized":"2024-03-06T07:23:42.389474+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:42.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.903194+0000","last_change":"2024-03-06T07:23:40.897961+0000","last_active":"2024-03-06T07:23:41.903194+0000","last_peered":"2024-03-06T07:23:41.903194+0000","last_clean":"2024-03-06T07:23:41.903194+0000","last_became_active":"2024-03-06T07:23:40.897696+0000","last_became_peered":"2024-03-06T07:23:40.897696+0000","last_unstale":"2024-03-06T07:23:41.903194+0000","last_undegraded":"2024-03-06T07:23:41.903194+0000","last_fullsized":"2024-03-06T07:23:41.903194+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:07.105791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887802+0000","last_change":"2024-03-06T07:23:39.909408+0000","last_active":"2024-03-06T07:23:42.887802+0000","last_peered":"2024-03-06T07:23:42.887802+0000","last_clean":"2024-03-06T07:23:42.887802+0000","last_became_active":"2024-03-06T07:23:39.909128+0000","last_became_peered":"2024-03-06T07:23:39.909128+0000","last_unstale":"2024-03-06T07:23:42.887802+0000","last_undegraded":"2024-03-06T07:23:42.887802+0000","last_fullsized":"2024-03-06T07:23:42.887802+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:05.084281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888012+0000","last_change":"2024-03-06T07:23:40.918746+0000","last_active":"2024-03-06T07:23:42.888012+0000","last_peered":"2024-03-06T07:23:42.888012+0000","last_clean":"2024-03-06T07:23:42.888012+0000","last_became_active":"2024-03-06T07:23:40.918483+0000","last_became_peered":"2024-03-06T07:23:40.918483+0000","last_unstale":"2024-03-06T07:23:42.888012+0000","last_undegraded":"2024-03-06T07:23:42.888012+0000","last_fullsized":"2024-03-06T07:23:42.888012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:45.724467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540708+0000","last_change":"2024-03-06T07:23:39.905798+0000","last_active":"2024-03-06T07:23:42.540708+0000","last_peered":"2024-03-06T07:23:42.540708+0000","last_clean":"2024-03-06T07:23:42.540708+0000","last_became_active":"2024-03-06T07:23:39.905513+0000","last_became_peered":"2024-03-06T07:23:39.905513+0000","last_unstale":"2024-03-06T07:23:42.540708+0000","last_undegraded":"2024-03-06T07:23:42.540708+0000","last_fullsized":"2024-03-06T07:23:42.540708+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:42.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.904851+0000","last_change":"2024-03-06T07:23:40.906891+0000","last_active":"2024-03-06T07:23:41.904851+0000","last_peered":"2024-03-06T07:23:41.904851+0000","last_clean":"2024-03-06T07:23:41.904851+0000","last_became_active":"2024-03-06T07:23:40.906365+0000","last_became_peered":"2024-03-06T07:23:40.906365+0000","last_unstale":"2024-03-06T07:23:41.904851+0000","last_undegraded":"2024-03-06T07:23:41.904851+0000","last_fullsized":"2024-03-06T07:23:41.904851+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:58.883802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888444+0000","last_change":"2024-03-06T07:23:39.905335+0000","last_active":"2024-03-06T07:23:42.888444+0000","last_peered":"2024-03-06T07:23:42.888444+0000","last_clean":"2024-03-06T07:23:42.888444+0000","last_became_active":"2024-03-06T07:23:39.903169+0000","last_became_peered":"2024-03-06T07:23:39.903169+0000","last_unstale":"2024-03-06T07:23:42.888444+0000","last_undegraded":"2024-03-06T07:23:42.888444+0000","last_fullsized":"2024-03-06T07:23:42.888444+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:25.168410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888708+0000","last_change":"2024-03-06T07:23:40.913603+0000","last_active":"2024-03-06T07:23:42.888708+0000","last_peered":"2024-03-06T07:23:42.888708+0000","last_clean":"2024-03-06T07:23:42.888708+0000","last_became_active":"2024-03-06T07:23:40.913338+0000","last_became_peered":"2024-03-06T07:23:40.913338+0000","last_unstale":"2024-03-06T07:23:42.888708+0000","last_undegraded":"2024-03-06T07:23:42.888708+0000","last_fullsized":"2024-03-06T07:23:42.888708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:14.637055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.541112+0000","last_change":"2024-03-06T07:23:39.908894+0000","last_active":"2024-03-06T07:23:42.541112+0000","last_peered":"2024-03-06T07:23:42.541112+0000","last_clean":"2024-03-06T07:23:42.541112+0000","last_became_active":"2024-03-06T07:23:39.908206+0000","last_became_peered":"2024-03-06T07:23:39.908206+0000","last_unstale":"2024-03-06T07:23:42.541112+0000","last_undegraded":"2024-03-06T07:23:42.541112+0000","last_fullsized":"2024-03-06T07:23:42.541112+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:46.994267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540987+0000","last_change":"2024-03-06T07:23:40.901113+0000","last_active":"2024-03-06T07:23:42.540987+0000","last_peered":"2024-03-06T07:23:42.540987+0000","last_clean":"2024-03-06T07:23:42.540987+0000","last_became_active":"2024-03-06T07:23:40.900860+0000","last_became_peered":"2024-03-06T07:23:40.900860+0000","last_unstale":"2024-03-06T07:23:42.540987+0000","last_undegraded":"2024-03-06T07:23:42.540987+0000","last_fullsized":"2024-03-06T07:23:42.540987+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:14.637055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.252597+0000","last_change":"2024-03-06T07:23:39.909279+0000","last_active":"2024-03-06T07:23:42.252597+0000","last_peered":"2024-03-06T07:23:42.252597+0000","last_clean":"2024-03-06T07:23:42.252597+0000","last_became_active":"2024-03-06T07:23:39.908955+0000","last_became_peered":"2024-03-06T07:23:39.908955+0000","last_unstale":"2024-03-06T07:23:42.252597+0000","last_undegraded":"2024-03-06T07:23:42.252597+0000","last_fullsized":"2024-03-06T07:23:42.252597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:22.417874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.887678+0000","last_change":"2024-03-06T07:23:40.917865+0000","last_active":"2024-03-06T07:23:42.887678+0000","last_peered":"2024-03-06T07:23:42.887678+0000","last_clean":"2024-03-06T07:23:42.887678+0000","last_became_active":"2024-03-06T07:23:40.916474+0000","last_became_peered":"2024-03-06T07:23:40.916474+0000","last_unstale":"2024-03-06T07:23:42.887678+0000","last_undegraded":"2024-03-06T07:23:42.887678+0000","last_fullsized":"2024-03-06T07:23:42.887678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:08.381016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540925+0000","last_change":"2024-03-06T07:23:39.908492+0000","last_active":"2024-03-06T07:23:42.540925+0000","last_peered":"2024-03-06T07:23:42.540925+0000","last_clean":"2024-03-06T07:23:42.540925+0000","last_became_active":"2024-03-06T07:23:39.908214+0000","last_became_peered":"2024-03-06T07:23:39.908214+0000","last_unstale":"2024-03-06T07:23:42.540925+0000","last_undegraded":"2024-03-06T07:23:42.540925+0000","last_fullsized":"2024-03-06T07:23:42.540925+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:48.343179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888553+0000","last_change":"2024-03-06T07:23:40.907011+0000","last_active":"2024-03-06T07:23:42.888553+0000","last_peered":"2024-03-06T07:23:42.888553+0000","last_clean":"2024-03-06T07:23:42.888553+0000","last_became_active":"2024-03-06T07:23:40.906722+0000","last_became_peered":"2024-03-06T07:23:40.906722+0000","last_unstale":"2024-03-06T07:23:42.888553+0000","last_undegraded":"2024-03-06T07:23:42.888553+0000","last_fullsized":"2024-03-06T07:23:42.888553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:02.677516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905641+0000","last_change":"2024-03-06T07:23:39.901912+0000","last_active":"2024-03-06T07:23:41.905641+0000","last_peered":"2024-03-06T07:23:41.905641+0000","last_clean":"2024-03-06T07:23:41.905641+0000","last_became_active":"2024-03-06T07:23:39.901713+0000","last_became_peered":"2024-03-06T07:23:39.901713+0000","last_unstale":"2024-03-06T07:23:41.905641+0000","last_undegraded":"2024-03-06T07:23:41.905641+0000","last_fullsized":"2024-03-06T07:23:41.905641+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:46.721045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389477+0000","last_change":"2024-03-06T07:23:40.916841+0000","last_active":"2024-03-06T07:23:42.389477+0000","last_peered":"2024-03-06T07:23:42.389477+0000","last_clean":"2024-03-06T07:23:42.389477+0000","last_became_active":"2024-03-06T07:23:40.916503+0000","last_became_peered":"2024-03-06T07:23:40.916503+0000","last_unstale":"2024-03-06T07:23:42.389477+0000","last_undegraded":"2024-03-06T07:23:42.389477+0000","last_fullsized":"2024-03-06T07:23:42.389477+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:48.431072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.252541+0000","last_change":"2024-03-06T07:23:39.908817+0000","last_active":"2024-03-06T07:23:42.252541+0000","last_peered":"2024-03-06T07:23:42.252541+0000","last_clean":"2024-03-06T07:23:42.252541+0000","last_became_active":"2024-03-06T07:23:39.908523+0000","last_became_peered":"2024-03-06T07:23:39.908523+0000","last_unstale":"2024-03-06T07:23:42.252541+0000","last_undegraded":"2024-03-06T07:23:42.252541+0000","last_fullsized":"2024-03-06T07:23:42.252541+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:04.465403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:41.905243+0000","last_change":"2024-03-06T07:23:40.908212+0000","last_active":"2024-03-06T07:23:41.905243+0000","last_peered":"2024-03-06T07:23:41.905243+0000","last_clean":"2024-03-06T07:23:41.905243+0000","last_became_active":"2024-03-06T07:23:40.907940+0000","last_became_peered":"2024-03-06T07:23:40.907940+0000","last_unstale":"2024-03-06T07:23:41.905243+0000","last_undegraded":"2024-03-06T07:23:41.905243+0000","last_fullsized":"2024-03-06T07:23:41.905243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:15.993890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888621+0000","last_change":"2024-03-06T07:23:39.902253+0000","last_active":"2024-03-06T07:23:42.888621+0000","last_peered":"2024-03-06T07:23:42.888621+0000","last_clean":"2024-03-06T07:23:42.888621+0000","last_became_active":"2024-03-06T07:23:39.901938+0000","last_became_peered":"2024-03-06T07:23:39.901938+0000","last_unstale":"2024-03-06T07:23:42.888621+0000","last_undegraded":"2024-03-06T07:23:42.888621+0000","last_fullsized":"2024-03-06T07:23:42.888621+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:07.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.888207+0000","last_change":"2024-03-06T07:23:40.906482+0000","last_active":"2024-03-06T07:23:42.888207+0000","last_peered":"2024-03-06T07:23:42.888207+0000","last_clean":"2024-03-06T07:23:42.888207+0000","last_became_active":"2024-03-06T07:23:40.906239+0000","last_became_peered":"2024-03-06T07:23:40.906239+0000","last_unstale":"2024-03-06T07:23:42.888207+0000","last_undegraded":"2024-03-06T07:23:42.888207+0000","last_fullsized":"2024-03-06T07:23:42.888207+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:39.880214+0000","last_clean_scrub_stamp":"2024-03-06T07:23:39.880214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:47.996183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.389784+0000","last_change":"2024-03-06T07:23:39.908459+0000","last_active":"2024-03-06T07:23:42.389784+0000","last_peered":"2024-03-06T07:23:42.389784+0000","last_clean":"2024-03-06T07:23:42.389784+0000","last_became_active":"2024-03-06T07:23:39.908193+0000","last_became_peered":"2024-03-06T07:23:39.908193+0000","last_unstale":"2024-03-06T07:23:42.389784+0000","last_undegraded":"2024-03-06T07:23:42.389784+0000","last_fullsized":"2024-03-06T07:23:42.389784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:38.878298+0000","last_clean_scrub_stamp":"2024-03-06T07:23:38.878298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:48.343179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":216,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:42.540761+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:23:42.540761+0000","last_peered":"2024-03-06T07:23:42.540761+0000","last_clean":"2024-03-06T07:23:42.540761+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:23:42.540761+0000","last_undegraded":"2024-03-06T07:23:42.540761+0000","last_fullsized":"2024-03-06T07:23:42.540761+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":805573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":536576,"data_stored":215787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":215185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":216727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":215753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":216749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":215185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":805517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:48.563 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:48.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:48.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.705+0000 7fdc9dd19640 1 Processor -- start 2024-03-06T07:23:48.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.705+0000 7fdc9dd19640 1 -- start start 2024-03-06T07:23:48.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc9dd19640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdc981614d0 0x7fdc980d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc9dd19640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdc980d24b0 0x7fdc980b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc9dd19640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc981618b0 con 0x7fdc980d1910 2024-03-06T07:23:48.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc9dd19640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc980cf400 con 0x7fdc981614d0 2024-03-06T07:23:48.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc9dd19640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc980cf6a0 con 0x7fdc980d24b0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdc981614d0 0x7fdc980d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdc981614d0 0x7fdc980d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47148/0 (socket says 172.21.15.122:47148) 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 -- 172.21.15.122:0/2629970163 learned_addr learned my addr 172.21.15.122:0/2629970163 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc97fff640 1 --2- 172.21.15.122:0/2629970163 >> v2:172.21.15.169:3300/0 conn(0x7fdc980d24b0 0x7fdc980b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 -- 172.21.15.122:0/2629970163 >> v2:172.21.15.169:3300/0 conn(0x7fdc980d24b0 msgr2=0x7fdc980b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 --2- 172.21.15.122:0/2629970163 >> v2:172.21.15.169:3300/0 conn(0x7fdc980d24b0 0x7fdc980b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 -- 172.21.15.122:0/2629970163 >> v1:172.21.15.169:6789/0 conn(0x7fdc980d1910 legacy=0x7fdc980d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc977fe640 1 -- 172.21.15.122:0/2629970163 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc980cf980 con 0x7fdc981614d0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.706+0000 7fdc97fff640 1 --2- 172.21.15.122:0/2629970163 >> v2:172.21.15.169:3300/0 conn(0x7fdc980d24b0 0x7fdc980b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.707+0000 7fdc977fe640 1 --2- 172.21.15.122:0/2629970163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc980d4860 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fdc980b2a70 tx=0x7fdc8802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52334ac70218cd70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.707+0000 7fdc967fc640 1 -- 172.21.15.122:0/2629970163 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88034070 con 0x7fdc981614d0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.707+0000 7fdc967fc640 1 -- 172.21.15.122:0/2629970163 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88030860 con 0x7fdc981614d0 2024-03-06T07:23:48.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.707+0000 7fdc967fc640 1 -- 172.21.15.122:0/2629970163 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88030c30 con 0x7fdc981614d0 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 -- 172.21.15.122:0/2629970163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 msgr2=0x7fdc980d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/2629970163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc980d4860 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fdc980b2a70 tx=0x7fdc8802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 -- 172.21.15.122:0/2629970163 shutdown_connections 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/2629970163 >> v2:172.21.15.169:3300/0 conn(0x7fdc980d24b0 0x7fdc980b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/2629970163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc980d4860 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 -- 172.21.15.122:0/2629970163 >> 172.21.15.122:0/2629970163 conn(0x7fdc980d3d40 msgr2=0x7fdc980d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 -- 172.21.15.122:0/2629970163 shutdown_connections 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.708+0000 7fdc9dd19640 1 -- 172.21.15.122:0/2629970163 wait complete. 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.709+0000 7fdc9dd19640 1 Processor -- start 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.709+0000 7fdc9dd19640 1 -- start start 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc9dd19640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc980d1910 0x7fdc980ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc9dd19640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc980d24b0 0x7fdc980b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc9dd19640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc9806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc9dd19640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980b1370 con 0x7fdc981614d0 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc9dd19640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980b0910 con 0x7fdc980d1910 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc9dd19640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980b14f0 con 0x7fdc980d24b0 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc9806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc9806cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47156/0 (socket says 172.21.15.122:47156) 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 -- 172.21.15.122:0/1810988267 learned_addr learned my addr 172.21.15.122:0/1810988267 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc977fe640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc980d1910 0x7fdc980ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc96ffd640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc980d24b0 0x7fdc980b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 -- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc980d24b0 msgr2=0x7fdc980b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc980d24b0 0x7fdc980b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 -- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc980d1910 msgr2=0x7fdc980ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc980d1910 0x7fdc980ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 -- 172.21.15.122:0/1810988267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc980b8c90 con 0x7fdc981614d0 2024-03-06T07:23:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc97fff640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc9806cc00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000efc0 tx=0x7fdc90009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90018030 con 0x7fdc981614d0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.710+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9000cde0 con 0x7fdc981614d0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.711+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc980b8f70 con 0x7fdc981614d0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.711+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90003bb0 con 0x7fdc981614d0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.711+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc980b9530 con 0x7fdc981614d0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.711+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc98068ff0 con 0x7fdc981614d0 2024-03-06T07:23:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.712+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc9001b070 con 0x7fdc981614d0 2024-03-06T07:23:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.712+0000 7fdc7ffff640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8406b4d0 0x7fdc8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.712+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc9008ab00 con 0x7fdc981614d0 2024-03-06T07:23:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.712+0000 7fdc977fe640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8406b4d0 0x7fdc8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.713+0000 7fdc977fe640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8406b4d0 0x7fdc8406d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdc88036000 tx=0x7fdc8803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.717+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc90010040 con 0x7fdc981614d0 2024-03-06T07:23:48.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.957+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fdc980d0d30 con 0x7fdc981614d0 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.959+0000 7fdc7ffff640 1 -- 172.21.15.122:0/1810988267 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fdc90059330 con 0x7fdc981614d0 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8406b4d0 msgr2=0x7fdc8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8406b4d0 0x7fdc8406d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdc88036000 tx=0x7fdc8803b040 comp rx=0 tx=0).stop 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 msgr2=0x7fdc9806cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc9806cc00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000efc0 tx=0x7fdc90009f90 comp rx=0 tx=0).stop 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 shutdown_connections 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdc8406b4d0 0x7fdc8406d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc981614d0 0x7fdc9806cc00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdc980d24b0 0x7fdc980b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 --2- 172.21.15.122:0/1810988267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc980d1910 0x7fdc980ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 >> 172.21.15.122:0/1810988267 conn(0x7fdc980d3d40 msgr2=0x7fdc980b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 shutdown_connections 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:48.961+0000 7fdc9dd19640 1 -- 172.21.15.122:0/1810988267 wait complete. 2024-03-06T07:23:48.974 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:49.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 Processor -- start 2024-03-06T07:23:49.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 -- start start 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa80815dfe0 0x7fa8080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa8080d29c0 0x7fa8080b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8080b3430 con 0x7fa8080cf830 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8080b31e0 con 0x7fa80815dfe0 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa80e815640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8080b3d60 con 0x7fa8080d29c0 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa807fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa80815dfe0 0x7fa8080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.072+0000 7fa807fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa80815dfe0 0x7fa8080cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47162/0 (socket says 172.21.15.122:47162) 2024-03-06T07:23:49.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa807fff640 1 -- 172.21.15.122:0/1180710487 learned_addr learned my addr 172.21.15.122:0/1180710487 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa80cdb2640 1 --2- 172.21.15.122:0/1180710487 >> v2:172.21.15.169:3300/0 conn(0x7fa8080d29c0 0x7fa8080b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa807fff640 1 -- 172.21.15.122:0/1180710487 >> v2:172.21.15.169:3300/0 conn(0x7fa8080d29c0 msgr2=0x7fa8080b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa807fff640 1 --2- 172.21.15.122:0/1180710487 >> v2:172.21.15.169:3300/0 conn(0x7fa8080d29c0 0x7fa8080b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa807fff640 1 -- 172.21.15.122:0/1180710487 >> v1:172.21.15.122:6789/0 conn(0x7fa8080cf830 legacy=0x7fa8080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa807fff640 1 -- 172.21.15.122:0/1180710487 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8080b3710 con 0x7fa80815dfe0 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.073+0000 7fa806ffd640 1 -- 172.21.15.122:0/1180710487 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601913425 0 0) 0x7fa8080b3430 con 0x7fa8080cf830 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa807fff640 1 --2- 172.21.15.122:0/1180710487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080cf2f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa8080b3cd0 tx=0x7fa7f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316a258599e2df13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa806ffd640 1 -- 172.21.15.122:0/1180710487 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0034070 con 0x7fa80815dfe0 2024-03-06T07:23:49.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa806ffd640 1 -- 172.21.15.122:0/1180710487 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f0030860 con 0x7fa80815dfe0 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa806ffd640 1 -- 172.21.15.122:0/1180710487 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0030c30 con 0x7fa80815dfe0 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 -- 172.21.15.122:0/1180710487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 msgr2=0x7fa8080cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 --2- 172.21.15.122:0/1180710487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080cf2f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa8080b3cd0 tx=0x7fa7f002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 -- 172.21.15.122:0/1180710487 shutdown_connections 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 --2- 172.21.15.122:0/1180710487 >> v2:172.21.15.169:3300/0 conn(0x7fa8080d29c0 0x7fa8080b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 --2- 172.21.15.122:0/1180710487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080cf2f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 -- 172.21.15.122:0/1180710487 >> 172.21.15.122:0/1180710487 conn(0x7fa8080d3cd0 msgr2=0x7fa8080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 -- 172.21.15.122:0/1180710487 shutdown_connections 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.074+0000 7fa80e815640 1 -- 172.21.15.122:0/1180710487 wait complete. 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.075+0000 7fa80e815640 1 Processor -- start 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 -- start start 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8080cf830 0x7fa80815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8080d29c0 0x7fa8081f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080d1df0 con 0x7fa80815dfe0 2024-03-06T07:23:49.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080d1c70 con 0x7fa8080d29c0 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80e815640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80815e3c0 con 0x7fa8080cf830 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80cdb2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80cdb2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47174/0 (socket says 172.21.15.122:47174) 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa80cdb2640 1 -- 172.21.15.122:0/2990663834 learned_addr learned my addr 172.21.15.122:0/2990663834 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa8077fe640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8080d29c0 0x7fa8081f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.076+0000 7fa807fff640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8080cf830 0x7fa80815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80cdb2640 1 -- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8080cf830 msgr2=0x7fa80815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80cdb2640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8080cf830 0x7fa80815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80cdb2640 1 -- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8080d29c0 msgr2=0x7fa8081f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80cdb2640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8080d29c0 0x7fa8081f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80cdb2640 1 -- 172.21.15.122:0/2990663834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8081f7280 con 0x7fa80815dfe0 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80cdb2640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080b68d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa80000efc0 tx=0x7fa80000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800018020 con 0x7fa80815dfe0 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800014480 con 0x7fa80815dfe0 2024-03-06T07:23:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8081f7560 con 0x7fa80815dfe0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800014870 con 0x7fa80815dfe0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.077+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8081f7b20 con 0x7fa80815dfe0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.078+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80804f330 con 0x7fa80815dfe0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.078+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa800005d10 con 0x7fa80815dfe0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.079+0000 7fa804ff9640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa7e806b500 0x7fa7e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.079+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa80008aeb0 con 0x7fa80815dfe0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.079+0000 7fa807fff640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa7e806b500 0x7fa7e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.079+0000 7fa807fff640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa7e806b500 0x7fa7e806d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0036000 tx=0x7fa7f003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.083+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa800022030 con 0x7fa80815dfe0 2024-03-06T07:23:49.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.294+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8081f7d20 con 0x7fa80815dfe0 2024-03-06T07:23:49.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.295+0000 7fa804ff9640 1 -- 172.21.15.122:0/2990663834 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4989 (secure 0 0 0) 0x7fa800010040 con 0x7fa80815dfe0 2024-03-06T07:23:49.300 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:23:49.300 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:40.243216+0000","modified":"2024-03-06T07:23:42.237773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/3905558085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3905558085},{"type":"v1","addr":"172.21.15.169:6835","nonce":3905558085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":3}]} 2024-03-06T07:23:49.302 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 21 2024-03-06T07:23:49.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.296+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa7e806b500 msgr2=0x7fa7e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.296+0000 7fa80e815640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa7e806b500 0x7fa7e806d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0036000 tx=0x7fa7f003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.296+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 msgr2=0x7fa8080b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.296+0000 7fa80e815640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080b68d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa80000efc0 tx=0x7fa80000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:49.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 shutdown_connections 2024-03-06T07:23:49.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa7e806b500 0x7fa7e806d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa80815dfe0 0x7fa8080b68d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8080d29c0 0x7fa8081f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 --2- 172.21.15.122:0/2990663834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8080cf830 0x7fa80815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 >> 172.21.15.122:0/2990663834 conn(0x7fa8080d3cd0 msgr2=0x7fa8080ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 shutdown_connections 2024-03-06T07:23:49.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.297+0000 7fa80e815640 1 -- 172.21.15.122:0/2990663834 wait complete. 2024-03-06T07:23:49.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:40.243216+0000', 'modified': '2024-03-06T07:23:42.237773+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5769}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5769': {'gid': 5769, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.169:6835/3905558085', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6834', 'nonce': 3905558085}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 3905558085}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5769, 'qdb_cluster': [5769]} 2024-03-06T07:23:49.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:49.714 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:23:49.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.815+0000 7f54cf71e640 1 Processor -- start 2024-03-06T07:23:49.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.815+0000 7f54cf71e640 1 -- start start 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cf71e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54c81444c0 0x7f54c806ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cf71e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c8070fe0 0x7f54c80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cf71e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c806d370 con 0x7f54c815b2d0 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cf71e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c81448a0 con 0x7f54c81444c0 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cf71e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c806d650 con 0x7f54c8070fe0 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cd4ba640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54c81444c0 0x7f54c806ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c8070fe0 0x7f54c80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cccb9640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f54c815b2d0 0x7f54c80708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:43514/0 (socket says 172.21.15.122:43514) 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cd4ba640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54c81444c0 0x7f54c806ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:34822/0 (socket says 172.21.15.122:34822) 2024-03-06T07:23:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.816+0000 7f54cccb9640 1 -- 172.21.15.122:0/3710907894 learned_addr learned my addr 172.21.15.122:0/3710907894 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.817+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3710907894 >> v2:172.21.15.169:3300/0 conn(0x7f54c81444c0 msgr2=0x7f54c806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.817+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/3710907894 >> v2:172.21.15.169:3300/0 conn(0x7f54c81444c0 0x7f54c806ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.817+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3710907894 >> v1:172.21.15.169:6789/0 conn(0x7f54c815b2d0 legacy=0x7f54c80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.817+0000 7f54bffff640 1 -- 172.21.15.122:0/3710907894 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543412869 0 0) 0x7f54c806d370 con 0x7f54c815b2d0 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.817+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3710907894 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c806d930 con 0x7f54c8070fe0 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.817+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/3710907894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c8070fe0 0x7f54c80cda60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ed23f297f340e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54bffff640 1 -- 172.21.15.122:0/3710907894 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0035020 con 0x7f54c8070fe0 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54bffff640 1 -- 172.21.15.122:0/3710907894 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0031830 con 0x7f54c8070fe0 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54bffff640 1 -- 172.21.15.122:0/3710907894 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0031c00 con 0x7f54c8070fe0 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 -- 172.21.15.122:0/3710907894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c8070fe0 msgr2=0x7f54c80cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3710907894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c8070fe0 0x7f54c80cda60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 -- 172.21.15.122:0/3710907894 shutdown_connections 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3710907894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c8070fe0 0x7f54c80cda60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3710907894 >> v2:172.21.15.169:3300/0 conn(0x7f54c81444c0 0x7f54c806ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 -- 172.21.15.122:0/3710907894 >> 172.21.15.122:0/3710907894 conn(0x7f54c80b1c80 msgr2=0x7f54c80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 -- 172.21.15.122:0/3710907894 shutdown_connections 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.818+0000 7f54cf71e640 1 -- 172.21.15.122:0/3710907894 wait complete. 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.819+0000 7f54cf71e640 1 Processor -- start 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.819+0000 7f54cf71e640 1 -- start start 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cf71e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54c8070fe0 0x7f54c815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cf71e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c81444c0 0x7f54c81f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cf71e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 0x7f54c80cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cf71e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c806cf40 con 0x7f54c815b2d0 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cf71e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c806def0 con 0x7f54c8070fe0 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cf71e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c806d0c0 con 0x7f54c81444c0 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 0x7f54c80cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 0x7f54c80cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47188/0 (socket says 172.21.15.122:47188) 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3372201456 learned_addr learned my addr 172.21.15.122:0/3372201456 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54c8070fe0 0x7f54c815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cccb9640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c81444c0 0x7f54c81f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.820+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c81444c0 msgr2=0x7f54c81f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c81444c0 0x7f54c81f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54c8070fe0 msgr2=0x7f54c815a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54c8070fe0 0x7f54c815a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cdcbb640 1 -- 172.21.15.122:0/3372201456 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c813b650 con 0x7f54c815b2d0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54c8070fe0 0x7f54c815a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 0x7f54c80cdb60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54bdffb640 1 -- 172.21.15.122:0/3372201456 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0035070 con 0x7f54c815b2d0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54bdffb640 1 -- 172.21.15.122:0/3372201456 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0004230 con 0x7f54c815b2d0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c813b930 con 0x7f54c815b2d0 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54bdffb640 1 -- 172.21.15.122:0/3372201456 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c003a630 con 0x7f54c815b2d0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.821+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c813bef0 con 0x7f54c815b2d0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.822+0000 7f54bdffb640 1 -- 172.21.15.122:0/3372201456 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c0038020 con 0x7f54c815b2d0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.823+0000 7f54bdffb640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.823+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.823+0000 7f54bdffb640 1 -- 172.21.15.122:0/3372201456 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54c00abd00 con 0x7f54c815b2d0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.823+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54ac06b4d0 0x7f54ac06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f54c80c6710 tx=0x7f54c400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.824+0000 7f54d23e1740 1 Processor -- start 2024-03-06T07:23:49.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.824+0000 7f54d23e1740 1 -- start start 2024-03-06T07:23:49.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175792b870 0x56175792bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175792f8c0 0x561757931360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x56175795ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54d23e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617578956b0 con 0x56175795b600 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56175783ea60 con 0x56175792f8c0 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x561757891750 con 0x56175792b870 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x56175795ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x56175795ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47198/0 (socket says 172.21.15.122:47198) 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 -- 172.21.15.122:0/2615548378 learned_addr learned my addr 172.21.15.122:0/2615548378 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cccb9640 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175792f8c0 0x561757931360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175792b870 0x56175792bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 -- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175792b870 msgr2=0x56175792bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175792b870 0x56175792bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 -- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175792f8c0 msgr2=0x561757931360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175792f8c0 0x561757931360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 -- 172.21.15.122:0/2615548378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56175785cf30 con 0x56175795b600 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x56175795ba80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f54c0082660 tx=0x7f54c007a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50c5d7a296b3a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54b2ffd640 1 -- 172.21.15.122:0/2615548378 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0078830 con 0x56175795b600 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54b2ffd640 1 -- 172.21.15.122:0/2615548378 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0076ba0 con 0x56175795b600 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.825+0000 7f54b2ffd640 1 -- 172.21.15.122:0/2615548378 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c007e3a0 con 0x56175795b600 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 -- 172.21.15.122:0/2615548378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 msgr2=0x56175795ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x56175795ba80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f54c0082660 tx=0x7f54c007a5f0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 -- 172.21.15.122:0/2615548378 shutdown_connections 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x56175795ba80 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175792f8c0 0x561757931360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 --2- 172.21.15.122:0/2615548378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175792b870 0x56175792bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 -- 172.21.15.122:0/2615548378 >> 172.21.15.122:0/2615548378 conn(0x56175792d8f0 msgr2=0x56175792f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 -- 172.21.15.122:0/2615548378 shutdown_connections 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.826+0000 7f54d23e1740 1 -- 172.21.15.122:0/2615548378 wait complete. 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.827+0000 7f54d23e1740 1 -- start start 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.827+0000 7f54d23e1740 1 Processor -- start 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.827+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795b600 0x56175795d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.827+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x56175795ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.827+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175795a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561757891750 con 0x56175795ebb0 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561757851ea0 con 0x56175795b600 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617578956b0 con 0x56175795a3b0 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 10 client.0 _ll_register_callbacks cb 0x561757851870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x56175795ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x56175795ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47204/0 (socket says 172.21.15.122:47204) 2024-03-06T07:23:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 -- 172.21.15.122:0/1306895405 learned_addr learned my addr 172.21.15.122:0/1306895405 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795b600 0x56175795d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175795a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 -- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 msgr2=0x56175795a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175795a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 -- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795b600 msgr2=0x56175795d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795b600 0x56175795d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 -- 172.21.15.122:0/1306895405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561757852270 con 0x56175795ebb0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54cccb9640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x56175795ef90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f54b8000f60 tx=0x7f54b80015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba56c9674628fabf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54a7fff640 1 -- 172.21.15.122:0/1306895405 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8015650 con 0x56175795ebb0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54a7fff640 1 -- 172.21.15.122:0/1306895405 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54b8010040 con 0x56175795ebb0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54c0076ba0 con 0x56175795ebb0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54a7fff640 1 -- 172.21.15.122:0/1306895405 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54b80157f0 con 0x56175795ebb0 2024-03-06T07:23:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 10 client.5934 fetch_fsmap learned FSMap version 21 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x5617575aa2d0 con 0x56175795ebb0 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.828+0000 7f54a7fff640 1 -- 172.21.15.122:0/1306895405 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0076ba0 con 0x56175795ebb0 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54a7fff640 1 -- 172.21.15.122:0/1306895405 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 21) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f54b8015990 con 0x56175795ebb0 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54d23e1740 10 client.5934 fetch_fsmap finished waiting for FSMap version 21 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54d23e1740 10 client.5934 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54d23e1740 20 client.5934 populate_metadata read hostname 'smithi122' 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54d23e1740 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x561757956220 0x561757995130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54d23e1740 4 client.5934 mds_command: new command op to 5769 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 --> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617578a2bf0 con 0x561757956220 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x561757956220 0x561757995130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54a7fff640 10 client.5934 ms_handle_connect on v2:172.21.15.169:6834/3905558085 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.829+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x561757956220 0x561757995130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af3ad8c1c3f7dd3 server_cookie=4f2a5ebc90bb2fca in_seq=0 out_seq=0 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54a7fff640 1 -- 172.21.15.122:0/1306895405 <== mds.0 v2:172.21.15.169:6834/3905558085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5617578a2bf0 con 0x561757956220 2024-03-06T07:23:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54a7fff640 10 client.5934 handle_command_reply: tid=0 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 client.5934 shutdown 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 msgr2=0x56175795ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x56175795ef90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f54b8000f60 tx=0x7f54b80015e0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 shutdown_connections 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x561757956220 0x561757995130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175795a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x56175795ef90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54a7fff640 0 client.5934 ms_handle_reset on v2:172.21.15.169:6834/3905558085 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 --2- 172.21.15.122:0/1306895405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795b600 0x56175795d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 >> 172.21.15.122:0/1306895405 conn(0x56175792b870 msgr2=0x561757977a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 shutdown_connections 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 1 -- 172.21.15.122:0/1306895405 wait complete. 2024-03-06T07:23:49.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.830+0000 7f54d23e1740 20 client.5934 trim_cache size 0 max 16384 2024-03-06T07:23:49.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.847+0000 7f54d23e1740 1 Processor -- start 2024-03-06T07:23:49.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.847+0000 7f54d23e1740 1 -- start start 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.847+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175792fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x5617579306b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795ebb0 0x56175795ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54d23e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c81fdda0 con 0x56175795b600 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c81fd0d0 con 0x56175795ebb0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c81f5040 con 0x56175795a3b0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x5617579306b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x5617579306b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47216/0 (socket says 172.21.15.122:47216) 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 -- 172.21.15.122:0/4277269741 learned_addr learned my addr 172.21.15.122:0/4277269741 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795ebb0 0x56175795ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175792fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 -- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 msgr2=0x56175792fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175792fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 -- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795ebb0 msgr2=0x56175795ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795ebb0 0x56175795ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 -- 172.21.15.122:0/4277269741 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617576746b0 con 0x56175795b600 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175792fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54cccb9640 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x5617579306b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f54b8001eb0 tx=0x7f54b800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea4dd95390f0295 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54b27fc640 1 -- 172.21.15.122:0/4277269741 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8019020 con 0x56175795b600 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54b27fc640 1 -- 172.21.15.122:0/4277269741 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8007e50 con 0x56175795b600 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.848+0000 7f54b27fc640 1 -- 172.21.15.122:0/4277269741 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8001ae0 con 0x56175795b600 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 -- 172.21.15.122:0/4277269741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 msgr2=0x5617579306b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x5617579306b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f54b8001eb0 tx=0x7f54b800f000 comp rx=0 tx=0).stop 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 -- 172.21.15.122:0/4277269741 shutdown_connections 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795ebb0 0x56175795ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795b600 0x5617579306b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 --2- 172.21.15.122:0/4277269741 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795a3b0 0x56175792fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 -- 172.21.15.122:0/4277269741 >> 172.21.15.122:0/4277269741 conn(0x56175792b870 msgr2=0x56175795a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 -- 172.21.15.122:0/4277269741 shutdown_connections 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 -- 172.21.15.122:0/4277269741 wait complete. 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 -- start start 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.849+0000 7f54d23e1740 1 Processor -- start 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795a3b0 0x5617579e9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795b600 0x561757958950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x5617579ed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56175783ea60 con 0x56175795ebb0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c81f5640 con 0x56175795a3b0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x56175793e150 con 0x56175795b600 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54d23e1740 10 client.0 _ll_register_callbacks cb 0x7f54b8004a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x5617579ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54cdcbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x5617579ed270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47232/0 (socket says 172.21.15.122:47232) 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54cd4ba640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795a3b0 0x5617579e9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54cdcbb640 1 -- 172.21.15.122:0/921896466 learned_addr learned my addr 172.21.15.122:0/921896466 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.850+0000 7f54cccb9640 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795b600 0x561757958950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cdcbb640 1 -- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795b600 msgr2=0x561757958950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795b600 0x561757958950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cdcbb640 1 -- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795a3b0 msgr2=0x5617579e9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795a3b0 0x5617579e9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cdcbb640 1 -- 172.21.15.122:0/921896466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617579553a0 con 0x56175795ebb0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cdcbb640 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x5617579ed270 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f54c002b910 tx=0x7f54c0031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2616ac5b1cc10dac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54a7fff640 1 -- 172.21.15.122:0/921896466 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c007a300 con 0x56175795ebb0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54a7fff640 1 -- 172.21.15.122:0/921896466 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54c007e160 con 0x56175795ebb0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54b8007e50 con 0x56175795ebb0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54a7fff640 1 -- 172.21.15.122:0/921896466 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54c007e360 con 0x56175795ebb0 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 10 client.5940 fetch_fsmap learned FSMap version 21 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x561757990850 con 0x56175795ebb0 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54a7fff640 1 -- 172.21.15.122:0/921896466 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8007e50 con 0x56175795ebb0 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54a7fff640 1 -- 172.21.15.122:0/921896466 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 21) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f54c0095020 con 0x56175795ebb0 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 10 client.5940 fetch_fsmap finished waiting for FSMap version 21 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 10 client.5940 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 20 client.5940 populate_metadata read hostname 'smithi122' 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x56175795cf00 0x56175795d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 4 client.5940 mds_command: new command op to 5769 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 --> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56175789cf40 con 0x56175795cf00 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.851+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x56175795cf00 0x56175795d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54a7fff640 10 client.5940 ms_handle_connect on v2:172.21.15.169:6834/3905558085 2024-03-06T07:23:49.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54cd4ba640 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x56175795cf00 0x56175795d2e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16df42f45bccd14a server_cookie=38912e02a4839331 in_seq=0 out_seq=0 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54a7fff640 1 -- 172.21.15.122:0/921896466 <== mds.0 v2:172.21.15.169:6834/3905558085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56175789cf40 con 0x56175795cf00 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54a7fff640 10 client.5940 handle_command_reply: tid=0 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 client.5940 shutdown 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 msgr2=0x5617579ed270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x5617579ed270 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f54c002b910 tx=0x7f54c0031f90 comp rx=0 tx=0).stop 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 shutdown_connections 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56175795ebb0 0x5617579ed270 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:6834/3905558085,v1:172.21.15.169:6835/3905558085] conn(0x56175795cf00 0x56175795d2e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x56175795b600 0x561757958950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 --2- 172.21.15.122:0/921896466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56175795a3b0 0x5617579e9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54a7fff640 0 client.5940 ms_handle_reset on v2:172.21.15.169:6834/3905558085 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 >> 172.21.15.122:0/921896466 conn(0x56175792b870 msgr2=0x56175795ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 shutdown_connections 2024-03-06T07:23:49.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 1 -- 172.21.15.122:0/921896466 wait complete. 2024-03-06T07:23:49.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.852+0000 7f54d23e1740 20 client.5940 trim_cache size 0 max 16384 2024-03-06T07:23:49.862 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:49.862 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:23:49.862 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 5769, 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 8.9646394600000008, 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 21, 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 45, 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 45, 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 14.292926171 2024-03-06T07:23:49.863 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:49.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54ac06b4d0 msgr2=0x7f54ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54ac06b4d0 0x7f54ac06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f54c80c6710 tx=0x7f54c400b040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 msgr2=0x7f54c80cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 0x7f54c80cdb60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097a0 tx=0x7f54c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 shutdown_connections 2024-03-06T07:23:49.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54c815b2d0 0x7f54c80cdb60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54c81444c0 0x7f54c81f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 --2- 172.21.15.122:0/3372201456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54c8070fe0 0x7f54c815a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 >> 172.21.15.122:0/3372201456 conn(0x7f54c80b1c80 msgr2=0x7f54c8141340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 shutdown_connections 2024-03-06T07:23:49.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.853+0000 7f54cf71e640 1 -- 172.21.15.122:0/3372201456 wait complete. 2024-03-06T07:23:49.866 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:49.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.964+0000 7f45379bf640 1 Processor -- start 2024-03-06T07:23:49.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.964+0000 7f45379bf640 1 -- start start 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f45379bf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45300cce40 0x7f453015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f45379bf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4530071f40 con 0x7f45301515e0 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f45379bf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4530070a70 con 0x7f45300cd7a0 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f45379bf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45300cd2f0 con 0x7f45300cce40 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f4534f5a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45300cce40 0x7f453015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f453575b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f45301515e0 0x7f4530071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41322/0 (socket says 172.21.15.122:41322) 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f453575b640 1 -- 172.21.15.122:0/1752981343 learned_addr learned my addr 172.21.15.122:0/1752981343 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f4534f5a640 1 -- 172.21.15.122:0/1752981343 >> v1:172.21.15.169:6789/0 conn(0x7f45300cd7a0 legacy=0x7f45300cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f4534f5a640 1 -- 172.21.15.122:0/1752981343 >> v1:172.21.15.122:6789/0 conn(0x7f45301515e0 legacy=0x7f4530071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.965+0000 7f4534f5a640 1 -- 172.21.15.122:0/1752981343 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530070d50 con 0x7f45300cce40 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f4534f5a640 1 --2- 172.21.15.122:0/1752981343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cce40 0x7f453015b430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f452000a940 tx=0x7f452002ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580f3d5940cde94c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f4527fff640 1 -- 172.21.15.122:0/1752981343 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452003a070 con 0x7f45300cce40 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f4527fff640 1 -- 172.21.15.122:0/1752981343 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452002dbe0 con 0x7f45300cce40 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f4527fff640 1 -- 172.21.15.122:0/1752981343 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452000b040 con 0x7f45300cce40 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f45379bf640 1 -- 172.21.15.122:0/1752981343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cce40 msgr2=0x7f453015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f45379bf640 1 --2- 172.21.15.122:0/1752981343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cce40 0x7f453015b430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f452000a940 tx=0x7f452002ce30 comp rx=0 tx=0).stop 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f45379bf640 1 -- 172.21.15.122:0/1752981343 shutdown_connections 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.966+0000 7f45379bf640 1 --2- 172.21.15.122:0/1752981343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cce40 0x7f453015b430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.967+0000 7f45379bf640 1 -- 172.21.15.122:0/1752981343 >> 172.21.15.122:0/1752981343 conn(0x7f45300cf8a0 msgr2=0x7f45300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.967+0000 7f45379bf640 1 -- 172.21.15.122:0/1752981343 shutdown_connections 2024-03-06T07:23:49.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.967+0000 7f45379bf640 1 -- 172.21.15.122:0/1752981343 wait complete. 2024-03-06T07:23:49.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 Processor -- start 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 -- start start 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45300cce40 0x7f4530145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 0x7f45301559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45301515e0 0x7f4530155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45300cc3a0 con 0x7f45300cd7a0 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4530071310 con 0x7f45300cce40 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.968+0000 7f45379bf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45300cc520 con 0x7f45301515e0 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 0x7f45301559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 0x7f45301559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47240/0 (socket says 172.21.15.122:47240) 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 -- 172.21.15.122:0/1982865796 learned_addr learned my addr 172.21.15.122:0/1982865796 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f453575b640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45300cce40 0x7f4530145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4535f5c640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45301515e0 0x7f4530155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 -- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45301515e0 msgr2=0x7f4530155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45301515e0 0x7f4530155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 -- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45300cce40 msgr2=0x7f4530145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45300cce40 0x7f4530145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 -- 172.21.15.122:0/1982865796 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530145840 con 0x7f45300cd7a0 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4534f5a640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 0x7f45301559b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4520038040 tx=0x7f45200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520045070 con 0x7f45300cd7a0 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452003e750 con 0x7f45300cd7a0 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452003a070 con 0x7f45300cd7a0 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453014d5b0 con 0x7f45300cd7a0 2024-03-06T07:23:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.969+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4530156800 con 0x7f45300cd7a0 2024-03-06T07:23:49.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.970+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4530068840 con 0x7f45300cd7a0 2024-03-06T07:23:49.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.971+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4520048070 con 0x7f45300cd7a0 2024-03-06T07:23:49.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.971+0000 7f4525ffb640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f451406b4d0 0x7f451406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.971+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45200abc90 con 0x7f45300cd7a0 2024-03-06T07:23:49.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.971+0000 7f453575b640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f451406b4d0 0x7f451406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.972+0000 7f453575b640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f451406b4d0 0x7f451406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f452c000f60 tx=0x7f452c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:49.976+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4530145840 con 0x7f45300cd7a0 2024-03-06T07:23:50.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.184+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f453015a4f0 con 0x7f45300cd7a0 2024-03-06T07:23:50.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.184+0000 7f4525ffb640 1 -- 172.21.15.122:0/1982865796 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+73 (secure 0 0 0) 0x7f453015a4f0 con 0x7f45300cd7a0 2024-03-06T07:23:50.187 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f451406b4d0 msgr2=0x7f451406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f451406b4d0 0x7f451406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f452c000f60 tx=0x7f452c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 msgr2=0x7f45301559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 0x7f45301559b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4520038040 tx=0x7f45200046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 shutdown_connections 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f451406b4d0 0x7f451406d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45301515e0 0x7f4530155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45300cd7a0 0x7f45301559b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 --2- 172.21.15.122:0/1982865796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45300cce40 0x7f4530145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.186+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 >> 172.21.15.122:0/1982865796 conn(0x7f45300cf8a0 msgr2=0x7f45300b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.187+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 shutdown_connections 2024-03-06T07:23:50.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:50.187+0000 7f45379bf640 1 -- 172.21.15.122:0/1982865796 wait complete. 2024-03-06T07:23:50.199 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:50.199 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:50.199 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:23:50.200 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:50.200 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:50.200 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:50.200 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:50.200 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:50.200 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:50.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.200 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42611sec preferred_lft 42611sec 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.217 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:50.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.218 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:50.283 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:23:50.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.284 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:23:50.339 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:23:50.339 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:50.339 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:50.339 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:50.339 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:50.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.340 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:23:50.340 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:23:50.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:50.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:50.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:50.340 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:23:50.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.434 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:23:50.448 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:50.448 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:50.449 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:50.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:23:50.515 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:50.516 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:50.516 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:50.572 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:50.572 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:50.626 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:23:50.694 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:50.750 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:23:50.750 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:23:50.750 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:23:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:50.752 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:23:50.753 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:50.753 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:50.753 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:50.753 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:50.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.753 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:23:50.805 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:50.805 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:50.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:23:50.875 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:23:50.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:23:50.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:50.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:50.935 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:23:50.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:23:50.932+0000 7f9ca4b83480 -1 init, newargv = 0x5574722136e0 newargc=15 2024-03-06T07:23:50.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[56162]: starting ceph client 2024-03-06T07:23:50.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[56162]: starting fuse 2024-03-06T07:23:50.949 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:23:50.949 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:23:50.950 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:50.951 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:23:51.069 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.56162.asok 2024-03-06T07:23:51.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:51.080 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56162.asok 2024-03-06T07:23:51.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56162.asok status 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "56162", 2024-03-06T07:23:51.216 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 5955, 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 5955 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:51.217 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 2911356589 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 2911356589 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.5955 192.168.0.1:0/2911356589", 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/2911356589", 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 21, 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 45, 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:51.218 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:23:51.219 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:23:51.219 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.225 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:23:51.225 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:23:51.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:51.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:23:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.447+0000 7efcfb79a640 1 Processor -- start 2024-03-06T07:23:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.447+0000 7efcfb79a640 1 -- start start 2024-03-06T07:23:51.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcfb79a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf406afb0 0x7efcf406b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcfb79a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efcf4163870 0x7efcf4165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcfb79a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf41459b0 con 0x7efcf41455d0 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcfb79a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf40cfdf0 con 0x7efcf4163870 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcfb79a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406b930 con 0x7efcf406afb0 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcf9536640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7efcf41455d0 0x7efcf4072480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:43542/0 (socket says 172.21.15.122:43542) 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcf8d35640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf406afb0 0x7efcf406b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcf9536640 1 -- 172.21.15.122:0/772390800 learned_addr learned my addr 172.21.15.122:0/772390800 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.448+0000 7efcf9d37640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efcf4163870 0x7efcf4165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.449+0000 7efcf8d35640 1 -- 172.21.15.122:0/772390800 >> v2:172.21.15.169:3300/0 conn(0x7efcf4163870 msgr2=0x7efcf4165c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.449+0000 7efcf8d35640 1 --2- 172.21.15.122:0/772390800 >> v2:172.21.15.169:3300/0 conn(0x7efcf4163870 0x7efcf4165c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.449+0000 7efcf8d35640 1 -- 172.21.15.122:0/772390800 >> v1:172.21.15.169:6789/0 conn(0x7efcf41455d0 legacy=0x7efcf4072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.449+0000 7efcebfff640 1 -- 172.21.15.122:0/772390800 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206015115 0 0) 0x7efcf41459b0 con 0x7efcf41455d0 2024-03-06T07:23:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.449+0000 7efcf8d35640 1 -- 172.21.15.122:0/772390800 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf40cf1e0 con 0x7efcf406afb0 2024-03-06T07:23:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.449+0000 7efcf9d37640 1 --2- 172.21.15.122:0/772390800 >> v2:172.21.15.169:3300/0 conn(0x7efcf4163870 0x7efcf4165c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcf8d35640 1 --2- 172.21.15.122:0/772390800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf406afb0 0x7efcf406b3f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efce40097a0 tx=0x7efce402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a91142adb41ec943 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcebfff640 1 -- 172.21.15.122:0/772390800 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4035020 con 0x7efcf406afb0 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcebfff640 1 -- 172.21.15.122:0/772390800 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce4031830 con 0x7efcf406afb0 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcebfff640 1 -- 172.21.15.122:0/772390800 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4031c00 con 0x7efcf406afb0 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 -- 172.21.15.122:0/772390800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf406afb0 msgr2=0x7efcf406b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 --2- 172.21.15.122:0/772390800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf406afb0 0x7efcf406b3f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efce40097a0 tx=0x7efce402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 -- 172.21.15.122:0/772390800 shutdown_connections 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 --2- 172.21.15.122:0/772390800 >> v2:172.21.15.169:3300/0 conn(0x7efcf4163870 0x7efcf4165c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 --2- 172.21.15.122:0/772390800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf406afb0 0x7efcf406b3f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 -- 172.21.15.122:0/772390800 >> 172.21.15.122:0/772390800 conn(0x7efcf40b2660 msgr2=0x7efcf40b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 -- 172.21.15.122:0/772390800 shutdown_connections 2024-03-06T07:23:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.450+0000 7efcfb79a640 1 -- 172.21.15.122:0/772390800 wait complete. 2024-03-06T07:23:51.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.451+0000 7efcfb79a640 1 Processor -- start 2024-03-06T07:23:51.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.451+0000 7efcfb79a640 1 -- start start 2024-03-06T07:23:51.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcfb79a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 0x7efcf415af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcfb79a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf41455d0 0x7efcf4139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcfb79a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcf4163870 0x7efcf4134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcfb79a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf40cfa80 con 0x7efcf406afb0 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcfb79a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf40cf7a0 con 0x7efcf4163870 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcfb79a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf40b3400 con 0x7efcf41455d0 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcf9536640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 0x7efcf415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcf9536640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 0x7efcf415af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47242/0 (socket says 172.21.15.122:47242) 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcf9536640 1 -- 172.21.15.122:0/1633907655 learned_addr learned my addr 172.21.15.122:0/1633907655 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcf9d37640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcf4163870 0x7efcf4134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.452+0000 7efcf8d35640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf41455d0 0x7efcf4139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcf9536640 1 -- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf41455d0 msgr2=0x7efcf4139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcf9536640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf41455d0 0x7efcf4139460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcf9536640 1 -- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcf4163870 msgr2=0x7efcf4134520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcf9536640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcf4163870 0x7efcf4134520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcf9536640 1 -- 172.21.15.122:0/1633907655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf4139f20 con 0x7efcf406afb0 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcf9536640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 0x7efcf415af10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c890 tx=0x7efcf000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0018070 con 0x7efcf406afb0 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf000a640 con 0x7efcf406afb0 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf413a200 con 0x7efcf406afb0 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf000aa30 con 0x7efcf406afb0 2024-03-06T07:23:51.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.453+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf4134000 con 0x7efcf406afb0 2024-03-06T07:23:51.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.454+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4073390 con 0x7efcf406afb0 2024-03-06T07:23:51.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.454+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcf001b070 con 0x7efcf406afb0 2024-03-06T07:23:51.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.455+0000 7efce9ffb640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcd806b4d0 0x7efcd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.455+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcf0089840 con 0x7efcf406afb0 2024-03-06T07:23:51.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.455+0000 7efcf8d35640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcd806b4d0 0x7efcd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.455+0000 7efcf8d35640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcd806b4d0 0x7efcd806d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efce4036000 tx=0x7efce4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.459+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcf0058070 con 0x7efcf406afb0 2024-03-06T07:23:51.631 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.629+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7efcf40cedc0 con 0x7efcf406afb0 2024-03-06T07:23:51.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.632+0000 7efce9ffb640 1 -- 172.21.15.122:0/1633907655 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7efcf005c110 con 0x7efcf406afb0 2024-03-06T07:23:51.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcd806b4d0 msgr2=0x7efcd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcd806b4d0 0x7efcd806d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efce4036000 tx=0x7efce4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 msgr2=0x7efcf415af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 0x7efcf415af10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c890 tx=0x7efcf000cd50 comp rx=0 tx=0).stop 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 shutdown_connections 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcd806b4d0 0x7efcd806d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efcf4163870 0x7efcf4134520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efcf41455d0 0x7efcf4139460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 --2- 172.21.15.122:0/1633907655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcf406afb0 0x7efcf415af10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 >> 172.21.15.122:0/1633907655 conn(0x7efcf40b2660 msgr2=0x7efcf406f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 shutdown_connections 2024-03-06T07:23:51.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:23:51.634+0000 7efcfb79a640 1 -- 172.21.15.122:0/1633907655 wait complete. 2024-03-06T07:23:53.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:53.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:01.648 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:01.648 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:01.667 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> import uuid 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> n = 51 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = False 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> if 0 > 0: 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = True 2024-03-06T07:24:01.668 DEBUG:teuthology.orchestra.run.smithi122:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> dpath = os.path.dirname(path) 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> try: 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(n): 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> f.write(f"{i}") 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:01.669 DEBUG:teuthology.orchestra.run.smithi122:> f.flush() 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(f.fileno()) 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> os.unlink(fpath) 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> if create_hard_links: 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> for j in range(0): 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:24:01.670 DEBUG:teuthology.orchestra.run.smithi122:> finally: 2024-03-06T07:24:01.671 DEBUG:teuthology.orchestra.run.smithi122:> os.close(dirfd) 2024-03-06T07:24:01.671 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:24:01.876 INFO:teuthology.orchestra.run.smithi122.stderr:Traceback (most recent call last): 2024-03-06T07:24:01.876 INFO:teuthology.orchestra.run.smithi122.stderr: File "", line 20, in 2024-03-06T07:24:01.876 INFO:teuthology.orchestra.run.smithi122.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-03-06T07:24:01.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:01 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:24:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:01.886 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:01.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:01.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:02.034 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:02.034 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:02.034 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:02.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:02.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.188+0000 7ffac741c640 1 Processor -- start 2024-03-06T07:24:02.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.188+0000 7ffac741c640 1 -- start start 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac741c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac015bb20 0x7ffac00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac741c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffac00cd8c0 0x7ffac00d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac741c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0161980 con 0x7ffac00cfff0 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac741c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac0161c60 con 0x7ffac00cd8c0 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac741c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac0161f40 con 0x7ffac015bb20 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac59b9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffac00cd8c0 0x7ffac00d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac49b7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffac00cfff0 0x7ffac00ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42520/0 (socket says 172.21.15.122:42520) 2024-03-06T07:24:02.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac49b7640 1 -- 172.21.15.122:0/1489679518 learned_addr learned my addr 172.21.15.122:0/1489679518 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.189+0000 7ffac51b8640 1 --2- 172.21.15.122:0/1489679518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac015bb20 0x7ffac00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.190+0000 7ffac59b9640 1 -- 172.21.15.122:0/1489679518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac015bb20 msgr2=0x7ffac00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.190+0000 7ffac59b9640 1 --2- 172.21.15.122:0/1489679518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac015bb20 0x7ffac00d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.190+0000 7ffac59b9640 1 -- 172.21.15.122:0/1489679518 >> v1:172.21.15.122:6789/0 conn(0x7ffac00cfff0 legacy=0x7ffac00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.190+0000 7ffab7fff640 1 -- 172.21.15.122:0/1489679518 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49333856 0 0) 0x7ffac0161980 con 0x7ffac00cfff0 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.190+0000 7ffac59b9640 1 -- 172.21.15.122:0/1489679518 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0162220 con 0x7ffac00cd8c0 2024-03-06T07:24:02.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.190+0000 7ffac59b9640 1 --2- 172.21.15.122:0/1489679518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cd8c0 0x7ffac00d0470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7856fc784c586804 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffab7fff640 1 -- 172.21.15.122:0/1489679518 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80315a0 con 0x7ffac00cd8c0 2024-03-06T07:24:02.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffab7fff640 1 -- 172.21.15.122:0/1489679518 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8037070 con 0x7ffac00cd8c0 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffab7fff640 1 -- 172.21.15.122:0/1489679518 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8031960 con 0x7ffac00cd8c0 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 -- 172.21.15.122:0/1489679518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cd8c0 msgr2=0x7ffac00d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 --2- 172.21.15.122:0/1489679518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cd8c0 0x7ffac00d0470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 -- 172.21.15.122:0/1489679518 shutdown_connections 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 --2- 172.21.15.122:0/1489679518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cd8c0 0x7ffac00d0470 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 --2- 172.21.15.122:0/1489679518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac015bb20 0x7ffac00d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 -- 172.21.15.122:0/1489679518 >> 172.21.15.122:0/1489679518 conn(0x7ffac009a9f0 msgr2=0x7ffac009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 -- 172.21.15.122:0/1489679518 shutdown_connections 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.191+0000 7ffac741c640 1 -- 172.21.15.122:0/1489679518 wait complete. 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.192+0000 7ffac741c640 1 Processor -- start 2024-03-06T07:24:02.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.192+0000 7ffac741c640 1 -- start start 2024-03-06T07:24:02.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac741c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac00cd8c0 0x7ffac0154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac741c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 0x7ffac006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac741c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffac015bb20 0x7ffac015a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac741c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d24b0 con 0x7ffac00cfff0 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac741c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d2330 con 0x7ffac015bb20 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac741c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d1980 con 0x7ffac00cd8c0 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 0x7ffac006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 0x7ffac006ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45966/0 (socket says 172.21.15.122:45966) 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 -- 172.21.15.122:0/3553057129 learned_addr learned my addr 172.21.15.122:0/3553057129 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac59b9640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffac015bb20 0x7ffac015a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac51b8640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac00cd8c0 0x7ffac0154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 -- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac00cd8c0 msgr2=0x7ffac0154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac00cd8c0 0x7ffac0154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 -- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffac015bb20 msgr2=0x7ffac015a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffac015bb20 0x7ffac015a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac49b7640 1 -- 172.21.15.122:0/3553057129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00c6180 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac59b9640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffac015bb20 0x7ffac015a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.193+0000 7ffac51b8640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac00cd8c0 0x7ffac0154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.194+0000 7ffac49b7640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 0x7ffac006ceb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c500 tx=0x7ffab000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.194+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0018070 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.194+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab0007580 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.194+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac00c6460 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.194+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0004a20 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.194+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac00c6a20 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.195+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac004f330 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.195+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffab001b070 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.196+0000 7ffab5ffb640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaa406b4d0 0x7ffaa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.196+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffab00894a0 con 0x7ffac00cfff0 2024-03-06T07:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.196+0000 7ffac51b8640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaa406b4d0 0x7ffaa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.196+0000 7ffac51b8640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaa406b4d0 0x7ffaa406d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009fd0 tx=0x7ffabc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.200+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffab00583b0 con 0x7ffac00cfff0 2024-03-06T07:24:02.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.372+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffac00b1820 con 0x7ffac00cfff0 2024-03-06T07:24:02.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.373+0000 7ffab5ffb640 1 -- 172.21.15.122:0/3553057129 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16613 (secure 0 0 0) 0x7ffab0057cb0 con 0x7ffac00cfff0 2024-03-06T07:24:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:02.376 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":45,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:41.897227+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:02.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaa406b4d0 msgr2=0x7ffaa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaa406b4d0 0x7ffaa406d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009fd0 tx=0x7ffabc00b040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 msgr2=0x7ffac006ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 0x7ffac006ceb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c500 tx=0x7ffab000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 shutdown_connections 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffaa406b4d0 0x7ffaa406d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffac015bb20 0x7ffac015a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffac00cfff0 0x7ffac006ceb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 --2- 172.21.15.122:0/3553057129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffac00cd8c0 0x7ffac0154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 >> 172.21.15.122:0/3553057129 conn(0x7ffac009a9f0 msgr2=0x7ffac006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 shutdown_connections 2024-03-06T07:24:02.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.375+0000 7ffac741c640 1 -- 172.21.15.122:0/3553057129 wait complete. 2024-03-06T07:24:02.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:38.463254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:39.205152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:02.389 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 Processor -- start 2024-03-06T07:24:02.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 -- start start 2024-03-06T07:24:02.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85940cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f85940712d0 0x7f8594071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 0x7f859415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85940cf610 con 0x7f859415de60 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8594071c70 con 0x7f85940712d0 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.491+0000 7f859a250640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85940b3320 con 0x7f859415a220 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f8593fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 0x7f859415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85940cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f8592ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f85940712d0 0x7f8594071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85940cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:60258/0 (socket says 172.21.15.122:60258) 2024-03-06T07:24:02.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 -- 172.21.15.122:0/271975601 learned_addr learned my addr 172.21.15.122:0/271975601 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f8593fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 0x7f859415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45972/0 (socket says 172.21.15.122:45972) 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 -- 172.21.15.122:0/271975601 >> v2:172.21.15.169:3300/0 conn(0x7f85940712d0 msgr2=0x7f8594071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 --2- 172.21.15.122:0/271975601 >> v2:172.21.15.169:3300/0 conn(0x7f85940712d0 0x7f8594071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 -- 172.21.15.122:0/271975601 >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 msgr2=0x7f859415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 --2- 172.21.15.122:0/271975601 >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 0x7f859415e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f85937fe640 1 -- 172.21.15.122:0/271975601 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85940728d0 con 0x7f859415a220 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.492+0000 7f8593fff640 1 --2- 172.21.15.122:0/271975601 >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 0x7f859415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f85937fe640 1 --2- 172.21.15.122:0/271975601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85940cf0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f859406eac0 tx=0x7f858402da90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a3ad8d5d1f9a36c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f85927fc640 1 -- 172.21.15.122:0/271975601 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858402f850 con 0x7f859415a220 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f85927fc640 1 -- 172.21.15.122:0/271975601 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858402f9f0 con 0x7f859415a220 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f85927fc640 1 -- 172.21.15.122:0/271975601 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858402fdc0 con 0x7f859415a220 2024-03-06T07:24:02.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 -- 172.21.15.122:0/271975601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 msgr2=0x7f85940cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 --2- 172.21.15.122:0/271975601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85940cf0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f859406eac0 tx=0x7f858402da90 comp rx=0 tx=0).stop 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 -- 172.21.15.122:0/271975601 shutdown_connections 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 --2- 172.21.15.122:0/271975601 >> v2:172.21.15.122:3300/0 conn(0x7f859415de60 0x7f859415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 --2- 172.21.15.122:0/271975601 >> v2:172.21.15.169:3300/0 conn(0x7f85940712d0 0x7f8594071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 --2- 172.21.15.122:0/271975601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85940cf0d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.493+0000 7f859a250640 1 -- 172.21.15.122:0/271975601 >> 172.21.15.122:0/271975601 conn(0x7f859406b190 msgr2=0x7f859406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.494+0000 7f859a250640 1 -- 172.21.15.122:0/271975601 shutdown_connections 2024-03-06T07:24:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.494+0000 7f859a250640 1 -- 172.21.15.122:0/271975601 wait complete. 2024-03-06T07:24:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.495+0000 7f859a250640 1 Processor -- start 2024-03-06T07:24:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.495+0000 7f859a250640 1 -- start start 2024-03-06T07:24:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.495+0000 7f859a250640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 0x7f85941475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.495+0000 7f859a250640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85941f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.495+0000 7f859a250640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f859415de60 0x7f85940b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.495+0000 7f859a250640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594072030 con 0x7f85940712d0 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f859a250640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594072bb0 con 0x7f859415de60 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f859a250640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8594072530 con 0x7f859415a220 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 0x7f85941475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 0x7f85941475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45974/0 (socket says 172.21.15.122:45974) 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 -- 172.21.15.122:0/1717822887 learned_addr learned my addr 172.21.15.122:0/1717822887 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f8593fff640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f859415de60 0x7f85940b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f8592ffd640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85941f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 -- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 msgr2=0x7f85941f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85941f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 -- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f859415de60 msgr2=0x7f85940b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f859415de60 0x7f85940b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 -- 172.21.15.122:0/1717822887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859414a0a0 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f8592ffd640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85941f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f85937fe640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 0x7f85941475d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8584009960 tx=0x7f8584004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584049070 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8584039890 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859414a380 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584039c80 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.496+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859414a940 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.497+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85940732b0 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.498+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8584043390 con 0x7f85940712d0 2024-03-06T07:24:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.498+0000 7f857bfff640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f858006b4d0 0x7f858006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.498+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85840adb30 con 0x7f85940712d0 2024-03-06T07:24:02.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.498+0000 7f8592ffd640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f858006b4d0 0x7f858006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.499+0000 7f8592ffd640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f858006b4d0 0x7f858006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8594158a00 tx=0x7f8588008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.502+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8584039e20 con 0x7f85940712d0 2024-03-06T07:24:02.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.673+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f859406b570 con 0x7f85940712d0 2024-03-06T07:24:02.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.673+0000 7f857bfff640 1 -- 172.21.15.122:0/1717822887 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f858407ca40 con 0x7f85940712d0 2024-03-06T07:24:02.676 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f858006b4d0 msgr2=0x7f858006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f858006b4d0 0x7f858006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8594158a00 tx=0x7f8588008040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 msgr2=0x7f85941475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 0x7f85941475d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8584009960 tx=0x7f8584004730 comp rx=0 tx=0).stop 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 shutdown_connections 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f858006b4d0 0x7f858006d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f859415de60 0x7f85940b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f859415a220 0x7f85941f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.675+0000 7f859a250640 1 --2- 172.21.15.122:0/1717822887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85940712d0 0x7f85941475d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.676+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 >> 172.21.15.122:0/1717822887 conn(0x7f859406b190 msgr2=0x7f8594159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.676+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 shutdown_connections 2024-03-06T07:24:02.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.676+0000 7f859a250640 1 -- 172.21.15.122:0/1717822887 wait complete. 2024-03-06T07:24:02.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.789+0000 7f8c16ff6640 1 Processor -- start 2024-03-06T07:24:02.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.789+0000 7f8c16ff6640 1 -- start start 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c16ff6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8c1015bb40 0x7f8c100b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c16ff6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1015ff90 con 0x7f8c100cfa90 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c16ff6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c10160270 con 0x7f8c1015f7d0 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c16ff6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1006c910 con 0x7f8c1015bb40 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c15593640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8c100cfa90 0x7f8c100cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42526/0 (socket says 172.21.15.122:42526) 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c15593640 1 -- 172.21.15.122:0/4016875950 learned_addr learned my addr 172.21.15.122:0/4016875950 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c14d92640 1 --2- 172.21.15.122:0/4016875950 >> v2:172.21.15.169:3300/0 conn(0x7f8c1015bb40 0x7f8c100b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c0f7fe640 1 -- 172.21.15.122:0/4016875950 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278279567 0 0) 0x7f8c1015ff90 con 0x7f8c100cfa90 2024-03-06T07:24:02.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c0f7fe640 1 -- 172.21.15.122:0/4016875950 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c00003520 con 0x7f8c100cfa90 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c14d92640 1 -- 172.21.15.122:0/4016875950 >> v1:172.21.15.169:6789/0 conn(0x7f8c1015f7d0 legacy=0x7f8c1006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c14d92640 1 -- 172.21.15.122:0/4016875950 >> v1:172.21.15.122:6789/0 conn(0x7f8c100cfa90 legacy=0x7f8c100cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.790+0000 7f8c14d92640 1 -- 172.21.15.122:0/4016875950 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10160550 con 0x7f8c1015bb40 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.791+0000 7f8c14d92640 1 --2- 172.21.15.122:0/4016875950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c100b4970 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8c1006ca90 tx=0x7f8c0402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5beb37afefa89c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.791+0000 7f8c0f7fe640 1 -- 172.21.15.122:0/4016875950 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04034070 con 0x7f8c1015bb40 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.791+0000 7f8c0f7fe640 1 -- 172.21.15.122:0/4016875950 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04030610 con 0x7f8c1015bb40 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.791+0000 7f8c0f7fe640 1 -- 172.21.15.122:0/4016875950 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040309e0 con 0x7f8c1015bb40 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 -- 172.21.15.122:0/4016875950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 msgr2=0x7f8c100b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/4016875950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c100b4970 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8c1006ca90 tx=0x7f8c0402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 -- 172.21.15.122:0/4016875950 shutdown_connections 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/4016875950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c100b4970 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 -- 172.21.15.122:0/4016875950 >> 172.21.15.122:0/4016875950 conn(0x7f8c1006b190 msgr2=0x7f8c1006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 -- 172.21.15.122:0/4016875950 shutdown_connections 2024-03-06T07:24:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.792+0000 7f8c16ff6640 1 -- 172.21.15.122:0/4016875950 wait complete. 2024-03-06T07:24:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.793+0000 7f8c16ff6640 1 Processor -- start 2024-03-06T07:24:02.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.793+0000 7f8c16ff6640 1 -- start start 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c100cfa90 0x7f8c10159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c101504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 0x7f8c10157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1006d790 con 0x7f8c1015f7d0 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1015fc30 con 0x7f8c1015bb40 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1015bf20 con 0x7f8c100cfa90 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 0x7f8c10157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 0x7f8c10157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45976/0 (socket says 172.21.15.122:45976) 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 -- 172.21.15.122:0/3254889282 learned_addr learned my addr 172.21.15.122:0/3254889282 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c14d92640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c100cfa90 0x7f8c10159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c0ffff640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c101504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 -- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c100cfa90 msgr2=0x7f8c10159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c100cfa90 0x7f8c10159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 -- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 msgr2=0x7f8c101504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c101504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 -- 172.21.15.122:0/3254889282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10158570 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c15593640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 0x7f8c10157b20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800cb40 tx=0x7f8c0800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08017070 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0800aca0 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c10158850 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08012620 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.794+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c10158e10 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.795+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c100732b0 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.796+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c0801a070 con 0x7f8c1015f7d0 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.796+0000 7f8c0d7fa640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf406b4d0 0x7f8bf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.796+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c08089f60 con 0x7f8c1015f7d0 2024-03-06T07:24:02.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.796+0000 7f8c14d92640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf406b4d0 0x7f8bf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.797+0000 7f8c14d92640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf406b4d0 0x7f8bf406d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8c04036000 tx=0x7f8c0403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.800+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c08058790 con 0x7f8c1015f7d0 2024-03-06T07:24:02.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.970+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c1006e130 con 0x7f8c1015f7d0 2024-03-06T07:24:02.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.970+0000 7f8c0d7fa640 1 -- 172.21.15.122:0/3254889282 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c0805c830 con 0x7f8c1015f7d0 2024-03-06T07:24:02.972 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf406b4d0 msgr2=0x7f8bf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf406b4d0 0x7f8bf406d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8c04036000 tx=0x7f8c0403b040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 msgr2=0x7f8c10157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 0x7f8c10157b20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800cb40 tx=0x7f8c0800a060 comp rx=0 tx=0).stop 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 shutdown_connections 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf406b4d0 0x7f8bf406d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c1015f7d0 0x7f8c10157b20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c1015bb40 0x7f8c101504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 --2- 172.21.15.122:0/3254889282 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c100cfa90 0x7f8c10159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 >> 172.21.15.122:0/3254889282 conn(0x7f8c1006b190 msgr2=0x7f8c101564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 shutdown_connections 2024-03-06T07:24:02.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:02.972+0000 7f8c16ff6640 1 -- 172.21.15.122:0/3254889282 wait complete. 2024-03-06T07:24:02.985 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:03.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 Processor -- start 2024-03-06T07:24:03.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 -- start start 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f137c0d4860 0x7f137c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f137c0b0e30 0x7f137c0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f137c1618b0 con 0x7f137c1614d0 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137c0d4cf0 con 0x7f137c0b0e30 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.086+0000 7f1382b02640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137c0cf400 con 0x7f137c0d4860 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f137c0b0e30 0x7f137c0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f137c0b0e30 0x7f137c0b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45990/0 (socket says 172.21.15.122:45990) 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 -- 172.21.15.122:0/1493884637 learned_addr learned my addr 172.21.15.122:0/1493884637 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f137bfff640 1 --2- 172.21.15.122:0/1493884637 >> v2:172.21.15.169:3300/0 conn(0x7f137c0d4860 0x7f137c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 -- 172.21.15.122:0/1493884637 >> v2:172.21.15.169:3300/0 conn(0x7f137c0d4860 msgr2=0x7f137c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 --2- 172.21.15.122:0/1493884637 >> v2:172.21.15.169:3300/0 conn(0x7f137c0d4860 0x7f137c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 -- 172.21.15.122:0/1493884637 >> v1:172.21.15.169:6789/0 conn(0x7f137c1614d0 legacy=0x7f137c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.087+0000 7f138109f640 1 -- 172.21.15.122:0/1493884637 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c0cf6a0 con 0x7f137c0b0e30 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f138109f640 1 --2- 172.21.15.122:0/1493884637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0b0e30 0x7f137c0b1260 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f137c0ce550 tx=0x7f137402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee0e54b025a1731 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f137b7fe640 1 -- 172.21.15.122:0/1493884637 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137402cb20 con 0x7f137c0b0e30 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f137b7fe640 1 -- 172.21.15.122:0/1493884637 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137402ccc0 con 0x7f137c0b0e30 2024-03-06T07:24:03.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f137b7fe640 1 -- 172.21.15.122:0/1493884637 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1374034610 con 0x7f137c0b0e30 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 -- 172.21.15.122:0/1493884637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0b0e30 msgr2=0x7f137c0b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 --2- 172.21.15.122:0/1493884637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0b0e30 0x7f137c0b1260 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f137c0ce550 tx=0x7f137402bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 -- 172.21.15.122:0/1493884637 shutdown_connections 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 --2- 172.21.15.122:0/1493884637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0b0e30 0x7f137c0b1260 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 --2- 172.21.15.122:0/1493884637 >> v2:172.21.15.169:3300/0 conn(0x7f137c0d4860 0x7f137c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 -- 172.21.15.122:0/1493884637 >> 172.21.15.122:0/1493884637 conn(0x7f137c0d3d40 msgr2=0x7f137c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 -- 172.21.15.122:0/1493884637 shutdown_connections 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.088+0000 7f1382b02640 1 -- 172.21.15.122:0/1493884637 wait complete. 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.089+0000 7f1382b02640 1 Processor -- start 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 -- start start 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f137c0b0e30 0x7f137c0cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 0x7f137c0b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f137c1614d0 0x7f137c15ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0b0910 con 0x7f137c0d4860 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0cfc60 con 0x7f137c0b0e30 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f1382b02640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0d2560 con 0x7f137c1614d0 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.090+0000 7f137bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 0x7f137c0b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 0x7f137c0b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46004/0 (socket says 172.21.15.122:46004) 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 -- 172.21.15.122:0/1994933554 learned_addr learned my addr 172.21.15.122:0/1994933554 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:03.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f138089e640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f137c0b0e30 0x7f137c0cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f138109f640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f137c1614d0 0x7f137c15ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 -- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f137c1614d0 msgr2=0x7f137c15ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f137c1614d0 0x7f137c15ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 -- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f137c0b0e30 msgr2=0x7f137c0cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f137c0b0e30 0x7f137c0cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 -- 172.21.15.122:0/1994933554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c0b7c20 con 0x7f137c0d4860 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f137bfff640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 0x7f137c0b8df0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f137000e9b0 tx=0x7f137000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370017070 con 0x7f137c0d4860 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137000c9a0 con 0x7f137c0d4860 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c0b87d0 con 0x7f137c0d4860 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137000cd90 con 0x7f137c0d4860 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.091+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c0cbd10 con 0x7f137c0d4860 2024-03-06T07:24:03.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.092+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c068ff0 con 0x7f137c0d4860 2024-03-06T07:24:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.093+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f137001a070 con 0x7f137c0d4860 2024-03-06T07:24:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.093+0000 7f13797fa640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f136806b4d0 0x7f136806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.093+0000 7f138089e640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f136806b4d0 0x7f136806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.093+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f137008a190 con 0x7f137c0d4860 2024-03-06T07:24:03.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.093+0000 7f138089e640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f136806b4d0 0x7f136806d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f136c000f60 tx=0x7f136c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.097+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13700589c0 con 0x7f137c0d4860 2024-03-06T07:24:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.267+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f137c0b25b0 con 0x7f137c0d4860 2024-03-06T07:24:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.268+0000 7f13797fa640 1 -- 172.21.15.122:0/1994933554 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7f137005ca60 con 0x7f137c0d4860 2024-03-06T07:24:03.270 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:24:03.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f136806b4d0 msgr2=0x7f136806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f136806b4d0 0x7f136806d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f136c000f60 tx=0x7f136c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 msgr2=0x7f137c0b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 0x7f137c0b8df0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f137000e9b0 tx=0x7f137000ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:03.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 shutdown_connections 2024-03-06T07:24:03.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f136806b4d0 0x7f136806d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f137c1614d0 0x7f137c15ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f137c0d4860 0x7f137c0b8df0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 --2- 172.21.15.122:0/1994933554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f137c0b0e30 0x7f137c0cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 >> 172.21.15.122:0/1994933554 conn(0x7f137c0d3d40 msgr2=0x7f137c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 shutdown_connections 2024-03-06T07:24:03.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.270+0000 7f1382b02640 1 -- 172.21.15.122:0/1994933554 wait complete. 2024-03-06T07:24:03.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:03.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.382+0000 7fafbe44e640 1 Processor -- start 2024-03-06T07:24:03.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafbe44e640 1 -- start start 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafbe44e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb80b5c30 0x7fafb815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafbe44e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb806c460 con 0x7fafb8072520 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafbe44e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb806c740 con 0x7fafb815d040 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafbe44e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb815fb50 con 0x7fafb80b5c30 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafb7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fafb8072520 0x7fafb8072900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42536/0 (socket says 172.21.15.122:42536) 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.383+0000 7fafb7fff640 1 -- 172.21.15.122:0/312059598 learned_addr learned my addr 172.21.15.122:0/312059598 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:03.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafbc9eb640 1 --2- 172.21.15.122:0/312059598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb80b5c30 0x7fafb815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681976469 0 0) 0x7fafb806c460 con 0x7fafb8072520 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa0003520 con 0x7fafb8072520 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9491525 0 0) 0x7fafb806c740 con 0x7fafb815d040 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb806c460 con 0x7fafb815d040 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafbc9eb640 1 -- 172.21.15.122:0/312059598 >> v1:172.21.15.169:6789/0 conn(0x7fafb815d040 legacy=0x7fafb815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafbc9eb640 1 -- 172.21.15.122:0/312059598 >> v1:172.21.15.122:6789/0 conn(0x7fafb8072520 legacy=0x7fafb8072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.384+0000 7fafbc9eb640 1 -- 172.21.15.122:0/312059598 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb815fe30 con 0x7fafb80b5c30 2024-03-06T07:24:03.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbc9eb640 1 --2- 172.21.15.122:0/312059598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb80b5c30 0x7fafb815dc00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009dc0 tx=0x7fafb002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5ec7fe2dbd18c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0039070 con 0x7fafb80b5c30 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb002cbb0 con 0x7fafb80b5c30 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafb6ffd640 1 -- 172.21.15.122:0/312059598 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0034440 con 0x7fafb80b5c30 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 -- 172.21.15.122:0/312059598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb80b5c30 msgr2=0x7fafb815dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 --2- 172.21.15.122:0/312059598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb80b5c30 0x7fafb815dc00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009dc0 tx=0x7fafb002be00 comp rx=0 tx=0).stop 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 -- 172.21.15.122:0/312059598 shutdown_connections 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 --2- 172.21.15.122:0/312059598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb80b5c30 0x7fafb815dc00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 -- 172.21.15.122:0/312059598 >> 172.21.15.122:0/312059598 conn(0x7fafb80b4e60 msgr2=0x7fafb80b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 -- 172.21.15.122:0/312059598 shutdown_connections 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.385+0000 7fafbe44e640 1 -- 172.21.15.122:0/312059598 wait complete. 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.386+0000 7fafbe44e640 1 Processor -- start 2024-03-06T07:24:03.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 -- start start 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafb8072520 0x7fafb814fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 0x7fafb81587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb815d040 0x7fafb815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8067110 con 0x7fafb80b5c30 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8066f90 con 0x7fafb8072520 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbe44e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8167390 con 0x7fafb815d040 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafb77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 0x7fafb81587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafb77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 0x7fafb81587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46010/0 (socket says 172.21.15.122:46010) 2024-03-06T07:24:03.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafb77fe640 1 -- 172.21.15.122:0/1865900471 learned_addr learned my addr 172.21.15.122:0/1865900471 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafb7fff640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafb8072520 0x7fafb814fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.387+0000 7fafbc9eb640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb815d040 0x7fafb815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb77fe640 1 -- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb815d040 msgr2=0x7fafb815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb77fe640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb815d040 0x7fafb815a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb77fe640 1 -- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafb8072520 msgr2=0x7fafb814fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb77fe640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafb8072520 0x7fafb814fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb77fe640 1 -- 172.21.15.122:0/1865900471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8158560 con 0x7fafb80b5c30 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb7fff640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafb8072520 0x7fafb814fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb77fe640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 0x7fafb81587d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c390 tx=0x7fafac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac017070 con 0x7fafb80b5c30 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac0052d0 con 0x7fafb80b5c30 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb8158280 con 0x7fafb80b5c30 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac003810 con 0x7fafb80b5c30 2024-03-06T07:24:03.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.388+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb8154650 con 0x7fafb80b5c30 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.389+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb804f330 con 0x7fafb80b5c30 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.390+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafac01a070 con 0x7fafb80b5c30 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.390+0000 7fafb4ff9640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa406b4d0 0x7fafa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.390+0000 7fafb7fff640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa406b4d0 0x7fafa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.390+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafac0894f0 con 0x7fafb80b5c30 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.390+0000 7fafb7fff640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa406b4d0 0x7fafa406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fafa8000f60 tx=0x7fafa8007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.394+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafac057d20 con 0x7fafb80b5c30 2024-03-06T07:24:03.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.578+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fafb80cd090 con 0x7fafb80b5c30 2024-03-06T07:24:03.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.578+0000 7fafb4ff9640 1 -- 172.21.15.122:0/1865900471 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) v1 ==== 80+0+0 (secure 0 0 0) 0x7fafac05bdc0 con 0x7fafb80b5c30 2024-03-06T07:24:03.580 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:24:03.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa406b4d0 msgr2=0x7fafa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa406b4d0 0x7fafa406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fafa8000f60 tx=0x7fafa8007d30 comp rx=0 tx=0).stop 2024-03-06T07:24:03.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 msgr2=0x7fafb81587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 0x7fafb81587d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c390 tx=0x7fafac00c850 comp rx=0 tx=0).stop 2024-03-06T07:24:03.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 shutdown_connections 2024-03-06T07:24:03.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa406b4d0 0x7fafa406d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafb815d040 0x7fafb815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb80b5c30 0x7fafb81587d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 --2- 172.21.15.122:0/1865900471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafb8072520 0x7fafb814fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 >> 172.21.15.122:0/1865900471 conn(0x7fafb80b4e60 msgr2=0x7fafb815d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 shutdown_connections 2024-03-06T07:24:03.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:03.580+0000 7fafbe44e640 1 -- 172.21.15.122:0/1865900471 wait complete. 2024-03-06T07:24:03.592 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:24:03.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:03.593 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:03.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[56162]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:03.619 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:24:03.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:03.620 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:03.738 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:24:03.739 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:03.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:03.754 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:24:03.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:03.754 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:03.809 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:03.809 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:03.809 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:03.825 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:03.825 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:24:03.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:03.825 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:03.880 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:03.881 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:24:03.881 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:03.896 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:03.896 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:24:03.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:03.897 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:03.951 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:03.952 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:04.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.049+0000 7f23a5aeb640 1 Processor -- start 2024-03-06T07:24:04.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.049+0000 7f23a5aeb640 1 -- start start 2024-03-06T07:24:04.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f23a5aeb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f23a00b6550 0x7f23a00cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f23a5aeb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00d06a0 0x7f23a00cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f23a5aeb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a015c620 con 0x7f23a00d02c0 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f23a5aeb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a00b50b0 con 0x7f23a00b6550 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f23a5aeb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00cd800 con 0x7f23a00d06a0 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f239ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00d06a0 0x7f23a00cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f239effd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f23a00b6550 0x7f23a00cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f239f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f23a00d02c0 0x7f23a00ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:49222/0 (socket says 172.21.15.122:49222) 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.050+0000 7f239f7fe640 1 -- 172.21.15.122:0/16566521 learned_addr learned my addr 172.21.15.122:0/16566521 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239e7fc640 1 -- 172.21.15.122:0/16566521 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686259182 0 0) 0x7f23a015c620 con 0x7f23a00d02c0 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239e7fc640 1 -- 172.21.15.122:0/16566521 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2388003520 con 0x7f23a00d02c0 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239ffff640 1 -- 172.21.15.122:0/16566521 >> v2:172.21.15.169:3300/0 conn(0x7f23a00b6550 msgr2=0x7f23a00cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239ffff640 1 --2- 172.21.15.122:0/16566521 >> v2:172.21.15.169:3300/0 conn(0x7f23a00b6550 0x7f23a00cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239ffff640 1 -- 172.21.15.122:0/16566521 >> v1:172.21.15.169:6789/0 conn(0x7f23a00d02c0 legacy=0x7f23a00ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239ffff640 1 -- 172.21.15.122:0/16566521 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00b4a60 con 0x7f23a00d06a0 2024-03-06T07:24:04.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.051+0000 7f239effd640 1 --2- 172.21.15.122:0/16566521 >> v2:172.21.15.169:3300/0 conn(0x7f23a00b6550 0x7f23a00cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:04.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f239ffff640 1 --2- 172.21.15.122:0/16566521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00d06a0 0x7f23a00cd0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2398009d90 tx=0x7f239802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc4e6c273008a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f239e7fc640 1 -- 172.21.15.122:0/16566521 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398039070 con 0x7f23a00d06a0 2024-03-06T07:24:04.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f239e7fc640 1 -- 172.21.15.122:0/16566521 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239802cbc0 con 0x7f23a00d06a0 2024-03-06T07:24:04.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f239e7fc640 1 -- 172.21.15.122:0/16566521 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398034440 con 0x7f23a00d06a0 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 -- 172.21.15.122:0/16566521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00d06a0 msgr2=0x7f23a00cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/16566521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00d06a0 0x7f23a00cd0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2398009d90 tx=0x7f239802be10 comp rx=0 tx=0).stop 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 -- 172.21.15.122:0/16566521 shutdown_connections 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/16566521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00d06a0 0x7f23a00cd0d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/16566521 >> v2:172.21.15.169:3300/0 conn(0x7f23a00b6550 0x7f23a00cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 -- 172.21.15.122:0/16566521 >> 172.21.15.122:0/16566521 conn(0x7f23a00b35a0 msgr2=0x7f23a00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 -- 172.21.15.122:0/16566521 shutdown_connections 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.052+0000 7f23a5aeb640 1 -- 172.21.15.122:0/16566521 wait complete. 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.053+0000 7f23a5aeb640 1 Processor -- start 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 -- start start 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 0x7f23a00c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23a00d02c0 0x7f23a00c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 0x7f23a00c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00ce9c0 con 0x7f23a00d06a0 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00ce840 con 0x7f23a00d02c0 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f23a5aeb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00ceb40 con 0x7f23a00b6550 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f239ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 0x7f23a00c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 0x7f23a00c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 0x7f23a00c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:60324/0 (socket says 172.21.15.122:60324) 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239f7fe640 1 -- 172.21.15.122:0/201720549 learned_addr learned my addr 172.21.15.122:0/201720549 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.054+0000 7f239ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 0x7f23a00c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46020/0 (socket says 172.21.15.122:46020) 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239effd640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23a00d02c0 0x7f23a00c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239ffff640 1 -- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 msgr2=0x7f23a00c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239ffff640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 0x7f23a00c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239ffff640 1 -- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23a00d02c0 msgr2=0x7f23a00c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239ffff640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23a00d02c0 0x7f23a00c6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239ffff640 1 -- 172.21.15.122:0/201720549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00c5b80 con 0x7f23a00d06a0 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239f7fe640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 0x7f23a00c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f239ffff640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 0x7f23a00c9af0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f23a00cc070 tx=0x7f2398004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398039050 con 0x7f23a00d06a0 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a00c5e60 con 0x7f23a00d06a0 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239803a040 con 0x7f23a00d06a0 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398009450 con 0x7f23a00d06a0 2024-03-06T07:24:04.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.055+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a00c6420 con 0x7f23a00d06a0 2024-03-06T07:24:04.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.056+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a004f330 con 0x7f23a00d06a0 2024-03-06T07:24:04.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.057+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2398047020 con 0x7f23a00d06a0 2024-03-06T07:24:04.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.057+0000 7f2387fff640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f238c06b500 0x7f238c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.057+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23980ac210 con 0x7f23a00d06a0 2024-03-06T07:24:04.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.057+0000 7f239f7fe640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f238c06b500 0x7f238c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.057+0000 7f239f7fe640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f238c06b500 0x7f238c06d9c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2390005880 tx=0x7f2390009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.061+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f239804e020 con 0x7f23a00d06a0 2024-03-06T07:24:04.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.270+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23a00cf3d0 con 0x7f23a00d06a0 2024-03-06T07:24:04.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.271+0000 7f2387fff640 1 -- 172.21.15.122:0/201720549 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f239807a9c0 con 0x7f23a00d06a0 2024-03-06T07:24:04.273 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:04.273 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:40.243216+0000","modified":"2024-03-06T07:23:42.237773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/3905558085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3905558085},{"type":"v1","addr":"172.21.15.169:6835","nonce":3905558085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":3}]} 2024-03-06T07:24:04.273 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 21 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f238c06b500 msgr2=0x7f238c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f238c06b500 0x7f238c06d9c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2390005880 tx=0x7f2390009040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 msgr2=0x7f23a00c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 0x7f23a00c9af0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f23a00cc070 tx=0x7f2398004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 shutdown_connections 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f238c06b500 0x7f238c06d9c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23a00d06a0 0x7f23a00c9af0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23a00d02c0 0x7f23a00c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 --2- 172.21.15.122:0/201720549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23a00b6550 0x7f23a00c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 >> 172.21.15.122:0/201720549 conn(0x7f23a00b35a0 msgr2=0x7f23a00b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 shutdown_connections 2024-03-06T07:24:04.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.273+0000 7f23a5aeb640 1 -- 172.21.15.122:0/201720549 wait complete. 2024-03-06T07:24:04.286 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:04.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.384+0000 7fe9ef1e4640 1 Processor -- start 2024-03-06T07:24:04.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.384+0000 7fe9ef1e4640 1 -- start start 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ef1e4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ef1e4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe9e80d0700 0x7fe9e80b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ef1e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9e80ccae0 0x7fe9e80ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ef1e4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e80b04e0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ef1e4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e815c270 con 0x7fe9e80d0700 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ef1e4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80d0e90 con 0x7fe9e80d0320 2024-03-06T07:24:04.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ed781640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9e80ccae0 0x7fe9e80ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ed781640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9e80ccae0 0x7fe9e80ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46024/0 (socket says 172.21.15.122:46024) 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ed781640 1 -- 172.21.15.122:0/569664418 learned_addr learned my addr 172.21.15.122:0/569664418 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9e7fff640 1 --2- 172.21.15.122:0/569664418 >> v2:172.21.15.169:3300/0 conn(0x7fe9e80d0700 0x7fe9e80b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.385+0000 7fe9ecf80640 1 --2- 172.21.15.122:0/569664418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9ed781640 1 -- 172.21.15.122:0/569664418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 msgr2=0x7fe9e80b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9ed781640 1 --2- 172.21.15.122:0/569664418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9ed781640 1 -- 172.21.15.122:0/569664418 >> v2:172.21.15.169:3300/0 conn(0x7fe9e80d0700 msgr2=0x7fe9e80b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9ed781640 1 --2- 172.21.15.122:0/569664418 >> v2:172.21.15.169:3300/0 conn(0x7fe9e80d0700 0x7fe9e80b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9ed781640 1 -- 172.21.15.122:0/569664418 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80b5ae0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9ed781640 1 --2- 172.21.15.122:0/569664418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e80ccf40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80cff90 tx=0x7fe9e002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4833e74d6e9b9c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9e77fe640 1 -- 172.21.15.122:0/569664418 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e002cb00 con 0x7fe9e80ccae0 2024-03-06T07:24:04.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9e77fe640 1 -- 172.21.15.122:0/569664418 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e002cca0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.386+0000 7fe9e77fe640 1 -- 172.21.15.122:0/569664418 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0034610 con 0x7fe9e80ccae0 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/569664418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 msgr2=0x7fe9e80ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/569664418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e80ccf40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80cff90 tx=0x7fe9e002bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/569664418 shutdown_connections 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/569664418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e80ccf40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/569664418 >> v2:172.21.15.169:3300/0 conn(0x7fe9e80d0700 0x7fe9e80b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/569664418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/569664418 >> 172.21.15.122:0/569664418 conn(0x7fe9e80724d0 msgr2=0x7fe9e80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/569664418 shutdown_connections 2024-03-06T07:24:04.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.387+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/569664418 wait complete. 2024-03-06T07:24:04.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.388+0000 7fe9ef1e4640 1 Processor -- start 2024-03-06T07:24:04.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.388+0000 7fe9ef1e4640 1 -- start start 2024-03-06T07:24:04.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ef1e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e814b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ef1e4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ef1e4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9e80d0700 0x7fe9e80c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ef1e4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b60a0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ef1e4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b0170 con 0x7fe9e80d0700 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ef1e4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80ca790 con 0x7fe9e80d0320 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e814b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e814b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46040/0 (socket says 172.21.15.122:46040) 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 -- 172.21.15.122:0/2899813241 learned_addr learned my addr 172.21.15.122:0/2899813241 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9e7fff640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ed781640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9e80d0700 0x7fe9e80c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 -- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 msgr2=0x7fe9e80c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 -- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9e80d0700 msgr2=0x7fe9e80c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9e80d0700 0x7fe9e80c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.389+0000 7fe9ecf80640 1 -- 172.21.15.122:0/2899813241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80cf750 con 0x7fe9e80ccae0 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.390+0000 7fe9ecf80640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e814b720 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800e890 tx=0x7fe9d800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.390+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8017070 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.390+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d800c940 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.390+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e81f4bf0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.390+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800cd10 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.390+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e80cebd0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.391+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e804f330 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.391+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9d801a070 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.391+0000 7fe9e57fa640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9d406b4d0 0x7fe9d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.392+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9d808a110 con 0x7fe9e80ccae0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.392+0000 7fe9e7fff640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9d406b4d0 0x7fe9d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.392+0000 7fe9e7fff640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9d406b4d0 0x7fe9d406d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80cfef0 tx=0x7fe9dc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.396+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9d8058940 con 0x7fe9e80ccae0 2024-03-06T07:24:04.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.568+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9e80cbad0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.569+0000 7fe9e57fa640 1 -- 172.21.15.122:0/2899813241 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16613 (secure 0 0 0) 0x7fe9d805c9e0 con 0x7fe9e80ccae0 2024-03-06T07:24:04.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:04.571 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":45,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:41.897227+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9d406b4d0 msgr2=0x7fe9d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9d406b4d0 0x7fe9d406d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80cfef0 tx=0x7fe9dc009190 comp rx=0 tx=0).stop 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 msgr2=0x7fe9e814b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e814b720 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800e890 tx=0x7fe9d800ed50 comp rx=0 tx=0).stop 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 shutdown_connections 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9d406b4d0 0x7fe9d406d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9e80d0700 0x7fe9e80c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9e80d0320 0x7fe9e80c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 --2- 172.21.15.122:0/2899813241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e80ccae0 0x7fe9e814b720 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 >> 172.21.15.122:0/2899813241 conn(0x7fe9e80724d0 msgr2=0x7fe9e806b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 shutdown_connections 2024-03-06T07:24:04.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.571+0000 7fe9ef1e4640 1 -- 172.21.15.122:0/2899813241 wait complete. 2024-03-06T07:24:04.584 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:38.463254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:39.205152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:04.585 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.686+0000 7fc707a6b640 1 Processor -- start 2024-03-06T07:24:04.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.686+0000 7fc707a6b640 1 -- start start 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc707a6b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc700066d90 0x7fc7000b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc707a6b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70006b710 con 0x7fc7000b5410 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc707a6b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70006b150 con 0x7fc7000cf9a0 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc707a6b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000b61c0 con 0x7fc700066d90 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc706008640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc7000b5410 0x7fc7000ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36950/0 (socket says 172.21.15.122:36950) 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc706008640 1 -- 172.21.15.122:0/1665486338 learned_addr learned my addr 172.21.15.122:0/1665486338 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc705006640 1 --2- 172.21.15.122:0/1665486338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc700066d90 0x7fc7000b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056673592 0 0) 0x7fc70006b710 con 0x7fc7000b5410 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003520 con 0x7fc7000b5410 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645715001 0 0) 0x7fc70006b150 con 0x7fc7000cf9a0 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70006b710 con 0x7fc7000cf9a0 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc705006640 1 -- 172.21.15.122:0/1665486338 >> v1:172.21.15.169:6789/0 conn(0x7fc7000cf9a0 legacy=0x7fc7000b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc705006640 1 -- 172.21.15.122:0/1665486338 >> v1:172.21.15.122:6789/0 conn(0x7fc7000b5410 legacy=0x7fc7000ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.687+0000 7fc705006640 1 -- 172.21.15.122:0/1665486338 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70006b430 con 0x7fc700066d90 2024-03-06T07:24:04.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.688+0000 7fc705006640 1 --2- 172.21.15.122:0/1665486338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc700066d90 0x7fc7000b4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6f00097d0 tx=0x7fc6f002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a575c11895dc117e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.688+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0035020 con 0x7fc700066d90 2024-03-06T07:24:04.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.688+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f0031890 con 0x7fc700066d90 2024-03-06T07:24:04.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.688+0000 7fc704805640 1 -- 172.21.15.122:0/1665486338 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0031c40 con 0x7fc700066d90 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 -- 172.21.15.122:0/1665486338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc700066d90 msgr2=0x7fc7000b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 --2- 172.21.15.122:0/1665486338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc700066d90 0x7fc7000b4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6f00097d0 tx=0x7fc6f002b930 comp rx=0 tx=0).stop 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 -- 172.21.15.122:0/1665486338 shutdown_connections 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 --2- 172.21.15.122:0/1665486338 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc700066d90 0x7fc7000b4ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 -- 172.21.15.122:0/1665486338 >> 172.21.15.122:0/1665486338 conn(0x7fc7000b1d70 msgr2=0x7fc7000b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 -- 172.21.15.122:0/1665486338 shutdown_connections 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.689+0000 7fc707a6b640 1 -- 172.21.15.122:0/1665486338 wait complete. 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 Processor -- start 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 -- start start 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 0x7fc700158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7000b5410 0x7fc70015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7000cf9a0 0x7fc7000ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000cfd80 con 0x7fc700066d90 2024-03-06T07:24:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000b0c50 con 0x7fc7000b5410 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.690+0000 7fc707a6b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000cff00 con 0x7fc7000cf9a0 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 0x7fc700158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 0x7fc700158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38540/0 (socket says 172.21.15.122:38540) 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 -- 172.21.15.122:0/2653147576 learned_addr learned my addr 172.21.15.122:0/2653147576 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc706008640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7000cf9a0 0x7fc7000ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705006640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7000b5410 0x7fc70015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 -- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7000cf9a0 msgr2=0x7fc7000ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7000cf9a0 0x7fc7000ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 -- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7000b5410 msgr2=0x7fc70015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7000b5410 0x7fc70015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 -- 172.21.15.122:0/2653147576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700157c90 con 0x7fc700066d90 2024-03-06T07:24:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc705807640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 0x7fc700158730 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00c430 tx=0x7fc6fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc017070 con 0x7fc700066d90 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0052f0 con 0x7fc700066d90 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc003830 con 0x7fc700066d90 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70014afc0 con 0x7fc700066d90 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.691+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70015a1b0 con 0x7fc700066d90 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.692+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70009e3d0 con 0x7fc700066d90 2024-03-06T07:24:04.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.693+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6fc01a070 con 0x7fc700066d90 2024-03-06T07:24:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.693+0000 7fc6f67fc640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6e406b4d0 0x7fc6e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.693+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6fc0894d0 con 0x7fc700066d90 2024-03-06T07:24:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.693+0000 7fc705006640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6e406b4d0 0x7fc6e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.694+0000 7fc705006640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6e406b4d0 0x7fc6e406d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0036000 tx=0x7fc6f003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.697+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc700157c90 con 0x7fc700066d90 2024-03-06T07:24:04.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.866+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc700068ff0 con 0x7fc700066d90 2024-03-06T07:24:04.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.866+0000 7fc6f67fc640 1 -- 172.21.15.122:0/2653147576 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6fc087e20 con 0x7fc700066d90 2024-03-06T07:24:04.868 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:04.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6e406b4d0 msgr2=0x7fc6e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6e406b4d0 0x7fc6e406d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0036000 tx=0x7fc6f003d040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 msgr2=0x7fc700158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 0x7fc700158730 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00c430 tx=0x7fc6fc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 shutdown_connections 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6e406b4d0 0x7fc6e406d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7000cf9a0 0x7fc7000ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7000b5410 0x7fc70015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 --2- 172.21.15.122:0/2653147576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc700066d90 0x7fc700158730 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 >> 172.21.15.122:0/2653147576 conn(0x7fc7000b1d70 msgr2=0x7fc70006e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 shutdown_connections 2024-03-06T07:24:04.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.868+0000 7fc707a6b640 1 -- 172.21.15.122:0/2653147576 wait complete. 2024-03-06T07:24:04.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:04.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.982+0000 7f0e235e2640 1 Processor -- start 2024-03-06T07:24:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e235e2640 1 -- start start 2024-03-06T07:24:04.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e235e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c1614d0 0x7f0e1c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e235e2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e1c0d1910 0x7f0e1c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e235e2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1c0d2740 con 0x7f0e1c0d22b0 2024-03-06T07:24:04.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e235e2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c1618b0 con 0x7f0e1c0d1910 2024-03-06T07:24:04.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e235e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0cf400 con 0x7f0e1c1614d0 2024-03-06T07:24:04.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e20b7d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e1c0d1910 0x7f0e1c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.983+0000 7f0e20b7d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e1c0d1910 0x7f0e1c0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38546/0 (socket says 172.21.15.122:38546) 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e20b7d640 1 -- 172.21.15.122:0/3968206820 learned_addr learned my addr 172.21.15.122:0/3968206820 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e2137e640 1 --2- 172.21.15.122:0/3968206820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c1614d0 0x7f0e1c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e20b7d640 1 -- 172.21.15.122:0/3968206820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c1614d0 msgr2=0x7f0e1c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e20b7d640 1 --2- 172.21.15.122:0/3968206820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c1614d0 0x7f0e1c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e20b7d640 1 -- 172.21.15.122:0/3968206820 >> v1:172.21.15.169:6789/0 conn(0x7f0e1c0d22b0 legacy=0x7f0e1c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e20b7d640 1 -- 172.21.15.122:0/3968206820 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0cf6a0 con 0x7f0e1c0d1910 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.984+0000 7f0e20b7d640 1 --2- 172.21.15.122:0/3968206820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0d1d70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009870 tx=0x7f0e0c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18adf6b727899283 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e13fff640 1 -- 172.21.15.122:0/3968206820 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c034070 con 0x7f0e1c0d1910 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e13fff640 1 -- 172.21.15.122:0/3968206820 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0c030840 con 0x7f0e1c0d1910 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e13fff640 1 -- 172.21.15.122:0/3968206820 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c030c10 con 0x7f0e1c0d1910 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 -- 172.21.15.122:0/3968206820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 msgr2=0x7f0e1c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 --2- 172.21.15.122:0/3968206820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0d1d70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009870 tx=0x7f0e0c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 -- 172.21.15.122:0/3968206820 shutdown_connections 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 --2- 172.21.15.122:0/3968206820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0d1d70 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 --2- 172.21.15.122:0/3968206820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c1614d0 0x7f0e1c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 -- 172.21.15.122:0/3968206820 >> 172.21.15.122:0/3968206820 conn(0x7f0e1c0d3d40 msgr2=0x7f0e1c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 -- 172.21.15.122:0/3968206820 shutdown_connections 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.985+0000 7f0e235e2640 1 -- 172.21.15.122:0/3968206820 wait complete. 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.986+0000 7f0e235e2640 1 Processor -- start 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.986+0000 7f0e235e2640 1 -- start start 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e235e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e235e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c0d22b0 0x7f0e1c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e235e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e1c1614d0 0x7f0e1c14fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e235e2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0b0910 con 0x7f0e1c0d1910 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e235e2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0cfc60 con 0x7f0e1c1614d0 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e235e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0b1e40 con 0x7f0e1c0d22b0 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38550/0 (socket says 172.21.15.122:38550) 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 -- 172.21.15.122:0/920300659 learned_addr learned my addr 172.21.15.122:0/920300659 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e20b7d640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c0d22b0 0x7f0e1c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e21b7f640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e1c1614d0 0x7f0e1c14fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 -- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c0d22b0 msgr2=0x7f0e1c0ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c0d22b0 0x7f0e1c0ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 -- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e1c1614d0 msgr2=0x7f0e1c14fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e1c1614d0 0x7f0e1c14fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e2137e640 1 -- 172.21.15.122:0/920300659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0b8460 con 0x7f0e1c0d1910 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.987+0000 7f0e20b7d640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c0d22b0 0x7f0e1c0ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.988+0000 7f0e2137e640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0b42a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009d10 tx=0x7f0e1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.988+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18018020 con 0x7f0e1c0d1910 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.988+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e18014480 con 0x7f0e1c0d1910 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.988+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c0b8740 con 0x7f0e1c0d1910 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.988+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18014830 con 0x7f0e1c0d1910 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.988+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c0b8d00 con 0x7f0e1c0d1910 2024-03-06T07:24:04.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.989+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1c068ff0 con 0x7f0e1c0d1910 2024-03-06T07:24:04.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.989+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e18005d10 con 0x7f0e1c0d1910 2024-03-06T07:24:04.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.990+0000 7f0e11ffb640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0006b4d0 0x7f0e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.990+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e1808ac60 con 0x7f0e1c0d1910 2024-03-06T07:24:04.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.990+0000 7f0e20b7d640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0006b4d0 0x7f0e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.990+0000 7f0e20b7d640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0006b4d0 0x7f0e0006d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c036000 tx=0x7f0e0c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:04.994+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e18022070 con 0x7f0e1c0d1910 2024-03-06T07:24:05.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.162+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e1c0b9b10 con 0x7f0e1c0d1910 2024-03-06T07:24:05.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.163+0000 7f0e11ffb640 1 -- 172.21.15.122:0/920300659 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e18010040 con 0x7f0e1c0d1910 2024-03-06T07:24:05.165 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:24:05.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0006b4d0 msgr2=0x7f0e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0006b4d0 0x7f0e0006d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c036000 tx=0x7f0e0c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 msgr2=0x7f0e1c0b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0b42a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009d10 tx=0x7f0e1800c490 comp rx=0 tx=0).stop 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 shutdown_connections 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0e0006b4d0 0x7f0e0006d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e1c1614d0 0x7f0e1c14fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0e1c0d22b0 0x7f0e1c0ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 --2- 172.21.15.122:0/920300659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c0d1910 0x7f0e1c0b42a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 >> 172.21.15.122:0/920300659 conn(0x7f0e1c0d3d40 msgr2=0x7f0e1c06ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 shutdown_connections 2024-03-06T07:24:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.165+0000 7f0e235e2640 1 -- 172.21.15.122:0/920300659 wait complete. 2024-03-06T07:24:05.178 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.279+0000 7f5b5e2d4640 1 Processor -- start 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.279+0000 7f5b5e2d4640 1 -- start start 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5e2d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b580aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5e2d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5b580aef80 0x7f5b580cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5e2d4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b580d03f0 con 0x7f5b580cffd0 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5e2d4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b58158ac0 con 0x7f5b580aef80 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5e2d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5815b2a0 con 0x7f5b581586e0 2024-03-06T07:24:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5b580aef80 0x7f5b580cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5c871640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5b580cffd0 0x7f5b5806da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36954/0 (socket says 172.21.15.122:36954) 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b5c871640 1 -- 172.21.15.122:0/750222473 learned_addr learned my addr 172.21.15.122:0/750222473 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b57fff640 1 --2- 172.21.15.122:0/750222473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b580aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b577fe640 1 -- 172.21.15.122:0/750222473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 msgr2=0x7f5b580aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b577fe640 1 --2- 172.21.15.122:0/750222473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b580aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b577fe640 1 -- 172.21.15.122:0/750222473 >> v1:172.21.15.122:6789/0 conn(0x7f5b580cffd0 legacy=0x7f5b5806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b56ffd640 1 -- 172.21.15.122:0/750222473 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021428235 0 0) 0x7f5b580d03f0 con 0x7f5b580cffd0 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.280+0000 7f5b577fe640 1 -- 172.21.15.122:0/750222473 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5815b540 con 0x7f5b580aef80 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.281+0000 7f5b577fe640 1 --2- 172.21.15.122:0/750222473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580aef80 0x7f5b580cfa90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097a0 tx=0x7f5b4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3928ea3375accb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.281+0000 7f5b56ffd640 1 -- 172.21.15.122:0/750222473 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c035020 con 0x7f5b580aef80 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.281+0000 7f5b56ffd640 1 -- 172.21.15.122:0/750222473 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c031830 con 0x7f5b580aef80 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.281+0000 7f5b56ffd640 1 -- 172.21.15.122:0/750222473 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c031c00 con 0x7f5b580aef80 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/750222473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580aef80 msgr2=0x7f5b580cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/750222473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580aef80 0x7f5b580cfa90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097a0 tx=0x7f5b4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/750222473 shutdown_connections 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/750222473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580aef80 0x7f5b580cfa90 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/750222473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b580aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/750222473 >> 172.21.15.122:0/750222473 conn(0x7f5b5806b190 msgr2=0x7f5b5806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/750222473 shutdown_connections 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.282+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/750222473 wait complete. 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.283+0000 7f5b5e2d4640 1 Processor -- start 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.283+0000 7f5b5e2d4640 1 -- start start 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5e2d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b580aef80 0x7f5b580cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5e2d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 0x7f5b580ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5e2d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b5806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5e2d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5815bde0 con 0x7f5b580cffd0 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5e2d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5815bb00 con 0x7f5b580aef80 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5e2d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58071350 con 0x7f5b581586e0 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 0x7f5b580ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 0x7f5b580ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38574/0 (socket says 172.21.15.122:38574) 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 -- 172.21.15.122:0/985183093 learned_addr learned my addr 172.21.15.122:0/985183093 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b57fff640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b580aef80 0x7f5b580cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b5c871640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b5806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 -- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 msgr2=0x7f5b5806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b5806d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 -- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b580aef80 msgr2=0x7f5b580cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b580aef80 0x7f5b580cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 -- 172.21.15.122:0/985183093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580ac140 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b577fe640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 0x7f5b580ab7d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c02bee0 tx=0x7f5b4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.284+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c035040 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.285+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c03b920 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.285+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c03bd10 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.285+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b580ac420 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.285+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b580ac9e0 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.285+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b580732b0 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.286+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b4c038020 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.286+0000 7f5b54ff9640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4406b4d0 0x7f5b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.286+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b4c0ac1f0 con 0x7f5b580cffd0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.286+0000 7f5b57fff640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4406b4d0 0x7f5b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.287+0000 7f5b57fff640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4406b4d0 0x7f5b4406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5b5806b950 tx=0x7f5b48009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.291+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b4c031800 con 0x7f5b580cffd0 2024-03-06T07:24:05.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.463+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b58068ff0 con 0x7f5b580cffd0 2024-03-06T07:24:05.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.464+0000 7f5b54ff9640 1 -- 172.21.15.122:0/985183093 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b4c07aa30 con 0x7f5b580cffd0 2024-03-06T07:24:05.466 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:24:05.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4406b4d0 msgr2=0x7f5b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4406b4d0 0x7f5b4406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5b5806b950 tx=0x7f5b48009190 comp rx=0 tx=0).stop 2024-03-06T07:24:05.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 msgr2=0x7f5b580ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 0x7f5b580ab7d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c02bee0 tx=0x7f5b4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 shutdown_connections 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4406b4d0 0x7f5b4406d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b581586e0 0x7f5b5806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b580cffd0 0x7f5b580ab7d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 --2- 172.21.15.122:0/985183093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b580aef80 0x7f5b580cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 >> 172.21.15.122:0/985183093 conn(0x7f5b5806b190 msgr2=0x7f5b581570c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 shutdown_connections 2024-03-06T07:24:05.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.466+0000 7f5b5e2d4640 1 -- 172.21.15.122:0/985183093 wait complete. 2024-03-06T07:24:05.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.577+0000 7f81f23ac640 1 Processor -- start 2024-03-06T07:24:05.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.577+0000 7f81f23ac640 1 -- start start 2024-03-06T07:24:05.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81f23ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81f23ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81ec0b41f0 0x7f81ec0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81f23ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec0b3430 con 0x7f81ec15dfe0 2024-03-06T07:24:05.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81f23ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ec0d2c50 con 0x7f81ec0b41f0 2024-03-06T07:24:05.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81f23ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0b31e0 con 0x7f81ec0d22b0 2024-03-06T07:24:05.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81f0949640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81ec0b41f0 0x7f81ec0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81ebfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f81ec15dfe0 0x7f81ec0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36970/0 (socket says 172.21.15.122:36970) 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81ebfff640 1 -- 172.21.15.122:0/3192604805 learned_addr learned my addr 172.21.15.122:0/3192604805 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.578+0000 7f81eb7fe640 1 --2- 172.21.15.122:0/3192604805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81f0949640 1 -- 172.21.15.122:0/3192604805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 msgr2=0x7f81ec0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81f0949640 1 --2- 172.21.15.122:0/3192604805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81f0949640 1 -- 172.21.15.122:0/3192604805 >> v1:172.21.15.122:6789/0 conn(0x7f81ec15dfe0 legacy=0x7f81ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81f0949640 1 -- 172.21.15.122:0/3192604805 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec0b3710 con 0x7f81ec0b41f0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81eaffd640 1 -- 172.21.15.122:0/3192604805 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149115293 0 0) 0x7f81ec0b3430 con 0x7f81ec15dfe0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81f0949640 1 --2- 172.21.15.122:0/3192604805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec0b4650 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009d90 tx=0x7f81e402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307b07f1f8a1d6fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.579+0000 7f81eaffd640 1 -- 172.21.15.122:0/3192604805 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4039070 con 0x7f81ec0b41f0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81eaffd640 1 -- 172.21.15.122:0/3192604805 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e402cc50 con 0x7f81ec0b41f0 2024-03-06T07:24:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81eaffd640 1 -- 172.21.15.122:0/3192604805 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4034440 con 0x7f81ec0b41f0 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 -- 172.21.15.122:0/3192604805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 msgr2=0x7f81ec0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 --2- 172.21.15.122:0/3192604805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec0b4650 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009d90 tx=0x7f81e402bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 -- 172.21.15.122:0/3192604805 shutdown_connections 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 --2- 172.21.15.122:0/3192604805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec0b4650 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 --2- 172.21.15.122:0/3192604805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 -- 172.21.15.122:0/3192604805 >> 172.21.15.122:0/3192604805 conn(0x7f81ec0d3cd0 msgr2=0x7f81ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 -- 172.21.15.122:0/3192604805 shutdown_connections 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.580+0000 7f81f23ac640 1 -- 172.21.15.122:0/3192604805 wait complete. 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.581+0000 7f81f23ac640 1 Processor -- start 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.581+0000 7f81f23ac640 1 -- start start 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f23ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f23ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f23ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81ec15dfe0 0x7f81ec1f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f23ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0cfa00 con 0x7f81ec0b41f0 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f23ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0b3d60 con 0x7f81ec15dfe0 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f23ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0d1c70 con 0x7f81ec0d22b0 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81ebfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81ebfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec1f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38582/0 (socket says 172.21.15.122:38582) 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81ebfff640 1 -- 172.21.15.122:0/2399079964 learned_addr learned my addr 172.21.15.122:0/2399079964 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81eb7fe640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81f0949640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81ec15dfe0 0x7f81ec1f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81ebfff640 1 -- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 msgr2=0x7f81ec1522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81ebfff640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec1522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.582+0000 7f81ebfff640 1 -- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81ec15dfe0 msgr2=0x7f81ec1f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81ebfff640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81ec15dfe0 0x7f81ec1f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81ebfff640 1 -- 172.21.15.122:0/2399079964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec0b7a20 con 0x7f81ec0b41f0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81ebfff640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec1f6570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c460 tx=0x7f81dc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0074e0 con 0x7f81ec0b41f0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc0047e0 con 0x7f81ec0b41f0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec0cbf70 con 0x7f81ec0b41f0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc004bd0 con 0x7f81ec0b41f0 2024-03-06T07:24:05.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.583+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec0cc530 con 0x7f81ec0b41f0 2024-03-06T07:24:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.584+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec04f330 con 0x7f81ec0b41f0 2024-03-06T07:24:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.584+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81dc011070 con 0x7f81ec0b41f0 2024-03-06T07:24:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.585+0000 7f81e8ff9640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f81d806b4d0 0x7f81d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.585+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81dc089cc0 con 0x7f81ec0b41f0 2024-03-06T07:24:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.585+0000 7f81eb7fe640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f81d806b4d0 0x7f81d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.585+0000 7f81eb7fe640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f81d806b4d0 0x7f81d806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0d27c0 tx=0x7f81e0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.589+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81dc0584f0 con 0x7f81ec0b41f0 2024-03-06T07:24:05.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.800+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81ec0ce910 con 0x7f81ec0b41f0 2024-03-06T07:24:05.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.800+0000 7f81e8ff9640 1 -- 172.21.15.122:0/2399079964 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f81dc05c590 con 0x7f81ec0b41f0 2024-03-06T07:24:05.803 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:05.803 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:40.243216+0000","modified":"2024-03-06T07:23:42.237773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/3905558085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3905558085},{"type":"v1","addr":"172.21.15.169:6835","nonce":3905558085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":3}]} 2024-03-06T07:24:05.803 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 21 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f81d806b4d0 msgr2=0x7f81d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f81d806b4d0 0x7f81d806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0d27c0 tx=0x7f81e0009190 comp rx=0 tx=0).stop 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 msgr2=0x7f81ec1f6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec1f6570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c460 tx=0x7f81dc00c920 comp rx=0 tx=0).stop 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 shutdown_connections 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f81d806b4d0 0x7f81d806d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81ec15dfe0 0x7f81ec1f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81ec0d22b0 0x7f81ec1522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 --2- 172.21.15.122:0/2399079964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81ec0b41f0 0x7f81ec1f6570 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 >> 172.21.15.122:0/2399079964 conn(0x7f81ec0d3cd0 msgr2=0x7f81ec06cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 shutdown_connections 2024-03-06T07:24:05.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.803+0000 7f81f23ac640 1 -- 172.21.15.122:0/2399079964 wait complete. 2024-03-06T07:24:05.816 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:05.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.914+0000 7fb261dc7640 1 Processor -- start 2024-03-06T07:24:05.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.914+0000 7fb261dc7640 1 -- start start 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb261dc7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb25c1424f0 0x7fb25c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb261dc7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c0cc980 con 0x7fb25c06da40 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb261dc7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c0ccc60 con 0x7fb25c0cd7a0 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb261dc7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25c0cd220 con 0x7fb25c1424f0 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb25bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb25c06da40 0x7fb25c071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36974/0 (socket says 172.21.15.122:36974) 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb25bfff640 1 -- 172.21.15.122:0/2450421133 learned_addr learned my addr 172.21.15.122:0/2450421133 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb25b7fe640 1 --2- 172.21.15.122:0/2450421133 >> v2:172.21.15.169:3300/0 conn(0x7fb25c1424f0 0x7fb25c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.915+0000 7fb25a7fc640 1 -- 172.21.15.122:0/2450421133 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540098779 0 0) 0x7fb25c0cc980 con 0x7fb25c06da40 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.916+0000 7fb25a7fc640 1 -- 172.21.15.122:0/2450421133 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb244003520 con 0x7fb25c06da40 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.916+0000 7fb25b7fe640 1 -- 172.21.15.122:0/2450421133 >> v1:172.21.15.169:6789/0 conn(0x7fb25c0cd7a0 legacy=0x7fb25c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.916+0000 7fb25a7fc640 1 -- 172.21.15.122:0/2450421133 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728218014 0 0) 0x7fb25c0ccc60 con 0x7fb25c0cd7a0 2024-03-06T07:24:05.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.916+0000 7fb25b7fe640 1 -- 172.21.15.122:0/2450421133 >> v1:172.21.15.122:6789/0 conn(0x7fb25c06da40 legacy=0x7fb25c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.916+0000 7fb25b7fe640 1 -- 172.21.15.122:0/2450421133 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0ccf40 con 0x7fb25c1424f0 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.916+0000 7fb25b7fe640 1 --2- 172.21.15.122:0/2450421133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c1424f0 0x7fb25c0cc2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0097a0 tx=0x7fb24c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ca34b6e56cebf10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb25a7fc640 1 -- 172.21.15.122:0/2450421133 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c034070 con 0x7fb25c1424f0 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb25a7fc640 1 -- 172.21.15.122:0/2450421133 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c030610 con 0x7fb25c1424f0 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb25a7fc640 1 -- 172.21.15.122:0/2450421133 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0309e0 con 0x7fb25c1424f0 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 -- 172.21.15.122:0/2450421133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c1424f0 msgr2=0x7fb25c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 --2- 172.21.15.122:0/2450421133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c1424f0 0x7fb25c0cc2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0097a0 tx=0x7fb24c02b960 comp rx=0 tx=0).stop 2024-03-06T07:24:05.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 -- 172.21.15.122:0/2450421133 shutdown_connections 2024-03-06T07:24:05.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 --2- 172.21.15.122:0/2450421133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c1424f0 0x7fb25c0cc2f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 -- 172.21.15.122:0/2450421133 >> 172.21.15.122:0/2450421133 conn(0x7fb25c0cf8a0 msgr2=0x7fb25c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 -- 172.21.15.122:0/2450421133 shutdown_connections 2024-03-06T07:24:05.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.917+0000 7fb261dc7640 1 -- 172.21.15.122:0/2450421133 wait complete. 2024-03-06T07:24:05.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.918+0000 7fb261dc7640 1 Processor -- start 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 -- start start 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb25c06da40 0x7fb25c13e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c0cd7a0 0x7fb25c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 0x7fb25c15c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c15d070 con 0x7fb25c1424f0 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c15cef0 con 0x7fb25c0cd7a0 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb261dc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c15d1f0 con 0x7fb25c06da40 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 0x7fb25c15c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 0x7fb25c15c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38588/0 (socket says 172.21.15.122:38588) 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb25affd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c0cd7a0 0x7fb25c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb25bfff640 1 -- 172.21.15.122:0/704813306 learned_addr learned my addr 172.21.15.122:0/704813306 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.919+0000 7fb25b7fe640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb25c06da40 0x7fb25c13e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb25bfff640 1 -- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb25c06da40 msgr2=0x7fb25c13e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb25bfff640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb25c06da40 0x7fb25c13e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb25bfff640 1 -- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c0cd7a0 msgr2=0x7fb25c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb25bfff640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c0cd7a0 0x7fb25c13f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb25bfff640 1 -- 172.21.15.122:0/704813306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c1400c0 con 0x7fb25c1424f0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb25bfff640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 0x7fb25c15c300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb25400cae0 tx=0x7fb25400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254017070 con 0x7fb25c1424f0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25400ac40 con 0x7fb25c1424f0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254012640 con 0x7fb25c1424f0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c1403a0 con 0x7fb25c1424f0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.920+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c140960 con 0x7fb25c1424f0 2024-03-06T07:24:05.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.921+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c15bf10 con 0x7fb25c1424f0 2024-03-06T07:24:05.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.922+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb25401a070 con 0x7fb25c1424f0 2024-03-06T07:24:05.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.922+0000 7fb243fff640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.922+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb254089fa0 con 0x7fb25c1424f0 2024-03-06T07:24:05.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.922+0000 7fb25b7fe640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.923+0000 7fb25b7fe640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb24806b4d0 0x7fb24806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb24c036000 tx=0x7fb24c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:05.926+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb25c15bf10 con 0x7fb25c1424f0 2024-03-06T07:24:06.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.099+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb25c04f330 con 0x7fb25c1424f0 2024-03-06T07:24:06.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.101+0000 7fb243fff640 1 -- 172.21.15.122:0/704813306 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16613 (secure 0 0 0) 0x7fb2540587e0 con 0x7fb25c1424f0 2024-03-06T07:24:06.103 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:06.103 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":45,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:41.897227+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb24806b4d0 msgr2=0x7fb24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb24806b4d0 0x7fb24806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb24c036000 tx=0x7fb24c03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 msgr2=0x7fb25c15c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 0x7fb25c15c300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb25400cae0 tx=0x7fb25400a060 comp rx=0 tx=0).stop 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 shutdown_connections 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb25c1424f0 0x7fb25c15c300 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb25c0cd7a0 0x7fb25c13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 --2- 172.21.15.122:0/704813306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb25c06da40 0x7fb25c13e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.102+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 >> 172.21.15.122:0/704813306 conn(0x7fb25c0cf8a0 msgr2=0x7fb25c0cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.103+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 shutdown_connections 2024-03-06T07:24:06.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.103+0000 7fb261dc7640 1 -- 172.21.15.122:0/704813306 wait complete. 2024-03-06T07:24:06.116 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:06.116 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:06.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.215+0000 7f69dbd9a640 1 Processor -- start 2024-03-06T07:24:06.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.215+0000 7f69dbd9a640 1 -- start start 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69dbd9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69d40d22b0 0x7f69d40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69dbd9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40b41f0 0x7f69d40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69dbd9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d40b3430 con 0x7f69d415dfe0 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69dbd9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d40d2c50 con 0x7f69d40d22b0 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69dbd9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d40b31e0 con 0x7f69d40b41f0 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69d40d22b0 0x7f69d40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69d40d22b0 0x7f69d40d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38596/0 (socket says 172.21.15.122:38596) 2024-03-06T07:24:06.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 -- 172.21.15.122:0/2523082156 learned_addr learned my addr 172.21.15.122:0/2523082156 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:06.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69da337640 1 --2- 172.21.15.122:0/2523082156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40b41f0 0x7f69d40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 -- 172.21.15.122:0/2523082156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40b41f0 msgr2=0x7f69d40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 --2- 172.21.15.122:0/2523082156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40b41f0 0x7f69d40b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 -- 172.21.15.122:0/2523082156 >> v1:172.21.15.169:6789/0 conn(0x7f69d415dfe0 legacy=0x7f69d40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d9335640 1 -- 172.21.15.122:0/2523082156 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d40b3710 con 0x7f69d40d22b0 2024-03-06T07:24:06.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.216+0000 7f69d8b34640 1 -- 172.21.15.122:0/2523082156 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775439780 0 0) 0x7f69d40b3430 con 0x7f69d415dfe0 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69d9335640 1 --2- 172.21.15.122:0/2523082156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40d22b0 0x7f69d40d2710 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009870 tx=0x7f69c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbfb79a206086df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69d8b34640 1 -- 172.21.15.122:0/2523082156 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8034070 con 0x7f69d40d22b0 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69d8b34640 1 -- 172.21.15.122:0/2523082156 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c8030840 con 0x7f69d40d22b0 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69d8b34640 1 -- 172.21.15.122:0/2523082156 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8030c10 con 0x7f69d40d22b0 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2523082156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40d22b0 msgr2=0x7f69d40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2523082156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40d22b0 0x7f69d40d2710 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009870 tx=0x7f69c802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2523082156 shutdown_connections 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2523082156 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40b41f0 0x7f69d40b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.217+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2523082156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40d22b0 0x7f69d40d2710 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.218+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2523082156 >> 172.21.15.122:0/2523082156 conn(0x7f69d40d3cd0 msgr2=0x7f69d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.218+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2523082156 shutdown_connections 2024-03-06T07:24:06.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.218+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2523082156 wait complete. 2024-03-06T07:24:06.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 Processor -- start 2024-03-06T07:24:06.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 -- start start 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 0x7f69d409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40d22b0 0x7f69d409f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d415dfe0 0x7f69d40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d40cfa00 con 0x7f69d40b41f0 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d40b3d60 con 0x7f69d415dfe0 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69dbd9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d40d1c70 con 0x7f69d40d22b0 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.219+0000 7f69d9b36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 0x7f69d409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 0x7f69d409ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38608/0 (socket says 172.21.15.122:38608) 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 -- 172.21.15.122:0/2452647786 learned_addr learned my addr 172.21.15.122:0/2452647786 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:06.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9335640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40d22b0 0x7f69d409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69da337640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d415dfe0 0x7f69d40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 -- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40d22b0 msgr2=0x7f69d409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40d22b0 0x7f69d409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 -- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d415dfe0 msgr2=0x7f69d40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d415dfe0 0x7f69d40cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 -- 172.21.15.122:0/2452647786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d409fa30 con 0x7f69d40b41f0 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9335640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40d22b0 0x7f69d409f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69da337640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d415dfe0 0x7f69d40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d9b36640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 0x7f69d409ecc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00c7f0 tx=0x7f69cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc017070 con 0x7f69d40b41f0 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69cc003da0 con 0x7f69d40b41f0 2024-03-06T07:24:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d409fd10 con 0x7f69d40b41f0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc005290 con 0x7f69d40b41f0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.220+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d40a02d0 con 0x7f69d40b41f0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.221+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d404f330 con 0x7f69d40b41f0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.221+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69cc01a070 con 0x7f69d40b41f0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.222+0000 7f69d27fc640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.222+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69cc0898d0 con 0x7f69d40b41f0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.222+0000 7f69d9335640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.222+0000 7f69d9335640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69c006b4d0 0x7f69c006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.227+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69cc058750 con 0x7f69d40b41f0 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.433+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69d40b2c70 con 0x7f69d40b41f0 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.434+0000 7f69d27fc640 1 -- 172.21.15.122:0/2452647786 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f69cc058050 con 0x7f69d40b41f0 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:40.243216+0000","modified":"2024-03-06T07:23:42.237773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/3905558085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3905558085},{"type":"v1","addr":"172.21.15.169:6835","nonce":3905558085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":3}]} 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 21 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.435+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69c006b4d0 msgr2=0x7f69c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.435+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69c006b4d0 0x7f69c006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.435+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 msgr2=0x7f69d409ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.435+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 0x7f69d409ecc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00c7f0 tx=0x7f69cc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 shutdown_connections 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d415dfe0 0x7f69d40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69d40d22b0 0x7f69d409f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 --2- 172.21.15.122:0/2452647786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69d40b41f0 0x7f69d409ecc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 >> 172.21.15.122:0/2452647786 conn(0x7f69d40d3cd0 msgr2=0x7f69d406c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 shutdown_connections 2024-03-06T07:24:06.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.436+0000 7f69dbd9a640 1 -- 172.21.15.122:0/2452647786 wait complete. 2024-03-06T07:24:06.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:06.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.547+0000 7f126df3e640 1 Processor -- start 2024-03-06T07:24:06.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.547+0000 7f126df3e640 1 -- start start 2024-03-06T07:24:06.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f126df3e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12680cc9e0 0x7f126806db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f126df3e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f126806e0a0 0x7f12680ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f126df3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126815e440 con 0x7f12680cf100 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f126df3e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126815e700 con 0x7f126806e0a0 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f126df3e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126815e9e0 con 0x7f12680cc9e0 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f126806e0a0 0x7f12680ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f126806e0a0 0x7f12680ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38610/0 (socket says 172.21.15.122:38610) 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 -- 172.21.15.122:0/777693162 learned_addr learned my addr 172.21.15.122:0/777693162 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1266ffd640 1 --2- 172.21.15.122:0/777693162 >> v2:172.21.15.169:3300/0 conn(0x7f12680cc9e0 0x7f126806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 -- 172.21.15.122:0/777693162 >> v2:172.21.15.169:3300/0 conn(0x7f12680cc9e0 msgr2=0x7f126806db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 --2- 172.21.15.122:0/777693162 >> v2:172.21.15.169:3300/0 conn(0x7f12680cc9e0 0x7f126806db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 -- 172.21.15.122:0/777693162 >> v1:172.21.15.169:6789/0 conn(0x7f12680cf100 legacy=0x7f12680cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.548+0000 7f1267fff640 1 -- 172.21.15.122:0/777693162 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126815ecc0 con 0x7f126806e0a0 2024-03-06T07:24:06.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.549+0000 7f1267fff640 1 --2- 172.21.15.122:0/777693162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f126806e0a0 0x7f12680ad860 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f12680731a0 tx=0x7f126002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c322323aead2c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.549+0000 7f12667fc640 1 -- 172.21.15.122:0/777693162 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126002ca80 con 0x7f126806e0a0 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.549+0000 7f12667fc640 1 -- 172.21.15.122:0/777693162 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126002cc20 con 0x7f126806e0a0 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.549+0000 7f12667fc640 1 -- 172.21.15.122:0/777693162 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12600343f0 con 0x7f126806e0a0 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.549+0000 7f126df3e640 1 -- 172.21.15.122:0/777693162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f126806e0a0 msgr2=0x7f12680ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.549+0000 7f126df3e640 1 --2- 172.21.15.122:0/777693162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f126806e0a0 0x7f12680ad860 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f12680731a0 tx=0x7f126002bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.550+0000 7f126df3e640 1 -- 172.21.15.122:0/777693162 shutdown_connections 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.550+0000 7f126df3e640 1 --2- 172.21.15.122:0/777693162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f126806e0a0 0x7f12680ad860 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.550+0000 7f126df3e640 1 --2- 172.21.15.122:0/777693162 >> v2:172.21.15.169:3300/0 conn(0x7f12680cc9e0 0x7f126806db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.550+0000 7f126df3e640 1 -- 172.21.15.122:0/777693162 >> 172.21.15.122:0/777693162 conn(0x7f12680cf8a0 msgr2=0x7f12680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.550+0000 7f126df3e640 1 -- 172.21.15.122:0/777693162 shutdown_connections 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.550+0000 7f126df3e640 1 -- 172.21.15.122:0/777693162 wait complete. 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.551+0000 7f126df3e640 1 Processor -- start 2024-03-06T07:24:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.551+0000 7f126df3e640 1 -- start start 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f126806e0a0 0x7f126814a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 0x7f1268071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12680cf100 0x7f1268071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126806e4f0 con 0x7f12680cc9e0 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12680cf4e0 con 0x7f12680cf100 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126806e670 con 0x7f126806e0a0 2024-03-06T07:24:06.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 0x7f1268071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 0x7f1268071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38622/0 (socket says 172.21.15.122:38622) 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 -- 172.21.15.122:0/312248858 learned_addr learned my addr 172.21.15.122:0/312248858 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1267fff640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12680cf100 0x7f1268071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f12677fe640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f126806e0a0 0x7f126814a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 -- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f126806e0a0 msgr2=0x7f126814a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f126806e0a0 0x7f126814a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 -- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12680cf100 msgr2=0x7f1268071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12680cf100 0x7f1268071d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 -- 172.21.15.122:0/312248858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126814a870 con 0x7f12680cc9e0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f1266ffd640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 0x7f1268071850 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f125c00e9b0 tx=0x7f125c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c017070 con 0x7f12680cc9e0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c00c9a0 con 0x7f12680cc9e0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12680afcc0 con 0x7f12680cc9e0 2024-03-06T07:24:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c00cd90 con 0x7f12680cc9e0 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.552+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12680b0280 con 0x7f12680cc9e0 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.553+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126804f330 con 0x7f12680cc9e0 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.554+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f125c01a070 con 0x7f12680cc9e0 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.554+0000 7f124ffff640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b500 0x7f125406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.554+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f125c08a450 con 0x7f12680cc9e0 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.554+0000 7f12677fe640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b500 0x7f125406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.555+0000 7f12677fe640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b500 0x7f125406d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1268157b80 tx=0x7f1258009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.558+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f125c058d10 con 0x7f12680cc9e0 2024-03-06T07:24:06.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.769+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12680d0770 con 0x7f12680cc9e0 2024-03-06T07:24:06.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.770+0000 7f124ffff640 1 -- 172.21.15.122:0/312248858 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f125c05cdb0 con 0x7f12680cc9e0 2024-03-06T07:24:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:06.772 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:40.243216+0000","modified":"2024-03-06T07:23:42.237773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/3905558085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3905558085},{"type":"v1","addr":"172.21.15.169:6835","nonce":3905558085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":3}]} 2024-03-06T07:24:06.772 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 21 2024-03-06T07:24:06.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b500 msgr2=0x7f125406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b500 0x7f125406d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1268157b80 tx=0x7f1258009040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 msgr2=0x7f1268071850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 0x7f1268071850 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f125c00e9b0 tx=0x7f125c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:06.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 shutdown_connections 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b500 0x7f125406d9c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12680cf100 0x7f1268071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12680cc9e0 0x7f1268071850 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 --2- 172.21.15.122:0/312248858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f126806e0a0 0x7f126814a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 >> 172.21.15.122:0/312248858 conn(0x7f12680cf8a0 msgr2=0x7f1268160c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 shutdown_connections 2024-03-06T07:24:06.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.772+0000 7f126df3e640 1 -- 172.21.15.122:0/312248858 wait complete. 2024-03-06T07:24:06.785 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:06.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.883+0000 7f600befb640 1 Processor -- start 2024-03-06T07:24:06.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.883+0000 7f600befb640 1 -- start start 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600befb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f600415dfe0 0x7f60040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600befb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040cf830 0x7f60040d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600befb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f60040d27f0 0x7f60040d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600befb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60040d2c50 con 0x7f600415dfe0 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600befb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60040b31e0 con 0x7f60040d27f0 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600befb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60040b3d60 con 0x7f60040cf830 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f6009c97640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f600415dfe0 0x7f60040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f6009c97640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f600415dfe0 0x7f60040cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38628/0 (socket says 172.21.15.122:38628) 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f6009c97640 1 -- 172.21.15.122:0/2061386638 learned_addr learned my addr 172.21.15.122:0/2061386638 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f6009496640 1 --2- 172.21.15.122:0/2061386638 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040cf830 0x7f60040d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.884+0000 7f600a498640 1 --2- 172.21.15.122:0/2061386638 >> v2:172.21.15.169:3300/0 conn(0x7f60040d27f0 0x7f60040d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.885+0000 7f6009c97640 1 -- 172.21.15.122:0/2061386638 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040cf830 msgr2=0x7f60040d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.885+0000 7f6009c97640 1 --2- 172.21.15.122:0/2061386638 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040cf830 0x7f60040d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.885+0000 7f6009c97640 1 -- 172.21.15.122:0/2061386638 >> v2:172.21.15.169:3300/0 conn(0x7f60040d27f0 msgr2=0x7f60040d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.885+0000 7f6009c97640 1 --2- 172.21.15.122:0/2061386638 >> v2:172.21.15.169:3300/0 conn(0x7f60040d27f0 0x7f60040d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.885+0000 7f6009c97640 1 -- 172.21.15.122:0/2061386638 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60040b3430 con 0x7f600415dfe0 2024-03-06T07:24:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.885+0000 7f6009c97640 1 --2- 172.21.15.122:0/2061386638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f600415dfe0 0x7f60040cf2f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6004067ad0 tx=0x7f600002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357c5d07998e74d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f6008c95640 1 -- 172.21.15.122:0/2061386638 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000035020 con 0x7f600415dfe0 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f6008c95640 1 -- 172.21.15.122:0/2061386638 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6000031870 con 0x7f600415dfe0 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f6008c95640 1 -- 172.21.15.122:0/2061386638 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000031c40 con 0x7f600415dfe0 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 -- 172.21.15.122:0/2061386638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f600415dfe0 msgr2=0x7f60040cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 --2- 172.21.15.122:0/2061386638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f600415dfe0 0x7f60040cf2f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6004067ad0 tx=0x7f600002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 -- 172.21.15.122:0/2061386638 shutdown_connections 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 --2- 172.21.15.122:0/2061386638 >> v2:172.21.15.169:3300/0 conn(0x7f60040d27f0 0x7f60040d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 --2- 172.21.15.122:0/2061386638 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040cf830 0x7f60040d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 --2- 172.21.15.122:0/2061386638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f600415dfe0 0x7f60040cf2f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 -- 172.21.15.122:0/2061386638 >> 172.21.15.122:0/2061386638 conn(0x7f60040d3cd0 msgr2=0x7f60040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 -- 172.21.15.122:0/2061386638 shutdown_connections 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.886+0000 7f600befb640 1 -- 172.21.15.122:0/2061386638 wait complete. 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.887+0000 7f600befb640 1 Processor -- start 2024-03-06T07:24:06.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.887+0000 7f600befb640 1 -- start start 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600befb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 0x7f60041f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600befb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040d27f0 0x7f60041f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600befb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f600415dfe0 0x7f60041f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600befb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60040d1c70 con 0x7f60040cf830 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600befb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60040b39f0 con 0x7f600415dfe0 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600befb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60040d1df0 con 0x7f60040d27f0 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f6009c97640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 0x7f60041f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f6009c97640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 0x7f60041f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38642/0 (socket says 172.21.15.122:38642) 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f6009c97640 1 -- 172.21.15.122:0/2510442554 learned_addr learned my addr 172.21.15.122:0/2510442554 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:06.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f600a498640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f600415dfe0 0x7f60041f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.888+0000 7f6009496640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040d27f0 0x7f60041f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f6009c97640 1 -- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040d27f0 msgr2=0x7f60041f7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f6009c97640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040d27f0 0x7f60041f7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f6009c97640 1 -- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f600415dfe0 msgr2=0x7f60041f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f6009c97640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f600415dfe0 0x7f60041f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f6009c97640 1 -- 172.21.15.122:0/2510442554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60040cae40 con 0x7f60040cf830 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f6009c97640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 0x7f60041f3bf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f600002b950 tx=0x7f60000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000040070 con 0x7f60040cf830 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6000031e10 con 0x7f60040cf830 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60041f4240 con 0x7f60040cf830 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000035040 con 0x7f60040cf830 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.889+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60041f4800 con 0x7f60040cf830 2024-03-06T07:24:06.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.890+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600404f330 con 0x7f60040cf830 2024-03-06T07:24:06.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.890+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f600004b060 con 0x7f60040cf830 2024-03-06T07:24:06.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.891+0000 7f5ffa7fc640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fe806b500 0x7f5fe806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.891+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60000abc20 con 0x7f60040cf830 2024-03-06T07:24:06.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.891+0000 7f6009496640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fe806b500 0x7f5fe806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.891+0000 7f6009496640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fe806b500 0x7f5fe806d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f600415c520 tx=0x7f5ff4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:06.896+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f600007a4e0 con 0x7f60040cf830 2024-03-06T07:24:07.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.068+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60040b0c10 con 0x7f60040cf830 2024-03-06T07:24:07.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.069+0000 7f5ffa7fc640 1 -- 172.21.15.122:0/2510442554 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16613 (secure 0 0 0) 0x7f600007e580 con 0x7f60040cf830 2024-03-06T07:24:07.071 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:07.071 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":45,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:23:41.897227+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:38.463254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:39.205152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:07.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fe806b500 msgr2=0x7f5fe806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fe806b500 0x7f5fe806d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f600415c520 tx=0x7f5ff4009190 comp rx=0 tx=0).stop 2024-03-06T07:24:07.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 msgr2=0x7f60041f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 0x7f60041f3bf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f600002b950 tx=0x7f60000046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 shutdown_connections 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fe806b500 0x7f5fe806d9c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f600415dfe0 0x7f60041f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60040d27f0 0x7f60041f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 --2- 172.21.15.122:0/2510442554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60040cf830 0x7f60041f3bf0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 >> 172.21.15.122:0/2510442554 conn(0x7f60040d3cd0 msgr2=0x7f60040d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 shutdown_connections 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.072+0000 7f600befb640 1 -- 172.21.15.122:0/2510442554 wait complete. 2024-03-06T07:24:07.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:07.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.184+0000 7f177ed7a640 1 Processor -- start 2024-03-06T07:24:07.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.184+0000 7f177ed7a640 1 -- start start 2024-03-06T07:24:07.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177ed7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177ed7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f17780b3240 0x7f17780b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177ed7a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177815e3f0 con 0x7f17780b2910 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177ed7a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780b1f40 con 0x7f17780b3240 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177ed7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780b1c10 con 0x7f177815e010 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177d317640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f17780b2910 0x7f17780d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36988/0 (socket says 172.21.15.122:36988) 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177d317640 1 -- 172.21.15.122:0/4098915464 learned_addr learned my addr 172.21.15.122:0/4098915464 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f177cb16640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f1777fff640 1 --2- 172.21.15.122:0/4098915464 >> v2:172.21.15.169:3300/0 conn(0x7f17780b3240 0x7f17780b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.185+0000 7f17777fe640 1 -- 172.21.15.122:0/4098915464 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048356023 0 0) 0x7f177815e3f0 con 0x7f17780b2910 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f17777fe640 1 -- 172.21.15.122:0/4098915464 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1760003520 con 0x7f17780b2910 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f1777fff640 1 -- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 msgr2=0x7f17780cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f1777fff640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f1777fff640 1 -- 172.21.15.122:0/4098915464 >> v1:172.21.15.122:6789/0 conn(0x7f17780b2910 legacy=0x7f17780d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f1777fff640 1 -- 172.21.15.122:0/4098915464 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177806d1b0 con 0x7f17780b3240 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f177cb16640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.186+0000 7f1777fff640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780b3240 0x7f17780b23d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097d0 tx=0x7f176c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bbf90e52c85c15e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f17777fe640 1 -- 172.21.15.122:0/4098915464 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c031770 con 0x7f17780b3240 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f17777fe640 1 -- 172.21.15.122:0/4098915464 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c037070 con 0x7f17780b3240 2024-03-06T07:24:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f17777fe640 1 -- 172.21.15.122:0/4098915464 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c031b10 con 0x7f17780b3240 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 -- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780b3240 msgr2=0x7f17780b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780b3240 0x7f17780b23d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097d0 tx=0x7f176c02b810 comp rx=0 tx=0).stop 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 -- 172.21.15.122:0/4098915464 shutdown_connections 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780b3240 0x7f17780b23d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 --2- 172.21.15.122:0/4098915464 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 -- 172.21.15.122:0/4098915464 >> 172.21.15.122:0/4098915464 conn(0x7f177809a9f0 msgr2=0x7f177809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 -- 172.21.15.122:0/4098915464 shutdown_connections 2024-03-06T07:24:07.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.187+0000 7f177ed7a640 1 -- 172.21.15.122:0/4098915464 wait complete. 2024-03-06T07:24:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.188+0000 7f177ed7a640 1 Processor -- start 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.188+0000 7f177ed7a640 1 -- start start 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177ed7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177ed7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780af800 0x7f17780b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177ed7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 0x7f17780b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177ed7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780cf310 con 0x7f177814f130 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177ed7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177806ced0 con 0x7f17780af800 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177ed7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780d21e0 con 0x7f177815e010 2024-03-06T07:24:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177d317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 0x7f17780b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177d317640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 0x7f17780b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38646/0 (socket says 172.21.15.122:38646) 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177d317640 1 -- 172.21.15.122:0/1056227575 learned_addr learned my addr 172.21.15.122:0/1056227575 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f177cb16640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.189+0000 7f1777fff640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780af800 0x7f17780b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177cb16640 1 -- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780af800 msgr2=0x7f17780b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177cb16640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780af800 0x7f17780b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177cb16640 1 -- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 msgr2=0x7f17780b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177cb16640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 0x7f17780b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177cb16640 1 -- 172.21.15.122:0/1056227575 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780afd10 con 0x7f177815e010 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177d317640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 0x7f17780b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:07.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f1777fff640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780af800 0x7f17780b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177cb16640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780c80c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f176800e8a0 tx=0x7f176800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768017070 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176800c940 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176800cd30 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17780afff0 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.190+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17780b05b0 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.191+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177804f330 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.191+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f176801a070 con 0x7f177815e010 2024-03-06T07:24:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.192+0000 7f17757fa640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f176406b4d0 0x7f176406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.192+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f176808a3b0 con 0x7f177815e010 2024-03-06T07:24:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.192+0000 7f1777fff640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f176406b4d0 0x7f176406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.192+0000 7f1777fff640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f176406b4d0 0x7f176406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f176c036000 tx=0x7f176c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.197+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1768058b50 con 0x7f177815e010 2024-03-06T07:24:07.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.404+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f17780d2980 con 0x7f177815e010 2024-03-06T07:24:07.405 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:24:07.405+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:07.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.409+0000 7f17757fa640 1 -- 172.21.15.122:0/1056227575 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) v1 ==== 161+0+0 (secure 0 0 0) 0x7f176805cbf0 con 0x7f177815e010 2024-03-06T07:24:07.410 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:07.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f176406b4d0 msgr2=0x7f176406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f176406b4d0 0x7f176406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f176c036000 tx=0x7f176c03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 msgr2=0x7f17780c80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780c80c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f176800e8a0 tx=0x7f176800ed60 comp rx=0 tx=0).stop 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 shutdown_connections 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f176406b4d0 0x7f176406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f177814f130 0x7f17780b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17780af800 0x7f17780b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 --2- 172.21.15.122:0/1056227575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f177815e010 0x7f17780c80c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 >> 172.21.15.122:0/1056227575 conn(0x7f177809a9f0 msgr2=0x7f17780ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 shutdown_connections 2024-03-06T07:24:07.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.411+0000 7f177ed7a640 1 -- 172.21.15.122:0/1056227575 wait complete. 2024-03-06T07:24:07.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:07.446 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:24:07.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.525+0000 7f5355e88640 1 Processor -- start 2024-03-06T07:24:07.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.525+0000 7f5355e88640 1 -- start start 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f5355e88640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f53500cfa90 0x7f53500cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f5355e88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53500b33a0 0x7f53500b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f5355e88640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5350158ac0 con 0x7f53501586e0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f5355e88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535015b2a0 con 0x7f53500b33a0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f5355e88640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535015b540 con 0x7f53500cfa90 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53500b33a0 0x7f53500b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53500b33a0 0x7f53500b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38658/0 (socket says 172.21.15.122:38658) 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 -- 172.21.15.122:0/4130357546 learned_addr learned my addr 172.21.15.122:0/4130357546 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534effd640 1 --2- 172.21.15.122:0/4130357546 >> v2:172.21.15.169:3300/0 conn(0x7f53500cfa90 0x7f53500cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 -- 172.21.15.122:0/4130357546 >> v2:172.21.15.169:3300/0 conn(0x7f53500cfa90 msgr2=0x7f53500cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 --2- 172.21.15.122:0/4130357546 >> v2:172.21.15.169:3300/0 conn(0x7f53500cfa90 0x7f53500cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 -- 172.21.15.122:0/4130357546 >> v1:172.21.15.169:6789/0 conn(0x7f53501586e0 legacy=0x7f53500aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.526+0000 7f534ffff640 1 -- 172.21.15.122:0/4130357546 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535015b820 con 0x7f53500b33a0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.527+0000 7f534ffff640 1 --2- 172.21.15.122:0/4130357546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53500b37f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5350072e00 tx=0x7f534802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befa8296e53fd5bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.527+0000 7f534e7fc640 1 -- 172.21.15.122:0/4130357546 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534802ca80 con 0x7f53500b33a0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.527+0000 7f534e7fc640 1 -- 172.21.15.122:0/4130357546 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534802cc20 con 0x7f53500b33a0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.527+0000 7f534e7fc640 1 -- 172.21.15.122:0/4130357546 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53480343f0 con 0x7f53500b33a0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 -- 172.21.15.122:0/4130357546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 msgr2=0x7f53500b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 --2- 172.21.15.122:0/4130357546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53500b37f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5350072e00 tx=0x7f534802bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 -- 172.21.15.122:0/4130357546 shutdown_connections 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 --2- 172.21.15.122:0/4130357546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53500b37f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 --2- 172.21.15.122:0/4130357546 >> v2:172.21.15.169:3300/0 conn(0x7f53500cfa90 0x7f53500cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 -- 172.21.15.122:0/4130357546 >> 172.21.15.122:0/4130357546 conn(0x7f535006b190 msgr2=0x7f535006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 -- 172.21.15.122:0/4130357546 shutdown_connections 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.528+0000 7f5355e88640 1 -- 172.21.15.122:0/4130357546 wait complete. 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.529+0000 7f5355e88640 1 Processor -- start 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.529+0000 7f5355e88640 1 -- start start 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53501462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53500cfa90 0x7f53501f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53501586e0 0x7f53501f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500af150 con 0x7f53500b33a0 2024-03-06T07:24:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535015bde0 con 0x7f53500cfa90 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5350071350 con 0x7f53501586e0 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53501462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53501462a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38660/0 (socket says 172.21.15.122:38660) 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 -- 172.21.15.122:0/507518299 learned_addr learned my addr 172.21.15.122:0/507518299 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534ffff640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53501586e0 0x7f53501f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534effd640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53500cfa90 0x7f53501f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 -- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53501586e0 msgr2=0x7f53501f5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53501586e0 0x7f53501f5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 -- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53500cfa90 msgr2=0x7f53501f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53500cfa90 0x7f53501f4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 -- 172.21.15.122:0/507518299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5350152d60 con 0x7f53500b33a0 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534effd640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53500cfa90 0x7f53501f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534ffff640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53501586e0 0x7f53501f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f534f7fe640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53501462a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f534400c430 tx=0x7f534400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344011790 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5344017070 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5350153040 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344011bb0 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.530+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5350150da0 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.531+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53500732b0 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.532+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f534401c460 con 0x7f53500b33a0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.532+0000 7f5337fff640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f533c06b4d0 0x7f533c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.532+0000 7f534effd640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f533c06b4d0 0x7f533c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.532+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f534408a150 con 0x7f53500b33a0 2024-03-06T07:24:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.533+0000 7f534effd640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f533c06b4d0 0x7f533c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5350154740 tx=0x7f5340008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.536+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5344058f70 con 0x7f53500b33a0 2024-03-06T07:24:07.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:07.748+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5350068ff0 con 0x7f53500b33a0 2024-03-06T07:24:08.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.407+0000 7f5337fff640 1 -- 172.21.15.122:0/507518299 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5344058870 con 0x7f53500b33a0 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.409+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f533c06b4d0 msgr2=0x7f533c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.409+0000 7f5355e88640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f533c06b4d0 0x7f533c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5350154740 tx=0x7f5340008040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.409+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 msgr2=0x7f53501462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.409+0000 7f5355e88640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53501462a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f534400c430 tx=0x7f534400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 shutdown_connections 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f533c06b4d0 0x7f533c06d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53501586e0 0x7f53501f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53500cfa90 0x7f53501f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 --2- 172.21.15.122:0/507518299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53500b33a0 0x7f53501462a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 >> 172.21.15.122:0/507518299 conn(0x7f535006b190 msgr2=0x7f535006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 shutdown_connections 2024-03-06T07:24:08.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.410+0000 7f5355e88640 1 -- 172.21.15.122:0/507518299 wait complete. 2024-03-06T07:24:08.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.529+0000 7f5eb62df640 1 Processor -- start 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.529+0000 7f5eb62df640 1 -- start start 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.529+0000 7f5eb62df640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5eb00b4e10 0x7f5eb0168c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.529+0000 7f5eb62df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00cfb30 con 0x7f5eb015dff0 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.529+0000 7f5eb62df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00b3320 con 0x7f5eb00d1d90 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.529+0000 7f5eb62df640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00b3600 con 0x7f5eb00b4e10 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5eb015dff0 0x7f5eb00cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37002/0 (socket says 172.21.15.122:37002) 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaffff640 1 -- 172.21.15.122:0/1987167152 learned_addr learned my addr 172.21.15.122:0/1987167152 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 --2- 172.21.15.122:0/1987167152 >> v2:172.21.15.169:3300/0 conn(0x7f5eb00b4e10 0x7f5eb0168c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724610245 0 0) 0x7f5eb00cfb30 con 0x7f5eb015dff0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e98003520 con 0x7f5eb015dff0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815739207 0 0) 0x7f5e98003520 con 0x7f5eb015dff0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 >> v2:172.21.15.169:3300/0 conn(0x7f5eb00b4e10 msgr2=0x7f5eb0168c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 --2- 172.21.15.122:0/1987167152 >> v2:172.21.15.169:3300/0 conn(0x7f5eb00b4e10 0x7f5eb0168c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 >> v1:172.21.15.169:6789/0 conn(0x7f5eb00d1d90 legacy=0x7f5eb00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00b38e0 con 0x7f5eb015dff0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f5ea0001670 con 0x7f5eb015dff0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 >> v1:172.21.15.122:6789/0 conn(0x7f5eb015dff0 legacy=0x7f5eb00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e980039b0 0x7f5e98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e980043b0 0x7f5e98004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5e9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00b3600 con 0x7f5e98005080 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9800bf30 con 0x7f5e980039b0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9800c130 con 0x7f5e980043b0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5e9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaffff640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e980039b0 0x7f5e98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaf7fe640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e980043b0 0x7f5e98004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 -- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e980043b0 msgr2=0x7f5e98004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e980043b0 0x7f5e98004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 -- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e980039b0 msgr2=0x7f5e98003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e980039b0 0x7f5e98003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eb487c640 1 -- 172.21.15.122:0/1987167152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e98003520 con 0x7f5e98005080 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.530+0000 7f5eaffff640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e980039b0 0x7f5e98003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.531+0000 7f5eb487c640 1 -- 172.21.15.122:0/1987167152 reap_dead start 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.531+0000 7f5eb487c640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5e9800b9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009fc0 tx=0x7f5e98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33a24ae8f97b191 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.531+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8009c10 con 0x7f5e98005080 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.531+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea802cca0 con 0x7f5e98005080 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.531+0000 7f5eaeffd640 1 -- 172.21.15.122:0/1987167152 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8003bd0 con 0x7f5e98005080 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 -- 172.21.15.122:0/1987167152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 msgr2=0x7f5e9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5e9800b9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009fc0 tx=0x7f5e98005830 comp rx=0 tx=0).stop 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 -- 172.21.15.122:0/1987167152 shutdown_connections 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 --2- 172.21.15.122:0/1987167152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5e9800b9c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 -- 172.21.15.122:0/1987167152 >> 172.21.15.122:0/1987167152 conn(0x7f5eb00d3d50 msgr2=0x7f5eb00d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 -- 172.21.15.122:0/1987167152 shutdown_connections 2024-03-06T07:24:08.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.532+0000 7f5eb62df640 1 -- 172.21.15.122:0/1987167152 wait complete. 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.533+0000 7f5eb62df640 1 Processor -- start 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.533+0000 7f5eb62df640 1 -- start start 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb62df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5eb00b9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb62df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5eb00cc620 0x7f5eb015f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb62df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb00b1ab0 0x7f5eb00b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb62df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00cf4f0 con 0x7f5e98005080 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb62df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00cf370 con 0x7f5eb00b1ab0 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb62df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00cf670 con 0x7f5eb00cc620 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5eb00b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5eb00b9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38674/0 (socket says 172.21.15.122:38674) 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 -- 172.21.15.122:0/3931372594 learned_addr learned my addr 172.21.15.122:0/3931372594 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:08.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eb487c640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb00b1ab0 0x7f5eb00b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaf7fe640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5eb00cc620 0x7f5eb015f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 -- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5eb00cc620 msgr2=0x7f5eb015f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5eb00cc620 0x7f5eb015f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 -- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb00b1ab0 msgr2=0x7f5eb00b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb00b1ab0 0x7f5eb00b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 -- 172.21.15.122:0/3931372594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0160070 con 0x7f5e98005080 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.534+0000 7f5eaffff640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5eb00b9180 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000e700 tx=0x7f5ea000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.535+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000c620 con 0x7f5e98005080 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.535+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea0018070 con 0x7f5e98005080 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.535+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb015e830 con 0x7f5e98005080 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.535+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000c9c0 con 0x7f5e98005080 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.535+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb015edf0 con 0x7f5e98005080 2024-03-06T07:24:08.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.536+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb015eff0 con 0x7f5e98005080 2024-03-06T07:24:08.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.536+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ea000cb60 con 0x7f5e98005080 2024-03-06T07:24:08.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.536+0000 7f5eacff9640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e9c06b500 0x7f5e9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.537+0000 7f5eaf7fe640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e9c06b500 0x7f5e9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.537+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5ea0089f40 con 0x7f5e98005080 2024-03-06T07:24:08.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.537+0000 7f5eaf7fe640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e9c06b500 0x7f5e9c06d9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5eb006bd20 tx=0x7f5ea4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.541+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5eb015eff0 con 0x7f5e98005080 2024-03-06T07:24:08.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:08.715+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5eb004f330 con 0x7f5e98005080 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.671+0000 7f5eacff9640 1 -- 172.21.15.122:0/3931372594 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5ea0058df0 con 0x7f5e98005080 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:09.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e9c06b500 msgr2=0x7f5e9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e9c06b500 0x7f5e9c06d9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5eb006bd20 tx=0x7f5ea4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 msgr2=0x7f5eb00b9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5eb00b9180 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000e700 tx=0x7f5ea000eb00 comp rx=0 tx=0).stop 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 shutdown_connections 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e9c06b500 0x7f5e9c06d9c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb00b1ab0 0x7f5eb00b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5eb00cc620 0x7f5eb015f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 --2- 172.21.15.122:0/3931372594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e98005080 0x7f5eb00b9180 secure :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000e700 tx=0x7f5ea000eb00 comp rx=0 tx=0).stop 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 >> 172.21.15.122:0/3931372594 conn(0x7f5eb00d3d50 msgr2=0x7f5eb006bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.675+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 shutdown_connections 2024-03-06T07:24:09.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.676+0000 7f5eb62df640 1 -- 172.21.15.122:0/3931372594 wait complete. 2024-03-06T07:24:09.691 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:09.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.798+0000 7fc29606b640 1 Processor -- start 2024-03-06T07:24:09.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.798+0000 7fc29606b640 1 -- start start 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc29606b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2900b3060 0x7fc290165000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc29606b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2900b4490 con 0x7fc2900b3fd0 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc29606b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2900cf4a0 con 0x7fc29015a490 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc29606b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2900b4750 con 0x7fc2900b3060 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc28ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2900b3060 0x7fc290165000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc28effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc2900b3fd0 0x7fc2900b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37018/0 (socket says 172.21.15.122:37018) 2024-03-06T07:24:09.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.799+0000 7fc28effd640 1 -- 172.21.15.122:0/3895464033 learned_addr learned my addr 172.21.15.122:0/3895464033 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:09.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28ffff640 1 -- 172.21.15.122:0/3895464033 >> v1:172.21.15.169:6789/0 conn(0x7fc29015a490 legacy=0x7fc29006df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28ffff640 1 -- 172.21.15.122:0/3895464033 >> v1:172.21.15.122:6789/0 conn(0x7fc2900b3fd0 legacy=0x7fc2900b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28ffff640 1 -- 172.21.15.122:0/3895464033 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2900b4a30 con 0x7fc2900b3060 2024-03-06T07:24:09.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28ffff640 1 --2- 172.21.15.122:0/3895464033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2900b3060 0x7fc290165000 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc288009d90 tx=0x7fc28802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aeb660cfd45ed16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28e7fc640 1 -- 172.21.15.122:0/3895464033 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288039070 con 0x7fc2900b3060 2024-03-06T07:24:09.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28e7fc640 1 -- 172.21.15.122:0/3895464033 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28802cd00 con 0x7fc2900b3060 2024-03-06T07:24:09.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.800+0000 7fc28e7fc640 1 -- 172.21.15.122:0/3895464033 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288034660 con 0x7fc2900b3060 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 -- 172.21.15.122:0/3895464033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2900b3060 msgr2=0x7fc290165000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 --2- 172.21.15.122:0/3895464033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2900b3060 0x7fc290165000 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc288009d90 tx=0x7fc28802bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 -- 172.21.15.122:0/3895464033 shutdown_connections 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 --2- 172.21.15.122:0/3895464033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2900b3060 0x7fc290165000 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 -- 172.21.15.122:0/3895464033 >> 172.21.15.122:0/3895464033 conn(0x7fc2900b1650 msgr2=0x7fc2900b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 -- 172.21.15.122:0/3895464033 shutdown_connections 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.801+0000 7fc29606b640 1 -- 172.21.15.122:0/3895464033 wait complete. 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.802+0000 7fc29606b640 1 Processor -- start 2024-03-06T07:24:09.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.802+0000 7fc29606b640 1 -- start start 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc29606b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2900b3060 0x7fc2901470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc29606b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2900b3fd0 0x7fc29015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc29606b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 0x7fc29015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc29606b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2900cf260 con 0x7fc29015a490 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc29606b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2900cf0e0 con 0x7fc2900b3fd0 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc29606b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29006c910 con 0x7fc2900b3060 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 0x7fc29015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 0x7fc29015d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38692/0 (socket says 172.21.15.122:38692) 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 -- 172.21.15.122:0/2669145875 learned_addr learned my addr 172.21.15.122:0/2669145875 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28effd640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2900b3fd0 0x7fc29015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28f7fe640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2900b3060 0x7fc2901470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 -- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2900b3060 msgr2=0x7fc2901470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2900b3060 0x7fc2901470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 -- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2900b3fd0 msgr2=0x7fc29015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.803+0000 7fc28ffff640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2900b3fd0 0x7fc29015cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc28ffff640 1 -- 172.21.15.122:0/2669145875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2901f51e0 con 0x7fc29015a490 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc28ffff640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 0x7fc29015d3e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc288009d90 tx=0x7fc288004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288039050 con 0x7fc29015a490 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288045070 con 0x7fc29015a490 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2901f54c0 con 0x7fc29015a490 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28803e830 con 0x7fc29015a490 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.804+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2901f5a80 con 0x7fc29015a490 2024-03-06T07:24:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.805+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2900687d0 con 0x7fc29015a490 2024-03-06T07:24:09.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.805+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc28803e9d0 con 0x7fc29015a490 2024-03-06T07:24:09.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.806+0000 7fc277fff640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27c06b4d0 0x7fc27c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.806+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fc2880acec0 con 0x7fc29015a490 2024-03-06T07:24:09.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.806+0000 7fc28f7fe640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27c06b4d0 0x7fc27c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.806+0000 7fc28f7fe640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27c06b4d0 0x7fc27c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc280007fd0 tx=0x7fc280009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.810+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc28807bea0 con 0x7fc29015a490 2024-03-06T07:24:09.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:09.985+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc29014f1d0 con 0x7fc29015a490 2024-03-06T07:24:10.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.676+0000 7fc277fff640 1 -- 172.21.15.122:0/2669145875 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc28807b7a0 con 0x7fc29015a490 2024-03-06T07:24:10.678 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:10.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27c06b4d0 msgr2=0x7fc27c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27c06b4d0 0x7fc27c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc280007fd0 tx=0x7fc280009040 comp rx=0 tx=0).stop 2024-03-06T07:24:10.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 msgr2=0x7fc29015d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 0x7fc29015d3e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc288009d90 tx=0x7fc288004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:10.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 shutdown_connections 2024-03-06T07:24:10.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc27c06b4d0 0x7fc27c06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29015a490 0x7fc29015d3e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2900b3fd0 0x7fc29015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 --2- 172.21.15.122:0/2669145875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2900b3060 0x7fc2901470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 >> 172.21.15.122:0/2669145875 conn(0x7fc2900b1650 msgr2=0x7fc29014f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 shutdown_connections 2024-03-06T07:24:10.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.680+0000 7fc29606b640 1 -- 172.21.15.122:0/2669145875 wait complete. 2024-03-06T07:24:10.696 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:10.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.804+0000 7fc906f0d640 1 Processor -- start 2024-03-06T07:24:10.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.804+0000 7fc906f0d640 1 -- start start 2024-03-06T07:24:10.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.804+0000 7fc906f0d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc90015c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.804+0000 7fc906f0d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc90006e2c0 0x7fc90006e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.804+0000 7fc906f0d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900070f50 con 0x7fc9000cdd10 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc906f0d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90006ec70 con 0x7fc90006e2c0 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc906f0d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc900071b60 con 0x7fc900145700 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8fffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc9000cdd10 0x7fc9000cf4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37030/0 (socket says 172.21.15.122:37030) 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8fffff640 1 -- 172.21.15.122:0/1428912653 learned_addr learned my addr 172.21.15.122:0/1428912653 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327423254 0 0) 0x7fc900070f50 con 0x7fc9000cdd10 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc9054aa640 1 --2- 172.21.15.122:0/1428912653 >> v2:172.21.15.169:3300/0 conn(0x7fc90006e2c0 0x7fc90006e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec003520 con 0x7fc9000cdd10 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc904ca9640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc90015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143536087 0 0) 0x7fc8ec003520 con 0x7fc9000cdd10 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 msgr2=0x7fc90015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc90015c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 >> v2:172.21.15.169:3300/0 conn(0x7fc90006e2c0 msgr2=0x7fc90006e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 --2- 172.21.15.122:0/1428912653 >> v2:172.21.15.169:3300/0 conn(0x7fc90006e2c0 0x7fc90006e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900071230 con 0x7fc9000cdd10 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fc8e8001670 con 0x7fc9000cdd10 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc9054aa640 1 --2- 172.21.15.122:0/1428912653 >> v2:172.21.15.169:3300/0 conn(0x7fc90006e2c0 0x7fc90006e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc904ca9640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc90015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 >> v1:172.21.15.122:6789/0 conn(0x7fc9000cdd10 legacy=0x7fc9000cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8ec0039b0 0x7fc8ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc8ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8ec004d40 0x7fc8ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90006ec70 con 0x7fc8ec004d40 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900071b60 con 0x7fc8ec0039b0 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec00bac0 con 0x7fc900145700 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc9054aa640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8ec004d40 0x7fc8ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc904ca9640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8ec0039b0 0x7fc8ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.805+0000 7fc8fffff640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc8ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc9054aa640 1 -- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 msgr2=0x7fc8ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc9054aa640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc900145700 0x7fc8ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc9054aa640 1 -- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8ec0039b0 msgr2=0x7fc8ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc9054aa640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8ec0039b0 0x7fc8ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc9054aa640 1 -- 172.21.15.122:0/1428912653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec003520 con 0x7fc8ec004d40 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc904ca9640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8ec0039b0 0x7fc8ec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc9054aa640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8ec004d40 0x7fc8ec0051c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00098a0 tx=0x7fc8ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590b112066f7f6d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f000f040 con 0x7fc8ec004d40 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0004230 con 0x7fc8ec004d40 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.806+0000 7fc8ff7fe640 1 -- 172.21.15.122:0/1428912653 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0003610 con 0x7fc8ec004d40 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc906f0d640 1 -- 172.21.15.122:0/1428912653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8ec004d40 msgr2=0x7fc8ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc906f0d640 1 --2- 172.21.15.122:0/1428912653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8ec004d40 0x7fc8ec0051c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00098a0 tx=0x7fc8ec005830 comp rx=0 tx=0).stop 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc9054aa640 1 -- 172.21.15.122:0/1428912653 reap_dead start 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc906f0d640 1 -- 172.21.15.122:0/1428912653 shutdown_connections 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc906f0d640 1 -- 172.21.15.122:0/1428912653 >> 172.21.15.122:0/1428912653 conn(0x7fc90006b830 msgr2=0x7fc9000b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc906f0d640 1 -- 172.21.15.122:0/1428912653 shutdown_connections 2024-03-06T07:24:10.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.807+0000 7fc906f0d640 1 -- 172.21.15.122:0/1428912653 wait complete. 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.808+0000 7fc906f0d640 1 Processor -- start 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.808+0000 7fc906f0d640 1 -- start start 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc906f0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 0x7fc9000aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc906f0d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc90013fd00 0x7fc900133120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc906f0d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc90015d8f0 0x7fc90015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc906f0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000cfbc0 con 0x7fc90015bae0 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc906f0d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000717f0 con 0x7fc90013fd00 2024-03-06T07:24:10.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc906f0d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc900145ae0 con 0x7fc90015d8f0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 0x7fc9000aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 0x7fc9000aa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38710/0 (socket says 172.21.15.122:38710) 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 -- 172.21.15.122:0/2153005883 learned_addr learned my addr 172.21.15.122:0/2153005883 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc9054aa640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc90015d8f0 0x7fc90015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc8fffff640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc90013fd00 0x7fc900133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 -- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc90015d8f0 msgr2=0x7fc90015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc90015d8f0 0x7fc90015dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 -- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc90013fd00 msgr2=0x7fc900133120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc90013fd00 0x7fc900133120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc904ca9640 1 -- 172.21.15.122:0/2153005883 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000ab0f0 con 0x7fc90015bae0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.809+0000 7fc8fffff640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc90013fd00 0x7fc900133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.810+0000 7fc904ca9640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 0x7fc9000aa9a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8004710 tx=0x7fc8f8007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.810+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8007e10 con 0x7fc90015bae0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.810+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8019070 con 0x7fc90015bae0 2024-03-06T07:24:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.810+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000ab3d0 con 0x7fc90015bae0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.810+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80137c0 con 0x7fc90015bae0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.810+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9000ab990 con 0x7fc90015bae0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.811+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90006e0b0 con 0x7fc90015bae0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.811+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8f8013960 con 0x7fc90015bae0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.811+0000 7fc8fd7fa640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc8e006b4d0 0x7fc8e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.812+0000 7fc8fffff640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc8e006b4d0 0x7fc8e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.812+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8f808a790 con 0x7fc90015bae0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.812+0000 7fc8fffff640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc8e006b4d0 0x7fc8e006d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8001050 tx=0x7fc8e80014c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.816+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8f80599a0 con 0x7fc90015bae0 2024-03-06T07:24:10.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.988+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fc90015eb30 con 0x7fc90015bae0 2024-03-06T07:24:10.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.991+0000 7fc8fd7fa640 1 -- 172.21.15.122:0/2153005883 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v14) v1 ==== 108+0+0 (secure 0 0 0) 0x7fc8f80592a0 con 0x7fc90015bae0 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc8e006b4d0 msgr2=0x7fc8e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc8e006b4d0 0x7fc8e006d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8001050 tx=0x7fc8e80014c0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 msgr2=0x7fc9000aa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 0x7fc9000aa9a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8004710 tx=0x7fc8f8007700 comp rx=0 tx=0).stop 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 shutdown_connections 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc8e006b4d0 0x7fc8e006d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc90015d8f0 0x7fc90015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc90013fd00 0x7fc900133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 --2- 172.21.15.122:0/2153005883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc90015bae0 0x7fc9000aa9a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 >> 172.21.15.122:0/2153005883 conn(0x7fc90006b830 msgr2=0x7fc9000726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 shutdown_connections 2024-03-06T07:24:10.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:10.994+0000 7fc906f0d640 1 -- 172.21.15.122:0/2153005883 wait complete. 2024-03-06T07:24:11.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:11.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 Processor -- start 2024-03-06T07:24:11.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 -- start start 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb840d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb840d1910 0x7ffb840d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb840d2740 con 0x7ffb840d22b0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb841618b0 con 0x7ffb840d1910 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.107+0000 7ffb8a684640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb840cf400 con 0x7ffb841614d0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb88c21640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffb840d22b0 0x7ffb840d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37040/0 (socket says 172.21.15.122:37040) 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb88c21640 1 -- 172.21.15.122:0/3332611278 learned_addr learned my addr 172.21.15.122:0/3332611278 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb837fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb840d1910 0x7ffb840d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb83fff640 1 --2- 172.21.15.122:0/3332611278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb840d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb82ffd640 1 -- 172.21.15.122:0/3332611278 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286324174 0 0) 0x7ffb7c001490 con 0x7ffb840d22b0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb82ffd640 1 -- 172.21.15.122:0/3332611278 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c003520 con 0x7ffb840d22b0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb837fe640 1 -- 172.21.15.122:0/3332611278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 msgr2=0x7ffb840d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb837fe640 1 --2- 172.21.15.122:0/3332611278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb840d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb837fe640 1 -- 172.21.15.122:0/3332611278 >> v1:172.21.15.122:6789/0 conn(0x7ffb840d22b0 legacy=0x7ffb840d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.108+0000 7ffb837fe640 1 -- 172.21.15.122:0/3332611278 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb840cf6a0 con 0x7ffb840d1910 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb837fe640 1 --2- 172.21.15.122:0/3332611278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d1910 0x7ffb840d1d70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ffb740097a0 tx=0x7ffb7402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0da0831b13b7bff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb82ffd640 1 -- 172.21.15.122:0/3332611278 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74035020 con 0x7ffb840d1910 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb82ffd640 1 -- 172.21.15.122:0/3332611278 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb74031830 con 0x7ffb840d1910 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb82ffd640 1 -- 172.21.15.122:0/3332611278 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74031c00 con 0x7ffb840d1910 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 -- 172.21.15.122:0/3332611278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d1910 msgr2=0x7ffb840d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 --2- 172.21.15.122:0/3332611278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d1910 0x7ffb840d1d70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ffb740097a0 tx=0x7ffb7402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 -- 172.21.15.122:0/3332611278 shutdown_connections 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 --2- 172.21.15.122:0/3332611278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d1910 0x7ffb840d1d70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 --2- 172.21.15.122:0/3332611278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb840d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 -- 172.21.15.122:0/3332611278 >> 172.21.15.122:0/3332611278 conn(0x7ffb840d3d40 msgr2=0x7ffb840d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 -- 172.21.15.122:0/3332611278 shutdown_connections 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.109+0000 7ffb8a684640 1 -- 172.21.15.122:0/3332611278 wait complete. 2024-03-06T07:24:11.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.110+0000 7ffb8a684640 1 Processor -- start 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 -- start start 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb840d1910 0x7ffb840c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 0x7ffb84150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb8415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb840b0910 con 0x7ffb840d22b0 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb840cfc60 con 0x7ffb840d1910 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb8a684640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb840b1e40 con 0x7ffb841614d0 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.111+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 0x7ffb84150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 0x7ffb84150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38730/0 (socket says 172.21.15.122:38730) 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 -- 172.21.15.122:0/423431134 learned_addr learned my addr 172.21.15.122:0/423431134 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb83fff640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb840d1910 0x7ffb840c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb88c21640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb8415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 -- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 msgr2=0x7ffb8415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb8415ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 -- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb840d1910 msgr2=0x7ffb840c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb840d1910 0x7ffb840c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 -- 172.21.15.122:0/423431134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb841506c0 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb88c21640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb8415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb837fe640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 0x7ffb84150180 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ffb7402bee0 tx=0x7ffb740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74035040 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7403b920 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb840c7a80 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7403bd10 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.112+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb840c8040 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.113+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb840c7810 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.114+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb74038020 con 0x7ffb840d22b0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.114+0000 7ffb80ff9640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb7006b4d0 0x7ffb7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.114+0000 7ffb83fff640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb7006b4d0 0x7ffb7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.114+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffb740ac0a0 con 0x7ffb840d22b0 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.114+0000 7ffb83fff640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb7006b4d0 0x7ffb7006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009a30 tx=0x7ffb78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.119+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb7407b190 con 0x7ffb840d22b0 2024-03-06T07:24:11.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.290+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ffb840ce050 con 0x7ffb840d22b0 2024-03-06T07:24:11.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.293+0000 7ffb80ff9640 1 -- 172.21.15.122:0/423431134 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffb7407aa90 con 0x7ffb840d22b0 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.295+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb7006b4d0 msgr2=0x7ffb7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.295+0000 7ffb8a684640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb7006b4d0 0x7ffb7006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009a30 tx=0x7ffb78008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.295+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 msgr2=0x7ffb84150180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.295+0000 7ffb8a684640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 0x7ffb84150180 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ffb7402bee0 tx=0x7ffb740046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 shutdown_connections 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb7006b4d0 0x7ffb7006d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb841614d0 0x7ffb8415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb840d22b0 0x7ffb84150180 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 --2- 172.21.15.122:0/423431134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb840d1910 0x7ffb840c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 >> 172.21.15.122:0/423431134 conn(0x7ffb840d3d40 msgr2=0x7ffb8406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 shutdown_connections 2024-03-06T07:24:11.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.296+0000 7ffb8a684640 1 -- 172.21.15.122:0/423431134 wait complete. 2024-03-06T07:24:11.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:24:11.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.407+0000 7f90da935640 1 Processor -- start 2024-03-06T07:24:11.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.407+0000 7f90da935640 1 -- start start 2024-03-06T07:24:11.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90da935640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f90d40cf100 0x7f90d40cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90da935640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90d40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90da935640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40ccc70 con 0x7f90d40cc810 2024-03-06T07:24:11.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90da935640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d415e440 con 0x7f90d40cf100 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90da935640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d415e700 con 0x7f90d406e270 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d37fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f90d40cc810 0x7f90d406db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37044/0 (socket says 172.21.15.122:37044) 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d37fe640 1 -- 172.21.15.122:0/2017581442 learned_addr learned my addr 172.21.15.122:0/2017581442 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d8ed2640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90d40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198499436 0 0) 0x7f90d40ccc70 con 0x7f90d40cc810 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90bc003520 con 0x7f90d40cc810 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d3fff640 1 --2- 172.21.15.122:0/2017581442 >> v2:172.21.15.169:3300/0 conn(0x7f90d40cf100 0x7f90d40cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769892857 0 0) 0x7f90bc003520 con 0x7f90d40cc810 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 msgr2=0x7f90d40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90d40ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 >> v2:172.21.15.169:3300/0 conn(0x7f90d40cf100 msgr2=0x7f90d40cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 --2- 172.21.15.122:0/2017581442 >> v2:172.21.15.169:3300/0 conn(0x7f90d40cf100 0x7f90d40cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d415e9e0 con 0x7f90d40cc810 2024-03-06T07:24:11.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f90c8001670 con 0x7f90d40cc810 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d8ed2640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90d40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 >> v1:172.21.15.122:6789/0 conn(0x7f90d40cc810 legacy=0x7f90d406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90bc0039b0 0x7f90bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc004800 0x7f90bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d415e440 con 0x7f90bc0039b0 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d415e700 con 0x7f90bc004800 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc00ba40 con 0x7f90d406e270 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.408+0000 7f90d3fff640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90bc0039b0 0x7f90bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d37fe640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc004800 0x7f90bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d8ed2640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d3fff640 1 -- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 msgr2=0x7f90bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d3fff640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d406e270 0x7f90bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d3fff640 1 -- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc004800 msgr2=0x7f90bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d3fff640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc004800 0x7f90bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d3fff640 1 -- 172.21.15.122:0/2017581442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc003520 con 0x7f90bc0039b0 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.409+0000 7f90d3fff640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90bc0039b0 0x7f90bc003dd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009ac0 tx=0x7f90bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e836c1f328871 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4039070 con 0x7f90bc0039b0 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c402c710 con 0x7f90bc0039b0 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90d2ffd640 1 -- 172.21.15.122:0/2017581442 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4009690 con 0x7f90bc0039b0 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90da935640 1 -- 172.21.15.122:0/2017581442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90bc0039b0 msgr2=0x7f90bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90da935640 1 --2- 172.21.15.122:0/2017581442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90bc0039b0 0x7f90bc003dd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009ac0 tx=0x7f90bc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90d8ed2640 1 -- 172.21.15.122:0/2017581442 reap_dead start 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90da935640 1 -- 172.21.15.122:0/2017581442 shutdown_connections 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90da935640 1 -- 172.21.15.122:0/2017581442 >> 172.21.15.122:0/2017581442 conn(0x7f90d40cf8a0 msgr2=0x7f90d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90da935640 1 -- 172.21.15.122:0/2017581442 shutdown_connections 2024-03-06T07:24:11.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.410+0000 7f90da935640 1 -- 172.21.15.122:0/2017581442 wait complete. 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.411+0000 7f90da935640 1 Processor -- start 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.411+0000 7f90da935640 1 -- start start 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90da935640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc0039b0 0x7f90d4145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90da935640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 0x7f90d4158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90da935640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d415c410 0x7f90d415c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90da935640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d40cf4e0 con 0x7f90d40b1180 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90da935640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d415efa0 con 0x7f90bc0039b0 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90da935640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d40ad860 con 0x7f90d415c410 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90d37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 0x7f90d4158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90d37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 0x7f90d4158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38746/0 (socket says 172.21.15.122:38746) 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90d37fe640 1 -- 172.21.15.122:0/532345067 learned_addr learned my addr 172.21.15.122:0/532345067 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90d8ed2640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d415c410 0x7f90d415c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.412+0000 7f90d3fff640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc0039b0 0x7f90d4145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d37fe640 1 -- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d415c410 msgr2=0x7f90d415c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d37fe640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d415c410 0x7f90d415c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d37fe640 1 -- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc0039b0 msgr2=0x7f90d4145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d37fe640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc0039b0 0x7f90d4145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d37fe640 1 -- 172.21.15.122:0/532345067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d41f4260 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d37fe640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 0x7f90d4158fd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f90c8007340 tx=0x7f90c8007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800f910 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c8018070 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800fcf0 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d41f4540 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.413+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d41f4b00 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.414+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d404f330 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.414+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90c801d460 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.415+0000 7f90d0ff9640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90c006b500 0x7f90c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.415+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f90c8089f80 con 0x7f90d40b1180 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.415+0000 7f90d3fff640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90c006b500 0x7f90c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.415+0000 7f90d3fff640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90c006b500 0x7f90c006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f90c4036000 tx=0x7f90c4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.419+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90c8058ab0 con 0x7f90d40b1180 2024-03-06T07:24:11.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.656+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f90d415de80 con 0x7f90d40b1180 2024-03-06T07:24:11.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.677+0000 7f90d0ff9640 1 -- 172.21.15.122:0/532345067 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v145) v1 ==== 133+0+0 (secure 0 0 0) 0x7f90c805cb50 con 0x7f90d40b1180 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90c006b500 msgr2=0x7f90c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90c006b500 0x7f90c006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f90c4036000 tx=0x7f90c4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 msgr2=0x7f90d4158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 0x7f90d4158fd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f90c8007340 tx=0x7f90c8007800 comp rx=0 tx=0).stop 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 shutdown_connections 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90c006b500 0x7f90c006d9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90d415c410 0x7f90d415c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90d40b1180 0x7f90d4158fd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 --2- 172.21.15.122:0/532345067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90bc0039b0 0x7f90d4145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.679+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 >> 172.21.15.122:0/532345067 conn(0x7f90d40cf8a0 msgr2=0x7f90d415f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.680+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 shutdown_connections 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.680+0000 7f90da935640 1 -- 172.21.15.122:0/532345067 wait complete. 2024-03-06T07:24:11.692 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:24:11.692 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:24:11.693 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:11.693 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:11.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.791+0000 7fb6e3bb5640 1 Processor -- start 2024-03-06T07:24:11.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.791+0000 7fb6e3bb5640 1 -- start start 2024-03-06T07:24:11.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e3bb5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb6dc13dc10 0x7fb6dc0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e3bb5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6dc15bf30 0x7fb6dc0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e3bb5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc15f580 con 0x7fb6dc15e850 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e3bb5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc0cf3f0 con 0x7fb6dc15bf30 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e3bb5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc15b210 con 0x7fb6dc13dc10 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6dc15bf30 0x7fb6dc0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6dc15bf30 0x7fb6dc0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38752/0 (socket says 172.21.15.122:38752) 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 -- 172.21.15.122:0/2053046531 learned_addr learned my addr 172.21.15.122:0/2053046531 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1951640 1 --2- 172.21.15.122:0/2053046531 >> v2:172.21.15.169:3300/0 conn(0x7fb6dc13dc10 0x7fb6dc0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 -- 172.21.15.122:0/2053046531 >> v2:172.21.15.169:3300/0 conn(0x7fb6dc13dc10 msgr2=0x7fb6dc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 --2- 172.21.15.122:0/2053046531 >> v2:172.21.15.169:3300/0 conn(0x7fb6dc13dc10 0x7fb6dc0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 -- 172.21.15.122:0/2053046531 >> v1:172.21.15.169:6789/0 conn(0x7fb6dc15e850 legacy=0x7fb6dc15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1150640 1 -- 172.21.15.122:0/2053046531 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc15b450 con 0x7fb6dc15bf30 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.792+0000 7fb6e1951640 1 --2- 172.21.15.122:0/2053046531 >> v2:172.21.15.169:3300/0 conn(0x7fb6dc13dc10 0x7fb6dc0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e1150640 1 --2- 172.21.15.122:0/2053046531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc15bf30 0x7fb6dc0b2e90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0731a0 tx=0x7fb6cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffdfe4c188dbe9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e094f640 1 -- 172.21.15.122:0/2053046531 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc035020 con 0x7fb6dc15bf30 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e094f640 1 -- 172.21.15.122:0/2053046531 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc031870 con 0x7fb6dc15bf30 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e094f640 1 -- 172.21.15.122:0/2053046531 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc031c40 con 0x7fb6dc15bf30 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/2053046531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc15bf30 msgr2=0x7fb6dc0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/2053046531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc15bf30 0x7fb6dc0b2e90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0731a0 tx=0x7fb6cc02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/2053046531 shutdown_connections 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.793+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/2053046531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc15bf30 0x7fb6dc0b2e90 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.794+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/2053046531 >> v2:172.21.15.169:3300/0 conn(0x7fb6dc13dc10 0x7fb6dc0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.794+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/2053046531 >> 172.21.15.122:0/2053046531 conn(0x7fb6dc0cf8a0 msgr2=0x7fb6dc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.794+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/2053046531 shutdown_connections 2024-03-06T07:24:11.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.794+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/2053046531 wait complete. 2024-03-06T07:24:11.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 Processor -- start 2024-03-06T07:24:11.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 -- start start 2024-03-06T07:24:11.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 0x7fb6dc13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6dc15bf30 0x7fb6dc15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6dc15e850 0x7fb6dc13f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0708d0 con 0x7fb6dc13dc10 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc13dff0 con 0x7fb6dc15e850 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e3bb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc070a50 con 0x7fb6dc15bf30 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e1951640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 0x7fb6dc13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e1951640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 0x7fb6dc13e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38764/0 (socket says 172.21.15.122:38764) 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.795+0000 7fb6e1951640 1 -- 172.21.15.122:0/533462893 learned_addr learned my addr 172.21.15.122:0/533462893 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1150640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6dc15bf30 0x7fb6dc15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e2152640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6dc15e850 0x7fb6dc13f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1951640 1 -- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6dc15bf30 msgr2=0x7fb6dc15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1951640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6dc15bf30 0x7fb6dc15df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1951640 1 -- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6dc15e850 msgr2=0x7fb6dc13f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1951640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6dc15e850 0x7fb6dc13f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1951640 1 -- 172.21.15.122:0/533462893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc15e470 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e1951640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 0x7fb6dc13e170 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800e7e0 tx=0x7fb6d800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d800f040 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8004030 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc13e750 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8004400 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.796+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc13ed10 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.797+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc04f330 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.797+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6d800c6d0 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.798+0000 7fb6d27fc640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6c006b4d0 0x7fb6c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.798+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb6d8089cc0 con 0x7fb6dc13dc10 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.798+0000 7fb6e1150640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6c006b4d0 0x7fb6c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.798+0000 7fb6e1150640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6c006b4d0 0x7fb6c006d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc036000 tx=0x7fb6cc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.802+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6d8017020 con 0x7fb6dc13dc10 2024-03-06T07:24:11.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.974+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6dc06e770 con 0x7fb6dc13dc10 2024-03-06T07:24:11.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.975+0000 7fb6d27fc640 1 -- 172.21.15.122:0/533462893 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13022 (secure 0 0 0) 0x7fb6d8058e40 con 0x7fb6dc13dc10 2024-03-06T07:24:11.977 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:11.977 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":48,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:10.672697+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:6835/3905558085":"2024-03-07T07:24:07.405098+0000","172.21.15.169:6834/3905558085":"2024-03-07T07:24:07.405098+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:11.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6c006b4d0 msgr2=0x7fb6c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6c006b4d0 0x7fb6c006d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc036000 tx=0x7fb6cc03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 msgr2=0x7fb6dc13e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 0x7fb6dc13e170 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800e7e0 tx=0x7fb6d800eca0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 shutdown_connections 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6c006b4d0 0x7fb6c006d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6dc15e850 0x7fb6dc13f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6dc15bf30 0x7fb6dc15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 --2- 172.21.15.122:0/533462893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6dc13dc10 0x7fb6dc13e170 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 >> 172.21.15.122:0/533462893 conn(0x7fb6dc0cf8a0 msgr2=0x7fb6dc071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 shutdown_connections 2024-03-06T07:24:11.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:11.977+0000 7fb6e3bb5640 1 -- 172.21.15.122:0/533462893 wait complete. 2024-03-06T07:24:11.990 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:11.990 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:12.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.092+0000 7f8b9da9a640 1 Processor -- start 2024-03-06T07:24:12.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.092+0000 7f8b9da9a640 1 -- start start 2024-03-06T07:24:12.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b9da9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b9806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b9da9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980b3320 con 0x7f8b980719e0 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b9da9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980cf560 con 0x7f8b9815a220 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b9da9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980728d0 con 0x7f8b980cf0d0 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b96ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8b980719e0 0x7f8b9806c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37058/0 (socket says 172.21.15.122:37058) 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b96ffd640 1 -- 172.21.15.122:0/2453665462 learned_addr learned my addr 172.21.15.122:0/2453665462 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b97fff640 1 --2- 172.21.15.122:0/2453665462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b9806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997249769 0 0) 0x7f8b980b3320 con 0x7f8b980719e0 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b84003520 con 0x7f8b980719e0 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955044790 0 0) 0x7f8b980cf560 con 0x7f8b9815a220 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b980b3320 con 0x7f8b9815a220 2024-03-06T07:24:12.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b97fff640 1 -- 172.21.15.122:0/2453665462 >> v1:172.21.15.169:6789/0 conn(0x7f8b9815a220 legacy=0x7f8b980712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b97fff640 1 -- 172.21.15.122:0/2453665462 >> v1:172.21.15.122:6789/0 conn(0x7f8b980719e0 legacy=0x7f8b9806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.093+0000 7f8b97fff640 1 -- 172.21.15.122:0/2453665462 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980b3590 con 0x7f8b980cf0d0 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.094+0000 7f8b97fff640 1 --2- 172.21.15.122:0/2453665462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b9806d100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b880097d0 tx=0x7f8b8802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1357e6d992c5db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.094+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88035020 con 0x7f8b980cf0d0 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.094+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b88031800 con 0x7f8b980cf0d0 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.094+0000 7f8b967fc640 1 -- 172.21.15.122:0/2453665462 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031bb0 con 0x7f8b980cf0d0 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 -- 172.21.15.122:0/2453665462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 msgr2=0x7f8b9806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/2453665462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b9806d100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b880097d0 tx=0x7f8b8802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 -- 172.21.15.122:0/2453665462 shutdown_connections 2024-03-06T07:24:12.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/2453665462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b9806d100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 -- 172.21.15.122:0/2453665462 >> 172.21.15.122:0/2453665462 conn(0x7f8b9806b190 msgr2=0x7f8b9806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 -- 172.21.15.122:0/2453665462 shutdown_connections 2024-03-06T07:24:12.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.095+0000 7f8b9da9a640 1 -- 172.21.15.122:0/2453665462 wait complete. 2024-03-06T07:24:12.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.096+0000 7f8b9da9a640 1 Processor -- start 2024-03-06T07:24:12.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.096+0000 7f8b9da9a640 1 -- start start 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b9da9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 0x7f8b980cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b9da9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b980cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b9da9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b9815a220 0x7f8b9814fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b9da9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98072530 con 0x7f8b980719e0 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b9da9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98072030 con 0x7f8b9815a220 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b9da9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980726b0 con 0x7f8b980cf0d0 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 0x7f8b980cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 0x7f8b980cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38772/0 (socket says 172.21.15.122:38772) 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 -- 172.21.15.122:0/3139888273 learned_addr learned my addr 172.21.15.122:0/3139888273 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b96ffd640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b980cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b97fff640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b9815a220 0x7f8b9814fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 -- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 msgr2=0x7f8b980cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b980cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 -- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b9815a220 msgr2=0x7f8b9814fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b9815a220 0x7f8b9814fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b977fe640 1 -- 172.21.15.122:0/3139888273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b981537d0 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.097+0000 7f8b96ffd640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b980cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.098+0000 7f8b977fe640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 0x7f8b980cba60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000c7a0 tx=0x7f8b9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.098+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90003b70 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.098+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b90003d30 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.098+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b98153ab0 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.098+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000aa30 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.098+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98152b00 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.099+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b980732b0 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.099+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b9000abd0 con 0x7f8b980719e0 2024-03-06T07:24:12.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.099+0000 7f8b8ffff640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b7806b4d0 0x7f8b7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.100+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8b90089e40 con 0x7f8b980719e0 2024-03-06T07:24:12.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.100+0000 7f8b96ffd640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b7806b4d0 0x7f8b7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.100+0000 7f8b96ffd640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b7806b4d0 0x7f8b7806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8b98152410 tx=0x7f8b80009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.104+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b90059050 con 0x7f8b980719e0 2024-03-06T07:24:12.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.276+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b980ad3b0 con 0x7f8b980719e0 2024-03-06T07:24:12.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.276+0000 7f8b8ffff640 1 -- 172.21.15.122:0/3139888273 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b980ad3b0 con 0x7f8b980719e0 2024-03-06T07:24:12.279 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:12.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b7806b4d0 msgr2=0x7f8b7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b7806b4d0 0x7f8b7806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8b98152410 tx=0x7f8b80009040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 msgr2=0x7f8b980cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 0x7f8b980cba60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000c7a0 tx=0x7f8b9000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 shutdown_connections 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b7806b4d0 0x7f8b7806d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b9815a220 0x7f8b9814fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8b980cf0d0 0x7f8b980cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 --2- 172.21.15.122:0/3139888273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b980719e0 0x7f8b980cba60 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.278+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 >> 172.21.15.122:0/3139888273 conn(0x7f8b9806b190 msgr2=0x7f8b9815c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.279+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 shutdown_connections 2024-03-06T07:24:12.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.279+0000 7f8b9da9a640 1 -- 172.21.15.122:0/3139888273 wait complete. 2024-03-06T07:24:12.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:12.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.390+0000 7f99469c7640 1 Processor -- start 2024-03-06T07:24:12.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.390+0000 7f99469c7640 1 -- start start 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f99469c7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f994013dc10 0x7f99400b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f99469c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015bf30 0x7f99400b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f99469c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f994015e850 0x7f994015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f99469c7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99400cf3f0 con 0x7f994013dc10 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f99469c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994015b210 con 0x7f994015e850 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f99469c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994015f580 con 0x7f994015bf30 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f994013dc10 0x7f99400b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f994013dc10 0x7f99400b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38776/0 (socket says 172.21.15.122:38776) 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 -- 172.21.15.122:0/3702856286 learned_addr learned my addr 172.21.15.122:0/3702856286 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 -- 172.21.15.122:0/3702856286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015bf30 msgr2=0x7f99400b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993f7fe640 1 --2- 172.21.15.122:0/3702856286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015bf30 0x7f99400b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f9944f64640 1 --2- 172.21.15.122:0/3702856286 >> v2:172.21.15.169:3300/0 conn(0x7f994015e850 0x7f994015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 --2- 172.21.15.122:0/3702856286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015bf30 0x7f99400b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 -- 172.21.15.122:0/3702856286 >> v2:172.21.15.169:3300/0 conn(0x7f994015e850 msgr2=0x7f994015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 --2- 172.21.15.122:0/3702856286 >> v2:172.21.15.169:3300/0 conn(0x7f994015e850 0x7f994015eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f993ffff640 1 -- 172.21.15.122:0/3702856286 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994015b450 con 0x7f994013dc10 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.391+0000 7f9944f64640 1 --2- 172.21.15.122:0/3702856286 >> v2:172.21.15.169:3300/0 conn(0x7f994015e850 0x7f994015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.392+0000 7f993ffff640 1 --2- 172.21.15.122:0/3702856286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f99400b2950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f994006c790 tx=0x7f993002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba33e75e626bdb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.392+0000 7f993effd640 1 -- 172.21.15.122:0/3702856286 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99300317c0 con 0x7f994013dc10 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.392+0000 7f993effd640 1 -- 172.21.15.122:0/3702856286 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930037070 con 0x7f994013dc10 2024-03-06T07:24:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.392+0000 7f993effd640 1 -- 172.21.15.122:0/3702856286 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930031b80 con 0x7f994013dc10 2024-03-06T07:24:12.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 -- 172.21.15.122:0/3702856286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 msgr2=0x7f99400b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 --2- 172.21.15.122:0/3702856286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f99400b2950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f994006c790 tx=0x7f993002b860 comp rx=0 tx=0).stop 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 -- 172.21.15.122:0/3702856286 shutdown_connections 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 --2- 172.21.15.122:0/3702856286 >> v2:172.21.15.169:3300/0 conn(0x7f994015e850 0x7f994015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 --2- 172.21.15.122:0/3702856286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015bf30 0x7f99400b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 --2- 172.21.15.122:0/3702856286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f99400b2950 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 -- 172.21.15.122:0/3702856286 >> 172.21.15.122:0/3702856286 conn(0x7f99400cf8a0 msgr2=0x7f99400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 -- 172.21.15.122:0/3702856286 shutdown_connections 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.393+0000 7f99469c7640 1 -- 172.21.15.122:0/3702856286 wait complete. 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.394+0000 7f99469c7640 1 Processor -- start 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.394+0000 7f99469c7640 1 -- start start 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f9940071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994015bf30 0x7f994013c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015e850 0x7f994013fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994015f1e0 con 0x7f994013dc10 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994013dff0 con 0x7f994015bf30 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994015f360 con 0x7f994015e850 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f9940071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f9940071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38790/0 (socket says 172.21.15.122:38790) 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 -- 172.21.15.122:0/3775529959 learned_addr learned my addr 172.21.15.122:0/3775529959 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 -- 172.21.15.122:0/3775529959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015e850 msgr2=0x7f994013fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015e850 0x7f994013fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 -- 172.21.15.122:0/3775529959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994015bf30 msgr2=0x7f994013c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994015bf30 0x7f994013c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 -- 172.21.15.122:0/3775529959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994006e710 con 0x7f994013dc10 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993ffff640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f9940071d60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f993002b8a0 tx=0x7f99300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930032860 con 0x7f994013dc10 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930032a00 con 0x7f994013dc10 2024-03-06T07:24:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994006e430 con 0x7f994013dc10 2024-03-06T07:24:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930032df0 con 0x7f994013dc10 2024-03-06T07:24:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.395+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9940071560 con 0x7f994013dc10 2024-03-06T07:24:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.396+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994004f330 con 0x7f994013dc10 2024-03-06T07:24:12.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.397+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993004c060 con 0x7f994013dc10 2024-03-06T07:24:12.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.397+0000 7f993cff9640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f992c06b4d0 0x7f992c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.397+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f99300abbb0 con 0x7f994013dc10 2024-03-06T07:24:12.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.397+0000 7f993f7fe640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f992c06b4d0 0x7f992c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.398+0000 7f993f7fe640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f992c06b4d0 0x7f992c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9934006b10 tx=0x7f9934008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.401+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9930040070 con 0x7f994013dc10 2024-03-06T07:24:12.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.636+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f994006eb00 con 0x7f994013dc10 2024-03-06T07:24:12.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.679+0000 7f993cff9640 1 -- 172.21.15.122:0/3775529959 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v146) v1 ==== 147+0+0 (secure 0 0 0) 0x7f993007adc0 con 0x7f994013dc10 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f992c06b4d0 msgr2=0x7f992c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f992c06b4d0 0x7f992c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9934006b10 tx=0x7f9934008040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 msgr2=0x7f9940071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f9940071d60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f993002b8a0 tx=0x7f99300046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 shutdown_connections 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f992c06b4d0 0x7f992c06d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f994015e850 0x7f994013fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f994015bf30 0x7f994013c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 --2- 172.21.15.122:0/3775529959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f994013dc10 0x7f9940071d60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 >> 172.21.15.122:0/3775529959 conn(0x7f99400cf8a0 msgr2=0x7f99400b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 shutdown_connections 2024-03-06T07:24:12.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.681+0000 7f99469c7640 1 -- 172.21.15.122:0/3775529959 wait complete. 2024-03-06T07:24:12.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:12.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.795+0000 7f81ab3f6640 1 Processor -- start 2024-03-06T07:24:12.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.796+0000 7f81ab3f6640 1 -- start start 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.796+0000 7f81ab3f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a415a220 0x7f81a40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.796+0000 7f81ab3f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81a406da00 0x7f81a4071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.796+0000 7f81ab3f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40b3320 con 0x7f81a40712d0 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.796+0000 7f81ab3f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a40cf610 con 0x7f81a406da00 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.796+0000 7f81ab3f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4071c70 con 0x7f81a415a220 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a9993640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81a406da00 0x7f81a4071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a8991640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f81a40712d0 0x7f81a415de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37062/0 (socket says 172.21.15.122:37062) 2024-03-06T07:24:12.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a8991640 1 -- 172.21.15.122:0/164146434 learned_addr learned my addr 172.21.15.122:0/164146434 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:12.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a9192640 1 --2- 172.21.15.122:0/164146434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a415a220 0x7f81a40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a9993640 1 -- 172.21.15.122:0/164146434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a415a220 msgr2=0x7f81a40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a9993640 1 --2- 172.21.15.122:0/164146434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a415a220 0x7f81a40cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a9993640 1 -- 172.21.15.122:0/164146434 >> v1:172.21.15.122:6789/0 conn(0x7f81a40712d0 legacy=0x7f81a415de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f81a9993640 1 -- 172.21.15.122:0/164146434 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a40728d0 con 0x7f81a406da00 2024-03-06T07:24:12.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.797+0000 7f819bfff640 1 -- 172.21.15.122:0/164146434 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846582145 0 0) 0x7f81a40b3320 con 0x7f81a40712d0 2024-03-06T07:24:12.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81a9993640 1 --2- 172.21.15.122:0/164146434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a406da00 0x7f81a4071730 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f819c009d90 tx=0x7f819c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d76e79f7ce74b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f819bfff640 1 -- 172.21.15.122:0/164146434 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c039070 con 0x7f81a406da00 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f819bfff640 1 -- 172.21.15.122:0/164146434 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c02cbc0 con 0x7f81a406da00 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f819bfff640 1 -- 172.21.15.122:0/164146434 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c034440 con 0x7f81a406da00 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81ab3f6640 1 -- 172.21.15.122:0/164146434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a406da00 msgr2=0x7f81a4071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/164146434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a406da00 0x7f81a4071730 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f819c009d90 tx=0x7f819c02be10 comp rx=0 tx=0).stop 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81ab3f6640 1 -- 172.21.15.122:0/164146434 shutdown_connections 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/164146434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a406da00 0x7f81a4071730 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/164146434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a415a220 0x7f81a40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.798+0000 7f81ab3f6640 1 -- 172.21.15.122:0/164146434 >> 172.21.15.122:0/164146434 conn(0x7f81a406b190 msgr2=0x7f81a406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.799+0000 7f81ab3f6640 1 -- 172.21.15.122:0/164146434 shutdown_connections 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.799+0000 7f81ab3f6640 1 -- 172.21.15.122:0/164146434 wait complete. 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 Processor -- start 2024-03-06T07:24:12.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 -- start start 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a406da00 0x7f81a41f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 0x7f81a4157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81a415a220 0x7f81a40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4072030 con 0x7f81a40712d0 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4072bb0 con 0x7f81a415a220 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81ab3f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4072530 con 0x7f81a406da00 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81a8991640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 0x7f81a4157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81a8991640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 0x7f81a4157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38806/0 (socket says 172.21.15.122:38806) 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81a8991640 1 -- 172.21.15.122:0/798239326 learned_addr learned my addr 172.21.15.122:0/798239326 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.800+0000 7f81a9192640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a406da00 0x7f81a41f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a9993640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81a415a220 0x7f81a40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a8991640 1 -- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a406da00 msgr2=0x7f81a41f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a8991640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a406da00 0x7f81a41f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a8991640 1 -- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81a415a220 msgr2=0x7f81a40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a8991640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81a415a220 0x7f81a40b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a8991640 1 -- 172.21.15.122:0/798239326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a40b0510 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81a8991640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 0x7f81a4157e80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f819400c4d0 tx=0x7f819400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194018070 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81940074e0 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a40b07f0 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194004a20 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.801+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a40b0db0 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.802+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a40732b0 con 0x7f81a40712d0 2024-03-06T07:24:12.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.802+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f819401b070 con 0x7f81a40712d0 2024-03-06T07:24:12.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.803+0000 7f8199ffb640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f818806b4d0 0x7f818806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.803+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f81940891b0 con 0x7f81a40712d0 2024-03-06T07:24:12.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.803+0000 7f81a9192640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f818806b4d0 0x7f818806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.803+0000 7f81a9192640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f818806b4d0 0x7f818806d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f81a000aa20 tx=0x7f81a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.807+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8194057ce0 con 0x7f81a40712d0 2024-03-06T07:24:12.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.981+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f81a4070ba0 con 0x7f81a40712d0 2024-03-06T07:24:12.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.984+0000 7f8199ffb640 1 -- 172.21.15.122:0/798239326 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f819405bd80 con 0x7f81a40712d0 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f818806b4d0 msgr2=0x7f818806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f818806b4d0 0x7f818806d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f81a000aa20 tx=0x7f81a0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 msgr2=0x7f81a4157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 0x7f81a4157e80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f819400c4d0 tx=0x7f819400c990 comp rx=0 tx=0).stop 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 shutdown_connections 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f818806b4d0 0x7f818806d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81a415a220 0x7f81a40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81a40712d0 0x7f81a4157e80 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 --2- 172.21.15.122:0/798239326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81a406da00 0x7f81a41f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 >> 172.21.15.122:0/798239326 conn(0x7f81a406b190 msgr2=0x7f81a40ccf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 shutdown_connections 2024-03-06T07:24:12.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:12.987+0000 7f81ab3f6640 1 -- 172.21.15.122:0/798239326 wait complete. 2024-03-06T07:24:12.999 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:13.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:13.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:13.098 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:13.098 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:13.098 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:13.114 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:13.114 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:24:13.114 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:13.131 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:13.131 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:13.131 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:13.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:13.169 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:24:13.169 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:13.188 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:13.188 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:13.189 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:13.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.307+0000 7f37b1555640 1 Processor -- start 2024-03-06T07:24:13.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.307+0000 7f37b1555640 1 -- start start 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37b1555640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 0x7f37ac0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37b1555640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f37ac162400 0x7f37ac0cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37b1555640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b1070 con 0x7f37ac162a00 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37b1555640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac0ce9d0 con 0x7f37ac162400 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37b1555640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0d2da0 con 0x7f37ac162020 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37aa7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f37ac162a00 0x7f37ac0b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37072/0 (socket says 172.21.15.122:37072) 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37aa7fc640 1 -- 172.21.15.122:0/2630030011 learned_addr learned my addr 172.21.15.122:0/2630030011 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37ab7fe640 1 --2- 172.21.15.122:0/2630030011 >> v2:172.21.15.169:3300/0 conn(0x7f37ac162400 0x7f37ac0cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225545604 0 0) 0x7f37ac0b1070 con 0x7f37ac162a00 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37aaffd640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 0x7f37ac0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003520 con 0x7f37ac162a00 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202625888 0 0) 0x7f3794003520 con 0x7f37ac162a00 2024-03-06T07:24:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 msgr2=0x7f37ac0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 0x7f37ac0d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 >> v2:172.21.15.169:3300/0 conn(0x7f37ac162400 msgr2=0x7f37ac0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 --2- 172.21.15.122:0/2630030011 >> v2:172.21.15.169:3300/0 conn(0x7f37ac162400 0x7f37ac0cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac0d2500 con 0x7f37ac162a00 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f37a0001670 con 0x7f37ac162a00 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 >> v1:172.21.15.122:6789/0 conn(0x7f37ac162a00 legacy=0x7f37ac0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37940039b0 0x7f3794003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3794004800 0x7f3794004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 0x7f37940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0d2da0 con 0x7f3794004800 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379400b9c0 con 0x7f37940039b0 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.308+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379400bb40 con 0x7f37ac162020 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aa7fc640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3794004800 0x7f3794004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aaffd640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37940039b0 0x7f3794003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37ab7fe640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 0x7f37940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aaffd640 1 -- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 msgr2=0x7f37940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aaffd640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac162020 0x7f37940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aaffd640 1 -- 172.21.15.122:0/2630030011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3794004800 msgr2=0x7f3794004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aaffd640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3794004800 0x7f3794004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aaffd640 1 -- 172.21.15.122:0/2630030011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794003520 con 0x7f37940039b0 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.309+0000 7f37aa7fc640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3794004800 0x7f3794004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:13.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37aaffd640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37940039b0 0x7f3794003dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f379c009ac0 tx=0x7f3794005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cdce38ea04dbca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c039070 con 0x7f37940039b0 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c02c6b0 con 0x7f37940039b0 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37a9ffb640 1 -- 172.21.15.122:0/2630030011 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c009690 con 0x7f37940039b0 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37b1555640 1 -- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37940039b0 msgr2=0x7f3794003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37b1555640 1 --2- 172.21.15.122:0/2630030011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37940039b0 0x7f3794003dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f379c009ac0 tx=0x7f3794005830 comp rx=0 tx=0).stop 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37ab7fe640 1 -- 172.21.15.122:0/2630030011 reap_dead start 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37b1555640 1 -- 172.21.15.122:0/2630030011 shutdown_connections 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37b1555640 1 -- 172.21.15.122:0/2630030011 >> 172.21.15.122:0/2630030011 conn(0x7f37ac0d3d10 msgr2=0x7f37ac0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37b1555640 1 -- 172.21.15.122:0/2630030011 shutdown_connections 2024-03-06T07:24:13.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.310+0000 7f37b1555640 1 -- 172.21.15.122:0/2630030011 wait complete. 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 Processor -- start 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 -- start start 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 0x7f37ac1f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ac06cb50 0x7f37ac147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac06cfb0 0x7f37ac147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0cec00 con 0x7f37940039b0 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0d2ac0 con 0x7f37ac06cb50 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37b1555640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0d4830 con 0x7f37ac06cfb0 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37aaffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 0x7f37ac1f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37aaffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 0x7f37ac1f7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38828/0 (socket says 172.21.15.122:38828) 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.312+0000 7f37aaffd640 1 -- 172.21.15.122:0/3243091049 learned_addr learned my addr 172.21.15.122:0/3243091049 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:13.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aa7fc640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ac06cb50 0x7f37ac147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37ab7fe640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac06cfb0 0x7f37ac147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aaffd640 1 -- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac06cfb0 msgr2=0x7f37ac147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aaffd640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac06cfb0 0x7f37ac147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aaffd640 1 -- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ac06cb50 msgr2=0x7f37ac147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aaffd640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ac06cb50 0x7f37ac147040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aaffd640 1 -- 172.21.15.122:0/3243091049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac1f8080 con 0x7f37940039b0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aa7fc640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ac06cb50 0x7f37ac147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37aaffd640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 0x7f37ac1f7b40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f379c02c320 tx=0x7f379c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c039050 con 0x7f37940039b0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c045070 con 0x7f37940039b0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac0c96a0 con 0x7f37940039b0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c003a50 con 0x7f37940039b0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.313+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac0c9c60 con 0x7f37940039b0 2024-03-06T07:24:13.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.314+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac04f330 con 0x7f37940039b0 2024-03-06T07:24:13.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.315+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f379c003bf0 con 0x7f37940039b0 2024-03-06T07:24:13.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.315+0000 7f37937fe640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f379806b4d0 0x7f379806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.315+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f379c0abb70 con 0x7f37940039b0 2024-03-06T07:24:13.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.315+0000 7f37aa7fc640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f379806b4d0 0x7f379806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.315+0000 7f37aa7fc640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f379806b4d0 0x7f379806d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0d3810 tx=0x7f37a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.319+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f379c04e020 con 0x7f37940039b0 2024-03-06T07:24:13.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.528+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37ac08ddb0 con 0x7f37940039b0 2024-03-06T07:24:13.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.529+0000 7f37937fe640 1 -- 172.21.15.122:0/3243091049 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+3711 (secure 0 0 0) 0x7f37ac1f8080 con 0x7f37940039b0 2024-03-06T07:24:13.531 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:13.531 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1066881068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1066881068},{"type":"v1","addr":"172.21.15.169:6833","nonce":1066881068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5754,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/2277739599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2277739599},{"type":"v1","addr":"172.21.15.122:6836","nonce":2277739599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3149261246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":3149261246},{"type":"v1","addr":"172.21.15.122:6837","nonce":3149261246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5833,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3765713747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3765713747},{"type":"v1","addr":"172.21.15.169:6835","nonce":3765713747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[]} 2024-03-06T07:24:13.531 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 23 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f379806b4d0 msgr2=0x7f379806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f379806b4d0 0x7f379806d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0d3810 tx=0x7f37a0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 msgr2=0x7f37ac1f7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 0x7f37ac1f7b40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f379c02c320 tx=0x7f379c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 shutdown_connections 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f379806b4d0 0x7f379806d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37ac06cfb0 0x7f37ac147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ac06cb50 0x7f37ac147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 --2- 172.21.15.122:0/3243091049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37940039b0 0x7f37ac1f7b40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 >> 172.21.15.122:0/3243091049 conn(0x7f37ac0d3d10 msgr2=0x7f37ac0cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 shutdown_connections 2024-03-06T07:24:13.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.531+0000 7f37b1555640 1 -- 172.21.15.122:0/3243091049 wait complete. 2024-03-06T07:24:13.544 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:13.544 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:13.544 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:13.544 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:13.544 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:13.545 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:13.545 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:13.545 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:13.545 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:13.545 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:13.545 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:13.545 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:13.546 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:13.546 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:13.546 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:13.554 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:13.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:13.555 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:13.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:13.557 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:13.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:13.558 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:13.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:13.597 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:13.597 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:13.598 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:13.598 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:13.600 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:13.600 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:13.601 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:13.601 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:13.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:13.673 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:24:13.673 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:24:13.686 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:24:13.686 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:24:13.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.703+0000 7f17292f4640 1 Processor -- start 2024-03-06T07:24:13.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.703+0000 7f17292f4640 1 -- start start 2024-03-06T07:24:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17292f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f17240b4460 0x7f17240d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17292f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1724162970 0x7f17240b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17292f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17240d22c0 con 0x7f1724161cf0 2024-03-06T07:24:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17292f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1724161b00 con 0x7f1724162970 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17292f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17240cecc0 con 0x7f17240b4460 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17227fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1724162970 0x7f17240b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17237fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1724161cf0 0x7f17240cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37084/0 (socket says 172.21.15.122:37084) 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f17237fe640 1 -- 172.21.15.122:0/2340913278 learned_addr learned my addr 172.21.15.122:0/2340913278 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.704+0000 7f1722ffd640 1 --2- 172.21.15.122:0/2340913278 >> v2:172.21.15.169:3300/0 conn(0x7f17240b4460 0x7f17240d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.705+0000 7f17227fc640 1 -- 172.21.15.122:0/2340913278 >> v2:172.21.15.169:3300/0 conn(0x7f17240b4460 msgr2=0x7f17240d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.705+0000 7f17227fc640 1 --2- 172.21.15.122:0/2340913278 >> v2:172.21.15.169:3300/0 conn(0x7f17240b4460 0x7f17240d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.705+0000 7f17227fc640 1 -- 172.21.15.122:0/2340913278 >> v1:172.21.15.122:6789/0 conn(0x7f1724161cf0 legacy=0x7f17240cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.705+0000 7f17227fc640 1 -- 172.21.15.122:0/2340913278 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17240d25a0 con 0x7f1724162970 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.705+0000 7f1721ffb640 1 -- 172.21.15.122:0/2340913278 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371011255 0 0) 0x7f17240d22c0 con 0x7f1724161cf0 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.705+0000 7f17227fc640 1 --2- 172.21.15.122:0/2340913278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724162970 0x7f17240b3580 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f172408ddd0 tx=0x7f170c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851d7e6092343d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f1721ffb640 1 -- 172.21.15.122:0/2340913278 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c035020 con 0x7f1724162970 2024-03-06T07:24:13.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f1721ffb640 1 -- 172.21.15.122:0/2340913278 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c031830 con 0x7f1724162970 2024-03-06T07:24:13.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f1721ffb640 1 -- 172.21.15.122:0/2340913278 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c031c00 con 0x7f1724162970 2024-03-06T07:24:13.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 -- 172.21.15.122:0/2340913278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724162970 msgr2=0x7f17240b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 --2- 172.21.15.122:0/2340913278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724162970 0x7f17240b3580 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f172408ddd0 tx=0x7f170c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 -- 172.21.15.122:0/2340913278 shutdown_connections 2024-03-06T07:24:13.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 --2- 172.21.15.122:0/2340913278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724162970 0x7f17240b3580 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 --2- 172.21.15.122:0/2340913278 >> v2:172.21.15.169:3300/0 conn(0x7f17240b4460 0x7f17240d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 -- 172.21.15.122:0/2340913278 >> 172.21.15.122:0/2340913278 conn(0x7f17240d3ce0 msgr2=0x7f17240d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 -- 172.21.15.122:0/2340913278 shutdown_connections 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.706+0000 7f17292f4640 1 -- 172.21.15.122:0/2340913278 wait complete. 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.707+0000 7f17292f4640 1 Processor -- start 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.707+0000 7f17292f4640 1 -- start start 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17292f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17240b4460 0x7f17240cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17292f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 0x7f17240c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17292f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1724162970 0x7f172414bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17292f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240ce950 con 0x7f1724161cf0 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17292f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240d2b60 con 0x7f17240b4460 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17292f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17240b3ac0 con 0x7f1724162970 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17227fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 0x7f17240c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17227fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 0x7f17240c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38870/0 (socket says 172.21.15.122:38870) 2024-03-06T07:24:13.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f17227fc640 1 -- 172.21.15.122:0/831087822 learned_addr learned my addr 172.21.15.122:0/831087822 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.708+0000 7f1722ffd640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17240b4460 0x7f17240cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17237fe640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1724162970 0x7f172414bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17227fc640 1 -- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1724162970 msgr2=0x7f172414bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17227fc640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1724162970 0x7f172414bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17227fc640 1 -- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17240b4460 msgr2=0x7f17240cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17227fc640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17240b4460 0x7f17240cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17227fc640 1 -- 172.21.15.122:0/831087822 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17240ca310 con 0x7f1724161cf0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f1722ffd640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17240b4460 0x7f17240cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17237fe640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1724162970 0x7f172414bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17227fc640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 0x7f17240c9c10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f170c009890 tx=0x7f170c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c041070 con 0x7f1724161cf0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c03b920 con 0x7f1724161cf0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17240ca5f0 con 0x7f1724161cf0 2024-03-06T07:24:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c035040 con 0x7f1724161cf0 2024-03-06T07:24:13.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.709+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17240cb380 con 0x7f1724161cf0 2024-03-06T07:24:13.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.710+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17240cb580 con 0x7f1724161cf0 2024-03-06T07:24:13.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.710+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f170c038020 con 0x7f1724161cf0 2024-03-06T07:24:13.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.711+0000 7f171b7fe640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.711+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f170c0ab7e0 con 0x7f1724161cf0 2024-03-06T07:24:13.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.711+0000 7f1722ffd640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.711+0000 7f1722ffd640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1714009a20 tx=0x7f1714008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.715+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17240cb580 con 0x7f1724161cf0 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:13.897+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f172404f330 con 0x7f1724161cf0 2024-03-06T07:24:15.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:14.999+0000 7f171b7fe640 1 -- 172.21.15.122:0/831087822 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v53) v1 ==== 97+0+0 (secure 0 0 0) 0x7f170c031800 con 0x7f1724161cf0 2024-03-06T07:24:15.001 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.001+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 msgr2=0x7f170406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.001+0000 7f17292f4640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1714009a20 tx=0x7f1714008040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.001+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 msgr2=0x7f17240c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 0x7f17240c9c10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f170c009890 tx=0x7f170c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 shutdown_connections 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f170406b4d0 0x7f170406d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1724162970 0x7f172414bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1724161cf0 0x7f17240c9c10 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 --2- 172.21.15.122:0/831087822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17240b4460 0x7f17240cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 >> 172.21.15.122:0/831087822 conn(0x7f17240d3ce0 msgr2=0x7f172415ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 shutdown_connections 2024-03-06T07:24:15.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.002+0000 7f17292f4640 1 -- 172.21.15.122:0/831087822 wait complete. 2024-03-06T07:24:15.017 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:15.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.121+0000 7f91b5cc3640 1 Processor -- start 2024-03-06T07:24:15.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.121+0000 7f91b5cc3640 1 -- start start 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91b5cc3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f91b013dc10 0x7f91b00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91b5cc3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b015f580 con 0x7f91b015ef60 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91b5cc3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b015b450 con 0x7f91b015bf30 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91b5cc3640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b00cf3f0 con 0x7f91b013dc10 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91af7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f91b013dc10 0x7f91b00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91af7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f91b013dc10 0x7f91b00b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46792/0 (socket says 172.21.15.122:46792) 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91affff640 1 --1- 172.21.15.122:0/3983123402 >> v1:172.21.15.122:6789/0 conn(0x7f91b015ef60 0x7f91b00708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33352/0 (socket says 172.21.15.122:33352) 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91af7fe640 1 -- 172.21.15.122:0/3983123402 learned_addr learned my addr 172.21.15.122:0/3983123402 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91af7fe640 1 -- 172.21.15.122:0/3983123402 >> v1:172.21.15.169:6789/0 conn(0x7f91b015bf30 legacy=0x7f91b015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91af7fe640 1 -- 172.21.15.122:0/3983123402 >> v1:172.21.15.122:6789/0 conn(0x7f91b015ef60 legacy=0x7f91b00708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91af7fe640 1 -- 172.21.15.122:0/3983123402 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b015b730 con 0x7f91b013dc10 2024-03-06T07:24:15.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.122+0000 7f91ae7fc640 1 -- 172.21.15.122:0/3983123402 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328590908 0 0) 0x7f91b015f580 con 0x7f91b015ef60 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.123+0000 7f91af7fe640 1 --2- 172.21.15.122:0/3983123402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b013dc10 0x7f91b00b2950 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009870 tx=0x7f91a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a9eb14ce6a6950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.123+0000 7f91ae7fc640 1 -- 172.21.15.122:0/3983123402 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0035020 con 0x7f91b013dc10 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.123+0000 7f91ae7fc640 1 -- 172.21.15.122:0/3983123402 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a0031870 con 0x7f91b013dc10 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.123+0000 7f91ae7fc640 1 -- 172.21.15.122:0/3983123402 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0031c40 con 0x7f91b013dc10 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 -- 172.21.15.122:0/3983123402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b013dc10 msgr2=0x7f91b00b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/3983123402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b013dc10 0x7f91b00b2950 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009870 tx=0x7f91a002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 -- 172.21.15.122:0/3983123402 shutdown_connections 2024-03-06T07:24:15.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/3983123402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b013dc10 0x7f91b00b2950 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 -- 172.21.15.122:0/3983123402 >> 172.21.15.122:0/3983123402 conn(0x7f91b00cf8a0 msgr2=0x7f91b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 -- 172.21.15.122:0/3983123402 shutdown_connections 2024-03-06T07:24:15.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.124+0000 7f91b5cc3640 1 -- 172.21.15.122:0/3983123402 wait complete. 2024-03-06T07:24:15.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.125+0000 7f91b5cc3640 1 Processor -- start 2024-03-06T07:24:15.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.125+0000 7f91b5cc3640 1 -- start start 2024-03-06T07:24:15.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91b5cc3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91b013dc10 0x7f91b01543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91b5cc3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91b015bf30 0x7f91b0156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91b5cc3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 0x7f91b01567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91b5cc3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00b2e90 con 0x7f91b015ef60 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91b5cc3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b015b210 con 0x7f91b013dc10 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91b5cc3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00b3010 con 0x7f91b015bf30 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 0x7f91b01567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 0x7f91b01567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46796/0 (socket says 172.21.15.122:46796) 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 -- 172.21.15.122:0/427624805 learned_addr learned my addr 172.21.15.122:0/427624805 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91af7fe640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91b013dc10 0x7f91b01543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91aeffd640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91b015bf30 0x7f91b0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 -- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91b015bf30 msgr2=0x7f91b0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91b015bf30 0x7f91b0156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 -- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91b013dc10 msgr2=0x7f91b01543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91b013dc10 0x7f91b01543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91affff640 1 -- 172.21.15.122:0/427624805 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0159cd0 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91af7fe640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91b013dc10 0x7f91b01543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.126+0000 7f91aeffd640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91b015bf30 0x7f91b0156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.127+0000 7f91affff640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 0x7f91b01567d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f91a800c780 tx=0x7f91a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.127+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8018070 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.127+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a800a640 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.127+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a800aa30 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.127+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b0159fb0 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.127+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b015a570 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.128+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b004f330 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.128+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91a801b070 con 0x7f91b015ef60 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.128+0000 7f9197fff640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f919c06b4d0 0x7f919c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.128+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91a80893c0 con 0x7f91b015ef60 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.129+0000 7f91af7fe640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f919c06b4d0 0x7f919c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.129+0000 7f91af7fe640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f919c06b4d0 0x7f919c06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f91a0036000 tx=0x7f91a003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.133+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91a8058620 con 0x7f91b015ef60 2024-03-06T07:24:15.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.368+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f91b006eb00 con 0x7f91b015ef60 2024-03-06T07:24:15.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.368+0000 7f9197fff640 1 -- 172.21.15.122:0/427624805 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f91a8057f20 con 0x7f91b015ef60 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.374 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:24:15.377 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.378 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:24:15.379 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:15.380 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.381 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:24:15.382 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:24:15.384 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:24:15.385 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:15.385 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:24:15.385 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.385 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:15.385 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:24:15.385 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:24:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f919c06b4d0 msgr2=0x7f919c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f919c06b4d0 0x7f919c06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f91a0036000 tx=0x7f91a003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 msgr2=0x7f91b01567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 0x7f91b01567d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f91a800c780 tx=0x7f91a800cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 shutdown_connections 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f919c06b4d0 0x7f919c06d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b015ef60 0x7f91b01567d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91b015bf30 0x7f91b0156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 --2- 172.21.15.122:0/427624805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91b013dc10 0x7f91b01543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.370+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 >> 172.21.15.122:0/427624805 conn(0x7f91b00cf8a0 msgr2=0x7f91b0071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.371+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 shutdown_connections 2024-03-06T07:24:15.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.371+0000 7f91b5cc3640 1 -- 172.21.15.122:0/427624805 wait complete. 2024-03-06T07:24:15.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.485+0000 7f74bb20f640 1 Processor -- start 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.485+0000 7f74bb20f640 1 -- start start 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74bb20f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f74b40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74bb20f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f74b406da40 0x7f74b415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74bb20f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40cc980 con 0x7f74b40cd7a0 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74bb20f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b40cd220 con 0x7f74b406da40 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74bb20f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40ccc60 con 0x7f74b41424f0 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74b3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f74b40cd7a0 0x7f74b406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33356/0 (socket says 172.21.15.122:33356) 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.486+0000 7f74b3fff640 1 -- 172.21.15.122:0/938418514 learned_addr learned my addr 172.21.15.122:0/938418514 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 --2- 172.21.15.122:0/938418514 >> v2:172.21.15.169:3300/0 conn(0x7f74b406da40 0x7f74b415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b8fab640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f74b40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398947405 0 0) 0x7f74b40cc980 con 0x7f74b40cd7a0 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f749c003520 con 0x7f74b40cd7a0 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636517439 0 0) 0x7f749c003520 con 0x7f74b40cd7a0 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 msgr2=0x7f74b40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f74b40cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 >> v2:172.21.15.169:3300/0 conn(0x7f74b406da40 msgr2=0x7f74b415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 --2- 172.21.15.122:0/938418514 >> v2:172.21.15.169:3300/0 conn(0x7f74b406da40 0x7f74b415cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40ccf40 con 0x7f74b40cd7a0 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f74a8001670 con 0x7f74b40cd7a0 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 >> v1:172.21.15.122:6789/0 conn(0x7f74b40cd7a0 legacy=0x7f74b406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f749c0039b0 0x7f749c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f749c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f749c004d40 0x7f749c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40cd220 con 0x7f749c004d40 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40ccc60 con 0x7f749c0039b0 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c00b9c0 con 0x7f74b41424f0 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f749c004d40 0x7f749c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b8fab640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f749c0039b0 0x7f749c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b3fff640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f749c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 -- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 msgr2=0x7f749c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f749c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 -- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f749c0039b0 msgr2=0x7f749c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f749c0039b0 0x7f749c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b97ac640 1 -- 172.21.15.122:0/938418514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c003520 con 0x7f749c004d40 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.487+0000 7f74b3fff640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74b41424f0 0x7f749c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.488+0000 7f74b97ac640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f749c004d40 0x7f749c0051c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009fc0 tx=0x7f749c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17e759b04bf130f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.488+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac039070 con 0x7f749c004d40 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.488+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac02cc60 con 0x7f749c004d40 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.488+0000 7f74b37fe640 1 -- 172.21.15.122:0/938418514 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac009c10 con 0x7f749c004d40 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74bb20f640 1 -- 172.21.15.122:0/938418514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f749c004d40 msgr2=0x7f749c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74bb20f640 1 --2- 172.21.15.122:0/938418514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f749c004d40 0x7f749c0051c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009fc0 tx=0x7f749c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74b97ac640 1 -- 172.21.15.122:0/938418514 reap_dead start 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74bb20f640 1 -- 172.21.15.122:0/938418514 shutdown_connections 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74bb20f640 1 -- 172.21.15.122:0/938418514 >> 172.21.15.122:0/938418514 conn(0x7f74b40cf8a0 msgr2=0x7f74b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74bb20f640 1 -- 172.21.15.122:0/938418514 shutdown_connections 2024-03-06T07:24:15.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.489+0000 7f74bb20f640 1 -- 172.21.15.122:0/938418514 wait complete. 2024-03-06T07:24:15.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.490+0000 7f74bb20f640 1 Processor -- start 2024-03-06T07:24:15.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.490+0000 7f74bb20f640 1 -- start start 2024-03-06T07:24:15.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74bb20f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f749c004d40 0x7f74b40c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74bb20f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 0x7f74b40c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74bb20f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74b41f3c70 0x7f74b41f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74bb20f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40cc470 con 0x7f74b40c48c0 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74bb20f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40cc2f0 con 0x7f74b41f3c70 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74bb20f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40cc5f0 con 0x7f749c004d40 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 0x7f74b40c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 0x7f74b40c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46814/0 (socket says 172.21.15.122:46814) 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 -- 172.21.15.122:0/99385099 learned_addr learned my addr 172.21.15.122:0/99385099 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 -- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f749c004d40 msgr2=0x7f74b40c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b8fab640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f749c004d40 0x7f74b40c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b97ac640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74b41f3c70 0x7f74b41f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f749c004d40 0x7f74b40c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 -- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74b41f3c70 msgr2=0x7f74b41f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74b41f3c70 0x7f74b41f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 -- 172.21.15.122:0/99385099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40b3a20 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b97ac640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74b41f3c70 0x7f74b41f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.491+0000 7f74b3fff640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 0x7f74b40c4d40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f74a800e7d0 tx=0x7f74a800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.492+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a800c8d0 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.492+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8018070 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.492+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b40c5430 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.492+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a800ccb0 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.492+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b40c59f0 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.493+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b404f330 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.493+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74a801d460 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.493+0000 7f74b17fa640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f74a006b500 0x7f74a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.493+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f74a8089df0 con 0x7f74b40c48c0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.493+0000 7f74b8fab640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f74a006b500 0x7f74a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.494+0000 7f74b8fab640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f74a006b500 0x7f74a006d9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009980 tx=0x7f74a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.498+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74a8058fc0 con 0x7f74b40c48c0 2024-03-06T07:24:15.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.669+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b4160650 con 0x7f74b40c48c0 2024-03-06T07:24:15.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.670+0000 7f74b17fa640 1 -- 172.21.15.122:0/99385099 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12891 (secure 0 0 0) 0x7f74a80588c0 con 0x7f74b40c48c0 2024-03-06T07:24:15.672 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:15.672 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":53,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:14.997031+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:15.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f74a006b500 msgr2=0x7f74a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f74a006b500 0x7f74a006d9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009980 tx=0x7f74a4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 msgr2=0x7f74b40c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 0x7f74b40c4d40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f74a800e7d0 tx=0x7f74a800ec90 comp rx=0 tx=0).stop 2024-03-06T07:24:15.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 shutdown_connections 2024-03-06T07:24:15.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f74a006b500 0x7f74a006d9c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74b41f3c70 0x7f74b41f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74b40c48c0 0x7f74b40c4d40 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 --2- 172.21.15.122:0/99385099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f749c004d40 0x7f74b40c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 >> 172.21.15.122:0/99385099 conn(0x7f74b40cf8a0 msgr2=0x7f74b415c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 shutdown_connections 2024-03-06T07:24:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.672+0000 7f74bb20f640 1 -- 172.21.15.122:0/99385099 wait complete. 2024-03-06T07:24:15.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:15.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:15.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.789+0000 7fe9c16e1640 1 Processor -- start 2024-03-06T07:24:15.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.789+0000 7fe9c16e1640 1 -- start start 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9c16e1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9bc0cea50 0x7fe9bc0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9c16e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9c16e1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc06d030 con 0x7fe9bc15def0 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9c16e1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9bc0cf1c0 con 0x7fe9bc0cea50 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9c16e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc0cc760 con 0x7fe9bc06cab0 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9ba7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9bc0cea50 0x7fe9bc0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.790+0000 7fe9ba7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9bc0cea50 0x7fe9bc0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46816/0 (socket says 172.21.15.122:46816) 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9ba7fc640 1 -- 172.21.15.122:0/1679419192 learned_addr learned my addr 172.21.15.122:0/1679419192 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9bb7fe640 1 --2- 172.21.15.122:0/1679419192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9ba7fc640 1 -- 172.21.15.122:0/1679419192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 msgr2=0x7fe9bc0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9ba7fc640 1 --2- 172.21.15.122:0/1679419192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9ba7fc640 1 -- 172.21.15.122:0/1679419192 >> v1:172.21.15.169:6789/0 conn(0x7fe9bc15def0 legacy=0x7fe9bc0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9ba7fc640 1 -- 172.21.15.122:0/1679419192 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc06d310 con 0x7fe9bc0cea50 2024-03-06T07:24:15.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.791+0000 7fe9ba7fc640 1 --2- 172.21.15.122:0/1679419192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc0b41b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009870 tx=0x7fe9b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d1921f0e5b6519 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9b9ffb640 1 -- 172.21.15.122:0/1679419192 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0034070 con 0x7fe9bc0cea50 2024-03-06T07:24:15.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9b9ffb640 1 -- 172.21.15.122:0/1679419192 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b0030840 con 0x7fe9bc0cea50 2024-03-06T07:24:15.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9b9ffb640 1 -- 172.21.15.122:0/1679419192 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0030c10 con 0x7fe9bc0cea50 2024-03-06T07:24:15.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1679419192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc0cea50 msgr2=0x7fe9bc0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1679419192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc0b41b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009870 tx=0x7fe9b002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1679419192 shutdown_connections 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1679419192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1679419192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc0b41b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1679419192 >> 172.21.15.122:0/1679419192 conn(0x7fe9bc0b3690 msgr2=0x7fe9bc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1679419192 shutdown_connections 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.792+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1679419192 wait complete. 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.793+0000 7fe9c16e1640 1 Processor -- start 2024-03-06T07:24:15.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 -- start start 2024-03-06T07:24:15.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 0x7fe9bc0ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc15e2d0 con 0x7fe9bc15def0 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc0cc3f0 con 0x7fe9bc0cea50 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9c16e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc15e450 con 0x7fe9bc06cab0 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9bb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 0x7fe9bc0ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9bb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 0x7fe9bc0ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46832/0 (socket says 172.21.15.122:46832) 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.794+0000 7fe9bb7fe640 1 -- 172.21.15.122:0/1967260488 learned_addr learned my addr 172.21.15.122:0/1967260488 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9baffd640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9ba7fc640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9bb7fe640 1 -- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 msgr2=0x7fe9bc0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9bb7fe640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9bb7fe640 1 -- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9bc0cea50 msgr2=0x7fe9bc153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9bb7fe640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9bb7fe640 1 -- 172.21.15.122:0/1967260488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0c9020 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9bb7fe640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 0x7fe9bc0ca0c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4007e40 tx=0x7fe9b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4014620 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b4019070 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc0c9300 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4014a20 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.795+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc0c98c0 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.796+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc09fcf0 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.796+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9b4014bc0 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.797+0000 7fe9a37fe640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9a806b4d0 0x7fe9a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.797+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe9b408a760 con 0x7fe9bc15def0 2024-03-06T07:24:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.797+0000 7fe9baffd640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9a806b4d0 0x7fe9a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.797+0000 7fe9baffd640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9a806b4d0 0x7fe9a806d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac006fd0 tx=0x7fe9ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.801+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9b40599c0 con 0x7fe9bc15def0 2024-03-06T07:24:15.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.974+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9bc068ff0 con 0x7fe9bc15def0 2024-03-06T07:24:15.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.974+0000 7fe9a37fe640 1 -- 172.21.15.122:0/1967260488 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9b40592c0 con 0x7fe9bc15def0 2024-03-06T07:24:15.976 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9a806b4d0 msgr2=0x7fe9a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9a806b4d0 0x7fe9a806d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac006fd0 tx=0x7fe9ac008040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 msgr2=0x7fe9bc0ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 0x7fe9bc0ca0c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4007e40 tx=0x7fe9b400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 shutdown_connections 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe9a806b4d0 0x7fe9a806d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9bc15def0 0x7fe9bc0ca0c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9bc0cea50 0x7fe9bc153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 --2- 172.21.15.122:0/1967260488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9bc06cab0 0x7fe9bc0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 >> 172.21.15.122:0/1967260488 conn(0x7fe9bc0b3690 msgr2=0x7fe9bc0d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 shutdown_connections 2024-03-06T07:24:15.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:15.976+0000 7fe9c16e1640 1 -- 172.21.15.122:0/1967260488 wait complete. 2024-03-06T07:24:15.989 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.087+0000 7fa711ced640 1 Processor -- start 2024-03-06T07:24:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.087+0000 7fa711ced640 1 -- start start 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa711ced640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 0x7fa70c15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa711ced640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c15f580 con 0x7fa70c13dc10 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa711ced640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c15b450 con 0x7fa70c0b3060 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa711ced640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c0cf3f0 con 0x7fa70c15bf30 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa70c13dc10 0x7fa70c0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33366/0 (socket says 172.21.15.122:33366) 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70b7fe640 1 -- 172.21.15.122:0/3710679684 learned_addr learned my addr 172.21.15.122:0/3710679684 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845925226 0 0) 0x7fa70c15f580 con 0x7fa70c13dc10 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f4005730 con 0x7fa70c13dc10 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70bfff640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 0x7fa70c15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987861556 0 0) 0x7fa6f4005730 con 0x7fa70c13dc10 2024-03-06T07:24:16.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 msgr2=0x7fa70c15ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 0x7fa70c15ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 >> v1:172.21.15.169:6789/0 conn(0x7fa70c0b3060 legacy=0x7fa70c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c15b730 con 0x7fa70c13dc10 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa6fc001670 con 0x7fa70c13dc10 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 >> v1:172.21.15.122:6789/0 conn(0x7fa70c13dc10 legacy=0x7fa70c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6f4005bf0 0x7fa6f4006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 0x7fa6f4006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4006f50 0x7fa6f40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.088+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c0cf3f0 con 0x7fa6f4005bf0 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f400ddb0 con 0x7fa6f4006f50 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f400dfb0 con 0x7fa70c15bf30 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6f4005bf0 0x7fa6f4006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70affd640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 0x7fa6f4006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70bfff640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4006f50 0x7fa6f40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70b7fe640 1 -- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 msgr2=0x7fa6f4006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c15bf30 0x7fa6f4006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70b7fe640 1 -- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4006f50 msgr2=0x7fa6f40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4006f50 0x7fa6f40073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.089+0000 7fa70b7fe640 1 -- 172.21.15.122:0/3710679684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f4005730 con 0x7fa6f4005bf0 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6f4005bf0 0x7fa6f4006010 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00c240 tx=0x7fa6f4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca4434c75ed514c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc007cf0 con 0x7fa6f4005bf0 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc030070 con 0x7fa6f4005bf0 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa70a7fc640 1 -- 172.21.15.122:0/3710679684 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc005d30 con 0x7fa6f4005bf0 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa711ced640 1 -- 172.21.15.122:0/3710679684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6f4005bf0 msgr2=0x7fa6f4006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa711ced640 1 --2- 172.21.15.122:0/3710679684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6f4005bf0 0x7fa6f4006010 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00c240 tx=0x7fa6f4007a40 comp rx=0 tx=0).stop 2024-03-06T07:24:16.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa70bfff640 1 -- 172.21.15.122:0/3710679684 reap_dead start 2024-03-06T07:24:16.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa711ced640 1 -- 172.21.15.122:0/3710679684 shutdown_connections 2024-03-06T07:24:16.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa711ced640 1 -- 172.21.15.122:0/3710679684 >> 172.21.15.122:0/3710679684 conn(0x7fa70c0cf8a0 msgr2=0x7fa70c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa711ced640 1 -- 172.21.15.122:0/3710679684 shutdown_connections 2024-03-06T07:24:16.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.090+0000 7fa711ced640 1 -- 172.21.15.122:0/3710679684 wait complete. 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.091+0000 7fa711ced640 1 Processor -- start 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 -- start start 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4005bf0 0x7fa70c140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 0x7fa70c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c13f810 0x7fa70c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c15ef60 con 0x7fa70c13d380 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c15b210 con 0x7fa6f4005bf0 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa711ced640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c15f0e0 con 0x7fa70c13f810 2024-03-06T07:24:16.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa70affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 0x7fa70c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa70affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 0x7fa70c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46852/0 (socket says 172.21.15.122:46852) 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa70affd640 1 -- 172.21.15.122:0/2365880431 learned_addr learned my addr 172.21.15.122:0/2365880431 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4005bf0 0x7fa70c140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.092+0000 7fa70bfff640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c13f810 0x7fa70c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70affd640 1 -- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c13f810 msgr2=0x7fa70c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70affd640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c13f810 0x7fa70c06dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70affd640 1 -- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4005bf0 msgr2=0x7fa70c140de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70affd640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4005bf0 0x7fa70c140de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70affd640 1 -- 172.21.15.122:0/2365880431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c13fd40 con 0x7fa70c13d380 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70bfff640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c13f810 0x7fa70c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa70affd640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 0x7fa70c13f2d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa70000f820 tx=0x7fa70000fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000050d0 con 0x7fa70c13d380 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa700018070 con 0x7fa70c13d380 2024-03-06T07:24:16.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c140020 con 0x7fa70c13d380 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700005490 con 0x7fa70c13d380 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.093+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c1405e0 con 0x7fa70c13d380 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.094+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c04f330 con 0x7fa70c13d380 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.094+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa70000d730 con 0x7fa70c13d380 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.095+0000 7fa6f3fff640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f806b4d0 0x7fa6f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.095+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa700089cc0 con 0x7fa70c13d380 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.095+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f806b4d0 0x7fa6f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.095+0000 7fa70b7fe640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f806b4d0 0x7fa6f806d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc034000 tx=0x7fa6fc007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.099+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa700058e90 con 0x7fa70c13d380 2024-03-06T07:24:16.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.271+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa70c06eb00 con 0x7fa70c13d380 2024-03-06T07:24:16.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.271+0000 7fa6f3fff640 1 -- 172.21.15.122:0/2365880431 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa700058790 con 0x7fa70c13d380 2024-03-06T07:24:16.273 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:16.273 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:24:16.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:24:16.274 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:24:16.274 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:24:16.274 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:16.274 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:24:16.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f806b4d0 msgr2=0x7fa6f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f806b4d0 0x7fa6f806d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc034000 tx=0x7fa6fc007000 comp rx=0 tx=0).stop 2024-03-06T07:24:16.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 msgr2=0x7fa70c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 0x7fa70c13f2d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa70000f820 tx=0x7fa70000fce0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 shutdown_connections 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f806b4d0 0x7fa6f806d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa70c13f810 0x7fa70c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa70c13d380 0x7fa70c13f2d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 --2- 172.21.15.122:0/2365880431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6f4005bf0 0x7fa70c140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 >> 172.21.15.122:0/2365880431 conn(0x7fa70c0cf8a0 msgr2=0x7fa70c15c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 shutdown_connections 2024-03-06T07:24:16.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.273+0000 7fa711ced640 1 -- 172.21.15.122:0/2365880431 wait complete. 2024-03-06T07:24:16.286 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:16.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:16.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.389+0000 7fd1759bc640 1 Processor -- start 2024-03-06T07:24:16.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.389+0000 7fd1759bc640 1 -- start start 2024-03-06T07:24:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd1759bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd170161630 0x7fd1700ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd1759bc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd1700cfa50 0x7fd1700cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd1759bc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd170161a10 con 0x7fd1700cf3c0 2024-03-06T07:24:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd1759bc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1700d12e0 con 0x7fd1700cfa50 2024-03-06T07:24:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd1759bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700d29e0 con 0x7fd170161630 2024-03-06T07:24:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd1700cf3c0 0x7fd1700cfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33378/0 (socket says 172.21.15.122:33378) 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16f7fe640 1 -- 172.21.15.122:0/3578933581 learned_addr learned my addr 172.21.15.122:0/3578933581 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16effd640 1 --2- 172.21.15.122:0/3578933581 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd170161630 0x7fd1700ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd167fff640 1 --2- 172.21.15.122:0/3578933581 >> v2:172.21.15.169:3300/0 conn(0x7fd1700cfa50 0x7fd1700cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16e7fc640 1 -- 172.21.15.122:0/3578933581 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203513970 0 0) 0x7fd170161a10 con 0x7fd1700cf3c0 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16e7fc640 1 -- 172.21.15.122:0/3578933581 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c003520 con 0x7fd1700cf3c0 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16effd640 1 -- 172.21.15.122:0/3578933581 >> v2:172.21.15.169:3300/0 conn(0x7fd1700cfa50 msgr2=0x7fd1700cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16effd640 1 --2- 172.21.15.122:0/3578933581 >> v2:172.21.15.169:3300/0 conn(0x7fd1700cfa50 0x7fd1700cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16effd640 1 -- 172.21.15.122:0/3578933581 >> v1:172.21.15.122:6789/0 conn(0x7fd1700cf3c0 legacy=0x7fd1700cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.390+0000 7fd16effd640 1 -- 172.21.15.122:0/3578933581 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1700d2cc0 con 0x7fd170161630 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.391+0000 7fd16effd640 1 --2- 172.21.15.122:0/3578933581 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd170161630 0x7fd1700ce940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd1600097a0 tx=0x7fd16002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2759d410577c2849 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.391+0000 7fd16e7fc640 1 -- 172.21.15.122:0/3578933581 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1600315a0 con 0x7fd170161630 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.391+0000 7fd16e7fc640 1 -- 172.21.15.122:0/3578933581 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd160037070 con 0x7fd170161630 2024-03-06T07:24:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.391+0000 7fd16e7fc640 1 -- 172.21.15.122:0/3578933581 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160031940 con 0x7fd170161630 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 -- 172.21.15.122:0/3578933581 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd170161630 msgr2=0x7fd1700ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 --2- 172.21.15.122:0/3578933581 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd170161630 0x7fd1700ce940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd1600097a0 tx=0x7fd16002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 -- 172.21.15.122:0/3578933581 shutdown_connections 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 --2- 172.21.15.122:0/3578933581 >> v2:172.21.15.169:3300/0 conn(0x7fd1700cfa50 0x7fd1700cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 --2- 172.21.15.122:0/3578933581 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd170161630 0x7fd1700ce940 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 -- 172.21.15.122:0/3578933581 >> 172.21.15.122:0/3578933581 conn(0x7fd1700d3f30 msgr2=0x7fd1700d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 -- 172.21.15.122:0/3578933581 shutdown_connections 2024-03-06T07:24:16.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.392+0000 7fd1759bc640 1 -- 172.21.15.122:0/3578933581 wait complete. 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.393+0000 7fd1759bc640 1 Processor -- start 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.393+0000 7fd1759bc640 1 -- start start 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1700cf3c0 0x7fd1700b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1700cfa50 0x7fd17015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 0x7fd1701f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700b4e00 con 0x7fd170161630 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700b4c80 con 0x7fd1700cfa50 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700b4f80 con 0x7fd1700cf3c0 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 0x7fd1701f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 0x7fd1701f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46864/0 (socket says 172.21.15.122:46864) 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 -- 172.21.15.122:0/2647225118 learned_addr learned my addr 172.21.15.122:0/2647225118 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 -- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1700cf3c0 msgr2=0x7fd1700b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16effd640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1700cf3c0 0x7fd1700b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1700cf3c0 0x7fd1700b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 -- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1700cfa50 msgr2=0x7fd17015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1700cfa50 0x7fd17015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 -- 172.21.15.122:0/2647225118 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1700c7fb0 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16f7fe640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 0x7fd1701f3c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd16800cb40 tx=0x7fd16800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168017070 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16800aca0 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700c82f0 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.394+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168012620 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.395+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1700c8850 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.395+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17004f330 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.396+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd16801a070 con 0x7fd170161630 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.396+0000 7fd16cff9640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd15006b4d0 0x7fd15006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.396+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd168089d30 con 0x7fd170161630 2024-03-06T07:24:16.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.396+0000 7fd16effd640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd15006b4d0 0x7fd15006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.397+0000 7fd16effd640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd15006b4d0 0x7fd15006d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd160036000 tx=0x7fd160008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.401+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1700c82f0 con 0x7fd170161630 2024-03-06T07:24:16.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:16.572+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd1700741f0 con 0x7fd170161630 2024-03-06T07:24:17.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.005+0000 7fd16cff9640 1 -- 172.21.15.122:0/2647225118 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v54) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd168058e80 con 0x7fd170161630 2024-03-06T07:24:17.007 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd15006b4d0 msgr2=0x7fd15006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd15006b4d0 0x7fd15006d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd160036000 tx=0x7fd160008030 comp rx=0 tx=0).stop 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 msgr2=0x7fd1701f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 0x7fd1701f3c90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd16800cb40 tx=0x7fd16800a060 comp rx=0 tx=0).stop 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 shutdown_connections 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd15006b4d0 0x7fd15006d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd170161630 0x7fd1701f3c90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1700cfa50 0x7fd17015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 --2- 172.21.15.122:0/2647225118 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1700cf3c0 0x7fd1700b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.007+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 >> 172.21.15.122:0/2647225118 conn(0x7fd1700d3f30 msgr2=0x7fd1700d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.008+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 shutdown_connections 2024-03-06T07:24:17.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.008+0000 7fd1759bc640 1 -- 172.21.15.122:0/2647225118 wait complete. 2024-03-06T07:24:17.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:17.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 Processor -- start 2024-03-06T07:24:17.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 -- start start 2024-03-06T07:24:17.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fefa806fd10 0x7fefa80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 0x7fefa80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa80cd1e0 con 0x7fefa815c050 2024-03-06T07:24:17.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa80cd4b0 con 0x7fefa806fd10 2024-03-06T07:24:17.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.130+0000 7fefadd3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80cd790 con 0x7fefa80d0af0 2024-03-06T07:24:17.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fefa815c050 0x7fefa806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33394/0 (socket says 172.21.15.122:33394) 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa77fe640 1 -- 172.21.15.122:0/2133922773 learned_addr learned my addr 172.21.15.122:0/2133922773 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000433386 0 0) 0x7fefa80cd1e0 con 0x7fefa815c050 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94003520 con 0x7fefa815c050 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa7fff640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 0x7fefa80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fef9ffff640 1 --2- 172.21.15.122:0/2133922773 >> v2:172.21.15.169:3300/0 conn(0x7fefa806fd10 0x7fefa80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 460665695 0 0) 0x7fef94003520 con 0x7fefa815c050 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 msgr2=0x7fefa80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 0x7fefa80ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 >> v2:172.21.15.169:3300/0 conn(0x7fefa806fd10 msgr2=0x7fefa80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 --2- 172.21.15.122:0/2133922773 >> v2:172.21.15.169:3300/0 conn(0x7fefa806fd10 0x7fefa80cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa80cda70 con 0x7fefa815c050 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fef98001670 con 0x7fefa815c050 2024-03-06T07:24:17.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 >> v1:172.21.15.122:6789/0 conn(0x7fefa815c050 legacy=0x7fefa806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 0x7fef940039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94003f30 0x7fef94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef94004d40 0x7fef940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80cd4b0 con 0x7fef94004d40 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80cd790 con 0x7fef94003f30 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400ba10 con 0x7fefa80d0af0 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa7fff640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef94004d40 0x7fef940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fefa77fe640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 0x7fef940039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.131+0000 7fef9ffff640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94003f30 0x7fef94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa7fff640 1 -- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 msgr2=0x7fef940039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa7fff640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80d0af0 0x7fef940039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa7fff640 1 -- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94003f30 msgr2=0x7fef94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa7fff640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94003f30 0x7fef94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa7fff640 1 -- 172.21.15.122:0/2133922773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94003520 con 0x7fef94004d40 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa7fff640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef94004d40 0x7fef940051c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009fc0 tx=0x7fef94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fef89ed6ed3e0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0039070 con 0x7fef94004d40 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa002cbd0 con 0x7fef94004d40 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.132+0000 7fefa6ffd640 1 -- 172.21.15.122:0/2133922773 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0009c10 con 0x7fef94004d40 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefadd3e640 1 -- 172.21.15.122:0/2133922773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef94004d40 msgr2=0x7fef940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2133922773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef94004d40 0x7fef940051c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009fc0 tx=0x7fef94005830 comp rx=0 tx=0).stop 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefa7fff640 1 -- 172.21.15.122:0/2133922773 reap_dead start 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefadd3e640 1 -- 172.21.15.122:0/2133922773 shutdown_connections 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefadd3e640 1 -- 172.21.15.122:0/2133922773 >> 172.21.15.122:0/2133922773 conn(0x7fefa80b3800 msgr2=0x7fefa80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefadd3e640 1 -- 172.21.15.122:0/2133922773 shutdown_connections 2024-03-06T07:24:17.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.133+0000 7fefadd3e640 1 -- 172.21.15.122:0/2133922773 wait complete. 2024-03-06T07:24:17.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.134+0000 7fefadd3e640 1 Processor -- start 2024-03-06T07:24:17.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.134+0000 7fefadd3e640 1 -- start start 2024-03-06T07:24:17.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefadd3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94004d40 0x7fefa8150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefadd3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 0x7fefa8157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefadd3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80b18d0 0x7fefa80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefadd3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80cce20 con 0x7fefa8155c20 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefadd3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80ccca0 con 0x7fef94004d40 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefadd3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80ccfa0 con 0x7fefa80b18d0 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 0x7fefa8157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 0x7fefa8157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46870/0 (socket says 172.21.15.122:46870) 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 -- 172.21.15.122:0/2779816903 learned_addr learned my addr 172.21.15.122:0/2779816903 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefa7fff640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80b18d0 0x7fefa80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 -- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80b18d0 msgr2=0x7fefa80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80b18d0 0x7fefa80b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 -- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94004d40 msgr2=0x7fefa8150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefa77fe640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94004d40 0x7fefa8150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94004d40 0x7fefa8150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 -- 172.21.15.122:0/2779816903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa81584b0 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fefa77fe640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94004d40 0x7fefa8150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.135+0000 7fef9ffff640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 0x7fefa8157f70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fef90011990 tx=0x7fef90011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.136+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000fa70 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.136+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9000fc10 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.136+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa8158780 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.136+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90014440 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.136+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa80b4ec0 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.137+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa80b2290 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.137+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef90012040 con 0x7fefa8155c20 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.138+0000 7fefa4ff9640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef8806b500 0x7fef8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.138+0000 7fefa77fe640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef8806b500 0x7fef8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.138+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fef9008a560 con 0x7fefa8155c20 2024-03-06T07:24:17.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.138+0000 7fefa77fe640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef8806b500 0x7fef8806d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef98001eb0 tx=0x7fef98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.142+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef90019030 con 0x7fefa8155c20 2024-03-06T07:24:17.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:17.313+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fefa80b4650 con 0x7fefa8155c20 2024-03-06T07:24:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.010+0000 7fefa4ff9640 1 -- 172.21.15.122:0/2779816903 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v55) v1 ==== 170+0+0 (secure 0 0 0) 0x7fefa80b4650 con 0x7fefa8155c20 2024-03-06T07:24:18.013 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef8806b500 msgr2=0x7fef8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef8806b500 0x7fef8806d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef98001eb0 tx=0x7fef98009040 comp rx=0 tx=0).stop 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 msgr2=0x7fefa8157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 0x7fefa8157f70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fef90011990 tx=0x7fef90011e50 comp rx=0 tx=0).stop 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 shutdown_connections 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef8806b500 0x7fef8806d9c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fefa80b18d0 0x7fefa80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefa8155c20 0x7fefa8157f70 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 --2- 172.21.15.122:0/2779816903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef94004d40 0x7fefa8150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 >> 172.21.15.122:0/2779816903 conn(0x7fefa80b3800 msgr2=0x7fefa80d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 shutdown_connections 2024-03-06T07:24:18.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.014+0000 7fefadd3e640 1 -- 172.21.15.122:0/2779816903 wait complete. 2024-03-06T07:24:18.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.145+0000 7f4d1079a640 1 Processor -- start 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.145+0000 7f4d1079a640 1 -- start start 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d1079a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4d0815bc50 0x7f4d080cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d1079a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4d080cc280 0x7f4d080cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d1079a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080aee20 0x7f4d0815f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d1079a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080b4850 con 0x7f4d0815bc50 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d1079a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0806d5b0 con 0x7f4d080cc280 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d1079a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080cc7b0 con 0x7f4d080aee20 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.146+0000 7f4d0e536640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4d0815bc50 0x7f4d080cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4d0815bc50 0x7f4d080cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46884/0 (socket says 172.21.15.122:46884) 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 -- 172.21.15.122:0/1527479257 learned_addr learned my addr 172.21.15.122:0/1527479257 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0ed37640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080aee20 0x7f4d0815f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 -- 172.21.15.122:0/1527479257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080aee20 msgr2=0x7f4d0815f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080aee20 0x7f4d0815f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0dd35640 1 --2- 172.21.15.122:0/1527479257 >> v2:172.21.15.169:3300/0 conn(0x7f4d080cc280 0x7f4d080cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 -- 172.21.15.122:0/1527479257 >> v2:172.21.15.169:3300/0 conn(0x7f4d080cc280 msgr2=0x7f4d080cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 --2- 172.21.15.122:0/1527479257 >> v2:172.21.15.169:3300/0 conn(0x7f4d080cc280 0x7f4d080cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 -- 172.21.15.122:0/1527479257 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d080b4ce0 con 0x7f4d0815bc50 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0ed37640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080aee20 0x7f4d0815f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0dd35640 1 --2- 172.21.15.122:0/1527479257 >> v2:172.21.15.169:3300/0 conn(0x7f4d080cc280 0x7f4d080cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.147+0000 7f4d0e536640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d0815bc50 0x7f4d080cf430 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4d080cd6e0 tx=0x7f4d0402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6878f27217081e5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d0d534640 1 -- 172.21.15.122:0/1527479257 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d040317d0 con 0x7f4d0815bc50 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d0d534640 1 -- 172.21.15.122:0/1527479257 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d04037070 con 0x7f4d0815bc50 2024-03-06T07:24:18.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d0d534640 1 -- 172.21.15.122:0/1527479257 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04031b90 con 0x7f4d0815bc50 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 -- 172.21.15.122:0/1527479257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d0815bc50 msgr2=0x7f4d080cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d0815bc50 0x7f4d080cf430 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4d080cd6e0 tx=0x7f4d0402b870 comp rx=0 tx=0).stop 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 -- 172.21.15.122:0/1527479257 shutdown_connections 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d080aee20 0x7f4d0815f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1527479257 >> v2:172.21.15.169:3300/0 conn(0x7f4d080cc280 0x7f4d080cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1527479257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d0815bc50 0x7f4d080cf430 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 -- 172.21.15.122:0/1527479257 >> 172.21.15.122:0/1527479257 conn(0x7f4d0806afb0 msgr2=0x7f4d0806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 -- 172.21.15.122:0/1527479257 shutdown_connections 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.148+0000 7f4d1079a640 1 -- 172.21.15.122:0/1527479257 wait complete. 2024-03-06T07:24:18.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.149+0000 7f4d1079a640 1 Processor -- start 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 -- start start 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 0x7f4d0814e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d080cc280 0x7f4d081552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d0815bc50 0x7f4d080719f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0815c030 con 0x7f4d080aee20 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080ae900 con 0x7f4d080cc280 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d1079a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0815c1b0 con 0x7f4d0815bc50 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d0e536640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 0x7f4d0814e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d0e536640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 0x7f4d0814e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46890/0 (socket says 172.21.15.122:46890) 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d0e536640 1 -- 172.21.15.122:0/1536490572 learned_addr learned my addr 172.21.15.122:0/1536490572 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:18.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d0dd35640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d080cc280 0x7f4d081552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.150+0000 7f4d0ed37640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d0815bc50 0x7f4d080719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d0e536640 1 -- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d0815bc50 msgr2=0x7f4d080719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d0e536640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d0815bc50 0x7f4d080719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d0e536640 1 -- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d080cc280 msgr2=0x7f4d081552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d0e536640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d080cc280 0x7f4d081552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d0e536640 1 -- 172.21.15.122:0/1536490572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08153e10 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d0e536640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 0x7f4d0814e4f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4d0402b8b0 tx=0x7f4d040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04032870 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d04032a10 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d08154840 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04032e00 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.151+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d08154ae0 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.152+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0804f330 con 0x7f4d080aee20 2024-03-06T07:24:18.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.152+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d040453f0 con 0x7f4d080aee20 2024-03-06T07:24:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.153+0000 7f4cfeffd640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b4d0 0x7f4cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.153+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4d040ac120 con 0x7f4d080aee20 2024-03-06T07:24:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.153+0000 7f4d0dd35640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b4d0 0x7f4cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.153+0000 7f4d0dd35640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b4d0 0x7f4cec06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4d081546a0 tx=0x7f4cf8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.157+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d04040070 con 0x7f4d080aee20 2024-03-06T07:24:18.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:18.365+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4d0806b530 con 0x7f4d080aee20 2024-03-06T07:24:19.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.012+0000 7f4cfeffd640 1 -- 172.21.15.122:0/1536490572 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v25) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4d0407b090 con 0x7f4d080aee20 2024-03-06T07:24:19.014 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:24:19.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.016+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b4d0 msgr2=0x7f4cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.016+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b4d0 0x7f4cec06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4d081546a0 tx=0x7f4cf8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:19.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.016+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 msgr2=0x7f4d0814e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.016+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 0x7f4d0814e4f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4d0402b8b0 tx=0x7f4d040046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 shutdown_connections 2024-03-06T07:24:19.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4cec06b4d0 0x7f4cec06d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d0815bc50 0x7f4d080719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d080cc280 0x7f4d081552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 --2- 172.21.15.122:0/1536490572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d080aee20 0x7f4d0814e4f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 >> 172.21.15.122:0/1536490572 conn(0x7f4d0806afb0 msgr2=0x7f4d080ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 shutdown_connections 2024-03-06T07:24:19.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.017+0000 7f4d1079a640 1 -- 172.21.15.122:0/1536490572 wait complete. 2024-03-06T07:24:19.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:19.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.145+0000 7f83bf965640 1 Processor -- start 2024-03-06T07:24:19.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.145+0000 7f83bf965640 1 -- start start 2024-03-06T07:24:19.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.146+0000 7f83bf965640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 0x7f83b815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.146+0000 7f83bf965640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b815f580 con 0x7f83b815bf30 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.146+0000 7f83bf965640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b815b450 con 0x7f83b813dc10 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.146+0000 7f83bf965640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80cf3f0 con 0x7f83b80b3060 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.146+0000 7f83bdf02640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f83b815bf30 0x7f83b815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33398/0 (socket says 172.21.15.122:33398) 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.146+0000 7f83bdf02640 1 -- 172.21.15.122:0/218643881 learned_addr learned my addr 172.21.15.122:0/218643881 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bcf00640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 0x7f83b815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216421536 0 0) 0x7f83b815f580 con 0x7f83b815bf30 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a4003520 con 0x7f83b815bf30 2024-03-06T07:24:19.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523509794 0 0) 0x7f83b815b450 con 0x7f83b813dc10 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b815f580 con 0x7f83b813dc10 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685075203 0 0) 0x7f83a4003520 con 0x7f83b815bf30 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 msgr2=0x7f83b815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 0x7f83b815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 >> v1:172.21.15.169:6789/0 conn(0x7f83b813dc10 legacy=0x7f83b80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b815b730 con 0x7f83b815bf30 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f83b0001670 con 0x7f83b815bf30 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 >> v1:172.21.15.122:6789/0 conn(0x7f83b815bf30 legacy=0x7f83b815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 0x7f83a4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83a40040d0 0x7f83a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83a4004d40 0x7f83a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80cf3f0 con 0x7f83a4004d40 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a400bb50 con 0x7f83a40040d0 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a400bd20 con 0x7f83b80b3060 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bdf02640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83a4004d40 0x7f83a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bd701640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 0x7f83a4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bcf00640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83a40040d0 0x7f83a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bdf02640 1 -- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 msgr2=0x7f83a4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bdf02640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b80b3060 0x7f83a4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bdf02640 1 -- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83a40040d0 msgr2=0x7f83a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bdf02640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83a40040d0 0x7f83a4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.147+0000 7f83bdf02640 1 -- 172.21.15.122:0/218643881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4003520 con 0x7f83a4004d40 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.148+0000 7f83bdf02640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83a4004d40 0x7f83a40051c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f83b0007fd0 tx=0x7f83a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3688e993d31afc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.148+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0007ca0 con 0x7f83a4004d40 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.148+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b0030070 con 0x7f83a4004d40 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.148+0000 7f83affff640 1 -- 172.21.15.122:0/218643881 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0005d70 con 0x7f83a4004d40 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bf965640 1 -- 172.21.15.122:0/218643881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83a4004d40 msgr2=0x7f83a40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bf965640 1 --2- 172.21.15.122:0/218643881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83a4004d40 0x7f83a40051c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f83b0007fd0 tx=0x7f83a4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bdf02640 1 -- 172.21.15.122:0/218643881 reap_dead start 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bf965640 1 -- 172.21.15.122:0/218643881 shutdown_connections 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bf965640 1 -- 172.21.15.122:0/218643881 >> 172.21.15.122:0/218643881 conn(0x7f83b80cf8a0 msgr2=0x7f83b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bf965640 1 -- 172.21.15.122:0/218643881 shutdown_connections 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.149+0000 7f83bf965640 1 -- 172.21.15.122:0/218643881 wait complete. 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.150+0000 7f83bf965640 1 Processor -- start 2024-03-06T07:24:19.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.150+0000 7f83bf965640 1 -- start start 2024-03-06T07:24:19.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bf965640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 0x7f83b8135550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bf965640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b8151170 0x7f83b806df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bf965640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83b81515f0 0x7f83b806e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bf965640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b815e850 con 0x7f83b8135170 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bf965640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b815b210 con 0x7f83b81515f0 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bf965640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b815e9d0 con 0x7f83b8151170 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 0x7f83b8135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 0x7f83b8135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46912/0 (socket says 172.21.15.122:46912) 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 -- 172.21.15.122:0/844760332 learned_addr learned my addr 172.21.15.122:0/844760332 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bcf00640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b8151170 0x7f83b806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bdf02640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83b81515f0 0x7f83b806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 -- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b8151170 msgr2=0x7f83b806df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b8151170 0x7f83b806df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 -- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83b81515f0 msgr2=0x7f83b806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83b81515f0 0x7f83b806e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 -- 172.21.15.122:0/844760332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b8151a70 con 0x7f83b8135170 2024-03-06T07:24:19.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.151+0000 7f83bd701640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 0x7f83b8135550 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f83b400fae0 tx=0x7f83b400fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.152+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4003b80 con 0x7f83b8135170 2024-03-06T07:24:19.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.152+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b4018070 con 0x7f83b8135170 2024-03-06T07:24:19.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.152+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b8151d50 con 0x7f83b8135170 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.152+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b400d850 con 0x7f83b8135170 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.152+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b8152310 con 0x7f83b8135170 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.153+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b804f330 con 0x7f83b8135170 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.153+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83b400d9f0 con 0x7f83b8135170 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.153+0000 7f83adffb640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f839c06b4d0 0x7f839c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.154+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83b408a360 con 0x7f83b8135170 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.154+0000 7f83bcf00640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f839c06b4d0 0x7f839c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.154+0000 7f83bcf00640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f839c06b4d0 0x7f839c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f83b80b1550 tx=0x7f83a8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.158+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83b40591e0 con 0x7f83b8135170 2024-03-06T07:24:19.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.332+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83b806eb00 con 0x7f83b8135170 2024-03-06T07:24:19.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.334+0000 7f83adffb640 1 -- 172.21.15.122:0/844760332 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16678 (secure 0 0 0) 0x7f83b4058ae0 con 0x7f83b8135170 2024-03-06T07:24:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:19.336 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":56,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:18.366948+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.335+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f839c06b4d0 msgr2=0x7f839c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.335+0000 7f83bf965640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f839c06b4d0 0x7f839c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f83b80b1550 tx=0x7f83a8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.335+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 msgr2=0x7f83b8135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.335+0000 7f83bf965640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 0x7f83b8135550 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f83b400fae0 tx=0x7f83b400fee0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 shutdown_connections 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f839c06b4d0 0x7f839c06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83b81515f0 0x7f83b806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83b8151170 0x7f83b806df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 --2- 172.21.15.122:0/844760332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83b8135170 0x7f83b8135550 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 >> 172.21.15.122:0/844760332 conn(0x7f83b80cf8a0 msgr2=0x7f83b80cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 shutdown_connections 2024-03-06T07:24:19.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.336+0000 7f83bf965640 1 -- 172.21.15.122:0/844760332 wait complete. 2024-03-06T07:24:19.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.447+0000 7f85062c2640 1 Processor -- start 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.447+0000 7f85062c2640 1 -- start start 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f85062c2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f85000d0320 0x7f85000b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f85062c2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f85000b54b0 0x7f85000cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f85062c2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850015c270 con 0x7f85000d0700 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f85062c2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000b04e0 con 0x7f85000d0320 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f85062c2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000d0e90 con 0x7f85000b54b0 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f84fffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f85000d0320 0x7f85000b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f84ff7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f85000d0700 0x7f85000ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33402/0 (socket says 172.21.15.122:33402) 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f84fffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f85000d0320 0x7f85000b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46914/0 (socket says 172.21.15.122:46914) 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.448+0000 7f84ff7fe640 1 -- 172.21.15.122:0/2223201786 learned_addr learned my addr 172.21.15.122:0/2223201786 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84fffff640 1 -- 172.21.15.122:0/2223201786 >> v2:172.21.15.169:3300/0 conn(0x7f85000b54b0 msgr2=0x7f85000cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f850485f640 1 --2- 172.21.15.122:0/2223201786 >> v2:172.21.15.169:3300/0 conn(0x7f85000b54b0 0x7f85000cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84feffd640 1 -- 172.21.15.122:0/2223201786 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769148229 0 0) 0x7f850015c270 con 0x7f85000d0700 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84fffff640 1 --2- 172.21.15.122:0/2223201786 >> v2:172.21.15.169:3300/0 conn(0x7f85000b54b0 0x7f85000cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84fffff640 1 -- 172.21.15.122:0/2223201786 >> v1:172.21.15.122:6789/0 conn(0x7f85000d0700 legacy=0x7f85000ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84fffff640 1 -- 172.21.15.122:0/2223201786 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000b5ae0 con 0x7f85000d0320 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84fffff640 1 --2- 172.21.15.122:0/2223201786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85000b4f70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8500073570 tx=0x7f84f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528960eba84fe1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84feffd640 1 -- 172.21.15.122:0/2223201786 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0035020 con 0x7f85000d0320 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84feffd640 1 -- 172.21.15.122:0/2223201786 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0031830 con 0x7f85000d0320 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.449+0000 7f84feffd640 1 -- 172.21.15.122:0/2223201786 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0031c00 con 0x7f85000d0320 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 -- 172.21.15.122:0/2223201786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 msgr2=0x7f85000b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 --2- 172.21.15.122:0/2223201786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85000b4f70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8500073570 tx=0x7f84f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 -- 172.21.15.122:0/2223201786 shutdown_connections 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 --2- 172.21.15.122:0/2223201786 >> v2:172.21.15.169:3300/0 conn(0x7f85000b54b0 0x7f85000cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 --2- 172.21.15.122:0/2223201786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85000b4f70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 -- 172.21.15.122:0/2223201786 >> 172.21.15.122:0/2223201786 conn(0x7f85000724d0 msgr2=0x7f85000728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 -- 172.21.15.122:0/2223201786 shutdown_connections 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.450+0000 7f85062c2640 1 -- 172.21.15.122:0/2223201786 wait complete. 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.451+0000 7f85062c2640 1 Processor -- start 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.451+0000 7f85062c2640 1 -- start start 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f85062c2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f85000b54b0 0x7f85000cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f85062c2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85001e1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f85062c2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85000d0700 0x7f850006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f85062c2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85000b60a0 con 0x7f85000d0320 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f85062c2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85000b0170 con 0x7f85000d0700 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f85062c2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85000ca790 con 0x7f85000b54b0 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85001e1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85001e1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46920/0 (socket says 172.21.15.122:46920) 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 -- 172.21.15.122:0/508156485 learned_addr learned my addr 172.21.15.122:0/508156485 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:19.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f850485f640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85000d0700 0x7f850006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84fffff640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f85000b54b0 0x7f85000cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 -- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f85000b54b0 msgr2=0x7f85000cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f85000b54b0 0x7f85000cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 -- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85000d0700 msgr2=0x7f850006ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85000d0700 0x7f850006ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.452+0000 7f84ff7fe640 1 -- 172.21.15.122:0/508156485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85001502c0 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.453+0000 7f84ff7fe640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85001e1170 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f84f400c4b0 tx=0x7f84f400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.453+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4018070 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.453+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4007500 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.453+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85001505a0 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.453+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4004a20 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.453+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8500150e40 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.454+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8500150990 con 0x7f85000d0320 2024-03-06T07:24:19.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.454+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84f4004bc0 con 0x7f85000d0320 2024-03-06T07:24:19.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.454+0000 7f84fcff9640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.455+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84f40894d0 con 0x7f85000d0320 2024-03-06T07:24:19.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.455+0000 7f84fffff640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.455+0000 7f84fffff640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ec06b4d0 0x7f84ec06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f84f0036000 tx=0x7f84f003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.459+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84f4058420 con 0x7f85000d0320 2024-03-06T07:24:19.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.633+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f850004f330 con 0x7f85000d0320 2024-03-06T07:24:19.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.634+0000 7f84fcff9640 1 -- 172.21.15.122:0/508156485 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16678 (secure 0 0 0) 0x7f84f4087e20 con 0x7f85000d0320 2024-03-06T07:24:19.636 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:19.636 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":56,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:18.366948+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ec06b4d0 msgr2=0x7f84ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ec06b4d0 0x7f84ec06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f84f0036000 tx=0x7f84f003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 msgr2=0x7f85001e1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85001e1170 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f84f400c4b0 tx=0x7f84f400c970 comp rx=0 tx=0).stop 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 shutdown_connections 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85000d0700 0x7f850006ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f85000d0320 0x7f85001e1170 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 --2- 172.21.15.122:0/508156485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f85000b54b0 0x7f85000cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 >> 172.21.15.122:0/508156485 conn(0x7f85000724d0 msgr2=0x7f85000b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 shutdown_connections 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.636+0000 7f85062c2640 1 -- 172.21.15.122:0/508156485 wait complete. 2024-03-06T07:24:19.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:19.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.750+0000 7f05b1c5c640 1 Processor -- start 2024-03-06T07:24:19.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.750+0000 7f05b1c5c640 1 -- start start 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05b1c5c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05ac0d4860 0x7f05ac0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05b1c5c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f05ac0b0e30 0x7f05ac0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05b1c5c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac1618b0 con 0x7f05ac1614d0 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05b1c5c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac0d4cf0 con 0x7f05ac0d4860 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05b1c5c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac0cf400 con 0x7f05ac0b0e30 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05ac0d4860 0x7f05ac0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05ac0d4860 0x7f05ac0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46936/0 (socket says 172.21.15.122:46936) 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 -- 172.21.15.122:0/4087138430 learned_addr learned my addr 172.21.15.122:0/4087138430 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:19.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05abfff640 1 --2- 172.21.15.122:0/4087138430 >> v2:172.21.15.169:3300/0 conn(0x7f05ac0b0e30 0x7f05ac0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 -- 172.21.15.122:0/4087138430 >> v2:172.21.15.169:3300/0 conn(0x7f05ac0b0e30 msgr2=0x7f05ac0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 --2- 172.21.15.122:0/4087138430 >> v2:172.21.15.169:3300/0 conn(0x7f05ac0b0e30 0x7f05ac0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 -- 172.21.15.122:0/4087138430 >> v1:172.21.15.169:6789/0 conn(0x7f05ac1614d0 legacy=0x7f05ac0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05aaffd640 1 -- 172.21.15.122:0/4087138430 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0cf6a0 con 0x7f05ac0d4860 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.751+0000 7f05abfff640 1 --2- 172.21.15.122:0/4087138430 >> v2:172.21.15.169:3300/0 conn(0x7f05ac0b0e30 0x7f05ac0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.752+0000 7f05aaffd640 1 --2- 172.21.15.122:0/4087138430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac0d4860 0x7f05ac0d2020 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0ce550 tx=0x7f05a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3eeea3fe509eaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.752+0000 7f05aa7fc640 1 -- 172.21.15.122:0/4087138430 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0035020 con 0x7f05ac0d4860 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.752+0000 7f05aa7fc640 1 -- 172.21.15.122:0/4087138430 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0031640 con 0x7f05ac0d4860 2024-03-06T07:24:19.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.752+0000 7f05aa7fc640 1 -- 172.21.15.122:0/4087138430 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a002d040 con 0x7f05ac0d4860 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.752+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4087138430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac0d4860 msgr2=0x7f05ac0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.752+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4087138430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac0d4860 0x7f05ac0d2020 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0ce550 tx=0x7f05a002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.753+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4087138430 shutdown_connections 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.753+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4087138430 >> v2:172.21.15.169:3300/0 conn(0x7f05ac0b0e30 0x7f05ac0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.753+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4087138430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac0d4860 0x7f05ac0d2020 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.753+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4087138430 >> 172.21.15.122:0/4087138430 conn(0x7f05ac0d3d40 msgr2=0x7f05ac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.753+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4087138430 shutdown_connections 2024-03-06T07:24:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.753+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4087138430 wait complete. 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.754+0000 7f05b1c5c640 1 Processor -- start 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.754+0000 7f05b1c5c640 1 -- start start 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0b0e30 0x7f05ac15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0d4860 0x7f05ac0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 0x7f05ac0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0b0910 con 0x7f05ac1614d0 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0cfc60 con 0x7f05ac0d4860 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0d2560 con 0x7f05ac0b0e30 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 0x7f05ac0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 0x7f05ac0b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46938/0 (socket says 172.21.15.122:46938) 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 -- 172.21.15.122:0/4223273561 learned_addr learned my addr 172.21.15.122:0/4223273561 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05ab7fe640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0b0e30 0x7f05ac15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05aaffd640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0d4860 0x7f05ac0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 -- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0b0e30 msgr2=0x7f05ac15eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0b0e30 0x7f05ac15eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 -- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0d4860 msgr2=0x7f05ac0c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0d4860 0x7f05ac0c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 -- 172.21.15.122:0/4223273561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0b7800 con 0x7f05ac1614d0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05abfff640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 0x7f05ac0b4590 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f05a400cf20 tx=0x7f05a400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a401a070 con 0x7f05ac1614d0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a400d840 con 0x7f05ac1614d0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400dc30 con 0x7f05ac1614d0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac0b7ae0 con 0x7f05ac1614d0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.755+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac0b80a0 con 0x7f05ac1614d0 2024-03-06T07:24:19.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.756+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac068ff0 con 0x7f05ac1614d0 2024-03-06T07:24:19.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.757+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05a401d070 con 0x7f05ac1614d0 2024-03-06T07:24:19.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.757+0000 7f0593fff640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f059806b4d0 0x7f059806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.757+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05a408aaf0 con 0x7f05ac1614d0 2024-03-06T07:24:19.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.757+0000 7f05ab7fe640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f059806b4d0 0x7f059806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.757+0000 7f05ab7fe640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f059806b4d0 0x7f059806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f059c006fd0 tx=0x7f059c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.761+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05a4059970 con 0x7f05ac1614d0 2024-03-06T07:24:19.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:19.967+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f05ac0d2250 con 0x7f05ac1614d0 2024-03-06T07:24:20.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.027+0000 7f0593fff640 1 -- 172.21.15.122:0/4223273561 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7f05a4059270 con 0x7f05ac1614d0 2024-03-06T07:24:20.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f059806b4d0 msgr2=0x7f059806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f059806b4d0 0x7f059806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f059c006fd0 tx=0x7f059c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 msgr2=0x7f05ac0b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 0x7f05ac0b4590 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f05a400cf20 tx=0x7f05a400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 shutdown_connections 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f059806b4d0 0x7f059806d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac1614d0 0x7f05ac0b4590 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0d4860 0x7f05ac0c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 --2- 172.21.15.122:0/4223273561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0b0e30 0x7f05ac15eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.030+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 >> 172.21.15.122:0/4223273561 conn(0x7f05ac0d3d40 msgr2=0x7f05ac0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.031+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 shutdown_connections 2024-03-06T07:24:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.031+0000 7f05b1c5c640 1 -- 172.21.15.122:0/4223273561 wait complete. 2024-03-06T07:24:20.043 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:20.044 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:20.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.145+0000 7f09db7f6640 1 Processor -- start 2024-03-06T07:24:20.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09db7f6640 1 -- start start 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09db7f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d41586e0 0x7f09d40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09db7f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09d40aef80 0x7f09d40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09db7f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40d03f0 con 0x7f09d40cffd0 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09db7f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d4158ac0 con 0x7f09d40aef80 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09db7f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415b2a0 con 0x7f09d41586e0 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.146+0000 7f09d8d91640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09d40aef80 0x7f09d40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09d40aef80 0x7f09d40cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46940/0 (socket says 172.21.15.122:46940) 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 -- 172.21.15.122:0/243731477 learned_addr learned my addr 172.21.15.122:0/243731477 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d9592640 1 --2- 172.21.15.122:0/243731477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d41586e0 0x7f09d40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 -- 172.21.15.122:0/243731477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d41586e0 msgr2=0x7f09d40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 --2- 172.21.15.122:0/243731477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d41586e0 0x7f09d40aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 -- 172.21.15.122:0/243731477 >> v1:172.21.15.169:6789/0 conn(0x7f09d40cffd0 legacy=0x7f09d406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 -- 172.21.15.122:0/243731477 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d415b540 con 0x7f09d40aef80 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.147+0000 7f09d8d91640 1 --2- 172.21.15.122:0/243731477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d40cfa90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009870 tx=0x7f09c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb15c44657be30af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09cbfff640 1 -- 172.21.15.122:0/243731477 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4034070 con 0x7f09d40aef80 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09cbfff640 1 -- 172.21.15.122:0/243731477 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c4030840 con 0x7f09d40aef80 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09cbfff640 1 -- 172.21.15.122:0/243731477 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4030c10 con 0x7f09d40aef80 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 -- 172.21.15.122:0/243731477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 msgr2=0x7f09d40cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 --2- 172.21.15.122:0/243731477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d40cfa90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009870 tx=0x7f09c402b980 comp rx=0 tx=0).stop 2024-03-06T07:24:20.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 -- 172.21.15.122:0/243731477 shutdown_connections 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 --2- 172.21.15.122:0/243731477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d40cfa90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 --2- 172.21.15.122:0/243731477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d41586e0 0x7f09d40aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 -- 172.21.15.122:0/243731477 >> 172.21.15.122:0/243731477 conn(0x7f09d406b190 msgr2=0x7f09d406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 -- 172.21.15.122:0/243731477 shutdown_connections 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.148+0000 7f09db7f6640 1 -- 172.21.15.122:0/243731477 wait complete. 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.149+0000 7f09db7f6640 1 Processor -- start 2024-03-06T07:24:20.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.149+0000 7f09db7f6640 1 -- start start 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09db7f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09db7f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d40cffd0 0x7f09d40b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09db7f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09d41586e0 0x7f09d41f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09db7f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415bde0 con 0x7f09d40aef80 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09db7f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415bb00 con 0x7f09d41586e0 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09db7f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4071350 con 0x7f09d40cffd0 2024-03-06T07:24:20.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09d9592640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09d9592640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d414f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46954/0 (socket says 172.21.15.122:46954) 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09d9592640 1 -- 172.21.15.122:0/3070106455 learned_addr learned my addr 172.21.15.122:0/3070106455 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09d8d91640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d40cffd0 0x7f09d40b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.150+0000 7f09d9d93640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09d41586e0 0x7f09d41f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09d9592640 1 -- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d40cffd0 msgr2=0x7f09d40b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09d9592640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d40cffd0 0x7f09d40b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09d9592640 1 -- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09d41586e0 msgr2=0x7f09d41f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09d9592640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09d41586e0 0x7f09d41f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09d9592640 1 -- 172.21.15.122:0/3070106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40b1240 con 0x7f09d40aef80 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09d9592640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d414f980 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f09d000e870 tx=0x7f09d000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0017070 con 0x7f09d40aef80 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d000c940 con 0x7f09d40aef80 2024-03-06T07:24:20.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d406e840 con 0x7f09d40aef80 2024-03-06T07:24:20.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d000cd30 con 0x7f09d40aef80 2024-03-06T07:24:20.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.151+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d406e280 con 0x7f09d40aef80 2024-03-06T07:24:20.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.152+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d40732b0 con 0x7f09d40aef80 2024-03-06T07:24:20.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.152+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09d001a070 con 0x7f09d40aef80 2024-03-06T07:24:20.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.153+0000 7f09c9ffb640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09b806b4d0 0x7f09b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.153+0000 7f09d8d91640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09b806b4d0 0x7f09b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.153+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09d0089fa0 con 0x7f09d40aef80 2024-03-06T07:24:20.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.154+0000 7f09d8d91640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09b806b4d0 0x7f09b806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f09c4036000 tx=0x7f09c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.157+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09d0058860 con 0x7f09d40aef80 2024-03-06T07:24:20.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:20.367+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f09d4070ec0 con 0x7f09d40aef80 2024-03-06T07:24:21.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.037+0000 7f09c9ffb640 1 -- 172.21.15.122:0/3070106455 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09d005c900 con 0x7f09d40aef80 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09b806b4d0 msgr2=0x7f09b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09b806b4d0 0x7f09b806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f09c4036000 tx=0x7f09c4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 msgr2=0x7f09d414f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d414f980 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f09d000e870 tx=0x7f09d000ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 shutdown_connections 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09b806b4d0 0x7f09b806d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09d41586e0 0x7f09d41f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09d40cffd0 0x7f09d40b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 --2- 172.21.15.122:0/3070106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09d40aef80 0x7f09d414f980 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.039+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 >> 172.21.15.122:0/3070106455 conn(0x7f09d406b190 msgr2=0x7f09d4154510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.040+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 shutdown_connections 2024-03-06T07:24:21.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.040+0000 7f09db7f6640 1 -- 172.21.15.122:0/3070106455 wait complete. 2024-03-06T07:24:21.053 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:21.054 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:21.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.156+0000 7f8f9cbf6640 1 Processor -- start 2024-03-06T07:24:21.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.156+0000 7f8f9cbf6640 1 -- start start 2024-03-06T07:24:21.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9cbf6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f9813dc10 0x7f8f980b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9cbf6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8f9815bf30 0x7f8f980b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9cbf6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9815f580 con 0x7f8f9815e850 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9cbf6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f980cf3f0 con 0x7f8f9813dc10 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9cbf6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9815b210 con 0x7f8f9815bf30 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f9813dc10 0x7f8f980b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f96d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8f9815e850 0x7f8f9815ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33418/0 (socket says 172.21.15.122:33418) 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f9813dc10 0x7f8f980b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46962/0 (socket says 172.21.15.122:46962) 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f96d9d640 1 -- 172.21.15.122:0/488619606 learned_addr learned my addr 172.21.15.122:0/488619606 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f8ffff640 1 --2- 172.21.15.122:0/488619606 >> v2:172.21.15.169:3300/0 conn(0x7f8f9815bf30 0x7f8f980b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9659c640 1 -- 172.21.15.122:0/488619606 >> v2:172.21.15.169:3300/0 conn(0x7f8f9815bf30 msgr2=0x7f8f980b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9659c640 1 --2- 172.21.15.122:0/488619606 >> v2:172.21.15.169:3300/0 conn(0x7f8f9815bf30 0x7f8f980b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9659c640 1 -- 172.21.15.122:0/488619606 >> v1:172.21.15.122:6789/0 conn(0x7f8f9815e850 legacy=0x7f8f9815ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f95d9b640 1 -- 172.21.15.122:0/488619606 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600528589 0 0) 0x7f8f9815f580 con 0x7f8f9815e850 2024-03-06T07:24:21.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.157+0000 7f8f9659c640 1 -- 172.21.15.122:0/488619606 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9815b450 con 0x7f8f9813dc10 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.158+0000 7f8f9659c640 1 --2- 172.21.15.122:0/488619606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9813dc10 0x7f8f980b2950 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8f9806c790 tx=0x7f8f8802e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d78f45cea04a6f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.158+0000 7f8f95d9b640 1 -- 172.21.15.122:0/488619606 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8803b070 con 0x7f8f9813dc10 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.158+0000 7f8f95d9b640 1 -- 172.21.15.122:0/488619606 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8802edc0 con 0x7f8f9813dc10 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.158+0000 7f8f95d9b640 1 -- 172.21.15.122:0/488619606 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88036660 con 0x7f8f9813dc10 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/488619606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9813dc10 msgr2=0x7f8f980b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/488619606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9813dc10 0x7f8f980b2950 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8f9806c790 tx=0x7f8f8802e010 comp rx=0 tx=0).stop 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/488619606 shutdown_connections 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/488619606 >> v2:172.21.15.169:3300/0 conn(0x7f8f9815bf30 0x7f8f980b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/488619606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9813dc10 0x7f8f980b2950 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/488619606 >> 172.21.15.122:0/488619606 conn(0x7f8f980cf8a0 msgr2=0x7f8f980cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/488619606 shutdown_connections 2024-03-06T07:24:21.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.159+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/488619606 wait complete. 2024-03-06T07:24:21.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.160+0000 7f8f9cbf6640 1 Processor -- start 2024-03-06T07:24:21.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.160+0000 7f8f9cbf6640 1 -- start start 2024-03-06T07:24:21.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9cbf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f9813dc10 0x7f8f9813c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9cbf6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f9815bf30 0x7f8f9813c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9cbf6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 0x7f8f9813ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9cbf6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980cd7a0 con 0x7f8f9815e850 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9cbf6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9813dff0 con 0x7f8f9813dc10 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9cbf6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980cd920 con 0x7f8f9815bf30 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 0x7f8f9813ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 0x7f8f9813ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46966/0 (socket says 172.21.15.122:46966) 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 -- 172.21.15.122:0/3244378886 learned_addr learned my addr 172.21.15.122:0/3244378886 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f8ffff640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f9815bf30 0x7f8f9813c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f9659c640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f9813dc10 0x7f8f9813c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 -- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f9815bf30 msgr2=0x7f8f9813c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f9815bf30 0x7f8f9813c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 -- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f9813dc10 msgr2=0x7f8f9813c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f9813dc10 0x7f8f9813c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 -- 172.21.15.122:0/3244378886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98071900 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f96d9d640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 0x7f8f9813ef80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000c700 tx=0x7f8f9000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90018070 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.161+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9000ce10 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.162+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9000a860 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.162+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f98071be0 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.162+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f980721a0 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.163+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f980724d0 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.163+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f9001b070 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.163+0000 7f8f8effd640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f7806b4d0 0x7f8f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.163+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f900897e0 con 0x7f8f9815e850 2024-03-06T07:24:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.163+0000 7f8f9659c640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f7806b4d0 0x7f8f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.164+0000 7f8f9659c640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f7806b4d0 0x7f8f7806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8f88038000 tx=0x7f8f88032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.167+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f90057f80 con 0x7f8f9815e850 2024-03-06T07:24:21.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.375+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f98155d40 con 0x7f8f9815e850 2024-03-06T07:24:21.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.375+0000 7f8f8effd640 1 -- 172.21.15.122:0/3244378886 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f8f98155d40 con 0x7f8f9815e850 2024-03-06T07:24:21.377 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:21.377 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:21.378 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:21.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f7806b4d0 msgr2=0x7f8f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f7806b4d0 0x7f8f7806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8f88038000 tx=0x7f8f88032000 comp rx=0 tx=0).stop 2024-03-06T07:24:21.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 msgr2=0x7f8f9813ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 0x7f8f9813ef80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000c700 tx=0x7f8f9000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 shutdown_connections 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f7806b4d0 0x7f8f7806d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9815e850 0x7f8f9813ef80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f9815bf30 0x7f8f9813c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 --2- 172.21.15.122:0/3244378886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f9813dc10 0x7f8f9813c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 >> 172.21.15.122:0/3244378886 conn(0x7f8f980cf8a0 msgr2=0x7f8f9815d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 shutdown_connections 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.377+0000 7f8f9cbf6640 1 -- 172.21.15.122:0/3244378886 wait complete. 2024-03-06T07:24:21.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:21.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.488+0000 7fb86cf3a640 1 Processor -- start 2024-03-06T07:24:21.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.488+0000 7fb86cf3a640 1 -- start start 2024-03-06T07:24:21.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86cf3a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb86815bb20 0x7fb8680d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86cf3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8680d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86cf3a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb8680ce4b0 0x7fb8680ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86cf3a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb868161980 con 0x7fb86815bb20 2024-03-06T07:24:21.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86cf3a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb868161c60 con 0x7fb8680ce4b0 2024-03-06T07:24:21.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86cf3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb868161f40 con 0x7fb8680cfff0 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb86815bb20 0x7fb8680d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb86815bb20 0x7fb8680d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46982/0 (socket says 172.21.15.122:46982) 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86659c640 1 -- 172.21.15.122:0/2685645965 learned_addr learned my addr 172.21.15.122:0/2685645965 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb866d9d640 1 --2- 172.21.15.122:0/2685645965 >> v2:172.21.15.169:3300/0 conn(0x7fb8680ce4b0 0x7fb8680ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb865d9b640 1 --2- 172.21.15.122:0/2685645965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8680d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.489+0000 7fb86659c640 1 -- 172.21.15.122:0/2685645965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 msgr2=0x7fb8680d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86659c640 1 --2- 172.21.15.122:0/2685645965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8680d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86659c640 1 -- 172.21.15.122:0/2685645965 >> v2:172.21.15.169:3300/0 conn(0x7fb8680ce4b0 msgr2=0x7fb8680ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86659c640 1 --2- 172.21.15.122:0/2685645965 >> v2:172.21.15.169:3300/0 conn(0x7fb8680ce4b0 0x7fb8680ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86659c640 1 -- 172.21.15.122:0/2685645965 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868162220 con 0x7fb86815bb20 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86659c640 1 --2- 172.21.15.122:0/2685645965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86815bb20 0x7fb8680d1df0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb8680b13b0 tx=0x7fb85802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66553ab031acb816 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86559a640 1 -- 172.21.15.122:0/2685645965 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858035020 con 0x7fb86815bb20 2024-03-06T07:24:21.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86559a640 1 -- 172.21.15.122:0/2685645965 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb858031870 con 0x7fb86815bb20 2024-03-06T07:24:21.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.490+0000 7fb86559a640 1 -- 172.21.15.122:0/2685645965 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858031c40 con 0x7fb86815bb20 2024-03-06T07:24:21.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2685645965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86815bb20 msgr2=0x7fb8680d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2685645965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86815bb20 0x7fb8680d1df0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb8680b13b0 tx=0x7fb85802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:21.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2685645965 shutdown_connections 2024-03-06T07:24:21.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2685645965 >> v2:172.21.15.169:3300/0 conn(0x7fb8680ce4b0 0x7fb8680ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2685645965 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8680d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2685645965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86815bb20 0x7fb8680d1df0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2685645965 >> 172.21.15.122:0/2685645965 conn(0x7fb86809a9f0 msgr2=0x7fb86809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2685645965 shutdown_connections 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.491+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2685645965 wait complete. 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.492+0000 7fb86cf3a640 1 Processor -- start 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.492+0000 7fb86cf3a640 1 -- start start 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86cf3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 0x7fb86815af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86cf3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8681f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86cf3a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86815bb20 0x7fb8680c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86cf3a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8680d24b0 con 0x7fb8680ce4b0 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86cf3a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8680d2330 con 0x7fb86815bb20 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86cf3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8680d1980 con 0x7fb8680cfff0 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 0x7fb86815af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 0x7fb86815af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46996/0 (socket says 172.21.15.122:46996) 2024-03-06T07:24:21.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 -- 172.21.15.122:0/2377568106 learned_addr learned my addr 172.21.15.122:0/2377568106 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb865d9b640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8681f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb866d9d640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86815bb20 0x7fb8680c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 -- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 msgr2=0x7fb8681f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8681f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 -- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86815bb20 msgr2=0x7fb8680c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86815bb20 0x7fb8680c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.493+0000 7fb86659c640 1 -- 172.21.15.122:0/2377568106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8680c5690 con 0x7fb8680ce4b0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.494+0000 7fb86659c640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 0x7fb86815af60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb85802b950 tx=0x7fb8580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.494+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858040070 con 0x7fb8680ce4b0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.494+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb858031e10 con 0x7fb8680ce4b0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.494+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8680c5900 con 0x7fb8680ce4b0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.494+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858035040 con 0x7fb8680ce4b0 2024-03-06T07:24:21.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.494+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8680c5ec0 con 0x7fb8680ce4b0 2024-03-06T07:24:21.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.495+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86804f330 con 0x7fb8680ce4b0 2024-03-06T07:24:21.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.495+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb85804b060 con 0x7fb8680ce4b0 2024-03-06T07:24:21.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.495+0000 7fb856ffd640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb84806b4d0 0x7fb84806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.496+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8580abce0 con 0x7fb8680ce4b0 2024-03-06T07:24:21.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.496+0000 7fb865d9b640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb84806b4d0 0x7fb84806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.496+0000 7fb865d9b640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb84806b4d0 0x7fb84806d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb86809aed0 tx=0x7fb85c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.500+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb85807a510 con 0x7fb8680ce4b0 2024-03-06T07:24:21.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.673+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8680b1820 con 0x7fb8680ce4b0 2024-03-06T07:24:21.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.674+0000 7fb856ffd640 1 -- 172.21.15.122:0/2377568106 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7fb85807e5b0 con 0x7fb8680ce4b0 2024-03-06T07:24:21.676 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:21.676 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb84806b4d0 msgr2=0x7fb84806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb84806b4d0 0x7fb84806d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb86809aed0 tx=0x7fb85c009190 comp rx=0 tx=0).stop 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 msgr2=0x7fb86815af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 0x7fb86815af60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb85802b950 tx=0x7fb8580046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 shutdown_connections 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb84806b4d0 0x7fb84806d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86815bb20 0x7fb8680c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb8680cfff0 0x7fb8681f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 --2- 172.21.15.122:0/2377568106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8680ce4b0 0x7fb86815af60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 >> 172.21.15.122:0/2377568106 conn(0x7fb86809a9f0 msgr2=0x7fb868070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 shutdown_connections 2024-03-06T07:24:21.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.676+0000 7fb86cf3a640 1 -- 172.21.15.122:0/2377568106 wait complete. 2024-03-06T07:24:21.689 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:21.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:21.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.788+0000 7f6351bf7640 1 Processor -- start 2024-03-06T07:24:21.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.788+0000 7f6351bf7640 1 -- start start 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f6351bf7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f634c1424f0 0x7f634c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f6351bf7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f6351bf7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c0cc980 con 0x7f634c0cd7a0 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f6351bf7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634c0cd220 con 0x7f634c1424f0 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f6351bf7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0ccc60 con 0x7f634c06da40 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f634c1424f0 0x7f634c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634bfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f634c1424f0 0x7f634c0cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:34812/0 (socket says 172.21.15.122:34812) 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 -- 172.21.15.122:0/2240024804 learned_addr learned my addr 172.21.15.122:0/2240024804 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634bfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c15cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:55598/0 (socket says 172.21.15.122:55598) 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 -- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 msgr2=0x7f634c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 --2- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 -- 172.21.15.122:0/2240024804 >> v1:172.21.15.169:6789/0 conn(0x7f634c0cd7a0 legacy=0x7f634c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.789+0000 7f634b7fe640 1 -- 172.21.15.122:0/2240024804 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0ccf40 con 0x7f634c1424f0 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.790+0000 7f634b7fe640 1 --2- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0cc2f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f633c009870 tx=0x7f633c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55dcf311e241d7e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.790+0000 7f634a7fc640 1 -- 172.21.15.122:0/2240024804 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c034070 con 0x7f634c1424f0 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.790+0000 7f634a7fc640 1 -- 172.21.15.122:0/2240024804 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c030840 con 0x7f634c1424f0 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.790+0000 7f634a7fc640 1 -- 172.21.15.122:0/2240024804 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c030c10 con 0x7f634c1424f0 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 -- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 msgr2=0x7f634c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 --2- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0cc2f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f633c009870 tx=0x7f633c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 -- 172.21.15.122:0/2240024804 shutdown_connections 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 --2- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 --2- 172.21.15.122:0/2240024804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0cc2f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 -- 172.21.15.122:0/2240024804 >> 172.21.15.122:0/2240024804 conn(0x7f634c0cf8a0 msgr2=0x7f634c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 -- 172.21.15.122:0/2240024804 shutdown_connections 2024-03-06T07:24:21.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.791+0000 7f6351bf7640 1 -- 172.21.15.122:0/2240024804 wait complete. 2024-03-06T07:24:21.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.792+0000 7f6351bf7640 1 Processor -- start 2024-03-06T07:24:21.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.792+0000 7f6351bf7640 1 -- start start 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f6351bf7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c1594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f6351bf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 0x7f634c1e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f6351bf7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f6351bf7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c071ce0 con 0x7f634c0cd7a0 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f6351bf7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c071b60 con 0x7f634c1424f0 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f6351bf7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c071e60 con 0x7f634c06da40 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 0x7f634c1e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 0x7f634c1e72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47010/0 (socket says 172.21.15.122:47010) 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 -- 172.21.15.122:0/3139726113 learned_addr learned my addr 172.21.15.122:0/3139726113 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634b7fe640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c1594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634bfff640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 -- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 msgr2=0x7f634c1594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c1594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 -- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 msgr2=0x7f634c0abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.793+0000 7f634affd640 1 -- 172.21.15.122:0/3139726113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0a97a0 con 0x7f634c0cd7a0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.794+0000 7f634affd640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 0x7f634c1e72e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f634000c450 tx=0x7f634000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.794+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63400052d0 con 0x7f634c0cd7a0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.794+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6340005470 con 0x7f634c0cd7a0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.794+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340012870 con 0x7f634c0cd7a0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.794+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c1e7d30 con 0x7f634c0cd7a0 2024-03-06T07:24:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.794+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c0a9a60 con 0x7f634c0cd7a0 2024-03-06T07:24:21.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.795+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c0682b0 con 0x7f634c0cd7a0 2024-03-06T07:24:21.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.795+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6340012a10 con 0x7f634c0cd7a0 2024-03-06T07:24:21.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.795+0000 7f6333fff640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f633806b500 0x7f633806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.796+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6340089c20 con 0x7f634c0cd7a0 2024-03-06T07:24:21.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.796+0000 7f634b7fe640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f633806b500 0x7f633806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.796+0000 7f634b7fe640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f633806b500 0x7f633806d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f633c036000 tx=0x7f633c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.800+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f634c0682b0 con 0x7f634c0cd7a0 2024-03-06T07:24:21.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.968+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f634c1e8250 con 0x7f633806b500 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.974+0000 7f6333fff640 1 -- 172.21.15.122:0/3139726113 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251951 (secure 0 0 0) 0x7f633c030810 con 0x7f633806b500 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:21.980 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f633806b500 msgr2=0x7f633806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f633806b500 0x7f633806d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f633c036000 tx=0x7f633c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 msgr2=0x7f634c1e72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 0x7f634c1e72e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f634000c450 tx=0x7f634000c910 comp rx=0 tx=0).stop 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 shutdown_connections 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f633806b500 0x7f633806d9c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f634c1424f0 0x7f634c0abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f634c0cd7a0 0x7f634c1e72e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 --2- 172.21.15.122:0/3139726113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f634c06da40 0x7f634c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 >> 172.21.15.122:0/3139726113 conn(0x7f634c0cf8a0 msgr2=0x7f634c15aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 shutdown_connections 2024-03-06T07:24:21.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:21.977+0000 7f6351bf7640 1 -- 172.21.15.122:0/3139726113 wait complete. 2024-03-06T07:24:21.990 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-03-06T07:24:21.360243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255980,"kb_used_data":6508,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718740,"statfs":{"total":773094113280,"available":772831989760,"internally_reserved":0,"allocated":6664192,"data_stored":3423032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.351256"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031202+0000","last_change":"2024-03-06T07:24:19.027740+0000","last_active":"2024-03-06T07:24:20.031202+0000","last_peered":"2024-03-06T07:24:20.031202+0000","last_clean":"2024-03-06T07:24:20.031202+0000","last_became_active":"2024-03-06T07:24:19.027500+0000","last_became_peered":"2024-03-06T07:24:19.027500+0000","last_unstale":"2024-03-06T07:24:20.031202+0000","last_undegraded":"2024-03-06T07:24:20.031202+0000","last_fullsized":"2024-03-06T07:24:20.031202+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:05.810668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058355+0000","last_change":"2024-03-06T07:24:19.029315+0000","last_active":"2024-03-06T07:24:20.058355+0000","last_peered":"2024-03-06T07:24:20.058355+0000","last_clean":"2024-03-06T07:24:20.058355+0000","last_became_active":"2024-03-06T07:24:19.029028+0000","last_became_peered":"2024-03-06T07:24:19.029028+0000","last_unstale":"2024-03-06T07:24:20.058355+0000","last_undegraded":"2024-03-06T07:24:20.058355+0000","last_fullsized":"2024-03-06T07:24:20.058355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:49.151798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059046+0000","last_change":"2024-03-06T07:24:19.037420+0000","last_active":"2024-03-06T07:24:20.059046+0000","last_peered":"2024-03-06T07:24:20.059046+0000","last_clean":"2024-03-06T07:24:20.059046+0000","last_became_active":"2024-03-06T07:24:19.037097+0000","last_became_peered":"2024-03-06T07:24:19.037097+0000","last_unstale":"2024-03-06T07:24:20.059046+0000","last_undegraded":"2024-03-06T07:24:20.059046+0000","last_fullsized":"2024-03-06T07:24:20.059046+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:54.420081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058887+0000","last_change":"2024-03-06T07:24:19.043360+0000","last_active":"2024-03-06T07:24:20.058887+0000","last_peered":"2024-03-06T07:24:20.058887+0000","last_clean":"2024-03-06T07:24:20.058887+0000","last_became_active":"2024-03-06T07:24:19.043085+0000","last_became_peered":"2024-03-06T07:24:19.043085+0000","last_unstale":"2024-03-06T07:24:20.058887+0000","last_undegraded":"2024-03-06T07:24:20.058887+0000","last_fullsized":"2024-03-06T07:24:20.058887+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:36.057457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031124+0000","last_change":"2024-03-06T07:24:19.039788+0000","last_active":"2024-03-06T07:24:20.031124+0000","last_peered":"2024-03-06T07:24:20.031124+0000","last_clean":"2024-03-06T07:24:20.031124+0000","last_became_active":"2024-03-06T07:24:19.039544+0000","last_became_peered":"2024-03-06T07:24:19.039544+0000","last_unstale":"2024-03-06T07:24:20.031124+0000","last_undegraded":"2024-03-06T07:24:20.031124+0000","last_fullsized":"2024-03-06T07:24:20.031124+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:26.122956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057801+0000","last_change":"2024-03-06T07:24:19.031360+0000","last_active":"2024-03-06T07:24:20.057801+0000","last_peered":"2024-03-06T07:24:20.057801+0000","last_clean":"2024-03-06T07:24:20.057801+0000","last_became_active":"2024-03-06T07:24:19.031084+0000","last_became_peered":"2024-03-06T07:24:19.031084+0000","last_unstale":"2024-03-06T07:24:20.057801+0000","last_undegraded":"2024-03-06T07:24:20.057801+0000","last_fullsized":"2024-03-06T07:24:20.057801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:36.057457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058038+0000","last_change":"2024-03-06T07:24:19.039716+0000","last_active":"2024-03-06T07:24:20.058038+0000","last_peered":"2024-03-06T07:24:20.058038+0000","last_clean":"2024-03-06T07:24:20.058038+0000","last_became_active":"2024-03-06T07:24:19.039219+0000","last_became_peered":"2024-03-06T07:24:19.039219+0000","last_unstale":"2024-03-06T07:24:20.058038+0000","last_undegraded":"2024-03-06T07:24:20.058038+0000","last_fullsized":"2024-03-06T07:24:20.058038+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:46.638512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057346+0000","last_change":"2024-03-06T07:24:19.037366+0000","last_active":"2024-03-06T07:24:20.057346+0000","last_peered":"2024-03-06T07:24:20.057346+0000","last_clean":"2024-03-06T07:24:20.057346+0000","last_became_active":"2024-03-06T07:24:19.037082+0000","last_became_peered":"2024-03-06T07:24:19.037082+0000","last_unstale":"2024-03-06T07:24:20.057346+0000","last_undegraded":"2024-03-06T07:24:20.057346+0000","last_fullsized":"2024-03-06T07:24:20.057346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:41.642813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057873+0000","last_change":"2024-03-06T07:24:19.037375+0000","last_active":"2024-03-06T07:24:20.057873+0000","last_peered":"2024-03-06T07:24:20.057873+0000","last_clean":"2024-03-06T07:24:20.057873+0000","last_became_active":"2024-03-06T07:24:19.036754+0000","last_became_peered":"2024-03-06T07:24:19.036754+0000","last_unstale":"2024-03-06T07:24:20.057873+0000","last_undegraded":"2024-03-06T07:24:20.057873+0000","last_fullsized":"2024-03-06T07:24:20.057873+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:31.236685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032035+0000","last_change":"2024-03-06T07:24:19.025580+0000","last_active":"2024-03-06T07:24:20.032035+0000","last_peered":"2024-03-06T07:24:20.032035+0000","last_clean":"2024-03-06T07:24:20.032035+0000","last_became_active":"2024-03-06T07:24:19.025310+0000","last_became_peered":"2024-03-06T07:24:19.025310+0000","last_unstale":"2024-03-06T07:24:20.032035+0000","last_undegraded":"2024-03-06T07:24:20.032035+0000","last_fullsized":"2024-03-06T07:24:20.032035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:40.804289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032108+0000","last_change":"2024-03-06T07:24:19.025140+0000","last_active":"2024-03-06T07:24:20.032108+0000","last_peered":"2024-03-06T07:24:20.032108+0000","last_clean":"2024-03-06T07:24:20.032108+0000","last_became_active":"2024-03-06T07:24:19.024817+0000","last_became_peered":"2024-03-06T07:24:19.024817+0000","last_unstale":"2024-03-06T07:24:20.032108+0000","last_undegraded":"2024-03-06T07:24:20.032108+0000","last_fullsized":"2024-03-06T07:24:20.032108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:21.771804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058963+0000","last_change":"2024-03-06T07:24:19.038114+0000","last_active":"2024-03-06T07:24:20.058963+0000","last_peered":"2024-03-06T07:24:20.058963+0000","last_clean":"2024-03-06T07:24:20.058963+0000","last_became_active":"2024-03-06T07:24:19.037712+0000","last_became_peered":"2024-03-06T07:24:19.037712+0000","last_unstale":"2024-03-06T07:24:20.058963+0000","last_undegraded":"2024-03-06T07:24:20.058963+0000","last_fullsized":"2024-03-06T07:24:20.058963+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:21.490775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032285+0000","last_change":"2024-03-06T07:24:19.023898+0000","last_active":"2024-03-06T07:24:20.032285+0000","last_peered":"2024-03-06T07:24:20.032285+0000","last_clean":"2024-03-06T07:24:20.032285+0000","last_became_active":"2024-03-06T07:24:19.023046+0000","last_became_peered":"2024-03-06T07:24:19.023046+0000","last_unstale":"2024-03-06T07:24:20.032285+0000","last_undegraded":"2024-03-06T07:24:20.032285+0000","last_fullsized":"2024-03-06T07:24:20.032285+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:43.236972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058432+0000","last_change":"2024-03-06T07:24:19.035379+0000","last_active":"2024-03-06T07:24:20.058432+0000","last_peered":"2024-03-06T07:24:20.058432+0000","last_clean":"2024-03-06T07:24:20.058432+0000","last_became_active":"2024-03-06T07:24:19.035031+0000","last_became_peered":"2024-03-06T07:24:19.035031+0000","last_unstale":"2024-03-06T07:24:20.058432+0000","last_undegraded":"2024-03-06T07:24:20.058432+0000","last_fullsized":"2024-03-06T07:24:20.058432+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:24.573580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058218+0000","last_change":"2024-03-06T07:24:19.036472+0000","last_active":"2024-03-06T07:24:20.058218+0000","last_peered":"2024-03-06T07:24:20.058218+0000","last_clean":"2024-03-06T07:24:20.058218+0000","last_became_active":"2024-03-06T07:24:19.036219+0000","last_became_peered":"2024-03-06T07:24:19.036219+0000","last_unstale":"2024-03-06T07:24:20.058218+0000","last_undegraded":"2024-03-06T07:24:20.058218+0000","last_fullsized":"2024-03-06T07:24:20.058218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:19.040346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057208+0000","last_change":"2024-03-06T07:24:19.027077+0000","last_active":"2024-03-06T07:24:20.057208+0000","last_peered":"2024-03-06T07:24:20.057208+0000","last_clean":"2024-03-06T07:24:20.057208+0000","last_became_active":"2024-03-06T07:24:19.026549+0000","last_became_peered":"2024-03-06T07:24:19.026549+0000","last_unstale":"2024-03-06T07:24:20.057208+0000","last_undegraded":"2024-03-06T07:24:20.057208+0000","last_fullsized":"2024-03-06T07:24:20.057208+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:27.221192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058209+0000","last_change":"2024-03-06T07:24:19.035221+0000","last_active":"2024-03-06T07:24:20.058209+0000","last_peered":"2024-03-06T07:24:20.058209+0000","last_clean":"2024-03-06T07:24:20.058209+0000","last_became_active":"2024-03-06T07:24:19.034944+0000","last_became_peered":"2024-03-06T07:24:19.034944+0000","last_unstale":"2024-03-06T07:24:20.058209+0000","last_undegraded":"2024-03-06T07:24:20.058209+0000","last_fullsized":"2024-03-06T07:24:20.058209+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:31.477413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057954+0000","last_change":"2024-03-06T07:24:19.043642+0000","last_active":"2024-03-06T07:24:20.057954+0000","last_peered":"2024-03-06T07:24:20.057954+0000","last_clean":"2024-03-06T07:24:20.057954+0000","last_became_active":"2024-03-06T07:24:19.043368+0000","last_became_peered":"2024-03-06T07:24:19.043368+0000","last_unstale":"2024-03-06T07:24:20.057954+0000","last_undegraded":"2024-03-06T07:24:20.057954+0000","last_fullsized":"2024-03-06T07:24:20.057954+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:30.673686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.030958+0000","last_change":"2024-03-06T07:24:19.027130+0000","last_active":"2024-03-06T07:24:20.030958+0000","last_peered":"2024-03-06T07:24:20.030958+0000","last_clean":"2024-03-06T07:24:20.030958+0000","last_became_active":"2024-03-06T07:24:19.026878+0000","last_became_peered":"2024-03-06T07:24:19.026878+0000","last_unstale":"2024-03-06T07:24:20.030958+0000","last_undegraded":"2024-03-06T07:24:20.030958+0000","last_fullsized":"2024-03-06T07:24:20.030958+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:23.851240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057808+0000","last_change":"2024-03-06T07:24:19.033048+0000","last_active":"2024-03-06T07:24:20.057808+0000","last_peered":"2024-03-06T07:24:20.057808+0000","last_clean":"2024-03-06T07:24:20.057808+0000","last_became_active":"2024-03-06T07:24:19.032763+0000","last_became_peered":"2024-03-06T07:24:19.032763+0000","last_unstale":"2024-03-06T07:24:20.057808+0000","last_undegraded":"2024-03-06T07:24:20.057808+0000","last_fullsized":"2024-03-06T07:24:20.057808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:30.287028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058343+0000","last_change":"2024-03-06T07:24:19.042682+0000","last_active":"2024-03-06T07:24:20.058343+0000","last_peered":"2024-03-06T07:24:20.058343+0000","last_clean":"2024-03-06T07:24:20.058343+0000","last_became_active":"2024-03-06T07:24:19.042396+0000","last_became_peered":"2024-03-06T07:24:19.042396+0000","last_unstale":"2024-03-06T07:24:20.058343+0000","last_undegraded":"2024-03-06T07:24:20.058343+0000","last_fullsized":"2024-03-06T07:24:20.058343+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:21.490775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057795+0000","last_change":"2024-03-06T07:24:19.034470+0000","last_active":"2024-03-06T07:24:20.057795+0000","last_peered":"2024-03-06T07:24:20.057795+0000","last_clean":"2024-03-06T07:24:20.057795+0000","last_became_active":"2024-03-06T07:24:19.033557+0000","last_became_peered":"2024-03-06T07:24:19.033557+0000","last_unstale":"2024-03-06T07:24:20.057795+0000","last_undegraded":"2024-03-06T07:24:20.057795+0000","last_fullsized":"2024-03-06T07:24:20.057795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:30.745625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058815+0000","last_change":"2024-03-06T07:24:19.039038+0000","last_active":"2024-03-06T07:24:20.058815+0000","last_peered":"2024-03-06T07:24:20.058815+0000","last_clean":"2024-03-06T07:24:20.058815+0000","last_became_active":"2024-03-06T07:24:19.038399+0000","last_became_peered":"2024-03-06T07:24:19.038399+0000","last_unstale":"2024-03-06T07:24:20.058815+0000","last_undegraded":"2024-03-06T07:24:20.058815+0000","last_fullsized":"2024-03-06T07:24:20.058815+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:49.991577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032193+0000","last_change":"2024-03-06T07:24:19.022261+0000","last_active":"2024-03-06T07:24:20.032193+0000","last_peered":"2024-03-06T07:24:20.032193+0000","last_clean":"2024-03-06T07:24:20.032193+0000","last_became_active":"2024-03-06T07:24:19.021730+0000","last_became_peered":"2024-03-06T07:24:19.021730+0000","last_unstale":"2024-03-06T07:24:20.032193+0000","last_undegraded":"2024-03-06T07:24:20.032193+0000","last_fullsized":"2024-03-06T07:24:20.032193+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:05.921214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031095+0000","last_change":"2024-03-06T07:24:19.029373+0000","last_active":"2024-03-06T07:24:20.031095+0000","last_peered":"2024-03-06T07:24:20.031095+0000","last_clean":"2024-03-06T07:24:20.031095+0000","last_became_active":"2024-03-06T07:24:19.029127+0000","last_became_peered":"2024-03-06T07:24:19.029127+0000","last_unstale":"2024-03-06T07:24:20.031095+0000","last_undegraded":"2024-03-06T07:24:20.031095+0000","last_fullsized":"2024-03-06T07:24:20.031095+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:27.471868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059217+0000","last_change":"2024-03-06T07:24:19.040759+0000","last_active":"2024-03-06T07:24:20.059217+0000","last_peered":"2024-03-06T07:24:20.059217+0000","last_clean":"2024-03-06T07:24:20.059217+0000","last_became_active":"2024-03-06T07:24:19.040485+0000","last_became_peered":"2024-03-06T07:24:19.040485+0000","last_unstale":"2024-03-06T07:24:20.059217+0000","last_undegraded":"2024-03-06T07:24:20.059217+0000","last_fullsized":"2024-03-06T07:24:20.059217+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:24.090559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058250+0000","last_change":"2024-03-06T07:24:19.025051+0000","last_active":"2024-03-06T07:24:20.058250+0000","last_peered":"2024-03-06T07:24:20.058250+0000","last_clean":"2024-03-06T07:24:20.058250+0000","last_became_active":"2024-03-06T07:24:19.024793+0000","last_became_peered":"2024-03-06T07:24:19.024793+0000","last_unstale":"2024-03-06T07:24:20.058250+0000","last_undegraded":"2024-03-06T07:24:20.058250+0000","last_fullsized":"2024-03-06T07:24:20.058250+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:24.090559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031103+0000","last_change":"2024-03-06T07:24:19.038797+0000","last_active":"2024-03-06T07:24:20.031103+0000","last_peered":"2024-03-06T07:24:20.031103+0000","last_clean":"2024-03-06T07:24:20.031103+0000","last_became_active":"2024-03-06T07:24:19.038542+0000","last_became_peered":"2024-03-06T07:24:19.038542+0000","last_unstale":"2024-03-06T07:24:20.031103+0000","last_undegraded":"2024-03-06T07:24:20.031103+0000","last_fullsized":"2024-03-06T07:24:20.031103+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:45.511066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058640+0000","last_change":"2024-03-06T07:24:19.042728+0000","last_active":"2024-03-06T07:24:20.058640+0000","last_peered":"2024-03-06T07:24:20.058640+0000","last_clean":"2024-03-06T07:24:20.058640+0000","last_became_active":"2024-03-06T07:24:19.042336+0000","last_became_peered":"2024-03-06T07:24:19.042336+0000","last_unstale":"2024-03-06T07:24:20.058640+0000","last_undegraded":"2024-03-06T07:24:20.058640+0000","last_fullsized":"2024-03-06T07:24:20.058640+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:57.706180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057931+0000","last_change":"2024-03-06T07:24:19.040616+0000","last_active":"2024-03-06T07:24:20.057931+0000","last_peered":"2024-03-06T07:24:20.057931+0000","last_clean":"2024-03-06T07:24:20.057931+0000","last_became_active":"2024-03-06T07:24:19.040344+0000","last_became_peered":"2024-03-06T07:24:19.040344+0000","last_unstale":"2024-03-06T07:24:20.057931+0000","last_undegraded":"2024-03-06T07:24:20.057931+0000","last_fullsized":"2024-03-06T07:24:20.057931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:04.603675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.030832+0000","last_change":"2024-03-06T07:24:19.028425+0000","last_active":"2024-03-06T07:24:20.030832+0000","last_peered":"2024-03-06T07:24:20.030832+0000","last_clean":"2024-03-06T07:24:20.030832+0000","last_became_active":"2024-03-06T07:24:19.028188+0000","last_became_peered":"2024-03-06T07:24:19.028188+0000","last_unstale":"2024-03-06T07:24:20.030832+0000","last_undegraded":"2024-03-06T07:24:20.030832+0000","last_fullsized":"2024-03-06T07:24:20.030832+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:54.120663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057268+0000","last_change":"2024-03-06T07:24:19.026862+0000","last_active":"2024-03-06T07:24:20.057268+0000","last_peered":"2024-03-06T07:24:20.057268+0000","last_clean":"2024-03-06T07:24:20.057268+0000","last_became_active":"2024-03-06T07:24:19.026218+0000","last_became_peered":"2024-03-06T07:24:19.026218+0000","last_unstale":"2024-03-06T07:24:20.057268+0000","last_undegraded":"2024-03-06T07:24:20.057268+0000","last_fullsized":"2024-03-06T07:24:20.057268+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:04.603675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058401+0000","last_change":"2024-03-06T07:24:19.035388+0000","last_active":"2024-03-06T07:24:20.058401+0000","last_peered":"2024-03-06T07:24:20.058401+0000","last_clean":"2024-03-06T07:24:20.058401+0000","last_became_active":"2024-03-06T07:24:19.034581+0000","last_became_peered":"2024-03-06T07:24:19.034581+0000","last_unstale":"2024-03-06T07:24:20.058401+0000","last_undegraded":"2024-03-06T07:24:20.058401+0000","last_fullsized":"2024-03-06T07:24:20.058401+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:24.648318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058249+0000","last_change":"2024-03-06T07:24:18.032384+0000","last_active":"2024-03-06T07:24:20.058249+0000","last_peered":"2024-03-06T07:24:20.058249+0000","last_clean":"2024-03-06T07:24:20.058249+0000","last_became_active":"2024-03-06T07:24:18.032039+0000","last_became_peered":"2024-03-06T07:24:18.032039+0000","last_unstale":"2024-03-06T07:24:20.058249+0000","last_undegraded":"2024-03-06T07:24:20.058249+0000","last_fullsized":"2024-03-06T07:24:20.058249+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:45.634728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032155+0000","last_change":"2024-03-06T07:24:19.032261+0000","last_active":"2024-03-06T07:24:20.032155+0000","last_peered":"2024-03-06T07:24:20.032155+0000","last_clean":"2024-03-06T07:24:20.032155+0000","last_became_active":"2024-03-06T07:24:19.031901+0000","last_became_peered":"2024-03-06T07:24:19.031901+0000","last_unstale":"2024-03-06T07:24:20.032155+0000","last_undegraded":"2024-03-06T07:24:20.032155+0000","last_fullsized":"2024-03-06T07:24:20.032155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:19.134434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058445+0000","last_change":"2024-03-06T07:24:18.023224+0000","last_active":"2024-03-06T07:24:20.058445+0000","last_peered":"2024-03-06T07:24:20.058445+0000","last_clean":"2024-03-06T07:24:20.058445+0000","last_became_active":"2024-03-06T07:24:18.022955+0000","last_became_peered":"2024-03-06T07:24:18.022955+0000","last_unstale":"2024-03-06T07:24:20.058445+0000","last_undegraded":"2024-03-06T07:24:20.058445+0000","last_fullsized":"2024-03-06T07:24:20.058445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:15.685665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058093+0000","last_change":"2024-03-06T07:24:19.027823+0000","last_active":"2024-03-06T07:24:20.058093+0000","last_peered":"2024-03-06T07:24:20.058093+0000","last_clean":"2024-03-06T07:24:20.058093+0000","last_became_active":"2024-03-06T07:24:19.027574+0000","last_became_peered":"2024-03-06T07:24:19.027574+0000","last_unstale":"2024-03-06T07:24:20.058093+0000","last_undegraded":"2024-03-06T07:24:20.058093+0000","last_fullsized":"2024-03-06T07:24:20.058093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:21.490775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058378+0000","last_change":"2024-03-06T07:24:18.030819+0000","last_active":"2024-03-06T07:24:20.058378+0000","last_peered":"2024-03-06T07:24:20.058378+0000","last_clean":"2024-03-06T07:24:20.058378+0000","last_became_active":"2024-03-06T07:24:18.030562+0000","last_became_peered":"2024-03-06T07:24:18.030562+0000","last_unstale":"2024-03-06T07:24:20.058378+0000","last_undegraded":"2024-03-06T07:24:20.058378+0000","last_fullsized":"2024-03-06T07:24:20.058378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:23.569796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058283+0000","last_change":"2024-03-06T07:24:19.039664+0000","last_active":"2024-03-06T07:24:20.058283+0000","last_peered":"2024-03-06T07:24:20.058283+0000","last_clean":"2024-03-06T07:24:20.058283+0000","last_became_active":"2024-03-06T07:24:19.039385+0000","last_became_peered":"2024-03-06T07:24:19.039385+0000","last_unstale":"2024-03-06T07:24:20.058283+0000","last_undegraded":"2024-03-06T07:24:20.058283+0000","last_fullsized":"2024-03-06T07:24:20.058283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:29.406311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058993+0000","last_change":"2024-03-06T07:24:18.031578+0000","last_active":"2024-03-06T07:24:20.058993+0000","last_peered":"2024-03-06T07:24:20.058993+0000","last_clean":"2024-03-06T07:24:20.058993+0000","last_became_active":"2024-03-06T07:24:18.031277+0000","last_became_peered":"2024-03-06T07:24:18.031277+0000","last_unstale":"2024-03-06T07:24:20.058993+0000","last_undegraded":"2024-03-06T07:24:20.058993+0000","last_fullsized":"2024-03-06T07:24:20.058993+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:18.258769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057931+0000","last_change":"2024-03-06T07:24:19.029566+0000","last_active":"2024-03-06T07:24:20.057931+0000","last_peered":"2024-03-06T07:24:20.057931+0000","last_clean":"2024-03-06T07:24:20.057931+0000","last_became_active":"2024-03-06T07:24:19.029290+0000","last_became_peered":"2024-03-06T07:24:19.029290+0000","last_unstale":"2024-03-06T07:24:20.057931+0000","last_undegraded":"2024-03-06T07:24:20.057931+0000","last_fullsized":"2024-03-06T07:24:20.057931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:43.931737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057997+0000","last_change":"2024-03-06T07:24:18.032504+0000","last_active":"2024-03-06T07:24:20.057997+0000","last_peered":"2024-03-06T07:24:20.057997+0000","last_clean":"2024-03-06T07:24:20.057997+0000","last_became_active":"2024-03-06T07:24:18.032189+0000","last_became_peered":"2024-03-06T07:24:18.032189+0000","last_unstale":"2024-03-06T07:24:20.057997+0000","last_undegraded":"2024-03-06T07:24:20.057997+0000","last_fullsized":"2024-03-06T07:24:20.057997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:48.456986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058552+0000","last_change":"2024-03-06T07:24:19.038983+0000","last_active":"2024-03-06T07:24:20.058552+0000","last_peered":"2024-03-06T07:24:20.058552+0000","last_clean":"2024-03-06T07:24:20.058552+0000","last_became_active":"2024-03-06T07:24:19.038729+0000","last_became_peered":"2024-03-06T07:24:19.038729+0000","last_unstale":"2024-03-06T07:24:20.058552+0000","last_undegraded":"2024-03-06T07:24:20.058552+0000","last_fullsized":"2024-03-06T07:24:20.058552+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:46.343704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057897+0000","last_change":"2024-03-06T07:24:18.025876+0000","last_active":"2024-03-06T07:24:20.057897+0000","last_peered":"2024-03-06T07:24:20.057897+0000","last_clean":"2024-03-06T07:24:20.057897+0000","last_became_active":"2024-03-06T07:24:18.024854+0000","last_became_peered":"2024-03-06T07:24:18.024854+0000","last_unstale":"2024-03-06T07:24:20.057897+0000","last_undegraded":"2024-03-06T07:24:20.057897+0000","last_fullsized":"2024-03-06T07:24:20.057897+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:18.036562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032192+0000","last_change":"2024-03-06T07:24:19.029063+0000","last_active":"2024-03-06T07:24:20.032192+0000","last_peered":"2024-03-06T07:24:20.032192+0000","last_clean":"2024-03-06T07:24:20.032192+0000","last_became_active":"2024-03-06T07:24:19.028783+0000","last_became_peered":"2024-03-06T07:24:19.028783+0000","last_unstale":"2024-03-06T07:24:20.032192+0000","last_undegraded":"2024-03-06T07:24:20.032192+0000","last_fullsized":"2024-03-06T07:24:20.032192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:00.387592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058157+0000","last_change":"2024-03-06T07:24:18.031884+0000","last_active":"2024-03-06T07:24:20.058157+0000","last_peered":"2024-03-06T07:24:20.058157+0000","last_clean":"2024-03-06T07:24:20.058157+0000","last_became_active":"2024-03-06T07:24:18.030235+0000","last_became_peered":"2024-03-06T07:24:18.030235+0000","last_unstale":"2024-03-06T07:24:20.058157+0000","last_undegraded":"2024-03-06T07:24:20.058157+0000","last_fullsized":"2024-03-06T07:24:20.058157+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:18.036562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057542+0000","last_change":"2024-03-06T07:24:19.029878+0000","last_active":"2024-03-06T07:24:20.057542+0000","last_peered":"2024-03-06T07:24:20.057542+0000","last_clean":"2024-03-06T07:24:20.057542+0000","last_became_active":"2024-03-06T07:24:19.029627+0000","last_became_peered":"2024-03-06T07:24:19.029627+0000","last_unstale":"2024-03-06T07:24:20.057542+0000","last_undegraded":"2024-03-06T07:24:20.057542+0000","last_fullsized":"2024-03-06T07:24:20.057542+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:24.179032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057512+0000","last_change":"2024-03-06T07:24:18.029703+0000","last_active":"2024-03-06T07:24:20.057512+0000","last_peered":"2024-03-06T07:24:20.057512+0000","last_clean":"2024-03-06T07:24:20.057512+0000","last_became_active":"2024-03-06T07:24:18.029157+0000","last_became_peered":"2024-03-06T07:24:18.029157+0000","last_unstale":"2024-03-06T07:24:20.057512+0000","last_undegraded":"2024-03-06T07:24:20.057512+0000","last_fullsized":"2024-03-06T07:24:20.057512+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:07.820313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058214+0000","last_change":"2024-03-06T07:24:19.037987+0000","last_active":"2024-03-06T07:24:20.058214+0000","last_peered":"2024-03-06T07:24:20.058214+0000","last_clean":"2024-03-06T07:24:20.058214+0000","last_became_active":"2024-03-06T07:24:19.036822+0000","last_became_peered":"2024-03-06T07:24:19.036822+0000","last_unstale":"2024-03-06T07:24:20.058214+0000","last_undegraded":"2024-03-06T07:24:20.058214+0000","last_fullsized":"2024-03-06T07:24:20.058214+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:54.764511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058073+0000","last_change":"2024-03-06T07:24:18.027726+0000","last_active":"2024-03-06T07:24:20.058073+0000","last_peered":"2024-03-06T07:24:20.058073+0000","last_clean":"2024-03-06T07:24:20.058073+0000","last_became_active":"2024-03-06T07:24:18.027418+0000","last_became_peered":"2024-03-06T07:24:18.027418+0000","last_unstale":"2024-03-06T07:24:20.058073+0000","last_undegraded":"2024-03-06T07:24:20.058073+0000","last_fullsized":"2024-03-06T07:24:20.058073+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:23.569796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057970+0000","last_change":"2024-03-06T07:24:19.042974+0000","last_active":"2024-03-06T07:24:20.057970+0000","last_peered":"2024-03-06T07:24:20.057970+0000","last_clean":"2024-03-06T07:24:20.057970+0000","last_became_active":"2024-03-06T07:24:19.042676+0000","last_became_peered":"2024-03-06T07:24:19.042676+0000","last_unstale":"2024-03-06T07:24:20.057970+0000","last_undegraded":"2024-03-06T07:24:20.057970+0000","last_fullsized":"2024-03-06T07:24:20.057970+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:16.154081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058533+0000","last_change":"2024-03-06T07:24:18.029944+0000","last_active":"2024-03-06T07:24:20.058533+0000","last_peered":"2024-03-06T07:24:20.058533+0000","last_clean":"2024-03-06T07:24:20.058533+0000","last_became_active":"2024-03-06T07:24:18.028944+0000","last_became_peered":"2024-03-06T07:24:18.028944+0000","last_unstale":"2024-03-06T07:24:20.058533+0000","last_undegraded":"2024-03-06T07:24:20.058533+0000","last_fullsized":"2024-03-06T07:24:20.058533+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.217408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031872+0000","last_change":"2024-03-06T07:24:19.038088+0000","last_active":"2024-03-06T07:24:20.031872+0000","last_peered":"2024-03-06T07:24:20.031872+0000","last_clean":"2024-03-06T07:24:20.031872+0000","last_became_active":"2024-03-06T07:24:19.037831+0000","last_became_peered":"2024-03-06T07:24:19.037831+0000","last_unstale":"2024-03-06T07:24:20.031872+0000","last_undegraded":"2024-03-06T07:24:20.031872+0000","last_fullsized":"2024-03-06T07:24:20.031872+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:18.564746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059204+0000","last_change":"2024-03-06T07:24:18.031108+0000","last_active":"2024-03-06T07:24:20.059204+0000","last_peered":"2024-03-06T07:24:20.059204+0000","last_clean":"2024-03-06T07:24:20.059204+0000","last_became_active":"2024-03-06T07:24:18.030684+0000","last_became_peered":"2024-03-06T07:24:18.030684+0000","last_unstale":"2024-03-06T07:24:20.059204+0000","last_undegraded":"2024-03-06T07:24:20.059204+0000","last_fullsized":"2024-03-06T07:24:20.059204+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:18.036562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059138+0000","last_change":"2024-03-06T07:24:19.042374+0000","last_active":"2024-03-06T07:24:20.059138+0000","last_peered":"2024-03-06T07:24:20.059138+0000","last_clean":"2024-03-06T07:24:20.059138+0000","last_became_active":"2024-03-06T07:24:19.041946+0000","last_became_peered":"2024-03-06T07:24:19.041946+0000","last_unstale":"2024-03-06T07:24:20.059138+0000","last_undegraded":"2024-03-06T07:24:20.059138+0000","last_fullsized":"2024-03-06T07:24:20.059138+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:12.788879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057894+0000","last_change":"2024-03-06T07:24:18.033183+0000","last_active":"2024-03-06T07:24:20.057894+0000","last_peered":"2024-03-06T07:24:20.057894+0000","last_clean":"2024-03-06T07:24:20.057894+0000","last_became_active":"2024-03-06T07:24:18.032923+0000","last_became_peered":"2024-03-06T07:24:18.032923+0000","last_unstale":"2024-03-06T07:24:20.057894+0000","last_undegraded":"2024-03-06T07:24:20.057894+0000","last_fullsized":"2024-03-06T07:24:20.057894+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:10.352057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057742+0000","last_change":"2024-03-06T07:24:19.039970+0000","last_active":"2024-03-06T07:24:20.057742+0000","last_peered":"2024-03-06T07:24:20.057742+0000","last_clean":"2024-03-06T07:24:20.057742+0000","last_became_active":"2024-03-06T07:24:19.039647+0000","last_became_peered":"2024-03-06T07:24:19.039647+0000","last_unstale":"2024-03-06T07:24:20.057742+0000","last_undegraded":"2024-03-06T07:24:20.057742+0000","last_fullsized":"2024-03-06T07:24:20.057742+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:27.221192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058637+0000","last_change":"2024-03-06T07:24:18.030989+0000","last_active":"2024-03-06T07:24:20.058637+0000","last_peered":"2024-03-06T07:24:20.058637+0000","last_clean":"2024-03-06T07:24:20.058637+0000","last_became_active":"2024-03-06T07:24:18.029227+0000","last_became_peered":"2024-03-06T07:24:18.029227+0000","last_unstale":"2024-03-06T07:24:20.058637+0000","last_undegraded":"2024-03-06T07:24:20.058637+0000","last_fullsized":"2024-03-06T07:24:20.058637+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:04.439937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058493+0000","last_change":"2024-03-06T07:24:19.034704+0000","last_active":"2024-03-06T07:24:20.058493+0000","last_peered":"2024-03-06T07:24:20.058493+0000","last_clean":"2024-03-06T07:24:20.058493+0000","last_became_active":"2024-03-06T07:24:19.034421+0000","last_became_peered":"2024-03-06T07:24:19.034421+0000","last_unstale":"2024-03-06T07:24:20.058493+0000","last_undegraded":"2024-03-06T07:24:20.058493+0000","last_fullsized":"2024-03-06T07:24:20.058493+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:24.090559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032048+0000","last_change":"2024-03-06T07:24:18.027111+0000","last_active":"2024-03-06T07:24:20.032048+0000","last_peered":"2024-03-06T07:24:20.032048+0000","last_clean":"2024-03-06T07:24:20.032048+0000","last_became_active":"2024-03-06T07:24:18.026809+0000","last_became_peered":"2024-03-06T07:24:18.026809+0000","last_unstale":"2024-03-06T07:24:20.032048+0000","last_undegraded":"2024-03-06T07:24:20.032048+0000","last_fullsized":"2024-03-06T07:24:20.032048+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:43.188495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057529+0000","last_change":"2024-03-06T07:24:19.038333+0000","last_active":"2024-03-06T07:24:20.057529+0000","last_peered":"2024-03-06T07:24:20.057529+0000","last_clean":"2024-03-06T07:24:20.057529+0000","last_became_active":"2024-03-06T07:24:19.038070+0000","last_became_peered":"2024-03-06T07:24:19.038070+0000","last_unstale":"2024-03-06T07:24:20.057529+0000","last_undegraded":"2024-03-06T07:24:20.057529+0000","last_fullsized":"2024-03-06T07:24:20.057529+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:46.648542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058246+0000","last_change":"2024-03-06T07:24:18.028849+0000","last_active":"2024-03-06T07:24:20.058246+0000","last_peered":"2024-03-06T07:24:20.058246+0000","last_clean":"2024-03-06T07:24:20.058246+0000","last_became_active":"2024-03-06T07:24:18.028513+0000","last_became_peered":"2024-03-06T07:24:18.028513+0000","last_unstale":"2024-03-06T07:24:20.058246+0000","last_undegraded":"2024-03-06T07:24:20.058246+0000","last_fullsized":"2024-03-06T07:24:20.058246+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:23.569796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058887+0000","last_change":"2024-03-06T07:24:19.039089+0000","last_active":"2024-03-06T07:24:20.058887+0000","last_peered":"2024-03-06T07:24:20.058887+0000","last_clean":"2024-03-06T07:24:20.058887+0000","last_became_active":"2024-03-06T07:24:19.038794+0000","last_became_peered":"2024-03-06T07:24:19.038794+0000","last_unstale":"2024-03-06T07:24:20.058887+0000","last_undegraded":"2024-03-06T07:24:20.058887+0000","last_fullsized":"2024-03-06T07:24:20.058887+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:24.648318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058317+0000","last_change":"2024-03-06T07:24:18.030751+0000","last_active":"2024-03-06T07:24:20.058317+0000","last_peered":"2024-03-06T07:24:20.058317+0000","last_clean":"2024-03-06T07:24:20.058317+0000","last_became_active":"2024-03-06T07:24:18.030471+0000","last_became_peered":"2024-03-06T07:24:18.030471+0000","last_unstale":"2024-03-06T07:24:20.058317+0000","last_undegraded":"2024-03-06T07:24:20.058317+0000","last_fullsized":"2024-03-06T07:24:20.058317+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:53.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058105+0000","last_change":"2024-03-06T07:24:19.040200+0000","last_active":"2024-03-06T07:24:20.058105+0000","last_peered":"2024-03-06T07:24:20.058105+0000","last_clean":"2024-03-06T07:24:20.058105+0000","last_became_active":"2024-03-06T07:24:19.039891+0000","last_became_peered":"2024-03-06T07:24:19.039891+0000","last_unstale":"2024-03-06T07:24:20.058105+0000","last_undegraded":"2024-03-06T07:24:20.058105+0000","last_fullsized":"2024-03-06T07:24:20.058105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:36.057457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058214+0000","last_change":"2024-03-06T07:24:18.028752+0000","last_active":"2024-03-06T07:24:20.058214+0000","last_peered":"2024-03-06T07:24:20.058214+0000","last_clean":"2024-03-06T07:24:20.058214+0000","last_became_active":"2024-03-06T07:24:18.028477+0000","last_became_peered":"2024-03-06T07:24:18.028477+0000","last_unstale":"2024-03-06T07:24:20.058214+0000","last_undegraded":"2024-03-06T07:24:20.058214+0000","last_fullsized":"2024-03-06T07:24:20.058214+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:43.231950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058555+0000","last_change":"2024-03-06T07:24:19.041500+0000","last_active":"2024-03-06T07:24:20.058555+0000","last_peered":"2024-03-06T07:24:20.058555+0000","last_clean":"2024-03-06T07:24:20.058555+0000","last_became_active":"2024-03-06T07:24:19.041204+0000","last_became_peered":"2024-03-06T07:24:19.041204+0000","last_unstale":"2024-03-06T07:24:20.058555+0000","last_undegraded":"2024-03-06T07:24:20.058555+0000","last_fullsized":"2024-03-06T07:24:20.058555+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:30.673686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058392+0000","last_change":"2024-03-06T07:24:18.028025+0000","last_active":"2024-03-06T07:24:20.058392+0000","last_peered":"2024-03-06T07:24:20.058392+0000","last_clean":"2024-03-06T07:24:20.058392+0000","last_became_active":"2024-03-06T07:24:18.027758+0000","last_became_peered":"2024-03-06T07:24:18.027758+0000","last_unstale":"2024-03-06T07:24:20.058392+0000","last_undegraded":"2024-03-06T07:24:20.058392+0000","last_fullsized":"2024-03-06T07:24:20.058392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.102294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057938+0000","last_change":"2024-03-06T07:24:19.035342+0000","last_active":"2024-03-06T07:24:20.057938+0000","last_peered":"2024-03-06T07:24:20.057938+0000","last_clean":"2024-03-06T07:24:20.057938+0000","last_became_active":"2024-03-06T07:24:19.035037+0000","last_became_peered":"2024-03-06T07:24:19.035037+0000","last_unstale":"2024-03-06T07:24:20.057938+0000","last_undegraded":"2024-03-06T07:24:20.057938+0000","last_fullsized":"2024-03-06T07:24:20.057938+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:56.856056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058629+0000","last_change":"2024-03-06T07:24:18.033385+0000","last_active":"2024-03-06T07:24:20.058629+0000","last_peered":"2024-03-06T07:24:20.058629+0000","last_clean":"2024-03-06T07:24:20.058629+0000","last_became_active":"2024-03-06T07:24:18.033080+0000","last_became_peered":"2024-03-06T07:24:18.033080+0000","last_unstale":"2024-03-06T07:24:20.058629+0000","last_undegraded":"2024-03-06T07:24:20.058629+0000","last_fullsized":"2024-03-06T07:24:20.058629+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:43.113655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059217+0000","last_change":"2024-03-06T07:24:19.042625+0000","last_active":"2024-03-06T07:24:20.059217+0000","last_peered":"2024-03-06T07:24:20.059217+0000","last_clean":"2024-03-06T07:24:20.059217+0000","last_became_active":"2024-03-06T07:24:19.042060+0000","last_became_peered":"2024-03-06T07:24:19.042060+0000","last_unstale":"2024-03-06T07:24:20.059217+0000","last_undegraded":"2024-03-06T07:24:20.059217+0000","last_fullsized":"2024-03-06T07:24:20.059217+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:28.413135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058094+0000","last_change":"2024-03-06T07:24:18.021164+0000","last_active":"2024-03-06T07:24:20.058094+0000","last_peered":"2024-03-06T07:24:20.058094+0000","last_clean":"2024-03-06T07:24:20.058094+0000","last_became_active":"2024-03-06T07:24:18.020859+0000","last_became_peered":"2024-03-06T07:24:18.020859+0000","last_unstale":"2024-03-06T07:24:20.058094+0000","last_undegraded":"2024-03-06T07:24:20.058094+0000","last_fullsized":"2024-03-06T07:24:20.058094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:30.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057898+0000","last_change":"2024-03-06T07:24:19.036202+0000","last_active":"2024-03-06T07:24:20.057898+0000","last_peered":"2024-03-06T07:24:20.057898+0000","last_clean":"2024-03-06T07:24:20.057898+0000","last_became_active":"2024-03-06T07:24:19.035941+0000","last_became_peered":"2024-03-06T07:24:19.035941+0000","last_unstale":"2024-03-06T07:24:20.057898+0000","last_undegraded":"2024-03-06T07:24:20.057898+0000","last_fullsized":"2024-03-06T07:24:20.057898+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:04.603675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058703+0000","last_change":"2024-03-06T07:24:18.030789+0000","last_active":"2024-03-06T07:24:20.058703+0000","last_peered":"2024-03-06T07:24:20.058703+0000","last_clean":"2024-03-06T07:24:20.058703+0000","last_became_active":"2024-03-06T07:24:18.030535+0000","last_became_peered":"2024-03-06T07:24:18.030535+0000","last_unstale":"2024-03-06T07:24:20.058703+0000","last_undegraded":"2024-03-06T07:24:20.058703+0000","last_fullsized":"2024-03-06T07:24:20.058703+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:03.599891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057562+0000","last_change":"2024-03-06T07:24:19.025616+0000","last_active":"2024-03-06T07:24:20.057562+0000","last_peered":"2024-03-06T07:24:20.057562+0000","last_clean":"2024-03-06T07:24:20.057562+0000","last_became_active":"2024-03-06T07:24:19.025356+0000","last_became_peered":"2024-03-06T07:24:19.025356+0000","last_unstale":"2024-03-06T07:24:20.057562+0000","last_undegraded":"2024-03-06T07:24:20.057562+0000","last_fullsized":"2024-03-06T07:24:20.057562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:16.154081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031727+0000","last_change":"2024-03-06T07:24:18.030819+0000","last_active":"2024-03-06T07:24:20.031727+0000","last_peered":"2024-03-06T07:24:20.031727+0000","last_clean":"2024-03-06T07:24:20.031727+0000","last_became_active":"2024-03-06T07:24:18.030562+0000","last_became_peered":"2024-03-06T07:24:18.030562+0000","last_unstale":"2024-03-06T07:24:20.031727+0000","last_undegraded":"2024-03-06T07:24:20.031727+0000","last_fullsized":"2024-03-06T07:24:20.031727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:20.278916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057297+0000","last_change":"2024-03-06T07:24:19.034596+0000","last_active":"2024-03-06T07:24:20.057297+0000","last_peered":"2024-03-06T07:24:20.057297+0000","last_clean":"2024-03-06T07:24:20.057297+0000","last_became_active":"2024-03-06T07:24:19.034324+0000","last_became_peered":"2024-03-06T07:24:19.034324+0000","last_unstale":"2024-03-06T07:24:20.057297+0000","last_undegraded":"2024-03-06T07:24:20.057297+0000","last_fullsized":"2024-03-06T07:24:20.057297+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:30.673686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058365+0000","last_change":"2024-03-06T07:24:18.032864+0000","last_active":"2024-03-06T07:24:20.058365+0000","last_peered":"2024-03-06T07:24:20.058365+0000","last_clean":"2024-03-06T07:24:20.058365+0000","last_became_active":"2024-03-06T07:24:18.032601+0000","last_became_peered":"2024-03-06T07:24:18.032601+0000","last_unstale":"2024-03-06T07:24:20.058365+0000","last_undegraded":"2024-03-06T07:24:20.058365+0000","last_fullsized":"2024-03-06T07:24:20.058365+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.102294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059333+0000","last_change":"2024-03-06T07:24:19.042578+0000","last_active":"2024-03-06T07:24:20.059333+0000","last_peered":"2024-03-06T07:24:20.059333+0000","last_clean":"2024-03-06T07:24:20.059333+0000","last_became_active":"2024-03-06T07:24:19.042327+0000","last_became_peered":"2024-03-06T07:24:19.042327+0000","last_unstale":"2024-03-06T07:24:20.059333+0000","last_undegraded":"2024-03-06T07:24:20.059333+0000","last_fullsized":"2024-03-06T07:24:20.059333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:29.406311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032191+0000","last_change":"2024-03-06T07:24:18.034298+0000","last_active":"2024-03-06T07:24:20.032191+0000","last_peered":"2024-03-06T07:24:20.032191+0000","last_clean":"2024-03-06T07:24:20.032191+0000","last_became_active":"2024-03-06T07:24:18.034028+0000","last_became_peered":"2024-03-06T07:24:18.034028+0000","last_unstale":"2024-03-06T07:24:20.032191+0000","last_undegraded":"2024-03-06T07:24:20.032191+0000","last_fullsized":"2024-03-06T07:24:20.032191+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:04.917430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058665+0000","last_change":"2024-03-06T07:24:19.039261+0000","last_active":"2024-03-06T07:24:20.058665+0000","last_peered":"2024-03-06T07:24:20.058665+0000","last_clean":"2024-03-06T07:24:20.058665+0000","last_became_active":"2024-03-06T07:24:19.038976+0000","last_became_peered":"2024-03-06T07:24:19.038976+0000","last_unstale":"2024-03-06T07:24:20.058665+0000","last_undegraded":"2024-03-06T07:24:20.058665+0000","last_fullsized":"2024-03-06T07:24:20.058665+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:16.154081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058486+0000","last_change":"2024-03-06T07:24:18.025191+0000","last_active":"2024-03-06T07:24:20.058486+0000","last_peered":"2024-03-06T07:24:20.058486+0000","last_clean":"2024-03-06T07:24:20.058486+0000","last_became_active":"2024-03-06T07:24:18.024907+0000","last_became_peered":"2024-03-06T07:24:18.024907+0000","last_unstale":"2024-03-06T07:24:20.058486+0000","last_undegraded":"2024-03-06T07:24:20.058486+0000","last_fullsized":"2024-03-06T07:24:20.058486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:53.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031922+0000","last_change":"2024-03-06T07:24:19.037469+0000","last_active":"2024-03-06T07:24:20.031922+0000","last_peered":"2024-03-06T07:24:20.031922+0000","last_clean":"2024-03-06T07:24:20.031922+0000","last_became_active":"2024-03-06T07:24:19.037213+0000","last_became_peered":"2024-03-06T07:24:19.037213+0000","last_unstale":"2024-03-06T07:24:20.031922+0000","last_undegraded":"2024-03-06T07:24:20.031922+0000","last_fullsized":"2024-03-06T07:24:20.031922+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:43.020758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032329+0000","last_change":"2024-03-06T07:24:18.030559+0000","last_active":"2024-03-06T07:24:20.032329+0000","last_peered":"2024-03-06T07:24:20.032329+0000","last_clean":"2024-03-06T07:24:20.032329+0000","last_became_active":"2024-03-06T07:24:18.030152+0000","last_became_peered":"2024-03-06T07:24:18.030152+0000","last_unstale":"2024-03-06T07:24:20.032329+0000","last_undegraded":"2024-03-06T07:24:20.032329+0000","last_fullsized":"2024-03-06T07:24:20.032329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:51.760044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058393+0000","last_change":"2024-03-06T07:24:19.039223+0000","last_active":"2024-03-06T07:24:20.058393+0000","last_peered":"2024-03-06T07:24:20.058393+0000","last_clean":"2024-03-06T07:24:20.058393+0000","last_became_active":"2024-03-06T07:24:19.038852+0000","last_became_peered":"2024-03-06T07:24:19.038852+0000","last_unstale":"2024-03-06T07:24:20.058393+0000","last_undegraded":"2024-03-06T07:24:20.058393+0000","last_fullsized":"2024-03-06T07:24:20.058393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:49.991577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059067+0000","last_change":"2024-03-06T07:24:18.030680+0000","last_active":"2024-03-06T07:24:20.059067+0000","last_peered":"2024-03-06T07:24:20.059067+0000","last_clean":"2024-03-06T07:24:20.059067+0000","last_became_active":"2024-03-06T07:24:18.030413+0000","last_became_peered":"2024-03-06T07:24:18.030413+0000","last_unstale":"2024-03-06T07:24:20.059067+0000","last_undegraded":"2024-03-06T07:24:20.059067+0000","last_fullsized":"2024-03-06T07:24:20.059067+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:53.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058079+0000","last_change":"2024-03-06T07:24:19.027980+0000","last_active":"2024-03-06T07:24:20.058079+0000","last_peered":"2024-03-06T07:24:20.058079+0000","last_clean":"2024-03-06T07:24:20.058079+0000","last_became_active":"2024-03-06T07:24:19.027703+0000","last_became_peered":"2024-03-06T07:24:19.027703+0000","last_unstale":"2024-03-06T07:24:20.058079+0000","last_undegraded":"2024-03-06T07:24:20.058079+0000","last_fullsized":"2024-03-06T07:24:20.058079+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:50.936956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031537+0000","last_change":"2024-03-06T07:24:18.033531+0000","last_active":"2024-03-06T07:24:20.031537+0000","last_peered":"2024-03-06T07:24:20.031537+0000","last_clean":"2024-03-06T07:24:20.031537+0000","last_became_active":"2024-03-06T07:24:18.033258+0000","last_became_peered":"2024-03-06T07:24:18.033258+0000","last_unstale":"2024-03-06T07:24:20.031537+0000","last_undegraded":"2024-03-06T07:24:20.031537+0000","last_fullsized":"2024-03-06T07:24:20.031537+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:38.211170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058677+0000","last_change":"2024-03-06T07:24:19.040244+0000","last_active":"2024-03-06T07:24:20.058677+0000","last_peered":"2024-03-06T07:24:20.058677+0000","last_clean":"2024-03-06T07:24:20.058677+0000","last_became_active":"2024-03-06T07:24:19.039924+0000","last_became_peered":"2024-03-06T07:24:19.039924+0000","last_unstale":"2024-03-06T07:24:20.058677+0000","last_undegraded":"2024-03-06T07:24:20.058677+0000","last_fullsized":"2024-03-06T07:24:20.058677+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:29.406311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058537+0000","last_change":"2024-03-06T07:24:18.033917+0000","last_active":"2024-03-06T07:24:20.058537+0000","last_peered":"2024-03-06T07:24:20.058537+0000","last_clean":"2024-03-06T07:24:20.058537+0000","last_became_active":"2024-03-06T07:24:18.033614+0000","last_became_peered":"2024-03-06T07:24:18.033614+0000","last_unstale":"2024-03-06T07:24:20.058537+0000","last_undegraded":"2024-03-06T07:24:20.058537+0000","last_fullsized":"2024-03-06T07:24:20.058537+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:24.939681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031618+0000","last_change":"2024-03-06T07:24:19.024493+0000","last_active":"2024-03-06T07:24:20.031618+0000","last_peered":"2024-03-06T07:24:20.031618+0000","last_clean":"2024-03-06T07:24:20.031618+0000","last_became_active":"2024-03-06T07:24:19.024250+0000","last_became_peered":"2024-03-06T07:24:19.024250+0000","last_unstale":"2024-03-06T07:24:20.031618+0000","last_undegraded":"2024-03-06T07:24:20.031618+0000","last_fullsized":"2024-03-06T07:24:20.031618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.507789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031687+0000","last_change":"2024-03-06T07:24:18.028930+0000","last_active":"2024-03-06T07:24:20.031687+0000","last_peered":"2024-03-06T07:24:20.031687+0000","last_clean":"2024-03-06T07:24:20.031687+0000","last_became_active":"2024-03-06T07:24:18.028531+0000","last_became_peered":"2024-03-06T07:24:18.028531+0000","last_unstale":"2024-03-06T07:24:20.031687+0000","last_undegraded":"2024-03-06T07:24:20.031687+0000","last_fullsized":"2024-03-06T07:24:20.031687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:59.383808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057673+0000","last_change":"2024-03-06T07:24:19.027235+0000","last_active":"2024-03-06T07:24:20.057673+0000","last_peered":"2024-03-06T07:24:20.057673+0000","last_clean":"2024-03-06T07:24:20.057673+0000","last_became_active":"2024-03-06T07:24:19.025852+0000","last_became_peered":"2024-03-06T07:24:19.025852+0000","last_unstale":"2024-03-06T07:24:20.057673+0000","last_undegraded":"2024-03-06T07:24:20.057673+0000","last_fullsized":"2024-03-06T07:24:20.057673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:42.676172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057638+0000","last_change":"2024-03-06T07:24:18.030948+0000","last_active":"2024-03-06T07:24:20.057638+0000","last_peered":"2024-03-06T07:24:20.057638+0000","last_clean":"2024-03-06T07:24:20.057638+0000","last_became_active":"2024-03-06T07:24:18.030667+0000","last_became_peered":"2024-03-06T07:24:18.030667+0000","last_unstale":"2024-03-06T07:24:20.057638+0000","last_undegraded":"2024-03-06T07:24:20.057638+0000","last_fullsized":"2024-03-06T07:24:20.057638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:43.836803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057551+0000","last_change":"2024-03-06T07:24:19.035719+0000","last_active":"2024-03-06T07:24:20.057551+0000","last_peered":"2024-03-06T07:24:20.057551+0000","last_clean":"2024-03-06T07:24:20.057551+0000","last_became_active":"2024-03-06T07:24:19.028978+0000","last_became_peered":"2024-03-06T07:24:19.028978+0000","last_unstale":"2024-03-06T07:24:20.057551+0000","last_undegraded":"2024-03-06T07:24:20.057551+0000","last_fullsized":"2024-03-06T07:24:20.057551+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:18.187937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032029+0000","last_change":"2024-03-06T07:24:18.030115+0000","last_active":"2024-03-06T07:24:20.032029+0000","last_peered":"2024-03-06T07:24:20.032029+0000","last_clean":"2024-03-06T07:24:20.032029+0000","last_became_active":"2024-03-06T07:24:18.029832+0000","last_became_peered":"2024-03-06T07:24:18.029832+0000","last_unstale":"2024-03-06T07:24:20.032029+0000","last_undegraded":"2024-03-06T07:24:20.032029+0000","last_fullsized":"2024-03-06T07:24:20.032029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:03.293315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":237,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058382+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:24:20.058382+0000","last_peered":"2024-03-06T07:24:20.058382+0000","last_clean":"2024-03-06T07:24:20.058382+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:24:20.058382+0000","last_undegraded":"2024-03-06T07:24:20.058382+0000","last_fullsized":"2024-03-06T07:24:20.058382+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1286144,"data_stored":874150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":671744,"data_stored":277322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":277974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":285234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":277378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":277378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":277412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":876184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:21.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:22.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.128+0000 7fb8734a4640 1 Processor -- start 2024-03-06T07:24:22.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.128+0000 7fb8734a4640 1 -- start start 2024-03-06T07:24:22.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb8734a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb86c15bed0 0x7fb86c0726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb8734a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c06d3b0 con 0x7fb86c0b50a0 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb8734a4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c06d690 con 0x7fb86c06e170 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb8734a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c0ccaa0 con 0x7fb86c15bed0 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb870a3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb86c15bed0 0x7fb86c0726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb870a3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb86c15bed0 0x7fb86c0726a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47014/0 (socket says 172.21.15.122:47014) 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb870a3f640 1 -- 172.21.15.122:0/4232974769 learned_addr learned my addr 172.21.15.122:0/4232974769 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb871240640 1 --1- 172.21.15.122:0/4232974769 >> v1:172.21.15.122:6789/0 conn(0x7fb86c0b50a0 0x7fb86c0b5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33424/0 (socket says 172.21.15.122:33424) 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.129+0000 7fb870a3f640 1 -- 172.21.15.122:0/4232974769 >> v1:172.21.15.169:6789/0 conn(0x7fb86c06e170 legacy=0x7fb86c0ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb870a3f640 1 -- 172.21.15.122:0/4232974769 >> v1:172.21.15.122:6789/0 conn(0x7fb86c0b50a0 legacy=0x7fb86c0b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb870a3f640 1 -- 172.21.15.122:0/4232974769 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c06d970 con 0x7fb86c15bed0 2024-03-06T07:24:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb863fff640 1 -- 172.21.15.122:0/4232974769 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027234968 0 0) 0x7fb86c06d3b0 con 0x7fb86c0b50a0 2024-03-06T07:24:22.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb870a3f640 1 --2- 172.21.15.122:0/4232974769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c15bed0 0x7fb86c0726a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009870 tx=0x7fb85c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e884f85f418c291a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb863fff640 1 -- 172.21.15.122:0/4232974769 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c02f840 con 0x7fb86c15bed0 2024-03-06T07:24:22.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb863fff640 1 -- 172.21.15.122:0/4232974769 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c02f9e0 con 0x7fb86c15bed0 2024-03-06T07:24:22.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.130+0000 7fb863fff640 1 -- 172.21.15.122:0/4232974769 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c02fdb0 con 0x7fb86c15bed0 2024-03-06T07:24:22.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 -- 172.21.15.122:0/4232974769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c15bed0 msgr2=0x7fb86c0726a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 --2- 172.21.15.122:0/4232974769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c15bed0 0x7fb86c0726a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009870 tx=0x7fb85c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 -- 172.21.15.122:0/4232974769 shutdown_connections 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 --2- 172.21.15.122:0/4232974769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c15bed0 0x7fb86c0726a0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 -- 172.21.15.122:0/4232974769 >> 172.21.15.122:0/4232974769 conn(0x7fb86c0b1a90 msgr2=0x7fb86c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 -- 172.21.15.122:0/4232974769 shutdown_connections 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.131+0000 7fb8734a4640 1 -- 172.21.15.122:0/4232974769 wait complete. 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.132+0000 7fb8734a4640 1 Processor -- start 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.132+0000 7fb8734a4640 1 -- start start 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb8734a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb86c06e170 0x7fb86c142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb8734a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 0x7fb86c153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb8734a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86c155130 0x7fb86c1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb8734a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c06f5b0 con 0x7fb86c0b50a0 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb8734a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c072be0 con 0x7fb86c155130 2024-03-06T07:24:22.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb8734a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c06f730 con 0x7fb86c06e170 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 0x7fb86c153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 0x7fb86c153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47016/0 (socket says 172.21.15.122:47016) 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 -- 172.21.15.122:0/3998685003 learned_addr learned my addr 172.21.15.122:0/3998685003 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb871240640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb86c06e170 0x7fb86c142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb871a41640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86c155130 0x7fb86c1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 -- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb86c06e170 msgr2=0x7fb86c142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb86c06e170 0x7fb86c142760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 -- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86c155130 msgr2=0x7fb86c1431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86c155130 0x7fb86c1431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 -- 172.21.15.122:0/3998685003 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c142ca0 con 0x7fb86c0b50a0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.133+0000 7fb870a3f640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 0x7fb86c153160 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb85c0090b0 tx=0x7fb85c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.134+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c047070 con 0x7fb86c0b50a0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.134+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c037cc0 con 0x7fb86c0b50a0 2024-03-06T07:24:22.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.134+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c153b00 con 0x7fb86c0b50a0 2024-03-06T07:24:22.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.134+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c004270 con 0x7fb86c0b50a0 2024-03-06T07:24:22.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.134+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c1540c0 con 0x7fb86c0b50a0 2024-03-06T07:24:22.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.135+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c04f330 con 0x7fb86c0b50a0 2024-03-06T07:24:22.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.135+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb85c04a070 con 0x7fb86c0b50a0 2024-03-06T07:24:22.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.135+0000 7fb861ffb640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb85006b4d0 0x7fb85006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.136+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb85c0ac230 con 0x7fb86c0b50a0 2024-03-06T07:24:22.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.136+0000 7fb871240640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb85006b4d0 0x7fb85006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.136+0000 7fb871240640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb85006b4d0 0x7fb85006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb868001920 tx=0x7fb8680018b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.140+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb85c033030 con 0x7fb86c0b50a0 2024-03-06T07:24:22.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.311+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb86c06d0f0 con 0x7fb85006b4d0 2024-03-06T07:24:22.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.317+0000 7fb861ffb640 1 -- 172.21.15.122:0/3998685003 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251951 (secure 0 0 0) 0x7fb86c06d0f0 con 0x7fb85006b4d0 2024-03-06T07:24:22.322 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:22.323 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:24:22.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb85006b4d0 msgr2=0x7fb85006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb85006b4d0 0x7fb85006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb868001920 tx=0x7fb8680018b0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 msgr2=0x7fb86c153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 0x7fb86c153160 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb85c0090b0 tx=0x7fb85c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 shutdown_connections 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb85006b4d0 0x7fb85006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86c155130 0x7fb86c1431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb86c0b50a0 0x7fb86c153160 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 --2- 172.21.15.122:0/3998685003 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb86c06e170 0x7fb86c142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 >> 172.21.15.122:0/3998685003 conn(0x7fb86c0b1a90 msgr2=0x7fb86c0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 shutdown_connections 2024-03-06T07:24:22.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.320+0000 7fb8734a4640 1 -- 172.21.15.122:0/3998685003 wait complete. 2024-03-06T07:24:22.333 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-03-06T07:24:21.360243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255980,"kb_used_data":6508,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718740,"statfs":{"total":773094113280,"available":772831989760,"internally_reserved":0,"allocated":6664192,"data_stored":3423032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.351256"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031202+0000","last_change":"2024-03-06T07:24:19.027740+0000","last_active":"2024-03-06T07:24:20.031202+0000","last_peered":"2024-03-06T07:24:20.031202+0000","last_clean":"2024-03-06T07:24:20.031202+0000","last_became_active":"2024-03-06T07:24:19.027500+0000","last_became_peered":"2024-03-06T07:24:19.027500+0000","last_unstale":"2024-03-06T07:24:20.031202+0000","last_undegraded":"2024-03-06T07:24:20.031202+0000","last_fullsized":"2024-03-06T07:24:20.031202+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:05.810668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058355+0000","last_change":"2024-03-06T07:24:19.029315+0000","last_active":"2024-03-06T07:24:20.058355+0000","last_peered":"2024-03-06T07:24:20.058355+0000","last_clean":"2024-03-06T07:24:20.058355+0000","last_became_active":"2024-03-06T07:24:19.029028+0000","last_became_peered":"2024-03-06T07:24:19.029028+0000","last_unstale":"2024-03-06T07:24:20.058355+0000","last_undegraded":"2024-03-06T07:24:20.058355+0000","last_fullsized":"2024-03-06T07:24:20.058355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:49.151798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059046+0000","last_change":"2024-03-06T07:24:19.037420+0000","last_active":"2024-03-06T07:24:20.059046+0000","last_peered":"2024-03-06T07:24:20.059046+0000","last_clean":"2024-03-06T07:24:20.059046+0000","last_became_active":"2024-03-06T07:24:19.037097+0000","last_became_peered":"2024-03-06T07:24:19.037097+0000","last_unstale":"2024-03-06T07:24:20.059046+0000","last_undegraded":"2024-03-06T07:24:20.059046+0000","last_fullsized":"2024-03-06T07:24:20.059046+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:54.420081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058887+0000","last_change":"2024-03-06T07:24:19.043360+0000","last_active":"2024-03-06T07:24:20.058887+0000","last_peered":"2024-03-06T07:24:20.058887+0000","last_clean":"2024-03-06T07:24:20.058887+0000","last_became_active":"2024-03-06T07:24:19.043085+0000","last_became_peered":"2024-03-06T07:24:19.043085+0000","last_unstale":"2024-03-06T07:24:20.058887+0000","last_undegraded":"2024-03-06T07:24:20.058887+0000","last_fullsized":"2024-03-06T07:24:20.058887+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:36.057457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031124+0000","last_change":"2024-03-06T07:24:19.039788+0000","last_active":"2024-03-06T07:24:20.031124+0000","last_peered":"2024-03-06T07:24:20.031124+0000","last_clean":"2024-03-06T07:24:20.031124+0000","last_became_active":"2024-03-06T07:24:19.039544+0000","last_became_peered":"2024-03-06T07:24:19.039544+0000","last_unstale":"2024-03-06T07:24:20.031124+0000","last_undegraded":"2024-03-06T07:24:20.031124+0000","last_fullsized":"2024-03-06T07:24:20.031124+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:26.122956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057801+0000","last_change":"2024-03-06T07:24:19.031360+0000","last_active":"2024-03-06T07:24:20.057801+0000","last_peered":"2024-03-06T07:24:20.057801+0000","last_clean":"2024-03-06T07:24:20.057801+0000","last_became_active":"2024-03-06T07:24:19.031084+0000","last_became_peered":"2024-03-06T07:24:19.031084+0000","last_unstale":"2024-03-06T07:24:20.057801+0000","last_undegraded":"2024-03-06T07:24:20.057801+0000","last_fullsized":"2024-03-06T07:24:20.057801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:36.057457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058038+0000","last_change":"2024-03-06T07:24:19.039716+0000","last_active":"2024-03-06T07:24:20.058038+0000","last_peered":"2024-03-06T07:24:20.058038+0000","last_clean":"2024-03-06T07:24:20.058038+0000","last_became_active":"2024-03-06T07:24:19.039219+0000","last_became_peered":"2024-03-06T07:24:19.039219+0000","last_unstale":"2024-03-06T07:24:20.058038+0000","last_undegraded":"2024-03-06T07:24:20.058038+0000","last_fullsized":"2024-03-06T07:24:20.058038+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:46.638512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057346+0000","last_change":"2024-03-06T07:24:19.037366+0000","last_active":"2024-03-06T07:24:20.057346+0000","last_peered":"2024-03-06T07:24:20.057346+0000","last_clean":"2024-03-06T07:24:20.057346+0000","last_became_active":"2024-03-06T07:24:19.037082+0000","last_became_peered":"2024-03-06T07:24:19.037082+0000","last_unstale":"2024-03-06T07:24:20.057346+0000","last_undegraded":"2024-03-06T07:24:20.057346+0000","last_fullsized":"2024-03-06T07:24:20.057346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:41.642813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057873+0000","last_change":"2024-03-06T07:24:19.037375+0000","last_active":"2024-03-06T07:24:20.057873+0000","last_peered":"2024-03-06T07:24:20.057873+0000","last_clean":"2024-03-06T07:24:20.057873+0000","last_became_active":"2024-03-06T07:24:19.036754+0000","last_became_peered":"2024-03-06T07:24:19.036754+0000","last_unstale":"2024-03-06T07:24:20.057873+0000","last_undegraded":"2024-03-06T07:24:20.057873+0000","last_fullsized":"2024-03-06T07:24:20.057873+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:31.236685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032035+0000","last_change":"2024-03-06T07:24:19.025580+0000","last_active":"2024-03-06T07:24:20.032035+0000","last_peered":"2024-03-06T07:24:20.032035+0000","last_clean":"2024-03-06T07:24:20.032035+0000","last_became_active":"2024-03-06T07:24:19.025310+0000","last_became_peered":"2024-03-06T07:24:19.025310+0000","last_unstale":"2024-03-06T07:24:20.032035+0000","last_undegraded":"2024-03-06T07:24:20.032035+0000","last_fullsized":"2024-03-06T07:24:20.032035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:40.804289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032108+0000","last_change":"2024-03-06T07:24:19.025140+0000","last_active":"2024-03-06T07:24:20.032108+0000","last_peered":"2024-03-06T07:24:20.032108+0000","last_clean":"2024-03-06T07:24:20.032108+0000","last_became_active":"2024-03-06T07:24:19.024817+0000","last_became_peered":"2024-03-06T07:24:19.024817+0000","last_unstale":"2024-03-06T07:24:20.032108+0000","last_undegraded":"2024-03-06T07:24:20.032108+0000","last_fullsized":"2024-03-06T07:24:20.032108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:21.771804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058963+0000","last_change":"2024-03-06T07:24:19.038114+0000","last_active":"2024-03-06T07:24:20.058963+0000","last_peered":"2024-03-06T07:24:20.058963+0000","last_clean":"2024-03-06T07:24:20.058963+0000","last_became_active":"2024-03-06T07:24:19.037712+0000","last_became_peered":"2024-03-06T07:24:19.037712+0000","last_unstale":"2024-03-06T07:24:20.058963+0000","last_undegraded":"2024-03-06T07:24:20.058963+0000","last_fullsized":"2024-03-06T07:24:20.058963+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:21.490775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032285+0000","last_change":"2024-03-06T07:24:19.023898+0000","last_active":"2024-03-06T07:24:20.032285+0000","last_peered":"2024-03-06T07:24:20.032285+0000","last_clean":"2024-03-06T07:24:20.032285+0000","last_became_active":"2024-03-06T07:24:19.023046+0000","last_became_peered":"2024-03-06T07:24:19.023046+0000","last_unstale":"2024-03-06T07:24:20.032285+0000","last_undegraded":"2024-03-06T07:24:20.032285+0000","last_fullsized":"2024-03-06T07:24:20.032285+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:43.236972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058432+0000","last_change":"2024-03-06T07:24:19.035379+0000","last_active":"2024-03-06T07:24:20.058432+0000","last_peered":"2024-03-06T07:24:20.058432+0000","last_clean":"2024-03-06T07:24:20.058432+0000","last_became_active":"2024-03-06T07:24:19.035031+0000","last_became_peered":"2024-03-06T07:24:19.035031+0000","last_unstale":"2024-03-06T07:24:20.058432+0000","last_undegraded":"2024-03-06T07:24:20.058432+0000","last_fullsized":"2024-03-06T07:24:20.058432+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:24.573580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058218+0000","last_change":"2024-03-06T07:24:19.036472+0000","last_active":"2024-03-06T07:24:20.058218+0000","last_peered":"2024-03-06T07:24:20.058218+0000","last_clean":"2024-03-06T07:24:20.058218+0000","last_became_active":"2024-03-06T07:24:19.036219+0000","last_became_peered":"2024-03-06T07:24:19.036219+0000","last_unstale":"2024-03-06T07:24:20.058218+0000","last_undegraded":"2024-03-06T07:24:20.058218+0000","last_fullsized":"2024-03-06T07:24:20.058218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:19.040346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057208+0000","last_change":"2024-03-06T07:24:19.027077+0000","last_active":"2024-03-06T07:24:20.057208+0000","last_peered":"2024-03-06T07:24:20.057208+0000","last_clean":"2024-03-06T07:24:20.057208+0000","last_became_active":"2024-03-06T07:24:19.026549+0000","last_became_peered":"2024-03-06T07:24:19.026549+0000","last_unstale":"2024-03-06T07:24:20.057208+0000","last_undegraded":"2024-03-06T07:24:20.057208+0000","last_fullsized":"2024-03-06T07:24:20.057208+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:27.221192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058209+0000","last_change":"2024-03-06T07:24:19.035221+0000","last_active":"2024-03-06T07:24:20.058209+0000","last_peered":"2024-03-06T07:24:20.058209+0000","last_clean":"2024-03-06T07:24:20.058209+0000","last_became_active":"2024-03-06T07:24:19.034944+0000","last_became_peered":"2024-03-06T07:24:19.034944+0000","last_unstale":"2024-03-06T07:24:20.058209+0000","last_undegraded":"2024-03-06T07:24:20.058209+0000","last_fullsized":"2024-03-06T07:24:20.058209+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:31.477413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057954+0000","last_change":"2024-03-06T07:24:19.043642+0000","last_active":"2024-03-06T07:24:20.057954+0000","last_peered":"2024-03-06T07:24:20.057954+0000","last_clean":"2024-03-06T07:24:20.057954+0000","last_became_active":"2024-03-06T07:24:19.043368+0000","last_became_peered":"2024-03-06T07:24:19.043368+0000","last_unstale":"2024-03-06T07:24:20.057954+0000","last_undegraded":"2024-03-06T07:24:20.057954+0000","last_fullsized":"2024-03-06T07:24:20.057954+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:30.673686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.030958+0000","last_change":"2024-03-06T07:24:19.027130+0000","last_active":"2024-03-06T07:24:20.030958+0000","last_peered":"2024-03-06T07:24:20.030958+0000","last_clean":"2024-03-06T07:24:20.030958+0000","last_became_active":"2024-03-06T07:24:19.026878+0000","last_became_peered":"2024-03-06T07:24:19.026878+0000","last_unstale":"2024-03-06T07:24:20.030958+0000","last_undegraded":"2024-03-06T07:24:20.030958+0000","last_fullsized":"2024-03-06T07:24:20.030958+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:23.851240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057808+0000","last_change":"2024-03-06T07:24:19.033048+0000","last_active":"2024-03-06T07:24:20.057808+0000","last_peered":"2024-03-06T07:24:20.057808+0000","last_clean":"2024-03-06T07:24:20.057808+0000","last_became_active":"2024-03-06T07:24:19.032763+0000","last_became_peered":"2024-03-06T07:24:19.032763+0000","last_unstale":"2024-03-06T07:24:20.057808+0000","last_undegraded":"2024-03-06T07:24:20.057808+0000","last_fullsized":"2024-03-06T07:24:20.057808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:30.287028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058343+0000","last_change":"2024-03-06T07:24:19.042682+0000","last_active":"2024-03-06T07:24:20.058343+0000","last_peered":"2024-03-06T07:24:20.058343+0000","last_clean":"2024-03-06T07:24:20.058343+0000","last_became_active":"2024-03-06T07:24:19.042396+0000","last_became_peered":"2024-03-06T07:24:19.042396+0000","last_unstale":"2024-03-06T07:24:20.058343+0000","last_undegraded":"2024-03-06T07:24:20.058343+0000","last_fullsized":"2024-03-06T07:24:20.058343+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:21.490775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057795+0000","last_change":"2024-03-06T07:24:19.034470+0000","last_active":"2024-03-06T07:24:20.057795+0000","last_peered":"2024-03-06T07:24:20.057795+0000","last_clean":"2024-03-06T07:24:20.057795+0000","last_became_active":"2024-03-06T07:24:19.033557+0000","last_became_peered":"2024-03-06T07:24:19.033557+0000","last_unstale":"2024-03-06T07:24:20.057795+0000","last_undegraded":"2024-03-06T07:24:20.057795+0000","last_fullsized":"2024-03-06T07:24:20.057795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:30.745625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058815+0000","last_change":"2024-03-06T07:24:19.039038+0000","last_active":"2024-03-06T07:24:20.058815+0000","last_peered":"2024-03-06T07:24:20.058815+0000","last_clean":"2024-03-06T07:24:20.058815+0000","last_became_active":"2024-03-06T07:24:19.038399+0000","last_became_peered":"2024-03-06T07:24:19.038399+0000","last_unstale":"2024-03-06T07:24:20.058815+0000","last_undegraded":"2024-03-06T07:24:20.058815+0000","last_fullsized":"2024-03-06T07:24:20.058815+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:49.991577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032193+0000","last_change":"2024-03-06T07:24:19.022261+0000","last_active":"2024-03-06T07:24:20.032193+0000","last_peered":"2024-03-06T07:24:20.032193+0000","last_clean":"2024-03-06T07:24:20.032193+0000","last_became_active":"2024-03-06T07:24:19.021730+0000","last_became_peered":"2024-03-06T07:24:19.021730+0000","last_unstale":"2024-03-06T07:24:20.032193+0000","last_undegraded":"2024-03-06T07:24:20.032193+0000","last_fullsized":"2024-03-06T07:24:20.032193+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:05.921214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031095+0000","last_change":"2024-03-06T07:24:19.029373+0000","last_active":"2024-03-06T07:24:20.031095+0000","last_peered":"2024-03-06T07:24:20.031095+0000","last_clean":"2024-03-06T07:24:20.031095+0000","last_became_active":"2024-03-06T07:24:19.029127+0000","last_became_peered":"2024-03-06T07:24:19.029127+0000","last_unstale":"2024-03-06T07:24:20.031095+0000","last_undegraded":"2024-03-06T07:24:20.031095+0000","last_fullsized":"2024-03-06T07:24:20.031095+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:27.471868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059217+0000","last_change":"2024-03-06T07:24:19.040759+0000","last_active":"2024-03-06T07:24:20.059217+0000","last_peered":"2024-03-06T07:24:20.059217+0000","last_clean":"2024-03-06T07:24:20.059217+0000","last_became_active":"2024-03-06T07:24:19.040485+0000","last_became_peered":"2024-03-06T07:24:19.040485+0000","last_unstale":"2024-03-06T07:24:20.059217+0000","last_undegraded":"2024-03-06T07:24:20.059217+0000","last_fullsized":"2024-03-06T07:24:20.059217+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:24.090559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058250+0000","last_change":"2024-03-06T07:24:19.025051+0000","last_active":"2024-03-06T07:24:20.058250+0000","last_peered":"2024-03-06T07:24:20.058250+0000","last_clean":"2024-03-06T07:24:20.058250+0000","last_became_active":"2024-03-06T07:24:19.024793+0000","last_became_peered":"2024-03-06T07:24:19.024793+0000","last_unstale":"2024-03-06T07:24:20.058250+0000","last_undegraded":"2024-03-06T07:24:20.058250+0000","last_fullsized":"2024-03-06T07:24:20.058250+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:24.090559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031103+0000","last_change":"2024-03-06T07:24:19.038797+0000","last_active":"2024-03-06T07:24:20.031103+0000","last_peered":"2024-03-06T07:24:20.031103+0000","last_clean":"2024-03-06T07:24:20.031103+0000","last_became_active":"2024-03-06T07:24:19.038542+0000","last_became_peered":"2024-03-06T07:24:19.038542+0000","last_unstale":"2024-03-06T07:24:20.031103+0000","last_undegraded":"2024-03-06T07:24:20.031103+0000","last_fullsized":"2024-03-06T07:24:20.031103+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:45.511066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058640+0000","last_change":"2024-03-06T07:24:19.042728+0000","last_active":"2024-03-06T07:24:20.058640+0000","last_peered":"2024-03-06T07:24:20.058640+0000","last_clean":"2024-03-06T07:24:20.058640+0000","last_became_active":"2024-03-06T07:24:19.042336+0000","last_became_peered":"2024-03-06T07:24:19.042336+0000","last_unstale":"2024-03-06T07:24:20.058640+0000","last_undegraded":"2024-03-06T07:24:20.058640+0000","last_fullsized":"2024-03-06T07:24:20.058640+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:57.706180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057931+0000","last_change":"2024-03-06T07:24:19.040616+0000","last_active":"2024-03-06T07:24:20.057931+0000","last_peered":"2024-03-06T07:24:20.057931+0000","last_clean":"2024-03-06T07:24:20.057931+0000","last_became_active":"2024-03-06T07:24:19.040344+0000","last_became_peered":"2024-03-06T07:24:19.040344+0000","last_unstale":"2024-03-06T07:24:20.057931+0000","last_undegraded":"2024-03-06T07:24:20.057931+0000","last_fullsized":"2024-03-06T07:24:20.057931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:04.603675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.030832+0000","last_change":"2024-03-06T07:24:19.028425+0000","last_active":"2024-03-06T07:24:20.030832+0000","last_peered":"2024-03-06T07:24:20.030832+0000","last_clean":"2024-03-06T07:24:20.030832+0000","last_became_active":"2024-03-06T07:24:19.028188+0000","last_became_peered":"2024-03-06T07:24:19.028188+0000","last_unstale":"2024-03-06T07:24:20.030832+0000","last_undegraded":"2024-03-06T07:24:20.030832+0000","last_fullsized":"2024-03-06T07:24:20.030832+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:54.120663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057268+0000","last_change":"2024-03-06T07:24:19.026862+0000","last_active":"2024-03-06T07:24:20.057268+0000","last_peered":"2024-03-06T07:24:20.057268+0000","last_clean":"2024-03-06T07:24:20.057268+0000","last_became_active":"2024-03-06T07:24:19.026218+0000","last_became_peered":"2024-03-06T07:24:19.026218+0000","last_unstale":"2024-03-06T07:24:20.057268+0000","last_undegraded":"2024-03-06T07:24:20.057268+0000","last_fullsized":"2024-03-06T07:24:20.057268+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:04.603675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058401+0000","last_change":"2024-03-06T07:24:19.035388+0000","last_active":"2024-03-06T07:24:20.058401+0000","last_peered":"2024-03-06T07:24:20.058401+0000","last_clean":"2024-03-06T07:24:20.058401+0000","last_became_active":"2024-03-06T07:24:19.034581+0000","last_became_peered":"2024-03-06T07:24:19.034581+0000","last_unstale":"2024-03-06T07:24:20.058401+0000","last_undegraded":"2024-03-06T07:24:20.058401+0000","last_fullsized":"2024-03-06T07:24:20.058401+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:24.648318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058249+0000","last_change":"2024-03-06T07:24:18.032384+0000","last_active":"2024-03-06T07:24:20.058249+0000","last_peered":"2024-03-06T07:24:20.058249+0000","last_clean":"2024-03-06T07:24:20.058249+0000","last_became_active":"2024-03-06T07:24:18.032039+0000","last_became_peered":"2024-03-06T07:24:18.032039+0000","last_unstale":"2024-03-06T07:24:20.058249+0000","last_undegraded":"2024-03-06T07:24:20.058249+0000","last_fullsized":"2024-03-06T07:24:20.058249+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:45.634728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032155+0000","last_change":"2024-03-06T07:24:19.032261+0000","last_active":"2024-03-06T07:24:20.032155+0000","last_peered":"2024-03-06T07:24:20.032155+0000","last_clean":"2024-03-06T07:24:20.032155+0000","last_became_active":"2024-03-06T07:24:19.031901+0000","last_became_peered":"2024-03-06T07:24:19.031901+0000","last_unstale":"2024-03-06T07:24:20.032155+0000","last_undegraded":"2024-03-06T07:24:20.032155+0000","last_fullsized":"2024-03-06T07:24:20.032155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:19.134434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058445+0000","last_change":"2024-03-06T07:24:18.023224+0000","last_active":"2024-03-06T07:24:20.058445+0000","last_peered":"2024-03-06T07:24:20.058445+0000","last_clean":"2024-03-06T07:24:20.058445+0000","last_became_active":"2024-03-06T07:24:18.022955+0000","last_became_peered":"2024-03-06T07:24:18.022955+0000","last_unstale":"2024-03-06T07:24:20.058445+0000","last_undegraded":"2024-03-06T07:24:20.058445+0000","last_fullsized":"2024-03-06T07:24:20.058445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:15.685665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058093+0000","last_change":"2024-03-06T07:24:19.027823+0000","last_active":"2024-03-06T07:24:20.058093+0000","last_peered":"2024-03-06T07:24:20.058093+0000","last_clean":"2024-03-06T07:24:20.058093+0000","last_became_active":"2024-03-06T07:24:19.027574+0000","last_became_peered":"2024-03-06T07:24:19.027574+0000","last_unstale":"2024-03-06T07:24:20.058093+0000","last_undegraded":"2024-03-06T07:24:20.058093+0000","last_fullsized":"2024-03-06T07:24:20.058093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:21.490775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058378+0000","last_change":"2024-03-06T07:24:18.030819+0000","last_active":"2024-03-06T07:24:20.058378+0000","last_peered":"2024-03-06T07:24:20.058378+0000","last_clean":"2024-03-06T07:24:20.058378+0000","last_became_active":"2024-03-06T07:24:18.030562+0000","last_became_peered":"2024-03-06T07:24:18.030562+0000","last_unstale":"2024-03-06T07:24:20.058378+0000","last_undegraded":"2024-03-06T07:24:20.058378+0000","last_fullsized":"2024-03-06T07:24:20.058378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:23.569796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058283+0000","last_change":"2024-03-06T07:24:19.039664+0000","last_active":"2024-03-06T07:24:20.058283+0000","last_peered":"2024-03-06T07:24:20.058283+0000","last_clean":"2024-03-06T07:24:20.058283+0000","last_became_active":"2024-03-06T07:24:19.039385+0000","last_became_peered":"2024-03-06T07:24:19.039385+0000","last_unstale":"2024-03-06T07:24:20.058283+0000","last_undegraded":"2024-03-06T07:24:20.058283+0000","last_fullsized":"2024-03-06T07:24:20.058283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:29.406311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058993+0000","last_change":"2024-03-06T07:24:18.031578+0000","last_active":"2024-03-06T07:24:20.058993+0000","last_peered":"2024-03-06T07:24:20.058993+0000","last_clean":"2024-03-06T07:24:20.058993+0000","last_became_active":"2024-03-06T07:24:18.031277+0000","last_became_peered":"2024-03-06T07:24:18.031277+0000","last_unstale":"2024-03-06T07:24:20.058993+0000","last_undegraded":"2024-03-06T07:24:20.058993+0000","last_fullsized":"2024-03-06T07:24:20.058993+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:18.258769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057931+0000","last_change":"2024-03-06T07:24:19.029566+0000","last_active":"2024-03-06T07:24:20.057931+0000","last_peered":"2024-03-06T07:24:20.057931+0000","last_clean":"2024-03-06T07:24:20.057931+0000","last_became_active":"2024-03-06T07:24:19.029290+0000","last_became_peered":"2024-03-06T07:24:19.029290+0000","last_unstale":"2024-03-06T07:24:20.057931+0000","last_undegraded":"2024-03-06T07:24:20.057931+0000","last_fullsized":"2024-03-06T07:24:20.057931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:43.931737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057997+0000","last_change":"2024-03-06T07:24:18.032504+0000","last_active":"2024-03-06T07:24:20.057997+0000","last_peered":"2024-03-06T07:24:20.057997+0000","last_clean":"2024-03-06T07:24:20.057997+0000","last_became_active":"2024-03-06T07:24:18.032189+0000","last_became_peered":"2024-03-06T07:24:18.032189+0000","last_unstale":"2024-03-06T07:24:20.057997+0000","last_undegraded":"2024-03-06T07:24:20.057997+0000","last_fullsized":"2024-03-06T07:24:20.057997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:48.456986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058552+0000","last_change":"2024-03-06T07:24:19.038983+0000","last_active":"2024-03-06T07:24:20.058552+0000","last_peered":"2024-03-06T07:24:20.058552+0000","last_clean":"2024-03-06T07:24:20.058552+0000","last_became_active":"2024-03-06T07:24:19.038729+0000","last_became_peered":"2024-03-06T07:24:19.038729+0000","last_unstale":"2024-03-06T07:24:20.058552+0000","last_undegraded":"2024-03-06T07:24:20.058552+0000","last_fullsized":"2024-03-06T07:24:20.058552+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:46.343704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057897+0000","last_change":"2024-03-06T07:24:18.025876+0000","last_active":"2024-03-06T07:24:20.057897+0000","last_peered":"2024-03-06T07:24:20.057897+0000","last_clean":"2024-03-06T07:24:20.057897+0000","last_became_active":"2024-03-06T07:24:18.024854+0000","last_became_peered":"2024-03-06T07:24:18.024854+0000","last_unstale":"2024-03-06T07:24:20.057897+0000","last_undegraded":"2024-03-06T07:24:20.057897+0000","last_fullsized":"2024-03-06T07:24:20.057897+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:18.036562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032192+0000","last_change":"2024-03-06T07:24:19.029063+0000","last_active":"2024-03-06T07:24:20.032192+0000","last_peered":"2024-03-06T07:24:20.032192+0000","last_clean":"2024-03-06T07:24:20.032192+0000","last_became_active":"2024-03-06T07:24:19.028783+0000","last_became_peered":"2024-03-06T07:24:19.028783+0000","last_unstale":"2024-03-06T07:24:20.032192+0000","last_undegraded":"2024-03-06T07:24:20.032192+0000","last_fullsized":"2024-03-06T07:24:20.032192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:00.387592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058157+0000","last_change":"2024-03-06T07:24:18.031884+0000","last_active":"2024-03-06T07:24:20.058157+0000","last_peered":"2024-03-06T07:24:20.058157+0000","last_clean":"2024-03-06T07:24:20.058157+0000","last_became_active":"2024-03-06T07:24:18.030235+0000","last_became_peered":"2024-03-06T07:24:18.030235+0000","last_unstale":"2024-03-06T07:24:20.058157+0000","last_undegraded":"2024-03-06T07:24:20.058157+0000","last_fullsized":"2024-03-06T07:24:20.058157+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:18.036562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057542+0000","last_change":"2024-03-06T07:24:19.029878+0000","last_active":"2024-03-06T07:24:20.057542+0000","last_peered":"2024-03-06T07:24:20.057542+0000","last_clean":"2024-03-06T07:24:20.057542+0000","last_became_active":"2024-03-06T07:24:19.029627+0000","last_became_peered":"2024-03-06T07:24:19.029627+0000","last_unstale":"2024-03-06T07:24:20.057542+0000","last_undegraded":"2024-03-06T07:24:20.057542+0000","last_fullsized":"2024-03-06T07:24:20.057542+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:24.179032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057512+0000","last_change":"2024-03-06T07:24:18.029703+0000","last_active":"2024-03-06T07:24:20.057512+0000","last_peered":"2024-03-06T07:24:20.057512+0000","last_clean":"2024-03-06T07:24:20.057512+0000","last_became_active":"2024-03-06T07:24:18.029157+0000","last_became_peered":"2024-03-06T07:24:18.029157+0000","last_unstale":"2024-03-06T07:24:20.057512+0000","last_undegraded":"2024-03-06T07:24:20.057512+0000","last_fullsized":"2024-03-06T07:24:20.057512+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:07.820313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058214+0000","last_change":"2024-03-06T07:24:19.037987+0000","last_active":"2024-03-06T07:24:20.058214+0000","last_peered":"2024-03-06T07:24:20.058214+0000","last_clean":"2024-03-06T07:24:20.058214+0000","last_became_active":"2024-03-06T07:24:19.036822+0000","last_became_peered":"2024-03-06T07:24:19.036822+0000","last_unstale":"2024-03-06T07:24:20.058214+0000","last_undegraded":"2024-03-06T07:24:20.058214+0000","last_fullsized":"2024-03-06T07:24:20.058214+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:54.764511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058073+0000","last_change":"2024-03-06T07:24:18.027726+0000","last_active":"2024-03-06T07:24:20.058073+0000","last_peered":"2024-03-06T07:24:20.058073+0000","last_clean":"2024-03-06T07:24:20.058073+0000","last_became_active":"2024-03-06T07:24:18.027418+0000","last_became_peered":"2024-03-06T07:24:18.027418+0000","last_unstale":"2024-03-06T07:24:20.058073+0000","last_undegraded":"2024-03-06T07:24:20.058073+0000","last_fullsized":"2024-03-06T07:24:20.058073+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:23.569796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057970+0000","last_change":"2024-03-06T07:24:19.042974+0000","last_active":"2024-03-06T07:24:20.057970+0000","last_peered":"2024-03-06T07:24:20.057970+0000","last_clean":"2024-03-06T07:24:20.057970+0000","last_became_active":"2024-03-06T07:24:19.042676+0000","last_became_peered":"2024-03-06T07:24:19.042676+0000","last_unstale":"2024-03-06T07:24:20.057970+0000","last_undegraded":"2024-03-06T07:24:20.057970+0000","last_fullsized":"2024-03-06T07:24:20.057970+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:16.154081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058533+0000","last_change":"2024-03-06T07:24:18.029944+0000","last_active":"2024-03-06T07:24:20.058533+0000","last_peered":"2024-03-06T07:24:20.058533+0000","last_clean":"2024-03-06T07:24:20.058533+0000","last_became_active":"2024-03-06T07:24:18.028944+0000","last_became_peered":"2024-03-06T07:24:18.028944+0000","last_unstale":"2024-03-06T07:24:20.058533+0000","last_undegraded":"2024-03-06T07:24:20.058533+0000","last_fullsized":"2024-03-06T07:24:20.058533+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.217408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031872+0000","last_change":"2024-03-06T07:24:19.038088+0000","last_active":"2024-03-06T07:24:20.031872+0000","last_peered":"2024-03-06T07:24:20.031872+0000","last_clean":"2024-03-06T07:24:20.031872+0000","last_became_active":"2024-03-06T07:24:19.037831+0000","last_became_peered":"2024-03-06T07:24:19.037831+0000","last_unstale":"2024-03-06T07:24:20.031872+0000","last_undegraded":"2024-03-06T07:24:20.031872+0000","last_fullsized":"2024-03-06T07:24:20.031872+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:18.564746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059204+0000","last_change":"2024-03-06T07:24:18.031108+0000","last_active":"2024-03-06T07:24:20.059204+0000","last_peered":"2024-03-06T07:24:20.059204+0000","last_clean":"2024-03-06T07:24:20.059204+0000","last_became_active":"2024-03-06T07:24:18.030684+0000","last_became_peered":"2024-03-06T07:24:18.030684+0000","last_unstale":"2024-03-06T07:24:20.059204+0000","last_undegraded":"2024-03-06T07:24:20.059204+0000","last_fullsized":"2024-03-06T07:24:20.059204+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:18.036562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059138+0000","last_change":"2024-03-06T07:24:19.042374+0000","last_active":"2024-03-06T07:24:20.059138+0000","last_peered":"2024-03-06T07:24:20.059138+0000","last_clean":"2024-03-06T07:24:20.059138+0000","last_became_active":"2024-03-06T07:24:19.041946+0000","last_became_peered":"2024-03-06T07:24:19.041946+0000","last_unstale":"2024-03-06T07:24:20.059138+0000","last_undegraded":"2024-03-06T07:24:20.059138+0000","last_fullsized":"2024-03-06T07:24:20.059138+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:12.788879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057894+0000","last_change":"2024-03-06T07:24:18.033183+0000","last_active":"2024-03-06T07:24:20.057894+0000","last_peered":"2024-03-06T07:24:20.057894+0000","last_clean":"2024-03-06T07:24:20.057894+0000","last_became_active":"2024-03-06T07:24:18.032923+0000","last_became_peered":"2024-03-06T07:24:18.032923+0000","last_unstale":"2024-03-06T07:24:20.057894+0000","last_undegraded":"2024-03-06T07:24:20.057894+0000","last_fullsized":"2024-03-06T07:24:20.057894+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:10.352057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057742+0000","last_change":"2024-03-06T07:24:19.039970+0000","last_active":"2024-03-06T07:24:20.057742+0000","last_peered":"2024-03-06T07:24:20.057742+0000","last_clean":"2024-03-06T07:24:20.057742+0000","last_became_active":"2024-03-06T07:24:19.039647+0000","last_became_peered":"2024-03-06T07:24:19.039647+0000","last_unstale":"2024-03-06T07:24:20.057742+0000","last_undegraded":"2024-03-06T07:24:20.057742+0000","last_fullsized":"2024-03-06T07:24:20.057742+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:27.221192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058637+0000","last_change":"2024-03-06T07:24:18.030989+0000","last_active":"2024-03-06T07:24:20.058637+0000","last_peered":"2024-03-06T07:24:20.058637+0000","last_clean":"2024-03-06T07:24:20.058637+0000","last_became_active":"2024-03-06T07:24:18.029227+0000","last_became_peered":"2024-03-06T07:24:18.029227+0000","last_unstale":"2024-03-06T07:24:20.058637+0000","last_undegraded":"2024-03-06T07:24:20.058637+0000","last_fullsized":"2024-03-06T07:24:20.058637+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:04.439937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058493+0000","last_change":"2024-03-06T07:24:19.034704+0000","last_active":"2024-03-06T07:24:20.058493+0000","last_peered":"2024-03-06T07:24:20.058493+0000","last_clean":"2024-03-06T07:24:20.058493+0000","last_became_active":"2024-03-06T07:24:19.034421+0000","last_became_peered":"2024-03-06T07:24:19.034421+0000","last_unstale":"2024-03-06T07:24:20.058493+0000","last_undegraded":"2024-03-06T07:24:20.058493+0000","last_fullsized":"2024-03-06T07:24:20.058493+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:24.090559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032048+0000","last_change":"2024-03-06T07:24:18.027111+0000","last_active":"2024-03-06T07:24:20.032048+0000","last_peered":"2024-03-06T07:24:20.032048+0000","last_clean":"2024-03-06T07:24:20.032048+0000","last_became_active":"2024-03-06T07:24:18.026809+0000","last_became_peered":"2024-03-06T07:24:18.026809+0000","last_unstale":"2024-03-06T07:24:20.032048+0000","last_undegraded":"2024-03-06T07:24:20.032048+0000","last_fullsized":"2024-03-06T07:24:20.032048+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:43.188495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057529+0000","last_change":"2024-03-06T07:24:19.038333+0000","last_active":"2024-03-06T07:24:20.057529+0000","last_peered":"2024-03-06T07:24:20.057529+0000","last_clean":"2024-03-06T07:24:20.057529+0000","last_became_active":"2024-03-06T07:24:19.038070+0000","last_became_peered":"2024-03-06T07:24:19.038070+0000","last_unstale":"2024-03-06T07:24:20.057529+0000","last_undegraded":"2024-03-06T07:24:20.057529+0000","last_fullsized":"2024-03-06T07:24:20.057529+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:46.648542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058246+0000","last_change":"2024-03-06T07:24:18.028849+0000","last_active":"2024-03-06T07:24:20.058246+0000","last_peered":"2024-03-06T07:24:20.058246+0000","last_clean":"2024-03-06T07:24:20.058246+0000","last_became_active":"2024-03-06T07:24:18.028513+0000","last_became_peered":"2024-03-06T07:24:18.028513+0000","last_unstale":"2024-03-06T07:24:20.058246+0000","last_undegraded":"2024-03-06T07:24:20.058246+0000","last_fullsized":"2024-03-06T07:24:20.058246+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:23.569796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058887+0000","last_change":"2024-03-06T07:24:19.039089+0000","last_active":"2024-03-06T07:24:20.058887+0000","last_peered":"2024-03-06T07:24:20.058887+0000","last_clean":"2024-03-06T07:24:20.058887+0000","last_became_active":"2024-03-06T07:24:19.038794+0000","last_became_peered":"2024-03-06T07:24:19.038794+0000","last_unstale":"2024-03-06T07:24:20.058887+0000","last_undegraded":"2024-03-06T07:24:20.058887+0000","last_fullsized":"2024-03-06T07:24:20.058887+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:24.648318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058317+0000","last_change":"2024-03-06T07:24:18.030751+0000","last_active":"2024-03-06T07:24:20.058317+0000","last_peered":"2024-03-06T07:24:20.058317+0000","last_clean":"2024-03-06T07:24:20.058317+0000","last_became_active":"2024-03-06T07:24:18.030471+0000","last_became_peered":"2024-03-06T07:24:18.030471+0000","last_unstale":"2024-03-06T07:24:20.058317+0000","last_undegraded":"2024-03-06T07:24:20.058317+0000","last_fullsized":"2024-03-06T07:24:20.058317+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:53.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058105+0000","last_change":"2024-03-06T07:24:19.040200+0000","last_active":"2024-03-06T07:24:20.058105+0000","last_peered":"2024-03-06T07:24:20.058105+0000","last_clean":"2024-03-06T07:24:20.058105+0000","last_became_active":"2024-03-06T07:24:19.039891+0000","last_became_peered":"2024-03-06T07:24:19.039891+0000","last_unstale":"2024-03-06T07:24:20.058105+0000","last_undegraded":"2024-03-06T07:24:20.058105+0000","last_fullsized":"2024-03-06T07:24:20.058105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:36.057457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058214+0000","last_change":"2024-03-06T07:24:18.028752+0000","last_active":"2024-03-06T07:24:20.058214+0000","last_peered":"2024-03-06T07:24:20.058214+0000","last_clean":"2024-03-06T07:24:20.058214+0000","last_became_active":"2024-03-06T07:24:18.028477+0000","last_became_peered":"2024-03-06T07:24:18.028477+0000","last_unstale":"2024-03-06T07:24:20.058214+0000","last_undegraded":"2024-03-06T07:24:20.058214+0000","last_fullsized":"2024-03-06T07:24:20.058214+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:43.231950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058555+0000","last_change":"2024-03-06T07:24:19.041500+0000","last_active":"2024-03-06T07:24:20.058555+0000","last_peered":"2024-03-06T07:24:20.058555+0000","last_clean":"2024-03-06T07:24:20.058555+0000","last_became_active":"2024-03-06T07:24:19.041204+0000","last_became_peered":"2024-03-06T07:24:19.041204+0000","last_unstale":"2024-03-06T07:24:20.058555+0000","last_undegraded":"2024-03-06T07:24:20.058555+0000","last_fullsized":"2024-03-06T07:24:20.058555+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:30.673686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058392+0000","last_change":"2024-03-06T07:24:18.028025+0000","last_active":"2024-03-06T07:24:20.058392+0000","last_peered":"2024-03-06T07:24:20.058392+0000","last_clean":"2024-03-06T07:24:20.058392+0000","last_became_active":"2024-03-06T07:24:18.027758+0000","last_became_peered":"2024-03-06T07:24:18.027758+0000","last_unstale":"2024-03-06T07:24:20.058392+0000","last_undegraded":"2024-03-06T07:24:20.058392+0000","last_fullsized":"2024-03-06T07:24:20.058392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.102294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057938+0000","last_change":"2024-03-06T07:24:19.035342+0000","last_active":"2024-03-06T07:24:20.057938+0000","last_peered":"2024-03-06T07:24:20.057938+0000","last_clean":"2024-03-06T07:24:20.057938+0000","last_became_active":"2024-03-06T07:24:19.035037+0000","last_became_peered":"2024-03-06T07:24:19.035037+0000","last_unstale":"2024-03-06T07:24:20.057938+0000","last_undegraded":"2024-03-06T07:24:20.057938+0000","last_fullsized":"2024-03-06T07:24:20.057938+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:56.856056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058629+0000","last_change":"2024-03-06T07:24:18.033385+0000","last_active":"2024-03-06T07:24:20.058629+0000","last_peered":"2024-03-06T07:24:20.058629+0000","last_clean":"2024-03-06T07:24:20.058629+0000","last_became_active":"2024-03-06T07:24:18.033080+0000","last_became_peered":"2024-03-06T07:24:18.033080+0000","last_unstale":"2024-03-06T07:24:20.058629+0000","last_undegraded":"2024-03-06T07:24:20.058629+0000","last_fullsized":"2024-03-06T07:24:20.058629+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:43.113655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059217+0000","last_change":"2024-03-06T07:24:19.042625+0000","last_active":"2024-03-06T07:24:20.059217+0000","last_peered":"2024-03-06T07:24:20.059217+0000","last_clean":"2024-03-06T07:24:20.059217+0000","last_became_active":"2024-03-06T07:24:19.042060+0000","last_became_peered":"2024-03-06T07:24:19.042060+0000","last_unstale":"2024-03-06T07:24:20.059217+0000","last_undegraded":"2024-03-06T07:24:20.059217+0000","last_fullsized":"2024-03-06T07:24:20.059217+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:28.413135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058094+0000","last_change":"2024-03-06T07:24:18.021164+0000","last_active":"2024-03-06T07:24:20.058094+0000","last_peered":"2024-03-06T07:24:20.058094+0000","last_clean":"2024-03-06T07:24:20.058094+0000","last_became_active":"2024-03-06T07:24:18.020859+0000","last_became_peered":"2024-03-06T07:24:18.020859+0000","last_unstale":"2024-03-06T07:24:20.058094+0000","last_undegraded":"2024-03-06T07:24:20.058094+0000","last_fullsized":"2024-03-06T07:24:20.058094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:30.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057898+0000","last_change":"2024-03-06T07:24:19.036202+0000","last_active":"2024-03-06T07:24:20.057898+0000","last_peered":"2024-03-06T07:24:20.057898+0000","last_clean":"2024-03-06T07:24:20.057898+0000","last_became_active":"2024-03-06T07:24:19.035941+0000","last_became_peered":"2024-03-06T07:24:19.035941+0000","last_unstale":"2024-03-06T07:24:20.057898+0000","last_undegraded":"2024-03-06T07:24:20.057898+0000","last_fullsized":"2024-03-06T07:24:20.057898+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:04.603675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058703+0000","last_change":"2024-03-06T07:24:18.030789+0000","last_active":"2024-03-06T07:24:20.058703+0000","last_peered":"2024-03-06T07:24:20.058703+0000","last_clean":"2024-03-06T07:24:20.058703+0000","last_became_active":"2024-03-06T07:24:18.030535+0000","last_became_peered":"2024-03-06T07:24:18.030535+0000","last_unstale":"2024-03-06T07:24:20.058703+0000","last_undegraded":"2024-03-06T07:24:20.058703+0000","last_fullsized":"2024-03-06T07:24:20.058703+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:03.599891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057562+0000","last_change":"2024-03-06T07:24:19.025616+0000","last_active":"2024-03-06T07:24:20.057562+0000","last_peered":"2024-03-06T07:24:20.057562+0000","last_clean":"2024-03-06T07:24:20.057562+0000","last_became_active":"2024-03-06T07:24:19.025356+0000","last_became_peered":"2024-03-06T07:24:19.025356+0000","last_unstale":"2024-03-06T07:24:20.057562+0000","last_undegraded":"2024-03-06T07:24:20.057562+0000","last_fullsized":"2024-03-06T07:24:20.057562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:16.154081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031727+0000","last_change":"2024-03-06T07:24:18.030819+0000","last_active":"2024-03-06T07:24:20.031727+0000","last_peered":"2024-03-06T07:24:20.031727+0000","last_clean":"2024-03-06T07:24:20.031727+0000","last_became_active":"2024-03-06T07:24:18.030562+0000","last_became_peered":"2024-03-06T07:24:18.030562+0000","last_unstale":"2024-03-06T07:24:20.031727+0000","last_undegraded":"2024-03-06T07:24:20.031727+0000","last_fullsized":"2024-03-06T07:24:20.031727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:20.278916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057297+0000","last_change":"2024-03-06T07:24:19.034596+0000","last_active":"2024-03-06T07:24:20.057297+0000","last_peered":"2024-03-06T07:24:20.057297+0000","last_clean":"2024-03-06T07:24:20.057297+0000","last_became_active":"2024-03-06T07:24:19.034324+0000","last_became_peered":"2024-03-06T07:24:19.034324+0000","last_unstale":"2024-03-06T07:24:20.057297+0000","last_undegraded":"2024-03-06T07:24:20.057297+0000","last_fullsized":"2024-03-06T07:24:20.057297+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:30.673686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058365+0000","last_change":"2024-03-06T07:24:18.032864+0000","last_active":"2024-03-06T07:24:20.058365+0000","last_peered":"2024-03-06T07:24:20.058365+0000","last_clean":"2024-03-06T07:24:20.058365+0000","last_became_active":"2024-03-06T07:24:18.032601+0000","last_became_peered":"2024-03-06T07:24:18.032601+0000","last_unstale":"2024-03-06T07:24:20.058365+0000","last_undegraded":"2024-03-06T07:24:20.058365+0000","last_fullsized":"2024-03-06T07:24:20.058365+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.102294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059333+0000","last_change":"2024-03-06T07:24:19.042578+0000","last_active":"2024-03-06T07:24:20.059333+0000","last_peered":"2024-03-06T07:24:20.059333+0000","last_clean":"2024-03-06T07:24:20.059333+0000","last_became_active":"2024-03-06T07:24:19.042327+0000","last_became_peered":"2024-03-06T07:24:19.042327+0000","last_unstale":"2024-03-06T07:24:20.059333+0000","last_undegraded":"2024-03-06T07:24:20.059333+0000","last_fullsized":"2024-03-06T07:24:20.059333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:29.406311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032191+0000","last_change":"2024-03-06T07:24:18.034298+0000","last_active":"2024-03-06T07:24:20.032191+0000","last_peered":"2024-03-06T07:24:20.032191+0000","last_clean":"2024-03-06T07:24:20.032191+0000","last_became_active":"2024-03-06T07:24:18.034028+0000","last_became_peered":"2024-03-06T07:24:18.034028+0000","last_unstale":"2024-03-06T07:24:20.032191+0000","last_undegraded":"2024-03-06T07:24:20.032191+0000","last_fullsized":"2024-03-06T07:24:20.032191+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:04.917430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058665+0000","last_change":"2024-03-06T07:24:19.039261+0000","last_active":"2024-03-06T07:24:20.058665+0000","last_peered":"2024-03-06T07:24:20.058665+0000","last_clean":"2024-03-06T07:24:20.058665+0000","last_became_active":"2024-03-06T07:24:19.038976+0000","last_became_peered":"2024-03-06T07:24:19.038976+0000","last_unstale":"2024-03-06T07:24:20.058665+0000","last_undegraded":"2024-03-06T07:24:20.058665+0000","last_fullsized":"2024-03-06T07:24:20.058665+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:16.154081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058486+0000","last_change":"2024-03-06T07:24:18.025191+0000","last_active":"2024-03-06T07:24:20.058486+0000","last_peered":"2024-03-06T07:24:20.058486+0000","last_clean":"2024-03-06T07:24:20.058486+0000","last_became_active":"2024-03-06T07:24:18.024907+0000","last_became_peered":"2024-03-06T07:24:18.024907+0000","last_unstale":"2024-03-06T07:24:20.058486+0000","last_undegraded":"2024-03-06T07:24:20.058486+0000","last_fullsized":"2024-03-06T07:24:20.058486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:53.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031922+0000","last_change":"2024-03-06T07:24:19.037469+0000","last_active":"2024-03-06T07:24:20.031922+0000","last_peered":"2024-03-06T07:24:20.031922+0000","last_clean":"2024-03-06T07:24:20.031922+0000","last_became_active":"2024-03-06T07:24:19.037213+0000","last_became_peered":"2024-03-06T07:24:19.037213+0000","last_unstale":"2024-03-06T07:24:20.031922+0000","last_undegraded":"2024-03-06T07:24:20.031922+0000","last_fullsized":"2024-03-06T07:24:20.031922+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:43.020758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032329+0000","last_change":"2024-03-06T07:24:18.030559+0000","last_active":"2024-03-06T07:24:20.032329+0000","last_peered":"2024-03-06T07:24:20.032329+0000","last_clean":"2024-03-06T07:24:20.032329+0000","last_became_active":"2024-03-06T07:24:18.030152+0000","last_became_peered":"2024-03-06T07:24:18.030152+0000","last_unstale":"2024-03-06T07:24:20.032329+0000","last_undegraded":"2024-03-06T07:24:20.032329+0000","last_fullsized":"2024-03-06T07:24:20.032329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:51.760044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058393+0000","last_change":"2024-03-06T07:24:19.039223+0000","last_active":"2024-03-06T07:24:20.058393+0000","last_peered":"2024-03-06T07:24:20.058393+0000","last_clean":"2024-03-06T07:24:20.058393+0000","last_became_active":"2024-03-06T07:24:19.038852+0000","last_became_peered":"2024-03-06T07:24:19.038852+0000","last_unstale":"2024-03-06T07:24:20.058393+0000","last_undegraded":"2024-03-06T07:24:20.058393+0000","last_fullsized":"2024-03-06T07:24:20.058393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:49.991577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.059067+0000","last_change":"2024-03-06T07:24:18.030680+0000","last_active":"2024-03-06T07:24:20.059067+0000","last_peered":"2024-03-06T07:24:20.059067+0000","last_clean":"2024-03-06T07:24:20.059067+0000","last_became_active":"2024-03-06T07:24:18.030413+0000","last_became_peered":"2024-03-06T07:24:18.030413+0000","last_unstale":"2024-03-06T07:24:20.059067+0000","last_undegraded":"2024-03-06T07:24:20.059067+0000","last_fullsized":"2024-03-06T07:24:20.059067+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:53.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058079+0000","last_change":"2024-03-06T07:24:19.027980+0000","last_active":"2024-03-06T07:24:20.058079+0000","last_peered":"2024-03-06T07:24:20.058079+0000","last_clean":"2024-03-06T07:24:20.058079+0000","last_became_active":"2024-03-06T07:24:19.027703+0000","last_became_peered":"2024-03-06T07:24:19.027703+0000","last_unstale":"2024-03-06T07:24:20.058079+0000","last_undegraded":"2024-03-06T07:24:20.058079+0000","last_fullsized":"2024-03-06T07:24:20.058079+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:50.936956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031537+0000","last_change":"2024-03-06T07:24:18.033531+0000","last_active":"2024-03-06T07:24:20.031537+0000","last_peered":"2024-03-06T07:24:20.031537+0000","last_clean":"2024-03-06T07:24:20.031537+0000","last_became_active":"2024-03-06T07:24:18.033258+0000","last_became_peered":"2024-03-06T07:24:18.033258+0000","last_unstale":"2024-03-06T07:24:20.031537+0000","last_undegraded":"2024-03-06T07:24:20.031537+0000","last_fullsized":"2024-03-06T07:24:20.031537+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:38.211170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058677+0000","last_change":"2024-03-06T07:24:19.040244+0000","last_active":"2024-03-06T07:24:20.058677+0000","last_peered":"2024-03-06T07:24:20.058677+0000","last_clean":"2024-03-06T07:24:20.058677+0000","last_became_active":"2024-03-06T07:24:19.039924+0000","last_became_peered":"2024-03-06T07:24:19.039924+0000","last_unstale":"2024-03-06T07:24:20.058677+0000","last_undegraded":"2024-03-06T07:24:20.058677+0000","last_fullsized":"2024-03-06T07:24:20.058677+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:29.406311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058537+0000","last_change":"2024-03-06T07:24:18.033917+0000","last_active":"2024-03-06T07:24:20.058537+0000","last_peered":"2024-03-06T07:24:20.058537+0000","last_clean":"2024-03-06T07:24:20.058537+0000","last_became_active":"2024-03-06T07:24:18.033614+0000","last_became_peered":"2024-03-06T07:24:18.033614+0000","last_unstale":"2024-03-06T07:24:20.058537+0000","last_undegraded":"2024-03-06T07:24:20.058537+0000","last_fullsized":"2024-03-06T07:24:20.058537+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:24.939681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031618+0000","last_change":"2024-03-06T07:24:19.024493+0000","last_active":"2024-03-06T07:24:20.031618+0000","last_peered":"2024-03-06T07:24:20.031618+0000","last_clean":"2024-03-06T07:24:20.031618+0000","last_became_active":"2024-03-06T07:24:19.024250+0000","last_became_peered":"2024-03-06T07:24:19.024250+0000","last_unstale":"2024-03-06T07:24:20.031618+0000","last_undegraded":"2024-03-06T07:24:20.031618+0000","last_fullsized":"2024-03-06T07:24:20.031618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.507789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.031687+0000","last_change":"2024-03-06T07:24:18.028930+0000","last_active":"2024-03-06T07:24:20.031687+0000","last_peered":"2024-03-06T07:24:20.031687+0000","last_clean":"2024-03-06T07:24:20.031687+0000","last_became_active":"2024-03-06T07:24:18.028531+0000","last_became_peered":"2024-03-06T07:24:18.028531+0000","last_unstale":"2024-03-06T07:24:20.031687+0000","last_undegraded":"2024-03-06T07:24:20.031687+0000","last_fullsized":"2024-03-06T07:24:20.031687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:59.383808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057673+0000","last_change":"2024-03-06T07:24:19.027235+0000","last_active":"2024-03-06T07:24:20.057673+0000","last_peered":"2024-03-06T07:24:20.057673+0000","last_clean":"2024-03-06T07:24:20.057673+0000","last_became_active":"2024-03-06T07:24:19.025852+0000","last_became_peered":"2024-03-06T07:24:19.025852+0000","last_unstale":"2024-03-06T07:24:20.057673+0000","last_undegraded":"2024-03-06T07:24:20.057673+0000","last_fullsized":"2024-03-06T07:24:20.057673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:42.676172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057638+0000","last_change":"2024-03-06T07:24:18.030948+0000","last_active":"2024-03-06T07:24:20.057638+0000","last_peered":"2024-03-06T07:24:20.057638+0000","last_clean":"2024-03-06T07:24:20.057638+0000","last_became_active":"2024-03-06T07:24:18.030667+0000","last_became_peered":"2024-03-06T07:24:18.030667+0000","last_unstale":"2024-03-06T07:24:20.057638+0000","last_undegraded":"2024-03-06T07:24:20.057638+0000","last_fullsized":"2024-03-06T07:24:20.057638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:43.836803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.057551+0000","last_change":"2024-03-06T07:24:19.035719+0000","last_active":"2024-03-06T07:24:20.057551+0000","last_peered":"2024-03-06T07:24:20.057551+0000","last_clean":"2024-03-06T07:24:20.057551+0000","last_became_active":"2024-03-06T07:24:19.028978+0000","last_became_peered":"2024-03-06T07:24:19.028978+0000","last_unstale":"2024-03-06T07:24:20.057551+0000","last_undegraded":"2024-03-06T07:24:20.057551+0000","last_fullsized":"2024-03-06T07:24:20.057551+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.006987+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.006987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:18.187937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.032029+0000","last_change":"2024-03-06T07:24:18.030115+0000","last_active":"2024-03-06T07:24:20.032029+0000","last_peered":"2024-03-06T07:24:20.032029+0000","last_clean":"2024-03-06T07:24:20.032029+0000","last_became_active":"2024-03-06T07:24:18.029832+0000","last_became_peered":"2024-03-06T07:24:18.029832+0000","last_unstale":"2024-03-06T07:24:20.032029+0000","last_undegraded":"2024-03-06T07:24:20.032029+0000","last_fullsized":"2024-03-06T07:24:20.032029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:17.003204+0000","last_clean_scrub_stamp":"2024-03-06T07:24:17.003204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:03.293315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":237,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:20.058382+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:24:20.058382+0000","last_peered":"2024-03-06T07:24:20.058382+0000","last_clean":"2024-03-06T07:24:20.058382+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:24:20.058382+0000","last_undegraded":"2024-03-06T07:24:20.058382+0000","last_fullsized":"2024-03-06T07:24:20.058382+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1286144,"data_stored":874150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":671744,"data_stored":277322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":277974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":285234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":277378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":277378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":277412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":876184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:22.337 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:22.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:22.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.480+0000 7f78acedf640 1 Processor -- start 2024-03-06T07:24:22.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.480+0000 7f78acedf640 1 -- start start 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78acedf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78a81586e0 0x7f78a80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78acedf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80aef80 0x7f78a80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78acedf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f78a80cffd0 0x7f78a806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78acedf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80d03f0 con 0x7f78a81586e0 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78acedf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a8158ac0 con 0x7f78a80cffd0 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78acedf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a815b2a0 con 0x7f78a80aef80 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78a81586e0 0x7f78a80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78a81586e0 0x7f78a80aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47022/0 (socket says 172.21.15.122:47022) 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 -- 172.21.15.122:0/1929209568 learned_addr learned my addr 172.21.15.122:0/1929209568 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a6d9d640 1 --2- 172.21.15.122:0/1929209568 >> v2:172.21.15.169:3300/0 conn(0x7f78a80cffd0 0x7f78a806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f789ffff640 1 --2- 172.21.15.122:0/1929209568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80aef80 0x7f78a80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 -- 172.21.15.122:0/1929209568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80aef80 msgr2=0x7f78a80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 --2- 172.21.15.122:0/1929209568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80aef80 0x7f78a80cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 -- 172.21.15.122:0/1929209568 >> v2:172.21.15.169:3300/0 conn(0x7f78a80cffd0 msgr2=0x7f78a806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 --2- 172.21.15.122:0/1929209568 >> v2:172.21.15.169:3300/0 conn(0x7f78a80cffd0 0x7f78a806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.481+0000 7f78a659c640 1 -- 172.21.15.122:0/1929209568 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a815b540 con 0x7f78a81586e0 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.482+0000 7f78a659c640 1 --2- 172.21.15.122:0/1929209568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80aea40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f78a8072e00 tx=0x7f789802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba644073adf8c9ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.482+0000 7f78a5d9b640 1 -- 172.21.15.122:0/1929209568 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898035020 con 0x7f78a81586e0 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.482+0000 7f78a5d9b640 1 -- 172.21.15.122:0/1929209568 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898031870 con 0x7f78a81586e0 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.482+0000 7f78a5d9b640 1 -- 172.21.15.122:0/1929209568 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898031c40 con 0x7f78a81586e0 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 -- 172.21.15.122:0/1929209568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 msgr2=0x7f78a80aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 --2- 172.21.15.122:0/1929209568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80aea40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f78a8072e00 tx=0x7f789802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 -- 172.21.15.122:0/1929209568 shutdown_connections 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 --2- 172.21.15.122:0/1929209568 >> v2:172.21.15.169:3300/0 conn(0x7f78a80cffd0 0x7f78a806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 --2- 172.21.15.122:0/1929209568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80aef80 0x7f78a80cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 --2- 172.21.15.122:0/1929209568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80aea40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 -- 172.21.15.122:0/1929209568 >> 172.21.15.122:0/1929209568 conn(0x7f78a806b190 msgr2=0x7f78a806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 -- 172.21.15.122:0/1929209568 shutdown_connections 2024-03-06T07:24:22.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.483+0000 7f78acedf640 1 -- 172.21.15.122:0/1929209568 wait complete. 2024-03-06T07:24:22.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.484+0000 7f78acedf640 1 Processor -- start 2024-03-06T07:24:22.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.484+0000 7f78acedf640 1 -- start start 2024-03-06T07:24:22.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78a80aef80 0x7f78a8145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80cffd0 0x7f78a80ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a815bde0 con 0x7f78a81586e0 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a815bb00 con 0x7f78a80aef80 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a8071350 con 0x7f78a80cffd0 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47030/0 (socket says 172.21.15.122:47030) 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 -- 172.21.15.122:0/4077475135 learned_addr learned my addr 172.21.15.122:0/4077475135 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f789ffff640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80cffd0 0x7f78a80ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a659c640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78a80aef80 0x7f78a8145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 -- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80cffd0 msgr2=0x7f78a80ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80cffd0 0x7f78a80ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 -- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78a80aef80 msgr2=0x7f78a8145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78a80aef80 0x7f78a8145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 -- 172.21.15.122:0/4077475135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8145840 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78a6d9d640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80b32a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f78a000efc0 tx=0x7f78a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0018020 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0014480 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a80adac0 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.485+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0014870 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.486+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a80ae080 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.486+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a815aff0 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.487+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78a0005d10 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.487+0000 7f789effd640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f788806b4d0 0x7f788806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.487+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a008ae30 con 0x7f78a81586e0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.487+0000 7f78a659c640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f788806b4d0 0x7f788806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.488+0000 7f78a659c640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f788806b4d0 0x7f788806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7898036000 tx=0x7f789803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.492+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78a815aff0 con 0x7f78a81586e0 2024-03-06T07:24:22.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.728+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f78a80732b0 con 0x7f78a81586e0 2024-03-06T07:24:22.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.729+0000 7f789effd640 1 -- 172.21.15.122:0/4077475135 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f78a0022030 con 0x7f78a81586e0 2024-03-06T07:24:22.731 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:24:22.731 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:24:22.732 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:24:22.732 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:24:22.732 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:22.732 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f788806b4d0 msgr2=0x7f788806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f788806b4d0 0x7f788806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7898036000 tx=0x7f789803c040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 msgr2=0x7f78a80b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80b32a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f78a000efc0 tx=0x7f78a000c490 comp rx=0 tx=0).stop 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 shutdown_connections 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f788806b4d0 0x7f788806d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78a81586e0 0x7f78a80b32a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78a80cffd0 0x7f78a80ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 --2- 172.21.15.122:0/4077475135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78a80aef80 0x7f78a8145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 >> 172.21.15.122:0/4077475135 conn(0x7f78a806b190 msgr2=0x7f78a8072210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 shutdown_connections 2024-03-06T07:24:22.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.732+0000 7f78acedf640 1 -- 172.21.15.122:0/4077475135 wait complete. 2024-03-06T07:24:22.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.842+0000 7fb6b0fbc640 1 Processor -- start 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.842+0000 7fb6b0fbc640 1 -- start start 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6b0fbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6b0fbc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 0x7fb6ac0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6b0fbc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac15f580 con 0x7fb6ac15e850 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6b0fbc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac0cf3f0 con 0x7fb6ac15bf30 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6b0fbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac15b210 con 0x7fb6ac13dc10 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6aa59c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6a9d9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 0x7fb6ac0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6aa59c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:55634/0 (socket says 172.21.15.122:55634) 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6a9d9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 0x7fb6ac0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:34862/0 (socket says 172.21.15.122:34862) 2024-03-06T07:24:22.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.843+0000 7fb6aa59c640 1 -- 172.21.15.122:0/2667093042 learned_addr learned my addr 172.21.15.122:0/2667093042 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.844+0000 7fb6aa59c640 1 -- 172.21.15.122:0/2667093042 >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 msgr2=0x7fb6ac0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.844+0000 7fb6aa59c640 1 --2- 172.21.15.122:0/2667093042 >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 0x7fb6ac0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.844+0000 7fb6aa59c640 1 -- 172.21.15.122:0/2667093042 >> v1:172.21.15.169:6789/0 conn(0x7fb6ac15e850 legacy=0x7fb6ac15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.844+0000 7fb6a959a640 1 -- 172.21.15.122:0/2667093042 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345187108 0 0) 0x7fb6ac15f580 con 0x7fb6ac15e850 2024-03-06T07:24:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.844+0000 7fb6aa59c640 1 -- 172.21.15.122:0/2667093042 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac15b450 con 0x7fb6ac13dc10 2024-03-06T07:24:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.844+0000 7fb6a9d9b640 1 --2- 172.21.15.122:0/2667093042 >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 0x7fb6ac0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6aa59c640 1 --2- 172.21.15.122:0/2667093042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac0b2950 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009e50 tx=0x7fb69c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537dabd6f1440d63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6a959a640 1 -- 172.21.15.122:0/2667093042 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c02ca70 con 0x7fb6ac13dc10 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6a959a640 1 -- 172.21.15.122:0/2667093042 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c02cc10 con 0x7fb6ac13dc10 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6a959a640 1 -- 172.21.15.122:0/2667093042 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0343f0 con 0x7fb6ac13dc10 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2667093042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 msgr2=0x7fb6ac0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2667093042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac0b2950 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009e50 tx=0x7fb69c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2667093042 shutdown_connections 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2667093042 >> v2:172.21.15.169:3300/0 conn(0x7fb6ac15bf30 0x7fb6ac0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2667093042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac0b2950 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2667093042 >> 172.21.15.122:0/2667093042 conn(0x7fb6ac0cf8a0 msgr2=0x7fb6ac0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2667093042 shutdown_connections 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.845+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2667093042 wait complete. 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 Processor -- start 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 -- start start 2024-03-06T07:24:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac15a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6ac15bf30 0x7fb6ac157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 0x7fb6ac0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac0708d0 con 0x7fb6ac15e850 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac13dff0 con 0x7fb6ac15bf30 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6b0fbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac070a50 con 0x7fb6ac13dc10 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6aad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 0x7fb6ac0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6aa59c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac15a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6a9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6ac15bf30 0x7fb6ac157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6aad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 0x7fb6ac0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47034/0 (socket says 172.21.15.122:47034) 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6aa59c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac15a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:55648/0 (socket says 172.21.15.122:55648) 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.847+0000 7fb6aad9d640 1 -- 172.21.15.122:0/2175149141 learned_addr learned my addr 172.21.15.122:0/2175149141 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aad9d640 1 -- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 msgr2=0x7fb6ac15a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aad9d640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac15a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aad9d640 1 -- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6ac15bf30 msgr2=0x7fb6ac157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aad9d640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6ac15bf30 0x7fb6ac157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aad9d640 1 -- 172.21.15.122:0/2175149141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac1f4710 con 0x7fb6ac15e850 2024-03-06T07:24:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aa59c640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac15a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6a9d9b640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6ac15bf30 0x7fb6ac157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6aad9d640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 0x7fb6ac0cd7a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400c3e0 tx=0x7fb6a400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4017070 con 0x7fb6ac15e850 2024-03-06T07:24:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a40035f0 con 0x7fb6ac15e850 2024-03-06T07:24:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac156c70 con 0x7fb6ac15e850 2024-03-06T07:24:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a40039e0 con 0x7fb6ac15e850 2024-03-06T07:24:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.848+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac15aa30 con 0x7fb6ac15e850 2024-03-06T07:24:22.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.849+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac04f330 con 0x7fb6ac15e850 2024-03-06T07:24:22.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.849+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6a401a070 con 0x7fb6ac15e850 2024-03-06T07:24:22.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.850+0000 7fb69affd640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb68c06b4d0 0x7fb68c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.850+0000 7fb6aa59c640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb68c06b4d0 0x7fb68c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.850+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6a4089500 con 0x7fb6ac15e850 2024-03-06T07:24:22.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.850+0000 7fb6aa59c640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb68c06b4d0 0x7fb68c06d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb69c035000 tx=0x7fb69c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:22.854+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6a4057d40 con 0x7fb6ac15e850 2024-03-06T07:24:23.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.061+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6ac06eb00 con 0x7fb6ac15e850 2024-03-06T07:24:23.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.062+0000 7fb69affd640 1 -- 172.21.15.122:0/2175149141 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fb6a405bde0 con 0x7fb6ac15e850 2024-03-06T07:24:23.064 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:23.064 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:23.064 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb68c06b4d0 msgr2=0x7fb68c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb68c06b4d0 0x7fb68c06d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb69c035000 tx=0x7fb69c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 msgr2=0x7fb6ac0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 0x7fb6ac0cd7a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400c3e0 tx=0x7fb6a400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 shutdown_connections 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb68c06b4d0 0x7fb68c06d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6ac15e850 0x7fb6ac0cd7a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb6ac15bf30 0x7fb6ac157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 --2- 172.21.15.122:0/2175149141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6ac13dc10 0x7fb6ac15a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 >> 172.21.15.122:0/2175149141 conn(0x7fb6ac0cf8a0 msgr2=0x7fb6ac0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 shutdown_connections 2024-03-06T07:24:23.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.064+0000 7fb6b0fbc640 1 -- 172.21.15.122:0/2175149141 wait complete. 2024-03-06T07:24:23.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:23.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:23.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:18.367265+0000', 'modified': '2024-03-06T07:24:20.368965+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6156}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6156': {'gid': 6156, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.169:6834/2303645671', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6832', 'nonce': 2303645671}, {'type': 'v1', 'addr': '172.21.15.169:6834', 'nonce': 2303645671}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6156, 'qdb_cluster': [6156]} 2024-03-06T07:24:23.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:23.480 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:24:23.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.578+0000 7f8cf8837640 1 Processor -- start 2024-03-06T07:24:23.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.578+0000 7f8cf8837640 1 -- start start 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.579+0000 7f8cf8837640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 0x7f8cf4071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.579+0000 7f8cf8837640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8cf40cd7c0 0x7f8cf40cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.579+0000 7f8cf8837640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf4072490 con 0x7f8cf40cebc0 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.579+0000 7f8cf8837640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf40ce170 con 0x7f8cf40cd7c0 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.579+0000 7f8cf8837640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40d0090 con 0x7f8cf415bbf0 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 0x7f8cf4071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf2d9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8cf40cd7c0 0x7f8cf40cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 0x7f8cf4071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:55664/0 (socket says 172.21.15.122:55664) 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 -- 172.21.15.122:0/965435645 learned_addr learned my addr 172.21.15.122:0/965435645 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 -- 172.21.15.122:0/965435645 >> v2:172.21.15.169:3300/0 conn(0x7f8cf40cd7c0 msgr2=0x7f8cf40cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 --2- 172.21.15.122:0/965435645 >> v2:172.21.15.169:3300/0 conn(0x7f8cf40cd7c0 0x7f8cf40cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 -- 172.21.15.122:0/965435645 >> v1:172.21.15.169:6789/0 conn(0x7f8cf40cebc0 legacy=0x7f8cf415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf259c640 1 -- 172.21.15.122:0/965435645 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf40d0370 con 0x7f8cf415bbf0 2024-03-06T07:24:23.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.580+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/965435645 >> v2:172.21.15.169:3300/0 conn(0x7f8cf40cd7c0 0x7f8cf40cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf259c640 1 --2- 172.21.15.122:0/965435645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 0x7f8cf4071e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009e50 tx=0x7f8ce402bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3537b427e34b4573 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf159a640 1 -- 172.21.15.122:0/965435645 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce402caf0 con 0x7f8cf415bbf0 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf159a640 1 -- 172.21.15.122:0/965435645 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce402cc90 con 0x7f8cf415bbf0 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf159a640 1 -- 172.21.15.122:0/965435645 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4034610 con 0x7f8cf415bbf0 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 -- 172.21.15.122:0/965435645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 msgr2=0x7f8cf4071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 --2- 172.21.15.122:0/965435645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 0x7f8cf4071e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009e50 tx=0x7f8ce402bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 -- 172.21.15.122:0/965435645 shutdown_connections 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 --2- 172.21.15.122:0/965435645 >> v2:172.21.15.169:3300/0 conn(0x7f8cf40cd7c0 0x7f8cf40cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 --2- 172.21.15.122:0/965435645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf415bbf0 0x7f8cf4071e00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 -- 172.21.15.122:0/965435645 >> 172.21.15.122:0/965435645 conn(0x7f8cf40b1c80 msgr2=0x7f8cf40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 -- 172.21.15.122:0/965435645 shutdown_connections 2024-03-06T07:24:23.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.581+0000 7f8cf8837640 1 -- 172.21.15.122:0/965435645 wait complete. 2024-03-06T07:24:23.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 Processor -- start 2024-03-06T07:24:23.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 -- start start 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cf40cd7c0 0x7f8cf41f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf40cebc0 0x7f8cf41443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 0x7f8cf406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40cefe0 con 0x7f8cf415bbf0 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf415c6e0 con 0x7f8cf40cd7c0 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.583+0000 7f8cf8837640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40b3680 con 0x7f8cf40cebc0 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 0x7f8cf406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 0x7f8cf406db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47038/0 (socket says 172.21.15.122:47038) 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cf40cd7c0 0x7f8cf41f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2239270321 learned_addr learned my addr 172.21.15.122:0/2239270321 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf40cebc0 0x7f8cf41443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf40cebc0 msgr2=0x7f8cf41443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf40cebc0 0x7f8cf41443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cf40cd7c0 msgr2=0x7f8cf41f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cf40cd7c0 0x7f8cf41f4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2239270321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf40c4b10 con 0x7f8cf415bbf0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cf40cd7c0 0x7f8cf41f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf40cebc0 0x7f8cf41443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 0x7f8cf406db70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00e960 tx=0x7f8cec00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8ce2ffd640 1 -- 172.21.15.122:0/2239270321 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec017070 con 0x7f8cf415bbf0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8ce2ffd640 1 -- 172.21.15.122:0/2239270321 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec004020 con 0x7f8cf415bbf0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf40c4df0 con 0x7f8cf415bbf0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8ce2ffd640 1 -- 172.21.15.122:0/2239270321 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0043f0 con 0x7f8cf415bbf0 2024-03-06T07:24:23.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.584+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf40c53b0 con 0x7f8cf415bbf0 2024-03-06T07:24:23.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.586+0000 7f8ce2ffd640 1 -- 172.21.15.122:0/2239270321 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cec00ca00 con 0x7f8cf415bbf0 2024-03-06T07:24:23.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.586+0000 7f8ce2ffd640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8cd406b4d0 0x7f8cd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.586+0000 7f8ce2ffd640 1 -- 172.21.15.122:0/2239270321 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cec08a330 con 0x7f8cf415bbf0 2024-03-06T07:24:23.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.586+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8cd406b4d0 0x7f8cd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.587+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8cd406b4d0 0x7f8cd406d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4035000 tx=0x7f8ce4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.587+0000 7f8cfb419740 1 Processor -- start 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.587+0000 7f8cfb419740 1 -- start start 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea64d870 0x55c4ea64dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 0x55c4ea653360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea67da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cfb419740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea5b76b0 con 0x55c4ea6518c0 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea560a60 con 0x55c4ea67d600 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea5b3750 con 0x55c4ea64d870 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 0x55c4ea653360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 0x55c4ea653360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47054/0 (socket says 172.21.15.122:47054) 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2481463603 learned_addr learned my addr 172.21.15.122:0/2481463603 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea64d870 0x55c4ea64dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea67da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea64d870 msgr2=0x55c4ea64dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea64d870 0x55c4ea64dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 msgr2=0x55c4ea67da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea67da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2481463603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4ea57ef30 con 0x55c4ea6518c0 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.588+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 0x55c4ea653360 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8001bf0 tx=0x7f8ce8007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bee98a4ab16fa30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cd3fff640 1 -- 172.21.15.122:0/2481463603 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800fe50 con 0x55c4ea6518c0 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cd3fff640 1 -- 172.21.15.122:0/2481463603 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8018070 con 0x55c4ea6518c0 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cd3fff640 1 -- 172.21.15.122:0/2481463603 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80052d0 con 0x55c4ea6518c0 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 -- 172.21.15.122:0/2481463603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 msgr2=0x55c4ea653360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 0x55c4ea653360 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8001bf0 tx=0x7f8ce8007e50 comp rx=0 tx=0).stop 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 -- 172.21.15.122:0/2481463603 shutdown_connections 2024-03-06T07:24:23.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea67da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6518c0 0x55c4ea653360 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2481463603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea64d870 0x55c4ea64dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 -- 172.21.15.122:0/2481463603 >> 172.21.15.122:0/2481463603 conn(0x55c4ea64f8f0 msgr2=0x55c4ea6511b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 -- 172.21.15.122:0/2481463603 shutdown_connections 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.589+0000 7f8cfb419740 1 -- 172.21.15.122:0/2481463603 wait complete. 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.590+0000 7f8cfb419740 1 -- start start 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.590+0000 7f8cfb419740 1 Processor -- start 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea6725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea67a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 0x55c4ea6760a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea651d50 con 0x55c4ea6709c0 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea573ea0 con 0x55c4ea67d600 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea652030 con 0x55c4ea67f1c0 2024-03-06T07:24:23.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cfb419740 10 client.0 _ll_register_callbacks cb 0x55c4ea573870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 0x55c4ea6760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 0x55c4ea6760a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47064/0 (socket says 172.21.15.122:47064) 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2036683106 learned_addr learned my addr 172.21.15.122:0/2036683106 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea6725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea67a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 msgr2=0x55c4ea67a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea67a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 msgr2=0x55c4ea6725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea6725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/2036683106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4ea574270 con 0x55c4ea6709c0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 0x55c4ea6760a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8cec054fc0 tx=0x7f8cec00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acad61c4a333958 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.591+0000 7f8cd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2036683106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec070020 con 0x55c4ea6709c0 2024-03-06T07:24:23.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2036683106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cec057fb0 con 0x55c4ea6709c0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ce8018070 con 0x55c4ea6709c0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2036683106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cec075050 con 0x55c4ea6709c0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 10 client.6285 fetch_fsmap learned FSMap version 27 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x55c4ea2cc2d0 con 0x55c4ea6709c0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2036683106 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8018070 con 0x55c4ea6709c0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2036683106 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 27) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8cec05db20 con 0x55c4ea6709c0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 10 client.6285 fetch_fsmap finished waiting for FSMap version 27 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 10 client.6285 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 20 client.6285 populate_metadata read hostname 'smithi122' 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea66fce0 0x55c4ea652ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 4 client.6285 mds_command: new command op to 6156 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 --> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4ea5c4bf0 con 0x55c4ea66fce0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.592+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea66fce0 0x55c4ea652ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cd17fa640 10 client.6285 ms_handle_connect on v2:172.21.15.169:6832/2303645671 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea66fce0 0x55c4ea652ad0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b38cd8b0f9b4168 server_cookie=b27b86edcdc38f11 in_seq=0 out_seq=0 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2036683106 <== mds.0 v2:172.21.15.169:6832/2303645671 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c4ea5c4bf0 con 0x55c4ea66fce0 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cd17fa640 10 client.6285 handle_command_reply: tid=0 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 client.6285 shutdown 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 msgr2=0x55c4ea6760a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 0x55c4ea6760a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8cec054fc0 tx=0x7f8cec00cdb0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 shutdown_connections 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea66fce0 0x55c4ea652ad0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea6709c0 0x55c4ea6760a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea67a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cd17fa640 0 client.6285 ms_handle_reset on v2:172.21.15.169:6832/2303645671 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2036683106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea67d600 0x55c4ea6725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.593+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 >> 172.21.15.122:0/2036683106 conn(0x55c4ea64d870 msgr2=0x55c4ea64c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.594+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 shutdown_connections 2024-03-06T07:24:23.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.594+0000 7f8cfb419740 1 -- 172.21.15.122:0/2036683106 wait complete. 2024-03-06T07:24:23.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.594+0000 7f8cfb419740 20 client.6285 trim_cache size 0 max 16384 2024-03-06T07:24:23.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.610+0000 7f8cfb419740 1 Processor -- start 2024-03-06T07:24:23.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.610+0000 7f8cfb419740 1 -- start start 2024-03-06T07:24:23.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea679f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67d600 0x55c4ea672840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cfb419740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf41ff400 con 0x55c4ea67f1c0 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40c6900 con 0x55c4ea6709c0 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf4160110 con 0x55c4ea67d600 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47066/0 (socket says 172.21.15.122:47066) 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/15247923 learned_addr learned my addr 172.21.15.122:0/15247923 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf259c640 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea679f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67d600 0x55c4ea672840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67d600 msgr2=0x55c4ea672840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67d600 0x55c4ea672840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 msgr2=0x55c4ea679f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea679f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 -- 172.21.15.122:0/15247923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4ea3966b0 con 0x55c4ea67f1c0 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 0x55c4ea679190 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0554e0 tx=0x7f8cec05b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345ceae02f0d7f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cd37fe640 1 -- 172.21.15.122:0/15247923 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec070040 con 0x55c4ea67f1c0 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cd37fe640 1 -- 172.21.15.122:0/15247923 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec057fb0 con 0x55c4ea67f1c0 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.611+0000 7f8cd37fe640 1 -- 172.21.15.122:0/15247923 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec020040 con 0x55c4ea67f1c0 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 -- 172.21.15.122:0/15247923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 msgr2=0x55c4ea679190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 0x55c4ea679190 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0554e0 tx=0x7f8cec05b000 comp rx=0 tx=0).stop 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 -- 172.21.15.122:0/15247923 shutdown_connections 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67d600 0x55c4ea672840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 --2- 172.21.15.122:0/15247923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea679f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 -- 172.21.15.122:0/15247923 >> 172.21.15.122:0/15247923 conn(0x55c4ea64d870 msgr2=0x55c4ea674810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 -- 172.21.15.122:0/15247923 shutdown_connections 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 -- 172.21.15.122:0/15247923 wait complete. 2024-03-06T07:24:23.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 -- start start 2024-03-06T07:24:23.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.612+0000 7f8cfb419740 1 Processor -- start 2024-03-06T07:24:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea673390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 0x55c4ea705b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea67f7b0 con 0x55c4ea67d600 2024-03-06T07:24:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea675980 con 0x55c4ea6709c0 2024-03-06T07:24:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4ea670110 con 0x55c4ea67f1c0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cfb419740 10 client.0 _ll_register_callbacks cb 0x7f8cec0584c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cf1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 0x55c4ea705b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cf1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 0x55c4ea705b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47070/0 (socket says 172.21.15.122:47070) 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2705849617 learned_addr learned my addr 172.21.15.122:0/2705849617 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.613+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea673390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf2d9d640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 msgr2=0x55c4ea679190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 msgr2=0x55c4ea673390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea673390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf1d9b640 1 -- 172.21.15.122:0/2705849617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4ea6744d0 con 0x55c4ea67d600 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cf1d9b640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 0x55c4ea705b50 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8015040 tx=0x7f8ce8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59c8d47fb8f65e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:24:23.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2705849617 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8007020 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2705849617 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ce8028ac0 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cec057fb0 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2705849617 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ce802f050 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 10 client.6291 fetch_fsmap learned FSMap version 27 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x55c4ea706090 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2705849617 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec057fb0 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2705849617 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 27) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8ce8032020 con 0x55c4ea67d600 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 10 client.6291 fetch_fsmap finished waiting for FSMap version 27 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 10 client.6291 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 20 client.6291 populate_metadata read hostname 'smithi122' 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea711e90 0x55c4ea714280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 4 client.6291 mds_command: new command op to 6156 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.614+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 --> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c4ea650050 con 0x55c4ea711e90 2024-03-06T07:24:23.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea711e90 0x55c4ea714280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cd17fa640 10 client.6291 ms_handle_connect on v2:172.21.15.169:6832/2303645671 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cf259c640 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea711e90 0x55c4ea714280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe04dd62eab57e50 server_cookie=73abdf7a630d91c in_seq=0 out_seq=0 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cd17fa640 1 -- 172.21.15.122:0/2705849617 <== mds.0 v2:172.21.15.169:6832/2303645671 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c4ea650050 con 0x55c4ea711e90 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cd17fa640 10 client.6291 handle_command_reply: tid=0 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 client.6291 shutdown 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 msgr2=0x55c4ea705b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 0x55c4ea705b50 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8015040 tx=0x7f8ce8000bc0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 shutdown_connections 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:6832/2303645671,v1:172.21.15.169:6834/2303645671] conn(0x55c4ea711e90 0x55c4ea714280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55c4ea67f1c0 0x55c4ea679190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c4ea67d600 0x55c4ea705b50 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cd17fa640 0 client.6291 ms_handle_reset on v2:172.21.15.169:6832/2303645671 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 --2- 172.21.15.122:0/2705849617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c4ea6709c0 0x55c4ea673390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 >> 172.21.15.122:0/2705849617 conn(0x55c4ea64d870 msgr2=0x55c4ea672840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 shutdown_connections 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 1 -- 172.21.15.122:0/2705849617 wait complete. 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.615+0000 7f8cfb419740 20 client.6291 trim_cache size 0 max 16384 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 6156, 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:24:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 4.6025634919999998, 2024-03-06T07:24:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 27, 2024-03-06T07:24:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 57, 2024-03-06T07:24:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 57, 2024-03-06T07:24:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 9.9297446699999998 2024-03-06T07:24:23.625 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8cd406b4d0 msgr2=0x7f8cd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8cd406b4d0 0x7f8cd406d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4035000 tx=0x7f8ce4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 msgr2=0x7f8cf406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 0x7f8cf406db70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00e960 tx=0x7f8cec00ee20 comp rx=0 tx=0).stop 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 shutdown_connections 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8cd406b4d0 0x7f8cd406d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf415bbf0 0x7f8cf406db70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8cf40cebc0 0x7f8cf41443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 --2- 172.21.15.122:0/2239270321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cf40cd7c0 0x7f8cf41f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 >> 172.21.15.122:0/2239270321 conn(0x7f8cf40b1c80 msgr2=0x7f8cf41507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 shutdown_connections 2024-03-06T07:24:23.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.616+0000 7f8cf8837640 1 -- 172.21.15.122:0/2239270321 wait complete. 2024-03-06T07:24:23.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:23.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.726+0000 7f8fdb92c640 1 Processor -- start 2024-03-06T07:24:23.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.726+0000 7f8fdb92c640 1 -- start start 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fdb92c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8fd40d00c0 0x7f8fd40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fdb92c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fdb92c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40d09e0 con 0x7f8fd40cf850 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fdb92c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd406ca90 con 0x7f8fd40d00c0 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fdb92c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd415e380 con 0x7f8fd406cf20 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8fd40d00c0 0x7f8fd40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8fd40d00c0 0x7f8fd40d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:34942/0 (socket says 172.21.15.122:34942) 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd9ec9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 -- 172.21.15.122:0/2492044891 learned_addr learned my addr 172.21.15.122:0/2492044891 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd8ec7640 1 --1- 172.21.15.122:0/2492044891 >> v1:172.21.15.169:6789/0 conn(0x7f8fd40cf850 0x7f8fd40cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:43564/0 (socket says 172.21.15.122:43564) 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 -- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 msgr2=0x7f8fd406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 --2- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd406d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 -- 172.21.15.122:0/2492044891 >> v1:172.21.15.169:6789/0 conn(0x7f8fd40cf850 legacy=0x7f8fd40cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd96c8640 1 -- 172.21.15.122:0/2492044891 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40cdc10 con 0x7f8fd40d00c0 2024-03-06T07:24:23.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.727+0000 7f8fd9ec9640 1 --2- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:23.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.728+0000 7f8fd96c8640 1 --2- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40d04a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009870 tx=0x7f8fd002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2021d6c89aed13d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.728+0000 7f8fcbfff640 1 -- 172.21.15.122:0/2492044891 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0034070 con 0x7f8fd40d00c0 2024-03-06T07:24:23.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.728+0000 7f8fcbfff640 1 -- 172.21.15.122:0/2492044891 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd0030840 con 0x7f8fd40d00c0 2024-03-06T07:24:23.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.728+0000 7f8fcbfff640 1 -- 172.21.15.122:0/2492044891 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0030c10 con 0x7f8fd40d00c0 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 -- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 msgr2=0x7f8fd40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40d04a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009870 tx=0x7f8fd002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 -- 172.21.15.122:0/2492044891 shutdown_connections 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/2492044891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40d04a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 -- 172.21.15.122:0/2492044891 >> 172.21.15.122:0/2492044891 conn(0x7f8fd40b35a0 msgr2=0x7f8fd40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 -- 172.21.15.122:0/2492044891 shutdown_connections 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.729+0000 7f8fdb92c640 1 -- 172.21.15.122:0/2492044891 wait complete. 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.730+0000 7f8fdb92c640 1 Processor -- start 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.730+0000 7f8fdb92c640 1 -- start start 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd40c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 0x7f8fd40c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd406c720 con 0x7f8fd40cf850 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd406c440 con 0x7f8fd40d00c0 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b1030 con 0x7f8fd406cf20 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 0x7f8fd40c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 0x7f8fd40c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47074/0 (socket says 172.21.15.122:47074) 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 -- 172.21.15.122:0/595563400 learned_addr learned my addr 172.21.15.122:0/595563400 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd96c8640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd40c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd9ec9640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 -- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 msgr2=0x7f8fd40c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd40c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 -- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 msgr2=0x7f8fd40c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 -- 172.21.15.122:0/595563400 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd415d340 con 0x7f8fd40cf850 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fd8ec7640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 0x7f8fd40c55a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c400 tx=0x7f8fc400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40074e0 con 0x7f8fd40cf850 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4007680 con 0x7f8fd40cf850 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4004a00 con 0x7f8fd40cf850 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.731+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd40b48a0 con 0x7f8fd40cf850 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.732+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd4156190 con 0x7f8fd40cf850 2024-03-06T07:24:23.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.733+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd40b4690 con 0x7f8fd40cf850 2024-03-06T07:24:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.733+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fc401a020 con 0x7f8fd40cf850 2024-03-06T07:24:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.733+0000 7f8fc9ffb640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fb806b4d0 0x7f8fb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.733+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fc4089800 con 0x7f8fd40cf850 2024-03-06T07:24:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.733+0000 7f8fd96c8640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fb806b4d0 0x7f8fb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.734+0000 7f8fd96c8640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fb806b4d0 0x7f8fb806d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0036000 tx=0x7f8fd0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.738+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fc4058710 con 0x7f8fd40cf850 2024-03-06T07:24:23.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.946+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8fd404f330 con 0x7f8fd40cf850 2024-03-06T07:24:23.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.947+0000 7f8fc9ffb640 1 -- 172.21.15.122:0/595563400 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8fc4058010 con 0x7f8fd40cf850 2024-03-06T07:24:23.949 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:23.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fb806b4d0 msgr2=0x7f8fb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fb806b4d0 0x7f8fb806d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0036000 tx=0x7f8fd0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:23.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 msgr2=0x7f8fd40c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 0x7f8fd40c55a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c400 tx=0x7f8fc400c800 comp rx=0 tx=0).stop 2024-03-06T07:24:23.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 shutdown_connections 2024-03-06T07:24:23.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fb806b4d0 0x7f8fb806d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8fd40d00c0 0x7f8fd40c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd40cf850 0x7f8fd40c55a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 --2- 172.21.15.122:0/595563400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8fd406cf20 0x7f8fd40c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 >> 172.21.15.122:0/595563400 conn(0x7f8fd40b35a0 msgr2=0x7f8fd40d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 shutdown_connections 2024-03-06T07:24:23.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:23.949+0000 7f8fdb92c640 1 -- 172.21.15.122:0/595563400 wait complete. 2024-03-06T07:24:23.961 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:23.962 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:23.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:23.963 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:24:23.977 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:23.978 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42577sec preferred_lft 42577sec 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:23.979 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:23.980 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:24:23.980 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.980 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:23.980 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:23.980 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:24:23.980 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:23.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:23.981 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.045 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:24:24.101 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:24:24.101 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:24.101 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:24.101 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:24.101 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:24.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.102 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:24:24.102 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:24:24.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:24.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:24.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:24.102 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:24:24.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.196 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:24:24.211 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:24.211 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:24.211 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:24.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:24:24.278 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:24.279 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:24.279 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:24.334 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:24.334 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:24.334 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:24.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:24:24.455 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:24.511 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:24:24.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:24:24.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:24:24.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:24:24.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:24:24.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:24.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:24.514 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:24.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.514 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:24:24.566 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:24.566 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:24.608 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:24:24.634 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:24.690 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:24:24.690 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:24:24.690 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:24:24.690 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:24.693 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:24.693 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:24.696 INFO:tasks.ceph.mds.b.smithi169.stderr:2024-03-06T07:24:24.694+0000 7fbf83148640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:24.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.697 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:24:24.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:24:24.691+0000 7f3fe000c480 -1 init, newargv = 0x55d1473219c0 newargc=15 2024-03-06T07:24:24.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[58887]: starting ceph client 2024-03-06T07:24:24.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[58887]: starting fuse 2024-03-06T07:24:24.711 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:24:24.712 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:24.712 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:24:24.713 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:24.714 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:24.714 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:24.714 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:24:24.832 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.58887.asok 2024-03-06T07:24:24.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:24.842 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58887.asok 2024-03-06T07:24:24.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58887.asok status 2024-03-06T07:24:24.977 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "58887", 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:24:24.978 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 6297, 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 6297 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 357806537 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:24.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 357806537 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.6297 192.168.0.1:0/357806537", 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/357806537", 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 27, 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 57, 2024-03-06T07:24:24.980 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:24.981 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:24:24.981 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:24:24.981 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:24:24.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:24.986 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:24:24.986 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:25.001 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:24:25.002 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:25.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:25.002 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:25.069 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:24:25.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.211+0000 7fcae94b1640 1 Processor -- start 2024-03-06T07:24:25.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.211+0000 7fcae94b1640 1 -- start start 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae94b1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcae40b1e40 0x7fcae4168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae94b1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae40d2730 con 0x7fcae40d4860 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae94b1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae41618b0 con 0x7fcae41614d0 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae94b1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae40d4cf0 con 0x7fcae40b1e40 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcada7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcae40d4860 0x7fcae40d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44852/0 (socket says 172.21.15.122:44852) 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcada7fc640 1 -- 172.21.15.122:0/1306930147 learned_addr learned my addr 172.21.15.122:0/1306930147 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae37fe640 1 --2- 172.21.15.122:0/1306930147 >> v2:172.21.15.169:3300/0 conn(0x7fcae40b1e40 0x7fcae4168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae27fc640 1 -- 172.21.15.122:0/1306930147 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605968244 0 0) 0x7fcae40d2730 con 0x7fcae40d4860 2024-03-06T07:24:25.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae27fc640 1 -- 172.21.15.122:0/1306930147 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad0003520 con 0x7fcae40d4860 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae37fe640 1 -- 172.21.15.122:0/1306930147 >> v1:172.21.15.169:6789/0 conn(0x7fcae41614d0 legacy=0x7fcae40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae37fe640 1 -- 172.21.15.122:0/1306930147 >> v1:172.21.15.122:6789/0 conn(0x7fcae40d4860 legacy=0x7fcae40d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.212+0000 7fcae37fe640 1 -- 172.21.15.122:0/1306930147 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae40cf400 con 0x7fcae40b1e40 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.213+0000 7fcae37fe640 1 --2- 172.21.15.122:0/1306930147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40b1e40 0x7fcae4168b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcae40cf5d0 tx=0x7fcadc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b8be11aeac8f67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.213+0000 7fcae27fc640 1 -- 172.21.15.122:0/1306930147 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc039070 con 0x7fcae40b1e40 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.213+0000 7fcae27fc640 1 -- 172.21.15.122:0/1306930147 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcadc02cc50 con 0x7fcae40b1e40 2024-03-06T07:24:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.213+0000 7fcae27fc640 1 -- 172.21.15.122:0/1306930147 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc034440 con 0x7fcae40b1e40 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 -- 172.21.15.122:0/1306930147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40b1e40 msgr2=0x7fcae4168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1306930147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40b1e40 0x7fcae4168b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcae40cf5d0 tx=0x7fcadc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 -- 172.21.15.122:0/1306930147 shutdown_connections 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1306930147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40b1e40 0x7fcae4168b60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 -- 172.21.15.122:0/1306930147 >> 172.21.15.122:0/1306930147 conn(0x7fcae40d3d40 msgr2=0x7fcae40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 -- 172.21.15.122:0/1306930147 shutdown_connections 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.214+0000 7fcae94b1640 1 -- 172.21.15.122:0/1306930147 wait complete. 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.215+0000 7fcae94b1640 1 Processor -- start 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.215+0000 7fcae94b1640 1 -- start start 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 0x7fcae41557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40d4860 0x7fcae40c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcae41614d0 0x7fcae40b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae40cfc60 con 0x7fcae40b1e40 2024-03-06T07:24:25.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae40cf980 con 0x7fcae40d4860 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae40b0910 con 0x7fcae41614d0 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 0x7fcae41557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 0x7fcae41557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56616/0 (socket says 172.21.15.122:56616) 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 -- 172.21.15.122:0/1167202843 learned_addr learned my addr 172.21.15.122:0/1167202843 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcada7fc640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40d4860 0x7fcae40c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae37fe640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcae41614d0 0x7fcae40b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 -- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcae41614d0 msgr2=0x7fcae40b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcae41614d0 0x7fcae40b8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 -- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40d4860 msgr2=0x7fcae40c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40d4860 0x7fcae40c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 -- 172.21.15.122:0/1167202843 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae40c91d0 con 0x7fcae40b1e40 2024-03-06T07:24:25.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae2ffd640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 0x7fcae41557f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fcad400c400 tx=0x7fcad400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad40052d0 con 0x7fcae40b1e40 2024-03-06T07:24:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad4005490 con 0x7fcae40b1e40 2024-03-06T07:24:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae4155ef0 con 0x7fcae40b1e40 2024-03-06T07:24:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.216+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad4007260 con 0x7fcae40b1e40 2024-03-06T07:24:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.217+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae40c9750 con 0x7fcae40b1e40 2024-03-06T07:24:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.218+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae40671f0 con 0x7fcae40b1e40 2024-03-06T07:24:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.218+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcad4007400 con 0x7fcae40b1e40 2024-03-06T07:24:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.218+0000 7fcadbfff640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcac406b4d0 0x7fcac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.218+0000 7fcada7fc640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcac406b4d0 0x7fcac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.218+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcad40896d0 con 0x7fcae40b1e40 2024-03-06T07:24:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.219+0000 7fcada7fc640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcac406b4d0 0x7fcac406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcae40c8fc0 tx=0x7fcacc007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.223+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcae40671f0 con 0x7fcae40b1e40 2024-03-06T07:24:25.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.395+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fcae40b99f0 con 0x7fcae40b1e40 2024-03-06T07:24:25.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.397+0000 7fcadbfff640 1 -- 172.21.15.122:0/1167202843 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcad4057f00 con 0x7fcae40b1e40 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcac406b4d0 msgr2=0x7fcac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcac406b4d0 0x7fcac406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcae40c8fc0 tx=0x7fcacc007d30 comp rx=0 tx=0).stop 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 msgr2=0x7fcae41557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 0x7fcae41557f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fcad400c400 tx=0x7fcad400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 shutdown_connections 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcac406b4d0 0x7fcac406d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcae41614d0 0x7fcae40b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcae40d4860 0x7fcae40c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 --2- 172.21.15.122:0/1167202843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae40b1e40 0x7fcae41557f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 >> 172.21.15.122:0/1167202843 conn(0x7fcae40d3d40 msgr2=0x7fcae40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 shutdown_connections 2024-03-06T07:24:25.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.400+0000 7fcae94b1640 1 -- 172.21.15.122:0/1167202843 wait complete. 2024-03-06T07:24:25.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:24:25.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.512+0000 7fedb4e78640 1 Processor -- start 2024-03-06T07:24:25.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.512+0000 7fedb4e78640 1 -- start start 2024-03-06T07:24:25.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedb4e78640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fedb00d06e0 0x7fedb015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedb4e78640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb00d47f0 con 0x7fedb00d0ac0 2024-03-06T07:24:25.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedb4e78640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb00d1ab0 con 0x7fedb00cf200 2024-03-06T07:24:25.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedb4e78640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb00ced70 con 0x7fedb00d06e0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedadd9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fedb00d0ac0 0x7fedb00b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44862/0 (socket says 172.21.15.122:44862) 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedadd9b640 1 -- 172.21.15.122:0/4117199775 learned_addr learned my addr 172.21.15.122:0/4117199775 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.513+0000 7fedae59c640 1 --2- 172.21.15.122:0/4117199775 >> v2:172.21.15.169:3300/0 conn(0x7fedb00d06e0 0x7fedb015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.514+0000 7fedad59a640 1 -- 172.21.15.122:0/4117199775 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434900929 0 0) 0x7fedb00d47f0 con 0x7fedb00d0ac0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.514+0000 7fedad59a640 1 -- 172.21.15.122:0/4117199775 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed98003520 con 0x7fedb00d0ac0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.514+0000 7fedae59c640 1 -- 172.21.15.122:0/4117199775 >> v1:172.21.15.169:6789/0 conn(0x7fedb00cf200 legacy=0x7fedb00cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.514+0000 7fedae59c640 1 -- 172.21.15.122:0/4117199775 >> v1:172.21.15.122:6789/0 conn(0x7fedb00d0ac0 legacy=0x7fedb00b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.514+0000 7fedad59a640 1 -- 172.21.15.122:0/4117199775 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191602120 0 0) 0x7fed98003520 con 0x7fedb00d0ac0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.514+0000 7fedae59c640 1 -- 172.21.15.122:0/4117199775 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00d1d90 con 0x7fedb00d06e0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedae59c640 1 --2- 172.21.15.122:0/4117199775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d06e0 0x7fedb015fc30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feda0009d90 tx=0x7feda002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6441154848e0813 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedad59a640 1 -- 172.21.15.122:0/4117199775 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda0039070 con 0x7fedb00d06e0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedad59a640 1 -- 172.21.15.122:0/4117199775 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda002c940 con 0x7fedb00d06e0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedad59a640 1 -- 172.21.15.122:0/4117199775 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda4001670 con 0x7fedb00d06e0 2024-03-06T07:24:25.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 -- 172.21.15.122:0/4117199775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d06e0 msgr2=0x7fedb015fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 --2- 172.21.15.122:0/4117199775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d06e0 0x7fedb015fc30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feda0009d90 tx=0x7feda002be10 comp rx=0 tx=0).stop 2024-03-06T07:24:25.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 -- 172.21.15.122:0/4117199775 shutdown_connections 2024-03-06T07:24:25.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 --2- 172.21.15.122:0/4117199775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d06e0 0x7fedb015fc30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 -- 172.21.15.122:0/4117199775 >> 172.21.15.122:0/4117199775 conn(0x7fedb00d3cd0 msgr2=0x7fedb00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 -- 172.21.15.122:0/4117199775 shutdown_connections 2024-03-06T07:24:25.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.515+0000 7fedb4e78640 1 -- 172.21.15.122:0/4117199775 wait complete. 2024-03-06T07:24:25.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 Processor -- start 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 -- start start 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedb00cf200 0x7fedb00b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 0x7fedb00b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d0ac0 0x7fedb00ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb00b44d0 con 0x7fedb00d06e0 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb00ce990 con 0x7fedb00d0ac0 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedb4e78640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb00d0f30 con 0x7fedb00cf200 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedadd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 0x7fedb00b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.517+0000 7fedadd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 0x7fedb00b35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56626/0 (socket says 172.21.15.122:56626) 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 -- 172.21.15.122:0/3093467711 learned_addr learned my addr 172.21.15.122:0/3093467711 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedae59c640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedb00cf200 0x7fedb00b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedaed9d640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d0ac0 0x7fedb00ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 -- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedb00cf200 msgr2=0x7fedb00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedb00cf200 0x7fedb00b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 -- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d0ac0 msgr2=0x7fedb00ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d0ac0 0x7fedb00ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 -- 172.21.15.122:0/3093467711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00b0ed0 con 0x7fedb00d06e0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedadd9b640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 0x7fedb00b35e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feda40017e0 tx=0x7feda400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda4005ad0 con 0x7fedb00d06e0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda4005c90 con 0x7fedb00d06e0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb00b11b0 con 0x7fedb00d06e0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda40072b0 con 0x7fedb00d06e0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.518+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb00b3b20 con 0x7fedb00d06e0 2024-03-06T07:24:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.519+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb00a1c80 con 0x7fedb00d06e0 2024-03-06T07:24:25.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.519+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feda4007450 con 0x7fedb00d06e0 2024-03-06T07:24:25.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.520+0000 7fed9effd640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed9006b4d0 0x7fed9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.520+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feda4089ef0 con 0x7fedb00d06e0 2024-03-06T07:24:25.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.520+0000 7fedae59c640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed9006b4d0 0x7fed9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.520+0000 7fedae59c640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed9006b4d0 0x7fed9006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7feda0036000 tx=0x7feda0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.524+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feda4058730 con 0x7fedb00d06e0 2024-03-06T07:24:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.695+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7fedb0068ff0 con 0x7fedb00d06e0 2024-03-06T07:24:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.698+0000 7fed9effd640 1 -- 172.21.15.122:0/3093467711 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7feda405c7d0 con 0x7fedb00d06e0 2024-03-06T07:24:25.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed9006b4d0 msgr2=0x7fed9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed9006b4d0 0x7fed9006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7feda0036000 tx=0x7feda0030000 comp rx=0 tx=0).stop 2024-03-06T07:24:25.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 msgr2=0x7fedb00b35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 0x7fedb00b35e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feda40017e0 tx=0x7feda400cbe0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 shutdown_connections 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed9006b4d0 0x7fed9006d990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb00d0ac0 0x7fedb00ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedb00d06e0 0x7fedb00b35e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 --2- 172.21.15.122:0/3093467711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedb00cf200 0x7fedb00b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 >> 172.21.15.122:0/3093467711 conn(0x7fedb00d3cd0 msgr2=0x7fedb00b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 shutdown_connections 2024-03-06T07:24:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:25.700+0000 7fedb4e78640 1 -- 172.21.15.122:0/3093467711 wait complete. 2024-03-06T07:24:35.713 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:35.713 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:35.732 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:24:35.732 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:24:35.732 DEBUG:teuthology.orchestra.run.smithi122:> import uuid 2024-03-06T07:24:35.732 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> n = 50 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = False 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> if 0 > 0: 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = True 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> dpath = os.path.dirname(path) 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> try: 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(n): 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:35.733 DEBUG:teuthology.orchestra.run.smithi122:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> f.write(f"{i}") 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> f.flush() 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(f.fileno()) 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> os.unlink(fpath) 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> if create_hard_links: 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> for j in range(0): 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:24:35.734 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:24:35.735 DEBUG:teuthology.orchestra.run.smithi122:> finally: 2024-03-06T07:24:35.735 DEBUG:teuthology.orchestra.run.smithi122:> os.close(dirfd) 2024-03-06T07:24:35.735 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:24:35.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:35.960 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:35.960 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:36.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:36.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.152+0000 7f4be2cdf640 1 Processor -- start 2024-03-06T07:24:36.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.152+0000 7f4be2cdf640 1 -- start start 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be2cdf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4bdc0d2710 0x7f4bdc0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be2cdf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0b3430 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be2cdf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0b3710 con 0x7f4bdc0d22b0 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be2cdf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc0b31e0 con 0x7f4bdc0d2710 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be0a7b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4bdc15dfe0 0x7f4bdc0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59230/0 (socket says 172.21.15.122:59230) 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be0a7b640 1 -- 172.21.15.122:0/2139467328 learned_addr learned my addr 172.21.15.122:0/2139467328 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.153+0000 7f4be127c640 1 --2- 172.21.15.122:0/2139467328 >> v2:172.21.15.169:3300/0 conn(0x7f4bdc0d2710 0x7f4bdc0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.154+0000 7f4bdb7fe640 1 -- 172.21.15.122:0/2139467328 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708957050 0 0) 0x7f4bdc0b3430 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.154+0000 7f4bdb7fe640 1 -- 172.21.15.122:0/2139467328 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc4003520 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.154+0000 7f4be127c640 1 -- 172.21.15.122:0/2139467328 >> v1:172.21.15.169:6789/0 conn(0x7f4bdc0d22b0 legacy=0x7f4bdc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.154+0000 7f4be127c640 1 -- 172.21.15.122:0/2139467328 >> v1:172.21.15.122:6789/0 conn(0x7f4bdc15dfe0 legacy=0x7f4bdc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.154+0000 7f4be127c640 1 -- 172.21.15.122:0/2139467328 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0b39f0 con 0x7f4bdc0d2710 2024-03-06T07:24:36.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be127c640 1 --2- 172.21.15.122:0/2139467328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc0b4900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009dc0 tx=0x7f4bd402bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=493b5eba8ca42320 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4bdb7fe640 1 -- 172.21.15.122:0/2139467328 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4039070 con 0x7f4bdc0d2710 2024-03-06T07:24:36.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4bdb7fe640 1 -- 172.21.15.122:0/2139467328 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd402ccd0 con 0x7f4bdc0d2710 2024-03-06T07:24:36.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4bdb7fe640 1 -- 172.21.15.122:0/2139467328 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4034640 con 0x7f4bdc0d2710 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2139467328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 msgr2=0x7f4bdc0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2139467328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc0b4900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009dc0 tx=0x7f4bd402bf20 comp rx=0 tx=0).stop 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2139467328 shutdown_connections 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2139467328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc0b4900 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2139467328 >> 172.21.15.122:0/2139467328 conn(0x7f4bdc0d3cd0 msgr2=0x7f4bdc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2139467328 shutdown_connections 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.155+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2139467328 wait complete. 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.156+0000 7f4be2cdf640 1 Processor -- start 2024-03-06T07:24:36.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.156+0000 7f4be2cdf640 1 -- start start 2024-03-06T07:24:36.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be2cdf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bdc0d22b0 0x7f4bdc14d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be2cdf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be2cdf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 0x7f4bdc1507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be2cdf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cfa00 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be2cdf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0b3d60 con 0x7f4bdc0d2710 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be2cdf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0d1c70 con 0x7f4bdc0d22b0 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be127c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 0x7f4bdc1507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be127c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 0x7f4bdc1507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45624/0 (socket says 172.21.15.122:45624) 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.157+0000 7f4be127c640 1 -- 172.21.15.122:0/2454860744 learned_addr learned my addr 172.21.15.122:0/2454860744 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be0a7b640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bdc0d22b0 0x7f4bdc14d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4bdbfff640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be127c640 1 -- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bdc0d22b0 msgr2=0x7f4bdc14d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be127c640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bdc0d22b0 0x7f4bdc14d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be127c640 1 -- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 msgr2=0x7f4bdc150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be127c640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be127c640 1 -- 172.21.15.122:0/2454860744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0cc8a0 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be127c640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 0x7f4bdc1507b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009ee0 tx=0x7f4bd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4044020 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd403a040 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc0ccb10 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4039050 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.158+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc0cd0d0 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.159+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc04f330 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.160+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bd4003b90 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.160+0000 7f4bd97fa640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.160+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bd40ac720 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.160+0000 7f4be0a7b640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.160+0000 7f4be0a7b640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0011e0 tx=0x7f4bcc007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.164+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bd404e020 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.336+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bdc0747a0 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.337+0000 7f4bd97fa640 1 -- 172.21.15.122:0/2454860744 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7f4bd407afe0 con 0x7f4bdc15dfe0 2024-03-06T07:24:36.340 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:36.340 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:36.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 msgr2=0x7f4bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0011e0 tx=0x7f4bcc007d30 comp rx=0 tx=0).stop 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 msgr2=0x7f4bdc1507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 0x7f4bdc1507b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009ee0 tx=0x7f4bd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 shutdown_connections 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bdc15dfe0 0x7f4bdc1507b0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bdc0d2710 0x7f4bdc150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 --2- 172.21.15.122:0/2454860744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bdc0d22b0 0x7f4bdc14d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 >> 172.21.15.122:0/2454860744 conn(0x7f4bdc0d3cd0 msgr2=0x7f4bdc15d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 shutdown_connections 2024-03-06T07:24:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.339+0000 7f4be2cdf640 1 -- 172.21.15.122:0/2454860744 wait complete. 2024-03-06T07:24:36.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:16.591753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:17.332050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:36.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:36.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.454+0000 7f4f7a756640 1 Processor -- start 2024-03-06T07:24:36.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.454+0000 7f4f7a756640 1 -- start start 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f7a756640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4f741586e0 0x7f4f740aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f7a756640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4f740aef80 0x7f4f740cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f7a756640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f74158ac0 con 0x7f4f740cffd0 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f7a756640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7415b2a0 con 0x7f4f741586e0 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f7a756640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7415b540 con 0x7f4f740aef80 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f73fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4f741586e0 0x7f4f740aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f78cf3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4f740cffd0 0x7f4f7406da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59244/0 (socket says 172.21.15.122:59244) 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f78cf3640 1 -- 172.21.15.122:0/2997455546 learned_addr learned my addr 172.21.15.122:0/2997455546 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f737fe640 1 --2- 172.21.15.122:0/2997455546 >> v2:172.21.15.169:3300/0 conn(0x7f4f740aef80 0x7f4f740cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f73fff640 1 -- 172.21.15.122:0/2997455546 >> v2:172.21.15.169:3300/0 conn(0x7f4f740aef80 msgr2=0x7f4f740cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f73fff640 1 --2- 172.21.15.122:0/2997455546 >> v2:172.21.15.169:3300/0 conn(0x7f4f740aef80 0x7f4f740cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f73fff640 1 -- 172.21.15.122:0/2997455546 >> v1:172.21.15.122:6789/0 conn(0x7f4f740cffd0 legacy=0x7f4f7406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f73fff640 1 -- 172.21.15.122:0/2997455546 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7415b820 con 0x7f4f741586e0 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.455+0000 7f4f72ffd640 1 -- 172.21.15.122:0/2997455546 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247973146 0 0) 0x7f4f74158ac0 con 0x7f4f740cffd0 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.456+0000 7f4f73fff640 1 --2- 172.21.15.122:0/2997455546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f741586e0 0x7f4f740aea40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4f74072e30 tx=0x7f4f6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9311756e725b36c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.456+0000 7f4f72ffd640 1 -- 172.21.15.122:0/2997455546 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64035020 con 0x7f4f741586e0 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.456+0000 7f4f72ffd640 1 -- 172.21.15.122:0/2997455546 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f64031830 con 0x7f4f741586e0 2024-03-06T07:24:36.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.456+0000 7f4f72ffd640 1 -- 172.21.15.122:0/2997455546 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64031c00 con 0x7f4f741586e0 2024-03-06T07:24:36.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 -- 172.21.15.122:0/2997455546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f741586e0 msgr2=0x7f4f740aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 --2- 172.21.15.122:0/2997455546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f741586e0 0x7f4f740aea40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4f74072e30 tx=0x7f4f6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 -- 172.21.15.122:0/2997455546 shutdown_connections 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 --2- 172.21.15.122:0/2997455546 >> v2:172.21.15.169:3300/0 conn(0x7f4f740aef80 0x7f4f740cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 --2- 172.21.15.122:0/2997455546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f741586e0 0x7f4f740aea40 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 -- 172.21.15.122:0/2997455546 >> 172.21.15.122:0/2997455546 conn(0x7f4f7406b190 msgr2=0x7f4f7406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 -- 172.21.15.122:0/2997455546 shutdown_connections 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.457+0000 7f4f7a756640 1 -- 172.21.15.122:0/2997455546 wait complete. 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.458+0000 7f4f7a756640 1 Processor -- start 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.458+0000 7f4f7a756640 1 -- start start 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f740aef80 0x7f4f7414e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 0x7f4f74156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f741586e0 0x7f4f74157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f74071350 con 0x7f4f740cffd0 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7415bde0 con 0x7f4f740aef80 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740714d0 con 0x7f4f741586e0 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 0x7f4f74156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 0x7f4f74156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45648/0 (socket says 172.21.15.122:45648) 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 -- 172.21.15.122:0/3098112048 learned_addr learned my addr 172.21.15.122:0/3098112048 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f78cf3640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f741586e0 0x7f4f74157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f73fff640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f740aef80 0x7f4f7414e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 -- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f741586e0 msgr2=0x7f4f74157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f741586e0 0x7f4f74157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 -- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f740aef80 msgr2=0x7f4f7414e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f740aef80 0x7f4f7414e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 -- 172.21.15.122:0/3098112048 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740aae90 con 0x7f4f740cffd0 2024-03-06T07:24:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f737fe640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 0x7f4f74156cc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800e9b0 tx=0x7f4f6800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68017070 con 0x7f4f740cffd0 2024-03-06T07:24:36.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6800c9a0 con 0x7f4f740cffd0 2024-03-06T07:24:36.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6800cd70 con 0x7f4f740cffd0 2024-03-06T07:24:36.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f740ab170 con 0x7f4f740cffd0 2024-03-06T07:24:36.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.459+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7414e680 con 0x7f4f740cffd0 2024-03-06T07:24:36.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.460+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f740b4820 con 0x7f4f740cffd0 2024-03-06T07:24:36.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.461+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f6801a070 con 0x7f4f740cffd0 2024-03-06T07:24:36.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.461+0000 7f4f70ff9640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f6006b4d0 0x7f4f6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.461+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f6808a470 con 0x7f4f740cffd0 2024-03-06T07:24:36.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.461+0000 7f4f73fff640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f6006b4d0 0x7f4f6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.462+0000 7f4f73fff640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f6006b4d0 0x7f4f6006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f64036000 tx=0x7f4f6403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.466+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f740b4820 con 0x7f4f740cffd0 2024-03-06T07:24:36.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.635+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f740732b0 con 0x7f4f740cffd0 2024-03-06T07:24:36.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.636+0000 7f4f70ff9640 1 -- 172.21.15.122:0/3098112048 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f68058d30 con 0x7f4f740cffd0 2024-03-06T07:24:36.638 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:36.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f6006b4d0 msgr2=0x7f4f6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f6006b4d0 0x7f4f6006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f64036000 tx=0x7f4f6403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 msgr2=0x7f4f74156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 0x7f4f74156cc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800e9b0 tx=0x7f4f6800ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 shutdown_connections 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f6006b4d0 0x7f4f6006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f741586e0 0x7f4f74157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f740cffd0 0x7f4f74156cc0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 --2- 172.21.15.122:0/3098112048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f740aef80 0x7f4f7414e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 >> 172.21.15.122:0/3098112048 conn(0x7f4f7406b190 msgr2=0x7f4f74158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 shutdown_connections 2024-03-06T07:24:36.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.638+0000 7f4f7a756640 1 -- 172.21.15.122:0/3098112048 wait complete. 2024-03-06T07:24:36.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:36.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.752+0000 7fb1e3ecf640 1 Processor -- start 2024-03-06T07:24:36.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.752+0000 7fb1e3ecf640 1 -- start start 2024-03-06T07:24:36.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e3ecf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc1586e0 0x7fb1dc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e3ecf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb1dc0aef80 0x7fb1dc0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e3ecf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc0d03f0 con 0x7fb1dc0cffd0 2024-03-06T07:24:36.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e3ecf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc158ac0 con 0x7fb1dc0aef80 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e3ecf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc15b2a0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e246c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb1dc0cffd0 0x7fb1dc06da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59254/0 (socket says 172.21.15.122:59254) 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e246c640 1 -- 172.21.15.122:0/905683247 learned_addr learned my addr 172.21.15.122:0/905683247 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e1c6b640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc1586e0 0x7fb1dc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e146a640 1 --2- 172.21.15.122:0/905683247 >> v2:172.21.15.169:3300/0 conn(0x7fb1dc0aef80 0x7fb1dc0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e0c69640 1 -- 172.21.15.122:0/905683247 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978396793 0 0) 0x7fb1dc0d03f0 con 0x7fb1dc0cffd0 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e0c69640 1 -- 172.21.15.122:0/905683247 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8003520 con 0x7fb1dc0cffd0 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e146a640 1 -- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc1586e0 msgr2=0x7fb1dc0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e146a640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc1586e0 0x7fb1dc0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e146a640 1 -- 172.21.15.122:0/905683247 >> v1:172.21.15.122:6789/0 conn(0x7fb1dc0cffd0 legacy=0x7fb1dc06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e146a640 1 -- 172.21.15.122:0/905683247 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc15b540 con 0x7fb1dc0aef80 2024-03-06T07:24:36.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.753+0000 7fb1e1c6b640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc1586e0 0x7fb1dc0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.754+0000 7fb1e146a640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0aef80 0x7fb1dc0cfa90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0097a0 tx=0x7fb1cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d546574aa503c64a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.754+0000 7fb1e0c69640 1 -- 172.21.15.122:0/905683247 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc035020 con 0x7fb1dc0aef80 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.754+0000 7fb1e0c69640 1 -- 172.21.15.122:0/905683247 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc031830 con 0x7fb1dc0aef80 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.754+0000 7fb1e0c69640 1 -- 172.21.15.122:0/905683247 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc031be0 con 0x7fb1dc0aef80 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0aef80 msgr2=0x7fb1dc0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0aef80 0x7fb1dc0cfa90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0097a0 tx=0x7fb1cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/905683247 shutdown_connections 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0aef80 0x7fb1dc0cfa90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/905683247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc1586e0 0x7fb1dc0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/905683247 >> 172.21.15.122:0/905683247 conn(0x7fb1dc06b190 msgr2=0x7fb1dc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/905683247 shutdown_connections 2024-03-06T07:24:36.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.755+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/905683247 wait complete. 2024-03-06T07:24:36.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.756+0000 7fb1e3ecf640 1 Processor -- start 2024-03-06T07:24:36.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.756+0000 7fb1e3ecf640 1 -- start start 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc0aef80 0x7fb1dc146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0cffd0 0x7fb1dc0b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 0x7fb1dc141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc15bde0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc15bb00 con 0x7fb1dc0cffd0 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc071350 con 0x7fb1dc0aef80 2024-03-06T07:24:36.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 0x7fb1dc141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 0x7fb1dc141050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45650/0 (socket says 172.21.15.122:45650) 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 -- 172.21.15.122:0/1600525227 learned_addr learned my addr 172.21.15.122:0/1600525227 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e1c6b640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc0aef80 0x7fb1dc146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e146a640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0cffd0 0x7fb1dc0b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 -- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc0aef80 msgr2=0x7fb1dc146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc0aef80 0x7fb1dc146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 -- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0cffd0 msgr2=0x7fb1dc0b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0cffd0 0x7fb1dc0b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 -- 172.21.15.122:0/1600525227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc14e1d0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e246c640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 0x7fb1dc141050 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400cb40 tx=0x7fb1d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4017070 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d400aca0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc14d130 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4012620 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.757+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc14d6f0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.758+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc0732b0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.759+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1d401a070 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.759+0000 7fb1d27fc640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.759+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1d4089fe0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.759+0000 7fb1e1c6b640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.760+0000 7fb1e1c6b640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1c006b4d0 0x7fb1c006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc14bb80 tx=0x7fb1d8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.764+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1d4058810 con 0x7fb1dc1586e0 2024-03-06T07:24:36.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.935+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1dc068ff0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.935+0000 7fb1d27fc640 1 -- 172.21.15.122:0/1600525227 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1d405c8b0 con 0x7fb1dc1586e0 2024-03-06T07:24:36.937 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:24:36.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1c006b4d0 msgr2=0x7fb1c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1c006b4d0 0x7fb1c006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc14bb80 tx=0x7fb1d8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 msgr2=0x7fb1dc141050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 0x7fb1dc141050 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400cb40 tx=0x7fb1d400a060 comp rx=0 tx=0).stop 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 shutdown_connections 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1dc1586e0 0x7fb1dc141050 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1dc0cffd0 0x7fb1dc0b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 --2- 172.21.15.122:0/1600525227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1dc0aef80 0x7fb1dc146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 >> 172.21.15.122:0/1600525227 conn(0x7fb1dc06b190 msgr2=0x7fb1dc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 shutdown_connections 2024-03-06T07:24:36.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:36.937+0000 7fb1e3ecf640 1 -- 172.21.15.122:0/1600525227 wait complete. 2024-03-06T07:24:36.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.050+0000 7fd8e0b8c640 1 Processor -- start 2024-03-06T07:24:37.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.050+0000 7fd8e0b8c640 1 -- start start 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8e0b8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8e0b8c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd8dc066d90 0x7fd8dc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8e0b8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc06b710 con 0x7fd8dc0b5410 2024-03-06T07:24:37.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8e0b8c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8dc06b150 con 0x7fd8dc066d90 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8e0b8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc06b430 con 0x7fd8dc0cf9a0 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8d9d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd8dc066d90 0x7fd8dc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8d9d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd8dc066d90 0x7fd8dc0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45658/0 (socket says 172.21.15.122:45658) 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8d9d9b640 1 -- 172.21.15.122:0/246772728 learned_addr learned my addr 172.21.15.122:0/246772728 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.051+0000 7fd8da59c640 1 --2- 172.21.15.122:0/246772728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d9d9b640 1 -- 172.21.15.122:0/246772728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0cf9a0 msgr2=0x7fd8dc0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d9d9b640 1 --2- 172.21.15.122:0/246772728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d9d9b640 1 -- 172.21.15.122:0/246772728 >> v1:172.21.15.169:6789/0 conn(0x7fd8dc0b5410 legacy=0x7fd8dc072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d9d9b640 1 -- 172.21.15.122:0/246772728 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0b0970 con 0x7fd8dc066d90 2024-03-06T07:24:37.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d9d9b640 1 --2- 172.21.15.122:0/246772728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc0b4ed0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009870 tx=0x7fd8d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e44dab77bd3803e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d959a640 1 -- 172.21.15.122:0/246772728 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0034070 con 0x7fd8dc066d90 2024-03-06T07:24:37.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d959a640 1 -- 172.21.15.122:0/246772728 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d0030840 con 0x7fd8dc066d90 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.052+0000 7fd8d959a640 1 -- 172.21.15.122:0/246772728 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0030c10 con 0x7fd8dc066d90 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/246772728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 msgr2=0x7fd8dc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/246772728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc0b4ed0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009870 tx=0x7fd8d002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/246772728 shutdown_connections 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/246772728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc0b4ed0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/246772728 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/246772728 >> 172.21.15.122:0/246772728 conn(0x7fd8dc0b1d70 msgr2=0x7fd8dc0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/246772728 shutdown_connections 2024-03-06T07:24:37.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.053+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/246772728 wait complete. 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.054+0000 7fd8e0b8c640 1 Processor -- start 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.054+0000 7fd8e0b8c640 1 -- start start 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8e0b8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8e0b8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0b5410 0x7fd8dc152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8e0b8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8e0b8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0b6170 con 0x7fd8dc066d90 2024-03-06T07:24:37.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8e0b8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0b5ff0 con 0x7fd8dc0cf9a0 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8e0b8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0cf050 con 0x7fd8dc0b5410 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8da59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8da59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc15b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45672/0 (socket says 172.21.15.122:45672) 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8da59c640 1 -- 172.21.15.122:0/2299767704 learned_addr learned my addr 172.21.15.122:0/2299767704 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8d9d9b640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0b5410 0x7fd8dc152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.055+0000 7fd8dad9d640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8da59c640 1 -- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0b5410 msgr2=0x7fd8dc152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8da59c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0b5410 0x7fd8dc152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8da59c640 1 -- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc0cf9a0 msgr2=0x7fd8dc0cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8da59c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8da59c640 1 -- 172.21.15.122:0/2299767704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0c74d0 con 0x7fd8dc066d90 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8da59c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc15b730 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e870 tx=0x7fd8cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc017070 con 0x7fd8dc066d90 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc00c940 con 0x7fd8dc066d90 2024-03-06T07:24:37.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc0cca60 con 0x7fd8dc066d90 2024-03-06T07:24:37.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00cd30 con 0x7fd8dc066d90 2024-03-06T07:24:37.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.056+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc0ca2c0 con 0x7fd8dc066d90 2024-03-06T07:24:37.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.057+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc09e3d0 con 0x7fd8dc066d90 2024-03-06T07:24:37.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.057+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8cc01a070 con 0x7fd8dc066d90 2024-03-06T07:24:37.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.058+0000 7fd8caffd640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8bc06b500 0x7fd8bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.058+0000 7fd8d9d9b640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8bc06b500 0x7fd8bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.058+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8cc089f30 con 0x7fd8dc066d90 2024-03-06T07:24:37.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.058+0000 7fd8d9d9b640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8bc06b500 0x7fd8bc06d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0036000 tx=0x7fd8d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.062+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8cc0587f0 con 0x7fd8dc066d90 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.233+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8dc0ca500 con 0x7fd8dc066d90 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.233+0000 7fd8caffd640 1 -- 172.21.15.122:0/2299767704 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd8cc05c890 con 0x7fd8dc066d90 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:24:37.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8bc06b500 msgr2=0x7fd8bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8bc06b500 0x7fd8bc06d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0036000 tx=0x7fd8d0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 msgr2=0x7fd8dc15b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc15b730 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e870 tx=0x7fd8cc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 shutdown_connections 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8bc06b500 0x7fd8bc06d9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc0cf9a0 0x7fd8dc0cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8dc0b5410 0x7fd8dc152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 --2- 172.21.15.122:0/2299767704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc066d90 0x7fd8dc15b730 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 >> 172.21.15.122:0/2299767704 conn(0x7fd8dc0b1d70 msgr2=0x7fd8dc0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 shutdown_connections 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.235+0000 7fd8e0b8c640 1 -- 172.21.15.122:0/2299767704 wait complete. 2024-03-06T07:24:37.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:37.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.346+0000 7f50b0513640 1 Processor -- start 2024-03-06T07:24:37.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.346+0000 7f50b0513640 1 -- start start 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.347+0000 7f50b0513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.347+0000 7f50b0513640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f50a8070e10 0x7f50a80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.347+0000 7f50b0513640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a806d370 con 0x7f50a81444c0 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.347+0000 7f50b0513640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a81448a0 con 0x7f50a8070e10 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.347+0000 7f50b0513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a8071260 con 0x7f50a815b2d0 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f50a8070e10 0x7f50a80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f50a8070e10 0x7f50a80cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45680/0 (socket says 172.21.15.122:45680) 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 -- 172.21.15.122:0/994622601 learned_addr learned my addr 172.21.15.122:0/994622601 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50adaae640 1 --2- 172.21.15.122:0/994622601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 -- 172.21.15.122:0/994622601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 msgr2=0x7f50a80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 --2- 172.21.15.122:0/994622601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 -- 172.21.15.122:0/994622601 >> v1:172.21.15.169:6789/0 conn(0x7f50a81444c0 legacy=0x7f50a806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.348+0000 7f50aeab0640 1 -- 172.21.15.122:0/994622601 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a806d650 con 0x7f50a8070e10 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50aeab0640 1 --2- 172.21.15.122:0/994622601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a8070e10 0x7f50a80cda60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f50a0009e50 tx=0x7f50a002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c3fb4fc8420629 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50ad2ad640 1 -- 172.21.15.122:0/994622601 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a002cb00 con 0x7f50a8070e10 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50ad2ad640 1 -- 172.21.15.122:0/994622601 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a002cca0 con 0x7f50a8070e10 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50ad2ad640 1 -- 172.21.15.122:0/994622601 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0034610 con 0x7f50a8070e10 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 -- 172.21.15.122:0/994622601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a8070e10 msgr2=0x7f50a80cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 --2- 172.21.15.122:0/994622601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a8070e10 0x7f50a80cda60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f50a0009e50 tx=0x7f50a002bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 -- 172.21.15.122:0/994622601 shutdown_connections 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 --2- 172.21.15.122:0/994622601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a8070e10 0x7f50a80cda60 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 --2- 172.21.15.122:0/994622601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 -- 172.21.15.122:0/994622601 >> 172.21.15.122:0/994622601 conn(0x7f50a80b1c80 msgr2=0x7f50a80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 -- 172.21.15.122:0/994622601 shutdown_connections 2024-03-06T07:24:37.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.349+0000 7f50b0513640 1 -- 172.21.15.122:0/994622601 wait complete. 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.350+0000 7f50b0513640 1 Processor -- start 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 -- start start 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50a8070e10 0x7f50a80b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 0x7f50a80cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a806dc10 con 0x7f50a81444c0 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a806d110 con 0x7f50a8070e10 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50b0513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a806def0 con 0x7f50a815b2d0 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50adaae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 0x7f50a80cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50adaae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 0x7f50a80cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45696/0 (socket says 172.21.15.122:45696) 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50adaae640 1 -- 172.21.15.122:0/2871060634 learned_addr learned my addr 172.21.15.122:0/2871060634 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50ae2af640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50a8070e10 0x7f50a80b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.351+0000 7f50aeab0640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50adaae640 1 -- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 msgr2=0x7f50a80a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50adaae640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50adaae640 1 -- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50a8070e10 msgr2=0x7f50a80b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50adaae640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50a8070e10 0x7f50a80b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50adaae640 1 -- 172.21.15.122:0/2871060634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a80c4dc0 con 0x7f50a81444c0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50adaae640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 0x7f50a80cb150 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f509800e870 tx=0x7f509800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098017070 con 0x7f50a81444c0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509800c940 con 0x7f50a81444c0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a80c50a0 con 0x7f50a81444c0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509800cd30 con 0x7f50a81444c0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.352+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a80c5660 con 0x7f50a81444c0 2024-03-06T07:24:37.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.353+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a804f330 con 0x7f50a81444c0 2024-03-06T07:24:37.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.353+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f509801a070 con 0x7f50a81444c0 2024-03-06T07:24:37.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.354+0000 7f509effd640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f508c06b4d0 0x7f508c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.354+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f509808a460 con 0x7f50a81444c0 2024-03-06T07:24:37.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.354+0000 7f50ae2af640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f508c06b4d0 0x7f508c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.354+0000 7f50ae2af640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f508c06b4d0 0x7f508c06d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f50a4001780 tx=0x7f50a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.358+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5098058c00 con 0x7f50a81444c0 2024-03-06T07:24:37.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.542+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f50a806cc10 con 0x7f50a81444c0 2024-03-06T07:24:37.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.543+0000 7f509effd640 1 -- 172.21.15.122:0/2871060634 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7f509805cca0 con 0x7f50a81444c0 2024-03-06T07:24:37.545 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f508c06b4d0 msgr2=0x7f508c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f508c06b4d0 0x7f508c06d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f50a4001780 tx=0x7f50a4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 msgr2=0x7f50a80cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 0x7f50a80cb150 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f509800e870 tx=0x7f509800ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 shutdown_connections 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f508c06b4d0 0x7f508c06d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50a815b2d0 0x7f50a80a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a81444c0 0x7f50a80cb150 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 --2- 172.21.15.122:0/2871060634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50a8070e10 0x7f50a80b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 >> 172.21.15.122:0/2871060634 conn(0x7f50a80b1c80 msgr2=0x7f50a815e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 shutdown_connections 2024-03-06T07:24:37.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.545+0000 7f50b0513640 1 -- 172.21.15.122:0/2871060634 wait complete. 2024-03-06T07:24:37.557 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:37.557 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:37.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:37.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.711+0000 7fb649381640 1 Processor -- start 2024-03-06T07:24:37.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.711+0000 7fb649381640 1 -- start start 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb649381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb6440cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb649381640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440cc980 con 0x7fb6441424f0 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb649381640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440ccc60 con 0x7fb64406da40 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb649381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cd220 con 0x7fb6440cd7a0 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb642ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb6441424f0 0x7fb64406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59262/0 (socket says 172.21.15.122:59262) 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb642ffd640 1 -- 172.21.15.122:0/1975738615 learned_addr learned my addr 172.21.15.122:0/1975738615 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb6437fe640 1 --2- 172.21.15.122:0/1975738615 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb6440cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508504728 0 0) 0x7fb6440cc980 con 0x7fb6441424f0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c003520 con 0x7fb6441424f0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342227387 0 0) 0x7fb6440ccc60 con 0x7fb64406da40 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6440cc980 con 0x7fb64406da40 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb6437fe640 1 -- 172.21.15.122:0/1975738615 >> v1:172.21.15.169:6789/0 conn(0x7fb64406da40 legacy=0x7fb644071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb6437fe640 1 -- 172.21.15.122:0/1975738615 >> v1:172.21.15.122:6789/0 conn(0x7fb6441424f0 legacy=0x7fb64406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.712+0000 7fb6437fe640 1 -- 172.21.15.122:0/1975738615 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440ccf40 con 0x7fb6440cd7a0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.713+0000 7fb6437fe640 1 --2- 172.21.15.122:0/1975738615 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb6440cc2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0097d0 tx=0x7fb63c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8de0172aaed6120 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.713+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c035020 con 0x7fb6440cd7a0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.713+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c031890 con 0x7fb6440cd7a0 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.713+0000 7fb641ffb640 1 -- 172.21.15.122:0/1975738615 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c031c60 con 0x7fb6440cd7a0 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 -- 172.21.15.122:0/1975738615 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 msgr2=0x7fb6440cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 --2- 172.21.15.122:0/1975738615 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb6440cc2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0097d0 tx=0x7fb63c02b930 comp rx=0 tx=0).stop 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 -- 172.21.15.122:0/1975738615 shutdown_connections 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 --2- 172.21.15.122:0/1975738615 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb6440cc2f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 -- 172.21.15.122:0/1975738615 >> 172.21.15.122:0/1975738615 conn(0x7fb6440cf8a0 msgr2=0x7fb6440cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 -- 172.21.15.122:0/1975738615 shutdown_connections 2024-03-06T07:24:37.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.714+0000 7fb649381640 1 -- 172.21.15.122:0/1975738615 wait complete. 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.715+0000 7fb649381640 1 Processor -- start 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.715+0000 7fb649381640 1 -- start start 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb649381640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb64406da40 0x7fb6440aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb649381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb644157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb649381640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 0x7fb644136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb649381640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64415d070 con 0x7fb6441424f0 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb649381640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64415cef0 con 0x7fb64406da40 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb649381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64415d1f0 con 0x7fb6440cd7a0 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 0x7fb644136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 0x7fb644136060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45702/0 (socket says 172.21.15.122:45702) 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 -- 172.21.15.122:0/2585979387 learned_addr learned my addr 172.21.15.122:0/2585979387 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:37.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6427fc640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb644157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb642ffd640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb64406da40 0x7fb6440aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 -- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 msgr2=0x7fb644157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb644157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 -- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb64406da40 msgr2=0x7fb6440aa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb64406da40 0x7fb6440aa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 -- 172.21.15.122:0/2585979387 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6441365a0 con 0x7fb6441424f0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.716+0000 7fb6437fe640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 0x7fb644136060 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0097d0 tx=0x7fb63c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.717+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c035040 con 0x7fb6441424f0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.717+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb644157f00 con 0x7fb6441424f0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.717+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c030ca0 con 0x7fb6441424f0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.717+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c006e10 con 0x7fb6441424f0 2024-03-06T07:24:37.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.717+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6441584c0 con 0x7fb6441424f0 2024-03-06T07:24:37.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.718+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64404f330 con 0x7fb6441424f0 2024-03-06T07:24:37.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.718+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb63c038020 con 0x7fb6441424f0 2024-03-06T07:24:37.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.719+0000 7fb62b7fe640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63006b4d0 0x7fb63006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.719+0000 7fb642ffd640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63006b4d0 0x7fb63006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.719+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb63c0ac810 con 0x7fb6441424f0 2024-03-06T07:24:37.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.719+0000 7fb642ffd640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63006b4d0 0x7fb63006d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb6340012e0 tx=0x7fb634001a10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.723+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb63c02d040 con 0x7fb6441424f0 2024-03-06T07:24:37.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.894+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6440d0620 con 0x7fb6441424f0 2024-03-06T07:24:37.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.895+0000 7fb62b7fe640 1 -- 172.21.15.122:0/2585979387 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7fb63c07af30 con 0x7fb6441424f0 2024-03-06T07:24:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:37.897 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63006b4d0 msgr2=0x7fb63006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63006b4d0 0x7fb63006d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb6340012e0 tx=0x7fb634001a10 comp rx=0 tx=0).stop 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 msgr2=0x7fb644136060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 0x7fb644136060 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0097d0 tx=0x7fb63c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 shutdown_connections 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63006b4d0 0x7fb63006d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6441424f0 0x7fb644136060 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6440cd7a0 0x7fb644157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 --2- 172.21.15.122:0/2585979387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb64406da40 0x7fb6440aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 >> 172.21.15.122:0/2585979387 conn(0x7fb6440cf8a0 msgr2=0x7fb64406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 shutdown_connections 2024-03-06T07:24:37.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:37.897+0000 7fb649381640 1 -- 172.21.15.122:0/2585979387 wait complete. 2024-03-06T07:24:37.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:16.591753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:17.332050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:37.910 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 Processor -- start 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 -- start start 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f53e40b4970 0x7f53e406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53e406d5c0 0x7f53e40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e415ff90 con 0x7f53e415bb40 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e40b4dd0 con 0x7f53e406d5c0 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.010+0000 7f53eaf93640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e406c910 con 0x7f53e40b4970 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e9530640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53e406d5c0 0x7f53e40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e8d2f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f53e415bb40 0x7f53e415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59264/0 (socket says 172.21.15.122:59264) 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e8d2f640 1 -- 172.21.15.122:0/942542205 learned_addr learned my addr 172.21.15.122:0/942542205 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53dbfff640 1 --2- 172.21.15.122:0/942542205 >> v2:172.21.15.169:3300/0 conn(0x7f53e40b4970 0x7f53e406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e9530640 1 -- 172.21.15.122:0/942542205 >> v2:172.21.15.169:3300/0 conn(0x7f53e40b4970 msgr2=0x7f53e406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e9530640 1 --2- 172.21.15.122:0/942542205 >> v2:172.21.15.169:3300/0 conn(0x7f53e40b4970 0x7f53e406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e9530640 1 -- 172.21.15.122:0/942542205 >> v1:172.21.15.122:6789/0 conn(0x7f53e415bb40 legacy=0x7f53e415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53db7fe640 1 -- 172.21.15.122:0/942542205 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244408022 0 0) 0x7f53e415ff90 con 0x7f53e415bb40 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.011+0000 7f53e9530640 1 -- 172.21.15.122:0/942542205 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e4160270 con 0x7f53e406d5c0 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53e9530640 1 --2- 172.21.15.122:0/942542205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e406d5c0 0x7f53e40cfa90 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f53e406bfe0 tx=0x7f53dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdde633baf195ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53db7fe640 1 -- 172.21.15.122:0/942542205 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc039070 con 0x7f53e406d5c0 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53db7fe640 1 -- 172.21.15.122:0/942542205 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc02cc80 con 0x7f53e406d5c0 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53db7fe640 1 -- 172.21.15.122:0/942542205 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc034660 con 0x7f53e406d5c0 2024-03-06T07:24:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 -- 172.21.15.122:0/942542205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e406d5c0 msgr2=0x7f53e40cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 --2- 172.21.15.122:0/942542205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e406d5c0 0x7f53e40cfa90 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f53e406bfe0 tx=0x7f53dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 -- 172.21.15.122:0/942542205 shutdown_connections 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 --2- 172.21.15.122:0/942542205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e406d5c0 0x7f53e40cfa90 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 --2- 172.21.15.122:0/942542205 >> v2:172.21.15.169:3300/0 conn(0x7f53e40b4970 0x7f53e406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 -- 172.21.15.122:0/942542205 >> 172.21.15.122:0/942542205 conn(0x7f53e406b190 msgr2=0x7f53e406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 -- 172.21.15.122:0/942542205 shutdown_connections 2024-03-06T07:24:38.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.012+0000 7f53eaf93640 1 -- 172.21.15.122:0/942542205 wait complete. 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.013+0000 7f53eaf93640 1 Processor -- start 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.013+0000 7f53eaf93640 1 -- start start 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53eaf93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53e406d5c0 0x7f53e414a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53eaf93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53e40b4970 0x7f53e40af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53eaf93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 0x7f53e40b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53eaf93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e415bf20 con 0x7f53e415bb40 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53eaf93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e406cb60 con 0x7f53e40b4970 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53eaf93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e415c0a0 con 0x7f53e406d5c0 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53e9530640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 0x7f53e40b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53e9530640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 0x7f53e40b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45726/0 (socket says 172.21.15.122:45726) 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53e9530640 1 -- 172.21.15.122:0/2008732240 learned_addr learned my addr 172.21.15.122:0/2008732240 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53dbfff640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53e40b4970 0x7f53e40af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.014+0000 7f53e8d2f640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53e406d5c0 0x7f53e414a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53e9530640 1 -- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53e406d5c0 msgr2=0x7f53e414a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53e9530640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53e406d5c0 0x7f53e414a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53e9530640 1 -- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53e40b4970 msgr2=0x7f53e40af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53e9530640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53e40b4970 0x7f53e40af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53e9530640 1 -- 172.21.15.122:0/2008732240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e40affe0 con 0x7f53e415bb40 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53e9530640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 0x7f53e40b2df0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009e80 tx=0x7f53dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc039050 con 0x7f53e415bb40 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc045070 con 0x7f53e415bb40 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e40b02c0 con 0x7f53e415bb40 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc009450 con 0x7f53e415bb40 2024-03-06T07:24:38.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.015+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e40b0880 con 0x7f53e415bb40 2024-03-06T07:24:38.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.016+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e40732b0 con 0x7f53e415bb40 2024-03-06T07:24:38.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.017+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53dc03e850 con 0x7f53e415bb40 2024-03-06T07:24:38.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.017+0000 7f53d97fa640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53d006b4d0 0x7f53d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.017+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53dc0ad190 con 0x7f53e415bb40 2024-03-06T07:24:38.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.017+0000 7f53e8d2f640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53d006b4d0 0x7f53d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.017+0000 7f53e8d2f640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53d006b4d0 0x7f53d006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53e0007fd0 tx=0x7f53e0007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.021+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53dc07bb70 con 0x7f53e415bb40 2024-03-06T07:24:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.192+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53e4068ff0 con 0x7f53e415bb40 2024-03-06T07:24:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.192+0000 7f53d97fa640 1 -- 172.21.15.122:0/2008732240 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53dc07fc10 con 0x7f53e415bb40 2024-03-06T07:24:38.195 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53d006b4d0 msgr2=0x7f53d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53d006b4d0 0x7f53d006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53e0007fd0 tx=0x7f53e0007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 msgr2=0x7f53e40b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 0x7f53e40b2df0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009e80 tx=0x7f53dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 shutdown_connections 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53d006b4d0 0x7f53d006d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53e415bb40 0x7f53e40b2df0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53e40b4970 0x7f53e40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 --2- 172.21.15.122:0/2008732240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53e406d5c0 0x7f53e414a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 >> 172.21.15.122:0/2008732240 conn(0x7f53e406b190 msgr2=0x7f53e415a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 shutdown_connections 2024-03-06T07:24:38.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.194+0000 7f53eaf93640 1 -- 172.21.15.122:0/2008732240 wait complete. 2024-03-06T07:24:38.207 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 Processor -- start 2024-03-06T07:24:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 -- start start 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe4a015f4f0 0x7fe4a0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a006b050 con 0x7fe4a00ce580 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a006b2c0 con 0x7fe4a015f4f0 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.308+0000 7fe4a6c66640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a00b0890 con 0x7fe4a00b0db0 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a00ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:48920/0 (socket says 172.21.15.122:48920) 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 -- 172.21.15.122:0/3942698566 learned_addr learned my addr 172.21.15.122:0/3942698566 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe4a4a02640 1 --2- 172.21.15.122:0/3942698566 >> v2:172.21.15.122:3300/0 conn(0x7fe4a015f4f0 0x7fe4a0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 -- 172.21.15.122:0/3942698566 >> v2:172.21.15.122:3300/0 conn(0x7fe4a015f4f0 msgr2=0x7fe4a0066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 --2- 172.21.15.122:0/3942698566 >> v2:172.21.15.122:3300/0 conn(0x7fe4a015f4f0 0x7fe4a0066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49f7fe640 1 -- 172.21.15.122:0/3942698566 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292941009 0 0) 0x7fe4a006b050 con 0x7fe4a00ce580 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 -- 172.21.15.122:0/3942698566 >> v1:172.21.15.122:6789/0 conn(0x7fe4a00ce580 legacy=0x7fe4a0072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.309+0000 7fe49ffff640 1 -- 172.21.15.122:0/3942698566 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a00672e0 con 0x7fe4a00b0db0 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.310+0000 7fe49ffff640 1 --2- 172.21.15.122:0/3942698566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a00ce040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009870 tx=0x7fe48c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=320a0fc11cf9076a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.310+0000 7fe49f7fe640 1 -- 172.21.15.122:0/3942698566 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c034070 con 0x7fe4a00b0db0 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.310+0000 7fe49f7fe640 1 -- 172.21.15.122:0/3942698566 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c030840 con 0x7fe4a00b0db0 2024-03-06T07:24:38.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.310+0000 7fe49f7fe640 1 -- 172.21.15.122:0/3942698566 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c030c10 con 0x7fe4a00b0db0 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 -- 172.21.15.122:0/3942698566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 msgr2=0x7fe4a00ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/3942698566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a00ce040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009870 tx=0x7fe48c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 -- 172.21.15.122:0/3942698566 shutdown_connections 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/3942698566 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a00ce040 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/3942698566 >> v2:172.21.15.122:3300/0 conn(0x7fe4a015f4f0 0x7fe4a0066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 -- 172.21.15.122:0/3942698566 >> 172.21.15.122:0/3942698566 conn(0x7fe4a00b1d70 msgr2=0x7fe4a00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 -- 172.21.15.122:0/3942698566 shutdown_connections 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.311+0000 7fe4a6c66640 1 -- 172.21.15.122:0/3942698566 wait complete. 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.312+0000 7fe4a6c66640 1 Processor -- start 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.312+0000 7fe4a6c66640 1 -- start start 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a0156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 0x7fe4a00cf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4a015f4f0 0x7fe4a00ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a015f8d0 con 0x7fe4a00ce580 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a006b5a0 con 0x7fe4a015f4f0 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0157530 con 0x7fe4a00b0db0 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 0x7fe4a00cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 0x7fe4a00cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45752/0 (socket says 172.21.15.122:45752) 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 -- 172.21.15.122:0/4179530709 learned_addr learned my addr 172.21.15.122:0/4179530709 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a5203640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4a015f4f0 0x7fe4a00ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a4a02640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a0156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 -- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 msgr2=0x7fe4a0156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a0156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 -- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4a015f4f0 msgr2=0x7fe4a00ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4a015f4f0 0x7fe4a00ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 -- 172.21.15.122:0/4179530709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a0155e00 con 0x7fe4a00ce580 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49ffff640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 0x7fe4a00cf780 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe48c000c00 tx=0x7fe48c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c034040 con 0x7fe4a00ce580 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c030640 con 0x7fe4a00ce580 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a01f5ad0 con 0x7fe4a00ce580 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c039d40 con 0x7fe4a00ce580 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.313+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a01f6090 con 0x7fe4a00ce580 2024-03-06T07:24:38.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.314+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a006f040 con 0x7fe4a00ce580 2024-03-06T07:24:38.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.315+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe48c04b060 con 0x7fe4a00ce580 2024-03-06T07:24:38.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.315+0000 7fe49d7fa640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe48406b4d0 0x7fe48406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.315+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe48c0abc60 con 0x7fe4a00ce580 2024-03-06T07:24:38.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.315+0000 7fe4a4a02640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe48406b4d0 0x7fe48406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.316+0000 7fe4a4a02640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe48406b4d0 0x7fe48406d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe494009aa0 tx=0x7fe494009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.320+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4a006f040 con 0x7fe4a00ce580 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.491+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4a009e3d0 con 0x7fe4a00ce580 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.491+0000 7fe49d7fa640 1 -- 172.21.15.122:0/4179530709 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe48c07a640 con 0x7fe4a00ce580 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:24:38.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe48406b4d0 msgr2=0x7fe48406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe48406b4d0 0x7fe48406d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe494009aa0 tx=0x7fe494009190 comp rx=0 tx=0).stop 2024-03-06T07:24:38.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 msgr2=0x7fe4a00cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 0x7fe4a00cf780 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe48c000c00 tx=0x7fe48c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 shutdown_connections 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe48406b4d0 0x7fe48406d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4a015f4f0 0x7fe4a00ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a00ce580 0x7fe4a00cf780 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 --2- 172.21.15.122:0/4179530709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4a00b0db0 0x7fe4a0156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 >> 172.21.15.122:0/4179530709 conn(0x7fe4a00b1d70 msgr2=0x7fe4a00b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 shutdown_connections 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.493+0000 7fe4a6c66640 1 -- 172.21.15.122:0/4179530709 wait complete. 2024-03-06T07:24:38.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:38.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 Processor -- start 2024-03-06T07:24:38.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 -- start start 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 0x7f52480d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f52480d1910 0x7f52480d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52481618b0 con 0x7f52480d22e0 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52480cf400 con 0x7f52481614d0 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.607+0000 7f524cfbf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52480d2710 con 0x7f52480d1910 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f524659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 0x7f52480d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f524659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 0x7f52480d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45768/0 (socket says 172.21.15.122:45768) 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f523ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f52480d1910 0x7f52480d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f524659c640 1 -- 172.21.15.122:0/2581798323 learned_addr learned my addr 172.21.15.122:0/2581798323 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:38.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f523ffff640 1 -- 172.21.15.122:0/2581798323 >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 msgr2=0x7f52480d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f523ffff640 1 --2- 172.21.15.122:0/2581798323 >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 0x7f52480d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f523ffff640 1 -- 172.21.15.122:0/2581798323 >> v1:172.21.15.169:6789/0 conn(0x7f52480d22e0 legacy=0x7f52480d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f523ffff640 1 -- 172.21.15.122:0/2581798323 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480cf6a0 con 0x7f52480d1910 2024-03-06T07:24:38.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.608+0000 7f524659c640 1 --2- 172.21.15.122:0/2581798323 >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 0x7f52480d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:38.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f523ffff640 1 --2- 172.21.15.122:0/2581798323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52480d1910 0x7f52480d1da0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f52480ce550 tx=0x7f523002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d8f5bb5a7606da6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f5245d9b640 1 -- 172.21.15.122:0/2581798323 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230035020 con 0x7f52480d1910 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f5245d9b640 1 -- 172.21.15.122:0/2581798323 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5230031830 con 0x7f52480d1910 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f5245d9b640 1 -- 172.21.15.122:0/2581798323 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230031c00 con 0x7f52480d1910 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 -- 172.21.15.122:0/2581798323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52480d1910 msgr2=0x7f52480d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 --2- 172.21.15.122:0/2581798323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52480d1910 0x7f52480d1da0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f52480ce550 tx=0x7f523002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 -- 172.21.15.122:0/2581798323 shutdown_connections 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 --2- 172.21.15.122:0/2581798323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52480d1910 0x7f52480d1da0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 --2- 172.21.15.122:0/2581798323 >> v2:172.21.15.122:3300/0 conn(0x7f52481614d0 0x7f52480d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 -- 172.21.15.122:0/2581798323 >> 172.21.15.122:0/2581798323 conn(0x7f52480d3d40 msgr2=0x7f52480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 -- 172.21.15.122:0/2581798323 shutdown_connections 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.609+0000 7f524cfbf640 1 -- 172.21.15.122:0/2581798323 wait complete. 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 Processor -- start 2024-03-06T07:24:38.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 -- start start 2024-03-06T07:24:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 0x7f52481552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52480d22e0 0x7f52480b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52481614d0 0x7f52480b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480b0910 con 0x7f52480d1910 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480cfc60 con 0x7f52481614d0 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.611+0000 7f524cfbf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52480b1e40 con 0x7f52480d22e0 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 0x7f52481552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 0x7f52481552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45778/0 (socket says 172.21.15.122:45778) 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 -- 172.21.15.122:0/390414299 learned_addr learned my addr 172.21.15.122:0/390414299 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f523ffff640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52480d22e0 0x7f52480b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f5246d9d640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52481614d0 0x7f52480b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 -- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52480d22e0 msgr2=0x7f52480b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52480d22e0 0x7f52480b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 -- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52481614d0 msgr2=0x7f52480b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52481614d0 0x7f52480b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 -- 172.21.15.122:0/390414299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480cbb40 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524659c640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 0x7f52481552f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f523800e9e0 tx=0x7f523800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238017070 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523800c9d0 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52480cbe20 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800cd80 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.612+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52480cc3e0 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.613+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5248068ff0 con 0x7f52480d1910 2024-03-06T07:24:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.614+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f523801a070 con 0x7f52480d1910 2024-03-06T07:24:38.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.614+0000 7f523effd640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f522806b500 0x7f522806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.614+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f523808a3f0 con 0x7f52480d1910 2024-03-06T07:24:38.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.614+0000 7f523ffff640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f522806b500 0x7f522806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.614+0000 7f523ffff640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f522806b500 0x7f522806d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5230036000 tx=0x7f5230008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.618+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5238058b90 con 0x7f52480d1910 2024-03-06T07:24:38.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.789+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f52480d3360 con 0x7f52480d1910 2024-03-06T07:24:38.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.789+0000 7f523effd640 1 -- 172.21.15.122:0/390414299 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f523805cc30 con 0x7f52480d1910 2024-03-06T07:24:38.791 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f522806b500 msgr2=0x7f522806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f522806b500 0x7f522806d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5230036000 tx=0x7f5230008030 comp rx=0 tx=0).stop 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 msgr2=0x7f52481552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 0x7f52481552f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f523800e9e0 tx=0x7f523800eea0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 shutdown_connections 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f522806b500 0x7f522806d9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52481614d0 0x7f52480b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52480d22e0 0x7f52480b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 --2- 172.21.15.122:0/390414299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52480d1910 0x7f52481552f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 >> 172.21.15.122:0/390414299 conn(0x7f52480d3d40 msgr2=0x7f52480d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.791+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 shutdown_connections 2024-03-06T07:24:38.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.792+0000 7f524cfbf640 1 -- 172.21.15.122:0/390414299 wait complete. 2024-03-06T07:24:38.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:38.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.902+0000 7fd39d9d7640 1 Processor -- start 2024-03-06T07:24:38.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.902+0000 7fd39d9d7640 1 -- start start 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd39d9d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398162970 0x7fd3980d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd39d9d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980d22c0 con 0x7fd3980b4460 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd39d9d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980d25a0 con 0x7fd398161cf0 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd39d9d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd398161b00 con 0x7fd398162970 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd396ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd3980b4460 0x7fd3980b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59290/0 (socket says 172.21.15.122:59290) 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd396ffd640 1 -- 172.21.15.122:0/192690591 learned_addr learned my addr 172.21.15.122:0/192690591 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd3967fc640 1 --2- 172.21.15.122:0/192690591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398162970 0x7fd3980d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd395ffb640 1 -- 172.21.15.122:0/192690591 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913502692 0 0) 0x7fd3980d22c0 con 0x7fd3980b4460 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd395ffb640 1 -- 172.21.15.122:0/192690591 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd380003520 con 0x7fd3980b4460 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd395ffb640 1 -- 172.21.15.122:0/192690591 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115655196 0 0) 0x7fd3980d25a0 con 0x7fd398161cf0 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd3967fc640 1 -- 172.21.15.122:0/192690591 >> v1:172.21.15.169:6789/0 conn(0x7fd398161cf0 legacy=0x7fd3980cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd3967fc640 1 -- 172.21.15.122:0/192690591 >> v1:172.21.15.122:6789/0 conn(0x7fd3980b4460 legacy=0x7fd3980b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.903+0000 7fd3967fc640 1 -- 172.21.15.122:0/192690591 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980d2880 con 0x7fd398162970 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.904+0000 7fd3967fc640 1 --2- 172.21.15.122:0/192690591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398162970 0x7fd3980d4800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0097d0 tx=0x7fd38c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fba4f9c229f364 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.904+0000 7fd395ffb640 1 -- 172.21.15.122:0/192690591 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c035020 con 0x7fd398162970 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.904+0000 7fd395ffb640 1 -- 172.21.15.122:0/192690591 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c031890 con 0x7fd398162970 2024-03-06T07:24:38.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.904+0000 7fd395ffb640 1 -- 172.21.15.122:0/192690591 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c031c60 con 0x7fd398162970 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 -- 172.21.15.122:0/192690591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398162970 msgr2=0x7fd3980d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/192690591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398162970 0x7fd3980d4800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0097d0 tx=0x7fd38c02b930 comp rx=0 tx=0).stop 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 -- 172.21.15.122:0/192690591 shutdown_connections 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/192690591 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398162970 0x7fd3980d4800 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 -- 172.21.15.122:0/192690591 >> 172.21.15.122:0/192690591 conn(0x7fd3980d3ce0 msgr2=0x7fd3980d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 -- 172.21.15.122:0/192690591 shutdown_connections 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.905+0000 7fd39d9d7640 1 -- 172.21.15.122:0/192690591 wait complete. 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.906+0000 7fd39d9d7640 1 Processor -- start 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.906+0000 7fd39d9d7640 1 -- start start 2024-03-06T07:24:38.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 0x7fd39815e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398161cf0 0x7fd398158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd398162970 0x7fd3981592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980ce950 con 0x7fd3980b4460 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980cecc0 con 0x7fd398162970 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980b4840 con 0x7fd398161cf0 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 0x7fd39815e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 0x7fd39815e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45790/0 (socket says 172.21.15.122:45790) 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 -- 172.21.15.122:0/2249452758 learned_addr learned my addr 172.21.15.122:0/2249452758 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd3967fc640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398161cf0 0x7fd398158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd3977fe640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd398162970 0x7fd3981592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 -- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398161cf0 msgr2=0x7fd398158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398161cf0 0x7fd398158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 -- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd398162970 msgr2=0x7fd3981592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd398162970 0x7fd3981592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 -- 172.21.15.122:0/2249452758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980b8820 con 0x7fd3980b4460 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd396ffd640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 0x7fd39815e740 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c390 tx=0x7fd38800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388017070 con 0x7fd3980b4460 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd388005ce0 con 0x7fd3980b4460 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3980b8b00 con 0x7fd3980b4460 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.907+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3880072f0 con 0x7fd3980b4460 2024-03-06T07:24:38.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.908+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3980b90c0 con 0x7fd3980b4460 2024-03-06T07:24:38.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.908+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39804f330 con 0x7fd3980b4460 2024-03-06T07:24:38.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.909+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd38801a070 con 0x7fd3980b4460 2024-03-06T07:24:38.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.909+0000 7fd37ffff640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38406b4d0 0x7fd38406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.909+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3880894c0 con 0x7fd3980b4460 2024-03-06T07:24:38.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.909+0000 7fd3967fc640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38406b4d0 0x7fd38406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.910+0000 7fd3967fc640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38406b4d0 0x7fd38406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd38c036000 tx=0x7fd38c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:38.913+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd388057d80 con 0x7fd3980b4460 2024-03-06T07:24:39.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.096+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd39806b180 con 0x7fd3980b4460 2024-03-06T07:24:39.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.096+0000 7fd37ffff640 1 -- 172.21.15.122:0/2249452758 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd38805e0d0 con 0x7fd3980b4460 2024-03-06T07:24:39.098 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:24:39.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38406b4d0 msgr2=0x7fd38406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38406b4d0 0x7fd38406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd38c036000 tx=0x7fd38c03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 msgr2=0x7fd39815e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 0x7fd39815e740 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c390 tx=0x7fd38800c850 comp rx=0 tx=0).stop 2024-03-06T07:24:39.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 shutdown_connections 2024-03-06T07:24:39.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd38406b4d0 0x7fd38406d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd398162970 0x7fd3981592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd398161cf0 0x7fd398158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 --2- 172.21.15.122:0/2249452758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3980b4460 0x7fd39815e740 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 >> 172.21.15.122:0/2249452758 conn(0x7fd3980d3ce0 msgr2=0x7fd3980d06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 shutdown_connections 2024-03-06T07:24:39.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:39.098+0000 7fd39d9d7640 1 -- 172.21.15.122:0/2249452758 wait complete. 2024-03-06T07:24:39.110 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:24:39.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:39.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[58887]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:39.139 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:45.240 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:24:45.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.241 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:45.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:45.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.416+0000 7f9ab051e640 1 Processor -- start 2024-03-06T07:24:45.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.416+0000 7f9ab051e640 1 -- start start 2024-03-06T07:24:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9ab051e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9aa80b4f70 0x7f9aa80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9ab051e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa806c6d0 0x7f9aa815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9ab051e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9aa8066d90 0x7f9aa80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9ab051e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa815e9f0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9ab051e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa80cda10 con 0x7f9aa8066d90 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9ab051e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80b5890 con 0x7f9aa806c6d0 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9aa80b4f70 0x7f9aa80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9aa80b4f70 0x7f9aa80b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39956/0 (socket says 172.21.15.122:39956) 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 -- 172.21.15.122:0/830590620 learned_addr learned my addr 172.21.15.122:0/830590620 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aadab9640 1 --2- 172.21.15.122:0/830590620 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa806c6d0 0x7f9aa815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aaeabb640 1 --2- 172.21.15.122:0/830590620 >> v2:172.21.15.169:3300/0 conn(0x7f9aa8066d90 0x7f9aa80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 -- 172.21.15.122:0/830590620 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa806c6d0 msgr2=0x7f9aa815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 --2- 172.21.15.122:0/830590620 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa806c6d0 0x7f9aa815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 -- 172.21.15.122:0/830590620 >> v2:172.21.15.169:3300/0 conn(0x7f9aa8066d90 msgr2=0x7f9aa80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 --2- 172.21.15.122:0/830590620 >> v2:172.21.15.169:3300/0 conn(0x7f9aa8066d90 0x7f9aa80ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.417+0000 7f9aae2ba640 1 -- 172.21.15.122:0/830590620 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80cd450 con 0x7f9aa80b4f70 2024-03-06T07:24:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.418+0000 7f9aae2ba640 1 --2- 172.21.15.122:0/830590620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa80b5350 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80723c0 tx=0x7f9aa402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbcc0e82349e0a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.418+0000 7f9aad2b8640 1 -- 172.21.15.122:0/830590620 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4035020 con 0x7f9aa80b4f70 2024-03-06T07:24:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.418+0000 7f9aad2b8640 1 -- 172.21.15.122:0/830590620 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa4031870 con 0x7f9aa80b4f70 2024-03-06T07:24:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.418+0000 7f9aad2b8640 1 -- 172.21.15.122:0/830590620 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4031c40 con 0x7f9aa80b4f70 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 -- 172.21.15.122:0/830590620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 msgr2=0x7f9aa80b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 --2- 172.21.15.122:0/830590620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa80b5350 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80723c0 tx=0x7f9aa402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 -- 172.21.15.122:0/830590620 shutdown_connections 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 --2- 172.21.15.122:0/830590620 >> v2:172.21.15.169:3300/0 conn(0x7f9aa8066d90 0x7f9aa80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 --2- 172.21.15.122:0/830590620 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa806c6d0 0x7f9aa815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 --2- 172.21.15.122:0/830590620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa80b5350 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 -- 172.21.15.122:0/830590620 >> 172.21.15.122:0/830590620 conn(0x7f9aa80724d0 msgr2=0x7f9aa80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 -- 172.21.15.122:0/830590620 shutdown_connections 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.419+0000 7f9ab051e640 1 -- 172.21.15.122:0/830590620 wait complete. 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.420+0000 7f9ab051e640 1 Processor -- start 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.420+0000 7f9ab051e640 1 -- start start 2024-03-06T07:24:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9ab051e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa8066d90 0x7f9aa815df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9ab051e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa81e9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9ab051e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa81e9aa0 0x7f9aa80cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9ab051e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa815f3c0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9ab051e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa815e680 con 0x7f9aa81e9aa0 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9ab051e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80cd090 con 0x7f9aa8066d90 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9aadab9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa81e9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9aadab9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa81e9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39960/0 (socket says 172.21.15.122:39960) 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9aadab9640 1 -- 172.21.15.122:0/3277192688 learned_addr learned my addr 172.21.15.122:0/3277192688 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9aae2ba640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa8066d90 0x7f9aa815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.421+0000 7f9aaeabb640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa81e9aa0 0x7f9aa80cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9aadab9640 1 -- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa8066d90 msgr2=0x7f9aa815df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9aadab9640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa8066d90 0x7f9aa815df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9aadab9640 1 -- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa81e9aa0 msgr2=0x7f9aa80cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9aadab9640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa81e9aa0 0x7f9aa80cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9aadab9640 1 -- 172.21.15.122:0/3277192688 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80d00e0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9aadab9640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa81e9560 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800e870 tx=0x7f9a9800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98017070 con 0x7f9aa80b4f70 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9800c8e0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa80d03c0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800ccd0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.422+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa80d0980 con 0x7f9aa80b4f70 2024-03-06T07:24:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.423+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa81e9eb0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.424+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a9801a070 con 0x7f9aa80b4f70 2024-03-06T07:24:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.424+0000 7f9a9effd640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8c06b500 0x7f9a8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.424+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a9808a310 con 0x7f9aa80b4f70 2024-03-06T07:24:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.424+0000 7f9aae2ba640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8c06b500 0x7f9a8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.425+0000 7f9aae2ba640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8c06b500 0x7f9a8c06d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4036000 tx=0x7f9aa403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.429+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9aa81e9eb0 con 0x7f9aa80b4f70 2024-03-06T07:24:45.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.638+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aa804f330 con 0x7f9aa80b4f70 2024-03-06T07:24:45.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.638+0000 7f9a9effd640 1 -- 172.21.15.122:0/3277192688 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f9a98058b40 con 0x7f9aa80b4f70 2024-03-06T07:24:45.640 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:45.640 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:45.641 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:45.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8c06b500 msgr2=0x7f9a8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8c06b500 0x7f9a8c06d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4036000 tx=0x7f9aa403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 msgr2=0x7f9aa81e9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa81e9560 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800e870 tx=0x7f9a9800ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 shutdown_connections 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a8c06b500 0x7f9a8c06d9c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9aa81e9aa0 0x7f9aa80cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa80b4f70 0x7f9aa81e9560 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 --2- 172.21.15.122:0/3277192688 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9aa8066d90 0x7f9aa815df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 >> 172.21.15.122:0/3277192688 conn(0x7f9aa80724d0 msgr2=0x7f9aa806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 shutdown_connections 2024-03-06T07:24:45.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.640+0000 7f9ab051e640 1 -- 172.21.15.122:0/3277192688 wait complete. 2024-03-06T07:24:45.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:24:45.814 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:24:45.815 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:45.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.913+0000 7f4b62f02640 1 Processor -- start 2024-03-06T07:24:45.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.913+0000 7f4b62f02640 1 -- start start 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b62f02640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4b5c06da40 0x7f4b5c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b62f02640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c0cc980 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b62f02640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c0ccc60 con 0x7f4b5c1424f0 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b62f02640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c0cd220 con 0x7f4b5c06da40 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b6149f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4b5c0cd7a0 0x7f4b5c071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43416/0 (socket says 172.21.15.122:43416) 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b6149f640 1 -- 172.21.15.122:0/2362112264 learned_addr learned my addr 172.21.15.122:0/2362112264 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5bfff640 1 --2- 172.21.15.122:0/2362112264 >> v2:172.21.15.169:3300/0 conn(0x7f4b5c06da40 0x7f4b5c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5b7fe640 1 -- 172.21.15.122:0/2362112264 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807317101 0 0) 0x7f4b5c0cc980 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5b7fe640 1 -- 172.21.15.122:0/2362112264 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b4c003520 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2362112264 >> v1:172.21.15.169:6789/0 conn(0x7f4b5c1424f0 legacy=0x7f4b5c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2362112264 >> v1:172.21.15.122:6789/0 conn(0x7f4b5c0cd7a0 legacy=0x7f4b5c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5b7fe640 1 -- 172.21.15.122:0/2362112264 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1404400417 0 0) 0x7f4b4c003520 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.914+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2362112264 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0ccf40 con 0x7f4b5c06da40 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.915+0000 7f4b5bfff640 1 --2- 172.21.15.122:0/2362112264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c06da40 0x7f4b5c0cc2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4b480097a0 tx=0x7f4b4802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be405f5a6f69b000 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.915+0000 7f4b5b7fe640 1 -- 172.21.15.122:0/2362112264 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48038070 con 0x7f4b5c06da40 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.915+0000 7f4b5b7fe640 1 -- 172.21.15.122:0/2362112264 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b48031500 con 0x7f4b5c06da40 2024-03-06T07:24:45.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.915+0000 7f4b5b7fe640 1 -- 172.21.15.122:0/2362112264 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b540019f0 con 0x7f4b5c06da40 2024-03-06T07:24:45.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 -- 172.21.15.122:0/2362112264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c06da40 msgr2=0x7f4b5c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2362112264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c06da40 0x7f4b5c0cc2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4b480097a0 tx=0x7f4b4802b820 comp rx=0 tx=0).stop 2024-03-06T07:24:45.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 -- 172.21.15.122:0/2362112264 shutdown_connections 2024-03-06T07:24:45.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2362112264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c06da40 0x7f4b5c0cc2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 -- 172.21.15.122:0/2362112264 >> 172.21.15.122:0/2362112264 conn(0x7f4b5c0cf8a0 msgr2=0x7f4b5c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 -- 172.21.15.122:0/2362112264 shutdown_connections 2024-03-06T07:24:45.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.916+0000 7f4b62f02640 1 -- 172.21.15.122:0/2362112264 wait complete. 2024-03-06T07:24:45.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.917+0000 7f4b62f02640 1 Processor -- start 2024-03-06T07:24:45.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.917+0000 7f4b62f02640 1 -- start start 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b62f02640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b5c06da40 0x7f4b5c1350d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b62f02640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 0x7f4b5c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b62f02640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c1424f0 0x7f4b5c158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b62f02640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c15d070 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b62f02640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c15cef0 con 0x7f4b5c1424f0 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b62f02640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c15d1f0 con 0x7f4b5c06da40 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 0x7f4b5c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 0x7f4b5c1585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39964/0 (socket says 172.21.15.122:39964) 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2219947095 learned_addr learned my addr 172.21.15.122:0/2219947095 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b60c9e640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b5c06da40 0x7f4b5c1350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b6149f640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c1424f0 0x7f4b5c158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b5c06da40 msgr2=0x7f4b5c1350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b5c06da40 0x7f4b5c1350d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c1424f0 msgr2=0x7f4b5c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c1424f0 0x7f4b5c158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.918+0000 7f4b5bfff640 1 -- 172.21.15.122:0/2219947095 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0b0be0 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.919+0000 7f4b5bfff640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 0x7f4b5c1585c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4b48036040 tx=0x7f4b480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.919+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48043070 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.919+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4803f870 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.919+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c0b0ec0 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.919+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48038040 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.919+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c154770 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.920+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c04f330 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.920+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b48046070 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.920+0000 7f4b597fa640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b4006b4d0 0x7f4b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.921+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b480abba0 con 0x7f4b5c0cd7a0 2024-03-06T07:24:45.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.921+0000 7f4b60c9e640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b4006b4d0 0x7f4b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.921+0000 7f4b60c9e640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b4006b4d0 0x7f4b4006d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4b50000f60 tx=0x7f4b50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:45.925+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b48030070 con 0x7f4b5c0cd7a0 2024-03-06T07:24:46.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.132+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b5c141d10 con 0x7f4b5c0cd7a0 2024-03-06T07:24:46.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.133+0000 7f4b597fa640 1 -- 172.21.15.122:0/2219947095 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f4b4807a580 con 0x7f4b5c0cd7a0 2024-03-06T07:24:46.135 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:46.135 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:46.135 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.134+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b4006b4d0 msgr2=0x7f4b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.134+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b4006b4d0 0x7f4b4006d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4b50000f60 tx=0x7f4b50009040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.134+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 msgr2=0x7f4b5c1585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.134+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 0x7f4b5c1585c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4b48036040 tx=0x7f4b480046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 shutdown_connections 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b4006b4d0 0x7f4b4006d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5c1424f0 0x7f4b5c158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b5c0cd7a0 0x7f4b5c1585c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 --2- 172.21.15.122:0/2219947095 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b5c06da40 0x7f4b5c1350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 >> 172.21.15.122:0/2219947095 conn(0x7f4b5c0cf8a0 msgr2=0x7f4b5c0ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 shutdown_connections 2024-03-06T07:24:46.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.135+0000 7f4b62f02640 1 -- 172.21.15.122:0/2219947095 wait complete. 2024-03-06T07:24:46.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:24:46.283 INFO:teuthology.orchestra.run.smithi169.stdout:{"message":"","return_code":0} 2024-03-06T07:24:46.283 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:46.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:46.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.382+0000 7fe156702640 1 Processor -- start 2024-03-06T07:24:46.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.382+0000 7fe156702640 1 -- start start 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe156702640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe15013dc10 0x7fe1500b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe156702640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe15015ef60 0x7fe1500b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe156702640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15015f580 con 0x7fe15015bf30 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe156702640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1500cf3f0 con 0x7fe15013dc10 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe156702640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15015b210 con 0x7fe15015ef60 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe14ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe15013dc10 0x7fe1500b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe14ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe15013dc10 0x7fe1500b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39976/0 (socket says 172.21.15.122:39976) 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe14ffff640 1 -- 172.21.15.122:0/2702622530 learned_addr learned my addr 172.21.15.122:0/2702622530 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe14f7fe640 1 --1- 172.21.15.122:0/2702622530 >> v1:172.21.15.122:6789/0 conn(0x7fe15015bf30 0x7fe15015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43422/0 (socket says 172.21.15.122:43422) 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe154c9f640 1 --2- 172.21.15.122:0/2702622530 >> v2:172.21.15.169:3300/0 conn(0x7fe15015ef60 0x7fe1500b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe14ffff640 1 -- 172.21.15.122:0/2702622530 >> v2:172.21.15.169:3300/0 conn(0x7fe15015ef60 msgr2=0x7fe1500b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.383+0000 7fe14ffff640 1 --2- 172.21.15.122:0/2702622530 >> v2:172.21.15.169:3300/0 conn(0x7fe15015ef60 0x7fe1500b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14ffff640 1 -- 172.21.15.122:0/2702622530 >> v1:172.21.15.122:6789/0 conn(0x7fe15015bf30 legacy=0x7fe15015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14ffff640 1 -- 172.21.15.122:0/2702622530 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15015b450 con 0x7fe15013dc10 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14effd640 1 -- 172.21.15.122:0/2702622530 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491170292 0 0) 0x7fe15015f580 con 0x7fe15015bf30 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14ffff640 1 --2- 172.21.15.122:0/2702622530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15013dc10 0x7fe1500b2950 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe150068700 tx=0x7fe14002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c9cf170550f87a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14effd640 1 -- 172.21.15.122:0/2702622530 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140035020 con 0x7fe15013dc10 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14effd640 1 -- 172.21.15.122:0/2702622530 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140031870 con 0x7fe15013dc10 2024-03-06T07:24:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.384+0000 7fe14effd640 1 -- 172.21.15.122:0/2702622530 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140031c40 con 0x7fe15013dc10 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 -- 172.21.15.122:0/2702622530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15013dc10 msgr2=0x7fe1500b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 --2- 172.21.15.122:0/2702622530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15013dc10 0x7fe1500b2950 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe150068700 tx=0x7fe14002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 -- 172.21.15.122:0/2702622530 shutdown_connections 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 --2- 172.21.15.122:0/2702622530 >> v2:172.21.15.169:3300/0 conn(0x7fe15015ef60 0x7fe1500b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 --2- 172.21.15.122:0/2702622530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15013dc10 0x7fe1500b2950 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 -- 172.21.15.122:0/2702622530 >> 172.21.15.122:0/2702622530 conn(0x7fe1500cf8a0 msgr2=0x7fe1500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 -- 172.21.15.122:0/2702622530 shutdown_connections 2024-03-06T07:24:46.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.385+0000 7fe156702640 1 -- 172.21.15.122:0/2702622530 wait complete. 2024-03-06T07:24:46.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.386+0000 7fe156702640 1 Processor -- start 2024-03-06T07:24:46.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.386+0000 7fe156702640 1 -- start start 2024-03-06T07:24:46.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe156702640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe15013dc10 0x7fe150154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe156702640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 0x7fe150071950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe156702640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe15015ef60 0x7fe150071e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe156702640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500cd7a0 con 0x7fe15015bf30 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe156702640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15013dff0 con 0x7fe15015ef60 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe156702640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500cd920 con 0x7fe15013dc10 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 0x7fe150071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 0x7fe150071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39988/0 (socket says 172.21.15.122:39988) 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 -- 172.21.15.122:0/2664055067 learned_addr learned my addr 172.21.15.122:0/2664055067 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14ffff640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe15013dc10 0x7fe150154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe154c9f640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe15015ef60 0x7fe150071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 -- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe15013dc10 msgr2=0x7fe150154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe15013dc10 0x7fe150154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 -- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe15015ef60 msgr2=0x7fe150071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe15015ef60 0x7fe150071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 -- 172.21.15.122:0/2664055067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1501f3f50 con 0x7fe15015bf30 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.387+0000 7fe14f7fe640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 0x7fe150071950 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe14400e6d0 tx=0x7fe14400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.388+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14401a070 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.388+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe144006c10 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.388+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe150139b20 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.388+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1440090d0 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.388+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15013a0a0 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.389+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15004f330 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.389+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe14401d070 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.389+0000 7fe14cff9640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe13c06b4d0 0x7fe13c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.389+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe14408b760 con 0x7fe15015bf30 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.389+0000 7fe14ffff640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe13c06b4d0 0x7fe13c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.390+0000 7fe14ffff640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe13c06b4d0 0x7fe13c06d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe140036000 tx=0x7fe14003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.394+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe14405a020 con 0x7fe15015bf30 2024-03-06T07:24:46.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.607+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe15006eb00 con 0x7fe15015bf30 2024-03-06T07:24:46.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.608+0000 7fe14cff9640 1 -- 172.21.15.122:0/2664055067 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe14405e0c0 con 0x7fe15015bf30 2024-03-06T07:24:46.610 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe13c06b4d0 msgr2=0x7fe13c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe13c06b4d0 0x7fe13c06d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe140036000 tx=0x7fe14003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 msgr2=0x7fe150071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 0x7fe150071950 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe14400e6d0 tx=0x7fe14400eb90 comp rx=0 tx=0).stop 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 shutdown_connections 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe13c06b4d0 0x7fe13c06d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe15015ef60 0x7fe150071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe15015bf30 0x7fe150071950 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 --2- 172.21.15.122:0/2664055067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe15013dc10 0x7fe150154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 >> 172.21.15.122:0/2664055067 conn(0x7fe1500cf8a0 msgr2=0x7fe15015c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 shutdown_connections 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:46.610+0000 7fe156702640 1 -- 172.21.15.122:0/2664055067 wait complete. 2024-03-06T07:24:46.622 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:46.623 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:46.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:46.623 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:24:46.638 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:46.639 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42554sec preferred_lft 42554sec 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:24:46.640 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:46.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:46.641 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:46.704 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:24:46.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:46.706 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:24:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:24:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:46.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:46.761 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:24:46.761 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:24:46.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:46.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:46.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:46.762 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:24:46.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:46.856 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:24:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:46.870 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:46.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:46.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:24:46.937 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:46.937 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:46.938 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:46.992 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:46.992 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:46.992 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:47.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:24:47.113 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:24:47.169 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:47.170 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:47.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.171 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:24:47.224 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:47.224 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:47.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:24:47.293 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:47.349 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:24:47.349 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:24:47.349 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:24:47.349 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:24:47.351 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:47.352 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.353 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:24:47.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:24:47.349+0000 7f3b242bb480 -1 init, newargv = 0x557570d83780 newargc=15 2024-03-06T07:24:47.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[59993]: starting ceph client 2024-03-06T07:24:47.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[59993]: starting fuse 2024-03-06T07:24:47.368 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:24:47.368 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:24:47.369 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:47.370 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:24:47.488 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.59993.asok 2024-03-06T07:24:47.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:47.500 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59993.asok 2024-03-06T07:24:47.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.500 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59993.asok status 2024-03-06T07:24:47.635 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:24:47.635 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:24:47.635 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "59993", 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 6408, 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:24:47.636 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 6408 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1666302974 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1666302974 2024-03-06T07:24:47.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.6408 192.168.0.1:0/1666302974", 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/1666302974", 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 27, 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 57, 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:24:47.638 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:24:47.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.644 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:24:47.644 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:47.659 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:24:47.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:47.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:47.725 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:47.726 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> import uuid 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> n = 75 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = False 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> if 0 > 0: 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = True 2024-03-06T07:24:47.781 DEBUG:teuthology.orchestra.run.smithi122:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> dpath = os.path.dirname(path) 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> try: 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(n): 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> f.write(f"{i}") 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> f.flush() 2024-03-06T07:24:47.782 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(f.fileno()) 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> os.unlink(fpath) 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> if create_hard_links: 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> for j in range(0): 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> finally: 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> os.close(dirfd) 2024-03-06T07:24:47.783 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:24:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:48.272 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:48.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:48.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:48.409 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:48.410 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:48.410 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:48.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:48.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.562+0000 7f5bbaea4640 1 Processor -- start 2024-03-06T07:24:48.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.562+0000 7f5bbaea4640 1 -- start start 2024-03-06T07:24:48.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bbaea4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb413dc10 0x7f5bb40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bbaea4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5bb415bf30 0x7f5bb40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bbaea4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb415f580 con 0x7f5bb415e850 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bbaea4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb40cf3f0 con 0x7f5bb415bf30 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bbaea4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb415b210 con 0x7f5bb413dc10 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb9441640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5bb415e850 0x7f5bb415eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43434/0 (socket says 172.21.15.122:43434) 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb9441640 1 -- 172.21.15.122:0/1734243117 learned_addr learned my addr 172.21.15.122:0/1734243117 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb8c40640 1 --2- 172.21.15.122:0/1734243117 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb413dc10 0x7f5bb40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb3fff640 1 --2- 172.21.15.122:0/1734243117 >> v2:172.21.15.169:3300/0 conn(0x7f5bb415bf30 0x7f5bb40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb37fe640 1 -- 172.21.15.122:0/1734243117 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338670005 0 0) 0x7f5bb415f580 con 0x7f5bb415e850 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb37fe640 1 -- 172.21.15.122:0/1734243117 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba4003520 con 0x7f5bb415e850 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb8c40640 1 -- 172.21.15.122:0/1734243117 >> v2:172.21.15.169:3300/0 conn(0x7f5bb415bf30 msgr2=0x7f5bb40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb8c40640 1 --2- 172.21.15.122:0/1734243117 >> v2:172.21.15.169:3300/0 conn(0x7f5bb415bf30 0x7f5bb40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb8c40640 1 -- 172.21.15.122:0/1734243117 >> v1:172.21.15.122:6789/0 conn(0x7f5bb415e850 legacy=0x7f5bb415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.563+0000 7f5bb8c40640 1 -- 172.21.15.122:0/1734243117 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb415b450 con 0x7f5bb413dc10 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.564+0000 7f5bb8c40640 1 --2- 172.21.15.122:0/1734243117 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb413dc10 0x7f5bb40b2950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80097a0 tx=0x7f5ba802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd018f906eee46ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.564+0000 7f5bb37fe640 1 -- 172.21.15.122:0/1734243117 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80315a0 con 0x7f5bb413dc10 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.564+0000 7f5bb37fe640 1 -- 172.21.15.122:0/1734243117 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba8037070 con 0x7f5bb413dc10 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.564+0000 7f5bb37fe640 1 -- 172.21.15.122:0/1734243117 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8031940 con 0x7f5bb413dc10 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 -- 172.21.15.122:0/1734243117 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb413dc10 msgr2=0x7f5bb40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/1734243117 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb413dc10 0x7f5bb40b2950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80097a0 tx=0x7f5ba802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 -- 172.21.15.122:0/1734243117 shutdown_connections 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/1734243117 >> v2:172.21.15.169:3300/0 conn(0x7f5bb415bf30 0x7f5bb40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/1734243117 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb413dc10 0x7f5bb40b2950 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 -- 172.21.15.122:0/1734243117 >> 172.21.15.122:0/1734243117 conn(0x7f5bb40cf8a0 msgr2=0x7f5bb40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 -- 172.21.15.122:0/1734243117 shutdown_connections 2024-03-06T07:24:48.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.565+0000 7f5bbaea4640 1 -- 172.21.15.122:0/1734243117 wait complete. 2024-03-06T07:24:48.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.566+0000 7f5bbaea4640 1 Processor -- start 2024-03-06T07:24:48.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.566+0000 7f5bbaea4640 1 -- start start 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb413dc10 0x7f5bb413c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 0x7f5bb415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb415e850 0x7f5bb4158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40708d0 con 0x7f5bb415bf30 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb413dff0 con 0x7f5bb413dc10 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4070a50 con 0x7f5bb415e850 2024-03-06T07:24:48.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 0x7f5bb415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 0x7f5bb415d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39990/0 (socket says 172.21.15.122:39990) 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 -- 172.21.15.122:0/4071771700 learned_addr learned my addr 172.21.15.122:0/4071771700 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb8c40640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb413dc10 0x7f5bb413c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb9441640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb415e850 0x7f5bb4158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 -- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb415e850 msgr2=0x7f5bb4158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb415e850 0x7f5bb4158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 -- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb413dc10 msgr2=0x7f5bb413c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb413dc10 0x7f5bb413c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 -- 172.21.15.122:0/4071771700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb41f3cb0 con 0x7f5bb415bf30 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb3fff640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 0x7f5bb415d4c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000e9b0 tx=0x7f5ba000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0017070 con 0x7f5bb415bf30 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba000c9a0 con 0x7f5bb415bf30 2024-03-06T07:24:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.567+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb415dcf0 con 0x7f5bb415bf30 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.568+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba000cd90 con 0x7f5bb415bf30 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.568+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb415e2b0 con 0x7f5bb415bf30 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.568+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb404f330 con 0x7f5bb415bf30 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.569+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ba001a070 con 0x7f5bb415bf30 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.569+0000 7f5bb17fa640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b9806b500 0x7f5b9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.569+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ba008a0b0 con 0x7f5bb415bf30 2024-03-06T07:24:48.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.569+0000 7f5bb8c40640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b9806b500 0x7f5b9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.570+0000 7f5bb8c40640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b9806b500 0x7f5b9806d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8036000 tx=0x7f5ba8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.573+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ba0058970 con 0x7f5bb415bf30 2024-03-06T07:24:48.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.744+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bb406bf40 con 0x7f5bb415bf30 2024-03-06T07:24:48.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.746+0000 7f5bb17fa640 1 -- 172.21.15.122:0/4071771700 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7f5ba005ca10 con 0x7f5bb415bf30 2024-03-06T07:24:48.748 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:48.748 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.747+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b9806b500 msgr2=0x7f5b9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.747+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b9806b500 0x7f5b9806d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8036000 tx=0x7f5ba8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.747+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 msgr2=0x7f5bb415d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.747+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 0x7f5bb415d4c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000e9b0 tx=0x7f5ba000ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 shutdown_connections 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b9806b500 0x7f5b9806d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bb415e850 0x7f5bb4158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bb415bf30 0x7f5bb415d4c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 --2- 172.21.15.122:0/4071771700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb413dc10 0x7f5bb413c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 >> 172.21.15.122:0/4071771700 conn(0x7f5bb40cf8a0 msgr2=0x7f5bb40af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 shutdown_connections 2024-03-06T07:24:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.748+0000 7f5bbaea4640 1 -- 172.21.15.122:0/4071771700 wait complete. 2024-03-06T07:24:48.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:16.591753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:17.332050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:48.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:48.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.861+0000 7f0dd508d640 1 Processor -- start 2024-03-06T07:24:48.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.861+0000 7f0dd508d640 1 -- start start 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.861+0000 7f0dd508d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0dd00719e0 0x7f0dd00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.861+0000 7f0dd508d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00b3320 con 0x7f0dd006c9f0 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.861+0000 7f0dd508d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00728d0 con 0x7f0dd015a220 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.861+0000 7f0dd508d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd00cf610 con 0x7f0dd00719e0 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dc7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0dd00719e0 0x7f0dd00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dc7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0dd00719e0 0x7f0dd00cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40004/0 (socket says 172.21.15.122:40004) 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dcf59e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0dd006c9f0 0x7f0dd006ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43450/0 (socket says 172.21.15.122:43450) 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dc7fff640 1 -- 172.21.15.122:0/3460626406 learned_addr learned my addr 172.21.15.122:0/3460626406 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dc7fff640 1 -- 172.21.15.122:0/3460626406 >> v1:172.21.15.169:6789/0 conn(0x7f0dd015a220 legacy=0x7f0dd00712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dc7fff640 1 -- 172.21.15.122:0/3460626406 >> v1:172.21.15.122:6789/0 conn(0x7f0dd006c9f0 legacy=0x7f0dd006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.862+0000 7f0dc7fff640 1 -- 172.21.15.122:0/3460626406 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00b3590 con 0x7f0dd00719e0 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dc7fff640 1 --2- 172.21.15.122:0/3460626406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd00cf0d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0db800aa10 tx=0x7f0db802ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18b899259bf549 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dce59c640 1 -- 172.21.15.122:0/3460626406 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db803a070 con 0x7f0dd00719e0 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dce59c640 1 -- 172.21.15.122:0/3460626406 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db802dc10 con 0x7f0dd00719e0 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dce59c640 1 -- 172.21.15.122:0/3460626406 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db800b040 con 0x7f0dd00719e0 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 -- 172.21.15.122:0/3460626406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 msgr2=0x7f0dd00cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 --2- 172.21.15.122:0/3460626406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd00cf0d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0db800aa10 tx=0x7f0db802ce60 comp rx=0 tx=0).stop 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 -- 172.21.15.122:0/3460626406 shutdown_connections 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 --2- 172.21.15.122:0/3460626406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd00cf0d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 -- 172.21.15.122:0/3460626406 >> 172.21.15.122:0/3460626406 conn(0x7f0dd006b190 msgr2=0x7f0dd006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 -- 172.21.15.122:0/3460626406 shutdown_connections 2024-03-06T07:24:48.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.863+0000 7f0dd508d640 1 -- 172.21.15.122:0/3460626406 wait complete. 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 Processor -- start 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 -- start start 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dd006c9f0 0x7f0dd00a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd0156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dd015a220 0x7f0dd0154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0072530 con 0x7f0dd00719e0 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0072030 con 0x7f0dd015a220 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dd508d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00726b0 con 0x7f0dd006c9f0 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dc7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd0156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dc7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd0156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40016/0 (socket says 172.21.15.122:40016) 2024-03-06T07:24:48.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dc7fff640 1 -- 172.21.15.122:0/2653117053 learned_addr learned my addr 172.21.15.122:0/2653117053 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dcf59e640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dd015a220 0x7f0dd0154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.865+0000 7f0dced9d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dd006c9f0 0x7f0dd00a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc7fff640 1 -- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dd006c9f0 msgr2=0x7f0dd00a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc7fff640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dd006c9f0 0x7f0dd00a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc7fff640 1 -- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dd015a220 msgr2=0x7f0dd0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc7fff640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dd015a220 0x7f0dd0154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc7fff640 1 -- 172.21.15.122:0/2653117053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd015aae0 con 0x7f0dd00719e0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc7fff640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd0156590 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0db800a1c0 tx=0x7f0db8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db803a040 con 0x7f0dd00719e0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8040650 con 0x7f0dd00719e0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8040a40 con 0x7f0dd00719e0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd015adc0 con 0x7f0dd00719e0 2024-03-06T07:24:48.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.866+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd015b380 con 0x7f0dd00719e0 2024-03-06T07:24:48.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.867+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd00732b0 con 0x7f0dd00719e0 2024-03-06T07:24:48.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.867+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0db8047060 con 0x7f0dd00719e0 2024-03-06T07:24:48.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.868+0000 7f0dc77fe640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0db006b4d0 0x7f0db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.868+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0db80ac640 con 0x7f0dd00719e0 2024-03-06T07:24:48.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.868+0000 7f0dced9d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0db006b4d0 0x7f0db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.868+0000 7f0dced9d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0db006b4d0 0x7f0db006d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00016e0 tx=0x7f0dc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:48.872+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0db807af00 con 0x7f0dd00719e0 2024-03-06T07:24:49.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.045+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0dd0070ba0 con 0x7f0dd00719e0 2024-03-06T07:24:49.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.046+0000 7f0dc77fe640 1 -- 172.21.15.122:0/2653117053 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0db807efa0 con 0x7f0dd00719e0 2024-03-06T07:24:49.048 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0db006b4d0 msgr2=0x7f0db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0db006b4d0 0x7f0db006d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00016e0 tx=0x7f0dc0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 msgr2=0x7f0dd0156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd0156590 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0db800a1c0 tx=0x7f0db8004730 comp rx=0 tx=0).stop 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 shutdown_connections 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0db006b4d0 0x7f0db006d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dd015a220 0x7f0dd0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd00719e0 0x7f0dd0156590 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 --2- 172.21.15.122:0/2653117053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dd006c9f0 0x7f0dd00a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.047+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 >> 172.21.15.122:0/2653117053 conn(0x7f0dd006b190 msgr2=0x7f0dd0159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.048+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 shutdown_connections 2024-03-06T07:24:49.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.048+0000 7f0dd508d640 1 -- 172.21.15.122:0/2653117053 wait complete. 2024-03-06T07:24:49.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:49.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.160+0000 7ff5a822a640 1 Processor -- start 2024-03-06T07:24:49.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.161+0000 7ff5a822a640 1 -- start start 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.161+0000 7ff5a822a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a00b4970 0x7ff5a006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.161+0000 7ff5a822a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a015ff90 con 0x7ff5a006d5c0 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.161+0000 7ff5a822a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a0160270 con 0x7ff5a015bb40 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.161+0000 7ff5a822a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a00b4dd0 con 0x7ff5a00b4970 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a67c7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff5a006d5c0 0x7ff5a00cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43464/0 (socket says 172.21.15.122:43464) 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a67c7640 1 -- 172.21.15.122:0/2497942222 learned_addr learned my addr 172.21.15.122:0/2497942222 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a57c5640 1 --2- 172.21.15.122:0/2497942222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a00b4970 0x7ff5a006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112754829 0 0) 0x7ff5a015ff90 con 0x7ff5a006d5c0 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff58c003520 con 0x7ff5a006d5c0 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418794887 0 0) 0x7ff5a0160270 con 0x7ff5a015bb40 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a015ff90 con 0x7ff5a015bb40 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a57c5640 1 -- 172.21.15.122:0/2497942222 >> v1:172.21.15.169:6789/0 conn(0x7ff5a015bb40 legacy=0x7ff5a015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a57c5640 1 -- 172.21.15.122:0/2497942222 >> v1:172.21.15.122:6789/0 conn(0x7ff5a006d5c0 legacy=0x7ff5a00cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.162+0000 7ff5a57c5640 1 -- 172.21.15.122:0/2497942222 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a0160550 con 0x7ff5a00b4970 2024-03-06T07:24:49.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a57c5640 1 --2- 172.21.15.122:0/2497942222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a00b4970 0x7ff5a006d080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff5900097d0 tx=0x7ff59002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7680fadbc1a8d965 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590035020 con 0x7ff5a00b4970 2024-03-06T07:24:49.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590031890 con 0x7ff5a00b4970 2024-03-06T07:24:49.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a4fc4640 1 -- 172.21.15.122:0/2497942222 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590031c40 con 0x7ff5a00b4970 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 -- 172.21.15.122:0/2497942222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a00b4970 msgr2=0x7ff5a006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 --2- 172.21.15.122:0/2497942222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a00b4970 0x7ff5a006d080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff5900097d0 tx=0x7ff59002b930 comp rx=0 tx=0).stop 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 -- 172.21.15.122:0/2497942222 shutdown_connections 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 --2- 172.21.15.122:0/2497942222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a00b4970 0x7ff5a006d080 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 -- 172.21.15.122:0/2497942222 >> 172.21.15.122:0/2497942222 conn(0x7ff5a006b190 msgr2=0x7ff5a006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 -- 172.21.15.122:0/2497942222 shutdown_connections 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.163+0000 7ff5a822a640 1 -- 172.21.15.122:0/2497942222 wait complete. 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 Processor -- start 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 -- start start 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a006d5c0 0x7ff5a01e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 0x7ff5a00b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5a015bb40 0x7ff5a0142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a015bf20 con 0x7ff5a00b4970 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a006c910 con 0x7ff5a015bb40 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a822a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a015c0a0 con 0x7ff5a006d5c0 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a57c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 0x7ff5a00b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a57c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 0x7ff5a00b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40026/0 (socket says 172.21.15.122:40026) 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a57c5640 1 -- 172.21.15.122:0/4177403919 learned_addr learned my addr 172.21.15.122:0/4177403919 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a5fc6640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a006d5c0 0x7ff5a01e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.165+0000 7ff5a67c7640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5a015bb40 0x7ff5a0142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a57c5640 1 -- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a006d5c0 msgr2=0x7ff5a01e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a57c5640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a006d5c0 0x7ff5a01e80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a57c5640 1 -- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5a015bb40 msgr2=0x7ff5a0142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a57c5640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5a015bb40 0x7ff5a0142f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a57c5640 1 -- 172.21.15.122:0/4177403919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a01e7020 con 0x7ff5a00b4970 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a57c5640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 0x7ff5a00b28a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff590000c00 tx=0x7ff5900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590035040 con 0x7ff5a00b4970 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590030d20 con 0x7ff5a00b4970 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a01e7300 con 0x7ff5a00b4970 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590006e10 con 0x7ff5a00b4970 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.166+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a01e78c0 con 0x7ff5a00b4970 2024-03-06T07:24:49.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.167+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a0157de0 con 0x7ff5a00b4970 2024-03-06T07:24:49.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.167+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff590038020 con 0x7ff5a00b4970 2024-03-06T07:24:49.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.168+0000 7ff5967fc640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58406b4d0 0x7ff58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.168+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5900ac7b0 con 0x7ff5a00b4970 2024-03-06T07:24:49.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.168+0000 7ff5a5fc6640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58406b4d0 0x7ff58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.168+0000 7ff5a5fc6640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58406b4d0 0x7ff58406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff59c007fd0 tx=0x7ff59c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.172+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff59002d040 con 0x7ff5a00b4970 2024-03-06T07:24:49.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.343+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff5a006e130 con 0x7ff5a00b4970 2024-03-06T07:24:49.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.343+0000 7ff5967fc640 1 -- 172.21.15.122:0/4177403919 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff59007afe0 con 0x7ff5a00b4970 2024-03-06T07:24:49.345 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:24:49.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58406b4d0 msgr2=0x7ff58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58406b4d0 0x7ff58406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff59c007fd0 tx=0x7ff59c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 msgr2=0x7ff5a00b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 0x7ff5a00b28a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff590000c00 tx=0x7ff5900046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 shutdown_connections 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58406b4d0 0x7ff58406d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5a015bb40 0x7ff5a0142f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5a00b4970 0x7ff5a00b28a0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 --2- 172.21.15.122:0/4177403919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5a006d5c0 0x7ff5a01e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 >> 172.21.15.122:0/4177403919 conn(0x7ff5a006b190 msgr2=0x7ff5a01595b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 shutdown_connections 2024-03-06T07:24:49.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.345+0000 7ff5a822a640 1 -- 172.21.15.122:0/4177403919 wait complete. 2024-03-06T07:24:49.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 Processor -- start 2024-03-06T07:24:49.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 -- start start 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4901614d0 0x7fe4900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe4900d2480 0x7fe4900b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4901618b0 con 0x7fe4900d1910 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4900cf400 con 0x7fe4900d2480 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe49593e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900cf6a0 con 0x7fe4901614d0 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.460+0000 7fe48f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe4900d2480 0x7fe4900b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48e7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe4900d1910 0x7fe4900d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43478/0 (socket says 172.21.15.122:43478) 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48e7fc640 1 -- 172.21.15.122:0/2720420595 learned_addr learned my addr 172.21.15.122:0/2720420595 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:49.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48effd640 1 --2- 172.21.15.122:0/2720420595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4901614d0 0x7fe4900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48dffb640 1 -- 172.21.15.122:0/2720420595 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037267954 0 0) 0x7fe4901618b0 con 0x7fe4900d1910 2024-03-06T07:24:49.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48dffb640 1 -- 172.21.15.122:0/2720420595 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe478003520 con 0x7fe4900d1910 2024-03-06T07:24:49.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2720420595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4901614d0 msgr2=0x7fe4900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48f7fe640 1 --2- 172.21.15.122:0/2720420595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4901614d0 0x7fe4900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2720420595 >> v1:172.21.15.122:6789/0 conn(0x7fe4900d1910 legacy=0x7fe4900d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.461+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2720420595 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900cf980 con 0x7fe4900d2480 2024-03-06T07:24:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe48f7fe640 1 --2- 172.21.15.122:0/2720420595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900d2480 0x7fe4900b0e30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe488009d90 tx=0x7fe48802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ee8b40f039e369 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe48dffb640 1 -- 172.21.15.122:0/2720420595 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488039070 con 0x7fe4900d2480 2024-03-06T07:24:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe48dffb640 1 -- 172.21.15.122:0/2720420595 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48802cc50 con 0x7fe4900d2480 2024-03-06T07:24:49.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe48dffb640 1 -- 172.21.15.122:0/2720420595 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488034440 con 0x7fe4900d2480 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 -- 172.21.15.122:0/2720420595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900d2480 msgr2=0x7fe4900b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 --2- 172.21.15.122:0/2720420595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900d2480 0x7fe4900b0e30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe488009d90 tx=0x7fe48802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 -- 172.21.15.122:0/2720420595 shutdown_connections 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 --2- 172.21.15.122:0/2720420595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900d2480 0x7fe4900b0e30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 --2- 172.21.15.122:0/2720420595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4901614d0 0x7fe4900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 -- 172.21.15.122:0/2720420595 >> 172.21.15.122:0/2720420595 conn(0x7fe4900d3d40 msgr2=0x7fe4900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 -- 172.21.15.122:0/2720420595 shutdown_connections 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.462+0000 7fe49593e640 1 -- 172.21.15.122:0/2720420595 wait complete. 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.463+0000 7fe49593e640 1 Processor -- start 2024-03-06T07:24:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 -- start start 2024-03-06T07:24:49.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4900d1910 0x7fe4900ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4900d2480 0x7fe49015d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 0x7fe4900b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900b1370 con 0x7fe4901614d0 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900b0910 con 0x7fe4900d1910 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe49593e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900b14f0 con 0x7fe4900d2480 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 0x7fe4900b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 0x7fe4900b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40050/0 (socket says 172.21.15.122:40050) 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.464+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2461309226 learned_addr learned my addr 172.21.15.122:0/2461309226 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48e7fc640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4900d2480 0x7fe49015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48effd640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4900d1910 0x7fe4900ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4900d2480 msgr2=0x7fe49015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48f7fe640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4900d2480 0x7fe49015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4900d1910 msgr2=0x7fe4900ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48f7fe640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4900d1910 0x7fe4900ca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48f7fe640 1 -- 172.21.15.122:0/2461309226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900b40d0 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe48f7fe640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 0x7fe4900b73d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe488009d90 tx=0x7fe488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488039050 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe488045070 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4900b4390 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488009450 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.465+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4900b4950 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.466+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490068ff0 con 0x7fe4901614d0 2024-03-06T07:24:49.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.467+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe48803e610 con 0x7fe4901614d0 2024-03-06T07:24:49.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.467+0000 7fe477fff640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe47c06b4d0 0x7fe47c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.467+0000 7fe48effd640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe47c06b4d0 0x7fe47c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.467+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4880aca10 con 0x7fe4901614d0 2024-03-06T07:24:49.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.468+0000 7fe48effd640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe47c06b4d0 0x7fe47c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe480009aa0 tx=0x7fe480009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.472+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe48807b2d0 con 0x7fe4901614d0 2024-03-06T07:24:49.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.642+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4900b4b20 con 0x7fe4901614d0 2024-03-06T07:24:49.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.642+0000 7fe477fff640 1 -- 172.21.15.122:0/2461309226 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe48807f370 con 0x7fe4901614d0 2024-03-06T07:24:49.645 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:24:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe47c06b4d0 msgr2=0x7fe47c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe47c06b4d0 0x7fe47c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe480009aa0 tx=0x7fe480009190 comp rx=0 tx=0).stop 2024-03-06T07:24:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 msgr2=0x7fe4900b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 0x7fe4900b73d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe488009d90 tx=0x7fe488004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 shutdown_connections 2024-03-06T07:24:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe47c06b4d0 0x7fe47c06d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4901614d0 0x7fe4900b73d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe4900d2480 0x7fe49015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 --2- 172.21.15.122:0/2461309226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4900d1910 0x7fe4900ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.644+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 >> 172.21.15.122:0/2461309226 conn(0x7fe4900d3d40 msgr2=0x7fe4900b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.645+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 shutdown_connections 2024-03-06T07:24:49.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.645+0000 7fe49593e640 1 -- 172.21.15.122:0/2461309226 wait complete. 2024-03-06T07:24:49.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:49.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.756+0000 7fbd66532640 1 Processor -- start 2024-03-06T07:24:49.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.756+0000 7fbd66532640 1 -- start start 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd66532640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbd60162400 0x7fbd600d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd66532640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600d22c0 con 0x7fbd600b4460 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd66532640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600d25a0 con 0x7fbd60162970 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd66532640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd60161b00 con 0x7fbd60162400 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd64acf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbd60162400 0x7fbd600d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd5ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbd600b4460 0x7fbd600b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43484/0 (socket says 172.21.15.122:43484) 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd5ffff640 1 -- 172.21.15.122:0/3506183049 learned_addr learned my addr 172.21.15.122:0/3506183049 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd64acf640 1 -- 172.21.15.122:0/3506183049 >> v1:172.21.15.169:6789/0 conn(0x7fbd60162970 legacy=0x7fbd60161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd64acf640 1 -- 172.21.15.122:0/3506183049 >> v1:172.21.15.122:6789/0 conn(0x7fbd600b4460 legacy=0x7fbd600b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd5effd640 1 -- 172.21.15.122:0/3506183049 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646545885 0 0) 0x7fbd600d22c0 con 0x7fbd600b4460 2024-03-06T07:24:49.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd64acf640 1 -- 172.21.15.122:0/3506183049 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd600d2880 con 0x7fbd60162400 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.758+0000 7fbd64acf640 1 --2- 172.21.15.122:0/3506183049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd60162400 0x7fbd600d4800 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009d90 tx=0x7fbd5802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab373839bfbe69ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd5effd640 1 -- 172.21.15.122:0/3506183049 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58039070 con 0x7fbd60162400 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd5effd640 1 -- 172.21.15.122:0/3506183049 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5802cd00 con 0x7fbd60162400 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd5effd640 1 -- 172.21.15.122:0/3506183049 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58034660 con 0x7fbd60162400 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 -- 172.21.15.122:0/3506183049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd60162400 msgr2=0x7fbd600d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 --2- 172.21.15.122:0/3506183049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd60162400 0x7fbd600d4800 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009d90 tx=0x7fbd5802bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 -- 172.21.15.122:0/3506183049 shutdown_connections 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 --2- 172.21.15.122:0/3506183049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd60162400 0x7fbd600d4800 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 -- 172.21.15.122:0/3506183049 >> 172.21.15.122:0/3506183049 conn(0x7fbd600d3ce0 msgr2=0x7fbd600d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 -- 172.21.15.122:0/3506183049 shutdown_connections 2024-03-06T07:24:49.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.759+0000 7fbd66532640 1 -- 172.21.15.122:0/3506183049 wait complete. 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.760+0000 7fbd66532640 1 Processor -- start 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 -- start start 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 0x7fbd60155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60162400 0x7fbd600cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd60162970 0x7fbd600b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600ce950 con 0x7fbd600b4460 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600cecc0 con 0x7fbd60162400 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd66532640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600b4840 con 0x7fbd60162970 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd5ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 0x7fbd60155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd5ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 0x7fbd60155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40070/0 (socket says 172.21.15.122:40070) 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd5ffff640 1 -- 172.21.15.122:0/2319989165 learned_addr learned my addr 172.21.15.122:0/2319989165 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd64acf640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd60162970 0x7fbd600b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.761+0000 7fbd5f7fe640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60162400 0x7fbd600cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5ffff640 1 -- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd60162970 msgr2=0x7fbd600b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5ffff640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd60162970 0x7fbd600b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5ffff640 1 -- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60162400 msgr2=0x7fbd600cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5ffff640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60162400 0x7fbd600cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5ffff640 1 -- 172.21.15.122:0/2319989165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd600c8720 con 0x7fbd600b4460 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5ffff640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 0x7fbd60155f00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c3f0 tx=0x7fbd5000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd500074e0 con 0x7fbd600b4460 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd500047e0 con 0x7fbd600b4460 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd600c8a00 con 0x7fbd600b4460 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50004c10 con 0x7fbd600b4460 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.762+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd600c8fc0 con 0x7fbd600b4460 2024-03-06T07:24:49.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.763+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6004f330 con 0x7fbd600b4460 2024-03-06T07:24:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.764+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd50011070 con 0x7fbd600b4460 2024-03-06T07:24:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.764+0000 7fbd5cff9640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd4c06b4d0 0x7fbd4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.764+0000 7fbd5f7fe640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd4c06b4d0 0x7fbd4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.764+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd50089c10 con 0x7fbd600b4460 2024-03-06T07:24:49.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.764+0000 7fbd5f7fe640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd4c06b4d0 0x7fbd4c06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbd6015d440 tx=0x7fbd54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.768+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd500584d0 con 0x7fbd600b4460 2024-03-06T07:24:49.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.952+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd600caa70 con 0x7fbd600b4460 2024-03-06T07:24:49.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.952+0000 7fbd5cff9640 1 -- 172.21.15.122:0/2319989165 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbd5005c570 con 0x7fbd600b4460 2024-03-06T07:24:49.954 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:24:49.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd4c06b4d0 msgr2=0x7fbd4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd4c06b4d0 0x7fbd4c06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbd6015d440 tx=0x7fbd54009040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 msgr2=0x7fbd60155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 0x7fbd60155f00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c3f0 tx=0x7fbd5000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 shutdown_connections 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd4c06b4d0 0x7fbd4c06d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd60162970 0x7fbd600b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60162400 0x7fbd600cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 --2- 172.21.15.122:0/2319989165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd600b4460 0x7fbd60155f00 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 >> 172.21.15.122:0/2319989165 conn(0x7fbd600d3ce0 msgr2=0x7fbd600d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 shutdown_connections 2024-03-06T07:24:49.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:49.954+0000 7fbd66532640 1 -- 172.21.15.122:0/2319989165 wait complete. 2024-03-06T07:24:49.966 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:24:49.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:49.967 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:49.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[59993]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:49.994 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:24:49.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:49.994 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:50.143 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:24:50.143 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:50.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:50.159 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:24:50.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:50.159 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:50.213 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:50.214 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:24:50.214 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:50.229 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:50.230 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:50.230 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:50.284 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:50.284 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:24:50.284 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:50.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:50.300 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:24:50.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:50.301 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:50.355 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:50.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:50.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.453+0000 7f3f54108640 1 Processor -- start 2024-03-06T07:24:50.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.453+0000 7f3f54108640 1 -- start start 2024-03-06T07:24:50.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.453+0000 7f3f54108640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3f4c06b220 0x7f3f4c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.453+0000 7f3f54108640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3f4c06b600 0x7f3f4c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f54108640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c15ea30 con 0x7f3f4c0ce920 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f54108640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c161b70 con 0x7f3f4c06b600 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f54108640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c15ed10 con 0x7f3f4c06b220 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f526a5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3f4c06b600 0x7f3f4c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f516a3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3f4c0ce920 0x7f3f4c071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43490/0 (socket says 172.21.15.122:43490) 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f526a5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3f4c06b600 0x7f3f4c162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40076/0 (socket says 172.21.15.122:40076) 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f516a3640 1 -- 172.21.15.122:0/981084663 learned_addr learned my addr 172.21.15.122:0/981084663 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f51ea4640 1 --2- 172.21.15.122:0/981084663 >> v2:172.21.15.169:3300/0 conn(0x7f3f4c06b220 0x7f3f4c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f526a5640 1 -- 172.21.15.122:0/981084663 >> v2:172.21.15.169:3300/0 conn(0x7f3f4c06b220 msgr2=0x7f3f4c15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f526a5640 1 --2- 172.21.15.122:0/981084663 >> v2:172.21.15.169:3300/0 conn(0x7f3f4c06b220 0x7f3f4c15e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f526a5640 1 -- 172.21.15.122:0/981084663 >> v1:172.21.15.122:6789/0 conn(0x7f3f4c0ce920 legacy=0x7f3f4c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f50ea2640 1 -- 172.21.15.122:0/981084663 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211850387 0 0) 0x7f3f4c15ea30 con 0x7f3f4c0ce920 2024-03-06T07:24:50.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.454+0000 7f3f526a5640 1 -- 172.21.15.122:0/981084663 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c15eff0 con 0x7f3f4c06b600 2024-03-06T07:24:50.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f526a5640 1 --2- 172.21.15.122:0/981084663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c06b600 0x7f3f4c162000 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0d0980 tx=0x7f3f4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71b8fb4488454d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f50ea2640 1 -- 172.21.15.122:0/981084663 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44039070 con 0x7f3f4c06b600 2024-03-06T07:24:50.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f50ea2640 1 -- 172.21.15.122:0/981084663 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4402cc80 con 0x7f3f4c06b600 2024-03-06T07:24:50.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f50ea2640 1 -- 172.21.15.122:0/981084663 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44034660 con 0x7f3f4c06b600 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f54108640 1 -- 172.21.15.122:0/981084663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c06b600 msgr2=0x7f3f4c162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f54108640 1 --2- 172.21.15.122:0/981084663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c06b600 0x7f3f4c162000 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0d0980 tx=0x7f3f4402bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f54108640 1 -- 172.21.15.122:0/981084663 shutdown_connections 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f54108640 1 --2- 172.21.15.122:0/981084663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c06b600 0x7f3f4c162000 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f54108640 1 --2- 172.21.15.122:0/981084663 >> v2:172.21.15.169:3300/0 conn(0x7f3f4c06b220 0x7f3f4c15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.455+0000 7f3f54108640 1 -- 172.21.15.122:0/981084663 >> 172.21.15.122:0/981084663 conn(0x7f3f4c06c0c0 msgr2=0x7f3f4c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.456+0000 7f3f54108640 1 -- 172.21.15.122:0/981084663 shutdown_connections 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.456+0000 7f3f54108640 1 -- 172.21.15.122:0/981084663 wait complete. 2024-03-06T07:24:50.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 Processor -- start 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 -- start start 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f4c06b220 0x7f3f4c1484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3f4c06b600 0x7f3f4c1f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 0x7f3f4c1f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c072090 con 0x7f3f4c0ce920 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0ce200 con 0x7f3f4c06b220 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f54108640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c1489e0 con 0x7f3f4c06b600 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.457+0000 7f3f526a5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 0x7f3f4c1f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 0x7f3f4c1f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40082/0 (socket says 172.21.15.122:40082) 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f51ea4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f4c06b220 0x7f3f4c1484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 -- 172.21.15.122:0/806401490 learned_addr learned my addr 172.21.15.122:0/806401490 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f516a3640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3f4c06b600 0x7f3f4c1f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 -- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3f4c06b600 msgr2=0x7f3f4c1f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3f4c06b600 0x7f3f4c1f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 -- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f4c06b220 msgr2=0x7f3f4c1484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f4c06b220 0x7f3f4c1484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 -- 172.21.15.122:0/806401490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c161050 con 0x7f3f4c0ce920 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f51ea4640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f4c06b220 0x7f3f4c1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f526a5640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 0x7f3f4c1f7750 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3f4402c4e0 tx=0x7f3f44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44039050 con 0x7f3f4c0ce920 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44045070 con 0x7f3f4c0ce920 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c0adda0 con 0x7f3f4c0ce920 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44009450 con 0x7f3f4c0ce920 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.458+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c0ae360 con 0x7f3f4c0ce920 2024-03-06T07:24:50.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.459+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c04f330 con 0x7f3f4c0ce920 2024-03-06T07:24:50.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.459+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f4403e850 con 0x7f3f4c0ce920 2024-03-06T07:24:50.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.461+0000 7f3f3e7fc640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f3006b4d0 0x7f3f3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.461+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f440ad130 con 0x7f3f4c0ce920 2024-03-06T07:24:50.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.461+0000 7f3f51ea4640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f3006b4d0 0x7f3f3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.461+0000 7f3f51ea4640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f3006b4d0 0x7f3f3006d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009e90 tx=0x7f3f40009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.464+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f4407bb10 con 0x7f3f4c0ce920 2024-03-06T07:24:50.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.672+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f4c0cdf00 con 0x7f3f4c0ce920 2024-03-06T07:24:50.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.672+0000 7f3f3e7fc640 1 -- 172.21.15.122:0/806401490 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f3f4407fbb0 con 0x7f3f4c0ce920 2024-03-06T07:24:50.674 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:50.674 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:50.675 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:50.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f3006b4d0 msgr2=0x7f3f3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f3006b4d0 0x7f3f3006d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009e90 tx=0x7f3f40009190 comp rx=0 tx=0).stop 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 msgr2=0x7f3f4c1f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 0x7f3f4c1f7750 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3f4402c4e0 tx=0x7f3f44004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 shutdown_connections 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f3006b4d0 0x7f3f3006d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f4c0ce920 0x7f3f4c1f7750 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3f4c06b600 0x7f3f4c1f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 --2- 172.21.15.122:0/806401490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f4c06b220 0x7f3f4c1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 >> 172.21.15.122:0/806401490 conn(0x7f3f4c06c0c0 msgr2=0x7f3f4c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 shutdown_connections 2024-03-06T07:24:50.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.674+0000 7f3f54108640 1 -- 172.21.15.122:0/806401490 wait complete. 2024-03-06T07:24:50.686 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.785+0000 7f20d7015640 1 Processor -- start 2024-03-06T07:24:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.785+0000 7f20d7015640 1 -- start start 2024-03-06T07:24:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20d7015640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f20d00b3060 0x7f20d015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20d7015640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d015bf30 0x7f20d015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20d7015640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d015f2d0 con 0x7f20d013dc10 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20d7015640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d00cf3f0 con 0x7f20d00b3060 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20d7015640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d015b210 con 0x7f20d015bf30 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f20d00b3060 0x7f20d015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f20d00b3060 0x7f20d015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40094/0 (socket says 172.21.15.122:40094) 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 -- 172.21.15.122:0/2496418726 learned_addr learned my addr 172.21.15.122:0/2496418726 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20d55b2640 1 --2- 172.21.15.122:0/2496418726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d015bf30 0x7f20d015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 -- 172.21.15.122:0/2496418726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d015bf30 msgr2=0x7f20d015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 --2- 172.21.15.122:0/2496418726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d015bf30 0x7f20d015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 -- 172.21.15.122:0/2496418726 >> v1:172.21.15.169:6789/0 conn(0x7f20d013dc10 legacy=0x7f20d00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.786+0000 7f20cffff640 1 -- 172.21.15.122:0/2496418726 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d015f580 con 0x7f20d00b3060 2024-03-06T07:24:50.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.787+0000 7f20cffff640 1 --2- 172.21.15.122:0/2496418726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d015e850 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f20c000bb10 tx=0x7f20c002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d322b0ad487343b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.787+0000 7f20cf7fe640 1 -- 172.21.15.122:0/2496418726 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c003b070 con 0x7f20d00b3060 2024-03-06T07:24:50.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.787+0000 7f20cf7fe640 1 -- 172.21.15.122:0/2496418726 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0036440 con 0x7f20d00b3060 2024-03-06T07:24:50.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.787+0000 7f20cf7fe640 1 -- 172.21.15.122:0/2496418726 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0036810 con 0x7f20d00b3060 2024-03-06T07:24:50.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 -- 172.21.15.122:0/2496418726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 msgr2=0x7f20d015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 --2- 172.21.15.122:0/2496418726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d015e850 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f20c000bb10 tx=0x7f20c002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 -- 172.21.15.122:0/2496418726 shutdown_connections 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 --2- 172.21.15.122:0/2496418726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d015bf30 0x7f20d015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 --2- 172.21.15.122:0/2496418726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d015e850 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 -- 172.21.15.122:0/2496418726 >> 172.21.15.122:0/2496418726 conn(0x7f20d00cf8a0 msgr2=0x7f20d00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 -- 172.21.15.122:0/2496418726 shutdown_connections 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.788+0000 7f20d7015640 1 -- 172.21.15.122:0/2496418726 wait complete. 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.789+0000 7f20d7015640 1 Processor -- start 2024-03-06T07:24:50.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.789+0000 7f20d7015640 1 -- start start 2024-03-06T07:24:50.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d7015640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d01408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d7015640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d013dc10 0x7f20d00b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d7015640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d015bf30 0x7f20d00b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d7015640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d013dff0 con 0x7f20d00b3060 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d7015640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d015b730 con 0x7f20d015bf30 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d7015640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d00cd7a0 con 0x7f20d013dc10 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d01408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d01408c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40102/0 (socket says 172.21.15.122:40102) 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 -- 172.21.15.122:0/605018588 learned_addr learned my addr 172.21.15.122:0/605018588 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20cffff640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d013dc10 0x7f20d00b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d55b2640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d015bf30 0x7f20d00b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 -- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d013dc10 msgr2=0x7f20d00b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d013dc10 0x7f20d00b0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 -- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d015bf30 msgr2=0x7f20d00b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d015bf30 0x7f20d00b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 -- 172.21.15.122:0/605018588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d013edc0 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d55b2640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d015bf30 0x7f20d00b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20d4db1640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d01408c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f20c800c790 tx=0x7f20c800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.790+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8016020 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.791+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8006d50 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.791+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d01f34e0 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.791+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8003bb0 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.791+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d0140e00 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.791+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d004f330 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.792+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20c800a850 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.792+0000 7f20cd7fa640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f20b006b4d0 0x7f20b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.792+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20c8089dc0 con 0x7f20d00b3060 2024-03-06T07:24:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.792+0000 7f20cffff640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f20b006b4d0 0x7f20b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.793+0000 7f20cffff640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f20b006b4d0 0x7f20b006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f20c0038000 tx=0x7f20c0032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.797+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20c8020020 con 0x7f20d00b3060 2024-03-06T07:24:50.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.969+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20d015ad30 con 0x7f20d00b3060 2024-03-06T07:24:50.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.970+0000 7f20cd7fa640 1 -- 172.21.15.122:0/605018588 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7f20c8058cd0 con 0x7f20d00b3060 2024-03-06T07:24:50.971 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:50.971 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:50.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f20b006b4d0 msgr2=0x7f20b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f20b006b4d0 0x7f20b006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f20c0038000 tx=0x7f20c0032000 comp rx=0 tx=0).stop 2024-03-06T07:24:50.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 msgr2=0x7f20d01408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d01408c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f20c800c790 tx=0x7f20c800cc50 comp rx=0 tx=0).stop 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 shutdown_connections 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f20b006b4d0 0x7f20b006d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d015bf30 0x7f20d00b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f20d013dc10 0x7f20d00b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 --2- 172.21.15.122:0/605018588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20d00b3060 0x7f20d01408c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 >> 172.21.15.122:0/605018588 conn(0x7f20d00cf8a0 msgr2=0x7f20d0154d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 shutdown_connections 2024-03-06T07:24:50.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:50.972+0000 7f20d7015640 1 -- 172.21.15.122:0/605018588 wait complete. 2024-03-06T07:24:50.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:16.591753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:17.332050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:50.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:51.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.088+0000 7f71a4888640 1 Processor -- start 2024-03-06T07:24:51.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.088+0000 7f71a4888640 1 -- start start 2024-03-06T07:24:51.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f71a4888640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f71a01586e0 0x7f71a00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f71a4888640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f71a4888640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71a00cffd0 0x7f71a006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f71a4888640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a00d03f0 con 0x7f71a00cffd0 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f71a4888640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a0158ac0 con 0x7f71a01586e0 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f71a4888640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a015b2a0 con 0x7f71a00aef80 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.089+0000 7f719ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71a00cffd0 0x7f71a006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71a00cffd0 0x7f71a006da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40112/0 (socket says 172.21.15.122:40112) 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 -- 172.21.15.122:0/2426295517 learned_addr learned my addr 172.21.15.122:0/2426295517 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719dd9b640 1 --2- 172.21.15.122:0/2426295517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719e59c640 1 --2- 172.21.15.122:0/2426295517 >> v2:172.21.15.169:3300/0 conn(0x7f71a01586e0 0x7f71a00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 -- 172.21.15.122:0/2426295517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 msgr2=0x7f71a00cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 --2- 172.21.15.122:0/2426295517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 -- 172.21.15.122:0/2426295517 >> v2:172.21.15.169:3300/0 conn(0x7f71a01586e0 msgr2=0x7f71a00aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 --2- 172.21.15.122:0/2426295517 >> v2:172.21.15.169:3300/0 conn(0x7f71a01586e0 0x7f71a00aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 -- 172.21.15.122:0/2426295517 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a015b540 con 0x7f71a00cffd0 2024-03-06T07:24:51.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.090+0000 7f719ed9d640 1 --2- 172.21.15.122:0/2426295517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006da20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f71a0070d10 tx=0x7f719002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18bc3dab79f8f119 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f719d59a640 1 -- 172.21.15.122:0/2426295517 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190035020 con 0x7f71a00cffd0 2024-03-06T07:24:51.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f719d59a640 1 -- 172.21.15.122:0/2426295517 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7190031870 con 0x7f71a00cffd0 2024-03-06T07:24:51.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f719d59a640 1 -- 172.21.15.122:0/2426295517 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190031c40 con 0x7f71a00cffd0 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 -- 172.21.15.122:0/2426295517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 msgr2=0x7f71a006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 --2- 172.21.15.122:0/2426295517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006da20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f71a0070d10 tx=0x7f719002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 -- 172.21.15.122:0/2426295517 shutdown_connections 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 --2- 172.21.15.122:0/2426295517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006da20 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 --2- 172.21.15.122:0/2426295517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 --2- 172.21.15.122:0/2426295517 >> v2:172.21.15.169:3300/0 conn(0x7f71a01586e0 0x7f71a00aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 -- 172.21.15.122:0/2426295517 >> 172.21.15.122:0/2426295517 conn(0x7f71a006b190 msgr2=0x7f71a006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 -- 172.21.15.122:0/2426295517 shutdown_connections 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.091+0000 7f71a4888640 1 -- 172.21.15.122:0/2426295517 wait complete. 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.092+0000 7f71a4888640 1 Processor -- start 2024-03-06T07:24:51.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 -- start start 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71a01586e0 0x7f71a014b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a015bde0 con 0x7f71a00cffd0 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a015bb00 con 0x7f71a01586e0 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f71a4888640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a0071350 con 0x7f71a00aef80 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f719dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f719dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40124/0 (socket says 172.21.15.122:40124) 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f719dd9b640 1 -- 172.21.15.122:0/3944910173 learned_addr learned my addr 172.21.15.122:0/3944910173 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f719e59c640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.093+0000 7f719ed9d640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71a01586e0 0x7f71a014b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f719dd9b640 1 -- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 msgr2=0x7f71a00ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f719dd9b640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f719dd9b640 1 -- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71a01586e0 msgr2=0x7f71a014b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f719dd9b640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71a01586e0 0x7f71a014b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f719dd9b640 1 -- 172.21.15.122:0/3944910173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a00ce790 con 0x7f71a00cffd0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f719dd9b640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006e4c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f719400eed0 tx=0x7f719400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194018020 con 0x7f71a00cffd0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7194014400 con 0x7f71a00cffd0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940147d0 con 0x7f71a00cffd0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a00cd110 con 0x7f71a00cffd0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.094+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a00cd6d0 con 0x7f71a00cffd0 2024-03-06T07:24:51.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.095+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a00732b0 con 0x7f71a00cffd0 2024-03-06T07:24:51.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.095+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7194005d10 con 0x7f71a00cffd0 2024-03-06T07:24:51.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.096+0000 7f718effd640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f718006b4d0 0x7f718006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.096+0000 7f719e59c640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f718006b4d0 0x7f718006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.096+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f719408a7b0 con 0x7f71a00cffd0 2024-03-06T07:24:51.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.096+0000 7f719e59c640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f718006b4d0 0x7f718006d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f71a015a5b0 tx=0x7f7198009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.100+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7194022070 con 0x7f71a00cffd0 2024-03-06T07:24:51.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.270+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71a0068ff0 con 0x7f71a00cffd0 2024-03-06T07:24:51.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.271+0000 7f718effd640 1 -- 172.21.15.122:0/3944910173 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7194058fe0 con 0x7f71a00cffd0 2024-03-06T07:24:51.272 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f718006b4d0 msgr2=0x7f718006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f718006b4d0 0x7f718006d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f71a015a5b0 tx=0x7f7198009190 comp rx=0 tx=0).stop 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 msgr2=0x7f71a006e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006e4c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f719400eed0 tx=0x7f719400b480 comp rx=0 tx=0).stop 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 shutdown_connections 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f718006b4d0 0x7f718006d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71a01586e0 0x7f71a014b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71a00cffd0 0x7f71a006e4c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 --2- 172.21.15.122:0/3944910173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71a00aef80 0x7f71a00ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.272+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 >> 172.21.15.122:0/3944910173 conn(0x7f71a006b190 msgr2=0x7f71a0155780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.273+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 shutdown_connections 2024-03-06T07:24:51.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.273+0000 7f71a4888640 1 -- 172.21.15.122:0/3944910173 wait complete. 2024-03-06T07:24:51.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:51.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.386+0000 7f5ec515e640 1 Processor -- start 2024-03-06T07:24:51.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.386+0000 7f5ec515e640 1 -- start start 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ec515e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5ec0066d90 0x7f5ec00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ec515e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec00b5410 0x7f5ec00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ec515e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec006b710 con 0x7f5ec00cf9a0 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ec515e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec00b61c0 con 0x7f5ec0066d90 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ec515e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec006b150 con 0x7f5ec00b5410 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5ec0066d90 0x7f5ec00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5ec0066d90 0x7f5ec00b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40132/0 (socket says 172.21.15.122:40132) 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 -- 172.21.15.122:0/452776675 learned_addr learned my addr 172.21.15.122:0/452776675 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebed9d640 1 --1- 172.21.15.122:0/452776675 >> v1:172.21.15.169:6789/0 conn(0x7f5ec00cf9a0 0x7f5ec00b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:43674/0 (socket says 172.21.15.122:43674) 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebf59e640 1 --2- 172.21.15.122:0/452776675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec00b5410 0x7f5ec00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 -- 172.21.15.122:0/452776675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec00b5410 msgr2=0x7f5ec00ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 --2- 172.21.15.122:0/452776675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec00b5410 0x7f5ec00ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 -- 172.21.15.122:0/452776675 >> v1:172.21.15.169:6789/0 conn(0x7f5ec00cf9a0 legacy=0x7f5ec00b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.387+0000 7f5ebe59c640 1 -- 172.21.15.122:0/452776675 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec006b430 con 0x7f5ec0066d90 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.388+0000 7f5ebe59c640 1 --2- 172.21.15.122:0/452776675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec0066d90 0x7f5ec00b4ed0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009870 tx=0x7f5eb402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0025ac5752e828 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.388+0000 7f5ebdd9b640 1 -- 172.21.15.122:0/452776675 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4034070 con 0x7f5ec0066d90 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.388+0000 7f5ebdd9b640 1 -- 172.21.15.122:0/452776675 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4030840 con 0x7f5ec0066d90 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.388+0000 7f5ebdd9b640 1 -- 172.21.15.122:0/452776675 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4030c10 con 0x7f5ec0066d90 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 -- 172.21.15.122:0/452776675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec0066d90 msgr2=0x7f5ec00b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 --2- 172.21.15.122:0/452776675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec0066d90 0x7f5ec00b4ed0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009870 tx=0x7f5eb402b980 comp rx=0 tx=0).stop 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 -- 172.21.15.122:0/452776675 shutdown_connections 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 --2- 172.21.15.122:0/452776675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec00b5410 0x7f5ec00ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 --2- 172.21.15.122:0/452776675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec0066d90 0x7f5ec00b4ed0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 -- 172.21.15.122:0/452776675 >> 172.21.15.122:0/452776675 conn(0x7f5ec00b1d70 msgr2=0x7f5ec00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 -- 172.21.15.122:0/452776675 shutdown_connections 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.389+0000 7f5ec515e640 1 -- 172.21.15.122:0/452776675 wait complete. 2024-03-06T07:24:51.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.390+0000 7f5ec515e640 1 Processor -- start 2024-03-06T07:24:51.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.390+0000 7f5ec515e640 1 -- start start 2024-03-06T07:24:51.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ec515e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec0066d90 0x7f5ec0156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ec515e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 0x7f5ec0157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ec515e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ec00cf9a0 0x7f5ec00c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ec515e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00cfd80 con 0x7f5ec00b5410 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ec515e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00b0c50 con 0x7f5ec00cf9a0 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ec515e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00cff00 con 0x7f5ec0066d90 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 0x7f5ec0157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 0x7f5ec0157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40142/0 (socket says 172.21.15.122:40142) 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 -- 172.21.15.122:0/3950763521 learned_addr learned my addr 172.21.15.122:0/3950763521 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebf59e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ec00cf9a0 0x7f5ec00c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebed9d640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec0066d90 0x7f5ec0156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 -- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec0066d90 msgr2=0x7f5ec0156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec0066d90 0x7f5ec0156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 -- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ec00cf9a0 msgr2=0x7f5ec00c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ec00cf9a0 0x7f5ec00c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 -- 172.21.15.122:0/3950763521 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec00c5160 con 0x7f5ec00b5410 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ebe59c640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 0x7f5ec0157360 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f5eb402b9c0 tx=0x7f5eb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.391+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4048070 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.392+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4030640 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.392+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4034040 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.392+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec00c9fb0 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.392+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec00c5420 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.392+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec009e3d0 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.393+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5eb4037070 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.393+0000 7f5ea77fe640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5eac06b4d0 0x7f5eac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.393+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5eb40ac030 con 0x7f5ec00b5410 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.393+0000 7f5ebed9d640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5eac06b4d0 0x7f5eac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.394+0000 7f5ebed9d640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5eac06b4d0 0x7f5eac06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0007fd0 tx=0x7f5eb0007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.398+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5eb407a900 con 0x7f5ec00b5410 2024-03-06T07:24:51.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.569+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ec0068ff0 con 0x7f5ec00b5410 2024-03-06T07:24:51.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.569+0000 7f5ea77fe640 1 -- 172.21.15.122:0/3950763521 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5eb407e9a0 con 0x7f5ec00b5410 2024-03-06T07:24:51.570 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:24:51.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5eac06b4d0 msgr2=0x7f5eac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5eac06b4d0 0x7f5eac06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0007fd0 tx=0x7f5eb0007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:51.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 msgr2=0x7f5ec0157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 0x7f5ec0157360 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f5eb402b9c0 tx=0x7f5eb4004730 comp rx=0 tx=0).stop 2024-03-06T07:24:51.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 shutdown_connections 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5eac06b4d0 0x7f5eac06d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ec00cf9a0 0x7f5ec00c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec00b5410 0x7f5ec0157360 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 --2- 172.21.15.122:0/3950763521 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ec0066d90 0x7f5ec0156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 >> 172.21.15.122:0/3950763521 conn(0x7f5ec00b1d70 msgr2=0x7f5ec006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 shutdown_connections 2024-03-06T07:24:51.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.571+0000 7f5ec515e640 1 -- 172.21.15.122:0/3950763521 wait complete. 2024-03-06T07:24:51.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:51.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.685+0000 7f0959b93640 1 Processor -- start 2024-03-06T07:24:51.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.685+0000 7f0959b93640 1 -- start start 2024-03-06T07:24:51.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0959b93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0954066d90 0x7f09540b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0959b93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f09540ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0959b93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095406b710 con 0x7f09540cf9a0 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0959b93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540b61c0 con 0x7f0954066d90 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0959b93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095406b150 con 0x7f09540b5410 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0954066d90 0x7f09540b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0954066d90 0x7f09540b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40156/0 (socket says 172.21.15.122:40156) 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 -- 172.21.15.122:0/2659325412 learned_addr learned my addr 172.21.15.122:0/2659325412 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0953fff640 1 --2- 172.21.15.122:0/2659325412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f09540ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 -- 172.21.15.122:0/2659325412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 msgr2=0x7f09540ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 --2- 172.21.15.122:0/2659325412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f09540ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 -- 172.21.15.122:0/2659325412 >> v1:172.21.15.169:6789/0 conn(0x7f09540cf9a0 legacy=0x7f09540b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.686+0000 7f0952ffd640 1 -- 172.21.15.122:0/2659325412 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095406b430 con 0x7f0954066d90 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.687+0000 7f0952ffd640 1 --2- 172.21.15.122:0/2659325412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0954066d90 0x7f09540b4ed0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0948009870 tx=0x7f094802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1045a0f8d5804552 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.687+0000 7f09527fc640 1 -- 172.21.15.122:0/2659325412 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948034070 con 0x7f0954066d90 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.687+0000 7f09527fc640 1 -- 172.21.15.122:0/2659325412 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0948030840 con 0x7f0954066d90 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.687+0000 7f09527fc640 1 -- 172.21.15.122:0/2659325412 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948030c10 con 0x7f0954066d90 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 -- 172.21.15.122:0/2659325412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0954066d90 msgr2=0x7f09540b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 --2- 172.21.15.122:0/2659325412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0954066d90 0x7f09540b4ed0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0948009870 tx=0x7f094802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 -- 172.21.15.122:0/2659325412 shutdown_connections 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 --2- 172.21.15.122:0/2659325412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f09540ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 --2- 172.21.15.122:0/2659325412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0954066d90 0x7f09540b4ed0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 -- 172.21.15.122:0/2659325412 >> 172.21.15.122:0/2659325412 conn(0x7f09540b1d70 msgr2=0x7f09540b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 -- 172.21.15.122:0/2659325412 shutdown_connections 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.688+0000 7f0959b93640 1 -- 172.21.15.122:0/2659325412 wait complete. 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.689+0000 7f0959b93640 1 Processor -- start 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.689+0000 7f0959b93640 1 -- start start 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0954066d90 0x7f095414a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f095414aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 0x7f09540cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540cfd80 con 0x7f09540cf9a0 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540b0c50 con 0x7f0954066d90 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540cff00 con 0x7f09540b5410 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 0x7f09540cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 0x7f09540cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40160/0 (socket says 172.21.15.122:40160) 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 -- 172.21.15.122:0/141055865 learned_addr learned my addr 172.21.15.122:0/141055865 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0952ffd640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f095414aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f09537fe640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0954066d90 0x7f095414a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 -- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 msgr2=0x7f095414aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f095414aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 -- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0954066d90 msgr2=0x7f095414a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0954066d90 0x7f095414a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 -- 172.21.15.122:0/141055865 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540adef0 con 0x7f09540cf9a0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0953fff640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 0x7f09540cd100 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f094c00efc0 tx=0x7f094c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c018020 con 0x7f09540cf9a0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c014480 con 0x7f09540cf9a0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09540ae1d0 con 0x7f09540cf9a0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c014870 con 0x7f09540cf9a0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.690+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09540ae790 con 0x7f09540cf9a0 2024-03-06T07:24:51.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.692+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095409e3d0 con 0x7f09540cf9a0 2024-03-06T07:24:51.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.692+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f094c005d10 con 0x7f09540cf9a0 2024-03-06T07:24:51.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.692+0000 7f093bfff640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f094006b4d0 0x7f094006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.692+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f094c08ab40 con 0x7f09540cf9a0 2024-03-06T07:24:51.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.692+0000 7f09537fe640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f094006b4d0 0x7f094006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.693+0000 7f09537fe640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f094006b4d0 0x7f094006d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f09440016e0 tx=0x7f0944009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.697+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f094c022070 con 0x7f09540cf9a0 2024-03-06T07:24:51.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.865+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0954068ff0 con 0x7f09540cf9a0 2024-03-06T07:24:51.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.865+0000 7f093bfff640 1 -- 172.21.15.122:0/141055865 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f094c010040 con 0x7f09540cf9a0 2024-03-06T07:24:51.867 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.867+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f094006b4d0 msgr2=0x7f094006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f094006b4d0 0x7f094006d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f09440016e0 tx=0x7f0944009040 comp rx=0 tx=0).stop 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 msgr2=0x7f09540cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 0x7f09540cd100 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f094c00efc0 tx=0x7f094c00c490 comp rx=0 tx=0).stop 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 shutdown_connections 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f094006b4d0 0x7f094006d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09540cf9a0 0x7f09540cd100 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09540b5410 0x7f095414aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 --2- 172.21.15.122:0/141055865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0954066d90 0x7f095414a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 >> 172.21.15.122:0/141055865 conn(0x7f09540b1d70 msgr2=0x7f09540730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 shutdown_connections 2024-03-06T07:24:51.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.868+0000 7f0959b93640 1 -- 172.21.15.122:0/141055865 wait complete. 2024-03-06T07:24:51.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:51.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.978+0000 7fb796fc9640 1 Processor -- start 2024-03-06T07:24:51.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.978+0000 7fb796fc9640 1 -- start start 2024-03-06T07:24:51.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb796fc9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb79006b220 0x7fb79015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb796fc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb79006b600 0x7fb790162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb796fc9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79015ea30 con 0x7fb7900ce920 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb796fc9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb790161b70 con 0x7fb79006b220 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb796fc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79015ed10 con 0x7fb79006b600 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb79006b220 0x7fb79015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb79006b220 0x7fb79015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40174/0 (socket says 172.21.15.122:40174) 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 -- 172.21.15.122:0/3154185369 learned_addr learned my addr 172.21.15.122:0/3154185369 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb795566640 1 --2- 172.21.15.122:0/3154185369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb79006b600 0x7fb790162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 -- 172.21.15.122:0/3154185369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb79006b600 msgr2=0x7fb790162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 --2- 172.21.15.122:0/3154185369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb79006b600 0x7fb790162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 -- 172.21.15.122:0/3154185369 >> v1:172.21.15.169:6789/0 conn(0x7fb7900ce920 legacy=0x7fb790071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb7877fe640 1 -- 172.21.15.122:0/3154185369 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173048684 0 0) 0x7fb79015ea30 con 0x7fb7900ce920 2024-03-06T07:24:51.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.979+0000 7fb794d65640 1 -- 172.21.15.122:0/3154185369 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79015eff0 con 0x7fb79006b220 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.980+0000 7fb794d65640 1 --2- 172.21.15.122:0/3154185369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79015e450 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009870 tx=0x7fb78c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4088e39b340b6499 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.980+0000 7fb7877fe640 1 -- 172.21.15.122:0/3154185369 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c035020 con 0x7fb79006b220 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.980+0000 7fb7877fe640 1 -- 172.21.15.122:0/3154185369 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c031640 con 0x7fb79006b220 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.980+0000 7fb7877fe640 1 -- 172.21.15.122:0/3154185369 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c02d040 con 0x7fb79006b220 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 -- 172.21.15.122:0/3154185369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 msgr2=0x7fb79015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 --2- 172.21.15.122:0/3154185369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79015e450 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009870 tx=0x7fb78c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 -- 172.21.15.122:0/3154185369 shutdown_connections 2024-03-06T07:24:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 --2- 172.21.15.122:0/3154185369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb79006b600 0x7fb790162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 --2- 172.21.15.122:0/3154185369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79015e450 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 -- 172.21.15.122:0/3154185369 >> 172.21.15.122:0/3154185369 conn(0x7fb79006c0c0 msgr2=0x7fb79006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 -- 172.21.15.122:0/3154185369 shutdown_connections 2024-03-06T07:24:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.981+0000 7fb796fc9640 1 -- 172.21.15.122:0/3154185369 wait complete. 2024-03-06T07:24:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.982+0000 7fb796fc9640 1 Processor -- start 2024-03-06T07:24:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.982+0000 7fb796fc9640 1 -- start start 2024-03-06T07:24:51.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79014a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79006b600 0x7fb7901605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7900ce920 0x7fb790160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790072090 con 0x7fb79006b220 2024-03-06T07:24:51.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900ce200 con 0x7fb79006b600 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900b1370 con 0x7fb7900ce920 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79014a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79014a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40184/0 (socket says 172.21.15.122:40184) 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 -- 172.21.15.122:0/1014627704 learned_addr learned my addr 172.21.15.122:0/1014627704 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb795566640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7900ce920 0x7fb790160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb787fff640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79006b600 0x7fb7901605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 -- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7900ce920 msgr2=0x7fb790160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7900ce920 0x7fb790160b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 -- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79006b600 msgr2=0x7fb7901605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79006b600 0x7fb7901605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 -- 172.21.15.122:0/1014627704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900b1650 con 0x7fb79006b220 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb794d65640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79014a570 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb78c02b930 tx=0x7fb78c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c040070 con 0x7fb79006b220 2024-03-06T07:24:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c03e070 con 0x7fb79006b220 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79014afb0 con 0x7fb79006b220 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c035040 con 0x7fb79006b220 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.983+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7900adb20 con 0x7fb79006b220 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.984+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79004f330 con 0x7fb79006b220 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.985+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb78c038020 con 0x7fb79006b220 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.985+0000 7fb7857fa640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.985+0000 7fb787fff640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.985+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb78c0ab8d0 con 0x7fb79006b220 2024-03-06T07:24:51.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.986+0000 7fb787fff640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb7901e8a90 tx=0x7fb780009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:51.990+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb78c07a190 con 0x7fb79006b220 2024-03-06T07:24:52.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.198+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7900b1820 con 0x7fb79006b220 2024-03-06T07:24:52.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.198+0000 7fb7857fa640 1 -- 172.21.15.122:0/1014627704 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fb78c07e230 con 0x7fb79006b220 2024-03-06T07:24:52.199 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:52.199 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:52.200 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:52.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 msgr2=0x7fb77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb7901e8a90 tx=0x7fb780009040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 msgr2=0x7fb79014a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79014a570 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb78c02b930 tx=0x7fb78c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 shutdown_connections 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7900ce920 0x7fb790160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79006b600 0x7fb7901605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 --2- 172.21.15.122:0/1014627704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb79006b220 0x7fb79014a570 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 >> 172.21.15.122:0/1014627704 conn(0x7fb79006c0c0 msgr2=0x7fb7900b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 shutdown_connections 2024-03-06T07:24:52.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.200+0000 7fb796fc9640 1 -- 172.21.15.122:0/1014627704 wait complete. 2024-03-06T07:24:52.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:52.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.311+0000 7fd6bbe39640 1 Processor -- start 2024-03-06T07:24:52.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.311+0000 7fd6bbe39640 1 -- start start 2024-03-06T07:24:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6bbe39640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd6b413dc10 0x7fd6b40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6bbe39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415bf30 0x7fd6b40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6bbe39640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b415f580 con 0x7fd6b415e850 2024-03-06T07:24:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6bbe39640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b40cf3f0 con 0x7fd6b413dc10 2024-03-06T07:24:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6bbe39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b415b210 con 0x7fd6b415bf30 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6ba3d6640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd6b415e850 0x7fd6b415eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43492/0 (socket says 172.21.15.122:43492) 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6ba3d6640 1 -- 172.21.15.122:0/3937060903 learned_addr learned my addr 172.21.15.122:0/3937060903 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b9bd5640 1 --2- 172.21.15.122:0/3937060903 >> v2:172.21.15.169:3300/0 conn(0x7fd6b413dc10 0x7fd6b40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b93d4640 1 --2- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415bf30 0x7fd6b40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b8bd3640 1 -- 172.21.15.122:0/3937060903 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267554753 0 0) 0x7fd6b415f580 con 0x7fd6b415e850 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b8bd3640 1 -- 172.21.15.122:0/3937060903 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a0003520 con 0x7fd6b415e850 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b9bd5640 1 -- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415bf30 msgr2=0x7fd6b40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b9bd5640 1 --2- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415bf30 0x7fd6b40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b9bd5640 1 -- 172.21.15.122:0/3937060903 >> v1:172.21.15.122:6789/0 conn(0x7fd6b415e850 legacy=0x7fd6b415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.312+0000 7fd6b9bd5640 1 -- 172.21.15.122:0/3937060903 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b415b450 con 0x7fd6b413dc10 2024-03-06T07:24:52.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.313+0000 7fd6b9bd5640 1 --2- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b40b2950 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00097a0 tx=0x7fd6b002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c74df9df036ab463 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.313+0000 7fd6b8bd3640 1 -- 172.21.15.122:0/3937060903 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00315a0 con 0x7fd6b413dc10 2024-03-06T07:24:52.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.313+0000 7fd6b8bd3640 1 -- 172.21.15.122:0/3937060903 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0037070 con 0x7fd6b413dc10 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.313+0000 7fd6b8bd3640 1 -- 172.21.15.122:0/3937060903 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0031940 con 0x7fd6b413dc10 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 -- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 msgr2=0x7fd6b40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b40b2950 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00097a0 tx=0x7fd6b002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 -- 172.21.15.122:0/3937060903 shutdown_connections 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415bf30 0x7fd6b40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/3937060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b40b2950 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 -- 172.21.15.122:0/3937060903 >> 172.21.15.122:0/3937060903 conn(0x7fd6b40cf8a0 msgr2=0x7fd6b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 -- 172.21.15.122:0/3937060903 shutdown_connections 2024-03-06T07:24:52.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.314+0000 7fd6bbe39640 1 -- 172.21.15.122:0/3937060903 wait complete. 2024-03-06T07:24:52.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.315+0000 7fd6bbe39640 1 Processor -- start 2024-03-06T07:24:52.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.315+0000 7fd6bbe39640 1 -- start start 2024-03-06T07:24:52.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b4159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 0x7fd6b4140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415e850 0x7fd6b41547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40708d0 con 0x7fd6b415bf30 2024-03-06T07:24:52.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b413dff0 con 0x7fd6b413dc10 2024-03-06T07:24:52.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4070a50 con 0x7fd6b415e850 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 0x7fd6b4140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 0x7fd6b4140170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40192/0 (socket says 172.21.15.122:40192) 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 -- 172.21.15.122:0/49603241 learned_addr learned my addr 172.21.15.122:0/49603241 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6ba3d6640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415e850 0x7fd6b41547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b9bd5640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b4159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 -- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415e850 msgr2=0x7fd6b41547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415e850 0x7fd6b41547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 -- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 msgr2=0x7fd6b4159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b4159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 -- 172.21.15.122:0/49603241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b41406b0 con 0x7fd6b415bf30 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6b93d4640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 0x7fd6b4140170 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400e9e0 tx=0x7fd6a400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4017070 con 0x7fd6b415bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b4159a90 con 0x7fd6b415bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a400c970 con 0x7fd6b415bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400cd60 con 0x7fd6b415bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.316+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b415a050 con 0x7fd6b415bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.317+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b404f330 con 0x7fd6b415bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.318+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6a401a070 con 0x7fd6b415bf30 2024-03-06T07:24:52.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.318+0000 7fd6aa7fc640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd69806b4d0 0x7fd69806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.318+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6a408a1f0 con 0x7fd6b415bf30 2024-03-06T07:24:52.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.318+0000 7fd6b9bd5640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd69806b4d0 0x7fd69806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.319+0000 7fd6b9bd5640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd69806b4d0 0x7fd69806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036000 tx=0x7fd6b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.322+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6a4058910 con 0x7fd6b415bf30 2024-03-06T07:24:52.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.497+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6b406eb00 con 0x7fd6b415bf30 2024-03-06T07:24:52.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.498+0000 7fd6aa7fc640 1 -- 172.21.15.122:0/49603241 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7fd6a405c9b0 con 0x7fd6b415bf30 2024-03-06T07:24:52.499 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:52.499 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd69806b4d0 msgr2=0x7fd69806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd69806b4d0 0x7fd69806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036000 tx=0x7fd6b0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 msgr2=0x7fd6b4140170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 0x7fd6b4140170 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400e9e0 tx=0x7fd6a400eea0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 shutdown_connections 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd69806b4d0 0x7fd69806d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6b415e850 0x7fd6b41547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6b415bf30 0x7fd6b4140170 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 --2- 172.21.15.122:0/49603241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6b413dc10 0x7fd6b4159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 >> 172.21.15.122:0/49603241 conn(0x7fd6b40cf8a0 msgr2=0x7fd6b40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 shutdown_connections 2024-03-06T07:24:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.500+0000 7fd6bbe39640 1 -- 172.21.15.122:0/49603241 wait complete. 2024-03-06T07:24:52.512 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:52.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:52.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.611+0000 7f604b71e640 1 Processor -- start 2024-03-06T07:24:52.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.612+0000 7f604b71e640 1 -- start start 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.612+0000 7f604b71e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f60440d04a0 0x7f60440ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.612+0000 7f604b71e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6044161980 con 0x7f60440cfff0 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.612+0000 7f604b71e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6044161c60 con 0x7f604415bb20 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.612+0000 7f604b71e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440d2500 con 0x7f60440d04a0 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f6048cb9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f60440cfff0 0x7f60440cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43504/0 (socket says 172.21.15.122:43504) 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f6048cb9640 1 -- 172.21.15.122:0/950604494 learned_addr learned my addr 172.21.15.122:0/950604494 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f6049cbb640 1 --2- 172.21.15.122:0/950604494 >> v2:172.21.15.169:3300/0 conn(0x7f60440d04a0 0x7f60440ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f603bfff640 1 -- 172.21.15.122:0/950604494 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260427058 0 0) 0x7f6044161980 con 0x7f60440cfff0 2024-03-06T07:24:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f603bfff640 1 -- 172.21.15.122:0/950604494 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030003520 con 0x7f60440cfff0 2024-03-06T07:24:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f6049cbb640 1 -- 172.21.15.122:0/950604494 >> v1:172.21.15.169:6789/0 conn(0x7f604415bb20 legacy=0x7f60440d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f603bfff640 1 -- 172.21.15.122:0/950604494 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871882927 0 0) 0x7f6044161c60 con 0x7f604415bb20 2024-03-06T07:24:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f6049cbb640 1 -- 172.21.15.122:0/950604494 >> v1:172.21.15.122:6789/0 conn(0x7f60440cfff0 legacy=0x7f60440cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.613+0000 7f6049cbb640 1 -- 172.21.15.122:0/950604494 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044161f40 con 0x7f60440d04a0 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f6049cbb640 1 --2- 172.21.15.122:0/950604494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440d04a0 0x7f60440ccfc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f603c009d90 tx=0x7f603c02bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101c0a9e018a3dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f603bfff640 1 -- 172.21.15.122:0/950604494 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c039070 con 0x7f60440d04a0 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f603bfff640 1 -- 172.21.15.122:0/950604494 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c02c860 con 0x7f60440d04a0 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f603bfff640 1 -- 172.21.15.122:0/950604494 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040001670 con 0x7f60440d04a0 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f604b71e640 1 -- 172.21.15.122:0/950604494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440d04a0 msgr2=0x7f60440ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f604b71e640 1 --2- 172.21.15.122:0/950604494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440d04a0 0x7f60440ccfc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f603c009d90 tx=0x7f603c02bd30 comp rx=0 tx=0).stop 2024-03-06T07:24:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f604b71e640 1 -- 172.21.15.122:0/950604494 shutdown_connections 2024-03-06T07:24:52.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f604b71e640 1 --2- 172.21.15.122:0/950604494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440d04a0 0x7f60440ccfc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.614+0000 7f604b71e640 1 -- 172.21.15.122:0/950604494 >> 172.21.15.122:0/950604494 conn(0x7f604409a9f0 msgr2=0x7f604409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.615+0000 7f604b71e640 1 -- 172.21.15.122:0/950604494 shutdown_connections 2024-03-06T07:24:52.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.615+0000 7f604b71e640 1 -- 172.21.15.122:0/950604494 wait complete. 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 Processor -- start 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 -- start start 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440cfff0 0x7f604415b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 0x7f604406fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f604415bb20 0x7f6044070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d08d0 con 0x7f60440d04a0 2024-03-06T07:24:52.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6044162500 con 0x7f60440cfff0 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f604b71e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d1980 con 0x7f604415bb20 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f6048cb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 0x7f604406fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f6048cb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 0x7f604406fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40208/0 (socket says 172.21.15.122:40208) 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f6048cb9640 1 -- 172.21.15.122:0/318824949 learned_addr learned my addr 172.21.15.122:0/318824949 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.616+0000 7f60494ba640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440cfff0 0x7f604415b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6049cbb640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f604415bb20 0x7f6044070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6048cb9640 1 -- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f604415bb20 msgr2=0x7f6044070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6048cb9640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f604415bb20 0x7f6044070400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6048cb9640 1 -- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440cfff0 msgr2=0x7f604415b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6048cb9640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440cfff0 0x7f604415b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6048cb9640 1 -- 172.21.15.122:0/318824949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604415b5a0 con 0x7f60440d04a0 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6048cb9640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 0x7f604406fec0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f604000c7f0 tx=0x7f604000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040018070 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040007500 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60440cb0c0 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040004a20 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.617+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60440cb680 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.618+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604404f330 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.618+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f604001b070 con 0x7f60440d04a0 2024-03-06T07:24:52.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.619+0000 7f6039ffb640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.619+0000 7f60494ba640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.619+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6040089720 con 0x7f60440d04a0 2024-03-06T07:24:52.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.619+0000 7f60494ba640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6034000f60 tx=0x7f6034009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.623+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f604415b5a0 con 0x7f60440d04a0 2024-03-06T07:24:52.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.833+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60440b1820 con 0x7f60440d04a0 2024-03-06T07:24:52.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.833+0000 7f6039ffb640 1 -- 172.21.15.122:0/318824949 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f6040057f50 con 0x7f60440d04a0 2024-03-06T07:24:52.835 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:52.835 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:52.835 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:52.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f602806b4d0 msgr2=0x7f602806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6034000f60 tx=0x7f6034009040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 msgr2=0x7f604406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 0x7f604406fec0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f604000c7f0 tx=0x7f604000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 shutdown_connections 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f604415bb20 0x7f6044070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60440d04a0 0x7f604406fec0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 --2- 172.21.15.122:0/318824949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60440cfff0 0x7f604415b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.835+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 >> 172.21.15.122:0/318824949 conn(0x7f604409a9f0 msgr2=0x7f604406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.836+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 shutdown_connections 2024-03-06T07:24:52.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.836+0000 7f604b71e640 1 -- 172.21.15.122:0/318824949 wait complete. 2024-03-06T07:24:52.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:52.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.945+0000 7f6db5bac640 1 Processor -- start 2024-03-06T07:24:52.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.945+0000 7f6db5bac640 1 -- start start 2024-03-06T07:24:52.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6db5bac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6db5bac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6db00cf830 0x7f6db00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6db5bac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00b3430 con 0x7f6db00d27f0 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6db5bac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db00d2c50 con 0x7f6db00cf830 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6db5bac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00b31e0 con 0x7f6db015dfe0 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6daeffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6db00cf830 0x7f6db00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6daeffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6db00cf830 0x7f6db00d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40216/0 (socket says 172.21.15.122:40216) 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6daeffd640 1 -- 172.21.15.122:0/3745304505 learned_addr learned my addr 172.21.15.122:0/3745304505 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.946+0000 7f6daf7fe640 1 --2- 172.21.15.122:0/3745304505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.947+0000 7f6daeffd640 1 -- 172.21.15.122:0/3745304505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 msgr2=0x7f6db00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.947+0000 7f6daeffd640 1 --2- 172.21.15.122:0/3745304505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.947+0000 7f6daeffd640 1 -- 172.21.15.122:0/3745304505 >> v1:172.21.15.169:6789/0 conn(0x7f6db00d27f0 legacy=0x7f6db00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.947+0000 7f6daeffd640 1 -- 172.21.15.122:0/3745304505 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00b3710 con 0x7f6db00cf830 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.947+0000 7f6daeffd640 1 --2- 172.21.15.122:0/3745304505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db00d22b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009870 tx=0x7f6da402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216a7315a1a9ea4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6dae7fc640 1 -- 172.21.15.122:0/3745304505 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4034070 con 0x7f6db00cf830 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6dae7fc640 1 -- 172.21.15.122:0/3745304505 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da4030840 con 0x7f6db00cf830 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6dae7fc640 1 -- 172.21.15.122:0/3745304505 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4030c10 con 0x7f6db00cf830 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 -- 172.21.15.122:0/3745304505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 msgr2=0x7f6db00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 --2- 172.21.15.122:0/3745304505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db00d22b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009870 tx=0x7f6da402b980 comp rx=0 tx=0).stop 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 -- 172.21.15.122:0/3745304505 shutdown_connections 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 --2- 172.21.15.122:0/3745304505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db00d22b0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 --2- 172.21.15.122:0/3745304505 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 -- 172.21.15.122:0/3745304505 >> 172.21.15.122:0/3745304505 conn(0x7f6db00d3cd0 msgr2=0x7f6db00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 -- 172.21.15.122:0/3745304505 shutdown_connections 2024-03-06T07:24:52.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.948+0000 7f6db5bac640 1 -- 172.21.15.122:0/3745304505 wait complete. 2024-03-06T07:24:52.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.949+0000 7f6db5bac640 1 Processor -- start 2024-03-06T07:24:52.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.949+0000 7f6db5bac640 1 -- start start 2024-03-06T07:24:52.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6db5bac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db0154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6db5bac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6db00d27f0 0x7f6db00cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6db5bac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6db5bac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00d1c70 con 0x7f6db00cf830 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6db5bac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00b3d60 con 0x7f6db00d27f0 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6db5bac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00d1df0 con 0x7f6db015dfe0 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6daf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db0154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6daf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db0154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40230/0 (socket says 172.21.15.122:40230) 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6daf7fe640 1 -- 172.21.15.122:0/1375591974 learned_addr learned my addr 172.21.15.122:0/1375591974 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6daffff640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.950+0000 7f6daeffd640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6db00d27f0 0x7f6db00cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6daf7fe640 1 -- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 msgr2=0x7f6db00cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6daf7fe640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6daf7fe640 1 -- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6db00d27f0 msgr2=0x7f6db00cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6daf7fe640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6db00d27f0 0x7f6db00cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6daf7fe640 1 -- 172.21.15.122:0/1375591974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00b7c50 con 0x7f6db00cf830 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6daf7fe640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db0154270 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e870 tx=0x7f6da000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da0017070 con 0x7f6db00cf830 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da000c940 con 0x7f6db00cf830 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db00b7f30 con 0x7f6db00cf830 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da000cd30 con 0x7f6db00cf830 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.951+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db00b84f0 con 0x7f6db00cf830 2024-03-06T07:24:52.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.952+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db004f330 con 0x7f6db00cf830 2024-03-06T07:24:52.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.952+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6da001a070 con 0x7f6db00cf830 2024-03-06T07:24:52.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.953+0000 7f6d97fff640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.953+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6da008a3d0 con 0x7f6db00cf830 2024-03-06T07:24:52.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.953+0000 7f6daeffd640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.953+0000 7f6daeffd640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6da4036000 tx=0x7f6da4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:52.957+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6da0058c00 con 0x7f6db00cf830 2024-03-06T07:24:53.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.167+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db015d570 con 0x7f6db00cf830 2024-03-06T07:24:53.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.168+0000 7f6d97fff640 1 -- 172.21.15.122:0/1375591974 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f6da005cca0 con 0x7f6db00cf830 2024-03-06T07:24:53.170 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:53.170 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:18.367265+0000","modified":"2024-03-06T07:24:20.368965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6156":{"gid":6156,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6834/2303645671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2303645671},{"type":"v1","addr":"172.21.15.169:6834","nonce":2303645671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6156,"qdb_cluster":[6156]},"id":4}]} 2024-03-06T07:24:53.170 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 27 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d9c06b4d0 msgr2=0x7f6d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6da4036000 tx=0x7f6da4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 msgr2=0x7f6db0154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db0154270 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e870 tx=0x7f6da000ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 shutdown_connections 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6db015dfe0 0x7f6db00cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6db00d27f0 0x7f6db00cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 --2- 172.21.15.122:0/1375591974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6db00cf830 0x7f6db0154270 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 >> 172.21.15.122:0/1375591974 conn(0x7f6db00d3cd0 msgr2=0x7f6db006cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 shutdown_connections 2024-03-06T07:24:53.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.170+0000 7f6db5bac640 1 -- 172.21.15.122:0/1375591974 wait complete. 2024-03-06T07:24:53.183 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.280+0000 7f2d827e9640 1 Processor -- start 2024-03-06T07:24:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.280+0000 7f2d827e9640 1 -- start start 2024-03-06T07:24:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d827e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d7c13dc10 0x7f2d7c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d827e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d827e9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c15f580 con 0x7f2d7c15bf30 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d827e9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0cf3f0 con 0x7f2d7c13dc10 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d827e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c15b210 con 0x7f2d7c15ef60 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d7c13dc10 0x7f2d7c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d7c13dc10 0x7f2d7c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40238/0 (socket says 172.21.15.122:40238) 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2d7c15bf30 0x7f2d7c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43510/0 (socket says 172.21.15.122:43510) 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 -- 172.21.15.122:0/3087655092 learned_addr learned my addr 172.21.15.122:0/3087655092 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d80d86640 1 --2- 172.21.15.122:0/3087655092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 -- 172.21.15.122:0/3087655092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 msgr2=0x7f2d7c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 --2- 172.21.15.122:0/3087655092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 -- 172.21.15.122:0/3087655092 >> v1:172.21.15.122:6789/0 conn(0x7f2d7c15bf30 legacy=0x7f2d7c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7bfff640 1 -- 172.21.15.122:0/3087655092 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c15b450 con 0x7f2d7c13dc10 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.281+0000 7f2d7affd640 1 -- 172.21.15.122:0/3087655092 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633699210 0 0) 0x7f2d7c15f580 con 0x7f2d7c15bf30 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d7bfff640 1 --2- 172.21.15.122:0/3087655092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c0b2950 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00bb10 tx=0x7f2d6c02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29446e5cf2422b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d7affd640 1 -- 172.21.15.122:0/3087655092 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c03b070 con 0x7f2d7c13dc10 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d7affd640 1 -- 172.21.15.122:0/3087655092 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c036440 con 0x7f2d7c13dc10 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d7affd640 1 -- 172.21.15.122:0/3087655092 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c036810 con 0x7f2d7c13dc10 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d827e9640 1 -- 172.21.15.122:0/3087655092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c13dc10 msgr2=0x7f2d7c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d827e9640 1 --2- 172.21.15.122:0/3087655092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c0b2950 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00bb10 tx=0x7f2d6c02e0c0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d827e9640 1 -- 172.21.15.122:0/3087655092 shutdown_connections 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d827e9640 1 --2- 172.21.15.122:0/3087655092 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d827e9640 1 --2- 172.21.15.122:0/3087655092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c0b2950 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.282+0000 7f2d827e9640 1 -- 172.21.15.122:0/3087655092 >> 172.21.15.122:0/3087655092 conn(0x7f2d7c0cf8a0 msgr2=0x7f2d7c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.283+0000 7f2d827e9640 1 -- 172.21.15.122:0/3087655092 shutdown_connections 2024-03-06T07:24:53.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.283+0000 7f2d827e9640 1 -- 172.21.15.122:0/3087655092 wait complete. 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.284+0000 7f2d827e9640 1 Processor -- start 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.284+0000 7f2d827e9640 1 -- start start 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 0x7f2d7c070d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0cd7a0 con 0x7f2d7c15bf30 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c13dff0 con 0x7f2d7c13dc10 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0cd920 con 0x7f2d7c15ef60 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 0x7f2d7c070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 0x7f2d7c070d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40252/0 (socket says 172.21.15.122:40252) 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 -- 172.21.15.122:0/2853055963 learned_addr learned my addr 172.21.15.122:0/2853055963 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7bfff640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d80d86640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 -- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 msgr2=0x7f2d7c159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 -- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c13dc10 msgr2=0x7f2d7c1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 -- 172.21.15.122:0/2853055963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c139210 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d7b7fe640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 0x7f2d7c070d00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000c4d0 tx=0x7f2d7000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70018070 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d70007500 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70004a20 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c071670 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.285+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c071c30 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.286+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c04f330 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.287+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d70004bc0 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.287+0000 7f2d78ff9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d6806b4d0 0x7f2d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.287+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d70089330 con 0x7f2d7c15bf30 2024-03-06T07:24:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.287+0000 7f2d7bfff640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d6806b4d0 0x7f2d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.287+0000 7f2d7bfff640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d6806b4d0 0x7f2d6806d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c038000 tx=0x7f2d6c032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.291+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d70057bf0 con 0x7f2d7c15bf30 2024-03-06T07:24:53.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.466+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d7c071fd0 con 0x7f2d7c15bf30 2024-03-06T07:24:53.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.467+0000 7f2d78ff9640 1 -- 172.21.15.122:0/2853055963 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16665 (secure 0 0 0) 0x7f2d7005bc90 con 0x7f2d7c15bf30 2024-03-06T07:24:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:53.470 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":57,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:20.025077+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:16.591753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:17.332050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:53.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d6806b4d0 msgr2=0x7f2d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d6806b4d0 0x7f2d6806d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c038000 tx=0x7f2d6c032000 comp rx=0 tx=0).stop 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 msgr2=0x7f2d7c070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 0x7f2d7c070d00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000c4d0 tx=0x7f2d7000c990 comp rx=0 tx=0).stop 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 shutdown_connections 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d6806b4d0 0x7f2d6806d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d7c15ef60 0x7f2d7c159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c15bf30 0x7f2d7c070d00 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 --2- 172.21.15.122:0/2853055963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c13dc10 0x7f2d7c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 >> 172.21.15.122:0/2853055963 conn(0x7f2d7c0cf8a0 msgr2=0x7f2d7c15ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 shutdown_connections 2024-03-06T07:24:53.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.469+0000 7f2d827e9640 1 -- 172.21.15.122:0/2853055963 wait complete. 2024-03-06T07:24:53.483 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:53.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:53.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:53.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.582+0000 7fc7e064e640 1 Processor -- start 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.582+0000 7fc7e064e640 1 -- start start 2024-03-06T07:24:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7e064e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d80b4c00 0x7fc7d80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7e064e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc7d80b28a0 0x7fc7d80b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7e064e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d815d510 con 0x7fc7d815e540 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7e064e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d815e0b0 con 0x7fc7d80b28a0 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7e064e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d815d780 con 0x7fc7d80b4c00 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7debeb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc7d815e540 0x7fc7d815e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43516/0 (socket says 172.21.15.122:43516) 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.583+0000 7fc7debeb640 1 -- 172.21.15.122:0/2067115968 learned_addr learned my addr 172.21.15.122:0/2067115968 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7ddbe9640 1 --2- 172.21.15.122:0/2067115968 >> v2:172.21.15.169:3300/0 conn(0x7fc7d80b28a0 0x7fc7d80b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7de3ea640 1 --2- 172.21.15.122:0/2067115968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d80b4c00 0x7fc7d80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7dd3e8640 1 -- 172.21.15.122:0/2067115968 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588235257 0 0) 0x7fc7d815d510 con 0x7fc7d815e540 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7dd3e8640 1 -- 172.21.15.122:0/2067115968 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c4003520 con 0x7fc7d815e540 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7de3ea640 1 -- 172.21.15.122:0/2067115968 >> v2:172.21.15.169:3300/0 conn(0x7fc7d80b28a0 msgr2=0x7fc7d80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7de3ea640 1 --2- 172.21.15.122:0/2067115968 >> v2:172.21.15.169:3300/0 conn(0x7fc7d80b28a0 0x7fc7d80b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7de3ea640 1 -- 172.21.15.122:0/2067115968 >> v1:172.21.15.122:6789/0 conn(0x7fc7d815e540 legacy=0x7fc7d815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7de3ea640 1 -- 172.21.15.122:0/2067115968 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d815da60 con 0x7fc7d80b4c00 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.584+0000 7fc7ddbe9640 1 --2- 172.21.15.122:0/2067115968 >> v2:172.21.15.169:3300/0 conn(0x7fc7d80b28a0 0x7fc7d80b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7de3ea640 1 --2- 172.21.15.122:0/2067115968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d80b4c00 0x7fc7d80b4fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097a0 tx=0x7fc7d402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6084c87458f6f8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7dd3e8640 1 -- 172.21.15.122:0/2067115968 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4034070 con 0x7fc7d80b4c00 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7dd3e8640 1 -- 172.21.15.122:0/2067115968 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d4030610 con 0x7fc7d80b4c00 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7dd3e8640 1 -- 172.21.15.122:0/2067115968 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40309c0 con 0x7fc7d80b4c00 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7e064e640 1 -- 172.21.15.122:0/2067115968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d80b4c00 msgr2=0x7fc7d80b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7e064e640 1 --2- 172.21.15.122:0/2067115968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d80b4c00 0x7fc7d80b4fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097a0 tx=0x7fc7d402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.585+0000 7fc7e064e640 1 -- 172.21.15.122:0/2067115968 shutdown_connections 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.586+0000 7fc7e064e640 1 --2- 172.21.15.122:0/2067115968 >> v2:172.21.15.169:3300/0 conn(0x7fc7d80b28a0 0x7fc7d80b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.586+0000 7fc7e064e640 1 --2- 172.21.15.122:0/2067115968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d80b4c00 0x7fc7d80b4fe0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.586+0000 7fc7e064e640 1 -- 172.21.15.122:0/2067115968 >> 172.21.15.122:0/2067115968 conn(0x7fc7d809a9f0 msgr2=0x7fc7d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.586+0000 7fc7e064e640 1 -- 172.21.15.122:0/2067115968 shutdown_connections 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.586+0000 7fc7e064e640 1 -- 172.21.15.122:0/2067115968 wait complete. 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 Processor -- start 2024-03-06T07:24:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 -- start start 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d80b28a0 0x7fc7d8154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 0x7fc7d813f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d815e540 0x7fc7d81510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80b33a0 con 0x7fc7d80b4c00 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80b3220 con 0x7fc7d80b28a0 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.587+0000 7fc7e064e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d815f090 con 0x7fc7d815e540 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 0x7fc7d813f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 0x7fc7d813f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40258/0 (socket says 172.21.15.122:40258) 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 -- 172.21.15.122:0/3229659392 learned_addr learned my addr 172.21.15.122:0/3229659392 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7debeb640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d815e540 0x7fc7d81510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7de3ea640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d80b28a0 0x7fc7d8154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 -- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d815e540 msgr2=0x7fc7d81510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d815e540 0x7fc7d81510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 -- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d80b28a0 msgr2=0x7fc7d8154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d80b28a0 0x7fc7d8154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 -- 172.21.15.122:0/3229659392 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d81f3070 con 0x7fc7d80b4c00 2024-03-06T07:24:53.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ddbe9640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 0x7fc7d813f540 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800ebe0 tx=0x7fc7c800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8017070 con 0x7fc7d80b4c00 2024-03-06T07:24:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c800cc30 con 0x7fc7d80b4c00 2024-03-06T07:24:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d81f22b0 con 0x7fc7d80b4c00 2024-03-06T07:24:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8013470 con 0x7fc7d80b4c00 2024-03-06T07:24:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.588+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d81f2870 con 0x7fc7d80b4c00 2024-03-06T07:24:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.589+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d804f330 con 0x7fc7d80b4c00 2024-03-06T07:24:53.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.590+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7c801a070 con 0x7fc7d80b4c00 2024-03-06T07:24:53.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.590+0000 7fc7ceffd640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7bc06b4d0 0x7fc7bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.590+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7c808a030 con 0x7fc7d80b4c00 2024-03-06T07:24:53.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.590+0000 7fc7de3ea640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7bc06b4d0 0x7fc7bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.590+0000 7fc7de3ea640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7bc06b4d0 0x7fc7bc06d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4036000 tx=0x7fc7d4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.594+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7c80588f0 con 0x7fc7d80b4c00 2024-03-06T07:24:53.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.805+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc7d815d210 con 0x7fc7d80b4c00 2024-03-06T07:24:53.808 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:24:53.806+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:53.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.807+0000 7fc7ceffd640 1 -- 172.21.15.122:0/3229659392 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc7c805c990 con 0x7fc7d80b4c00 2024-03-06T07:24:53.809 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7bc06b4d0 msgr2=0x7fc7bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7bc06b4d0 0x7fc7bc06d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4036000 tx=0x7fc7d4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 msgr2=0x7fc7d813f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 0x7fc7d813f540 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800ebe0 tx=0x7fc7c800c520 comp rx=0 tx=0).stop 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 shutdown_connections 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7bc06b4d0 0x7fc7bc06d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7d815e540 0x7fc7d81510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7d80b4c00 0x7fc7d813f540 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 --2- 172.21.15.122:0/3229659392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d80b28a0 0x7fc7d8154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 >> 172.21.15.122:0/3229659392 conn(0x7fc7d809a9f0 msgr2=0x7fc7d815b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 shutdown_connections 2024-03-06T07:24:53.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.810+0000 7fc7e064e640 1 -- 172.21.15.122:0/3229659392 wait complete. 2024-03-06T07:24:53.823 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:53.857 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:24:53.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.963+0000 7f5ad6eec640 1 Processor -- start 2024-03-06T07:24:53.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.963+0000 7f5ad6eec640 1 -- start start 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.963+0000 7f5ad6eec640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad015f4f0 0x7f5ad0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad6eec640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad006b050 con 0x7f5ad00ce750 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad6eec640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad006b2c0 con 0x7f5ad00b0db0 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad6eec640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00b0890 con 0x7f5ad015f4f0 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad5489640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5ad00ce750 0x7f5ad00b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43526/0 (socket says 172.21.15.122:43526) 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad5489640 1 -- 172.21.15.122:0/1830838632 learned_addr learned my addr 172.21.15.122:0/1830838632 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:53.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad4c88640 1 --2- 172.21.15.122:0/1830838632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad015f4f0 0x7f5ad0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5acf7fe640 1 -- 172.21.15.122:0/1830838632 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390147666 0 0) 0x7f5ad006b050 con 0x7f5ad00ce750 2024-03-06T07:24:53.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5acf7fe640 1 -- 172.21.15.122:0/1830838632 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5abc003520 con 0x7f5ad00ce750 2024-03-06T07:24:53.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5acf7fe640 1 -- 172.21.15.122:0/1830838632 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804223077 0 0) 0x7f5ad006b2c0 con 0x7f5ad00b0db0 2024-03-06T07:24:53.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad4c88640 1 -- 172.21.15.122:0/1830838632 >> v1:172.21.15.169:6789/0 conn(0x7f5ad00b0db0 legacy=0x7f5ad00ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad4c88640 1 -- 172.21.15.122:0/1830838632 >> v1:172.21.15.122:6789/0 conn(0x7f5ad00ce750 legacy=0x7f5ad00b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.964+0000 7f5ad4c88640 1 -- 172.21.15.122:0/1830838632 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad00672e0 con 0x7f5ad015f4f0 2024-03-06T07:24:53.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.965+0000 7f5ad4c88640 1 --2- 172.21.15.122:0/1830838632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad015f4f0 0x7f5ad0066d00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009dc0 tx=0x7f5ac802bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc6f17229112bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.965+0000 7f5acf7fe640 1 -- 172.21.15.122:0/1830838632 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8039070 con 0x7f5ad015f4f0 2024-03-06T07:24:53.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.965+0000 7f5acf7fe640 1 -- 172.21.15.122:0/1830838632 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac802c840 con 0x7f5ad015f4f0 2024-03-06T07:24:53.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.965+0000 7f5acf7fe640 1 -- 172.21.15.122:0/1830838632 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80015f0 con 0x7f5ad015f4f0 2024-03-06T07:24:53.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1830838632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad015f4f0 msgr2=0x7f5ad0066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1830838632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad015f4f0 0x7f5ad0066d00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009dc0 tx=0x7f5ac802bd10 comp rx=0 tx=0).stop 2024-03-06T07:24:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1830838632 shutdown_connections 2024-03-06T07:24:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1830838632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad015f4f0 0x7f5ad0066d00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1830838632 >> 172.21.15.122:0/1830838632 conn(0x7f5ad00b1d70 msgr2=0x7f5ad00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1830838632 shutdown_connections 2024-03-06T07:24:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.966+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1830838632 wait complete. 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.967+0000 7f5ad6eec640 1 Processor -- start 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.967+0000 7f5ad6eec640 1 -- start start 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad00b0db0 0x7f5ad014d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad00ce750 0x7f5ad01f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 0x7f5ad00c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015f8d0 con 0x7f5ad015f4f0 2024-03-06T07:24:53.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00cebe0 con 0x7f5ad00b0db0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad0156a10 con 0x7f5ad00ce750 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 0x7f5ad00c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 0x7f5ad00c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40272/0 (socket says 172.21.15.122:40272) 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 -- 172.21.15.122:0/1415994449 learned_addr learned my addr 172.21.15.122:0/1415994449 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 -- 172.21.15.122:0/1415994449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad00ce750 msgr2=0x7f5ad01f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad00ce750 0x7f5ad01f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 -- 172.21.15.122:0/1415994449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad00b0db0 msgr2=0x7f5ad014d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad00b0db0 0x7f5ad014d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 -- 172.21.15.122:0/1415994449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad00cbeb0 con 0x7f5ad015f4f0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad5489640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 0x7f5ad00c9a90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c750 tx=0x7f5ab800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8018070 con 0x7f5ad015f4f0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab80074e0 con 0x7f5ad015f4f0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad00cc190 con 0x7f5ad015f4f0 2024-03-06T07:24:53.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800ce60 con 0x7f5ad015f4f0 2024-03-06T07:24:53.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.968+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad00cc750 con 0x7f5ad015f4f0 2024-03-06T07:24:53.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.969+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad009e3d0 con 0x7f5ad015f4f0 2024-03-06T07:24:53.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.970+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ab801b070 con 0x7f5ad015f4f0 2024-03-06T07:24:53.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.970+0000 7f5acd7fa640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.971+0000 7f5ad4c88640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.971+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5ab8089790 con 0x7f5ad015f4f0 2024-03-06T07:24:53.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.971+0000 7f5ad4c88640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ab006b4d0 0x7f5ab006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8036000 tx=0x7f5ac802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:53.975+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ab80585a0 con 0x7f5ad015f4f0 2024-03-06T07:24:54.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.182+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5ad006cd40 con 0x7f5ad015f4f0 2024-03-06T07:24:54.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.808+0000 7f5acd7fa640 1 -- 172.21.15.122:0/1415994449 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5ab8057ea0 con 0x7f5ad015f4f0 2024-03-06T07:24:54.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ab006b4d0 msgr2=0x7f5ab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ab006b4d0 0x7f5ab006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8036000 tx=0x7f5ac802e000 comp rx=0 tx=0).stop 2024-03-06T07:24:54.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 msgr2=0x7f5ad00c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 0x7f5ad00c9a90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c750 tx=0x7f5ab800cc10 comp rx=0 tx=0).stop 2024-03-06T07:24:54.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 shutdown_connections 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ad015f4f0 0x7f5ad00c9a90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5ad00ce750 0x7f5ad01f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 --2- 172.21.15.122:0/1415994449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ad00b0db0 0x7f5ad014d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 >> 172.21.15.122:0/1415994449 conn(0x7f5ad00b1d70 msgr2=0x7f5ad00ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 shutdown_connections 2024-03-06T07:24:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.811+0000 7f5ad6eec640 1 -- 172.21.15.122:0/1415994449 wait complete. 2024-03-06T07:24:54.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.929+0000 7fdf09d0c640 1 Processor -- start 2024-03-06T07:24:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.929+0000 7fdf09d0c640 1 -- start start 2024-03-06T07:24:54.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf09d0c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 0x7fdf0406df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf09d0c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406e4d0 con 0x7fdf040b3060 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf09d0c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040b4490 con 0x7fdf040b3fd0 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf09d0c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040cf4a0 con 0x7fdf0415a490 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf03fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdf040b3060 0x7fdf0415dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33188/0 (socket says 172.21.15.122:33188) 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf03fff640 1 -- 172.21.15.122:0/3926971995 learned_addr learned my addr 172.21.15.122:0/3926971995 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686071938 0 0) 0x7fdefc001490 con 0x7fdf040b3060 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef0003440 con 0x7fdf040b3060 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.930+0000 7fdf037fe640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 0x7fdf0406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915115336 0 0) 0x7fdef0003440 con 0x7fdf040b3060 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 msgr2=0x7fdf0406df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 0x7fdf0406df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 >> v1:172.21.15.169:6789/0 conn(0x7fdf040b3fd0 legacy=0x7fdf040b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040b4750 con 0x7fdf040b3060 2024-03-06T07:24:54.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fdefc001bc0 con 0x7fdf040b3060 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 >> v1:172.21.15.122:6789/0 conn(0x7fdf040b3060 legacy=0x7fdf0415dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef0003ad0 0x7fdef0003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdef000aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 0x7fdef000d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040cf4a0 con 0x7fdef0008850 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000dc50 con 0x7fdef0003ad0 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000de50 con 0x7fdf0415a490 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdef000aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf037fe640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef0003ad0 0x7fdef0003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdf03fff640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 0x7fdef000d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdefbfff640 1 -- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 msgr2=0x7fdef000d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0415a490 0x7fdef000d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdefbfff640 1 -- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef0003ad0 msgr2=0x7fdef0003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdef0003ad0 0x7fdef0003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.931+0000 7fdefbfff640 1 -- 172.21.15.122:0/3926971995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef0003440 con 0x7fdef0008850 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdef000aca0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdeec001750 tx=0x7fdef0003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa93840b4b388856 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec001370 con 0x7fdef0008850 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec0052d0 con 0x7fdef0008850 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf02ffd640 1 -- 172.21.15.122:0/3926971995 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec003830 con 0x7fdef0008850 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3926971995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 msgr2=0x7fdef000aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3926971995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdef000aca0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdeec001750 tx=0x7fdef0003730 comp rx=0 tx=0).stop 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf03fff640 1 -- 172.21.15.122:0/3926971995 reap_dead start 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3926971995 shutdown_connections 2024-03-06T07:24:54.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.932+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3926971995 >> 172.21.15.122:0/3926971995 conn(0x7fdf040b1650 msgr2=0x7fdf040b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.933+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3926971995 shutdown_connections 2024-03-06T07:24:54.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.933+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3926971995 wait complete. 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 Processor -- start 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 -- start start 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdf041f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0414b810 0x7fdf0415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041e35a0 0x7fdf041e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040cf0e0 con 0x7fdef0008850 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b34f0 con 0x7fdf041e35a0 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf09d0c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040cf260 con 0x7fdf0414b810 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf037fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdf041f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf037fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdf041f0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56784/0 (socket says 172.21.15.122:56784) 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.934+0000 7fdf037fe640 1 -- 172.21.15.122:0/3147316711 learned_addr learned my addr 172.21.15.122:0/3147316711 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:54.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf03fff640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041e35a0 0x7fdf041e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf037fe640 1 -- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0414b810 msgr2=0x7fdf0415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0414b810 0x7fdf0415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf037fe640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0414b810 0x7fdf0415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf037fe640 1 -- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041e35a0 msgr2=0x7fdf041e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf037fe640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041e35a0 0x7fdf041e3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf037fe640 1 -- 172.21.15.122:0/3147316711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04153060 con 0x7fdef0008850 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf03fff640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041e35a0 0x7fdf041e3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0414b810 0x7fdf0415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf037fe640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdf041f0f60 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdef40119a0 tx=0x7fdef4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef400fa90 con 0x7fdef0008850 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef400fc30 con 0x7fdef0008850 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0415cbe0 con 0x7fdef0008850 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4014440 con 0x7fdef0008850 2024-03-06T07:24:54.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.935+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0415d1a0 con 0x7fdef0008850 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.936+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf040687d0 con 0x7fdef0008850 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.937+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdef4012040 con 0x7fdef0008850 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.937+0000 7fdf00ff9640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdee406b4d0 0x7fdee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.937+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdef408a880 con 0x7fdef0008850 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.937+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdee406b4d0 0x7fdee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.937+0000 7fdefbfff640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdee406b4d0 0x7fdee406d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdeec033000 tx=0x7fdeec009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:54.941+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf04153060 con 0x7fdef0008850 2024-03-06T07:24:55.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.112+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdf0404f330 con 0x7fdef0008850 2024-03-06T07:24:55.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.813+0000 7fdf00ff9640 1 -- 172.21.15.122:0/3147316711 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdef4019030 con 0x7fdef0008850 2024-03-06T07:24:55.815 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:55.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdee406b4d0 msgr2=0x7fdee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdee406b4d0 0x7fdee406d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdeec033000 tx=0x7fdeec009030 comp rx=0 tx=0).stop 2024-03-06T07:24:55.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 msgr2=0x7fdf041f0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdf041f0f60 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdef40119a0 tx=0x7fdef4011e60 comp rx=0 tx=0).stop 2024-03-06T07:24:55.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 shutdown_connections 2024-03-06T07:24:55.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdee406b4d0 0x7fdee406d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041e35a0 0x7fdf041e3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf0414b810 0x7fdf0415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 --2- 172.21.15.122:0/3147316711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdef0008850 0x7fdf041f0f60 secure :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdef40119a0 tx=0x7fdef4011e60 comp rx=0 tx=0).stop 2024-03-06T07:24:55.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.816+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 >> 172.21.15.122:0/3147316711 conn(0x7fdf040b1650 msgr2=0x7fdf0414be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.817+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 shutdown_connections 2024-03-06T07:24:55.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.817+0000 7fdf09d0c640 1 -- 172.21.15.122:0/3147316711 wait complete. 2024-03-06T07:24:55.832 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.942+0000 7f16bdd93640 1 Processor -- start 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.942+0000 7f16bdd93640 1 -- start start 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16bdd93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16b815ac00 0x7f16b806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16bdd93640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b815f5c0 con 0x7f16b8070d10 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16bdd93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80711c0 con 0x7f16b80708b0 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16bdd93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b80cdc10 con 0x7f16b815ac00 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16b77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16b815ac00 0x7f16b806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16b7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f16b8070d10 0x7f16b815eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33190/0 (socket says 172.21.15.122:33190) 2024-03-06T07:24:55.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.943+0000 7f16b7fff640 1 -- 172.21.15.122:0/1001949007 learned_addr learned my addr 172.21.15.122:0/1001949007 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:55.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.944+0000 7f16b77fe640 1 -- 172.21.15.122:0/1001949007 >> v1:172.21.15.169:6789/0 conn(0x7f16b80708b0 legacy=0x7f16b815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.944+0000 7f16b77fe640 1 -- 172.21.15.122:0/1001949007 >> v1:172.21.15.122:6789/0 conn(0x7f16b8070d10 legacy=0x7f16b815eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.944+0000 7f16b77fe640 1 -- 172.21.15.122:0/1001949007 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b815afe0 con 0x7f16b815ac00 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.944+0000 7f16b77fe640 1 --2- 172.21.15.122:0/1001949007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b806c910 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f16a80097a0 tx=0x7f16a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7775d4c353f14dcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16b67fc640 1 -- 172.21.15.122:0/1001949007 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a80315a0 con 0x7f16b815ac00 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16b67fc640 1 -- 172.21.15.122:0/1001949007 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a8037070 con 0x7f16b815ac00 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16b67fc640 1 -- 172.21.15.122:0/1001949007 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8031960 con 0x7f16b815ac00 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 -- 172.21.15.122:0/1001949007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 msgr2=0x7f16b806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1001949007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b806c910 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f16a80097a0 tx=0x7f16a802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 -- 172.21.15.122:0/1001949007 shutdown_connections 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1001949007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b806c910 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 -- 172.21.15.122:0/1001949007 >> 172.21.15.122:0/1001949007 conn(0x7f16b80b1650 msgr2=0x7f16b80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 -- 172.21.15.122:0/1001949007 shutdown_connections 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.945+0000 7f16bdd93640 1 -- 172.21.15.122:0/1001949007 wait complete. 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.946+0000 7f16bdd93640 1 Processor -- start 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.946+0000 7f16bdd93640 1 -- start start 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16bdd93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16b80708b0 0x7f16b81f1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16bdd93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16b8070d10 0x7f16b8141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16bdd93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b81586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16bdd93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80cd5c0 con 0x7f16b815ac00 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16bdd93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80cd2e0 con 0x7f16b8070d10 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16bdd93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80cd8a0 con 0x7f16b80708b0 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b81586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b81586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56808/0 (socket says 172.21.15.122:56808) 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16b7fff640 1 -- 172.21.15.122:0/1602061372 learned_addr learned my addr 172.21.15.122:0/1602061372 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16b6ffd640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16b8070d10 0x7f16b8141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16b77fe640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16b80708b0 0x7f16b81f1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.947+0000 7f16b7fff640 1 -- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16b80708b0 msgr2=0x7f16b81f1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b7fff640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16b80708b0 0x7f16b81f1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b7fff640 1 -- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16b8070d10 msgr2=0x7f16b8141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b7fff640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16b8070d10 0x7f16b8141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b7fff640 1 -- 172.21.15.122:0/1602061372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b81f17a0 con 0x7f16b815ac00 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b77fe640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16b80708b0 0x7f16b81f1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b6ffd640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16b8070d10 0x7f16b8141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16b7fff640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b81586f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f16b000ee60 tx=0x7f16b000c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0015070 con 0x7f16b815ac00 2024-03-06T07:24:55.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b0011880 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b8156ae0 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0011c70 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.948+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8157380 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.949+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80687d0 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.949+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16b0018070 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.950+0000 7f169ffff640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f16a406b500 0x7f16a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.950+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f16b008cd60 con 0x7f16b815ac00 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.950+0000 7f16b77fe640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f16a406b500 0x7f16a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.950+0000 7f16b77fe640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f16a406b500 0x7f16a406d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16a8036000 tx=0x7f16a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:55.954+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16b005bd40 con 0x7f16b815ac00 2024-03-06T07:24:56.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.125+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16b8155790 con 0x7f16b815ac00 2024-03-06T07:24:56.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.818+0000 7f169ffff640 1 -- 172.21.15.122:0/1602061372 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) v1 ==== 169+0+0 (secure 0 0 0) 0x7f16b005b640 con 0x7f16b815ac00 2024-03-06T07:24:56.820 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.821+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f16a406b500 msgr2=0x7f16a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.821+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f16a406b500 0x7f16a406d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16a8036000 tx=0x7f16a8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.821+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 msgr2=0x7f16b81586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.821+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b81586f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f16b000ee60 tx=0x7f16b000c640 comp rx=0 tx=0).stop 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 shutdown_connections 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f16a406b500 0x7f16a406d9c0 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16a8036000 tx=0x7f16a8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16b815ac00 0x7f16b81586f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16b8070d10 0x7f16b8141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 --2- 172.21.15.122:0/1602061372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16b80708b0 0x7f16b81f1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 >> 172.21.15.122:0/1602061372 conn(0x7f16b80b1650 msgr2=0x7f16b80b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 shutdown_connections 2024-03-06T07:24:56.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.822+0000 7f16bdd93640 1 -- 172.21.15.122:0/1602061372 wait complete. 2024-03-06T07:24:56.838 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:24:56.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.940+0000 7fbe2d415640 1 Processor -- start 2024-03-06T07:24:56.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.940+0000 7fbe2d415640 1 -- start start 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe2d415640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 0x7fbe280b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe2d415640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe280d47f0 con 0x7fbe280d06e0 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe2d415640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe28160070 con 0x7fbe280b4d10 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe2d415640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280ced70 con 0x7fbe2815fc30 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe26ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbe280d06e0 0x7fbe280d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33206/0 (socket says 172.21.15.122:33206) 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe26ffd640 1 -- 172.21.15.122:0/2941944821 learned_addr learned my addr 172.21.15.122:0/2941944821 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.941+0000 7fbe267fc640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 0x7fbe280b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287584182 0 0) 0x7fbe280d47f0 con 0x7fbe280d06e0 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe10003520 con 0x7fbe280d06e0 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2036547641 0 0) 0x7fbe10003520 con 0x7fbe280d06e0 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 msgr2=0x7fbe280b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 0x7fbe280b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 >> v1:172.21.15.169:6789/0 conn(0x7fbe280b4d10 legacy=0x7fbe280cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe280d1ab0 con 0x7fbe280d06e0 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fbe180019f0 con 0x7fbe280d06e0 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 >> v1:172.21.15.122:6789/0 conn(0x7fbe280d06e0 legacy=0x7fbe280d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe100039e0 0x7fbe10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe10004390 0x7fbe10004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 0x7fbe100050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280ced70 con 0x7fbe100039e0 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1000b9c0 con 0x7fbe10004390 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1000bbc0 con 0x7fbe2815fc30 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe100039e0 0x7fbe10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe267fc640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe10004390 0x7fbe10004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe277fe640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 0x7fbe100050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe26ffd640 1 -- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 msgr2=0x7fbe100050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe2815fc30 0x7fbe100050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe26ffd640 1 -- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe10004390 msgr2=0x7fbe10004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe10004390 0x7fbe10004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.942+0000 7fbe26ffd640 1 -- 172.21.15.122:0/2941944821 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10003520 con 0x7fbe100039e0 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.943+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe100039e0 0x7fbe10003dc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800a920 tx=0x7fbe10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b15563c1f4dfb8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.943+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1800a4c0 con 0x7fbe100039e0 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.943+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe18034650 con 0x7fbe100039e0 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.943+0000 7fbe25ffb640 1 -- 172.21.15.122:0/2941944821 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18034a40 con 0x7fbe100039e0 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe2d415640 1 -- 172.21.15.122:0/2941944821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe100039e0 msgr2=0x7fbe10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2941944821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe100039e0 0x7fbe10003dc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800a920 tx=0x7fbe10005830 comp rx=0 tx=0).stop 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe277fe640 1 -- 172.21.15.122:0/2941944821 reap_dead start 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe2d415640 1 -- 172.21.15.122:0/2941944821 shutdown_connections 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe2d415640 1 -- 172.21.15.122:0/2941944821 >> 172.21.15.122:0/2941944821 conn(0x7fbe280d3cd0 msgr2=0x7fbe280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe2d415640 1 -- 172.21.15.122:0/2941944821 shutdown_connections 2024-03-06T07:24:56.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.944+0000 7fbe2d415640 1 -- 172.21.15.122:0/2941944821 wait complete. 2024-03-06T07:24:56.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.945+0000 7fbe2d415640 1 Processor -- start 2024-03-06T07:24:56.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.945+0000 7fbe2d415640 1 -- start start 2024-03-06T07:24:56.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe100039e0 0x7fbe28153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe280cbdf0 0x7fbe280cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 0x7fbe280b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280ce990 con 0x7fbe280cc7b0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280d4ab0 con 0x7fbe100039e0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280b44d0 con 0x7fbe280cbdf0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 0x7fbe280b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 0x7fbe280b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56826/0 (socket says 172.21.15.122:56826) 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 -- 172.21.15.122:0/2329801549 learned_addr learned my addr 172.21.15.122:0/2329801549 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe267fc640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe280cbdf0 0x7fbe280cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe100039e0 0x7fbe28153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 -- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe280cbdf0 msgr2=0x7fbe280cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe280cbdf0 0x7fbe280cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 -- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe100039e0 msgr2=0x7fbe28153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe100039e0 0x7fbe28153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 -- 172.21.15.122:0/2329801549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe280ccc30 con 0x7fbe280cc7b0 2024-03-06T07:24:56.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe267fc640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe280cbdf0 0x7fbe280cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe100039e0 0x7fbe28153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe277fe640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 0x7fbe280b1a00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000e370 tx=0x7fbe2000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20019070 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe200052e0 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe280b09a0 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20005490 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.946+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe280b0f20 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.947+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe280a1c80 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.948+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe2001c070 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.948+0000 7fbe0f7fe640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe1406b4d0 0x7fbe1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.948+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe1406b4d0 0x7fbe1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.948+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbe200893e0 con 0x7fbe280cc7b0 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.949+0000 7fbe26ffd640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe1406b4d0 0x7fbe1406d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800a920 tx=0x7fbe1802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:56.953+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe200585f0 con 0x7fbe280cc7b0 2024-03-06T07:24:57.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.133+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7fbe280d2090 con 0x7fbe280cc7b0 2024-03-06T07:24:57.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.136+0000 7fbe0f7fe640 1 -- 172.21.15.122:0/2329801549 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v19) v1 ==== 101+0+0 (secure 0 0 0) 0x7fbe20057ef0 con 0x7fbe280cc7b0 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe1406b4d0 msgr2=0x7fbe1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe1406b4d0 0x7fbe1406d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800a920 tx=0x7fbe1802f000 comp rx=0 tx=0).stop 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 msgr2=0x7fbe280b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 0x7fbe280b1a00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000e370 tx=0x7fbe2000e830 comp rx=0 tx=0).stop 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 shutdown_connections 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe1406b4d0 0x7fbe1406d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe280cc7b0 0x7fbe280b1a00 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe280cbdf0 0x7fbe280cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 --2- 172.21.15.122:0/2329801549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe100039e0 0x7fbe28153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 >> 172.21.15.122:0/2329801549 conn(0x7fbe280d3cd0 msgr2=0x7fbe280cf830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 shutdown_connections 2024-03-06T07:24:57.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.139+0000 7fbe2d415640 1 -- 172.21.15.122:0/2329801549 wait complete. 2024-03-06T07:24:57.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:57.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.252+0000 7f7a4c9f6640 1 Processor -- start 2024-03-06T07:24:57.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.252+0000 7f7a4c9f6640 1 -- start start 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4c9f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 0x7f7a480aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4c9f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 0x7f7a480b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4c9f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a48158ac0 con 0x7f7a480aef80 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4c9f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4815b2a0 con 0x7f7a481586e0 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4c9f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4815b540 con 0x7f7a480d01a0 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4659c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 0x7f7a480aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a46d9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 0x7f7a480b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a46d9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 0x7f7a480b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44536/0 (socket says 172.21.15.122:44536) 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a4659c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 0x7f7a480aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:53374/0 (socket says 172.21.15.122:53374) 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.253+0000 7f7a46d9d640 1 -- 172.21.15.122:0/1088675465 learned_addr learned my addr 172.21.15.122:0/1088675465 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.254+0000 7f7a46d9d640 1 -- 172.21.15.122:0/1088675465 >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 msgr2=0x7f7a480aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.254+0000 7f7a46d9d640 1 --2- 172.21.15.122:0/1088675465 >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 0x7f7a480aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.254+0000 7f7a46d9d640 1 -- 172.21.15.122:0/1088675465 >> v1:172.21.15.169:6789/0 conn(0x7f7a480aef80 legacy=0x7f7a480cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.254+0000 7f7a46d9d640 1 -- 172.21.15.122:0/1088675465 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4815b820 con 0x7f7a480d01a0 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.254+0000 7f7a4659c640 1 --2- 172.21.15.122:0/1088675465 >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 0x7f7a480aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:57.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.254+0000 7f7a46d9d640 1 --2- 172.21.15.122:0/1088675465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 0x7f7a480b33a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009e50 tx=0x7f7a4002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7c161d152f7e8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a45d9b640 1 -- 172.21.15.122:0/1088675465 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4002cb00 con 0x7f7a480d01a0 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a45d9b640 1 -- 172.21.15.122:0/1088675465 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4002cca0 con 0x7f7a480d01a0 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a45d9b640 1 -- 172.21.15.122:0/1088675465 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40034610 con 0x7f7a480d01a0 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1088675465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 msgr2=0x7f7a480b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1088675465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 0x7f7a480b33a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009e50 tx=0x7f7a4002bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1088675465 shutdown_connections 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1088675465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a480d01a0 0x7f7a480b33a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1088675465 >> v2:172.21.15.169:3300/0 conn(0x7f7a481586e0 0x7f7a480aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1088675465 >> 172.21.15.122:0/1088675465 conn(0x7f7a4806b190 msgr2=0x7f7a4806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1088675465 shutdown_connections 2024-03-06T07:24:57.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.255+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1088675465 wait complete. 2024-03-06T07:24:57.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.256+0000 7f7a4c9f6640 1 Processor -- start 2024-03-06T07:24:57.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 -- start start 2024-03-06T07:24:57.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480aef80 0x7f7a481f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 0x7f7a480ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481586e0 0x7f7a480ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48071350 con 0x7f7a480d01a0 2024-03-06T07:24:57.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4815bde0 con 0x7f7a480aef80 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a4c9f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480714d0 con 0x7f7a481586e0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a3ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 0x7f7a480ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a3ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 0x7f7a480ac330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56832/0 (socket says 172.21.15.122:56832) 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.257+0000 7f7a3ffff640 1 -- 172.21.15.122:0/1004415476 learned_addr learned my addr 172.21.15.122:0/1004415476 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a46d9d640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481586e0 0x7f7a480ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a4659c640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480aef80 0x7f7a481f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3ffff640 1 -- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481586e0 msgr2=0x7f7a480ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3ffff640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481586e0 0x7f7a480ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3ffff640 1 -- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480aef80 msgr2=0x7f7a481f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3ffff640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480aef80 0x7f7a481f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3ffff640 1 -- 172.21.15.122:0/1004415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a481e1420 con 0x7f7a480d01a0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3ffff640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 0x7f7a480ac330 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c440 tx=0x7f7a30001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300052d0 con 0x7f7a480d01a0 2024-03-06T07:24:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30005470 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a480aced0 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30007260 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.258+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a480ad490 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.259+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4815c250 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.260+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a30007400 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.260+0000 7f7a3effd640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2806b4d0 0x7f7a2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.260+0000 7f7a4659c640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2806b4d0 0x7f7a2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.260+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7a30089380 con 0x7f7a480d01a0 2024-03-06T07:24:57.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.260+0000 7f7a4659c640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2806b4d0 0x7f7a2806d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009560 tx=0x7f7a38009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.265+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a4815c250 con 0x7f7a480d01a0 2024-03-06T07:24:57.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.437+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f7a48068ff0 con 0x7f7a480d01a0 2024-03-06T07:24:57.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.440+0000 7f7a3effd640 1 -- 172.21.15.122:0/1004415476 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v20) v1 ==== 108+0+0 (secure 0 0 0) 0x7f7a30057eb0 con 0x7f7a480d01a0 2024-03-06T07:24:57.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2806b4d0 msgr2=0x7f7a2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2806b4d0 0x7f7a2806d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009560 tx=0x7f7a38009190 comp rx=0 tx=0).stop 2024-03-06T07:24:57.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 msgr2=0x7f7a480ac330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 0x7f7a480ac330 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c440 tx=0x7f7a30001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 shutdown_connections 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2806b4d0 0x7f7a2806d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481586e0 0x7f7a480ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480d01a0 0x7f7a480ac330 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 --2- 172.21.15.122:0/1004415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480aef80 0x7f7a481f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 >> 172.21.15.122:0/1004415476 conn(0x7f7a4806b190 msgr2=0x7f7a480b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 shutdown_connections 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.442+0000 7f7a4c9f6640 1 -- 172.21.15.122:0/1004415476 wait complete. 2024-03-06T07:24:57.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:57.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.555+0000 7f161d5c7640 1 Processor -- start 2024-03-06T07:24:57.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.555+0000 7f161d5c7640 1 -- start start 2024-03-06T07:24:57.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f161d5c7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16181614d0 0x7f16180d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f161d5c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f16180d1910 0x7f16180d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f161d5c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d22e0 0x7f16180d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f161d5c7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16180d2750 con 0x7f16181614d0 2024-03-06T07:24:57.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f161d5c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16181618b0 con 0x7f16180d1910 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f161d5c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16180cf400 con 0x7f16180d22e0 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16181614d0 0x7f16180d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16181614d0 0x7f16180d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56834/0 (socket says 172.21.15.122:56834) 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 -- 172.21.15.122:0/3941671305 learned_addr learned my addr 172.21.15.122:0/3941671305 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f16167fc640 1 --2- 172.21.15.122:0/3941671305 >> v2:172.21.15.169:3300/0 conn(0x7f16180d1910 0x7f16180d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 -- 172.21.15.122:0/3941671305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d22e0 msgr2=0x7f16180d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f16177fe640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d22e0 0x7f16180d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d22e0 0x7f16180d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 -- 172.21.15.122:0/3941671305 >> v2:172.21.15.169:3300/0 conn(0x7f16180d1910 msgr2=0x7f16180d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 --2- 172.21.15.122:0/3941671305 >> v2:172.21.15.169:3300/0 conn(0x7f16180d1910 0x7f16180d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f1616ffd640 1 -- 172.21.15.122:0/3941671305 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180cf6a0 con 0x7f16181614d0 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.556+0000 7f16177fe640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d22e0 0x7f16180d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.557+0000 7f1616ffd640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16181614d0 0x7f16180d4860 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f16180ce550 tx=0x7f160802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb88e3312204636 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.557+0000 7f1615ffb640 1 -- 172.21.15.122:0/3941671305 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1608035020 con 0x7f16181614d0 2024-03-06T07:24:57.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.557+0000 7f1615ffb640 1 -- 172.21.15.122:0/3941671305 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1608031870 con 0x7f16181614d0 2024-03-06T07:24:57.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.557+0000 7f1615ffb640 1 -- 172.21.15.122:0/3941671305 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1608031c40 con 0x7f16181614d0 2024-03-06T07:24:57.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 -- 172.21.15.122:0/3941671305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16181614d0 msgr2=0x7f16180d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16181614d0 0x7f16180d4860 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f16180ce550 tx=0x7f160802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:57.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 -- 172.21.15.122:0/3941671305 shutdown_connections 2024-03-06T07:24:57.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d22e0 0x7f16180d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 --2- 172.21.15.122:0/3941671305 >> v2:172.21.15.169:3300/0 conn(0x7f16180d1910 0x7f16180d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 --2- 172.21.15.122:0/3941671305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16181614d0 0x7f16180d4860 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 -- 172.21.15.122:0/3941671305 >> 172.21.15.122:0/3941671305 conn(0x7f16180d3d40 msgr2=0x7f16180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 -- 172.21.15.122:0/3941671305 shutdown_connections 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.558+0000 7f161d5c7640 1 -- 172.21.15.122:0/3941671305 wait complete. 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.559+0000 7f161d5c7640 1 Processor -- start 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.559+0000 7f161d5c7640 1 -- start start 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f161d5c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 0x7f16181537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f161d5c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180d22e0 0x7f1618151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f161d5c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16181614d0 0x7f161814e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f161d5c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16180b0910 con 0x7f16180d1910 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f161d5c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16180cfc60 con 0x7f16180d22e0 2024-03-06T07:24:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f161d5c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16180d0be0 con 0x7f16181614d0 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 0x7f16181537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 0x7f16181537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56846/0 (socket says 172.21.15.122:56846) 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 -- 172.21.15.122:0/2624867575 learned_addr learned my addr 172.21.15.122:0/2624867575 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 -- 172.21.15.122:0/2624867575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16181614d0 msgr2=0x7f161814e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16181614d0 0x7f161814e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 -- 172.21.15.122:0/2624867575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180d22e0 msgr2=0x7f1618151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180d22e0 0x7f1618151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 -- 172.21.15.122:0/2624867575 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618152720 con 0x7f16180d1910 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f1616ffd640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 0x7f16181537c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f160802b950 tx=0x7f16080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1608040070 con 0x7f16180d1910 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.560+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1608030d00 con 0x7f16180d1910 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.561+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1608035040 con 0x7f16180d1910 2024-03-06T07:24:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.561+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1618152a00 con 0x7f16180d1910 2024-03-06T07:24:57.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.561+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1618152fc0 con 0x7f16180d1910 2024-03-06T07:24:57.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.561+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1618068ff0 con 0x7f16180d1910 2024-03-06T07:24:57.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.562+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f160804b060 con 0x7f16180d1910 2024-03-06T07:24:57.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.562+0000 7f15ff7fe640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f160406b4d0 0x7f160406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.562+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f16080abc40 con 0x7f16180d1910 2024-03-06T07:24:57.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.562+0000 7f16167fc640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f160406b4d0 0x7f160406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.563+0000 7f16167fc640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f160406b4d0 0x7f160406d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f16181e6870 tx=0x7f160c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.567+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f160807adc0 con 0x7f16180d1910 2024-03-06T07:24:57.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.740+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f16180d4250 con 0x7f16180d1910 2024-03-06T07:24:57.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.743+0000 7f15ff7fe640 1 -- 172.21.15.122:0/2624867575 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f160807a6c0 con 0x7f16180d1910 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f160406b4d0 msgr2=0x7f160406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f160406b4d0 0x7f160406d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f16181e6870 tx=0x7f160c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 msgr2=0x7f16181537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 0x7f16181537c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f160802b950 tx=0x7f16080046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 shutdown_connections 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f160406b4d0 0x7f160406d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16181614d0 0x7f161814e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180d22e0 0x7f1618151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 --2- 172.21.15.122:0/2624867575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d1910 0x7f16181537c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 >> 172.21.15.122:0/2624867575 conn(0x7f16180d3d40 msgr2=0x7f16180b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 shutdown_connections 2024-03-06T07:24:57.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.745+0000 7f161d5c7640 1 -- 172.21.15.122:0/2624867575 wait complete. 2024-03-06T07:24:57.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:57.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.855+0000 7f90a55a0640 1 Processor -- start 2024-03-06T07:24:57.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.856+0000 7f90a55a0640 1 -- start start 2024-03-06T07:24:57.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.856+0000 7f90a55a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 0x7f90a00b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.856+0000 7f90a55a0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f90a015cb60 0x7f90a00b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.856+0000 7f90a55a0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a015cff0 con 0x7f90a00b7370 2024-03-06T07:24:57.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.856+0000 7f90a55a0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a00b77d0 con 0x7f90a015cb60 2024-03-06T07:24:57.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.856+0000 7f90a55a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0152fa0 con 0x7f90a0152bc0 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f90a4d9f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f90a015cb60 0x7f90a00b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 0x7f90a00b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 0x7f90a00b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44560/0 (socket says 172.21.15.122:44560) 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 -- 172.21.15.122:0/3048802108 learned_addr learned my addr 172.21.15.122:0/3048802108 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f90a4d9f640 1 --2- 172.21.15.122:0/3048802108 >> v2:172.21.15.169:3300/0 conn(0x7f90a015cb60 0x7f90a00b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:53404/0 (socket says 172.21.15.122:53404) 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 -- 172.21.15.122:0/3048802108 >> v2:172.21.15.169:3300/0 conn(0x7f90a015cb60 msgr2=0x7f90a00b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 --2- 172.21.15.122:0/3048802108 >> v2:172.21.15.169:3300/0 conn(0x7f90a015cb60 0x7f90a00b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 -- 172.21.15.122:0/3048802108 >> v1:172.21.15.169:6789/0 conn(0x7f90a00b7370 legacy=0x7f90a015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.857+0000 7f909ffff640 1 -- 172.21.15.122:0/3048802108 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a00d2780 con 0x7f90a0152bc0 2024-03-06T07:24:57.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.858+0000 7f909ffff640 1 --2- 172.21.15.122:0/3048802108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 0x7f90a00b6e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9090009870 tx=0x7f909002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69d08f244f8b463 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.858+0000 7f909effd640 1 -- 172.21.15.122:0/3048802108 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090034070 con 0x7f90a0152bc0 2024-03-06T07:24:57.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.858+0000 7f909effd640 1 -- 172.21.15.122:0/3048802108 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090030840 con 0x7f90a0152bc0 2024-03-06T07:24:57.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.858+0000 7f909effd640 1 -- 172.21.15.122:0/3048802108 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090030c10 con 0x7f90a0152bc0 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 -- 172.21.15.122:0/3048802108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 msgr2=0x7f90a00b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 --2- 172.21.15.122:0/3048802108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 0x7f90a00b6e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9090009870 tx=0x7f909002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 -- 172.21.15.122:0/3048802108 shutdown_connections 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 --2- 172.21.15.122:0/3048802108 >> v2:172.21.15.169:3300/0 conn(0x7f90a015cb60 0x7f90a00b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 --2- 172.21.15.122:0/3048802108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a0152bc0 0x7f90a00b6e30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 -- 172.21.15.122:0/3048802108 >> 172.21.15.122:0/3048802108 conn(0x7f90a009a9f0 msgr2=0x7f90a009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 -- 172.21.15.122:0/3048802108 shutdown_connections 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.859+0000 7f90a55a0640 1 -- 172.21.15.122:0/3048802108 wait complete. 2024-03-06T07:24:57.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.860+0000 7f90a55a0640 1 Processor -- start 2024-03-06T07:24:57.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.860+0000 7f90a55a0640 1 -- start start 2024-03-06T07:24:57.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a55a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a00b7370 0x7f90a0146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a55a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 0x7f90a01e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a55a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90a015cb60 0x7f90a01e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a55a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a00d1ca0 con 0x7f90a0152bc0 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a55a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a00d19c0 con 0x7f90a015cb60 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a55a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a00d1f80 con 0x7f90a00b7370 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 0x7f90a01e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 0x7f90a01e54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56858/0 (socket says 172.21.15.122:56858) 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 -- 172.21.15.122:0/1696848511 learned_addr learned my addr 172.21.15.122:0/1696848511 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909ffff640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a00b7370 0x7f90a0146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f90a4d9f640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90a015cb60 0x7f90a01e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 -- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a00b7370 msgr2=0x7f90a0146fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a00b7370 0x7f90a0146fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 -- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90a015cb60 msgr2=0x7f90a01e5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90a015cb60 0x7f90a01e5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 -- 172.21.15.122:0/1696848511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a01e7380 con 0x7f90a0152bc0 2024-03-06T07:24:57.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.861+0000 7f909f7fe640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 0x7f90a01e54d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f909400c400 tx=0x7f909400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.862+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094017070 con 0x7f90a0152bc0 2024-03-06T07:24:57.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.862+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90940052d0 con 0x7f90a0152bc0 2024-03-06T07:24:57.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.862+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094003830 con 0x7f90a0152bc0 2024-03-06T07:24:57.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.862+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a01e7660 con 0x7f90a0152bc0 2024-03-06T07:24:57.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.862+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a01e6580 con 0x7f90a0152bc0 2024-03-06T07:24:57.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.863+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a01e6780 con 0x7f90a0152bc0 2024-03-06T07:24:57.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.863+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f909401a070 con 0x7f90a0152bc0 2024-03-06T07:24:57.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.864+0000 7f909cff9640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f908c06b500 0x7f908c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.864+0000 7f909ffff640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f908c06b500 0x7f908c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.864+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9094089210 con 0x7f90a0152bc0 2024-03-06T07:24:57.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.864+0000 7f909ffff640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f908c06b500 0x7f908c06d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9090036000 tx=0x7f909003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:57.868+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9094058390 con 0x7f90a0152bc0 2024-03-06T07:24:58.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.106+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f90a0068f80 con 0x7f90a0152bc0 2024-03-06T07:24:58.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.820+0000 7f909cff9640 1 -- 172.21.15.122:0/1696848511 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v183) v1 ==== 144+0+0 (secure 0 0 0) 0x7f9094057c90 con 0x7f90a0152bc0 2024-03-06T07:24:58.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f908c06b500 msgr2=0x7f908c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f908c06b500 0x7f908c06d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9090036000 tx=0x7f909003b040 comp rx=0 tx=0).stop 2024-03-06T07:24:58.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 msgr2=0x7f90a01e54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 0x7f90a01e54d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f909400c400 tx=0x7f909400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 shutdown_connections 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f908c06b500 0x7f908c06d9c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90a015cb60 0x7f90a01e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90a0152bc0 0x7f90a01e54d0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 --2- 172.21.15.122:0/1696848511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90a00b7370 0x7f90a0146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 >> 172.21.15.122:0/1696848511 conn(0x7f90a009a9f0 msgr2=0x7f90a009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 shutdown_connections 2024-03-06T07:24:58.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.823+0000 7f90a55a0640 1 -- 172.21.15.122:0/1696848511 wait complete. 2024-03-06T07:24:58.838 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:24:58.838 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:24:58.839 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:58.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:58.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.942+0000 7f275bb16640 1 Processor -- start 2024-03-06T07:24:58.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.942+0000 7f275bb16640 1 -- start start 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275bb16640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f275415bb20 0x7f27540d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275bb16640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f27540cd8c0 0x7f27540d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275bb16640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2754161980 con 0x7f27540cfff0 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275bb16640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2754161c60 con 0x7f27540cd8c0 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275bb16640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2754161f40 con 0x7f275415bb20 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275a0b3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f27540cd8c0 0x7f27540d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275a0b3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f27540cd8c0 0x7f27540d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56860/0 (socket says 172.21.15.122:56860) 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.943+0000 7f275a0b3640 1 -- 172.21.15.122:0/2710271306 learned_addr learned my addr 172.21.15.122:0/2710271306 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.944+0000 7f27598b2640 1 --2- 172.21.15.122:0/2710271306 >> v2:172.21.15.169:3300/0 conn(0x7f275415bb20 0x7f27540d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.944+0000 7f275a0b3640 1 -- 172.21.15.122:0/2710271306 >> v2:172.21.15.169:3300/0 conn(0x7f275415bb20 msgr2=0x7f27540d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.944+0000 7f275a0b3640 1 --2- 172.21.15.122:0/2710271306 >> v2:172.21.15.169:3300/0 conn(0x7f275415bb20 0x7f27540d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.944+0000 7f275a0b3640 1 -- 172.21.15.122:0/2710271306 >> v1:172.21.15.169:6789/0 conn(0x7f27540cfff0 legacy=0x7f27540ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.944+0000 7f275a0b3640 1 -- 172.21.15.122:0/2710271306 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2754162220 con 0x7f27540cd8c0 2024-03-06T07:24:58.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.944+0000 7f275a0b3640 1 --2- 172.21.15.122:0/2710271306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27540cd8c0 0x7f27540d04a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f27540d2bc0 tx=0x7f274c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4836a4ecd686e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f27588b0640 1 -- 172.21.15.122:0/2710271306 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c035020 con 0x7f27540cd8c0 2024-03-06T07:24:58.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f27588b0640 1 -- 172.21.15.122:0/2710271306 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c031870 con 0x7f27540cd8c0 2024-03-06T07:24:58.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f27588b0640 1 -- 172.21.15.122:0/2710271306 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c031c40 con 0x7f27540cd8c0 2024-03-06T07:24:58.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 -- 172.21.15.122:0/2710271306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27540cd8c0 msgr2=0x7f27540d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 --2- 172.21.15.122:0/2710271306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27540cd8c0 0x7f27540d04a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f27540d2bc0 tx=0x7f274c02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 -- 172.21.15.122:0/2710271306 shutdown_connections 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 --2- 172.21.15.122:0/2710271306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27540cd8c0 0x7f27540d04a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 --2- 172.21.15.122:0/2710271306 >> v2:172.21.15.169:3300/0 conn(0x7f275415bb20 0x7f27540d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 -- 172.21.15.122:0/2710271306 >> 172.21.15.122:0/2710271306 conn(0x7f275409a9f0 msgr2=0x7f275409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 -- 172.21.15.122:0/2710271306 shutdown_connections 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.945+0000 7f275bb16640 1 -- 172.21.15.122:0/2710271306 wait complete. 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.946+0000 7f275bb16640 1 Processor -- start 2024-03-06T07:24:58.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.946+0000 7f275bb16640 1 -- start start 2024-03-06T07:24:58.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275bb16640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27540cd8c0 0x7f2754154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275bb16640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27540cfff0 0x7f27540c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275bb16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 0x7f275415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275bb16640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27540d24b0 con 0x7f275415bb20 2024-03-06T07:24:58.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275bb16640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27540d2330 con 0x7f27540cfff0 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275bb16640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27540d1980 con 0x7f27540cd8c0 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 0x7f275415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 0x7f275415a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56868/0 (socket says 172.21.15.122:56868) 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 -- 172.21.15.122:0/250925382 learned_addr learned my addr 172.21.15.122:0/250925382 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f27590b1640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27540cfff0 0x7f27540c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f27598b2640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27540cd8c0 0x7f2754154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 -- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27540cd8c0 msgr2=0x7f2754154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27540cd8c0 0x7f2754154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 -- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27540cfff0 msgr2=0x7f27540c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27540cfff0 0x7f27540c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.947+0000 7f275a0b3640 1 -- 172.21.15.122:0/250925382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27540c9f80 con 0x7f275415bb20 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.948+0000 7f275a0b3640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 0x7f275415a0d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f274c02b950 tx=0x7f274c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.948+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c040070 con 0x7f275415bb20 2024-03-06T07:24:58.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.948+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c030d00 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.948+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27540ca260 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.948+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c035040 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.948+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27540ca820 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.949+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27540cac00 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.949+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f274c038020 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.950+0000 7f274a7fc640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f273806b500 0x7f273806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.950+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f274c0abcd0 con 0x7f275415bb20 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.950+0000 7f27598b2640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f273806b500 0x7f273806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.950+0000 7f27598b2640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f273806b500 0x7f273806d9c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f27500099c0 tx=0x7f2750003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:58.954+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f274c07ae50 con 0x7f275415bb20 2024-03-06T07:24:59.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.128+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27540cf740 con 0x7f275415bb20 2024-03-06T07:24:59.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.129+0000 7f274a7fc640 1 -- 172.21.15.122:0/250925382 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+13022 (secure 0 0 0) 0x7f274c07a750 con 0x7f275415bb20 2024-03-06T07:24:59.131 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:24:59.131 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":60,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:24:56.814556+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:6834/2303645671":"2024-03-07T07:24:53.807239+0000","172.21.15.169:6832/2303645671":"2024-03-07T07:24:53.807239+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:59.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f273806b500 msgr2=0x7f273806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f273806b500 0x7f273806d9c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f27500099c0 tx=0x7f2750003990 comp rx=0 tx=0).stop 2024-03-06T07:24:59.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 msgr2=0x7f275415a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 0x7f275415a0d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f274c02b950 tx=0x7f274c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 shutdown_connections 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f273806b500 0x7f273806d9c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f275415bb20 0x7f275415a0d0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27540cfff0 0x7f27540c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 --2- 172.21.15.122:0/250925382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27540cd8c0 0x7f2754154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 >> 172.21.15.122:0/250925382 conn(0x7f275409a9f0 msgr2=0x7f27540b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 shutdown_connections 2024-03-06T07:24:59.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.131+0000 7f275bb16640 1 -- 172.21.15.122:0/250925382 wait complete. 2024-03-06T07:24:59.144 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:59.144 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:59.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.246+0000 7f9a33fbd640 1 Processor -- start 2024-03-06T07:24:59.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.246+0000 7f9a33fbd640 1 -- start start 2024-03-06T07:24:59.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.246+0000 7f9a33fbd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c1586e0 0x7f9a2c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a33fbd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 0x7f9a2c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a33fbd640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c158ac0 con 0x7f9a2c0aef80 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a33fbd640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2c15b2a0 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a33fbd640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c15b540 con 0x7f9a2c1586e0 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a3255a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 0x7f9a2c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a31d59640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c1586e0 0x7f9a2c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a3255a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 0x7f9a2c0b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:53436/0 (socket says 172.21.15.122:53436) 2024-03-06T07:24:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a3255a640 1 -- 172.21.15.122:0/144153202 learned_addr learned my addr 172.21.15.122:0/144153202 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a31d59640 1 -- 172.21.15.122:0/144153202 >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 msgr2=0x7f9a2c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a31d59640 1 --2- 172.21.15.122:0/144153202 >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 0x7f9a2c0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a31d59640 1 -- 172.21.15.122:0/144153202 >> v1:172.21.15.169:6789/0 conn(0x7f9a2c0aef80 legacy=0x7f9a2c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a31d59640 1 -- 172.21.15.122:0/144153202 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c15b820 con 0x7f9a2c1586e0 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.247+0000 7f9a3255a640 1 --2- 172.21.15.122:0/144153202 >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 0x7f9a2c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.248+0000 7f9a31d59640 1 --2- 172.21.15.122:0/144153202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c1586e0 0x7f9a2c0aea40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9a200097a0 tx=0x7f9a2002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511dfa8fc9e3776e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.248+0000 7f9a30d57640 1 -- 172.21.15.122:0/144153202 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20035020 con 0x7f9a2c1586e0 2024-03-06T07:24:59.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.248+0000 7f9a30d57640 1 -- 172.21.15.122:0/144153202 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a20031830 con 0x7f9a2c1586e0 2024-03-06T07:24:59.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.248+0000 7f9a30d57640 1 -- 172.21.15.122:0/144153202 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20031c00 con 0x7f9a2c1586e0 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 -- 172.21.15.122:0/144153202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c1586e0 msgr2=0x7f9a2c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/144153202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c1586e0 0x7f9a2c0aea40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9a200097a0 tx=0x7f9a2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 -- 172.21.15.122:0/144153202 shutdown_connections 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/144153202 >> v2:172.21.15.169:3300/0 conn(0x7f9a2c0d01a0 0x7f9a2c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/144153202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c1586e0 0x7f9a2c0aea40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 -- 172.21.15.122:0/144153202 >> 172.21.15.122:0/144153202 conn(0x7f9a2c06b190 msgr2=0x7f9a2c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 -- 172.21.15.122:0/144153202 shutdown_connections 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.249+0000 7f9a33fbd640 1 -- 172.21.15.122:0/144153202 wait complete. 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.250+0000 7f9a33fbd640 1 Processor -- start 2024-03-06T07:24:59.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.250+0000 7f9a33fbd640 1 -- start start 2024-03-06T07:24:59.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c0aef80 0x7f9a2c144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 0x7f9a2c1e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a2c1586e0 0x7f9a2c144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c071350 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c15bde0 con 0x7f9a2c1586e0 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0714d0 con 0x7f9a2c0aef80 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 0x7f9a2c1e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 0x7f9a2c1e7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56882/0 (socket says 172.21.15.122:56882) 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 -- 172.21.15.122:0/2285311729 learned_addr learned my addr 172.21.15.122:0/2285311729 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31d59640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c0aef80 0x7f9a2c144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a3255a640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a2c1586e0 0x7f9a2c144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 -- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c0aef80 msgr2=0x7f9a2c144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c0aef80 0x7f9a2c144380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 -- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a2c1586e0 msgr2=0x7f9a2c144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a2c1586e0 0x7f9a2c144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 -- 172.21.15.122:0/2285311729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c1448c0 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a3255a640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a2c1586e0 0x7f9a2c144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a31558640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 0x7f9a2c1e7af0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800c3c0 tx=0x7f9a2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a280052d0 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a28005490 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c1e1d20 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a28007260 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.251+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c1e2580 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.252+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c0732b0 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.253+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a28007400 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.253+0000 7f9a1e7fc640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a1006b4d0 0x7f9a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.253+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9a28089330 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.253+0000 7f9a31d59640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a1006b4d0 0x7f9a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.254+0000 7f9a31d59640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a1006b4d0 0x7f9a1006d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9a20036000 tx=0x7f9a20008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.258+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a2c1448c0 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.427+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a2c1555b0 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.428+0000 7f9a1e7fc640 1 -- 172.21.15.122:0/2285311729 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a2805ce20 con 0x7f9a2c0d01a0 2024-03-06T07:24:59.430 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a1006b4d0 msgr2=0x7f9a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a1006b4d0 0x7f9a1006d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9a20036000 tx=0x7f9a20008030 comp rx=0 tx=0).stop 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 msgr2=0x7f9a2c1e7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 0x7f9a2c1e7af0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800c3c0 tx=0x7f9a2800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 shutdown_connections 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a1006b4d0 0x7f9a1006d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a2c1586e0 0x7f9a2c144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a2c0d01a0 0x7f9a2c1e7af0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 --2- 172.21.15.122:0/2285311729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a2c0aef80 0x7f9a2c144380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 >> 172.21.15.122:0/2285311729 conn(0x7f9a2c06b190 msgr2=0x7f9a2c0722d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 shutdown_connections 2024-03-06T07:24:59.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.430+0000 7f9a33fbd640 1 -- 172.21.15.122:0/2285311729 wait complete. 2024-03-06T07:24:59.443 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.540+0000 7f385e8e1640 1 Processor -- start 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.540+0000 7f385e8e1640 1 -- start start 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f385e8e1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f385813dc10 0x7f38580b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f385e8e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f38580b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f385e8e1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385815f580 con 0x7f385815bf30 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f385e8e1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38580cf3f0 con 0x7f385813dc10 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f385e8e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385815b210 con 0x7f385815ef60 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f3857fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f385813dc10 0x7f38580b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f3857fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f385813dc10 0x7f38580b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56886/0 (socket says 172.21.15.122:56886) 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f3857fff640 1 -- 172.21.15.122:0/2287943744 learned_addr learned my addr 172.21.15.122:0/2287943744 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f385ce7e640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f38580b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f3857fff640 1 -- 172.21.15.122:0/2287943744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 msgr2=0x7f38580b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.541+0000 7f3857fff640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f38580b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f3857fff640 1 -- 172.21.15.122:0/2287943744 >> v1:172.21.15.169:6789/0 conn(0x7f385815bf30 legacy=0x7f385815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f3857fff640 1 -- 172.21.15.122:0/2287943744 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385815b450 con 0x7f385813dc10 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f385ce7e640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f38580b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f3857fff640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38580b2950 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3848009870 tx=0x7f384802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63fbbf52ba1445 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f3856ffd640 1 -- 172.21.15.122:0/2287943744 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38480317c0 con 0x7f385813dc10 2024-03-06T07:24:59.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f3856ffd640 1 -- 172.21.15.122:0/2287943744 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3848037070 con 0x7f385813dc10 2024-03-06T07:24:59.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.542+0000 7f3856ffd640 1 -- 172.21.15.122:0/2287943744 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848031b80 con 0x7f385813dc10 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 -- 172.21.15.122:0/2287943744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 msgr2=0x7f38580b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38580b2950 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3848009870 tx=0x7f384802b860 comp rx=0 tx=0).stop 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 -- 172.21.15.122:0/2287943744 shutdown_connections 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f38580b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2287943744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38580b2950 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 -- 172.21.15.122:0/2287943744 >> 172.21.15.122:0/2287943744 conn(0x7f38580cf8a0 msgr2=0x7f38580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 -- 172.21.15.122:0/2287943744 shutdown_connections 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.543+0000 7f385e8e1640 1 -- 172.21.15.122:0/2287943744 wait complete. 2024-03-06T07:24:59.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.544+0000 7f385e8e1640 1 Processor -- start 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.544+0000 7f385e8e1640 1 -- start start 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385e8e1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38581f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385e8e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f385815bf30 0x7f385813e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385e8e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f385813e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385e8e1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38580708d0 con 0x7f385813dc10 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385e8e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385813dff0 con 0x7f385815bf30 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385e8e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3858070a50 con 0x7f385815ef60 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38581f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38581f4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56892/0 (socket says 172.21.15.122:56892) 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 -- 172.21.15.122:0/2825226680 learned_addr learned my addr 172.21.15.122:0/2825226680 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385ce7e640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f385813e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f38577fe640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f385815bf30 0x7f385813e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 -- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 msgr2=0x7f385813e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f385813e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 -- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f385815bf30 msgr2=0x7f385813e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f385815bf30 0x7f385813e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f3857fff640 1 -- 172.21.15.122:0/2825226680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385812cf50 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f385ce7e640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f385813e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.545+0000 7f38577fe640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f385815bf30 0x7f385813e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.546+0000 7f3857fff640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38581f4ad0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3848009870 tx=0x7f38480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.546+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848032860 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.546+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3848032a00 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.546+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385812d230 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.546+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848032df0 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.546+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385812d7f0 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.547+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385804f330 con 0x7f385813dc10 2024-03-06T07:24:59.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.547+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f384804c060 con 0x7f385813dc10 2024-03-06T07:24:59.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.547+0000 7f3854ff9640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f384406b4d0 0x7f384406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.548+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f38480abc20 con 0x7f385813dc10 2024-03-06T07:24:59.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.548+0000 7f38577fe640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f384406b4d0 0x7f384406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.548+0000 7f38577fe640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f384406b4d0 0x7f384406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f38580b34a0 tx=0x7f384c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.552+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3848040070 con 0x7f385813dc10 2024-03-06T07:24:59.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.788+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f385806eb00 con 0x7f385813dc10 2024-03-06T07:24:59.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.825+0000 7f3854ff9640 1 -- 172.21.15.122:0/2825226680 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v184) v1 ==== 143+0+0 (secure 0 0 0) 0x7f384807a7e0 con 0x7f385813dc10 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f384406b4d0 msgr2=0x7f384406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f384406b4d0 0x7f384406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f38580b34a0 tx=0x7f384c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 msgr2=0x7f38581f4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38581f4ad0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3848009870 tx=0x7f38480046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 shutdown_connections 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f384406b4d0 0x7f384406d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f385815ef60 0x7f385813e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f385815bf30 0x7f385813e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 --2- 172.21.15.122:0/2825226680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f385813dc10 0x7f38581f4ad0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 >> 172.21.15.122:0/2825226680 conn(0x7f38580cf8a0 msgr2=0x7f3858072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 shutdown_connections 2024-03-06T07:24:59.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.827+0000 7f385e8e1640 1 -- 172.21.15.122:0/2825226680 wait complete. 2024-03-06T07:24:59.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:59.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.941+0000 7fbb36edf640 1 Processor -- start 2024-03-06T07:24:59.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.941+0000 7fbb36edf640 1 -- start start 2024-03-06T07:24:59.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.942+0000 7fbb36edf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 0x7fbb300aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.942+0000 7fbb36edf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300d01a0 0x7fbb300b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.942+0000 7fbb36edf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb30158ac0 con 0x7fbb300aef80 2024-03-06T07:24:59.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.942+0000 7fbb36edf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3015b2a0 con 0x7fbb301586e0 2024-03-06T07:24:59.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.942+0000 7fbb36edf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3015b540 con 0x7fbb300d01a0 2024-03-06T07:24:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb34c7b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 0x7fbb300aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb3547c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300d01a0 0x7fbb300b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb34c7b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 0x7fbb300aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:53460/0 (socket says 172.21.15.122:53460) 2024-03-06T07:24:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb34c7b640 1 -- 172.21.15.122:0/59861079 learned_addr learned my addr 172.21.15.122:0/59861079 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb3547c640 1 -- 172.21.15.122:0/59861079 >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 msgr2=0x7fbb300aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb3547c640 1 --2- 172.21.15.122:0/59861079 >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 0x7fbb300aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb3547c640 1 -- 172.21.15.122:0/59861079 >> v1:172.21.15.169:6789/0 conn(0x7fbb300aef80 legacy=0x7fbb300cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb3547c640 1 -- 172.21.15.122:0/59861079 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3015b820 con 0x7fbb300d01a0 2024-03-06T07:24:59.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.943+0000 7fbb34c7b640 1 --2- 172.21.15.122:0/59861079 >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 0x7fbb300aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb3547c640 1 --2- 172.21.15.122:0/59861079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300d01a0 0x7fbb300b33a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009d90 tx=0x7fbb2802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34ac06b8412b1429 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb2f7fe640 1 -- 172.21.15.122:0/59861079 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28039070 con 0x7fbb300d01a0 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb2f7fe640 1 -- 172.21.15.122:0/59861079 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2802cc50 con 0x7fbb300d01a0 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb2f7fe640 1 -- 172.21.15.122:0/59861079 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28034440 con 0x7fbb300d01a0 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb36edf640 1 -- 172.21.15.122:0/59861079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300d01a0 msgr2=0x7fbb300b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb36edf640 1 --2- 172.21.15.122:0/59861079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300d01a0 0x7fbb300b33a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009d90 tx=0x7fbb2802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.944+0000 7fbb36edf640 1 -- 172.21.15.122:0/59861079 shutdown_connections 2024-03-06T07:24:59.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.945+0000 7fbb36edf640 1 --2- 172.21.15.122:0/59861079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300d01a0 0x7fbb300b33a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.945+0000 7fbb36edf640 1 --2- 172.21.15.122:0/59861079 >> v2:172.21.15.169:3300/0 conn(0x7fbb301586e0 0x7fbb300aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.945+0000 7fbb36edf640 1 -- 172.21.15.122:0/59861079 >> 172.21.15.122:0/59861079 conn(0x7fbb3006b190 msgr2=0x7fbb3006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.945+0000 7fbb36edf640 1 -- 172.21.15.122:0/59861079 shutdown_connections 2024-03-06T07:24:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.945+0000 7fbb36edf640 1 -- 172.21.15.122:0/59861079 wait complete. 2024-03-06T07:24:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 Processor -- start 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 -- start start 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300aef80 0x7fbb30152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 0x7fbb30155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb301586e0 0x7fbb30156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30071350 con 0x7fbb300d01a0 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3015bde0 con 0x7fbb301586e0 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.946+0000 7fbb36edf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300714d0 con 0x7fbb300aef80 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 0x7fbb30155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 0x7fbb30155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56902/0 (socket says 172.21.15.122:56902) 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 -- 172.21.15.122:0/2376643194 learned_addr learned my addr 172.21.15.122:0/2376643194 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb3547c640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb301586e0 0x7fbb30156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 -- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300aef80 msgr2=0x7fbb30152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb34c7b640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300aef80 0x7fbb30152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300aef80 0x7fbb30152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 -- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb301586e0 msgr2=0x7fbb30156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb301586e0 0x7fbb30156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 -- 172.21.15.122:0/2376643194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb301f4b60 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb3547c640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb301586e0 0x7fbb30156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb34c7b640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300aef80 0x7fbb30152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2ffff640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 0x7fbb30155ae0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400c3f0 tx=0x7fbb2400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24017070 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb240052f0 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb301f7600 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24003830 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.947+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb301f7bc0 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.948+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb300732b0 con 0x7fbb300d01a0 2024-03-06T07:24:59.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.949+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb2401a070 con 0x7fbb300d01a0 2024-03-06T07:24:59.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.949+0000 7fbb2d7fa640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.949+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbb24089260 con 0x7fbb300d01a0 2024-03-06T07:24:59.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.949+0000 7fbb34c7b640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.950+0000 7fbb34c7b640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009630 tx=0x7fbb20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:24:59.954+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb240583e0 con 0x7fbb300d01a0 2024-03-06T07:25:00.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.124+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbb30068ff0 con 0x7fbb300d01a0 2024-03-06T07:25:00.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.127+0000 7fbb2d7fa640 1 -- 172.21.15.122:0/2376643194 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbb24089080 con 0x7fbb300d01a0 2024-03-06T07:25:00.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb1c06b4d0 msgr2=0x7fbb1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009630 tx=0x7fbb20008040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 msgr2=0x7fbb30155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 0x7fbb30155ae0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400c3f0 tx=0x7fbb2400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 shutdown_connections 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb301586e0 0x7fbb30156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb300d01a0 0x7fbb30155ae0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 --2- 172.21.15.122:0/2376643194 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb300aef80 0x7fbb30152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 >> 172.21.15.122:0/2376643194 conn(0x7fbb3006b190 msgr2=0x7fbb300b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 shutdown_connections 2024-03-06T07:25:00.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.129+0000 7fbb36edf640 1 -- 172.21.15.122:0/2376643194 wait complete. 2024-03-06T07:25:00.142 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:00.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:00.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:00.227 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:00.227 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:00.228 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:00.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:00.243 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:25:00.243 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:00.260 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:00.260 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:00.260 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:00.298 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:00.298 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:25:00.298 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:00.317 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:00.317 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:00.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:00.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.436+0000 7f646dcc2640 1 Processor -- start 2024-03-06T07:25:00.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.436+0000 7f646dcc2640 1 -- start start 2024-03-06T07:25:00.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f646dcc2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646815bb20 0x7f64680d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f646dcc2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64680cfff0 0x7f64680d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f646dcc2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468161980 con 0x7f64680ce4b0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f646dcc2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6468161c60 con 0x7f64680cfff0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f646dcc2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468161f40 con 0x7f646815bb20 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f6466ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64680cfff0 0x7f64680d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f6467fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f64680ce4b0 0x7f64680ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33220/0 (socket says 172.21.15.122:33220) 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.437+0000 7f6467fff640 1 -- 172.21.15.122:0/4213047858 learned_addr learned my addr 172.21.15.122:0/4213047858 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f64677fe640 1 --2- 172.21.15.122:0/4213047858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646815bb20 0x7f64680d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f6466ffd640 1 -- 172.21.15.122:0/4213047858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646815bb20 msgr2=0x7f64680d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f6466ffd640 1 --2- 172.21.15.122:0/4213047858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646815bb20 0x7f64680d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f6466ffd640 1 -- 172.21.15.122:0/4213047858 >> v1:172.21.15.122:6789/0 conn(0x7f64680ce4b0 legacy=0x7f64680ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f6466ffd640 1 -- 172.21.15.122:0/4213047858 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468162220 con 0x7f64680cfff0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f64667fc640 1 -- 172.21.15.122:0/4213047858 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085568194 0 0) 0x7f6468161980 con 0x7f64680ce4b0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.438+0000 7f6466ffd640 1 --2- 172.21.15.122:0/4213047858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f64680d0470 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f64580097a0 tx=0x7f645802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b74556fb76ec9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f64667fc640 1 -- 172.21.15.122:0/4213047858 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458035020 con 0x7f64680cfff0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f64667fc640 1 -- 172.21.15.122:0/4213047858 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6458031830 con 0x7f64680cfff0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f64667fc640 1 -- 172.21.15.122:0/4213047858 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458031c00 con 0x7f64680cfff0 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 -- 172.21.15.122:0/4213047858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 msgr2=0x7f64680d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 --2- 172.21.15.122:0/4213047858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f64680d0470 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f64580097a0 tx=0x7f645802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 -- 172.21.15.122:0/4213047858 shutdown_connections 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 --2- 172.21.15.122:0/4213047858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f64680d0470 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 --2- 172.21.15.122:0/4213047858 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646815bb20 0x7f64680d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 -- 172.21.15.122:0/4213047858 >> 172.21.15.122:0/4213047858 conn(0x7f646809a9f0 msgr2=0x7f646809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 -- 172.21.15.122:0/4213047858 shutdown_connections 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.439+0000 7f646dcc2640 1 -- 172.21.15.122:0/4213047858 wait complete. 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.440+0000 7f646dcc2640 1 Processor -- start 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 -- start start 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64680ce4b0 0x7f64681f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f6468159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646815bb20 0x7f64680b6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680d24b0 con 0x7f64680cfff0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680d2330 con 0x7f646815bb20 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f646dcc2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64680d1980 con 0x7f64680ce4b0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f6466ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f6468159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f6466ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f6468159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56926/0 (socket says 172.21.15.122:56926) 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.441+0000 7f6466ffd640 1 -- 172.21.15.122:0/166428787 learned_addr learned my addr 172.21.15.122:0/166428787 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f64677fe640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64680ce4b0 0x7f64681f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6467fff640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646815bb20 0x7f64680b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6466ffd640 1 -- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64680ce4b0 msgr2=0x7f64681f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6466ffd640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64680ce4b0 0x7f64681f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6466ffd640 1 -- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646815bb20 msgr2=0x7f64680b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6466ffd640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646815bb20 0x7f64680b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6466ffd640 1 -- 172.21.15.122:0/166428787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646815d880 con 0x7f64680cfff0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f6466ffd640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f6468159450 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f64580097a0 tx=0x7f64580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458035040 con 0x7f64680cfff0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6458004230 con 0x7f64680cfff0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64680b6ee0 con 0x7f64680cfff0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645803a700 con 0x7f64680cfff0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.442+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646815d080 con 0x7f64680cfff0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.443+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646804f330 con 0x7f64680cfff0 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.444+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6458038020 con 0x7f64680cfff0 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.444+0000 7f644ffff640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645406b4d0 0x7f645406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.444+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f64580abce0 con 0x7f64680cfff0 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.444+0000 7f64677fe640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645406b4d0 0x7f645406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.444+0000 7f64677fe640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645406b4d0 0x7f645406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f645c006fd0 tx=0x7f645c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.448+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f646815d880 con 0x7f64680cfff0 2024-03-06T07:25:00.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.656+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64680b1820 con 0x7f64680cfff0 2024-03-06T07:25:00.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.656+0000 7f644ffff640 1 -- 172.21.15.122:0/166428787 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6458031800 con 0x7f64680cfff0 2024-03-06T07:25:00.659 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:00.659 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5771,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/2962352349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2962352349},{"type":"v1","addr":"172.21.15.169:6834","nonce":2962352349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/967487316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":967487316},{"type":"v1","addr":"172.21.15.169:6835","nonce":967487316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6147,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3545939761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3545939761},{"type":"v1","addr":"172.21.15.122:6836","nonce":3545939761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/817024731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":817024731},{"type":"v1","addr":"172.21.15.122:6837","nonce":817024731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[]} 2024-03-06T07:25:00.659 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 29 2024-03-06T07:25:00.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645406b4d0 msgr2=0x7f645406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645406b4d0 0x7f645406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f645c006fd0 tx=0x7f645c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 msgr2=0x7f6468159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f6468159450 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f64580097a0 tx=0x7f64580046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 shutdown_connections 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f645406b4d0 0x7f645406d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646815bb20 0x7f64680b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64680cfff0 0x7f6468159450 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 --2- 172.21.15.122:0/166428787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64680ce4b0 0x7f64681f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 >> 172.21.15.122:0/166428787 conn(0x7f646809a9f0 msgr2=0x7f64680d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 shutdown_connections 2024-03-06T07:25:00.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.658+0000 7f646dcc2640 1 -- 172.21.15.122:0/166428787 wait complete. 2024-03-06T07:25:00.671 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:00.671 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:00.672 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:00.672 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:00.672 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:00.672 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:00.672 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:00.672 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:00.673 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:00.673 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:00.673 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:00.673 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:00.673 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:00.673 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:00.673 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:00.673 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:00.681 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:00.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:00.682 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:00.682 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:00.686 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:00.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:00.686 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:00.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:00.724 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:00.724 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.725 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:00.725 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.729 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:00.729 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.730 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:00.730 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:00.800 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:25:00.800 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:25:00.815 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:25:00.815 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:25:00.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.833+0000 7f5991b93640 1 Processor -- start 2024-03-06T07:25:00.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.833+0000 7f5991b93640 1 -- start start 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f5991b93640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f598c162400 0x7f598c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f5991b93640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0d22c0 con 0x7f598c162970 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f5991b93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0d25a0 con 0x7f598c0b4460 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f5991b93640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c161b00 con 0x7f598c162400 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f598affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f598c162970 0x7f598c161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33258/0 (socket says 172.21.15.122:33258) 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f598affd640 1 -- 172.21.15.122:0/3341407438 learned_addr learned my addr 172.21.15.122:0/3341407438 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.834+0000 7f598bfff640 1 --2- 172.21.15.122:0/3341407438 >> v2:172.21.15.169:3300/0 conn(0x7f598c162400 0x7f598c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.835+0000 7f598a7fc640 1 -- 172.21.15.122:0/3341407438 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263113522 0 0) 0x7f598c0d22c0 con 0x7f598c162970 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.835+0000 7f598a7fc640 1 -- 172.21.15.122:0/3341407438 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5974003520 con 0x7f598c162970 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.835+0000 7f598bfff640 1 -- 172.21.15.122:0/3341407438 >> v1:172.21.15.169:6789/0 conn(0x7f598c0b4460 legacy=0x7f598c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.835+0000 7f598bfff640 1 -- 172.21.15.122:0/3341407438 >> v1:172.21.15.122:6789/0 conn(0x7f598c162970 legacy=0x7f598c161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.835+0000 7f598bfff640 1 -- 172.21.15.122:0/3341407438 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0d2880 con 0x7f598c162400 2024-03-06T07:25:00.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.835+0000 7f598bfff640 1 --2- 172.21.15.122:0/3341407438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162400 0x7f598c0d4800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5984009d90 tx=0x7f598402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afc90b9a310fea04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.837+0000 7f598a7fc640 1 -- 172.21.15.122:0/3341407438 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984039070 con 0x7f598c162400 2024-03-06T07:25:00.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.837+0000 7f598a7fc640 1 -- 172.21.15.122:0/3341407438 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598402cd00 con 0x7f598c162400 2024-03-06T07:25:00.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.837+0000 7f598a7fc640 1 -- 172.21.15.122:0/3341407438 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984034660 con 0x7f598c162400 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 -- 172.21.15.122:0/3341407438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162400 msgr2=0x7f598c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 --2- 172.21.15.122:0/3341407438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162400 0x7f598c0d4800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5984009d90 tx=0x7f598402bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 -- 172.21.15.122:0/3341407438 shutdown_connections 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 --2- 172.21.15.122:0/3341407438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162400 0x7f598c0d4800 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 -- 172.21.15.122:0/3341407438 >> 172.21.15.122:0/3341407438 conn(0x7f598c0d3ce0 msgr2=0x7f598c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 -- 172.21.15.122:0/3341407438 shutdown_connections 2024-03-06T07:25:00.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.838+0000 7f5991b93640 1 -- 172.21.15.122:0/3341407438 wait complete. 2024-03-06T07:25:00.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.839+0000 7f5991b93640 1 Processor -- start 2024-03-06T07:25:00.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.839+0000 7f5991b93640 1 -- start start 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f5991b93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 0x7f598c1561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f5991b93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f598c162400 0x7f598c0b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f5991b93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162970 0x7f598c0b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f5991b93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0ce950 con 0x7f598c0b4460 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f5991b93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0cecc0 con 0x7f598c162970 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f5991b93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0b4840 con 0x7f598c162400 2024-03-06T07:25:00.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 0x7f598c1561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 0x7f598c1561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56972/0 (socket says 172.21.15.122:56972) 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 -- 172.21.15.122:0/1761420950 learned_addr learned my addr 172.21.15.122:0/1761420950 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598affd640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f598c162400 0x7f598c0b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598bfff640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162970 0x7f598c0b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 -- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f598c162400 msgr2=0x7f598c0b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f598c162400 0x7f598c0b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 -- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162970 msgr2=0x7f598c0b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162970 0x7f598c0b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 -- 172.21.15.122:0/1761420950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0b7a30 con 0x7f598c0b4460 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.840+0000 7f598b7fe640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 0x7f598c1561a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f597c00c3c0 tx=0x7f597c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.841+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c0052d0 con 0x7f598c0b4460 2024-03-06T07:25:00.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.841+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c005490 con 0x7f598c0b4460 2024-03-06T07:25:00.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.841+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c007260 con 0x7f598c0b4460 2024-03-06T07:25:00.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.841+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0b7470 con 0x7f598c0b4460 2024-03-06T07:25:00.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.841+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0b7d90 con 0x7f598c0b4460 2024-03-06T07:25:00.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.842+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c04f330 con 0x7f598c0b4460 2024-03-06T07:25:00.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.842+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f597c007400 con 0x7f598c0b4460 2024-03-06T07:25:00.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.843+0000 7f5973fff640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f597806b4d0 0x7f597806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.843+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f597c0892a0 con 0x7f598c0b4460 2024-03-06T07:25:00.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.843+0000 7f598affd640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f597806b4d0 0x7f597806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.843+0000 7f598affd640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f597806b4d0 0x7f597806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f598c0cde20 tx=0x7f5980007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:00.847+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f597c057e60 con 0x7f598c0b4460 2024-03-06T07:25:01.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.033+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f598c06b180 con 0x7f598c0b4460 2024-03-06T07:25:01.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.840+0000 7f5973fff640 1 -- 172.21.15.122:0/1761420950 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v65) v1 ==== 97+0+0 (secure 0 0 0) 0x7f597c05ce20 con 0x7f598c0b4460 2024-03-06T07:25:01.842 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f597806b4d0 msgr2=0x7f597806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f597806b4d0 0x7f597806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f598c0cde20 tx=0x7f5980007d30 comp rx=0 tx=0).stop 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 msgr2=0x7f598c1561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 0x7f598c1561a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f597c00c3c0 tx=0x7f597c00c880 comp rx=0 tx=0).stop 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 shutdown_connections 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f597806b4d0 0x7f597806d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f598c162970 0x7f598c0b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f598c162400 0x7f598c0b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 --2- 172.21.15.122:0/1761420950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f598c0b4460 0x7f598c1561a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 >> 172.21.15.122:0/1761420950 conn(0x7f598c0d3ce0 msgr2=0x7f598c0d09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 shutdown_connections 2024-03-06T07:25:01.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.843+0000 7f5991b93640 1 -- 172.21.15.122:0/1761420950 wait complete. 2024-03-06T07:25:01.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:01.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.961+0000 7fe1fe615640 1 Processor -- start 2024-03-06T07:25:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.961+0000 7fe1fe615640 1 -- start start 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1fe615640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe1f815dfe0 0x7fe1f80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1fe615640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80b3430 con 0x7fe1f80cf830 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1fe615640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80b3710 con 0x7fe1f80d29c0 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1fe615640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f80b31e0 con 0x7fe1f815dfe0 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1f7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe1f815dfe0 0x7fe1f80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1f77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe1f80cf830 0x7fe1f80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33264/0 (socket says 172.21.15.122:33264) 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.962+0000 7fe1f77fe640 1 -- 172.21.15.122:0/850474963 learned_addr learned my addr 172.21.15.122:0/850474963 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.963+0000 7fe1f7fff640 1 -- 172.21.15.122:0/850474963 >> v1:172.21.15.169:6789/0 conn(0x7fe1f80d29c0 legacy=0x7fe1f80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.963+0000 7fe1f7fff640 1 -- 172.21.15.122:0/850474963 >> v1:172.21.15.122:6789/0 conn(0x7fe1f80cf830 legacy=0x7fe1f80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.963+0000 7fe1f6ffd640 1 -- 172.21.15.122:0/850474963 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322555713 0 0) 0x7fe1f80b3430 con 0x7fe1f80cf830 2024-03-06T07:25:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.963+0000 7fe1f7fff640 1 -- 172.21.15.122:0/850474963 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f80b39f0 con 0x7fe1f815dfe0 2024-03-06T07:25:01.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.963+0000 7fe1f7fff640 1 --2- 172.21.15.122:0/850474963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80cf2f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80b3360 tx=0x7fe1e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b12a00bfb4a1d8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1f6ffd640 1 -- 172.21.15.122:0/850474963 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8035020 con 0x7fe1f815dfe0 2024-03-06T07:25:01.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1f6ffd640 1 -- 172.21.15.122:0/850474963 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8031830 con 0x7fe1f815dfe0 2024-03-06T07:25:01.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1f6ffd640 1 -- 172.21.15.122:0/850474963 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8031c00 con 0x7fe1f815dfe0 2024-03-06T07:25:01.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 -- 172.21.15.122:0/850474963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f815dfe0 msgr2=0x7fe1f80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 --2- 172.21.15.122:0/850474963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80cf2f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80b3360 tx=0x7fe1e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 -- 172.21.15.122:0/850474963 shutdown_connections 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 --2- 172.21.15.122:0/850474963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80cf2f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 -- 172.21.15.122:0/850474963 >> 172.21.15.122:0/850474963 conn(0x7fe1f80d3cd0 msgr2=0x7fe1f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 -- 172.21.15.122:0/850474963 shutdown_connections 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.964+0000 7fe1fe615640 1 -- 172.21.15.122:0/850474963 wait complete. 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.965+0000 7fe1fe615640 1 Processor -- start 2024-03-06T07:25:01.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.965+0000 7fe1fe615640 1 -- start start 2024-03-06T07:25:01.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1fe615640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1f80cf830 0x7fe1f81f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1fe615640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 0x7fe1f815f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1fe615640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1fe615640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f815e3c0 con 0x7fe1f80d29c0 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1fe615640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80b3d60 con 0x7fe1f815dfe0 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1fe615640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f815e540 con 0x7fe1f80cf830 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1f77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 0x7fe1f815f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1f77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 0x7fe1f815f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56998/0 (socket says 172.21.15.122:56998) 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.966+0000 7fe1f77fe640 1 -- 172.21.15.122:0/4038065419 learned_addr learned my addr 172.21.15.122:0/4038065419 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f7fff640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1f80cf830 0x7fe1f81f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1fcbb2640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f77fe640 1 -- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1f80cf830 msgr2=0x7fe1f81f33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f77fe640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1f80cf830 0x7fe1f81f33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f77fe640 1 -- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f815dfe0 msgr2=0x7fe1f80ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f77fe640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f77fe640 1 -- 172.21.15.122:0/4038065419 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f815fec0 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1fcbb2640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f77fe640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 0x7fe1f815f660 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00c460 tx=0x7fe1ec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec018070 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec007520 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec004a20 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f81601a0 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.967+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f8160760 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.968+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f8160960 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.968+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1ec01b070 con 0x7fe1f80d29c0 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.969+0000 7fe1f4ff9640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe1e406b4d0 0x7fe1e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.969+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe1ec0891c0 con 0x7fe1f80d29c0 2024-03-06T07:25:01.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.969+0000 7fe1f7fff640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe1e406b4d0 0x7fe1e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.969+0000 7fe1f7fff640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe1e406b4d0 0x7fe1e406d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8036000 tx=0x7fe1e803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:01.973+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1ec058420 con 0x7fe1f80d29c0 2024-03-06T07:25:02.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.211+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe1f80b8d20 con 0x7fe1f80d29c0 2024-03-06T07:25:02.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.211+0000 7fe1f4ff9640 1 -- 172.21.15.122:0/4038065419 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe1ec057d20 con 0x7fe1f80d29c0 2024-03-06T07:25:02.213 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.214 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:02.215 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.217 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.219 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:02.224 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:25:02.227 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:02.227 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:02.227 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.227 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:02.227 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:25:02.227 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe1e406b4d0 msgr2=0x7fe1e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe1e406b4d0 0x7fe1e406d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8036000 tx=0x7fe1e803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 msgr2=0x7fe1f815f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 0x7fe1f815f660 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00c460 tx=0x7fe1ec00c920 comp rx=0 tx=0).stop 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 shutdown_connections 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe1e406b4d0 0x7fe1e406d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f815dfe0 0x7fe1f80ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1f80d29c0 0x7fe1f815f660 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 --2- 172.21.15.122:0/4038065419 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1f80cf830 0x7fe1f81f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 >> 172.21.15.122:0/4038065419 conn(0x7fe1f80d3cd0 msgr2=0x7fe1f80b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 shutdown_connections 2024-03-06T07:25:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.213+0000 7fe1fe615640 1 -- 172.21.15.122:0/4038065419 wait complete. 2024-03-06T07:25:02.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:02.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.328+0000 7f6006df7640 1 Processor -- start 2024-03-06T07:25:02.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.328+0000 7f6006df7640 1 -- start start 2024-03-06T07:25:02.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.329+0000 7f6006df7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f60000d06a0 0x7f60000cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.329+0000 7f6006df7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60000cef50 con 0x7f60000d02c0 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.329+0000 7f6006df7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600015c620 con 0x7f60000b6550 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.329+0000 7f6006df7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60000b50b0 con 0x7f60000d06a0 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.329+0000 7f6004b93640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f60000d02c0 0x7f60000ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33272/0 (socket says 172.21.15.122:33272) 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.329+0000 7f6004b93640 1 -- 172.21.15.122:0/1349347034 learned_addr learned my addr 172.21.15.122:0/1349347034 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 --2- 172.21.15.122:0/1349347034 >> v2:172.21.15.169:3300/0 conn(0x7f60000d06a0 0x7f60000cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149800043 0 0) 0x7f60000cef50 con 0x7f60000d02c0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8003520 con 0x7f60000d02c0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 347581343 0 0) 0x7f5fe8003520 con 0x7f60000d02c0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 >> v2:172.21.15.169:3300/0 conn(0x7f60000d06a0 msgr2=0x7f60000cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 --2- 172.21.15.122:0/1349347034 >> v2:172.21.15.169:3300/0 conn(0x7f60000d06a0 0x7f60000cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 >> v1:172.21.15.169:6789/0 conn(0x7f60000b6550 legacy=0x7f60000ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000cd800 con 0x7f60000d02c0 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f5ff0001670 con 0x7f60000d02c0 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 >> v1:172.21.15.122:6789/0 conn(0x7f60000d02c0 legacy=0x7f60000ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5fe80039b0 0x7f5fe8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5fe8004800 0x7f5fe8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f5fe800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60000b50b0 con 0x7f5fe80051c0 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe800c060 con 0x7f5fe80039b0 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe800c260 con 0x7f5fe8004800 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f5fe800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6004b93640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5fe80039b0 0x7f5fe8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5ffffff640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5fe8004800 0x7f5fe8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 -- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5fe8004800 msgr2=0x7f5fe8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5fe8004800 0x7f5fe8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 -- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5fe80039b0 msgr2=0x7f5fe8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5fe80039b0 0x7f5fe8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6005394640 1 -- 172.21.15.122:0/1349347034 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe8003520 con 0x7f5fe80051c0 2024-03-06T07:25:02.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f6004b93640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5fe80039b0 0x7f5fe8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:02.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.330+0000 7f5ffffff640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5fe8004800 0x7f5fe8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:02.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.331+0000 7f6005394640 1 -- 172.21.15.122:0/1349347034 reap_dead start 2024-03-06T07:25:02.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.331+0000 7f6005394640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f5fe800b9c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009fc0 tx=0x7f5fe8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6012c21e5ec32176 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.331+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8009c10 con 0x7f5fe80051c0 2024-03-06T07:25:02.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.331+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff802cc60 con 0x7f5fe80051c0 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.331+0000 7f5fff7fe640 1 -- 172.21.15.122:0/1349347034 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8003bd0 con 0x7f5fe80051c0 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 -- 172.21.15.122:0/1349347034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 msgr2=0x7f5fe800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f5fe800b9c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009fc0 tx=0x7f5fe8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 -- 172.21.15.122:0/1349347034 shutdown_connections 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 --2- 172.21.15.122:0/1349347034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f5fe800b9c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 -- 172.21.15.122:0/1349347034 >> 172.21.15.122:0/1349347034 conn(0x7f60000b35a0 msgr2=0x7f60000b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 -- 172.21.15.122:0/1349347034 shutdown_connections 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.332+0000 7f6006df7640 1 -- 172.21.15.122:0/1349347034 wait complete. 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.333+0000 7f6006df7640 1 Processor -- start 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.333+0000 7f6006df7640 1 -- start start 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f60000cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60000c8b40 0x7f6000157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f600006c0c0 0x7f600015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60000cf850 con 0x7f5fe80051c0 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60000b4d40 con 0x7f60000c8b40 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60000cf9d0 con 0x7f600006c0c0 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f60000cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f60000cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57012/0 (socket says 172.21.15.122:57012) 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 -- 172.21.15.122:0/3913636982 learned_addr learned my addr 172.21.15.122:0/3913636982 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6005394640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f600006c0c0 0x7f600015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f5ffffff640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60000c8b40 0x7f6000157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 -- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f600006c0c0 msgr2=0x7f600015a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f600006c0c0 0x7f600015a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 -- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60000c8b40 msgr2=0x7f6000157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60000c8b40 0x7f6000157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 -- 172.21.15.122:0/3913636982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000c9600 con 0x7f5fe80051c0 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f5ffffff640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60000c8b40 0x7f6000157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6004b93640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f60000cbbc0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000e7e0 tx=0x7f5ff000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff000c870 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff0018070 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60000c98e0 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff000cc50 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.334+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600006c570 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.336+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600006c890 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.336+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ff0013420 con 0x7f5fe80051c0 2024-03-06T07:25:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.336+0000 7f5ffd7fa640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fec06b500 0x7f5fec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.336+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5ff0089a10 con 0x7f5fe80051c0 2024-03-06T07:25:02.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.336+0000 7f5ffffff640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fec06b500 0x7f5fec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.337+0000 7f5ffffff640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fec06b500 0x7f5fec06d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4000c00 tx=0x7f5ff4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.340+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f600006c890 con 0x7f5fe80051c0 2024-03-06T07:25:02.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.513+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60000c8f20 con 0x7f5fe80051c0 2024-03-06T07:25:02.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.514+0000 7f5ffd7fa640 1 -- 172.21.15.122:0/3913636982 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+12891 (secure 0 0 0) 0x7f5ff0058c70 con 0x7f5fe80051c0 2024-03-06T07:25:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:02.517 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":65,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:01.839598+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:02.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fec06b500 msgr2=0x7f5fec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fec06b500 0x7f5fec06d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4000c00 tx=0x7f5ff4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 msgr2=0x7f60000cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f60000cbbc0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000e7e0 tx=0x7f5ff000eca0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 shutdown_connections 2024-03-06T07:25:02.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5fec06b500 0x7f5fec06d9c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f600006c0c0 0x7f600015a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60000c8b40 0x7f6000157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 --2- 172.21.15.122:0/3913636982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe80051c0 0x7f60000cbbc0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 >> 172.21.15.122:0/3913636982 conn(0x7f60000b35a0 msgr2=0x7f6000070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 shutdown_connections 2024-03-06T07:25:02.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.516+0000 7f6006df7640 1 -- 172.21.15.122:0/3913636982 wait complete. 2024-03-06T07:25:02.530 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:02.530 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:02.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.633+0000 7ffa341b6640 1 Processor -- start 2024-03-06T07:25:02.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.633+0000 7ffa341b6640 1 -- start start 2024-03-06T07:25:02.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa341b6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa2c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa341b6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0b3320 con 0x7ffa2c06da00 2024-03-06T07:25:02.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa341b6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c15ec80 con 0x7ffa2c0712d0 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa341b6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cf610 con 0x7ffa2c15a220 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa32753640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffa2c06da00 0x7ffa2c15e570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33274/0 (socket says 172.21.15.122:33274) 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa32753640 1 -- 172.21.15.122:0/1510998987 learned_addr learned my addr 172.21.15.122:0/1510998987 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa31f52640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa2c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872540154 0 0) 0x7ffa2c0b3320 con 0x7ffa2c06da00 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18003520 con 0x7ffa2c06da00 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997926717 0 0) 0x7ffa18003520 con 0x7ffa2c06da00 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 msgr2=0x7ffa2c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa2c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 >> v1:172.21.15.169:6789/0 conn(0x7ffa2c0712d0 legacy=0x7ffa2c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0728d0 con 0x7ffa2c06da00 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.634+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ffa240019f0 con 0x7ffa2c06da00 2024-03-06T07:25:02.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 >> v1:172.21.15.122:6789/0 conn(0x7ffa2c06da00 legacy=0x7ffa2c15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa18003f60 0x7ffa18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18004d40 0x7ffa180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cf610 con 0x7ffa18003f60 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1800cc60 con 0x7ffa18004d40 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1800ce60 con 0x7ffa2c15a220 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31751640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa18003f60 0x7ffa18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31f52640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa32753640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18004d40 0x7ffa180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31751640 1 -- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 msgr2=0x7ffa180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31751640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31751640 1 -- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18004d40 msgr2=0x7ffa180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31751640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18004d40 0x7ffa180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31751640 1 -- 172.21.15.122:0/1510998987 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18003520 con 0x7ffa18003f60 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.635+0000 7ffa31f52640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a220 0x7ffa180039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa31751640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa18003f60 0x7ffa18004800 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c001750 tx=0x7ffa18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3acc2dfb7ac6ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c001370 con 0x7ffa18003f60 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c0052d0 con 0x7ffa18003f60 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa30f50640 1 -- 172.21.15.122:0/1510998987 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003830 con 0x7ffa18003f60 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa341b6640 1 -- 172.21.15.122:0/1510998987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa18003f60 msgr2=0x7ffa18004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa341b6640 1 --2- 172.21.15.122:0/1510998987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa18003f60 0x7ffa18004800 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c001750 tx=0x7ffa18005830 comp rx=0 tx=0).stop 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa32753640 1 -- 172.21.15.122:0/1510998987 reap_dead start 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa341b6640 1 -- 172.21.15.122:0/1510998987 shutdown_connections 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa341b6640 1 -- 172.21.15.122:0/1510998987 >> 172.21.15.122:0/1510998987 conn(0x7ffa2c06b190 msgr2=0x7ffa2c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa341b6640 1 -- 172.21.15.122:0/1510998987 shutdown_connections 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.636+0000 7ffa341b6640 1 -- 172.21.15.122:0/1510998987 wait complete. 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 Processor -- start 2024-03-06T07:25:02.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 -- start start 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18003f60 0x7ffa2c15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a600 0x7ffa2c1f7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 0x7ffa2c155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c072030 con 0x7ffa2c1554e0 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c072bb0 con 0x7ffa18003f60 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa341b6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c072530 con 0x7ffa2c15a600 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa32753640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 0x7ffa2c155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa32753640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 0x7ffa2c155970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57034/0 (socket says 172.21.15.122:57034) 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.638+0000 7ffa32753640 1 -- 172.21.15.122:0/3501423164 learned_addr learned my addr 172.21.15.122:0/3501423164 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa31751640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a600 0x7ffa2c1f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa31f52640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18003f60 0x7ffa2c15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa32753640 1 -- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a600 msgr2=0x7ffa2c1f7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa32753640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a600 0x7ffa2c1f7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa32753640 1 -- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18003f60 msgr2=0x7ffa2c15c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa32753640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18003f60 0x7ffa2c15c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa32753640 1 -- 172.21.15.122:0/3501423164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c15cb70 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa32753640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 0x7ffa2c155970 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c15ef00 tx=0x7ffa2400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400eb30 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24018070 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c15ce50 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24013420 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.639+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c15d410 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.640+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c0732b0 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.640+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa240135c0 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.641+0000 7ffa227fc640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa1006b4d0 0x7ffa1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.641+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffa24089ee0 con 0x7ffa2c1554e0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.641+0000 7ffa31f52640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa1006b4d0 0x7ffa1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.641+0000 7ffa31f52640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa1006b4d0 0x7ffa1006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0cf4c0 tx=0x7ffa28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.645+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa24058bb0 con 0x7ffa2c1554e0 2024-03-06T07:25:02.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.814+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa2c155eb0 con 0x7ffa2c1554e0 2024-03-06T07:25:02.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.814+0000 7ffa227fc640 1 -- 172.21.15.122:0/3501423164 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa2405cc50 con 0x7ffa2c1554e0 2024-03-06T07:25:02.816 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa1006b4d0 msgr2=0x7ffa1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa1006b4d0 0x7ffa1006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0cf4c0 tx=0x7ffa28008040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 msgr2=0x7ffa2c155970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 0x7ffa2c155970 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c15ef00 tx=0x7ffa2400e530 comp rx=0 tx=0).stop 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 shutdown_connections 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa1006b4d0 0x7ffa1006d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c1554e0 0x7ffa2c155970 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa2c15a600 0x7ffa2c1f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 --2- 172.21.15.122:0/3501423164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa18003f60 0x7ffa2c15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 >> 172.21.15.122:0/3501423164 conn(0x7ffa2c06b190 msgr2=0x7ffa2c0ae240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 shutdown_connections 2024-03-06T07:25:02.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.816+0000 7ffa341b6640 1 -- 172.21.15.122:0/3501423164 wait complete. 2024-03-06T07:25:02.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:02.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 Processor -- start 2024-03-06T07:25:02.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 -- start start 2024-03-06T07:25:02.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff62015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff62015bf30 0x7ff62015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62015f2d0 con 0x7ff62013dc10 2024-03-06T07:25:02.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200cf3f0 con 0x7ff62015bf30 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff626c08640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015b210 con 0x7ff6200b3060 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.927+0000 7ff6249a4640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff62013dc10 0x7ff6200b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33276/0 (socket says 172.21.15.122:33276) 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff6249a4640 1 -- 172.21.15.122:0/86158193 learned_addr learned my addr 172.21.15.122:0/86158193 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff62015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff6251a5640 1 --2- 172.21.15.122:0/86158193 >> v2:172.21.15.169:3300/0 conn(0x7ff62015bf30 0x7ff62015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164744244 0 0) 0x7ff62015f2d0 con 0x7ff62013dc10 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff610003520 con 0x7ff62013dc10 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2894791913 0 0) 0x7ff610003520 con 0x7ff62013dc10 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 msgr2=0x7ff62015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff62015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 >> v2:172.21.15.169:3300/0 conn(0x7ff62015bf30 msgr2=0x7ff62015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 --2- 172.21.15.122:0/86158193 >> v2:172.21.15.169:3300/0 conn(0x7ff62015bf30 0x7ff62015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62015f580 con 0x7ff62013dc10 2024-03-06T07:25:02.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff614001670 con 0x7ff62013dc10 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 >> v1:172.21.15.122:6789/0 conn(0x7ff62013dc10 legacy=0x7ff6200b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff6100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff610004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff610004d40 0x7ff6100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200cf3f0 con 0x7ff610003f30 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015b210 con 0x7ff610004d40 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61000ba10 con 0x7ff6200b3060 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff610004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff6249a4640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff6100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff6251a5640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff610004d40 0x7ff6100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 -- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 msgr2=0x7ff6100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff6100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 -- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff610004d40 msgr2=0x7ff6100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff610004d40 0x7ff6100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff61ffff640 1 -- 172.21.15.122:0/86158193 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610003520 con 0x7ff610003f30 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.928+0000 7ff6249a4640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6200b3060 0x7ff6100039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.929+0000 7ff61ffff640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff610004800 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009af0 tx=0x7ff610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74cf6217e35a71b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.929+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c039070 con 0x7ff610003f30 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.929+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c02c6b0 con 0x7ff610003f30 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.929+0000 7ff61f7fe640 1 -- 172.21.15.122:0/86158193 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c009690 con 0x7ff610003f30 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff626c08640 1 -- 172.21.15.122:0/86158193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 msgr2=0x7ff610004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff626c08640 1 --2- 172.21.15.122:0/86158193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff610004800 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009af0 tx=0x7ff610005830 comp rx=0 tx=0).stop 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff6251a5640 1 -- 172.21.15.122:0/86158193 reap_dead start 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff626c08640 1 -- 172.21.15.122:0/86158193 shutdown_connections 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff626c08640 1 -- 172.21.15.122:0/86158193 >> 172.21.15.122:0/86158193 conn(0x7ff6200cf8a0 msgr2=0x7ff6200cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff626c08640 1 -- 172.21.15.122:0/86158193 shutdown_connections 2024-03-06T07:25:02.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.930+0000 7ff626c08640 1 -- 172.21.15.122:0/86158193 wait complete. 2024-03-06T07:25:02.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.931+0000 7ff626c08640 1 Processor -- start 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.931+0000 7ff626c08640 1 -- start start 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff626c08640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff6201592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff626c08640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6201f5130 0x7ff6201f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff626c08640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff62015cc10 0x7ff62015d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff626c08640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62013dff0 con 0x7ff610003f30 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff626c08640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015b730 con 0x7ff62015cc10 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff626c08640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015e850 con 0x7ff6201f5130 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff6201592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff6201592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57058/0 (socket says 172.21.15.122:57058) 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 -- 172.21.15.122:0/818317879 learned_addr learned my addr 172.21.15.122:0/818317879 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6251a5640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff62015cc10 0x7ff62015d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff61ffff640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6201f5130 0x7ff6201f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 -- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6201f5130 msgr2=0x7ff6201f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6201f5130 0x7ff6201f48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 -- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff62015cc10 msgr2=0x7ff62015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff62015cc10 0x7ff62015d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 -- 172.21.15.122:0/818317879 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6201f5630 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff61ffff640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6201f5130 0x7ff6201f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.932+0000 7ff6249a4640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff6201592a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff61400e820 tx=0x7ff61400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.933+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400c870 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.933+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff614018070 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.933+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6201f5910 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.933+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400cc10 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.933+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6201f5ed0 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.934+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62004f330 con 0x7ff610003f30 2024-03-06T07:25:02.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.934+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff614013420 con 0x7ff610003f30 2024-03-06T07:25:02.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.934+0000 7ff61d7fa640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b500 0x7ff60406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.934+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff614089970 con 0x7ff610003f30 2024-03-06T07:25:02.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.934+0000 7ff61ffff640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b500 0x7ff60406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.935+0000 7ff61ffff640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b500 0x7ff60406d9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff60c036000 tx=0x7ff60c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:02.939+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff614058bd0 con 0x7ff610003f30 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.112+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff62006eb00 con 0x7ff610003f30 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.112+0000 7ff61d7fa640 1 -- 172.21.15.122:0/818317879 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff6140584d0 con 0x7ff610003f30 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:03.114 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:25:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b500 msgr2=0x7ff60406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b500 0x7ff60406d9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff60c036000 tx=0x7ff60c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 msgr2=0x7ff6201592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff6201592a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff61400e820 tx=0x7ff61400ece0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 shutdown_connections 2024-03-06T07:25:03.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b500 0x7ff60406d9c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff62015cc10 0x7ff62015d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6201f5130 0x7ff6201f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 --2- 172.21.15.122:0/818317879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff610003f30 0x7ff6201592a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 >> 172.21.15.122:0/818317879 conn(0x7ff6200cf8a0 msgr2=0x7ff62006e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 shutdown_connections 2024-03-06T07:25:03.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.114+0000 7ff626c08640 1 -- 172.21.15.122:0/818317879 wait complete. 2024-03-06T07:25:03.126 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:03.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:03.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.228+0000 7fbfbe55e640 1 Processor -- start 2024-03-06T07:25:03.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfbe55e640 1 -- start start 2024-03-06T07:25:03.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfbe55e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfb8070d10 0x7fbfb80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfbe55e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80711c0 con 0x7fbfb80708b0 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfbe55e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb815afe0 con 0x7fbfb815ac00 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfbe55e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb80cdc10 con 0x7fbfb8070d10 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfbcafb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfb8070d10 0x7fbfb80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfb77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbfb80708b0 0x7fbfb80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33292/0 (socket says 172.21.15.122:33292) 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.229+0000 7fbfb77fe640 1 -- 172.21.15.122:0/559207925 learned_addr learned my addr 172.21.15.122:0/559207925 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.230+0000 7fbfbcafb640 1 -- 172.21.15.122:0/559207925 >> v1:172.21.15.169:6789/0 conn(0x7fbfb815ac00 legacy=0x7fbfb806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.230+0000 7fbfbcafb640 1 -- 172.21.15.122:0/559207925 >> v1:172.21.15.122:6789/0 conn(0x7fbfb80708b0 legacy=0x7fbfb80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.230+0000 7fbfbcafb640 1 -- 172.21.15.122:0/559207925 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb80cd000 con 0x7fbfb8070d10 2024-03-06T07:25:03.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.230+0000 7fbfb6ffd640 1 -- 172.21.15.122:0/559207925 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952101303 0 0) 0x7fbfb80711c0 con 0x7fbfb80708b0 2024-03-06T07:25:03.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.230+0000 7fbfbcafb640 1 --2- 172.21.15.122:0/559207925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb8070d10 0x7fbfb80ce7b0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009d90 tx=0x7fbfb002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce98d5e5edca550 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfb6ffd640 1 -- 172.21.15.122:0/559207925 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0039070 con 0x7fbfb8070d10 2024-03-06T07:25:03.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfb6ffd640 1 -- 172.21.15.122:0/559207925 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb002cbd0 con 0x7fbfb8070d10 2024-03-06T07:25:03.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfb6ffd640 1 -- 172.21.15.122:0/559207925 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0034440 con 0x7fbfb8070d10 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 -- 172.21.15.122:0/559207925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb8070d10 msgr2=0x7fbfb80ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/559207925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb8070d10 0x7fbfb80ce7b0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009d90 tx=0x7fbfb002be20 comp rx=0 tx=0).stop 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 -- 172.21.15.122:0/559207925 shutdown_connections 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/559207925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb8070d10 0x7fbfb80ce7b0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 -- 172.21.15.122:0/559207925 >> 172.21.15.122:0/559207925 conn(0x7fbfb80b1650 msgr2=0x7fbfb80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 -- 172.21.15.122:0/559207925 shutdown_connections 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.231+0000 7fbfbe55e640 1 -- 172.21.15.122:0/559207925 wait complete. 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.232+0000 7fbfbe55e640 1 Processor -- start 2024-03-06T07:25:03.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.232+0000 7fbfbe55e640 1 -- start start 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbe55e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 0x7fbfb81449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbe55e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfb8070d10 0x7fbfb81f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbe55e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfb815ac00 0x7fbfb8145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbe55e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80cd8a0 con 0x7fbfb80708b0 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbe55e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80cd5c0 con 0x7fbfb815ac00 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbe55e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80cf5d0 con 0x7fbfb8070d10 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 0x7fbfb81449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 0x7fbfb81449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57076/0 (socket says 172.21.15.122:57076) 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 -- 172.21.15.122:0/1736185306 learned_addr learned my addr 172.21.15.122:0/1736185306 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfbcafb640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfb815ac00 0x7fbfb8145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb77fe640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfb8070d10 0x7fbfb81f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 -- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfb8070d10 msgr2=0x7fbfb81f0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfb8070d10 0x7fbfb81f0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 -- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfb815ac00 msgr2=0x7fbfb8145410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.233+0000 7fbfb7fff640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfb815ac00 0x7fbfb8145410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfb7fff640 1 -- 172.21.15.122:0/1736185306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8144f00 con 0x7fbfb80708b0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfbcafb640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfb815ac00 0x7fbfb8145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfb77fe640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfb8070d10 0x7fbfb81f0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfb7fff640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 0x7fbfb81449c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b3e0 tx=0x7fbfa800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8015020 con 0x7fbfb80708b0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8011850 con 0x7fbfb80708b0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb8142320 con 0x7fbfb80708b0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8011c40 con 0x7fbfb80708b0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.234+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb8142b60 con 0x7fbfb80708b0 2024-03-06T07:25:03.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.235+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb804f330 con 0x7fbfb80708b0 2024-03-06T07:25:03.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.235+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfa8018020 con 0x7fbfb80708b0 2024-03-06T07:25:03.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.236+0000 7fbfb4ff9640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfa406b4d0 0x7fbfa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.236+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbfa8089de0 con 0x7fbfb80708b0 2024-03-06T07:25:03.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.236+0000 7fbfb77fe640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfa406b4d0 0x7fbfa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.236+0000 7fbfb77fe640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfa406b4d0 0x7fbfa406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8153860 tx=0x7fbfac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.240+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfa801f020 con 0x7fbfb80708b0 2024-03-06T07:25:03.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.411+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbfb8148250 con 0x7fbfb80708b0 2024-03-06T07:25:03.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.846+0000 7fbfb4ff9640 1 -- 172.21.15.122:0/1736185306 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbfb8142320 con 0x7fbfb80708b0 2024-03-06T07:25:03.848 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.848+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfa406b4d0 msgr2=0x7fbfa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.848+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfa406b4d0 0x7fbfa406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8153860 tx=0x7fbfac009040 comp rx=0 tx=0).stop 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.848+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 msgr2=0x7fbfb81449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.848+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 0x7fbfb81449c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b3e0 tx=0x7fbfa800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 shutdown_connections 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfa406b4d0 0x7fbfa406d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfb815ac00 0x7fbfb8145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfb8070d10 0x7fbfb81f0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 --2- 172.21.15.122:0/1736185306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfb80708b0 0x7fbfb81449c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 >> 172.21.15.122:0/1736185306 conn(0x7fbfb80b1650 msgr2=0x7fbfb80b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 shutdown_connections 2024-03-06T07:25:03.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.849+0000 7fbfbe55e640 1 -- 172.21.15.122:0/1736185306 wait complete. 2024-03-06T07:25:03.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:03.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.972+0000 7f5bc94c9640 1 Processor -- start 2024-03-06T07:25:03.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.972+0000 7f5bc94c9640 1 -- start start 2024-03-06T07:25:03.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc94c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 0x7f5bc40b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc94c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5bc40cea50 0x7f5bc40ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc94c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc406d030 con 0x7f5bc406cab0 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc94c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc40cf1c0 con 0x7f5bc40cea50 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc94c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40cc760 con 0x7f5bc415def0 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc37fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5bc40cea50 0x7f5bc40ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc2ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 0x7f5bc40b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc27fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5bc406cab0 0x7f5bc40ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:43502/0 (socket says 172.21.15.122:43502) 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc2ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 0x7f5bc40b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:44796/0 (socket says 172.21.15.122:44796) 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc37fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5bc40cea50 0x7f5bc40ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:53642/0 (socket says 172.21.15.122:53642) 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc27fc640 1 -- 172.21.15.122:0/2966195869 learned_addr learned my addr 172.21.15.122:0/2966195869 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc2ffd640 1 -- 172.21.15.122:0/2966195869 >> v2:172.21.15.169:3300/0 conn(0x7f5bc40cea50 msgr2=0x7f5bc40ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc2ffd640 1 --2- 172.21.15.122:0/2966195869 >> v2:172.21.15.169:3300/0 conn(0x7f5bc40cea50 0x7f5bc40ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc2ffd640 1 -- 172.21.15.122:0/2966195869 >> v1:172.21.15.169:6789/0 conn(0x7f5bc406cab0 legacy=0x7f5bc40ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.973+0000 7f5bc2ffd640 1 -- 172.21.15.122:0/2966195869 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc406d310 con 0x7f5bc415def0 2024-03-06T07:25:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.974+0000 7f5bc2ffd640 1 --2- 172.21.15.122:0/2966195869 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 0x7f5bc40b41b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40097a0 tx=0x7f5bb402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8360616bcbb87de0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc1ffb640 1 -- 172.21.15.122:0/2966195869 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4034070 con 0x7f5bc415def0 2024-03-06T07:25:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc1ffb640 1 -- 172.21.15.122:0/2966195869 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb4030610 con 0x7f5bc415def0 2024-03-06T07:25:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc1ffb640 1 -- 172.21.15.122:0/2966195869 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40309e0 con 0x7f5bc415def0 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 -- 172.21.15.122:0/2966195869 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 msgr2=0x7f5bc40b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/2966195869 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 0x7f5bc40b41b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40097a0 tx=0x7f5bb402b960 comp rx=0 tx=0).stop 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 -- 172.21.15.122:0/2966195869 shutdown_connections 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/2966195869 >> v2:172.21.15.169:3300/0 conn(0x7f5bc40cea50 0x7f5bc40ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/2966195869 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc415def0 0x7f5bc40b41b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 -- 172.21.15.122:0/2966195869 >> 172.21.15.122:0/2966195869 conn(0x7f5bc40b3690 msgr2=0x7f5bc40b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 -- 172.21.15.122:0/2966195869 shutdown_connections 2024-03-06T07:25:03.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.975+0000 7f5bc94c9640 1 -- 172.21.15.122:0/2966195869 wait complete. 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.976+0000 7f5bc94c9640 1 Processor -- start 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.976+0000 7f5bc94c9640 1 -- start start 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc94c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc406cab0 0x7f5bc409f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc94c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 0x7f5bc41f8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc94c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bc415def0 0x7f5bc40c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc94c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc415e2d0 con 0x7f5bc40cea50 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc94c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40cc3f0 con 0x7f5bc415def0 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc94c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc415e450 con 0x7f5bc406cab0 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 0x7f5bc41f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 0x7f5bc41f8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57086/0 (socket says 172.21.15.122:57086) 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc27fc640 1 -- 172.21.15.122:0/1526532451 learned_addr learned my addr 172.21.15.122:0/1526532451 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc2ffd640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc406cab0 0x7f5bc409f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.977+0000 7f5bc37fe640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bc415def0 0x7f5bc40c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc27fc640 1 -- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc406cab0 msgr2=0x7f5bc409f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc27fc640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc406cab0 0x7f5bc409f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc27fc640 1 -- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bc415def0 msgr2=0x7f5bc40c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc27fc640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bc415def0 0x7f5bc40c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc27fc640 1 -- 172.21.15.122:0/1526532451 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc40c55e0 con 0x7f5bc40cea50 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc27fc640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 0x7f5bc41f8340 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c400 tx=0x7f5bac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac017070 con 0x7f5bc40cea50 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac005ce0 con 0x7f5bc40cea50 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc40c5880 con 0x7f5bc40cea50 2024-03-06T07:25:03.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0072f0 con 0x7f5bc40cea50 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.978+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc40c5e40 con 0x7f5bc40cea50 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.979+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc409fcf0 con 0x7f5bc40cea50 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.979+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bac01a070 con 0x7f5bc40cea50 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.980+0000 7f5bbb7fe640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ba406b4d0 0x7f5ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.980+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5bac0893b0 con 0x7f5bc40cea50 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.980+0000 7f5bc2ffd640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ba406b4d0 0x7f5ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.980+0000 7f5bc2ffd640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ba406b4d0 0x7f5ba406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4036000 tx=0x7f5bb403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:03.984+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bac057dc0 con 0x7f5bc40cea50 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.156+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5bc4068ff0 con 0x7f5bc40cea50 2024-03-06T07:25:04.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.847+0000 7f5bbb7fe640 1 -- 172.21.15.122:0/1526532451 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5bac05be60 con 0x7f5bc40cea50 2024-03-06T07:25:04.850 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:25:04.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.850+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ba406b4d0 msgr2=0x7f5ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.850+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ba406b4d0 0x7f5ba406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4036000 tx=0x7f5bb403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 msgr2=0x7f5bc41f8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 0x7f5bc41f8340 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c400 tx=0x7f5bac00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 shutdown_connections 2024-03-06T07:25:04.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ba406b4d0 0x7f5ba406d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bc415def0 0x7f5bc40c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bc40cea50 0x7f5bc41f8340 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 --2- 172.21.15.122:0/1526532451 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bc406cab0 0x7f5bc409f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 >> 172.21.15.122:0/1526532451 conn(0x7f5bc40b3690 msgr2=0x7f5bc415cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 shutdown_connections 2024-03-06T07:25:04.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.851+0000 7f5bc94c9640 1 -- 172.21.15.122:0/1526532451 wait complete. 2024-03-06T07:25:04.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.978+0000 7f45cb037640 1 Processor -- start 2024-03-06T07:25:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.978+0000 7f45cb037640 1 -- start start 2024-03-06T07:25:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45cb037640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45cb037640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45c40d2a90 0x7f45c40cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45cb037640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c40b4290 con 0x7f45c40b3df0 2024-03-06T07:25:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45cb037640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c40ce990 con 0x7f45c40d2a90 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45cb037640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c40b4570 con 0x7f45c415e0b0 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45c95d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45c40d2a90 0x7f45c40cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45c95d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45c40d2a90 0x7f45c40cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47788/0 (socket says 172.21.15.122:47788) 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.979+0000 7f45c95d4640 1 -- 172.21.15.122:0/4015584430 learned_addr learned my addr 172.21.15.122:0/4015584430 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.980+0000 7f45c8dd3640 1 --2- 172.21.15.122:0/4015584430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.980+0000 7f45c95d4640 1 -- 172.21.15.122:0/4015584430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 msgr2=0x7f45c40b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.980+0000 7f45c95d4640 1 --2- 172.21.15.122:0/4015584430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.980+0000 7f45c95d4640 1 -- 172.21.15.122:0/4015584430 >> v1:172.21.15.169:6789/0 conn(0x7f45c40b3df0 legacy=0x7f45c40d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.980+0000 7f45c95d4640 1 -- 172.21.15.122:0/4015584430 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c40b4850 con 0x7f45c40d2a90 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.980+0000 7f45c95d4640 1 --2- 172.21.15.122:0/4015584430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c40cee00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009870 tx=0x7f45b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a4baf62f4a41e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45c37fe640 1 -- 172.21.15.122:0/4015584430 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b4034070 con 0x7f45c40d2a90 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45c37fe640 1 -- 172.21.15.122:0/4015584430 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b4030840 con 0x7f45c40d2a90 2024-03-06T07:25:04.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45c37fe640 1 -- 172.21.15.122:0/4015584430 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b4030c10 con 0x7f45c40d2a90 2024-03-06T07:25:04.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 -- 172.21.15.122:0/4015584430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 msgr2=0x7f45c40cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 --2- 172.21.15.122:0/4015584430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c40cee00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009870 tx=0x7f45b402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 -- 172.21.15.122:0/4015584430 shutdown_connections 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 --2- 172.21.15.122:0/4015584430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c40cee00 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 --2- 172.21.15.122:0/4015584430 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 -- 172.21.15.122:0/4015584430 >> 172.21.15.122:0/4015584430 conn(0x7f45c408d6f0 msgr2=0x7f45c409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 -- 172.21.15.122:0/4015584430 shutdown_connections 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.981+0000 7f45cb037640 1 -- 172.21.15.122:0/4015584430 wait complete. 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.982+0000 7f45cb037640 1 Processor -- start 2024-03-06T07:25:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.982+0000 7f45cb037640 1 -- start start 2024-03-06T07:25:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45cb037640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45c40b3df0 0x7f45c41610f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45cb037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c415a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45cb037640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45cb037640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c40d1cd0 con 0x7f45c40d2a90 2024-03-06T07:25:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45cb037640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c415e490 con 0x7f45c40b3df0 2024-03-06T07:25:04.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45cb037640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c40d1e50 con 0x7f45c415e0b0 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45c3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45c3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c415a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47794/0 (socket says 172.21.15.122:47794) 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45c3fff640 1 -- 172.21.15.122:0/1545866764 learned_addr learned my addr 172.21.15.122:0/1545866764 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45c8dd3640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45c40b3df0 0x7f45c41610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.983+0000 7f45c95d4640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c3fff640 1 -- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 msgr2=0x7f45c40cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c3fff640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c3fff640 1 -- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45c40b3df0 msgr2=0x7f45c41610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c3fff640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45c40b3df0 0x7f45c41610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c3fff640 1 -- 172.21.15.122:0/1545866764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c41f7e60 con 0x7f45c40d2a90 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c3fff640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c415a170 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00c400 tx=0x7f45ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0052d0 con 0x7f45c40d2a90 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac005490 con 0x7f45c40d2a90 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c415e950 con 0x7f45c40d2a90 2024-03-06T07:25:04.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac007260 con 0x7f45c40d2a90 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.984+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c415ebf0 con 0x7f45c40d2a90 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.985+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c404f330 con 0x7f45c40d2a90 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.985+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45ac007400 con 0x7f45c40d2a90 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.986+0000 7f45c17fa640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f45a406b4d0 0x7f45a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.986+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f45ac0895c0 con 0x7f45c40d2a90 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.986+0000 7f45c8dd3640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f45a406b4d0 0x7f45a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.986+0000 7f45c8dd3640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f45a406b4d0 0x7f45a406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f45c40b9630 tx=0x7f45bc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:04.990+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45ac057ee0 con 0x7f45c40d2a90 2024-03-06T07:25:05.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.199+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f45c40cd650 con 0x7f45c40d2a90 2024-03-06T07:25:05.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.869+0000 7f45c17fa640 1 -- 172.21.15.122:0/1545866764 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v31) v1 ==== 174+0+0 (secure 0 0 0) 0x7f45ac05bf80 con 0x7f45c40d2a90 2024-03-06T07:25:05.871 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:25:05.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.876+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f45a406b4d0 msgr2=0x7f45a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.876+0000 7f45cb037640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f45a406b4d0 0x7f45a406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f45c40b9630 tx=0x7f45bc00b040 comp rx=0 tx=0).stop 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.876+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 msgr2=0x7f45c415a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.876+0000 7f45cb037640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c415a170 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00c400 tx=0x7f45ac00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 shutdown_connections 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f45a406b4d0 0x7f45a406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45c415e0b0 0x7f45c40cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45c40d2a90 0x7f45c415a170 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 --2- 172.21.15.122:0/1545866764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45c40b3df0 0x7f45c41610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 >> 172.21.15.122:0/1545866764 conn(0x7f45c408d6f0 msgr2=0x7f45c40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 shutdown_connections 2024-03-06T07:25:05.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.877+0000 7f45cb037640 1 -- 172.21.15.122:0/1545866764 wait complete. 2024-03-06T07:25:05.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:05.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.996+0000 7f0a041bc640 1 Processor -- start 2024-03-06T07:25:05.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.997+0000 7f0a041bc640 1 -- start start 2024-03-06T07:25:06.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.997+0000 7f0a041bc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09fc0d22b0 0x7f09fc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.997+0000 7f0a041bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.997+0000 7f0a041bc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09fc0b3430 con 0x7f09fc15dfe0 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.997+0000 7f0a041bc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc0d2c50 con 0x7f09fc0d22b0 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.997+0000 7f0a041bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0b31e0 con 0x7f09fc0b41f0 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09fc0d22b0 0x7f09fc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09fc0d22b0 0x7f09fc0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47800/0 (socket says 172.21.15.122:47800) 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 -- 172.21.15.122:0/1113161431 learned_addr learned my addr 172.21.15.122:0/1113161431 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a02759640 1 --2- 172.21.15.122:0/1113161431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 -- 172.21.15.122:0/1113161431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 msgr2=0x7f09fc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 --2- 172.21.15.122:0/1113161431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 -- 172.21.15.122:0/1113161431 >> v1:172.21.15.169:6789/0 conn(0x7f09fc15dfe0 legacy=0x7f09fc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.998+0000 7f0a01757640 1 -- 172.21.15.122:0/1113161431 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0b3710 con 0x7f09fc0d22b0 2024-03-06T07:25:06.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a01757640 1 --2- 172.21.15.122:0/1113161431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc0d22b0 0x7f09fc0d2710 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009e50 tx=0x7f09f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9311414457968a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a00f56640 1 -- 172.21.15.122:0/1113161431 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f002cb00 con 0x7f09fc0d22b0 2024-03-06T07:25:06.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a00f56640 1 -- 172.21.15.122:0/1113161431 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f002cca0 con 0x7f09fc0d22b0 2024-03-06T07:25:06.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a00f56640 1 -- 172.21.15.122:0/1113161431 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0034610 con 0x7f09fc0d22b0 2024-03-06T07:25:06.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 -- 172.21.15.122:0/1113161431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc0d22b0 msgr2=0x7f09fc0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 --2- 172.21.15.122:0/1113161431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc0d22b0 0x7f09fc0d2710 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009e50 tx=0x7f09f002bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 -- 172.21.15.122:0/1113161431 shutdown_connections 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 --2- 172.21.15.122:0/1113161431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 --2- 172.21.15.122:0/1113161431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc0d22b0 0x7f09fc0d2710 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 -- 172.21.15.122:0/1113161431 >> 172.21.15.122:0/1113161431 conn(0x7f09fc0d3cd0 msgr2=0x7f09fc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 -- 172.21.15.122:0/1113161431 shutdown_connections 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:05.999+0000 7f0a041bc640 1 -- 172.21.15.122:0/1113161431 wait complete. 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.000+0000 7f0a041bc640 1 Processor -- start 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.000+0000 7f0a041bc640 1 -- start start 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a041bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a041bc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09fc0d22b0 0x7f09fc1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a041bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 0x7f09fc1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a041bc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0cfa00 con 0x7f09fc15dfe0 2024-03-06T07:25:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a041bc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0b3d60 con 0x7f09fc0d22b0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a041bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0d1c70 con 0x7f09fc0b41f0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a02759640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 0x7f09fc1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a02759640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 0x7f09fc1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47806/0 (socket says 172.21.15.122:47806) 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a02759640 1 -- 172.21.15.122:0/412762071 learned_addr learned my addr 172.21.15.122:0/412762071 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a01757640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09fc0d22b0 0x7f09fc1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.001+0000 7f0a01f58640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a02759640 1 -- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 msgr2=0x7f09fc0b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a02759640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a02759640 1 -- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09fc0d22b0 msgr2=0x7f09fc1f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a02759640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09fc0d22b0 0x7f09fc1f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a02759640 1 -- 172.21.15.122:0/412762071 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0b9490 con 0x7f09fc15dfe0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a02759640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 0x7f09fc1f3e40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00d830 tx=0x7f09ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00b6c0 con 0x7f09fc15dfe0 2024-03-06T07:25:06.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec019070 con 0x7f09fc15dfe0 2024-03-06T07:25:06.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00bac0 con 0x7f09fc15dfe0 2024-03-06T07:25:06.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc0b9770 con 0x7f09fc15dfe0 2024-03-06T07:25:06.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.002+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc0b1130 con 0x7f09fc15dfe0 2024-03-06T07:25:06.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.003+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc04f330 con 0x7f09fc15dfe0 2024-03-06T07:25:06.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.003+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09ec01e460 con 0x7f09fc15dfe0 2024-03-06T07:25:06.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.004+0000 7f09fa7fc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09e806b4d0 0x7f09e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.004+0000 7f0a01f58640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09e806b4d0 0x7f09e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.004+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09ec08a6a0 con 0x7f09fc15dfe0 2024-03-06T07:25:06.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.004+0000 7f0a01f58640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09e806b4d0 0x7f09e806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f09fc15e9e0 tx=0x7f09f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.008+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09ec058f60 con 0x7f09fc15dfe0 2024-03-06T07:25:06.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.180+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09fc0b2f40 con 0x7f09fc15dfe0 2024-03-06T07:25:06.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.181+0000 7f09fa7fc640 1 -- 172.21.15.122:0/412762071 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f09ec05d000 con 0x7f09fc15dfe0 2024-03-06T07:25:06.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:06.184 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":68,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:05.200878+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09e806b4d0 msgr2=0x7f09e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09e806b4d0 0x7f09e806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f09fc15e9e0 tx=0x7f09f4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 msgr2=0x7f09fc1f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 0x7f09fc1f3e40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00d830 tx=0x7f09ec00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 shutdown_connections 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09e806b4d0 0x7f09e806d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09fc15dfe0 0x7f09fc1f3e40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09fc0d22b0 0x7f09fc1f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 --2- 172.21.15.122:0/412762071 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09fc0b41f0 0x7f09fc0b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 >> 172.21.15.122:0/412762071 conn(0x7f09fc0d3cd0 msgr2=0x7f09fc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 shutdown_connections 2024-03-06T07:25:06.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.184+0000 7f0a041bc640 1 -- 172.21.15.122:0/412762071 wait complete. 2024-03-06T07:25:06.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:06.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.294+0000 7f12e9be2640 1 Processor -- start 2024-03-06T07:25:06.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.294+0000 7f12e9be2640 1 -- start start 2024-03-06T07:25:06.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.295+0000 7f12e9be2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12e415dfe0 0x7f12e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.295+0000 7f12e9be2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e40b3430 con 0x7f12e40d29c0 2024-03-06T07:25:06.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.295+0000 7f12e9be2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e40b3710 con 0x7f12e40cf830 2024-03-06T07:25:06.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.295+0000 7f12e9be2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e40b31e0 con 0x7f12e415dfe0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.295+0000 7f12e3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f12e40d29c0 0x7f12e40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36508/0 (socket says 172.21.15.122:36508) 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.295+0000 7f12e3fff640 1 -- 172.21.15.122:0/3470821539 learned_addr learned my addr 172.21.15.122:0/3470821539 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 --2- 172.21.15.122:0/3470821539 >> v2:172.21.15.169:3300/0 conn(0x7f12e415dfe0 0x7f12e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315425019 0 0) 0x7f12e40b3430 con 0x7f12e40d29c0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d0003520 con 0x7f12e40d29c0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 852323067 0 0) 0x7f12d0003520 con 0x7f12e40d29c0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 >> v2:172.21.15.169:3300/0 conn(0x7f12e415dfe0 msgr2=0x7f12e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 --2- 172.21.15.122:0/3470821539 >> v2:172.21.15.169:3300/0 conn(0x7f12e415dfe0 0x7f12e40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 >> v1:172.21.15.169:6789/0 conn(0x7f12e40cf830 legacy=0x7f12e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e40b39f0 con 0x7f12e40d29c0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f12dc0019f0 con 0x7f12e40d29c0 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 >> v1:172.21.15.122:6789/0 conn(0x7f12e40d29c0 legacy=0x7f12e40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d00039b0 0x7f12d0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d0004800 0x7f12d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00051c0 0x7f12d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40b31e0 con 0x7f12d00039b0 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d000c090 con 0x7f12d00051c0 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d000c290 con 0x7f12d0004800 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d00039b0 0x7f12d0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12daffd640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d0004800 0x7f12d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e3fff640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00051c0 0x7f12d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 -- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d0004800 msgr2=0x7f12d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d0004800 0x7f12d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 -- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00051c0 msgr2=0x7f12d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00051c0 0x7f12d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e37fe640 1 -- 172.21.15.122:0/3470821539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d0003520 con 0x7f12d00039b0 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12e3fff640 1 -- 172.21.15.122:0/3470821539 reap_dead start 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.296+0000 7f12daffd640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d0004800 0x7f12d0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.297+0000 7f12e37fe640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d00039b0 0x7f12d0003df0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ac0 tx=0x7f12d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a98d3ab04b1ead server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.297+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4039070 con 0x7f12d00039b0 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.297+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d402c6b0 con 0x7f12d00039b0 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.297+0000 7f12e2ffd640 1 -- 172.21.15.122:0/3470821539 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4009690 con 0x7f12d00039b0 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 -- 172.21.15.122:0/3470821539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d00039b0 msgr2=0x7f12d0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d00039b0 0x7f12d0003df0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ac0 tx=0x7f12d0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 -- 172.21.15.122:0/3470821539 shutdown_connections 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 --2- 172.21.15.122:0/3470821539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d00039b0 0x7f12d0003df0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 -- 172.21.15.122:0/3470821539 >> 172.21.15.122:0/3470821539 conn(0x7f12e40d3cd0 msgr2=0x7f12e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 -- 172.21.15.122:0/3470821539 shutdown_connections 2024-03-06T07:25:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.298+0000 7f12e9be2640 1 -- 172.21.15.122:0/3470821539 wait complete. 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.299+0000 7f12e9be2640 1 Processor -- start 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.299+0000 7f12e9be2640 1 -- start start 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e9be2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00039b0 0x7f12e41f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e9be2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12e41f4580 0x7f12e41f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e9be2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 0x7f12e40cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e9be2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e415e3c0 con 0x7f12e40cb7b0 2024-03-06T07:25:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e9be2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40b3d60 con 0x7f12d00039b0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e9be2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e415e540 con 0x7f12e41f4580 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 0x7f12e40cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 0x7f12e40cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47828/0 (socket says 172.21.15.122:47828) 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 -- 172.21.15.122:0/4178783148 learned_addr learned my addr 172.21.15.122:0/4178783148 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12daffd640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12e41f4580 0x7f12e41f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e37fe640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00039b0 0x7f12e41f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 -- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12e41f4580 msgr2=0x7f12e41f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12e41f4580 0x7f12e41f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 -- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00039b0 msgr2=0x7f12e41f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00039b0 0x7f12e41f4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 -- 172.21.15.122:0/4178783148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e40b6970 con 0x7f12e40cb7b0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e3fff640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 0x7f12e40cc630 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f12e4161bf0 tx=0x7f12dc007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.300+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00eaa0 con 0x7f12e40cb7b0 2024-03-06T07:25:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.301+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc018070 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.301+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e40b6c50 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.301+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00ee80 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.301+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e40b7210 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.302+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e404f330 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.302+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12dc01d460 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.302+0000 7f12e0ff9640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12c406b500 0x7f12c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.303+0000 7f12e37fe640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12c406b500 0x7f12c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.303+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12dc08a4f0 con 0x7f12e40cb7b0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.303+0000 7f12e37fe640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12c406b500 0x7f12c406d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ac0 tx=0x7f12d4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.307+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12dc059370 con 0x7f12e40cb7b0 2024-03-06T07:25:06.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.478+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12e406c450 con 0x7f12e40cb7b0 2024-03-06T07:25:06.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.479+0000 7f12e0ff9640 1 -- 172.21.15.122:0/4178783148 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f12dc058c70 con 0x7f12e40cb7b0 2024-03-06T07:25:06.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:06.482 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":68,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:05.200878+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12c406b500 msgr2=0x7f12c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12c406b500 0x7f12c406d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ac0 tx=0x7f12d4030000 comp rx=0 tx=0).stop 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 msgr2=0x7f12e40cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 0x7f12e40cc630 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f12e4161bf0 tx=0x7f12dc007bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 shutdown_connections 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12c406b500 0x7f12c406d9c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12e40cb7b0 0x7f12e40cc630 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12e41f4580 0x7f12e41f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 --2- 172.21.15.122:0/4178783148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00039b0 0x7f12e41f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 >> 172.21.15.122:0/4178783148 conn(0x7f12e40d3cd0 msgr2=0x7f12e40b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 shutdown_connections 2024-03-06T07:25:06.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.481+0000 7f12e9be2640 1 -- 172.21.15.122:0/4178783148 wait complete. 2024-03-06T07:25:06.494 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:06.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.596+0000 7f12d70d4640 1 Processor -- start 2024-03-06T07:25:06.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.596+0000 7f12d70d4640 1 -- start start 2024-03-06T07:25:06.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d70d4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 0x7f12d0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d70d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00b0db0 0x7f12d00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d70d4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d006b050 con 0x7f12d00ce580 2024-03-06T07:25:06.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d70d4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d006b2c0 con 0x7f12d015f4f0 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d70d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d00b0890 con 0x7f12d00b0db0 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00b0db0 0x7f12d00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d4e70640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 0x7f12d0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d5671640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f12d00ce580 0x7f12d0072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:48344/0 (socket says 172.21.15.122:48344) 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d4e70640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 0x7f12d0066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:42396/0 (socket says 172.21.15.122:42396) 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d5671640 1 -- 172.21.15.122:0/1970604819 learned_addr learned my addr 172.21.15.122:0/1970604819 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cf7fe640 1 -- 172.21.15.122:0/1970604819 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771193327 0 0) 0x7f12d006b050 con 0x7f12d00ce580 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cf7fe640 1 -- 172.21.15.122:0/1970604819 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0003520 con 0x7f12d00ce580 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cffff640 1 -- 172.21.15.122:0/1970604819 >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 msgr2=0x7f12d0066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cffff640 1 --2- 172.21.15.122:0/1970604819 >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 0x7f12d0066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cffff640 1 -- 172.21.15.122:0/1970604819 >> v1:172.21.15.169:6789/0 conn(0x7f12d00ce580 legacy=0x7f12d0072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12cffff640 1 -- 172.21.15.122:0/1970604819 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d00672e0 con 0x7f12d00b0db0 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.597+0000 7f12d4e70640 1 --2- 172.21.15.122:0/1970604819 >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 0x7f12d0066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.598+0000 7f12cffff640 1 --2- 172.21.15.122:0/1970604819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00b0db0 0x7f12d00ce040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0097a0 tx=0x7f12bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8992594027fbe528 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.598+0000 7f12cf7fe640 1 -- 172.21.15.122:0/1970604819 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc035020 con 0x7f12d00b0db0 2024-03-06T07:25:06.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.598+0000 7f12cf7fe640 1 -- 172.21.15.122:0/1970604819 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc031830 con 0x7f12d00b0db0 2024-03-06T07:25:06.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.598+0000 7f12cf7fe640 1 -- 172.21.15.122:0/1970604819 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc031be0 con 0x7f12d00b0db0 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 -- 172.21.15.122:0/1970604819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00b0db0 msgr2=0x7f12d00ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 --2- 172.21.15.122:0/1970604819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00b0db0 0x7f12d00ce040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0097a0 tx=0x7f12bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 -- 172.21.15.122:0/1970604819 shutdown_connections 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 --2- 172.21.15.122:0/1970604819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00b0db0 0x7f12d00ce040 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 --2- 172.21.15.122:0/1970604819 >> v2:172.21.15.169:3300/0 conn(0x7f12d015f4f0 0x7f12d0066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 -- 172.21.15.122:0/1970604819 >> 172.21.15.122:0/1970604819 conn(0x7f12d00b1d70 msgr2=0x7f12d00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 -- 172.21.15.122:0/1970604819 shutdown_connections 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.599+0000 7f12d70d4640 1 -- 172.21.15.122:0/1970604819 wait complete. 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.600+0000 7f12d70d4640 1 Processor -- start 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.600+0000 7f12d70d4640 1 -- start start 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d70d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00b0db0 0x7f12d0149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d70d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00ce580 0x7f12d00caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d70d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 0x7f12d0159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d70d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d015f8d0 con 0x7f12d015f4f0 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d70d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d006b5a0 con 0x7f12d00b0db0 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d70d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d00cda20 con 0x7f12d00ce580 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 0x7f12d0159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 0x7f12d0159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47838/0 (socket says 172.21.15.122:47838) 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 -- 172.21.15.122:0/3582796207 learned_addr learned my addr 172.21.15.122:0/3582796207 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12cffff640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00ce580 0x7f12d00caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d4e70640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00b0db0 0x7f12d0149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 -- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00ce580 msgr2=0x7f12d00caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00ce580 0x7f12d00caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 -- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00b0db0 msgr2=0x7f12d0149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00b0db0 0x7f12d0149060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d5671640 1 -- 172.21.15.122:0/3582796207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d01560e0 con 0x7f12d015f4f0 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.601+0000 7f12d4e70640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00b0db0 0x7f12d0149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.602+0000 7f12d5671640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 0x7f12d0159430 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f12c800cb70 tx=0x7f12c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.602+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8017070 con 0x7f12d015f4f0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.602+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c800ad40 con 0x7f12d015f4f0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.602+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d00ad180 con 0x7f12d015f4f0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.602+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8012620 con 0x7f12d015f4f0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.602+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d00ad420 con 0x7f12d015f4f0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.603+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d009e3d0 con 0x7f12d015f4f0 2024-03-06T07:25:06.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.603+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12c801a070 con 0x7f12d015f4f0 2024-03-06T07:25:06.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.603+0000 7f12cd7fa640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12b406b4d0 0x7f12b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.604+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12c808a070 con 0x7f12d015f4f0 2024-03-06T07:25:06.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.604+0000 7f12d4e70640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12b406b4d0 0x7f12b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.604+0000 7f12d4e70640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12b406b4d0 0x7f12b406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009a20 tx=0x7f12c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.608+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12c8058f00 con 0x7f12d015f4f0 2024-03-06T07:25:06.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.817+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f12d006cd40 con 0x7f12d015f4f0 2024-03-06T07:25:06.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.872+0000 7f12cd7fa640 1 -- 172.21.15.122:0/3582796207 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f12c8058800 con 0x7f12d015f4f0 2024-03-06T07:25:06.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12b406b4d0 msgr2=0x7f12b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12b406b4d0 0x7f12b406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009a20 tx=0x7f12c4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 msgr2=0x7f12d0159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 0x7f12d0159430 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f12c800cb70 tx=0x7f12c800a430 comp rx=0 tx=0).stop 2024-03-06T07:25:06.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 shutdown_connections 2024-03-06T07:25:06.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f12b406b4d0 0x7f12b406d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12d015f4f0 0x7f12d0159430 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12d00ce580 0x7f12d00caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 --2- 172.21.15.122:0/3582796207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12d00b0db0 0x7f12d0149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 >> 172.21.15.122:0/3582796207 conn(0x7f12d00b1d70 msgr2=0x7f12d0072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 shutdown_connections 2024-03-06T07:25:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.875+0000 7f12d70d4640 1 -- 172.21.15.122:0/3582796207 wait complete. 2024-03-06T07:25:06.888 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:06.889 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:06.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.991+0000 7fd0b253e640 1 Processor -- start 2024-03-06T07:25:06.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.991+0000 7fd0b253e640 1 -- start start 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.991+0000 7fd0b253e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd0ac15c050 0x7fd0ac06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.991+0000 7fd0b253e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac0cd4b0 con 0x7fd0ac06fb40 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.991+0000 7fd0b253e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac0cd790 con 0x7fd0ac0d0af0 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.991+0000 7fd0b253e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ac06ffb0 con 0x7fd0ac15c050 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0abfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd0ac15c050 0x7fd0ac06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0ab7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd0ac06fb40 0x7fd0ac0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36522/0 (socket says 172.21.15.122:36522) 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0ab7fe640 1 -- 172.21.15.122:0/2961052829 learned_addr learned my addr 172.21.15.122:0/2961052829 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0aaffd640 1 -- 172.21.15.122:0/2961052829 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485670966 0 0) 0x7fd0ac0cd4b0 con 0x7fd0ac06fb40 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0aaffd640 1 -- 172.21.15.122:0/2961052829 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd094003520 con 0x7fd0ac06fb40 2024-03-06T07:25:06.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0abfff640 1 -- 172.21.15.122:0/2961052829 >> v1:172.21.15.169:6789/0 conn(0x7fd0ac0d0af0 legacy=0x7fd0ac165ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0abfff640 1 -- 172.21.15.122:0/2961052829 >> v1:172.21.15.122:6789/0 conn(0x7fd0ac06fb40 legacy=0x7fd0ac0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.992+0000 7fd0abfff640 1 -- 172.21.15.122:0/2961052829 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac0cda70 con 0x7fd0ac15c050 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0abfff640 1 --2- 172.21.15.122:0/2961052829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac15c050 0x7fd0ac06f600 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0097a0 tx=0x7fd09c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a0c2f2e23e21f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0aaffd640 1 -- 172.21.15.122:0/2961052829 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c035020 con 0x7fd0ac15c050 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0aaffd640 1 -- 172.21.15.122:0/2961052829 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c031830 con 0x7fd0ac15c050 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0aaffd640 1 -- 172.21.15.122:0/2961052829 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c031be0 con 0x7fd0ac15c050 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0b253e640 1 -- 172.21.15.122:0/2961052829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac15c050 msgr2=0x7fd0ac06f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2961052829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac15c050 0x7fd0ac06f600 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0097a0 tx=0x7fd09c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0b253e640 1 -- 172.21.15.122:0/2961052829 shutdown_connections 2024-03-06T07:25:06.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2961052829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac15c050 0x7fd0ac06f600 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0b253e640 1 -- 172.21.15.122:0/2961052829 >> 172.21.15.122:0/2961052829 conn(0x7fd0ac0b3800 msgr2=0x7fd0ac0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.993+0000 7fd0b253e640 1 -- 172.21.15.122:0/2961052829 shutdown_connections 2024-03-06T07:25:06.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.994+0000 7fd0b253e640 1 -- 172.21.15.122:0/2961052829 wait complete. 2024-03-06T07:25:06.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.995+0000 7fd0b253e640 1 Processor -- start 2024-03-06T07:25:06.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.995+0000 7fd0b253e640 1 -- start start 2024-03-06T07:25:06.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 0x7fd0ac1f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0ac0d0af0 0x7fd0ac0b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0ac15c050 0x7fd0ac0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac0ccff0 con 0x7fd0ac06fb40 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac0cce70 con 0x7fd0ac15c050 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac0cd170 con 0x7fd0ac0d0af0 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 0x7fd0ac1f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 0x7fd0ac1f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47844/0 (socket says 172.21.15.122:47844) 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 -- 172.21.15.122:0/2919348744 learned_addr learned my addr 172.21.15.122:0/2919348744 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 -- 172.21.15.122:0/2919348744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0ac0d0af0 msgr2=0x7fd0ac0b7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0ac0d0af0 0x7fd0ac0b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 -- 172.21.15.122:0/2919348744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0ac15c050 msgr2=0x7fd0ac0b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0ac15c050 0x7fd0ac0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 -- 172.21.15.122:0/2919348744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac1f1e00 con 0x7fd0ac06fb40 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0abfff640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 0x7fd0ac1f2580 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd09c000c00 tx=0x7fd09c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c035070 con 0x7fd0ac06fb40 2024-03-06T07:25:07.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c004230 con 0x7fd0ac06fb40 2024-03-06T07:25:07.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac0b7960 con 0x7fd0ac06fb40 2024-03-06T07:25:07.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c03a630 con 0x7fd0ac06fb40 2024-03-06T07:25:07.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.997+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ac0b6940 con 0x7fd0ac06fb40 2024-03-06T07:25:07.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.999+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ac09fe60 con 0x7fd0ac06fb40 2024-03-06T07:25:07.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.999+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd09c038020 con 0x7fd0ac06fb40 2024-03-06T07:25:07.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.999+0000 7fd0a8ff9640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd09806b4d0 0x7fd09806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.999+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd09c0abe80 con 0x7fd0ac06fb40 2024-03-06T07:25:07.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:06.999+0000 7fd0ab7fe640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd09806b4d0 0x7fd09806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.000+0000 7fd0ab7fe640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd09806b4d0 0x7fd09806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac09fe00 tx=0x7fd0a0007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.004+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0ac0b7960 con 0x7fd0ac06fb40 2024-03-06T07:25:07.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.222+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd0ac068ff0 con 0x7fd0ac06fb40 2024-03-06T07:25:07.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.878+0000 7fd0a8ff9640 1 -- 172.21.15.122:0/2919348744 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd0ac1f1e00 con 0x7fd0ac06fb40 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd09806b4d0 msgr2=0x7fd09806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd09806b4d0 0x7fd09806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac09fe00 tx=0x7fd0a0007930 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 msgr2=0x7fd0ac1f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 0x7fd0ac1f2580 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd09c000c00 tx=0x7fd09c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 shutdown_connections 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd09806b4d0 0x7fd09806d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0ac15c050 0x7fd0ac0b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0ac0d0af0 0x7fd0ac0b7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 --2- 172.21.15.122:0/2919348744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0ac06fb40 0x7fd0ac1f2580 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 >> 172.21.15.122:0/2919348744 conn(0x7fd0ac0b3800 msgr2=0x7fd0ac158630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 shutdown_connections 2024-03-06T07:25:07.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.880+0000 7fd0b253e640 1 -- 172.21.15.122:0/2919348744 wait complete. 2024-03-06T07:25:07.893 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:07.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 Processor -- start 2024-03-06T07:25:07.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 -- start start 2024-03-06T07:25:07.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3fdc0b4f70 0x7f3fdc0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3fdc06c6d0 0x7f3fdc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fdc15e9f0 con 0x7f3fdc0b4f70 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fdc0cda10 con 0x7f3fdc06c6d0 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe2f1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0b5890 con 0x7f3fdc066d90 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.992+0000 7f3fe0cba640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3fdc0b4f70 0x7f3fdc0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3fdc0b4f70 0x7f3fdc0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47850/0 (socket says 172.21.15.122:47850) 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 -- 172.21.15.122:0/1902768742 learned_addr learned my addr 172.21.15.122:0/1902768742 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fdbfff640 1 --2- 172.21.15.122:0/1902768742 >> v2:172.21.15.169:3300/0 conn(0x7f3fdc06c6d0 0x7f3fdc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe14bb640 1 --2- 172.21.15.122:0/1902768742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 -- 172.21.15.122:0/1902768742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 msgr2=0x7f3fdc0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/1902768742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc0ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 -- 172.21.15.122:0/1902768742 >> v2:172.21.15.169:3300/0 conn(0x7f3fdc06c6d0 msgr2=0x7f3fdc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/1902768742 >> v2:172.21.15.169:3300/0 conn(0x7f3fdc06c6d0 0x7f3fdc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fe0cba640 1 -- 172.21.15.122:0/1902768742 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0cd450 con 0x7f3fdc0b4f70 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.993+0000 7f3fdbfff640 1 --2- 172.21.15.122:0/1902768742 >> v2:172.21.15.169:3300/0 conn(0x7f3fdc06c6d0 0x7f3fdc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/1902768742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0b5350 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0ccb00 tx=0x7f3fd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc2b682d3af2034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fdb7fe640 1 -- 172.21.15.122:0/1902768742 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd402ca70 con 0x7f3fdc0b4f70 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fdb7fe640 1 -- 172.21.15.122:0/1902768742 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd402cc10 con 0x7f3fdc0b4f70 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fdb7fe640 1 -- 172.21.15.122:0/1902768742 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40343f0 con 0x7f3fdc0b4f70 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/1902768742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 msgr2=0x7f3fdc0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/1902768742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0b5350 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0ccb00 tx=0x7f3fd402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/1902768742 shutdown_connections 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/1902768742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/1902768742 >> v2:172.21.15.169:3300/0 conn(0x7f3fdc06c6d0 0x7f3fdc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/1902768742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0b5350 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/1902768742 >> 172.21.15.122:0/1902768742 conn(0x7f3fdc0724d0 msgr2=0x7f3fdc0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/1902768742 shutdown_connections 2024-03-06T07:25:07.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.994+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/1902768742 wait complete. 2024-03-06T07:25:08.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.995+0000 7f3fe2f1e640 1 Processor -- start 2024-03-06T07:25:08.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 -- start start 2024-03-06T07:25:08.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fdc1555d0 0x7f3fdc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc15f3c0 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc15e680 con 0x7f3fdc1555d0 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.996+0000 7f3fe2f1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0cd090 con 0x7f3fdc066d90 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47860/0 (socket says 172.21.15.122:47860) 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 -- 172.21.15.122:0/3829121284 learned_addr learned my addr 172.21.15.122:0/3829121284 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fe14bb640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fdc1555d0 0x7f3fdc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 -- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 msgr2=0x7f3fdc1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 -- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fdc1555d0 msgr2=0x7f3fdc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fdc1555d0 0x7f3fdc15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 -- 172.21.15.122:0/3829121284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc159fe0 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fdbfff640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0cbb60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000d7d0 tx=0x7f3fd000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0004030 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd00041d0 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc15a2c0 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00096a0 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.997+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc15ab60 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.999+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc15a490 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.999+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fd0009840 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.999+0000 7f3fd97fa640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fc806b4d0 0x7f3fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.999+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fd008a470 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.999+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fc806b4d0 0x7f3fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:07.999+0000 7f3fe0cba640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fc806b4d0 0x7f3fc806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4035000 tx=0x7f3fd4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.003+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fd0059260 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.306+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fdc04f330 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.307+0000 7f3fd97fa640 1 -- 172.21.15.122:0/3829121284 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f3fd0058b60 con 0x7f3fdc0b4f70 2024-03-06T07:25:08.309 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:08.309 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:08.309 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fc806b4d0 msgr2=0x7f3fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fc806b4d0 0x7f3fc806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4035000 tx=0x7f3fd4030000 comp rx=0 tx=0).stop 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 msgr2=0x7f3fdc0cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0cbb60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000d7d0 tx=0x7f3fd000dc90 comp rx=0 tx=0).stop 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 shutdown_connections 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fc806b4d0 0x7f3fc806d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fdc1555d0 0x7f3fdc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fdc0b4f70 0x7f3fdc0cbb60 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 --2- 172.21.15.122:0/3829121284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fdc066d90 0x7f3fdc1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 >> 172.21.15.122:0/3829121284 conn(0x7f3fdc0724d0 msgr2=0x7f3fdc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 shutdown_connections 2024-03-06T07:25:08.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.309+0000 7f3fe2f1e640 1 -- 172.21.15.122:0/3829121284 wait complete. 2024-03-06T07:25:08.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:08.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.421+0000 7ff4c3664640 1 Processor -- start 2024-03-06T07:25:08.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.422+0000 7ff4c3664640 1 -- start start 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c3664640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4bc0b4170 0x7ff4bc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c3664640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff4bc0d4760 0x7ff4bc0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c3664640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc0d25d0 con 0x7ff4bc0b4dd0 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c3664640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4bc0d4bf0 con 0x7ff4bc0b4170 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c3664640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4bc0ce9e0 con 0x7ff4bc0d4760 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4bc0b4170 0x7ff4bc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4bc0b4170 0x7ff4bc0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47864/0 (socket says 172.21.15.122:47864) 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 -- 172.21.15.122:0/2910072328 learned_addr learned my addr 172.21.15.122:0/2910072328 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:08.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c0bff640 1 --2- 172.21.15.122:0/2910072328 >> v2:172.21.15.169:3300/0 conn(0x7ff4bc0d4760 0x7ff4bc0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 -- 172.21.15.122:0/2910072328 >> v2:172.21.15.169:3300/0 conn(0x7ff4bc0d4760 msgr2=0x7ff4bc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 --2- 172.21.15.122:0/2910072328 >> v2:172.21.15.169:3300/0 conn(0x7ff4bc0d4760 0x7ff4bc0d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 -- 172.21.15.122:0/2910072328 >> v1:172.21.15.169:6789/0 conn(0x7ff4bc0b4dd0 legacy=0x7ff4bc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c1400640 1 -- 172.21.15.122:0/2910072328 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc0d2870 con 0x7ff4bc0b4170 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.423+0000 7ff4c0bff640 1 --2- 172.21.15.122:0/2910072328 >> v2:172.21.15.169:3300/0 conn(0x7ff4bc0d4760 0x7ff4bc0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.424+0000 7ff4c1400640 1 --2- 172.21.15.122:0/2910072328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc0b6a90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc08d0c0 tx=0x7ff4b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2e36455f23ae63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.424+0000 7ff4b3fff640 1 -- 172.21.15.122:0/2910072328 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8035020 con 0x7ff4bc0b4170 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.424+0000 7ff4b3fff640 1 -- 172.21.15.122:0/2910072328 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8031870 con 0x7ff4bc0b4170 2024-03-06T07:25:08.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.424+0000 7ff4b3fff640 1 -- 172.21.15.122:0/2910072328 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8031c40 con 0x7ff4bc0b4170 2024-03-06T07:25:08.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 -- 172.21.15.122:0/2910072328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 msgr2=0x7ff4bc0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 --2- 172.21.15.122:0/2910072328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc0b6a90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc08d0c0 tx=0x7ff4b802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:08.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 -- 172.21.15.122:0/2910072328 shutdown_connections 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 --2- 172.21.15.122:0/2910072328 >> v2:172.21.15.169:3300/0 conn(0x7ff4bc0d4760 0x7ff4bc0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 --2- 172.21.15.122:0/2910072328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc0b6a90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 -- 172.21.15.122:0/2910072328 >> 172.21.15.122:0/2910072328 conn(0x7ff4bc0d3c40 msgr2=0x7ff4bc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 -- 172.21.15.122:0/2910072328 shutdown_connections 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.425+0000 7ff4c3664640 1 -- 172.21.15.122:0/2910072328 wait complete. 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.426+0000 7ff4c3664640 1 Processor -- start 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.426+0000 7ff4c3664640 1 -- start start 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c3664640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c3664640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4bc0b4dd0 0x7ff4bc15aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c3664640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4bc15b020 0x7ff4bc0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c3664640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc0b47d0 con 0x7ff4bc0b4170 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c3664640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc0b4550 con 0x7ff4bc0b4dd0 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c3664640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc0b4950 con 0x7ff4bc15b020 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47870/0 (socket says 172.21.15.122:47870) 2024-03-06T07:25:08.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 -- 172.21.15.122:0/159863697 learned_addr learned my addr 172.21.15.122:0/159863697 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c0bff640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4bc0b4dd0 0x7ff4bc15aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 -- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4bc15b020 msgr2=0x7ff4bc0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4bc15b020 0x7ff4bc0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 -- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4bc0b4dd0 msgr2=0x7ff4bc15aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4bc0b4dd0 0x7ff4bc15aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 -- 172.21.15.122:0/159863697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc157290 con 0x7ff4bc0b4170 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.427+0000 7ff4c1400640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc156d50 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8000c00 tx=0x7ff4b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.428+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8040070 con 0x7ff4bc0b4170 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.428+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8030d00 con 0x7ff4bc0b4170 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.428+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8035040 con 0x7ff4bc0b4170 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.428+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc159a40 con 0x7ff4bc0b4170 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.428+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc15a000 con 0x7ff4bc0b4170 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.429+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc0b6a90 con 0x7ff4bc0b4170 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.429+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4b804b060 con 0x7ff4bc0b4170 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.429+0000 7ff4b1ffb640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.430+0000 7ff4c0bff640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.430+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4b80ac040 con 0x7ff4bc0b4170 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.430+0000 7ff4c0bff640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4a006b4d0 0x7ff4a006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc1579e0 tx=0x7ff4ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.434+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4bc0b6a90 con 0x7ff4bc0b4170 2024-03-06T07:25:08.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.613+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4bc04f330 con 0x7ff4bc0b4170 2024-03-06T07:25:08.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.614+0000 7ff4b1ffb640 1 -- 172.21.15.122:0/159863697 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16668 (secure 0 0 0) 0x7ff4b807aec0 con 0x7ff4bc0b4170 2024-03-06T07:25:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:08.617 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":69,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:06.867998+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4a006b4d0 msgr2=0x7ff4a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4a006b4d0 0x7ff4a006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc1579e0 tx=0x7ff4ac008040 comp rx=0 tx=0).stop 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 msgr2=0x7ff4bc156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc156d50 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8000c00 tx=0x7ff4b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 shutdown_connections 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4bc15b020 0x7ff4bc0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4bc0b4dd0 0x7ff4bc15aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 --2- 172.21.15.122:0/159863697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc0b4170 0x7ff4bc156d50 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.616+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 >> 172.21.15.122:0/159863697 conn(0x7ff4bc0d3c40 msgr2=0x7ff4bc06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.617+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 shutdown_connections 2024-03-06T07:25:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.617+0000 7ff4c3664640 1 -- 172.21.15.122:0/159863697 wait complete. 2024-03-06T07:25:08.630 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:08.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.729+0000 7f97ef1f7640 1 Processor -- start 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.729+0000 7f97ef1f7640 1 -- start start 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97ef1f7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f97e8158600 0x7f97e8072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97ef1f7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f97e815e3a0 0x7f97e806d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97ef1f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e815eb40 con 0x7f97e806deb0 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97ef1f7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e8161ba0 con 0x7f97e815e3a0 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97ef1f7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e815ee20 con 0x7f97e8158600 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f97e815e3a0 0x7f97e806d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f97e815e3a0 0x7f97e806d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47882/0 (socket says 172.21.15.122:47882) 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 -- 172.21.15.122:0/2728771937 learned_addr learned my addr 172.21.15.122:0/2728771937 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97ecf93640 1 --2- 172.21.15.122:0/2728771937 >> v2:172.21.15.169:3300/0 conn(0x7f97e8158600 0x7f97e8072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 -- 172.21.15.122:0/2728771937 >> v2:172.21.15.169:3300/0 conn(0x7f97e8158600 msgr2=0x7f97e8072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 --2- 172.21.15.122:0/2728771937 >> v2:172.21.15.169:3300/0 conn(0x7f97e8158600 0x7f97e8072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 -- 172.21.15.122:0/2728771937 >> v1:172.21.15.169:6789/0 conn(0x7f97e806deb0 legacy=0x7f97e8162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.730+0000 7f97dffff640 1 -- 172.21.15.122:0/2728771937 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e815f100 con 0x7f97e815e3a0 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.731+0000 7f97dffff640 1 --2- 172.21.15.122:0/2728771937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e815e3a0 0x7f97e806d970 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f97e8161d20 tx=0x7f97d802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff0f8b00aeb9d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.731+0000 7f97df7fe640 1 -- 172.21.15.122:0/2728771937 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8038070 con 0x7f97e815e3a0 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.731+0000 7f97df7fe640 1 -- 172.21.15.122:0/2728771937 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d802f050 con 0x7f97e815e3a0 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.731+0000 7f97df7fe640 1 -- 172.21.15.122:0/2728771937 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8031970 con 0x7f97e815e3a0 2024-03-06T07:25:08.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2728771937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e815e3a0 msgr2=0x7f97e806d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2728771937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e815e3a0 0x7f97e806d970 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f97e8161d20 tx=0x7f97d802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2728771937 shutdown_connections 2024-03-06T07:25:08.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2728771937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e815e3a0 0x7f97e806d970 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2728771937 >> v2:172.21.15.169:3300/0 conn(0x7f97e8158600 0x7f97e8072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2728771937 >> 172.21.15.122:0/2728771937 conn(0x7f97e806c0c0 msgr2=0x7f97e806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2728771937 shutdown_connections 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.732+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2728771937 wait complete. 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.733+0000 7f97ef1f7640 1 Processor -- start 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.733+0000 7f97ef1f7640 1 -- start start 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ef1f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97e806deb0 0x7f97e815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ef1f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 0x7f97e8149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ef1f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97e815e3a0 0x7f97e814a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ef1f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e81628c0 con 0x7f97e8158600 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ef1f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e8162740 con 0x7f97e815e3a0 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ef1f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e806afb0 con 0x7f97e806deb0 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 0x7f97e8149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 0x7f97e8149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47898/0 (socket says 172.21.15.122:47898) 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 -- 172.21.15.122:0/2881904531 learned_addr learned my addr 172.21.15.122:0/2881904531 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:08.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ed794640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97e815e3a0 0x7f97e814a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ecf93640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97e806deb0 0x7f97e815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 -- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97e806deb0 msgr2=0x7f97e815d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97e806deb0 0x7f97e815d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 -- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97e815e3a0 msgr2=0x7f97e814a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97e815e3a0 0x7f97e814a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 -- 172.21.15.122:0/2881904531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e815d7b0 con 0x7f97e8158600 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97ed794640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97e815e3a0 0x7f97e814a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dffff640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 0x7f97e8149d80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f97d8000c00 tx=0x7f97d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8042070 con 0x7f97e8158600 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.734+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d803d870 con 0x7f97e8158600 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.735+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e81f41e0 con 0x7f97e8158600 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.735+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8038040 con 0x7f97e8158600 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.735+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e81e9800 con 0x7f97e8158600 2024-03-06T07:25:08.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.735+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e804f330 con 0x7f97e8158600 2024-03-06T07:25:08.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.736+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97d8045070 con 0x7f97e8158600 2024-03-06T07:25:08.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.737+0000 7f97dd7fa640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f97d406b4d0 0x7f97d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.737+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97d80ac100 con 0x7f97e8158600 2024-03-06T07:25:08.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.737+0000 7f97ecf93640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f97d406b4d0 0x7f97d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.737+0000 7f97ecf93640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f97d406b4d0 0x7f97d406d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009a40 tx=0x7f97e4003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.741+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97d807b0a0 con 0x7f97e8158600 2024-03-06T07:25:08.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.917+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97e80723f0 con 0x7f97d406b4d0 2024-03-06T07:25:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.923+0000 7f97dd7fa640 1 -- 172.21.15.122:0/2881904531 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f97e80723f0 con 0x7f97d406b4d0 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f97d406b4d0 msgr2=0x7f97d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f97d406b4d0 0x7f97d406d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009a40 tx=0x7f97e4003990 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 msgr2=0x7f97e8149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 0x7f97e8149d80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f97d8000c00 tx=0x7f97d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 shutdown_connections 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f97d406b4d0 0x7f97d406d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97e815e3a0 0x7f97e814a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97e8158600 0x7f97e8149d80 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 --2- 172.21.15.122:0/2881904531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97e806deb0 0x7f97e815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 >> 172.21.15.122:0/2881904531 conn(0x7f97e806c0c0 msgr2=0x7f97e80afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 shutdown_connections 2024-03-06T07:25:08.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:08.927+0000 7f97ef1f7640 1 -- 172.21.15.122:0/2881904531 wait complete. 2024-03-06T07:25:08.940 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-06T07:25:07.370084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256944,"kb_used_data":7480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717776,"statfs":{"total":773094113280,"available":772831002624,"internally_reserved":0,"allocated":7659520,"data_stored":3868132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000670"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858078+0000","last_change":"2024-03-06T07:25:04.858078+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858078+0000","last_undegraded":"2024-03-06T07:25:04.858078+0000","last_fullsized":"2024-03-06T07:25:04.858078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853918+0000","last_change":"2024-03-06T07:25:04.853918+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853918+0000","last_undegraded":"2024-03-06T07:25:04.853918+0000","last_fullsized":"2024-03-06T07:25:04.853918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856216+0000","last_change":"2024-03-06T07:25:04.856216+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856216+0000","last_undegraded":"2024-03-06T07:25:04.856216+0000","last_fullsized":"2024-03-06T07:25:04.856216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855094+0000","last_change":"2024-03-06T07:25:04.855094+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855094+0000","last_undegraded":"2024-03-06T07:25:04.855094+0000","last_fullsized":"2024-03-06T07:25:04.855094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857926+0000","last_change":"2024-03-06T07:25:04.857926+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857926+0000","last_undegraded":"2024-03-06T07:25:04.857926+0000","last_fullsized":"2024-03-06T07:25:04.857926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855162+0000","last_change":"2024-03-06T07:25:04.855162+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855162+0000","last_undegraded":"2024-03-06T07:25:04.855162+0000","last_fullsized":"2024-03-06T07:25:04.855162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859124+0000","last_change":"2024-03-06T07:25:04.859124+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859124+0000","last_undegraded":"2024-03-06T07:25:04.859124+0000","last_fullsized":"2024-03-06T07:25:04.859124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854959+0000","last_change":"2024-03-06T07:25:04.854959+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854959+0000","last_undegraded":"2024-03-06T07:25:04.854959+0000","last_fullsized":"2024-03-06T07:25:04.854959+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857036+0000","last_change":"2024-03-06T07:25:04.857036+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857036+0000","last_undegraded":"2024-03-06T07:25:04.857036+0000","last_fullsized":"2024-03-06T07:25:04.857036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858672+0000","last_change":"2024-03-06T07:25:04.858672+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858672+0000","last_undegraded":"2024-03-06T07:25:04.858672+0000","last_fullsized":"2024-03-06T07:25:04.858672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860968+0000","last_change":"2024-03-06T07:25:04.860968+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860968+0000","last_undegraded":"2024-03-06T07:25:04.860968+0000","last_fullsized":"2024-03-06T07:25:04.860968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854645+0000","last_change":"2024-03-06T07:25:04.854645+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854645+0000","last_undegraded":"2024-03-06T07:25:04.854645+0000","last_fullsized":"2024-03-06T07:25:04.854645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855112+0000","last_change":"2024-03-06T07:25:04.855112+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855112+0000","last_undegraded":"2024-03-06T07:25:04.855112+0000","last_fullsized":"2024-03-06T07:25:04.855112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856602+0000","last_change":"2024-03-06T07:25:04.856602+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856602+0000","last_undegraded":"2024-03-06T07:25:04.856602+0000","last_fullsized":"2024-03-06T07:25:04.856602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854677+0000","last_change":"2024-03-06T07:25:04.854677+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854677+0000","last_undegraded":"2024-03-06T07:25:04.854677+0000","last_fullsized":"2024-03-06T07:25:04.854677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858851+0000","last_change":"2024-03-06T07:25:04.858851+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858851+0000","last_undegraded":"2024-03-06T07:25:04.858851+0000","last_fullsized":"2024-03-06T07:25:04.858851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856578+0000","last_change":"2024-03-06T07:25:04.856578+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856578+0000","last_undegraded":"2024-03-06T07:25:04.856578+0000","last_fullsized":"2024-03-06T07:25:04.856578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855526+0000","last_change":"2024-03-06T07:25:04.855526+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855526+0000","last_undegraded":"2024-03-06T07:25:04.855526+0000","last_fullsized":"2024-03-06T07:25:04.855526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854605+0000","last_change":"2024-03-06T07:25:04.854605+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854605+0000","last_undegraded":"2024-03-06T07:25:04.854605+0000","last_fullsized":"2024-03-06T07:25:04.854605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856470+0000","last_change":"2024-03-06T07:25:04.856470+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856470+0000","last_undegraded":"2024-03-06T07:25:04.856470+0000","last_fullsized":"2024-03-06T07:25:04.856470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860123+0000","last_change":"2024-03-06T07:25:04.860123+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860123+0000","last_undegraded":"2024-03-06T07:25:04.860123+0000","last_fullsized":"2024-03-06T07:25:04.860123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858357+0000","last_change":"2024-03-06T07:25:04.858357+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858357+0000","last_undegraded":"2024-03-06T07:25:04.858357+0000","last_fullsized":"2024-03-06T07:25:04.858357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858229+0000","last_change":"2024-03-06T07:25:04.858229+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858229+0000","last_undegraded":"2024-03-06T07:25:04.858229+0000","last_fullsized":"2024-03-06T07:25:04.858229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857673+0000","last_change":"2024-03-06T07:25:04.857673+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857673+0000","last_undegraded":"2024-03-06T07:25:04.857673+0000","last_fullsized":"2024-03-06T07:25:04.857673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861040+0000","last_change":"2024-03-06T07:25:04.861040+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861040+0000","last_undegraded":"2024-03-06T07:25:04.861040+0000","last_fullsized":"2024-03-06T07:25:04.861040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859134+0000","last_change":"2024-03-06T07:25:04.859134+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859134+0000","last_undegraded":"2024-03-06T07:25:04.859134+0000","last_fullsized":"2024-03-06T07:25:04.859134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855782+0000","last_change":"2024-03-06T07:25:04.855782+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855782+0000","last_undegraded":"2024-03-06T07:25:04.855782+0000","last_fullsized":"2024-03-06T07:25:04.855782+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861710+0000","last_change":"2024-03-06T07:25:04.861710+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861710+0000","last_undegraded":"2024-03-06T07:25:04.861710+0000","last_fullsized":"2024-03-06T07:25:04.861710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859725+0000","last_change":"2024-03-06T07:25:04.859725+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859725+0000","last_undegraded":"2024-03-06T07:25:04.859725+0000","last_fullsized":"2024-03-06T07:25:04.859725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857152+0000","last_change":"2024-03-06T07:25:04.857152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857152+0000","last_undegraded":"2024-03-06T07:25:04.857152+0000","last_fullsized":"2024-03-06T07:25:04.857152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856377+0000","last_change":"2024-03-06T07:25:04.856377+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856377+0000","last_undegraded":"2024-03-06T07:25:04.856377+0000","last_fullsized":"2024-03-06T07:25:04.856377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854396+0000","last_change":"2024-03-06T07:25:04.854396+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854396+0000","last_undegraded":"2024-03-06T07:25:04.854396+0000","last_fullsized":"2024-03-06T07:25:04.854396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870547+0000","last_change":"2024-03-06T07:25:04.867874+0000","last_active":"2024-03-06T07:25:04.870547+0000","last_peered":"2024-03-06T07:25:04.870547+0000","last_clean":"2024-03-06T07:25:04.870547+0000","last_became_active":"2024-03-06T07:25:04.867619+0000","last_became_peered":"2024-03-06T07:25:04.867619+0000","last_unstale":"2024-03-06T07:25:04.870547+0000","last_undegraded":"2024-03-06T07:25:04.870547+0000","last_fullsized":"2024-03-06T07:25:04.870547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853353+0000","last_change":"2024-03-06T07:25:04.853353+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853353+0000","last_undegraded":"2024-03-06T07:25:04.853353+0000","last_fullsized":"2024-03-06T07:25:04.853353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865840+0000","last_change":"2024-03-06T07:25:04.864438+0000","last_active":"2024-03-06T07:25:04.865840+0000","last_peered":"2024-03-06T07:25:04.865840+0000","last_clean":"2024-03-06T07:25:04.865840+0000","last_became_active":"2024-03-06T07:25:04.864180+0000","last_became_peered":"2024-03-06T07:25:04.864180+0000","last_unstale":"2024-03-06T07:25:04.865840+0000","last_undegraded":"2024-03-06T07:25:04.865840+0000","last_fullsized":"2024-03-06T07:25:04.865840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856020+0000","last_change":"2024-03-06T07:25:04.856020+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856020+0000","last_undegraded":"2024-03-06T07:25:04.856020+0000","last_fullsized":"2024-03-06T07:25:04.856020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872656+0000","last_change":"2024-03-06T07:25:04.871271+0000","last_active":"2024-03-06T07:25:04.872656+0000","last_peered":"2024-03-06T07:25:04.872656+0000","last_clean":"2024-03-06T07:25:04.872656+0000","last_became_active":"2024-03-06T07:25:04.870981+0000","last_became_peered":"2024-03-06T07:25:04.870981+0000","last_unstale":"2024-03-06T07:25:04.872656+0000","last_undegraded":"2024-03-06T07:25:04.872656+0000","last_fullsized":"2024-03-06T07:25:04.872656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:21.893351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860599+0000","last_change":"2024-03-06T07:25:04.860599+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860599+0000","last_undegraded":"2024-03-06T07:25:04.860599+0000","last_fullsized":"2024-03-06T07:25:04.860599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869854+0000","last_change":"2024-03-06T07:25:04.868531+0000","last_active":"2024-03-06T07:25:04.869854+0000","last_peered":"2024-03-06T07:25:04.869854+0000","last_clean":"2024-03-06T07:25:04.869854+0000","last_became_active":"2024-03-06T07:25:04.868198+0000","last_became_peered":"2024-03-06T07:25:04.868198+0000","last_unstale":"2024-03-06T07:25:04.869854+0000","last_undegraded":"2024-03-06T07:25:04.869854+0000","last_fullsized":"2024-03-06T07:25:04.869854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:56.138604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856363+0000","last_change":"2024-03-06T07:25:04.856363+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856363+0000","last_undegraded":"2024-03-06T07:25:04.856363+0000","last_fullsized":"2024-03-06T07:25:04.856363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869200+0000","last_change":"2024-03-06T07:25:04.867446+0000","last_active":"2024-03-06T07:25:04.869200+0000","last_peered":"2024-03-06T07:25:04.869200+0000","last_clean":"2024-03-06T07:25:04.869200+0000","last_became_active":"2024-03-06T07:25:04.867198+0000","last_became_peered":"2024-03-06T07:25:04.867198+0000","last_unstale":"2024-03-06T07:25:04.869200+0000","last_undegraded":"2024-03-06T07:25:04.869200+0000","last_fullsized":"2024-03-06T07:25:04.869200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:32.313600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869433+0000","last_change":"2024-03-06T07:25:04.862460+0000","last_active":"2024-03-06T07:25:04.869433+0000","last_peered":"2024-03-06T07:25:04.869433+0000","last_clean":"2024-03-06T07:25:04.869433+0000","last_became_active":"2024-03-06T07:25:04.862179+0000","last_became_peered":"2024-03-06T07:25:04.862179+0000","last_unstale":"2024-03-06T07:25:04.869433+0000","last_undegraded":"2024-03-06T07:25:04.869433+0000","last_fullsized":"2024-03-06T07:25:04.869433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:04.876240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854255+0000","last_change":"2024-03-06T07:25:04.854255+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854255+0000","last_undegraded":"2024-03-06T07:25:04.854255+0000","last_fullsized":"2024-03-06T07:25:04.854255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865330+0000","last_change":"2024-03-06T07:25:04.862049+0000","last_active":"2024-03-06T07:25:04.865330+0000","last_peered":"2024-03-06T07:25:04.865330+0000","last_clean":"2024-03-06T07:25:04.865330+0000","last_became_active":"2024-03-06T07:25:04.861815+0000","last_became_peered":"2024-03-06T07:25:04.861815+0000","last_unstale":"2024-03-06T07:25:04.865330+0000","last_undegraded":"2024-03-06T07:25:04.865330+0000","last_fullsized":"2024-03-06T07:25:04.865330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:30.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854239+0000","last_change":"2024-03-06T07:25:04.854239+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854239+0000","last_undegraded":"2024-03-06T07:25:04.854239+0000","last_fullsized":"2024-03-06T07:25:04.854239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863920+0000","last_change":"2024-03-06T07:25:04.859156+0000","last_active":"2024-03-06T07:25:04.863920+0000","last_peered":"2024-03-06T07:25:04.863920+0000","last_clean":"2024-03-06T07:25:04.863920+0000","last_became_active":"2024-03-06T07:25:04.858589+0000","last_became_peered":"2024-03-06T07:25:04.858589+0000","last_unstale":"2024-03-06T07:25:04.863920+0000","last_undegraded":"2024-03-06T07:25:04.863920+0000","last_fullsized":"2024-03-06T07:25:04.863920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:46.223486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857051+0000","last_change":"2024-03-06T07:25:04.857051+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857051+0000","last_undegraded":"2024-03-06T07:25:04.857051+0000","last_fullsized":"2024-03-06T07:25:04.857051+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872201+0000","last_change":"2024-03-06T07:25:04.866854+0000","last_active":"2024-03-06T07:25:04.872201+0000","last_peered":"2024-03-06T07:25:04.872201+0000","last_clean":"2024-03-06T07:25:04.872201+0000","last_became_active":"2024-03-06T07:25:04.866566+0000","last_became_peered":"2024-03-06T07:25:04.866566+0000","last_unstale":"2024-03-06T07:25:04.872201+0000","last_undegraded":"2024-03-06T07:25:04.872201+0000","last_fullsized":"2024-03-06T07:25:04.872201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:50.439569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857731+0000","last_change":"2024-03-06T07:25:04.857731+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857731+0000","last_undegraded":"2024-03-06T07:25:04.857731+0000","last_fullsized":"2024-03-06T07:25:04.857731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869677+0000","last_change":"2024-03-06T07:25:04.862756+0000","last_active":"2024-03-06T07:25:04.869677+0000","last_peered":"2024-03-06T07:25:04.869677+0000","last_clean":"2024-03-06T07:25:04.869677+0000","last_became_active":"2024-03-06T07:25:04.862463+0000","last_became_peered":"2024-03-06T07:25:04.862463+0000","last_unstale":"2024-03-06T07:25:04.869677+0000","last_undegraded":"2024-03-06T07:25:04.869677+0000","last_fullsized":"2024-03-06T07:25:04.869677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:10.409474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855499+0000","last_change":"2024-03-06T07:25:04.855499+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855499+0000","last_undegraded":"2024-03-06T07:25:04.855499+0000","last_fullsized":"2024-03-06T07:25:04.855499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.866505+0000","last_change":"2024-03-06T07:25:04.864989+0000","last_active":"2024-03-06T07:25:04.866505+0000","last_peered":"2024-03-06T07:25:04.866505+0000","last_clean":"2024-03-06T07:25:04.866505+0000","last_became_active":"2024-03-06T07:25:04.864724+0000","last_became_peered":"2024-03-06T07:25:04.864724+0000","last_unstale":"2024-03-06T07:25:04.866505+0000","last_undegraded":"2024-03-06T07:25:04.866505+0000","last_fullsized":"2024-03-06T07:25:04.866505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:13.895735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854484+0000","last_change":"2024-03-06T07:25:04.854484+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854484+0000","last_undegraded":"2024-03-06T07:25:04.854484+0000","last_fullsized":"2024-03-06T07:25:04.854484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.867439+0000","last_change":"2024-03-06T07:25:04.866085+0000","last_active":"2024-03-06T07:25:04.867439+0000","last_peered":"2024-03-06T07:25:04.867439+0000","last_clean":"2024-03-06T07:25:04.867439+0000","last_became_active":"2024-03-06T07:25:04.865800+0000","last_became_peered":"2024-03-06T07:25:04.865800+0000","last_unstale":"2024-03-06T07:25:04.867439+0000","last_undegraded":"2024-03-06T07:25:04.867439+0000","last_fullsized":"2024-03-06T07:25:04.867439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:02.525342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858713+0000","last_change":"2024-03-06T07:25:04.858713+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858713+0000","last_undegraded":"2024-03-06T07:25:04.858713+0000","last_fullsized":"2024-03-06T07:25:04.858713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.868697+0000","last_change":"2024-03-06T07:25:04.866818+0000","last_active":"2024-03-06T07:25:04.868697+0000","last_peered":"2024-03-06T07:25:04.868697+0000","last_clean":"2024-03-06T07:25:04.868697+0000","last_became_active":"2024-03-06T07:25:04.866510+0000","last_became_peered":"2024-03-06T07:25:04.866510+0000","last_unstale":"2024-03-06T07:25:04.868697+0000","last_undegraded":"2024-03-06T07:25:04.868697+0000","last_fullsized":"2024-03-06T07:25:04.868697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.279615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857221+0000","last_change":"2024-03-06T07:25:04.857221+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857221+0000","last_undegraded":"2024-03-06T07:25:04.857221+0000","last_fullsized":"2024-03-06T07:25:04.857221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870181+0000","last_change":"2024-03-06T07:25:04.868290+0000","last_active":"2024-03-06T07:25:04.870181+0000","last_peered":"2024-03-06T07:25:04.870181+0000","last_clean":"2024-03-06T07:25:04.870181+0000","last_became_active":"2024-03-06T07:25:04.867100+0000","last_became_peered":"2024-03-06T07:25:04.867100+0000","last_unstale":"2024-03-06T07:25:04.870181+0000","last_undegraded":"2024-03-06T07:25:04.870181+0000","last_fullsized":"2024-03-06T07:25:04.870181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:24.492810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856137+0000","last_change":"2024-03-06T07:25:04.856137+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856137+0000","last_undegraded":"2024-03-06T07:25:04.856137+0000","last_fullsized":"2024-03-06T07:25:04.856137+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872446+0000","last_change":"2024-03-06T07:25:04.870029+0000","last_active":"2024-03-06T07:25:04.872446+0000","last_peered":"2024-03-06T07:25:04.872446+0000","last_clean":"2024-03-06T07:25:04.872446+0000","last_became_active":"2024-03-06T07:25:04.869729+0000","last_became_peered":"2024-03-06T07:25:04.869729+0000","last_unstale":"2024-03-06T07:25:04.872446+0000","last_undegraded":"2024-03-06T07:25:04.872446+0000","last_fullsized":"2024-03-06T07:25:04.872446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:36.772850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853796+0000","last_change":"2024-03-06T07:25:04.853796+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853796+0000","last_undegraded":"2024-03-06T07:25:04.853796+0000","last_fullsized":"2024-03-06T07:25:04.853796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872524+0000","last_change":"2024-03-06T07:25:04.869373+0000","last_active":"2024-03-06T07:25:04.872524+0000","last_peered":"2024-03-06T07:25:04.872524+0000","last_clean":"2024-03-06T07:25:04.872524+0000","last_became_active":"2024-03-06T07:25:04.869036+0000","last_became_peered":"2024-03-06T07:25:04.869036+0000","last_unstale":"2024-03-06T07:25:04.872524+0000","last_undegraded":"2024-03-06T07:25:04.872524+0000","last_fullsized":"2024-03-06T07:25:04.872524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858111+0000","last_change":"2024-03-06T07:25:04.858111+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858111+0000","last_undegraded":"2024-03-06T07:25:04.858111+0000","last_fullsized":"2024-03-06T07:25:04.858111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869904+0000","last_change":"2024-03-06T07:25:04.867842+0000","last_active":"2024-03-06T07:25:04.869904+0000","last_peered":"2024-03-06T07:25:04.869904+0000","last_clean":"2024-03-06T07:25:04.869904+0000","last_became_active":"2024-03-06T07:25:04.867564+0000","last_became_peered":"2024-03-06T07:25:04.867564+0000","last_unstale":"2024-03-06T07:25:04.869904+0000","last_undegraded":"2024-03-06T07:25:04.869904+0000","last_fullsized":"2024-03-06T07:25:04.869904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.953333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857270+0000","last_change":"2024-03-06T07:25:04.857270+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857270+0000","last_undegraded":"2024-03-06T07:25:04.857270+0000","last_fullsized":"2024-03-06T07:25:04.857270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872606+0000","last_change":"2024-03-06T07:25:04.870677+0000","last_active":"2024-03-06T07:25:04.872606+0000","last_peered":"2024-03-06T07:25:04.872606+0000","last_clean":"2024-03-06T07:25:04.872606+0000","last_became_active":"2024-03-06T07:25:04.870423+0000","last_became_peered":"2024-03-06T07:25:04.870423+0000","last_unstale":"2024-03-06T07:25:04.872606+0000","last_undegraded":"2024-03-06T07:25:04.872606+0000","last_fullsized":"2024-03-06T07:25:04.872606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:01.989975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854259+0000","last_change":"2024-03-06T07:25:04.854259+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854259+0000","last_undegraded":"2024-03-06T07:25:04.854259+0000","last_fullsized":"2024-03-06T07:25:04.854259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872449+0000","last_change":"2024-03-06T07:25:04.869978+0000","last_active":"2024-03-06T07:25:04.872449+0000","last_peered":"2024-03-06T07:25:04.872449+0000","last_clean":"2024-03-06T07:25:04.872449+0000","last_became_active":"2024-03-06T07:25:04.869501+0000","last_became_peered":"2024-03-06T07:25:04.869501+0000","last_unstale":"2024-03-06T07:25:04.872449+0000","last_undegraded":"2024-03-06T07:25:04.872449+0000","last_fullsized":"2024-03-06T07:25:04.872449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:59.249650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854014+0000","last_change":"2024-03-06T07:25:04.854014+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854014+0000","last_undegraded":"2024-03-06T07:25:04.854014+0000","last_fullsized":"2024-03-06T07:25:04.854014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872463+0000","last_change":"2024-03-06T07:25:04.870140+0000","last_active":"2024-03-06T07:25:04.872463+0000","last_peered":"2024-03-06T07:25:04.872463+0000","last_clean":"2024-03-06T07:25:04.872463+0000","last_became_active":"2024-03-06T07:25:04.869889+0000","last_became_peered":"2024-03-06T07:25:04.869889+0000","last_unstale":"2024-03-06T07:25:04.872463+0000","last_undegraded":"2024-03-06T07:25:04.872463+0000","last_fullsized":"2024-03-06T07:25:04.872463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859370+0000","last_change":"2024-03-06T07:25:04.859370+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859370+0000","last_undegraded":"2024-03-06T07:25:04.859370+0000","last_fullsized":"2024-03-06T07:25:04.859370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869412+0000","last_change":"2024-03-06T07:25:04.861518+0000","last_active":"2024-03-06T07:25:04.869412+0000","last_peered":"2024-03-06T07:25:04.869412+0000","last_clean":"2024-03-06T07:25:04.869412+0000","last_became_active":"2024-03-06T07:25:04.861183+0000","last_became_peered":"2024-03-06T07:25:04.861183+0000","last_unstale":"2024-03-06T07:25:04.869412+0000","last_undegraded":"2024-03-06T07:25:04.869412+0000","last_fullsized":"2024-03-06T07:25:04.869412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:40.600405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855390+0000","last_change":"2024-03-06T07:25:04.855390+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855390+0000","last_undegraded":"2024-03-06T07:25:04.855390+0000","last_fullsized":"2024-03-06T07:25:04.855390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869925+0000","last_change":"2024-03-06T07:25:04.865215+0000","last_active":"2024-03-06T07:25:04.869925+0000","last_peered":"2024-03-06T07:25:04.869925+0000","last_clean":"2024-03-06T07:25:04.869925+0000","last_became_active":"2024-03-06T07:25:04.864955+0000","last_became_peered":"2024-03-06T07:25:04.864955+0000","last_unstale":"2024-03-06T07:25:04.869925+0000","last_undegraded":"2024-03-06T07:25:04.869925+0000","last_fullsized":"2024-03-06T07:25:04.869925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:42.691950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857793+0000","last_change":"2024-03-06T07:25:04.857793+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857793+0000","last_undegraded":"2024-03-06T07:25:04.857793+0000","last_fullsized":"2024-03-06T07:25:04.857793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869331+0000","last_change":"2024-03-06T07:25:04.865095+0000","last_active":"2024-03-06T07:25:04.869331+0000","last_peered":"2024-03-06T07:25:04.869331+0000","last_clean":"2024-03-06T07:25:04.869331+0000","last_became_active":"2024-03-06T07:25:04.864783+0000","last_became_peered":"2024-03-06T07:25:04.864783+0000","last_unstale":"2024-03-06T07:25:04.869331+0000","last_undegraded":"2024-03-06T07:25:04.869331+0000","last_fullsized":"2024-03-06T07:25:04.869331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855883+0000","last_change":"2024-03-06T07:25:04.855883+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855883+0000","last_undegraded":"2024-03-06T07:25:04.855883+0000","last_fullsized":"2024-03-06T07:25:04.855883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869886+0000","last_change":"2024-03-06T07:25:04.866682+0000","last_active":"2024-03-06T07:25:04.869886+0000","last_peered":"2024-03-06T07:25:04.869886+0000","last_clean":"2024-03-06T07:25:04.869886+0000","last_became_active":"2024-03-06T07:25:04.866374+0000","last_became_peered":"2024-03-06T07:25:04.866374+0000","last_unstale":"2024-03-06T07:25:04.869886+0000","last_undegraded":"2024-03-06T07:25:04.869886+0000","last_fullsized":"2024-03-06T07:25:04.869886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:15.941972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858011+0000","last_change":"2024-03-06T07:25:04.858011+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858011+0000","last_undegraded":"2024-03-06T07:25:04.858011+0000","last_fullsized":"2024-03-06T07:25:04.858011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869185+0000","last_change":"2024-03-06T07:25:04.864465+0000","last_active":"2024-03-06T07:25:04.869185+0000","last_peered":"2024-03-06T07:25:04.869185+0000","last_clean":"2024-03-06T07:25:04.869185+0000","last_became_active":"2024-03-06T07:25:04.864206+0000","last_became_peered":"2024-03-06T07:25:04.864206+0000","last_unstale":"2024-03-06T07:25:04.869185+0000","last_undegraded":"2024-03-06T07:25:04.869185+0000","last_fullsized":"2024-03-06T07:25:04.869185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:29.767631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858764+0000","last_change":"2024-03-06T07:25:04.858764+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858764+0000","last_undegraded":"2024-03-06T07:25:04.858764+0000","last_fullsized":"2024-03-06T07:25:04.858764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863948+0000","last_change":"2024-03-06T07:25:04.859877+0000","last_active":"2024-03-06T07:25:04.863948+0000","last_peered":"2024-03-06T07:25:04.863948+0000","last_clean":"2024-03-06T07:25:04.863948+0000","last_became_active":"2024-03-06T07:25:04.859458+0000","last_became_peered":"2024-03-06T07:25:04.859458+0000","last_unstale":"2024-03-06T07:25:04.863948+0000","last_undegraded":"2024-03-06T07:25:04.863948+0000","last_fullsized":"2024-03-06T07:25:04.863948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:51.757108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860052+0000","last_change":"2024-03-06T07:25:04.860052+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860052+0000","last_undegraded":"2024-03-06T07:25:04.860052+0000","last_fullsized":"2024-03-06T07:25:04.860052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.871744+0000","last_change":"2024-03-06T07:25:04.869898+0000","last_active":"2024-03-06T07:25:04.871744+0000","last_peered":"2024-03-06T07:25:04.871744+0000","last_clean":"2024-03-06T07:25:04.871744+0000","last_became_active":"2024-03-06T07:25:04.869613+0000","last_became_peered":"2024-03-06T07:25:04.869613+0000","last_unstale":"2024-03-06T07:25:04.871744+0000","last_undegraded":"2024-03-06T07:25:04.871744+0000","last_fullsized":"2024-03-06T07:25:04.871744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:01.821292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853346+0000","last_change":"2024-03-06T07:25:04.853346+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853346+0000","last_undegraded":"2024-03-06T07:25:04.853346+0000","last_fullsized":"2024-03-06T07:25:04.853346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.864047+0000","last_change":"2024-03-06T07:25:04.860934+0000","last_active":"2024-03-06T07:25:04.864047+0000","last_peered":"2024-03-06T07:25:04.864047+0000","last_clean":"2024-03-06T07:25:04.864047+0000","last_became_active":"2024-03-06T07:25:04.860341+0000","last_became_peered":"2024-03-06T07:25:04.860341+0000","last_unstale":"2024-03-06T07:25:04.864047+0000","last_undegraded":"2024-03-06T07:25:04.864047+0000","last_fullsized":"2024-03-06T07:25:04.864047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:29.072866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853953+0000","last_change":"2024-03-06T07:25:04.853953+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853953+0000","last_undegraded":"2024-03-06T07:25:04.853953+0000","last_fullsized":"2024-03-06T07:25:04.853953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865333+0000","last_change":"2024-03-06T07:25:04.862667+0000","last_active":"2024-03-06T07:25:04.865333+0000","last_peered":"2024-03-06T07:25:04.865333+0000","last_clean":"2024-03-06T07:25:04.865333+0000","last_became_active":"2024-03-06T07:25:04.862428+0000","last_became_peered":"2024-03-06T07:25:04.862428+0000","last_unstale":"2024-03-06T07:25:04.865333+0000","last_undegraded":"2024-03-06T07:25:04.865333+0000","last_fullsized":"2024-03-06T07:25:04.865333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.676481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853833+0000","last_change":"2024-03-06T07:25:04.853833+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853833+0000","last_undegraded":"2024-03-06T07:25:04.853833+0000","last_fullsized":"2024-03-06T07:25:04.853833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872165+0000","last_change":"2024-03-06T07:25:04.866391+0000","last_active":"2024-03-06T07:25:04.872165+0000","last_peered":"2024-03-06T07:25:04.872165+0000","last_clean":"2024-03-06T07:25:04.872165+0000","last_became_active":"2024-03-06T07:25:04.866047+0000","last_became_peered":"2024-03-06T07:25:04.866047+0000","last_unstale":"2024-03-06T07:25:04.872165+0000","last_undegraded":"2024-03-06T07:25:04.872165+0000","last_fullsized":"2024-03-06T07:25:04.872165+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:13.057086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853896+0000","last_change":"2024-03-06T07:25:04.853896+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853896+0000","last_undegraded":"2024-03-06T07:25:04.853896+0000","last_fullsized":"2024-03-06T07:25:04.853896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870711+0000","last_change":"2024-03-06T07:25:04.868423+0000","last_active":"2024-03-06T07:25:04.870711+0000","last_peered":"2024-03-06T07:25:04.870711+0000","last_clean":"2024-03-06T07:25:04.870711+0000","last_became_active":"2024-03-06T07:25:04.868028+0000","last_became_peered":"2024-03-06T07:25:04.868028+0000","last_unstale":"2024-03-06T07:25:04.870711+0000","last_undegraded":"2024-03-06T07:25:04.870711+0000","last_fullsized":"2024-03-06T07:25:04.870711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:51.388578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859152+0000","last_change":"2024-03-06T07:25:04.859152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859152+0000","last_undegraded":"2024-03-06T07:25:04.859152+0000","last_fullsized":"2024-03-06T07:25:04.859152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869781+0000","last_change":"2024-03-06T07:25:04.867025+0000","last_active":"2024-03-06T07:25:04.869781+0000","last_peered":"2024-03-06T07:25:04.869781+0000","last_clean":"2024-03-06T07:25:04.869781+0000","last_became_active":"2024-03-06T07:25:04.866493+0000","last_became_peered":"2024-03-06T07:25:04.866493+0000","last_unstale":"2024-03-06T07:25:04.869781+0000","last_undegraded":"2024-03-06T07:25:04.869781+0000","last_fullsized":"2024-03-06T07:25:04.869781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:05.098447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.857899+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:04.857899+0000","last_peered":"2024-03-06T07:25:04.857899+0000","last_clean":"2024-03-06T07:25:04.857899+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:04.857899+0000","last_undegraded":"2024-03-06T07:25:04.857899+0000","last_fullsized":"2024-03-06T07:25:04.857899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":929430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":923155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:08.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.079+0000 7f61bda46640 1 Processor -- start 2024-03-06T07:25:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.079+0000 7f61bda46640 1 -- start start 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61bda46640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f61b815bd20 0x7f61b80724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61bda46640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 0x7f61b806b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61bda46640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b80ce9a0 con 0x7f61b815f8f0 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61bda46640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b8072ad0 con 0x7f61b815bd20 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61bda46640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8072db0 con 0x7f61b806afb0 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b67fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f61b815f8f0 0x7f61b80ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36526/0 (socket says 172.21.15.122:36526) 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b67fc640 1 -- 172.21.15.122:0/1285311256 learned_addr learned my addr 172.21.15.122:0/1285311256 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203075022 0 0) 0x7f61ac001110 con 0x7f61b815f8f0 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0003520 con 0x7f61b815f8f0 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/1285311256 >> v2:172.21.15.169:3300/0 conn(0x7f61b815bd20 0x7f61b80724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b77fe640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 0x7f61b806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3531687134 0 0) 0x7f61a0003520 con 0x7f61b815f8f0 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 msgr2=0x7f61b806b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 0x7f61b806b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 >> v2:172.21.15.169:3300/0 conn(0x7f61b815bd20 msgr2=0x7f61b80724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 --2- 172.21.15.122:0/1285311256 >> v2:172.21.15.169:3300/0 conn(0x7f61b815bd20 0x7f61b80724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.080+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b8073090 con 0x7f61b815f8f0 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f61ac001840 con 0x7f61b815f8f0 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 >> v1:172.21.15.122:6789/0 conn(0x7f61b815f8f0 legacy=0x7f61b80ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 0x7f61a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61a0004d40 0x7f61a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8072ad0 con 0x7f61a00039b0 2024-03-06T07:25:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8072db0 con 0x7f61a0004d40 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a000ba70 con 0x7f61b806afb0 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/1285311256 >> v2:172.21.15.169:3300/0 conn(0x7f61b815bd20 0x7f61b80724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b67fc640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 0x7f61a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b77fe640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61a0004d40 0x7f61a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 -- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 msgr2=0x7f61a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b806afb0 0x7f61a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 -- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61a0004d40 msgr2=0x7f61a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61a0004d40 0x7f61a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.081+0000 7f61b6ffd640 1 -- 172.21.15.122:0/1285311256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a0003520 con 0x7f61a00039b0 2024-03-06T07:25:09.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61a0003dd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f61a8007410 tx=0x7f61a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b86100aaf255e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8039070 con 0x7f61a00039b0 2024-03-06T07:25:09.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a802cdc0 con 0x7f61a00039b0 2024-03-06T07:25:09.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61b5ffb640 1 -- 172.21.15.122:0/1285311256 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8009c70 con 0x7f61a00039b0 2024-03-06T07:25:09.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61bda46640 1 -- 172.21.15.122:0/1285311256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 msgr2=0x7f61a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61bda46640 1 --2- 172.21.15.122:0/1285311256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61a0003dd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f61a8007410 tx=0x7f61a0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:09.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.082+0000 7f61b77fe640 1 -- 172.21.15.122:0/1285311256 reap_dead start 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.083+0000 7f61bda46640 1 -- 172.21.15.122:0/1285311256 shutdown_connections 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.083+0000 7f61bda46640 1 -- 172.21.15.122:0/1285311256 >> 172.21.15.122:0/1285311256 conn(0x7f61b80b1d00 msgr2=0x7f61b80b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.083+0000 7f61bda46640 1 -- 172.21.15.122:0/1285311256 shutdown_connections 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.083+0000 7f61bda46640 1 -- 172.21.15.122:0/1285311256 wait complete. 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.084+0000 7f61bda46640 1 Processor -- start 2024-03-06T07:25:09.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.085+0000 7f61bda46640 1 -- start start 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61bda46640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61b81542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61bda46640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b80c98c0 0x7f61b80c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61bda46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61b815a390 0x7f61b815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61bda46640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80672f0 con 0x7f61a00039b0 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61bda46640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80cca90 con 0x7f61b815a390 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61bda46640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b815c100 con 0x7f61b80c98c0 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61b81542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61b81542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47928/0 (socket says 172.21.15.122:47928) 2024-03-06T07:25:09.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 -- 172.21.15.122:0/857112893 learned_addr learned my addr 172.21.15.122:0/857112893 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 -- 172.21.15.122:0/857112893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b80c98c0 msgr2=0x7f61b80c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b80c98c0 0x7f61b80c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 -- 172.21.15.122:0/857112893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61b815a390 msgr2=0x7f61b815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61b815a390 0x7f61b815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 -- 172.21.15.122:0/857112893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b815d520 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f61b6ffd640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61b81542f0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f61a8007530 tx=0x7f61a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.086+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8039050 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.087+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8045070 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.087+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b815d7c0 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.087+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8009c70 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.087+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b8154830 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.088+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b80b44f0 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.088+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61a803e830 con 0x7f61a00039b0 2024-03-06T07:25:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.088+0000 7f619ffff640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61a406b500 0x7f61a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.088+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61a80ac8e0 con 0x7f61a00039b0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.088+0000 7f61b67fc640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61a406b500 0x7f61a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.089+0000 7f61b67fc640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61a406b500 0x7f61a406d9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f61ac001eb0 tx=0x7f61ac001d80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.093+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61b80b44f0 con 0x7f61a00039b0 2024-03-06T07:25:09.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.266+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61b81f1db0 con 0x7f61a406b500 2024-03-06T07:25:09.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.277+0000 7f619ffff640 1 -- 172.21.15.122:0/857112893 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f61b81f1db0 con 0x7f61a406b500 2024-03-06T07:25:09.282 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:09.283 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61a406b500 msgr2=0x7f61a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61a406b500 0x7f61a406d9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f61ac001eb0 tx=0x7f61ac001d80 comp rx=0 tx=0).stop 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 msgr2=0x7f61b81542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61b81542f0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f61a8007530 tx=0x7f61a8004ab0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 shutdown_connections 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f61a406b500 0x7f61a406d9c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61b815a390 0x7f61b815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61b80c98c0 0x7f61b80c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 --2- 172.21.15.122:0/857112893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61a00039b0 0x7f61b81542f0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 >> 172.21.15.122:0/857112893 conn(0x7f61b80b1d00 msgr2=0x7f61b80b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 shutdown_connections 2024-03-06T07:25:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.281+0000 7f61bda46640 1 -- 172.21.15.122:0/857112893 wait complete. 2024-03-06T07:25:09.294 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-06T07:25:07.370084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256944,"kb_used_data":7480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717776,"statfs":{"total":773094113280,"available":772831002624,"internally_reserved":0,"allocated":7659520,"data_stored":3868132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000670"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858078+0000","last_change":"2024-03-06T07:25:04.858078+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858078+0000","last_undegraded":"2024-03-06T07:25:04.858078+0000","last_fullsized":"2024-03-06T07:25:04.858078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853918+0000","last_change":"2024-03-06T07:25:04.853918+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853918+0000","last_undegraded":"2024-03-06T07:25:04.853918+0000","last_fullsized":"2024-03-06T07:25:04.853918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856216+0000","last_change":"2024-03-06T07:25:04.856216+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856216+0000","last_undegraded":"2024-03-06T07:25:04.856216+0000","last_fullsized":"2024-03-06T07:25:04.856216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855094+0000","last_change":"2024-03-06T07:25:04.855094+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855094+0000","last_undegraded":"2024-03-06T07:25:04.855094+0000","last_fullsized":"2024-03-06T07:25:04.855094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857926+0000","last_change":"2024-03-06T07:25:04.857926+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857926+0000","last_undegraded":"2024-03-06T07:25:04.857926+0000","last_fullsized":"2024-03-06T07:25:04.857926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855162+0000","last_change":"2024-03-06T07:25:04.855162+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855162+0000","last_undegraded":"2024-03-06T07:25:04.855162+0000","last_fullsized":"2024-03-06T07:25:04.855162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859124+0000","last_change":"2024-03-06T07:25:04.859124+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859124+0000","last_undegraded":"2024-03-06T07:25:04.859124+0000","last_fullsized":"2024-03-06T07:25:04.859124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854959+0000","last_change":"2024-03-06T07:25:04.854959+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854959+0000","last_undegraded":"2024-03-06T07:25:04.854959+0000","last_fullsized":"2024-03-06T07:25:04.854959+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857036+0000","last_change":"2024-03-06T07:25:04.857036+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857036+0000","last_undegraded":"2024-03-06T07:25:04.857036+0000","last_fullsized":"2024-03-06T07:25:04.857036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858672+0000","last_change":"2024-03-06T07:25:04.858672+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858672+0000","last_undegraded":"2024-03-06T07:25:04.858672+0000","last_fullsized":"2024-03-06T07:25:04.858672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860968+0000","last_change":"2024-03-06T07:25:04.860968+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860968+0000","last_undegraded":"2024-03-06T07:25:04.860968+0000","last_fullsized":"2024-03-06T07:25:04.860968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854645+0000","last_change":"2024-03-06T07:25:04.854645+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854645+0000","last_undegraded":"2024-03-06T07:25:04.854645+0000","last_fullsized":"2024-03-06T07:25:04.854645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855112+0000","last_change":"2024-03-06T07:25:04.855112+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855112+0000","last_undegraded":"2024-03-06T07:25:04.855112+0000","last_fullsized":"2024-03-06T07:25:04.855112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856602+0000","last_change":"2024-03-06T07:25:04.856602+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856602+0000","last_undegraded":"2024-03-06T07:25:04.856602+0000","last_fullsized":"2024-03-06T07:25:04.856602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854677+0000","last_change":"2024-03-06T07:25:04.854677+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854677+0000","last_undegraded":"2024-03-06T07:25:04.854677+0000","last_fullsized":"2024-03-06T07:25:04.854677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858851+0000","last_change":"2024-03-06T07:25:04.858851+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858851+0000","last_undegraded":"2024-03-06T07:25:04.858851+0000","last_fullsized":"2024-03-06T07:25:04.858851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856578+0000","last_change":"2024-03-06T07:25:04.856578+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856578+0000","last_undegraded":"2024-03-06T07:25:04.856578+0000","last_fullsized":"2024-03-06T07:25:04.856578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855526+0000","last_change":"2024-03-06T07:25:04.855526+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855526+0000","last_undegraded":"2024-03-06T07:25:04.855526+0000","last_fullsized":"2024-03-06T07:25:04.855526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854605+0000","last_change":"2024-03-06T07:25:04.854605+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854605+0000","last_undegraded":"2024-03-06T07:25:04.854605+0000","last_fullsized":"2024-03-06T07:25:04.854605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856470+0000","last_change":"2024-03-06T07:25:04.856470+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856470+0000","last_undegraded":"2024-03-06T07:25:04.856470+0000","last_fullsized":"2024-03-06T07:25:04.856470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860123+0000","last_change":"2024-03-06T07:25:04.860123+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860123+0000","last_undegraded":"2024-03-06T07:25:04.860123+0000","last_fullsized":"2024-03-06T07:25:04.860123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858357+0000","last_change":"2024-03-06T07:25:04.858357+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858357+0000","last_undegraded":"2024-03-06T07:25:04.858357+0000","last_fullsized":"2024-03-06T07:25:04.858357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858229+0000","last_change":"2024-03-06T07:25:04.858229+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858229+0000","last_undegraded":"2024-03-06T07:25:04.858229+0000","last_fullsized":"2024-03-06T07:25:04.858229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857673+0000","last_change":"2024-03-06T07:25:04.857673+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857673+0000","last_undegraded":"2024-03-06T07:25:04.857673+0000","last_fullsized":"2024-03-06T07:25:04.857673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861040+0000","last_change":"2024-03-06T07:25:04.861040+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861040+0000","last_undegraded":"2024-03-06T07:25:04.861040+0000","last_fullsized":"2024-03-06T07:25:04.861040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859134+0000","last_change":"2024-03-06T07:25:04.859134+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859134+0000","last_undegraded":"2024-03-06T07:25:04.859134+0000","last_fullsized":"2024-03-06T07:25:04.859134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855782+0000","last_change":"2024-03-06T07:25:04.855782+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855782+0000","last_undegraded":"2024-03-06T07:25:04.855782+0000","last_fullsized":"2024-03-06T07:25:04.855782+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861710+0000","last_change":"2024-03-06T07:25:04.861710+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861710+0000","last_undegraded":"2024-03-06T07:25:04.861710+0000","last_fullsized":"2024-03-06T07:25:04.861710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859725+0000","last_change":"2024-03-06T07:25:04.859725+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859725+0000","last_undegraded":"2024-03-06T07:25:04.859725+0000","last_fullsized":"2024-03-06T07:25:04.859725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857152+0000","last_change":"2024-03-06T07:25:04.857152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857152+0000","last_undegraded":"2024-03-06T07:25:04.857152+0000","last_fullsized":"2024-03-06T07:25:04.857152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856377+0000","last_change":"2024-03-06T07:25:04.856377+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856377+0000","last_undegraded":"2024-03-06T07:25:04.856377+0000","last_fullsized":"2024-03-06T07:25:04.856377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854396+0000","last_change":"2024-03-06T07:25:04.854396+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854396+0000","last_undegraded":"2024-03-06T07:25:04.854396+0000","last_fullsized":"2024-03-06T07:25:04.854396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870547+0000","last_change":"2024-03-06T07:25:04.867874+0000","last_active":"2024-03-06T07:25:04.870547+0000","last_peered":"2024-03-06T07:25:04.870547+0000","last_clean":"2024-03-06T07:25:04.870547+0000","last_became_active":"2024-03-06T07:25:04.867619+0000","last_became_peered":"2024-03-06T07:25:04.867619+0000","last_unstale":"2024-03-06T07:25:04.870547+0000","last_undegraded":"2024-03-06T07:25:04.870547+0000","last_fullsized":"2024-03-06T07:25:04.870547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853353+0000","last_change":"2024-03-06T07:25:04.853353+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853353+0000","last_undegraded":"2024-03-06T07:25:04.853353+0000","last_fullsized":"2024-03-06T07:25:04.853353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865840+0000","last_change":"2024-03-06T07:25:04.864438+0000","last_active":"2024-03-06T07:25:04.865840+0000","last_peered":"2024-03-06T07:25:04.865840+0000","last_clean":"2024-03-06T07:25:04.865840+0000","last_became_active":"2024-03-06T07:25:04.864180+0000","last_became_peered":"2024-03-06T07:25:04.864180+0000","last_unstale":"2024-03-06T07:25:04.865840+0000","last_undegraded":"2024-03-06T07:25:04.865840+0000","last_fullsized":"2024-03-06T07:25:04.865840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856020+0000","last_change":"2024-03-06T07:25:04.856020+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856020+0000","last_undegraded":"2024-03-06T07:25:04.856020+0000","last_fullsized":"2024-03-06T07:25:04.856020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872656+0000","last_change":"2024-03-06T07:25:04.871271+0000","last_active":"2024-03-06T07:25:04.872656+0000","last_peered":"2024-03-06T07:25:04.872656+0000","last_clean":"2024-03-06T07:25:04.872656+0000","last_became_active":"2024-03-06T07:25:04.870981+0000","last_became_peered":"2024-03-06T07:25:04.870981+0000","last_unstale":"2024-03-06T07:25:04.872656+0000","last_undegraded":"2024-03-06T07:25:04.872656+0000","last_fullsized":"2024-03-06T07:25:04.872656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:21.893351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860599+0000","last_change":"2024-03-06T07:25:04.860599+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860599+0000","last_undegraded":"2024-03-06T07:25:04.860599+0000","last_fullsized":"2024-03-06T07:25:04.860599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869854+0000","last_change":"2024-03-06T07:25:04.868531+0000","last_active":"2024-03-06T07:25:04.869854+0000","last_peered":"2024-03-06T07:25:04.869854+0000","last_clean":"2024-03-06T07:25:04.869854+0000","last_became_active":"2024-03-06T07:25:04.868198+0000","last_became_peered":"2024-03-06T07:25:04.868198+0000","last_unstale":"2024-03-06T07:25:04.869854+0000","last_undegraded":"2024-03-06T07:25:04.869854+0000","last_fullsized":"2024-03-06T07:25:04.869854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:56.138604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856363+0000","last_change":"2024-03-06T07:25:04.856363+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856363+0000","last_undegraded":"2024-03-06T07:25:04.856363+0000","last_fullsized":"2024-03-06T07:25:04.856363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869200+0000","last_change":"2024-03-06T07:25:04.867446+0000","last_active":"2024-03-06T07:25:04.869200+0000","last_peered":"2024-03-06T07:25:04.869200+0000","last_clean":"2024-03-06T07:25:04.869200+0000","last_became_active":"2024-03-06T07:25:04.867198+0000","last_became_peered":"2024-03-06T07:25:04.867198+0000","last_unstale":"2024-03-06T07:25:04.869200+0000","last_undegraded":"2024-03-06T07:25:04.869200+0000","last_fullsized":"2024-03-06T07:25:04.869200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:32.313600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869433+0000","last_change":"2024-03-06T07:25:04.862460+0000","last_active":"2024-03-06T07:25:04.869433+0000","last_peered":"2024-03-06T07:25:04.869433+0000","last_clean":"2024-03-06T07:25:04.869433+0000","last_became_active":"2024-03-06T07:25:04.862179+0000","last_became_peered":"2024-03-06T07:25:04.862179+0000","last_unstale":"2024-03-06T07:25:04.869433+0000","last_undegraded":"2024-03-06T07:25:04.869433+0000","last_fullsized":"2024-03-06T07:25:04.869433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:04.876240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854255+0000","last_change":"2024-03-06T07:25:04.854255+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854255+0000","last_undegraded":"2024-03-06T07:25:04.854255+0000","last_fullsized":"2024-03-06T07:25:04.854255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865330+0000","last_change":"2024-03-06T07:25:04.862049+0000","last_active":"2024-03-06T07:25:04.865330+0000","last_peered":"2024-03-06T07:25:04.865330+0000","last_clean":"2024-03-06T07:25:04.865330+0000","last_became_active":"2024-03-06T07:25:04.861815+0000","last_became_peered":"2024-03-06T07:25:04.861815+0000","last_unstale":"2024-03-06T07:25:04.865330+0000","last_undegraded":"2024-03-06T07:25:04.865330+0000","last_fullsized":"2024-03-06T07:25:04.865330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:30.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854239+0000","last_change":"2024-03-06T07:25:04.854239+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854239+0000","last_undegraded":"2024-03-06T07:25:04.854239+0000","last_fullsized":"2024-03-06T07:25:04.854239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863920+0000","last_change":"2024-03-06T07:25:04.859156+0000","last_active":"2024-03-06T07:25:04.863920+0000","last_peered":"2024-03-06T07:25:04.863920+0000","last_clean":"2024-03-06T07:25:04.863920+0000","last_became_active":"2024-03-06T07:25:04.858589+0000","last_became_peered":"2024-03-06T07:25:04.858589+0000","last_unstale":"2024-03-06T07:25:04.863920+0000","last_undegraded":"2024-03-06T07:25:04.863920+0000","last_fullsized":"2024-03-06T07:25:04.863920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:46.223486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857051+0000","last_change":"2024-03-06T07:25:04.857051+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857051+0000","last_undegraded":"2024-03-06T07:25:04.857051+0000","last_fullsized":"2024-03-06T07:25:04.857051+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872201+0000","last_change":"2024-03-06T07:25:04.866854+0000","last_active":"2024-03-06T07:25:04.872201+0000","last_peered":"2024-03-06T07:25:04.872201+0000","last_clean":"2024-03-06T07:25:04.872201+0000","last_became_active":"2024-03-06T07:25:04.866566+0000","last_became_peered":"2024-03-06T07:25:04.866566+0000","last_unstale":"2024-03-06T07:25:04.872201+0000","last_undegraded":"2024-03-06T07:25:04.872201+0000","last_fullsized":"2024-03-06T07:25:04.872201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:50.439569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857731+0000","last_change":"2024-03-06T07:25:04.857731+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857731+0000","last_undegraded":"2024-03-06T07:25:04.857731+0000","last_fullsized":"2024-03-06T07:25:04.857731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869677+0000","last_change":"2024-03-06T07:25:04.862756+0000","last_active":"2024-03-06T07:25:04.869677+0000","last_peered":"2024-03-06T07:25:04.869677+0000","last_clean":"2024-03-06T07:25:04.869677+0000","last_became_active":"2024-03-06T07:25:04.862463+0000","last_became_peered":"2024-03-06T07:25:04.862463+0000","last_unstale":"2024-03-06T07:25:04.869677+0000","last_undegraded":"2024-03-06T07:25:04.869677+0000","last_fullsized":"2024-03-06T07:25:04.869677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:10.409474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855499+0000","last_change":"2024-03-06T07:25:04.855499+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855499+0000","last_undegraded":"2024-03-06T07:25:04.855499+0000","last_fullsized":"2024-03-06T07:25:04.855499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.866505+0000","last_change":"2024-03-06T07:25:04.864989+0000","last_active":"2024-03-06T07:25:04.866505+0000","last_peered":"2024-03-06T07:25:04.866505+0000","last_clean":"2024-03-06T07:25:04.866505+0000","last_became_active":"2024-03-06T07:25:04.864724+0000","last_became_peered":"2024-03-06T07:25:04.864724+0000","last_unstale":"2024-03-06T07:25:04.866505+0000","last_undegraded":"2024-03-06T07:25:04.866505+0000","last_fullsized":"2024-03-06T07:25:04.866505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:13.895735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854484+0000","last_change":"2024-03-06T07:25:04.854484+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854484+0000","last_undegraded":"2024-03-06T07:25:04.854484+0000","last_fullsized":"2024-03-06T07:25:04.854484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.867439+0000","last_change":"2024-03-06T07:25:04.866085+0000","last_active":"2024-03-06T07:25:04.867439+0000","last_peered":"2024-03-06T07:25:04.867439+0000","last_clean":"2024-03-06T07:25:04.867439+0000","last_became_active":"2024-03-06T07:25:04.865800+0000","last_became_peered":"2024-03-06T07:25:04.865800+0000","last_unstale":"2024-03-06T07:25:04.867439+0000","last_undegraded":"2024-03-06T07:25:04.867439+0000","last_fullsized":"2024-03-06T07:25:04.867439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:02.525342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858713+0000","last_change":"2024-03-06T07:25:04.858713+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858713+0000","last_undegraded":"2024-03-06T07:25:04.858713+0000","last_fullsized":"2024-03-06T07:25:04.858713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.868697+0000","last_change":"2024-03-06T07:25:04.866818+0000","last_active":"2024-03-06T07:25:04.868697+0000","last_peered":"2024-03-06T07:25:04.868697+0000","last_clean":"2024-03-06T07:25:04.868697+0000","last_became_active":"2024-03-06T07:25:04.866510+0000","last_became_peered":"2024-03-06T07:25:04.866510+0000","last_unstale":"2024-03-06T07:25:04.868697+0000","last_undegraded":"2024-03-06T07:25:04.868697+0000","last_fullsized":"2024-03-06T07:25:04.868697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.279615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857221+0000","last_change":"2024-03-06T07:25:04.857221+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857221+0000","last_undegraded":"2024-03-06T07:25:04.857221+0000","last_fullsized":"2024-03-06T07:25:04.857221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870181+0000","last_change":"2024-03-06T07:25:04.868290+0000","last_active":"2024-03-06T07:25:04.870181+0000","last_peered":"2024-03-06T07:25:04.870181+0000","last_clean":"2024-03-06T07:25:04.870181+0000","last_became_active":"2024-03-06T07:25:04.867100+0000","last_became_peered":"2024-03-06T07:25:04.867100+0000","last_unstale":"2024-03-06T07:25:04.870181+0000","last_undegraded":"2024-03-06T07:25:04.870181+0000","last_fullsized":"2024-03-06T07:25:04.870181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:24.492810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856137+0000","last_change":"2024-03-06T07:25:04.856137+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856137+0000","last_undegraded":"2024-03-06T07:25:04.856137+0000","last_fullsized":"2024-03-06T07:25:04.856137+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872446+0000","last_change":"2024-03-06T07:25:04.870029+0000","last_active":"2024-03-06T07:25:04.872446+0000","last_peered":"2024-03-06T07:25:04.872446+0000","last_clean":"2024-03-06T07:25:04.872446+0000","last_became_active":"2024-03-06T07:25:04.869729+0000","last_became_peered":"2024-03-06T07:25:04.869729+0000","last_unstale":"2024-03-06T07:25:04.872446+0000","last_undegraded":"2024-03-06T07:25:04.872446+0000","last_fullsized":"2024-03-06T07:25:04.872446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:36.772850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853796+0000","last_change":"2024-03-06T07:25:04.853796+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853796+0000","last_undegraded":"2024-03-06T07:25:04.853796+0000","last_fullsized":"2024-03-06T07:25:04.853796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872524+0000","last_change":"2024-03-06T07:25:04.869373+0000","last_active":"2024-03-06T07:25:04.872524+0000","last_peered":"2024-03-06T07:25:04.872524+0000","last_clean":"2024-03-06T07:25:04.872524+0000","last_became_active":"2024-03-06T07:25:04.869036+0000","last_became_peered":"2024-03-06T07:25:04.869036+0000","last_unstale":"2024-03-06T07:25:04.872524+0000","last_undegraded":"2024-03-06T07:25:04.872524+0000","last_fullsized":"2024-03-06T07:25:04.872524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858111+0000","last_change":"2024-03-06T07:25:04.858111+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858111+0000","last_undegraded":"2024-03-06T07:25:04.858111+0000","last_fullsized":"2024-03-06T07:25:04.858111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869904+0000","last_change":"2024-03-06T07:25:04.867842+0000","last_active":"2024-03-06T07:25:04.869904+0000","last_peered":"2024-03-06T07:25:04.869904+0000","last_clean":"2024-03-06T07:25:04.869904+0000","last_became_active":"2024-03-06T07:25:04.867564+0000","last_became_peered":"2024-03-06T07:25:04.867564+0000","last_unstale":"2024-03-06T07:25:04.869904+0000","last_undegraded":"2024-03-06T07:25:04.869904+0000","last_fullsized":"2024-03-06T07:25:04.869904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.953333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857270+0000","last_change":"2024-03-06T07:25:04.857270+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857270+0000","last_undegraded":"2024-03-06T07:25:04.857270+0000","last_fullsized":"2024-03-06T07:25:04.857270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872606+0000","last_change":"2024-03-06T07:25:04.870677+0000","last_active":"2024-03-06T07:25:04.872606+0000","last_peered":"2024-03-06T07:25:04.872606+0000","last_clean":"2024-03-06T07:25:04.872606+0000","last_became_active":"2024-03-06T07:25:04.870423+0000","last_became_peered":"2024-03-06T07:25:04.870423+0000","last_unstale":"2024-03-06T07:25:04.872606+0000","last_undegraded":"2024-03-06T07:25:04.872606+0000","last_fullsized":"2024-03-06T07:25:04.872606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:01.989975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854259+0000","last_change":"2024-03-06T07:25:04.854259+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854259+0000","last_undegraded":"2024-03-06T07:25:04.854259+0000","last_fullsized":"2024-03-06T07:25:04.854259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872449+0000","last_change":"2024-03-06T07:25:04.869978+0000","last_active":"2024-03-06T07:25:04.872449+0000","last_peered":"2024-03-06T07:25:04.872449+0000","last_clean":"2024-03-06T07:25:04.872449+0000","last_became_active":"2024-03-06T07:25:04.869501+0000","last_became_peered":"2024-03-06T07:25:04.869501+0000","last_unstale":"2024-03-06T07:25:04.872449+0000","last_undegraded":"2024-03-06T07:25:04.872449+0000","last_fullsized":"2024-03-06T07:25:04.872449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:59.249650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854014+0000","last_change":"2024-03-06T07:25:04.854014+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854014+0000","last_undegraded":"2024-03-06T07:25:04.854014+0000","last_fullsized":"2024-03-06T07:25:04.854014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872463+0000","last_change":"2024-03-06T07:25:04.870140+0000","last_active":"2024-03-06T07:25:04.872463+0000","last_peered":"2024-03-06T07:25:04.872463+0000","last_clean":"2024-03-06T07:25:04.872463+0000","last_became_active":"2024-03-06T07:25:04.869889+0000","last_became_peered":"2024-03-06T07:25:04.869889+0000","last_unstale":"2024-03-06T07:25:04.872463+0000","last_undegraded":"2024-03-06T07:25:04.872463+0000","last_fullsized":"2024-03-06T07:25:04.872463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859370+0000","last_change":"2024-03-06T07:25:04.859370+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859370+0000","last_undegraded":"2024-03-06T07:25:04.859370+0000","last_fullsized":"2024-03-06T07:25:04.859370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869412+0000","last_change":"2024-03-06T07:25:04.861518+0000","last_active":"2024-03-06T07:25:04.869412+0000","last_peered":"2024-03-06T07:25:04.869412+0000","last_clean":"2024-03-06T07:25:04.869412+0000","last_became_active":"2024-03-06T07:25:04.861183+0000","last_became_peered":"2024-03-06T07:25:04.861183+0000","last_unstale":"2024-03-06T07:25:04.869412+0000","last_undegraded":"2024-03-06T07:25:04.869412+0000","last_fullsized":"2024-03-06T07:25:04.869412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:40.600405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855390+0000","last_change":"2024-03-06T07:25:04.855390+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855390+0000","last_undegraded":"2024-03-06T07:25:04.855390+0000","last_fullsized":"2024-03-06T07:25:04.855390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869925+0000","last_change":"2024-03-06T07:25:04.865215+0000","last_active":"2024-03-06T07:25:04.869925+0000","last_peered":"2024-03-06T07:25:04.869925+0000","last_clean":"2024-03-06T07:25:04.869925+0000","last_became_active":"2024-03-06T07:25:04.864955+0000","last_became_peered":"2024-03-06T07:25:04.864955+0000","last_unstale":"2024-03-06T07:25:04.869925+0000","last_undegraded":"2024-03-06T07:25:04.869925+0000","last_fullsized":"2024-03-06T07:25:04.869925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:42.691950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857793+0000","last_change":"2024-03-06T07:25:04.857793+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857793+0000","last_undegraded":"2024-03-06T07:25:04.857793+0000","last_fullsized":"2024-03-06T07:25:04.857793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869331+0000","last_change":"2024-03-06T07:25:04.865095+0000","last_active":"2024-03-06T07:25:04.869331+0000","last_peered":"2024-03-06T07:25:04.869331+0000","last_clean":"2024-03-06T07:25:04.869331+0000","last_became_active":"2024-03-06T07:25:04.864783+0000","last_became_peered":"2024-03-06T07:25:04.864783+0000","last_unstale":"2024-03-06T07:25:04.869331+0000","last_undegraded":"2024-03-06T07:25:04.869331+0000","last_fullsized":"2024-03-06T07:25:04.869331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855883+0000","last_change":"2024-03-06T07:25:04.855883+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855883+0000","last_undegraded":"2024-03-06T07:25:04.855883+0000","last_fullsized":"2024-03-06T07:25:04.855883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869886+0000","last_change":"2024-03-06T07:25:04.866682+0000","last_active":"2024-03-06T07:25:04.869886+0000","last_peered":"2024-03-06T07:25:04.869886+0000","last_clean":"2024-03-06T07:25:04.869886+0000","last_became_active":"2024-03-06T07:25:04.866374+0000","last_became_peered":"2024-03-06T07:25:04.866374+0000","last_unstale":"2024-03-06T07:25:04.869886+0000","last_undegraded":"2024-03-06T07:25:04.869886+0000","last_fullsized":"2024-03-06T07:25:04.869886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:15.941972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858011+0000","last_change":"2024-03-06T07:25:04.858011+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858011+0000","last_undegraded":"2024-03-06T07:25:04.858011+0000","last_fullsized":"2024-03-06T07:25:04.858011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869185+0000","last_change":"2024-03-06T07:25:04.864465+0000","last_active":"2024-03-06T07:25:04.869185+0000","last_peered":"2024-03-06T07:25:04.869185+0000","last_clean":"2024-03-06T07:25:04.869185+0000","last_became_active":"2024-03-06T07:25:04.864206+0000","last_became_peered":"2024-03-06T07:25:04.864206+0000","last_unstale":"2024-03-06T07:25:04.869185+0000","last_undegraded":"2024-03-06T07:25:04.869185+0000","last_fullsized":"2024-03-06T07:25:04.869185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:29.767631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858764+0000","last_change":"2024-03-06T07:25:04.858764+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858764+0000","last_undegraded":"2024-03-06T07:25:04.858764+0000","last_fullsized":"2024-03-06T07:25:04.858764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863948+0000","last_change":"2024-03-06T07:25:04.859877+0000","last_active":"2024-03-06T07:25:04.863948+0000","last_peered":"2024-03-06T07:25:04.863948+0000","last_clean":"2024-03-06T07:25:04.863948+0000","last_became_active":"2024-03-06T07:25:04.859458+0000","last_became_peered":"2024-03-06T07:25:04.859458+0000","last_unstale":"2024-03-06T07:25:04.863948+0000","last_undegraded":"2024-03-06T07:25:04.863948+0000","last_fullsized":"2024-03-06T07:25:04.863948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:51.757108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860052+0000","last_change":"2024-03-06T07:25:04.860052+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860052+0000","last_undegraded":"2024-03-06T07:25:04.860052+0000","last_fullsized":"2024-03-06T07:25:04.860052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.871744+0000","last_change":"2024-03-06T07:25:04.869898+0000","last_active":"2024-03-06T07:25:04.871744+0000","last_peered":"2024-03-06T07:25:04.871744+0000","last_clean":"2024-03-06T07:25:04.871744+0000","last_became_active":"2024-03-06T07:25:04.869613+0000","last_became_peered":"2024-03-06T07:25:04.869613+0000","last_unstale":"2024-03-06T07:25:04.871744+0000","last_undegraded":"2024-03-06T07:25:04.871744+0000","last_fullsized":"2024-03-06T07:25:04.871744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:01.821292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853346+0000","last_change":"2024-03-06T07:25:04.853346+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853346+0000","last_undegraded":"2024-03-06T07:25:04.853346+0000","last_fullsized":"2024-03-06T07:25:04.853346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.864047+0000","last_change":"2024-03-06T07:25:04.860934+0000","last_active":"2024-03-06T07:25:04.864047+0000","last_peered":"2024-03-06T07:25:04.864047+0000","last_clean":"2024-03-06T07:25:04.864047+0000","last_became_active":"2024-03-06T07:25:04.860341+0000","last_became_peered":"2024-03-06T07:25:04.860341+0000","last_unstale":"2024-03-06T07:25:04.864047+0000","last_undegraded":"2024-03-06T07:25:04.864047+0000","last_fullsized":"2024-03-06T07:25:04.864047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:29.072866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853953+0000","last_change":"2024-03-06T07:25:04.853953+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853953+0000","last_undegraded":"2024-03-06T07:25:04.853953+0000","last_fullsized":"2024-03-06T07:25:04.853953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865333+0000","last_change":"2024-03-06T07:25:04.862667+0000","last_active":"2024-03-06T07:25:04.865333+0000","last_peered":"2024-03-06T07:25:04.865333+0000","last_clean":"2024-03-06T07:25:04.865333+0000","last_became_active":"2024-03-06T07:25:04.862428+0000","last_became_peered":"2024-03-06T07:25:04.862428+0000","last_unstale":"2024-03-06T07:25:04.865333+0000","last_undegraded":"2024-03-06T07:25:04.865333+0000","last_fullsized":"2024-03-06T07:25:04.865333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.676481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853833+0000","last_change":"2024-03-06T07:25:04.853833+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853833+0000","last_undegraded":"2024-03-06T07:25:04.853833+0000","last_fullsized":"2024-03-06T07:25:04.853833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872165+0000","last_change":"2024-03-06T07:25:04.866391+0000","last_active":"2024-03-06T07:25:04.872165+0000","last_peered":"2024-03-06T07:25:04.872165+0000","last_clean":"2024-03-06T07:25:04.872165+0000","last_became_active":"2024-03-06T07:25:04.866047+0000","last_became_peered":"2024-03-06T07:25:04.866047+0000","last_unstale":"2024-03-06T07:25:04.872165+0000","last_undegraded":"2024-03-06T07:25:04.872165+0000","last_fullsized":"2024-03-06T07:25:04.872165+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:13.057086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853896+0000","last_change":"2024-03-06T07:25:04.853896+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853896+0000","last_undegraded":"2024-03-06T07:25:04.853896+0000","last_fullsized":"2024-03-06T07:25:04.853896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870711+0000","last_change":"2024-03-06T07:25:04.868423+0000","last_active":"2024-03-06T07:25:04.870711+0000","last_peered":"2024-03-06T07:25:04.870711+0000","last_clean":"2024-03-06T07:25:04.870711+0000","last_became_active":"2024-03-06T07:25:04.868028+0000","last_became_peered":"2024-03-06T07:25:04.868028+0000","last_unstale":"2024-03-06T07:25:04.870711+0000","last_undegraded":"2024-03-06T07:25:04.870711+0000","last_fullsized":"2024-03-06T07:25:04.870711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:51.388578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859152+0000","last_change":"2024-03-06T07:25:04.859152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859152+0000","last_undegraded":"2024-03-06T07:25:04.859152+0000","last_fullsized":"2024-03-06T07:25:04.859152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869781+0000","last_change":"2024-03-06T07:25:04.867025+0000","last_active":"2024-03-06T07:25:04.869781+0000","last_peered":"2024-03-06T07:25:04.869781+0000","last_clean":"2024-03-06T07:25:04.869781+0000","last_became_active":"2024-03-06T07:25:04.866493+0000","last_became_peered":"2024-03-06T07:25:04.866493+0000","last_unstale":"2024-03-06T07:25:04.869781+0000","last_undegraded":"2024-03-06T07:25:04.869781+0000","last_fullsized":"2024-03-06T07:25:04.869781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:05.098447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.857899+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:04.857899+0000","last_peered":"2024-03-06T07:25:04.857899+0000","last_clean":"2024-03-06T07:25:04.857899+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:04.857899+0000","last_undegraded":"2024-03-06T07:25:04.857899+0000","last_fullsized":"2024-03-06T07:25:04.857899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":929430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":923155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:09.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.432+0000 7f90efe88640 1 Processor -- start 2024-03-06T07:25:09.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.432+0000 7f90efe88640 1 -- start start 2024-03-06T07:25:09.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90efe88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80d22b0 0x7f90e80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90efe88640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e80b3430 con 0x7f90e80b41f0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90efe88640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e80b3710 con 0x7f90e815dfe0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90efe88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80d2c50 con 0x7f90e80d22b0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90ee425640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f90e80b41f0 0x7f90e80b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36540/0 (socket says 172.21.15.122:36540) 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90ee425640 1 -- 172.21.15.122:0/37260373 learned_addr learned my addr 172.21.15.122:0/37260373 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.433+0000 7f90ed423640 1 --2- 172.21.15.122:0/37260373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80d22b0 0x7f90e80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862204592 0 0) 0x7f90e80b3430 con 0x7f90e80b41f0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d4003520 con 0x7f90e80b41f0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584239781 0 0) 0x7f90e80b3710 con 0x7f90e815dfe0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e80b3430 con 0x7f90e815dfe0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ed423640 1 -- 172.21.15.122:0/37260373 >> v1:172.21.15.169:6789/0 conn(0x7f90e815dfe0 legacy=0x7f90e80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072718995 0 0) 0x7f90d4003520 con 0x7f90e80b41f0 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ed423640 1 -- 172.21.15.122:0/37260373 >> v1:172.21.15.122:6789/0 conn(0x7f90e80b41f0 legacy=0x7f90e80b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.434+0000 7f90ed423640 1 -- 172.21.15.122:0/37260373 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e80b39f0 con 0x7f90e80d22b0 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90ed423640 1 --2- 172.21.15.122:0/37260373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80d22b0 0x7f90e80d2710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f90d80097d0 tx=0x7f90d802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5088d8014e188e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d8037070 con 0x7f90e80d22b0 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8030510 con 0x7f90e80d22b0 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90ecc22640 1 -- 172.21.15.122:0/37260373 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e00019f0 con 0x7f90e80d22b0 2024-03-06T07:25:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 -- 172.21.15.122:0/37260373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80d22b0 msgr2=0x7f90e80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 --2- 172.21.15.122:0/37260373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80d22b0 0x7f90e80d2710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f90d80097d0 tx=0x7f90d802b800 comp rx=0 tx=0).stop 2024-03-06T07:25:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 -- 172.21.15.122:0/37260373 shutdown_connections 2024-03-06T07:25:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 --2- 172.21.15.122:0/37260373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80d22b0 0x7f90e80d2710 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 -- 172.21.15.122:0/37260373 >> 172.21.15.122:0/37260373 conn(0x7f90e80d3cd0 msgr2=0x7f90e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 -- 172.21.15.122:0/37260373 shutdown_connections 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.435+0000 7f90efe88640 1 -- 172.21.15.122:0/37260373 wait complete. 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.436+0000 7f90efe88640 1 Processor -- start 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 -- start start 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80b41f0 0x7f90e814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90e80d22b0 0x7f90e815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 0x7f90e80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80b3d60 con 0x7f90e815dfe0 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80b31e0 con 0x7f90e80d22b0 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90efe88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80d1c70 con 0x7f90e80b41f0 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90ee425640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 0x7f90e80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90ee425640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 0x7f90e80b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47930/0 (socket says 172.21.15.122:47930) 2024-03-06T07:25:09.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90ee425640 1 -- 172.21.15.122:0/3510194044 learned_addr learned my addr 172.21.15.122:0/3510194044 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90ed423640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90e80d22b0 0x7f90e815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.437+0000 7f90edc24640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80b41f0 0x7f90e814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90ee425640 1 -- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80b41f0 msgr2=0x7f90e814f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90ee425640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80b41f0 0x7f90e814f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90ee425640 1 -- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90e80d22b0 msgr2=0x7f90e815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90ee425640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90e80d22b0 0x7f90e815e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90ee425640 1 -- 172.21.15.122:0/3510194044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e815ed30 con 0x7f90e815dfe0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90ee425640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 0x7f90e80b2180 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f90e0001f30 tx=0x7f90e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0017070 con 0x7f90e815dfe0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e0012400 con 0x7f90e815dfe0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e815f010 con 0x7f90e815dfe0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0012830 con 0x7f90e815dfe0 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.438+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e8160240 con 0x7f90e815dfe0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.439+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e804f330 con 0x7f90e815dfe0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.439+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90e001a070 con 0x7f90e815dfe0 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.440+0000 7f90de7fc640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.440+0000 7f90edc24640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.440+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90e008a0a0 con 0x7f90e815dfe0 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.440+0000 7f90edc24640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90cc06b4d0 0x7f90cc06d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f90e40011e0 tx=0x7f90e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.444+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90e0058960 con 0x7f90e815dfe0 2024-03-06T07:25:09.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.614+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90e80ce940 con 0x7f90cc06b4d0 2024-03-06T07:25:09.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.620+0000 7f90de7fc640 1 -- 172.21.15.122:0/3510194044 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f90e80ce940 con 0x7f90cc06b4d0 2024-03-06T07:25:09.624 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:09.626 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90cc06b4d0 msgr2=0x7f90cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90cc06b4d0 0x7f90cc06d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f90e40011e0 tx=0x7f90e4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 msgr2=0x7f90e80b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 0x7f90e80b2180 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f90e0001f30 tx=0x7f90e000a430 comp rx=0 tx=0).stop 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 shutdown_connections 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90e815dfe0 0x7f90e80b2180 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90e80d22b0 0x7f90e815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 --2- 172.21.15.122:0/3510194044 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90e80b41f0 0x7f90e814f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 >> 172.21.15.122:0/3510194044 conn(0x7f90e80d3cd0 msgr2=0x7f90e806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 shutdown_connections 2024-03-06T07:25:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.623+0000 7f90efe88640 1 -- 172.21.15.122:0/3510194044 wait complete. 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-03-06T07:25:09.370347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256944,"kb_used_data":7480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717776,"statfs":{"total":773094113280,"available":772831002624,"internally_reserved":0,"allocated":7659520,"data_stored":3868132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.522808"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858078+0000","last_change":"2024-03-06T07:25:04.858078+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858078+0000","last_undegraded":"2024-03-06T07:25:04.858078+0000","last_fullsized":"2024-03-06T07:25:04.858078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853918+0000","last_change":"2024-03-06T07:25:04.853918+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853918+0000","last_undegraded":"2024-03-06T07:25:04.853918+0000","last_fullsized":"2024-03-06T07:25:04.853918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856216+0000","last_change":"2024-03-06T07:25:04.856216+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856216+0000","last_undegraded":"2024-03-06T07:25:04.856216+0000","last_fullsized":"2024-03-06T07:25:04.856216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855094+0000","last_change":"2024-03-06T07:25:04.855094+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855094+0000","last_undegraded":"2024-03-06T07:25:04.855094+0000","last_fullsized":"2024-03-06T07:25:04.855094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857926+0000","last_change":"2024-03-06T07:25:04.857926+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857926+0000","last_undegraded":"2024-03-06T07:25:04.857926+0000","last_fullsized":"2024-03-06T07:25:04.857926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855162+0000","last_change":"2024-03-06T07:25:04.855162+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855162+0000","last_undegraded":"2024-03-06T07:25:04.855162+0000","last_fullsized":"2024-03-06T07:25:04.855162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859124+0000","last_change":"2024-03-06T07:25:04.859124+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859124+0000","last_undegraded":"2024-03-06T07:25:04.859124+0000","last_fullsized":"2024-03-06T07:25:04.859124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854959+0000","last_change":"2024-03-06T07:25:04.854959+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854959+0000","last_undegraded":"2024-03-06T07:25:04.854959+0000","last_fullsized":"2024-03-06T07:25:04.854959+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857036+0000","last_change":"2024-03-06T07:25:04.857036+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857036+0000","last_undegraded":"2024-03-06T07:25:04.857036+0000","last_fullsized":"2024-03-06T07:25:04.857036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858672+0000","last_change":"2024-03-06T07:25:04.858672+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858672+0000","last_undegraded":"2024-03-06T07:25:04.858672+0000","last_fullsized":"2024-03-06T07:25:04.858672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860968+0000","last_change":"2024-03-06T07:25:04.860968+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860968+0000","last_undegraded":"2024-03-06T07:25:04.860968+0000","last_fullsized":"2024-03-06T07:25:04.860968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854645+0000","last_change":"2024-03-06T07:25:04.854645+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854645+0000","last_undegraded":"2024-03-06T07:25:04.854645+0000","last_fullsized":"2024-03-06T07:25:04.854645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855112+0000","last_change":"2024-03-06T07:25:04.855112+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855112+0000","last_undegraded":"2024-03-06T07:25:04.855112+0000","last_fullsized":"2024-03-06T07:25:04.855112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856602+0000","last_change":"2024-03-06T07:25:04.856602+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856602+0000","last_undegraded":"2024-03-06T07:25:04.856602+0000","last_fullsized":"2024-03-06T07:25:04.856602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854677+0000","last_change":"2024-03-06T07:25:04.854677+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854677+0000","last_undegraded":"2024-03-06T07:25:04.854677+0000","last_fullsized":"2024-03-06T07:25:04.854677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858851+0000","last_change":"2024-03-06T07:25:04.858851+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858851+0000","last_undegraded":"2024-03-06T07:25:04.858851+0000","last_fullsized":"2024-03-06T07:25:04.858851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856578+0000","last_change":"2024-03-06T07:25:04.856578+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856578+0000","last_undegraded":"2024-03-06T07:25:04.856578+0000","last_fullsized":"2024-03-06T07:25:04.856578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855526+0000","last_change":"2024-03-06T07:25:04.855526+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855526+0000","last_undegraded":"2024-03-06T07:25:04.855526+0000","last_fullsized":"2024-03-06T07:25:04.855526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854605+0000","last_change":"2024-03-06T07:25:04.854605+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854605+0000","last_undegraded":"2024-03-06T07:25:04.854605+0000","last_fullsized":"2024-03-06T07:25:04.854605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856470+0000","last_change":"2024-03-06T07:25:04.856470+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856470+0000","last_undegraded":"2024-03-06T07:25:04.856470+0000","last_fullsized":"2024-03-06T07:25:04.856470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860123+0000","last_change":"2024-03-06T07:25:04.860123+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860123+0000","last_undegraded":"2024-03-06T07:25:04.860123+0000","last_fullsized":"2024-03-06T07:25:04.860123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858357+0000","last_change":"2024-03-06T07:25:04.858357+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858357+0000","last_undegraded":"2024-03-06T07:25:04.858357+0000","last_fullsized":"2024-03-06T07:25:04.858357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858229+0000","last_change":"2024-03-06T07:25:04.858229+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858229+0000","last_undegraded":"2024-03-06T07:25:04.858229+0000","last_fullsized":"2024-03-06T07:25:04.858229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857673+0000","last_change":"2024-03-06T07:25:04.857673+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857673+0000","last_undegraded":"2024-03-06T07:25:04.857673+0000","last_fullsized":"2024-03-06T07:25:04.857673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861040+0000","last_change":"2024-03-06T07:25:04.861040+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861040+0000","last_undegraded":"2024-03-06T07:25:04.861040+0000","last_fullsized":"2024-03-06T07:25:04.861040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859134+0000","last_change":"2024-03-06T07:25:04.859134+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859134+0000","last_undegraded":"2024-03-06T07:25:04.859134+0000","last_fullsized":"2024-03-06T07:25:04.859134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855782+0000","last_change":"2024-03-06T07:25:04.855782+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855782+0000","last_undegraded":"2024-03-06T07:25:04.855782+0000","last_fullsized":"2024-03-06T07:25:04.855782+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861710+0000","last_change":"2024-03-06T07:25:04.861710+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861710+0000","last_undegraded":"2024-03-06T07:25:04.861710+0000","last_fullsized":"2024-03-06T07:25:04.861710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859725+0000","last_change":"2024-03-06T07:25:04.859725+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859725+0000","last_undegraded":"2024-03-06T07:25:04.859725+0000","last_fullsized":"2024-03-06T07:25:04.859725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857152+0000","last_change":"2024-03-06T07:25:04.857152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857152+0000","last_undegraded":"2024-03-06T07:25:04.857152+0000","last_fullsized":"2024-03-06T07:25:04.857152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856377+0000","last_change":"2024-03-06T07:25:04.856377+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856377+0000","last_undegraded":"2024-03-06T07:25:04.856377+0000","last_fullsized":"2024-03-06T07:25:04.856377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854396+0000","last_change":"2024-03-06T07:25:04.854396+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854396+0000","last_undegraded":"2024-03-06T07:25:04.854396+0000","last_fullsized":"2024-03-06T07:25:04.854396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870547+0000","last_change":"2024-03-06T07:25:04.867874+0000","last_active":"2024-03-06T07:25:04.870547+0000","last_peered":"2024-03-06T07:25:04.870547+0000","last_clean":"2024-03-06T07:25:04.870547+0000","last_became_active":"2024-03-06T07:25:04.867619+0000","last_became_peered":"2024-03-06T07:25:04.867619+0000","last_unstale":"2024-03-06T07:25:04.870547+0000","last_undegraded":"2024-03-06T07:25:04.870547+0000","last_fullsized":"2024-03-06T07:25:04.870547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853353+0000","last_change":"2024-03-06T07:25:04.853353+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853353+0000","last_undegraded":"2024-03-06T07:25:04.853353+0000","last_fullsized":"2024-03-06T07:25:04.853353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865840+0000","last_change":"2024-03-06T07:25:04.864438+0000","last_active":"2024-03-06T07:25:04.865840+0000","last_peered":"2024-03-06T07:25:04.865840+0000","last_clean":"2024-03-06T07:25:04.865840+0000","last_became_active":"2024-03-06T07:25:04.864180+0000","last_became_peered":"2024-03-06T07:25:04.864180+0000","last_unstale":"2024-03-06T07:25:04.865840+0000","last_undegraded":"2024-03-06T07:25:04.865840+0000","last_fullsized":"2024-03-06T07:25:04.865840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856020+0000","last_change":"2024-03-06T07:25:04.856020+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856020+0000","last_undegraded":"2024-03-06T07:25:04.856020+0000","last_fullsized":"2024-03-06T07:25:04.856020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872656+0000","last_change":"2024-03-06T07:25:04.871271+0000","last_active":"2024-03-06T07:25:04.872656+0000","last_peered":"2024-03-06T07:25:04.872656+0000","last_clean":"2024-03-06T07:25:04.872656+0000","last_became_active":"2024-03-06T07:25:04.870981+0000","last_became_peered":"2024-03-06T07:25:04.870981+0000","last_unstale":"2024-03-06T07:25:04.872656+0000","last_undegraded":"2024-03-06T07:25:04.872656+0000","last_fullsized":"2024-03-06T07:25:04.872656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:21.893351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860599+0000","last_change":"2024-03-06T07:25:04.860599+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860599+0000","last_undegraded":"2024-03-06T07:25:04.860599+0000","last_fullsized":"2024-03-06T07:25:04.860599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869854+0000","last_change":"2024-03-06T07:25:04.868531+0000","last_active":"2024-03-06T07:25:04.869854+0000","last_peered":"2024-03-06T07:25:04.869854+0000","last_clean":"2024-03-06T07:25:04.869854+0000","last_became_active":"2024-03-06T07:25:04.868198+0000","last_became_peered":"2024-03-06T07:25:04.868198+0000","last_unstale":"2024-03-06T07:25:04.869854+0000","last_undegraded":"2024-03-06T07:25:04.869854+0000","last_fullsized":"2024-03-06T07:25:04.869854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:56.138604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856363+0000","last_change":"2024-03-06T07:25:04.856363+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856363+0000","last_undegraded":"2024-03-06T07:25:04.856363+0000","last_fullsized":"2024-03-06T07:25:04.856363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869200+0000","last_change":"2024-03-06T07:25:04.867446+0000","last_active":"2024-03-06T07:25:04.869200+0000","last_peered":"2024-03-06T07:25:04.869200+0000","last_clean":"2024-03-06T07:25:04.869200+0000","last_became_active":"2024-03-06T07:25:04.867198+0000","last_became_peered":"2024-03-06T07:25:04.867198+0000","last_unstale":"2024-03-06T07:25:04.869200+0000","last_undegraded":"2024-03-06T07:25:04.869200+0000","last_fullsized":"2024-03-06T07:25:04.869200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:32.313600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869433+0000","last_change":"2024-03-06T07:25:04.862460+0000","last_active":"2024-03-06T07:25:04.869433+0000","last_peered":"2024-03-06T07:25:04.869433+0000","last_clean":"2024-03-06T07:25:04.869433+0000","last_became_active":"2024-03-06T07:25:04.862179+0000","last_became_peered":"2024-03-06T07:25:04.862179+0000","last_unstale":"2024-03-06T07:25:04.869433+0000","last_undegraded":"2024-03-06T07:25:04.869433+0000","last_fullsized":"2024-03-06T07:25:04.869433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:04.876240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854255+0000","last_change":"2024-03-06T07:25:04.854255+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854255+0000","last_undegraded":"2024-03-06T07:25:04.854255+0000","last_fullsized":"2024-03-06T07:25:04.854255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865330+0000","last_change":"2024-03-06T07:25:04.862049+0000","last_active":"2024-03-06T07:25:04.865330+0000","last_peered":"2024-03-06T07:25:04.865330+0000","last_clean":"2024-03-06T07:25:04.865330+0000","last_became_active":"2024-03-06T07:25:04.861815+0000","last_became_peered":"2024-03-06T07:25:04.861815+0000","last_unstale":"2024-03-06T07:25:04.865330+0000","last_undegraded":"2024-03-06T07:25:04.865330+0000","last_fullsized":"2024-03-06T07:25:04.865330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:30.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854239+0000","last_change":"2024-03-06T07:25:04.854239+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854239+0000","last_undegraded":"2024-03-06T07:25:04.854239+0000","last_fullsized":"2024-03-06T07:25:04.854239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863920+0000","last_change":"2024-03-06T07:25:04.859156+0000","last_active":"2024-03-06T07:25:04.863920+0000","last_peered":"2024-03-06T07:25:04.863920+0000","last_clean":"2024-03-06T07:25:04.863920+0000","last_became_active":"2024-03-06T07:25:04.858589+0000","last_became_peered":"2024-03-06T07:25:04.858589+0000","last_unstale":"2024-03-06T07:25:04.863920+0000","last_undegraded":"2024-03-06T07:25:04.863920+0000","last_fullsized":"2024-03-06T07:25:04.863920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:46.223486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857051+0000","last_change":"2024-03-06T07:25:04.857051+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857051+0000","last_undegraded":"2024-03-06T07:25:04.857051+0000","last_fullsized":"2024-03-06T07:25:04.857051+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872201+0000","last_change":"2024-03-06T07:25:04.866854+0000","last_active":"2024-03-06T07:25:04.872201+0000","last_peered":"2024-03-06T07:25:04.872201+0000","last_clean":"2024-03-06T07:25:04.872201+0000","last_became_active":"2024-03-06T07:25:04.866566+0000","last_became_peered":"2024-03-06T07:25:04.866566+0000","last_unstale":"2024-03-06T07:25:04.872201+0000","last_undegraded":"2024-03-06T07:25:04.872201+0000","last_fullsized":"2024-03-06T07:25:04.872201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:50.439569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857731+0000","last_change":"2024-03-06T07:25:04.857731+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857731+0000","last_undegraded":"2024-03-06T07:25:04.857731+0000","last_fullsized":"2024-03-06T07:25:04.857731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869677+0000","last_change":"2024-03-06T07:25:04.862756+0000","last_active":"2024-03-06T07:25:04.869677+0000","last_peered":"2024-03-06T07:25:04.869677+0000","last_clean":"2024-03-06T07:25:04.869677+0000","last_became_active":"2024-03-06T07:25:04.862463+0000","last_became_peered":"2024-03-06T07:25:04.862463+0000","last_unstale":"2024-03-06T07:25:04.869677+0000","last_undegraded":"2024-03-06T07:25:04.869677+0000","last_fullsized":"2024-03-06T07:25:04.869677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:10.409474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855499+0000","last_change":"2024-03-06T07:25:04.855499+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855499+0000","last_undegraded":"2024-03-06T07:25:04.855499+0000","last_fullsized":"2024-03-06T07:25:04.855499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.866505+0000","last_change":"2024-03-06T07:25:04.864989+0000","last_active":"2024-03-06T07:25:04.866505+0000","last_peered":"2024-03-06T07:25:04.866505+0000","last_clean":"2024-03-06T07:25:04.866505+0000","last_became_active":"2024-03-06T07:25:04.864724+0000","last_became_peered":"2024-03-06T07:25:04.864724+0000","last_unstale":"2024-03-06T07:25:04.866505+0000","last_undegraded":"2024-03-06T07:25:04.866505+0000","last_fullsized":"2024-03-06T07:25:04.866505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:13.895735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854484+0000","last_change":"2024-03-06T07:25:04.854484+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854484+0000","last_undegraded":"2024-03-06T07:25:04.854484+0000","last_fullsized":"2024-03-06T07:25:04.854484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.867439+0000","last_change":"2024-03-06T07:25:04.866085+0000","last_active":"2024-03-06T07:25:04.867439+0000","last_peered":"2024-03-06T07:25:04.867439+0000","last_clean":"2024-03-06T07:25:04.867439+0000","last_became_active":"2024-03-06T07:25:04.865800+0000","last_became_peered":"2024-03-06T07:25:04.865800+0000","last_unstale":"2024-03-06T07:25:04.867439+0000","last_undegraded":"2024-03-06T07:25:04.867439+0000","last_fullsized":"2024-03-06T07:25:04.867439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:02.525342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858713+0000","last_change":"2024-03-06T07:25:04.858713+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858713+0000","last_undegraded":"2024-03-06T07:25:04.858713+0000","last_fullsized":"2024-03-06T07:25:04.858713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.868697+0000","last_change":"2024-03-06T07:25:04.866818+0000","last_active":"2024-03-06T07:25:04.868697+0000","last_peered":"2024-03-06T07:25:04.868697+0000","last_clean":"2024-03-06T07:25:04.868697+0000","last_became_active":"2024-03-06T07:25:04.866510+0000","last_became_peered":"2024-03-06T07:25:04.866510+0000","last_unstale":"2024-03-06T07:25:04.868697+0000","last_undegraded":"2024-03-06T07:25:04.868697+0000","last_fullsized":"2024-03-06T07:25:04.868697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.279615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857221+0000","last_change":"2024-03-06T07:25:04.857221+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857221+0000","last_undegraded":"2024-03-06T07:25:04.857221+0000","last_fullsized":"2024-03-06T07:25:04.857221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870181+0000","last_change":"2024-03-06T07:25:04.868290+0000","last_active":"2024-03-06T07:25:04.870181+0000","last_peered":"2024-03-06T07:25:04.870181+0000","last_clean":"2024-03-06T07:25:04.870181+0000","last_became_active":"2024-03-06T07:25:04.867100+0000","last_became_peered":"2024-03-06T07:25:04.867100+0000","last_unstale":"2024-03-06T07:25:04.870181+0000","last_undegraded":"2024-03-06T07:25:04.870181+0000","last_fullsized":"2024-03-06T07:25:04.870181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:24.492810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856137+0000","last_change":"2024-03-06T07:25:04.856137+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856137+0000","last_undegraded":"2024-03-06T07:25:04.856137+0000","last_fullsized":"2024-03-06T07:25:04.856137+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872446+0000","last_change":"2024-03-06T07:25:04.870029+0000","last_active":"2024-03-06T07:25:04.872446+0000","last_peered":"2024-03-06T07:25:04.872446+0000","last_clean":"2024-03-06T07:25:04.872446+0000","last_became_active":"2024-03-06T07:25:04.869729+0000","last_became_peered":"2024-03-06T07:25:04.869729+0000","last_unstale":"2024-03-06T07:25:04.872446+0000","last_undegraded":"2024-03-06T07:25:04.872446+0000","last_fullsized":"2024-03-06T07:25:04.872446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:36.772850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853796+0000","last_change":"2024-03-06T07:25:04.853796+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853796+0000","last_undegraded":"2024-03-06T07:25:04.853796+0000","last_fullsized":"2024-03-06T07:25:04.853796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872524+0000","last_change":"2024-03-06T07:25:04.869373+0000","last_active":"2024-03-06T07:25:04.872524+0000","last_peered":"2024-03-06T07:25:04.872524+0000","last_clean":"2024-03-06T07:25:04.872524+0000","last_became_active":"2024-03-06T07:25:04.869036+0000","last_became_peered":"2024-03-06T07:25:04.869036+0000","last_unstale":"2024-03-06T07:25:04.872524+0000","last_undegraded":"2024-03-06T07:25:04.872524+0000","last_fullsized":"2024-03-06T07:25:04.872524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858111+0000","last_change":"2024-03-06T07:25:04.858111+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858111+0000","last_undegraded":"2024-03-06T07:25:04.858111+0000","last_fullsized":"2024-03-06T07:25:04.858111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869904+0000","last_change":"2024-03-06T07:25:04.867842+0000","last_active":"2024-03-06T07:25:04.869904+0000","last_peered":"2024-03-06T07:25:04.869904+0000","last_clean":"2024-03-06T07:25:04.869904+0000","last_became_active":"2024-03-06T07:25:04.867564+0000","last_became_peered":"2024-03-06T07:25:04.867564+0000","last_unstale":"2024-03-06T07:25:04.869904+0000","last_undegraded":"2024-03-06T07:25:04.869904+0000","last_fullsized":"2024-03-06T07:25:04.869904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.953333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857270+0000","last_change":"2024-03-06T07:25:04.857270+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857270+0000","last_undegraded":"2024-03-06T07:25:04.857270+0000","last_fullsized":"2024-03-06T07:25:04.857270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872606+0000","last_change":"2024-03-06T07:25:04.870677+0000","last_active":"2024-03-06T07:25:04.872606+0000","last_peered":"2024-03-06T07:25:04.872606+0000","last_clean":"2024-03-06T07:25:04.872606+0000","last_became_active":"2024-03-06T07:25:04.870423+0000","last_became_peered":"2024-03-06T07:25:04.870423+0000","last_unstale":"2024-03-06T07:25:04.872606+0000","last_undegraded":"2024-03-06T07:25:04.872606+0000","last_fullsized":"2024-03-06T07:25:04.872606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:01.989975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854259+0000","last_change":"2024-03-06T07:25:04.854259+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854259+0000","last_undegraded":"2024-03-06T07:25:04.854259+0000","last_fullsized":"2024-03-06T07:25:04.854259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872449+0000","last_change":"2024-03-06T07:25:04.869978+0000","last_active":"2024-03-06T07:25:04.872449+0000","last_peered":"2024-03-06T07:25:04.872449+0000","last_clean":"2024-03-06T07:25:04.872449+0000","last_became_active":"2024-03-06T07:25:04.869501+0000","last_became_peered":"2024-03-06T07:25:04.869501+0000","last_unstale":"2024-03-06T07:25:04.872449+0000","last_undegraded":"2024-03-06T07:25:04.872449+0000","last_fullsized":"2024-03-06T07:25:04.872449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:59.249650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854014+0000","last_change":"2024-03-06T07:25:04.854014+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854014+0000","last_undegraded":"2024-03-06T07:25:04.854014+0000","last_fullsized":"2024-03-06T07:25:04.854014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872463+0000","last_change":"2024-03-06T07:25:04.870140+0000","last_active":"2024-03-06T07:25:04.872463+0000","last_peered":"2024-03-06T07:25:04.872463+0000","last_clean":"2024-03-06T07:25:04.872463+0000","last_became_active":"2024-03-06T07:25:04.869889+0000","last_became_peered":"2024-03-06T07:25:04.869889+0000","last_unstale":"2024-03-06T07:25:04.872463+0000","last_undegraded":"2024-03-06T07:25:04.872463+0000","last_fullsized":"2024-03-06T07:25:04.872463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859370+0000","last_change":"2024-03-06T07:25:04.859370+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859370+0000","last_undegraded":"2024-03-06T07:25:04.859370+0000","last_fullsized":"2024-03-06T07:25:04.859370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869412+0000","last_change":"2024-03-06T07:25:04.861518+0000","last_active":"2024-03-06T07:25:04.869412+0000","last_peered":"2024-03-06T07:25:04.869412+0000","last_clean":"2024-03-06T07:25:04.869412+0000","last_became_active":"2024-03-06T07:25:04.861183+0000","last_became_peered":"2024-03-06T07:25:04.861183+0000","last_unstale":"2024-03-06T07:25:04.869412+0000","last_undegraded":"2024-03-06T07:25:04.869412+0000","last_fullsized":"2024-03-06T07:25:04.869412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:40.600405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855390+0000","last_change":"2024-03-06T07:25:04.855390+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855390+0000","last_undegraded":"2024-03-06T07:25:04.855390+0000","last_fullsized":"2024-03-06T07:25:04.855390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869925+0000","last_change":"2024-03-06T07:25:04.865215+0000","last_active":"2024-03-06T07:25:04.869925+0000","last_peered":"2024-03-06T07:25:04.869925+0000","last_clean":"2024-03-06T07:25:04.869925+0000","last_became_active":"2024-03-06T07:25:04.864955+0000","last_became_peered":"2024-03-06T07:25:04.864955+0000","last_unstale":"2024-03-06T07:25:04.869925+0000","last_undegraded":"2024-03-06T07:25:04.869925+0000","last_fullsized":"2024-03-06T07:25:04.869925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:42.691950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857793+0000","last_change":"2024-03-06T07:25:04.857793+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857793+0000","last_undegraded":"2024-03-06T07:25:04.857793+0000","last_fullsized":"2024-03-06T07:25:04.857793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869331+0000","last_change":"2024-03-06T07:25:04.865095+0000","last_active":"2024-03-06T07:25:04.869331+0000","last_peered":"2024-03-06T07:25:04.869331+0000","last_clean":"2024-03-06T07:25:04.869331+0000","last_became_active":"2024-03-06T07:25:04.864783+0000","last_became_peered":"2024-03-06T07:25:04.864783+0000","last_unstale":"2024-03-06T07:25:04.869331+0000","last_undegraded":"2024-03-06T07:25:04.869331+0000","last_fullsized":"2024-03-06T07:25:04.869331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855883+0000","last_change":"2024-03-06T07:25:04.855883+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855883+0000","last_undegraded":"2024-03-06T07:25:04.855883+0000","last_fullsized":"2024-03-06T07:25:04.855883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869886+0000","last_change":"2024-03-06T07:25:04.866682+0000","last_active":"2024-03-06T07:25:04.869886+0000","last_peered":"2024-03-06T07:25:04.869886+0000","last_clean":"2024-03-06T07:25:04.869886+0000","last_became_active":"2024-03-06T07:25:04.866374+0000","last_became_peered":"2024-03-06T07:25:04.866374+0000","last_unstale":"2024-03-06T07:25:04.869886+0000","last_undegraded":"2024-03-06T07:25:04.869886+0000","last_fullsized":"2024-03-06T07:25:04.869886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:15.941972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858011+0000","last_change":"2024-03-06T07:25:04.858011+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858011+0000","last_undegraded":"2024-03-06T07:25:04.858011+0000","last_fullsized":"2024-03-06T07:25:04.858011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869185+0000","last_change":"2024-03-06T07:25:04.864465+0000","last_active":"2024-03-06T07:25:04.869185+0000","last_peered":"2024-03-06T07:25:04.869185+0000","last_clean":"2024-03-06T07:25:04.869185+0000","last_became_active":"2024-03-06T07:25:04.864206+0000","last_became_peered":"2024-03-06T07:25:04.864206+0000","last_unstale":"2024-03-06T07:25:04.869185+0000","last_undegraded":"2024-03-06T07:25:04.869185+0000","last_fullsized":"2024-03-06T07:25:04.869185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:29.767631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858764+0000","last_change":"2024-03-06T07:25:04.858764+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858764+0000","last_undegraded":"2024-03-06T07:25:04.858764+0000","last_fullsized":"2024-03-06T07:25:04.858764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863948+0000","last_change":"2024-03-06T07:25:04.859877+0000","last_active":"2024-03-06T07:25:04.863948+0000","last_peered":"2024-03-06T07:25:04.863948+0000","last_clean":"2024-03-06T07:25:04.863948+0000","last_became_active":"2024-03-06T07:25:04.859458+0000","last_became_peered":"2024-03-06T07:25:04.859458+0000","last_unstale":"2024-03-06T07:25:04.863948+0000","last_undegraded":"2024-03-06T07:25:04.863948+0000","last_fullsized":"2024-03-06T07:25:04.863948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:51.757108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860052+0000","last_change":"2024-03-06T07:25:04.860052+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860052+0000","last_undegraded":"2024-03-06T07:25:04.860052+0000","last_fullsized":"2024-03-06T07:25:04.860052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.871744+0000","last_change":"2024-03-06T07:25:04.869898+0000","last_active":"2024-03-06T07:25:04.871744+0000","last_peered":"2024-03-06T07:25:04.871744+0000","last_clean":"2024-03-06T07:25:04.871744+0000","last_became_active":"2024-03-06T07:25:04.869613+0000","last_became_peered":"2024-03-06T07:25:04.869613+0000","last_unstale":"2024-03-06T07:25:04.871744+0000","last_undegraded":"2024-03-06T07:25:04.871744+0000","last_fullsized":"2024-03-06T07:25:04.871744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:01.821292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853346+0000","last_change":"2024-03-06T07:25:04.853346+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853346+0000","last_undegraded":"2024-03-06T07:25:04.853346+0000","last_fullsized":"2024-03-06T07:25:04.853346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.864047+0000","last_change":"2024-03-06T07:25:04.860934+0000","last_active":"2024-03-06T07:25:04.864047+0000","last_peered":"2024-03-06T07:25:04.864047+0000","last_clean":"2024-03-06T07:25:04.864047+0000","last_became_active":"2024-03-06T07:25:04.860341+0000","last_became_peered":"2024-03-06T07:25:04.860341+0000","last_unstale":"2024-03-06T07:25:04.864047+0000","last_undegraded":"2024-03-06T07:25:04.864047+0000","last_fullsized":"2024-03-06T07:25:04.864047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:29.072866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853953+0000","last_change":"2024-03-06T07:25:04.853953+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853953+0000","last_undegraded":"2024-03-06T07:25:04.853953+0000","last_fullsized":"2024-03-06T07:25:04.853953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865333+0000","last_change":"2024-03-06T07:25:04.862667+0000","last_active":"2024-03-06T07:25:04.865333+0000","last_peered":"2024-03-06T07:25:04.865333+0000","last_clean":"2024-03-06T07:25:04.865333+0000","last_became_active":"2024-03-06T07:25:04.862428+0000","last_became_peered":"2024-03-06T07:25:04.862428+0000","last_unstale":"2024-03-06T07:25:04.865333+0000","last_undegraded":"2024-03-06T07:25:04.865333+0000","last_fullsized":"2024-03-06T07:25:04.865333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.676481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853833+0000","last_change":"2024-03-06T07:25:04.853833+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853833+0000","last_undegraded":"2024-03-06T07:25:04.853833+0000","last_fullsized":"2024-03-06T07:25:04.853833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872165+0000","last_change":"2024-03-06T07:25:04.866391+0000","last_active":"2024-03-06T07:25:04.872165+0000","last_peered":"2024-03-06T07:25:04.872165+0000","last_clean":"2024-03-06T07:25:04.872165+0000","last_became_active":"2024-03-06T07:25:04.866047+0000","last_became_peered":"2024-03-06T07:25:04.866047+0000","last_unstale":"2024-03-06T07:25:04.872165+0000","last_undegraded":"2024-03-06T07:25:04.872165+0000","last_fullsized":"2024-03-06T07:25:04.872165+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:13.057086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853896+0000","last_change":"2024-03-06T07:25:04.853896+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853896+0000","last_undegraded":"2024-03-06T07:25:04.853896+0000","last_fullsized":"2024-03-06T07:25:04.853896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870711+0000","last_change":"2024-03-06T07:25:04.868423+0000","last_active":"2024-03-06T07:25:04.870711+0000","last_peered":"2024-03-06T07:25:04.870711+0000","last_clean":"2024-03-06T07:25:04.870711+0000","last_became_active":"2024-03-06T07:25:04.868028+0000","last_became_peered":"2024-03-06T07:25:04.868028+0000","last_unstale":"2024-03-06T07:25:04.870711+0000","last_undegraded":"2024-03-06T07:25:04.870711+0000","last_fullsized":"2024-03-06T07:25:04.870711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:51.388578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859152+0000","last_change":"2024-03-06T07:25:04.859152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859152+0000","last_undegraded":"2024-03-06T07:25:04.859152+0000","last_fullsized":"2024-03-06T07:25:04.859152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869781+0000","last_change":"2024-03-06T07:25:04.867025+0000","last_active":"2024-03-06T07:25:04.869781+0000","last_peered":"2024-03-06T07:25:04.869781+0000","last_clean":"2024-03-06T07:25:04.869781+0000","last_became_active":"2024-03-06T07:25:04.866493+0000","last_became_peered":"2024-03-06T07:25:04.866493+0000","last_unstale":"2024-03-06T07:25:04.869781+0000","last_undegraded":"2024-03-06T07:25:04.869781+0000","last_fullsized":"2024-03-06T07:25:04.869781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:05.098447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.857899+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:04.857899+0000","last_peered":"2024-03-06T07:25:04.857899+0000","last_clean":"2024-03-06T07:25:04.857899+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:04.857899+0000","last_undegraded":"2024-03-06T07:25:04.857899+0000","last_fullsized":"2024-03-06T07:25:04.857899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":929430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":923155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:09.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2024-03-06T07:25:09.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858078+0000', 'last_change': '2024-03-06T07:25:04.858078+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858078+0000', 'last_undegraded': '2024-03-06T07:25:04.858078+0000', 'last_fullsized': '2024-03-06T07:25:04.858078+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2024-03-06T07:25:09.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853918+0000', 'last_change': '2024-03-06T07:25:04.853918+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853918+0000', 'last_undegraded': '2024-03-06T07:25:04.853918+0000', 'last_fullsized': '2024-03-06T07:25:04.853918+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2024-03-06T07:25:09.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856216+0000', 'last_change': '2024-03-06T07:25:04.856216+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856216+0000', 'last_undegraded': '2024-03-06T07:25:04.856216+0000', 'last_fullsized': '2024-03-06T07:25:04.856216+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855094+0000', 'last_change': '2024-03-06T07:25:04.855094+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855094+0000', 'last_undegraded': '2024-03-06T07:25:04.855094+0000', 'last_fullsized': '2024-03-06T07:25:04.855094+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857926+0000', 'last_change': '2024-03-06T07:25:04.857926+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857926+0000', 'last_undegraded': '2024-03-06T07:25:04.857926+0000', 'last_fullsized': '2024-03-06T07:25:04.857926+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855162+0000', 'last_change': '2024-03-06T07:25:04.855162+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855162+0000', 'last_undegraded': '2024-03-06T07:25:04.855162+0000', 'last_fullsized': '2024-03-06T07:25:04.855162+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.39 is not active+clean 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859124+0000', 'last_change': '2024-03-06T07:25:04.859124+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859124+0000', 'last_undegraded': '2024-03-06T07:25:04.859124+0000', 'last_fullsized': '2024-03-06T07:25:04.859124+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2024-03-06T07:25:09.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854959+0000', 'last_change': '2024-03-06T07:25:04.854959+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854959+0000', 'last_undegraded': '2024-03-06T07:25:04.854959+0000', 'last_fullsized': '2024-03-06T07:25:04.854959+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857036+0000', 'last_change': '2024-03-06T07:25:04.857036+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857036+0000', 'last_undegraded': '2024-03-06T07:25:04.857036+0000', 'last_fullsized': '2024-03-06T07:25:04.857036+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858672+0000', 'last_change': '2024-03-06T07:25:04.858672+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858672+0000', 'last_undegraded': '2024-03-06T07:25:04.858672+0000', 'last_fullsized': '2024-03-06T07:25:04.858672+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.860968+0000', 'last_change': '2024-03-06T07:25:04.860968+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.860968+0000', 'last_undegraded': '2024-03-06T07:25:04.860968+0000', 'last_fullsized': '2024-03-06T07:25:04.860968+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854645+0000', 'last_change': '2024-03-06T07:25:04.854645+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854645+0000', 'last_undegraded': '2024-03-06T07:25:04.854645+0000', 'last_fullsized': '2024-03-06T07:25:04.854645+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855112+0000', 'last_change': '2024-03-06T07:25:04.855112+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855112+0000', 'last_undegraded': '2024-03-06T07:25:04.855112+0000', 'last_fullsized': '2024-03-06T07:25:04.855112+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:09.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856602+0000', 'last_change': '2024-03-06T07:25:04.856602+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856602+0000', 'last_undegraded': '2024-03-06T07:25:04.856602+0000', 'last_fullsized': '2024-03-06T07:25:04.856602+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854677+0000', 'last_change': '2024-03-06T07:25:04.854677+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854677+0000', 'last_undegraded': '2024-03-06T07:25:04.854677+0000', 'last_fullsized': '2024-03-06T07:25:04.854677+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858851+0000', 'last_change': '2024-03-06T07:25:04.858851+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858851+0000', 'last_undegraded': '2024-03-06T07:25:04.858851+0000', 'last_fullsized': '2024-03-06T07:25:04.858851+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856578+0000', 'last_change': '2024-03-06T07:25:04.856578+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856578+0000', 'last_undegraded': '2024-03-06T07:25:04.856578+0000', 'last_fullsized': '2024-03-06T07:25:04.856578+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855526+0000', 'last_change': '2024-03-06T07:25:04.855526+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855526+0000', 'last_undegraded': '2024-03-06T07:25:04.855526+0000', 'last_fullsized': '2024-03-06T07:25:04.855526+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2024-03-06T07:25:09.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854605+0000', 'last_change': '2024-03-06T07:25:04.854605+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854605+0000', 'last_undegraded': '2024-03-06T07:25:04.854605+0000', 'last_fullsized': '2024-03-06T07:25:04.854605+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856470+0000', 'last_change': '2024-03-06T07:25:04.856470+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856470+0000', 'last_undegraded': '2024-03-06T07:25:04.856470+0000', 'last_fullsized': '2024-03-06T07:25:04.856470+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.860123+0000', 'last_change': '2024-03-06T07:25:04.860123+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.860123+0000', 'last_undegraded': '2024-03-06T07:25:04.860123+0000', 'last_fullsized': '2024-03-06T07:25:04.860123+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858357+0000', 'last_change': '2024-03-06T07:25:04.858357+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858357+0000', 'last_undegraded': '2024-03-06T07:25:04.858357+0000', 'last_fullsized': '2024-03-06T07:25:04.858357+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858229+0000', 'last_change': '2024-03-06T07:25:04.858229+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858229+0000', 'last_undegraded': '2024-03-06T07:25:04.858229+0000', 'last_fullsized': '2024-03-06T07:25:04.858229+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2024-03-06T07:25:09.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857673+0000', 'last_change': '2024-03-06T07:25:04.857673+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857673+0000', 'last_undegraded': '2024-03-06T07:25:04.857673+0000', 'last_fullsized': '2024-03-06T07:25:04.857673+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.861040+0000', 'last_change': '2024-03-06T07:25:04.861040+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.861040+0000', 'last_undegraded': '2024-03-06T07:25:04.861040+0000', 'last_fullsized': '2024-03-06T07:25:04.861040+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.26 is not active+clean 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859134+0000', 'last_change': '2024-03-06T07:25:04.859134+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859134+0000', 'last_undegraded': '2024-03-06T07:25:04.859134+0000', 'last_fullsized': '2024-03-06T07:25:04.859134+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855782+0000', 'last_change': '2024-03-06T07:25:04.855782+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855782+0000', 'last_undegraded': '2024-03-06T07:25:04.855782+0000', 'last_fullsized': '2024-03-06T07:25:04.855782+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.861710+0000', 'last_change': '2024-03-06T07:25:04.861710+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.861710+0000', 'last_undegraded': '2024-03-06T07:25:04.861710+0000', 'last_fullsized': '2024-03-06T07:25:04.861710+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859725+0000', 'last_change': '2024-03-06T07:25:04.859725+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859725+0000', 'last_undegraded': '2024-03-06T07:25:04.859725+0000', 'last_fullsized': '2024-03-06T07:25:04.859725+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.22 is not active+clean 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857152+0000', 'last_change': '2024-03-06T07:25:04.857152+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857152+0000', 'last_undegraded': '2024-03-06T07:25:04.857152+0000', 'last_fullsized': '2024-03-06T07:25:04.857152+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.21 is not active+clean 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856377+0000', 'last_change': '2024-03-06T07:25:04.856377+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856377+0000', 'last_undegraded': '2024-03-06T07:25:04.856377+0000', 'last_fullsized': '2024-03-06T07:25:04.856377+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854396+0000', 'last_change': '2024-03-06T07:25:04.854396+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854396+0000', 'last_undegraded': '2024-03-06T07:25:04.854396+0000', 'last_fullsized': '2024-03-06T07:25:04.854396+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859683+0000', 'last_change': '2024-03-06T07:25:04.859683+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859683+0000', 'last_undegraded': '2024-03-06T07:25:04.859683+0000', 'last_fullsized': '2024-03-06T07:25:04.859683+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853353+0000', 'last_change': '2024-03-06T07:25:04.853353+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853353+0000', 'last_undegraded': '2024-03-06T07:25:04.853353+0000', 'last_fullsized': '2024-03-06T07:25:04.853353+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-03-06T07:25:09.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856020+0000', 'last_change': '2024-03-06T07:25:04.856020+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856020+0000', 'last_undegraded': '2024-03-06T07:25:04.856020+0000', 'last_fullsized': '2024-03-06T07:25:04.856020+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.860599+0000', 'last_change': '2024-03-06T07:25:04.860599+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.860599+0000', 'last_undegraded': '2024-03-06T07:25:04.860599+0000', 'last_fullsized': '2024-03-06T07:25:04.860599+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856363+0000', 'last_change': '2024-03-06T07:25:04.856363+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856363+0000', 'last_undegraded': '2024-03-06T07:25:04.856363+0000', 'last_fullsized': '2024-03-06T07:25:04.856363+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859683+0000', 'last_change': '2024-03-06T07:25:04.859683+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859683+0000', 'last_undegraded': '2024-03-06T07:25:04.859683+0000', 'last_fullsized': '2024-03-06T07:25:04.859683+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854255+0000', 'last_change': '2024-03-06T07:25:04.854255+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854255+0000', 'last_undegraded': '2024-03-06T07:25:04.854255+0000', 'last_fullsized': '2024-03-06T07:25:04.854255+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-03-06T07:25:09.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854239+0000', 'last_change': '2024-03-06T07:25:04.854239+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854239+0000', 'last_undegraded': '2024-03-06T07:25:04.854239+0000', 'last_fullsized': '2024-03-06T07:25:04.854239+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857051+0000', 'last_change': '2024-03-06T07:25:04.857051+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857051+0000', 'last_undegraded': '2024-03-06T07:25:04.857051+0000', 'last_fullsized': '2024-03-06T07:25:04.857051+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857731+0000', 'last_change': '2024-03-06T07:25:04.857731+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857731+0000', 'last_undegraded': '2024-03-06T07:25:04.857731+0000', 'last_fullsized': '2024-03-06T07:25:04.857731+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855499+0000', 'last_change': '2024-03-06T07:25:04.855499+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855499+0000', 'last_undegraded': '2024-03-06T07:25:04.855499+0000', 'last_fullsized': '2024-03-06T07:25:04.855499+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854484+0000', 'last_change': '2024-03-06T07:25:04.854484+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854484+0000', 'last_undegraded': '2024-03-06T07:25:04.854484+0000', 'last_fullsized': '2024-03-06T07:25:04.854484+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858713+0000', 'last_change': '2024-03-06T07:25:04.858713+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858713+0000', 'last_undegraded': '2024-03-06T07:25:04.858713+0000', 'last_fullsized': '2024-03-06T07:25:04.858713+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-03-06T07:25:09.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857221+0000', 'last_change': '2024-03-06T07:25:04.857221+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857221+0000', 'last_undegraded': '2024-03-06T07:25:04.857221+0000', 'last_fullsized': '2024-03-06T07:25:04.857221+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.856137+0000', 'last_change': '2024-03-06T07:25:04.856137+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.856137+0000', 'last_undegraded': '2024-03-06T07:25:04.856137+0000', 'last_fullsized': '2024-03-06T07:25:04.856137+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853796+0000', 'last_change': '2024-03-06T07:25:04.853796+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853796+0000', 'last_undegraded': '2024-03-06T07:25:04.853796+0000', 'last_fullsized': '2024-03-06T07:25:04.853796+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858111+0000', 'last_change': '2024-03-06T07:25:04.858111+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858111+0000', 'last_undegraded': '2024-03-06T07:25:04.858111+0000', 'last_fullsized': '2024-03-06T07:25:04.858111+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857270+0000', 'last_change': '2024-03-06T07:25:04.857270+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857270+0000', 'last_undegraded': '2024-03-06T07:25:04.857270+0000', 'last_fullsized': '2024-03-06T07:25:04.857270+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-03-06T07:25:09.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854259+0000', 'last_change': '2024-03-06T07:25:04.854259+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854259+0000', 'last_undegraded': '2024-03-06T07:25:04.854259+0000', 'last_fullsized': '2024-03-06T07:25:04.854259+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.854014+0000', 'last_change': '2024-03-06T07:25:04.854014+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.854014+0000', 'last_undegraded': '2024-03-06T07:25:04.854014+0000', 'last_fullsized': '2024-03-06T07:25:04.854014+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859370+0000', 'last_change': '2024-03-06T07:25:04.859370+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859370+0000', 'last_undegraded': '2024-03-06T07:25:04.859370+0000', 'last_fullsized': '2024-03-06T07:25:04.859370+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855390+0000', 'last_change': '2024-03-06T07:25:04.855390+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855390+0000', 'last_undegraded': '2024-03-06T07:25:04.855390+0000', 'last_fullsized': '2024-03-06T07:25:04.855390+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.857793+0000', 'last_change': '2024-03-06T07:25:04.857793+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.857793+0000', 'last_undegraded': '2024-03-06T07:25:04.857793+0000', 'last_fullsized': '2024-03-06T07:25:04.857793+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-03-06T07:25:09.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.855883+0000', 'last_change': '2024-03-06T07:25:04.855883+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.855883+0000', 'last_undegraded': '2024-03-06T07:25:04.855883+0000', 'last_fullsized': '2024-03-06T07:25:04.855883+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858011+0000', 'last_change': '2024-03-06T07:25:04.858011+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858011+0000', 'last_undegraded': '2024-03-06T07:25:04.858011+0000', 'last_fullsized': '2024-03-06T07:25:04.858011+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.858764+0000', 'last_change': '2024-03-06T07:25:04.858764+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.858764+0000', 'last_undegraded': '2024-03-06T07:25:04.858764+0000', 'last_fullsized': '2024-03-06T07:25:04.858764+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.860052+0000', 'last_change': '2024-03-06T07:25:04.860052+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.860052+0000', 'last_undegraded': '2024-03-06T07:25:04.860052+0000', 'last_fullsized': '2024-03-06T07:25:04.860052+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853346+0000', 'last_change': '2024-03-06T07:25:04.853346+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853346+0000', 'last_undegraded': '2024-03-06T07:25:04.853346+0000', 'last_fullsized': '2024-03-06T07:25:04.853346+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853953+0000', 'last_change': '2024-03-06T07:25:04.853953+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853953+0000', 'last_undegraded': '2024-03-06T07:25:04.853953+0000', 'last_fullsized': '2024-03-06T07:25:04.853953+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-03-06T07:25:09.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853833+0000', 'last_change': '2024-03-06T07:25:04.853833+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853833+0000', 'last_undegraded': '2024-03-06T07:25:04.853833+0000', 'last_fullsized': '2024-03-06T07:25:04.853833+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:09.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-03-06T07:25:09.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.853896+0000', 'last_change': '2024-03-06T07:25:04.853896+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.853896+0000', 'last_undegraded': '2024-03-06T07:25:04.853896+0000', 'last_fullsized': '2024-03-06T07:25:04.853896+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:09.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-03-06T07:25:09.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:04.859152+0000', 'last_change': '2024-03-06T07:25:04.859152+0000', 'last_active': '2024-03-06T07:25:04.847107+0000', 'last_peered': '2024-03-06T07:25:04.847107+0000', 'last_clean': '2024-03-06T07:25:04.847107+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:04.859152+0000', 'last_undegraded': '2024-03-06T07:25:04.859152+0000', 'last_fullsized': '2024-03-06T07:25:04.859152+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:04.847107+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:09.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:09.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.774+0000 7f5c83440640 1 Processor -- start 2024-03-06T07:25:09.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.774+0000 7f5c83440640 1 -- start start 2024-03-06T07:25:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c83440640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c7c0b4a50 0x7f5c7c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c83440640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c06e770 con 0x7f5c7c0cc920 2024-03-06T07:25:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c83440640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c06e490 con 0x7f5c7c06e000 2024-03-06T07:25:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c83440640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7c15e660 con 0x7f5c7c0b4a50 2024-03-06T07:25:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c819dd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c7c0b4a50 0x7f5c7c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c811dc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5c7c0cc920 0x7f5c7c072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36550/0 (socket says 172.21.15.122:36550) 2024-03-06T07:25:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c811dc640 1 -- 172.21.15.122:0/3074531255 learned_addr learned my addr 172.21.15.122:0/3074531255 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c819dd640 1 -- 172.21.15.122:0/3074531255 >> v1:172.21.15.169:6789/0 conn(0x7f5c7c06e000 legacy=0x7f5c7c150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c819dd640 1 -- 172.21.15.122:0/3074531255 >> v1:172.21.15.122:6789/0 conn(0x7f5c7c0cc920 legacy=0x7f5c7c072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.775+0000 7f5c819dd640 1 -- 172.21.15.122:0/3074531255 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c15e010 con 0x7f5c7c0b4a50 2024-03-06T07:25:09.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.776+0000 7f5c819dd640 1 --2- 172.21.15.122:0/3074531255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0b4a50 0x7f5c7c15eaf0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c15e7e0 tx=0x7f5c7402bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab7b8789c83562a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.776+0000 7f5c73fff640 1 -- 172.21.15.122:0/3074531255 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74039070 con 0x7f5c7c0b4a50 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.776+0000 7f5c73fff640 1 -- 172.21.15.122:0/3074531255 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7402ca10 con 0x7f5c7c0b4a50 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.776+0000 7f5c73fff640 1 -- 172.21.15.122:0/3074531255 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7402cde0 con 0x7f5c7c0b4a50 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 -- 172.21.15.122:0/3074531255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0b4a50 msgr2=0x7f5c7c15eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 --2- 172.21.15.122:0/3074531255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0b4a50 0x7f5c7c15eaf0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c15e7e0 tx=0x7f5c7402bce0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 -- 172.21.15.122:0/3074531255 shutdown_connections 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 --2- 172.21.15.122:0/3074531255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0b4a50 0x7f5c7c15eaf0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 -- 172.21.15.122:0/3074531255 >> 172.21.15.122:0/3074531255 conn(0x7f5c7c0b1c80 msgr2=0x7f5c7c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 -- 172.21.15.122:0/3074531255 shutdown_connections 2024-03-06T07:25:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.777+0000 7f5c83440640 1 -- 172.21.15.122:0/3074531255 wait complete. 2024-03-06T07:25:09.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.779+0000 7f5c83440640 1 Processor -- start 2024-03-06T07:25:09.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.779+0000 7f5c83440640 1 -- start start 2024-03-06T07:25:09.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c83440640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0b4a50 0x7f5c7c156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c83440640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 0x7f5c7c149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c83440640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c15a810 0x7f5c7c15aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c83440640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0cce80 con 0x7f5c7c0cc920 2024-03-06T07:25:09.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c83440640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0ccd00 con 0x7f5c7c15a810 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c83440640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c070940 con 0x7f5c7c0b4a50 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 0x7f5c7c149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 0x7f5c7c149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47940/0 (socket says 172.21.15.122:47940) 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 -- 172.21.15.122:0/2143586217 learned_addr learned my addr 172.21.15.122:0/2143586217 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c811dc640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0b4a50 0x7f5c7c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c819dd640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c15a810 0x7f5c7c15aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 -- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0b4a50 msgr2=0x7f5c7c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0b4a50 0x7f5c7c156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 -- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c15a810 msgr2=0x7f5c7c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c15a810 0x7f5c7c15aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.780+0000 7f5c809db640 1 -- 172.21.15.122:0/2143586217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c1f3510 con 0x7f5c7c0cc920 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.781+0000 7f5c809db640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 0x7f5c7c149410 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c400 tx=0x7f5c6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.781+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c011790 con 0x7f5c7c0cc920 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.781+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c017070 con 0x7f5c7c0cc920 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.781+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7c1f37f0 con 0x7f5c7c0cc920 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.781+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c011bb0 con 0x7f5c7c0cc920 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.781+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7c1f3db0 con 0x7f5c7c0cc920 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.782+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7c04f330 con 0x7f5c7c0cc920 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.782+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c6c01c460 con 0x7f5c7c0cc920 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.783+0000 7f5c71ffb640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.783+0000 7f5c811dc640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.783+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c6c08a0a0 con 0x7f5c7c0cc920 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.783+0000 7f5c811dc640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5c78007fd0 tx=0x7f5c78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:09.787+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c6c058f20 con 0x7f5c7c0cc920 2024-03-06T07:25:10.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.033+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5c7c0b40d0 con 0x7f5c7c0cc920 2024-03-06T07:25:10.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.033+0000 7f5c71ffb640 1 -- 172.21.15.122:0/2143586217 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7f5c6c058820 con 0x7f5c7c0cc920 2024-03-06T07:25:10.035 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:10.035 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":69,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":263110656,"bytes_avail":772831002624,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":33,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6600}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:22:47.341990+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:10.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 msgr2=0x7f5c6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5c78007fd0 tx=0x7f5c78009040 comp rx=0 tx=0).stop 2024-03-06T07:25:10.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 msgr2=0x7f5c7c149410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 0x7f5c7c149410 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c400 tx=0x7f5c6c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 shutdown_connections 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c15a810 0x7f5c7c15aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c0cc920 0x7f5c7c149410 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 --2- 172.21.15.122:0/2143586217 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0b4a50 0x7f5c7c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 >> 172.21.15.122:0/2143586217 conn(0x7f5c7c0b1c80 msgr2=0x7f5c7c1530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 shutdown_connections 2024-03-06T07:25:10.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.035+0000 7f5c83440640 1 -- 172.21.15.122:0/2143586217 wait complete. 2024-03-06T07:25:10.048 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:10.048 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:10.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa016bc9640 1 Processor -- start 2024-03-06T07:25:10.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa016bc9640 1 -- start start 2024-03-06T07:25:10.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa016bc9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa0100d22b0 0x7fa0100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa016bc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0100b3430 con 0x7fa0100b41f0 2024-03-06T07:25:10.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa016bc9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0100b3710 con 0x7fa01015dfe0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa016bc9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100d2c50 con 0x7fa0100d22b0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.168+0000 7fa015166640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa0100b41f0 0x7fa0100b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36552/0 (socket says 172.21.15.122:36552) 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa015166640 1 -- 172.21.15.122:0/3270189373 learned_addr learned my addr 172.21.15.122:0/3270189373 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00ffff640 1 --2- 172.21.15.122:0/3270189373 >> v2:172.21.15.169:3300/0 conn(0x7fa0100d22b0 0x7fa0100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321025216 0 0) 0x7fa0100b3430 con 0x7fa0100b41f0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa000003520 con 0x7fa0100b41f0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245830216 0 0) 0x7fa000003520 con 0x7fa0100b41f0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 >> v2:172.21.15.169:3300/0 conn(0x7fa0100d22b0 msgr2=0x7fa0100d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 --2- 172.21.15.122:0/3270189373 >> v2:172.21.15.169:3300/0 conn(0x7fa0100d22b0 0x7fa0100d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 >> v1:172.21.15.169:6789/0 conn(0x7fa01015dfe0 legacy=0x7fa0100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100b39f0 con 0x7fa0100b41f0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa0080019f0 con 0x7fa0100b41f0 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 >> v1:172.21.15.122:6789/0 conn(0x7fa0100b41f0 legacy=0x7fa0100b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0000039b0 0x7fa000003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa000004800 0x7fa000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa00000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c030 con 0x7fa0000051c0 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c1e0 con 0x7fa000004800 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c3e0 con 0x7fa0000039b0 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa015166640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa00000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa014965640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0000039b0 0x7fa000003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.169+0000 7fa00ffff640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa000004800 0x7fa000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 -- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0000039b0 msgr2=0x7fa000003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0000039b0 0x7fa000003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 -- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa000004800 msgr2=0x7fa000004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa000004800 0x7fa000004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 -- 172.21.15.122:0/3270189373 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000003520 con 0x7fa0000051c0 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 -- 172.21.15.122:0/3270189373 reap_dead start 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.170+0000 7fa015166640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa00000b9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa00800a860 tx=0x7fa000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a271c74b2df8c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00800a480 con 0x7fa0000051c0 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa008034630 con 0x7fa0000051c0 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa00f7fe640 1 -- 172.21.15.122:0/3270189373 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008034a20 con 0x7fa0000051c0 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 -- 172.21.15.122:0/3270189373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 msgr2=0x7fa00000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa00000b9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa00800a860 tx=0x7fa000005830 comp rx=0 tx=0).stop 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 -- 172.21.15.122:0/3270189373 shutdown_connections 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 --2- 172.21.15.122:0/3270189373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa00000b9c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 -- 172.21.15.122:0/3270189373 >> 172.21.15.122:0/3270189373 conn(0x7fa0100d3cd0 msgr2=0x7fa0100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 -- 172.21.15.122:0/3270189373 shutdown_connections 2024-03-06T07:25:10.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.171+0000 7fa016bc9640 1 -- 172.21.15.122:0/3270189373 wait complete. 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.172+0000 7fa016bc9640 1 Processor -- start 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.172+0000 7fa016bc9640 1 -- start start 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa016bc9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa0100cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa016bc9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa01015fff0 0x7fa010160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa016bc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0101609b0 0x7fa010160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa016bc9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b3d60 con 0x7fa0000051c0 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa016bc9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b31e0 con 0x7fa01015fff0 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa016bc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01015e3c0 con 0x7fa0101609b0 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa014965640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa0100cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa014965640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa0100cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47960/0 (socket says 172.21.15.122:47960) 2024-03-06T07:25:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa014965640 1 -- 172.21.15.122:0/1101186991 learned_addr learned my addr 172.21.15.122:0/1101186991 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa00ffff640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa01015fff0 0x7fa010160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.173+0000 7fa015166640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0101609b0 0x7fa010160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa014965640 1 -- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0101609b0 msgr2=0x7fa010160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa014965640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0101609b0 0x7fa010160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa014965640 1 -- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa01015fff0 msgr2=0x7fa010160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa014965640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa01015fff0 0x7fa010160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa014965640 1 -- 172.21.15.122:0/1101186991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100b9250 con 0x7fa0000051c0 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa014965640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa0100cb910 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa00400f620 tx=0x7fa00400fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0040052e0 con 0x7fa0000051c0 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004018070 con 0x7fa0000051c0 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0100b8f70 con 0x7fa0000051c0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004003810 con 0x7fa0000051c0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.174+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01015f7f0 con 0x7fa0000051c0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.175+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01004f330 con 0x7fa0000051c0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.175+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0040039b0 con 0x7fa0000051c0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.176+0000 7fa00d7fa640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.176+0000 7fa00ffff640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.176+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa004089e60 con 0x7fa0000051c0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.176+0000 7fa00ffff640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ff406b4d0 0x7f9ff406d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa0100cd070 tx=0x7f9ffc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.180+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa004058d70 con 0x7fa0000051c0 2024-03-06T07:25:10.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.354+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0100d0bb0 con 0x7f9ff406b4d0 2024-03-06T07:25:10.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.360+0000 7fa00d7fa640 1 -- 172.21.15.122:0/1101186991 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7fa0100d0bb0 con 0x7f9ff406b4d0 2024-03-06T07:25:10.403 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:10.405 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ff406b4d0 msgr2=0x7f9ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ff406b4d0 0x7f9ff406d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa0100cd070 tx=0x7f9ffc009190 comp rx=0 tx=0).stop 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 msgr2=0x7fa0100cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa0100cb910 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa00400f620 tx=0x7fa00400fa20 comp rx=0 tx=0).stop 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 shutdown_connections 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0101609b0 0x7fa010160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa01015fff0 0x7fa010160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 --2- 172.21.15.122:0/1101186991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0000051c0 0x7fa0100cb910 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 >> 172.21.15.122:0/1101186991 conn(0x7fa0100d3cd0 msgr2=0x7fa01006c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 shutdown_connections 2024-03-06T07:25:10.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:10.363+0000 7fa016bc9640 1 -- 172.21.15.122:0/1101186991 wait complete. 2024-03-06T07:25:10.408 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-03-06T07:25:09.370347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256944,"kb_used_data":7480,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717776,"statfs":{"total":773094113280,"available":772831002624,"internally_reserved":0,"allocated":7659520,"data_stored":3868132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.522808"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858078+0000","last_change":"2024-03-06T07:25:04.858078+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858078+0000","last_undegraded":"2024-03-06T07:25:04.858078+0000","last_fullsized":"2024-03-06T07:25:04.858078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853918+0000","last_change":"2024-03-06T07:25:04.853918+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853918+0000","last_undegraded":"2024-03-06T07:25:04.853918+0000","last_fullsized":"2024-03-06T07:25:04.853918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856216+0000","last_change":"2024-03-06T07:25:04.856216+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856216+0000","last_undegraded":"2024-03-06T07:25:04.856216+0000","last_fullsized":"2024-03-06T07:25:04.856216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855094+0000","last_change":"2024-03-06T07:25:04.855094+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855094+0000","last_undegraded":"2024-03-06T07:25:04.855094+0000","last_fullsized":"2024-03-06T07:25:04.855094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857926+0000","last_change":"2024-03-06T07:25:04.857926+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857926+0000","last_undegraded":"2024-03-06T07:25:04.857926+0000","last_fullsized":"2024-03-06T07:25:04.857926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855162+0000","last_change":"2024-03-06T07:25:04.855162+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855162+0000","last_undegraded":"2024-03-06T07:25:04.855162+0000","last_fullsized":"2024-03-06T07:25:04.855162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859124+0000","last_change":"2024-03-06T07:25:04.859124+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859124+0000","last_undegraded":"2024-03-06T07:25:04.859124+0000","last_fullsized":"2024-03-06T07:25:04.859124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854959+0000","last_change":"2024-03-06T07:25:04.854959+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854959+0000","last_undegraded":"2024-03-06T07:25:04.854959+0000","last_fullsized":"2024-03-06T07:25:04.854959+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857036+0000","last_change":"2024-03-06T07:25:04.857036+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857036+0000","last_undegraded":"2024-03-06T07:25:04.857036+0000","last_fullsized":"2024-03-06T07:25:04.857036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858672+0000","last_change":"2024-03-06T07:25:04.858672+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858672+0000","last_undegraded":"2024-03-06T07:25:04.858672+0000","last_fullsized":"2024-03-06T07:25:04.858672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860968+0000","last_change":"2024-03-06T07:25:04.860968+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860968+0000","last_undegraded":"2024-03-06T07:25:04.860968+0000","last_fullsized":"2024-03-06T07:25:04.860968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854645+0000","last_change":"2024-03-06T07:25:04.854645+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854645+0000","last_undegraded":"2024-03-06T07:25:04.854645+0000","last_fullsized":"2024-03-06T07:25:04.854645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855112+0000","last_change":"2024-03-06T07:25:04.855112+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855112+0000","last_undegraded":"2024-03-06T07:25:04.855112+0000","last_fullsized":"2024-03-06T07:25:04.855112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856602+0000","last_change":"2024-03-06T07:25:04.856602+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856602+0000","last_undegraded":"2024-03-06T07:25:04.856602+0000","last_fullsized":"2024-03-06T07:25:04.856602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854677+0000","last_change":"2024-03-06T07:25:04.854677+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854677+0000","last_undegraded":"2024-03-06T07:25:04.854677+0000","last_fullsized":"2024-03-06T07:25:04.854677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858851+0000","last_change":"2024-03-06T07:25:04.858851+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858851+0000","last_undegraded":"2024-03-06T07:25:04.858851+0000","last_fullsized":"2024-03-06T07:25:04.858851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856578+0000","last_change":"2024-03-06T07:25:04.856578+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856578+0000","last_undegraded":"2024-03-06T07:25:04.856578+0000","last_fullsized":"2024-03-06T07:25:04.856578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855526+0000","last_change":"2024-03-06T07:25:04.855526+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855526+0000","last_undegraded":"2024-03-06T07:25:04.855526+0000","last_fullsized":"2024-03-06T07:25:04.855526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854605+0000","last_change":"2024-03-06T07:25:04.854605+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854605+0000","last_undegraded":"2024-03-06T07:25:04.854605+0000","last_fullsized":"2024-03-06T07:25:04.854605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856470+0000","last_change":"2024-03-06T07:25:04.856470+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856470+0000","last_undegraded":"2024-03-06T07:25:04.856470+0000","last_fullsized":"2024-03-06T07:25:04.856470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860123+0000","last_change":"2024-03-06T07:25:04.860123+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860123+0000","last_undegraded":"2024-03-06T07:25:04.860123+0000","last_fullsized":"2024-03-06T07:25:04.860123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858357+0000","last_change":"2024-03-06T07:25:04.858357+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858357+0000","last_undegraded":"2024-03-06T07:25:04.858357+0000","last_fullsized":"2024-03-06T07:25:04.858357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858229+0000","last_change":"2024-03-06T07:25:04.858229+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858229+0000","last_undegraded":"2024-03-06T07:25:04.858229+0000","last_fullsized":"2024-03-06T07:25:04.858229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857673+0000","last_change":"2024-03-06T07:25:04.857673+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857673+0000","last_undegraded":"2024-03-06T07:25:04.857673+0000","last_fullsized":"2024-03-06T07:25:04.857673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861040+0000","last_change":"2024-03-06T07:25:04.861040+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861040+0000","last_undegraded":"2024-03-06T07:25:04.861040+0000","last_fullsized":"2024-03-06T07:25:04.861040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859134+0000","last_change":"2024-03-06T07:25:04.859134+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859134+0000","last_undegraded":"2024-03-06T07:25:04.859134+0000","last_fullsized":"2024-03-06T07:25:04.859134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855782+0000","last_change":"2024-03-06T07:25:04.855782+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855782+0000","last_undegraded":"2024-03-06T07:25:04.855782+0000","last_fullsized":"2024-03-06T07:25:04.855782+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.861710+0000","last_change":"2024-03-06T07:25:04.861710+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.861710+0000","last_undegraded":"2024-03-06T07:25:04.861710+0000","last_fullsized":"2024-03-06T07:25:04.861710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859725+0000","last_change":"2024-03-06T07:25:04.859725+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859725+0000","last_undegraded":"2024-03-06T07:25:04.859725+0000","last_fullsized":"2024-03-06T07:25:04.859725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857152+0000","last_change":"2024-03-06T07:25:04.857152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857152+0000","last_undegraded":"2024-03-06T07:25:04.857152+0000","last_fullsized":"2024-03-06T07:25:04.857152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856377+0000","last_change":"2024-03-06T07:25:04.856377+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856377+0000","last_undegraded":"2024-03-06T07:25:04.856377+0000","last_fullsized":"2024-03-06T07:25:04.856377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854396+0000","last_change":"2024-03-06T07:25:04.854396+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854396+0000","last_undegraded":"2024-03-06T07:25:04.854396+0000","last_fullsized":"2024-03-06T07:25:04.854396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870547+0000","last_change":"2024-03-06T07:25:04.867874+0000","last_active":"2024-03-06T07:25:04.870547+0000","last_peered":"2024-03-06T07:25:04.870547+0000","last_clean":"2024-03-06T07:25:04.870547+0000","last_became_active":"2024-03-06T07:25:04.867619+0000","last_became_peered":"2024-03-06T07:25:04.867619+0000","last_unstale":"2024-03-06T07:25:04.870547+0000","last_undegraded":"2024-03-06T07:25:04.870547+0000","last_fullsized":"2024-03-06T07:25:04.870547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853353+0000","last_change":"2024-03-06T07:25:04.853353+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853353+0000","last_undegraded":"2024-03-06T07:25:04.853353+0000","last_fullsized":"2024-03-06T07:25:04.853353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865840+0000","last_change":"2024-03-06T07:25:04.864438+0000","last_active":"2024-03-06T07:25:04.865840+0000","last_peered":"2024-03-06T07:25:04.865840+0000","last_clean":"2024-03-06T07:25:04.865840+0000","last_became_active":"2024-03-06T07:25:04.864180+0000","last_became_peered":"2024-03-06T07:25:04.864180+0000","last_unstale":"2024-03-06T07:25:04.865840+0000","last_undegraded":"2024-03-06T07:25:04.865840+0000","last_fullsized":"2024-03-06T07:25:04.865840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856020+0000","last_change":"2024-03-06T07:25:04.856020+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856020+0000","last_undegraded":"2024-03-06T07:25:04.856020+0000","last_fullsized":"2024-03-06T07:25:04.856020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872656+0000","last_change":"2024-03-06T07:25:04.871271+0000","last_active":"2024-03-06T07:25:04.872656+0000","last_peered":"2024-03-06T07:25:04.872656+0000","last_clean":"2024-03-06T07:25:04.872656+0000","last_became_active":"2024-03-06T07:25:04.870981+0000","last_became_peered":"2024-03-06T07:25:04.870981+0000","last_unstale":"2024-03-06T07:25:04.872656+0000","last_undegraded":"2024-03-06T07:25:04.872656+0000","last_fullsized":"2024-03-06T07:25:04.872656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:21.893351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860599+0000","last_change":"2024-03-06T07:25:04.860599+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860599+0000","last_undegraded":"2024-03-06T07:25:04.860599+0000","last_fullsized":"2024-03-06T07:25:04.860599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869854+0000","last_change":"2024-03-06T07:25:04.868531+0000","last_active":"2024-03-06T07:25:04.869854+0000","last_peered":"2024-03-06T07:25:04.869854+0000","last_clean":"2024-03-06T07:25:04.869854+0000","last_became_active":"2024-03-06T07:25:04.868198+0000","last_became_peered":"2024-03-06T07:25:04.868198+0000","last_unstale":"2024-03-06T07:25:04.869854+0000","last_undegraded":"2024-03-06T07:25:04.869854+0000","last_fullsized":"2024-03-06T07:25:04.869854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:56.138604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856363+0000","last_change":"2024-03-06T07:25:04.856363+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856363+0000","last_undegraded":"2024-03-06T07:25:04.856363+0000","last_fullsized":"2024-03-06T07:25:04.856363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869200+0000","last_change":"2024-03-06T07:25:04.867446+0000","last_active":"2024-03-06T07:25:04.869200+0000","last_peered":"2024-03-06T07:25:04.869200+0000","last_clean":"2024-03-06T07:25:04.869200+0000","last_became_active":"2024-03-06T07:25:04.867198+0000","last_became_peered":"2024-03-06T07:25:04.867198+0000","last_unstale":"2024-03-06T07:25:04.869200+0000","last_undegraded":"2024-03-06T07:25:04.869200+0000","last_fullsized":"2024-03-06T07:25:04.869200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:32.313600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859683+0000","last_change":"2024-03-06T07:25:04.859683+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859683+0000","last_undegraded":"2024-03-06T07:25:04.859683+0000","last_fullsized":"2024-03-06T07:25:04.859683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869433+0000","last_change":"2024-03-06T07:25:04.862460+0000","last_active":"2024-03-06T07:25:04.869433+0000","last_peered":"2024-03-06T07:25:04.869433+0000","last_clean":"2024-03-06T07:25:04.869433+0000","last_became_active":"2024-03-06T07:25:04.862179+0000","last_became_peered":"2024-03-06T07:25:04.862179+0000","last_unstale":"2024-03-06T07:25:04.869433+0000","last_undegraded":"2024-03-06T07:25:04.869433+0000","last_fullsized":"2024-03-06T07:25:04.869433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:04.876240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854255+0000","last_change":"2024-03-06T07:25:04.854255+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854255+0000","last_undegraded":"2024-03-06T07:25:04.854255+0000","last_fullsized":"2024-03-06T07:25:04.854255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865330+0000","last_change":"2024-03-06T07:25:04.862049+0000","last_active":"2024-03-06T07:25:04.865330+0000","last_peered":"2024-03-06T07:25:04.865330+0000","last_clean":"2024-03-06T07:25:04.865330+0000","last_became_active":"2024-03-06T07:25:04.861815+0000","last_became_peered":"2024-03-06T07:25:04.861815+0000","last_unstale":"2024-03-06T07:25:04.865330+0000","last_undegraded":"2024-03-06T07:25:04.865330+0000","last_fullsized":"2024-03-06T07:25:04.865330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:30.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854239+0000","last_change":"2024-03-06T07:25:04.854239+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854239+0000","last_undegraded":"2024-03-06T07:25:04.854239+0000","last_fullsized":"2024-03-06T07:25:04.854239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863920+0000","last_change":"2024-03-06T07:25:04.859156+0000","last_active":"2024-03-06T07:25:04.863920+0000","last_peered":"2024-03-06T07:25:04.863920+0000","last_clean":"2024-03-06T07:25:04.863920+0000","last_became_active":"2024-03-06T07:25:04.858589+0000","last_became_peered":"2024-03-06T07:25:04.858589+0000","last_unstale":"2024-03-06T07:25:04.863920+0000","last_undegraded":"2024-03-06T07:25:04.863920+0000","last_fullsized":"2024-03-06T07:25:04.863920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:46.223486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857051+0000","last_change":"2024-03-06T07:25:04.857051+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857051+0000","last_undegraded":"2024-03-06T07:25:04.857051+0000","last_fullsized":"2024-03-06T07:25:04.857051+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872201+0000","last_change":"2024-03-06T07:25:04.866854+0000","last_active":"2024-03-06T07:25:04.872201+0000","last_peered":"2024-03-06T07:25:04.872201+0000","last_clean":"2024-03-06T07:25:04.872201+0000","last_became_active":"2024-03-06T07:25:04.866566+0000","last_became_peered":"2024-03-06T07:25:04.866566+0000","last_unstale":"2024-03-06T07:25:04.872201+0000","last_undegraded":"2024-03-06T07:25:04.872201+0000","last_fullsized":"2024-03-06T07:25:04.872201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:50.439569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857731+0000","last_change":"2024-03-06T07:25:04.857731+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857731+0000","last_undegraded":"2024-03-06T07:25:04.857731+0000","last_fullsized":"2024-03-06T07:25:04.857731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869677+0000","last_change":"2024-03-06T07:25:04.862756+0000","last_active":"2024-03-06T07:25:04.869677+0000","last_peered":"2024-03-06T07:25:04.869677+0000","last_clean":"2024-03-06T07:25:04.869677+0000","last_became_active":"2024-03-06T07:25:04.862463+0000","last_became_peered":"2024-03-06T07:25:04.862463+0000","last_unstale":"2024-03-06T07:25:04.869677+0000","last_undegraded":"2024-03-06T07:25:04.869677+0000","last_fullsized":"2024-03-06T07:25:04.869677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:10.409474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855499+0000","last_change":"2024-03-06T07:25:04.855499+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855499+0000","last_undegraded":"2024-03-06T07:25:04.855499+0000","last_fullsized":"2024-03-06T07:25:04.855499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.866505+0000","last_change":"2024-03-06T07:25:04.864989+0000","last_active":"2024-03-06T07:25:04.866505+0000","last_peered":"2024-03-06T07:25:04.866505+0000","last_clean":"2024-03-06T07:25:04.866505+0000","last_became_active":"2024-03-06T07:25:04.864724+0000","last_became_peered":"2024-03-06T07:25:04.864724+0000","last_unstale":"2024-03-06T07:25:04.866505+0000","last_undegraded":"2024-03-06T07:25:04.866505+0000","last_fullsized":"2024-03-06T07:25:04.866505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:13.895735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854484+0000","last_change":"2024-03-06T07:25:04.854484+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854484+0000","last_undegraded":"2024-03-06T07:25:04.854484+0000","last_fullsized":"2024-03-06T07:25:04.854484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.867439+0000","last_change":"2024-03-06T07:25:04.866085+0000","last_active":"2024-03-06T07:25:04.867439+0000","last_peered":"2024-03-06T07:25:04.867439+0000","last_clean":"2024-03-06T07:25:04.867439+0000","last_became_active":"2024-03-06T07:25:04.865800+0000","last_became_peered":"2024-03-06T07:25:04.865800+0000","last_unstale":"2024-03-06T07:25:04.867439+0000","last_undegraded":"2024-03-06T07:25:04.867439+0000","last_fullsized":"2024-03-06T07:25:04.867439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:02.525342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858713+0000","last_change":"2024-03-06T07:25:04.858713+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858713+0000","last_undegraded":"2024-03-06T07:25:04.858713+0000","last_fullsized":"2024-03-06T07:25:04.858713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.868697+0000","last_change":"2024-03-06T07:25:04.866818+0000","last_active":"2024-03-06T07:25:04.868697+0000","last_peered":"2024-03-06T07:25:04.868697+0000","last_clean":"2024-03-06T07:25:04.868697+0000","last_became_active":"2024-03-06T07:25:04.866510+0000","last_became_peered":"2024-03-06T07:25:04.866510+0000","last_unstale":"2024-03-06T07:25:04.868697+0000","last_undegraded":"2024-03-06T07:25:04.868697+0000","last_fullsized":"2024-03-06T07:25:04.868697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.279615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857221+0000","last_change":"2024-03-06T07:25:04.857221+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857221+0000","last_undegraded":"2024-03-06T07:25:04.857221+0000","last_fullsized":"2024-03-06T07:25:04.857221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870181+0000","last_change":"2024-03-06T07:25:04.868290+0000","last_active":"2024-03-06T07:25:04.870181+0000","last_peered":"2024-03-06T07:25:04.870181+0000","last_clean":"2024-03-06T07:25:04.870181+0000","last_became_active":"2024-03-06T07:25:04.867100+0000","last_became_peered":"2024-03-06T07:25:04.867100+0000","last_unstale":"2024-03-06T07:25:04.870181+0000","last_undegraded":"2024-03-06T07:25:04.870181+0000","last_fullsized":"2024-03-06T07:25:04.870181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:24.492810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.856137+0000","last_change":"2024-03-06T07:25:04.856137+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.856137+0000","last_undegraded":"2024-03-06T07:25:04.856137+0000","last_fullsized":"2024-03-06T07:25:04.856137+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872446+0000","last_change":"2024-03-06T07:25:04.870029+0000","last_active":"2024-03-06T07:25:04.872446+0000","last_peered":"2024-03-06T07:25:04.872446+0000","last_clean":"2024-03-06T07:25:04.872446+0000","last_became_active":"2024-03-06T07:25:04.869729+0000","last_became_peered":"2024-03-06T07:25:04.869729+0000","last_unstale":"2024-03-06T07:25:04.872446+0000","last_undegraded":"2024-03-06T07:25:04.872446+0000","last_fullsized":"2024-03-06T07:25:04.872446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:36.772850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853796+0000","last_change":"2024-03-06T07:25:04.853796+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853796+0000","last_undegraded":"2024-03-06T07:25:04.853796+0000","last_fullsized":"2024-03-06T07:25:04.853796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872524+0000","last_change":"2024-03-06T07:25:04.869373+0000","last_active":"2024-03-06T07:25:04.872524+0000","last_peered":"2024-03-06T07:25:04.872524+0000","last_clean":"2024-03-06T07:25:04.872524+0000","last_became_active":"2024-03-06T07:25:04.869036+0000","last_became_peered":"2024-03-06T07:25:04.869036+0000","last_unstale":"2024-03-06T07:25:04.872524+0000","last_undegraded":"2024-03-06T07:25:04.872524+0000","last_fullsized":"2024-03-06T07:25:04.872524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858111+0000","last_change":"2024-03-06T07:25:04.858111+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858111+0000","last_undegraded":"2024-03-06T07:25:04.858111+0000","last_fullsized":"2024-03-06T07:25:04.858111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869904+0000","last_change":"2024-03-06T07:25:04.867842+0000","last_active":"2024-03-06T07:25:04.869904+0000","last_peered":"2024-03-06T07:25:04.869904+0000","last_clean":"2024-03-06T07:25:04.869904+0000","last_became_active":"2024-03-06T07:25:04.867564+0000","last_became_peered":"2024-03-06T07:25:04.867564+0000","last_unstale":"2024-03-06T07:25:04.869904+0000","last_undegraded":"2024-03-06T07:25:04.869904+0000","last_fullsized":"2024-03-06T07:25:04.869904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.953333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857270+0000","last_change":"2024-03-06T07:25:04.857270+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857270+0000","last_undegraded":"2024-03-06T07:25:04.857270+0000","last_fullsized":"2024-03-06T07:25:04.857270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872606+0000","last_change":"2024-03-06T07:25:04.870677+0000","last_active":"2024-03-06T07:25:04.872606+0000","last_peered":"2024-03-06T07:25:04.872606+0000","last_clean":"2024-03-06T07:25:04.872606+0000","last_became_active":"2024-03-06T07:25:04.870423+0000","last_became_peered":"2024-03-06T07:25:04.870423+0000","last_unstale":"2024-03-06T07:25:04.872606+0000","last_undegraded":"2024-03-06T07:25:04.872606+0000","last_fullsized":"2024-03-06T07:25:04.872606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:01.989975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854259+0000","last_change":"2024-03-06T07:25:04.854259+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854259+0000","last_undegraded":"2024-03-06T07:25:04.854259+0000","last_fullsized":"2024-03-06T07:25:04.854259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872449+0000","last_change":"2024-03-06T07:25:04.869978+0000","last_active":"2024-03-06T07:25:04.872449+0000","last_peered":"2024-03-06T07:25:04.872449+0000","last_clean":"2024-03-06T07:25:04.872449+0000","last_became_active":"2024-03-06T07:25:04.869501+0000","last_became_peered":"2024-03-06T07:25:04.869501+0000","last_unstale":"2024-03-06T07:25:04.872449+0000","last_undegraded":"2024-03-06T07:25:04.872449+0000","last_fullsized":"2024-03-06T07:25:04.872449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:59.249650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.854014+0000","last_change":"2024-03-06T07:25:04.854014+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.854014+0000","last_undegraded":"2024-03-06T07:25:04.854014+0000","last_fullsized":"2024-03-06T07:25:04.854014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872463+0000","last_change":"2024-03-06T07:25:04.870140+0000","last_active":"2024-03-06T07:25:04.872463+0000","last_peered":"2024-03-06T07:25:04.872463+0000","last_clean":"2024-03-06T07:25:04.872463+0000","last_became_active":"2024-03-06T07:25:04.869889+0000","last_became_peered":"2024-03-06T07:25:04.869889+0000","last_unstale":"2024-03-06T07:25:04.872463+0000","last_undegraded":"2024-03-06T07:25:04.872463+0000","last_fullsized":"2024-03-06T07:25:04.872463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859370+0000","last_change":"2024-03-06T07:25:04.859370+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859370+0000","last_undegraded":"2024-03-06T07:25:04.859370+0000","last_fullsized":"2024-03-06T07:25:04.859370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869412+0000","last_change":"2024-03-06T07:25:04.861518+0000","last_active":"2024-03-06T07:25:04.869412+0000","last_peered":"2024-03-06T07:25:04.869412+0000","last_clean":"2024-03-06T07:25:04.869412+0000","last_became_active":"2024-03-06T07:25:04.861183+0000","last_became_peered":"2024-03-06T07:25:04.861183+0000","last_unstale":"2024-03-06T07:25:04.869412+0000","last_undegraded":"2024-03-06T07:25:04.869412+0000","last_fullsized":"2024-03-06T07:25:04.869412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:40.600405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855390+0000","last_change":"2024-03-06T07:25:04.855390+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855390+0000","last_undegraded":"2024-03-06T07:25:04.855390+0000","last_fullsized":"2024-03-06T07:25:04.855390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869925+0000","last_change":"2024-03-06T07:25:04.865215+0000","last_active":"2024-03-06T07:25:04.869925+0000","last_peered":"2024-03-06T07:25:04.869925+0000","last_clean":"2024-03-06T07:25:04.869925+0000","last_became_active":"2024-03-06T07:25:04.864955+0000","last_became_peered":"2024-03-06T07:25:04.864955+0000","last_unstale":"2024-03-06T07:25:04.869925+0000","last_undegraded":"2024-03-06T07:25:04.869925+0000","last_fullsized":"2024-03-06T07:25:04.869925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:42.691950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.857793+0000","last_change":"2024-03-06T07:25:04.857793+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.857793+0000","last_undegraded":"2024-03-06T07:25:04.857793+0000","last_fullsized":"2024-03-06T07:25:04.857793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869331+0000","last_change":"2024-03-06T07:25:04.865095+0000","last_active":"2024-03-06T07:25:04.869331+0000","last_peered":"2024-03-06T07:25:04.869331+0000","last_clean":"2024-03-06T07:25:04.869331+0000","last_became_active":"2024-03-06T07:25:04.864783+0000","last_became_peered":"2024-03-06T07:25:04.864783+0000","last_unstale":"2024-03-06T07:25:04.869331+0000","last_undegraded":"2024-03-06T07:25:04.869331+0000","last_fullsized":"2024-03-06T07:25:04.869331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:43.542074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.855883+0000","last_change":"2024-03-06T07:25:04.855883+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.855883+0000","last_undegraded":"2024-03-06T07:25:04.855883+0000","last_fullsized":"2024-03-06T07:25:04.855883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869886+0000","last_change":"2024-03-06T07:25:04.866682+0000","last_active":"2024-03-06T07:25:04.869886+0000","last_peered":"2024-03-06T07:25:04.869886+0000","last_clean":"2024-03-06T07:25:04.869886+0000","last_became_active":"2024-03-06T07:25:04.866374+0000","last_became_peered":"2024-03-06T07:25:04.866374+0000","last_unstale":"2024-03-06T07:25:04.869886+0000","last_undegraded":"2024-03-06T07:25:04.869886+0000","last_fullsized":"2024-03-06T07:25:04.869886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:15.941972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858011+0000","last_change":"2024-03-06T07:25:04.858011+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858011+0000","last_undegraded":"2024-03-06T07:25:04.858011+0000","last_fullsized":"2024-03-06T07:25:04.858011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869185+0000","last_change":"2024-03-06T07:25:04.864465+0000","last_active":"2024-03-06T07:25:04.869185+0000","last_peered":"2024-03-06T07:25:04.869185+0000","last_clean":"2024-03-06T07:25:04.869185+0000","last_became_active":"2024-03-06T07:25:04.864206+0000","last_became_peered":"2024-03-06T07:25:04.864206+0000","last_unstale":"2024-03-06T07:25:04.869185+0000","last_undegraded":"2024-03-06T07:25:04.869185+0000","last_fullsized":"2024-03-06T07:25:04.869185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:29.767631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.858764+0000","last_change":"2024-03-06T07:25:04.858764+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.858764+0000","last_undegraded":"2024-03-06T07:25:04.858764+0000","last_fullsized":"2024-03-06T07:25:04.858764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.863948+0000","last_change":"2024-03-06T07:25:04.859877+0000","last_active":"2024-03-06T07:25:04.863948+0000","last_peered":"2024-03-06T07:25:04.863948+0000","last_clean":"2024-03-06T07:25:04.863948+0000","last_became_active":"2024-03-06T07:25:04.859458+0000","last_became_peered":"2024-03-06T07:25:04.859458+0000","last_unstale":"2024-03-06T07:25:04.863948+0000","last_undegraded":"2024-03-06T07:25:04.863948+0000","last_fullsized":"2024-03-06T07:25:04.863948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:51.757108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.860052+0000","last_change":"2024-03-06T07:25:04.860052+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.860052+0000","last_undegraded":"2024-03-06T07:25:04.860052+0000","last_fullsized":"2024-03-06T07:25:04.860052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.871744+0000","last_change":"2024-03-06T07:25:04.869898+0000","last_active":"2024-03-06T07:25:04.871744+0000","last_peered":"2024-03-06T07:25:04.871744+0000","last_clean":"2024-03-06T07:25:04.871744+0000","last_became_active":"2024-03-06T07:25:04.869613+0000","last_became_peered":"2024-03-06T07:25:04.869613+0000","last_unstale":"2024-03-06T07:25:04.871744+0000","last_undegraded":"2024-03-06T07:25:04.871744+0000","last_fullsized":"2024-03-06T07:25:04.871744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:01.821292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853346+0000","last_change":"2024-03-06T07:25:04.853346+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853346+0000","last_undegraded":"2024-03-06T07:25:04.853346+0000","last_fullsized":"2024-03-06T07:25:04.853346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.864047+0000","last_change":"2024-03-06T07:25:04.860934+0000","last_active":"2024-03-06T07:25:04.864047+0000","last_peered":"2024-03-06T07:25:04.864047+0000","last_clean":"2024-03-06T07:25:04.864047+0000","last_became_active":"2024-03-06T07:25:04.860341+0000","last_became_peered":"2024-03-06T07:25:04.860341+0000","last_unstale":"2024-03-06T07:25:04.864047+0000","last_undegraded":"2024-03-06T07:25:04.864047+0000","last_fullsized":"2024-03-06T07:25:04.864047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:29.072866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853953+0000","last_change":"2024-03-06T07:25:04.853953+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853953+0000","last_undegraded":"2024-03-06T07:25:04.853953+0000","last_fullsized":"2024-03-06T07:25:04.853953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.865333+0000","last_change":"2024-03-06T07:25:04.862667+0000","last_active":"2024-03-06T07:25:04.865333+0000","last_peered":"2024-03-06T07:25:04.865333+0000","last_clean":"2024-03-06T07:25:04.865333+0000","last_became_active":"2024-03-06T07:25:04.862428+0000","last_became_peered":"2024-03-06T07:25:04.862428+0000","last_unstale":"2024-03-06T07:25:04.865333+0000","last_undegraded":"2024-03-06T07:25:04.865333+0000","last_fullsized":"2024-03-06T07:25:04.865333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.676481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853833+0000","last_change":"2024-03-06T07:25:04.853833+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853833+0000","last_undegraded":"2024-03-06T07:25:04.853833+0000","last_fullsized":"2024-03-06T07:25:04.853833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.872165+0000","last_change":"2024-03-06T07:25:04.866391+0000","last_active":"2024-03-06T07:25:04.872165+0000","last_peered":"2024-03-06T07:25:04.872165+0000","last_clean":"2024-03-06T07:25:04.872165+0000","last_became_active":"2024-03-06T07:25:04.866047+0000","last_became_peered":"2024-03-06T07:25:04.866047+0000","last_unstale":"2024-03-06T07:25:04.872165+0000","last_undegraded":"2024-03-06T07:25:04.872165+0000","last_fullsized":"2024-03-06T07:25:04.872165+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:13.057086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.853896+0000","last_change":"2024-03-06T07:25:04.853896+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.853896+0000","last_undegraded":"2024-03-06T07:25:04.853896+0000","last_fullsized":"2024-03-06T07:25:04.853896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.870711+0000","last_change":"2024-03-06T07:25:04.868423+0000","last_active":"2024-03-06T07:25:04.870711+0000","last_peered":"2024-03-06T07:25:04.870711+0000","last_clean":"2024-03-06T07:25:04.870711+0000","last_became_active":"2024-03-06T07:25:04.868028+0000","last_became_peered":"2024-03-06T07:25:04.868028+0000","last_unstale":"2024-03-06T07:25:04.870711+0000","last_undegraded":"2024-03-06T07:25:04.870711+0000","last_fullsized":"2024-03-06T07:25:04.870711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:51.388578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:25:04.859152+0000","last_change":"2024-03-06T07:25:04.859152+0000","last_active":"2024-03-06T07:25:04.847107+0000","last_peered":"2024-03-06T07:25:04.847107+0000","last_clean":"2024-03-06T07:25:04.847107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:04.859152+0000","last_undegraded":"2024-03-06T07:25:04.859152+0000","last_fullsized":"2024-03-06T07:25:04.859152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.869781+0000","last_change":"2024-03-06T07:25:04.867025+0000","last_active":"2024-03-06T07:25:04.869781+0000","last_peered":"2024-03-06T07:25:04.869781+0000","last_clean":"2024-03-06T07:25:04.869781+0000","last_became_active":"2024-03-06T07:25:04.866493+0000","last_became_peered":"2024-03-06T07:25:04.866493+0000","last_unstale":"2024-03-06T07:25:04.869781+0000","last_undegraded":"2024-03-06T07:25:04.869781+0000","last_fullsized":"2024-03-06T07:25:04.869781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:05.098447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:25:04.857899+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:04.857899+0000","last_peered":"2024-03-06T07:25:04.857899+0000","last_clean":"2024-03-06T07:25:04.857899+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:04.857899+0000","last_undegraded":"2024-03-06T07:25:04.857899+0000","last_fullsized":"2024-03-06T07:25:04.857899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":929430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":339062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":923155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:11.135 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:25:11.133+0000 7f72aa740640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:13.412 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:13.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.511+0000 7f299eb25640 1 Processor -- start 2024-03-06T07:25:13.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f299eb25640 1 -- start start 2024-03-06T07:25:13.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f299eb25640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29980d22b0 0x7f29980d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f299eb25640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980b41f0 0x7f29980b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f299eb25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29980b3430 con 0x7f299815dfe0 2024-03-06T07:25:13.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f299eb25640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29980d2c50 con 0x7f29980d22b0 2024-03-06T07:25:13.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f299eb25640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29980b31e0 con 0x7f29980b41f0 2024-03-06T07:25:13.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.512+0000 7f2997fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29980d22b0 0x7f29980d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f299c8c1640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f299815dfe0 0x7f29980cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36562/0 (socket says 172.21.15.122:36562) 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f2997fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29980d22b0 0x7f29980d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47976/0 (socket says 172.21.15.122:47976) 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f299c8c1640 1 -- 172.21.15.122:0/3726209507 learned_addr learned my addr 172.21.15.122:0/3726209507 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f299d0c2640 1 --2- 172.21.15.122:0/3726209507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980b41f0 0x7f29980b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f2997fff640 1 -- 172.21.15.122:0/3726209507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980b41f0 msgr2=0x7f29980b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f2997fff640 1 --2- 172.21.15.122:0/3726209507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980b41f0 0x7f29980b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f2997fff640 1 -- 172.21.15.122:0/3726209507 >> v1:172.21.15.122:6789/0 conn(0x7f299815dfe0 legacy=0x7f29980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.513+0000 7f2997fff640 1 -- 172.21.15.122:0/3726209507 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29980b3710 con 0x7f29980d22b0 2024-03-06T07:25:13.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f2997fff640 1 --2- 172.21.15.122:0/3726209507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29980d22b0 0x7f29980d2710 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f298c0097a0 tx=0x7f298c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c30df864882df5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f29977fe640 1 -- 172.21.15.122:0/3726209507 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c035020 con 0x7f29980d22b0 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f29977fe640 1 -- 172.21.15.122:0/3726209507 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c031830 con 0x7f29980d22b0 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f29977fe640 1 -- 172.21.15.122:0/3726209507 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c031c00 con 0x7f29980d22b0 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 -- 172.21.15.122:0/3726209507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29980d22b0 msgr2=0x7f29980d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 --2- 172.21.15.122:0/3726209507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29980d22b0 0x7f29980d2710 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f298c0097a0 tx=0x7f298c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 -- 172.21.15.122:0/3726209507 shutdown_connections 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 --2- 172.21.15.122:0/3726209507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980b41f0 0x7f29980b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 --2- 172.21.15.122:0/3726209507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29980d22b0 0x7f29980d2710 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 -- 172.21.15.122:0/3726209507 >> 172.21.15.122:0/3726209507 conn(0x7f29980d3cd0 msgr2=0x7f29980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 -- 172.21.15.122:0/3726209507 shutdown_connections 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.514+0000 7f299eb25640 1 -- 172.21.15.122:0/3726209507 wait complete. 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.515+0000 7f299eb25640 1 Processor -- start 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.515+0000 7f299eb25640 1 -- start start 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299eb25640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29980b41f0 0x7f29980b2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299eb25640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980d22b0 0x7f29980b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299eb25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 0x7f299815ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299eb25640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29980cfa00 con 0x7f299815dfe0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299eb25640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29980b3d60 con 0x7f29980b41f0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299eb25640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29980d1c70 con 0x7f29980d22b0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299d0c2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 0x7f299815ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299d0c2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 0x7f299815ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47984/0 (socket says 172.21.15.122:47984) 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.516+0000 7f299d0c2640 1 -- 172.21.15.122:0/124743766 learned_addr learned my addr 172.21.15.122:0/124743766 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f2997fff640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980d22b0 0x7f29980b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299c8c1640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29980b41f0 0x7f29980b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299d0c2640 1 -- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980d22b0 msgr2=0x7f29980b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299d0c2640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980d22b0 0x7f29980b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299d0c2640 1 -- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29980b41f0 msgr2=0x7f29980b2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299d0c2640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29980b41f0 0x7f29980b2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299d0c2640 1 -- 172.21.15.122:0/124743766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998161650 con 0x7f299815dfe0 2024-03-06T07:25:13.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299d0c2640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 0x7f299815ea70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f298800d800 tx=0x7f298800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298800b6c0 con 0x7f299815dfe0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2988019070 con 0x7f299815dfe0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2998161090 con 0x7f299815dfe0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298800bac0 con 0x7f299815dfe0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.517+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29980b1c50 con 0x7f299815dfe0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.518+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299804f330 con 0x7f299815dfe0 2024-03-06T07:25:13.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.519+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f298800bc60 con 0x7f299815dfe0 2024-03-06T07:25:13.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.519+0000 7f29957fa640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f298406b4d0 0x7f298406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.519+0000 7f299c8c1640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f298406b4d0 0x7f298406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.519+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f298808a600 con 0x7f299815dfe0 2024-03-06T07:25:13.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.520+0000 7f299c8c1640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f298406b4d0 0x7f298406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f29900012e0 tx=0x7f2990009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.523+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29880594f0 con 0x7f299815dfe0 2024-03-06T07:25:13.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.693+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29980b2f80 con 0x7f298406b4d0 2024-03-06T07:25:13.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.699+0000 7f29957fa640 1 -- 172.21.15.122:0/124743766 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f29980b2f80 con 0x7f298406b4d0 2024-03-06T07:25:13.703 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:13.705 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:13.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f298406b4d0 msgr2=0x7f298406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f298406b4d0 0x7f298406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f29900012e0 tx=0x7f2990009040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 msgr2=0x7f299815ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 0x7f299815ea70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f298800d800 tx=0x7f298800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 shutdown_connections 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f298406b4d0 0x7f298406d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f299815dfe0 0x7f299815ea70 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29980d22b0 0x7f29980b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 --2- 172.21.15.122:0/124743766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29980b41f0 0x7f29980b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 >> 172.21.15.122:0/124743766 conn(0x7f29980d3cd0 msgr2=0x7f299806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 shutdown_connections 2024-03-06T07:25:13.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.702+0000 7f299eb25640 1 -- 172.21.15.122:0/124743766 wait complete. 2024-03-06T07:25:13.715 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-03-06T07:25:13.371158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257232,"kb_used_data":7784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717488,"statfs":{"total":773094113280,"available":772830707712,"internally_reserved":0,"allocated":7970816,"data_stored":4002652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001511"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097785+0000","last_change":"2024-03-06T07:25:05.876276+0000","last_active":"2024-03-06T07:25:07.097785+0000","last_peered":"2024-03-06T07:25:07.097785+0000","last_clean":"2024-03-06T07:25:07.097785+0000","last_became_active":"2024-03-06T07:25:05.875985+0000","last_became_peered":"2024-03-06T07:25:05.875985+0000","last_unstale":"2024-03-06T07:25:07.097785+0000","last_undegraded":"2024-03-06T07:25:07.097785+0000","last_fullsized":"2024-03-06T07:25:07.097785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:21.622814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877146+0000","last_change":"2024-03-06T07:25:05.880330+0000","last_active":"2024-03-06T07:25:06.877146+0000","last_peered":"2024-03-06T07:25:06.877146+0000","last_clean":"2024-03-06T07:25:06.877146+0000","last_became_active":"2024-03-06T07:25:05.880081+0000","last_became_peered":"2024-03-06T07:25:05.880081+0000","last_unstale":"2024-03-06T07:25:06.877146+0000","last_undegraded":"2024-03-06T07:25:06.877146+0000","last_fullsized":"2024-03-06T07:25:06.877146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.585744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098042+0000","last_change":"2024-03-06T07:25:05.874708+0000","last_active":"2024-03-06T07:25:07.098042+0000","last_peered":"2024-03-06T07:25:07.098042+0000","last_clean":"2024-03-06T07:25:07.098042+0000","last_became_active":"2024-03-06T07:25:05.874439+0000","last_became_peered":"2024-03-06T07:25:05.874439+0000","last_unstale":"2024-03-06T07:25:07.098042+0000","last_undegraded":"2024-03-06T07:25:07.098042+0000","last_fullsized":"2024-03-06T07:25:07.098042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:30.770960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097740+0000","last_change":"2024-03-06T07:25:05.881199+0000","last_active":"2024-03-06T07:25:07.097740+0000","last_peered":"2024-03-06T07:25:07.097740+0000","last_clean":"2024-03-06T07:25:07.097740+0000","last_became_active":"2024-03-06T07:25:05.880939+0000","last_became_peered":"2024-03-06T07:25:05.880939+0000","last_unstale":"2024-03-06T07:25:07.097740+0000","last_undegraded":"2024-03-06T07:25:07.097740+0000","last_fullsized":"2024-03-06T07:25:07.097740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:41.260201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.875956+0000","last_change":"2024-03-06T07:25:05.867838+0000","last_active":"2024-03-06T07:25:06.875956+0000","last_peered":"2024-03-06T07:25:06.875956+0000","last_clean":"2024-03-06T07:25:06.875956+0000","last_became_active":"2024-03-06T07:25:05.867569+0000","last_became_peered":"2024-03-06T07:25:05.867569+0000","last_unstale":"2024-03-06T07:25:06.875956+0000","last_undegraded":"2024-03-06T07:25:06.875956+0000","last_fullsized":"2024-03-06T07:25:06.875956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:05.880465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877304+0000","last_change":"2024-03-06T07:25:05.878856+0000","last_active":"2024-03-06T07:25:06.877304+0000","last_peered":"2024-03-06T07:25:06.877304+0000","last_clean":"2024-03-06T07:25:06.877304+0000","last_became_active":"2024-03-06T07:25:05.878510+0000","last_became_peered":"2024-03-06T07:25:05.878510+0000","last_unstale":"2024-03-06T07:25:06.877304+0000","last_undegraded":"2024-03-06T07:25:06.877304+0000","last_fullsized":"2024-03-06T07:25:06.877304+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:30.771856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.879731+0000","last_change":"2024-03-06T07:25:05.881852+0000","last_active":"2024-03-06T07:25:06.879731+0000","last_peered":"2024-03-06T07:25:06.879731+0000","last_clean":"2024-03-06T07:25:06.879731+0000","last_became_active":"2024-03-06T07:25:05.881484+0000","last_became_peered":"2024-03-06T07:25:05.881484+0000","last_unstale":"2024-03-06T07:25:06.879731+0000","last_undegraded":"2024-03-06T07:25:06.879731+0000","last_fullsized":"2024-03-06T07:25:06.879731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:36.513462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876560+0000","last_change":"2024-03-06T07:25:05.875198+0000","last_active":"2024-03-06T07:25:06.876560+0000","last_peered":"2024-03-06T07:25:06.876560+0000","last_clean":"2024-03-06T07:25:06.876560+0000","last_became_active":"2024-03-06T07:25:05.874926+0000","last_became_peered":"2024-03-06T07:25:05.874926+0000","last_unstale":"2024-03-06T07:25:06.876560+0000","last_undegraded":"2024-03-06T07:25:06.876560+0000","last_fullsized":"2024-03-06T07:25:06.876560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:11.019151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876397+0000","last_change":"2024-03-06T07:25:05.873563+0000","last_active":"2024-03-06T07:25:06.876397+0000","last_peered":"2024-03-06T07:25:06.876397+0000","last_clean":"2024-03-06T07:25:06.876397+0000","last_became_active":"2024-03-06T07:25:05.873294+0000","last_became_peered":"2024-03-06T07:25:05.873294+0000","last_unstale":"2024-03-06T07:25:06.876397+0000","last_undegraded":"2024-03-06T07:25:06.876397+0000","last_fullsized":"2024-03-06T07:25:06.876397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:25.497035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877944+0000","last_change":"2024-03-06T07:25:05.879652+0000","last_active":"2024-03-06T07:25:06.877944+0000","last_peered":"2024-03-06T07:25:06.877944+0000","last_clean":"2024-03-06T07:25:06.877944+0000","last_became_active":"2024-03-06T07:25:05.879383+0000","last_became_peered":"2024-03-06T07:25:05.879383+0000","last_unstale":"2024-03-06T07:25:06.877944+0000","last_undegraded":"2024-03-06T07:25:06.877944+0000","last_fullsized":"2024-03-06T07:25:06.877944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:37.777075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097640+0000","last_change":"2024-03-06T07:25:05.874667+0000","last_active":"2024-03-06T07:25:07.097640+0000","last_peered":"2024-03-06T07:25:07.097640+0000","last_clean":"2024-03-06T07:25:07.097640+0000","last_became_active":"2024-03-06T07:25:05.874265+0000","last_became_peered":"2024-03-06T07:25:05.874265+0000","last_unstale":"2024-03-06T07:25:07.097640+0000","last_undegraded":"2024-03-06T07:25:07.097640+0000","last_fullsized":"2024-03-06T07:25:07.097640+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:56.953650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097949+0000","last_change":"2024-03-06T07:25:05.875648+0000","last_active":"2024-03-06T07:25:07.097949+0000","last_peered":"2024-03-06T07:25:07.097949+0000","last_clean":"2024-03-06T07:25:07.097949+0000","last_became_active":"2024-03-06T07:25:05.875171+0000","last_became_peered":"2024-03-06T07:25:05.875171+0000","last_unstale":"2024-03-06T07:25:07.097949+0000","last_undegraded":"2024-03-06T07:25:07.097949+0000","last_fullsized":"2024-03-06T07:25:07.097949+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:42.387647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882179+0000","last_change":"2024-03-06T07:25:05.877262+0000","last_active":"2024-03-06T07:25:06.882179+0000","last_peered":"2024-03-06T07:25:06.882179+0000","last_clean":"2024-03-06T07:25:06.882179+0000","last_became_active":"2024-03-06T07:25:05.876988+0000","last_became_peered":"2024-03-06T07:25:05.876988+0000","last_unstale":"2024-03-06T07:25:06.882179+0000","last_undegraded":"2024-03-06T07:25:06.882179+0000","last_fullsized":"2024-03-06T07:25:06.882179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:24.325879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.875976+0000","last_change":"2024-03-06T07:25:05.882032+0000","last_active":"2024-03-06T07:25:06.875976+0000","last_peered":"2024-03-06T07:25:06.875976+0000","last_clean":"2024-03-06T07:25:06.875976+0000","last_became_active":"2024-03-06T07:25:05.881760+0000","last_became_peered":"2024-03-06T07:25:05.881760+0000","last_unstale":"2024-03-06T07:25:06.875976+0000","last_undegraded":"2024-03-06T07:25:06.875976+0000","last_fullsized":"2024-03-06T07:25:06.875976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:11.413699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882041+0000","last_change":"2024-03-06T07:25:05.868909+0000","last_active":"2024-03-06T07:25:06.882041+0000","last_peered":"2024-03-06T07:25:06.882041+0000","last_clean":"2024-03-06T07:25:06.882041+0000","last_became_active":"2024-03-06T07:25:05.868656+0000","last_became_peered":"2024-03-06T07:25:05.868656+0000","last_unstale":"2024-03-06T07:25:06.882041+0000","last_undegraded":"2024-03-06T07:25:06.882041+0000","last_fullsized":"2024-03-06T07:25:06.882041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:20.118069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876291+0000","last_change":"2024-03-06T07:25:05.876454+0000","last_active":"2024-03-06T07:25:06.876291+0000","last_peered":"2024-03-06T07:25:06.876291+0000","last_clean":"2024-03-06T07:25:06.876291+0000","last_became_active":"2024-03-06T07:25:05.876183+0000","last_became_peered":"2024-03-06T07:25:05.876183+0000","last_unstale":"2024-03-06T07:25:06.876291+0000","last_undegraded":"2024-03-06T07:25:06.876291+0000","last_fullsized":"2024-03-06T07:25:06.876291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:29.516291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876460+0000","last_change":"2024-03-06T07:25:05.873014+0000","last_active":"2024-03-06T07:25:06.876460+0000","last_peered":"2024-03-06T07:25:06.876460+0000","last_clean":"2024-03-06T07:25:06.876460+0000","last_became_active":"2024-03-06T07:25:05.872733+0000","last_became_peered":"2024-03-06T07:25:05.872733+0000","last_unstale":"2024-03-06T07:25:06.876460+0000","last_undegraded":"2024-03-06T07:25:06.876460+0000","last_fullsized":"2024-03-06T07:25:06.876460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:52.392803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097838+0000","last_change":"2024-03-06T07:25:05.880312+0000","last_active":"2024-03-06T07:25:07.097838+0000","last_peered":"2024-03-06T07:25:07.097838+0000","last_clean":"2024-03-06T07:25:07.097838+0000","last_became_active":"2024-03-06T07:25:05.880019+0000","last_became_peered":"2024-03-06T07:25:05.880019+0000","last_unstale":"2024-03-06T07:25:07.097838+0000","last_undegraded":"2024-03-06T07:25:07.097838+0000","last_fullsized":"2024-03-06T07:25:07.097838+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:59.628998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876738+0000","last_change":"2024-03-06T07:25:05.875744+0000","last_active":"2024-03-06T07:25:06.876738+0000","last_peered":"2024-03-06T07:25:06.876738+0000","last_clean":"2024-03-06T07:25:06.876738+0000","last_became_active":"2024-03-06T07:25:05.875223+0000","last_became_peered":"2024-03-06T07:25:05.875223+0000","last_unstale":"2024-03-06T07:25:06.876738+0000","last_undegraded":"2024-03-06T07:25:06.876738+0000","last_fullsized":"2024-03-06T07:25:06.876738+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:05.028057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882541+0000","last_change":"2024-03-06T07:25:05.881915+0000","last_active":"2024-03-06T07:25:06.882541+0000","last_peered":"2024-03-06T07:25:06.882541+0000","last_clean":"2024-03-06T07:25:06.882541+0000","last_became_active":"2024-03-06T07:25:05.881587+0000","last_became_peered":"2024-03-06T07:25:05.881587+0000","last_unstale":"2024-03-06T07:25:06.882541+0000","last_undegraded":"2024-03-06T07:25:06.882541+0000","last_fullsized":"2024-03-06T07:25:06.882541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:33.317825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877087+0000","last_change":"2024-03-06T07:25:05.875459+0000","last_active":"2024-03-06T07:25:06.877087+0000","last_peered":"2024-03-06T07:25:06.877087+0000","last_clean":"2024-03-06T07:25:06.877087+0000","last_became_active":"2024-03-06T07:25:05.875160+0000","last_became_peered":"2024-03-06T07:25:05.875160+0000","last_unstale":"2024-03-06T07:25:06.877087+0000","last_undegraded":"2024-03-06T07:25:06.877087+0000","last_fullsized":"2024-03-06T07:25:06.877087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:17.127147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876685+0000","last_change":"2024-03-06T07:25:05.877402+0000","last_active":"2024-03-06T07:25:06.876685+0000","last_peered":"2024-03-06T07:25:06.876685+0000","last_clean":"2024-03-06T07:25:06.876685+0000","last_became_active":"2024-03-06T07:25:05.877051+0000","last_became_peered":"2024-03-06T07:25:05.877051+0000","last_unstale":"2024-03-06T07:25:06.876685+0000","last_undegraded":"2024-03-06T07:25:06.876685+0000","last_fullsized":"2024-03-06T07:25:06.876685+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.585744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097409+0000","last_change":"2024-03-06T07:25:05.884844+0000","last_active":"2024-03-06T07:25:07.097409+0000","last_peered":"2024-03-06T07:25:07.097409+0000","last_clean":"2024-03-06T07:25:07.097409+0000","last_became_active":"2024-03-06T07:25:05.884552+0000","last_became_peered":"2024-03-06T07:25:05.884552+0000","last_unstale":"2024-03-06T07:25:07.097409+0000","last_undegraded":"2024-03-06T07:25:07.097409+0000","last_fullsized":"2024-03-06T07:25:07.097409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:01.866419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097254+0000","last_change":"2024-03-06T07:25:05.872645+0000","last_active":"2024-03-06T07:25:07.097254+0000","last_peered":"2024-03-06T07:25:07.097254+0000","last_clean":"2024-03-06T07:25:07.097254+0000","last_became_active":"2024-03-06T07:25:05.872316+0000","last_became_peered":"2024-03-06T07:25:05.872316+0000","last_unstale":"2024-03-06T07:25:07.097254+0000","last_undegraded":"2024-03-06T07:25:07.097254+0000","last_fullsized":"2024-03-06T07:25:07.097254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:14.899961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.878105+0000","last_change":"2024-03-06T07:25:05.880157+0000","last_active":"2024-03-06T07:25:06.878105+0000","last_peered":"2024-03-06T07:25:06.878105+0000","last_clean":"2024-03-06T07:25:06.878105+0000","last_became_active":"2024-03-06T07:25:05.879796+0000","last_became_peered":"2024-03-06T07:25:05.879796+0000","last_unstale":"2024-03-06T07:25:06.878105+0000","last_undegraded":"2024-03-06T07:25:06.878105+0000","last_fullsized":"2024-03-06T07:25:06.878105+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:35.991918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.879963+0000","last_change":"2024-03-06T07:25:05.883649+0000","last_active":"2024-03-06T07:25:06.879963+0000","last_peered":"2024-03-06T07:25:06.879963+0000","last_clean":"2024-03-06T07:25:06.879963+0000","last_became_active":"2024-03-06T07:25:05.883388+0000","last_became_peered":"2024-03-06T07:25:05.883388+0000","last_unstale":"2024-03-06T07:25:06.879963+0000","last_undegraded":"2024-03-06T07:25:06.879963+0000","last_fullsized":"2024-03-06T07:25:06.879963+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:16.270387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098189+0000","last_change":"2024-03-06T07:25:05.869645+0000","last_active":"2024-03-06T07:25:07.098189+0000","last_peered":"2024-03-06T07:25:07.098189+0000","last_clean":"2024-03-06T07:25:07.098189+0000","last_became_active":"2024-03-06T07:25:05.869351+0000","last_became_peered":"2024-03-06T07:25:05.869351+0000","last_unstale":"2024-03-06T07:25:07.098189+0000","last_undegraded":"2024-03-06T07:25:07.098189+0000","last_fullsized":"2024-03-06T07:25:07.098189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:40.717163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877674+0000","last_change":"2024-03-06T07:25:05.881455+0000","last_active":"2024-03-06T07:25:06.877674+0000","last_peered":"2024-03-06T07:25:06.877674+0000","last_clean":"2024-03-06T07:25:06.877674+0000","last_became_active":"2024-03-06T07:25:05.881199+0000","last_became_peered":"2024-03-06T07:25:05.881199+0000","last_unstale":"2024-03-06T07:25:06.877674+0000","last_undegraded":"2024-03-06T07:25:06.877674+0000","last_fullsized":"2024-03-06T07:25:06.877674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:43.696175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098219+0000","last_change":"2024-03-06T07:25:05.885680+0000","last_active":"2024-03-06T07:25:07.098219+0000","last_peered":"2024-03-06T07:25:07.098219+0000","last_clean":"2024-03-06T07:25:07.098219+0000","last_became_active":"2024-03-06T07:25:05.885406+0000","last_became_peered":"2024-03-06T07:25:05.885406+0000","last_unstale":"2024-03-06T07:25:07.098219+0000","last_undegraded":"2024-03-06T07:25:07.098219+0000","last_fullsized":"2024-03-06T07:25:07.098219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:28.749351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.879831+0000","last_change":"2024-03-06T07:25:05.879982+0000","last_active":"2024-03-06T07:25:06.879831+0000","last_peered":"2024-03-06T07:25:06.879831+0000","last_clean":"2024-03-06T07:25:06.879831+0000","last_became_active":"2024-03-06T07:25:05.879700+0000","last_became_peered":"2024-03-06T07:25:05.879700+0000","last_unstale":"2024-03-06T07:25:06.879831+0000","last_undegraded":"2024-03-06T07:25:06.879831+0000","last_fullsized":"2024-03-06T07:25:06.879831+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:48.133499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.880131+0000","last_change":"2024-03-06T07:25:05.881742+0000","last_active":"2024-03-06T07:25:06.880131+0000","last_peered":"2024-03-06T07:25:06.880131+0000","last_clean":"2024-03-06T07:25:06.880131+0000","last_became_active":"2024-03-06T07:25:05.881421+0000","last_became_peered":"2024-03-06T07:25:05.881421+0000","last_unstale":"2024-03-06T07:25:06.880131+0000","last_undegraded":"2024-03-06T07:25:06.880131+0000","last_fullsized":"2024-03-06T07:25:06.880131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:14.899961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876751+0000","last_change":"2024-03-06T07:25:05.871062+0000","last_active":"2024-03-06T07:25:06.876751+0000","last_peered":"2024-03-06T07:25:06.876751+0000","last_clean":"2024-03-06T07:25:06.876751+0000","last_became_active":"2024-03-06T07:25:05.870802+0000","last_became_peered":"2024-03-06T07:25:05.870802+0000","last_unstale":"2024-03-06T07:25:06.876751+0000","last_undegraded":"2024-03-06T07:25:06.876751+0000","last_fullsized":"2024-03-06T07:25:06.876751+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:29.516291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097695+0000","last_change":"2024-03-06T07:25:05.874955+0000","last_active":"2024-03-06T07:25:07.097695+0000","last_peered":"2024-03-06T07:25:07.097695+0000","last_clean":"2024-03-06T07:25:07.097695+0000","last_became_active":"2024-03-06T07:25:05.874681+0000","last_became_peered":"2024-03-06T07:25:05.874681+0000","last_unstale":"2024-03-06T07:25:07.097695+0000","last_undegraded":"2024-03-06T07:25:07.097695+0000","last_fullsized":"2024-03-06T07:25:07.097695+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:38.504357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097927+0000","last_change":"2024-03-06T07:25:04.867874+0000","last_active":"2024-03-06T07:25:07.097927+0000","last_peered":"2024-03-06T07:25:07.097927+0000","last_clean":"2024-03-06T07:25:07.097927+0000","last_became_active":"2024-03-06T07:25:04.867619+0000","last_became_peered":"2024-03-06T07:25:04.867619+0000","last_unstale":"2024-03-06T07:25:07.097927+0000","last_undegraded":"2024-03-06T07:25:07.097927+0000","last_fullsized":"2024-03-06T07:25:07.097927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:30.978788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876531+0000","last_change":"2024-03-06T07:25:05.872603+0000","last_active":"2024-03-06T07:25:06.876531+0000","last_peered":"2024-03-06T07:25:06.876531+0000","last_clean":"2024-03-06T07:25:06.876531+0000","last_became_active":"2024-03-06T07:25:05.872312+0000","last_became_peered":"2024-03-06T07:25:05.872312+0000","last_unstale":"2024-03-06T07:25:06.876531+0000","last_undegraded":"2024-03-06T07:25:06.876531+0000","last_fullsized":"2024-03-06T07:25:06.876531+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.142829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882504+0000","last_change":"2024-03-06T07:25:04.864438+0000","last_active":"2024-03-06T07:25:06.882504+0000","last_peered":"2024-03-06T07:25:06.882504+0000","last_clean":"2024-03-06T07:25:06.882504+0000","last_became_active":"2024-03-06T07:25:04.864180+0000","last_became_peered":"2024-03-06T07:25:04.864180+0000","last_unstale":"2024-03-06T07:25:06.882504+0000","last_undegraded":"2024-03-06T07:25:06.882504+0000","last_fullsized":"2024-03-06T07:25:06.882504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:30.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877252+0000","last_change":"2024-03-06T07:25:05.868536+0000","last_active":"2024-03-06T07:25:06.877252+0000","last_peered":"2024-03-06T07:25:06.877252+0000","last_clean":"2024-03-06T07:25:06.877252+0000","last_became_active":"2024-03-06T07:25:05.868254+0000","last_became_peered":"2024-03-06T07:25:05.868254+0000","last_unstale":"2024-03-06T07:25:06.877252+0000","last_undegraded":"2024-03-06T07:25:06.877252+0000","last_fullsized":"2024-03-06T07:25:06.877252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:43.696175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877786+0000","last_change":"2024-03-06T07:25:04.871271+0000","last_active":"2024-03-06T07:25:06.877786+0000","last_peered":"2024-03-06T07:25:06.877786+0000","last_clean":"2024-03-06T07:25:06.877786+0000","last_became_active":"2024-03-06T07:25:04.870981+0000","last_became_peered":"2024-03-06T07:25:04.870981+0000","last_unstale":"2024-03-06T07:25:06.877786+0000","last_undegraded":"2024-03-06T07:25:06.877786+0000","last_fullsized":"2024-03-06T07:25:06.877786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:10.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097663+0000","last_change":"2024-03-06T07:25:05.868805+0000","last_active":"2024-03-06T07:25:07.097663+0000","last_peered":"2024-03-06T07:25:07.097663+0000","last_clean":"2024-03-06T07:25:07.097663+0000","last_became_active":"2024-03-06T07:25:05.868507+0000","last_became_peered":"2024-03-06T07:25:05.868507+0000","last_unstale":"2024-03-06T07:25:07.097663+0000","last_undegraded":"2024-03-06T07:25:07.097663+0000","last_fullsized":"2024-03-06T07:25:07.097663+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:08.038444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877695+0000","last_change":"2024-03-06T07:25:04.868531+0000","last_active":"2024-03-06T07:25:06.877695+0000","last_peered":"2024-03-06T07:25:06.877695+0000","last_clean":"2024-03-06T07:25:06.877695+0000","last_became_active":"2024-03-06T07:25:04.868198+0000","last_became_peered":"2024-03-06T07:25:04.868198+0000","last_unstale":"2024-03-06T07:25:06.877695+0000","last_undegraded":"2024-03-06T07:25:06.877695+0000","last_fullsized":"2024-03-06T07:25:06.877695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:50.439569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.880007+0000","last_change":"2024-03-06T07:25:05.879799+0000","last_active":"2024-03-06T07:25:06.880007+0000","last_peered":"2024-03-06T07:25:06.880007+0000","last_clean":"2024-03-06T07:25:06.880007+0000","last_became_active":"2024-03-06T07:25:05.879247+0000","last_became_peered":"2024-03-06T07:25:05.879247+0000","last_unstale":"2024-03-06T07:25:06.880007+0000","last_undegraded":"2024-03-06T07:25:06.880007+0000","last_fullsized":"2024-03-06T07:25:06.880007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:20.118069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098382+0000","last_change":"2024-03-06T07:25:04.867446+0000","last_active":"2024-03-06T07:25:07.098382+0000","last_peered":"2024-03-06T07:25:07.098382+0000","last_clean":"2024-03-06T07:25:07.098382+0000","last_became_active":"2024-03-06T07:25:04.867198+0000","last_became_peered":"2024-03-06T07:25:04.867198+0000","last_unstale":"2024-03-06T07:25:07.098382+0000","last_undegraded":"2024-03-06T07:25:07.098382+0000","last_fullsized":"2024-03-06T07:25:07.098382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:37.319162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877229+0000","last_change":"2024-03-06T07:25:05.877141+0000","last_active":"2024-03-06T07:25:06.877229+0000","last_peered":"2024-03-06T07:25:06.877229+0000","last_clean":"2024-03-06T07:25:06.877229+0000","last_became_active":"2024-03-06T07:25:05.875766+0000","last_became_peered":"2024-03-06T07:25:05.875766+0000","last_unstale":"2024-03-06T07:25:06.877229+0000","last_undegraded":"2024-03-06T07:25:06.877229+0000","last_fullsized":"2024-03-06T07:25:06.877229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:30.771856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877423+0000","last_change":"2024-03-06T07:25:04.862460+0000","last_active":"2024-03-06T07:25:06.877423+0000","last_peered":"2024-03-06T07:25:06.877423+0000","last_clean":"2024-03-06T07:25:06.877423+0000","last_became_active":"2024-03-06T07:25:04.862179+0000","last_became_peered":"2024-03-06T07:25:04.862179+0000","last_unstale":"2024-03-06T07:25:06.877423+0000","last_undegraded":"2024-03-06T07:25:06.877423+0000","last_fullsized":"2024-03-06T07:25:06.877423+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:30.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.875729+0000","last_change":"2024-03-06T07:25:05.876851+0000","last_active":"2024-03-06T07:25:06.875729+0000","last_peered":"2024-03-06T07:25:06.875729+0000","last_clean":"2024-03-06T07:25:06.875729+0000","last_became_active":"2024-03-06T07:25:05.876592+0000","last_became_peered":"2024-03-06T07:25:05.876592+0000","last_unstale":"2024-03-06T07:25:06.875729+0000","last_undegraded":"2024-03-06T07:25:06.875729+0000","last_fullsized":"2024-03-06T07:25:06.875729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:33.478631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097695+0000","last_change":"2024-03-06T07:25:04.862049+0000","last_active":"2024-03-06T07:25:07.097695+0000","last_peered":"2024-03-06T07:25:07.097695+0000","last_clean":"2024-03-06T07:25:07.097695+0000","last_became_active":"2024-03-06T07:25:04.861815+0000","last_became_peered":"2024-03-06T07:25:04.861815+0000","last_unstale":"2024-03-06T07:25:07.097695+0000","last_undegraded":"2024-03-06T07:25:07.097695+0000","last_fullsized":"2024-03-06T07:25:07.097695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:01.989975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882100+0000","last_change":"2024-03-06T07:25:05.871402+0000","last_active":"2024-03-06T07:25:06.882100+0000","last_peered":"2024-03-06T07:25:06.882100+0000","last_clean":"2024-03-06T07:25:06.882100+0000","last_became_active":"2024-03-06T07:25:05.871142+0000","last_became_peered":"2024-03-06T07:25:05.871142+0000","last_unstale":"2024-03-06T07:25:06.882100+0000","last_undegraded":"2024-03-06T07:25:06.882100+0000","last_fullsized":"2024-03-06T07:25:06.882100+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:22.911697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.875721+0000","last_change":"2024-03-06T07:25:04.859156+0000","last_active":"2024-03-06T07:25:06.875721+0000","last_peered":"2024-03-06T07:25:06.875721+0000","last_clean":"2024-03-06T07:25:06.875721+0000","last_became_active":"2024-03-06T07:25:04.858589+0000","last_became_peered":"2024-03-06T07:25:04.858589+0000","last_unstale":"2024-03-06T07:25:06.875721+0000","last_undegraded":"2024-03-06T07:25:06.875721+0000","last_fullsized":"2024-03-06T07:25:06.875721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:52.775771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.880033+0000","last_change":"2024-03-06T07:25:05.882761+0000","last_active":"2024-03-06T07:25:06.880033+0000","last_peered":"2024-03-06T07:25:06.880033+0000","last_clean":"2024-03-06T07:25:06.880033+0000","last_became_active":"2024-03-06T07:25:05.882453+0000","last_became_peered":"2024-03-06T07:25:05.882453+0000","last_unstale":"2024-03-06T07:25:06.880033+0000","last_undegraded":"2024-03-06T07:25:06.880033+0000","last_fullsized":"2024-03-06T07:25:06.880033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:22.911697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.878053+0000","last_change":"2024-03-06T07:25:04.866854+0000","last_active":"2024-03-06T07:25:06.878053+0000","last_peered":"2024-03-06T07:25:06.878053+0000","last_clean":"2024-03-06T07:25:06.878053+0000","last_became_active":"2024-03-06T07:25:04.866566+0000","last_became_peered":"2024-03-06T07:25:04.866566+0000","last_unstale":"2024-03-06T07:25:06.878053+0000","last_undegraded":"2024-03-06T07:25:06.878053+0000","last_fullsized":"2024-03-06T07:25:06.878053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:07.326669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097830+0000","last_change":"2024-03-06T07:25:05.874779+0000","last_active":"2024-03-06T07:25:07.097830+0000","last_peered":"2024-03-06T07:25:07.097830+0000","last_clean":"2024-03-06T07:25:07.097830+0000","last_became_active":"2024-03-06T07:25:05.874393+0000","last_became_peered":"2024-03-06T07:25:05.874393+0000","last_unstale":"2024-03-06T07:25:07.097830+0000","last_undegraded":"2024-03-06T07:25:07.097830+0000","last_fullsized":"2024-03-06T07:25:07.097830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:50.784377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877263+0000","last_change":"2024-03-06T07:25:04.862756+0000","last_active":"2024-03-06T07:25:06.877263+0000","last_peered":"2024-03-06T07:25:06.877263+0000","last_clean":"2024-03-06T07:25:06.877263+0000","last_became_active":"2024-03-06T07:25:04.862463+0000","last_became_peered":"2024-03-06T07:25:04.862463+0000","last_unstale":"2024-03-06T07:25:06.877263+0000","last_undegraded":"2024-03-06T07:25:06.877263+0000","last_fullsized":"2024-03-06T07:25:06.877263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:01.989975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877562+0000","last_change":"2024-03-06T07:25:05.879523+0000","last_active":"2024-03-06T07:25:06.877562+0000","last_peered":"2024-03-06T07:25:06.877562+0000","last_clean":"2024-03-06T07:25:06.877562+0000","last_became_active":"2024-03-06T07:25:05.879251+0000","last_became_peered":"2024-03-06T07:25:05.879251+0000","last_unstale":"2024-03-06T07:25:06.877562+0000","last_undegraded":"2024-03-06T07:25:06.877562+0000","last_fullsized":"2024-03-06T07:25:06.877562+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:37.777075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098274+0000","last_change":"2024-03-06T07:25:04.864989+0000","last_active":"2024-03-06T07:25:07.098274+0000","last_peered":"2024-03-06T07:25:07.098274+0000","last_clean":"2024-03-06T07:25:07.098274+0000","last_became_active":"2024-03-06T07:25:04.864724+0000","last_became_peered":"2024-03-06T07:25:04.864724+0000","last_unstale":"2024-03-06T07:25:07.098274+0000","last_undegraded":"2024-03-06T07:25:07.098274+0000","last_fullsized":"2024-03-06T07:25:07.098274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:55.949425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876099+0000","last_change":"2024-03-06T07:25:05.875353+0000","last_active":"2024-03-06T07:25:06.876099+0000","last_peered":"2024-03-06T07:25:06.876099+0000","last_clean":"2024-03-06T07:25:06.876099+0000","last_became_active":"2024-03-06T07:25:05.875079+0000","last_became_peered":"2024-03-06T07:25:05.875079+0000","last_unstale":"2024-03-06T07:25:06.876099+0000","last_undegraded":"2024-03-06T07:25:06.876099+0000","last_fullsized":"2024-03-06T07:25:06.876099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:06.102672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097962+0000","last_change":"2024-03-06T07:25:04.866085+0000","last_active":"2024-03-06T07:25:07.097962+0000","last_peered":"2024-03-06T07:25:07.097962+0000","last_clean":"2024-03-06T07:25:07.097962+0000","last_became_active":"2024-03-06T07:25:04.865800+0000","last_became_peered":"2024-03-06T07:25:04.865800+0000","last_unstale":"2024-03-06T07:25:07.097962+0000","last_undegraded":"2024-03-06T07:25:07.097962+0000","last_fullsized":"2024-03-06T07:25:07.097962+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:50.439569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097334+0000","last_change":"2024-03-06T07:25:05.873388+0000","last_active":"2024-03-06T07:25:07.097334+0000","last_peered":"2024-03-06T07:25:07.097334+0000","last_clean":"2024-03-06T07:25:07.097334+0000","last_became_active":"2024-03-06T07:25:05.873164+0000","last_became_peered":"2024-03-06T07:25:05.873164+0000","last_unstale":"2024-03-06T07:25:07.097334+0000","last_undegraded":"2024-03-06T07:25:07.097334+0000","last_fullsized":"2024-03-06T07:25:07.097334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:33.317825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097844+0000","last_change":"2024-03-06T07:25:04.866818+0000","last_active":"2024-03-06T07:25:07.097844+0000","last_peered":"2024-03-06T07:25:07.097844+0000","last_clean":"2024-03-06T07:25:07.097844+0000","last_became_active":"2024-03-06T07:25:04.866510+0000","last_became_peered":"2024-03-06T07:25:04.866510+0000","last_unstale":"2024-03-06T07:25:07.097844+0000","last_undegraded":"2024-03-06T07:25:07.097844+0000","last_fullsized":"2024-03-06T07:25:07.097844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:13.057086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.879959+0000","last_change":"2024-03-06T07:25:05.879981+0000","last_active":"2024-03-06T07:25:06.879959+0000","last_peered":"2024-03-06T07:25:06.879959+0000","last_clean":"2024-03-06T07:25:06.879959+0000","last_became_active":"2024-03-06T07:25:05.877897+0000","last_became_peered":"2024-03-06T07:25:05.877897+0000","last_unstale":"2024-03-06T07:25:06.879959+0000","last_undegraded":"2024-03-06T07:25:06.879959+0000","last_fullsized":"2024-03-06T07:25:06.879959+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:47.912785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877424+0000","last_change":"2024-03-06T07:25:04.868290+0000","last_active":"2024-03-06T07:25:06.877424+0000","last_peered":"2024-03-06T07:25:06.877424+0000","last_clean":"2024-03-06T07:25:06.877424+0000","last_became_active":"2024-03-06T07:25:04.867100+0000","last_became_peered":"2024-03-06T07:25:04.867100+0000","last_unstale":"2024-03-06T07:25:06.877424+0000","last_undegraded":"2024-03-06T07:25:06.877424+0000","last_fullsized":"2024-03-06T07:25:06.877424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:02.525342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.881982+0000","last_change":"2024-03-06T07:25:05.876428+0000","last_active":"2024-03-06T07:25:06.881982+0000","last_peered":"2024-03-06T07:25:06.881982+0000","last_clean":"2024-03-06T07:25:06.881982+0000","last_became_active":"2024-03-06T07:25:05.876168+0000","last_became_peered":"2024-03-06T07:25:05.876168+0000","last_unstale":"2024-03-06T07:25:06.881982+0000","last_undegraded":"2024-03-06T07:25:06.881982+0000","last_fullsized":"2024-03-06T07:25:06.881982+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:47.912785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.879724+0000","last_change":"2024-03-06T07:25:04.870029+0000","last_active":"2024-03-06T07:25:06.879724+0000","last_peered":"2024-03-06T07:25:06.879724+0000","last_clean":"2024-03-06T07:25:06.879724+0000","last_became_active":"2024-03-06T07:25:04.869729+0000","last_became_peered":"2024-03-06T07:25:04.869729+0000","last_unstale":"2024-03-06T07:25:06.879724+0000","last_undegraded":"2024-03-06T07:25:06.879724+0000","last_fullsized":"2024-03-06T07:25:06.879724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:34.987692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.875870+0000","last_change":"2024-03-06T07:25:05.873413+0000","last_active":"2024-03-06T07:25:06.875870+0000","last_peered":"2024-03-06T07:25:06.875870+0000","last_clean":"2024-03-06T07:25:06.875870+0000","last_became_active":"2024-03-06T07:25:05.873152+0000","last_became_peered":"2024-03-06T07:25:05.873152+0000","last_unstale":"2024-03-06T07:25:06.875870+0000","last_undegraded":"2024-03-06T07:25:06.875870+0000","last_fullsized":"2024-03-06T07:25:06.875870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:41.604630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876320+0000","last_change":"2024-03-06T07:25:04.869373+0000","last_active":"2024-03-06T07:25:06.876320+0000","last_peered":"2024-03-06T07:25:06.876320+0000","last_clean":"2024-03-06T07:25:06.876320+0000","last_became_active":"2024-03-06T07:25:04.869036+0000","last_became_peered":"2024-03-06T07:25:04.869036+0000","last_unstale":"2024-03-06T07:25:06.876320+0000","last_undegraded":"2024-03-06T07:25:06.876320+0000","last_fullsized":"2024-03-06T07:25:06.876320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:09.926453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.878011+0000","last_change":"2024-03-06T07:25:05.877706+0000","last_active":"2024-03-06T07:25:06.878011+0000","last_peered":"2024-03-06T07:25:06.878011+0000","last_clean":"2024-03-06T07:25:06.878011+0000","last_became_active":"2024-03-06T07:25:05.877334+0000","last_became_peered":"2024-03-06T07:25:05.877334+0000","last_unstale":"2024-03-06T07:25:06.878011+0000","last_undegraded":"2024-03-06T07:25:06.878011+0000","last_fullsized":"2024-03-06T07:25:06.878011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.317533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877421+0000","last_change":"2024-03-06T07:25:04.867842+0000","last_active":"2024-03-06T07:25:06.877421+0000","last_peered":"2024-03-06T07:25:06.877421+0000","last_clean":"2024-03-06T07:25:06.877421+0000","last_became_active":"2024-03-06T07:25:04.867564+0000","last_became_peered":"2024-03-06T07:25:04.867564+0000","last_unstale":"2024-03-06T07:25:06.877421+0000","last_undegraded":"2024-03-06T07:25:06.877421+0000","last_fullsized":"2024-03-06T07:25:06.877421+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097951+0000","last_change":"2024-03-06T07:25:05.877608+0000","last_active":"2024-03-06T07:25:07.097951+0000","last_peered":"2024-03-06T07:25:07.097951+0000","last_clean":"2024-03-06T07:25:07.097951+0000","last_became_active":"2024-03-06T07:25:05.876617+0000","last_became_peered":"2024-03-06T07:25:05.876617+0000","last_unstale":"2024-03-06T07:25:07.097951+0000","last_undegraded":"2024-03-06T07:25:07.097951+0000","last_fullsized":"2024-03-06T07:25:07.097951+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:47.547544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.878090+0000","last_change":"2024-03-06T07:25:04.870677+0000","last_active":"2024-03-06T07:25:06.878090+0000","last_peered":"2024-03-06T07:25:06.878090+0000","last_clean":"2024-03-06T07:25:06.878090+0000","last_became_active":"2024-03-06T07:25:04.870423+0000","last_became_peered":"2024-03-06T07:25:04.870423+0000","last_unstale":"2024-03-06T07:25:06.878090+0000","last_undegraded":"2024-03-06T07:25:06.878090+0000","last_fullsized":"2024-03-06T07:25:06.878090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:35.827471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097878+0000","last_change":"2024-03-06T07:25:05.875443+0000","last_active":"2024-03-06T07:25:07.097878+0000","last_peered":"2024-03-06T07:25:07.097878+0000","last_clean":"2024-03-06T07:25:07.097878+0000","last_became_active":"2024-03-06T07:25:05.874876+0000","last_became_peered":"2024-03-06T07:25:05.874876+0000","last_unstale":"2024-03-06T07:25:07.097878+0000","last_undegraded":"2024-03-06T07:25:07.097878+0000","last_fullsized":"2024-03-06T07:25:07.097878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:15.253254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876850+0000","last_change":"2024-03-06T07:25:04.869978+0000","last_active":"2024-03-06T07:25:06.876850+0000","last_peered":"2024-03-06T07:25:06.876850+0000","last_clean":"2024-03-06T07:25:06.876850+0000","last_became_active":"2024-03-06T07:25:04.869501+0000","last_became_peered":"2024-03-06T07:25:04.869501+0000","last_unstale":"2024-03-06T07:25:06.876850+0000","last_undegraded":"2024-03-06T07:25:06.876850+0000","last_fullsized":"2024-03-06T07:25:06.876850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:07.326669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876193+0000","last_change":"2024-03-06T07:25:05.880162+0000","last_active":"2024-03-06T07:25:06.876193+0000","last_peered":"2024-03-06T07:25:06.876193+0000","last_clean":"2024-03-06T07:25:06.876193+0000","last_became_active":"2024-03-06T07:25:05.879849+0000","last_became_peered":"2024-03-06T07:25:05.879849+0000","last_unstale":"2024-03-06T07:25:06.876193+0000","last_undegraded":"2024-03-06T07:25:06.876193+0000","last_fullsized":"2024-03-06T07:25:06.876193+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:30.957558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097509+0000","last_change":"2024-03-06T07:25:04.870140+0000","last_active":"2024-03-06T07:25:07.097509+0000","last_peered":"2024-03-06T07:25:07.097509+0000","last_clean":"2024-03-06T07:25:07.097509+0000","last_became_active":"2024-03-06T07:25:04.869889+0000","last_became_peered":"2024-03-06T07:25:04.869889+0000","last_unstale":"2024-03-06T07:25:07.097509+0000","last_undegraded":"2024-03-06T07:25:07.097509+0000","last_fullsized":"2024-03-06T07:25:07.097509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:30.862079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098359+0000","last_change":"2024-03-06T07:25:05.876498+0000","last_active":"2024-03-06T07:25:07.098359+0000","last_peered":"2024-03-06T07:25:07.098359+0000","last_clean":"2024-03-06T07:25:07.098359+0000","last_became_active":"2024-03-06T07:25:05.876230+0000","last_became_peered":"2024-03-06T07:25:05.876230+0000","last_unstale":"2024-03-06T07:25:07.098359+0000","last_undegraded":"2024-03-06T07:25:07.098359+0000","last_fullsized":"2024-03-06T07:25:07.098359+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:19.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877543+0000","last_change":"2024-03-06T07:25:04.861518+0000","last_active":"2024-03-06T07:25:06.877543+0000","last_peered":"2024-03-06T07:25:06.877543+0000","last_clean":"2024-03-06T07:25:06.877543+0000","last_became_active":"2024-03-06T07:25:04.861183+0000","last_became_peered":"2024-03-06T07:25:04.861183+0000","last_unstale":"2024-03-06T07:25:06.877543+0000","last_undegraded":"2024-03-06T07:25:06.877543+0000","last_fullsized":"2024-03-06T07:25:06.877543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.279615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097394+0000","last_change":"2024-03-06T07:25:05.878205+0000","last_active":"2024-03-06T07:25:07.097394+0000","last_peered":"2024-03-06T07:25:07.097394+0000","last_clean":"2024-03-06T07:25:07.097394+0000","last_became_active":"2024-03-06T07:25:05.877910+0000","last_became_peered":"2024-03-06T07:25:05.877910+0000","last_unstale":"2024-03-06T07:25:07.097394+0000","last_undegraded":"2024-03-06T07:25:07.097394+0000","last_fullsized":"2024-03-06T07:25:07.097394+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:23.714325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.879887+0000","last_change":"2024-03-06T07:25:04.865215+0000","last_active":"2024-03-06T07:25:06.879887+0000","last_peered":"2024-03-06T07:25:06.879887+0000","last_clean":"2024-03-06T07:25:06.879887+0000","last_became_active":"2024-03-06T07:25:04.864955+0000","last_became_peered":"2024-03-06T07:25:04.864955+0000","last_unstale":"2024-03-06T07:25:06.879887+0000","last_undegraded":"2024-03-06T07:25:06.879887+0000","last_fullsized":"2024-03-06T07:25:06.879887+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:32.179598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097615+0000","last_change":"2024-03-06T07:25:05.877465+0000","last_active":"2024-03-06T07:25:07.097615+0000","last_peered":"2024-03-06T07:25:07.097615+0000","last_clean":"2024-03-06T07:25:07.097615+0000","last_became_active":"2024-03-06T07:25:05.877224+0000","last_became_peered":"2024-03-06T07:25:05.877224+0000","last_unstale":"2024-03-06T07:25:07.097615+0000","last_undegraded":"2024-03-06T07:25:07.097615+0000","last_fullsized":"2024-03-06T07:25:07.097615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:01.017990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.881914+0000","last_change":"2024-03-06T07:25:04.865095+0000","last_active":"2024-03-06T07:25:06.881914+0000","last_peered":"2024-03-06T07:25:06.881914+0000","last_clean":"2024-03-06T07:25:06.881914+0000","last_became_active":"2024-03-06T07:25:04.864783+0000","last_became_peered":"2024-03-06T07:25:04.864783+0000","last_unstale":"2024-03-06T07:25:06.881914+0000","last_undegraded":"2024-03-06T07:25:06.881914+0000","last_fullsized":"2024-03-06T07:25:06.881914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:30.978788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882431+0000","last_change":"2024-03-06T07:25:05.869129+0000","last_active":"2024-03-06T07:25:06.882431+0000","last_peered":"2024-03-06T07:25:06.882431+0000","last_clean":"2024-03-06T07:25:06.882431+0000","last_became_active":"2024-03-06T07:25:05.868540+0000","last_became_peered":"2024-03-06T07:25:05.868540+0000","last_unstale":"2024-03-06T07:25:06.882431+0000","last_undegraded":"2024-03-06T07:25:06.882431+0000","last_fullsized":"2024-03-06T07:25:06.882431+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:14.899961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877549+0000","last_change":"2024-03-06T07:25:04.866682+0000","last_active":"2024-03-06T07:25:06.877549+0000","last_peered":"2024-03-06T07:25:06.877549+0000","last_clean":"2024-03-06T07:25:06.877549+0000","last_became_active":"2024-03-06T07:25:04.866374+0000","last_became_peered":"2024-03-06T07:25:04.866374+0000","last_unstale":"2024-03-06T07:25:06.877549+0000","last_undegraded":"2024-03-06T07:25:06.877549+0000","last_fullsized":"2024-03-06T07:25:06.877549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:21.893351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.880054+0000","last_change":"2024-03-06T07:25:05.880353+0000","last_active":"2024-03-06T07:25:06.880054+0000","last_peered":"2024-03-06T07:25:06.880054+0000","last_clean":"2024-03-06T07:25:06.880054+0000","last_became_active":"2024-03-06T07:25:05.880074+0000","last_became_peered":"2024-03-06T07:25:05.880074+0000","last_unstale":"2024-03-06T07:25:06.880054+0000","last_undegraded":"2024-03-06T07:25:06.880054+0000","last_fullsized":"2024-03-06T07:25:06.880054+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:24.325879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.880172+0000","last_change":"2024-03-06T07:25:04.864465+0000","last_active":"2024-03-06T07:25:06.880172+0000","last_peered":"2024-03-06T07:25:06.880172+0000","last_clean":"2024-03-06T07:25:06.880172+0000","last_became_active":"2024-03-06T07:25:04.864206+0000","last_became_peered":"2024-03-06T07:25:04.864206+0000","last_unstale":"2024-03-06T07:25:06.880172+0000","last_undegraded":"2024-03-06T07:25:06.880172+0000","last_fullsized":"2024-03-06T07:25:06.880172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:17.313307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097552+0000","last_change":"2024-03-06T07:25:05.873369+0000","last_active":"2024-03-06T07:25:07.097552+0000","last_peered":"2024-03-06T07:25:07.097552+0000","last_clean":"2024-03-06T07:25:07.097552+0000","last_became_active":"2024-03-06T07:25:05.873117+0000","last_became_peered":"2024-03-06T07:25:05.873117+0000","last_unstale":"2024-03-06T07:25:07.097552+0000","last_undegraded":"2024-03-06T07:25:07.097552+0000","last_fullsized":"2024-03-06T07:25:07.097552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:38.323387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876179+0000","last_change":"2024-03-06T07:25:04.859877+0000","last_active":"2024-03-06T07:25:06.876179+0000","last_peered":"2024-03-06T07:25:06.876179+0000","last_clean":"2024-03-06T07:25:06.876179+0000","last_became_active":"2024-03-06T07:25:04.859458+0000","last_became_peered":"2024-03-06T07:25:04.859458+0000","last_unstale":"2024-03-06T07:25:06.876179+0000","last_undegraded":"2024-03-06T07:25:06.876179+0000","last_fullsized":"2024-03-06T07:25:06.876179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:54.659991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097833+0000","last_change":"2024-03-06T07:25:05.883644+0000","last_active":"2024-03-06T07:25:07.097833+0000","last_peered":"2024-03-06T07:25:07.097833+0000","last_clean":"2024-03-06T07:25:07.097833+0000","last_became_active":"2024-03-06T07:25:05.883295+0000","last_became_peered":"2024-03-06T07:25:05.883295+0000","last_unstale":"2024-03-06T07:25:07.097833+0000","last_undegraded":"2024-03-06T07:25:07.097833+0000","last_fullsized":"2024-03-06T07:25:07.097833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:20.777081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.882491+0000","last_change":"2024-03-06T07:25:04.869898+0000","last_active":"2024-03-06T07:25:06.882491+0000","last_peered":"2024-03-06T07:25:06.882491+0000","last_clean":"2024-03-06T07:25:06.882491+0000","last_became_active":"2024-03-06T07:25:04.869613+0000","last_became_peered":"2024-03-06T07:25:04.869613+0000","last_unstale":"2024-03-06T07:25:06.882491+0000","last_undegraded":"2024-03-06T07:25:06.882491+0000","last_fullsized":"2024-03-06T07:25:06.882491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:11.035476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876020+0000","last_change":"2024-03-06T07:25:05.873708+0000","last_active":"2024-03-06T07:25:06.876020+0000","last_peered":"2024-03-06T07:25:06.876020+0000","last_clean":"2024-03-06T07:25:06.876020+0000","last_became_active":"2024-03-06T07:25:05.873451+0000","last_became_peered":"2024-03-06T07:25:05.873451+0000","last_unstale":"2024-03-06T07:25:06.876020+0000","last_undegraded":"2024-03-06T07:25:06.876020+0000","last_fullsized":"2024-03-06T07:25:06.876020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:16.946197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.875904+0000","last_change":"2024-03-06T07:25:04.860934+0000","last_active":"2024-03-06T07:25:06.875904+0000","last_peered":"2024-03-06T07:25:06.875904+0000","last_clean":"2024-03-06T07:25:06.875904+0000","last_became_active":"2024-03-06T07:25:04.860341+0000","last_became_peered":"2024-03-06T07:25:04.860341+0000","last_unstale":"2024-03-06T07:25:06.875904+0000","last_undegraded":"2024-03-06T07:25:06.875904+0000","last_fullsized":"2024-03-06T07:25:06.875904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.296664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877268+0000","last_change":"2024-03-06T07:25:05.869196+0000","last_active":"2024-03-06T07:25:06.877268+0000","last_peered":"2024-03-06T07:25:06.877268+0000","last_clean":"2024-03-06T07:25:06.877268+0000","last_became_active":"2024-03-06T07:25:05.868916+0000","last_became_peered":"2024-03-06T07:25:05.868916+0000","last_unstale":"2024-03-06T07:25:06.877268+0000","last_undegraded":"2024-03-06T07:25:06.877268+0000","last_fullsized":"2024-03-06T07:25:06.877268+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:17.127147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097309+0000","last_change":"2024-03-06T07:25:04.862667+0000","last_active":"2024-03-06T07:25:07.097309+0000","last_peered":"2024-03-06T07:25:07.097309+0000","last_clean":"2024-03-06T07:25:07.097309+0000","last_became_active":"2024-03-06T07:25:04.862428+0000","last_became_peered":"2024-03-06T07:25:04.862428+0000","last_unstale":"2024-03-06T07:25:07.097309+0000","last_undegraded":"2024-03-06T07:25:07.097309+0000","last_fullsized":"2024-03-06T07:25:07.097309+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097391+0000","last_change":"2024-03-06T07:25:05.880488+0000","last_active":"2024-03-06T07:25:07.097391+0000","last_peered":"2024-03-06T07:25:07.097391+0000","last_clean":"2024-03-06T07:25:07.097391+0000","last_became_active":"2024-03-06T07:25:05.880183+0000","last_became_peered":"2024-03-06T07:25:05.880183+0000","last_unstale":"2024-03-06T07:25:07.097391+0000","last_undegraded":"2024-03-06T07:25:07.097391+0000","last_fullsized":"2024-03-06T07:25:07.097391+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:11.488437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877326+0000","last_change":"2024-03-06T07:25:04.866391+0000","last_active":"2024-03-06T07:25:06.877326+0000","last_peered":"2024-03-06T07:25:06.877326+0000","last_clean":"2024-03-06T07:25:06.877326+0000","last_became_active":"2024-03-06T07:25:04.866047+0000","last_became_peered":"2024-03-06T07:25:04.866047+0000","last_unstale":"2024-03-06T07:25:06.877326+0000","last_undegraded":"2024-03-06T07:25:06.877326+0000","last_fullsized":"2024-03-06T07:25:06.877326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:15.242205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876503+0000","last_change":"2024-03-06T07:25:05.876008+0000","last_active":"2024-03-06T07:25:06.876503+0000","last_peered":"2024-03-06T07:25:06.876503+0000","last_clean":"2024-03-06T07:25:06.876503+0000","last_became_active":"2024-03-06T07:25:05.875572+0000","last_became_peered":"2024-03-06T07:25:05.875572+0000","last_unstale":"2024-03-06T07:25:06.876503+0000","last_undegraded":"2024-03-06T07:25:06.876503+0000","last_fullsized":"2024-03-06T07:25:06.876503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:28.482932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.876855+0000","last_change":"2024-03-06T07:25:04.868423+0000","last_active":"2024-03-06T07:25:06.876855+0000","last_peered":"2024-03-06T07:25:06.876855+0000","last_clean":"2024-03-06T07:25:06.876855+0000","last_became_active":"2024-03-06T07:25:04.868028+0000","last_became_peered":"2024-03-06T07:25:04.868028+0000","last_unstale":"2024-03-06T07:25:06.876855+0000","last_undegraded":"2024-03-06T07:25:06.876855+0000","last_fullsized":"2024-03-06T07:25:06.876855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:30.676481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.098046+0000","last_change":"2024-03-06T07:25:05.878084+0000","last_active":"2024-03-06T07:25:07.098046+0000","last_peered":"2024-03-06T07:25:07.098046+0000","last_clean":"2024-03-06T07:25:07.098046+0000","last_became_active":"2024-03-06T07:25:05.877799+0000","last_became_peered":"2024-03-06T07:25:05.877799+0000","last_unstale":"2024-03-06T07:25:07.098046+0000","last_undegraded":"2024-03-06T07:25:07.098046+0000","last_fullsized":"2024-03-06T07:25:07.098046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.847107+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.847107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:52.051062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.877383+0000","last_change":"2024-03-06T07:25:04.867025+0000","last_active":"2024-03-06T07:25:06.877383+0000","last_peered":"2024-03-06T07:25:06.877383+0000","last_clean":"2024-03-06T07:25:06.877383+0000","last_became_active":"2024-03-06T07:25:04.866493+0000","last_became_peered":"2024-03-06T07:25:04.866493+0000","last_unstale":"2024-03-06T07:25:06.877383+0000","last_undegraded":"2024-03-06T07:25:06.877383+0000","last_fullsized":"2024-03-06T07:25:06.877383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.842881+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.842881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:13.057086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":259,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:07.097256+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:07.097256+0000","last_peered":"2024-03-06T07:25:07.097256+0000","last_clean":"2024-03-06T07:25:07.097256+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:07.097256+0000","last_undegraded":"2024-03-06T07:25:07.097256+0000","last_fullsized":"2024-03-06T07:25:07.097256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1441792,"data_stored":943121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":843776,"data_stored":352227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":353749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":354365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":942505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:13.719 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:13.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:13.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 Processor -- start 2024-03-06T07:25:13.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 -- start start 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2dc80b0db0 0x7f2dc8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2dc80ce750 0x7f2dc80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc806b050 con 0x7f2dc815f4f0 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc806b2c0 con 0x7f2dc80ce750 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.862+0000 7f2dcf647640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc80b0890 con 0x7f2dc80b0db0 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcdbe4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2dc80ce750 0x7f2dc80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcd3e3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2dc815f4f0 0x7f2dc80ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36572/0 (socket says 172.21.15.122:36572) 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcd3e3640 1 -- 172.21.15.122:0/2712912017 learned_addr learned my addr 172.21.15.122:0/2712912017 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:13.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dccbe2640 1 --2- 172.21.15.122:0/2712912017 >> v2:172.21.15.169:3300/0 conn(0x7f2dc80b0db0 0x7f2dc8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/2712912017 >> v2:172.21.15.169:3300/0 conn(0x7f2dc80b0db0 msgr2=0x7f2dc8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcdbe4640 1 --2- 172.21.15.122:0/2712912017 >> v2:172.21.15.169:3300/0 conn(0x7f2dc80b0db0 0x7f2dc8066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dbffff640 1 -- 172.21.15.122:0/2712912017 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217671093 0 0) 0x7f2dc806b050 con 0x7f2dc815f4f0 2024-03-06T07:25:13.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/2712912017 >> v1:172.21.15.122:6789/0 conn(0x7f2dc815f4f0 legacy=0x7f2dc80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.863+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/2712912017 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80672e0 con 0x7f2dc80ce750 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.864+0000 7f2dcdbe4640 1 --2- 172.21.15.122:0/2712912017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc80ce750 0x7f2dc80b4ed0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80d11d0 tx=0x7f2dc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eb064e3a2d7a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.864+0000 7f2dbffff640 1 -- 172.21.15.122:0/2712912017 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0039070 con 0x7f2dc80ce750 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.864+0000 7f2dbffff640 1 -- 172.21.15.122:0/2712912017 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc002cc80 con 0x7f2dc80ce750 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.864+0000 7f2dbffff640 1 -- 172.21.15.122:0/2712912017 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0034660 con 0x7f2dc80ce750 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.864+0000 7f2dcf647640 1 -- 172.21.15.122:0/2712912017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc80ce750 msgr2=0x7f2dc80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.864+0000 7f2dcf647640 1 --2- 172.21.15.122:0/2712912017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc80ce750 0x7f2dc80b4ed0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80d11d0 tx=0x7f2dc002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.865+0000 7f2dcf647640 1 -- 172.21.15.122:0/2712912017 shutdown_connections 2024-03-06T07:25:13.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.865+0000 7f2dcf647640 1 --2- 172.21.15.122:0/2712912017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc80ce750 0x7f2dc80b4ed0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.865+0000 7f2dcf647640 1 --2- 172.21.15.122:0/2712912017 >> v2:172.21.15.169:3300/0 conn(0x7f2dc80b0db0 0x7f2dc8066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.865+0000 7f2dcf647640 1 -- 172.21.15.122:0/2712912017 >> 172.21.15.122:0/2712912017 conn(0x7f2dc80b1d70 msgr2=0x7f2dc80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.865+0000 7f2dcf647640 1 -- 172.21.15.122:0/2712912017 shutdown_connections 2024-03-06T07:25:13.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.865+0000 7f2dcf647640 1 -- 172.21.15.122:0/2712912017 wait complete. 2024-03-06T07:25:13.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.866+0000 7f2dcf647640 1 Processor -- start 2024-03-06T07:25:13.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.866+0000 7f2dcf647640 1 -- start start 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dc80b0db0 0x7f2dc80c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dc80ce750 0x7f2dc815bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 0x7f2dc814d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc815f8d0 con 0x7f2dc815f4f0 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc806b5a0 con 0x7f2dc80b0db0 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc80c8ea0 con 0x7f2dc80ce750 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 0x7f2dc814d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 0x7f2dc814d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47996/0 (socket says 172.21.15.122:47996) 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/340226971 learned_addr learned my addr 172.21.15.122:0/340226971 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcd3e3640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dc80b0db0 0x7f2dc80c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dccbe2640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dc80ce750 0x7f2dc815bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dc80ce750 msgr2=0x7f2dc815bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dc80ce750 0x7f2dc815bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dc80b0db0 msgr2=0x7f2dc80c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dc80b0db0 0x7f2dc80c7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 -- 172.21.15.122:0/340226971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc815c410 con 0x7f2dc815f4f0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcd3e3640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dc80b0db0 0x7f2dc80c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcdbe4640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 0x7f2dc814d7d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009e80 tx=0x7f2dc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0039050 con 0x7f2dc815f4f0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0045070 con 0x7f2dc815f4f0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc80c8290 con 0x7f2dc815f4f0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.868+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0009450 con 0x7f2dc815f4f0 2024-03-06T07:25:13.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.869+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc80c8b30 con 0x7f2dc815f4f0 2024-03-06T07:25:13.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.870+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80c8680 con 0x7f2dc815f4f0 2024-03-06T07:25:13.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.870+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dc003e850 con 0x7f2dc815f4f0 2024-03-06T07:25:13.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.871+0000 7f2dbdffb640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dac06b4d0 0x7f2dac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.871+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dc00ad0b0 con 0x7f2dc815f4f0 2024-03-06T07:25:13.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.871+0000 7f2dcd3e3640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dac06b4d0 0x7f2dac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.871+0000 7f2dcd3e3640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dac06b4d0 0x7f2dac06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4000f60 tx=0x7f2dc4007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:13.877+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dc007c0e0 con 0x7f2dc815f4f0 2024-03-06T07:25:14.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.123+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2dc80c9020 con 0x7f2dc815f4f0 2024-03-06T07:25:14.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.125+0000 7f2dbdffb640 1 -- 172.21.15.122:0/340226971 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2dc007b9e0 con 0x7f2dc815f4f0 2024-03-06T07:25:14.127 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:25:14.127 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:25:14.127 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:25:14.127 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:25:14.127 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:14.128 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dac06b4d0 msgr2=0x7f2dac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dac06b4d0 0x7f2dac06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4000f60 tx=0x7f2dc4007ca0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 msgr2=0x7f2dc814d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 0x7f2dc814d7d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009e80 tx=0x7f2dc0004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 shutdown_connections 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dac06b4d0 0x7f2dac06d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dc815f4f0 0x7f2dc814d7d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2dc80ce750 0x7f2dc815bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 --2- 172.21.15.122:0/340226971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dc80b0db0 0x7f2dc80c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.127+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 >> 172.21.15.122:0/340226971 conn(0x7f2dc80b1d70 msgr2=0x7f2dc80b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.128+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 shutdown_connections 2024-03-06T07:25:14.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.128+0000 7f2dcf647640 1 -- 172.21.15.122:0/340226971 wait complete. 2024-03-06T07:25:14.140 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:14.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 Processor -- start 2024-03-06T07:25:14.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 -- start start 2024-03-06T07:25:14.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f956415bb20 0x7f95640d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95640cfff0 0x7f95640d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f95640ce4b0 0x7f95640ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9564161980 con 0x7f95640cfff0 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9564161c60 con 0x7f95640ce4b0 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.240+0000 7f9569215640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9564161f40 con 0x7f956415bb20 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95640cfff0 0x7f95640d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95640cfff0 0x7f95640d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48000/0 (socket says 172.21.15.122:48000) 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 -- 172.21.15.122:0/1348450958 learned_addr learned my addr 172.21.15.122:0/1348450958 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f9562d9d640 1 --2- 172.21.15.122:0/1348450958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f956415bb20 0x7f95640d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f956359e640 1 --2- 172.21.15.122:0/1348450958 >> v2:172.21.15.169:3300/0 conn(0x7f95640ce4b0 0x7f95640ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 -- 172.21.15.122:0/1348450958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f956415bb20 msgr2=0x7f95640d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 --2- 172.21.15.122:0/1348450958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f956415bb20 0x7f95640d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 -- 172.21.15.122:0/1348450958 >> v2:172.21.15.169:3300/0 conn(0x7f95640ce4b0 msgr2=0x7f95640ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 --2- 172.21.15.122:0/1348450958 >> v2:172.21.15.169:3300/0 conn(0x7f95640ce4b0 0x7f95640ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f955bfff640 1 -- 172.21.15.122:0/1348450958 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564162220 con 0x7f95640cfff0 2024-03-06T07:25:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.241+0000 7f956359e640 1 --2- 172.21.15.122:0/1348450958 >> v2:172.21.15.169:3300/0 conn(0x7f95640ce4b0 0x7f95640ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f955bfff640 1 --2- 172.21.15.122:0/1348450958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95640cfff0 0x7f95640d0470 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f95640b13b0 tx=0x7f954c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f0aa259ea61d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f956259c640 1 -- 172.21.15.122:0/1348450958 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c035020 con 0x7f95640cfff0 2024-03-06T07:25:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f956259c640 1 -- 172.21.15.122:0/1348450958 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c02d040 con 0x7f95640cfff0 2024-03-06T07:25:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f956259c640 1 -- 172.21.15.122:0/1348450958 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c031a80 con 0x7f95640cfff0 2024-03-06T07:25:14.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 -- 172.21.15.122:0/1348450958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95640cfff0 msgr2=0x7f95640d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 --2- 172.21.15.122:0/1348450958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95640cfff0 0x7f95640d0470 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f95640b13b0 tx=0x7f954c02b900 comp rx=0 tx=0).stop 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 -- 172.21.15.122:0/1348450958 shutdown_connections 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 --2- 172.21.15.122:0/1348450958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95640cfff0 0x7f95640d0470 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 --2- 172.21.15.122:0/1348450958 >> v2:172.21.15.169:3300/0 conn(0x7f95640ce4b0 0x7f95640ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 --2- 172.21.15.122:0/1348450958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f956415bb20 0x7f95640d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 -- 172.21.15.122:0/1348450958 >> 172.21.15.122:0/1348450958 conn(0x7f956409a9f0 msgr2=0x7f956409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 -- 172.21.15.122:0/1348450958 shutdown_connections 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.242+0000 7f9569215640 1 -- 172.21.15.122:0/1348450958 wait complete. 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 Processor -- start 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 -- start start 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95640ce4b0 0x7f956415b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95640cfff0 0x7f956415e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 0x7f956406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640d24b0 con 0x7f956415bb20 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640d2330 con 0x7f95640ce4b0 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f9569215640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640d1980 con 0x7f95640cfff0 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f956359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 0x7f956406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f956359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 0x7f956406ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48004/0 (socket says 172.21.15.122:48004) 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.244+0000 7f956359e640 1 -- 172.21.15.122:0/155865608 learned_addr learned my addr 172.21.15.122:0/155865608 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f955bfff640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95640cfff0 0x7f956415e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f9562d9d640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95640ce4b0 0x7f956415b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f956359e640 1 -- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95640cfff0 msgr2=0x7f956415e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f956359e640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95640cfff0 0x7f956415e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f956359e640 1 -- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95640ce4b0 msgr2=0x7f956415b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f956359e640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95640ce4b0 0x7f956415b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f956359e640 1 -- 172.21.15.122:0/155865608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956415b5a0 con 0x7f956415bb20 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f9562d9d640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95640ce4b0 0x7f956415b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:14.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f956359e640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 0x7f956406ceb0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f955c00cef0 tx=0x7f955c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c01a070 con 0x7f956415bb20 2024-03-06T07:25:14.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c00d840 con 0x7f956415bb20 2024-03-06T07:25:14.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956415edc0 con 0x7f956415bb20 2024-03-06T07:25:14.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c00dc30 con 0x7f956415bb20 2024-03-06T07:25:14.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.245+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956415f380 con 0x7f956415bb20 2024-03-06T07:25:14.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.246+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956404f330 con 0x7f956415bb20 2024-03-06T07:25:14.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.247+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f955c01d070 con 0x7f956415bb20 2024-03-06T07:25:14.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.247+0000 7f955b7fe640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f954406b500 0x7f954406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.247+0000 7f9562d9d640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f954406b500 0x7f954406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.247+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f955c08a980 con 0x7f956415bb20 2024-03-06T07:25:14.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.248+0000 7f9562d9d640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f954406b500 0x7f954406d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9554006fd0 tx=0x7f9554008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.252+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f956415b5a0 con 0x7f956415bb20 2024-03-06T07:25:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.479+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95640b1820 con 0x7f956415bb20 2024-03-06T07:25:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.480+0000 7f955b7fe640 1 -- 172.21.15.122:0/155865608 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f955c0591b0 con 0x7f956415bb20 2024-03-06T07:25:14.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:14.482 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:14.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f954406b500 msgr2=0x7f954406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f954406b500 0x7f954406d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9554006fd0 tx=0x7f9554008040 comp rx=0 tx=0).stop 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 msgr2=0x7f956406ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 0x7f956406ceb0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f955c00cef0 tx=0x7f955c00a710 comp rx=0 tx=0).stop 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 shutdown_connections 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f954406b500 0x7f954406d9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f956415bb20 0x7f956406ceb0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95640cfff0 0x7f956415e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 --2- 172.21.15.122:0/155865608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95640ce4b0 0x7f956415b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 >> 172.21.15.122:0/155865608 conn(0x7f956409a9f0 msgr2=0x7f95640cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 shutdown_connections 2024-03-06T07:25:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:14.482+0000 7f9569215640 1 -- 172.21.15.122:0/155865608 wait complete. 2024-03-06T07:25:14.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:05.201235+0000', 'modified': '2024-03-06T07:25:07.224230+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6600}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6600': {'gid': 6600, 'name': 'a', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/323983625', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 323983625}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 323983625}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6600, 'qdb_cluster': [6600]} 2024-03-06T07:25:14.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:14.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.002+0000 7f11a0c55640 1 Processor -- start 2024-03-06T07:25:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.002+0000 7f11a0c55640 1 -- start start 2024-03-06T07:25:15.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f11a0c55640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f119c0aef20 0x7f119c072c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f11a0c55640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c066d00 con 0x7f119c161670 2024-03-06T07:25:15.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f11a0c55640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c066fc0 con 0x7f119c073170 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f11a0c55640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f119c0b6170 con 0x7f119c0aef20 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f1199d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f119c0aef20 0x7f119c072c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f119a59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f119c161670 0x7f119c072520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45770/0 (socket says 172.21.15.122:45770) 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f119a59c640 1 -- 172.21.15.122:0/943570164 learned_addr learned my addr 172.21.15.122:0/943570164 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f1199d9b640 1 -- 172.21.15.122:0/943570164 >> v1:172.21.15.169:6789/0 conn(0x7f119c073170 legacy=0x7f119c0afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f1199d9b640 1 -- 172.21.15.122:0/943570164 >> v1:172.21.15.122:6789/0 conn(0x7f119c161670 legacy=0x7f119c072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f1199d9b640 1 -- 172.21.15.122:0/943570164 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c06b050 con 0x7f119c0aef20 2024-03-06T07:25:15.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.003+0000 7f119959a640 1 -- 172.21.15.122:0/943570164 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780180039 0 0) 0x7f119c066d00 con 0x7f119c161670 2024-03-06T07:25:15.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f1199d9b640 1 --2- 172.21.15.122:0/943570164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c072c30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5268b4fc66e2e19e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f119959a640 1 -- 172.21.15.122:0/943570164 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c035020 con 0x7f119c0aef20 2024-03-06T07:25:15.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f119959a640 1 -- 172.21.15.122:0/943570164 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c031830 con 0x7f119c0aef20 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f119959a640 1 -- 172.21.15.122:0/943570164 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c031c00 con 0x7f119c0aef20 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 -- 172.21.15.122:0/943570164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 msgr2=0x7f119c072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 --2- 172.21.15.122:0/943570164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c072c30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 -- 172.21.15.122:0/943570164 shutdown_connections 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 --2- 172.21.15.122:0/943570164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c072c30 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 -- 172.21.15.122:0/943570164 >> 172.21.15.122:0/943570164 conn(0x7f119c0b4e60 msgr2=0x7f119c0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 -- 172.21.15.122:0/943570164 shutdown_connections 2024-03-06T07:25:15.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.005+0000 7f11a0c55640 1 -- 172.21.15.122:0/943570164 wait complete. 2024-03-06T07:25:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 Processor -- start 2024-03-06T07:25:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 -- start start 2024-03-06T07:25:15.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f119c073170 0x7f119c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f119c161670 0x7f119c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c161a50 con 0x7f119c0aef20 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c0ada80 con 0x7f119c161670 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f11a0c55640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119c06b610 con 0x7f119c073170 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f1199d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f1199d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57660/0 (socket says 172.21.15.122:57660) 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.007+0000 7f1199d9b640 1 -- 172.21.15.122:0/3549475578 learned_addr learned my addr 172.21.15.122:0/3549475578 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f119a59c640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f119c073170 0x7f119c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f119ad9d640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f119c161670 0x7f119c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f1199d9b640 1 -- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f119c073170 msgr2=0x7f119c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f1199d9b640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f119c073170 0x7f119c1568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f1199d9b640 1 -- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f119c161670 msgr2=0x7f119c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f1199d9b640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f119c161670 0x7f119c15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f1199d9b640 1 -- 172.21.15.122:0/3549475578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c158a20 con 0x7f119c0aef20 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f119ad9d640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f119c161670 0x7f119c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f1199d9b640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c159c70 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f118affd640 1 -- 172.21.15.122:0/3549475578 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c004230 con 0x7f119c0aef20 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f118affd640 1 -- 172.21.15.122:0/3549475578 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c0043d0 con 0x7f119c0aef20 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119c158d00 con 0x7f119c0aef20 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f118affd640 1 -- 172.21.15.122:0/3549475578 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c03cad0 con 0x7f119c0aef20 2024-03-06T07:25:15.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.008+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119c0ca300 con 0x7f119c0aef20 2024-03-06T07:25:15.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.010+0000 7f118affd640 1 -- 172.21.15.122:0/3549475578 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f118c038020 con 0x7f119c0aef20 2024-03-06T07:25:15.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.010+0000 7f118affd640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f117c06b4d0 0x7f117c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.010+0000 7f118affd640 1 -- 172.21.15.122:0/3549475578 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f118c0ac440 con 0x7f119c0aef20 2024-03-06T07:25:15.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.010+0000 7f119a59c640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f117c06b4d0 0x7f117c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.010+0000 7f119a59c640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f117c06b4d0 0x7f117c06d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1190007fd0 tx=0x7f1190007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.011+0000 7f11a383e740 1 Processor -- start 2024-03-06T07:25:15.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.012+0000 7f11a383e740 1 -- start start 2024-03-06T07:25:15.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a62870 0x559cf7a62c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f11a383e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 0x559cf7a68360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a92a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f11a383e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf79cc6b0 con 0x559cf7a668c0 2024-03-06T07:25:15.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7975a60 con 0x559cf7a92600 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cf79c8750 con 0x559cf7a62870 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 0x559cf7a68360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 0x559cf7a68360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57668/0 (socket says 172.21.15.122:57668) 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 -- 172.21.15.122:0/1704158922 learned_addr learned my addr 172.21.15.122:0/1704158922 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 -- 172.21.15.122:0/1704158922 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a62870 msgr2=0x559cf7a62c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a62870 0x559cf7a62c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 -- 172.21.15.122:0/1704158922 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 msgr2=0x559cf7a92a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a92a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 -- 172.21.15.122:0/1704158922 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cf7993f30 con 0x559cf7a668c0 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f1199d9b640 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 0x559cf7a68360 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f118c0aa990 tx=0x7f118c006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3082e9b704abd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f117bfff640 1 -- 172.21.15.122:0/1704158922 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c03cc70 con 0x559cf7a668c0 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f117bfff640 1 -- 172.21.15.122:0/1704158922 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c07abc0 con 0x559cf7a668c0 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.013+0000 7f117bfff640 1 -- 172.21.15.122:0/1704158922 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c07ed10 con 0x559cf7a668c0 2024-03-06T07:25:15.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 -- 172.21.15.122:0/1704158922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 msgr2=0x559cf7a68360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 0x559cf7a68360 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f118c0aa990 tx=0x7f118c006f40 comp rx=0 tx=0).stop 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 -- 172.21.15.122:0/1704158922 shutdown_connections 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a92a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a668c0 0x559cf7a68360 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 --2- 172.21.15.122:0/1704158922 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a62870 0x559cf7a62c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 -- 172.21.15.122:0/1704158922 >> 172.21.15.122:0/1704158922 conn(0x559cf7a648f0 msgr2=0x559cf7a661b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 -- 172.21.15.122:0/1704158922 shutdown_connections 2024-03-06T07:25:15.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.014+0000 7f11a383e740 1 -- 172.21.15.122:0/1704158922 wait complete. 2024-03-06T07:25:15.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.015+0000 7f11a383e740 1 -- start start 2024-03-06T07:25:15.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.015+0000 7f11a383e740 1 Processor -- start 2024-03-06T07:25:15.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.016+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7aadd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.016+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a83240 0x559cf7a8cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 0x559cf7a66f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7975a60 con 0x559cf7a8d2e0 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7988ea0 con 0x559cf7a92600 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cf79cc6b0 con 0x559cf7a83240 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 10 client.0 _ll_register_callbacks cb 0x559cf7988870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 0x559cf7a66f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 0x559cf7a66f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57670/0 (socket says 172.21.15.122:57670) 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 -- 172.21.15.122:0/1756941968 learned_addr learned my addr 172.21.15.122:0/1756941968 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119a59c640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7aadd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f1199d9b640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a83240 0x559cf7a8cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 -- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a83240 msgr2=0x559cf7a8cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a83240 0x559cf7a8cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 -- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 msgr2=0x559cf7aadd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7aadd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 -- 172.21.15.122:0/1756941968 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559cf7989270 con 0x559cf7a8d2e0 2024-03-06T07:25:15.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119a59c640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7aadd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f119ad9d640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 0x559cf7a66f60 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f119400e490 tx=0x7f119400e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8fcdd94de88dc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11797fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11797fa640 1 -- 172.21.15.122:0/1756941968 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194019060 con 0x559cf7a8d2e0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11797fa640 1 -- 172.21.15.122:0/1756941968 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1194003b80 con 0x559cf7a8d2e0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f118c07abc0 con 0x559cf7a8d2e0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11797fa640 1 -- 172.21.15.122:0/1756941968 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f119401f050 con 0x559cf7a8d2e0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 10 client.6765 fetch_fsmap learned FSMap version 33 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.017+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x559cf76e12d0 con 0x559cf7a8d2e0 2024-03-06T07:25:15.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11797fa640 1 -- 172.21.15.122:0/1756941968 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c07abc0 con 0x559cf7a8d2e0 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11797fa640 1 -- 172.21.15.122:0/1756941968 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1194022020 con 0x559cf7a8d2e0 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11a383e740 10 client.6765 fetch_fsmap finished waiting for FSMap version 33 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11a383e740 10 client.6765 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11a383e740 20 client.6765 populate_metadata read hostname 'smithi122' 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11a383e740 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a67970 0x559cf7a879c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11a383e740 4 client.6765 mds_command: new command op to 6600 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 --> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559cf79d9bf0 con 0x559cf7a67970 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f119a59c640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a67970 0x559cf7a879c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f11797fa640 10 client.6765 ms_handle_connect on v2:172.21.15.122:6835/323983625 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.018+0000 7f119a59c640 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a67970 0x559cf7a879c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fc8499fad939119 server_cookie=e4a2d1c9686e3bc2 in_seq=0 out_seq=0 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11797fa640 1 -- 172.21.15.122:0/1756941968 <== mds.0 v2:172.21.15.122:6835/323983625 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559cf79d9bf0 con 0x559cf7a67970 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11797fa640 10 client.6765 handle_command_reply: tid=0 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 client.6765 shutdown 2024-03-06T07:25:15.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 msgr2=0x559cf7a66f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 0x559cf7a66f60 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f119400e490 tx=0x7f119400e890 comp rx=0 tx=0).stop 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 shutdown_connections 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a67970 0x559cf7a879c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a8d2e0 0x559cf7a66f60 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a83240 0x559cf7a8cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11797fa640 0 client.6765 ms_handle_reset on v2:172.21.15.122:6835/323983625 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 --2- 172.21.15.122:0/1756941968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7aadd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 >> 172.21.15.122:0/1756941968 conn(0x559cf7a62870 msgr2=0x559cf7a61510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 shutdown_connections 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 1 -- 172.21.15.122:0/1756941968 wait complete. 2024-03-06T07:25:15.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.019+0000 7f11a383e740 20 client.6765 trim_cache size 0 max 16384 2024-03-06T07:25:15.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.051+0000 7f11a383e740 1 Processor -- start 2024-03-06T07:25:15.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.051+0000 7f11a383e740 1 -- start start 2024-03-06T07:25:15.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a83240 0x559cf7a61700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7a61c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f11a383e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 0x559cf7a85720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f11a383e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c1ff2b0 con 0x559cf7a92600 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c1fe6b0 con 0x559cf7a83240 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119c1fecb0 con 0x559cf7a8d2e0 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 0x559cf7a85720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 0x559cf7a85720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57680/0 (socket says 172.21.15.122:57680) 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 -- 172.21.15.122:0/1067207503 learned_addr learned my addr 172.21.15.122:0/1067207503 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 -- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 msgr2=0x559cf7a61c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f1199d9b640 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7a61c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119a59c640 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a83240 0x559cf7a61700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7a61c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 -- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a83240 msgr2=0x559cf7a61700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a83240 0x559cf7a61700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 -- 172.21.15.122:0/1067207503 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cf77ab6b0 con 0x559cf7a92600 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f1199d9b640 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7a61c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f119ad9d640 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 0x559cf7a85720 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1194005170 tx=0x7f119400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ce894349f3491c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f117b7fe640 1 -- 172.21.15.122:0/1067207503 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194003b80 con 0x559cf7a92600 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f117b7fe640 1 -- 172.21.15.122:0/1067207503 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194003e50 con 0x559cf7a92600 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.052+0000 7f117b7fe640 1 -- 172.21.15.122:0/1067207503 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194005410 con 0x559cf7a92600 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 -- 172.21.15.122:0/1067207503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 msgr2=0x559cf7a85720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 0x559cf7a85720 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1194005170 tx=0x7f119400a000 comp rx=0 tx=0).stop 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 -- 172.21.15.122:0/1067207503 shutdown_connections 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a92600 0x559cf7a85720 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7a61c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 --2- 172.21.15.122:0/1067207503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a83240 0x559cf7a61700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 -- 172.21.15.122:0/1067207503 >> 172.21.15.122:0/1067207503 conn(0x559cf7a62870 msgr2=0x559cf7ac0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 -- 172.21.15.122:0/1067207503 shutdown_connections 2024-03-06T07:25:15.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 -- 172.21.15.122:0/1067207503 wait complete. 2024-03-06T07:25:15.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.053+0000 7f11a383e740 1 -- start start 2024-03-06T07:25:15.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 Processor -- start 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 0x559cf7b15680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7b17200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a85d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7a964d0 con 0x559cf7a83240 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cf7a96150 con 0x559cf7a92600 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cf7a67df0 con 0x559cf7a8d2e0 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.054+0000 7f11a383e740 10 client.0 _ll_register_callbacks cb 0x7f1194002060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 0x559cf7b15680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 0x559cf7b15680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57694/0 (socket says 172.21.15.122:57694) 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 -- 172.21.15.122:0/414196367 learned_addr learned my addr 172.21.15.122:0/414196367 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119ad9d640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a85d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f1199d9b640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7b17200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 -- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 msgr2=0x559cf7b17200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7b17200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 -- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 msgr2=0x559cf7a85d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a85d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 -- 172.21.15.122:0/414196367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559cf7a82c50 con 0x559cf7a83240 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119ad9d640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a85d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f119a59c640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 0x559cf7b15680 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x559cf7a8da30 tx=0x7f1190000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f26bc0ad3962a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11797fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11797fa640 1 -- 172.21.15.122:0/414196367 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190027020 con 0x559cf7a83240 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11797fa640 1 -- 172.21.15.122:0/414196367 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1190014c40 con 0x559cf7a83240 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1194003e50 con 0x559cf7a83240 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11797fa640 1 -- 172.21.15.122:0/414196367 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1190031050 con 0x559cf7a83240 2024-03-06T07:25:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11a383e740 10 client.6771 fetch_fsmap learned FSMap version 33 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x559cf7a82960 con 0x559cf7a83240 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.055+0000 7f11797fa640 1 -- 172.21.15.122:0/414196367 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194003e50 con 0x559cf7a83240 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11797fa640 1 -- 172.21.15.122:0/414196367 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1190034020 con 0x559cf7a83240 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 10 client.6771 fetch_fsmap finished waiting for FSMap version 33 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 10 client.6771 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 20 client.6771 populate_metadata read hostname 'smithi122' 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a864d0 0x559cf7a88150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 4 client.6771 mds_command: new command op to 6600 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 --> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559cf7a956a0 con 0x559cf7a864d0 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f1199d9b640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a864d0 0x559cf7a88150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11797fa640 10 client.6771 ms_handle_connect on v2:172.21.15.122:6835/323983625 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f1199d9b640 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a864d0 0x559cf7a88150 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b7cdbbaf689eb8e server_cookie=a5a48ecb16a6622 in_seq=0 out_seq=0 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11797fa640 1 -- 172.21.15.122:0/414196367 <== mds.0 v2:172.21.15.122:6835/323983625 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559cf7a956a0 con 0x559cf7a864d0 2024-03-06T07:25:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11797fa640 10 client.6771 handle_command_reply: tid=0 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 client.6771 shutdown 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 msgr2=0x559cf7b15680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 0x559cf7b15680 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x559cf7a8da30 tx=0x7f1190000ed0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 shutdown_connections 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:6835/323983625,v1:172.21.15.122:6837/323983625] conn(0x559cf7a864d0 0x559cf7a88150 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559cf7a92600 0x559cf7a85d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559cf7a8d2e0 0x559cf7b17200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11797fa640 0 client.6771 ms_handle_reset on v2:172.21.15.122:6835/323983625 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 --2- 172.21.15.122:0/414196367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559cf7a83240 0x559cf7b15680 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.056+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 >> 172.21.15.122:0/414196367 conn(0x559cf7a62870 msgr2=0x559cf7aad170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 shutdown_connections 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a383e740 1 -- 172.21.15.122:0/414196367 wait complete. 2024-03-06T07:25:15.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a383e740 20 client.6771 trim_cache size 0 max 16384 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 6600, 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 9.1890066279999996, 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 33, 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 69, 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 69, 2024-03-06T07:25:15.066 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 14.256377227 2024-03-06T07:25:15.067 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:25:15.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f117c06b4d0 msgr2=0x7f117c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f117c06b4d0 0x7f117c06d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1190007fd0 tx=0x7f1190007de0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 msgr2=0x7f119c159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c159c70 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 shutdown_connections 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f117c06b4d0 0x7f117c06d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f119c161670 0x7f119c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c0aef20 0x7f119c159c70 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 --2- 172.21.15.122:0/3549475578 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f119c073170 0x7f119c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 >> 172.21.15.122:0/3549475578 conn(0x7f119c0b4e60 msgr2=0x7f119c0b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 shutdown_connections 2024-03-06T07:25:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.057+0000 7f11a0c55640 1 -- 172.21.15.122:0/3549475578 wait complete. 2024-03-06T07:25:15.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.169+0000 7ff0e959a640 1 Processor -- start 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.169+0000 7ff0e959a640 1 -- start start 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e959a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff0e413dc10 0x7ff0e40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e959a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e415f580 con 0x7ff0e415bf30 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e959a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e415b450 con 0x7ff0e415ef60 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e959a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e40cf3f0 con 0x7ff0e413dc10 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e27fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff0e415bf30 0x7ff0e415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45782/0 (socket says 172.21.15.122:45782) 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e27fc640 1 -- 172.21.15.122:0/3828236187 learned_addr learned my addr 172.21.15.122:0/3828236187 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674958071 0 0) 0x7ff0e415f580 con 0x7ff0e415bf30 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0cc003520 con 0x7ff0e415bf30 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e2ffd640 1 --2- 172.21.15.122:0/3828236187 >> v2:172.21.15.169:3300/0 conn(0x7ff0e413dc10 0x7ff0e40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2942371784 0 0) 0x7ff0cc003520 con 0x7ff0e415bf30 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 >> v2:172.21.15.169:3300/0 conn(0x7ff0e413dc10 msgr2=0x7ff0e40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 --2- 172.21.15.122:0/3828236187 >> v2:172.21.15.169:3300/0 conn(0x7ff0e413dc10 0x7ff0e40b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 >> v1:172.21.15.169:6789/0 conn(0x7ff0e415ef60 legacy=0x7ff0e40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e415b730 con 0x7ff0e415bf30 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff0d8001670 con 0x7ff0e415bf30 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 >> v1:172.21.15.122:6789/0 conn(0x7ff0e415bf30 legacy=0x7ff0e415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0cc003a00 0x7ff0cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0cc0052e0 0x7ff0cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40cf3f0 con 0x7ff0cc004800 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc00c200 con 0x7ff0cc003a00 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc00c400 con 0x7ff0cc0052e0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.170+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e2ffd640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0cc003a00 0x7ff0cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e37fe640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0cc0052e0 0x7ff0cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e27fc640 1 -- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0cc0052e0 msgr2=0x7ff0cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0cc0052e0 0x7ff0cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e27fc640 1 -- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0cc003a00 msgr2=0x7ff0cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0cc003a00 0x7ff0cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e27fc640 1 -- 172.21.15.122:0/3828236187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc003520 con 0x7ff0cc004800 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e37fe640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0cc0052e0 0x7ff0cc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e2ffd640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0cc003a00 0x7ff0cc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e37fe640 1 -- 172.21.15.122:0/3828236187 reap_dead start 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.171+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0cc004c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c0e0 tx=0x7ff0cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de03f6c93471e6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8007c20 con 0x7ff0cc004800 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8030070 con 0x7ff0cc004800 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e1ffb640 1 -- 172.21.15.122:0/3828236187 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8005d70 con 0x7ff0cc004800 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 -- 172.21.15.122:0/3828236187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 msgr2=0x7ff0cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0cc004c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c0e0 tx=0x7ff0cc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 -- 172.21.15.122:0/3828236187 shutdown_connections 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 --2- 172.21.15.122:0/3828236187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0cc004c80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 -- 172.21.15.122:0/3828236187 >> 172.21.15.122:0/3828236187 conn(0x7ff0e40cf8a0 msgr2=0x7ff0e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 -- 172.21.15.122:0/3828236187 shutdown_connections 2024-03-06T07:25:15.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.172+0000 7ff0e959a640 1 -- 172.21.15.122:0/3828236187 wait complete. 2024-03-06T07:25:15.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.173+0000 7ff0e959a640 1 Processor -- start 2024-03-06T07:25:15.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.173+0000 7ff0e959a640 1 -- start start 2024-03-06T07:25:15.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e959a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0e413d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e959a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e41536c0 0x7ff0e4140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e959a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e4140980 0x7ff0e41428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e959a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e415e850 con 0x7ff0cc004800 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e959a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e415b210 con 0x7ff0e41536c0 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e959a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e415e9d0 con 0x7ff0e4140980 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0e413d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0e413d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57702/0 (socket says 172.21.15.122:57702) 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e2ffd640 1 -- 172.21.15.122:0/917266115 learned_addr learned my addr 172.21.15.122:0/917266115 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e41536c0 0x7ff0e4140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.174+0000 7ff0e37fe640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e4140980 0x7ff0e41428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e2ffd640 1 -- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e4140980 msgr2=0x7ff0e41428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e2ffd640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e4140980 0x7ff0e41428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e2ffd640 1 -- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e41536c0 msgr2=0x7ff0e4140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e2ffd640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e41536c0 0x7ff0e4140440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e2ffd640 1 -- 172.21.15.122:0/917266115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4152620 con 0x7ff0cc004800 2024-03-06T07:25:15.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e2ffd640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0e413d410 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400f5b0 tx=0x7ff0d400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4019020 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d4014870 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e4152900 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4014c60 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.175+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e4152ec0 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.176+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e404f330 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.177+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0d401c020 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.177+0000 7ff0cb7fe640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d006b4d0 0x7ff0d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.177+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0d408a830 con 0x7ff0cc004800 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.177+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d006b4d0 0x7ff0d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.178+0000 7ff0e27fc640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d006b4d0 0x7ff0d006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8034000 tx=0x7ff0d8007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.181+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0d4023020 con 0x7ff0cc004800 2024-03-06T07:25:15.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.441+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff0e406eb00 con 0x7ff0cc004800 2024-03-06T07:25:15.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.441+0000 7ff0cb7fe640 1 -- 172.21.15.122:0/917266115 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff0d4059060 con 0x7ff0cc004800 2024-03-06T07:25:15.444 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d006b4d0 msgr2=0x7ff0d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d006b4d0 0x7ff0d006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8034000 tx=0x7ff0d8007000 comp rx=0 tx=0).stop 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 msgr2=0x7ff0e413d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0e413d410 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400f5b0 tx=0x7ff0d400fa70 comp rx=0 tx=0).stop 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 shutdown_connections 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d006b4d0 0x7ff0d006d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e4140980 0x7ff0e41428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e41536c0 0x7ff0e4140440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 --2- 172.21.15.122:0/917266115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0cc004800 0x7ff0e413d410 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 >> 172.21.15.122:0/917266115 conn(0x7ff0e40cf8a0 msgr2=0x7ff0e40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 shutdown_connections 2024-03-06T07:25:15.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:15.443+0000 7ff0e959a640 1 -- 172.21.15.122:0/917266115 wait complete. 2024-03-06T07:25:15.456 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:15.456 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:15.456 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:25:15.457 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:15.457 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:15.457 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:15.457 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:15.457 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:15.457 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:15.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:15.457 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:25:15.472 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:15.472 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:15.472 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:15.473 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:25:15.474 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42525sec preferred_lft 42525sec 2024-03-06T07:25:15.474 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:25:15.474 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.474 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:15.474 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:15.475 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:25:15.476 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.476 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:15.476 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:15.476 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:25:15.476 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:15.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:15.476 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:15.539 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:25:15.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:15.541 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:25:15.599 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:25:15.599 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:15.599 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:25:15.599 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:25:15.599 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:15.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:15.600 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:25:15.600 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:25:15.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:25:15.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:25:15.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:25:15.600 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:25:15.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:15.696 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:25:15.710 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:15.710 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:15.711 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:15.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:15.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:25:15.777 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:15.777 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:15.778 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:15.832 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:15.832 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:15.832 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:15.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:25:15.953 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:16.009 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:25:16.009 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:25:16.009 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:25:16.009 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:25:16.009 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:16.010 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:16.011 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:16.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:16.012 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:25:16.067 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:16.067 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:16.110 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:25:16.136 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:25:16.192 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:16.193 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:16.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:16.194 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:25:16.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:25:16.197+0000 7fa117043480 -1 init, newargv = 0x55dc913a8430 newargc=15 2024-03-06T07:25:16.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[62880]: starting ceph client 2024-03-06T07:25:16.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[62880]: starting fuse 2024-03-06T07:25:16.215 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:25:16.215 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:25:16.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:16.215 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:25:16.215 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:25:16.215 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:25:16.215 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:16.216 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:16.217 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:25:16.334 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.62880.asok 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:16.345 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62880.asok 2024-03-06T07:25:16.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:16.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62880.asok status 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "62880", 2024-03-06T07:25:16.481 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 6786, 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 6786 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1856886955 2024-03-06T07:25:16.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1856886955 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.6786 192.168.0.1:0/1856886955", 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/1856886955", 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 33, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 69, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:25:16.483 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:25:16.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:16.489 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:25:16.489 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:16.504 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:25:16.505 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:16.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:16.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:16.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:25:16.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.715+0000 7f05e47b5640 1 Processor -- start 2024-03-06T07:25:16.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.715+0000 7f05e47b5640 1 -- start start 2024-03-06T07:25:16.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e47b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f05dc15dd00 0x7f05dc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e47b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05dc15e0e0 0x7f05dc06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e47b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05dc161bd0 con 0x7f05dc0b0000 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e47b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05dc070220 con 0x7f05dc15e0e0 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e47b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05dc161e90 con 0x7f05dc15dd00 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05dc15e0e0 0x7f05dc06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05dc15e0e0 0x7f05dc06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57716/0 (socket says 172.21.15.122:57716) 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 -- 172.21.15.122:0/3853811634 learned_addr learned my addr 172.21.15.122:0/3853811634 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e2d52640 1 --1- 172.21.15.122:0/3853811634 >> v1:172.21.15.122:6789/0 conn(0x7f05dc0b0000 0x7f05dc0b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45786/0 (socket says 172.21.15.122:45786) 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 -- 172.21.15.122:0/3853811634 >> v2:172.21.15.169:3300/0 conn(0x7f05dc15dd00 msgr2=0x7f05dc06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 --2- 172.21.15.122:0/3853811634 >> v2:172.21.15.169:3300/0 conn(0x7f05dc15dd00 0x7f05dc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 -- 172.21.15.122:0/3853811634 >> v1:172.21.15.122:6789/0 conn(0x7f05dc0b0000 legacy=0x7f05dc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e1d50640 1 -- 172.21.15.122:0/3853811634 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc162170 con 0x7f05dc15e0e0 2024-03-06T07:25:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.716+0000 7f05e154f640 1 -- 172.21.15.122:0/3853811634 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977815631 0 0) 0x7f05dc161bd0 con 0x7f05dc0b0000 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.717+0000 7f05e1d50640 1 --2- 172.21.15.122:0/3853811634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15e0e0 0x7f05dc06afb0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f05dc070620 tx=0x7f05d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cbfe71742bb9b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.717+0000 7f05e154f640 1 -- 172.21.15.122:0/3853811634 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8034070 con 0x7f05dc15e0e0 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.717+0000 7f05e154f640 1 -- 172.21.15.122:0/3853811634 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d8030860 con 0x7f05dc15e0e0 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.717+0000 7f05e154f640 1 -- 172.21.15.122:0/3853811634 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8030c30 con 0x7f05dc15e0e0 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 -- 172.21.15.122:0/3853811634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15e0e0 msgr2=0x7f05dc06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 --2- 172.21.15.122:0/3853811634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15e0e0 0x7f05dc06afb0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f05dc070620 tx=0x7f05d802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 -- 172.21.15.122:0/3853811634 shutdown_connections 2024-03-06T07:25:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 --2- 172.21.15.122:0/3853811634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15e0e0 0x7f05dc06afb0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 --2- 172.21.15.122:0/3853811634 >> v2:172.21.15.169:3300/0 conn(0x7f05dc15dd00 0x7f05dc06e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 -- 172.21.15.122:0/3853811634 >> 172.21.15.122:0/3853811634 conn(0x7f05dc0b2660 msgr2=0x7f05dc0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 -- 172.21.15.122:0/3853811634 shutdown_connections 2024-03-06T07:25:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.718+0000 7f05e47b5640 1 -- 172.21.15.122:0/3853811634 wait complete. 2024-03-06T07:25:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.719+0000 7f05e47b5640 1 Processor -- start 2024-03-06T07:25:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.719+0000 7f05e47b5640 1 -- start start 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e47b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05dc0b0000 0x7f05dc1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e47b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05dc15dd00 0x7f05dc15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e47b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 0x7f05dc1600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e47b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc162810 con 0x7f05dc15cbf0 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e47b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc162690 con 0x7f05dc0b0000 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e47b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc162990 con 0x7f05dc15dd00 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 0x7f05dc1600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 0x7f05dc1600b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:57718/0 (socket says 172.21.15.122:57718) 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 -- 172.21.15.122:0/2406731111 learned_addr learned my addr 172.21.15.122:0/2406731111 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e1d50640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05dc15dd00 0x7f05dc15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2551640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05dc0b0000 0x7f05dc1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 -- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05dc15dd00 msgr2=0x7f05dc15c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05dc15dd00 0x7f05dc15c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 -- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05dc0b0000 msgr2=0x7f05dc1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05dc0b0000 0x7f05dc1558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 -- 172.21.15.122:0/2406731111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc160e30 con 0x7f05dc15cbf0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.720+0000 7f05e2d52640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 0x7f05dc1600b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f05d400c730 tx=0x7f05d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.721+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4018070 con 0x7f05dc15cbf0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.721+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d400ce40 con 0x7f05dc15cbf0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.721+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05dc156400 con 0x7f05dc15cbf0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.721+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d400a860 con 0x7f05dc15cbf0 2024-03-06T07:25:16.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.721+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05dc1569c0 con 0x7f05dc15cbf0 2024-03-06T07:25:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.722+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05dc073390 con 0x7f05dc15cbf0 2024-03-06T07:25:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.722+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05d401b070 con 0x7f05dc15cbf0 2024-03-06T07:25:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.723+0000 7f05d2ffd640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05c006b4d0 0x7f05c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.723+0000 7f05e2551640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05c006b4d0 0x7f05c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.723+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05d4089770 con 0x7f05dc15cbf0 2024-03-06T07:25:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.723+0000 7f05e2551640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05c006b4d0 0x7f05c006d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009890 tx=0x7f05cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.727+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05d4058680 con 0x7f05dc15cbf0 2024-03-06T07:25:16.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.905+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f05dc068ff0 con 0x7f05dc15cbf0 2024-03-06T07:25:16.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.907+0000 7f05d2ffd640 1 -- 172.21.15.122:0/2406731111 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f05d4057f80 con 0x7f05dc15cbf0 2024-03-06T07:25:16.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05c006b4d0 msgr2=0x7f05c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05c006b4d0 0x7f05c006d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009890 tx=0x7f05cc008040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 msgr2=0x7f05dc1600b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 0x7f05dc1600b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f05d400c730 tx=0x7f05d400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 shutdown_connections 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05c006b4d0 0x7f05c006d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05dc15cbf0 0x7f05dc1600b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05dc15dd00 0x7f05dc15c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 --2- 172.21.15.122:0/2406731111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05dc0b0000 0x7f05dc1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 >> 172.21.15.122:0/2406731111 conn(0x7f05dc0b2660 msgr2=0x7f05dc06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 shutdown_connections 2024-03-06T07:25:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:16.909+0000 7f05e47b5640 1 -- 172.21.15.122:0/2406731111 wait complete. 2024-03-06T07:25:23.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:23.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:26.926 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.026+0000 7f25fd4b1640 1 Processor -- start 2024-03-06T07:25:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.026+0000 7f25fd4b1640 1 -- start start 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25fd4b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f25f81515e0 0x7f25f80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25fd4b1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f8071f40 con 0x7f25f815b430 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25fd4b1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f815b8e0 con 0x7f25f80cd7a0 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25fd4b1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f8070a70 con 0x7f25f81515e0 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25f6ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f25f81515e0 0x7f25f80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25ee7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f25f815b430 0x7f25f8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40922/0 (socket says 172.21.15.122:40922) 2024-03-06T07:25:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25ee7fc640 1 -- 172.21.15.122:0/3268746546 learned_addr learned my addr 172.21.15.122:0/3268746546 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25f6ffd640 1 -- 172.21.15.122:0/3268746546 >> v1:172.21.15.169:6789/0 conn(0x7f25f80cd7a0 legacy=0x7f25f80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25f6ffd640 1 -- 172.21.15.122:0/3268746546 >> v1:172.21.15.122:6789/0 conn(0x7f25f815b430 legacy=0x7f25f8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.027+0000 7f25f6ffd640 1 -- 172.21.15.122:0/3268746546 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8070d50 con 0x7f25f81515e0 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.028+0000 7f25f6ffd640 1 --2- 172.21.15.122:0/3268746546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f80cce40 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f25e80097a0 tx=0x7f25e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb1dce70f761992 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.028+0000 7f25f67fc640 1 -- 172.21.15.122:0/3268746546 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8035020 con 0x7f25f81515e0 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.028+0000 7f25f67fc640 1 -- 172.21.15.122:0/3268746546 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e8031830 con 0x7f25f81515e0 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.028+0000 7f25f67fc640 1 -- 172.21.15.122:0/3268746546 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8031c00 con 0x7f25f81515e0 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.028+0000 7f25fd4b1640 1 -- 172.21.15.122:0/3268746546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 msgr2=0x7f25f80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.028+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/3268746546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f80cce40 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f25e80097a0 tx=0x7f25e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.029+0000 7f25fd4b1640 1 -- 172.21.15.122:0/3268746546 shutdown_connections 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.029+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/3268746546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f80cce40 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.029+0000 7f25fd4b1640 1 -- 172.21.15.122:0/3268746546 >> 172.21.15.122:0/3268746546 conn(0x7f25f80cf8a0 msgr2=0x7f25f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.029+0000 7f25fd4b1640 1 -- 172.21.15.122:0/3268746546 shutdown_connections 2024-03-06T07:25:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.029+0000 7f25fd4b1640 1 -- 172.21.15.122:0/3268746546 wait complete. 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.030+0000 7f25fd4b1640 1 Processor -- start 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.030+0000 7f25fd4b1640 1 -- start start 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.030+0000 7f25fd4b1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25f80cd7a0 0x7f25f8158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.030+0000 7f25fd4b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f8141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25fd4b1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25f815b430 0x7f25f8140c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25fd4b1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80cc3a0 con 0x7f25f81515e0 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25fd4b1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8071310 con 0x7f25f80cd7a0 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25fd4b1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80cc520 con 0x7f25f815b430 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f8141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f8141df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41450/0 (socket says 172.21.15.122:41450) 2024-03-06T07:25:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 -- 172.21.15.122:0/922741109 learned_addr learned my addr 172.21.15.122:0/922741109 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25f6ffd640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25f80cd7a0 0x7f25f8158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25f77fe640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25f815b430 0x7f25f8140c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 -- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25f815b430 msgr2=0x7f25f8140c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25f815b430 0x7f25f8140c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 -- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25f80cd7a0 msgr2=0x7f25f8158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25f80cd7a0 0x7f25f8158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 -- 172.21.15.122:0/922741109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8142330 con 0x7f25f81515e0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25f6ffd640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25f80cd7a0 0x7f25f8158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25ee7fc640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f8141df0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c3c0 tx=0x7f25e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0017070 con 0x7f25f81515e0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0005ce0 con 0x7f25f81515e0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0007310 con 0x7f25f81515e0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f81425e0 con 0x7f25f81515e0 2024-03-06T07:25:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.031+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f81556e0 con 0x7f25f81515e0 2024-03-06T07:25:27.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.032+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f81429d0 con 0x7f25f81515e0 2024-03-06T07:25:27.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.033+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25e001a070 con 0x7f25f81515e0 2024-03-06T07:25:27.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.033+0000 7f25effff640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25d806b4d0 0x7f25d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.033+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25e00895a0 con 0x7f25f81515e0 2024-03-06T07:25:27.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.033+0000 7f25f6ffd640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25d806b4d0 0x7f25d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.034+0000 7f25f6ffd640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25d806b4d0 0x7f25d806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f25e8036000 tx=0x7f25e8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.038+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25e0058420 con 0x7f25f81515e0 2024-03-06T07:25:27.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.287+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f8068840 con 0x7f25f81515e0 2024-03-06T07:25:27.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.287+0000 7f25effff640 1 -- 172.21.15.122:0/922741109 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f25e0057d20 con 0x7f25f81515e0 2024-03-06T07:25:27.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:27.290 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:27.290 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.289+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25d806b4d0 msgr2=0x7f25d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25d806b4d0 0x7f25d806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f25e8036000 tx=0x7f25e8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 msgr2=0x7f25f8141df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f8141df0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c3c0 tx=0x7f25e000c880 comp rx=0 tx=0).stop 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 shutdown_connections 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f25d806b4d0 0x7f25d806d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f25f815b430 0x7f25f8140c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f81515e0 0x7f25f8141df0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 --2- 172.21.15.122:0/922741109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25f80cd7a0 0x7f25f8158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 >> 172.21.15.122:0/922741109 conn(0x7f25f80cf8a0 msgr2=0x7f25f80b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 shutdown_connections 2024-03-06T07:25:27.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.290+0000 7f25fd4b1640 1 -- 172.21.15.122:0/922741109 wait complete. 2024-03-06T07:25:27.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:27.423 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:25:27.423 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:25:27.424 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:27.424 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> import uuid 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> n = 100 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = False 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> if 0 > 0: 2024-03-06T07:25:27.439 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = True 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> dpath = os.path.dirname(path) 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> try: 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(n): 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> f.write(f"{i}") 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:25:27.440 DEBUG:teuthology.orchestra.run.smithi122:> f.flush() 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(f.fileno()) 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> os.unlink(fpath) 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> if create_hard_links: 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> for j in range(0): 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> finally: 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> os.close(dirfd) 2024-03-06T07:25:27.441 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:25:27.587 INFO:teuthology.orchestra.run.smithi122.stderr:Traceback (most recent call last): 2024-03-06T07:25:27.587 INFO:teuthology.orchestra.run.smithi122.stderr: File "", line 26, in 2024-03-06T07:25:27.587 INFO:teuthology.orchestra.run.smithi122.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:25:27.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:25:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:27.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:27.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.737+0000 7f11bc793640 1 Processor -- start 2024-03-06T07:25:27.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.737+0000 7f11bc793640 1 -- start start 2024-03-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11bc793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 0x7f11b40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11bc793640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b40b3430 con 0x7f11b415dfe0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11bc793640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b40b3710 con 0x7f11b40d22b0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11bc793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40b31e0 con 0x7f11b40d2710 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11ba52f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f11b415dfe0 0x7f11b40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40926/0 (socket says 172.21.15.122:40926) 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11ba52f640 1 -- 172.21.15.122:0/1359623744 learned_addr learned my addr 172.21.15.122:0/1359623744 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11bad30640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 0x7f11b40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153145824 0 0) 0x7f11b40b3430 con 0x7f11b415dfe0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a0003520 con 0x7f11b415dfe0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4062346418 0 0) 0x7f11a0003520 con 0x7f11b415dfe0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 msgr2=0x7f11b40b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 0x7f11b40b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 >> v1:172.21.15.169:6789/0 conn(0x7f11b40d22b0 legacy=0x7f11b40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b40b39f0 con 0x7f11b415dfe0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f11b0001670 con 0x7f11b415dfe0 2024-03-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 >> v1:172.21.15.122:6789/0 conn(0x7f11b415dfe0 legacy=0x7f11b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11a0004800 0x7f11a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 0x7f11a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40b31e0 con 0x7f11a00039e0 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a000baa0 con 0x7f11a0004800 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.738+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a000bca0 con 0x7f11b40d2710 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11b9d2e640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11a0004800 0x7f11a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11bad30640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 0x7f11a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11ba52f640 1 -- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 msgr2=0x7f11a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40d2710 0x7f11a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11ba52f640 1 -- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11a0004800 msgr2=0x7f11a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11a0004800 0x7f11a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.739+0000 7f11ba52f640 1 -- 172.21.15.122:0/1359623744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0003520 con 0x7f11a00039e0 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11a0003e20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f11b000c210 tx=0x7f11a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb239e21d23aef92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0007d10 con 0x7f11a00039e0 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b0030070 con 0x7f11a00039e0 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11b952d640 1 -- 172.21.15.122:0/1359623744 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0005cf0 con 0x7f11a00039e0 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bc793640 1 -- 172.21.15.122:0/1359623744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 msgr2=0x7f11a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bc793640 1 --2- 172.21.15.122:0/1359623744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11a0003e20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f11b000c210 tx=0x7f11a0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bad30640 1 -- 172.21.15.122:0/1359623744 reap_dead start 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bc793640 1 -- 172.21.15.122:0/1359623744 shutdown_connections 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bc793640 1 -- 172.21.15.122:0/1359623744 >> 172.21.15.122:0/1359623744 conn(0x7f11b40d3cd0 msgr2=0x7f11b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bc793640 1 -- 172.21.15.122:0/1359623744 shutdown_connections 2024-03-06T07:25:27.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.740+0000 7f11bc793640 1 -- 172.21.15.122:0/1359623744 wait complete. 2024-03-06T07:25:27.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.741+0000 7f11bc793640 1 Processor -- start 2024-03-06T07:25:27.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 -- start start 2024-03-06T07:25:27.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11b41f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40cbdf0 0x7f11b41f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11b41510a0 0x7f11b41f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40cfa00 con 0x7f11a00039e0 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40b3d60 con 0x7f11b41510a0 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bc793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40d1c70 con 0x7f11b40cbdf0 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11ba52f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11b41f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11ba52f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11b41f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41480/0 (socket says 172.21.15.122:41480) 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11ba52f640 1 -- 172.21.15.122:0/1750322756 learned_addr learned my addr 172.21.15.122:0/1750322756 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11bad30640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11b41510a0 0x7f11b41f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.742+0000 7f11b9d2e640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40cbdf0 0x7f11b41f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11ba52f640 1 -- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40cbdf0 msgr2=0x7f11b41f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40cbdf0 0x7f11b41f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11ba52f640 1 -- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11b41510a0 msgr2=0x7f11b41f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11b41510a0 0x7f11b41f7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11ba52f640 1 -- 172.21.15.122:0/1750322756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b41612b0 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11b9d2e640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40cbdf0 0x7f11b41f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11ba52f640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11b41f3470 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f11b0000f60 tx=0x7f11b40cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0007d10 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b0004bf0 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b4161590 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0007070 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.743+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b4150df0 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.744+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b404f330 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.745+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11b0007210 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.745+0000 7f11aaffd640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f119806b500 0x7f119806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.745+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11b00ab730 con 0x7f11a00039e0 2024-03-06T07:25:27.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.745+0000 7f11b9d2e640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f119806b500 0x7f119806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.746+0000 7f11b9d2e640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f119806b500 0x7f119806d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f11b40cc210 tx=0x7f11a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.749+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b007a640 con 0x7f11a00039e0 2024-03-06T07:25:27.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.963+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b415d3e0 con 0x7f11a00039e0 2024-03-06T07:25:27.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.964+0000 7f11aaffd640 1 -- 172.21.15.122:0/1750322756 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f11b0079f40 con 0x7f11a00039e0 2024-03-06T07:25:27.966 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:27.966 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:27.966 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f119806b500 msgr2=0x7f119806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f119806b500 0x7f119806d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f11b40cc210 tx=0x7f11a4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 msgr2=0x7f11b41f3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11b41f3470 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f11b0000f60 tx=0x7f11b40cfd40 comp rx=0 tx=0).stop 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 shutdown_connections 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f119806b500 0x7f119806d9c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11b41510a0 0x7f11b41f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11b40cbdf0 0x7f11b41f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 --2- 172.21.15.122:0/1750322756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11a00039e0 0x7f11b41f3470 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 >> 172.21.15.122:0/1750322756 conn(0x7f11b40d3cd0 msgr2=0x7f11b40d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 shutdown_connections 2024-03-06T07:25:27.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:27.966+0000 7f11bc793640 1 -- 172.21.15.122:0/1750322756 wait complete. 2024-03-06T07:25:27.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:28.093 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:25:28.094 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:25:28.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:28.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.193+0000 7fbcc560e640 1 Processor -- start 2024-03-06T07:25:28.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.193+0000 7fbcc560e640 1 -- start start 2024-03-06T07:25:28.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcc560e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbcc01424f0 0x7fbcc00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcc560e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc00cd7a0 0x7fbcc015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcc560e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc00cc980 con 0x7fbcc006d330 2024-03-06T07:25:28.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcc560e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc00cd220 con 0x7fbcc01424f0 2024-03-06T07:25:28.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcc560e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00ccc60 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcbeffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbcc01424f0 0x7fbcc00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcbeffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbcc01424f0 0x7fbcc00cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41492/0 (socket says 172.21.15.122:41492) 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcbf7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbcc006d330 0x7fbcc0071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40928/0 (socket says 172.21.15.122:40928) 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.194+0000 7fbcbeffd640 1 -- 172.21.15.122:0/3814016119 learned_addr learned my addr 172.21.15.122:0/3814016119 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcb7fff640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc00cd7a0 0x7fbcc015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcbeffd640 1 -- 172.21.15.122:0/3814016119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc00cd7a0 msgr2=0x7fbcc015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcbeffd640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc00cd7a0 0x7fbcc015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcbeffd640 1 -- 172.21.15.122:0/3814016119 >> v1:172.21.15.122:6789/0 conn(0x7fbcc006d330 legacy=0x7fbcc0071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcbeffd640 1 -- 172.21.15.122:0/3814016119 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00ccf40 con 0x7fbcc01424f0 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcbe7fc640 1 -- 172.21.15.122:0/3814016119 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588689231 0 0) 0x7fbcc00cc980 con 0x7fbcc006d330 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcb7fff640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc00cd7a0 0x7fbcc015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:28.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.195+0000 7fbcbeffd640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc01424f0 0x7fbcc00cc2f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009870 tx=0x7fbcb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0f9e9d98b132c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcbe7fc640 1 -- 172.21.15.122:0/3814016119 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0034070 con 0x7fbcc01424f0 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcbe7fc640 1 -- 172.21.15.122:0/3814016119 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb0030840 con 0x7fbcc01424f0 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcbe7fc640 1 -- 172.21.15.122:0/3814016119 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0030c10 con 0x7fbcc01424f0 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 -- 172.21.15.122:0/3814016119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc01424f0 msgr2=0x7fbcc00cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc01424f0 0x7fbcc00cc2f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009870 tx=0x7fbcb002b980 comp rx=0 tx=0).stop 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 -- 172.21.15.122:0/3814016119 shutdown_connections 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc00cd7a0 0x7fbcc015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 --2- 172.21.15.122:0/3814016119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc01424f0 0x7fbcc00cc2f0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 -- 172.21.15.122:0/3814016119 >> 172.21.15.122:0/3814016119 conn(0x7fbcc00cf8a0 msgr2=0x7fbcc00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 -- 172.21.15.122:0/3814016119 shutdown_connections 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.196+0000 7fbcc560e640 1 -- 172.21.15.122:0/3814016119 wait complete. 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 Processor -- start 2024-03-06T07:25:28.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 -- start start 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcc006d330 0x7fbcc015b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 0x7fbcc01432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc01424f0 0x7fbcc0143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc006d940 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc006d7c0 con 0x7fbcc006d330 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcc560e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc006dac0 con 0x7fbcc01424f0 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcb7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 0x7fbcc01432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcb7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 0x7fbcc01432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41494/0 (socket says 172.21.15.122:41494) 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.198+0000 7fbcb7fff640 1 -- 172.21.15.122:0/2405107298 learned_addr learned my addr 172.21.15.122:0/2405107298 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcbeffd640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcc006d330 0x7fbcc015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcbf7fe640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc01424f0 0x7fbcc0143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb7fff640 1 -- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc01424f0 msgr2=0x7fbcc0143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb7fff640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc01424f0 0x7fbcc0143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb7fff640 1 -- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcc006d330 msgr2=0x7fbcc015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb7fff640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcc006d330 0x7fbcc015b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb7fff640 1 -- 172.21.15.122:0/2405107298 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc01f1ce0 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcbeffd640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcc006d330 0x7fbcc015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb7fff640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 0x7fbcc01432e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7f0 tx=0x7fbca800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8015070 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8007a30 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc01459e0 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8007e00 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.199+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc01f2230 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.200+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc004f330 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.200+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbca8018070 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.201+0000 7fbcb77fe640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbca006b4d0 0x7fbca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.201+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbca808a0e0 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.201+0000 7fbcbeffd640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbca006b4d0 0x7fbca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.201+0000 7fbcbeffd640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbca006b4d0 0x7fbca006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0036000 tx=0x7fbcb0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.210+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbca8058ff0 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.423+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcc015f3a0 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.424+0000 7fbcb77fe640 1 -- 172.21.15.122:0/2405107298 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fbca80588f0 con 0x7fbcc00cd7a0 2024-03-06T07:25:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:28.426 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:28.426 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:28.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbca006b4d0 msgr2=0x7fbca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbca006b4d0 0x7fbca006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0036000 tx=0x7fbcb0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 msgr2=0x7fbcc01432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 0x7fbcc01432e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7f0 tx=0x7fbca800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 shutdown_connections 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbca006b4d0 0x7fbca006d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbcc01424f0 0x7fbcc0143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcc00cd7a0 0x7fbcc01432e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 --2- 172.21.15.122:0/2405107298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcc006d330 0x7fbcc015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 >> 172.21.15.122:0/2405107298 conn(0x7fbcc00cf8a0 msgr2=0x7fbcc00cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 shutdown_connections 2024-03-06T07:25:28.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:28.426+0000 7fbcc560e640 1 -- 172.21.15.122:0/2405107298 wait complete. 2024-03-06T07:25:28.442 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:28.558 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:25:28.559 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:25:28.559 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:25:33.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.675+0000 7f6752a9a640 1 Processor -- start 2024-03-06T07:25:33.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.675+0000 7f6752a9a640 1 -- start start 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6752a9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6752a9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f674c0ce920 0x7f674c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6752a9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c15ea30 con 0x7f674c071ec0 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6752a9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674c161b70 con 0x7f674c0ce920 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6752a9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c15ed10 con 0x7f674c06b220 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f674bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f674c0ce920 0x7f674c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6751037640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f674c071ec0 0x7f674c162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40942/0 (socket says 172.21.15.122:40942) 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6751037640 1 -- 172.21.15.122:0/2697280879 learned_addr learned my addr 172.21.15.122:0/2697280879 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:33.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f6750836640 1 --2- 172.21.15.122:0/2697280879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f674bfff640 1 -- 172.21.15.122:0/2697280879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 msgr2=0x7f674c15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f674bfff640 1 --2- 172.21.15.122:0/2697280879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c15e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f674bfff640 1 -- 172.21.15.122:0/2697280879 >> v1:172.21.15.122:6789/0 conn(0x7f674c071ec0 legacy=0x7f674c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.676+0000 7f674bfff640 1 -- 172.21.15.122:0/2697280879 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c15eff0 con 0x7f674c0ce920 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.677+0000 7f674bfff640 1 --2- 172.21.15.122:0/2697280879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c071980 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f67380097a0 tx=0x7f673802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329d51bd224e0ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.677+0000 7f674b7fe640 1 -- 172.21.15.122:0/2697280879 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738035020 con 0x7f674c0ce920 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.677+0000 7f674b7fe640 1 -- 172.21.15.122:0/2697280879 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738031830 con 0x7f674c0ce920 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.677+0000 7f674b7fe640 1 -- 172.21.15.122:0/2697280879 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738031c00 con 0x7f674c0ce920 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 -- 172.21.15.122:0/2697280879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 msgr2=0x7f674c071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 --2- 172.21.15.122:0/2697280879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c071980 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f67380097a0 tx=0x7f673802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 -- 172.21.15.122:0/2697280879 shutdown_connections 2024-03-06T07:25:33.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 --2- 172.21.15.122:0/2697280879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c071980 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 --2- 172.21.15.122:0/2697280879 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 -- 172.21.15.122:0/2697280879 >> 172.21.15.122:0/2697280879 conn(0x7f674c06c0c0 msgr2=0x7f674c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 -- 172.21.15.122:0/2697280879 shutdown_connections 2024-03-06T07:25:33.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.678+0000 7f6752a9a640 1 -- 172.21.15.122:0/2697280879 wait complete. 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.679+0000 7f6752a9a640 1 Processor -- start 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.679+0000 7f6752a9a640 1 -- start start 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c1504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f674c071ec0 0x7f674c1e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c15ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c0ce380 con 0x7f674c0ce920 2024-03-06T07:25:33.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c0ce200 con 0x7f674c071ec0 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c162710 con 0x7f674c06b220 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c15ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c15ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41510/0 (socket says 172.21.15.122:41510) 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 -- 172.21.15.122:0/3650994556 learned_addr learned my addr 172.21.15.122:0/3650994556 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f674bfff640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f674c071ec0 0x7f674c1e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6750836640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c1504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 -- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 msgr2=0x7f674c1504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c1504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 -- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f674c071ec0 msgr2=0x7f674c1e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f674c071ec0 0x7f674c1e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 -- 172.21.15.122:0/3650994556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c1e87b0 con 0x7f674c0ce920 2024-03-06T07:25:33.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6751037640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c15ffb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f674400cab0 tx=0x7f674400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744017070 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674400ac30 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c157d90 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744012400 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.680+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c158350 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.681+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c072b80 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.682+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f674401a070 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.682+0000 7f67497fa640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f673006b4d0 0x7f673006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.682+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6744089de0 con 0x7f674c0ce920 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.682+0000 7f6750836640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f673006b4d0 0x7f673006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.683+0000 7f6750836640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f673006b4d0 0x7f673006d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6740006fd0 tx=0x7f6740008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.687+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f674c072b80 con 0x7f674c0ce920 2024-03-06T07:25:33.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.958+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f674c06be60 con 0x7f674c0ce920 2024-03-06T07:25:33.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.958+0000 7f67497fa640 1 -- 172.21.15.122:0/3650994556 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f674c06be60 con 0x7f674c0ce920 2024-03-06T07:25:33.961 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:33.961 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:33.961 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:33.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f673006b4d0 msgr2=0x7f673006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f673006b4d0 0x7f673006d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6740006fd0 tx=0x7f6740008040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 msgr2=0x7f674c15ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c15ffb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f674400cab0 tx=0x7f674400cf70 comp rx=0 tx=0).stop 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 shutdown_connections 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f673006b4d0 0x7f673006d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c0ce920 0x7f674c15ffb0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f674c071ec0 0x7f674c1e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 --2- 172.21.15.122:0/3650994556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f674c06b220 0x7f674c1504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 >> 172.21.15.122:0/3650994556 conn(0x7f674c06c0c0 msgr2=0x7f674c0ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 shutdown_connections 2024-03-06T07:25:33.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:33.960+0000 7f6752a9a640 1 -- 172.21.15.122:0/3650994556 wait complete. 2024-03-06T07:25:33.973 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:34.088 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:25:34.088 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:25:34.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:34.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.190+0000 7ff827956640 1 Processor -- start 2024-03-06T07:25:34.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.190+0000 7ff827956640 1 -- start start 2024-03-06T07:25:34.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff827956640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff8200cfff0 0x7ff8200d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff827956640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820161980 con 0x7ff8200cc8b0 2024-03-06T07:25:34.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff827956640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820161c60 con 0x7ff82015bb20 2024-03-06T07:25:34.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff827956640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8200d2500 con 0x7ff8200cfff0 2024-03-06T07:25:34.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff824ef1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff8200cfff0 0x7ff8200d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff825ef3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff8200cc8b0 0x7ff8200ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40956/0 (socket says 172.21.15.122:40956) 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff825ef3640 1 -- 172.21.15.122:0/263122340 learned_addr learned my addr 172.21.15.122:0/263122340 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff817fff640 1 -- 172.21.15.122:0/263122340 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934017964 0 0) 0x7ff820161980 con 0x7ff8200cc8b0 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff817fff640 1 -- 172.21.15.122:0/263122340 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c003520 con 0x7ff8200cc8b0 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff824ef1640 1 -- 172.21.15.122:0/263122340 >> v1:172.21.15.169:6789/0 conn(0x7ff82015bb20 legacy=0x7ff8200d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff824ef1640 1 -- 172.21.15.122:0/263122340 >> v1:172.21.15.122:6789/0 conn(0x7ff8200cc8b0 legacy=0x7ff8200ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.191+0000 7ff824ef1640 1 -- 172.21.15.122:0/263122340 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff820161f40 con 0x7ff8200cfff0 2024-03-06T07:25:34.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.192+0000 7ff824ef1640 1 --2- 172.21.15.122:0/263122340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8200cfff0 0x7ff8200d04a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff8100097a0 tx=0x7ff81002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991caea0efa94623 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.192+0000 7ff817fff640 1 -- 172.21.15.122:0/263122340 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810034070 con 0x7ff8200cfff0 2024-03-06T07:25:34.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.192+0000 7ff817fff640 1 -- 172.21.15.122:0/263122340 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff810030610 con 0x7ff8200cfff0 2024-03-06T07:25:34.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.192+0000 7ff817fff640 1 -- 172.21.15.122:0/263122340 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8100309c0 con 0x7ff8200cfff0 2024-03-06T07:25:34.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 -- 172.21.15.122:0/263122340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8200cfff0 msgr2=0x7ff8200d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 --2- 172.21.15.122:0/263122340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8200cfff0 0x7ff8200d04a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff8100097a0 tx=0x7ff81002b960 comp rx=0 tx=0).stop 2024-03-06T07:25:34.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 -- 172.21.15.122:0/263122340 shutdown_connections 2024-03-06T07:25:34.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 --2- 172.21.15.122:0/263122340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8200cfff0 0x7ff8200d04a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 -- 172.21.15.122:0/263122340 >> 172.21.15.122:0/263122340 conn(0x7ff82009a9f0 msgr2=0x7ff82009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 -- 172.21.15.122:0/263122340 shutdown_connections 2024-03-06T07:25:34.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.193+0000 7ff827956640 1 -- 172.21.15.122:0/263122340 wait complete. 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.194+0000 7ff827956640 1 Processor -- start 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.194+0000 7ff827956640 1 -- start start 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8200cc8b0 0x7ff82014aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8200cfff0 0x7ff82015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 0x7ff82015a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8200d1980 con 0x7ff82015bb20 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820162500 con 0x7ff8200cfff0 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8200d1b00 con 0x7ff8200cc8b0 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 0x7ff82015a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 0x7ff82015a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41532/0 (socket says 172.21.15.122:41532) 2024-03-06T07:25:34.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 -- 172.21.15.122:0/2058962539 learned_addr learned my addr 172.21.15.122:0/2058962539 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 -- 172.21.15.122:0/2058962539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8200cc8b0 msgr2=0x7ff82014aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8200cc8b0 0x7ff82014aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 -- 172.21.15.122:0/2058962539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8200cfff0 msgr2=0x7ff82015a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8200cfff0 0x7ff82015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 -- 172.21.15.122:0/2058962539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82014b430 con 0x7ff82015bb20 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff825ef3640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 0x7ff82015a5f0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff81800caa0 tx=0x7ff81800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818017070 con 0x7ff82015bb20 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81800ac00 con 0x7ff82015bb20 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8200c8f00 con 0x7ff82015bb20 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.195+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818012400 con 0x7ff82015bb20 2024-03-06T07:25:34.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.196+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8200c94c0 con 0x7ff82015bb20 2024-03-06T07:25:34.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.197+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82004f330 con 0x7ff82015bb20 2024-03-06T07:25:34.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.197+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff81801a070 con 0x7ff82015bb20 2024-03-06T07:25:34.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.197+0000 7ff815ffb640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff80406b4d0 0x7ff80406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.197+0000 7ff8256f2640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff80406b4d0 0x7ff80406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.197+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff81808a040 con 0x7ff82015bb20 2024-03-06T07:25:34.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.198+0000 7ff8256f2640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff80406b4d0 0x7ff80406d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff81c000f60 tx=0x7ff81c0079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.202+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff82014b430 con 0x7ff82015bb20 2024-03-06T07:25:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.412+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8200747a0 con 0x7ff82015bb20 2024-03-06T07:25:34.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.413+0000 7ff815ffb640 1 -- 172.21.15.122:0/2058962539 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7ff818058e80 con 0x7ff82015bb20 2024-03-06T07:25:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:34.414 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:34.414 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.414+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff80406b4d0 msgr2=0x7ff80406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.414+0000 7ff827956640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff80406b4d0 0x7ff80406d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff81c000f60 tx=0x7ff81c0079d0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.414+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 msgr2=0x7ff82015a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.414+0000 7ff827956640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 0x7ff82015a5f0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff81800caa0 tx=0x7ff81800cf60 comp rx=0 tx=0).stop 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 shutdown_connections 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff80406b4d0 0x7ff80406d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82015bb20 0x7ff82015a5f0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8200cfff0 0x7ff82015a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 --2- 172.21.15.122:0/2058962539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8200cc8b0 0x7ff82014aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 >> 172.21.15.122:0/2058962539 conn(0x7ff82009a9f0 msgr2=0x7ff820070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 shutdown_connections 2024-03-06T07:25:34.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34.415+0000 7ff827956640 1 -- 172.21.15.122:0/2058962539 wait complete. 2024-03-06T07:25:34.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:25:34.545 INFO:teuthology.orchestra.run.smithi122.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:25:34.545 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:25:34.546 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:34.546 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:34.561 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:25:34.561 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:25:34.561 DEBUG:teuthology.orchestra.run.smithi122:> import uuid 2024-03-06T07:25:34.561 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> n = 10 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = False 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> if 0 > 0: 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> create_hard_links = True 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> dpath = os.path.dirname(path) 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> try: 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:25:34.562 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(n): 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> f.write(f"{i}") 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> f.flush() 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(f.fileno()) 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> os.unlink(fpath) 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> if True: 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> if create_hard_links: 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> for j in range(0): 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> if False: 2024-03-06T07:25:34.563 DEBUG:teuthology.orchestra.run.smithi122:> os.fsync(dirfd) 2024-03-06T07:25:34.564 DEBUG:teuthology.orchestra.run.smithi122:> finally: 2024-03-06T07:25:34.564 DEBUG:teuthology.orchestra.run.smithi122:> os.close(dirfd) 2024-03-06T07:25:34.564 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:34.736 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:34.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:34.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:34.864 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:34.864 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:34.864 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:34.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.022+0000 7fcc027d7640 1 Processor -- start 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.022+0000 7fcc027d7640 1 -- start start 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcc027d7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcbfc15dfe0 0x7fcbfc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcc027d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 0x7fcbfc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcc027d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbfc0b3430 con 0x7fcbfc0cf830 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcc027d7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbfc0b31e0 con 0x7fcbfc15dfe0 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcc027d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0b3d60 con 0x7fcbfc0d29c0 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcbfb7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcbfc0cf830 0x7fcbfc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36422/0 (socket says 172.21.15.122:36422) 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcbfb7fe640 1 -- 172.21.15.122:0/939996487 learned_addr learned my addr 172.21.15.122:0/939996487 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181157587 0 0) 0x7fcbfc0b3430 con 0x7fcbfc0cf830 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe4003520 con 0x7fcbfc0cf830 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcc00d74640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 0x7fcbfc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.023+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/939996487 >> v2:172.21.15.169:3300/0 conn(0x7fcbfc15dfe0 0x7fcbfc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765261925 0 0) 0x7fcbe4003520 con 0x7fcbfc0cf830 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 msgr2=0x7fcbfc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 0x7fcbfc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 >> v2:172.21.15.169:3300/0 conn(0x7fcbfc15dfe0 msgr2=0x7fcbfc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 --2- 172.21.15.122:0/939996487 >> v2:172.21.15.169:3300/0 conn(0x7fcbfc15dfe0 0x7fcbfc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc0b3710 con 0x7fcbfc0cf830 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fcbf0001670 con 0x7fcbfc0cf830 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/939996487 >> v2:172.21.15.169:3300/0 conn(0x7fcbfc15dfe0 0x7fcbfc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 >> v1:172.21.15.122:6789/0 conn(0x7fcbfc0cf830 legacy=0x7fcbfc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 0x7fcbe40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbe4003fe0 0x7fcbe4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbe4004d40 0x7fcbe40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0b31e0 con 0x7fcbe4003fe0 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0b3d60 con 0x7fcbe4004d40 2024-03-06T07:25:35.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe400bac0 con 0x7fcbfc0d29c0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfb7fe640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbe4003fe0 0x7fcbe4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 0x7fcbe40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcc00d74640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbe4004d40 0x7fcbe40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfb7fe640 1 -- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 msgr2=0x7fcbe40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfb7fe640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbfc0d29c0 0x7fcbe40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfb7fe640 1 -- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbe4004d40 msgr2=0x7fcbe40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfb7fe640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbe4004d40 0x7fcbe40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.024+0000 7fcbfb7fe640 1 -- 172.21.15.122:0/939996487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe4003520 con 0x7fcbe4003fe0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcbfb7fe640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbe4003fe0 0x7fcbe4004800 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0cfc70 tx=0x7fcbe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106ce083765c8303 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0007d50 con 0x7fcbe4003fe0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf0030070 con 0x7fcbe4003fe0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcbfaffd640 1 -- 172.21.15.122:0/939996487 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0005d10 con 0x7fcbe4003fe0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcc027d7640 1 -- 172.21.15.122:0/939996487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbe4003fe0 msgr2=0x7fcbe4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcc027d7640 1 --2- 172.21.15.122:0/939996487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbe4003fe0 0x7fcbe4004800 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0cfc70 tx=0x7fcbe4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.025+0000 7fcc00d74640 1 -- 172.21.15.122:0/939996487 reap_dead start 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.026+0000 7fcc027d7640 1 -- 172.21.15.122:0/939996487 shutdown_connections 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.026+0000 7fcc027d7640 1 -- 172.21.15.122:0/939996487 >> 172.21.15.122:0/939996487 conn(0x7fcbfc0d3cd0 msgr2=0x7fcbfc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.026+0000 7fcc027d7640 1 -- 172.21.15.122:0/939996487 shutdown_connections 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.026+0000 7fcc027d7640 1 -- 172.21.15.122:0/939996487 wait complete. 2024-03-06T07:25:35.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.027+0000 7fcc027d7640 1 Processor -- start 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.027+0000 7fcc027d7640 1 -- start start 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbe4003fe0 0x7fcbfc14e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbfc14fbe0 0x7fcbfc14cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 0x7fcbfc14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0d1df0 con 0x7fcbfc0cd130 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0d1c70 con 0x7fcbfc14fbe0 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc15e3c0 con 0x7fcbe4003fe0 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 0x7fcbfc14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 0x7fcbfc14efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60644/0 (socket says 172.21.15.122:60644) 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 -- 172.21.15.122:0/1212523982 learned_addr learned my addr 172.21.15.122:0/1212523982 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbe4003fe0 0x7fcbfc14e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbfb7fe640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbfc14fbe0 0x7fcbfc14cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 -- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbe4003fe0 msgr2=0x7fcbfc14e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbe4003fe0 0x7fcbfc14e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 -- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbfc14fbe0 msgr2=0x7fcbfc14cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbfc14fbe0 0x7fcbfc14cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 -- 172.21.15.122:0/1212523982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc14ed70 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbe4003fe0 0x7fcbfc14e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbfb7fe640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbfc14fbe0 0x7fcbfc14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc00d74640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 0x7fcbfc14efd0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc161bf0 tx=0x7fcbf4007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40046a0 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf401a070 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbfc14d790 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4015640 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.028+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbfc14dd50 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.030+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbfc04f330 con 0x7fcbfc0cd130 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.030+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbf40157e0 con 0x7fcbfc0cd130 2024-03-06T07:25:35.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.030+0000 7fcbf8ff9640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcbe806b500 0x7fcbe806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.030+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcbf408a8e0 con 0x7fcbfc0cd130 2024-03-06T07:25:35.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.030+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcbe806b500 0x7fcbe806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.031+0000 7fcbfbfff640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcbe806b500 0x7fcbe806d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006fd0 tx=0x7fcbec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.038+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbf4024020 con 0x7fcbfc0cd130 2024-03-06T07:25:35.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.234+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbfc0b4e80 con 0x7fcbfc0cd130 2024-03-06T07:25:35.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.235+0000 7fcbf8ff9640 1 -- 172.21.15.122:0/1212523982 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16765 (secure 0 0 0) 0x7fcbf4059720 con 0x7fcbfc0cd130 2024-03-06T07:25:35.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:35.237 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":71,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:34.941157+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcbe806b500 msgr2=0x7fcbe806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcbe806b500 0x7fcbe806d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006fd0 tx=0x7fcbec008040 comp rx=0 tx=0).stop 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 msgr2=0x7fcbfc14efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 0x7fcbfc14efd0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc161bf0 tx=0x7fcbf4007700 comp rx=0 tx=0).stop 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 shutdown_connections 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcbe806b500 0x7fcbe806d9c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbfc0cd130 0x7fcbfc14efd0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbfc14fbe0 0x7fcbfc14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 --2- 172.21.15.122:0/1212523982 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcbe4003fe0 0x7fcbfc14e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 >> 172.21.15.122:0/1212523982 conn(0x7fcbfc0d3cd0 msgr2=0x7fcbfc0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 shutdown_connections 2024-03-06T07:25:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.238+0000 7fcc027d7640 1 -- 172.21.15.122:0/1212523982 wait complete. 2024-03-06T07:25:35.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:03.431204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:04.175007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:35.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:35.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.352+0000 7f5c82c66640 1 Processor -- start 2024-03-06T07:25:35.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.352+0000 7f5c82c66640 1 -- start start 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c82c66640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5c7c1614d0 0x7f5c7c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c82c66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d1910 0x7f5c7c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c82c66640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c0d2740 con 0x7f5c7c0d22b0 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c82c66640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7c1618b0 con 0x7f5c7c1614d0 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c82c66640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0cf400 con 0x7f5c7c0d1910 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c81203640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5c7c0d22b0 0x7f5c7c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36436/0 (socket says 172.21.15.122:36436) 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c81203640 1 -- 172.21.15.122:0/2275773295 learned_addr learned my addr 172.21.15.122:0/2275773295 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c7bfff640 1 --2- 172.21.15.122:0/2275773295 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d1910 0x7f5c7c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.353+0000 7f5c80a02640 1 --2- 172.21.15.122:0/2275773295 >> v2:172.21.15.169:3300/0 conn(0x7f5c7c1614d0 0x7f5c7c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c7b7fe640 1 -- 172.21.15.122:0/2275773295 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245379117 0 0) 0x7f5c74001490 con 0x7f5c7c0d22b0 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c7b7fe640 1 -- 172.21.15.122:0/2275773295 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c003520 con 0x7f5c7c0d22b0 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c7bfff640 1 -- 172.21.15.122:0/2275773295 >> v2:172.21.15.169:3300/0 conn(0x7f5c7c1614d0 msgr2=0x7f5c7c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c7bfff640 1 --2- 172.21.15.122:0/2275773295 >> v2:172.21.15.169:3300/0 conn(0x7f5c7c1614d0 0x7f5c7c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c7bfff640 1 -- 172.21.15.122:0/2275773295 >> v1:172.21.15.122:6789/0 conn(0x7f5c7c0d22b0 legacy=0x7f5c7c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c7bfff640 1 -- 172.21.15.122:0/2275773295 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c0cf6a0 con 0x7f5c7c0d1910 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.354+0000 7f5c80a02640 1 --2- 172.21.15.122:0/2275773295 >> v2:172.21.15.169:3300/0 conn(0x7f5c7c1614d0 0x7f5c7c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:35.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c7bfff640 1 --2- 172.21.15.122:0/2275773295 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d1910 0x7f5c7c0d1d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c680097a0 tx=0x7f5c6802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=739007f11bb45d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c7b7fe640 1 -- 172.21.15.122:0/2275773295 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68035020 con 0x7f5c7c0d1910 2024-03-06T07:25:35.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c7b7fe640 1 -- 172.21.15.122:0/2275773295 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68031830 con 0x7f5c7c0d1910 2024-03-06T07:25:35.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c7b7fe640 1 -- 172.21.15.122:0/2275773295 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68031c00 con 0x7f5c7c0d1910 2024-03-06T07:25:35.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 -- 172.21.15.122:0/2275773295 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d1910 msgr2=0x7f5c7c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 --2- 172.21.15.122:0/2275773295 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d1910 0x7f5c7c0d1d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c680097a0 tx=0x7f5c6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 -- 172.21.15.122:0/2275773295 shutdown_connections 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 --2- 172.21.15.122:0/2275773295 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d1910 0x7f5c7c0d1d70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 --2- 172.21.15.122:0/2275773295 >> v2:172.21.15.169:3300/0 conn(0x7f5c7c1614d0 0x7f5c7c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 -- 172.21.15.122:0/2275773295 >> 172.21.15.122:0/2275773295 conn(0x7f5c7c0d3d40 msgr2=0x7f5c7c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 -- 172.21.15.122:0/2275773295 shutdown_connections 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.355+0000 7f5c82c66640 1 -- 172.21.15.122:0/2275773295 wait complete. 2024-03-06T07:25:35.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.357+0000 7f5c82c66640 1 Processor -- start 2024-03-06T07:25:35.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.357+0000 7f5c82c66640 1 -- start start 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c0d1910 0x7f5c7c153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d22b0 0x7f5c7c15ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 0x7f5c7c160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0b0910 con 0x7f5c7c1614d0 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0cfc60 con 0x7f5c7c0d1910 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0b1e40 con 0x7f5c7c0d22b0 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 0x7f5c7c160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 0x7f5c7c160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60650/0 (socket says 172.21.15.122:60650) 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 -- 172.21.15.122:0/3172364999 learned_addr learned my addr 172.21.15.122:0/3172364999 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:35.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c7bfff640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d22b0 0x7f5c7c15ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c80a02640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c0d1910 0x7f5c7c153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 -- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d22b0 msgr2=0x7f5c7c15ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d22b0 0x7f5c7c15ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 -- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c0d1910 msgr2=0x7f5c7c153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c0d1910 0x7f5c7c153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 -- 172.21.15.122:0/3172364999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c0c7ac0 con 0x7f5c7c1614d0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c80a02640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c0d1910 0x7f5c7c153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c81203640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 0x7f5c7c160520 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400cb50 tx=0x7f5c7400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74017070 con 0x7f5c7c1614d0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7400ad40 con 0x7f5c7c1614d0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7c0c7da0 con 0x7f5c7c1614d0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74012620 con 0x7f5c7c1614d0 2024-03-06T07:25:35.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.359+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7c0c8360 con 0x7f5c7c1614d0 2024-03-06T07:25:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.360+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7c068ff0 con 0x7f5c7c1614d0 2024-03-06T07:25:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.361+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c7401a070 con 0x7f5c7c1614d0 2024-03-06T07:25:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.361+0000 7f5c797fa640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.361+0000 7f5c80a02640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.361+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5c7408a0c0 con 0x7f5c7c1614d0 2024-03-06T07:25:35.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.362+0000 7f5c80a02640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009a20 tx=0x7f5c70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.366+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c7c0d2740 con 0x7f5c7c1614d0 2024-03-06T07:25:35.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.541+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c7c0b5040 con 0x7f5c7c1614d0 2024-03-06T07:25:35.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.541+0000 7f5c797fa640 1 -- 172.21.15.122:0/3172364999 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c74058f00 con 0x7f5c7c1614d0 2024-03-06T07:25:35.543 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 msgr2=0x7f5c6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009a20 tx=0x7f5c70008040 comp rx=0 tx=0).stop 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 msgr2=0x7f5c7c160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 0x7f5c7c160520 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400cb50 tx=0x7f5c7400a430 comp rx=0 tx=0).stop 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 shutdown_connections 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c6006b4d0 0x7f5c6006d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c7c1614d0 0x7f5c7c160520 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c7c0d22b0 0x7f5c7c15ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 --2- 172.21.15.122:0/3172364999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c7c0d1910 0x7f5c7c153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 >> 172.21.15.122:0/3172364999 conn(0x7f5c7c0d3d40 msgr2=0x7f5c7c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 shutdown_connections 2024-03-06T07:25:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.544+0000 7f5c82c66640 1 -- 172.21.15.122:0/3172364999 wait complete. 2024-03-06T07:25:35.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:35.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.659+0000 7f05ca381640 1 Processor -- start 2024-03-06T07:25:35.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.659+0000 7f05ca381640 1 -- start start 2024-03-06T07:25:35.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05ca381640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f05c40d4860 0x7f05c40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05ca381640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c41618b0 con 0x7f05c40b0e30 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05ca381640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c40cf400 con 0x7f05c41614d0 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05ca381640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c40d4cf0 con 0x7f05c40d4860 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c891e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f05c40b0e30 0x7f05c40b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36446/0 (socket says 172.21.15.122:36446) 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c891e640 1 -- 172.21.15.122:0/165696544 learned_addr learned my addr 172.21.15.122:0/165696544 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772546744 0 0) 0x7f05c41618b0 con 0x7f05c40b0e30 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ac003520 con 0x7f05c40b0e30 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c37fe640 1 --2- 172.21.15.122:0/165696544 >> v2:172.21.15.169:3300/0 conn(0x7f05c40d4860 0x7f05c40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680914616 0 0) 0x7f05ac003520 con 0x7f05c40b0e30 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 >> v2:172.21.15.169:3300/0 conn(0x7f05c40d4860 msgr2=0x7f05c40d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 --2- 172.21.15.122:0/165696544 >> v2:172.21.15.169:3300/0 conn(0x7f05c40d4860 0x7f05c40d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 >> v1:172.21.15.169:6789/0 conn(0x7f05c41614d0 legacy=0x7f05c40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c40cf6a0 con 0x7f05c40b0e30 2024-03-06T07:25:35.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f05bc001a10 con 0x7f05c40b0e30 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c37fe640 1 --2- 172.21.15.122:0/165696544 >> v2:172.21.15.169:3300/0 conn(0x7f05c40d4860 0x7f05c40d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 >> v1:172.21.15.122:6789/0 conn(0x7f05c40b0e30 legacy=0x7f05c40b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0039b0 0x7f05ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0051c0 0x7f05ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40d4cf0 con 0x7f05ac004800 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac00c110 con 0x7f05ac0051c0 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.660+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac00c310 con 0x7f05ac0039b0 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c37fe640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c3fff640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0039b0 0x7f05ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c891e640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0051c0 0x7f05ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c37fe640 1 -- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0039b0 msgr2=0x7f05ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c37fe640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac0039b0 0x7f05ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c37fe640 1 -- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0051c0 msgr2=0x7f05ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c37fe640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05ac0051c0 0x7f05ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c891e640 1 -- 172.21.15.122:0/165696544 reap_dead start 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.661+0000 7f05c37fe640 1 -- 172.21.15.122:0/165696544 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac003520 con 0x7f05ac004800 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05c37fe640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f05b40098a0 tx=0x7f05ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218a72a48aaff6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400f040 con 0x7f05ac004800 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4004230 con 0x7f05ac004800 2024-03-06T07:25:35.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05c2ffd640 1 -- 172.21.15.122:0/165696544 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4003610 con 0x7f05ac004800 2024-03-06T07:25:35.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 -- 172.21.15.122:0/165696544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac004800 msgr2=0x7f05ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f05b40098a0 tx=0x7f05ac005830 comp rx=0 tx=0).stop 2024-03-06T07:25:35.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 -- 172.21.15.122:0/165696544 shutdown_connections 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 --2- 172.21.15.122:0/165696544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 -- 172.21.15.122:0/165696544 >> 172.21.15.122:0/165696544 conn(0x7f05c40d3d40 msgr2=0x7f05c40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 -- 172.21.15.122:0/165696544 shutdown_connections 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.662+0000 7f05ca381640 1 -- 172.21.15.122:0/165696544 wait complete. 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.663+0000 7f05ca381640 1 Processor -- start 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 -- start start 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac004800 0x7f05c415a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 0x7f05c40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05c40c88f0 0x7f05c40c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40b0910 con 0x7f05c40c96c0 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40cfc60 con 0x7f05c40c88f0 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05ca381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40d1910 con 0x7f05ac004800 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 0x7f05c40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05c37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 0x7f05c40cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60658/0 (socket says 172.21.15.122:60658) 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05c37fe640 1 -- 172.21.15.122:0/2719258229 learned_addr learned my addr 172.21.15.122:0/2719258229 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05c3fff640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac004800 0x7f05c415a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.664+0000 7f05c891e640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05c40c88f0 0x7f05c40c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c37fe640 1 -- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac004800 msgr2=0x7f05c415a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c37fe640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac004800 0x7f05c415a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c37fe640 1 -- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05c40c88f0 msgr2=0x7f05c40c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c37fe640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05c40c88f0 0x7f05c40c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c37fe640 1 -- 172.21.15.122:0/2719258229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c40c85c0 con 0x7f05c40c96c0 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c37fe640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 0x7f05c40cae10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f05b40099c0 tx=0x7f05b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4048070 con 0x7f05c40c96c0 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4030430 con 0x7f05c40c96c0 2024-03-06T07:25:35.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c40c8000 con 0x7f05c40c96c0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b40307e0 con 0x7f05c40c96c0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.665+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c40cdb20 con 0x7f05c40c96c0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.666+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c4068ff0 con 0x7f05c40c96c0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.667+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05b404b070 con 0x7f05c40c96c0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.667+0000 7f05c0ff9640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05b006b500 0x7f05b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.667+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05b40abcf0 con 0x7f05c40c96c0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.667+0000 7f05c3fff640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05b006b500 0x7f05b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.668+0000 7f05c3fff640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05b006b500 0x7f05b006d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f05c40b5480 tx=0x7f05b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.672+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05b4034020 con 0x7f05c40c96c0 2024-03-06T07:25:35.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.909+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05c40cdec0 con 0x7f05c40c96c0 2024-03-06T07:25:35.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.909+0000 7f05c0ff9640 1 -- 172.21.15.122:0/2719258229 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f05b407a4f0 con 0x7f05c40c96c0 2024-03-06T07:25:35.911 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05b006b500 msgr2=0x7f05b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05b006b500 0x7f05b006d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f05c40b5480 tx=0x7f05b8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 msgr2=0x7f05c40cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 0x7f05c40cae10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f05b40099c0 tx=0x7f05b4004730 comp rx=0 tx=0).stop 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 shutdown_connections 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f05b006b500 0x7f05b006d9c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05c40c88f0 0x7f05c40c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05c40c96c0 0x7f05c40cae10 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 --2- 172.21.15.122:0/2719258229 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f05ac004800 0x7f05c415a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.911+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 >> 172.21.15.122:0/2719258229 conn(0x7f05c40d3d40 msgr2=0x7f05c406c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.912+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 shutdown_connections 2024-03-06T07:25:35.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:35.912+0000 7f05ca381640 1 -- 172.21.15.122:0/2719258229 wait complete. 2024-03-06T07:25:35.923 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:36.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 Processor -- start 2024-03-06T07:25:36.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 -- start start 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0b440d4860 0x7f0b440d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0b440b0e30 0x7f0b440b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b441618b0 con 0x7f0b441614d0 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b440d4cf0 con 0x7f0b440b0e30 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b4a5d7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b440cf400 con 0x7f0b440d4860 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b48b74640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0b440b0e30 0x7f0b440b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.027+0000 7f0b43fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0b441614d0 0x7f0b440d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36450/0 (socket says 172.21.15.122:36450) 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b48b74640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0b440b0e30 0x7f0b440b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60668/0 (socket says 172.21.15.122:60668) 2024-03-06T07:25:36.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b43fff640 1 -- 172.21.15.122:0/953197455 learned_addr learned my addr 172.21.15.122:0/953197455 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:36.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b437fe640 1 --2- 172.21.15.122:0/953197455 >> v2:172.21.15.169:3300/0 conn(0x7f0b440d4860 0x7f0b440d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b48b74640 1 -- 172.21.15.122:0/953197455 >> v2:172.21.15.169:3300/0 conn(0x7f0b440d4860 msgr2=0x7f0b440d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b48b74640 1 --2- 172.21.15.122:0/953197455 >> v2:172.21.15.169:3300/0 conn(0x7f0b440d4860 0x7f0b440d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b48b74640 1 -- 172.21.15.122:0/953197455 >> v1:172.21.15.122:6789/0 conn(0x7f0b441614d0 legacy=0x7f0b440d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b48b74640 1 -- 172.21.15.122:0/953197455 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b440cf6a0 con 0x7f0b440b0e30 2024-03-06T07:25:36.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.028+0000 7f0b42ffd640 1 -- 172.21.15.122:0/953197455 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845566599 0 0) 0x7f0b441618b0 con 0x7f0b441614d0 2024-03-06T07:25:36.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b48b74640 1 --2- 172.21.15.122:0/953197455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b440b0e30 0x7f0b440b1260 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0b440ce550 tx=0x7f0b3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8edb391979c2770 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b42ffd640 1 -- 172.21.15.122:0/953197455 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c039070 con 0x7f0b440b0e30 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b42ffd640 1 -- 172.21.15.122:0/953197455 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c02cc80 con 0x7f0b440b0e30 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b42ffd640 1 -- 172.21.15.122:0/953197455 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c034660 con 0x7f0b440b0e30 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/953197455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b440b0e30 msgr2=0x7f0b440b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/953197455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b440b0e30 0x7f0b440b1260 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0b440ce550 tx=0x7f0b3c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/953197455 shutdown_connections 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/953197455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b440b0e30 0x7f0b440b1260 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/953197455 >> v2:172.21.15.169:3300/0 conn(0x7f0b440d4860 0x7f0b440d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/953197455 >> 172.21.15.122:0/953197455 conn(0x7f0b440d3d40 msgr2=0x7f0b440d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/953197455 shutdown_connections 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.029+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/953197455 wait complete. 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.030+0000 7f0b4a5d7640 1 Processor -- start 2024-03-06T07:25:36.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 -- start start 2024-03-06T07:25:36.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b440b0e30 0x7f0b440cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b440d4860 0x7f0b440b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 0x7f0b440b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b440b0910 con 0x7f0b441614d0 2024-03-06T07:25:36.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b440cfc60 con 0x7f0b440b0e30 2024-03-06T07:25:36.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b4a5d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b440d2560 con 0x7f0b440d4860 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b48b74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 0x7f0b440b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b48b74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 0x7f0b440b8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60670/0 (socket says 172.21.15.122:60670) 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b48b74640 1 -- 172.21.15.122:0/1694583951 learned_addr learned my addr 172.21.15.122:0/1694583951 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b43fff640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b440b0e30 0x7f0b440cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.031+0000 7f0b437fe640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b440d4860 0x7f0b440b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b48b74640 1 -- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b440d4860 msgr2=0x7f0b440b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b48b74640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b440d4860 0x7f0b440b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b48b74640 1 -- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b440b0e30 msgr2=0x7f0b440cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b48b74640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b440b0e30 0x7f0b440cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b48b74640 1 -- 172.21.15.122:0/1694583951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b440cdd50 con 0x7f0b441614d0 2024-03-06T07:25:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b48b74640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 0x7f0b440b8ed0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c037040 tx=0x7f0b3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c044020 con 0x7f0b441614d0 2024-03-06T07:25:36.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c03a040 con 0x7f0b441614d0 2024-03-06T07:25:36.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b440c9490 con 0x7f0b441614d0 2024-03-06T07:25:36.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c039050 con 0x7f0b441614d0 2024-03-06T07:25:36.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.032+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b440c9a50 con 0x7f0b441614d0 2024-03-06T07:25:36.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.033+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b44068ff0 con 0x7f0b441614d0 2024-03-06T07:25:36.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.034+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b3c003b90 con 0x7f0b441614d0 2024-03-06T07:25:36.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.034+0000 7f0b40ff9640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b3006b4d0 0x7f0b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.034+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0b3c0ac9e0 con 0x7f0b441614d0 2024-03-06T07:25:36.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.034+0000 7f0b43fff640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b3006b4d0 0x7f0b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.035+0000 7f0b43fff640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b3006b4d0 0x7f0b3006d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0b34000f60 tx=0x7f0b34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.039+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b3c04e020 con 0x7f0b441614d0 2024-03-06T07:25:36.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.213+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b440d20a0 con 0x7f0b441614d0 2024-03-06T07:25:36.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.214+0000 7f0b40ff9640 1 -- 172.21.15.122:0/1694583951 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b3c07b260 con 0x7f0b441614d0 2024-03-06T07:25:36.215 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b3006b4d0 msgr2=0x7f0b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b3006b4d0 0x7f0b3006d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0b34000f60 tx=0x7f0b34009040 comp rx=0 tx=0).stop 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 msgr2=0x7f0b440b8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 0x7f0b440b8ed0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c037040 tx=0x7f0b3c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 shutdown_connections 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b3006b4d0 0x7f0b3006d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b441614d0 0x7f0b440b8ed0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b440d4860 0x7f0b440b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 --2- 172.21.15.122:0/1694583951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b440b0e30 0x7f0b440cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 >> 172.21.15.122:0/1694583951 conn(0x7f0b440d3d40 msgr2=0x7f0b440d0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 shutdown_connections 2024-03-06T07:25:36.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.216+0000 7f0b4a5d7640 1 -- 172.21.15.122:0/1694583951 wait complete. 2024-03-06T07:25:36.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:36.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.332+0000 7f9d0180e640 1 Processor -- start 2024-03-06T07:25:36.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.332+0000 7f9d0180e640 1 -- start start 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9d0180e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9cfc0c90f0 0x7f9cfc0ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9d0180e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc0d25e0 con 0x7f9cfc0d0d90 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9d0180e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc0d2880 con 0x7f9cfc15e000 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9d0180e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0c7c50 con 0x7f9cfc0c90f0 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cfb7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9cfc0c90f0 0x7f9cfc0ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cf3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9cfc0d0d90 0x7f9cfc0ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36464/0 (socket says 172.21.15.122:36464) 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cf3fff640 1 -- 172.21.15.122:0/2592836304 learned_addr learned my addr 172.21.15.122:0/2592836304 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/2592836304 >> v1:172.21.15.169:6789/0 conn(0x7f9cfc15e000 legacy=0x7f9cfc0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/2592836304 >> v1:172.21.15.122:6789/0 conn(0x7f9cfc0d0d90 legacy=0x7f9cfc0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/2592836304 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160645193 0 0) 0x7f9cfc0d25e0 con 0x7f9cfc0d0d90 2024-03-06T07:25:36.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.333+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/2592836304 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0d2b40 con 0x7f9cfc0c90f0 2024-03-06T07:25:36.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.334+0000 7f9cfb7fe640 1 --2- 172.21.15.122:0/2592836304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc0c90f0 0x7f9cfc0ceed0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009d90 tx=0x7f9cf402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea1af1c9758d675 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.334+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/2592836304 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4039070 con 0x7f9cfc0c90f0 2024-03-06T07:25:36.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.334+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/2592836304 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf402cd00 con 0x7f9cfc0c90f0 2024-03-06T07:25:36.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.334+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/2592836304 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4034660 con 0x7f9cfc0c90f0 2024-03-06T07:25:36.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 -- 172.21.15.122:0/2592836304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc0c90f0 msgr2=0x7f9cfc0ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 --2- 172.21.15.122:0/2592836304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc0c90f0 0x7f9cfc0ceed0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009d90 tx=0x7f9cf402bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 -- 172.21.15.122:0/2592836304 shutdown_connections 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 --2- 172.21.15.122:0/2592836304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc0c90f0 0x7f9cfc0ceed0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 -- 172.21.15.122:0/2592836304 >> 172.21.15.122:0/2592836304 conn(0x7f9cfc0d3cf0 msgr2=0x7f9cfc0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 -- 172.21.15.122:0/2592836304 shutdown_connections 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.335+0000 7f9d0180e640 1 -- 172.21.15.122:0/2592836304 wait complete. 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.336+0000 7f9d0180e640 1 Processor -- start 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.336+0000 7f9d0180e640 1 -- start start 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.336+0000 7f9d0180e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cfc0c90f0 0x7f9cfc1f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cfc0d0d90 0x7f9cfc0b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 0x7f9cfc1569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc15e3e0 con 0x7f9cfc15e000 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0d22d0 con 0x7f9cfc0d0d90 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0d4810 con 0x7f9cfc0c90f0 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 0x7f9cfc1569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 0x7f9cfc1569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60688/0 (socket says 172.21.15.122:60688) 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/1323028507 learned_addr learned my addr 172.21.15.122:0/1323028507 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cf3fff640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cfc0d0d90 0x7f9cfc0b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfaffd640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cfc0c90f0 0x7f9cfc1f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cfc0c90f0 msgr2=0x7f9cfc1f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cfc0c90f0 0x7f9cfc1f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cfc0d0d90 msgr2=0x7f9cfc0b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cfc0d0d90 0x7f9cfc0b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 -- 172.21.15.122:0/1323028507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc1f80f0 con 0x7f9cfc15e000 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cfb7fe640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 0x7f9cfc1569d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4037040 tx=0x7f9cf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4044030 con 0x7f9cfc15e000 2024-03-06T07:25:36.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf402cb00 con 0x7f9cfc15e000 2024-03-06T07:25:36.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc09f940 con 0x7f9cfc15e000 2024-03-06T07:25:36.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4039040 con 0x7f9cfc15e000 2024-03-06T07:25:36.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.337+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc0b1810 con 0x7f9cfc15e000 2024-03-06T07:25:36.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.338+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc04f330 con 0x7f9cfc15e000 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.339+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cf403f470 con 0x7f9cfc15e000 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.339+0000 7f9cf8ff9640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cdc06b4d0 0x7f9cdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.340+0000 7f9cfaffd640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cdc06b4d0 0x7f9cdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.340+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9cf40acd00 con 0x7f9cfc15e000 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.340+0000 7f9cfaffd640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cdc06b4d0 0x7f9cdc06d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9cec006fd0 tx=0x7f9cec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.344+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cfc1f80f0 con 0x7f9cfc15e000 2024-03-06T07:25:36.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.531+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9cfc0b6ab0 con 0x7f9cfc15e000 2024-03-06T07:25:36.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.531+0000 7f9cf8ff9640 1 -- 172.21.15.122:0/1323028507 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v71) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9cf407bb30 con 0x7f9cfc15e000 2024-03-06T07:25:36.533 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:25:36.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cdc06b4d0 msgr2=0x7f9cdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cdc06b4d0 0x7f9cdc06d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9cec006fd0 tx=0x7f9cec008040 comp rx=0 tx=0).stop 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 msgr2=0x7f9cfc1569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 0x7f9cfc1569d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4037040 tx=0x7f9cf4004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 shutdown_connections 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cdc06b4d0 0x7f9cdc06d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cfc15e000 0x7f9cfc1569d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cfc0d0d90 0x7f9cfc0b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 --2- 172.21.15.122:0/1323028507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cfc0c90f0 0x7f9cfc1f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.533+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 >> 172.21.15.122:0/1323028507 conn(0x7f9cfc0d3cf0 msgr2=0x7f9cfc06cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.534+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 shutdown_connections 2024-03-06T07:25:36.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.534+0000 7f9d0180e640 1 -- 172.21.15.122:0/1323028507 wait complete. 2024-03-06T07:25:36.546 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:25:36.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:36.546 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:36.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[62880]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:36.573 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:25:36.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:36.573 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:36.665 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:25:36.665 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:36.682 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:36.682 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:25:36.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:36.682 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:36.738 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:36.739 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:36.739 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:36.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:36.755 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:25:36.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:36.755 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:36.809 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:36.809 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:25:36.810 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:36.826 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:36.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:25:36.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:36.827 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:36.883 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:36.883 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:37.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 Processor -- start 2024-03-06T07:25:37.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 -- start start 2024-03-06T07:25:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f95ec15dfe0 0x7f95ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95ec0d29c0 0x7f95ec0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0b3430 con 0x7f95ec0cf830 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0b31e0 con 0x7f95ec0d29c0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95f0ad4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0b3d60 con 0x7f95ec15dfe0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:36.999+0000 7f95ead9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95ec0d29c0 0x7f95ec0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95e1d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f95ec0cf830 0x7f95ec0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36470/0 (socket says 172.21.15.122:36470) 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95e1d9b640 1 -- 172.21.15.122:0/683857147 learned_addr learned my addr 172.21.15.122:0/683857147 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95ea59c640 1 --2- 172.21.15.122:0/683857147 >> v2:172.21.15.169:3300/0 conn(0x7f95ec15dfe0 0x7f95ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95ead9d640 1 -- 172.21.15.122:0/683857147 >> v2:172.21.15.169:3300/0 conn(0x7f95ec15dfe0 msgr2=0x7f95ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95ead9d640 1 --2- 172.21.15.122:0/683857147 >> v2:172.21.15.169:3300/0 conn(0x7f95ec15dfe0 0x7f95ec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95ead9d640 1 -- 172.21.15.122:0/683857147 >> v1:172.21.15.122:6789/0 conn(0x7f95ec0cf830 legacy=0x7f95ec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.000+0000 7f95ead9d640 1 -- 172.21.15.122:0/683857147 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0b3710 con 0x7f95ec0d29c0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95ead9d640 1 --2- 172.21.15.122:0/683857147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0d29c0 0x7f95ec0b41f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f95ec067990 tx=0x7f95e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91780a05f9d6587e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95e9d9b640 1 -- 172.21.15.122:0/683857147 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4039070 con 0x7f95ec0d29c0 2024-03-06T07:25:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95e9d9b640 1 -- 172.21.15.122:0/683857147 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e402cc80 con 0x7f95ec0d29c0 2024-03-06T07:25:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95e9d9b640 1 -- 172.21.15.122:0/683857147 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4034660 con 0x7f95ec0d29c0 2024-03-06T07:25:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 -- 172.21.15.122:0/683857147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0d29c0 msgr2=0x7f95ec0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/683857147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0d29c0 0x7f95ec0b41f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f95ec067990 tx=0x7f95e402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 -- 172.21.15.122:0/683857147 shutdown_connections 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/683857147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0d29c0 0x7f95ec0b41f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/683857147 >> v2:172.21.15.169:3300/0 conn(0x7f95ec15dfe0 0x7f95ec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 -- 172.21.15.122:0/683857147 >> 172.21.15.122:0/683857147 conn(0x7f95ec0d3cd0 msgr2=0x7f95ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 -- 172.21.15.122:0/683857147 shutdown_connections 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.001+0000 7f95f0ad4640 1 -- 172.21.15.122:0/683857147 wait complete. 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.003+0000 7f95f0ad4640 1 Processor -- start 2024-03-06T07:25:37.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.003+0000 7f95f0ad4640 1 -- start start 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec0cf830 0x7f95ec1f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0d29c0 0x7f95ec1520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 0x7f95ec0b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0d1df0 con 0x7f95ec15dfe0 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0d1c70 con 0x7f95ec0cf830 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec15e3c0 con 0x7f95ec0d29c0 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 0x7f95ec0b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 0x7f95ec0b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60704/0 (socket says 172.21.15.122:60704) 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 -- 172.21.15.122:0/2922036486 learned_addr learned my addr 172.21.15.122:0/2922036486 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ea59c640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec0cf830 0x7f95ec1f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95e1d9b640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0d29c0 0x7f95ec1520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 -- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0d29c0 msgr2=0x7f95ec1520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0d29c0 0x7f95ec1520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 -- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec0cf830 msgr2=0x7f95ec1f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec0cf830 0x7f95ec1f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 -- 172.21.15.122:0/2922036486 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec15fb30 con 0x7f95ec15dfe0 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ea59c640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec0cf830 0x7f95ec1f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95e1d9b640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0d29c0 0x7f95ec1520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95ead9d640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 0x7f95ec0b7b10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009e80 tx=0x7f95e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4039050 con 0x7f95ec15dfe0 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e4045070 con 0x7f95ec15dfe0 2024-03-06T07:25:37.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e403e830 con 0x7f95ec15dfe0 2024-03-06T07:25:37.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec15fe10 con 0x7f95ec15dfe0 2024-03-06T07:25:37.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.005+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0b9420 con 0x7f95ec15dfe0 2024-03-06T07:25:37.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.007+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec04f330 con 0x7f95ec15dfe0 2024-03-06T07:25:37.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.007+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95e403e9d0 con 0x7f95ec15dfe0 2024-03-06T07:25:37.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.007+0000 7f95e37fe640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95cc06b4d0 0x7f95cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.007+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f95e40ad180 con 0x7f95ec15dfe0 2024-03-06T07:25:37.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.007+0000 7f95ea59c640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95cc06b4d0 0x7f95cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.008+0000 7f95ea59c640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95cc06b4d0 0x7f95cc06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f95dc006fd0 tx=0x7f95dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.016+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95e407c170 con 0x7f95ec15dfe0 2024-03-06T07:25:37.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.238+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95ec0b2e60 con 0x7f95ec15dfe0 2024-03-06T07:25:37.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.238+0000 7f95e37fe640 1 -- 172.21.15.122:0/2922036486 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f95e407ba70 con 0x7f95ec15dfe0 2024-03-06T07:25:37.241 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:37.241 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:37.243 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:37.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95cc06b4d0 msgr2=0x7f95cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95cc06b4d0 0x7f95cc06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f95dc006fd0 tx=0x7f95dc008040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 msgr2=0x7f95ec0b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 0x7f95ec0b7b10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009e80 tx=0x7f95e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:37.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 shutdown_connections 2024-03-06T07:25:37.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95cc06b4d0 0x7f95cc06d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15dfe0 0x7f95ec0b7b10 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0d29c0 0x7f95ec1520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 --2- 172.21.15.122:0/2922036486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec0cf830 0x7f95ec1f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 >> 172.21.15.122:0/2922036486 conn(0x7f95ec0d3cd0 msgr2=0x7f95ec0b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 shutdown_connections 2024-03-06T07:25:37.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.240+0000 7f95f0ad4640 1 -- 172.21.15.122:0/2922036486 wait complete. 2024-03-06T07:25:37.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.364+0000 7fe91497a640 1 Processor -- start 2024-03-06T07:25:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.364+0000 7fe91497a640 1 -- start start 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe91497a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe9100d22b0 0x7fe9100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe91497a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9100b41f0 0x7fe9100b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe91497a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9100b3430 con 0x7fe91015dfe0 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe91497a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100d2c50 con 0x7fe9100b41f0 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe91497a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100b31e0 con 0x7fe9100d22b0 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9100b41f0 0x7fe9100b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe9100b41f0 0x7fe9100b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60708/0 (socket says 172.21.15.122:60708) 2024-03-06T07:25:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 -- 172.21.15.122:0/1389959647 learned_addr learned my addr 172.21.15.122:0/1389959647 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90dd9b640 1 --2- 172.21.15.122:0/1389959647 >> v2:172.21.15.169:3300/0 conn(0x7fe9100d22b0 0x7fe9100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 -- 172.21.15.122:0/1389959647 >> v2:172.21.15.169:3300/0 conn(0x7fe9100d22b0 msgr2=0x7fe9100d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 --2- 172.21.15.122:0/1389959647 >> v2:172.21.15.169:3300/0 conn(0x7fe9100d22b0 0x7fe9100d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 -- 172.21.15.122:0/1389959647 >> v1:172.21.15.169:6789/0 conn(0x7fe91015dfe0 legacy=0x7fe9100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.365+0000 7fe90ed9d640 1 -- 172.21.15.122:0/1389959647 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9100b3710 con 0x7fe9100b41f0 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.366+0000 7fe90ed9d640 1 --2- 172.21.15.122:0/1389959647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b4680 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe900009870 tx=0x7fe90002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49bd09cd0cd18c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.366+0000 7fe90d59a640 1 -- 172.21.15.122:0/1389959647 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900034070 con 0x7fe9100b41f0 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.366+0000 7fe90d59a640 1 -- 172.21.15.122:0/1389959647 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe900030860 con 0x7fe9100b41f0 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.366+0000 7fe90d59a640 1 -- 172.21.15.122:0/1389959647 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900030c30 con 0x7fe9100b41f0 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.366+0000 7fe91497a640 1 -- 172.21.15.122:0/1389959647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 msgr2=0x7fe9100b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.366+0000 7fe91497a640 1 --2- 172.21.15.122:0/1389959647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b4680 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe900009870 tx=0x7fe90002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.367+0000 7fe91497a640 1 -- 172.21.15.122:0/1389959647 shutdown_connections 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.367+0000 7fe91497a640 1 --2- 172.21.15.122:0/1389959647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b4680 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.367+0000 7fe91497a640 1 --2- 172.21.15.122:0/1389959647 >> v2:172.21.15.169:3300/0 conn(0x7fe9100d22b0 0x7fe9100d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.367+0000 7fe91497a640 1 -- 172.21.15.122:0/1389959647 >> 172.21.15.122:0/1389959647 conn(0x7fe9100d3cd0 msgr2=0x7fe9100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.367+0000 7fe91497a640 1 -- 172.21.15.122:0/1389959647 shutdown_connections 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.367+0000 7fe91497a640 1 -- 172.21.15.122:0/1389959647 wait complete. 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 Processor -- start 2024-03-06T07:25:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 -- start start 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9100d22b0 0x7fe9101f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91015dfe0 0x7fe9100cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100cfa00 con 0x7fe9100b41f0 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100b3d60 con 0x7fe91015dfe0 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.369+0000 7fe91497a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100d1c70 con 0x7fe9100d22b0 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60710/0 (socket says 172.21.15.122:60710) 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 -- 172.21.15.122:0/3902891120 learned_addr learned my addr 172.21.15.122:0/3902891120 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:37.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 -- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9100d22b0 msgr2=0x7fe9101f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9100d22b0 0x7fe9101f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 -- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91015dfe0 msgr2=0x7fe9100cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90ed9d640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91015dfe0 0x7fe9100cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91015dfe0 0x7fe9100cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 -- 172.21.15.122:0/3902891120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9101f4370 con 0x7fe9100b41f0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe90e59c640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b6f10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c730 tx=0x7fe90800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908016020 con 0x7fe9100b41f0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe908006d50 con 0x7fe9100b41f0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9101f4650 con 0x7fe9100b41f0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908003bb0 con 0x7fe9100b41f0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.370+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9101f4030 con 0x7fe9100b41f0 2024-03-06T07:25:37.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.371+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91015e830 con 0x7fe9100b41f0 2024-03-06T07:25:37.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.372+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe90800a850 con 0x7fe9100b41f0 2024-03-06T07:25:37.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.372+0000 7fe906ffd640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.372+0000 7fe90dd9b640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.372+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe908089e50 con 0x7fe9100b41f0 2024-03-06T07:25:37.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.373+0000 7fe90dd9b640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe9100ceb80 tx=0x7fe8f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.377+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe91015e830 con 0x7fe9100b41f0 2024-03-06T07:25:37.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.552+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9100b0a60 con 0x7fe9100b41f0 2024-03-06T07:25:37.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.553+0000 7fe906ffd640 1 -- 172.21.15.122:0/3902891120 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16765 (secure 0 0 0) 0x7fe908020020 con 0x7fe9100b41f0 2024-03-06T07:25:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:37.555 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":71,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:34.941157+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:37.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 msgr2=0x7fe8f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe9100ceb80 tx=0x7fe8f8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 msgr2=0x7fe9100b6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b6f10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c730 tx=0x7fe90800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 shutdown_connections 2024-03-06T07:25:37.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91015dfe0 0x7fe9100cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe9100d22b0 0x7fe9101f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 --2- 172.21.15.122:0/3902891120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9100b41f0 0x7fe9100b6f10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 >> 172.21.15.122:0/3902891120 conn(0x7fe9100d3cd0 msgr2=0x7fe9100b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 shutdown_connections 2024-03-06T07:25:37.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.555+0000 7fe91497a640 1 -- 172.21.15.122:0/3902891120 wait complete. 2024-03-06T07:25:37.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:03.431204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:04.175007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:37.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.670+0000 7f687c50c640 1 Processor -- start 2024-03-06T07:25:37.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.670+0000 7f687c50c640 1 -- start start 2024-03-06T07:25:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687c50c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f68741614d0 0x7f68740d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687c50c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f68740d24b0 0x7f68740b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687c50c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68741618b0 con 0x7f68740d1910 2024-03-06T07:25:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687c50c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68740cf400 con 0x7f68740d24b0 2024-03-06T07:25:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687c50c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68740cf6a0 con 0x7f68741614d0 2024-03-06T07:25:37.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687aaa9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f68740d24b0 0x7f68740b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f6879aa7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f68740d1910 0x7f68740d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36472/0 (socket says 172.21.15.122:36472) 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f6879aa7640 1 -- 172.21.15.122:0/520723420 learned_addr learned my addr 172.21.15.122:0/520723420 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687a2a8640 1 --2- 172.21.15.122:0/520723420 >> v2:172.21.15.169:3300/0 conn(0x7f68741614d0 0x7f68740d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687aaa9640 1 -- 172.21.15.122:0/520723420 >> v2:172.21.15.169:3300/0 conn(0x7f68741614d0 msgr2=0x7f68740d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687aaa9640 1 --2- 172.21.15.122:0/520723420 >> v2:172.21.15.169:3300/0 conn(0x7f68741614d0 0x7f68740d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687aaa9640 1 -- 172.21.15.122:0/520723420 >> v1:172.21.15.122:6789/0 conn(0x7f68740d1910 legacy=0x7f68740d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f68792a6640 1 -- 172.21.15.122:0/520723420 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111688062 0 0) 0x7f68741618b0 con 0x7f68740d1910 2024-03-06T07:25:37.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.671+0000 7f687aaa9640 1 -- 172.21.15.122:0/520723420 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68740cf980 con 0x7f68740d24b0 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.672+0000 7f687aaa9640 1 --2- 172.21.15.122:0/520723420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68740d24b0 0x7f68740b0e30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f687408c470 tx=0x7f686c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c519c5bf29a740e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.672+0000 7f68792a6640 1 -- 172.21.15.122:0/520723420 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c039070 con 0x7f68740d24b0 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.672+0000 7f68792a6640 1 -- 172.21.15.122:0/520723420 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c02cc80 con 0x7f68740d24b0 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.672+0000 7f68792a6640 1 -- 172.21.15.122:0/520723420 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c034660 con 0x7f68740d24b0 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.672+0000 7f687c50c640 1 -- 172.21.15.122:0/520723420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68740d24b0 msgr2=0x7f68740b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.672+0000 7f687c50c640 1 --2- 172.21.15.122:0/520723420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68740d24b0 0x7f68740b0e30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f687408c470 tx=0x7f686c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.673+0000 7f687c50c640 1 -- 172.21.15.122:0/520723420 shutdown_connections 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.673+0000 7f687c50c640 1 --2- 172.21.15.122:0/520723420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68740d24b0 0x7f68740b0e30 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.673+0000 7f687c50c640 1 --2- 172.21.15.122:0/520723420 >> v2:172.21.15.169:3300/0 conn(0x7f68741614d0 0x7f68740d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.673+0000 7f687c50c640 1 -- 172.21.15.122:0/520723420 >> 172.21.15.122:0/520723420 conn(0x7f68740d3d40 msgr2=0x7f68740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.673+0000 7f687c50c640 1 -- 172.21.15.122:0/520723420 shutdown_connections 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.673+0000 7f687c50c640 1 -- 172.21.15.122:0/520723420 wait complete. 2024-03-06T07:25:37.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.674+0000 7f687c50c640 1 Processor -- start 2024-03-06T07:25:37.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.674+0000 7f687c50c640 1 -- start start 2024-03-06T07:25:37.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68740d1910 0x7f68741521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68740d24b0 0x7f68740b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 0x7f68740b9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68740b1370 con 0x7f68741614d0 2024-03-06T07:25:37.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68740b0910 con 0x7f68740d24b0 2024-03-06T07:25:37.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68740b14f0 con 0x7f68740d1910 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 0x7f68740b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 0x7f68740b9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60732/0 (socket says 172.21.15.122:60732) 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 -- 172.21.15.122:0/69554773 learned_addr learned my addr 172.21.15.122:0/69554773 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f6879aa7640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68740d24b0 0x7f68740b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687a2a8640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68740d1910 0x7f68741521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 -- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68740d1910 msgr2=0x7f68741521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68740d1910 0x7f68741521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 -- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68740d24b0 msgr2=0x7f68740b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68740d24b0 0x7f68740b8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 -- 172.21.15.122:0/69554773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874152700 con 0x7f68741614d0 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f6879aa7640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68740d24b0 0x7f68740b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687aaa9640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 0x7f68740b9040 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f686c009e80 tx=0x7f686c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c039050 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c045070 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68741f7860 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c03e830 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.675+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68741f7de0 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.676+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68740b9c00 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.677+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f686c03e9d0 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.677+0000 7f686affd640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f685806b4d0 0x7f685806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.677+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f686c0ad180 con 0x7f68741614d0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.677+0000 7f687a2a8640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f685806b4d0 0x7f685806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.678+0000 7f687a2a8640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f685806b4d0 0x7f685806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6870006fd0 tx=0x7f6870008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.682+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6874152700 con 0x7f68741614d0 2024-03-06T07:25:37.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.859+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6874068ff0 con 0x7f68741614d0 2024-03-06T07:25:37.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.859+0000 7f686affd640 1 -- 172.21.15.122:0/69554773 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f686c07c170 con 0x7f68741614d0 2024-03-06T07:25:37.861 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:25:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f685806b4d0 msgr2=0x7f685806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f685806b4d0 0x7f685806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6870006fd0 tx=0x7f6870008040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 msgr2=0x7f68740b9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 0x7f68740b9040 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f686c009e80 tx=0x7f686c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 shutdown_connections 2024-03-06T07:25:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f685806b4d0 0x7f685806d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68741614d0 0x7f68740b9040 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68740d24b0 0x7f68740b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 --2- 172.21.15.122:0/69554773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68740d1910 0x7f68741521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 >> 172.21.15.122:0/69554773 conn(0x7f68740d3d40 msgr2=0x7f68740b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 shutdown_connections 2024-03-06T07:25:37.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.861+0000 7f687c50c640 1 -- 172.21.15.122:0/69554773 wait complete. 2024-03-06T07:25:37.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:37.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.980+0000 7fd5c442a640 1 Processor -- start 2024-03-06T07:25:37.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.980+0000 7fd5c442a640 1 -- start start 2024-03-06T07:25:37.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c442a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd5bc06fd10 0x7fd5bc0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c442a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5bc0cd4b0 con 0x7fd5bc0d0af0 2024-03-06T07:25:37.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c442a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5bc0cd790 con 0x7fd5bc15c050 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c442a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5bc0cda70 con 0x7fd5bc06fd10 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c29c7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd5bc0d0af0 0x7fd5bc0ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36484/0 (socket says 172.21.15.122:36484) 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c29c7640 1 -- 172.21.15.122:0/1231223514 learned_addr learned my addr 172.21.15.122:0/1231223514 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c19c5640 1 --2- 172.21.15.122:0/1231223514 >> v2:172.21.15.169:3300/0 conn(0x7fd5bc06fd10 0x7fd5bc0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c11c4640 1 -- 172.21.15.122:0/1231223514 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598727527 0 0) 0x7fd5bc0cd4b0 con 0x7fd5bc0d0af0 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c11c4640 1 -- 172.21.15.122:0/1231223514 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a8003520 con 0x7fd5bc0d0af0 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c19c5640 1 -- 172.21.15.122:0/1231223514 >> v1:172.21.15.169:6789/0 conn(0x7fd5bc15c050 legacy=0x7fd5bc06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c11c4640 1 -- 172.21.15.122:0/1231223514 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25691349 0 0) 0x7fd5bc0cd790 con 0x7fd5bc15c050 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c19c5640 1 -- 172.21.15.122:0/1231223514 >> v1:172.21.15.122:6789/0 conn(0x7fd5bc0d0af0 legacy=0x7fd5bc0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.981+0000 7fd5c19c5640 1 -- 172.21.15.122:0/1231223514 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc0cdd50 con 0x7fd5bc06fd10 2024-03-06T07:25:37.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.982+0000 7fd5c19c5640 1 --2- 172.21.15.122:0/1231223514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc0cc760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0097a0 tx=0x7fd5ac02b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39349706f27cfb00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.982+0000 7fd5c11c4640 1 -- 172.21.15.122:0/1231223514 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac037070 con 0x7fd5bc06fd10 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.982+0000 7fd5c11c4640 1 -- 172.21.15.122:0/1231223514 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ac030510 con 0x7fd5bc06fd10 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.982+0000 7fd5c11c4640 1 -- 172.21.15.122:0/1231223514 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4001970 con 0x7fd5bc06fd10 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 -- 172.21.15.122:0/1231223514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc06fd10 msgr2=0x7fd5bc0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 --2- 172.21.15.122:0/1231223514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc0cc760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0097a0 tx=0x7fd5ac02b6b0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 -- 172.21.15.122:0/1231223514 shutdown_connections 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 --2- 172.21.15.122:0/1231223514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc0cc760 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 -- 172.21.15.122:0/1231223514 >> 172.21.15.122:0/1231223514 conn(0x7fd5bc0b3800 msgr2=0x7fd5bc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 -- 172.21.15.122:0/1231223514 shutdown_connections 2024-03-06T07:25:37.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.983+0000 7fd5c442a640 1 -- 172.21.15.122:0/1231223514 wait complete. 2024-03-06T07:25:37.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.984+0000 7fd5c442a640 1 Processor -- start 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.984+0000 7fd5c442a640 1 -- start start 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c442a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc06d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c442a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5bc0d0af0 0x7fd5bc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c442a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc15c050 0x7fd5bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c442a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc166e50 con 0x7fd5bc06fd10 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c442a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc166cd0 con 0x7fd5bc15c050 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c442a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc166fd0 con 0x7fd5bc0d0af0 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc06d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc06d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60746/0 (socket says 172.21.15.122:60746) 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 -- 172.21.15.122:0/932046090 learned_addr learned my addr 172.21.15.122:0/932046090 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c29c7640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc15c050 0x7fd5bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c19c5640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5bc0d0af0 0x7fd5bc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 -- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5bc0d0af0 msgr2=0x7fd5bc0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5bc0d0af0 0x7fd5bc0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 -- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc15c050 msgr2=0x7fd5bc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc15c050 0x7fd5bc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.985+0000 7fd5c21c6640 1 -- 172.21.15.122:0/932046090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc154860 con 0x7fd5bc06fd10 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.986+0000 7fd5c21c6640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc06d9a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8001eb0 tx=0x7fd5b800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.986+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8017070 con 0x7fd5bc06fd10 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.986+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8005480 con 0x7fd5bc06fd10 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.986+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc0b2610 con 0x7fd5bc06fd10 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.986+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8003810 con 0x7fd5bc06fd10 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.986+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc155ee0 con 0x7fd5bc06fd10 2024-03-06T07:25:37.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.987+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc09fe60 con 0x7fd5bc06fd10 2024-03-06T07:25:37.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.988+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5b801a070 con 0x7fd5bc06fd10 2024-03-06T07:25:37.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.988+0000 7fd5b2ffd640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd5a006b4d0 0x7fd5a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.988+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd5b8089700 con 0x7fd5bc06fd10 2024-03-06T07:25:37.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.988+0000 7fd5c19c5640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd5a006b4d0 0x7fd5a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.989+0000 7fd5c19c5640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd5a006b4d0 0x7fd5a006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac034000 tx=0x7fd5ac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:37.992+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5b80584a0 con 0x7fd5bc06fd10 2024-03-06T07:25:38.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.189+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5bc068ff0 con 0x7fd5bc06fd10 2024-03-06T07:25:38.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.189+0000 7fd5b2ffd640 1 -- 172.21.15.122:0/932046090 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd5b8057da0 con 0x7fd5bc06fd10 2024-03-06T07:25:38.191 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:25:38.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd5a006b4d0 msgr2=0x7fd5a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd5a006b4d0 0x7fd5a006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac034000 tx=0x7fd5ac008030 comp rx=0 tx=0).stop 2024-03-06T07:25:38.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 msgr2=0x7fd5bc06d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc06d9a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8001eb0 tx=0x7fd5b800c9b0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 shutdown_connections 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd5a006b4d0 0x7fd5a006d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5bc15c050 0x7fd5bc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd5bc0d0af0 0x7fd5bc0cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 --2- 172.21.15.122:0/932046090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5bc06fd10 0x7fd5bc06d9a0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 >> 172.21.15.122:0/932046090 conn(0x7fd5bc0b3800 msgr2=0x7fd5bc0cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 shutdown_connections 2024-03-06T07:25:38.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.191+0000 7fd5c442a640 1 -- 172.21.15.122:0/932046090 wait complete. 2024-03-06T07:25:38.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:38.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 Processor -- start 2024-03-06T07:25:38.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 -- start start 2024-03-06T07:25:38.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd32c1614d0 0x7fd32c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd32c0d1910 0x7fd32c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c1618b0 con 0x7fd32c0d22e0 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0cf400 con 0x7fd32c0d1910 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd3309ac640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0d2710 con 0x7fd32c1614d0 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.306+0000 7fd323fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd32c0d1910 0x7fd32c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd323fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd32c0d1910 0x7fd32c0d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60754/0 (socket says 172.21.15.122:60754) 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd323fff640 1 -- 172.21.15.122:0/857512801 learned_addr learned my addr 172.21.15.122:0/857512801 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd32a59c640 1 --2- 172.21.15.122:0/857512801 >> v2:172.21.15.169:3300/0 conn(0x7fd32c1614d0 0x7fd32c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd323fff640 1 -- 172.21.15.122:0/857512801 >> v2:172.21.15.169:3300/0 conn(0x7fd32c1614d0 msgr2=0x7fd32c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd323fff640 1 --2- 172.21.15.122:0/857512801 >> v2:172.21.15.169:3300/0 conn(0x7fd32c1614d0 0x7fd32c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd323fff640 1 -- 172.21.15.122:0/857512801 >> v1:172.21.15.169:6789/0 conn(0x7fd32c0d22e0 legacy=0x7fd32c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.307+0000 7fd323fff640 1 -- 172.21.15.122:0/857512801 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0cf6a0 con 0x7fd32c0d1910 2024-03-06T07:25:38.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd323fff640 1 --2- 172.21.15.122:0/857512801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c0d1910 0x7fd32c0d1da0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd32c08c470 tx=0x7fd31402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b581b768d85c86b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd329d9b640 1 -- 172.21.15.122:0/857512801 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314034070 con 0x7fd32c0d1910 2024-03-06T07:25:38.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd329d9b640 1 -- 172.21.15.122:0/857512801 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd314030860 con 0x7fd32c0d1910 2024-03-06T07:25:38.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd329d9b640 1 -- 172.21.15.122:0/857512801 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314030c30 con 0x7fd32c0d1910 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 -- 172.21.15.122:0/857512801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c0d1910 msgr2=0x7fd32c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 --2- 172.21.15.122:0/857512801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c0d1910 0x7fd32c0d1da0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd32c08c470 tx=0x7fd31402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 -- 172.21.15.122:0/857512801 shutdown_connections 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 --2- 172.21.15.122:0/857512801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c0d1910 0x7fd32c0d1da0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 --2- 172.21.15.122:0/857512801 >> v2:172.21.15.169:3300/0 conn(0x7fd32c1614d0 0x7fd32c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 -- 172.21.15.122:0/857512801 >> 172.21.15.122:0/857512801 conn(0x7fd32c0d3d40 msgr2=0x7fd32c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 -- 172.21.15.122:0/857512801 shutdown_connections 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.308+0000 7fd3309ac640 1 -- 172.21.15.122:0/857512801 wait complete. 2024-03-06T07:25:38.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 Processor -- start 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 -- start start 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd32c0d1910 0x7fd32c0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd32c0d22e0 0x7fd32c14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 0x7fd32c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0b0910 con 0x7fd32c1614d0 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0cfc60 con 0x7fd32c0d22e0 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd3309ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0b1e40 con 0x7fd32c0d1910 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 0x7fd32c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 0x7fd32c0b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60760/0 (socket says 172.21.15.122:60760) 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd32ad9d640 1 -- 172.21.15.122:0/2802402948 learned_addr learned my addr 172.21.15.122:0/2802402948 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd32a59c640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd32c0d1910 0x7fd32c0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.310+0000 7fd323fff640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd32c0d22e0 0x7fd32c14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd32ad9d640 1 -- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd32c0d1910 msgr2=0x7fd32c0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd32ad9d640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd32c0d1910 0x7fd32c0cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd32ad9d640 1 -- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd32c0d22e0 msgr2=0x7fd32c14f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd32ad9d640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd32c0d22e0 0x7fd32c14f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd32ad9d640 1 -- 172.21.15.122:0/2802402948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0b63e0 con 0x7fd32c1614d0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd32ad9d640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 0x7fd32c0b4af0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd32400c760 tx=0x7fd32400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd324003b90 con 0x7fd32c1614d0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd324003d30 con 0x7fd32c1614d0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c0b5340 con 0x7fd32c1614d0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32400aa90 con 0x7fd32c1614d0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.311+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c0b5900 con 0x7fd32c1614d0 2024-03-06T07:25:38.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.312+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c068ff0 con 0x7fd32c1614d0 2024-03-06T07:25:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.313+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd32400ac30 con 0x7fd32c1614d0 2024-03-06T07:25:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.313+0000 7fd322ffd640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd30c06b500 0x7fd30c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.313+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd324089f90 con 0x7fd32c1614d0 2024-03-06T07:25:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.313+0000 7fd32a59c640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd30c06b500 0x7fd30c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.313+0000 7fd32a59c640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd30c06b500 0x7fd30c06d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0079a0 tx=0x7fd31c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.318+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd324058780 con 0x7fd32c1614d0 2024-03-06T07:25:38.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.495+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd32c0caab0 con 0x7fd32c1614d0 2024-03-06T07:25:38.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.495+0000 7fd322ffd640 1 -- 172.21.15.122:0/2802402948 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd32405c820 con 0x7fd32c1614d0 2024-03-06T07:25:38.497 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:25:38.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd30c06b500 msgr2=0x7fd30c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd30c06b500 0x7fd30c06d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0079a0 tx=0x7fd31c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 msgr2=0x7fd32c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 0x7fd32c0b4af0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd32400c760 tx=0x7fd32400cc20 comp rx=0 tx=0).stop 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 shutdown_connections 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd30c06b500 0x7fd30c06d9c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd32c1614d0 0x7fd32c0b4af0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd32c0d22e0 0x7fd32c14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 --2- 172.21.15.122:0/2802402948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd32c0d1910 0x7fd32c0cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 >> 172.21.15.122:0/2802402948 conn(0x7fd32c0d3d40 msgr2=0x7fd32c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 shutdown_connections 2024-03-06T07:25:38.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.497+0000 7fd3309ac640 1 -- 172.21.15.122:0/2802402948 wait complete. 2024-03-06T07:25:38.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:38.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.610+0000 7fb3060fa640 1 Processor -- start 2024-03-06T07:25:38.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.610+0000 7fb3060fa640 1 -- start start 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb3060fa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb30015dfe0 0x7fb3000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb3060fa640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb3000cf830 0x7fb3000d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb3060fa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000d27f0 0x7fb3000d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb3060fa640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3000b31e0 con 0x7fb30015dfe0 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb3060fa640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3000b3d60 con 0x7fb3000cf830 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb3060fa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3000b3430 con 0x7fb3000d27f0 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb30015dfe0 0x7fb3000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb30015dfe0 0x7fb3000cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60770/0 (socket says 172.21.15.122:60770) 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 -- 172.21.15.122:0/3871782033 learned_addr learned my addr 172.21.15.122:0/3871782033 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2feffd640 1 --2- 172.21.15.122:0/3871782033 >> v2:172.21.15.169:3300/0 conn(0x7fb3000cf830 0x7fb3000d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2fffff640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000d27f0 0x7fb3000d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 -- 172.21.15.122:0/3871782033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000d27f0 msgr2=0x7fb3000d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000d27f0 0x7fb3000d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 -- 172.21.15.122:0/3871782033 >> v2:172.21.15.169:3300/0 conn(0x7fb3000cf830 msgr2=0x7fb3000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 --2- 172.21.15.122:0/3871782033 >> v2:172.21.15.169:3300/0 conn(0x7fb3000cf830 0x7fb3000d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2ff7fe640 1 -- 172.21.15.122:0/3871782033 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000b3710 con 0x7fb30015dfe0 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.611+0000 7fb2fffff640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000d27f0 0x7fb3000d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb2ff7fe640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb30015dfe0 0x7fb3000cf2f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb3000d16b0 tx=0x7fb2f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3fa58312189087 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb2fe7fc640 1 -- 172.21.15.122:0/3871782033 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0035020 con 0x7fb30015dfe0 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb2fe7fc640 1 -- 172.21.15.122:0/3871782033 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0031870 con 0x7fb30015dfe0 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb2fe7fc640 1 -- 172.21.15.122:0/3871782033 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0031c40 con 0x7fb30015dfe0 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 -- 172.21.15.122:0/3871782033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb30015dfe0 msgr2=0x7fb3000cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb30015dfe0 0x7fb3000cf2f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb3000d16b0 tx=0x7fb2f002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 -- 172.21.15.122:0/3871782033 shutdown_connections 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000d27f0 0x7fb3000d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 --2- 172.21.15.122:0/3871782033 >> v2:172.21.15.169:3300/0 conn(0x7fb3000cf830 0x7fb3000d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 --2- 172.21.15.122:0/3871782033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb30015dfe0 0x7fb3000cf2f0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 -- 172.21.15.122:0/3871782033 >> 172.21.15.122:0/3871782033 conn(0x7fb3000d3cd0 msgr2=0x7fb3000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.612+0000 7fb3060fa640 1 -- 172.21.15.122:0/3871782033 shutdown_connections 2024-03-06T07:25:38.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.613+0000 7fb3060fa640 1 -- 172.21.15.122:0/3871782033 wait complete. 2024-03-06T07:25:38.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.614+0000 7fb3060fa640 1 Processor -- start 2024-03-06T07:25:38.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.614+0000 7fb3060fa640 1 -- start start 2024-03-06T07:25:38.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb3060fa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000cf830 0x7fb3000cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb3060fa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 0x7fb3000cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb3060fa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb30015dfe0 0x7fb3000cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb3060fa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3000d1df0 con 0x7fb3000d27f0 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb3060fa640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3000d1c70 con 0x7fb30015dfe0 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb3060fa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30015e3c0 con 0x7fb3000cf830 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb2feffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 0x7fb3000cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.615+0000 7fb2feffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 0x7fb3000cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60778/0 (socket says 172.21.15.122:60778) 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 -- 172.21.15.122:0/1552722157 learned_addr learned my addr 172.21.15.122:0/1552722157 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2fffff640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb30015dfe0 0x7fb3000cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 -- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000cf830 msgr2=0x7fb3000cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2ff7fe640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000cf830 0x7fb3000cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000cf830 0x7fb3000cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 -- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb30015dfe0 msgr2=0x7fb3000cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb30015dfe0 0x7fb3000cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 -- 172.21.15.122:0/1552722157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000cb410 con 0x7fb3000d27f0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2fffff640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb30015dfe0 0x7fb3000cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2feffd640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 0x7fb3000cada0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400d800 tx=0x7fb2f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f40096a0 con 0x7fb3000d27f0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f4019070 con 0x7fb3000d27f0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3000cb6f0 con 0x7fb3000d27f0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4009a60 con 0x7fb3000d27f0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.616+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3000cbcb0 con 0x7fb3000d27f0 2024-03-06T07:25:38.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.617+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30004f330 con 0x7fb3000d27f0 2024-03-06T07:25:38.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.618+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2f4009c00 con 0x7fb3000d27f0 2024-03-06T07:25:38.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.618+0000 7fb2e7fff640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2ec06b4d0 0x7fb2ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.618+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb2f408a5c0 con 0x7fb3000d27f0 2024-03-06T07:25:38.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.618+0000 7fb2ff7fe640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2ec06b4d0 0x7fb2ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.619+0000 7fb2ff7fe640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2ec06b4d0 0x7fb2ec06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0036000 tx=0x7fb2f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.623+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2f4059490 con 0x7fb3000d27f0 2024-03-06T07:25:38.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.852+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3000ce9b0 con 0x7fb3000d27f0 2024-03-06T07:25:38.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.853+0000 7fb2e7fff640 1 -- 172.21.15.122:0/1552722157 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fb2f4058d90 con 0x7fb3000d27f0 2024-03-06T07:25:38.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:38.855 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:38.855 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:38.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2ec06b4d0 msgr2=0x7fb2ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2ec06b4d0 0x7fb2ec06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0036000 tx=0x7fb2f0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:38.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 msgr2=0x7fb3000cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 0x7fb3000cada0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400d800 tx=0x7fb2f400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 shutdown_connections 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2ec06b4d0 0x7fb2ec06d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb30015dfe0 0x7fb3000cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3000d27f0 0x7fb3000cada0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 --2- 172.21.15.122:0/1552722157 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb3000cf830 0x7fb3000cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 >> 172.21.15.122:0/1552722157 conn(0x7fb3000d3cd0 msgr2=0x7fb30006bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 shutdown_connections 2024-03-06T07:25:38.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.855+0000 7fb3060fa640 1 -- 172.21.15.122:0/1552722157 wait complete. 2024-03-06T07:25:38.868 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.967+0000 7fa5ddd3e640 1 Processor -- start 2024-03-06T07:25:38.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.967+0000 7fa5ddd3e640 1 -- start start 2024-03-06T07:25:38.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5ddd3e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa5d80d4760 0x7fa5d80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5ddd3e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80d25d0 con 0x7fa5d80b4170 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5ddd3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80d2870 con 0x7fa5d80d0a80 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5ddd3e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d80d4bf0 con 0x7fa5d80d4760 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa5d80b4170 0x7fa5d80d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36496/0 (socket says 172.21.15.122:36496) 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d77fe640 1 -- 172.21.15.122:0/1428311406 learned_addr learned my addr 172.21.15.122:0/1428311406 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136726742 0 0) 0x7fa5d80d25d0 con 0x7fa5d80b4170 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c4003520 con 0x7fa5d80b4170 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5ceffd640 1 --2- 172.21.15.122:0/1428311406 >> v2:172.21.15.169:3300/0 conn(0x7fa5d80d4760 0x7fa5d80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813514630 0 0) 0x7fa5c4003520 con 0x7fa5d80b4170 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 >> v2:172.21.15.169:3300/0 conn(0x7fa5d80d4760 msgr2=0x7fa5d80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 --2- 172.21.15.122:0/1428311406 >> v2:172.21.15.169:3300/0 conn(0x7fa5d80d4760 0x7fa5d80b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 >> v1:172.21.15.169:6789/0 conn(0x7fa5d80d0a80 legacy=0x7fa5d80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80d2b50 con 0x7fa5d80b4170 2024-03-06T07:25:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa5c8001690 con 0x7fa5d80b4170 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 >> v1:172.21.15.122:6789/0 conn(0x7fa5d80b4170 legacy=0x7fa5d80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5c40039b0 0x7fa5c4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5c4004800 0x7fa5c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40051c0 0x7fa5c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80d4bf0 con 0x7fa5c40039b0 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c400c0f0 con 0x7fa5c40051c0 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c400c2f0 con 0x7fa5c4004800 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.968+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5c40039b0 0x7fa5c4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5ceffd640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5c4004800 0x7fa5c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d7fff640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40051c0 0x7fa5c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d77fe640 1 -- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5c4004800 msgr2=0x7fa5c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5c4004800 0x7fa5c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d77fe640 1 -- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40051c0 msgr2=0x7fa5c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40051c0 0x7fa5c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d7fff640 1 -- 172.21.15.122:0/1428311406 reap_dead start 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5d77fe640 1 -- 172.21.15.122:0/1428311406 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4003520 con 0x7fa5c40039b0 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.969+0000 7fa5ceffd640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5c4004800 0x7fa5c4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5c40039b0 0x7fa5c4003db0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80d27a0 tx=0x7fa5c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ddf8182a21b9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8007d60 con 0x7fa5c40039b0 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c8030070 con 0x7fa5c40039b0 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5d6ffd640 1 -- 172.21.15.122:0/1428311406 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8005d10 con 0x7fa5c40039b0 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1428311406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5c40039b0 msgr2=0x7fa5c4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5c40039b0 0x7fa5c4003db0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80d27a0 tx=0x7fa5c4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1428311406 shutdown_connections 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1428311406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5c40039b0 0x7fa5c4003db0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1428311406 >> 172.21.15.122:0/1428311406 conn(0x7fa5d80d3c40 msgr2=0x7fa5d80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1428311406 shutdown_connections 2024-03-06T07:25:38.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.970+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1428311406 wait complete. 2024-03-06T07:25:38.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.972+0000 7fa5ddd3e640 1 Processor -- start 2024-03-06T07:25:38.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.972+0000 7fa5ddd3e640 1 -- start start 2024-03-06T07:25:38.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40039b0 0x7fa5d81571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5d80cc8b0 0x7fa5d80ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 0x7fa5d81f7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80b4dd0 con 0x7fa5d81f7370 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80ce9e0 con 0x7fa5c40039b0 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80b4f50 con 0x7fa5d80cc8b0 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 0x7fa5d81f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 0x7fa5d81f7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60786/0 (socket says 172.21.15.122:60786) 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 -- 172.21.15.122:0/1747648769 learned_addr learned my addr 172.21.15.122:0/1747648769 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40039b0 0x7fa5d81571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ceffd640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5d80cc8b0 0x7fa5d80ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 -- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5d80cc8b0 msgr2=0x7fa5d80ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5d80cc8b0 0x7fa5d80ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 -- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40039b0 msgr2=0x7fa5d81571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40039b0 0x7fa5d81571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 -- 172.21.15.122:0/1747648769 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d81f80a0 con 0x7fa5d81f7370 2024-03-06T07:25:38.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40039b0 0x7fa5d81571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d7fff640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 0x7fa5d81f7820 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80b67e0 tx=0x7fa5d000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0006d60 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d0018070 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d80b0990 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0007270 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.974+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d80b0f50 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.976+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d804f330 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.976+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5d0007410 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.976+0000 7fa5d4ff9640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5b806b4d0 0x7fa5b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.976+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa5d0089db0 con 0x7fa5d81f7370 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.976+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5b806b4d0 0x7fa5b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.977+0000 7fa5d77fe640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5b806b4d0 0x7fa5b806d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8034000 tx=0x7fa5c8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:38.984+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5d0058c80 con 0x7fa5d81f7370 2024-03-06T07:25:39.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.162+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5d80b1120 con 0x7fa5d81f7370 2024-03-06T07:25:39.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.163+0000 7fa5d4ff9640 1 -- 172.21.15.122:0/1747648769 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16765 (secure 0 0 0) 0x7fa5d0058580 con 0x7fa5d81f7370 2024-03-06T07:25:39.166 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:39.166 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":71,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:34.941157+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5b806b4d0 msgr2=0x7fa5b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5b806b4d0 0x7fa5b806d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8034000 tx=0x7fa5c8009030 comp rx=0 tx=0).stop 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 msgr2=0x7fa5d81f7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 0x7fa5d81f7820 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80b67e0 tx=0x7fa5d000e9e0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 shutdown_connections 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa5b806b4d0 0x7fa5b806d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d81f7370 0x7fa5d81f7820 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5d80cc8b0 0x7fa5d80ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 --2- 172.21.15.122:0/1747648769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5c40039b0 0x7fa5d81571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 >> 172.21.15.122:0/1747648769 conn(0x7fa5d80d3c40 msgr2=0x7fa5d80d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 shutdown_connections 2024-03-06T07:25:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.165+0000 7fa5ddd3e640 1 -- 172.21.15.122:0/1747648769 wait complete. 2024-03-06T07:25:39.179 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:39.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:39.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.300+0000 7ff8067e9640 1 Processor -- start 2024-03-06T07:25:39.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.300+0000 7ff8067e9640 1 -- start start 2024-03-06T07:25:39.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.301+0000 7ff8067e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff80015abf0 0x7ff80006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.301+0000 7ff8067e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff8000cfd20 0x7ff80006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff8067e9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff800066e30 con 0x7ff80006f000 2024-03-06T07:25:39.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff8067e9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80006eb70 con 0x7ff8000cfd20 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff8067e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80006bba0 con 0x7ff80015abf0 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff804d86640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff8000cfd20 0x7ff80006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff7ff7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff80006f000 0x7ff8000b4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36510/0 (socket says 172.21.15.122:36510) 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff7ff7fe640 1 -- 172.21.15.122:0/2616888130 learned_addr learned my addr 172.21.15.122:0/2616888130 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff7fffff640 1 --2- 172.21.15.122:0/2616888130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff80015abf0 0x7ff80006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff804d86640 1 -- 172.21.15.122:0/2616888130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff80015abf0 msgr2=0x7ff80006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff804d86640 1 --2- 172.21.15.122:0/2616888130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff80015abf0 0x7ff80006d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff804d86640 1 -- 172.21.15.122:0/2616888130 >> v1:172.21.15.122:6789/0 conn(0x7ff80006f000 legacy=0x7ff8000b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff7feffd640 1 -- 172.21.15.122:0/2616888130 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563952607 0 0) 0x7ff800066e30 con 0x7ff80006f000 2024-03-06T07:25:39.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.302+0000 7ff804d86640 1 -- 172.21.15.122:0/2616888130 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80006be80 con 0x7ff8000cfd20 2024-03-06T07:25:39.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.303+0000 7ff804d86640 1 --2- 172.21.15.122:0/2616888130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8000cfd20 0x7ff80006c140 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009d90 tx=0x7ff7f802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991d03891b348933 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.303+0000 7ff7feffd640 1 -- 172.21.15.122:0/2616888130 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8039070 con 0x7ff8000cfd20 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.303+0000 7ff7feffd640 1 -- 172.21.15.122:0/2616888130 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f802cbc0 con 0x7ff8000cfd20 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.303+0000 7ff7feffd640 1 -- 172.21.15.122:0/2616888130 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8034440 con 0x7ff8000cfd20 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 -- 172.21.15.122:0/2616888130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8000cfd20 msgr2=0x7ff80006c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 --2- 172.21.15.122:0/2616888130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8000cfd20 0x7ff80006c140 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009d90 tx=0x7ff7f802be10 comp rx=0 tx=0).stop 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 -- 172.21.15.122:0/2616888130 shutdown_connections 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 --2- 172.21.15.122:0/2616888130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8000cfd20 0x7ff80006c140 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 --2- 172.21.15.122:0/2616888130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff80015abf0 0x7ff80006d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 -- 172.21.15.122:0/2616888130 >> 172.21.15.122:0/2616888130 conn(0x7ff8000b1d00 msgr2=0x7ff8000b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 -- 172.21.15.122:0/2616888130 shutdown_connections 2024-03-06T07:25:39.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.304+0000 7ff8067e9640 1 -- 172.21.15.122:0/2616888130 wait complete. 2024-03-06T07:25:39.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.306+0000 7ff8067e9640 1 Processor -- start 2024-03-06T07:25:39.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.306+0000 7ff8067e9640 1 -- start start 2024-03-06T07:25:39.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.306+0000 7ff8067e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff80006f000 0x7ff8000cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.306+0000 7ff8067e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8000cfd20 0x7ff8001ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.306+0000 7ff8067e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 0x7ff8001efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff8067e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80006c800 con 0x7ff80015abf0 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff8067e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80006c680 con 0x7ff80006f000 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff8067e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80015afd0 con 0x7ff8000cfd20 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 0x7ff8001efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 0x7ff8001efcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60798/0 (socket says 172.21.15.122:60798) 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 -- 172.21.15.122:0/130899384 learned_addr learned my addr 172.21.15.122:0/130899384 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7fffff640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff80006f000 0x7ff8000cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7ff7fe640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8000cfd20 0x7ff8001ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 -- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8000cfd20 msgr2=0x7ff8001ef770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8000cfd20 0x7ff8001ef770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 -- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff80006f000 msgr2=0x7ff8000cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff80006f000 0x7ff8000cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 -- 172.21.15.122:0/130899384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800157550 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7fffff640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff80006f000 0x7ff8000cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7ff7fe640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8000cfd20 0x7ff8001ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff804d86640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 0x7ff8001efcb0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009eb0 tx=0x7ff7f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8039050 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f8045070 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8000afd60 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f803e830 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.307+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8000ccc20 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.308+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80004f330 con 0x7ff80015abf0 2024-03-06T07:25:39.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.309+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7f803e9d0 con 0x7ff80015abf0 2024-03-06T07:25:39.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.309+0000 7ff7fcff9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.309+0000 7ff7fffff640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.309+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff7f80aced0 con 0x7ff80015abf0 2024-03-06T07:25:39.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.310+0000 7ff7fffff640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff8000b19b0 tx=0x7ff7f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.314+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7f8032e20 con 0x7ff80015abf0 2024-03-06T07:25:39.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.625+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8000b2610 con 0x7ff80015abf0 2024-03-06T07:25:39.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.625+0000 7ff7fcff9640 1 -- 172.21.15.122:0/130899384 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7ff7f807bec0 con 0x7ff80015abf0 2024-03-06T07:25:39.628 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:39.628 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:39.628 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:39.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7ec06b500 msgr2=0x7ff7ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff8000b19b0 tx=0x7ff7f0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 msgr2=0x7ff8001efcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 0x7ff8001efcb0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009eb0 tx=0x7ff7f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 shutdown_connections 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff80015abf0 0x7ff8001efcb0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff8000cfd20 0x7ff8001ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 --2- 172.21.15.122:0/130899384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff80006f000 0x7ff8000cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.627+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 >> 172.21.15.122:0/130899384 conn(0x7ff8000b1d00 msgr2=0x7ff8000af210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.628+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 shutdown_connections 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.628+0000 7ff8067e9640 1 -- 172.21.15.122:0/130899384 wait complete. 2024-03-06T07:25:39.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:39.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 Processor -- start 2024-03-06T07:25:39.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 -- start start 2024-03-06T07:25:39.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f618015bb20 0x7f61800d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f61800cfff0 0x7f61800d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800ce4b0 0x7f61800ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6180161980 con 0x7f618015bb20 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6180161c60 con 0x7f61800cfff0 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f6186078640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6180161f40 con 0x7f61800ce4b0 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.740+0000 7f617f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f618015bb20 0x7f61800d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f618015bb20 0x7f61800d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60800/0 (socket says 172.21.15.122:60800) 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 -- 172.21.15.122:0/3072583145 learned_addr learned my addr 172.21.15.122:0/3072583145 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617effd640 1 --2- 172.21.15.122:0/3072583145 >> v2:172.21.15.169:3300/0 conn(0x7f61800cfff0 0x7f61800d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617ffff640 1 --2- 172.21.15.122:0/3072583145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800ce4b0 0x7f61800ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 -- 172.21.15.122:0/3072583145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800ce4b0 msgr2=0x7f61800ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 --2- 172.21.15.122:0/3072583145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800ce4b0 0x7f61800ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 -- 172.21.15.122:0/3072583145 >> v2:172.21.15.169:3300/0 conn(0x7f61800cfff0 msgr2=0x7f61800d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 --2- 172.21.15.122:0/3072583145 >> v2:172.21.15.169:3300/0 conn(0x7f61800cfff0 0x7f61800d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 -- 172.21.15.122:0/3072583145 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180162220 con 0x7f618015bb20 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617effd640 1 --2- 172.21.15.122:0/3072583145 >> v2:172.21.15.169:3300/0 conn(0x7f61800cfff0 0x7f61800d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.741+0000 7f617f7fe640 1 --2- 172.21.15.122:0/3072583145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f61800d1df0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6180071290 tx=0x7f616802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e80611342cad424 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f617e7fc640 1 -- 172.21.15.122:0/3072583145 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168035020 con 0x7f618015bb20 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f617e7fc640 1 -- 172.21.15.122:0/3072583145 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168031870 con 0x7f618015bb20 2024-03-06T07:25:39.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f617e7fc640 1 -- 172.21.15.122:0/3072583145 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168031c40 con 0x7f618015bb20 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 -- 172.21.15.122:0/3072583145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 msgr2=0x7f61800d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 --2- 172.21.15.122:0/3072583145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f61800d1df0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6180071290 tx=0x7f616802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 -- 172.21.15.122:0/3072583145 shutdown_connections 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 --2- 172.21.15.122:0/3072583145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800ce4b0 0x7f61800ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 --2- 172.21.15.122:0/3072583145 >> v2:172.21.15.169:3300/0 conn(0x7f61800cfff0 0x7f61800d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 --2- 172.21.15.122:0/3072583145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f61800d1df0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 -- 172.21.15.122:0/3072583145 >> 172.21.15.122:0/3072583145 conn(0x7f618009a9f0 msgr2=0x7f618009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 -- 172.21.15.122:0/3072583145 shutdown_connections 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.742+0000 7f6186078640 1 -- 172.21.15.122:0/3072583145 wait complete. 2024-03-06T07:25:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.743+0000 7f6186078640 1 Processor -- start 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 -- start start 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61800ce4b0 0x7f6180156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800cfff0 0x7f61800c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f618006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800d24b0 con 0x7f618015bb20 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800d2330 con 0x7f61800ce4b0 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f6186078640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61800d1980 con 0x7f61800cfff0 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f617ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f618006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f617ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f618006d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60810/0 (socket says 172.21.15.122:60810) 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f617ffff640 1 -- 172.21.15.122:0/2060545439 learned_addr learned my addr 172.21.15.122:0/2060545439 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f617f7fe640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61800ce4b0 0x7f6180156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.744+0000 7f617effd640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800cfff0 0x7f61800c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f617ffff640 1 -- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800cfff0 msgr2=0x7f61800c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f617ffff640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800cfff0 0x7f61800c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f617ffff640 1 -- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61800ce4b0 msgr2=0x7f6180156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f617ffff640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61800ce4b0 0x7f6180156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f617ffff640 1 -- 172.21.15.122:0/2060545439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180157130 con 0x7f618015bb20 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f617ffff640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f618006d450 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f617800ccf0 tx=0x7f617800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178003b90 con 0x7f618015bb20 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6178003d30 con 0x7f618015bb20 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61800c55e0 con 0x7f618015bb20 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178004240 con 0x7f618015bb20 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.745+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61800c5ba0 con 0x7f618015bb20 2024-03-06T07:25:39.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.746+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618004f330 con 0x7f618015bb20 2024-03-06T07:25:39.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.746+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61780043e0 con 0x7f618015bb20 2024-03-06T07:25:39.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.747+0000 7f6177fff640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616006b4d0 0x7f616006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.747+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f617808ab80 con 0x7f618015bb20 2024-03-06T07:25:39.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.747+0000 7f617f7fe640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616006b4d0 0x7f616006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.747+0000 7f617f7fe640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616006b4d0 0x7f616006d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6168036000 tx=0x7f616803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.751+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6178059360 con 0x7f618015bb20 2024-03-06T07:25:39.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.965+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61800b2b40 con 0x7f618015bb20 2024-03-06T07:25:39.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.965+0000 7f6177fff640 1 -- 172.21.15.122:0/2060545439 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f617805d400 con 0x7f618015bb20 2024-03-06T07:25:39.968 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:39.968 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:05.201235+0000","modified":"2024-03-06T07:25:07.224230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6600":{"gid":6600,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/323983625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":323983625},{"type":"v1","addr":"172.21.15.122:6837","nonce":323983625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6600,"qdb_cluster":[6600]},"id":5}]} 2024-03-06T07:25:39.968 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 33 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616006b4d0 msgr2=0x7f616006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616006b4d0 0x7f616006d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6168036000 tx=0x7f616803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 msgr2=0x7f618006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f618006d450 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f617800ccf0 tx=0x7f617800a660 comp rx=0 tx=0).stop 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 shutdown_connections 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f616006b4d0 0x7f616006d990 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618015bb20 0x7f618006d450 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f61800cfff0 0x7f61800c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 --2- 172.21.15.122:0/2060545439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61800ce4b0 0x7f6180156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.967+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 >> 172.21.15.122:0/2060545439 conn(0x7f618009a9f0 msgr2=0x7f618006f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.968+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 shutdown_connections 2024-03-06T07:25:39.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:39.968+0000 7f6186078640 1 -- 172.21.15.122:0/2060545439 wait complete. 2024-03-06T07:25:39.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:40.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.078+0000 7f5650ac9640 1 Processor -- start 2024-03-06T07:25:40.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.078+0000 7f5650ac9640 1 -- start start 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f5650ac9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f564c0b4f70 0x7f564c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f5650ac9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c06c6d0 0x7f564c15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f5650ac9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f564c066d90 0x7f564c0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f5650ac9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c15e9f0 con 0x7f564c0b4f70 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f5650ac9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c0cda10 con 0x7f564c066d90 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f5650ac9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0b5890 con 0x7f564c06c6d0 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f564a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f564c0b4f70 0x7f564c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f564a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f564c0b4f70 0x7f564c0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60826/0 (socket says 172.21.15.122:60826) 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.079+0000 7f564a59c640 1 -- 172.21.15.122:0/3273683744 learned_addr learned my addr 172.21.15.122:0/3273683744 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:40.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564a59c640 1 -- 172.21.15.122:0/3273683744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c06c6d0 msgr2=0x7f564c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564a59c640 1 --2- 172.21.15.122:0/3273683744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c06c6d0 0x7f564c15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564a59c640 1 -- 172.21.15.122:0/3273683744 >> v2:172.21.15.169:3300/0 conn(0x7f564c066d90 msgr2=0x7f564c0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564ad9d640 1 --2- 172.21.15.122:0/3273683744 >> v2:172.21.15.169:3300/0 conn(0x7f564c066d90 0x7f564c0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564a59c640 1 --2- 172.21.15.122:0/3273683744 >> v2:172.21.15.169:3300/0 conn(0x7f564c066d90 0x7f564c0ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564a59c640 1 -- 172.21.15.122:0/3273683744 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0cd450 con 0x7f564c0b4f70 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.080+0000 7f564a59c640 1 --2- 172.21.15.122:0/3273683744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c0b4f70 0x7f564c0b5350 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f564c0723c0 tx=0x7f564402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c8221f7f5054a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f564959a640 1 -- 172.21.15.122:0/3273683744 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644035020 con 0x7f564c0b4f70 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f564959a640 1 -- 172.21.15.122:0/3273683744 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5644031870 con 0x7f564c0b4f70 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f564959a640 1 -- 172.21.15.122:0/3273683744 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644031c40 con 0x7f564c0b4f70 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 -- 172.21.15.122:0/3273683744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c0b4f70 msgr2=0x7f564c0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 --2- 172.21.15.122:0/3273683744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c0b4f70 0x7f564c0b5350 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f564c0723c0 tx=0x7f564402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 -- 172.21.15.122:0/3273683744 shutdown_connections 2024-03-06T07:25:40.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 --2- 172.21.15.122:0/3273683744 >> v2:172.21.15.169:3300/0 conn(0x7f564c066d90 0x7f564c0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 --2- 172.21.15.122:0/3273683744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c06c6d0 0x7f564c15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 --2- 172.21.15.122:0/3273683744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c0b4f70 0x7f564c0b5350 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 -- 172.21.15.122:0/3273683744 >> 172.21.15.122:0/3273683744 conn(0x7f564c0724d0 msgr2=0x7f564c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 -- 172.21.15.122:0/3273683744 shutdown_connections 2024-03-06T07:25:40.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.081+0000 7f5650ac9640 1 -- 172.21.15.122:0/3273683744 wait complete. 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 Processor -- start 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 -- start start 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 0x7f564c0adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c0b4f70 0x7f564c160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f564c160c90 0x7f564c143680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c15f3c0 con 0x7f564c066d90 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c15e680 con 0x7f564c160c90 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f5650ac9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0cd090 con 0x7f564c0b4f70 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.083+0000 7f564a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 0x7f564c0adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 0x7f564c0adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60836/0 (socket says 172.21.15.122:60836) 2024-03-06T07:25:40.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 -- 172.21.15.122:0/2276499149 learned_addr learned my addr 172.21.15.122:0/2276499149 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564ad9d640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f564c160c90 0x7f564c143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 -- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c0b4f70 msgr2=0x7f564c160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f5649d9b640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c0b4f70 0x7f564c160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c0b4f70 0x7f564c160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 -- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f564c160c90 msgr2=0x7f564c143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f564c160c90 0x7f564c143680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 -- 172.21.15.122:0/2276499149 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c15a960 con 0x7f564c066d90 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564ad9d640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f564c160c90 0x7f564c143680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f564a59c640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 0x7f564c0adc30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f564402b950 tx=0x7f56440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644040070 con 0x7f564c066d90 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564403a420 con 0x7f564c066d90 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644035040 con 0x7f564c066d90 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c0ae3a0 con 0x7f564c066d90 2024-03-06T07:25:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.084+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c0ae960 con 0x7f564c066d90 2024-03-06T07:25:40.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.085+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c1610a0 con 0x7f564c066d90 2024-03-06T07:25:40.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.086+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5644038020 con 0x7f564c066d90 2024-03-06T07:25:40.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.086+0000 7f563affd640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f562c06b4d0 0x7f562c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.086+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56440ac260 con 0x7f564c066d90 2024-03-06T07:25:40.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.086+0000 7f5649d9b640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f562c06b4d0 0x7f562c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.087+0000 7f5649d9b640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f562c06b4d0 0x7f562c06d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f564c15ba60 tx=0x7f5640008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.091+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f564c1610a0 con 0x7f564c066d90 2024-03-06T07:25:40.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.269+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f564c04f330 con 0x7f564c066d90 2024-03-06T07:25:40.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.270+0000 7f563affd640 1 -- 172.21.15.122:0/2276499149 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16765 (secure 0 0 0) 0x7f564407b0a0 con 0x7f564c066d90 2024-03-06T07:25:40.272 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:40.272 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":71,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:34.941157+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:03.431204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:04.175007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f562c06b4d0 msgr2=0x7f562c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f562c06b4d0 0x7f562c06d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f564c15ba60 tx=0x7f5640008040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 msgr2=0x7f564c0adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 0x7f564c0adc30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f564402b950 tx=0x7f56440046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 shutdown_connections 2024-03-06T07:25:40.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f562c06b4d0 0x7f562c06d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f564c160c90 0x7f564c143680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f564c0b4f70 0x7f564c160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 --2- 172.21.15.122:0/2276499149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f564c066d90 0x7f564c0adc30 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 >> 172.21.15.122:0/2276499149 conn(0x7f564c0724d0 msgr2=0x7f564c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 shutdown_connections 2024-03-06T07:25:40.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.272+0000 7f5650ac9640 1 -- 172.21.15.122:0/2276499149 wait complete. 2024-03-06T07:25:40.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:40.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.387+0000 7ff5e4fe2640 1 Processor -- start 2024-03-06T07:25:40.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.387+0000 7ff5e4fe2640 1 -- start start 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5e4fe2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5e015e010 0x7ff5e00cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5e4fe2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff5e00b3240 0x7ff5e00b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5e4fe2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e015e3f0 con 0x7ff5e00b2910 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5e4fe2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e00b1f40 con 0x7ff5e015e010 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5e4fe2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e00b1c10 con 0x7ff5e00b3240 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5de59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5e015e010 0x7ff5e00cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5ded9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff5e00b2910 0x7ff5e00d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36514/0 (socket says 172.21.15.122:36514) 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.388+0000 7ff5ded9d640 1 -- 172.21.15.122:0/1118134991 learned_addr learned my addr 172.21.15.122:0/1118134991 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.389+0000 7ff5ddd9b640 1 --2- 172.21.15.122:0/1118134991 >> v2:172.21.15.169:3300/0 conn(0x7ff5e00b3240 0x7ff5e00b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.389+0000 7ff5de59c640 1 -- 172.21.15.122:0/1118134991 >> v2:172.21.15.169:3300/0 conn(0x7ff5e00b3240 msgr2=0x7ff5e00b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.389+0000 7ff5de59c640 1 --2- 172.21.15.122:0/1118134991 >> v2:172.21.15.169:3300/0 conn(0x7ff5e00b3240 0x7ff5e00b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.389+0000 7ff5de59c640 1 -- 172.21.15.122:0/1118134991 >> v1:172.21.15.122:6789/0 conn(0x7ff5e00b2910 legacy=0x7ff5e00d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.389+0000 7ff5de59c640 1 -- 172.21.15.122:0/1118134991 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e006d1b0 con 0x7ff5e015e010 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.390+0000 7ff5de59c640 1 --2- 172.21.15.122:0/1118134991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e00cedd0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff5e006e2a0 tx=0x7ff5d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6f6f315c1f4a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.390+0000 7ff5dd59a640 1 -- 172.21.15.122:0/1118134991 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0035020 con 0x7ff5e015e010 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.390+0000 7ff5dd59a640 1 -- 172.21.15.122:0/1118134991 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0031830 con 0x7ff5e015e010 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.390+0000 7ff5dd59a640 1 -- 172.21.15.122:0/1118134991 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0031c00 con 0x7ff5e015e010 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.390+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/1118134991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 msgr2=0x7ff5e00cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.390+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/1118134991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e00cedd0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff5e006e2a0 tx=0x7ff5d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.391+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/1118134991 shutdown_connections 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.391+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/1118134991 >> v2:172.21.15.169:3300/0 conn(0x7ff5e00b3240 0x7ff5e00b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.391+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/1118134991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e00cedd0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.391+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/1118134991 >> 172.21.15.122:0/1118134991 conn(0x7ff5e009a9f0 msgr2=0x7ff5e009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.391+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/1118134991 shutdown_connections 2024-03-06T07:25:40.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.391+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/1118134991 wait complete. 2024-03-06T07:25:40.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.392+0000 7ff5e4fe2640 1 Processor -- start 2024-03-06T07:25:40.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.392+0000 7ff5e4fe2640 1 -- start start 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e01f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5e00b6860 0x7ff5e01521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5e0156750 0x7ff5e00d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00cf310 con 0x7ff5e015e010 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e006ced0 con 0x7ff5e00b6860 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00d21e0 con 0x7ff5e0156750 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e01f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e01f7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60848/0 (socket says 172.21.15.122:60848) 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 -- 172.21.15.122:0/2955990330 learned_addr learned my addr 172.21.15.122:0/2955990330 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 -- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5e0156750 msgr2=0x7ff5e00d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5ddd9b640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5e00b6860 0x7ff5e01521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5e0156750 0x7ff5e00d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 -- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5e00b6860 msgr2=0x7ff5e01521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5e00b6860 0x7ff5e01521d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 -- 172.21.15.122:0/2955990330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e01f7cf0 con 0x7ff5e015e010 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5de59c640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e01f7710 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009890 tx=0x7ff5d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0041070 con 0x7ff5e015e010 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d003aa30 con 0x7ff5e015e010 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0035040 con 0x7ff5e015e010 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.393+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e01f7fd0 con 0x7ff5e015e010 2024-03-06T07:25:40.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.394+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e01f8590 con 0x7ff5e015e010 2024-03-06T07:25:40.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.394+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e00c5560 con 0x7ff5e015e010 2024-03-06T07:25:40.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.395+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5d0038020 con 0x7ff5e015e010 2024-03-06T07:25:40.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.396+0000 7ff5ceffd640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff5c006b4d0 0x7ff5c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.396+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff5d00abca0 con 0x7ff5e015e010 2024-03-06T07:25:40.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.396+0000 7ff5ddd9b640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff5c006b4d0 0x7ff5c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.396+0000 7ff5ddd9b640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff5c006b4d0 0x7ff5c006d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4006fd0 tx=0x7ff5d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.400+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5d007aae0 con 0x7ff5e015e010 2024-03-06T07:25:40.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.616+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff5e004f330 con 0x7ff5e015e010 2024-03-06T07:25:40.618 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:25:40.616+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:40.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.618+0000 7ff5ceffd640 1 -- 172.21.15.122:0/2955990330 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff5d007a3e0 con 0x7ff5e015e010 2024-03-06T07:25:40.619 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:40.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff5c006b4d0 msgr2=0x7ff5c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff5c006b4d0 0x7ff5c006d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4006fd0 tx=0x7ff5d4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 msgr2=0x7ff5e01f7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e01f7710 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009890 tx=0x7ff5d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 shutdown_connections 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff5c006b4d0 0x7ff5c006d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5e0156750 0x7ff5e00d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5e00b6860 0x7ff5e01521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 --2- 172.21.15.122:0/2955990330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5e015e010 0x7ff5e01f7710 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.621+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 >> 172.21.15.122:0/2955990330 conn(0x7ff5e009a9f0 msgr2=0x7ff5e015d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.622+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 shutdown_connections 2024-03-06T07:25:40.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.622+0000 7ff5e4fe2640 1 -- 172.21.15.122:0/2955990330 wait complete. 2024-03-06T07:25:40.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:40.662 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:25:40.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 Processor -- start 2024-03-06T07:25:40.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 -- start start 2024-03-06T07:25:40.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcab80d01a0 0x7fcab815e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcab815ea20 0x7fcab8070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab80ad900 con 0x7fcab815b940 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab815bd20 con 0x7fcab815ea20 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.736+0000 7fcabdcdf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab80adb70 con 0x7fcab80d01a0 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcab815ea20 0x7fcab8070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcab815b940 0x7fcab80cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36536/0 (socket says 172.21.15.122:36536) 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab77fe640 1 -- 172.21.15.122:0/3881528216 learned_addr learned my addr 172.21.15.122:0/3881528216 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab6ffd640 1 --2- 172.21.15.122:0/3881528216 >> v2:172.21.15.169:3300/0 conn(0x7fcab80d01a0 0x7fcab815e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab7fff640 1 -- 172.21.15.122:0/3881528216 >> v2:172.21.15.169:3300/0 conn(0x7fcab80d01a0 msgr2=0x7fcab815e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab7fff640 1 --2- 172.21.15.122:0/3881528216 >> v2:172.21.15.169:3300/0 conn(0x7fcab80d01a0 0x7fcab815e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab7fff640 1 -- 172.21.15.122:0/3881528216 >> v1:172.21.15.122:6789/0 conn(0x7fcab815b940 legacy=0x7fcab80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab67fc640 1 -- 172.21.15.122:0/3881528216 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98877071 0 0) 0x7fcab80ad900 con 0x7fcab815b940 2024-03-06T07:25:40.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.737+0000 7fcab7fff640 1 -- 172.21.15.122:0/3881528216 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab80ade50 con 0x7fcab815ea20 2024-03-06T07:25:40.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcab7fff640 1 --2- 172.21.15.122:0/3881528216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815ea20 0x7fcab8070b90 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcab80d05e0 tx=0x7fcab002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7a512530e51b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcab67fc640 1 -- 172.21.15.122:0/3881528216 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0039070 con 0x7fcab815ea20 2024-03-06T07:25:40.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcab67fc640 1 -- 172.21.15.122:0/3881528216 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab002cc80 con 0x7fcab815ea20 2024-03-06T07:25:40.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcab67fc640 1 -- 172.21.15.122:0/3881528216 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0034660 con 0x7fcab815ea20 2024-03-06T07:25:40.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 -- 172.21.15.122:0/3881528216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815ea20 msgr2=0x7fcab8070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/3881528216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815ea20 0x7fcab8070b90 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcab80d05e0 tx=0x7fcab002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 -- 172.21.15.122:0/3881528216 shutdown_connections 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/3881528216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815ea20 0x7fcab8070b90 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/3881528216 >> v2:172.21.15.169:3300/0 conn(0x7fcab80d01a0 0x7fcab815e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 -- 172.21.15.122:0/3881528216 >> 172.21.15.122:0/3881528216 conn(0x7fcab806b190 msgr2=0x7fcab806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 -- 172.21.15.122:0/3881528216 shutdown_connections 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.738+0000 7fcabdcdf640 1 -- 172.21.15.122:0/3881528216 wait complete. 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.739+0000 7fcabdcdf640 1 Processor -- start 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 -- start start 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab80d01a0 0x7fcab8145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 0x7fcab81592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcab815ea20 0x7fcab81606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab815ee90 con 0x7fcab815b940 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab815f210 con 0x7fcab80d01a0 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcabdcdf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80ae650 con 0x7fcab815ea20 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcab6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 0x7fcab81592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcab6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 0x7fcab81592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60878/0 (socket says 172.21.15.122:60878) 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.740+0000 7fcab6ffd640 1 -- 172.21.15.122:0/1987952878 learned_addr learned my addr 172.21.15.122:0/1987952878 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab77fe640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab80d01a0 0x7fcab8145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab7fff640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcab815ea20 0x7fcab81606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab6ffd640 1 -- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcab815ea20 msgr2=0x7fcab81606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab6ffd640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcab815ea20 0x7fcab81606d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab6ffd640 1 -- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab80d01a0 msgr2=0x7fcab8145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab6ffd640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab80d01a0 0x7fcab8145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab6ffd640 1 -- 172.21.15.122:0/1987952878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab81597e0 con 0x7fcab815b940 2024-03-06T07:25:40.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcab6ffd640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 0x7fcab81592a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00d5f0 tx=0x7fcaac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0125f0 con 0x7fcab815b940 2024-03-06T07:25:40.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac019070 con 0x7fcab815b940 2024-03-06T07:25:40.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab8161240 con 0x7fcab815b940 2024-03-06T07:25:40.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0129d0 con 0x7fcab815b940 2024-03-06T07:25:40.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.741+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab815c3d0 con 0x7fcab815b940 2024-03-06T07:25:40.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.742+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab80732b0 con 0x7fcab815b940 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.743+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcaac012b70 con 0x7fcab815b940 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.743+0000 7fca9ffff640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcaa406b4d0 0x7fcaa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.743+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcaac08a690 con 0x7fcab815b940 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.743+0000 7fcab77fe640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcaa406b4d0 0x7fcaa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.743+0000 7fcab77fe640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcaa406b4d0 0x7fcaa406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000f60 tx=0x7fcaa8007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.747+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcaac059480 con 0x7fcab815b940 2024-03-06T07:25:40.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:40.960+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcab806b610 con 0x7fcab815b940 2024-03-06T07:25:41.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.619+0000 7fca9ffff640 1 -- 172.21.15.122:0/1987952878 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcaac058d80 con 0x7fcab815b940 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.621+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcaa406b4d0 msgr2=0x7fcaa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.621+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcaa406b4d0 0x7fcaa406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000f60 tx=0x7fcaa8007c00 comp rx=0 tx=0).stop 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 msgr2=0x7fcab81592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 0x7fcab81592a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00d5f0 tx=0x7fcaac00dab0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 shutdown_connections 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcaa406b4d0 0x7fcaa406d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcab815ea20 0x7fcab81606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcab815b940 0x7fcab81592a0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 --2- 172.21.15.122:0/1987952878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab80d01a0 0x7fcab8145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 >> 172.21.15.122:0/1987952878 conn(0x7fcab806b190 msgr2=0x7fcab8070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 shutdown_connections 2024-03-06T07:25:41.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.622+0000 7fcabdcdf640 1 -- 172.21.15.122:0/1987952878 wait complete. 2024-03-06T07:25:41.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:41.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.778+0000 7fe18537a640 1 Processor -- start 2024-03-06T07:25:41.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.778+0000 7fe18537a640 1 -- start start 2024-03-06T07:25:41.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe18537a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe18006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe18537a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800d1860 con 0x7fe18015e050 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe18537a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800cfc00 con 0x7fe18006b4f0 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe18537a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18015e430 con 0x7fe1800ce170 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe18015e050 0x7fe1800cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36544/0 (socket says 172.21.15.122:36544) 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17effd640 1 -- 172.21.15.122:0/3851647985 learned_addr learned my addr 172.21.15.122:0/3851647985 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071034433 0 0) 0x7fe1800d1860 con 0x7fe18015e050 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c003520 con 0x7fe18015e050 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe177fff640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe18006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790311059 0 0) 0x7fe16c003520 con 0x7fe18015e050 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 msgr2=0x7fe18006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe18006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 >> v1:172.21.15.169:6789/0 conn(0x7fe18006b4f0 legacy=0x7fe18006e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800cd690 con 0x7fe18015e050 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fe170001670 con 0x7fe18015e050 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 >> v1:172.21.15.122:6789/0 conn(0x7fe18015e050 legacy=0x7fe1800cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0039e0 0x7fe16c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe16c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c004d40 0x7fe16c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18015e430 con 0x7fe16c004d40 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c00ba70 con 0x7fe16c0039e0 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c00bc70 con 0x7fe1800ce170 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.779+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c004d40 0x7fe16c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17effd640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0039e0 0x7fe16c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe177fff640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe16c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17f7fe640 1 -- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 msgr2=0x7fe16c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe16c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17f7fe640 1 -- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0039e0 msgr2=0x7fe16c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0039e0 0x7fe16c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17f7fe640 1 -- 172.21.15.122:0/3851647985 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c003520 con 0x7fe16c004d40 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe177fff640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1800ce170 0x7fe16c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.780+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c004d40 0x7fe16c0051c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe178001e20 tx=0x7fe16c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647ed7cf751b3fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178001a70 con 0x7fe16c004d40 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178003b70 con 0x7fe16c004d40 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe17e7fc640 1 -- 172.21.15.122:0/3851647985 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1780050d0 con 0x7fe16c004d40 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe18537a640 1 -- 172.21.15.122:0/3851647985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c004d40 msgr2=0x7fe16c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe18537a640 1 --2- 172.21.15.122:0/3851647985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c004d40 0x7fe16c0051c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe178001e20 tx=0x7fe16c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe17f7fe640 1 -- 172.21.15.122:0/3851647985 reap_dead start 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe18537a640 1 -- 172.21.15.122:0/3851647985 shutdown_connections 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe18537a640 1 -- 172.21.15.122:0/3851647985 >> 172.21.15.122:0/3851647985 conn(0x7fe1800d0a30 msgr2=0x7fe1800d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe18537a640 1 -- 172.21.15.122:0/3851647985 shutdown_connections 2024-03-06T07:25:41.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.781+0000 7fe18537a640 1 -- 172.21.15.122:0/3851647985 wait complete. 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.782+0000 7fe18537a640 1 Processor -- start 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.782+0000 7fe18537a640 1 -- start start 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe18537a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe180155a60 0x7fe1800ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe18537a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1801f4f00 0x7fe180158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe18537a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 0x7fe1800af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe18537a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800cdce0 con 0x7fe180158f80 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe18537a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800cd440 con 0x7fe180155a60 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe18537a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800cf7c0 con 0x7fe1801f4f00 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 0x7fe1800af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 0x7fe1800af4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60896/0 (socket says 172.21.15.122:60896) 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 -- 172.21.15.122:0/282871458 learned_addr learned my addr 172.21.15.122:0/282871458 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe177fff640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1801f4f00 0x7fe180158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17effd640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe180155a60 0x7fe1800ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 -- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1801f4f00 msgr2=0x7fe180158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1801f4f00 0x7fe180158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 -- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe180155a60 msgr2=0x7fe1800ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe180155a60 0x7fe1800ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe17f7fe640 1 -- 172.21.15.122:0/282871458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800adda0 con 0x7fe180158f80 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.783+0000 7fe177fff640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1801f4f00 0x7fe180158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.784+0000 7fe17f7fe640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 0x7fe1800af4c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe178005040 tx=0x7fe1780014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.784+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178001a70 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.784+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178003970 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.784+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe180159420 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.784+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178007db0 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.784+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe180156740 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.785+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1800753f0 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.785+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe178041030 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.785+0000 7fe1777fe640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.785+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe1780ab100 con 0x7fe180158f80 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.786+0000 7fe17effd640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.786+0000 7fe17effd640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe16006b4d0 0x7fe16006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe170007c50 tx=0x7fe170009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.790+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1800753f0 con 0x7fe180158f80 2024-03-06T07:25:41.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:41.961+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe1800cd1a0 con 0x7fe180158f80 2024-03-06T07:25:42.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.620+0000 7fe1777fe640 1 -- 172.21.15.122:0/282871458 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe178033020 con 0x7fe180158f80 2024-03-06T07:25:42.623 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe16006b4d0 msgr2=0x7fe16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe16006b4d0 0x7fe16006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe170007c50 tx=0x7fe170009040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 msgr2=0x7fe1800af4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 0x7fe1800af4c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe178005040 tx=0x7fe1780014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 shutdown_connections 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe180158f80 0x7fe1800af4c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe1801f4f00 0x7fe180158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 --2- 172.21.15.122:0/282871458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe180155a60 0x7fe1800ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 >> 172.21.15.122:0/282871458 conn(0x7fe1800d0a30 msgr2=0x7fe18015d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 shutdown_connections 2024-03-06T07:25:42.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.623+0000 7fe18537a640 1 -- 172.21.15.122:0/282871458 wait complete. 2024-03-06T07:25:42.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:42.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.740+0000 7fe7f9a84640 1 Processor -- start 2024-03-06T07:25:42.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.741+0000 7fe7f9a84640 1 -- start start 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.741+0000 7fe7f9a84640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe7f415ac00 0x7fe7f406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.741+0000 7fe7f9a84640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7f4070d10 0x7fe7f415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.741+0000 7fe7f9a84640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f40711c0 con 0x7fe7f40708b0 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.741+0000 7fe7f9a84640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f40cdc10 con 0x7fe7f4070d10 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.741+0000 7fe7f9a84640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f415afe0 con 0x7fe7f415ac00 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7f4070d10 0x7fe7f415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7f4070d10 0x7fe7f415eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60912/0 (socket says 172.21.15.122:60912) 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 -- 172.21.15.122:0/269018441 learned_addr learned my addr 172.21.15.122:0/269018441 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f37fe640 1 --2- 172.21.15.122:0/269018441 >> v2:172.21.15.169:3300/0 conn(0x7fe7f415ac00 0x7fe7f406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 -- 172.21.15.122:0/269018441 >> v2:172.21.15.169:3300/0 conn(0x7fe7f415ac00 msgr2=0x7fe7f406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 --2- 172.21.15.122:0/269018441 >> v2:172.21.15.169:3300/0 conn(0x7fe7f415ac00 0x7fe7f406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 -- 172.21.15.122:0/269018441 >> v1:172.21.15.169:6789/0 conn(0x7fe7f40708b0 legacy=0x7fe7f415e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.742+0000 7fe7f3fff640 1 -- 172.21.15.122:0/269018441 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40cd000 con 0x7fe7f4070d10 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f3fff640 1 --2- 172.21.15.122:0/269018441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f415eeb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40730c0 tx=0x7fe7ec02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65df201b1feec98b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f27fc640 1 -- 172.21.15.122:0/269018441 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec02c9f0 con 0x7fe7f4070d10 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f27fc640 1 -- 172.21.15.122:0/269018441 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec02cb90 con 0x7fe7f4070d10 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f27fc640 1 -- 172.21.15.122:0/269018441 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec0343f0 con 0x7fe7f4070d10 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 -- 172.21.15.122:0/269018441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 msgr2=0x7fe7f415eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/269018441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f415eeb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40730c0 tx=0x7fe7ec02be50 comp rx=0 tx=0).stop 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 -- 172.21.15.122:0/269018441 shutdown_connections 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/269018441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f415eeb0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/269018441 >> v2:172.21.15.169:3300/0 conn(0x7fe7f415ac00 0x7fe7f406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 -- 172.21.15.122:0/269018441 >> 172.21.15.122:0/269018441 conn(0x7fe7f40b1650 msgr2=0x7fe7f40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 -- 172.21.15.122:0/269018441 shutdown_connections 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.743+0000 7fe7f9a84640 1 -- 172.21.15.122:0/269018441 wait complete. 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 Processor -- start 2024-03-06T07:25:42.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 -- start start 2024-03-06T07:25:42.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7f40708b0 0x7fe7f4153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f4154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7f415ac00 0x7fe7f4155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40cd8a0 con 0x7fe7f4070d10 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40cd5c0 con 0x7fe7f40708b0 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.745+0000 7fe7f9a84640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f406ce50 con 0x7fe7f415ac00 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f4154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f4154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60924/0 (socket says 172.21.15.122:60924) 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 -- 172.21.15.122:0/2121204113 learned_addr learned my addr 172.21.15.122:0/2121204113 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f37fe640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7f40708b0 0x7fe7f4153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f3fff640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7f415ac00 0x7fe7f4155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 -- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7f415ac00 msgr2=0x7fe7f4155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7f415ac00 0x7fe7f4155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 -- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7f40708b0 msgr2=0x7fe7f4153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7f40708b0 0x7fe7f4153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 -- 172.21.15.122:0/2121204113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4153a80 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f2ffd640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f4154ed0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c400 tx=0x7fe7e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8011790 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e8017070 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f4156a70 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8011bb0 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.746+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f4157030 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.747+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f404f330 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.748+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7e801c460 con 0x7fe7f4070d10 2024-03-06T07:25:42.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.748+0000 7fe7dbfff640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe7e006b4d0 0x7fe7e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.748+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fe7e8089e20 con 0x7fe7f4070d10 2024-03-06T07:25:42.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.748+0000 7fe7f37fe640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe7e006b4d0 0x7fe7e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.748+0000 7fe7f37fe640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe7e006b4d0 0x7fe7e006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40079a0 tx=0x7fe7e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.752+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7f4153a80 con 0x7fe7f4070d10 2024-03-06T07:25:42.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:42.926+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe7f4074880 con 0x7fe7f4070d10 2024-03-06T07:25:43.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.622+0000 7fe7dbfff640 1 -- 172.21.15.122:0/2121204113 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe7e8058e50 con 0x7fe7f4070d10 2024-03-06T07:25:43.624 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:43.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe7e006b4d0 msgr2=0x7fe7e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe7e006b4d0 0x7fe7e006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40079a0 tx=0x7fe7e4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:43.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 msgr2=0x7fe7f4154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f4154ed0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c400 tx=0x7fe7e800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 shutdown_connections 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe7e006b4d0 0x7fe7e006d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7f415ac00 0x7fe7f4155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7f4070d10 0x7fe7f4154ed0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 --2- 172.21.15.122:0/2121204113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7f40708b0 0x7fe7f4153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.625+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 >> 172.21.15.122:0/2121204113 conn(0x7fe7f40b1650 msgr2=0x7fe7f40af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.626+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 shutdown_connections 2024-03-06T07:25:43.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.626+0000 7fe7f9a84640 1 -- 172.21.15.122:0/2121204113 wait complete. 2024-03-06T07:25:43.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:43.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.743+0000 7f99726f4640 1 Processor -- start 2024-03-06T07:25:43.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.743+0000 7f99726f4640 1 -- start start 2024-03-06T07:25:43.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f99726f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 0x7f996c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f99726f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 0x7f996c0b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f99726f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996c158ac0 con 0x7f996c1586e0 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f99726f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c15b2a0 con 0x7f996c0cfa90 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f99726f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c15b540 con 0x7f996c0b33a0 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f996b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 0x7f996c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f9970c91640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 0x7f996c0b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f996bfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f996c1586e0 0x7f996c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:43316/0 (socket says 172.21.15.122:43316) 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f996b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 0x7f996c0cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:45068/0 (socket says 172.21.15.122:45068) 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f9970c91640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 0x7f996c0b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:55824/0 (socket says 172.21.15.122:55824) 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.744+0000 7f996bfff640 1 -- 172.21.15.122:0/2417923551 learned_addr learned my addr 172.21.15.122:0/2417923551 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.745+0000 7f9970c91640 1 -- 172.21.15.122:0/2417923551 >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 msgr2=0x7f996c0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.745+0000 7f9970c91640 1 --2- 172.21.15.122:0/2417923551 >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 0x7f996c0cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.745+0000 7f9970c91640 1 -- 172.21.15.122:0/2417923551 >> v1:172.21.15.169:6789/0 conn(0x7f996c1586e0 legacy=0x7f996c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.745+0000 7f9970c91640 1 -- 172.21.15.122:0/2417923551 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c15b820 con 0x7f996c0b33a0 2024-03-06T07:25:43.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.745+0000 7f996b7fe640 1 --2- 172.21.15.122:0/2417923551 >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 0x7f996c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:43.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.745+0000 7f9970c91640 1 --2- 172.21.15.122:0/2417923551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 0x7f996c0b37c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9964009d90 tx=0x7f996402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d352f1368785ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f996affd640 1 -- 172.21.15.122:0/2417923551 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964039070 con 0x7f996c0b33a0 2024-03-06T07:25:43.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f996affd640 1 -- 172.21.15.122:0/2417923551 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996402cc50 con 0x7f996c0b33a0 2024-03-06T07:25:43.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f996affd640 1 -- 172.21.15.122:0/2417923551 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964034440 con 0x7f996c0b33a0 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 -- 172.21.15.122:0/2417923551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 msgr2=0x7f996c0b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 --2- 172.21.15.122:0/2417923551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 0x7f996c0b37c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9964009d90 tx=0x7f996402bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 -- 172.21.15.122:0/2417923551 shutdown_connections 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 --2- 172.21.15.122:0/2417923551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c0b33a0 0x7f996c0b37c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 --2- 172.21.15.122:0/2417923551 >> v2:172.21.15.169:3300/0 conn(0x7f996c0cfa90 0x7f996c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 -- 172.21.15.122:0/2417923551 >> 172.21.15.122:0/2417923551 conn(0x7f996c06b190 msgr2=0x7f996c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 -- 172.21.15.122:0/2417923551 shutdown_connections 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.746+0000 7f99726f4640 1 -- 172.21.15.122:0/2417923551 wait complete. 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.747+0000 7f99726f4640 1 Processor -- start 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.747+0000 7f99726f4640 1 -- start start 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f99726f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f996c0b33a0 0x7f996c141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f99726f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 0x7f996c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f99726f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c1586e0 0x7f996c157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f99726f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0af150 con 0x7f996c0cfa90 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f99726f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c15bde0 con 0x7f996c0b33a0 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f99726f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c071350 con 0x7f996c1586e0 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f996b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 0x7f996c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f996b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 0x7f996c156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60934/0 (socket says 172.21.15.122:60934) 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f996b7fe640 1 -- 172.21.15.122:0/3140123653 learned_addr learned my addr 172.21.15.122:0/3140123653 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f996bfff640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f996c0b33a0 0x7f996c141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.748+0000 7f9970c91640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c1586e0 0x7f996c157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f996b7fe640 1 -- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c1586e0 msgr2=0x7f996c157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f996b7fe640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c1586e0 0x7f996c157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f996b7fe640 1 -- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f996c0b33a0 msgr2=0x7f996c141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f996b7fe640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f996c0b33a0 0x7f996c141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f996b7fe640 1 -- 172.21.15.122:0/3140123653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c1e4ca0 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f996b7fe640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 0x7f996c156dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f996000ec10 tx=0x7f996000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960017070 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996000cc30 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960013470 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c1e4f80 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.749+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c1e5540 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.750+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c0732b0 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.750+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f996001a070 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.751+0000 7f9968ff9640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f995806b4d0 0x7f995806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.751+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9960089de0 con 0x7f996c0cfa90 2024-03-06T07:25:43.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.751+0000 7f996bfff640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f995806b4d0 0x7f995806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.751+0000 7f996bfff640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f995806b4d0 0x7f995806d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f995c007fd0 tx=0x7f995c007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.755+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9960058910 con 0x7f996c0cfa90 2024-03-06T07:25:43.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.926+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f996c0aeca0 con 0x7f996c0cfa90 2024-03-06T07:25:43.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.930+0000 7f9968ff9640 1 -- 172.21.15.122:0/3140123653 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v24) v1 ==== 108+0+0 (secure 0 0 0) 0x7f996005c9b0 con 0x7f996c0cfa90 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f995806b4d0 msgr2=0x7f995806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f995806b4d0 0x7f995806d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f995c007fd0 tx=0x7f995c007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 msgr2=0x7f996c156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 0x7f996c156dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f996000ec10 tx=0x7f996000c520 comp rx=0 tx=0).stop 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 shutdown_connections 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f995806b4d0 0x7f995806d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f996c1586e0 0x7f996c157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f996c0cfa90 0x7f996c156dc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 --2- 172.21.15.122:0/3140123653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f996c0b33a0 0x7f996c141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 >> 172.21.15.122:0/3140123653 conn(0x7f996c06b190 msgr2=0x7f996c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 shutdown_connections 2024-03-06T07:25:43.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:43.932+0000 7f99726f4640 1 -- 172.21.15.122:0/3140123653 wait complete. 2024-03-06T07:25:43.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:44.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.045+0000 7ff0b82df640 1 Processor -- start 2024-03-06T07:25:44.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.045+0000 7ff0b82df640 1 -- start start 2024-03-06T07:25:44.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b82df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff0b015ecd0 0x7ff0b00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b82df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b00cfa50 con 0x7ff0b00d14a0 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b82df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b00b45c0 con 0x7ff0b00d10c0 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b82df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b015c050 con 0x7ff0b015ecd0 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b687c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff0b015ecd0 0x7ff0b00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b587a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff0b00d14a0 0x7ff0b015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36554/0 (socket says 172.21.15.122:36554) 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.046+0000 7ff0b587a640 1 -- 172.21.15.122:0/3408006156 learned_addr learned my addr 172.21.15.122:0/3408006156 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.047+0000 7ff0b687c640 1 -- 172.21.15.122:0/3408006156 >> v1:172.21.15.169:6789/0 conn(0x7ff0b00d10c0 legacy=0x7ff0b00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.047+0000 7ff0b687c640 1 -- 172.21.15.122:0/3408006156 >> v1:172.21.15.122:6789/0 conn(0x7ff0b00d14a0 legacy=0x7ff0b015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.047+0000 7ff0b687c640 1 -- 172.21.15.122:0/3408006156 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b015f160 con 0x7ff0b015ecd0 2024-03-06T07:25:44.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.047+0000 7ff0b687c640 1 --2- 172.21.15.122:0/3408006156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b015ecd0 0x7ff0b00cae80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80097a0 tx=0x7ff0a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c0c37a99c3fd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b5079640 1 -- 172.21.15.122:0/3408006156 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8035020 con 0x7ff0b015ecd0 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b5079640 1 -- 172.21.15.122:0/3408006156 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8031830 con 0x7ff0b015ecd0 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b5079640 1 -- 172.21.15.122:0/3408006156 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8031c00 con 0x7ff0b015ecd0 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 -- 172.21.15.122:0/3408006156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b015ecd0 msgr2=0x7ff0b00cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 --2- 172.21.15.122:0/3408006156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b015ecd0 0x7ff0b00cae80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80097a0 tx=0x7ff0a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 -- 172.21.15.122:0/3408006156 shutdown_connections 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 --2- 172.21.15.122:0/3408006156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b015ecd0 0x7ff0b00cae80 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 -- 172.21.15.122:0/3408006156 >> 172.21.15.122:0/3408006156 conn(0x7ff0b00b3800 msgr2=0x7ff0b00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 -- 172.21.15.122:0/3408006156 shutdown_connections 2024-03-06T07:25:44.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.048+0000 7ff0b82df640 1 -- 172.21.15.122:0/3408006156 wait complete. 2024-03-06T07:25:44.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.049+0000 7ff0b82df640 1 Processor -- start 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.049+0000 7ff0b82df640 1 -- start start 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b82df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0b00d10c0 0x7ff0b00b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b82df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0b00d14a0 0x7ff0b00b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b82df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 0x7ff0b006c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b82df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00cfcf0 con 0x7ff0b0145830 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b82df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00d0060 con 0x7ff0b00d10c0 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b82df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00cb3c0 con 0x7ff0b00d14a0 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b687c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 0x7ff0b006c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b687c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 0x7ff0b006c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60952/0 (socket says 172.21.15.122:60952) 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b687c640 1 -- 172.21.15.122:0/553620042 learned_addr learned my addr 172.21.15.122:0/553620042 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b607b640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0b00d10c0 0x7ff0b00b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b587a640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0b00d14a0 0x7ff0b00b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.050+0000 7ff0b687c640 1 -- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0b00d14a0 msgr2=0x7ff0b00b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b687c640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0b00d14a0 0x7ff0b00b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b687c640 1 -- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0b00d10c0 msgr2=0x7ff0b00b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b687c640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0b00d10c0 0x7ff0b00b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b687c640 1 -- 172.21.15.122:0/553620042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0160550 con 0x7ff0b0145830 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b607b640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0b00d10c0 0x7ff0b00b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b587a640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0b00d14a0 0x7ff0b00b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b687c640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 0x7ff0b006c0a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80097a0 tx=0x7ff0a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8040070 con 0x7ff0b0145830 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a80398c0 con 0x7ff0b0145830 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b00b5440 con 0x7ff0b0145830 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8035040 con 0x7ff0b0145830 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.051+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b00b5a00 con 0x7ff0b0145830 2024-03-06T07:25:44.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.052+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b009fe60 con 0x7ff0b0145830 2024-03-06T07:25:44.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.052+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0a8038020 con 0x7ff0b0145830 2024-03-06T07:25:44.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.053+0000 7ff0a6ffd640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff09406b500 0x7ff09406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.053+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff0a80ab900 con 0x7ff0b0145830 2024-03-06T07:25:44.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.053+0000 7ff0b607b640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff09406b500 0x7ff09406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.053+0000 7ff0b607b640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff09406b500 0x7ff09406d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac004520 tx=0x7ff0ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.057+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0a807aa00 con 0x7ff0b0145830 2024-03-06T07:25:44.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.236+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff0b006f730 con 0x7ff0b0145830 2024-03-06T07:25:44.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.239+0000 7ff0a6ffd640 1 -- 172.21.15.122:0/553620042 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff0a807a300 con 0x7ff0b0145830 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff09406b500 msgr2=0x7ff09406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff09406b500 0x7ff09406d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac004520 tx=0x7ff0ac009040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 msgr2=0x7ff0b006c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 0x7ff0b006c0a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80097a0 tx=0x7ff0a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 shutdown_connections 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff09406b500 0x7ff09406d9c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0b0145830 0x7ff0b006c0a0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0b00d14a0 0x7ff0b00b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 --2- 172.21.15.122:0/553620042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0b00d10c0 0x7ff0b00b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 >> 172.21.15.122:0/553620042 conn(0x7ff0b00b3800 msgr2=0x7ff0b015e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 shutdown_connections 2024-03-06T07:25:44.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.241+0000 7ff0b82df640 1 -- 172.21.15.122:0/553620042 wait complete. 2024-03-06T07:25:44.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:25:44.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.353+0000 7fbe9ff63640 1 Processor -- start 2024-03-06T07:25:44.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.353+0000 7fbe9ff63640 1 -- start start 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9ff63640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbe980b3060 0x7fbe9815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9ff63640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbe9815bf30 0x7fbe9815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9ff63640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9815f2d0 con 0x7fbe9813dc10 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9ff63640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980cf3f0 con 0x7fbe980b3060 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9ff63640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9815b210 con 0x7fbe9815bf30 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9d4fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbe980b3060 0x7fbe9815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9dcff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbe9813dc10 0x7fbe980b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36556/0 (socket says 172.21.15.122:36556) 2024-03-06T07:25:44.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9dcff640 1 -- 172.21.15.122:0/1117927431 learned_addr learned my addr 172.21.15.122:0/1117927431 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:44.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9d4fe640 1 -- 172.21.15.122:0/1117927431 >> v2:172.21.15.169:3300/0 conn(0x7fbe9815bf30 msgr2=0x7fbe9815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9d4fe640 1 --2- 172.21.15.122:0/1117927431 >> v2:172.21.15.169:3300/0 conn(0x7fbe9815bf30 0x7fbe9815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9d4fe640 1 -- 172.21.15.122:0/1117927431 >> v1:172.21.15.122:6789/0 conn(0x7fbe9813dc10 legacy=0x7fbe980b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9ccfd640 1 -- 172.21.15.122:0/1117927431 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540466387 0 0) 0x7fbe9815f2d0 con 0x7fbe9813dc10 2024-03-06T07:25:44.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.354+0000 7fbe9d4fe640 1 -- 172.21.15.122:0/1117927431 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9815f580 con 0x7fbe980b3060 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.355+0000 7fbe9d4fe640 1 --2- 172.21.15.122:0/1117927431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9815e850 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbe980b4e00 tx=0x7fbe8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad90962d28abccf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.355+0000 7fbe9ccfd640 1 -- 172.21.15.122:0/1117927431 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe880315a0 con 0x7fbe980b3060 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.355+0000 7fbe9ccfd640 1 -- 172.21.15.122:0/1117927431 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe88037070 con 0x7fbe980b3060 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.355+0000 7fbe9ccfd640 1 -- 172.21.15.122:0/1117927431 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88031960 con 0x7fbe980b3060 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.355+0000 7fbe9ff63640 1 -- 172.21.15.122:0/1117927431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 msgr2=0x7fbe9815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.355+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/1117927431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9815e850 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbe980b4e00 tx=0x7fbe8802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.356+0000 7fbe9ff63640 1 -- 172.21.15.122:0/1117927431 shutdown_connections 2024-03-06T07:25:44.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.356+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/1117927431 >> v2:172.21.15.169:3300/0 conn(0x7fbe9815bf30 0x7fbe9815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.356+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/1117927431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9815e850 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.356+0000 7fbe9ff63640 1 -- 172.21.15.122:0/1117927431 >> 172.21.15.122:0/1117927431 conn(0x7fbe980cf8a0 msgr2=0x7fbe980cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.356+0000 7fbe9ff63640 1 -- 172.21.15.122:0/1117927431 shutdown_connections 2024-03-06T07:25:44.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.356+0000 7fbe9ff63640 1 -- 172.21.15.122:0/1117927431 wait complete. 2024-03-06T07:25:44.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 Processor -- start 2024-03-06T07:25:44.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 -- start start 2024-03-06T07:25:44.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9813e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe9813dc10 0x7fbe98132cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe9815bf30 0x7fbe9813ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9813dff0 con 0x7fbe980b3060 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9815b730 con 0x7fbe9813dc10 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.357+0000 7fbe9ff63640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980708d0 con 0x7fbe9815bf30 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9813e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9813e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60972/0 (socket says 172.21.15.122:60972) 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 -- 172.21.15.122:0/2486332610 learned_addr learned my addr 172.21.15.122:0/2486332610 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9d4fe640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe9813dc10 0x7fbe98132cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 -- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe9815bf30 msgr2=0x7fbe9813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe9815bf30 0x7fbe9813ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 -- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe9813dc10 msgr2=0x7fbe98132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe9813dc10 0x7fbe98132cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 -- 172.21.15.122:0/2486332610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9813f9c0 con 0x7fbe980b3060 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9d4fe640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe9813dc10 0x7fbe98132cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9dcff640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9813e3e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c4d0 tx=0x7fbe9400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94018070 con 0x7fbe980b3060 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe94007500 con 0x7fbe980b3060 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9815cbe0 con 0x7fbe980b3060 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94004a20 con 0x7fbe980b3060 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.358+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9815d1a0 con 0x7fbe980b3060 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.359+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9804f330 con 0x7fbe980b3060 2024-03-06T07:25:44.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.360+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe9401b070 con 0x7fbe980b3060 2024-03-06T07:25:44.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.360+0000 7fbe8e7fc640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe7c06b4d0 0x7fbe7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.360+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbe94089240 con 0x7fbe980b3060 2024-03-06T07:25:44.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.360+0000 7fbe9d4fe640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe7c06b4d0 0x7fbe7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.360+0000 7fbe9d4fe640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe7c06b4d0 0x7fbe7c06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbe88036000 tx=0x7fbe88008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.364+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe940583c0 con 0x7fbe980b3060 2024-03-06T07:25:44.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.602+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fbe980b2950 con 0x7fbe980b3060 2024-03-06T07:25:44.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.624+0000 7fbe8e7fc640 1 -- 172.21.15.122:0/2486332610 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v221) v1 ==== 140+0+0 (secure 0 0 0) 0x7fbe9405e0c0 con 0x7fbe980b3060 2024-03-06T07:25:44.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe7c06b4d0 msgr2=0x7fbe7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe7c06b4d0 0x7fbe7c06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbe88036000 tx=0x7fbe88008030 comp rx=0 tx=0).stop 2024-03-06T07:25:44.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 msgr2=0x7fbe9813e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9813e3e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c4d0 tx=0x7fbe9400c990 comp rx=0 tx=0).stop 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 shutdown_connections 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbe7c06b4d0 0x7fbe7c06d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbe9815bf30 0x7fbe9813ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe9813dc10 0x7fbe98132cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 --2- 172.21.15.122:0/2486332610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe980b3060 0x7fbe9813e3e0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 >> 172.21.15.122:0/2486332610 conn(0x7fbe980cf8a0 msgr2=0x7fbe98071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 shutdown_connections 2024-03-06T07:25:44.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.626+0000 7fbe9ff63640 1 -- 172.21.15.122:0/2486332610 wait complete. 2024-03-06T07:25:44.639 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:25:44.639 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:25:44.639 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:44.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:44.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.738+0000 7f4ff457a640 1 Processor -- start 2024-03-06T07:25:44.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.738+0000 7f4ff457a640 1 -- start start 2024-03-06T07:25:44.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff457a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 0x7f4fec15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff457a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 0x7f4fec15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff457a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec15f2d0 con 0x7f4fec13dc10 2024-03-06T07:25:44.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff457a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0cf3f0 con 0x7f4fec0b3060 2024-03-06T07:25:44.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff457a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec15b210 con 0x7f4fec15bf30 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff1b15640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 0x7f4fec15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff2b17640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 0x7f4fec15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.739+0000 7f4ff1b15640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 0x7f4fec15e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54758/0 (socket says 172.21.15.122:54758) 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff1b15640 1 -- 172.21.15.122:0/452520284 learned_addr learned my addr 172.21.15.122:0/452520284 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff2316640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4fec13dc10 0x7f4fec0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:34904/0 (socket says 172.21.15.122:34904) 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff2b17640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 0x7f4fec15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:37358/0 (socket says 172.21.15.122:37358) 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff2b17640 1 -- 172.21.15.122:0/452520284 >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 msgr2=0x7f4fec15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff2b17640 1 --2- 172.21.15.122:0/452520284 >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 0x7f4fec15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff2b17640 1 -- 172.21.15.122:0/452520284 >> v1:172.21.15.169:6789/0 conn(0x7f4fec13dc10 legacy=0x7f4fec0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff2b17640 1 -- 172.21.15.122:0/452520284 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec15f580 con 0x7f4fec15bf30 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.740+0000 7f4ff1b15640 1 --2- 172.21.15.122:0/452520284 >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 0x7f4fec15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:44.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff2b17640 1 --2- 172.21.15.122:0/452520284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 0x7f4fec15ed90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40097a0 tx=0x7f4fe402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846b2cf89f007cb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff1314640 1 -- 172.21.15.122:0/452520284 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40315a0 con 0x7f4fec15bf30 2024-03-06T07:25:44.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff1314640 1 -- 172.21.15.122:0/452520284 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4037070 con 0x7f4fec15bf30 2024-03-06T07:25:44.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff1314640 1 -- 172.21.15.122:0/452520284 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4031960 con 0x7f4fec15bf30 2024-03-06T07:25:44.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff457a640 1 -- 172.21.15.122:0/452520284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 msgr2=0x7f4fec15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff457a640 1 --2- 172.21.15.122:0/452520284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 0x7f4fec15ed90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40097a0 tx=0x7f4fe402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff457a640 1 -- 172.21.15.122:0/452520284 shutdown_connections 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff457a640 1 --2- 172.21.15.122:0/452520284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec15bf30 0x7f4fec15ed90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff457a640 1 --2- 172.21.15.122:0/452520284 >> v2:172.21.15.169:3300/0 conn(0x7f4fec0b3060 0x7f4fec15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.741+0000 7f4ff457a640 1 -- 172.21.15.122:0/452520284 >> 172.21.15.122:0/452520284 conn(0x7f4fec0cf8a0 msgr2=0x7f4fec0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.742+0000 7f4ff457a640 1 -- 172.21.15.122:0/452520284 shutdown_connections 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.742+0000 7f4ff457a640 1 -- 172.21.15.122:0/452520284 wait complete. 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 Processor -- start 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 -- start start 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec0b3060 0x7f4fec13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fec13dc10 0x7f4fec1443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 0x7f4fec141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec13dff0 con 0x7f4fec15bf30 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec15b730 con 0x7f4fec13dc10 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff457a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0cd7a0 con 0x7f4fec0b3060 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff2b17640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 0x7f4fec141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff2b17640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 0x7f4fec141500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50844/0 (socket says 172.21.15.122:50844) 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.743+0000 7f4ff2b17640 1 -- 172.21.15.122:0/410513060 learned_addr learned my addr 172.21.15.122:0/410513060 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff1b15640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fec13dc10 0x7f4fec1443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2316640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec0b3060 0x7f4fec13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2b17640 1 -- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec0b3060 msgr2=0x7f4fec13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2b17640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec0b3060 0x7f4fec13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2b17640 1 -- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fec13dc10 msgr2=0x7f4fec1443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2b17640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fec13dc10 0x7f4fec1443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2b17640 1 -- 172.21.15.122:0/410513060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec140dc0 con 0x7f4fec15bf30 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff2b17640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 0x7f4fec141500 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4034040 tx=0x7f4fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4032630 con 0x7f4fec15bf30 2024-03-06T07:25:44.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe40327f0 con 0x7f4fec15bf30 2024-03-06T07:25:44.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec1410a0 con 0x7f4fec15bf30 2024-03-06T07:25:44.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4032be0 con 0x7f4fec15bf30 2024-03-06T07:25:44.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.744+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec13c850 con 0x7f4fec15bf30 2024-03-06T07:25:44.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.745+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec04f330 con 0x7f4fec15bf30 2024-03-06T07:25:44.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.745+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fe403b3f0 con 0x7f4fec15bf30 2024-03-06T07:25:44.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.746+0000 7f4fe2ffd640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.746+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4fe402f030 con 0x7f4fec15bf30 2024-03-06T07:25:44.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.746+0000 7f4ff2316640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.746+0000 7f4ff2316640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8007fd0 tx=0x7f4fe80016d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.750+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fe4040070 con 0x7f4fec15bf30 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.924+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fec0afaa0 con 0x7f4fec15bf30 2024-03-06T07:25:44.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.925+0000 7f4fe2ffd640 1 -- 172.21.15.122:0/410513060 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13021 (secure 0 0 0) 0x7f4fe407a6c0 con 0x7f4fec15bf30 2024-03-06T07:25:44.927 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:44.927 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":74,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:43.621986+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/323983625":"2024-03-07T07:25:40.617632+0000","172.21.15.122:6835/323983625":"2024-03-07T07:25:40.617632+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:44.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fd006b4d0 msgr2=0x7f4fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8007fd0 tx=0x7f4fe80016d0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 msgr2=0x7f4fec141500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 0x7f4fec141500 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4034040 tx=0x7f4fe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 shutdown_connections 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec15bf30 0x7f4fec141500 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fec13dc10 0x7f4fec1443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 --2- 172.21.15.122:0/410513060 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fec0b3060 0x7f4fec13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 >> 172.21.15.122:0/410513060 conn(0x7f4fec0cf8a0 msgr2=0x7f4fec0b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 shutdown_connections 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:44.927+0000 7f4ff457a640 1 -- 172.21.15.122:0/410513060 wait complete. 2024-03-06T07:25:44.940 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:44.940 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:45.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.040+0000 7fa81e85c640 1 Processor -- start 2024-03-06T07:25:45.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.040+0000 7fa81e85c640 1 -- start start 2024-03-06T07:25:45.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81e85c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa81815a220 0x7fa8180cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81e85c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa8180712d0 0x7fa818071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81e85c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa81815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81e85c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180cf610 con 0x7fa81815a220 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81e85c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa818071c70 con 0x7fa8180712d0 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81e85c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180b3320 con 0x7fa81815de60 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa81815a220 0x7fa8180cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa81815a220 0x7fa8180cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50856/0 (socket says 172.21.15.122:50856) 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 -- 172.21.15.122:0/3915781736 learned_addr learned my addr 172.21.15.122:0/3915781736 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa8177fe640 1 --2- 172.21.15.122:0/3915781736 >> v2:172.21.15.169:3300/0 conn(0x7fa8180712d0 0x7fa818071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa81cdf9640 1 --2- 172.21.15.122:0/3915781736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa81815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 -- 172.21.15.122:0/3915781736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 msgr2=0x7fa81815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 --2- 172.21.15.122:0/3915781736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa81815e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 -- 172.21.15.122:0/3915781736 >> v2:172.21.15.169:3300/0 conn(0x7fa8180712d0 msgr2=0x7fa818071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 --2- 172.21.15.122:0/3915781736 >> v2:172.21.15.169:3300/0 conn(0x7fa8180712d0 0x7fa818071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.041+0000 7fa817fff640 1 -- 172.21.15.122:0/3915781736 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180728d0 con 0x7fa81815a220 2024-03-06T07:25:45.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.042+0000 7fa817fff640 1 --2- 172.21.15.122:0/3915781736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa81815a220 0x7fa8180cf0d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa8180d0980 tx=0x7fa80802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c40d584982eaab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.042+0000 7fa816ffd640 1 -- 172.21.15.122:0/3915781736 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80802e070 con 0x7fa81815a220 2024-03-06T07:25:45.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.042+0000 7fa816ffd640 1 -- 172.21.15.122:0/3915781736 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa808037070 con 0x7fa81815a220 2024-03-06T07:25:45.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.042+0000 7fa816ffd640 1 -- 172.21.15.122:0/3915781736 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808031a00 con 0x7fa81815a220 2024-03-06T07:25:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 -- 172.21.15.122:0/3915781736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa81815a220 msgr2=0x7fa8180cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 --2- 172.21.15.122:0/3915781736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa81815a220 0x7fa8180cf0d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa8180d0980 tx=0x7fa80802b880 comp rx=0 tx=0).stop 2024-03-06T07:25:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 -- 172.21.15.122:0/3915781736 shutdown_connections 2024-03-06T07:25:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 --2- 172.21.15.122:0/3915781736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa81815e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 --2- 172.21.15.122:0/3915781736 >> v2:172.21.15.169:3300/0 conn(0x7fa8180712d0 0x7fa818071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 --2- 172.21.15.122:0/3915781736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa81815a220 0x7fa8180cf0d0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 -- 172.21.15.122:0/3915781736 >> 172.21.15.122:0/3915781736 conn(0x7fa81806b190 msgr2=0x7fa81806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 -- 172.21.15.122:0/3915781736 shutdown_connections 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.043+0000 7fa81e85c640 1 -- 172.21.15.122:0/3915781736 wait complete. 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.044+0000 7fa81e85c640 1 Processor -- start 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.044+0000 7fa81e85c640 1 -- start start 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa81e85c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 0x7fa8181f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa81e85c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815a220 0x7fa818159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa81e85c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa8181471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa81e85c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa818072030 con 0x7fa8180712d0 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa81e85c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa818072bb0 con 0x7fa81815a220 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa81e85c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa818072530 con 0x7fa81815de60 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 0x7fa8181f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 0x7fa8181f0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50866/0 (socket says 172.21.15.122:50866) 2024-03-06T07:25:45.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 -- 172.21.15.122:0/2118426674 learned_addr learned my addr 172.21.15.122:0/2118426674 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 -- 172.21.15.122:0/2118426674 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 msgr2=0x7fa8181471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa8181471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 -- 172.21.15.122:0/2118426674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815a220 msgr2=0x7fa818159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815a220 0x7fa818159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.045+0000 7fa817fff640 1 -- 172.21.15.122:0/2118426674 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180a95f0 con 0x7fa8180712d0 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.046+0000 7fa817fff640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 0x7fa8181f0010 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa80802b8c0 tx=0x7fa8080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.046+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808032880 con 0x7fa8180712d0 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.046+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa808032a20 con 0x7fa8180712d0 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.046+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8180a98d0 con 0x7fa8180712d0 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.046+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808032e10 con 0x7fa8180712d0 2024-03-06T07:25:45.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.046+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8180a9e90 con 0x7fa8180712d0 2024-03-06T07:25:45.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.047+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8180732b0 con 0x7fa8180712d0 2024-03-06T07:25:45.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.047+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8080453f0 con 0x7fa8180712d0 2024-03-06T07:25:45.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.047+0000 7fa814ff9640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.047+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa8080abef0 con 0x7fa8180712d0 2024-03-06T07:25:45.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.047+0000 7fa8177fe640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.048+0000 7fa8177fe640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa80c006f00 tx=0x7fa80c006e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.052+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa808040070 con 0x7fa8180712d0 2024-03-06T07:25:45.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.224+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa818070ba0 con 0x7fa8180712d0 2024-03-06T07:25:45.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.224+0000 7fa814ff9640 1 -- 172.21.15.122:0/2118426674 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa80807b190 con 0x7fa8180712d0 2024-03-06T07:25:45.226 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:25:45.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 msgr2=0x7fa80406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa80c006f00 tx=0x7fa80c006e10 comp rx=0 tx=0).stop 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 msgr2=0x7fa8181f0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 0x7fa8181f0010 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa80802b8c0 tx=0x7fa8080046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 shutdown_connections 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815de60 0x7fa8181471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815a220 0x7fa818159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 --2- 172.21.15.122:0/2118426674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180712d0 0x7fa8181f0010 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 >> 172.21.15.122:0/2118426674 conn(0x7fa81806b190 msgr2=0x7fa8180cee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 shutdown_connections 2024-03-06T07:25:45.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.226+0000 7fa81e85c640 1 -- 172.21.15.122:0/2118426674 wait complete. 2024-03-06T07:25:45.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:25:45.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.338+0000 7fd2a8c4e640 1 Processor -- start 2024-03-06T07:25:45.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.338+0000 7fd2a8c4e640 1 -- start start 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a8c4e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2a415ee80 0x7fd2a40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a8c4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a40b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a8c4e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a4070d70 con 0x7fd2a41443d0 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a8c4e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40b5060 con 0x7fd2a415ee80 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a8c4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40ce610 con 0x7fd2a4070930 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2a415ee80 0x7fd2a40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2a415ee80 0x7fd2a40b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50868/0 (socket says 172.21.15.122:50868) 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/1710226663 learned_addr learned my addr 172.21.15.122:0/1710226663 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/1710226663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 msgr2=0x7fd2a40b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a2d9d640 1 --2- 172.21.15.122:0/1710226663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a40b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 --2- 172.21.15.122:0/1710226663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a40b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/1710226663 >> v1:172.21.15.169:6789/0 conn(0x7fd2a41443d0 legacy=0x7fd2a415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.339+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/1710226663 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40b3bd0 con 0x7fd2a415ee80 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a1d9b640 1 --2- 172.21.15.122:0/1710226663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a415ee80 0x7fd2a40b34d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd294009870 tx=0x7fd29402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece6893372dcc886 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a159a640 1 -- 172.21.15.122:0/1710226663 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294035020 con 0x7fd2a415ee80 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a159a640 1 -- 172.21.15.122:0/1710226663 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd294031640 con 0x7fd2a415ee80 2024-03-06T07:25:45.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a159a640 1 -- 172.21.15.122:0/1710226663 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29402d040 con 0x7fd2a415ee80 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/1710226663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a415ee80 msgr2=0x7fd2a40b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/1710226663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a415ee80 0x7fd2a40b34d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd294009870 tx=0x7fd29402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/1710226663 shutdown_connections 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/1710226663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a40b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.340+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/1710226663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a415ee80 0x7fd2a40b34d0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.341+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/1710226663 >> 172.21.15.122:0/1710226663 conn(0x7fd2a406b3d0 msgr2=0x7fd2a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.341+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/1710226663 shutdown_connections 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.341+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/1710226663 wait complete. 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.342+0000 7fd2a8c4e640 1 Processor -- start 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.342+0000 7fd2a8c4e640 1 -- start start 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.342+0000 7fd2a8c4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.342+0000 7fd2a8c4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 0x7fd2a413bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.342+0000 7fd2a8c4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2a415ee80 0x7fd2a4159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a8c4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41447b0 con 0x7fd2a41443d0 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a8c4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40ce2a0 con 0x7fd2a415ee80 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a8c4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40cdee0 con 0x7fd2a4070930 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 0x7fd2a413bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 0x7fd2a413bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50880/0 (socket says 172.21.15.122:50880) 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/930750040 learned_addr learned my addr 172.21.15.122:0/930750040 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a2d9d640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2a415ee80 0x7fd2a4159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a259c640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 msgr2=0x7fd2a41543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a41543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2a415ee80 msgr2=0x7fd2a4159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2a415ee80 0x7fd2a4159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 -- 172.21.15.122:0/930750040 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a413c7a0 con 0x7fd2a41443d0 2024-03-06T07:25:45.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a1d9b640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 0x7fd2a413bd00 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd29402b930 tx=0x7fd2940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294040070 con 0x7fd2a41443d0 2024-03-06T07:25:45.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd294039c10 con 0x7fd2a41443d0 2024-03-06T07:25:45.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a413ca80 con 0x7fd2a41443d0 2024-03-06T07:25:45.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294035040 con 0x7fd2a41443d0 2024-03-06T07:25:45.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.343+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a413d040 con 0x7fd2a41443d0 2024-03-06T07:25:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.344+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a413d520 con 0x7fd2a41443d0 2024-03-06T07:25:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.345+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd29404b060 con 0x7fd2a41443d0 2024-03-06T07:25:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.345+0000 7fd292ffd640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd28406b4d0 0x7fd28406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.345+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd2940ab980 con 0x7fd2a41443d0 2024-03-06T07:25:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.345+0000 7fd2a259c640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd28406b4d0 0x7fd28406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.346+0000 7fd2a259c640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd28406b4d0 0x7fd28406d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd298004ea0 tx=0x7fd298009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.349+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2a413d520 con 0x7fd2a41443d0 2024-03-06T07:25:45.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.588+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7fd2a4072650 con 0x7fd2a41443d0 2024-03-06T07:25:45.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.628+0000 7fd292ffd640 1 -- 172.21.15.122:0/930750040 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v222) v1 ==== 135+0+0 (secure 0 0 0) 0x7fd29402f300 con 0x7fd2a41443d0 2024-03-06T07:25:45.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd28406b4d0 msgr2=0x7fd28406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd28406b4d0 0x7fd28406d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd298004ea0 tx=0x7fd298009040 comp rx=0 tx=0).stop 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 msgr2=0x7fd2a413bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 0x7fd2a413bd00 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd29402b930 tx=0x7fd2940046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 shutdown_connections 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd28406b4d0 0x7fd28406d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2a415ee80 0x7fd2a4159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2a41443d0 0x7fd2a413bd00 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 --2- 172.21.15.122:0/930750040 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2a4070930 0x7fd2a41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 >> 172.21.15.122:0/930750040 conn(0x7fd2a406b3d0 msgr2=0x7fd2a406d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 shutdown_connections 2024-03-06T07:25:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.630+0000 7fd2a8c4e640 1 -- 172.21.15.122:0/930750040 wait complete. 2024-03-06T07:25:45.643 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:45.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.744+0000 7f5cd8c02640 1 Processor -- start 2024-03-06T07:25:45.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.744+0000 7f5cd8c02640 1 -- start start 2024-03-06T07:25:45.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd8c02640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5cd415ee10 0x7f5cd4168d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd8c02640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd40cea90 con 0x7f5cd406b6c0 2024-03-06T07:25:45.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd8c02640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd4161bc0 con 0x7f5cd415bbb0 2024-03-06T07:25:45.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd8c02640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd4161970 con 0x7f5cd415ee10 2024-03-06T07:25:45.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd1d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5cd406b6c0 0x7f5cd415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47516/0 (socket says 172.21.15.122:47516) 2024-03-06T07:25:45.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd1d9b640 1 -- 172.21.15.122:0/1061651011 learned_addr learned my addr 172.21.15.122:0/1061651011 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:45.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.745+0000 7f5cd2d9d640 1 --2- 172.21.15.122:0/1061651011 >> v2:172.21.15.122:3300/0 conn(0x7f5cd415ee10 0x7f5cd4168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd2d9d640 1 -- 172.21.15.122:0/1061651011 >> v1:172.21.15.169:6789/0 conn(0x7f5cd415bbb0 legacy=0x7f5cd406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd2d9d640 1 -- 172.21.15.122:0/1061651011 >> v1:172.21.15.122:6789/0 conn(0x7f5cd406b6c0 legacy=0x7f5cd415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd2d9d640 1 -- 172.21.15.122:0/1061651011 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4161ea0 con 0x7f5cd415ee10 2024-03-06T07:25:45.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd2d9d640 1 --2- 172.21.15.122:0/1061651011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd415ee10 0x7f5cd4168d00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0097a0 tx=0x7f5ccc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af94ddcdee8b48d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd159a640 1 -- 172.21.15.122:0/1061651011 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc034070 con 0x7f5cd415ee10 2024-03-06T07:25:45.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd159a640 1 -- 172.21.15.122:0/1061651011 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc030610 con 0x7f5cd415ee10 2024-03-06T07:25:45.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.746+0000 7f5cd159a640 1 -- 172.21.15.122:0/1061651011 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0309e0 con 0x7f5cd415ee10 2024-03-06T07:25:45.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 -- 172.21.15.122:0/1061651011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd415ee10 msgr2=0x7f5cd4168d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/1061651011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd415ee10 0x7f5cd4168d00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0097a0 tx=0x7f5ccc02b960 comp rx=0 tx=0).stop 2024-03-06T07:25:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 -- 172.21.15.122:0/1061651011 shutdown_connections 2024-03-06T07:25:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/1061651011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd415ee10 0x7f5cd4168d00 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 -- 172.21.15.122:0/1061651011 >> 172.21.15.122:0/1061651011 conn(0x7f5cd40b2660 msgr2=0x7f5cd40b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 -- 172.21.15.122:0/1061651011 shutdown_connections 2024-03-06T07:25:45.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.747+0000 7f5cd8c02640 1 -- 172.21.15.122:0/1061651011 wait complete. 2024-03-06T07:25:45.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.748+0000 7f5cd8c02640 1 Processor -- start 2024-03-06T07:25:45.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.748+0000 7f5cd8c02640 1 -- start start 2024-03-06T07:25:45.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd8c02640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 0x7f5cd414ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd8c02640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cd415ee10 0x7f5cd41f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd8c02640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cd41f8120 0x7f5cd415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd8c02640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd4072cb0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd8c02640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd41624f0 con 0x7f5cd415ee10 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd8c02640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd4169240 con 0x7f5cd41f8120 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 0x7f5cd414ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 0x7f5cd414ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50892/0 (socket says 172.21.15.122:50892) 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 -- 172.21.15.122:0/864294695 learned_addr learned my addr 172.21.15.122:0/864294695 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 -- 172.21.15.122:0/864294695 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cd41f8120 msgr2=0x7f5cd415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cd41f8120 0x7f5cd415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 -- 172.21.15.122:0/864294695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cd415ee10 msgr2=0x7f5cd41f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cd415ee10 0x7f5cd41f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 -- 172.21.15.122:0/864294695 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd41f85a0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.749+0000 7f5cd259c640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 0x7f5cd414ade0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400c7c0 tx=0x7f5cc400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.750+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4016020 con 0x7f5cd406b6c0 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.750+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc4006d50 con 0x7f5cd406b6c0 2024-03-06T07:25:45.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.750+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd41f8880 con 0x7f5cd406b6c0 2024-03-06T07:25:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.750+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4003bb0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.750+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd414dde0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.751+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd40733b0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.751+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cc400a850 con 0x7f5cd406b6c0 2024-03-06T07:25:45.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.751+0000 7f5cc2ffd640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb406b500 0x7f5cb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.752+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5cc4089bd0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.752+0000 7f5cd1d9b640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb406b500 0x7f5cb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.752+0000 7f5cd1d9b640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb406b500 0x7f5cb406d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40b63e0 tx=0x7f5cc8007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.756+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cc4020020 con 0x7f5cd406b6c0 2024-03-06T07:25:45.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.929+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5cd4068ff0 con 0x7f5cd406b6c0 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.932+0000 7f5cc2ffd640 1 -- 172.21.15.122:0/864294695 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5cc4019310 con 0x7f5cd406b6c0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb406b500 msgr2=0x7f5cb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb406b500 0x7f5cb406d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40b63e0 tx=0x7f5cc8007a40 comp rx=0 tx=0).stop 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 msgr2=0x7f5cd414ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 0x7f5cd414ade0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400c7c0 tx=0x7f5cc400cc80 comp rx=0 tx=0).stop 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 shutdown_connections 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5cb406b500 0x7f5cb406d9c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cd41f8120 0x7f5cd415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cd415ee10 0x7f5cd41f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 --2- 172.21.15.122:0/864294695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cd406b6c0 0x7f5cd414ade0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 >> 172.21.15.122:0/864294695 conn(0x7f5cd40b2660 msgr2=0x7f5cd40cd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 shutdown_connections 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:45.934+0000 7f5cd8c02640 1 -- 172.21.15.122:0/864294695 wait complete. 2024-03-06T07:25:45.947 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:45.947 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:45.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:46.039 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:46.039 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:46.039 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:46.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:46.055 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:25:46.055 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:46.072 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:46.072 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:25:46.072 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:46.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:46.110 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:25:46.110 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:46.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:46.130 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:46.130 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:46.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.247+0000 7f5d99972640 1 Processor -- start 2024-03-06T07:25:46.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.248+0000 7f5d99972640 1 -- start start 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.248+0000 7f5d99972640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d940ce920 0x7f5d9415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.248+0000 7f5d99972640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d9406b600 0x7f5d94162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.248+0000 7f5d99972640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9415ea30 con 0x7f5d9406b220 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.248+0000 7f5d99972640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d94161b70 con 0x7f5d9406b600 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.248+0000 7f5d99972640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9415ed10 con 0x7f5d940ce920 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d9406b600 0x7f5d94162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d9406b600 0x7f5d94162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50896/0 (socket says 172.21.15.122:50896) 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 -- 172.21.15.122:0/2746818981 learned_addr learned my addr 172.21.15.122:0/2746818981 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:46.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d927fc640 1 --2- 172.21.15.122:0/2746818981 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d940ce920 0x7f5d9415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 -- 172.21.15.122:0/2746818981 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d940ce920 msgr2=0x7f5d9415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 --2- 172.21.15.122:0/2746818981 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d940ce920 0x7f5d9415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 -- 172.21.15.122:0/2746818981 >> v1:172.21.15.169:6789/0 conn(0x7f5d9406b220 legacy=0x7f5d94071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d937fe640 1 -- 172.21.15.122:0/2746818981 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9415eff0 con 0x7f5d9406b600 2024-03-06T07:25:46.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.249+0000 7f5d91ffb640 1 -- 172.21.15.122:0/2746818981 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643874279 0 0) 0x7f5d9415ea30 con 0x7f5d9406b220 2024-03-06T07:25:46.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d937fe640 1 --2- 172.21.15.122:0/2746818981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d9406b600 0x7f5d94162000 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009e50 tx=0x7f5d8c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbffe37e0964ae93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d91ffb640 1 -- 172.21.15.122:0/2746818981 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c02cb00 con 0x7f5d9406b600 2024-03-06T07:25:46.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d91ffb640 1 -- 172.21.15.122:0/2746818981 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c02cca0 con 0x7f5d9406b600 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d91ffb640 1 -- 172.21.15.122:0/2746818981 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c034610 con 0x7f5d9406b600 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 -- 172.21.15.122:0/2746818981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d9406b600 msgr2=0x7f5d94162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 --2- 172.21.15.122:0/2746818981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d9406b600 0x7f5d94162000 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009e50 tx=0x7f5d8c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 -- 172.21.15.122:0/2746818981 shutdown_connections 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 --2- 172.21.15.122:0/2746818981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d9406b600 0x7f5d94162000 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 --2- 172.21.15.122:0/2746818981 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d940ce920 0x7f5d9415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 -- 172.21.15.122:0/2746818981 >> 172.21.15.122:0/2746818981 conn(0x7f5d9406c0c0 msgr2=0x7f5d9406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 -- 172.21.15.122:0/2746818981 shutdown_connections 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.250+0000 7f5d99972640 1 -- 172.21.15.122:0/2746818981 wait complete. 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.251+0000 7f5d99972640 1 Processor -- start 2024-03-06T07:25:46.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 -- start start 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d9406b220 0x7f5d9415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d9406b600 0x7f5d941e6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 0x7f5d941e7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94072090 con 0x7f5d940ce920 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ce200 con 0x7f5d9406b220 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d99972640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d941f5cb0 con 0x7f5d9406b600 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 0x7f5d941e7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 0x7f5d941e7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50906/0 (socket says 172.21.15.122:50906) 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.252+0000 7f5d937fe640 1 -- 172.21.15.122:0/683415476 learned_addr learned my addr 172.21.15.122:0/683415476 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d92ffd640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d9406b220 0x7f5d9415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d927fc640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d9406b600 0x7f5d941e6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d937fe640 1 -- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d9406b600 msgr2=0x7f5d941e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d937fe640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d9406b600 0x7f5d941e6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d937fe640 1 -- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d9406b220 msgr2=0x7f5d9415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d937fe640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d9406b220 0x7f5d9415a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d937fe640 1 -- 172.21.15.122:0/683415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d941f5f90 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d937fe640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 0x7f5d941e7250 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c036040 tx=0x7f5d8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c043030 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c02ce50 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d941f6270 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c009510 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.253+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d941f6830 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.254+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d9404f330 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.255+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d8c046070 con 0x7f5d940ce920 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.255+0000 7f5d7bfff640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d8006b500 0x7f5d8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.255+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5d8c0ac000 con 0x7f5d940ce920 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.255+0000 7f5d92ffd640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d8006b500 0x7f5d8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.255+0000 7f5d92ffd640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d8006b500 0x7f5d8006d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5d84007fd0 tx=0x7f5d84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.259+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d8c04d070 con 0x7f5d940ce920 2024-03-06T07:25:46.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.468+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d940cdf00 con 0x7f5d940ce920 2024-03-06T07:25:46.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.468+0000 7f5d7bfff640 1 -- 172.21.15.122:0/683415476 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f5d8c0333a0 con 0x7f5d940ce920 2024-03-06T07:25:46.470 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:46.470 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2750860737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2750860737},{"type":"v1","addr":"172.21.15.169:6835","nonce":2750860737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/369818189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":369818189},{"type":"v1","addr":"172.21.15.169:6834","nonce":369818189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6597,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1707640048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1707640048},{"type":"v1","addr":"172.21.15.122:6836","nonce":1707640048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6957,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2047265686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2047265686},{"type":"v1","addr":"172.21.15.122:6837","nonce":2047265686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2024-03-06T07:25:46.471 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 35 2024-03-06T07:25:46.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d8006b500 msgr2=0x7f5d8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d8006b500 0x7f5d8006d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5d84007fd0 tx=0x7f5d84009040 comp rx=0 tx=0).stop 2024-03-06T07:25:46.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 msgr2=0x7f5d941e7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 0x7f5d941e7250 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c036040 tx=0x7f5d8c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 shutdown_connections 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5d8006b500 0x7f5d8006d9c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d940ce920 0x7f5d941e7250 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5d9406b600 0x7f5d941e6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 --2- 172.21.15.122:0/683415476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d9406b220 0x7f5d9415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 >> 172.21.15.122:0/683415476 conn(0x7f5d9406c0c0 msgr2=0x7f5d9415c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 shutdown_connections 2024-03-06T07:25:46.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.470+0000 7f5d99972640 1 -- 172.21.15.122:0/683415476 wait complete. 2024-03-06T07:25:46.484 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:46.484 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:46.484 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:46.484 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:46.484 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:46.484 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:46.485 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:46.485 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:46.485 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:46.485 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:46.485 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:46.485 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:46.485 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:46.486 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:46.486 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:46.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:46.494 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:46.494 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:46.494 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:46.495 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:46.498 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:46.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:46.499 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:46.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:46.536 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:46.536 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:46.537 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:46.537 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:46.541 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:46.541 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:46.541 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:46.542 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:46.542 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:46.613 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:25:46.613 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:25:46.625 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:25:46.625 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:25:46.626 INFO:tasks.ceph.mds.b.smithi169.stderr:2024-03-06T07:25:46.624+0000 7fca3ad09640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:25:46.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.648+0000 7f2919fac640 1 Processor -- start 2024-03-06T07:25:46.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.648+0000 7f2919fac640 1 -- start start 2024-03-06T07:25:46.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.648+0000 7f2919fac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29140b4460 0x7f29140d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.648+0000 7f2919fac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2914162970 0x7f29140b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2919fac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140d22c0 con 0x7f2914161cf0 2024-03-06T07:25:46.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2919fac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2914161b00 con 0x7f2914162970 2024-03-06T07:25:46.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2919fac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29140cecc0 con 0x7f29140b4460 2024-03-06T07:25:46.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2912ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2914162970 0x7f29140b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2913fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2914161cf0 0x7f29140cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47536/0 (socket says 172.21.15.122:47536) 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2913fff640 1 -- 172.21.15.122:0/2369102545 learned_addr learned my addr 172.21.15.122:0/2369102545 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f29137fe640 1 --2- 172.21.15.122:0/2369102545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29140b4460 0x7f29140d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2912ffd640 1 -- 172.21.15.122:0/2369102545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29140b4460 msgr2=0x7f29140d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2912ffd640 1 --2- 172.21.15.122:0/2369102545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29140b4460 0x7f29140d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2912ffd640 1 -- 172.21.15.122:0/2369102545 >> v1:172.21.15.122:6789/0 conn(0x7f2914161cf0 legacy=0x7f29140cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f29127fc640 1 -- 172.21.15.122:0/2369102545 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079464805 0 0) 0x7f29140d22c0 con 0x7f2914161cf0 2024-03-06T07:25:46.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.649+0000 7f2912ffd640 1 -- 172.21.15.122:0/2369102545 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140d25a0 con 0x7f2914162970 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2912ffd640 1 --2- 172.21.15.122:0/2369102545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914162970 0x7f29140b3580 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f29080097a0 tx=0x7f290802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffff9ad9cc2429be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f29127fc640 1 -- 172.21.15.122:0/2369102545 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908035020 con 0x7f2914162970 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f29127fc640 1 -- 172.21.15.122:0/2369102545 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2908031830 con 0x7f2914162970 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f29127fc640 1 -- 172.21.15.122:0/2369102545 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908031c00 con 0x7f2914162970 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2919fac640 1 -- 172.21.15.122:0/2369102545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914162970 msgr2=0x7f29140b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2919fac640 1 --2- 172.21.15.122:0/2369102545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914162970 0x7f29140b3580 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f29080097a0 tx=0x7f290802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2919fac640 1 -- 172.21.15.122:0/2369102545 shutdown_connections 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2919fac640 1 --2- 172.21.15.122:0/2369102545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914162970 0x7f29140b3580 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2919fac640 1 --2- 172.21.15.122:0/2369102545 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29140b4460 0x7f29140d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.650+0000 7f2919fac640 1 -- 172.21.15.122:0/2369102545 >> 172.21.15.122:0/2369102545 conn(0x7f29140d3ce0 msgr2=0x7f29140d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.651+0000 7f2919fac640 1 -- 172.21.15.122:0/2369102545 shutdown_connections 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.651+0000 7f2919fac640 1 -- 172.21.15.122:0/2369102545 wait complete. 2024-03-06T07:25:46.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.652+0000 7f2919fac640 1 Processor -- start 2024-03-06T07:25:46.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.652+0000 7f2919fac640 1 -- start start 2024-03-06T07:25:46.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29140b4460 0x7f2914148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 0x7f29140c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2914162970 0x7f29140b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 0x7f29140c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 0x7f29140c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50956/0 (socket says 172.21.15.122:50956) 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 -- 172.21.15.122:0/1729533771 learned_addr learned my addr 172.21.15.122:0/1729533771 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140ce950 con 0x7f2914161cf0 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140d2b60 con 0x7f29140b4460 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29140b3ac0 con 0x7f2914162970 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f29137fe640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29140b4460 0x7f2914148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2913fff640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2914162970 0x7f29140b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 -- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2914162970 msgr2=0x7f29140b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2914162970 0x7f29140b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 -- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29140b4460 msgr2=0x7f2914148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29140b4460 0x7f2914148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140caee0 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2912ffd640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 0x7f29140c8c80 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2908009890 tx=0x7f29080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908041070 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29140cb1c0 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290803b920 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.653+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908035040 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.654+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29140cb780 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.655+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291404f330 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.655+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f290804c060 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.655+0000 7f28fbfff640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f290006b500 0x7f290006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.655+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f29080ab730 con 0x7f2914161cf0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.655+0000 7f29137fe640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f290006b500 0x7f290006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.656+0000 7f29137fe640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f290006b500 0x7f290006d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2904009aa0 tx=0x7f2904009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.660+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f290807a1e0 con 0x7f2914161cf0 2024-03-06T07:25:46.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:46.846+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f291406b180 con 0x7f2914161cf0 2024-03-06T07:25:47.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.636+0000 7f28fbfff640 1 -- 172.21.15.122:0/1729533771 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7f290807e280 con 0x7f2914161cf0 2024-03-06T07:25:47.638 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:25:47.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f290006b500 msgr2=0x7f290006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f290006b500 0x7f290006d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2904009aa0 tx=0x7f2904009190 comp rx=0 tx=0).stop 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 msgr2=0x7f29140c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 0x7f29140c8c80 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2908009890 tx=0x7f29080046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 shutdown_connections 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f290006b500 0x7f290006d9c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2914162970 0x7f29140b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2914161cf0 0x7f29140c8c80 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 --2- 172.21.15.122:0/1729533771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29140b4460 0x7f2914148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 >> 172.21.15.122:0/1729533771 conn(0x7f29140d3ce0 msgr2=0x7f29140cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.639+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 shutdown_connections 2024-03-06T07:25:47.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.640+0000 7f2919fac640 1 -- 172.21.15.122:0/1729533771 wait complete. 2024-03-06T07:25:47.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:47.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.754+0000 7f36b9adf640 1 Processor -- start 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.754+0000 7f36b9adf640 1 -- start start 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36b9adf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f36b40d22b0 0x7f36b40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36b9adf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b3430 con 0x7f36b40b41f0 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36b9adf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b3710 con 0x7f36b415dfe0 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36b9adf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b40d2c50 con 0x7f36b40d22b0 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36aaffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f36b40d22b0 0x7f36b40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36b3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f36b40b41f0 0x7f36b40b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47540/0 (socket says 172.21.15.122:47540) 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.755+0000 7f36b3fff640 1 -- 172.21.15.122:0/292058176 learned_addr learned my addr 172.21.15.122:0/292058176 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.756+0000 7f36aaffd640 1 -- 172.21.15.122:0/292058176 >> v1:172.21.15.169:6789/0 conn(0x7f36b415dfe0 legacy=0x7f36b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.756+0000 7f36aaffd640 1 -- 172.21.15.122:0/292058176 >> v1:172.21.15.122:6789/0 conn(0x7f36b40b41f0 legacy=0x7f36b40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.756+0000 7f36aaffd640 1 -- 172.21.15.122:0/292058176 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40b39f0 con 0x7f36b40d22b0 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.756+0000 7f36b2ffd640 1 -- 172.21.15.122:0/292058176 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719990837 0 0) 0x7f36b40b3430 con 0x7f36b40b41f0 2024-03-06T07:25:47.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.756+0000 7f36aaffd640 1 --2- 172.21.15.122:0/292058176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40d22b0 0x7f36b40d2710 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f369c0097d0 tx=0x7f369c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af982ddc217796e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b2ffd640 1 -- 172.21.15.122:0/292058176 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c035020 con 0x7f36b40d22b0 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b2ffd640 1 -- 172.21.15.122:0/292058176 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369c031830 con 0x7f36b40d22b0 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b2ffd640 1 -- 172.21.15.122:0/292058176 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c031c00 con 0x7f36b40d22b0 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 -- 172.21.15.122:0/292058176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40d22b0 msgr2=0x7f36b40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 --2- 172.21.15.122:0/292058176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40d22b0 0x7f36b40d2710 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f369c0097d0 tx=0x7f369c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 -- 172.21.15.122:0/292058176 shutdown_connections 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 --2- 172.21.15.122:0/292058176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40d22b0 0x7f36b40d2710 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 -- 172.21.15.122:0/292058176 >> 172.21.15.122:0/292058176 conn(0x7f36b40d3cd0 msgr2=0x7f36b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 -- 172.21.15.122:0/292058176 shutdown_connections 2024-03-06T07:25:47.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.757+0000 7f36b9adf640 1 -- 172.21.15.122:0/292058176 wait complete. 2024-03-06T07:25:47.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.758+0000 7f36b9adf640 1 Processor -- start 2024-03-06T07:25:47.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.758+0000 7f36b9adf640 1 -- start start 2024-03-06T07:25:47.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b9adf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 0x7f36b41610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b9adf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f36b40d22b0 0x7f36b40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b9adf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b415dfe0 0x7f36b4149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b9adf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40b3d60 con 0x7f36b40b41f0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b9adf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40b31e0 con 0x7f36b415dfe0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b9adf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40d1c70 con 0x7f36b40d22b0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 0x7f36b41610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 0x7f36b41610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50970/0 (socket says 172.21.15.122:50970) 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b37fe640 1 -- 172.21.15.122:0/3479182293 learned_addr learned my addr 172.21.15.122:0/3479182293 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b37fe640 1 -- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f36b40d22b0 msgr2=0x7f36b40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b37fe640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f36b40d22b0 0x7f36b40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.759+0000 7f36b3fff640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b415dfe0 0x7f36b4149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b37fe640 1 -- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b415dfe0 msgr2=0x7f36b4149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b37fe640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b415dfe0 0x7f36b4149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b37fe640 1 -- 172.21.15.122:0/3479182293 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40b9070 con 0x7f36b40b41f0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b37fe640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 0x7f36b41610c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f36a400b3f0 tx=0x7f36a400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4015020 con 0x7f36b40b41f0 2024-03-06T07:25:47.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a4011830 con 0x7f36b40b41f0 2024-03-06T07:25:47.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b40b9350 con 0x7f36b40b41f0 2024-03-06T07:25:47.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4011c20 con 0x7f36b40b41f0 2024-03-06T07:25:47.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.760+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b40b9910 con 0x7f36b40b41f0 2024-03-06T07:25:47.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.761+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b404f330 con 0x7f36b40b41f0 2024-03-06T07:25:47.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.761+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36a4018020 con 0x7f36b40b41f0 2024-03-06T07:25:47.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.762+0000 7f36b0ff9640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f369406b4d0 0x7f369406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.762+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f36a4089b00 con 0x7f36b40b41f0 2024-03-06T07:25:47.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.762+0000 7f36aaffd640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f369406b4d0 0x7f369406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.762+0000 7f36aaffd640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f369406b4d0 0x7f369406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f369c036000 tx=0x7f369c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:47.766+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36a401f030 con 0x7f36b40b41f0 2024-03-06T07:25:48.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.000+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f36b415d4c0 con 0x7f36b40b41f0 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.001+0000 7f36b0ff9640 1 -- 172.21.15.122:0/3479182293 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f36a40172c0 con 0x7f36b40b41f0 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.003 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:48.004 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.005 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.006 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:25:48.007 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.009 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.010 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:48.011 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:25:48.013 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:25:48.015 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:48.016 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:25:48.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.016 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:25:48.016 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:25:48.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f369406b4d0 msgr2=0x7f369406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f369406b4d0 0x7f369406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f369c036000 tx=0x7f369c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:48.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 msgr2=0x7f36b41610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 0x7f36b41610c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f36a400b3f0 tx=0x7f36a400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 shutdown_connections 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f369406b4d0 0x7f369406d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b415dfe0 0x7f36b4149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f36b40d22b0 0x7f36b40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 --2- 172.21.15.122:0/3479182293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36b40b41f0 0x7f36b41610c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 >> 172.21.15.122:0/3479182293 conn(0x7f36b40d3cd0 msgr2=0x7f36b40b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 shutdown_connections 2024-03-06T07:25:48.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.003+0000 7f36b9adf640 1 -- 172.21.15.122:0/3479182293 wait complete. 2024-03-06T07:25:48.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:48.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.117+0000 7f7dd97f6640 1 Processor -- start 2024-03-06T07:25:48.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.117+0000 7f7dd97f6640 1 -- start start 2024-03-06T07:25:48.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd97f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd40d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd97f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7dd40d0650 0x7f7dd40d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd97f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd415e470 con 0x7f7dd40d2920 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd97f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd40d18f0 con 0x7f7dd40d0650 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd97f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40b3cf0 con 0x7f7dd415e090 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7dd40d0650 0x7f7dd40d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7dd40d0650 0x7f7dd40d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50986/0 (socket says 172.21.15.122:50986) 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 -- 172.21.15.122:0/2244871215 learned_addr learned my addr 172.21.15.122:0/2244871215 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd2ffd640 1 --2- 172.21.15.122:0/2244871215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd40d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 -- 172.21.15.122:0/2244871215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 msgr2=0x7f7dd40d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 --2- 172.21.15.122:0/2244871215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd40d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 -- 172.21.15.122:0/2244871215 >> v1:172.21.15.169:6789/0 conn(0x7f7dd40d2920 legacy=0x7f7dd40b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.118+0000 7f7dd37fe640 1 -- 172.21.15.122:0/2244871215 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd40d1b90 con 0x7f7dd40d0650 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd37fe640 1 --2- 172.21.15.122:0/2244871215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d0650 0x7f7dd40d0ae0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009e50 tx=0x7f7dcc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dd97ab6aab4644 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd1ffb640 1 -- 172.21.15.122:0/2244871215 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02ca70 con 0x7f7dd40d0650 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd1ffb640 1 -- 172.21.15.122:0/2244871215 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc02cc10 con 0x7f7dd40d0650 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd1ffb640 1 -- 172.21.15.122:0/2244871215 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc0343f0 con 0x7f7dd40d0650 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 -- 172.21.15.122:0/2244871215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d0650 msgr2=0x7f7dd40d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/2244871215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d0650 0x7f7dd40d0ae0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009e50 tx=0x7f7dcc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 -- 172.21.15.122:0/2244871215 shutdown_connections 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/2244871215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d0650 0x7f7dd40d0ae0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/2244871215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd40d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 -- 172.21.15.122:0/2244871215 >> 172.21.15.122:0/2244871215 conn(0x7f7dd40d3c80 msgr2=0x7f7dd40d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 -- 172.21.15.122:0/2244871215 shutdown_connections 2024-03-06T07:25:48.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.119+0000 7f7dd97f6640 1 -- 172.21.15.122:0/2244871215 wait complete. 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 Processor -- start 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 -- start start 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dd40d0650 0x7f7dd40cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 0x7f7dd414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd4160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40b2df0 con 0x7f7dd40d2920 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40cee30 con 0x7f7dd40d0650 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.121+0000 7f7dd97f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40ce800 con 0x7f7dd415e090 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 0x7f7dd414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 0x7f7dd414bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50996/0 (socket says 172.21.15.122:50996) 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 -- 172.21.15.122:0/3934789445 learned_addr learned my addr 172.21.15.122:0/3934789445 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:48.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd2ffd640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dd40d0650 0x7f7dd40cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd37fe640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd4160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 -- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 msgr2=0x7f7dd4160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd4160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 -- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dd40d0650 msgr2=0x7f7dd40cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dd40d0650 0x7f7dd40cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 -- 172.21.15.122:0/3934789445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd415fc40 con 0x7f7dd40d2920 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd27fc640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 0x7f7dd414bf70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400b3f0 tx=0x7f7dc400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4015020 con 0x7f7dd40d2920 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc4011830 con 0x7f7dd40d2920 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd415ff20 con 0x7f7dd40d2920 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4011c20 con 0x7f7dd40d2920 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.122+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd41604e0 con 0x7f7dd40d2920 2024-03-06T07:25:48.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.123+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd40b33f0 con 0x7f7dd40d2920 2024-03-06T07:25:48.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.124+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dc4018020 con 0x7f7dd40d2920 2024-03-06T07:25:48.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.124+0000 7f7dbb7fe640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dc006b4d0 0x7f7dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.124+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7dc40899a0 con 0x7f7dd40d2920 2024-03-06T07:25:48.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.124+0000 7f7dd2ffd640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dc006b4d0 0x7f7dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.124+0000 7f7dd2ffd640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dc006b4d0 0x7f7dc006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8006fd0 tx=0x7f7dc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.128+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dc401f030 con 0x7f7dd40d2920 2024-03-06T07:25:48.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.299+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd404f330 con 0x7f7dd40d2920 2024-03-06T07:25:48.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.300+0000 7f7dbb7fe640 1 -- 172.21.15.122:0/3934789445 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12892 (secure 0 0 0) 0x7f7dc4058520 con 0x7f7dd40d2920 2024-03-06T07:25:48.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:48.302 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":78,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:47.634688+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dc006b4d0 msgr2=0x7f7dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dc006b4d0 0x7f7dc006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8006fd0 tx=0x7f7dc8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 msgr2=0x7f7dd414bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 0x7f7dd414bf70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400b3f0 tx=0x7f7dc400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 shutdown_connections 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7dc006b4d0 0x7f7dc006d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7dd415e090 0x7f7dd4160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7dd40d2920 0x7f7dd414bf70 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 --2- 172.21.15.122:0/3934789445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7dd40d0650 0x7f7dd40cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.302+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 >> 172.21.15.122:0/3934789445 conn(0x7f7dd40d3c80 msgr2=0x7f7dd40b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.303+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 shutdown_connections 2024-03-06T07:25:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.303+0000 7f7dd97f6640 1 -- 172.21.15.122:0/3934789445 wait complete. 2024-03-06T07:25:48.316 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:48.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:48.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.417+0000 7f9912488640 1 Processor -- start 2024-03-06T07:25:48.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.417+0000 7f9912488640 1 -- start start 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9912488640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c15a220 0x7f990c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9912488640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f990c06da00 0x7f990c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9912488640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990c0b3320 con 0x7f990c0712d0 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9912488640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990c0cf610 con 0x7f990c06da00 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9912488640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c071c70 con 0x7f990c15a220 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9910a25640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f990c06da00 0x7f990c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f990b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f990c0712d0 0x7f990c15de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47550/0 (socket says 172.21.15.122:47550) 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f990b7fe640 1 -- 172.21.15.122:0/212250513 learned_addr learned my addr 172.21.15.122:0/212250513 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f990bfff640 1 --2- 172.21.15.122:0/212250513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c15a220 0x7f990c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9910a25640 1 -- 172.21.15.122:0/212250513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c15a220 msgr2=0x7f990c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9910a25640 1 --2- 172.21.15.122:0/212250513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c15a220 0x7f990c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9910a25640 1 -- 172.21.15.122:0/212250513 >> v1:172.21.15.122:6789/0 conn(0x7f990c0712d0 legacy=0x7f990c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f990affd640 1 -- 172.21.15.122:0/212250513 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032871722 0 0) 0x7f990c0b3320 con 0x7f990c0712d0 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.418+0000 7f9910a25640 1 -- 172.21.15.122:0/212250513 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c0728d0 con 0x7f990c06da00 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.419+0000 7f9910a25640 1 --2- 172.21.15.122:0/212250513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c06da00 0x7f990c071730 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f9904009d90 tx=0x7f990402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d7f1a0f114f970 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.419+0000 7f990affd640 1 -- 172.21.15.122:0/212250513 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904039070 con 0x7f990c06da00 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.419+0000 7f990affd640 1 -- 172.21.15.122:0/212250513 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990402cbc0 con 0x7f990c06da00 2024-03-06T07:25:48.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.419+0000 7f990affd640 1 -- 172.21.15.122:0/212250513 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904034440 con 0x7f990c06da00 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 -- 172.21.15.122:0/212250513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c06da00 msgr2=0x7f990c071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 --2- 172.21.15.122:0/212250513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c06da00 0x7f990c071730 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f9904009d90 tx=0x7f990402be10 comp rx=0 tx=0).stop 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 -- 172.21.15.122:0/212250513 shutdown_connections 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 --2- 172.21.15.122:0/212250513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c06da00 0x7f990c071730 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 --2- 172.21.15.122:0/212250513 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c15a220 0x7f990c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 -- 172.21.15.122:0/212250513 >> 172.21.15.122:0/212250513 conn(0x7f990c06b190 msgr2=0x7f990c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 -- 172.21.15.122:0/212250513 shutdown_connections 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.420+0000 7f9912488640 1 -- 172.21.15.122:0/212250513 wait complete. 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.421+0000 7f9912488640 1 Processor -- start 2024-03-06T07:25:48.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.421+0000 7f9912488640 1 -- start start 2024-03-06T07:25:48.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c06da00 0x7f990c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f990c0712d0 0x7f990c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 0x7f990c1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c072030 con 0x7f990c15a220 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c072bb0 con 0x7f990c0712d0 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c072530 con 0x7f990c06da00 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 0x7f990c1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 0x7f990c1524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51014/0 (socket says 172.21.15.122:51014) 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 -- 172.21.15.122:0/941793778 learned_addr learned my addr 172.21.15.122:0/941793778 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 -- 172.21.15.122:0/941793778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c06da00 msgr2=0x7f990c15aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c06da00 0x7f990c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 -- 172.21.15.122:0/941793778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f990c0712d0 msgr2=0x7f990c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f990c0712d0 0x7f990c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 -- 172.21.15.122:0/941793778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c1f70c0 con 0x7f990c15a220 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9910a25640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 0x7f990c1524a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f9904009eb0 tx=0x7f9904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904039050 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904045070 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c151c30 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990403e830 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.422+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c1f7360 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.423+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c0732b0 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.424+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f990403e9d0 con 0x7f990c15a220 2024-03-06T07:25:48.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.424+0000 7f9908ff9640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98f806b4d0 0x7f98f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.424+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f99040acc70 con 0x7f990c15a220 2024-03-06T07:25:48.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.424+0000 7f990bfff640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98f806b4d0 0x7f98f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.425+0000 7f990bfff640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98f806b4d0 0x7f98f806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9900006fd0 tx=0x7f9900008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.429+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f990407bff0 con 0x7f990c15a220 2024-03-06T07:25:48.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.599+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f990c070ba0 con 0x7f990c15a220 2024-03-06T07:25:48.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.600+0000 7f9908ff9640 1 -- 172.21.15.122:0/941793778 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f990407b8f0 con 0x7f990c15a220 2024-03-06T07:25:48.602 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98f806b4d0 msgr2=0x7f98f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98f806b4d0 0x7f98f806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9900006fd0 tx=0x7f9900008040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 msgr2=0x7f990c1524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 0x7f990c1524a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f9904009eb0 tx=0x7f9904004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 shutdown_connections 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98f806b4d0 0x7f98f806d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f990c15a220 0x7f990c1524a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f990c0712d0 0x7f990c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 --2- 172.21.15.122:0/941793778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f990c06da00 0x7f990c15aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 >> 172.21.15.122:0/941793778 conn(0x7f990c06b190 msgr2=0x7f990c0af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 shutdown_connections 2024-03-06T07:25:48.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.602+0000 7f9912488640 1 -- 172.21.15.122:0/941793778 wait complete. 2024-03-06T07:25:48.615 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.713+0000 7f787849d640 1 Processor -- start 2024-03-06T07:25:48.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.713+0000 7f787849d640 1 -- start start 2024-03-06T07:25:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f787849d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f78700cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f787849d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 0x7f7870071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f787849d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787015b8b0 con 0x7f78700cd7a0 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f787849d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7870070a70 con 0x7f787015b430 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f787849d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7870070d50 con 0x7f78701515e0 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f7875a38640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 0x7f7870071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f7876239640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f78700cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f7876239640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f78700cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:37536/0 (socket says 172.21.15.122:37536) 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f7875a38640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 0x7f7870071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54884/0 (socket says 172.21.15.122:54884) 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f7876239640 1 -- 172.21.15.122:0/2714398255 learned_addr learned my addr 172.21.15.122:0/2714398255 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.714+0000 7f7876a3a640 1 --1- 172.21.15.122:0/2714398255 >> v1:172.21.15.169:6789/0 conn(0x7f78700cd7a0 0x7f78700cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:34962/0 (socket says 172.21.15.122:34962) 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.715+0000 7f7876239640 1 -- 172.21.15.122:0/2714398255 >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 msgr2=0x7f7870071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.715+0000 7f7876239640 1 --2- 172.21.15.122:0/2714398255 >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 0x7f7870071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.715+0000 7f7876239640 1 -- 172.21.15.122:0/2714398255 >> v1:172.21.15.169:6789/0 conn(0x7f78700cd7a0 legacy=0x7f78700cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.715+0000 7f7876239640 1 -- 172.21.15.122:0/2714398255 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870071030 con 0x7f78701515e0 2024-03-06T07:25:48.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.715+0000 7f7875a38640 1 --2- 172.21.15.122:0/2714398255 >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 0x7f7870071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.715+0000 7f7876239640 1 --2- 172.21.15.122:0/2714398255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f78700cce40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f786c009e50 tx=0x7f786c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab72b94dc3fef5f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f7875237640 1 -- 172.21.15.122:0/2714398255 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c02cb00 con 0x7f78701515e0 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f7875237640 1 -- 172.21.15.122:0/2714398255 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c02cca0 con 0x7f78701515e0 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f7875237640 1 -- 172.21.15.122:0/2714398255 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c034610 con 0x7f78701515e0 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 -- 172.21.15.122:0/2714398255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 msgr2=0x7f78700cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 --2- 172.21.15.122:0/2714398255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f78700cce40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f786c009e50 tx=0x7f786c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 -- 172.21.15.122:0/2714398255 shutdown_connections 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 --2- 172.21.15.122:0/2714398255 >> v2:172.21.15.169:3300/0 conn(0x7f787015b430 0x7f7870071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 --2- 172.21.15.122:0/2714398255 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f78700cce40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 -- 172.21.15.122:0/2714398255 >> 172.21.15.122:0/2714398255 conn(0x7f78700cf8a0 msgr2=0x7f78700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 -- 172.21.15.122:0/2714398255 shutdown_connections 2024-03-06T07:25:48.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.716+0000 7f787849d640 1 -- 172.21.15.122:0/2714398255 wait complete. 2024-03-06T07:25:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.717+0000 7f787849d640 1 Processor -- start 2024-03-06T07:25:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.717+0000 7f787849d640 1 -- start start 2024-03-06T07:25:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f787849d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78700cd7a0 0x7f7870153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f787849d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f7870143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f787849d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 0x7f78701562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f787849d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7870071ef0 con 0x7f787015b430 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f787849d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7870071d70 con 0x7f78700cd7a0 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f787849d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78700cc3a0 con 0x7f78701515e0 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f7876a3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 0x7f78701562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f7876a3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 0x7f78701562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51024/0 (socket says 172.21.15.122:51024) 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f7876a3a640 1 -- 172.21.15.122:0/3193798625 learned_addr learned my addr 172.21.15.122:0/3193798625 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f7875a38640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f7870143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.718+0000 7f7876239640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78700cd7a0 0x7f7870153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7876a3a640 1 -- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 msgr2=0x7f7870143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7876a3a640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f7870143270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7876a3a640 1 -- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78700cd7a0 msgr2=0x7f7870153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7876a3a640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78700cd7a0 0x7f7870153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7876a3a640 1 -- 172.21.15.122:0/3193798625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870154120 con 0x7f787015b430 2024-03-06T07:25:48.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7876a3a640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 0x7f78701562b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f786800c430 tx=0x7f786800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680074e0 con 0x7f787015b430 2024-03-06T07:25:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7868007680 con 0x7f787015b430 2024-03-06T07:25:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7870155210 con 0x7f787015b430 2024-03-06T07:25:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680049e0 con 0x7f787015b430 2024-03-06T07:25:48.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.719+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78701557d0 con 0x7f787015b430 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.720+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7870068840 con 0x7f787015b430 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.720+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f786801a020 con 0x7f787015b430 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.721+0000 7f7866ffd640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f785406b4d0 0x7f785406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.721+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f78680895a0 con 0x7f787015b430 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.721+0000 7f7876239640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f785406b4d0 0x7f785406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.721+0000 7f7876239640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f785406b4d0 0x7f785406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f786c035000 tx=0x7f786c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.725+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78680580a0 con 0x7f787015b430 2024-03-06T07:25:48.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.899+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f787004f330 con 0x7f787015b430 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.900+0000 7f7866ffd640 1 -- 172.21.15.122:0/3193798625 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f786805c140 con 0x7f787015b430 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:25:48.902 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:25:48.903 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f785406b4d0 msgr2=0x7f785406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f785406b4d0 0x7f785406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f786c035000 tx=0x7f786c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 msgr2=0x7f78701562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 0x7f78701562b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f786800c430 tx=0x7f786800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 shutdown_connections 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f785406b4d0 0x7f785406d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f787015b430 0x7f78701562b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78701515e0 0x7f7870143270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 --2- 172.21.15.122:0/3193798625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78700cd7a0 0x7f7870153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 >> 172.21.15.122:0/3193798625 conn(0x7f78700cf8a0 msgr2=0x7f7870072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 shutdown_connections 2024-03-06T07:25:48.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:48.902+0000 7f787849d640 1 -- 172.21.15.122:0/3193798625 wait complete. 2024-03-06T07:25:48.914 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:48.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:49.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.016+0000 7fcd019e9640 1 Processor -- start 2024-03-06T07:25:49.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.016+0000 7fcd019e9640 1 -- start start 2024-03-06T07:25:49.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fcd019e9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fccfc0708b0 0x7fccfc070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fcd019e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fccfc0ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fcd019e9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc15afe0 con 0x7fccfc15ac00 2024-03-06T07:25:49.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fcd019e9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccfc071250 con 0x7fccfc0708b0 2024-03-06T07:25:49.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fcd019e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0cdc10 con 0x7fccfc0ce0a0 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fccfaffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fccfc15ac00 0x7fccfc06c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47566/0 (socket says 172.21.15.122:47566) 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.017+0000 7fccfaffd640 1 -- 172.21.15.122:0/1212437154 learned_addr learned my addr 172.21.15.122:0/1212437154 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fccfc0ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfa7fc640 1 --2- 172.21.15.122:0/1212437154 >> v2:172.21.15.169:3300/0 conn(0x7fccfc0708b0 0x7fccfc070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050486897 0 0) 0x7fccfc15afe0 con 0x7fccfc15ac00 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce4003520 con 0x7fccfc15ac00 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1309865666 0 0) 0x7fcce4003520 con 0x7fccfc15ac00 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 msgr2=0x7fccfc0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fccfc0ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 >> v2:172.21.15.169:3300/0 conn(0x7fccfc0708b0 msgr2=0x7fccfc070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 --2- 172.21.15.122:0/1212437154 >> v2:172.21.15.169:3300/0 conn(0x7fccfc0708b0 0x7fccfc070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc0cd000 con 0x7fccfc15ac00 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fccfc0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:49.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fccec0019f0 con 0x7fccfc15ac00 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 >> v1:172.21.15.122:6789/0 conn(0x7fccfc15ac00 legacy=0x7fccfc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fcce40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcce4003ef0 0x7fcce4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcce4004d40 0x7fcce40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0cdc10 con 0x7fcce4004d40 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400b9c0 con 0x7fcce4003ef0 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400bb40 con 0x7fccfc0ce0a0 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfa7fc640 1 --2- 172.21.15.122:0/1212437154 >> v2:172.21.15.169:3300/0 conn(0x7fccfc0708b0 0x7fccfc070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcce4004d40 0x7fcce40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfaffd640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fcce40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfa7fc640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcce4003ef0 0x7fcce4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 -- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 msgr2=0x7fcce40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc0ce0a0 0x7fcce40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 -- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcce4003ef0 msgr2=0x7fcce4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcce4003ef0 0x7fcce4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.018+0000 7fccfb7fe640 1 -- 172.21.15.122:0/1212437154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4003520 con 0x7fcce4004d40 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.019+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcce4004d40 0x7fcce40051c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009c90 tx=0x7fcce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7751bc37d0f2d2d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.019+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4039070 con 0x7fcce4004d40 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.019+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf402c8f0 con 0x7fcce4004d40 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.019+0000 7fccf9ffb640 1 -- 172.21.15.122:0/1212437154 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4009860 con 0x7fcce4004d40 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fcd019e9640 1 -- 172.21.15.122:0/1212437154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcce4004d40 msgr2=0x7fcce40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fcd019e9640 1 --2- 172.21.15.122:0/1212437154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcce4004d40 0x7fcce40051c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009c90 tx=0x7fcce4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fccfb7fe640 1 -- 172.21.15.122:0/1212437154 reap_dead start 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fcd019e9640 1 -- 172.21.15.122:0/1212437154 shutdown_connections 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fcd019e9640 1 -- 172.21.15.122:0/1212437154 >> 172.21.15.122:0/1212437154 conn(0x7fccfc0b1650 msgr2=0x7fccfc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fcd019e9640 1 -- 172.21.15.122:0/1212437154 shutdown_connections 2024-03-06T07:25:49.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.020+0000 7fcd019e9640 1 -- 172.21.15.122:0/1212437154 wait complete. 2024-03-06T07:25:49.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.021+0000 7fcd019e9640 1 Processor -- start 2024-03-06T07:25:49.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.021+0000 7fcd019e9640 1 -- start start 2024-03-06T07:25:49.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcd019e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc152550 0x7fccfc159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcd019e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fccfc0a9450 0x7fccfc0a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcd019e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 0x7fccfc155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcd019e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0cd8a0 con 0x7fccfc1552f0 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcd019e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0cd5c0 con 0x7fccfc0a9450 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcd019e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc0cf5d0 con 0x7fccfc152550 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 0x7fccfc155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 0x7fccfc155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51056/0 (socket says 172.21.15.122:51056) 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 -- 172.21.15.122:0/3412246775 learned_addr learned my addr 172.21.15.122:0/3412246775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfaffd640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc152550 0x7fccfc159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfa7fc640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fccfc0a9450 0x7fccfc0a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 -- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc152550 msgr2=0x7fccfc159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc152550 0x7fccfc159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 -- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fccfc0a9450 msgr2=0x7fccfc0a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fccfc0a9450 0x7fccfc0a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 -- 172.21.15.122:0/3412246775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc1588a0 con 0x7fccfc1552f0 2024-03-06T07:25:49.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfaffd640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc152550 0x7fccfc159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fccfb7fe640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 0x7fccfc155770 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009db0 tx=0x7fccf4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.022+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4039030 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.023+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf403a040 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.023+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc158b80 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.023+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4009860 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.023+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc159140 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.024+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc155cb0 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.024+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccf4045030 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.024+0000 7fcce3fff640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcce806b4d0 0x7fcce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.024+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fccf40aa9b0 con 0x7fccfc1552f0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.025+0000 7fccfaffd640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcce806b4d0 0x7fcce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.025+0000 7fccfaffd640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcce806b4d0 0x7fcce806d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fccec001b60 tx=0x7fccec006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.029+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccf404c060 con 0x7fccfc1552f0 2024-03-06T07:25:49.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.201+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fccfc0687d0 con 0x7fccfc1552f0 2024-03-06T07:25:49.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.642+0000 7fcce3fff640 1 -- 172.21.15.122:0/3412246775 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7fccf4079c10 con 0x7fccfc1552f0 2024-03-06T07:25:49.644 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcce806b4d0 msgr2=0x7fcce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcce806b4d0 0x7fcce806d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fccec001b60 tx=0x7fccec006040 comp rx=0 tx=0).stop 2024-03-06T07:25:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 msgr2=0x7fccfc155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 0x7fccfc155770 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009db0 tx=0x7fccf4004730 comp rx=0 tx=0).stop 2024-03-06T07:25:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 shutdown_connections 2024-03-06T07:25:49.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcce806b4d0 0x7fcce806d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccfc1552f0 0x7fccfc155770 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fccfc0a9450 0x7fccfc0a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 --2- 172.21.15.122:0/3412246775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fccfc152550 0x7fccfc159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 >> 172.21.15.122:0/3412246775 conn(0x7fccfc0b1650 msgr2=0x7fccfc06d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 shutdown_connections 2024-03-06T07:25:49.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.645+0000 7fcd019e9640 1 -- 172.21.15.122:0/3412246775 wait complete. 2024-03-06T07:25:49.662 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:49.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.767+0000 7f9cfc6b5640 1 Processor -- start 2024-03-06T07:25:49.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.767+0000 7f9cfc6b5640 1 -- start start 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cfc6b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9cf40cfa90 0x7f9cf40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cfc6b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9cf40b33a0 0x7f9cf40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cfc6b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf4158ac0 con 0x7f9cf41586e0 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cfc6b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf415b2a0 con 0x7f9cf40cfa90 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cfc6b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf415b540 con 0x7f9cf40b33a0 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cf9c50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9cf40cfa90 0x7f9cf40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cf9c50640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9cf40cfa90 0x7f9cf40cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51064/0 (socket says 172.21.15.122:51064) 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cf9c50640 1 -- 172.21.15.122:0/3185858724 learned_addr learned my addr 172.21.15.122:0/3185858724 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.768+0000 7f9cfac52640 1 --2- 172.21.15.122:0/3185858724 >> v2:172.21.15.169:3300/0 conn(0x7f9cf40b33a0 0x7f9cf40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf9c50640 1 -- 172.21.15.122:0/3185858724 >> v2:172.21.15.169:3300/0 conn(0x7f9cf40b33a0 msgr2=0x7f9cf40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf9c50640 1 --2- 172.21.15.122:0/3185858724 >> v2:172.21.15.169:3300/0 conn(0x7f9cf40b33a0 0x7f9cf40b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf9c50640 1 -- 172.21.15.122:0/3185858724 >> v1:172.21.15.169:6789/0 conn(0x7f9cf41586e0 legacy=0x7f9cf40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf9c50640 1 -- 172.21.15.122:0/3185858724 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf415b820 con 0x7f9cf40cfa90 2024-03-06T07:25:49.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf9c50640 1 --2- 172.21.15.122:0/3185858724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40cfa90 0x7f9cf40cff40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40b2ae0 tx=0x7f9ce402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cab2b9dc0cd8944 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf944f640 1 -- 172.21.15.122:0/3185858724 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce403b070 con 0x7f9cf40cfa90 2024-03-06T07:25:49.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.769+0000 7f9cf944f640 1 -- 172.21.15.122:0/3185858724 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce4036440 con 0x7f9cf40cfa90 2024-03-06T07:25:49.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cf944f640 1 -- 172.21.15.122:0/3185858724 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4036810 con 0x7f9cf40cfa90 2024-03-06T07:25:49.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/3185858724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40cfa90 msgr2=0x7f9cf40cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/3185858724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40cfa90 0x7f9cf40cff40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40b2ae0 tx=0x7f9ce402e170 comp rx=0 tx=0).stop 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/3185858724 shutdown_connections 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/3185858724 >> v2:172.21.15.169:3300/0 conn(0x7f9cf40b33a0 0x7f9cf40b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/3185858724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40cfa90 0x7f9cf40cff40 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/3185858724 >> 172.21.15.122:0/3185858724 conn(0x7f9cf406b190 msgr2=0x7f9cf406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/3185858724 shutdown_connections 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.770+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/3185858724 wait complete. 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.771+0000 7f9cfc6b5640 1 Processor -- start 2024-03-06T07:25:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.771+0000 7f9cfc6b5640 1 -- start start 2024-03-06T07:25:49.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfc6b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 0x7f9cf40abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfc6b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cf40cfa90 0x7f9cf40ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfc6b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cf41586e0 0x7f9cf41475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfc6b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40af150 con 0x7f9cf40b33a0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfc6b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf415bde0 con 0x7f9cf40cfa90 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfc6b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf4071350 con 0x7f9cf41586e0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 0x7f9cf40abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 0x7f9cf40abbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51074/0 (socket says 172.21.15.122:51074) 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 -- 172.21.15.122:0/666791286 learned_addr learned my addr 172.21.15.122:0/666791286 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 -- 172.21.15.122:0/666791286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cf41586e0 msgr2=0x7f9cf41475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cf41586e0 0x7f9cf41475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 -- 172.21.15.122:0/666791286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cf40cfa90 msgr2=0x7f9cf40ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cf40cfa90 0x7f9cf40ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 -- 172.21.15.122:0/666791286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4148c10 con 0x7f9cf40b33a0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9cfa451640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 0x7f9cf40abbc0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000c790 tx=0x7f9cf000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.772+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0016020 con 0x7f9cf40b33a0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.773+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0006d50 con 0x7f9cf40b33a0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.773+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf4148650 con 0x7f9cf40b33a0 2024-03-06T07:25:49.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.773+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0003bb0 con 0x7f9cf40b33a0 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.773+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf40ccc60 con 0x7f9cf40b33a0 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.773+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf40732b0 con 0x7f9cf40b33a0 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.774+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cf000a850 con 0x7f9cf40b33a0 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.774+0000 7f9ceaffd640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cd806b500 0x7f9cd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.774+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9cf0089cc0 con 0x7f9cf40b33a0 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.775+0000 7f9cf9c50640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cd806b500 0x7f9cd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.775+0000 7f9cf9c50640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cd806b500 0x7f9cd806d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4038000 tx=0x7f9ce4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.779+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cf0020020 con 0x7f9cf40b33a0 2024-03-06T07:25:49.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:49.950+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9cf40aee50 con 0x7f9cf40b33a0 2024-03-06T07:25:50.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.644+0000 7f9ceaffd640 1 -- 172.21.15.122:0/666791286 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9cf0019310 con 0x7f9cf40b33a0 2024-03-06T07:25:50.646 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.646+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cd806b500 msgr2=0x7f9cd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.646+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cd806b500 0x7f9cd806d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4038000 tx=0x7f9ce4030000 comp rx=0 tx=0).stop 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.646+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 msgr2=0x7f9cf40abbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.646+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 0x7f9cf40abbc0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000c790 tx=0x7f9cf000cc50 comp rx=0 tx=0).stop 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 shutdown_connections 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9cd806b500 0x7f9cd806d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9cf41586e0 0x7f9cf41475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9cf40cfa90 0x7f9cf40ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 --2- 172.21.15.122:0/666791286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cf40b33a0 0x7f9cf40abbc0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 >> 172.21.15.122:0/666791286 conn(0x7f9cf406b190 msgr2=0x7f9cf41578b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 shutdown_connections 2024-03-06T07:25:50.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.647+0000 7f9cfc6b5640 1 -- 172.21.15.122:0/666791286 wait complete. 2024-03-06T07:25:50.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:50.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.775+0000 7f8c13593640 1 Processor -- start 2024-03-06T07:25:50.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.775+0000 7f8c13593640 1 -- start start 2024-03-06T07:25:50.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c13593640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 0x7f8c0c072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c13593640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 0x7f8c0c0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c13593640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c071c20 con 0x7f8c0c071810 2024-03-06T07:25:50.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c13593640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c06caa0 con 0x7f8c0c0cfc50 2024-03-06T07:25:50.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c13593640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0b3340 con 0x7f8c0c0cf870 2024-03-06T07:25:50.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c1132f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 0x7f8c0c072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c10b2e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 0x7f8c0c0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c10b2e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 0x7f8c0c0712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54960/0 (socket says 172.21.15.122:54960) 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c1132f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 0x7f8c0c072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:37584/0 (socket says 172.21.15.122:37584) 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c10b2e640 1 -- 172.21.15.122:0/740145391 learned_addr learned my addr 172.21.15.122:0/740145391 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c1132f640 1 -- 172.21.15.122:0/740145391 >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 msgr2=0x7f8c0c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c1132f640 1 --2- 172.21.15.122:0/740145391 >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 0x7f8c0c0712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c1132f640 1 -- 172.21.15.122:0/740145391 >> v1:172.21.15.169:6789/0 conn(0x7f8c0c071810 legacy=0x7f8c0c06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c1132f640 1 -- 172.21.15.122:0/740145391 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c0b3620 con 0x7f8c0c0cf870 2024-03-06T07:25:50.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.776+0000 7f8c10b2e640 1 --2- 172.21.15.122:0/740145391 >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 0x7f8c0c0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:50.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.777+0000 7f8c1132f640 1 --2- 172.21.15.122:0/740145391 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 0x7f8c0c072530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8c08009d90 tx=0x7f8c0802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f71cae0d1b629f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.777+0000 7f8c03fff640 1 -- 172.21.15.122:0/740145391 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08039070 con 0x7f8c0c0cf870 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.777+0000 7f8c03fff640 1 -- 172.21.15.122:0/740145391 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0802cce0 con 0x7f8c0c0cf870 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.777+0000 7f8c03fff640 1 -- 172.21.15.122:0/740145391 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08034660 con 0x7f8c0c0cf870 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 -- 172.21.15.122:0/740145391 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 msgr2=0x7f8c0c072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 --2- 172.21.15.122:0/740145391 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 0x7f8c0c072530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8c08009d90 tx=0x7f8c0802bf30 comp rx=0 tx=0).stop 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 -- 172.21.15.122:0/740145391 shutdown_connections 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 --2- 172.21.15.122:0/740145391 >> v2:172.21.15.169:3300/0 conn(0x7f8c0c0cfc50 0x7f8c0c0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 --2- 172.21.15.122:0/740145391 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cf870 0x7f8c0c072530 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 -- 172.21.15.122:0/740145391 >> 172.21.15.122:0/740145391 conn(0x7f8c0c06afb0 msgr2=0x7f8c0c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 -- 172.21.15.122:0/740145391 shutdown_connections 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.778+0000 7f8c13593640 1 -- 172.21.15.122:0/740145391 wait complete. 2024-03-06T07:25:50.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.779+0000 7f8c13593640 1 Processor -- start 2024-03-06T07:25:50.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.779+0000 7f8c13593640 1 -- start start 2024-03-06T07:25:50.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c13593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 0x7f8c0c147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c13593640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c0c0cf870 0x7f8c0c149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c13593640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cfc50 0x7f8c0c14d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c13593640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0b3a80 con 0x7f8c0c071810 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c13593640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0b3900 con 0x7f8c0c0cf870 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c13593640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c15b5a0 con 0x7f8c0c0cfc50 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 0x7f8c0c147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 0x7f8c0c147f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51078/0 (socket says 172.21.15.122:51078) 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 -- 172.21.15.122:0/4120478758 learned_addr learned my addr 172.21.15.122:0/4120478758 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c11b30640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cfc50 0x7f8c0c14d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c10b2e640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c0c0cf870 0x7f8c0c149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 -- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cfc50 msgr2=0x7f8c0c14d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cfc50 0x7f8c0c14d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 -- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c0c0cf870 msgr2=0x7f8c0c149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c0c0cf870 0x7f8c0c149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 -- 172.21.15.122:0/4120478758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c14a3a0 con 0x7f8c0c071810 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c1132f640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 0x7f8c0c147f60 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8c08037040 tx=0x7f8c08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.780+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08039050 con 0x7f8c0c071810 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.781+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c08045070 con 0x7f8c0c071810 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.781+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c0cca50 con 0x7f8c0c071810 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.781+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08009450 con 0x7f8c0c071810 2024-03-06T07:25:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.781+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c0cd010 con 0x7f8c0c071810 2024-03-06T07:25:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.782+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c0cd210 con 0x7f8c0c071810 2024-03-06T07:25:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.782+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c0803e850 con 0x7f8c0c071810 2024-03-06T07:25:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.783+0000 7f8c01ffb640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf006b4d0 0x7f8bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.783+0000 7f8c10b2e640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf006b4d0 0x7f8bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.783+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8c080ad140 con 0x7f8c0c071810 2024-03-06T07:25:50.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.783+0000 7f8c10b2e640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf006b4d0 0x7f8bf006d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c14a730 tx=0x7f8bfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.788+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c0807c150 con 0x7f8c0c071810 2024-03-06T07:25:50.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:50.996+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8c0c04f330 con 0x7f8c0c071810 2024-03-06T07:25:51.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.667+0000 7f8c01ffb640 1 -- 172.21.15.122:0/4120478758 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8c0807ba50 con 0x7f8c0c071810 2024-03-06T07:25:51.669 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf006b4d0 msgr2=0x7f8bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf006b4d0 0x7f8bf006d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c14a730 tx=0x7f8bfc008040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 msgr2=0x7f8c0c147f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 0x7f8c0c147f60 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8c08037040 tx=0x7f8c08004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 shutdown_connections 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bf006b4d0 0x7f8bf006d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.675+0000 7f8c13593640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8c0c0cfc50 0x7f8c0c14d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.676+0000 7f8c13593640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c0c0cf870 0x7f8c0c149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.676+0000 7f8c13593640 1 --2- 172.21.15.122:0/4120478758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c0c071810 0x7f8c0c147f60 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.676+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 >> 172.21.15.122:0/4120478758 conn(0x7f8c0c06afb0 msgr2=0x7f8c0c0b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.677+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 shutdown_connections 2024-03-06T07:25:51.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.677+0000 7f8c13593640 1 -- 172.21.15.122:0/4120478758 wait complete. 2024-03-06T07:25:51.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:51.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.795+0000 7f2ee1d0d640 1 Processor -- start 2024-03-06T07:25:51.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.795+0000 7f2ee1d0d640 1 -- start start 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2ee1d0d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2edc0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2ee1d0d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2edc0cfff0 0x7f2edc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2ee1d0d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc161980 con 0x7f2edc0ce4b0 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2ee1d0d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc161c60 con 0x7f2edc0cfff0 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2ee1d0d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc161f40 con 0x7f2edc15bb20 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2edbfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2edc0ce4b0 0x7f2edc0ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47578/0 (socket says 172.21.15.122:47578) 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2edbfff640 1 -- 172.21.15.122:0/467944265 learned_addr learned my addr 172.21.15.122:0/467944265 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2edb7fe640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2edc0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.796+0000 7f2edaffd640 1 --2- 172.21.15.122:0/467944265 >> v2:172.21.15.169:3300/0 conn(0x7f2edc0cfff0 0x7f2edc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091977627 0 0) 0x7f2edc161980 con 0x7f2edc0ce4b0 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec4003520 con 0x7f2edc0ce4b0 2024-03-06T07:25:51.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237715955 0 0) 0x7f2ec4003520 con 0x7f2edc0ce4b0 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 msgr2=0x7f2edc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2edc0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 >> v2:172.21.15.169:3300/0 conn(0x7f2edc0cfff0 msgr2=0x7f2edc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 --2- 172.21.15.122:0/467944265 >> v2:172.21.15.169:3300/0 conn(0x7f2edc0cfff0 0x7f2edc0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc162220 con 0x7f2edc0ce4b0 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f2ed4001670 con 0x7f2edc0ce4b0 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 >> v1:172.21.15.122:6789/0 conn(0x7f2edc0ce4b0 legacy=0x7f2edc0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ec40039b0 0x7f2ec4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec4004800 0x7f2ec4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2ec40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc161c60 con 0x7f2ec4004800 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc161f40 con 0x7f2ec40039b0 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec400b9c0 con 0x7f2edc15bb20 2024-03-06T07:25:51.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edaffd640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec4004800 0x7f2ec4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edb7fe640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ec40039b0 0x7f2ec4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edbfff640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2ec40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edaffd640 1 -- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 msgr2=0x7f2ec40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edaffd640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2ec40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edaffd640 1 -- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ec40039b0 msgr2=0x7f2ec4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edaffd640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ec40039b0 0x7f2ec4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edaffd640 1 -- 172.21.15.122:0/467944265 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec4003520 con 0x7f2ec4004800 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edb7fe640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ec40039b0 0x7f2ec4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.797+0000 7f2edbfff640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2edc15bb20 0x7f2ec40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.798+0000 7f2edaffd640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec4004800 0x7f2ec4004c80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009ac0 tx=0x7f2ec4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d089a418b6de3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.798+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0039070 con 0x7f2ec4004800 2024-03-06T07:25:51.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.798+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed002c710 con 0x7f2ec4004800 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.798+0000 7f2eda7fc640 1 -- 172.21.15.122:0/467944265 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0009690 con 0x7f2ec4004800 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/467944265 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec4004800 msgr2=0x7f2ec4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/467944265 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec4004800 0x7f2ec4004c80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009ac0 tx=0x7f2ec4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2edbfff640 1 -- 172.21.15.122:0/467944265 reap_dead start 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/467944265 shutdown_connections 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/467944265 >> 172.21.15.122:0/467944265 conn(0x7f2edc09a9f0 msgr2=0x7f2edc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/467944265 shutdown_connections 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.799+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/467944265 wait complete. 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.800+0000 7f2ee1d0d640 1 Processor -- start 2024-03-06T07:25:51.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.800+0000 7f2ee1d0d640 1 -- start start 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ee1d0d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2ec4004800 0x7f2edc0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ee1d0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 0x7f2edc0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ee1d0d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2edc0b6660 0x7f2edc06c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ee1d0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0d1b00 con 0x7f2edc0c7f10 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ee1d0d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0d1980 con 0x7f2edc0b6660 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ee1d0d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0cf050 con 0x7f2ec4004800 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 0x7f2edc0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 0x7f2edc0b6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51102/0 (socket says 172.21.15.122:51102) 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 -- 172.21.15.122:0/3571441061 learned_addr learned my addr 172.21.15.122:0/3571441061 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:51.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edb7fe640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2ec4004800 0x7f2edc0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edbfff640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2edc0b6660 0x7f2edc06c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 -- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2ec4004800 msgr2=0x7f2edc0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2ec4004800 0x7f2edc0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 -- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2edc0b6660 msgr2=0x7f2edc06c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2edc0b6660 0x7f2edc06c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 -- 172.21.15.122:0/3571441061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc1591b0 con 0x7f2edc0c7f10 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edbfff640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2edc0b6660 0x7f2edc06c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2edaffd640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 0x7f2edc0b6120 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0000c00 tx=0x7f2ed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.801+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0039050 con 0x7f2edc0c7f10 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.802+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed0045070 con 0x7f2edc0c7f10 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.802+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc159490 con 0x7f2edc0c7f10 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.802+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00044e0 con 0x7f2edc0c7f10 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.802+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc0b7030 con 0x7f2edc0c7f10 2024-03-06T07:25:51.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.803+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc0d2510 con 0x7f2edc0c7f10 2024-03-06T07:25:51.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.803+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ed0003a50 con 0x7f2edc0c7f10 2024-03-06T07:25:51.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.803+0000 7f2ec3fff640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2ec806b4d0 0x7f2ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.804+0000 7f2edb7fe640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2ec806b4d0 0x7f2ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.804+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ed00abf10 con 0x7f2edc0c7f10 2024-03-06T07:25:51.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.804+0000 7f2edb7fe640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2ec806b4d0 0x7f2ec806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc007440 tx=0x7f2ecc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.808+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2edc0d2510 con 0x7f2edc0c7f10 2024-03-06T07:25:51.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.981+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2edc04f330 con 0x7f2edc0c7f10 2024-03-06T07:25:51.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.982+0000 7f2ec3fff640 1 -- 172.21.15.122:0/3571441061 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16673 (secure 0 0 0) 0x7f2ed004e020 con 0x7f2edc0c7f10 2024-03-06T07:25:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:51.985 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:50.997984+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2ec806b4d0 msgr2=0x7f2ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2ec806b4d0 0x7f2ec806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc007440 tx=0x7f2ecc00b040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 msgr2=0x7f2edc0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 0x7f2edc0b6120 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0000c00 tx=0x7f2ed0004730 comp rx=0 tx=0).stop 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 shutdown_connections 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2ec806b4d0 0x7f2ec806d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2edc0b6660 0x7f2edc06c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2edc0c7f10 0x7f2edc0b6120 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 --2- 172.21.15.122:0/3571441061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2ec4004800 0x7f2edc0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 >> 172.21.15.122:0/3571441061 conn(0x7f2edc09a9f0 msgr2=0x7f2edc06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 shutdown_connections 2024-03-06T07:25:51.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:51.984+0000 7f2ee1d0d640 1 -- 172.21.15.122:0/3571441061 wait complete. 2024-03-06T07:25:51.998 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.095+0000 7fc6e16e1640 1 Processor -- start 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.095+0000 7fc6e16e1640 1 -- start start 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6e16e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 0x7fc6dc0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6e16e1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc6dc0cd7a0 0x7fc6dc0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6e16e1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc071f40 con 0x7fc6dc15b430 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6e16e1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc15b8b0 con 0x7fc6dc0cd7a0 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6e16e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc070a70 con 0x7fc6dc1515e0 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6d3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc6dc15b430 0x7fc6dc071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47582/0 (socket says 172.21.15.122:47582) 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6d3fff640 1 -- 172.21.15.122:0/3939688497 learned_addr learned my addr 172.21.15.122:0/3939688497 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6daffd640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 0x7fc6dc0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145388251 0 0) 0x7fc6dc071f40 con 0x7fc6dc15b430 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c8003520 con 0x7fc6dc15b430 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6db7fe640 1 --2- 172.21.15.122:0/3939688497 >> v2:172.21.15.169:3300/0 conn(0x7fc6dc0cd7a0 0x7fc6dc0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165066593 0 0) 0x7fc6c8003520 con 0x7fc6dc15b430 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 msgr2=0x7fc6dc0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 0x7fc6dc0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 >> v2:172.21.15.169:3300/0 conn(0x7fc6dc0cd7a0 msgr2=0x7fc6dc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 --2- 172.21.15.122:0/3939688497 >> v2:172.21.15.169:3300/0 conn(0x7fc6dc0cd7a0 0x7fc6dc0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc070d50 con 0x7fc6dc15b430 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.096+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fc6c4001670 con 0x7fc6dc15b430 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 >> v1:172.21.15.122:6789/0 conn(0x7fc6dc15b430 legacy=0x7fc6dc071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c80039b0 0x7fc6c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6c8004800 0x7fc6c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 0x7fc6c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc15b8b0 con 0x7fc6c8004800 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc070a70 con 0x7fc6c80039b0 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c800b9c0 con 0x7fc6dc1515e0 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6c8004800 0x7fc6c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6daffd640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c80039b0 0x7fc6c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6db7fe640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 0x7fc6c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6d3fff640 1 -- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 msgr2=0x7fc6c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc1515e0 0x7fc6c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6d3fff640 1 -- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c80039b0 msgr2=0x7fc6c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c80039b0 0x7fc6c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.097+0000 7fc6d3fff640 1 -- 172.21.15.122:0/3939688497 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8003520 con 0x7fc6c8004800 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6c8004800 0x7fc6c8004c80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c210 tx=0x7fc6c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8c8f0d03d7bdea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4007d50 con 0x7fc6c8004800 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4030070 con 0x7fc6c8004800 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6da7fc640 1 -- 172.21.15.122:0/3939688497 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c402cd80 con 0x7fc6c8004800 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6e16e1640 1 -- 172.21.15.122:0/3939688497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6c8004800 msgr2=0x7fc6c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/3939688497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6c8004800 0x7fc6c8004c80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c210 tx=0x7fc6c8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:52.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6db7fe640 1 -- 172.21.15.122:0/3939688497 reap_dead start 2024-03-06T07:25:52.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6e16e1640 1 -- 172.21.15.122:0/3939688497 shutdown_connections 2024-03-06T07:25:52.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6e16e1640 1 -- 172.21.15.122:0/3939688497 >> 172.21.15.122:0/3939688497 conn(0x7fc6dc0cf8a0 msgr2=0x7fc6dc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6e16e1640 1 -- 172.21.15.122:0/3939688497 shutdown_connections 2024-03-06T07:25:52.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.098+0000 7fc6e16e1640 1 -- 172.21.15.122:0/3939688497 wait complete. 2024-03-06T07:25:52.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 Processor -- start 2024-03-06T07:25:52.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 -- start start 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c8004800 0x7fc6dc1421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 0x7fc6dc14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc14e2d0 0x7fc6dc166110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0ce110 con 0x7fc6dc14aae0 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc071310 con 0x7fc6c8004800 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6e16e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0cce40 con 0x7fc6dc14e2d0 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 0x7fc6dc14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 0x7fc6dc14dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51106/0 (socket says 172.21.15.122:51106) 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6d3fff640 1 -- 172.21.15.122:0/121305625 learned_addr learned my addr 172.21.15.122:0/121305625 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.100+0000 7fc6daffd640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c8004800 0x7fc6dc1421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6db7fe640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc14e2d0 0x7fc6dc166110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d3fff640 1 -- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc14e2d0 msgr2=0x7fc6dc166110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc14e2d0 0x7fc6dc166110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d3fff640 1 -- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c8004800 msgr2=0x7fc6dc1421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c8004800 0x7fc6dc1421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d3fff640 1 -- 172.21.15.122:0/121305625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc14af70 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d3fff640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 0x7fc6dc14dd90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c330 tx=0x7fc6c4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4001c90 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4004bf0 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc14b250 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4007070 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.101+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc14b810 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.102+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc068840 con 0x7fc6dc14aae0 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.103+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6c4047020 con 0x7fc6dc14aae0 2024-03-06T07:25:52.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.103+0000 7fc6d37fe640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6bc06b4d0 0x7fc6bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.103+0000 7fc6daffd640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6bc06b4d0 0x7fc6bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.103+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6c40ab380 con 0x7fc6dc14aae0 2024-03-06T07:25:52.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.103+0000 7fc6daffd640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6bc06b4d0 0x7fc6bc06d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009a60 tx=0x7fc6cc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.107+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6c407a2c0 con 0x7fc6dc14aae0 2024-03-06T07:25:52.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.281+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6dc04f330 con 0x7fc6dc14aae0 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.282+0000 7fc6d37fe640 1 -- 172.21.15.122:0/121305625 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16673 (secure 0 0 0) 0x7fc6c4079bc0 con 0x7fc6dc14aae0 2024-03-06T07:25:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:52.284 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:50.997984+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6bc06b4d0 msgr2=0x7fc6bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6bc06b4d0 0x7fc6bc06d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009a60 tx=0x7fc6cc009190 comp rx=0 tx=0).stop 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 msgr2=0x7fc6dc14dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 0x7fc6dc14dd90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c330 tx=0x7fc6c4001a40 comp rx=0 tx=0).stop 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 shutdown_connections 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc6bc06b4d0 0x7fc6bc06d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc6dc14e2d0 0x7fc6dc166110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6dc14aae0 0x7fc6dc14dd90 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 --2- 172.21.15.122:0/121305625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6c8004800 0x7fc6dc1421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 >> 172.21.15.122:0/121305625 conn(0x7fc6dc0cf8a0 msgr2=0x7fc6dc071730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 shutdown_connections 2024-03-06T07:25:52.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.284+0000 7fc6e16e1640 1 -- 172.21.15.122:0/121305625 wait complete. 2024-03-06T07:25:52.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:52.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.398+0000 7f3132535640 1 Processor -- start 2024-03-06T07:25:52.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.398+0000 7f3132535640 1 -- start start 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3132535640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f312c15bb40 0x7f312c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3132535640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c15f7d0 0x7f312c06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3132535640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f312c06d5c0 0x7f312c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3132535640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c06c910 con 0x7f312c06d5c0 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3132535640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c15ff90 con 0x7f312c15bb40 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3132535640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c160270 con 0x7f312c15f7d0 2024-03-06T07:25:52.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f312c06d5c0 0x7f312c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f312c06d5c0 0x7f312c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51116/0 (socket says 172.21.15.122:51116) 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 -- 172.21.15.122:0/2145174599 learned_addr learned my addr 172.21.15.122:0/2145174599 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f312bfff640 1 --2- 172.21.15.122:0/2145174599 >> v2:172.21.15.169:3300/0 conn(0x7f312c15bb40 0x7f312c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f312b7fe640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c15f7d0 0x7f312c06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 -- 172.21.15.122:0/2145174599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c15f7d0 msgr2=0x7f312c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c15f7d0 0x7f312c06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 -- 172.21.15.122:0/2145174599 >> v2:172.21.15.169:3300/0 conn(0x7f312c15bb40 msgr2=0x7f312c0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 --2- 172.21.15.122:0/2145174599 >> v2:172.21.15.169:3300/0 conn(0x7f312c15bb40 0x7f312c0b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f3130ad2640 1 -- 172.21.15.122:0/2145174599 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c160550 con 0x7f312c06d5c0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f312bfff640 1 --2- 172.21.15.122:0/2145174599 >> v2:172.21.15.169:3300/0 conn(0x7f312c15bb40 0x7f312c0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.399+0000 7f312b7fe640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c15f7d0 0x7f312c06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.400+0000 7f3130ad2640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c06d5c0 0x7f312c0cfa90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f312c0b4f60 tx=0x7f312402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d65feef710585f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.400+0000 7f312affd640 1 -- 172.21.15.122:0/2145174599 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312402ca70 con 0x7f312c06d5c0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.400+0000 7f312affd640 1 -- 172.21.15.122:0/2145174599 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312402cc10 con 0x7f312c06d5c0 2024-03-06T07:25:52.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.400+0000 7f312affd640 1 -- 172.21.15.122:0/2145174599 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240343f0 con 0x7f312c06d5c0 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 -- 172.21.15.122:0/2145174599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c06d5c0 msgr2=0x7f312c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c06d5c0 0x7f312c0cfa90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f312c0b4f60 tx=0x7f312402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 -- 172.21.15.122:0/2145174599 shutdown_connections 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c06d5c0 0x7f312c0cfa90 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 --2- 172.21.15.122:0/2145174599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c15f7d0 0x7f312c06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 --2- 172.21.15.122:0/2145174599 >> v2:172.21.15.169:3300/0 conn(0x7f312c15bb40 0x7f312c0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 -- 172.21.15.122:0/2145174599 >> 172.21.15.122:0/2145174599 conn(0x7f312c06b190 msgr2=0x7f312c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 -- 172.21.15.122:0/2145174599 shutdown_connections 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.401+0000 7f3132535640 1 -- 172.21.15.122:0/2145174599 wait complete. 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.402+0000 7f3132535640 1 Processor -- start 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.402+0000 7f3132535640 1 -- start start 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3132535640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c06d5c0 0x7f312c0add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3132535640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 0x7f312c15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3132535640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f312c15f7d0 0x7f312c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3132535640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15bf20 con 0x7f312c15bb40 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3132535640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15fc30 con 0x7f312c15f7d0 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3132535640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15c0a0 con 0x7f312c06d5c0 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 0x7f312c15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 0x7f312c15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51130/0 (socket says 172.21.15.122:51130) 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 -- 172.21.15.122:0/1249589228 learned_addr learned my addr 172.21.15.122:0/1249589228 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312bfff640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c06d5c0 0x7f312c0add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3130ad2640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f312c15f7d0 0x7f312c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 -- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c06d5c0 msgr2=0x7f312c0add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c06d5c0 0x7f312c0add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 -- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f312c15f7d0 msgr2=0x7f312c15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f312c15f7d0 0x7f312c15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312b7fe640 1 -- 172.21.15.122:0/1249589228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c1f28f0 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f312bfff640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c06d5c0 0x7f312c0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.403+0000 7f3130ad2640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f312c15f7d0 0x7f312c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.404+0000 7f312b7fe640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 0x7f312c15cc00 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f312000c7f0 tx=0x7f312000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.404+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120015070 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.404+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3120007a30 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.404+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c1f1b30 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.404+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120007e00 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.404+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c1f20f0 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.405+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c0732b0 con 0x7f312c15bb40 2024-03-06T07:25:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.405+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3120018070 con 0x7f312c15bb40 2024-03-06T07:25:52.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.405+0000 7f3128ff9640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311806b500 0x7f311806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.406+0000 7f312bfff640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311806b500 0x7f311806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.406+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f312008a100 con 0x7f312c15bb40 2024-03-06T07:25:52.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.406+0000 7f312bfff640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311806b500 0x7f311806d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f311c006fd0 tx=0x7f311c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.410+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3120058f80 con 0x7f312c15bb40 2024-03-06T07:25:52.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.616+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f312c068ff0 con 0x7f312c15bb40 2024-03-06T07:25:52.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.677+0000 7f3128ff9640 1 -- 172.21.15.122:0/1249589228 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3120058880 con 0x7f312c15bb40 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311806b500 msgr2=0x7f311806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311806b500 0x7f311806d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f311c006fd0 tx=0x7f311c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 msgr2=0x7f312c15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 0x7f312c15cc00 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f312000c7f0 tx=0x7f312000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 shutdown_connections 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311806b500 0x7f311806d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f312c15f7d0 0x7f312c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f312c15bb40 0x7f312c15cc00 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 --2- 172.21.15.122:0/1249589228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f312c06d5c0 0x7f312c0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 >> 172.21.15.122:0/1249589228 conn(0x7f312c06b190 msgr2=0x7f312c0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 shutdown_connections 2024-03-06T07:25:52.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.680+0000 7f3132535640 1 -- 172.21.15.122:0/1249589228 wait complete. 2024-03-06T07:25:52.693 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:52.693 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:52.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.795+0000 7f037741c640 1 Processor -- start 2024-03-06T07:25:52.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.795+0000 7f037741c640 1 -- start start 2024-03-06T07:25:52.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f037741c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f03700d14a0 0x7f037015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f037741c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f037015eb00 0x7f03700cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f037741c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700cfa50 con 0x7f03700d10c0 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f037741c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037015c050 con 0x7f03700d14a0 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f037741c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03700d0060 con 0x7f037015eb00 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f03700d14a0 0x7f037015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f03700d14a0 0x7f037015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51142/0 (socket says 172.21.15.122:51142) 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 -- 172.21.15.122:0/3424618948 learned_addr learned my addr 172.21.15.122:0/3424618948 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03759b9640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f037015eb00 0x7f03700cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 -- 172.21.15.122:0/3424618948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f037015eb00 msgr2=0x7f03700cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f037015eb00 0x7f03700cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 -- 172.21.15.122:0/3424618948 >> v1:172.21.15.169:6789/0 conn(0x7f03700d10c0 legacy=0x7f03700d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03749b7640 1 -- 172.21.15.122:0/3424618948 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03700b45c0 con 0x7f03700d14a0 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.796+0000 7f03759b9640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f037015eb00 0x7f03700cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.797+0000 7f03749b7640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d14a0 0x7f037015e5c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0360009870 tx=0x7f036002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67b5a2f3ed0d68b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.797+0000 7f0367fff640 1 -- 172.21.15.122:0/3424618948 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360035020 con 0x7f03700d14a0 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.797+0000 7f0367fff640 1 -- 172.21.15.122:0/3424618948 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360031640 con 0x7f03700d14a0 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.797+0000 7f0367fff640 1 -- 172.21.15.122:0/3424618948 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036002d040 con 0x7f03700d14a0 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.797+0000 7f037741c640 1 -- 172.21.15.122:0/3424618948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d14a0 msgr2=0x7f037015e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.797+0000 7f037741c640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d14a0 0x7f037015e5c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0360009870 tx=0x7f036002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.798+0000 7f037741c640 1 -- 172.21.15.122:0/3424618948 shutdown_connections 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.798+0000 7f037741c640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f037015eb00 0x7f03700cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.798+0000 7f037741c640 1 --2- 172.21.15.122:0/3424618948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d14a0 0x7f037015e5c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.798+0000 7f037741c640 1 -- 172.21.15.122:0/3424618948 >> 172.21.15.122:0/3424618948 conn(0x7f03700b3800 msgr2=0x7f03700b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.798+0000 7f037741c640 1 -- 172.21.15.122:0/3424618948 shutdown_connections 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.798+0000 7f037741c640 1 -- 172.21.15.122:0/3424618948 wait complete. 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.799+0000 7f037741c640 1 Processor -- start 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.799+0000 7f037741c640 1 -- start start 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f037741c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 0x7f03700ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f037741c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03700d14a0 0x7f03700b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f037741c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f037015eb00 0x7f03701f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f037741c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03700cfcf0 con 0x7f03700d10c0 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f037741c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037015f440 con 0x7f037015eb00 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f037741c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03700cb3c0 con 0x7f03700d14a0 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f03751b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 0x7f03700ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f03751b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 0x7f03700ceb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51158/0 (socket says 172.21.15.122:51158) 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f03751b8640 1 -- 172.21.15.122:0/616536550 learned_addr learned my addr 172.21.15.122:0/616536550 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f03759b9640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f037015eb00 0x7f03701f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.800+0000 7f03749b7640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03700d14a0 0x7f03700b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f03751b8640 1 -- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03700d14a0 msgr2=0x7f03700b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f03751b8640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03700d14a0 0x7f03700b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f03751b8640 1 -- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f037015eb00 msgr2=0x7f03701f7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f03751b8640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f037015eb00 0x7f03701f7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f03751b8640 1 -- 172.21.15.122:0/616536550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03701f8de0 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f03751b8640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 0x7f03700ceb20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f036800c7f0 tx=0x7f036800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368016020 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036800aa50 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03700cd070 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036800ae40 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.801+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03700cd630 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.802+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0370068ff0 con 0x7f03700d10c0 2024-03-06T07:25:52.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.803+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0368019020 con 0x7f03700d10c0 2024-03-06T07:25:52.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.803+0000 7f0365ffb640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f035406b4d0 0x7f035406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.803+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0368089e00 con 0x7f03700d10c0 2024-03-06T07:25:52.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.803+0000 7f03749b7640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f035406b4d0 0x7f035406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.804+0000 7f03749b7640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f035406b4d0 0x7f035406d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0360036000 tx=0x7f036003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:52.812+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0368020070 con 0x7f03700d10c0 2024-03-06T07:25:53.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.111+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f03700d0c30 con 0x7f03700d10c0 2024-03-06T07:25:53.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.682+0000 7f0365ffb640 1 -- 172.21.15.122:0/616536550 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0368058630 con 0x7f03700d10c0 2024-03-06T07:25:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f035406b4d0 msgr2=0x7f035406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f035406b4d0 0x7f035406d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0360036000 tx=0x7f036003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 msgr2=0x7f03700ceb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 0x7f03700ceb20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f036800c7f0 tx=0x7f036800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 shutdown_connections 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f035406b4d0 0x7f035406d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f037015eb00 0x7f03701f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03700d14a0 0x7f03700b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 --2- 172.21.15.122:0/616536550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700d10c0 0x7f03700ceb20 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 >> 172.21.15.122:0/616536550 conn(0x7f03700b3800 msgr2=0x7f03700b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 shutdown_connections 2024-03-06T07:25:53.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.684+0000 7f037741c640 1 -- 172.21.15.122:0/616536550 wait complete. 2024-03-06T07:25:53.697 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:53.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:53.772 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:53.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:53.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.795+0000 7f7aa93f6640 1 Processor -- start 2024-03-06T07:25:53.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.795+0000 7f7aa93f6640 1 -- start start 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa93f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7aa4158600 0x7f7aa4072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa93f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7aa4162030 0x7f7aa4162480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa93f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa415eb40 con 0x7f7aa415e3a0 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa93f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa4161ba0 con 0x7f7aa4158600 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa93f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa415ee20 con 0x7f7aa4162030 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7aa4158600 0x7f7aa4072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7a9bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7aa415e3a0 0x7f7aa406d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47590/0 (socket says 172.21.15.122:47590) 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7a9bfff640 1 -- 172.21.15.122:0/2266743700 learned_addr learned my addr 172.21.15.122:0/2266743700 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.796+0000 7f7aa37fe640 1 --2- 172.21.15.122:0/2266743700 >> v2:172.21.15.169:3300/0 conn(0x7f7aa4162030 0x7f7aa4162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.797+0000 7f7aa27fc640 1 -- 172.21.15.122:0/2266743700 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784754149 0 0) 0x7f7aa415eb40 con 0x7f7aa415e3a0 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.797+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/2266743700 >> v2:172.21.15.169:3300/0 conn(0x7f7aa4162030 msgr2=0x7f7aa4162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.797+0000 7f7aa2ffd640 1 --2- 172.21.15.122:0/2266743700 >> v2:172.21.15.169:3300/0 conn(0x7f7aa4162030 0x7f7aa4162480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.797+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/2266743700 >> v1:172.21.15.122:6789/0 conn(0x7f7aa415e3a0 legacy=0x7f7aa406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.797+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/2266743700 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa415f100 con 0x7f7aa4158600 2024-03-06T07:25:53.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.798+0000 7f7aa2ffd640 1 --2- 172.21.15.122:0/2266743700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4072730 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4161d20 tx=0x7f7a9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fdf23098572cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.798+0000 7f7aa27fc640 1 -- 172.21.15.122:0/2266743700 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940315a0 con 0x7f7aa4158600 2024-03-06T07:25:53.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.798+0000 7f7aa27fc640 1 -- 172.21.15.122:0/2266743700 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a94037070 con 0x7f7aa4158600 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.798+0000 7f7aa27fc640 1 -- 172.21.15.122:0/2266743700 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94031960 con 0x7f7aa4158600 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 -- 172.21.15.122:0/2266743700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 msgr2=0x7f7aa4072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/2266743700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4072730 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4161d20 tx=0x7f7a9402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 -- 172.21.15.122:0/2266743700 shutdown_connections 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/2266743700 >> v2:172.21.15.169:3300/0 conn(0x7f7aa4162030 0x7f7aa4162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/2266743700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4072730 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 -- 172.21.15.122:0/2266743700 >> 172.21.15.122:0/2266743700 conn(0x7f7aa406c0c0 msgr2=0x7f7aa406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 -- 172.21.15.122:0/2266743700 shutdown_connections 2024-03-06T07:25:53.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.799+0000 7f7aa93f6640 1 -- 172.21.15.122:0/2266743700 wait complete. 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.800+0000 7f7aa93f6640 1 Processor -- start 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.800+0000 7f7aa93f6640 1 -- start start 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa93f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa93f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aa415e3a0 0x7f7aa41e9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa93f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7aa4162030 0x7f7aa415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa93f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa406e080 con 0x7f7aa4158600 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa93f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa406afb0 con 0x7f7aa415e3a0 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa93f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa41e73a0 con 0x7f7aa4162030 2024-03-06T07:25:53.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51168/0 (socket says 172.21.15.122:51168) 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/1098862158 learned_addr learned my addr 172.21.15.122:0/1098862158 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa37fe640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7aa4162030 0x7f7aa415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7a9bfff640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aa415e3a0 0x7f7aa41e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7aa4162030 msgr2=0x7f7aa415a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7aa4162030 0x7f7aa415a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aa415e3a0 msgr2=0x7f7aa41e9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aa415e3a0 0x7f7aa41e9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.801+0000 7f7aa2ffd640 1 -- 172.21.15.122:0/1098862158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa4158f50 con 0x7f7aa4158600 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.802+0000 7f7aa2ffd640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4158a10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f7a94000c00 tx=0x7f7a940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.802+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94032630 con 0x7f7aa4158600 2024-03-06T07:25:53.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.802+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a940327f0 con 0x7f7aa4158600 2024-03-06T07:25:53.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.802+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94032be0 con 0x7f7aa4158600 2024-03-06T07:25:53.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.802+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa41e9da0 con 0x7f7aa4158600 2024-03-06T07:25:53.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.802+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa41e75f0 con 0x7f7aa4158600 2024-03-06T07:25:53.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.803+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa404f330 con 0x7f7aa4158600 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.803+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a9404c020 con 0x7f7aa4158600 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.804+0000 7f7a9b7fe640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a8406b4d0 0x7f7a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.804+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a9402f020 con 0x7f7aa4158600 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.804+0000 7f7a9bfff640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a8406b4d0 0x7f7a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.804+0000 7f7a9bfff640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a8406b4d0 0x7f7a8406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40cf180 tx=0x7f7a8c007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:53.808+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a94040070 con 0x7f7aa4158600 2024-03-06T07:25:54.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.063+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa41e9fd0 con 0x7f7aa4158600 2024-03-06T07:25:54.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.063+0000 7f7a9b7fe640 1 -- 172.21.15.122:0/1098862158 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f7a9407a6c0 con 0x7f7aa4158600 2024-03-06T07:25:54.066 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:54.066 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:25:54.066 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:25:54.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a8406b4d0 msgr2=0x7f7a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a8406b4d0 0x7f7a8406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40cf180 tx=0x7f7a8c007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 msgr2=0x7f7aa4158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4158a10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f7a94000c00 tx=0x7f7a940046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 shutdown_connections 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a8406b4d0 0x7f7a8406d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7aa4162030 0x7f7aa415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7aa415e3a0 0x7f7aa41e9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 --2- 172.21.15.122:0/1098862158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7aa4158600 0x7f7aa4158a10 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 >> 172.21.15.122:0/1098862158 conn(0x7f7aa406c0c0 msgr2=0x7f7aa4074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 shutdown_connections 2024-03-06T07:25:54.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.065+0000 7f7aa93f6640 1 -- 172.21.15.122:0/1098862158 wait complete. 2024-03-06T07:25:54.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:54.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.177+0000 7fa8f226f640 1 Processor -- start 2024-03-06T07:25:54.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.177+0000 7fa8f226f640 1 -- start start 2024-03-06T07:25:54.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f226f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f226f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8ec0cf830 0x7fa8ec0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f226f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec0b3430 con 0x7fa8ec0d27f0 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f226f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0d2c50 con 0x7fa8ec0cf830 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f226f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b31e0 con 0x7fa8ec15dfe0 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8eb7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8ec0cf830 0x7fa8ec0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f080c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa8ec0d27f0 0x7fa8ec0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47596/0 (socket says 172.21.15.122:47596) 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8eb7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8ec0cf830 0x7fa8ec0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51184/0 (socket says 172.21.15.122:51184) 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8f080c640 1 -- 172.21.15.122:0/248500778 learned_addr learned my addr 172.21.15.122:0/248500778 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.178+0000 7fa8ebfff640 1 --2- 172.21.15.122:0/248500778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.179+0000 7fa8eb7fe640 1 -- 172.21.15.122:0/248500778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 msgr2=0x7fa8ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.179+0000 7fa8eb7fe640 1 --2- 172.21.15.122:0/248500778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.179+0000 7fa8eb7fe640 1 -- 172.21.15.122:0/248500778 >> v1:172.21.15.122:6789/0 conn(0x7fa8ec0d27f0 legacy=0x7fa8ec0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.179+0000 7fa8eb7fe640 1 -- 172.21.15.122:0/248500778 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0b3710 con 0x7fa8ec0cf830 2024-03-06T07:25:54.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.179+0000 7fa8eaffd640 1 -- 172.21.15.122:0/248500778 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441709387 0 0) 0x7fa8ec0b3430 con 0x7fa8ec0d27f0 2024-03-06T07:25:54.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.179+0000 7fa8eb7fe640 1 --2- 172.21.15.122:0/248500778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec0d22b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00097a0 tx=0x7fa8e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fe516dad303de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8eaffd640 1 -- 172.21.15.122:0/248500778 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0035020 con 0x7fa8ec0cf830 2024-03-06T07:25:54.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8eaffd640 1 -- 172.21.15.122:0/248500778 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0031830 con 0x7fa8ec0cf830 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8eaffd640 1 -- 172.21.15.122:0/248500778 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0031c00 con 0x7fa8ec0cf830 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8f226f640 1 -- 172.21.15.122:0/248500778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 msgr2=0x7fa8ec0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8f226f640 1 --2- 172.21.15.122:0/248500778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec0d22b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00097a0 tx=0x7fa8e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8f226f640 1 -- 172.21.15.122:0/248500778 shutdown_connections 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8f226f640 1 --2- 172.21.15.122:0/248500778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec0d22b0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8f226f640 1 --2- 172.21.15.122:0/248500778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.180+0000 7fa8f226f640 1 -- 172.21.15.122:0/248500778 >> 172.21.15.122:0/248500778 conn(0x7fa8ec0d3cd0 msgr2=0x7fa8ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.181+0000 7fa8f226f640 1 -- 172.21.15.122:0/248500778 shutdown_connections 2024-03-06T07:25:54.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.181+0000 7fa8f226f640 1 -- 172.21.15.122:0/248500778 wait complete. 2024-03-06T07:25:54.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.182+0000 7fa8f226f640 1 Processor -- start 2024-03-06T07:25:54.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.183+0000 7fa8f226f640 1 -- start start 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f226f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f226f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ec0d27f0 0x7fa8ec0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f226f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec14f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f226f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0d1c70 con 0x7fa8ec0cf830 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f226f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b3d60 con 0x7fa8ec0d27f0 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f226f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0d1df0 con 0x7fa8ec15dfe0 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51198/0 (socket says 172.21.15.122:51198) 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 -- 172.21.15.122:0/1869069273 learned_addr learned my addr 172.21.15.122:0/1869069273 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8f080c640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec14f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8eb7fe640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ec0d27f0 0x7fa8ec0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 -- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 msgr2=0x7fa8ec14f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec14f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 -- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ec0d27f0 msgr2=0x7fa8ec0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ec0d27f0 0x7fa8ec0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 -- 172.21.15.122:0/1869069273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0cb5f0 con 0x7fa8ec0cf830 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8ebfff640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec161130 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e870 tx=0x7fa8dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc017070 con 0x7fa8ec0cf830 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.184+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc00c940 con 0x7fa8ec0cf830 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.185+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec0cb8d0 con 0x7fa8ec0cf830 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.185+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00cd30 con 0x7fa8ec0cf830 2024-03-06T07:25:54.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.185+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec0cb2e0 con 0x7fa8ec0cf830 2024-03-06T07:25:54.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.186+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec04f330 con 0x7fa8ec0cf830 2024-03-06T07:25:54.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.186+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8dc01a070 con 0x7fa8ec0cf830 2024-03-06T07:25:54.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.186+0000 7fa8e8ff9640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8d806b4d0 0x7fa8d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.186+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8dc08a0e0 con 0x7fa8ec0cf830 2024-03-06T07:25:54.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.186+0000 7fa8eb7fe640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8d806b4d0 0x7fa8d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.187+0000 7fa8eb7fe640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8d806b4d0 0x7fa8d806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0036000 tx=0x7fa8e0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.191+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8dc058f60 con 0x7fa8ec0cf830 2024-03-06T07:25:54.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.434+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8ec0cbdf0 con 0x7fa8ec0cf830 2024-03-06T07:25:54.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.435+0000 7fa8e8ff9640 1 -- 172.21.15.122:0/1869069273 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16660 (secure 0 0 0) 0x7fa8dc058860 con 0x7fa8ec0cf830 2024-03-06T07:25:54.438 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:54.438 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":82,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:52.668124+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8d806b4d0 msgr2=0x7fa8d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8d806b4d0 0x7fa8d806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0036000 tx=0x7fa8e0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 msgr2=0x7fa8ec161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec161130 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e870 tx=0x7fa8dc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 shutdown_connections 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8d806b4d0 0x7fa8d806d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8ec15dfe0 0x7fa8ec14f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ec0d27f0 0x7fa8ec0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 --2- 172.21.15.122:0/1869069273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8ec0cf830 0x7fa8ec161130 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 >> 172.21.15.122:0/1869069273 conn(0x7fa8ec0d3cd0 msgr2=0x7fa8ec06ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 shutdown_connections 2024-03-06T07:25:54.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.437+0000 7fa8f226f640 1 -- 172.21.15.122:0/1869069273 wait complete. 2024-03-06T07:25:54.450 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:54.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:54.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.550+0000 7fcc5c17a640 1 Processor -- start 2024-03-06T07:25:54.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.550+0000 7fcc5c17a640 1 -- start start 2024-03-06T07:25:54.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5c17a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcc540cdd40 0x7fcc5406dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5c17a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc5406e500 0x7fcc540b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5c17a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5406eaa0 con 0x7fcc5406b9a0 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5c17a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540b4e60 con 0x7fcc540cdd40 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5c17a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5406ed80 con 0x7fcc5406e500 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59f16640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcc5406b9a0 0x7fcc5406bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59798/0 (socket says 172.21.15.122:59798) 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59f16640 1 -- 172.21.15.122:0/833911950 learned_addr learned my addr 172.21.15.122:0/833911950 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59715640 1 --2- 172.21.15.122:0/833911950 >> v2:172.21.15.169:3300/0 conn(0x7fcc540cdd40 0x7fcc5406dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5a717640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc5406e500 0x7fcc540b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc58f14640 1 -- 172.21.15.122:0/833911950 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602516087 0 0) 0x7fcc5406eaa0 con 0x7fcc5406b9a0 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc58f14640 1 -- 172.21.15.122:0/833911950 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc40003520 con 0x7fcc5406b9a0 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59715640 1 -- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc5406e500 msgr2=0x7fcc540b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59715640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc5406e500 0x7fcc540b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59715640 1 -- 172.21.15.122:0/833911950 >> v1:172.21.15.122:6789/0 conn(0x7fcc5406b9a0 legacy=0x7fcc5406bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc59715640 1 -- 172.21.15.122:0/833911950 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5406f060 con 0x7fcc540cdd40 2024-03-06T07:25:54.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.552+0000 7fcc5a717640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc5406e500 0x7fcc540b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:54.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.553+0000 7fcc59715640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc540cdd40 0x7fcc5406dfc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcc440097a0 tx=0x7fcc4402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=322d739e434e3e16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.553+0000 7fcc58f14640 1 -- 172.21.15.122:0/833911950 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44035020 con 0x7fcc540cdd40 2024-03-06T07:25:54.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.553+0000 7fcc58f14640 1 -- 172.21.15.122:0/833911950 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44031830 con 0x7fcc540cdd40 2024-03-06T07:25:54.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.553+0000 7fcc58f14640 1 -- 172.21.15.122:0/833911950 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44031be0 con 0x7fcc540cdd40 2024-03-06T07:25:54.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 -- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc540cdd40 msgr2=0x7fcc5406dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc540cdd40 0x7fcc5406dfc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcc440097a0 tx=0x7fcc4402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 -- 172.21.15.122:0/833911950 shutdown_connections 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc5406e500 0x7fcc540b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/833911950 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc540cdd40 0x7fcc5406dfc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 -- 172.21.15.122:0/833911950 >> 172.21.15.122:0/833911950 conn(0x7fcc540b1d00 msgr2=0x7fcc540b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 -- 172.21.15.122:0/833911950 shutdown_connections 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.554+0000 7fcc5c17a640 1 -- 172.21.15.122:0/833911950 wait complete. 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.555+0000 7fcc5c17a640 1 Processor -- start 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.555+0000 7fcc5c17a640 1 -- start start 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5406b9a0 0x7fcc540994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 0x7fcc540ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc540cdd40 0x7fcc54153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5415b810 con 0x7fcc5406e500 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5415b690 con 0x7fcc5406b9a0 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5406b4a0 con 0x7fcc540cdd40 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 0x7fcc540ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 0x7fcc540ae7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59368/0 (socket says 172.21.15.122:59368) 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 -- 172.21.15.122:0/1850251603 learned_addr learned my addr 172.21.15.122:0/1850251603 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59f16640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5406b9a0 0x7fcc540994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5a717640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc540cdd40 0x7fcc54153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 -- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc540cdd40 msgr2=0x7fcc54153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc540cdd40 0x7fcc54153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 -- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5406b9a0 msgr2=0x7fcc540994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5406b9a0 0x7fcc540994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 -- 172.21.15.122:0/1850251603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc541562f0 con 0x7fcc5406e500 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5a717640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc540cdd40 0x7fcc54153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc59715640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 0x7fcc540ae7f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44035050 con 0x7fcc5406e500 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4403e070 con 0x7fcc5406e500 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.556+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc541565d0 con 0x7fcc5406e500 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.557+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44003610 con 0x7fcc5406e500 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.557+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc54156b90 con 0x7fcc5406e500 2024-03-06T07:25:54.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.557+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5404f330 con 0x7fcc5406e500 2024-03-06T07:25:54.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.558+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc44038020 con 0x7fcc5406e500 2024-03-06T07:25:54.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.558+0000 7fcc4a7fc640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc3806b4d0 0x7fcc3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.558+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc440abbc0 con 0x7fcc5406e500 2024-03-06T07:25:54.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.558+0000 7fcc59f16640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc3806b4d0 0x7fcc3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.559+0000 7fcc59f16640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc3806b4d0 0x7fcc3806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcc50005c00 tx=0x7fcc50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.563+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc44031800 con 0x7fcc5406e500 2024-03-06T07:25:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.739+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc54072650 con 0x7fcc3806b4d0 2024-03-06T07:25:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.745+0000 7fcc4a7fc640 1 -- 172.21.15.122:0/1850251603 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240168 (secure 0 0 0) 0x7fcc54072650 con 0x7fcc3806b4d0 2024-03-06T07:25:54.749 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc3806b4d0 msgr2=0x7fcc3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc3806b4d0 0x7fcc3806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcc50005c00 tx=0x7fcc50009040 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 msgr2=0x7fcc540ae7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 0x7fcc540ae7f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc440046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 shutdown_connections 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc3806b4d0 0x7fcc3806d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc540cdd40 0x7fcc54153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc5406e500 0x7fcc540ae7f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 --2- 172.21.15.122:0/1850251603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc5406b9a0 0x7fcc540994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 >> 172.21.15.122:0/1850251603 conn(0x7fcc540b1d00 msgr2=0x7fcc54072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 shutdown_connections 2024-03-06T07:25:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.748+0000 7fcc5c17a640 1 -- 172.21.15.122:0/1850251603 wait complete. 2024-03-06T07:25:54.760 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-03-06T07:25:53.378242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258200,"kb_used_data":8732,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716520,"statfs":{"total":773094113280,"available":772829716480,"internally_reserved":0,"allocated":8941568,"data_stored":4441455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000797"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647015+0000","last_change":"2024-03-06T07:25:49.647015+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647015+0000","last_undegraded":"2024-03-06T07:25:49.647015+0000","last_fullsized":"2024-03-06T07:25:49.647015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647946+0000","last_change":"2024-03-06T07:25:49.647946+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647946+0000","last_undegraded":"2024-03-06T07:25:49.647946+0000","last_fullsized":"2024-03-06T07:25:49.647946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650634+0000","last_change":"2024-03-06T07:25:49.650634+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650634+0000","last_undegraded":"2024-03-06T07:25:49.650634+0000","last_fullsized":"2024-03-06T07:25:49.650634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650144+0000","last_change":"2024-03-06T07:25:49.650144+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650144+0000","last_undegraded":"2024-03-06T07:25:49.650144+0000","last_fullsized":"2024-03-06T07:25:49.650144+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648056+0000","last_change":"2024-03-06T07:25:49.648056+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648056+0000","last_undegraded":"2024-03-06T07:25:49.648056+0000","last_fullsized":"2024-03-06T07:25:49.648056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648675+0000","last_change":"2024-03-06T07:25:49.648675+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648675+0000","last_undegraded":"2024-03-06T07:25:49.648675+0000","last_fullsized":"2024-03-06T07:25:49.648675+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650020+0000","last_change":"2024-03-06T07:25:49.650020+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650020+0000","last_undegraded":"2024-03-06T07:25:49.650020+0000","last_fullsized":"2024-03-06T07:25:49.650020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645899+0000","last_change":"2024-03-06T07:25:49.645899+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645899+0000","last_undegraded":"2024-03-06T07:25:49.645899+0000","last_fullsized":"2024-03-06T07:25:49.645899+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.646523+0000","last_change":"2024-03-06T07:25:49.646523+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.646523+0000","last_undegraded":"2024-03-06T07:25:49.646523+0000","last_fullsized":"2024-03-06T07:25:49.646523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645845+0000","last_change":"2024-03-06T07:25:49.645845+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645845+0000","last_undegraded":"2024-03-06T07:25:49.645845+0000","last_fullsized":"2024-03-06T07:25:49.645845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647343+0000","last_change":"2024-03-06T07:25:49.647343+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647343+0000","last_undegraded":"2024-03-06T07:25:49.647343+0000","last_fullsized":"2024-03-06T07:25:49.647343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649340+0000","last_change":"2024-03-06T07:25:49.649340+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649340+0000","last_undegraded":"2024-03-06T07:25:49.649340+0000","last_fullsized":"2024-03-06T07:25:49.649340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.646087+0000","last_change":"2024-03-06T07:25:49.646087+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.646087+0000","last_undegraded":"2024-03-06T07:25:49.646087+0000","last_fullsized":"2024-03-06T07:25:49.646087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649838+0000","last_change":"2024-03-06T07:25:49.649838+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649838+0000","last_undegraded":"2024-03-06T07:25:49.649838+0000","last_fullsized":"2024-03-06T07:25:49.649838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649317+0000","last_change":"2024-03-06T07:25:49.649317+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649317+0000","last_undegraded":"2024-03-06T07:25:49.649317+0000","last_fullsized":"2024-03-06T07:25:49.649317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650301+0000","last_change":"2024-03-06T07:25:49.650301+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650301+0000","last_undegraded":"2024-03-06T07:25:49.650301+0000","last_fullsized":"2024-03-06T07:25:49.650301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645800+0000","last_change":"2024-03-06T07:25:49.645800+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645800+0000","last_undegraded":"2024-03-06T07:25:49.645800+0000","last_fullsized":"2024-03-06T07:25:49.645800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648988+0000","last_change":"2024-03-06T07:25:49.648988+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648988+0000","last_undegraded":"2024-03-06T07:25:49.648988+0000","last_fullsized":"2024-03-06T07:25:49.648988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649014+0000","last_change":"2024-03-06T07:25:49.649014+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649014+0000","last_undegraded":"2024-03-06T07:25:49.649014+0000","last_fullsized":"2024-03-06T07:25:49.649014+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649421+0000","last_change":"2024-03-06T07:25:49.649421+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649421+0000","last_undegraded":"2024-03-06T07:25:49.649421+0000","last_fullsized":"2024-03-06T07:25:49.649421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648959+0000","last_change":"2024-03-06T07:25:49.648959+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648959+0000","last_undegraded":"2024-03-06T07:25:49.648959+0000","last_fullsized":"2024-03-06T07:25:49.648959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648943+0000","last_change":"2024-03-06T07:25:49.648943+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648943+0000","last_undegraded":"2024-03-06T07:25:49.648943+0000","last_fullsized":"2024-03-06T07:25:49.648943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648946+0000","last_change":"2024-03-06T07:25:49.648946+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648946+0000","last_undegraded":"2024-03-06T07:25:49.648946+0000","last_fullsized":"2024-03-06T07:25:49.648946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645916+0000","last_change":"2024-03-06T07:25:49.645916+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645916+0000","last_undegraded":"2024-03-06T07:25:49.645916+0000","last_fullsized":"2024-03-06T07:25:49.645916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645929+0000","last_change":"2024-03-06T07:25:49.645929+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645929+0000","last_undegraded":"2024-03-06T07:25:49.645929+0000","last_fullsized":"2024-03-06T07:25:49.645929+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645828+0000","last_change":"2024-03-06T07:25:49.645828+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645828+0000","last_undegraded":"2024-03-06T07:25:49.645828+0000","last_fullsized":"2024-03-06T07:25:49.645828+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645816+0000","last_change":"2024-03-06T07:25:49.645816+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645816+0000","last_undegraded":"2024-03-06T07:25:49.645816+0000","last_fullsized":"2024-03-06T07:25:49.645816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648943+0000","last_change":"2024-03-06T07:25:49.648943+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648943+0000","last_undegraded":"2024-03-06T07:25:49.648943+0000","last_fullsized":"2024-03-06T07:25:49.648943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648141+0000","last_change":"2024-03-06T07:25:49.648141+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648141+0000","last_undegraded":"2024-03-06T07:25:49.648141+0000","last_fullsized":"2024-03-06T07:25:49.648141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645825+0000","last_change":"2024-03-06T07:25:49.645825+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645825+0000","last_undegraded":"2024-03-06T07:25:49.645825+0000","last_fullsized":"2024-03-06T07:25:49.645825+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647888+0000","last_change":"2024-03-06T07:25:49.647888+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647888+0000","last_undegraded":"2024-03-06T07:25:49.647888+0000","last_fullsized":"2024-03-06T07:25:49.647888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647775+0000","last_change":"2024-03-06T07:25:49.647775+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647775+0000","last_undegraded":"2024-03-06T07:25:49.647775+0000","last_fullsized":"2024-03-06T07:25:49.647775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":280,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.648126+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:49.648126+0000","last_peered":"2024-03-06T07:25:49.648126+0000","last_clean":"2024-03-06T07:25:49.648126+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:49.648126+0000","last_undegraded":"2024-03-06T07:25:49.648126+0000","last_fullsized":"2024-03-06T07:25:49.648126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1552384,"data_stored":992868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":998685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:54.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:54.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.904+0000 7fd7b29c7640 1 Processor -- start 2024-03-06T07:25:54.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.904+0000 7fd7b29c7640 1 -- start start 2024-03-06T07:25:54.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b29c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 0x7fd7ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b29c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 0x7fd7ac0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b29c7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0b3430 con 0x7fd7ac0cf830 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b29c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ac0b31e0 con 0x7fd7ac15dfe0 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b29c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0b3d60 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b0f64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 0x7fd7ac0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7abfff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 0x7fd7ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7abfff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 0x7fd7ac0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:58714/0 (socket says 172.21.15.122:58714) 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7ab7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd7ac0cf830 0x7fd7ac0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:44492/0 (socket says 172.21.15.122:44492) 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b0f64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 0x7fd7ac0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:38564/0 (socket says 172.21.15.122:38564) 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7abfff640 1 -- 172.21.15.122:0/693650275 learned_addr learned my addr 172.21.15.122:0/693650275 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7aaffd640 1 -- 172.21.15.122:0/693650275 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662326703 0 0) 0x7fd7ac0b3430 con 0x7fd7ac0cf830 2024-03-06T07:25:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7aaffd640 1 -- 172.21.15.122:0/693650275 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd794003520 con 0x7fd7ac0cf830 2024-03-06T07:25:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b0f64640 1 -- 172.21.15.122:0/693650275 >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 msgr2=0x7fd7ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b0f64640 1 --2- 172.21.15.122:0/693650275 >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 0x7fd7ac0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b0f64640 1 -- 172.21.15.122:0/693650275 >> v1:172.21.15.169:6789/0 conn(0x7fd7ac0cf830 legacy=0x7fd7ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7b0f64640 1 -- 172.21.15.122:0/693650275 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac0b3710 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.905+0000 7fd7abfff640 1 --2- 172.21.15.122:0/693650275 >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 0x7fd7ac0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:54.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.906+0000 7fd7b0f64640 1 --2- 172.21.15.122:0/693650275 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 0x7fd7ac0b41f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4009d90 tx=0x7fd7a402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9beb353eee3f0cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.906+0000 7fd7aaffd640 1 -- 172.21.15.122:0/693650275 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4039070 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.906+0000 7fd7aaffd640 1 -- 172.21.15.122:0/693650275 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a402cc50 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7aaffd640 1 -- 172.21.15.122:0/693650275 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4034440 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 -- 172.21.15.122:0/693650275 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 msgr2=0x7fd7ac0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/693650275 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 0x7fd7ac0b41f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4009d90 tx=0x7fd7a402bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 -- 172.21.15.122:0/693650275 shutdown_connections 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/693650275 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac0d29c0 0x7fd7ac0b41f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/693650275 >> v2:172.21.15.169:3300/0 conn(0x7fd7ac15dfe0 0x7fd7ac0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 -- 172.21.15.122:0/693650275 >> 172.21.15.122:0/693650275 conn(0x7fd7ac0d3cd0 msgr2=0x7fd7ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 -- 172.21.15.122:0/693650275 shutdown_connections 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.907+0000 7fd7b29c7640 1 -- 172.21.15.122:0/693650275 wait complete. 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.908+0000 7fd7b29c7640 1 Processor -- start 2024-03-06T07:25:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.908+0000 7fd7b29c7640 1 -- start start 2024-03-06T07:25:54.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b29c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7ac0cf830 0x7fd7ac0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b29c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 0x7fd7ac14d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b29c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac15dfe0 0x7fd7ac1f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b29c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0d1df0 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b29c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0d1c70 con 0x7fd7ac0cf830 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b29c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac15e3c0 con 0x7fd7ac15dfe0 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 0x7fd7ac14d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 0x7fd7ac14d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59372/0 (socket says 172.21.15.122:59372) 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 -- 172.21.15.122:0/4291566689 learned_addr learned my addr 172.21.15.122:0/4291566689 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7b0f64640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac15dfe0 0x7fd7ac1f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7abfff640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7ac0cf830 0x7fd7ac0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 -- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac15dfe0 msgr2=0x7fd7ac1f7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac15dfe0 0x7fd7ac1f7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 -- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7ac0cf830 msgr2=0x7fd7ac0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7ac0cf830 0x7fd7ac0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.909+0000 7fd7ab7fe640 1 -- 172.21.15.122:0/4291566689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac1f79a0 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7b0f64640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac15dfe0 0x7fd7ac1f7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7abfff640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7ac0cf830 0x7fd7ac0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7ab7fe640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 0x7fd7ac14d790 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000c430 tx=0x7fd7a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00074e0 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a00047e0 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac1f7c80 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0004bd0 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.910+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac14d270 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.911+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac14d470 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.911+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7a001a020 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.912+0000 7fd7a8ff9640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd79806b4d0 0x7fd79806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.912+0000 7fd7abfff640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd79806b4d0 0x7fd79806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.912+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7a0089b00 con 0x7fd7ac0d29c0 2024-03-06T07:25:54.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.912+0000 7fd7abfff640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd79806b4d0 0x7fd79806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac06a360 tx=0x7fd79c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:54.916+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7ac14d470 con 0x7fd7ac0d29c0 2024-03-06T07:25:55.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.091+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7ac04f330 con 0x7fd79806b4d0 2024-03-06T07:25:55.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.097+0000 7fd7a8ff9640 1 -- 172.21.15.122:0/4291566689 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240168 (secure 0 0 0) 0x7fd7ac04f330 con 0x7fd79806b4d0 2024-03-06T07:25:55.101 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:55.102 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:55.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd79806b4d0 msgr2=0x7fd79806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd79806b4d0 0x7fd79806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac06a360 tx=0x7fd79c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 msgr2=0x7fd7ac14d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 0x7fd7ac14d790 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000c430 tx=0x7fd7a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 shutdown_connections 2024-03-06T07:25:55.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd79806b4d0 0x7fd79806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7ac15dfe0 0x7fd7ac1f7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7ac0d29c0 0x7fd7ac14d790 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 --2- 172.21.15.122:0/4291566689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7ac0cf830 0x7fd7ac0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 >> 172.21.15.122:0/4291566689 conn(0x7fd7ac0d3cd0 msgr2=0x7fd7ac0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 shutdown_connections 2024-03-06T07:25:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.100+0000 7fd7b29c7640 1 -- 172.21.15.122:0/4291566689 wait complete. 2024-03-06T07:25:55.113 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-03-06T07:25:53.378242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258200,"kb_used_data":8732,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716520,"statfs":{"total":773094113280,"available":772829716480,"internally_reserved":0,"allocated":8941568,"data_stored":4441455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000797"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647015+0000","last_change":"2024-03-06T07:25:49.647015+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647015+0000","last_undegraded":"2024-03-06T07:25:49.647015+0000","last_fullsized":"2024-03-06T07:25:49.647015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647946+0000","last_change":"2024-03-06T07:25:49.647946+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647946+0000","last_undegraded":"2024-03-06T07:25:49.647946+0000","last_fullsized":"2024-03-06T07:25:49.647946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650634+0000","last_change":"2024-03-06T07:25:49.650634+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650634+0000","last_undegraded":"2024-03-06T07:25:49.650634+0000","last_fullsized":"2024-03-06T07:25:49.650634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650144+0000","last_change":"2024-03-06T07:25:49.650144+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650144+0000","last_undegraded":"2024-03-06T07:25:49.650144+0000","last_fullsized":"2024-03-06T07:25:49.650144+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648056+0000","last_change":"2024-03-06T07:25:49.648056+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648056+0000","last_undegraded":"2024-03-06T07:25:49.648056+0000","last_fullsized":"2024-03-06T07:25:49.648056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648675+0000","last_change":"2024-03-06T07:25:49.648675+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648675+0000","last_undegraded":"2024-03-06T07:25:49.648675+0000","last_fullsized":"2024-03-06T07:25:49.648675+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650020+0000","last_change":"2024-03-06T07:25:49.650020+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650020+0000","last_undegraded":"2024-03-06T07:25:49.650020+0000","last_fullsized":"2024-03-06T07:25:49.650020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645899+0000","last_change":"2024-03-06T07:25:49.645899+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645899+0000","last_undegraded":"2024-03-06T07:25:49.645899+0000","last_fullsized":"2024-03-06T07:25:49.645899+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.646523+0000","last_change":"2024-03-06T07:25:49.646523+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.646523+0000","last_undegraded":"2024-03-06T07:25:49.646523+0000","last_fullsized":"2024-03-06T07:25:49.646523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645845+0000","last_change":"2024-03-06T07:25:49.645845+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645845+0000","last_undegraded":"2024-03-06T07:25:49.645845+0000","last_fullsized":"2024-03-06T07:25:49.645845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647343+0000","last_change":"2024-03-06T07:25:49.647343+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647343+0000","last_undegraded":"2024-03-06T07:25:49.647343+0000","last_fullsized":"2024-03-06T07:25:49.647343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649340+0000","last_change":"2024-03-06T07:25:49.649340+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649340+0000","last_undegraded":"2024-03-06T07:25:49.649340+0000","last_fullsized":"2024-03-06T07:25:49.649340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.646087+0000","last_change":"2024-03-06T07:25:49.646087+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.646087+0000","last_undegraded":"2024-03-06T07:25:49.646087+0000","last_fullsized":"2024-03-06T07:25:49.646087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649838+0000","last_change":"2024-03-06T07:25:49.649838+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649838+0000","last_undegraded":"2024-03-06T07:25:49.649838+0000","last_fullsized":"2024-03-06T07:25:49.649838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649317+0000","last_change":"2024-03-06T07:25:49.649317+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649317+0000","last_undegraded":"2024-03-06T07:25:49.649317+0000","last_fullsized":"2024-03-06T07:25:49.649317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.650301+0000","last_change":"2024-03-06T07:25:49.650301+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.650301+0000","last_undegraded":"2024-03-06T07:25:49.650301+0000","last_fullsized":"2024-03-06T07:25:49.650301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645800+0000","last_change":"2024-03-06T07:25:49.645800+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645800+0000","last_undegraded":"2024-03-06T07:25:49.645800+0000","last_fullsized":"2024-03-06T07:25:49.645800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648988+0000","last_change":"2024-03-06T07:25:49.648988+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648988+0000","last_undegraded":"2024-03-06T07:25:49.648988+0000","last_fullsized":"2024-03-06T07:25:49.648988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649014+0000","last_change":"2024-03-06T07:25:49.649014+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649014+0000","last_undegraded":"2024-03-06T07:25:49.649014+0000","last_fullsized":"2024-03-06T07:25:49.649014+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.649421+0000","last_change":"2024-03-06T07:25:49.649421+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.649421+0000","last_undegraded":"2024-03-06T07:25:49.649421+0000","last_fullsized":"2024-03-06T07:25:49.649421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648959+0000","last_change":"2024-03-06T07:25:49.648959+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648959+0000","last_undegraded":"2024-03-06T07:25:49.648959+0000","last_fullsized":"2024-03-06T07:25:49.648959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648943+0000","last_change":"2024-03-06T07:25:49.648943+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648943+0000","last_undegraded":"2024-03-06T07:25:49.648943+0000","last_fullsized":"2024-03-06T07:25:49.648943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648946+0000","last_change":"2024-03-06T07:25:49.648946+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648946+0000","last_undegraded":"2024-03-06T07:25:49.648946+0000","last_fullsized":"2024-03-06T07:25:49.648946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645916+0000","last_change":"2024-03-06T07:25:49.645916+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645916+0000","last_undegraded":"2024-03-06T07:25:49.645916+0000","last_fullsized":"2024-03-06T07:25:49.645916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645929+0000","last_change":"2024-03-06T07:25:49.645929+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645929+0000","last_undegraded":"2024-03-06T07:25:49.645929+0000","last_fullsized":"2024-03-06T07:25:49.645929+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645828+0000","last_change":"2024-03-06T07:25:49.645828+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645828+0000","last_undegraded":"2024-03-06T07:25:49.645828+0000","last_fullsized":"2024-03-06T07:25:49.645828+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645816+0000","last_change":"2024-03-06T07:25:49.645816+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645816+0000","last_undegraded":"2024-03-06T07:25:49.645816+0000","last_fullsized":"2024-03-06T07:25:49.645816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648943+0000","last_change":"2024-03-06T07:25:49.648943+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648943+0000","last_undegraded":"2024-03-06T07:25:49.648943+0000","last_fullsized":"2024-03-06T07:25:49.648943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.648141+0000","last_change":"2024-03-06T07:25:49.648141+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.648141+0000","last_undegraded":"2024-03-06T07:25:49.648141+0000","last_fullsized":"2024-03-06T07:25:49.648141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.645825+0000","last_change":"2024-03-06T07:25:49.645825+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.645825+0000","last_undegraded":"2024-03-06T07:25:49.645825+0000","last_fullsized":"2024-03-06T07:25:49.645825+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647888+0000","last_change":"2024-03-06T07:25:49.647888+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647888+0000","last_undegraded":"2024-03-06T07:25:49.647888+0000","last_fullsized":"2024-03-06T07:25:49.647888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:50.643283+0000","last_change":"2024-03-06T07:25:50.643283+0000","last_active":"2024-03-06T07:25:50.643283+0000","last_peered":"2024-03-06T07:25:50.643283+0000","last_clean":"2024-03-06T07:25:50.643283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:50.643283+0000","last_undegraded":"2024-03-06T07:25:50.643283+0000","last_fullsized":"2024-03-06T07:25:50.643283+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-03-06T07:25:49.647775+0000","last_change":"2024-03-06T07:25:49.647775+0000","last_active":"2024-03-06T07:25:49.637079+0000","last_peered":"2024-03-06T07:25:49.637079+0000","last_clean":"2024-03-06T07:25:49.637079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:49.647775+0000","last_undegraded":"2024-03-06T07:25:49.647775+0000","last_fullsized":"2024-03-06T07:25:49.647775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":280,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.648126+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:49.648126+0000","last_peered":"2024-03-06T07:25:49.648126+0000","last_clean":"2024-03-06T07:25:49.648126+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:49.648126+0000","last_undegraded":"2024-03-06T07:25:49.648126+0000","last_fullsized":"2024-03-06T07:25:49.648126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1552384,"data_stored":992868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":408317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":998685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:55.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:55.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.251+0000 7fcca5935640 1 Processor -- start 2024-03-06T07:25:55.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.251+0000 7fcca5935640 1 -- start start 2024-03-06T07:25:55.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcca5935640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcca015fa10 0x7fcca00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcca5935640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca00cebe0 con 0x7fcca00d16e0 2024-03-06T07:25:55.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcca5935640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca00ceec0 con 0x7fcca015bdd0 2024-03-06T07:25:55.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcca5935640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca00cf1a0 con 0x7fcca015fa10 2024-03-06T07:25:55.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcc9f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcca00d16e0 0x7fcca00d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59800/0 (socket says 172.21.15.122:59800) 2024-03-06T07:25:55.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcc9f7fe640 1 -- 172.21.15.122:0/938561021 learned_addr learned my addr 172.21.15.122:0/938561021 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcc967fc640 1 --2- 172.21.15.122:0/938561021 >> v2:172.21.15.169:3300/0 conn(0x7fcca015fa10 0x7fcca00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcc9e7fc640 1 -- 172.21.15.122:0/938561021 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732264932 0 0) 0x7fcca00cebe0 con 0x7fcca00d16e0 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.252+0000 7fcc9e7fc640 1 -- 172.21.15.122:0/938561021 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc8c003520 con 0x7fcca00d16e0 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.253+0000 7fcc967fc640 1 -- 172.21.15.122:0/938561021 >> v1:172.21.15.169:6789/0 conn(0x7fcca015bdd0 legacy=0x7fcca00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.253+0000 7fcc9e7fc640 1 -- 172.21.15.122:0/938561021 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953404433 0 0) 0x7fcca00ceec0 con 0x7fcca015bdd0 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.253+0000 7fcc967fc640 1 -- 172.21.15.122:0/938561021 >> v1:172.21.15.122:6789/0 conn(0x7fcca00d16e0 legacy=0x7fcca00d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.253+0000 7fcc967fc640 1 -- 172.21.15.122:0/938561021 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca00cc8e0 con 0x7fcca015fa10 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.253+0000 7fcc967fc640 1 --2- 172.21.15.122:0/938561021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015fa10 0x7fcca00ce270 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc880097a0 tx=0x7fcc8802b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=587081e622ffe172 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcc9e7fc640 1 -- 172.21.15.122:0/938561021 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88037070 con 0x7fcca015fa10 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcc9e7fc640 1 -- 172.21.15.122:0/938561021 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc88030510 con 0x7fcca015fa10 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcc9e7fc640 1 -- 172.21.15.122:0/938561021 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98001a10 con 0x7fcca015fa10 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 -- 172.21.15.122:0/938561021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015fa10 msgr2=0x7fcca00ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 --2- 172.21.15.122:0/938561021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015fa10 0x7fcca00ce270 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc880097a0 tx=0x7fcc8802b6b0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 -- 172.21.15.122:0/938561021 shutdown_connections 2024-03-06T07:25:55.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 --2- 172.21.15.122:0/938561021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015fa10 0x7fcca00ce270 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 -- 172.21.15.122:0/938561021 >> 172.21.15.122:0/938561021 conn(0x7fcca009a9f0 msgr2=0x7fcca009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 -- 172.21.15.122:0/938561021 shutdown_connections 2024-03-06T07:25:55.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.254+0000 7fcca5935640 1 -- 172.21.15.122:0/938561021 wait complete. 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.255+0000 7fcca5935640 1 Processor -- start 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.255+0000 7fcca5935640 1 -- start start 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcca5935640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 0x7fcca0154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcca5935640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015bdd0 0x7fcca015dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcca5935640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcca015fa10 0x7fcca015e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcca5935640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca00ce7b0 con 0x7fcca00d16e0 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcca5935640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca015c1b0 con 0x7fcca015bdd0 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcca5935640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca00ce930 con 0x7fcca015fa10 2024-03-06T07:25:55.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 0x7fcca0154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 0x7fcca0154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59378/0 (socket says 172.21.15.122:59378) 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 -- 172.21.15.122:0/346160853 learned_addr learned my addr 172.21.15.122:0/346160853 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9f7fe640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcca015fa10 0x7fcca015e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc967fc640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015bdd0 0x7fcca015dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 -- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcca015fa10 msgr2=0x7fcca015e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcca015fa10 0x7fcca015e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 -- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015bdd0 msgr2=0x7fcca015dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015bdd0 0x7fcca015dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.256+0000 7fcc9effd640 1 -- 172.21.15.122:0/346160853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca01554d0 con 0x7fcca00d16e0 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcc967fc640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015bdd0 0x7fcca015dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcc9effd640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 0x7fcca0154f90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000c4e0 tx=0x7fcc9000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc90017070 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc900035f0 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca01e6c00 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc900039e0 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.257+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca00b6040 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.258+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca00d0be0 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.258+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc9001a070 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.259+0000 7fcc9cff9640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.259+0000 7fcc967fc640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.259+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc900894e0 con 0x7fcca00d16e0 2024-03-06T07:25:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.259+0000 7fcc967fc640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcc88034000 tx=0x7fcc88008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.263+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcca00d0be0 con 0x7fcca00d16e0 2024-03-06T07:25:55.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.496+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcca004f330 con 0x7fcc8006b4d0 2024-03-06T07:25:55.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.502+0000 7fcc9cff9640 1 -- 172.21.15.122:0/346160853 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252079 (secure 0 0 0) 0x7fcca004f330 con 0x7fcc8006b4d0 2024-03-06T07:25:55.507 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:55.508 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc8006b4d0 msgr2=0x7fcc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcc88034000 tx=0x7fcc88008030 comp rx=0 tx=0).stop 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 msgr2=0x7fcca0154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 0x7fcca0154f90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000c4e0 tx=0x7fcc9000c9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 shutdown_connections 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcca015fa10 0x7fcca015e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcca015bdd0 0x7fcca015dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 --2- 172.21.15.122:0/346160853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcca00d16e0 0x7fcca0154f90 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 >> 172.21.15.122:0/346160853 conn(0x7fcca009a9f0 msgr2=0x7fcca00d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.505+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 shutdown_connections 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.506+0000 7fcca5935640 1 -- 172.21.15.122:0/346160853 wait complete. 2024-03-06T07:25:55.522 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-03-06T07:25:55.378537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258576,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716144,"statfs":{"total":773094113280,"available":772829331456,"internally_reserved":0,"allocated":9314304,"data_stored":4606892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.734795"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675857+0000","last_change":"2024-03-06T07:25:51.678317+0000","last_active":"2024-03-06T07:25:52.675857+0000","last_peered":"2024-03-06T07:25:52.675857+0000","last_clean":"2024-03-06T07:25:52.675857+0000","last_became_active":"2024-03-06T07:25:51.678048+0000","last_became_peered":"2024-03-06T07:25:51.678048+0000","last_unstale":"2024-03-06T07:25:52.675857+0000","last_undegraded":"2024-03-06T07:25:52.675857+0000","last_fullsized":"2024-03-06T07:25:52.675857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:25.036381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005035+0000","last_change":"2024-03-06T07:25:51.680521+0000","last_active":"2024-03-06T07:25:53.005035+0000","last_peered":"2024-03-06T07:25:53.005035+0000","last_clean":"2024-03-06T07:25:53.005035+0000","last_became_active":"2024-03-06T07:25:51.680252+0000","last_became_peered":"2024-03-06T07:25:51.680252+0000","last_unstale":"2024-03-06T07:25:53.005035+0000","last_undegraded":"2024-03-06T07:25:53.005035+0000","last_fullsized":"2024-03-06T07:25:53.005035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:46.814166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005336+0000","last_change":"2024-03-06T07:25:51.674847+0000","last_active":"2024-03-06T07:25:53.005336+0000","last_peered":"2024-03-06T07:25:53.005336+0000","last_clean":"2024-03-06T07:25:53.005336+0000","last_became_active":"2024-03-06T07:25:51.674004+0000","last_became_peered":"2024-03-06T07:25:51.674004+0000","last_unstale":"2024-03-06T07:25:53.005336+0000","last_undegraded":"2024-03-06T07:25:53.005336+0000","last_fullsized":"2024-03-06T07:25:53.005336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:53.438813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801245+0000","last_change":"2024-03-06T07:25:51.671416+0000","last_active":"2024-03-06T07:25:52.801245+0000","last_peered":"2024-03-06T07:25:52.801245+0000","last_clean":"2024-03-06T07:25:52.801245+0000","last_became_active":"2024-03-06T07:25:51.671151+0000","last_became_peered":"2024-03-06T07:25:51.671151+0000","last_unstale":"2024-03-06T07:25:52.801245+0000","last_undegraded":"2024-03-06T07:25:52.801245+0000","last_fullsized":"2024-03-06T07:25:52.801245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:48.621694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675343+0000","last_change":"2024-03-06T07:25:51.682772+0000","last_active":"2024-03-06T07:25:52.675343+0000","last_peered":"2024-03-06T07:25:52.675343+0000","last_clean":"2024-03-06T07:25:52.675343+0000","last_became_active":"2024-03-06T07:25:51.682504+0000","last_became_peered":"2024-03-06T07:25:51.682504+0000","last_unstale":"2024-03-06T07:25:52.675343+0000","last_undegraded":"2024-03-06T07:25:52.675343+0000","last_fullsized":"2024-03-06T07:25:52.675343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005031+0000","last_change":"2024-03-06T07:25:51.675616+0000","last_active":"2024-03-06T07:25:53.005031+0000","last_peered":"2024-03-06T07:25:53.005031+0000","last_clean":"2024-03-06T07:25:53.005031+0000","last_became_active":"2024-03-06T07:25:51.675306+0000","last_became_peered":"2024-03-06T07:25:51.675306+0000","last_unstale":"2024-03-06T07:25:53.005031+0000","last_undegraded":"2024-03-06T07:25:53.005031+0000","last_fullsized":"2024-03-06T07:25:53.005031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800816+0000","last_change":"2024-03-06T07:25:51.670665+0000","last_active":"2024-03-06T07:25:52.800816+0000","last_peered":"2024-03-06T07:25:52.800816+0000","last_clean":"2024-03-06T07:25:52.800816+0000","last_became_active":"2024-03-06T07:25:51.670401+0000","last_became_peered":"2024-03-06T07:25:51.670401+0000","last_unstale":"2024-03-06T07:25:52.800816+0000","last_undegraded":"2024-03-06T07:25:52.800816+0000","last_fullsized":"2024-03-06T07:25:52.800816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:29.492351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004988+0000","last_change":"2024-03-06T07:25:51.681909+0000","last_active":"2024-03-06T07:25:53.004988+0000","last_peered":"2024-03-06T07:25:53.004988+0000","last_clean":"2024-03-06T07:25:53.004988+0000","last_became_active":"2024-03-06T07:25:51.681317+0000","last_became_peered":"2024-03-06T07:25:51.681317+0000","last_unstale":"2024-03-06T07:25:53.004988+0000","last_undegraded":"2024-03-06T07:25:53.004988+0000","last_fullsized":"2024-03-06T07:25:53.004988+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005033+0000","last_change":"2024-03-06T07:25:51.682383+0000","last_active":"2024-03-06T07:25:53.005033+0000","last_peered":"2024-03-06T07:25:53.005033+0000","last_clean":"2024-03-06T07:25:53.005033+0000","last_became_active":"2024-03-06T07:25:51.682090+0000","last_became_peered":"2024-03-06T07:25:51.682090+0000","last_unstale":"2024-03-06T07:25:53.005033+0000","last_undegraded":"2024-03-06T07:25:53.005033+0000","last_fullsized":"2024-03-06T07:25:53.005033+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675059+0000","last_change":"2024-03-06T07:25:51.680536+0000","last_active":"2024-03-06T07:25:52.675059+0000","last_peered":"2024-03-06T07:25:52.675059+0000","last_clean":"2024-03-06T07:25:52.675059+0000","last_became_active":"2024-03-06T07:25:51.679636+0000","last_became_peered":"2024-03-06T07:25:51.679636+0000","last_unstale":"2024-03-06T07:25:52.675059+0000","last_undegraded":"2024-03-06T07:25:52.675059+0000","last_fullsized":"2024-03-06T07:25:52.675059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004844+0000","last_change":"2024-03-06T07:25:51.679595+0000","last_active":"2024-03-06T07:25:53.004844+0000","last_peered":"2024-03-06T07:25:53.004844+0000","last_clean":"2024-03-06T07:25:53.004844+0000","last_became_active":"2024-03-06T07:25:51.679267+0000","last_became_peered":"2024-03-06T07:25:51.679267+0000","last_unstale":"2024-03-06T07:25:53.004844+0000","last_undegraded":"2024-03-06T07:25:53.004844+0000","last_fullsized":"2024-03-06T07:25:53.004844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:36.580554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675672+0000","last_change":"2024-03-06T07:25:51.680345+0000","last_active":"2024-03-06T07:25:52.675672+0000","last_peered":"2024-03-06T07:25:52.675672+0000","last_clean":"2024-03-06T07:25:52.675672+0000","last_became_active":"2024-03-06T07:25:51.680024+0000","last_became_peered":"2024-03-06T07:25:51.680024+0000","last_unstale":"2024-03-06T07:25:52.675672+0000","last_undegraded":"2024-03-06T07:25:52.675672+0000","last_fullsized":"2024-03-06T07:25:52.675672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005403+0000","last_change":"2024-03-06T07:25:51.680932+0000","last_active":"2024-03-06T07:25:53.005403+0000","last_peered":"2024-03-06T07:25:53.005403+0000","last_clean":"2024-03-06T07:25:53.005403+0000","last_became_active":"2024-03-06T07:25:51.680638+0000","last_became_peered":"2024-03-06T07:25:51.680638+0000","last_unstale":"2024-03-06T07:25:53.005403+0000","last_undegraded":"2024-03-06T07:25:53.005403+0000","last_fullsized":"2024-03-06T07:25:53.005403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:48.659969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801037+0000","last_change":"2024-03-06T07:25:51.682488+0000","last_active":"2024-03-06T07:25:52.801037+0000","last_peered":"2024-03-06T07:25:52.801037+0000","last_clean":"2024-03-06T07:25:52.801037+0000","last_became_active":"2024-03-06T07:25:51.682199+0000","last_became_peered":"2024-03-06T07:25:51.682199+0000","last_unstale":"2024-03-06T07:25:52.801037+0000","last_undegraded":"2024-03-06T07:25:52.801037+0000","last_fullsized":"2024-03-06T07:25:52.801037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:04.113709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994217+0000","last_change":"2024-03-06T07:25:51.676566+0000","last_active":"2024-03-06T07:25:52.994217+0000","last_peered":"2024-03-06T07:25:52.994217+0000","last_clean":"2024-03-06T07:25:52.994217+0000","last_became_active":"2024-03-06T07:25:51.676318+0000","last_became_peered":"2024-03-06T07:25:51.676318+0000","last_unstale":"2024-03-06T07:25:52.994217+0000","last_undegraded":"2024-03-06T07:25:52.994217+0000","last_fullsized":"2024-03-06T07:25:52.994217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:53.438813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674313+0000","last_change":"2024-03-06T07:25:51.676990+0000","last_active":"2024-03-06T07:25:52.674313+0000","last_peered":"2024-03-06T07:25:52.674313+0000","last_clean":"2024-03-06T07:25:52.674313+0000","last_became_active":"2024-03-06T07:25:51.676709+0000","last_became_peered":"2024-03-06T07:25:51.676709+0000","last_unstale":"2024-03-06T07:25:52.674313+0000","last_undegraded":"2024-03-06T07:25:52.674313+0000","last_fullsized":"2024-03-06T07:25:52.674313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:21.738142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674764+0000","last_change":"2024-03-06T07:25:51.679322+0000","last_active":"2024-03-06T07:25:52.674764+0000","last_peered":"2024-03-06T07:25:52.674764+0000","last_clean":"2024-03-06T07:25:52.674764+0000","last_became_active":"2024-03-06T07:25:51.679022+0000","last_became_peered":"2024-03-06T07:25:51.679022+0000","last_unstale":"2024-03-06T07:25:52.674764+0000","last_undegraded":"2024-03-06T07:25:52.674764+0000","last_fullsized":"2024-03-06T07:25:52.674764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:33.451809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801077+0000","last_change":"2024-03-06T07:25:51.672332+0000","last_active":"2024-03-06T07:25:52.801077+0000","last_peered":"2024-03-06T07:25:52.801077+0000","last_clean":"2024-03-06T07:25:52.801077+0000","last_became_active":"2024-03-06T07:25:51.672086+0000","last_became_peered":"2024-03-06T07:25:51.672086+0000","last_unstale":"2024-03-06T07:25:52.801077+0000","last_undegraded":"2024-03-06T07:25:52.801077+0000","last_fullsized":"2024-03-06T07:25:52.801077+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:21.788094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671368+0000","last_change":"2024-03-06T07:25:51.680243+0000","last_active":"2024-03-06T07:25:52.671368+0000","last_peered":"2024-03-06T07:25:52.671368+0000","last_clean":"2024-03-06T07:25:52.671368+0000","last_became_active":"2024-03-06T07:25:51.679981+0000","last_became_peered":"2024-03-06T07:25:51.679981+0000","last_unstale":"2024-03-06T07:25:52.671368+0000","last_undegraded":"2024-03-06T07:25:52.671368+0000","last_fullsized":"2024-03-06T07:25:52.671368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:49.195508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801303+0000","last_change":"2024-03-06T07:25:51.681850+0000","last_active":"2024-03-06T07:25:52.801303+0000","last_peered":"2024-03-06T07:25:52.801303+0000","last_clean":"2024-03-06T07:25:52.801303+0000","last_became_active":"2024-03-06T07:25:51.681568+0000","last_became_peered":"2024-03-06T07:25:51.681568+0000","last_unstale":"2024-03-06T07:25:52.801303+0000","last_undegraded":"2024-03-06T07:25:52.801303+0000","last_fullsized":"2024-03-06T07:25:52.801303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:17.662481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801098+0000","last_change":"2024-03-06T07:25:51.681241+0000","last_active":"2024-03-06T07:25:52.801098+0000","last_peered":"2024-03-06T07:25:52.801098+0000","last_clean":"2024-03-06T07:25:52.801098+0000","last_became_active":"2024-03-06T07:25:51.680940+0000","last_became_peered":"2024-03-06T07:25:51.680940+0000","last_unstale":"2024-03-06T07:25:52.801098+0000","last_undegraded":"2024-03-06T07:25:52.801098+0000","last_fullsized":"2024-03-06T07:25:52.801098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:18.980000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004740+0000","last_change":"2024-03-06T07:25:51.681877+0000","last_active":"2024-03-06T07:25:53.004740+0000","last_peered":"2024-03-06T07:25:53.004740+0000","last_clean":"2024-03-06T07:25:53.004740+0000","last_became_active":"2024-03-06T07:25:51.681626+0000","last_became_peered":"2024-03-06T07:25:51.681626+0000","last_unstale":"2024-03-06T07:25:53.004740+0000","last_undegraded":"2024-03-06T07:25:53.004740+0000","last_fullsized":"2024-03-06T07:25:53.004740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:07.418991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674861+0000","last_change":"2024-03-06T07:25:51.683454+0000","last_active":"2024-03-06T07:25:52.674861+0000","last_peered":"2024-03-06T07:25:52.674861+0000","last_clean":"2024-03-06T07:25:52.674861+0000","last_became_active":"2024-03-06T07:25:51.682932+0000","last_became_peered":"2024-03-06T07:25:51.682932+0000","last_unstale":"2024-03-06T07:25:52.674861+0000","last_undegraded":"2024-03-06T07:25:52.674861+0000","last_fullsized":"2024-03-06T07:25:52.674861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800882+0000","last_change":"2024-03-06T07:25:51.684204+0000","last_active":"2024-03-06T07:25:52.800882+0000","last_peered":"2024-03-06T07:25:52.800882+0000","last_clean":"2024-03-06T07:25:52.800882+0000","last_became_active":"2024-03-06T07:25:51.683940+0000","last_became_peered":"2024-03-06T07:25:51.683940+0000","last_unstale":"2024-03-06T07:25:52.800882+0000","last_undegraded":"2024-03-06T07:25:52.800882+0000","last_fullsized":"2024-03-06T07:25:52.800882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:23.573251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004849+0000","last_change":"2024-03-06T07:25:51.680652+0000","last_active":"2024-03-06T07:25:53.004849+0000","last_peered":"2024-03-06T07:25:53.004849+0000","last_clean":"2024-03-06T07:25:53.004849+0000","last_became_active":"2024-03-06T07:25:51.680402+0000","last_became_peered":"2024-03-06T07:25:51.680402+0000","last_unstale":"2024-03-06T07:25:53.004849+0000","last_undegraded":"2024-03-06T07:25:53.004849+0000","last_fullsized":"2024-03-06T07:25:53.004849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:33.343720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005369+0000","last_change":"2024-03-06T07:25:51.682124+0000","last_active":"2024-03-06T07:25:53.005369+0000","last_peered":"2024-03-06T07:25:53.005369+0000","last_clean":"2024-03-06T07:25:53.005369+0000","last_became_active":"2024-03-06T07:25:51.681851+0000","last_became_peered":"2024-03-06T07:25:51.681851+0000","last_unstale":"2024-03-06T07:25:53.005369+0000","last_undegraded":"2024-03-06T07:25:53.005369+0000","last_fullsized":"2024-03-06T07:25:53.005369+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:55.427087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674874+0000","last_change":"2024-03-06T07:25:51.678817+0000","last_active":"2024-03-06T07:25:52.674874+0000","last_peered":"2024-03-06T07:25:52.674874+0000","last_clean":"2024-03-06T07:25:52.674874+0000","last_became_active":"2024-03-06T07:25:51.677774+0000","last_became_peered":"2024-03-06T07:25:51.677774+0000","last_unstale":"2024-03-06T07:25:52.674874+0000","last_undegraded":"2024-03-06T07:25:52.674874+0000","last_fullsized":"2024-03-06T07:25:52.674874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675705+0000","last_change":"2024-03-06T07:25:51.680054+0000","last_active":"2024-03-06T07:25:52.675705+0000","last_peered":"2024-03-06T07:25:52.675705+0000","last_clean":"2024-03-06T07:25:52.675705+0000","last_became_active":"2024-03-06T07:25:51.679787+0000","last_became_peered":"2024-03-06T07:25:51.679787+0000","last_unstale":"2024-03-06T07:25:52.675705+0000","last_undegraded":"2024-03-06T07:25:52.675705+0000","last_fullsized":"2024-03-06T07:25:52.675705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004704+0000","last_change":"2024-03-06T07:25:51.677373+0000","last_active":"2024-03-06T07:25:53.004704+0000","last_peered":"2024-03-06T07:25:53.004704+0000","last_clean":"2024-03-06T07:25:53.004704+0000","last_became_active":"2024-03-06T07:25:51.677137+0000","last_became_peered":"2024-03-06T07:25:51.677137+0000","last_unstale":"2024-03-06T07:25:53.004704+0000","last_undegraded":"2024-03-06T07:25:53.004704+0000","last_fullsized":"2024-03-06T07:25:53.004704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:24.300534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004894+0000","last_change":"2024-03-06T07:25:51.672433+0000","last_active":"2024-03-06T07:25:53.004894+0000","last_peered":"2024-03-06T07:25:53.004894+0000","last_clean":"2024-03-06T07:25:53.004894+0000","last_became_active":"2024-03-06T07:25:51.672152+0000","last_became_peered":"2024-03-06T07:25:51.672152+0000","last_unstale":"2024-03-06T07:25:53.004894+0000","last_undegraded":"2024-03-06T07:25:53.004894+0000","last_fullsized":"2024-03-06T07:25:53.004894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:45.422326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674713+0000","last_change":"2024-03-06T07:25:51.685203+0000","last_active":"2024-03-06T07:25:52.674713+0000","last_peered":"2024-03-06T07:25:52.674713+0000","last_clean":"2024-03-06T07:25:52.674713+0000","last_became_active":"2024-03-06T07:25:51.684174+0000","last_became_peered":"2024-03-06T07:25:51.684174+0000","last_unstale":"2024-03-06T07:25:52.674713+0000","last_undegraded":"2024-03-06T07:25:52.674713+0000","last_fullsized":"2024-03-06T07:25:52.674713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:18.482302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674638+0000","last_change":"2024-03-06T07:25:51.686828+0000","last_active":"2024-03-06T07:25:52.674638+0000","last_peered":"2024-03-06T07:25:52.674638+0000","last_clean":"2024-03-06T07:25:52.674638+0000","last_became_active":"2024-03-06T07:25:51.686529+0000","last_became_peered":"2024-03-06T07:25:51.686529+0000","last_unstale":"2024-03-06T07:25:52.674638+0000","last_undegraded":"2024-03-06T07:25:52.674638+0000","last_fullsized":"2024-03-06T07:25:52.674638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674836+0000","last_change":"2024-03-06T07:25:51.675516+0000","last_active":"2024-03-06T07:25:52.674836+0000","last_peered":"2024-03-06T07:25:52.674836+0000","last_clean":"2024-03-06T07:25:52.674836+0000","last_became_active":"2024-03-06T07:25:51.675166+0000","last_became_peered":"2024-03-06T07:25:51.675166+0000","last_unstale":"2024-03-06T07:25:52.674836+0000","last_undegraded":"2024-03-06T07:25:52.674836+0000","last_fullsized":"2024-03-06T07:25:52.674836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:27.077831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675927+0000","last_change":"2024-03-06T07:25:50.665724+0000","last_active":"2024-03-06T07:25:52.675927+0000","last_peered":"2024-03-06T07:25:52.675927+0000","last_clean":"2024-03-06T07:25:52.675927+0000","last_became_active":"2024-03-06T07:25:50.665446+0000","last_became_peered":"2024-03-06T07:25:50.665446+0000","last_unstale":"2024-03-06T07:25:52.675927+0000","last_undegraded":"2024-03-06T07:25:52.675927+0000","last_fullsized":"2024-03-06T07:25:52.675927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:35.574350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675869+0000","last_change":"2024-03-06T07:25:51.680584+0000","last_active":"2024-03-06T07:25:52.675869+0000","last_peered":"2024-03-06T07:25:52.675869+0000","last_clean":"2024-03-06T07:25:52.675869+0000","last_became_active":"2024-03-06T07:25:51.680260+0000","last_became_peered":"2024-03-06T07:25:51.680260+0000","last_unstale":"2024-03-06T07:25:52.675869+0000","last_undegraded":"2024-03-06T07:25:52.675869+0000","last_fullsized":"2024-03-06T07:25:52.675869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.712380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675077+0000","last_change":"2024-03-06T07:25:50.667936+0000","last_active":"2024-03-06T07:25:52.675077+0000","last_peered":"2024-03-06T07:25:52.675077+0000","last_clean":"2024-03-06T07:25:52.675077+0000","last_became_active":"2024-03-06T07:25:50.667577+0000","last_became_peered":"2024-03-06T07:25:50.667577+0000","last_unstale":"2024-03-06T07:25:52.675077+0000","last_undegraded":"2024-03-06T07:25:52.675077+0000","last_fullsized":"2024-03-06T07:25:52.675077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:41.743623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675771+0000","last_change":"2024-03-06T07:25:51.685616+0000","last_active":"2024-03-06T07:25:52.675771+0000","last_peered":"2024-03-06T07:25:52.675771+0000","last_clean":"2024-03-06T07:25:52.675771+0000","last_became_active":"2024-03-06T07:25:51.685322+0000","last_became_peered":"2024-03-06T07:25:51.685322+0000","last_unstale":"2024-03-06T07:25:52.675771+0000","last_undegraded":"2024-03-06T07:25:52.675771+0000","last_fullsized":"2024-03-06T07:25:52.675771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671402+0000","last_change":"2024-03-06T07:25:50.667866+0000","last_active":"2024-03-06T07:25:52.671402+0000","last_peered":"2024-03-06T07:25:52.671402+0000","last_clean":"2024-03-06T07:25:52.671402+0000","last_became_active":"2024-03-06T07:25:50.667614+0000","last_became_peered":"2024-03-06T07:25:50.667614+0000","last_unstale":"2024-03-06T07:25:52.671402+0000","last_undegraded":"2024-03-06T07:25:52.671402+0000","last_fullsized":"2024-03-06T07:25:52.671402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:55.559459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005458+0000","last_change":"2024-03-06T07:25:51.674998+0000","last_active":"2024-03-06T07:25:53.005458+0000","last_peered":"2024-03-06T07:25:53.005458+0000","last_clean":"2024-03-06T07:25:53.005458+0000","last_became_active":"2024-03-06T07:25:51.674720+0000","last_became_peered":"2024-03-06T07:25:51.674720+0000","last_unstale":"2024-03-06T07:25:53.005458+0000","last_undegraded":"2024-03-06T07:25:53.005458+0000","last_fullsized":"2024-03-06T07:25:53.005458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:23.266106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671229+0000","last_change":"2024-03-06T07:25:50.667000+0000","last_active":"2024-03-06T07:25:52.671229+0000","last_peered":"2024-03-06T07:25:52.671229+0000","last_clean":"2024-03-06T07:25:52.671229+0000","last_became_active":"2024-03-06T07:25:50.666740+0000","last_became_peered":"2024-03-06T07:25:50.666740+0000","last_unstale":"2024-03-06T07:25:52.671229+0000","last_undegraded":"2024-03-06T07:25:52.671229+0000","last_fullsized":"2024-03-06T07:25:52.671229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:37.494888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004961+0000","last_change":"2024-03-06T07:25:51.670449+0000","last_active":"2024-03-06T07:25:53.004961+0000","last_peered":"2024-03-06T07:25:53.004961+0000","last_clean":"2024-03-06T07:25:53.004961+0000","last_became_active":"2024-03-06T07:25:51.670204+0000","last_became_peered":"2024-03-06T07:25:51.670204+0000","last_unstale":"2024-03-06T07:25:53.004961+0000","last_undegraded":"2024-03-06T07:25:53.004961+0000","last_fullsized":"2024-03-06T07:25:53.004961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:09.510502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675637+0000","last_change":"2024-03-06T07:25:50.665732+0000","last_active":"2024-03-06T07:25:52.675637+0000","last_peered":"2024-03-06T07:25:52.675637+0000","last_clean":"2024-03-06T07:25:52.675637+0000","last_became_active":"2024-03-06T07:25:50.664895+0000","last_became_peered":"2024-03-06T07:25:50.664895+0000","last_unstale":"2024-03-06T07:25:52.675637+0000","last_undegraded":"2024-03-06T07:25:52.675637+0000","last_fullsized":"2024-03-06T07:25:52.675637+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:06.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675488+0000","last_change":"2024-03-06T07:25:51.685945+0000","last_active":"2024-03-06T07:25:52.675488+0000","last_peered":"2024-03-06T07:25:52.675488+0000","last_clean":"2024-03-06T07:25:52.675488+0000","last_became_active":"2024-03-06T07:25:51.685619+0000","last_became_peered":"2024-03-06T07:25:51.685619+0000","last_unstale":"2024-03-06T07:25:52.675488+0000","last_undegraded":"2024-03-06T07:25:52.675488+0000","last_fullsized":"2024-03-06T07:25:52.675488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674446+0000","last_change":"2024-03-06T07:25:50.665220+0000","last_active":"2024-03-06T07:25:52.674446+0000","last_peered":"2024-03-06T07:25:52.674446+0000","last_clean":"2024-03-06T07:25:52.674446+0000","last_became_active":"2024-03-06T07:25:50.664947+0000","last_became_peered":"2024-03-06T07:25:50.664947+0000","last_unstale":"2024-03-06T07:25:52.674446+0000","last_undegraded":"2024-03-06T07:25:52.674446+0000","last_fullsized":"2024-03-06T07:25:52.674446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:23.294330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004723+0000","last_change":"2024-03-06T07:25:51.681340+0000","last_active":"2024-03-06T07:25:53.004723+0000","last_peered":"2024-03-06T07:25:53.004723+0000","last_clean":"2024-03-06T07:25:53.004723+0000","last_became_active":"2024-03-06T07:25:51.681056+0000","last_became_peered":"2024-03-06T07:25:51.681056+0000","last_unstale":"2024-03-06T07:25:53.004723+0000","last_undegraded":"2024-03-06T07:25:53.004723+0000","last_fullsized":"2024-03-06T07:25:53.004723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:12.789933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.670990+0000","last_change":"2024-03-06T07:25:50.665789+0000","last_active":"2024-03-06T07:25:52.670990+0000","last_peered":"2024-03-06T07:25:52.670990+0000","last_clean":"2024-03-06T07:25:52.670990+0000","last_became_active":"2024-03-06T07:25:50.665358+0000","last_became_peered":"2024-03-06T07:25:50.665358+0000","last_unstale":"2024-03-06T07:25:52.670990+0000","last_undegraded":"2024-03-06T07:25:52.670990+0000","last_fullsized":"2024-03-06T07:25:52.670990+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:26.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994238+0000","last_change":"2024-03-06T07:25:51.674367+0000","last_active":"2024-03-06T07:25:52.994238+0000","last_peered":"2024-03-06T07:25:52.994238+0000","last_clean":"2024-03-06T07:25:52.994238+0000","last_became_active":"2024-03-06T07:25:51.674050+0000","last_became_peered":"2024-03-06T07:25:51.674050+0000","last_unstale":"2024-03-06T07:25:52.994238+0000","last_undegraded":"2024-03-06T07:25:52.994238+0000","last_fullsized":"2024-03-06T07:25:52.994238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:55.427087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004492+0000","last_change":"2024-03-06T07:25:50.664211+0000","last_active":"2024-03-06T07:25:53.004492+0000","last_peered":"2024-03-06T07:25:53.004492+0000","last_clean":"2024-03-06T07:25:53.004492+0000","last_became_active":"2024-03-06T07:25:50.663925+0000","last_became_peered":"2024-03-06T07:25:50.663925+0000","last_unstale":"2024-03-06T07:25:53.004492+0000","last_undegraded":"2024-03-06T07:25:53.004492+0000","last_fullsized":"2024-03-06T07:25:53.004492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:29.439664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994502+0000","last_change":"2024-03-06T07:25:51.681731+0000","last_active":"2024-03-06T07:25:52.994502+0000","last_peered":"2024-03-06T07:25:52.994502+0000","last_clean":"2024-03-06T07:25:52.994502+0000","last_became_active":"2024-03-06T07:25:51.681445+0000","last_became_peered":"2024-03-06T07:25:51.681445+0000","last_unstale":"2024-03-06T07:25:52.994502+0000","last_undegraded":"2024-03-06T07:25:52.994502+0000","last_fullsized":"2024-03-06T07:25:52.994502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:51.665911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.676005+0000","last_change":"2024-03-06T07:25:50.659033+0000","last_active":"2024-03-06T07:25:52.676005+0000","last_peered":"2024-03-06T07:25:52.676005+0000","last_clean":"2024-03-06T07:25:52.676005+0000","last_became_active":"2024-03-06T07:25:50.658341+0000","last_became_peered":"2024-03-06T07:25:50.658341+0000","last_unstale":"2024-03-06T07:25:52.676005+0000","last_undegraded":"2024-03-06T07:25:52.676005+0000","last_fullsized":"2024-03-06T07:25:52.676005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:44.416122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005205+0000","last_change":"2024-03-06T07:25:51.675730+0000","last_active":"2024-03-06T07:25:53.005205+0000","last_peered":"2024-03-06T07:25:53.005205+0000","last_clean":"2024-03-06T07:25:53.005205+0000","last_became_active":"2024-03-06T07:25:51.675375+0000","last_became_peered":"2024-03-06T07:25:51.675375+0000","last_unstale":"2024-03-06T07:25:53.005205+0000","last_undegraded":"2024-03-06T07:25:53.005205+0000","last_fullsized":"2024-03-06T07:25:53.005205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675117+0000","last_change":"2024-03-06T07:25:50.666708+0000","last_active":"2024-03-06T07:25:52.675117+0000","last_peered":"2024-03-06T07:25:52.675117+0000","last_clean":"2024-03-06T07:25:52.675117+0000","last_became_active":"2024-03-06T07:25:50.666385+0000","last_became_peered":"2024-03-06T07:25:50.666385+0000","last_unstale":"2024-03-06T07:25:52.675117+0000","last_undegraded":"2024-03-06T07:25:52.675117+0000","last_fullsized":"2024-03-06T07:25:52.675117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:05.567054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671255+0000","last_change":"2024-03-06T07:25:51.675676+0000","last_active":"2024-03-06T07:25:52.671255+0000","last_peered":"2024-03-06T07:25:52.671255+0000","last_clean":"2024-03-06T07:25:52.671255+0000","last_became_active":"2024-03-06T07:25:51.675431+0000","last_became_peered":"2024-03-06T07:25:51.675431+0000","last_unstale":"2024-03-06T07:25:52.671255+0000","last_undegraded":"2024-03-06T07:25:52.671255+0000","last_fullsized":"2024-03-06T07:25:52.671255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:52.903718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675236+0000","last_change":"2024-03-06T07:25:50.658006+0000","last_active":"2024-03-06T07:25:52.675236+0000","last_peered":"2024-03-06T07:25:52.675236+0000","last_clean":"2024-03-06T07:25:52.675236+0000","last_became_active":"2024-03-06T07:25:50.657716+0000","last_became_peered":"2024-03-06T07:25:50.657716+0000","last_unstale":"2024-03-06T07:25:52.675236+0000","last_undegraded":"2024-03-06T07:25:52.675236+0000","last_fullsized":"2024-03-06T07:25:52.675236+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:52.828417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674944+0000","last_change":"2024-03-06T07:25:51.686127+0000","last_active":"2024-03-06T07:25:52.674944+0000","last_peered":"2024-03-06T07:25:52.674944+0000","last_clean":"2024-03-06T07:25:52.674944+0000","last_became_active":"2024-03-06T07:25:51.685846+0000","last_became_peered":"2024-03-06T07:25:51.685846+0000","last_unstale":"2024-03-06T07:25:52.674944+0000","last_undegraded":"2024-03-06T07:25:52.674944+0000","last_fullsized":"2024-03-06T07:25:52.674944+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005222+0000","last_change":"2024-03-06T07:25:50.668266+0000","last_active":"2024-03-06T07:25:53.005222+0000","last_peered":"2024-03-06T07:25:53.005222+0000","last_clean":"2024-03-06T07:25:53.005222+0000","last_became_active":"2024-03-06T07:25:50.667485+0000","last_became_peered":"2024-03-06T07:25:50.667485+0000","last_unstale":"2024-03-06T07:25:53.005222+0000","last_undegraded":"2024-03-06T07:25:53.005222+0000","last_fullsized":"2024-03-06T07:25:53.005222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:25.507135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671140+0000","last_change":"2024-03-06T07:25:51.679153+0000","last_active":"2024-03-06T07:25:52.671140+0000","last_peered":"2024-03-06T07:25:52.671140+0000","last_clean":"2024-03-06T07:25:52.671140+0000","last_became_active":"2024-03-06T07:25:51.678289+0000","last_became_peered":"2024-03-06T07:25:51.678289+0000","last_unstale":"2024-03-06T07:25:52.671140+0000","last_undegraded":"2024-03-06T07:25:52.671140+0000","last_fullsized":"2024-03-06T07:25:52.671140+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:15.538278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004757+0000","last_change":"2024-03-06T07:25:50.665487+0000","last_active":"2024-03-06T07:25:53.004757+0000","last_peered":"2024-03-06T07:25:53.004757+0000","last_clean":"2024-03-06T07:25:53.004757+0000","last_became_active":"2024-03-06T07:25:50.665171+0000","last_became_peered":"2024-03-06T07:25:50.665171+0000","last_unstale":"2024-03-06T07:25:53.004757+0000","last_undegraded":"2024-03-06T07:25:53.004757+0000","last_fullsized":"2024-03-06T07:25:53.004757+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:21.303434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005362+0000","last_change":"2024-03-06T07:25:51.680433+0000","last_active":"2024-03-06T07:25:53.005362+0000","last_peered":"2024-03-06T07:25:53.005362+0000","last_clean":"2024-03-06T07:25:53.005362+0000","last_became_active":"2024-03-06T07:25:51.680124+0000","last_became_peered":"2024-03-06T07:25:51.680124+0000","last_unstale":"2024-03-06T07:25:53.005362+0000","last_undegraded":"2024-03-06T07:25:53.005362+0000","last_fullsized":"2024-03-06T07:25:53.005362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:51.665911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994254+0000","last_change":"2024-03-06T07:25:50.660720+0000","last_active":"2024-03-06T07:25:52.994254+0000","last_peered":"2024-03-06T07:25:52.994254+0000","last_clean":"2024-03-06T07:25:52.994254+0000","last_became_active":"2024-03-06T07:25:50.660452+0000","last_became_peered":"2024-03-06T07:25:50.660452+0000","last_unstale":"2024-03-06T07:25:52.994254+0000","last_undegraded":"2024-03-06T07:25:52.994254+0000","last_fullsized":"2024-03-06T07:25:52.994254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:02.606511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004561+0000","last_change":"2024-03-06T07:25:51.671565+0000","last_active":"2024-03-06T07:25:53.004561+0000","last_peered":"2024-03-06T07:25:53.004561+0000","last_clean":"2024-03-06T07:25:53.004561+0000","last_became_active":"2024-03-06T07:25:51.671341+0000","last_became_peered":"2024-03-06T07:25:51.671341+0000","last_unstale":"2024-03-06T07:25:53.004561+0000","last_undegraded":"2024-03-06T07:25:53.004561+0000","last_fullsized":"2024-03-06T07:25:53.004561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:53.834620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005007+0000","last_change":"2024-03-06T07:25:50.667664+0000","last_active":"2024-03-06T07:25:53.005007+0000","last_peered":"2024-03-06T07:25:53.005007+0000","last_clean":"2024-03-06T07:25:53.005007+0000","last_became_active":"2024-03-06T07:25:50.667347+0000","last_became_peered":"2024-03-06T07:25:50.667347+0000","last_unstale":"2024-03-06T07:25:53.005007+0000","last_undegraded":"2024-03-06T07:25:53.005007+0000","last_fullsized":"2024-03-06T07:25:53.005007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:36.841035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004529+0000","last_change":"2024-03-06T07:25:51.680371+0000","last_active":"2024-03-06T07:25:53.004529+0000","last_peered":"2024-03-06T07:25:53.004529+0000","last_clean":"2024-03-06T07:25:53.004529+0000","last_became_active":"2024-03-06T07:25:51.680102+0000","last_became_peered":"2024-03-06T07:25:51.680102+0000","last_unstale":"2024-03-06T07:25:53.004529+0000","last_undegraded":"2024-03-06T07:25:53.004529+0000","last_fullsized":"2024-03-06T07:25:53.004529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:42.749827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.670947+0000","last_change":"2024-03-06T07:25:50.663501+0000","last_active":"2024-03-06T07:25:52.670947+0000","last_peered":"2024-03-06T07:25:52.670947+0000","last_clean":"2024-03-06T07:25:52.670947+0000","last_became_active":"2024-03-06T07:25:50.663232+0000","last_became_peered":"2024-03-06T07:25:50.663232+0000","last_unstale":"2024-03-06T07:25:52.670947+0000","last_undegraded":"2024-03-06T07:25:52.670947+0000","last_fullsized":"2024-03-06T07:25:52.670947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:49.142433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005373+0000","last_change":"2024-03-06T07:25:51.675519+0000","last_active":"2024-03-06T07:25:53.005373+0000","last_peered":"2024-03-06T07:25:53.005373+0000","last_clean":"2024-03-06T07:25:53.005373+0000","last_became_active":"2024-03-06T07:25:51.675270+0000","last_became_peered":"2024-03-06T07:25:51.675270+0000","last_unstale":"2024-03-06T07:25:53.005373+0000","last_undegraded":"2024-03-06T07:25:53.005373+0000","last_fullsized":"2024-03-06T07:25:53.005373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:24.533672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.676059+0000","last_change":"2024-03-06T07:25:50.659845+0000","last_active":"2024-03-06T07:25:52.676059+0000","last_peered":"2024-03-06T07:25:52.676059+0000","last_clean":"2024-03-06T07:25:52.676059+0000","last_became_active":"2024-03-06T07:25:50.659527+0000","last_became_peered":"2024-03-06T07:25:50.659527+0000","last_unstale":"2024-03-06T07:25:52.676059+0000","last_undegraded":"2024-03-06T07:25:52.676059+0000","last_fullsized":"2024-03-06T07:25:52.676059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:32.337516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675974+0000","last_change":"2024-03-06T07:25:51.673226+0000","last_active":"2024-03-06T07:25:52.675974+0000","last_peered":"2024-03-06T07:25:52.675974+0000","last_clean":"2024-03-06T07:25:52.675974+0000","last_became_active":"2024-03-06T07:25:51.672946+0000","last_became_peered":"2024-03-06T07:25:51.672946+0000","last_unstale":"2024-03-06T07:25:52.675974+0000","last_undegraded":"2024-03-06T07:25:52.675974+0000","last_fullsized":"2024-03-06T07:25:52.675974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:15.538278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671294+0000","last_change":"2024-03-06T07:25:50.668525+0000","last_active":"2024-03-06T07:25:52.671294+0000","last_peered":"2024-03-06T07:25:52.671294+0000","last_clean":"2024-03-06T07:25:52.671294+0000","last_became_active":"2024-03-06T07:25:50.668291+0000","last_became_peered":"2024-03-06T07:25:50.668291+0000","last_unstale":"2024-03-06T07:25:52.671294+0000","last_undegraded":"2024-03-06T07:25:52.671294+0000","last_fullsized":"2024-03-06T07:25:52.671294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:31.002940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675006+0000","last_change":"2024-03-06T07:25:51.683425+0000","last_active":"2024-03-06T07:25:52.675006+0000","last_peered":"2024-03-06T07:25:52.675006+0000","last_clean":"2024-03-06T07:25:52.675006+0000","last_became_active":"2024-03-06T07:25:51.683168+0000","last_became_peered":"2024-03-06T07:25:51.683168+0000","last_unstale":"2024-03-06T07:25:52.675006+0000","last_undegraded":"2024-03-06T07:25:52.675006+0000","last_fullsized":"2024-03-06T07:25:52.675006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671310+0000","last_change":"2024-03-06T07:25:50.665805+0000","last_active":"2024-03-06T07:25:52.671310+0000","last_peered":"2024-03-06T07:25:52.671310+0000","last_clean":"2024-03-06T07:25:52.671310+0000","last_became_active":"2024-03-06T07:25:50.665533+0000","last_became_peered":"2024-03-06T07:25:50.665533+0000","last_unstale":"2024-03-06T07:25:52.671310+0000","last_undegraded":"2024-03-06T07:25:52.671310+0000","last_fullsized":"2024-03-06T07:25:52.671310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:32.208112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801202+0000","last_change":"2024-03-06T07:25:51.670633+0000","last_active":"2024-03-06T07:25:52.801202+0000","last_peered":"2024-03-06T07:25:52.801202+0000","last_clean":"2024-03-06T07:25:52.801202+0000","last_became_active":"2024-03-06T07:25:51.670383+0000","last_became_peered":"2024-03-06T07:25:51.670383+0000","last_unstale":"2024-03-06T07:25:52.801202+0000","last_undegraded":"2024-03-06T07:25:52.801202+0000","last_fullsized":"2024-03-06T07:25:52.801202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:30.342476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994321+0000","last_change":"2024-03-06T07:25:50.665180+0000","last_active":"2024-03-06T07:25:52.994321+0000","last_peered":"2024-03-06T07:25:52.994321+0000","last_clean":"2024-03-06T07:25:52.994321+0000","last_became_active":"2024-03-06T07:25:50.664910+0000","last_became_peered":"2024-03-06T07:25:50.664910+0000","last_unstale":"2024-03-06T07:25:52.994321+0000","last_undegraded":"2024-03-06T07:25:52.994321+0000","last_fullsized":"2024-03-06T07:25:52.994321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:52.993794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004875+0000","last_change":"2024-03-06T07:25:51.676865+0000","last_active":"2024-03-06T07:25:53.004875+0000","last_peered":"2024-03-06T07:25:53.004875+0000","last_clean":"2024-03-06T07:25:53.004875+0000","last_became_active":"2024-03-06T07:25:51.676514+0000","last_became_peered":"2024-03-06T07:25:51.676514+0000","last_unstale":"2024-03-06T07:25:53.004875+0000","last_undegraded":"2024-03-06T07:25:53.004875+0000","last_fullsized":"2024-03-06T07:25:53.004875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801155+0000","last_change":"2024-03-06T07:25:50.662536+0000","last_active":"2024-03-06T07:25:52.801155+0000","last_peered":"2024-03-06T07:25:52.801155+0000","last_clean":"2024-03-06T07:25:52.801155+0000","last_became_active":"2024-03-06T07:25:50.662248+0000","last_became_peered":"2024-03-06T07:25:50.662248+0000","last_unstale":"2024-03-06T07:25:52.801155+0000","last_undegraded":"2024-03-06T07:25:52.801155+0000","last_fullsized":"2024-03-06T07:25:52.801155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:26.050173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800815+0000","last_change":"2024-03-06T07:25:51.675362+0000","last_active":"2024-03-06T07:25:52.800815+0000","last_peered":"2024-03-06T07:25:52.800815+0000","last_clean":"2024-03-06T07:25:52.800815+0000","last_became_active":"2024-03-06T07:25:51.674936+0000","last_became_peered":"2024-03-06T07:25:51.674936+0000","last_unstale":"2024-03-06T07:25:52.800815+0000","last_undegraded":"2024-03-06T07:25:52.800815+0000","last_fullsized":"2024-03-06T07:25:52.800815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:16.568032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800780+0000","last_change":"2024-03-06T07:25:50.660498+0000","last_active":"2024-03-06T07:25:52.800780+0000","last_peered":"2024-03-06T07:25:52.800780+0000","last_clean":"2024-03-06T07:25:52.800780+0000","last_became_active":"2024-03-06T07:25:50.660246+0000","last_became_peered":"2024-03-06T07:25:50.660246+0000","last_unstale":"2024-03-06T07:25:52.800780+0000","last_undegraded":"2024-03-06T07:25:52.800780+0000","last_fullsized":"2024-03-06T07:25:52.800780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.278410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004907+0000","last_change":"2024-03-06T07:25:51.680574+0000","last_active":"2024-03-06T07:25:53.004907+0000","last_peered":"2024-03-06T07:25:53.004907+0000","last_clean":"2024-03-06T07:25:53.004907+0000","last_became_active":"2024-03-06T07:25:51.680281+0000","last_became_peered":"2024-03-06T07:25:51.680281+0000","last_unstale":"2024-03-06T07:25:53.004907+0000","last_undegraded":"2024-03-06T07:25:53.004907+0000","last_fullsized":"2024-03-06T07:25:53.004907+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994090+0000","last_change":"2024-03-06T07:25:50.666257+0000","last_active":"2024-03-06T07:25:52.994090+0000","last_peered":"2024-03-06T07:25:52.994090+0000","last_clean":"2024-03-06T07:25:52.994090+0000","last_became_active":"2024-03-06T07:25:50.664875+0000","last_became_peered":"2024-03-06T07:25:50.664875+0000","last_unstale":"2024-03-06T07:25:52.994090+0000","last_undegraded":"2024-03-06T07:25:52.994090+0000","last_fullsized":"2024-03-06T07:25:52.994090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.612866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674975+0000","last_change":"2024-03-06T07:25:51.678510+0000","last_active":"2024-03-06T07:25:52.674975+0000","last_peered":"2024-03-06T07:25:52.674975+0000","last_clean":"2024-03-06T07:25:52.674975+0000","last_became_active":"2024-03-06T07:25:51.678049+0000","last_became_peered":"2024-03-06T07:25:51.678049+0000","last_unstale":"2024-03-06T07:25:52.674975+0000","last_undegraded":"2024-03-06T07:25:52.674975+0000","last_fullsized":"2024-03-06T07:25:52.674975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004908+0000","last_change":"2024-03-06T07:25:50.664028+0000","last_active":"2024-03-06T07:25:53.004908+0000","last_peered":"2024-03-06T07:25:53.004908+0000","last_clean":"2024-03-06T07:25:53.004908+0000","last_became_active":"2024-03-06T07:25:50.663698+0000","last_became_peered":"2024-03-06T07:25:50.663698+0000","last_unstale":"2024-03-06T07:25:53.004908+0000","last_undegraded":"2024-03-06T07:25:53.004908+0000","last_fullsized":"2024-03-06T07:25:53.004908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:54.976768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005163+0000","last_change":"2024-03-06T07:25:51.674498+0000","last_active":"2024-03-06T07:25:53.005163+0000","last_peered":"2024-03-06T07:25:53.005163+0000","last_clean":"2024-03-06T07:25:53.005163+0000","last_became_active":"2024-03-06T07:25:51.673628+0000","last_became_peered":"2024-03-06T07:25:51.673628+0000","last_unstale":"2024-03-06T07:25:53.005163+0000","last_undegraded":"2024-03-06T07:25:53.005163+0000","last_fullsized":"2024-03-06T07:25:53.005163+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:41.203954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005010+0000","last_change":"2024-03-06T07:25:50.665322+0000","last_active":"2024-03-06T07:25:53.005010+0000","last_peered":"2024-03-06T07:25:53.005010+0000","last_clean":"2024-03-06T07:25:53.005010+0000","last_became_active":"2024-03-06T07:25:50.665051+0000","last_became_peered":"2024-03-06T07:25:50.665051+0000","last_unstale":"2024-03-06T07:25:53.005010+0000","last_undegraded":"2024-03-06T07:25:53.005010+0000","last_fullsized":"2024-03-06T07:25:53.005010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:03.252965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675860+0000","last_change":"2024-03-06T07:25:51.670848+0000","last_active":"2024-03-06T07:25:52.675860+0000","last_peered":"2024-03-06T07:25:52.675860+0000","last_clean":"2024-03-06T07:25:52.675860+0000","last_became_active":"2024-03-06T07:25:51.670601+0000","last_became_peered":"2024-03-06T07:25:51.670601+0000","last_unstale":"2024-03-06T07:25:52.675860+0000","last_undegraded":"2024-03-06T07:25:52.675860+0000","last_fullsized":"2024-03-06T07:25:52.675860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:42.061998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675408+0000","last_change":"2024-03-06T07:25:50.659022+0000","last_active":"2024-03-06T07:25:52.675408+0000","last_peered":"2024-03-06T07:25:52.675408+0000","last_clean":"2024-03-06T07:25:52.675408+0000","last_became_active":"2024-03-06T07:25:50.658737+0000","last_became_peered":"2024-03-06T07:25:50.658737+0000","last_unstale":"2024-03-06T07:25:52.675408+0000","last_undegraded":"2024-03-06T07:25:52.675408+0000","last_fullsized":"2024-03-06T07:25:52.675408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:41.743623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675106+0000","last_change":"2024-03-06T07:25:51.685417+0000","last_active":"2024-03-06T07:25:52.675106+0000","last_peered":"2024-03-06T07:25:52.675106+0000","last_clean":"2024-03-06T07:25:52.675106+0000","last_became_active":"2024-03-06T07:25:51.685153+0000","last_became_peered":"2024-03-06T07:25:51.685153+0000","last_unstale":"2024-03-06T07:25:52.675106+0000","last_undegraded":"2024-03-06T07:25:52.675106+0000","last_fullsized":"2024-03-06T07:25:52.675106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675812+0000","last_change":"2024-03-06T07:25:50.658414+0000","last_active":"2024-03-06T07:25:52.675812+0000","last_peered":"2024-03-06T07:25:52.675812+0000","last_clean":"2024-03-06T07:25:52.675812+0000","last_became_active":"2024-03-06T07:25:50.658117+0000","last_became_peered":"2024-03-06T07:25:50.658117+0000","last_unstale":"2024-03-06T07:25:52.675812+0000","last_undegraded":"2024-03-06T07:25:52.675812+0000","last_fullsized":"2024-03-06T07:25:52.675812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:06.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005316+0000","last_change":"2024-03-06T07:25:51.680659+0000","last_active":"2024-03-06T07:25:53.005316+0000","last_peered":"2024-03-06T07:25:53.005316+0000","last_clean":"2024-03-06T07:25:53.005316+0000","last_became_active":"2024-03-06T07:25:51.680365+0000","last_became_peered":"2024-03-06T07:25:51.680365+0000","last_unstale":"2024-03-06T07:25:53.005316+0000","last_undegraded":"2024-03-06T07:25:53.005316+0000","last_fullsized":"2024-03-06T07:25:53.005316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671168+0000","last_change":"2024-03-06T07:25:50.664527+0000","last_active":"2024-03-06T07:25:52.671168+0000","last_peered":"2024-03-06T07:25:52.671168+0000","last_clean":"2024-03-06T07:25:52.671168+0000","last_became_active":"2024-03-06T07:25:50.664117+0000","last_became_peered":"2024-03-06T07:25:50.664117+0000","last_unstale":"2024-03-06T07:25:52.671168+0000","last_undegraded":"2024-03-06T07:25:52.671168+0000","last_fullsized":"2024-03-06T07:25:52.671168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:15.731978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671218+0000","last_change":"2024-03-06T07:25:51.680138+0000","last_active":"2024-03-06T07:25:52.671218+0000","last_peered":"2024-03-06T07:25:52.671218+0000","last_clean":"2024-03-06T07:25:52.671218+0000","last_became_active":"2024-03-06T07:25:51.679882+0000","last_became_peered":"2024-03-06T07:25:51.679882+0000","last_unstale":"2024-03-06T07:25:52.671218+0000","last_undegraded":"2024-03-06T07:25:52.671218+0000","last_fullsized":"2024-03-06T07:25:52.671218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:17.142904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674507+0000","last_change":"2024-03-06T07:25:50.664868+0000","last_active":"2024-03-06T07:25:52.674507+0000","last_peered":"2024-03-06T07:25:52.674507+0000","last_clean":"2024-03-06T07:25:52.674507+0000","last_became_active":"2024-03-06T07:25:50.664449+0000","last_became_peered":"2024-03-06T07:25:50.664449+0000","last_unstale":"2024-03-06T07:25:52.674507+0000","last_undegraded":"2024-03-06T07:25:52.674507+0000","last_fullsized":"2024-03-06T07:25:52.674507+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:05.567054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005047+0000","last_change":"2024-03-06T07:25:51.681998+0000","last_active":"2024-03-06T07:25:53.005047+0000","last_peered":"2024-03-06T07:25:53.005047+0000","last_clean":"2024-03-06T07:25:53.005047+0000","last_became_active":"2024-03-06T07:25:51.681705+0000","last_became_peered":"2024-03-06T07:25:51.681705+0000","last_unstale":"2024-03-06T07:25:53.005047+0000","last_undegraded":"2024-03-06T07:25:53.005047+0000","last_fullsized":"2024-03-06T07:25:53.005047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:29.536548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675118+0000","last_change":"2024-03-06T07:25:50.657136+0000","last_active":"2024-03-06T07:25:52.675118+0000","last_peered":"2024-03-06T07:25:52.675118+0000","last_clean":"2024-03-06T07:25:52.675118+0000","last_became_active":"2024-03-06T07:25:50.656776+0000","last_became_peered":"2024-03-06T07:25:50.656776+0000","last_unstale":"2024-03-06T07:25:52.675118+0000","last_undegraded":"2024-03-06T07:25:52.675118+0000","last_fullsized":"2024-03-06T07:25:52.675118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:23.294330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675003+0000","last_change":"2024-03-06T07:25:51.670201+0000","last_active":"2024-03-06T07:25:52.675003+0000","last_peered":"2024-03-06T07:25:52.675003+0000","last_clean":"2024-03-06T07:25:52.675003+0000","last_became_active":"2024-03-06T07:25:51.669930+0000","last_became_peered":"2024-03-06T07:25:51.669930+0000","last_unstale":"2024-03-06T07:25:52.675003+0000","last_undegraded":"2024-03-06T07:25:52.675003+0000","last_fullsized":"2024-03-06T07:25:52.675003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674310+0000","last_change":"2024-03-06T07:25:50.666076+0000","last_active":"2024-03-06T07:25:52.674310+0000","last_peered":"2024-03-06T07:25:52.674310+0000","last_clean":"2024-03-06T07:25:52.674310+0000","last_became_active":"2024-03-06T07:25:50.665365+0000","last_became_peered":"2024-03-06T07:25:50.665365+0000","last_unstale":"2024-03-06T07:25:52.674310+0000","last_undegraded":"2024-03-06T07:25:52.674310+0000","last_fullsized":"2024-03-06T07:25:52.674310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:52.828417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004803+0000","last_change":"2024-03-06T07:25:51.674771+0000","last_active":"2024-03-06T07:25:53.004803+0000","last_peered":"2024-03-06T07:25:53.004803+0000","last_clean":"2024-03-06T07:25:53.004803+0000","last_became_active":"2024-03-06T07:25:51.674526+0000","last_became_peered":"2024-03-06T07:25:51.674526+0000","last_unstale":"2024-03-06T07:25:53.004803+0000","last_undegraded":"2024-03-06T07:25:53.004803+0000","last_fullsized":"2024-03-06T07:25:53.004803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:06.573258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005403+0000","last_change":"2024-03-06T07:25:50.665597+0000","last_active":"2024-03-06T07:25:53.005403+0000","last_peered":"2024-03-06T07:25:53.005403+0000","last_clean":"2024-03-06T07:25:53.005403+0000","last_became_active":"2024-03-06T07:25:50.665303+0000","last_became_peered":"2024-03-06T07:25:50.665303+0000","last_unstale":"2024-03-06T07:25:53.005403+0000","last_undegraded":"2024-03-06T07:25:53.005403+0000","last_fullsized":"2024-03-06T07:25:53.005403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:14.532074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":286,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005108+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:53.005108+0000","last_peered":"2024-03-06T07:25:53.005108+0000","last_clean":"2024-03-06T07:25:53.005108+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:53.005108+0000","last_undegraded":"2024-03-06T07:25:53.005108+0000","last_fullsized":"2024-03-06T07:25:53.005108+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1018105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":428361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":429301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":427667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":429279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1011712,"data_stored":428249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":427861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1018069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:55.527 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.661+0000 7f23babf6640 1 Processor -- start 2024-03-06T07:25:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.661+0000 7f23babf6640 1 -- start start 2024-03-06T07:25:55.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23babf6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23b415e000 0x7f23b40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23babf6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f23b40cf7d0 0x7f23b40d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23babf6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b40d25e0 con 0x7f23b40d4810 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23babf6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b40c7c50 con 0x7f23b415e000 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23babf6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b40d2880 con 0x7f23b40cf7d0 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b8992640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23b415e000 0x7f23b40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f23b40d4810 0x7f23b40c80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59802/0 (socket says 172.21.15.122:59802) 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b3fff640 1 -- 172.21.15.122:0/3822993037 learned_addr learned my addr 172.21.15.122:0/3822993037 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b8992640 1 -- 172.21.15.122:0/3822993037 >> v2:172.21.15.169:3300/0 conn(0x7f23b40cf7d0 msgr2=0x7f23b40d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b8992640 1 --2- 172.21.15.122:0/3822993037 >> v2:172.21.15.169:3300/0 conn(0x7f23b40cf7d0 0x7f23b40d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b8992640 1 -- 172.21.15.122:0/3822993037 >> v1:172.21.15.122:6789/0 conn(0x7f23b40d4810 legacy=0x7f23b40c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b37fe640 1 -- 172.21.15.122:0/3822993037 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801875308 0 0) 0x7f23b40d25e0 con 0x7f23b40d4810 2024-03-06T07:25:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.662+0000 7f23b8992640 1 -- 172.21.15.122:0/3822993037 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b40d2b40 con 0x7f23b415e000 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.663+0000 7f23b8992640 1 --2- 172.21.15.122:0/3822993037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40d0680 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f23b408df20 tx=0x7f23a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db606abe4081c815 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.663+0000 7f23b37fe640 1 -- 172.21.15.122:0/3822993037 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8034070 con 0x7f23b415e000 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.663+0000 7f23b37fe640 1 -- 172.21.15.122:0/3822993037 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8030610 con 0x7f23b415e000 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.663+0000 7f23b37fe640 1 -- 172.21.15.122:0/3822993037 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a80309e0 con 0x7f23b415e000 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 -- 172.21.15.122:0/3822993037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 msgr2=0x7f23b40d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 --2- 172.21.15.122:0/3822993037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40d0680 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f23b408df20 tx=0x7f23a802b960 comp rx=0 tx=0).stop 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 -- 172.21.15.122:0/3822993037 shutdown_connections 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 --2- 172.21.15.122:0/3822993037 >> v2:172.21.15.169:3300/0 conn(0x7f23b40cf7d0 0x7f23b40d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 --2- 172.21.15.122:0/3822993037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40d0680 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 -- 172.21.15.122:0/3822993037 >> 172.21.15.122:0/3822993037 conn(0x7f23b40d3cf0 msgr2=0x7f23b40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 -- 172.21.15.122:0/3822993037 shutdown_connections 2024-03-06T07:25:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.664+0000 7f23babf6640 1 -- 172.21.15.122:0/3822993037 wait complete. 2024-03-06T07:25:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.665+0000 7f23babf6640 1 Processor -- start 2024-03-06T07:25:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.665+0000 7f23babf6640 1 -- start start 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23babf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23b40cf7d0 0x7f23b40b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23babf6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23b40d4810 0x7f23b40cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23babf6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23babf6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b415e3e0 con 0x7f23b415e000 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23babf6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b40d22d0 con 0x7f23b40cf7d0 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23babf6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b40c87f0 con 0x7f23b40d4810 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40cd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59400/0 (socket says 172.21.15.122:59400) 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 -- 172.21.15.122:0/4162084554 learned_addr learned my addr 172.21.15.122:0/4162084554 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:55.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b3fff640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23b40d4810 0x7f23b40cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 -- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23b40d4810 msgr2=0x7f23b40cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b8992640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23b40cf7d0 0x7f23b40b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23b40d4810 0x7f23b40cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 -- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23b40cf7d0 msgr2=0x7f23b40b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23b40cf7d0 0x7f23b40b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 -- 172.21.15.122:0/4162084554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b40b2100 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b8992640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23b40cf7d0 0x7f23b40b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b3fff640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23b40d4810 0x7f23b40cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.666+0000 7f23b9193640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40cd9f0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00b970 tx=0x7f23ac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.667+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac019070 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.667+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00cb60 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.667+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b40b23e0 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.667+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac014440 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.667+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b40b29a0 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.668+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b404f330 con 0x7f23b415e000 2024-03-06T07:25:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.668+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23ac01c070 con 0x7f23b415e000 2024-03-06T07:25:55.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.668+0000 7f23b17fa640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f239806b500 0x7f239806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.668+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23ac089f00 con 0x7f23b415e000 2024-03-06T07:25:55.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.668+0000 7f23b8992640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f239806b500 0x7f239806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.669+0000 7f23b8992640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f239806b500 0x7f239806d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f23a8036000 tx=0x7f23a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.673+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23ac058d80 con 0x7f23b415e000 2024-03-06T07:25:55.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.920+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f23b40b4e80 con 0x7f23b415e000 2024-03-06T07:25:55.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.921+0000 7f23b17fa640 1 -- 172.21.15.122:0/4162084554 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7f23ac058680 con 0x7f23b415e000 2024-03-06T07:25:55.923 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:55.923 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":266,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":82,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":264781824,"bytes_avail":772829331456,"bytes_total":773094113280,"write_bytes_sec":2499,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":40,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":7041}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-06T07:25:41.376130+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:55.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.922+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f239806b500 msgr2=0x7f239806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.922+0000 7f23babf6640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f239806b500 0x7f239806d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f23a8036000 tx=0x7f23a8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.922+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 msgr2=0x7f23b40cd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.922+0000 7f23babf6640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40cd9f0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00b970 tx=0x7f23ac00be30 comp rx=0 tx=0).stop 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 shutdown_connections 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f239806b500 0x7f239806d9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23b415e000 0x7f23b40cd9f0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23b40d4810 0x7f23b40cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 --2- 172.21.15.122:0/4162084554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23b40cf7d0 0x7f23b40b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 >> 172.21.15.122:0/4162084554 conn(0x7f23b40d3cf0 msgr2=0x7f23b40b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 shutdown_connections 2024-03-06T07:25:55.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:55.923+0000 7f23babf6640 1 -- 172.21.15.122:0/4162084554 wait complete. 2024-03-06T07:25:55.935 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:55.936 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:56.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.046+0000 7fcb511e9640 1 Processor -- start 2024-03-06T07:25:56.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.046+0000 7fcb511e9640 1 -- start start 2024-03-06T07:25:56.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb511e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb511e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcb4c15bf30 0x7fcb4c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb511e9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcb4c15e850 0x7fcb4c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb511e9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c0cf3f0 con 0x7fcb4c15bf30 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb511e9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c15b210 con 0x7fcb4c15e850 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb511e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c15f580 con 0x7fcb4c13dc10 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcb4c15bf30 0x7fcb4c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcb4c15bf30 0x7fcb4c0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59404/0 (socket says 172.21.15.122:59404) 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 -- 172.21.15.122:0/3593826530 learned_addr learned my addr 172.21.15.122:0/3593826530 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4ad9d640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 -- 172.21.15.122:0/3593826530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 msgr2=0x7fcb4c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 -- 172.21.15.122:0/3593826530 >> v2:172.21.15.169:3300/0 conn(0x7fcb4c15e850 msgr2=0x7fcb4c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4b59e640 1 --2- 172.21.15.122:0/3593826530 >> v2:172.21.15.169:3300/0 conn(0x7fcb4c15e850 0x7fcb4c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 --2- 172.21.15.122:0/3593826530 >> v2:172.21.15.169:3300/0 conn(0x7fcb4c15e850 0x7fcb4c15eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4a59c640 1 -- 172.21.15.122:0/3593826530 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c15b450 con 0x7fcb4c15bf30 2024-03-06T07:25:56.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4ad9d640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.047+0000 7fcb4b59e640 1 --2- 172.21.15.122:0/3593826530 >> v2:172.21.15.169:3300/0 conn(0x7fcb4c15e850 0x7fcb4c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.048+0000 7fcb4a59c640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c0b2e90 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c06c790 tx=0x7fcb3402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ad38cd94244c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.048+0000 7fcb49d9b640 1 -- 172.21.15.122:0/3593826530 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3403b070 con 0x7fcb4c15bf30 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.048+0000 7fcb49d9b640 1 -- 172.21.15.122:0/3593826530 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3402ede0 con 0x7fcb4c15bf30 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.048+0000 7fcb49d9b640 1 -- 172.21.15.122:0/3593826530 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34036660 con 0x7fcb4c15bf30 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 -- 172.21.15.122:0/3593826530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 msgr2=0x7fcb4c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c0b2e90 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c06c790 tx=0x7fcb3402e030 comp rx=0 tx=0).stop 2024-03-06T07:25:56.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 -- 172.21.15.122:0/3593826530 shutdown_connections 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c0b2e90 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 --2- 172.21.15.122:0/3593826530 >> v2:172.21.15.169:3300/0 conn(0x7fcb4c15e850 0x7fcb4c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 --2- 172.21.15.122:0/3593826530 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 -- 172.21.15.122:0/3593826530 >> 172.21.15.122:0/3593826530 conn(0x7fcb4c0cf8a0 msgr2=0x7fcb4c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 -- 172.21.15.122:0/3593826530 shutdown_connections 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.049+0000 7fcb511e9640 1 -- 172.21.15.122:0/3593826530 wait complete. 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.050+0000 7fcb511e9640 1 Processor -- start 2024-03-06T07:25:56.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.050+0000 7fcb511e9640 1 -- start start 2024-03-06T07:25:56.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb511e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb511e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c135420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb511e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb4c15e850 0x7fcb4c15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb511e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c15f1e0 con 0x7fcb4c15bf30 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb511e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c13dff0 con 0x7fcb4c15e850 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb511e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c15f360 con 0x7fcb4c13dc10 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c135420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c135420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59412/0 (socket says 172.21.15.122:59412) 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 -- 172.21.15.122:0/2295371550 learned_addr learned my addr 172.21.15.122:0/2295371550 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4b59e640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb4c15e850 0x7fcb4c15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 -- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 msgr2=0x7fcb4c139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4ad9d640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 -- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb4c15e850 msgr2=0x7fcb4c15a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb4c15e850 0x7fcb4c15a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4a59c640 1 -- 172.21.15.122:0/2295371550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c15ab10 con 0x7fcb4c15bf30 2024-03-06T07:25:56.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.051+0000 7fcb4b59e640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb4c15e850 0x7fcb4c15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.052+0000 7fcb4a59c640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c135420 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fcb3402e070 tx=0x7fcb34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.052+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3403b030 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.052+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c13a500 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.052+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3403d050 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.052+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34034790 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.052+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c159dd0 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.053+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c04f330 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.053+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb34048060 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.053+0000 7fcb437fe640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.054+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb340aba40 con 0x7fcb4c15bf30 2024-03-06T07:25:56.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.054+0000 7fcb4ad9d640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.054+0000 7fcb4ad9d640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcb44008e60 tx=0x7fcb440083b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.060+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb3407a840 con 0x7fcb4c15bf30 2024-03-06T07:25:56.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.234+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb4c06eb00 con 0x7fcb2c06b4d0 2024-03-06T07:25:56.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.240+0000 7fcb437fe640 1 -- 172.21.15.122:0/2295371550 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252079 (secure 0 0 0) 0x7fcb4c06eb00 con 0x7fcb2c06b4d0 2024-03-06T07:25:56.244 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:56.245 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.242+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb2c06b4d0 msgr2=0x7fcb2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.242+0000 7fcb511e9640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcb44008e60 tx=0x7fcb440083b0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.242+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 msgr2=0x7fcb4c135420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.242+0000 7fcb511e9640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c135420 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fcb3402e070 tx=0x7fcb34004730 comp rx=0 tx=0).stop 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 shutdown_connections 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb4c15e850 0x7fcb4c15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb4c15bf30 0x7fcb4c135420 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 --2- 172.21.15.122:0/2295371550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb4c13dc10 0x7fcb4c139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 >> 172.21.15.122:0/2295371550 conn(0x7fcb4c0cf8a0 msgr2=0x7fcb4c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 shutdown_connections 2024-03-06T07:25:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:56.243+0000 7fcb511e9640 1 -- 172.21.15.122:0/2295371550 wait complete. 2024-03-06T07:25:56.256 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-03-06T07:25:55.378537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258576,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716144,"statfs":{"total":773094113280,"available":772829331456,"internally_reserved":0,"allocated":9314304,"data_stored":4606892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.734795"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675857+0000","last_change":"2024-03-06T07:25:51.678317+0000","last_active":"2024-03-06T07:25:52.675857+0000","last_peered":"2024-03-06T07:25:52.675857+0000","last_clean":"2024-03-06T07:25:52.675857+0000","last_became_active":"2024-03-06T07:25:51.678048+0000","last_became_peered":"2024-03-06T07:25:51.678048+0000","last_unstale":"2024-03-06T07:25:52.675857+0000","last_undegraded":"2024-03-06T07:25:52.675857+0000","last_fullsized":"2024-03-06T07:25:52.675857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:25.036381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005035+0000","last_change":"2024-03-06T07:25:51.680521+0000","last_active":"2024-03-06T07:25:53.005035+0000","last_peered":"2024-03-06T07:25:53.005035+0000","last_clean":"2024-03-06T07:25:53.005035+0000","last_became_active":"2024-03-06T07:25:51.680252+0000","last_became_peered":"2024-03-06T07:25:51.680252+0000","last_unstale":"2024-03-06T07:25:53.005035+0000","last_undegraded":"2024-03-06T07:25:53.005035+0000","last_fullsized":"2024-03-06T07:25:53.005035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:46.814166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005336+0000","last_change":"2024-03-06T07:25:51.674847+0000","last_active":"2024-03-06T07:25:53.005336+0000","last_peered":"2024-03-06T07:25:53.005336+0000","last_clean":"2024-03-06T07:25:53.005336+0000","last_became_active":"2024-03-06T07:25:51.674004+0000","last_became_peered":"2024-03-06T07:25:51.674004+0000","last_unstale":"2024-03-06T07:25:53.005336+0000","last_undegraded":"2024-03-06T07:25:53.005336+0000","last_fullsized":"2024-03-06T07:25:53.005336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:53.438813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801245+0000","last_change":"2024-03-06T07:25:51.671416+0000","last_active":"2024-03-06T07:25:52.801245+0000","last_peered":"2024-03-06T07:25:52.801245+0000","last_clean":"2024-03-06T07:25:52.801245+0000","last_became_active":"2024-03-06T07:25:51.671151+0000","last_became_peered":"2024-03-06T07:25:51.671151+0000","last_unstale":"2024-03-06T07:25:52.801245+0000","last_undegraded":"2024-03-06T07:25:52.801245+0000","last_fullsized":"2024-03-06T07:25:52.801245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:48.621694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675343+0000","last_change":"2024-03-06T07:25:51.682772+0000","last_active":"2024-03-06T07:25:52.675343+0000","last_peered":"2024-03-06T07:25:52.675343+0000","last_clean":"2024-03-06T07:25:52.675343+0000","last_became_active":"2024-03-06T07:25:51.682504+0000","last_became_peered":"2024-03-06T07:25:51.682504+0000","last_unstale":"2024-03-06T07:25:52.675343+0000","last_undegraded":"2024-03-06T07:25:52.675343+0000","last_fullsized":"2024-03-06T07:25:52.675343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005031+0000","last_change":"2024-03-06T07:25:51.675616+0000","last_active":"2024-03-06T07:25:53.005031+0000","last_peered":"2024-03-06T07:25:53.005031+0000","last_clean":"2024-03-06T07:25:53.005031+0000","last_became_active":"2024-03-06T07:25:51.675306+0000","last_became_peered":"2024-03-06T07:25:51.675306+0000","last_unstale":"2024-03-06T07:25:53.005031+0000","last_undegraded":"2024-03-06T07:25:53.005031+0000","last_fullsized":"2024-03-06T07:25:53.005031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800816+0000","last_change":"2024-03-06T07:25:51.670665+0000","last_active":"2024-03-06T07:25:52.800816+0000","last_peered":"2024-03-06T07:25:52.800816+0000","last_clean":"2024-03-06T07:25:52.800816+0000","last_became_active":"2024-03-06T07:25:51.670401+0000","last_became_peered":"2024-03-06T07:25:51.670401+0000","last_unstale":"2024-03-06T07:25:52.800816+0000","last_undegraded":"2024-03-06T07:25:52.800816+0000","last_fullsized":"2024-03-06T07:25:52.800816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:29.492351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004988+0000","last_change":"2024-03-06T07:25:51.681909+0000","last_active":"2024-03-06T07:25:53.004988+0000","last_peered":"2024-03-06T07:25:53.004988+0000","last_clean":"2024-03-06T07:25:53.004988+0000","last_became_active":"2024-03-06T07:25:51.681317+0000","last_became_peered":"2024-03-06T07:25:51.681317+0000","last_unstale":"2024-03-06T07:25:53.004988+0000","last_undegraded":"2024-03-06T07:25:53.004988+0000","last_fullsized":"2024-03-06T07:25:53.004988+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005033+0000","last_change":"2024-03-06T07:25:51.682383+0000","last_active":"2024-03-06T07:25:53.005033+0000","last_peered":"2024-03-06T07:25:53.005033+0000","last_clean":"2024-03-06T07:25:53.005033+0000","last_became_active":"2024-03-06T07:25:51.682090+0000","last_became_peered":"2024-03-06T07:25:51.682090+0000","last_unstale":"2024-03-06T07:25:53.005033+0000","last_undegraded":"2024-03-06T07:25:53.005033+0000","last_fullsized":"2024-03-06T07:25:53.005033+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675059+0000","last_change":"2024-03-06T07:25:51.680536+0000","last_active":"2024-03-06T07:25:52.675059+0000","last_peered":"2024-03-06T07:25:52.675059+0000","last_clean":"2024-03-06T07:25:52.675059+0000","last_became_active":"2024-03-06T07:25:51.679636+0000","last_became_peered":"2024-03-06T07:25:51.679636+0000","last_unstale":"2024-03-06T07:25:52.675059+0000","last_undegraded":"2024-03-06T07:25:52.675059+0000","last_fullsized":"2024-03-06T07:25:52.675059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004844+0000","last_change":"2024-03-06T07:25:51.679595+0000","last_active":"2024-03-06T07:25:53.004844+0000","last_peered":"2024-03-06T07:25:53.004844+0000","last_clean":"2024-03-06T07:25:53.004844+0000","last_became_active":"2024-03-06T07:25:51.679267+0000","last_became_peered":"2024-03-06T07:25:51.679267+0000","last_unstale":"2024-03-06T07:25:53.004844+0000","last_undegraded":"2024-03-06T07:25:53.004844+0000","last_fullsized":"2024-03-06T07:25:53.004844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:36.580554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675672+0000","last_change":"2024-03-06T07:25:51.680345+0000","last_active":"2024-03-06T07:25:52.675672+0000","last_peered":"2024-03-06T07:25:52.675672+0000","last_clean":"2024-03-06T07:25:52.675672+0000","last_became_active":"2024-03-06T07:25:51.680024+0000","last_became_peered":"2024-03-06T07:25:51.680024+0000","last_unstale":"2024-03-06T07:25:52.675672+0000","last_undegraded":"2024-03-06T07:25:52.675672+0000","last_fullsized":"2024-03-06T07:25:52.675672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005403+0000","last_change":"2024-03-06T07:25:51.680932+0000","last_active":"2024-03-06T07:25:53.005403+0000","last_peered":"2024-03-06T07:25:53.005403+0000","last_clean":"2024-03-06T07:25:53.005403+0000","last_became_active":"2024-03-06T07:25:51.680638+0000","last_became_peered":"2024-03-06T07:25:51.680638+0000","last_unstale":"2024-03-06T07:25:53.005403+0000","last_undegraded":"2024-03-06T07:25:53.005403+0000","last_fullsized":"2024-03-06T07:25:53.005403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:48.659969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801037+0000","last_change":"2024-03-06T07:25:51.682488+0000","last_active":"2024-03-06T07:25:52.801037+0000","last_peered":"2024-03-06T07:25:52.801037+0000","last_clean":"2024-03-06T07:25:52.801037+0000","last_became_active":"2024-03-06T07:25:51.682199+0000","last_became_peered":"2024-03-06T07:25:51.682199+0000","last_unstale":"2024-03-06T07:25:52.801037+0000","last_undegraded":"2024-03-06T07:25:52.801037+0000","last_fullsized":"2024-03-06T07:25:52.801037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:04.113709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994217+0000","last_change":"2024-03-06T07:25:51.676566+0000","last_active":"2024-03-06T07:25:52.994217+0000","last_peered":"2024-03-06T07:25:52.994217+0000","last_clean":"2024-03-06T07:25:52.994217+0000","last_became_active":"2024-03-06T07:25:51.676318+0000","last_became_peered":"2024-03-06T07:25:51.676318+0000","last_unstale":"2024-03-06T07:25:52.994217+0000","last_undegraded":"2024-03-06T07:25:52.994217+0000","last_fullsized":"2024-03-06T07:25:52.994217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:53.438813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674313+0000","last_change":"2024-03-06T07:25:51.676990+0000","last_active":"2024-03-06T07:25:52.674313+0000","last_peered":"2024-03-06T07:25:52.674313+0000","last_clean":"2024-03-06T07:25:52.674313+0000","last_became_active":"2024-03-06T07:25:51.676709+0000","last_became_peered":"2024-03-06T07:25:51.676709+0000","last_unstale":"2024-03-06T07:25:52.674313+0000","last_undegraded":"2024-03-06T07:25:52.674313+0000","last_fullsized":"2024-03-06T07:25:52.674313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:21.738142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674764+0000","last_change":"2024-03-06T07:25:51.679322+0000","last_active":"2024-03-06T07:25:52.674764+0000","last_peered":"2024-03-06T07:25:52.674764+0000","last_clean":"2024-03-06T07:25:52.674764+0000","last_became_active":"2024-03-06T07:25:51.679022+0000","last_became_peered":"2024-03-06T07:25:51.679022+0000","last_unstale":"2024-03-06T07:25:52.674764+0000","last_undegraded":"2024-03-06T07:25:52.674764+0000","last_fullsized":"2024-03-06T07:25:52.674764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:33.451809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801077+0000","last_change":"2024-03-06T07:25:51.672332+0000","last_active":"2024-03-06T07:25:52.801077+0000","last_peered":"2024-03-06T07:25:52.801077+0000","last_clean":"2024-03-06T07:25:52.801077+0000","last_became_active":"2024-03-06T07:25:51.672086+0000","last_became_peered":"2024-03-06T07:25:51.672086+0000","last_unstale":"2024-03-06T07:25:52.801077+0000","last_undegraded":"2024-03-06T07:25:52.801077+0000","last_fullsized":"2024-03-06T07:25:52.801077+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:21.788094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671368+0000","last_change":"2024-03-06T07:25:51.680243+0000","last_active":"2024-03-06T07:25:52.671368+0000","last_peered":"2024-03-06T07:25:52.671368+0000","last_clean":"2024-03-06T07:25:52.671368+0000","last_became_active":"2024-03-06T07:25:51.679981+0000","last_became_peered":"2024-03-06T07:25:51.679981+0000","last_unstale":"2024-03-06T07:25:52.671368+0000","last_undegraded":"2024-03-06T07:25:52.671368+0000","last_fullsized":"2024-03-06T07:25:52.671368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:49.195508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801303+0000","last_change":"2024-03-06T07:25:51.681850+0000","last_active":"2024-03-06T07:25:52.801303+0000","last_peered":"2024-03-06T07:25:52.801303+0000","last_clean":"2024-03-06T07:25:52.801303+0000","last_became_active":"2024-03-06T07:25:51.681568+0000","last_became_peered":"2024-03-06T07:25:51.681568+0000","last_unstale":"2024-03-06T07:25:52.801303+0000","last_undegraded":"2024-03-06T07:25:52.801303+0000","last_fullsized":"2024-03-06T07:25:52.801303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:17.662481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801098+0000","last_change":"2024-03-06T07:25:51.681241+0000","last_active":"2024-03-06T07:25:52.801098+0000","last_peered":"2024-03-06T07:25:52.801098+0000","last_clean":"2024-03-06T07:25:52.801098+0000","last_became_active":"2024-03-06T07:25:51.680940+0000","last_became_peered":"2024-03-06T07:25:51.680940+0000","last_unstale":"2024-03-06T07:25:52.801098+0000","last_undegraded":"2024-03-06T07:25:52.801098+0000","last_fullsized":"2024-03-06T07:25:52.801098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:18.980000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004740+0000","last_change":"2024-03-06T07:25:51.681877+0000","last_active":"2024-03-06T07:25:53.004740+0000","last_peered":"2024-03-06T07:25:53.004740+0000","last_clean":"2024-03-06T07:25:53.004740+0000","last_became_active":"2024-03-06T07:25:51.681626+0000","last_became_peered":"2024-03-06T07:25:51.681626+0000","last_unstale":"2024-03-06T07:25:53.004740+0000","last_undegraded":"2024-03-06T07:25:53.004740+0000","last_fullsized":"2024-03-06T07:25:53.004740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:07.418991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674861+0000","last_change":"2024-03-06T07:25:51.683454+0000","last_active":"2024-03-06T07:25:52.674861+0000","last_peered":"2024-03-06T07:25:52.674861+0000","last_clean":"2024-03-06T07:25:52.674861+0000","last_became_active":"2024-03-06T07:25:51.682932+0000","last_became_peered":"2024-03-06T07:25:51.682932+0000","last_unstale":"2024-03-06T07:25:52.674861+0000","last_undegraded":"2024-03-06T07:25:52.674861+0000","last_fullsized":"2024-03-06T07:25:52.674861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800882+0000","last_change":"2024-03-06T07:25:51.684204+0000","last_active":"2024-03-06T07:25:52.800882+0000","last_peered":"2024-03-06T07:25:52.800882+0000","last_clean":"2024-03-06T07:25:52.800882+0000","last_became_active":"2024-03-06T07:25:51.683940+0000","last_became_peered":"2024-03-06T07:25:51.683940+0000","last_unstale":"2024-03-06T07:25:52.800882+0000","last_undegraded":"2024-03-06T07:25:52.800882+0000","last_fullsized":"2024-03-06T07:25:52.800882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:23.573251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004849+0000","last_change":"2024-03-06T07:25:51.680652+0000","last_active":"2024-03-06T07:25:53.004849+0000","last_peered":"2024-03-06T07:25:53.004849+0000","last_clean":"2024-03-06T07:25:53.004849+0000","last_became_active":"2024-03-06T07:25:51.680402+0000","last_became_peered":"2024-03-06T07:25:51.680402+0000","last_unstale":"2024-03-06T07:25:53.004849+0000","last_undegraded":"2024-03-06T07:25:53.004849+0000","last_fullsized":"2024-03-06T07:25:53.004849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:33.343720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005369+0000","last_change":"2024-03-06T07:25:51.682124+0000","last_active":"2024-03-06T07:25:53.005369+0000","last_peered":"2024-03-06T07:25:53.005369+0000","last_clean":"2024-03-06T07:25:53.005369+0000","last_became_active":"2024-03-06T07:25:51.681851+0000","last_became_peered":"2024-03-06T07:25:51.681851+0000","last_unstale":"2024-03-06T07:25:53.005369+0000","last_undegraded":"2024-03-06T07:25:53.005369+0000","last_fullsized":"2024-03-06T07:25:53.005369+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:55.427087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674874+0000","last_change":"2024-03-06T07:25:51.678817+0000","last_active":"2024-03-06T07:25:52.674874+0000","last_peered":"2024-03-06T07:25:52.674874+0000","last_clean":"2024-03-06T07:25:52.674874+0000","last_became_active":"2024-03-06T07:25:51.677774+0000","last_became_peered":"2024-03-06T07:25:51.677774+0000","last_unstale":"2024-03-06T07:25:52.674874+0000","last_undegraded":"2024-03-06T07:25:52.674874+0000","last_fullsized":"2024-03-06T07:25:52.674874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675705+0000","last_change":"2024-03-06T07:25:51.680054+0000","last_active":"2024-03-06T07:25:52.675705+0000","last_peered":"2024-03-06T07:25:52.675705+0000","last_clean":"2024-03-06T07:25:52.675705+0000","last_became_active":"2024-03-06T07:25:51.679787+0000","last_became_peered":"2024-03-06T07:25:51.679787+0000","last_unstale":"2024-03-06T07:25:52.675705+0000","last_undegraded":"2024-03-06T07:25:52.675705+0000","last_fullsized":"2024-03-06T07:25:52.675705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004704+0000","last_change":"2024-03-06T07:25:51.677373+0000","last_active":"2024-03-06T07:25:53.004704+0000","last_peered":"2024-03-06T07:25:53.004704+0000","last_clean":"2024-03-06T07:25:53.004704+0000","last_became_active":"2024-03-06T07:25:51.677137+0000","last_became_peered":"2024-03-06T07:25:51.677137+0000","last_unstale":"2024-03-06T07:25:53.004704+0000","last_undegraded":"2024-03-06T07:25:53.004704+0000","last_fullsized":"2024-03-06T07:25:53.004704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:24.300534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004894+0000","last_change":"2024-03-06T07:25:51.672433+0000","last_active":"2024-03-06T07:25:53.004894+0000","last_peered":"2024-03-06T07:25:53.004894+0000","last_clean":"2024-03-06T07:25:53.004894+0000","last_became_active":"2024-03-06T07:25:51.672152+0000","last_became_peered":"2024-03-06T07:25:51.672152+0000","last_unstale":"2024-03-06T07:25:53.004894+0000","last_undegraded":"2024-03-06T07:25:53.004894+0000","last_fullsized":"2024-03-06T07:25:53.004894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:45.422326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674713+0000","last_change":"2024-03-06T07:25:51.685203+0000","last_active":"2024-03-06T07:25:52.674713+0000","last_peered":"2024-03-06T07:25:52.674713+0000","last_clean":"2024-03-06T07:25:52.674713+0000","last_became_active":"2024-03-06T07:25:51.684174+0000","last_became_peered":"2024-03-06T07:25:51.684174+0000","last_unstale":"2024-03-06T07:25:52.674713+0000","last_undegraded":"2024-03-06T07:25:52.674713+0000","last_fullsized":"2024-03-06T07:25:52.674713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:18.482302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674638+0000","last_change":"2024-03-06T07:25:51.686828+0000","last_active":"2024-03-06T07:25:52.674638+0000","last_peered":"2024-03-06T07:25:52.674638+0000","last_clean":"2024-03-06T07:25:52.674638+0000","last_became_active":"2024-03-06T07:25:51.686529+0000","last_became_peered":"2024-03-06T07:25:51.686529+0000","last_unstale":"2024-03-06T07:25:52.674638+0000","last_undegraded":"2024-03-06T07:25:52.674638+0000","last_fullsized":"2024-03-06T07:25:52.674638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674836+0000","last_change":"2024-03-06T07:25:51.675516+0000","last_active":"2024-03-06T07:25:52.674836+0000","last_peered":"2024-03-06T07:25:52.674836+0000","last_clean":"2024-03-06T07:25:52.674836+0000","last_became_active":"2024-03-06T07:25:51.675166+0000","last_became_peered":"2024-03-06T07:25:51.675166+0000","last_unstale":"2024-03-06T07:25:52.674836+0000","last_undegraded":"2024-03-06T07:25:52.674836+0000","last_fullsized":"2024-03-06T07:25:52.674836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:27.077831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675927+0000","last_change":"2024-03-06T07:25:50.665724+0000","last_active":"2024-03-06T07:25:52.675927+0000","last_peered":"2024-03-06T07:25:52.675927+0000","last_clean":"2024-03-06T07:25:52.675927+0000","last_became_active":"2024-03-06T07:25:50.665446+0000","last_became_peered":"2024-03-06T07:25:50.665446+0000","last_unstale":"2024-03-06T07:25:52.675927+0000","last_undegraded":"2024-03-06T07:25:52.675927+0000","last_fullsized":"2024-03-06T07:25:52.675927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:35.574350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675869+0000","last_change":"2024-03-06T07:25:51.680584+0000","last_active":"2024-03-06T07:25:52.675869+0000","last_peered":"2024-03-06T07:25:52.675869+0000","last_clean":"2024-03-06T07:25:52.675869+0000","last_became_active":"2024-03-06T07:25:51.680260+0000","last_became_peered":"2024-03-06T07:25:51.680260+0000","last_unstale":"2024-03-06T07:25:52.675869+0000","last_undegraded":"2024-03-06T07:25:52.675869+0000","last_fullsized":"2024-03-06T07:25:52.675869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.712380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675077+0000","last_change":"2024-03-06T07:25:50.667936+0000","last_active":"2024-03-06T07:25:52.675077+0000","last_peered":"2024-03-06T07:25:52.675077+0000","last_clean":"2024-03-06T07:25:52.675077+0000","last_became_active":"2024-03-06T07:25:50.667577+0000","last_became_peered":"2024-03-06T07:25:50.667577+0000","last_unstale":"2024-03-06T07:25:52.675077+0000","last_undegraded":"2024-03-06T07:25:52.675077+0000","last_fullsized":"2024-03-06T07:25:52.675077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:41.743623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675771+0000","last_change":"2024-03-06T07:25:51.685616+0000","last_active":"2024-03-06T07:25:52.675771+0000","last_peered":"2024-03-06T07:25:52.675771+0000","last_clean":"2024-03-06T07:25:52.675771+0000","last_became_active":"2024-03-06T07:25:51.685322+0000","last_became_peered":"2024-03-06T07:25:51.685322+0000","last_unstale":"2024-03-06T07:25:52.675771+0000","last_undegraded":"2024-03-06T07:25:52.675771+0000","last_fullsized":"2024-03-06T07:25:52.675771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671402+0000","last_change":"2024-03-06T07:25:50.667866+0000","last_active":"2024-03-06T07:25:52.671402+0000","last_peered":"2024-03-06T07:25:52.671402+0000","last_clean":"2024-03-06T07:25:52.671402+0000","last_became_active":"2024-03-06T07:25:50.667614+0000","last_became_peered":"2024-03-06T07:25:50.667614+0000","last_unstale":"2024-03-06T07:25:52.671402+0000","last_undegraded":"2024-03-06T07:25:52.671402+0000","last_fullsized":"2024-03-06T07:25:52.671402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:55.559459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005458+0000","last_change":"2024-03-06T07:25:51.674998+0000","last_active":"2024-03-06T07:25:53.005458+0000","last_peered":"2024-03-06T07:25:53.005458+0000","last_clean":"2024-03-06T07:25:53.005458+0000","last_became_active":"2024-03-06T07:25:51.674720+0000","last_became_peered":"2024-03-06T07:25:51.674720+0000","last_unstale":"2024-03-06T07:25:53.005458+0000","last_undegraded":"2024-03-06T07:25:53.005458+0000","last_fullsized":"2024-03-06T07:25:53.005458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:23.266106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671229+0000","last_change":"2024-03-06T07:25:50.667000+0000","last_active":"2024-03-06T07:25:52.671229+0000","last_peered":"2024-03-06T07:25:52.671229+0000","last_clean":"2024-03-06T07:25:52.671229+0000","last_became_active":"2024-03-06T07:25:50.666740+0000","last_became_peered":"2024-03-06T07:25:50.666740+0000","last_unstale":"2024-03-06T07:25:52.671229+0000","last_undegraded":"2024-03-06T07:25:52.671229+0000","last_fullsized":"2024-03-06T07:25:52.671229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:37.494888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004961+0000","last_change":"2024-03-06T07:25:51.670449+0000","last_active":"2024-03-06T07:25:53.004961+0000","last_peered":"2024-03-06T07:25:53.004961+0000","last_clean":"2024-03-06T07:25:53.004961+0000","last_became_active":"2024-03-06T07:25:51.670204+0000","last_became_peered":"2024-03-06T07:25:51.670204+0000","last_unstale":"2024-03-06T07:25:53.004961+0000","last_undegraded":"2024-03-06T07:25:53.004961+0000","last_fullsized":"2024-03-06T07:25:53.004961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:09.510502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675637+0000","last_change":"2024-03-06T07:25:50.665732+0000","last_active":"2024-03-06T07:25:52.675637+0000","last_peered":"2024-03-06T07:25:52.675637+0000","last_clean":"2024-03-06T07:25:52.675637+0000","last_became_active":"2024-03-06T07:25:50.664895+0000","last_became_peered":"2024-03-06T07:25:50.664895+0000","last_unstale":"2024-03-06T07:25:52.675637+0000","last_undegraded":"2024-03-06T07:25:52.675637+0000","last_fullsized":"2024-03-06T07:25:52.675637+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:06.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675488+0000","last_change":"2024-03-06T07:25:51.685945+0000","last_active":"2024-03-06T07:25:52.675488+0000","last_peered":"2024-03-06T07:25:52.675488+0000","last_clean":"2024-03-06T07:25:52.675488+0000","last_became_active":"2024-03-06T07:25:51.685619+0000","last_became_peered":"2024-03-06T07:25:51.685619+0000","last_unstale":"2024-03-06T07:25:52.675488+0000","last_undegraded":"2024-03-06T07:25:52.675488+0000","last_fullsized":"2024-03-06T07:25:52.675488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674446+0000","last_change":"2024-03-06T07:25:50.665220+0000","last_active":"2024-03-06T07:25:52.674446+0000","last_peered":"2024-03-06T07:25:52.674446+0000","last_clean":"2024-03-06T07:25:52.674446+0000","last_became_active":"2024-03-06T07:25:50.664947+0000","last_became_peered":"2024-03-06T07:25:50.664947+0000","last_unstale":"2024-03-06T07:25:52.674446+0000","last_undegraded":"2024-03-06T07:25:52.674446+0000","last_fullsized":"2024-03-06T07:25:52.674446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:23.294330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004723+0000","last_change":"2024-03-06T07:25:51.681340+0000","last_active":"2024-03-06T07:25:53.004723+0000","last_peered":"2024-03-06T07:25:53.004723+0000","last_clean":"2024-03-06T07:25:53.004723+0000","last_became_active":"2024-03-06T07:25:51.681056+0000","last_became_peered":"2024-03-06T07:25:51.681056+0000","last_unstale":"2024-03-06T07:25:53.004723+0000","last_undegraded":"2024-03-06T07:25:53.004723+0000","last_fullsized":"2024-03-06T07:25:53.004723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:12.789933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.670990+0000","last_change":"2024-03-06T07:25:50.665789+0000","last_active":"2024-03-06T07:25:52.670990+0000","last_peered":"2024-03-06T07:25:52.670990+0000","last_clean":"2024-03-06T07:25:52.670990+0000","last_became_active":"2024-03-06T07:25:50.665358+0000","last_became_peered":"2024-03-06T07:25:50.665358+0000","last_unstale":"2024-03-06T07:25:52.670990+0000","last_undegraded":"2024-03-06T07:25:52.670990+0000","last_fullsized":"2024-03-06T07:25:52.670990+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:26.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994238+0000","last_change":"2024-03-06T07:25:51.674367+0000","last_active":"2024-03-06T07:25:52.994238+0000","last_peered":"2024-03-06T07:25:52.994238+0000","last_clean":"2024-03-06T07:25:52.994238+0000","last_became_active":"2024-03-06T07:25:51.674050+0000","last_became_peered":"2024-03-06T07:25:51.674050+0000","last_unstale":"2024-03-06T07:25:52.994238+0000","last_undegraded":"2024-03-06T07:25:52.994238+0000","last_fullsized":"2024-03-06T07:25:52.994238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:55.427087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004492+0000","last_change":"2024-03-06T07:25:50.664211+0000","last_active":"2024-03-06T07:25:53.004492+0000","last_peered":"2024-03-06T07:25:53.004492+0000","last_clean":"2024-03-06T07:25:53.004492+0000","last_became_active":"2024-03-06T07:25:50.663925+0000","last_became_peered":"2024-03-06T07:25:50.663925+0000","last_unstale":"2024-03-06T07:25:53.004492+0000","last_undegraded":"2024-03-06T07:25:53.004492+0000","last_fullsized":"2024-03-06T07:25:53.004492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:29.439664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994502+0000","last_change":"2024-03-06T07:25:51.681731+0000","last_active":"2024-03-06T07:25:52.994502+0000","last_peered":"2024-03-06T07:25:52.994502+0000","last_clean":"2024-03-06T07:25:52.994502+0000","last_became_active":"2024-03-06T07:25:51.681445+0000","last_became_peered":"2024-03-06T07:25:51.681445+0000","last_unstale":"2024-03-06T07:25:52.994502+0000","last_undegraded":"2024-03-06T07:25:52.994502+0000","last_fullsized":"2024-03-06T07:25:52.994502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:51.665911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.676005+0000","last_change":"2024-03-06T07:25:50.659033+0000","last_active":"2024-03-06T07:25:52.676005+0000","last_peered":"2024-03-06T07:25:52.676005+0000","last_clean":"2024-03-06T07:25:52.676005+0000","last_became_active":"2024-03-06T07:25:50.658341+0000","last_became_peered":"2024-03-06T07:25:50.658341+0000","last_unstale":"2024-03-06T07:25:52.676005+0000","last_undegraded":"2024-03-06T07:25:52.676005+0000","last_fullsized":"2024-03-06T07:25:52.676005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:44.416122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005205+0000","last_change":"2024-03-06T07:25:51.675730+0000","last_active":"2024-03-06T07:25:53.005205+0000","last_peered":"2024-03-06T07:25:53.005205+0000","last_clean":"2024-03-06T07:25:53.005205+0000","last_became_active":"2024-03-06T07:25:51.675375+0000","last_became_peered":"2024-03-06T07:25:51.675375+0000","last_unstale":"2024-03-06T07:25:53.005205+0000","last_undegraded":"2024-03-06T07:25:53.005205+0000","last_fullsized":"2024-03-06T07:25:53.005205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675117+0000","last_change":"2024-03-06T07:25:50.666708+0000","last_active":"2024-03-06T07:25:52.675117+0000","last_peered":"2024-03-06T07:25:52.675117+0000","last_clean":"2024-03-06T07:25:52.675117+0000","last_became_active":"2024-03-06T07:25:50.666385+0000","last_became_peered":"2024-03-06T07:25:50.666385+0000","last_unstale":"2024-03-06T07:25:52.675117+0000","last_undegraded":"2024-03-06T07:25:52.675117+0000","last_fullsized":"2024-03-06T07:25:52.675117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:05.567054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671255+0000","last_change":"2024-03-06T07:25:51.675676+0000","last_active":"2024-03-06T07:25:52.671255+0000","last_peered":"2024-03-06T07:25:52.671255+0000","last_clean":"2024-03-06T07:25:52.671255+0000","last_became_active":"2024-03-06T07:25:51.675431+0000","last_became_peered":"2024-03-06T07:25:51.675431+0000","last_unstale":"2024-03-06T07:25:52.671255+0000","last_undegraded":"2024-03-06T07:25:52.671255+0000","last_fullsized":"2024-03-06T07:25:52.671255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:52.903718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675236+0000","last_change":"2024-03-06T07:25:50.658006+0000","last_active":"2024-03-06T07:25:52.675236+0000","last_peered":"2024-03-06T07:25:52.675236+0000","last_clean":"2024-03-06T07:25:52.675236+0000","last_became_active":"2024-03-06T07:25:50.657716+0000","last_became_peered":"2024-03-06T07:25:50.657716+0000","last_unstale":"2024-03-06T07:25:52.675236+0000","last_undegraded":"2024-03-06T07:25:52.675236+0000","last_fullsized":"2024-03-06T07:25:52.675236+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:52.828417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674944+0000","last_change":"2024-03-06T07:25:51.686127+0000","last_active":"2024-03-06T07:25:52.674944+0000","last_peered":"2024-03-06T07:25:52.674944+0000","last_clean":"2024-03-06T07:25:52.674944+0000","last_became_active":"2024-03-06T07:25:51.685846+0000","last_became_peered":"2024-03-06T07:25:51.685846+0000","last_unstale":"2024-03-06T07:25:52.674944+0000","last_undegraded":"2024-03-06T07:25:52.674944+0000","last_fullsized":"2024-03-06T07:25:52.674944+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005222+0000","last_change":"2024-03-06T07:25:50.668266+0000","last_active":"2024-03-06T07:25:53.005222+0000","last_peered":"2024-03-06T07:25:53.005222+0000","last_clean":"2024-03-06T07:25:53.005222+0000","last_became_active":"2024-03-06T07:25:50.667485+0000","last_became_peered":"2024-03-06T07:25:50.667485+0000","last_unstale":"2024-03-06T07:25:53.005222+0000","last_undegraded":"2024-03-06T07:25:53.005222+0000","last_fullsized":"2024-03-06T07:25:53.005222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:25.507135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671140+0000","last_change":"2024-03-06T07:25:51.679153+0000","last_active":"2024-03-06T07:25:52.671140+0000","last_peered":"2024-03-06T07:25:52.671140+0000","last_clean":"2024-03-06T07:25:52.671140+0000","last_became_active":"2024-03-06T07:25:51.678289+0000","last_became_peered":"2024-03-06T07:25:51.678289+0000","last_unstale":"2024-03-06T07:25:52.671140+0000","last_undegraded":"2024-03-06T07:25:52.671140+0000","last_fullsized":"2024-03-06T07:25:52.671140+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:15.538278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004757+0000","last_change":"2024-03-06T07:25:50.665487+0000","last_active":"2024-03-06T07:25:53.004757+0000","last_peered":"2024-03-06T07:25:53.004757+0000","last_clean":"2024-03-06T07:25:53.004757+0000","last_became_active":"2024-03-06T07:25:50.665171+0000","last_became_peered":"2024-03-06T07:25:50.665171+0000","last_unstale":"2024-03-06T07:25:53.004757+0000","last_undegraded":"2024-03-06T07:25:53.004757+0000","last_fullsized":"2024-03-06T07:25:53.004757+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:21.303434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005362+0000","last_change":"2024-03-06T07:25:51.680433+0000","last_active":"2024-03-06T07:25:53.005362+0000","last_peered":"2024-03-06T07:25:53.005362+0000","last_clean":"2024-03-06T07:25:53.005362+0000","last_became_active":"2024-03-06T07:25:51.680124+0000","last_became_peered":"2024-03-06T07:25:51.680124+0000","last_unstale":"2024-03-06T07:25:53.005362+0000","last_undegraded":"2024-03-06T07:25:53.005362+0000","last_fullsized":"2024-03-06T07:25:53.005362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:51.665911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994254+0000","last_change":"2024-03-06T07:25:50.660720+0000","last_active":"2024-03-06T07:25:52.994254+0000","last_peered":"2024-03-06T07:25:52.994254+0000","last_clean":"2024-03-06T07:25:52.994254+0000","last_became_active":"2024-03-06T07:25:50.660452+0000","last_became_peered":"2024-03-06T07:25:50.660452+0000","last_unstale":"2024-03-06T07:25:52.994254+0000","last_undegraded":"2024-03-06T07:25:52.994254+0000","last_fullsized":"2024-03-06T07:25:52.994254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:02.606511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004561+0000","last_change":"2024-03-06T07:25:51.671565+0000","last_active":"2024-03-06T07:25:53.004561+0000","last_peered":"2024-03-06T07:25:53.004561+0000","last_clean":"2024-03-06T07:25:53.004561+0000","last_became_active":"2024-03-06T07:25:51.671341+0000","last_became_peered":"2024-03-06T07:25:51.671341+0000","last_unstale":"2024-03-06T07:25:53.004561+0000","last_undegraded":"2024-03-06T07:25:53.004561+0000","last_fullsized":"2024-03-06T07:25:53.004561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:53.834620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005007+0000","last_change":"2024-03-06T07:25:50.667664+0000","last_active":"2024-03-06T07:25:53.005007+0000","last_peered":"2024-03-06T07:25:53.005007+0000","last_clean":"2024-03-06T07:25:53.005007+0000","last_became_active":"2024-03-06T07:25:50.667347+0000","last_became_peered":"2024-03-06T07:25:50.667347+0000","last_unstale":"2024-03-06T07:25:53.005007+0000","last_undegraded":"2024-03-06T07:25:53.005007+0000","last_fullsized":"2024-03-06T07:25:53.005007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:36.841035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004529+0000","last_change":"2024-03-06T07:25:51.680371+0000","last_active":"2024-03-06T07:25:53.004529+0000","last_peered":"2024-03-06T07:25:53.004529+0000","last_clean":"2024-03-06T07:25:53.004529+0000","last_became_active":"2024-03-06T07:25:51.680102+0000","last_became_peered":"2024-03-06T07:25:51.680102+0000","last_unstale":"2024-03-06T07:25:53.004529+0000","last_undegraded":"2024-03-06T07:25:53.004529+0000","last_fullsized":"2024-03-06T07:25:53.004529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:42.749827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.670947+0000","last_change":"2024-03-06T07:25:50.663501+0000","last_active":"2024-03-06T07:25:52.670947+0000","last_peered":"2024-03-06T07:25:52.670947+0000","last_clean":"2024-03-06T07:25:52.670947+0000","last_became_active":"2024-03-06T07:25:50.663232+0000","last_became_peered":"2024-03-06T07:25:50.663232+0000","last_unstale":"2024-03-06T07:25:52.670947+0000","last_undegraded":"2024-03-06T07:25:52.670947+0000","last_fullsized":"2024-03-06T07:25:52.670947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:49.142433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005373+0000","last_change":"2024-03-06T07:25:51.675519+0000","last_active":"2024-03-06T07:25:53.005373+0000","last_peered":"2024-03-06T07:25:53.005373+0000","last_clean":"2024-03-06T07:25:53.005373+0000","last_became_active":"2024-03-06T07:25:51.675270+0000","last_became_peered":"2024-03-06T07:25:51.675270+0000","last_unstale":"2024-03-06T07:25:53.005373+0000","last_undegraded":"2024-03-06T07:25:53.005373+0000","last_fullsized":"2024-03-06T07:25:53.005373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:24.533672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.676059+0000","last_change":"2024-03-06T07:25:50.659845+0000","last_active":"2024-03-06T07:25:52.676059+0000","last_peered":"2024-03-06T07:25:52.676059+0000","last_clean":"2024-03-06T07:25:52.676059+0000","last_became_active":"2024-03-06T07:25:50.659527+0000","last_became_peered":"2024-03-06T07:25:50.659527+0000","last_unstale":"2024-03-06T07:25:52.676059+0000","last_undegraded":"2024-03-06T07:25:52.676059+0000","last_fullsized":"2024-03-06T07:25:52.676059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:32.337516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675974+0000","last_change":"2024-03-06T07:25:51.673226+0000","last_active":"2024-03-06T07:25:52.675974+0000","last_peered":"2024-03-06T07:25:52.675974+0000","last_clean":"2024-03-06T07:25:52.675974+0000","last_became_active":"2024-03-06T07:25:51.672946+0000","last_became_peered":"2024-03-06T07:25:51.672946+0000","last_unstale":"2024-03-06T07:25:52.675974+0000","last_undegraded":"2024-03-06T07:25:52.675974+0000","last_fullsized":"2024-03-06T07:25:52.675974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:15.538278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671294+0000","last_change":"2024-03-06T07:25:50.668525+0000","last_active":"2024-03-06T07:25:52.671294+0000","last_peered":"2024-03-06T07:25:52.671294+0000","last_clean":"2024-03-06T07:25:52.671294+0000","last_became_active":"2024-03-06T07:25:50.668291+0000","last_became_peered":"2024-03-06T07:25:50.668291+0000","last_unstale":"2024-03-06T07:25:52.671294+0000","last_undegraded":"2024-03-06T07:25:52.671294+0000","last_fullsized":"2024-03-06T07:25:52.671294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:31.002940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675006+0000","last_change":"2024-03-06T07:25:51.683425+0000","last_active":"2024-03-06T07:25:52.675006+0000","last_peered":"2024-03-06T07:25:52.675006+0000","last_clean":"2024-03-06T07:25:52.675006+0000","last_became_active":"2024-03-06T07:25:51.683168+0000","last_became_peered":"2024-03-06T07:25:51.683168+0000","last_unstale":"2024-03-06T07:25:52.675006+0000","last_undegraded":"2024-03-06T07:25:52.675006+0000","last_fullsized":"2024-03-06T07:25:52.675006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671310+0000","last_change":"2024-03-06T07:25:50.665805+0000","last_active":"2024-03-06T07:25:52.671310+0000","last_peered":"2024-03-06T07:25:52.671310+0000","last_clean":"2024-03-06T07:25:52.671310+0000","last_became_active":"2024-03-06T07:25:50.665533+0000","last_became_peered":"2024-03-06T07:25:50.665533+0000","last_unstale":"2024-03-06T07:25:52.671310+0000","last_undegraded":"2024-03-06T07:25:52.671310+0000","last_fullsized":"2024-03-06T07:25:52.671310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:32.208112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801202+0000","last_change":"2024-03-06T07:25:51.670633+0000","last_active":"2024-03-06T07:25:52.801202+0000","last_peered":"2024-03-06T07:25:52.801202+0000","last_clean":"2024-03-06T07:25:52.801202+0000","last_became_active":"2024-03-06T07:25:51.670383+0000","last_became_peered":"2024-03-06T07:25:51.670383+0000","last_unstale":"2024-03-06T07:25:52.801202+0000","last_undegraded":"2024-03-06T07:25:52.801202+0000","last_fullsized":"2024-03-06T07:25:52.801202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:30.342476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994321+0000","last_change":"2024-03-06T07:25:50.665180+0000","last_active":"2024-03-06T07:25:52.994321+0000","last_peered":"2024-03-06T07:25:52.994321+0000","last_clean":"2024-03-06T07:25:52.994321+0000","last_became_active":"2024-03-06T07:25:50.664910+0000","last_became_peered":"2024-03-06T07:25:50.664910+0000","last_unstale":"2024-03-06T07:25:52.994321+0000","last_undegraded":"2024-03-06T07:25:52.994321+0000","last_fullsized":"2024-03-06T07:25:52.994321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:52.993794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004875+0000","last_change":"2024-03-06T07:25:51.676865+0000","last_active":"2024-03-06T07:25:53.004875+0000","last_peered":"2024-03-06T07:25:53.004875+0000","last_clean":"2024-03-06T07:25:53.004875+0000","last_became_active":"2024-03-06T07:25:51.676514+0000","last_became_peered":"2024-03-06T07:25:51.676514+0000","last_unstale":"2024-03-06T07:25:53.004875+0000","last_undegraded":"2024-03-06T07:25:53.004875+0000","last_fullsized":"2024-03-06T07:25:53.004875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801155+0000","last_change":"2024-03-06T07:25:50.662536+0000","last_active":"2024-03-06T07:25:52.801155+0000","last_peered":"2024-03-06T07:25:52.801155+0000","last_clean":"2024-03-06T07:25:52.801155+0000","last_became_active":"2024-03-06T07:25:50.662248+0000","last_became_peered":"2024-03-06T07:25:50.662248+0000","last_unstale":"2024-03-06T07:25:52.801155+0000","last_undegraded":"2024-03-06T07:25:52.801155+0000","last_fullsized":"2024-03-06T07:25:52.801155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:26.050173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800815+0000","last_change":"2024-03-06T07:25:51.675362+0000","last_active":"2024-03-06T07:25:52.800815+0000","last_peered":"2024-03-06T07:25:52.800815+0000","last_clean":"2024-03-06T07:25:52.800815+0000","last_became_active":"2024-03-06T07:25:51.674936+0000","last_became_peered":"2024-03-06T07:25:51.674936+0000","last_unstale":"2024-03-06T07:25:52.800815+0000","last_undegraded":"2024-03-06T07:25:52.800815+0000","last_fullsized":"2024-03-06T07:25:52.800815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:16.568032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800780+0000","last_change":"2024-03-06T07:25:50.660498+0000","last_active":"2024-03-06T07:25:52.800780+0000","last_peered":"2024-03-06T07:25:52.800780+0000","last_clean":"2024-03-06T07:25:52.800780+0000","last_became_active":"2024-03-06T07:25:50.660246+0000","last_became_peered":"2024-03-06T07:25:50.660246+0000","last_unstale":"2024-03-06T07:25:52.800780+0000","last_undegraded":"2024-03-06T07:25:52.800780+0000","last_fullsized":"2024-03-06T07:25:52.800780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.278410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004907+0000","last_change":"2024-03-06T07:25:51.680574+0000","last_active":"2024-03-06T07:25:53.004907+0000","last_peered":"2024-03-06T07:25:53.004907+0000","last_clean":"2024-03-06T07:25:53.004907+0000","last_became_active":"2024-03-06T07:25:51.680281+0000","last_became_peered":"2024-03-06T07:25:51.680281+0000","last_unstale":"2024-03-06T07:25:53.004907+0000","last_undegraded":"2024-03-06T07:25:53.004907+0000","last_fullsized":"2024-03-06T07:25:53.004907+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994090+0000","last_change":"2024-03-06T07:25:50.666257+0000","last_active":"2024-03-06T07:25:52.994090+0000","last_peered":"2024-03-06T07:25:52.994090+0000","last_clean":"2024-03-06T07:25:52.994090+0000","last_became_active":"2024-03-06T07:25:50.664875+0000","last_became_peered":"2024-03-06T07:25:50.664875+0000","last_unstale":"2024-03-06T07:25:52.994090+0000","last_undegraded":"2024-03-06T07:25:52.994090+0000","last_fullsized":"2024-03-06T07:25:52.994090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.612866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674975+0000","last_change":"2024-03-06T07:25:51.678510+0000","last_active":"2024-03-06T07:25:52.674975+0000","last_peered":"2024-03-06T07:25:52.674975+0000","last_clean":"2024-03-06T07:25:52.674975+0000","last_became_active":"2024-03-06T07:25:51.678049+0000","last_became_peered":"2024-03-06T07:25:51.678049+0000","last_unstale":"2024-03-06T07:25:52.674975+0000","last_undegraded":"2024-03-06T07:25:52.674975+0000","last_fullsized":"2024-03-06T07:25:52.674975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004908+0000","last_change":"2024-03-06T07:25:50.664028+0000","last_active":"2024-03-06T07:25:53.004908+0000","last_peered":"2024-03-06T07:25:53.004908+0000","last_clean":"2024-03-06T07:25:53.004908+0000","last_became_active":"2024-03-06T07:25:50.663698+0000","last_became_peered":"2024-03-06T07:25:50.663698+0000","last_unstale":"2024-03-06T07:25:53.004908+0000","last_undegraded":"2024-03-06T07:25:53.004908+0000","last_fullsized":"2024-03-06T07:25:53.004908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:54.976768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005163+0000","last_change":"2024-03-06T07:25:51.674498+0000","last_active":"2024-03-06T07:25:53.005163+0000","last_peered":"2024-03-06T07:25:53.005163+0000","last_clean":"2024-03-06T07:25:53.005163+0000","last_became_active":"2024-03-06T07:25:51.673628+0000","last_became_peered":"2024-03-06T07:25:51.673628+0000","last_unstale":"2024-03-06T07:25:53.005163+0000","last_undegraded":"2024-03-06T07:25:53.005163+0000","last_fullsized":"2024-03-06T07:25:53.005163+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:41.203954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005010+0000","last_change":"2024-03-06T07:25:50.665322+0000","last_active":"2024-03-06T07:25:53.005010+0000","last_peered":"2024-03-06T07:25:53.005010+0000","last_clean":"2024-03-06T07:25:53.005010+0000","last_became_active":"2024-03-06T07:25:50.665051+0000","last_became_peered":"2024-03-06T07:25:50.665051+0000","last_unstale":"2024-03-06T07:25:53.005010+0000","last_undegraded":"2024-03-06T07:25:53.005010+0000","last_fullsized":"2024-03-06T07:25:53.005010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:03.252965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675860+0000","last_change":"2024-03-06T07:25:51.670848+0000","last_active":"2024-03-06T07:25:52.675860+0000","last_peered":"2024-03-06T07:25:52.675860+0000","last_clean":"2024-03-06T07:25:52.675860+0000","last_became_active":"2024-03-06T07:25:51.670601+0000","last_became_peered":"2024-03-06T07:25:51.670601+0000","last_unstale":"2024-03-06T07:25:52.675860+0000","last_undegraded":"2024-03-06T07:25:52.675860+0000","last_fullsized":"2024-03-06T07:25:52.675860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:42.061998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675408+0000","last_change":"2024-03-06T07:25:50.659022+0000","last_active":"2024-03-06T07:25:52.675408+0000","last_peered":"2024-03-06T07:25:52.675408+0000","last_clean":"2024-03-06T07:25:52.675408+0000","last_became_active":"2024-03-06T07:25:50.658737+0000","last_became_peered":"2024-03-06T07:25:50.658737+0000","last_unstale":"2024-03-06T07:25:52.675408+0000","last_undegraded":"2024-03-06T07:25:52.675408+0000","last_fullsized":"2024-03-06T07:25:52.675408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:41.743623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675106+0000","last_change":"2024-03-06T07:25:51.685417+0000","last_active":"2024-03-06T07:25:52.675106+0000","last_peered":"2024-03-06T07:25:52.675106+0000","last_clean":"2024-03-06T07:25:52.675106+0000","last_became_active":"2024-03-06T07:25:51.685153+0000","last_became_peered":"2024-03-06T07:25:51.685153+0000","last_unstale":"2024-03-06T07:25:52.675106+0000","last_undegraded":"2024-03-06T07:25:52.675106+0000","last_fullsized":"2024-03-06T07:25:52.675106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675812+0000","last_change":"2024-03-06T07:25:50.658414+0000","last_active":"2024-03-06T07:25:52.675812+0000","last_peered":"2024-03-06T07:25:52.675812+0000","last_clean":"2024-03-06T07:25:52.675812+0000","last_became_active":"2024-03-06T07:25:50.658117+0000","last_became_peered":"2024-03-06T07:25:50.658117+0000","last_unstale":"2024-03-06T07:25:52.675812+0000","last_undegraded":"2024-03-06T07:25:52.675812+0000","last_fullsized":"2024-03-06T07:25:52.675812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:06.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005316+0000","last_change":"2024-03-06T07:25:51.680659+0000","last_active":"2024-03-06T07:25:53.005316+0000","last_peered":"2024-03-06T07:25:53.005316+0000","last_clean":"2024-03-06T07:25:53.005316+0000","last_became_active":"2024-03-06T07:25:51.680365+0000","last_became_peered":"2024-03-06T07:25:51.680365+0000","last_unstale":"2024-03-06T07:25:53.005316+0000","last_undegraded":"2024-03-06T07:25:53.005316+0000","last_fullsized":"2024-03-06T07:25:53.005316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671168+0000","last_change":"2024-03-06T07:25:50.664527+0000","last_active":"2024-03-06T07:25:52.671168+0000","last_peered":"2024-03-06T07:25:52.671168+0000","last_clean":"2024-03-06T07:25:52.671168+0000","last_became_active":"2024-03-06T07:25:50.664117+0000","last_became_peered":"2024-03-06T07:25:50.664117+0000","last_unstale":"2024-03-06T07:25:52.671168+0000","last_undegraded":"2024-03-06T07:25:52.671168+0000","last_fullsized":"2024-03-06T07:25:52.671168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:15.731978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671218+0000","last_change":"2024-03-06T07:25:51.680138+0000","last_active":"2024-03-06T07:25:52.671218+0000","last_peered":"2024-03-06T07:25:52.671218+0000","last_clean":"2024-03-06T07:25:52.671218+0000","last_became_active":"2024-03-06T07:25:51.679882+0000","last_became_peered":"2024-03-06T07:25:51.679882+0000","last_unstale":"2024-03-06T07:25:52.671218+0000","last_undegraded":"2024-03-06T07:25:52.671218+0000","last_fullsized":"2024-03-06T07:25:52.671218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:17.142904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674507+0000","last_change":"2024-03-06T07:25:50.664868+0000","last_active":"2024-03-06T07:25:52.674507+0000","last_peered":"2024-03-06T07:25:52.674507+0000","last_clean":"2024-03-06T07:25:52.674507+0000","last_became_active":"2024-03-06T07:25:50.664449+0000","last_became_peered":"2024-03-06T07:25:50.664449+0000","last_unstale":"2024-03-06T07:25:52.674507+0000","last_undegraded":"2024-03-06T07:25:52.674507+0000","last_fullsized":"2024-03-06T07:25:52.674507+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:05.567054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005047+0000","last_change":"2024-03-06T07:25:51.681998+0000","last_active":"2024-03-06T07:25:53.005047+0000","last_peered":"2024-03-06T07:25:53.005047+0000","last_clean":"2024-03-06T07:25:53.005047+0000","last_became_active":"2024-03-06T07:25:51.681705+0000","last_became_peered":"2024-03-06T07:25:51.681705+0000","last_unstale":"2024-03-06T07:25:53.005047+0000","last_undegraded":"2024-03-06T07:25:53.005047+0000","last_fullsized":"2024-03-06T07:25:53.005047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:29.536548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675118+0000","last_change":"2024-03-06T07:25:50.657136+0000","last_active":"2024-03-06T07:25:52.675118+0000","last_peered":"2024-03-06T07:25:52.675118+0000","last_clean":"2024-03-06T07:25:52.675118+0000","last_became_active":"2024-03-06T07:25:50.656776+0000","last_became_peered":"2024-03-06T07:25:50.656776+0000","last_unstale":"2024-03-06T07:25:52.675118+0000","last_undegraded":"2024-03-06T07:25:52.675118+0000","last_fullsized":"2024-03-06T07:25:52.675118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:23.294330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675003+0000","last_change":"2024-03-06T07:25:51.670201+0000","last_active":"2024-03-06T07:25:52.675003+0000","last_peered":"2024-03-06T07:25:52.675003+0000","last_clean":"2024-03-06T07:25:52.675003+0000","last_became_active":"2024-03-06T07:25:51.669930+0000","last_became_peered":"2024-03-06T07:25:51.669930+0000","last_unstale":"2024-03-06T07:25:52.675003+0000","last_undegraded":"2024-03-06T07:25:52.675003+0000","last_fullsized":"2024-03-06T07:25:52.675003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674310+0000","last_change":"2024-03-06T07:25:50.666076+0000","last_active":"2024-03-06T07:25:52.674310+0000","last_peered":"2024-03-06T07:25:52.674310+0000","last_clean":"2024-03-06T07:25:52.674310+0000","last_became_active":"2024-03-06T07:25:50.665365+0000","last_became_peered":"2024-03-06T07:25:50.665365+0000","last_unstale":"2024-03-06T07:25:52.674310+0000","last_undegraded":"2024-03-06T07:25:52.674310+0000","last_fullsized":"2024-03-06T07:25:52.674310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:52.828417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004803+0000","last_change":"2024-03-06T07:25:51.674771+0000","last_active":"2024-03-06T07:25:53.004803+0000","last_peered":"2024-03-06T07:25:53.004803+0000","last_clean":"2024-03-06T07:25:53.004803+0000","last_became_active":"2024-03-06T07:25:51.674526+0000","last_became_peered":"2024-03-06T07:25:51.674526+0000","last_unstale":"2024-03-06T07:25:53.004803+0000","last_undegraded":"2024-03-06T07:25:53.004803+0000","last_fullsized":"2024-03-06T07:25:53.004803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:06.573258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005403+0000","last_change":"2024-03-06T07:25:50.665597+0000","last_active":"2024-03-06T07:25:53.005403+0000","last_peered":"2024-03-06T07:25:53.005403+0000","last_clean":"2024-03-06T07:25:53.005403+0000","last_became_active":"2024-03-06T07:25:50.665303+0000","last_became_peered":"2024-03-06T07:25:50.665303+0000","last_unstale":"2024-03-06T07:25:53.005403+0000","last_undegraded":"2024-03-06T07:25:53.005403+0000","last_fullsized":"2024-03-06T07:25:53.005403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:14.532074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":286,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005108+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:53.005108+0000","last_peered":"2024-03-06T07:25:53.005108+0000","last_clean":"2024-03-06T07:25:53.005108+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:53.005108+0000","last_undegraded":"2024-03-06T07:25:53.005108+0000","last_fullsized":"2024-03-06T07:25:53.005108+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1018105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":428361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":429301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":427667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":429279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1011712,"data_stored":428249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":427861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1018069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:56.941 INFO:tasks.ceph.mds.c.smithi122.stderr:2024-03-06T07:25:56.938+0000 7f816cbea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:59.295 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:59.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.395+0000 7f1084833640 1 Processor -- start 2024-03-06T07:25:59.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.395+0000 7f1084833640 1 -- start start 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.396+0000 7f1084833640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f10800d2920 0x7f10800d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.396+0000 7f1084833640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f10800d0650 0x7f10800d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f1084833640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108015e470 con 0x7f108015e090 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f1084833640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800d0fe0 con 0x7f10800d0650 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f1084833640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800d18f0 con 0x7f10800d2920 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f10800d0650 0x7f10800d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107e59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f108015e090 0x7f10800b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59806/0 (socket says 172.21.15.122:59806) 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107e59c640 1 -- 172.21.15.122:0/4269708022 learned_addr learned my addr 172.21.15.122:0/4269708022 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107dd9b640 1 --2- 172.21.15.122:0/4269708022 >> v2:172.21.15.169:3300/0 conn(0x7f10800d2920 0x7f10800d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107d59a640 1 -- 172.21.15.122:0/4269708022 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869759971 0 0) 0x7f108015e470 con 0x7f108015e090 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107d59a640 1 -- 172.21.15.122:0/4269708022 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c003520 con 0x7f108015e090 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107ed9d640 1 -- 172.21.15.122:0/4269708022 >> v2:172.21.15.169:3300/0 conn(0x7f10800d2920 msgr2=0x7f10800d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107ed9d640 1 --2- 172.21.15.122:0/4269708022 >> v2:172.21.15.169:3300/0 conn(0x7f10800d2920 0x7f10800d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107ed9d640 1 -- 172.21.15.122:0/4269708022 >> v1:172.21.15.122:6789/0 conn(0x7f108015e090 legacy=0x7f10800b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.397+0000 7f107ed9d640 1 -- 172.21.15.122:0/4269708022 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10800b3cf0 con 0x7f10800d0650 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.398+0000 7f107ed9d640 1 --2- 172.21.15.122:0/4269708022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d0650 0x7f10800d0aa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f10800d1700 tx=0x7f107802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d5a8e96d3611fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.398+0000 7f107d59a640 1 -- 172.21.15.122:0/4269708022 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078039070 con 0x7f10800d0650 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.398+0000 7f107d59a640 1 -- 172.21.15.122:0/4269708022 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107802cc80 con 0x7f10800d0650 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.398+0000 7f107d59a640 1 -- 172.21.15.122:0/4269708022 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078034660 con 0x7f10800d0650 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 -- 172.21.15.122:0/4269708022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d0650 msgr2=0x7f10800d0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 --2- 172.21.15.122:0/4269708022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d0650 0x7f10800d0aa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f10800d1700 tx=0x7f107802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 -- 172.21.15.122:0/4269708022 shutdown_connections 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 --2- 172.21.15.122:0/4269708022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d0650 0x7f10800d0aa0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 --2- 172.21.15.122:0/4269708022 >> v2:172.21.15.169:3300/0 conn(0x7f10800d2920 0x7f10800d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 -- 172.21.15.122:0/4269708022 >> 172.21.15.122:0/4269708022 conn(0x7f10800d3c80 msgr2=0x7f10800d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 -- 172.21.15.122:0/4269708022 shutdown_connections 2024-03-06T07:25:59.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.399+0000 7f1084833640 1 -- 172.21.15.122:0/4269708022 wait complete. 2024-03-06T07:25:59.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.401+0000 7f1084833640 1 Processor -- start 2024-03-06T07:25:59.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.401+0000 7f1084833640 1 -- start start 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f1084833640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10800d0650 0x7f108014d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f1084833640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 0x7f108015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f1084833640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f108015e090 0x7f10800cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f1084833640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800cee30 con 0x7f10800d2920 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f1084833640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800ceb50 con 0x7f108015e090 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f1084833640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10800b2df0 con 0x7f10800d0650 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 0x7f108015ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 0x7f108015ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59430/0 (socket says 172.21.15.122:59430) 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 -- 172.21.15.122:0/2808797355 learned_addr learned my addr 172.21.15.122:0/2808797355 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107ed9d640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f108015e090 0x7f10800cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107e59c640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10800d0650 0x7f108014d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 -- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10800d0650 msgr2=0x7f108014d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10800d0650 0x7f108014d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 -- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f108015e090 msgr2=0x7f10800cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f108015e090 0x7f10800cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.402+0000 7f107dd9b640 1 -- 172.21.15.122:0/2808797355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108015e8f0 con 0x7f10800d2920 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.403+0000 7f107dd9b640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 0x7f108015ec50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f107000e870 tx=0x7f107000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.403+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070017070 con 0x7f10800d2920 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.403+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107000c940 con 0x7f10800d2920 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.403+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108015f2e0 con 0x7f10800d2920 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.403+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107000cd30 con 0x7f10800d2920 2024-03-06T07:25:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.403+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108015f8a0 con 0x7f10800d2920 2024-03-06T07:25:59.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.404+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108004f330 con 0x7f10800d2920 2024-03-06T07:25:59.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.404+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f107001a070 con 0x7f10800d2920 2024-03-06T07:25:59.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.405+0000 7f1076ffd640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f106006b4d0 0x7f106006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.405+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f107008a070 con 0x7f10800d2920 2024-03-06T07:25:59.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.405+0000 7f107e59c640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f106006b4d0 0x7f106006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.405+0000 7f107e59c640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f106006b4d0 0x7f106006d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1068005880 tx=0x7f1068009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.409+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1070058ef0 con 0x7f10800d2920 2024-03-06T07:25:59.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.603+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10800d1230 con 0x7f106006b4d0 2024-03-06T07:25:59.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.609+0000 7f1076ffd640 1 -- 172.21.15.122:0/2808797355 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252079 (secure 0 0 0) 0x7f10800d1230 con 0x7f106006b4d0 2024-03-06T07:25:59.613 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:25:59.614 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.611+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f106006b4d0 msgr2=0x7f106006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.611+0000 7f1084833640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f106006b4d0 0x7f106006d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1068005880 tx=0x7f1068009040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.611+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 msgr2=0x7f108015ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 0x7f108015ec50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f107000e870 tx=0x7f107000ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 shutdown_connections 2024-03-06T07:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f106006b4d0 0x7f106006d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f108015e090 0x7f10800cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10800d2920 0x7f108015ec50 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 --2- 172.21.15.122:0/2808797355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10800d0650 0x7f108014d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 >> 172.21.15.122:0/2808797355 conn(0x7f10800d3c80 msgr2=0x7f108006cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 shutdown_connections 2024-03-06T07:25:59.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.612+0000 7f1084833640 1 -- 172.21.15.122:0/2808797355 wait complete. 2024-03-06T07:25:59.625 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-03-06T07:25:59.379194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258576,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716144,"statfs":{"total":773094113280,"available":772829331456,"internally_reserved":0,"allocated":9314304,"data_stored":4606892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001432"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675857+0000","last_change":"2024-03-06T07:25:51.678317+0000","last_active":"2024-03-06T07:25:52.675857+0000","last_peered":"2024-03-06T07:25:52.675857+0000","last_clean":"2024-03-06T07:25:52.675857+0000","last_became_active":"2024-03-06T07:25:51.678048+0000","last_became_peered":"2024-03-06T07:25:51.678048+0000","last_unstale":"2024-03-06T07:25:52.675857+0000","last_undegraded":"2024-03-06T07:25:52.675857+0000","last_fullsized":"2024-03-06T07:25:52.675857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:25.036381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005035+0000","last_change":"2024-03-06T07:25:51.680521+0000","last_active":"2024-03-06T07:25:53.005035+0000","last_peered":"2024-03-06T07:25:53.005035+0000","last_clean":"2024-03-06T07:25:53.005035+0000","last_became_active":"2024-03-06T07:25:51.680252+0000","last_became_peered":"2024-03-06T07:25:51.680252+0000","last_unstale":"2024-03-06T07:25:53.005035+0000","last_undegraded":"2024-03-06T07:25:53.005035+0000","last_fullsized":"2024-03-06T07:25:53.005035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:46.814166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005336+0000","last_change":"2024-03-06T07:25:51.674847+0000","last_active":"2024-03-06T07:25:53.005336+0000","last_peered":"2024-03-06T07:25:53.005336+0000","last_clean":"2024-03-06T07:25:53.005336+0000","last_became_active":"2024-03-06T07:25:51.674004+0000","last_became_peered":"2024-03-06T07:25:51.674004+0000","last_unstale":"2024-03-06T07:25:53.005336+0000","last_undegraded":"2024-03-06T07:25:53.005336+0000","last_fullsized":"2024-03-06T07:25:53.005336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:53.438813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801245+0000","last_change":"2024-03-06T07:25:51.671416+0000","last_active":"2024-03-06T07:25:52.801245+0000","last_peered":"2024-03-06T07:25:52.801245+0000","last_clean":"2024-03-06T07:25:52.801245+0000","last_became_active":"2024-03-06T07:25:51.671151+0000","last_became_peered":"2024-03-06T07:25:51.671151+0000","last_unstale":"2024-03-06T07:25:52.801245+0000","last_undegraded":"2024-03-06T07:25:52.801245+0000","last_fullsized":"2024-03-06T07:25:52.801245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:48.621694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675343+0000","last_change":"2024-03-06T07:25:51.682772+0000","last_active":"2024-03-06T07:25:52.675343+0000","last_peered":"2024-03-06T07:25:52.675343+0000","last_clean":"2024-03-06T07:25:52.675343+0000","last_became_active":"2024-03-06T07:25:51.682504+0000","last_became_peered":"2024-03-06T07:25:51.682504+0000","last_unstale":"2024-03-06T07:25:52.675343+0000","last_undegraded":"2024-03-06T07:25:52.675343+0000","last_fullsized":"2024-03-06T07:25:52.675343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005031+0000","last_change":"2024-03-06T07:25:51.675616+0000","last_active":"2024-03-06T07:25:53.005031+0000","last_peered":"2024-03-06T07:25:53.005031+0000","last_clean":"2024-03-06T07:25:53.005031+0000","last_became_active":"2024-03-06T07:25:51.675306+0000","last_became_peered":"2024-03-06T07:25:51.675306+0000","last_unstale":"2024-03-06T07:25:53.005031+0000","last_undegraded":"2024-03-06T07:25:53.005031+0000","last_fullsized":"2024-03-06T07:25:53.005031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800816+0000","last_change":"2024-03-06T07:25:51.670665+0000","last_active":"2024-03-06T07:25:52.800816+0000","last_peered":"2024-03-06T07:25:52.800816+0000","last_clean":"2024-03-06T07:25:52.800816+0000","last_became_active":"2024-03-06T07:25:51.670401+0000","last_became_peered":"2024-03-06T07:25:51.670401+0000","last_unstale":"2024-03-06T07:25:52.800816+0000","last_undegraded":"2024-03-06T07:25:52.800816+0000","last_fullsized":"2024-03-06T07:25:52.800816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:29.492351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004988+0000","last_change":"2024-03-06T07:25:51.681909+0000","last_active":"2024-03-06T07:25:53.004988+0000","last_peered":"2024-03-06T07:25:53.004988+0000","last_clean":"2024-03-06T07:25:53.004988+0000","last_became_active":"2024-03-06T07:25:51.681317+0000","last_became_peered":"2024-03-06T07:25:51.681317+0000","last_unstale":"2024-03-06T07:25:53.004988+0000","last_undegraded":"2024-03-06T07:25:53.004988+0000","last_fullsized":"2024-03-06T07:25:53.004988+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005033+0000","last_change":"2024-03-06T07:25:51.682383+0000","last_active":"2024-03-06T07:25:53.005033+0000","last_peered":"2024-03-06T07:25:53.005033+0000","last_clean":"2024-03-06T07:25:53.005033+0000","last_became_active":"2024-03-06T07:25:51.682090+0000","last_became_peered":"2024-03-06T07:25:51.682090+0000","last_unstale":"2024-03-06T07:25:53.005033+0000","last_undegraded":"2024-03-06T07:25:53.005033+0000","last_fullsized":"2024-03-06T07:25:53.005033+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675059+0000","last_change":"2024-03-06T07:25:51.680536+0000","last_active":"2024-03-06T07:25:52.675059+0000","last_peered":"2024-03-06T07:25:52.675059+0000","last_clean":"2024-03-06T07:25:52.675059+0000","last_became_active":"2024-03-06T07:25:51.679636+0000","last_became_peered":"2024-03-06T07:25:51.679636+0000","last_unstale":"2024-03-06T07:25:52.675059+0000","last_undegraded":"2024-03-06T07:25:52.675059+0000","last_fullsized":"2024-03-06T07:25:52.675059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004844+0000","last_change":"2024-03-06T07:25:51.679595+0000","last_active":"2024-03-06T07:25:53.004844+0000","last_peered":"2024-03-06T07:25:53.004844+0000","last_clean":"2024-03-06T07:25:53.004844+0000","last_became_active":"2024-03-06T07:25:51.679267+0000","last_became_peered":"2024-03-06T07:25:51.679267+0000","last_unstale":"2024-03-06T07:25:53.004844+0000","last_undegraded":"2024-03-06T07:25:53.004844+0000","last_fullsized":"2024-03-06T07:25:53.004844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:36.580554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675672+0000","last_change":"2024-03-06T07:25:51.680345+0000","last_active":"2024-03-06T07:25:52.675672+0000","last_peered":"2024-03-06T07:25:52.675672+0000","last_clean":"2024-03-06T07:25:52.675672+0000","last_became_active":"2024-03-06T07:25:51.680024+0000","last_became_peered":"2024-03-06T07:25:51.680024+0000","last_unstale":"2024-03-06T07:25:52.675672+0000","last_undegraded":"2024-03-06T07:25:52.675672+0000","last_fullsized":"2024-03-06T07:25:52.675672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005403+0000","last_change":"2024-03-06T07:25:51.680932+0000","last_active":"2024-03-06T07:25:53.005403+0000","last_peered":"2024-03-06T07:25:53.005403+0000","last_clean":"2024-03-06T07:25:53.005403+0000","last_became_active":"2024-03-06T07:25:51.680638+0000","last_became_peered":"2024-03-06T07:25:51.680638+0000","last_unstale":"2024-03-06T07:25:53.005403+0000","last_undegraded":"2024-03-06T07:25:53.005403+0000","last_fullsized":"2024-03-06T07:25:53.005403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:48.659969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801037+0000","last_change":"2024-03-06T07:25:51.682488+0000","last_active":"2024-03-06T07:25:52.801037+0000","last_peered":"2024-03-06T07:25:52.801037+0000","last_clean":"2024-03-06T07:25:52.801037+0000","last_became_active":"2024-03-06T07:25:51.682199+0000","last_became_peered":"2024-03-06T07:25:51.682199+0000","last_unstale":"2024-03-06T07:25:52.801037+0000","last_undegraded":"2024-03-06T07:25:52.801037+0000","last_fullsized":"2024-03-06T07:25:52.801037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:04.113709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994217+0000","last_change":"2024-03-06T07:25:51.676566+0000","last_active":"2024-03-06T07:25:52.994217+0000","last_peered":"2024-03-06T07:25:52.994217+0000","last_clean":"2024-03-06T07:25:52.994217+0000","last_became_active":"2024-03-06T07:25:51.676318+0000","last_became_peered":"2024-03-06T07:25:51.676318+0000","last_unstale":"2024-03-06T07:25:52.994217+0000","last_undegraded":"2024-03-06T07:25:52.994217+0000","last_fullsized":"2024-03-06T07:25:52.994217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:53.438813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674313+0000","last_change":"2024-03-06T07:25:51.676990+0000","last_active":"2024-03-06T07:25:52.674313+0000","last_peered":"2024-03-06T07:25:52.674313+0000","last_clean":"2024-03-06T07:25:52.674313+0000","last_became_active":"2024-03-06T07:25:51.676709+0000","last_became_peered":"2024-03-06T07:25:51.676709+0000","last_unstale":"2024-03-06T07:25:52.674313+0000","last_undegraded":"2024-03-06T07:25:52.674313+0000","last_fullsized":"2024-03-06T07:25:52.674313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:21.738142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674764+0000","last_change":"2024-03-06T07:25:51.679322+0000","last_active":"2024-03-06T07:25:52.674764+0000","last_peered":"2024-03-06T07:25:52.674764+0000","last_clean":"2024-03-06T07:25:52.674764+0000","last_became_active":"2024-03-06T07:25:51.679022+0000","last_became_peered":"2024-03-06T07:25:51.679022+0000","last_unstale":"2024-03-06T07:25:52.674764+0000","last_undegraded":"2024-03-06T07:25:52.674764+0000","last_fullsized":"2024-03-06T07:25:52.674764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:33.451809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801077+0000","last_change":"2024-03-06T07:25:51.672332+0000","last_active":"2024-03-06T07:25:52.801077+0000","last_peered":"2024-03-06T07:25:52.801077+0000","last_clean":"2024-03-06T07:25:52.801077+0000","last_became_active":"2024-03-06T07:25:51.672086+0000","last_became_peered":"2024-03-06T07:25:51.672086+0000","last_unstale":"2024-03-06T07:25:52.801077+0000","last_undegraded":"2024-03-06T07:25:52.801077+0000","last_fullsized":"2024-03-06T07:25:52.801077+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:21.788094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671368+0000","last_change":"2024-03-06T07:25:51.680243+0000","last_active":"2024-03-06T07:25:52.671368+0000","last_peered":"2024-03-06T07:25:52.671368+0000","last_clean":"2024-03-06T07:25:52.671368+0000","last_became_active":"2024-03-06T07:25:51.679981+0000","last_became_peered":"2024-03-06T07:25:51.679981+0000","last_unstale":"2024-03-06T07:25:52.671368+0000","last_undegraded":"2024-03-06T07:25:52.671368+0000","last_fullsized":"2024-03-06T07:25:52.671368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:49.195508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801303+0000","last_change":"2024-03-06T07:25:51.681850+0000","last_active":"2024-03-06T07:25:52.801303+0000","last_peered":"2024-03-06T07:25:52.801303+0000","last_clean":"2024-03-06T07:25:52.801303+0000","last_became_active":"2024-03-06T07:25:51.681568+0000","last_became_peered":"2024-03-06T07:25:51.681568+0000","last_unstale":"2024-03-06T07:25:52.801303+0000","last_undegraded":"2024-03-06T07:25:52.801303+0000","last_fullsized":"2024-03-06T07:25:52.801303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:17.662481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801098+0000","last_change":"2024-03-06T07:25:51.681241+0000","last_active":"2024-03-06T07:25:52.801098+0000","last_peered":"2024-03-06T07:25:52.801098+0000","last_clean":"2024-03-06T07:25:52.801098+0000","last_became_active":"2024-03-06T07:25:51.680940+0000","last_became_peered":"2024-03-06T07:25:51.680940+0000","last_unstale":"2024-03-06T07:25:52.801098+0000","last_undegraded":"2024-03-06T07:25:52.801098+0000","last_fullsized":"2024-03-06T07:25:52.801098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:18.980000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004740+0000","last_change":"2024-03-06T07:25:51.681877+0000","last_active":"2024-03-06T07:25:53.004740+0000","last_peered":"2024-03-06T07:25:53.004740+0000","last_clean":"2024-03-06T07:25:53.004740+0000","last_became_active":"2024-03-06T07:25:51.681626+0000","last_became_peered":"2024-03-06T07:25:51.681626+0000","last_unstale":"2024-03-06T07:25:53.004740+0000","last_undegraded":"2024-03-06T07:25:53.004740+0000","last_fullsized":"2024-03-06T07:25:53.004740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:07.418991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674861+0000","last_change":"2024-03-06T07:25:51.683454+0000","last_active":"2024-03-06T07:25:52.674861+0000","last_peered":"2024-03-06T07:25:52.674861+0000","last_clean":"2024-03-06T07:25:52.674861+0000","last_became_active":"2024-03-06T07:25:51.682932+0000","last_became_peered":"2024-03-06T07:25:51.682932+0000","last_unstale":"2024-03-06T07:25:52.674861+0000","last_undegraded":"2024-03-06T07:25:52.674861+0000","last_fullsized":"2024-03-06T07:25:52.674861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800882+0000","last_change":"2024-03-06T07:25:51.684204+0000","last_active":"2024-03-06T07:25:52.800882+0000","last_peered":"2024-03-06T07:25:52.800882+0000","last_clean":"2024-03-06T07:25:52.800882+0000","last_became_active":"2024-03-06T07:25:51.683940+0000","last_became_peered":"2024-03-06T07:25:51.683940+0000","last_unstale":"2024-03-06T07:25:52.800882+0000","last_undegraded":"2024-03-06T07:25:52.800882+0000","last_fullsized":"2024-03-06T07:25:52.800882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:23.573251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004849+0000","last_change":"2024-03-06T07:25:51.680652+0000","last_active":"2024-03-06T07:25:53.004849+0000","last_peered":"2024-03-06T07:25:53.004849+0000","last_clean":"2024-03-06T07:25:53.004849+0000","last_became_active":"2024-03-06T07:25:51.680402+0000","last_became_peered":"2024-03-06T07:25:51.680402+0000","last_unstale":"2024-03-06T07:25:53.004849+0000","last_undegraded":"2024-03-06T07:25:53.004849+0000","last_fullsized":"2024-03-06T07:25:53.004849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:33.343720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005369+0000","last_change":"2024-03-06T07:25:51.682124+0000","last_active":"2024-03-06T07:25:53.005369+0000","last_peered":"2024-03-06T07:25:53.005369+0000","last_clean":"2024-03-06T07:25:53.005369+0000","last_became_active":"2024-03-06T07:25:51.681851+0000","last_became_peered":"2024-03-06T07:25:51.681851+0000","last_unstale":"2024-03-06T07:25:53.005369+0000","last_undegraded":"2024-03-06T07:25:53.005369+0000","last_fullsized":"2024-03-06T07:25:53.005369+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:55.427087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674874+0000","last_change":"2024-03-06T07:25:51.678817+0000","last_active":"2024-03-06T07:25:52.674874+0000","last_peered":"2024-03-06T07:25:52.674874+0000","last_clean":"2024-03-06T07:25:52.674874+0000","last_became_active":"2024-03-06T07:25:51.677774+0000","last_became_peered":"2024-03-06T07:25:51.677774+0000","last_unstale":"2024-03-06T07:25:52.674874+0000","last_undegraded":"2024-03-06T07:25:52.674874+0000","last_fullsized":"2024-03-06T07:25:52.674874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675705+0000","last_change":"2024-03-06T07:25:51.680054+0000","last_active":"2024-03-06T07:25:52.675705+0000","last_peered":"2024-03-06T07:25:52.675705+0000","last_clean":"2024-03-06T07:25:52.675705+0000","last_became_active":"2024-03-06T07:25:51.679787+0000","last_became_peered":"2024-03-06T07:25:51.679787+0000","last_unstale":"2024-03-06T07:25:52.675705+0000","last_undegraded":"2024-03-06T07:25:52.675705+0000","last_fullsized":"2024-03-06T07:25:52.675705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004704+0000","last_change":"2024-03-06T07:25:51.677373+0000","last_active":"2024-03-06T07:25:53.004704+0000","last_peered":"2024-03-06T07:25:53.004704+0000","last_clean":"2024-03-06T07:25:53.004704+0000","last_became_active":"2024-03-06T07:25:51.677137+0000","last_became_peered":"2024-03-06T07:25:51.677137+0000","last_unstale":"2024-03-06T07:25:53.004704+0000","last_undegraded":"2024-03-06T07:25:53.004704+0000","last_fullsized":"2024-03-06T07:25:53.004704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:24.300534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004894+0000","last_change":"2024-03-06T07:25:51.672433+0000","last_active":"2024-03-06T07:25:53.004894+0000","last_peered":"2024-03-06T07:25:53.004894+0000","last_clean":"2024-03-06T07:25:53.004894+0000","last_became_active":"2024-03-06T07:25:51.672152+0000","last_became_peered":"2024-03-06T07:25:51.672152+0000","last_unstale":"2024-03-06T07:25:53.004894+0000","last_undegraded":"2024-03-06T07:25:53.004894+0000","last_fullsized":"2024-03-06T07:25:53.004894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:45.422326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674713+0000","last_change":"2024-03-06T07:25:51.685203+0000","last_active":"2024-03-06T07:25:52.674713+0000","last_peered":"2024-03-06T07:25:52.674713+0000","last_clean":"2024-03-06T07:25:52.674713+0000","last_became_active":"2024-03-06T07:25:51.684174+0000","last_became_peered":"2024-03-06T07:25:51.684174+0000","last_unstale":"2024-03-06T07:25:52.674713+0000","last_undegraded":"2024-03-06T07:25:52.674713+0000","last_fullsized":"2024-03-06T07:25:52.674713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:18.482302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674638+0000","last_change":"2024-03-06T07:25:51.686828+0000","last_active":"2024-03-06T07:25:52.674638+0000","last_peered":"2024-03-06T07:25:52.674638+0000","last_clean":"2024-03-06T07:25:52.674638+0000","last_became_active":"2024-03-06T07:25:51.686529+0000","last_became_peered":"2024-03-06T07:25:51.686529+0000","last_unstale":"2024-03-06T07:25:52.674638+0000","last_undegraded":"2024-03-06T07:25:52.674638+0000","last_fullsized":"2024-03-06T07:25:52.674638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674836+0000","last_change":"2024-03-06T07:25:51.675516+0000","last_active":"2024-03-06T07:25:52.674836+0000","last_peered":"2024-03-06T07:25:52.674836+0000","last_clean":"2024-03-06T07:25:52.674836+0000","last_became_active":"2024-03-06T07:25:51.675166+0000","last_became_peered":"2024-03-06T07:25:51.675166+0000","last_unstale":"2024-03-06T07:25:52.674836+0000","last_undegraded":"2024-03-06T07:25:52.674836+0000","last_fullsized":"2024-03-06T07:25:52.674836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:27.077831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675927+0000","last_change":"2024-03-06T07:25:50.665724+0000","last_active":"2024-03-06T07:25:52.675927+0000","last_peered":"2024-03-06T07:25:52.675927+0000","last_clean":"2024-03-06T07:25:52.675927+0000","last_became_active":"2024-03-06T07:25:50.665446+0000","last_became_peered":"2024-03-06T07:25:50.665446+0000","last_unstale":"2024-03-06T07:25:52.675927+0000","last_undegraded":"2024-03-06T07:25:52.675927+0000","last_fullsized":"2024-03-06T07:25:52.675927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:35.574350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675869+0000","last_change":"2024-03-06T07:25:51.680584+0000","last_active":"2024-03-06T07:25:52.675869+0000","last_peered":"2024-03-06T07:25:52.675869+0000","last_clean":"2024-03-06T07:25:52.675869+0000","last_became_active":"2024-03-06T07:25:51.680260+0000","last_became_peered":"2024-03-06T07:25:51.680260+0000","last_unstale":"2024-03-06T07:25:52.675869+0000","last_undegraded":"2024-03-06T07:25:52.675869+0000","last_fullsized":"2024-03-06T07:25:52.675869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.712380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675077+0000","last_change":"2024-03-06T07:25:50.667936+0000","last_active":"2024-03-06T07:25:52.675077+0000","last_peered":"2024-03-06T07:25:52.675077+0000","last_clean":"2024-03-06T07:25:52.675077+0000","last_became_active":"2024-03-06T07:25:50.667577+0000","last_became_peered":"2024-03-06T07:25:50.667577+0000","last_unstale":"2024-03-06T07:25:52.675077+0000","last_undegraded":"2024-03-06T07:25:52.675077+0000","last_fullsized":"2024-03-06T07:25:52.675077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:41.743623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675771+0000","last_change":"2024-03-06T07:25:51.685616+0000","last_active":"2024-03-06T07:25:52.675771+0000","last_peered":"2024-03-06T07:25:52.675771+0000","last_clean":"2024-03-06T07:25:52.675771+0000","last_became_active":"2024-03-06T07:25:51.685322+0000","last_became_peered":"2024-03-06T07:25:51.685322+0000","last_unstale":"2024-03-06T07:25:52.675771+0000","last_undegraded":"2024-03-06T07:25:52.675771+0000","last_fullsized":"2024-03-06T07:25:52.675771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671402+0000","last_change":"2024-03-06T07:25:50.667866+0000","last_active":"2024-03-06T07:25:52.671402+0000","last_peered":"2024-03-06T07:25:52.671402+0000","last_clean":"2024-03-06T07:25:52.671402+0000","last_became_active":"2024-03-06T07:25:50.667614+0000","last_became_peered":"2024-03-06T07:25:50.667614+0000","last_unstale":"2024-03-06T07:25:52.671402+0000","last_undegraded":"2024-03-06T07:25:52.671402+0000","last_fullsized":"2024-03-06T07:25:52.671402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:55.559459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005458+0000","last_change":"2024-03-06T07:25:51.674998+0000","last_active":"2024-03-06T07:25:53.005458+0000","last_peered":"2024-03-06T07:25:53.005458+0000","last_clean":"2024-03-06T07:25:53.005458+0000","last_became_active":"2024-03-06T07:25:51.674720+0000","last_became_peered":"2024-03-06T07:25:51.674720+0000","last_unstale":"2024-03-06T07:25:53.005458+0000","last_undegraded":"2024-03-06T07:25:53.005458+0000","last_fullsized":"2024-03-06T07:25:53.005458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:23.266106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671229+0000","last_change":"2024-03-06T07:25:50.667000+0000","last_active":"2024-03-06T07:25:52.671229+0000","last_peered":"2024-03-06T07:25:52.671229+0000","last_clean":"2024-03-06T07:25:52.671229+0000","last_became_active":"2024-03-06T07:25:50.666740+0000","last_became_peered":"2024-03-06T07:25:50.666740+0000","last_unstale":"2024-03-06T07:25:52.671229+0000","last_undegraded":"2024-03-06T07:25:52.671229+0000","last_fullsized":"2024-03-06T07:25:52.671229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:37.494888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004961+0000","last_change":"2024-03-06T07:25:51.670449+0000","last_active":"2024-03-06T07:25:53.004961+0000","last_peered":"2024-03-06T07:25:53.004961+0000","last_clean":"2024-03-06T07:25:53.004961+0000","last_became_active":"2024-03-06T07:25:51.670204+0000","last_became_peered":"2024-03-06T07:25:51.670204+0000","last_unstale":"2024-03-06T07:25:53.004961+0000","last_undegraded":"2024-03-06T07:25:53.004961+0000","last_fullsized":"2024-03-06T07:25:53.004961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:09.510502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675637+0000","last_change":"2024-03-06T07:25:50.665732+0000","last_active":"2024-03-06T07:25:52.675637+0000","last_peered":"2024-03-06T07:25:52.675637+0000","last_clean":"2024-03-06T07:25:52.675637+0000","last_became_active":"2024-03-06T07:25:50.664895+0000","last_became_peered":"2024-03-06T07:25:50.664895+0000","last_unstale":"2024-03-06T07:25:52.675637+0000","last_undegraded":"2024-03-06T07:25:52.675637+0000","last_fullsized":"2024-03-06T07:25:52.675637+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:06.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675488+0000","last_change":"2024-03-06T07:25:51.685945+0000","last_active":"2024-03-06T07:25:52.675488+0000","last_peered":"2024-03-06T07:25:52.675488+0000","last_clean":"2024-03-06T07:25:52.675488+0000","last_became_active":"2024-03-06T07:25:51.685619+0000","last_became_peered":"2024-03-06T07:25:51.685619+0000","last_unstale":"2024-03-06T07:25:52.675488+0000","last_undegraded":"2024-03-06T07:25:52.675488+0000","last_fullsized":"2024-03-06T07:25:52.675488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674446+0000","last_change":"2024-03-06T07:25:50.665220+0000","last_active":"2024-03-06T07:25:52.674446+0000","last_peered":"2024-03-06T07:25:52.674446+0000","last_clean":"2024-03-06T07:25:52.674446+0000","last_became_active":"2024-03-06T07:25:50.664947+0000","last_became_peered":"2024-03-06T07:25:50.664947+0000","last_unstale":"2024-03-06T07:25:52.674446+0000","last_undegraded":"2024-03-06T07:25:52.674446+0000","last_fullsized":"2024-03-06T07:25:52.674446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:23.294330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004723+0000","last_change":"2024-03-06T07:25:51.681340+0000","last_active":"2024-03-06T07:25:53.004723+0000","last_peered":"2024-03-06T07:25:53.004723+0000","last_clean":"2024-03-06T07:25:53.004723+0000","last_became_active":"2024-03-06T07:25:51.681056+0000","last_became_peered":"2024-03-06T07:25:51.681056+0000","last_unstale":"2024-03-06T07:25:53.004723+0000","last_undegraded":"2024-03-06T07:25:53.004723+0000","last_fullsized":"2024-03-06T07:25:53.004723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:12.789933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.670990+0000","last_change":"2024-03-06T07:25:50.665789+0000","last_active":"2024-03-06T07:25:52.670990+0000","last_peered":"2024-03-06T07:25:52.670990+0000","last_clean":"2024-03-06T07:25:52.670990+0000","last_became_active":"2024-03-06T07:25:50.665358+0000","last_became_peered":"2024-03-06T07:25:50.665358+0000","last_unstale":"2024-03-06T07:25:52.670990+0000","last_undegraded":"2024-03-06T07:25:52.670990+0000","last_fullsized":"2024-03-06T07:25:52.670990+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:26.071628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994238+0000","last_change":"2024-03-06T07:25:51.674367+0000","last_active":"2024-03-06T07:25:52.994238+0000","last_peered":"2024-03-06T07:25:52.994238+0000","last_clean":"2024-03-06T07:25:52.994238+0000","last_became_active":"2024-03-06T07:25:51.674050+0000","last_became_peered":"2024-03-06T07:25:51.674050+0000","last_unstale":"2024-03-06T07:25:52.994238+0000","last_undegraded":"2024-03-06T07:25:52.994238+0000","last_fullsized":"2024-03-06T07:25:52.994238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:55.427087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004492+0000","last_change":"2024-03-06T07:25:50.664211+0000","last_active":"2024-03-06T07:25:53.004492+0000","last_peered":"2024-03-06T07:25:53.004492+0000","last_clean":"2024-03-06T07:25:53.004492+0000","last_became_active":"2024-03-06T07:25:50.663925+0000","last_became_peered":"2024-03-06T07:25:50.663925+0000","last_unstale":"2024-03-06T07:25:53.004492+0000","last_undegraded":"2024-03-06T07:25:53.004492+0000","last_fullsized":"2024-03-06T07:25:53.004492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:29.439664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994502+0000","last_change":"2024-03-06T07:25:51.681731+0000","last_active":"2024-03-06T07:25:52.994502+0000","last_peered":"2024-03-06T07:25:52.994502+0000","last_clean":"2024-03-06T07:25:52.994502+0000","last_became_active":"2024-03-06T07:25:51.681445+0000","last_became_peered":"2024-03-06T07:25:51.681445+0000","last_unstale":"2024-03-06T07:25:52.994502+0000","last_undegraded":"2024-03-06T07:25:52.994502+0000","last_fullsized":"2024-03-06T07:25:52.994502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:51.665911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.676005+0000","last_change":"2024-03-06T07:25:50.659033+0000","last_active":"2024-03-06T07:25:52.676005+0000","last_peered":"2024-03-06T07:25:52.676005+0000","last_clean":"2024-03-06T07:25:52.676005+0000","last_became_active":"2024-03-06T07:25:50.658341+0000","last_became_peered":"2024-03-06T07:25:50.658341+0000","last_unstale":"2024-03-06T07:25:52.676005+0000","last_undegraded":"2024-03-06T07:25:52.676005+0000","last_fullsized":"2024-03-06T07:25:52.676005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:44.416122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005205+0000","last_change":"2024-03-06T07:25:51.675730+0000","last_active":"2024-03-06T07:25:53.005205+0000","last_peered":"2024-03-06T07:25:53.005205+0000","last_clean":"2024-03-06T07:25:53.005205+0000","last_became_active":"2024-03-06T07:25:51.675375+0000","last_became_peered":"2024-03-06T07:25:51.675375+0000","last_unstale":"2024-03-06T07:25:53.005205+0000","last_undegraded":"2024-03-06T07:25:53.005205+0000","last_fullsized":"2024-03-06T07:25:53.005205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:56.565662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675117+0000","last_change":"2024-03-06T07:25:50.666708+0000","last_active":"2024-03-06T07:25:52.675117+0000","last_peered":"2024-03-06T07:25:52.675117+0000","last_clean":"2024-03-06T07:25:52.675117+0000","last_became_active":"2024-03-06T07:25:50.666385+0000","last_became_peered":"2024-03-06T07:25:50.666385+0000","last_unstale":"2024-03-06T07:25:52.675117+0000","last_undegraded":"2024-03-06T07:25:52.675117+0000","last_fullsized":"2024-03-06T07:25:52.675117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:05.567054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671255+0000","last_change":"2024-03-06T07:25:51.675676+0000","last_active":"2024-03-06T07:25:52.671255+0000","last_peered":"2024-03-06T07:25:52.671255+0000","last_clean":"2024-03-06T07:25:52.671255+0000","last_became_active":"2024-03-06T07:25:51.675431+0000","last_became_peered":"2024-03-06T07:25:51.675431+0000","last_unstale":"2024-03-06T07:25:52.671255+0000","last_undegraded":"2024-03-06T07:25:52.671255+0000","last_fullsized":"2024-03-06T07:25:52.671255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:52.903718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675236+0000","last_change":"2024-03-06T07:25:50.658006+0000","last_active":"2024-03-06T07:25:52.675236+0000","last_peered":"2024-03-06T07:25:52.675236+0000","last_clean":"2024-03-06T07:25:52.675236+0000","last_became_active":"2024-03-06T07:25:50.657716+0000","last_became_peered":"2024-03-06T07:25:50.657716+0000","last_unstale":"2024-03-06T07:25:52.675236+0000","last_undegraded":"2024-03-06T07:25:52.675236+0000","last_fullsized":"2024-03-06T07:25:52.675236+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:52.828417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674944+0000","last_change":"2024-03-06T07:25:51.686127+0000","last_active":"2024-03-06T07:25:52.674944+0000","last_peered":"2024-03-06T07:25:52.674944+0000","last_clean":"2024-03-06T07:25:52.674944+0000","last_became_active":"2024-03-06T07:25:51.685846+0000","last_became_peered":"2024-03-06T07:25:51.685846+0000","last_unstale":"2024-03-06T07:25:52.674944+0000","last_undegraded":"2024-03-06T07:25:52.674944+0000","last_fullsized":"2024-03-06T07:25:52.674944+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005222+0000","last_change":"2024-03-06T07:25:50.668266+0000","last_active":"2024-03-06T07:25:53.005222+0000","last_peered":"2024-03-06T07:25:53.005222+0000","last_clean":"2024-03-06T07:25:53.005222+0000","last_became_active":"2024-03-06T07:25:50.667485+0000","last_became_peered":"2024-03-06T07:25:50.667485+0000","last_unstale":"2024-03-06T07:25:53.005222+0000","last_undegraded":"2024-03-06T07:25:53.005222+0000","last_fullsized":"2024-03-06T07:25:53.005222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:25.507135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671140+0000","last_change":"2024-03-06T07:25:51.679153+0000","last_active":"2024-03-06T07:25:52.671140+0000","last_peered":"2024-03-06T07:25:52.671140+0000","last_clean":"2024-03-06T07:25:52.671140+0000","last_became_active":"2024-03-06T07:25:51.678289+0000","last_became_peered":"2024-03-06T07:25:51.678289+0000","last_unstale":"2024-03-06T07:25:52.671140+0000","last_undegraded":"2024-03-06T07:25:52.671140+0000","last_fullsized":"2024-03-06T07:25:52.671140+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:15.538278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004757+0000","last_change":"2024-03-06T07:25:50.665487+0000","last_active":"2024-03-06T07:25:53.004757+0000","last_peered":"2024-03-06T07:25:53.004757+0000","last_clean":"2024-03-06T07:25:53.004757+0000","last_became_active":"2024-03-06T07:25:50.665171+0000","last_became_peered":"2024-03-06T07:25:50.665171+0000","last_unstale":"2024-03-06T07:25:53.004757+0000","last_undegraded":"2024-03-06T07:25:53.004757+0000","last_fullsized":"2024-03-06T07:25:53.004757+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:21.303434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005362+0000","last_change":"2024-03-06T07:25:51.680433+0000","last_active":"2024-03-06T07:25:53.005362+0000","last_peered":"2024-03-06T07:25:53.005362+0000","last_clean":"2024-03-06T07:25:53.005362+0000","last_became_active":"2024-03-06T07:25:51.680124+0000","last_became_peered":"2024-03-06T07:25:51.680124+0000","last_unstale":"2024-03-06T07:25:53.005362+0000","last_undegraded":"2024-03-06T07:25:53.005362+0000","last_fullsized":"2024-03-06T07:25:53.005362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:51.665911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994254+0000","last_change":"2024-03-06T07:25:50.660720+0000","last_active":"2024-03-06T07:25:52.994254+0000","last_peered":"2024-03-06T07:25:52.994254+0000","last_clean":"2024-03-06T07:25:52.994254+0000","last_became_active":"2024-03-06T07:25:50.660452+0000","last_became_peered":"2024-03-06T07:25:50.660452+0000","last_unstale":"2024-03-06T07:25:52.994254+0000","last_undegraded":"2024-03-06T07:25:52.994254+0000","last_fullsized":"2024-03-06T07:25:52.994254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:02.606511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004561+0000","last_change":"2024-03-06T07:25:51.671565+0000","last_active":"2024-03-06T07:25:53.004561+0000","last_peered":"2024-03-06T07:25:53.004561+0000","last_clean":"2024-03-06T07:25:53.004561+0000","last_became_active":"2024-03-06T07:25:51.671341+0000","last_became_peered":"2024-03-06T07:25:51.671341+0000","last_unstale":"2024-03-06T07:25:53.004561+0000","last_undegraded":"2024-03-06T07:25:53.004561+0000","last_fullsized":"2024-03-06T07:25:53.004561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:53.834620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005007+0000","last_change":"2024-03-06T07:25:50.667664+0000","last_active":"2024-03-06T07:25:53.005007+0000","last_peered":"2024-03-06T07:25:53.005007+0000","last_clean":"2024-03-06T07:25:53.005007+0000","last_became_active":"2024-03-06T07:25:50.667347+0000","last_became_peered":"2024-03-06T07:25:50.667347+0000","last_unstale":"2024-03-06T07:25:53.005007+0000","last_undegraded":"2024-03-06T07:25:53.005007+0000","last_fullsized":"2024-03-06T07:25:53.005007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:36.841035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004529+0000","last_change":"2024-03-06T07:25:51.680371+0000","last_active":"2024-03-06T07:25:53.004529+0000","last_peered":"2024-03-06T07:25:53.004529+0000","last_clean":"2024-03-06T07:25:53.004529+0000","last_became_active":"2024-03-06T07:25:51.680102+0000","last_became_peered":"2024-03-06T07:25:51.680102+0000","last_unstale":"2024-03-06T07:25:53.004529+0000","last_undegraded":"2024-03-06T07:25:53.004529+0000","last_fullsized":"2024-03-06T07:25:53.004529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:42.749827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.670947+0000","last_change":"2024-03-06T07:25:50.663501+0000","last_active":"2024-03-06T07:25:52.670947+0000","last_peered":"2024-03-06T07:25:52.670947+0000","last_clean":"2024-03-06T07:25:52.670947+0000","last_became_active":"2024-03-06T07:25:50.663232+0000","last_became_peered":"2024-03-06T07:25:50.663232+0000","last_unstale":"2024-03-06T07:25:52.670947+0000","last_undegraded":"2024-03-06T07:25:52.670947+0000","last_fullsized":"2024-03-06T07:25:52.670947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:49.142433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005373+0000","last_change":"2024-03-06T07:25:51.675519+0000","last_active":"2024-03-06T07:25:53.005373+0000","last_peered":"2024-03-06T07:25:53.005373+0000","last_clean":"2024-03-06T07:25:53.005373+0000","last_became_active":"2024-03-06T07:25:51.675270+0000","last_became_peered":"2024-03-06T07:25:51.675270+0000","last_unstale":"2024-03-06T07:25:53.005373+0000","last_undegraded":"2024-03-06T07:25:53.005373+0000","last_fullsized":"2024-03-06T07:25:53.005373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:24.533672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.676059+0000","last_change":"2024-03-06T07:25:50.659845+0000","last_active":"2024-03-06T07:25:52.676059+0000","last_peered":"2024-03-06T07:25:52.676059+0000","last_clean":"2024-03-06T07:25:52.676059+0000","last_became_active":"2024-03-06T07:25:50.659527+0000","last_became_peered":"2024-03-06T07:25:50.659527+0000","last_unstale":"2024-03-06T07:25:52.676059+0000","last_undegraded":"2024-03-06T07:25:52.676059+0000","last_fullsized":"2024-03-06T07:25:52.676059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:32.337516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675974+0000","last_change":"2024-03-06T07:25:51.673226+0000","last_active":"2024-03-06T07:25:52.675974+0000","last_peered":"2024-03-06T07:25:52.675974+0000","last_clean":"2024-03-06T07:25:52.675974+0000","last_became_active":"2024-03-06T07:25:51.672946+0000","last_became_peered":"2024-03-06T07:25:51.672946+0000","last_unstale":"2024-03-06T07:25:52.675974+0000","last_undegraded":"2024-03-06T07:25:52.675974+0000","last_fullsized":"2024-03-06T07:25:52.675974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:15.538278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671294+0000","last_change":"2024-03-06T07:25:50.668525+0000","last_active":"2024-03-06T07:25:52.671294+0000","last_peered":"2024-03-06T07:25:52.671294+0000","last_clean":"2024-03-06T07:25:52.671294+0000","last_became_active":"2024-03-06T07:25:50.668291+0000","last_became_peered":"2024-03-06T07:25:50.668291+0000","last_unstale":"2024-03-06T07:25:52.671294+0000","last_undegraded":"2024-03-06T07:25:52.671294+0000","last_fullsized":"2024-03-06T07:25:52.671294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:31.002940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675006+0000","last_change":"2024-03-06T07:25:51.683425+0000","last_active":"2024-03-06T07:25:52.675006+0000","last_peered":"2024-03-06T07:25:52.675006+0000","last_clean":"2024-03-06T07:25:52.675006+0000","last_became_active":"2024-03-06T07:25:51.683168+0000","last_became_peered":"2024-03-06T07:25:51.683168+0000","last_unstale":"2024-03-06T07:25:52.675006+0000","last_undegraded":"2024-03-06T07:25:52.675006+0000","last_fullsized":"2024-03-06T07:25:52.675006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:33.214315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671310+0000","last_change":"2024-03-06T07:25:50.665805+0000","last_active":"2024-03-06T07:25:52.671310+0000","last_peered":"2024-03-06T07:25:52.671310+0000","last_clean":"2024-03-06T07:25:52.671310+0000","last_became_active":"2024-03-06T07:25:50.665533+0000","last_became_peered":"2024-03-06T07:25:50.665533+0000","last_unstale":"2024-03-06T07:25:52.671310+0000","last_undegraded":"2024-03-06T07:25:52.671310+0000","last_fullsized":"2024-03-06T07:25:52.671310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:32.208112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801202+0000","last_change":"2024-03-06T07:25:51.670633+0000","last_active":"2024-03-06T07:25:52.801202+0000","last_peered":"2024-03-06T07:25:52.801202+0000","last_clean":"2024-03-06T07:25:52.801202+0000","last_became_active":"2024-03-06T07:25:51.670383+0000","last_became_peered":"2024-03-06T07:25:51.670383+0000","last_unstale":"2024-03-06T07:25:52.801202+0000","last_undegraded":"2024-03-06T07:25:52.801202+0000","last_fullsized":"2024-03-06T07:25:52.801202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:30.342476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994321+0000","last_change":"2024-03-06T07:25:50.665180+0000","last_active":"2024-03-06T07:25:52.994321+0000","last_peered":"2024-03-06T07:25:52.994321+0000","last_clean":"2024-03-06T07:25:52.994321+0000","last_became_active":"2024-03-06T07:25:50.664910+0000","last_became_peered":"2024-03-06T07:25:50.664910+0000","last_unstale":"2024-03-06T07:25:52.994321+0000","last_undegraded":"2024-03-06T07:25:52.994321+0000","last_fullsized":"2024-03-06T07:25:52.994321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:52.993794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004875+0000","last_change":"2024-03-06T07:25:51.676865+0000","last_active":"2024-03-06T07:25:53.004875+0000","last_peered":"2024-03-06T07:25:53.004875+0000","last_clean":"2024-03-06T07:25:53.004875+0000","last_became_active":"2024-03-06T07:25:51.676514+0000","last_became_peered":"2024-03-06T07:25:51.676514+0000","last_unstale":"2024-03-06T07:25:53.004875+0000","last_undegraded":"2024-03-06T07:25:53.004875+0000","last_fullsized":"2024-03-06T07:25:53.004875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:16.738182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.801155+0000","last_change":"2024-03-06T07:25:50.662536+0000","last_active":"2024-03-06T07:25:52.801155+0000","last_peered":"2024-03-06T07:25:52.801155+0000","last_clean":"2024-03-06T07:25:52.801155+0000","last_became_active":"2024-03-06T07:25:50.662248+0000","last_became_peered":"2024-03-06T07:25:50.662248+0000","last_unstale":"2024-03-06T07:25:52.801155+0000","last_undegraded":"2024-03-06T07:25:52.801155+0000","last_fullsized":"2024-03-06T07:25:52.801155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:26.050173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800815+0000","last_change":"2024-03-06T07:25:51.675362+0000","last_active":"2024-03-06T07:25:52.800815+0000","last_peered":"2024-03-06T07:25:52.800815+0000","last_clean":"2024-03-06T07:25:52.800815+0000","last_became_active":"2024-03-06T07:25:51.674936+0000","last_became_peered":"2024-03-06T07:25:51.674936+0000","last_unstale":"2024-03-06T07:25:52.800815+0000","last_undegraded":"2024-03-06T07:25:52.800815+0000","last_fullsized":"2024-03-06T07:25:52.800815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:16.568032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.800780+0000","last_change":"2024-03-06T07:25:50.660498+0000","last_active":"2024-03-06T07:25:52.800780+0000","last_peered":"2024-03-06T07:25:52.800780+0000","last_clean":"2024-03-06T07:25:52.800780+0000","last_became_active":"2024-03-06T07:25:50.660246+0000","last_became_peered":"2024-03-06T07:25:50.660246+0000","last_unstale":"2024-03-06T07:25:52.800780+0000","last_undegraded":"2024-03-06T07:25:52.800780+0000","last_fullsized":"2024-03-06T07:25:52.800780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.278410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004907+0000","last_change":"2024-03-06T07:25:51.680574+0000","last_active":"2024-03-06T07:25:53.004907+0000","last_peered":"2024-03-06T07:25:53.004907+0000","last_clean":"2024-03-06T07:25:53.004907+0000","last_became_active":"2024-03-06T07:25:51.680281+0000","last_became_peered":"2024-03-06T07:25:51.680281+0000","last_unstale":"2024-03-06T07:25:53.004907+0000","last_undegraded":"2024-03-06T07:25:53.004907+0000","last_fullsized":"2024-03-06T07:25:53.004907+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:50.148637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.994090+0000","last_change":"2024-03-06T07:25:50.666257+0000","last_active":"2024-03-06T07:25:52.994090+0000","last_peered":"2024-03-06T07:25:52.994090+0000","last_clean":"2024-03-06T07:25:52.994090+0000","last_became_active":"2024-03-06T07:25:50.664875+0000","last_became_peered":"2024-03-06T07:25:50.664875+0000","last_unstale":"2024-03-06T07:25:52.994090+0000","last_undegraded":"2024-03-06T07:25:52.994090+0000","last_fullsized":"2024-03-06T07:25:52.994090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.612866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674975+0000","last_change":"2024-03-06T07:25:51.678510+0000","last_active":"2024-03-06T07:25:52.674975+0000","last_peered":"2024-03-06T07:25:52.674975+0000","last_clean":"2024-03-06T07:25:52.674975+0000","last_became_active":"2024-03-06T07:25:51.678049+0000","last_became_peered":"2024-03-06T07:25:51.678049+0000","last_unstale":"2024-03-06T07:25:52.674975+0000","last_undegraded":"2024-03-06T07:25:52.674975+0000","last_fullsized":"2024-03-06T07:25:52.674975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004908+0000","last_change":"2024-03-06T07:25:50.664028+0000","last_active":"2024-03-06T07:25:53.004908+0000","last_peered":"2024-03-06T07:25:53.004908+0000","last_clean":"2024-03-06T07:25:53.004908+0000","last_became_active":"2024-03-06T07:25:50.663698+0000","last_became_peered":"2024-03-06T07:25:50.663698+0000","last_unstale":"2024-03-06T07:25:53.004908+0000","last_undegraded":"2024-03-06T07:25:53.004908+0000","last_fullsized":"2024-03-06T07:25:53.004908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:54.976768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005163+0000","last_change":"2024-03-06T07:25:51.674498+0000","last_active":"2024-03-06T07:25:53.005163+0000","last_peered":"2024-03-06T07:25:53.005163+0000","last_clean":"2024-03-06T07:25:53.005163+0000","last_became_active":"2024-03-06T07:25:51.673628+0000","last_became_peered":"2024-03-06T07:25:51.673628+0000","last_unstale":"2024-03-06T07:25:53.005163+0000","last_undegraded":"2024-03-06T07:25:53.005163+0000","last_fullsized":"2024-03-06T07:25:53.005163+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:41.203954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005010+0000","last_change":"2024-03-06T07:25:50.665322+0000","last_active":"2024-03-06T07:25:53.005010+0000","last_peered":"2024-03-06T07:25:53.005010+0000","last_clean":"2024-03-06T07:25:53.005010+0000","last_became_active":"2024-03-06T07:25:50.665051+0000","last_became_peered":"2024-03-06T07:25:50.665051+0000","last_unstale":"2024-03-06T07:25:53.005010+0000","last_undegraded":"2024-03-06T07:25:53.005010+0000","last_fullsized":"2024-03-06T07:25:53.005010+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:03.252965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675860+0000","last_change":"2024-03-06T07:25:51.670848+0000","last_active":"2024-03-06T07:25:52.675860+0000","last_peered":"2024-03-06T07:25:52.675860+0000","last_clean":"2024-03-06T07:25:52.675860+0000","last_became_active":"2024-03-06T07:25:51.670601+0000","last_became_peered":"2024-03-06T07:25:51.670601+0000","last_unstale":"2024-03-06T07:25:52.675860+0000","last_undegraded":"2024-03-06T07:25:52.675860+0000","last_fullsized":"2024-03-06T07:25:52.675860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:42.061998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675408+0000","last_change":"2024-03-06T07:25:50.659022+0000","last_active":"2024-03-06T07:25:52.675408+0000","last_peered":"2024-03-06T07:25:52.675408+0000","last_clean":"2024-03-06T07:25:52.675408+0000","last_became_active":"2024-03-06T07:25:50.658737+0000","last_became_peered":"2024-03-06T07:25:50.658737+0000","last_unstale":"2024-03-06T07:25:52.675408+0000","last_undegraded":"2024-03-06T07:25:52.675408+0000","last_fullsized":"2024-03-06T07:25:52.675408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:41.743623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675106+0000","last_change":"2024-03-06T07:25:51.685417+0000","last_active":"2024-03-06T07:25:52.675106+0000","last_peered":"2024-03-06T07:25:52.675106+0000","last_clean":"2024-03-06T07:25:52.675106+0000","last_became_active":"2024-03-06T07:25:51.685153+0000","last_became_peered":"2024-03-06T07:25:51.685153+0000","last_unstale":"2024-03-06T07:25:52.675106+0000","last_undegraded":"2024-03-06T07:25:52.675106+0000","last_fullsized":"2024-03-06T07:25:52.675106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:32.009144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675812+0000","last_change":"2024-03-06T07:25:50.658414+0000","last_active":"2024-03-06T07:25:52.675812+0000","last_peered":"2024-03-06T07:25:52.675812+0000","last_clean":"2024-03-06T07:25:52.675812+0000","last_became_active":"2024-03-06T07:25:50.658117+0000","last_became_peered":"2024-03-06T07:25:50.658117+0000","last_unstale":"2024-03-06T07:25:52.675812+0000","last_undegraded":"2024-03-06T07:25:52.675812+0000","last_fullsized":"2024-03-06T07:25:52.675812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:06.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005316+0000","last_change":"2024-03-06T07:25:51.680659+0000","last_active":"2024-03-06T07:25:53.005316+0000","last_peered":"2024-03-06T07:25:53.005316+0000","last_clean":"2024-03-06T07:25:53.005316+0000","last_became_active":"2024-03-06T07:25:51.680365+0000","last_became_peered":"2024-03-06T07:25:51.680365+0000","last_unstale":"2024-03-06T07:25:53.005316+0000","last_undegraded":"2024-03-06T07:25:53.005316+0000","last_fullsized":"2024-03-06T07:25:53.005316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:52.693088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671168+0000","last_change":"2024-03-06T07:25:50.664527+0000","last_active":"2024-03-06T07:25:52.671168+0000","last_peered":"2024-03-06T07:25:52.671168+0000","last_clean":"2024-03-06T07:25:52.671168+0000","last_became_active":"2024-03-06T07:25:50.664117+0000","last_became_peered":"2024-03-06T07:25:50.664117+0000","last_unstale":"2024-03-06T07:25:52.671168+0000","last_undegraded":"2024-03-06T07:25:52.671168+0000","last_fullsized":"2024-03-06T07:25:52.671168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:15.731978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.671218+0000","last_change":"2024-03-06T07:25:51.680138+0000","last_active":"2024-03-06T07:25:52.671218+0000","last_peered":"2024-03-06T07:25:52.671218+0000","last_clean":"2024-03-06T07:25:52.671218+0000","last_became_active":"2024-03-06T07:25:51.679882+0000","last_became_peered":"2024-03-06T07:25:51.679882+0000","last_unstale":"2024-03-06T07:25:52.671218+0000","last_undegraded":"2024-03-06T07:25:52.671218+0000","last_fullsized":"2024-03-06T07:25:52.671218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:17.142904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674507+0000","last_change":"2024-03-06T07:25:50.664868+0000","last_active":"2024-03-06T07:25:52.674507+0000","last_peered":"2024-03-06T07:25:52.674507+0000","last_clean":"2024-03-06T07:25:52.674507+0000","last_became_active":"2024-03-06T07:25:50.664449+0000","last_became_peered":"2024-03-06T07:25:50.664449+0000","last_unstale":"2024-03-06T07:25:52.674507+0000","last_undegraded":"2024-03-06T07:25:52.674507+0000","last_fullsized":"2024-03-06T07:25:52.674507+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:05.567054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005047+0000","last_change":"2024-03-06T07:25:51.681998+0000","last_active":"2024-03-06T07:25:53.005047+0000","last_peered":"2024-03-06T07:25:53.005047+0000","last_clean":"2024-03-06T07:25:53.005047+0000","last_became_active":"2024-03-06T07:25:51.681705+0000","last_became_peered":"2024-03-06T07:25:51.681705+0000","last_unstale":"2024-03-06T07:25:53.005047+0000","last_undegraded":"2024-03-06T07:25:53.005047+0000","last_fullsized":"2024-03-06T07:25:53.005047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:29.536548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675118+0000","last_change":"2024-03-06T07:25:50.657136+0000","last_active":"2024-03-06T07:25:52.675118+0000","last_peered":"2024-03-06T07:25:52.675118+0000","last_clean":"2024-03-06T07:25:52.675118+0000","last_became_active":"2024-03-06T07:25:50.656776+0000","last_became_peered":"2024-03-06T07:25:50.656776+0000","last_unstale":"2024-03-06T07:25:52.675118+0000","last_undegraded":"2024-03-06T07:25:52.675118+0000","last_fullsized":"2024-03-06T07:25:52.675118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:23.294330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.675003+0000","last_change":"2024-03-06T07:25:51.670201+0000","last_active":"2024-03-06T07:25:52.675003+0000","last_peered":"2024-03-06T07:25:52.675003+0000","last_clean":"2024-03-06T07:25:52.675003+0000","last_became_active":"2024-03-06T07:25:51.669930+0000","last_became_peered":"2024-03-06T07:25:51.669930+0000","last_unstale":"2024-03-06T07:25:52.675003+0000","last_undegraded":"2024-03-06T07:25:52.675003+0000","last_fullsized":"2024-03-06T07:25:52.675003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:38.501092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:52.674310+0000","last_change":"2024-03-06T07:25:50.666076+0000","last_active":"2024-03-06T07:25:52.674310+0000","last_peered":"2024-03-06T07:25:52.674310+0000","last_clean":"2024-03-06T07:25:52.674310+0000","last_became_active":"2024-03-06T07:25:50.665365+0000","last_became_peered":"2024-03-06T07:25:50.665365+0000","last_unstale":"2024-03-06T07:25:52.674310+0000","last_undegraded":"2024-03-06T07:25:52.674310+0000","last_fullsized":"2024-03-06T07:25:52.674310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:52.828417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.004803+0000","last_change":"2024-03-06T07:25:51.674771+0000","last_active":"2024-03-06T07:25:53.004803+0000","last_peered":"2024-03-06T07:25:53.004803+0000","last_clean":"2024-03-06T07:25:53.004803+0000","last_became_active":"2024-03-06T07:25:51.674526+0000","last_became_peered":"2024-03-06T07:25:51.674526+0000","last_unstale":"2024-03-06T07:25:53.004803+0000","last_undegraded":"2024-03-06T07:25:53.004803+0000","last_fullsized":"2024-03-06T07:25:53.004803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:50.643283+0000","last_clean_scrub_stamp":"2024-03-06T07:25:50.643283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:06.573258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005403+0000","last_change":"2024-03-06T07:25:50.665597+0000","last_active":"2024-03-06T07:25:53.005403+0000","last_peered":"2024-03-06T07:25:53.005403+0000","last_clean":"2024-03-06T07:25:53.005403+0000","last_became_active":"2024-03-06T07:25:50.665303+0000","last_became_peered":"2024-03-06T07:25:50.665303+0000","last_unstale":"2024-03-06T07:25:53.005403+0000","last_undegraded":"2024-03-06T07:25:53.005403+0000","last_fullsized":"2024-03-06T07:25:53.005403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:49.637079+0000","last_clean_scrub_stamp":"2024-03-06T07:25:49.637079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:14.532074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":286,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:25:53.005108+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:25:53.005108+0000","last_peered":"2024-03-06T07:25:53.005108+0000","last_clean":"2024-03-06T07:25:53.005108+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:25:53.005108+0000","last_undegraded":"2024-03-06T07:25:53.005108+0000","last_fullsized":"2024-03-06T07:25:53.005108+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1018105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":428361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":429301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":427667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":429279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1011712,"data_stored":428249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1019904,"data_stored":427861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1018069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:59.629 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:59.629 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:59.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.797+0000 7f70041bc640 1 Processor -- start 2024-03-06T07:25:59.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.797+0000 7f70041bc640 1 -- start start 2024-03-06T07:25:59.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f70041bc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6ffc06b600 0x7f6ffc0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f70041bc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ffc15ecb0 0x7f6ffc157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f70041bc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc0b3a00 con 0x7f6ffc06b220 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f70041bc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0b3ce0 con 0x7f6ffc15ecb0 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f70041bc640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc15e4b0 con 0x7f6ffc06b600 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7001f58640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6ffc06b220 0x7f6ffc072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59822/0 (socket says 172.21.15.122:59822) 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7001f58640 1 -- 172.21.15.122:0/1142087812 learned_addr learned my addr 172.21.15.122:0/1142087812 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7002759640 1 --2- 172.21.15.122:0/1142087812 >> v2:172.21.15.122:3300/0 conn(0x7f6ffc15ecb0 0x7f6ffc157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7001757640 1 --2- 172.21.15.122:0/1142087812 >> v2:172.21.15.169:3300/0 conn(0x7f6ffc06b600 0x7f6ffc0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7002759640 1 -- 172.21.15.122:0/1142087812 >> v2:172.21.15.169:3300/0 conn(0x7f6ffc06b600 msgr2=0x7f6ffc0b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7002759640 1 --2- 172.21.15.122:0/1142087812 >> v2:172.21.15.169:3300/0 conn(0x7f6ffc06b600 0x7f6ffc0b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7000f56640 1 -- 172.21.15.122:0/1142087812 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889191806 0 0) 0x7f6ffc0b3a00 con 0x7f6ffc06b220 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7002759640 1 -- 172.21.15.122:0/1142087812 >> v1:172.21.15.122:6789/0 conn(0x7f6ffc06b220 legacy=0x7f6ffc072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7002759640 1 -- 172.21.15.122:0/1142087812 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc15e790 con 0x7f6ffc15ecb0 2024-03-06T07:25:59.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.798+0000 7f7001757640 1 --2- 172.21.15.122:0/1142087812 >> v2:172.21.15.169:3300/0 conn(0x7f6ffc06b600 0x7f6ffc0b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f7002759640 1 --2- 172.21.15.122:0/1142087812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc15ecb0 0x7f6ffc157e50 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc06fb00 tx=0x7f6ff402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de58ae8726f53e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f7000f56640 1 -- 172.21.15.122:0/1142087812 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4039070 con 0x7f6ffc15ecb0 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f7000f56640 1 -- 172.21.15.122:0/1142087812 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff402cbf0 con 0x7f6ffc15ecb0 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f7000f56640 1 -- 172.21.15.122:0/1142087812 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4034440 con 0x7f6ffc15ecb0 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 -- 172.21.15.122:0/1142087812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc15ecb0 msgr2=0x7f6ffc157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 --2- 172.21.15.122:0/1142087812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc15ecb0 0x7f6ffc157e50 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc06fb00 tx=0x7f6ff402be40 comp rx=0 tx=0).stop 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 -- 172.21.15.122:0/1142087812 shutdown_connections 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 --2- 172.21.15.122:0/1142087812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc15ecb0 0x7f6ffc157e50 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 --2- 172.21.15.122:0/1142087812 >> v2:172.21.15.169:3300/0 conn(0x7f6ffc06b600 0x7f6ffc0b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 -- 172.21.15.122:0/1142087812 >> 172.21.15.122:0/1142087812 conn(0x7f6ffc06c0c0 msgr2=0x7f6ffc06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 -- 172.21.15.122:0/1142087812 shutdown_connections 2024-03-06T07:25:59.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.800+0000 7f70041bc640 1 -- 172.21.15.122:0/1142087812 wait complete. 2024-03-06T07:25:59.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.802+0000 7f70041bc640 1 Processor -- start 2024-03-06T07:25:59.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.802+0000 7f70041bc640 1 -- start start 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 0x7f6ffc14f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ffc06b600 0x7f6ffc0af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ffc15ecb0 0x7f6ffc1f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc15f2c0 con 0x7f6ffc06b220 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc15f140 con 0x7f6ffc06b600 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc158390 con 0x7f6ffc15ecb0 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 0x7f6ffc14f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 0x7f6ffc14f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59448/0 (socket says 172.21.15.122:59448) 2024-03-06T07:25:59.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 -- 172.21.15.122:0/1333392527 learned_addr learned my addr 172.21.15.122:0/1333392527 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001757640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ffc06b600 0x7f6ffc0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7002759640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ffc15ecb0 0x7f6ffc1f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 -- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ffc15ecb0 msgr2=0x7f6ffc1f7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ffc15ecb0 0x7f6ffc1f7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 -- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ffc06b600 msgr2=0x7f6ffc0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ffc06b600 0x7f6ffc0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 -- 172.21.15.122:0/1333392527 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc1f7850 con 0x7f6ffc06b220 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f7001f58640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 0x7f6ffc14f8f0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800c4d0 tx=0x7f6ff800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8018070 con 0x7f6ffc06b220 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff8007500 con 0x7f6ffc06b220 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc1f7b20 con 0x7f6ffc06b220 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8004a20 con 0x7f6ffc06b220 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.803+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc1f80e0 con 0x7f6ffc06b220 2024-03-06T07:25:59.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.804+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc0b4f00 con 0x7f6ffc06b220 2024-03-06T07:25:59.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.805+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ff801b070 con 0x7f6ffc06b220 2024-03-06T07:25:59.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.805+0000 7f6ff27fc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.806+0000 7f7001757640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.806+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ff80893d0 con 0x7f6ffc06b220 2024-03-06T07:25:59.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.806+0000 7f7001757640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6fe006b4d0 0x7f6fe006d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1e84b0 tx=0x7f6fec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:25:59.810+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ffc0b4f00 con 0x7f6ffc06b220 2024-03-06T07:26:00.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.175+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6ffc0733b0 con 0x7f6ffc06b220 2024-03-06T07:26:00.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.178+0000 7f6ff27fc640 1 -- 172.21.15.122:0/1333392527 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6ff80582e0 con 0x7f6ffc06b220 2024-03-06T07:26:00.180 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:26:00.180 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:26:00.180 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:26:00.180 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:26:00.180 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:00.181 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:26:00.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6fe006b4d0 msgr2=0x7f6fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6fe006b4d0 0x7f6fe006d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1e84b0 tx=0x7f6fec008040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 msgr2=0x7f6ffc14f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 0x7f6ffc14f8f0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800c4d0 tx=0x7f6ff800c990 comp rx=0 tx=0).stop 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 shutdown_connections 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ffc15ecb0 0x7f6ffc1f7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ffc06b600 0x7f6ffc0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 --2- 172.21.15.122:0/1333392527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ffc06b220 0x7f6ffc14f8f0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 >> 172.21.15.122:0/1333392527 conn(0x7f6ffc06c0c0 msgr2=0x7f6ffc0b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 shutdown_connections 2024-03-06T07:26:00.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.180+0000 7f70041bc640 1 -- 172.21.15.122:0/1333392527 wait complete. 2024-03-06T07:26:00.193 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:00.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.292+0000 7f5c6b629640 1 Processor -- start 2024-03-06T07:26:00.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.292+0000 7f5c6b629640 1 -- start start 2024-03-06T07:26:00.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c6b629640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c6415dfe0 0x7f5c640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c6b629640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5c640d29c0 0x7f5c640b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c6b629640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c640b3430 con 0x7f5c640cf830 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c6b629640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c640b31e0 con 0x7f5c6415dfe0 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c6b629640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c640b3d60 con 0x7f5c640d29c0 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c693c5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c6415dfe0 0x7f5c640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c68bc4640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5c640cf830 0x7f5c640d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59838/0 (socket says 172.21.15.122:59838) 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c693c5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c6415dfe0 0x7f5c640cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59450/0 (socket says 172.21.15.122:59450) 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c68bc4640 1 -- 172.21.15.122:0/3926338820 learned_addr learned my addr 172.21.15.122:0/3926338820 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.293+0000 7f5c69bc6640 1 --2- 172.21.15.122:0/3926338820 >> v2:172.21.15.169:3300/0 conn(0x7f5c640d29c0 0x7f5c640b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.294+0000 7f5c693c5640 1 -- 172.21.15.122:0/3926338820 >> v2:172.21.15.169:3300/0 conn(0x7f5c640d29c0 msgr2=0x7f5c640b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.294+0000 7f5c693c5640 1 --2- 172.21.15.122:0/3926338820 >> v2:172.21.15.169:3300/0 conn(0x7f5c640d29c0 0x7f5c640b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.294+0000 7f5c693c5640 1 -- 172.21.15.122:0/3926338820 >> v1:172.21.15.122:6789/0 conn(0x7f5c640cf830 legacy=0x7f5c640d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.294+0000 7f5c693c5640 1 -- 172.21.15.122:0/3926338820 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640b3710 con 0x7f5c6415dfe0 2024-03-06T07:26:00.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.294+0000 7f5c5bfff640 1 -- 172.21.15.122:0/3926338820 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763953143 0 0) 0x7f5c640b3430 con 0x7f5c640cf830 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.294+0000 7f5c693c5640 1 --2- 172.21.15.122:0/3926338820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c6415dfe0 0x7f5c640cf2f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f5c640b3cd0 tx=0x7f5c6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2870692d36835df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c5bfff640 1 -- 172.21.15.122:0/3926338820 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60035020 con 0x7f5c6415dfe0 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c5bfff640 1 -- 172.21.15.122:0/3926338820 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60031830 con 0x7f5c6415dfe0 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c5bfff640 1 -- 172.21.15.122:0/3926338820 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60031c00 con 0x7f5c6415dfe0 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 -- 172.21.15.122:0/3926338820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c6415dfe0 msgr2=0x7f5c640cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 --2- 172.21.15.122:0/3926338820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c6415dfe0 0x7f5c640cf2f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f5c640b3cd0 tx=0x7f5c6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 -- 172.21.15.122:0/3926338820 shutdown_connections 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 --2- 172.21.15.122:0/3926338820 >> v2:172.21.15.169:3300/0 conn(0x7f5c640d29c0 0x7f5c640b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 --2- 172.21.15.122:0/3926338820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c6415dfe0 0x7f5c640cf2f0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 -- 172.21.15.122:0/3926338820 >> 172.21.15.122:0/3926338820 conn(0x7f5c640d3cd0 msgr2=0x7f5c640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 -- 172.21.15.122:0/3926338820 shutdown_connections 2024-03-06T07:26:00.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.295+0000 7f5c6b629640 1 -- 172.21.15.122:0/3926338820 wait complete. 2024-03-06T07:26:00.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.297+0000 7f5c6b629640 1 Processor -- start 2024-03-06T07:26:00.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.297+0000 7f5c6b629640 1 -- start start 2024-03-06T07:26:00.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c6b629640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c640cf830 0x7f5c6415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c6b629640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 0x7f5c641f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c6b629640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c6415dfe0 0x7f5c640b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c6b629640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640d1df0 con 0x7f5c640d29c0 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c6b629640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640d1c70 con 0x7f5c6415dfe0 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c6b629640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6415e3c0 con 0x7f5c640cf830 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 0x7f5c641f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 0x7f5c641f4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59452/0 (socket says 172.21.15.122:59452) 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 -- 172.21.15.122:0/1811894917 learned_addr learned my addr 172.21.15.122:0/1811894917 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 -- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c640cf830 msgr2=0x7f5c6415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c69bc6640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c6415dfe0 0x7f5c640b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c693c5640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c640cf830 0x7f5c6415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c640cf830 0x7f5c6415bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 -- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c6415dfe0 msgr2=0x7f5c640b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c6415dfe0 0x7f5c640b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 -- 172.21.15.122:0/1811894917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c641f7280 con 0x7f5c640d29c0 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c693c5640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c640cf830 0x7f5c6415bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c69bc6640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c6415dfe0 0x7f5c640b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.298+0000 7f5c68bc4640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 0x7f5c641f4300 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400c4b0 tx=0x7f5c5400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.299+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54018070 con 0x7f5c640d29c0 2024-03-06T07:26:00.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.299+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54007500 con 0x7f5c640d29c0 2024-03-06T07:26:00.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.299+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c641f7560 con 0x7f5c640d29c0 2024-03-06T07:26:00.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.299+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54004a20 con 0x7f5c640d29c0 2024-03-06T07:26:00.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.299+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c641f7b20 con 0x7f5c640d29c0 2024-03-06T07:26:00.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.300+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c641f7d60 con 0x7f5c640d29c0 2024-03-06T07:26:00.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.300+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c5401b070 con 0x7f5c640d29c0 2024-03-06T07:26:00.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.301+0000 7f5c59ffb640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c4806b500 0x7f5c4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.301+0000 7f5c693c5640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c4806b500 0x7f5c4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.301+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c54089400 con 0x7f5c640d29c0 2024-03-06T07:26:00.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.301+0000 7f5c693c5640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c4806b500 0x7f5c4806d9c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5c60036000 tx=0x7f5c60008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.305+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c641f7d60 con 0x7f5c640d29c0 2024-03-06T07:26:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.520+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c6404f330 con 0x7f5c640d29c0 2024-03-06T07:26:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.520+0000 7f5c59ffb640 1 -- 172.21.15.122:0/1811894917 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f5c540583d0 con 0x7f5c640d29c0 2024-03-06T07:26:00.522 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:00.522 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:26:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:26:00.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c4806b500 msgr2=0x7f5c4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c4806b500 0x7f5c4806d9c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5c60036000 tx=0x7f5c60008030 comp rx=0 tx=0).stop 2024-03-06T07:26:00.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 msgr2=0x7f5c641f4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 0x7f5c641f4300 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400c4b0 tx=0x7f5c5400c970 comp rx=0 tx=0).stop 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 shutdown_connections 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5c4806b500 0x7f5c4806d9c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c6415dfe0 0x7f5c640b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c640d29c0 0x7f5c641f4300 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 --2- 172.21.15.122:0/1811894917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c640cf830 0x7f5c6415bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 >> 172.21.15.122:0/1811894917 conn(0x7f5c640d3cd0 msgr2=0x7f5c640ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 shutdown_connections 2024-03-06T07:26:00.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:00.522+0000 7f5c6b629640 1 -- 172.21.15.122:0/1811894917 wait complete. 2024-03-06T07:26:00.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:50.998325+0000', 'modified': '2024-03-06T07:25:53.113303+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7041}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7041': {'gid': 7041, 'name': 'c', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6836/4292480577', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 4292480577}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 4292480577}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7041, 'qdb_cluster': [7041]} 2024-03-06T07:26:00.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:00.937 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:26:01.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 Processor -- start 2024-03-06T07:26:01.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 -- start start 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f829c15b2d0 0x7f829c0708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c071230 con 0x7f829c1444c0 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c1448a0 con 0x7f829c15b2d0 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.038+0000 7f82a2313640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c06d110 con 0x7f829c070e10 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f829c15b2d0 0x7f829c0708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f829c1444c0 0x7f829c06ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59842/0 (socket says 172.21.15.122:59842) 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829bfff640 1 -- 172.21.15.122:0/3161413440 learned_addr learned my addr 172.21.15.122:0/3161413440 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f82a08b0640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829b7fe640 1 -- 172.21.15.122:0/3161413440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 msgr2=0x7f829c0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829b7fe640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829b7fe640 1 -- 172.21.15.122:0/3161413440 >> v1:172.21.15.122:6789/0 conn(0x7f829c1444c0 legacy=0x7f829c06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829b7fe640 1 -- 172.21.15.122:0/3161413440 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c06d370 con 0x7f829c15b2d0 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f829affd640 1 -- 172.21.15.122:0/3161413440 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915529822 0 0) 0x7f829c071230 con 0x7f829c1444c0 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.039+0000 7f82a08b0640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:01.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.040+0000 7f829b7fe640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c15b2d0 0x7f829c0708d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f82900097d0 tx=0x7f829002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53732092091d8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.040+0000 7f829affd640 1 -- 172.21.15.122:0/3161413440 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290035020 con 0x7f829c15b2d0 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.040+0000 7f829affd640 1 -- 172.21.15.122:0/3161413440 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290031800 con 0x7f829c15b2d0 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.040+0000 7f829affd640 1 -- 172.21.15.122:0/3161413440 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290031bd0 con 0x7f829c15b2d0 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 -- 172.21.15.122:0/3161413440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c15b2d0 msgr2=0x7f829c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c15b2d0 0x7f829c0708d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f82900097d0 tx=0x7f829002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 -- 172.21.15.122:0/3161413440 shutdown_connections 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 --2- 172.21.15.122:0/3161413440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c15b2d0 0x7f829c0708d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 -- 172.21.15.122:0/3161413440 >> 172.21.15.122:0/3161413440 conn(0x7f829c0b1c80 msgr2=0x7f829c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 -- 172.21.15.122:0/3161413440 shutdown_connections 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.041+0000 7f82a2313640 1 -- 172.21.15.122:0/3161413440 wait complete. 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.042+0000 7f82a2313640 1 Processor -- start 2024-03-06T07:26:01.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.042+0000 7f82a2313640 1 -- start start 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f82a2313640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c15a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f82a2313640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 0x7f829c0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f82a2313640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f829c15b2d0 0x7f829c13ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f82a2313640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c06dc10 con 0x7f829c1444c0 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f82a2313640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c06d930 con 0x7f829c15b2d0 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f82a2313640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c06def0 con 0x7f829c070e10 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 0x7f829c0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 0x7f829c0b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59472/0 (socket says 172.21.15.122:59472) 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 -- 172.21.15.122:0/1360798314 learned_addr learned my addr 172.21.15.122:0/1360798314 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829bfff640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c15a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 -- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 msgr2=0x7f829c15a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c15a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 -- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f829c15b2d0 msgr2=0x7f829c13ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f829c15b2d0 0x7f829c13ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 -- 172.21.15.122:0/1360798314 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0b2e20 con 0x7f829c1444c0 2024-03-06T07:26:01.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.043+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 0x7f829c0b23d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8290034040 tx=0x7f82900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.044+0000 7f8298ff9640 1 -- 172.21.15.122:0/1360798314 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290040070 con 0x7f829c1444c0 2024-03-06T07:26:01.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.044+0000 7f8298ff9640 1 -- 172.21.15.122:0/1360798314 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290039cf0 con 0x7f829c1444c0 2024-03-06T07:26:01.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.044+0000 7f8298ff9640 1 -- 172.21.15.122:0/1360798314 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290035040 con 0x7f829c1444c0 2024-03-06T07:26:01.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.044+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c0b3100 con 0x7f829c1444c0 2024-03-06T07:26:01.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.044+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c0b36c0 con 0x7f829c1444c0 2024-03-06T07:26:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.045+0000 7f8298ff9640 1 -- 172.21.15.122:0/1360798314 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f829004b060 con 0x7f829c1444c0 2024-03-06T07:26:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.046+0000 7f8298ff9640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f828806b4d0 0x7f828806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.046+0000 7f8298ff9640 1 -- 172.21.15.122:0/1360798314 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82900abe30 con 0x7f829c1444c0 2024-03-06T07:26:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.046+0000 7f829bfff640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f828806b4d0 0x7f828806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.046+0000 7f829bfff640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f828806b4d0 0x7f828806d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f828c007fd0 tx=0x7f828c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 Processor -- start 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 -- start start 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 0x559e19d14c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d188c0 0x559e19d1a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d44a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e19c7e6b0 con 0x559e19d14870 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e19c27a60 con 0x559e19d188c0 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x559e19c7a750 con 0x559e19d44600 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f829bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 0x559e19d14c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f829bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 0x559e19d14c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59482/0 (socket says 172.21.15.122:59482) 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.047+0000 7f829bfff640 1 -- 172.21.15.122:0/4254539835 learned_addr learned my addr 172.21.15.122:0/4254539835 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f829bfff640 1 -- 172.21.15.122:0/4254539835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 msgr2=0x559e19d44a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f829bfff640 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d44a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f829bfff640 1 -- 172.21.15.122:0/4254539835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d188c0 msgr2=0x559e19d1a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f829bfff640 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d188c0 0x559e19d1a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f829bfff640 1 -- 172.21.15.122:0/4254539835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e19c45f30 con 0x559e19d14870 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f829bfff640 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 0x559e19d14c50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f828c007390 tx=0x7f828c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dac1a4dc535f50c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f8281ffb640 1 -- 172.21.15.122:0/4254539835 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c00d9a0 con 0x559e19d14870 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f8281ffb640 1 -- 172.21.15.122:0/4254539835 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c00db40 con 0x559e19d14870 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.048+0000 7f8281ffb640 1 -- 172.21.15.122:0/4254539835 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c020450 con 0x559e19d14870 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 -- 172.21.15.122:0/4254539835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 msgr2=0x559e19d14c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 0x559e19d14c50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f828c007390 tx=0x7f828c010000 comp rx=0 tx=0).stop 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 -- 172.21.15.122:0/4254539835 shutdown_connections 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d44a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d188c0 0x559e19d1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/4254539835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d14870 0x559e19d14c50 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 -- 172.21.15.122:0/4254539835 >> 172.21.15.122:0/4254539835 conn(0x559e19d168f0 msgr2=0x559e19d181b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 -- 172.21.15.122:0/4254539835 shutdown_connections 2024-03-06T07:26:01.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 -- 172.21.15.122:0/4254539835 wait complete. 2024-03-06T07:26:01.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.049+0000 7f82a4fd9740 1 -- start start 2024-03-06T07:26:01.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 Processor -- start 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d44600 0x559e19d739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d461b0 0x559e19d3e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e19d18d50 con 0x559e19d72e50 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e19c3aea0 con 0x559e19d44600 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x559e19d19030 con 0x559e19d461b0 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.050+0000 7f82a4fd9740 10 client.0 _ll_register_callbacks cb 0x559e19c3a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59496/0 (socket says 172.21.15.122:59496) 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 -- 172.21.15.122:0/1132886220 learned_addr learned my addr 172.21.15.122:0/1132886220 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d461b0 0x559e19d3e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f829bfff640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d44600 0x559e19d739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 -- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d461b0 msgr2=0x559e19d3e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d461b0 0x559e19d3e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 -- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d44600 msgr2=0x559e19d739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d44600 0x559e19d739f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 -- 172.21.15.122:0/1132886220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e19c3b270 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a08b0640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d732d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f829400daa0 tx=0x7f829400df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab70fa565eea095 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f827affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f827affd640 1 -- 172.21.15.122:0/1132886220 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829401b060 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f827affd640 1 -- 172.21.15.122:0/1132886220 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f829400bdd0 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f828c00db40 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f827affd640 1 -- 172.21.15.122:0/1132886220 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8294021050 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a4fd9740 10 client.7224 fetch_fsmap learned FSMap version 40 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x559e199932d0 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.051+0000 7f827affd640 1 -- 172.21.15.122:0/1132886220 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c00db40 con 0x559e19d72e50 2024-03-06T07:26:01.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f827affd640 1 -- 172.21.15.122:0/1132886220 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 40) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8294024020 con 0x559e19d72e50 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f82a4fd9740 10 client.7224 fetch_fsmap finished waiting for FSMap version 40 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f82a4fd9740 10 client.7224 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f82a4fd9740 20 client.7224 populate_metadata read hostname 'smithi122' 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d3f270 0x559e19d7e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f82a4fd9740 4 client.7224 mds_command: new command op to 7041 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 --> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e19c8bbf0 con 0x559e19d3f270 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f829bfff640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d3f270 0x559e19d7e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f827affd640 10 client.7224 ms_handle_connect on v2:172.21.15.122:6834/4292480577 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.052+0000 7f829bfff640 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d3f270 0x559e19d7e2e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e8b782772f888b server_cookie=4bd48c122f1f0795 in_seq=0 out_seq=0 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f827affd640 1 -- 172.21.15.122:0/1132886220 <== mds.0 v2:172.21.15.122:6834/4292480577 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559e19c8bbf0 con 0x559e19d3f270 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f827affd640 10 client.7224 handle_command_reply: tid=0 2024-03-06T07:26:01.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 client.7224 shutdown 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 msgr2=0x559e19d732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d732d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f829400daa0 tx=0x7f829400df60 comp rx=0 tx=0).stop 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 shutdown_connections 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d732d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d461b0 0x559e19d3e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d3f270 0x559e19d7e2e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1132886220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d44600 0x559e19d739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f827affd640 0 client.7224 ms_handle_reset on v2:172.21.15.122:6834/4292480577 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 >> 172.21.15.122:0/1132886220 conn(0x559e19d14870 msgr2=0x559e19d425c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 shutdown_connections 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1132886220 wait complete. 2024-03-06T07:26:01.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.053+0000 7f82a4fd9740 20 client.7224 trim_cache size 0 max 16384 2024-03-06T07:26:01.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.070+0000 7f82a4fd9740 1 Processor -- start 2024-03-06T07:26:01.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.070+0000 7f82a4fd9740 1 -- start start 2024-03-06T07:26:01.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d3f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d461b0 0x559e19d449e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d44f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c13e940 con 0x559e19d72e50 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c1378c0 con 0x559e19d461b0 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c13e340 con 0x559e19d44600 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d44f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d44f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59502/0 (socket says 172.21.15.122:59502) 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 -- 172.21.15.122:0/2017203261 learned_addr learned my addr 172.21.15.122:0/2017203261 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f829b7fe640 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d461b0 0x559e19d449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f829bfff640 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d3f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 -- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 msgr2=0x559e19d3f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d3f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 -- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d461b0 msgr2=0x559e19d449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d461b0 0x559e19d449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.071+0000 7f82a08b0640 1 -- 172.21.15.122:0/2017203261 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e19a5d6b0 con 0x559e19d72e50 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a08b0640 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d44f20 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8294002230 tx=0x7f8294010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be307a75c8b09863 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82817fa640 1 -- 172.21.15.122:0/2017203261 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400bbd0 con 0x559e19d72e50 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82817fa640 1 -- 172.21.15.122:0/2017203261 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829400bd70 con 0x559e19d72e50 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82817fa640 1 -- 172.21.15.122:0/2017203261 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294003b90 con 0x559e19d72e50 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 -- 172.21.15.122:0/2017203261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 msgr2=0x559e19d44f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d44f20 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8294002230 tx=0x7f8294010000 comp rx=0 tx=0).stop 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 -- 172.21.15.122:0/2017203261 shutdown_connections 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d72e50 0x559e19d44f20 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d461b0 0x559e19d449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/2017203261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d3f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 -- 172.21.15.122:0/2017203261 >> 172.21.15.122:0/2017203261 conn(0x559e19d14870 msgr2=0x559e19d18080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 -- 172.21.15.122:0/2017203261 shutdown_connections 2024-03-06T07:26:01.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.072+0000 7f82a4fd9740 1 -- 172.21.15.122:0/2017203261 wait complete. 2024-03-06T07:26:01.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.073+0000 7f82a4fd9740 1 -- start start 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.073+0000 7f82a4fd9740 1 Processor -- start 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d45280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 0x559e19dd1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d72e50 0x559e19d5ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e19d471a0 con 0x559e19d461b0 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e19d5b670 con 0x559e19d72e50 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x559e19d47490 con 0x559e19d44600 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a4fd9740 10 client.0 _ll_register_callbacks cb 0x7f829400d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:01.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 0x559e19dd1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 0x559e19dd1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59510/0 (socket says 172.21.15.122:59510) 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 -- 172.21.15.122:0/1393970639 learned_addr learned my addr 172.21.15.122:0/1393970639 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829bfff640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d45280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a08b0640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d72e50 0x559e19d5ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 -- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 msgr2=0x559e19d45280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d45280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 -- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d72e50 msgr2=0x559e19d5ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d72e50 0x559e19d5ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 -- 172.21.15.122:0/1393970639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e19d60470 con 0x559e19d461b0 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829bfff640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d45280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f82a08b0640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d72e50 0x559e19d5ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f829b7fe640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 0x559e19dd1d00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f829007a230 tx=0x7f82900767d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17b02fc801979da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.074+0000 7f827affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:26:01.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f827affd640 1 -- 172.21.15.122:0/1393970639 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829007e680 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f827affd640 1 -- 172.21.15.122:0/1393970639 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f829007e150 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f829400bd70 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f827affd640 1 -- 172.21.15.122:0/1393970639 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8290082c40 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 10 client.7230 fetch_fsmap learned FSMap version 40 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x559e19d5f720 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f827affd640 1 -- 172.21.15.122:0/1393970639 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829400bd70 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f827affd640 1 -- 172.21.15.122:0/1393970639 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 40) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8290079980 con 0x559e19d461b0 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 10 client.7230 fetch_fsmap finished waiting for FSMap version 40 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 10 client.7230 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 20 client.7230 populate_metadata read hostname 'smithi122' 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d19ea0 0x559e19d59c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 4 client.7230 mds_command: new command op to 7041 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 --> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559e19c85f40 con 0x559e19d19ea0 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f829bfff640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d19ea0 0x559e19d59c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f827affd640 10 client.7230 ms_handle_connect on v2:172.21.15.122:6834/4292480577 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.075+0000 7f829bfff640 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d19ea0 0x559e19d59c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8912cae03d7b4fd server_cookie=f0e0664275355ff0 in_seq=0 out_seq=0 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f827affd640 1 -- 172.21.15.122:0/1393970639 <== mds.0 v2:172.21.15.122:6834/4292480577 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559e19c85f40 con 0x559e19d19ea0 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f827affd640 10 client.7230 handle_command_reply: tid=0 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 client.7230 shutdown 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 msgr2=0x559e19dd1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 0x559e19dd1d00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f829007a230 tx=0x7f82900767d0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 shutdown_connections 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:6834/4292480577,v1:172.21.15.122:6836/4292480577] conn(0x559e19d19ea0 0x559e19d59c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559e19d72e50 0x559e19d5ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f827affd640 0 client.7230 ms_handle_reset on v2:172.21.15.122:6834/4292480577 2024-03-06T07:26:01.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e19d461b0 0x559e19dd1d00 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 --2- 172.21.15.122:0/1393970639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x559e19d44600 0x559e19d45280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 >> 172.21.15.122:0/1393970639 conn(0x559e19d14870 msgr2=0x559e19d3f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 shutdown_connections 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 1 -- 172.21.15.122:0/1393970639 wait complete. 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a4fd9740 20 client.7230 trim_cache size 0 max 16384 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 7041, 2024-03-06T07:26:01.085 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 9.4080366610000006, 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 40, 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 82, 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 82, 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 14.463344511000001 2024-03-06T07:26:01.086 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f828806b4d0 msgr2=0x7f828806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a2313640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f828806b4d0 0x7f828806d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f828c007fd0 tx=0x7f828c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 msgr2=0x7f829c0b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.076+0000 7f82a2313640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 0x7f829c0b23d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8290034040 tx=0x7f82900046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 shutdown_connections 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f828806b4d0 0x7f828806d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f829c15b2d0 0x7f829c13ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f829c1444c0 0x7f829c0b23d0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 --2- 172.21.15.122:0/1360798314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f829c070e10 0x7f829c15a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 >> 172.21.15.122:0/1360798314 conn(0x7f829c0b1c80 msgr2=0x7f829c0ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 shutdown_connections 2024-03-06T07:26:01.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.077+0000 7f82a2313640 1 -- 172.21.15.122:0/1360798314 wait complete. 2024-03-06T07:26:01.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:01.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.188+0000 7fcffb793640 1 Processor -- start 2024-03-06T07:26:01.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcffb793640 1 -- start start 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcffb793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcffb793640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcff40b41f0 0x7fcff40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcffb793640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff40b3430 con 0x7fcff415dfe0 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcffb793640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff40d2c50 con 0x7fcff40b41f0 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcffb793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40b31e0 con 0x7fcff40d22b0 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.189+0000 7fcff9d30640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcff40b41f0 0x7fcff40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff952f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcff415dfe0 0x7fcff40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59858/0 (socket says 172.21.15.122:59858) 2024-03-06T07:26:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff952f640 1 -- 172.21.15.122:0/2540484499 learned_addr learned my addr 172.21.15.122:0/2540484499 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff9d30640 1 -- 172.21.15.122:0/2540484499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 msgr2=0x7fcff40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff8d2e640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcfebfff640 1 -- 172.21.15.122:0/2540484499 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295566254 0 0) 0x7fcff40b3430 con 0x7fcff415dfe0 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff9d30640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff9d30640 1 -- 172.21.15.122:0/2540484499 >> v1:172.21.15.122:6789/0 conn(0x7fcff415dfe0 legacy=0x7fcff40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff9d30640 1 -- 172.21.15.122:0/2540484499 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff40b3710 con 0x7fcff40b41f0 2024-03-06T07:26:01.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.190+0000 7fcff8d2e640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:01.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcff9d30640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff40b41f0 0x7fcff40b4650 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d90 tx=0x7fcfec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a90103b25f3e312 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcfebfff640 1 -- 172.21.15.122:0/2540484499 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec039070 con 0x7fcff40b41f0 2024-03-06T07:26:01.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcfebfff640 1 -- 172.21.15.122:0/2540484499 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec02cc50 con 0x7fcff40b41f0 2024-03-06T07:26:01.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcfebfff640 1 -- 172.21.15.122:0/2540484499 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec034440 con 0x7fcff40b41f0 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 -- 172.21.15.122:0/2540484499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff40b41f0 msgr2=0x7fcff40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff40b41f0 0x7fcff40b4650 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d90 tx=0x7fcfec02bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 -- 172.21.15.122:0/2540484499 shutdown_connections 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff40b41f0 0x7fcff40b4650 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 --2- 172.21.15.122:0/2540484499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 -- 172.21.15.122:0/2540484499 >> 172.21.15.122:0/2540484499 conn(0x7fcff40d3cd0 msgr2=0x7fcff40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 -- 172.21.15.122:0/2540484499 shutdown_connections 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.191+0000 7fcffb793640 1 -- 172.21.15.122:0/2540484499 wait complete. 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.192+0000 7fcffb793640 1 Processor -- start 2024-03-06T07:26:01.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.192+0000 7fcffb793640 1 -- start start 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcffb793640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcff40b41f0 0x7fcff41f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcffb793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff41f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcffb793640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 0x7fcff41605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcffb793640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40cfa00 con 0x7fcff415dfe0 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcffb793640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40b3d60 con 0x7fcff40b41f0 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcffb793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40d1c70 con 0x7fcff40d22b0 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 0x7fcff41605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 0x7fcff41605e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59536/0 (socket says 172.21.15.122:59536) 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 -- 172.21.15.122:0/4004521506 learned_addr learned my addr 172.21.15.122:0/4004521506 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 -- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 msgr2=0x7fcff41f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff8d2e640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff41f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff41f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 -- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcff40b41f0 msgr2=0x7fcff41f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcff40b41f0 0x7fcff41f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.193+0000 7fcff9d30640 1 -- 172.21.15.122:0/4004521506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff41f4830 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.194+0000 7fcff9d30640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 0x7fcff41605e0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d90 tx=0x7fcfec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.194+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec039050 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.194+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec045070 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.194+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff415f540 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.194+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec03e610 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.194+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff415fb00 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.195+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff404f330 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.195+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfec03e7b0 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.196+0000 7fcfe9ffb640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfd806b4d0 0x7fcfd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.196+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfec0acd80 con 0x7fcff415dfe0 2024-03-06T07:26:01.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.196+0000 7fcff952f640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfd806b4d0 0x7fcfd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.196+0000 7fcff952f640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfd806b4d0 0x7fcfd806d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcff0007fd0 tx=0x7fcff0007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.200+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcff41f4830 con 0x7fcff415dfe0 2024-03-06T07:26:01.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.499+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcff406bfe0 con 0x7fcff415dfe0 2024-03-06T07:26:01.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.500+0000 7fcfe9ffb640 1 -- 172.21.15.122:0/4004521506 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcfec032e20 con 0x7fcff415dfe0 2024-03-06T07:26:01.504 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:01.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfd806b4d0 msgr2=0x7fcfd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfd806b4d0 0x7fcfd806d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcff0007fd0 tx=0x7fcff0007960 comp rx=0 tx=0).stop 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 msgr2=0x7fcff41605e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 0x7fcff41605e0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d90 tx=0x7fcfec004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 shutdown_connections 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfd806b4d0 0x7fcfd806d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcff415dfe0 0x7fcff41605e0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcff40d22b0 0x7fcff41f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 --2- 172.21.15.122:0/4004521506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcff40b41f0 0x7fcff41f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 >> 172.21.15.122:0/4004521506 conn(0x7fcff40d3cd0 msgr2=0x7fcff40d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 shutdown_connections 2024-03-06T07:26:01.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:01.502+0000 7fcffb793640 1 -- 172.21.15.122:0/4004521506 wait complete. 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:01.515 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:01.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.515 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42479sec preferred_lft 42479sec 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.532 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:01.596 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:26:01.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.597 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:26:01.653 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:26:01.654 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:01.654 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:26:01.654 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:26:01.654 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:01.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:26:01.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.748 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:01.764 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:01.764 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:01.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:26:01.830 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:01.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:01.831 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:01.886 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.942 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:26:02.008 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:02.064 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:26:02.065 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:02.066 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.067 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:26:02.119 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:02.119 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:02.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:26:02.188 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:26:02.243 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:02.244 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:02.247 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:26:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:02.248 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:02.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.249 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:26:02.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:26:02.246+0000 7efdeca38480 -1 init, newargv = 0x5601330446d0 newargc=15 2024-03-06T07:26:02.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[66148]: starting ceph client 2024-03-06T07:26:02.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[66148]: starting fuse 2024-03-06T07:26:02.266 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:26:02.266 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:26:02.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:02.267 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:02.268 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:26:02.386 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.66148.asok 2024-03-06T07:26:02.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:02.396 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.66148.asok 2024-03-06T07:26:02.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.66148.asok status 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:26:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "66148", 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 7245, 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 7245 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:26:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1769990003 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1769990003 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.7245 192.168.0.1:0/1769990003", 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/1769990003", 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:26:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 40, 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 82, 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:26:02.537 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:02.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.542 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:26:02.543 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:02.557 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:26:02.558 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:02.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:02.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.625 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:26:02.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.685 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:26:02.750 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:02.750 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:02.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:02.805 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:26:03.035 INFO:teuthology.orchestra.run.smithi122.stderr:35+0 records in 2024-03-06T07:26:03.035 INFO:teuthology.orchestra.run.smithi122.stderr:35+0 records out 2024-03-06T07:26:03.035 INFO:teuthology.orchestra.run.smithi122.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.172147 s, 213 MB/s 2024-03-06T07:26:03.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:03.036 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:26:03.054 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.193+0000 7f594310d640 1 Processor -- start 2024-03-06T07:26:03.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.193+0000 7f594310d640 1 -- start start 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f594310d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f593c15bb20 0x7f593c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f594310d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f593c0cd8c0 0x7f593c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f594310d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c161980 con 0x7f593c0cfff0 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f594310d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c161c60 con 0x7f593c0cd8c0 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f594310d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c161f40 con 0x7f593c15bb20 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f59416aa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f593c0cd8c0 0x7f593c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f593bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f593c0cfff0 0x7f593c0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59860/0 (socket says 172.21.15.122:59860) 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f593bfff640 1 -- 172.21.15.122:0/3187481309 learned_addr learned my addr 172.21.15.122:0/3187481309 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f5940ea9640 1 --2- 172.21.15.122:0/3187481309 >> v2:172.21.15.169:3300/0 conn(0x7f593c15bb20 0x7f593c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f59416aa640 1 -- 172.21.15.122:0/3187481309 >> v2:172.21.15.169:3300/0 conn(0x7f593c15bb20 msgr2=0x7f593c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f59416aa640 1 --2- 172.21.15.122:0/3187481309 >> v2:172.21.15.169:3300/0 conn(0x7f593c15bb20 0x7f593c0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f59416aa640 1 -- 172.21.15.122:0/3187481309 >> v1:172.21.15.122:6789/0 conn(0x7f593c0cfff0 legacy=0x7f593c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.194+0000 7f59416aa640 1 -- 172.21.15.122:0/3187481309 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c162220 con 0x7f593c0cd8c0 2024-03-06T07:26:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f59416aa640 1 --2- 172.21.15.122:0/3187481309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0d04a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f593c0d2bc0 tx=0x7f593402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154d5fc4405f93ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f593b7fe640 1 -- 172.21.15.122:0/3187481309 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934039070 con 0x7f593c0cd8c0 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f593b7fe640 1 -- 172.21.15.122:0/3187481309 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593402cc80 con 0x7f593c0cd8c0 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f593b7fe640 1 -- 172.21.15.122:0/3187481309 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934034660 con 0x7f593c0cd8c0 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 -- 172.21.15.122:0/3187481309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 msgr2=0x7f593c0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 --2- 172.21.15.122:0/3187481309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0d04a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f593c0d2bc0 tx=0x7f593402bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 -- 172.21.15.122:0/3187481309 shutdown_connections 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 --2- 172.21.15.122:0/3187481309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0d04a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 --2- 172.21.15.122:0/3187481309 >> v2:172.21.15.169:3300/0 conn(0x7f593c15bb20 0x7f593c0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 -- 172.21.15.122:0/3187481309 >> 172.21.15.122:0/3187481309 conn(0x7f593c09a9f0 msgr2=0x7f593c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 -- 172.21.15.122:0/3187481309 shutdown_connections 2024-03-06T07:26:03.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.196+0000 7f594310d640 1 -- 172.21.15.122:0/3187481309 wait complete. 2024-03-06T07:26:03.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.197+0000 7f594310d640 1 Processor -- start 2024-03-06T07:26:03.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.197+0000 7f594310d640 1 -- start start 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f594310d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f594310d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f593c0cfff0 0x7f593c151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f594310d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f593c15bb20 0x7f593c0c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f594310d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0d24b0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f594310d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0d2330 con 0x7f593c15bb20 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f594310d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0d1980 con 0x7f593c0cfff0 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f5940ea9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f5940ea9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59550/0 (socket says 172.21.15.122:59550) 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f5940ea9640 1 -- 172.21.15.122:0/3819651073 learned_addr learned my addr 172.21.15.122:0/3819651073 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f59416aa640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f593c15bb20 0x7f593c0c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.198+0000 7f593bfff640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f593c0cfff0 0x7f593c151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f5940ea9640 1 -- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f593c0cfff0 msgr2=0x7f593c151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f5940ea9640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f593c0cfff0 0x7f593c151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f5940ea9640 1 -- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f593c15bb20 msgr2=0x7f593c0c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f5940ea9640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f593c15bb20 0x7f593c0c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f5940ea9640 1 -- 172.21.15.122:0/3819651073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0c9b70 con 0x7f593c0cd8c0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f5940ea9640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0ca450 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f593000d5f0 tx=0x7f593000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300125f0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5930019070 con 0x7f593c0cd8c0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c0caae0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300129d0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.199+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c0c58d0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.200+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c04f330 con 0x7f593c0cd8c0 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.201+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5930012b70 con 0x7f593c0cd8c0 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.201+0000 7f59397fa640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f592006b4d0 0x7f592006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.201+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f593008a510 con 0x7f593c0cd8c0 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.201+0000 7f593bfff640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f592006b4d0 0x7f592006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.202+0000 7f593bfff640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f592006b4d0 0x7f592006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f593c15ae90 tx=0x7f5928007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.210+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5930058d40 con 0x7f593c0cd8c0 2024-03-06T07:26:03.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.425+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f593c1524d0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.426+0000 7f59397fa640 1 -- 172.21.15.122:0/3819651073 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f593005cde0 con 0x7f593c0cd8c0 2024-03-06T07:26:03.428 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:03.428 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:26:03.429 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:26:03.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f592006b4d0 msgr2=0x7f592006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f592006b4d0 0x7f592006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f593c15ae90 tx=0x7f5928007e80 comp rx=0 tx=0).stop 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 msgr2=0x7f593c0ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0ca450 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f593000d5f0 tx=0x7f593000dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 shutdown_connections 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f592006b4d0 0x7f592006d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f593c15bb20 0x7f593c0c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f593c0cfff0 0x7f593c151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 --2- 172.21.15.122:0/3819651073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0cd8c0 0x7f593c0ca450 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 >> 172.21.15.122:0/3819651073 conn(0x7f593c09a9f0 msgr2=0x7f593c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 shutdown_connections 2024-03-06T07:26:03.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.428+0000 7f594310d640 1 -- 172.21.15.122:0/3819651073 wait complete. 2024-03-06T07:26:03.441 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:26:03.572 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:26:03.572 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:03.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:26:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.670+0000 7f14777e4640 1 Processor -- start 2024-03-06T07:26:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.670+0000 7f14777e4640 1 -- start start 2024-03-06T07:26:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f14777e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f14700b4460 0x7f14700d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f14777e4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1470161cf0 0x7f1470162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f14777e4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700d22c0 con 0x7f1470162970 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f14777e4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1470161b00 con 0x7f14700b4460 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f14777e4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14700cecc0 con 0x7f1470161cf0 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475580640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f14700b4460 0x7f14700d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1474d7f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1470162970 0x7f14700b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59874/0 (socket says 172.21.15.122:59874) 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1474d7f640 1 -- 172.21.15.122:0/2551543594 learned_addr learned my addr 172.21.15.122:0/2551543594 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475d81640 1 --2- 172.21.15.122:0/2551543594 >> v2:172.21.15.169:3300/0 conn(0x7f1470161cf0 0x7f1470162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475580640 1 -- 172.21.15.122:0/2551543594 >> v2:172.21.15.169:3300/0 conn(0x7f1470161cf0 msgr2=0x7f1470162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475580640 1 --2- 172.21.15.122:0/2551543594 >> v2:172.21.15.169:3300/0 conn(0x7f1470161cf0 0x7f1470162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475580640 1 -- 172.21.15.122:0/2551543594 >> v1:172.21.15.122:6789/0 conn(0x7f1470162970 legacy=0x7f14700b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475580640 1 -- 172.21.15.122:0/2551543594 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700d25a0 con 0x7f14700b4460 2024-03-06T07:26:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.671+0000 7f1475d81640 1 --2- 172.21.15.122:0/2551543594 >> v2:172.21.15.169:3300/0 conn(0x7f1470161cf0 0x7f1470162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.672+0000 7f1475580640 1 --2- 172.21.15.122:0/2551543594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14700b4460 0x7f14700d4800 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f147008c900 tx=0x7f146002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b84089a6493e6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.672+0000 7f1467fff640 1 -- 172.21.15.122:0/2551543594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460035020 con 0x7f14700b4460 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.672+0000 7f1467fff640 1 -- 172.21.15.122:0/2551543594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1460031830 con 0x7f14700b4460 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.672+0000 7f1467fff640 1 -- 172.21.15.122:0/2551543594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460031c00 con 0x7f14700b4460 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 -- 172.21.15.122:0/2551543594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14700b4460 msgr2=0x7f14700d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 --2- 172.21.15.122:0/2551543594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14700b4460 0x7f14700d4800 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f147008c900 tx=0x7f146002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 -- 172.21.15.122:0/2551543594 shutdown_connections 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 --2- 172.21.15.122:0/2551543594 >> v2:172.21.15.169:3300/0 conn(0x7f1470161cf0 0x7f1470162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 --2- 172.21.15.122:0/2551543594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14700b4460 0x7f14700d4800 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 -- 172.21.15.122:0/2551543594 >> 172.21.15.122:0/2551543594 conn(0x7f14700d3ce0 msgr2=0x7f14700d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 -- 172.21.15.122:0/2551543594 shutdown_connections 2024-03-06T07:26:03.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.673+0000 7f14777e4640 1 -- 172.21.15.122:0/2551543594 wait complete. 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.674+0000 7f14777e4640 1 Processor -- start 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.674+0000 7f14777e4640 1 -- start start 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f14777e4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14700b4460 0x7f147014cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f14777e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 0x7f14700c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f14777e4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1470162970 0x7f147015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f14777e4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700ce950 con 0x7f1470161cf0 2024-03-06T07:26:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f14777e4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700d2b60 con 0x7f14700b4460 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f14777e4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700b4840 con 0x7f1470162970 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 0x7f14700c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 0x7f14700c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59568/0 (socket says 172.21.15.122:59568) 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 -- 172.21.15.122:0/4091839228 learned_addr learned my addr 172.21.15.122:0/4091839228 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1475580640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14700b4460 0x7f147014cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1475d81640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1470162970 0x7f147015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 -- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1470162970 msgr2=0x7f147015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1470162970 0x7f147015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 -- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14700b4460 msgr2=0x7f147014cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14700b4460 0x7f147014cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1474d7f640 1 -- 172.21.15.122:0/4091839228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700c8110 con 0x7f1470161cf0 2024-03-06T07:26:03.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.675+0000 7f1475580640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14700b4460 0x7f147014cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f1475d81640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1470162970 0x7f147015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f1474d7f640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 0x7f14700c7810 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f146c00c440 tx=0x7f146c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c017070 con 0x7f1470161cf0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c005ce0 con 0x7f1470161cf0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0072f0 con 0x7f1470161cf0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14700c83f0 con 0x7f1470161cf0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.676+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14700c89b0 con 0x7f1470161cf0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.677+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147004f330 con 0x7f1470161cf0 2024-03-06T07:26:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.677+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f146c01a070 con 0x7f1470161cf0 2024-03-06T07:26:03.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.678+0000 7f1465ffb640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f145406b4d0 0x7f145406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.678+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f146c0895f0 con 0x7f1470161cf0 2024-03-06T07:26:03.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.678+0000 7f1475580640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f145406b4d0 0x7f145406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.678+0000 7f1475580640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f145406b4d0 0x7f145406d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1460036000 tx=0x7f146003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.682+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f146c058470 con 0x7f1470161cf0 2024-03-06T07:26:03.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.934+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f147006b180 con 0x7f1470161cf0 2024-03-06T07:26:03.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.935+0000 7f1465ffb640 1 -- 172.21.15.122:0/4091839228 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1653 (secure 0 0 0) 0x7f146c057d70 con 0x7f1470161cf0 2024-03-06T07:26:03.937 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:03.937 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:03.937 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:26:03.937 INFO:teuthology.orchestra.run.smithi122.stdout: "total_bytes": 773094113280, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "total_avail_bytes": 772829331456, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_bytes": 264781824, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_bytes": 264781824, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_ratio": 0.00034249623422510922, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "num_osds": 8, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "num_per_pool_osds": 8, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "stats_by_class": { 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "ssd": { 2024-03-06T07:26:03.938 INFO:teuthology.orchestra.run.smithi122.stdout: "total_bytes": 773094113280, 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "total_avail_bytes": 772829331456, 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_bytes": 264781824, 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_bytes": 264781824, 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "total_used_raw_ratio": 0.00034249623422510922 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "pools": [ 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "name": ".mgr", 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 1, 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 590368, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 2, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 1160, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 1187840, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 1.6179310478037223e-06, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367085518848 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "name": "cephfs_metadata", 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 12, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 2410, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 22, 2024-03-06T07:26:03.940 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 64, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 65536, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 8.9265292047002731e-08, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367085518848 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "name": "cephfs_data", 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 13, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "stats": { 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "stored": 0, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "objects": 0, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "kb_used": 0, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "bytes_used": 0, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "percent_used": 0, 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: "max_avail": 367085518848 2024-03-06T07:26:03.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:03.942 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:03.942 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:26:03.942 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:03.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f145406b4d0 msgr2=0x7f145406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f145406b4d0 0x7f145406d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1460036000 tx=0x7f146003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 msgr2=0x7f14700c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 0x7f14700c7810 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f146c00c440 tx=0x7f146c001e70 comp rx=0 tx=0).stop 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 shutdown_connections 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f145406b4d0 0x7f145406d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1470162970 0x7f147015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1470161cf0 0x7f14700c7810 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 --2- 172.21.15.122:0/4091839228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14700b4460 0x7f147014cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 >> 172.21.15.122:0/4091839228 conn(0x7f14700d3ce0 msgr2=0x7f147015ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 shutdown_connections 2024-03-06T07:26:03.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:03.937+0000 7f14777e4640 1 -- 172.21.15.122:0/4091839228 wait complete. 2024-03-06T07:26:03.949 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:26:03.949 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:03.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:03.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:04.043 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:04.043 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:04.043 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.058 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:04.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.216+0000 7fc4a08c9640 1 Processor -- start 2024-03-06T07:26:04.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.216+0000 7fc4a08c9640 1 -- start start 2024-03-06T07:26:04.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc4a08c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc49c0b4420 0x7fc49c0cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc4a08c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c0d0a70 con 0x7fc49c0ce0d0 2024-03-06T07:26:04.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc4a08c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c0d0d10 con 0x7fc49c06afb0 2024-03-06T07:26:04.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc4a08c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc49c0d0850 con 0x7fc49c0b4420 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc499d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc49c0b4420 0x7fc49c0cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc49ad9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc49c0ce0d0 0x7fc49c15c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59880/0 (socket says 172.21.15.122:59880) 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc49ad9d640 1 -- 172.21.15.122:0/3026757588 learned_addr learned my addr 172.21.15.122:0/3026757588 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.217+0000 7fc499d9b640 1 -- 172.21.15.122:0/3026757588 >> v1:172.21.15.169:6789/0 conn(0x7fc49c06afb0 legacy=0x7fc49c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.218+0000 7fc499d9b640 1 -- 172.21.15.122:0/3026757588 >> v1:172.21.15.122:6789/0 conn(0x7fc49c0ce0d0 legacy=0x7fc49c15c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.218+0000 7fc499d9b640 1 -- 172.21.15.122:0/3026757588 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c0d0ff0 con 0x7fc49c0b4420 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.218+0000 7fc499d9b640 1 --2- 172.21.15.122:0/3026757588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c0cdb90 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc490009d90 tx=0x7fc49002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a250b034359bf0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.218+0000 7fc49959a640 1 -- 172.21.15.122:0/3026757588 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490039070 con 0x7fc49c0b4420 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.218+0000 7fc49959a640 1 -- 172.21.15.122:0/3026757588 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49002cbd0 con 0x7fc49c0b4420 2024-03-06T07:26:04.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.218+0000 7fc49959a640 1 -- 172.21.15.122:0/3026757588 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490034440 con 0x7fc49c0b4420 2024-03-06T07:26:04.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 -- 172.21.15.122:0/3026757588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 msgr2=0x7fc49c0cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/3026757588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c0cdb90 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc490009d90 tx=0x7fc49002be20 comp rx=0 tx=0).stop 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 -- 172.21.15.122:0/3026757588 shutdown_connections 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/3026757588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c0cdb90 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 -- 172.21.15.122:0/3026757588 >> 172.21.15.122:0/3026757588 conn(0x7fc49c06c0c0 msgr2=0x7fc49c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 -- 172.21.15.122:0/3026757588 shutdown_connections 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.219+0000 7fc4a08c9640 1 -- 172.21.15.122:0/3026757588 wait complete. 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.220+0000 7fc4a08c9640 1 Processor -- start 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.220+0000 7fc4a08c9640 1 -- start start 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc4a08c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc49c06afb0 0x7fc49c1e48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc4a08c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c13dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc4a08c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc49c0ce0d0 0x7fc49c155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc4a08c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c0ceb90 con 0x7fc49c0b4420 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc4a08c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c145840 con 0x7fc49c0ce0d0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc4a08c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c15cba0 con 0x7fc49c06afb0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c13dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c13dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59596/0 (socket says 172.21.15.122:59596) 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 -- 172.21.15.122:0/328397086 learned_addr learned my addr 172.21.15.122:0/328397086 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc49a59c640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc49c06afb0 0x7fc49c1e48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 -- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc49c06afb0 msgr2=0x7fc49c1e48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc49c06afb0 0x7fc49c1e48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc49ad9d640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc49c0ce0d0 0x7fc49c155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 -- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc49c0ce0d0 msgr2=0x7fc49c155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc49c0ce0d0 0x7fc49c155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc499d9b640 1 -- 172.21.15.122:0/328397086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c13e530 con 0x7fc49c0b4420 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc49a59c640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc49c06afb0 0x7fc49c1e48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.221+0000 7fc49ad9d640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc49c0ce0d0 0x7fc49c155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.222+0000 7fc499d9b640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c13dff0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc490037040 tx=0x7fc490004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.222+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490044020 con 0x7fc49c0b4420 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.222+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49003f670 con 0x7fc49c0b4420 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.222+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49c13e7d0 con 0x7fc49c0b4420 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.222+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490039040 con 0x7fc49c0b4420 2024-03-06T07:26:04.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.222+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49c13ed90 con 0x7fc49c0b4420 2024-03-06T07:26:04.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.223+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49c04f330 con 0x7fc49c0b4420 2024-03-06T07:26:04.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.223+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc490047020 con 0x7fc49c0b4420 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.223+0000 7fc48affd640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc47c06b4d0 0x7fc47c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.224+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4900ace70 con 0x7fc49c0b4420 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.224+0000 7fc49a59c640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc47c06b4d0 0x7fc47c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.224+0000 7fc49a59c640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc47c06b4d0 0x7fc47c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc494007fd0 tx=0x7fc494009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.228+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc49004e030 con 0x7fc49c0b4420 2024-03-06T07:26:04.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.401+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc49c0b5cb0 con 0x7fc49c0b4420 2024-03-06T07:26:04.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.403+0000 7fc48affd640 1 -- 172.21.15.122:0/328397086 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16660 (secure 0 0 0) 0x7fc49007be00 con 0x7fc49c0b4420 2024-03-06T07:26:04.405 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:04.405 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":82,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:52.668124+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:04.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc47c06b4d0 msgr2=0x7fc47c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc47c06b4d0 0x7fc47c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc494007fd0 tx=0x7fc494009040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 msgr2=0x7fc49c13dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c13dff0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc490037040 tx=0x7fc490004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 shutdown_connections 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc47c06b4d0 0x7fc47c06d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc49c0ce0d0 0x7fc49c155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc49c0b4420 0x7fc49c13dff0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 --2- 172.21.15.122:0/328397086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc49c06afb0 0x7fc49c1e48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 >> 172.21.15.122:0/328397086 conn(0x7fc49c06c0c0 msgr2=0x7fc49c142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 shutdown_connections 2024-03-06T07:26:04.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.405+0000 7fc4a08c9640 1 -- 172.21.15.122:0/328397086 wait complete. 2024-03-06T07:26:04.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:49.219949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:49.969609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:04.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:04.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.520+0000 7f0b637d7640 1 Processor -- start 2024-03-06T07:26:04.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.520+0000 7f0b637d7640 1 -- start start 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b637d7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0b5c15ecd0 0x7f0b5c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b637d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0cfa50 con 0x7f0b5c0d14a0 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b637d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0b45c0 con 0x7f0b5c0d10c0 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b637d7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5c15c050 con 0x7f0b5c15ecd0 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b60d72640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0b5c0d14a0 0x7f0b5c15e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47580/0 (socket says 172.21.15.122:47580) 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b60d72640 1 -- 172.21.15.122:0/2552477683 learned_addr learned my addr 172.21.15.122:0/2552477683 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943264414 0 0) 0x7f0b5c0cfa50 con 0x7f0b5c0d14a0 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b48003520 con 0x7f0b5c0d14a0 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b61d74640 1 --2- 172.21.15.122:0/2552477683 >> v2:172.21.15.169:3300/0 conn(0x7f0b5c15ecd0 0x7f0b5c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782479291 0 0) 0x7f0b48003520 con 0x7f0b5c0d14a0 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 >> v2:172.21.15.169:3300/0 conn(0x7f0b5c15ecd0 msgr2=0x7f0b5c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 --2- 172.21.15.122:0/2552477683 >> v2:172.21.15.169:3300/0 conn(0x7f0b5c15ecd0 0x7f0b5c0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 >> v1:172.21.15.169:6789/0 conn(0x7f0b5c0d10c0 legacy=0x7f0b5c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c15f160 con 0x7f0b5c0d14a0 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f0b4c001670 con 0x7f0b5c0d14a0 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 >> v1:172.21.15.122:6789/0 conn(0x7f0b5c0d14a0 legacy=0x7f0b5c15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b48003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b48004800 0x7f0b48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b480051c0 0x7f0b4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4800c0c0 con 0x7f0b480039b0 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4800c270 con 0x7f0b480051c0 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4800c470 con 0x7f0b48004800 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.521+0000 7f0b61573640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b48003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b60d72640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b48004800 0x7f0b48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61d74640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b480051c0 0x7f0b4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61573640 1 -- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b48004800 msgr2=0x7f0b48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61573640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b48004800 0x7f0b48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61573640 1 -- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b480051c0 msgr2=0x7f0b4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61573640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b480051c0 0x7f0b4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61573640 1 -- 172.21.15.122:0/2552477683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b48003520 con 0x7f0b480039b0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61d74640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b480051c0 0x7f0b4800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61d74640 1 -- 172.21.15.122:0/2552477683 reap_dead start 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.522+0000 7f0b61573640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b48003db0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0b58003d80 tx=0x7f0b48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edebff960a05698b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58037070 con 0x7f0b480039b0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58032600 con 0x7f0b480039b0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b53fff640 1 -- 172.21.15.122:0/2552477683 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58003980 con 0x7f0b480039b0 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 -- 172.21.15.122:0/2552477683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 msgr2=0x7f0b48003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b48003db0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0b58003d80 tx=0x7f0b48005830 comp rx=0 tx=0).stop 2024-03-06T07:26:04.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 -- 172.21.15.122:0/2552477683 shutdown_connections 2024-03-06T07:26:04.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 --2- 172.21.15.122:0/2552477683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b48003db0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 -- 172.21.15.122:0/2552477683 >> 172.21.15.122:0/2552477683 conn(0x7f0b5c0b3800 msgr2=0x7f0b5c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 -- 172.21.15.122:0/2552477683 shutdown_connections 2024-03-06T07:26:04.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.523+0000 7f0b637d7640 1 -- 172.21.15.122:0/2552477683 wait complete. 2024-03-06T07:26:04.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.524+0000 7f0b637d7640 1 Processor -- start 2024-03-06T07:26:04.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.524+0000 7f0b637d7640 1 -- start start 2024-03-06T07:26:04.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b637d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b5c1e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b637d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b5c0cf1e0 0x7f0b5c14ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b637d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b5c0b60c0 0x7f0b5c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b637d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c0cfcf0 con 0x7f0b480039b0 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b637d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c0d0060 con 0x7f0b5c0b60c0 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b637d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c0d04f0 con 0x7f0b5c0cf1e0 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61573640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b5c1e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61573640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b5c1e6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54792/0 (socket says 172.21.15.122:54792) 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61573640 1 -- 172.21.15.122:0/1031591826 learned_addr learned my addr 172.21.15.122:0/1031591826 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b60d72640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b5c0cf1e0 0x7f0b5c14ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61573640 1 -- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b5c0cf1e0 msgr2=0x7f0b5c14ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61d74640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b5c0b60c0 0x7f0b5c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61573640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b5c0cf1e0 0x7f0b5c14ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.525+0000 7f0b61573640 1 -- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b5c0b60c0 msgr2=0x7f0b5c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b61573640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b5c0b60c0 0x7f0b5c0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b61573640 1 -- 172.21.15.122:0/1031591826 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c0cd2c0 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b61573640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b5c1e6850 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0b58003ea0 tx=0x7f0b580036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58042030 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58032400 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c0cd5a0 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58037040 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.526+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c0cdb60 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.527+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c09fe60 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.527+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b58045070 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.528+0000 7f0b51ffb640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b4006b500 0x7f0b4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.528+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b580abcb0 con 0x7f0b480039b0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.528+0000 7f0b60d72640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b4006b500 0x7f0b4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.528+0000 7f0b60d72640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b4006b500 0x7f0b4006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c000f60 tx=0x7f0b4c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.532+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b5807aaa0 con 0x7f0b480039b0 2024-03-06T07:26:04.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.704+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b5c068ff0 con 0x7f0b480039b0 2024-03-06T07:26:04.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.704+0000 7f0b51ffb640 1 -- 172.21.15.122:0/1031591826 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b5807a3a0 con 0x7f0b480039b0 2024-03-06T07:26:04.706 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b4006b500 msgr2=0x7f0b4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b4006b500 0x7f0b4006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c000f60 tx=0x7f0b4c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 msgr2=0x7f0b5c1e6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b5c1e6850 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0b58003ea0 tx=0x7f0b580036f0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 shutdown_connections 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0b4006b500 0x7f0b4006d9c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b5c0b60c0 0x7f0b5c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0b5c0cf1e0 0x7f0b5c14ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 --2- 172.21.15.122:0/1031591826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b480039b0 0x7f0b5c1e6850 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 >> 172.21.15.122:0/1031591826 conn(0x7f0b5c0b3800 msgr2=0x7f0b5c070170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 shutdown_connections 2024-03-06T07:26:04.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.706+0000 7f0b637d7640 1 -- 172.21.15.122:0/1031591826 wait complete. 2024-03-06T07:26:04.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:04.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.820+0000 7f1bfebe9640 1 Processor -- start 2024-03-06T07:26:04.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfebe9640 1 -- start start 2024-03-06T07:26:04.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfebe9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 0x7f1bf80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfebe9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf81618b0 con 0x7f1bf81614d0 2024-03-06T07:26:04.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfebe9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80cf400 con 0x7f1bf80b0e30 2024-03-06T07:26:04.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfebe9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80d4cc0 con 0x7f1bf80d4860 2024-03-06T07:26:04.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfc985640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1bf81614d0 0x7f1bf80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47592/0 (socket says 172.21.15.122:47592) 2024-03-06T07:26:04.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.821+0000 7f1bfc985640 1 -- 172.21.15.122:0/1856278008 learned_addr learned my addr 172.21.15.122:0/1856278008 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:04.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 0x7f1bf80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49383133 0 0) 0x7f1bf81618b0 con 0x7f1bf81614d0 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be8003520 con 0x7f1bf81614d0 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1803327862 0 0) 0x7f1be8003520 con 0x7f1bf81614d0 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 msgr2=0x7f1bf80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 0x7f1bf80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 >> v1:172.21.15.169:6789/0 conn(0x7f1bf80b0e30 legacy=0x7f1bf80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf80cf6a0 con 0x7f1bf81614d0 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f1bec001670 con 0x7f1bf81614d0 2024-03-06T07:26:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 >> v1:172.21.15.122:6789/0 conn(0x7f1bf81614d0 legacy=0x7f1bf80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be80039e0 0x7f1be8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1be8004800 0x7f1be8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 0x7f1be80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80d4cc0 con 0x7f1be8004800 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be800baa0 con 0x7f1be80039e0 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be800bca0 con 0x7f1bf80d4860 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1be8004800 0x7f1be8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bfc985640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be80039e0 0x7f1be8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bfd186640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 0x7f1be80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 -- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 msgr2=0x7f1be80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80d4860 0x7f1be80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 -- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be80039e0 msgr2=0x7f1be8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be80039e0 0x7f1be8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.822+0000 7f1bf7fff640 1 -- 172.21.15.122:0/1856278008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be8003520 con 0x7f1be8004800 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.823+0000 7f1bf7fff640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1be8004800 0x7f1be8004c80 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009a90 tx=0x7f1be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34eb7b1dcc12aa41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.823+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4039070 con 0x7f1be8004800 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.823+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be402c680 con 0x7f1be8004800 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.823+0000 7f1bf77fe640 1 -- 172.21.15.122:0/1856278008 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4009690 con 0x7f1be8004800 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfebe9640 1 -- 172.21.15.122:0/1856278008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1be8004800 msgr2=0x7f1be8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/1856278008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1be8004800 0x7f1be8004c80 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009a90 tx=0x7f1be8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfd186640 1 -- 172.21.15.122:0/1856278008 reap_dead start 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfebe9640 1 -- 172.21.15.122:0/1856278008 shutdown_connections 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfebe9640 1 -- 172.21.15.122:0/1856278008 >> 172.21.15.122:0/1856278008 conn(0x7f1bf80d3d40 msgr2=0x7f1bf80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfebe9640 1 -- 172.21.15.122:0/1856278008 shutdown_connections 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.824+0000 7f1bfebe9640 1 -- 172.21.15.122:0/1856278008 wait complete. 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.825+0000 7f1bfebe9640 1 Processor -- start 2024-03-06T07:26:04.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.825+0000 7f1bfebe9640 1 -- start start 2024-03-06T07:26:04.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfebe9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be8004800 0x7f1bf80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfebe9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80c7870 0x7f1bf815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfebe9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 0x7f1bf80cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfebe9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80b0910 con 0x7f1bf80cc400 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfebe9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80cfc60 con 0x7f1be8004800 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfebe9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80d2020 con 0x7f1bf80c7870 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 0x7f1bf80cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 0x7f1bf80cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54808/0 (socket says 172.21.15.122:54808) 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 -- 172.21.15.122:0/4261283230 learned_addr learned my addr 172.21.15.122:0/4261283230 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfc985640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be8004800 0x7f1bf80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bf7fff640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80c7870 0x7f1bf815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 -- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80c7870 msgr2=0x7f1bf815a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80c7870 0x7f1bf815a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 -- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be8004800 msgr2=0x7f1bf80ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be8004800 0x7f1bf80ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfd186640 1 -- 172.21.15.122:0/4261283230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf80c8050 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.826+0000 7f1bfc985640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be8004800 0x7f1bf80ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.827+0000 7f1bfd186640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 0x7f1bf80cc880 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80cadc0 tx=0x7f1bf000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.827+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf001a070 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.827+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf0006d60 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.827+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf80c8330 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.827+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0003b90 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.827+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf80b6e60 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.828+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf80b7060 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.828+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bf001d070 con 0x7f1bf80cc400 2024-03-06T07:26:04.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.828+0000 7f1bf57fa640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1bdc06b500 0x7f1bdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.829+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bf0089740 con 0x7f1bf80cc400 2024-03-06T07:26:04.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.829+0000 7f1bfc985640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1bdc06b500 0x7f1bdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.829+0000 7f1bfc985640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1bdc06b500 0x7f1bdc06d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000c00 tx=0x7f1bec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:04.833+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bf80b7060 con 0x7f1bf80cc400 2024-03-06T07:26:05.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.104+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1bf8068ff0 con 0x7f1bf80cc400 2024-03-06T07:26:05.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.105+0000 7f1bf57fa640 1 -- 172.21.15.122:0/4261283230 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1bf0057f70 con 0x7f1bf80cc400 2024-03-06T07:26:05.107 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1bdc06b500 msgr2=0x7f1bdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1bdc06b500 0x7f1bdc06d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000c00 tx=0x7f1bec009040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 msgr2=0x7f1bf80cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 0x7f1bf80cc880 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80cadc0 tx=0x7f1bf000fd40 comp rx=0 tx=0).stop 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 shutdown_connections 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1bdc06b500 0x7f1bdc06d9c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf80cc400 0x7f1bf80cc880 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1bf80c7870 0x7f1bf815a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 --2- 172.21.15.122:0/4261283230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be8004800 0x7f1bf80ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.107+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 >> 172.21.15.122:0/4261283230 conn(0x7f1bf80d3d40 msgr2=0x7f1bf806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.108+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 shutdown_connections 2024-03-06T07:26:05.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.108+0000 7f1bfebe9640 1 -- 172.21.15.122:0/4261283230 wait complete. 2024-03-06T07:26:05.121 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:05.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.223+0000 7fa65e847640 1 Processor -- start 2024-03-06T07:26:05.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.223+0000 7fa65e847640 1 -- start start 2024-03-06T07:26:05.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa65e847640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6581614d0 0x7fa6580d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa65e847640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa6580d24b0 0x7fa6580b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa65e847640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6581618b0 con 0x7fa6580d1910 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa65e847640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6580cf400 con 0x7fa6581614d0 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa65e847640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6580cf6a0 con 0x7fa6580d24b0 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6581614d0 0x7fa6580d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6581614d0 0x7fa6580d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54810/0 (socket says 172.21.15.122:54810) 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 -- 172.21.15.122:0/13372965 learned_addr learned my addr 172.21.15.122:0/13372965 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:05.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa65cde4640 1 --2- 172.21.15.122:0/13372965 >> v2:172.21.15.169:3300/0 conn(0x7fa6580d24b0 0x7fa6580b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 -- 172.21.15.122:0/13372965 >> v2:172.21.15.169:3300/0 conn(0x7fa6580d24b0 msgr2=0x7fa6580b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 --2- 172.21.15.122:0/13372965 >> v2:172.21.15.169:3300/0 conn(0x7fa6580d24b0 0x7fa6580b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 -- 172.21.15.122:0/13372965 >> v1:172.21.15.169:6789/0 conn(0x7fa6580d1910 legacy=0x7fa6580d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.224+0000 7fa657fff640 1 -- 172.21.15.122:0/13372965 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6580cf980 con 0x7fa6581614d0 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.225+0000 7fa65cde4640 1 --2- 172.21.15.122:0/13372965 >> v2:172.21.15.169:3300/0 conn(0x7fa6580d24b0 0x7fa6580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.225+0000 7fa657fff640 1 --2- 172.21.15.122:0/13372965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6581614d0 0x7fa6580d4860 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa6580b2a70 tx=0x7fa64c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc0658ded965b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.225+0000 7fa656ffd640 1 -- 172.21.15.122:0/13372965 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c035020 con 0x7fa6581614d0 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.225+0000 7fa656ffd640 1 -- 172.21.15.122:0/13372965 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c031870 con 0x7fa6581614d0 2024-03-06T07:26:05.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.225+0000 7fa656ffd640 1 -- 172.21.15.122:0/13372965 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c031c40 con 0x7fa6581614d0 2024-03-06T07:26:05.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 -- 172.21.15.122:0/13372965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6581614d0 msgr2=0x7fa6580d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 --2- 172.21.15.122:0/13372965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6581614d0 0x7fa6580d4860 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa6580b2a70 tx=0x7fa64c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 -- 172.21.15.122:0/13372965 shutdown_connections 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 --2- 172.21.15.122:0/13372965 >> v2:172.21.15.169:3300/0 conn(0x7fa6580d24b0 0x7fa6580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 --2- 172.21.15.122:0/13372965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6581614d0 0x7fa6580d4860 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 -- 172.21.15.122:0/13372965 >> 172.21.15.122:0/13372965 conn(0x7fa6580d3d40 msgr2=0x7fa6580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 -- 172.21.15.122:0/13372965 shutdown_connections 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.226+0000 7fa65e847640 1 -- 172.21.15.122:0/13372965 wait complete. 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.227+0000 7fa65e847640 1 Processor -- start 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 -- start start 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 0x7fa65814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6580d24b0 0x7fa6580b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6581614d0 0x7fa6580b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580b1370 con 0x7fa6580d1910 2024-03-06T07:26:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580b0910 con 0x7fa6581614d0 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65e847640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580b14f0 con 0x7fa6580d24b0 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa657fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 0x7fa65814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa657fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 0x7fa65814f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54826/0 (socket says 172.21.15.122:54826) 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa657fff640 1 -- 172.21.15.122:0/1657691120 learned_addr learned my addr 172.21.15.122:0/1657691120 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa65cde4640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6581614d0 0x7fa6580b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.228+0000 7fa6577fe640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6580d24b0 0x7fa6580b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa657fff640 1 -- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6580d24b0 msgr2=0x7fa6580b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa657fff640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6580d24b0 0x7fa6580b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa657fff640 1 -- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6581614d0 msgr2=0x7fa6580b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa657fff640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6581614d0 0x7fa6580b5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa657fff640 1 -- 172.21.15.122:0/1657691120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6580b69d0 con 0x7fa6580d1910 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa657fff640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 0x7fa65814f220 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c040070 con 0x7fa6580d1910 2024-03-06T07:26:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c03a420 con 0x7fa6580d1910 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6580b6cb0 con 0x7fa6580d1910 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c035040 con 0x7fa6580d1910 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.229+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6580b7270 con 0x7fa6580d1910 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.230+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6580b36b0 con 0x7fa6580d1910 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.231+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa64c038020 con 0x7fa6580d1910 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.231+0000 7fa654ff9640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa64406b4d0 0x7fa64406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.231+0000 7fa6577fe640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa64406b4d0 0x7fa64406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.231+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa64c0ac110 con 0x7fa6580d1910 2024-03-06T07:26:05.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.232+0000 7fa6577fe640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa64406b4d0 0x7fa64406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa65815ec70 tx=0x7fa648007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.235+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6580b36b0 con 0x7fa6580d1910 2024-03-06T07:26:05.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.458+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa658068ff0 con 0x7fa6580d1910 2024-03-06T07:26:05.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.459+0000 7fa654ff9640 1 -- 172.21.15.122:0/1657691120 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa64c07a940 con 0x7fa6580d1910 2024-03-06T07:26:05.461 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:26:05.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa64406b4d0 msgr2=0x7fa64406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa64406b4d0 0x7fa64406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa65815ec70 tx=0x7fa648007e30 comp rx=0 tx=0).stop 2024-03-06T07:26:05.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 msgr2=0x7fa65814f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 0x7fa65814f220 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 shutdown_connections 2024-03-06T07:26:05.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa64406b4d0 0x7fa64406d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6581614d0 0x7fa6580b5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6580d24b0 0x7fa6580b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 --2- 172.21.15.122:0/1657691120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6580d1910 0x7fa65814f220 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 >> 172.21.15.122:0/1657691120 conn(0x7fa6580d3d40 msgr2=0x7fa6580d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 shutdown_connections 2024-03-06T07:26:05.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.461+0000 7fa65e847640 1 -- 172.21.15.122:0/1657691120 wait complete. 2024-03-06T07:26:05.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:05.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.573+0000 7f98ddd8c640 1 Processor -- start 2024-03-06T07:26:05.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.573+0000 7f98ddd8c640 1 -- start start 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98ddd8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98ddd8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d80d25e0 con 0x7f98d80c90f0 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98ddd8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d80d2880 con 0x7f98d815e000 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98ddd8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80c7c50 con 0x7f98d80d0d90 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98d7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f98d80c90f0 0x7f98d80ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47598/0 (socket says 172.21.15.122:47598) 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98d7fff640 1 -- 172.21.15.122:0/1756356651 learned_addr learned my addr 172.21.15.122:0/1756356651 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98cffff640 1 --2- 172.21.15.122:0/1756356651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98d6ffd640 1 -- 172.21.15.122:0/1756356651 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161693484 0 0) 0x7f98d80d25e0 con 0x7f98d80c90f0 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.574+0000 7f98d6ffd640 1 -- 172.21.15.122:0/1756356651 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4003520 con 0x7f98d80c90f0 2024-03-06T07:26:05.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.575+0000 7f98d6ffd640 1 -- 172.21.15.122:0/1756356651 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224821628 0 0) 0x7f98c4003520 con 0x7f98d80c90f0 2024-03-06T07:26:05.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.575+0000 7f98cffff640 1 -- 172.21.15.122:0/1756356651 >> v1:172.21.15.169:6789/0 conn(0x7f98d815e000 legacy=0x7f98d80d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.575+0000 7f98cffff640 1 -- 172.21.15.122:0/1756356651 >> v1:172.21.15.122:6789/0 conn(0x7f98d80c90f0 legacy=0x7f98d80ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.575+0000 7f98cffff640 1 -- 172.21.15.122:0/1756356651 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d80d2b40 con 0x7f98d80d0d90 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.575+0000 7f98cffff640 1 --2- 172.21.15.122:0/1756356651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80ce7c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009800 tx=0x7f98c002b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b592a25f857c958b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98d6ffd640 1 -- 172.21.15.122:0/1756356651 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0037070 con 0x7f98d80d0d90 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98d6ffd640 1 -- 172.21.15.122:0/1756356651 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c0030510 con 0x7f98d80d0d90 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98d6ffd640 1 -- 172.21.15.122:0/1756356651 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d00019f0 con 0x7f98d80d0d90 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 -- 172.21.15.122:0/1756356651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 msgr2=0x7f98d80ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/1756356651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80ce7c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009800 tx=0x7f98c002b730 comp rx=0 tx=0).stop 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 -- 172.21.15.122:0/1756356651 shutdown_connections 2024-03-06T07:26:05.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/1756356651 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80ce7c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 -- 172.21.15.122:0/1756356651 >> 172.21.15.122:0/1756356651 conn(0x7f98d80d3cf0 msgr2=0x7f98d80d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 -- 172.21.15.122:0/1756356651 shutdown_connections 2024-03-06T07:26:05.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.576+0000 7f98ddd8c640 1 -- 172.21.15.122:0/1756356651 wait complete. 2024-03-06T07:26:05.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.578+0000 7f98ddd8c640 1 Processor -- start 2024-03-06T07:26:05.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.578+0000 7f98ddd8c640 1 -- start start 2024-03-06T07:26:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98ddd8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98d80c90f0 0x7f98d81612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98ddd8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98ddd8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 0x7f98d80cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98ddd8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d815e3e0 con 0x7f98d815e000 2024-03-06T07:26:05.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98ddd8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80d22d0 con 0x7f98d80c90f0 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98ddd8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80d4810 con 0x7f98d80d0d90 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 0x7f98d80cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 0x7f98d80cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54832/0 (socket says 172.21.15.122:54832) 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 -- 172.21.15.122:0/4020628147 learned_addr learned my addr 172.21.15.122:0/4020628147 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 -- 172.21.15.122:0/4020628147 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 msgr2=0x7f98d80cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 -- 172.21.15.122:0/4020628147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98d80c90f0 msgr2=0x7f98d81612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98d80c90f0 0x7f98d81612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.579+0000 7f98d7fff640 1 -- 172.21.15.122:0/4020628147 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d809dbc0 con 0x7f98d815e000 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.580+0000 7f98d7fff640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 0x7f98d80cc180 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f98d000ca70 tx=0x7f98d000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.580+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0018070 con 0x7f98d815e000 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.580+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d000a6a0 con 0x7f98d815e000 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.580+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d000aad0 con 0x7f98d815e000 2024-03-06T07:26:05.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.580+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d809dea0 con 0x7f98d815e000 2024-03-06T07:26:05.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.580+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d80ccd50 con 0x7f98d815e000 2024-03-06T07:26:05.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.581+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d80cd7b0 con 0x7f98d815e000 2024-03-06T07:26:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.581+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98d001b070 con 0x7f98d815e000 2024-03-06T07:26:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.582+0000 7f98d4ff9640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.582+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98d00899a0 con 0x7f98d815e000 2024-03-06T07:26:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.582+0000 7f98d77fe640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.582+0000 7f98d77fe640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f98c8001690 tx=0x7f98c8007950 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.586+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98d000ac70 con 0x7f98d815e000 2024-03-06T07:26:05.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.781+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f98d804f330 con 0x7f98d815e000 2024-03-06T07:26:05.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.781+0000 7f98d4ff9640 1 -- 172.21.15.122:0/4020628147 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7f98d00588b0 con 0x7f98d815e000 2024-03-06T07:26:05.783 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:26:05.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98b806b4d0 msgr2=0x7f98b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f98c8001690 tx=0x7f98c8007950 comp rx=0 tx=0).stop 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 msgr2=0x7f98d80cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 0x7f98d80cc180 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f98d000ca70 tx=0x7f98d000cf30 comp rx=0 tx=0).stop 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 shutdown_connections 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98d815e000 0x7f98d80cc180 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98d80d0d90 0x7f98d80cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 --2- 172.21.15.122:0/4020628147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98d80c90f0 0x7f98d81612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 >> 172.21.15.122:0/4020628147 conn(0x7f98d80d3cf0 msgr2=0x7f98d80b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 shutdown_connections 2024-03-06T07:26:05.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:05.783+0000 7f98ddd8c640 1 -- 172.21.15.122:0/4020628147 wait complete. 2024-03-06T07:26:05.796 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:26:05.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:05.796 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:05.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[66148]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:05.824 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:26:05.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:05.824 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:05.896 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:26:05.896 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:05.915 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:05.915 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:05.915 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:05.973 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:05.973 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:05.974 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:05.988 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:05.989 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:26:05.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:05.989 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:06.045 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:06.045 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:26:06.045 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:06.062 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:06.062 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:26:06.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:06.063 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:06.119 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:06.120 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:06.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.222+0000 7f101a22a640 1 Processor -- start 2024-03-06T07:26:06.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.222+0000 7f101a22a640 1 -- start start 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f101a22a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140d4760 0x7f10140b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f101a22a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10140d25d0 con 0x7f10140b4170 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f101a22a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10140d2870 con 0x7f10140d0a80 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f101a22a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140d4bf0 con 0x7f10140d4760 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f10137fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f10140b4170 0x7f10140d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47608/0 (socket says 172.21.15.122:47608) 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f10137fe640 1 -- 172.21.15.122:0/3832883205 learned_addr learned my addr 172.21.15.122:0/3832883205 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:06.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.223+0000 7f100affd640 1 --2- 172.21.15.122:0/3832883205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140d4760 0x7f10140b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898807993 0 0) 0x7f10140d25d0 con 0x7f10140b4170 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff4003520 con 0x7f10140b4170 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805124436 0 0) 0x7f10140d2870 con 0x7f10140d0a80 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10140d25d0 con 0x7f10140d0a80 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2876990298 0 0) 0x7f0ff4003520 con 0x7f10140b4170 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f100affd640 1 -- 172.21.15.122:0/3832883205 >> v1:172.21.15.169:6789/0 conn(0x7f10140d0a80 legacy=0x7f10140cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f100affd640 1 -- 172.21.15.122:0/3832883205 >> v1:172.21.15.122:6789/0 conn(0x7f10140b4170 legacy=0x7f10140d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.224+0000 7f100affd640 1 -- 172.21.15.122:0/3832883205 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10140d2b50 con 0x7f10140d4760 2024-03-06T07:26:06.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.225+0000 7f100affd640 1 --2- 172.21.15.122:0/3832883205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140d4760 0x7f10140b6a90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0097d0 tx=0x7f0ffc02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecdafeda07bafe1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.225+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc037070 con 0x7f10140d4760 2024-03-06T07:26:06.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.225+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc030510 con 0x7f10140d4760 2024-03-06T07:26:06.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.225+0000 7f1012ffd640 1 -- 172.21.15.122:0/3832883205 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004001670 con 0x7f10140d4760 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 -- 172.21.15.122:0/3832883205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140d4760 msgr2=0x7f10140b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 --2- 172.21.15.122:0/3832883205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140d4760 0x7f10140b6a90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0097d0 tx=0x7f0ffc02b800 comp rx=0 tx=0).stop 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 -- 172.21.15.122:0/3832883205 shutdown_connections 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 --2- 172.21.15.122:0/3832883205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140d4760 0x7f10140b6a90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 -- 172.21.15.122:0/3832883205 >> 172.21.15.122:0/3832883205 conn(0x7f10140d3c40 msgr2=0x7f10140d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 -- 172.21.15.122:0/3832883205 shutdown_connections 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.226+0000 7f101a22a640 1 -- 172.21.15.122:0/3832883205 wait complete. 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.227+0000 7f101a22a640 1 Processor -- start 2024-03-06T07:26:06.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.227+0000 7f101a22a640 1 -- start start 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f101a22a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10140b4170 0x7f10140b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f101a22a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 0x7f10140cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f101a22a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140cbf70 0x7f1014157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f101a22a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140b4dd0 con 0x7f10140d0a80 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f101a22a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140ce9e0 con 0x7f10140b4170 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f101a22a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140b4f50 con 0x7f10140cbf70 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 0x7f10140cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 0x7f10140cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54844/0 (socket says 172.21.15.122:54844) 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 -- 172.21.15.122:0/3684817281 learned_addr learned my addr 172.21.15.122:0/3684817281 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f1013fff640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140cbf70 0x7f1014157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f10137fe640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10140b4170 0x7f10140b85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 -- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140cbf70 msgr2=0x7f1014157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140cbf70 0x7f1014157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 -- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10140b4170 msgr2=0x7f10140b85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10140b4170 0x7f10140b85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f100affd640 1 -- 172.21.15.122:0/3684817281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10140cc420 con 0x7f10140d0a80 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f1013fff640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140cbf70 0x7f1014157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.228+0000 7f10137fe640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10140b4170 0x7f10140b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.229+0000 7f100affd640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 0x7f10140cba30 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc035040 tx=0x7f0ffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.229+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc042020 con 0x7f10140d0a80 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.229+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc038040 con 0x7f10140d0a80 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.229+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10140c94c0 con 0x7f10140d0a80 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.229+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc037070 con 0x7f10140d0a80 2024-03-06T07:26:06.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.229+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10140c9d60 con 0x7f10140d0a80 2024-03-06T07:26:06.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.230+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10140c98b0 con 0x7f10140d0a80 2024-03-06T07:26:06.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.230+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ffc045020 con 0x7f10140d0a80 2024-03-06T07:26:06.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.230+0000 7f1010ff9640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ff806b4d0 0x7f0ff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.231+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ffc0abd10 con 0x7f10140d0a80 2024-03-06T07:26:06.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.231+0000 7f10137fe640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ff806b4d0 0x7f0ff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.231+0000 7f10137fe640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ff806b4d0 0x7f0ff806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1004000fd0 tx=0x7f1004009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.235+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ffc04c030 con 0x7f10140d0a80 2024-03-06T07:26:06.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.449+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f101404f330 con 0x7f10140d0a80 2024-03-06T07:26:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.450+0000 7f1010ff9640 1 -- 172.21.15.122:0/3684817281 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f0ffc07ac20 con 0x7f10140d0a80 2024-03-06T07:26:06.452 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:06.452 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:26:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ff806b4d0 msgr2=0x7f0ff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ff806b4d0 0x7f0ff806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1004000fd0 tx=0x7f1004009040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 msgr2=0x7f10140cba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 0x7f10140cba30 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc035040 tx=0x7f0ffc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 shutdown_connections 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ff806b4d0 0x7f0ff806d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10140cbf70 0x7f1014157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10140d0a80 0x7f10140cba30 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 --2- 172.21.15.122:0/3684817281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10140b4170 0x7f10140b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 >> 172.21.15.122:0/3684817281 conn(0x7f10140d3c40 msgr2=0x7f10140b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 shutdown_connections 2024-03-06T07:26:06.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.452+0000 7f101a22a640 1 -- 172.21.15.122:0/3684817281 wait complete. 2024-03-06T07:26:06.465 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:06.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.563+0000 7fa988c9a640 1 Processor -- start 2024-03-06T07:26:06.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.563+0000 7fa988c9a640 1 -- start start 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa988c9a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa98415dfe0 0x7fa9840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa988c9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa9840d29c0 0x7fa9840b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa988c9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9840b3430 con 0x7fa9840cf830 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa988c9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9840b31e0 con 0x7fa9840d29c0 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa988c9a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9840b3d60 con 0x7fa98415dfe0 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa9840d29c0 0x7fa9840b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa9840d29c0 0x7fa9840b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54854/0 (socket says 172.21.15.122:54854) 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 -- 172.21.15.122:0/558535725 learned_addr learned my addr 172.21.15.122:0/558535725 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa97bfff640 1 --1- 172.21.15.122:0/558535725 >> v1:172.21.15.122:6789/0 conn(0x7fa9840cf830 0x7fa9840d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47616/0 (socket says 172.21.15.122:47616) 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa98259c640 1 --2- 172.21.15.122:0/558535725 >> v2:172.21.15.169:3300/0 conn(0x7fa98415dfe0 0x7fa9840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 -- 172.21.15.122:0/558535725 >> v2:172.21.15.169:3300/0 conn(0x7fa98415dfe0 msgr2=0x7fa9840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 --2- 172.21.15.122:0/558535725 >> v2:172.21.15.169:3300/0 conn(0x7fa98415dfe0 0x7fa9840cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 -- 172.21.15.122:0/558535725 >> v1:172.21.15.122:6789/0 conn(0x7fa9840cf830 legacy=0x7fa9840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa981d9b640 1 -- 172.21.15.122:0/558535725 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804498019 0 0) 0x7fa9840b3430 con 0x7fa9840cf830 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa982d9d640 1 -- 172.21.15.122:0/558535725 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840b3710 con 0x7fa9840d29c0 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.564+0000 7fa98259c640 1 --2- 172.21.15.122:0/558535725 >> v2:172.21.15.169:3300/0 conn(0x7fa98415dfe0 0x7fa9840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.565+0000 7fa982d9d640 1 --2- 172.21.15.122:0/558535725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9840d29c0 0x7fa9840b41f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa984067990 tx=0x7fa97c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1006276347ef9f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.565+0000 7fa981d9b640 1 -- 172.21.15.122:0/558535725 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c02cb20 con 0x7fa9840d29c0 2024-03-06T07:26:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.565+0000 7fa981d9b640 1 -- 172.21.15.122:0/558535725 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c02ccc0 con 0x7fa9840d29c0 2024-03-06T07:26:06.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.565+0000 7fa981d9b640 1 -- 172.21.15.122:0/558535725 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c034610 con 0x7fa9840d29c0 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 -- 172.21.15.122:0/558535725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9840d29c0 msgr2=0x7fa9840b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 --2- 172.21.15.122:0/558535725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9840d29c0 0x7fa9840b41f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa984067990 tx=0x7fa97c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 -- 172.21.15.122:0/558535725 shutdown_connections 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 --2- 172.21.15.122:0/558535725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9840d29c0 0x7fa9840b41f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 --2- 172.21.15.122:0/558535725 >> v2:172.21.15.169:3300/0 conn(0x7fa98415dfe0 0x7fa9840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 -- 172.21.15.122:0/558535725 >> 172.21.15.122:0/558535725 conn(0x7fa9840d3cd0 msgr2=0x7fa9840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 -- 172.21.15.122:0/558535725 shutdown_connections 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.566+0000 7fa988c9a640 1 -- 172.21.15.122:0/558535725 wait complete. 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.567+0000 7fa988c9a640 1 Processor -- start 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.567+0000 7fa988c9a640 1 -- start start 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa988c9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9840cf830 0x7fa9840a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa988c9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9840d29c0 0x7fa98415f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa988c9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 0x7fa9841f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa988c9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840d1df0 con 0x7fa98415dfe0 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa988c9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840d1c70 con 0x7fa9840d29c0 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa988c9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98415e3c0 con 0x7fa9840cf830 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 0x7fa9841f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 0x7fa9841f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54864/0 (socket says 172.21.15.122:54864) 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 -- 172.21.15.122:0/4106911179 learned_addr learned my addr 172.21.15.122:0/4106911179 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa98259c640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9840cf830 0x7fa9840a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa97bfff640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9840d29c0 0x7fa98415f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 -- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9840cf830 msgr2=0x7fa9840a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9840cf830 0x7fa9840a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 -- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9840d29c0 msgr2=0x7fa98415f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9840d29c0 0x7fa98415f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 -- 172.21.15.122:0/4106911179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840a1aa0 con 0x7fa98415dfe0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.568+0000 7fa982d9d640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 0x7fa9841f3a70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa97c036040 tx=0x7fa97c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.569+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c043020 con 0x7fa98415dfe0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.569+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c03e450 con 0x7fa98415dfe0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.569+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c03e860 con 0x7fa98415dfe0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.569+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9841f45a0 con 0x7fa98415dfe0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.569+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98415e960 con 0x7fa98415dfe0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.570+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98415e750 con 0x7fa98415dfe0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.570+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa97c046020 con 0x7fa98415dfe0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.570+0000 7fa97affd640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96406b4d0 0x7fa96406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.571+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa97c0acb70 con 0x7fa98415dfe0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.571+0000 7fa98259c640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96406b4d0 0x7fa96406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.571+0000 7fa98259c640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96406b4d0 0x7fa96406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa9840cf7a0 tx=0x7fa974008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.575+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa97c04d030 con 0x7fa98415dfe0 2024-03-06T07:26:06.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.757+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9840b4ce0 con 0x7fa98415dfe0 2024-03-06T07:26:06.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.758+0000 7fa97affd640 1 -- 172.21.15.122:0/4106911179 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16660 (secure 0 0 0) 0x7fa9840a1aa0 con 0x7fa98415dfe0 2024-03-06T07:26:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:06.761 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":82,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:52.668124+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:06.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96406b4d0 msgr2=0x7fa96406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96406b4d0 0x7fa96406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa9840cf7a0 tx=0x7fa974008040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 msgr2=0x7fa9841f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 0x7fa9841f3a70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa97c036040 tx=0x7fa97c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 shutdown_connections 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa96406b4d0 0x7fa96406d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa98415dfe0 0x7fa9841f3a70 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9840d29c0 0x7fa98415f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 --2- 172.21.15.122:0/4106911179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9840cf830 0x7fa9840a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 >> 172.21.15.122:0/4106911179 conn(0x7fa9840d3cd0 msgr2=0x7fa9840d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 shutdown_connections 2024-03-06T07:26:06.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.760+0000 7fa988c9a640 1 -- 172.21.15.122:0/4106911179 wait complete. 2024-03-06T07:26:06.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:49.219949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:49.969609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:06.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.876+0000 7f7929b56640 1 Processor -- start 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.876+0000 7f7929b56640 1 -- start start 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7929b56640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79240cfa90 0x7f79240cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7929b56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792415ff90 con 0x7f792415bb40 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7929b56640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7924160270 con 0x7f79240b4970 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7929b56640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79240d0440 con 0x7f79240cfa90 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7923fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79240cfa90 0x7f79240cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f79237fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f792415bb40 0x7f792415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47632/0 (socket says 172.21.15.122:47632) 2024-03-06T07:26:06.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f79237fe640 1 -- 172.21.15.122:0/2830492065 learned_addr learned my addr 172.21.15.122:0/2830492065 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:06.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7923fff640 1 -- 172.21.15.122:0/2830492065 >> v1:172.21.15.169:6789/0 conn(0x7f79240b4970 legacy=0x7f792406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7923fff640 1 -- 172.21.15.122:0/2830492065 >> v1:172.21.15.122:6789/0 conn(0x7f792415bb40 legacy=0x7f792415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f79227fc640 1 -- 172.21.15.122:0/2830492065 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664218155 0 0) 0x7f792415ff90 con 0x7f792415bb40 2024-03-06T07:26:06.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.878+0000 7f7923fff640 1 -- 172.21.15.122:0/2830492065 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7924160550 con 0x7f79240cfa90 2024-03-06T07:26:06.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7923fff640 1 --2- 172.21.15.122:0/2830492065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f79240cff00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f791c0097a0 tx=0x7f791c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5eb3e89f70379af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f79227fc640 1 -- 172.21.15.122:0/2830492065 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c035020 con 0x7f79240cfa90 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f79227fc640 1 -- 172.21.15.122:0/2830492065 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791c031830 con 0x7f79240cfa90 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f79227fc640 1 -- 172.21.15.122:0/2830492065 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c031c00 con 0x7f79240cfa90 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 -- 172.21.15.122:0/2830492065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 msgr2=0x7f79240cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 --2- 172.21.15.122:0/2830492065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f79240cff00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f791c0097a0 tx=0x7f791c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 -- 172.21.15.122:0/2830492065 shutdown_connections 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 --2- 172.21.15.122:0/2830492065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f79240cff00 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 -- 172.21.15.122:0/2830492065 >> 172.21.15.122:0/2830492065 conn(0x7f792406b190 msgr2=0x7f792406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 -- 172.21.15.122:0/2830492065 shutdown_connections 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.879+0000 7f7929b56640 1 -- 172.21.15.122:0/2830492065 wait complete. 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.881+0000 7f7929b56640 1 Processor -- start 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.881+0000 7f7929b56640 1 -- start start 2024-03-06T07:26:06.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79240b4970 0x7f792415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f792413e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f792415bb40 0x7f792413b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792406c910 con 0x7f79240cfa90 2024-03-06T07:26:06.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792406d790 con 0x7f792415bb40 2024-03-06T07:26:06.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792415bf20 con 0x7f79240b4970 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f792413e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f792413e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54886/0 (socket says 172.21.15.122:54886) 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 -- 172.21.15.122:0/553504152 learned_addr learned my addr 172.21.15.122:0/553504152 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f79237fe640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79240b4970 0x7f792415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7923fff640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f792415bb40 0x7f792413b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 -- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79240b4970 msgr2=0x7f792415d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79240b4970 0x7f792415d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 -- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f792415bb40 msgr2=0x7f792413b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f792415bb40 0x7f792413b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 -- 172.21.15.122:0/553504152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792413ed70 con 0x7f79240cfa90 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7922ffd640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f792413e830 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f790c00b3e0 tx=0x7f790c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c015020 con 0x7f79240cfa90 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c011850 con 0x7f79240cfa90 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c011c40 con 0x7f79240cfa90 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792415def0 con 0x7f79240cfa90 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.883+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792415e4b0 con 0x7f79240cfa90 2024-03-06T07:26:06.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.884+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79240732b0 con 0x7f79240cfa90 2024-03-06T07:26:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.885+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f790c018020 con 0x7f79240cfa90 2024-03-06T07:26:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.885+0000 7f791bfff640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f790406b4d0 0x7f790406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.885+0000 7f79237fe640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f790406b4d0 0x7f790406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.885+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f790c08a090 con 0x7f79240cfa90 2024-03-06T07:26:06.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.886+0000 7f79237fe640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f790406b4d0 0x7f790406d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f79140012e0 tx=0x7f7914007ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:06.894+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f790c01f020 con 0x7f79240cfa90 2024-03-06T07:26:07.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.070+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79240cf590 con 0x7f79240cfa90 2024-03-06T07:26:07.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.071+0000 7f791bfff640 1 -- 172.21.15.122:0/553504152 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f790c0172c0 con 0x7f79240cfa90 2024-03-06T07:26:07.073 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:26:07.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f790406b4d0 msgr2=0x7f790406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f790406b4d0 0x7f790406d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f79140012e0 tx=0x7f7914007ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 msgr2=0x7f792413e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f792413e830 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f790c00b3e0 tx=0x7f790c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 shutdown_connections 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f790406b4d0 0x7f790406d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f792415bb40 0x7f792413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79240cfa90 0x7f792413e830 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 --2- 172.21.15.122:0/553504152 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79240b4970 0x7f792415d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 >> 172.21.15.122:0/553504152 conn(0x7f792406b190 msgr2=0x7f7924159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 shutdown_connections 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.073+0000 7f7929b56640 1 -- 172.21.15.122:0/553504152 wait complete. 2024-03-06T07:26:07.086 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.188+0000 7f3121e2a640 1 Processor -- start 2024-03-06T07:26:07.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.188+0000 7f3121e2a640 1 -- start start 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f3121e2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0d06e0 0x7f311c15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f3121e2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f311c0cf200 0x7f311c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f3121e2a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0d47f0 con 0x7f311c0d0ac0 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f3121e2a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0ced70 con 0x7f311c0cf200 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f3121e2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0d1ab0 con 0x7f311c0d06e0 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f311c0cf200 0x7f311c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f311c0cf200 0x7f311c0cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54892/0 (socket says 172.21.15.122:54892) 2024-03-06T07:26:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 -- 172.21.15.122:0/2696175656 learned_addr learned my addr 172.21.15.122:0/2696175656 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 -- 172.21.15.122:0/2696175656 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0d06e0 msgr2=0x7f311c15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311b7fe640 1 --2- 172.21.15.122:0/2696175656 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0d06e0 0x7f311c15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 --2- 172.21.15.122:0/2696175656 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0d06e0 0x7f311c15fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 -- 172.21.15.122:0/2696175656 >> v1:172.21.15.169:6789/0 conn(0x7f311c0d0ac0 legacy=0x7f311c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.189+0000 7f311bfff640 1 -- 172.21.15.122:0/2696175656 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0d1d90 con 0x7f311c0cf200 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.190+0000 7f311bfff640 1 --2- 172.21.15.122:0/2696175656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0cf200 0x7f311c0cf690 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3114009e50 tx=0x7f311402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d826bd024e3041 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.190+0000 7f311affd640 1 -- 172.21.15.122:0/2696175656 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311402ca70 con 0x7f311c0cf200 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.190+0000 7f311affd640 1 -- 172.21.15.122:0/2696175656 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311402cc10 con 0x7f311c0cf200 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.190+0000 7f311affd640 1 -- 172.21.15.122:0/2696175656 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31140343f0 con 0x7f311c0cf200 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 -- 172.21.15.122:0/2696175656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0cf200 msgr2=0x7f311c0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 --2- 172.21.15.122:0/2696175656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0cf200 0x7f311c0cf690 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3114009e50 tx=0x7f311402bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 -- 172.21.15.122:0/2696175656 shutdown_connections 2024-03-06T07:26:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 --2- 172.21.15.122:0/2696175656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0cf200 0x7f311c0cf690 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 --2- 172.21.15.122:0/2696175656 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0d06e0 0x7f311c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 -- 172.21.15.122:0/2696175656 >> 172.21.15.122:0/2696175656 conn(0x7f311c0d3cd0 msgr2=0x7f311c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 -- 172.21.15.122:0/2696175656 shutdown_connections 2024-03-06T07:26:07.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.191+0000 7f3121e2a640 1 -- 172.21.15.122:0/2696175656 wait complete. 2024-03-06T07:26:07.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.193+0000 7f3121e2a640 1 Processor -- start 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.193+0000 7f3121e2a640 1 -- start start 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3121e2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0cf200 0x7f311c0c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3121e2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f311c0d06e0 0x7f311c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3121e2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 0x7f311c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3121e2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0b44d0 con 0x7f311c0d0ac0 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3121e2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0ce990 con 0x7f311c0d06e0 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3121e2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0d0f00 con 0x7f311c0cf200 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f311bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 0x7f311c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f311bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 0x7f311c0cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54898/0 (socket says 172.21.15.122:54898) 2024-03-06T07:26:07.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f311bfff640 1 -- 172.21.15.122:0/1578560317 learned_addr learned my addr 172.21.15.122:0/1578560317 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f311b7fe640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0cf200 0x7f311c0c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.194+0000 7f3112ffd640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f311c0d06e0 0x7f311c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f311bfff640 1 -- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0cf200 msgr2=0x7f311c0c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f311bfff640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0cf200 0x7f311c0c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f311bfff640 1 -- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f311c0d06e0 msgr2=0x7f311c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f311bfff640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f311c0d06e0 0x7f311c0ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f311bfff640 1 -- 172.21.15.122:0/1578560317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0c7960 con 0x7f311c0d0ac0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f311bfff640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 0x7f311c0cd4b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3114009e50 tx=0x7f3114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114043020 con 0x7f311c0d0ac0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311403e450 con 0x7f311c0d0ac0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114009510 con 0x7f311c0d0ac0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c159320 con 0x7f311c0d0ac0 2024-03-06T07:26:07.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.195+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c1598e0 con 0x7f311c0d0ac0 2024-03-06T07:26:07.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.196+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c068ff0 con 0x7f311c0d0ac0 2024-03-06T07:26:07.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.196+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3114046020 con 0x7f311c0d0ac0 2024-03-06T07:26:07.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.197+0000 7f3118ff9640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f310006b4d0 0x7f310006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.197+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31140ac940 con 0x7f311c0d0ac0 2024-03-06T07:26:07.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.197+0000 7f311b7fe640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f310006b4d0 0x7f310006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.197+0000 7f311b7fe640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f310006b4d0 0x7f310006d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f311c0b5dc0 tx=0x7f3104008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.201+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f311404d030 con 0x7f311c0d0ac0 2024-03-06T07:26:07.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.377+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f311c08d7a0 con 0x7f311c0d0ac0 2024-03-06T07:26:07.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.378+0000 7f3118ff9640 1 -- 172.21.15.122:0/1578560317 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f311c0c7960 con 0x7f311c0d0ac0 2024-03-06T07:26:07.380 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:26:07.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f310006b4d0 msgr2=0x7f310006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f310006b4d0 0x7f310006d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f311c0b5dc0 tx=0x7f3104008040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 msgr2=0x7f311c0cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 0x7f311c0cd4b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3114009e50 tx=0x7f3114004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:07.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 shutdown_connections 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f310006b4d0 0x7f310006d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0d0ac0 0x7f311c0cd4b0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f311c0d06e0 0x7f311c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 --2- 172.21.15.122:0/1578560317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f311c0cf200 0x7f311c0c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 >> 172.21.15.122:0/1578560317 conn(0x7f311c0d3cd0 msgr2=0x7f311c06d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 shutdown_connections 2024-03-06T07:26:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.380+0000 7f3121e2a640 1 -- 172.21.15.122:0/1578560317 wait complete. 2024-03-06T07:26:07.393 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:07.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.494+0000 7fac9b6a4640 1 Processor -- start 2024-03-06T07:26:07.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac9b6a4640 1 -- start start 2024-03-06T07:26:07.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac9b6a4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fac941614d0 0x7fac940d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac9b6a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fac940d24b0 0x7fac940b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac9b6a4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac941618b0 con 0x7fac940d1910 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac9b6a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940cf400 con 0x7fac940d24b0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac9b6a4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940cf6a0 con 0x7fac941614d0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac99c41640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fac940d24b0 0x7fac940b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac99c41640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fac940d24b0 0x7fac940b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54906/0 (socket says 172.21.15.122:54906) 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.495+0000 7fac99c41640 1 -- 172.21.15.122:0/1649617939 learned_addr learned my addr 172.21.15.122:0/1649617939 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.496+0000 7fac99440640 1 --2- 172.21.15.122:0/1649617939 >> v2:172.21.15.169:3300/0 conn(0x7fac941614d0 0x7fac940d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.496+0000 7fac99c41640 1 -- 172.21.15.122:0/1649617939 >> v2:172.21.15.169:3300/0 conn(0x7fac941614d0 msgr2=0x7fac940d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.496+0000 7fac99c41640 1 --2- 172.21.15.122:0/1649617939 >> v2:172.21.15.169:3300/0 conn(0x7fac941614d0 0x7fac940d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.496+0000 7fac99c41640 1 -- 172.21.15.122:0/1649617939 >> v1:172.21.15.169:6789/0 conn(0x7fac940d1910 legacy=0x7fac940d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.496+0000 7fac99c41640 1 -- 172.21.15.122:0/1649617939 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940cf980 con 0x7fac940d24b0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.496+0000 7fac99c41640 1 --2- 172.21.15.122:0/1649617939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b0e30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fac9408c470 tx=0x7fac8c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178c233304859f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac8bfff640 1 -- 172.21.15.122:0/1649617939 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c02cb20 con 0x7fac940d24b0 2024-03-06T07:26:07.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac8bfff640 1 -- 172.21.15.122:0/1649617939 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c02ccc0 con 0x7fac940d24b0 2024-03-06T07:26:07.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac8bfff640 1 -- 172.21.15.122:0/1649617939 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c034610 con 0x7fac940d24b0 2024-03-06T07:26:07.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 -- 172.21.15.122:0/1649617939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 msgr2=0x7fac940b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/1649617939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b0e30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fac9408c470 tx=0x7fac8c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:07.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 -- 172.21.15.122:0/1649617939 shutdown_connections 2024-03-06T07:26:07.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/1649617939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b0e30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/1649617939 >> v2:172.21.15.169:3300/0 conn(0x7fac941614d0 0x7fac940d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 -- 172.21.15.122:0/1649617939 >> 172.21.15.122:0/1649617939 conn(0x7fac940d3d40 msgr2=0x7fac940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 -- 172.21.15.122:0/1649617939 shutdown_connections 2024-03-06T07:26:07.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.497+0000 7fac9b6a4640 1 -- 172.21.15.122:0/1649617939 wait complete. 2024-03-06T07:26:07.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.499+0000 7fac9b6a4640 1 Processor -- start 2024-03-06T07:26:07.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.499+0000 7fac9b6a4640 1 -- start start 2024-03-06T07:26:07.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac9b6a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac940d1910 0x7fac940b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac9b6a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac9b6a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fac941614d0 0x7fac940ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac9b6a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac940b1370 con 0x7fac940d24b0 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac9b6a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac940b0910 con 0x7fac940d1910 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac9b6a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac940b14f0 con 0x7fac941614d0 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.500+0000 7fac98c3f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54916/0 (socket says 172.21.15.122:54916) 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 -- 172.21.15.122:0/16244466 learned_addr learned my addr 172.21.15.122:0/16244466 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac99440640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac940d1910 0x7fac940b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac99c41640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fac941614d0 0x7fac940ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 -- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fac941614d0 msgr2=0x7fac940ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fac941614d0 0x7fac940ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 -- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac940d1910 msgr2=0x7fac940b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac940d1910 0x7fac940b98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 -- 172.21.15.122:0/16244466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940cd7d0 con 0x7fac940d24b0 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac98c3f640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b6930 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c430 tx=0x7fac8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840052d0 con 0x7fac940d24b0 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84005490 con 0x7fac940d24b0 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac940b7160 con 0x7fac940d24b0 2024-03-06T07:26:07.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84007260 con 0x7fac940d24b0 2024-03-06T07:26:07.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.501+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac940ccfd0 con 0x7fac940d24b0 2024-03-06T07:26:07.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.502+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac940ccb20 con 0x7fac940d24b0 2024-03-06T07:26:07.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.503+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac84007400 con 0x7fac940d24b0 2024-03-06T07:26:07.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.503+0000 7fac89ffb640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fac7806b4d0 0x7fac7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.503+0000 7fac99440640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fac7806b4d0 0x7fac7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.503+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac840896f0 con 0x7fac940d24b0 2024-03-06T07:26:07.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.504+0000 7fac99440640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fac7806b4d0 0x7fac7806d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fac90009a30 tx=0x7fac90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.507+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac84058570 con 0x7fac940d24b0 2024-03-06T07:26:07.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.705+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fac94068ff0 con 0x7fac940d24b0 2024-03-06T07:26:07.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.705+0000 7fac89ffb640 1 -- 172.21.15.122:0/16244466 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fac84057e70 con 0x7fac940d24b0 2024-03-06T07:26:07.707 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:26:07.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fac7806b4d0 msgr2=0x7fac7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fac7806b4d0 0x7fac7806d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fac90009a30 tx=0x7fac90008040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 msgr2=0x7fac940b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b6930 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c430 tx=0x7fac8400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 shutdown_connections 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fac7806b4d0 0x7fac7806d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fac941614d0 0x7fac940ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac940d24b0 0x7fac940b6930 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 --2- 172.21.15.122:0/16244466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac940d1910 0x7fac940b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 >> 172.21.15.122:0/16244466 conn(0x7fac940d3d40 msgr2=0x7fac940d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 shutdown_connections 2024-03-06T07:26:07.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.708+0000 7fac9b6a4640 1 -- 172.21.15.122:0/16244466 wait complete. 2024-03-06T07:26:07.722 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:07.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.820+0000 7ff93ff5e640 1 Processor -- start 2024-03-06T07:26:07.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93ff5e640 1 -- start start 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93ff5e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff9380d2710 0x7ff9380b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93ff5e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380b3430 con 0x7ff9380d22b0 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93ff5e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380b3710 con 0x7ff93815dfe0 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93ff5e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9380b31e0 con 0x7ff9380d2710 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93d4f9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff9380d22b0 0x7ff9380b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47646/0 (socket says 172.21.15.122:47646) 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.821+0000 7ff93d4f9640 1 -- 172.21.15.122:0/4191305863 learned_addr learned my addr 172.21.15.122:0/4191305863 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93e4fb640 1 --2- 172.21.15.122:0/4191305863 >> v2:172.21.15.169:3300/0 conn(0x7ff9380d2710 0x7ff9380b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027189703 0 0) 0x7ff9380b3430 con 0x7ff9380d22b0 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff924003520 con 0x7ff9380d22b0 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2356427261 0 0) 0x7ff924003520 con 0x7ff9380d22b0 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 >> v2:172.21.15.169:3300/0 conn(0x7ff9380d2710 msgr2=0x7ff9380b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 --2- 172.21.15.122:0/4191305863 >> v2:172.21.15.169:3300/0 conn(0x7ff9380d2710 0x7ff9380b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 >> v1:172.21.15.169:6789/0 conn(0x7ff93815dfe0 legacy=0x7ff9380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9380b39f0 con 0x7ff9380d22b0 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff928001670 con 0x7ff9380d22b0 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 >> v1:172.21.15.122:6789/0 conn(0x7ff9380d22b0 legacy=0x7ff9380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9240039e0 0x7ff924003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff924004800 0x7ff924004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9240051c0 0x7ff92400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380b31e0 con 0x7ff9240051c0 2024-03-06T07:26:07.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92400c090 con 0x7ff9240039e0 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92400c290 con 0x7ff924004800 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93e4fb640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9240051c0 0x7ff92400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93dcfa640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9240039e0 0x7ff924003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.822+0000 7ff93d4f9640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff924004800 0x7ff924004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 -- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff924004800 msgr2=0x7ff924004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff924004800 0x7ff924004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 -- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9240039e0 msgr2=0x7ff924003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9240039e0 0x7ff924003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 -- 172.21.15.122:0/4191305863 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff924003520 con 0x7ff9240051c0 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93d4f9640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff924004800 0x7ff924004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 -- 172.21.15.122:0/4191305863 reap_dead start 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93e4fb640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9240051c0 0x7ff92400b9c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff930009fc0 tx=0x7ff924005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74450647ee2c6852 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930009c10 con 0x7ff9240051c0 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93002cca0 con 0x7ff9240051c0 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.823+0000 7ff93ccf8640 1 -- 172.21.15.122:0/4191305863 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930003bd0 con 0x7ff9240051c0 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 -- 172.21.15.122:0/4191305863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9240051c0 msgr2=0x7ff92400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9240051c0 0x7ff92400b9c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff930009fc0 tx=0x7ff924005830 comp rx=0 tx=0).stop 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 -- 172.21.15.122:0/4191305863 shutdown_connections 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/4191305863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9240051c0 0x7ff92400b9c0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 -- 172.21.15.122:0/4191305863 >> 172.21.15.122:0/4191305863 conn(0x7ff9380d3cd0 msgr2=0x7ff9380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 -- 172.21.15.122:0/4191305863 shutdown_connections 2024-03-06T07:26:07.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.824+0000 7ff93ff5e640 1 -- 172.21.15.122:0/4191305863 wait complete. 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.825+0000 7ff93ff5e640 1 Processor -- start 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.825+0000 7ff93ff5e640 1 -- start start 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93ff5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9240051c0 0x7ff93815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93ff5e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 0x7ff9380b78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93ff5e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff93809e6a0 0x7ff9380b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93ff5e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380d1c70 con 0x7ff93809e220 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93ff5e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380b3d60 con 0x7ff93809e6a0 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93ff5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380d1df0 con 0x7ff9240051c0 2024-03-06T07:26:07.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 0x7ff9380b78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 0x7ff9380b78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54940/0 (socket says 172.21.15.122:54940) 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 -- 172.21.15.122:0/3261675811 learned_addr learned my addr 172.21.15.122:0/3261675811 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93dcfa640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9240051c0 0x7ff93815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 -- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9240051c0 msgr2=0x7ff93815c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9240051c0 0x7ff93815c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 -- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff93809e6a0 msgr2=0x7ff9380b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff93809e6a0 0x7ff9380b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 -- 172.21.15.122:0/3261675811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93815c680 con 0x7ff93809e220 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.826+0000 7ff93d4f9640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 0x7ff9380b78a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff92800e780 tx=0x7ff92800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.827+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92800ee90 con 0x7ff93809e220 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.827+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff928018070 con 0x7ff93809e220 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.827+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92800ca70 con 0x7ff93809e220 2024-03-06T07:26:07.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.827+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9380b6ae0 con 0x7ff93809e220 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.827+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9380b70a0 con 0x7ff93809e220 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.828+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9380d4c10 con 0x7ff93809e220 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.828+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff92800cc10 con 0x7ff93809e220 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.829+0000 7ff92e7fc640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff91c06b6a0 0x7ff91c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.829+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff928089c50 con 0x7ff93809e220 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.829+0000 7ff93dcfa640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff91c06b6a0 0x7ff91c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.829+0000 7ff93dcfa640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff91c06b6a0 0x7ff91c06db60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff934000c00 tx=0x7ff934009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:07.833+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9380d4c10 con 0x7ff93809e220 2024-03-06T07:26:08.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.050+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff93804f330 con 0x7ff93809e220 2024-03-06T07:26:08.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.050+0000 7ff92e7fc640 1 -- 172.21.15.122:0/3261675811 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7ff92808f020 con 0x7ff93809e220 2024-03-06T07:26:08.053 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:08.053 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:26:08.053 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:26:08.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff91c06b6a0 msgr2=0x7ff91c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff91c06b6a0 0x7ff91c06db60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff934000c00 tx=0x7ff934009040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 msgr2=0x7ff9380b78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 0x7ff9380b78a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff92800e780 tx=0x7ff92800ec40 comp rx=0 tx=0).stop 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 shutdown_connections 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff91c06b6a0 0x7ff91c06db60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff93809e6a0 0x7ff9380b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff93809e220 0x7ff9380b78a0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 --2- 172.21.15.122:0/3261675811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9240051c0 0x7ff93815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 >> 172.21.15.122:0/3261675811 conn(0x7ff9380d3cd0 msgr2=0x7ff9380b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 shutdown_connections 2024-03-06T07:26:08.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.052+0000 7ff93ff5e640 1 -- 172.21.15.122:0/3261675811 wait complete. 2024-03-06T07:26:08.065 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:08.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.165+0000 7f595d35e640 1 Processor -- start 2024-03-06T07:26:08.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.165+0000 7f595d35e640 1 -- start start 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f595d35e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59581597b0 0x7f5958066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f595d35e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f595806c600 0x7f595815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f595d35e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59580b6060 con 0x7f595806b9c0 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f595d35e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595815c2a0 con 0x7f595806c600 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f595d35e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595815c560 con 0x7f59581597b0 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f595806c600 0x7f595815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f595806c600 0x7f595815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54952/0 (socket says 172.21.15.122:54952) 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 -- 172.21.15.122:0/3733836264 learned_addr learned my addr 172.21.15.122:0/3733836264 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f5956ffd640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59581597b0 0x7f5958066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 -- 172.21.15.122:0/3733836264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59581597b0 msgr2=0x7f5958066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59581597b0 0x7f5958066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 -- 172.21.15.122:0/3733836264 >> v1:172.21.15.169:6789/0 conn(0x7f595806b9c0 legacy=0x7f595806bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f59577fe640 1 -- 172.21.15.122:0/3733836264 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595815c840 con 0x7f595806c600 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.167+0000 7f5956ffd640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59581597b0 0x7f5958066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.168+0000 7f59577fe640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f595806c600 0x7f595815f890 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5950009e50 tx=0x7f595002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43910f23c2aed6aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.168+0000 7f5955ffb640 1 -- 172.21.15.122:0/3733836264 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595002ca70 con 0x7f595806c600 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.168+0000 7f5955ffb640 1 -- 172.21.15.122:0/3733836264 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595002cc10 con 0x7f595806c600 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.168+0000 7f5955ffb640 1 -- 172.21.15.122:0/3733836264 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59500343f0 con 0x7f595806c600 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 -- 172.21.15.122:0/3733836264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f595806c600 msgr2=0x7f595815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f595806c600 0x7f595815f890 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5950009e50 tx=0x7f595002bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 -- 172.21.15.122:0/3733836264 shutdown_connections 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f595806c600 0x7f595815f890 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 --2- 172.21.15.122:0/3733836264 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59581597b0 0x7f5958066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 -- 172.21.15.122:0/3733836264 >> 172.21.15.122:0/3733836264 conn(0x7f59580b4fe0 msgr2=0x7f59580b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 -- 172.21.15.122:0/3733836264 shutdown_connections 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.169+0000 7f595d35e640 1 -- 172.21.15.122:0/3733836264 wait complete. 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.170+0000 7f595d35e640 1 Processor -- start 2024-03-06T07:26:08.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.170+0000 7f595d35e640 1 -- start start 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f595d35e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f595806b9c0 0x7f595814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f595d35e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595806c600 0x7f5958154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f595d35e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 0x7f59580cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f595d35e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59580b1e10 con 0x7f59581597b0 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f595d35e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595815ce00 con 0x7f595806c600 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f595d35e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5958159b90 con 0x7f595806b9c0 2024-03-06T07:26:08.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 0x7f59580cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 0x7f59580cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54960/0 (socket says 172.21.15.122:54960) 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 -- 172.21.15.122:0/2611244077 learned_addr learned my addr 172.21.15.122:0/2611244077 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f5956ffd640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f595806b9c0 0x7f595814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59567fc640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595806c600 0x7f5958154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 -- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f595806b9c0 msgr2=0x7f595814fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f595806b9c0 0x7f595814fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 -- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595806c600 msgr2=0x7f5958154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595806c600 0x7f5958154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59577fe640 1 -- 172.21.15.122:0/2611244077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5958150da0 con 0x7f59581597b0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.171+0000 7f59567fc640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595806c600 0x7f5958154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.172+0000 7f59577fe640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 0x7f59580cb2a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f5950009e50 tx=0x7f5950004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.172+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950043020 con 0x7f59581597b0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.172+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595003e450 con 0x7f59581597b0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.172+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59580b1720 con 0x7f59581597b0 2024-03-06T07:26:08.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.172+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595003e820 con 0x7f59581597b0 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.172+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59581543b0 con 0x7f59581597b0 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.173+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59580ae1e0 con 0x7f59581597b0 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.173+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5950046020 con 0x7f59581597b0 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.174+0000 7f593f7fe640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f594406b500 0x7f594406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.174+0000 7f5956ffd640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f594406b500 0x7f594406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.174+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59500ac790 con 0x7f59581597b0 2024-03-06T07:26:08.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.174+0000 7f5956ffd640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f594406b500 0x7f594406d9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f594800ab60 tx=0x7f5948008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.178+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f595004d070 con 0x7f59581597b0 2024-03-06T07:26:08.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.356+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f595804f330 con 0x7f59581597b0 2024-03-06T07:26:08.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.357+0000 7f593f7fe640 1 -- 172.21.15.122:0/2611244077 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16660 (secure 0 0 0) 0x7f59580b6060 con 0x7f59581597b0 2024-03-06T07:26:08.359 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:08.360 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":82,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:52.668124+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:08.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f594406b500 msgr2=0x7f594406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f594406b500 0x7f594406d9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f594800ab60 tx=0x7f5948008040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 msgr2=0x7f59580cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 0x7f59580cb2a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f5950009e50 tx=0x7f5950004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 shutdown_connections 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f594406b500 0x7f594406d9c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59581597b0 0x7f59580cb2a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595806c600 0x7f5958154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 --2- 172.21.15.122:0/2611244077 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f595806b9c0 0x7f595814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 >> 172.21.15.122:0/2611244077 conn(0x7f59580b4fe0 msgr2=0x7f59580ae6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 shutdown_connections 2024-03-06T07:26:08.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.359+0000 7f595d35e640 1 -- 172.21.15.122:0/2611244077 wait complete. 2024-03-06T07:26:08.373 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:08.373 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:08.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.471+0000 7f9ee97dd640 1 Processor -- start 2024-03-06T07:26:08.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.471+0000 7f9ee97dd640 1 -- start start 2024-03-06T07:26:08.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee97dd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ee415dfe0 0x7f9ee40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee97dd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9ee40d29c0 0x7f9ee40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee97dd640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40b3430 con 0x7f9ee40cf830 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee97dd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40b31e0 con 0x7f9ee415dfe0 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee97dd640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40b3d60 con 0x7f9ee40d29c0 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ee415dfe0 0x7f9ee40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ee415dfe0 0x7f9ee40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54966/0 (socket says 172.21.15.122:54966) 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.472+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1656246304 learned_addr learned my addr 172.21.15.122:0/1656246304 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.473+0000 7f9ee37fe640 1 --2- 172.21.15.122:0/1656246304 >> v2:172.21.15.169:3300/0 conn(0x7f9ee40d29c0 0x7f9ee40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.473+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1656246304 >> v2:172.21.15.169:3300/0 conn(0x7f9ee40d29c0 msgr2=0x7f9ee40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.473+0000 7f9ee2ffd640 1 --2- 172.21.15.122:0/1656246304 >> v2:172.21.15.169:3300/0 conn(0x7f9ee40d29c0 0x7f9ee40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.473+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1656246304 >> v1:172.21.15.169:6789/0 conn(0x7f9ee40cf830 legacy=0x7f9ee40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.473+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1656246304 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee40b3710 con 0x7f9ee415dfe0 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.473+0000 7f9ee2ffd640 1 --2- 172.21.15.122:0/1656246304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40cf2f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40b3cd0 tx=0x7f9ed402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64222d8fef3348ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee27fc640 1 -- 172.21.15.122:0/1656246304 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4034070 con 0x7f9ee415dfe0 2024-03-06T07:26:08.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee27fc640 1 -- 172.21.15.122:0/1656246304 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed4030860 con 0x7f9ee415dfe0 2024-03-06T07:26:08.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee27fc640 1 -- 172.21.15.122:0/1656246304 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4030c30 con 0x7f9ee415dfe0 2024-03-06T07:26:08.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1656246304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee415dfe0 msgr2=0x7f9ee40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1656246304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40cf2f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40b3cd0 tx=0x7f9ed402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1656246304 shutdown_connections 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1656246304 >> v2:172.21.15.169:3300/0 conn(0x7f9ee40d29c0 0x7f9ee40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1656246304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40cf2f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1656246304 >> 172.21.15.122:0/1656246304 conn(0x7f9ee40d3cd0 msgr2=0x7f9ee40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1656246304 shutdown_connections 2024-03-06T07:26:08.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.474+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1656246304 wait complete. 2024-03-06T07:26:08.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.476+0000 7f9ee97dd640 1 Processor -- start 2024-03-06T07:26:08.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.476+0000 7f9ee97dd640 1 -- start start 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee97dd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 0x7f9ee40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee97dd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ee40d29c0 0x7f9ee40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee97dd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee97dd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40d1df0 con 0x7f9ee40cf830 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee97dd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40d1c70 con 0x7f9ee415dfe0 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee97dd640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee415e3c0 con 0x7f9ee40d29c0 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.477+0000 7f9ee2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 0x7f9ee40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 0x7f9ee40cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54972/0 (socket says 172.21.15.122:54972) 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1410987613 learned_addr learned my addr 172.21.15.122:0/1410987613 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:08.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9edbfff640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ee40d29c0 0x7f9ee40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee37fe640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ee40d29c0 msgr2=0x7f9ee40b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ee40d29c0 0x7f9ee40b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee415dfe0 msgr2=0x7f9ee40c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 -- 172.21.15.122:0/1410987613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee415fef0 con 0x7f9ee40cf830 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee2ffd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 0x7f9ee40cada0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f9ed402b9e0 tx=0x7f9ed40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4034040 con 0x7f9ee40cf830 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed4039d90 con 0x7f9ee40cf830 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee40cdcf0 con 0x7f9ee40cf830 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4043610 con 0x7f9ee40cf830 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.478+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee4160190 con 0x7f9ee40cf830 2024-03-06T07:26:08.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.479+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee404f330 con 0x7f9ee40cf830 2024-03-06T07:26:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.480+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ed404a060 con 0x7f9ee40cf830 2024-03-06T07:26:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.480+0000 7f9edb7fe640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ec406b4d0 0x7f9ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.480+0000 7f9edbfff640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ec406b4d0 0x7f9ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.480+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ed40ac2f0 con 0x7f9ee40cf830 2024-03-06T07:26:08.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.481+0000 7f9edbfff640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ec406b4d0 0x7f9ec406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40b5a30 tx=0x7f9ecc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.484+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ed403e070 con 0x7f9ee40cf830 2024-03-06T07:26:08.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.697+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ee406a1b0 con 0x7f9ee40cf830 2024-03-06T07:26:08.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.698+0000 7f9edb7fe640 1 -- 172.21.15.122:0/1410987613 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f9ed404be90 con 0x7f9ee40cf830 2024-03-06T07:26:08.700 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:08.700 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:26:08.700 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ec406b4d0 msgr2=0x7f9ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ec406b4d0 0x7f9ec406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40b5a30 tx=0x7f9ecc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 msgr2=0x7f9ee40cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 0x7f9ee40cada0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f9ed402b9e0 tx=0x7f9ed40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 shutdown_connections 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ec406b4d0 0x7f9ec406d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ee415dfe0 0x7f9ee40c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ee40d29c0 0x7f9ee40b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.699+0000 7f9ee97dd640 1 --2- 172.21.15.122:0/1410987613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ee40cf830 0x7f9ee40cada0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.700+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 >> 172.21.15.122:0/1410987613 conn(0x7f9ee40d3cd0 msgr2=0x7f9ee40cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.700+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 shutdown_connections 2024-03-06T07:26:08.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.700+0000 7f9ee97dd640 1 -- 172.21.15.122:0/1410987613 wait complete. 2024-03-06T07:26:08.713 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:08.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 Processor -- start 2024-03-06T07:26:08.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 -- start start 2024-03-06T07:26:08.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec440b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fec440d4760 0x7fec440d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440d25d0 con 0x7fec440b4dd0 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec440d4bf0 con 0x7fec440d4760 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.813+0000 7fec499ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec440ce9e0 con 0x7fec440b4170 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec3bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fec440d4760 0x7fec440d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec437fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fec440b4dd0 0x7fec440cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47654/0 (socket says 172.21.15.122:47654) 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec437fe640 1 -- 172.21.15.122:0/2666952359 learned_addr learned my addr 172.21.15.122:0/2666952359 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec42ffd640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec440b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec3bfff640 1 -- 172.21.15.122:0/2666952359 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 msgr2=0x7fec440b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec3bfff640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec440b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec3bfff640 1 -- 172.21.15.122:0/2666952359 >> v1:172.21.15.122:6789/0 conn(0x7fec440b4dd0 legacy=0x7fec440cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec3bfff640 1 -- 172.21.15.122:0/2666952359 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440d2870 con 0x7fec440d4760 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec427fc640 1 -- 172.21.15.122:0/2666952359 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31570872 0 0) 0x7fec440d25d0 con 0x7fec440b4dd0 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.814+0000 7fec42ffd640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec440b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec3bfff640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440d4760 0x7fec440d0370 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0097a0 tx=0x7fec2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f886d96b5840aca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec427fc640 1 -- 172.21.15.122:0/2666952359 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c035020 con 0x7fec440d4760 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec427fc640 1 -- 172.21.15.122:0/2666952359 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c031830 con 0x7fec440d4760 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec427fc640 1 -- 172.21.15.122:0/2666952359 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c031c00 con 0x7fec440d4760 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 -- 172.21.15.122:0/2666952359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440d4760 msgr2=0x7fec440d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440d4760 0x7fec440d0370 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0097a0 tx=0x7fec2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 -- 172.21.15.122:0/2666952359 shutdown_connections 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440d4760 0x7fec440d0370 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 --2- 172.21.15.122:0/2666952359 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec440b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 -- 172.21.15.122:0/2666952359 >> 172.21.15.122:0/2666952359 conn(0x7fec440d3c40 msgr2=0x7fec440d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 -- 172.21.15.122:0/2666952359 shutdown_connections 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.815+0000 7fec499ac640 1 -- 172.21.15.122:0/2666952359 wait complete. 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.816+0000 7fec499ac640 1 Processor -- start 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 -- start start 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec4414a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 0x7fec44147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec440ca100 0x7fec4415ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440b47d0 con 0x7fec440b4dd0 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440b4550 con 0x7fec440ca100 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec499ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec440b4950 con 0x7fec440b4170 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 0x7fec44147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec3bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 0x7fec44147c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54978/0 (socket says 172.21.15.122:54978) 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.817+0000 7fec3bfff640 1 -- 172.21.15.122:0/406605369 learned_addr learned my addr 172.21.15.122:0/406605369 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec42ffd640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec4414a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec3bfff640 1 -- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 msgr2=0x7fec4414a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec3bfff640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec4414a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec437fe640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec440ca100 0x7fec4415ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec3bfff640 1 -- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec440ca100 msgr2=0x7fec4415ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec3bfff640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec440ca100 0x7fec4415ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec3bfff640 1 -- 172.21.15.122:0/406605369 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440cbe00 con 0x7fec440b4dd0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec42ffd640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec4414a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec437fe640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec440ca100 0x7fec4415ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec3bfff640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 0x7fec44147c30 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fec2c034040 tx=0x7fec2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c035040 con 0x7fec440b4dd0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c004230 con 0x7fec440b4dd0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec440cc0e0 con 0x7fec440b4dd0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c03a700 con 0x7fec440b4dd0 2024-03-06T07:26:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.818+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec440cc6a0 con 0x7fec440b4dd0 2024-03-06T07:26:08.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.819+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4404f330 con 0x7fec440b4dd0 2024-03-06T07:26:08.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.819+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec2c038020 con 0x7fec440b4dd0 2024-03-06T07:26:08.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.820+0000 7fec40ff9640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2406b4d0 0x7fec2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.820+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2c0ac070 con 0x7fec440b4dd0 2024-03-06T07:26:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.820+0000 7fec42ffd640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2406b4d0 0x7fec2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.820+0000 7fec42ffd640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2406b4d0 0x7fec2406d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fec34006fd0 tx=0x7fec34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:08.828+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec2c07aef0 con 0x7fec440b4dd0 2024-03-06T07:26:09.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.043+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec440733b0 con 0x7fec440b4dd0 2024-03-06T07:26:09.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.044+0000 7fec40ff9640 1 -- 172.21.15.122:0/406605369 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fec2c07a7f0 con 0x7fec440b4dd0 2024-03-06T07:26:09.046 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:09.046 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:50.998325+0000","modified":"2024-03-06T07:25:53.113303+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6836/4292480577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":4292480577},{"type":"v1","addr":"172.21.15.122:6836","nonce":4292480577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":6}]} 2024-03-06T07:26:09.046 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 40 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2406b4d0 msgr2=0x7fec2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2406b4d0 0x7fec2406d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fec34006fd0 tx=0x7fec34008040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 msgr2=0x7fec44147c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 0x7fec44147c30 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fec2c034040 tx=0x7fec2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 shutdown_connections 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec2406b4d0 0x7fec2406d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec440ca100 0x7fec4415ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec440b4dd0 0x7fec44147c30 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 --2- 172.21.15.122:0/406605369 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec440b4170 0x7fec4414a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 >> 172.21.15.122:0/406605369 conn(0x7fec440d3c40 msgr2=0x7fec440b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 shutdown_connections 2024-03-06T07:26:09.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.046+0000 7fec499ac640 1 -- 172.21.15.122:0/406605369 wait complete. 2024-03-06T07:26:09.059 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:09.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.157+0000 7fda5d8c9640 1 Processor -- start 2024-03-06T07:26:09.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.157+0000 7fda5d8c9640 1 -- start start 2024-03-06T07:26:09.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda5d8c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fda58071ce0 0x7fda58072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda5d8c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda5806d400 0x7fda5806d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda5d8c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580adf70 con 0x7fda580cf790 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda5d8c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda58072680 con 0x7fda5806d400 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda5d8c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5806cf70 con 0x7fda58071ce0 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda5806d400 0x7fda5806d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda5806d400 0x7fda5806d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54988/0 (socket says 172.21.15.122:54988) 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 -- 172.21.15.122:0/2012493276 learned_addr learned my addr 172.21.15.122:0/2012493276 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda567fc640 1 --2- 172.21.15.122:0/2012493276 >> v2:172.21.15.169:3300/0 conn(0x7fda58071ce0 0x7fda58072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 -- 172.21.15.122:0/2012493276 >> v2:172.21.15.169:3300/0 conn(0x7fda58071ce0 msgr2=0x7fda58072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 --2- 172.21.15.122:0/2012493276 >> v2:172.21.15.169:3300/0 conn(0x7fda58071ce0 0x7fda58072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 -- 172.21.15.122:0/2012493276 >> v1:172.21.15.169:6789/0 conn(0x7fda580cf790 legacy=0x7fda580ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.158+0000 7fda577fe640 1 -- 172.21.15.122:0/2012493276 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580ae240 con 0x7fda5806d400 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.159+0000 7fda577fe640 1 --2- 172.21.15.122:0/2012493276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda5806d890 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fda580737d0 tx=0x7fda5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5283b2ef3edf631 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.159+0000 7fda55ffb640 1 -- 172.21.15.122:0/2012493276 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50035020 con 0x7fda5806d400 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.159+0000 7fda55ffb640 1 -- 172.21.15.122:0/2012493276 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda50031870 con 0x7fda5806d400 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.159+0000 7fda55ffb640 1 -- 172.21.15.122:0/2012493276 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50031c40 con 0x7fda5806d400 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 -- 172.21.15.122:0/2012493276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 msgr2=0x7fda5806d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/2012493276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda5806d890 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fda580737d0 tx=0x7fda5002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:09.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 -- 172.21.15.122:0/2012493276 shutdown_connections 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/2012493276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda5806d890 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/2012493276 >> v2:172.21.15.169:3300/0 conn(0x7fda58071ce0 0x7fda58072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 -- 172.21.15.122:0/2012493276 >> 172.21.15.122:0/2012493276 conn(0x7fda580b1a90 msgr2=0x7fda580b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 -- 172.21.15.122:0/2012493276 shutdown_connections 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.160+0000 7fda5d8c9640 1 -- 172.21.15.122:0/2012493276 wait complete. 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.161+0000 7fda5d8c9640 1 Processor -- start 2024-03-06T07:26:09.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.161+0000 7fda5d8c9640 1 -- start start 2024-03-06T07:26:09.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda580c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda58071ce0 0x7fda58159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580cf790 0x7fda580c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580aeae0 con 0x7fda5806d400 2024-03-06T07:26:09.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580ae800 con 0x7fda580cf790 2024-03-06T07:26:09.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5815e580 con 0x7fda58071ce0 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda580c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda580c5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55000/0 (socket says 172.21.15.122:55000) 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 -- 172.21.15.122:0/1441529998 learned_addr learned my addr 172.21.15.122:0/1441529998 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda567fc640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda58071ce0 0x7fda58159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda577fe640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580cf790 0x7fda580c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 -- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda58071ce0 msgr2=0x7fda58159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda58071ce0 0x7fda58159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 -- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580cf790 msgr2=0x7fda580c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580cf790 0x7fda580c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 -- 172.21.15.122:0/1441529998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5815a660 con 0x7fda5806d400 2024-03-06T07:26:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda577fe640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580cf790 0x7fda580c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda56ffd640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda580c5100 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fda4800ba80 tx=0x7fda4800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4800cc00 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4800cda0 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5815a940 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.162+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48003bb0 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.163+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5815a350 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.164+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5815b930 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.164+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda48014440 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.164+0000 7fda3ffff640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda4406b4d0 0x7fda4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.164+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda4808a150 con 0x7fda5806d400 2024-03-06T07:26:09.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.164+0000 7fda567fc640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda4406b4d0 0x7fda4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.165+0000 7fda567fc640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda4406b4d0 0x7fda4406d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fda5815b1f0 tx=0x7fda4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.169+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda48059060 con 0x7fda5806d400 2024-03-06T07:26:09.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.348+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda58072dd0 con 0x7fda5806d400 2024-03-06T07:26:09.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.349+0000 7fda3ffff640 1 -- 172.21.15.122:0/1441529998 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16660 (secure 0 0 0) 0x7fda48058960 con 0x7fda5806d400 2024-03-06T07:26:09.352 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:09.352 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":82,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:25:52.668124+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:49.219949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:49.969609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.351+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda4406b4d0 msgr2=0x7fda4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.351+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda4406b4d0 0x7fda4406d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fda5815b1f0 tx=0x7fda4c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.351+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 msgr2=0x7fda580c5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.351+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda580c5100 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fda4800ba80 tx=0x7fda4800bf40 comp rx=0 tx=0).stop 2024-03-06T07:26:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.351+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 shutdown_connections 2024-03-06T07:26:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.351+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda4406b4d0 0x7fda4406d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.352+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580cf790 0x7fda580c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.352+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda58071ce0 0x7fda58159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.352+0000 7fda5d8c9640 1 --2- 172.21.15.122:0/1441529998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5806d400 0x7fda580c5100 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.352+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 >> 172.21.15.122:0/1441529998 conn(0x7fda580b1a90 msgr2=0x7fda5815e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.352+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 shutdown_connections 2024-03-06T07:26:09.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.352+0000 7fda5d8c9640 1 -- 172.21.15.122:0/1441529998 wait complete. 2024-03-06T07:26:09.365 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:09.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.465+0000 7fc36e165640 1 Processor -- start 2024-03-06T07:26:09.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.465+0000 7fc36e165640 1 -- start start 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc36e165640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3680b4460 0x7fc3680d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc36e165640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc368162970 0x7fc3680b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc36e165640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc368161cf0 0x7fc368162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc36e165640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc368161b00 con 0x7fc368161cf0 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc36e165640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3680cecc0 con 0x7fc368162970 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc36e165640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680d22c0 con 0x7fc3680b4460 2024-03-06T07:26:09.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc367fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc368161cf0 0x7fc368162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc367fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc368161cf0 0x7fc368162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55002/0 (socket says 172.21.15.122:55002) 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.466+0000 7fc367fff640 1 -- 172.21.15.122:0/3160624035 learned_addr learned my addr 172.21.15.122:0/3160624035 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc3677fe640 1 --2- 172.21.15.122:0/3160624035 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3680b4460 0x7fc3680d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc366ffd640 1 --2- 172.21.15.122:0/3160624035 >> v2:172.21.15.169:3300/0 conn(0x7fc368162970 0x7fc3680b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc367fff640 1 -- 172.21.15.122:0/3160624035 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3680b4460 msgr2=0x7fc3680d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc367fff640 1 --2- 172.21.15.122:0/3160624035 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3680b4460 0x7fc3680d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc367fff640 1 -- 172.21.15.122:0/3160624035 >> v2:172.21.15.169:3300/0 conn(0x7fc368162970 msgr2=0x7fc3680b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc367fff640 1 --2- 172.21.15.122:0/3160624035 >> v2:172.21.15.169:3300/0 conn(0x7fc368162970 0x7fc3680b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc367fff640 1 -- 172.21.15.122:0/3160624035 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680d25a0 con 0x7fc368161cf0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc366ffd640 1 --2- 172.21.15.122:0/3160624035 >> v2:172.21.15.169:3300/0 conn(0x7fc368162970 0x7fc3680b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc367fff640 1 --2- 172.21.15.122:0/3160624035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc368161cf0 0x7fc368162180 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc3680b6840 tx=0x7fc36002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bfa82c887827ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.467+0000 7fc3667fc640 1 -- 172.21.15.122:0/3160624035 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36002ca70 con 0x7fc368161cf0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc3667fc640 1 -- 172.21.15.122:0/3160624035 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36002cc10 con 0x7fc368161cf0 2024-03-06T07:26:09.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc3667fc640 1 -- 172.21.15.122:0/3160624035 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600343f0 con 0x7fc368161cf0 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 -- 172.21.15.122:0/3160624035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc368161cf0 msgr2=0x7fc368162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 --2- 172.21.15.122:0/3160624035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc368161cf0 0x7fc368162180 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc3680b6840 tx=0x7fc36002bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 -- 172.21.15.122:0/3160624035 shutdown_connections 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 --2- 172.21.15.122:0/3160624035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc368161cf0 0x7fc368162180 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 --2- 172.21.15.122:0/3160624035 >> v2:172.21.15.169:3300/0 conn(0x7fc368162970 0x7fc3680b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 --2- 172.21.15.122:0/3160624035 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3680b4460 0x7fc3680d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 -- 172.21.15.122:0/3160624035 >> 172.21.15.122:0/3160624035 conn(0x7fc3680d3ce0 msgr2=0x7fc3680d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 -- 172.21.15.122:0/3160624035 shutdown_connections 2024-03-06T07:26:09.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.468+0000 7fc36e165640 1 -- 172.21.15.122:0/3160624035 wait complete. 2024-03-06T07:26:09.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.469+0000 7fc36e165640 1 Processor -- start 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.469+0000 7fc36e165640 1 -- start start 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc36e165640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 0x7fc3680cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc36e165640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc368161cf0 0x7fc3681f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc36e165640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc368162970 0x7fc3681f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc36e165640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680ce950 con 0x7fc3680b4460 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc36e165640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680d2b60 con 0x7fc368161cf0 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc36e165640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680b3ac0 con 0x7fc368162970 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc3677fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 0x7fc3680cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc3677fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 0x7fc3680cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55018/0 (socket says 172.21.15.122:55018) 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc3677fe640 1 -- 172.21.15.122:0/2178200655 learned_addr learned my addr 172.21.15.122:0/2178200655 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc366ffd640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc368161cf0 0x7fc3681f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.470+0000 7fc367fff640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc368162970 0x7fc3681f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc3677fe640 1 -- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc368162970 msgr2=0x7fc3681f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc3677fe640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc368162970 0x7fc3681f7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc3677fe640 1 -- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc368161cf0 msgr2=0x7fc3681f6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc3677fe640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc368161cf0 0x7fc3681f6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc3677fe640 1 -- 172.21.15.122:0/2178200655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680cbbe0 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc3677fe640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 0x7fc3680cca80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc35800d620 tx=0x7fc35800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800b620 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc358019070 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3680cbec0 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800ba00 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.471+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3680cc480 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.472+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36804f330 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.472+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc35800bba0 con 0x7fc3680b4460 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.473+0000 7fc34ffff640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc35406b4d0 0x7fc35406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.473+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc35808a4c0 con 0x7fc3680b4460 2024-03-06T07:26:09.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.473+0000 7fc366ffd640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc35406b4d0 0x7fc35406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.473+0000 7fc366ffd640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc35406b4d0 0x7fc35406d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc368162db0 tx=0x7fc35c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.477+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc358058cf0 con 0x7fc3680b4460 2024-03-06T07:26:09.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.693+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc36806b180 con 0x7fc3680b4460 2024-03-06T07:26:09.695 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:26:09.693+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.694+0000 7fc34ffff640 1 -- 172.21.15.122:0/2178200655 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc35805cd90 con 0x7fc3680b4460 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:09.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc35406b4d0 msgr2=0x7fc35406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc35406b4d0 0x7fc35406d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc368162db0 tx=0x7fc35c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 msgr2=0x7fc3680cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 0x7fc3680cca80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc35800d620 tx=0x7fc35800dae0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 shutdown_connections 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc35406b4d0 0x7fc35406d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc368162970 0x7fc3681f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc368161cf0 0x7fc3681f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 --2- 172.21.15.122:0/2178200655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3680b4460 0x7fc3680cca80 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 >> 172.21.15.122:0/2178200655 conn(0x7fc3680d3ce0 msgr2=0x7fc368160bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 shutdown_connections 2024-03-06T07:26:09.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.697+0000 7fc36e165640 1 -- 172.21.15.122:0/2178200655 wait complete. 2024-03-06T07:26:09.701 INFO:tasks.ceph.mds.c.smithi122.stderr: -2791> 2024-03-06T07:25:56.938+0000 7f816cbea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:09.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:09.729 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:26:09.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.812+0000 7fa177adf640 1 Processor -- start 2024-03-06T07:26:09.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa177adf640 1 -- start start 2024-03-06T07:26:09.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa177adf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17015c050 0x7fa17006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa177adf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa17006fb40 0x7fa1700cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa177adf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700cd4b0 con 0x7fa1700ccca0 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa177adf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17006ff80 con 0x7fa17006fb40 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa177adf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700cd790 con 0x7fa17015c050 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa17507a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa17006fb40 0x7fa1700cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa17507a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa17006fb40 0x7fa1700cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55034/0 (socket says 172.21.15.122:55034) 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.813+0000 7fa17507a640 1 -- 172.21.15.122:0/112931607 learned_addr learned my addr 172.21.15.122:0/112931607 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa17587b640 1 --2- 172.21.15.122:0/112931607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17015c050 0x7fa17006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa17507a640 1 -- 172.21.15.122:0/112931607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17015c050 msgr2=0x7fa17006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa17507a640 1 --2- 172.21.15.122:0/112931607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17015c050 0x7fa17006f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa17507a640 1 -- 172.21.15.122:0/112931607 >> v1:172.21.15.169:6789/0 conn(0x7fa1700ccca0 legacy=0x7fa1700cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa17507a640 1 -- 172.21.15.122:0/112931607 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1700cda70 con 0x7fa17006fb40 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa17507a640 1 --2- 172.21.15.122:0/112931607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17006fb40 0x7fa1700cc760 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa160009870 tx=0x7fa16002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606a81c9df354be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.814+0000 7fa174879640 1 -- 172.21.15.122:0/112931607 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160035020 con 0x7fa17006fb40 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa174879640 1 -- 172.21.15.122:0/112931607 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160031640 con 0x7fa17006fb40 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa174879640 1 -- 172.21.15.122:0/112931607 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16002d040 con 0x7fa17006fb40 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 -- 172.21.15.122:0/112931607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17006fb40 msgr2=0x7fa1700cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 --2- 172.21.15.122:0/112931607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17006fb40 0x7fa1700cc760 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa160009870 tx=0x7fa16002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 -- 172.21.15.122:0/112931607 shutdown_connections 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 --2- 172.21.15.122:0/112931607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17006fb40 0x7fa1700cc760 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 --2- 172.21.15.122:0/112931607 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17015c050 0x7fa17006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 -- 172.21.15.122:0/112931607 >> 172.21.15.122:0/112931607 conn(0x7fa1700b3800 msgr2=0x7fa1700b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 -- 172.21.15.122:0/112931607 shutdown_connections 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.815+0000 7fa177adf640 1 -- 172.21.15.122:0/112931607 wait complete. 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.816+0000 7fa177adf640 1 Processor -- start 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.816+0000 7fa177adf640 1 -- start start 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa177adf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17006fb40 0x7fa170145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa177adf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1700ccca0 0x7fa170141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa177adf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 0x7fa170155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa177adf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700d01f0 con 0x7fa17015c050 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa177adf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700cd130 con 0x7fa1700ccca0 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa177adf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700d0370 con 0x7fa17006fb40 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 0x7fa170155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 0x7fa170155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55046/0 (socket says 172.21.15.122:55046) 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 -- 172.21.15.122:0/2603549730 learned_addr learned my addr 172.21.15.122:0/2603549730 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17587b640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17006fb40 0x7fa170145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17507a640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1700ccca0 0x7fa170141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 -- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17006fb40 msgr2=0x7fa170145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17006fb40 0x7fa170145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 -- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1700ccca0 msgr2=0x7fa170141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1700ccca0 0x7fa170141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17607c640 1 -- 172.21.15.122:0/2603549730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1701421c0 con 0x7fa17015c050 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17587b640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17006fb40 0x7fa170145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.817+0000 7fa17507a640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1700ccca0 0x7fa170141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.818+0000 7fa17607c640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 0x7fa170155ca0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c7a0 tx=0x7fa16800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.818+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168003b70 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.818+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168003d30 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.818+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1701585a0 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.818+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800aa90 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.818+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa170158b60 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.819+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17009fe60 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.819+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa16800ac30 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.819+0000 7fa1667fc640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa15406b4d0 0x7fa15406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.819+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa16808a1a0 con 0x7fa17015c050 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.819+0000 7fa17587b640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa15406b4d0 0x7fa15406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.820+0000 7fa17587b640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa15406b4d0 0x7fa15406d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa16c006fd0 tx=0x7fa16c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:09.824+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa168058fc0 con 0x7fa17015c050 2024-03-06T07:26:10.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.033+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa170068ff0 con 0x7fa17015c050 2024-03-06T07:26:10.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.741+0000 7fa1667fc640 1 -- 172.21.15.122:0/2603549730 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa1680588c0 con 0x7fa17015c050 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa15406b4d0 msgr2=0x7fa15406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa15406b4d0 0x7fa15406d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa16c006fd0 tx=0x7fa16c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 msgr2=0x7fa170155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 0x7fa170155ca0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c7a0 tx=0x7fa16800cc60 comp rx=0 tx=0).stop 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 shutdown_connections 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa15406b4d0 0x7fa15406d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa17015c050 0x7fa170155ca0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1700ccca0 0x7fa170141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 --2- 172.21.15.122:0/2603549730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa17006fb40 0x7fa170145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.743+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 >> 172.21.15.122:0/2603549730 conn(0x7fa1700b3800 msgr2=0x7fa1700d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.744+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 shutdown_connections 2024-03-06T07:26:10.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.744+0000 7fa177adf640 1 -- 172.21.15.122:0/2603549730 wait complete. 2024-03-06T07:26:10.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:10.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.906+0000 7f883ba9a640 1 Processor -- start 2024-03-06T07:26:10.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.906+0000 7f883ba9a640 1 -- start start 2024-03-06T07:26:10.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883ba9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8834070d10 0x7f88340ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883ba9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88340711c0 con 0x7f883415ac00 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883ba9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883415afe0 con 0x7f88340708b0 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883ba9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88340cdc10 con 0x7f8834070d10 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883a037640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8834070d10 0x7f88340ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883a037640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8834070d10 0x7f88340ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55058/0 (socket says 172.21.15.122:55058) 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f8839836640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f883415ac00 0x7f883406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47682/0 (socket says 172.21.15.122:47682) 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883a037640 1 -- 172.21.15.122:0/1721527567 learned_addr learned my addr 172.21.15.122:0/1721527567 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883a037640 1 -- 172.21.15.122:0/1721527567 >> v1:172.21.15.169:6789/0 conn(0x7f88340708b0 legacy=0x7f88340ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883a037640 1 -- 172.21.15.122:0/1721527567 >> v1:172.21.15.122:6789/0 conn(0x7f883415ac00 legacy=0x7f883406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f8838834640 1 -- 172.21.15.122:0/1721527567 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17436101 0 0) 0x7f88340711c0 con 0x7f883415ac00 2024-03-06T07:26:10.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.907+0000 7f883a037640 1 -- 172.21.15.122:0/1721527567 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340cd000 con 0x7f8834070d10 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.908+0000 7f883a037640 1 --2- 172.21.15.122:0/1721527567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8834070d10 0x7f88340ce7b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f882c009e50 tx=0x7f882c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efe4a22157176eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.908+0000 7f8838834640 1 -- 172.21.15.122:0/1721527567 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c02ca80 con 0x7f8834070d10 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.908+0000 7f8838834640 1 -- 172.21.15.122:0/1721527567 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c02cc20 con 0x7f8834070d10 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.908+0000 7f8838834640 1 -- 172.21.15.122:0/1721527567 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c0343f0 con 0x7f8834070d10 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 -- 172.21.15.122:0/1721527567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8834070d10 msgr2=0x7f88340ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 --2- 172.21.15.122:0/1721527567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8834070d10 0x7f88340ce7b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f882c009e50 tx=0x7f882c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 -- 172.21.15.122:0/1721527567 shutdown_connections 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 --2- 172.21.15.122:0/1721527567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8834070d10 0x7f88340ce7b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 -- 172.21.15.122:0/1721527567 >> 172.21.15.122:0/1721527567 conn(0x7f88340b1650 msgr2=0x7f88340b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 -- 172.21.15.122:0/1721527567 shutdown_connections 2024-03-06T07:26:10.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.909+0000 7f883ba9a640 1 -- 172.21.15.122:0/1721527567 wait complete. 2024-03-06T07:26:10.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.910+0000 7f883ba9a640 1 Processor -- start 2024-03-06T07:26:10.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.910+0000 7f883ba9a640 1 -- start start 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88340708b0 0x7f8834155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8834070d10 0x7f88340a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 0x7f88340a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88340cd8a0 con 0x7f883415ac00 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88340cd5c0 con 0x7f8834070d10 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88340cf5d0 con 0x7f88340708b0 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 0x7f88340a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 0x7f88340a9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55066/0 (socket says 172.21.15.122:55066) 2024-03-06T07:26:10.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 -- 172.21.15.122:0/2903810525 learned_addr learned my addr 172.21.15.122:0/2903810525 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 -- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88340708b0 msgr2=0x7f8834155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f8839836640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88340708b0 0x7f8834155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88340708b0 0x7f8834155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 -- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8834070d10 msgr2=0x7f88340a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8834070d10 0x7f88340a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 -- 172.21.15.122:0/2903810525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340aa010 con 0x7f883415ac00 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883a037640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 0x7f88340a9990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f882c036040 tx=0x7f882c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c043020 con 0x7f883415ac00 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c03e450 con 0x7f883415ac00 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88340aa2f0 con 0x7f883415ac00 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c009510 con 0x7f883415ac00 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.911+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88340aa8b0 con 0x7f883415ac00 2024-03-06T07:26:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.912+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88340ad800 con 0x7f883415ac00 2024-03-06T07:26:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.913+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f882c046020 con 0x7f883415ac00 2024-03-06T07:26:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.913+0000 7f882a7fc640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f881806b4d0 0x7f881806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.913+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f882c0acc60 con 0x7f883415ac00 2024-03-06T07:26:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.913+0000 7f8839836640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f881806b4d0 0x7f881806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.914+0000 7f8839836640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f881806b4d0 0x7f881806d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8830000f60 tx=0x7f8830009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:10.917+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f882c04d030 con 0x7f883415ac00 2024-03-06T07:26:11.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.088+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f88340687d0 con 0x7f883415ac00 2024-03-06T07:26:11.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.747+0000 7f882a7fc640 1 -- 172.21.15.122:0/2903810525 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f882c07b9f0 con 0x7f883415ac00 2024-03-06T07:26:11.749 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:11.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f881806b4d0 msgr2=0x7f881806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f881806b4d0 0x7f881806d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8830000f60 tx=0x7f8830009040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 msgr2=0x7f88340a9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 0x7f88340a9990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f882c036040 tx=0x7f882c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 shutdown_connections 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f881806b4d0 0x7f881806d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f883415ac00 0x7f88340a9990 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8834070d10 0x7f88340a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 --2- 172.21.15.122:0/2903810525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88340708b0 0x7f8834155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 >> 172.21.15.122:0/2903810525 conn(0x7f88340b1650 msgr2=0x7f883415ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 shutdown_connections 2024-03-06T07:26:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.751+0000 7f883ba9a640 1 -- 172.21.15.122:0/2903810525 wait complete. 2024-03-06T07:26:11.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:11.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.870+0000 7f77a2dc9640 1 Processor -- start 2024-03-06T07:26:11.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.870+0000 7f77a2dc9640 1 -- start start 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.870+0000 7f77a2dc9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f779c15ac00 0x7f779c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.870+0000 7f77a2dc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f779c15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.870+0000 7f77a2dc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c15afe0 con 0x7f779c0708b0 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f77a2dc9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779c0cdc10 con 0x7f779c15ac00 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f77a2dc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c0cd000 con 0x7f779c070d10 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f779c0708b0 0x7f779c15e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47696/0 (socket says 172.21.15.122:47696) 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779bfff640 1 -- 172.21.15.122:0/6283317 learned_addr learned my addr 172.21.15.122:0/6283317 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f77a1366640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f779c15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f77a0b65640 1 --2- 172.21.15.122:0/6283317 >> v2:172.21.15.169:3300/0 conn(0x7f779c15ac00 0x7f779c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240924435 0 0) 0x7f779c15afe0 con 0x7f779c0708b0 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c003520 con 0x7f779c0708b0 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3901800824 0 0) 0x7f778c003520 con 0x7f779c0708b0 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 msgr2=0x7f779c15eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f779c15eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 >> v2:172.21.15.169:3300/0 conn(0x7f779c15ac00 msgr2=0x7f779c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 --2- 172.21.15.122:0/6283317 >> v2:172.21.15.169:3300/0 conn(0x7f779c15ac00 0x7f779c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0cd2e0 con 0x7f779c0708b0 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7788001670 con 0x7f779c0708b0 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 >> v1:172.21.15.122:6789/0 conn(0x7f779c0708b0 legacy=0x7f779c15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c0039b0 0x7f778c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f778c004800 0x7f778c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f778c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c0cdc10 con 0x7f778c004800 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c0cd000 con 0x7f778c0039b0 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c00b9f0 con 0x7f779c070d10 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f779bfff640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f778c004800 0x7f778c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f77a0b65640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c0039b0 0x7f778c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.871+0000 7f77a1366640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f778c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779bfff640 1 -- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 msgr2=0x7f778c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779bfff640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f778c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779bfff640 1 -- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c0039b0 msgr2=0x7f778c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779bfff640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c0039b0 0x7f778c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779bfff640 1 -- 172.21.15.122:0/6283317 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c003520 con 0x7f778c004800 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f77a1366640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c070d10 0x7f778c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779bfff640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f778c004800 0x7f778c004c80 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f778800c210 tx=0x7f778c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bb4d2a42a162a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788007d30 con 0x7f778c004800 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7788030070 con 0x7f778c004800 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.872+0000 7f779b7fe640 1 -- 172.21.15.122:0/6283317 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788005d30 con 0x7f778c004800 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a2dc9640 1 -- 172.21.15.122:0/6283317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f778c004800 msgr2=0x7f778c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/6283317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f778c004800 0x7f778c004c80 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f778800c210 tx=0x7f778c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a1366640 1 -- 172.21.15.122:0/6283317 reap_dead start 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a2dc9640 1 -- 172.21.15.122:0/6283317 shutdown_connections 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a2dc9640 1 -- 172.21.15.122:0/6283317 >> 172.21.15.122:0/6283317 conn(0x7f779c0b1650 msgr2=0x7f779c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a2dc9640 1 -- 172.21.15.122:0/6283317 shutdown_connections 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.873+0000 7f77a2dc9640 1 -- 172.21.15.122:0/6283317 wait complete. 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.874+0000 7f77a2dc9640 1 Processor -- start 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.874+0000 7f77a2dc9640 1 -- start start 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a2dc9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c004800 0x7f779c148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a2dc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c14e980 0x7f779c13c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a2dc9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 0x7f779c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a2dc9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c071190 con 0x7f779c15d980 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a2dc9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c0cd8a0 con 0x7f778c004800 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a2dc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c06ce50 con 0x7f779c14e980 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 0x7f779c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 0x7f779c06d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55082/0 (socket says 172.21.15.122:55082) 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 -- 172.21.15.122:0/4173353892 learned_addr learned my addr 172.21.15.122:0/4173353892 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 -- 172.21.15.122:0/4173353892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c14e980 msgr2=0x7f779c13c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c14e980 0x7f779c13c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 -- 172.21.15.122:0/4173353892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c004800 msgr2=0x7f779c148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c004800 0x7f779c148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.875+0000 7f77a1366640 1 -- 172.21.15.122:0/4173353892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c15c8e0 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.876+0000 7f77a1366640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 0x7f779c06d1a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f779c071600 tx=0x7f7794010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.876+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794010da0 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.876+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794003990 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.876+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c15cbc0 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.876+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794003dc0 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.876+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c15d180 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.877+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c0687d0 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.877+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7794005d10 con 0x7f779c15d980 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.877+0000 7f77997fa640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f778006b4d0 0x7f778006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.878+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f779408a6f0 con 0x7f779c15d980 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.878+0000 7f77a0b65640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f778006b4d0 0x7f778006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.878+0000 7f77a0b65640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f778006b4d0 0x7f778006d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7790006b10 tx=0x7f7790006aa0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:11.882+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7794025020 con 0x7f779c15d980 2024-03-06T07:26:12.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.055+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f779c04f330 con 0x7f779c15d980 2024-03-06T07:26:12.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.749+0000 7f77997fa640 1 -- 172.21.15.122:0/4173353892 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f77940596d0 con 0x7f779c15d980 2024-03-06T07:26:12.751 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:12.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.756+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f778006b4d0 msgr2=0x7f778006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.756+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f778006b4d0 0x7f778006d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7790006b10 tx=0x7f7790006aa0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.756+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 msgr2=0x7f779c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.756+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 0x7f779c06d1a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f779c071600 tx=0x7f7794010590 comp rx=0 tx=0).stop 2024-03-06T07:26:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.757+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 shutdown_connections 2024-03-06T07:26:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.757+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f778006b4d0 0x7f778006d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.757+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f779c15d980 0x7f779c06d1a0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.757+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f779c14e980 0x7f779c13c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.757+0000 7f77a2dc9640 1 --2- 172.21.15.122:0/4173353892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f778c004800 0x7f779c148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.757+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 >> 172.21.15.122:0/4173353892 conn(0x7f779c0b1650 msgr2=0x7f779c14b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.758+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 shutdown_connections 2024-03-06T07:26:12.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.758+0000 7f77a2dc9640 1 -- 172.21.15.122:0/4173353892 wait complete. 2024-03-06T07:26:12.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.881+0000 7f469470f640 1 Processor -- start 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.881+0000 7f469470f640 1 -- start start 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f469470f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f468c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f469470f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f468c0d01a0 0x7f468c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f469470f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c158ac0 con 0x7f468c0aef80 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f469470f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468c15b2a0 con 0x7f468c0d01a0 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f469470f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c15b540 con 0x7f468c1586e0 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f4691caa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f468c0aef80 0x7f468c0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47698/0 (socket says 172.21.15.122:47698) 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f4691caa640 1 -- 172.21.15.122:0/1239688186 learned_addr learned my addr 172.21.15.122:0/1239688186 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46885627 0 0) 0x7f468c158ac0 con 0x7f468c0aef80 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4678003520 con 0x7f468c0aef80 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f4692cac640 1 --2- 172.21.15.122:0/1239688186 >> v2:172.21.15.169:3300/0 conn(0x7f468c0d01a0 0x7f468c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46924ab640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f468c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1036640113 0 0) 0x7f4678003520 con 0x7f468c0aef80 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 msgr2=0x7f468c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f468c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 >> v2:172.21.15.169:3300/0 conn(0x7f468c0d01a0 msgr2=0x7f468c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 --2- 172.21.15.122:0/1239688186 >> v2:172.21.15.169:3300/0 conn(0x7f468c0d01a0 0x7f468c0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c15b820 con 0x7f468c0aef80 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46924ab640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f468c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f4692cac640 1 --2- 172.21.15.122:0/1239688186 >> v2:172.21.15.169:3300/0 conn(0x7f468c0d01a0 0x7f468c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.882+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f467c001670 con 0x7f468c0aef80 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 >> v1:172.21.15.122:6789/0 conn(0x7f468c0aef80 legacy=0x7f468c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46780039b0 0x7f4678003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4678004800 0x7f4678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f46780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c15b2a0 con 0x7f46780039b0 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c15b540 con 0x7f4678004800 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467800ba40 con 0x7f468c1586e0 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46924ab640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46780039b0 0x7f4678003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f4691caa640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4678004800 0x7f4678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f4692cac640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f46780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46924ab640 1 -- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 msgr2=0x7f46780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46924ab640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1586e0 0x7f46780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46924ab640 1 -- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4678004800 msgr2=0x7f4678004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46924ab640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4678004800 0x7f4678004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.883+0000 7f46924ab640 1 -- 172.21.15.122:0/1239688186 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678003520 con 0x7f46780039b0 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f46924ab640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46780039b0 0x7f4678003dd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f46880098a0 tx=0x7f4678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591bc2a9f6a4a524 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468800f040 con 0x7f46780039b0 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4688004230 con 0x7f46780039b0 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f46914a9640 1 -- 172.21.15.122:0/1239688186 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688003610 con 0x7f46780039b0 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f469470f640 1 -- 172.21.15.122:0/1239688186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46780039b0 msgr2=0x7f4678003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f469470f640 1 --2- 172.21.15.122:0/1239688186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46780039b0 0x7f4678003dd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f46880098a0 tx=0x7f4678005830 comp rx=0 tx=0).stop 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f4692cac640 1 -- 172.21.15.122:0/1239688186 reap_dead start 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f469470f640 1 -- 172.21.15.122:0/1239688186 shutdown_connections 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f469470f640 1 -- 172.21.15.122:0/1239688186 >> 172.21.15.122:0/1239688186 conn(0x7f468c06b190 msgr2=0x7f468c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f469470f640 1 -- 172.21.15.122:0/1239688186 shutdown_connections 2024-03-06T07:26:12.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.884+0000 7f469470f640 1 -- 172.21.15.122:0/1239688186 wait complete. 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 Processor -- start 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 -- start start 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46780039b0 0x7f468c145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1f3d10 0x7f468c1f4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 0x7f468c159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c071350 con 0x7f468c1595a0 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c15bde0 con 0x7f46780039b0 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f469470f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0714d0 con 0x7f468c1f3d10 2024-03-06T07:26:12.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f4692cac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 0x7f468c159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f4692cac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 0x7f468c159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55090/0 (socket says 172.21.15.122:55090) 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.886+0000 7f4692cac640 1 -- 172.21.15.122:0/3490588151 learned_addr learned my addr 172.21.15.122:0/3490588151 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4691caa640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1f3d10 0x7f468c1f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f46924ab640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46780039b0 0x7f468c145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4692cac640 1 -- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1f3d10 msgr2=0x7f468c1f4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4692cac640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1f3d10 0x7f468c1f4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4692cac640 1 -- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46780039b0 msgr2=0x7f468c145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4692cac640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46780039b0 0x7f468c145300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4692cac640 1 -- 172.21.15.122:0/3490588151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c1f0710 con 0x7f468c1595a0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4692cac640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 0x7f468c159a30 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f468c06bf00 tx=0x7f4684010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684010e10 con 0x7f468c1595a0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4684003990 con 0x7f468c1595a0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c1f09f0 con 0x7f468c1595a0 2024-03-06T07:26:12.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684003b80 con 0x7f468c1595a0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.887+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c1f0fb0 con 0x7f468c1595a0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.888+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c0732b0 con 0x7f468c1595a0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.889+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f468401e030 con 0x7f468c1595a0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.889+0000 7f4682ffd640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f467006b4d0 0x7f467006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.889+0000 7f46924ab640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f467006b4d0 0x7f467006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.889+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f468408a580 con 0x7f468c1595a0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.889+0000 7f46924ab640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f467006b4d0 0x7f467006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f46880098a0 tx=0x7f468803a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:12.893+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4684025020 con 0x7f468c1595a0 2024-03-06T07:26:13.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.065+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f468c068ff0 con 0x7f468c1595a0 2024-03-06T07:26:13.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.068+0000 7f4682ffd640 1 -- 172.21.15.122:0/3490588151 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4684059790 con 0x7f468c1595a0 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f467006b4d0 msgr2=0x7f467006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f467006b4d0 0x7f467006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f46880098a0 tx=0x7f468803a040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 msgr2=0x7f468c159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 0x7f468c159a30 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f468c06bf00 tx=0x7f4684010530 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 shutdown_connections 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f467006b4d0 0x7f467006d990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468c1595a0 0x7f468c159a30 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f468c1f3d10 0x7f468c1f4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 --2- 172.21.15.122:0/3490588151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46780039b0 0x7f468c145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 >> 172.21.15.122:0/3490588151 conn(0x7f468c06b190 msgr2=0x7f468c0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 shutdown_connections 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.070+0000 7f469470f640 1 -- 172.21.15.122:0/3490588151 wait complete. 2024-03-06T07:26:13.083 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:26:13.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.181+0000 7f09fb381640 1 Processor -- start 2024-03-06T07:26:13.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.181+0000 7f09fb381640 1 -- start start 2024-03-06T07:26:13.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09fb381640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09f40b3060 0x7f09f415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09fb381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f415bf30 0x7f09f415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09fb381640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f415f2d0 con 0x7f09f413dc10 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09fb381640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f40cf3f0 con 0x7f09f40b3060 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09fb381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f415b210 con 0x7f09f415bf30 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09f40b3060 0x7f09f415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09f40b3060 0x7f09f415e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55092/0 (socket says 172.21.15.122:55092) 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 -- 172.21.15.122:0/3416935261 learned_addr learned my addr 172.21.15.122:0/3416935261 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f991e640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f415bf30 0x7f09f415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 -- 172.21.15.122:0/3416935261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f415bf30 msgr2=0x7f09f415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f415bf30 0x7f09f415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 -- 172.21.15.122:0/3416935261 >> v1:172.21.15.169:6789/0 conn(0x7f09f413dc10 legacy=0x7f09f40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f891c640 1 -- 172.21.15.122:0/3416935261 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f415f580 con 0x7f09f40b3060 2024-03-06T07:26:13.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.182+0000 7f09f991e640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f415bf30 0x7f09f415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.183+0000 7f09f891c640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f40b3060 0x7f09f415e850 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f09e400bb10 tx=0x7f09e402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ee39d14c76939 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.183+0000 7f09ebfff640 1 -- 172.21.15.122:0/3416935261 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e403b070 con 0x7f09f40b3060 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.183+0000 7f09ebfff640 1 -- 172.21.15.122:0/3416935261 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e4036440 con 0x7f09f40b3060 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.183+0000 7f09ebfff640 1 -- 172.21.15.122:0/3416935261 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4036810 con 0x7f09f40b3060 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 -- 172.21.15.122:0/3416935261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f40b3060 msgr2=0x7f09f415e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f40b3060 0x7f09f415e850 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f09e400bb10 tx=0x7f09e402e0c0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 -- 172.21.15.122:0/3416935261 shutdown_connections 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f415bf30 0x7f09f415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 --2- 172.21.15.122:0/3416935261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f40b3060 0x7f09f415e850 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 -- 172.21.15.122:0/3416935261 >> 172.21.15.122:0/3416935261 conn(0x7f09f40cf8a0 msgr2=0x7f09f40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 -- 172.21.15.122:0/3416935261 shutdown_connections 2024-03-06T07:26:13.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.184+0000 7f09fb381640 1 -- 172.21.15.122:0/3416935261 wait complete. 2024-03-06T07:26:13.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.185+0000 7f09fb381640 1 Processor -- start 2024-03-06T07:26:13.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.185+0000 7f09fb381640 1 -- start start 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09f40b3060 0x7f09f413d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f413dc10 0x7f09f4141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 0x7f09f41f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f413dff0 con 0x7f09f415bf30 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f415b730 con 0x7f09f40b3060 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f40cd7a0 con 0x7f09f413dc10 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 0x7f09f41f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 0x7f09f41f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55104/0 (socket says 172.21.15.122:55104) 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 -- 172.21.15.122:0/1064757641 learned_addr learned my addr 172.21.15.122:0/1064757641 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f891c640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f413dc10 0x7f09f4141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f911d640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09f40b3060 0x7f09f413d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 -- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f413dc10 msgr2=0x7f09f4141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f413dc10 0x7f09f4141500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 -- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09f40b3060 msgr2=0x7f09f413d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09f40b3060 0x7f09f413d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 -- 172.21.15.122:0/1064757641 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4153410 con 0x7f09f415bf30 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f891c640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f413dc10 0x7f09f4141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f911d640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09f40b3060 0x7f09f413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09f991e640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 0x7f09f41f3c30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00d7d0 tx=0x7f09ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec004030 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec0041d0 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f41f4680 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.186+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0096a0 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.187+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f41536b0 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.188+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f41f4ba0 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.188+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09ec009840 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.188+0000 7f09e9ffb640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09d806b4d0 0x7f09d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.188+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09ec08a240 con 0x7f09f415bf30 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.188+0000 7f09f911d640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09d806b4d0 0x7f09d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.189+0000 7f09f911d640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09d806b4d0 0x7f09d806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f09f0001d70 tx=0x7f09f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.192+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09f41f4ba0 con 0x7f09f415bf30 2024-03-06T07:26:13.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.441+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f09f406eb00 con 0x7f09f415bf30 2024-03-06T07:26:13.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.758+0000 7f09e9ffb640 1 -- 172.21.15.122:0/1064757641 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v249) v1 ==== 132+0+0 (secure 0 0 0) 0x7f09ec0593c0 con 0x7f09f415bf30 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09d806b4d0 msgr2=0x7f09d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09d806b4d0 0x7f09d806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f09f0001d70 tx=0x7f09f0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 msgr2=0x7f09f41f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 0x7f09f41f3c30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00d7d0 tx=0x7f09ec00dc90 comp rx=0 tx=0).stop 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 shutdown_connections 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09d806b4d0 0x7f09d806d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f415bf30 0x7f09f41f3c30 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f413dc10 0x7f09f4141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 --2- 172.21.15.122:0/1064757641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09f40b3060 0x7f09f413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 >> 172.21.15.122:0/1064757641 conn(0x7f09f40cf8a0 msgr2=0x7f09f40b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 shutdown_connections 2024-03-06T07:26:13.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.761+0000 7f09fb381640 1 -- 172.21.15.122:0/1064757641 wait complete. 2024-03-06T07:26:13.774 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:26:13.774 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:13.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:13.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.874+0000 7f247f913640 1 Processor -- start 2024-03-06T07:26:13.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.874+0000 7f247f913640 1 -- start start 2024-03-06T07:26:13.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247f913640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f24781424f0 0x7f24780cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247f913640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 0x7f247815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247f913640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24780cc980 con 0x7f24780cd7a0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247f913640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24780cd220 con 0x7f24781424f0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247f913640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780ccc60 con 0x7f247806da40 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247ceae640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f24780cd7a0 0x7f247806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47702/0 (socket says 172.21.15.122:47702) 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247ceae640 1 -- 172.21.15.122:0/1027185433 learned_addr learned my addr 172.21.15.122:0/1027185433 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247deb0640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 0x7f247815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247d6af640 1 --2- 172.21.15.122:0/1027185433 >> v2:172.21.15.169:3300/0 conn(0x7f24781424f0 0x7f24780cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147816483 0 0) 0x7f24780cc980 con 0x7f24780cd7a0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2464003520 con 0x7f24780cd7a0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2965731208 0 0) 0x7f2464003520 con 0x7f24780cd7a0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 msgr2=0x7f247815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 0x7f247815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 >> v2:172.21.15.169:3300/0 conn(0x7f24781424f0 msgr2=0x7f24780cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 --2- 172.21.15.122:0/1027185433 >> v2:172.21.15.169:3300/0 conn(0x7f24781424f0 0x7f24780cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780ccf40 con 0x7f24780cd7a0 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f2468001670 con 0x7f24780cd7a0 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f247d6af640 1 --2- 172.21.15.122:0/1027185433 >> v2:172.21.15.169:3300/0 conn(0x7f24781424f0 0x7f24780cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 >> v1:172.21.15.122:6789/0 conn(0x7f24780cd7a0 legacy=0x7f247806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 0x7f24640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464003f30 0x7f2464004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2464004d40 0x7f24640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780cd220 con 0x7f2464004d40 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780ccc60 con 0x7f2464003f30 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.875+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246400b9c0 con 0x7f247806da40 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247deb0640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2464004d40 0x7f24640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247ceae640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464003f30 0x7f2464004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247d6af640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 0x7f24640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247deb0640 1 -- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 msgr2=0x7f24640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247deb0640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f247806da40 0x7f24640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247deb0640 1 -- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464003f30 msgr2=0x7f2464004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247deb0640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464003f30 0x7f2464004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.876+0000 7f247deb0640 1 -- 172.21.15.122:0/1027185433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464003520 con 0x7f2464004d40 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247deb0640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2464004d40 0x7f24640051c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2470009f90 tx=0x7f2464005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639168a7b097a54b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470039070 con 0x7f2464004d40 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247002cbc0 con 0x7f2464004d40 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f246ffff640 1 -- 172.21.15.122:0/1027185433 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470009c10 con 0x7f2464004d40 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247f913640 1 -- 172.21.15.122:0/1027185433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2464004d40 msgr2=0x7f24640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247f913640 1 --2- 172.21.15.122:0/1027185433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2464004d40 0x7f24640051c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2470009f90 tx=0x7f2464005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247deb0640 1 -- 172.21.15.122:0/1027185433 reap_dead start 2024-03-06T07:26:13.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247f913640 1 -- 172.21.15.122:0/1027185433 shutdown_connections 2024-03-06T07:26:13.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247f913640 1 -- 172.21.15.122:0/1027185433 >> 172.21.15.122:0/1027185433 conn(0x7f24780cf8a0 msgr2=0x7f24780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247f913640 1 -- 172.21.15.122:0/1027185433 shutdown_connections 2024-03-06T07:26:13.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.877+0000 7f247f913640 1 -- 172.21.15.122:0/1027185433 wait complete. 2024-03-06T07:26:13.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.878+0000 7f247f913640 1 Processor -- start 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.878+0000 7f247f913640 1 -- start start 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247f913640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464004d40 0x7f24780cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247f913640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f24780b0ea0 0x7f24780ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247f913640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 0x7f2478140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247f913640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247815d070 con 0x7f2478140450 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247f913640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247815cef0 con 0x7f2464004d40 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247f913640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247815d1f0 con 0x7f24780b0ea0 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247deb0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 0x7f2478140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247deb0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 0x7f2478140900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55118/0 (socket says 172.21.15.122:55118) 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247deb0640 1 -- 172.21.15.122:0/3269281408 learned_addr learned my addr 172.21.15.122:0/3269281408 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.879+0000 7f247deb0640 1 -- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f24780b0ea0 msgr2=0x7f24780ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247ceae640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f24780b0ea0 0x7f24780ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247d6af640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464004d40 0x7f24780cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247deb0640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f24780b0ea0 0x7f24780ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247deb0640 1 -- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464004d40 msgr2=0x7f24780cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247deb0640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464004d40 0x7f24780cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247deb0640 1 -- 172.21.15.122:0/3269281408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780b45e0 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247deb0640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 0x7f2478140900 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f2470009f90 tx=0x7f2470009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470039030 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247003a040 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24780b48c0 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470003990 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.880+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24780b2f80 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.881+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24780b3180 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.882+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2470045030 con 0x7f2478140450 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.882+0000 7f246dffb640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f245c06b500 0x7f245c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.882+0000 7f247d6af640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f245c06b500 0x7f245c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.882+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f24700abdf0 con 0x7f2478140450 2024-03-06T07:26:13.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.882+0000 7f247d6af640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f245c06b500 0x7f245c06d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2474009bf0 tx=0x7f2474008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:13.887+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24780b3180 con 0x7f2478140450 2024-03-06T07:26:14.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.058+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f247804f330 con 0x7f2478140450 2024-03-06T07:26:14.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.059+0000 7f246dffb640 1 -- 172.21.15.122:0/3269281408 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13023 (secure 0 0 0) 0x7f247004c020 con 0x7f2478140450 2024-03-06T07:26:14.061 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:14.061 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":85,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:12.748652+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6836/4292480577":"2024-03-07T07:26:09.694580+0000","172.21.15.122:6834/4292480577":"2024-03-07T07:26:09.694580+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:14.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f245c06b500 msgr2=0x7f245c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f245c06b500 0x7f245c06d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2474009bf0 tx=0x7f2474008040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 msgr2=0x7f2478140900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 0x7f2478140900 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f2470009f90 tx=0x7f2470009750 comp rx=0 tx=0).stop 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 shutdown_connections 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f245c06b500 0x7f245c06d9c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2478140450 0x7f2478140900 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f24780b0ea0 0x7f24780ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 --2- 172.21.15.122:0/3269281408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464004d40 0x7f24780cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 >> 172.21.15.122:0/3269281408 conn(0x7f24780cf8a0 msgr2=0x7f24780cea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 shutdown_connections 2024-03-06T07:26:14.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.061+0000 7f247f913640 1 -- 172.21.15.122:0/3269281408 wait complete. 2024-03-06T07:26:14.074 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:14.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:14.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.175+0000 7fedf11a0640 1 Processor -- start 2024-03-06T07:26:14.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedf11a0640 1 -- start start 2024-03-06T07:26:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedf11a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 0x7fedec06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedf11a0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fedec06d5c0 0x7fedec0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedf11a0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec15ff90 con 0x7fedec15bb40 2024-03-06T07:26:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedf11a0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedec0b4dd0 con 0x7fedec06d5c0 2024-03-06T07:26:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedf11a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec06c910 con 0x7fedec0b4970 2024-03-06T07:26:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedead9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fedec15bb40 0x7fedec15f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47706/0 (socket says 172.21.15.122:47706) 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.176+0000 7fedead9d640 1 -- 172.21.15.122:0/2886807283 learned_addr learned my addr 172.21.15.122:0/2886807283 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fede3fff640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 0x7fedec06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedeb59e640 1 --2- 172.21.15.122:0/2886807283 >> v2:172.21.15.169:3300/0 conn(0x7fedec06d5c0 0x7fedec0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939684451 0 0) 0x7fedec15ff90 con 0x7fedec15bb40 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd8003520 con 0x7fedec15bb40 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447440044 0 0) 0x7fedd8003520 con 0x7fedec15bb40 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 msgr2=0x7fedec06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 0x7fedec06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 >> v2:172.21.15.169:3300/0 conn(0x7fedec06d5c0 msgr2=0x7fedec0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 --2- 172.21.15.122:0/2886807283 >> v2:172.21.15.169:3300/0 conn(0x7fedec06d5c0 0x7fedec0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec160270 con 0x7fedec15bb40 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7feddc001670 con 0x7fedec15bb40 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 >> v1:172.21.15.122:6789/0 conn(0x7fedec15bb40 legacy=0x7fedec15f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedd8004800 0x7fedd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 0x7fedd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec06c910 con 0x7fedd80039b0 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd800b9c0 con 0x7fedd8004800 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd800bb40 con 0x7fedec0b4970 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedead9d640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fede3fff640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedd8004800 0x7fedd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedeb59e640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 0x7fedd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedead9d640 1 -- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 msgr2=0x7fedd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedead9d640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec0b4970 0x7fedd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedead9d640 1 -- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedd8004800 msgr2=0x7fedd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedead9d640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedd8004800 0x7fedd8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.177+0000 7fedead9d640 1 -- 172.21.15.122:0/2886807283 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8003520 con 0x7fedd80039b0 2024-03-06T07:26:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.178+0000 7fedead9d640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedd8003dd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7feddc001050 tx=0x7fedd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b6091c0f4a5820 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.178+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc007d70 con 0x7fedd80039b0 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.178+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc030070 con 0x7fedd80039b0 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.178+0000 7fedea59c640 1 -- 172.21.15.122:0/2886807283 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc02cd80 con 0x7fedd80039b0 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedf11a0640 1 -- 172.21.15.122:0/2886807283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 msgr2=0x7fedd8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedf11a0640 1 --2- 172.21.15.122:0/2886807283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedd8003dd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7feddc001050 tx=0x7fedd8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedeb59e640 1 -- 172.21.15.122:0/2886807283 reap_dead start 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedf11a0640 1 -- 172.21.15.122:0/2886807283 shutdown_connections 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedf11a0640 1 -- 172.21.15.122:0/2886807283 >> 172.21.15.122:0/2886807283 conn(0x7fedec06b190 msgr2=0x7fedec06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedf11a0640 1 -- 172.21.15.122:0/2886807283 shutdown_connections 2024-03-06T07:26:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.179+0000 7fedf11a0640 1 -- 172.21.15.122:0/2886807283 wait complete. 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.180+0000 7fedf11a0640 1 Processor -- start 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.180+0000 7fedf11a0640 1 -- start start 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedf11a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedec152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedf11a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedec15cc60 0x7fedec15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedf11a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec1e55d0 0x7fedec1e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedf11a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec15bf20 con 0x7fedd80039b0 2024-03-06T07:26:14.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedf11a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec06cb60 con 0x7fedec15cc60 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedf11a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec15c0a0 con 0x7fedec1e55d0 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedec152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedec152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55140/0 (socket says 172.21.15.122:55140) 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 -- 172.21.15.122:0/1449493108 learned_addr learned my addr 172.21.15.122:0/1449493108 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 -- 172.21.15.122:0/1449493108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec1e55d0 msgr2=0x7fedec1e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec1e55d0 0x7fedec1e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 -- 172.21.15.122:0/1449493108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedec15cc60 msgr2=0x7fedec15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedec15cc60 0x7fedec15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 -- 172.21.15.122:0/1449493108 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec14e3b0 con 0x7fedd80039b0 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.181+0000 7fedead9d640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedec152990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7feddc02c590 tx=0x7feddc000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.182+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc045070 con 0x7fedd80039b0 2024-03-06T07:26:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.182+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc004bf0 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.182+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec15d800 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.182+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc001cc0 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.182+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec15ddc0 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.183+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec0732b0 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.183+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feddc007070 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.184+0000 7fede37fe640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fedcc06b500 0x7fedcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.184+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feddc0aaf80 con 0x7fedd80039b0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.184+0000 7fede3fff640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fedcc06b500 0x7fedcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.184+0000 7fede3fff640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fedcc06b500 0x7fedcc06d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fedec0ccee0 tx=0x7fedd4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.188+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feddc07a1e0 con 0x7fedd80039b0 2024-03-06T07:26:14.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.360+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fedec068ff0 con 0x7fedd80039b0 2024-03-06T07:26:14.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.360+0000 7fede37fe640 1 -- 172.21.15.122:0/1449493108 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7feddc079ae0 con 0x7fedd80039b0 2024-03-06T07:26:14.362 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:26:14.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fedcc06b500 msgr2=0x7fedcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fedcc06b500 0x7fedcc06d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fedec0ccee0 tx=0x7fedd4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 msgr2=0x7fedec152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedec152990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7feddc02c590 tx=0x7feddc000fe0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 shutdown_connections 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fedcc06b500 0x7fedcc06d9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fedec1e55d0 0x7fedec1e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedec15cc60 0x7fedec15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 --2- 172.21.15.122:0/1449493108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedd80039b0 0x7fedec152990 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 >> 172.21.15.122:0/1449493108 conn(0x7fedec06b190 msgr2=0x7fedec0b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 shutdown_connections 2024-03-06T07:26:14.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.362+0000 7fedf11a0640 1 -- 172.21.15.122:0/1449493108 wait complete. 2024-03-06T07:26:14.375 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:14.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.472+0000 7f87234c9640 1 Processor -- start 2024-03-06T07:26:14.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.472+0000 7f87234c9640 1 -- start start 2024-03-06T07:26:14.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f87234c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f871c0cf420 0x7f871c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f87234c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c0ccac0 0x7f871c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f87234c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f871c06afb0 0x7f871c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f87234c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c0d03b0 con 0x7f871c0cf420 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f87234c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c06b950 con 0x7f871c06afb0 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f87234c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c0ce780 con 0x7f871c0ccac0 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f871c0cf420 0x7f871c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f871c0cf420 0x7f871c0cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55152/0 (socket says 172.21.15.122:55152) 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 -- 172.21.15.122:0/2497852626 learned_addr learned my addr 172.21.15.122:0/2497852626 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8720a64640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c0ccac0 0x7f871c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721a66640 1 --2- 172.21.15.122:0/2497852626 >> v2:172.21.15.169:3300/0 conn(0x7f871c06afb0 0x7f871c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 -- 172.21.15.122:0/2497852626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c0ccac0 msgr2=0x7f871c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c0ccac0 0x7f871c0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 -- 172.21.15.122:0/2497852626 >> v2:172.21.15.169:3300/0 conn(0x7f871c06afb0 msgr2=0x7f871c06b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 --2- 172.21.15.122:0/2497852626 >> v2:172.21.15.169:3300/0 conn(0x7f871c06afb0 0x7f871c06b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721265640 1 -- 172.21.15.122:0/2497852626 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0d05e0 con 0x7f871c0cf420 2024-03-06T07:26:14.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8721a66640 1 --2- 172.21.15.122:0/2497852626 >> v2:172.21.15.169:3300/0 conn(0x7f871c06afb0 0x7f871c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.473+0000 7f8720a64640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c0ccac0 0x7f871c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.474+0000 7f8721265640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c0cf420 0x7f871c0cfe70 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f871c09e480 tx=0x7f871802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc24d6ffbcadf42e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.474+0000 7f8713fff640 1 -- 172.21.15.122:0/2497852626 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871802ca70 con 0x7f871c0cf420 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.474+0000 7f8713fff640 1 -- 172.21.15.122:0/2497852626 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871802cc10 con 0x7f871c0cf420 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.474+0000 7f8713fff640 1 -- 172.21.15.122:0/2497852626 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180343f0 con 0x7f871c0cf420 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 -- 172.21.15.122:0/2497852626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c0cf420 msgr2=0x7f871c0cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c0cf420 0x7f871c0cfe70 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f871c09e480 tx=0x7f871802bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 -- 172.21.15.122:0/2497852626 shutdown_connections 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 --2- 172.21.15.122:0/2497852626 >> v2:172.21.15.169:3300/0 conn(0x7f871c06afb0 0x7f871c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c0ccac0 0x7f871c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 --2- 172.21.15.122:0/2497852626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c0cf420 0x7f871c0cfe70 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 -- 172.21.15.122:0/2497852626 >> 172.21.15.122:0/2497852626 conn(0x7f871c0725f0 msgr2=0x7f871c072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 -- 172.21.15.122:0/2497852626 shutdown_connections 2024-03-06T07:26:14.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.475+0000 7f87234c9640 1 -- 172.21.15.122:0/2497852626 wait complete. 2024-03-06T07:26:14.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.476+0000 7f87234c9640 1 Processor -- start 2024-03-06T07:26:14.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.476+0000 7f87234c9640 1 -- start start 2024-03-06T07:26:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f87234c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c06afb0 0x7f871c0ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f87234c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f871c0ccac0 0x7f871c1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f87234c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 0x7f871c09bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f87234c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c0d0e80 con 0x7f871c1f4270 2024-03-06T07:26:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f87234c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c0d0ba0 con 0x7f871c0ccac0 2024-03-06T07:26:14.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f87234c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c15c270 con 0x7f871c06afb0 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 0x7f871c09bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 0x7f871c09bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56468/0 (socket says 172.21.15.122:56468) 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 -- 172.21.15.122:0/2142380949 learned_addr learned my addr 172.21.15.122:0/2142380949 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721265640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c06afb0 0x7f871c0ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8720a64640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f871c0ccac0 0x7f871c1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 -- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c06afb0 msgr2=0x7f871c0ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c06afb0 0x7f871c0ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 -- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f871c0ccac0 msgr2=0x7f871c1565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f871c0ccac0 0x7f871c1565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 -- 172.21.15.122:0/2142380949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0ad860 con 0x7f871c1f4270 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8720a64640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f871c0ccac0 0x7f871c1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.477+0000 7f8721a66640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 0x7f871c09bbb0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f871400afd0 tx=0x7f871400ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.478+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714015020 con 0x7f871c1f4270 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.478+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8714004440 con 0x7f871c1f4270 2024-03-06T07:26:14.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.478+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c0adb40 con 0x7f871c1f4270 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.478+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714003850 con 0x7f871c1f4270 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.478+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c0ae100 con 0x7f871c1f4270 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.479+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c04f330 con 0x7f871c1f4270 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.479+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8714018020 con 0x7f871c1f4270 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.479+0000 7f8711ffb640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f870006b4d0 0x7f870006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.479+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8714089ea0 con 0x7f871c1f4270 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.479+0000 7f8721265640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f870006b4d0 0x7f870006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.480+0000 7f8721265640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f870006b4d0 0x7f870006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8718035000 tx=0x7f8718030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.484+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f871401f020 con 0x7f871c1f4270 2024-03-06T07:26:14.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.724+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f871c1f4700 con 0x7f871c1f4270 2024-03-06T07:26:14.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.760+0000 7f8711ffb640 1 -- 172.21.15.122:0/2142380949 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v250) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8714059020 con 0x7f871c1f4270 2024-03-06T07:26:14.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f870006b4d0 msgr2=0x7f870006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f870006b4d0 0x7f870006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8718035000 tx=0x7f8718030000 comp rx=0 tx=0).stop 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 msgr2=0x7f871c09bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 0x7f871c09bbb0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f871400afd0 tx=0x7f871400ca20 comp rx=0 tx=0).stop 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 shutdown_connections 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f870006b4d0 0x7f870006d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f871c1f4270 0x7f871c09bbb0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f871c0ccac0 0x7f871c1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 --2- 172.21.15.122:0/2142380949 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f871c06afb0 0x7f871c0ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 >> 172.21.15.122:0/2142380949 conn(0x7f871c0725f0 msgr2=0x7f871c0b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 shutdown_connections 2024-03-06T07:26:14.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.762+0000 7f87234c9640 1 -- 172.21.15.122:0/2142380949 wait complete. 2024-03-06T07:26:14.775 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:14.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.876+0000 7ffbc1f1e640 1 Processor -- start 2024-03-06T07:26:14.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.876+0000 7ffbc1f1e640 1 -- start start 2024-03-06T07:26:14.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbc1f1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc145590 0x7ffbbc06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbc1f1e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc06b970 con 0x7ffbbc06f6e0 2024-03-06T07:26:14.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbc1f1e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc15f7e0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbc1f1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc145970 con 0x7ffbbc145590 2024-03-06T07:26:14.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbbbfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffbbc06f6e0 0x7ffbbc0cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50254/0 (socket says 172.21.15.122:50254) 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbbbfff640 1 -- 172.21.15.122:0/3714802889 learned_addr learned my addr 172.21.15.122:0/3714802889 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbbb7fe640 1 --2- 172.21.15.122:0/3714802889 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc145590 0x7ffbbc06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607471512 0 0) 0x7ffbbc06b970 con 0x7ffbbc06f6e0 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.877+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba4003520 con 0x7ffbbc06f6e0 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153295383 0 0) 0x7ffbbc15f7e0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc06b970 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 82132854 0 0) 0x7ffba4003520 con 0x7ffbbc06f6e0 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3714802889 >> v1:172.21.15.169:6789/0 conn(0x7ffbbc06b4f0 legacy=0x7ffbbc06efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3714802889 >> v1:172.21.15.122:6789/0 conn(0x7ffbbc06f6e0 legacy=0x7ffbbc0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3714802889 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc15fac0 con 0x7ffbbc145590 2024-03-06T07:26:14.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.878+0000 7ffbbb7fe640 1 --2- 172.21.15.122:0/3714802889 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc145590 0x7ffbbc06afb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0097d0 tx=0x7ffbac02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9885cd608ec82763 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac037070 con 0x7ffbbc145590 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac030510 con 0x7ffbbc145590 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbba7fc640 1 -- 172.21.15.122:0/3714802889 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40019f0 con 0x7ffbbc145590 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3714802889 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc145590 msgr2=0x7ffbbc06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3714802889 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc145590 0x7ffbbc06afb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0097d0 tx=0x7ffbac02b800 comp rx=0 tx=0).stop 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3714802889 shutdown_connections 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3714802889 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc145590 0x7ffbbc06afb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3714802889 >> 172.21.15.122:0/3714802889 conn(0x7ffbbc0b2660 msgr2=0x7ffbbc0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3714802889 shutdown_connections 2024-03-06T07:26:14.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.879+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3714802889 wait complete. 2024-03-06T07:26:14.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.880+0000 7ffbc1f1e640 1 Processor -- start 2024-03-06T07:26:14.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 -- start start 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 0x7ffbbc0d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc06f6e0 0x7ffbbc15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbbc145590 0x7ffbbc0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1605a0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc160110 con 0x7ffbbc145590 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbc1f1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc0cf020 con 0x7ffbbc06f6e0 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbbb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 0x7ffbbc0d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbbb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 0x7ffbbc0d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56480/0 (socket says 172.21.15.122:56480) 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.881+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3317106331 learned_addr learned my addr 172.21.15.122:0/3317106331 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbaffd640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc06f6e0 0x7ffbbc15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbbfff640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbbc145590 0x7ffbbc0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc06f6e0 msgr2=0x7ffbbc15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbb7fe640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc06f6e0 0x7ffbbc15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbbc145590 msgr2=0x7ffbbc0ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbb7fe640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbbc145590 0x7ffbbc0ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbb7fe640 1 -- 172.21.15.122:0/3317106331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc0cf300 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbaffd640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc06f6e0 0x7ffbbc15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbbb7fe640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 0x7ffbbc0d00c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ffbac000c00 tx=0x7ffbac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac041020 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac03c870 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc0cf5e0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac037040 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.882+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc0cfba0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.883+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc0733b0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.883+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbac044020 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.884+0000 7ffba3fff640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffba806b4d0 0x7ffba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.884+0000 7ffbbaffd640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffba806b4d0 0x7ffba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.884+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffbac0ab8e0 con 0x7ffbbc06b4f0 2024-03-06T07:26:14.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.884+0000 7ffbbaffd640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffba806b4d0 0x7ffba806d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc139d80 tx=0x7ffbb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:14.888+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbac04b030 con 0x7ffbbc06b4f0 2024-03-06T07:26:15.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.061+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ffbbc068ff0 con 0x7ffbbc06b4f0 2024-03-06T07:26:15.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.064+0000 7ffba3fff640 1 -- 172.21.15.122:0/3317106331 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7ffbac07aa60 con 0x7ffbbc06b4f0 2024-03-06T07:26:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffba806b4d0 msgr2=0x7ffba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffba806b4d0 0x7ffba806d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc139d80 tx=0x7ffbb0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 msgr2=0x7ffbbc0d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 0x7ffbbc0d00c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ffbac000c00 tx=0x7ffbac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 shutdown_connections 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffba806b4d0 0x7ffba806d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbbc145590 0x7ffbbc0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffbbc06f6e0 0x7ffbbc15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 --2- 172.21.15.122:0/3317106331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbbc06b4f0 0x7ffbbc0d00c0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 >> 172.21.15.122:0/3317106331 conn(0x7ffbbc0b2660 msgr2=0x7ffbbc143d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 shutdown_connections 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.066+0000 7ffbc1f1e640 1 -- 172.21.15.122:0/3317106331 wait complete. 2024-03-06T07:26:15.079 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:15.079 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:15.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:15.171 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:15.171 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:15.172 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:15.187 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:15.187 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:26:15.187 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:15.204 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:15.204 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:15.204 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:15.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:15.242 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:26:15.242 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:15.260 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:15.261 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:15.261 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:15.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.380+0000 7f595ba9d640 1 Processor -- start 2024-03-06T07:26:15.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.380+0000 7f595ba9d640 1 -- start start 2024-03-06T07:26:15.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595ba9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595ba9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f59540cd7a0 0x7f595415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595ba9d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59540cc980 con 0x7f595406d330 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595ba9d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59540cd220 con 0x7f59540cd7a0 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595ba9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59540ccc60 con 0x7f59541424f0 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595a03a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f595406d330 0x7f5954071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50260/0 (socket says 172.21.15.122:50260) 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f595a03a640 1 -- 172.21.15.122:0/679362107 learned_addr learned my addr 172.21.15.122:0/679362107 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f5959839640 1 --2- 172.21.15.122:0/679362107 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f5959038640 1 --2- 172.21.15.122:0/679362107 >> v2:172.21.15.169:3300/0 conn(0x7f59540cd7a0 0x7f595415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f5958837640 1 -- 172.21.15.122:0/679362107 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391663295 0 0) 0x7f59540cc980 con 0x7f595406d330 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.381+0000 7f5958837640 1 -- 172.21.15.122:0/679362107 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5940003520 con 0x7f595406d330 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5959839640 1 -- 172.21.15.122:0/679362107 >> v2:172.21.15.169:3300/0 conn(0x7f59540cd7a0 msgr2=0x7f595415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5959839640 1 --2- 172.21.15.122:0/679362107 >> v2:172.21.15.169:3300/0 conn(0x7f59540cd7a0 0x7f595415cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5959839640 1 -- 172.21.15.122:0/679362107 >> v1:172.21.15.122:6789/0 conn(0x7f595406d330 legacy=0x7f5954071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5958837640 1 -- 172.21.15.122:0/679362107 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1720802786 0 0) 0x7f5940003520 con 0x7f595406d330 2024-03-06T07:26:15.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5959839640 1 -- 172.21.15.122:0/679362107 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59540ccf40 con 0x7f59541424f0 2024-03-06T07:26:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5959038640 1 --2- 172.21.15.122:0/679362107 >> v2:172.21.15.169:3300/0 conn(0x7f59540cd7a0 0x7f595415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.382+0000 7f5959839640 1 --2- 172.21.15.122:0/679362107 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540cc2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f59500097a0 tx=0x7f595002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58a169c437fb835 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f5958837640 1 -- 172.21.15.122:0/679362107 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950037070 con 0x7f59541424f0 2024-03-06T07:26:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f5958837640 1 -- 172.21.15.122:0/679362107 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5950030510 con 0x7f59541424f0 2024-03-06T07:26:15.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f5958837640 1 -- 172.21.15.122:0/679362107 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c0019f0 con 0x7f59541424f0 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 -- 172.21.15.122:0/679362107 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 msgr2=0x7f59540cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 --2- 172.21.15.122:0/679362107 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540cc2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f59500097a0 tx=0x7f595002b790 comp rx=0 tx=0).stop 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 -- 172.21.15.122:0/679362107 shutdown_connections 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 --2- 172.21.15.122:0/679362107 >> v2:172.21.15.169:3300/0 conn(0x7f59540cd7a0 0x7f595415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 --2- 172.21.15.122:0/679362107 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540cc2f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 -- 172.21.15.122:0/679362107 >> 172.21.15.122:0/679362107 conn(0x7f59540cf8a0 msgr2=0x7f59540cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 -- 172.21.15.122:0/679362107 shutdown_connections 2024-03-06T07:26:15.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.383+0000 7f595ba9d640 1 -- 172.21.15.122:0/679362107 wait complete. 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.384+0000 7f595ba9d640 1 Processor -- start 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.384+0000 7f595ba9d640 1 -- start start 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595ba9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595406d330 0x7f59541f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595ba9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 0x7f5954144370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595ba9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595ba9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595406d940 con 0x7f59540cd7a0 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595ba9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595406d7c0 con 0x7f595406d330 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595ba9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595406dac0 con 0x7f59541424f0 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 0x7f5954144370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 0x7f5954144370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56484/0 (socket says 172.21.15.122:56484) 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 -- 172.21.15.122:0/3763029350 learned_addr learned my addr 172.21.15.122:0/3763029350 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f595a03a640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959839640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595406d330 0x7f59541f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 -- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 msgr2=0x7f59540ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 -- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595406d330 msgr2=0x7f59541f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595406d330 0x7f59541f5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.385+0000 7f5959038640 1 -- 172.21.15.122:0/3763029350 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5954144c10 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.386+0000 7f5959038640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 0x7f5954144370 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f594400ebe0 tx=0x7f594400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.386+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944017070 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.386+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400cbd0 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.386+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944013470 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.386+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5954144ef0 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.386+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59541448b0 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.387+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59541f5200 con 0x7f59540cd7a0 2024-03-06T07:26:15.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.387+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f594401a070 con 0x7f59540cd7a0 2024-03-06T07:26:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.387+0000 7f594a7fc640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f593806b4d0 0x7f593806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.388+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5944089e60 con 0x7f59540cd7a0 2024-03-06T07:26:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.388+0000 7f5959839640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f593806b4d0 0x7f593806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.388+0000 7f5959839640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f593806b4d0 0x7f593806d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5950034000 tx=0x7f595002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.392+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5944058fe0 con 0x7f59540cd7a0 2024-03-06T07:26:15.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.599+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f595404f330 con 0x7f59540cd7a0 2024-03-06T07:26:15.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.599+0000 7f594a7fc640 1 -- 172.21.15.122:0/3763029350 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f59440588e0 con 0x7f59540cd7a0 2024-03-06T07:26:15.601 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:15.601 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2583981903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2583981903},{"type":"v1","addr":"172.21.15.122:6837","nonce":2583981903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6020,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2530532852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2530532852},{"type":"v1","addr":"172.21.15.169:6835","nonce":2530532852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/84611680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":84611680},{"type":"v1","addr":"172.21.15.169:6834","nonce":84611680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":7380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1914516818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1914516818},{"type":"v1","addr":"172.21.15.122:6836","nonce":1914516818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[]} 2024-03-06T07:26:15.602 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 42 2024-03-06T07:26:15.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f593806b4d0 msgr2=0x7f593806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f593806b4d0 0x7f593806d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5950034000 tx=0x7f595002f000 comp rx=0 tx=0).stop 2024-03-06T07:26:15.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 msgr2=0x7f5954144370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 0x7f5954144370 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f594400ebe0 tx=0x7f594400c520 comp rx=0 tx=0).stop 2024-03-06T07:26:15.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 shutdown_connections 2024-03-06T07:26:15.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f593806b4d0 0x7f593806d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f59541424f0 0x7f59540ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59540cd7a0 0x7f5954144370 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 --2- 172.21.15.122:0/3763029350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f595406d330 0x7f59541f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 >> 172.21.15.122:0/3763029350 conn(0x7f59540cf8a0 msgr2=0x7f59540afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 shutdown_connections 2024-03-06T07:26:15.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.601+0000 7f595ba9d640 1 -- 172.21.15.122:0/3763029350 wait complete. 2024-03-06T07:26:15.616 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:15.617 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:15.617 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:15.617 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:15.617 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:15.617 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:15.617 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:15.617 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:15.618 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:15.618 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:15.618 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:15.618 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:15.618 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:15.618 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:15.618 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:15.619 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:15.626 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:15.626 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:15.627 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:15.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:15.630 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:15.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:15.631 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:15.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:15.669 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:15.669 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:15.669 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:15.670 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:15.672 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:15.673 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:15.673 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:15.673 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:15.674 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:15.744 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:26:15.745 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:26:15.754 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:26:15.754 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:26:15.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.779+0000 7f18e962a640 1 Processor -- start 2024-03-06T07:26:15.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.779+0000 7f18e962a640 1 -- start start 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e962a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f18e40b4460 0x7f18e40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e962a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f18e4161cf0 0x7f18e4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e962a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e40d22c0 con 0x7f18e4162970 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e962a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e4161b00 con 0x7f18e40b4460 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e962a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e40cecc0 con 0x7f18e4161cf0 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f18e40b4460 0x7f18e40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e2ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f18e40b4460 0x7f18e40d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56520/0 (socket says 172.21.15.122:56520) 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e2ffd640 1 -- 172.21.15.122:0/2803673121 learned_addr learned my addr 172.21.15.122:0/2803673121 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e37fe640 1 --2- 172.21.15.122:0/2803673121 >> v2:172.21.15.169:3300/0 conn(0x7f18e4161cf0 0x7f18e4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.780+0000 7f18e2ffd640 1 -- 172.21.15.122:0/2803673121 >> v2:172.21.15.169:3300/0 conn(0x7f18e4161cf0 msgr2=0x7f18e4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e2ffd640 1 --2- 172.21.15.122:0/2803673121 >> v2:172.21.15.169:3300/0 conn(0x7f18e4161cf0 0x7f18e4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e2ffd640 1 -- 172.21.15.122:0/2803673121 >> v1:172.21.15.122:6789/0 conn(0x7f18e4162970 legacy=0x7f18e40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e1ffb640 1 -- 172.21.15.122:0/2803673121 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175258962 0 0) 0x7f18e40d22c0 con 0x7f18e4162970 2024-03-06T07:26:15.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e2ffd640 1 -- 172.21.15.122:0/2803673121 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40d25a0 con 0x7f18e40b4460 2024-03-06T07:26:15.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e37fe640 1 --2- 172.21.15.122:0/2803673121 >> v2:172.21.15.169:3300/0 conn(0x7f18e4161cf0 0x7f18e4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:15.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e2ffd640 1 --2- 172.21.15.122:0/2803673121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e40d4800 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f18e408c900 tx=0x7f18dc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e4ded6a033e653 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e1ffb640 1 -- 172.21.15.122:0/2803673121 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc02cb20 con 0x7f18e40b4460 2024-03-06T07:26:15.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e1ffb640 1 -- 172.21.15.122:0/2803673121 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc02ccc0 con 0x7f18e40b4460 2024-03-06T07:26:15.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.781+0000 7f18e1ffb640 1 -- 172.21.15.122:0/2803673121 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc034610 con 0x7f18e40b4460 2024-03-06T07:26:15.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 -- 172.21.15.122:0/2803673121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 msgr2=0x7f18e40d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 --2- 172.21.15.122:0/2803673121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e40d4800 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f18e408c900 tx=0x7f18dc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 -- 172.21.15.122:0/2803673121 shutdown_connections 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 --2- 172.21.15.122:0/2803673121 >> v2:172.21.15.169:3300/0 conn(0x7f18e4161cf0 0x7f18e4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 --2- 172.21.15.122:0/2803673121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e40d4800 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 -- 172.21.15.122:0/2803673121 >> 172.21.15.122:0/2803673121 conn(0x7f18e40d3ce0 msgr2=0x7f18e40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 -- 172.21.15.122:0/2803673121 shutdown_connections 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.782+0000 7f18e962a640 1 -- 172.21.15.122:0/2803673121 wait complete. 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.783+0000 7f18e962a640 1 Processor -- start 2024-03-06T07:26:15.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.783+0000 7f18e962a640 1 -- start start 2024-03-06T07:26:15.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e962a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e414dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e962a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18e4161cf0 0x7f18e40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e962a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18e4162970 0x7f18e40c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e962a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40ce950 con 0x7f18e40b4460 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e962a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40d2b60 con 0x7f18e4162970 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e962a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40b4840 con 0x7f18e4161cf0 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e414dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e414dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56526/0 (socket says 172.21.15.122:56526) 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 -- 172.21.15.122:0/3507962062 learned_addr learned my addr 172.21.15.122:0/3507962062 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e37fe640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18e4162970 0x7f18e40c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 -- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18e4161cf0 msgr2=0x7f18e40cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18e4161cf0 0x7f18e40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 -- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18e4162970 msgr2=0x7f18e40c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18e4162970 0x7f18e40c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.784+0000 7f18e2ffd640 1 -- 172.21.15.122:0/3507962062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40b7200 con 0x7f18e40b4460 2024-03-06T07:26:15.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.785+0000 7f18e2ffd640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e414dfd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f18dc000f80 tx=0x7f18dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.785+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc043020 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.785+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc03e450 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.785+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc009510 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.785+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e41514d0 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.785+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e40cd080 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.786+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e404f330 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.786+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18dc046020 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.787+0000 7f18db7fe640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f18c406b4d0 0x7f18c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.787+0000 7f18e27fc640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f18c406b4d0 0x7f18c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.787+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f18dc0ac780 con 0x7f18e40b4460 2024-03-06T07:26:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.787+0000 7f18e27fc640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f18c406b4d0 0x7f18c406d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000f60 tx=0x7f18cc0078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.791+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18dc04d030 con 0x7f18e40b4460 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:15.973+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f18e406b180 con 0x7f18e40b4460 2024-03-06T07:26:16.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.079+0000 7f18db7fe640 1 -- 172.21.15.122:0/3507962062 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7f18dc07ba20 con 0x7f18e40b4460 2024-03-06T07:26:16.081 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:26:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f18c406b4d0 msgr2=0x7f18c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f18c406b4d0 0x7f18c406d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000f60 tx=0x7f18cc0078f0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 msgr2=0x7f18e414dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e414dfd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f18dc000f80 tx=0x7f18dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 shutdown_connections 2024-03-06T07:26:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f18c406b4d0 0x7f18c406d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18e4162970 0x7f18e40c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18e4161cf0 0x7f18e40cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 --2- 172.21.15.122:0/3507962062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18e40b4460 0x7f18e414dfd0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 >> 172.21.15.122:0/3507962062 conn(0x7f18e40d3ce0 msgr2=0x7f18e40d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 shutdown_connections 2024-03-06T07:26:16.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.081+0000 7f18e962a640 1 -- 172.21.15.122:0/3507962062 wait complete. 2024-03-06T07:26:16.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:16.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.194+0000 7fab8357a640 1 Processor -- start 2024-03-06T07:26:16.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.194+0000 7fab8357a640 1 -- start start 2024-03-06T07:26:16.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab8357a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fab7c1515e0 0x7fab7c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab8357a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c15b430 0x7fab7c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab8357a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c15b8b0 con 0x7fab7c0cd7a0 2024-03-06T07:26:16.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab8357a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c070a70 con 0x7fab7c1515e0 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab8357a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c070d50 con 0x7fab7c15b430 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab80b15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c15b430 0x7fab7c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81316640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fab7c1515e0 0x7fab7c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81b17640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fab7c0cd7a0 0x7fab7c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:60290/0 (socket says 172.21.15.122:60290) 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81b17640 1 -- 172.21.15.122:0/1321353265 learned_addr learned my addr 172.21.15.122:0/1321353265 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab73fff640 1 -- 172.21.15.122:0/1321353265 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297745260 0 0) 0x7fab7c15b8b0 con 0x7fab7c0cd7a0 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab73fff640 1 -- 172.21.15.122:0/1321353265 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab68003520 con 0x7fab7c0cd7a0 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81316640 1 -- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c15b430 msgr2=0x7fab7c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81316640 1 --2- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c15b430 0x7fab7c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81316640 1 -- 172.21.15.122:0/1321353265 >> v1:172.21.15.169:6789/0 conn(0x7fab7c0cd7a0 legacy=0x7fab7c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab81316640 1 -- 172.21.15.122:0/1321353265 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c071030 con 0x7fab7c1515e0 2024-03-06T07:26:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.195+0000 7fab80b15640 1 --2- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c15b430 0x7fab7c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.196+0000 7fab81316640 1 --2- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cce40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fab780097a0 tx=0x7fab7802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbee3854b9b29588 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.196+0000 7fab73fff640 1 -- 172.21.15.122:0/1321353265 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78035020 con 0x7fab7c1515e0 2024-03-06T07:26:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.196+0000 7fab73fff640 1 -- 172.21.15.122:0/1321353265 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78031830 con 0x7fab7c1515e0 2024-03-06T07:26:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.196+0000 7fab73fff640 1 -- 172.21.15.122:0/1321353265 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78031be0 con 0x7fab7c1515e0 2024-03-06T07:26:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 -- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c1515e0 msgr2=0x7fab7c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 --2- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cce40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fab780097a0 tx=0x7fab7802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 -- 172.21.15.122:0/1321353265 shutdown_connections 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 --2- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c15b430 0x7fab7c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 --2- 172.21.15.122:0/1321353265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cce40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 -- 172.21.15.122:0/1321353265 >> 172.21.15.122:0/1321353265 conn(0x7fab7c0cf8a0 msgr2=0x7fab7c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 -- 172.21.15.122:0/1321353265 shutdown_connections 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.197+0000 7fab8357a640 1 -- 172.21.15.122:0/1321353265 wait complete. 2024-03-06T07:26:16.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.198+0000 7fab8357a640 1 Processor -- start 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.198+0000 7fab8357a640 1 -- start start 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c0cd7a0 0x7fab7c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c15b430 0x7fab7c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c071ef0 con 0x7fab7c1515e0 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c071d70 con 0x7fab7c15b430 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0cc3a0 con 0x7fab7c0cd7a0 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56536/0 (socket says 172.21.15.122:56536) 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 -- 172.21.15.122:0/2195959835 learned_addr learned my addr 172.21.15.122:0/2195959835 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab81b17640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c15b430 0x7fab7c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab81316640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c0cd7a0 0x7fab7c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 -- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c0cd7a0 msgr2=0x7fab7c0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c0cd7a0 0x7fab7c0c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 -- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c15b430 msgr2=0x7fab7c0c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c15b430 0x7fab7c0c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 -- 172.21.15.122:0/2195959835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c1400d0 con 0x7fab7c1515e0 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab81b17640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c15b430 0x7fab7c0c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab80b15640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cb6a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00e860 tx=0x7fab6c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c016070 con 0x7fab7c1515e0 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c00cc40 con 0x7fab7c1515e0 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c1403b0 con 0x7fab7c1515e0 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.199+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c00b040 con 0x7fab7c1515e0 2024-03-06T07:26:16.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.200+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c140970 con 0x7fab7c1515e0 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.201+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7c068840 con 0x7fab7c1515e0 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.201+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab6c019070 con 0x7fab7c1515e0 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.201+0000 7fab71ffb640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab6006b4d0 0x7fab6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.201+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab6c089da0 con 0x7fab7c1515e0 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.201+0000 7fab81316640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab6006b4d0 0x7fab6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.202+0000 7fab81316640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab6006b4d0 0x7fab6006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fab78036000 tx=0x7fab78008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.206+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab6c058f70 con 0x7fab7c1515e0 2024-03-06T07:26:16.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.441+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fab7c0b4c30 con 0x7fab7c1515e0 2024-03-06T07:26:16.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.441+0000 7fab71ffb640 1 -- 172.21.15.122:0/2195959835 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fab6c058870 con 0x7fab7c1515e0 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.444 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.445 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.446 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.448 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.449 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.450 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.451 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:26:16.453 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:26:16.456 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab6006b4d0 msgr2=0x7fab6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab6006b4d0 0x7fab6006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fab78036000 tx=0x7fab78008030 comp rx=0 tx=0).stop 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 msgr2=0x7fab7c0cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cb6a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00e860 tx=0x7fab6c00ed20 comp rx=0 tx=0).stop 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 shutdown_connections 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab6006b4d0 0x7fab6006d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab7c15b430 0x7fab7c0c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab7c1515e0 0x7fab7c0cb6a0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 --2- 172.21.15.122:0/2195959835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab7c0cd7a0 0x7fab7c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 >> 172.21.15.122:0/2195959835 conn(0x7fab7c0cf8a0 msgr2=0x7fab7c14e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.443+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 shutdown_connections 2024-03-06T07:26:16.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.444+0000 7fab8357a640 1 -- 172.21.15.122:0/2195959835 wait complete. 2024-03-06T07:26:16.459 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.557+0000 7f0995d47640 1 Processor -- start 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.557+0000 7f0995d47640 1 -- start start 2024-03-06T07:26:16.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f0995d47640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09900cce40 0x7f099015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f0995d47640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0990071f40 con 0x7f09901515e0 2024-03-06T07:26:16.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f0995d47640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0990070a70 con 0x7f09900cd7a0 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f0995d47640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09900cd2f0 con 0x7f09900cce40 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09900cce40 0x7f099015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09900cce40 0x7f099015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56550/0 (socket says 172.21.15.122:56550) 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098effd640 1 -- 172.21.15.122:0/2160349828 learned_addr learned my addr 172.21.15.122:0/2160349828 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098f7fe640 1 --1- 172.21.15.122:0/2160349828 >> v1:172.21.15.122:6789/0 conn(0x7f09901515e0 0x7f0990071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50300/0 (socket says 172.21.15.122:50300) 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098effd640 1 -- 172.21.15.122:0/2160349828 >> v1:172.21.15.169:6789/0 conn(0x7f09900cd7a0 legacy=0x7f09900cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098effd640 1 -- 172.21.15.122:0/2160349828 >> v1:172.21.15.122:6789/0 conn(0x7f09901515e0 legacy=0x7f0990071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.558+0000 7f098effd640 1 -- 172.21.15.122:0/2160349828 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990070d50 con 0x7f09900cce40 2024-03-06T07:26:16.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.559+0000 7f098effd640 1 --2- 172.21.15.122:0/2160349828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09900cce40 0x7f099015b430 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f098400bb00 tx=0x7f098402df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7bfda122a3113c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.559+0000 7f098e7fc640 1 -- 172.21.15.122:0/2160349828 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984034070 con 0x7f09900cce40 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.559+0000 7f098e7fc640 1 -- 172.21.15.122:0/2160349828 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098402ece0 con 0x7f09900cce40 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.559+0000 7f098e7fc640 1 -- 172.21.15.122:0/2160349828 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984038620 con 0x7f09900cce40 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 -- 172.21.15.122:0/2160349828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09900cce40 msgr2=0x7f099015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 --2- 172.21.15.122:0/2160349828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09900cce40 0x7f099015b430 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f098400bb00 tx=0x7f098402df30 comp rx=0 tx=0).stop 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 -- 172.21.15.122:0/2160349828 shutdown_connections 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 --2- 172.21.15.122:0/2160349828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09900cce40 0x7f099015b430 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 -- 172.21.15.122:0/2160349828 >> 172.21.15.122:0/2160349828 conn(0x7f09900cf8a0 msgr2=0x7f09900cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 -- 172.21.15.122:0/2160349828 shutdown_connections 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.560+0000 7f0995d47640 1 -- 172.21.15.122:0/2160349828 wait complete. 2024-03-06T07:26:16.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.561+0000 7f0995d47640 1 Processor -- start 2024-03-06T07:26:16.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.561+0000 7f0995d47640 1 -- start start 2024-03-06T07:26:16.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f0995d47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09900cce40 0x7f09901597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f0995d47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09900cd7a0 0x7f09900ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f0995d47640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 0x7f09900cee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f0995d47640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09900cc3a0 con 0x7f09901515e0 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f0995d47640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0990071310 con 0x7f09900cce40 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f0995d47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09900cc520 con 0x7f09900cd7a0 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 0x7f09900cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 0x7f09900cee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56554/0 (socket says 172.21.15.122:56554) 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 -- 172.21.15.122:0/2708943485 learned_addr learned my addr 172.21.15.122:0/2708943485 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 -- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09900cd7a0 msgr2=0x7f09900ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09900cd7a0 0x7f09900ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098f7fe640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09900cce40 0x7f09901597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 -- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09900cce40 msgr2=0x7f09901597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09900cce40 0x7f09901597f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 -- 172.21.15.122:0/2708943485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990159d30 con 0x7f09901515e0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.562+0000 7f098ffff640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 0x7f09900cee20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f098800c780 tx=0x7f098800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.563+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988004b60 con 0x7f09901515e0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.563+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0988004d00 con 0x7f09901515e0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.563+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099015e2e0 con 0x7f09901515e0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.563+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098800aa90 con 0x7f09901515e0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.563+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09901f0100 con 0x7f09901515e0 2024-03-06T07:26:16.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.564+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0990068840 con 0x7f09901515e0 2024-03-06T07:26:16.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.564+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f098800ac30 con 0x7f09901515e0 2024-03-06T07:26:16.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.564+0000 7f0977fff640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f097c06b4d0 0x7f097c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.565+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f098808a080 con 0x7f09901515e0 2024-03-06T07:26:16.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.565+0000 7f098f7fe640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f097c06b4d0 0x7f097c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.565+0000 7f098f7fe640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f097c06b4d0 0x7f097c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0980000f60 tx=0x7f0980009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.570+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0988020070 con 0x7f09901515e0 2024-03-06T07:26:16.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.740+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f099004f330 con 0x7f09901515e0 2024-03-06T07:26:16.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.741+0000 7f0977fff640 1 -- 172.21.15.122:0/2708943485 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12892 (secure 0 0 0) 0x7f09880592e0 con 0x7f09901515e0 2024-03-06T07:26:16.743 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:16.743 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":89,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:16.076562+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:16.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f097c06b4d0 msgr2=0x7f097c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f097c06b4d0 0x7f097c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0980000f60 tx=0x7f0980009040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 msgr2=0x7f09900cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 0x7f09900cee20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f098800c780 tx=0x7f098800cc40 comp rx=0 tx=0).stop 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 shutdown_connections 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f097c06b4d0 0x7f097c06d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09901515e0 0x7f09900cee20 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09900cd7a0 0x7f09900ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 --2- 172.21.15.122:0/2708943485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09900cce40 0x7f09901597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 >> 172.21.15.122:0/2708943485 conn(0x7f09900cf8a0 msgr2=0x7f09900cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 shutdown_connections 2024-03-06T07:26:16.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.744+0000 7f0995d47640 1 -- 172.21.15.122:0/2708943485 wait complete. 2024-03-06T07:26:16.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:16.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:16.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.857+0000 7f26bf057640 1 Processor -- start 2024-03-06T07:26:16.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.857+0000 7f26bf057640 1 -- start start 2024-03-06T07:26:16.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26bf057640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b81586e0 0x7f26b80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26bf057640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f26b80aef80 0x7f26b80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26bf057640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b80d03f0 con 0x7f26b80cffd0 2024-03-06T07:26:16.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26bf057640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b8158ac0 con 0x7f26b80aef80 2024-03-06T07:26:16.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26bf057640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b815b2a0 con 0x7f26b81586e0 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f26b80aef80 0x7f26b80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f26b80aef80 0x7f26b80cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56560/0 (socket says 172.21.15.122:56560) 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 -- 172.21.15.122:0/1463525013 learned_addr learned my addr 172.21.15.122:0/1463525013 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 -- 172.21.15.122:0/1463525013 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b81586e0 msgr2=0x7f26b80aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 --2- 172.21.15.122:0/1463525013 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b81586e0 0x7f26b80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 -- 172.21.15.122:0/1463525013 >> v1:172.21.15.169:6789/0 conn(0x7f26b80cffd0 legacy=0x7f26b806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.858+0000 7f26b7fff640 1 -- 172.21.15.122:0/1463525013 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b815b540 con 0x7f26b80aef80 2024-03-06T07:26:16.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.859+0000 7f26b7fff640 1 --2- 172.21.15.122:0/1463525013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b80cfa90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f26a400bb10 tx=0x7f26a402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90ae7a751b51c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.859+0000 7f26b77fe640 1 -- 172.21.15.122:0/1463525013 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a403b070 con 0x7f26b80aef80 2024-03-06T07:26:16.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.859+0000 7f26b77fe640 1 -- 172.21.15.122:0/1463525013 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4036440 con 0x7f26b80aef80 2024-03-06T07:26:16.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.859+0000 7f26b77fe640 1 -- 172.21.15.122:0/1463525013 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4036810 con 0x7f26b80aef80 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 -- 172.21.15.122:0/1463525013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 msgr2=0x7f26b80cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 --2- 172.21.15.122:0/1463525013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b80cfa90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f26a400bb10 tx=0x7f26a402e0c0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 -- 172.21.15.122:0/1463525013 shutdown_connections 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 --2- 172.21.15.122:0/1463525013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b80cfa90 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 --2- 172.21.15.122:0/1463525013 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b81586e0 0x7f26b80aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 -- 172.21.15.122:0/1463525013 >> 172.21.15.122:0/1463525013 conn(0x7f26b806b190 msgr2=0x7f26b806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 -- 172.21.15.122:0/1463525013 shutdown_connections 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.860+0000 7f26bf057640 1 -- 172.21.15.122:0/1463525013 wait complete. 2024-03-06T07:26:16.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.861+0000 7f26bf057640 1 Processor -- start 2024-03-06T07:26:16.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.861+0000 7f26bf057640 1 -- start start 2024-03-06T07:26:16.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bf057640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b81e1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bf057640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b80cffd0 0x7f26b815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bf057640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26b81586e0 0x7f26b806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bf057640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b815bde0 con 0x7f26b80aef80 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bf057640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b815bb00 con 0x7f26b81586e0 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bf057640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8071350 con 0x7f26b80cffd0 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b81e1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b81e1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56562/0 (socket says 172.21.15.122:56562) 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 -- 172.21.15.122:0/976510773 learned_addr learned my addr 172.21.15.122:0/976510773 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 -- 172.21.15.122:0/976510773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b80cffd0 msgr2=0x7f26b815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b80cffd0 0x7f26b815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 -- 172.21.15.122:0/976510773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26b81586e0 msgr2=0x7f26b806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26b81586e0 0x7f26b806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 -- 172.21.15.122:0/976510773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8147d80 con 0x7f26b80aef80 2024-03-06T07:26:16.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26bcdf3640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b81e1170 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00b410 tx=0x7f26ac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac015020 con 0x7f26b80aef80 2024-03-06T07:26:16.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.862+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac011850 con 0x7f26b80aef80 2024-03-06T07:26:16.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.863+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b8148060 con 0x7f26b80aef80 2024-03-06T07:26:16.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.863+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac011c40 con 0x7f26b80aef80 2024-03-06T07:26:16.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.863+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b8148620 con 0x7f26b80aef80 2024-03-06T07:26:16.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.864+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b80732b0 con 0x7f26b80aef80 2024-03-06T07:26:16.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.864+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26ac018020 con 0x7f26b80aef80 2024-03-06T07:26:16.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.864+0000 7f26b57fa640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f269c06b4d0 0x7f269c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.864+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f26ac089dd0 con 0x7f26b80aef80 2024-03-06T07:26:16.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.864+0000 7f26b7fff640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f269c06b4d0 0x7f269c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.865+0000 7f26b7fff640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f269c06b4d0 0x7f269c06d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f26a4038000 tx=0x7f26a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:16.869+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26ac01f020 con 0x7f26b80aef80 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.040+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26b8068ff0 con 0x7f26b80aef80 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.041+0000 7f26b57fa640 1 -- 172.21.15.122:0/976510773 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26ac08b020 con 0x7f26b80aef80 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f269c06b4d0 msgr2=0x7f269c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f269c06b4d0 0x7f269c06d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f26a4038000 tx=0x7f26a4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 msgr2=0x7f26b81e1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b81e1170 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00b410 tx=0x7f26ac00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 shutdown_connections 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f269c06b4d0 0x7f269c06d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26b81586e0 0x7f26b806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26b80cffd0 0x7f26b815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 --2- 172.21.15.122:0/976510773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26b80aef80 0x7f26b81e1170 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 >> 172.21.15.122:0/976510773 conn(0x7f26b806b190 msgr2=0x7f26b80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 shutdown_connections 2024-03-06T07:26:17.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.043+0000 7f26bf057640 1 -- 172.21.15.122:0/976510773 wait complete. 2024-03-06T07:26:17.056 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:17.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2f0dc7640 1 Processor -- start 2024-03-06T07:26:17.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2f0dc7640 1 -- start start 2024-03-06T07:26:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2f0dc7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb2ec15bf30 0x7fb2ec15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2f0dc7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec15f580 con 0x7fb2ec13dc10 2024-03-06T07:26:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2f0dc7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec15b450 con 0x7fb2ec0b3060 2024-03-06T07:26:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2f0dc7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec0cf3f0 con 0x7fb2ec15bf30 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2ead9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb2ec15bf30 0x7fb2ec15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2ead9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb2ec15bf30 0x7fb2ec15ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56564/0 (socket says 172.21.15.122:56564) 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2ea59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb2ec13dc10 0x7fb2ec0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50314/0 (socket says 172.21.15.122:50314) 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.155+0000 7fb2ead9d640 1 -- 172.21.15.122:0/1314936137 learned_addr learned my addr 172.21.15.122:0/1314936137 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.156+0000 7fb2ead9d640 1 -- 172.21.15.122:0/1314936137 >> v1:172.21.15.169:6789/0 conn(0x7fb2ec0b3060 legacy=0x7fb2ec15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.156+0000 7fb2ead9d640 1 -- 172.21.15.122:0/1314936137 >> v1:172.21.15.122:6789/0 conn(0x7fb2ec13dc10 legacy=0x7fb2ec0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.156+0000 7fb2ead9d640 1 -- 172.21.15.122:0/1314936137 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec15b730 con 0x7fb2ec15bf30 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.156+0000 7fb2e959a640 1 -- 172.21.15.122:0/1314936137 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147417751 0 0) 0x7fb2ec15f580 con 0x7fb2ec13dc10 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.156+0000 7fb2ead9d640 1 --2- 172.21.15.122:0/1314936137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15ef60 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009870 tx=0x7fb2dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b137736b1a19e0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2e959a640 1 -- 172.21.15.122:0/1314936137 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc035020 con 0x7fb2ec15bf30 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2e959a640 1 -- 172.21.15.122:0/1314936137 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2dc031870 con 0x7fb2ec15bf30 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2e959a640 1 -- 172.21.15.122:0/1314936137 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc031c40 con 0x7fb2ec15bf30 2024-03-06T07:26:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/1314936137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 msgr2=0x7fb2ec15ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/1314936137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15ef60 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009870 tx=0x7fb2dc02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/1314936137 shutdown_connections 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/1314936137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15ef60 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/1314936137 >> 172.21.15.122:0/1314936137 conn(0x7fb2ec0cf8a0 msgr2=0x7fb2ec0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/1314936137 shutdown_connections 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.157+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/1314936137 wait complete. 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.158+0000 7fb2f0dc7640 1 Processor -- start 2024-03-06T07:26:17.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 -- start start 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2ec0b3060 0x7fb2ec159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2ec13dc10 0x7fb2ec150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec0718e0 con 0x7fb2ec15bf30 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec15b210 con 0x7fb2ec0b3060 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2f0dc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec071a60 con 0x7fb2ec13dc10 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2ead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2ead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56574/0 (socket says 172.21.15.122:56574) 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2ead9d640 1 -- 172.21.15.122:0/2045621434 learned_addr learned my addr 172.21.15.122:0/2045621434 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.159+0000 7fb2ea59c640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2ec0b3060 0x7fb2ec159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2e9d9b640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2ec13dc10 0x7fb2ec150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2ead9d640 1 -- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2ec13dc10 msgr2=0x7fb2ec150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2ead9d640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2ec13dc10 0x7fb2ec150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2ead9d640 1 -- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2ec0b3060 msgr2=0x7fb2ec159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2ead9d640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2ec0b3060 0x7fb2ec159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2ead9d640 1 -- 172.21.15.122:0/2045621434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec152d90 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2ead9d640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15c340 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc02b950 tx=0x7fb2dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc0443f0 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2dc0445b0 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec151400 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc0449a0 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.160+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec13f420 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.161+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec04f330 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.161+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2dc04b060 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.162+0000 7fb2daffd640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.162+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb2dc0ac0d0 con 0x7fb2ec15bf30 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.162+0000 7fb2ea59c640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.162+0000 7fb2ea59c640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4000f60 tx=0x7fb2e4007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.167+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2dc03f070 con 0x7fb2ec15bf30 2024-03-06T07:26:17.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.340+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb2ec15a970 con 0x7fb2ec15bf30 2024-03-06T07:26:17.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.340+0000 7fb2daffd640 1 -- 172.21.15.122:0/2045621434 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb2dc02f320 con 0x7fb2ec15bf30 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:17.343 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b4d0 msgr2=0x7fb2cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4000f60 tx=0x7fb2e4007b50 comp rx=0 tx=0).stop 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 msgr2=0x7fb2ec15c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15c340 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc02b950 tx=0x7fb2dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 shutdown_connections 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2ec15bf30 0x7fb2ec15c340 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2ec13dc10 0x7fb2ec150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 --2- 172.21.15.122:0/2045621434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2ec0b3060 0x7fb2ec159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.342+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 >> 172.21.15.122:0/2045621434 conn(0x7fb2ec0cf8a0 msgr2=0x7fb2ec06de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.343+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 shutdown_connections 2024-03-06T07:26:17.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.343+0000 7fb2f0dc7640 1 -- 172.21.15.122:0/2045621434 wait complete. 2024-03-06T07:26:17.355 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:17.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:17.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.457+0000 7f76003d0640 1 Processor -- start 2024-03-06T07:26:17.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.457+0000 7f76003d0640 1 -- start start 2024-03-06T07:26:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f76003d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75f80cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f76003d0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f75f80d1560 0x7f75f806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f76003d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f80b0390 con 0x7f75f80afbb0 2024-03-06T07:26:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f76003d0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f80cdba0 con 0x7f75f80d1560 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f76003d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80b00b0 con 0x7f75f815e1c0 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fe96d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f75f80afbb0 0x7f75f80719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50326/0 (socket says 172.21.15.122:50326) 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fe96d640 1 -- 172.21.15.122:0/3087566512 learned_addr learned my addr 172.21.15.122:0/3087566512 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fe16c640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75f80cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd96b640 1 --2- 172.21.15.122:0/3087566512 >> v2:172.21.15.169:3300/0 conn(0x7f75f80d1560 0x7f75f806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912959611 0 0) 0x7f75f80b0390 con 0x7f75f80afbb0 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e4003520 con 0x7f75f80afbb0 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881796620 0 0) 0x7f75e4003520 con 0x7f75f80afbb0 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 msgr2=0x7f75f80cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75f80cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 >> v2:172.21.15.169:3300/0 conn(0x7f75f80d1560 msgr2=0x7f75f806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 --2- 172.21.15.122:0/3087566512 >> v2:172.21.15.169:3300/0 conn(0x7f75f80d1560 0x7f75f806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f80cd550 con 0x7f75f80afbb0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f75f00019f0 con 0x7f75f80afbb0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.458+0000 7f75fe16c640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75f80cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 >> v1:172.21.15.122:6789/0 conn(0x7f75f80afbb0 legacy=0x7f75f80719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75e40039b0 0x7f75e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75e4004800 0x7f75e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80cdba0 con 0x7f75e40039b0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80b00b0 con 0x7f75e4004800 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e400ba10 con 0x7f75f815e1c0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe16c640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75e40039b0 0x7f75e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe96d640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fd96b640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75e4004800 0x7f75e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe16c640 1 -- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 msgr2=0x7f75e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe16c640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75f815e1c0 0x7f75e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe16c640 1 -- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75e4004800 msgr2=0x7f75e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe16c640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75e4004800 0x7f75e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.459+0000 7f75fe16c640 1 -- 172.21.15.122:0/3087566512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4003520 con 0x7f75e40039b0 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f75fe16c640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75e40039b0 0x7f75e4003dd0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009c90 tx=0x7f75e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea2cf92ab4aa7c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4039070 con 0x7f75e40039b0 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f402c8f0 con 0x7f75e40039b0 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f75fd16a640 1 -- 172.21.15.122:0/3087566512 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4009860 con 0x7f75e40039b0 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f76003d0640 1 -- 172.21.15.122:0/3087566512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75e40039b0 msgr2=0x7f75e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f76003d0640 1 --2- 172.21.15.122:0/3087566512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75e40039b0 0x7f75e4003dd0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009c90 tx=0x7f75e4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f75fe96d640 1 -- 172.21.15.122:0/3087566512 reap_dead start 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f76003d0640 1 -- 172.21.15.122:0/3087566512 shutdown_connections 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f76003d0640 1 -- 172.21.15.122:0/3087566512 >> 172.21.15.122:0/3087566512 conn(0x7f75f80d09b0 msgr2=0x7f75f80d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f76003d0640 1 -- 172.21.15.122:0/3087566512 shutdown_connections 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.460+0000 7f76003d0640 1 -- 172.21.15.122:0/3087566512 wait complete. 2024-03-06T07:26:17.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.461+0000 7f76003d0640 1 Processor -- start 2024-03-06T07:26:17.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 -- start start 2024-03-06T07:26:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75e40039b0 0x7f75f81556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75f815f400 0x7f75f814a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 0x7f75f80b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80ce1b0 con 0x7f75f8153e40 2024-03-06T07:26:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80ce030 con 0x7f75f815f400 2024-03-06T07:26:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f76003d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80cfc80 con 0x7f75e40039b0 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f75fe96d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 0x7f75f80b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f75fe96d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 0x7f75f80b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56592/0 (socket says 172.21.15.122:56592) 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f75fe96d640 1 -- 172.21.15.122:0/1704024583 learned_addr learned my addr 172.21.15.122:0/1704024583 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f75fd96b640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75f815f400 0x7f75f814a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.462+0000 7f75fe16c640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75e40039b0 0x7f75f81556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75fe96d640 1 -- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75e40039b0 msgr2=0x7f75f81556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75fe96d640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75e40039b0 0x7f75f81556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75fe96d640 1 -- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75f815f400 msgr2=0x7f75f814a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75fe96d640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75f815f400 0x7f75f814a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75fe96d640 1 -- 172.21.15.122:0/1704024583 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f8154650 con 0x7f75f8153e40 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75fe96d640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 0x7f75f80b4860 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f75f80affe0 tx=0x7f75f0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f000ead0 con 0x7f75f8153e40 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f000ec90 con 0x7f75f8153e40 2024-03-06T07:26:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f8154930 con 0x7f75f8153e40 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0014430 con 0x7f75f8153e40 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.463+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f8154ef0 con 0x7f75f8153e40 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.464+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f804f330 con 0x7f75f8153e40 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.464+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75f00145d0 con 0x7f75f8153e40 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.465+0000 7f75eeffd640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f75dc06b4d0 0x7f75dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.465+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f75f0089ac0 con 0x7f75f8153e40 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.465+0000 7f75fe16c640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f75dc06b4d0 0x7f75dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.465+0000 7f75fe16c640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f75dc06b4d0 0x7f75dc06d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f75f4036000 tx=0x7f75f4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.469+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75f0058640 con 0x7f75f8153e40 2024-03-06T07:26:17.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:17.640+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f75f81593d0 con 0x7f75f8153e40 2024-03-06T07:26:18.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.086+0000 7f75eeffd640 1 -- 172.21.15.122:0/1704024583 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7f75f005c6e0 con 0x7f75f8153e40 2024-03-06T07:26:18.087 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:18.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f75dc06b4d0 msgr2=0x7f75dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f75dc06b4d0 0x7f75dc06d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f75f4036000 tx=0x7f75f4030000 comp rx=0 tx=0).stop 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 msgr2=0x7f75f80b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 0x7f75f80b4860 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f75f80affe0 tx=0x7f75f0007bc0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 shutdown_connections 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f75dc06b4d0 0x7f75dc06d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75f8153e40 0x7f75f80b4860 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75f815f400 0x7f75f814a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 --2- 172.21.15.122:0/1704024583 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f75e40039b0 0x7f75f81556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 >> 172.21.15.122:0/1704024583 conn(0x7f75f80d09b0 msgr2=0x7f75f80ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 shutdown_connections 2024-03-06T07:26:18.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.088+0000 7f76003d0640 1 -- 172.21.15.122:0/1704024583 wait complete. 2024-03-06T07:26:18.105 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 Processor -- start 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 -- start start 2024-03-06T07:26:18.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189815def0 0x7f18980b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f189806cab0 0x7f18980ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f18980ce880 0x7f18980cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18980cecf0 con 0x7f18980ce880 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18980cf1c0 con 0x7f189806cab0 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.214+0000 7f189ef65640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18980cc760 con 0x7f189815def0 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f18980ce880 0x7f18980cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f18980ce880 0x7f18980cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56594/0 (socket says 172.21.15.122:56594) 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 -- 172.21.15.122:0/370392043 learned_addr learned my addr 172.21.15.122:0/370392043 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189cd01640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189815def0 0x7f18980b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f1897fff640 1 --2- 172.21.15.122:0/370392043 >> v2:172.21.15.169:3300/0 conn(0x7f189806cab0 0x7f18980ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 -- 172.21.15.122:0/370392043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189815def0 msgr2=0x7f18980b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189815def0 0x7f18980b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 -- 172.21.15.122:0/370392043 >> v2:172.21.15.169:3300/0 conn(0x7f189806cab0 msgr2=0x7f18980ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 --2- 172.21.15.122:0/370392043 >> v2:172.21.15.169:3300/0 conn(0x7f189806cab0 0x7f18980ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189d502640 1 -- 172.21.15.122:0/370392043 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189806d030 con 0x7f18980ce880 2024-03-06T07:26:18.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f189cd01640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189815def0 0x7f18980b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.215+0000 7f1897fff640 1 --2- 172.21.15.122:0/370392043 >> v2:172.21.15.169:3300/0 conn(0x7f189806cab0 0x7f18980ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189d502640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18980ce880 0x7f18980cfd00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f1898068970 tx=0x7f189002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19bc1d04b33a4e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f18977fe640 1 -- 172.21.15.122:0/370392043 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189002cb00 con 0x7f18980ce880 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f18977fe640 1 -- 172.21.15.122:0/370392043 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189002cca0 con 0x7f18980ce880 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f18977fe640 1 -- 172.21.15.122:0/370392043 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890034610 con 0x7f18980ce880 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 -- 172.21.15.122:0/370392043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18980ce880 msgr2=0x7f18980cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18980ce880 0x7f18980cfd00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f1898068970 tx=0x7f189002bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 -- 172.21.15.122:0/370392043 shutdown_connections 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18980ce880 0x7f18980cfd00 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 --2- 172.21.15.122:0/370392043 >> v2:172.21.15.169:3300/0 conn(0x7f189806cab0 0x7f18980ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 --2- 172.21.15.122:0/370392043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189815def0 0x7f18980b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 -- 172.21.15.122:0/370392043 >> 172.21.15.122:0/370392043 conn(0x7f18980b3690 msgr2=0x7f18980b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 -- 172.21.15.122:0/370392043 shutdown_connections 2024-03-06T07:26:18.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.216+0000 7f189ef65640 1 -- 172.21.15.122:0/370392043 wait complete. 2024-03-06T07:26:18.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.217+0000 7f189ef65640 1 Processor -- start 2024-03-06T07:26:18.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 -- start start 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189806cab0 0x7f18980c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18980ce880 0x7f18980c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 0x7f189814ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18980cc3f0 con 0x7f189815def0 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18980cc110 con 0x7f18980ce880 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189ef65640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189815e2d0 con 0x7f189806cab0 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189d502640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 0x7f189814ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189d502640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 0x7f189814ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56608/0 (socket says 172.21.15.122:56608) 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.218+0000 7f189d502640 1 -- 172.21.15.122:0/2649750613 learned_addr learned my addr 172.21.15.122:0/2649750613 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f1897fff640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18980ce880 0x7f18980c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189cd01640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189806cab0 0x7f18980c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189d502640 1 -- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189806cab0 msgr2=0x7f18980c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189d502640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189806cab0 0x7f18980c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189d502640 1 -- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18980ce880 msgr2=0x7f18980c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189d502640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18980ce880 0x7f18980c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189d502640 1 -- 172.21.15.122:0/2649750613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980c5a80 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f1897fff640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18980ce880 0x7f18980c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189cd01640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189806cab0 0x7f18980c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189d502640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 0x7f189814ff40 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f1890036040 tx=0x7f1890004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890043020 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189002ce50 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1898150480 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890009510 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.219+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18980c8d70 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.220+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189809fcf0 con 0x7f189815def0 2024-03-06T07:26:18.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.221+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1890046020 con 0x7f189815def0 2024-03-06T07:26:18.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.221+0000 7f18957fa640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f187c06b4d0 0x7f187c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.221+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f18900ac720 con 0x7f189815def0 2024-03-06T07:26:18.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.221+0000 7f189cd01640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f187c06b4d0 0x7f187c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.222+0000 7f189cd01640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f187c06b4d0 0x7f187c06d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f188c006fd0 tx=0x7f188c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.226+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f189004d030 con 0x7f189815def0 2024-03-06T07:26:18.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:18.397+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1898068ff0 con 0x7f189815def0 2024-03-06T07:26:19.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.091+0000 7f18957fa640 1 -- 172.21.15.122:0/2649750613 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1898150480 con 0x7f189815def0 2024-03-06T07:26:19.092 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:26:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.094+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f187c06b4d0 msgr2=0x7f187c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.094+0000 7f189ef65640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f187c06b4d0 0x7f187c06d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f188c006fd0 tx=0x7f188c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.094+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 msgr2=0x7f189814ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.094+0000 7f189ef65640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 0x7f189814ff40 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f1890036040 tx=0x7f1890004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:19.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 shutdown_connections 2024-03-06T07:26:19.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f187c06b4d0 0x7f187c06d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f189815def0 0x7f189814ff40 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18980ce880 0x7f18980c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 --2- 172.21.15.122:0/2649750613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f189806cab0 0x7f18980c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 >> 172.21.15.122:0/2649750613 conn(0x7f18980b3690 msgr2=0x7f189806fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.095+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 shutdown_connections 2024-03-06T07:26:19.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.097+0000 7f189ef65640 1 -- 172.21.15.122:0/2649750613 wait complete. 2024-03-06T07:26:19.124 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:19.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.230+0000 7f583cbf6640 1 Processor -- start 2024-03-06T07:26:19.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.230+0000 7f583cbf6640 1 -- start start 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583cbf6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f583815a490 0x7f5838072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583cbf6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583806caa0 con 0x7f583806e270 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583cbf6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380b3340 con 0x7f58380712d0 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583cbf6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58380b3620 con 0x7f583815a490 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f5836d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f583806e270 0x7f583815af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50330/0 (socket says 172.21.15.122:50330) 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f5836d9d640 1 -- 172.21.15.122:0/1097840670 learned_addr learned my addr 172.21.15.122:0/1097840670 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583659c640 1 --2- 172.21.15.122:0/1097840670 >> v2:172.21.15.169:3300/0 conn(0x7f583815a490 0x7f5838072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694311095 0 0) 0x7f583806caa0 con 0x7f583806e270 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820004630 con 0x7f583806e270 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787338857 0 0) 0x7f5820004630 con 0x7f583806e270 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 >> v2:172.21.15.169:3300/0 conn(0x7f583815a490 msgr2=0x7f5838072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 --2- 172.21.15.122:0/1097840670 >> v2:172.21.15.169:3300/0 conn(0x7f583815a490 0x7f5838072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 >> v1:172.21.15.169:6789/0 conn(0x7f58380712d0 legacy=0x7f583815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583806cd40 con 0x7f583806e270 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f5830001a10 con 0x7f583806e270 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 >> v1:172.21.15.122:6789/0 conn(0x7f583806e270 legacy=0x7f583815af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5820004ac0 0x7f5820004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58200054c0 0x7f5820005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820006190 0x7f582000cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58380b3620 con 0x7f5820004ac0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582000d040 con 0x7f58200054c0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582000d240 con 0x7f5820006190 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f583659c640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5820004ac0 0x7f5820004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f5835d9b640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58200054c0 0x7f5820005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.231+0000 7f5836d9d640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820006190 0x7f582000cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583659c640 1 -- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820006190 msgr2=0x7f582000cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583659c640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820006190 0x7f582000cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583659c640 1 -- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58200054c0 msgr2=0x7f5820005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583659c640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58200054c0 0x7f5820005990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583659c640 1 -- 172.21.15.122:0/1097840670 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820004630 con 0x7f5820004ac0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f5836d9d640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820006190 0x7f582000cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f5836d9d640 1 -- 172.21.15.122:0/1097840670 reap_dead start 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583659c640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5820004ac0 0x7f5820004ea0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5828009aa0 tx=0x7f5820006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6800c348f4288107 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.232+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828039070 con 0x7f5820004ac0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582802c6b0 con 0x7f5820004ac0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583559a640 1 -- 172.21.15.122:0/1097840670 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828009670 con 0x7f5820004ac0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 -- 172.21.15.122:0/1097840670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5820004ac0 msgr2=0x7f5820004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5820004ac0 0x7f5820004ea0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5828009aa0 tx=0x7f5820006940 comp rx=0 tx=0).stop 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 -- 172.21.15.122:0/1097840670 shutdown_connections 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1097840670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5820004ac0 0x7f5820004ea0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 -- 172.21.15.122:0/1097840670 >> 172.21.15.122:0/1097840670 conn(0x7f583806afb0 msgr2=0x7f583806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 -- 172.21.15.122:0/1097840670 shutdown_connections 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.233+0000 7f583cbf6640 1 -- 172.21.15.122:0/1097840670 wait complete. 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.234+0000 7f583cbf6640 1 Processor -- start 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.234+0000 7f583cbf6640 1 -- start start 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583cbf6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820004ac0 0x7f5838155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583cbf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58381525e0 0x7f583814e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583cbf6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 0x7f58381e7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583cbf6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5838072130 con 0x7f58381e6ef0 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583cbf6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5838071fb0 con 0x7f58381525e0 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583cbf6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5838071740 con 0x7f5820004ac0 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f5836d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 0x7f58381e7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f5836d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 0x7f58381e7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56620/0 (socket says 172.21.15.122:56620) 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f5836d9d640 1 -- 172.21.15.122:0/1804060903 learned_addr learned my addr 172.21.15.122:0/1804060903 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f5835d9b640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58381525e0 0x7f583814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.235+0000 7f583659c640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820004ac0 0x7f5838155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5836d9d640 1 -- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820004ac0 msgr2=0x7f5838155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5836d9d640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820004ac0 0x7f5838155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5836d9d640 1 -- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58381525e0 msgr2=0x7f583814e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5836d9d640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58381525e0 0x7f583814e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5836d9d640 1 -- 172.21.15.122:0/1804060903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838153250 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5836d9d640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 0x7f58381e7370 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f58380b25c0 tx=0x7f583000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830007a80 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830018070 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830013420 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5838151cb0 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.236+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5838152270 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.237+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583804f330 con 0x7f58381e6ef0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.238+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58300135c0 con 0x7f58381e6ef0 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.238+0000 7f5826ffd640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f581806b500 0x7f581806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.238+0000 7f583659c640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f581806b500 0x7f581806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.238+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f583008a270 con 0x7f58381e6ef0 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.238+0000 7f583659c640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f581806b500 0x7f581806d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5828036000 tx=0x7f5828030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.243+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5830058b90 con 0x7f58381e6ef0 2024-03-06T07:26:19.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:19.451+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f58380cbc90 con 0x7f58381e6ef0 2024-03-06T07:26:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.100+0000 7f5826ffd640 1 -- 172.21.15.122:0/1804060903 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7f583005cc30 con 0x7f58381e6ef0 2024-03-06T07:26:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.109+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f581806b500 msgr2=0x7f581806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.109+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f581806b500 0x7f581806d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5828036000 tx=0x7f5828030000 comp rx=0 tx=0).stop 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.109+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 msgr2=0x7f58381e7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.109+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 0x7f58381e7370 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f58380b25c0 tx=0x7f583000e530 comp rx=0 tx=0).stop 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.109+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 shutdown_connections 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f581806b500 0x7f581806d9c0 secure :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5828036000 tx=0x7f5828030000 comp rx=0 tx=0).stop 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f58381e6ef0 0x7f58381e7370 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58381525e0 0x7f583814e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 --2- 172.21.15.122:0/1804060903 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5820004ac0 0x7f5838155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 >> 172.21.15.122:0/1804060903 conn(0x7f583806afb0 msgr2=0x7f583815ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 shutdown_connections 2024-03-06T07:26:20.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.110+0000 7f583cbf6640 1 -- 172.21.15.122:0/1804060903 wait complete. 2024-03-06T07:26:20.129 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.227+0000 7f445926f640 1 Processor -- start 2024-03-06T07:26:20.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.227+0000 7f445926f640 1 -- start start 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f445926f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f44541515e0 0x7f44540cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f445926f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4454071f40 con 0x7f445415b430 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f445926f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445415b8e0 con 0x7f44540cd7a0 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f445926f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4454070a70 con 0x7f44541515e0 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f44527fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f445415b430 0x7f4454071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50336/0 (socket says 172.21.15.122:50336) 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f44527fc640 1 -- 172.21.15.122:0/1708138286 learned_addr learned my addr 172.21.15.122:0/1708138286 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486232531 0 0) 0x7f4454071f40 con 0x7f445415b430 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f443c003520 con 0x7f445415b430 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.228+0000 7f4452ffd640 1 --2- 172.21.15.122:0/1708138286 >> v2:172.21.15.169:3300/0 conn(0x7f44541515e0 0x7f44540cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2268299267 0 0) 0x7f443c003520 con 0x7f445415b430 2024-03-06T07:26:20.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 >> v2:172.21.15.169:3300/0 conn(0x7f44541515e0 msgr2=0x7f44540cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 --2- 172.21.15.122:0/1708138286 >> v2:172.21.15.169:3300/0 conn(0x7f44541515e0 0x7f44540cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 >> v1:172.21.15.169:6789/0 conn(0x7f44540cd7a0 legacy=0x7f44540cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454070d50 con 0x7f445415b430 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4448001690 con 0x7f445415b430 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 >> v1:172.21.15.122:6789/0 conn(0x7f445415b430 legacy=0x7f4454071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f443c0039b0 0x7f443c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c004800 0x7f443c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f443c0051c0 0x7f443c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4454070a70 con 0x7f443c0039b0 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c00c060 con 0x7f443c004800 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c00c260 con 0x7f443c0051c0 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4452ffd640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f443c0039b0 0x7f443c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f44527fc640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c004800 0x7f443c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f44537fe640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f443c0051c0 0x7f443c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4452ffd640 1 -- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f443c0051c0 msgr2=0x7f443c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4452ffd640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f443c0051c0 0x7f443c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4452ffd640 1 -- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c004800 msgr2=0x7f443c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4452ffd640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c004800 0x7f443c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f4452ffd640 1 -- 172.21.15.122:0/1708138286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c003520 con 0x7f443c0039b0 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f44537fe640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f443c0051c0 0x7f443c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.229+0000 7f44537fe640 1 -- 172.21.15.122:0/1708138286 reap_dead start 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.230+0000 7f4452ffd640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f443c0039b0 0x7f443c003dd0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f4444009fc0 tx=0x7f443c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b00c5ce93743351 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.230+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444039070 con 0x7f443c0039b0 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.230+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444402cc20 con 0x7f443c0039b0 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.230+0000 7f4451ffb640 1 -- 172.21.15.122:0/1708138286 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444009c10 con 0x7f443c0039b0 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.230+0000 7f445926f640 1 -- 172.21.15.122:0/1708138286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f443c0039b0 msgr2=0x7f443c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.230+0000 7f445926f640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f443c0039b0 0x7f443c003dd0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f4444009fc0 tx=0x7f443c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:20.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.231+0000 7f445926f640 1 -- 172.21.15.122:0/1708138286 shutdown_connections 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.231+0000 7f445926f640 1 --2- 172.21.15.122:0/1708138286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f443c0039b0 0x7f443c003dd0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.231+0000 7f445926f640 1 -- 172.21.15.122:0/1708138286 >> 172.21.15.122:0/1708138286 conn(0x7f44540cf8a0 msgr2=0x7f44540cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.231+0000 7f445926f640 1 -- 172.21.15.122:0/1708138286 shutdown_connections 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.231+0000 7f445926f640 1 -- 172.21.15.122:0/1708138286 wait complete. 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.232+0000 7f445926f640 1 Processor -- start 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.232+0000 7f445926f640 1 -- start start 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c0039b0 0x7f4454144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f445413d150 0x7f44541454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 0x7f445406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44540cce40 con 0x7f445406c800 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4454071310 con 0x7f443c0039b0 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44540ccfc0 con 0x7f445413d150 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 0x7f445406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 0x7f445406ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56640/0 (socket says 172.21.15.122:56640) 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 -- 172.21.15.122:0/935210172 learned_addr learned my addr 172.21.15.122:0/935210172 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 -- 172.21.15.122:0/935210172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f445413d150 msgr2=0x7f44541454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f445413d150 0x7f44541454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 -- 172.21.15.122:0/935210172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c0039b0 msgr2=0x7f4454144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c0039b0 0x7f4454144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 -- 172.21.15.122:0/935210172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454145d00 con 0x7f445406c800 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f44537fe640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 0x7f445406ccb0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f44540731f0 tx=0x7f444c00e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c019070 con 0x7f445406c800 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c0050d0 con 0x7f445406c800 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c0035f0 con 0x7f445406c800 2024-03-06T07:26:20.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4454145fe0 con 0x7f445406c800 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.233+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44541465a0 con 0x7f445406c800 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.234+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454068840 con 0x7f445406c800 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.235+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f444c01c070 con 0x7f445406c800 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.235+0000 7f443b7fe640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f444006b4d0 0x7f444006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.235+0000 7f4452ffd640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f444006b4d0 0x7f444006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.235+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f444c0895d0 con 0x7f445406c800 2024-03-06T07:26:20.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.236+0000 7f4452ffd640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f444006b4d0 0x7f444006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4444036000 tx=0x7f444402e070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.240+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f444c0584e0 con 0x7f445406c800 2024-03-06T07:26:20.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.414+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f445404f330 con 0x7f445406c800 2024-03-06T07:26:20.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.415+0000 7f443b7fe640 1 -- 172.21.15.122:0/935210172 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f444c057de0 con 0x7f445406c800 2024-03-06T07:26:20.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:20.418 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":92,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:19.451372+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f444006b4d0 msgr2=0x7f444006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f444006b4d0 0x7f444006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4444036000 tx=0x7f444402e070 comp rx=0 tx=0).stop 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 msgr2=0x7f445406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 0x7f445406ccb0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f44540731f0 tx=0x7f444c00e820 comp rx=0 tx=0).stop 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 shutdown_connections 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f444006b4d0 0x7f444006d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f445406c800 0x7f445406ccb0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f445413d150 0x7f44541454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 --2- 172.21.15.122:0/935210172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f443c0039b0 0x7f4454144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 >> 172.21.15.122:0/935210172 conn(0x7f44540cf8a0 msgr2=0x7f44540cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 shutdown_connections 2024-03-06T07:26:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.417+0000 7f445926f640 1 -- 172.21.15.122:0/935210172 wait complete. 2024-03-06T07:26:20.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.528+0000 7fea9060e640 1 Processor -- start 2024-03-06T07:26:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.528+0000 7fea9060e640 1 -- start start 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.529+0000 7fea9060e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea880d22b0 0x7fea880d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.529+0000 7fea9060e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea880b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.529+0000 7fea9060e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea880b3430 con 0x7fea8815dfe0 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.529+0000 7fea9060e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea880d2c50 con 0x7fea880d22b0 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.529+0000 7fea9060e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b31e0 con 0x7fea880b41f0 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea880d22b0 0x7fea880d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea880d22b0 0x7fea880d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56648/0 (socket says 172.21.15.122:56648) 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8e3aa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fea8815dfe0 0x7fea880cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50340/0 (socket says 172.21.15.122:50340) 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 -- 172.21.15.122:0/1163912374 learned_addr learned my addr 172.21.15.122:0/1163912374 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 -- 172.21.15.122:0/1163912374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 msgr2=0x7fea880b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 --2- 172.21.15.122:0/1163912374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea880b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 -- 172.21.15.122:0/1163912374 >> v1:172.21.15.122:6789/0 conn(0x7fea8815dfe0 legacy=0x7fea880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8dba9640 1 -- 172.21.15.122:0/1163912374 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880b3710 con 0x7fea880d22b0 2024-03-06T07:26:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.530+0000 7fea8d3a8640 1 -- 172.21.15.122:0/1163912374 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318334074 0 0) 0x7fea880b3430 con 0x7fea8815dfe0 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea8dba9640 1 --2- 172.21.15.122:0/1163912374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880d2710 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fea78009870 tx=0x7fea7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df97c7e14d2ba186 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea8d3a8640 1 -- 172.21.15.122:0/1163912374 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78034070 con 0x7fea880d22b0 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea8d3a8640 1 -- 172.21.15.122:0/1163912374 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea78030840 con 0x7fea880d22b0 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea8d3a8640 1 -- 172.21.15.122:0/1163912374 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78030c10 con 0x7fea880d22b0 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 -- 172.21.15.122:0/1163912374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 msgr2=0x7fea880d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 --2- 172.21.15.122:0/1163912374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880d2710 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fea78009870 tx=0x7fea7802b980 comp rx=0 tx=0).stop 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 -- 172.21.15.122:0/1163912374 shutdown_connections 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 --2- 172.21.15.122:0/1163912374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea880b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 --2- 172.21.15.122:0/1163912374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880d2710 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 -- 172.21.15.122:0/1163912374 >> 172.21.15.122:0/1163912374 conn(0x7fea880d3cd0 msgr2=0x7fea880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 -- 172.21.15.122:0/1163912374 shutdown_connections 2024-03-06T07:26:20.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.531+0000 7fea9060e640 1 -- 172.21.15.122:0/1163912374 wait complete. 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 Processor -- start 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 -- start start 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea8814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea8815dfe0 0x7fea8814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880cfa00 con 0x7fea880d22b0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b3d60 con 0x7fea8815dfe0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea9060e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880d1c70 con 0x7fea880b41f0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea8dba9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea8dba9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56662/0 (socket says 172.21.15.122:56662) 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea8dba9640 1 -- 172.21.15.122:0/3976807652 learned_addr learned my addr 172.21.15.122:0/3976807652 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea8e3aa640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea8814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.533+0000 7fea8ebab640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea8815dfe0 0x7fea8814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8dba9640 1 -- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 msgr2=0x7fea8814af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8dba9640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea8814af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8dba9640 1 -- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea8815dfe0 msgr2=0x7fea8814c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8dba9640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea8815dfe0 0x7fea8814c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8dba9640 1 -- 172.21.15.122:0/3976807652 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880ca670 con 0x7fea880d22b0 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8ebab640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea8815dfe0 0x7fea8814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea8dba9640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880ca130 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fea78009870 tx=0x7fea78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78048070 con 0x7fea880d22b0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea78030640 con 0x7fea880d22b0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8814b6c0 con 0x7fea880d22b0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78034040 con 0x7fea880d22b0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.534+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8814bc80 con 0x7fea880d22b0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.535+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8804f330 con 0x7fea880d22b0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.535+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea7804b070 con 0x7fea880d22b0 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.536+0000 7fea7effd640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.536+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea780acba0 con 0x7fea880d22b0 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.536+0000 7fea8e3aa640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.536+0000 7fea8e3aa640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea6c06b4d0 0x7fea6c06d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fea84007fd0 tx=0x7fea84007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.540+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea7807aab0 con 0x7fea880d22b0 2024-03-06T07:26:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.714+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea8806bfe0 con 0x7fea880d22b0 2024-03-06T07:26:20.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.715+0000 7fea7effd640 1 -- 172.21.15.122:0/3976807652 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16621 (secure 0 0 0) 0x7fea7807eb50 con 0x7fea880d22b0 2024-03-06T07:26:20.717 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:20.717 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":92,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:19.451372+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea6c06b4d0 msgr2=0x7fea6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea6c06b4d0 0x7fea6c06d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fea84007fd0 tx=0x7fea84007de0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 msgr2=0x7fea880ca130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880ca130 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fea78009870 tx=0x7fea78004730 comp rx=0 tx=0).stop 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 shutdown_connections 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea8815dfe0 0x7fea8814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea880d22b0 0x7fea880ca130 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 --2- 172.21.15.122:0/3976807652 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea880b41f0 0x7fea8814af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 >> 172.21.15.122:0/3976807652 conn(0x7fea880d3cd0 msgr2=0x7fea8806c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 shutdown_connections 2024-03-06T07:26:20.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.717+0000 7fea9060e640 1 -- 172.21.15.122:0/3976807652 wait complete. 2024-03-06T07:26:20.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:20.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.831+0000 7f3717dc9640 1 Processor -- start 2024-03-06T07:26:20.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.831+0000 7f3717dc9640 1 -- start start 2024-03-06T07:26:20.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3717dc9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f37100b4970 0x7f371006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3717dc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371015ff90 con 0x7f371015bb40 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3717dc9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3710160270 con 0x7f371006d5c0 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3717dc9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37100b4dd0 con 0x7f37100b4970 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3715b65640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f371015bb40 0x7f371015f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50356/0 (socket says 172.21.15.122:50356) 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3715b65640 1 -- 172.21.15.122:0/3233357894 learned_addr learned my addr 172.21.15.122:0/3233357894 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3715364640 1 --2- 172.21.15.122:0/3233357894 >> v2:172.21.15.169:3300/0 conn(0x7f37100b4970 0x7f371006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3714b63640 1 -- 172.21.15.122:0/3233357894 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934208473 0 0) 0x7f371015ff90 con 0x7f371015bb40 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.832+0000 7f3714b63640 1 -- 172.21.15.122:0/3233357894 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc003520 con 0x7f371015bb40 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.833+0000 7f3715364640 1 -- 172.21.15.122:0/3233357894 >> v1:172.21.15.169:6789/0 conn(0x7f371006d5c0 legacy=0x7f37100cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.833+0000 7f3715364640 1 -- 172.21.15.122:0/3233357894 >> v1:172.21.15.122:6789/0 conn(0x7f371015bb40 legacy=0x7f371015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.833+0000 7f3715364640 1 -- 172.21.15.122:0/3233357894 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710160550 con 0x7f37100b4970 2024-03-06T07:26:20.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.833+0000 7f3715364640 1 --2- 172.21.15.122:0/3233357894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37100b4970 0x7f371006d080 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f37000097a0 tx=0x7f370002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc021540a99571a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3714b63640 1 -- 172.21.15.122:0/3233357894 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700034070 con 0x7f37100b4970 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3714b63640 1 -- 172.21.15.122:0/3233357894 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700030610 con 0x7f37100b4970 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3714b63640 1 -- 172.21.15.122:0/3233357894 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37000309c0 con 0x7f37100b4970 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 -- 172.21.15.122:0/3233357894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37100b4970 msgr2=0x7f371006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 --2- 172.21.15.122:0/3233357894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37100b4970 0x7f371006d080 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f37000097a0 tx=0x7f370002b960 comp rx=0 tx=0).stop 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 -- 172.21.15.122:0/3233357894 shutdown_connections 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 --2- 172.21.15.122:0/3233357894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37100b4970 0x7f371006d080 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 -- 172.21.15.122:0/3233357894 >> 172.21.15.122:0/3233357894 conn(0x7f371006b190 msgr2=0x7f371006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 -- 172.21.15.122:0/3233357894 shutdown_connections 2024-03-06T07:26:20.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.834+0000 7f3717dc9640 1 -- 172.21.15.122:0/3233357894 wait complete. 2024-03-06T07:26:20.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.835+0000 7f3717dc9640 1 Processor -- start 2024-03-06T07:26:20.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.835+0000 7f3717dc9640 1 -- start start 2024-03-06T07:26:20.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3717dc9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 0x7f3710148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3717dc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37100b4970 0x7f3710147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3717dc9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371015bb40 0x7f371015ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3717dc9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371015bf20 con 0x7f371006d5c0 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3717dc9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371006c910 con 0x7f371015bb40 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3717dc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371015c0a0 con 0x7f37100b4970 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 0x7f3710148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 0x7f3710148030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56668/0 (socket says 172.21.15.122:56668) 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 -- 172.21.15.122:0/1912913734 learned_addr learned my addr 172.21.15.122:0/1912913734 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3716366640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371015bb40 0x7f371015ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715364640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37100b4970 0x7f3710147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 -- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37100b4970 msgr2=0x7f3710147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37100b4970 0x7f3710147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 -- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371015bb40 msgr2=0x7f371015ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371015bb40 0x7f371015ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3715b65640 1 -- 172.21.15.122:0/1912913734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710148570 con 0x7f371006d5c0 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.836+0000 7f3716366640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371015bb40 0x7f371015ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.837+0000 7f3715b65640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 0x7f3710148030 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f370c00c410 tx=0x7f370c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.837+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c017070 con 0x7f371006d5c0 2024-03-06T07:26:20.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.837+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c005ce0 con 0x7f371006d5c0 2024-03-06T07:26:20.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.837+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c0072f0 con 0x7f371006d5c0 2024-03-06T07:26:20.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.837+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3710070d60 con 0x7f371006d5c0 2024-03-06T07:26:20.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.837+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3710071600 con 0x7f371006d5c0 2024-03-06T07:26:20.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.838+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37101553d0 con 0x7f371006d5c0 2024-03-06T07:26:20.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.838+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f370c01a070 con 0x7f371006d5c0 2024-03-06T07:26:20.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.838+0000 7f37067fc640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f36f406b4d0 0x7f36f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.838+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f370c0895a0 con 0x7f371006d5c0 2024-03-06T07:26:20.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.839+0000 7f3715364640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f36f406b4d0 0x7f36f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.839+0000 7f3715364640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f36f406b4d0 0x7f36f406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3700036000 tx=0x7f3700008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:20.843+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37101553d0 con 0x7f371006d5c0 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.053+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3710068ff0 con 0x7f371006d5c0 2024-03-06T07:26:21.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.100+0000 7f37067fc640 1 -- 172.21.15.122:0/1912913734 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v46) v1 ==== 118+0+0 (secure 0 0 0) 0x7f370c058570 con 0x7f371006d5c0 2024-03-06T07:26:21.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f36f406b4d0 msgr2=0x7f36f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f36f406b4d0 0x7f36f406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3700036000 tx=0x7f3700008030 comp rx=0 tx=0).stop 2024-03-06T07:26:21.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 msgr2=0x7f3710148030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 0x7f3710148030 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f370c00c410 tx=0x7f370c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 shutdown_connections 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f36f406b4d0 0x7f36f406d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f371015bb40 0x7f371015ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f37100b4970 0x7f3710147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 --2- 172.21.15.122:0/1912913734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f371006d5c0 0x7f3710148030 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 >> 172.21.15.122:0/1912913734 conn(0x7f371006b190 msgr2=0x7f37101594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 shutdown_connections 2024-03-06T07:26:21.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.102+0000 7f3717dc9640 1 -- 172.21.15.122:0/1912913734 wait complete. 2024-03-06T07:26:21.116 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:21.116 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.218+0000 7fad3d44e640 1 Processor -- start 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.218+0000 7fad3d44e640 1 -- start start 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad3d44e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fad3815fc30 0x7fad380b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad3d44e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad381600a0 con 0x7fad380b4d10 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad3d44e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380d47f0 con 0x7fad380d06e0 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad3d44e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad380ced70 con 0x7fad3815fc30 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad377fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fad380b4d10 0x7fad380cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50362/0 (socket says 172.21.15.122:50362) 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad377fe640 1 -- 172.21.15.122:0/1234421089 learned_addr learned my addr 172.21.15.122:0/1234421089 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad367fc640 1 --2- 172.21.15.122:0/1234421089 >> v2:172.21.15.169:3300/0 conn(0x7fad3815fc30 0x7fad380b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad35ffb640 1 -- 172.21.15.122:0/1234421089 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112281349 0 0) 0x7fad381600a0 con 0x7fad380b4d10 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad35ffb640 1 -- 172.21.15.122:0/1234421089 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad20003520 con 0x7fad380b4d10 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad367fc640 1 -- 172.21.15.122:0/1234421089 >> v1:172.21.15.169:6789/0 conn(0x7fad380d06e0 legacy=0x7fad380d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad35ffb640 1 -- 172.21.15.122:0/1234421089 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894477096 0 0) 0x7fad380d47f0 con 0x7fad380d06e0 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad367fc640 1 -- 172.21.15.122:0/1234421089 >> v1:172.21.15.122:6789/0 conn(0x7fad380b4d10 legacy=0x7fad380cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.219+0000 7fad367fc640 1 -- 172.21.15.122:0/1234421089 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380d1ab0 con 0x7fad3815fc30 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.220+0000 7fad367fc640 1 --2- 172.21.15.122:0/1234421089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b47d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0097a0 tx=0x7fad2c02b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2f4f519943daaa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.220+0000 7fad35ffb640 1 -- 172.21.15.122:0/1234421089 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c037070 con 0x7fad3815fc30 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.220+0000 7fad35ffb640 1 -- 172.21.15.122:0/1234421089 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c030510 con 0x7fad3815fc30 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.220+0000 7fad35ffb640 1 -- 172.21.15.122:0/1234421089 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300019f0 con 0x7fad3815fc30 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 -- 172.21.15.122:0/1234421089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 msgr2=0x7fad380b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 --2- 172.21.15.122:0/1234421089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b47d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0097a0 tx=0x7fad2c02b740 comp rx=0 tx=0).stop 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 -- 172.21.15.122:0/1234421089 shutdown_connections 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 --2- 172.21.15.122:0/1234421089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b47d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 -- 172.21.15.122:0/1234421089 >> 172.21.15.122:0/1234421089 conn(0x7fad380d3cd0 msgr2=0x7fad380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 -- 172.21.15.122:0/1234421089 shutdown_connections 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.221+0000 7fad3d44e640 1 -- 172.21.15.122:0/1234421089 wait complete. 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.222+0000 7fad3d44e640 1 Processor -- start 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.222+0000 7fad3d44e640 1 -- start start 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad3d44e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad380b4d10 0x7fad38154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad3d44e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 0x7fad380b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad3d44e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad3d44e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380ce990 con 0x7fad380d06e0 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad3d44e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380d4ab0 con 0x7fad3815fc30 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad3d44e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380b44d0 con 0x7fad380b4d10 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 0x7fad380b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 0x7fad380b6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56670/0 (socket says 172.21.15.122:56670) 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 -- 172.21.15.122:0/738968364 learned_addr learned my addr 172.21.15.122:0/738968364 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad377fe640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad36ffd640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad380b4d10 0x7fad38154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 -- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad380b4d10 msgr2=0x7fad38154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad380b4d10 0x7fad38154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 -- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 msgr2=0x7fad380b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.223+0000 7fad367fc640 1 -- 172.21.15.122:0/738968364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad381e7ea0 con 0x7fad380d06e0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.224+0000 7fad367fc640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 0x7fad380b6d50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fad2c035040 tx=0x7fad2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.224+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c042070 con 0x7fad380d06e0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.224+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c004220 con 0x7fad380d06e0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.224+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad380b3580 con 0x7fad380d06e0 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.224+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c037040 con 0x7fad380d06e0 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.224+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad380b3860 con 0x7fad380d06e0 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.225+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad380a1c80 con 0x7fad380d06e0 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.225+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad2c03a040 con 0x7fad380d06e0 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.225+0000 7fad1f7fe640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad2406b4d0 0x7fad2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.225+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad2c0aba70 con 0x7fad380d06e0 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.226+0000 7fad36ffd640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad2406b4d0 0x7fad2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.226+0000 7fad36ffd640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad2406b4d0 0x7fad2406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fad28000f60 tx=0x7fad28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.230+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad2c07a220 con 0x7fad380d06e0 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:21.440+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fad38068ff0 con 0x7fad380d06e0 2024-03-06T07:26:22.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.114+0000 7fad1f7fe640 1 -- 172.21.15.122:0/738968364 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v47) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad2c07e2c0 con 0x7fad380d06e0 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad2406b4d0 msgr2=0x7fad2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad2406b4d0 0x7fad2406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fad28000f60 tx=0x7fad28009040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 msgr2=0x7fad380b6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 0x7fad380b6d50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fad2c035040 tx=0x7fad2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 shutdown_connections 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fad2406b4d0 0x7fad2406d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad3815fc30 0x7fad380b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad380d06e0 0x7fad380b6d50 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 --2- 172.21.15.122:0/738968364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fad380b4d10 0x7fad38154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 >> 172.21.15.122:0/738968364 conn(0x7fad380d3cd0 msgr2=0x7fad3806b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 shutdown_connections 2024-03-06T07:26:22.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.117+0000 7fad3d44e640 1 -- 172.21.15.122:0/738968364 wait complete. 2024-03-06T07:26:22.130 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:22.130 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:22.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.229+0000 7f3b0c556640 1 Processor -- start 2024-03-06T07:26:22.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.229+0000 7f3b0c556640 1 -- start start 2024-03-06T07:26:22.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0c556640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3b041515e0 0x7f3b040cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0c556640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3b0415b430 0x7f3b04071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0c556640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0415b8e0 con 0x7f3b040cd7a0 2024-03-06T07:26:22.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0c556640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b04070a70 con 0x7f3b041515e0 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0c556640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b04070d50 con 0x7f3b0415b430 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3b041515e0 0x7f3b040cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3b041515e0 0x7f3b040cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56682/0 (socket says 172.21.15.122:56682) 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 -- 172.21.15.122:0/2898647512 learned_addr learned my addr 172.21.15.122:0/2898647512 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b09af1640 1 --2- 172.21.15.122:0/2898647512 >> v2:172.21.15.169:3300/0 conn(0x7f3b0415b430 0x7f3b04071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 -- 172.21.15.122:0/2898647512 >> v2:172.21.15.169:3300/0 conn(0x7f3b0415b430 msgr2=0x7f3b04071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 --2- 172.21.15.122:0/2898647512 >> v2:172.21.15.169:3300/0 conn(0x7f3b0415b430 0x7f3b04071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 -- 172.21.15.122:0/2898647512 >> v1:172.21.15.169:6789/0 conn(0x7f3b040cd7a0 legacy=0x7f3b040cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.230+0000 7f3b0a2f2640 1 -- 172.21.15.122:0/2898647512 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04071030 con 0x7f3b041515e0 2024-03-06T07:26:22.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.231+0000 7f3b0a2f2640 1 --2- 172.21.15.122:0/2898647512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b041515e0 0x7f3b040cce40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f3b040cc110 tx=0x7f3b0002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f31439199d35afb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.231+0000 7f3b092f0640 1 -- 172.21.15.122:0/2898647512 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00034070 con 0x7f3b041515e0 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.231+0000 7f3b092f0640 1 -- 172.21.15.122:0/2898647512 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00030860 con 0x7f3b041515e0 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.231+0000 7f3b092f0640 1 -- 172.21.15.122:0/2898647512 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00030c30 con 0x7f3b041515e0 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.231+0000 7f3b0c556640 1 -- 172.21.15.122:0/2898647512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b041515e0 msgr2=0x7f3b040cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.231+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2898647512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b041515e0 0x7f3b040cce40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f3b040cc110 tx=0x7f3b0002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.232+0000 7f3b0c556640 1 -- 172.21.15.122:0/2898647512 shutdown_connections 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.232+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2898647512 >> v2:172.21.15.169:3300/0 conn(0x7f3b0415b430 0x7f3b04071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.232+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2898647512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b041515e0 0x7f3b040cce40 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.232+0000 7f3b0c556640 1 -- 172.21.15.122:0/2898647512 >> 172.21.15.122:0/2898647512 conn(0x7f3b040cf8a0 msgr2=0x7f3b040cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.232+0000 7f3b0c556640 1 -- 172.21.15.122:0/2898647512 shutdown_connections 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.232+0000 7f3b0c556640 1 -- 172.21.15.122:0/2898647512 wait complete. 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.233+0000 7f3b0c556640 1 Processor -- start 2024-03-06T07:26:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.233+0000 7f3b0c556640 1 -- start start 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.233+0000 7f3b0c556640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b040cd7a0 0x7f3b041474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.233+0000 7f3b0c556640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b041515e0 0x7f3b040c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0c556640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 0x7f3b040ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0c556640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04071ef0 con 0x7f3b0415b430 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0c556640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04071d70 con 0x7f3b040cd7a0 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0c556640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b040cc3a0 con 0x7f3b041515e0 2024-03-06T07:26:22.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 0x7f3b040ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 0x7f3b040ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56686/0 (socket says 172.21.15.122:56686) 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 -- 172.21.15.122:0/2030132673 learned_addr learned my addr 172.21.15.122:0/2030132673 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0a2f2640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b040cd7a0 0x7f3b041474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b09af1640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b041515e0 0x7f3b040c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 -- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b041515e0 msgr2=0x7f3b040c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b041515e0 0x7f3b040c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 -- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b040cd7a0 msgr2=0x7f3b041474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b040cd7a0 0x7f3b041474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 -- 172.21.15.122:0/2030132673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b040c95d0 con 0x7f3b0415b430 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0aaf3640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 0x7f3b040ce7b0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c730 tx=0x7f3afc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc003b90 con 0x7f3b0415b430 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc003d30 con 0x7f3b0415b430 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc00aa90 con 0x7f3b0415b430 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b040c98b0 con 0x7f3b0415b430 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.234+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0414bfb0 con 0x7f3b0415b430 2024-03-06T07:26:22.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.235+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b04068840 con 0x7f3b0415b430 2024-03-06T07:26:22.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.236+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3afc00ac30 con 0x7f3b0415b430 2024-03-06T07:26:22.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.236+0000 7f3afaffd640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ae806b4d0 0x7f3ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.236+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3afc089f90 con 0x7f3b0415b430 2024-03-06T07:26:22.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.236+0000 7f3b0a2f2640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ae806b4d0 0x7f3ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.236+0000 7f3b0a2f2640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ae806b4d0 0x7f3ae806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3b00036000 tx=0x7f3b00008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.240+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3afc058850 con 0x7f3b0415b430 2024-03-06T07:26:22.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.447+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b0404f330 con 0x7f3b0415b430 2024-03-06T07:26:22.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.448+0000 7f3afaffd640 1 -- 172.21.15.122:0/2030132673 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f3afc05c8f0 con 0x7f3b0415b430 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":47,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/2641321695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2641321695},{"type":"v1","addr":"172.21.15.169:6834","nonce":2641321695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6805,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3022103545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3022103545},{"type":"v1","addr":"172.21.15.122:6836","nonce":3022103545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2175287267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2175287267},{"type":"v1","addr":"172.21.15.122:6837","nonce":2175287267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:21.441727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6817":{"gid":6817,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/2187235474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2187235474},{"type":"v1","addr":"172.21.15.169:6835","nonce":2187235474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6817,"qdb_cluster":[6817]},"id":7}]} 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 47 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.449+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ae806b4d0 msgr2=0x7f3ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.449+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ae806b4d0 0x7f3ae806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3b00036000 tx=0x7f3b00008030 comp rx=0 tx=0).stop 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.449+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 msgr2=0x7f3b040ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.449+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 0x7f3b040ce7b0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c730 tx=0x7f3afc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 shutdown_connections 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ae806b4d0 0x7f3ae806d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b0415b430 0x7f3b040ce7b0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b041515e0 0x7f3b040c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 --2- 172.21.15.122:0/2030132673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b040cd7a0 0x7f3b041474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 >> 172.21.15.122:0/2030132673 conn(0x7f3b040cf8a0 msgr2=0x7f3b040cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 shutdown_connections 2024-03-06T07:26:22.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.450+0000 7f3b0c556640 1 -- 172.21.15.122:0/2030132673 wait complete. 2024-03-06T07:26:22.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:22.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.559+0000 7f218e00e640 1 Processor -- start 2024-03-06T07:26:22.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.559+0000 7f218e00e640 1 -- start start 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f218e00e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f21880cdeb0 0x7f218815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f218e00e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2188071f40 con 0x7f21881515e0 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f218e00e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2188070a70 con 0x7f21880cce40 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f218e00e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21880cd2f0 con 0x7f21880cdeb0 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f2187fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f21880cdeb0 0x7f218815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f2187fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f21880cdeb0 0x7f218815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56692/0 (socket says 172.21.15.122:56692) 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f21877fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f21881515e0 0x7f2188071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50372/0 (socket says 172.21.15.122:50372) 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f2187fff640 1 -- 172.21.15.122:0/3735567313 learned_addr learned my addr 172.21.15.122:0/3735567313 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f2187fff640 1 -- 172.21.15.122:0/3735567313 >> v1:172.21.15.169:6789/0 conn(0x7f21880cce40 legacy=0x7f21880cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f2187fff640 1 -- 172.21.15.122:0/3735567313 >> v1:172.21.15.122:6789/0 conn(0x7f21881515e0 legacy=0x7f2188071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.560+0000 7f2187fff640 1 -- 172.21.15.122:0/3735567313 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188070d50 con 0x7f21880cdeb0 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f2187fff640 1 --2- 172.21.15.122:0/3735567313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21880cdeb0 0x7f218815b430 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f218000af60 tx=0x7f218002d110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa1dc54a6a02f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f21867fc640 1 -- 172.21.15.122:0/3735567313 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218002dab0 con 0x7f21880cdeb0 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f21867fc640 1 -- 172.21.15.122:0/3735567313 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218002dc50 con 0x7f21880cdeb0 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f21867fc640 1 -- 172.21.15.122:0/3735567313 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180008500 con 0x7f21880cdeb0 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f218e00e640 1 -- 172.21.15.122:0/3735567313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21880cdeb0 msgr2=0x7f218815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f218e00e640 1 --2- 172.21.15.122:0/3735567313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21880cdeb0 0x7f218815b430 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f218000af60 tx=0x7f218002d110 comp rx=0 tx=0).stop 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f218e00e640 1 -- 172.21.15.122:0/3735567313 shutdown_connections 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f218e00e640 1 --2- 172.21.15.122:0/3735567313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21880cdeb0 0x7f218815b430 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.561+0000 7f218e00e640 1 -- 172.21.15.122:0/3735567313 >> 172.21.15.122:0/3735567313 conn(0x7f21880cf8a0 msgr2=0x7f21880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.562+0000 7f218e00e640 1 -- 172.21.15.122:0/3735567313 shutdown_connections 2024-03-06T07:26:22.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.562+0000 7f218e00e640 1 -- 172.21.15.122:0/3735567313 wait complete. 2024-03-06T07:26:22.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.563+0000 7f218e00e640 1 Processor -- start 2024-03-06T07:26:22.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.563+0000 7f218e00e640 1 -- start start 2024-03-06T07:26:22.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.563+0000 7f218e00e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f21880cce40 0x7f218814d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.563+0000 7f218e00e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21880cdeb0 0x7f21881f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.563+0000 7f218e00e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 0x7f21881589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f218e00e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880cc3a0 con 0x7f21881515e0 2024-03-06T07:26:22.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f218e00e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2188071310 con 0x7f21880cdeb0 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f218e00e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21880cc520 con 0x7f21880cce40 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 0x7f21881589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 0x7f21881589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56694/0 (socket says 172.21.15.122:56694) 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 -- 172.21.15.122:0/3836521859 learned_addr learned my addr 172.21.15.122:0/3836521859 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f21877fe640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f21880cce40 0x7f218814d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2186ffd640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21880cdeb0 0x7f21881f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 -- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f21880cce40 msgr2=0x7f218814d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f21880cce40 0x7f218814d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 -- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21880cdeb0 msgr2=0x7f21881f2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21880cdeb0 0x7f21881f2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 -- 172.21.15.122:0/3836521859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21881562d0 con 0x7f21881515e0 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f2187fff640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 0x7f21881589e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2180037040 tx=0x7f2180004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180044020 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2180040470 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180040880 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21881565b0 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.564+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2188156b70 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.565+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2188068840 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.566+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2180047020 con 0x7f21881515e0 2024-03-06T07:26:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.566+0000 7f216ffff640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f217406b4d0 0x7f217406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.566+0000 7f21877fe640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f217406b4d0 0x7f217406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.566+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21800ace80 con 0x7f21881515e0 2024-03-06T07:26:22.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.567+0000 7f21877fe640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f217406b4d0 0x7f217406d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c007e00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.571+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f218004e030 con 0x7f21881515e0 2024-03-06T07:26:22.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.743+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f218804f330 con 0x7f21881515e0 2024-03-06T07:26:22.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.744+0000 7f216ffff640 1 -- 172.21.15.122:0/3836521859 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16608 (secure 0 0 0) 0x7f218007b7d0 con 0x7f21881515e0 2024-03-06T07:26:22.746 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:22.746 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":93,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:21.100498+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f217406b4d0 msgr2=0x7f217406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f217406b4d0 0x7f217406d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c007e00 comp rx=0 tx=0).stop 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 msgr2=0x7f21881589e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 0x7f21881589e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f2180037040 tx=0x7f2180004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 shutdown_connections 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f217406b4d0 0x7f217406d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21881515e0 0x7f21881589e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21880cdeb0 0x7f21881f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 --2- 172.21.15.122:0/3836521859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f21880cce40 0x7f218814d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 >> 172.21.15.122:0/3836521859 conn(0x7f21880cf8a0 msgr2=0x7f218814ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 shutdown_connections 2024-03-06T07:26:22.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.746+0000 7f218e00e640 1 -- 172.21.15.122:0/3836521859 wait complete. 2024-03-06T07:26:22.759 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:22.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:22.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.857+0000 7f7f75b56640 1 Processor -- start 2024-03-06T07:26:22.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.858+0000 7f7f75b56640 1 -- start start 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.858+0000 7f7f75b56640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7f7015dfe0 0x7f7f700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.858+0000 7f7f75b56640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f700d29c0 0x7f7f700b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.858+0000 7f7f75b56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f700b3430 con 0x7f7f700cf830 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.858+0000 7f7f75b56640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f700b31e0 con 0x7f7f700d29c0 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.858+0000 7f7f75b56640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f700b3d60 con 0x7f7f7015dfe0 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f700d29c0 0x7f7f700b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7f700cf830 0x7f7f700d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50388/0 (socket says 172.21.15.122:50388) 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f700d29c0 0x7f7f700b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56702/0 (socket says 172.21.15.122:56702) 2024-03-06T07:26:22.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6effd640 1 -- 172.21.15.122:0/1588753334 learned_addr learned my addr 172.21.15.122:0/1588753334 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6f7fe640 1 --2- 172.21.15.122:0/1588753334 >> v2:172.21.15.169:3300/0 conn(0x7f7f7015dfe0 0x7f7f700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6ffff640 1 -- 172.21.15.122:0/1588753334 >> v2:172.21.15.169:3300/0 conn(0x7f7f7015dfe0 msgr2=0x7f7f700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6ffff640 1 --2- 172.21.15.122:0/1588753334 >> v2:172.21.15.169:3300/0 conn(0x7f7f7015dfe0 0x7f7f700cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6ffff640 1 -- 172.21.15.122:0/1588753334 >> v1:172.21.15.122:6789/0 conn(0x7f7f700cf830 legacy=0x7f7f700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6ffff640 1 -- 172.21.15.122:0/1588753334 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f700b3710 con 0x7f7f700d29c0 2024-03-06T07:26:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.859+0000 7f7f6e7fc640 1 -- 172.21.15.122:0/1588753334 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398469835 0 0) 0x7f7f700b3430 con 0x7f7f700cf830 2024-03-06T07:26:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f6ffff640 1 --2- 172.21.15.122:0/1588753334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f700d29c0 0x7f7f700b41f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7f70067990 tx=0x7f7f6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808dc0f332b86ea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f6e7fc640 1 -- 172.21.15.122:0/1588753334 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68035020 con 0x7f7f700d29c0 2024-03-06T07:26:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f6e7fc640 1 -- 172.21.15.122:0/1588753334 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f68031830 con 0x7f7f700d29c0 2024-03-06T07:26:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f6e7fc640 1 -- 172.21.15.122:0/1588753334 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68031c00 con 0x7f7f700d29c0 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 -- 172.21.15.122:0/1588753334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f700d29c0 msgr2=0x7f7f700b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 --2- 172.21.15.122:0/1588753334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f700d29c0 0x7f7f700b41f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7f70067990 tx=0x7f7f6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 -- 172.21.15.122:0/1588753334 shutdown_connections 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 --2- 172.21.15.122:0/1588753334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f700d29c0 0x7f7f700b41f0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 --2- 172.21.15.122:0/1588753334 >> v2:172.21.15.169:3300/0 conn(0x7f7f7015dfe0 0x7f7f700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 -- 172.21.15.122:0/1588753334 >> 172.21.15.122:0/1588753334 conn(0x7f7f700d3cd0 msgr2=0x7f7f700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 -- 172.21.15.122:0/1588753334 shutdown_connections 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.860+0000 7f7f75b56640 1 -- 172.21.15.122:0/1588753334 wait complete. 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.861+0000 7f7f75b56640 1 Processor -- start 2024-03-06T07:26:22.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 -- start start 2024-03-06T07:26:22.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f700cf830 0x7f7f700b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f700d29c0 0x7f7f7015a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 0x7f7f700b9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f700d1df0 con 0x7f7f7015dfe0 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f700d1c70 con 0x7f7f700cf830 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f75b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7015e3c0 con 0x7f7f700d29c0 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f6ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 0x7f7f700b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f6ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 0x7f7f700b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56714/0 (socket says 172.21.15.122:56714) 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f6ffff640 1 -- 172.21.15.122:0/674938219 learned_addr learned my addr 172.21.15.122:0/674938219 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f6f7fe640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f700cf830 0x7f7f700b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.862+0000 7f7f6effd640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f700d29c0 0x7f7f7015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6ffff640 1 -- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f700d29c0 msgr2=0x7f7f7015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6ffff640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f700d29c0 0x7f7f7015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6ffff640 1 -- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f700cf830 msgr2=0x7f7f700b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6ffff640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f700cf830 0x7f7f700b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6ffff640 1 -- 172.21.15.122:0/674938219 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f700cc340 con 0x7f7f7015dfe0 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6f7fe640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f700cf830 0x7f7f700b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:22.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f6ffff640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 0x7f7f700b9380 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009890 tx=0x7f7f680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68041070 con 0x7f7f7015dfe0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6803b860 con 0x7f7f7015dfe0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68035040 con 0x7f7f7015dfe0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f700b78c0 con 0x7f7f7015dfe0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.863+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f700b7e80 con 0x7f7f7015dfe0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.864+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7004f330 con 0x7f7f7015dfe0 2024-03-06T07:26:22.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.864+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f68038020 con 0x7f7f7015dfe0 2024-03-06T07:26:22.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.865+0000 7f7f67fff640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f5006b4d0 0x7f7f5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.865+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f680aba40 con 0x7f7f7015dfe0 2024-03-06T07:26:22.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.865+0000 7f7f6f7fe640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f5006b4d0 0x7f7f5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.865+0000 7f7f6f7fe640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f5006b4d0 0x7f7f5006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009fd0 tx=0x7f7f6000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:22.869+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f6807a300 con 0x7f7f7015dfe0 2024-03-06T07:26:23.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.039+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f700d0230 con 0x7f7f5006b4d0 2024-03-06T07:26:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.045+0000 7f7f67fff640 1 -- 172.21.15.122:0/674938219 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252013 (secure 0 0 0) 0x7f7f700d0230 con 0x7f7f5006b4d0 2024-03-06T07:26:23.050 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:26:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f5006b4d0 msgr2=0x7f7f5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f5006b4d0 0x7f7f5006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009fd0 tx=0x7f7f6000b040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 msgr2=0x7f7f700b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 0x7f7f700b9380 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009890 tx=0x7f7f680046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 shutdown_connections 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f5006b4d0 0x7f7f5006d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f7015dfe0 0x7f7f700b9380 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f700d29c0 0x7f7f7015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 --2- 172.21.15.122:0/674938219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f700cf830 0x7f7f700b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 >> 172.21.15.122:0/674938219 conn(0x7f7f700d3cd0 msgr2=0x7f7f700d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 shutdown_connections 2024-03-06T07:26:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.048+0000 7f7f75b56640 1 -- 172.21.15.122:0/674938219 wait complete. 2024-03-06T07:26:23.061 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-03-06T07:26:21.383242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259452,"kb_used_data":9996,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715268,"statfs":{"total":773094113280,"available":772828434432,"internally_reserved":0,"allocated":10235904,"data_stored":5018692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000639"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.119217+0000","last_change":"2024-03-06T07:26:20.115502+0000","last_active":"2024-03-06T07:26:20.119217+0000","last_peered":"2024-03-06T07:26:20.119217+0000","last_clean":"2024-03-06T07:26:20.119217+0000","last_became_active":"2024-03-06T07:26:20.115270+0000","last_became_peered":"2024-03-06T07:26:20.115270+0000","last_unstale":"2024-03-06T07:26:20.119217+0000","last_undegraded":"2024-03-06T07:26:20.119217+0000","last_fullsized":"2024-03-06T07:26:20.119217+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:02.362057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123975+0000","last_change":"2024-03-06T07:26:20.118442+0000","last_active":"2024-03-06T07:26:20.123975+0000","last_peered":"2024-03-06T07:26:20.123975+0000","last_clean":"2024-03-06T07:26:20.123975+0000","last_became_active":"2024-03-06T07:26:20.118191+0000","last_became_peered":"2024-03-06T07:26:20.118191+0000","last_unstale":"2024-03-06T07:26:20.123975+0000","last_undegraded":"2024-03-06T07:26:20.123975+0000","last_fullsized":"2024-03-06T07:26:20.123975+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:13.090726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.119101+0000","last_change":"2024-03-06T07:26:20.114809+0000","last_active":"2024-03-06T07:26:20.119101+0000","last_peered":"2024-03-06T07:26:20.119101+0000","last_clean":"2024-03-06T07:26:20.119101+0000","last_became_active":"2024-03-06T07:26:20.114535+0000","last_became_peered":"2024-03-06T07:26:20.114535+0000","last_unstale":"2024-03-06T07:26:20.119101+0000","last_undegraded":"2024-03-06T07:26:20.119101+0000","last_fullsized":"2024-03-06T07:26:20.119101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:17.638953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110669+0000","last_change":"2024-03-06T07:26:20.100026+0000","last_active":"2024-03-06T07:26:20.110669+0000","last_peered":"2024-03-06T07:26:20.110669+0000","last_clean":"2024-03-06T07:26:20.110669+0000","last_became_active":"2024-03-06T07:26:20.099710+0000","last_became_peered":"2024-03-06T07:26:20.099710+0000","last_unstale":"2024-03-06T07:26:20.110669+0000","last_undegraded":"2024-03-06T07:26:20.110669+0000","last_fullsized":"2024-03-06T07:26:20.110669+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:57.584951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118104+0000","last_change":"2024-03-06T07:26:20.115782+0000","last_active":"2024-03-06T07:26:20.118104+0000","last_peered":"2024-03-06T07:26:20.118104+0000","last_clean":"2024-03-06T07:26:20.118104+0000","last_became_active":"2024-03-06T07:26:20.115459+0000","last_became_peered":"2024-03-06T07:26:20.115459+0000","last_unstale":"2024-03-06T07:26:20.118104+0000","last_undegraded":"2024-03-06T07:26:20.118104+0000","last_fullsized":"2024-03-06T07:26:20.118104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:17.269474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117759+0000","last_change":"2024-03-06T07:26:20.110804+0000","last_active":"2024-03-06T07:26:20.117759+0000","last_peered":"2024-03-06T07:26:20.117759+0000","last_clean":"2024-03-06T07:26:20.117759+0000","last_became_active":"2024-03-06T07:26:20.110506+0000","last_became_peered":"2024-03-06T07:26:20.110506+0000","last_unstale":"2024-03-06T07:26:20.117759+0000","last_undegraded":"2024-03-06T07:26:20.117759+0000","last_fullsized":"2024-03-06T07:26:20.117759+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:55.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118088+0000","last_change":"2024-03-06T07:26:20.108671+0000","last_active":"2024-03-06T07:26:20.118088+0000","last_peered":"2024-03-06T07:26:20.118088+0000","last_clean":"2024-03-06T07:26:20.118088+0000","last_became_active":"2024-03-06T07:26:20.108313+0000","last_became_peered":"2024-03-06T07:26:20.108313+0000","last_unstale":"2024-03-06T07:26:20.118088+0000","last_undegraded":"2024-03-06T07:26:20.118088+0000","last_fullsized":"2024-03-06T07:26:20.118088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:25.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110504+0000","last_change":"2024-03-06T07:26:20.106615+0000","last_active":"2024-03-06T07:26:20.110504+0000","last_peered":"2024-03-06T07:26:20.110504+0000","last_clean":"2024-03-06T07:26:20.110504+0000","last_became_active":"2024-03-06T07:26:20.106373+0000","last_became_peered":"2024-03-06T07:26:20.106373+0000","last_unstale":"2024-03-06T07:26:20.110504+0000","last_undegraded":"2024-03-06T07:26:20.110504+0000","last_fullsized":"2024-03-06T07:26:20.110504+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:25.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123455+0000","last_change":"2024-03-06T07:26:20.117860+0000","last_active":"2024-03-06T07:26:20.123455+0000","last_peered":"2024-03-06T07:26:20.123455+0000","last_clean":"2024-03-06T07:26:20.123455+0000","last_became_active":"2024-03-06T07:26:20.117394+0000","last_became_peered":"2024-03-06T07:26:20.117394+0000","last_unstale":"2024-03-06T07:26:20.123455+0000","last_undegraded":"2024-03-06T07:26:20.123455+0000","last_fullsized":"2024-03-06T07:26:20.123455+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:20.109355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.114179+0000","last_change":"2024-03-06T07:26:20.112053+0000","last_active":"2024-03-06T07:26:20.114179+0000","last_peered":"2024-03-06T07:26:20.114179+0000","last_clean":"2024-03-06T07:26:20.114179+0000","last_became_active":"2024-03-06T07:26:20.111765+0000","last_became_peered":"2024-03-06T07:26:20.111765+0000","last_unstale":"2024-03-06T07:26:20.114179+0000","last_undegraded":"2024-03-06T07:26:20.114179+0000","last_fullsized":"2024-03-06T07:26:20.114179+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:29.411645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120467+0000","last_change":"2024-03-06T07:26:20.113734+0000","last_active":"2024-03-06T07:26:20.120467+0000","last_peered":"2024-03-06T07:26:20.120467+0000","last_clean":"2024-03-06T07:26:20.120467+0000","last_became_active":"2024-03-06T07:26:20.113085+0000","last_became_peered":"2024-03-06T07:26:20.113085+0000","last_unstale":"2024-03-06T07:26:20.120467+0000","last_undegraded":"2024-03-06T07:26:20.120467+0000","last_fullsized":"2024-03-06T07:26:20.120467+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:33.335668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.114477+0000","last_change":"2024-03-06T07:26:20.108964+0000","last_active":"2024-03-06T07:26:20.114477+0000","last_peered":"2024-03-06T07:26:20.114477+0000","last_clean":"2024-03-06T07:26:20.114477+0000","last_became_active":"2024-03-06T07:26:20.108685+0000","last_became_peered":"2024-03-06T07:26:20.108685+0000","last_unstale":"2024-03-06T07:26:20.114477+0000","last_undegraded":"2024-03-06T07:26:20.114477+0000","last_fullsized":"2024-03-06T07:26:20.114477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:41.233378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113591+0000","last_change":"2024-03-06T07:26:20.107142+0000","last_active":"2024-03-06T07:26:20.113591+0000","last_peered":"2024-03-06T07:26:20.113591+0000","last_clean":"2024-03-06T07:26:20.113591+0000","last_became_active":"2024-03-06T07:26:20.106903+0000","last_became_peered":"2024-03-06T07:26:20.106903+0000","last_unstale":"2024-03-06T07:26:20.113591+0000","last_undegraded":"2024-03-06T07:26:20.113591+0000","last_fullsized":"2024-03-06T07:26:20.113591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:07.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.108612+0000","last_change":"2024-03-06T07:26:20.102980+0000","last_active":"2024-03-06T07:26:20.108612+0000","last_peered":"2024-03-06T07:26:20.108612+0000","last_clean":"2024-03-06T07:26:20.108612+0000","last_became_active":"2024-03-06T07:26:20.102723+0000","last_became_peered":"2024-03-06T07:26:20.102723+0000","last_unstale":"2024-03-06T07:26:20.108612+0000","last_undegraded":"2024-03-06T07:26:20.108612+0000","last_fullsized":"2024-03-06T07:26:20.108612+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:16.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113258+0000","last_change":"2024-03-06T07:26:20.105701+0000","last_active":"2024-03-06T07:26:20.113258+0000","last_peered":"2024-03-06T07:26:20.113258+0000","last_clean":"2024-03-06T07:26:20.113258+0000","last_became_active":"2024-03-06T07:26:20.105446+0000","last_became_peered":"2024-03-06T07:26:20.105446+0000","last_unstale":"2024-03-06T07:26:20.113258+0000","last_undegraded":"2024-03-06T07:26:20.113258+0000","last_fullsized":"2024-03-06T07:26:20.113258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:37.165234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107257+0000","last_change":"2024-03-06T07:26:20.105930+0000","last_active":"2024-03-06T07:26:20.107257+0000","last_peered":"2024-03-06T07:26:20.107257+0000","last_clean":"2024-03-06T07:26:20.107257+0000","last_became_active":"2024-03-06T07:26:20.105676+0000","last_became_peered":"2024-03-06T07:26:20.105676+0000","last_unstale":"2024-03-06T07:26:20.107257+0000","last_undegraded":"2024-03-06T07:26:20.107257+0000","last_fullsized":"2024-03-06T07:26:20.107257+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:37.165234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121520+0000","last_change":"2024-03-06T07:26:20.114404+0000","last_active":"2024-03-06T07:26:20.121520+0000","last_peered":"2024-03-06T07:26:20.121520+0000","last_clean":"2024-03-06T07:26:20.121520+0000","last_became_active":"2024-03-06T07:26:20.114137+0000","last_became_peered":"2024-03-06T07:26:20.114137+0000","last_unstale":"2024-03-06T07:26:20.121520+0000","last_undegraded":"2024-03-06T07:26:20.121520+0000","last_fullsized":"2024-03-06T07:26:20.121520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:09.734012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110704+0000","last_change":"2024-03-06T07:26:20.100452+0000","last_active":"2024-03-06T07:26:20.110704+0000","last_peered":"2024-03-06T07:26:20.110704+0000","last_clean":"2024-03-06T07:26:20.110704+0000","last_became_active":"2024-03-06T07:26:20.100210+0000","last_became_peered":"2024-03-06T07:26:20.100210+0000","last_unstale":"2024-03-06T07:26:20.110704+0000","last_undegraded":"2024-03-06T07:26:20.110704+0000","last_fullsized":"2024-03-06T07:26:20.110704+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:34.904820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113315+0000","last_change":"2024-03-06T07:26:20.108035+0000","last_active":"2024-03-06T07:26:20.113315+0000","last_peered":"2024-03-06T07:26:20.113315+0000","last_clean":"2024-03-06T07:26:20.113315+0000","last_became_active":"2024-03-06T07:26:20.107587+0000","last_became_peered":"2024-03-06T07:26:20.107587+0000","last_unstale":"2024-03-06T07:26:20.113315+0000","last_undegraded":"2024-03-06T07:26:20.113315+0000","last_fullsized":"2024-03-06T07:26:20.113315+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:05.023999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123476+0000","last_change":"2024-03-06T07:26:20.117910+0000","last_active":"2024-03-06T07:26:20.123476+0000","last_peered":"2024-03-06T07:26:20.123476+0000","last_clean":"2024-03-06T07:26:20.123476+0000","last_became_active":"2024-03-06T07:26:20.117666+0000","last_became_peered":"2024-03-06T07:26:20.117666+0000","last_unstale":"2024-03-06T07:26:20.123476+0000","last_undegraded":"2024-03-06T07:26:20.123476+0000","last_fullsized":"2024-03-06T07:26:20.123476+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.977117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118308+0000","last_change":"2024-03-06T07:26:20.111165+0000","last_active":"2024-03-06T07:26:20.118308+0000","last_peered":"2024-03-06T07:26:20.118308+0000","last_clean":"2024-03-06T07:26:20.118308+0000","last_became_active":"2024-03-06T07:26:20.110898+0000","last_became_peered":"2024-03-06T07:26:20.110898+0000","last_unstale":"2024-03-06T07:26:20.118308+0000","last_undegraded":"2024-03-06T07:26:20.118308+0000","last_fullsized":"2024-03-06T07:26:20.118308+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:15.257611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124040+0000","last_change":"2024-03-06T07:26:20.119588+0000","last_active":"2024-03-06T07:26:20.124040+0000","last_peered":"2024-03-06T07:26:20.124040+0000","last_clean":"2024-03-06T07:26:20.124040+0000","last_became_active":"2024-03-06T07:26:20.119364+0000","last_became_peered":"2024-03-06T07:26:20.119364+0000","last_unstale":"2024-03-06T07:26:20.124040+0000","last_undegraded":"2024-03-06T07:26:20.124040+0000","last_fullsized":"2024-03-06T07:26:20.124040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:47.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121666+0000","last_change":"2024-03-06T07:26:20.117755+0000","last_active":"2024-03-06T07:26:20.121666+0000","last_peered":"2024-03-06T07:26:20.121666+0000","last_clean":"2024-03-06T07:26:20.121666+0000","last_became_active":"2024-03-06T07:26:20.117490+0000","last_became_peered":"2024-03-06T07:26:20.117490+0000","last_unstale":"2024-03-06T07:26:20.121666+0000","last_undegraded":"2024-03-06T07:26:20.121666+0000","last_fullsized":"2024-03-06T07:26:20.121666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:08.325075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117821+0000","last_change":"2024-03-06T07:26:20.110112+0000","last_active":"2024-03-06T07:26:20.117821+0000","last_peered":"2024-03-06T07:26:20.117821+0000","last_clean":"2024-03-06T07:26:20.117821+0000","last_became_active":"2024-03-06T07:26:20.109083+0000","last_became_peered":"2024-03-06T07:26:20.109083+0000","last_unstale":"2024-03-06T07:26:20.117821+0000","last_undegraded":"2024-03-06T07:26:20.117821+0000","last_fullsized":"2024-03-06T07:26:20.117821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:07.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110440+0000","last_change":"2024-03-06T07:26:20.106989+0000","last_active":"2024-03-06T07:26:20.110440+0000","last_peered":"2024-03-06T07:26:20.110440+0000","last_clean":"2024-03-06T07:26:20.110440+0000","last_became_active":"2024-03-06T07:26:20.106708+0000","last_became_peered":"2024-03-06T07:26:20.106708+0000","last_unstale":"2024-03-06T07:26:20.110440+0000","last_undegraded":"2024-03-06T07:26:20.110440+0000","last_fullsized":"2024-03-06T07:26:20.110440+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117925+0000","last_change":"2024-03-06T07:26:20.112204+0000","last_active":"2024-03-06T07:26:20.117925+0000","last_peered":"2024-03-06T07:26:20.117925+0000","last_clean":"2024-03-06T07:26:20.117925+0000","last_became_active":"2024-03-06T07:26:20.111911+0000","last_became_peered":"2024-03-06T07:26:20.111911+0000","last_unstale":"2024-03-06T07:26:20.117925+0000","last_undegraded":"2024-03-06T07:26:20.117925+0000","last_fullsized":"2024-03-06T07:26:20.117925+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:55.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107290+0000","last_change":"2024-03-06T07:26:20.104134+0000","last_active":"2024-03-06T07:26:20.107290+0000","last_peered":"2024-03-06T07:26:20.107290+0000","last_clean":"2024-03-06T07:26:20.107290+0000","last_became_active":"2024-03-06T07:26:20.103719+0000","last_became_peered":"2024-03-06T07:26:20.103719+0000","last_unstale":"2024-03-06T07:26:20.107290+0000","last_undegraded":"2024-03-06T07:26:20.107290+0000","last_fullsized":"2024-03-06T07:26:20.107290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:24.084552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.116776+0000","last_change":"2024-03-06T07:26:20.115034+0000","last_active":"2024-03-06T07:26:20.116776+0000","last_peered":"2024-03-06T07:26:20.116776+0000","last_clean":"2024-03-06T07:26:20.116776+0000","last_became_active":"2024-03-06T07:26:20.112033+0000","last_became_peered":"2024-03-06T07:26:20.112033+0000","last_unstale":"2024-03-06T07:26:20.116776+0000","last_undegraded":"2024-03-06T07:26:20.116776+0000","last_fullsized":"2024-03-06T07:26:20.116776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:29.012780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110473+0000","last_change":"2024-03-06T07:26:20.106755+0000","last_active":"2024-03-06T07:26:20.110473+0000","last_peered":"2024-03-06T07:26:20.110473+0000","last_clean":"2024-03-06T07:26:20.110473+0000","last_became_active":"2024-03-06T07:26:20.106521+0000","last_became_peered":"2024-03-06T07:26:20.106521+0000","last_unstale":"2024-03-06T07:26:20.110473+0000","last_undegraded":"2024-03-06T07:26:20.110473+0000","last_fullsized":"2024-03-06T07:26:20.110473+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:07.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113714+0000","last_change":"2024-03-06T07:26:20.109914+0000","last_active":"2024-03-06T07:26:20.113714+0000","last_peered":"2024-03-06T07:26:20.113714+0000","last_clean":"2024-03-06T07:26:20.113714+0000","last_became_active":"2024-03-06T07:26:20.109643+0000","last_became_peered":"2024-03-06T07:26:20.109643+0000","last_unstale":"2024-03-06T07:26:20.113714+0000","last_undegraded":"2024-03-06T07:26:20.113714+0000","last_fullsized":"2024-03-06T07:26:20.113714+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:37.953947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113236+0000","last_change":"2024-03-06T07:26:20.104526+0000","last_active":"2024-03-06T07:26:20.113236+0000","last_peered":"2024-03-06T07:26:20.113236+0000","last_clean":"2024-03-06T07:26:20.113236+0000","last_became_active":"2024-03-06T07:26:20.104295+0000","last_became_peered":"2024-03-06T07:26:20.104295+0000","last_unstale":"2024-03-06T07:26:20.113236+0000","last_undegraded":"2024-03-06T07:26:20.113236+0000","last_fullsized":"2024-03-06T07:26:20.113236+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:24.084552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113178+0000","last_change":"2024-03-06T07:26:20.109817+0000","last_active":"2024-03-06T07:26:20.113178+0000","last_peered":"2024-03-06T07:26:20.113178+0000","last_clean":"2024-03-06T07:26:20.113178+0000","last_became_active":"2024-03-06T07:26:20.109556+0000","last_became_peered":"2024-03-06T07:26:20.109556+0000","last_unstale":"2024-03-06T07:26:20.113178+0000","last_undegraded":"2024-03-06T07:26:20.113178+0000","last_fullsized":"2024-03-06T07:26:20.113178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:45.586349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113346+0000","last_change":"2024-03-06T07:26:20.109783+0000","last_active":"2024-03-06T07:26:20.113346+0000","last_peered":"2024-03-06T07:26:20.113346+0000","last_clean":"2024-03-06T07:26:20.113346+0000","last_became_active":"2024-03-06T07:26:20.109466+0000","last_became_peered":"2024-03-06T07:26:20.109466+0000","last_unstale":"2024-03-06T07:26:20.113346+0000","last_undegraded":"2024-03-06T07:26:20.113346+0000","last_fullsized":"2024-03-06T07:26:20.113346+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:13.865771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123079+0000","last_change":"2024-03-06T07:26:19.113658+0000","last_active":"2024-03-06T07:26:20.123079+0000","last_peered":"2024-03-06T07:26:20.123079+0000","last_clean":"2024-03-06T07:26:20.123079+0000","last_became_active":"2024-03-06T07:26:19.113388+0000","last_became_peered":"2024-03-06T07:26:19.113388+0000","last_unstale":"2024-03-06T07:26:20.123079+0000","last_undegraded":"2024-03-06T07:26:20.123079+0000","last_fullsized":"2024-03-06T07:26:20.123079+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:00.789303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113068+0000","last_change":"2024-03-06T07:26:20.103920+0000","last_active":"2024-03-06T07:26:20.113068+0000","last_peered":"2024-03-06T07:26:20.113068+0000","last_clean":"2024-03-06T07:26:20.113068+0000","last_became_active":"2024-03-06T07:26:20.103683+0000","last_became_peered":"2024-03-06T07:26:20.103683+0000","last_unstale":"2024-03-06T07:26:20.113068+0000","last_undegraded":"2024-03-06T07:26:20.113068+0000","last_fullsized":"2024-03-06T07:26:20.113068+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:26.692331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121604+0000","last_change":"2024-03-06T07:26:19.110988+0000","last_active":"2024-03-06T07:26:20.121604+0000","last_peered":"2024-03-06T07:26:20.121604+0000","last_clean":"2024-03-06T07:26:20.121604+0000","last_became_active":"2024-03-06T07:26:19.110672+0000","last_became_peered":"2024-03-06T07:26:19.110672+0000","last_unstale":"2024-03-06T07:26:20.121604+0000","last_undegraded":"2024-03-06T07:26:20.121604+0000","last_fullsized":"2024-03-06T07:26:20.121604+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:00.789303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123843+0000","last_change":"2024-03-06T07:26:20.120735+0000","last_active":"2024-03-06T07:26:20.123843+0000","last_peered":"2024-03-06T07:26:20.123843+0000","last_clean":"2024-03-06T07:26:20.123843+0000","last_became_active":"2024-03-06T07:26:20.120444+0000","last_became_peered":"2024-03-06T07:26:20.120444+0000","last_unstale":"2024-03-06T07:26:20.123843+0000","last_undegraded":"2024-03-06T07:26:20.123843+0000","last_fullsized":"2024-03-06T07:26:20.123843+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:37.277117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124193+0000","last_change":"2024-03-06T07:26:19.114482+0000","last_active":"2024-03-06T07:26:20.124193+0000","last_peered":"2024-03-06T07:26:20.124193+0000","last_clean":"2024-03-06T07:26:20.124193+0000","last_became_active":"2024-03-06T07:26:19.114178+0000","last_became_peered":"2024-03-06T07:26:19.114178+0000","last_unstale":"2024-03-06T07:26:20.124193+0000","last_undegraded":"2024-03-06T07:26:20.124193+0000","last_fullsized":"2024-03-06T07:26:20.124193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:32.777885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121165+0000","last_change":"2024-03-06T07:26:20.116993+0000","last_active":"2024-03-06T07:26:20.121165+0000","last_peered":"2024-03-06T07:26:20.121165+0000","last_clean":"2024-03-06T07:26:20.121165+0000","last_became_active":"2024-03-06T07:26:20.116744+0000","last_became_peered":"2024-03-06T07:26:20.116744+0000","last_unstale":"2024-03-06T07:26:20.121165+0000","last_undegraded":"2024-03-06T07:26:20.121165+0000","last_fullsized":"2024-03-06T07:26:20.121165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.870532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124328+0000","last_change":"2024-03-06T07:26:19.110182+0000","last_active":"2024-03-06T07:26:20.124328+0000","last_peered":"2024-03-06T07:26:20.124328+0000","last_clean":"2024-03-06T07:26:20.124328+0000","last_became_active":"2024-03-06T07:26:19.109898+0000","last_became_peered":"2024-03-06T07:26:19.109898+0000","last_unstale":"2024-03-06T07:26:20.124328+0000","last_undegraded":"2024-03-06T07:26:20.124328+0000","last_fullsized":"2024-03-06T07:26:20.124328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:56.576898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120935+0000","last_change":"2024-03-06T07:26:20.115512+0000","last_active":"2024-03-06T07:26:20.120935+0000","last_peered":"2024-03-06T07:26:20.120935+0000","last_clean":"2024-03-06T07:26:20.120935+0000","last_became_active":"2024-03-06T07:26:20.115195+0000","last_became_peered":"2024-03-06T07:26:20.115195+0000","last_unstale":"2024-03-06T07:26:20.120935+0000","last_undegraded":"2024-03-06T07:26:20.120935+0000","last_fullsized":"2024-03-06T07:26:20.120935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:57.979993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.119153+0000","last_change":"2024-03-06T07:26:19.107806+0000","last_active":"2024-03-06T07:26:20.119153+0000","last_peered":"2024-03-06T07:26:20.119153+0000","last_clean":"2024-03-06T07:26:20.119153+0000","last_became_active":"2024-03-06T07:26:19.107555+0000","last_became_peered":"2024-03-06T07:26:19.107555+0000","last_unstale":"2024-03-06T07:26:20.119153+0000","last_undegraded":"2024-03-06T07:26:20.119153+0000","last_fullsized":"2024-03-06T07:26:20.119153+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:17.147773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107072+0000","last_change":"2024-03-06T07:26:20.103519+0000","last_active":"2024-03-06T07:26:20.107072+0000","last_peered":"2024-03-06T07:26:20.107072+0000","last_clean":"2024-03-06T07:26:20.107072+0000","last_became_active":"2024-03-06T07:26:20.103251+0000","last_became_peered":"2024-03-06T07:26:20.103251+0000","last_unstale":"2024-03-06T07:26:20.107072+0000","last_undegraded":"2024-03-06T07:26:20.107072+0000","last_fullsized":"2024-03-06T07:26:20.107072+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:26.692331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124571+0000","last_change":"2024-03-06T07:26:19.108714+0000","last_active":"2024-03-06T07:26:20.124571+0000","last_peered":"2024-03-06T07:26:20.124571+0000","last_clean":"2024-03-06T07:26:20.124571+0000","last_became_active":"2024-03-06T07:26:19.108441+0000","last_became_peered":"2024-03-06T07:26:19.108441+0000","last_unstale":"2024-03-06T07:26:20.124571+0000","last_undegraded":"2024-03-06T07:26:20.124571+0000","last_fullsized":"2024-03-06T07:26:20.124571+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:22.183800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113583+0000","last_change":"2024-03-06T07:26:20.108907+0000","last_active":"2024-03-06T07:26:20.113583+0000","last_peered":"2024-03-06T07:26:20.113583+0000","last_clean":"2024-03-06T07:26:20.113583+0000","last_became_active":"2024-03-06T07:26:20.108196+0000","last_became_peered":"2024-03-06T07:26:20.108196+0000","last_unstale":"2024-03-06T07:26:20.113583+0000","last_undegraded":"2024-03-06T07:26:20.113583+0000","last_fullsized":"2024-03-06T07:26:20.113583+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:55.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.099874+0000","last_change":"2024-03-06T07:26:19.114206+0000","last_active":"2024-03-06T07:26:20.099874+0000","last_peered":"2024-03-06T07:26:20.099874+0000","last_clean":"2024-03-06T07:26:20.099874+0000","last_became_active":"2024-03-06T07:26:19.113991+0000","last_became_peered":"2024-03-06T07:26:19.113991+0000","last_unstale":"2024-03-06T07:26:20.099874+0000","last_undegraded":"2024-03-06T07:26:20.099874+0000","last_fullsized":"2024-03-06T07:26:20.099874+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:22.862479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118174+0000","last_change":"2024-03-06T07:26:20.110357+0000","last_active":"2024-03-06T07:26:20.118174+0000","last_peered":"2024-03-06T07:26:20.118174+0000","last_clean":"2024-03-06T07:26:20.118174+0000","last_became_active":"2024-03-06T07:26:20.110096+0000","last_became_peered":"2024-03-06T07:26:20.110096+0000","last_unstale":"2024-03-06T07:26:20.118174+0000","last_undegraded":"2024-03-06T07:26:20.118174+0000","last_fullsized":"2024-03-06T07:26:20.118174+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:57.979993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.106487+0000","last_change":"2024-03-06T07:26:19.112408+0000","last_active":"2024-03-06T07:26:20.106487+0000","last_peered":"2024-03-06T07:26:20.106487+0000","last_clean":"2024-03-06T07:26:20.106487+0000","last_became_active":"2024-03-06T07:26:19.112142+0000","last_became_peered":"2024-03-06T07:26:19.112142+0000","last_unstale":"2024-03-06T07:26:20.106487+0000","last_undegraded":"2024-03-06T07:26:20.106487+0000","last_fullsized":"2024-03-06T07:26:20.106487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:24.001055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121629+0000","last_change":"2024-03-06T07:26:20.116535+0000","last_active":"2024-03-06T07:26:20.121629+0000","last_peered":"2024-03-06T07:26:20.121629+0000","last_clean":"2024-03-06T07:26:20.121629+0000","last_became_active":"2024-03-06T07:26:20.116167+0000","last_became_peered":"2024-03-06T07:26:20.116167+0000","last_unstale":"2024-03-06T07:26:20.121629+0000","last_undegraded":"2024-03-06T07:26:20.121629+0000","last_fullsized":"2024-03-06T07:26:20.121629+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:33.335668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.108368+0000","last_change":"2024-03-06T07:26:19.104265+0000","last_active":"2024-03-06T07:26:20.108368+0000","last_peered":"2024-03-06T07:26:20.108368+0000","last_clean":"2024-03-06T07:26:20.108368+0000","last_became_active":"2024-03-06T07:26:19.103989+0000","last_became_peered":"2024-03-06T07:26:19.103989+0000","last_unstale":"2024-03-06T07:26:20.108368+0000","last_undegraded":"2024-03-06T07:26:20.108368+0000","last_fullsized":"2024-03-06T07:26:20.108368+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:42.973670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120128+0000","last_change":"2024-03-06T07:26:20.112073+0000","last_active":"2024-03-06T07:26:20.120128+0000","last_peered":"2024-03-06T07:26:20.120128+0000","last_clean":"2024-03-06T07:26:20.120128+0000","last_became_active":"2024-03-06T07:26:20.111778+0000","last_became_peered":"2024-03-06T07:26:20.111778+0000","last_unstale":"2024-03-06T07:26:20.120128+0000","last_undegraded":"2024-03-06T07:26:20.120128+0000","last_fullsized":"2024-03-06T07:26:20.120128+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:21.882258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.095249+0000","last_change":"2024-03-06T07:26:19.110192+0000","last_active":"2024-03-06T07:26:20.095249+0000","last_peered":"2024-03-06T07:26:20.095249+0000","last_clean":"2024-03-06T07:26:20.095249+0000","last_became_active":"2024-03-06T07:26:19.109927+0000","last_became_peered":"2024-03-06T07:26:19.109927+0000","last_unstale":"2024-03-06T07:26:20.095249+0000","last_undegraded":"2024-03-06T07:26:20.095249+0000","last_fullsized":"2024-03-06T07:26:20.095249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:34.827670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110356+0000","last_change":"2024-03-06T07:26:20.108055+0000","last_active":"2024-03-06T07:26:20.110356+0000","last_peered":"2024-03-06T07:26:20.110356+0000","last_clean":"2024-03-06T07:26:20.110356+0000","last_became_active":"2024-03-06T07:26:20.107783+0000","last_became_peered":"2024-03-06T07:26:20.107783+0000","last_unstale":"2024-03-06T07:26:20.110356+0000","last_undegraded":"2024-03-06T07:26:20.110356+0000","last_fullsized":"2024-03-06T07:26:20.110356+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:02.917862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124511+0000","last_change":"2024-03-06T07:26:19.109632+0000","last_active":"2024-03-06T07:26:20.124511+0000","last_peered":"2024-03-06T07:26:20.124511+0000","last_clean":"2024-03-06T07:26:20.124511+0000","last_became_active":"2024-03-06T07:26:19.109342+0000","last_became_peered":"2024-03-06T07:26:19.109342+0000","last_unstale":"2024-03-06T07:26:20.124511+0000","last_undegraded":"2024-03-06T07:26:20.124511+0000","last_fullsized":"2024-03-06T07:26:20.124511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:56.576898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118064+0000","last_change":"2024-03-06T07:26:20.112776+0000","last_active":"2024-03-06T07:26:20.118064+0000","last_peered":"2024-03-06T07:26:20.118064+0000","last_clean":"2024-03-06T07:26:20.118064+0000","last_became_active":"2024-03-06T07:26:20.112480+0000","last_became_peered":"2024-03-06T07:26:20.112480+0000","last_unstale":"2024-03-06T07:26:20.118064+0000","last_undegraded":"2024-03-06T07:26:20.118064+0000","last_fullsized":"2024-03-06T07:26:20.118064+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:09.734012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.128420+0000","last_change":"2024-03-06T07:26:19.108018+0000","last_active":"2024-03-06T07:26:20.128420+0000","last_peered":"2024-03-06T07:26:20.128420+0000","last_clean":"2024-03-06T07:26:20.128420+0000","last_became_active":"2024-03-06T07:26:19.107638+0000","last_became_peered":"2024-03-06T07:26:19.107638+0000","last_unstale":"2024-03-06T07:26:20.128420+0000","last_undegraded":"2024-03-06T07:26:20.128420+0000","last_fullsized":"2024-03-06T07:26:20.128420+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:21.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113457+0000","last_change":"2024-03-06T07:26:20.106544+0000","last_active":"2024-03-06T07:26:20.113457+0000","last_peered":"2024-03-06T07:26:20.113457+0000","last_clean":"2024-03-06T07:26:20.113457+0000","last_became_active":"2024-03-06T07:26:20.106307+0000","last_became_peered":"2024-03-06T07:26:20.106307+0000","last_unstale":"2024-03-06T07:26:20.113457+0000","last_undegraded":"2024-03-06T07:26:20.113457+0000","last_fullsized":"2024-03-06T07:26:20.113457+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:25.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124281+0000","last_change":"2024-03-06T07:26:19.112397+0000","last_active":"2024-03-06T07:26:20.124281+0000","last_peered":"2024-03-06T07:26:20.124281+0000","last_clean":"2024-03-06T07:26:20.124281+0000","last_became_active":"2024-03-06T07:26:19.112023+0000","last_became_peered":"2024-03-06T07:26:19.112023+0000","last_unstale":"2024-03-06T07:26:20.124281+0000","last_undegraded":"2024-03-06T07:26:20.124281+0000","last_fullsized":"2024-03-06T07:26:20.124281+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:46.549394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113045+0000","last_change":"2024-03-06T07:26:20.110618+0000","last_active":"2024-03-06T07:26:20.113045+0000","last_peered":"2024-03-06T07:26:20.113045+0000","last_clean":"2024-03-06T07:26:20.113045+0000","last_became_active":"2024-03-06T07:26:20.110377+0000","last_became_peered":"2024-03-06T07:26:20.110377+0000","last_unstale":"2024-03-06T07:26:20.113045+0000","last_undegraded":"2024-03-06T07:26:20.113045+0000","last_fullsized":"2024-03-06T07:26:20.113045+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:21.347163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.099172+0000","last_change":"2024-03-06T07:26:19.113525+0000","last_active":"2024-03-06T07:26:20.099172+0000","last_peered":"2024-03-06T07:26:20.099172+0000","last_clean":"2024-03-06T07:26:20.099172+0000","last_became_active":"2024-03-06T07:26:19.113249+0000","last_became_peered":"2024-03-06T07:26:19.113249+0000","last_unstale":"2024-03-06T07:26:20.099172+0000","last_undegraded":"2024-03-06T07:26:20.099172+0000","last_fullsized":"2024-03-06T07:26:20.099172+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:28.131529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118249+0000","last_change":"2024-03-06T07:26:20.111248+0000","last_active":"2024-03-06T07:26:20.118249+0000","last_peered":"2024-03-06T07:26:20.118249+0000","last_clean":"2024-03-06T07:26:20.118249+0000","last_became_active":"2024-03-06T07:26:20.110952+0000","last_became_peered":"2024-03-06T07:26:20.110952+0000","last_unstale":"2024-03-06T07:26:20.118249+0000","last_undegraded":"2024-03-06T07:26:20.118249+0000","last_fullsized":"2024-03-06T07:26:20.118249+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:17.103413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.106497+0000","last_change":"2024-03-06T07:26:19.107891+0000","last_active":"2024-03-06T07:26:20.106497+0000","last_peered":"2024-03-06T07:26:20.106497+0000","last_clean":"2024-03-06T07:26:20.106497+0000","last_became_active":"2024-03-06T07:26:19.107600+0000","last_became_peered":"2024-03-06T07:26:19.107600+0000","last_unstale":"2024-03-06T07:26:20.106497+0000","last_undegraded":"2024-03-06T07:26:20.106497+0000","last_fullsized":"2024-03-06T07:26:20.106497+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:44.054462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110551+0000","last_change":"2024-03-06T07:26:20.105866+0000","last_active":"2024-03-06T07:26:20.110551+0000","last_peered":"2024-03-06T07:26:20.110551+0000","last_clean":"2024-03-06T07:26:20.110551+0000","last_became_active":"2024-03-06T07:26:20.105594+0000","last_became_peered":"2024-03-06T07:26:20.105594+0000","last_unstale":"2024-03-06T07:26:20.110551+0000","last_undegraded":"2024-03-06T07:26:20.110551+0000","last_fullsized":"2024-03-06T07:26:20.110551+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:35.862436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"92'8","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.125008+0000","last_change":"2024-03-06T07:26:19.110099+0000","last_active":"2024-03-06T07:26:20.125008+0000","last_peered":"2024-03-06T07:26:20.125008+0000","last_clean":"2024-03-06T07:26:20.125008+0000","last_became_active":"2024-03-06T07:26:19.109815+0000","last_became_peered":"2024-03-06T07:26:19.109815+0000","last_unstale":"2024-03-06T07:26:20.125008+0000","last_undegraded":"2024-03-06T07:26:20.125008+0000","last_fullsized":"2024-03-06T07:26:20.125008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:20.128480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.116321+0000","last_change":"2024-03-06T07:26:20.114001+0000","last_active":"2024-03-06T07:26:20.116321+0000","last_peered":"2024-03-06T07:26:20.116321+0000","last_clean":"2024-03-06T07:26:20.116321+0000","last_became_active":"2024-03-06T07:26:20.113033+0000","last_became_peered":"2024-03-06T07:26:20.113033+0000","last_unstale":"2024-03-06T07:26:20.116321+0000","last_undegraded":"2024-03-06T07:26:20.116321+0000","last_fullsized":"2024-03-06T07:26:20.116321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:51.214905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124347+0000","last_change":"2024-03-06T07:26:19.110045+0000","last_active":"2024-03-06T07:26:20.124347+0000","last_peered":"2024-03-06T07:26:20.124347+0000","last_clean":"2024-03-06T07:26:20.124347+0000","last_became_active":"2024-03-06T07:26:19.109745+0000","last_became_peered":"2024-03-06T07:26:19.109745+0000","last_unstale":"2024-03-06T07:26:20.124347+0000","last_undegraded":"2024-03-06T07:26:20.124347+0000","last_fullsized":"2024-03-06T07:26:20.124347+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:31.048107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117638+0000","last_change":"2024-03-06T07:26:20.110006+0000","last_active":"2024-03-06T07:26:20.117638+0000","last_peered":"2024-03-06T07:26:20.117638+0000","last_clean":"2024-03-06T07:26:20.117638+0000","last_became_active":"2024-03-06T07:26:20.109702+0000","last_became_peered":"2024-03-06T07:26:20.109702+0000","last_unstale":"2024-03-06T07:26:20.117638+0000","last_undegraded":"2024-03-06T07:26:20.117638+0000","last_fullsized":"2024-03-06T07:26:20.117638+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:41.233378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123886+0000","last_change":"2024-03-06T07:26:19.109958+0000","last_active":"2024-03-06T07:26:20.123886+0000","last_peered":"2024-03-06T07:26:20.123886+0000","last_clean":"2024-03-06T07:26:20.123886+0000","last_became_active":"2024-03-06T07:26:19.109672+0000","last_became_peered":"2024-03-06T07:26:19.109672+0000","last_unstale":"2024-03-06T07:26:20.123886+0000","last_undegraded":"2024-03-06T07:26:20.123886+0000","last_fullsized":"2024-03-06T07:26:20.123886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:19.101303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120503+0000","last_change":"2024-03-06T07:26:20.113947+0000","last_active":"2024-03-06T07:26:20.120503+0000","last_peered":"2024-03-06T07:26:20.120503+0000","last_clean":"2024-03-06T07:26:20.120503+0000","last_became_active":"2024-03-06T07:26:20.113465+0000","last_became_peered":"2024-03-06T07:26:20.113465+0000","last_unstale":"2024-03-06T07:26:20.120503+0000","last_undegraded":"2024-03-06T07:26:20.120503+0000","last_fullsized":"2024-03-06T07:26:20.120503+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:08.325075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.103165+0000","last_change":"2024-03-06T07:26:19.110420+0000","last_active":"2024-03-06T07:26:20.103165+0000","last_peered":"2024-03-06T07:26:20.103165+0000","last_clean":"2024-03-06T07:26:20.103165+0000","last_became_active":"2024-03-06T07:26:19.109525+0000","last_became_peered":"2024-03-06T07:26:19.109525+0000","last_unstale":"2024-03-06T07:26:20.103165+0000","last_undegraded":"2024-03-06T07:26:20.103165+0000","last_fullsized":"2024-03-06T07:26:20.103165+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:33.896767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109518+0000","last_change":"2024-03-06T07:26:20.104809+0000","last_active":"2024-03-06T07:26:20.109518+0000","last_peered":"2024-03-06T07:26:20.109518+0000","last_clean":"2024-03-06T07:26:20.109518+0000","last_became_active":"2024-03-06T07:26:20.104567+0000","last_became_peered":"2024-03-06T07:26:20.104567+0000","last_unstale":"2024-03-06T07:26:20.109518+0000","last_undegraded":"2024-03-06T07:26:20.109518+0000","last_fullsized":"2024-03-06T07:26:20.109518+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:22.278065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124100+0000","last_change":"2024-03-06T07:26:19.108073+0000","last_active":"2024-03-06T07:26:20.124100+0000","last_peered":"2024-03-06T07:26:20.124100+0000","last_clean":"2024-03-06T07:26:20.124100+0000","last_became_active":"2024-03-06T07:26:19.107792+0000","last_became_peered":"2024-03-06T07:26:19.107792+0000","last_unstale":"2024-03-06T07:26:20.124100+0000","last_undegraded":"2024-03-06T07:26:20.124100+0000","last_fullsized":"2024-03-06T07:26:20.124100+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:25.684278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109499+0000","last_change":"2024-03-06T07:26:20.105001+0000","last_active":"2024-03-06T07:26:20.109499+0000","last_peered":"2024-03-06T07:26:20.109499+0000","last_clean":"2024-03-06T07:26:20.109499+0000","last_became_active":"2024-03-06T07:26:20.104717+0000","last_became_peered":"2024-03-06T07:26:20.104717+0000","last_unstale":"2024-03-06T07:26:20.109499+0000","last_undegraded":"2024-03-06T07:26:20.109499+0000","last_fullsized":"2024-03-06T07:26:20.109499+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:52.743978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101487+0000","last_change":"2024-03-06T07:26:19.108061+0000","last_active":"2024-03-06T07:26:20.101487+0000","last_peered":"2024-03-06T07:26:20.101487+0000","last_clean":"2024-03-06T07:26:20.101487+0000","last_became_active":"2024-03-06T07:26:19.107764+0000","last_became_peered":"2024-03-06T07:26:19.107764+0000","last_unstale":"2024-03-06T07:26:20.101487+0000","last_undegraded":"2024-03-06T07:26:20.101487+0000","last_fullsized":"2024-03-06T07:26:20.101487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:31.048107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109584+0000","last_change":"2024-03-06T07:26:20.104546+0000","last_active":"2024-03-06T07:26:20.109584+0000","last_peered":"2024-03-06T07:26:20.109584+0000","last_clean":"2024-03-06T07:26:20.109584+0000","last_became_active":"2024-03-06T07:26:20.104281+0000","last_became_peered":"2024-03-06T07:26:20.104281+0000","last_unstale":"2024-03-06T07:26:20.109584+0000","last_undegraded":"2024-03-06T07:26:20.109584+0000","last_fullsized":"2024-03-06T07:26:20.109584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:11.193272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101395+0000","last_change":"2024-03-06T07:26:19.105855+0000","last_active":"2024-03-06T07:26:20.101395+0000","last_peered":"2024-03-06T07:26:20.101395+0000","last_clean":"2024-03-06T07:26:20.101395+0000","last_became_active":"2024-03-06T07:26:19.105503+0000","last_became_peered":"2024-03-06T07:26:19.105503+0000","last_unstale":"2024-03-06T07:26:20.101395+0000","last_undegraded":"2024-03-06T07:26:20.101395+0000","last_fullsized":"2024-03-06T07:26:20.101395+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:44.054462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107199+0000","last_change":"2024-03-06T07:26:20.104890+0000","last_active":"2024-03-06T07:26:20.107199+0000","last_peered":"2024-03-06T07:26:20.107199+0000","last_clean":"2024-03-06T07:26:20.107199+0000","last_became_active":"2024-03-06T07:26:20.104613+0000","last_became_peered":"2024-03-06T07:26:20.104613+0000","last_unstale":"2024-03-06T07:26:20.107199+0000","last_undegraded":"2024-03-06T07:26:20.107199+0000","last_fullsized":"2024-03-06T07:26:20.107199+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.977117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101439+0000","last_change":"2024-03-06T07:26:19.102520+0000","last_active":"2024-03-06T07:26:20.101439+0000","last_peered":"2024-03-06T07:26:20.101439+0000","last_clean":"2024-03-06T07:26:20.101439+0000","last_became_active":"2024-03-06T07:26:19.102212+0000","last_became_peered":"2024-03-06T07:26:19.102212+0000","last_unstale":"2024-03-06T07:26:20.101439+0000","last_undegraded":"2024-03-06T07:26:20.101439+0000","last_fullsized":"2024-03-06T07:26:20.101439+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:09.370568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123995+0000","last_change":"2024-03-06T07:26:20.118894+0000","last_active":"2024-03-06T07:26:20.123995+0000","last_peered":"2024-03-06T07:26:20.123995+0000","last_clean":"2024-03-06T07:26:20.123995+0000","last_became_active":"2024-03-06T07:26:20.118636+0000","last_became_peered":"2024-03-06T07:26:20.118636+0000","last_unstale":"2024-03-06T07:26:20.123995+0000","last_undegraded":"2024-03-06T07:26:20.123995+0000","last_fullsized":"2024-03-06T07:26:20.123995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:55.482297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124277+0000","last_change":"2024-03-06T07:26:19.110031+0000","last_active":"2024-03-06T07:26:20.124277+0000","last_peered":"2024-03-06T07:26:20.124277+0000","last_clean":"2024-03-06T07:26:20.124277+0000","last_became_active":"2024-03-06T07:26:19.109745+0000","last_became_peered":"2024-03-06T07:26:19.109745+0000","last_unstale":"2024-03-06T07:26:20.124277+0000","last_undegraded":"2024-03-06T07:26:20.124277+0000","last_fullsized":"2024-03-06T07:26:20.124277+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:21.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.111523+0000","last_change":"2024-03-06T07:26:20.107533+0000","last_active":"2024-03-06T07:26:20.111523+0000","last_peered":"2024-03-06T07:26:20.111523+0000","last_clean":"2024-03-06T07:26:20.111523+0000","last_became_active":"2024-03-06T07:26:20.106910+0000","last_became_peered":"2024-03-06T07:26:20.106910+0000","last_unstale":"2024-03-06T07:26:20.111523+0000","last_undegraded":"2024-03-06T07:26:20.111523+0000","last_fullsized":"2024-03-06T07:26:20.111523+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:53.479826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.104740+0000","last_change":"2024-03-06T07:26:19.107899+0000","last_active":"2024-03-06T07:26:20.104740+0000","last_peered":"2024-03-06T07:26:20.104740+0000","last_clean":"2024-03-06T07:26:20.104740+0000","last_became_active":"2024-03-06T07:26:19.107531+0000","last_became_peered":"2024-03-06T07:26:19.107531+0000","last_unstale":"2024-03-06T07:26:20.104740+0000","last_undegraded":"2024-03-06T07:26:20.104740+0000","last_fullsized":"2024-03-06T07:26:20.104740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:36.157182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.111454+0000","last_change":"2024-03-06T07:26:20.107330+0000","last_active":"2024-03-06T07:26:20.111454+0000","last_peered":"2024-03-06T07:26:20.111454+0000","last_clean":"2024-03-06T07:26:20.111454+0000","last_became_active":"2024-03-06T07:26:20.106877+0000","last_became_peered":"2024-03-06T07:26:20.106877+0000","last_unstale":"2024-03-06T07:26:20.111454+0000","last_undegraded":"2024-03-06T07:26:20.111454+0000","last_fullsized":"2024-03-06T07:26:20.111454+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:10.505443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101580+0000","last_change":"2024-03-06T07:26:19.111449+0000","last_active":"2024-03-06T07:26:20.101580+0000","last_peered":"2024-03-06T07:26:20.101580+0000","last_clean":"2024-03-06T07:26:20.101580+0000","last_became_active":"2024-03-06T07:26:19.110513+0000","last_became_peered":"2024-03-06T07:26:19.110513+0000","last_unstale":"2024-03-06T07:26:20.101580+0000","last_undegraded":"2024-03-06T07:26:20.101580+0000","last_fullsized":"2024-03-06T07:26:20.101580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:34.827670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117508+0000","last_change":"2024-03-06T07:26:20.115860+0000","last_active":"2024-03-06T07:26:20.117508+0000","last_peered":"2024-03-06T07:26:20.117508+0000","last_clean":"2024-03-06T07:26:20.117508+0000","last_became_active":"2024-03-06T07:26:20.115536+0000","last_became_peered":"2024-03-06T07:26:20.115536+0000","last_unstale":"2024-03-06T07:26:20.117508+0000","last_undegraded":"2024-03-06T07:26:20.117508+0000","last_fullsized":"2024-03-06T07:26:20.117508+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:33.785937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.096942+0000","last_change":"2024-03-06T07:26:19.109180+0000","last_active":"2024-03-06T07:26:20.096942+0000","last_peered":"2024-03-06T07:26:20.096942+0000","last_clean":"2024-03-06T07:26:20.096942+0000","last_became_active":"2024-03-06T07:26:19.108881+0000","last_became_peered":"2024-03-06T07:26:19.108881+0000","last_unstale":"2024-03-06T07:26:20.096942+0000","last_undegraded":"2024-03-06T07:26:20.096942+0000","last_fullsized":"2024-03-06T07:26:20.096942+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:20.874205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113693+0000","last_change":"2024-03-06T07:26:20.108331+0000","last_active":"2024-03-06T07:26:20.113693+0000","last_peered":"2024-03-06T07:26:20.113693+0000","last_clean":"2024-03-06T07:26:20.113693+0000","last_became_active":"2024-03-06T07:26:20.107429+0000","last_became_peered":"2024-03-06T07:26:20.107429+0000","last_unstale":"2024-03-06T07:26:20.113693+0000","last_undegraded":"2024-03-06T07:26:20.113693+0000","last_fullsized":"2024-03-06T07:26:20.113693+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:01.787165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.103911+0000","last_change":"2024-03-06T07:26:19.103430+0000","last_active":"2024-03-06T07:26:20.103911+0000","last_peered":"2024-03-06T07:26:20.103911+0000","last_clean":"2024-03-06T07:26:20.103911+0000","last_became_active":"2024-03-06T07:26:19.102977+0000","last_became_peered":"2024-03-06T07:26:19.102977+0000","last_unstale":"2024-03-06T07:26:20.103911+0000","last_undegraded":"2024-03-06T07:26:20.103911+0000","last_fullsized":"2024-03-06T07:26:20.103911+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:23.076499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118300+0000","last_change":"2024-03-06T07:26:20.111066+0000","last_active":"2024-03-06T07:26:20.118300+0000","last_peered":"2024-03-06T07:26:20.118300+0000","last_clean":"2024-03-06T07:26:20.118300+0000","last_became_active":"2024-03-06T07:26:20.110753+0000","last_became_peered":"2024-03-06T07:26:20.110753+0000","last_unstale":"2024-03-06T07:26:20.118300+0000","last_undegraded":"2024-03-06T07:26:20.118300+0000","last_fullsized":"2024-03-06T07:26:20.118300+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:09.647399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.096039+0000","last_change":"2024-03-06T07:26:19.110996+0000","last_active":"2024-03-06T07:26:20.096039+0000","last_peered":"2024-03-06T07:26:20.096039+0000","last_clean":"2024-03-06T07:26:20.096039+0000","last_became_active":"2024-03-06T07:26:19.110738+0000","last_became_peered":"2024-03-06T07:26:19.110738+0000","last_unstale":"2024-03-06T07:26:20.096039+0000","last_undegraded":"2024-03-06T07:26:20.096039+0000","last_fullsized":"2024-03-06T07:26:20.096039+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:16.095361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110583+0000","last_change":"2024-03-06T07:26:20.107196+0000","last_active":"2024-03-06T07:26:20.110583+0000","last_peered":"2024-03-06T07:26:20.110583+0000","last_clean":"2024-03-06T07:26:20.110583+0000","last_became_active":"2024-03-06T07:26:20.106251+0000","last_became_peered":"2024-03-06T07:26:20.106251+0000","last_unstale":"2024-03-06T07:26:20.110583+0000","last_undegraded":"2024-03-06T07:26:20.110583+0000","last_fullsized":"2024-03-06T07:26:20.110583+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:35.016702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.095385+0000","last_change":"2024-03-06T07:26:19.109512+0000","last_active":"2024-03-06T07:26:20.095385+0000","last_peered":"2024-03-06T07:26:20.095385+0000","last_clean":"2024-03-06T07:26:20.095385+0000","last_became_active":"2024-03-06T07:26:19.109004+0000","last_became_peered":"2024-03-06T07:26:19.109004+0000","last_unstale":"2024-03-06T07:26:20.095385+0000","last_undegraded":"2024-03-06T07:26:20.095385+0000","last_fullsized":"2024-03-06T07:26:20.095385+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:00.789303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117709+0000","last_change":"2024-03-06T07:26:20.115910+0000","last_active":"2024-03-06T07:26:20.117709+0000","last_peered":"2024-03-06T07:26:20.117709+0000","last_clean":"2024-03-06T07:26:20.117709+0000","last_became_active":"2024-03-06T07:26:20.115620+0000","last_became_peered":"2024-03-06T07:26:20.115620+0000","last_unstale":"2024-03-06T07:26:20.117709+0000","last_undegraded":"2024-03-06T07:26:20.117709+0000","last_fullsized":"2024-03-06T07:26:20.117709+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:10.378621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.098956+0000","last_change":"2024-03-06T07:26:19.106142+0000","last_active":"2024-03-06T07:26:20.098956+0000","last_peered":"2024-03-06T07:26:20.098956+0000","last_clean":"2024-03-06T07:26:20.098956+0000","last_became_active":"2024-03-06T07:26:19.105867+0000","last_became_peered":"2024-03-06T07:26:19.105867+0000","last_unstale":"2024-03-06T07:26:20.098956+0000","last_undegraded":"2024-03-06T07:26:20.098956+0000","last_fullsized":"2024-03-06T07:26:20.098956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:37.557448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109674+0000","last_change":"2024-03-06T07:26:20.103753+0000","last_active":"2024-03-06T07:26:20.109674+0000","last_peered":"2024-03-06T07:26:20.109674+0000","last_clean":"2024-03-06T07:26:20.109674+0000","last_became_active":"2024-03-06T07:26:20.103483+0000","last_became_peered":"2024-03-06T07:26:20.103483+0000","last_unstale":"2024-03-06T07:26:20.109674+0000","last_undegraded":"2024-03-06T07:26:20.109674+0000","last_fullsized":"2024-03-06T07:26:20.109674+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:52.563008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121429+0000","last_change":"2024-03-06T07:26:19.112799+0000","last_active":"2024-03-06T07:26:20.121429+0000","last_peered":"2024-03-06T07:26:20.121429+0000","last_clean":"2024-03-06T07:26:20.121429+0000","last_became_active":"2024-03-06T07:26:19.112583+0000","last_became_peered":"2024-03-06T07:26:19.112583+0000","last_unstale":"2024-03-06T07:26:20.121429+0000","last_undegraded":"2024-03-06T07:26:20.121429+0000","last_fullsized":"2024-03-06T07:26:20.121429+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:33.896767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":305,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.096395+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:26:20.096395+0000","last_peered":"2024-03-06T07:26:20.096395+0000","last_clean":"2024-03-06T07:26:20.096395+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:26:20.096395+0000","last_undegraded":"2024-03-06T07:26:20.096395+0000","last_fullsized":"2024-03-06T07:26:20.096395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1740800,"data_stored":1075219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1736704,"data_stored":1075689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:23.065 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:23.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.200+0000 7f6806a42640 1 Processor -- start 2024-03-06T07:26:23.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.200+0000 7f6806a42640 1 -- start start 2024-03-06T07:26:23.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f6806a42640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68001515e0 0x7f68000cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f6806a42640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800071f40 con 0x7f680015b430 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f6806a42640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680015b8b0 con 0x7f68000cd7a0 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f6806a42640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6800070a70 con 0x7f68001515e0 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f67ff7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f680015b430 0x7f6800071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50402/0 (socket says 172.21.15.122:50402) 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f67ff7fe640 1 -- 172.21.15.122:0/1363509978 learned_addr learned my addr 172.21.15.122:0/1363509978 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f67fffff640 1 --2- 172.21.15.122:0/1363509978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68001515e0 0x7f68000cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.201+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344911969 0 0) 0x7f6800071f40 con 0x7f680015b430 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e8003520 con 0x7f680015b430 2024-03-06T07:26:23.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089230487 0 0) 0x7f680015b8b0 con 0x7f68000cd7a0 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6800071f40 con 0x7f68000cd7a0 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67fffff640 1 -- 172.21.15.122:0/1363509978 >> v1:172.21.15.169:6789/0 conn(0x7f68000cd7a0 legacy=0x7f68000cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67fffff640 1 -- 172.21.15.122:0/1363509978 >> v1:172.21.15.122:6789/0 conn(0x7f680015b430 legacy=0x7f6800071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67fffff640 1 -- 172.21.15.122:0/1363509978 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800070d50 con 0x7f68001515e0 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.202+0000 7f67fffff640 1 --2- 172.21.15.122:0/1363509978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68001515e0 0x7f68000cce40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67f000a970 tx=0x7f67f002df00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d495115b3afc417f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f003b070 con 0x7f68001515e0 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f002ecb0 con 0x7f68001515e0 2024-03-06T07:26:23.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f67feffd640 1 -- 172.21.15.122:0/1363509978 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f000b040 con 0x7f68001515e0 2024-03-06T07:26:23.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 -- 172.21.15.122:0/1363509978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68001515e0 msgr2=0x7f68000cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 --2- 172.21.15.122:0/1363509978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68001515e0 0x7f68000cce40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67f000a970 tx=0x7f67f002df00 comp rx=0 tx=0).stop 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 -- 172.21.15.122:0/1363509978 shutdown_connections 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 --2- 172.21.15.122:0/1363509978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68001515e0 0x7f68000cce40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 -- 172.21.15.122:0/1363509978 >> 172.21.15.122:0/1363509978 conn(0x7f68000cf8a0 msgr2=0x7f68000cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 -- 172.21.15.122:0/1363509978 shutdown_connections 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.203+0000 7f6806a42640 1 -- 172.21.15.122:0/1363509978 wait complete. 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.204+0000 7f6806a42640 1 Processor -- start 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.204+0000 7f6806a42640 1 -- start start 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6806a42640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68000cd7a0 0x7f68000ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6806a42640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 0x7f680013e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6806a42640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f680015b430 0x7f68000c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6806a42640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000cc3a0 con 0x7f68001515e0 2024-03-06T07:26:23.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6806a42640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800071310 con 0x7f680015b430 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6806a42640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000cc520 con 0x7f68000cd7a0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f67ff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 0x7f680013e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f67ff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 0x7f680013e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56720/0 (socket says 172.21.15.122:56720) 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f67ff7fe640 1 -- 172.21.15.122:0/2356762721 learned_addr learned my addr 172.21.15.122:0/2356762721 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f67fffff640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68000cd7a0 0x7f68000ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.205+0000 7f6804fdf640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f680015b430 0x7f68000c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67ff7fe640 1 -- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68000cd7a0 msgr2=0x7f68000ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67ff7fe640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68000cd7a0 0x7f68000ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67ff7fe640 1 -- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f680015b430 msgr2=0x7f68000c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67ff7fe640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f680015b430 0x7f68000c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67ff7fe640 1 -- 172.21.15.122:0/2356762721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680013eb50 con 0x7f68001515e0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67ff7fe640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 0x7f680013e610 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c410 tx=0x7f67f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4017070 con 0x7f68001515e0 2024-03-06T07:26:23.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4005ce0 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68001f0780 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f40072f0 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.206+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680006cce0 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.207+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680006d4b0 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.207+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67f401a070 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.208+0000 7f67fcff9640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.208+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67f4089560 con 0x7f68001515e0 2024-03-06T07:26:23.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.208+0000 7f67fffff640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.208+0000 7f67fffff640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f67ec06b4d0 0x7f67ec06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f67f0038000 tx=0x7f67f0032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.212+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67f4057ee0 con 0x7f68001515e0 2024-03-06T07:26:23.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.380+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6800068840 con 0x7f67ec06b4d0 2024-03-06T07:26:23.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.386+0000 7f67fcff9640 1 -- 172.21.15.122:0/2356762721 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252013 (secure 0 0 0) 0x7f67f0084880 con 0x7f67ec06b4d0 2024-03-06T07:26:23.390 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:23.392 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f67ec06b4d0 msgr2=0x7f67ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f67ec06b4d0 0x7f67ec06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f67f0038000 tx=0x7f67f0032000 comp rx=0 tx=0).stop 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 msgr2=0x7f680013e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 0x7f680013e610 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c410 tx=0x7f67f400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 shutdown_connections 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f680015b430 0x7f68000c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68001515e0 0x7f680013e610 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 --2- 172.21.15.122:0/2356762721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f68000cd7a0 0x7f68000ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 >> 172.21.15.122:0/2356762721 conn(0x7f68000cf8a0 msgr2=0x7f680014e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 shutdown_connections 2024-03-06T07:26:23.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.389+0000 7f6806a42640 1 -- 172.21.15.122:0/2356762721 wait complete. 2024-03-06T07:26:23.402 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-03-06T07:26:21.383242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259452,"kb_used_data":9996,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715268,"statfs":{"total":773094113280,"available":772828434432,"internally_reserved":0,"allocated":10235904,"data_stored":5018692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000639"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.119217+0000","last_change":"2024-03-06T07:26:20.115502+0000","last_active":"2024-03-06T07:26:20.119217+0000","last_peered":"2024-03-06T07:26:20.119217+0000","last_clean":"2024-03-06T07:26:20.119217+0000","last_became_active":"2024-03-06T07:26:20.115270+0000","last_became_peered":"2024-03-06T07:26:20.115270+0000","last_unstale":"2024-03-06T07:26:20.119217+0000","last_undegraded":"2024-03-06T07:26:20.119217+0000","last_fullsized":"2024-03-06T07:26:20.119217+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:02.362057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123975+0000","last_change":"2024-03-06T07:26:20.118442+0000","last_active":"2024-03-06T07:26:20.123975+0000","last_peered":"2024-03-06T07:26:20.123975+0000","last_clean":"2024-03-06T07:26:20.123975+0000","last_became_active":"2024-03-06T07:26:20.118191+0000","last_became_peered":"2024-03-06T07:26:20.118191+0000","last_unstale":"2024-03-06T07:26:20.123975+0000","last_undegraded":"2024-03-06T07:26:20.123975+0000","last_fullsized":"2024-03-06T07:26:20.123975+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:13.090726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.119101+0000","last_change":"2024-03-06T07:26:20.114809+0000","last_active":"2024-03-06T07:26:20.119101+0000","last_peered":"2024-03-06T07:26:20.119101+0000","last_clean":"2024-03-06T07:26:20.119101+0000","last_became_active":"2024-03-06T07:26:20.114535+0000","last_became_peered":"2024-03-06T07:26:20.114535+0000","last_unstale":"2024-03-06T07:26:20.119101+0000","last_undegraded":"2024-03-06T07:26:20.119101+0000","last_fullsized":"2024-03-06T07:26:20.119101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:17.638953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110669+0000","last_change":"2024-03-06T07:26:20.100026+0000","last_active":"2024-03-06T07:26:20.110669+0000","last_peered":"2024-03-06T07:26:20.110669+0000","last_clean":"2024-03-06T07:26:20.110669+0000","last_became_active":"2024-03-06T07:26:20.099710+0000","last_became_peered":"2024-03-06T07:26:20.099710+0000","last_unstale":"2024-03-06T07:26:20.110669+0000","last_undegraded":"2024-03-06T07:26:20.110669+0000","last_fullsized":"2024-03-06T07:26:20.110669+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:57.584951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118104+0000","last_change":"2024-03-06T07:26:20.115782+0000","last_active":"2024-03-06T07:26:20.118104+0000","last_peered":"2024-03-06T07:26:20.118104+0000","last_clean":"2024-03-06T07:26:20.118104+0000","last_became_active":"2024-03-06T07:26:20.115459+0000","last_became_peered":"2024-03-06T07:26:20.115459+0000","last_unstale":"2024-03-06T07:26:20.118104+0000","last_undegraded":"2024-03-06T07:26:20.118104+0000","last_fullsized":"2024-03-06T07:26:20.118104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:17.269474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117759+0000","last_change":"2024-03-06T07:26:20.110804+0000","last_active":"2024-03-06T07:26:20.117759+0000","last_peered":"2024-03-06T07:26:20.117759+0000","last_clean":"2024-03-06T07:26:20.117759+0000","last_became_active":"2024-03-06T07:26:20.110506+0000","last_became_peered":"2024-03-06T07:26:20.110506+0000","last_unstale":"2024-03-06T07:26:20.117759+0000","last_undegraded":"2024-03-06T07:26:20.117759+0000","last_fullsized":"2024-03-06T07:26:20.117759+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:55.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118088+0000","last_change":"2024-03-06T07:26:20.108671+0000","last_active":"2024-03-06T07:26:20.118088+0000","last_peered":"2024-03-06T07:26:20.118088+0000","last_clean":"2024-03-06T07:26:20.118088+0000","last_became_active":"2024-03-06T07:26:20.108313+0000","last_became_peered":"2024-03-06T07:26:20.108313+0000","last_unstale":"2024-03-06T07:26:20.118088+0000","last_undegraded":"2024-03-06T07:26:20.118088+0000","last_fullsized":"2024-03-06T07:26:20.118088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:25.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110504+0000","last_change":"2024-03-06T07:26:20.106615+0000","last_active":"2024-03-06T07:26:20.110504+0000","last_peered":"2024-03-06T07:26:20.110504+0000","last_clean":"2024-03-06T07:26:20.110504+0000","last_became_active":"2024-03-06T07:26:20.106373+0000","last_became_peered":"2024-03-06T07:26:20.106373+0000","last_unstale":"2024-03-06T07:26:20.110504+0000","last_undegraded":"2024-03-06T07:26:20.110504+0000","last_fullsized":"2024-03-06T07:26:20.110504+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:25.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123455+0000","last_change":"2024-03-06T07:26:20.117860+0000","last_active":"2024-03-06T07:26:20.123455+0000","last_peered":"2024-03-06T07:26:20.123455+0000","last_clean":"2024-03-06T07:26:20.123455+0000","last_became_active":"2024-03-06T07:26:20.117394+0000","last_became_peered":"2024-03-06T07:26:20.117394+0000","last_unstale":"2024-03-06T07:26:20.123455+0000","last_undegraded":"2024-03-06T07:26:20.123455+0000","last_fullsized":"2024-03-06T07:26:20.123455+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:20.109355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.114179+0000","last_change":"2024-03-06T07:26:20.112053+0000","last_active":"2024-03-06T07:26:20.114179+0000","last_peered":"2024-03-06T07:26:20.114179+0000","last_clean":"2024-03-06T07:26:20.114179+0000","last_became_active":"2024-03-06T07:26:20.111765+0000","last_became_peered":"2024-03-06T07:26:20.111765+0000","last_unstale":"2024-03-06T07:26:20.114179+0000","last_undegraded":"2024-03-06T07:26:20.114179+0000","last_fullsized":"2024-03-06T07:26:20.114179+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:29.411645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120467+0000","last_change":"2024-03-06T07:26:20.113734+0000","last_active":"2024-03-06T07:26:20.120467+0000","last_peered":"2024-03-06T07:26:20.120467+0000","last_clean":"2024-03-06T07:26:20.120467+0000","last_became_active":"2024-03-06T07:26:20.113085+0000","last_became_peered":"2024-03-06T07:26:20.113085+0000","last_unstale":"2024-03-06T07:26:20.120467+0000","last_undegraded":"2024-03-06T07:26:20.120467+0000","last_fullsized":"2024-03-06T07:26:20.120467+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:33.335668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.114477+0000","last_change":"2024-03-06T07:26:20.108964+0000","last_active":"2024-03-06T07:26:20.114477+0000","last_peered":"2024-03-06T07:26:20.114477+0000","last_clean":"2024-03-06T07:26:20.114477+0000","last_became_active":"2024-03-06T07:26:20.108685+0000","last_became_peered":"2024-03-06T07:26:20.108685+0000","last_unstale":"2024-03-06T07:26:20.114477+0000","last_undegraded":"2024-03-06T07:26:20.114477+0000","last_fullsized":"2024-03-06T07:26:20.114477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:41.233378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113591+0000","last_change":"2024-03-06T07:26:20.107142+0000","last_active":"2024-03-06T07:26:20.113591+0000","last_peered":"2024-03-06T07:26:20.113591+0000","last_clean":"2024-03-06T07:26:20.113591+0000","last_became_active":"2024-03-06T07:26:20.106903+0000","last_became_peered":"2024-03-06T07:26:20.106903+0000","last_unstale":"2024-03-06T07:26:20.113591+0000","last_undegraded":"2024-03-06T07:26:20.113591+0000","last_fullsized":"2024-03-06T07:26:20.113591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:07.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.108612+0000","last_change":"2024-03-06T07:26:20.102980+0000","last_active":"2024-03-06T07:26:20.108612+0000","last_peered":"2024-03-06T07:26:20.108612+0000","last_clean":"2024-03-06T07:26:20.108612+0000","last_became_active":"2024-03-06T07:26:20.102723+0000","last_became_peered":"2024-03-06T07:26:20.102723+0000","last_unstale":"2024-03-06T07:26:20.108612+0000","last_undegraded":"2024-03-06T07:26:20.108612+0000","last_fullsized":"2024-03-06T07:26:20.108612+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:16.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113258+0000","last_change":"2024-03-06T07:26:20.105701+0000","last_active":"2024-03-06T07:26:20.113258+0000","last_peered":"2024-03-06T07:26:20.113258+0000","last_clean":"2024-03-06T07:26:20.113258+0000","last_became_active":"2024-03-06T07:26:20.105446+0000","last_became_peered":"2024-03-06T07:26:20.105446+0000","last_unstale":"2024-03-06T07:26:20.113258+0000","last_undegraded":"2024-03-06T07:26:20.113258+0000","last_fullsized":"2024-03-06T07:26:20.113258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:37.165234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107257+0000","last_change":"2024-03-06T07:26:20.105930+0000","last_active":"2024-03-06T07:26:20.107257+0000","last_peered":"2024-03-06T07:26:20.107257+0000","last_clean":"2024-03-06T07:26:20.107257+0000","last_became_active":"2024-03-06T07:26:20.105676+0000","last_became_peered":"2024-03-06T07:26:20.105676+0000","last_unstale":"2024-03-06T07:26:20.107257+0000","last_undegraded":"2024-03-06T07:26:20.107257+0000","last_fullsized":"2024-03-06T07:26:20.107257+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:37.165234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121520+0000","last_change":"2024-03-06T07:26:20.114404+0000","last_active":"2024-03-06T07:26:20.121520+0000","last_peered":"2024-03-06T07:26:20.121520+0000","last_clean":"2024-03-06T07:26:20.121520+0000","last_became_active":"2024-03-06T07:26:20.114137+0000","last_became_peered":"2024-03-06T07:26:20.114137+0000","last_unstale":"2024-03-06T07:26:20.121520+0000","last_undegraded":"2024-03-06T07:26:20.121520+0000","last_fullsized":"2024-03-06T07:26:20.121520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:09.734012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110704+0000","last_change":"2024-03-06T07:26:20.100452+0000","last_active":"2024-03-06T07:26:20.110704+0000","last_peered":"2024-03-06T07:26:20.110704+0000","last_clean":"2024-03-06T07:26:20.110704+0000","last_became_active":"2024-03-06T07:26:20.100210+0000","last_became_peered":"2024-03-06T07:26:20.100210+0000","last_unstale":"2024-03-06T07:26:20.110704+0000","last_undegraded":"2024-03-06T07:26:20.110704+0000","last_fullsized":"2024-03-06T07:26:20.110704+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:34.904820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113315+0000","last_change":"2024-03-06T07:26:20.108035+0000","last_active":"2024-03-06T07:26:20.113315+0000","last_peered":"2024-03-06T07:26:20.113315+0000","last_clean":"2024-03-06T07:26:20.113315+0000","last_became_active":"2024-03-06T07:26:20.107587+0000","last_became_peered":"2024-03-06T07:26:20.107587+0000","last_unstale":"2024-03-06T07:26:20.113315+0000","last_undegraded":"2024-03-06T07:26:20.113315+0000","last_fullsized":"2024-03-06T07:26:20.113315+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:05.023999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123476+0000","last_change":"2024-03-06T07:26:20.117910+0000","last_active":"2024-03-06T07:26:20.123476+0000","last_peered":"2024-03-06T07:26:20.123476+0000","last_clean":"2024-03-06T07:26:20.123476+0000","last_became_active":"2024-03-06T07:26:20.117666+0000","last_became_peered":"2024-03-06T07:26:20.117666+0000","last_unstale":"2024-03-06T07:26:20.123476+0000","last_undegraded":"2024-03-06T07:26:20.123476+0000","last_fullsized":"2024-03-06T07:26:20.123476+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.977117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118308+0000","last_change":"2024-03-06T07:26:20.111165+0000","last_active":"2024-03-06T07:26:20.118308+0000","last_peered":"2024-03-06T07:26:20.118308+0000","last_clean":"2024-03-06T07:26:20.118308+0000","last_became_active":"2024-03-06T07:26:20.110898+0000","last_became_peered":"2024-03-06T07:26:20.110898+0000","last_unstale":"2024-03-06T07:26:20.118308+0000","last_undegraded":"2024-03-06T07:26:20.118308+0000","last_fullsized":"2024-03-06T07:26:20.118308+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:15.257611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124040+0000","last_change":"2024-03-06T07:26:20.119588+0000","last_active":"2024-03-06T07:26:20.124040+0000","last_peered":"2024-03-06T07:26:20.124040+0000","last_clean":"2024-03-06T07:26:20.124040+0000","last_became_active":"2024-03-06T07:26:20.119364+0000","last_became_peered":"2024-03-06T07:26:20.119364+0000","last_unstale":"2024-03-06T07:26:20.124040+0000","last_undegraded":"2024-03-06T07:26:20.124040+0000","last_fullsized":"2024-03-06T07:26:20.124040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:47.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121666+0000","last_change":"2024-03-06T07:26:20.117755+0000","last_active":"2024-03-06T07:26:20.121666+0000","last_peered":"2024-03-06T07:26:20.121666+0000","last_clean":"2024-03-06T07:26:20.121666+0000","last_became_active":"2024-03-06T07:26:20.117490+0000","last_became_peered":"2024-03-06T07:26:20.117490+0000","last_unstale":"2024-03-06T07:26:20.121666+0000","last_undegraded":"2024-03-06T07:26:20.121666+0000","last_fullsized":"2024-03-06T07:26:20.121666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:08.325075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117821+0000","last_change":"2024-03-06T07:26:20.110112+0000","last_active":"2024-03-06T07:26:20.117821+0000","last_peered":"2024-03-06T07:26:20.117821+0000","last_clean":"2024-03-06T07:26:20.117821+0000","last_became_active":"2024-03-06T07:26:20.109083+0000","last_became_peered":"2024-03-06T07:26:20.109083+0000","last_unstale":"2024-03-06T07:26:20.117821+0000","last_undegraded":"2024-03-06T07:26:20.117821+0000","last_fullsized":"2024-03-06T07:26:20.117821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:07.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110440+0000","last_change":"2024-03-06T07:26:20.106989+0000","last_active":"2024-03-06T07:26:20.110440+0000","last_peered":"2024-03-06T07:26:20.110440+0000","last_clean":"2024-03-06T07:26:20.110440+0000","last_became_active":"2024-03-06T07:26:20.106708+0000","last_became_peered":"2024-03-06T07:26:20.106708+0000","last_unstale":"2024-03-06T07:26:20.110440+0000","last_undegraded":"2024-03-06T07:26:20.110440+0000","last_fullsized":"2024-03-06T07:26:20.110440+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117925+0000","last_change":"2024-03-06T07:26:20.112204+0000","last_active":"2024-03-06T07:26:20.117925+0000","last_peered":"2024-03-06T07:26:20.117925+0000","last_clean":"2024-03-06T07:26:20.117925+0000","last_became_active":"2024-03-06T07:26:20.111911+0000","last_became_peered":"2024-03-06T07:26:20.111911+0000","last_unstale":"2024-03-06T07:26:20.117925+0000","last_undegraded":"2024-03-06T07:26:20.117925+0000","last_fullsized":"2024-03-06T07:26:20.117925+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:55.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107290+0000","last_change":"2024-03-06T07:26:20.104134+0000","last_active":"2024-03-06T07:26:20.107290+0000","last_peered":"2024-03-06T07:26:20.107290+0000","last_clean":"2024-03-06T07:26:20.107290+0000","last_became_active":"2024-03-06T07:26:20.103719+0000","last_became_peered":"2024-03-06T07:26:20.103719+0000","last_unstale":"2024-03-06T07:26:20.107290+0000","last_undegraded":"2024-03-06T07:26:20.107290+0000","last_fullsized":"2024-03-06T07:26:20.107290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:24.084552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.116776+0000","last_change":"2024-03-06T07:26:20.115034+0000","last_active":"2024-03-06T07:26:20.116776+0000","last_peered":"2024-03-06T07:26:20.116776+0000","last_clean":"2024-03-06T07:26:20.116776+0000","last_became_active":"2024-03-06T07:26:20.112033+0000","last_became_peered":"2024-03-06T07:26:20.112033+0000","last_unstale":"2024-03-06T07:26:20.116776+0000","last_undegraded":"2024-03-06T07:26:20.116776+0000","last_fullsized":"2024-03-06T07:26:20.116776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:29.012780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110473+0000","last_change":"2024-03-06T07:26:20.106755+0000","last_active":"2024-03-06T07:26:20.110473+0000","last_peered":"2024-03-06T07:26:20.110473+0000","last_clean":"2024-03-06T07:26:20.110473+0000","last_became_active":"2024-03-06T07:26:20.106521+0000","last_became_peered":"2024-03-06T07:26:20.106521+0000","last_unstale":"2024-03-06T07:26:20.110473+0000","last_undegraded":"2024-03-06T07:26:20.110473+0000","last_fullsized":"2024-03-06T07:26:20.110473+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:07.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113714+0000","last_change":"2024-03-06T07:26:20.109914+0000","last_active":"2024-03-06T07:26:20.113714+0000","last_peered":"2024-03-06T07:26:20.113714+0000","last_clean":"2024-03-06T07:26:20.113714+0000","last_became_active":"2024-03-06T07:26:20.109643+0000","last_became_peered":"2024-03-06T07:26:20.109643+0000","last_unstale":"2024-03-06T07:26:20.113714+0000","last_undegraded":"2024-03-06T07:26:20.113714+0000","last_fullsized":"2024-03-06T07:26:20.113714+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:37.953947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113236+0000","last_change":"2024-03-06T07:26:20.104526+0000","last_active":"2024-03-06T07:26:20.113236+0000","last_peered":"2024-03-06T07:26:20.113236+0000","last_clean":"2024-03-06T07:26:20.113236+0000","last_became_active":"2024-03-06T07:26:20.104295+0000","last_became_peered":"2024-03-06T07:26:20.104295+0000","last_unstale":"2024-03-06T07:26:20.113236+0000","last_undegraded":"2024-03-06T07:26:20.113236+0000","last_fullsized":"2024-03-06T07:26:20.113236+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:24.084552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113178+0000","last_change":"2024-03-06T07:26:20.109817+0000","last_active":"2024-03-06T07:26:20.113178+0000","last_peered":"2024-03-06T07:26:20.113178+0000","last_clean":"2024-03-06T07:26:20.113178+0000","last_became_active":"2024-03-06T07:26:20.109556+0000","last_became_peered":"2024-03-06T07:26:20.109556+0000","last_unstale":"2024-03-06T07:26:20.113178+0000","last_undegraded":"2024-03-06T07:26:20.113178+0000","last_fullsized":"2024-03-06T07:26:20.113178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:45.586349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113346+0000","last_change":"2024-03-06T07:26:20.109783+0000","last_active":"2024-03-06T07:26:20.113346+0000","last_peered":"2024-03-06T07:26:20.113346+0000","last_clean":"2024-03-06T07:26:20.113346+0000","last_became_active":"2024-03-06T07:26:20.109466+0000","last_became_peered":"2024-03-06T07:26:20.109466+0000","last_unstale":"2024-03-06T07:26:20.113346+0000","last_undegraded":"2024-03-06T07:26:20.113346+0000","last_fullsized":"2024-03-06T07:26:20.113346+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:13.865771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123079+0000","last_change":"2024-03-06T07:26:19.113658+0000","last_active":"2024-03-06T07:26:20.123079+0000","last_peered":"2024-03-06T07:26:20.123079+0000","last_clean":"2024-03-06T07:26:20.123079+0000","last_became_active":"2024-03-06T07:26:19.113388+0000","last_became_peered":"2024-03-06T07:26:19.113388+0000","last_unstale":"2024-03-06T07:26:20.123079+0000","last_undegraded":"2024-03-06T07:26:20.123079+0000","last_fullsized":"2024-03-06T07:26:20.123079+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:00.789303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113068+0000","last_change":"2024-03-06T07:26:20.103920+0000","last_active":"2024-03-06T07:26:20.113068+0000","last_peered":"2024-03-06T07:26:20.113068+0000","last_clean":"2024-03-06T07:26:20.113068+0000","last_became_active":"2024-03-06T07:26:20.103683+0000","last_became_peered":"2024-03-06T07:26:20.103683+0000","last_unstale":"2024-03-06T07:26:20.113068+0000","last_undegraded":"2024-03-06T07:26:20.113068+0000","last_fullsized":"2024-03-06T07:26:20.113068+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:26.692331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121604+0000","last_change":"2024-03-06T07:26:19.110988+0000","last_active":"2024-03-06T07:26:20.121604+0000","last_peered":"2024-03-06T07:26:20.121604+0000","last_clean":"2024-03-06T07:26:20.121604+0000","last_became_active":"2024-03-06T07:26:19.110672+0000","last_became_peered":"2024-03-06T07:26:19.110672+0000","last_unstale":"2024-03-06T07:26:20.121604+0000","last_undegraded":"2024-03-06T07:26:20.121604+0000","last_fullsized":"2024-03-06T07:26:20.121604+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:00.789303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123843+0000","last_change":"2024-03-06T07:26:20.120735+0000","last_active":"2024-03-06T07:26:20.123843+0000","last_peered":"2024-03-06T07:26:20.123843+0000","last_clean":"2024-03-06T07:26:20.123843+0000","last_became_active":"2024-03-06T07:26:20.120444+0000","last_became_peered":"2024-03-06T07:26:20.120444+0000","last_unstale":"2024-03-06T07:26:20.123843+0000","last_undegraded":"2024-03-06T07:26:20.123843+0000","last_fullsized":"2024-03-06T07:26:20.123843+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:37.277117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124193+0000","last_change":"2024-03-06T07:26:19.114482+0000","last_active":"2024-03-06T07:26:20.124193+0000","last_peered":"2024-03-06T07:26:20.124193+0000","last_clean":"2024-03-06T07:26:20.124193+0000","last_became_active":"2024-03-06T07:26:19.114178+0000","last_became_peered":"2024-03-06T07:26:19.114178+0000","last_unstale":"2024-03-06T07:26:20.124193+0000","last_undegraded":"2024-03-06T07:26:20.124193+0000","last_fullsized":"2024-03-06T07:26:20.124193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:32.777885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121165+0000","last_change":"2024-03-06T07:26:20.116993+0000","last_active":"2024-03-06T07:26:20.121165+0000","last_peered":"2024-03-06T07:26:20.121165+0000","last_clean":"2024-03-06T07:26:20.121165+0000","last_became_active":"2024-03-06T07:26:20.116744+0000","last_became_peered":"2024-03-06T07:26:20.116744+0000","last_unstale":"2024-03-06T07:26:20.121165+0000","last_undegraded":"2024-03-06T07:26:20.121165+0000","last_fullsized":"2024-03-06T07:26:20.121165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.870532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124328+0000","last_change":"2024-03-06T07:26:19.110182+0000","last_active":"2024-03-06T07:26:20.124328+0000","last_peered":"2024-03-06T07:26:20.124328+0000","last_clean":"2024-03-06T07:26:20.124328+0000","last_became_active":"2024-03-06T07:26:19.109898+0000","last_became_peered":"2024-03-06T07:26:19.109898+0000","last_unstale":"2024-03-06T07:26:20.124328+0000","last_undegraded":"2024-03-06T07:26:20.124328+0000","last_fullsized":"2024-03-06T07:26:20.124328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:56.576898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120935+0000","last_change":"2024-03-06T07:26:20.115512+0000","last_active":"2024-03-06T07:26:20.120935+0000","last_peered":"2024-03-06T07:26:20.120935+0000","last_clean":"2024-03-06T07:26:20.120935+0000","last_became_active":"2024-03-06T07:26:20.115195+0000","last_became_peered":"2024-03-06T07:26:20.115195+0000","last_unstale":"2024-03-06T07:26:20.120935+0000","last_undegraded":"2024-03-06T07:26:20.120935+0000","last_fullsized":"2024-03-06T07:26:20.120935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:57.979993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.119153+0000","last_change":"2024-03-06T07:26:19.107806+0000","last_active":"2024-03-06T07:26:20.119153+0000","last_peered":"2024-03-06T07:26:20.119153+0000","last_clean":"2024-03-06T07:26:20.119153+0000","last_became_active":"2024-03-06T07:26:19.107555+0000","last_became_peered":"2024-03-06T07:26:19.107555+0000","last_unstale":"2024-03-06T07:26:20.119153+0000","last_undegraded":"2024-03-06T07:26:20.119153+0000","last_fullsized":"2024-03-06T07:26:20.119153+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:17.147773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107072+0000","last_change":"2024-03-06T07:26:20.103519+0000","last_active":"2024-03-06T07:26:20.107072+0000","last_peered":"2024-03-06T07:26:20.107072+0000","last_clean":"2024-03-06T07:26:20.107072+0000","last_became_active":"2024-03-06T07:26:20.103251+0000","last_became_peered":"2024-03-06T07:26:20.103251+0000","last_unstale":"2024-03-06T07:26:20.107072+0000","last_undegraded":"2024-03-06T07:26:20.107072+0000","last_fullsized":"2024-03-06T07:26:20.107072+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:26.692331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124571+0000","last_change":"2024-03-06T07:26:19.108714+0000","last_active":"2024-03-06T07:26:20.124571+0000","last_peered":"2024-03-06T07:26:20.124571+0000","last_clean":"2024-03-06T07:26:20.124571+0000","last_became_active":"2024-03-06T07:26:19.108441+0000","last_became_peered":"2024-03-06T07:26:19.108441+0000","last_unstale":"2024-03-06T07:26:20.124571+0000","last_undegraded":"2024-03-06T07:26:20.124571+0000","last_fullsized":"2024-03-06T07:26:20.124571+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:22.183800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113583+0000","last_change":"2024-03-06T07:26:20.108907+0000","last_active":"2024-03-06T07:26:20.113583+0000","last_peered":"2024-03-06T07:26:20.113583+0000","last_clean":"2024-03-06T07:26:20.113583+0000","last_became_active":"2024-03-06T07:26:20.108196+0000","last_became_peered":"2024-03-06T07:26:20.108196+0000","last_unstale":"2024-03-06T07:26:20.113583+0000","last_undegraded":"2024-03-06T07:26:20.113583+0000","last_fullsized":"2024-03-06T07:26:20.113583+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:55.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.099874+0000","last_change":"2024-03-06T07:26:19.114206+0000","last_active":"2024-03-06T07:26:20.099874+0000","last_peered":"2024-03-06T07:26:20.099874+0000","last_clean":"2024-03-06T07:26:20.099874+0000","last_became_active":"2024-03-06T07:26:19.113991+0000","last_became_peered":"2024-03-06T07:26:19.113991+0000","last_unstale":"2024-03-06T07:26:20.099874+0000","last_undegraded":"2024-03-06T07:26:20.099874+0000","last_fullsized":"2024-03-06T07:26:20.099874+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:22.862479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118174+0000","last_change":"2024-03-06T07:26:20.110357+0000","last_active":"2024-03-06T07:26:20.118174+0000","last_peered":"2024-03-06T07:26:20.118174+0000","last_clean":"2024-03-06T07:26:20.118174+0000","last_became_active":"2024-03-06T07:26:20.110096+0000","last_became_peered":"2024-03-06T07:26:20.110096+0000","last_unstale":"2024-03-06T07:26:20.118174+0000","last_undegraded":"2024-03-06T07:26:20.118174+0000","last_fullsized":"2024-03-06T07:26:20.118174+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:57.979993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.106487+0000","last_change":"2024-03-06T07:26:19.112408+0000","last_active":"2024-03-06T07:26:20.106487+0000","last_peered":"2024-03-06T07:26:20.106487+0000","last_clean":"2024-03-06T07:26:20.106487+0000","last_became_active":"2024-03-06T07:26:19.112142+0000","last_became_peered":"2024-03-06T07:26:19.112142+0000","last_unstale":"2024-03-06T07:26:20.106487+0000","last_undegraded":"2024-03-06T07:26:20.106487+0000","last_fullsized":"2024-03-06T07:26:20.106487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:24.001055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121629+0000","last_change":"2024-03-06T07:26:20.116535+0000","last_active":"2024-03-06T07:26:20.121629+0000","last_peered":"2024-03-06T07:26:20.121629+0000","last_clean":"2024-03-06T07:26:20.121629+0000","last_became_active":"2024-03-06T07:26:20.116167+0000","last_became_peered":"2024-03-06T07:26:20.116167+0000","last_unstale":"2024-03-06T07:26:20.121629+0000","last_undegraded":"2024-03-06T07:26:20.121629+0000","last_fullsized":"2024-03-06T07:26:20.121629+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:33.335668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.108368+0000","last_change":"2024-03-06T07:26:19.104265+0000","last_active":"2024-03-06T07:26:20.108368+0000","last_peered":"2024-03-06T07:26:20.108368+0000","last_clean":"2024-03-06T07:26:20.108368+0000","last_became_active":"2024-03-06T07:26:19.103989+0000","last_became_peered":"2024-03-06T07:26:19.103989+0000","last_unstale":"2024-03-06T07:26:20.108368+0000","last_undegraded":"2024-03-06T07:26:20.108368+0000","last_fullsized":"2024-03-06T07:26:20.108368+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:42.973670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120128+0000","last_change":"2024-03-06T07:26:20.112073+0000","last_active":"2024-03-06T07:26:20.120128+0000","last_peered":"2024-03-06T07:26:20.120128+0000","last_clean":"2024-03-06T07:26:20.120128+0000","last_became_active":"2024-03-06T07:26:20.111778+0000","last_became_peered":"2024-03-06T07:26:20.111778+0000","last_unstale":"2024-03-06T07:26:20.120128+0000","last_undegraded":"2024-03-06T07:26:20.120128+0000","last_fullsized":"2024-03-06T07:26:20.120128+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:21.882258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.095249+0000","last_change":"2024-03-06T07:26:19.110192+0000","last_active":"2024-03-06T07:26:20.095249+0000","last_peered":"2024-03-06T07:26:20.095249+0000","last_clean":"2024-03-06T07:26:20.095249+0000","last_became_active":"2024-03-06T07:26:19.109927+0000","last_became_peered":"2024-03-06T07:26:19.109927+0000","last_unstale":"2024-03-06T07:26:20.095249+0000","last_undegraded":"2024-03-06T07:26:20.095249+0000","last_fullsized":"2024-03-06T07:26:20.095249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:34.827670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110356+0000","last_change":"2024-03-06T07:26:20.108055+0000","last_active":"2024-03-06T07:26:20.110356+0000","last_peered":"2024-03-06T07:26:20.110356+0000","last_clean":"2024-03-06T07:26:20.110356+0000","last_became_active":"2024-03-06T07:26:20.107783+0000","last_became_peered":"2024-03-06T07:26:20.107783+0000","last_unstale":"2024-03-06T07:26:20.110356+0000","last_undegraded":"2024-03-06T07:26:20.110356+0000","last_fullsized":"2024-03-06T07:26:20.110356+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:02.917862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124511+0000","last_change":"2024-03-06T07:26:19.109632+0000","last_active":"2024-03-06T07:26:20.124511+0000","last_peered":"2024-03-06T07:26:20.124511+0000","last_clean":"2024-03-06T07:26:20.124511+0000","last_became_active":"2024-03-06T07:26:19.109342+0000","last_became_peered":"2024-03-06T07:26:19.109342+0000","last_unstale":"2024-03-06T07:26:20.124511+0000","last_undegraded":"2024-03-06T07:26:20.124511+0000","last_fullsized":"2024-03-06T07:26:20.124511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:56.576898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118064+0000","last_change":"2024-03-06T07:26:20.112776+0000","last_active":"2024-03-06T07:26:20.118064+0000","last_peered":"2024-03-06T07:26:20.118064+0000","last_clean":"2024-03-06T07:26:20.118064+0000","last_became_active":"2024-03-06T07:26:20.112480+0000","last_became_peered":"2024-03-06T07:26:20.112480+0000","last_unstale":"2024-03-06T07:26:20.118064+0000","last_undegraded":"2024-03-06T07:26:20.118064+0000","last_fullsized":"2024-03-06T07:26:20.118064+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:09.734012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.128420+0000","last_change":"2024-03-06T07:26:19.108018+0000","last_active":"2024-03-06T07:26:20.128420+0000","last_peered":"2024-03-06T07:26:20.128420+0000","last_clean":"2024-03-06T07:26:20.128420+0000","last_became_active":"2024-03-06T07:26:19.107638+0000","last_became_peered":"2024-03-06T07:26:19.107638+0000","last_unstale":"2024-03-06T07:26:20.128420+0000","last_undegraded":"2024-03-06T07:26:20.128420+0000","last_fullsized":"2024-03-06T07:26:20.128420+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:21.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113457+0000","last_change":"2024-03-06T07:26:20.106544+0000","last_active":"2024-03-06T07:26:20.113457+0000","last_peered":"2024-03-06T07:26:20.113457+0000","last_clean":"2024-03-06T07:26:20.113457+0000","last_became_active":"2024-03-06T07:26:20.106307+0000","last_became_peered":"2024-03-06T07:26:20.106307+0000","last_unstale":"2024-03-06T07:26:20.113457+0000","last_undegraded":"2024-03-06T07:26:20.113457+0000","last_fullsized":"2024-03-06T07:26:20.113457+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:25.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124281+0000","last_change":"2024-03-06T07:26:19.112397+0000","last_active":"2024-03-06T07:26:20.124281+0000","last_peered":"2024-03-06T07:26:20.124281+0000","last_clean":"2024-03-06T07:26:20.124281+0000","last_became_active":"2024-03-06T07:26:19.112023+0000","last_became_peered":"2024-03-06T07:26:19.112023+0000","last_unstale":"2024-03-06T07:26:20.124281+0000","last_undegraded":"2024-03-06T07:26:20.124281+0000","last_fullsized":"2024-03-06T07:26:20.124281+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:46.549394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113045+0000","last_change":"2024-03-06T07:26:20.110618+0000","last_active":"2024-03-06T07:26:20.113045+0000","last_peered":"2024-03-06T07:26:20.113045+0000","last_clean":"2024-03-06T07:26:20.113045+0000","last_became_active":"2024-03-06T07:26:20.110377+0000","last_became_peered":"2024-03-06T07:26:20.110377+0000","last_unstale":"2024-03-06T07:26:20.113045+0000","last_undegraded":"2024-03-06T07:26:20.113045+0000","last_fullsized":"2024-03-06T07:26:20.113045+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:21.347163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.099172+0000","last_change":"2024-03-06T07:26:19.113525+0000","last_active":"2024-03-06T07:26:20.099172+0000","last_peered":"2024-03-06T07:26:20.099172+0000","last_clean":"2024-03-06T07:26:20.099172+0000","last_became_active":"2024-03-06T07:26:19.113249+0000","last_became_peered":"2024-03-06T07:26:19.113249+0000","last_unstale":"2024-03-06T07:26:20.099172+0000","last_undegraded":"2024-03-06T07:26:20.099172+0000","last_fullsized":"2024-03-06T07:26:20.099172+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:28.131529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118249+0000","last_change":"2024-03-06T07:26:20.111248+0000","last_active":"2024-03-06T07:26:20.118249+0000","last_peered":"2024-03-06T07:26:20.118249+0000","last_clean":"2024-03-06T07:26:20.118249+0000","last_became_active":"2024-03-06T07:26:20.110952+0000","last_became_peered":"2024-03-06T07:26:20.110952+0000","last_unstale":"2024-03-06T07:26:20.118249+0000","last_undegraded":"2024-03-06T07:26:20.118249+0000","last_fullsized":"2024-03-06T07:26:20.118249+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:17.103413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.106497+0000","last_change":"2024-03-06T07:26:19.107891+0000","last_active":"2024-03-06T07:26:20.106497+0000","last_peered":"2024-03-06T07:26:20.106497+0000","last_clean":"2024-03-06T07:26:20.106497+0000","last_became_active":"2024-03-06T07:26:19.107600+0000","last_became_peered":"2024-03-06T07:26:19.107600+0000","last_unstale":"2024-03-06T07:26:20.106497+0000","last_undegraded":"2024-03-06T07:26:20.106497+0000","last_fullsized":"2024-03-06T07:26:20.106497+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:44.054462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110551+0000","last_change":"2024-03-06T07:26:20.105866+0000","last_active":"2024-03-06T07:26:20.110551+0000","last_peered":"2024-03-06T07:26:20.110551+0000","last_clean":"2024-03-06T07:26:20.110551+0000","last_became_active":"2024-03-06T07:26:20.105594+0000","last_became_peered":"2024-03-06T07:26:20.105594+0000","last_unstale":"2024-03-06T07:26:20.110551+0000","last_undegraded":"2024-03-06T07:26:20.110551+0000","last_fullsized":"2024-03-06T07:26:20.110551+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:35.862436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"92'8","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.125008+0000","last_change":"2024-03-06T07:26:19.110099+0000","last_active":"2024-03-06T07:26:20.125008+0000","last_peered":"2024-03-06T07:26:20.125008+0000","last_clean":"2024-03-06T07:26:20.125008+0000","last_became_active":"2024-03-06T07:26:19.109815+0000","last_became_peered":"2024-03-06T07:26:19.109815+0000","last_unstale":"2024-03-06T07:26:20.125008+0000","last_undegraded":"2024-03-06T07:26:20.125008+0000","last_fullsized":"2024-03-06T07:26:20.125008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:20.128480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.116321+0000","last_change":"2024-03-06T07:26:20.114001+0000","last_active":"2024-03-06T07:26:20.116321+0000","last_peered":"2024-03-06T07:26:20.116321+0000","last_clean":"2024-03-06T07:26:20.116321+0000","last_became_active":"2024-03-06T07:26:20.113033+0000","last_became_peered":"2024-03-06T07:26:20.113033+0000","last_unstale":"2024-03-06T07:26:20.116321+0000","last_undegraded":"2024-03-06T07:26:20.116321+0000","last_fullsized":"2024-03-06T07:26:20.116321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:51.214905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124347+0000","last_change":"2024-03-06T07:26:19.110045+0000","last_active":"2024-03-06T07:26:20.124347+0000","last_peered":"2024-03-06T07:26:20.124347+0000","last_clean":"2024-03-06T07:26:20.124347+0000","last_became_active":"2024-03-06T07:26:19.109745+0000","last_became_peered":"2024-03-06T07:26:19.109745+0000","last_unstale":"2024-03-06T07:26:20.124347+0000","last_undegraded":"2024-03-06T07:26:20.124347+0000","last_fullsized":"2024-03-06T07:26:20.124347+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:31.048107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117638+0000","last_change":"2024-03-06T07:26:20.110006+0000","last_active":"2024-03-06T07:26:20.117638+0000","last_peered":"2024-03-06T07:26:20.117638+0000","last_clean":"2024-03-06T07:26:20.117638+0000","last_became_active":"2024-03-06T07:26:20.109702+0000","last_became_peered":"2024-03-06T07:26:20.109702+0000","last_unstale":"2024-03-06T07:26:20.117638+0000","last_undegraded":"2024-03-06T07:26:20.117638+0000","last_fullsized":"2024-03-06T07:26:20.117638+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:41.233378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123886+0000","last_change":"2024-03-06T07:26:19.109958+0000","last_active":"2024-03-06T07:26:20.123886+0000","last_peered":"2024-03-06T07:26:20.123886+0000","last_clean":"2024-03-06T07:26:20.123886+0000","last_became_active":"2024-03-06T07:26:19.109672+0000","last_became_peered":"2024-03-06T07:26:19.109672+0000","last_unstale":"2024-03-06T07:26:20.123886+0000","last_undegraded":"2024-03-06T07:26:20.123886+0000","last_fullsized":"2024-03-06T07:26:20.123886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:19.101303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.120503+0000","last_change":"2024-03-06T07:26:20.113947+0000","last_active":"2024-03-06T07:26:20.120503+0000","last_peered":"2024-03-06T07:26:20.120503+0000","last_clean":"2024-03-06T07:26:20.120503+0000","last_became_active":"2024-03-06T07:26:20.113465+0000","last_became_peered":"2024-03-06T07:26:20.113465+0000","last_unstale":"2024-03-06T07:26:20.120503+0000","last_undegraded":"2024-03-06T07:26:20.120503+0000","last_fullsized":"2024-03-06T07:26:20.120503+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:08.325075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.103165+0000","last_change":"2024-03-06T07:26:19.110420+0000","last_active":"2024-03-06T07:26:20.103165+0000","last_peered":"2024-03-06T07:26:20.103165+0000","last_clean":"2024-03-06T07:26:20.103165+0000","last_became_active":"2024-03-06T07:26:19.109525+0000","last_became_peered":"2024-03-06T07:26:19.109525+0000","last_unstale":"2024-03-06T07:26:20.103165+0000","last_undegraded":"2024-03-06T07:26:20.103165+0000","last_fullsized":"2024-03-06T07:26:20.103165+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:33.896767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109518+0000","last_change":"2024-03-06T07:26:20.104809+0000","last_active":"2024-03-06T07:26:20.109518+0000","last_peered":"2024-03-06T07:26:20.109518+0000","last_clean":"2024-03-06T07:26:20.109518+0000","last_became_active":"2024-03-06T07:26:20.104567+0000","last_became_peered":"2024-03-06T07:26:20.104567+0000","last_unstale":"2024-03-06T07:26:20.109518+0000","last_undegraded":"2024-03-06T07:26:20.109518+0000","last_fullsized":"2024-03-06T07:26:20.109518+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:22.278065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124100+0000","last_change":"2024-03-06T07:26:19.108073+0000","last_active":"2024-03-06T07:26:20.124100+0000","last_peered":"2024-03-06T07:26:20.124100+0000","last_clean":"2024-03-06T07:26:20.124100+0000","last_became_active":"2024-03-06T07:26:19.107792+0000","last_became_peered":"2024-03-06T07:26:19.107792+0000","last_unstale":"2024-03-06T07:26:20.124100+0000","last_undegraded":"2024-03-06T07:26:20.124100+0000","last_fullsized":"2024-03-06T07:26:20.124100+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:25.684278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109499+0000","last_change":"2024-03-06T07:26:20.105001+0000","last_active":"2024-03-06T07:26:20.109499+0000","last_peered":"2024-03-06T07:26:20.109499+0000","last_clean":"2024-03-06T07:26:20.109499+0000","last_became_active":"2024-03-06T07:26:20.104717+0000","last_became_peered":"2024-03-06T07:26:20.104717+0000","last_unstale":"2024-03-06T07:26:20.109499+0000","last_undegraded":"2024-03-06T07:26:20.109499+0000","last_fullsized":"2024-03-06T07:26:20.109499+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:52.743978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101487+0000","last_change":"2024-03-06T07:26:19.108061+0000","last_active":"2024-03-06T07:26:20.101487+0000","last_peered":"2024-03-06T07:26:20.101487+0000","last_clean":"2024-03-06T07:26:20.101487+0000","last_became_active":"2024-03-06T07:26:19.107764+0000","last_became_peered":"2024-03-06T07:26:19.107764+0000","last_unstale":"2024-03-06T07:26:20.101487+0000","last_undegraded":"2024-03-06T07:26:20.101487+0000","last_fullsized":"2024-03-06T07:26:20.101487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:31.048107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109584+0000","last_change":"2024-03-06T07:26:20.104546+0000","last_active":"2024-03-06T07:26:20.109584+0000","last_peered":"2024-03-06T07:26:20.109584+0000","last_clean":"2024-03-06T07:26:20.109584+0000","last_became_active":"2024-03-06T07:26:20.104281+0000","last_became_peered":"2024-03-06T07:26:20.104281+0000","last_unstale":"2024-03-06T07:26:20.109584+0000","last_undegraded":"2024-03-06T07:26:20.109584+0000","last_fullsized":"2024-03-06T07:26:20.109584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:11.193272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101395+0000","last_change":"2024-03-06T07:26:19.105855+0000","last_active":"2024-03-06T07:26:20.101395+0000","last_peered":"2024-03-06T07:26:20.101395+0000","last_clean":"2024-03-06T07:26:20.101395+0000","last_became_active":"2024-03-06T07:26:19.105503+0000","last_became_peered":"2024-03-06T07:26:19.105503+0000","last_unstale":"2024-03-06T07:26:20.101395+0000","last_undegraded":"2024-03-06T07:26:20.101395+0000","last_fullsized":"2024-03-06T07:26:20.101395+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:44.054462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.107199+0000","last_change":"2024-03-06T07:26:20.104890+0000","last_active":"2024-03-06T07:26:20.107199+0000","last_peered":"2024-03-06T07:26:20.107199+0000","last_clean":"2024-03-06T07:26:20.107199+0000","last_became_active":"2024-03-06T07:26:20.104613+0000","last_became_peered":"2024-03-06T07:26:20.104613+0000","last_unstale":"2024-03-06T07:26:20.107199+0000","last_undegraded":"2024-03-06T07:26:20.107199+0000","last_fullsized":"2024-03-06T07:26:20.107199+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.977117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101439+0000","last_change":"2024-03-06T07:26:19.102520+0000","last_active":"2024-03-06T07:26:20.101439+0000","last_peered":"2024-03-06T07:26:20.101439+0000","last_clean":"2024-03-06T07:26:20.101439+0000","last_became_active":"2024-03-06T07:26:19.102212+0000","last_became_peered":"2024-03-06T07:26:19.102212+0000","last_unstale":"2024-03-06T07:26:20.101439+0000","last_undegraded":"2024-03-06T07:26:20.101439+0000","last_fullsized":"2024-03-06T07:26:20.101439+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:09.370568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.123995+0000","last_change":"2024-03-06T07:26:20.118894+0000","last_active":"2024-03-06T07:26:20.123995+0000","last_peered":"2024-03-06T07:26:20.123995+0000","last_clean":"2024-03-06T07:26:20.123995+0000","last_became_active":"2024-03-06T07:26:20.118636+0000","last_became_peered":"2024-03-06T07:26:20.118636+0000","last_unstale":"2024-03-06T07:26:20.123995+0000","last_undegraded":"2024-03-06T07:26:20.123995+0000","last_fullsized":"2024-03-06T07:26:20.123995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:55.482297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.124277+0000","last_change":"2024-03-06T07:26:19.110031+0000","last_active":"2024-03-06T07:26:20.124277+0000","last_peered":"2024-03-06T07:26:20.124277+0000","last_clean":"2024-03-06T07:26:20.124277+0000","last_became_active":"2024-03-06T07:26:19.109745+0000","last_became_peered":"2024-03-06T07:26:19.109745+0000","last_unstale":"2024-03-06T07:26:20.124277+0000","last_undegraded":"2024-03-06T07:26:20.124277+0000","last_fullsized":"2024-03-06T07:26:20.124277+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:21.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.111523+0000","last_change":"2024-03-06T07:26:20.107533+0000","last_active":"2024-03-06T07:26:20.111523+0000","last_peered":"2024-03-06T07:26:20.111523+0000","last_clean":"2024-03-06T07:26:20.111523+0000","last_became_active":"2024-03-06T07:26:20.106910+0000","last_became_peered":"2024-03-06T07:26:20.106910+0000","last_unstale":"2024-03-06T07:26:20.111523+0000","last_undegraded":"2024-03-06T07:26:20.111523+0000","last_fullsized":"2024-03-06T07:26:20.111523+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:53.479826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.104740+0000","last_change":"2024-03-06T07:26:19.107899+0000","last_active":"2024-03-06T07:26:20.104740+0000","last_peered":"2024-03-06T07:26:20.104740+0000","last_clean":"2024-03-06T07:26:20.104740+0000","last_became_active":"2024-03-06T07:26:19.107531+0000","last_became_peered":"2024-03-06T07:26:19.107531+0000","last_unstale":"2024-03-06T07:26:20.104740+0000","last_undegraded":"2024-03-06T07:26:20.104740+0000","last_fullsized":"2024-03-06T07:26:20.104740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:36.157182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.111454+0000","last_change":"2024-03-06T07:26:20.107330+0000","last_active":"2024-03-06T07:26:20.111454+0000","last_peered":"2024-03-06T07:26:20.111454+0000","last_clean":"2024-03-06T07:26:20.111454+0000","last_became_active":"2024-03-06T07:26:20.106877+0000","last_became_peered":"2024-03-06T07:26:20.106877+0000","last_unstale":"2024-03-06T07:26:20.111454+0000","last_undegraded":"2024-03-06T07:26:20.111454+0000","last_fullsized":"2024-03-06T07:26:20.111454+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:10.505443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.101580+0000","last_change":"2024-03-06T07:26:19.111449+0000","last_active":"2024-03-06T07:26:20.101580+0000","last_peered":"2024-03-06T07:26:20.101580+0000","last_clean":"2024-03-06T07:26:20.101580+0000","last_became_active":"2024-03-06T07:26:19.110513+0000","last_became_peered":"2024-03-06T07:26:19.110513+0000","last_unstale":"2024-03-06T07:26:20.101580+0000","last_undegraded":"2024-03-06T07:26:20.101580+0000","last_fullsized":"2024-03-06T07:26:20.101580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:34.827670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117508+0000","last_change":"2024-03-06T07:26:20.115860+0000","last_active":"2024-03-06T07:26:20.117508+0000","last_peered":"2024-03-06T07:26:20.117508+0000","last_clean":"2024-03-06T07:26:20.117508+0000","last_became_active":"2024-03-06T07:26:20.115536+0000","last_became_peered":"2024-03-06T07:26:20.115536+0000","last_unstale":"2024-03-06T07:26:20.117508+0000","last_undegraded":"2024-03-06T07:26:20.117508+0000","last_fullsized":"2024-03-06T07:26:20.117508+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:33.785937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.096942+0000","last_change":"2024-03-06T07:26:19.109180+0000","last_active":"2024-03-06T07:26:20.096942+0000","last_peered":"2024-03-06T07:26:20.096942+0000","last_clean":"2024-03-06T07:26:20.096942+0000","last_became_active":"2024-03-06T07:26:19.108881+0000","last_became_peered":"2024-03-06T07:26:19.108881+0000","last_unstale":"2024-03-06T07:26:20.096942+0000","last_undegraded":"2024-03-06T07:26:20.096942+0000","last_fullsized":"2024-03-06T07:26:20.096942+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:20.874205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.113693+0000","last_change":"2024-03-06T07:26:20.108331+0000","last_active":"2024-03-06T07:26:20.113693+0000","last_peered":"2024-03-06T07:26:20.113693+0000","last_clean":"2024-03-06T07:26:20.113693+0000","last_became_active":"2024-03-06T07:26:20.107429+0000","last_became_peered":"2024-03-06T07:26:20.107429+0000","last_unstale":"2024-03-06T07:26:20.113693+0000","last_undegraded":"2024-03-06T07:26:20.113693+0000","last_fullsized":"2024-03-06T07:26:20.113693+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:01.787165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.103911+0000","last_change":"2024-03-06T07:26:19.103430+0000","last_active":"2024-03-06T07:26:20.103911+0000","last_peered":"2024-03-06T07:26:20.103911+0000","last_clean":"2024-03-06T07:26:20.103911+0000","last_became_active":"2024-03-06T07:26:19.102977+0000","last_became_peered":"2024-03-06T07:26:19.102977+0000","last_unstale":"2024-03-06T07:26:20.103911+0000","last_undegraded":"2024-03-06T07:26:20.103911+0000","last_fullsized":"2024-03-06T07:26:20.103911+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:23.076499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.118300+0000","last_change":"2024-03-06T07:26:20.111066+0000","last_active":"2024-03-06T07:26:20.118300+0000","last_peered":"2024-03-06T07:26:20.118300+0000","last_clean":"2024-03-06T07:26:20.118300+0000","last_became_active":"2024-03-06T07:26:20.110753+0000","last_became_peered":"2024-03-06T07:26:20.110753+0000","last_unstale":"2024-03-06T07:26:20.118300+0000","last_undegraded":"2024-03-06T07:26:20.118300+0000","last_fullsized":"2024-03-06T07:26:20.118300+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:09.647399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.096039+0000","last_change":"2024-03-06T07:26:19.110996+0000","last_active":"2024-03-06T07:26:20.096039+0000","last_peered":"2024-03-06T07:26:20.096039+0000","last_clean":"2024-03-06T07:26:20.096039+0000","last_became_active":"2024-03-06T07:26:19.110738+0000","last_became_peered":"2024-03-06T07:26:19.110738+0000","last_unstale":"2024-03-06T07:26:20.096039+0000","last_undegraded":"2024-03-06T07:26:20.096039+0000","last_fullsized":"2024-03-06T07:26:20.096039+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:16.095361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.110583+0000","last_change":"2024-03-06T07:26:20.107196+0000","last_active":"2024-03-06T07:26:20.110583+0000","last_peered":"2024-03-06T07:26:20.110583+0000","last_clean":"2024-03-06T07:26:20.110583+0000","last_became_active":"2024-03-06T07:26:20.106251+0000","last_became_peered":"2024-03-06T07:26:20.106251+0000","last_unstale":"2024-03-06T07:26:20.110583+0000","last_undegraded":"2024-03-06T07:26:20.110583+0000","last_fullsized":"2024-03-06T07:26:20.110583+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:35.016702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.095385+0000","last_change":"2024-03-06T07:26:19.109512+0000","last_active":"2024-03-06T07:26:20.095385+0000","last_peered":"2024-03-06T07:26:20.095385+0000","last_clean":"2024-03-06T07:26:20.095385+0000","last_became_active":"2024-03-06T07:26:19.109004+0000","last_became_peered":"2024-03-06T07:26:19.109004+0000","last_unstale":"2024-03-06T07:26:20.095385+0000","last_undegraded":"2024-03-06T07:26:20.095385+0000","last_fullsized":"2024-03-06T07:26:20.095385+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:00.789303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.117709+0000","last_change":"2024-03-06T07:26:20.115910+0000","last_active":"2024-03-06T07:26:20.117709+0000","last_peered":"2024-03-06T07:26:20.117709+0000","last_clean":"2024-03-06T07:26:20.117709+0000","last_became_active":"2024-03-06T07:26:20.115620+0000","last_became_peered":"2024-03-06T07:26:20.115620+0000","last_unstale":"2024-03-06T07:26:20.117709+0000","last_undegraded":"2024-03-06T07:26:20.117709+0000","last_fullsized":"2024-03-06T07:26:20.117709+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:10.378621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.098956+0000","last_change":"2024-03-06T07:26:19.106142+0000","last_active":"2024-03-06T07:26:20.098956+0000","last_peered":"2024-03-06T07:26:20.098956+0000","last_clean":"2024-03-06T07:26:20.098956+0000","last_became_active":"2024-03-06T07:26:19.105867+0000","last_became_peered":"2024-03-06T07:26:19.105867+0000","last_unstale":"2024-03-06T07:26:20.098956+0000","last_undegraded":"2024-03-06T07:26:20.098956+0000","last_fullsized":"2024-03-06T07:26:20.098956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:37.557448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.109674+0000","last_change":"2024-03-06T07:26:20.103753+0000","last_active":"2024-03-06T07:26:20.109674+0000","last_peered":"2024-03-06T07:26:20.109674+0000","last_clean":"2024-03-06T07:26:20.109674+0000","last_became_active":"2024-03-06T07:26:20.103483+0000","last_became_peered":"2024-03-06T07:26:20.103483+0000","last_unstale":"2024-03-06T07:26:20.109674+0000","last_undegraded":"2024-03-06T07:26:20.109674+0000","last_fullsized":"2024-03-06T07:26:20.109674+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.086728+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.086728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:52.563008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.121429+0000","last_change":"2024-03-06T07:26:19.112799+0000","last_active":"2024-03-06T07:26:20.121429+0000","last_peered":"2024-03-06T07:26:20.121429+0000","last_clean":"2024-03-06T07:26:20.121429+0000","last_became_active":"2024-03-06T07:26:19.112583+0000","last_became_peered":"2024-03-06T07:26:19.112583+0000","last_unstale":"2024-03-06T07:26:20.121429+0000","last_undegraded":"2024-03-06T07:26:20.121429+0000","last_fullsized":"2024-03-06T07:26:20.121429+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.078675+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.078675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:33.896767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":305,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:20.096395+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:26:20.096395+0000","last_peered":"2024-03-06T07:26:20.096395+0000","last_clean":"2024-03-06T07:26:20.096395+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:26:20.096395+0000","last_undegraded":"2024-03-06T07:26:20.096395+0000","last_fullsized":"2024-03-06T07:26:20.096395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1740800,"data_stored":1075219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":477964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1736704,"data_stored":1075689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:23.406 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:23.406 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:23.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.547+0000 7f7f4213e640 1 Processor -- start 2024-03-06T07:26:23.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.547+0000 7f7f4213e640 1 -- start start 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f4213e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7f3c1586e0 0x7f7f3c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f4213e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f3c0d01a0 0x7f7f3c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f4213e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c158ac0 con 0x7f7f3c0aef80 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f4213e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c15b2a0 con 0x7f7f3c0d01a0 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f4213e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c15b540 con 0x7f7f3c1586e0 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f3bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f3c0d01a0 0x7f7f3c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f3b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7f3c1586e0 0x7f7f3c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f3bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f3c0d01a0 0x7f7f3c0b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56728/0 (socket says 172.21.15.122:56728) 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f3bfff640 1 -- 172.21.15.122:0/3944485151 learned_addr learned my addr 172.21.15.122:0/3944485151 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.548+0000 7f7f3affd640 1 --1- 172.21.15.122:0/3944485151 >> v1:172.21.15.169:6789/0 conn(0x7f7f3c0aef80 0x7f7f3c0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:60378/0 (socket says 172.21.15.122:60378) 2024-03-06T07:26:23.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.549+0000 7f7f3bfff640 1 -- 172.21.15.122:0/3944485151 >> v2:172.21.15.169:3300/0 conn(0x7f7f3c1586e0 msgr2=0x7f7f3c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.549+0000 7f7f3bfff640 1 --2- 172.21.15.122:0/3944485151 >> v2:172.21.15.169:3300/0 conn(0x7f7f3c1586e0 0x7f7f3c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.549+0000 7f7f3bfff640 1 -- 172.21.15.122:0/3944485151 >> v1:172.21.15.169:6789/0 conn(0x7f7f3c0aef80 legacy=0x7f7f3c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.549+0000 7f7f3bfff640 1 -- 172.21.15.122:0/3944485151 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c15b820 con 0x7f7f3c0d01a0 2024-03-06T07:26:23.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.549+0000 7f7f3b7fe640 1 --2- 172.21.15.122:0/3944485151 >> v2:172.21.15.169:3300/0 conn(0x7f7f3c1586e0 0x7f7f3c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:23.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.549+0000 7f7f3bfff640 1 --2- 172.21.15.122:0/3944485151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c0d01a0 0x7f7f3c0b33a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c070d10 tx=0x7f7f3402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e363f5976194fe07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f3a7fc640 1 -- 172.21.15.122:0/3944485151 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3402ea90 con 0x7f7f3c0d01a0 2024-03-06T07:26:23.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f3a7fc640 1 -- 172.21.15.122:0/3944485151 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3402ec30 con 0x7f7f3c0d01a0 2024-03-06T07:26:23.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f3a7fc640 1 -- 172.21.15.122:0/3944485151 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34009600 con 0x7f7f3c0d01a0 2024-03-06T07:26:23.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 -- 172.21.15.122:0/3944485151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c0d01a0 msgr2=0x7f7f3c0b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 --2- 172.21.15.122:0/3944485151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c0d01a0 0x7f7f3c0b33a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c070d10 tx=0x7f7f3402e0f0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 -- 172.21.15.122:0/3944485151 shutdown_connections 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 --2- 172.21.15.122:0/3944485151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c0d01a0 0x7f7f3c0b33a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 --2- 172.21.15.122:0/3944485151 >> v2:172.21.15.169:3300/0 conn(0x7f7f3c1586e0 0x7f7f3c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 -- 172.21.15.122:0/3944485151 >> 172.21.15.122:0/3944485151 conn(0x7f7f3c06b190 msgr2=0x7f7f3c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 -- 172.21.15.122:0/3944485151 shutdown_connections 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.550+0000 7f7f4213e640 1 -- 172.21.15.122:0/3944485151 wait complete. 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.551+0000 7f7f4213e640 1 Processor -- start 2024-03-06T07:26:23.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.551+0000 7f7f4213e640 1 -- start start 2024-03-06T07:26:23.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f4213e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f3c0aef80 0x7f7f3c06cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f4213e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f3c0d01a0 0x7f7f3c1f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f4213e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 0x7f7f3c1f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f4213e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c071350 con 0x7f7f3c1586e0 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f4213e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c15bde0 con 0x7f7f3c0aef80 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f4213e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0714d0 con 0x7f7f3c0d01a0 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 0x7f7f3c1f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 0x7f7f3c1f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56740/0 (socket says 172.21.15.122:56740) 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f3bfff640 1 -- 172.21.15.122:0/221664866 learned_addr learned my addr 172.21.15.122:0/221664866 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f3b7fe640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f3c0aef80 0x7f7f3c06cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.552+0000 7f7f3affd640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f3c0d01a0 0x7f7f3c1f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f3bfff640 1 -- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f3c0d01a0 msgr2=0x7f7f3c1f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f3bfff640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f3c0d01a0 0x7f7f3c1f33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f3bfff640 1 -- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f3c0aef80 msgr2=0x7f7f3c06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f3bfff640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f3c0aef80 0x7f7f3c06cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f3bfff640 1 -- 172.21.15.122:0/221664866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c06d770 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f3bfff640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 0x7f7f3c1f3910 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7f34038040 tx=0x7f7f34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34045020 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f34009e30 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c06da50 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3402ea90 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.553+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c06e010 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.554+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c0732b0 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.554+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f34048020 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.555+0000 7f7f23fff640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2806b4d0 0x7f7f2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.555+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f340aeaa0 con 0x7f7f3c1586e0 2024-03-06T07:26:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.555+0000 7f7f3b7fe640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2806b4d0 0x7f7f2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.555+0000 7f7f3b7fe640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2806b4d0 0x7f7f2806d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c071db0 tx=0x7f7f2c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.559+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f3404f030 con 0x7f7f3c1586e0 2024-03-06T07:26:23.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.796+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7f3c0724c0 con 0x7f7f3c1586e0 2024-03-06T07:26:23.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.798+0000 7f7f23fff640 1 -- 172.21.15.122:0/221664866 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7f3407d250 con 0x7f7f3c1586e0 2024-03-06T07:26:23.801 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:26:23.801 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:26:23.801 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:26:23.801 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:26:23.801 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:23.801 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2806b4d0 msgr2=0x7f7f2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2806b4d0 0x7f7f2806d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c071db0 tx=0x7f7f2c009190 comp rx=0 tx=0).stop 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 msgr2=0x7f7f3c1f3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 0x7f7f3c1f3910 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7f34038040 tx=0x7f7f34004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 shutdown_connections 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2806b4d0 0x7f7f2806d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f3c1586e0 0x7f7f3c1f3910 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f3c0d01a0 0x7f7f3c1f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 --2- 172.21.15.122:0/221664866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f3c0aef80 0x7f7f3c06cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.801+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 >> 172.21.15.122:0/221664866 conn(0x7f7f3c06b190 msgr2=0x7f7f3c0cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.802+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 shutdown_connections 2024-03-06T07:26:23.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.802+0000 7f7f4213e640 1 -- 172.21.15.122:0/221664866 wait complete. 2024-03-06T07:26:23.814 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:23.835 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:23.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:23.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.914+0000 7f52d373e640 1 Processor -- start 2024-03-06T07:26:23.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.914+0000 7f52d373e640 1 -- start start 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d373e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0d00c0 0x7f52cc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d373e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f52cc0cf850 0x7f52cc0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d373e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52cc0d09e0 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d373e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52cc06ca90 con 0x7f52cc0cf850 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d373e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc06df30 con 0x7f52cc0d00c0 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d0cd9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f52cc0cf850 0x7f52cc0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d0cd9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f52cc0cf850 0x7f52cc0cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56742/0 (socket says 172.21.15.122:56742) 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d0cd9640 1 -- 172.21.15.122:0/2343861678 learned_addr learned my addr 172.21.15.122:0/2343861678 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.915+0000 7f52d14da640 1 --2- 172.21.15.122:0/2343861678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0d00c0 0x7f52cc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52d0cd9640 1 -- 172.21.15.122:0/2343861678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0d00c0 msgr2=0x7f52cc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52d0cd9640 1 --2- 172.21.15.122:0/2343861678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0d00c0 0x7f52cc0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52d0cd9640 1 -- 172.21.15.122:0/2343861678 >> v1:172.21.15.169:6789/0 conn(0x7f52cc0ce1b0 legacy=0x7f52cc0ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52d0cd9640 1 -- 172.21.15.122:0/2343861678 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc15e380 con 0x7f52cc0cf850 2024-03-06T07:26:23.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52d0cd9640 1 --2- 172.21.15.122:0/2343861678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0cf850 0x7f52cc0cd3c0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009870 tx=0x7f52c002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af68c982536ac857 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52cbfff640 1 -- 172.21.15.122:0/2343861678 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0034070 con 0x7f52cc0cf850 2024-03-06T07:26:23.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52cbfff640 1 -- 172.21.15.122:0/2343861678 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0030840 con 0x7f52cc0cf850 2024-03-06T07:26:23.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.916+0000 7f52cbfff640 1 -- 172.21.15.122:0/2343861678 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0030c10 con 0x7f52cc0cf850 2024-03-06T07:26:23.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 -- 172.21.15.122:0/2343861678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0cf850 msgr2=0x7f52cc0cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 --2- 172.21.15.122:0/2343861678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0cf850 0x7f52cc0cd3c0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009870 tx=0x7f52c002b980 comp rx=0 tx=0).stop 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 -- 172.21.15.122:0/2343861678 shutdown_connections 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 --2- 172.21.15.122:0/2343861678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0cf850 0x7f52cc0cd3c0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 --2- 172.21.15.122:0/2343861678 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0d00c0 0x7f52cc0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 -- 172.21.15.122:0/2343861678 >> 172.21.15.122:0/2343861678 conn(0x7f52cc0b35a0 msgr2=0x7f52cc0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 -- 172.21.15.122:0/2343861678 shutdown_connections 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.917+0000 7f52d373e640 1 -- 172.21.15.122:0/2343861678 wait complete. 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.918+0000 7f52d373e640 1 Processor -- start 2024-03-06T07:26:23.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.918+0000 7f52d373e640 1 -- start start 2024-03-06T07:26:23.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d373e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 0x7f52cc14be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d373e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0cf850 0x7f52cc0b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d373e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52cc0d00c0 0x7f52cc0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d373e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc06c440 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d373e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc06c160 con 0x7f52cc0d00c0 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d373e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc06c720 con 0x7f52cc0cf850 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 0x7f52cc14be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 0x7f52cc14be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:56754/0 (socket says 172.21.15.122:56754) 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 -- 172.21.15.122:0/255475953 learned_addr learned my addr 172.21.15.122:0/255475953 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d0cd9640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0cf850 0x7f52cc0b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d1cdb640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52cc0d00c0 0x7f52cc0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 -- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0cf850 msgr2=0x7f52cc0b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0cf850 0x7f52cc0b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 -- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52cc0d00c0 msgr2=0x7f52cc0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52cc0d00c0 0x7f52cc0b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 -- 172.21.15.122:0/255475953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc0c7e70 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.919+0000 7f52d14da640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 0x7f52cc14be00 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00e870 tx=0x7f52bc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.920+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc017070 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.920+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00c940 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.920+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52cc0c8150 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.920+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00cd30 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.920+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc14c850 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.921+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52cc068150 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.921+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52bc01a070 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.922+0000 7f52c9ffb640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f52b806b500 0x7f52b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.922+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52bc08a030 con 0x7f52cc0ce1b0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.922+0000 7f52d0cd9640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f52b806b500 0x7f52b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.922+0000 7f52d0cd9640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f52b806b500 0x7f52b806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f52c0036000 tx=0x7f52c0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:23.926+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52cc068150 con 0x7f52cc0ce1b0 2024-03-06T07:26:24.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.134+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52cc0b5610 con 0x7f52cc0ce1b0 2024-03-06T07:26:24.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.135+0000 7f52c9ffb640 1 -- 172.21.15.122:0/255475953 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f52bc058860 con 0x7f52cc0ce1b0 2024-03-06T07:26:24.137 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:24.137 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":47,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/2641321695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":2641321695},{"type":"v1","addr":"172.21.15.169:6834","nonce":2641321695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6805,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/3022103545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3022103545},{"type":"v1","addr":"172.21.15.122:6836","nonce":3022103545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2175287267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2175287267},{"type":"v1","addr":"172.21.15.122:6837","nonce":2175287267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:21.441727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6817":{"gid":6817,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.169:6835/2187235474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2187235474},{"type":"v1","addr":"172.21.15.169:6835","nonce":2187235474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6817,"qdb_cluster":[6817]},"id":7}]} 2024-03-06T07:26:24.137 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 47 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f52b806b500 msgr2=0x7f52b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f52b806b500 0x7f52b806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f52c0036000 tx=0x7f52c0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 msgr2=0x7f52cc14be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 0x7f52cc14be00 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00e870 tx=0x7f52bc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 shutdown_connections 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f52b806b500 0x7f52b806d9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52cc0d00c0 0x7f52cc0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f52cc0cf850 0x7f52cc0b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 --2- 172.21.15.122:0/255475953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52cc0ce1b0 0x7f52cc14be00 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 >> 172.21.15.122:0/255475953 conn(0x7f52cc0b35a0 msgr2=0x7f52cc0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 shutdown_connections 2024-03-06T07:26:24.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.137+0000 7f52d373e640 1 -- 172.21.15.122:0/255475953 wait complete. 2024-03-06T07:26:24.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.451683+0000', 'modified': '2024-03-06T07:26:21.441727+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6817}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6817': {'gid': 6817, 'name': 'd', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.169:6835/2187235474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6833', 'nonce': 2187235474}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 2187235474}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6817, 'qdb_cluster': [6817]} 2024-03-06T07:26:24.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:24.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:26:24.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.654+0000 7f356cc9d640 1 Processor -- start 2024-03-06T07:26:24.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.654+0000 7f356cc9d640 1 -- start start 2024-03-06T07:26:24.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356cc9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 0x7f3568071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356cc9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f35680cebc0 0x7f356815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356cc9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3568072490 con 0x7f356815c510 2024-03-06T07:26:24.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356cc9d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35680d0090 con 0x7f35680cebc0 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356cc9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680d0370 con 0x7f356815bbf0 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f355ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f35680cebc0 0x7f356815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356659c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 0x7f3568071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f3566d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f356815c510 0x7f35680cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:32796/0 (socket says 172.21.15.122:32796) 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356659c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 0x7f3568071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47526/0 (socket says 172.21.15.122:47526) 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f355ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f35680cebc0 0x7f356815bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:60668/0 (socket says 172.21.15.122:60668) 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f3566d9d640 1 -- 172.21.15.122:0/3960269333 learned_addr learned my addr 172.21.15.122:0/3960269333 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356659c640 1 -- 172.21.15.122:0/3960269333 >> v2:172.21.15.169:3300/0 conn(0x7f35680cebc0 msgr2=0x7f356815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356659c640 1 --2- 172.21.15.122:0/3960269333 >> v2:172.21.15.169:3300/0 conn(0x7f35680cebc0 0x7f356815bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356659c640 1 -- 172.21.15.122:0/3960269333 >> v1:172.21.15.169:6789/0 conn(0x7f356815c510 legacy=0x7f35680cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.655+0000 7f356659c640 1 -- 172.21.15.122:0/3960269333 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35680d0650 con 0x7f356815bbf0 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.656+0000 7f356659c640 1 --2- 172.21.15.122:0/3960269333 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 0x7f3568071e00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f35580097a0 tx=0x7f355802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c109f6b26621bbf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.656+0000 7f3565d9b640 1 -- 172.21.15.122:0/3960269333 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558035020 con 0x7f356815bbf0 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.656+0000 7f3565d9b640 1 -- 172.21.15.122:0/3960269333 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558031830 con 0x7f356815bbf0 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.656+0000 7f3565d9b640 1 -- 172.21.15.122:0/3960269333 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558031c00 con 0x7f356815bbf0 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 -- 172.21.15.122:0/3960269333 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 msgr2=0x7f3568071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 --2- 172.21.15.122:0/3960269333 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 0x7f3568071e00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f35580097a0 tx=0x7f355802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 -- 172.21.15.122:0/3960269333 shutdown_connections 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 --2- 172.21.15.122:0/3960269333 >> v2:172.21.15.169:3300/0 conn(0x7f35680cebc0 0x7f356815bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 --2- 172.21.15.122:0/3960269333 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815bbf0 0x7f3568071e00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 -- 172.21.15.122:0/3960269333 >> 172.21.15.122:0/3960269333 conn(0x7f35680b1c80 msgr2=0x7f35680cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 -- 172.21.15.122:0/3960269333 shutdown_connections 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.657+0000 7f356cc9d640 1 -- 172.21.15.122:0/3960269333 wait complete. 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.658+0000 7f356cc9d640 1 Processor -- start 2024-03-06T07:26:24.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.658+0000 7f356cc9d640 1 -- start start 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356cc9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 0x7f356814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356cc9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f356815bbf0 0x7f356813a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356cc9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815c510 0x7f3568149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356cc9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680b3800 con 0x7f35680cebc0 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356cc9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680b3680 con 0x7f356815bbf0 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356cc9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680cded0 con 0x7f356815c510 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 0x7f356814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 0x7f356814c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58874/0 (socket says 172.21.15.122:58874) 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 -- 172.21.15.122:0/1478340956 learned_addr learned my addr 172.21.15.122:0/1478340956 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f355ffff640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f356815bbf0 0x7f356813a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f3566d9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815c510 0x7f3568149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 -- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815c510 msgr2=0x7f3568149300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815c510 0x7f3568149300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 -- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f356815bbf0 msgr2=0x7f356813a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f356815bbf0 0x7f356813a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 -- 172.21.15.122:0/1478340956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356814cf10 con 0x7f35680cebc0 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.659+0000 7f356659c640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 0x7f356814c9d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f355802bee0 tx=0x7f35580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.660+0000 7f355effd640 1 -- 172.21.15.122:0/1478340956 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558035040 con 0x7f35680cebc0 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.660+0000 7f355effd640 1 -- 172.21.15.122:0/1478340956 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355803b920 con 0x7f35680cebc0 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.660+0000 7f355effd640 1 -- 172.21.15.122:0/1478340956 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355803bd10 con 0x7f35680cebc0 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.660+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356814d360 con 0x7f35680cebc0 2024-03-06T07:26:24.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.660+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356814d920 con 0x7f35680cebc0 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.661+0000 7f355effd640 1 -- 172.21.15.122:0/1478340956 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3558038020 con 0x7f35680cebc0 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.662+0000 7f355effd640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f354806b4d0 0x7f354806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.662+0000 7f355effd640 1 -- 172.21.15.122:0/1478340956 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35580ac1a0 con 0x7f35680cebc0 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.662+0000 7f355ffff640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f354806b4d0 0x7f354806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.662+0000 7f355ffff640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f354806b4d0 0x7f354806d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3568144bb0 tx=0x7f3550009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 Processor -- start 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 -- start start 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b2c870 0x563d53b2cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b308c0 0x563d53b32360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b5ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53a966b0 con 0x563d53b5c600 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53a3fa60 con 0x563d53b308c0 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.663+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53a92750 con 0x563d53b2c870 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b5ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b5ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58878/0 (socket says 172.21.15.122:58878) 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 -- 172.21.15.122:0/1543799926 learned_addr learned my addr 172.21.15.122:0/1543799926 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f356659c640 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b2c870 0x563d53b2cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f355ffff640 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b308c0 0x563d53b32360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 -- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b2c870 msgr2=0x563d53b2cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b2c870 0x563d53b2cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 -- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b308c0 msgr2=0x563d53b32360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b308c0 0x563d53b32360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 -- 172.21.15.122:0/1543799926 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d53a5df30 con 0x563d53b5c600 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3566d9d640 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b5ca80 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f356000f280 tx=0x7f35600019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc6bb8263695dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3547fff640 1 -- 172.21.15.122:0/1543799926 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356000fea0 con 0x563d53b5c600 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3547fff640 1 -- 172.21.15.122:0/1543799926 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560018070 con 0x563d53b5c600 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.664+0000 7f3547fff640 1 -- 172.21.15.122:0/1543799926 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35600074e0 con 0x563d53b5c600 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 -- 172.21.15.122:0/1543799926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 msgr2=0x563d53b5ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b5ca80 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f356000f280 tx=0x7f35600019f0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 -- 172.21.15.122:0/1543799926 shutdown_connections 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b5ca80 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b308c0 0x563d53b32360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 --2- 172.21.15.122:0/1543799926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b2c870 0x563d53b2cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 -- 172.21.15.122:0/1543799926 >> 172.21.15.122:0/1543799926 conn(0x563d53b2e8f0 msgr2=0x563d53b301b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 -- 172.21.15.122:0/1543799926 shutdown_connections 2024-03-06T07:26:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 -- 172.21.15.122:0/1543799926 wait complete. 2024-03-06T07:26:24.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.665+0000 7f356f964740 1 -- start start 2024-03-06T07:26:24.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 Processor -- start 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b57860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b51490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b50d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53b30d50 con 0x563d53b5c600 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53a52ea0 con 0x563d53b50920 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.666+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53b31030 con 0x563d53b5ea30 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356f964740 10 client.0 _ll_register_callbacks cb 0x563d53a52870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b57860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b57860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58884/0 (socket says 172.21.15.122:58884) 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 -- 172.21.15.122:0/2208246894 learned_addr learned my addr 172.21.15.122:0/2208246894 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f355ffff640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b51490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f3566d9d640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b50d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 -- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 msgr2=0x563d53b51490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b51490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 -- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 msgr2=0x563d53b50d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b50d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 -- 172.21.15.122:0/2208246894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d53a53270 con 0x563d53b5c600 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f3566d9d640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b50d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356659c640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b57860 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f355802b910 tx=0x7f355807a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410a4e9b933594c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f35457fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f35457fa640 1 -- 172.21.15.122:0/2208246894 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355804b030 con 0x563d53b5c600 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f35457fa640 1 -- 172.21.15.122:0/2208246894 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3558093d70 con 0x563d53b5c600 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3560018070 con 0x563d53b5c600 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f35457fa640 1 -- 172.21.15.122:0/2208246894 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f355807a9e0 con 0x563d53b5c600 2024-03-06T07:26:24.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356f964740 10 client.7605 fetch_fsmap learned FSMap version 47 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x563d537ab2d0 con 0x563d53b5c600 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f35457fa640 1 -- 172.21.15.122:0/2208246894 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560018070 con 0x563d53b5c600 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f35457fa640 1 -- 172.21.15.122:0/2208246894 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3558082bb0 con 0x563d53b5c600 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.667+0000 7f356f964740 10 client.7605 fetch_fsmap finished waiting for FSMap version 47 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f356f964740 10 client.7605 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f356f964740 20 client.7605 populate_metadata read hostname 'smithi122' 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f356f964740 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b58da0 0x563d53b591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f356f964740 4 client.7605 mds_command: new command op to 6817 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 --> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d53aa3bf0 con 0x563d53b58da0 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f355ffff640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b58da0 0x563d53b591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f355ffff640 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b58da0 0x563d53b591c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=917e4f67b1274b33 server_cookie=9c46a8b7d4c55908 in_seq=0 out_seq=0 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.668+0000 7f35457fa640 10 client.7605 ms_handle_connect on v2:172.21.15.169:6833/2187235474 2024-03-06T07:26:24.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f35457fa640 1 -- 172.21.15.122:0/2208246894 <== mds.0 v2:172.21.15.169:6833/2187235474 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d53aa3bf0 con 0x563d53b58da0 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f35457fa640 10 client.7605 handle_command_reply: tid=0 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 client.7605 shutdown 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 msgr2=0x563d53b57860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b57860 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f355802b910 tx=0x7f355807a520 comp rx=0 tx=0).stop 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 shutdown_connections 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b50d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b51490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b58da0 0x563d53b591c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 --2- 172.21.15.122:0/2208246894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5c600 0x563d53b57860 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f35457fa640 0 client.7605 ms_handle_reset on v2:172.21.15.169:6833/2187235474 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 >> 172.21.15.122:0/2208246894 conn(0x563d53b2c870 msgr2=0x563d53b5a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 shutdown_connections 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 1 -- 172.21.15.122:0/2208246894 wait complete. 2024-03-06T07:26:24.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.669+0000 7f356f964740 20 client.7605 trim_cache size 0 max 16384 2024-03-06T07:26:24.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 Processor -- start 2024-03-06T07:26:24.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 -- start start 2024-03-06T07:26:24.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b5c600 0x563d53b5b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b5bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35681fd1c0 con 0x563d53b50920 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35681fc5c0 con 0x563d53b5c600 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35681fcbc0 con 0x563d53b5ea30 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.686+0000 7f356659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58898/0 (socket says 172.21.15.122:58898) 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 -- 172.21.15.122:0/581964131 learned_addr learned my addr 172.21.15.122:0/581964131 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f355ffff640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b5c600 0x563d53b5b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f3566d9d640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b5bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 -- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 msgr2=0x563d53b5bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b5bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 -- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b5c600 msgr2=0x563d53b5b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b5c600 0x563d53b5b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 -- 172.21.15.122:0/581964131 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d53b7b930 con 0x563d53b50920 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f3566d9d640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b5bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f355ffff640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b5c600 0x563d53b5b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f356659c640 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 0x563d53b8a070 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f3558076d00 tx=0x7f355807d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ee46ddcf27e599 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f35477fe640 1 -- 172.21.15.122:0/581964131 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355804b070 con 0x563d53b50920 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f35477fe640 1 -- 172.21.15.122:0/581964131 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355807ea80 con 0x563d53b50920 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.687+0000 7f35477fe640 1 -- 172.21.15.122:0/581964131 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558083040 con 0x563d53b50920 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 -- 172.21.15.122:0/581964131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 msgr2=0x563d53b8a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 0x563d53b8a070 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f3558076d00 tx=0x7f355807d000 comp rx=0 tx=0).stop 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 -- 172.21.15.122:0/581964131 shutdown_connections 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5ea30 0x563d53b5bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b5c600 0x563d53b5b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 --2- 172.21.15.122:0/581964131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 -- 172.21.15.122:0/581964131 >> 172.21.15.122:0/581964131 conn(0x563d53b2c870 msgr2=0x563d53b50d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 -- 172.21.15.122:0/581964131 shutdown_connections 2024-03-06T07:26:24.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 -- 172.21.15.122:0/581964131 wait complete. 2024-03-06T07:26:24.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 -- start start 2024-03-06T07:26:24.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.688+0000 7f356f964740 1 Processor -- start 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5c600 0x563d53bca030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 0x563d53b91120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53b30d50 con 0x563d53b5ea30 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d53a52ea0 con 0x563d53b50920 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d53b520a0 con 0x563d53b5c600 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356f964740 10 client.0 _ll_register_callbacks cb 0x7f3558076d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 0x563d53b91120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 0x563d53b91120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58900/0 (socket says 172.21.15.122:58900) 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 -- 172.21.15.122:0/4149326923 learned_addr learned my addr 172.21.15.122:0/4149326923 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f355ffff640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5c600 0x563d53bca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f356659c640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 -- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5c600 msgr2=0x563d53bca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5c600 0x563d53bca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 -- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 msgr2=0x563d53b8a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.689+0000 7f3566d9d640 1 -- 172.21.15.122:0/4149326923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d538756b0 con 0x563d53b5ea30 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f3566d9d640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 0x563d53b91120 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3560015040 tx=0x7f3560001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65169ad6c9c63570 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f35457fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f35457fa640 1 -- 172.21.15.122:0/4149326923 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560007820 con 0x563d53b5ea30 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f35457fa640 1 -- 172.21.15.122:0/4149326923 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35600079c0 con 0x563d53b5ea30 2024-03-06T07:26:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f355807ea80 con 0x563d53b5ea30 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f35457fa640 1 -- 172.21.15.122:0/4149326923 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3560030050 con 0x563d53b5ea30 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 10 client.7611 fetch_fsmap learned FSMap version 47 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x563d53b90c00 con 0x563d53b5ea30 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f35457fa640 1 -- 172.21.15.122:0/4149326923 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355807ea80 con 0x563d53b5ea30 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f35457fa640 1 -- 172.21.15.122:0/4149326923 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3560033020 con 0x563d53b5ea30 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 10 client.7611 fetch_fsmap finished waiting for FSMap version 47 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 10 client.7611 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 20 client.7611 populate_metadata read hostname 'smithi122' 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b4e090 0x563d53b30f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 4 client.7611 mds_command: new command op to 6817 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 --> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d53b572b0 con 0x563d53b4e090 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.690+0000 7f356659c640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b4e090 0x563d53b30f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f35457fa640 10 client.7611 ms_handle_connect on v2:172.21.15.169:6833/2187235474 2024-03-06T07:26:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356659c640 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b4e090 0x563d53b30f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f0bc66bfff100b7 server_cookie=ba57473086acfd7 in_seq=0 out_seq=0 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f35457fa640 1 -- 172.21.15.122:0/4149326923 <== mds.0 v2:172.21.15.169:6833/2187235474 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x563d53b572b0 con 0x563d53b4e090 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f35457fa640 10 client.7611 handle_command_reply: tid=0 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 client.7611 shutdown 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 msgr2=0x563d53b91120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 0x563d53b91120 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3560015040 tx=0x7f3560001250 comp rx=0 tx=0).stop 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 shutdown_connections 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:6833/2187235474,v1:172.21.15.169:6835/2187235474] conn(0x563d53b4e090 0x563d53b30f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d53b5ea30 0x563d53b91120 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563d53b5c600 0x563d53bca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f35457fa640 0 client.7611 ms_handle_reset on v2:172.21.15.169:6833/2187235474 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 --2- 172.21.15.122:0/4149326923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563d53b50920 0x563d53b8a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 >> 172.21.15.122:0/4149326923 conn(0x563d53b2c870 msgr2=0x563d53b5b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 shutdown_connections 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 1 -- 172.21.15.122:0/4149326923 wait complete. 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.691+0000 7f356f964740 20 client.7611 trim_cache size 0 max 16384 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 6817, 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:26:24.701 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 4.5902477160000004, 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 47, 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 93, 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 93, 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 8.9368029530000008 2024-03-06T07:26:24.702 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f354806b4d0 msgr2=0x7f354806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f354806b4d0 0x7f354806d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3568144bb0 tx=0x7f3550009190 comp rx=0 tx=0).stop 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 msgr2=0x7f356814c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 0x7f356814c9d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f355802bee0 tx=0x7f35580046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 shutdown_connections 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f354806b4d0 0x7f354806d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f356815c510 0x7f3568149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f356815bbf0 0x7f356813a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 --2- 172.21.15.122:0/1478340956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35680cebc0 0x7f356814c9d0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 >> 172.21.15.122:0/1478340956 conn(0x7f35680b1c80 msgr2=0x7f3568071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 shutdown_connections 2024-03-06T07:26:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.692+0000 7f356cc9d640 1 -- 172.21.15.122:0/1478340956 wait complete. 2024-03-06T07:26:24.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.802+0000 7f6927f5e640 1 Processor -- start 2024-03-06T07:26:24.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.802+0000 7f6927f5e640 1 -- start start 2024-03-06T07:26:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6927f5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f692015e0f0 0x7f69200cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6927f5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200d0660 0x7f69200b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6927f5e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692015e4d0 con 0x7f69200cf850 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6927f5e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200b1410 con 0x7f692015e0f0 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6927f5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692006c8b0 con 0x7f69200d0660 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6925cfa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f692015e0f0 0x7f69200cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6925cfa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f692015e0f0 0x7f69200cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58908/0 (socket says 172.21.15.122:58908) 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f6925cfa640 1 -- 172.21.15.122:0/3278229469 learned_addr learned my addr 172.21.15.122:0/3278229469 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.803+0000 7f69264fb640 1 --2- 172.21.15.122:0/3278229469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200d0660 0x7f69200b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6925cfa640 1 -- 172.21.15.122:0/3278229469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200d0660 msgr2=0x7f69200b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6925cfa640 1 --2- 172.21.15.122:0/3278229469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200d0660 0x7f69200b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6925cfa640 1 -- 172.21.15.122:0/3278229469 >> v1:172.21.15.169:6789/0 conn(0x7f69200cf850 legacy=0x7f69200cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6925cfa640 1 -- 172.21.15.122:0/3278229469 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69200cdac0 con 0x7f692015e0f0 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6925cfa640 1 --2- 172.21.15.122:0/3278229469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200cd3c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f691c009e50 tx=0x7f691c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634ce12f8b761455 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6924cf8640 1 -- 172.21.15.122:0/3278229469 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c02cb00 con 0x7f692015e0f0 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6924cf8640 1 -- 172.21.15.122:0/3278229469 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c02cca0 con 0x7f692015e0f0 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.804+0000 7f6924cf8640 1 -- 172.21.15.122:0/3278229469 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c034610 con 0x7f692015e0f0 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 -- 172.21.15.122:0/3278229469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 msgr2=0x7f69200cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 --2- 172.21.15.122:0/3278229469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200cd3c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f691c009e50 tx=0x7f691c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 -- 172.21.15.122:0/3278229469 shutdown_connections 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 --2- 172.21.15.122:0/3278229469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200d0660 0x7f69200b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 --2- 172.21.15.122:0/3278229469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200cd3c0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 -- 172.21.15.122:0/3278229469 >> 172.21.15.122:0/3278229469 conn(0x7f69200b35a0 msgr2=0x7f69200b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 -- 172.21.15.122:0/3278229469 shutdown_connections 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.805+0000 7f6927f5e640 1 -- 172.21.15.122:0/3278229469 wait complete. 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.806+0000 7f6927f5e640 1 Processor -- start 2024-03-06T07:26:24.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.806+0000 7f6927f5e640 1 -- start start 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6927f5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200cf850 0x7f6920153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6927f5e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69200d0660 0x7f69200b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6927f5e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6927f5e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69200d0cc0 con 0x7f692015e0f0 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6927f5e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692006c540 con 0x7f69200d0660 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6927f5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69200b2af0 con 0x7f69200cf850 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200d1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58916/0 (socket says 172.21.15.122:58916) 2024-03-06T07:26:24.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 -- 172.21.15.122:0/1233779136 learned_addr learned my addr 172.21.15.122:0/1233779136 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f6925cfa640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200cf850 0x7f6920153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69254f9640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69200d0660 0x7f69200b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 -- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200cf850 msgr2=0x7f6920153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200cf850 0x7f6920153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 -- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69200d0660 msgr2=0x7f69200b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69200d0660 0x7f69200b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 -- 172.21.15.122:0/1233779136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69200b7040 con 0x7f692015e0f0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.807+0000 7f69264fb640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200d1a50 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f691800e850 tx=0x7f691800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.808+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918017070 con 0x7f692015e0f0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.808+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691800c8e0 con 0x7f692015e0f0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.808+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69200b7320 con 0x7f692015e0f0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.808+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691800ccd0 con 0x7f692015e0f0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.808+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69200ca310 con 0x7f692015e0f0 2024-03-06T07:26:24.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.809+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69200ca510 con 0x7f692015e0f0 2024-03-06T07:26:24.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.809+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f691801a070 con 0x7f692015e0f0 2024-03-06T07:26:24.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.809+0000 7f69167fc640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f690406b4d0 0x7f690406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.810+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f691808a030 con 0x7f692015e0f0 2024-03-06T07:26:24.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.810+0000 7f6925cfa640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f690406b4d0 0x7f690406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.810+0000 7f6925cfa640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f690406b4d0 0x7f690406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f691c035000 tx=0x7f691c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:24.814+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69200ca510 con 0x7f692015e0f0 2024-03-06T07:26:25.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.022+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f692004f330 con 0x7f692015e0f0 2024-03-06T07:26:25.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.023+0000 7f69167fc640 1 -- 172.21.15.122:0/1233779136 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7f69180588f0 con 0x7f692015e0f0 2024-03-06T07:26:25.025 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f690406b4d0 msgr2=0x7f690406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f690406b4d0 0x7f690406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f691c035000 tx=0x7f691c030000 comp rx=0 tx=0).stop 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 msgr2=0x7f69200d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200d1a50 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f691800e850 tx=0x7f691800ed10 comp rx=0 tx=0).stop 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 shutdown_connections 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f690406b4d0 0x7f690406d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f692015e0f0 0x7f69200d1a50 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69200d0660 0x7f69200b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 --2- 172.21.15.122:0/1233779136 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69200cf850 0x7f6920153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 >> 172.21.15.122:0/1233779136 conn(0x7f69200b35a0 msgr2=0x7f692006cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 shutdown_connections 2024-03-06T07:26:25.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25.025+0000 7f6927f5e640 1 -- 172.21.15.122:0/1233779136 wait complete. 2024-03-06T07:26:25.037 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:25.037 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:25.038 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:25.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.038 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:26:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42456sec preferred_lft 42456sec 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.055 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:26:25.056 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:25.056 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.056 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:26:25.056 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:25.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.056 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:26:25.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.124 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:26:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:26:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:26:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:26:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:25.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.180 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:26:25.180 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:26:25.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:26:25.181 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:26:25.181 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:26:25.181 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:26:25.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.275 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:26:25.290 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:25.290 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:25.290 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:25.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.290 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:26:25.357 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:25.357 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:25.357 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.413 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:25.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:25.413 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.466 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:26:25.535 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:25.589 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:26:25.589 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:26:25.589 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:26:25.589 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:26:25.590 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:26:25.591 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.592 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.592 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.592 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.592 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:26:25.644 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:25.644 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:25.687 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:26:25.712 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:26:25.769 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:26:25.770 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.771 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.772 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:26:25.773 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:26:25.771+0000 7f717d356480 -1 init, newargv = 0x556ca71e0f60 newargc=15 2024-03-06T07:26:25.773 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[68944]: starting ceph client 2024-03-06T07:26:25.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[68944]: starting fuse 2024-03-06T07:26:25.787 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:26:25.787 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:26:25.788 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:25.789 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:26:25.907 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.68944.asok 2024-03-06T07:26:25.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:25.917 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68944.asok 2024-03-06T07:26:25.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68944.asok status 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "68944", 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 7623, 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 7623 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 4144397787 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 4144397787 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.7623 192.168.0.1:0/4144397787", 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/4144397787", 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 47, 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 93, 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:26.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:26.060 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:26:26.060 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:26.075 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:26:26.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:26.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:26.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:26.081 INFO:tasks.ceph.mds.d.smithi169.stderr:2024-03-06T07:26:26.079+0000 7fb5b172d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:26.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.232 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:26:26.232 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:26:26.235 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:26:26.236 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.237 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.238 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:26:26.239 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.240 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:26:26.241 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:26:26.242 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:26:26.243 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.244 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.245 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:26:26.246 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:26:26.247 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:26:26.248 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.249 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:26:26.250 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.251 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.252 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:26:26.253 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.254 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.256 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.345 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:26:26.345 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:26:26.346 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.351 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.353 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.358 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.361 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:26:26.361 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:26:26.361 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.361 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:26:26.361 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.362 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.363 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:26:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:26:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.369 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.370 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:26:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.376 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:26.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.516+0000 7fe70c466640 1 Processor -- start 2024-03-06T07:26:26.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.516+0000 7fe70c466640 1 -- start start 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.517+0000 7fe70c466640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe70415e3b0 0x7fe7040d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.517+0000 7fe70c466640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7040d2b70 0x7fe7040d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.517+0000 7fe70c466640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040cebb0 con 0x7fe70415e790 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.517+0000 7fe70c466640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7040b7c60 con 0x7fe70415e3b0 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.517+0000 7fe70c466640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cee90 con 0x7fe7040d2b70 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe70415e3b0 0x7fe7040d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe70415e3b0 0x7fe7040d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58920/0 (socket says 172.21.15.122:58920) 2024-03-06T07:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 -- 172.21.15.122:0/55274923 learned_addr learned my addr 172.21.15.122:0/55274923 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70aa03640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7040d2b70 0x7fe7040d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 -- 172.21.15.122:0/55274923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7040d2b70 msgr2=0x7fe7040d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7040d2b70 0x7fe7040d0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 -- 172.21.15.122:0/55274923 >> v1:172.21.15.122:6789/0 conn(0x7fe70415e790 legacy=0x7fe704162140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70a202640 1 -- 172.21.15.122:0/55274923 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040cf170 con 0x7fe70415e3b0 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe709200640 1 -- 172.21.15.122:0/55274923 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259311175 0 0) 0x7fe7040cebb0 con 0x7fe70415e790 2024-03-06T07:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.518+0000 7fe70aa03640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7040d2b70 0x7fe7040d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.519+0000 7fe70a202640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e3b0 0x7fe7040d05b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fe700009e80 tx=0x7fe70002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697179538be4810a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.519+0000 7fe709200640 1 -- 172.21.15.122:0/55274923 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70002ca50 con 0x7fe70415e3b0 2024-03-06T07:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.519+0000 7fe709200640 1 -- 172.21.15.122:0/55274923 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70002cbf0 con 0x7fe70415e3b0 2024-03-06T07:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.519+0000 7fe709200640 1 -- 172.21.15.122:0/55274923 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7000343f0 con 0x7fe70415e3b0 2024-03-06T07:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 -- 172.21.15.122:0/55274923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e3b0 msgr2=0x7fe7040d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e3b0 0x7fe7040d05b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fe700009e80 tx=0x7fe70002beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 -- 172.21.15.122:0/55274923 shutdown_connections 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7040d2b70 0x7fe7040d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 --2- 172.21.15.122:0/55274923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e3b0 0x7fe7040d05b0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 -- 172.21.15.122:0/55274923 >> 172.21.15.122:0/55274923 conn(0x7fe7040d3e00 msgr2=0x7fe7040d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 -- 172.21.15.122:0/55274923 shutdown_connections 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.520+0000 7fe70c466640 1 -- 172.21.15.122:0/55274923 wait complete. 2024-03-06T07:26:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.521+0000 7fe70c466640 1 Processor -- start 2024-03-06T07:26:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.521+0000 7fe70c466640 1 -- start start 2024-03-06T07:26:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d2b70 0x7fe70415fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415e3b0 0x7fe7040cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 0x7fe704156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cfc50 con 0x7fe70415e790 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cf730 con 0x7fe7040d2b70 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cdac0 con 0x7fe70415e3b0 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 0x7fe704156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 0x7fe704156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58924/0 (socket says 172.21.15.122:58924) 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 -- 172.21.15.122:0/3202993364 learned_addr learned my addr 172.21.15.122:0/3202993364 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe709a01640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415e3b0 0x7fe7040cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70a202640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d2b70 0x7fe70415fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 -- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415e3b0 msgr2=0x7fe7040cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415e3b0 0x7fe7040cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 -- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d2b70 msgr2=0x7fe70415fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d2b70 0x7fe70415fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 -- 172.21.15.122:0/3202993364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040c9e20 con 0x7fe70415e790 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70a202640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d2b70 0x7fe70415fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70aa03640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 0x7fe704156840 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00d5b0 tx=0x7fe6fc00da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc018070 con 0x7fe70415e790 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc004030 con 0x7fe70415e790 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe704161a10 con 0x7fe70415e790 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.522+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc004400 con 0x7fe70415e790 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.523+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7040cd750 con 0x7fe70415e790 2024-03-06T07:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.523+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe704068fb0 con 0x7fe70415e790 2024-03-06T07:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.524+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6fc014620 con 0x7fe70415e790 2024-03-06T07:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.524+0000 7fe6faffd640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6e806b4d0 0x7fe6e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.524+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6fc08a3e0 con 0x7fe70415e790 2024-03-06T07:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.524+0000 7fe70a202640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6e806b4d0 0x7fe6e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.525+0000 7fe70a202640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6e806b4d0 0x7fe6e806d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe700035000 tx=0x7fe700030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.529+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6fc0592f0 con 0x7fe70415e790 2024-03-06T07:26:26.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.703+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7040d2950 con 0x7fe70415e790 2024-03-06T07:26:26.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.704+0000 7fe6faffd640 1 -- 172.21.15.122:0/3202993364 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe6fc058bf0 con 0x7fe70415e790 2024-03-06T07:26:26.706 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.705+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6e806b4d0 msgr2=0x7fe6e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.705+0000 7fe70c466640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6e806b4d0 0x7fe6e806d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe700035000 tx=0x7fe700030000 comp rx=0 tx=0).stop 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.705+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 msgr2=0x7fe704156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.705+0000 7fe70c466640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 0x7fe704156840 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00d5b0 tx=0x7fe6fc00da70 comp rx=0 tx=0).stop 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 shutdown_connections 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6e806b4d0 0x7fe6e806d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe70415e790 0x7fe704156840 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415e3b0 0x7fe7040cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 --2- 172.21.15.122:0/3202993364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d2b70 0x7fe70415fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 >> 172.21.15.122:0/3202993364 conn(0x7fe7040d3e00 msgr2=0x7fe7040b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 shutdown_connections 2024-03-06T07:26:26.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.706+0000 7fe70c466640 1 -- 172.21.15.122:0/3202993364 wait complete. 2024-03-06T07:26:26.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.770 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.772 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.773 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.774 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.775 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:26:26.776 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.777 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.778 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.779 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.780 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.781 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.782 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.783 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.784 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.785 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.786 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.788 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:26.795 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.795 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.795 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:26.796 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.796 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:26.796 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.845 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:26.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:26:26.846 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:26.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:26:26.848 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:26.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:26:26.849 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:26.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:26:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.950+0000 7f1275439640 1 Processor -- start 2024-03-06T07:26:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.950+0000 7f1275439640 1 -- start start 2024-03-06T07:26:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f1275439640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1270161670 0x7f1270161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f1275439640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f12700b2da0 0x7f12701623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f1275439640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270162c70 con 0x7f12700cc680 2024-03-06T07:26:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f1275439640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12700b3590 con 0x7f1270161670 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f1275439640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127006cd80 con 0x7f12700b2da0 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f126effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1270161670 0x7f1270161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f126effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1270161670 0x7f1270161e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58926/0 (socket says 172.21.15.122:58926) 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.951+0000 7f126effd640 1 -- 172.21.15.122:0/3841245082 learned_addr learned my addr 172.21.15.122:0/3841245082 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126e7fc640 1 --2- 172.21.15.122:0/3841245082 >> v2:172.21.15.169:3300/0 conn(0x7f12700b2da0 0x7f12701623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126effd640 1 -- 172.21.15.122:0/3841245082 >> v2:172.21.15.169:3300/0 conn(0x7f12700b2da0 msgr2=0x7f12701623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126effd640 1 --2- 172.21.15.122:0/3841245082 >> v2:172.21.15.169:3300/0 conn(0x7f12700b2da0 0x7f12701623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126effd640 1 -- 172.21.15.122:0/3841245082 >> v1:172.21.15.169:6789/0 conn(0x7f12700cc680 legacy=0x7f12700ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126dffb640 1 -- 172.21.15.122:0/3841245082 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242756831 0 0) 0x7f1270162c70 con 0x7f12700cc680 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126effd640 1 -- 172.21.15.122:0/3841245082 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127006d060 con 0x7f1270161670 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.952+0000 7f126e7fc640 1 --2- 172.21.15.122:0/3841245082 >> v2:172.21.15.169:3300/0 conn(0x7f12700b2da0 0x7f12701623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.953+0000 7f126effd640 1 --2- 172.21.15.122:0/3841245082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270161e60 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1260009870 tx=0x7f126002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65cc9c272144618 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.953+0000 7f126dffb640 1 -- 172.21.15.122:0/3841245082 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260034070 con 0x7f1270161670 2024-03-06T07:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.953+0000 7f126dffb640 1 -- 172.21.15.122:0/3841245082 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260030860 con 0x7f1270161670 2024-03-06T07:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.953+0000 7f126dffb640 1 -- 172.21.15.122:0/3841245082 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260030c30 con 0x7f1270161670 2024-03-06T07:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 -- 172.21.15.122:0/3841245082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 msgr2=0x7f1270161e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 --2- 172.21.15.122:0/3841245082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270161e60 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1260009870 tx=0x7f126002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 -- 172.21.15.122:0/3841245082 shutdown_connections 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 --2- 172.21.15.122:0/3841245082 >> v2:172.21.15.169:3300/0 conn(0x7f12700b2da0 0x7f12701623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 --2- 172.21.15.122:0/3841245082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270161e60 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 -- 172.21.15.122:0/3841245082 >> 172.21.15.122:0/3841245082 conn(0x7f12700d1d50 msgr2=0x7f12700d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 -- 172.21.15.122:0/3841245082 shutdown_connections 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.954+0000 7f1275439640 1 -- 172.21.15.122:0/3841245082 wait complete. 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.956+0000 7f1275439640 1 Processor -- start 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.956+0000 7f1275439640 1 -- start start 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f1275439640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b2da0 0x7f12700c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f1275439640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12700cc680 0x7f12700b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f1275439640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f1275439640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127006d340 con 0x7f1270161670 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f1275439640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12701628e0 con 0x7f12700cc680 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f1275439640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127006d4c0 con 0x7f12700b2da0 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58936/0 (socket says 172.21.15.122:58936) 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 -- 172.21.15.122:0/1556950716 learned_addr learned my addr 172.21.15.122:0/1556950716 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126e7fc640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12700cc680 0x7f12700b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126effd640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b2da0 0x7f12700c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 -- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b2da0 msgr2=0x7f12700c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b2da0 0x7f12700c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 -- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12700cc680 msgr2=0x7f12700b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12700cc680 0x7f12700b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 -- 172.21.15.122:0/1556950716 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700c6cb0 con 0x7f1270161670 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.957+0000 7f126f7fe640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270155940 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f126800c760 tx=0x7f126800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.958+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268003da0 con 0x7f1270161670 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.958+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268005070 con 0x7f1270161670 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.958+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12700c6f90 con 0x7f1270161670 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.959+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268005460 con 0x7f1270161670 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.959+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12700c7550 con 0x7f1270161670 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.959+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127004f3d0 con 0x7f1270161670 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.960+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1268005600 con 0x7f1270161670 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.960+0000 7f12577fe640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125c06b4d0 0x7f125c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.960+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12680896b0 con 0x7f1270161670 2024-03-06T07:26:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.960+0000 7f126effd640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125c06b4d0 0x7f125c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.961+0000 7f126effd640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125c06b4d0 0x7f125c06d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1260036000 tx=0x7f126003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.969+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f127004f3d0 con 0x7f1270161670 2024-03-06T07:26:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.993+0000 7fc7c93f7640 1 Processor -- start 2024-03-06T07:26:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.993+0000 7ffb15ee1640 1 Processor -- start 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.993+0000 7fc7c93f7640 1 -- start start 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.993+0000 7ffb15ee1640 1 -- start start 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb15ee1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb1015e150 0x7ffb100cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c93f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c40723a0 0x7fc7c4165c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c93f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c415f0d0 con 0x7fc7c4155560 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c93f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c415f3b0 con 0x7fc7c415b300 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c93f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c415f690 con 0x7fc7c40723a0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb15ee1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ffb100cfba0 0x7ffb100d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb15ee1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100d1ea0 con 0x7ffb100d0e90 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb15ee1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb100d2ab0 con 0x7ffb100cfba0 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb15ee1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100d2180 con 0x7ffb1015e150 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb0ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffb100d0e90 0x7ffb100b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47796/0 (socket says 172.21.15.122:47796) 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb0ffff640 1 -- 172.21.15.122:0/296689161 learned_addr learned my addr 172.21.15.122:0/296689161 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c2ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc7c4155560 0x7fc7c415e8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47798/0 (socket says 172.21.15.122:47798) 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c2ffd640 1 -- 172.21.15.122:0/383682291 learned_addr learned my addr 172.21.15.122:0/383682291 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb0effd640 1 --2- 172.21.15.122:0/296689161 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cfba0 0x7ffb100d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7ffb0f7fe640 1 --2- 172.21.15.122:0/296689161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb1015e150 0x7ffb100cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.994+0000 7fc7c37fe640 1 --2- 172.21.15.122:0/383682291 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c40723a0 0x7fc7c4165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7ffb0e7fc640 1 -- 172.21.15.122:0/296689161 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643699234 0 0) 0x7ffb100d1ea0 con 0x7ffb100d0e90 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007710413 0 0) 0x7fc7c415f0d0 con 0x7fc7c4155560 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7ffb0e7fc640 1 -- 172.21.15.122:0/296689161 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf8003520 con 0x7ffb100d0e90 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ac003520 con 0x7fc7c4155560 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870018007 0 0) 0x7fc7c415f3b0 con 0x7fc7c415b300 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c415f0d0 con 0x7fc7c415b300 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/296689161 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cfba0 msgr2=0x7ffb100d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7ffb0f7fe640 1 --2- 172.21.15.122:0/296689161 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cfba0 0x7ffb100d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/296689161 >> v1:172.21.15.122:6789/0 conn(0x7ffb100d0e90 legacy=0x7ffb100b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c37fe640 1 -- 172.21.15.122:0/383682291 >> v1:172.21.15.169:6789/0 conn(0x7fc7c415b300 legacy=0x7fc7c4071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c37fe640 1 -- 172.21.15.122:0/383682291 >> v1:172.21.15.122:6789/0 conn(0x7fc7c4155560 legacy=0x7fc7c415e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7fc7c37fe640 1 -- 172.21.15.122:0/383682291 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c40b3da0 con 0x7fc7c40723a0 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.995+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/296689161 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb100d2460 con 0x7ffb1015e150 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7ffb0f7fe640 1 --2- 172.21.15.122:0/296689161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb1015e150 0x7ffb100cf660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb000097a0 tx=0x7ffb0002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed806a9a5dafa5ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc7c37fe640 1 --2- 172.21.15.122:0/383682291 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c40723a0 0x7fc7c4165c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009dc0 tx=0x7fc7bc02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffe022df28a64bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7ffb0e7fc640 1 -- 172.21.15.122:0/296689161 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb000315a0 con 0x7ffb1015e150 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7ffb0e7fc640 1 -- 172.21.15.122:0/296689161 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00037070 con 0x7ffb1015e150 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7ffb0e7fc640 1 -- 172.21.15.122:0/296689161 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00031960 con 0x7ffb1015e150 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc039070 con 0x7fc7c40723a0 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc02ccd0 con 0x7fc7c40723a0 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc7c1ffb640 1 -- 172.21.15.122:0/383682291 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc034660 con 0x7fc7c40723a0 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc40a9d6640 1 Processor -- start 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc7c93f7640 1 -- 172.21.15.122:0/383682291 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c40723a0 msgr2=0x7fc7c4165c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/383682291 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c40723a0 0x7fc7c4165c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009dc0 tx=0x7fc7bc02bf20 comp rx=0 tx=0).stop 2024-03-06T07:26:27.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.996+0000 7fc40a9d6640 1 -- start start 2024-03-06T07:26:27.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc7c93f7640 1 -- 172.21.15.122:0/383682291 shutdown_connections 2024-03-06T07:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/383682291 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c40723a0 0x7fc7c4165c80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc7c93f7640 1 -- 172.21.15.122:0/383682291 >> 172.21.15.122:0/383682291 conn(0x7fc7c40b2890 msgr2=0x7fc7c40b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc7c93f7640 1 -- 172.21.15.122:0/383682291 shutdown_connections 2024-03-06T07:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 -- 172.21.15.122:0/296689161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb1015e150 msgr2=0x7ffb100cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/296689161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb1015e150 0x7ffb100cf660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb000097a0 tx=0x7ffb0002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc7c93f7640 1 -- 172.21.15.122:0/383682291 wait complete. 2024-03-06T07:26:27.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 -- 172.21.15.122:0/296689161 shutdown_connections 2024-03-06T07:26:27.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/296689161 >> v2:172.21.15.169:3300/0 conn(0x7ffb100cfba0 0x7ffb100d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/296689161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb1015e150 0x7ffb100cf660 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 -- 172.21.15.122:0/296689161 >> 172.21.15.122:0/296689161 conn(0x7ffb1009d230 msgr2=0x7ffb100d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 -- 172.21.15.122:0/296689161 shutdown_connections 2024-03-06T07:26:27.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7ffb15ee1640 1 -- 172.21.15.122:0/296689161 wait complete. 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc40a9d6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc4040d0a60 0x7fc4040ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc40a9d6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc404155840 con 0x7fc404155460 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc40a9d6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40415bab0 con 0x7fc4040aeb20 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.997+0000 7fc40a9d6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40415c340 con 0x7fc4040d0a60 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc403fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc404155460 0x7fc40415b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47814/0 (socket says 172.21.15.122:47814) 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc403fff640 1 -- 172.21.15.122:0/1150886552 learned_addr learned my addr 172.21.15.122:0/1150886552 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc4037fe640 1 --2- 172.21.15.122:0/1150886552 >> v2:172.21.15.169:3300/0 conn(0x7fc4040d0a60 0x7fc4040ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc402ffd640 1 -- 172.21.15.122:0/1150886552 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873167774 0 0) 0x7fc404155840 con 0x7fc404155460 2024-03-06T07:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc402ffd640 1 -- 172.21.15.122:0/1150886552 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ec003520 con 0x7fc404155460 2024-03-06T07:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc4037fe640 1 -- 172.21.15.122:0/1150886552 >> v1:172.21.15.169:6789/0 conn(0x7fc4040aeb20 legacy=0x7fc4040b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc402ffd640 1 -- 172.21.15.122:0/1150886552 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309078135 0 0) 0x7fc40415bab0 con 0x7fc4040aeb20 2024-03-06T07:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc4037fe640 1 -- 172.21.15.122:0/1150886552 >> v1:172.21.15.122:6789/0 conn(0x7fc404155460 legacy=0x7fc40415b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc4037fe640 1 -- 172.21.15.122:0/1150886552 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40415bd80 con 0x7fc4040d0a60 2024-03-06T07:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc7c93f7640 1 Processor -- start 2024-03-06T07:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.998+0000 7fc7c93f7640 1 -- start start 2024-03-06T07:26:27.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc4037fe640 1 --2- 172.21.15.122:0/1150886552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc4040ae5e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097a0 tx=0x7fc3f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=462cdeb03eedbd74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7ffb15ee1640 1 Processor -- start 2024-03-06T07:26:27.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc402ffd640 1 -- 172.21.15.122:0/1150886552 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8035020 con 0x7fc4040d0a60 2024-03-06T07:26:27.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c93f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c40723a0 0x7fc7c413fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc402ffd640 1 -- 172.21.15.122:0/1150886552 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f8031830 con 0x7fc4040d0a60 2024-03-06T07:26:27.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc402ffd640 1 -- 172.21.15.122:0/1150886552 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8031c00 con 0x7fc4040d0a60 2024-03-06T07:26:27.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c93f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7c4155560 0x7fc7c415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c40723a0 0x7fc7c413fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7ffb15ee1640 1 -- start start 2024-03-06T07:26:27.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c40723a0 0x7fc7c413fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58946/0 (socket says 172.21.15.122:58946) 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c93f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c415b300 0x7fc7c415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c2ffd640 1 -- 172.21.15.122:0/3340475845 learned_addr learned my addr 172.21.15.122:0/3340475845 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c406af80 con 0x7fc7c40723a0 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40b5560 con 0x7fc7c4155560 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c406b100 con 0x7fc7c415b300 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c27fc640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7c4155560 0x7fc7c415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:26.999+0000 7fc7c37fe640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c415b300 0x7fc7c415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c27fc640 1 -- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c415b300 msgr2=0x7fc7c415c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c27fc640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c415b300 0x7fc7c415c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c27fc640 1 -- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c40723a0 msgr2=0x7fc7c413fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c27fc640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c40723a0 0x7fc7c413fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c27fc640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c4143600 con 0x7fc7c4155560 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c27fc640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7c4155560 0x7fc7c415c4a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8001e20 tx=0x7fc7b800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80052d0 con 0x7fc7c4155560 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b8005490 con 0x7fc7c4155560 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc40a9d6640 1 -- 172.21.15.122:0/1150886552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 msgr2=0x7fc4040ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/1150886552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc4040ae5e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097a0 tx=0x7fc3f802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c4142840 con 0x7fc7c4155560 2024-03-06T07:26:27.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc40a9d6640 1 -- 172.21.15.122:0/1150886552 shutdown_connections 2024-03-06T07:26:27.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/1150886552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc4040ae5e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc40a9d6640 1 -- 172.21.15.122:0/1150886552 >> 172.21.15.122:0/1150886552 conn(0x7fc40406c090 msgr2=0x7fc40406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8011a70 con 0x7fc7c4155560 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc40a9d6640 1 -- 172.21.15.122:0/1150886552 shutdown_connections 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c4142e00 con 0x7fc7c4155560 2024-03-06T07:26:27.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.000+0000 7ffb15ee1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 0x7ffb100c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb15ee1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb100d0e90 0x7ffb100b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7fc40a9d6640 1 -- 172.21.15.122:0/1150886552 wait complete. 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb0f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 0x7ffb100c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb0f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 0x7ffb100c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47660/0 (socket says 172.21.15.122:47660) 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/1669712655 learned_addr learned my addr 172.21.15.122:0/1669712655 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015e150 0x7ffb100b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100d1b30 con 0x7ffb100d0e90 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100d2f40 con 0x7ffb1015e150 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb0effd640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb100d0e90 0x7ffb100b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1015e530 con 0x7ffb100cfba0 2024-03-06T07:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.001+0000 7ffb0ffff640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015e150 0x7ffb100b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c4068050 con 0x7fc7c4155560 2024-03-06T07:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015e150 msgr2=0x7ffb100b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0f7fe640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015e150 0x7ffb100b6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb100d0e90 msgr2=0x7ffb100b9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0f7fe640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb100d0e90 0x7ffb100b9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0f7fe640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb101f4e30 con 0x7ffb100cfba0 2024-03-06T07:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0effd640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb100d0e90 0x7ffb100b9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7b8011c10 con 0x7fc7c4155560 2024-03-06T07:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb0f7fe640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 0x7ffb100c80b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb00034040 tx=0x7ffb000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7fc40a9d6640 1 Processor -- start 2024-03-06T07:26:27.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00032630 con 0x7ffb100cfba0 2024-03-06T07:26:27.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb000327f0 con 0x7ffb100cfba0 2024-03-06T07:26:27.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7fc7ab7fe640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7b006b4d0 0x7fc7b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb101f4070 con 0x7ffb100cfba0 2024-03-06T07:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7b808d990 con 0x7fc7c4155560 2024-03-06T07:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00032be0 con 0x7ffb100cfba0 2024-03-06T07:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7fc7c2ffd640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7b006b4d0 0x7fc7b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.002+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb101f4630 con 0x7ffb100cfba0 2024-03-06T07:26:27.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.003+0000 7fc40a9d6640 1 -- start start 2024-03-06T07:26:27.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.003+0000 7fc7c2ffd640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7b006b4d0 0x7fc7b006d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4000f60 tx=0x7fc7b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc40a9d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 0x7fc404157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10069ee0 con 0x7ffb100cfba0 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc403fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 0x7fc404157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc403fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 0x7fc404157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58958/0 (socket says 172.21.15.122:58958) 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc403fff640 1 -- 172.21.15.122:0/3435568032 learned_addr learned my addr 172.21.15.122:0/3435568032 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc40406f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc404155460 0x7fc404158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40406ecc0 con 0x7fc4040aeb20 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40406e9e0 con 0x7fc4040d0a60 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4040b4f30 con 0x7fc404155460 2024-03-06T07:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc4037fe640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc40406f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb0004e430 con 0x7ffb100cfba0 2024-03-06T07:26:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.004+0000 7fc408f73640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc404155460 0x7fc404158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc403fff640 1 -- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc404155460 msgr2=0x7fc404158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc403fff640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc404155460 0x7fc404158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc403fff640 1 -- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 msgr2=0x7fc40406f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc403fff640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc40406f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc403fff640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404156280 con 0x7fc4040aeb20 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc408f73640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc404155460 0x7fc404158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc403fff640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 0x7fc404157300 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400c390 tx=0x7fc3f400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4005ce0 con 0x7fc4040aeb20 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f4015030 con 0x7fc4040aeb20 2024-03-06T07:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7ffaf7fff640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffafc06b4d0 0x7ffafc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40072f0 con 0x7fc4040aeb20 2024-03-06T07:26:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7ffb0effd640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffafc06b4d0 0x7ffafc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc404156560 con 0x7fc4040aeb20 2024-03-06T07:26:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb0002f030 con 0x7ffb100cfba0 2024-03-06T07:26:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.005+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc404156b20 con 0x7fc4040aeb20 2024-03-06T07:26:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.006+0000 7ffb0effd640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffafc06b4d0 0x7ffafc06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ffb100c8900 tx=0x7ffb04009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.006+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc404068050 con 0x7fc4040aeb20 2024-03-06T07:26:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.007+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7b805c0b0 con 0x7fc7c4155560 2024-03-06T07:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.007+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3f4007490 con 0x7fc4040aeb20 2024-03-06T07:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.007+0000 7fc400ff9640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f006b4d0 0x7fc3f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.007+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3f408da70 con 0x7fc4040aeb20 2024-03-06T07:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.007+0000 7fc4037fe640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f006b4d0 0x7fc3f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.008+0000 7fc4037fe640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f006b4d0 0x7fc3f006d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8036000 tx=0x7fc3f8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.010+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb00040070 con 0x7ffb100cfba0 2024-03-06T07:26:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.012+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3f405c180 con 0x7fc4040aeb20 2024-03-06T07:26:27.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.256+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f1270068050 con 0x7f1270161670 2024-03-06T07:26:27.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.274+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fc7c415e110 con 0x7fc7c4155560 2024-03-06T07:26:27.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.280+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fc40414c390 con 0x7fc4040aeb20 2024-03-06T07:26:27.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:27.292+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7ffb100cadf0 con 0x7ffb100cfba0 2024-03-06T07:26:28.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.134+0000 7f12577fe640 1 -- 172.21.15.122:0/1556950716 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6221 v48) v1 ==== 95+0+0 (secure 0 0 0) 0x7f1268057f70 con 0x7f1270161670 2024-03-06T07:26:28.136 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6221 2024-03-06T07:26:28.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125c06b4d0 msgr2=0x7f125c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125c06b4d0 0x7f125c06d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1260036000 tx=0x7f126003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:28.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 msgr2=0x7f1270155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270155940 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f126800c760 tx=0x7f126800cc20 comp rx=0 tx=0).stop 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 shutdown_connections 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125c06b4d0 0x7f125c06d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1270161670 0x7f1270155940 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12700cc680 0x7f12700b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 --2- 172.21.15.122:0/1556950716 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b2da0 0x7f12700c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 >> 172.21.15.122:0/1556950716 conn(0x7f12700d1d50 msgr2=0x7f12700d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 shutdown_connections 2024-03-06T07:26:28.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.136+0000 7f1275439640 1 -- 172.21.15.122:0/1556950716 wait complete. 2024-03-06T07:26:28.145 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:26:28.144+0000 7fdd670e6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:28.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.147+0000 7fc400ff9640 1 -- 172.21.15.122:0/3435568032 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6817 v49) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc3f4060220 con 0x7fc4040aeb20 2024-03-06T07:26:28.148 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6817 2024-03-06T07:26:28.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f006b4d0 msgr2=0x7fc3f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f006b4d0 0x7fc3f006d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8036000 tx=0x7fc3f8008030 comp rx=0 tx=0).stop 2024-03-06T07:26:28.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 msgr2=0x7fc404157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 0x7fc404157300 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400c390 tx=0x7fc3f400c850 comp rx=0 tx=0).stop 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 shutdown_connections 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f006b4d0 0x7fc3f006d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc404155460 0x7fc404158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4040d0a60 0x7fc40406f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 --2- 172.21.15.122:0/3435568032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4040aeb20 0x7fc404157300 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 >> 172.21.15.122:0/3435568032 conn(0x7fc40406c090 msgr2=0x7fc40415b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 shutdown_connections 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc40a9d6640 1 -- 172.21.15.122:0/3435568032 wait complete. 2024-03-06T07:26:28.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.149+0000 7fc7ab7fe640 1 -- 172.21.15.122:0/3340475845 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6805 v49) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc7b8060150 con 0x7fc7c4155560 2024-03-06T07:26:28.153 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6805 2024-03-06T07:26:28.153 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:28.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7b006b4d0 msgr2=0x7fc7b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7b006b4d0 0x7fc7b006d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4000f60 tx=0x7fc7b4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7c4155560 msgr2=0x7fc7c415c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7c4155560 0x7fc7c415c4a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8001e20 tx=0x7fc7b800c850 comp rx=0 tx=0).stop 2024-03-06T07:26:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 shutdown_connections 2024-03-06T07:26:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7b006b4d0 0x7fc7b006d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7c415b300 0x7fc7c415c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7c4155560 0x7fc7c415c4a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 --2- 172.21.15.122:0/3340475845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c40723a0 0x7fc7c413fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 >> 172.21.15.122:0/3340475845 conn(0x7fc7c40b2890 msgr2=0x7fc7c40d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 shutdown_connections 2024-03-06T07:26:28.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.153+0000 7fc7c93f7640 1 -- 172.21.15.122:0/3340475845 wait complete. 2024-03-06T07:26:28.156 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:28.157 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:28.166 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:28.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:28.169 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:28.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:28.169 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:28.170 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:28.211 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:28.211 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:28.236 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:26:28.248 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:26:28.288 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:26:28.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.962+0000 7ffaf7fff640 1 -- 172.21.15.122:0/1669712655 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6808 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7ffb0007ad90 con 0x7ffb100cfba0 2024-03-06T07:26:28.964 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6808 2024-03-06T07:26:28.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffafc06b4d0 msgr2=0x7ffafc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffafc06b4d0 0x7ffafc06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ffb100c8900 tx=0x7ffb04009190 comp rx=0 tx=0).stop 2024-03-06T07:26:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 msgr2=0x7ffb100c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 0x7ffb100c80b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb00034040 tx=0x7ffb000046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 shutdown_connections 2024-03-06T07:26:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffafc06b4d0 0x7ffafc06d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb1015e150 0x7ffb100b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb100d0e90 0x7ffb100b9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 --2- 172.21.15.122:0/1669712655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb100cfba0 0x7ffb100c80b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 >> 172.21.15.122:0/1669712655 conn(0x7ffb1009d230 msgr2=0x7ffb1015f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 shutdown_connections 2024-03-06T07:26:28.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:28.965+0000 7ffb15ee1640 1 -- 172.21.15.122:0/1669712655 wait complete. 2024-03-06T07:26:28.981 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:28.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:29.023 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:29.023 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:29.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:29.100 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:26:29.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.125+0000 7f350afe9640 1 Processor -- start 2024-03-06T07:26:29.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.125+0000 7f350afe9640 1 -- start start 2024-03-06T07:26:29.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f350afe9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3504161760 0x7f35040b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f350afe9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 0x7f35040c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f350afe9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3504161b40 con 0x7f35040b5240 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f350afe9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35040d27e0 con 0x7f3504161760 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f350afe9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35040cf580 con 0x7f35040d2240 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f3503fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f35040b5240 0x7f35040d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47838/0 (socket says 172.21.15.122:47838) 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f3503fff640 1 -- 172.21.15.122:0/3578694479 learned_addr learned my addr 172.21.15.122:0/3578694479 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f3509586640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 0x7f35040c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f3508d85640 1 --2- 172.21.15.122:0/3578694479 >> v2:172.21.15.169:3300/0 conn(0x7f3504161760 0x7f35040b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066485435 0 0) 0x7f3504161b40 con 0x7f35040b5240 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f0003520 con 0x7f35040b5240 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3605526354 0 0) 0x7f34f0003520 con 0x7f35040b5240 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 msgr2=0x7f35040c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 0x7f35040c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 >> v2:172.21.15.169:3300/0 conn(0x7f3504161760 msgr2=0x7f35040b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 --2- 172.21.15.122:0/3578694479 >> v2:172.21.15.169:3300/0 conn(0x7f3504161760 0x7f35040b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35040cf860 con 0x7f35040b5240 2024-03-06T07:26:29.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.126+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f34f40019f0 con 0x7f35040b5240 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 >> v1:172.21.15.122:6789/0 conn(0x7f35040b5240 legacy=0x7f35040d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34f00039b0 0x7f34f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f0004800 0x7f34f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 0x7f34f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35040d27e0 con 0x7f34f00039b0 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35040cf580 con 0x7f34f0004800 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f000b9c0 con 0x7f35040d2240 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3508d85640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34f00039b0 0x7f34f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3503fff640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f0004800 0x7f34f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3509586640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 0x7f34f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3508d85640 1 -- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 msgr2=0x7f34f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3508d85640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040d2240 0x7f34f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3508d85640 1 -- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f0004800 msgr2=0x7f34f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3508d85640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f0004800 0x7f34f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.127+0000 7f3508d85640 1 -- 172.21.15.122:0/3578694479 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f0003520 con 0x7f34f00039b0 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f3508d85640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34f00039b0 0x7f34f0003dd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ac0 tx=0x7f34f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9585cbf3300cf3ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc039070 con 0x7f34f00039b0 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc02c710 con 0x7f34f00039b0 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f35037fe640 1 -- 172.21.15.122:0/3578694479 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc009690 con 0x7f34f00039b0 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f350afe9640 1 -- 172.21.15.122:0/3578694479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34f00039b0 msgr2=0x7f34f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f350afe9640 1 --2- 172.21.15.122:0/3578694479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34f00039b0 0x7f34f0003dd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ac0 tx=0x7f34f0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f3509586640 1 -- 172.21.15.122:0/3578694479 reap_dead start 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f350afe9640 1 -- 172.21.15.122:0/3578694479 shutdown_connections 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f350afe9640 1 -- 172.21.15.122:0/3578694479 >> 172.21.15.122:0/3578694479 conn(0x7f35040d3f60 msgr2=0x7f35040d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.128+0000 7f350afe9640 1 -- 172.21.15.122:0/3578694479 shutdown_connections 2024-03-06T07:26:29.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.129+0000 7f350afe9640 1 -- 172.21.15.122:0/3578694479 wait complete. 2024-03-06T07:26:29.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.130+0000 7f350afe9640 1 Processor -- start 2024-03-06T07:26:29.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.130+0000 7f350afe9640 1 -- start start 2024-03-06T07:26:29.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.131+0000 7f350afe9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f00039b0 0x7f350415b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.131+0000 7f350afe9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040c8c60 0x7f35040ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.131+0000 7f350afe9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 0x7f35041f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.131+0000 7f350afe9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35040d2c40 con 0x7f35040ca5b0 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f350afe9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35040d2ac0 con 0x7f34f00039b0 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f350afe9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35040d4a80 con 0x7f35040c8c60 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 0x7f35041f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 0x7f35041f4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59000/0 (socket says 172.21.15.122:59000) 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 -- 172.21.15.122:0/1664552606 learned_addr learned my addr 172.21.15.122:0/1664552606 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3503fff640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040c8c60 0x7f35040ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3508d85640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f00039b0 0x7f350415b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 -- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040c8c60 msgr2=0x7f35040ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040c8c60 0x7f35040ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 -- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f00039b0 msgr2=0x7f350415b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f00039b0 0x7f350415b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 -- 172.21.15.122:0/1664552606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35040c7bc0 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3503fff640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040c8c60 0x7f35040ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3508d85640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f00039b0 0x7f350415b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f3509586640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 0x7f35041f4d70 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f35040a1e80 tx=0x7f34ec011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec019020 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec003610 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35040c7ea0 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec003a00 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.132+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35040c8460 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.134+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3504069ee0 con 0x7f35040ca5b0 2024-03-06T07:26:29.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.134+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34ec01c020 con 0x7f35040ca5b0 2024-03-06T07:26:29.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.134+0000 7f35017fa640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e406b4d0 0x7f34e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.134+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f34ec08a680 con 0x7f35040ca5b0 2024-03-06T07:26:29.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.134+0000 7f3508d85640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e406b4d0 0x7f34e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.135+0000 7f3508d85640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e406b4d0 0x7f34e406d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f34fc036000 tx=0x7f34fc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.139+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34ec023020 con 0x7f35040ca5b0 2024-03-06T07:26:29.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.344+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35040d47c0 con 0x7f35040ca5b0 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.345+0000 7f35017fa640 1 -- 172.21.15.122:0/1664552606 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4185 (secure 0 0 0) 0x7f34ec059530 con 0x7f35040ca5b0 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:28.959824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 50 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e406b4d0 msgr2=0x7f34e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e406b4d0 0x7f34e406d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f34fc036000 tx=0x7f34fc030000 comp rx=0 tx=0).stop 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 msgr2=0x7f35041f4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 0x7f35041f4d70 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f35040a1e80 tx=0x7f34ec011920 comp rx=0 tx=0).stop 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 shutdown_connections 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e406b4d0 0x7f34e406d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35040ca5b0 0x7f35041f4d70 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f35040c8c60 0x7f35040ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 --2- 172.21.15.122:0/1664552606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34f00039b0 0x7f350415b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 >> 172.21.15.122:0/1664552606 conn(0x7f35040d3f60 msgr2=0x7f350415faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 shutdown_connections 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:29.347+0000 7f350afe9640 1 -- 172.21.15.122:0/1664552606 wait complete. 2024-03-06T07:26:29.762 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.451683+0000', 'modified': '2024-03-06T07:26:28.959824+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.122:6835/498310660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 498310660}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 498310660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:29.762 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7671:up:replay 2024-03-06T07:26:30.763 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:30.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.873+0000 7f1e8d4eb640 1 Processor -- start 2024-03-06T07:26:30.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.873+0000 7f1e8d4eb640 1 -- start start 2024-03-06T07:26:30.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e8d4eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1e8815e210 0x7f1e880d49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e8d4eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1e880d2d10 0x7f1e880b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e8d4eb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e880cef60 con 0x7f1e8815e5f0 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e8d4eb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e880d0000 con 0x7f1e880d2d10 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e8d4eb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e880cf240 con 0x7f1e8815e210 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1e880d2d10 0x7f1e880b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1e880d2d10 0x7f1e880b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59002/0 (socket says 172.21.15.122:59002) 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 -- 172.21.15.122:0/2483089554 learned_addr learned my addr 172.21.15.122:0/2483089554 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e86ffd640 1 --2- 172.21.15.122:0/2483089554 >> v2:172.21.15.169:3300/0 conn(0x7f1e8815e210 0x7f1e880d49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 -- 172.21.15.122:0/2483089554 >> v2:172.21.15.169:3300/0 conn(0x7f1e8815e210 msgr2=0x7f1e880d49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 --2- 172.21.15.122:0/2483089554 >> v2:172.21.15.169:3300/0 conn(0x7f1e8815e210 0x7f1e880d49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 -- 172.21.15.122:0/2483089554 >> v1:172.21.15.169:6789/0 conn(0x7f1e8815e5f0 legacy=0x7f1e880d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e877fe640 1 -- 172.21.15.122:0/2483089554 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880cf520 con 0x7f1e880d2d10 2024-03-06T07:26:30.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.874+0000 7f1e86ffd640 1 --2- 172.21.15.122:0/2483089554 >> v2:172.21.15.169:3300/0 conn(0x7f1e8815e210 0x7f1e880d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.875+0000 7f1e877fe640 1 --2- 172.21.15.122:0/2483089554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e880d2d10 0x7f1e880b3470 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1e8808e300 tx=0x7f1e8002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a756daefd2991f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.875+0000 7f1e867fc640 1 -- 172.21.15.122:0/2483089554 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8002ca80 con 0x7f1e880d2d10 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.875+0000 7f1e867fc640 1 -- 172.21.15.122:0/2483089554 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8002cc20 con 0x7f1e880d2d10 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.875+0000 7f1e867fc640 1 -- 172.21.15.122:0/2483089554 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800343f0 con 0x7f1e880d2d10 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/2483089554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e880d2d10 msgr2=0x7f1e880b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/2483089554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e880d2d10 0x7f1e880b3470 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1e8808e300 tx=0x7f1e8002bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/2483089554 shutdown_connections 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/2483089554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e880d2d10 0x7f1e880b3470 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/2483089554 >> v2:172.21.15.169:3300/0 conn(0x7f1e8815e210 0x7f1e880d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/2483089554 >> 172.21.15.122:0/2483089554 conn(0x7f1e8809d230 msgr2=0x7f1e880d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/2483089554 shutdown_connections 2024-03-06T07:26:30.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.876+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/2483089554 wait complete. 2024-03-06T07:26:30.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.877+0000 7f1e8d4eb640 1 Processor -- start 2024-03-06T07:26:30.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.877+0000 7f1e8d4eb640 1 -- start start 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e880d2d10 0x7f1e88154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 0x7f1e880b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e8815e5f0 0x7f1e88156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e880d4ef0 con 0x7f1e8815e210 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e880cfae0 con 0x7f1e880d2d10 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e880cec10 con 0x7f1e8815e5f0 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 0x7f1e880b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 0x7f1e880b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59012/0 (socket says 172.21.15.122:59012) 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 -- 172.21.15.122:0/429612925 learned_addr learned my addr 172.21.15.122:0/429612925 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 -- 172.21.15.122:0/429612925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e8815e5f0 msgr2=0x7f1e88156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e8815e5f0 0x7f1e88156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 -- 172.21.15.122:0/429612925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e880d2d10 msgr2=0x7f1e88154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e880d2d10 0x7f1e88154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 -- 172.21.15.122:0/429612925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88155b50 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7ffff640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 0x7f1e880b6bd0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000b3f0 tx=0x7f1e7000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70015020 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70011830 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e88155e30 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70011c20 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.878+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e881563f0 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.879+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e88069ee0 con 0x7f1e8815e210 2024-03-06T07:26:30.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.880+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e70018020 con 0x7f1e8815e210 2024-03-06T07:26:30.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.880+0000 7f1e7f7fe640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e6806b4d0 0x7f1e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.880+0000 7f1e86ffd640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e6806b4d0 0x7f1e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.880+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1e70089e90 con 0x7f1e8815e210 2024-03-06T07:26:30.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.881+0000 7f1e86ffd640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e6806b4d0 0x7f1e6806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:30.885+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e7001f030 con 0x7f1e8815e210 2024-03-06T07:26:31.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.095+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e880d1740 con 0x7f1e8815e210 2024-03-06T07:26:31.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.095+0000 7f1e7f7fe640 1 -- 172.21.15.122:0/429612925 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f1e700172c0 con 0x7f1e8815e210 2024-03-06T07:26:31.098 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:31.098 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":52,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:29.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:31.098 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 52 2024-03-06T07:26:31.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.097+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e6806b4d0 msgr2=0x7f1e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.097+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e6806b4d0 0x7f1e6806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78008040 comp rx=0 tx=0).stop 2024-03-06T07:26:31.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.097+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 msgr2=0x7f1e880b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.097+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 0x7f1e880b6bd0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000b3f0 tx=0x7f1e7000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 shutdown_connections 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1e6806b4d0 0x7f1e6806d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1e8815e5f0 0x7f1e88156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e8815e210 0x7f1e880b6bd0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 --2- 172.21.15.122:0/429612925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e880d2d10 0x7f1e88154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 >> 172.21.15.122:0/429612925 conn(0x7f1e8809d230 msgr2=0x7f1e880d4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 shutdown_connections 2024-03-06T07:26:31.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:31.098+0000 7f1e8d4eb640 1 -- 172.21.15.122:0/429612925 wait complete. 2024-03-06T07:26:31.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.451683+0000', 'modified': '2024-03-06T07:26:29.965725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.122:6835/498310660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 498310660}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 498310660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:31.513 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7671:up:rejoin 2024-03-06T07:26:32.514 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.624+0000 7fafc17a0640 1 Processor -- start 2024-03-06T07:26:32.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.624+0000 7fafc17a0640 1 -- start start 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc17a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fafbc15e0a0 0x7fafbc0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc17a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0cef90 0x7fafbc0d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc17a0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafbc0cf610 con 0x7fafbc0d4380 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc17a0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc0cf3f0 con 0x7fafbc15e0a0 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc17a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0cf8f0 con 0x7fafbc0cef90 2024-03-06T07:26:32.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fafbc15e0a0 0x7fafbc0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc0f9f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fafbc0d4380 0x7fafbc0d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47840/0 (socket says 172.21.15.122:47840) 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafc0f9f640 1 -- 172.21.15.122:0/913435959 learned_addr learned my addr 172.21.15.122:0/913435959 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbb7fe640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0cef90 0x7fafbc0d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbbfff640 1 -- 172.21.15.122:0/913435959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0cef90 msgr2=0x7fafbc0d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbbfff640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0cef90 0x7fafbc0d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbbfff640 1 -- 172.21.15.122:0/913435959 >> v1:172.21.15.122:6789/0 conn(0x7fafbc0d4380 legacy=0x7fafbc0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbbfff640 1 -- 172.21.15.122:0/913435959 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc0cfbd0 con 0x7fafbc15e0a0 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbaffd640 1 -- 172.21.15.122:0/913435959 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802681379 0 0) 0x7fafbc0cf610 con 0x7fafbc0d4380 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.625+0000 7fafbb7fe640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0cef90 0x7fafbc0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.626+0000 7fafbbfff640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0cea50 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fafac0097a0 tx=0x7fafac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a4da0b5d3eb7ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.626+0000 7fafbaffd640 1 -- 172.21.15.122:0/913435959 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac0315a0 con 0x7fafbc15e0a0 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.626+0000 7fafbaffd640 1 -- 172.21.15.122:0/913435959 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac037070 con 0x7fafbc15e0a0 2024-03-06T07:26:32.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.626+0000 7fafbaffd640 1 -- 172.21.15.122:0/913435959 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac031960 con 0x7fafbc15e0a0 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 -- 172.21.15.122:0/913435959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc15e0a0 msgr2=0x7fafbc0cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0cea50 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fafac0097a0 tx=0x7fafac02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:32.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 -- 172.21.15.122:0/913435959 shutdown_connections 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0cef90 0x7fafbc0d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 --2- 172.21.15.122:0/913435959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0cea50 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 -- 172.21.15.122:0/913435959 >> 172.21.15.122:0/913435959 conn(0x7fafbc08d7f0 msgr2=0x7fafbc09d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 -- 172.21.15.122:0/913435959 shutdown_connections 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.627+0000 7fafc17a0640 1 -- 172.21.15.122:0/913435959 wait complete. 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.628+0000 7fafc17a0640 1 Processor -- start 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.628+0000 7fafc17a0640 1 -- start start 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc17a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 0x7fafbc0c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc17a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0d4380 0x7fafbc0b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc17a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0b9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc17a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc15e480 con 0x7fafbc0cef90 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc17a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0d1cb0 con 0x7fafbc15e0a0 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc17a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc15e600 con 0x7fafbc0d4380 2024-03-06T07:26:32.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 0x7fafbc0c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 0x7fafbc0c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59026/0 (socket says 172.21.15.122:59026) 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 -- 172.21.15.122:0/1130112519 learned_addr learned my addr 172.21.15.122:0/1130112519 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 -- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0d4380 msgr2=0x7fafbc0b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbb7fe640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0d4380 0x7fafbc0b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0d4380 0x7fafbc0b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc0f9f640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 -- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafbc15e0a0 msgr2=0x7fafbc0b9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0b9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 -- 172.21.15.122:0/1130112519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc0b72c0 con 0x7fafbc0cef90 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafc0f9f640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.629+0000 7fafbbfff640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 0x7fafbc0c9c30 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fafac034040 tx=0x7fafac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.630+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac032840 con 0x7fafbc0cef90 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.630+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac032a00 con 0x7fafbc0cef90 2024-03-06T07:26:32.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.630+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac032df0 con 0x7fafbc0cef90 2024-03-06T07:26:32.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.630+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc0b75a0 con 0x7fafbc0cef90 2024-03-06T07:26:32.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.630+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc0b7b60 con 0x7fafbc0cef90 2024-03-06T07:26:32.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.631+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc04f330 con 0x7fafbc0cef90 2024-03-06T07:26:32.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.631+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafac04e430 con 0x7fafbc0cef90 2024-03-06T07:26:32.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.631+0000 7fafb8ff9640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa806b4d0 0x7fafa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.632+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fafac02f030 con 0x7fafbc0cef90 2024-03-06T07:26:32.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.632+0000 7fafbb7fe640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa806b4d0 0x7fafa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.632+0000 7fafbb7fe640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa806b4d0 0x7fafa806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0a1e30 tx=0x7fafb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.636+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafac040070 con 0x7fafbc0cef90 2024-03-06T07:26:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.845+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafbc0ced70 con 0x7fafbc0cef90 2024-03-06T07:26:32.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.845+0000 7fafb8ff9640 1 -- 172.21.15.122:0/1130112519 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fafac07afe0 con 0x7fafbc0cef90 2024-03-06T07:26:32.848 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:32.848 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:30.968285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:clientreplay","state_seq":4,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:32.848 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 53 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.847+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa806b4d0 msgr2=0x7fafa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.847+0000 7fafc17a0640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa806b4d0 0x7fafa806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0a1e30 tx=0x7fafb0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.847+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 msgr2=0x7fafbc0c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.847+0000 7fafc17a0640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 0x7fafbc0c9c30 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fafac034040 tx=0x7fafac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.847+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 shutdown_connections 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fafa806b4d0 0x7fafa806d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafbc15e0a0 0x7fafbc0b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fafbc0d4380 0x7fafbc0b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 --2- 172.21.15.122:0/1130112519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafbc0cef90 0x7fafbc0c9c30 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 >> 172.21.15.122:0/1130112519 conn(0x7fafbc08d7f0 msgr2=0x7fafbc15e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:32.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 shutdown_connections 2024-03-06T07:26:32.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:32.848+0000 7fafc17a0640 1 -- 172.21.15.122:0/1130112519 wait complete. 2024-03-06T07:26:33.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.451683+0000', 'modified': '2024-03-06T07:26:30.968285+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:clientreplay', 'state_seq': 4, 'addr': '172.21.15.122:6835/498310660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 498310660}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 498310660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:33.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7671:up:clientreplay 2024-03-06T07:26:33.969 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:26:33.967+0000 7f9350414640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:34.264 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:34.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.363+0000 7f95e3466640 1 Processor -- start 2024-03-06T07:26:34.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.363+0000 7f95e3466640 1 -- start start 2024-03-06T07:26:34.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e3466640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f95dc0d0950 0x7f95dc0d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e3466640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95dc0d1ea0 con 0x7f95dc0b9060 2024-03-06T07:26:34.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e3466640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95dc0d2180 con 0x7f95dc15e150 2024-03-06T07:26:34.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e3466640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95dc0d12f0 con 0x7f95dc0d0950 2024-03-06T07:26:34.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e1a03640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f95dc0b9060 0x7f95dc0b94d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47842/0 (socket says 172.21.15.122:47842) 2024-03-06T07:26:34.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e1a03640 1 -- 172.21.15.122:0/1961608717 learned_addr learned my addr 172.21.15.122:0/1961608717 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:34.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95e0a01640 1 --2- 172.21.15.122:0/1961608717 >> v2:172.21.15.169:3300/0 conn(0x7f95dc0d0950 0x7f95dc0d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24741259 0 0) 0x7f95dc0d1ea0 con 0x7f95dc0b9060 2024-03-06T07:26:34.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.364+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c8003520 con 0x7f95dc0b9060 2024-03-06T07:26:34.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506212978 0 0) 0x7f95dc0d2180 con 0x7f95dc15e150 2024-03-06T07:26:34.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95dc0d1ea0 con 0x7f95dc15e150 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716069332 0 0) 0x7f95c8003520 con 0x7f95dc0b9060 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 >> v2:172.21.15.169:3300/0 conn(0x7f95dc0d0950 msgr2=0x7f95dc0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 --2- 172.21.15.122:0/1961608717 >> v2:172.21.15.169:3300/0 conn(0x7f95dc0d0950 0x7f95dc0d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 >> v1:172.21.15.169:6789/0 conn(0x7f95dc15e150 legacy=0x7f95dc0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc0d2460 con 0x7f95dc0b9060 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f95d0001670 con 0x7f95dc0b9060 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 >> v1:172.21.15.122:6789/0 conn(0x7f95dc0b9060 legacy=0x7f95dc0b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8003b40 0x7f95c8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95c8004800 0x7f95c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95c80051c0 0x7f95c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c800c0c0 con 0x7f95c8004800 2024-03-06T07:26:34.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c800c270 con 0x7f95c80051c0 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c800c470 con 0x7f95c8003b40 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e0a01640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95c8004800 0x7f95c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e1202640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8003b40 0x7f95c8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e1a03640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95c80051c0 0x7f95c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e0a01640 1 -- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8003b40 msgr2=0x7f95c8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e0a01640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8003b40 0x7f95c8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e0a01640 1 -- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95c80051c0 msgr2=0x7f95c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e0a01640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95c80051c0 0x7f95c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e0a01640 1 -- 172.21.15.122:0/1961608717 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c8003520 con 0x7f95c8004800 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e1a03640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95c80051c0 0x7f95c800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:34.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e1202640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8003b40 0x7f95c8003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.365+0000 7f95e1a03640 1 -- 172.21.15.122:0/1961608717 reap_dead start 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.366+0000 7f95e0a01640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95c8004800 0x7f95c8004c80 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009ac0 tx=0x7f95c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f427af64e2472f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.366+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8039070 con 0x7f95c8004800 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.366+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d802c6b0 con 0x7f95c8004800 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.366+0000 7f95cffff640 1 -- 172.21.15.122:0/1961608717 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8009690 con 0x7f95c8004800 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 -- 172.21.15.122:0/1961608717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95c8004800 msgr2=0x7f95c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95c8004800 0x7f95c8004c80 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009ac0 tx=0x7f95c8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 -- 172.21.15.122:0/1961608717 shutdown_connections 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 --2- 172.21.15.122:0/1961608717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95c8004800 0x7f95c8004c80 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 -- 172.21.15.122:0/1961608717 >> 172.21.15.122:0/1961608717 conn(0x7f95dc09d230 msgr2=0x7f95dc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 -- 172.21.15.122:0/1961608717 shutdown_connections 2024-03-06T07:26:34.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.367+0000 7f95e3466640 1 -- 172.21.15.122:0/1961608717 wait complete. 2024-03-06T07:26:34.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.368+0000 7f95e3466640 1 Processor -- start 2024-03-06T07:26:34.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.368+0000 7f95e3466640 1 -- start start 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e3466640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8004800 0x7f95dc150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e3466640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 0x7f95dc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e3466640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95dc0cbd30 0x7f95dc0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e3466640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0d2f40 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e3466640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0d2ab0 con 0x7f95dc0cbd30 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e3466640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0d1b30 con 0x7f95c8004800 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 0x7f95dc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 0x7f95dc0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59038/0 (socket says 172.21.15.122:59038) 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 -- 172.21.15.122:0/682281777 learned_addr learned my addr 172.21.15.122:0/682281777 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e1202640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8004800 0x7f95dc150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e1a03640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95dc0cbd30 0x7f95dc0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 -- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8004800 msgr2=0x7f95dc150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8004800 0x7f95dc150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 -- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95dc0cbd30 msgr2=0x7f95dc0cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95dc0cbd30 0x7f95dc0cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 -- 172.21.15.122:0/682281777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc0cc290 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95e0a01640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 0x7f95dc0ceaa0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f95d8037040 tx=0x7f95d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.369+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8042020 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.370+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d803a040 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.370+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8039070 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.370+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95dc0cc570 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.370+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95dc0c7c60 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.371+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95dc0c77a0 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.371+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95d8045020 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.372+0000 7f95cdffb640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95c006b4d0 0x7f95c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.372+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f95d80ab5a0 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.372+0000 7f95e1202640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95c006b4d0 0x7f95c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.372+0000 7f95e1202640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95c006b4d0 0x7f95c006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f95dc068700 tx=0x7f95d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.376+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95dc0c77a0 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.606+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95dc069ee0 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.606+0000 7f95cdffb640 1 -- 172.21.15.122:0/682281777 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f95d804c070 con 0x7f95dc0cb8b0 2024-03-06T07:26:34.608 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:34.608 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:34.609 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:34.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95c006b4d0 msgr2=0x7f95c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95c006b4d0 0x7f95c006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f95dc068700 tx=0x7f95d4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:34.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 msgr2=0x7f95dc0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 0x7f95dc0ceaa0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f95d8037040 tx=0x7f95d8004730 comp rx=0 tx=0).stop 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 shutdown_connections 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95c006b4d0 0x7f95c006d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95dc0cbd30 0x7f95dc0cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95dc0cb8b0 0x7f95dc0ceaa0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 --2- 172.21.15.122:0/682281777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95c8004800 0x7f95dc150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 >> 172.21.15.122:0/682281777 conn(0x7f95dc09d230 msgr2=0x7f95dc0b6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 shutdown_connections 2024-03-06T07:26:34.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:34.608+0000 7f95e3466640 1 -- 172.21.15.122:0/682281777 wait complete. 2024-03-06T07:26:35.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.451683+0000', 'modified': '2024-03-06T07:26:32.965725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.122:6835/498310660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 498310660}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 498310660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7671, 'qdb_cluster': [7671]} 2024-03-06T07:26:35.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:35.025 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:35.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.132+0000 7ff66975c640 1 Processor -- start 2024-03-06T07:26:35.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.132+0000 7ff66975c640 1 -- start start 2024-03-06T07:26:35.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff66975c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff664161670 0x7ff664161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff66975c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff6640cd690 0x7ff6640cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff66975c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff664162c70 con 0x7ff6640b2da0 2024-03-06T07:26:35.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff66975c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6640b3590 con 0x7ff6640cd690 2024-03-06T07:26:35.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff66975c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff66406cd80 con 0x7ff664161670 2024-03-06T07:26:35.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff6637fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff6640cd690 0x7ff6640cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff6627fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff6640b2da0 0x7ff6641623a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38430/0 (socket says 172.21.15.122:38430) 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff6627fc640 1 -- 172.21.15.122:0/3130585833 learned_addr learned my addr 172.21.15.122:0/3130585833 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.133+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3130585833 >> v2:172.21.15.169:3300/0 conn(0x7ff664161670 0x7ff664161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff6637fe640 1 -- 172.21.15.122:0/3130585833 >> v2:172.21.15.169:3300/0 conn(0x7ff664161670 msgr2=0x7ff664161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff6637fe640 1 --2- 172.21.15.122:0/3130585833 >> v2:172.21.15.169:3300/0 conn(0x7ff664161670 0x7ff664161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff6637fe640 1 -- 172.21.15.122:0/3130585833 >> v1:172.21.15.122:6789/0 conn(0x7ff6640b2da0 legacy=0x7ff6641623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff6637fe640 1 -- 172.21.15.122:0/3130585833 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66406d060 con 0x7ff6640cd690 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff661ffb640 1 -- 172.21.15.122:0/3130585833 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655571740 0 0) 0x7ff664162c70 con 0x7ff6640b2da0 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff6637fe640 1 --2- 172.21.15.122:0/3130585833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640cd690 0x7ff6640cff50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7ff6540097a0 tx=0x7ff65402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b2bed58b54260f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff661ffb640 1 -- 172.21.15.122:0/3130585833 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654035020 con 0x7ff6640cd690 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff661ffb640 1 -- 172.21.15.122:0/3130585833 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff654031830 con 0x7ff6640cd690 2024-03-06T07:26:35.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.134+0000 7ff661ffb640 1 -- 172.21.15.122:0/3130585833 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654031c00 con 0x7ff6640cd690 2024-03-06T07:26:35.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 -- 172.21.15.122:0/3130585833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640cd690 msgr2=0x7ff6640cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 --2- 172.21.15.122:0/3130585833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640cd690 0x7ff6640cff50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7ff6540097a0 tx=0x7ff65402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 -- 172.21.15.122:0/3130585833 shutdown_connections 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 --2- 172.21.15.122:0/3130585833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640cd690 0x7ff6640cff50 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 --2- 172.21.15.122:0/3130585833 >> v2:172.21.15.169:3300/0 conn(0x7ff664161670 0x7ff664161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 -- 172.21.15.122:0/3130585833 >> 172.21.15.122:0/3130585833 conn(0x7ff6640d1d50 msgr2=0x7ff6640d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 -- 172.21.15.122:0/3130585833 shutdown_connections 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.135+0000 7ff66975c640 1 -- 172.21.15.122:0/3130585833 wait complete. 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.136+0000 7ff66975c640 1 Processor -- start 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.136+0000 7ff66975c640 1 -- start start 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff66975c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 0x7ff664152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff66975c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6640cd690 0x7ff6640b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff66975c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff664161670 0x7ff66415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff66975c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66406d4c0 con 0x7ff6640b2da0 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff66975c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66406d340 con 0x7ff6640cd690 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff66975c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff664161a50 con 0x7ff664161670 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 0x7ff664152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 0x7ff664152410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53566/0 (socket says 172.21.15.122:53566) 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 -- 172.21.15.122:0/3155812165 learned_addr learned my addr 172.21.15.122:0/3155812165 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff6637fe640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff664161670 0x7ff66415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6640cd690 0x7ff6640b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 -- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff664161670 msgr2=0x7ff66415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff664161670 0x7ff66415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 -- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6640cd690 msgr2=0x7ff6640b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6640cd690 0x7ff6640b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff662ffd640 1 -- 172.21.15.122:0/3155812165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640b6da0 con 0x7ff6640b2da0 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.137+0000 7ff6637fe640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff664161670 0x7ff66415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:35.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6640cd690 0x7ff6640b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:35.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 0x7ff664152410 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00d800 tx=0x7ff65c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff64b7fe640 1 -- 172.21.15.122:0/3155812165 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00b6c0 con 0x7ff6640b2da0 2024-03-06T07:26:35.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff64b7fe640 1 -- 172.21.15.122:0/3155812165 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c019070 con 0x7ff6640b2da0 2024-03-06T07:26:35.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6640b7080 con 0x7ff6640b2da0 2024-03-06T07:26:35.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff64b7fe640 1 -- 172.21.15.122:0/3155812165 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00ba80 con 0x7ff6640b2da0 2024-03-06T07:26:35.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.138+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6640b7640 con 0x7ff6640b2da0 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.140+0000 7ff64b7fe640 1 -- 172.21.15.122:0/3155812165 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff65c00bc20 con 0x7ff6640b2da0 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.140+0000 7ff64b7fe640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff65006b4d0 0x7ff65006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.140+0000 7ff64b7fe640 1 -- 172.21.15.122:0/3155812165 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff65c08a710 con 0x7ff6640b2da0 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.140+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff65006b4d0 0x7ff65006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.141+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff65006b4d0 0x7ff65006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff658007fd0 tx=0x7ff658009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.141+0000 7ff66c415740 1 Processor -- start 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.141+0000 7ff66c415740 1 -- start start 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff66c415740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 0x56512157ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56512157e8c0 0x565121580360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff66c415740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651214e46b0 con 0x56512157a870 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512148da60 con 0x56512157e8c0 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651214e0750 con 0x5651215aa600 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 0x56512157ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 0x56512157ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53582/0 (socket says 172.21.15.122:53582) 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 -- 172.21.15.122:0/2190400045 learned_addr learned my addr 172.21.15.122:0/2190400045 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 -- 172.21.15.122:0/2190400045 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 msgr2=0x5651215aaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 -- 172.21.15.122:0/2190400045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56512157e8c0 msgr2=0x565121580360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56512157e8c0 0x565121580360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 -- 172.21.15.122:0/2190400045 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651214abf30 con 0x56512157a870 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff662ffd640 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 0x56512157ac50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7ff65c061010 tx=0x7ff65c011340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b79d6fdca43ab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff648ff9640 1 -- 172.21.15.122:0/2190400045 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c05cf80 con 0x56512157a870 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff648ff9640 1 -- 172.21.15.122:0/2190400045 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c061540 con 0x56512157a870 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.142+0000 7ff648ff9640 1 -- 172.21.15.122:0/2190400045 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0554a0 con 0x56512157a870 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 -- 172.21.15.122:0/2190400045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 msgr2=0x56512157ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 0x56512157ac50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7ff65c061010 tx=0x7ff65c011340 comp rx=0 tx=0).stop 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 -- 172.21.15.122:0/2190400045 shutdown_connections 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215aaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56512157e8c0 0x565121580360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 --2- 172.21.15.122:0/2190400045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56512157a870 0x56512157ac50 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 -- 172.21.15.122:0/2190400045 >> 172.21.15.122:0/2190400045 conn(0x56512157c8f0 msgr2=0x56512157e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 -- 172.21.15.122:0/2190400045 shutdown_connections 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.143+0000 7ff66c415740 1 -- 172.21.15.122:0/2190400045 wait complete. 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 -- start start 2024-03-06T07:26:35.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 Processor -- start 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215c00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 0x5651215adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215a56b0 0x5651215c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512157ed50 con 0x5651215c3160 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651214a0ea0 con 0x5651215a56b0 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.144+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x56512157f030 con 0x5651215aa600 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff66c415740 10 client.0 _ll_register_callbacks cb 0x5651214a0870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 0x5651215adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 0x5651215adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53592/0 (socket says 172.21.15.122:53592) 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 -- 172.21.15.122:0/343094385 learned_addr learned my addr 172.21.15.122:0/343094385 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff662ffd640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215c00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6637fe640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215a56b0 0x5651215c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 -- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 msgr2=0x5651215c00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215c00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 -- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215a56b0 msgr2=0x5651215c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215a56b0 0x5651215c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 -- 172.21.15.122:0/343094385 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651214a1270 con 0x5651215c3160 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff662ffd640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215c00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6637fe640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215a56b0 0x5651215c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff6627fc640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 0x5651215adf90 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7ff6580012e0 tx=0x7ff6580074a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1665d954987e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff641ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff641ffb640 1 -- 172.21.15.122:0/343094385 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff658021420 con 0x5651215c3160 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff641ffb640 1 -- 172.21.15.122:0/343094385 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff658022cd0 con 0x5651215c3160 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff65c061540 con 0x5651215c3160 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff641ffb640 1 -- 172.21.15.122:0/343094385 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff658005050 con 0x5651215c3160 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff66c415740 10 client.7722 fetch_fsmap learned FSMap version 54 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x5651211f92d0 con 0x5651215c3160 2024-03-06T07:26:35.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.145+0000 7ff641ffb640 1 -- 172.21.15.122:0/343094385 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c061540 con 0x5651215c3160 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff641ffb640 1 -- 172.21.15.122:0/343094385 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 54) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff658029050 con 0x5651215c3160 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff66c415740 10 client.7722 fetch_fsmap finished waiting for FSMap version 54 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff66c415740 10 client.7722 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff66c415740 20 client.7722 populate_metadata read hostname 'smithi122' 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff66c415740 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x5651215abf70 0x5651215e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff66c415740 4 client.7722 mds_command: new command op to 7671 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 --> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651214f1bf0 con 0x5651215abf70 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff662ffd640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x5651215abf70 0x5651215e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff641ffb640 10 client.7722 ms_handle_connect on v2:172.21.15.122:6834/498310660 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.146+0000 7ff662ffd640 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x5651215abf70 0x5651215e48f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e575fe19b52ce84 server_cookie=5117a4e68a38b65 in_seq=0 out_seq=0 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff641ffb640 1 -- 172.21.15.122:0/343094385 <== mds.0 v2:172.21.15.122:6834/498310660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5651214f1bf0 con 0x5651215abf70 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff641ffb640 10 client.7722 handle_command_reply: tid=0 2024-03-06T07:26:35.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 client.7722 shutdown 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 msgr2=0x5651215adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 0x5651215adf90 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7ff6580012e0 tx=0x7ff6580074a0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 shutdown_connections 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x5651215abf70 0x5651215e48f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215a56b0 0x5651215c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff641ffb640 0 client.7722 ms_handle_reset on v2:172.21.15.122:6834/498310660 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215c3160 0x5651215adf90 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 --2- 172.21.15.122:0/343094385 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215aa600 0x5651215c00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 >> 172.21.15.122:0/343094385 conn(0x56512157a870 msgr2=0x5651215a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 shutdown_connections 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 1 -- 172.21.15.122:0/343094385 wait complete. 2024-03-06T07:26:35.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.147+0000 7ff66c415740 20 client.7722 trim_cache size 0 max 16384 2024-03-06T07:26:35.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.164+0000 7ff66c415740 1 Processor -- start 2024-03-06T07:26:35.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.164+0000 7ff66c415740 1 -- start start 2024-03-06T07:26:35.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215a56b0 0x56512157e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff66c415740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 0x5651215a8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215c3160 0x5651215a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff66c415740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6641ff5a0 con 0x5651215aa600 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640c9cf0 con 0x5651215c3160 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640ca2f0 con 0x5651215a56b0 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 0x5651215a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 0x5651215a8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53608/0 (socket says 172.21.15.122:53608) 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 -- 172.21.15.122:0/3265858506 learned_addr learned my addr 172.21.15.122:0/3265858506 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 -- 172.21.15.122:0/3265858506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215a56b0 msgr2=0x56512157e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215a56b0 0x56512157e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 -- 172.21.15.122:0/3265858506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215c3160 msgr2=0x5651215a88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215c3160 0x5651215a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 -- 172.21.15.122:0/3265858506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651212c36b0 con 0x5651215aa600 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 0x5651215a8370 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff658001370 tx=0x7ff658019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f7de1df0514fec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff643fff640 1 -- 172.21.15.122:0/3265858506 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6580053e0 con 0x5651215aa600 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff643fff640 1 -- 172.21.15.122:0/3265858506 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff658018040 con 0x5651215aa600 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.165+0000 7ff643fff640 1 -- 172.21.15.122:0/3265858506 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6580057a0 con 0x5651215aa600 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 -- 172.21.15.122:0/3265858506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 msgr2=0x5651215a8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 0x5651215a8370 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff658001370 tx=0x7ff658019000 comp rx=0 tx=0).stop 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 -- 172.21.15.122:0/3265858506 shutdown_connections 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215c3160 0x5651215a88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215aa600 0x5651215a8370 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 --2- 172.21.15.122:0/3265858506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215a56b0 0x56512157e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 -- 172.21.15.122:0/3265858506 >> 172.21.15.122:0/3265858506 conn(0x56512157a870 msgr2=0x5651215c9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 -- 172.21.15.122:0/3265858506 shutdown_connections 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 -- 172.21.15.122:0/3265858506 wait complete. 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.166+0000 7ff66c415740 1 -- start start 2024-03-06T07:26:35.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.167+0000 7ff66c415740 1 Processor -- start 2024-03-06T07:26:35.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.167+0000 7ff66c415740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 0x565121638d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.167+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215aa600 0x565121579510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.167+0000 7ff66c415740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215c3160 0x565121579a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.167+0000 7ff66c415740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651215c06c0 con 0x5651215a56b0 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.167+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651215acec0 con 0x5651215aa600 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff66c415740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651215aba90 con 0x5651215c3160 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff66c415740 10 client.0 _ll_register_callbacks cb 0x7ff65800db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 0x565121638d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff6627fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215aa600 0x565121579510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 0x565121638d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53616/0 (socket says 172.21.15.122:53616) 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 -- 172.21.15.122:0/3012283675 learned_addr learned my addr 172.21.15.122:0/3012283675 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff6637fe640 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215c3160 0x565121579a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 -- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215c3160 msgr2=0x565121579a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215c3160 0x565121579a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 -- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215aa600 msgr2=0x565121579510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215aa600 0x565121579510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 -- 172.21.15.122:0/3012283675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651215e8ab0 con 0x5651215a56b0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff662ffd640 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 0x565121638d30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff65c058dc0 tx=0x7ff65c088ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5e6235d58cc295 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff641ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff641ffb640 1 -- 172.21.15.122:0/3012283675 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0550d0 con 0x5651215a56b0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff641ffb640 1 -- 172.21.15.122:0/3012283675 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff65c056ed0 con 0x5651215a56b0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff658018040 con 0x5651215a56b0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff641ffb640 1 -- 172.21.15.122:0/3012283675 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff65c061540 con 0x5651215a56b0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff66c415740 10 client.7728 fetch_fsmap learned FSMap version 54 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.168+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x5651215a3930 con 0x5651215a56b0 2024-03-06T07:26:35.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff641ffb640 1 -- 172.21.15.122:0/3012283675 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff658018040 con 0x5651215a56b0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff641ffb640 1 -- 172.21.15.122:0/3012283675 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 54) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff65c013070 con 0x5651215a56b0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 10 client.7728 fetch_fsmap finished waiting for FSMap version 54 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 10 client.7728 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 20 client.7728 populate_metadata read hostname 'smithi122' 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x56512159e2b0 0x56512159e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 4 client.7728 mds_command: new command op to 7671 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 --> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56512157d050 con 0x56512159e2b0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x56512159e2b0 0x56512159e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff641ffb640 10 client.7728 ms_handle_connect on v2:172.21.15.122:6834/498310660 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff6627fc640 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x56512159e2b0 0x56512159e690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0e5a8a127afe42e server_cookie=abef4c15271d7a9f in_seq=0 out_seq=0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff641ffb640 1 -- 172.21.15.122:0/3012283675 <== mds.0 v2:172.21.15.122:6834/498310660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56512157d050 con 0x56512159e2b0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff641ffb640 10 client.7728 handle_command_reply: tid=0 2024-03-06T07:26:35.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 client.7728 shutdown 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 msgr2=0x565121638d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 0x565121638d30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff65c058dc0 tx=0x7ff65c088ab0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 shutdown_connections 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.169+0000 7ff66c415740 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:6834/498310660,v1:172.21.15.122:6835/498310660] conn(0x56512159e2b0 0x56512159e690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5651215c3160 0x565121579a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5651215aa600 0x565121579510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff641ffb640 0 client.7728 ms_handle_reset on v2:172.21.15.122:6834/498310660 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 1 --2- 172.21.15.122:0/3012283675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5651215a56b0 0x565121638d30 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 >> 172.21.15.122:0/3012283675 conn(0x56512157a870 msgr2=0x5651215c57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 shutdown_connections 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 1 -- 172.21.15.122:0/3012283675 wait complete. 2024-03-06T07:26:35.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66c415740 20 client.7728 trim_cache size 0 max 16384 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 7671, 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 6.2089503969999997, 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 54, 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 97, 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 97, 2024-03-06T07:26:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 6.8820449310000003 2024-03-06T07:26:35.180 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:26:35.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff65006b4d0 msgr2=0x7ff65006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff65006b4d0 0x7ff65006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff658007fd0 tx=0x7ff658009040 comp rx=0 tx=0).stop 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 msgr2=0x7ff664152410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 0x7ff664152410 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00d800 tx=0x7ff65c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 shutdown_connections 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff65006b4d0 0x7ff65006d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff664161670 0x7ff66415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6640cd690 0x7ff6640b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 --2- 172.21.15.122:0/3155812165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6640b2da0 0x7ff664152410 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 >> 172.21.15.122:0/3155812165 conn(0x7ff6640d1d50 msgr2=0x7ff6640d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 shutdown_connections 2024-03-06T07:26:35.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:35.170+0000 7ff66975c640 1 -- 172.21.15.122:0/3155812165 wait complete. 2024-03-06T07:26:35.183 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:26:35.183 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:26:35.183 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:26:35.183 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> subdir = "delete_me" 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> size_unit = 1024 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> file_multiplier = 200 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> for i in range(0, file_multiplier): 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> f.write(size * '"'"'x'"'"') 2024-03-06T07:26:35.184 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:26:41.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:41.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_ops 2024-03-06T07:26:41.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:41.613 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:26:41.727 INFO:teuthology.orchestra.run.smithi169.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:26:41.727 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:26:41.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json config get mds_max_purge_files 2024-03-06T07:26:41.842 INFO:teuthology.orchestra.run.smithi169.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:26:41.842 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:26:41.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:41.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.950+0000 7f53815ee640 1 Processor -- start 2024-03-06T07:26:41.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.950+0000 7f53815ee640 1 -- start start 2024-03-06T07:26:41.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.951+0000 7f53815ee640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 0x7f537c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.951+0000 7f53815ee640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0d1ea0 con 0x7f537c0cfba0 2024-03-06T07:26:41.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.951+0000 7f53815ee640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0d2180 con 0x7f537c0d1060 2024-03-06T07:26:41.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.951+0000 7f53815ee640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c0d2ab0 con 0x7f537c15e150 2024-03-06T07:26:41.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f537a7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f537c0cfba0 0x7f537c0d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38446/0 (socket says 172.21.15.122:38446) 2024-03-06T07:26:41.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f537a7fc640 1 -- 172.21.15.122:0/1415917253 learned_addr learned my addr 172.21.15.122:0/1415917253 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:41.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f537affd640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 0x7f537c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529509451 0 0) 0x7f537c0d1ea0 con 0x7f537c0cfba0 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5364003520 con 0x7f537c0cfba0 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894221398 0 0) 0x7f5364003520 con 0x7f537c0cfba0 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 msgr2=0x7f537c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 0x7f537c0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 >> v1:172.21.15.169:6789/0 conn(0x7f537c0d1060 legacy=0x7f537c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c0d2460 con 0x7f537c0cfba0 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f5370001670 con 0x7f537c0cfba0 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 >> v1:172.21.15.122:6789/0 conn(0x7f537c0cfba0 legacy=0x7f537c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 0x7f53640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5364003f20 0x7f5364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c0d2ab0 con 0x7f5364003f20 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536400bad0 con 0x7f5364004d40 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.952+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536400bcd0 con 0x7f537c15e150 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5364003f20 0x7f5364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537b7fe640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537affd640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 0x7f53640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537a7fc640 1 -- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 msgr2=0x7f53640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c15e150 0x7f53640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537a7fc640 1 -- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364004d40 msgr2=0x7f53640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.953+0000 7f537a7fc640 1 -- 172.21.15.122:0/1415917253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364003520 con 0x7f5364003f20 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5364003f20 0x7f5364004800 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f537000c210 tx=0x7f5364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb5c625baf43d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370007ce0 con 0x7f5364003f20 2024-03-06T07:26:41.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5370030070 con 0x7f5364003f20 2024-03-06T07:26:41.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f5379ffb640 1 -- 172.21.15.122:0/1415917253 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370005d70 con 0x7f5364003f20 2024-03-06T07:26:41.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f53815ee640 1 -- 172.21.15.122:0/1415917253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5364003f20 msgr2=0x7f5364004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f53815ee640 1 --2- 172.21.15.122:0/1415917253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5364003f20 0x7f5364004800 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f537000c210 tx=0x7f5364005830 comp rx=0 tx=0).stop 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f537b7fe640 1 -- 172.21.15.122:0/1415917253 reap_dead start 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f53815ee640 1 -- 172.21.15.122:0/1415917253 shutdown_connections 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f53815ee640 1 -- 172.21.15.122:0/1415917253 >> 172.21.15.122:0/1415917253 conn(0x7f537c09d230 msgr2=0x7f537c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f53815ee640 1 -- 172.21.15.122:0/1415917253 shutdown_connections 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.954+0000 7f53815ee640 1 -- 172.21.15.122:0/1415917253 wait complete. 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.956+0000 7f53815ee640 1 Processor -- start 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.956+0000 7f53815ee640 1 -- start start 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.956+0000 7f53815ee640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364003f20 0x7f537c15a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53815ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 0x7f537c0cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53815ee640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c1f46d0 0x7f537c1f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53815ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c0b7dc0 con 0x7f537c0cd090 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53815ee640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c0d2f40 con 0x7f5364003f20 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53815ee640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c0d1b30 con 0x7f537c1f46d0 2024-03-06T07:26:41.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 0x7f537c0cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 0x7f537c0cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53644/0 (socket says 172.21.15.122:53644) 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 -- 172.21.15.122:0/1285307958 learned_addr learned my addr 172.21.15.122:0/1285307958 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537b7fe640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c1f46d0 0x7f537c1f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537affd640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364003f20 0x7f537c15a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 -- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c1f46d0 msgr2=0x7f537c1f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c1f46d0 0x7f537c1f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 -- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364003f20 msgr2=0x7f537c15a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364003f20 0x7f537c15a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 -- 172.21.15.122:0/1285307958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c0a1480 con 0x7f537c0cd090 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f537a7fc640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 0x7f537c0cd510 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f537000c300 tx=0x7f537c0d2a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370007ce0 con 0x7f537c0cd090 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.957+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53700049f0 con 0x7f537c0cd090 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.958+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370004de0 con 0x7f537c0cd090 2024-03-06T07:26:41.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.958+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537c0a1760 con 0x7f537c0cd090 2024-03-06T07:26:41.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.958+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c1f41b0 con 0x7f537c0cd090 2024-03-06T07:26:41.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.959+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537c0a1c80 con 0x7f537c0cd090 2024-03-06T07:26:41.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.960+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5370048020 con 0x7f537c0cd090 2024-03-06T07:26:41.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.960+0000 7f53637fe640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f536806b4d0 0x7f536806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.960+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f53700ab320 con 0x7f537c0cd090 2024-03-06T07:26:41.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.960+0000 7f537affd640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f536806b4d0 0x7f536806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.961+0000 7f537affd640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f536806b4d0 0x7f536806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f537c0ccd70 tx=0x7f536c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:41.965+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f537c0a1c80 con 0x7f537c0cd090 2024-03-06T07:26:42.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.212+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f537c1f3fa0 con 0x7f537c0cd090 2024-03-06T07:26:42.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.213+0000 7f53637fe640 1 -- 172.21.15.122:0/1285307958 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f537007a200 con 0x7f537c0cd090 2024-03-06T07:26:42.215 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:42.215 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:42.215 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:42.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f536806b4d0 msgr2=0x7f536806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f536806b4d0 0x7f536806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f537c0ccd70 tx=0x7f536c009190 comp rx=0 tx=0).stop 2024-03-06T07:26:42.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 msgr2=0x7f537c0cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 0x7f537c0cd510 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f537000c300 tx=0x7f537c0d2a20 comp rx=0 tx=0).stop 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 shutdown_connections 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f536806b4d0 0x7f536806d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f537c1f46d0 0x7f537c1f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f537c0cd090 0x7f537c0cd510 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 --2- 172.21.15.122:0/1285307958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5364003f20 0x7f537c15a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 >> 172.21.15.122:0/1285307958 conn(0x7f537c09d230 msgr2=0x7f537c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 shutdown_connections 2024-03-06T07:26:42.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:42.215+0000 7f53815ee640 1 -- 172.21.15.122:0/1285307958 wait complete. 2024-03-06T07:26:42.229 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:42.360 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":35,"msgr_send_messages":31,"msgr_recv_bytes":113281,"msgr_send_bytes":2796119,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.012344255,"msgr_running_send_time":0.004893420,"msgr_running_recv_time":0.008235585,"msgr_running_fast_dispatch_time":0.001610487,"msgr_send_messages_queue_lat":{"avgcount":30,"sum":0.004131244,"avgtime":0.000137708},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":106576,"msgr_send_encrypted_bytes":6672},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4278,"msgr_send_messages":10102,"msgr_recv_bytes":1479803,"msgr_send_bytes":10129920,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.435958415,"msgr_running_send_time":0.228902331,"msgr_running_recv_time":0.241871834,"msgr_running_fast_dispatch_time":0.004430280,"msgr_send_messages_queue_lat":{"avgcount":10101,"sum":0.149064012,"avgtime":0.000014757},"msgr_handle_ack_lat":{"avgcount":14197,"sum":0.002192156,"avgtime":0.000000154},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92672},"AsyncMessenger::Worker-2":{"msgr_recv_messages":12,"msgr_send_messages":13,"msgr_recv_bytes":10211,"msgr_send_bytes":4101,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.002337022,"msgr_running_send_time":0.000242556,"msgr_running_recv_time":0.001733236,"msgr_running_fast_dispatch_time":0.000242284,"msgr_send_messages_queue_lat":{"avgcount":12,"sum":0.004535460,"avgtime":0.000377955},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":71,"sum":0.933808334,"avgtime":0.013152230}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000203015,"avgtime":0.000050753}},"mds":{"request":3286,"reply":3283,"reply_latency":{"avgcount":3283,"sum":3.533219903,"avgtime":0.001076216},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3292,"traverse_hit":3291,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3012,"root_rbytes":2590720,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":55,"ceph_cap_op_grant":3199,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":938,"handle_client_caps_dirty":873,"handle_client_cap_release":0,"process_request_cap_release":3349},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":74,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":74,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4215,"evex":0,"evtrm":0,"ev":4218,"evexg":0,"evexd":0,"segadd":5,"segex":0,"segtrm":0,"seg":7,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":12532298,"rdpos":4195244,"jlat":{"avgcount":10,"sum":0.046407440,"avgtime":0.004640744},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3284,"dn+":3284,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":89760,"heap":248164},"mds_server":{"dispatch_client_request":3289,"dispatch_server_request":0,"handle_client_request":3286,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":73,"sum":0.097219171,"avgtime":0.001331769},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3134,"average_load":3134,"avg_session_uptime":776,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1477803,"buffer_anon_items":492,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5304152,"mds_co_items":26214,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":62,"op_send_bytes":7311166,"op_resend":0,"op_reply":62,"op_latency":{"avgcount":62,"sum":0.126997081,"avgtime":0.002048340},"op_inflight":0,"oplen_avg":{"avgcount":62,"sum":117},"op":62,"op_r":36,"op_w":26,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":10,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":557,"max":104857600,"get_started":0,"get":4320,"get_sum":1211761,"get_or_fail_fail":0,"get_or_fail_success":4320,"take":0,"take_sum":0,"put":4318,"put_sum":1211204,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":7312283,"put":62,"put_sum":7312283,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":62,"put":62,"put_sum":62,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":1380983,"max":3758096384,"get_started":0,"get":4215,"get_sum":8336613,"get_or_fail_fail":0,"get_or_fail_success":4215,"take":0,"take_sum":0,"put":10,"put_sum":6955630,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:42.361 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 113281, "msgr_recv_encrypted_bytes": 106576, "msgr_recv_messages": 35, "msgr_running_fast_dispatch_time": 0.001610487, "msgr_running_recv_time": 0.008235585, "msgr_running_send_time": 0.00489342, "msgr_running_total_time": 0.012344255, "msgr_send_bytes": 2796119, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 30, "avgtime": 0.000137708, "sum": 0.004131244 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 14197, "avgtime": 1.54e-07, "sum": 0.002192156 }, "msgr_recv_bytes": 1479803, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 4278, "msgr_running_fast_dispatch_time": 0.00443028, "msgr_running_recv_time": 0.241871834, "msgr_running_send_time": 0.228902331, "msgr_running_total_time": 0.435958415, "msgr_send_bytes": 10129920, "msgr_send_encrypted_bytes": 92672, "msgr_send_messages": 10102, "msgr_send_messages_queue_lat": { "avgcount": 10101, "avgtime": 1.4757e-05, "sum": 0.149064012 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 10211, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 12, "msgr_running_fast_dispatch_time": 0.000242284, "msgr_running_recv_time": 0.001733236, "msgr_running_send_time": 0.000242556, "msgr_running_total_time": 0.002337022, "msgr_send_bytes": 4101, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 13, "msgr_send_messages_queue_lat": { "avgcount": 12, "avgtime": 0.000377955, "sum": 0.00453546 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 71, "avgtime": 0.01315223, "sum": 0.933808334 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 5.0753e-05, "sum": 0.000203015 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3199, "ceph_cap_op_revoke": 55, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 938, "handle_client_caps_dirty": 873, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3349, "q": 0, "reply": 3283, "reply_latency": { "avgcount": 3283, "avgtime": 0.001076216, "sum": 3.533219903 }, "request": 3286, "root_rbytes": 2590720, "root_rfiles": 3012, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3292, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3291, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 74, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 74, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 4218, "evadd": 4215, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 10, "avgtime": 0.004640744, "sum": 0.04640744 }, "rdpos": 4195244, "replayed": 3, "seg": 7, "segadd": 5, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 12532298 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3284, "dn+": 3284, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 89760 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3289, "dispatch_server_request": 0, "handle_client_request": 3286, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 73, "avgtime": 0.001331769, "sum": 0.097219171 } }, "mds_sessions": { "average_load": 3134, "avg_session_uptime": 776, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3134 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1477803, "buffer_anon_items": 492, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5304152, "mds_co_items": 26214, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 15, "op": 62, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 62, "avgtime": 0.00204834, "sum": 0.126997081 }, "op_pg": 0, "op_r": 36, "op_reply": 62, "op_resend": 0, "op_rmw": 0, "op_send": 62, "op_send_bytes": 7311166, "op_w": 26, "oplen_avg": { "avgcount": 62, "sum": 117 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 6, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 10, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4320, "get_or_fail_fail": 0, "get_or_fail_success": 4320, "get_started": 0, "get_sum": 1211761, "max": 104857600, "put": 4318, "put_sum": 1211204, "take": 0, "take_sum": 0, "val": 557, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 62, "put_sum": 7312283, "take": 62, "take_sum": 7312283, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 62, "put_sum": 62, "take": 62, "take_sum": 62, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 4215, "get_or_fail_fail": 0, "get_or_fail_success": 4215, "get_started": 0, "get_sum": 8336613, "max": 3758096384, "put": 10, "put_sum": 6955630, "take": 0, "take_sum": 0, "val": 1380983, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:42.362 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/74, 0/74 2024-03-06T07:26:43.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.491+0000 7fdfe0edf640 1 Processor -- start 2024-03-06T07:26:43.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.491+0000 7fdfe0edf640 1 -- start start 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfe0edf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfe0edf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdfdc1622d0 0x7fdfdc162730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfe0edf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc0cf2f0 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfe0edf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc0b5140 con 0x7fdfdc1622d0 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfe0edf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0cf8b0 con 0x7fdfdc15e310 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdfdc1622d0 0x7fdfdc162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdfdc1622d0 0x7fdfdc162730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53658/0 (socket says 172.21.15.122:53658) 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 -- 172.21.15.122:0/4229244577 learned_addr learned my addr 172.21.15.122:0/4229244577 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfda59c640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 -- 172.21.15.122:0/4229244577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 msgr2=0x7fdfdc161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 -- 172.21.15.122:0/4229244577 >> v1:172.21.15.169:6789/0 conn(0x7fdfdc0b55d0 legacy=0x7fdfdc0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfd9d9b640 1 -- 172.21.15.122:0/4229244577 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0cf5d0 con 0x7fdfdc1622d0 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.492+0000 7fdfda59c640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:43.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.493+0000 7fdfd9d9b640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc162730 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009870 tx=0x7fdfd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498f48cf95ae8c88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.493+0000 7fdfd959a640 1 -- 172.21.15.122:0/4229244577 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0035020 con 0x7fdfdc1622d0 2024-03-06T07:26:43.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.493+0000 7fdfd959a640 1 -- 172.21.15.122:0/4229244577 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd0031640 con 0x7fdfdc1622d0 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.493+0000 7fdfd959a640 1 -- 172.21.15.122:0/4229244577 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd002d040 con 0x7fdfdc1622d0 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 -- 172.21.15.122:0/4229244577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc1622d0 msgr2=0x7fdfdc162730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc162730 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009870 tx=0x7fdfd002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 -- 172.21.15.122:0/4229244577 shutdown_connections 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc162730 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/4229244577 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 -- 172.21.15.122:0/4229244577 >> 172.21.15.122:0/4229244577 conn(0x7fdfdc09aa40 msgr2=0x7fdfdc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 -- 172.21.15.122:0/4229244577 shutdown_connections 2024-03-06T07:26:43.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.494+0000 7fdfe0edf640 1 -- 172.21.15.122:0/4229244577 wait complete. 2024-03-06T07:26:43.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.496+0000 7fdfe0edf640 1 Processor -- start 2024-03-06T07:26:43.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.496+0000 7fdfe0edf640 1 -- start start 2024-03-06T07:26:43.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfe0edf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 0x7fdfdc0b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfe0edf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfe0edf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfe0edf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc09b7c0 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfe0edf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc09b640 con 0x7fdfdc1622d0 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfe0edf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0d22e0 con 0x7fdfdc15e310 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfda59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 0x7fdfdc0b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfda59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 0x7fdfdc0b1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53668/0 (socket says 172.21.15.122:53668) 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfda59c640 1 -- 172.21.15.122:0/1957650780 learned_addr learned my addr 172.21.15.122:0/1957650780 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfd9d9b640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.497+0000 7fdfdad9d640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfda59c640 1 -- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 msgr2=0x7fdfdc15fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfda59c640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc15fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfda59c640 1 -- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfdc1622d0 msgr2=0x7fdfdc0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfda59c640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc0b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfda59c640 1 -- 172.21.15.122:0/1957650780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc160000 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfda59c640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 0x7fdfdc0b1810 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00e7e0 tx=0x7fdfcc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00f040 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc004030 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc004400 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc0b1ea0 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.498+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc0b2460 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.500+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc069ee0 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.500+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfcc00c6d0 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.501+0000 7fdfcaffd640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.501+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdfcc089f70 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.501+0000 7fdfd9d9b640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.501+0000 7fdfd9d9b640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0036000 tx=0x7fdfd003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.506+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfcc017020 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.818+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfdc159bf0 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.819+0000 7fdfcaffd640 1 -- 172.21.15.122:0/1957650780 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fdfcc058740 con 0x7fdfdc0b55d0 2024-03-06T07:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:43.821 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdfbc06b4d0 msgr2=0x7fdfbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0036000 tx=0x7fdfd003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 msgr2=0x7fdfdc0b1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 0x7fdfdc0b1810 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00e7e0 tx=0x7fdfcc00eca0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 shutdown_connections 2024-03-06T07:26:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdfbc06b4d0 0x7fdfbc06d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfdc1622d0 0x7fdfdc0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdfdc15e310 0x7fdfdc15fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 --2- 172.21.15.122:0/1957650780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfdc0b55d0 0x7fdfdc0b1810 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 >> 172.21.15.122:0/1957650780 conn(0x7fdfdc09aa40 msgr2=0x7fdfdc06efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 shutdown_connections 2024-03-06T07:26:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:43.821+0000 7fdfe0edf640 1 -- 172.21.15.122:0/1957650780 wait complete. 2024-03-06T07:26:43.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:43.993 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":37,"msgr_send_messages":34,"msgr_recv_bytes":113799,"msgr_send_bytes":4228976,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.016120008,"msgr_running_send_time":0.008019153,"msgr_running_recv_time":0.008271847,"msgr_running_fast_dispatch_time":0.002211364,"msgr_send_messages_queue_lat":{"avgcount":33,"sum":0.004430018,"avgtime":0.000134242},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":106848,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6183,"msgr_send_messages":13190,"msgr_recv_bytes":2168666,"msgr_send_bytes":13726370,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.573172318,"msgr_running_send_time":0.306395385,"msgr_running_recv_time":0.313276592,"msgr_running_fast_dispatch_time":0.005409333,"msgr_send_messages_queue_lat":{"avgcount":13189,"sum":0.190766692,"avgtime":0.000014464},"msgr_handle_ack_lat":{"avgcount":18789,"sum":0.003010632,"avgtime":0.000000160},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":12,"msgr_send_messages":15,"msgr_recv_bytes":10211,"msgr_send_bytes":4101,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.002337022,"msgr_running_send_time":0.000242556,"msgr_running_recv_time":0.001733236,"msgr_running_fast_dispatch_time":0.000242284,"msgr_send_messages_queue_lat":{"avgcount":13,"sum":0.004892553,"avgtime":0.000376350},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1047,"complete_latency":{"avgcount":76,"sum":1.563608817,"avgtime":0.020573800}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000203015,"avgtime":0.000050753}},"mds":{"request":3891,"reply":3889,"reply_latency":{"avgcount":3889,"sum":4.720380363,"avgtime":0.001213777},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3865,"inodes_top":0,"inodes_bottom":3865,"inodes_pin_tail":0,"inodes_pinned":3865,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3897,"traverse_hit":3896,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3178,"root_rbytes":6738944,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":439,"ceph_cap_op_grant":4066,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1891,"handle_client_caps_dirty":1770,"handle_client_cap_release":1,"process_request_cap_release":4559},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":679,"num_strays_delayed":0,"num_strays_enqueuing":6,"strays_created":679,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":5775,"evex":0,"evtrm":0,"ev":5778,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":9,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16323947,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.065789358,"avgtime":0.005482446},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3890,"dn+":3890,"dn-":0,"cap":3196,"cap+":3202,"cap-":6,"rss":93344,"heap":248164},"mds_server":{"dispatch_client_request":3894,"dispatch_server_request":0,"handle_client_request":3891,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":679,"sum":1.284379631,"avgtime":0.001891575},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3679,"average_load":3679,"avg_session_uptime":781,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":15530203,"buffer_anon_items":4204,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13160482,"mds_co_items":89689,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":8,"op_laggy":0,"op_send":74,"op_send_bytes":11203917,"op_resend":0,"op_reply":66,"op_latency":{"avgcount":66,"sum":0.147996598,"avgtime":0.002242372},"op_inflight":0,"oplen_avg":{"avgcount":74,"sum":133},"op":74,"op_r":36,"op_w":38,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":13,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":88958,"max":104857600,"get_started":0,"get":6227,"get_sum":1729460,"get_or_fail_fail":0,"get_or_fail_success":6227,"take":0,"take_sum":0,"put":5885,"put_sum":1640502,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":9991,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":74,"take_sum":11205034,"put":66,"put_sum":11195043,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":8,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":74,"take_sum":74,"put":66,"put_sum":66,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":606,"max":3758096384,"get_started":0,"get":6,"get_sum":606,"get_or_fail_fail":0,"get_or_fail_success":6,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":1301012,"max":3758096384,"get_started":0,"get":5776,"get_sum":12130194,"get_or_fail_fail":0,"get_or_fail_success":5776,"take":0,"take_sum":0,"put":12,"put_sum":10829182,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:43.994 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 113799, "msgr_recv_encrypted_bytes": 106848, "msgr_recv_messages": 37, "msgr_running_fast_dispatch_time": 0.002211364, "msgr_running_recv_time": 0.008271847, "msgr_running_send_time": 0.008019153, "msgr_running_total_time": 0.016120008, "msgr_send_bytes": 4228976, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 34, "msgr_send_messages_queue_lat": { "avgcount": 33, "avgtime": 0.000134242, "sum": 0.004430018 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18789, "avgtime": 1.6e-07, "sum": 0.003010632 }, "msgr_recv_bytes": 2168666, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 6183, "msgr_running_fast_dispatch_time": 0.005409333, "msgr_running_recv_time": 0.313276592, "msgr_running_send_time": 0.306395385, "msgr_running_total_time": 0.573172318, "msgr_send_bytes": 13726370, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 13190, "msgr_send_messages_queue_lat": { "avgcount": 13189, "avgtime": 1.4464e-05, "sum": 0.190766692 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 10211, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 12, "msgr_running_fast_dispatch_time": 0.000242284, "msgr_running_recv_time": 0.001733236, "msgr_running_send_time": 0.000242556, "msgr_running_total_time": 0.002337022, "msgr_send_bytes": 4101, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 15, "msgr_send_messages_queue_lat": { "avgcount": 13, "avgtime": 0.00037635, "sum": 0.004892553 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 76, "avgtime": 0.0205738, "sum": 1.563608817 }, "queue_len": 1047 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 5.0753e-05, "sum": 0.000203015 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4066, "ceph_cap_op_revoke": 439, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 1891, "handle_client_caps_dirty": 1770, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3865, "inodes_bottom": 3865, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3865, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 4559, "q": 0, "reply": 3889, "reply_latency": { "avgcount": 3889, "avgtime": 0.001213777, "sum": 4.720380363 }, "request": 3891, "root_rbytes": 6738944, "root_rfiles": 3178, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3897, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3896, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 679, "num_strays_delayed": 0, "num_strays_enqueuing": 6, "recovery_completed": 0, "recovery_started": 0, "strays_created": 679, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5778, "evadd": 5775, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.005482446, "sum": 0.065789358 }, "rdpos": 4195244, "replayed": 3, "seg": 9, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 16323947 }, "mds_mem": { "cap": 3196, "cap+": 3202, "cap-": 6, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3890, "dn+": 3890, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 93344 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3894, "dispatch_server_request": 0, "handle_client_request": 3891, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 679, "avgtime": 0.001891575, "sum": 1.284379631 } }, "mds_sessions": { "average_load": 3679, "avg_session_uptime": 781, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3679 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 15530203, "buffer_anon_items": 4204, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13160482, "mds_co_items": 89689, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 19, "op": 74, "op_Infinitylight": 0, "op_active": 8, "op_laggy": 0, "op_latency": { "avgcount": 66, "avgtime": 0.002242372, "sum": 0.147996598 }, "op_pg": 0, "op_r": 36, "op_reply": 66, "op_resend": 0, "op_rmw": 0, "op_send": 74, "op_send_bytes": 11203917, "op_w": 38, "oplen_avg": { "avgcount": 74, "sum": 133 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 13, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6227, "get_or_fail_fail": 0, "get_or_fail_success": 6227, "get_started": 0, "get_sum": 1729460, "max": 104857600, "put": 5885, "put_sum": 1640502, "take": 0, "take_sum": 0, "val": 88958, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 66, "put_sum": 11195043, "take": 74, "take_sum": 11205034, "val": 9991, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 66, "put_sum": 66, "take": 74, "take_sum": 74, "val": 8, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 6, "get_or_fail_fail": 0, "get_or_fail_success": 6, "get_started": 0, "get_sum": 606, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 606, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 5776, "get_or_fail_fail": 0, "get_or_fail_success": 5776, "get_started": 0, "get_sum": 12130194, "max": 3758096384, "put": 12, "put_sum": 10829182, "take": 0, "take_sum": 0, "val": 1301012, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:43.995 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/679, 0/679 2024-03-06T07:26:44.996 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:45.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.120+0000 7ff0516df640 1 Processor -- start 2024-03-06T07:26:45.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.120+0000 7ff0516df640 1 -- start start 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff0516df640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff04c161670 0x7ff04c161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff0516df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff04c0b2da0 0x7ff04c1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff0516df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0cc680 0x7ff04c0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff0516df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04c0b3590 con 0x7ff04c0b2da0 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff0516df640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04c162c70 con 0x7ff04c161670 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff0516df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c06cd80 con 0x7ff04c0cc680 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff04c0b2da0 0x7ff04c1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff04c0b2da0 0x7ff04c1623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52000/0 (socket says 172.21.15.122:52000) 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 -- 172.21.15.122:0/777050515 learned_addr learned my addr 172.21.15.122:0/777050515 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:45.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff04b7fe640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0cc680 0x7ff04c0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff04affd640 1 --2- 172.21.15.122:0/777050515 >> v2:172.21.15.169:3300/0 conn(0x7ff04c161670 0x7ff04c161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 -- 172.21.15.122:0/777050515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0cc680 msgr2=0x7ff04c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0cc680 0x7ff04c0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 -- 172.21.15.122:0/777050515 >> v2:172.21.15.169:3300/0 conn(0x7ff04c161670 msgr2=0x7ff04c161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 --2- 172.21.15.122:0/777050515 >> v2:172.21.15.169:3300/0 conn(0x7ff04c161670 0x7ff04c161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff043fff640 1 -- 172.21.15.122:0/777050515 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c06d060 con 0x7ff04c0b2da0 2024-03-06T07:26:45.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.121+0000 7ff04b7fe640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0cc680 0x7ff04c0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:45.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.123+0000 7ff043fff640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0b2da0 0x7ff04c1623a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d981f3d262ead7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.123+0000 7ff04a7fc640 1 -- 172.21.15.122:0/777050515 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034035020 con 0x7ff04c0b2da0 2024-03-06T07:26:45.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.123+0000 7ff04a7fc640 1 -- 172.21.15.122:0/777050515 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034031640 con 0x7ff04c0b2da0 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.123+0000 7ff04a7fc640 1 -- 172.21.15.122:0/777050515 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03402d040 con 0x7ff04c0b2da0 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.123+0000 7ff0516df640 1 -- 172.21.15.122:0/777050515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0b2da0 msgr2=0x7ff04c1623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.123+0000 7ff0516df640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0b2da0 0x7ff04c1623a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 -- 172.21.15.122:0/777050515 shutdown_connections 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0b2da0 0x7ff04c1623a0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 --2- 172.21.15.122:0/777050515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0cc680 0x7ff04c0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 --2- 172.21.15.122:0/777050515 >> v2:172.21.15.169:3300/0 conn(0x7ff04c161670 0x7ff04c161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 -- 172.21.15.122:0/777050515 >> 172.21.15.122:0/777050515 conn(0x7ff04c0d1d50 msgr2=0x7ff04c0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 -- 172.21.15.122:0/777050515 shutdown_connections 2024-03-06T07:26:45.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.124+0000 7ff0516df640 1 -- 172.21.15.122:0/777050515 wait complete. 2024-03-06T07:26:45.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.125+0000 7ff0516df640 1 Processor -- start 2024-03-06T07:26:45.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.125+0000 7ff0516df640 1 -- start start 2024-03-06T07:26:45.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff0516df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0b2da0 0x7ff04c15a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff0516df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 0x7ff04c0c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff0516df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff04c161670 0x7ff04c0b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff0516df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c06d340 con 0x7ff04c0cc680 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff0516df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1628e0 con 0x7ff04c161670 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff0516df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c06d4c0 con 0x7ff04c0b2da0 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff043fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 0x7ff04c0c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff043fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 0x7ff04c0c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52016/0 (socket says 172.21.15.122:52016) 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff043fff640 1 -- 172.21.15.122:0/413043316 learned_addr learned my addr 172.21.15.122:0/413043316 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff04b7fe640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff04c161670 0x7ff04c0b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.126+0000 7ff04affd640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0b2da0 0x7ff04c15a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff043fff640 1 -- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0b2da0 msgr2=0x7ff04c15a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff043fff640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0b2da0 0x7ff04c15a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff043fff640 1 -- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff04c161670 msgr2=0x7ff04c0b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff043fff640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff04c161670 0x7ff04c0b6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff043fff640 1 -- 172.21.15.122:0/413043316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c0c6d30 con 0x7ff04c0cc680 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff043fff640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 0x7ff04c0c6260 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7ff034000c00 tx=0x7ff0340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034040070 con 0x7ff04c0cc680 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0340399f0 con 0x7ff04c0cc680 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034035040 con 0x7ff04c0cc680 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c0c7010 con 0x7ff04c0cc680 2024-03-06T07:26:45.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.127+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c0c75d0 con 0x7ff04c0cc680 2024-03-06T07:26:45.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.129+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c068050 con 0x7ff04c0cc680 2024-03-06T07:26:45.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.129+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff03404b060 con 0x7ff04c0cc680 2024-03-06T07:26:45.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.129+0000 7ff0437fe640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.130+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff0340ab9d0 con 0x7ff04c0cc680 2024-03-06T07:26:45.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.130+0000 7ff04affd640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.130+0000 7ff04affd640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff02c06b4d0 0x7ff02c06d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff03c006fd0 tx=0x7ff03c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.137+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff03407a1a0 con 0x7ff04c0cc680 2024-03-06T07:26:45.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.469+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04c0b7cd0 con 0x7ff04c0cc680 2024-03-06T07:26:45.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.470+0000 7ff0437fe640 1 -- 172.21.15.122:0/413043316 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7ff03407e240 con 0x7ff04c0cc680 2024-03-06T07:26:45.472 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:45.472 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:45.472 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:45.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.472+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff02c06b4d0 msgr2=0x7ff02c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.472+0000 7ff0516df640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff02c06b4d0 0x7ff02c06d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff03c006fd0 tx=0x7ff03c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.472+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 msgr2=0x7ff04c0c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.472+0000 7ff0516df640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 0x7ff04c0c6260 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7ff034000c00 tx=0x7ff0340046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 shutdown_connections 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff04c161670 0x7ff04c0b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff04c0cc680 0x7ff04c0c6260 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 --2- 172.21.15.122:0/413043316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff04c0b2da0 0x7ff04c15a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 >> 172.21.15.122:0/413043316 conn(0x7ff04c0d1d50 msgr2=0x7ff04c070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 shutdown_connections 2024-03-06T07:26:45.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:45.473+0000 7ff0516df640 1 -- 172.21.15.122:0/413043316 wait complete. 2024-03-06T07:26:45.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:45.640 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":133,"msgr_send_messages":129,"msgr_recv_bytes":138175,"msgr_send_bytes":4259946,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.033047088,"msgr_running_send_time":0.018376596,"msgr_running_recv_time":0.010839400,"msgr_running_fast_dispatch_time":0.005557942,"msgr_send_messages_queue_lat":{"avgcount":128,"sum":0.014290608,"avgtime":0.000111645},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":106848,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7316,"msgr_send_messages":16347,"msgr_recv_bytes":2582129,"msgr_send_bytes":19007653,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.709725718,"msgr_running_send_time":0.388957419,"msgr_running_recv_time":0.365134366,"msgr_running_fast_dispatch_time":0.009977182,"msgr_send_messages_queue_lat":{"avgcount":16346,"sum":0.256589637,"avgtime":0.000015697},"msgr_handle_ack_lat":{"avgcount":22247,"sum":0.003678523,"avgtime":0.000000165},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":180,"msgr_send_messages":181,"msgr_recv_bytes":86917,"msgr_send_bytes":93173,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.028290415,"msgr_running_send_time":0.015570504,"msgr_running_recv_time":0.005651051,"msgr_running_fast_dispatch_time":0.005436538,"msgr_send_messages_queue_lat":{"avgcount":180,"sum":0.026164574,"avgtime":0.000145358},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1055,"complete_latency":{"avgcount":82,"sum":3.519377774,"avgtime":0.042919241}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":521,"sum":0.334620936,"avgtime":0.000642266}},"mds":{"request":4390,"reply":4387,"reply_latency":{"avgcount":4387,"sum":6.111846614,"avgtime":0.001393172},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4382,"inodes_top":0,"inodes_bottom":4382,"inodes_pin_tail":0,"inodes_pinned":4382,"inodes_expired":0,"inodes_with_caps":2864,"caps":2864,"subtrees":2,"traverse":4396,"traverse_hit":4395,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2462,"root_rbytes":11721728,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":598,"ceph_cap_op_grant":4922,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2768,"handle_client_caps_dirty":2588,"handle_client_cap_release":2,"process_request_cap_release":5557},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":840,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1178,"strays_enqueued":338,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7493,"evex":0,"evtrm":0,"ev":7496,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":19821302,"rdpos":4195244,"jlat":{"avgcount":15,"sum":0.278652408,"avgtime":0.018576827},"replayed":3},"mds_mem":{"ino":3208,"ino+":3214,"ino-":6,"dir":13,"dir+":13,"dir-":0,"dn":4382,"dn+":4388,"dn-":6,"cap":2864,"cap+":3202,"cap-":338,"rss":93344,"heap":248164},"mds_server":{"dispatch_client_request":4393,"dispatch_server_request":0,"handle_client_request":4390,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1177,"sum":2.675845882,"avgtime":0.002273445},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4100,"average_load":4100,"avg_session_uptime":781,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":15530203,"buffer_anon_items":4204,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13160482,"mds_co_items":89689,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":427,"op_send_bytes":15445022,"op_resend":0,"op_reply":427,"op_latency":{"avgcount":427,"sum":0.799981615,"avgtime":0.001873493},"op_inflight":0,"oplen_avg":{"avgcount":427,"sum":488},"op":427,"op_r":40,"op_w":387,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":12,"osdop_write":19,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":352,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":21,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":338,"pq_executed":338,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":7624,"get_sum":2118275,"get_or_fail_fail":0,"get_or_fail_success":7624,"take":0,"take_sum":0,"put":7623,"put_sum":2117979,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":427,"take_sum":15480277,"put":427,"put_sum":15480277,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":427,"take_sum":427,"put":427,"put_sum":427,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":338,"get_sum":34138,"get_or_fail_fail":0,"get_or_fail_success":338,"take":0,"take_sum":0,"put":4,"put_sum":34138,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":597553,"max":3758096384,"get_started":0,"get":7493,"get_sum":15625617,"get_or_fail_fail":0,"get_or_fail_success":7493,"take":0,"take_sum":0,"put":15,"put_sum":15028064,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:45.641 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 138175, "msgr_recv_encrypted_bytes": 106848, "msgr_recv_messages": 133, "msgr_running_fast_dispatch_time": 0.005557942, "msgr_running_recv_time": 0.0108394, "msgr_running_send_time": 0.018376596, "msgr_running_total_time": 0.033047088, "msgr_send_bytes": 4259946, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 129, "msgr_send_messages_queue_lat": { "avgcount": 128, "avgtime": 0.000111645, "sum": 0.014290608 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22247, "avgtime": 1.65e-07, "sum": 0.003678523 }, "msgr_recv_bytes": 2582129, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 7316, "msgr_running_fast_dispatch_time": 0.009977182, "msgr_running_recv_time": 0.365134366, "msgr_running_send_time": 0.388957419, "msgr_running_total_time": 0.709725718, "msgr_send_bytes": 19007653, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 16347, "msgr_send_messages_queue_lat": { "avgcount": 16346, "avgtime": 1.5697e-05, "sum": 0.256589637 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 86917, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 180, "msgr_running_fast_dispatch_time": 0.005436538, "msgr_running_recv_time": 0.005651051, "msgr_running_send_time": 0.015570504, "msgr_running_total_time": 0.028290415, "msgr_send_bytes": 93173, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 181, "msgr_send_messages_queue_lat": { "avgcount": 180, "avgtime": 0.000145358, "sum": 0.026164574 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 82, "avgtime": 0.042919241, "sum": 3.519377774 }, "queue_len": 1055 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 521, "avgtime": 0.000642266, "sum": 0.334620936 }, "queue_len": 0 }, "mds": { "caps": 2864, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4922, "ceph_cap_op_revoke": 598, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 2768, "handle_client_caps_dirty": 2588, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4382, "inodes_bottom": 4382, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4382, "inodes_top": 0, "inodes_with_caps": 2864, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 5557, "q": 0, "reply": 4387, "reply_latency": { "avgcount": 4387, "avgtime": 0.001393172, "sum": 6.111846614 }, "request": 4390, "root_rbytes": 11721728, "root_rfiles": 2462, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4396, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4395, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 840, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1178, "strays_enqueued": 338, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7496, "evadd": 7493, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 15, "avgtime": 0.018576827, "sum": 0.278652408 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 19821302 }, "mds_mem": { "cap": 2864, "cap+": 3202, "cap-": 338, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4382, "dn+": 4388, "dn-": 6, "heap": 248164, "ino": 3208, "ino+": 3214, "ino-": 6, "rss": 93344 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4393, "dispatch_server_request": 0, "handle_client_request": 4390, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1177, "avgtime": 0.002273445, "sum": 2.675845882 } }, "mds_sessions": { "average_load": 4100, "avg_session_uptime": 781, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4100 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 15530203, "buffer_anon_items": 4204, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13160482, "mds_co_items": 89689, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 21, "op": 427, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 427, "avgtime": 0.001873493, "sum": 0.799981615 }, "op_pg": 0, "op_r": 40, "op_reply": 427, "op_resend": 0, "op_rmw": 0, "op_send": 427, "op_send_bytes": 15445022, "op_w": 387, "oplen_avg": { "avgcount": 427, "sum": 488 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 352, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 12, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 19, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 338, "pq_executed_ops": 338, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7624, "get_or_fail_fail": 0, "get_or_fail_success": 7624, "get_started": 0, "get_sum": 2118275, "max": 104857600, "put": 7623, "put_sum": 2117979, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 427, "put_sum": 15480277, "take": 427, "take_sum": 15480277, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 427, "put_sum": 427, "take": 427, "take_sum": 427, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 338, "get_or_fail_fail": 0, "get_or_fail_success": 338, "get_started": 0, "get_sum": 34138, "max": 3758096384, "put": 4, "put_sum": 34138, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 7493, "get_or_fail_fail": 0, "get_or_fail_success": 7493, "get_started": 0, "get_sum": 15625617, "max": 3758096384, "put": 15, "put_sum": 15028064, "take": 0, "take_sum": 0, "val": 597553, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:45.642 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/840, 338/1178 2024-03-06T07:26:46.643 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.767+0000 7fb7d464e640 1 Processor -- start 2024-03-06T07:26:46.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.767+0000 7fb7d464e640 1 -- start start 2024-03-06T07:26:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d464e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 0x7fb7cc0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d464e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb7cc0cfba0 0x7fb7cc0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d464e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc0d1ea0 con 0x7fb7cc0d0e90 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d464e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7cc0d2ab0 con 0x7fb7cc0cfba0 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d464e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0d2180 con 0x7fb7cc15e150 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d2beb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb7cc0d0e90 0x7fb7cc0b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59004/0 (socket says 172.21.15.122:59004) 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.768+0000 7fb7d2beb640 1 -- 172.21.15.122:0/3152322854 learned_addr learned my addr 172.21.15.122:0/3152322854 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d23ea640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 0x7fb7cc0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3152322854 >> v2:172.21.15.169:3300/0 conn(0x7fb7cc0cfba0 0x7fb7cc0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48766151 0 0) 0x7fb7cc0d1ea0 con 0x7fb7cc0d0e90 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b8003520 con 0x7fb7cc0d0e90 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565631568 0 0) 0x7fb7b8003520 con 0x7fb7cc0d0e90 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 msgr2=0x7fb7cc0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 0x7fb7cc0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 >> v2:172.21.15.169:3300/0 conn(0x7fb7cc0cfba0 msgr2=0x7fb7cc0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 --2- 172.21.15.122:0/3152322854 >> v2:172.21.15.169:3300/0 conn(0x7fb7cc0cfba0 0x7fb7cc0d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0d2460 con 0x7fb7cc0d0e90 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb7c40019f0 con 0x7fb7cc0d0e90 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 >> v1:172.21.15.122:6789/0 conn(0x7fb7cc0d0e90 legacy=0x7fb7cc0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7b80039b0 0x7fb7b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7b8004800 0x7fb7b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 0x7fb7b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0d2ab0 con 0x7fb7b8004800 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0d2180 con 0x7fb7b80039b0 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b800b9c0 con 0x7fb7cc15e150 2024-03-06T07:26:46.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.769+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7b8004800 0x7fb7b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d2beb640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 0x7fb7b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d23ea640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7b80039b0 0x7fb7b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 msgr2=0x7fb7b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7cc15e150 0x7fb7b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7b80039b0 msgr2=0x7fb7b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7b80039b0 0x7fb7b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3152322854 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8003520 con 0x7fb7b8004800 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.770+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7b8004800 0x7fb7b8004c80 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009af0 tx=0x7fb7b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa3de093fffdd03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.771+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc039070 con 0x7fb7b8004800 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.771+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc02c710 con 0x7fb7b8004800 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.771+0000 7fb7d13e8640 1 -- 172.21.15.122:0/3152322854 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc009690 con 0x7fb7b8004800 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.771+0000 7fb7d464e640 1 -- 172.21.15.122:0/3152322854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7b8004800 msgr2=0x7fb7b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.771+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3152322854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7b8004800 0x7fb7b8004c80 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009af0 tx=0x7fb7b8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.771+0000 7fb7d2beb640 1 -- 172.21.15.122:0/3152322854 reap_dead start 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.772+0000 7fb7d464e640 1 -- 172.21.15.122:0/3152322854 shutdown_connections 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.772+0000 7fb7d464e640 1 -- 172.21.15.122:0/3152322854 >> 172.21.15.122:0/3152322854 conn(0x7fb7cc09d230 msgr2=0x7fb7cc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.772+0000 7fb7d464e640 1 -- 172.21.15.122:0/3152322854 shutdown_connections 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.772+0000 7fb7d464e640 1 -- 172.21.15.122:0/3152322854 wait complete. 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.773+0000 7fb7d464e640 1 Processor -- start 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.773+0000 7fb7d464e640 1 -- start start 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d464e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7b8004800 0x7fb7cc0b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d464e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 0x7fb7cc0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d1be9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 0x7fb7cc0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d1be9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 0x7fb7cc0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52036/0 (socket says 172.21.15.122:52036) 2024-03-06T07:26:46.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d464e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7cc0b10d0 0x7fb7cc0cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3060930543 learned_addr learned my addr 172.21.15.122:0/3060930543 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0d1b30 con 0x7fb7cc15c560 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0d2f40 con 0x7fb7cc0b10d0 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc15e530 con 0x7fb7b8004800 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.774+0000 7fb7d23ea640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7b8004800 0x7fb7cc0b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d2beb640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7cc0b10d0 0x7fb7cc0cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7b8004800 msgr2=0x7fb7cc0b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7b8004800 0x7fb7cc0b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7cc0b10d0 msgr2=0x7fb7cc0cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7cc0b10d0 0x7fb7cc0cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d1be9640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0b37c0 con 0x7fb7cc15c560 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d2beb640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7cc0b10d0 0x7fb7cc0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7d1be9640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 0x7fb7cc0b0b90 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc037040 tx=0x7fb7bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc039050 con 0x7fb7cc15c560 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc045070 con 0x7fb7cc15c560 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.775+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0044e0 con 0x7fb7cc15c560 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.776+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc0b3aa0 con 0x7fb7cc15c560 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.776+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc0b9c40 con 0x7fb7cc15c560 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.777+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc069ee0 con 0x7fb7cc15c560 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.777+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7bc003a50 con 0x7fb7cc15c560 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.778+0000 7fb7c2ffd640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb7b006b500 0x7fb7b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.778+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7bc0abd10 con 0x7fb7cc15c560 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.778+0000 7fb7d23ea640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb7b006b500 0x7fb7b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.778+0000 7fb7d23ea640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb7b006b500 0x7fb7b006d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80099e0 tx=0x7fb7c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:46.784+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7bc04e020 con 0x7fb7cc15c560 2024-03-06T07:26:47.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.111+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7cc0d0560 con 0x7fb7cc15c560 2024-03-06T07:26:47.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.112+0000 7fb7c2ffd640 1 -- 172.21.15.122:0/3060930543 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fb7bc07a690 con 0x7fb7cc15c560 2024-03-06T07:26:47.114 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:47.115 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:47.115 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:47.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.116+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb7b006b500 msgr2=0x7fb7b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.116+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb7b006b500 0x7fb7b006d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80099e0 tx=0x7fb7c8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:47.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.116+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 msgr2=0x7fb7cc0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.116+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 0x7fb7cc0b0b90 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc037040 tx=0x7fb7bc004730 comp rx=0 tx=0).stop 2024-03-06T07:26:47.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.116+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 shutdown_connections 2024-03-06T07:26:47.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.116+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb7b006b500 0x7fb7b006d9c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.117+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7cc0b10d0 0x7fb7cc0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.117+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7cc15c560 0x7fb7cc0b0b90 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.117+0000 7fb7d464e640 1 --2- 172.21.15.122:0/3060930543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7b8004800 0x7fb7cc0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.117+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 >> 172.21.15.122:0/3060930543 conn(0x7fb7cc09d230 msgr2=0x7fb7cc06cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.117+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 shutdown_connections 2024-03-06T07:26:47.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:47.117+0000 7fb7d464e640 1 -- 172.21.15.122:0/3060930543 wait complete. 2024-03-06T07:26:47.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:47.309 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":315,"msgr_send_messages":311,"msgr_recv_bytes":184403,"msgr_send_bytes":4319278,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.052610861,"msgr_running_send_time":0.026807632,"msgr_running_recv_time":0.015208772,"msgr_running_fast_dispatch_time":0.010305432,"msgr_send_messages_queue_lat":{"avgcount":310,"sum":0.026490816,"avgtime":0.000085454},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":106848,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8720,"msgr_send_messages":19015,"msgr_recv_bytes":3100415,"msgr_send_bytes":21762607,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.848863701,"msgr_running_send_time":0.470307836,"msgr_running_recv_time":0.425548851,"msgr_running_fast_dispatch_time":0.015760065,"msgr_send_messages_queue_lat":{"avgcount":19014,"sum":0.314665863,"avgtime":0.000016549},"msgr_handle_ack_lat":{"avgcount":25454,"sum":0.004219845,"avgtime":0.000000165},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":524,"msgr_send_messages":525,"msgr_recv_bytes":244368,"msgr_send_bytes":275572,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.066970840,"msgr_running_send_time":0.036146141,"msgr_running_recv_time":0.013814247,"msgr_running_fast_dispatch_time":0.013362316,"msgr_send_messages_queue_lat":{"avgcount":524,"sum":0.047407637,"avgtime":0.000090472},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":86,"sum":5.124310664,"avgtime":0.059585007}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1597,"sum":0.656565747,"avgtime":0.000411124}},"mds":{"request":5166,"reply":5163,"reply_latency":{"avgcount":5163,"sum":7.311916174,"avgtime":0.001416214},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4522,"inodes_top":0,"inodes_bottom":4522,"inodes_pin_tail":0,"inodes_pinned":4522,"inodes_expired":0,"inodes_with_caps":2523,"caps":2523,"subtrees":2,"traverse":5172,"traverse_hit":5171,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2167,"root_rbytes":12989440,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":666,"ceph_cap_op_grant":5473,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3204,"handle_client_caps_dirty":3000,"handle_client_cap_release":4,"process_request_cap_release":7109},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":921,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1954,"strays_enqueued":1033,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9403,"evex":0,"evtrm":0,"ev":9406,"evexg":0,"evexd":0,"segadd":12,"segex":0,"segtrm":0,"seg":14,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23557270,"rdpos":4195244,"jlat":{"avgcount":16,"sum":0.344316092,"avgtime":0.021519755},"replayed":3},"mds_mem":{"ino":2718,"ino+":3214,"ino-":496,"dir":13,"dir+":13,"dir-":0,"dn":4668,"dn+":5164,"dn-":496,"cap":2169,"cap+":3202,"cap-":1033,"rss":93344,"heap":248164},"mds_server":{"dispatch_client_request":5169,"dispatch_server_request":0,"handle_client_request":5166,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1953,"sum":3.875915442,"avgtime":0.001984595},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4791,"average_load":4791,"avg_session_uptime":781,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":15530203,"buffer_anon_items":4204,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13160482,"mds_co_items":89689,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1140,"op_send_bytes":17273894,"op_resend":0,"op_reply":1140,"op_latency":{"avgcount":1140,"sum":2.034953211,"avgtime":0.001785046},"op_inflight":0,"oplen_avg":{"avgcount":1140,"sum":1205},"op":1140,"op_r":46,"op_w":1094,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":18,"osdop_write":26,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1048,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":25,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1033,"pq_executed":1033,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":9554,"get_sum":2666540,"get_or_fail_fail":0,"get_or_fail_success":9554,"take":0,"take_sum":0,"put":9554,"put_sum":2666540,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1140,"take_sum":17379344,"put":1140,"put_sum":17379344,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1140,"take_sum":1140,"put":1140,"put_sum":1140,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1033,"get_sum":104333,"get_or_fail_fail":0,"get_or_fail_success":1033,"take":0,"take_sum":0,"put":10,"put_sum":104333,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":2588934,"max":3758096384,"get_started":0,"get":9404,"get_sum":19364769,"get_or_fail_fail":0,"get_or_fail_success":9404,"take":0,"take_sum":0,"put":16,"put_sum":16775835,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:47.310 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 184403, "msgr_recv_encrypted_bytes": 106848, "msgr_recv_messages": 315, "msgr_running_fast_dispatch_time": 0.010305432, "msgr_running_recv_time": 0.015208772, "msgr_running_send_time": 0.026807632, "msgr_running_total_time": 0.052610861, "msgr_send_bytes": 4319278, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 311, "msgr_send_messages_queue_lat": { "avgcount": 310, "avgtime": 8.5454e-05, "sum": 0.026490816 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 25454, "avgtime": 1.65e-07, "sum": 0.004219845 }, "msgr_recv_bytes": 3100415, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 8720, "msgr_running_fast_dispatch_time": 0.015760065, "msgr_running_recv_time": 0.425548851, "msgr_running_send_time": 0.470307836, "msgr_running_total_time": 0.848863701, "msgr_send_bytes": 21762607, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 19015, "msgr_send_messages_queue_lat": { "avgcount": 19014, "avgtime": 1.6549e-05, "sum": 0.314665863 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 244368, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 524, "msgr_running_fast_dispatch_time": 0.013362316, "msgr_running_recv_time": 0.013814247, "msgr_running_send_time": 0.036146141, "msgr_running_total_time": 0.06697084, "msgr_send_bytes": 275572, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 525, "msgr_send_messages_queue_lat": { "avgcount": 524, "avgtime": 9.0472e-05, "sum": 0.047407637 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 86, "avgtime": 0.059585007, "sum": 5.124310664 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1597, "avgtime": 0.000411124, "sum": 0.656565747 }, "queue_len": 0 }, "mds": { "caps": 2523, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5473, "ceph_cap_op_revoke": 666, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 3204, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4522, "inodes_bottom": 4522, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4522, "inodes_top": 0, "inodes_with_caps": 2523, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 7109, "q": 0, "reply": 5163, "reply_latency": { "avgcount": 5163, "avgtime": 0.001416214, "sum": 7.311916174 }, "request": 5166, "root_rbytes": 12989440, "root_rfiles": 2167, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5172, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5171, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 921, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1954, "strays_enqueued": 1033, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9406, "evadd": 9403, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.021519755, "sum": 0.344316092 }, "rdpos": 4195244, "replayed": 3, "seg": 14, "segadd": 12, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23557270 }, "mds_mem": { "cap": 2169, "cap+": 3202, "cap-": 1033, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4668, "dn+": 5164, "dn-": 496, "heap": 248164, "ino": 2718, "ino+": 3214, "ino-": 496, "rss": 93344 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5169, "dispatch_server_request": 0, "handle_client_request": 5166, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1953, "avgtime": 0.001984595, "sum": 3.875915442 } }, "mds_sessions": { "average_load": 4791, "avg_session_uptime": 781, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4791 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 15530203, "buffer_anon_items": 4204, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13160482, "mds_co_items": 89689, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 25, "op": 1140, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1140, "avgtime": 0.001785046, "sum": 2.034953211 }, "op_pg": 0, "op_r": 46, "op_reply": 1140, "op_resend": 0, "op_rmw": 0, "op_send": 1140, "op_send_bytes": 17273894, "op_w": 1094, "oplen_avg": { "avgcount": 1140, "sum": 1205 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1048, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 18, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 26, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1033, "pq_executed_ops": 1033, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9554, "get_or_fail_fail": 0, "get_or_fail_success": 9554, "get_started": 0, "get_sum": 2666540, "max": 104857600, "put": 9554, "put_sum": 2666540, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1140, "put_sum": 17379344, "take": 1140, "take_sum": 17379344, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1140, "put_sum": 1140, "take": 1140, "take_sum": 1140, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1033, "get_or_fail_fail": 0, "get_or_fail_success": 1033, "get_started": 0, "get_sum": 104333, "max": 3758096384, "put": 10, "put_sum": 104333, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 9404, "get_or_fail_fail": 0, "get_or_fail_success": 9404, "get_started": 0, "get_sum": 19364769, "max": 3758096384, "put": 16, "put_sum": 16775835, "take": 0, "take_sum": 0, "val": 2588934, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:47.310 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/921, 1033/1954 2024-03-06T07:26:48.312 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:48.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.441+0000 7fec2b850640 1 Processor -- start 2024-03-06T07:26:48.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.442+0000 7fec2b850640 1 -- start start 2024-03-06T07:26:48.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec2b850640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 0x7fec240cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec2b850640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 0x7fec240d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec2b850640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec240d1ea0 con 0x7fec240d0e90 2024-03-06T07:26:48.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec2b850640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec240d2ab0 con 0x7fec240cfba0 2024-03-06T07:26:48.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec2b850640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec240d2180 con 0x7fec2415e150 2024-03-06T07:26:48.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec295ec640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 0x7fec240cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec28deb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 0x7fec240d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec28deb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 0x7fec240d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:36324/0 (socket says 172.21.15.122:36324) 2024-03-06T07:26:48.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec295ec640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 0x7fec240cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:60638/0 (socket says 172.21.15.122:60638) 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec28deb640 1 -- 172.21.15.122:0/3780655418 learned_addr learned my addr 172.21.15.122:0/3780655418 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec295ec640 1 -- 172.21.15.122:0/3780655418 >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 msgr2=0x7fec240d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec295ec640 1 --2- 172.21.15.122:0/3780655418 >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 0x7fec240d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.443+0000 7fec295ec640 1 -- 172.21.15.122:0/3780655418 >> v1:172.21.15.169:6789/0 conn(0x7fec240d0e90 legacy=0x7fec240b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.444+0000 7fec1bfff640 1 -- 172.21.15.122:0/3780655418 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452821245 0 0) 0x7fec240d1ea0 con 0x7fec240d0e90 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.444+0000 7fec295ec640 1 -- 172.21.15.122:0/3780655418 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240d2460 con 0x7fec2415e150 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.444+0000 7fec28deb640 1 --2- 172.21.15.122:0/3780655418 >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 0x7fec240d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.444+0000 7fec295ec640 1 --2- 172.21.15.122:0/3780655418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 0x7fec240cf660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fec200097a0 tx=0x7fec2002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=976977f2bce06c19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.444+0000 7fec1bfff640 1 -- 172.21.15.122:0/3780655418 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec200315a0 con 0x7fec2415e150 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.444+0000 7fec1bfff640 1 -- 172.21.15.122:0/3780655418 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20037070 con 0x7fec2415e150 2024-03-06T07:26:48.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec1bfff640 1 -- 172.21.15.122:0/3780655418 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20031960 con 0x7fec2415e150 2024-03-06T07:26:48.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 -- 172.21.15.122:0/3780655418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 msgr2=0x7fec240cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 --2- 172.21.15.122:0/3780655418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 0x7fec240cf660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fec200097a0 tx=0x7fec2002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:48.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 -- 172.21.15.122:0/3780655418 shutdown_connections 2024-03-06T07:26:48.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 --2- 172.21.15.122:0/3780655418 >> v2:172.21.15.169:3300/0 conn(0x7fec240cfba0 0x7fec240d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 --2- 172.21.15.122:0/3780655418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec2415e150 0x7fec240cf660 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 -- 172.21.15.122:0/3780655418 >> 172.21.15.122:0/3780655418 conn(0x7fec2409d230 msgr2=0x7fec240d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 -- 172.21.15.122:0/3780655418 shutdown_connections 2024-03-06T07:26:48.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.445+0000 7fec2b850640 1 -- 172.21.15.122:0/3780655418 wait complete. 2024-03-06T07:26:48.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.447+0000 7fec2b850640 1 Processor -- start 2024-03-06T07:26:48.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.447+0000 7fec2b850640 1 -- start start 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec2b850640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec240cfba0 0x7fec240c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec2b850640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 0x7fec2415be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec2b850640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec2415e150 0x7fec241f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec2b850640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec240d1b30 con 0x7fec240d0e90 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec2b850640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec240d2f40 con 0x7fec2415e150 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec2b850640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2415e530 con 0x7fec240cfba0 2024-03-06T07:26:48.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 0x7fec2415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 0x7fec2415be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52050/0 (socket says 172.21.15.122:52050) 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 -- 172.21.15.122:0/2488985426 learned_addr learned my addr 172.21.15.122:0/2488985426 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec29ded640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec2415e150 0x7fec241f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec295ec640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec240cfba0 0x7fec240c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 -- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec240cfba0 msgr2=0x7fec240c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec240cfba0 0x7fec240c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 -- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec2415e150 msgr2=0x7fec241f83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec2415e150 0x7fec241f83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.448+0000 7fec28deb640 1 -- 172.21.15.122:0/2488985426 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240c91b0 con 0x7fec240d0e90 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.449+0000 7fec28deb640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 0x7fec2415be90 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e9d0 tx=0x7fec1400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.449+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14017070 con 0x7fec240d0e90 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.449+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1400ca60 con 0x7fec240d0e90 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.449+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1400ce10 con 0x7fec240d0e90 2024-03-06T07:26:48.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.449+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec241f8e80 con 0x7fec240d0e90 2024-03-06T07:26:48.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.449+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec240b7060 con 0x7fec240d0e90 2024-03-06T07:26:48.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.450+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec1401a070 con 0x7fec240d0e90 2024-03-06T07:26:48.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.451+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec24069ee0 con 0x7fec240d0e90 2024-03-06T07:26:48.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.451+0000 7fec19ffb640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec0806b4d0 0x7fec0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.451+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fec1408a110 con 0x7fec240d0e90 2024-03-06T07:26:48.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.451+0000 7fec295ec640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec0806b4d0 0x7fec0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.452+0000 7fec295ec640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec0806b4d0 0x7fec0806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fec20036000 tx=0x7fec20008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.458+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec140588e0 con 0x7fec240d0e90 2024-03-06T07:26:48.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.761+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec240d42f0 con 0x7fec240d0e90 2024-03-06T07:26:48.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.762+0000 7fec19ffb640 1 -- 172.21.15.122:0/2488985426 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fec1405c980 con 0x7fec240d0e90 2024-03-06T07:26:48.764 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:48.764 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:48.764 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.764+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec0806b4d0 msgr2=0x7fec0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec0806b4d0 0x7fec0806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fec20036000 tx=0x7fec20008030 comp rx=0 tx=0).stop 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 msgr2=0x7fec2415be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 0x7fec2415be90 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e9d0 tx=0x7fec1400ee90 comp rx=0 tx=0).stop 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 shutdown_connections 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fec0806b4d0 0x7fec0806d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec2415e150 0x7fec241f83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec240d0e90 0x7fec2415be90 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 --2- 172.21.15.122:0/2488985426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fec240cfba0 0x7fec240c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 >> 172.21.15.122:0/2488985426 conn(0x7fec2409d230 msgr2=0x7fec24161240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 shutdown_connections 2024-03-06T07:26:48.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:48.765+0000 7fec2b850640 1 -- 172.21.15.122:0/2488985426 wait complete. 2024-03-06T07:26:48.779 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:48.921 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":384,"msgr_send_messages":380,"msgr_recv_bytes":201939,"msgr_send_bytes":4342062,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.059590970,"msgr_running_send_time":0.028686790,"msgr_running_recv_time":0.016971171,"msgr_running_fast_dispatch_time":0.012770597,"msgr_send_messages_queue_lat":{"avgcount":379,"sum":0.029924467,"avgtime":0.000078956},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":107120,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9801,"msgr_send_messages":21224,"msgr_recv_bytes":3512275,"msgr_send_bytes":26834329,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.966475739,"msgr_running_send_time":0.538103899,"msgr_running_recv_time":0.482140762,"msgr_running_fast_dispatch_time":0.020116845,"msgr_send_messages_queue_lat":{"avgcount":21223,"sum":0.371216118,"avgtime":0.000017491},"msgr_handle_ack_lat":{"avgcount":28040,"sum":0.004678352,"avgtime":0.000000166},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":693,"msgr_send_messages":694,"msgr_recv_bytes":320451,"msgr_send_bytes":1023080,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.087639732,"msgr_running_send_time":0.047326922,"msgr_running_recv_time":0.017467105,"msgr_running_fast_dispatch_time":0.018391607,"msgr_send_messages_queue_lat":{"avgcount":693,"sum":0.056904303,"avgtime":0.000082112},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1833,"complete_latency":{"avgcount":89,"sum":5.124832176,"avgtime":0.057582384}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2078,"sum":0.893477444,"avgtime":0.000429969}},"mds":{"request":6140,"reply":6137,"reply_latency":{"avgcount":6137,"sum":8.435031453,"avgtime":0.001374455},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5277,"inodes_top":0,"inodes_bottom":5277,"inodes_pin_tail":0,"inodes_pinned":5277,"inodes_expired":0,"inodes_with_caps":1840,"caps":1840,"subtrees":2,"traverse":6146,"traverse_hit":6145,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1826,"root_rbytes":13126656,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":666,"ceph_cap_op_grant":5960,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3204,"handle_client_caps_dirty":3000,"handle_client_cap_release":5,"process_request_cap_release":9057},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1565,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":2927,"strays_enqueued":1362,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10708,"evex":0,"evtrm":0,"ev":10711,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26863527,"rdpos":4195244,"jlat":{"avgcount":17,"sum":0.367504067,"avgtime":0.021617886},"replayed":3},"mds_mem":{"ino":2346,"ino+":3214,"ino-":868,"dir":13,"dir+":13,"dir-":0,"dn":5270,"dn+":6138,"dn-":868,"cap":1840,"cap+":3202,"cap-":1362,"rss":117152,"heap":248164},"mds_server":{"dispatch_client_request":6143,"dispatch_server_request":0,"handle_client_request":6140,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2927,"sum":4.999030721,"avgtime":0.001707902},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5667,"average_load":5667,"avg_session_uptime":786,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":6145030,"buffer_anon_items":2696,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14081045,"mds_co_items":147701,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1482,"op_send_bytes":22162252,"op_resend":0,"op_reply":1482,"op_latency":{"avgcount":1482,"sum":2.668938629,"avgtime":0.001800903},"op_inflight":0,"oplen_avg":{"avgcount":1482,"sum":1548},"op":1482,"op_r":49,"op_w":1433,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":21,"osdop_write":32,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1378,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":26,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1362,"pq_executed":1362,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":10873,"get_sum":3053257,"get_or_fail_fail":0,"get_or_fail_success":10873,"take":0,"take_sum":0,"put":10872,"put_sum":3052960,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1482,"take_sum":22300931,"put":1482,"put_sum":22300931,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1482,"take_sum":1482,"put":1482,"put_sum":1482,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1362,"get_sum":137562,"get_or_fail_fail":0,"get_or_fail_success":1362,"take":0,"take_sum":0,"put":13,"put_sum":137562,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":1038536,"max":3758096384,"get_started":0,"get":10708,"get_sum":22667842,"get_or_fail_fail":0,"get_or_fail_success":10708,"take":0,"take_sum":0,"put":19,"put_sum":21629306,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:48.922 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 201939, "msgr_recv_encrypted_bytes": 107120, "msgr_recv_messages": 384, "msgr_running_fast_dispatch_time": 0.012770597, "msgr_running_recv_time": 0.016971171, "msgr_running_send_time": 0.02868679, "msgr_running_total_time": 0.05959097, "msgr_send_bytes": 4342062, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 380, "msgr_send_messages_queue_lat": { "avgcount": 379, "avgtime": 7.8956e-05, "sum": 0.029924467 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 28040, "avgtime": 1.66e-07, "sum": 0.004678352 }, "msgr_recv_bytes": 3512275, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 9801, "msgr_running_fast_dispatch_time": 0.020116845, "msgr_running_recv_time": 0.482140762, "msgr_running_send_time": 0.538103899, "msgr_running_total_time": 0.966475739, "msgr_send_bytes": 26834329, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 21224, "msgr_send_messages_queue_lat": { "avgcount": 21223, "avgtime": 1.7491e-05, "sum": 0.371216118 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 320451, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 693, "msgr_running_fast_dispatch_time": 0.018391607, "msgr_running_recv_time": 0.017467105, "msgr_running_send_time": 0.047326922, "msgr_running_total_time": 0.087639732, "msgr_send_bytes": 1023080, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 694, "msgr_send_messages_queue_lat": { "avgcount": 693, "avgtime": 8.2112e-05, "sum": 0.056904303 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 89, "avgtime": 0.057582384, "sum": 5.124832176 }, "queue_len": 1833 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2078, "avgtime": 0.000429969, "sum": 0.893477444 }, "queue_len": 0 }, "mds": { "caps": 1840, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5960, "ceph_cap_op_revoke": 666, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 3204, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5277, "inodes_bottom": 5277, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5277, "inodes_top": 0, "inodes_with_caps": 1840, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9057, "q": 0, "reply": 6137, "reply_latency": { "avgcount": 6137, "avgtime": 0.001374455, "sum": 8.435031453 }, "request": 6140, "root_rbytes": 13126656, "root_rfiles": 1826, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6146, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6145, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1565, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2927, "strays_enqueued": 1362, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10711, "evadd": 10708, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.021617886, "sum": 0.367504067 }, "rdpos": 4195244, "replayed": 3, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26863527 }, "mds_mem": { "cap": 1840, "cap+": 3202, "cap-": 1362, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5270, "dn+": 6138, "dn-": 868, "heap": 248164, "ino": 2346, "ino+": 3214, "ino-": 868, "rss": 117152 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6143, "dispatch_server_request": 0, "handle_client_request": 6140, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2927, "avgtime": 0.001707902, "sum": 4.999030721 } }, "mds_sessions": { "average_load": 5667, "avg_session_uptime": 786, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5667 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 6145030, "buffer_anon_items": 2696, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14081045, "mds_co_items": 147701, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 26, "op": 1482, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1482, "avgtime": 0.001800903, "sum": 2.668938629 }, "op_pg": 0, "op_r": 49, "op_reply": 1482, "op_resend": 0, "op_rmw": 0, "op_send": 1482, "op_send_bytes": 22162252, "op_w": 1433, "oplen_avg": { "avgcount": 1482, "sum": 1548 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1378, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 21, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 32, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1362, "pq_executed_ops": 1362, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10873, "get_or_fail_fail": 0, "get_or_fail_success": 10873, "get_started": 0, "get_sum": 3053257, "max": 104857600, "put": 10872, "put_sum": 3052960, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1482, "put_sum": 22300931, "take": 1482, "take_sum": 22300931, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1482, "put_sum": 1482, "take": 1482, "take_sum": 1482, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1362, "get_or_fail_fail": 0, "get_or_fail_success": 1362, "get_started": 0, "get_sum": 137562, "max": 3758096384, "put": 13, "put_sum": 137562, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 10708, "get_or_fail_fail": 0, "get_or_fail_success": 10708, "get_started": 0, "get_sum": 22667842, "max": 3758096384, "put": 19, "put_sum": 21629306, "take": 0, "take_sum": 0, "val": 1038536, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:48.922 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1565, 1362/2927 2024-03-06T07:26:49.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.923 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:50.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.001+0000 7fd8f78df640 1 Processor -- start 2024-03-06T07:26:50.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.001+0000 7fd8f78df640 1 -- start start 2024-03-06T07:26:50.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f78df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 0x7fd8f00cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f78df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00b5300 con 0x7fd8f0162940 2024-03-06T07:26:50.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f78df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00ccbd0 con 0x7fd8f00ce5a0 2024-03-06T07:26:50.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f78df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00b3b00 con 0x7fd8f00ce980 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f5e7c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd8f0162940 0x7fd8f0164d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59018/0 (socket says 172.21.15.122:59018) 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f5e7c640 1 -- 172.21.15.122:0/388289410 learned_addr learned my addr 172.21.15.122:0/388289410 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8f4e7a640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 0x7fd8f00cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276883077 0 0) 0x7fd8f00b5300 con 0x7fd8f0162940 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.002+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc003520 con 0x7fd8f0162940 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 271837021 0 0) 0x7fd8dc003520 con 0x7fd8f0162940 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 msgr2=0x7fd8f00cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 0x7fd8f00cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 >> v1:172.21.15.169:6789/0 conn(0x7fd8f00ce5a0 legacy=0x7fd8f00b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00b3de0 con 0x7fd8f0162940 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fd8ec001670 con 0x7fd8f0162940 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 >> v1:172.21.15.122:6789/0 conn(0x7fd8f0162940 legacy=0x7fd8f0164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 0x7fd8dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc003f20 0x7fd8dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc004d40 0x7fd8dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00b3b00 con 0x7fd8dc004d40 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc00ba20 con 0x7fd8dc003f20 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc00bc20 con 0x7fd8f00ce980 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f567b640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 0x7fd8dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f5e7c640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc004d40 0x7fd8dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f4e7a640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc003f20 0x7fd8dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f5e7c640 1 -- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 msgr2=0x7fd8dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f5e7c640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f00ce980 0x7fd8dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f5e7c640 1 -- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc003f20 msgr2=0x7fd8dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f5e7c640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8dc003f20 0x7fd8dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.003+0000 7fd8f5e7c640 1 -- 172.21.15.122:0/388289410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc003520 con 0x7fd8dc004d40 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.004+0000 7fd8f5e7c640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc004d40 0x7fd8dc0051c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c210 tx=0x7fd8dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768effb37230863e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.004+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec007d10 con 0x7fd8dc004d40 2024-03-06T07:26:50.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.004+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec030070 con 0x7fd8dc004d40 2024-03-06T07:26:50.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.004+0000 7fd8e7fff640 1 -- 172.21.15.122:0/388289410 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec005d90 con 0x7fd8dc004d40 2024-03-06T07:26:50.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f78df640 1 -- 172.21.15.122:0/388289410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc004d40 msgr2=0x7fd8dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f78df640 1 --2- 172.21.15.122:0/388289410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8dc004d40 0x7fd8dc0051c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c210 tx=0x7fd8dc005830 comp rx=0 tx=0).stop 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f5e7c640 1 -- 172.21.15.122:0/388289410 reap_dead start 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f78df640 1 -- 172.21.15.122:0/388289410 shutdown_connections 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f78df640 1 -- 172.21.15.122:0/388289410 >> 172.21.15.122:0/388289410 conn(0x7fd8f006b160 msgr2=0x7fd8f006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f78df640 1 -- 172.21.15.122:0/388289410 shutdown_connections 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.005+0000 7fd8f78df640 1 -- 172.21.15.122:0/388289410 wait complete. 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.007+0000 7fd8f78df640 1 Processor -- start 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.007+0000 7fd8f78df640 1 -- start start 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f78df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8f0159c10 0x7fd8f01efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f78df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 0x7fd8f0140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f78df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f01f0230 0x7fd8f0140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f78df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00cc9b0 con 0x7fd8f013c120 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f78df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00cc830 con 0x7fd8f0159c10 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f78df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00cceb0 con 0x7fd8f01f0230 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f4e7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 0x7fd8f0140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f4e7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 0x7fd8f0140990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52072/0 (socket says 172.21.15.122:52072) 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f4e7a640 1 -- 172.21.15.122:0/2082936196 learned_addr learned my addr 172.21.15.122:0/2082936196 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f567b640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8f0159c10 0x7fd8f01efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.008+0000 7fd8f5e7c640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f01f0230 0x7fd8f0140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f4e7a640 1 -- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f01f0230 msgr2=0x7fd8f0140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f4e7a640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f01f0230 0x7fd8f0140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f4e7a640 1 -- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8f0159c10 msgr2=0x7fd8f01efcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f4e7a640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8f0159c10 0x7fd8f01efcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f4e7a640 1 -- 172.21.15.122:0/2082936196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f013c5b0 con 0x7fd8f013c120 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f4e7a640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 0x7fd8f0140990 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00119a0 tx=0x7fd8e0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000fa70 con 0x7fd8f013c120 2024-03-06T07:26:50.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000fc10 con 0x7fd8f013c120 2024-03-06T07:26:50.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f0133840 con 0x7fd8f013c120 2024-03-06T07:26:50.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0014440 con 0x7fd8f013c120 2024-03-06T07:26:50.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.009+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f0133dc0 con 0x7fd8f013c120 2024-03-06T07:26:50.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.010+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f0068050 con 0x7fd8f013c120 2024-03-06T07:26:50.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.011+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8e0012040 con 0x7fd8f013c120 2024-03-06T07:26:50.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.011+0000 7fd8e5ffb640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8d406b470 0x7fd8d406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.011+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd8e008a750 con 0x7fd8f013c120 2024-03-06T07:26:50.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.011+0000 7fd8f567b640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8d406b470 0x7fd8d406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.012+0000 7fd8f567b640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8d406b470 0x7fd8d406d930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8002010 tx=0x7fd8e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.016+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8f0068050 con 0x7fd8f013c120 2024-03-06T07:26:50.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.079+0000 7fe171f8c640 1 Processor -- start 2024-03-06T07:26:50.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.079+0000 7fe171f8c640 1 -- start start 2024-03-06T07:26:50.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.081+0000 7fe171f8c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d0690 0x7fe16c06af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.081+0000 7fe171f8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c071d30 con 0x7fe16c06b4c0 2024-03-06T07:26:50.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.081+0000 7fe171f8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c071fa0 con 0x7fe16c163240 2024-03-06T07:26:50.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.081+0000 7fe171f8c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c0725f0 con 0x7fe16c0d0690 2024-03-06T07:26:50.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.081+0000 7fe16bfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe16c163240 0x7fe16c06da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:42770/0 (socket says 172.21.15.122:42770) 2024-03-06T07:26:50.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16bfff640 1 -- 172.21.15.122:0/2215957022 learned_addr learned my addr 172.21.15.122:0/2215957022 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:50.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16b7fe640 1 --2- 172.21.15.122:0/2215957022 >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d0690 0x7fe16c06af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48495333 0 0) 0x7fe16c071fa0 con 0x7fe16c163240 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe158003520 con 0x7fe16c163240 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261180830 0 0) 0x7fe16c071d30 con 0x7fe16c06b4c0 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe158003880 con 0x7fe16c06b4c0 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16b7fe640 1 -- 172.21.15.122:0/2215957022 >> v1:172.21.15.169:6789/0 conn(0x7fe16c163240 legacy=0x7fe16c06da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16b7fe640 1 -- 172.21.15.122:0/2215957022 >> v1:172.21.15.122:6789/0 conn(0x7fe16c06b4c0 legacy=0x7fe16c162b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.082+0000 7fe16b7fe640 1 -- 172.21.15.122:0/2215957022 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c072280 con 0x7fe16c0d0690 2024-03-06T07:26:50.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.083+0000 7fe16b7fe640 1 --2- 172.21.15.122:0/2215957022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0d0690 0x7fe16c06af80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0097a0 tx=0x7fe15c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a1a096e9ead98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.084+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c034070 con 0x7fe16c0d0690 2024-03-06T07:26:50.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.084+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c030610 con 0x7fe16c0d0690 2024-03-06T07:26:50.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.084+0000 7fe16affd640 1 -- 172.21.15.122:0/2215957022 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c0309e0 con 0x7fe16c0d0690 2024-03-06T07:26:50.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.085+0000 7fe171f8c640 1 -- 172.21.15.122:0/2215957022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0d0690 msgr2=0x7fe16c06af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.085+0000 7fe171f8c640 1 --2- 172.21.15.122:0/2215957022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0d0690 0x7fe16c06af80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0097a0 tx=0x7fe15c02b960 comp rx=0 tx=0).stop 2024-03-06T07:26:50.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.086+0000 7fe171f8c640 1 -- 172.21.15.122:0/2215957022 shutdown_connections 2024-03-06T07:26:50.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.086+0000 7fe171f8c640 1 --2- 172.21.15.122:0/2215957022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0d0690 0x7fe16c06af80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.086+0000 7fe171f8c640 1 -- 172.21.15.122:0/2215957022 >> 172.21.15.122:0/2215957022 conn(0x7fe16c0b2890 msgr2=0x7fe16c0b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.086+0000 7fe171f8c640 1 -- 172.21.15.122:0/2215957022 shutdown_connections 2024-03-06T07:26:50.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.086+0000 7fe171f8c640 1 -- 172.21.15.122:0/2215957022 wait complete. 2024-03-06T07:26:50.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.087+0000 7fe171f8c640 1 Processor -- start 2024-03-06T07:26:50.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.087+0000 7fe171f8c640 1 -- start start 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.088+0000 7fe171f8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c06b4c0 0x7fe16c15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.088+0000 7fe171f8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 0x7fe16c149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.088+0000 7fe171f8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c163240 0x7fe16c0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe171f8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c072c00 con 0x7fe16c0d0690 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe171f8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c072a80 con 0x7fe16c163240 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe171f8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c06e120 con 0x7fe16c06b4c0 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 0x7fe16c149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 0x7fe16c149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52078/0 (socket says 172.21.15.122:52078) 2024-03-06T07:26:50.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 -- 172.21.15.122:0/390471 learned_addr learned my addr 172.21.15.122:0/390471 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe16b7fe640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c06b4c0 0x7fe16c15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe16bfff640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c163240 0x7fe16c0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 -- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c06b4c0 msgr2=0x7fe16c15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c06b4c0 0x7fe16c15f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 -- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c163240 msgr2=0x7fe16c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c163240 0x7fe16c0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 -- 172.21.15.122:0/390471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c147170 con 0x7fe16c0d0690 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.089+0000 7fe163fff640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 0x7fe16c149470 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe15400c490 tx=0x7fe15400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.090+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154017070 con 0x7fe16c0d0690 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.090+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1540035f0 con 0x7fe16c0d0690 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.090+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c147450 con 0x7fe16c0d0690 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.090+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c147a10 con 0x7fe16c0d0690 2024-03-06T07:26:50.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.090+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154003a60 con 0x7fe16c0d0690 2024-03-06T07:26:50.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.092+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe150005290 con 0x7fe16c0d0690 2024-03-06T07:26:50.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.093+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe15401a070 con 0x7fe16c0d0690 2024-03-06T07:26:50.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.094+0000 7fe168ff9640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe14c06b550 0x7fe14c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.094+0000 7fe16b7fe640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe14c06b550 0x7fe14c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.094+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe1540895f0 con 0x7fe16c0d0690 2024-03-06T07:26:50.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.095+0000 7fe16b7fe640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe14c06b550 0x7fe14c06da10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe15c036000 tx=0x7fe15c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.101+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe154057d10 con 0x7fe16c0d0690 2024-03-06T07:26:50.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.366+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f0072c90 con 0x7fd8f013c120 2024-03-06T07:26:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.367+0000 7fd8e5ffb640 1 -- 172.21.15.122:0/2082936196 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fd8f0072c90 con 0x7fd8f013c120 2024-03-06T07:26:50.369 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:50.369 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:50.369 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8d406b470 msgr2=0x7fd8d406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8d406b470 0x7fd8d406d930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8002010 tx=0x7fd8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 msgr2=0x7fd8f0140990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 0x7fd8f0140990 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00119a0 tx=0x7fd8e0011e60 comp rx=0 tx=0).stop 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 shutdown_connections 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8d406b470 0x7fd8d406d930 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8f01f0230 0x7fd8f0140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8f013c120 0x7fd8f0140990 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 --2- 172.21.15.122:0/2082936196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8f0159c10 0x7fd8f01efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 >> 172.21.15.122:0/2082936196 conn(0x7fd8f006b160 msgr2=0x7fd8f006d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 shutdown_connections 2024-03-06T07:26:50.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.370+0000 7fd8f78df640 1 -- 172.21.15.122:0/2082936196 wait complete. 2024-03-06T07:26:50.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:26:50.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.507+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe150005580 con 0x7fe16c0d0690 2024-03-06T07:26:50.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.507+0000 7fe168ff9640 1 -- 172.21.15.122:0/390471 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fe15405bdb0 con 0x7fe16c0d0690 2024-03-06T07:26:50.509 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:50.509 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:50.510 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:50.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe14c06b550 msgr2=0x7fe14c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe14c06b550 0x7fe14c06da10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe15c036000 tx=0x7fe15c03b040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 msgr2=0x7fe16c149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 0x7fe16c149470 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe15400c490 tx=0x7fe15400c950 comp rx=0 tx=0).stop 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 shutdown_connections 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe14c06b550 0x7fe14c06da10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c163240 0x7fe16c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0d0690 0x7fe16c149470 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 --2- 172.21.15.122:0/390471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c06b4c0 0x7fe16c15f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.509+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 >> 172.21.15.122:0/390471 conn(0x7fe16c0b2890 msgr2=0x7fe16c06e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.510+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 shutdown_connections 2024-03-06T07:26:50.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:50.510+0000 7fe171f8c640 1 -- 172.21.15.122:0/390471 wait complete. 2024-03-06T07:26:50.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:50.776 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:26:50.776 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:50.793 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":843,"msgr_send_messages":839,"msgr_recv_bytes":346853,"msgr_send_bytes":5262718,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.087625926,"msgr_running_send_time":0.041163170,"msgr_running_recv_time":0.020988913,"msgr_running_fast_dispatch_time":0.021660517,"msgr_send_messages_queue_lat":{"avgcount":838,"sum":0.487232324,"avgtime":0.000581422},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":107120,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10584,"msgr_send_messages":24767,"msgr_recv_bytes":3791157,"msgr_send_bytes":27902924,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.065345398,"msgr_running_send_time":0.591616121,"msgr_running_recv_time":0.509159203,"msgr_running_fast_dispatch_time":0.034743841,"msgr_send_messages_queue_lat":{"avgcount":24766,"sum":0.518112446,"avgtime":0.000020920},"msgr_handle_ack_lat":{"avgcount":29959,"sum":0.005038586,"avgtime":0.000000168},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1616,"msgr_send_messages":1617,"msgr_recv_bytes":660020,"msgr_send_bytes":2458247,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.149765584,"msgr_running_send_time":0.072915558,"msgr_running_recv_time":0.029570069,"msgr_running_fast_dispatch_time":0.037721354,"msgr_send_messages_queue_lat":{"avgcount":1616,"sum":0.505665299,"avgtime":0.000312911},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":115,"sum":7.227954059,"avgtime":0.062851774}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2900,"sum":0.961848102,"avgtime":0.000331671}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.280122065,"avgtime":0.001447531},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":5050,"inodes_top":0,"inodes_bottom":5050,"inodes_pin_tail":0,"inodes_pinned":5050,"inodes_expired":0,"inodes_with_caps":1330,"caps":1330,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":668,"ceph_cap_op_grant":7341,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":6,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1329,"num_strays_delayed":1329,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":1872,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11496,"evex":11496,"evtrm":11496,"ev":11498,"evexg":0,"evexd":11495,"segadd":15,"segex":16,"segtrm":16,"seg":1,"segmjr":1,"segexg":0,"segexd":15,"expos":28042820,"wrpos":28046501,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.162674279,"avgtime":0.137507577},"replayed":3},"mds_mem":{"ino":1852,"ino+":3214,"ino-":1362,"dir":13,"dir+":13,"dir-":0,"dn":5050,"dn+":6412,"dn-":1362,"cap":1330,"cap+":3202,"cap-":1872,"rss":117152,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.489485060,"avgtime":0.489485060},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.354636273,"avgtime":0.001673323},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":786,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":6145030,"buffer_anon_items":2696,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14081045,"mds_co_items":147701,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3369,"op_send_bytes":25374594,"op_resend":0,"op_reply":3369,"op_latency":{"avgcount":3369,"sum":104.059606565,"avgtime":0.030887386},"op_inflight":0,"oplen_avg":{"avgcount":3369,"sum":6136},"op":3369,"op_r":52,"op_w":3317,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1340,"osdop_read":24,"osdop_write":39,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1893,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2680,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":50,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1872,"pq_executed":1872,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13038,"get_sum":3621772,"get_or_fail_fail":0,"get_or_fail_success":13038,"take":0,"take_sum":0,"put":13038,"put_sum":3621772,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3369,"take_sum":25564783,"put":3369,"put_sum":25564783,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3369,"take_sum":3369,"put":3369,"put_sum":3369,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1872,"get_sum":189072,"get_or_fail_fail":0,"get_or_fail_success":1872,"take":0,"take_sum":0,"put":16,"put_sum":189072,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":0,"max":3758096384,"get_started":0,"get":11496,"get_sum":23850816,"get_or_fail_fail":0,"get_or_fail_success":11496,"take":0,"take_sum":0,"put":23,"put_sum":23850816,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:50.795 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 346853, "msgr_recv_encrypted_bytes": 107120, "msgr_recv_messages": 843, "msgr_running_fast_dispatch_time": 0.021660517, "msgr_running_recv_time": 0.020988913, "msgr_running_send_time": 0.04116317, "msgr_running_total_time": 0.087625926, "msgr_send_bytes": 5262718, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 839, "msgr_send_messages_queue_lat": { "avgcount": 838, "avgtime": 0.000581422, "sum": 0.487232324 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29959, "avgtime": 1.68e-07, "sum": 0.005038586 }, "msgr_recv_bytes": 3791157, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 10584, "msgr_running_fast_dispatch_time": 0.034743841, "msgr_running_recv_time": 0.509159203, "msgr_running_send_time": 0.591616121, "msgr_running_total_time": 1.065345398, "msgr_send_bytes": 27902924, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 24767, "msgr_send_messages_queue_lat": { "avgcount": 24766, "avgtime": 2.092e-05, "sum": 0.518112446 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 660020, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1616, "msgr_running_fast_dispatch_time": 0.037721354, "msgr_running_recv_time": 0.029570069, "msgr_running_send_time": 0.072915558, "msgr_running_total_time": 0.149765584, "msgr_send_bytes": 2458247, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1617, "msgr_send_messages_queue_lat": { "avgcount": 1616, "avgtime": 0.000312911, "sum": 0.505665299 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 115, "avgtime": 0.062851774, "sum": 7.227954059 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2900, "avgtime": 0.000331671, "sum": 0.961848102 }, "queue_len": 0 }, "mds": { "caps": 1330, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7341, "ceph_cap_op_revoke": 668, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 6, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5050, "inodes_bottom": 5050, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5050, "inodes_top": 0, "inodes_with_caps": 1330, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001447531, "sum": 9.280122065 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1329, "num_strays_delayed": 1329, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 1872, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11498, "evadd": 11496, "evex": 11496, "evexd": 11495, "evexg": 0, "evlrg": 0, "evtrm": 11496, "expos": 28042820, "jlat": { "avgcount": 23, "avgtime": 0.137507577, "sum": 3.162674279 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 15, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28046501 }, "mds_mem": { "cap": 1330, "cap+": 3202, "cap-": 1872, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5050, "dn+": 6412, "dn-": 1362, "heap": 248164, "ino": 1852, "ino+": 3214, "ino-": 1362, "rss": 117152 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.48948506, "sum": 0.48948506 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001673323, "sum": 5.354636273 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 786, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 6145030, "buffer_anon_items": 2696, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14081045, "mds_co_items": 147701, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 50, "op": 3369, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3369, "avgtime": 0.030887386, "sum": 104.059606565 }, "op_pg": 0, "op_r": 52, "op_reply": 3369, "op_resend": 0, "op_rmw": 0, "op_send": 3369, "op_send_bytes": 25374594, "op_w": 3317, "oplen_avg": { "avgcount": 3369, "sum": 6136 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1340, "osdop_delete": 1893, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2680, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 39, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1872, "pq_executed_ops": 1872, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13038, "get_or_fail_fail": 0, "get_or_fail_success": 13038, "get_started": 0, "get_sum": 3621772, "max": 104857600, "put": 13038, "put_sum": 3621772, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3369, "put_sum": 25564783, "take": 3369, "take_sum": 25564783, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3369, "put_sum": 3369, "take": 3369, "take_sum": 3369, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1872, "get_or_fail_fail": 0, "get_or_fail_success": 1872, "get_started": 0, "get_sum": 189072, "max": 3758096384, "put": 16, "put_sum": 189072, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 11496, "get_or_fail_fail": 0, "get_or_fail_success": 11496, "get_started": 0, "get_sum": 23850816, "max": 3758096384, "put": 23, "put_sum": 23850816, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:50.795 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1329, 1872/3201 2024-03-06T07:26:51.796 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:51.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.937+0000 7f51ae2ed640 1 Processor -- start 2024-03-06T07:26:51.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.937+0000 7f51ae2ed640 1 -- start start 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ae2ed640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f51a81625f0 0x7f51a8166200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ae2ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a8166740 0x7f51a80c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ae2ed640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f51a806c670 0x7f51a80c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ae2ed640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a8166be0 con 0x7f51a806c670 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ae2ed640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a8166ec0 con 0x7f51a81625f0 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ae2ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80cc490 con 0x7f51a8166740 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ac88a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f51a806c670 0x7f51a80c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ac88a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f51a806c670 0x7f51a80c51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52084/0 (socket says 172.21.15.122:52084) 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.938+0000 7f51ac88a640 1 -- 172.21.15.122:0/1644978512 learned_addr learned my addr 172.21.15.122:0/1644978512 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51a77fe640 1 --2- 172.21.15.122:0/1644978512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a8166740 0x7f51a80c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51a7fff640 1 --2- 172.21.15.122:0/1644978512 >> v2:172.21.15.169:3300/0 conn(0x7f51a81625f0 0x7f51a8166200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51ac88a640 1 -- 172.21.15.122:0/1644978512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a8166740 msgr2=0x7f51a80c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51ac88a640 1 --2- 172.21.15.122:0/1644978512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a8166740 0x7f51a80c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51ac88a640 1 -- 172.21.15.122:0/1644978512 >> v2:172.21.15.169:3300/0 conn(0x7f51a81625f0 msgr2=0x7f51a8166200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51ac88a640 1 --2- 172.21.15.122:0/1644978512 >> v2:172.21.15.169:3300/0 conn(0x7f51a81625f0 0x7f51a8166200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.939+0000 7f51ac88a640 1 -- 172.21.15.122:0/1644978512 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a80cc770 con 0x7f51a806c670 2024-03-06T07:26:51.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ac88a640 1 --2- 172.21.15.122:0/1644978512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a806c670 0x7f51a80c51b0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f51a8067ce0 tx=0x7f51a002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadeb68dbbcee00b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51a6ffd640 1 -- 172.21.15.122:0/1644978512 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a002cb00 con 0x7f51a806c670 2024-03-06T07:26:51.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51a6ffd640 1 -- 172.21.15.122:0/1644978512 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a002cca0 con 0x7f51a806c670 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51a6ffd640 1 -- 172.21.15.122:0/1644978512 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a0034610 con 0x7f51a806c670 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 -- 172.21.15.122:0/1644978512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a806c670 msgr2=0x7f51a80c51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/1644978512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a806c670 0x7f51a80c51b0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f51a8067ce0 tx=0x7f51a002bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 -- 172.21.15.122:0/1644978512 shutdown_connections 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/1644978512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a806c670 0x7f51a80c51b0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/1644978512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a8166740 0x7f51a80c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/1644978512 >> v2:172.21.15.169:3300/0 conn(0x7f51a81625f0 0x7f51a8166200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 -- 172.21.15.122:0/1644978512 >> 172.21.15.122:0/1644978512 conn(0x7f51a809aa40 msgr2=0x7f51a809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.940+0000 7f51ae2ed640 1 -- 172.21.15.122:0/1644978512 shutdown_connections 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.941+0000 7f51ae2ed640 1 -- 172.21.15.122:0/1644978512 wait complete. 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.942+0000 7f51ae2ed640 1 Processor -- start 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.942+0000 7f51ae2ed640 1 -- start start 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ae2ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a806c670 0x7f51a80c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ae2ed640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51a81625f0 0x7f51a814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ae2ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 0x7f51a814cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ae2ed640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80d1d50 con 0x7f51a8166740 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ae2ed640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80d1bd0 con 0x7f51a81625f0 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ae2ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80d1ed0 con 0x7f51a806c670 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ac88a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 0x7f51a814cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.943+0000 7f51ac88a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 0x7f51a814cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52096/0 (socket says 172.21.15.122:52096) 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 -- 172.21.15.122:0/16349039 learned_addr learned my addr 172.21.15.122:0/16349039 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51a7fff640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a806c670 0x7f51a80c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51a77fe640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51a81625f0 0x7f51a814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 -- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a806c670 msgr2=0x7f51a80c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a806c670 0x7f51a80c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 -- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51a81625f0 msgr2=0x7f51a814c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51a81625f0 0x7f51a814c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 -- 172.21.15.122:0/16349039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a80b65f0 con 0x7f51a8166740 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ac88a640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 0x7f51a814cd40 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f51a002c570 tx=0x7f51a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a0043020 con 0x7f51a8166740 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a002ce50 con 0x7f51a8166740 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a80b68d0 con 0x7f51a8166740 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a0009510 con 0x7f51a8166740 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.944+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a80b6e90 con 0x7f51a8166740 2024-03-06T07:26:51.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.945+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a8068050 con 0x7f51a8166740 2024-03-06T07:26:51.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.946+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51a0046020 con 0x7f51a8166740 2024-03-06T07:26:51.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.946+0000 7f51a4ff9640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f519406b500 0x7f519406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.946+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f51a00ac8e0 con 0x7f51a8166740 2024-03-06T07:26:51.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.946+0000 7f51a7fff640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f519406b500 0x7f519406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.946+0000 7f51a7fff640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f519406b500 0x7f519406d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f519800aaf0 tx=0x7f5198008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:51.955+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51a007b030 con 0x7f51a8166740 2024-03-06T07:26:52.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.307+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51a80cec10 con 0x7f51a8166740 2024-03-06T07:26:52.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.308+0000 7f51a4ff9640 1 -- 172.21.15.122:0/16349039 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f51a007f0d0 con 0x7f51a8166740 2024-03-06T07:26:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:52.310 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:52.310 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f519406b500 msgr2=0x7f519406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f519406b500 0x7f519406d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f519800aaf0 tx=0x7f5198008040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 msgr2=0x7f51a814cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 0x7f51a814cd40 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f51a002c570 tx=0x7f51a0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:52.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 shutdown_connections 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f519406b500 0x7f519406d9c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51a8166740 0x7f51a814cd40 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51a81625f0 0x7f51a814c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 --2- 172.21.15.122:0/16349039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f51a806c670 0x7f51a80c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 >> 172.21.15.122:0/16349039 conn(0x7f51a809aa40 msgr2=0x7f51a80d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.311+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 shutdown_connections 2024-03-06T07:26:52.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.312+0000 7f51ae2ed640 1 -- 172.21.15.122:0/16349039 wait complete. 2024-03-06T07:26:52.329 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:52.502 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1176,"msgr_send_messages":1172,"msgr_recv_bytes":431453,"msgr_send_bytes":5371574,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.113220508,"msgr_running_send_time":0.051569146,"msgr_running_recv_time":0.026729881,"msgr_running_fast_dispatch_time":0.028677735,"msgr_send_messages_queue_lat":{"avgcount":1171,"sum":0.501393184,"avgtime":0.000428175},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":107392,"msgr_send_encrypted_bytes":8544},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10951,"msgr_send_messages":25130,"msgr_recv_bytes":3916325,"msgr_send_bytes":28021356,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.095143717,"msgr_running_send_time":0.603180492,"msgr_running_recv_time":0.515438042,"msgr_running_fast_dispatch_time":0.043500760,"msgr_send_messages_queue_lat":{"avgcount":25129,"sum":0.532941557,"avgtime":0.000021208},"msgr_handle_ack_lat":{"avgcount":29963,"sum":0.005039003,"avgtime":0.000000168},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2258,"msgr_send_messages":2259,"msgr_recv_bytes":957261,"msgr_send_bytes":2801802,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.196961906,"msgr_running_send_time":0.092094010,"msgr_running_recv_time":0.040467040,"msgr_running_fast_dispatch_time":0.051816679,"msgr_send_messages_queue_lat":{"avgcount":2258,"sum":0.527903853,"avgtime":0.000233792},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":116,"sum":7.227965045,"avgtime":0.062310043}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5163,"sum":1.092110670,"avgtime":0.000211526}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.280122065,"avgtime":0.001447531},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":5050,"inodes_top":0,"inodes_bottom":5050,"inodes_pin_tail":0,"inodes_pinned":5050,"inodes_expired":0,"inodes_with_caps":1330,"caps":1330,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":668,"ceph_cap_op_grant":7341,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12826,"evex":11496,"evtrm":11496,"ev":1333,"evexg":0,"evexd":11495,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28042820,"wrpos":28844318,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.162674279,"avgtime":0.137507577},"replayed":3},"mds_mem":{"ino":1342,"ino+":3214,"ino-":1872,"dir":13,"dir+":13,"dir-":0,"dn":1340,"dn+":6412,"dn-":5072,"cap":1,"cap+":3202,"cap-":3201,"rss":117408,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.489485060,"avgtime":0.489485060},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.354636273,"avgtime":0.001673323},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":786,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":6145030,"buffer_anon_items":2696,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14081045,"mds_co_items":147701,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4706,"op_send_bytes":25508973,"op_resend":0,"op_reply":4706,"op_latency":{"avgcount":4706,"sum":106.155558357,"avgtime":0.022557492},"op_inflight":0,"oplen_avg":{"avgcount":4706,"sum":7474},"op":4706,"op_r":55,"op_w":4651,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1340,"osdop_read":27,"osdop_write":42,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2680,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14380,"get_sum":4007949,"get_or_fail_fail":0,"get_or_fail_success":14380,"take":0,"take_sum":0,"put":14380,"put_sum":4007949,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4706,"take_sum":25833391,"put":4706,"put_sum":25833391,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4706,"take_sum":4706,"put":4706,"put_sum":4706,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":19,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":797817,"max":3758096384,"get_started":0,"get":12826,"get_sum":24648633,"get_or_fail_fail":0,"get_or_fail_success":12826,"take":0,"take_sum":0,"put":23,"put_sum":23850816,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:52.503 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 431453, "msgr_recv_encrypted_bytes": 107392, "msgr_recv_messages": 1176, "msgr_running_fast_dispatch_time": 0.028677735, "msgr_running_recv_time": 0.026729881, "msgr_running_send_time": 0.051569146, "msgr_running_total_time": 0.113220508, "msgr_send_bytes": 5371574, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 1172, "msgr_send_messages_queue_lat": { "avgcount": 1171, "avgtime": 0.000428175, "sum": 0.501393184 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29963, "avgtime": 1.68e-07, "sum": 0.005039003 }, "msgr_recv_bytes": 3916325, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 10951, "msgr_running_fast_dispatch_time": 0.04350076, "msgr_running_recv_time": 0.515438042, "msgr_running_send_time": 0.603180492, "msgr_running_total_time": 1.095143717, "msgr_send_bytes": 28021356, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 25130, "msgr_send_messages_queue_lat": { "avgcount": 25129, "avgtime": 2.1208e-05, "sum": 0.532941557 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 957261, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2258, "msgr_running_fast_dispatch_time": 0.051816679, "msgr_running_recv_time": 0.04046704, "msgr_running_send_time": 0.09209401, "msgr_running_total_time": 0.196961906, "msgr_send_bytes": 2801802, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2259, "msgr_send_messages_queue_lat": { "avgcount": 2258, "avgtime": 0.000233792, "sum": 0.527903853 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 116, "avgtime": 0.062310043, "sum": 7.227965045 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5163, "avgtime": 0.000211526, "sum": 1.09211067 }, "queue_len": 0 }, "mds": { "caps": 1330, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7341, "ceph_cap_op_revoke": 668, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5050, "inodes_bottom": 5050, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5050, "inodes_top": 0, "inodes_with_caps": 1330, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001447531, "sum": 9.280122065 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1333, "evadd": 12826, "evex": 11496, "evexd": 11495, "evexg": 0, "evlrg": 0, "evtrm": 11496, "expos": 28042820, "jlat": { "avgcount": 23, "avgtime": 0.137507577, "sum": 3.162674279 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28844318 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1340, "dn+": 6412, "dn-": 5072, "heap": 248164, "ino": 1342, "ino+": 3214, "ino-": 1872, "rss": 117408 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.48948506, "sum": 0.48948506 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001673323, "sum": 5.354636273 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 786, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 6145030, "buffer_anon_items": 2696, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14081045, "mds_co_items": 147701, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 4706, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4706, "avgtime": 0.022557492, "sum": 106.155558357 }, "op_pg": 0, "op_r": 55, "op_reply": 4706, "op_resend": 0, "op_rmw": 0, "op_send": 4706, "op_send_bytes": 25508973, "op_w": 4651, "oplen_avg": { "avgcount": 4706, "sum": 7474 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1340, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 27, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2680, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 42, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14380, "get_or_fail_fail": 0, "get_or_fail_success": 14380, "get_started": 0, "get_sum": 4007949, "max": 104857600, "put": 14380, "put_sum": 4007949, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4706, "put_sum": 25833391, "take": 4706, "take_sum": 25833391, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4706, "put_sum": 4706, "take": 4706, "take_sum": 4706, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 19, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 12826, "get_or_fail_fail": 0, "get_or_fail_success": 12826, "get_started": 0, "get_sum": 24648633, "max": 3758096384, "put": 23, "put_sum": 23850816, "take": 0, "take_sum": 0, "val": 797817, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:52.503 INFO:tasks.cephfs.test_strays:Complete purge in 6 seconds 2024-03-06T07:26:52.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:52.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.602+0000 7f7b25ca4640 1 Processor -- start 2024-03-06T07:26:52.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.602+0000 7f7b25ca4640 1 -- start start 2024-03-06T07:26:52.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b25ca4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 0x7f7b200cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b25ca4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b200ce190 con 0x7f7b2015abc0 2024-03-06T07:26:52.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b25ca4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2015b0c0 con 0x7f7b2006d9e0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b25ca4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200cf9a0 con 0x7f7b20154af0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7b2015abc0 0x7f7b20072aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59048/0 (socket says 172.21.15.122:59048) 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1effd640 1 -- 172.21.15.122:0/1750853303 learned_addr learned my addr 172.21.15.122:0/1750853303 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 0x7f7b200cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151051588 0 0) 0x7f7b200ce190 con 0x7f7b2015abc0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08003520 con 0x7f7b2015abc0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3128797853 0 0) 0x7f7b08003520 con 0x7f7b2015abc0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 msgr2=0x7f7b200cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 0x7f7b200cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 >> v1:172.21.15.169:6789/0 conn(0x7f7b2006d9e0 legacy=0x7f7b20071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2015b3a0 con 0x7f7b2015abc0 2024-03-06T07:26:52.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7b10001690 con 0x7f7b2015abc0 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 >> v1:172.21.15.122:6789/0 conn(0x7f7b2015abc0 legacy=0x7f7b20072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.603+0000 7f7b1e7fc640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b080039e0 0x7f7b08003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1e7fc640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 0x7f7b08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1e7fc640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b08004d40 0x7f7b080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0800b9c0 con 0x7f7b080039e0 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0800bb40 con 0x7f7b08004d40 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0800df50 con 0x7f7b20154af0 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b080039e0 0x7f7b08003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1effd640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 0x7f7b08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1ffff640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b08004d40 0x7f7b080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1f7fe640 1 -- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 msgr2=0x7f7b08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b20154af0 0x7f7b08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1f7fe640 1 -- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b08004d40 msgr2=0x7f7b080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b08004d40 0x7f7b080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1f7fe640 1 -- 172.21.15.122:0/1750853303 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08003520 con 0x7f7b080039e0 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.604+0000 7f7b1ffff640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b08004d40 0x7f7b080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b080039e0 0x7f7b08003dc0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009ac0 tx=0x7f7b08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e420b0b2abe2273c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14039070 con 0x7f7b080039e0 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1402c600 con 0x7f7b080039e0 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b1e7fc640 1 -- 172.21.15.122:0/1750853303 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14009690 con 0x7f7b080039e0 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b25ca4640 1 -- 172.21.15.122:0/1750853303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b080039e0 msgr2=0x7f7b08003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/1750853303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b080039e0 0x7f7b08003dc0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009ac0 tx=0x7f7b08005830 comp rx=0 tx=0).stop 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b1ffff640 1 -- 172.21.15.122:0/1750853303 reap_dead start 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b25ca4640 1 -- 172.21.15.122:0/1750853303 shutdown_connections 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b25ca4640 1 -- 172.21.15.122:0/1750853303 >> 172.21.15.122:0/1750853303 conn(0x7f7b2006b160 msgr2=0x7f7b2006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b25ca4640 1 -- 172.21.15.122:0/1750853303 shutdown_connections 2024-03-06T07:26:52.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.605+0000 7f7b25ca4640 1 -- 172.21.15.122:0/1750853303 wait complete. 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.606+0000 7f7b25ca4640 1 Processor -- start 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 -- start start 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b080039e0 0x7f7b2015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 0x7f7b20149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b201374d0 0x7f7b200ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2015b960 con 0x7f7b2014aeb0 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20070d70 con 0x7f7b080039e0 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b25ca4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20154ed0 con 0x7f7b201374d0 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b1effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 0x7f7b20149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b1effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 0x7f7b20149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52114/0 (socket says 172.21.15.122:52114) 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.607+0000 7f7b1effd640 1 -- 172.21.15.122:0/3330915998 learned_addr learned my addr 172.21.15.122:0/3330915998 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1ffff640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b201374d0 0x7f7b200ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b080039e0 0x7f7b2015ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1effd640 1 -- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b201374d0 msgr2=0x7f7b200ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1effd640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b201374d0 0x7f7b200ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1effd640 1 -- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b080039e0 msgr2=0x7f7b2015ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1effd640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b080039e0 0x7f7b2015ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1effd640 1 -- 172.21.15.122:0/3330915998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20149b70 con 0x7f7b2014aeb0 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1ffff640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b201374d0 0x7f7b200ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b080039e0 0x7f7b2015ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b1effd640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 0x7f7b20149630 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007320 tx=0x7f7b100077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1001a070 con 0x7f7b2014aeb0 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1000fa80 con 0x7f7b2014aeb0 2024-03-06T07:26:52.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000fe70 con 0x7f7b2014aeb0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b20149e10 con 0x7f7b2014aeb0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.608+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2014a3d0 con 0x7f7b2014aeb0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.609+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b20069ee0 con 0x7f7b2014aeb0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.610+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b1001d070 con 0x7f7b2014aeb0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.610+0000 7f7b07fff640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b0c06b500 0x7f7b0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.610+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b0c06b500 0x7f7b0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.610+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7b10089cc0 con 0x7f7b2014aeb0 2024-03-06T07:26:52.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.611+0000 7f7b1f7fe640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b0c06b500 0x7f7b0c06d9c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b14036000 tx=0x7f7b14030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.614+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b10058ae0 con 0x7f7b2014aeb0 2024-03-06T07:26:52.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.828+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b20072370 con 0x7f7b2014aeb0 2024-03-06T07:26:52.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.829+0000 7f7b07fff640 1 -- 172.21.15.122:0/3330915998 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f7b100583e0 con 0x7f7b2014aeb0 2024-03-06T07:26:52.831 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:52.831 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:52.831 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b0c06b500 msgr2=0x7f7b0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b0c06b500 0x7f7b0c06d9c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b14036000 tx=0x7f7b14030000 comp rx=0 tx=0).stop 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 msgr2=0x7f7b20149630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 0x7f7b20149630 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007320 tx=0x7f7b100077e0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 shutdown_connections 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b0c06b500 0x7f7b0c06d9c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b201374d0 0x7f7b200ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.830+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b2014aeb0 0x7f7b20149630 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.831+0000 7f7b25ca4640 1 --2- 172.21.15.122:0/3330915998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b080039e0 0x7f7b2015ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.831+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 >> 172.21.15.122:0/3330915998 conn(0x7f7b2006b160 msgr2=0x7f7b2014f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.831+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 shutdown_connections 2024-03-06T07:26:52.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:52.831+0000 7f7b25ca4640 1 -- 172.21.15.122:0/3330915998 wait complete. 2024-03-06T07:26:52.843 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:52.959 INFO:teuthology.orchestra.run.smithi122.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1176,"msgr_send_messages":1172,"msgr_recv_bytes":431453,"msgr_send_bytes":5371574,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.113220508,"msgr_running_send_time":0.051569146,"msgr_running_recv_time":0.026729881,"msgr_running_fast_dispatch_time":0.028677735,"msgr_send_messages_queue_lat":{"avgcount":1171,"sum":0.501393184,"avgtime":0.000428175},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000242,"avgtime":0.000000121},"msgr_recv_encrypted_bytes":107392,"msgr_send_encrypted_bytes":8544},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10951,"msgr_send_messages":25130,"msgr_recv_bytes":3916325,"msgr_send_bytes":28021356,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.095143717,"msgr_running_send_time":0.603180492,"msgr_running_recv_time":0.515438042,"msgr_running_fast_dispatch_time":0.043500760,"msgr_send_messages_queue_lat":{"avgcount":25129,"sum":0.532941557,"avgtime":0.000021208},"msgr_handle_ack_lat":{"avgcount":29963,"sum":0.005039003,"avgtime":0.000000168},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2258,"msgr_send_messages":2259,"msgr_recv_bytes":957261,"msgr_send_bytes":2801802,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.196961906,"msgr_running_send_time":0.092094010,"msgr_running_recv_time":0.040467040,"msgr_running_fast_dispatch_time":0.051816679,"msgr_send_messages_queue_lat":{"avgcount":2258,"sum":0.527903853,"avgtime":0.000233792},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":116,"sum":7.227965045,"avgtime":0.062310043}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5163,"sum":1.092110670,"avgtime":0.000211526}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.280122065,"avgtime":0.001447531},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":5050,"inodes_top":0,"inodes_bottom":5050,"inodes_pin_tail":0,"inodes_pinned":5050,"inodes_expired":0,"inodes_with_caps":1330,"caps":1330,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":668,"ceph_cap_op_grant":7341,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12826,"evex":11496,"evtrm":11496,"ev":1333,"evexg":0,"evexd":11495,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28042820,"wrpos":28844318,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.162674279,"avgtime":0.137507577},"replayed":3},"mds_mem":{"ino":1342,"ino+":3214,"ino-":1872,"dir":13,"dir+":13,"dir-":0,"dn":1340,"dn+":6412,"dn-":5072,"cap":1,"cap+":3202,"cap-":3201,"rss":117408,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.035885336,"avgtime":0.000948714},"req_getattr_latency":{"avgcount":6,"sum":0.001172485,"avgtime":0.000195414},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002595468,"avgtime":0.002595468},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.396347443,"avgtime":0.132115814},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.489485060,"avgtime":0.489485060},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.354636273,"avgtime":0.001673323},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":786,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":6145030,"buffer_anon_items":2696,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14081045,"mds_co_items":147701,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4706,"op_send_bytes":25508973,"op_resend":0,"op_reply":4706,"op_latency":{"avgcount":4706,"sum":106.155558357,"avgtime":0.022557492},"op_inflight":0,"oplen_avg":{"avgcount":4706,"sum":7474},"op":4706,"op_r":55,"op_w":4651,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1340,"osdop_read":27,"osdop_write":42,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2680,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14380,"get_sum":4007949,"get_or_fail_fail":0,"get_or_fail_success":14380,"take":0,"take_sum":0,"put":14380,"put_sum":4007949,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4706,"take_sum":25833391,"put":4706,"put_sum":25833391,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4706,"take_sum":4706,"put":4706,"put_sum":4706,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":19,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55f5bd20a820":{"val":797817,"max":3758096384,"get_started":0,"get":12826,"get_sum":24648633,"get_or_fail_fail":0,"get_or_fail_success":12826,"take":0,"take_sum":0,"put":23,"put_sum":23850816,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:52.960 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.21e-07, "sum": 2.42e-07 }, "msgr_recv_bytes": 431453, "msgr_recv_encrypted_bytes": 107392, "msgr_recv_messages": 1176, "msgr_running_fast_dispatch_time": 0.028677735, "msgr_running_recv_time": 0.026729881, "msgr_running_send_time": 0.051569146, "msgr_running_total_time": 0.113220508, "msgr_send_bytes": 5371574, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 1172, "msgr_send_messages_queue_lat": { "avgcount": 1171, "avgtime": 0.000428175, "sum": 0.501393184 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29963, "avgtime": 1.68e-07, "sum": 0.005039003 }, "msgr_recv_bytes": 3916325, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 10951, "msgr_running_fast_dispatch_time": 0.04350076, "msgr_running_recv_time": 0.515438042, "msgr_running_send_time": 0.603180492, "msgr_running_total_time": 1.095143717, "msgr_send_bytes": 28021356, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 25130, "msgr_send_messages_queue_lat": { "avgcount": 25129, "avgtime": 2.1208e-05, "sum": 0.532941557 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 957261, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2258, "msgr_running_fast_dispatch_time": 0.051816679, "msgr_running_recv_time": 0.04046704, "msgr_running_send_time": 0.09209401, "msgr_running_total_time": 0.196961906, "msgr_send_bytes": 2801802, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2259, "msgr_send_messages_queue_lat": { "avgcount": 2258, "avgtime": 0.000233792, "sum": 0.527903853 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 116, "avgtime": 0.062310043, "sum": 7.227965045 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5163, "avgtime": 0.000211526, "sum": 1.09211067 }, "queue_len": 0 }, "mds": { "caps": 1330, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7341, "ceph_cap_op_revoke": 668, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5050, "inodes_bottom": 5050, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5050, "inodes_top": 0, "inodes_with_caps": 1330, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001447531, "sum": 9.280122065 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1333, "evadd": 12826, "evex": 11496, "evexd": 11495, "evexg": 0, "evlrg": 0, "evtrm": 11496, "expos": 28042820, "jlat": { "avgcount": 23, "avgtime": 0.137507577, "sum": 3.162674279 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28844318 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1340, "dn+": 6412, "dn-": 5072, "heap": 248164, "ino": 1342, "ino+": 3214, "ino-": 1872, "rss": 117408 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000948714, "sum": 3.035885336 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195414, "sum": 0.001172485 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002595468, "sum": 0.002595468 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.132115814, "sum": 0.396347443 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.48948506, "sum": 0.48948506 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001673323, "sum": 5.354636273 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 786, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 6145030, "buffer_anon_items": 2696, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14081045, "mds_co_items": 147701, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 4706, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4706, "avgtime": 0.022557492, "sum": 106.155558357 }, "op_pg": 0, "op_r": 55, "op_reply": 4706, "op_resend": 0, "op_rmw": 0, "op_send": 4706, "op_send_bytes": 25508973, "op_w": 4651, "oplen_avg": { "avgcount": 4706, "sum": 7474 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1340, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 27, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2680, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 42, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14380, "get_or_fail_fail": 0, "get_or_fail_success": 14380, "get_started": 0, "get_sum": 4007949, "max": 104857600, "put": 14380, "put_sum": 4007949, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4706, "put_sum": 25833391, "take": 4706, "take_sum": 25833391, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4706, "put_sum": 4706, "take": 4706, "take_sum": 4706, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 19, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55f5bd20a820": { "get": 12826, "get_or_fail_fail": 0, "get_or_fail_success": 12826, "get_started": 0, "get_sum": 24648633, "max": 3758096384, "put": 23, "put_sum": 23850816, "take": 0, "take_sum": 0, "val": 797817, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:52.960 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:52.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:53.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:53.041 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:53.041 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:53.041 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:53.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:53.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.196+0000 7fd15b98a640 1 Processor -- start 2024-03-06T07:26:53.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.196+0000 7fd15b98a640 1 -- start start 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd15b98a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd1540b4d20 0x7fd15406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd15b98a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1540b37b0 con 0x7fd15415e370 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd15b98a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1540b31f0 con 0x7fd1540ae350 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd15b98a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15415e750 con 0x7fd1540b4d20 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd158f25640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd1540b4d20 0x7fd15406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd159726640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd15415e370 0x7fd15406e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59060/0 (socket says 172.21.15.122:59060) 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd159726640 1 -- 172.21.15.122:0/1669934894 learned_addr learned my addr 172.21.15.122:0/1669934894 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd158f25640 1 -- 172.21.15.122:0/1669934894 >> v1:172.21.15.169:6789/0 conn(0x7fd1540ae350 legacy=0x7fd1540ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd158f25640 1 -- 172.21.15.122:0/1669934894 >> v1:172.21.15.122:6789/0 conn(0x7fd15415e370 legacy=0x7fd15406e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd158f25640 1 -- 172.21.15.122:0/1669934894 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1540b34d0 con 0x7fd1540b4d20 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.197+0000 7fd14bfff640 1 -- 172.21.15.122:0/1669934894 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39672854 0 0) 0x7fd1540b37b0 con 0x7fd15415e370 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.198+0000 7fd158f25640 1 --2- 172.21.15.122:0/1669934894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540b4d20 0x7fd15406ec30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd1440097a0 tx=0x7fd14402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951883bca481ee77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.198+0000 7fd14bfff640 1 -- 172.21.15.122:0/1669934894 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1440315a0 con 0x7fd1540b4d20 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.198+0000 7fd14bfff640 1 -- 172.21.15.122:0/1669934894 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd144037070 con 0x7fd1540b4d20 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.198+0000 7fd14bfff640 1 -- 172.21.15.122:0/1669934894 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144031960 con 0x7fd1540b4d20 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 -- 172.21.15.122:0/1669934894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540b4d20 msgr2=0x7fd15406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 --2- 172.21.15.122:0/1669934894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540b4d20 0x7fd15406ec30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd1440097a0 tx=0x7fd14402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 -- 172.21.15.122:0/1669934894 shutdown_connections 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 --2- 172.21.15.122:0/1669934894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540b4d20 0x7fd15406ec30 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 -- 172.21.15.122:0/1669934894 >> 172.21.15.122:0/1669934894 conn(0x7fd1540b1dc0 msgr2=0x7fd1540b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 -- 172.21.15.122:0/1669934894 shutdown_connections 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.199+0000 7fd15b98a640 1 -- 172.21.15.122:0/1669934894 wait complete. 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.200+0000 7fd15b98a640 1 Processor -- start 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.200+0000 7fd15b98a640 1 -- start start 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd15b98a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 0x7fd154147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd15b98a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1540b4d20 0x7fd15409ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd15b98a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15415e370 0x7fd15409d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd15b98a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540b29c0 con 0x7fd1540ae350 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd15b98a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540ade30 con 0x7fd1540b4d20 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd15b98a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540b2b40 con 0x7fd15415e370 2024-03-06T07:26:53.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 0x7fd154147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 0x7fd154147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52130/0 (socket says 172.21.15.122:52130) 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 -- 172.21.15.122:0/3967630772 learned_addr learned my addr 172.21.15.122:0/3967630772 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159f27640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15415e370 0x7fd15409d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd158f25640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1540b4d20 0x7fd15409ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 -- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15415e370 msgr2=0x7fd15409d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15415e370 0x7fd15409d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 -- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1540b4d20 msgr2=0x7fd15409ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1540b4d20 0x7fd15409ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 -- 172.21.15.122:0/3967630772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154161320 con 0x7fd1540ae350 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd158f25640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1540b4d20 0x7fd15409ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.201+0000 7fd159726640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 0x7fd154147350 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd15000c500 tx=0x7fd15000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.202+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150018070 con 0x7fd1540ae350 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.202+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd150007580 con 0x7fd1540ae350 2024-03-06T07:26:53.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.202+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd154160560 con 0x7fd1540ae350 2024-03-06T07:26:53.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.202+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150004a20 con 0x7fd1540ae350 2024-03-06T07:26:53.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.202+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd154160b20 con 0x7fd1540ae350 2024-03-06T07:26:53.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.203+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1540b1a20 con 0x7fd1540ae350 2024-03-06T07:26:53.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.203+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd15001b070 con 0x7fd1540ae350 2024-03-06T07:26:53.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.203+0000 7fd149ffb640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13806b4d0 0x7fd13806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.204+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd150089430 con 0x7fd1540ae350 2024-03-06T07:26:53.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.204+0000 7fd158f25640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13806b4d0 0x7fd13806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.204+0000 7fd158f25640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13806b4d0 0x7fd13806d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd144036000 tx=0x7fd144008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.208+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1540b1a20 con 0x7fd1540ae350 2024-03-06T07:26:53.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.429+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd154069ee0 con 0x7fd1540ae350 2024-03-06T07:26:53.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.431+0000 7fd149ffb640 1 -- 172.21.15.122:0/3967630772 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16739 (secure 0 0 0) 0x7fd150057c90 con 0x7fd1540ae350 2024-03-06T07:26:53.434 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:53.434 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":97,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:28.153647+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:6835/2187235474":"2024-03-07T07:26:28.144746+0000","172.21.15.169:6833/2187235474":"2024-03-07T07:26:28.144746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:53.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13806b4d0 msgr2=0x7fd13806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13806b4d0 0x7fd13806d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd144036000 tx=0x7fd144008030 comp rx=0 tx=0).stop 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 msgr2=0x7fd154147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 0x7fd154147350 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd15000c500 tx=0x7fd15000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 shutdown_connections 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13806b4d0 0x7fd13806d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15415e370 0x7fd15409d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1540b4d20 0x7fd15409ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 --2- 172.21.15.122:0/3967630772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1540ae350 0x7fd154147350 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 >> 172.21.15.122:0/3967630772 conn(0x7fd1540b1dc0 msgr2=0x7fd1540aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 shutdown_connections 2024-03-06T07:26:53.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.433+0000 7fd15b98a640 1 -- 172.21.15.122:0/3967630772 wait complete. 2024-03-06T07:26:53.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:17.659781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:18.415605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:53.447 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:53.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.565+0000 7f1afaca4640 1 Processor -- start 2024-03-06T07:26:53.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.565+0000 7f1afaca4640 1 -- start start 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.566+0000 7f1afaca4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 0x7f1af40b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.566+0000 7f1afaca4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1af40cec60 0x7f1af40cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.566+0000 7f1afaca4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af40cd4d0 con 0x7f1af40b16b0 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.566+0000 7f1afaca4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af40cf920 con 0x7f1af40cec60 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.566+0000 7f1afaca4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40cd7b0 con 0x7f1af415f5e0 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 0x7f1af40b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af9241640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1af40cec60 0x7f1af40cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 0x7f1af40b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:60754/0 (socket says 172.21.15.122:60754) 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 -- 172.21.15.122:0/2978657659 learned_addr learned my addr 172.21.15.122:0/2978657659 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 -- 172.21.15.122:0/2978657659 >> v2:172.21.15.169:3300/0 conn(0x7f1af40cec60 msgr2=0x7f1af40cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 --2- 172.21.15.122:0/2978657659 >> v2:172.21.15.169:3300/0 conn(0x7f1af40cec60 0x7f1af40cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 -- 172.21.15.122:0/2978657659 >> v1:172.21.15.169:6789/0 conn(0x7f1af40b16b0 legacy=0x7f1af40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af8a40640 1 -- 172.21.15.122:0/2978657659 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40cda90 con 0x7f1af415f5e0 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.567+0000 7f1af9241640 1 --2- 172.21.15.122:0/2978657659 >> v2:172.21.15.169:3300/0 conn(0x7f1af40cec60 0x7f1af40cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.568+0000 7f1af8a40640 1 --2- 172.21.15.122:0/2978657659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 0x7f1af40b2bf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009870 tx=0x7f1ae802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=108560a607033495 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.568+0000 7f1af37fe640 1 -- 172.21.15.122:0/2978657659 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8035020 con 0x7f1af415f5e0 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.568+0000 7f1af37fe640 1 -- 172.21.15.122:0/2978657659 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae8031640 con 0x7f1af415f5e0 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.568+0000 7f1af37fe640 1 -- 172.21.15.122:0/2978657659 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae802d040 con 0x7f1af415f5e0 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 -- 172.21.15.122:0/2978657659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 msgr2=0x7f1af40b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 --2- 172.21.15.122:0/2978657659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 0x7f1af40b2bf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009870 tx=0x7f1ae802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 -- 172.21.15.122:0/2978657659 shutdown_connections 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 --2- 172.21.15.122:0/2978657659 >> v2:172.21.15.169:3300/0 conn(0x7f1af40cec60 0x7f1af40cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 --2- 172.21.15.122:0/2978657659 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af415f5e0 0x7f1af40b2bf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 -- 172.21.15.122:0/2978657659 >> 172.21.15.122:0/2978657659 conn(0x7f1af40d1d50 msgr2=0x7f1af40d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 -- 172.21.15.122:0/2978657659 shutdown_connections 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.569+0000 7f1afaca4640 1 -- 172.21.15.122:0/2978657659 wait complete. 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.571+0000 7f1afaca4640 1 Processor -- start 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.571+0000 7f1afaca4640 1 -- start start 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1afaca4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af40b16b0 0x7f1af4154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1afaca4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af40cec60 0x7f1af40c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1afaca4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 0x7f1af40c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1afaca4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40cf270 con 0x7f1af415f5e0 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1afaca4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40cf0f0 con 0x7f1af40cec60 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1afaca4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40d2870 con 0x7f1af40b16b0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1af9241640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 0x7f1af40c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1af9241640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 0x7f1af40c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52136/0 (socket says 172.21.15.122:52136) 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1af9241640 1 -- 172.21.15.122:0/1869561572 learned_addr learned my addr 172.21.15.122:0/1869561572 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1af3fff640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af40cec60 0x7f1af40c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.572+0000 7f1af8a40640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af40b16b0 0x7f1af4154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af9241640 1 -- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af40b16b0 msgr2=0x7f1af4154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af9241640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af40b16b0 0x7f1af4154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af9241640 1 -- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af40cec60 msgr2=0x7f1af40c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af9241640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af40cec60 0x7f1af40c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af9241640 1 -- 172.21.15.122:0/1869561572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40c6f00 con 0x7f1af415f5e0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af9241640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 0x7f1af40c92f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009ea0 tx=0x7f1aec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec018030 con 0x7f1af415f5e0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec00cde0 con 0x7f1af415f5e0 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af40c83c0 con 0x7f1af415f5e0 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec003bb0 con 0x7f1af415f5e0 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.573+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af40b7010 con 0x7f1af415f5e0 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.574+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af40b7210 con 0x7f1af415f5e0 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.574+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aec005d10 con 0x7f1af415f5e0 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.575+0000 7f1af17fa640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ad806b4d0 0x7f1ad806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.575+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1aec08aa40 con 0x7f1af415f5e0 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.575+0000 7f1af8a40640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ad806b4d0 0x7f1ad806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.575+0000 7f1af8a40640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ad806b4d0 0x7f1ad806d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8036000 tx=0x7f1ae803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.579+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1af40b7210 con 0x7f1af415f5e0 2024-03-06T07:26:53.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.757+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1af4068050 con 0x7f1af415f5e0 2024-03-06T07:26:53.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.757+0000 7f1af17fa640 1 -- 172.21.15.122:0/1869561572 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1aec08ae20 con 0x7f1af415f5e0 2024-03-06T07:26:53.760 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.759+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ad806b4d0 msgr2=0x7f1ad806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.759+0000 7f1afaca4640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ad806b4d0 0x7f1ad806d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8036000 tx=0x7f1ae803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.759+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 msgr2=0x7f1af40c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.759+0000 7f1afaca4640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 0x7f1af40c92f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009ea0 tx=0x7f1aec009f90 comp rx=0 tx=0).stop 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 shutdown_connections 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ad806b4d0 0x7f1ad806d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af415f5e0 0x7f1af40c92f0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af40cec60 0x7f1af40c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 --2- 172.21.15.122:0/1869561572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1af40b16b0 0x7f1af4154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 >> 172.21.15.122:0/1869561572 conn(0x7f1af40d1d50 msgr2=0x7f1af406f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 shutdown_connections 2024-03-06T07:26:53.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.760+0000 7f1afaca4640 1 -- 172.21.15.122:0/1869561572 wait complete. 2024-03-06T07:26:53.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:53.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.876+0000 7f80286f4640 1 Processor -- start 2024-03-06T07:26:53.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.876+0000 7f80286f4640 1 -- start start 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.877+0000 7f80286f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f802015f5e0 0x7f80200b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.877+0000 7f80286f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f80200b16b0 0x7f80200ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.877+0000 7f80286f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200cd4d0 con 0x7f80200d2870 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.877+0000 7f80286f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80200cf920 con 0x7f802015f5e0 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.877+0000 7f80286f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80200cd7b0 con 0x7f80200b16b0 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.877+0000 7f8026490640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f802015f5e0 0x7f80200b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8026490640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f802015f5e0 0x7f80200b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52150/0 (socket says 172.21.15.122:52150) 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8026490640 1 -- 172.21.15.122:0/2039930309 learned_addr learned my addr 172.21.15.122:0/2039930309 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8025c8f640 1 --2- 172.21.15.122:0/2039930309 >> v2:172.21.15.169:3300/0 conn(0x7f80200b16b0 0x7f80200ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8026490640 1 -- 172.21.15.122:0/2039930309 >> v2:172.21.15.169:3300/0 conn(0x7f80200b16b0 msgr2=0x7f80200ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8026490640 1 --2- 172.21.15.122:0/2039930309 >> v2:172.21.15.169:3300/0 conn(0x7f80200b16b0 0x7f80200ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8026490640 1 -- 172.21.15.122:0/2039930309 >> v1:172.21.15.169:6789/0 conn(0x7f80200d2870 legacy=0x7f802015fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8026490640 1 -- 172.21.15.122:0/2039930309 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80200cda90 con 0x7f802015f5e0 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.878+0000 7f8025c8f640 1 --2- 172.21.15.122:0/2039930309 >> v2:172.21.15.169:3300/0 conn(0x7f80200b16b0 0x7f80200ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f8026490640 1 --2- 172.21.15.122:0/2039930309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f802015f5e0 0x7f80200b2bf0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f8020071dc0 tx=0x7f801002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956143597396a924 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f802548e640 1 -- 172.21.15.122:0/2039930309 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010035020 con 0x7f802015f5e0 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f802548e640 1 -- 172.21.15.122:0/2039930309 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010031870 con 0x7f802015f5e0 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f802548e640 1 -- 172.21.15.122:0/2039930309 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010031c40 con 0x7f802015f5e0 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 -- 172.21.15.122:0/2039930309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f802015f5e0 msgr2=0x7f80200b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 --2- 172.21.15.122:0/2039930309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f802015f5e0 0x7f80200b2bf0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f8020071dc0 tx=0x7f801002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 -- 172.21.15.122:0/2039930309 shutdown_connections 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 --2- 172.21.15.122:0/2039930309 >> v2:172.21.15.169:3300/0 conn(0x7f80200b16b0 0x7f80200ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 --2- 172.21.15.122:0/2039930309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f802015f5e0 0x7f80200b2bf0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 -- 172.21.15.122:0/2039930309 >> 172.21.15.122:0/2039930309 conn(0x7f80200d1d50 msgr2=0x7f80200d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 -- 172.21.15.122:0/2039930309 shutdown_connections 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.879+0000 7f80286f4640 1 -- 172.21.15.122:0/2039930309 wait complete. 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.880+0000 7f80286f4640 1 Processor -- start 2024-03-06T07:26:53.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 -- start start 2024-03-06T07:26:53.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 0x7f802015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80200d2870 0x7f80200cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802015f5e0 0x7f80200ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200cf270 con 0x7f80200b16b0 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200cf0f0 con 0x7f80200d2870 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f80286f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80200cea90 con 0x7f802015f5e0 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f8026490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 0x7f802015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f8026490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 0x7f802015a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52156/0 (socket says 172.21.15.122:52156) 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.881+0000 7f8026490640 1 -- 172.21.15.122:0/258863528 learned_addr learned my addr 172.21.15.122:0/258863528 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8025c8f640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80200d2870 0x7f80200cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026c91640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802015f5e0 0x7f80200ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026490640 1 -- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802015f5e0 msgr2=0x7f80200ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026490640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802015f5e0 0x7f80200ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026490640 1 -- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80200d2870 msgr2=0x7f80200cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026490640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80200d2870 0x7f80200cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026490640 1 -- 172.21.15.122:0/258863528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80200cbc40 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8026490640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 0x7f802015a520 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8010000c00 tx=0x7f80100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010035040 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010030d00 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80200caba0 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100456a0 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.882+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80200cb160 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.883+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8020068050 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.883+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8010038020 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.884+0000 7f8016ffd640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800406b4d0 0x7f800406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.884+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f80100ac4f0 con 0x7f80200b16b0 2024-03-06T07:26:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.884+0000 7f8025c8f640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800406b4d0 0x7f800406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.884+0000 7f8025c8f640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800406b4d0 0x7f800406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f802015ea20 tx=0x7f801c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:53.888+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f801007b3a0 con 0x7f80200b16b0 2024-03-06T07:26:54.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:54.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:54.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.101+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80200b7cd0 con 0x7f80200b16b0 2024-03-06T07:26:54.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.102+0000 7f8016ffd640 1 -- 172.21.15.122:0/258863528 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7f801007aca0 con 0x7f80200b16b0 2024-03-06T07:26:54.104 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:26:54.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800406b4d0 msgr2=0x7f800406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800406b4d0 0x7f800406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f802015ea20 tx=0x7f801c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 msgr2=0x7f802015a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 0x7f802015a520 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8010000c00 tx=0x7f80100046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 shutdown_connections 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800406b4d0 0x7f800406d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802015f5e0 0x7f80200ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80200d2870 0x7f80200cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 --2- 172.21.15.122:0/258863528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80200b16b0 0x7f802015a520 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 >> 172.21.15.122:0/258863528 conn(0x7f80200d1d50 msgr2=0x7f802006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 shutdown_connections 2024-03-06T07:26:54.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.104+0000 7f80286f4640 1 -- 172.21.15.122:0/258863528 wait complete. 2024-03-06T07:26:54.116 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:54.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.224+0000 7f3cde6a4640 1 Processor -- start 2024-03-06T07:26:54.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.224+0000 7f3cde6a4640 1 -- start start 2024-03-06T07:26:54.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cde6a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3cd80d2240 0x7f3cd80b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cde6a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd8161b40 con 0x7f3cd8161760 2024-03-06T07:26:54.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cde6a4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd80d27e0 con 0x7f3cd80b5240 2024-03-06T07:26:54.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cde6a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd80cf580 con 0x7f3cd80d2240 2024-03-06T07:26:54.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cdcc41640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3cd80d2240 0x7f3cd80b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cd7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3cd8161760 0x7f3cd80d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59074/0 (socket says 172.21.15.122:59074) 2024-03-06T07:26:54.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cd7fff640 1 -- 172.21.15.122:0/1263685536 learned_addr learned my addr 172.21.15.122:0/1263685536 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:54.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cdcc41640 1 -- 172.21.15.122:0/1263685536 >> v1:172.21.15.169:6789/0 conn(0x7f3cd80b5240 legacy=0x7f3cd80c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cdcc41640 1 -- 172.21.15.122:0/1263685536 >> v1:172.21.15.122:6789/0 conn(0x7f3cd8161760 legacy=0x7f3cd80d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cdcc41640 1 -- 172.21.15.122:0/1263685536 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd80cf860 con 0x7f3cd80d2240 2024-03-06T07:26:54.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.225+0000 7f3cd6ffd640 1 -- 172.21.15.122:0/1263685536 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171293557 0 0) 0x7f3cd8161b40 con 0x7f3cd8161760 2024-03-06T07:26:54.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.226+0000 7f3cdcc41640 1 --2- 172.21.15.122:0/1263685536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80d2240 0x7f3cd80b4cd0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009d90 tx=0x7f3cd002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4371d4b83ec2f0f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.226+0000 7f3cd6ffd640 1 -- 172.21.15.122:0/1263685536 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0039070 con 0x7f3cd80d2240 2024-03-06T07:26:54.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.226+0000 7f3cd6ffd640 1 -- 172.21.15.122:0/1263685536 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd002cbd0 con 0x7f3cd80d2240 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.226+0000 7f3cd6ffd640 1 -- 172.21.15.122:0/1263685536 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0034440 con 0x7f3cd80d2240 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1263685536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80d2240 msgr2=0x7f3cd80b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1263685536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80d2240 0x7f3cd80b4cd0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009d90 tx=0x7f3cd002be20 comp rx=0 tx=0).stop 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1263685536 shutdown_connections 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1263685536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80d2240 0x7f3cd80b4cd0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1263685536 >> 172.21.15.122:0/1263685536 conn(0x7f3cd80d3f60 msgr2=0x7f3cd80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1263685536 shutdown_connections 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.227+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1263685536 wait complete. 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.228+0000 7f3cde6a4640 1 Processor -- start 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.228+0000 7f3cde6a4640 1 -- start start 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cde6a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 0x7f3cd815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cde6a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cd80d2240 0x7f3cd809e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cde6a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cd8161760 0x7f3cd809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cde6a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80d2c40 con 0x7f3cd80b5240 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cde6a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80d2ac0 con 0x7f3cd80d2240 2024-03-06T07:26:54.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cde6a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80d4a80 con 0x7f3cd8161760 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 0x7f3cd815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 0x7f3cd815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52174/0 (socket says 172.21.15.122:52174) 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 -- 172.21.15.122:0/1947556881 learned_addr learned my addr 172.21.15.122:0/1947556881 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cdcc41640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cd8161760 0x7f3cd809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd77fe640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cd80d2240 0x7f3cd809e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 -- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cd8161760 msgr2=0x7f3cd809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cd8161760 0x7f3cd809ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 -- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cd80d2240 msgr2=0x7f3cd809e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cd80d2240 0x7f3cd809e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 -- 172.21.15.122:0/1947556881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd80b70b0 con 0x7f3cd80b5240 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.229+0000 7f3cd7fff640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 0x7f3cd815cf90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c430 tx=0x7f3ccc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.230+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0074e0 con 0x7f3cd80b5240 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.230+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0047e0 con 0x7f3cd80b5240 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.230+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc004bd0 con 0x7f3cd80b5240 2024-03-06T07:26:54.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.230+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd80b7390 con 0x7f3cd80b5240 2024-03-06T07:26:54.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.230+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd80b80c0 con 0x7f3cd80b5240 2024-03-06T07:26:54.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.231+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd808dd20 con 0x7f3cd80b5240 2024-03-06T07:26:54.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.232+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ccc011070 con 0x7f3cd80b5240 2024-03-06T07:26:54.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.232+0000 7f3cd4ff9640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3cc406b4d0 0x7f3cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.232+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ccc089f20 con 0x7f3cd80b5240 2024-03-06T07:26:54.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.232+0000 7f3cd77fe640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3cc406b4d0 0x7f3cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.232+0000 7f3cd77fe640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3cc406b4d0 0x7f3cc406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80c9e80 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.236+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cd808dd20 con 0x7f3cd80b5240 2024-03-06T07:26:54.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.410+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3cd8069ee0 con 0x7f3cd80b5240 2024-03-06T07:26:54.412 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:26:54.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.410+0000 7f3cd4ff9640 1 -- 172.21.15.122:0/1947556881 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3ccc058d40 con 0x7f3cd80b5240 2024-03-06T07:26:54.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3cc406b4d0 msgr2=0x7f3cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3cc406b4d0 0x7f3cc406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80c9e80 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 msgr2=0x7f3cd815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 0x7f3cd815cf90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c430 tx=0x7f3ccc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 shutdown_connections 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3cc406b4d0 0x7f3cc406d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3cd8161760 0x7f3cd809ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cd80d2240 0x7f3cd809e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 --2- 172.21.15.122:0/1947556881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cd80b5240 0x7f3cd815cf90 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 >> 172.21.15.122:0/1947556881 conn(0x7f3cd80d3f60 msgr2=0x7f3cd80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 shutdown_connections 2024-03-06T07:26:54.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.412+0000 7f3cde6a4640 1 -- 172.21.15.122:0/1947556881 wait complete. 2024-03-06T07:26:54.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:54.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.577+0000 7fb712d0f640 1 Processor -- start 2024-03-06T07:26:54.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.577+0000 7fb712d0f640 1 -- start start 2024-03-06T07:26:54.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb712d0f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c154af0 0x7fb70c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb712d0f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb70c15abc0 0x7fb70c072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb712d0f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb70c06d9e0 0x7fb70c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb712d0f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70c06e390 con 0x7fb70c06d9e0 2024-03-06T07:26:54.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb712d0f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70c0cf9a0 con 0x7fb70c15abc0 2024-03-06T07:26:54.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb712d0f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c070d70 con 0x7fb70c154af0 2024-03-06T07:26:54.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb70c06d9e0 0x7fb70c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb70c06d9e0 0x7fb70c06de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54980/0 (socket says 172.21.15.122:54980) 2024-03-06T07:26:54.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 -- 172.21.15.122:0/1394071384 learned_addr learned my addr 172.21.15.122:0/1394071384 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:54.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb710aab640 1 --2- 172.21.15.122:0/1394071384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c154af0 0x7fb70c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb70bfff640 1 --2- 172.21.15.122:0/1394071384 >> v2:172.21.15.169:3300/0 conn(0x7fb70c15abc0 0x7fb70c072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 -- 172.21.15.122:0/1394071384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c154af0 msgr2=0x7fb70c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 --2- 172.21.15.122:0/1394071384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c154af0 0x7fb70c0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 -- 172.21.15.122:0/1394071384 >> v2:172.21.15.169:3300/0 conn(0x7fb70c15abc0 msgr2=0x7fb70c072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 --2- 172.21.15.122:0/1394071384 >> v2:172.21.15.169:3300/0 conn(0x7fb70c15abc0 0x7fb70c072aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb7112ac640 1 -- 172.21.15.122:0/1394071384 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0ce190 con 0x7fb70c06d9e0 2024-03-06T07:26:54.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.578+0000 7fb70bfff640 1 --2- 172.21.15.122:0/1394071384 >> v2:172.21.15.169:3300/0 conn(0x7fb70c15abc0 0x7fb70c072aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:54.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.579+0000 7fb7112ac640 1 --2- 172.21.15.122:0/1394071384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c06d9e0 0x7fb70c06de50 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0ce360 tx=0x7fb70402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a816289bed324ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.579+0000 7fb70b7fe640 1 -- 172.21.15.122:0/1394071384 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70402ea80 con 0x7fb70c06d9e0 2024-03-06T07:26:54.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.579+0000 7fb70b7fe640 1 -- 172.21.15.122:0/1394071384 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70402ec20 con 0x7fb70c06d9e0 2024-03-06T07:26:54.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.579+0000 7fb70b7fe640 1 -- 172.21.15.122:0/1394071384 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704009600 con 0x7fb70c06d9e0 2024-03-06T07:26:54.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 -- 172.21.15.122:0/1394071384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c06d9e0 msgr2=0x7fb70c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 --2- 172.21.15.122:0/1394071384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c06d9e0 0x7fb70c06de50 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0ce360 tx=0x7fb70402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 -- 172.21.15.122:0/1394071384 shutdown_connections 2024-03-06T07:26:54.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 --2- 172.21.15.122:0/1394071384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c06d9e0 0x7fb70c06de50 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 --2- 172.21.15.122:0/1394071384 >> v2:172.21.15.169:3300/0 conn(0x7fb70c15abc0 0x7fb70c072aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 --2- 172.21.15.122:0/1394071384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c154af0 0x7fb70c0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 -- 172.21.15.122:0/1394071384 >> 172.21.15.122:0/1394071384 conn(0x7fb70c06b160 msgr2=0x7fb70c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 -- 172.21.15.122:0/1394071384 shutdown_connections 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.580+0000 7fb712d0f640 1 -- 172.21.15.122:0/1394071384 wait complete. 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.582+0000 7fb712d0f640 1 Processor -- start 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.582+0000 7fb712d0f640 1 -- start start 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb712d0f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c06d9e0 0x7fb70c155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb712d0f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb70c154af0 0x7fb70c152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb712d0f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 0x7fb70c1404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb712d0f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c15b3a0 con 0x7fb70c15abc0 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb712d0f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c15b0c0 con 0x7fb70c154af0 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb712d0f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c15b680 con 0x7fb70c06d9e0 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 0x7fb70c1404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 0x7fb70c1404d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54996/0 (socket says 172.21.15.122:54996) 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 -- 172.21.15.122:0/2185130683 learned_addr learned my addr 172.21.15.122:0/2185130683 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb710aab640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c06d9e0 0x7fb70c155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb70bfff640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb70c154af0 0x7fb70c152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 -- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c06d9e0 msgr2=0x7fb70c155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c06d9e0 0x7fb70c155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 -- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb70c154af0 msgr2=0x7fb70c152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb70c154af0 0x7fb70c152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.583+0000 7fb7112ac640 1 -- 172.21.15.122:0/2185130683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c1534b0 con 0x7fb70c15abc0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.584+0000 7fb7112ac640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 0x7fb70c1404d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fb704038040 tx=0x7fb704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.584+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704045020 con 0x7fb70c15abc0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.584+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb704009e30 con 0x7fb70c15abc0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.584+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70402ea80 con 0x7fb70c15abc0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.584+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c153790 con 0x7fb70c15abc0 2024-03-06T07:26:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.584+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c153d50 con 0x7fb70c15abc0 2024-03-06T07:26:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.585+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c069ee0 con 0x7fb70c15abc0 2024-03-06T07:26:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.585+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb704048020 con 0x7fb70c15abc0 2024-03-06T07:26:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.586+0000 7fb7097fa640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6f006b4d0 0x7fb6f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.586+0000 7fb710aab640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6f006b4d0 0x7fb6f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.586+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7040aec00 con 0x7fb70c15abc0 2024-03-06T07:26:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.586+0000 7fb710aab640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6f006b4d0 0x7fb6f006d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb70c152210 tx=0x7fb700008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.590+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7040b5080 con 0x7fb70c15abc0 2024-03-06T07:26:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.880+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb70c072370 con 0x7fb70c15abc0 2024-03-06T07:26:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.880+0000 7fb7097fa640 1 -- 172.21.15.122:0/2185130683 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v97) v1 ==== 80+0+124 (secure 0 0 0) 0x7fb70404f030 con 0x7fb70c15abc0 2024-03-06T07:26:54.883 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.169:6835/2187235474 2024-03-07T07:26:28.144746+0000 2024-03-06T07:26:54.883 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.169:6833/2187235474 2024-03-07T07:26:28.144746+0000 2024-03-06T07:26:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:listed 2 entries 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6f006b4d0 msgr2=0x7fb6f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6f006b4d0 0x7fb6f006d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb70c152210 tx=0x7fb700008040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 msgr2=0x7fb70c1404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 0x7fb70c1404d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fb704038040 tx=0x7fb704004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 shutdown_connections 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb6f006b4d0 0x7fb6f006d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb70c15abc0 0x7fb70c1404d0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb70c154af0 0x7fb70c152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 --2- 172.21.15.122:0/2185130683 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb70c06d9e0 0x7fb70c155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.882+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 >> 172.21.15.122:0/2185130683 conn(0x7fb70c06b160 msgr2=0x7fb70c0cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.883+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 shutdown_connections 2024-03-06T07:26:54.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:54.883+0000 7fb712d0f640 1 -- 172.21.15.122:0/2185130683 wait complete. 2024-03-06T07:26:54.895 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:26:54.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:54.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:54.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[68944]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:54.922 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:26:54.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:54.922 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:54.991 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:26:54.992 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:55.009 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:55.010 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:26:55.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:55.010 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:55.065 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:55.066 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:26:55.066 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:55.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:55.081 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:26:55.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:55.081 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:55.136 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:55.136 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:26:55.136 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:55.152 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:55.152 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:26:55.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:55.152 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:55.208 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:55.209 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:55.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.308+0000 7fb302855640 1 Processor -- start 2024-03-06T07:26:55.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.308+0000 7fb302855640 1 -- start start 2024-03-06T07:26:55.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb302855640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc0b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb302855640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0cd4d0 con 0x7fb2fc0b16b0 2024-03-06T07:26:55.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb302855640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0cd7b0 con 0x7fb2fc15f5e0 2024-03-06T07:26:55.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb302855640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cf920 con 0x7fb2fc0cec60 2024-03-06T07:26:55.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb300df2640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb2fc0b16b0 0x7fb2fc0cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59890/0 (socket says 172.21.15.122:59890) 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb300df2640 1 -- 172.21.15.122:0/3604389286 learned_addr learned my addr 172.21.15.122:0/3604389286 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb2fb7fe640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc0b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787494336 0 0) 0x7fb2fc0cd4d0 con 0x7fb2fc0b16b0 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.309+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e4003520 con 0x7fb2fc0b16b0 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655323352 0 0) 0x7fb2fc0cd7b0 con 0x7fb2fc15f5e0 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc0cd4d0 con 0x7fb2fc15f5e0 2024-03-06T07:26:55.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729272350 0 0) 0x7fb2e4003520 con 0x7fb2fc0b16b0 2024-03-06T07:26:55.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 msgr2=0x7fb2fc0b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc0b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 >> v1:172.21.15.169:6789/0 conn(0x7fb2fc15f5e0 legacy=0x7fb2fc0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0cda90 con 0x7fb2fc0b16b0 2024-03-06T07:26:55.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb2f40019f0 con 0x7fb2fc0b16b0 2024-03-06T07:26:55.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 >> v1:172.21.15.122:6789/0 conn(0x7fb2fc0b16b0 legacy=0x7fb2fc0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2e4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e4004120 0x7fb2e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2e4004d40 0x7fb2e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cf920 con 0x7fb2e4004120 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e400bb80 con 0x7fb2e4004d40 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e400bd80 con 0x7fb2fc0cec60 2024-03-06T07:26:55.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2e4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fb7fe640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e4004120 0x7fb2e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb300df2640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2e4004d40 0x7fb2e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fbfff640 1 -- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2e4004d40 msgr2=0x7fb2e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2e4004d40 0x7fb2e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fbfff640 1 -- 172.21.15.122:0/3604389286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e4004120 msgr2=0x7fb2e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e4004120 0x7fb2e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fbfff640 1 -- 172.21.15.122:0/3604389286 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e4003520 con 0x7fb2fc0cec60 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.310+0000 7fb2fb7fe640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e4004120 0x7fb2e4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.311+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2e4003b70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00018d0 tx=0x7fb2e4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76cb08dfbc3cf2d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.311+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0037070 con 0x7fb2fc0cec60 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.311+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f00052d0 con 0x7fb2fc0cec60 2024-03-06T07:26:55.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.311+0000 7fb2faffd640 1 -- 172.21.15.122:0/3604389286 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00014f0 con 0x7fb2fc0cec60 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb302855640 1 -- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 msgr2=0x7fb2e4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb302855640 1 --2- 172.21.15.122:0/3604389286 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2e4003b70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00018d0 tx=0x7fb2e4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb300df2640 1 -- 172.21.15.122:0/3604389286 reap_dead start 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb302855640 1 -- 172.21.15.122:0/3604389286 shutdown_connections 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb302855640 1 -- 172.21.15.122:0/3604389286 >> 172.21.15.122:0/3604389286 conn(0x7fb2fc0d1d50 msgr2=0x7fb2fc0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb302855640 1 -- 172.21.15.122:0/3604389286 shutdown_connections 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.312+0000 7fb302855640 1 -- 172.21.15.122:0/3604389286 wait complete. 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.313+0000 7fb302855640 1 Processor -- start 2024-03-06T07:26:55.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.313+0000 7fb302855640 1 -- start start 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb302855640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb302855640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 0x7fb2fc0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb302855640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc06ddf0 0x7fb2fc0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb302855640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cf270 con 0x7fb2fc06d970 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb302855640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cf0f0 con 0x7fb2fc06ddf0 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb302855640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0d2870 con 0x7fb2fc0cec60 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 0x7fb2fc0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 0x7fb2fc0afaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55020/0 (socket says 172.21.15.122:55020) 2024-03-06T07:26:55.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 -- 172.21.15.122:0/1593292164 learned_addr learned my addr 172.21.15.122:0/1593292164 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:55.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb300df2640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc06ddf0 0x7fb2fc0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 -- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 msgr2=0x7fb2fc150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 -- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc06ddf0 msgr2=0x7fb2fc0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc06ddf0 0x7fb2fc0affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb2fb7fe640 1 -- 172.21.15.122:0/1593292164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0b6010 con 0x7fb2fc06d970 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.314+0000 7fb300df2640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc06ddf0 0x7fb2fc0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.315+0000 7fb2fb7fe640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 0x7fb2fc0afaa0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec011ac0 tx=0x7fb2ec011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.315+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00fbb0 con 0x7fb2fc06d970 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.315+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec00fd50 con 0x7fb2fc06d970 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.315+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc0b62f0 con 0x7fb2fc06d970 2024-03-06T07:26:55.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.315+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec014660 con 0x7fb2fc06d970 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.315+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc0b68b0 con 0x7fb2fc06d970 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.316+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc068050 con 0x7fb2fc06d970 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.317+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2ec012040 con 0x7fb2fc06d970 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.317+0000 7fb2f8ff9640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2e806b4d0 0x7fb2e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.317+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2e806b4d0 0x7fb2e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.317+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb2ec08a880 con 0x7fb2fc06d970 2024-03-06T07:26:55.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.318+0000 7fb2fbfff640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2e806b4d0 0x7fb2e806d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0034000 tx=0x7fb2f0031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.322+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2ec019030 con 0x7fb2fc06d970 2024-03-06T07:26:55.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.537+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2fc0b7cd0 con 0x7fb2fc06d970 2024-03-06T07:26:55.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.538+0000 7fb2f8ff9640 1 -- 172.21.15.122:0/1593292164 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fb2ec059110 con 0x7fb2fc06d970 2024-03-06T07:26:55.540 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:55.540 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:55.540 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:55.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2e806b4d0 msgr2=0x7fb2e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2e806b4d0 0x7fb2e806d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0034000 tx=0x7fb2f0031000 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 msgr2=0x7fb2fc0afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 0x7fb2fc0afaa0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec011ac0 tx=0x7fb2ec011f80 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 shutdown_connections 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2e806b4d0 0x7fb2e806d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc06ddf0 0x7fb2fc0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2fc06d970 0x7fb2fc0afaa0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 --2- 172.21.15.122:0/1593292164 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2fc0cec60 0x7fb2fc150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 >> 172.21.15.122:0/1593292164 conn(0x7fb2fc0d1d50 msgr2=0x7fb2fc160520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 shutdown_connections 2024-03-06T07:26:55.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.540+0000 7fb302855640 1 -- 172.21.15.122:0/1593292164 wait complete. 2024-03-06T07:26:55.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:55.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.656+0000 7f9bb8e29640 1 Processor -- start 2024-03-06T07:26:55.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.656+0000 7f9bb8e29640 1 -- start start 2024-03-06T07:26:55.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb8e29640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9bb415e150 0x7f9bb40cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb8e29640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9bb40cfba0 0x7f9bb40d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb8e29640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb40d1ea0 con 0x7f9bb40d0e90 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb8e29640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb40d1300 con 0x7f9bb415e150 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb8e29640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb40d2ab0 con 0x7f9bb40cfba0 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9bb415e150 0x7f9bb40cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9bb415e150 0x7f9bb40cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55026/0 (socket says 172.21.15.122:55026) 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 -- 172.21.15.122:0/3403788111 learned_addr learned my addr 172.21.15.122:0/3403788111 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb1d9b640 1 --2- 172.21.15.122:0/3403788111 >> v2:172.21.15.169:3300/0 conn(0x7f9bb40cfba0 0x7f9bb40d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 -- 172.21.15.122:0/3403788111 >> v2:172.21.15.169:3300/0 conn(0x7f9bb40cfba0 msgr2=0x7f9bb40d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 --2- 172.21.15.122:0/3403788111 >> v2:172.21.15.169:3300/0 conn(0x7f9bb40cfba0 0x7f9bb40d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 -- 172.21.15.122:0/3403788111 >> v1:172.21.15.169:6789/0 conn(0x7f9bb40d0e90 legacy=0x7f9bb40b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb259c640 1 -- 172.21.15.122:0/3403788111 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40d2180 con 0x7f9bb415e150 2024-03-06T07:26:55.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.657+0000 7f9bb1d9b640 1 --2- 172.21.15.122:0/3403788111 >> v2:172.21.15.169:3300/0 conn(0x7f9bb40cfba0 0x7f9bb40d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.658+0000 7f9bb259c640 1 --2- 172.21.15.122:0/3403788111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb415e150 0x7f9bb40cf660 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40b6b50 tx=0x7f9ba402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe6533bed2f3565 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.658+0000 7f9bb159a640 1 -- 172.21.15.122:0/3403788111 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba402e070 con 0x7f9bb415e150 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.658+0000 7f9bb159a640 1 -- 172.21.15.122:0/3403788111 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba4037070 con 0x7f9bb415e150 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.658+0000 7f9bb159a640 1 -- 172.21.15.122:0/3403788111 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4031a00 con 0x7f9bb415e150 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 -- 172.21.15.122:0/3403788111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb415e150 msgr2=0x7f9bb40cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/3403788111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb415e150 0x7f9bb40cf660 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40b6b50 tx=0x7f9ba402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 -- 172.21.15.122:0/3403788111 shutdown_connections 2024-03-06T07:26:55.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/3403788111 >> v2:172.21.15.169:3300/0 conn(0x7f9bb40cfba0 0x7f9bb40d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/3403788111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb415e150 0x7f9bb40cf660 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 -- 172.21.15.122:0/3403788111 >> 172.21.15.122:0/3403788111 conn(0x7f9bb409d230 msgr2=0x7f9bb40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 -- 172.21.15.122:0/3403788111 shutdown_connections 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.659+0000 7f9bb8e29640 1 -- 172.21.15.122:0/3403788111 wait complete. 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.661+0000 7f9bb8e29640 1 Processor -- start 2024-03-06T07:26:55.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.661+0000 7f9bb8e29640 1 -- start start 2024-03-06T07:26:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb8e29640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bb40cfba0 0x7f9bb40c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb8e29640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 0x7f9bb414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb8e29640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bb415e150 0x7f9bb415c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb8e29640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40d2f40 con 0x7f9bb40d0e90 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb8e29640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40d2740 con 0x7f9bb415e150 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb8e29640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40d1b30 con 0x7f9bb40cfba0 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 0x7f9bb414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 0x7f9bb414bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55036/0 (socket says 172.21.15.122:55036) 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.662+0000 7f9bb1d9b640 1 -- 172.21.15.122:0/2065232215 learned_addr learned my addr 172.21.15.122:0/2065232215 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb259c640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bb40cfba0 0x7f9bb40c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb2d9d640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bb415e150 0x7f9bb415c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb1d9b640 1 -- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bb40cfba0 msgr2=0x7f9bb40c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb1d9b640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bb40cfba0 0x7f9bb40c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb1d9b640 1 -- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bb415e150 msgr2=0x7f9bb415c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb1d9b640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bb415e150 0x7f9bb415c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb1d9b640 1 -- 172.21.15.122:0/2065232215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40cbc20 con 0x7f9bb40d0e90 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb1d9b640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 0x7f9bb414bfc0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c820 tx=0x7f9ba800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8015070 con 0x7f9bb40d0e90 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba8007a30 con 0x7f9bb40d0e90 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb40cbf00 con 0x7f9bb40d0e90 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8007e00 con 0x7f9bb40d0e90 2024-03-06T07:26:55.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.663+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb40cc4c0 con 0x7f9bb40d0e90 2024-03-06T07:26:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.664+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb4069ee0 con 0x7f9bb40d0e90 2024-03-06T07:26:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.665+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ba8018070 con 0x7f9bb40d0e90 2024-03-06T07:26:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.665+0000 7f9ba2ffd640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.665+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9ba808a1b0 con 0x7f9bb40d0e90 2024-03-06T07:26:55.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.665+0000 7f9bb259c640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.666+0000 7f9bb259c640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4036000 tx=0x7f9ba403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.669+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ba8059060 con 0x7f9bb40d0e90 2024-03-06T07:26:55.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.846+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bb40d4670 con 0x7f9bb40d0e90 2024-03-06T07:26:55.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.847+0000 7f9ba2ffd640 1 -- 172.21.15.122:0/2065232215 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16739 (secure 0 0 0) 0x7f9ba8058960 con 0x7f9bb40d0e90 2024-03-06T07:26:55.849 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:55.850 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":97,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:28.153647+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:6835/2187235474":"2024-03-07T07:26:28.144746+0000","172.21.15.169:6833/2187235474":"2024-03-07T07:26:28.144746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:55.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b9406b4d0 msgr2=0x7f9b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4036000 tx=0x7f9ba403d040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 msgr2=0x7f9bb414bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 0x7f9bb414bfc0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c820 tx=0x7f9ba800cc20 comp rx=0 tx=0).stop 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 shutdown_connections 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bb415e150 0x7f9bb415c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb40d0e90 0x7f9bb414bfc0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 --2- 172.21.15.122:0/2065232215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9bb40cfba0 0x7f9bb40c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 >> 172.21.15.122:0/2065232215 conn(0x7f9bb409d230 msgr2=0x7f9bb40ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 shutdown_connections 2024-03-06T07:26:55.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.849+0000 7f9bb8e29640 1 -- 172.21.15.122:0/2065232215 wait complete. 2024-03-06T07:26:55.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:17.659781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:18.415605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:55.864 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:55.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.972+0000 7f7ecf6df640 1 Processor -- start 2024-03-06T07:26:55.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.973+0000 7f7ecf6df640 1 -- start start 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.973+0000 7f7ecf6df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec80d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.973+0000 7f7ecf6df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7ec80b9060 0x7f7ec80b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.973+0000 7f7ecf6df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80d1ea0 con 0x7f7ec815e150 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecf6df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec80d12f0 con 0x7f7ec80b9060 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecf6df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d2ab0 con 0x7f7ec80d0950 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7ec80b9060 0x7f7ec80b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7ec80b9060 0x7f7ec80b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55038/0 (socket says 172.21.15.122:55038) 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 -- 172.21.15.122:0/778248926 learned_addr learned my addr 172.21.15.122:0/778248926 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 -- 172.21.15.122:0/778248926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 msgr2=0x7f7ec80d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 --2- 172.21.15.122:0/778248926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec80d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 -- 172.21.15.122:0/778248926 >> v1:172.21.15.169:6789/0 conn(0x7f7ec815e150 legacy=0x7f7ec80cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.974+0000 7f7ecdc7c640 1 -- 172.21.15.122:0/778248926 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80d2180 con 0x7f7ec80b9060 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecdc7c640 1 --2- 172.21.15.122:0/778248926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80b94f0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009e50 tx=0x7f7ec402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc6b989a06611cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ebffff640 1 -- 172.21.15.122:0/778248926 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec402c9e0 con 0x7f7ec80b9060 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ebffff640 1 -- 172.21.15.122:0/778248926 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec402cb80 con 0x7f7ec80b9060 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ebffff640 1 -- 172.21.15.122:0/778248926 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40343f0 con 0x7f7ec80b9060 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 -- 172.21.15.122:0/778248926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 msgr2=0x7f7ec80b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/778248926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80b94f0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009e50 tx=0x7f7ec402be40 comp rx=0 tx=0).stop 2024-03-06T07:26:55.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 -- 172.21.15.122:0/778248926 shutdown_connections 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/778248926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80b94f0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/778248926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec80d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 -- 172.21.15.122:0/778248926 >> 172.21.15.122:0/778248926 conn(0x7f7ec809d230 msgr2=0x7f7ec80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.975+0000 7f7ecf6df640 1 -- 172.21.15.122:0/778248926 shutdown_connections 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.976+0000 7f7ecf6df640 1 -- 172.21.15.122:0/778248926 wait complete. 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.977+0000 7f7ecf6df640 1 Processor -- start 2024-03-06T07:26:55.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.977+0000 7f7ecf6df640 1 -- start start 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecf6df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecf6df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec81e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecf6df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ec815e150 0x7f7ec8149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecf6df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d2f40 con 0x7f7ec80b9060 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecf6df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d2740 con 0x7f7ec815e150 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecf6df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d1b30 con 0x7f7ec80d0950 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecd47b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecd47b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55052/0 (socket says 172.21.15.122:55052) 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.978+0000 7f7ecd47b640 1 -- 172.21.15.122:0/2601359917 learned_addr learned my addr 172.21.15.122:0/2601359917 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7eccc7a640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec81e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecdc7c640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ec815e150 0x7f7ec8149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecd47b640 1 -- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 msgr2=0x7f7ec81e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecd47b640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec81e9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecd47b640 1 -- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ec815e150 msgr2=0x7f7ec8149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecd47b640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ec815e150 0x7f7ec8149ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecd47b640 1 -- 172.21.15.122:0/2601359917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec815f780 con 0x7f7ec80b9060 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecd47b640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80c77e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000b3f0 tx=0x7f7ec000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0015020 con 0x7f7ec80b9060 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec0011830 con 0x7f7ec80b9060 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0011c20 con 0x7f7ec80b9060 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec815fa60 con 0x7f7ec80b9060 2024-03-06T07:26:55.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.979+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec80c9f80 con 0x7f7ec80b9060 2024-03-06T07:26:55.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.980+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8069ee0 con 0x7f7ec80b9060 2024-03-06T07:26:55.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.980+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ec0018020 con 0x7f7ec80b9060 2024-03-06T07:26:55.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.981+0000 7f7ebdffb640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7eac06b4d0 0x7f7eac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.981+0000 7f7eccc7a640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7eac06b4d0 0x7f7eac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.981+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7ec0089f20 con 0x7f7ec80b9060 2024-03-06T07:26:55.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.981+0000 7f7eccc7a640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7eac06b4d0 0x7f7eac06d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7ec806c640 tx=0x7f7eb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:55.985+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ec001f030 con 0x7f7ec80b9060 2024-03-06T07:26:56.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.162+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ec80d4670 con 0x7f7ec80b9060 2024-03-06T07:26:56.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.162+0000 7f7ebdffb640 1 -- 172.21.15.122:0/2601359917 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ec00172c0 con 0x7f7ec80b9060 2024-03-06T07:26:56.164 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7eac06b4d0 msgr2=0x7f7eac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7eac06b4d0 0x7f7eac06d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7ec806c640 tx=0x7f7eb8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 msgr2=0x7f7ec80c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80c77e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000b3f0 tx=0x7f7ec000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 shutdown_connections 2024-03-06T07:26:56.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7eac06b4d0 0x7f7eac06d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ec815e150 0x7f7ec8149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7ec80d0950 0x7f7ec81e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 --2- 172.21.15.122:0/2601359917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec80b9060 0x7f7ec80c77e0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 >> 172.21.15.122:0/2601359917 conn(0x7f7ec809d230 msgr2=0x7f7ec80b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 shutdown_connections 2024-03-06T07:26:56.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.165+0000 7f7ecf6df640 1 -- 172.21.15.122:0/2601359917 wait complete. 2024-03-06T07:26:56.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:56.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.294+0000 7ff5a119a640 1 Processor -- start 2024-03-06T07:26:56.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.294+0000 7ff5a119a640 1 -- start start 2024-03-06T07:26:56.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff5a119a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff59c0d0db0 0x7ff59c0b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff5a119a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0d1ea0 con 0x7ff59c15e150 2024-03-06T07:26:56.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff5a119a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0d2180 con 0x7ff59c0d0950 2024-03-06T07:26:56.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff5a119a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59c0d2ab0 con 0x7ff59c0d0db0 2024-03-06T07:26:56.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff59b59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff59c0d0db0 0x7ff59c0b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff59b59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff59c0d0db0 0x7ff59c0b9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55056/0 (socket says 172.21.15.122:55056) 2024-03-06T07:26:56.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff59b59e640 1 -- 172.21.15.122:0/2415586298 learned_addr learned my addr 172.21.15.122:0/2415586298 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:56.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff59b59e640 1 -- 172.21.15.122:0/2415586298 >> v1:172.21.15.169:6789/0 conn(0x7ff59c0d0950 legacy=0x7ff59c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff59b59e640 1 -- 172.21.15.122:0/2415586298 >> v1:172.21.15.122:6789/0 conn(0x7ff59c15e150 legacy=0x7ff59c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.295+0000 7ff59b59e640 1 -- 172.21.15.122:0/2415586298 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0d2460 con 0x7ff59c0d0db0 2024-03-06T07:26:56.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.296+0000 7ff59b59e640 1 --2- 172.21.15.122:0/2415586298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c0d0db0 0x7ff59c0b9770 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009e50 tx=0x7ff58c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbb4a499b2b2bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.296+0000 7ff599d9b640 1 -- 172.21.15.122:0/2415586298 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c02ca80 con 0x7ff59c0d0db0 2024-03-06T07:26:56.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.296+0000 7ff599d9b640 1 -- 172.21.15.122:0/2415586298 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c02cc20 con 0x7ff59c0d0db0 2024-03-06T07:26:56.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.296+0000 7ff599d9b640 1 -- 172.21.15.122:0/2415586298 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0343f0 con 0x7ff59c0d0db0 2024-03-06T07:26:56.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 -- 172.21.15.122:0/2415586298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c0d0db0 msgr2=0x7ff59c0b9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 --2- 172.21.15.122:0/2415586298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c0d0db0 0x7ff59c0b9770 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009e50 tx=0x7ff58c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 -- 172.21.15.122:0/2415586298 shutdown_connections 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 --2- 172.21.15.122:0/2415586298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c0d0db0 0x7ff59c0b9770 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 -- 172.21.15.122:0/2415586298 >> 172.21.15.122:0/2415586298 conn(0x7ff59c09d230 msgr2=0x7ff59c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 -- 172.21.15.122:0/2415586298 shutdown_connections 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.297+0000 7ff5a119a640 1 -- 172.21.15.122:0/2415586298 wait complete. 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.298+0000 7ff5a119a640 1 Processor -- start 2024-03-06T07:26:56.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.298+0000 7ff5a119a640 1 -- start start 2024-03-06T07:26:56.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff5a119a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff59c0d0950 0x7ff59c151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff5a119a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff59c0d0db0 0x7ff59c0cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff5a119a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 0x7ff59c0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff5a119a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0d2f40 con 0x7ff59c15e150 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff5a119a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0d1220 con 0x7ff59c0d0950 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff5a119a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0b7dc0 con 0x7ff59c0d0db0 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff59b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 0x7ff59c0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff59b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 0x7ff59c0cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55060/0 (socket says 172.21.15.122:55060) 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff59b59e640 1 -- 172.21.15.122:0/1397183955 learned_addr learned my addr 172.21.15.122:0/1397183955 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff59ad9d640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff59c0d0950 0x7ff59c151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.299+0000 7ff59a59c640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff59c0d0db0 0x7ff59c0cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59b59e640 1 -- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff59c0d0db0 msgr2=0x7ff59c0cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59b59e640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff59c0d0db0 0x7ff59c0cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59b59e640 1 -- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff59c0d0950 msgr2=0x7ff59c151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59b59e640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff59c0d0950 0x7ff59c151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59b59e640 1 -- 172.21.15.122:0/1397183955 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c1601e0 con 0x7ff59c15e150 2024-03-06T07:26:56.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59ad9d640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff59c0d0950 0x7ff59c151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:56.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff59b59e640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 0x7ff59c0cd530 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff58c036040 tx=0x7ff58c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c043020 con 0x7ff59c15e150 2024-03-06T07:26:56.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c03e450 con 0x7ff59c15e150 2024-03-06T07:26:56.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c1604c0 con 0x7ff59c15e150 2024-03-06T07:26:56.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c03e860 con 0x7ff59c15e150 2024-03-06T07:26:56.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.300+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c1518a0 con 0x7ff59c15e150 2024-03-06T07:26:56.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.301+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c069ee0 con 0x7ff59c15e150 2024-03-06T07:26:56.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.301+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff58c046020 con 0x7ff59c15e150 2024-03-06T07:26:56.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.302+0000 7ff5837fe640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58806b4d0 0x7ff58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.302+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff58c0acaf0 con 0x7ff59c15e150 2024-03-06T07:26:56.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.302+0000 7ff59ad9d640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58806b4d0 0x7ff58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.302+0000 7ff59ad9d640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58806b4d0 0x7ff58806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff5940015e0 tx=0x7ff594009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.306+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff58c04d030 con 0x7ff59c15e150 2024-03-06T07:26:56.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.561+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff59c0d4670 con 0x7ff59c15e150 2024-03-06T07:26:56.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.562+0000 7ff5837fe640 1 -- 172.21.15.122:0/1397183955 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff58c07b9a0 con 0x7ff59c15e150 2024-03-06T07:26:56.564 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:26:56.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58806b4d0 msgr2=0x7ff58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58806b4d0 0x7ff58806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff5940015e0 tx=0x7ff594009040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 msgr2=0x7ff59c0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 0x7ff59c0cd530 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff58c036040 tx=0x7ff58c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 shutdown_connections 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff58806b4d0 0x7ff58806d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59c15e150 0x7ff59c0cd530 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff59c0d0db0 0x7ff59c0cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 --2- 172.21.15.122:0/1397183955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff59c0d0950 0x7ff59c151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 >> 172.21.15.122:0/1397183955 conn(0x7ff59c09d230 msgr2=0x7ff59c0b9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 shutdown_connections 2024-03-06T07:26:56.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.564+0000 7ff5a119a640 1 -- 172.21.15.122:0/1397183955 wait complete. 2024-03-06T07:26:56.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:56.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.709+0000 7ff286237640 1 Processor -- start 2024-03-06T07:26:56.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.709+0000 7ff286237640 1 -- start start 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.709+0000 7ff286237640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 0x7ff2800d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.709+0000 7ff286237640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800d1ea0 con 0x7ff2800b9060 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.709+0000 7ff286237640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800d2180 con 0x7ff28015e150 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.709+0000 7ff286237640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800d12f0 con 0x7ff2800d0950 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff2800b9060 0x7ff2800b94f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59914/0 (socket says 172.21.15.122:59914) 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27ffff640 1 -- 172.21.15.122:0/1717501425 learned_addr learned my addr 172.21.15.122:0/1717501425 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119617141 0 0) 0x7ff2800d1ea0 con 0x7ff2800b9060 2024-03-06T07:26:56.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff268003520 con 0x7ff2800b9060 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27effd640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 0x7ff2800d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4212813476 0 0) 0x7ff268003520 con 0x7ff2800b9060 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 msgr2=0x7ff2800d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 0x7ff2800d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 >> v1:172.21.15.169:6789/0 conn(0x7ff28015e150 legacy=0x7ff2800cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800d2460 con 0x7ff2800b9060 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff2780019f0 con 0x7ff2800b9060 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 >> v1:172.21.15.122:6789/0 conn(0x7ff2800b9060 legacy=0x7ff2800b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2680039e0 0x7ff268003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 0x7ff268004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff268004d40 0x7ff2680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26800bac0 con 0x7ff2680039e0 2024-03-06T07:26:56.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26800bc70 con 0x7ff268004d40 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26800be70 con 0x7ff2800d0950 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2680039e0 0x7ff268003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27effd640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 0x7ff268004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.710+0000 7ff27ffff640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff268004d40 0x7ff2680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27f7fe640 1 -- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 msgr2=0x7ff268004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2800d0950 0x7ff268004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27f7fe640 1 -- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff268004d40 msgr2=0x7ff2680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff268004d40 0x7ff2680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27f7fe640 1 -- 172.21.15.122:0/1717501425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268003520 con 0x7ff2680039e0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27ffff640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff268004d40 0x7ff2680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.711+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2680039e0 0x7ff268003e00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7ff270001330 tx=0x7ff268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c851db79cb08091 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270001370 con 0x7ff2680039e0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2700052d0 con 0x7ff2680039e0 2024-03-06T07:26:56.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff27e7fc640 1 -- 172.21.15.122:0/1717501425 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270034ae0 con 0x7ff2680039e0 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff286237640 1 -- 172.21.15.122:0/1717501425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2680039e0 msgr2=0x7ff268003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff286237640 1 --2- 172.21.15.122:0/1717501425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2680039e0 0x7ff268003e00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7ff270001330 tx=0x7ff268005830 comp rx=0 tx=0).stop 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff27ffff640 1 -- 172.21.15.122:0/1717501425 reap_dead start 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff286237640 1 -- 172.21.15.122:0/1717501425 shutdown_connections 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff286237640 1 -- 172.21.15.122:0/1717501425 >> 172.21.15.122:0/1717501425 conn(0x7ff28009d230 msgr2=0x7ff2800d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff286237640 1 -- 172.21.15.122:0/1717501425 shutdown_connections 2024-03-06T07:26:56.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.712+0000 7ff286237640 1 -- 172.21.15.122:0/1717501425 wait complete. 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.713+0000 7ff286237640 1 Processor -- start 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.713+0000 7ff286237640 1 -- start start 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff286237640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2680039e0 0x7ff2800cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff286237640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 0x7ff28015b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff286237640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2800cf970 0x7ff2800cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff286237640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800d2f40 con 0x7ff2800c77e0 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff286237640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800d2ab0 con 0x7ff2800cf970 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff286237640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800b7dc0 con 0x7ff2680039e0 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff27effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 0x7ff28015b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff27effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 0x7ff28015b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55074/0 (socket says 172.21.15.122:55074) 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff27effd640 1 -- 172.21.15.122:0/2354378781 learned_addr learned my addr 172.21.15.122:0/2354378781 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2680039e0 0x7ff2800cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff27ffff640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2800cf970 0x7ff2800cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.714+0000 7ff27effd640 1 -- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2680039e0 msgr2=0x7ff2800cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff27effd640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2680039e0 0x7ff2800cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff27effd640 1 -- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2800cf970 msgr2=0x7ff2800cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff27effd640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2800cf970 0x7ff2800cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff27effd640 1 -- 172.21.15.122:0/2354378781 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800c7c60 con 0x7ff2800c77e0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff27effd640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 0x7ff28015b350 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7ff2740119d0 tx=0x7ff274011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27400fb50 con 0x7ff2800c77e0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27400fcf0 con 0x7ff2800c77e0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2800c7ee0 con 0x7ff2800c77e0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27400fea0 con 0x7ff2800c77e0 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.715+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2800c84a0 con 0x7ff2800c77e0 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.716+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff280069ee0 con 0x7ff2800c77e0 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.716+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff274012040 con 0x7ff2800c77e0 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.717+0000 7ff267fff640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff26c06b4d0 0x7ff26c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.717+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff27408a820 con 0x7ff2800c77e0 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.717+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff26c06b4d0 0x7ff26c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.717+0000 7ff27f7fe640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff26c06b4d0 0x7ff26c06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff27003c000 tx=0x7ff270038000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.725+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2800c7c60 con 0x7ff2800c77e0 2024-03-06T07:26:56.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.900+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2800d4670 con 0x7ff2800c77e0 2024-03-06T07:26:56.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.901+0000 7ff267fff640 1 -- 172.21.15.122:0/2354378781 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff274019030 con 0x7ff2800c77e0 2024-03-06T07:26:56.903 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:26:56.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff26c06b4d0 msgr2=0x7ff26c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff26c06b4d0 0x7ff26c06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff27003c000 tx=0x7ff270038000 comp rx=0 tx=0).stop 2024-03-06T07:26:56.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 msgr2=0x7ff28015b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 0x7ff28015b350 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7ff2740119d0 tx=0x7ff274011e90 comp rx=0 tx=0).stop 2024-03-06T07:26:56.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 shutdown_connections 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff26c06b4d0 0x7ff26c06d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2800cf970 0x7ff2800cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2800c77e0 0x7ff28015b350 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 --2- 172.21.15.122:0/2354378781 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2680039e0 0x7ff2800cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 >> 172.21.15.122:0/2354378781 conn(0x7ff28009d230 msgr2=0x7ff280160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 shutdown_connections 2024-03-06T07:26:56.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:56.903+0000 7ff286237640 1 -- 172.21.15.122:0/2354378781 wait complete. 2024-03-06T07:26:56.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.019+0000 7f80250fa640 1 Processor -- start 2024-03-06T07:26:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.019+0000 7f80250fa640 1 -- start start 2024-03-06T07:26:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.020+0000 7f80250fa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 0x7f80200b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.020+0000 7f80250fa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200d1ea0 con 0x7f80200d0950 2024-03-06T07:26:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.020+0000 7f80250fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200d2180 con 0x7f802015e150 2024-03-06T07:26:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.020+0000 7f80250fa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80200d2ab0 con 0x7f80200d0db0 2024-03-06T07:26:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.020+0000 7f801659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f80200d0950 0x7f80200b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59924/0 (socket says 172.21.15.122:59924) 2024-03-06T07:26:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.020+0000 7f801659c640 1 -- 172.21.15.122:0/3557284 learned_addr learned my addr 172.21.15.122:0/3557284 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801f59e640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 0x7f80200b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659062153 0 0) 0x7f80200d1ea0 con 0x7f80200d0950 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f800c003520 con 0x7f80200d0950 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309607445 0 0) 0x7f800c003520 con 0x7f80200d0950 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 msgr2=0x7f80200b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 0x7f80200b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 >> v1:172.21.15.169:6789/0 conn(0x7f802015e150 legacy=0x7f80200cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80200d2460 con 0x7f80200d0950 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f8008001670 con 0x7f80200d0950 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 >> v1:172.21.15.122:6789/0 conn(0x7f80200d0950 legacy=0x7f80200b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f800c0039e0 0x7f800c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c004800 0x7f800c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 0x7f800c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200d2ab0 con 0x7f800c0039e0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c00baf0 con 0x7f800c004800 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c00bcf0 con 0x7f80200d0db0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801ed9d640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f800c0039e0 0x7f800c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801659c640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c004800 0x7f800c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801f59e640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 0x7f800c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801ed9d640 1 -- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 msgr2=0x7f800c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801ed9d640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200d0db0 0x7f800c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801ed9d640 1 -- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c004800 msgr2=0x7f800c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801ed9d640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c004800 0x7f800c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.021+0000 7f801ed9d640 1 -- 172.21.15.122:0/3557284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c003520 con 0x7f800c0039e0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.022+0000 7f801ed9d640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f800c0039e0 0x7f800c003e00 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8010001750 tx=0x7f800c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4914745d9776cd16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.022+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010001370 con 0x7f800c0039e0 2024-03-06T07:26:57.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.022+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80100052d0 con 0x7f800c0039e0 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.022+0000 7f801e59c640 1 -- 172.21.15.122:0/3557284 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010003830 con 0x7f800c0039e0 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f80250fa640 1 -- 172.21.15.122:0/3557284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f800c0039e0 msgr2=0x7f800c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f80250fa640 1 --2- 172.21.15.122:0/3557284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f800c0039e0 0x7f800c003e00 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8010001750 tx=0x7f800c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f801f59e640 1 -- 172.21.15.122:0/3557284 reap_dead start 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f80250fa640 1 -- 172.21.15.122:0/3557284 shutdown_connections 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f80250fa640 1 -- 172.21.15.122:0/3557284 >> 172.21.15.122:0/3557284 conn(0x7f802009d230 msgr2=0x7f80200d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f80250fa640 1 -- 172.21.15.122:0/3557284 shutdown_connections 2024-03-06T07:26:57.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.023+0000 7f80250fa640 1 -- 172.21.15.122:0/3557284 wait complete. 2024-03-06T07:26:57.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.024+0000 7f80250fa640 1 Processor -- start 2024-03-06T07:26:57.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.024+0000 7f80250fa640 1 -- start start 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f80250fa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c0039e0 0x7f80200ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f80250fa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 0x7f80200c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f80250fa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200c7b50 0x7f8020160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f80250fa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200d2f40 con 0x7f80201549c0 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f80250fa640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200d1240 con 0x7f800c0039e0 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f80250fa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80200b7dc0 con 0x7f80200c7b50 2024-03-06T07:26:57.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 0x7f80200c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 0x7f80200c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55096/0 (socket says 172.21.15.122:55096) 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801659c640 1 -- 172.21.15.122:0/1981078024 learned_addr learned my addr 172.21.15.122:0/1981078024 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801ed9d640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c0039e0 0x7f80200ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801f59e640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200c7b50 0x7f8020160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801659c640 1 -- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200c7b50 msgr2=0x7f8020160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.025+0000 7f801659c640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200c7b50 0x7f8020160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f801659c640 1 -- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c0039e0 msgr2=0x7f80200ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f801659c640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c0039e0 0x7f80200ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f801659c640 1 -- 172.21.15.122:0/1981078024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802015f7b0 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f801659c640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 0x7f80200c7610 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f80080072c0 tx=0x7f8008007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800801a070 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800800f960 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80200ce050 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800800fd30 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.026+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802015fd30 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.027+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8020069ee0 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.027+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f800801d070 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.028+0000 7f8017fff640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800006b4d0 0x7f800006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.028+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8008089b00 con 0x7f80201549c0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.028+0000 7f801ed9d640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800006b4d0 0x7f800006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.028+0000 7f801ed9d640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800006b4d0 0x7f800006d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8010033000 tx=0x7f8010009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.032+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8008058240 con 0x7f80201549c0 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.249+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80200d4670 con 0x7f80201549c0 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.250+0000 7f8017fff640 1 -- 172.21.15.122:0/1981078024 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f800805c2e0 con 0x7f80201549c0 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:57.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800006b4d0 msgr2=0x7f800006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800006b4d0 0x7f800006d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8010033000 tx=0x7f8010009030 comp rx=0 tx=0).stop 2024-03-06T07:26:57.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 msgr2=0x7f80200c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 0x7f80200c7610 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f80080072c0 tx=0x7f8008007780 comp rx=0 tx=0).stop 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 shutdown_connections 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800006b4d0 0x7f800006d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80200c7b50 0x7f8020160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80201549c0 0x7f80200c7610 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 --2- 172.21.15.122:0/1981078024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f800c0039e0 0x7f80200ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 >> 172.21.15.122:0/1981078024 conn(0x7f802009d230 msgr2=0x7f80200b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 shutdown_connections 2024-03-06T07:26:57.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.252+0000 7f80250fa640 1 -- 172.21.15.122:0/1981078024 wait complete. 2024-03-06T07:26:57.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:57.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.364+0000 7f06a051e640 1 Processor -- start 2024-03-06T07:26:57.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.364+0000 7f06a051e640 1 -- start start 2024-03-06T07:26:57.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.365+0000 7f06a051e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06980cd690 0x7f0698162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.365+0000 7f06a051e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0698162c70 con 0x7f06980b2da0 2024-03-06T07:26:57.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.365+0000 7f06a051e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069806cd80 con 0x7f0698161670 2024-03-06T07:26:57.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.365+0000 7f06a051e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06980b3590 con 0x7f06980cd690 2024-03-06T07:26:57.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.365+0000 7f069dab9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06980cd690 0x7f0698162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.366+0000 7f069dab9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06980cd690 0x7f0698162570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55108/0 (socket says 172.21.15.122:55108) 2024-03-06T07:26:57.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.366+0000 7f069dab9640 1 -- 172.21.15.122:0/630625978 learned_addr learned my addr 172.21.15.122:0/630625978 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:57.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.366+0000 7f069dab9640 1 -- 172.21.15.122:0/630625978 >> v1:172.21.15.169:6789/0 conn(0x7f0698161670 legacy=0x7f0698161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.366+0000 7f069dab9640 1 -- 172.21.15.122:0/630625978 >> v1:172.21.15.122:6789/0 conn(0x7f06980b2da0 legacy=0x7f06980cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.366+0000 7f069dab9640 1 -- 172.21.15.122:0/630625978 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069806d060 con 0x7f06980cd690 2024-03-06T07:26:57.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f069dab9640 1 --2- 172.21.15.122:0/630625978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980cd690 0x7f0698162570 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f06880098a0 tx=0x7f068802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad147d99847266e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f069d2b8640 1 -- 172.21.15.122:0/630625978 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688034070 con 0x7f06980cd690 2024-03-06T07:26:57.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f069d2b8640 1 -- 172.21.15.122:0/630625978 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688030840 con 0x7f06980cd690 2024-03-06T07:26:57.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f069d2b8640 1 -- 172.21.15.122:0/630625978 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688030c10 con 0x7f06980cd690 2024-03-06T07:26:57.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 -- 172.21.15.122:0/630625978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980cd690 msgr2=0x7f0698162570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 --2- 172.21.15.122:0/630625978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980cd690 0x7f0698162570 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f06880098a0 tx=0x7f068802b980 comp rx=0 tx=0).stop 2024-03-06T07:26:57.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 -- 172.21.15.122:0/630625978 shutdown_connections 2024-03-06T07:26:57.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 --2- 172.21.15.122:0/630625978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980cd690 0x7f0698162570 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 -- 172.21.15.122:0/630625978 >> 172.21.15.122:0/630625978 conn(0x7f06980d1d50 msgr2=0x7f06980d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 -- 172.21.15.122:0/630625978 shutdown_connections 2024-03-06T07:26:57.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.367+0000 7f06a051e640 1 -- 172.21.15.122:0/630625978 wait complete. 2024-03-06T07:26:57.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 Processor -- start 2024-03-06T07:26:57.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 -- start start 2024-03-06T07:26:57.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 0x7f0698152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06980cd690 0x7f069806fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0698161670 0x7f0698070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069806d4c0 con 0x7f06980b2da0 2024-03-06T07:26:57.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069806d340 con 0x7f0698161670 2024-03-06T07:26:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f06a051e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0698161a50 con 0x7f06980cd690 2024-03-06T07:26:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f069e2ba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 0x7f0698152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f069e2ba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 0x7f0698152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55120/0 (socket says 172.21.15.122:55120) 2024-03-06T07:26:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.369+0000 7f069e2ba640 1 -- 172.21.15.122:0/3201468706 learned_addr learned my addr 172.21.15.122:0/3201468706 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069dab9640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06980cd690 0x7f069806fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069eabb640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0698161670 0x7f0698070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069e2ba640 1 -- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06980cd690 msgr2=0x7f069806fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069e2ba640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06980cd690 0x7f069806fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069e2ba640 1 -- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0698161670 msgr2=0x7f0698070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069e2ba640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0698161670 0x7f0698070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069e2ba640 1 -- 172.21.15.122:0/3201468706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980ca140 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069dab9640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06980cd690 0x7f069806fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069eabb640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0698161670 0x7f0698070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f069e2ba640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 0x7f0698152260 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f069400c410 tx=0x7f069400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694017070 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06940035f0 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980ca420 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06940039e0 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.370+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06980ca9e0 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.371+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0698068050 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.372+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0694003b80 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.372+0000 7f068effd640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f067c06b500 0x7f067c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.372+0000 7f069dab9640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f067c06b500 0x7f067c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.372+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f06940894f0 con 0x7f06980b2da0 2024-03-06T07:26:57.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.373+0000 7f069dab9640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f067c06b500 0x7f067c06d9c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0688036000 tx=0x7f068803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.381+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0694058320 con 0x7f06980b2da0 2024-03-06T07:26:57.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.559+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06980b7cd0 con 0x7f06980b2da0 2024-03-06T07:26:57.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.560+0000 7f068effd640 1 -- 172.21.15.122:0/3201468706 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16739 (secure 0 0 0) 0x7f0694057c20 con 0x7f06980b2da0 2024-03-06T07:26:57.562 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:57.563 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":97,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:28.153647+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:6835/2187235474":"2024-03-07T07:26:28.144746+0000","172.21.15.169:6833/2187235474":"2024-03-07T07:26:28.144746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f067c06b500 msgr2=0x7f067c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f067c06b500 0x7f067c06d9c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0688036000 tx=0x7f068803b040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 msgr2=0x7f0698152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 0x7f0698152260 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f069400c410 tx=0x7f069400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 shutdown_connections 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f067c06b500 0x7f067c06d9c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0698161670 0x7f0698070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06980cd690 0x7f069806fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 --2- 172.21.15.122:0/3201468706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06980b2da0 0x7f0698152260 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 >> 172.21.15.122:0/3201468706 conn(0x7f06980d1d50 msgr2=0x7f06980d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 shutdown_connections 2024-03-06T07:26:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.562+0000 7f06a051e640 1 -- 172.21.15.122:0/3201468706 wait complete. 2024-03-06T07:26:57.576 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:57.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:57.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.679+0000 7f720310c640 1 Processor -- start 2024-03-06T07:26:57.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.679+0000 7f720310c640 1 -- start start 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f720310c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f71fc15f5e0 0x7f71fc0b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f720310c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71fc0cec60 0x7f71fc0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f720310c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc0cd4d0 con 0x7f71fc0b16b0 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f720310c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71fc0cf920 con 0x7f71fc0cec60 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f720310c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71fc0cd7b0 con 0x7f71fc15f5e0 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71fc0cec60 0x7f71fc0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71fc0cec60 0x7f71fc0cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55126/0 (socket says 172.21.15.122:55126) 2024-03-06T07:26:57.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 -- 172.21.15.122:0/815681781 learned_addr learned my addr 172.21.15.122:0/815681781 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 -- 172.21.15.122:0/815681781 >> v2:172.21.15.169:3300/0 conn(0x7f71fc15f5e0 msgr2=0x7f71fc0b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 --2- 172.21.15.122:0/815681781 >> v2:172.21.15.169:3300/0 conn(0x7f71fc15f5e0 0x7f71fc0b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 -- 172.21.15.122:0/815681781 >> v1:172.21.15.169:6789/0 conn(0x7f71fc0b16b0 legacy=0x7f71fc0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.680+0000 7f72016a9640 1 -- 172.21.15.122:0/815681781 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc0cda90 con 0x7f71fc0cec60 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.681+0000 7f72016a9640 1 --2- 172.21.15.122:0/815681781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0cfdb0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0cf490 tx=0x7f71e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd339b7a997e76d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.681+0000 7f71fb7fe640 1 -- 172.21.15.122:0/815681781 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e402e070 con 0x7f71fc0cec60 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.681+0000 7f71fb7fe640 1 -- 172.21.15.122:0/815681781 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4037070 con 0x7f71fc0cec60 2024-03-06T07:26:57.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.681+0000 7f71fb7fe640 1 -- 172.21.15.122:0/815681781 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4031a00 con 0x7f71fc0cec60 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 -- 172.21.15.122:0/815681781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 msgr2=0x7f71fc0cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 --2- 172.21.15.122:0/815681781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0cfdb0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0cf490 tx=0x7f71e402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 -- 172.21.15.122:0/815681781 shutdown_connections 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 --2- 172.21.15.122:0/815681781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0cfdb0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 --2- 172.21.15.122:0/815681781 >> v2:172.21.15.169:3300/0 conn(0x7f71fc15f5e0 0x7f71fc0b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 -- 172.21.15.122:0/815681781 >> 172.21.15.122:0/815681781 conn(0x7f71fc0d1d50 msgr2=0x7f71fc0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 -- 172.21.15.122:0/815681781 shutdown_connections 2024-03-06T07:26:57.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.682+0000 7f720310c640 1 -- 172.21.15.122:0/815681781 wait complete. 2024-03-06T07:26:57.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.683+0000 7f720310c640 1 Processor -- start 2024-03-06T07:26:57.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.683+0000 7f720310c640 1 -- start start 2024-03-06T07:26:57.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f720310c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71fc0b16b0 0x7f71fc158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f720310c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f720310c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71fc15f5e0 0x7f71fc15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f720310c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc0cf270 con 0x7f71fc0cec60 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f720310c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc0cf0f0 con 0x7f71fc15f5e0 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f720310c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc0d2870 con 0x7f71fc0b16b0 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55138/0 (socket says 172.21.15.122:55138) 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 -- 172.21.15.122:0/629747233 learned_addr learned my addr 172.21.15.122:0/629747233 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f7200ea8640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71fc0b16b0 0x7f71fc158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f72016a9640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71fc15f5e0 0x7f71fc15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 -- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71fc0b16b0 msgr2=0x7f71fc158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71fc0b16b0 0x7f71fc158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 -- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71fc15f5e0 msgr2=0x7f71fc15ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71fc15f5e0 0x7f71fc15ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 -- 172.21.15.122:0/629747233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc0b7880 con 0x7f71fc0cec60 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.684+0000 7f71fbfff640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0b6e30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00c7f0 tx=0x7f71ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.685+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec017070 con 0x7f71fc0cec60 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.685+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec003da0 con 0x7f71fc0cec60 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.685+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec0052b0 con 0x7f71fc0cec60 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.685+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71fc1f5d00 con 0x7f71fc0cec60 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.685+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71fc158660 con 0x7f71fc0cec60 2024-03-06T07:26:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.686+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71fc068050 con 0x7f71fc0cec60 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.686+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71ec01a070 con 0x7f71fc0cec60 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.687+0000 7f71f97fa640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.687+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f71ec089780 con 0x7f71fc0cec60 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.687+0000 7f7200ea8640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.687+0000 7f7200ea8640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f71dc06b4d0 0x7f71dc06d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f71f400a770 tx=0x7f71f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.691+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71ec057fe0 con 0x7f71fc0cec60 2024-03-06T07:26:57.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.916+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71fc0b7cd0 con 0x7f71fc0cec60 2024-03-06T07:26:57.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.917+0000 7f71f97fa640 1 -- 172.21.15.122:0/629747233 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f71ec05c080 con 0x7f71fc0cec60 2024-03-06T07:26:57.919 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:57.919 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:57.919 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:57.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f71dc06b4d0 msgr2=0x7f71dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f71dc06b4d0 0x7f71dc06d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f71f400a770 tx=0x7f71f4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 msgr2=0x7f71fc0b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0b6e30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00c7f0 tx=0x7f71ec00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 shutdown_connections 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71fc15f5e0 0x7f71fc15ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71fc0cec60 0x7f71fc0b6e30 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 --2- 172.21.15.122:0/629747233 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71fc0b16b0 0x7f71fc158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 >> 172.21.15.122:0/629747233 conn(0x7f71fc0d1d50 msgr2=0x7f71fc06f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 shutdown_connections 2024-03-06T07:26:57.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:57.919+0000 7f720310c640 1 -- 172.21.15.122:0/629747233 wait complete. 2024-03-06T07:26:57.933 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.031+0000 7f76d4afb640 1 Processor -- start 2024-03-06T07:26:58.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.031+0000 7f76d4afb640 1 -- start start 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76d4afb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f76d0158690 0x7f76d00b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76d4afb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f76d00b40d0 0x7f76d00afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76d4afb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d00b5e20 con 0x7f76d00affa0 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76d4afb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d015f4d0 con 0x7f76d00b40d0 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76d4afb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d006ddc0 con 0x7f76d0158690 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76c5d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f76d00b40d0 0x7f76d00afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76c5d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f76d00b40d0 0x7f76d00afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55150/0 (socket says 172.21.15.122:55150) 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76c5d9b640 1 -- 172.21.15.122:0/595784732 learned_addr learned my addr 172.21.15.122:0/595784732 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76ced9d640 1 --1- 172.21.15.122:0/595784732 >> v1:172.21.15.122:6789/0 conn(0x7f76d00affa0 0x7f76d0161b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59938/0 (socket says 172.21.15.122:59938) 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.032+0000 7f76ce59c640 1 --2- 172.21.15.122:0/595784732 >> v2:172.21.15.169:3300/0 conn(0x7f76d0158690 0x7f76d00b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76c5d9b640 1 -- 172.21.15.122:0/595784732 >> v2:172.21.15.169:3300/0 conn(0x7f76d0158690 msgr2=0x7f76d00b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76c5d9b640 1 --2- 172.21.15.122:0/595784732 >> v2:172.21.15.169:3300/0 conn(0x7f76d0158690 0x7f76d00b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76c5d9b640 1 -- 172.21.15.122:0/595784732 >> v1:172.21.15.122:6789/0 conn(0x7f76d00affa0 legacy=0x7f76d0161b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76c5d9b640 1 -- 172.21.15.122:0/595784732 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d015e710 con 0x7f76d00b40d0 2024-03-06T07:26:58.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76c5d9b640 1 --2- 172.21.15.122:0/595784732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d00afa60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f76d006be40 tx=0x7f76b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7767b488960aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76cdd9b640 1 -- 172.21.15.122:0/595784732 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8034070 con 0x7f76d00b40d0 2024-03-06T07:26:58.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76cdd9b640 1 -- 172.21.15.122:0/595784732 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8030860 con 0x7f76d00b40d0 2024-03-06T07:26:58.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.033+0000 7f76cdd9b640 1 -- 172.21.15.122:0/595784732 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8030c30 con 0x7f76d00b40d0 2024-03-06T07:26:58.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 -- 172.21.15.122:0/595784732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 msgr2=0x7f76d00afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 --2- 172.21.15.122:0/595784732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d00afa60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f76d006be40 tx=0x7f76b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 -- 172.21.15.122:0/595784732 shutdown_connections 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 --2- 172.21.15.122:0/595784732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d00afa60 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 --2- 172.21.15.122:0/595784732 >> v2:172.21.15.169:3300/0 conn(0x7f76d0158690 0x7f76d00b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 -- 172.21.15.122:0/595784732 >> 172.21.15.122:0/595784732 conn(0x7f76d00b2890 msgr2=0x7f76d00b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 -- 172.21.15.122:0/595784732 shutdown_connections 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.034+0000 7f76d4afb640 1 -- 172.21.15.122:0/595784732 wait complete. 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.035+0000 7f76d4afb640 1 Processor -- start 2024-03-06T07:26:58.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.035+0000 7f76d4afb640 1 -- start start 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76d4afb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76d00affa0 0x7f76d0142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76d4afb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d01556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76d4afb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76d0158690 0x7f76d01440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76d4afb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d015f040 con 0x7f76d00b40d0 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76d4afb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d015ecd0 con 0x7f76d00affa0 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76d4afb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0155c20 con 0x7f76d0158690 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76c5d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d01556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76c5d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d01556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55158/0 (socket says 172.21.15.122:55158) 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.036+0000 7f76c5d9b640 1 -- 172.21.15.122:0/4016934489 learned_addr learned my addr 172.21.15.122:0/4016934489 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:58.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76ced9d640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76d0158690 0x7f76d01440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76ce59c640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76d00affa0 0x7f76d0142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c5d9b640 1 -- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76d0158690 msgr2=0x7f76d01440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c5d9b640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76d0158690 0x7f76d01440b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c5d9b640 1 -- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76d00affa0 msgr2=0x7f76d0142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c5d9b640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76d00affa0 0x7f76d0142a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c5d9b640 1 -- 172.21.15.122:0/4016934489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0143010 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c5d9b640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d01556e0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f76b8000c00 tx=0x7f76b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8048070 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8030660 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8034040 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d01432f0 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.037+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d01438b0 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.038+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d0068050 con 0x7f76d00b40d0 2024-03-06T07:26:58.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.038+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76b804b070 con 0x7f76d00b40d0 2024-03-06T07:26:58.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.039+0000 7f76c77fe640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76b006b500 0x7f76b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.039+0000 7f76ce59c640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76b006b500 0x7f76b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.039+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f76b80ac260 con 0x7f76d00b40d0 2024-03-06T07:26:58.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.039+0000 7f76ce59c640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76b006b500 0x7f76b006d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.043+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76b807b230 con 0x7f76d00b40d0 2024-03-06T07:26:58.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.256+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d00b3450 con 0x7f76d00b40d0 2024-03-06T07:26:58.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.257+0000 7f76c77fe640 1 -- 172.21.15.122:0/4016934489 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f76b807ab30 con 0x7f76d00b40d0 2024-03-06T07:26:58.259 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:58.259 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.451683+0000","modified":"2024-03-06T07:26:32.965725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/498310660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":498310660},{"type":"v1","addr":"172.21.15.122:6835","nonce":498310660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7671,"qdb_cluster":[7671]},"id":7}]} 2024-03-06T07:26:58.259 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 54 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76b006b500 msgr2=0x7f76b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76b006b500 0x7f76b006d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 msgr2=0x7f76d01556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d01556e0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f76b8000c00 tx=0x7f76b8004730 comp rx=0 tx=0).stop 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 shutdown_connections 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f76b006b500 0x7f76b006d9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f76d0158690 0x7f76d01440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76d00b40d0 0x7f76d01556e0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 --2- 172.21.15.122:0/4016934489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76d00affa0 0x7f76d0142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 >> 172.21.15.122:0/4016934489 conn(0x7f76d00b2890 msgr2=0x7f76d00ae380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 shutdown_connections 2024-03-06T07:26:58.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.259+0000 7f76d4afb640 1 -- 172.21.15.122:0/4016934489 wait complete. 2024-03-06T07:26:58.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.371+0000 7f1a176af640 1 Processor -- start 2024-03-06T07:26:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.371+0000 7f1a176af640 1 -- start start 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a176af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a10158570 0x7f1a1006f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a176af640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a100b3cc0 0x7f1a100afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a176af640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1a100affa0 0x7f1a10071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a176af640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a100b0410 con 0x7f1a100b3cc0 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a176af640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1006b190 con 0x7f1a100affa0 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a176af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1006b470 con 0x7f1a10158570 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a100b3cc0 0x7f1a100afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a100b3cc0 0x7f1a100afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55174/0 (socket says 172.21.15.122:55174) 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 -- 172.21.15.122:0/3484459971 learned_addr learned my addr 172.21.15.122:0/3484459971 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a1544b640 1 --2- 172.21.15.122:0/3484459971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a10158570 0x7f1a1006f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a15c4c640 1 --2- 172.21.15.122:0/3484459971 >> v2:172.21.15.169:3300/0 conn(0x7f1a100affa0 0x7f1a10071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 -- 172.21.15.122:0/3484459971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a10158570 msgr2=0x7f1a1006f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 --2- 172.21.15.122:0/3484459971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a10158570 0x7f1a1006f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 -- 172.21.15.122:0/3484459971 >> v2:172.21.15.169:3300/0 conn(0x7f1a100affa0 msgr2=0x7f1a10071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 --2- 172.21.15.122:0/3484459971 >> v2:172.21.15.169:3300/0 conn(0x7f1a100affa0 0x7f1a10071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.373+0000 7f1a14c4a640 1 -- 172.21.15.122:0/3484459971 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1006b750 con 0x7f1a100b3cc0 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.374+0000 7f1a14c4a640 1 --2- 172.21.15.122:0/3484459971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100afa60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f1a1006d820 tx=0x7f1a0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d5281f81288263 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.374+0000 7f1a07fff640 1 -- 172.21.15.122:0/3484459971 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c035020 con 0x7f1a100b3cc0 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.374+0000 7f1a07fff640 1 -- 172.21.15.122:0/3484459971 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c031870 con 0x7f1a100b3cc0 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.374+0000 7f1a07fff640 1 -- 172.21.15.122:0/3484459971 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c031c40 con 0x7f1a100b3cc0 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.374+0000 7f1a176af640 1 -- 172.21.15.122:0/3484459971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 msgr2=0x7f1a100afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.374+0000 7f1a176af640 1 --2- 172.21.15.122:0/3484459971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100afa60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f1a1006d820 tx=0x7f1a0c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 -- 172.21.15.122:0/3484459971 shutdown_connections 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 --2- 172.21.15.122:0/3484459971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100afa60 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 --2- 172.21.15.122:0/3484459971 >> v2:172.21.15.169:3300/0 conn(0x7f1a100affa0 0x7f1a10071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 --2- 172.21.15.122:0/3484459971 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a10158570 0x7f1a1006f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 -- 172.21.15.122:0/3484459971 >> 172.21.15.122:0/3484459971 conn(0x7f1a100b29c0 msgr2=0x7f1a100b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 -- 172.21.15.122:0/3484459971 shutdown_connections 2024-03-06T07:26:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.375+0000 7f1a176af640 1 -- 172.21.15.122:0/3484459971 wait complete. 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 Processor -- start 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 -- start start 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a100affa0 0x7f1a101415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a10158570 0x7f1a100b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10158950 con 0x7f1a100b3cc0 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1006fa70 con 0x7f1a10158570 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.376+0000 7f1a176af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100722b0 con 0x7f1a100affa0 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55184/0 (socket says 172.21.15.122:55184) 2024-03-06T07:26:58.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 -- 172.21.15.122:0/2234950328 learned_addr learned my addr 172.21.15.122:0/2234950328 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a1544b640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a100affa0 0x7f1a101415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a15c4c640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a10158570 0x7f1a100b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 -- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a100affa0 msgr2=0x7f1a101415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a100affa0 0x7f1a101415e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 -- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a10158570 msgr2=0x7f1a100b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a10158570 0x7f1a100b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 -- 172.21.15.122:0/2234950328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10151060 con 0x7f1a100b3cc0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a15c4c640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a10158570 0x7f1a100b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a14c4a640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100b1a60 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c02b950 tx=0x7f1a0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c040070 con 0x7f1a100b3cc0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c030d00 con 0x7f1a100b3cc0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a10151340 con 0x7f1a100b3cc0 2024-03-06T07:26:58.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.377+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c035040 con 0x7f1a100b3cc0 2024-03-06T07:26:58.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.378+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a10151900 con 0x7f1a100b3cc0 2024-03-06T07:26:58.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.379+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10069ee0 con 0x7f1a100b3cc0 2024-03-06T07:26:58.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.379+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a0c04b060 con 0x7f1a100b3cc0 2024-03-06T07:26:58.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.380+0000 7f1a05ffb640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.380+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1a0c0ac020 con 0x7f1a100b3cc0 2024-03-06T07:26:58.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.380+0000 7f1a1544b640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.380+0000 7f1a1544b640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19f406b4d0 0x7f19f406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.384+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a0c07ae40 con 0x7f1a100b3cc0 2024-03-06T07:26:58.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.560+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a100af4a0 con 0x7f1a100b3cc0 2024-03-06T07:26:58.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.561+0000 7f1a05ffb640 1 -- 172.21.15.122:0/2234950328 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16739 (secure 0 0 0) 0x7f1a0c07a740 con 0x7f1a100b3cc0 2024-03-06T07:26:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:26:58.563 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":97,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:26:28.153647+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.659781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.415605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:6835/2187235474":"2024-03-07T07:26:28.144746+0000","172.21.15.169:6833/2187235474":"2024-03-07T07:26:28.144746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19f406b4d0 msgr2=0x7f19f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19f406b4d0 0x7f19f406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00008040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 msgr2=0x7f1a100b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100b1a60 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c02b950 tx=0x7f1a0c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 shutdown_connections 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a10158570 0x7f1a100b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a100b3cc0 0x7f1a100b1a60 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 --2- 172.21.15.122:0/2234950328 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a100affa0 0x7f1a101415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 >> 172.21.15.122:0/2234950328 conn(0x7f1a100b29c0 msgr2=0x7f1a10073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 shutdown_connections 2024-03-06T07:26:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.563+0000 7f1a176af640 1 -- 172.21.15.122:0/2234950328 wait complete. 2024-03-06T07:26:58.576 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.677+0000 7fb79ef1e640 1 Processor -- start 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.677+0000 7fb79ef1e640 1 -- start start 2024-03-06T07:26:58.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb79ef1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb7980d0950 0x7fb7980d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb79ef1e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980d1ea0 con 0x7fb79815e150 2024-03-06T07:26:58.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb79ef1e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980d2180 con 0x7fb7980b9060 2024-03-06T07:26:58.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb79ef1e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7980d12f0 con 0x7fb7980d0950 2024-03-06T07:26:58.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb797fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb7980d0950 0x7fb7980d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb797fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb7980d0950 0x7fb7980d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55186/0 (socket says 172.21.15.122:55186) 2024-03-06T07:26:58.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb797fff640 1 -- 172.21.15.122:0/203329213 learned_addr learned my addr 172.21.15.122:0/203329213 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:58.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb797fff640 1 -- 172.21.15.122:0/203329213 >> v1:172.21.15.169:6789/0 conn(0x7fb7980b9060 legacy=0x7fb7980b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb797fff640 1 -- 172.21.15.122:0/203329213 >> v1:172.21.15.122:6789/0 conn(0x7fb79815e150 legacy=0x7fb7980cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.678+0000 7fb797fff640 1 -- 172.21.15.122:0/203329213 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980d2460 con 0x7fb7980d0950 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.679+0000 7fb797fff640 1 --2- 172.21.15.122:0/203329213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980d0db0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fb784009870 tx=0x7fb78402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557609319ad45b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.679+0000 7fb7977fe640 1 -- 172.21.15.122:0/203329213 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784034070 con 0x7fb7980d0950 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.679+0000 7fb7977fe640 1 -- 172.21.15.122:0/203329213 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784030860 con 0x7fb7980d0950 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.679+0000 7fb7977fe640 1 -- 172.21.15.122:0/203329213 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784030c30 con 0x7fb7980d0950 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 -- 172.21.15.122:0/203329213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 msgr2=0x7fb7980d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/203329213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980d0db0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fb784009870 tx=0x7fb78402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 -- 172.21.15.122:0/203329213 shutdown_connections 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/203329213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980d0db0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 -- 172.21.15.122:0/203329213 >> 172.21.15.122:0/203329213 conn(0x7fb79809d230 msgr2=0x7fb7980d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 -- 172.21.15.122:0/203329213 shutdown_connections 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.680+0000 7fb79ef1e640 1 -- 172.21.15.122:0/203329213 wait complete. 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.681+0000 7fb79ef1e640 1 Processor -- start 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.681+0000 7fb79ef1e640 1 -- start start 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ef1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7980b9060 0x7fb798154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ef1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ef1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79815e150 0x7fb79815ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ef1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980d2f40 con 0x7fb7980d0950 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ef1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980d2ab0 con 0x7fb79815e150 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ef1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980b7dc0 con 0x7fb7980b9060 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55198/0 (socket says 172.21.15.122:55198) 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 -- 172.21.15.122:0/788936015 learned_addr learned my addr 172.21.15.122:0/788936015 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79ccba640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7980b9060 0x7fb798154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79d4bb640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79815e150 0x7fb79815ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 -- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7980b9060 msgr2=0x7fb798154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7980b9060 0x7fb798154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 -- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79815e150 msgr2=0x7fb79815ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79815e150 0x7fb79815ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 -- 172.21.15.122:0/788936015 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980c7fd0 con 0x7fb7980d0950 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb79d4bb640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79815e150 0x7fb79815ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb797fff640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980c77a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb784000c00 tx=0x7fb784004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784048070 con 0x7fb7980d0950 2024-03-06T07:26:58.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784039d90 con 0x7fb7980d0950 2024-03-06T07:26:58.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.682+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784034040 con 0x7fb7980d0950 2024-03-06T07:26:58.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.683+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7980c82b0 con 0x7fb7980d0950 2024-03-06T07:26:58.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.683+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7980c8870 con 0x7fb7980d0950 2024-03-06T07:26:58.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.683+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb798069ee0 con 0x7fb7980d0950 2024-03-06T07:26:58.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.684+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb78404b070 con 0x7fb7980d0950 2024-03-06T07:26:58.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.684+0000 7fb7957fa640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.684+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7840ac320 con 0x7fb7980d0950 2024-03-06T07:26:58.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.684+0000 7fb79ccba640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.685+0000 7fb79ccba640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb78c000f60 tx=0x7fb78c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.689+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb78407aaf0 con 0x7fb7980d0950 2024-03-06T07:26:58.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.901+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb79806d7c0 con 0x7fb7980d0950 2024-03-06T07:26:58.903 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:26:58.902+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:58.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.903+0000 7fb7957fa640 1 -- 172.21.15.122:0/788936015 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb78407eb90 con 0x7fb7980d0950 2024-03-06T07:26:58.905 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 msgr2=0x7fb77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb78c000f60 tx=0x7fb78c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 msgr2=0x7fb7980c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980c77a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb784000c00 tx=0x7fb784004730 comp rx=0 tx=0).stop 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 shutdown_connections 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb79815e150 0x7fb79815ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7980d0950 0x7fb7980c77a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 --2- 172.21.15.122:0/788936015 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb7980b9060 0x7fb798154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 >> 172.21.15.122:0/788936015 conn(0x7fb79809d230 msgr2=0x7fb7980b6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 shutdown_connections 2024-03-06T07:26:58.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:58.907+0000 7fb79ef1e640 1 -- 172.21.15.122:0/788936015 wait complete. 2024-03-06T07:26:58.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:58.948 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:26:59.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.022+0000 7f3ff12c9640 1 Processor -- start 2024-03-06T07:26:59.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.022+0000 7f3ff12c9640 1 -- start start 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3ff12c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3fec15e150 0x7f3fec0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3ff12c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 0x7f3fec0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3ff12c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fec0d1ea0 con 0x7f3fec0cfba0 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3ff12c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec0d2ab0 con 0x7f3fec15e150 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3ff12c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d2180 con 0x7f3fec0d1060 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3fea7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3fec0cfba0 0x7f3fec0d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59956/0 (socket says 172.21.15.122:59956) 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.023+0000 7f3fea7fc640 1 -- 172.21.15.122:0/2867471169 learned_addr learned my addr 172.21.15.122:0/2867471169 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3feb7fe640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 0x7f3fec0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3feaffd640 1 --2- 172.21.15.122:0/2867471169 >> v2:172.21.15.169:3300/0 conn(0x7f3fec15e150 0x7f3fec0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588681472 0 0) 0x7f3fec0d1ea0 con 0x7f3fec0cfba0 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd4003520 con 0x7f3fec0cfba0 2024-03-06T07:26:59.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888018105 0 0) 0x7f3fd4003520 con 0x7f3fec0cfba0 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 msgr2=0x7f3fec0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 0x7f3fec0b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 >> v2:172.21.15.169:3300/0 conn(0x7f3fec15e150 msgr2=0x7f3fec0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 --2- 172.21.15.122:0/2867471169 >> v2:172.21.15.169:3300/0 conn(0x7f3fec15e150 0x7f3fec0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0d2460 con 0x7f3fec0cfba0 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f3fe0001670 con 0x7f3fec0cfba0 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3feaffd640 1 --2- 172.21.15.122:0/2867471169 >> v2:172.21.15.169:3300/0 conn(0x7f3fec15e150 0x7f3fec0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 >> v1:172.21.15.122:6789/0 conn(0x7f3fec0cfba0 legacy=0x7f3fec0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd40039b0 0x7f3fd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fd4004800 0x7f3fd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 0x7f3fd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d2ab0 con 0x7f3fd4004800 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d2180 con 0x7f3fd40039b0 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd400b9f0 con 0x7f3fec0d1060 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fea7fc640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fd4004800 0x7f3fd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3feaffd640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd40039b0 0x7f3fd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3feb7fe640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 0x7f3fd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fea7fc640 1 -- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 msgr2=0x7f3fd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fea7fc640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fec0d1060 0x7f3fd40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fea7fc640 1 -- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd40039b0 msgr2=0x7f3fd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fea7fc640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd40039b0 0x7f3fd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3fea7fc640 1 -- 172.21.15.122:0/2867471169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd4003520 con 0x7f3fd4004800 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.024+0000 7f3feaffd640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd40039b0 0x7f3fd4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.025+0000 7f3fea7fc640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fd4004800 0x7f3fd4004c80 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c290 tx=0x7f3fd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2b72de2057d0d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.025+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0007d50 con 0x7f3fd4004800 2024-03-06T07:26:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.025+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe0030070 con 0x7f3fd4004800 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.025+0000 7f3fe9ffb640 1 -- 172.21.15.122:0/2867471169 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe002cd50 con 0x7f3fd4004800 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3ff12c9640 1 -- 172.21.15.122:0/2867471169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fd4004800 msgr2=0x7f3fd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/2867471169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fd4004800 0x7f3fd4004c80 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c290 tx=0x7f3fd4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3feb7fe640 1 -- 172.21.15.122:0/2867471169 reap_dead start 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3ff12c9640 1 -- 172.21.15.122:0/2867471169 shutdown_connections 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3ff12c9640 1 -- 172.21.15.122:0/2867471169 >> 172.21.15.122:0/2867471169 conn(0x7f3fec09d230 msgr2=0x7f3fec0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3ff12c9640 1 -- 172.21.15.122:0/2867471169 shutdown_connections 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.026+0000 7f3ff12c9640 1 -- 172.21.15.122:0/2867471169 wait complete. 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.027+0000 7f3ff12c9640 1 Processor -- start 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.027+0000 7f3ff12c9640 1 -- start start 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fd4004800 0x7f3fec1f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fec09dd80 0x7f3fec1f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 0x7f3fec0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d1b30 con 0x7f3fec0c9040 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d2f40 con 0x7f3fec09dd80 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec15e530 con 0x7f3fd4004800 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 0x7f3fec0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 0x7f3fec0c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55234/0 (socket says 172.21.15.122:55234) 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 -- 172.21.15.122:0/3519587 learned_addr learned my addr 172.21.15.122:0/3519587 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 -- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fd4004800 msgr2=0x7f3fec1f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feaffd640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fd4004800 0x7f3fec1f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fd4004800 0x7f3fec1f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 -- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fec09dd80 msgr2=0x7f3fec1f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fec09dd80 0x7f3fec1f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 -- 172.21.15.122:0/3519587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec1f3c70 con 0x7f3fec0c9040 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3feb7fe640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 0x7f3fec0c94f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3fec08ca60 tx=0x7f3fe4011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4003610 con 0x7f3fec0c9040 2024-03-06T07:26:59.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe401b070 con 0x7f3fec0c9040 2024-03-06T07:26:59.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec1f81b0 con 0x7f3fec0c9040 2024-03-06T07:26:59.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.028+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4003a10 con 0x7f3fec0c9040 2024-03-06T07:26:59.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.029+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec1f41d0 con 0x7f3fec0c9040 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.029+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec069ee0 con 0x7f3fec0c9040 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.030+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fe4003bb0 con 0x7f3fec0c9040 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.030+0000 7f3fd37fe640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fd806b4d0 0x7f3fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.030+0000 7f3feaffd640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fd806b4d0 0x7f3fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.030+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3fe408a210 con 0x7f3fec0c9040 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.031+0000 7f3feaffd640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fd806b4d0 0x7f3fd806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc007440 tx=0x7f3fdc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.035+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fec1f81b0 con 0x7f3fec0c9040 2024-03-06T07:26:59.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.247+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fec0d4670 con 0x7f3fec0c9040 2024-03-06T07:26:59.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.905+0000 7f3fd37fe640 1 -- 172.21.15.122:0/3519587 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fec1f3c70 con 0x7f3fec0c9040 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fd806b4d0 msgr2=0x7f3fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fd806b4d0 0x7f3fd806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc007440 tx=0x7f3fdc00b040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 msgr2=0x7f3fec0c94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 0x7f3fec0c94f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3fec08ca60 tx=0x7f3fe4011920 comp rx=0 tx=0).stop 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 shutdown_connections 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3fd806b4d0 0x7f3fd806d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fec0c9040 0x7f3fec0c94f0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fec09dd80 0x7f3fec1f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 --2- 172.21.15.122:0/3519587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fd4004800 0x7f3fec1f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 >> 172.21.15.122:0/3519587 conn(0x7f3fec09d230 msgr2=0x7f3fec0cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 shutdown_connections 2024-03-06T07:26:59.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:26:59.908+0000 7f3ff12c9640 1 -- 172.21.15.122:0/3519587 wait complete. 2024-03-06T07:26:59.923 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:27:00.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.071+0000 7fe670f8c640 1 Processor -- start 2024-03-06T07:27:00.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.071+0000 7fe670f8c640 1 -- start start 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe670f8c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe66c145900 0x7fe66c0cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe670f8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c0cf610 0x7fe66c15c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe670f8c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe66c15c970 0x7fe66c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe670f8c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c06d150 con 0x7fe66c145900 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe670f8c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c06d3a0 con 0x7fe66c15c970 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe670f8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c06d680 con 0x7fe66c0cf610 2024-03-06T07:27:00.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe66c145900 0x7fe66c0cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe66c145900 0x7fe66c0cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55242/0 (socket says 172.21.15.122:55242) 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 -- 172.21.15.122:0/2350023963 learned_addr learned my addr 172.21.15.122:0/2350023963 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe663fff640 1 --2- 172.21.15.122:0/2350023963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c0cf610 0x7fe66c15c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66ad9d640 1 --2- 172.21.15.122:0/2350023963 >> v2:172.21.15.169:3300/0 conn(0x7fe66c15c970 0x7fe66c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 -- 172.21.15.122:0/2350023963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c0cf610 msgr2=0x7fe66c15c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 --2- 172.21.15.122:0/2350023963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c0cf610 0x7fe66c15c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 -- 172.21.15.122:0/2350023963 >> v2:172.21.15.169:3300/0 conn(0x7fe66c15c970 msgr2=0x7fe66c06e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 --2- 172.21.15.122:0/2350023963 >> v2:172.21.15.169:3300/0 conn(0x7fe66c15c970 0x7fe66c06e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.072+0000 7fe66a59c640 1 -- 172.21.15.122:0/2350023963 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c06d960 con 0x7fe66c145900 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe66a59c640 1 --2- 172.21.15.122:0/2350023963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c145900 0x7fe66c0cdb30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0b39a0 tx=0x7fe65c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5297e47219cfba45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe669d9b640 1 -- 172.21.15.122:0/2350023963 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c038070 con 0x7fe66c145900 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe669d9b640 1 -- 172.21.15.122:0/2350023963 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c02f050 con 0x7fe66c145900 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe669d9b640 1 -- 172.21.15.122:0/2350023963 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c031970 con 0x7fe66c145900 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 -- 172.21.15.122:0/2350023963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c145900 msgr2=0x7fe66c0cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 --2- 172.21.15.122:0/2350023963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c145900 0x7fe66c0cdb30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0b39a0 tx=0x7fe65c02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 -- 172.21.15.122:0/2350023963 shutdown_connections 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 --2- 172.21.15.122:0/2350023963 >> v2:172.21.15.169:3300/0 conn(0x7fe66c15c970 0x7fe66c06e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 --2- 172.21.15.122:0/2350023963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c0cf610 0x7fe66c15c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 --2- 172.21.15.122:0/2350023963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c145900 0x7fe66c0cdb30 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.073+0000 7fe670f8c640 1 -- 172.21.15.122:0/2350023963 >> 172.21.15.122:0/2350023963 conn(0x7fe66c06b160 msgr2=0x7fe66c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.074+0000 7fe670f8c640 1 -- 172.21.15.122:0/2350023963 shutdown_connections 2024-03-06T07:27:00.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.074+0000 7fe670f8c640 1 -- 172.21.15.122:0/2350023963 wait complete. 2024-03-06T07:27:00.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.075+0000 7fe670f8c640 1 Processor -- start 2024-03-06T07:27:00.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.075+0000 7fe670f8c640 1 -- start start 2024-03-06T07:27:00.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 0x7fe66c15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe66c145900 0x7fe66c1f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c15c970 0x7fe66c0c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0cfbf0 con 0x7fe66c0cf610 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0cfa70 con 0x7fe66c145900 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c06e6a0 con 0x7fe66c15c970 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 0x7fe66c15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 0x7fe66c15bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55244/0 (socket says 172.21.15.122:55244) 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 -- 172.21.15.122:0/1741494121 learned_addr learned my addr 172.21.15.122:0/1741494121 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 -- 172.21.15.122:0/1741494121 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c15c970 msgr2=0x7fe66c0c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c15c970 0x7fe66c0c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 -- 172.21.15.122:0/1741494121 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe66c145900 msgr2=0x7fe66c1f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe66c145900 0x7fe66c1f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 -- 172.21.15.122:0/1741494121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c0aeff0 con 0x7fe66c0cf610 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe66a59c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 0x7fe66c15bb40 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe65c02b830 tx=0x7fe65c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c042020 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c03d890 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c0aa740 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c038040 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.076+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c0aad00 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.077+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c069ee0 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.078+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe65c045020 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.078+0000 7fe662ffd640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.078+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe65c0abda0 con 0x7fe66c0cf610 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.078+0000 7fe663fff640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.078+0000 7fe663fff640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe64c06b4d0 0x7fe64c06d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0ac000 tx=0x7fe654008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.082+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe65c04c030 con 0x7fe66c0cf610 2024-03-06T07:27:00.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:00.260+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe66c0aaed0 con 0x7fe66c0cf610 2024-03-06T07:27:01.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.212+0000 7fe662ffd640 1 -- 172.21.15.122:0/1741494121 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe65c07ac00 con 0x7fe66c0cf610 2024-03-06T07:27:01.213 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe64c06b4d0 msgr2=0x7fe64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe64c06b4d0 0x7fe64c06d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0ac000 tx=0x7fe654008040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 msgr2=0x7fe66c15bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 0x7fe66c15bb40 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe65c02b830 tx=0x7fe65c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 shutdown_connections 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe66c15c970 0x7fe66c0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe66c145900 0x7fe66c1f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 --2- 172.21.15.122:0/1741494121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe66c0cf610 0x7fe66c15bb40 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 >> 172.21.15.122:0/1741494121 conn(0x7fe66c06b160 msgr2=0x7fe66c15dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 shutdown_connections 2024-03-06T07:27:01.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.216+0000 7fe670f8c640 1 -- 172.21.15.122:0/1741494121 wait complete. 2024-03-06T07:27:01.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:27:01.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.341+0000 7fca59793640 1 Processor -- start 2024-03-06T07:27:01.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.341+0000 7fca59793640 1 -- start start 2024-03-06T07:27:01.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca59793640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 0x7fca5406d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca59793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 0x7fca540aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca59793640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5406ded0 con 0x7fca5406d950 2024-03-06T07:27:01.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca59793640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca540ae430 con 0x7fca54155380 2024-03-06T07:27:01.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca59793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5406e770 con 0x7fca5406c6a0 2024-03-06T07:27:01.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca537fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 0x7fca540aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca52ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 0x7fca5406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca537fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 0x7fca540aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:52410/0 (socket says 172.21.15.122:52410) 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca52ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 0x7fca5406d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:33628/0 (socket says 172.21.15.122:33628) 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.342+0000 7fca537fe640 1 -- 172.21.15.122:0/1064945650 learned_addr learned my addr 172.21.15.122:0/1064945650 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.343+0000 7fca537fe640 1 -- 172.21.15.122:0/1064945650 >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 msgr2=0x7fca5406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.343+0000 7fca537fe640 1 --2- 172.21.15.122:0/1064945650 >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 0x7fca5406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.343+0000 7fca537fe640 1 -- 172.21.15.122:0/1064945650 >> v1:172.21.15.169:6789/0 conn(0x7fca5406d950 legacy=0x7fca5406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.343+0000 7fca537fe640 1 -- 172.21.15.122:0/1064945650 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5406e1b0 con 0x7fca5406c6a0 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.343+0000 7fca52ffd640 1 --2- 172.21.15.122:0/1064945650 >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 0x7fca5406d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.343+0000 7fca537fe640 1 --2- 172.21.15.122:0/1064945650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 0x7fca540aeb80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca4c0098a0 tx=0x7fca4c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a37b2e9f4a8c85e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca51ffb640 1 -- 172.21.15.122:0/1064945650 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c034070 con 0x7fca5406c6a0 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca51ffb640 1 -- 172.21.15.122:0/1064945650 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c030840 con 0x7fca5406c6a0 2024-03-06T07:27:01.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca51ffb640 1 -- 172.21.15.122:0/1064945650 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c030c10 con 0x7fca5406c6a0 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 -- 172.21.15.122:0/1064945650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 msgr2=0x7fca540aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 --2- 172.21.15.122:0/1064945650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 0x7fca540aeb80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca4c0098a0 tx=0x7fca4c02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 -- 172.21.15.122:0/1064945650 shutdown_connections 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 --2- 172.21.15.122:0/1064945650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca5406c6a0 0x7fca540aeb80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 --2- 172.21.15.122:0/1064945650 >> v2:172.21.15.169:3300/0 conn(0x7fca54155380 0x7fca5406d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 -- 172.21.15.122:0/1064945650 >> 172.21.15.122:0/1064945650 conn(0x7fca540b2980 msgr2=0x7fca54071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 -- 172.21.15.122:0/1064945650 shutdown_connections 2024-03-06T07:27:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.344+0000 7fca59793640 1 -- 172.21.15.122:0/1064945650 wait complete. 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.345+0000 7fca59793640 1 Processor -- start 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.345+0000 7fca59793640 1 -- start start 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca59793640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca5406c6a0 0x7fca54152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca59793640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 0x7fca541532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca59793640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca54155380 0x7fca5414a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca59793640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5406f450 con 0x7fca5406d950 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca59793640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca540ae660 con 0x7fca5406c6a0 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca59793640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5406f5d0 con 0x7fca54155380 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca527fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 0x7fca541532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca527fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 0x7fca541532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55256/0 (socket says 172.21.15.122:55256) 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca527fc640 1 -- 172.21.15.122:0/880413708 learned_addr learned my addr 172.21.15.122:0/880413708 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca537fe640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca54155380 0x7fca5414a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.346+0000 7fca52ffd640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca5406c6a0 0x7fca54152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca527fc640 1 -- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca54155380 msgr2=0x7fca5414a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca527fc640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca54155380 0x7fca5414a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca527fc640 1 -- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca5406c6a0 msgr2=0x7fca54152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca527fc640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca5406c6a0 0x7fca54152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca527fc640 1 -- 172.21.15.122:0/880413708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54153830 con 0x7fca5406d950 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca52ffd640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca5406c6a0 0x7fca54152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca527fc640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 0x7fca541532f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fca4800c400 tx=0x7fca4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca480052d0 con 0x7fca5406d950 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca48005490 con 0x7fca5406d950 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5414ba20 con 0x7fca5406d950 2024-03-06T07:27:01.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48007260 con 0x7fca5406d950 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.347+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5414bfe0 con 0x7fca5406d950 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.348+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca54069ee0 con 0x7fca5406d950 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.348+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca48007400 con 0x7fca5406d950 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.349+0000 7fca3b7fe640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fca4006b500 0x7fca4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.349+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5443+0+0 (secure 0 0 0) 0x7fca48089690 con 0x7fca5406d950 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.349+0000 7fca52ffd640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fca4006b500 0x7fca4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.349+0000 7fca52ffd640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fca4006b500 0x7fca4006d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca4400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.353+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca54153830 con 0x7fca5406d950 2024-03-06T07:27:01.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:01.528+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fca54072780 con 0x7fca5406d950 2024-03-06T07:27:02.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.223+0000 7fca3b7fe640 1 -- 172.21.15.122:0/880413708 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) v1 ==== 169+0+0 (secure 0 0 0) 0x7fca48057eb0 con 0x7fca5406d950 2024-03-06T07:27:02.224 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fca4006b500 msgr2=0x7fca4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fca4006b500 0x7fca4006d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca4400b040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 msgr2=0x7fca541532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 0x7fca541532f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fca4800c400 tx=0x7fca4800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 shutdown_connections 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fca4006b500 0x7fca4006d9c0 secure :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca4400b040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fca54155380 0x7fca5414a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca5406d950 0x7fca541532f0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 --2- 172.21.15.122:0/880413708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca5406c6a0 0x7fca54152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 >> 172.21.15.122:0/880413708 conn(0x7fca540b2980 msgr2=0x7fca5415baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.227+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 shutdown_connections 2024-03-06T07:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.228+0000 7fca59793640 1 -- 172.21.15.122:0/880413708 wait complete. 2024-03-06T07:27:02.244 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.296 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:27:02.296 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:27:02.296 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:27:02.297 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.298 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.304 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:27:02.308 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.309 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.310 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.311 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.313 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.314 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.315 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:27:02.321 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.413 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:27:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:27:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:27:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:27:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:27:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:27:02.424 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.425 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.426 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:27:02.427 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:27:02.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.429 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.430 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.431 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.432 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.433 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:27:02.434 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:27:02.435 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.436 INFO:teuthology.orchestra.run.smithi169.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.438 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.527 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-06T07:27:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-06T07:27:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:02.536 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-06T07:27:02.537 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.539 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.540 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.122,172.21.15.169,[v2:172.21.15.169:3301,v1:172.21.15.169:6790]" 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 12542aa0-5a71-4218-ae26-0a8da740ed3f 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.542 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.543 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.544 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-06T07:27:02.545 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.546 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: mon op complaint time = 120 2024-03-06T07:27:02.547 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: client mount timeout = 600 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: debug client = 20 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = "" 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.548 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout:[mds] 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds = 20 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout: debug mds balancer = 20 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug frag = True 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.549 INFO:teuthology.orchestra.run.smithi169.stdout: mds op complaint time = 180 2024-03-06T07:27:02.550 INFO:teuthology.orchestra.run.smithi169.stdout: mds verify scatter = True 2024-03-06T07:27:02.550 INFO:teuthology.orchestra.run.smithi169.stdout: osd op complaint time = 180 2024-03-06T07:27:02.550 INFO:teuthology.orchestra.run.smithi169.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.550 INFO:teuthology.orchestra.run.smithi169.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:27:02.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.694+0000 7fb49be66640 1 Processor -- start 2024-03-06T07:27:02.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.695+0000 7fb49be66640 1 -- start start 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.695+0000 7fb49be66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4941614d0 0x7fb4940d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.695+0000 7fb49be66640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb4940d1910 0x7fb4940d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.695+0000 7fb49be66640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4940d2740 con 0x7fb4940d22b0 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.695+0000 7fb49be66640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4941618b0 con 0x7fb4940d1910 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.695+0000 7fb49be66640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940cf400 con 0x7fb4941614d0 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb4940d1910 0x7fb4940d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb4940d1910 0x7fb4940d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55268/0 (socket says 172.21.15.122:55268) 2024-03-06T07:27:02.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 -- 172.21.15.122:0/314205664 learned_addr learned my addr 172.21.15.122:0/314205664 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb49a403640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb4940d22b0 0x7fb4940d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59964/0 (socket says 172.21.15.122:59964) 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499c02640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4941614d0 0x7fb4940d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 -- 172.21.15.122:0/314205664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4941614d0 msgr2=0x7fb4940d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4941614d0 0x7fb4940d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 -- 172.21.15.122:0/314205664 >> v1:172.21.15.122:6789/0 conn(0x7fb4940d22b0 legacy=0x7fb4940d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499401640 1 -- 172.21.15.122:0/314205664 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4940cf6a0 con 0x7fb4940d1910 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb498c00640 1 -- 172.21.15.122:0/314205664 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262752171 0 0) 0x7fb48c001490 con 0x7fb4940d22b0 2024-03-06T07:27:02.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.696+0000 7fb499c02640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4941614d0 0x7fb4940d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.697+0000 7fb499401640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d1910 0x7fb4940d1d70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb490009870 tx=0x7fb49002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c2e8da5580da92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.697+0000 7fb498c00640 1 -- 172.21.15.122:0/314205664 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490034070 con 0x7fb4940d1910 2024-03-06T07:27:02.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.697+0000 7fb498c00640 1 -- 172.21.15.122:0/314205664 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490030840 con 0x7fb4940d1910 2024-03-06T07:27:02.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.697+0000 7fb498c00640 1 -- 172.21.15.122:0/314205664 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490030c10 con 0x7fb4940d1910 2024-03-06T07:27:02.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 -- 172.21.15.122:0/314205664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d1910 msgr2=0x7fb4940d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d1910 0x7fb4940d1d70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb490009870 tx=0x7fb49002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:02.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 -- 172.21.15.122:0/314205664 shutdown_connections 2024-03-06T07:27:02.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d1910 0x7fb4940d1d70 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 --2- 172.21.15.122:0/314205664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4941614d0 0x7fb4940d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 -- 172.21.15.122:0/314205664 >> 172.21.15.122:0/314205664 conn(0x7fb4940d3d40 msgr2=0x7fb4940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 -- 172.21.15.122:0/314205664 shutdown_connections 2024-03-06T07:27:02.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.698+0000 7fb49be66640 1 -- 172.21.15.122:0/314205664 wait complete. 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.699+0000 7fb49be66640 1 Processor -- start 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.699+0000 7fb49be66640 1 -- start start 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4940d1910 0x7fb4941501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 0x7fb4940c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4941614d0 0x7fb4941600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940b0910 con 0x7fb4940d22b0 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940cfc60 con 0x7fb4941614d0 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940b1e40 con 0x7fb4940d1910 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 0x7fb4940c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 0x7fb4940c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55282/0 (socket says 172.21.15.122:55282) 2024-03-06T07:27:02.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 -- 172.21.15.122:0/486660663 learned_addr learned my addr 172.21.15.122:0/486660663 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499c02640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4940d1910 0x7fb4941501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49a403640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4941614d0 0x7fb4941600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 -- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4940d1910 msgr2=0x7fb4941501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4940d1910 0x7fb4941501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 -- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4941614d0 msgr2=0x7fb4941600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4941614d0 0x7fb4941600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 -- 172.21.15.122:0/486660663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4940c82c0 con 0x7fb4940d22b0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499c02640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4940d1910 0x7fb4941501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49a403640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4941614d0 0x7fb4941600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb499401640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 0x7fb4940c7870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fb490000c00 tx=0x7fb490004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490048070 con 0x7fb4940d22b0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490030640 con 0x7fb4940d22b0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.700+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4940c85a0 con 0x7fb4940d22b0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.701+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490034040 con 0x7fb4940d22b0 2024-03-06T07:27:02.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.701+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4940c8b60 con 0x7fb4940d22b0 2024-03-06T07:27:02.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.701+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb494068ff0 con 0x7fb4940d22b0 2024-03-06T07:27:02.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.702+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb49004b070 con 0x7fb4940d22b0 2024-03-06T07:27:02.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.702+0000 7fb48a7fc640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb47806b4d0 0x7fb47806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.702+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb4900ac020 con 0x7fb4940d22b0 2024-03-06T07:27:02.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.702+0000 7fb499c02640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb47806b4d0 0x7fb47806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.703+0000 7fb499c02640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb47806b4d0 0x7fb47806d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb484006fd0 tx=0x7fb484008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.707+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb49007b260 con 0x7fb4940d22b0 2024-03-06T07:27:02.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.887+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb494069e80 con 0x7fb4940d22b0 2024-03-06T07:27:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.890+0000 7fb48a7fc640 1 -- 172.21.15.122:0/486660663 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb49007ab60 con 0x7fb4940d22b0 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb47806b4d0 msgr2=0x7fb47806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb47806b4d0 0x7fb47806d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb484006fd0 tx=0x7fb484008040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 msgr2=0x7fb4940c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 0x7fb4940c7870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fb490000c00 tx=0x7fb490004730 comp rx=0 tx=0).stop 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 shutdown_connections 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb47806b4d0 0x7fb47806d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4941614d0 0x7fb4941600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4940d22b0 0x7fb4940c7870 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 --2- 172.21.15.122:0/486660663 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4940d1910 0x7fb4941501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 >> 172.21.15.122:0/486660663 conn(0x7fb4940d3d40 msgr2=0x7fb4940d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 shutdown_connections 2024-03-06T07:27:02.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:02.893+0000 7fb49be66640 1 -- 172.21.15.122:0/486660663 wait complete. 2024-03-06T07:27:02.905 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:27:03.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.004+0000 7f111dc78640 1 Processor -- start 2024-03-06T07:27:03.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.004+0000 7f111dc78640 1 -- start start 2024-03-06T07:27:03.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f111dc78640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118144640 0x7f11180b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f111dc78640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1118070930 0x7f11180b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f111dc78640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1118070d90 con 0x7f111815b090 2024-03-06T07:27:03.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f111dc78640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11180b4fe0 con 0x7f1118070930 2024-03-06T07:27:03.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f111dc78640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11180ce5d0 con 0x7f1118144640 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f1117fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f111815b090 0x7f111815b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59968/0 (socket says 172.21.15.122:59968) 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f1117fff640 1 -- 172.21.15.122:0/353497477 learned_addr learned my addr 172.21.15.122:0/353497477 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f11177fe640 1 --2- 172.21.15.122:0/353497477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118144640 0x7f11180b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f1116ffd640 1 --2- 172.21.15.122:0/353497477 >> v2:172.21.15.169:3300/0 conn(0x7f1118070930 0x7f11180b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f11167fc640 1 -- 172.21.15.122:0/353497477 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408118874 0 0) 0x7f1118070d90 con 0x7f111815b090 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.005+0000 7f11167fc640 1 -- 172.21.15.122:0/353497477 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1100003520 con 0x7f111815b090 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11177fe640 1 -- 172.21.15.122:0/353497477 >> v2:172.21.15.169:3300/0 conn(0x7f1118070930 msgr2=0x7f11180b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11177fe640 1 --2- 172.21.15.122:0/353497477 >> v2:172.21.15.169:3300/0 conn(0x7f1118070930 0x7f11180b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11167fc640 1 -- 172.21.15.122:0/353497477 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009991540 0 0) 0x7f1100003520 con 0x7f111815b090 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11177fe640 1 -- 172.21.15.122:0/353497477 >> v1:172.21.15.122:6789/0 conn(0x7f111815b090 legacy=0x7f111815b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11177fe640 1 -- 172.21.15.122:0/353497477 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11180b3bb0 con 0x7f1118144640 2024-03-06T07:27:03.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f1116ffd640 1 --2- 172.21.15.122:0/353497477 >> v2:172.21.15.169:3300/0 conn(0x7f1118070930 0x7f11180b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11177fe640 1 --2- 172.21.15.122:0/353497477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118144640 0x7f11180b34d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f11180ce750 tx=0x7f110802d8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7015357140b6ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.006+0000 7f11167fc640 1 -- 172.21.15.122:0/353497477 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100019f0 con 0x7f1118144640 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f11167fc640 1 -- 172.21.15.122:0/353497477 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1108039070 con 0x7f1118144640 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f11167fc640 1 -- 172.21.15.122:0/353497477 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11080337f0 con 0x7f1118144640 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 -- 172.21.15.122:0/353497477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118144640 msgr2=0x7f11180b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 --2- 172.21.15.122:0/353497477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118144640 0x7f11180b34d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f11180ce750 tx=0x7f110802d8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 -- 172.21.15.122:0/353497477 shutdown_connections 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 --2- 172.21.15.122:0/353497477 >> v2:172.21.15.169:3300/0 conn(0x7f1118070930 0x7f11180b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 --2- 172.21.15.122:0/353497477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118144640 0x7f11180b34d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 -- 172.21.15.122:0/353497477 >> 172.21.15.122:0/353497477 conn(0x7f111806b3d0 msgr2=0x7f111806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 -- 172.21.15.122:0/353497477 shutdown_connections 2024-03-06T07:27:03.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.007+0000 7f111dc78640 1 -- 172.21.15.122:0/353497477 wait complete. 2024-03-06T07:27:03.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.008+0000 7f111dc78640 1 Processor -- start 2024-03-06T07:27:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.008+0000 7f111dc78640 1 -- start start 2024-03-06T07:27:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f111dc78640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118070930 0x7f111815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f111dc78640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 0x7f1118141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f111dc78640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f111815b090 0x7f1118141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f111dc78640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180cdee0 con 0x7f1118144640 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f111dc78640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180ce260 con 0x7f111815b090 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f111dc78640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11180cea60 con 0x7f1118070930 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f1116ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 0x7f1118141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f1116ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 0x7f1118141300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55288/0 (socket says 172.21.15.122:55288) 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f11177fe640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118070930 0x7f111815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f1116ffd640 1 -- 172.21.15.122:0/961349251 learned_addr learned my addr 172.21.15.122:0/961349251 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f1117fff640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f111815b090 0x7f1118141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f1116ffd640 1 -- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118070930 msgr2=0x7f111815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.009+0000 7f1116ffd640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118070930 0x7f111815a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f1116ffd640 1 -- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f111815b090 msgr2=0x7f1118141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f1116ffd640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f111815b090 0x7f1118141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f1116ffd640 1 -- 172.21.15.122:0/961349251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11181406d0 con 0x7f1118144640 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f11177fe640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118070930 0x7f111815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f1117fff640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f111815b090 0x7f1118141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f1116ffd640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 0x7f1118141300 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f110c00ebe0 tx=0x7f110c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c017070 con 0x7f1118144640 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c00cc30 con 0x7f1118144640 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11180a9660 con 0x7f1118144640 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c013470 con 0x7f1118144640 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.010+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11180a9c20 con 0x7f1118144640 2024-03-06T07:27:03.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.011+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111804f330 con 0x7f1118144640 2024-03-06T07:27:03.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.011+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f110c01a070 con 0x7f1118144640 2024-03-06T07:27:03.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.012+0000 7f10fffff640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f110406b500 0x7f110406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.012+0000 7f11177fe640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f110406b500 0x7f110406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.012+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f110c089dd0 con 0x7f1118144640 2024-03-06T07:27:03.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.012+0000 7f11177fe640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f110406b500 0x7f110406d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1108038000 tx=0x7f110803f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.016+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f110c0588a0 con 0x7f1118144640 2024-03-06T07:27:03.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:03.254+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f11180b1530 con 0x7f1118144640 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.228+0000 7f10fffff640 1 -- 172.21.15.122:0/961349251 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v290) v1 ==== 134+0+0 (secure 0 0 0) 0x7f110c05c940 con 0x7f1118144640 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f110406b500 msgr2=0x7f110406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f110406b500 0x7f110406d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1108038000 tx=0x7f110803f040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 msgr2=0x7f1118141300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 0x7f1118141300 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f110c00ebe0 tx=0x7f110c00c520 comp rx=0 tx=0).stop 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 shutdown_connections 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f110406b500 0x7f110406d9c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f111815b090 0x7f1118141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1118144640 0x7f1118141300 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 --2- 172.21.15.122:0/961349251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1118070930 0x7f111815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 >> 172.21.15.122:0/961349251 conn(0x7f111806b3d0 msgr2=0x7f1118070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.230+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 shutdown_connections 2024-03-06T07:27:04.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.231+0000 7f111dc78640 1 -- 172.21.15.122:0/961349251 wait complete. 2024-03-06T07:27:04.245 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:27:04.245 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:27:04.246 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:27:04.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:04.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.347+0000 7f1def081640 1 Processor -- start 2024-03-06T07:27:04.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.347+0000 7f1def081640 1 -- start start 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1def081640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1de80cce40 0x7f1de815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1def081640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 0x7f1de80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1def081640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de8071f40 con 0x7f1de81515e0 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1def081640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80cd2f0 con 0x7f1de80cce40 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1def081640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8070a70 con 0x7f1de80cd7a0 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1de80cce40 0x7f1de815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1de80cce40 0x7f1de815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55302/0 (socket says 172.21.15.122:55302) 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 -- 172.21.15.122:0/4262663135 learned_addr learned my addr 172.21.15.122:0/4262663135 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 -- 172.21.15.122:0/4262663135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 msgr2=0x7f1de80cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 --2- 172.21.15.122:0/4262663135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 0x7f1de80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 -- 172.21.15.122:0/4262663135 >> v1:172.21.15.169:6789/0 conn(0x7f1de81515e0 legacy=0x7f1de8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.348+0000 7f1de7fff640 1 -- 172.21.15.122:0/4262663135 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8070d50 con 0x7f1de80cce40 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1de7fff640 1 --2- 172.21.15.122:0/4262663135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de815b430 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1de80cc110 tx=0x7f1dd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646d78618c405ff1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1de77fe640 1 -- 172.21.15.122:0/4262663135 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0035020 con 0x7f1de80cce40 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1de77fe640 1 -- 172.21.15.122:0/4262663135 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0031640 con 0x7f1de80cce40 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1de77fe640 1 -- 172.21.15.122:0/4262663135 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd002d040 con 0x7f1de80cce40 2024-03-06T07:27:04.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1def081640 1 -- 172.21.15.122:0/4262663135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 msgr2=0x7f1de815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1def081640 1 --2- 172.21.15.122:0/4262663135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de815b430 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1de80cc110 tx=0x7f1dd002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1def081640 1 -- 172.21.15.122:0/4262663135 shutdown_connections 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1def081640 1 --2- 172.21.15.122:0/4262663135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 0x7f1de80cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1def081640 1 --2- 172.21.15.122:0/4262663135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de815b430 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.349+0000 7f1def081640 1 -- 172.21.15.122:0/4262663135 >> 172.21.15.122:0/4262663135 conn(0x7f1de80cf8a0 msgr2=0x7f1de80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.350+0000 7f1def081640 1 -- 172.21.15.122:0/4262663135 shutdown_connections 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.350+0000 7f1def081640 1 -- 172.21.15.122:0/4262663135 wait complete. 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.351+0000 7f1def081640 1 Processor -- start 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.351+0000 7f1def081640 1 -- start start 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de8145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 0x7f1de814b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de81515e0 0x7f1de814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80ce0f0 con 0x7f1de80cce40 2024-03-06T07:27:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8071310 con 0x7f1de81515e0 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80cc3a0 con 0x7f1de80cd7a0 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de8145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de8145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55316/0 (socket says 172.21.15.122:55316) 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 -- 172.21.15.122:0/2569433242 learned_addr learned my addr 172.21.15.122:0/2569433242 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 -- 172.21.15.122:0/2569433242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 msgr2=0x7f1de814b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 0x7f1de814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 -- 172.21.15.122:0/2569433242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de81515e0 msgr2=0x7f1de814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de81515e0 0x7f1de814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 -- 172.21.15.122:0/2569433242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8144bd0 con 0x7f1de80cce40 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1dece1d640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de8145c70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b410 tx=0x7f1dd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8015020 con 0x7f1de80cce40 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd8011830 con 0x7f1de80cce40 2024-03-06T07:27:04.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de8144eb0 con 0x7f1de80cce40 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8011c20 con 0x7f1de80cce40 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.352+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de8145470 con 0x7f1de80cce40 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.353+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8068840 con 0x7f1de80cce40 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.354+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dd8018020 con 0x7f1de80cce40 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.354+0000 7f1de57fa640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dc806b4d0 0x7f1dc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.354+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1dd8089ec0 con 0x7f1de80cce40 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.354+0000 7f1de7fff640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dc806b4d0 0x7f1dc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.354+0000 7f1de7fff640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dc806b4d0 0x7f1dc806d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0036000 tx=0x7f1dd0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.358+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1dd801f020 con 0x7f1de80cce40 2024-03-06T07:27:04.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.530+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1de804f330 con 0x7f1de80cce40 2024-03-06T07:27:04.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.531+0000 7f1de57fa640 1 -- 172.21.15.122:0/2569433242 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13154 (secure 0 0 0) 0x7f1dd80172c0 con 0x7f1de80cce40 2024-03-06T07:27:04.533 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:04.533 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:02.219461+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6835/498310660":"2024-03-07T07:26:58.903017+0000","172.21.15.122:6834/498310660":"2024-03-07T07:26:58.903017+0000","172.21.15.169:6835/2187235474":"2024-03-07T07:26:28.144746+0000","172.21.15.169:6833/2187235474":"2024-03-07T07:26:28.144746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:04.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dc806b4d0 msgr2=0x7f1dc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dc806b4d0 0x7f1dc806d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0036000 tx=0x7f1dd0030000 comp rx=0 tx=0).stop 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 msgr2=0x7f1de8145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de8145c70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b410 tx=0x7f1dd800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 shutdown_connections 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dc806b4d0 0x7f1dc806d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de81515e0 0x7f1de814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de80cd7a0 0x7f1de814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 --2- 172.21.15.122:0/2569433242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de80cce40 0x7f1de8145c70 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 >> 172.21.15.122:0/2569433242 conn(0x7f1de80cf8a0 msgr2=0x7f1de815d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 shutdown_connections 2024-03-06T07:27:04.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.533+0000 7f1def081640 1 -- 172.21.15.122:0/2569433242 wait complete. 2024-03-06T07:27:04.546 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:04.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:04.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.648+0000 7f1277be2640 1 Processor -- start 2024-03-06T07:27:04.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.648+0000 7f1277be2640 1 -- start start 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f1277be2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700cfa90 0x7f12700cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f1277be2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f12700b33a0 0x7f12700b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f1277be2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270158ac0 con 0x7f12701586e0 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f1277be2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127015b2a0 con 0x7f12700b33a0 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f1277be2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127015b540 con 0x7f12700cfa90 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f12700b33a0 0x7f12700b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f12700b33a0 0x7f12700b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44346/0 (socket says 172.21.15.122:44346) 2024-03-06T07:27:04.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 -- 172.21.15.122:0/922189069 learned_addr learned my addr 172.21.15.122:0/922189069 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127517d640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700cfa90 0x7f12700cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 -- 172.21.15.122:0/922189069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700cfa90 msgr2=0x7f12700cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700cfa90 0x7f12700cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 -- 172.21.15.122:0/922189069 >> v1:172.21.15.169:6789/0 conn(0x7f12701586e0 legacy=0x7f12700aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127617f640 1 -- 172.21.15.122:0/922189069 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127015b820 con 0x7f12700b33a0 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.649+0000 7f127517d640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700cfa90 0x7f12700cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.650+0000 7f127617f640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700b33a0 0x7f12700b37f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f1268009e50 tx=0x7f126802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3240cd15afe0011 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.650+0000 7f127497c640 1 -- 172.21.15.122:0/922189069 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126802cb00 con 0x7f12700b33a0 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.650+0000 7f127497c640 1 -- 172.21.15.122:0/922189069 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126802cca0 con 0x7f12700b33a0 2024-03-06T07:27:04.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.650+0000 7f127497c640 1 -- 172.21.15.122:0/922189069 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268034610 con 0x7f12700b33a0 2024-03-06T07:27:04.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 -- 172.21.15.122:0/922189069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700b33a0 msgr2=0x7f12700b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700b33a0 0x7f12700b37f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f1268009e50 tx=0x7f126802bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 -- 172.21.15.122:0/922189069 shutdown_connections 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700b33a0 0x7f12700b37f0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 --2- 172.21.15.122:0/922189069 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700cfa90 0x7f12700cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 -- 172.21.15.122:0/922189069 >> 172.21.15.122:0/922189069 conn(0x7f127006b190 msgr2=0x7f127006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 -- 172.21.15.122:0/922189069 shutdown_connections 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.651+0000 7f1277be2640 1 -- 172.21.15.122:0/922189069 wait complete. 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.652+0000 7f1277be2640 1 Processor -- start 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.652+0000 7f1277be2640 1 -- start start 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.652+0000 7f1277be2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b33a0 0x7f1270148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 0x7f12700ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12701586e0 0x7f12700adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700af150 con 0x7f12700cfa90 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127015bde0 con 0x7f12701586e0 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1270071350 con 0x7f12700b33a0 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 0x7f12700ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 0x7f12700ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44354/0 (socket says 172.21.15.122:44354) 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 -- 172.21.15.122:0/1282683030 learned_addr learned my addr 172.21.15.122:0/1282683030 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127597e640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b33a0 0x7f1270148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127617f640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12701586e0 0x7f12700adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 -- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b33a0 msgr2=0x7f1270148650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b33a0 0x7f1270148650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 -- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12701586e0 msgr2=0x7f12700adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12701586e0 0x7f12700adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 -- 172.21.15.122:0/1282683030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12701f5090 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127597e640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b33a0 0x7f1270148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127617f640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12701586e0 0x7f12700adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f127517d640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 0x7f12700ad860 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f126000e7e0 tx=0x7f126000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126000f040 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260004030 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1270148b90 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260004400 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.653+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12701520b0 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.654+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12700732b0 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.655+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f126000c6d0 con 0x7f12700cfa90 2024-03-06T07:27:04.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.655+0000 7f12667fc640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b4d0 0x7f125406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.655+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1260089c90 con 0x7f12700cfa90 2024-03-06T07:27:04.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.655+0000 7f127597e640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b4d0 0x7f125406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.655+0000 7f127597e640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b4d0 0x7f125406d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f126c001680 tx=0x7f126c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.659+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12701f5090 con 0x7f12700cfa90 2024-03-06T07:27:04.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.828+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1270068ff0 con 0x7f12700cfa90 2024-03-06T07:27:04.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.829+0000 7f12667fc640 1 -- 172.21.15.122:0/1282683030 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1260017020 con 0x7f12700cfa90 2024-03-06T07:27:04.831 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b4d0 msgr2=0x7f125406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b4d0 0x7f125406d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f126c001680 tx=0x7f126c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 msgr2=0x7f12700ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 0x7f12700ad860 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f126000e7e0 tx=0x7f126000eca0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 shutdown_connections 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f125406b4d0 0x7f125406d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12701586e0 0x7f12700adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12700cfa90 0x7f12700ad860 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 --2- 172.21.15.122:0/1282683030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12700b33a0 0x7f1270148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 >> 172.21.15.122:0/1282683030 conn(0x7f127006b190 msgr2=0x7f1270159930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 shutdown_connections 2024-03-06T07:27:04.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.831+0000 7f1277be2640 1 -- 172.21.15.122:0/1282683030 wait complete. 2024-03-06T07:27:04.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:27:04.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.946+0000 7f3a24f65640 1 Processor -- start 2024-03-06T07:27:04.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.946+0000 7f3a24f65640 1 -- start start 2024-03-06T07:27:04.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a24f65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a201424f0 0x7f3a200cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a24f65640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200cc980 con 0x7f3a2006da40 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a24f65640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200ccc60 con 0x7f3a200cd7a0 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a24f65640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a200cd220 con 0x7f3a201424f0 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a1e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a201424f0 0x7f3a200cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a1ed9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3a2006da40 0x7f3a20071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43428/0 (socket says 172.21.15.122:43428) 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a1e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a201424f0 0x7f3a200cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44362/0 (socket says 172.21.15.122:44362) 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.947+0000 7f3a1ed9d640 1 -- 172.21.15.122:0/3664389211 learned_addr learned my addr 172.21.15.122:0/3664389211 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1e59c640 1 -- 172.21.15.122:0/3664389211 >> v1:172.21.15.169:6789/0 conn(0x7f3a200cd7a0 legacy=0x7f3a2006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1e59c640 1 -- 172.21.15.122:0/3664389211 >> v1:172.21.15.122:6789/0 conn(0x7f3a2006da40 legacy=0x7f3a20071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1e59c640 1 -- 172.21.15.122:0/3664389211 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a200ccf40 con 0x7f3a201424f0 2024-03-06T07:27:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1d59a640 1 -- 172.21.15.122:0/3664389211 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711571781 0 0) 0x7f3a200cc980 con 0x7f3a2006da40 2024-03-06T07:27:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1e59c640 1 --2- 172.21.15.122:0/3664389211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200cc2f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009d90 tx=0x7f3a1002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472e124026a9561c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1d59a640 1 -- 172.21.15.122:0/3664389211 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10039070 con 0x7f3a201424f0 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1d59a640 1 -- 172.21.15.122:0/3664389211 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1002cc60 con 0x7f3a201424f0 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.948+0000 7f3a1d59a640 1 -- 172.21.15.122:0/3664389211 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10034660 con 0x7f3a201424f0 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 -- 172.21.15.122:0/3664389211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 msgr2=0x7f3a200cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 --2- 172.21.15.122:0/3664389211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200cc2f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009d90 tx=0x7f3a1002beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 -- 172.21.15.122:0/3664389211 shutdown_connections 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 --2- 172.21.15.122:0/3664389211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200cc2f0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 -- 172.21.15.122:0/3664389211 >> 172.21.15.122:0/3664389211 conn(0x7f3a200cf8a0 msgr2=0x7f3a200cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 -- 172.21.15.122:0/3664389211 shutdown_connections 2024-03-06T07:27:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.949+0000 7f3a24f65640 1 -- 172.21.15.122:0/3664389211 wait complete. 2024-03-06T07:27:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.950+0000 7f3a24f65640 1 Processor -- start 2024-03-06T07:27:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.950+0000 7f3a24f65640 1 -- start start 2024-03-06T07:27:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a24f65640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a2006da40 0x7f3a2015fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a24f65640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a200cd7a0 0x7f3a200ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a24f65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a24f65640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2015d070 con 0x7f3a201424f0 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a24f65640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2015cef0 con 0x7f3a200cd7a0 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a24f65640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2015d1f0 con 0x7f3a2006da40 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a1ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a1ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44370/0 (socket says 172.21.15.122:44370) 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a1ed9d640 1 -- 172.21.15.122:0/865945528 learned_addr learned my addr 172.21.15.122:0/865945528 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:04.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a1dd9b640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a200cd7a0 0x7f3a200ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.951+0000 7f3a1e59c640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a2006da40 0x7f3a2015fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1ed9d640 1 -- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a2006da40 msgr2=0x7f3a2015fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1ed9d640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a2006da40 0x7f3a2015fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1ed9d640 1 -- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a200cd7a0 msgr2=0x7f3a200ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1ed9d640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a200cd7a0 0x7f3a200ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1ed9d640 1 -- 172.21.15.122:0/865945528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2013eb20 con 0x7f3a201424f0 2024-03-06T07:27:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1e59c640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a2006da40 0x7f3a2015fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a1ed9d640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200ae590 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800c450 tx=0x7f3a1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a180074e0 con 0x7f3a201424f0 2024-03-06T07:27:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a180047e0 con 0x7f3a201424f0 2024-03-06T07:27:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a200aebc0 con 0x7f3a201424f0 2024-03-06T07:27:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004bd0 con 0x7f3a201424f0 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.952+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a200b2c60 con 0x7f3a201424f0 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.953+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a200b2a50 con 0x7f3a201424f0 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.953+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a18011070 con 0x7f3a201424f0 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.954+0000 7f3a0effd640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a0006b4d0 0x7f3a0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.954+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3a18089aa0 con 0x7f3a201424f0 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.954+0000 7f3a1e59c640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a0006b4d0 0x7f3a0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.954+0000 7f3a1e59c640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a0006b4d0 0x7f3a0006d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3a10036000 tx=0x7f3a10030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:04.958+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a18058bc0 con 0x7f3a201424f0 2024-03-06T07:27:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.197+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f3a200d0620 con 0x7f3a201424f0 2024-03-06T07:27:05.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.229+0000 7f3a0effd640 1 -- 172.21.15.122:0/865945528 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v291) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3a180584c0 con 0x7f3a201424f0 2024-03-06T07:27:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a0006b4d0 msgr2=0x7f3a0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a0006b4d0 0x7f3a0006d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3a10036000 tx=0x7f3a10030000 comp rx=0 tx=0).stop 2024-03-06T07:27:05.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 msgr2=0x7f3a200ae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200ae590 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800c450 tx=0x7f3a1800c910 comp rx=0 tx=0).stop 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 shutdown_connections 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a0006b4d0 0x7f3a0006d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a201424f0 0x7f3a200ae590 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a200cd7a0 0x7f3a200ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 --2- 172.21.15.122:0/865945528 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a2006da40 0x7f3a2015fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 >> 172.21.15.122:0/865945528 conn(0x7f3a200cf8a0 msgr2=0x7f3a200ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 shutdown_connections 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.231+0000 7f3a24f65640 1 -- 172.21.15.122:0/865945528 wait complete. 2024-03-06T07:27:05.244 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:05.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.345+0000 7f6d557a0640 1 Processor -- start 2024-03-06T07:27:05.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.345+0000 7f6d557a0640 1 -- start start 2024-03-06T07:27:05.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d557a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d5006cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d557a0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6d500d0710 0x7f6d5006d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d557a0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500d0c60 con 0x7f6d50161850 2024-03-06T07:27:05.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d557a0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5006c160 con 0x7f6d500d0710 2024-03-06T07:27:05.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d557a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5006c440 con 0x7f6d5015de90 2024-03-06T07:27:05.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d4ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6d50161850 0x7f6d50161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43434/0 (socket says 172.21.15.122:43434) 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d4ffff640 1 -- 172.21.15.122:0/3781742487 learned_addr learned my addr 172.21.15.122:0/3781742487 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d4f7fe640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d5006cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.346+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/3781742487 >> v2:172.21.15.169:3300/0 conn(0x7f6d500d0710 0x7f6d5006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281697412 0 0) 0x7f6d500d0c60 con 0x7f6d50161850 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d38003520 con 0x7f6d50161850 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221453280 0 0) 0x7f6d38003520 con 0x7f6d50161850 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 msgr2=0x7f6d5006cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d5006cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 >> v2:172.21.15.169:3300/0 conn(0x7f6d500d0710 msgr2=0x7f6d5006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 --2- 172.21.15.122:0/3781742487 >> v2:172.21.15.169:3300/0 conn(0x7f6d500d0710 0x7f6d5006d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500d0f40 con 0x7f6d50161850 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f6d40001670 con 0x7f6d50161850 2024-03-06T07:27:05.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/3781742487 >> v2:172.21.15.169:3300/0 conn(0x7f6d500d0710 0x7f6d5006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 >> v1:172.21.15.122:6789/0 conn(0x7f6d50161850 legacy=0x7f6d50161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380039b0 0x7f6d38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d38004d40 0x7f6d380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5006c160 con 0x7f6d38004d40 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5006c440 con 0x7f6d380039b0 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3800b9c0 con 0x7f6d5015de90 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d38004d40 0x7f6d380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4ffff640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380039b0 0x7f6d38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4f7fe640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 -- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 msgr2=0x7f6d38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 -- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380039b0 msgr2=0x7f6d38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380039b0 0x7f6d38003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d54f9f640 1 -- 172.21.15.122:0/3781742487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d38003520 con 0x7f6d38004d40 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.347+0000 7f6d4f7fe640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d5015de90 0x7f6d38004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.348+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d38004d40 0x7f6d380051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800e380 tx=0x7f6d38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6371a57fe68bc39a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.348+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48039070 con 0x7f6d38004d40 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.348+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4802cde0 con 0x7f6d38004d40 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.348+0000 7f6d4effd640 1 -- 172.21.15.122:0/3781742487 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48009de0 con 0x7f6d38004d40 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d557a0640 1 -- 172.21.15.122:0/3781742487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d38004d40 msgr2=0x7f6d380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d557a0640 1 --2- 172.21.15.122:0/3781742487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d38004d40 0x7f6d380051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800e380 tx=0x7f6d38005830 comp rx=0 tx=0).stop 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d54f9f640 1 -- 172.21.15.122:0/3781742487 reap_dead start 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d557a0640 1 -- 172.21.15.122:0/3781742487 shutdown_connections 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d557a0640 1 -- 172.21.15.122:0/3781742487 >> 172.21.15.122:0/3781742487 conn(0x7f6d500d1930 msgr2=0x7f6d500d1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d557a0640 1 -- 172.21.15.122:0/3781742487 shutdown_connections 2024-03-06T07:27:05.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.349+0000 7f6d557a0640 1 -- 172.21.15.122:0/3781742487 wait complete. 2024-03-06T07:27:05.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.350+0000 7f6d557a0640 1 Processor -- start 2024-03-06T07:27:05.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.350+0000 7f6d557a0640 1 -- start start 2024-03-06T07:27:05.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d500b5b60 0x7f6d500b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 0x7f6d500ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d500aebe0 0x7f6d50149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50162340 con 0x7f6d500b6480 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5006ecc0 con 0x7f6d500b5b60 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d501624c0 con 0x7f6d500aebe0 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 0x7f6d500ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 0x7f6d500ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44390/0 (socket says 172.21.15.122:44390) 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 -- 172.21.15.122:0/2177980972 learned_addr learned my addr 172.21.15.122:0/2177980972 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4ffff640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d500b5b60 0x7f6d500b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 -- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d500aebe0 msgr2=0x7f6d50149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d54f9f640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d500aebe0 0x7f6d50149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d500aebe0 0x7f6d50149dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 -- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d500b5b60 msgr2=0x7f6d500b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d500b5b60 0x7f6d500b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 -- 172.21.15.122:0/2177980972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500af410 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4f7fe640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 0x7f6d500ae6a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f6d44011730 tx=0x7f6d44011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400f960 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4400fb00 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44015460 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d500af6f0 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.351+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d500afcb0 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.352+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d501495d0 con 0x7f6d500b6480 2024-03-06T07:27:05.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.353+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d44015600 con 0x7f6d500b6480 2024-03-06T07:27:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.353+0000 7f6d4cff9640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.353+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6d4408a880 con 0x7f6d500b6480 2024-03-06T07:27:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.353+0000 7f6d4ffff640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.354+0000 7f6d4ffff640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6d40004840 tx=0x7f6d40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.358+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d501495d0 con 0x7f6d500b6480 2024-03-06T07:27:05.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.530+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6d50068f10 con 0x7f6d500b6480 2024-03-06T07:27:05.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.534+0000 7f6d4cff9640 1 -- 172.21.15.122:0/2177980972 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6d44019030 con 0x7f6d500b6480 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d3c06b4d0 msgr2=0x7f6d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6d40004840 tx=0x7f6d40009040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 msgr2=0x7f6d500ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 0x7f6d500ae6a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f6d44011730 tx=0x7f6d44011bf0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 shutdown_connections 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d500aebe0 0x7f6d50149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d500b6480 0x7f6d500ae6a0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 --2- 172.21.15.122:0/2177980972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d500b5b60 0x7f6d500b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 >> 172.21.15.122:0/2177980972 conn(0x7f6d500d1930 msgr2=0x7f6d500b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 shutdown_connections 2024-03-06T07:27:05.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.536+0000 7f6d557a0640 1 -- 172.21.15.122:0/2177980972 wait complete. 2024-03-06T07:27:05.549 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:05.549 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:05.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:05.641 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:05.641 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:27:05.641 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:05.657 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:27:05.657 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:05.674 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:27:05.674 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:05.712 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:27:05.712 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:05.732 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:27:05.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:05.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.849+0000 7fe584295640 1 Processor -- start 2024-03-06T07:27:05.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.849+0000 7fe584295640 1 -- start start 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe584295640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe57c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe584295640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c0cc980 con 0x7fe57c0cd7a0 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe584295640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c0ccc60 con 0x7fe57c06da40 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe584295640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0cd220 con 0x7fe57c1424f0 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe581830640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe57c0cd7a0 0x7fe57c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43438/0 (socket says 172.21.15.122:43438) 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe581830640 1 -- 172.21.15.122:0/4240779017 learned_addr learned my addr 172.21.15.122:0/4240779017 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe582031640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe57c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596777248 0 0) 0x7fe57c0cc980 con 0x7fe57c0cd7a0 2024-03-06T07:27:05.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.850+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe568003520 con 0x7fe57c0cd7a0 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994244087 0 0) 0x7fe568003520 con 0x7fe57c0cd7a0 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 msgr2=0x7fe57c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe57c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 >> v1:172.21.15.169:6789/0 conn(0x7fe57c06da40 legacy=0x7fe57c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c0ccf40 con 0x7fe57c0cd7a0 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fe56c001670 con 0x7fe57c0cd7a0 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe57c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 >> v1:172.21.15.122:6789/0 conn(0x7fe57c0cd7a0 legacy=0x7fe57c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5680039e0 0x7fe568003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe568004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe568004d40 0x7fe5680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0cd220 con 0x7fe5680039e0 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56800b9c0 con 0x7fe568004d40 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56800bb70 con 0x7fe57c1424f0 2024-03-06T07:27:05.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5680039e0 0x7fe568003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe581830640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe568004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582832640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe568004d40 0x7fe5680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 -- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 msgr2=0x7fe568004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe568004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 -- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe568004d40 msgr2=0x7fe5680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe568004d40 0x7fe5680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe582031640 1 -- 172.21.15.122:0/4240779017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568003520 con 0x7fe5680039e0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.851+0000 7fe581830640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c1424f0 0x7fe568004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe582031640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5680039e0 0x7fe568003dc0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe57800e350 tx=0x7fe568005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a489cdbc1445565 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578039070 con 0x7fe5680039e0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57802cd70 con 0x7fe5680039e0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe58102f640 1 -- 172.21.15.122:0/4240779017 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578009de0 con 0x7fe5680039e0 2024-03-06T07:27:05.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe584295640 1 -- 172.21.15.122:0/4240779017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5680039e0 msgr2=0x7fe568003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe584295640 1 --2- 172.21.15.122:0/4240779017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5680039e0 0x7fe568003dc0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe57800e350 tx=0x7fe568005830 comp rx=0 tx=0).stop 2024-03-06T07:27:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.852+0000 7fe582832640 1 -- 172.21.15.122:0/4240779017 reap_dead start 2024-03-06T07:27:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.853+0000 7fe584295640 1 -- 172.21.15.122:0/4240779017 shutdown_connections 2024-03-06T07:27:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.853+0000 7fe584295640 1 -- 172.21.15.122:0/4240779017 >> 172.21.15.122:0/4240779017 conn(0x7fe57c0cf8a0 msgr2=0x7fe57c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.853+0000 7fe584295640 1 -- 172.21.15.122:0/4240779017 shutdown_connections 2024-03-06T07:27:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.853+0000 7fe584295640 1 -- 172.21.15.122:0/4240779017 wait complete. 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 Processor -- start 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 -- start start 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5680039e0 0x7fe57c13cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c143fb0 0x7fe57c144410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 0x7fe57c1f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0cc470 con 0x7fe57c1f1f10 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0cc2f0 con 0x7fe5680039e0 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe584295640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0cc5f0 con 0x7fe57c143fb0 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.854+0000 7fe582832640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 0x7fe57c1f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 0x7fe57c1f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44416/0 (socket says 172.21.15.122:44416) 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 -- 172.21.15.122:0/3839099379 learned_addr learned my addr 172.21.15.122:0/3839099379 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582031640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5680039e0 0x7fe57c13cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe581830640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c143fb0 0x7fe57c144410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 -- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c143fb0 msgr2=0x7fe57c144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c143fb0 0x7fe57c144410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 -- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5680039e0 msgr2=0x7fe57c13cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5680039e0 0x7fe57c13cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 -- 172.21.15.122:0/3839099379 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c13d140 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe582832640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 0x7fe57c1f2390 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe57c145d50 tx=0x7fe57400e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574019070 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5740052e0 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c13dcf0 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574005490 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.855+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c13e2b0 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.856+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c071010 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.856+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe57401c070 con 0x7fe57c1f1f10 2024-03-06T07:27:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.857+0000 7fe572ffd640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe56006b500 0x7fe56006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.857+0000 7fe582031640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe56006b500 0x7fe56006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.857+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe574089270 con 0x7fe57c1f1f10 2024-03-06T07:27:05.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.857+0000 7fe582031640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe56006b500 0x7fe56006d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe578036000 tx=0x7fe57802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:05.861+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe57c071010 con 0x7fe57c1f1f10 2024-03-06T07:27:06.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.068+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe57c04f330 con 0x7fe57c1f1f10 2024-03-06T07:27:06.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.068+0000 7fe572ffd640 1 -- 172.21.15.122:0/3839099379 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fe574057dd0 con 0x7fe57c1f1f10 2024-03-06T07:27:06.071 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:06.071 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":56,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/287322342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":287322342},{"type":"v1","addr":"172.21.15.169:6835","nonce":287322342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6991,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1138390958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1138390958},{"type":"v1","addr":"172.21.15.169:6833","nonce":1138390958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3278594725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3278594725},{"type":"v1","addr":"172.21.15.122:6837","nonce":3278594725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/993969119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":993969119},{"type":"v1","addr":"172.21.15.122:6835","nonce":993969119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[]} 2024-03-06T07:27:06.072 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 56 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe56006b500 msgr2=0x7fe56006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe56006b500 0x7fe56006d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe578036000 tx=0x7fe57802e000 comp rx=0 tx=0).stop 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 msgr2=0x7fe57c1f2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 0x7fe57c1f2390 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe57c145d50 tx=0x7fe57400e800 comp rx=0 tx=0).stop 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 shutdown_connections 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe56006b500 0x7fe56006d9c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe57c1f1f10 0x7fe57c1f2390 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe57c143fb0 0x7fe57c144410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 --2- 172.21.15.122:0/3839099379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5680039e0 0x7fe57c13cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 >> 172.21.15.122:0/3839099379 conn(0x7fe57c0cf8a0 msgr2=0x7fe57c06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 shutdown_connections 2024-03-06T07:27:06.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.070+0000 7fe584295640 1 -- 172.21.15.122:0/3839099379 wait complete. 2024-03-06T07:27:06.084 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:06.084 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:06.084 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:06.084 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:06.084 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:06.084 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:06.085 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:06.085 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:06.085 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:06.085 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:06.085 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:06.085 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:06.085 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:06.086 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:06.086 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:06.086 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:06.093 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:06.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:06.094 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:06.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:06.097 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:06.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:06.098 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:06.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:06.136 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:06.136 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:06.137 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:06.137 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:06.140 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:06.140 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:06.140 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:06.141 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:06.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:06.213 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:27:06.213 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:27:06.222 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:27:06.222 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.247+0000 7f2dee7f7640 1 Processor -- start 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.247+0000 7f2dee7f7640 1 -- start start 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.247+0000 7f2dee7f7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2de80b4460 0x7f2de80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.247+0000 7f2dee7f7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2de8162970 0x7f2de80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2dee7f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80d22c0 con 0x7f2de8161cf0 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2dee7f7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de8161b00 con 0x7f2de80b4460 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2dee7f7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de80cecc0 con 0x7f2de8162970 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2de80b4460 0x7f2de80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2decd94640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2de8161cf0 0x7f2de80cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43446/0 (socket says 172.21.15.122:43446) 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2decd94640 1 -- 172.21.15.122:0/1601967991 learned_addr learned my addr 172.21.15.122:0/1601967991 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de77fe640 1 --2- 172.21.15.122:0/1601967991 >> v2:172.21.15.169:3300/0 conn(0x7f2de8162970 0x7f2de80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de7fff640 1 -- 172.21.15.122:0/1601967991 >> v2:172.21.15.169:3300/0 conn(0x7f2de8162970 msgr2=0x7f2de80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de7fff640 1 --2- 172.21.15.122:0/1601967991 >> v2:172.21.15.169:3300/0 conn(0x7f2de8162970 0x7f2de80b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de7fff640 1 -- 172.21.15.122:0/1601967991 >> v1:172.21.15.122:6789/0 conn(0x7f2de8161cf0 legacy=0x7f2de80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de6ffd640 1 -- 172.21.15.122:0/1601967991 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125899948 0 0) 0x7f2de80d22c0 con 0x7f2de8161cf0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.248+0000 7f2de7fff640 1 -- 172.21.15.122:0/1601967991 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80d25a0 con 0x7f2de80b4460 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2de7fff640 1 --2- 172.21.15.122:0/1601967991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de80b4460 0x7f2de80d4800 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f2de80d17e0 tx=0x7f2ddc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07b074b784f8722 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2de6ffd640 1 -- 172.21.15.122:0/1601967991 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc035020 con 0x7f2de80b4460 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2de6ffd640 1 -- 172.21.15.122:0/1601967991 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc031830 con 0x7f2de80b4460 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2de6ffd640 1 -- 172.21.15.122:0/1601967991 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc031c00 con 0x7f2de80b4460 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 -- 172.21.15.122:0/1601967991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de80b4460 msgr2=0x7f2de80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/1601967991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de80b4460 0x7f2de80d4800 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f2de80d17e0 tx=0x7f2ddc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 -- 172.21.15.122:0/1601967991 shutdown_connections 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/1601967991 >> v2:172.21.15.169:3300/0 conn(0x7f2de8162970 0x7f2de80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/1601967991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de80b4460 0x7f2de80d4800 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 -- 172.21.15.122:0/1601967991 >> 172.21.15.122:0/1601967991 conn(0x7f2de80d3ce0 msgr2=0x7f2de80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 -- 172.21.15.122:0/1601967991 shutdown_connections 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.249+0000 7f2dee7f7640 1 -- 172.21.15.122:0/1601967991 wait complete. 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.251+0000 7f2dee7f7640 1 Processor -- start 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.251+0000 7f2dee7f7640 1 -- start start 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2de80b4460 0x7f2de80cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2de8161cf0 0x7f2de80c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 0x7f2de80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80ce950 con 0x7f2de8162970 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80d2b60 con 0x7f2de8161cf0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80b3ac0 con 0x7f2de80b4460 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 0x7f2de80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 0x7f2de80b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44468/0 (socket says 172.21.15.122:44468) 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 -- 172.21.15.122:0/3021961715 learned_addr learned my addr 172.21.15.122:0/3021961715 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de77fe640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2de8161cf0 0x7f2de80c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de7fff640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2de80b4460 0x7f2de80cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 -- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2de80b4460 msgr2=0x7f2de80cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2de80b4460 0x7f2de80cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 -- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2de8161cf0 msgr2=0x7f2de80c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2de8161cf0 0x7f2de80c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 -- 172.21.15.122:0/3021961715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80c9610 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de77fe640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2de8161cf0 0x7f2de80c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de7fff640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2de80b4460 0x7f2de80cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2decd94640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 0x7f2de80b2170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f2de000c7a0 tx=0x7f2de000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0018070 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de000a640 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de000aab0 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de80c98f0 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.252+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de80c9eb0 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.253+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de804f330 con 0x7f2de8162970 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.254+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2de000ac50 con 0x7f2de8162970 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.254+0000 7f2de4ff9640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dd406b4d0 0x7f2dd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.254+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2de0089580 con 0x7f2de8162970 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.254+0000 7f2de7fff640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dd406b4d0 0x7f2dd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.255+0000 7f2de7fff640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dd406b4d0 0x7f2dd406d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc036000 tx=0x7f2ddc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.259+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2de00586a0 con 0x7f2de8162970 2024-03-06T07:27:06.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:06.445+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2de806b180 con 0x7f2de8162970 2024-03-06T07:27:07.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.239+0000 7f2de4ff9640 1 -- 172.21.15.122:0/3021961715 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2de0057fa0 con 0x7f2de8162970 2024-03-06T07:27:07.241 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dd406b4d0 msgr2=0x7f2dd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dd406b4d0 0x7f2dd406d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc036000 tx=0x7f2ddc008030 comp rx=0 tx=0).stop 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 msgr2=0x7f2de80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 0x7f2de80b2170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f2de000c7a0 tx=0x7f2de000cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 shutdown_connections 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2dd406b4d0 0x7f2dd406d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2de8162970 0x7f2de80b2170 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2de8161cf0 0x7f2de80c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 --2- 172.21.15.122:0/3021961715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2de80b4460 0x7f2de80cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 >> 172.21.15.122:0/3021961715 conn(0x7f2de80d3ce0 msgr2=0x7f2de80d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 shutdown_connections 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.242+0000 7f2dee7f7640 1 -- 172.21.15.122:0/3021961715 wait complete. 2024-03-06T07:27:07.258 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:07.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.360+0000 7fce34078640 1 Processor -- start 2024-03-06T07:27:07.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.361+0000 7fce34078640 1 -- start start 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.361+0000 7fce34078640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fce2c0b3060 0x7fce2c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce34078640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fce2c15bf30 0x7fce2c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce34078640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c15f2d0 con 0x7fce2c13dc10 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce34078640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2c0cf3f0 con 0x7fce2c0b3060 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce34078640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2c15b210 con 0x7fce2c15bf30 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fce2c0b3060 0x7fce2c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fce2c0b3060 0x7fce2c15e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44470/0 (socket says 172.21.15.122:44470) 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 -- 172.21.15.122:0/1033511439 learned_addr learned my addr 172.21.15.122:0/1033511439 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce32615640 1 --2- 172.21.15.122:0/1033511439 >> v2:172.21.15.169:3300/0 conn(0x7fce2c15bf30 0x7fce2c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 -- 172.21.15.122:0/1033511439 >> v2:172.21.15.169:3300/0 conn(0x7fce2c15bf30 msgr2=0x7fce2c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 --2- 172.21.15.122:0/1033511439 >> v2:172.21.15.169:3300/0 conn(0x7fce2c15bf30 0x7fce2c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 -- 172.21.15.122:0/1033511439 >> v1:172.21.15.169:6789/0 conn(0x7fce2c13dc10 legacy=0x7fce2c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.362+0000 7fce31613640 1 -- 172.21.15.122:0/1033511439 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c15f580 con 0x7fce2c0b3060 2024-03-06T07:27:07.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce31613640 1 --2- 172.21.15.122:0/1033511439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c0b3060 0x7fce2c15e850 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0b4e00 tx=0x7fce1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ee9d3b69725df0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce30e12640 1 -- 172.21.15.122:0/1033511439 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c02e070 con 0x7fce2c0b3060 2024-03-06T07:27:07.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce30e12640 1 -- 172.21.15.122:0/1033511439 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c037070 con 0x7fce2c0b3060 2024-03-06T07:27:07.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce30e12640 1 -- 172.21.15.122:0/1033511439 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c031a00 con 0x7fce2c0b3060 2024-03-06T07:27:07.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 -- 172.21.15.122:0/1033511439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c0b3060 msgr2=0x7fce2c15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 --2- 172.21.15.122:0/1033511439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c0b3060 0x7fce2c15e850 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0b4e00 tx=0x7fce1c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:07.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 -- 172.21.15.122:0/1033511439 shutdown_connections 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 --2- 172.21.15.122:0/1033511439 >> v2:172.21.15.169:3300/0 conn(0x7fce2c15bf30 0x7fce2c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 --2- 172.21.15.122:0/1033511439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c0b3060 0x7fce2c15e850 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 -- 172.21.15.122:0/1033511439 >> 172.21.15.122:0/1033511439 conn(0x7fce2c0cf8a0 msgr2=0x7fce2c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 -- 172.21.15.122:0/1033511439 shutdown_connections 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.363+0000 7fce34078640 1 -- 172.21.15.122:0/1033511439 wait complete. 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 Processor -- start 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 -- start start 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce2c0b3060 0x7fce2c12aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 0x7fce2c13e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce2c15bf30 0x7fce2c15d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c13dff0 con 0x7fce2c13dc10 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c15b730 con 0x7fce2c0b3060 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce34078640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0cd7a0 con 0x7fce2c15bf30 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce31613640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 0x7fce2c13e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce31613640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 0x7fce2c13e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44486/0 (socket says 172.21.15.122:44486) 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.365+0000 7fce31613640 1 -- 172.21.15.122:0/744396841 learned_addr learned my addr 172.21.15.122:0/744396841 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31e14640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce2c0b3060 0x7fce2c12aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce32615640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce2c15bf30 0x7fce2c15d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31613640 1 -- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce2c15bf30 msgr2=0x7fce2c15d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31613640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce2c15bf30 0x7fce2c15d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31613640 1 -- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce2c0b3060 msgr2=0x7fce2c12aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31613640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce2c0b3060 0x7fce2c12aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31613640 1 -- 172.21.15.122:0/744396841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c133730 con 0x7fce2c13dc10 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31e14640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce2c0b3060 0x7fce2c12aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce31613640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 0x7fce2c13e3e0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fce1c02b8c0 tx=0x7fce1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c032880 con 0x7fce2c13dc10 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c032a20 con 0x7fce2c13dc10 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c151e30 con 0x7fce2c13dc10 2024-03-06T07:27:07.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c032e10 con 0x7fce2c13dc10 2024-03-06T07:27:07.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.366+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c15c8c0 con 0x7fce2c13dc10 2024-03-06T07:27:07.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.367+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c04f330 con 0x7fce2c13dc10 2024-03-06T07:27:07.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.367+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce1c0453f0 con 0x7fce2c13dc10 2024-03-06T07:27:07.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.368+0000 7fce227fc640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce1006b4d0 0x7fce1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.368+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fce1c0abdf0 con 0x7fce2c13dc10 2024-03-06T07:27:07.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.368+0000 7fce31e14640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce1006b4d0 0x7fce1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.368+0000 7fce31e14640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce1006b4d0 0x7fce1006d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fce240045b0 tx=0x7fce24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.372+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce1c040070 con 0x7fce2c13dc10 2024-03-06T07:27:07.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.607+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fce2c06eb00 con 0x7fce2c13dc10 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.608+0000 7fce227fc640 1 -- 172.21.15.122:0/744396841 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fce1c07ab20 con 0x7fce2c13dc10 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:27:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:27:07.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.609+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce1006b4d0 msgr2=0x7fce1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.609+0000 7fce34078640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce1006b4d0 0x7fce1006d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fce240045b0 tx=0x7fce24009040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 msgr2=0x7fce2c13e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 0x7fce2c13e3e0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fce1c02b8c0 tx=0x7fce1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 shutdown_connections 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce1006b4d0 0x7fce1006d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce2c15bf30 0x7fce2c15d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce2c13dc10 0x7fce2c13e3e0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 --2- 172.21.15.122:0/744396841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce2c0b3060 0x7fce2c12aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 >> 172.21.15.122:0/744396841 conn(0x7fce2c0cf8a0 msgr2=0x7fce2c071670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 shutdown_connections 2024-03-06T07:27:07.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.610+0000 7fce34078640 1 -- 172.21.15.122:0/744396841 wait complete. 2024-03-06T07:27:07.626 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:07.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.723+0000 7fdac4feb640 1 Processor -- start 2024-03-06T07:27:07.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.723+0000 7fdac4feb640 1 -- start start 2024-03-06T07:27:07.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.724+0000 7fdac4feb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 0x7fdac015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.724+0000 7fdac4feb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac015f580 con 0x7fdac013dc10 2024-03-06T07:27:07.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.724+0000 7fdac4feb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac015b450 con 0x7fdac015bf30 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.724+0000 7fdac4feb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00cf3f0 con 0x7fdac00b3060 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.724+0000 7fdabe59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdac013dc10 0x7fdac00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43456/0 (socket says 172.21.15.122:43456) 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabe59c640 1 -- 172.21.15.122:0/3788130507 learned_addr learned my addr 172.21.15.122:0/3788130507 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabdd9b640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 0x7fdac015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780644407 0 0) 0x7fdac015f580 con 0x7fdac013dc10 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa8003520 con 0x7fdac013dc10 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335785973 0 0) 0x7fdac015b450 con 0x7fdac015bf30 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac015f580 con 0x7fdac015bf30 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129535917 0 0) 0x7fdaa8003520 con 0x7fdac013dc10 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 msgr2=0x7fdac015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 0x7fdac015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 >> v1:172.21.15.169:6789/0 conn(0x7fdac015bf30 legacy=0x7fdac015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac015b730 con 0x7fdac013dc10 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fdab8001670 con 0x7fdac013dc10 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 >> v1:172.21.15.122:6789/0 conn(0x7fdac013dc10 legacy=0x7fdac00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 0x7fdaa8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004120 0x7fdaa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa8004d40 0x7fdaa80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00cf3f0 con 0x7fdaa8004d40 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa800bb80 con 0x7fdaa8004120 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa800bd80 con 0x7fdac00b3060 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa8004d40 0x7fdaa80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabdd9b640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004120 0x7fdaa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabe59c640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 0x7fdaa8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabed9d640 1 -- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 msgr2=0x7fdaa8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac00b3060 0x7fdaa8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabed9d640 1 -- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004120 msgr2=0x7fdaa8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.725+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004120 0x7fdaa8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.726+0000 7fdabed9d640 1 -- 172.21.15.122:0/3788130507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8003520 con 0x7fdaa8004d40 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.726+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa8004d40 0x7fdaa80051c0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fdab00018d0 tx=0x7fdaa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677ce891d51f60f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.726+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0037070 con 0x7fdaa8004d40 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.726+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab00052d0 con 0x7fdaa8004d40 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.726+0000 7fdabd59a640 1 -- 172.21.15.122:0/3788130507 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab00014f0 con 0x7fdaa8004d40 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdac4feb640 1 -- 172.21.15.122:0/3788130507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa8004d40 msgr2=0x7fdaa80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3788130507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaa8004d40 0x7fdaa80051c0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fdab00018d0 tx=0x7fdaa8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdabed9d640 1 -- 172.21.15.122:0/3788130507 reap_dead start 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdac4feb640 1 -- 172.21.15.122:0/3788130507 shutdown_connections 2024-03-06T07:27:07.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdac4feb640 1 -- 172.21.15.122:0/3788130507 >> 172.21.15.122:0/3788130507 conn(0x7fdac00cf8a0 msgr2=0x7fdac00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdac4feb640 1 -- 172.21.15.122:0/3788130507 shutdown_connections 2024-03-06T07:27:07.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.727+0000 7fdac4feb640 1 -- 172.21.15.122:0/3788130507 wait complete. 2024-03-06T07:27:07.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.728+0000 7fdac4feb640 1 Processor -- start 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.728+0000 7fdac4feb640 1 -- start start 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdac4feb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004d40 0x7fdac01568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdac4feb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac01f1310 0x7fdac0153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdac4feb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 0x7fdac015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdac4feb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac015e850 con 0x7fdac01efe10 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdac4feb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac015b210 con 0x7fdaa8004d40 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdac4feb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac015e9d0 con 0x7fdac01f1310 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 0x7fdac015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 0x7fdac015d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44496/0 (socket says 172.21.15.122:44496) 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 -- 172.21.15.122:0/3752424729 learned_addr learned my addr 172.21.15.122:0/3752424729 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabdd9b640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac01f1310 0x7fdac0153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabe59c640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004d40 0x7fdac01568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 -- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac01f1310 msgr2=0x7fdac0153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac01f1310 0x7fdac0153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 -- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004d40 msgr2=0x7fdac01568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004d40 0x7fdac01568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.729+0000 7fdabed9d640 1 -- 172.21.15.122:0/3752424729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac0155b60 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.730+0000 7fdabed9d640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 0x7fdac015d730 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fdab0035040 tx=0x7fdab00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.730+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab00014f0 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.730+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab0005480 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.730+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab00035f0 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.730+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac0154410 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.730+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac01560e0 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.731+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac004f330 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.731+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdab0003790 con 0x7fdac01efe10 2024-03-06T07:27:07.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.732+0000 7fdaaeffd640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdaa006b4d0 0x7fdaa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.732+0000 7fdabe59c640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdaa006b4d0 0x7fdaa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.732+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdab00ad820 con 0x7fdac01efe10 2024-03-06T07:27:07.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.732+0000 7fdabe59c640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdaa006b4d0 0x7fdaa006d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fdab8007b20 tx=0x7fdab8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.736+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdab004a060 con 0x7fdac01efe10 2024-03-06T07:27:07.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.909+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdac006eb00 con 0x7fdac01efe10 2024-03-06T07:27:07.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.910+0000 7fdaaeffd640 1 -- 172.21.15.122:0/3752424729 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12893 (secure 0 0 0) 0x7fdab007c960 con 0x7fdac01efe10 2024-03-06T07:27:07.912 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:07.912 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":104,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:07.239009+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:07.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdaa006b4d0 msgr2=0x7fdaa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdaa006b4d0 0x7fdaa006d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fdab8007b20 tx=0x7fdab8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 msgr2=0x7fdac015d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 0x7fdac015d730 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fdab0035040 tx=0x7fdab00012d0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 shutdown_connections 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdaa006b4d0 0x7fdaa006d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdac01efe10 0x7fdac015d730 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdac01f1310 0x7fdac0153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 --2- 172.21.15.122:0/3752424729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdaa8004d40 0x7fdac01568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 >> 172.21.15.122:0/3752424729 conn(0x7fdac00cf8a0 msgr2=0x7fdac0072210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 shutdown_connections 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:07.912+0000 7fdac4feb640 1 -- 172.21.15.122:0/3752424729 wait complete. 2024-03-06T07:27:07.925 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:07.926 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:08.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.026+0000 7fa5bad7a640 1 Processor -- start 2024-03-06T07:27:08.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.026+0000 7fa5bad7a640 1 -- start start 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.026+0000 7fa5bad7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5b40b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.026+0000 7fa5bad7a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b4158ac0 con 0x7fa5b40cfa90 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.026+0000 7fa5bad7a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b415b2a0 con 0x7fa5b41586e0 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.026+0000 7fa5bad7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415b540 con 0x7fa5b40cff40 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa5b40cfa90 0x7fa5b40b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43464/0 (socket says 172.21.15.122:43464) 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b3fff640 1 -- 172.21.15.122:0/2132017374 learned_addr learned my addr 172.21.15.122:0/2132017374 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b9317640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5b40b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183009991 0 0) 0x7fa5b4158ac0 con 0x7fa5b40cfa90 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4003520 con 0x7fa5b40cfa90 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176782099 0 0) 0x7fa5b415b2a0 con 0x7fa5b41586e0 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b4158ac0 con 0x7fa5b41586e0 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663048457 0 0) 0x7fa5a4003520 con 0x7fa5b40cfa90 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 msgr2=0x7fa5b40b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5b40b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 >> v1:172.21.15.169:6789/0 conn(0x7fa5b41586e0 legacy=0x7fa5b40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b415b820 con 0x7fa5b40cfa90 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa5a0001670 con 0x7fa5b40cfa90 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 >> v1:172.21.15.122:6789/0 conn(0x7fa5b40cfa90 legacy=0x7fa5b40b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5a4003b70 0x7fa5a4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415b540 con 0x7fa5a4004800 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a400dc80 con 0x7fa5a4003b70 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a400de50 con 0x7fa5b40cff40 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b8b16640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5a4003b70 0x7fa5a4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.027+0000 7fa5b9317640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b3fff640 1 -- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 msgr2=0x7fa5a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5a40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b3fff640 1 -- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5a4003b70 msgr2=0x7fa5a4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5a4003b70 0x7fa5a4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b3fff640 1 -- 172.21.15.122:0/2132017374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a4003520 con 0x7fa5a4004800 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b8b16640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5a4003b70 0x7fa5a4003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b9317640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b40cff40 0x7fa5a40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5a4004c80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c210 tx=0x7fa5a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da7862d2c182cda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0007ce0 con 0x7fa5a4004800 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0030070 con 0x7fa5a4004800 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.028+0000 7fa5b37fe640 1 -- 172.21.15.122:0/2132017374 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0005d50 con 0x7fa5a4004800 2024-03-06T07:27:08.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5bad7a640 1 -- 172.21.15.122:0/2132017374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 msgr2=0x7fa5a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/2132017374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5a4004c80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c210 tx=0x7fa5a4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5b9317640 1 -- 172.21.15.122:0/2132017374 reap_dead start 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5bad7a640 1 -- 172.21.15.122:0/2132017374 shutdown_connections 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5bad7a640 1 -- 172.21.15.122:0/2132017374 >> 172.21.15.122:0/2132017374 conn(0x7fa5b406b190 msgr2=0x7fa5b406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5bad7a640 1 -- 172.21.15.122:0/2132017374 shutdown_connections 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.029+0000 7fa5bad7a640 1 -- 172.21.15.122:0/2132017374 wait complete. 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.030+0000 7fa5bad7a640 1 Processor -- start 2024-03-06T07:27:08.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.030+0000 7fa5bad7a640 1 -- start start 2024-03-06T07:27:08.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5bad7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5b4145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5bad7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b41491d0 0x7fa5b4150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5bad7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b41512c0 0x7fa5b40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5bad7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40d0360 con 0x7fa5a4004800 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5bad7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415bde0 con 0x7fa5b41491d0 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5bad7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40af150 con 0x7fa5b41512c0 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5b4145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5b4145520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44514/0 (socket says 172.21.15.122:44514) 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 -- 172.21.15.122:0/538214025 learned_addr learned my addr 172.21.15.122:0/538214025 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 -- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b41512c0 msgr2=0x7fa5b40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b9317640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b41512c0 0x7fa5b40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b41512c0 0x7fa5b40ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 -- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b41491d0 msgr2=0x7fa5b4150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b41491d0 0x7fa5b4150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 -- 172.21.15.122:0/538214025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4145a60 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.031+0000 7fa5b8b16640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5b4145520 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800e780 tx=0x7fa5a800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.032+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80050d0 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.032+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8005270 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.032+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8005420 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.032+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b4151880 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.032+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b4155fd0 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.033+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b40732b0 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.033+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5a801a020 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.033+0000 7fa5b17fa640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa59806b4d0 0x7fa59806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.034+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa5a8089680 con 0x7fa5a4004800 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.034+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa59806b4d0 0x7fa59806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.034+0000 7fa5b3fff640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa59806b4d0 0x7fa59806d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0034000 tx=0x7fa5a0039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.038+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5a8058880 con 0x7fa5a4004800 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.209+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5b40b3de0 con 0x7fa5a4004800 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.209+0000 7fa5b17fa640 1 -- 172.21.15.122:0/538214025 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5a8058180 con 0x7fa5a4004800 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa59806b4d0 msgr2=0x7fa59806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa59806b4d0 0x7fa59806d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0034000 tx=0x7fa5a0039040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 msgr2=0x7fa5b4145520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5b4145520 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800e780 tx=0x7fa5a800ec40 comp rx=0 tx=0).stop 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 shutdown_connections 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa59806b4d0 0x7fa59806d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa5b41512c0 0x7fa5b40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5b41491d0 0x7fa5b4150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 --2- 172.21.15.122:0/538214025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5a4004800 0x7fa5b4145520 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 >> 172.21.15.122:0/538214025 conn(0x7fa5b406b190 msgr2=0x7fa5b406dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 shutdown_connections 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.211+0000 7fa5bad7a640 1 -- 172.21.15.122:0/538214025 wait complete. 2024-03-06T07:27:08.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:08.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.323+0000 7f4b56bd7640 1 Processor -- start 2024-03-06T07:27:08.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.323+0000 7f4b56bd7640 1 -- start start 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b56bd7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b501424f0 0x7f4b500cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b56bd7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4b500cd7a0 0x7f4b5015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b56bd7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b500cc980 con 0x7f4b5006d330 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b56bd7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b500cd220 con 0x7f4b500cd7a0 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b56bd7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b500ccc60 con 0x7f4b501424f0 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b4ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4b500cd7a0 0x7f4b5015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b55174640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4b5006d330 0x7f4b50071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43466/0 (socket says 172.21.15.122:43466) 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b55174640 1 -- 172.21.15.122:0/1523550645 learned_addr learned my addr 172.21.15.122:0/1523550645 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b54973640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b501424f0 0x7f4b500cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b4ffff640 1 -- 172.21.15.122:0/1523550645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b501424f0 msgr2=0x7f4b500cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b4ffff640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b501424f0 0x7f4b500cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b4ffff640 1 -- 172.21.15.122:0/1523550645 >> v1:172.21.15.122:6789/0 conn(0x7f4b5006d330 legacy=0x7f4b50071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b4ffff640 1 -- 172.21.15.122:0/1523550645 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b500ccf40 con 0x7f4b500cd7a0 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b4f7fe640 1 -- 172.21.15.122:0/1523550645 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931858571 0 0) 0x7f4b500cc980 con 0x7f4b5006d330 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.324+0000 7f4b54973640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b501424f0 0x7f4b500cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b4ffff640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b500cd7a0 0x7f4b5015cef0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0097a0 tx=0x7f4b3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2258dbfb6641ae15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b4f7fe640 1 -- 172.21.15.122:0/1523550645 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c035020 con 0x7f4b500cd7a0 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b4f7fe640 1 -- 172.21.15.122:0/1523550645 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c031830 con 0x7f4b500cd7a0 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b4f7fe640 1 -- 172.21.15.122:0/1523550645 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c031c00 con 0x7f4b500cd7a0 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b56bd7640 1 -- 172.21.15.122:0/1523550645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b500cd7a0 msgr2=0x7f4b5015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b500cd7a0 0x7f4b5015cef0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0097a0 tx=0x7f4b3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b56bd7640 1 -- 172.21.15.122:0/1523550645 shutdown_connections 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b500cd7a0 0x7f4b5015cef0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/1523550645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b501424f0 0x7f4b500cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.325+0000 7f4b56bd7640 1 -- 172.21.15.122:0/1523550645 >> 172.21.15.122:0/1523550645 conn(0x7f4b500cf8a0 msgr2=0x7f4b500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.326+0000 7f4b56bd7640 1 -- 172.21.15.122:0/1523550645 shutdown_connections 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.326+0000 7f4b56bd7640 1 -- 172.21.15.122:0/1523550645 wait complete. 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 Processor -- start 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 -- start start 2024-03-06T07:27:08.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5006d330 0x7f4b501451f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b500cd7a0 0x7f4b501433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 0x7f4b501f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5006d940 con 0x7f4b501424f0 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5006d7c0 con 0x7f4b5006d330 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.327+0000 7f4b56bd7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5006dac0 con 0x7f4b500cd7a0 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 0x7f4b501f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 0x7f4b501f2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44536/0 (socket says 172.21.15.122:44536) 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 -- 172.21.15.122:0/895932722 learned_addr learned my addr 172.21.15.122:0/895932722 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b54973640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5006d330 0x7f4b501451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b4ffff640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b500cd7a0 0x7f4b501433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 -- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b500cd7a0 msgr2=0x7f4b501433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b500cd7a0 0x7f4b501433c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 -- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5006d330 msgr2=0x7f4b501451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5006d330 0x7f4b501451f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 -- 172.21.15.122:0/895932722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b501599f0 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b55174640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 0x7f4b501f2020 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800c730 tx=0x7f4b4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48018070 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4800ce40 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4800a860 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50159cd0 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.328+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5015a290 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.329+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5004f330 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.330+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b4801b070 con 0x7f4b501424f0 2024-03-06T07:27:08.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.330+0000 7f4b4d7fa640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b3406b4d0 0x7f4b3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.330+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b48089400 con 0x7f4b501424f0 2024-03-06T07:27:08.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.330+0000 7f4b54973640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b3406b4d0 0x7f4b3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.331+0000 7f4b54973640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b3406b4d0 0x7f4b3406d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.334+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b48057f80 con 0x7f4b501424f0 2024-03-06T07:27:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.506+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4b501605a0 con 0x7f4b501424f0 2024-03-06T07:27:08.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.507+0000 7f4b4d7fa640 1 -- 172.21.15.122:0/895932722 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4b4805c020 con 0x7f4b501424f0 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:08.536 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:27:08.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.508+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b3406b4d0 msgr2=0x7f4b3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.508+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b3406b4d0 0x7f4b3406d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44008040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.508+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 msgr2=0x7f4b501f2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.508+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 0x7f4b501f2020 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800c730 tx=0x7f4b4800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 shutdown_connections 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b3406b4d0 0x7f4b3406d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b501424f0 0x7f4b501f2020 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b500cd7a0 0x7f4b501433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 --2- 172.21.15.122:0/895932722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b5006d330 0x7f4b501451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 >> 172.21.15.122:0/895932722 conn(0x7f4b500cf8a0 msgr2=0x7f4b50141830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 shutdown_connections 2024-03-06T07:27:08.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.509+0000 7f4b56bd7640 1 -- 172.21.15.122:0/895932722 wait complete. 2024-03-06T07:27:08.538 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:08.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:08.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.640+0000 7f99f5c88640 1 Processor -- start 2024-03-06T07:27:08.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99f5c88640 1 -- start start 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99f5c88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99f015dfa0 0x7f99f00d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99f5c88640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f99f00d29f0 0x7f99f00cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99f5c88640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f015e380 con 0x7f99f00d4630 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99f5c88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f00d2100 con 0x7f99f015dfa0 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99f5c88640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f00d1e30 con 0x7f99f00d29f0 2024-03-06T07:27:08.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99ef7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99f015dfa0 0x7f99f00d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99ef7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99f015dfa0 0x7f99f00d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44540/0 (socket says 172.21.15.122:44540) 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.641+0000 7f99ef7fe640 1 -- 172.21.15.122:0/3089523744 learned_addr learned my addr 172.21.15.122:0/3089523744 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.642+0000 7f99effff640 1 --2- 172.21.15.122:0/3089523744 >> v2:172.21.15.169:3300/0 conn(0x7f99f00d29f0 0x7f99f00cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.642+0000 7f99ef7fe640 1 -- 172.21.15.122:0/3089523744 >> v2:172.21.15.169:3300/0 conn(0x7f99f00d29f0 msgr2=0x7f99f00cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.642+0000 7f99ef7fe640 1 --2- 172.21.15.122:0/3089523744 >> v2:172.21.15.169:3300/0 conn(0x7f99f00d29f0 0x7f99f00cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.642+0000 7f99ef7fe640 1 -- 172.21.15.122:0/3089523744 >> v1:172.21.15.169:6789/0 conn(0x7f99f00d4630 legacy=0x7f99f00d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.642+0000 7f99ef7fe640 1 -- 172.21.15.122:0/3089523744 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00cf470 con 0x7f99f015dfa0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.642+0000 7f99ef7fe640 1 --2- 172.21.15.122:0/3089523744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f00d18f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f99f00ce580 tx=0x7f99e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640cbd3d7086db26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99ee7fc640 1 -- 172.21.15.122:0/3089523744 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e002e070 con 0x7f99f015dfa0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99ee7fc640 1 -- 172.21.15.122:0/3089523744 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e0037070 con 0x7f99f015dfa0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99ee7fc640 1 -- 172.21.15.122:0/3089523744 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0031a00 con 0x7f99f015dfa0 2024-03-06T07:27:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 -- 172.21.15.122:0/3089523744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 msgr2=0x7f99f00d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 --2- 172.21.15.122:0/3089523744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f00d18f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f99f00ce580 tx=0x7f99e002b880 comp rx=0 tx=0).stop 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 -- 172.21.15.122:0/3089523744 shutdown_connections 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 --2- 172.21.15.122:0/3089523744 >> v2:172.21.15.169:3300/0 conn(0x7f99f00d29f0 0x7f99f00cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 --2- 172.21.15.122:0/3089523744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f00d18f0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 -- 172.21.15.122:0/3089523744 >> 172.21.15.122:0/3089523744 conn(0x7f99f009d000 msgr2=0x7f99f00d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 -- 172.21.15.122:0/3089523744 shutdown_connections 2024-03-06T07:27:08.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.643+0000 7f99f5c88640 1 -- 172.21.15.122:0/3089523744 wait complete. 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.644+0000 7f99f5c88640 1 Processor -- start 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.644+0000 7f99f5c88640 1 -- start start 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99f5c88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99f00d29f0 0x7f99f014c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99f5c88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99f00d4630 0x7f99f014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99f5c88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f009f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99f5c88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00cb6d0 con 0x7f99f015dfa0 2024-03-06T07:27:08.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99f5c88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00cfa30 con 0x7f99f00d4630 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99f5c88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00cb850 con 0x7f99f00d29f0 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f009f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f009f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44550/0 (socket says 172.21.15.122:44550) 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99effff640 1 -- 172.21.15.122:0/1985649710 learned_addr learned my addr 172.21.15.122:0/1985649710 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99ef7fe640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99f00d29f0 0x7f99f014c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.645+0000 7f99eeffd640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99f00d4630 0x7f99f014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99effff640 1 -- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99f00d29f0 msgr2=0x7f99f014c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99effff640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99f00d29f0 0x7f99f014c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99effff640 1 -- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99f00d4630 msgr2=0x7f99f014f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99effff640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99f00d4630 0x7f99f014f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99effff640 1 -- 172.21.15.122:0/1985649710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f014ced0 con 0x7f99f015dfa0 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99eeffd640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99f00d4630 0x7f99f014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99ef7fe640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99f00d29f0 0x7f99f014c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99effff640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f009f2c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f99e8007e30 tx=0x7f99e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8014620 con 0x7f99f015dfa0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8019070 con 0x7f99f015dfa0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f014cbd0 con 0x7f99f015dfa0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8014a00 con 0x7f99f015dfa0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.646+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f014d170 con 0x7f99f015dfa0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.647+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f004f330 con 0x7f99f015dfa0 2024-03-06T07:27:08.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.647+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99e8014ba0 con 0x7f99f015dfa0 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.648+0000 7f99d7fff640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99dc06b4d0 0x7f99dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.648+0000 7f99ef7fe640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99dc06b4d0 0x7f99dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.648+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f99e808a770 con 0x7f99f015dfa0 2024-03-06T07:27:08.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.648+0000 7f99ef7fe640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99dc06b4d0 0x7f99dc06d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f99e0036000 tx=0x7f99e003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.652+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99e8059940 con 0x7f99f015dfa0 2024-03-06T07:27:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:08.824+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f99f006d640 con 0x7f99f015dfa0 2024-03-06T07:27:09.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.252+0000 7f99d7fff640 1 -- 172.21.15.122:0/1985649710 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f99e8059240 con 0x7f99f015dfa0 2024-03-06T07:27:09.254 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99dc06b4d0 msgr2=0x7f99dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99dc06b4d0 0x7f99dc06d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f99e0036000 tx=0x7f99e003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 msgr2=0x7f99f009f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f009f2c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f99e8007e30 tx=0x7f99e800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 shutdown_connections 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99dc06b4d0 0x7f99dc06d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99f015dfa0 0x7f99f009f2c0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99f00d4630 0x7f99f014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 --2- 172.21.15.122:0/1985649710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99f00d29f0 0x7f99f014c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 >> 172.21.15.122:0/1985649710 conn(0x7f99f009d000 msgr2=0x7f99f01605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 shutdown_connections 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.254+0000 7f99f5c88640 1 -- 172.21.15.122:0/1985649710 wait complete. 2024-03-06T07:27:09.271 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:09.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.376+0000 7f230d35e640 1 Processor -- start 2024-03-06T07:27:09.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 -- start start 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f23081614d0 0x7f23080d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23080d1910 0x7f23080d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23080d22b0 0x7f23080d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080d2740 con 0x7f23080d22b0 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23081618b0 con 0x7f23081614d0 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.377+0000 7f230d35e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23080cf400 con 0x7f23080d1910 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23080d22b0 0x7f23080d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23080d22b0 0x7f23080d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44566/0 (socket says 172.21.15.122:44566) 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 -- 172.21.15.122:0/1706775936 learned_addr learned my addr 172.21.15.122:0/1706775936 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f2306ffd640 1 --2- 172.21.15.122:0/1706775936 >> v2:172.21.15.169:3300/0 conn(0x7f23081614d0 0x7f23080d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23067fc640 1 --2- 172.21.15.122:0/1706775936 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23080d1910 0x7f23080d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 -- 172.21.15.122:0/1706775936 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23080d1910 msgr2=0x7f23080d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 --2- 172.21.15.122:0/1706775936 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23080d1910 0x7f23080d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 -- 172.21.15.122:0/1706775936 >> v2:172.21.15.169:3300/0 conn(0x7f23081614d0 msgr2=0x7f23080d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 --2- 172.21.15.122:0/1706775936 >> v2:172.21.15.169:3300/0 conn(0x7f23081614d0 0x7f23080d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.378+0000 7f23077fe640 1 -- 172.21.15.122:0/1706775936 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080cf6a0 con 0x7f23080d22b0 2024-03-06T07:27:09.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f23077fe640 1 --2- 172.21.15.122:0/1706775936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d22b0 0x7f23080d06a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f230808c470 tx=0x7f230002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd8bcdc9d4e1e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f2305ffb640 1 -- 172.21.15.122:0/1706775936 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300035020 con 0x7f23080d22b0 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f2305ffb640 1 -- 172.21.15.122:0/1706775936 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2300031870 con 0x7f23080d22b0 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f2305ffb640 1 -- 172.21.15.122:0/1706775936 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300031c40 con 0x7f23080d22b0 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 -- 172.21.15.122:0/1706775936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d22b0 msgr2=0x7f23080d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 --2- 172.21.15.122:0/1706775936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d22b0 0x7f23080d06a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f230808c470 tx=0x7f230002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 -- 172.21.15.122:0/1706775936 shutdown_connections 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 --2- 172.21.15.122:0/1706775936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d22b0 0x7f23080d06a0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 --2- 172.21.15.122:0/1706775936 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23080d1910 0x7f23080d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 --2- 172.21.15.122:0/1706775936 >> v2:172.21.15.169:3300/0 conn(0x7f23081614d0 0x7f23080d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 -- 172.21.15.122:0/1706775936 >> 172.21.15.122:0/1706775936 conn(0x7f23080d3d40 msgr2=0x7f23080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 -- 172.21.15.122:0/1706775936 shutdown_connections 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.379+0000 7f230d35e640 1 -- 172.21.15.122:0/1706775936 wait complete. 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 Processor -- start 2024-03-06T07:27:09.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 -- start start 2024-03-06T07:27:09.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 0x7f23080c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23080d22b0 0x7f230815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23081614d0 0x7f2308159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080b0910 con 0x7f23080d1910 2024-03-06T07:27:09.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080cfc60 con 0x7f23080d22b0 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f230d35e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23080d0be0 con 0x7f23081614d0 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 0x7f23080c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 0x7f23080c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44580/0 (socket says 172.21.15.122:44580) 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.381+0000 7f2306ffd640 1 -- 172.21.15.122:0/1030698076 learned_addr learned my addr 172.21.15.122:0/1030698076 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f23067fc640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23080d22b0 0x7f230815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f23077fe640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23081614d0 0x7f2308159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f2306ffd640 1 -- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23081614d0 msgr2=0x7f2308159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f2306ffd640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23081614d0 0x7f2308159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f2306ffd640 1 -- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23080d22b0 msgr2=0x7f230815ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f2306ffd640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23080d22b0 0x7f230815ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f2306ffd640 1 -- 172.21.15.122:0/1030698076 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080cdf10 con 0x7f23080d1910 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f2306ffd640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 0x7f23080c8860 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f22f8007e40 tx=0x7f22f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8014620 con 0x7f23080d1910 2024-03-06T07:27:09.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8019070 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230815b4c0 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8014a00 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.382+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230815ba80 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.383+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2308068ff0 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.383+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22f8014ba0 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.384+0000 7f22ef7fe640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f22f406b4d0 0x7f22f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.384+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f22f808a6c0 con 0x7f23080d1910 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.384+0000 7f23067fc640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f22f406b4d0 0x7f22f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.384+0000 7f23067fc640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f22f406b4d0 0x7f22f406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f230815cd30 tx=0x7f22fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.389+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22f80590d0 con 0x7f23080d1910 2024-03-06T07:27:09.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:09.560+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f23080d0730 con 0x7f23080d1910 2024-03-06T07:27:10.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.257+0000 7f22ef7fe640 1 -- 172.21.15.122:0/1030698076 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f22f805d170 con 0x7f23080d1910 2024-03-06T07:27:10.259 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f22f406b4d0 msgr2=0x7f22f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f22f406b4d0 0x7f22f406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f230815cd30 tx=0x7f22fc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 msgr2=0x7f23080c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 0x7f23080c8860 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f22f8007e40 tx=0x7f22f800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 shutdown_connections 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f22f406b4d0 0x7f22f406d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23081614d0 0x7f2308159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23080d22b0 0x7f230815ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 --2- 172.21.15.122:0/1030698076 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23080d1910 0x7f23080c8860 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 >> 172.21.15.122:0/1030698076 conn(0x7f23080d3d40 msgr2=0x7f23080d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 shutdown_connections 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.260+0000 7f230d35e640 1 -- 172.21.15.122:0/1030698076 wait complete. 2024-03-06T07:27:10.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:10.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.383+0000 7f313f49d640 1 Processor -- start 2024-03-06T07:27:10.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.383+0000 7f313f49d640 1 -- start start 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313f49d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3138158420 0x7f31380b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313f49d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3138158800 0x7f31380b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313f49d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313815e8a0 con 0x7f313815b370 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313f49d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313815e650 con 0x7f3138158420 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313f49d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313815eb80 con 0x7f3138158800 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313da3a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f313815b370 0x7f313815b820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43480/0 (socket says 172.21.15.122:43480) 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313da3a640 1 -- 172.21.15.122:0/1601591161 learned_addr learned my addr 172.21.15.122:0/1601591161 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698293392 0 0) 0x7f313815e8a0 con 0x7f313815b370 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3124003520 con 0x7f313815b370 2024-03-06T07:27:10.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313ca38640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3138158800 0x7f31380b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.384+0000 7f313d239640 1 --2- 172.21.15.122:0/1601591161 >> v2:172.21.15.169:3300/0 conn(0x7f3138158420 0x7f31380b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1214648460 0 0) 0x7f3124003520 con 0x7f313815b370 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3138158800 msgr2=0x7f31380b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3138158800 0x7f31380b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 >> v2:172.21.15.169:3300/0 conn(0x7f3138158420 msgr2=0x7f31380b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 --2- 172.21.15.122:0/1601591161 >> v2:172.21.15.169:3300/0 conn(0x7f3138158420 0x7f31380b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313815ee60 con 0x7f313815b370 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f31300019f0 con 0x7f313815b370 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 >> v1:172.21.15.122:6789/0 conn(0x7f313815b370 legacy=0x7f313815b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31240039b0 0x7f3124003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f3124004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3124004e50 0x7f31240052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313815e650 con 0x7f3124004310 2024-03-06T07:27:10.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313815eb80 con 0x7f3124004e50 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312400bde0 con 0x7f31240039b0 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313ca38640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f3124004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313da3a640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3124004e50 0x7f31240052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313d239640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31240039b0 0x7f3124003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313ca38640 1 -- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31240039b0 msgr2=0x7f3124003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313ca38640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31240039b0 0x7f3124003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313ca38640 1 -- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3124004e50 msgr2=0x7f31240052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313ca38640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3124004e50 0x7f31240052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.385+0000 7f313ca38640 1 -- 172.21.15.122:0/1601591161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124003520 con 0x7f3124004310 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.386+0000 7f313ca38640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f3124004800 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3128009ac0 tx=0x7f3124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f3df6657d1785b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.386+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128039070 con 0x7f3124004310 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.386+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312802c6b0 con 0x7f3124004310 2024-03-06T07:27:10.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.386+0000 7f312ffff640 1 -- 172.21.15.122:0/1601591161 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128009690 con 0x7f3124004310 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313f49d640 1 -- 172.21.15.122:0/1601591161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 msgr2=0x7f3124004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313f49d640 1 --2- 172.21.15.122:0/1601591161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f3124004800 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3128009ac0 tx=0x7f3124005830 comp rx=0 tx=0).stop 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313da3a640 1 -- 172.21.15.122:0/1601591161 reap_dead start 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313f49d640 1 -- 172.21.15.122:0/1601591161 shutdown_connections 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313f49d640 1 -- 172.21.15.122:0/1601591161 >> 172.21.15.122:0/1601591161 conn(0x7f313806afb0 msgr2=0x7f313806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313f49d640 1 -- 172.21.15.122:0/1601591161 shutdown_connections 2024-03-06T07:27:10.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.387+0000 7f313f49d640 1 -- 172.21.15.122:0/1601591161 wait complete. 2024-03-06T07:27:10.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.388+0000 7f313f49d640 1 Processor -- start 2024-03-06T07:27:10.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.388+0000 7f313f49d640 1 -- start start 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f31380ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31380adda0 0x7f3138155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3138156300 0x7f3138156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380b3c80 con 0x7f3124004310 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380b3b00 con 0x7f3138156300 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380b3e00 con 0x7f31380adda0 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f31380ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f31380ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44584/0 (socket says 172.21.15.122:44584) 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 -- 172.21.15.122:0/927013800 learned_addr learned my addr 172.21.15.122:0/927013800 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 -- 172.21.15.122:0/927013800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31380adda0 msgr2=0x7f3138155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31380adda0 0x7f3138155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 -- 172.21.15.122:0/927013800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3138156300 msgr2=0x7f3138156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3138156300 0x7f3138156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 -- 172.21.15.122:0/927013800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138146940 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313d239640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f31380ad860 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3134011990 tx=0x7f3134011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313400fa50 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313400fbf0 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134014440 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.389+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3138150f80 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.390+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3138151540 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.390+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31380ae1e0 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.391+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3134012040 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.391+0000 7f312dffb640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311c06b4a0 0x7f311c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.391+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f313408a850 con 0x7f3124004310 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.391+0000 7f313ca38640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311c06b4a0 0x7f311c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.392+0000 7f313ca38640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311c06b4a0 0x7f311c06d960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3128036000 tx=0x7f312802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.396+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3134019030 con 0x7f3124004310 2024-03-06T07:27:10.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:10.602+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f313804f330 con 0x7f3124004310 2024-03-06T07:27:11.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.259+0000 7f312dffb640 1 -- 172.21.15.122:0/927013800 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v59) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3134059730 con 0x7f3124004310 2024-03-06T07:27:11.261 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:27:11.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311c06b4a0 msgr2=0x7f311c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311c06b4a0 0x7f311c06d960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3128036000 tx=0x7f312802e000 comp rx=0 tx=0).stop 2024-03-06T07:27:11.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 msgr2=0x7f31380ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f31380ad860 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3134011990 tx=0x7f3134011e50 comp rx=0 tx=0).stop 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 shutdown_connections 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f311c06b4a0 0x7f311c06d960 secure :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3128036000 tx=0x7f312802e000 comp rx=0 tx=0).stop 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3138156300 0x7f3138156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31380adda0 0x7f3138155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 --2- 172.21.15.122:0/927013800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3124004310 0x7f31380ad860 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.263+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 >> 172.21.15.122:0/927013800 conn(0x7f313806afb0 msgr2=0x7f31380b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.265+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 shutdown_connections 2024-03-06T07:27:11.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.266+0000 7f313f49d640 1 -- 172.21.15.122:0/927013800 wait complete. 2024-03-06T07:27:11.295 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:11.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.396+0000 7fb2ef288640 1 Processor -- start 2024-03-06T07:27:11.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.396+0000 7fb2ef288640 1 -- start start 2024-03-06T07:27:11.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ef288640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb2e80d4760 0x7fb2e80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ef288640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80d25d0 con 0x7fb2e80d0a80 2024-03-06T07:27:11.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ef288640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80d2870 con 0x7fb2e80b4170 2024-03-06T07:27:11.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ef288640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e80d4bf0 con 0x7fb2e80d4760 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ed825640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb2e80d0a80 0x7fb2e80cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43482/0 (socket says 172.21.15.122:43482) 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ed825640 1 -- 172.21.15.122:0/501793539 learned_addr learned my addr 172.21.15.122:0/501793539 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245190140 0 0) 0x7fb2e80d25d0 con 0x7fb2e80d0a80 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d4003520 con 0x7fb2e80d0a80 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ec823640 1 --2- 172.21.15.122:0/501793539 >> v2:172.21.15.169:3300/0 conn(0x7fb2e80d4760 0x7fb2e80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159216896 0 0) 0x7fb2d4003520 con 0x7fb2e80d0a80 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 >> v2:172.21.15.169:3300/0 conn(0x7fb2e80d4760 msgr2=0x7fb2e80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 --2- 172.21.15.122:0/501793539 >> v2:172.21.15.169:3300/0 conn(0x7fb2e80d4760 0x7fb2e80b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 >> v1:172.21.15.169:6789/0 conn(0x7fb2e80b4170 legacy=0x7fb2e80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80d2b50 con 0x7fb2e80d0a80 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb2e0001a10 con 0x7fb2e80d0a80 2024-03-06T07:27:11.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 >> v1:172.21.15.122:6789/0 conn(0x7fb2e80d0a80 legacy=0x7fb2e80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2d40039b0 0x7fb2d4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d4004800 0x7fb2d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2d40051c0 0x7fb2d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80d4bf0 con 0x7fb2d40039b0 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d400c0f0 con 0x7fb2d4004800 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d400c2f0 con 0x7fb2d40051c0 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ed024640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2d40039b0 0x7fb2d4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.397+0000 7fb2ec823640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d4004800 0x7fb2d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed825640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2d40051c0 0x7fb2d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed024640 1 -- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2d40051c0 msgr2=0x7fb2d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed024640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2d40051c0 0x7fb2d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed024640 1 -- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d4004800 msgr2=0x7fb2d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed024640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d4004800 0x7fb2d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed024640 1 -- 172.21.15.122:0/501793539 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4003520 con 0x7fb2d40039b0 2024-03-06T07:27:11.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed825640 1 -- 172.21.15.122:0/501793539 reap_dead start 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.398+0000 7fb2ed024640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2d40039b0 0x7fb2d4003db0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4003d80 tx=0x7fb2d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9460f149ff810122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4037070 con 0x7fb2d40039b0 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e4032600 con 0x7fb2d40039b0 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2dffff640 1 -- 172.21.15.122:0/501793539 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4003980 con 0x7fb2d40039b0 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 -- 172.21.15.122:0/501793539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2d40039b0 msgr2=0x7fb2d4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2d40039b0 0x7fb2d4003db0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4003d80 tx=0x7fb2d4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 -- 172.21.15.122:0/501793539 shutdown_connections 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 --2- 172.21.15.122:0/501793539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2d40039b0 0x7fb2d4003db0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 -- 172.21.15.122:0/501793539 >> 172.21.15.122:0/501793539 conn(0x7fb2e80d3c40 msgr2=0x7fb2e80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 -- 172.21.15.122:0/501793539 shutdown_connections 2024-03-06T07:27:11.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.399+0000 7fb2ef288640 1 -- 172.21.15.122:0/501793539 wait complete. 2024-03-06T07:27:11.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.400+0000 7fb2ef288640 1 Processor -- start 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.400+0000 7fb2ef288640 1 -- start start 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ef288640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d40039b0 0x7fb2e80a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ef288640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 0x7fb2e8151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ef288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2e8152ec0 0x7fb2e8153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ef288640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80b4dd0 con 0x7fb2e8150e30 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ef288640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80ce9e0 con 0x7fb2d40039b0 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ef288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80b4f50 con 0x7fb2e8152ec0 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 0x7fb2e8151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 0x7fb2e8151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44606/0 (socket says 172.21.15.122:44606) 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 -- 172.21.15.122:0/1956428630 learned_addr learned my addr 172.21.15.122:0/1956428630 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ed024640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d40039b0 0x7fb2e80a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ed825640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2e8152ec0 0x7fb2e8153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 -- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2e8152ec0 msgr2=0x7fb2e8153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2e8152ec0 0x7fb2e8153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 -- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d40039b0 msgr2=0x7fb2e80a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d40039b0 0x7fb2e80a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ec823640 1 -- 172.21.15.122:0/1956428630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8160750 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.401+0000 7fb2ed024640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d40039b0 0x7fb2e80a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.402+0000 7fb2ec823640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 0x7fb2e8151270 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800b820 tx=0x7fb2d800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.402+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8018070 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.402+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d800be10 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.402+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e81519f0 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.402+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8010c60 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.402+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e8151fb0 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.403+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e804f330 con 0x7fb2e8150e30 2024-03-06T07:27:11.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.403+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d801b070 con 0x7fb2e8150e30 2024-03-06T07:27:11.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.404+0000 7fb2ddffb640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b500 0x7fb2cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.404+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2d808a5e0 con 0x7fb2e8150e30 2024-03-06T07:27:11.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.404+0000 7fb2ed024640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b500 0x7fb2cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.404+0000 7fb2ed024640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b500 0x7fb2cc06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4034000 tx=0x7fb2e4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.408+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2d8059340 con 0x7fb2e8150e30 2024-03-06T07:27:11.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.580+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2e80733b0 con 0x7fb2e8150e30 2024-03-06T07:27:11.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.582+0000 7fb2ddffb640 1 -- 172.21.15.122:0/1956428630 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16640 (secure 0 0 0) 0x7fb2d8058c40 con 0x7fb2e8150e30 2024-03-06T07:27:11.584 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:11.584 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":107,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:10.603858+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.583+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b500 msgr2=0x7fb2cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.583+0000 7fb2ef288640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b500 0x7fb2cc06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4034000 tx=0x7fb2e4009030 comp rx=0 tx=0).stop 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.583+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 msgr2=0x7fb2e8151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.583+0000 7fb2ef288640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 0x7fb2e8151270 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800b820 tx=0x7fb2d800bce0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 shutdown_connections 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb2cc06b500 0x7fb2cc06d9c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb2e8152ec0 0x7fb2e8153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2e8150e30 0x7fb2e8151270 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 --2- 172.21.15.122:0/1956428630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2d40039b0 0x7fb2e80a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 >> 172.21.15.122:0/1956428630 conn(0x7fb2e80d3c40 msgr2=0x7fb2e806b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 shutdown_connections 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.584+0000 7fb2ef288640 1 -- 172.21.15.122:0/1956428630 wait complete. 2024-03-06T07:27:11.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:11.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.696+0000 7f26ec3f6640 1 Processor -- start 2024-03-06T07:27:11.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.696+0000 7f26ec3f6640 1 -- start start 2024-03-06T07:27:11.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ec3f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f26e415bf30 0x7f26e415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ec3f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e415f580 con 0x7f26e413dc10 2024-03-06T07:27:11.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ec3f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e415b450 con 0x7f26e40b3060 2024-03-06T07:27:11.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ec3f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e40cf3f0 con 0x7f26e415bf30 2024-03-06T07:27:11.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ea993640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f26e415bf30 0x7f26e415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ea192640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f26e413dc10 0x7f26e40b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43490/0 (socket says 172.21.15.122:43490) 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ea192640 1 -- 172.21.15.122:0/1578903887 learned_addr learned my addr 172.21.15.122:0/1578903887 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ea993640 1 -- 172.21.15.122:0/1578903887 >> v1:172.21.15.169:6789/0 conn(0x7f26e40b3060 legacy=0x7f26e415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ea993640 1 -- 172.21.15.122:0/1578903887 >> v1:172.21.15.122:6789/0 conn(0x7f26e413dc10 legacy=0x7f26e40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26ea993640 1 -- 172.21.15.122:0/1578903887 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e415b730 con 0x7f26e415bf30 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.697+0000 7f26e9190640 1 -- 172.21.15.122:0/1578903887 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200573094 0 0) 0x7f26e415f580 con 0x7f26e413dc10 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.698+0000 7f26ea993640 1 --2- 172.21.15.122:0/1578903887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e415bf30 0x7f26e415ef60 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097a0 tx=0x7f26dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d894c2c0baac7d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.698+0000 7f26e9190640 1 -- 172.21.15.122:0/1578903887 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0315a0 con 0x7f26e415bf30 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.698+0000 7f26e9190640 1 -- 172.21.15.122:0/1578903887 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc037070 con 0x7f26e415bf30 2024-03-06T07:27:11.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.698+0000 7f26e9190640 1 -- 172.21.15.122:0/1578903887 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc031960 con 0x7f26e415bf30 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1578903887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e415bf30 msgr2=0x7f26e415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1578903887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e415bf30 0x7f26e415ef60 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097a0 tx=0x7f26dc02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1578903887 shutdown_connections 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1578903887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e415bf30 0x7f26e415ef60 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1578903887 >> 172.21.15.122:0/1578903887 conn(0x7f26e40cf8a0 msgr2=0x7f26e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1578903887 shutdown_connections 2024-03-06T07:27:11.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.699+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1578903887 wait complete. 2024-03-06T07:27:11.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.700+0000 7f26ec3f6640 1 Processor -- start 2024-03-06T07:27:11.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.700+0000 7f26ec3f6640 1 -- start start 2024-03-06T07:27:11.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 0x7f26e4154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26e413dc10 0x7f26e413cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26e415bf30 0x7f26e413d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40718e0 con 0x7f26e40b3060 2024-03-06T07:27:11.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e415b210 con 0x7f26e413dc10 2024-03-06T07:27:11.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4071a60 con 0x7f26e415bf30 2024-03-06T07:27:11.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 0x7f26e4154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 0x7f26e4154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44628/0 (socket says 172.21.15.122:44628) 2024-03-06T07:27:11.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 -- 172.21.15.122:0/1197403562 learned_addr learned my addr 172.21.15.122:0/1197403562 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:11.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 -- 172.21.15.122:0/1197403562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26e415bf30 msgr2=0x7f26e413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26e415bf30 0x7f26e413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 -- 172.21.15.122:0/1197403562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26e413dc10 msgr2=0x7f26e413cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26e413dc10 0x7f26e413cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 -- 172.21.15.122:0/1197403562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e413ee20 con 0x7f26e40b3060 2024-03-06T07:27:11.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ea192640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 0x7f26e4154b20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c890 tx=0x7f26e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0018070 con 0x7f26e40b3060 2024-03-06T07:27:11.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000a640 con 0x7f26e40b3060 2024-03-06T07:27:11.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e413f100 con 0x7f26e40b3060 2024-03-06T07:27:11.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000aa30 con 0x7f26e40b3060 2024-03-06T07:27:11.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.701+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e4155060 con 0x7f26e40b3060 2024-03-06T07:27:11.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.702+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e404f330 con 0x7f26e40b3060 2024-03-06T07:27:11.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.703+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26e001b070 con 0x7f26e40b3060 2024-03-06T07:27:11.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.703+0000 7f26daffd640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f26c806b4d0 0x7f26c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.703+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26e0089900 con 0x7f26e40b3060 2024-03-06T07:27:11.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.703+0000 7f26e9991640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f26c806b4d0 0x7f26c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.704+0000 7f26e9991640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f26c806b4d0 0x7f26c806d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f26e40b0fd0 tx=0x7f26d4007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.707+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26e0058780 con 0x7f26e40b3060 2024-03-06T07:27:11.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.879+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e415a970 con 0x7f26e40b3060 2024-03-06T07:27:11.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.880+0000 7f26daffd640 1 -- 172.21.15.122:0/1197403562 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16640 (secure 0 0 0) 0x7f26e0058080 con 0x7f26e40b3060 2024-03-06T07:27:11.882 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:11.882 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":107,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:10.603858+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f26c806b4d0 msgr2=0x7f26c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f26c806b4d0 0x7f26c806d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f26e40b0fd0 tx=0x7f26d4007990 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 msgr2=0x7f26e4154b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 0x7f26e4154b20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c890 tx=0x7f26e000cd50 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 shutdown_connections 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f26c806b4d0 0x7f26c806d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f26e415bf30 0x7f26e413d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26e413dc10 0x7f26e413cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 --2- 172.21.15.122:0/1197403562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26e40b3060 0x7f26e4154b20 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 >> 172.21.15.122:0/1197403562 conn(0x7f26e40cf8a0 msgr2=0x7f26e406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 shutdown_connections 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.882+0000 7f26ec3f6640 1 -- 172.21.15.122:0/1197403562 wait complete. 2024-03-06T07:27:11.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:11.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.996+0000 7fc464da0640 1 Processor -- start 2024-03-06T07:27:11.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.996+0000 7fc464da0640 1 -- start start 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc464da0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc46015e0d0 0x7fc4600b4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc464da0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46015e690 con 0x7fc4600b4e90 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc464da0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46015e970 con 0x7fc460158030 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc464da0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46015ec50 con 0x7fc46015e0d0 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc45ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc4600b4e90 0x7fc46006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43504/0 (socket says 172.21.15.122:43504) 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc45ffff640 1 -- 172.21.15.122:0/3949108014 learned_addr learned my addr 172.21.15.122:0/3949108014 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.997+0000 7fc456ffd640 1 --2- 172.21.15.122:0/3949108014 >> v2:172.21.15.169:3300/0 conn(0x7fc46015e0d0 0x7fc4600b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.998+0000 7fc45effd640 1 -- 172.21.15.122:0/3949108014 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7180373 0 0) 0x7fc46015e690 con 0x7fc4600b4e90 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.998+0000 7fc45effd640 1 -- 172.21.15.122:0/3949108014 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c003520 con 0x7fc4600b4e90 2024-03-06T07:27:12.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.998+0000 7fc45effd640 1 -- 172.21.15.122:0/3949108014 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711920348 0 0) 0x7fc44c003520 con 0x7fc4600b4e90 2024-03-06T07:27:12.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.998+0000 7fc456ffd640 1 -- 172.21.15.122:0/3949108014 >> v1:172.21.15.169:6789/0 conn(0x7fc460158030 legacy=0x7fc460072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.998+0000 7fc456ffd640 1 -- 172.21.15.122:0/3949108014 >> v1:172.21.15.122:6789/0 conn(0x7fc4600b4e90 legacy=0x7fc46006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.998+0000 7fc456ffd640 1 -- 172.21.15.122:0/3949108014 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46015ef30 con 0x7fc46015e0d0 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc456ffd640 1 --2- 172.21.15.122:0/3949108014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46015e0d0 0x7fc4600b4950 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc4480097a0 tx=0x7fc44802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bef6463539207d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc45effd640 1 -- 172.21.15.122:0/3949108014 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448038070 con 0x7fc46015e0d0 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc45effd640 1 -- 172.21.15.122:0/3949108014 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448031500 con 0x7fc46015e0d0 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc45effd640 1 -- 172.21.15.122:0/3949108014 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4580019f0 con 0x7fc46015e0d0 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 -- 172.21.15.122:0/3949108014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46015e0d0 msgr2=0x7fc4600b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 --2- 172.21.15.122:0/3949108014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46015e0d0 0x7fc4600b4950 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc4480097a0 tx=0x7fc44802b820 comp rx=0 tx=0).stop 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 -- 172.21.15.122:0/3949108014 shutdown_connections 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 --2- 172.21.15.122:0/3949108014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc46015e0d0 0x7fc4600b4950 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 -- 172.21.15.122:0/3949108014 >> 172.21.15.122:0/3949108014 conn(0x7fc4600b2510 msgr2=0x7fc4600b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 -- 172.21.15.122:0/3949108014 shutdown_connections 2024-03-06T07:27:12.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:11.999+0000 7fc464da0640 1 -- 172.21.15.122:0/3949108014 wait complete. 2024-03-06T07:27:12.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 Processor -- start 2024-03-06T07:27:12.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 -- start start 2024-03-06T07:27:12.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 0x7fc46014c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc460158030 0x7fc46015f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc46015e0d0 0x7fc460156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc460158590 con 0x7fc4600b4e90 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc460158410 con 0x7fc460158030 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.001+0000 7fc464da0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46006b6c0 con 0x7fc46015e0d0 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 0x7fc46014c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 0x7fc46014c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44630/0 (socket says 172.21.15.122:44630) 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 -- 172.21.15.122:0/4273061345 learned_addr learned my addr 172.21.15.122:0/4273061345 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc456ffd640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc460158030 0x7fc46015f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45ffff640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc46015e0d0 0x7fc460156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 -- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc46015e0d0 msgr2=0x7fc460156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc46015e0d0 0x7fc460156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 -- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc460158030 msgr2=0x7fc46015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc460158030 0x7fc46015f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 -- 172.21.15.122:0/4273061345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46015fa70 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45f7fe640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 0x7fc46014c4c0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fc45000c450 tx=0x7fc45000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500052d0 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450005470 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46015fd50 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450012870 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.002+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46015f6b0 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.003+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc460068f10 con 0x7fc4600b4e90 2024-03-06T07:27:12.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.004+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc450012a10 con 0x7fc4600b4e90 2024-03-06T07:27:12.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.004+0000 7fc45cff9640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc44006b500 0x7fc44006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.004+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc450089c60 con 0x7fc4600b4e90 2024-03-06T07:27:12.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.004+0000 7fc456ffd640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc44006b500 0x7fc44006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.004+0000 7fc456ffd640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc44006b500 0x7fc44006d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc448035000 tx=0x7fc448008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.008+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc450058520 con 0x7fc4600b4e90 2024-03-06T07:27:12.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.218+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc460070210 con 0x7fc4600b4e90 2024-03-06T07:27:12.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.275+0000 7fc45cff9640 1 -- 172.21.15.122:0/4273061345 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc45005c5c0 con 0x7fc4600b4e90 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc44006b500 msgr2=0x7fc44006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc44006b500 0x7fc44006d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc448035000 tx=0x7fc448008030 comp rx=0 tx=0).stop 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 msgr2=0x7fc46014c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 0x7fc46014c4c0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fc45000c450 tx=0x7fc45000c910 comp rx=0 tx=0).stop 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 shutdown_connections 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc44006b500 0x7fc44006d9c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc46015e0d0 0x7fc460156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc460158030 0x7fc46015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 --2- 172.21.15.122:0/4273061345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4600b4e90 0x7fc46014c4c0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 >> 172.21.15.122:0/4273061345 conn(0x7fc4600b2510 msgr2=0x7fc460072440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 shutdown_connections 2024-03-06T07:27:12.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.277+0000 7fc464da0640 1 -- 172.21.15.122:0/4273061345 wait complete. 2024-03-06T07:27:12.291 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:12.291 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.393+0000 7fcd646f4640 1 Processor -- start 2024-03-06T07:27:12.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.393+0000 7fcd646f4640 1 -- start start 2024-03-06T07:27:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.394+0000 7fcd646f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 0x7fcd5c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.394+0000 7fcd646f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5c0cd4b0 con 0x7fcd5c15c050 2024-03-06T07:27:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.394+0000 7fcd646f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5c0cd790 con 0x7fcd5c06fd10 2024-03-06T07:27:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.394+0000 7fcd646f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c0cda70 con 0x7fcd5c0d0af0 2024-03-06T07:27:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.394+0000 7fcd62490640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcd5c15c050 0x7fcd5c06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43508/0 (socket says 172.21.15.122:43508) 2024-03-06T07:27:12.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.394+0000 7fcd62490640 1 -- 172.21.15.122:0/2239759551 learned_addr learned my addr 172.21.15.122:0/2239759551 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd62c91640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 0x7fcd5c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441706054 0 0) 0x7fcd5c0cd4b0 con 0x7fcd5c15c050 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd48003520 con 0x7fcd5c15c050 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4188167632 0 0) 0x7fcd48003520 con 0x7fcd5c15c050 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 msgr2=0x7fcd5c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 0x7fcd5c0cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 >> v1:172.21.15.169:6789/0 conn(0x7fcd5c06fd10 legacy=0x7fcd5c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c0cdd50 con 0x7fcd5c15c050 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fcd58001670 con 0x7fcd5c15c050 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 >> v1:172.21.15.122:6789/0 conn(0x7fcd5c15c050 legacy=0x7fcd5c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 0x7fcd480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd48004d40 0x7fcd480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c0cda70 con 0x7fcd48003fb0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4800bbb0 con 0x7fcd48004d40 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4800bdb0 con 0x7fcd5c0d0af0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd62490640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 0x7fcd480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd62c91640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd48004d40 0x7fcd480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd61c8f640 1 -- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 msgr2=0x7fcd480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c0d0af0 0x7fcd480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd61c8f640 1 -- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd48004d40 msgr2=0x7fcd480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd48004d40 0x7fcd480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.395+0000 7fcd61c8f640 1 -- 172.21.15.122:0/2239759551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48003520 con 0x7fcd48003fb0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.396+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd48004800 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c001750 tx=0x7fcd48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c56e9066a5e7092 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.396+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c001370 con 0x7fcd48003fb0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.396+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4c0052d0 con 0x7fcd48003fb0 2024-03-06T07:27:12.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.396+0000 7fcd6148e640 1 -- 172.21.15.122:0/2239759551 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c003830 con 0x7fcd48003fb0 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd646f4640 1 -- 172.21.15.122:0/2239759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 msgr2=0x7fcd48004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd646f4640 1 --2- 172.21.15.122:0/2239759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd48004800 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c001750 tx=0x7fcd48005830 comp rx=0 tx=0).stop 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd62c91640 1 -- 172.21.15.122:0/2239759551 reap_dead start 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd646f4640 1 -- 172.21.15.122:0/2239759551 shutdown_connections 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd646f4640 1 -- 172.21.15.122:0/2239759551 >> 172.21.15.122:0/2239759551 conn(0x7fcd5c0b3800 msgr2=0x7fcd5c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd646f4640 1 -- 172.21.15.122:0/2239759551 shutdown_connections 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.397+0000 7fcd646f4640 1 -- 172.21.15.122:0/2239759551 wait complete. 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.399+0000 7fcd646f4640 1 Processor -- start 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.399+0000 7fcd646f4640 1 -- start start 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd646f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd5c14c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd646f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c1f2c40 0x7fcd5c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd646f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd5c155050 0x7fcd5c155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd646f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c0cd100 con 0x7fcd48003fb0 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd646f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c164ab0 con 0x7fcd5c155050 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd646f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c154bc0 con 0x7fcd5c1f2c40 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd5c14c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd5c14c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44650/0 (socket says 172.21.15.122:44650) 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 -- 172.21.15.122:0/3948531203 learned_addr learned my addr 172.21.15.122:0/3948531203 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c1f2c40 0x7fcd5c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62c91640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd5c155050 0x7fcd5c155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 -- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c1f2c40 msgr2=0x7fcd5c0b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c1f2c40 0x7fcd5c0b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 -- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd5c155050 msgr2=0x7fcd5c155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd5c155050 0x7fcd5c155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 -- 172.21.15.122:0/3948531203 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c153fb0 con 0x7fcd48003fb0 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.400+0000 7fcd62490640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd5c14c470 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fcd580072c0 tx=0x7fcd58007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.401+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800f960 con 0x7fcd48003fb0 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.401+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd58018070 con 0x7fcd48003fb0 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.401+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800fd20 con 0x7fcd48003fb0 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.401+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5c154290 con 0x7fcd48003fb0 2024-03-06T07:27:12.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.401+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c154850 con 0x7fcd48003fb0 2024-03-06T07:27:12.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.402+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c09fe60 con 0x7fcd48003fb0 2024-03-06T07:27:12.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.402+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd5801d460 con 0x7fcd48003fb0 2024-03-06T07:27:12.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.402+0000 7fcd52ffd640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcd4006b4d0 0x7fcd4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.403+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd5808a250 con 0x7fcd48003fb0 2024-03-06T07:27:12.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.403+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcd4006b4d0 0x7fcd4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.403+0000 7fcd61c8f640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcd4006b4d0 0x7fcd4006d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c033000 tx=0x7fcd4c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.407+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd58059160 con 0x7fcd48003fb0 2024-03-06T07:27:12.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:12.633+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcd5c15a7d0 con 0x7fcd48003fb0 2024-03-06T07:27:13.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.279+0000 7fcd52ffd640 1 -- 172.21.15.122:0/3948531203 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd58058a60 con 0x7fcd48003fb0 2024-03-06T07:27:13.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.281+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcd4006b4d0 msgr2=0x7fcd4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.281+0000 7fcd646f4640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcd4006b4d0 0x7fcd4006d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c033000 tx=0x7fcd4c009030 comp rx=0 tx=0).stop 2024-03-06T07:27:13.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.281+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 msgr2=0x7fcd5c14c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.281+0000 7fcd646f4640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd5c14c470 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fcd580072c0 tx=0x7fcd58007780 comp rx=0 tx=0).stop 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 shutdown_connections 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcd4006b4d0 0x7fcd4006d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd5c155050 0x7fcd5c155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcd5c1f2c40 0x7fcd5c0b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 --2- 172.21.15.122:0/3948531203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48003fb0 0x7fcd5c14c470 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 >> 172.21.15.122:0/3948531203 conn(0x7fcd5c0b3800 msgr2=0x7fcd5c0aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 shutdown_connections 2024-03-06T07:27:13.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.282+0000 7fcd646f4640 1 -- 172.21.15.122:0/3948531203 wait complete. 2024-03-06T07:27:13.298 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:13.298 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:13.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 Processor -- start 2024-03-06T07:27:13.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 -- start start 2024-03-06T07:27:13.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ffa680d22b0 0x7ffa680d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa680b41f0 0x7ffa680b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa680b3430 con 0x7ffa6815dfe0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680d2c50 con 0x7ffa680b41f0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6ecdf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680b31e0 con 0x7ffa680d22b0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6d27c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa680b41f0 0x7ffa680b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6d27c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa680b41f0 0x7ffa680b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44662/0 (socket says 172.21.15.122:44662) 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.398+0000 7ffa6d27c640 1 -- 172.21.15.122:0/3326483923 learned_addr learned my addr 172.21.15.122:0/3326483923 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa67fff640 1 --2- 172.21.15.122:0/3326483923 >> v2:172.21.15.169:3300/0 conn(0x7ffa680d22b0 0x7ffa680d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa6d27c640 1 -- 172.21.15.122:0/3326483923 >> v2:172.21.15.169:3300/0 conn(0x7ffa680d22b0 msgr2=0x7ffa680d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa6d27c640 1 --2- 172.21.15.122:0/3326483923 >> v2:172.21.15.169:3300/0 conn(0x7ffa680d22b0 0x7ffa680d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa6d27c640 1 -- 172.21.15.122:0/3326483923 >> v1:172.21.15.169:6789/0 conn(0x7ffa6815dfe0 legacy=0x7ffa680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa6d27c640 1 -- 172.21.15.122:0/3326483923 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680b3710 con 0x7ffa680b41f0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa67fff640 1 --2- 172.21.15.122:0/3326483923 >> v2:172.21.15.169:3300/0 conn(0x7ffa680d22b0 0x7ffa680d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.399+0000 7ffa6d27c640 1 --2- 172.21.15.122:0/3326483923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa680b41f0 0x7ffa680b4680 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009e50 tx=0x7ffa6002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b462765488dabc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa677fe640 1 -- 172.21.15.122:0/3326483923 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6002ca80 con 0x7ffa680b41f0 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa677fe640 1 -- 172.21.15.122:0/3326483923 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6002cc20 con 0x7ffa680b41f0 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa677fe640 1 -- 172.21.15.122:0/3326483923 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600343f0 con 0x7ffa680b41f0 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/3326483923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa680b41f0 msgr2=0x7ffa680b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/3326483923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa680b41f0 0x7ffa680b4680 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009e50 tx=0x7ffa6002bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/3326483923 shutdown_connections 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/3326483923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa680b41f0 0x7ffa680b4680 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/3326483923 >> v2:172.21.15.169:3300/0 conn(0x7ffa680d22b0 0x7ffa680d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/3326483923 >> 172.21.15.122:0/3326483923 conn(0x7ffa680d3cd0 msgr2=0x7ffa680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/3326483923 shutdown_connections 2024-03-06T07:27:13.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.400+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/3326483923 wait complete. 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.401+0000 7ffa6ecdf640 1 Processor -- start 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.401+0000 7ffa6ecdf640 1 -- start start 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ecdf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa680b41f0 0x7ffa68161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ecdf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa680d22b0 0x7ffa680cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ecdf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 0x7ffa680b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ecdf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680cfa00 con 0x7ffa6815dfe0 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ecdf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680b3d60 con 0x7ffa680b41f0 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ecdf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680d1c70 con 0x7ffa680d22b0 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6d27c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 0x7ffa680b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6d27c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 0x7ffa680b9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44670/0 (socket says 172.21.15.122:44670) 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6d27c640 1 -- 172.21.15.122:0/221952226 learned_addr learned my addr 172.21.15.122:0/221952226 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa6ca7b640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa680b41f0 0x7ffa68161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.402+0000 7ffa67fff640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa680d22b0 0x7ffa680cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6d27c640 1 -- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa680d22b0 msgr2=0x7ffa680cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6d27c640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa680d22b0 0x7ffa680cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6d27c640 1 -- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa680b41f0 msgr2=0x7ffa68161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6d27c640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa680b41f0 0x7ffa68161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6d27c640 1 -- 172.21.15.122:0/221952226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680cc250 con 0x7ffa6815dfe0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6d27c640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 0x7ffa680b9320 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009e50 tx=0x7ffa60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60043020 con 0x7ffa6815dfe0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6003e450 con 0x7ffa6815dfe0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa680cc530 con 0x7ffa6815dfe0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60009510 con 0x7ffa6815dfe0 2024-03-06T07:27:13.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.403+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa680ccaf0 con 0x7ffa6815dfe0 2024-03-06T07:27:13.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.404+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6804f330 con 0x7ffa6815dfe0 2024-03-06T07:27:13.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.404+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa60046020 con 0x7ffa6815dfe0 2024-03-06T07:27:13.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.405+0000 7ffa657fa640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.405+0000 7ffa6ca7b640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.405+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa600aca90 con 0x7ffa6815dfe0 2024-03-06T07:27:13.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.405+0000 7ffa6ca7b640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c006fd0 tx=0x7ffa5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.409+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa6004d030 con 0x7ffa6815dfe0 2024-03-06T07:27:13.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.618+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa6815d500 con 0x7ffa6815dfe0 2024-03-06T07:27:13.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.619+0000 7ffa657fa640 1 -- 172.21.15.122:0/221952226 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4988 (secure 0 0 0) 0x7ffa6007b9a0 con 0x7ffa6815dfe0 2024-03-06T07:27:13.621 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:13.621 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3571233755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3571233755},{"type":"v1","addr":"172.21.15.169:6834","nonce":3571233755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1631232650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1631232650},{"type":"v1","addr":"172.21.15.169:6835","nonce":1631232650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/2235394450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2235394450},{"type":"v1","addr":"172.21.15.122:6835","nonce":2235394450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:12.635136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7962},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7962":{"gid":7962,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/386534812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":386534812},{"type":"v1","addr":"172.21.15.122:6837","nonce":386534812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7962,"qdb_cluster":[7962]},"id":8}]} 2024-03-06T07:27:13.621 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 61 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa4c06b4d0 msgr2=0x7ffa4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c006fd0 tx=0x7ffa5c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 msgr2=0x7ffa680b9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 0x7ffa680b9320 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009e50 tx=0x7ffa60004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 shutdown_connections 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffa4c06b4d0 0x7ffa4c06d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa6815dfe0 0x7ffa680b9320 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffa680d22b0 0x7ffa680cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 --2- 172.21.15.122:0/221952226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa680b41f0 0x7ffa68161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 >> 172.21.15.122:0/221952226 conn(0x7ffa680d3cd0 msgr2=0x7ffa680d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 shutdown_connections 2024-03-06T07:27:13.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.621+0000 7ffa6ecdf640 1 -- 172.21.15.122:0/221952226 wait complete. 2024-03-06T07:27:13.634 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:13.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.733+0000 7f886b55e640 1 Processor -- start 2024-03-06T07:27:13.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.733+0000 7f886b55e640 1 -- start start 2024-03-06T07:27:13.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f886b55e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886415dfe0 0x7f88640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f886b55e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f88640cf830 0x7f88640d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f886b55e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f88640d27f0 0x7f88640d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f886b55e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88640d2c50 con 0x7f88640d27f0 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f886b55e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88640b31e0 con 0x7f88640cf830 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f886b55e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88640b3d60 con 0x7f886415dfe0 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f88640d27f0 0x7f88640d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f88640d27f0 0x7f88640d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44684/0 (socket says 172.21.15.122:44684) 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 -- 172.21.15.122:0/3259039585 learned_addr learned my addr 172.21.15.122:0/3259039585 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8868af9640 1 --2- 172.21.15.122:0/3259039585 >> v2:172.21.15.169:3300/0 conn(0x7f88640cf830 0x7f88640d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f88692fa640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886415dfe0 0x7f88640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 -- 172.21.15.122:0/3259039585 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886415dfe0 msgr2=0x7f88640cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886415dfe0 0x7f88640cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 -- 172.21.15.122:0/3259039585 >> v2:172.21.15.169:3300/0 conn(0x7f88640cf830 msgr2=0x7f88640d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 --2- 172.21.15.122:0/3259039585 >> v2:172.21.15.169:3300/0 conn(0x7f88640cf830 0x7f88640d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8869afb640 1 -- 172.21.15.122:0/3259039585 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88640b3430 con 0x7f88640d27f0 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f8868af9640 1 --2- 172.21.15.122:0/3259039585 >> v2:172.21.15.169:3300/0 conn(0x7f88640cf830 0x7f88640d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.734+0000 7f88692fa640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886415dfe0 0x7f88640cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f8869afb640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88640d27f0 0x7f88640d0300 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f886408cba0 tx=0x7f885c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a0e70f00c22f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f885bfff640 1 -- 172.21.15.122:0/3259039585 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c02ca70 con 0x7f88640d27f0 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f885bfff640 1 -- 172.21.15.122:0/3259039585 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c02cc10 con 0x7f88640d27f0 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f885bfff640 1 -- 172.21.15.122:0/3259039585 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0343f0 con 0x7f88640d27f0 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 -- 172.21.15.122:0/3259039585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88640d27f0 msgr2=0x7f88640d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88640d27f0 0x7f88640d0300 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f886408cba0 tx=0x7f885c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 -- 172.21.15.122:0/3259039585 shutdown_connections 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88640d27f0 0x7f88640d0300 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 --2- 172.21.15.122:0/3259039585 >> v2:172.21.15.169:3300/0 conn(0x7f88640cf830 0x7f88640d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 --2- 172.21.15.122:0/3259039585 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f886415dfe0 0x7f88640cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.735+0000 7f886b55e640 1 -- 172.21.15.122:0/3259039585 >> 172.21.15.122:0/3259039585 conn(0x7f88640d3cd0 msgr2=0x7f88640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.736+0000 7f886b55e640 1 -- 172.21.15.122:0/3259039585 shutdown_connections 2024-03-06T07:27:13.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.736+0000 7f886b55e640 1 -- 172.21.15.122:0/3259039585 wait complete. 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 Processor -- start 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 -- start start 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88640cf830 0x7f886415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88640d27f0 0x7f88640b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 0x7f88640ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88640d1c70 con 0x7f886415dfe0 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88640b39f0 con 0x7f88640cf830 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.737+0000 7f886b55e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88640d1df0 con 0x7f88640d27f0 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 0x7f88640ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 0x7f88640ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44698/0 (socket says 172.21.15.122:44698) 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 -- 172.21.15.122:0/197177378 learned_addr learned my addr 172.21.15.122:0/197177378 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f88692fa640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88640cf830 0x7f886415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8868af9640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88640d27f0 0x7f88640b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 -- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88640d27f0 msgr2=0x7f88640b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88640d27f0 0x7f88640b7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 -- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88640cf830 msgr2=0x7f886415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88640cf830 0x7f886415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 -- 172.21.15.122:0/197177378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886415e490 con 0x7f886415dfe0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8869afb640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 0x7f88640ccd10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f885c036040 tx=0x7f885c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c043020 con 0x7f886415dfe0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c03e450 con 0x7f886415dfe0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886415e770 con 0x7f886415dfe0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c009510 con 0x7f886415dfe0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.738+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886415ed30 con 0x7f886415dfe0 2024-03-06T07:27:13.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.739+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886404f330 con 0x7f886415dfe0 2024-03-06T07:27:13.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.740+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f885c046020 con 0x7f886415dfe0 2024-03-06T07:27:13.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.741+0000 7f8859ffb640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f884806b4d0 0x7f884806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.741+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f885c0acaf0 con 0x7f886415dfe0 2024-03-06T07:27:13.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.741+0000 7f88692fa640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f884806b4d0 0x7f884806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.741+0000 7f88692fa640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f884806b4d0 0x7f884806d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f886000a950 tx=0x7f8860008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.745+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f885c04d030 con 0x7f886415dfe0 2024-03-06T07:27:13.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.923+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f886415aa30 con 0x7f886415dfe0 2024-03-06T07:27:13.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.924+0000 7f8859ffb640 1 -- 172.21.15.122:0/197177378 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f885c07b970 con 0x7f886415dfe0 2024-03-06T07:27:13.926 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:13.926 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":108,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:12.269669+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:13.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f884806b4d0 msgr2=0x7f884806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f884806b4d0 0x7f884806d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f886000a950 tx=0x7f8860008040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 msgr2=0x7f88640ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 0x7f88640ccd10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f885c036040 tx=0x7f885c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 shutdown_connections 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f884806b4d0 0x7f884806d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f886415dfe0 0x7f88640ccd10 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88640d27f0 0x7f88640b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 --2- 172.21.15.122:0/197177378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88640cf830 0x7f886415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 >> 172.21.15.122:0/197177378 conn(0x7f88640d3cd0 msgr2=0x7f886406cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 shutdown_connections 2024-03-06T07:27:13.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:13.926+0000 7f886b55e640 1 -- 172.21.15.122:0/197177378 wait complete. 2024-03-06T07:27:13.940 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:13.940 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.039+0000 7f3a84288640 1 Processor -- start 2024-03-06T07:27:14.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.039+0000 7f3a84288640 1 -- start start 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a84288640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3a7c0d4760 0x7f3a7c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a84288640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a7c0d0a80 0x7f3a7c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a84288640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c0d25d0 con 0x7f3a7c0b4170 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a84288640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c0d4bf0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a84288640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c0ce9e0 con 0x7f3a7c0d4760 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a82825640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a7c0d0a80 0x7f3a7c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a82825640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3a7c0d0a80 0x7f3a7c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44700/0 (socket says 172.21.15.122:44700) 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.040+0000 7f3a82825640 1 -- 172.21.15.122:0/2397494043 learned_addr learned my addr 172.21.15.122:0/2397494043 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a81823640 1 --2- 172.21.15.122:0/2397494043 >> v2:172.21.15.169:3300/0 conn(0x7f3a7c0d4760 0x7f3a7c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a82825640 1 -- 172.21.15.122:0/2397494043 >> v2:172.21.15.169:3300/0 conn(0x7f3a7c0d4760 msgr2=0x7f3a7c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a82825640 1 --2- 172.21.15.122:0/2397494043 >> v2:172.21.15.169:3300/0 conn(0x7f3a7c0d4760 0x7f3a7c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a82825640 1 -- 172.21.15.122:0/2397494043 >> v1:172.21.15.169:6789/0 conn(0x7f3a7c0b4170 legacy=0x7f3a7c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a82825640 1 -- 172.21.15.122:0/2397494043 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c0d2870 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a81823640 1 --2- 172.21.15.122:0/2397494043 >> v2:172.21.15.169:3300/0 conn(0x7f3a7c0d4760 0x7f3a7c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.041+0000 7f3a82825640 1 --2- 172.21.15.122:0/2397494043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0cee70 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c08df30 tx=0x7f3a7402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09827c7685a5f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a81022640 1 -- 172.21.15.122:0/2397494043 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7402ca50 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a81022640 1 -- 172.21.15.122:0/2397494043 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7402cbf0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a81022640 1 -- 172.21.15.122:0/2397494043 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a740343f0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 -- 172.21.15.122:0/2397494043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 msgr2=0x7f3a7c0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 --2- 172.21.15.122:0/2397494043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0cee70 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c08df30 tx=0x7f3a7402beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 -- 172.21.15.122:0/2397494043 shutdown_connections 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 --2- 172.21.15.122:0/2397494043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0cee70 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 --2- 172.21.15.122:0/2397494043 >> v2:172.21.15.169:3300/0 conn(0x7f3a7c0d4760 0x7f3a7c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 -- 172.21.15.122:0/2397494043 >> 172.21.15.122:0/2397494043 conn(0x7f3a7c0d3c40 msgr2=0x7f3a7c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 -- 172.21.15.122:0/2397494043 shutdown_connections 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.042+0000 7f3a84288640 1 -- 172.21.15.122:0/2397494043 wait complete. 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.043+0000 7f3a84288640 1 Processor -- start 2024-03-06T07:27:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.043+0000 7f3a84288640 1 -- start start 2024-03-06T07:27:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a84288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a7c0b4170 0x7f3a7c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a84288640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a84288640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a7c09f150 0x7f3a7c09f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a84288640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0b4dd0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a84288640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0b4550 con 0x7f3a7c09f150 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a84288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0b4f50 con 0x7f3a7c0b4170 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44716/0 (socket says 172.21.15.122:44716) 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 -- 172.21.15.122:0/1587214926 learned_addr learned my addr 172.21.15.122:0/1587214926 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a82024640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a7c0b4170 0x7f3a7c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a82825640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a7c09f150 0x7f3a7c09f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 -- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a7c0b4170 msgr2=0x7f3a7c1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a7c0b4170 0x7f3a7c1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 -- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a7c09f150 msgr2=0x7f3a7c09f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a7c09f150 0x7f3a7c09f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a81823640 1 -- 172.21.15.122:0/1587214926 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c0b2340 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a82024640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a7c0b4170 0x7f3a7c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.044+0000 7f3a82825640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a7c09f150 0x7f3a7c09f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.045+0000 7f3a81823640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0b0d50 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d830 tx=0x7f3a6c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.045+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c004030 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.045+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6c0041d0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.045+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c0b2620 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.045+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0096a0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.045+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c0b2be0 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.046+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c08dc90 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.046+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a6c009a50 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.047+0000 7f3a72ffd640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a6006b500 0x7f3a6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.047+0000 7f3a82024640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a6006b500 0x7f3a6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.047+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a6c08a610 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.047+0000 7f3a82024640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a6006b500 0x7f3a6006d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3a78003eb0 tx=0x7f3a78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.051+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a7c08dc90 con 0x7f3a7c0d0a80 2024-03-06T07:27:14.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.226+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a7c0b20f0 con 0x7f3a6006b500 2024-03-06T07:27:14.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.232+0000 7f3a72ffd640 1 -- 172.21.15.122:0/1587214926 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247087 (secure 0 0 0) 0x7f3a78054540 con 0x7f3a6006b500 2024-03-06T07:27:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:14.238 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:27:14.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a6006b500 msgr2=0x7f3a6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a6006b500 0x7f3a6006d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3a78003eb0 tx=0x7f3a78009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 msgr2=0x7f3a7c0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0b0d50 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d830 tx=0x7f3a6c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 shutdown_connections 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a6006b500 0x7f3a6006d9c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a7c09f150 0x7f3a7c09f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a7c0d0a80 0x7f3a7c0b0d50 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 --2- 172.21.15.122:0/1587214926 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a7c0b4170 0x7f3a7c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 >> 172.21.15.122:0/1587214926 conn(0x7f3a7c0d3c40 msgr2=0x7f3a7c0b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 shutdown_connections 2024-03-06T07:27:14.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.235+0000 7f3a84288640 1 -- 172.21.15.122:0/1587214926 wait complete. 2024-03-06T07:27:14.248 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:13.396067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323280,"kb_used_data":11380,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651440,"statfs":{"total":773094113280,"available":772763074560,"internally_reserved":0,"allocated":11653120,"data_stored":5663339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.140367"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267598+0000","last_change":"2024-03-06T07:27:10.267598+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267598+0000","last_undegraded":"2024-03-06T07:27:10.267598+0000","last_fullsized":"2024-03-06T07:27:10.267598+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261734+0000","last_change":"2024-03-06T07:27:10.261734+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261734+0000","last_undegraded":"2024-03-06T07:27:10.261734+0000","last_fullsized":"2024-03-06T07:27:10.261734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273806+0000","last_change":"2024-03-06T07:27:10.273806+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273806+0000","last_undegraded":"2024-03-06T07:27:10.273806+0000","last_fullsized":"2024-03-06T07:27:10.273806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265833+0000","last_change":"2024-03-06T07:27:10.265833+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265833+0000","last_undegraded":"2024-03-06T07:27:10.265833+0000","last_fullsized":"2024-03-06T07:27:10.265833+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266357+0000","last_change":"2024-03-06T07:27:10.266357+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266357+0000","last_undegraded":"2024-03-06T07:27:10.266357+0000","last_fullsized":"2024-03-06T07:27:10.266357+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265033+0000","last_change":"2024-03-06T07:27:10.265033+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265033+0000","last_undegraded":"2024-03-06T07:27:10.265033+0000","last_fullsized":"2024-03-06T07:27:10.265033+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265851+0000","last_change":"2024-03-06T07:27:10.265851+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265851+0000","last_undegraded":"2024-03-06T07:27:10.265851+0000","last_fullsized":"2024-03-06T07:27:10.265851+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269538+0000","last_change":"2024-03-06T07:27:10.269538+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269538+0000","last_undegraded":"2024-03-06T07:27:10.269538+0000","last_fullsized":"2024-03-06T07:27:10.269538+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267258+0000","last_change":"2024-03-06T07:27:10.267258+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267258+0000","last_undegraded":"2024-03-06T07:27:10.267258+0000","last_fullsized":"2024-03-06T07:27:10.267258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273029+0000","last_change":"2024-03-06T07:27:10.273029+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273029+0000","last_undegraded":"2024-03-06T07:27:10.273029+0000","last_fullsized":"2024-03-06T07:27:10.273029+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261449+0000","last_change":"2024-03-06T07:27:10.261449+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261449+0000","last_undegraded":"2024-03-06T07:27:10.261449+0000","last_fullsized":"2024-03-06T07:27:10.261449+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265454+0000","last_change":"2024-03-06T07:27:10.265454+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265454+0000","last_undegraded":"2024-03-06T07:27:10.265454+0000","last_fullsized":"2024-03-06T07:27:10.265454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268184+0000","last_change":"2024-03-06T07:27:10.268184+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268184+0000","last_undegraded":"2024-03-06T07:27:10.268184+0000","last_fullsized":"2024-03-06T07:27:10.268184+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268837+0000","last_change":"2024-03-06T07:27:10.268837+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268837+0000","last_undegraded":"2024-03-06T07:27:10.268837+0000","last_fullsized":"2024-03-06T07:27:10.268837+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264950+0000","last_change":"2024-03-06T07:27:10.264950+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264950+0000","last_undegraded":"2024-03-06T07:27:10.264950+0000","last_fullsized":"2024-03-06T07:27:10.264950+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268981+0000","last_change":"2024-03-06T07:27:10.268981+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268981+0000","last_undegraded":"2024-03-06T07:27:10.268981+0000","last_fullsized":"2024-03-06T07:27:10.268981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.262813+0000","last_change":"2024-03-06T07:27:10.262813+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.262813+0000","last_undegraded":"2024-03-06T07:27:10.262813+0000","last_fullsized":"2024-03-06T07:27:10.262813+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264573+0000","last_change":"2024-03-06T07:27:10.264573+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264573+0000","last_undegraded":"2024-03-06T07:27:10.264573+0000","last_fullsized":"2024-03-06T07:27:10.264573+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273199+0000","last_change":"2024-03-06T07:27:10.273199+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273199+0000","last_undegraded":"2024-03-06T07:27:10.273199+0000","last_fullsized":"2024-03-06T07:27:10.273199+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264185+0000","last_change":"2024-03-06T07:27:10.264185+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264185+0000","last_undegraded":"2024-03-06T07:27:10.264185+0000","last_fullsized":"2024-03-06T07:27:10.264185+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261404+0000","last_change":"2024-03-06T07:27:10.261404+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261404+0000","last_undegraded":"2024-03-06T07:27:10.261404+0000","last_fullsized":"2024-03-06T07:27:10.261404+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268216+0000","last_change":"2024-03-06T07:27:10.268216+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268216+0000","last_undegraded":"2024-03-06T07:27:10.268216+0000","last_fullsized":"2024-03-06T07:27:10.268216+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261822+0000","last_change":"2024-03-06T07:27:10.261822+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261822+0000","last_undegraded":"2024-03-06T07:27:10.261822+0000","last_fullsized":"2024-03-06T07:27:10.261822+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265508+0000","last_change":"2024-03-06T07:27:10.265508+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265508+0000","last_undegraded":"2024-03-06T07:27:10.265508+0000","last_fullsized":"2024-03-06T07:27:10.265508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.262335+0000","last_change":"2024-03-06T07:27:10.262335+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.262335+0000","last_undegraded":"2024-03-06T07:27:10.262335+0000","last_fullsized":"2024-03-06T07:27:10.262335+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.272656+0000","last_change":"2024-03-06T07:27:10.272656+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.272656+0000","last_undegraded":"2024-03-06T07:27:10.272656+0000","last_fullsized":"2024-03-06T07:27:10.272656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265513+0000","last_change":"2024-03-06T07:27:10.265513+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265513+0000","last_undegraded":"2024-03-06T07:27:10.265513+0000","last_fullsized":"2024-03-06T07:27:10.265513+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264450+0000","last_change":"2024-03-06T07:27:10.264450+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264450+0000","last_undegraded":"2024-03-06T07:27:10.264450+0000","last_fullsized":"2024-03-06T07:27:10.264450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264152+0000","last_change":"2024-03-06T07:27:10.264152+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264152+0000","last_undegraded":"2024-03-06T07:27:10.264152+0000","last_fullsized":"2024-03-06T07:27:10.264152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265454+0000","last_change":"2024-03-06T07:27:10.265454+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265454+0000","last_undegraded":"2024-03-06T07:27:10.265454+0000","last_fullsized":"2024-03-06T07:27:10.265454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265052+0000","last_change":"2024-03-06T07:27:10.265052+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265052+0000","last_undegraded":"2024-03-06T07:27:10.265052+0000","last_fullsized":"2024-03-06T07:27:10.265052+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267742+0000","last_change":"2024-03-06T07:27:10.267742+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267742+0000","last_undegraded":"2024-03-06T07:27:10.267742+0000","last_fullsized":"2024-03-06T07:27:10.267742+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265433+0000","last_change":"2024-03-06T07:27:10.265433+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265433+0000","last_undegraded":"2024-03-06T07:27:10.265433+0000","last_fullsized":"2024-03-06T07:27:10.265433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276306+0000","last_change":"2024-03-06T07:27:10.271295+0000","last_active":"2024-03-06T07:27:10.276306+0000","last_peered":"2024-03-06T07:27:10.276306+0000","last_clean":"2024-03-06T07:27:10.276306+0000","last_became_active":"2024-03-06T07:27:10.270894+0000","last_became_peered":"2024-03-06T07:27:10.270894+0000","last_unstale":"2024-03-06T07:27:10.276306+0000","last_undegraded":"2024-03-06T07:27:10.276306+0000","last_fullsized":"2024-03-06T07:27:10.276306+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:47.748004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265204+0000","last_change":"2024-03-06T07:27:10.265204+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265204+0000","last_undegraded":"2024-03-06T07:27:10.265204+0000","last_fullsized":"2024-03-06T07:27:10.265204+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280189+0000","last_change":"2024-03-06T07:27:10.279732+0000","last_active":"2024-03-06T07:27:10.280189+0000","last_peered":"2024-03-06T07:27:10.280189+0000","last_clean":"2024-03-06T07:27:10.280189+0000","last_became_active":"2024-03-06T07:27:10.279431+0000","last_became_peered":"2024-03-06T07:27:10.279431+0000","last_unstale":"2024-03-06T07:27:10.280189+0000","last_undegraded":"2024-03-06T07:27:10.280189+0000","last_fullsized":"2024-03-06T07:27:10.280189+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:10.859022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266552+0000","last_change":"2024-03-06T07:27:10.266552+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266552+0000","last_undegraded":"2024-03-06T07:27:10.266552+0000","last_fullsized":"2024-03-06T07:27:10.266552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279750+0000","last_change":"2024-03-06T07:27:10.278840+0000","last_active":"2024-03-06T07:27:10.279750+0000","last_peered":"2024-03-06T07:27:10.279750+0000","last_clean":"2024-03-06T07:27:10.279750+0000","last_became_active":"2024-03-06T07:27:10.278584+0000","last_became_peered":"2024-03-06T07:27:10.278584+0000","last_unstale":"2024-03-06T07:27:10.279750+0000","last_undegraded":"2024-03-06T07:27:10.279750+0000","last_fullsized":"2024-03-06T07:27:10.279750+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:12.045311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.271902+0000","last_change":"2024-03-06T07:27:10.271902+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.271902+0000","last_undegraded":"2024-03-06T07:27:10.271902+0000","last_fullsized":"2024-03-06T07:27:10.271902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279717+0000","last_change":"2024-03-06T07:27:10.278593+0000","last_active":"2024-03-06T07:27:10.279717+0000","last_peered":"2024-03-06T07:27:10.279717+0000","last_clean":"2024-03-06T07:27:10.279717+0000","last_became_active":"2024-03-06T07:27:10.278332+0000","last_became_peered":"2024-03-06T07:27:10.278332+0000","last_unstale":"2024-03-06T07:27:10.279717+0000","last_undegraded":"2024-03-06T07:27:10.279717+0000","last_fullsized":"2024-03-06T07:27:10.279717+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:59.810451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264549+0000","last_change":"2024-03-06T07:27:10.264549+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264549+0000","last_undegraded":"2024-03-06T07:27:10.264549+0000","last_fullsized":"2024-03-06T07:27:10.264549+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276275+0000","last_change":"2024-03-06T07:27:10.271671+0000","last_active":"2024-03-06T07:27:10.276275+0000","last_peered":"2024-03-06T07:27:10.276275+0000","last_clean":"2024-03-06T07:27:10.276275+0000","last_became_active":"2024-03-06T07:27:10.271376+0000","last_became_peered":"2024-03-06T07:27:10.271376+0000","last_unstale":"2024-03-06T07:27:10.276275+0000","last_undegraded":"2024-03-06T07:27:10.276275+0000","last_fullsized":"2024-03-06T07:27:10.276275+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:25.067873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263608+0000","last_change":"2024-03-06T07:27:10.263608+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263608+0000","last_undegraded":"2024-03-06T07:27:10.263608+0000","last_fullsized":"2024-03-06T07:27:10.263608+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.272607+0000","last_change":"2024-03-06T07:27:10.269066+0000","last_active":"2024-03-06T07:27:10.272607+0000","last_peered":"2024-03-06T07:27:10.272607+0000","last_clean":"2024-03-06T07:27:10.272607+0000","last_became_active":"2024-03-06T07:27:10.268772+0000","last_became_peered":"2024-03-06T07:27:10.268772+0000","last_unstale":"2024-03-06T07:27:10.272607+0000","last_undegraded":"2024-03-06T07:27:10.272607+0000","last_fullsized":"2024-03-06T07:27:10.272607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:50.615510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268623+0000","last_change":"2024-03-06T07:27:10.268623+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268623+0000","last_undegraded":"2024-03-06T07:27:10.268623+0000","last_fullsized":"2024-03-06T07:27:10.268623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.269106+0000","last_change":"2024-03-06T07:27:10.268260+0000","last_active":"2024-03-06T07:27:10.269106+0000","last_peered":"2024-03-06T07:27:10.269106+0000","last_clean":"2024-03-06T07:27:10.269106+0000","last_became_active":"2024-03-06T07:27:10.267951+0000","last_became_peered":"2024-03-06T07:27:10.267951+0000","last_unstale":"2024-03-06T07:27:10.269106+0000","last_undegraded":"2024-03-06T07:27:10.269106+0000","last_fullsized":"2024-03-06T07:27:10.269106+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:23.498721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266447+0000","last_change":"2024-03-06T07:27:10.266447+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266447+0000","last_undegraded":"2024-03-06T07:27:10.266447+0000","last_fullsized":"2024-03-06T07:27:10.266447+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280949+0000","last_change":"2024-03-06T07:27:10.278526+0000","last_active":"2024-03-06T07:27:10.280949+0000","last_peered":"2024-03-06T07:27:10.280949+0000","last_clean":"2024-03-06T07:27:10.280949+0000","last_became_active":"2024-03-06T07:27:10.277985+0000","last_became_peered":"2024-03-06T07:27:10.277985+0000","last_unstale":"2024-03-06T07:27:10.280949+0000","last_undegraded":"2024-03-06T07:27:10.280949+0000","last_fullsized":"2024-03-06T07:27:10.280949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:23.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264647+0000","last_change":"2024-03-06T07:27:10.264647+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264647+0000","last_undegraded":"2024-03-06T07:27:10.264647+0000","last_fullsized":"2024-03-06T07:27:10.264647+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278297+0000","last_change":"2024-03-06T07:27:10.277574+0000","last_active":"2024-03-06T07:27:10.278297+0000","last_peered":"2024-03-06T07:27:10.278297+0000","last_clean":"2024-03-06T07:27:10.278297+0000","last_became_active":"2024-03-06T07:27:10.277328+0000","last_became_peered":"2024-03-06T07:27:10.277328+0000","last_unstale":"2024-03-06T07:27:10.278297+0000","last_undegraded":"2024-03-06T07:27:10.278297+0000","last_fullsized":"2024-03-06T07:27:10.278297+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:48.143046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264239+0000","last_change":"2024-03-06T07:27:10.264239+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264239+0000","last_undegraded":"2024-03-06T07:27:10.264239+0000","last_fullsized":"2024-03-06T07:27:10.264239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.272084+0000","last_change":"2024-03-06T07:27:10.269028+0000","last_active":"2024-03-06T07:27:10.272084+0000","last_peered":"2024-03-06T07:27:10.272084+0000","last_clean":"2024-03-06T07:27:10.272084+0000","last_became_active":"2024-03-06T07:27:10.268733+0000","last_became_peered":"2024-03-06T07:27:10.268733+0000","last_unstale":"2024-03-06T07:27:10.272084+0000","last_undegraded":"2024-03-06T07:27:10.272084+0000","last_fullsized":"2024-03-06T07:27:10.272084+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:05.178263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263939+0000","last_change":"2024-03-06T07:27:10.263939+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263939+0000","last_undegraded":"2024-03-06T07:27:10.263939+0000","last_fullsized":"2024-03-06T07:27:10.263939+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281719+0000","last_change":"2024-03-06T07:27:10.281030+0000","last_active":"2024-03-06T07:27:10.281719+0000","last_peered":"2024-03-06T07:27:10.281719+0000","last_clean":"2024-03-06T07:27:10.281719+0000","last_became_active":"2024-03-06T07:27:10.280767+0000","last_became_peered":"2024-03-06T07:27:10.280767+0000","last_unstale":"2024-03-06T07:27:10.281719+0000","last_undegraded":"2024-03-06T07:27:10.281719+0000","last_fullsized":"2024-03-06T07:27:10.281719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:58.940827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264009+0000","last_change":"2024-03-06T07:27:10.264009+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264009+0000","last_undegraded":"2024-03-06T07:27:10.264009+0000","last_fullsized":"2024-03-06T07:27:10.264009+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280068+0000","last_change":"2024-03-06T07:27:10.278124+0000","last_active":"2024-03-06T07:27:10.280068+0000","last_peered":"2024-03-06T07:27:10.280068+0000","last_clean":"2024-03-06T07:27:10.280068+0000","last_became_active":"2024-03-06T07:27:10.277686+0000","last_became_peered":"2024-03-06T07:27:10.277686+0000","last_unstale":"2024-03-06T07:27:10.280068+0000","last_undegraded":"2024-03-06T07:27:10.280068+0000","last_fullsized":"2024-03-06T07:27:10.280068+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:55.423671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263293+0000","last_change":"2024-03-06T07:27:10.263293+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263293+0000","last_undegraded":"2024-03-06T07:27:10.263293+0000","last_fullsized":"2024-03-06T07:27:10.263293+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276302+0000","last_change":"2024-03-06T07:27:10.272261+0000","last_active":"2024-03-06T07:27:10.276302+0000","last_peered":"2024-03-06T07:27:10.276302+0000","last_clean":"2024-03-06T07:27:10.276302+0000","last_became_active":"2024-03-06T07:27:10.272009+0000","last_became_peered":"2024-03-06T07:27:10.272009+0000","last_unstale":"2024-03-06T07:27:10.276302+0000","last_undegraded":"2024-03-06T07:27:10.276302+0000","last_fullsized":"2024-03-06T07:27:10.276302+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:13.354906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263736+0000","last_change":"2024-03-06T07:27:10.263736+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263736+0000","last_undegraded":"2024-03-06T07:27:10.263736+0000","last_fullsized":"2024-03-06T07:27:10.263736+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279136+0000","last_change":"2024-03-06T07:27:10.275499+0000","last_active":"2024-03-06T07:27:10.279136+0000","last_peered":"2024-03-06T07:27:10.279136+0000","last_clean":"2024-03-06T07:27:10.279136+0000","last_became_active":"2024-03-06T07:27:10.275217+0000","last_became_peered":"2024-03-06T07:27:10.275217+0000","last_unstale":"2024-03-06T07:27:10.279136+0000","last_undegraded":"2024-03-06T07:27:10.279136+0000","last_fullsized":"2024-03-06T07:27:10.279136+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:24.392230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264215+0000","last_change":"2024-03-06T07:27:10.264215+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264215+0000","last_undegraded":"2024-03-06T07:27:10.264215+0000","last_fullsized":"2024-03-06T07:27:10.264215+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278410+0000","last_change":"2024-03-06T07:27:10.276787+0000","last_active":"2024-03-06T07:27:10.278410+0000","last_peered":"2024-03-06T07:27:10.278410+0000","last_clean":"2024-03-06T07:27:10.278410+0000","last_became_active":"2024-03-06T07:27:10.276489+0000","last_became_peered":"2024-03-06T07:27:10.276489+0000","last_unstale":"2024-03-06T07:27:10.278410+0000","last_undegraded":"2024-03-06T07:27:10.278410+0000","last_fullsized":"2024-03-06T07:27:10.278410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:12.606496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.270279+0000","last_change":"2024-03-06T07:27:10.270279+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.270279+0000","last_undegraded":"2024-03-06T07:27:10.270279+0000","last_fullsized":"2024-03-06T07:27:10.270279+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281716+0000","last_change":"2024-03-06T07:27:10.280931+0000","last_active":"2024-03-06T07:27:10.281716+0000","last_peered":"2024-03-06T07:27:10.281716+0000","last_clean":"2024-03-06T07:27:10.281716+0000","last_became_active":"2024-03-06T07:27:10.280631+0000","last_became_peered":"2024-03-06T07:27:10.280631+0000","last_unstale":"2024-03-06T07:27:10.281716+0000","last_undegraded":"2024-03-06T07:27:10.281716+0000","last_fullsized":"2024-03-06T07:27:10.281716+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:44.772144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265754+0000","last_change":"2024-03-06T07:27:10.265754+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265754+0000","last_undegraded":"2024-03-06T07:27:10.265754+0000","last_fullsized":"2024-03-06T07:27:10.265754+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278330+0000","last_change":"2024-03-06T07:27:10.275684+0000","last_active":"2024-03-06T07:27:10.278330+0000","last_peered":"2024-03-06T07:27:10.278330+0000","last_clean":"2024-03-06T07:27:10.278330+0000","last_became_active":"2024-03-06T07:27:10.275341+0000","last_became_peered":"2024-03-06T07:27:10.275341+0000","last_unstale":"2024-03-06T07:27:10.278330+0000","last_undegraded":"2024-03-06T07:27:10.278330+0000","last_fullsized":"2024-03-06T07:27:10.278330+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:22.219213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269219+0000","last_change":"2024-03-06T07:27:10.269219+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269219+0000","last_undegraded":"2024-03-06T07:27:10.269219+0000","last_fullsized":"2024-03-06T07:27:10.269219+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276412+0000","last_change":"2024-03-06T07:27:10.273231+0000","last_active":"2024-03-06T07:27:10.276412+0000","last_peered":"2024-03-06T07:27:10.276412+0000","last_clean":"2024-03-06T07:27:10.276412+0000","last_became_active":"2024-03-06T07:27:10.272878+0000","last_became_peered":"2024-03-06T07:27:10.272878+0000","last_unstale":"2024-03-06T07:27:10.276412+0000","last_undegraded":"2024-03-06T07:27:10.276412+0000","last_fullsized":"2024-03-06T07:27:10.276412+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:28.515777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269745+0000","last_change":"2024-03-06T07:27:10.269745+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269745+0000","last_undegraded":"2024-03-06T07:27:10.269745+0000","last_fullsized":"2024-03-06T07:27:10.269745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281876+0000","last_change":"2024-03-06T07:27:10.277902+0000","last_active":"2024-03-06T07:27:10.281876+0000","last_peered":"2024-03-06T07:27:10.281876+0000","last_clean":"2024-03-06T07:27:10.281876+0000","last_became_active":"2024-03-06T07:27:10.277622+0000","last_became_peered":"2024-03-06T07:27:10.277622+0000","last_unstale":"2024-03-06T07:27:10.281876+0000","last_undegraded":"2024-03-06T07:27:10.281876+0000","last_fullsized":"2024-03-06T07:27:10.281876+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:14.247605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267319+0000","last_change":"2024-03-06T07:27:10.267319+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267319+0000","last_undegraded":"2024-03-06T07:27:10.267319+0000","last_fullsized":"2024-03-06T07:27:10.267319+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279670+0000","last_change":"2024-03-06T07:27:10.277544+0000","last_active":"2024-03-06T07:27:10.279670+0000","last_peered":"2024-03-06T07:27:10.279670+0000","last_clean":"2024-03-06T07:27:10.279670+0000","last_became_active":"2024-03-06T07:27:10.276523+0000","last_became_peered":"2024-03-06T07:27:10.276523+0000","last_unstale":"2024-03-06T07:27:10.279670+0000","last_undegraded":"2024-03-06T07:27:10.279670+0000","last_fullsized":"2024-03-06T07:27:10.279670+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:25.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269843+0000","last_change":"2024-03-06T07:27:10.269843+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269843+0000","last_undegraded":"2024-03-06T07:27:10.269843+0000","last_fullsized":"2024-03-06T07:27:10.269843+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279775+0000","last_change":"2024-03-06T07:27:10.278830+0000","last_active":"2024-03-06T07:27:10.279775+0000","last_peered":"2024-03-06T07:27:10.279775+0000","last_clean":"2024-03-06T07:27:10.279775+0000","last_became_active":"2024-03-06T07:27:10.278106+0000","last_became_peered":"2024-03-06T07:27:10.278106+0000","last_unstale":"2024-03-06T07:27:10.279775+0000","last_undegraded":"2024-03-06T07:27:10.279775+0000","last_fullsized":"2024-03-06T07:27:10.279775+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.266466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269136+0000","last_change":"2024-03-06T07:27:10.269136+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269136+0000","last_undegraded":"2024-03-06T07:27:10.269136+0000","last_fullsized":"2024-03-06T07:27:10.269136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274650+0000","last_change":"2024-03-06T07:27:10.271560+0000","last_active":"2024-03-06T07:27:10.274650+0000","last_peered":"2024-03-06T07:27:10.274650+0000","last_clean":"2024-03-06T07:27:10.274650+0000","last_became_active":"2024-03-06T07:27:10.271296+0000","last_became_peered":"2024-03-06T07:27:10.271296+0000","last_unstale":"2024-03-06T07:27:10.274650+0000","last_undegraded":"2024-03-06T07:27:10.274650+0000","last_fullsized":"2024-03-06T07:27:10.274650+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:00.541674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263530+0000","last_change":"2024-03-06T07:27:10.263530+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263530+0000","last_undegraded":"2024-03-06T07:27:10.263530+0000","last_fullsized":"2024-03-06T07:27:10.263530+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281370+0000","last_change":"2024-03-06T07:27:10.279864+0000","last_active":"2024-03-06T07:27:10.281370+0000","last_peered":"2024-03-06T07:27:10.281370+0000","last_clean":"2024-03-06T07:27:10.281370+0000","last_became_active":"2024-03-06T07:27:10.279622+0000","last_became_peered":"2024-03-06T07:27:10.279622+0000","last_unstale":"2024-03-06T07:27:10.281370+0000","last_undegraded":"2024-03-06T07:27:10.281370+0000","last_fullsized":"2024-03-06T07:27:10.281370+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:33.259974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264152+0000","last_change":"2024-03-06T07:27:10.264152+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264152+0000","last_undegraded":"2024-03-06T07:27:10.264152+0000","last_fullsized":"2024-03-06T07:27:10.264152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276867+0000","last_change":"2024-03-06T07:27:10.273788+0000","last_active":"2024-03-06T07:27:10.276867+0000","last_peered":"2024-03-06T07:27:10.276867+0000","last_clean":"2024-03-06T07:27:10.276867+0000","last_became_active":"2024-03-06T07:27:10.273542+0000","last_became_peered":"2024-03-06T07:27:10.273542+0000","last_unstale":"2024-03-06T07:27:10.276867+0000","last_undegraded":"2024-03-06T07:27:10.276867+0000","last_fullsized":"2024-03-06T07:27:10.276867+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:10.176108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266917+0000","last_change":"2024-03-06T07:27:10.266917+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266917+0000","last_undegraded":"2024-03-06T07:27:10.266917+0000","last_fullsized":"2024-03-06T07:27:10.266917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.270719+0000","last_change":"2024-03-06T07:27:10.269615+0000","last_active":"2024-03-06T07:27:10.270719+0000","last_peered":"2024-03-06T07:27:10.270719+0000","last_clean":"2024-03-06T07:27:10.270719+0000","last_became_active":"2024-03-06T07:27:10.269331+0000","last_became_peered":"2024-03-06T07:27:10.269331+0000","last_unstale":"2024-03-06T07:27:10.270719+0000","last_undegraded":"2024-03-06T07:27:10.270719+0000","last_fullsized":"2024-03-06T07:27:10.270719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:31.221037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265082+0000","last_change":"2024-03-06T07:27:10.265082+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265082+0000","last_undegraded":"2024-03-06T07:27:10.265082+0000","last_fullsized":"2024-03-06T07:27:10.265082+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274598+0000","last_change":"2024-03-06T07:27:10.273864+0000","last_active":"2024-03-06T07:27:10.274598+0000","last_peered":"2024-03-06T07:27:10.274598+0000","last_clean":"2024-03-06T07:27:10.274598+0000","last_became_active":"2024-03-06T07:27:10.273611+0000","last_became_peered":"2024-03-06T07:27:10.273611+0000","last_unstale":"2024-03-06T07:27:10.274598+0000","last_undegraded":"2024-03-06T07:27:10.274598+0000","last_fullsized":"2024-03-06T07:27:10.274598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:49.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263562+0000","last_change":"2024-03-06T07:27:10.263562+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263562+0000","last_undegraded":"2024-03-06T07:27:10.263562+0000","last_fullsized":"2024-03-06T07:27:10.263562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276271+0000","last_change":"2024-03-06T07:27:10.271689+0000","last_active":"2024-03-06T07:27:10.276271+0000","last_peered":"2024-03-06T07:27:10.276271+0000","last_clean":"2024-03-06T07:27:10.276271+0000","last_became_active":"2024-03-06T07:27:10.271421+0000","last_became_peered":"2024-03-06T07:27:10.271421+0000","last_unstale":"2024-03-06T07:27:10.276271+0000","last_undegraded":"2024-03-06T07:27:10.276271+0000","last_fullsized":"2024-03-06T07:27:10.276271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:20.049932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261430+0000","last_change":"2024-03-06T07:27:10.261430+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261430+0000","last_undegraded":"2024-03-06T07:27:10.261430+0000","last_fullsized":"2024-03-06T07:27:10.261430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.269674+0000","last_change":"2024-03-06T07:27:10.268891+0000","last_active":"2024-03-06T07:27:10.269674+0000","last_peered":"2024-03-06T07:27:10.269674+0000","last_clean":"2024-03-06T07:27:10.269674+0000","last_became_active":"2024-03-06T07:27:10.268536+0000","last_became_peered":"2024-03-06T07:27:10.268536+0000","last_unstale":"2024-03-06T07:27:10.269674+0000","last_undegraded":"2024-03-06T07:27:10.269674+0000","last_fullsized":"2024-03-06T07:27:10.269674+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:51.960409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263082+0000","last_change":"2024-03-06T07:27:10.263082+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263082+0000","last_undegraded":"2024-03-06T07:27:10.263082+0000","last_fullsized":"2024-03-06T07:27:10.263082+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276122+0000","last_change":"2024-03-06T07:27:10.272914+0000","last_active":"2024-03-06T07:27:10.276122+0000","last_peered":"2024-03-06T07:27:10.276122+0000","last_clean":"2024-03-06T07:27:10.276122+0000","last_became_active":"2024-03-06T07:27:10.272635+0000","last_became_peered":"2024-03-06T07:27:10.272635+0000","last_unstale":"2024-03-06T07:27:10.276122+0000","last_undegraded":"2024-03-06T07:27:10.276122+0000","last_fullsized":"2024-03-06T07:27:10.276122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:58.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268710+0000","last_change":"2024-03-06T07:27:10.268710+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268710+0000","last_undegraded":"2024-03-06T07:27:10.268710+0000","last_fullsized":"2024-03-06T07:27:10.268710+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278239+0000","last_change":"2024-03-06T07:27:10.276957+0000","last_active":"2024-03-06T07:27:10.278239+0000","last_peered":"2024-03-06T07:27:10.278239+0000","last_clean":"2024-03-06T07:27:10.278239+0000","last_became_active":"2024-03-06T07:27:10.276707+0000","last_became_peered":"2024-03-06T07:27:10.276707+0000","last_unstale":"2024-03-06T07:27:10.278239+0000","last_undegraded":"2024-03-06T07:27:10.278239+0000","last_fullsized":"2024-03-06T07:27:10.278239+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:31.396430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264427+0000","last_change":"2024-03-06T07:27:10.264427+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264427+0000","last_undegraded":"2024-03-06T07:27:10.264427+0000","last_fullsized":"2024-03-06T07:27:10.264427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276211+0000","last_change":"2024-03-06T07:27:10.275390+0000","last_active":"2024-03-06T07:27:10.276211+0000","last_peered":"2024-03-06T07:27:10.276211+0000","last_clean":"2024-03-06T07:27:10.276211+0000","last_became_active":"2024-03-06T07:27:10.274903+0000","last_became_peered":"2024-03-06T07:27:10.274903+0000","last_unstale":"2024-03-06T07:27:10.276211+0000","last_undegraded":"2024-03-06T07:27:10.276211+0000","last_fullsized":"2024-03-06T07:27:10.276211+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:02.332832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263304+0000","last_change":"2024-03-06T07:27:10.263304+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263304+0000","last_undegraded":"2024-03-06T07:27:10.263304+0000","last_fullsized":"2024-03-06T07:27:10.263304+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281890+0000","last_change":"2024-03-06T07:27:10.277891+0000","last_active":"2024-03-06T07:27:10.281890+0000","last_peered":"2024-03-06T07:27:10.281890+0000","last_clean":"2024-03-06T07:27:10.281890+0000","last_became_active":"2024-03-06T07:27:10.277582+0000","last_became_peered":"2024-03-06T07:27:10.277582+0000","last_unstale":"2024-03-06T07:27:10.281890+0000","last_undegraded":"2024-03-06T07:27:10.281890+0000","last_fullsized":"2024-03-06T07:27:10.281890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:16.855383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":332,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274060+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:27:10.274060+0000","last_peered":"2024-03-06T07:27:10.274060+0000","last_clean":"2024-03-06T07:27:10.274060+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:27:10.274060+0000","last_undegraded":"2024-03-06T07:27:10.274060+0000","last_fullsized":"2024-03-06T07:27:10.274060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":1156184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40776,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331064,"statfs":{"total":96636764160,"available":96595009536,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40792,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331048,"statfs":{"total":96636764160,"available":96594993152,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41356,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330484,"statfs":{"total":96636764160,"available":96594415616,"internally_reserved":0,"allocated":1900544,"data_stored":1149909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.412+0000 7fd2e4dac640 1 Processor -- start 2024-03-06T07:27:14.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.412+0000 7fd2e4dac640 1 -- start start 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2e4dac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2e015dfe0 0x7fd2e00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2e4dac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00b3430 con 0x7fd2e00cf830 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2e4dac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00b3710 con 0x7fd2e00d29c0 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2e4dac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e00b31e0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2de59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2e015dfe0 0x7fd2e00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2d7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd2e00cf830 0x7fd2e00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43512/0 (socket says 172.21.15.122:43512) 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2d7fff640 1 -- 172.21.15.122:0/2050048099 learned_addr learned my addr 172.21.15.122:0/2050048099 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2de59c640 1 -- 172.21.15.122:0/2050048099 >> v1:172.21.15.169:6789/0 conn(0x7fd2e00d29c0 legacy=0x7fd2e00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2de59c640 1 -- 172.21.15.122:0/2050048099 >> v1:172.21.15.122:6789/0 conn(0x7fd2e00cf830 legacy=0x7fd2e00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2de59c640 1 -- 172.21.15.122:0/2050048099 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00b39f0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.413+0000 7fd2ddd9b640 1 -- 172.21.15.122:0/2050048099 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536083485 0 0) 0x7fd2e00b3430 con 0x7fd2e00cf830 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.414+0000 7fd2de59c640 1 --2- 172.21.15.122:0/2050048099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cf2f0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00b3360 tx=0x7fd2d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9791beba64bcf86d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.414+0000 7fd2ddd9b640 1 -- 172.21.15.122:0/2050048099 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0035020 con 0x7fd2e015dfe0 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.414+0000 7fd2ddd9b640 1 -- 172.21.15.122:0/2050048099 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0031830 con 0x7fd2e015dfe0 2024-03-06T07:27:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.414+0000 7fd2ddd9b640 1 -- 172.21.15.122:0/2050048099 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0031c00 con 0x7fd2e015dfe0 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2050048099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 msgr2=0x7fd2e00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2050048099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cf2f0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00b3360 tx=0x7fd2d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2050048099 shutdown_connections 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2050048099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cf2f0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2050048099 >> 172.21.15.122:0/2050048099 conn(0x7fd2e00d3cd0 msgr2=0x7fd2e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2050048099 shutdown_connections 2024-03-06T07:27:14.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.415+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2050048099 wait complete. 2024-03-06T07:27:14.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.416+0000 7fd2e4dac640 1 Processor -- start 2024-03-06T07:27:14.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.416+0000 7fd2e4dac640 1 -- start start 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2e00cf830 0x7fd2e015c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2e00d29c0 0x7fd2e015b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e015e3c0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00b3d60 con 0x7fd2e00d29c0 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e015e540 con 0x7fd2e00cf830 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44734/0 (socket says 172.21.15.122:44734) 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 -- 172.21.15.122:0/2348046613 learned_addr learned my addr 172.21.15.122:0/2348046613 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2d7fff640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2e00d29c0 0x7fd2e015b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2de59c640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2e00cf830 0x7fd2e015c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 -- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2e00cf830 msgr2=0x7fd2e015c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2e00cf830 0x7fd2e015c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 -- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2e00d29c0 msgr2=0x7fd2e015b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2e00d29c0 0x7fd2e015b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 -- 172.21.15.122:0/2348046613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00cbaf0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2ded9d640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cd310 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800c780 tx=0x7fd2d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8017070 con 0x7fd2e015dfe0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8003da0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80052b0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e00b79c0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.417+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e00cbdd0 con 0x7fd2e015dfe0 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.418+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e004f330 con 0x7fd2e015dfe0 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.419+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2d801a070 con 0x7fd2e015dfe0 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.419+0000 7fd2d6ffd640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b500 0x7fd2c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.419+0000 7fd2de59c640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b500 0x7fd2c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.419+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2d8089810 con 0x7fd2e015dfe0 2024-03-06T07:27:14.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.420+0000 7fd2de59c640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b500 0x7fd2c006d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0036000 tx=0x7fd2d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.424+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2d8058690 con 0x7fd2e015dfe0 2024-03-06T07:27:14.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.600+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2e00d4a80 con 0x7fd2c006b500 2024-03-06T07:27:14.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.606+0000 7fd2d6ffd640 1 -- 172.21.15.122:0/2348046613 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247087 (secure 0 0 0) 0x7fd2e00d4a80 con 0x7fd2c006b500 2024-03-06T07:27:14.611 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:14.613 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:27:14.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b500 msgr2=0x7fd2c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b500 0x7fd2c006d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0036000 tx=0x7fd2d0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 msgr2=0x7fd2e00cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cd310 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800c780 tx=0x7fd2d800cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 shutdown_connections 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd2c006b500 0x7fd2c006d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2e015dfe0 0x7fd2e00cd310 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2e00d29c0 0x7fd2e015b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 --2- 172.21.15.122:0/2348046613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2e00cf830 0x7fd2e015c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 >> 172.21.15.122:0/2348046613 conn(0x7fd2e00d3cd0 msgr2=0x7fd2e006c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 shutdown_connections 2024-03-06T07:27:14.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.609+0000 7fd2e4dac640 1 -- 172.21.15.122:0/2348046613 wait complete. 2024-03-06T07:27:14.622 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:13.396067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323280,"kb_used_data":11380,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651440,"statfs":{"total":773094113280,"available":772763074560,"internally_reserved":0,"allocated":11653120,"data_stored":5663339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.140367"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267598+0000","last_change":"2024-03-06T07:27:10.267598+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267598+0000","last_undegraded":"2024-03-06T07:27:10.267598+0000","last_fullsized":"2024-03-06T07:27:10.267598+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261734+0000","last_change":"2024-03-06T07:27:10.261734+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261734+0000","last_undegraded":"2024-03-06T07:27:10.261734+0000","last_fullsized":"2024-03-06T07:27:10.261734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273806+0000","last_change":"2024-03-06T07:27:10.273806+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273806+0000","last_undegraded":"2024-03-06T07:27:10.273806+0000","last_fullsized":"2024-03-06T07:27:10.273806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265833+0000","last_change":"2024-03-06T07:27:10.265833+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265833+0000","last_undegraded":"2024-03-06T07:27:10.265833+0000","last_fullsized":"2024-03-06T07:27:10.265833+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266357+0000","last_change":"2024-03-06T07:27:10.266357+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266357+0000","last_undegraded":"2024-03-06T07:27:10.266357+0000","last_fullsized":"2024-03-06T07:27:10.266357+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265033+0000","last_change":"2024-03-06T07:27:10.265033+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265033+0000","last_undegraded":"2024-03-06T07:27:10.265033+0000","last_fullsized":"2024-03-06T07:27:10.265033+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265851+0000","last_change":"2024-03-06T07:27:10.265851+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265851+0000","last_undegraded":"2024-03-06T07:27:10.265851+0000","last_fullsized":"2024-03-06T07:27:10.265851+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269538+0000","last_change":"2024-03-06T07:27:10.269538+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269538+0000","last_undegraded":"2024-03-06T07:27:10.269538+0000","last_fullsized":"2024-03-06T07:27:10.269538+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267258+0000","last_change":"2024-03-06T07:27:10.267258+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267258+0000","last_undegraded":"2024-03-06T07:27:10.267258+0000","last_fullsized":"2024-03-06T07:27:10.267258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273029+0000","last_change":"2024-03-06T07:27:10.273029+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273029+0000","last_undegraded":"2024-03-06T07:27:10.273029+0000","last_fullsized":"2024-03-06T07:27:10.273029+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261449+0000","last_change":"2024-03-06T07:27:10.261449+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261449+0000","last_undegraded":"2024-03-06T07:27:10.261449+0000","last_fullsized":"2024-03-06T07:27:10.261449+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265454+0000","last_change":"2024-03-06T07:27:10.265454+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265454+0000","last_undegraded":"2024-03-06T07:27:10.265454+0000","last_fullsized":"2024-03-06T07:27:10.265454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268184+0000","last_change":"2024-03-06T07:27:10.268184+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268184+0000","last_undegraded":"2024-03-06T07:27:10.268184+0000","last_fullsized":"2024-03-06T07:27:10.268184+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268837+0000","last_change":"2024-03-06T07:27:10.268837+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268837+0000","last_undegraded":"2024-03-06T07:27:10.268837+0000","last_fullsized":"2024-03-06T07:27:10.268837+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264950+0000","last_change":"2024-03-06T07:27:10.264950+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264950+0000","last_undegraded":"2024-03-06T07:27:10.264950+0000","last_fullsized":"2024-03-06T07:27:10.264950+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268981+0000","last_change":"2024-03-06T07:27:10.268981+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268981+0000","last_undegraded":"2024-03-06T07:27:10.268981+0000","last_fullsized":"2024-03-06T07:27:10.268981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.262813+0000","last_change":"2024-03-06T07:27:10.262813+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.262813+0000","last_undegraded":"2024-03-06T07:27:10.262813+0000","last_fullsized":"2024-03-06T07:27:10.262813+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264573+0000","last_change":"2024-03-06T07:27:10.264573+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264573+0000","last_undegraded":"2024-03-06T07:27:10.264573+0000","last_fullsized":"2024-03-06T07:27:10.264573+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273199+0000","last_change":"2024-03-06T07:27:10.273199+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273199+0000","last_undegraded":"2024-03-06T07:27:10.273199+0000","last_fullsized":"2024-03-06T07:27:10.273199+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264185+0000","last_change":"2024-03-06T07:27:10.264185+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264185+0000","last_undegraded":"2024-03-06T07:27:10.264185+0000","last_fullsized":"2024-03-06T07:27:10.264185+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261404+0000","last_change":"2024-03-06T07:27:10.261404+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261404+0000","last_undegraded":"2024-03-06T07:27:10.261404+0000","last_fullsized":"2024-03-06T07:27:10.261404+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268216+0000","last_change":"2024-03-06T07:27:10.268216+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268216+0000","last_undegraded":"2024-03-06T07:27:10.268216+0000","last_fullsized":"2024-03-06T07:27:10.268216+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261822+0000","last_change":"2024-03-06T07:27:10.261822+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261822+0000","last_undegraded":"2024-03-06T07:27:10.261822+0000","last_fullsized":"2024-03-06T07:27:10.261822+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265508+0000","last_change":"2024-03-06T07:27:10.265508+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265508+0000","last_undegraded":"2024-03-06T07:27:10.265508+0000","last_fullsized":"2024-03-06T07:27:10.265508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.262335+0000","last_change":"2024-03-06T07:27:10.262335+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.262335+0000","last_undegraded":"2024-03-06T07:27:10.262335+0000","last_fullsized":"2024-03-06T07:27:10.262335+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.272656+0000","last_change":"2024-03-06T07:27:10.272656+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.272656+0000","last_undegraded":"2024-03-06T07:27:10.272656+0000","last_fullsized":"2024-03-06T07:27:10.272656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265513+0000","last_change":"2024-03-06T07:27:10.265513+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265513+0000","last_undegraded":"2024-03-06T07:27:10.265513+0000","last_fullsized":"2024-03-06T07:27:10.265513+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264450+0000","last_change":"2024-03-06T07:27:10.264450+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264450+0000","last_undegraded":"2024-03-06T07:27:10.264450+0000","last_fullsized":"2024-03-06T07:27:10.264450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264152+0000","last_change":"2024-03-06T07:27:10.264152+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264152+0000","last_undegraded":"2024-03-06T07:27:10.264152+0000","last_fullsized":"2024-03-06T07:27:10.264152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265454+0000","last_change":"2024-03-06T07:27:10.265454+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265454+0000","last_undegraded":"2024-03-06T07:27:10.265454+0000","last_fullsized":"2024-03-06T07:27:10.265454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265052+0000","last_change":"2024-03-06T07:27:10.265052+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265052+0000","last_undegraded":"2024-03-06T07:27:10.265052+0000","last_fullsized":"2024-03-06T07:27:10.265052+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267742+0000","last_change":"2024-03-06T07:27:10.267742+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267742+0000","last_undegraded":"2024-03-06T07:27:10.267742+0000","last_fullsized":"2024-03-06T07:27:10.267742+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265433+0000","last_change":"2024-03-06T07:27:10.265433+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265433+0000","last_undegraded":"2024-03-06T07:27:10.265433+0000","last_fullsized":"2024-03-06T07:27:10.265433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276306+0000","last_change":"2024-03-06T07:27:10.271295+0000","last_active":"2024-03-06T07:27:10.276306+0000","last_peered":"2024-03-06T07:27:10.276306+0000","last_clean":"2024-03-06T07:27:10.276306+0000","last_became_active":"2024-03-06T07:27:10.270894+0000","last_became_peered":"2024-03-06T07:27:10.270894+0000","last_unstale":"2024-03-06T07:27:10.276306+0000","last_undegraded":"2024-03-06T07:27:10.276306+0000","last_fullsized":"2024-03-06T07:27:10.276306+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:47.748004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265204+0000","last_change":"2024-03-06T07:27:10.265204+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265204+0000","last_undegraded":"2024-03-06T07:27:10.265204+0000","last_fullsized":"2024-03-06T07:27:10.265204+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280189+0000","last_change":"2024-03-06T07:27:10.279732+0000","last_active":"2024-03-06T07:27:10.280189+0000","last_peered":"2024-03-06T07:27:10.280189+0000","last_clean":"2024-03-06T07:27:10.280189+0000","last_became_active":"2024-03-06T07:27:10.279431+0000","last_became_peered":"2024-03-06T07:27:10.279431+0000","last_unstale":"2024-03-06T07:27:10.280189+0000","last_undegraded":"2024-03-06T07:27:10.280189+0000","last_fullsized":"2024-03-06T07:27:10.280189+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:10.859022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266552+0000","last_change":"2024-03-06T07:27:10.266552+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266552+0000","last_undegraded":"2024-03-06T07:27:10.266552+0000","last_fullsized":"2024-03-06T07:27:10.266552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279750+0000","last_change":"2024-03-06T07:27:10.278840+0000","last_active":"2024-03-06T07:27:10.279750+0000","last_peered":"2024-03-06T07:27:10.279750+0000","last_clean":"2024-03-06T07:27:10.279750+0000","last_became_active":"2024-03-06T07:27:10.278584+0000","last_became_peered":"2024-03-06T07:27:10.278584+0000","last_unstale":"2024-03-06T07:27:10.279750+0000","last_undegraded":"2024-03-06T07:27:10.279750+0000","last_fullsized":"2024-03-06T07:27:10.279750+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:12.045311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.271902+0000","last_change":"2024-03-06T07:27:10.271902+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.271902+0000","last_undegraded":"2024-03-06T07:27:10.271902+0000","last_fullsized":"2024-03-06T07:27:10.271902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279717+0000","last_change":"2024-03-06T07:27:10.278593+0000","last_active":"2024-03-06T07:27:10.279717+0000","last_peered":"2024-03-06T07:27:10.279717+0000","last_clean":"2024-03-06T07:27:10.279717+0000","last_became_active":"2024-03-06T07:27:10.278332+0000","last_became_peered":"2024-03-06T07:27:10.278332+0000","last_unstale":"2024-03-06T07:27:10.279717+0000","last_undegraded":"2024-03-06T07:27:10.279717+0000","last_fullsized":"2024-03-06T07:27:10.279717+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:59.810451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264549+0000","last_change":"2024-03-06T07:27:10.264549+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264549+0000","last_undegraded":"2024-03-06T07:27:10.264549+0000","last_fullsized":"2024-03-06T07:27:10.264549+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276275+0000","last_change":"2024-03-06T07:27:10.271671+0000","last_active":"2024-03-06T07:27:10.276275+0000","last_peered":"2024-03-06T07:27:10.276275+0000","last_clean":"2024-03-06T07:27:10.276275+0000","last_became_active":"2024-03-06T07:27:10.271376+0000","last_became_peered":"2024-03-06T07:27:10.271376+0000","last_unstale":"2024-03-06T07:27:10.276275+0000","last_undegraded":"2024-03-06T07:27:10.276275+0000","last_fullsized":"2024-03-06T07:27:10.276275+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:25.067873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263608+0000","last_change":"2024-03-06T07:27:10.263608+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263608+0000","last_undegraded":"2024-03-06T07:27:10.263608+0000","last_fullsized":"2024-03-06T07:27:10.263608+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.272607+0000","last_change":"2024-03-06T07:27:10.269066+0000","last_active":"2024-03-06T07:27:10.272607+0000","last_peered":"2024-03-06T07:27:10.272607+0000","last_clean":"2024-03-06T07:27:10.272607+0000","last_became_active":"2024-03-06T07:27:10.268772+0000","last_became_peered":"2024-03-06T07:27:10.268772+0000","last_unstale":"2024-03-06T07:27:10.272607+0000","last_undegraded":"2024-03-06T07:27:10.272607+0000","last_fullsized":"2024-03-06T07:27:10.272607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:50.615510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268623+0000","last_change":"2024-03-06T07:27:10.268623+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268623+0000","last_undegraded":"2024-03-06T07:27:10.268623+0000","last_fullsized":"2024-03-06T07:27:10.268623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.269106+0000","last_change":"2024-03-06T07:27:10.268260+0000","last_active":"2024-03-06T07:27:10.269106+0000","last_peered":"2024-03-06T07:27:10.269106+0000","last_clean":"2024-03-06T07:27:10.269106+0000","last_became_active":"2024-03-06T07:27:10.267951+0000","last_became_peered":"2024-03-06T07:27:10.267951+0000","last_unstale":"2024-03-06T07:27:10.269106+0000","last_undegraded":"2024-03-06T07:27:10.269106+0000","last_fullsized":"2024-03-06T07:27:10.269106+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:23.498721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266447+0000","last_change":"2024-03-06T07:27:10.266447+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266447+0000","last_undegraded":"2024-03-06T07:27:10.266447+0000","last_fullsized":"2024-03-06T07:27:10.266447+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280949+0000","last_change":"2024-03-06T07:27:10.278526+0000","last_active":"2024-03-06T07:27:10.280949+0000","last_peered":"2024-03-06T07:27:10.280949+0000","last_clean":"2024-03-06T07:27:10.280949+0000","last_became_active":"2024-03-06T07:27:10.277985+0000","last_became_peered":"2024-03-06T07:27:10.277985+0000","last_unstale":"2024-03-06T07:27:10.280949+0000","last_undegraded":"2024-03-06T07:27:10.280949+0000","last_fullsized":"2024-03-06T07:27:10.280949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:23.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264647+0000","last_change":"2024-03-06T07:27:10.264647+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264647+0000","last_undegraded":"2024-03-06T07:27:10.264647+0000","last_fullsized":"2024-03-06T07:27:10.264647+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278297+0000","last_change":"2024-03-06T07:27:10.277574+0000","last_active":"2024-03-06T07:27:10.278297+0000","last_peered":"2024-03-06T07:27:10.278297+0000","last_clean":"2024-03-06T07:27:10.278297+0000","last_became_active":"2024-03-06T07:27:10.277328+0000","last_became_peered":"2024-03-06T07:27:10.277328+0000","last_unstale":"2024-03-06T07:27:10.278297+0000","last_undegraded":"2024-03-06T07:27:10.278297+0000","last_fullsized":"2024-03-06T07:27:10.278297+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:48.143046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264239+0000","last_change":"2024-03-06T07:27:10.264239+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264239+0000","last_undegraded":"2024-03-06T07:27:10.264239+0000","last_fullsized":"2024-03-06T07:27:10.264239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.272084+0000","last_change":"2024-03-06T07:27:10.269028+0000","last_active":"2024-03-06T07:27:10.272084+0000","last_peered":"2024-03-06T07:27:10.272084+0000","last_clean":"2024-03-06T07:27:10.272084+0000","last_became_active":"2024-03-06T07:27:10.268733+0000","last_became_peered":"2024-03-06T07:27:10.268733+0000","last_unstale":"2024-03-06T07:27:10.272084+0000","last_undegraded":"2024-03-06T07:27:10.272084+0000","last_fullsized":"2024-03-06T07:27:10.272084+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:05.178263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263939+0000","last_change":"2024-03-06T07:27:10.263939+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263939+0000","last_undegraded":"2024-03-06T07:27:10.263939+0000","last_fullsized":"2024-03-06T07:27:10.263939+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281719+0000","last_change":"2024-03-06T07:27:10.281030+0000","last_active":"2024-03-06T07:27:10.281719+0000","last_peered":"2024-03-06T07:27:10.281719+0000","last_clean":"2024-03-06T07:27:10.281719+0000","last_became_active":"2024-03-06T07:27:10.280767+0000","last_became_peered":"2024-03-06T07:27:10.280767+0000","last_unstale":"2024-03-06T07:27:10.281719+0000","last_undegraded":"2024-03-06T07:27:10.281719+0000","last_fullsized":"2024-03-06T07:27:10.281719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:58.940827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264009+0000","last_change":"2024-03-06T07:27:10.264009+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264009+0000","last_undegraded":"2024-03-06T07:27:10.264009+0000","last_fullsized":"2024-03-06T07:27:10.264009+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280068+0000","last_change":"2024-03-06T07:27:10.278124+0000","last_active":"2024-03-06T07:27:10.280068+0000","last_peered":"2024-03-06T07:27:10.280068+0000","last_clean":"2024-03-06T07:27:10.280068+0000","last_became_active":"2024-03-06T07:27:10.277686+0000","last_became_peered":"2024-03-06T07:27:10.277686+0000","last_unstale":"2024-03-06T07:27:10.280068+0000","last_undegraded":"2024-03-06T07:27:10.280068+0000","last_fullsized":"2024-03-06T07:27:10.280068+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:55.423671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263293+0000","last_change":"2024-03-06T07:27:10.263293+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263293+0000","last_undegraded":"2024-03-06T07:27:10.263293+0000","last_fullsized":"2024-03-06T07:27:10.263293+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276302+0000","last_change":"2024-03-06T07:27:10.272261+0000","last_active":"2024-03-06T07:27:10.276302+0000","last_peered":"2024-03-06T07:27:10.276302+0000","last_clean":"2024-03-06T07:27:10.276302+0000","last_became_active":"2024-03-06T07:27:10.272009+0000","last_became_peered":"2024-03-06T07:27:10.272009+0000","last_unstale":"2024-03-06T07:27:10.276302+0000","last_undegraded":"2024-03-06T07:27:10.276302+0000","last_fullsized":"2024-03-06T07:27:10.276302+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:13.354906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263736+0000","last_change":"2024-03-06T07:27:10.263736+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263736+0000","last_undegraded":"2024-03-06T07:27:10.263736+0000","last_fullsized":"2024-03-06T07:27:10.263736+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279136+0000","last_change":"2024-03-06T07:27:10.275499+0000","last_active":"2024-03-06T07:27:10.279136+0000","last_peered":"2024-03-06T07:27:10.279136+0000","last_clean":"2024-03-06T07:27:10.279136+0000","last_became_active":"2024-03-06T07:27:10.275217+0000","last_became_peered":"2024-03-06T07:27:10.275217+0000","last_unstale":"2024-03-06T07:27:10.279136+0000","last_undegraded":"2024-03-06T07:27:10.279136+0000","last_fullsized":"2024-03-06T07:27:10.279136+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:24.392230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264215+0000","last_change":"2024-03-06T07:27:10.264215+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264215+0000","last_undegraded":"2024-03-06T07:27:10.264215+0000","last_fullsized":"2024-03-06T07:27:10.264215+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278410+0000","last_change":"2024-03-06T07:27:10.276787+0000","last_active":"2024-03-06T07:27:10.278410+0000","last_peered":"2024-03-06T07:27:10.278410+0000","last_clean":"2024-03-06T07:27:10.278410+0000","last_became_active":"2024-03-06T07:27:10.276489+0000","last_became_peered":"2024-03-06T07:27:10.276489+0000","last_unstale":"2024-03-06T07:27:10.278410+0000","last_undegraded":"2024-03-06T07:27:10.278410+0000","last_fullsized":"2024-03-06T07:27:10.278410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:12.606496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.270279+0000","last_change":"2024-03-06T07:27:10.270279+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.270279+0000","last_undegraded":"2024-03-06T07:27:10.270279+0000","last_fullsized":"2024-03-06T07:27:10.270279+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281716+0000","last_change":"2024-03-06T07:27:10.280931+0000","last_active":"2024-03-06T07:27:10.281716+0000","last_peered":"2024-03-06T07:27:10.281716+0000","last_clean":"2024-03-06T07:27:10.281716+0000","last_became_active":"2024-03-06T07:27:10.280631+0000","last_became_peered":"2024-03-06T07:27:10.280631+0000","last_unstale":"2024-03-06T07:27:10.281716+0000","last_undegraded":"2024-03-06T07:27:10.281716+0000","last_fullsized":"2024-03-06T07:27:10.281716+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:44.772144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265754+0000","last_change":"2024-03-06T07:27:10.265754+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265754+0000","last_undegraded":"2024-03-06T07:27:10.265754+0000","last_fullsized":"2024-03-06T07:27:10.265754+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278330+0000","last_change":"2024-03-06T07:27:10.275684+0000","last_active":"2024-03-06T07:27:10.278330+0000","last_peered":"2024-03-06T07:27:10.278330+0000","last_clean":"2024-03-06T07:27:10.278330+0000","last_became_active":"2024-03-06T07:27:10.275341+0000","last_became_peered":"2024-03-06T07:27:10.275341+0000","last_unstale":"2024-03-06T07:27:10.278330+0000","last_undegraded":"2024-03-06T07:27:10.278330+0000","last_fullsized":"2024-03-06T07:27:10.278330+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:22.219213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269219+0000","last_change":"2024-03-06T07:27:10.269219+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269219+0000","last_undegraded":"2024-03-06T07:27:10.269219+0000","last_fullsized":"2024-03-06T07:27:10.269219+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276412+0000","last_change":"2024-03-06T07:27:10.273231+0000","last_active":"2024-03-06T07:27:10.276412+0000","last_peered":"2024-03-06T07:27:10.276412+0000","last_clean":"2024-03-06T07:27:10.276412+0000","last_became_active":"2024-03-06T07:27:10.272878+0000","last_became_peered":"2024-03-06T07:27:10.272878+0000","last_unstale":"2024-03-06T07:27:10.276412+0000","last_undegraded":"2024-03-06T07:27:10.276412+0000","last_fullsized":"2024-03-06T07:27:10.276412+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:28.515777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269745+0000","last_change":"2024-03-06T07:27:10.269745+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269745+0000","last_undegraded":"2024-03-06T07:27:10.269745+0000","last_fullsized":"2024-03-06T07:27:10.269745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281876+0000","last_change":"2024-03-06T07:27:10.277902+0000","last_active":"2024-03-06T07:27:10.281876+0000","last_peered":"2024-03-06T07:27:10.281876+0000","last_clean":"2024-03-06T07:27:10.281876+0000","last_became_active":"2024-03-06T07:27:10.277622+0000","last_became_peered":"2024-03-06T07:27:10.277622+0000","last_unstale":"2024-03-06T07:27:10.281876+0000","last_undegraded":"2024-03-06T07:27:10.281876+0000","last_fullsized":"2024-03-06T07:27:10.281876+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:14.247605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267319+0000","last_change":"2024-03-06T07:27:10.267319+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267319+0000","last_undegraded":"2024-03-06T07:27:10.267319+0000","last_fullsized":"2024-03-06T07:27:10.267319+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279670+0000","last_change":"2024-03-06T07:27:10.277544+0000","last_active":"2024-03-06T07:27:10.279670+0000","last_peered":"2024-03-06T07:27:10.279670+0000","last_clean":"2024-03-06T07:27:10.279670+0000","last_became_active":"2024-03-06T07:27:10.276523+0000","last_became_peered":"2024-03-06T07:27:10.276523+0000","last_unstale":"2024-03-06T07:27:10.279670+0000","last_undegraded":"2024-03-06T07:27:10.279670+0000","last_fullsized":"2024-03-06T07:27:10.279670+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:25.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269843+0000","last_change":"2024-03-06T07:27:10.269843+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269843+0000","last_undegraded":"2024-03-06T07:27:10.269843+0000","last_fullsized":"2024-03-06T07:27:10.269843+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279775+0000","last_change":"2024-03-06T07:27:10.278830+0000","last_active":"2024-03-06T07:27:10.279775+0000","last_peered":"2024-03-06T07:27:10.279775+0000","last_clean":"2024-03-06T07:27:10.279775+0000","last_became_active":"2024-03-06T07:27:10.278106+0000","last_became_peered":"2024-03-06T07:27:10.278106+0000","last_unstale":"2024-03-06T07:27:10.279775+0000","last_undegraded":"2024-03-06T07:27:10.279775+0000","last_fullsized":"2024-03-06T07:27:10.279775+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.266466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269136+0000","last_change":"2024-03-06T07:27:10.269136+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269136+0000","last_undegraded":"2024-03-06T07:27:10.269136+0000","last_fullsized":"2024-03-06T07:27:10.269136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274650+0000","last_change":"2024-03-06T07:27:10.271560+0000","last_active":"2024-03-06T07:27:10.274650+0000","last_peered":"2024-03-06T07:27:10.274650+0000","last_clean":"2024-03-06T07:27:10.274650+0000","last_became_active":"2024-03-06T07:27:10.271296+0000","last_became_peered":"2024-03-06T07:27:10.271296+0000","last_unstale":"2024-03-06T07:27:10.274650+0000","last_undegraded":"2024-03-06T07:27:10.274650+0000","last_fullsized":"2024-03-06T07:27:10.274650+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:00.541674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263530+0000","last_change":"2024-03-06T07:27:10.263530+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263530+0000","last_undegraded":"2024-03-06T07:27:10.263530+0000","last_fullsized":"2024-03-06T07:27:10.263530+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281370+0000","last_change":"2024-03-06T07:27:10.279864+0000","last_active":"2024-03-06T07:27:10.281370+0000","last_peered":"2024-03-06T07:27:10.281370+0000","last_clean":"2024-03-06T07:27:10.281370+0000","last_became_active":"2024-03-06T07:27:10.279622+0000","last_became_peered":"2024-03-06T07:27:10.279622+0000","last_unstale":"2024-03-06T07:27:10.281370+0000","last_undegraded":"2024-03-06T07:27:10.281370+0000","last_fullsized":"2024-03-06T07:27:10.281370+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:33.259974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264152+0000","last_change":"2024-03-06T07:27:10.264152+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264152+0000","last_undegraded":"2024-03-06T07:27:10.264152+0000","last_fullsized":"2024-03-06T07:27:10.264152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276867+0000","last_change":"2024-03-06T07:27:10.273788+0000","last_active":"2024-03-06T07:27:10.276867+0000","last_peered":"2024-03-06T07:27:10.276867+0000","last_clean":"2024-03-06T07:27:10.276867+0000","last_became_active":"2024-03-06T07:27:10.273542+0000","last_became_peered":"2024-03-06T07:27:10.273542+0000","last_unstale":"2024-03-06T07:27:10.276867+0000","last_undegraded":"2024-03-06T07:27:10.276867+0000","last_fullsized":"2024-03-06T07:27:10.276867+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:10.176108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266917+0000","last_change":"2024-03-06T07:27:10.266917+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266917+0000","last_undegraded":"2024-03-06T07:27:10.266917+0000","last_fullsized":"2024-03-06T07:27:10.266917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.270719+0000","last_change":"2024-03-06T07:27:10.269615+0000","last_active":"2024-03-06T07:27:10.270719+0000","last_peered":"2024-03-06T07:27:10.270719+0000","last_clean":"2024-03-06T07:27:10.270719+0000","last_became_active":"2024-03-06T07:27:10.269331+0000","last_became_peered":"2024-03-06T07:27:10.269331+0000","last_unstale":"2024-03-06T07:27:10.270719+0000","last_undegraded":"2024-03-06T07:27:10.270719+0000","last_fullsized":"2024-03-06T07:27:10.270719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:31.221037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265082+0000","last_change":"2024-03-06T07:27:10.265082+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265082+0000","last_undegraded":"2024-03-06T07:27:10.265082+0000","last_fullsized":"2024-03-06T07:27:10.265082+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274598+0000","last_change":"2024-03-06T07:27:10.273864+0000","last_active":"2024-03-06T07:27:10.274598+0000","last_peered":"2024-03-06T07:27:10.274598+0000","last_clean":"2024-03-06T07:27:10.274598+0000","last_became_active":"2024-03-06T07:27:10.273611+0000","last_became_peered":"2024-03-06T07:27:10.273611+0000","last_unstale":"2024-03-06T07:27:10.274598+0000","last_undegraded":"2024-03-06T07:27:10.274598+0000","last_fullsized":"2024-03-06T07:27:10.274598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:49.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263562+0000","last_change":"2024-03-06T07:27:10.263562+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263562+0000","last_undegraded":"2024-03-06T07:27:10.263562+0000","last_fullsized":"2024-03-06T07:27:10.263562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276271+0000","last_change":"2024-03-06T07:27:10.271689+0000","last_active":"2024-03-06T07:27:10.276271+0000","last_peered":"2024-03-06T07:27:10.276271+0000","last_clean":"2024-03-06T07:27:10.276271+0000","last_became_active":"2024-03-06T07:27:10.271421+0000","last_became_peered":"2024-03-06T07:27:10.271421+0000","last_unstale":"2024-03-06T07:27:10.276271+0000","last_undegraded":"2024-03-06T07:27:10.276271+0000","last_fullsized":"2024-03-06T07:27:10.276271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:20.049932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261430+0000","last_change":"2024-03-06T07:27:10.261430+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261430+0000","last_undegraded":"2024-03-06T07:27:10.261430+0000","last_fullsized":"2024-03-06T07:27:10.261430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.269674+0000","last_change":"2024-03-06T07:27:10.268891+0000","last_active":"2024-03-06T07:27:10.269674+0000","last_peered":"2024-03-06T07:27:10.269674+0000","last_clean":"2024-03-06T07:27:10.269674+0000","last_became_active":"2024-03-06T07:27:10.268536+0000","last_became_peered":"2024-03-06T07:27:10.268536+0000","last_unstale":"2024-03-06T07:27:10.269674+0000","last_undegraded":"2024-03-06T07:27:10.269674+0000","last_fullsized":"2024-03-06T07:27:10.269674+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:51.960409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263082+0000","last_change":"2024-03-06T07:27:10.263082+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263082+0000","last_undegraded":"2024-03-06T07:27:10.263082+0000","last_fullsized":"2024-03-06T07:27:10.263082+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276122+0000","last_change":"2024-03-06T07:27:10.272914+0000","last_active":"2024-03-06T07:27:10.276122+0000","last_peered":"2024-03-06T07:27:10.276122+0000","last_clean":"2024-03-06T07:27:10.276122+0000","last_became_active":"2024-03-06T07:27:10.272635+0000","last_became_peered":"2024-03-06T07:27:10.272635+0000","last_unstale":"2024-03-06T07:27:10.276122+0000","last_undegraded":"2024-03-06T07:27:10.276122+0000","last_fullsized":"2024-03-06T07:27:10.276122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:58.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268710+0000","last_change":"2024-03-06T07:27:10.268710+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268710+0000","last_undegraded":"2024-03-06T07:27:10.268710+0000","last_fullsized":"2024-03-06T07:27:10.268710+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278239+0000","last_change":"2024-03-06T07:27:10.276957+0000","last_active":"2024-03-06T07:27:10.278239+0000","last_peered":"2024-03-06T07:27:10.278239+0000","last_clean":"2024-03-06T07:27:10.278239+0000","last_became_active":"2024-03-06T07:27:10.276707+0000","last_became_peered":"2024-03-06T07:27:10.276707+0000","last_unstale":"2024-03-06T07:27:10.278239+0000","last_undegraded":"2024-03-06T07:27:10.278239+0000","last_fullsized":"2024-03-06T07:27:10.278239+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:31.396430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264427+0000","last_change":"2024-03-06T07:27:10.264427+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264427+0000","last_undegraded":"2024-03-06T07:27:10.264427+0000","last_fullsized":"2024-03-06T07:27:10.264427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276211+0000","last_change":"2024-03-06T07:27:10.275390+0000","last_active":"2024-03-06T07:27:10.276211+0000","last_peered":"2024-03-06T07:27:10.276211+0000","last_clean":"2024-03-06T07:27:10.276211+0000","last_became_active":"2024-03-06T07:27:10.274903+0000","last_became_peered":"2024-03-06T07:27:10.274903+0000","last_unstale":"2024-03-06T07:27:10.276211+0000","last_undegraded":"2024-03-06T07:27:10.276211+0000","last_fullsized":"2024-03-06T07:27:10.276211+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:02.332832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263304+0000","last_change":"2024-03-06T07:27:10.263304+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263304+0000","last_undegraded":"2024-03-06T07:27:10.263304+0000","last_fullsized":"2024-03-06T07:27:10.263304+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281890+0000","last_change":"2024-03-06T07:27:10.277891+0000","last_active":"2024-03-06T07:27:10.281890+0000","last_peered":"2024-03-06T07:27:10.281890+0000","last_clean":"2024-03-06T07:27:10.281890+0000","last_became_active":"2024-03-06T07:27:10.277582+0000","last_became_peered":"2024-03-06T07:27:10.277582+0000","last_unstale":"2024-03-06T07:27:10.281890+0000","last_undegraded":"2024-03-06T07:27:10.281890+0000","last_fullsized":"2024-03-06T07:27:10.281890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:16.855383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":332,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274060+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:27:10.274060+0000","last_peered":"2024-03-06T07:27:10.274060+0000","last_clean":"2024-03-06T07:27:10.274060+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:27:10.274060+0000","last_undegraded":"2024-03-06T07:27:10.274060+0000","last_fullsized":"2024-03-06T07:27:10.274060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":1156184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40776,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331064,"statfs":{"total":96636764160,"available":96595009536,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40792,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331048,"statfs":{"total":96636764160,"available":96594993152,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41356,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330484,"statfs":{"total":96636764160,"available":96594415616,"internally_reserved":0,"allocated":1900544,"data_stored":1149909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.626 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.764+0000 7f5c1820e640 1 Processor -- start 2024-03-06T07:27:14.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.765+0000 7f5c1820e640 1 -- start start 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c1820e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5c100d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c1820e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100d18d0 con 0x7f5c1015ddf0 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c1820e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100d1bb0 con 0x7f5c100cfb50 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c1820e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100d27c0 con 0x7f5c1015e1d0 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c15faa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5c1015ddf0 0x7f5c100cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49962/0 (socket says 172.21.15.122:49962) 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c15faa640 1 -- 172.21.15.122:0/1231565 learned_addr learned my addr 172.21.15.122:0/1231565 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c157a9640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5c100d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097494313 0 0) 0x7f5c100d18d0 con 0x7f5c1015ddf0 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bfc003520 con 0x7f5c1015ddf0 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286265557 0 0) 0x7f5bfc003520 con 0x7f5c1015ddf0 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 msgr2=0x7f5c100d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5c100d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 >> v1:172.21.15.169:6789/0 conn(0x7f5c100cfb50 legacy=0x7f5c100b6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100d1e90 con 0x7f5c1015ddf0 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f5c00001670 con 0x7f5c1015ddf0 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.766+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 >> v1:172.21.15.122:6789/0 conn(0x7f5c1015ddf0 legacy=0x7f5c100cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c14fa8640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5bfc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c14fa8640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bfc003f40 0x7f5bfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c14fa8640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bfc004d40 0x7f5bfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100d27c0 con 0x7f5bfc003f40 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc00ba70 con 0x7f5bfc004d40 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc00bc40 con 0x7f5c1015e1d0 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c157a9640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bfc003f40 0x7f5bfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c15faa640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5bfc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c167ab640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bfc004d40 0x7f5bfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c157a9640 1 -- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 msgr2=0x7f5bfc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c157a9640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5bfc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c157a9640 1 -- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bfc004d40 msgr2=0x7f5bfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c157a9640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bfc004d40 0x7f5bfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c157a9640 1 -- 172.21.15.122:0/1231565 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc003520 con 0x7f5bfc003f40 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c15faa640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5c1015e1d0 0x7f5bfc0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.767+0000 7f5c167ab640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bfc004d40 0x7f5bfc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c157a9640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bfc003f40 0x7f5bfc004800 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009ac0 tx=0x7f5bfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c336b1276d1548 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c039070 con 0x7f5bfc003f40 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c02c680 con 0x7f5bfc003f40 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c14fa8640 1 -- 172.21.15.122:0/1231565 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c009690 con 0x7f5bfc003f40 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c1820e640 1 -- 172.21.15.122:0/1231565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bfc003f40 msgr2=0x7f5bfc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c1820e640 1 --2- 172.21.15.122:0/1231565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bfc003f40 0x7f5bfc004800 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009ac0 tx=0x7f5bfc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c167ab640 1 -- 172.21.15.122:0/1231565 reap_dead start 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c1820e640 1 -- 172.21.15.122:0/1231565 shutdown_connections 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c1820e640 1 -- 172.21.15.122:0/1231565 >> 172.21.15.122:0/1231565 conn(0x7f5c1009cff0 msgr2=0x7f5c100d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c1820e640 1 -- 172.21.15.122:0/1231565 shutdown_connections 2024-03-06T07:27:14.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.768+0000 7f5c1820e640 1 -- 172.21.15.122:0/1231565 wait complete. 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 Processor -- start 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 -- start start 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bfc003f40 0x7f5c100cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 0x7f5c100c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c10154e50 0x7f5c101552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100d2c50 con 0x7f5c100ca350 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100d2450 con 0x7f5c10154e50 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c1820e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100b7260 con 0x7f5bfc003f40 2024-03-06T07:27:14.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c157a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 0x7f5c100c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c157a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 0x7f5c100c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51826/0 (socket says 172.21.15.122:51826) 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.770+0000 7f5c157a9640 1 -- 172.21.15.122:0/2040058535 learned_addr learned my addr 172.21.15.122:0/2040058535 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c15faa640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bfc003f40 0x7f5c100cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c167ab640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c10154e50 0x7f5c101552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c157a9640 1 -- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bfc003f40 msgr2=0x7f5c100cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c157a9640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bfc003f40 0x7f5c100cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c157a9640 1 -- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c10154e50 msgr2=0x7f5c101552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c157a9640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c10154e50 0x7f5c101552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c157a9640 1 -- 172.21.15.122:0/2040058535 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100cb620 con 0x7f5c100ca350 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c157a9640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 0x7f5c100c98f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c000c00 tx=0x7f5c0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c042020 con 0x7f5c100ca350 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c03a040 con 0x7f5c100ca350 2024-03-06T07:27:14.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c039070 con 0x7f5c100ca350 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c100cb900 con 0x7f5c100ca350 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.771+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c100cb310 con 0x7f5c100ca350 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.772+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c10155810 con 0x7f5c100ca350 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.773+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c0c045020 con 0x7f5c100ca350 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.773+0000 7f5c067fc640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5bf406b4d0 0x7f5bf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.773+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c0c0aba00 con 0x7f5c100ca350 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.773+0000 7f5c15faa640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5bf406b4d0 0x7f5bf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.774+0000 7f5c15faa640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5bf406b4d0 0x7f5bf406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000f60 tx=0x7f5c00000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.777+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c0c04c030 con 0x7f5c100ca350 2024-03-06T07:27:14.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:14.996+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c1004f330 con 0x7f5bf406b4d0 2024-03-06T07:27:15.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.002+0000 7f5c067fc640 1 -- 172.21.15.122:0/2040058535 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247087 (secure 0 0 0) 0x7f5c1004f330 con 0x7f5bf406b4d0 2024-03-06T07:27:15.006 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:15.007 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5bf406b4d0 msgr2=0x7f5bf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5bf406b4d0 0x7f5bf406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000f60 tx=0x7f5c00000ed0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 msgr2=0x7f5c100c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 0x7f5c100c98f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c000c00 tx=0x7f5c0c004730 comp rx=0 tx=0).stop 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 shutdown_connections 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5bf406b4d0 0x7f5bf406d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5c10154e50 0x7f5c101552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c100ca350 0x7f5c100c98f0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 --2- 172.21.15.122:0/2040058535 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5bfc003f40 0x7f5c100cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 >> 172.21.15.122:0/2040058535 conn(0x7f5c1009cff0 msgr2=0x7f5c101618a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 shutdown_connections 2024-03-06T07:27:15.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.005+0000 7f5c1820e640 1 -- 172.21.15.122:0/2040058535 wait complete. 2024-03-06T07:27:15.021 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:13.396067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323280,"kb_used_data":11380,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651440,"statfs":{"total":773094113280,"available":772763074560,"internally_reserved":0,"allocated":11653120,"data_stored":5663339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.140367"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267598+0000","last_change":"2024-03-06T07:27:10.267598+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267598+0000","last_undegraded":"2024-03-06T07:27:10.267598+0000","last_fullsized":"2024-03-06T07:27:10.267598+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261734+0000","last_change":"2024-03-06T07:27:10.261734+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261734+0000","last_undegraded":"2024-03-06T07:27:10.261734+0000","last_fullsized":"2024-03-06T07:27:10.261734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273806+0000","last_change":"2024-03-06T07:27:10.273806+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273806+0000","last_undegraded":"2024-03-06T07:27:10.273806+0000","last_fullsized":"2024-03-06T07:27:10.273806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265833+0000","last_change":"2024-03-06T07:27:10.265833+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265833+0000","last_undegraded":"2024-03-06T07:27:10.265833+0000","last_fullsized":"2024-03-06T07:27:10.265833+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266357+0000","last_change":"2024-03-06T07:27:10.266357+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266357+0000","last_undegraded":"2024-03-06T07:27:10.266357+0000","last_fullsized":"2024-03-06T07:27:10.266357+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265033+0000","last_change":"2024-03-06T07:27:10.265033+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265033+0000","last_undegraded":"2024-03-06T07:27:10.265033+0000","last_fullsized":"2024-03-06T07:27:10.265033+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265851+0000","last_change":"2024-03-06T07:27:10.265851+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265851+0000","last_undegraded":"2024-03-06T07:27:10.265851+0000","last_fullsized":"2024-03-06T07:27:10.265851+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269538+0000","last_change":"2024-03-06T07:27:10.269538+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269538+0000","last_undegraded":"2024-03-06T07:27:10.269538+0000","last_fullsized":"2024-03-06T07:27:10.269538+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267258+0000","last_change":"2024-03-06T07:27:10.267258+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267258+0000","last_undegraded":"2024-03-06T07:27:10.267258+0000","last_fullsized":"2024-03-06T07:27:10.267258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273029+0000","last_change":"2024-03-06T07:27:10.273029+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273029+0000","last_undegraded":"2024-03-06T07:27:10.273029+0000","last_fullsized":"2024-03-06T07:27:10.273029+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261449+0000","last_change":"2024-03-06T07:27:10.261449+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261449+0000","last_undegraded":"2024-03-06T07:27:10.261449+0000","last_fullsized":"2024-03-06T07:27:10.261449+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265454+0000","last_change":"2024-03-06T07:27:10.265454+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265454+0000","last_undegraded":"2024-03-06T07:27:10.265454+0000","last_fullsized":"2024-03-06T07:27:10.265454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268184+0000","last_change":"2024-03-06T07:27:10.268184+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268184+0000","last_undegraded":"2024-03-06T07:27:10.268184+0000","last_fullsized":"2024-03-06T07:27:10.268184+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268837+0000","last_change":"2024-03-06T07:27:10.268837+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268837+0000","last_undegraded":"2024-03-06T07:27:10.268837+0000","last_fullsized":"2024-03-06T07:27:10.268837+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264950+0000","last_change":"2024-03-06T07:27:10.264950+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264950+0000","last_undegraded":"2024-03-06T07:27:10.264950+0000","last_fullsized":"2024-03-06T07:27:10.264950+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268981+0000","last_change":"2024-03-06T07:27:10.268981+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268981+0000","last_undegraded":"2024-03-06T07:27:10.268981+0000","last_fullsized":"2024-03-06T07:27:10.268981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.262813+0000","last_change":"2024-03-06T07:27:10.262813+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.262813+0000","last_undegraded":"2024-03-06T07:27:10.262813+0000","last_fullsized":"2024-03-06T07:27:10.262813+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264573+0000","last_change":"2024-03-06T07:27:10.264573+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264573+0000","last_undegraded":"2024-03-06T07:27:10.264573+0000","last_fullsized":"2024-03-06T07:27:10.264573+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.273199+0000","last_change":"2024-03-06T07:27:10.273199+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.273199+0000","last_undegraded":"2024-03-06T07:27:10.273199+0000","last_fullsized":"2024-03-06T07:27:10.273199+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264185+0000","last_change":"2024-03-06T07:27:10.264185+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264185+0000","last_undegraded":"2024-03-06T07:27:10.264185+0000","last_fullsized":"2024-03-06T07:27:10.264185+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261404+0000","last_change":"2024-03-06T07:27:10.261404+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261404+0000","last_undegraded":"2024-03-06T07:27:10.261404+0000","last_fullsized":"2024-03-06T07:27:10.261404+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268216+0000","last_change":"2024-03-06T07:27:10.268216+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268216+0000","last_undegraded":"2024-03-06T07:27:10.268216+0000","last_fullsized":"2024-03-06T07:27:10.268216+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261822+0000","last_change":"2024-03-06T07:27:10.261822+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261822+0000","last_undegraded":"2024-03-06T07:27:10.261822+0000","last_fullsized":"2024-03-06T07:27:10.261822+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265508+0000","last_change":"2024-03-06T07:27:10.265508+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265508+0000","last_undegraded":"2024-03-06T07:27:10.265508+0000","last_fullsized":"2024-03-06T07:27:10.265508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.262335+0000","last_change":"2024-03-06T07:27:10.262335+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.262335+0000","last_undegraded":"2024-03-06T07:27:10.262335+0000","last_fullsized":"2024-03-06T07:27:10.262335+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.272656+0000","last_change":"2024-03-06T07:27:10.272656+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.272656+0000","last_undegraded":"2024-03-06T07:27:10.272656+0000","last_fullsized":"2024-03-06T07:27:10.272656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265513+0000","last_change":"2024-03-06T07:27:10.265513+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265513+0000","last_undegraded":"2024-03-06T07:27:10.265513+0000","last_fullsized":"2024-03-06T07:27:10.265513+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264450+0000","last_change":"2024-03-06T07:27:10.264450+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264450+0000","last_undegraded":"2024-03-06T07:27:10.264450+0000","last_fullsized":"2024-03-06T07:27:10.264450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264152+0000","last_change":"2024-03-06T07:27:10.264152+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264152+0000","last_undegraded":"2024-03-06T07:27:10.264152+0000","last_fullsized":"2024-03-06T07:27:10.264152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265454+0000","last_change":"2024-03-06T07:27:10.265454+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265454+0000","last_undegraded":"2024-03-06T07:27:10.265454+0000","last_fullsized":"2024-03-06T07:27:10.265454+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265052+0000","last_change":"2024-03-06T07:27:10.265052+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265052+0000","last_undegraded":"2024-03-06T07:27:10.265052+0000","last_fullsized":"2024-03-06T07:27:10.265052+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267742+0000","last_change":"2024-03-06T07:27:10.267742+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267742+0000","last_undegraded":"2024-03-06T07:27:10.267742+0000","last_fullsized":"2024-03-06T07:27:10.267742+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265433+0000","last_change":"2024-03-06T07:27:10.265433+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265433+0000","last_undegraded":"2024-03-06T07:27:10.265433+0000","last_fullsized":"2024-03-06T07:27:10.265433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276306+0000","last_change":"2024-03-06T07:27:10.271295+0000","last_active":"2024-03-06T07:27:10.276306+0000","last_peered":"2024-03-06T07:27:10.276306+0000","last_clean":"2024-03-06T07:27:10.276306+0000","last_became_active":"2024-03-06T07:27:10.270894+0000","last_became_peered":"2024-03-06T07:27:10.270894+0000","last_unstale":"2024-03-06T07:27:10.276306+0000","last_undegraded":"2024-03-06T07:27:10.276306+0000","last_fullsized":"2024-03-06T07:27:10.276306+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:47.748004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265204+0000","last_change":"2024-03-06T07:27:10.265204+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265204+0000","last_undegraded":"2024-03-06T07:27:10.265204+0000","last_fullsized":"2024-03-06T07:27:10.265204+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280189+0000","last_change":"2024-03-06T07:27:10.279732+0000","last_active":"2024-03-06T07:27:10.280189+0000","last_peered":"2024-03-06T07:27:10.280189+0000","last_clean":"2024-03-06T07:27:10.280189+0000","last_became_active":"2024-03-06T07:27:10.279431+0000","last_became_peered":"2024-03-06T07:27:10.279431+0000","last_unstale":"2024-03-06T07:27:10.280189+0000","last_undegraded":"2024-03-06T07:27:10.280189+0000","last_fullsized":"2024-03-06T07:27:10.280189+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:10.859022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266552+0000","last_change":"2024-03-06T07:27:10.266552+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266552+0000","last_undegraded":"2024-03-06T07:27:10.266552+0000","last_fullsized":"2024-03-06T07:27:10.266552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279750+0000","last_change":"2024-03-06T07:27:10.278840+0000","last_active":"2024-03-06T07:27:10.279750+0000","last_peered":"2024-03-06T07:27:10.279750+0000","last_clean":"2024-03-06T07:27:10.279750+0000","last_became_active":"2024-03-06T07:27:10.278584+0000","last_became_peered":"2024-03-06T07:27:10.278584+0000","last_unstale":"2024-03-06T07:27:10.279750+0000","last_undegraded":"2024-03-06T07:27:10.279750+0000","last_fullsized":"2024-03-06T07:27:10.279750+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:12.045311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.271902+0000","last_change":"2024-03-06T07:27:10.271902+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.271902+0000","last_undegraded":"2024-03-06T07:27:10.271902+0000","last_fullsized":"2024-03-06T07:27:10.271902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279717+0000","last_change":"2024-03-06T07:27:10.278593+0000","last_active":"2024-03-06T07:27:10.279717+0000","last_peered":"2024-03-06T07:27:10.279717+0000","last_clean":"2024-03-06T07:27:10.279717+0000","last_became_active":"2024-03-06T07:27:10.278332+0000","last_became_peered":"2024-03-06T07:27:10.278332+0000","last_unstale":"2024-03-06T07:27:10.279717+0000","last_undegraded":"2024-03-06T07:27:10.279717+0000","last_fullsized":"2024-03-06T07:27:10.279717+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:59.810451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264549+0000","last_change":"2024-03-06T07:27:10.264549+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264549+0000","last_undegraded":"2024-03-06T07:27:10.264549+0000","last_fullsized":"2024-03-06T07:27:10.264549+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276275+0000","last_change":"2024-03-06T07:27:10.271671+0000","last_active":"2024-03-06T07:27:10.276275+0000","last_peered":"2024-03-06T07:27:10.276275+0000","last_clean":"2024-03-06T07:27:10.276275+0000","last_became_active":"2024-03-06T07:27:10.271376+0000","last_became_peered":"2024-03-06T07:27:10.271376+0000","last_unstale":"2024-03-06T07:27:10.276275+0000","last_undegraded":"2024-03-06T07:27:10.276275+0000","last_fullsized":"2024-03-06T07:27:10.276275+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:25.067873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263608+0000","last_change":"2024-03-06T07:27:10.263608+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263608+0000","last_undegraded":"2024-03-06T07:27:10.263608+0000","last_fullsized":"2024-03-06T07:27:10.263608+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.272607+0000","last_change":"2024-03-06T07:27:10.269066+0000","last_active":"2024-03-06T07:27:10.272607+0000","last_peered":"2024-03-06T07:27:10.272607+0000","last_clean":"2024-03-06T07:27:10.272607+0000","last_became_active":"2024-03-06T07:27:10.268772+0000","last_became_peered":"2024-03-06T07:27:10.268772+0000","last_unstale":"2024-03-06T07:27:10.272607+0000","last_undegraded":"2024-03-06T07:27:10.272607+0000","last_fullsized":"2024-03-06T07:27:10.272607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:50.615510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268623+0000","last_change":"2024-03-06T07:27:10.268623+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268623+0000","last_undegraded":"2024-03-06T07:27:10.268623+0000","last_fullsized":"2024-03-06T07:27:10.268623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.269106+0000","last_change":"2024-03-06T07:27:10.268260+0000","last_active":"2024-03-06T07:27:10.269106+0000","last_peered":"2024-03-06T07:27:10.269106+0000","last_clean":"2024-03-06T07:27:10.269106+0000","last_became_active":"2024-03-06T07:27:10.267951+0000","last_became_peered":"2024-03-06T07:27:10.267951+0000","last_unstale":"2024-03-06T07:27:10.269106+0000","last_undegraded":"2024-03-06T07:27:10.269106+0000","last_fullsized":"2024-03-06T07:27:10.269106+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:23.498721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266447+0000","last_change":"2024-03-06T07:27:10.266447+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266447+0000","last_undegraded":"2024-03-06T07:27:10.266447+0000","last_fullsized":"2024-03-06T07:27:10.266447+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280949+0000","last_change":"2024-03-06T07:27:10.278526+0000","last_active":"2024-03-06T07:27:10.280949+0000","last_peered":"2024-03-06T07:27:10.280949+0000","last_clean":"2024-03-06T07:27:10.280949+0000","last_became_active":"2024-03-06T07:27:10.277985+0000","last_became_peered":"2024-03-06T07:27:10.277985+0000","last_unstale":"2024-03-06T07:27:10.280949+0000","last_undegraded":"2024-03-06T07:27:10.280949+0000","last_fullsized":"2024-03-06T07:27:10.280949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:23.505985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264647+0000","last_change":"2024-03-06T07:27:10.264647+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264647+0000","last_undegraded":"2024-03-06T07:27:10.264647+0000","last_fullsized":"2024-03-06T07:27:10.264647+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278297+0000","last_change":"2024-03-06T07:27:10.277574+0000","last_active":"2024-03-06T07:27:10.278297+0000","last_peered":"2024-03-06T07:27:10.278297+0000","last_clean":"2024-03-06T07:27:10.278297+0000","last_became_active":"2024-03-06T07:27:10.277328+0000","last_became_peered":"2024-03-06T07:27:10.277328+0000","last_unstale":"2024-03-06T07:27:10.278297+0000","last_undegraded":"2024-03-06T07:27:10.278297+0000","last_fullsized":"2024-03-06T07:27:10.278297+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:48.143046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264239+0000","last_change":"2024-03-06T07:27:10.264239+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264239+0000","last_undegraded":"2024-03-06T07:27:10.264239+0000","last_fullsized":"2024-03-06T07:27:10.264239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.272084+0000","last_change":"2024-03-06T07:27:10.269028+0000","last_active":"2024-03-06T07:27:10.272084+0000","last_peered":"2024-03-06T07:27:10.272084+0000","last_clean":"2024-03-06T07:27:10.272084+0000","last_became_active":"2024-03-06T07:27:10.268733+0000","last_became_peered":"2024-03-06T07:27:10.268733+0000","last_unstale":"2024-03-06T07:27:10.272084+0000","last_undegraded":"2024-03-06T07:27:10.272084+0000","last_fullsized":"2024-03-06T07:27:10.272084+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:05.178263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263939+0000","last_change":"2024-03-06T07:27:10.263939+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263939+0000","last_undegraded":"2024-03-06T07:27:10.263939+0000","last_fullsized":"2024-03-06T07:27:10.263939+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281719+0000","last_change":"2024-03-06T07:27:10.281030+0000","last_active":"2024-03-06T07:27:10.281719+0000","last_peered":"2024-03-06T07:27:10.281719+0000","last_clean":"2024-03-06T07:27:10.281719+0000","last_became_active":"2024-03-06T07:27:10.280767+0000","last_became_peered":"2024-03-06T07:27:10.280767+0000","last_unstale":"2024-03-06T07:27:10.281719+0000","last_undegraded":"2024-03-06T07:27:10.281719+0000","last_fullsized":"2024-03-06T07:27:10.281719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:58.940827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264009+0000","last_change":"2024-03-06T07:27:10.264009+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264009+0000","last_undegraded":"2024-03-06T07:27:10.264009+0000","last_fullsized":"2024-03-06T07:27:10.264009+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.280068+0000","last_change":"2024-03-06T07:27:10.278124+0000","last_active":"2024-03-06T07:27:10.280068+0000","last_peered":"2024-03-06T07:27:10.280068+0000","last_clean":"2024-03-06T07:27:10.280068+0000","last_became_active":"2024-03-06T07:27:10.277686+0000","last_became_peered":"2024-03-06T07:27:10.277686+0000","last_unstale":"2024-03-06T07:27:10.280068+0000","last_undegraded":"2024-03-06T07:27:10.280068+0000","last_fullsized":"2024-03-06T07:27:10.280068+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:55.423671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263293+0000","last_change":"2024-03-06T07:27:10.263293+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263293+0000","last_undegraded":"2024-03-06T07:27:10.263293+0000","last_fullsized":"2024-03-06T07:27:10.263293+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276302+0000","last_change":"2024-03-06T07:27:10.272261+0000","last_active":"2024-03-06T07:27:10.276302+0000","last_peered":"2024-03-06T07:27:10.276302+0000","last_clean":"2024-03-06T07:27:10.276302+0000","last_became_active":"2024-03-06T07:27:10.272009+0000","last_became_peered":"2024-03-06T07:27:10.272009+0000","last_unstale":"2024-03-06T07:27:10.276302+0000","last_undegraded":"2024-03-06T07:27:10.276302+0000","last_fullsized":"2024-03-06T07:27:10.276302+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:13.354906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263736+0000","last_change":"2024-03-06T07:27:10.263736+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263736+0000","last_undegraded":"2024-03-06T07:27:10.263736+0000","last_fullsized":"2024-03-06T07:27:10.263736+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279136+0000","last_change":"2024-03-06T07:27:10.275499+0000","last_active":"2024-03-06T07:27:10.279136+0000","last_peered":"2024-03-06T07:27:10.279136+0000","last_clean":"2024-03-06T07:27:10.279136+0000","last_became_active":"2024-03-06T07:27:10.275217+0000","last_became_peered":"2024-03-06T07:27:10.275217+0000","last_unstale":"2024-03-06T07:27:10.279136+0000","last_undegraded":"2024-03-06T07:27:10.279136+0000","last_fullsized":"2024-03-06T07:27:10.279136+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:24.392230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264215+0000","last_change":"2024-03-06T07:27:10.264215+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264215+0000","last_undegraded":"2024-03-06T07:27:10.264215+0000","last_fullsized":"2024-03-06T07:27:10.264215+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278410+0000","last_change":"2024-03-06T07:27:10.276787+0000","last_active":"2024-03-06T07:27:10.278410+0000","last_peered":"2024-03-06T07:27:10.278410+0000","last_clean":"2024-03-06T07:27:10.278410+0000","last_became_active":"2024-03-06T07:27:10.276489+0000","last_became_peered":"2024-03-06T07:27:10.276489+0000","last_unstale":"2024-03-06T07:27:10.278410+0000","last_undegraded":"2024-03-06T07:27:10.278410+0000","last_fullsized":"2024-03-06T07:27:10.278410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:12.606496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.270279+0000","last_change":"2024-03-06T07:27:10.270279+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.270279+0000","last_undegraded":"2024-03-06T07:27:10.270279+0000","last_fullsized":"2024-03-06T07:27:10.270279+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281716+0000","last_change":"2024-03-06T07:27:10.280931+0000","last_active":"2024-03-06T07:27:10.281716+0000","last_peered":"2024-03-06T07:27:10.281716+0000","last_clean":"2024-03-06T07:27:10.281716+0000","last_became_active":"2024-03-06T07:27:10.280631+0000","last_became_peered":"2024-03-06T07:27:10.280631+0000","last_unstale":"2024-03-06T07:27:10.281716+0000","last_undegraded":"2024-03-06T07:27:10.281716+0000","last_fullsized":"2024-03-06T07:27:10.281716+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:44.772144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265754+0000","last_change":"2024-03-06T07:27:10.265754+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265754+0000","last_undegraded":"2024-03-06T07:27:10.265754+0000","last_fullsized":"2024-03-06T07:27:10.265754+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278330+0000","last_change":"2024-03-06T07:27:10.275684+0000","last_active":"2024-03-06T07:27:10.278330+0000","last_peered":"2024-03-06T07:27:10.278330+0000","last_clean":"2024-03-06T07:27:10.278330+0000","last_became_active":"2024-03-06T07:27:10.275341+0000","last_became_peered":"2024-03-06T07:27:10.275341+0000","last_unstale":"2024-03-06T07:27:10.278330+0000","last_undegraded":"2024-03-06T07:27:10.278330+0000","last_fullsized":"2024-03-06T07:27:10.278330+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:22.219213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269219+0000","last_change":"2024-03-06T07:27:10.269219+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269219+0000","last_undegraded":"2024-03-06T07:27:10.269219+0000","last_fullsized":"2024-03-06T07:27:10.269219+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276412+0000","last_change":"2024-03-06T07:27:10.273231+0000","last_active":"2024-03-06T07:27:10.276412+0000","last_peered":"2024-03-06T07:27:10.276412+0000","last_clean":"2024-03-06T07:27:10.276412+0000","last_became_active":"2024-03-06T07:27:10.272878+0000","last_became_peered":"2024-03-06T07:27:10.272878+0000","last_unstale":"2024-03-06T07:27:10.276412+0000","last_undegraded":"2024-03-06T07:27:10.276412+0000","last_fullsized":"2024-03-06T07:27:10.276412+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:28.515777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269745+0000","last_change":"2024-03-06T07:27:10.269745+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269745+0000","last_undegraded":"2024-03-06T07:27:10.269745+0000","last_fullsized":"2024-03-06T07:27:10.269745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281876+0000","last_change":"2024-03-06T07:27:10.277902+0000","last_active":"2024-03-06T07:27:10.281876+0000","last_peered":"2024-03-06T07:27:10.281876+0000","last_clean":"2024-03-06T07:27:10.281876+0000","last_became_active":"2024-03-06T07:27:10.277622+0000","last_became_peered":"2024-03-06T07:27:10.277622+0000","last_unstale":"2024-03-06T07:27:10.281876+0000","last_undegraded":"2024-03-06T07:27:10.281876+0000","last_fullsized":"2024-03-06T07:27:10.281876+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:14.247605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.267319+0000","last_change":"2024-03-06T07:27:10.267319+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.267319+0000","last_undegraded":"2024-03-06T07:27:10.267319+0000","last_fullsized":"2024-03-06T07:27:10.267319+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279670+0000","last_change":"2024-03-06T07:27:10.277544+0000","last_active":"2024-03-06T07:27:10.279670+0000","last_peered":"2024-03-06T07:27:10.279670+0000","last_clean":"2024-03-06T07:27:10.279670+0000","last_became_active":"2024-03-06T07:27:10.276523+0000","last_became_peered":"2024-03-06T07:27:10.276523+0000","last_unstale":"2024-03-06T07:27:10.279670+0000","last_undegraded":"2024-03-06T07:27:10.279670+0000","last_fullsized":"2024-03-06T07:27:10.279670+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:25.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269843+0000","last_change":"2024-03-06T07:27:10.269843+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269843+0000","last_undegraded":"2024-03-06T07:27:10.269843+0000","last_fullsized":"2024-03-06T07:27:10.269843+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.279775+0000","last_change":"2024-03-06T07:27:10.278830+0000","last_active":"2024-03-06T07:27:10.279775+0000","last_peered":"2024-03-06T07:27:10.279775+0000","last_clean":"2024-03-06T07:27:10.279775+0000","last_became_active":"2024-03-06T07:27:10.278106+0000","last_became_peered":"2024-03-06T07:27:10.278106+0000","last_unstale":"2024-03-06T07:27:10.279775+0000","last_undegraded":"2024-03-06T07:27:10.279775+0000","last_fullsized":"2024-03-06T07:27:10.279775+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.266466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.269136+0000","last_change":"2024-03-06T07:27:10.269136+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.269136+0000","last_undegraded":"2024-03-06T07:27:10.269136+0000","last_fullsized":"2024-03-06T07:27:10.269136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274650+0000","last_change":"2024-03-06T07:27:10.271560+0000","last_active":"2024-03-06T07:27:10.274650+0000","last_peered":"2024-03-06T07:27:10.274650+0000","last_clean":"2024-03-06T07:27:10.274650+0000","last_became_active":"2024-03-06T07:27:10.271296+0000","last_became_peered":"2024-03-06T07:27:10.271296+0000","last_unstale":"2024-03-06T07:27:10.274650+0000","last_undegraded":"2024-03-06T07:27:10.274650+0000","last_fullsized":"2024-03-06T07:27:10.274650+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:00.541674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263530+0000","last_change":"2024-03-06T07:27:10.263530+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263530+0000","last_undegraded":"2024-03-06T07:27:10.263530+0000","last_fullsized":"2024-03-06T07:27:10.263530+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281370+0000","last_change":"2024-03-06T07:27:10.279864+0000","last_active":"2024-03-06T07:27:10.281370+0000","last_peered":"2024-03-06T07:27:10.281370+0000","last_clean":"2024-03-06T07:27:10.281370+0000","last_became_active":"2024-03-06T07:27:10.279622+0000","last_became_peered":"2024-03-06T07:27:10.279622+0000","last_unstale":"2024-03-06T07:27:10.281370+0000","last_undegraded":"2024-03-06T07:27:10.281370+0000","last_fullsized":"2024-03-06T07:27:10.281370+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:33.259974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264152+0000","last_change":"2024-03-06T07:27:10.264152+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264152+0000","last_undegraded":"2024-03-06T07:27:10.264152+0000","last_fullsized":"2024-03-06T07:27:10.264152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276867+0000","last_change":"2024-03-06T07:27:10.273788+0000","last_active":"2024-03-06T07:27:10.276867+0000","last_peered":"2024-03-06T07:27:10.276867+0000","last_clean":"2024-03-06T07:27:10.276867+0000","last_became_active":"2024-03-06T07:27:10.273542+0000","last_became_peered":"2024-03-06T07:27:10.273542+0000","last_unstale":"2024-03-06T07:27:10.276867+0000","last_undegraded":"2024-03-06T07:27:10.276867+0000","last_fullsized":"2024-03-06T07:27:10.276867+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:10.176108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.266917+0000","last_change":"2024-03-06T07:27:10.266917+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.266917+0000","last_undegraded":"2024-03-06T07:27:10.266917+0000","last_fullsized":"2024-03-06T07:27:10.266917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.270719+0000","last_change":"2024-03-06T07:27:10.269615+0000","last_active":"2024-03-06T07:27:10.270719+0000","last_peered":"2024-03-06T07:27:10.270719+0000","last_clean":"2024-03-06T07:27:10.270719+0000","last_became_active":"2024-03-06T07:27:10.269331+0000","last_became_peered":"2024-03-06T07:27:10.269331+0000","last_unstale":"2024-03-06T07:27:10.270719+0000","last_undegraded":"2024-03-06T07:27:10.270719+0000","last_fullsized":"2024-03-06T07:27:10.270719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:31.221037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.265082+0000","last_change":"2024-03-06T07:27:10.265082+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.265082+0000","last_undegraded":"2024-03-06T07:27:10.265082+0000","last_fullsized":"2024-03-06T07:27:10.265082+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274598+0000","last_change":"2024-03-06T07:27:10.273864+0000","last_active":"2024-03-06T07:27:10.274598+0000","last_peered":"2024-03-06T07:27:10.274598+0000","last_clean":"2024-03-06T07:27:10.274598+0000","last_became_active":"2024-03-06T07:27:10.273611+0000","last_became_peered":"2024-03-06T07:27:10.273611+0000","last_unstale":"2024-03-06T07:27:10.274598+0000","last_undegraded":"2024-03-06T07:27:10.274598+0000","last_fullsized":"2024-03-06T07:27:10.274598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:49.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263562+0000","last_change":"2024-03-06T07:27:10.263562+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263562+0000","last_undegraded":"2024-03-06T07:27:10.263562+0000","last_fullsized":"2024-03-06T07:27:10.263562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276271+0000","last_change":"2024-03-06T07:27:10.271689+0000","last_active":"2024-03-06T07:27:10.276271+0000","last_peered":"2024-03-06T07:27:10.276271+0000","last_clean":"2024-03-06T07:27:10.276271+0000","last_became_active":"2024-03-06T07:27:10.271421+0000","last_became_peered":"2024-03-06T07:27:10.271421+0000","last_unstale":"2024-03-06T07:27:10.276271+0000","last_undegraded":"2024-03-06T07:27:10.276271+0000","last_fullsized":"2024-03-06T07:27:10.276271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:20.049932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.261430+0000","last_change":"2024-03-06T07:27:10.261430+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.261430+0000","last_undegraded":"2024-03-06T07:27:10.261430+0000","last_fullsized":"2024-03-06T07:27:10.261430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.269674+0000","last_change":"2024-03-06T07:27:10.268891+0000","last_active":"2024-03-06T07:27:10.269674+0000","last_peered":"2024-03-06T07:27:10.269674+0000","last_clean":"2024-03-06T07:27:10.269674+0000","last_became_active":"2024-03-06T07:27:10.268536+0000","last_became_peered":"2024-03-06T07:27:10.268536+0000","last_unstale":"2024-03-06T07:27:10.269674+0000","last_undegraded":"2024-03-06T07:27:10.269674+0000","last_fullsized":"2024-03-06T07:27:10.269674+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:51.960409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263082+0000","last_change":"2024-03-06T07:27:10.263082+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263082+0000","last_undegraded":"2024-03-06T07:27:10.263082+0000","last_fullsized":"2024-03-06T07:27:10.263082+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276122+0000","last_change":"2024-03-06T07:27:10.272914+0000","last_active":"2024-03-06T07:27:10.276122+0000","last_peered":"2024-03-06T07:27:10.276122+0000","last_clean":"2024-03-06T07:27:10.276122+0000","last_became_active":"2024-03-06T07:27:10.272635+0000","last_became_peered":"2024-03-06T07:27:10.272635+0000","last_unstale":"2024-03-06T07:27:10.276122+0000","last_undegraded":"2024-03-06T07:27:10.276122+0000","last_fullsized":"2024-03-06T07:27:10.276122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:58.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.268710+0000","last_change":"2024-03-06T07:27:10.268710+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.268710+0000","last_undegraded":"2024-03-06T07:27:10.268710+0000","last_fullsized":"2024-03-06T07:27:10.268710+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.278239+0000","last_change":"2024-03-06T07:27:10.276957+0000","last_active":"2024-03-06T07:27:10.278239+0000","last_peered":"2024-03-06T07:27:10.278239+0000","last_clean":"2024-03-06T07:27:10.278239+0000","last_became_active":"2024-03-06T07:27:10.276707+0000","last_became_peered":"2024-03-06T07:27:10.276707+0000","last_unstale":"2024-03-06T07:27:10.278239+0000","last_undegraded":"2024-03-06T07:27:10.278239+0000","last_fullsized":"2024-03-06T07:27:10.278239+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:31.396430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.264427+0000","last_change":"2024-03-06T07:27:10.264427+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.264427+0000","last_undegraded":"2024-03-06T07:27:10.264427+0000","last_fullsized":"2024-03-06T07:27:10.264427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.276211+0000","last_change":"2024-03-06T07:27:10.275390+0000","last_active":"2024-03-06T07:27:10.276211+0000","last_peered":"2024-03-06T07:27:10.276211+0000","last_clean":"2024-03-06T07:27:10.276211+0000","last_became_active":"2024-03-06T07:27:10.274903+0000","last_became_peered":"2024-03-06T07:27:10.274903+0000","last_unstale":"2024-03-06T07:27:10.276211+0000","last_undegraded":"2024-03-06T07:27:10.276211+0000","last_fullsized":"2024-03-06T07:27:10.276211+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:02.332832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.263304+0000","last_change":"2024-03-06T07:27:10.263304+0000","last_active":"2024-03-06T07:27:10.253454+0000","last_peered":"2024-03-06T07:27:10.253454+0000","last_clean":"2024-03-06T07:27:10.253454+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.263304+0000","last_undegraded":"2024-03-06T07:27:10.263304+0000","last_fullsized":"2024-03-06T07:27:10.263304+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.281890+0000","last_change":"2024-03-06T07:27:10.277891+0000","last_active":"2024-03-06T07:27:10.281890+0000","last_peered":"2024-03-06T07:27:10.281890+0000","last_clean":"2024-03-06T07:27:10.281890+0000","last_became_active":"2024-03-06T07:27:10.277582+0000","last_became_peered":"2024-03-06T07:27:10.277582+0000","last_unstale":"2024-03-06T07:27:10.281890+0000","last_undegraded":"2024-03-06T07:27:10.281890+0000","last_fullsized":"2024-03-06T07:27:10.281890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:16.855383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":332,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.274060+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:27:10.274060+0000","last_peered":"2024-03-06T07:27:10.274060+0000","last_clean":"2024-03-06T07:27:10.274060+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:27:10.274060+0000","last_undegraded":"2024-03-06T07:27:10.274060+0000","last_fullsized":"2024-03-06T07:27:10.274060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":1156184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40776,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331064,"statfs":{"total":96636764160,"available":96595009536,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40792,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331048,"statfs":{"total":96636764160,"available":96594993152,"internally_reserved":0,"allocated":1306624,"data_stored":559541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41356,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330484,"statfs":{"total":96636764160,"available":96594415616,"internally_reserved":0,"allocated":1900544,"data_stored":1149909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:15.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3f is not active+clean 2024-03-06T07:27:15.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.267598+0000', 'last_change': '2024-03-06T07:27:10.267598+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.267598+0000', 'last_undegraded': '2024-03-06T07:27:10.267598+0000', 'last_fullsized': '2024-03-06T07:27:10.267598+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3e is not active+clean 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.261734+0000', 'last_change': '2024-03-06T07:27:10.261734+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.261734+0000', 'last_undegraded': '2024-03-06T07:27:10.261734+0000', 'last_fullsized': '2024-03-06T07:27:10.261734+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3d is not active+clean 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.273806+0000', 'last_change': '2024-03-06T07:27:10.273806+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.273806+0000', 'last_undegraded': '2024-03-06T07:27:10.273806+0000', 'last_fullsized': '2024-03-06T07:27:10.273806+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3c is not active+clean 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265833+0000', 'last_change': '2024-03-06T07:27:10.265833+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265833+0000', 'last_undegraded': '2024-03-06T07:27:10.265833+0000', 'last_fullsized': '2024-03-06T07:27:10.265833+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3b is not active+clean 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.266357+0000', 'last_change': '2024-03-06T07:27:10.266357+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.266357+0000', 'last_undegraded': '2024-03-06T07:27:10.266357+0000', 'last_fullsized': '2024-03-06T07:27:10.266357+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3a is not active+clean 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265033+0000', 'last_change': '2024-03-06T07:27:10.265033+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265033+0000', 'last_undegraded': '2024-03-06T07:27:10.265033+0000', 'last_fullsized': '2024-03-06T07:27:10.265033+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.39 is not active+clean 2024-03-06T07:27:15.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265851+0000', 'last_change': '2024-03-06T07:27:10.265851+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265851+0000', 'last_undegraded': '2024-03-06T07:27:10.265851+0000', 'last_fullsized': '2024-03-06T07:27:10.265851+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.38 is not active+clean 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.269538+0000', 'last_change': '2024-03-06T07:27:10.269538+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.269538+0000', 'last_undegraded': '2024-03-06T07:27:10.269538+0000', 'last_fullsized': '2024-03-06T07:27:10.269538+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.37 is not active+clean 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.267258+0000', 'last_change': '2024-03-06T07:27:10.267258+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.267258+0000', 'last_undegraded': '2024-03-06T07:27:10.267258+0000', 'last_fullsized': '2024-03-06T07:27:10.267258+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.36 is not active+clean 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.273029+0000', 'last_change': '2024-03-06T07:27:10.273029+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.273029+0000', 'last_undegraded': '2024-03-06T07:27:10.273029+0000', 'last_fullsized': '2024-03-06T07:27:10.273029+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.35 is not active+clean 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.261449+0000', 'last_change': '2024-03-06T07:27:10.261449+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.261449+0000', 'last_undegraded': '2024-03-06T07:27:10.261449+0000', 'last_fullsized': '2024-03-06T07:27:10.261449+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.34 is not active+clean 2024-03-06T07:27:15.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265454+0000', 'last_change': '2024-03-06T07:27:10.265454+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265454+0000', 'last_undegraded': '2024-03-06T07:27:10.265454+0000', 'last_fullsized': '2024-03-06T07:27:10.265454+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.33 is not active+clean 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.268184+0000', 'last_change': '2024-03-06T07:27:10.268184+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.268184+0000', 'last_undegraded': '2024-03-06T07:27:10.268184+0000', 'last_fullsized': '2024-03-06T07:27:10.268184+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.32 is not active+clean 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.268837+0000', 'last_change': '2024-03-06T07:27:10.268837+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.268837+0000', 'last_undegraded': '2024-03-06T07:27:10.268837+0000', 'last_fullsized': '2024-03-06T07:27:10.268837+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.31 is not active+clean 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264950+0000', 'last_change': '2024-03-06T07:27:10.264950+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264950+0000', 'last_undegraded': '2024-03-06T07:27:10.264950+0000', 'last_fullsized': '2024-03-06T07:27:10.264950+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.30 is not active+clean 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.268981+0000', 'last_change': '2024-03-06T07:27:10.268981+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.268981+0000', 'last_undegraded': '2024-03-06T07:27:10.268981+0000', 'last_fullsized': '2024-03-06T07:27:10.268981+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2f is not active+clean 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.262813+0000', 'last_change': '2024-03-06T07:27:10.262813+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.262813+0000', 'last_undegraded': '2024-03-06T07:27:10.262813+0000', 'last_fullsized': '2024-03-06T07:27:10.262813+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2e is not active+clean 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264573+0000', 'last_change': '2024-03-06T07:27:10.264573+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264573+0000', 'last_undegraded': '2024-03-06T07:27:10.264573+0000', 'last_fullsized': '2024-03-06T07:27:10.264573+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2d is not active+clean 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.273199+0000', 'last_change': '2024-03-06T07:27:10.273199+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.273199+0000', 'last_undegraded': '2024-03-06T07:27:10.273199+0000', 'last_fullsized': '2024-03-06T07:27:10.273199+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2c is not active+clean 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264185+0000', 'last_change': '2024-03-06T07:27:10.264185+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264185+0000', 'last_undegraded': '2024-03-06T07:27:10.264185+0000', 'last_fullsized': '2024-03-06T07:27:10.264185+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2b is not active+clean 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.261404+0000', 'last_change': '2024-03-06T07:27:10.261404+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.261404+0000', 'last_undegraded': '2024-03-06T07:27:10.261404+0000', 'last_fullsized': '2024-03-06T07:27:10.261404+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2a is not active+clean 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.268216+0000', 'last_change': '2024-03-06T07:27:10.268216+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.268216+0000', 'last_undegraded': '2024-03-06T07:27:10.268216+0000', 'last_fullsized': '2024-03-06T07:27:10.268216+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.29 is not active+clean 2024-03-06T07:27:15.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.261822+0000', 'last_change': '2024-03-06T07:27:10.261822+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.261822+0000', 'last_undegraded': '2024-03-06T07:27:10.261822+0000', 'last_fullsized': '2024-03-06T07:27:10.261822+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.28 is not active+clean 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265508+0000', 'last_change': '2024-03-06T07:27:10.265508+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265508+0000', 'last_undegraded': '2024-03-06T07:27:10.265508+0000', 'last_fullsized': '2024-03-06T07:27:10.265508+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.27 is not active+clean 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.262335+0000', 'last_change': '2024-03-06T07:27:10.262335+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.262335+0000', 'last_undegraded': '2024-03-06T07:27:10.262335+0000', 'last_fullsized': '2024-03-06T07:27:10.262335+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.26 is not active+clean 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.272656+0000', 'last_change': '2024-03-06T07:27:10.272656+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.272656+0000', 'last_undegraded': '2024-03-06T07:27:10.272656+0000', 'last_fullsized': '2024-03-06T07:27:10.272656+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.25 is not active+clean 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265513+0000', 'last_change': '2024-03-06T07:27:10.265513+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265513+0000', 'last_undegraded': '2024-03-06T07:27:10.265513+0000', 'last_fullsized': '2024-03-06T07:27:10.265513+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.24 is not active+clean 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264450+0000', 'last_change': '2024-03-06T07:27:10.264450+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264450+0000', 'last_undegraded': '2024-03-06T07:27:10.264450+0000', 'last_fullsized': '2024-03-06T07:27:10.264450+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.23 is not active+clean 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264152+0000', 'last_change': '2024-03-06T07:27:10.264152+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264152+0000', 'last_undegraded': '2024-03-06T07:27:10.264152+0000', 'last_fullsized': '2024-03-06T07:27:10.264152+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.22 is not active+clean 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265454+0000', 'last_change': '2024-03-06T07:27:10.265454+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265454+0000', 'last_undegraded': '2024-03-06T07:27:10.265454+0000', 'last_fullsized': '2024-03-06T07:27:10.265454+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.21 is not active+clean 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265052+0000', 'last_change': '2024-03-06T07:27:10.265052+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265052+0000', 'last_undegraded': '2024-03-06T07:27:10.265052+0000', 'last_fullsized': '2024-03-06T07:27:10.265052+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.20 is not active+clean 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.267742+0000', 'last_change': '2024-03-06T07:27:10.267742+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.267742+0000', 'last_undegraded': '2024-03-06T07:27:10.267742+0000', 'last_fullsized': '2024-03-06T07:27:10.267742+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265433+0000', 'last_change': '2024-03-06T07:27:10.265433+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265433+0000', 'last_undegraded': '2024-03-06T07:27:10.265433+0000', 'last_fullsized': '2024-03-06T07:27:10.265433+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1f is not active+clean 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265204+0000', 'last_change': '2024-03-06T07:27:10.265204+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265204+0000', 'last_undegraded': '2024-03-06T07:27:10.265204+0000', 'last_fullsized': '2024-03-06T07:27:10.265204+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.266552+0000', 'last_change': '2024-03-06T07:27:10.266552+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.266552+0000', 'last_undegraded': '2024-03-06T07:27:10.266552+0000', 'last_fullsized': '2024-03-06T07:27:10.266552+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.271902+0000', 'last_change': '2024-03-06T07:27:10.271902+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.271902+0000', 'last_undegraded': '2024-03-06T07:27:10.271902+0000', 'last_fullsized': '2024-03-06T07:27:10.271902+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1a is not active+clean 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264549+0000', 'last_change': '2024-03-06T07:27:10.264549+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264549+0000', 'last_undegraded': '2024-03-06T07:27:10.264549+0000', 'last_fullsized': '2024-03-06T07:27:10.264549+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263608+0000', 'last_change': '2024-03-06T07:27:10.263608+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263608+0000', 'last_undegraded': '2024-03-06T07:27:10.263608+0000', 'last_fullsized': '2024-03-06T07:27:10.263608+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.18 is not active+clean 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.268623+0000', 'last_change': '2024-03-06T07:27:10.268623+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.268623+0000', 'last_undegraded': '2024-03-06T07:27:10.268623+0000', 'last_fullsized': '2024-03-06T07:27:10.268623+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-03-06T07:27:15.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.266447+0000', 'last_change': '2024-03-06T07:27:10.266447+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.266447+0000', 'last_undegraded': '2024-03-06T07:27:10.266447+0000', 'last_fullsized': '2024-03-06T07:27:10.266447+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264647+0000', 'last_change': '2024-03-06T07:27:10.264647+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264647+0000', 'last_undegraded': '2024-03-06T07:27:10.264647+0000', 'last_fullsized': '2024-03-06T07:27:10.264647+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.17 is not active+clean 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264239+0000', 'last_change': '2024-03-06T07:27:10.264239+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264239+0000', 'last_undegraded': '2024-03-06T07:27:10.264239+0000', 'last_fullsized': '2024-03-06T07:27:10.264239+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263939+0000', 'last_change': '2024-03-06T07:27:10.263939+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263939+0000', 'last_undegraded': '2024-03-06T07:27:10.263939+0000', 'last_fullsized': '2024-03-06T07:27:10.263939+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-03-06T07:27:15.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264009+0000', 'last_change': '2024-03-06T07:27:10.264009+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264009+0000', 'last_undegraded': '2024-03-06T07:27:10.264009+0000', 'last_fullsized': '2024-03-06T07:27:10.264009+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263293+0000', 'last_change': '2024-03-06T07:27:10.263293+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263293+0000', 'last_undegraded': '2024-03-06T07:27:10.263293+0000', 'last_fullsized': '2024-03-06T07:27:10.263293+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263736+0000', 'last_change': '2024-03-06T07:27:10.263736+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263736+0000', 'last_undegraded': '2024-03-06T07:27:10.263736+0000', 'last_fullsized': '2024-03-06T07:27:10.263736+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264215+0000', 'last_change': '2024-03-06T07:27:10.264215+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264215+0000', 'last_undegraded': '2024-03-06T07:27:10.264215+0000', 'last_fullsized': '2024-03-06T07:27:10.264215+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.270279+0000', 'last_change': '2024-03-06T07:27:10.270279+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.270279+0000', 'last_undegraded': '2024-03-06T07:27:10.270279+0000', 'last_fullsized': '2024-03-06T07:27:10.270279+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265754+0000', 'last_change': '2024-03-06T07:27:10.265754+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265754+0000', 'last_undegraded': '2024-03-06T07:27:10.265754+0000', 'last_fullsized': '2024-03-06T07:27:10.265754+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-03-06T07:27:15.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.269219+0000', 'last_change': '2024-03-06T07:27:10.269219+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.269219+0000', 'last_undegraded': '2024-03-06T07:27:10.269219+0000', 'last_fullsized': '2024-03-06T07:27:10.269219+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.269745+0000', 'last_change': '2024-03-06T07:27:10.269745+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.269745+0000', 'last_undegraded': '2024-03-06T07:27:10.269745+0000', 'last_fullsized': '2024-03-06T07:27:10.269745+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.267319+0000', 'last_change': '2024-03-06T07:27:10.267319+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.267319+0000', 'last_undegraded': '2024-03-06T07:27:10.267319+0000', 'last_fullsized': '2024-03-06T07:27:10.267319+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.b is not active+clean 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.269843+0000', 'last_change': '2024-03-06T07:27:10.269843+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.269843+0000', 'last_undegraded': '2024-03-06T07:27:10.269843+0000', 'last_fullsized': '2024-03-06T07:27:10.269843+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.269136+0000', 'last_change': '2024-03-06T07:27:10.269136+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.269136+0000', 'last_undegraded': '2024-03-06T07:27:10.269136+0000', 'last_fullsized': '2024-03-06T07:27:10.269136+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263530+0000', 'last_change': '2024-03-06T07:27:10.263530+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263530+0000', 'last_undegraded': '2024-03-06T07:27:10.263530+0000', 'last_fullsized': '2024-03-06T07:27:10.263530+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.6 is not active+clean 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264152+0000', 'last_change': '2024-03-06T07:27:10.264152+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264152+0000', 'last_undegraded': '2024-03-06T07:27:10.264152+0000', 'last_fullsized': '2024-03-06T07:27:10.264152+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.7 is not active+clean 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.266917+0000', 'last_change': '2024-03-06T07:27:10.266917+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.266917+0000', 'last_undegraded': '2024-03-06T07:27:10.266917+0000', 'last_fullsized': '2024-03-06T07:27:10.266917+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.4 is not active+clean 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.265082+0000', 'last_change': '2024-03-06T07:27:10.265082+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.265082+0000', 'last_undegraded': '2024-03-06T07:27:10.265082+0000', 'last_fullsized': '2024-03-06T07:27:10.265082+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.5 is not active+clean 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263562+0000', 'last_change': '2024-03-06T07:27:10.263562+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263562+0000', 'last_undegraded': '2024-03-06T07:27:10.263562+0000', 'last_fullsized': '2024-03-06T07:27:10.263562+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.261430+0000', 'last_change': '2024-03-06T07:27:10.261430+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.261430+0000', 'last_undegraded': '2024-03-06T07:27:10.261430+0000', 'last_fullsized': '2024-03-06T07:27:10.261430+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263082+0000', 'last_change': '2024-03-06T07:27:10.263082+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263082+0000', 'last_undegraded': '2024-03-06T07:27:10.263082+0000', 'last_fullsized': '2024-03-06T07:27:10.263082+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.268710+0000', 'last_change': '2024-03-06T07:27:10.268710+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.268710+0000', 'last_undegraded': '2024-03-06T07:27:10.268710+0000', 'last_fullsized': '2024-03-06T07:27:10.268710+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1 is not active+clean 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.264427+0000', 'last_change': '2024-03-06T07:27:10.264427+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.264427+0000', 'last_undegraded': '2024-03-06T07:27:10.264427+0000', 'last_fullsized': '2024-03-06T07:27:10.264427+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-03-06T07:27:15.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.263304+0000', 'last_change': '2024-03-06T07:27:10.263304+0000', 'last_active': '2024-03-06T07:27:10.253454+0000', 'last_peered': '2024-03-06T07:27:10.253454+0000', 'last_clean': '2024-03-06T07:27:10.253454+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.263304+0000', 'last_undegraded': '2024-03-06T07:27:10.263304+0000', 'last_fullsized': '2024-03-06T07:27:10.263304+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.253454+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:27:15.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.159+0000 7f838e3d7640 1 Processor -- start 2024-03-06T07:27:15.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.159+0000 7f838e3d7640 1 -- start start 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.159+0000 7f838e3d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 0x7f83880d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.159+0000 7f838e3d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880d22c0 con 0x7f83880b4460 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.159+0000 7f838e3d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880d25a0 con 0x7f8388162970 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.159+0000 7f838e3d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8388161b00 con 0x7f8388162400 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8387fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f83880b4460 0x7f83880b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49964/0 (socket says 172.21.15.122:49964) 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8387fff640 1 -- 172.21.15.122:0/6047344 learned_addr learned my addr 172.21.15.122:0/6047344 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:15.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f838c974640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 0x7f83880d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888234704 0 0) 0x7f83880d22c0 con 0x7f83880b4460 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8370003520 con 0x7f83880b4460 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916122882 0 0) 0x7f83880d25a0 con 0x7f8388162970 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83880d22c0 con 0x7f8388162970 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1412274121 0 0) 0x7f8370003520 con 0x7f83880b4460 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 msgr2=0x7f83880d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 0x7f83880d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 >> v1:172.21.15.169:6789/0 conn(0x7f8388162970 legacy=0x7f8388161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880d2880 con 0x7f83880b4460 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f8378001670 con 0x7f83880b4460 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 >> v1:172.21.15.122:6789/0 conn(0x7f83880b4460 legacy=0x7f83880b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 0x7f8370003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83700040d0 0x7f8370004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8370004d40 0x7f83700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388161b00 con 0x7f83700040d0 2024-03-06T07:27:15.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837000baa0 con 0x7f8370004d40 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837000bca0 con 0x7f8388162400 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f83877fe640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83700040d0 0x7f8370004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f8387fff640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 0x7f8370003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.160+0000 7f838c974640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8370004d40 0x7f83700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f83877fe640 1 -- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 msgr2=0x7f8370003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f83877fe640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8388162400 0x7f8370003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f83877fe640 1 -- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8370004d40 msgr2=0x7f83700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f83877fe640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8370004d40 0x7f83700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f83877fe640 1 -- 172.21.15.122:0/6047344 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370003520 con 0x7f83700040d0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f83877fe640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83700040d0 0x7f8370004800 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f837c0018a0 tx=0x7f8370005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d83cffffb8b8685 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.161+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c037070 con 0x7f83700040d0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c0052d0 con 0x7f83700040d0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f8386ffd640 1 -- 172.21.15.122:0/6047344 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c0014f0 con 0x7f83700040d0 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838e3d7640 1 -- 172.21.15.122:0/6047344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83700040d0 msgr2=0x7f8370004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838e3d7640 1 --2- 172.21.15.122:0/6047344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83700040d0 0x7f8370004800 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f837c0018a0 tx=0x7f8370005830 comp rx=0 tx=0).stop 2024-03-06T07:27:15.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838c974640 1 -- 172.21.15.122:0/6047344 reap_dead start 2024-03-06T07:27:15.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838e3d7640 1 -- 172.21.15.122:0/6047344 shutdown_connections 2024-03-06T07:27:15.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838e3d7640 1 -- 172.21.15.122:0/6047344 >> 172.21.15.122:0/6047344 conn(0x7f83880d3ce0 msgr2=0x7f83880d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838e3d7640 1 -- 172.21.15.122:0/6047344 shutdown_connections 2024-03-06T07:27:15.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.162+0000 7f838e3d7640 1 -- 172.21.15.122:0/6047344 wait complete. 2024-03-06T07:27:15.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.163+0000 7f838e3d7640 1 Processor -- start 2024-03-06T07:27:15.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.163+0000 7f838e3d7640 1 -- start start 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838e3d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83700040d0 0x7f83881f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838e3d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83880b6900 0x7f83881f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838e3d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 0x7f8388153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838e3d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880ce950 con 0x7f83880b6d80 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838e3d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880cecc0 con 0x7f83880b6900 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838e3d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b4840 con 0x7f83700040d0 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838c974640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 0x7f8388153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838c974640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 0x7f8388153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51844/0 (socket says 172.21.15.122:51844) 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838c974640 1 -- 172.21.15.122:0/3068515091 learned_addr learned my addr 172.21.15.122:0/3068515091 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f8387fff640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83700040d0 0x7f83881f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f83877fe640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83880b6900 0x7f83881f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838c974640 1 -- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83700040d0 msgr2=0x7f83881f7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.164+0000 7f838c974640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83700040d0 0x7f83881f7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f838c974640 1 -- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83880b6900 msgr2=0x7f83881f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f838c974640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83880b6900 0x7f83881f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f838c974640 1 -- 172.21.15.122:0/3068515091 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880c8740 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f838c974640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 0x7f8388153ea0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f83880c7610 tx=0x7f838000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838000fdb0 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380015430 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8380015820 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83880c8a20 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.165+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880c8fe0 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.166+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388161800 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.167+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83800159c0 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.167+0000 7f8384ff9640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f837406b4d0 0x7f837406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.167+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f838008aed0 con 0x7f83880b6d80 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.167+0000 7f8387fff640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f837406b4d0 0x7f837406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.168+0000 7f8387fff640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f837406b4d0 0x7f837406d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f83880c9980 tx=0x7f8378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.176+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8388161800 con 0x7f83880b6d80 2024-03-06T07:27:15.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.431+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f838804f330 con 0x7f83880b6d80 2024-03-06T07:27:15.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.431+0000 7f8384ff9640 1 -- 172.21.15.122:0/3068515091 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1454 (secure 0 0 0) 0x7f8380019030 con 0x7f83880b6d80 2024-03-06T07:27:15.433 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:15.433 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":108,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":331038720,"bytes_avail":772763074560,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":61,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":7962}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-06T07:26:59.393962+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f837406b4d0 msgr2=0x7f837406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f837406b4d0 0x7f837406d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f83880c9980 tx=0x7f8378009040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 msgr2=0x7f8388153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 0x7f8388153ea0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f83880c7610 tx=0x7f838000f580 comp rx=0 tx=0).stop 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 shutdown_connections 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f837406b4d0 0x7f837406d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83880b6d80 0x7f8388153ea0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f83880b6900 0x7f83881f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 --2- 172.21.15.122:0/3068515091 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f83700040d0 0x7f83881f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 >> 172.21.15.122:0/3068515091 conn(0x7f83880d3ce0 msgr2=0x7f83880d1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 shutdown_connections 2024-03-06T07:27:15.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.434+0000 7f838e3d7640 1 -- 172.21.15.122:0/3068515091 wait complete. 2024-03-06T07:27:15.447 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:27:15.447 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:15.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.546+0000 7f8d97c0e640 1 Processor -- start 2024-03-06T07:27:15.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.546+0000 7f8d97c0e640 1 -- start start 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d97c0e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d900d4760 0x7f8d900b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d97c0e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8d900d0a80 0x7f8d900cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d97c0e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d900d25d0 con 0x7f8d900b4170 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d97c0e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d900d4bf0 con 0x7f8d900d4760 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d97c0e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d900ce9e0 con 0x7f8d900d0a80 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d959aa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8d900b4170 0x7f8d900d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49974/0 (socket says 172.21.15.122:49974) 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d951a9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d900d4760 0x7f8d900b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d959aa640 1 -- 172.21.15.122:0/3895730766 learned_addr learned my addr 172.21.15.122:0/3895730766 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d951a9640 1 -- 172.21.15.122:0/3895730766 >> v2:172.21.15.169:3300/0 conn(0x7f8d900d0a80 msgr2=0x7f8d900cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d951a9640 1 --2- 172.21.15.122:0/3895730766 >> v2:172.21.15.169:3300/0 conn(0x7f8d900d0a80 0x7f8d900cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d951a9640 1 -- 172.21.15.122:0/3895730766 >> v1:172.21.15.122:6789/0 conn(0x7f8d900b4170 legacy=0x7f8d900d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d951a9640 1 -- 172.21.15.122:0/3895730766 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900d2870 con 0x7f8d900d4760 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.548+0000 7f8d949a8640 1 -- 172.21.15.122:0/3895730766 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732253458 0 0) 0x7f8d900d25d0 con 0x7f8d900b4170 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d951a9640 1 --2- 172.21.15.122:0/3895730766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900d4760 0x7f8d900b6a90 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8d9008df30 tx=0x7f8d8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3a1ab87878837d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d949a8640 1 -- 172.21.15.122:0/3895730766 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84035020 con 0x7f8d900d4760 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d949a8640 1 -- 172.21.15.122:0/3895730766 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d84031830 con 0x7f8d900d4760 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d949a8640 1 -- 172.21.15.122:0/3895730766 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84031c00 con 0x7f8d900d4760 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d97c0e640 1 -- 172.21.15.122:0/3895730766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900d4760 msgr2=0x7f8d900b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/3895730766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900d4760 0x7f8d900b6a90 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8d9008df30 tx=0x7f8d8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d97c0e640 1 -- 172.21.15.122:0/3895730766 shutdown_connections 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/3895730766 >> v2:172.21.15.169:3300/0 conn(0x7f8d900d0a80 0x7f8d900cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/3895730766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900d4760 0x7f8d900b6a90 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.549+0000 7f8d97c0e640 1 -- 172.21.15.122:0/3895730766 >> 172.21.15.122:0/3895730766 conn(0x7f8d900d3c40 msgr2=0x7f8d900d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.550+0000 7f8d97c0e640 1 -- 172.21.15.122:0/3895730766 shutdown_connections 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.550+0000 7f8d97c0e640 1 -- 172.21.15.122:0/3895730766 wait complete. 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 Processor -- start 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 -- start start 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 0x7f8d9015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8d900d0a80 0x7f8d9006be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d9006c380 0x7f8d9015c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900b4dd0 con 0x7f8d900b4170 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900b4550 con 0x7f8d9006c380 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d97c0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900b4f50 con 0x7f8d900d0a80 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.551+0000 7f8d959aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 0x7f8d9015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 0x7f8d9015b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51858/0 (socket says 172.21.15.122:51858) 2024-03-06T07:27:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 -- 172.21.15.122:0/2924266351 learned_addr learned my addr 172.21.15.122:0/2924266351 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d951a9640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8d900d0a80 0x7f8d9006be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d961ab640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d9006c380 0x7f8d9015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 -- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8d900d0a80 msgr2=0x7f8d9006be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8d900d0a80 0x7f8d9006be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 -- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d9006c380 msgr2=0x7f8d9015c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d9006c380 0x7f8d9015c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 -- 172.21.15.122:0/2924266351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900b07f0 con 0x7f8d900b4170 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d951a9640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8d900d0a80 0x7f8d9006be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d959aa640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 0x7f8d9015b370 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00c4a0 tx=0x7f8d8c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c017070 con 0x7f8d900b4170 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c005ce0 con 0x7f8d900b4170 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0072b0 con 0x7f8d900b4170 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d900b0a90 con 0x7f8d900b4170 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.552+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d900b1050 con 0x7f8d900b4170 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.553+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9006c790 con 0x7f8d900b4170 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.553+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d8c01a070 con 0x7f8d900b4170 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.554+0000 7f8d827fc640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8d7406b4d0 0x7f8d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.554+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d8c0894c0 con 0x7f8d900b4170 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.554+0000 7f8d951a9640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8d7406b4d0 0x7f8d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.554+0000 7f8d951a9640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8d7406b4d0 0x7f8d7406d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8d84036000 tx=0x7f8d84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.558+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d9006c790 con 0x7f8d900b4170 2024-03-06T07:27:15.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.734+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d9004f330 con 0x7f8d7406b4d0 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.740+0000 7f8d827fc640 1 -- 172.21.15.122:0/2924266351 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f8d9004f330 con 0x7f8d7406b4d0 2024-03-06T07:27:15.744 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8d7406b4d0 msgr2=0x7f8d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8d7406b4d0 0x7f8d7406d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8d84036000 tx=0x7f8d84008030 comp rx=0 tx=0).stop 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 msgr2=0x7f8d9015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 0x7f8d9015b370 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00c4a0 tx=0x7f8d8c00c960 comp rx=0 tx=0).stop 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 shutdown_connections 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8d7406b4d0 0x7f8d7406d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d9006c380 0x7f8d9015c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8d900d0a80 0x7f8d9006be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 --2- 172.21.15.122:0/2924266351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d900b4170 0x7f8d9015b370 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 >> 172.21.15.122:0/2924266351 conn(0x7f8d900d3c40 msgr2=0x7f8d900b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 shutdown_connections 2024-03-06T07:27:15.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:15.743+0000 7f8d97c0e640 1 -- 172.21.15.122:0/2924266351 wait complete. 2024-03-06T07:27:15.756 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-03-06T07:27:15.396306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323584,"kb_used_data":11720,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651136,"statfs":{"total":773094113280,"available":772762763264,"internally_reserved":0,"allocated":12001280,"data_stored":5816684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136473+0000","last_change":"2024-03-06T07:27:11.270188+0000","last_active":"2024-03-06T07:27:13.136473+0000","last_peered":"2024-03-06T07:27:13.136473+0000","last_clean":"2024-03-06T07:27:13.136473+0000","last_became_active":"2024-03-06T07:27:11.269882+0000","last_became_peered":"2024-03-06T07:27:11.269882+0000","last_unstale":"2024-03-06T07:27:13.136473+0000","last_undegraded":"2024-03-06T07:27:13.136473+0000","last_fullsized":"2024-03-06T07:27:13.136473+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:47.558402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281328+0000","last_change":"2024-03-06T07:27:11.286409+0000","last_active":"2024-03-06T07:27:12.281328+0000","last_peered":"2024-03-06T07:27:12.281328+0000","last_clean":"2024-03-06T07:27:12.281328+0000","last_became_active":"2024-03-06T07:27:11.286166+0000","last_became_peered":"2024-03-06T07:27:11.286166+0000","last_unstale":"2024-03-06T07:27:12.281328+0000","last_undegraded":"2024-03-06T07:27:12.281328+0000","last_fullsized":"2024-03-06T07:27:12.281328+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:27.683733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574149+0000","last_change":"2024-03-06T07:27:11.284747+0000","last_active":"2024-03-06T07:27:12.574149+0000","last_peered":"2024-03-06T07:27:12.574149+0000","last_clean":"2024-03-06T07:27:12.574149+0000","last_became_active":"2024-03-06T07:27:11.284397+0000","last_became_peered":"2024-03-06T07:27:11.284397+0000","last_unstale":"2024-03-06T07:27:12.574149+0000","last_undegraded":"2024-03-06T07:27:12.574149+0000","last_fullsized":"2024-03-06T07:27:12.574149+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:20.907525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136168+0000","last_change":"2024-03-06T07:27:11.270434+0000","last_active":"2024-03-06T07:27:13.136168+0000","last_peered":"2024-03-06T07:27:13.136168+0000","last_clean":"2024-03-06T07:27:13.136168+0000","last_became_active":"2024-03-06T07:27:11.270124+0000","last_became_peered":"2024-03-06T07:27:11.270124+0000","last_unstale":"2024-03-06T07:27:13.136168+0000","last_undegraded":"2024-03-06T07:27:13.136168+0000","last_fullsized":"2024-03-06T07:27:13.136168+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:34.760563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280243+0000","last_change":"2024-03-06T07:27:11.283197+0000","last_active":"2024-03-06T07:27:12.280243+0000","last_peered":"2024-03-06T07:27:12.280243+0000","last_clean":"2024-03-06T07:27:12.280243+0000","last_became_active":"2024-03-06T07:27:11.282920+0000","last_became_peered":"2024-03-06T07:27:11.282920+0000","last_unstale":"2024-03-06T07:27:12.280243+0000","last_undegraded":"2024-03-06T07:27:12.280243+0000","last_fullsized":"2024-03-06T07:27:12.280243+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:01.545346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316052+0000","last_change":"2024-03-06T07:27:11.286485+0000","last_active":"2024-03-06T07:27:13.316052+0000","last_peered":"2024-03-06T07:27:13.316052+0000","last_clean":"2024-03-06T07:27:13.316052+0000","last_became_active":"2024-03-06T07:27:11.286208+0000","last_became_peered":"2024-03-06T07:27:11.286208+0000","last_unstale":"2024-03-06T07:27:13.316052+0000","last_undegraded":"2024-03-06T07:27:13.316052+0000","last_fullsized":"2024-03-06T07:27:13.316052+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:46.649023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266207+0000","last_change":"2024-03-06T07:27:11.276239+0000","last_active":"2024-03-06T07:27:13.266207+0000","last_peered":"2024-03-06T07:27:13.266207+0000","last_clean":"2024-03-06T07:27:13.266207+0000","last_became_active":"2024-03-06T07:27:11.275834+0000","last_became_peered":"2024-03-06T07:27:11.275834+0000","last_unstale":"2024-03-06T07:27:13.266207+0000","last_undegraded":"2024-03-06T07:27:13.266207+0000","last_fullsized":"2024-03-06T07:27:13.266207+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.180576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574038+0000","last_change":"2024-03-06T07:27:11.288114+0000","last_active":"2024-03-06T07:27:12.574038+0000","last_peered":"2024-03-06T07:27:12.574038+0000","last_clean":"2024-03-06T07:27:12.574038+0000","last_became_active":"2024-03-06T07:27:11.287836+0000","last_became_peered":"2024-03-06T07:27:11.287836+0000","last_unstale":"2024-03-06T07:27:12.574038+0000","last_undegraded":"2024-03-06T07:27:12.574038+0000","last_fullsized":"2024-03-06T07:27:12.574038+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:24.979042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136459+0000","last_change":"2024-03-06T07:27:11.284596+0000","last_active":"2024-03-06T07:27:13.136459+0000","last_peered":"2024-03-06T07:27:13.136459+0000","last_clean":"2024-03-06T07:27:13.136459+0000","last_became_active":"2024-03-06T07:27:11.284348+0000","last_became_peered":"2024-03-06T07:27:11.284348+0000","last_unstale":"2024-03-06T07:27:13.136459+0000","last_undegraded":"2024-03-06T07:27:13.136459+0000","last_fullsized":"2024-03-06T07:27:13.136459+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:07.229953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136378+0000","last_change":"2024-03-06T07:27:11.282051+0000","last_active":"2024-03-06T07:27:13.136378+0000","last_peered":"2024-03-06T07:27:13.136378+0000","last_clean":"2024-03-06T07:27:13.136378+0000","last_became_active":"2024-03-06T07:27:11.281607+0000","last_became_peered":"2024-03-06T07:27:11.281607+0000","last_unstale":"2024-03-06T07:27:13.136378+0000","last_undegraded":"2024-03-06T07:27:13.136378+0000","last_fullsized":"2024-03-06T07:27:13.136378+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:03.588248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281552+0000","last_change":"2024-03-06T07:27:11.271046+0000","last_active":"2024-03-06T07:27:12.281552+0000","last_peered":"2024-03-06T07:27:12.281552+0000","last_clean":"2024-03-06T07:27:12.281552+0000","last_became_active":"2024-03-06T07:27:11.267879+0000","last_became_peered":"2024-03-06T07:27:11.267879+0000","last_unstale":"2024-03-06T07:27:12.281552+0000","last_undegraded":"2024-03-06T07:27:12.281552+0000","last_fullsized":"2024-03-06T07:27:12.281552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:13.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136193+0000","last_change":"2024-03-06T07:27:11.274321+0000","last_active":"2024-03-06T07:27:13.136193+0000","last_peered":"2024-03-06T07:27:13.136193+0000","last_clean":"2024-03-06T07:27:13.136193+0000","last_became_active":"2024-03-06T07:27:11.274001+0000","last_became_peered":"2024-03-06T07:27:11.274001+0000","last_unstale":"2024-03-06T07:27:13.136193+0000","last_undegraded":"2024-03-06T07:27:13.136193+0000","last_fullsized":"2024-03-06T07:27:13.136193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:10.749862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574200+0000","last_change":"2024-03-06T07:27:11.289018+0000","last_active":"2024-03-06T07:27:12.574200+0000","last_peered":"2024-03-06T07:27:12.574200+0000","last_clean":"2024-03-06T07:27:12.574200+0000","last_became_active":"2024-03-06T07:27:11.288683+0000","last_became_peered":"2024-03-06T07:27:11.288683+0000","last_unstale":"2024-03-06T07:27:12.574200+0000","last_undegraded":"2024-03-06T07:27:12.574200+0000","last_fullsized":"2024-03-06T07:27:12.574200+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:12.081033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.573891+0000","last_change":"2024-03-06T07:27:11.282681+0000","last_active":"2024-03-06T07:27:12.573891+0000","last_peered":"2024-03-06T07:27:12.573891+0000","last_clean":"2024-03-06T07:27:12.573891+0000","last_became_active":"2024-03-06T07:27:11.282346+0000","last_became_peered":"2024-03-06T07:27:11.282346+0000","last_unstale":"2024-03-06T07:27:12.573891+0000","last_undegraded":"2024-03-06T07:27:12.573891+0000","last_fullsized":"2024-03-06T07:27:12.573891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:07.548742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315706+0000","last_change":"2024-03-06T07:27:11.283783+0000","last_active":"2024-03-06T07:27:13.315706+0000","last_peered":"2024-03-06T07:27:13.315706+0000","last_clean":"2024-03-06T07:27:13.315706+0000","last_became_active":"2024-03-06T07:27:11.283528+0000","last_became_peered":"2024-03-06T07:27:11.283528+0000","last_unstale":"2024-03-06T07:27:13.315706+0000","last_undegraded":"2024-03-06T07:27:13.315706+0000","last_fullsized":"2024-03-06T07:27:13.315706+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:04.257452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135778+0000","last_change":"2024-03-06T07:27:11.286570+0000","last_active":"2024-03-06T07:27:13.135778+0000","last_peered":"2024-03-06T07:27:13.135778+0000","last_clean":"2024-03-06T07:27:13.135778+0000","last_became_active":"2024-03-06T07:27:11.286311+0000","last_became_peered":"2024-03-06T07:27:11.286311+0000","last_unstale":"2024-03-06T07:27:13.135778+0000","last_undegraded":"2024-03-06T07:27:13.135778+0000","last_fullsized":"2024-03-06T07:27:13.135778+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:17.667299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281796+0000","last_change":"2024-03-06T07:27:11.268436+0000","last_active":"2024-03-06T07:27:12.281796+0000","last_peered":"2024-03-06T07:27:12.281796+0000","last_clean":"2024-03-06T07:27:12.281796+0000","last_became_active":"2024-03-06T07:27:11.267761+0000","last_became_peered":"2024-03-06T07:27:11.267761+0000","last_unstale":"2024-03-06T07:27:12.281796+0000","last_undegraded":"2024-03-06T07:27:12.281796+0000","last_fullsized":"2024-03-06T07:27:12.281796+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:21.053605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281828+0000","last_change":"2024-03-06T07:27:11.268216+0000","last_active":"2024-03-06T07:27:12.281828+0000","last_peered":"2024-03-06T07:27:12.281828+0000","last_clean":"2024-03-06T07:27:12.281828+0000","last_became_active":"2024-03-06T07:27:11.267898+0000","last_became_peered":"2024-03-06T07:27:11.267898+0000","last_unstale":"2024-03-06T07:27:12.281828+0000","last_undegraded":"2024-03-06T07:27:12.281828+0000","last_fullsized":"2024-03-06T07:27:12.281828+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:21.053605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574223+0000","last_change":"2024-03-06T07:27:11.288773+0000","last_active":"2024-03-06T07:27:12.574223+0000","last_peered":"2024-03-06T07:27:12.574223+0000","last_clean":"2024-03-06T07:27:12.574223+0000","last_became_active":"2024-03-06T07:27:11.288478+0000","last_became_peered":"2024-03-06T07:27:11.288478+0000","last_unstale":"2024-03-06T07:27:12.574223+0000","last_undegraded":"2024-03-06T07:27:12.574223+0000","last_fullsized":"2024-03-06T07:27:12.574223+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:17.558475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281285+0000","last_change":"2024-03-06T07:27:11.274854+0000","last_active":"2024-03-06T07:27:12.281285+0000","last_peered":"2024-03-06T07:27:12.281285+0000","last_clean":"2024-03-06T07:27:12.281285+0000","last_became_active":"2024-03-06T07:27:11.274529+0000","last_became_peered":"2024-03-06T07:27:11.274529+0000","last_unstale":"2024-03-06T07:27:12.281285+0000","last_undegraded":"2024-03-06T07:27:12.281285+0000","last_fullsized":"2024-03-06T07:27:12.281285+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.737415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281814+0000","last_change":"2024-03-06T07:27:11.281523+0000","last_active":"2024-03-06T07:27:12.281814+0000","last_peered":"2024-03-06T07:27:12.281814+0000","last_clean":"2024-03-06T07:27:12.281814+0000","last_became_active":"2024-03-06T07:27:11.281217+0000","last_became_peered":"2024-03-06T07:27:11.281217+0000","last_unstale":"2024-03-06T07:27:12.281814+0000","last_undegraded":"2024-03-06T07:27:12.281814+0000","last_fullsized":"2024-03-06T07:27:12.281814+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:11.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315546+0000","last_change":"2024-03-06T07:27:11.282691+0000","last_active":"2024-03-06T07:27:13.315546+0000","last_peered":"2024-03-06T07:27:13.315546+0000","last_clean":"2024-03-06T07:27:13.315546+0000","last_became_active":"2024-03-06T07:27:11.282220+0000","last_became_peered":"2024-03-06T07:27:11.282220+0000","last_unstale":"2024-03-06T07:27:13.315546+0000","last_undegraded":"2024-03-06T07:27:13.315546+0000","last_fullsized":"2024-03-06T07:27:13.315546+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:39.081159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280855+0000","last_change":"2024-03-06T07:27:11.274164+0000","last_active":"2024-03-06T07:27:12.280855+0000","last_peered":"2024-03-06T07:27:12.280855+0000","last_clean":"2024-03-06T07:27:12.280855+0000","last_became_active":"2024-03-06T07:27:11.273889+0000","last_became_peered":"2024-03-06T07:27:11.273889+0000","last_unstale":"2024-03-06T07:27:12.280855+0000","last_undegraded":"2024-03-06T07:27:12.280855+0000","last_fullsized":"2024-03-06T07:27:12.280855+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:23.578628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315433+0000","last_change":"2024-03-06T07:27:11.282564+0000","last_active":"2024-03-06T07:27:13.315433+0000","last_peered":"2024-03-06T07:27:13.315433+0000","last_clean":"2024-03-06T07:27:13.315433+0000","last_became_active":"2024-03-06T07:27:11.282279+0000","last_became_peered":"2024-03-06T07:27:11.282279+0000","last_unstale":"2024-03-06T07:27:13.315433+0000","last_undegraded":"2024-03-06T07:27:13.315433+0000","last_fullsized":"2024-03-06T07:27:13.315433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:45.775817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281935+0000","last_change":"2024-03-06T07:27:11.273360+0000","last_active":"2024-03-06T07:27:12.281935+0000","last_peered":"2024-03-06T07:27:12.281935+0000","last_clean":"2024-03-06T07:27:12.281935+0000","last_became_active":"2024-03-06T07:27:11.270163+0000","last_became_peered":"2024-03-06T07:27:11.270163+0000","last_unstale":"2024-03-06T07:27:12.281935+0000","last_undegraded":"2024-03-06T07:27:12.281935+0000","last_fullsized":"2024-03-06T07:27:12.281935+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:09.697689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136445+0000","last_change":"2024-03-06T07:27:11.284060+0000","last_active":"2024-03-06T07:27:13.136445+0000","last_peered":"2024-03-06T07:27:13.136445+0000","last_clean":"2024-03-06T07:27:13.136445+0000","last_became_active":"2024-03-06T07:27:11.283793+0000","last_became_peered":"2024-03-06T07:27:11.283793+0000","last_unstale":"2024-03-06T07:27:13.136445+0000","last_undegraded":"2024-03-06T07:27:13.136445+0000","last_fullsized":"2024-03-06T07:27:13.136445+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:08.243462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316427+0000","last_change":"2024-03-06T07:27:11.290580+0000","last_active":"2024-03-06T07:27:13.316427+0000","last_peered":"2024-03-06T07:27:13.316427+0000","last_clean":"2024-03-06T07:27:13.316427+0000","last_became_active":"2024-03-06T07:27:11.290313+0000","last_became_peered":"2024-03-06T07:27:11.290313+0000","last_unstale":"2024-03-06T07:27:13.316427+0000","last_undegraded":"2024-03-06T07:27:13.316427+0000","last_fullsized":"2024-03-06T07:27:13.316427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:54.818102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315593+0000","last_change":"2024-03-06T07:27:11.289550+0000","last_active":"2024-03-06T07:27:13.315593+0000","last_peered":"2024-03-06T07:27:13.315593+0000","last_clean":"2024-03-06T07:27:13.315593+0000","last_became_active":"2024-03-06T07:27:11.289305+0000","last_became_peered":"2024-03-06T07:27:11.289305+0000","last_unstale":"2024-03-06T07:27:13.315593+0000","last_undegraded":"2024-03-06T07:27:13.315593+0000","last_fullsized":"2024-03-06T07:27:13.315593+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:07.747073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281108+0000","last_change":"2024-03-06T07:27:11.275511+0000","last_active":"2024-03-06T07:27:12.281108+0000","last_peered":"2024-03-06T07:27:12.281108+0000","last_clean":"2024-03-06T07:27:12.281108+0000","last_became_active":"2024-03-06T07:27:11.275231+0000","last_became_peered":"2024-03-06T07:27:11.275231+0000","last_unstale":"2024-03-06T07:27:12.281108+0000","last_undegraded":"2024-03-06T07:27:12.281108+0000","last_fullsized":"2024-03-06T07:27:12.281108+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:51.619182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266300+0000","last_change":"2024-03-06T07:27:11.284090+0000","last_active":"2024-03-06T07:27:13.266300+0000","last_peered":"2024-03-06T07:27:13.266300+0000","last_clean":"2024-03-06T07:27:13.266300+0000","last_became_active":"2024-03-06T07:27:11.283788+0000","last_became_peered":"2024-03-06T07:27:11.283788+0000","last_unstale":"2024-03-06T07:27:13.266300+0000","last_undegraded":"2024-03-06T07:27:13.266300+0000","last_fullsized":"2024-03-06T07:27:13.266300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:59.601907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135872+0000","last_change":"2024-03-06T07:27:11.271614+0000","last_active":"2024-03-06T07:27:13.135872+0000","last_peered":"2024-03-06T07:27:13.135872+0000","last_clean":"2024-03-06T07:27:13.135872+0000","last_became_active":"2024-03-06T07:27:11.271257+0000","last_became_peered":"2024-03-06T07:27:11.271257+0000","last_unstale":"2024-03-06T07:27:13.135872+0000","last_undegraded":"2024-03-06T07:27:13.135872+0000","last_fullsized":"2024-03-06T07:27:13.135872+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:54.325681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315415+0000","last_change":"2024-03-06T07:27:11.286558+0000","last_active":"2024-03-06T07:27:13.315415+0000","last_peered":"2024-03-06T07:27:13.315415+0000","last_clean":"2024-03-06T07:27:13.315415+0000","last_became_active":"2024-03-06T07:27:11.286280+0000","last_became_peered":"2024-03-06T07:27:11.286280+0000","last_unstale":"2024-03-06T07:27:13.315415+0000","last_undegraded":"2024-03-06T07:27:13.315415+0000","last_fullsized":"2024-03-06T07:27:13.315415+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:59.491801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316481+0000","last_change":"2024-03-06T07:27:11.293773+0000","last_active":"2024-03-06T07:27:13.316481+0000","last_peered":"2024-03-06T07:27:13.316481+0000","last_clean":"2024-03-06T07:27:13.316481+0000","last_became_active":"2024-03-06T07:27:11.293529+0000","last_became_peered":"2024-03-06T07:27:11.293529+0000","last_unstale":"2024-03-06T07:27:13.316481+0000","last_undegraded":"2024-03-06T07:27:13.316481+0000","last_fullsized":"2024-03-06T07:27:13.316481+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.180576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266483+0000","last_change":"2024-03-06T07:27:10.271295+0000","last_active":"2024-03-06T07:27:13.266483+0000","last_peered":"2024-03-06T07:27:13.266483+0000","last_clean":"2024-03-06T07:27:13.266483+0000","last_became_active":"2024-03-06T07:27:10.270894+0000","last_became_peered":"2024-03-06T07:27:10.270894+0000","last_unstale":"2024-03-06T07:27:13.266483+0000","last_undegraded":"2024-03-06T07:27:13.266483+0000","last_fullsized":"2024-03-06T07:27:13.266483+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:15.407609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.282303+0000","last_change":"2024-03-06T07:27:11.273041+0000","last_active":"2024-03-06T07:27:12.282303+0000","last_peered":"2024-03-06T07:27:12.282303+0000","last_clean":"2024-03-06T07:27:12.282303+0000","last_became_active":"2024-03-06T07:27:11.272703+0000","last_became_peered":"2024-03-06T07:27:11.272703+0000","last_unstale":"2024-03-06T07:27:12.282303+0000","last_undegraded":"2024-03-06T07:27:12.282303+0000","last_fullsized":"2024-03-06T07:27:12.282303+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:09.697689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574136+0000","last_change":"2024-03-06T07:27:10.279732+0000","last_active":"2024-03-06T07:27:12.574136+0000","last_peered":"2024-03-06T07:27:12.574136+0000","last_clean":"2024-03-06T07:27:12.574136+0000","last_became_active":"2024-03-06T07:27:10.279431+0000","last_became_peered":"2024-03-06T07:27:10.279431+0000","last_unstale":"2024-03-06T07:27:12.574136+0000","last_undegraded":"2024-03-06T07:27:12.574136+0000","last_fullsized":"2024-03-06T07:27:12.574136+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:49.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316366+0000","last_change":"2024-03-06T07:27:11.287232+0000","last_active":"2024-03-06T07:27:13.316366+0000","last_peered":"2024-03-06T07:27:13.316366+0000","last_clean":"2024-03-06T07:27:13.316366+0000","last_became_active":"2024-03-06T07:27:11.286175+0000","last_became_peered":"2024-03-06T07:27:11.286175+0000","last_unstale":"2024-03-06T07:27:13.316366+0000","last_undegraded":"2024-03-06T07:27:13.316366+0000","last_fullsized":"2024-03-06T07:27:13.316366+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:38.385046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280211+0000","last_change":"2024-03-06T07:27:10.278840+0000","last_active":"2024-03-06T07:27:12.280211+0000","last_peered":"2024-03-06T07:27:12.280211+0000","last_clean":"2024-03-06T07:27:12.280211+0000","last_became_active":"2024-03-06T07:27:10.278584+0000","last_became_peered":"2024-03-06T07:27:10.278584+0000","last_unstale":"2024-03-06T07:27:12.280211+0000","last_undegraded":"2024-03-06T07:27:12.280211+0000","last_fullsized":"2024-03-06T07:27:12.280211+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:41.872604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574430+0000","last_change":"2024-03-06T07:27:11.288847+0000","last_active":"2024-03-06T07:27:12.574430+0000","last_peered":"2024-03-06T07:27:12.574430+0000","last_clean":"2024-03-06T07:27:12.574430+0000","last_became_active":"2024-03-06T07:27:11.288570+0000","last_became_peered":"2024-03-06T07:27:11.288570+0000","last_unstale":"2024-03-06T07:27:12.574430+0000","last_undegraded":"2024-03-06T07:27:12.574430+0000","last_fullsized":"2024-03-06T07:27:12.574430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:21.833851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280135+0000","last_change":"2024-03-06T07:27:10.278593+0000","last_active":"2024-03-06T07:27:12.280135+0000","last_peered":"2024-03-06T07:27:12.280135+0000","last_clean":"2024-03-06T07:27:12.280135+0000","last_became_active":"2024-03-06T07:27:10.278332+0000","last_became_peered":"2024-03-06T07:27:10.278332+0000","last_unstale":"2024-03-06T07:27:12.280135+0000","last_undegraded":"2024-03-06T07:27:12.280135+0000","last_fullsized":"2024-03-06T07:27:12.280135+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:53.080914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135897+0000","last_change":"2024-03-06T07:27:11.285918+0000","last_active":"2024-03-06T07:27:13.135897+0000","last_peered":"2024-03-06T07:27:13.135897+0000","last_clean":"2024-03-06T07:27:13.135897+0000","last_became_active":"2024-03-06T07:27:11.285459+0000","last_became_peered":"2024-03-06T07:27:11.285459+0000","last_unstale":"2024-03-06T07:27:13.135897+0000","last_undegraded":"2024-03-06T07:27:13.135897+0000","last_fullsized":"2024-03-06T07:27:13.135897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.750370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266175+0000","last_change":"2024-03-06T07:27:10.271671+0000","last_active":"2024-03-06T07:27:13.266175+0000","last_peered":"2024-03-06T07:27:13.266175+0000","last_clean":"2024-03-06T07:27:13.266175+0000","last_became_active":"2024-03-06T07:27:10.271376+0000","last_became_peered":"2024-03-06T07:27:10.271376+0000","last_unstale":"2024-03-06T07:27:13.266175+0000","last_undegraded":"2024-03-06T07:27:13.266175+0000","last_fullsized":"2024-03-06T07:27:13.266175+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:27.328287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281464+0000","last_change":"2024-03-06T07:27:11.286632+0000","last_active":"2024-03-06T07:27:12.281464+0000","last_peered":"2024-03-06T07:27:12.281464+0000","last_clean":"2024-03-06T07:27:12.281464+0000","last_became_active":"2024-03-06T07:27:11.286250+0000","last_became_peered":"2024-03-06T07:27:11.286250+0000","last_unstale":"2024-03-06T07:27:12.281464+0000","last_undegraded":"2024-03-06T07:27:12.281464+0000","last_fullsized":"2024-03-06T07:27:12.281464+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:35.801091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316041+0000","last_change":"2024-03-06T07:27:10.269066+0000","last_active":"2024-03-06T07:27:13.316041+0000","last_peered":"2024-03-06T07:27:13.316041+0000","last_clean":"2024-03-06T07:27:13.316041+0000","last_became_active":"2024-03-06T07:27:10.268772+0000","last_became_peered":"2024-03-06T07:27:10.268772+0000","last_unstale":"2024-03-06T07:27:13.316041+0000","last_undegraded":"2024-03-06T07:27:13.316041+0000","last_fullsized":"2024-03-06T07:27:13.316041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:58.449261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135847+0000","last_change":"2024-03-06T07:27:11.290987+0000","last_active":"2024-03-06T07:27:13.135847+0000","last_peered":"2024-03-06T07:27:13.135847+0000","last_clean":"2024-03-06T07:27:13.135847+0000","last_became_active":"2024-03-06T07:27:11.290745+0000","last_became_peered":"2024-03-06T07:27:11.290745+0000","last_unstale":"2024-03-06T07:27:13.135847+0000","last_undegraded":"2024-03-06T07:27:13.135847+0000","last_fullsized":"2024-03-06T07:27:13.135847+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:49.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280841+0000","last_change":"2024-03-06T07:27:10.268260+0000","last_active":"2024-03-06T07:27:12.280841+0000","last_peered":"2024-03-06T07:27:12.280841+0000","last_clean":"2024-03-06T07:27:12.280841+0000","last_became_active":"2024-03-06T07:27:10.267951+0000","last_became_peered":"2024-03-06T07:27:10.267951+0000","last_unstale":"2024-03-06T07:27:12.280841+0000","last_undegraded":"2024-03-06T07:27:12.280841+0000","last_fullsized":"2024-03-06T07:27:12.280841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:37.381374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266119+0000","last_change":"2024-03-06T07:27:11.274420+0000","last_active":"2024-03-06T07:27:13.266119+0000","last_peered":"2024-03-06T07:27:13.266119+0000","last_clean":"2024-03-06T07:27:13.266119+0000","last_became_active":"2024-03-06T07:27:11.273786+0000","last_became_peered":"2024-03-06T07:27:11.273786+0000","last_unstale":"2024-03-06T07:27:13.266119+0000","last_undegraded":"2024-03-06T07:27:13.266119+0000","last_fullsized":"2024-03-06T07:27:13.266119+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:44.665708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135940+0000","last_change":"2024-03-06T07:27:10.278526+0000","last_active":"2024-03-06T07:27:13.135940+0000","last_peered":"2024-03-06T07:27:13.135940+0000","last_clean":"2024-03-06T07:27:13.135940+0000","last_became_active":"2024-03-06T07:27:10.277985+0000","last_became_peered":"2024-03-06T07:27:10.277985+0000","last_unstale":"2024-03-06T07:27:13.135940+0000","last_undegraded":"2024-03-06T07:27:13.135940+0000","last_fullsized":"2024-03-06T07:27:13.135940+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:53.219915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266492+0000","last_change":"2024-03-06T07:27:11.270127+0000","last_active":"2024-03-06T07:27:13.266492+0000","last_peered":"2024-03-06T07:27:13.266492+0000","last_clean":"2024-03-06T07:27:13.266492+0000","last_became_active":"2024-03-06T07:27:11.269853+0000","last_became_peered":"2024-03-06T07:27:11.269853+0000","last_unstale":"2024-03-06T07:27:13.266492+0000","last_undegraded":"2024-03-06T07:27:13.266492+0000","last_fullsized":"2024-03-06T07:27:13.266492+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:15.070965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280091+0000","last_change":"2024-03-06T07:27:10.277574+0000","last_active":"2024-03-06T07:27:12.280091+0000","last_peered":"2024-03-06T07:27:12.280091+0000","last_clean":"2024-03-06T07:27:12.280091+0000","last_became_active":"2024-03-06T07:27:10.277328+0000","last_became_peered":"2024-03-06T07:27:10.277328+0000","last_unstale":"2024-03-06T07:27:12.280091+0000","last_undegraded":"2024-03-06T07:27:12.280091+0000","last_fullsized":"2024-03-06T07:27:12.280091+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:19.574698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.282368+0000","last_change":"2024-03-06T07:27:11.274532+0000","last_active":"2024-03-06T07:27:12.282368+0000","last_peered":"2024-03-06T07:27:12.282368+0000","last_clean":"2024-03-06T07:27:12.282368+0000","last_became_active":"2024-03-06T07:27:11.274236+0000","last_became_peered":"2024-03-06T07:27:11.274236+0000","last_unstale":"2024-03-06T07:27:12.282368+0000","last_undegraded":"2024-03-06T07:27:12.282368+0000","last_fullsized":"2024-03-06T07:27:12.282368+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:27.211413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315851+0000","last_change":"2024-03-06T07:27:10.269028+0000","last_active":"2024-03-06T07:27:13.315851+0000","last_peered":"2024-03-06T07:27:13.315851+0000","last_clean":"2024-03-06T07:27:13.315851+0000","last_became_active":"2024-03-06T07:27:10.268733+0000","last_became_peered":"2024-03-06T07:27:10.268733+0000","last_unstale":"2024-03-06T07:27:13.315851+0000","last_undegraded":"2024-03-06T07:27:13.315851+0000","last_fullsized":"2024-03-06T07:27:13.315851+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:26.207740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315625+0000","last_change":"2024-03-06T07:27:11.283211+0000","last_active":"2024-03-06T07:27:13.315625+0000","last_peered":"2024-03-06T07:27:13.315625+0000","last_clean":"2024-03-06T07:27:13.315625+0000","last_became_active":"2024-03-06T07:27:11.282770+0000","last_became_peered":"2024-03-06T07:27:11.282770+0000","last_unstale":"2024-03-06T07:27:13.315625+0000","last_undegraded":"2024-03-06T07:27:13.315625+0000","last_fullsized":"2024-03-06T07:27:13.315625+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.210495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136167+0000","last_change":"2024-03-06T07:27:10.281030+0000","last_active":"2024-03-06T07:27:13.136167+0000","last_peered":"2024-03-06T07:27:13.136167+0000","last_clean":"2024-03-06T07:27:13.136167+0000","last_became_active":"2024-03-06T07:27:10.280767+0000","last_became_peered":"2024-03-06T07:27:10.280767+0000","last_unstale":"2024-03-06T07:27:13.136167+0000","last_undegraded":"2024-03-06T07:27:13.136167+0000","last_fullsized":"2024-03-06T07:27:13.136167+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:38.077486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266360+0000","last_change":"2024-03-06T07:27:11.278923+0000","last_active":"2024-03-06T07:27:13.266360+0000","last_peered":"2024-03-06T07:27:13.266360+0000","last_clean":"2024-03-06T07:27:13.266360+0000","last_became_active":"2024-03-06T07:27:11.278610+0000","last_became_peered":"2024-03-06T07:27:11.278610+0000","last_unstale":"2024-03-06T07:27:13.266360+0000","last_undegraded":"2024-03-06T07:27:13.266360+0000","last_fullsized":"2024-03-06T07:27:13.266360+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:45.688316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136292+0000","last_change":"2024-03-06T07:27:10.278124+0000","last_active":"2024-03-06T07:27:13.136292+0000","last_peered":"2024-03-06T07:27:13.136292+0000","last_clean":"2024-03-06T07:27:13.136292+0000","last_became_active":"2024-03-06T07:27:10.277686+0000","last_became_peered":"2024-03-06T07:27:10.277686+0000","last_unstale":"2024-03-06T07:27:13.136292+0000","last_undegraded":"2024-03-06T07:27:13.136292+0000","last_fullsized":"2024-03-06T07:27:13.136292+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:18.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315680+0000","last_change":"2024-03-06T07:27:11.282476+0000","last_active":"2024-03-06T07:27:13.315680+0000","last_peered":"2024-03-06T07:27:13.315680+0000","last_clean":"2024-03-06T07:27:13.315680+0000","last_became_active":"2024-03-06T07:27:11.281936+0000","last_became_peered":"2024-03-06T07:27:11.281936+0000","last_unstale":"2024-03-06T07:27:13.315680+0000","last_undegraded":"2024-03-06T07:27:13.315680+0000","last_fullsized":"2024-03-06T07:27:13.315680+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:07.548742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316147+0000","last_change":"2024-03-06T07:27:10.272261+0000","last_active":"2024-03-06T07:27:13.316147+0000","last_peered":"2024-03-06T07:27:13.316147+0000","last_clean":"2024-03-06T07:27:13.316147+0000","last_became_active":"2024-03-06T07:27:10.272009+0000","last_became_peered":"2024-03-06T07:27:10.272009+0000","last_unstale":"2024-03-06T07:27:13.316147+0000","last_undegraded":"2024-03-06T07:27:13.316147+0000","last_fullsized":"2024-03-06T07:27:13.316147+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:47.748004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315636+0000","last_change":"2024-03-06T07:27:11.282879+0000","last_active":"2024-03-06T07:27:13.315636+0000","last_peered":"2024-03-06T07:27:13.315636+0000","last_clean":"2024-03-06T07:27:13.315636+0000","last_became_active":"2024-03-06T07:27:11.282259+0000","last_became_peered":"2024-03-06T07:27:11.282259+0000","last_unstale":"2024-03-06T07:27:13.315636+0000","last_undegraded":"2024-03-06T07:27:13.315636+0000","last_fullsized":"2024-03-06T07:27:13.315636+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:54.223587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281444+0000","last_change":"2024-03-06T07:27:10.275499+0000","last_active":"2024-03-06T07:27:12.281444+0000","last_peered":"2024-03-06T07:27:12.281444+0000","last_clean":"2024-03-06T07:27:12.281444+0000","last_became_active":"2024-03-06T07:27:10.275217+0000","last_became_peered":"2024-03-06T07:27:10.275217+0000","last_unstale":"2024-03-06T07:27:12.281444+0000","last_undegraded":"2024-03-06T07:27:12.281444+0000","last_fullsized":"2024-03-06T07:27:12.281444+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:03.349214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315688+0000","last_change":"2024-03-06T07:27:11.284094+0000","last_active":"2024-03-06T07:27:13.315688+0000","last_peered":"2024-03-06T07:27:13.315688+0000","last_clean":"2024-03-06T07:27:13.315688+0000","last_became_active":"2024-03-06T07:27:11.283834+0000","last_became_peered":"2024-03-06T07:27:11.283834+0000","last_unstale":"2024-03-06T07:27:13.315688+0000","last_undegraded":"2024-03-06T07:27:13.315688+0000","last_fullsized":"2024-03-06T07:27:13.315688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.846507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.279966+0000","last_change":"2024-03-06T07:27:10.276787+0000","last_active":"2024-03-06T07:27:12.279966+0000","last_peered":"2024-03-06T07:27:12.279966+0000","last_clean":"2024-03-06T07:27:12.279966+0000","last_became_active":"2024-03-06T07:27:10.276489+0000","last_became_peered":"2024-03-06T07:27:10.276489+0000","last_unstale":"2024-03-06T07:27:12.279966+0000","last_undegraded":"2024-03-06T07:27:12.279966+0000","last_fullsized":"2024-03-06T07:27:12.279966+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:10.272408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280415+0000","last_change":"2024-03-06T07:27:11.289647+0000","last_active":"2024-03-06T07:27:12.280415+0000","last_peered":"2024-03-06T07:27:12.280415+0000","last_clean":"2024-03-06T07:27:12.280415+0000","last_became_active":"2024-03-06T07:27:11.289047+0000","last_became_peered":"2024-03-06T07:27:11.289047+0000","last_unstale":"2024-03-06T07:27:12.280415+0000","last_undegraded":"2024-03-06T07:27:12.280415+0000","last_fullsized":"2024-03-06T07:27:12.280415+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:14.358579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136590+0000","last_change":"2024-03-06T07:27:10.280931+0000","last_active":"2024-03-06T07:27:13.136590+0000","last_peered":"2024-03-06T07:27:13.136590+0000","last_clean":"2024-03-06T07:27:13.136590+0000","last_became_active":"2024-03-06T07:27:10.280631+0000","last_became_peered":"2024-03-06T07:27:10.280631+0000","last_unstale":"2024-03-06T07:27:13.136590+0000","last_undegraded":"2024-03-06T07:27:13.136590+0000","last_fullsized":"2024-03-06T07:27:13.136590+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:47.398871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316471+0000","last_change":"2024-03-06T07:27:11.292080+0000","last_active":"2024-03-06T07:27:13.316471+0000","last_peered":"2024-03-06T07:27:13.316471+0000","last_clean":"2024-03-06T07:27:13.316471+0000","last_became_active":"2024-03-06T07:27:11.291812+0000","last_became_peered":"2024-03-06T07:27:11.291812+0000","last_unstale":"2024-03-06T07:27:13.316471+0000","last_undegraded":"2024-03-06T07:27:13.316471+0000","last_fullsized":"2024-03-06T07:27:13.316471+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:44.665708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280387+0000","last_change":"2024-03-06T07:27:10.275684+0000","last_active":"2024-03-06T07:27:12.280387+0000","last_peered":"2024-03-06T07:27:12.280387+0000","last_clean":"2024-03-06T07:27:12.280387+0000","last_became_active":"2024-03-06T07:27:10.275341+0000","last_became_peered":"2024-03-06T07:27:10.275341+0000","last_unstale":"2024-03-06T07:27:12.280387+0000","last_undegraded":"2024-03-06T07:27:12.280387+0000","last_fullsized":"2024-03-06T07:27:12.280387+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:43.140170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574054+0000","last_change":"2024-03-06T07:27:11.287870+0000","last_active":"2024-03-06T07:27:12.574054+0000","last_peered":"2024-03-06T07:27:12.574054+0000","last_clean":"2024-03-06T07:27:12.574054+0000","last_became_active":"2024-03-06T07:27:11.287618+0000","last_became_peered":"2024-03-06T07:27:11.287618+0000","last_unstale":"2024-03-06T07:27:12.574054+0000","last_undegraded":"2024-03-06T07:27:12.574054+0000","last_fullsized":"2024-03-06T07:27:12.574054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:22.881185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281510+0000","last_change":"2024-03-06T07:27:10.273231+0000","last_active":"2024-03-06T07:27:12.281510+0000","last_peered":"2024-03-06T07:27:12.281510+0000","last_clean":"2024-03-06T07:27:12.281510+0000","last_became_active":"2024-03-06T07:27:10.272878+0000","last_became_peered":"2024-03-06T07:27:10.272878+0000","last_unstale":"2024-03-06T07:27:12.281510+0000","last_undegraded":"2024-03-06T07:27:12.281510+0000","last_fullsized":"2024-03-06T07:27:12.281510+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:38.441829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574444+0000","last_change":"2024-03-06T07:27:11.287627+0000","last_active":"2024-03-06T07:27:12.574444+0000","last_peered":"2024-03-06T07:27:12.574444+0000","last_clean":"2024-03-06T07:27:12.574444+0000","last_became_active":"2024-03-06T07:27:11.287333+0000","last_became_peered":"2024-03-06T07:27:11.287333+0000","last_unstale":"2024-03-06T07:27:12.574444+0000","last_undegraded":"2024-03-06T07:27:12.574444+0000","last_fullsized":"2024-03-06T07:27:12.574444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:17.002710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266183+0000","last_change":"2024-03-06T07:27:10.277902+0000","last_active":"2024-03-06T07:27:13.266183+0000","last_peered":"2024-03-06T07:27:13.266183+0000","last_clean":"2024-03-06T07:27:13.266183+0000","last_became_active":"2024-03-06T07:27:10.277622+0000","last_became_peered":"2024-03-06T07:27:10.277622+0000","last_unstale":"2024-03-06T07:27:13.266183+0000","last_undegraded":"2024-03-06T07:27:13.266183+0000","last_fullsized":"2024-03-06T07:27:13.266183+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:57.522954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.573914+0000","last_change":"2024-03-06T07:27:11.281911+0000","last_active":"2024-03-06T07:27:12.573914+0000","last_peered":"2024-03-06T07:27:12.573914+0000","last_clean":"2024-03-06T07:27:12.573914+0000","last_became_active":"2024-03-06T07:27:11.281531+0000","last_became_peered":"2024-03-06T07:27:11.281531+0000","last_unstale":"2024-03-06T07:27:12.573914+0000","last_undegraded":"2024-03-06T07:27:12.573914+0000","last_fullsized":"2024-03-06T07:27:12.573914+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:10.645129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316216+0000","last_change":"2024-03-06T07:27:10.277544+0000","last_active":"2024-03-06T07:27:13.316216+0000","last_peered":"2024-03-06T07:27:13.316216+0000","last_clean":"2024-03-06T07:27:13.316216+0000","last_became_active":"2024-03-06T07:27:10.276523+0000","last_became_peered":"2024-03-06T07:27:10.276523+0000","last_unstale":"2024-03-06T07:27:13.316216+0000","last_undegraded":"2024-03-06T07:27:13.316216+0000","last_fullsized":"2024-03-06T07:27:13.316216+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:25.067873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135917+0000","last_change":"2024-03-06T07:27:11.273670+0000","last_active":"2024-03-06T07:27:13.135917+0000","last_peered":"2024-03-06T07:27:13.135917+0000","last_clean":"2024-03-06T07:27:13.135917+0000","last_became_active":"2024-03-06T07:27:11.273411+0000","last_became_peered":"2024-03-06T07:27:11.273411+0000","last_unstale":"2024-03-06T07:27:13.135917+0000","last_undegraded":"2024-03-06T07:27:13.135917+0000","last_fullsized":"2024-03-06T07:27:13.135917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:08.581865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280047+0000","last_change":"2024-03-06T07:27:10.278830+0000","last_active":"2024-03-06T07:27:12.280047+0000","last_peered":"2024-03-06T07:27:12.280047+0000","last_clean":"2024-03-06T07:27:12.280047+0000","last_became_active":"2024-03-06T07:27:10.278106+0000","last_became_peered":"2024-03-06T07:27:10.278106+0000","last_unstale":"2024-03-06T07:27:12.280047+0000","last_undegraded":"2024-03-06T07:27:12.280047+0000","last_fullsized":"2024-03-06T07:27:12.280047+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:19.175833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.573960+0000","last_change":"2024-03-06T07:27:11.288954+0000","last_active":"2024-03-06T07:27:12.573960+0000","last_peered":"2024-03-06T07:27:12.573960+0000","last_clean":"2024-03-06T07:27:12.573960+0000","last_became_active":"2024-03-06T07:27:11.288655+0000","last_became_peered":"2024-03-06T07:27:11.288655+0000","last_unstale":"2024-03-06T07:27:12.573960+0000","last_undegraded":"2024-03-06T07:27:12.573960+0000","last_fullsized":"2024-03-06T07:27:12.573960+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:08.886485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315789+0000","last_change":"2024-03-06T07:27:10.271560+0000","last_active":"2024-03-06T07:27:13.315789+0000","last_peered":"2024-03-06T07:27:13.315789+0000","last_clean":"2024-03-06T07:27:13.315789+0000","last_became_active":"2024-03-06T07:27:10.271296+0000","last_became_peered":"2024-03-06T07:27:10.271296+0000","last_unstale":"2024-03-06T07:27:13.315789+0000","last_undegraded":"2024-03-06T07:27:13.315789+0000","last_fullsized":"2024-03-06T07:27:13.315789+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:51.960409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315819+0000","last_change":"2024-03-06T07:27:11.289924+0000","last_active":"2024-03-06T07:27:13.315819+0000","last_peered":"2024-03-06T07:27:13.315819+0000","last_clean":"2024-03-06T07:27:13.315819+0000","last_became_active":"2024-03-06T07:27:11.288953+0000","last_became_peered":"2024-03-06T07:27:11.288953+0000","last_unstale":"2024-03-06T07:27:13.315819+0000","last_undegraded":"2024-03-06T07:27:13.315819+0000","last_fullsized":"2024-03-06T07:27:13.315819+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:48.402544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136012+0000","last_change":"2024-03-06T07:27:10.279864+0000","last_active":"2024-03-06T07:27:13.136012+0000","last_peered":"2024-03-06T07:27:13.136012+0000","last_clean":"2024-03-06T07:27:13.136012+0000","last_became_active":"2024-03-06T07:27:10.279622+0000","last_became_peered":"2024-03-06T07:27:10.279622+0000","last_unstale":"2024-03-06T07:27:13.136012+0000","last_undegraded":"2024-03-06T07:27:13.136012+0000","last_fullsized":"2024-03-06T07:27:13.136012+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:00.842834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281970+0000","last_change":"2024-03-06T07:27:11.284561+0000","last_active":"2024-03-06T07:27:12.281970+0000","last_peered":"2024-03-06T07:27:12.281970+0000","last_clean":"2024-03-06T07:27:12.281970+0000","last_became_active":"2024-03-06T07:27:11.284292+0000","last_became_peered":"2024-03-06T07:27:11.284292+0000","last_unstale":"2024-03-06T07:27:12.281970+0000","last_undegraded":"2024-03-06T07:27:12.281970+0000","last_fullsized":"2024-03-06T07:27:12.281970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:11.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315882+0000","last_change":"2024-03-06T07:27:10.273788+0000","last_active":"2024-03-06T07:27:13.315882+0000","last_peered":"2024-03-06T07:27:13.315882+0000","last_clean":"2024-03-06T07:27:13.315882+0000","last_became_active":"2024-03-06T07:27:10.273542+0000","last_became_peered":"2024-03-06T07:27:10.273542+0000","last_unstale":"2024-03-06T07:27:13.315882+0000","last_undegraded":"2024-03-06T07:27:13.315882+0000","last_fullsized":"2024-03-06T07:27:13.315882+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:08.694016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136542+0000","last_change":"2024-03-06T07:27:11.271394+0000","last_active":"2024-03-06T07:27:13.136542+0000","last_peered":"2024-03-06T07:27:13.136542+0000","last_clean":"2024-03-06T07:27:13.136542+0000","last_became_active":"2024-03-06T07:27:11.270251+0000","last_became_peered":"2024-03-06T07:27:11.270251+0000","last_unstale":"2024-03-06T07:27:13.136542+0000","last_undegraded":"2024-03-06T07:27:13.136542+0000","last_fullsized":"2024-03-06T07:27:13.136542+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:59.836515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.282096+0000","last_change":"2024-03-06T07:27:10.269615+0000","last_active":"2024-03-06T07:27:12.282096+0000","last_peered":"2024-03-06T07:27:12.282096+0000","last_clean":"2024-03-06T07:27:12.282096+0000","last_became_active":"2024-03-06T07:27:10.269331+0000","last_became_peered":"2024-03-06T07:27:10.269331+0000","last_unstale":"2024-03-06T07:27:12.282096+0000","last_undegraded":"2024-03-06T07:27:12.282096+0000","last_fullsized":"2024-03-06T07:27:12.282096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:22.574956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136276+0000","last_change":"2024-03-06T07:27:11.289894+0000","last_active":"2024-03-06T07:27:13.136276+0000","last_peered":"2024-03-06T07:27:13.136276+0000","last_clean":"2024-03-06T07:27:13.136276+0000","last_became_active":"2024-03-06T07:27:11.289625+0000","last_became_peered":"2024-03-06T07:27:11.289625+0000","last_unstale":"2024-03-06T07:27:13.136276+0000","last_undegraded":"2024-03-06T07:27:13.136276+0000","last_fullsized":"2024-03-06T07:27:13.136276+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:58.272911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136197+0000","last_change":"2024-03-06T07:27:10.273864+0000","last_active":"2024-03-06T07:27:13.136197+0000","last_peered":"2024-03-06T07:27:13.136197+0000","last_clean":"2024-03-06T07:27:13.136197+0000","last_became_active":"2024-03-06T07:27:10.273611+0000","last_became_peered":"2024-03-06T07:27:10.273611+0000","last_unstale":"2024-03-06T07:27:13.136197+0000","last_undegraded":"2024-03-06T07:27:13.136197+0000","last_fullsized":"2024-03-06T07:27:13.136197+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:06.743401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281413+0000","last_change":"2024-03-06T07:27:11.274612+0000","last_active":"2024-03-06T07:27:12.281413+0000","last_peered":"2024-03-06T07:27:12.281413+0000","last_clean":"2024-03-06T07:27:12.281413+0000","last_became_active":"2024-03-06T07:27:11.274304+0000","last_became_peered":"2024-03-06T07:27:11.274304+0000","last_unstale":"2024-03-06T07:27:12.281413+0000","last_undegraded":"2024-03-06T07:27:12.281413+0000","last_fullsized":"2024-03-06T07:27:12.281413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:51.619182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315690+0000","last_change":"2024-03-06T07:27:10.271689+0000","last_active":"2024-03-06T07:27:13.315690+0000","last_peered":"2024-03-06T07:27:13.315690+0000","last_clean":"2024-03-06T07:27:13.315690+0000","last_became_active":"2024-03-06T07:27:10.271421+0000","last_became_peered":"2024-03-06T07:27:10.271421+0000","last_unstale":"2024-03-06T07:27:13.315690+0000","last_undegraded":"2024-03-06T07:27:13.315690+0000","last_fullsized":"2024-03-06T07:27:13.315690+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:44.962932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280921+0000","last_change":"2024-03-06T07:27:11.276400+0000","last_active":"2024-03-06T07:27:12.280921+0000","last_peered":"2024-03-06T07:27:12.280921+0000","last_clean":"2024-03-06T07:27:12.280921+0000","last_became_active":"2024-03-06T07:27:11.276124+0000","last_became_peered":"2024-03-06T07:27:11.276124+0000","last_unstale":"2024-03-06T07:27:12.280921+0000","last_undegraded":"2024-03-06T07:27:12.280921+0000","last_fullsized":"2024-03-06T07:27:12.280921+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:19.052096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266351+0000","last_change":"2024-03-06T07:27:10.268891+0000","last_active":"2024-03-06T07:27:13.266351+0000","last_peered":"2024-03-06T07:27:13.266351+0000","last_clean":"2024-03-06T07:27:13.266351+0000","last_became_active":"2024-03-06T07:27:10.268536+0000","last_became_peered":"2024-03-06T07:27:10.268536+0000","last_unstale":"2024-03-06T07:27:13.266351+0000","last_undegraded":"2024-03-06T07:27:13.266351+0000","last_fullsized":"2024-03-06T07:27:13.266351+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:45.518696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281623+0000","last_change":"2024-03-06T07:27:11.271113+0000","last_active":"2024-03-06T07:27:12.281623+0000","last_peered":"2024-03-06T07:27:12.281623+0000","last_clean":"2024-03-06T07:27:12.281623+0000","last_became_active":"2024-03-06T07:27:11.270835+0000","last_became_peered":"2024-03-06T07:27:11.270835+0000","last_unstale":"2024-03-06T07:27:12.281623+0000","last_undegraded":"2024-03-06T07:27:12.281623+0000","last_fullsized":"2024-03-06T07:27:12.281623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:06.181936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280991+0000","last_change":"2024-03-06T07:27:10.272914+0000","last_active":"2024-03-06T07:27:12.280991+0000","last_peered":"2024-03-06T07:27:12.280991+0000","last_clean":"2024-03-06T07:27:12.280991+0000","last_became_active":"2024-03-06T07:27:10.272635+0000","last_became_peered":"2024-03-06T07:27:10.272635+0000","last_unstale":"2024-03-06T07:27:12.280991+0000","last_undegraded":"2024-03-06T07:27:12.280991+0000","last_fullsized":"2024-03-06T07:27:12.280991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:45.645350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574187+0000","last_change":"2024-03-06T07:27:11.283189+0000","last_active":"2024-03-06T07:27:12.574187+0000","last_peered":"2024-03-06T07:27:12.574187+0000","last_clean":"2024-03-06T07:27:12.574187+0000","last_became_active":"2024-03-06T07:27:11.282804+0000","last_became_peered":"2024-03-06T07:27:11.282804+0000","last_unstale":"2024-03-06T07:27:12.574187+0000","last_undegraded":"2024-03-06T07:27:12.574187+0000","last_fullsized":"2024-03-06T07:27:12.574187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.192766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.279981+0000","last_change":"2024-03-06T07:27:10.276957+0000","last_active":"2024-03-06T07:27:12.279981+0000","last_peered":"2024-03-06T07:27:12.279981+0000","last_clean":"2024-03-06T07:27:12.279981+0000","last_became_active":"2024-03-06T07:27:10.276707+0000","last_became_peered":"2024-03-06T07:27:10.276707+0000","last_unstale":"2024-03-06T07:27:12.279981+0000","last_undegraded":"2024-03-06T07:27:12.279981+0000","last_fullsized":"2024-03-06T07:27:12.279981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:41.377958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135901+0000","last_change":"2024-03-06T07:27:11.290890+0000","last_active":"2024-03-06T07:27:13.135901+0000","last_peered":"2024-03-06T07:27:13.135901+0000","last_clean":"2024-03-06T07:27:13.135901+0000","last_became_active":"2024-03-06T07:27:11.290643+0000","last_became_peered":"2024-03-06T07:27:11.290643+0000","last_unstale":"2024-03-06T07:27:13.135901+0000","last_undegraded":"2024-03-06T07:27:13.135901+0000","last_fullsized":"2024-03-06T07:27:13.135901+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:35.227519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136038+0000","last_change":"2024-03-06T07:27:10.275390+0000","last_active":"2024-03-06T07:27:13.136038+0000","last_peered":"2024-03-06T07:27:13.136038+0000","last_clean":"2024-03-06T07:27:13.136038+0000","last_became_active":"2024-03-06T07:27:10.274903+0000","last_became_peered":"2024-03-06T07:27:10.274903+0000","last_unstale":"2024-03-06T07:27:13.136038+0000","last_undegraded":"2024-03-06T07:27:13.136038+0000","last_fullsized":"2024-03-06T07:27:13.136038+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:06.545069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315472+0000","last_change":"2024-03-06T07:27:11.281365+0000","last_active":"2024-03-06T07:27:13.315472+0000","last_peered":"2024-03-06T07:27:13.315472+0000","last_clean":"2024-03-06T07:27:13.315472+0000","last_became_active":"2024-03-06T07:27:11.281108+0000","last_became_peered":"2024-03-06T07:27:11.281108+0000","last_unstale":"2024-03-06T07:27:13.315472+0000","last_undegraded":"2024-03-06T07:27:13.315472+0000","last_fullsized":"2024-03-06T07:27:13.315472+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.944500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266252+0000","last_change":"2024-03-06T07:27:10.277891+0000","last_active":"2024-03-06T07:27:13.266252+0000","last_peered":"2024-03-06T07:27:13.266252+0000","last_clean":"2024-03-06T07:27:13.266252+0000","last_became_active":"2024-03-06T07:27:10.277582+0000","last_became_peered":"2024-03-06T07:27:10.277582+0000","last_unstale":"2024-03-06T07:27:13.266252+0000","last_undegraded":"2024-03-06T07:27:13.266252+0000","last_fullsized":"2024-03-06T07:27:13.266252+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:59.897065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574391+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:27:12.574391+0000","last_peered":"2024-03-06T07:27:12.574391+0000","last_clean":"2024-03-06T07:27:12.574391+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:27:12.574391+0000","last_undegraded":"2024-03-06T07:27:12.574391+0000","last_fullsized":"2024-03-06T07:27:12.574391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":1169875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1347584,"data_stored":578961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1351680,"data_stored":579495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1343488,"data_stored":578891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574919,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40828,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331012,"statfs":{"total":96636764160,"available":96594956288,"internally_reserved":0,"allocated":1359872,"data_stored":580573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574919,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1343488,"data_stored":578891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1351680,"data_stored":579015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574920,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41412,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330428,"statfs":{"total":96636764160,"available":96594358272,"internally_reserved":0,"allocated":1957888,"data_stored":1170983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:16.542 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:27:16.540+0000 7fab949fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:18.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:18.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 Processor -- start 2024-03-06T07:27:18.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 -- start start 2024-03-06T07:27:18.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7bf41515e0 0x7f7bf40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 0x7f7bf4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf415b8b0 con 0x7f7bf40cc340 2024-03-06T07:27:18.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf4070a70 con 0x7f7bf41515e0 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.860+0000 7f7bf9a39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4070d50 con 0x7f7bf415b430 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bebfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 0x7f7bf4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7bf41515e0 0x7f7bf40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7bf41515e0 0x7f7bf40cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:52586/0 (socket says 172.21.15.122:52586) 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bebfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 0x7f7bf4071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:46628/0 (socket says 172.21.15.122:46628) 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 -- 172.21.15.122:0/444359051 learned_addr learned my addr 172.21.15.122:0/444359051 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 -- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 msgr2=0x7f7bf4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 --2- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 0x7f7bf4071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 -- 172.21.15.122:0/444359051 >> v1:172.21.15.169:6789/0 conn(0x7f7bf40cc340 legacy=0x7f7bf40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bf2ffd640 1 -- 172.21.15.122:0/444359051 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4071030 con 0x7f7bf41515e0 2024-03-06T07:27:18.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.861+0000 7f7bebfff640 1 --2- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 0x7f7bf4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.862+0000 7f7bf2ffd640 1 --2- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40cce40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7936410b4cf97467 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.862+0000 7f7bf27fc640 1 -- 172.21.15.122:0/444359051 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4034070 con 0x7f7bf41515e0 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.862+0000 7f7bf27fc640 1 -- 172.21.15.122:0/444359051 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be4030840 con 0x7f7bf41515e0 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.862+0000 7f7bf27fc640 1 -- 172.21.15.122:0/444359051 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4030c10 con 0x7f7bf41515e0 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 -- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 msgr2=0x7f7bf40cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40cce40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402b980 comp rx=0 tx=0).stop 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 -- 172.21.15.122:0/444359051 shutdown_connections 2024-03-06T07:27:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf415b430 0x7f7bf4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/444359051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40cce40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 -- 172.21.15.122:0/444359051 >> 172.21.15.122:0/444359051 conn(0x7f7bf40cf8a0 msgr2=0x7f7bf40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 -- 172.21.15.122:0/444359051 shutdown_connections 2024-03-06T07:27:18.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.863+0000 7f7bf9a39640 1 -- 172.21.15.122:0/444359051 wait complete. 2024-03-06T07:27:18.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.864+0000 7f7bf9a39640 1 Processor -- start 2024-03-06T07:27:18.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.864+0000 7f7bf9a39640 1 -- start start 2024-03-06T07:27:18.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf40cc340 0x7f7bf41423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 0x7f7bf4153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4071ef0 con 0x7f7bf415b430 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4071d70 con 0x7f7bf41515e0 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40cc770 con 0x7f7bf40cc340 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 0x7f7bf4153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 0x7f7bf4153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51866/0 (socket says 172.21.15.122:51866) 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 -- 172.21.15.122:0/264371547 learned_addr learned my addr 172.21.15.122:0/264371547 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bebfff640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf2ffd640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf40cc340 0x7f7bf41423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 -- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf40cc340 msgr2=0x7f7bf41423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf40cc340 0x7f7bf41423b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 -- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 msgr2=0x7f7bf40ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 -- 172.21.15.122:0/264371547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf41428f0 con 0x7f7bf415b430 2024-03-06T07:27:18.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bebfff640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf37fe640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 0x7f7bf4153fc0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c7d0 tx=0x7f7bec00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec003b70 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf4142b90 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec003d30 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00aa90 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.865+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf4143150 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.866+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf4068840 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.867+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bec00ac30 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.867+0000 7f7bf0ff9640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7bd406b4d0 0x7f7bd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.867+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bec08a110 con 0x7f7bf415b430 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.867+0000 7f7bf2ffd640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7bd406b4d0 0x7f7bd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.868+0000 7f7bf2ffd640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7bd406b4d0 0x7f7bd406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7be4036000 tx=0x7f7be403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:18.872+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bec058fa0 con 0x7f7bf415b430 2024-03-06T07:27:19.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.081+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7bf404f330 con 0x7f7bd406b4d0 2024-03-06T07:27:19.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.087+0000 7f7bf0ff9640 1 -- 172.21.15.122:0/264371547 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f7bf404f330 con 0x7f7bd406b4d0 2024-03-06T07:27:19.091 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:19.092 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7bd406b4d0 msgr2=0x7f7bd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7bd406b4d0 0x7f7bd406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7be4036000 tx=0x7f7be403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 msgr2=0x7f7bf4153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 0x7f7bf4153fc0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c7d0 tx=0x7f7bec00cc90 comp rx=0 tx=0).stop 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 shutdown_connections 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7bd406b4d0 0x7f7bd406d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf415b430 0x7f7bf4153fc0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf41515e0 0x7f7bf40ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 --2- 172.21.15.122:0/264371547 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf40cc340 0x7f7bf41423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 >> 172.21.15.122:0/264371547 conn(0x7f7bf40cf8a0 msgr2=0x7f7bf4150130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 shutdown_connections 2024-03-06T07:27:19.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.090+0000 7f7bf9a39640 1 -- 172.21.15.122:0/264371547 wait complete. 2024-03-06T07:27:19.103 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-03-06T07:27:17.396664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323584,"kb_used_data":11720,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651136,"statfs":{"total":773094113280,"available":772762763264,"internally_reserved":0,"allocated":12001280,"data_stored":5816684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.137757"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136473+0000","last_change":"2024-03-06T07:27:11.270188+0000","last_active":"2024-03-06T07:27:13.136473+0000","last_peered":"2024-03-06T07:27:13.136473+0000","last_clean":"2024-03-06T07:27:13.136473+0000","last_became_active":"2024-03-06T07:27:11.269882+0000","last_became_peered":"2024-03-06T07:27:11.269882+0000","last_unstale":"2024-03-06T07:27:13.136473+0000","last_undegraded":"2024-03-06T07:27:13.136473+0000","last_fullsized":"2024-03-06T07:27:13.136473+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:47.558402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281328+0000","last_change":"2024-03-06T07:27:11.286409+0000","last_active":"2024-03-06T07:27:12.281328+0000","last_peered":"2024-03-06T07:27:12.281328+0000","last_clean":"2024-03-06T07:27:12.281328+0000","last_became_active":"2024-03-06T07:27:11.286166+0000","last_became_peered":"2024-03-06T07:27:11.286166+0000","last_unstale":"2024-03-06T07:27:12.281328+0000","last_undegraded":"2024-03-06T07:27:12.281328+0000","last_fullsized":"2024-03-06T07:27:12.281328+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:27.683733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574149+0000","last_change":"2024-03-06T07:27:11.284747+0000","last_active":"2024-03-06T07:27:12.574149+0000","last_peered":"2024-03-06T07:27:12.574149+0000","last_clean":"2024-03-06T07:27:12.574149+0000","last_became_active":"2024-03-06T07:27:11.284397+0000","last_became_peered":"2024-03-06T07:27:11.284397+0000","last_unstale":"2024-03-06T07:27:12.574149+0000","last_undegraded":"2024-03-06T07:27:12.574149+0000","last_fullsized":"2024-03-06T07:27:12.574149+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:20.907525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136168+0000","last_change":"2024-03-06T07:27:11.270434+0000","last_active":"2024-03-06T07:27:13.136168+0000","last_peered":"2024-03-06T07:27:13.136168+0000","last_clean":"2024-03-06T07:27:13.136168+0000","last_became_active":"2024-03-06T07:27:11.270124+0000","last_became_peered":"2024-03-06T07:27:11.270124+0000","last_unstale":"2024-03-06T07:27:13.136168+0000","last_undegraded":"2024-03-06T07:27:13.136168+0000","last_fullsized":"2024-03-06T07:27:13.136168+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:34.760563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280243+0000","last_change":"2024-03-06T07:27:11.283197+0000","last_active":"2024-03-06T07:27:12.280243+0000","last_peered":"2024-03-06T07:27:12.280243+0000","last_clean":"2024-03-06T07:27:12.280243+0000","last_became_active":"2024-03-06T07:27:11.282920+0000","last_became_peered":"2024-03-06T07:27:11.282920+0000","last_unstale":"2024-03-06T07:27:12.280243+0000","last_undegraded":"2024-03-06T07:27:12.280243+0000","last_fullsized":"2024-03-06T07:27:12.280243+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:01.545346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316052+0000","last_change":"2024-03-06T07:27:11.286485+0000","last_active":"2024-03-06T07:27:13.316052+0000","last_peered":"2024-03-06T07:27:13.316052+0000","last_clean":"2024-03-06T07:27:13.316052+0000","last_became_active":"2024-03-06T07:27:11.286208+0000","last_became_peered":"2024-03-06T07:27:11.286208+0000","last_unstale":"2024-03-06T07:27:13.316052+0000","last_undegraded":"2024-03-06T07:27:13.316052+0000","last_fullsized":"2024-03-06T07:27:13.316052+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:46.649023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266207+0000","last_change":"2024-03-06T07:27:11.276239+0000","last_active":"2024-03-06T07:27:13.266207+0000","last_peered":"2024-03-06T07:27:13.266207+0000","last_clean":"2024-03-06T07:27:13.266207+0000","last_became_active":"2024-03-06T07:27:11.275834+0000","last_became_peered":"2024-03-06T07:27:11.275834+0000","last_unstale":"2024-03-06T07:27:13.266207+0000","last_undegraded":"2024-03-06T07:27:13.266207+0000","last_fullsized":"2024-03-06T07:27:13.266207+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.180576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574038+0000","last_change":"2024-03-06T07:27:11.288114+0000","last_active":"2024-03-06T07:27:12.574038+0000","last_peered":"2024-03-06T07:27:12.574038+0000","last_clean":"2024-03-06T07:27:12.574038+0000","last_became_active":"2024-03-06T07:27:11.287836+0000","last_became_peered":"2024-03-06T07:27:11.287836+0000","last_unstale":"2024-03-06T07:27:12.574038+0000","last_undegraded":"2024-03-06T07:27:12.574038+0000","last_fullsized":"2024-03-06T07:27:12.574038+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:24.979042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136459+0000","last_change":"2024-03-06T07:27:11.284596+0000","last_active":"2024-03-06T07:27:13.136459+0000","last_peered":"2024-03-06T07:27:13.136459+0000","last_clean":"2024-03-06T07:27:13.136459+0000","last_became_active":"2024-03-06T07:27:11.284348+0000","last_became_peered":"2024-03-06T07:27:11.284348+0000","last_unstale":"2024-03-06T07:27:13.136459+0000","last_undegraded":"2024-03-06T07:27:13.136459+0000","last_fullsized":"2024-03-06T07:27:13.136459+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:07.229953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136378+0000","last_change":"2024-03-06T07:27:11.282051+0000","last_active":"2024-03-06T07:27:13.136378+0000","last_peered":"2024-03-06T07:27:13.136378+0000","last_clean":"2024-03-06T07:27:13.136378+0000","last_became_active":"2024-03-06T07:27:11.281607+0000","last_became_peered":"2024-03-06T07:27:11.281607+0000","last_unstale":"2024-03-06T07:27:13.136378+0000","last_undegraded":"2024-03-06T07:27:13.136378+0000","last_fullsized":"2024-03-06T07:27:13.136378+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:03.588248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281552+0000","last_change":"2024-03-06T07:27:11.271046+0000","last_active":"2024-03-06T07:27:12.281552+0000","last_peered":"2024-03-06T07:27:12.281552+0000","last_clean":"2024-03-06T07:27:12.281552+0000","last_became_active":"2024-03-06T07:27:11.267879+0000","last_became_peered":"2024-03-06T07:27:11.267879+0000","last_unstale":"2024-03-06T07:27:12.281552+0000","last_undegraded":"2024-03-06T07:27:12.281552+0000","last_fullsized":"2024-03-06T07:27:12.281552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:13.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136193+0000","last_change":"2024-03-06T07:27:11.274321+0000","last_active":"2024-03-06T07:27:13.136193+0000","last_peered":"2024-03-06T07:27:13.136193+0000","last_clean":"2024-03-06T07:27:13.136193+0000","last_became_active":"2024-03-06T07:27:11.274001+0000","last_became_peered":"2024-03-06T07:27:11.274001+0000","last_unstale":"2024-03-06T07:27:13.136193+0000","last_undegraded":"2024-03-06T07:27:13.136193+0000","last_fullsized":"2024-03-06T07:27:13.136193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:10.749862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574200+0000","last_change":"2024-03-06T07:27:11.289018+0000","last_active":"2024-03-06T07:27:12.574200+0000","last_peered":"2024-03-06T07:27:12.574200+0000","last_clean":"2024-03-06T07:27:12.574200+0000","last_became_active":"2024-03-06T07:27:11.288683+0000","last_became_peered":"2024-03-06T07:27:11.288683+0000","last_unstale":"2024-03-06T07:27:12.574200+0000","last_undegraded":"2024-03-06T07:27:12.574200+0000","last_fullsized":"2024-03-06T07:27:12.574200+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:12.081033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.573891+0000","last_change":"2024-03-06T07:27:11.282681+0000","last_active":"2024-03-06T07:27:12.573891+0000","last_peered":"2024-03-06T07:27:12.573891+0000","last_clean":"2024-03-06T07:27:12.573891+0000","last_became_active":"2024-03-06T07:27:11.282346+0000","last_became_peered":"2024-03-06T07:27:11.282346+0000","last_unstale":"2024-03-06T07:27:12.573891+0000","last_undegraded":"2024-03-06T07:27:12.573891+0000","last_fullsized":"2024-03-06T07:27:12.573891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:07.548742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315706+0000","last_change":"2024-03-06T07:27:11.283783+0000","last_active":"2024-03-06T07:27:13.315706+0000","last_peered":"2024-03-06T07:27:13.315706+0000","last_clean":"2024-03-06T07:27:13.315706+0000","last_became_active":"2024-03-06T07:27:11.283528+0000","last_became_peered":"2024-03-06T07:27:11.283528+0000","last_unstale":"2024-03-06T07:27:13.315706+0000","last_undegraded":"2024-03-06T07:27:13.315706+0000","last_fullsized":"2024-03-06T07:27:13.315706+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:04.257452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135778+0000","last_change":"2024-03-06T07:27:11.286570+0000","last_active":"2024-03-06T07:27:13.135778+0000","last_peered":"2024-03-06T07:27:13.135778+0000","last_clean":"2024-03-06T07:27:13.135778+0000","last_became_active":"2024-03-06T07:27:11.286311+0000","last_became_peered":"2024-03-06T07:27:11.286311+0000","last_unstale":"2024-03-06T07:27:13.135778+0000","last_undegraded":"2024-03-06T07:27:13.135778+0000","last_fullsized":"2024-03-06T07:27:13.135778+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:17.667299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281796+0000","last_change":"2024-03-06T07:27:11.268436+0000","last_active":"2024-03-06T07:27:12.281796+0000","last_peered":"2024-03-06T07:27:12.281796+0000","last_clean":"2024-03-06T07:27:12.281796+0000","last_became_active":"2024-03-06T07:27:11.267761+0000","last_became_peered":"2024-03-06T07:27:11.267761+0000","last_unstale":"2024-03-06T07:27:12.281796+0000","last_undegraded":"2024-03-06T07:27:12.281796+0000","last_fullsized":"2024-03-06T07:27:12.281796+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:21.053605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281828+0000","last_change":"2024-03-06T07:27:11.268216+0000","last_active":"2024-03-06T07:27:12.281828+0000","last_peered":"2024-03-06T07:27:12.281828+0000","last_clean":"2024-03-06T07:27:12.281828+0000","last_became_active":"2024-03-06T07:27:11.267898+0000","last_became_peered":"2024-03-06T07:27:11.267898+0000","last_unstale":"2024-03-06T07:27:12.281828+0000","last_undegraded":"2024-03-06T07:27:12.281828+0000","last_fullsized":"2024-03-06T07:27:12.281828+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:21.053605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574223+0000","last_change":"2024-03-06T07:27:11.288773+0000","last_active":"2024-03-06T07:27:12.574223+0000","last_peered":"2024-03-06T07:27:12.574223+0000","last_clean":"2024-03-06T07:27:12.574223+0000","last_became_active":"2024-03-06T07:27:11.288478+0000","last_became_peered":"2024-03-06T07:27:11.288478+0000","last_unstale":"2024-03-06T07:27:12.574223+0000","last_undegraded":"2024-03-06T07:27:12.574223+0000","last_fullsized":"2024-03-06T07:27:12.574223+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:17.558475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281285+0000","last_change":"2024-03-06T07:27:11.274854+0000","last_active":"2024-03-06T07:27:12.281285+0000","last_peered":"2024-03-06T07:27:12.281285+0000","last_clean":"2024-03-06T07:27:12.281285+0000","last_became_active":"2024-03-06T07:27:11.274529+0000","last_became_peered":"2024-03-06T07:27:11.274529+0000","last_unstale":"2024-03-06T07:27:12.281285+0000","last_undegraded":"2024-03-06T07:27:12.281285+0000","last_fullsized":"2024-03-06T07:27:12.281285+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.737415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281814+0000","last_change":"2024-03-06T07:27:11.281523+0000","last_active":"2024-03-06T07:27:12.281814+0000","last_peered":"2024-03-06T07:27:12.281814+0000","last_clean":"2024-03-06T07:27:12.281814+0000","last_became_active":"2024-03-06T07:27:11.281217+0000","last_became_peered":"2024-03-06T07:27:11.281217+0000","last_unstale":"2024-03-06T07:27:12.281814+0000","last_undegraded":"2024-03-06T07:27:12.281814+0000","last_fullsized":"2024-03-06T07:27:12.281814+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:11.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315546+0000","last_change":"2024-03-06T07:27:11.282691+0000","last_active":"2024-03-06T07:27:13.315546+0000","last_peered":"2024-03-06T07:27:13.315546+0000","last_clean":"2024-03-06T07:27:13.315546+0000","last_became_active":"2024-03-06T07:27:11.282220+0000","last_became_peered":"2024-03-06T07:27:11.282220+0000","last_unstale":"2024-03-06T07:27:13.315546+0000","last_undegraded":"2024-03-06T07:27:13.315546+0000","last_fullsized":"2024-03-06T07:27:13.315546+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:39.081159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280855+0000","last_change":"2024-03-06T07:27:11.274164+0000","last_active":"2024-03-06T07:27:12.280855+0000","last_peered":"2024-03-06T07:27:12.280855+0000","last_clean":"2024-03-06T07:27:12.280855+0000","last_became_active":"2024-03-06T07:27:11.273889+0000","last_became_peered":"2024-03-06T07:27:11.273889+0000","last_unstale":"2024-03-06T07:27:12.280855+0000","last_undegraded":"2024-03-06T07:27:12.280855+0000","last_fullsized":"2024-03-06T07:27:12.280855+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:23.578628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315433+0000","last_change":"2024-03-06T07:27:11.282564+0000","last_active":"2024-03-06T07:27:13.315433+0000","last_peered":"2024-03-06T07:27:13.315433+0000","last_clean":"2024-03-06T07:27:13.315433+0000","last_became_active":"2024-03-06T07:27:11.282279+0000","last_became_peered":"2024-03-06T07:27:11.282279+0000","last_unstale":"2024-03-06T07:27:13.315433+0000","last_undegraded":"2024-03-06T07:27:13.315433+0000","last_fullsized":"2024-03-06T07:27:13.315433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:45.775817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281935+0000","last_change":"2024-03-06T07:27:11.273360+0000","last_active":"2024-03-06T07:27:12.281935+0000","last_peered":"2024-03-06T07:27:12.281935+0000","last_clean":"2024-03-06T07:27:12.281935+0000","last_became_active":"2024-03-06T07:27:11.270163+0000","last_became_peered":"2024-03-06T07:27:11.270163+0000","last_unstale":"2024-03-06T07:27:12.281935+0000","last_undegraded":"2024-03-06T07:27:12.281935+0000","last_fullsized":"2024-03-06T07:27:12.281935+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:09.697689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136445+0000","last_change":"2024-03-06T07:27:11.284060+0000","last_active":"2024-03-06T07:27:13.136445+0000","last_peered":"2024-03-06T07:27:13.136445+0000","last_clean":"2024-03-06T07:27:13.136445+0000","last_became_active":"2024-03-06T07:27:11.283793+0000","last_became_peered":"2024-03-06T07:27:11.283793+0000","last_unstale":"2024-03-06T07:27:13.136445+0000","last_undegraded":"2024-03-06T07:27:13.136445+0000","last_fullsized":"2024-03-06T07:27:13.136445+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:08.243462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316427+0000","last_change":"2024-03-06T07:27:11.290580+0000","last_active":"2024-03-06T07:27:13.316427+0000","last_peered":"2024-03-06T07:27:13.316427+0000","last_clean":"2024-03-06T07:27:13.316427+0000","last_became_active":"2024-03-06T07:27:11.290313+0000","last_became_peered":"2024-03-06T07:27:11.290313+0000","last_unstale":"2024-03-06T07:27:13.316427+0000","last_undegraded":"2024-03-06T07:27:13.316427+0000","last_fullsized":"2024-03-06T07:27:13.316427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:54.818102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315593+0000","last_change":"2024-03-06T07:27:11.289550+0000","last_active":"2024-03-06T07:27:13.315593+0000","last_peered":"2024-03-06T07:27:13.315593+0000","last_clean":"2024-03-06T07:27:13.315593+0000","last_became_active":"2024-03-06T07:27:11.289305+0000","last_became_peered":"2024-03-06T07:27:11.289305+0000","last_unstale":"2024-03-06T07:27:13.315593+0000","last_undegraded":"2024-03-06T07:27:13.315593+0000","last_fullsized":"2024-03-06T07:27:13.315593+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:07.747073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281108+0000","last_change":"2024-03-06T07:27:11.275511+0000","last_active":"2024-03-06T07:27:12.281108+0000","last_peered":"2024-03-06T07:27:12.281108+0000","last_clean":"2024-03-06T07:27:12.281108+0000","last_became_active":"2024-03-06T07:27:11.275231+0000","last_became_peered":"2024-03-06T07:27:11.275231+0000","last_unstale":"2024-03-06T07:27:12.281108+0000","last_undegraded":"2024-03-06T07:27:12.281108+0000","last_fullsized":"2024-03-06T07:27:12.281108+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:51.619182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266300+0000","last_change":"2024-03-06T07:27:11.284090+0000","last_active":"2024-03-06T07:27:13.266300+0000","last_peered":"2024-03-06T07:27:13.266300+0000","last_clean":"2024-03-06T07:27:13.266300+0000","last_became_active":"2024-03-06T07:27:11.283788+0000","last_became_peered":"2024-03-06T07:27:11.283788+0000","last_unstale":"2024-03-06T07:27:13.266300+0000","last_undegraded":"2024-03-06T07:27:13.266300+0000","last_fullsized":"2024-03-06T07:27:13.266300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:59.601907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135872+0000","last_change":"2024-03-06T07:27:11.271614+0000","last_active":"2024-03-06T07:27:13.135872+0000","last_peered":"2024-03-06T07:27:13.135872+0000","last_clean":"2024-03-06T07:27:13.135872+0000","last_became_active":"2024-03-06T07:27:11.271257+0000","last_became_peered":"2024-03-06T07:27:11.271257+0000","last_unstale":"2024-03-06T07:27:13.135872+0000","last_undegraded":"2024-03-06T07:27:13.135872+0000","last_fullsized":"2024-03-06T07:27:13.135872+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:54.325681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315415+0000","last_change":"2024-03-06T07:27:11.286558+0000","last_active":"2024-03-06T07:27:13.315415+0000","last_peered":"2024-03-06T07:27:13.315415+0000","last_clean":"2024-03-06T07:27:13.315415+0000","last_became_active":"2024-03-06T07:27:11.286280+0000","last_became_peered":"2024-03-06T07:27:11.286280+0000","last_unstale":"2024-03-06T07:27:13.315415+0000","last_undegraded":"2024-03-06T07:27:13.315415+0000","last_fullsized":"2024-03-06T07:27:13.315415+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:59.491801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316481+0000","last_change":"2024-03-06T07:27:11.293773+0000","last_active":"2024-03-06T07:27:13.316481+0000","last_peered":"2024-03-06T07:27:13.316481+0000","last_clean":"2024-03-06T07:27:13.316481+0000","last_became_active":"2024-03-06T07:27:11.293529+0000","last_became_peered":"2024-03-06T07:27:11.293529+0000","last_unstale":"2024-03-06T07:27:13.316481+0000","last_undegraded":"2024-03-06T07:27:13.316481+0000","last_fullsized":"2024-03-06T07:27:13.316481+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:41.180576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266483+0000","last_change":"2024-03-06T07:27:10.271295+0000","last_active":"2024-03-06T07:27:13.266483+0000","last_peered":"2024-03-06T07:27:13.266483+0000","last_clean":"2024-03-06T07:27:13.266483+0000","last_became_active":"2024-03-06T07:27:10.270894+0000","last_became_peered":"2024-03-06T07:27:10.270894+0000","last_unstale":"2024-03-06T07:27:13.266483+0000","last_undegraded":"2024-03-06T07:27:13.266483+0000","last_fullsized":"2024-03-06T07:27:13.266483+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:15.407609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.282303+0000","last_change":"2024-03-06T07:27:11.273041+0000","last_active":"2024-03-06T07:27:12.282303+0000","last_peered":"2024-03-06T07:27:12.282303+0000","last_clean":"2024-03-06T07:27:12.282303+0000","last_became_active":"2024-03-06T07:27:11.272703+0000","last_became_peered":"2024-03-06T07:27:11.272703+0000","last_unstale":"2024-03-06T07:27:12.282303+0000","last_undegraded":"2024-03-06T07:27:12.282303+0000","last_fullsized":"2024-03-06T07:27:12.282303+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:09.697689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574136+0000","last_change":"2024-03-06T07:27:10.279732+0000","last_active":"2024-03-06T07:27:12.574136+0000","last_peered":"2024-03-06T07:27:12.574136+0000","last_clean":"2024-03-06T07:27:12.574136+0000","last_became_active":"2024-03-06T07:27:10.279431+0000","last_became_peered":"2024-03-06T07:27:10.279431+0000","last_unstale":"2024-03-06T07:27:12.574136+0000","last_undegraded":"2024-03-06T07:27:12.574136+0000","last_fullsized":"2024-03-06T07:27:12.574136+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:49.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316366+0000","last_change":"2024-03-06T07:27:11.287232+0000","last_active":"2024-03-06T07:27:13.316366+0000","last_peered":"2024-03-06T07:27:13.316366+0000","last_clean":"2024-03-06T07:27:13.316366+0000","last_became_active":"2024-03-06T07:27:11.286175+0000","last_became_peered":"2024-03-06T07:27:11.286175+0000","last_unstale":"2024-03-06T07:27:13.316366+0000","last_undegraded":"2024-03-06T07:27:13.316366+0000","last_fullsized":"2024-03-06T07:27:13.316366+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:38.385046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280211+0000","last_change":"2024-03-06T07:27:10.278840+0000","last_active":"2024-03-06T07:27:12.280211+0000","last_peered":"2024-03-06T07:27:12.280211+0000","last_clean":"2024-03-06T07:27:12.280211+0000","last_became_active":"2024-03-06T07:27:10.278584+0000","last_became_peered":"2024-03-06T07:27:10.278584+0000","last_unstale":"2024-03-06T07:27:12.280211+0000","last_undegraded":"2024-03-06T07:27:12.280211+0000","last_fullsized":"2024-03-06T07:27:12.280211+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:41.872604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574430+0000","last_change":"2024-03-06T07:27:11.288847+0000","last_active":"2024-03-06T07:27:12.574430+0000","last_peered":"2024-03-06T07:27:12.574430+0000","last_clean":"2024-03-06T07:27:12.574430+0000","last_became_active":"2024-03-06T07:27:11.288570+0000","last_became_peered":"2024-03-06T07:27:11.288570+0000","last_unstale":"2024-03-06T07:27:12.574430+0000","last_undegraded":"2024-03-06T07:27:12.574430+0000","last_fullsized":"2024-03-06T07:27:12.574430+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:21.833851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280135+0000","last_change":"2024-03-06T07:27:10.278593+0000","last_active":"2024-03-06T07:27:12.280135+0000","last_peered":"2024-03-06T07:27:12.280135+0000","last_clean":"2024-03-06T07:27:12.280135+0000","last_became_active":"2024-03-06T07:27:10.278332+0000","last_became_peered":"2024-03-06T07:27:10.278332+0000","last_unstale":"2024-03-06T07:27:12.280135+0000","last_undegraded":"2024-03-06T07:27:12.280135+0000","last_fullsized":"2024-03-06T07:27:12.280135+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:53.080914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135897+0000","last_change":"2024-03-06T07:27:11.285918+0000","last_active":"2024-03-06T07:27:13.135897+0000","last_peered":"2024-03-06T07:27:13.135897+0000","last_clean":"2024-03-06T07:27:13.135897+0000","last_became_active":"2024-03-06T07:27:11.285459+0000","last_became_peered":"2024-03-06T07:27:11.285459+0000","last_unstale":"2024-03-06T07:27:13.135897+0000","last_undegraded":"2024-03-06T07:27:13.135897+0000","last_fullsized":"2024-03-06T07:27:13.135897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.750370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266175+0000","last_change":"2024-03-06T07:27:10.271671+0000","last_active":"2024-03-06T07:27:13.266175+0000","last_peered":"2024-03-06T07:27:13.266175+0000","last_clean":"2024-03-06T07:27:13.266175+0000","last_became_active":"2024-03-06T07:27:10.271376+0000","last_became_peered":"2024-03-06T07:27:10.271376+0000","last_unstale":"2024-03-06T07:27:13.266175+0000","last_undegraded":"2024-03-06T07:27:13.266175+0000","last_fullsized":"2024-03-06T07:27:13.266175+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:27.328287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281464+0000","last_change":"2024-03-06T07:27:11.286632+0000","last_active":"2024-03-06T07:27:12.281464+0000","last_peered":"2024-03-06T07:27:12.281464+0000","last_clean":"2024-03-06T07:27:12.281464+0000","last_became_active":"2024-03-06T07:27:11.286250+0000","last_became_peered":"2024-03-06T07:27:11.286250+0000","last_unstale":"2024-03-06T07:27:12.281464+0000","last_undegraded":"2024-03-06T07:27:12.281464+0000","last_fullsized":"2024-03-06T07:27:12.281464+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:35.801091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316041+0000","last_change":"2024-03-06T07:27:10.269066+0000","last_active":"2024-03-06T07:27:13.316041+0000","last_peered":"2024-03-06T07:27:13.316041+0000","last_clean":"2024-03-06T07:27:13.316041+0000","last_became_active":"2024-03-06T07:27:10.268772+0000","last_became_peered":"2024-03-06T07:27:10.268772+0000","last_unstale":"2024-03-06T07:27:13.316041+0000","last_undegraded":"2024-03-06T07:27:13.316041+0000","last_fullsized":"2024-03-06T07:27:13.316041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:58.449261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135847+0000","last_change":"2024-03-06T07:27:11.290987+0000","last_active":"2024-03-06T07:27:13.135847+0000","last_peered":"2024-03-06T07:27:13.135847+0000","last_clean":"2024-03-06T07:27:13.135847+0000","last_became_active":"2024-03-06T07:27:11.290745+0000","last_became_peered":"2024-03-06T07:27:11.290745+0000","last_unstale":"2024-03-06T07:27:13.135847+0000","last_undegraded":"2024-03-06T07:27:13.135847+0000","last_fullsized":"2024-03-06T07:27:13.135847+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:49.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280841+0000","last_change":"2024-03-06T07:27:10.268260+0000","last_active":"2024-03-06T07:27:12.280841+0000","last_peered":"2024-03-06T07:27:12.280841+0000","last_clean":"2024-03-06T07:27:12.280841+0000","last_became_active":"2024-03-06T07:27:10.267951+0000","last_became_peered":"2024-03-06T07:27:10.267951+0000","last_unstale":"2024-03-06T07:27:12.280841+0000","last_undegraded":"2024-03-06T07:27:12.280841+0000","last_fullsized":"2024-03-06T07:27:12.280841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:37.381374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266119+0000","last_change":"2024-03-06T07:27:11.274420+0000","last_active":"2024-03-06T07:27:13.266119+0000","last_peered":"2024-03-06T07:27:13.266119+0000","last_clean":"2024-03-06T07:27:13.266119+0000","last_became_active":"2024-03-06T07:27:11.273786+0000","last_became_peered":"2024-03-06T07:27:11.273786+0000","last_unstale":"2024-03-06T07:27:13.266119+0000","last_undegraded":"2024-03-06T07:27:13.266119+0000","last_fullsized":"2024-03-06T07:27:13.266119+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:44.665708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135940+0000","last_change":"2024-03-06T07:27:10.278526+0000","last_active":"2024-03-06T07:27:13.135940+0000","last_peered":"2024-03-06T07:27:13.135940+0000","last_clean":"2024-03-06T07:27:13.135940+0000","last_became_active":"2024-03-06T07:27:10.277985+0000","last_became_peered":"2024-03-06T07:27:10.277985+0000","last_unstale":"2024-03-06T07:27:13.135940+0000","last_undegraded":"2024-03-06T07:27:13.135940+0000","last_fullsized":"2024-03-06T07:27:13.135940+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:53.219915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266492+0000","last_change":"2024-03-06T07:27:11.270127+0000","last_active":"2024-03-06T07:27:13.266492+0000","last_peered":"2024-03-06T07:27:13.266492+0000","last_clean":"2024-03-06T07:27:13.266492+0000","last_became_active":"2024-03-06T07:27:11.269853+0000","last_became_peered":"2024-03-06T07:27:11.269853+0000","last_unstale":"2024-03-06T07:27:13.266492+0000","last_undegraded":"2024-03-06T07:27:13.266492+0000","last_fullsized":"2024-03-06T07:27:13.266492+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:15.070965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280091+0000","last_change":"2024-03-06T07:27:10.277574+0000","last_active":"2024-03-06T07:27:12.280091+0000","last_peered":"2024-03-06T07:27:12.280091+0000","last_clean":"2024-03-06T07:27:12.280091+0000","last_became_active":"2024-03-06T07:27:10.277328+0000","last_became_peered":"2024-03-06T07:27:10.277328+0000","last_unstale":"2024-03-06T07:27:12.280091+0000","last_undegraded":"2024-03-06T07:27:12.280091+0000","last_fullsized":"2024-03-06T07:27:12.280091+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:19.574698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.282368+0000","last_change":"2024-03-06T07:27:11.274532+0000","last_active":"2024-03-06T07:27:12.282368+0000","last_peered":"2024-03-06T07:27:12.282368+0000","last_clean":"2024-03-06T07:27:12.282368+0000","last_became_active":"2024-03-06T07:27:11.274236+0000","last_became_peered":"2024-03-06T07:27:11.274236+0000","last_unstale":"2024-03-06T07:27:12.282368+0000","last_undegraded":"2024-03-06T07:27:12.282368+0000","last_fullsized":"2024-03-06T07:27:12.282368+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:27.211413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315851+0000","last_change":"2024-03-06T07:27:10.269028+0000","last_active":"2024-03-06T07:27:13.315851+0000","last_peered":"2024-03-06T07:27:13.315851+0000","last_clean":"2024-03-06T07:27:13.315851+0000","last_became_active":"2024-03-06T07:27:10.268733+0000","last_became_peered":"2024-03-06T07:27:10.268733+0000","last_unstale":"2024-03-06T07:27:13.315851+0000","last_undegraded":"2024-03-06T07:27:13.315851+0000","last_fullsized":"2024-03-06T07:27:13.315851+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:26.207740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315625+0000","last_change":"2024-03-06T07:27:11.283211+0000","last_active":"2024-03-06T07:27:13.315625+0000","last_peered":"2024-03-06T07:27:13.315625+0000","last_clean":"2024-03-06T07:27:13.315625+0000","last_became_active":"2024-03-06T07:27:11.282770+0000","last_became_peered":"2024-03-06T07:27:11.282770+0000","last_unstale":"2024-03-06T07:27:13.315625+0000","last_undegraded":"2024-03-06T07:27:13.315625+0000","last_fullsized":"2024-03-06T07:27:13.315625+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.210495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136167+0000","last_change":"2024-03-06T07:27:10.281030+0000","last_active":"2024-03-06T07:27:13.136167+0000","last_peered":"2024-03-06T07:27:13.136167+0000","last_clean":"2024-03-06T07:27:13.136167+0000","last_became_active":"2024-03-06T07:27:10.280767+0000","last_became_peered":"2024-03-06T07:27:10.280767+0000","last_unstale":"2024-03-06T07:27:13.136167+0000","last_undegraded":"2024-03-06T07:27:13.136167+0000","last_fullsized":"2024-03-06T07:27:13.136167+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:38.077486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266360+0000","last_change":"2024-03-06T07:27:11.278923+0000","last_active":"2024-03-06T07:27:13.266360+0000","last_peered":"2024-03-06T07:27:13.266360+0000","last_clean":"2024-03-06T07:27:13.266360+0000","last_became_active":"2024-03-06T07:27:11.278610+0000","last_became_peered":"2024-03-06T07:27:11.278610+0000","last_unstale":"2024-03-06T07:27:13.266360+0000","last_undegraded":"2024-03-06T07:27:13.266360+0000","last_fullsized":"2024-03-06T07:27:13.266360+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:45.688316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136292+0000","last_change":"2024-03-06T07:27:10.278124+0000","last_active":"2024-03-06T07:27:13.136292+0000","last_peered":"2024-03-06T07:27:13.136292+0000","last_clean":"2024-03-06T07:27:13.136292+0000","last_became_active":"2024-03-06T07:27:10.277686+0000","last_became_peered":"2024-03-06T07:27:10.277686+0000","last_unstale":"2024-03-06T07:27:13.136292+0000","last_undegraded":"2024-03-06T07:27:13.136292+0000","last_fullsized":"2024-03-06T07:27:13.136292+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:18.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315680+0000","last_change":"2024-03-06T07:27:11.282476+0000","last_active":"2024-03-06T07:27:13.315680+0000","last_peered":"2024-03-06T07:27:13.315680+0000","last_clean":"2024-03-06T07:27:13.315680+0000","last_became_active":"2024-03-06T07:27:11.281936+0000","last_became_peered":"2024-03-06T07:27:11.281936+0000","last_unstale":"2024-03-06T07:27:13.315680+0000","last_undegraded":"2024-03-06T07:27:13.315680+0000","last_fullsized":"2024-03-06T07:27:13.315680+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:07.548742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316147+0000","last_change":"2024-03-06T07:27:10.272261+0000","last_active":"2024-03-06T07:27:13.316147+0000","last_peered":"2024-03-06T07:27:13.316147+0000","last_clean":"2024-03-06T07:27:13.316147+0000","last_became_active":"2024-03-06T07:27:10.272009+0000","last_became_peered":"2024-03-06T07:27:10.272009+0000","last_unstale":"2024-03-06T07:27:13.316147+0000","last_undegraded":"2024-03-06T07:27:13.316147+0000","last_fullsized":"2024-03-06T07:27:13.316147+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:47.748004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315636+0000","last_change":"2024-03-06T07:27:11.282879+0000","last_active":"2024-03-06T07:27:13.315636+0000","last_peered":"2024-03-06T07:27:13.315636+0000","last_clean":"2024-03-06T07:27:13.315636+0000","last_became_active":"2024-03-06T07:27:11.282259+0000","last_became_peered":"2024-03-06T07:27:11.282259+0000","last_unstale":"2024-03-06T07:27:13.315636+0000","last_undegraded":"2024-03-06T07:27:13.315636+0000","last_fullsized":"2024-03-06T07:27:13.315636+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:54.223587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281444+0000","last_change":"2024-03-06T07:27:10.275499+0000","last_active":"2024-03-06T07:27:12.281444+0000","last_peered":"2024-03-06T07:27:12.281444+0000","last_clean":"2024-03-06T07:27:12.281444+0000","last_became_active":"2024-03-06T07:27:10.275217+0000","last_became_peered":"2024-03-06T07:27:10.275217+0000","last_unstale":"2024-03-06T07:27:12.281444+0000","last_undegraded":"2024-03-06T07:27:12.281444+0000","last_fullsized":"2024-03-06T07:27:12.281444+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:03.349214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315688+0000","last_change":"2024-03-06T07:27:11.284094+0000","last_active":"2024-03-06T07:27:13.315688+0000","last_peered":"2024-03-06T07:27:13.315688+0000","last_clean":"2024-03-06T07:27:13.315688+0000","last_became_active":"2024-03-06T07:27:11.283834+0000","last_became_peered":"2024-03-06T07:27:11.283834+0000","last_unstale":"2024-03-06T07:27:13.315688+0000","last_undegraded":"2024-03-06T07:27:13.315688+0000","last_fullsized":"2024-03-06T07:27:13.315688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:01.846507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.279966+0000","last_change":"2024-03-06T07:27:10.276787+0000","last_active":"2024-03-06T07:27:12.279966+0000","last_peered":"2024-03-06T07:27:12.279966+0000","last_clean":"2024-03-06T07:27:12.279966+0000","last_became_active":"2024-03-06T07:27:10.276489+0000","last_became_peered":"2024-03-06T07:27:10.276489+0000","last_unstale":"2024-03-06T07:27:12.279966+0000","last_undegraded":"2024-03-06T07:27:12.279966+0000","last_fullsized":"2024-03-06T07:27:12.279966+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:10.272408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280415+0000","last_change":"2024-03-06T07:27:11.289647+0000","last_active":"2024-03-06T07:27:12.280415+0000","last_peered":"2024-03-06T07:27:12.280415+0000","last_clean":"2024-03-06T07:27:12.280415+0000","last_became_active":"2024-03-06T07:27:11.289047+0000","last_became_peered":"2024-03-06T07:27:11.289047+0000","last_unstale":"2024-03-06T07:27:12.280415+0000","last_undegraded":"2024-03-06T07:27:12.280415+0000","last_fullsized":"2024-03-06T07:27:12.280415+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:14.358579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136590+0000","last_change":"2024-03-06T07:27:10.280931+0000","last_active":"2024-03-06T07:27:13.136590+0000","last_peered":"2024-03-06T07:27:13.136590+0000","last_clean":"2024-03-06T07:27:13.136590+0000","last_became_active":"2024-03-06T07:27:10.280631+0000","last_became_peered":"2024-03-06T07:27:10.280631+0000","last_unstale":"2024-03-06T07:27:13.136590+0000","last_undegraded":"2024-03-06T07:27:13.136590+0000","last_fullsized":"2024-03-06T07:27:13.136590+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:47.398871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316471+0000","last_change":"2024-03-06T07:27:11.292080+0000","last_active":"2024-03-06T07:27:13.316471+0000","last_peered":"2024-03-06T07:27:13.316471+0000","last_clean":"2024-03-06T07:27:13.316471+0000","last_became_active":"2024-03-06T07:27:11.291812+0000","last_became_peered":"2024-03-06T07:27:11.291812+0000","last_unstale":"2024-03-06T07:27:13.316471+0000","last_undegraded":"2024-03-06T07:27:13.316471+0000","last_fullsized":"2024-03-06T07:27:13.316471+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:44.665708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280387+0000","last_change":"2024-03-06T07:27:10.275684+0000","last_active":"2024-03-06T07:27:12.280387+0000","last_peered":"2024-03-06T07:27:12.280387+0000","last_clean":"2024-03-06T07:27:12.280387+0000","last_became_active":"2024-03-06T07:27:10.275341+0000","last_became_peered":"2024-03-06T07:27:10.275341+0000","last_unstale":"2024-03-06T07:27:12.280387+0000","last_undegraded":"2024-03-06T07:27:12.280387+0000","last_fullsized":"2024-03-06T07:27:12.280387+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:43.140170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574054+0000","last_change":"2024-03-06T07:27:11.287870+0000","last_active":"2024-03-06T07:27:12.574054+0000","last_peered":"2024-03-06T07:27:12.574054+0000","last_clean":"2024-03-06T07:27:12.574054+0000","last_became_active":"2024-03-06T07:27:11.287618+0000","last_became_peered":"2024-03-06T07:27:11.287618+0000","last_unstale":"2024-03-06T07:27:12.574054+0000","last_undegraded":"2024-03-06T07:27:12.574054+0000","last_fullsized":"2024-03-06T07:27:12.574054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:22.881185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281510+0000","last_change":"2024-03-06T07:27:10.273231+0000","last_active":"2024-03-06T07:27:12.281510+0000","last_peered":"2024-03-06T07:27:12.281510+0000","last_clean":"2024-03-06T07:27:12.281510+0000","last_became_active":"2024-03-06T07:27:10.272878+0000","last_became_peered":"2024-03-06T07:27:10.272878+0000","last_unstale":"2024-03-06T07:27:12.281510+0000","last_undegraded":"2024-03-06T07:27:12.281510+0000","last_fullsized":"2024-03-06T07:27:12.281510+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:38.441829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574444+0000","last_change":"2024-03-06T07:27:11.287627+0000","last_active":"2024-03-06T07:27:12.574444+0000","last_peered":"2024-03-06T07:27:12.574444+0000","last_clean":"2024-03-06T07:27:12.574444+0000","last_became_active":"2024-03-06T07:27:11.287333+0000","last_became_peered":"2024-03-06T07:27:11.287333+0000","last_unstale":"2024-03-06T07:27:12.574444+0000","last_undegraded":"2024-03-06T07:27:12.574444+0000","last_fullsized":"2024-03-06T07:27:12.574444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:17.002710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266183+0000","last_change":"2024-03-06T07:27:10.277902+0000","last_active":"2024-03-06T07:27:13.266183+0000","last_peered":"2024-03-06T07:27:13.266183+0000","last_clean":"2024-03-06T07:27:13.266183+0000","last_became_active":"2024-03-06T07:27:10.277622+0000","last_became_peered":"2024-03-06T07:27:10.277622+0000","last_unstale":"2024-03-06T07:27:13.266183+0000","last_undegraded":"2024-03-06T07:27:13.266183+0000","last_fullsized":"2024-03-06T07:27:13.266183+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:57.522954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.573914+0000","last_change":"2024-03-06T07:27:11.281911+0000","last_active":"2024-03-06T07:27:12.573914+0000","last_peered":"2024-03-06T07:27:12.573914+0000","last_clean":"2024-03-06T07:27:12.573914+0000","last_became_active":"2024-03-06T07:27:11.281531+0000","last_became_peered":"2024-03-06T07:27:11.281531+0000","last_unstale":"2024-03-06T07:27:12.573914+0000","last_undegraded":"2024-03-06T07:27:12.573914+0000","last_fullsized":"2024-03-06T07:27:12.573914+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:10.645129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.316216+0000","last_change":"2024-03-06T07:27:10.277544+0000","last_active":"2024-03-06T07:27:13.316216+0000","last_peered":"2024-03-06T07:27:13.316216+0000","last_clean":"2024-03-06T07:27:13.316216+0000","last_became_active":"2024-03-06T07:27:10.276523+0000","last_became_peered":"2024-03-06T07:27:10.276523+0000","last_unstale":"2024-03-06T07:27:13.316216+0000","last_undegraded":"2024-03-06T07:27:13.316216+0000","last_fullsized":"2024-03-06T07:27:13.316216+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:25.067873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135917+0000","last_change":"2024-03-06T07:27:11.273670+0000","last_active":"2024-03-06T07:27:13.135917+0000","last_peered":"2024-03-06T07:27:13.135917+0000","last_clean":"2024-03-06T07:27:13.135917+0000","last_became_active":"2024-03-06T07:27:11.273411+0000","last_became_peered":"2024-03-06T07:27:11.273411+0000","last_unstale":"2024-03-06T07:27:13.135917+0000","last_undegraded":"2024-03-06T07:27:13.135917+0000","last_fullsized":"2024-03-06T07:27:13.135917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:08.581865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280047+0000","last_change":"2024-03-06T07:27:10.278830+0000","last_active":"2024-03-06T07:27:12.280047+0000","last_peered":"2024-03-06T07:27:12.280047+0000","last_clean":"2024-03-06T07:27:12.280047+0000","last_became_active":"2024-03-06T07:27:10.278106+0000","last_became_peered":"2024-03-06T07:27:10.278106+0000","last_unstale":"2024-03-06T07:27:12.280047+0000","last_undegraded":"2024-03-06T07:27:12.280047+0000","last_fullsized":"2024-03-06T07:27:12.280047+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:19.175833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.573960+0000","last_change":"2024-03-06T07:27:11.288954+0000","last_active":"2024-03-06T07:27:12.573960+0000","last_peered":"2024-03-06T07:27:12.573960+0000","last_clean":"2024-03-06T07:27:12.573960+0000","last_became_active":"2024-03-06T07:27:11.288655+0000","last_became_peered":"2024-03-06T07:27:11.288655+0000","last_unstale":"2024-03-06T07:27:12.573960+0000","last_undegraded":"2024-03-06T07:27:12.573960+0000","last_fullsized":"2024-03-06T07:27:12.573960+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:08.886485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315789+0000","last_change":"2024-03-06T07:27:10.271560+0000","last_active":"2024-03-06T07:27:13.315789+0000","last_peered":"2024-03-06T07:27:13.315789+0000","last_clean":"2024-03-06T07:27:13.315789+0000","last_became_active":"2024-03-06T07:27:10.271296+0000","last_became_peered":"2024-03-06T07:27:10.271296+0000","last_unstale":"2024-03-06T07:27:13.315789+0000","last_undegraded":"2024-03-06T07:27:13.315789+0000","last_fullsized":"2024-03-06T07:27:13.315789+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:51.960409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315819+0000","last_change":"2024-03-06T07:27:11.289924+0000","last_active":"2024-03-06T07:27:13.315819+0000","last_peered":"2024-03-06T07:27:13.315819+0000","last_clean":"2024-03-06T07:27:13.315819+0000","last_became_active":"2024-03-06T07:27:11.288953+0000","last_became_peered":"2024-03-06T07:27:11.288953+0000","last_unstale":"2024-03-06T07:27:13.315819+0000","last_undegraded":"2024-03-06T07:27:13.315819+0000","last_fullsized":"2024-03-06T07:27:13.315819+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:48.402544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136012+0000","last_change":"2024-03-06T07:27:10.279864+0000","last_active":"2024-03-06T07:27:13.136012+0000","last_peered":"2024-03-06T07:27:13.136012+0000","last_clean":"2024-03-06T07:27:13.136012+0000","last_became_active":"2024-03-06T07:27:10.279622+0000","last_became_peered":"2024-03-06T07:27:10.279622+0000","last_unstale":"2024-03-06T07:27:13.136012+0000","last_undegraded":"2024-03-06T07:27:13.136012+0000","last_fullsized":"2024-03-06T07:27:13.136012+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:00.842834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281970+0000","last_change":"2024-03-06T07:27:11.284561+0000","last_active":"2024-03-06T07:27:12.281970+0000","last_peered":"2024-03-06T07:27:12.281970+0000","last_clean":"2024-03-06T07:27:12.281970+0000","last_became_active":"2024-03-06T07:27:11.284292+0000","last_became_peered":"2024-03-06T07:27:11.284292+0000","last_unstale":"2024-03-06T07:27:12.281970+0000","last_undegraded":"2024-03-06T07:27:12.281970+0000","last_fullsized":"2024-03-06T07:27:12.281970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:11.179780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315882+0000","last_change":"2024-03-06T07:27:10.273788+0000","last_active":"2024-03-06T07:27:13.315882+0000","last_peered":"2024-03-06T07:27:13.315882+0000","last_clean":"2024-03-06T07:27:13.315882+0000","last_became_active":"2024-03-06T07:27:10.273542+0000","last_became_peered":"2024-03-06T07:27:10.273542+0000","last_unstale":"2024-03-06T07:27:13.315882+0000","last_undegraded":"2024-03-06T07:27:13.315882+0000","last_fullsized":"2024-03-06T07:27:13.315882+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:08.694016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136542+0000","last_change":"2024-03-06T07:27:11.271394+0000","last_active":"2024-03-06T07:27:13.136542+0000","last_peered":"2024-03-06T07:27:13.136542+0000","last_clean":"2024-03-06T07:27:13.136542+0000","last_became_active":"2024-03-06T07:27:11.270251+0000","last_became_peered":"2024-03-06T07:27:11.270251+0000","last_unstale":"2024-03-06T07:27:13.136542+0000","last_undegraded":"2024-03-06T07:27:13.136542+0000","last_fullsized":"2024-03-06T07:27:13.136542+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:59.836515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.282096+0000","last_change":"2024-03-06T07:27:10.269615+0000","last_active":"2024-03-06T07:27:12.282096+0000","last_peered":"2024-03-06T07:27:12.282096+0000","last_clean":"2024-03-06T07:27:12.282096+0000","last_became_active":"2024-03-06T07:27:10.269331+0000","last_became_peered":"2024-03-06T07:27:10.269331+0000","last_unstale":"2024-03-06T07:27:12.282096+0000","last_undegraded":"2024-03-06T07:27:12.282096+0000","last_fullsized":"2024-03-06T07:27:12.282096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:22.574956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136276+0000","last_change":"2024-03-06T07:27:11.289894+0000","last_active":"2024-03-06T07:27:13.136276+0000","last_peered":"2024-03-06T07:27:13.136276+0000","last_clean":"2024-03-06T07:27:13.136276+0000","last_became_active":"2024-03-06T07:27:11.289625+0000","last_became_peered":"2024-03-06T07:27:11.289625+0000","last_unstale":"2024-03-06T07:27:13.136276+0000","last_undegraded":"2024-03-06T07:27:13.136276+0000","last_fullsized":"2024-03-06T07:27:13.136276+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:58.272911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136197+0000","last_change":"2024-03-06T07:27:10.273864+0000","last_active":"2024-03-06T07:27:13.136197+0000","last_peered":"2024-03-06T07:27:13.136197+0000","last_clean":"2024-03-06T07:27:13.136197+0000","last_became_active":"2024-03-06T07:27:10.273611+0000","last_became_peered":"2024-03-06T07:27:10.273611+0000","last_unstale":"2024-03-06T07:27:13.136197+0000","last_undegraded":"2024-03-06T07:27:13.136197+0000","last_fullsized":"2024-03-06T07:27:13.136197+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:06.743401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281413+0000","last_change":"2024-03-06T07:27:11.274612+0000","last_active":"2024-03-06T07:27:12.281413+0000","last_peered":"2024-03-06T07:27:12.281413+0000","last_clean":"2024-03-06T07:27:12.281413+0000","last_became_active":"2024-03-06T07:27:11.274304+0000","last_became_peered":"2024-03-06T07:27:11.274304+0000","last_unstale":"2024-03-06T07:27:12.281413+0000","last_undegraded":"2024-03-06T07:27:12.281413+0000","last_fullsized":"2024-03-06T07:27:12.281413+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:51.619182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315690+0000","last_change":"2024-03-06T07:27:10.271689+0000","last_active":"2024-03-06T07:27:13.315690+0000","last_peered":"2024-03-06T07:27:13.315690+0000","last_clean":"2024-03-06T07:27:13.315690+0000","last_became_active":"2024-03-06T07:27:10.271421+0000","last_became_peered":"2024-03-06T07:27:10.271421+0000","last_unstale":"2024-03-06T07:27:13.315690+0000","last_undegraded":"2024-03-06T07:27:13.315690+0000","last_fullsized":"2024-03-06T07:27:13.315690+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:44.962932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280921+0000","last_change":"2024-03-06T07:27:11.276400+0000","last_active":"2024-03-06T07:27:12.280921+0000","last_peered":"2024-03-06T07:27:12.280921+0000","last_clean":"2024-03-06T07:27:12.280921+0000","last_became_active":"2024-03-06T07:27:11.276124+0000","last_became_peered":"2024-03-06T07:27:11.276124+0000","last_unstale":"2024-03-06T07:27:12.280921+0000","last_undegraded":"2024-03-06T07:27:12.280921+0000","last_fullsized":"2024-03-06T07:27:12.280921+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:19.052096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266351+0000","last_change":"2024-03-06T07:27:10.268891+0000","last_active":"2024-03-06T07:27:13.266351+0000","last_peered":"2024-03-06T07:27:13.266351+0000","last_clean":"2024-03-06T07:27:13.266351+0000","last_became_active":"2024-03-06T07:27:10.268536+0000","last_became_peered":"2024-03-06T07:27:10.268536+0000","last_unstale":"2024-03-06T07:27:13.266351+0000","last_undegraded":"2024-03-06T07:27:13.266351+0000","last_fullsized":"2024-03-06T07:27:13.266351+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:45.518696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.281623+0000","last_change":"2024-03-06T07:27:11.271113+0000","last_active":"2024-03-06T07:27:12.281623+0000","last_peered":"2024-03-06T07:27:12.281623+0000","last_clean":"2024-03-06T07:27:12.281623+0000","last_became_active":"2024-03-06T07:27:11.270835+0000","last_became_peered":"2024-03-06T07:27:11.270835+0000","last_unstale":"2024-03-06T07:27:12.281623+0000","last_undegraded":"2024-03-06T07:27:12.281623+0000","last_fullsized":"2024-03-06T07:27:12.281623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:06.181936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.280991+0000","last_change":"2024-03-06T07:27:10.272914+0000","last_active":"2024-03-06T07:27:12.280991+0000","last_peered":"2024-03-06T07:27:12.280991+0000","last_clean":"2024-03-06T07:27:12.280991+0000","last_became_active":"2024-03-06T07:27:10.272635+0000","last_became_peered":"2024-03-06T07:27:10.272635+0000","last_unstale":"2024-03-06T07:27:12.280991+0000","last_undegraded":"2024-03-06T07:27:12.280991+0000","last_fullsized":"2024-03-06T07:27:12.280991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:45.645350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574187+0000","last_change":"2024-03-06T07:27:11.283189+0000","last_active":"2024-03-06T07:27:12.574187+0000","last_peered":"2024-03-06T07:27:12.574187+0000","last_clean":"2024-03-06T07:27:12.574187+0000","last_became_active":"2024-03-06T07:27:11.282804+0000","last_became_peered":"2024-03-06T07:27:11.282804+0000","last_unstale":"2024-03-06T07:27:12.574187+0000","last_undegraded":"2024-03-06T07:27:12.574187+0000","last_fullsized":"2024-03-06T07:27:12.574187+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:36.192766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.279981+0000","last_change":"2024-03-06T07:27:10.276957+0000","last_active":"2024-03-06T07:27:12.279981+0000","last_peered":"2024-03-06T07:27:12.279981+0000","last_clean":"2024-03-06T07:27:12.279981+0000","last_became_active":"2024-03-06T07:27:10.276707+0000","last_became_peered":"2024-03-06T07:27:10.276707+0000","last_unstale":"2024-03-06T07:27:12.279981+0000","last_undegraded":"2024-03-06T07:27:12.279981+0000","last_fullsized":"2024-03-06T07:27:12.279981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:41.377958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.135901+0000","last_change":"2024-03-06T07:27:11.290890+0000","last_active":"2024-03-06T07:27:13.135901+0000","last_peered":"2024-03-06T07:27:13.135901+0000","last_clean":"2024-03-06T07:27:13.135901+0000","last_became_active":"2024-03-06T07:27:11.290643+0000","last_became_peered":"2024-03-06T07:27:11.290643+0000","last_unstale":"2024-03-06T07:27:13.135901+0000","last_undegraded":"2024-03-06T07:27:13.135901+0000","last_fullsized":"2024-03-06T07:27:13.135901+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:35.227519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.136038+0000","last_change":"2024-03-06T07:27:10.275390+0000","last_active":"2024-03-06T07:27:13.136038+0000","last_peered":"2024-03-06T07:27:13.136038+0000","last_clean":"2024-03-06T07:27:13.136038+0000","last_became_active":"2024-03-06T07:27:10.274903+0000","last_became_peered":"2024-03-06T07:27:10.274903+0000","last_unstale":"2024-03-06T07:27:13.136038+0000","last_undegraded":"2024-03-06T07:27:13.136038+0000","last_fullsized":"2024-03-06T07:27:13.136038+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:06.545069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.315472+0000","last_change":"2024-03-06T07:27:11.281365+0000","last_active":"2024-03-06T07:27:13.315472+0000","last_peered":"2024-03-06T07:27:13.315472+0000","last_clean":"2024-03-06T07:27:13.315472+0000","last_became_active":"2024-03-06T07:27:11.281108+0000","last_became_peered":"2024-03-06T07:27:11.281108+0000","last_unstale":"2024-03-06T07:27:13.315472+0000","last_undegraded":"2024-03-06T07:27:13.315472+0000","last_fullsized":"2024-03-06T07:27:13.315472+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.253454+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.253454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.944500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:13.266252+0000","last_change":"2024-03-06T07:27:10.277891+0000","last_active":"2024-03-06T07:27:13.266252+0000","last_peered":"2024-03-06T07:27:13.266252+0000","last_clean":"2024-03-06T07:27:13.266252+0000","last_became_active":"2024-03-06T07:27:10.277582+0000","last_became_peered":"2024-03-06T07:27:10.277582+0000","last_unstale":"2024-03-06T07:27:13.266252+0000","last_undegraded":"2024-03-06T07:27:13.266252+0000","last_fullsized":"2024-03-06T07:27:13.266252+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.249781+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.249781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:59.897065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.574391+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:27:12.574391+0000","last_peered":"2024-03-06T07:27:12.574391+0000","last_clean":"2024-03-06T07:27:12.574391+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:27:12.574391+0000","last_undegraded":"2024-03-06T07:27:12.574391+0000","last_fullsized":"2024-03-06T07:27:12.574391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":1169875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1347584,"data_stored":578961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1351680,"data_stored":579495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1343488,"data_stored":578891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574919,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40828,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331012,"statfs":{"total":96636764160,"available":96594956288,"internally_reserved":0,"allocated":1359872,"data_stored":580573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574919,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1343488,"data_stored":578891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1351680,"data_stored":579015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574920,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41412,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330428,"statfs":{"total":96636764160,"available":96594358272,"internally_reserved":0,"allocated":1957888,"data_stored":1170983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:19.107 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:19.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:19.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.250+0000 7f50de5eb640 1 Processor -- start 2024-03-06T07:27:19.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.250+0000 7f50de5eb640 1 -- start start 2024-03-06T07:27:19.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50de5eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f50d815c050 0x7f50d806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50de5eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f50d806fb40 0x7f50d80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50de5eb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d80cd4b0 con 0x7f50d80ccca0 2024-03-06T07:27:19.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50de5eb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d806ffb0 con 0x7f50d806fb40 2024-03-06T07:27:19.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50de5eb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d80cd790 con 0x7f50d815c050 2024-03-06T07:27:19.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50dcb88640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f50d80ccca0 0x7f50d80cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49988/0 (socket says 172.21.15.122:49988) 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50dcb88640 1 -- 172.21.15.122:0/2066525467 learned_addr learned my addr 172.21.15.122:0/2066525467 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d77fe640 1 --2- 172.21.15.122:0/2066525467 >> v2:172.21.15.122:3300/0 conn(0x7f50d806fb40 0x7f50d80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d7fff640 1 --2- 172.21.15.122:0/2066525467 >> v2:172.21.15.169:3300/0 conn(0x7f50d815c050 0x7f50d806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d6ffd640 1 -- 172.21.15.122:0/2066525467 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531766726 0 0) 0x7f50d80cd4b0 con 0x7f50d80ccca0 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d6ffd640 1 -- 172.21.15.122:0/2066525467 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c0003520 con 0x7f50d80ccca0 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d77fe640 1 -- 172.21.15.122:0/2066525467 >> v2:172.21.15.169:3300/0 conn(0x7f50d815c050 msgr2=0x7f50d806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d77fe640 1 --2- 172.21.15.122:0/2066525467 >> v2:172.21.15.169:3300/0 conn(0x7f50d815c050 0x7f50d806f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d77fe640 1 -- 172.21.15.122:0/2066525467 >> v1:172.21.15.122:6789/0 conn(0x7f50d80ccca0 legacy=0x7f50d80cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.251+0000 7f50d77fe640 1 -- 172.21.15.122:0/2066525467 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80cda70 con 0x7f50d806fb40 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.252+0000 7f50d77fe640 1 --2- 172.21.15.122:0/2066525467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d806fb40 0x7f50d80cc760 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f50d8070ef0 tx=0x7f50c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05fd4ce059bafa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.252+0000 7f50d6ffd640 1 -- 172.21.15.122:0/2066525467 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80315a0 con 0x7f50d806fb40 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.252+0000 7f50d6ffd640 1 -- 172.21.15.122:0/2066525467 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c8037070 con 0x7f50d806fb40 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.252+0000 7f50d6ffd640 1 -- 172.21.15.122:0/2066525467 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8031960 con 0x7f50d806fb40 2024-03-06T07:27:19.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 -- 172.21.15.122:0/2066525467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d806fb40 msgr2=0x7f50d80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 --2- 172.21.15.122:0/2066525467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d806fb40 0x7f50d80cc760 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f50d8070ef0 tx=0x7f50c802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 -- 172.21.15.122:0/2066525467 shutdown_connections 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 --2- 172.21.15.122:0/2066525467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d806fb40 0x7f50d80cc760 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 --2- 172.21.15.122:0/2066525467 >> v2:172.21.15.169:3300/0 conn(0x7f50d815c050 0x7f50d806f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 -- 172.21.15.122:0/2066525467 >> 172.21.15.122:0/2066525467 conn(0x7f50d80b3800 msgr2=0x7f50d80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 -- 172.21.15.122:0/2066525467 shutdown_connections 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.253+0000 7f50de5eb640 1 -- 172.21.15.122:0/2066525467 wait complete. 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.254+0000 7f50de5eb640 1 Processor -- start 2024-03-06T07:27:19.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 -- start start 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50d806fb40 0x7f50d8151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 0x7f50d80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50d815c050 0x7f50d80cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80d01f0 con 0x7f50d80ccca0 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80cd130 con 0x7f50d815c050 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50de5eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80d0370 con 0x7f50d806fb40 2024-03-06T07:27:19.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50d77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 0x7f50d80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50d77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 0x7f50d80cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51886/0 (socket says 172.21.15.122:51886) 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.255+0000 7f50d77fe640 1 -- 172.21.15.122:0/1681790168 learned_addr learned my addr 172.21.15.122:0/1681790168 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d7fff640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50d806fb40 0x7f50d8151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50dcb88640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50d815c050 0x7f50d80cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d77fe640 1 -- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50d806fb40 msgr2=0x7f50d8151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d77fe640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50d806fb40 0x7f50d8151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d77fe640 1 -- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50d815c050 msgr2=0x7f50d80cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d77fe640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50d815c050 0x7f50d80cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d77fe640 1 -- 172.21.15.122:0/1681790168 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d806dd50 con 0x7f50d80ccca0 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d7fff640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50d806fb40 0x7f50d8151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50dcb88640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50d815c050 0x7f50d80cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d77fe640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 0x7f50d80cae80 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f50c802be50 tx=0x7f50c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8032630 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c80327d0 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d809ce90 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8032bc0 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.256+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d8151dc0 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.257+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d8070d40 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.257+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50c804c020 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.258+0000 7f50d4ff9640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50c406b4d0 0x7f50c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.258+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50c802f020 con 0x7f50d80ccca0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.258+0000 7f50d7fff640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50c406b4d0 0x7f50c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.258+0000 7f50d7fff640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50c406b4d0 0x7f50c406d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009ac0 tx=0x7f50cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.267+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50d8070d40 con 0x7f50d80ccca0 2024-03-06T07:27:19.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.522+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f50d8068ff0 con 0x7f50d80ccca0 2024-03-06T07:27:19.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.524+0000 7f50d4ff9640 1 -- 172.21.15.122:0/1681790168 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f50c8040070 con 0x7f50d80ccca0 2024-03-06T07:27:19.526 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:27:19.526 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:27:19.526 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:27:19.526 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:27:19.526 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:19.526 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:27:19.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50c406b4d0 msgr2=0x7f50c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50c406b4d0 0x7f50c406d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009ac0 tx=0x7f50cc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 msgr2=0x7f50d80cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 0x7f50d80cae80 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f50c802be50 tx=0x7f50c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 shutdown_connections 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f50c406b4d0 0x7f50c406d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50d815c050 0x7f50d80cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50d80ccca0 0x7f50d80cae80 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 --2- 172.21.15.122:0/1681790168 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f50d806fb40 0x7f50d8151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 >> 172.21.15.122:0/1681790168 conn(0x7f50d80b3800 msgr2=0x7f50d806ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.526+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 shutdown_connections 2024-03-06T07:27:19.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.527+0000 7f50de5eb640 1 -- 172.21.15.122:0/1681790168 wait complete. 2024-03-06T07:27:19.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:19.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.638+0000 7f1b22295640 1 Processor -- start 2024-03-06T07:27:19.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.638+0000 7f1b22295640 1 -- start start 2024-03-06T07:27:19.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b22295640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1b1c0d2710 0x7f1b1c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b22295640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c0b3430 con 0x7f1b1c0d22b0 2024-03-06T07:27:19.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b22295640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c0b3710 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b22295640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1c0b31e0 con 0x7f1b1c0d2710 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b20832640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1b1c0d2710 0x7f1b1c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b1b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1b1c0d22b0 0x7f1b1c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49992/0 (socket says 172.21.15.122:49992) 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b1b7fe640 1 -- 172.21.15.122:0/2389957242 learned_addr learned my addr 172.21.15.122:0/2389957242 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b20832640 1 -- 172.21.15.122:0/2389957242 >> v1:172.21.15.169:6789/0 conn(0x7f1b1c15dfe0 legacy=0x7f1b1c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b20832640 1 -- 172.21.15.122:0/2389957242 >> v1:172.21.15.122:6789/0 conn(0x7f1b1c0d22b0 legacy=0x7f1b1c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b20832640 1 -- 172.21.15.122:0/2389957242 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c0b39f0 con 0x7f1b1c0d2710 2024-03-06T07:27:19.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.639+0000 7f1b1affd640 1 -- 172.21.15.122:0/2389957242 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660559767 0 0) 0x7f1b1c0b3430 con 0x7f1b1c0d22b0 2024-03-06T07:27:19.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.640+0000 7f1b20832640 1 --2- 172.21.15.122:0/2389957242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c0b4900 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009dc0 tx=0x7f1b1402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a87ae36f99bd5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.640+0000 7f1b1affd640 1 -- 172.21.15.122:0/2389957242 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14039070 con 0x7f1b1c0d2710 2024-03-06T07:27:19.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.640+0000 7f1b1affd640 1 -- 172.21.15.122:0/2389957242 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1402cc40 con 0x7f1b1c0d2710 2024-03-06T07:27:19.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.640+0000 7f1b1affd640 1 -- 172.21.15.122:0/2389957242 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14034440 con 0x7f1b1c0d2710 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 -- 172.21.15.122:0/2389957242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c0d2710 msgr2=0x7f1b1c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 --2- 172.21.15.122:0/2389957242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c0b4900 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009dc0 tx=0x7f1b1402be90 comp rx=0 tx=0).stop 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 -- 172.21.15.122:0/2389957242 shutdown_connections 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 --2- 172.21.15.122:0/2389957242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c0b4900 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 -- 172.21.15.122:0/2389957242 >> 172.21.15.122:0/2389957242 conn(0x7f1b1c0d3cd0 msgr2=0x7f1b1c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 -- 172.21.15.122:0/2389957242 shutdown_connections 2024-03-06T07:27:19.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.641+0000 7f1b22295640 1 -- 172.21.15.122:0/2389957242 wait complete. 2024-03-06T07:27:19.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.642+0000 7f1b22295640 1 Processor -- start 2024-03-06T07:27:19.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.642+0000 7f1b22295640 1 -- start start 2024-03-06T07:27:19.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b22295640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b1c0d22b0 0x7f1b1c1f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b22295640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c1f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b22295640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 0x7f1b1c0b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b22295640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c0cfa00 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b22295640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c0b3d60 con 0x7f1b1c0d2710 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b22295640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c0d1c70 con 0x7f1b1c0d22b0 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 0x7f1b1c0b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 0x7f1b1c0b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51912/0 (socket says 172.21.15.122:51912) 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 -- 172.21.15.122:0/2926058904 learned_addr learned my addr 172.21.15.122:0/2926058904 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b1bfff640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b1c0d22b0 0x7f1b1c1f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b1b7fe640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c1f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 -- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b1c0d22b0 msgr2=0x7f1b1c1f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b1c0d22b0 0x7f1b1c1f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 -- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b1c0d2710 msgr2=0x7f1b1c1f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c1f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 -- 172.21.15.122:0/2926058904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c0b7e50 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.643+0000 7f1b20832640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 0x7f1b1c0b89d0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1b14037040 tx=0x7f1b14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.644+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14044020 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.644+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b14034df0 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.644+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1c0b6db0 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.644+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14039040 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.644+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1c0b7370 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.645+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1c04f330 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.646+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b14047020 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.646+0000 7f1b18ff9640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b0806b4d0 0x7f1b0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.646+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b140acc00 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.646+0000 7f1b1bfff640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b0806b4d0 0x7f1b0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.647+0000 7f1b1bfff640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b0806b4d0 0x7f1b0806d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c006fd0 tx=0x7f1b0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.651+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b140b3080 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.860+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b1c0747a0 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.861+0000 7f1b18ff9640 1 -- 172.21.15.122:0/2926058904 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f1b1404e030 con 0x7f1b1c15dfe0 2024-03-06T07:27:19.863 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:19.863 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3571233755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3571233755},{"type":"v1","addr":"172.21.15.169:6834","nonce":3571233755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1631232650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1631232650},{"type":"v1","addr":"172.21.15.169:6835","nonce":1631232650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/2235394450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2235394450},{"type":"v1","addr":"172.21.15.122:6835","nonce":2235394450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:12.635136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7962},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7962":{"gid":7962,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/386534812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":386534812},{"type":"v1","addr":"172.21.15.122:6837","nonce":386534812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7962,"qdb_cluster":[7962]},"id":8}]} 2024-03-06T07:27:19.863 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 61 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b0806b4d0 msgr2=0x7f1b0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b0806b4d0 0x7f1b0806d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c006fd0 tx=0x7f1b0c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 msgr2=0x7f1b1c0b89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 0x7f1b1c0b89d0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1b14037040 tx=0x7f1b14004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 shutdown_connections 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b0806b4d0 0x7f1b0806d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b1c15dfe0 0x7f1b1c0b89d0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b1c0d2710 0x7f1b1c1f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 --2- 172.21.15.122:0/2926058904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b1c0d22b0 0x7f1b1c1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 >> 172.21.15.122:0/2926058904 conn(0x7f1b1c0d3cd0 msgr2=0x7f1b1c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 shutdown_connections 2024-03-06T07:27:19.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:19.863+0000 7f1b22295640 1 -- 172.21.15.122:0/2926058904 wait complete. 2024-03-06T07:27:20.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.604168+0000', 'modified': '2024-03-06T07:27:12.635136+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7962}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7962': {'gid': 7962, 'name': 'a', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/386534812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 386534812}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 386534812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7962, 'qdb_cluster': [7962]} 2024-03-06T07:27:20.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:20.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:27:20.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 Processor -- start 2024-03-06T07:27:20.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 -- start start 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7f340b4460 0x7f7f340d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 0x7f7f340b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f340d22c0 con 0x7f7f34161cf0 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f34161b00 con 0x7f7f340b4460 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.381+0000 7f7f3a581640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340cecc0 con 0x7f7f34162970 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 0x7f7f340b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f33fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7f340b4460 0x7f7f340d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 0x7f7f340b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:46656/0 (socket says 172.21.15.122:46656) 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 -- 172.21.15.122:0/3276320411 learned_addr learned my addr 172.21.15.122:0/3276320411 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 -- 172.21.15.122:0/3276320411 >> v2:172.21.15.169:3300/0 conn(0x7f7f340b4460 msgr2=0x7f7f340d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 --2- 172.21.15.122:0/3276320411 >> v2:172.21.15.169:3300/0 conn(0x7f7f340b4460 0x7f7f340d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 -- 172.21.15.122:0/3276320411 >> v1:172.21.15.169:6789/0 conn(0x7f7f34161cf0 legacy=0x7f7f340cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.382+0000 7f7f337fe640 1 -- 172.21.15.122:0/3276320411 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340d25a0 con 0x7f7f34162970 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.383+0000 7f7f337fe640 1 --2- 172.21.15.122:0/3276320411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 0x7f7f340b3580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009870 tx=0x7f7f2802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36c0dcbf84b68951 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.383+0000 7f7f32ffd640 1 -- 172.21.15.122:0/3276320411 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28034070 con 0x7f7f34162970 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.383+0000 7f7f32ffd640 1 -- 172.21.15.122:0/3276320411 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f28030840 con 0x7f7f34162970 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.383+0000 7f7f32ffd640 1 -- 172.21.15.122:0/3276320411 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28030c10 con 0x7f7f34162970 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 -- 172.21.15.122:0/3276320411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 msgr2=0x7f7f340b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3276320411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 0x7f7f340b3580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009870 tx=0x7f7f2802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:20.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 -- 172.21.15.122:0/3276320411 shutdown_connections 2024-03-06T07:27:20.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3276320411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34162970 0x7f7f340b3580 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3276320411 >> v2:172.21.15.169:3300/0 conn(0x7f7f340b4460 0x7f7f340d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 -- 172.21.15.122:0/3276320411 >> 172.21.15.122:0/3276320411 conn(0x7f7f340d3ce0 msgr2=0x7f7f340d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 -- 172.21.15.122:0/3276320411 shutdown_connections 2024-03-06T07:27:20.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.384+0000 7f7f3a581640 1 -- 172.21.15.122:0/3276320411 wait complete. 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.385+0000 7f7f3a581640 1 Processor -- start 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.385+0000 7f7f3a581640 1 -- start start 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f3a581640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f340b4460 0x7f7f340cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f3a581640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34161cf0 0x7f7f34156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f3a581640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 0x7f7f340cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f3a581640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340ce950 con 0x7f7f34162970 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f3a581640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340d2b60 con 0x7f7f340b4460 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f3a581640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340b3ac0 con 0x7f7f34161cf0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 0x7f7f340cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 0x7f7f340cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51914/0 (socket says 172.21.15.122:51914) 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 -- 172.21.15.122:0/3223378804 learned_addr learned my addr 172.21.15.122:0/3223378804 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f33fff640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f340b4460 0x7f7f340cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 -- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34161cf0 msgr2=0x7f7f34156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f337fe640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34161cf0 0x7f7f34156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34161cf0 0x7f7f34156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 -- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f340b4460 msgr2=0x7f7f340cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f340b4460 0x7f7f340cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 -- 172.21.15.122:0/3223378804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340cb850 con 0x7f7f34162970 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 0x7f7f340cadb0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00c7a0 tx=0x7f7f2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.386+0000 7f7f30ff9640 1 -- 172.21.15.122:0/3223378804 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c003b70 con 0x7f7f34162970 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.387+0000 7f7f30ff9640 1 -- 172.21.15.122:0/3223378804 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c003d30 con 0x7f7f34162970 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.387+0000 7f7f30ff9640 1 -- 172.21.15.122:0/3223378804 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c00aa90 con 0x7f7f34162970 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.387+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f340cbb30 con 0x7f7f34162970 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.387+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f340cc0f0 con 0x7f7f34162970 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.388+0000 7f7f30ff9640 1 -- 172.21.15.122:0/3223378804 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f2c00ac30 con 0x7f7f34162970 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.388+0000 7f7f30ff9640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2006b500 0x7f7f2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.388+0000 7f7f30ff9640 1 -- 172.21.15.122:0/3223378804 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f2c08a130 con 0x7f7f34162970 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.388+0000 7f7f33fff640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2006b500 0x7f7f2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.389+0000 7f7f33fff640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2006b500 0x7f7f2006d9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7f34156b30 tx=0x7f7f24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.390+0000 7f7f3d16b740 1 Processor -- start 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.390+0000 7f7f3d16b740 1 -- start start 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 0x55b82e41fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e4238c0 0x55b82e425360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e3896b0 con 0x55b82e41f870 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e332a60 con 0x55b82e44f600 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b82e385750 con 0x55b82e4238c0 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f33fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 0x55b82e41fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.391+0000 7f7f33fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 0x55b82e41fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51926/0 (socket says 172.21.15.122:51926) 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 -- 172.21.15.122:0/807021990 learned_addr learned my addr 172.21.15.122:0/807021990 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f337fe640 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e4238c0 0x55b82e425360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 -- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e4238c0 msgr2=0x55b82e425360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e4238c0 0x55b82e425360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 -- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 msgr2=0x55b82e44fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 -- 172.21.15.122:0/807021990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b82e350f30 con 0x55b82e41f870 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f337fe640 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e4238c0 0x55b82e425360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 0x55b82e41fc50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7f24010a10 tx=0x7f7f24013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d970a7e5807e685e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f19ffb640 1 -- 172.21.15.122:0/807021990 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2401c050 con 0x55b82e41f870 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f19ffb640 1 -- 172.21.15.122:0/807021990 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24026070 con 0x55b82e41f870 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.392+0000 7f7f19ffb640 1 -- 172.21.15.122:0/807021990 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2400f040 con 0x55b82e41f870 2024-03-06T07:27:20.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807021990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 msgr2=0x55b82e41fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 0x55b82e41fc50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7f24010a10 tx=0x7f7f24013000 comp rx=0 tx=0).stop 2024-03-06T07:27:20.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807021990 shutdown_connections 2024-03-06T07:27:20.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e4238c0 0x55b82e425360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807021990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e41f870 0x55b82e41fc50 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807021990 >> 172.21.15.122:0/807021990 conn(0x55b82e4218f0 msgr2=0x55b82e4231b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807021990 shutdown_connections 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.393+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807021990 wait complete. 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.394+0000 7f7f3d16b740 1 -- start start 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.394+0000 7f7f3d16b740 1 Processor -- start 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.394+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e448500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.394+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e445f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.394+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e449e10 0x55b82e44a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.394+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e423d50 con 0x55b82e442350 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e345ea0 con 0x55b82e449e10 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b82e424030 con 0x55b82e44f600 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f3d16b740 10 client.0 _ll_register_callbacks cb 0x55b82e345870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e445f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e445f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51940/0 (socket says 172.21.15.122:51940) 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 -- 172.21.15.122:0/1469802090 learned_addr learned my addr 172.21.15.122:0/1469802090 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f33fff640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e448500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e449e10 0x55b82e44a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 -- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 msgr2=0x55b82e448500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e448500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 -- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e449e10 msgr2=0x55b82e44a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e449e10 0x55b82e44a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 -- 172.21.15.122:0/1469802090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b82e346270 con 0x55b82e442350 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f337fe640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e445f70 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009960 tx=0x7f7f28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9288247e063dfa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f12ffd640 1 -- 172.21.15.122:0/1469802090 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2804a060 con 0x55b82e442350 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f12ffd640 1 -- 172.21.15.122:0/1469802090 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f28039860 con 0x55b82e442350 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f24026070 con 0x55b82e442350 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.395+0000 7f7f12ffd640 1 -- 172.21.15.122:0/1469802090 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f28050050 con 0x55b82e442350 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 10 client.8130 fetch_fsmap learned FSMap version 61 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55b82e09e2d0 con 0x55b82e442350 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f12ffd640 1 -- 172.21.15.122:0/1469802090 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24026070 con 0x55b82e442350 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f12ffd640 1 -- 172.21.15.122:0/1469802090 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7f28053020 con 0x55b82e442350 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 10 client.8130 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 10 client.8130 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 20 client.8130 populate_metadata read hostname 'smithi122' 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e4442b0 0x55b82e4898f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 4 client.8130 mds_command: new command op to 7962 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 --> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b82e396bf0 con 0x55b82e4442b0 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f33fff640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e4442b0 0x55b82e4898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f12ffd640 10 client.8130 ms_handle_connect on v2:172.21.15.122:6836/386534812 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.396+0000 7f7f33fff640 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e4442b0 0x55b82e4898f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=541c29dc39914c1d server_cookie=49a5f528b66de959 in_seq=0 out_seq=0 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f12ffd640 1 -- 172.21.15.122:0/1469802090 <== mds.0 v2:172.21.15.122:6836/386534812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b82e396bf0 con 0x55b82e4442b0 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f12ffd640 10 client.8130 handle_command_reply: tid=0 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 client.8130 shutdown 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 msgr2=0x55b82e445f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e445f70 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009960 tx=0x7f7f28004730 comp rx=0 tx=0).stop 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 shutdown_connections 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e449e10 0x55b82e44a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e4442b0 0x55b82e4898f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e445f70 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/1469802090 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e448500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f12ffd640 0 client.8130 ms_handle_reset on v2:172.21.15.122:6836/386534812 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 >> 172.21.15.122:0/1469802090 conn(0x55b82e41f870 msgr2=0x55b82e44d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 shutdown_connections 2024-03-06T07:27:20.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.397+0000 7f7f3d16b740 1 -- 172.21.15.122:0/1469802090 wait complete. 2024-03-06T07:27:20.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.398+0000 7f7f3d16b740 20 client.8130 trim_cache size 0 max 16384 2024-03-06T07:27:20.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.414+0000 7f7f3d16b740 1 Processor -- start 2024-03-06T07:27:20.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.414+0000 7f7f3d16b740 1 -- start start 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e44c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e449e10 0x55b82e4484e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f34200190 con 0x55b82e442350 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340c8160 con 0x55b82e44f600 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340b7b70 con 0x55b82e449e10 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e44c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e44c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51948/0 (socket says 172.21.15.122:51948) 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 -- 172.21.15.122:0/2670935570 learned_addr learned my addr 172.21.15.122:0/2670935570 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 -- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e449e10 msgr2=0x55b82e4484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f337fe640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e449e10 0x55b82e4484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e449e10 0x55b82e4484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 -- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 msgr2=0x55b82e44d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f33fff640 1 -- 172.21.15.122:0/2670935570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b82e4262a0 con 0x55b82e442350 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.415+0000 7f7f337fe640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e449e10 0x55b82e4484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f33fff640 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e44c7e0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x55b82e453cf0 tx=0x7f7f24015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9549d90d716d0b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f197fa640 1 -- 172.21.15.122:0/2670935570 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2403c070 con 0x55b82e442350 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f197fa640 1 -- 172.21.15.122:0/2670935570 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24026040 con 0x55b82e442350 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f197fa640 1 -- 172.21.15.122:0/2670935570 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2401c040 con 0x55b82e442350 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f3d16b740 1 -- 172.21.15.122:0/2670935570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 msgr2=0x55b82e44c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e44c7e0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x55b82e453cf0 tx=0x7f7f24015000 comp rx=0 tx=0).stop 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f3d16b740 1 -- 172.21.15.122:0/2670935570 shutdown_connections 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e449e10 0x55b82e4484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.416+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/2670935570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e442350 0x55b82e44c7e0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.417+0000 7f7f3d16b740 1 -- 172.21.15.122:0/2670935570 >> 172.21.15.122:0/2670935570 conn(0x55b82e41f870 msgr2=0x55b82e41ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.417+0000 7f7f3d16b740 1 -- 172.21.15.122:0/2670935570 shutdown_connections 2024-03-06T07:27:20.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.417+0000 7f7f3d16b740 1 -- 172.21.15.122:0/2670935570 wait complete. 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.417+0000 7f7f3d16b740 1 -- start start 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.418+0000 7f7f3d16b740 1 Processor -- start 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e442350 0x55b82e48df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 0x55b82e4d98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e447460 con 0x55b82e449e10 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340c7b60 con 0x55b82e442350 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b82e447d00 con 0x55b82e44f600 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f3d16b740 10 client.0 _ll_register_callbacks cb 0x7f7f28030970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 0x55b82e4d98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 0x55b82e4d98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51954/0 (socket says 172.21.15.122:51954) 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 -- 172.21.15.122:0/807392662 learned_addr learned my addr 172.21.15.122:0/807392662 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e442350 0x55b82e48df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f38b1e640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 -- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 msgr2=0x55b82e44d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 -- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e442350 msgr2=0x55b82e48df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e442350 0x55b82e48df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f337fe640 1 -- 172.21.15.122:0/807392662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b82e1686b0 con 0x55b82e449e10 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.419+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e442350 0x55b82e48df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f337fe640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 0x55b82e4d98e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009990 tx=0x7f7f28031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc75005bf90490ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f12ffd640 1 -- 172.21.15.122:0/807392662 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28034040 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f12ffd640 1 -- 172.21.15.122:0/807392662 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f28039860 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f24026040 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f12ffd640 1 -- 172.21.15.122:0/807392662 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f2802c000 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 10 client.8136 fetch_fsmap learned FSMap version 61 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55b82e48e500 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f12ffd640 1 -- 172.21.15.122:0/807392662 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24026040 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f12ffd640 1 -- 172.21.15.122:0/807392662 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7f28053070 con 0x55b82e449e10 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 10 client.8136 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:27:20.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 10 client.8136 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 20 client.8136 populate_metadata read hostname 'smithi122' 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e424300 0x55b82e4e5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 4 client.8136 mds_command: new command op to 7962 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 --> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b82e451810 con 0x55b82e424300 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.420+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e424300 0x55b82e4e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f12ffd640 10 client.8136 ms_handle_connect on v2:172.21.15.122:6836/386534812 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f33fff640 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e424300 0x55b82e4e5820 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a565075e01fbf204 server_cookie=576d059c8f641503 in_seq=0 out_seq=0 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f12ffd640 1 -- 172.21.15.122:0/807392662 <== mds.0 v2:172.21.15.122:6836/386534812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b82e451810 con 0x55b82e424300 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f12ffd640 10 client.8136 handle_command_reply: tid=0 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 client.8136 shutdown 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 msgr2=0x55b82e4d98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 0x55b82e4d98e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009990 tx=0x7f7f28031000 comp rx=0 tx=0).stop 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 shutdown_connections 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55b82e44f600 0x55b82e44d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:6836/386534812,v1:172.21.15.122:6837/386534812] conn(0x55b82e424300 0x55b82e4e5820 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b82e449e10 0x55b82e4d98e0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 --2- 172.21.15.122:0/807392662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b82e442350 0x55b82e48df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f12ffd640 0 client.8136 ms_handle_reset on v2:172.21.15.122:6836/386534812 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 >> 172.21.15.122:0/807392662 conn(0x55b82e41f870 msgr2=0x55b82e41adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 shutdown_connections 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 1 -- 172.21.15.122:0/807392662 wait complete. 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.421+0000 7f7f3d16b740 20 client.8136 trim_cache size 0 max 16384 2024-03-06T07:27:20.430 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 7962, 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 9.1615669460000007, 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 61, 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 108, 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 14.207280223 2024-03-06T07:27:20.431 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:27:20.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2006b500 msgr2=0x7f7f2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2006b500 0x7f7f2006d9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7f34156b30 tx=0x7f7f24008040 comp rx=0 tx=0).stop 2024-03-06T07:27:20.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 msgr2=0x7f7f340cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 0x7f7f340cadb0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00c7a0 tx=0x7f7f2c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 shutdown_connections 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7f2006b500 0x7f7f2006d9c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f34162970 0x7f7f340cadb0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f34161cf0 0x7f7f34156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 --2- 172.21.15.122:0/3223378804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f340b4460 0x7f7f340cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 >> 172.21.15.122:0/3223378804 conn(0x7f7f340d3ce0 msgr2=0x7f7f340d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 shutdown_connections 2024-03-06T07:27:20.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.422+0000 7f7f3a581640 1 -- 172.21.15.122:0/3223378804 wait complete. 2024-03-06T07:27:20.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:20.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.534+0000 7fc41d7d0640 1 Processor -- start 2024-03-06T07:27:20.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.534+0000 7fc41d7d0640 1 -- start start 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.534+0000 7fc41d7d0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc4181424f0 0x7fc4180cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.534+0000 7fc41d7d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180cc980 con 0x7fc41806da40 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.534+0000 7fc41d7d0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180ccc60 con 0x7fc4180cd7a0 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.534+0000 7fc41d7d0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180cd220 con 0x7fc4181424f0 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc4177fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc41806da40 0x7fc418071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50008/0 (socket says 172.21.15.122:50008) 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc4177fe640 1 -- 172.21.15.122:0/2900249686 learned_addr learned my addr 172.21.15.122:0/2900249686 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc416ffd640 1 --2- 172.21.15.122:0/2900249686 >> v2:172.21.15.169:3300/0 conn(0x7fc4181424f0 0x7fc4180cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc4167fc640 1 -- 172.21.15.122:0/2900249686 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948962149 0 0) 0x7fc4180cc980 con 0x7fc41806da40 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc4167fc640 1 -- 172.21.15.122:0/2900249686 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc404003520 con 0x7fc41806da40 2024-03-06T07:27:20.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc416ffd640 1 -- 172.21.15.122:0/2900249686 >> v1:172.21.15.169:6789/0 conn(0x7fc4180cd7a0 legacy=0x7fc41806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc416ffd640 1 -- 172.21.15.122:0/2900249686 >> v1:172.21.15.122:6789/0 conn(0x7fc41806da40 legacy=0x7fc418071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.535+0000 7fc416ffd640 1 -- 172.21.15.122:0/2900249686 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180ccf40 con 0x7fc4181424f0 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.536+0000 7fc416ffd640 1 --2- 172.21.15.122:0/2900249686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4181424f0 0x7fc4180cc2f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc4080097a0 tx=0x7fc40802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=528d33dc1e5becde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.536+0000 7fc4167fc640 1 -- 172.21.15.122:0/2900249686 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408034070 con 0x7fc4181424f0 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.536+0000 7fc4167fc640 1 -- 172.21.15.122:0/2900249686 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc408030610 con 0x7fc4181424f0 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.536+0000 7fc4167fc640 1 -- 172.21.15.122:0/2900249686 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4080309c0 con 0x7fc4181424f0 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 -- 172.21.15.122:0/2900249686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4181424f0 msgr2=0x7fc4180cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/2900249686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4181424f0 0x7fc4180cc2f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc4080097a0 tx=0x7fc40802b960 comp rx=0 tx=0).stop 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 -- 172.21.15.122:0/2900249686 shutdown_connections 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/2900249686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4181424f0 0x7fc4180cc2f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 -- 172.21.15.122:0/2900249686 >> 172.21.15.122:0/2900249686 conn(0x7fc4180cf8a0 msgr2=0x7fc4180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 -- 172.21.15.122:0/2900249686 shutdown_connections 2024-03-06T07:27:20.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.537+0000 7fc41d7d0640 1 -- 172.21.15.122:0/2900249686 wait complete. 2024-03-06T07:27:20.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.538+0000 7fc41d7d0640 1 Processor -- start 2024-03-06T07:27:20.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.538+0000 7fc41d7d0640 1 -- start start 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc41806da40 0x7fc4181400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4180cd7a0 0x7fc4181598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 0x7fc418159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815d070 con 0x7fc4181424f0 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815cef0 con 0x7fc4180cd7a0 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815d1f0 con 0x7fc41806da40 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 0x7fc418159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 0x7fc418159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51958/0 (socket says 172.21.15.122:51958) 2024-03-06T07:27:20.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 -- 172.21.15.122:0/315329398 learned_addr learned my addr 172.21.15.122:0/315329398 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc40ffff640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4180cd7a0 0x7fc4181598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc416ffd640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc41806da40 0x7fc4181400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 -- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc41806da40 msgr2=0x7fc4181400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc41806da40 0x7fc4181400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 -- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4180cd7a0 msgr2=0x7fc4181598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4180cd7a0 0x7fc4181598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 -- 172.21.15.122:0/315329398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41815a4e0 con 0x7fc4181424f0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc40ffff640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4180cd7a0 0x7fc4181598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc4177fe640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 0x7fc418159e10 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc41000cae0 tx=0x7fc41000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410017070 con 0x7fc4181424f0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41000ac40 con 0x7fc4181424f0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41815a7c0 con 0x7fc4181424f0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410012640 con 0x7fc4181424f0 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.539+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41815ad80 con 0x7fc4181424f0 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.540+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41804f330 con 0x7fc4181424f0 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.541+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc41001a070 con 0x7fc4181424f0 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.541+0000 7fc40f7fe640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f806b4d0 0x7fc3f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.542+0000 7fc416ffd640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f806b4d0 0x7fc3f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.542+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc41008a0e0 con 0x7fc4181424f0 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.542+0000 7fc416ffd640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f806b4d0 0x7fc3f806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc408036000 tx=0x7fc408008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.546+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc410058ee0 con 0x7fc4181424f0 2024-03-06T07:27:20.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.762+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc4180cbee0 con 0x7fc4181424f0 2024-03-06T07:27:20.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.763+0000 7fc40f7fe640 1 -- 172.21.15.122:0/315329398 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc4100587e0 con 0x7fc4181424f0 2024-03-06T07:27:20.765 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f806b4d0 msgr2=0x7fc3f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f806b4d0 0x7fc3f806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc408036000 tx=0x7fc408008030 comp rx=0 tx=0).stop 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 msgr2=0x7fc418159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 0x7fc418159e10 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc41000cae0 tx=0x7fc41000a060 comp rx=0 tx=0).stop 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 shutdown_connections 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3f806b4d0 0x7fc3f806d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4181424f0 0x7fc418159e10 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4180cd7a0 0x7fc4181598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 --2- 172.21.15.122:0/315329398 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc41806da40 0x7fc4181400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 >> 172.21.15.122:0/315329398 conn(0x7fc4180cf8a0 msgr2=0x7fc418073fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 shutdown_connections 2024-03-06T07:27:20.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:20.765+0000 7fc41d7d0640 1 -- 172.21.15.122:0/315329398 wait complete. 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:20.778 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:20.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.779 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.794 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42400sec preferred_lft 42400sec 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:27:20.795 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.796 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:20.859 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:27:20.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.861 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:27:20.915 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:27:20.916 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:20.916 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:20.916 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:20.916 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:20.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.916 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:27:20.916 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:27:20.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:20.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:20.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:20.916 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:27:21.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.010 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:27:21.025 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:21.025 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:21.025 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:21.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.025 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:27:21.091 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:21.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:21.091 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.146 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:21.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:21.146 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:27:21.267 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:27:21.323 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:27:21.324 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.325 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:27:21.377 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:21.377 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:21.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:27:21.445 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:21.500 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:27:21.500 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:27:21.500 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:27:21.500 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:27:21.500 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:27:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.503 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.503 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:27:21.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:27:21.503+0000 7f4413436480 -1 init, newargv = 0x56545600f700 newargc=15 2024-03-06T07:27:21.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[73035]: starting ceph client 2024-03-06T07:27:21.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[73035]: starting fuse 2024-03-06T07:27:21.520 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:27:21.520 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:27:21.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:27:21.521 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:21.522 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:27:21.640 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.73035.asok 2024-03-06T07:27:21.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:21.651 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.73035.asok 2024-03-06T07:27:21.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.73035.asok status 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "73035", 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:27:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 8148, 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 8148 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 357682647 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 357682647 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.8148 192.168.0.1:0/357682647", 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/357682647", 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 61, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 108, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:27:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:27:21.809 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:27:21.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.815 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:27:21.815 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.830 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:27:21.830 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.831 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.898 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:27:21.956 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:27:21.957 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:22.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.012 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:27:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:8+0 records in 2024-03-06T07:27:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:8+0 records out 2024-03-06T07:27:22.132 INFO:teuthology.orchestra.run.smithi122.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0641828 s, 131 MB/s 2024-03-06T07:27:22.133 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:27:22.133 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:27:22.133 DEBUG:teuthology.orchestra.run.smithi122:> import stat 2024-03-06T07:27:22.133 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:22.133 DEBUG:teuthology.orchestra.run.smithi122:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:27:22.133 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi122.stdout:1099511627777 2024-03-06T07:27:22.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:22.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.207 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:27:22.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.263 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:27:22.324 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:27:22.324 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:27:22.324 DEBUG:teuthology.orchestra.run.smithi122:> import stat 2024-03-06T07:27:22.324 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:22.324 DEBUG:teuthology.orchestra.run.smithi122:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:27:22.324 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:27:22.428 INFO:teuthology.orchestra.run.smithi122.stdout:1099511627777 2024-03-06T07:27:22.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:22.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:22.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.576+0000 7fb45328f640 1 Processor -- start 2024-03-06T07:27:22.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45328f640 1 -- start start 2024-03-06T07:27:22.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45328f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb44c1621c0 0x7fb44c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45328f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 0x7fb44c0d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45328f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0ceac0 con 0x7fb44c0d2bf0 2024-03-06T07:27:22.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45328f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c0cf690 con 0x7fb44c1621c0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45328f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0ced60 con 0x7fb44c0d4920 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45182c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb44c0d2bf0 0x7fb44c15dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50018/0 (socket says 172.21.15.122:50018) 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.577+0000 7fb45182c640 1 -- 172.21.15.122:0/3353284414 learned_addr learned my addr 172.21.15.122:0/3353284414 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395989448 0 0) 0x7fb44c0ceac0 con 0x7fb44c0d2bf0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb45082a640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 0x7fb44c0d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb45102b640 1 --2- 172.21.15.122:0/3353284414 >> v2:172.21.15.169:3300/0 conn(0x7fb44c1621c0 0x7fb44c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb438003520 con 0x7fb44c0d2bf0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 930005133 0 0) 0x7fb438003520 con 0x7fb44c0d2bf0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 msgr2=0x7fb44c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 0x7fb44c0d26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 >> v2:172.21.15.169:3300/0 conn(0x7fb44c1621c0 msgr2=0x7fb44c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 --2- 172.21.15.122:0/3353284414 >> v2:172.21.15.169:3300/0 conn(0x7fb44c1621c0 0x7fb44c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c0cf040 con 0x7fb44c0d2bf0 2024-03-06T07:27:22.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb4440019f0 con 0x7fb44c0d2bf0 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 >> v1:172.21.15.122:6789/0 conn(0x7fb44c0d2bf0 legacy=0x7fb44c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4380039b0 0x7fb438003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb438004800 0x7fb438004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 0x7fb4380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0cf690 con 0x7fb4380039b0 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0ced60 con 0x7fb438004800 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43800b9c0 con 0x7fb44c0d4920 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb45102b640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4380039b0 0x7fb438003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb45082a640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb438004800 0x7fb438004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.578+0000 7fb45182c640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 0x7fb4380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb45102b640 1 -- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 msgr2=0x7fb4380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb45102b640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb44c0d4920 0x7fb4380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb45102b640 1 -- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb438004800 msgr2=0x7fb438004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb45102b640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb438004800 0x7fb438004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb45102b640 1 -- 172.21.15.122:0/3353284414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438003520 con 0x7fb4380039b0 2024-03-06T07:27:22.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb45102b640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4380039b0 0x7fb438003dd0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb448009ac0 tx=0x7fb438005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcdac5860a54803c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448039070 con 0x7fb4380039b0 2024-03-06T07:27:22.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44802c710 con 0x7fb4380039b0 2024-03-06T07:27:22.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.579+0000 7fb443fff640 1 -- 172.21.15.122:0/3353284414 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448009690 con 0x7fb4380039b0 2024-03-06T07:27:22.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45328f640 1 -- 172.21.15.122:0/3353284414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4380039b0 msgr2=0x7fb438003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45328f640 1 --2- 172.21.15.122:0/3353284414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4380039b0 0x7fb438003dd0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb448009ac0 tx=0x7fb438005830 comp rx=0 tx=0).stop 2024-03-06T07:27:22.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45182c640 1 -- 172.21.15.122:0/3353284414 reap_dead start 2024-03-06T07:27:22.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45328f640 1 -- 172.21.15.122:0/3353284414 shutdown_connections 2024-03-06T07:27:22.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45328f640 1 -- 172.21.15.122:0/3353284414 >> 172.21.15.122:0/3353284414 conn(0x7fb44c0d3d10 msgr2=0x7fb44c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45328f640 1 -- 172.21.15.122:0/3353284414 shutdown_connections 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.580+0000 7fb45328f640 1 -- 172.21.15.122:0/3353284414 wait complete. 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.581+0000 7fb45328f640 1 Processor -- start 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.581+0000 7fb45328f640 1 -- start start 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45328f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4380039b0 0x7fb44c14d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45328f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb44c1500f0 0x7fb44c15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45328f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 0x7fb44c15f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45328f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c162720 con 0x7fb44c15f0a0 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45328f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1625a0 con 0x7fb44c1500f0 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45328f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1628a0 con 0x7fb4380039b0 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45182c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 0x7fb44c15f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45182c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 0x7fb44c15f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51980/0 (socket says 172.21.15.122:51980) 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45182c640 1 -- 172.21.15.122:0/1408799487 learned_addr learned my addr 172.21.15.122:0/1408799487 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45082a640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb44c1500f0 0x7fb44c15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.582+0000 7fb45102b640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4380039b0 0x7fb44c14d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45182c640 1 -- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4380039b0 msgr2=0x7fb44c14d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45182c640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4380039b0 0x7fb44c14d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45182c640 1 -- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb44c1500f0 msgr2=0x7fb44c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45182c640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb44c1500f0 0x7fb44c15eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45182c640 1 -- 172.21.15.122:0/1408799487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c09da70 con 0x7fb44c15f0a0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45102b640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4380039b0 0x7fb44c14d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45082a640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb44c1500f0 0x7fb44c15eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45182c640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 0x7fb44c15f520 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fb4440013d0 tx=0x7fb444007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb444017020 con 0x7fb44c15f0a0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44400abc0 con 0x7fb44c15f0a0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c09dd50 con 0x7fb44c15f0a0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4440133f0 con 0x7fb44c15f0a0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.583+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c09fa10 con 0x7fb44c15f0a0 2024-03-06T07:27:22.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.584+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb44401a020 con 0x7fb44c15f0a0 2024-03-06T07:27:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.584+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c04f330 con 0x7fb44c15f0a0 2024-03-06T07:27:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.585+0000 7fb441ffb640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb43006b4d0 0x7fb43006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.585+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb44408aa50 con 0x7fb44c15f0a0 2024-03-06T07:27:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.585+0000 7fb45102b640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb43006b4d0 0x7fb43006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.585+0000 7fb45102b640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb43006b4d0 0x7fb43006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb448036000 tx=0x7fb448030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.589+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb444021020 con 0x7fb44c15f0a0 2024-03-06T07:27:22.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.857+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb44c0ca020 con 0x7fb44c15f0a0 2024-03-06T07:27:22.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.857+0000 7fb441ffb640 1 -- 172.21.15.122:0/1408799487 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fb444059310 con 0x7fb44c15f0a0 2024-03-06T07:27:22.860 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:22.860 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3571233755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3571233755},{"type":"v1","addr":"172.21.15.169:6834","nonce":3571233755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1631232650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1631232650},{"type":"v1","addr":"172.21.15.169:6835","nonce":1631232650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/2235394450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2235394450},{"type":"v1","addr":"172.21.15.122:6835","nonce":2235394450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:12.635136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7962},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7962":{"gid":7962,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/386534812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":386534812},{"type":"v1","addr":"172.21.15.122:6837","nonce":386534812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7962,"qdb_cluster":[7962]},"id":8}]} 2024-03-06T07:27:22.860 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 61 2024-03-06T07:27:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb43006b4d0 msgr2=0x7fb43006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb43006b4d0 0x7fb43006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb448036000 tx=0x7fb448030000 comp rx=0 tx=0).stop 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 msgr2=0x7fb44c15f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 0x7fb44c15f520 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fb4440013d0 tx=0x7fb444007b20 comp rx=0 tx=0).stop 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 shutdown_connections 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb43006b4d0 0x7fb43006d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb44c15f0a0 0x7fb44c15f520 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb44c1500f0 0x7fb44c15eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 --2- 172.21.15.122:0/1408799487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4380039b0 0x7fb44c14d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 >> 172.21.15.122:0/1408799487 conn(0x7fb44c0d3d10 msgr2=0x7fb44c0b7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 shutdown_connections 2024-03-06T07:27:22.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:22.860+0000 7fb45328f640 1 -- 172.21.15.122:0/1408799487 wait complete. 2024-03-06T07:27:22.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:27:23.003 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:27:23.003 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:23.003 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:23.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.042+0000 7fbc6f81cf40 1 Processor -- start 2024-03-06T07:27:23.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.042+0000 7fbc6f81cf40 1 -- start start 2024-03-06T07:27:23.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6f81cf40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 0x563e575055c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6f81cf40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 0x563e5751ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6f81cf40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e575aa8f0 0x563e575032b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6f81cf40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5751d020 con 0x563e574d57d0 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6f81cf40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e575ab210 con 0x563e575aa8f0 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6f81cf40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e5751d2c0 con 0x563e574d8da0 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 0x563e575055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 0x563e575055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51992/0 (socket says 172.21.15.122:51992) 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 -- 172.21.15.122:0/1631938410 learned_addr learned my addr 172.21.15.122:0/1631938410 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 -- 172.21.15.122:0/1631938410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 msgr2=0x563e5751ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 0x563e5751ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 -- 172.21.15.122:0/1631938410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e575aa8f0 msgr2=0x563e575032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e575aa8f0 0x563e575032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 -- 172.21.15.122:0/1631938410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e57402c50 con 0x563e574d57d0 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc6d31d640 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 0x563e575055c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000a440 tx=0x7fbc6000a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc9d69df12b93e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.043+0000 7fbc5f7fe640 1 -- 172.21.15.122:0/1631938410 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6000b930 con 0x563e574d57d0 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc5f7fe640 1 -- 172.21.15.122:0/1631938410 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6000f030 con 0x563e574d57d0 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc5f7fe640 1 -- 172.21.15.122:0/1631938410 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6000bcf0 con 0x563e574d57d0 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1631938410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 msgr2=0x563e575055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 0x563e575055c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000a440 tx=0x7fbc6000a900 comp rx=0 tx=0).stop 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1631938410 shutdown_connections 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e575aa8f0 0x563e575032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 0x563e5751ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1631938410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e574d57d0 0x563e575055c0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1631938410 >> 172.21.15.122:0/1631938410 conn(0x563e57500790 msgr2=0x563e575a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1631938410 shutdown_connections 2024-03-06T07:27:23.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.044+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1631938410 wait complete. 2024-03-06T07:27:23.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.045+0000 7fbc6f81cf40 1 Processor -- start 2024-03-06T07:27:23.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.045+0000 7fbc6f81cf40 1 -- start start 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6f81cf40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e574d57d0 0x563e576310f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6f81cf40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 0x563e575a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6f81cf40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e575aa8f0 0x563e575032b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6f81cf40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5751c750 con 0x563e575aa8f0 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6f81cf40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5751cca0 con 0x563e574d57d0 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6f81cf40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e57505120 con 0x563e574d8da0 2024-03-06T07:27:23.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6db1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e575aa8f0 0x563e575032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6db1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e575aa8f0 0x563e575032b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51998/0 (socket says 172.21.15.122:51998) 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6db1e640 1 -- 172.21.15.122:0/1624400445 learned_addr learned my addr 172.21.15.122:0/1624400445 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6d31d640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e574d57d0 0x563e576310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.046+0000 7fbc6cb1c640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 0x563e575a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6db1e640 1 -- 172.21.15.122:0/1624400445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 msgr2=0x563e575a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6db1e640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x563e574d8da0 0x563e575a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6db1e640 1 -- 172.21.15.122:0/1624400445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e574d57d0 msgr2=0x563e576310f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6db1e640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e574d57d0 0x563e576310f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6db1e640 1 -- 172.21.15.122:0/1624400445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e576316d0 con 0x563e575aa8f0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6db1e640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e575aa8f0 0x563e575032b0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fbc68005f90 tx=0x7fbc680084f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc5e7fc640 1 -- 172.21.15.122:0/1624400445 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68010640 con 0x563e575aa8f0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc5e7fc640 1 -- 172.21.15.122:0/1624400445 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc68015070 con 0x563e575aa8f0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563e575a5290 con 0x563e575aa8f0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc5e7fc640 1 -- 172.21.15.122:0/1624400445 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68010a20 con 0x563e575aa8f0 2024-03-06T07:27:23.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.047+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e5751b080 con 0x563e575aa8f0 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.048+0000 7fbc5e7fc640 1 -- 172.21.15.122:0/1624400445 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc68010bc0 con 0x563e575aa8f0 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc5e7fc640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc50069eb0 0x7fbc5006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc5e7fc640 1 -- 172.21.15.122:0/1624400445 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc68088b30 con 0x563e575aa8f0 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc6d31d640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc50069eb0 0x7fbc5006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x563e575a1810 0x563e575193c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x563e5751b5a0 con 0x563e575a1810 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc6cb1c640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x563e575a1810 0x563e575193c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc6d31d640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc50069eb0 0x7fbc5006c370 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000a940 tx=0x7fbc60008000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.049+0000 7fbc6cb1c640 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x563e575a1810 0x563e575193c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6cb1c640 1 -- 172.21.15.122:0/1624400445 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fbc6405d040 con 0x563e575a1810 2024-03-06T07:27:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x563e575a1810 msgr2=0x563e575193c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x563e575a1810 0x563e575193c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc50069eb0 msgr2=0x7fbc5006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc50069eb0 0x7fbc5006c370 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000a940 tx=0x7fbc60008000 comp rx=0 tx=0).stop 2024-03-06T07:27:23.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e575aa8f0 msgr2=0x563e575032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 --2- 172.21.15.122:0/1624400445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e575aa8f0 0x563e575032b0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fbc68005f90 tx=0x7fbc680084f0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6db1e640 1 -- 172.21.15.122:0/1624400445 reap_dead start 2024-03-06T07:27:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 shutdown_connections 2024-03-06T07:27:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 >> 172.21.15.122:0/1624400445 conn(0x563e57500790 msgr2=0x563e575a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 shutdown_connections 2024-03-06T07:27:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.050+0000 7fbc6f81cf40 1 -- 172.21.15.122:0/1624400445 wait complete. 2024-03-06T07:27:23.056 INFO:teuthology.orchestra.run.smithi122.stdout:Wfile_a 2024-03-06T07:27:23.056 INFO:teuthology.orchestra.run.smithi122.stdout:dir_1 2024-03-06T07:27:23.056 DEBUG:teuthology.orchestra.run.smithi122:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:27:23.190 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:27:23.190 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 1099511627777, 2024-03-06T07:27:23.190 INFO:teuthology.orchestra.run.smithi122.stdout: "ancestors": [ 2024-03-06T07:27:23.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:23.190 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1099511627776, 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "file_a", 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 10 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1, 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "dir_1", 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 17 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "pool": 17, 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: "old_pools": [] 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:27:23.191 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:23.215 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:27:23.216 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:23.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:23.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 Processor -- start 2024-03-06T07:27:23.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 -- start start 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00d4760 0x7ff4c00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4c00d0a80 0x7ff4c00cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00d25d0 con 0x7ff4c00b4170 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00d4bf0 con 0x7ff4c00d0a80 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.370+0000 7ff4c5b79640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00ce9e0 con 0x7ff4c00d4760 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4c00d0a80 0x7ff4c00cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4c00d0a80 0x7ff4c00cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52002/0 (socket says 172.21.15.122:52002) 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 -- 172.21.15.122:0/2838649808 learned_addr learned my addr 172.21.15.122:0/2838649808 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:23.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4beffd640 1 --2- 172.21.15.122:0/2838649808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00d4760 0x7ff4c00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 -- 172.21.15.122:0/2838649808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00d4760 msgr2=0x7ff4c00b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 --2- 172.21.15.122:0/2838649808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00d4760 0x7ff4c00b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 -- 172.21.15.122:0/2838649808 >> v1:172.21.15.169:6789/0 conn(0x7ff4c00b4170 legacy=0x7ff4c00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.371+0000 7ff4bffff640 1 -- 172.21.15.122:0/2838649808 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00d2870 con 0x7ff4c00d0a80 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4bffff640 1 --2- 172.21.15.122:0/2838649808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00cee70 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009e50 tx=0x7ff4b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b11ba8ee59c572 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4be7fc640 1 -- 172.21.15.122:0/2838649808 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b802ca70 con 0x7ff4c00d0a80 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4be7fc640 1 -- 172.21.15.122:0/2838649808 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b802cc10 con 0x7ff4c00d0a80 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4be7fc640 1 -- 172.21.15.122:0/2838649808 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80343f0 con 0x7ff4c00d0a80 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 -- 172.21.15.122:0/2838649808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00d0a80 msgr2=0x7ff4c00cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/2838649808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00cee70 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009e50 tx=0x7ff4b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 -- 172.21.15.122:0/2838649808 shutdown_connections 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/2838649808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00cee70 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/2838649808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00d4760 0x7ff4c00b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 -- 172.21.15.122:0/2838649808 >> 172.21.15.122:0/2838649808 conn(0x7ff4c00d3c40 msgr2=0x7ff4c00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 -- 172.21.15.122:0/2838649808 shutdown_connections 2024-03-06T07:27:23.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.372+0000 7ff4c5b79640 1 -- 172.21.15.122:0/2838649808 wait complete. 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.373+0000 7ff4c5b79640 1 Processor -- start 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 -- start start 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00b4170 0x7ff4c01f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 0x7ff4c01555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00b4dd0 con 0x7ff4c00b9490 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00b4550 con 0x7ff4c00d0a80 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4c5b79640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00b4f50 con 0x7ff4c00b4170 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 0x7ff4c01555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 0x7ff4c01555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52018/0 (socket says 172.21.15.122:52018) 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.374+0000 7ff4bffff640 1 -- 172.21.15.122:0/1086397471 learned_addr learned my addr 172.21.15.122:0/1086397471 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bf7fe640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00b4170 0x7ff4c01f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4beffd640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bffff640 1 -- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00b4170 msgr2=0x7ff4c01f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bffff640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00b4170 0x7ff4c01f7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bffff640 1 -- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c00d0a80 msgr2=0x7ff4c00b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bffff640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bffff640 1 -- 172.21.15.122:0/1086397471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00cbcb0 con 0x7ff4c00b9490 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4bffff640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 0x7ff4c01555f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8036040 tx=0x7ff4b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8043020 con 0x7ff4c00b9490 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b803e450 con 0x7ff4c00b9490 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8009510 con 0x7ff4c00b9490 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c00c9470 con 0x7ff4c00b9490 2024-03-06T07:27:23.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.375+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c00c9a30 con 0x7ff4c00b9490 2024-03-06T07:27:23.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.376+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c004f330 con 0x7ff4c00b9490 2024-03-06T07:27:23.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.377+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4b8046020 con 0x7ff4c00b9490 2024-03-06T07:27:23.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.377+0000 7ff4a7fff640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.377+0000 7ff4bf7fe640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.377+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4b80ac8a0 con 0x7ff4c00b9490 2024-03-06T07:27:23.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.378+0000 7ff4bf7fe640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0004f60 tx=0x7ff4b0007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.382+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4b804d030 con 0x7ff4c00b9490 2024-03-06T07:27:23.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.617+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c00733b0 con 0x7ff4c00b9490 2024-03-06T07:27:23.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.618+0000 7ff4a7fff640 1 -- 172.21.15.122:0/1086397471 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16627 (secure 0 0 0) 0x7ff4c00cbcb0 con 0x7ff4c00b9490 2024-03-06T07:27:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:23.621 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":108,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:12.269669+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.620+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4ac06b4d0 msgr2=0x7ff4ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.620+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0004f60 tx=0x7ff4b0007e30 comp rx=0 tx=0).stop 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.620+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 msgr2=0x7ff4c01555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.620+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 0x7ff4c01555f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8036040 tx=0x7ff4b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 shutdown_connections 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c00b9490 0x7ff4c01555f0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c00d0a80 0x7ff4c00b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 --2- 172.21.15.122:0/1086397471 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff4c00b4170 0x7ff4c01f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 >> 172.21.15.122:0/1086397471 conn(0x7ff4c00d3c40 msgr2=0x7ff4c00d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 shutdown_connections 2024-03-06T07:27:23.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.621+0000 7ff4c5b79640 1 -- 172.21.15.122:0/1086397471 wait complete. 2024-03-06T07:27:23.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.843386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.579477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:23.634 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.736+0000 7fc34ef93640 1 Processor -- start 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.737+0000 7fc34ef93640 1 -- start start 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34ef93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc3480b1e40 0x7fc348168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34ef93640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3480d2730 con 0x7fc3480d4860 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34ef93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3481618b0 con 0x7fc3481614d0 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34ef93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3480d4cf0 con 0x7fc3480b1e40 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34d530640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc3480b1e40 0x7fc348168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34d530640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc3480b1e40 0x7fc348168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52028/0 (socket says 172.21.15.122:52028) 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc347fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc3480d4860 0x7fc3480d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50032/0 (socket says 172.21.15.122:50032) 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34d530640 1 -- 172.21.15.122:0/3082730018 learned_addr learned my addr 172.21.15.122:0/3082730018 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34d530640 1 -- 172.21.15.122:0/3082730018 >> v1:172.21.15.169:6789/0 conn(0x7fc3481614d0 legacy=0x7fc3480d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34d530640 1 -- 172.21.15.122:0/3082730018 >> v1:172.21.15.122:6789/0 conn(0x7fc3480d4860 legacy=0x7fc3480d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc3477fe640 1 -- 172.21.15.122:0/3082730018 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180026439 0 0) 0x7fc3480d2730 con 0x7fc3480d4860 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.738+0000 7fc34d530640 1 -- 172.21.15.122:0/3082730018 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3480cf400 con 0x7fc3480b1e40 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.739+0000 7fc34d530640 1 --2- 172.21.15.122:0/3082730018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480b1e40 0x7fc348168b60 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc3480cf5d0 tx=0x7fc34002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a8d205fa468420 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.739+0000 7fc3477fe640 1 -- 172.21.15.122:0/3082730018 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34002caf0 con 0x7fc3480b1e40 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.739+0000 7fc3477fe640 1 -- 172.21.15.122:0/3082730018 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34002cc90 con 0x7fc3480b1e40 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.739+0000 7fc3477fe640 1 -- 172.21.15.122:0/3082730018 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc340034610 con 0x7fc3480b1e40 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 -- 172.21.15.122:0/3082730018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480b1e40 msgr2=0x7fc348168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 --2- 172.21.15.122:0/3082730018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480b1e40 0x7fc348168b60 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc3480cf5d0 tx=0x7fc34002bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 -- 172.21.15.122:0/3082730018 shutdown_connections 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 --2- 172.21.15.122:0/3082730018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480b1e40 0x7fc348168b60 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 -- 172.21.15.122:0/3082730018 >> 172.21.15.122:0/3082730018 conn(0x7fc3480d3d40 msgr2=0x7fc3480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 -- 172.21.15.122:0/3082730018 shutdown_connections 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.740+0000 7fc34ef93640 1 -- 172.21.15.122:0/3082730018 wait complete. 2024-03-06T07:27:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.741+0000 7fc34ef93640 1 Processor -- start 2024-03-06T07:27:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.741+0000 7fc34ef93640 1 -- start start 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3480b1e40 0x7fc3480b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 0x7fc3481540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3481614d0 0x7fc34815e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3480cfc60 con 0x7fc3480d4860 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3480cf980 con 0x7fc3480b1e40 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3480b0910 con 0x7fc3481614d0 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 0x7fc3481540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 0x7fc3481540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52042/0 (socket says 172.21.15.122:52042) 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 -- 172.21.15.122:0/2484876414 learned_addr learned my addr 172.21.15.122:0/2484876414 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34cd2f640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3480b1e40 0x7fc3480b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34d530640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3481614d0 0x7fc34815e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 -- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3481614d0 msgr2=0x7fc34815e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3481614d0 0x7fc34815e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 -- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3480b1e40 msgr2=0x7fc3480b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3480b1e40 0x7fc3480b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 -- 172.21.15.122:0/2484876414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3480b9070 con 0x7fc3480d4860 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc347fff640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 0x7fc3481540b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c4d0 tx=0x7fc33400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334018070 con 0x7fc3480d4860 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3340074e0 con 0x7fc3480d4860 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.742+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3480b9350 con 0x7fc3480d4860 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.743+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334004a20 con 0x7fc3480d4860 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.743+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3480b9910 con 0x7fc3480d4860 2024-03-06T07:27:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.744+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc348068ff0 con 0x7fc3480d4860 2024-03-06T07:27:23.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.744+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc33401b070 con 0x7fc3480d4860 2024-03-06T07:27:23.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.744+0000 7fc3457fa640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc32c06b500 0x7fc32c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.745+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3340894c0 con 0x7fc3480d4860 2024-03-06T07:27:23.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.745+0000 7fc34cd2f640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc32c06b500 0x7fc32c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.745+0000 7fc34cd2f640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc32c06b500 0x7fc32c06d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc33c000f60 tx=0x7fc33c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.753+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc334058340 con 0x7fc3480d4860 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.941+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3480d0950 con 0x7fc3480d4860 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.941+0000 7fc3457fa640 1 -- 172.21.15.122:0/2484876414 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc334057c40 con 0x7fc3480d4860 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.943+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc32c06b500 msgr2=0x7fc32c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.943+0000 7fc34ef93640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc32c06b500 0x7fc32c06d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc33c000f60 tx=0x7fc33c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 msgr2=0x7fc3481540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 0x7fc3481540b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c4d0 tx=0x7fc33400c990 comp rx=0 tx=0).stop 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 shutdown_connections 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc32c06b500 0x7fc32c06d9c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3481614d0 0x7fc34815e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3480d4860 0x7fc3481540b0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 --2- 172.21.15.122:0/2484876414 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3480b1e40 0x7fc3480b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 >> 172.21.15.122:0/2484876414 conn(0x7fc3480d3d40 msgr2=0x7fc3480cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 shutdown_connections 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:23.944+0000 7fc34ef93640 1 -- 172.21.15.122:0/2484876414 wait complete. 2024-03-06T07:27:23.957 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:24.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.061+0000 7ff585113640 1 Processor -- start 2024-03-06T07:27:24.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.061+0000 7ff585113640 1 -- start start 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff585113640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff5800d10c0 0x7ff5800d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff585113640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5800d14a0 0x7ff58015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff585113640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff58015eb00 0x7ff5800cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff585113640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58015c050 con 0x7ff5800d14a0 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff585113640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5800d0060 con 0x7ff5800d10c0 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff585113640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800cfa50 con 0x7ff58015eb00 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff577fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5800d14a0 0x7ff58015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff577fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5800d14a0 0x7ff58015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52052/0 (socket says 172.21.15.122:52052) 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.062+0000 7ff577fff640 1 -- 172.21.15.122:0/1265218209 learned_addr learned my addr 172.21.15.122:0/1265218209 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff577fff640 1 -- 172.21.15.122:0/1265218209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff58015eb00 msgr2=0x7ff5800cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/1265218209 >> v2:172.21.15.169:3300/0 conn(0x7ff5800d10c0 0x7ff5800d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff57f59e640 1 --2- 172.21.15.122:0/1265218209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff58015eb00 0x7ff5800cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff577fff640 1 --2- 172.21.15.122:0/1265218209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff58015eb00 0x7ff5800cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff577fff640 1 -- 172.21.15.122:0/1265218209 >> v2:172.21.15.169:3300/0 conn(0x7ff5800d10c0 msgr2=0x7ff5800d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff577fff640 1 --2- 172.21.15.122:0/1265218209 >> v2:172.21.15.169:3300/0 conn(0x7ff5800d10c0 0x7ff5800d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.063+0000 7ff577fff640 1 -- 172.21.15.122:0/1265218209 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800d0a30 con 0x7ff5800d14a0 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.064+0000 7ff577fff640 1 --2- 172.21.15.122:0/1265218209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800d14a0 0x7ff58015e5c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7ff5800718d0 tx=0x7ff56802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad0e34da4b4024d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.064+0000 7ff57e59c640 1 -- 172.21.15.122:0/1265218209 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568035020 con 0x7ff5800d14a0 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.064+0000 7ff57e59c640 1 -- 172.21.15.122:0/1265218209 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568031870 con 0x7ff5800d14a0 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.064+0000 7ff57e59c640 1 -- 172.21.15.122:0/1265218209 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568031c40 con 0x7ff5800d14a0 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.064+0000 7ff585113640 1 -- 172.21.15.122:0/1265218209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800d14a0 msgr2=0x7ff58015e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.064+0000 7ff585113640 1 --2- 172.21.15.122:0/1265218209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800d14a0 0x7ff58015e5c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7ff5800718d0 tx=0x7ff56802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 -- 172.21.15.122:0/1265218209 shutdown_connections 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 --2- 172.21.15.122:0/1265218209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800d14a0 0x7ff58015e5c0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 --2- 172.21.15.122:0/1265218209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff58015eb00 0x7ff5800cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 --2- 172.21.15.122:0/1265218209 >> v2:172.21.15.169:3300/0 conn(0x7ff5800d10c0 0x7ff5800d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 -- 172.21.15.122:0/1265218209 >> 172.21.15.122:0/1265218209 conn(0x7ff5800b3800 msgr2=0x7ff5800b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 -- 172.21.15.122:0/1265218209 shutdown_connections 2024-03-06T07:27:24.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.065+0000 7ff585113640 1 -- 172.21.15.122:0/1265218209 wait complete. 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.066+0000 7ff585113640 1 Processor -- start 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.066+0000 7ff585113640 1 -- start start 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d10c0 0x7ff5800ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d14a0 0x7ff58006c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 0x7ff580152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015f440 con 0x7ff58015eb00 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015f160 con 0x7ff5800d10c0 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800cfcf0 con 0x7ff5800d14a0 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 0x7ff580152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 0x7ff580152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52062/0 (socket says 172.21.15.122:52062) 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 -- 172.21.15.122:0/3608720174 learned_addr learned my addr 172.21.15.122:0/3608720174 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d10c0 0x7ff5800ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff577fff640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d14a0 0x7ff58006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 -- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d14a0 msgr2=0x7ff58006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d14a0 0x7ff58006c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 -- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d10c0 msgr2=0x7ff5800ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d10c0 0x7ff5800ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 -- 172.21.15.122:0/3608720174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800b6460 con 0x7ff58015eb00 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff57f59e640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 0x7ff580152670 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff57800cd70 tx=0x7ff57800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578003b90 con 0x7ff58015eb00 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578003d30 con 0x7ff58015eb00 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5800b5e00 con 0x7ff58015eb00 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578004240 con 0x7ff58015eb00 2024-03-06T07:27:24.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.067+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5800b5060 con 0x7ff58015eb00 2024-03-06T07:27:24.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.069+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5800b7dc0 con 0x7ff58015eb00 2024-03-06T07:27:24.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.069+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5780043e0 con 0x7ff58015eb00 2024-03-06T07:27:24.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.069+0000 7ff5777fe640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.069+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff57808abf0 con 0x7ff58015eb00 2024-03-06T07:27:24.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.069+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.070+0000 7ff57ed9d640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff57000a9b0 tx=0x7ff570008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.074+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5800b7dc0 con 0x7ff58015eb00 2024-03-06T07:27:24.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.252+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff58009fe60 con 0x7ff58015eb00 2024-03-06T07:27:24.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.252+0000 7ff5777fe640 1 -- 172.21.15.122:0/3608720174 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff578059420 con 0x7ff58015eb00 2024-03-06T07:27:24.254 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:27:24.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 msgr2=0x7ff56006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff57000a9b0 tx=0x7ff570008040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 msgr2=0x7ff580152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 0x7ff580152670 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff57800cd70 tx=0x7ff57800a660 comp rx=0 tx=0).stop 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 shutdown_connections 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff56006b4d0 0x7ff56006d990 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff58015eb00 0x7ff580152670 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5800d14a0 0x7ff58006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 --2- 172.21.15.122:0/3608720174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800d10c0 0x7ff5800ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 >> 172.21.15.122:0/3608720174 conn(0x7ff5800b3800 msgr2=0x7ff58015cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 shutdown_connections 2024-03-06T07:27:24.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.254+0000 7ff585113640 1 -- 172.21.15.122:0/3608720174 wait complete. 2024-03-06T07:27:24.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.369+0000 7f5448fb5640 1 Processor -- start 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.369+0000 7f5448fb5640 1 -- start start 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.370+0000 7f5448fb5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54440d4860 0x7f54440d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.370+0000 7f5448fb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 0x7f54440b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.370+0000 7f5448fb5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54441618b0 con 0x7f54441614d0 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.370+0000 7f5448fb5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54440d4cf0 con 0x7f54440d4860 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.370+0000 7f5448fb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54440cf400 con 0x7f54440b0e30 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544259c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f54441614d0 0x7f54440d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50036/0 (socket says 172.21.15.122:50036) 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544259c640 1 -- 172.21.15.122:0/2816642014 learned_addr learned my addr 172.21.15.122:0/2816642014 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:24.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700959313 0 0) 0x7f54441618b0 con 0x7f54441614d0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c003520 con 0x7f54441614d0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f5442d9d640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 0x7f54440b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f5441d9b640 1 --2- 172.21.15.122:0/2816642014 >> v2:172.21.15.169:3300/0 conn(0x7f54440d4860 0x7f54440d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 845464946 0 0) 0x7f542c003520 con 0x7f54441614d0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 msgr2=0x7f54440b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 0x7f54440b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 >> v2:172.21.15.169:3300/0 conn(0x7f54440d4860 msgr2=0x7f54440d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 --2- 172.21.15.122:0/2816642014 >> v2:172.21.15.169:3300/0 conn(0x7f54440d4860 0x7f54440d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440cf6a0 con 0x7f54441614d0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f543c001670 con 0x7f54441614d0 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 >> v1:172.21.15.122:6789/0 conn(0x7f54441614d0 legacy=0x7f54440d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f542c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f542c004800 0x7f542c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 0x7f542c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440d4cf0 con 0x7f542c0039b0 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440cf400 con 0x7f542c004800 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c00b9c0 con 0x7f54440b0e30 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f544259c640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f542c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f5441d9b640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f542c004800 0x7f542c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.371+0000 7f5442d9d640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 0x7f542c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544259c640 1 -- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 msgr2=0x7f542c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544259c640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b0e30 0x7f542c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544259c640 1 -- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f542c004800 msgr2=0x7f542c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544259c640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f542c004800 0x7f542c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544259c640 1 -- 172.21.15.122:0/2816642014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c003520 con 0x7f542c0039b0 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544259c640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f542c003dd0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f543c001050 tx=0x7f542c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fdae98b3afe143 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c007d70 con 0x7f542c0039b0 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c030070 con 0x7f542c0039b0 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.372+0000 7f544159a640 1 -- 172.21.15.122:0/2816642014 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c02cd80 con 0x7f542c0039b0 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5448fb5640 1 -- 172.21.15.122:0/2816642014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 msgr2=0x7f542c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5448fb5640 1 --2- 172.21.15.122:0/2816642014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f542c003dd0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f543c001050 tx=0x7f542c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5442d9d640 1 -- 172.21.15.122:0/2816642014 reap_dead start 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5448fb5640 1 -- 172.21.15.122:0/2816642014 shutdown_connections 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5448fb5640 1 -- 172.21.15.122:0/2816642014 >> 172.21.15.122:0/2816642014 conn(0x7f54440d3d40 msgr2=0x7f54440d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5448fb5640 1 -- 172.21.15.122:0/2816642014 shutdown_connections 2024-03-06T07:27:24.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.373+0000 7f5448fb5640 1 -- 172.21.15.122:0/2816642014 wait complete. 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.374+0000 7f5448fb5640 1 Processor -- start 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.374+0000 7f5448fb5640 1 -- start start 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5448fb5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f54440caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5448fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5444152250 0x7f54440cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5448fb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b3540 0x7f54440b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5448fb5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440b0910 con 0x7f542c0039b0 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5448fb5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440cfc60 con 0x7f5444152250 2024-03-06T07:27:24.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5448fb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54440d2560 con 0x7f54440b3540 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f544259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f54440caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f544259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f54440caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52076/0 (socket says 172.21.15.122:52076) 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f544259c640 1 -- 172.21.15.122:0/3117490661 learned_addr learned my addr 172.21.15.122:0/3117490661 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5441d9b640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5444152250 0x7f54440cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.375+0000 7f5442d9d640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b3540 0x7f54440b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f544259c640 1 -- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b3540 msgr2=0x7f54440b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f544259c640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b3540 0x7f54440b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f544259c640 1 -- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5444152250 msgr2=0x7f54440cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f544259c640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5444152250 0x7f54440cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f544259c640 1 -- 172.21.15.122:0/3117490661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54441526d0 con 0x7f542c0039b0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f5442d9d640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b3540 0x7f54440b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f544259c640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f54440caf30 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f543c00c300 tx=0x7f543c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c045070 con 0x7f542c0039b0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c004bf0 con 0x7f542c0039b0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54440cdf10 con 0x7f542c0039b0 2024-03-06T07:27:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c007d50 con 0x7f542c0039b0 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.376+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54440b6d40 con 0x7f542c0039b0 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.377+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5444068ff0 con 0x7f542c0039b0 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.377+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f543c007070 con 0x7f542c0039b0 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.378+0000 7f5432ffd640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f542406b4d0 0x7f542406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.378+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f543c0ab250 con 0x7f542c0039b0 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.378+0000 7f5441d9b640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f542406b4d0 0x7f542406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.378+0000 7f5441d9b640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f542406b4d0 0x7f542406d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f54440b99a0 tx=0x7f5438009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.382+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f543c07a300 con 0x7f542c0039b0 2024-03-06T07:27:24.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:24.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:24.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.581+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f54440c71c0 con 0x7f542c0039b0 2024-03-06T07:27:24.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.581+0000 7f5432ffd640 1 -- 172.21.15.122:0/3117490661 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f543c079c00 con 0x7f542c0039b0 2024-03-06T07:27:24.583 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:27:24.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f542406b4d0 msgr2=0x7f542406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f542406b4d0 0x7f542406d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f54440b99a0 tx=0x7f5438009190 comp rx=0 tx=0).stop 2024-03-06T07:27:24.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 msgr2=0x7f54440caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f54440caf30 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f543c00c300 tx=0x7f543c000fe0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 shutdown_connections 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f542406b4d0 0x7f542406d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54440b3540 0x7f54440b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5444152250 0x7f54440cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 --2- 172.21.15.122:0/3117490661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f542c0039b0 0x7f54440caf30 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 >> 172.21.15.122:0/3117490661 conn(0x7f54440d3d40 msgr2=0x7f54440cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 shutdown_connections 2024-03-06T07:27:24.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.583+0000 7f5448fb5640 1 -- 172.21.15.122:0/3117490661 wait complete. 2024-03-06T07:27:24.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:24.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 Processor -- start 2024-03-06T07:27:24.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 -- start start 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f80b4c00 0x7fa0f80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa0f815ba50 0x7fa0f815be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f815d510 con 0x7fa0f80b28a0 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f815e0b0 con 0x7fa0f815ba50 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.695+0000 7fa0fd46b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80b33f0 con 0x7fa0f80b4c00 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa0f815ba50 0x7fa0f815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0ee7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa0f80b28a0 0x7fa0f80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42926/0 (socket says 172.21.15.122:42926) 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0ee7fc640 1 -- 172.21.15.122:0/59429436 learned_addr learned my addr 172.21.15.122:0/59429436 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:24.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f6ffd640 1 --2- 172.21.15.122:0/59429436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f80b4c00 0x7fa0f80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f67fc640 1 -- 172.21.15.122:0/59429436 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293673903 0 0) 0x7fa0f815d510 con 0x7fa0f80b28a0 2024-03-06T07:27:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f67fc640 1 -- 172.21.15.122:0/59429436 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e4003520 con 0x7fa0f80b28a0 2024-03-06T07:27:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f77fe640 1 -- 172.21.15.122:0/59429436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f80b4c00 msgr2=0x7fa0f80b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f77fe640 1 --2- 172.21.15.122:0/59429436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f80b4c00 0x7fa0f80b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f77fe640 1 -- 172.21.15.122:0/59429436 >> v1:172.21.15.122:6789/0 conn(0x7fa0f80b28a0 legacy=0x7fa0f80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.696+0000 7fa0f77fe640 1 -- 172.21.15.122:0/59429436 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f815d780 con 0x7fa0f815ba50 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0f77fe640 1 --2- 172.21.15.122:0/59429436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f815ba50 0x7fa0f815be90 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009d90 tx=0x7fa0f002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aea0e0dacd9a7b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0f67fc640 1 -- 172.21.15.122:0/59429436 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0039070 con 0x7fa0f815ba50 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0f67fc640 1 -- 172.21.15.122:0/59429436 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f002cce0 con 0x7fa0f815ba50 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0f67fc640 1 -- 172.21.15.122:0/59429436 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0034660 con 0x7fa0f815ba50 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0fd46b640 1 -- 172.21.15.122:0/59429436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f815ba50 msgr2=0x7fa0f815be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/59429436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f815ba50 0x7fa0f815be90 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009d90 tx=0x7fa0f002bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0fd46b640 1 -- 172.21.15.122:0/59429436 shutdown_connections 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/59429436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f815ba50 0x7fa0f815be90 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/59429436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f80b4c00 0x7fa0f80b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.697+0000 7fa0fd46b640 1 -- 172.21.15.122:0/59429436 >> 172.21.15.122:0/59429436 conn(0x7fa0f809a9f0 msgr2=0x7fa0f809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.698+0000 7fa0fd46b640 1 -- 172.21.15.122:0/59429436 shutdown_connections 2024-03-06T07:27:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.698+0000 7fa0fd46b640 1 -- 172.21.15.122:0/59429436 wait complete. 2024-03-06T07:27:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.699+0000 7fa0fd46b640 1 Processor -- start 2024-03-06T07:27:24.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.700+0000 7fa0fd46b640 1 -- start start 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0f80b28a0 0x7fa0f81499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 0x7fa0f8145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f815ba50 0x7fa0f814c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f815f550 con 0x7fa0f80b4c00 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f815dd40 con 0x7fa0f80b28a0 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f815f6d0 con 0x7fa0f815ba50 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 0x7fa0f8145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 0x7fa0f8145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37442/0 (socket says 172.21.15.122:37442) 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 -- 172.21.15.122:0/2948814685 learned_addr learned my addr 172.21.15.122:0/2948814685 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0f77fe640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f815ba50 0x7fa0f814c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0f6ffd640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0f80b28a0 0x7fa0f81499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 -- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f815ba50 msgr2=0x7fa0f814c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f815ba50 0x7fa0f814c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 -- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0f80b28a0 msgr2=0x7fa0f81499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0f80b28a0 0x7fa0f81499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 -- 172.21.15.122:0/2948814685 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f8146590 con 0x7fa0f80b4c00 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0ee7fc640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 0x7fa0f8145c50 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c460 tx=0x7fa0e000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0017070 con 0x7fa0f80b4c00 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0005ce0 con 0x7fa0f80b4c00 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0007290 con 0x7fa0f80b4c00 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.701+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f8146870 con 0x7fa0f80b4c00 2024-03-06T07:27:24.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.702+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f8146e30 con 0x7fa0f80b4c00 2024-03-06T07:27:24.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.702+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f804f330 con 0x7fa0f80b4c00 2024-03-06T07:27:24.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.703+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0e001a070 con 0x7fa0f80b4c00 2024-03-06T07:27:24.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.704+0000 7fa0effff640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0d806b500 0x7fa0d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.704+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0e00894b0 con 0x7fa0f80b4c00 2024-03-06T07:27:24.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.704+0000 7fa0f6ffd640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0d806b500 0x7fa0d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.704+0000 7fa0f6ffd640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0d806b500 0x7fa0d806d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009ad0 tx=0x7fa0e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.708+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0e00583c0 con 0x7fa0f80b4c00 2024-03-06T07:27:24.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.897+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa0f8146190 con 0x7fa0f80b4c00 2024-03-06T07:27:24.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.897+0000 7fa0effff640 1 -- 172.21.15.122:0/2948814685 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa0e0057cc0 con 0x7fa0f80b4c00 2024-03-06T07:27:24.899 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:27:24.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0d806b500 msgr2=0x7fa0d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0d806b500 0x7fa0d806d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009ad0 tx=0x7fa0e8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 msgr2=0x7fa0f8145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 0x7fa0f8145c50 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c460 tx=0x7fa0e000c920 comp rx=0 tx=0).stop 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 shutdown_connections 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa0d806b500 0x7fa0d806d9c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa0f815ba50 0x7fa0f814c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0f80b4c00 0x7fa0f8145c50 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 --2- 172.21.15.122:0/2948814685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0f80b28a0 0x7fa0f81499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 >> 172.21.15.122:0/2948814685 conn(0x7fa0f809a9f0 msgr2=0x7fa0f806f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.899+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 shutdown_connections 2024-03-06T07:27:24.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:24.900+0000 7fa0fd46b640 1 -- 172.21.15.122:0/2948814685 wait complete. 2024-03-06T07:27:24.912 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:27:24.912 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:24.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.068+0000 7f610db1e640 1 Processor -- start 2024-03-06T07:27:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.068+0000 7f610db1e640 1 -- start start 2024-03-06T07:27:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f610db1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f61080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f610db1e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f61080d29c0 0x7f61080b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f610db1e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61080b3430 con 0x7f61080cf830 2024-03-06T07:27:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f610db1e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61080b31e0 con 0x7f61080d29c0 2024-03-06T07:27:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f610db1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61080b3d60 con 0x7f610815dfe0 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f61077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f61080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f6107fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f61080d29c0 0x7f61080b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f6107fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f61080d29c0 0x7f61080b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54690/0 (socket says 172.21.15.122:54690) 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f61077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f61080cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47826/0 (socket says 172.21.15.122:47826) 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.069+0000 7f6107fff640 1 -- 172.21.15.122:0/314865322 learned_addr learned my addr 172.21.15.122:0/314865322 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.070+0000 7f6107fff640 1 -- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 msgr2=0x7f61080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.070+0000 7f6107fff640 1 --2- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f61080cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.070+0000 7f6107fff640 1 -- 172.21.15.122:0/314865322 >> v1:172.21.15.169:6789/0 conn(0x7f61080cf830 legacy=0x7f61080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.070+0000 7f6107fff640 1 -- 172.21.15.122:0/314865322 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61080b3710 con 0x7f61080d29c0 2024-03-06T07:27:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.070+0000 7f61077fe640 1 --2- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f61080cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.070+0000 7f6107fff640 1 --2- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b41f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6100009e50 tx=0x7f610002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36afbc153c0c1a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f6106ffd640 1 -- 172.21.15.122:0/314865322 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610002cb00 con 0x7f61080d29c0 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f6106ffd640 1 -- 172.21.15.122:0/314865322 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610002cca0 con 0x7f61080d29c0 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f6106ffd640 1 -- 172.21.15.122:0/314865322 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100034610 con 0x7f61080d29c0 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 -- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 msgr2=0x7f61080b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 --2- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b41f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6100009e50 tx=0x7f610002bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 -- 172.21.15.122:0/314865322 shutdown_connections 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 --2- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b41f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 --2- 172.21.15.122:0/314865322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f61080cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 -- 172.21.15.122:0/314865322 >> 172.21.15.122:0/314865322 conn(0x7f61080d3cd0 msgr2=0x7f61080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 -- 172.21.15.122:0/314865322 shutdown_connections 2024-03-06T07:27:25.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.071+0000 7f610db1e640 1 -- 172.21.15.122:0/314865322 wait complete. 2024-03-06T07:27:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.072+0000 7f610db1e640 1 Processor -- start 2024-03-06T07:27:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 -- start start 2024-03-06T07:27:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 0x7f61081572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f610815f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080d1df0 con 0x7f61080cf830 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080d1c70 con 0x7f61080d29c0 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f610db1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610815e3c0 con 0x7f610815dfe0 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f61077fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 0x7f61081572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f61077fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 0x7f61081572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37458/0 (socket says 172.21.15.122:37458) 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.073+0000 7f61077fe640 1 -- 172.21.15.122:0/643872033 learned_addr learned my addr 172.21.15.122:0/643872033 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f6107fff640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f610815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f60feffd640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f61077fe640 1 -- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 msgr2=0x7f610815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f61077fe640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f610815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f61077fe640 1 -- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 msgr2=0x7f61080b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f61077fe640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f61077fe640 1 -- 172.21.15.122:0/643872033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610815e790 con 0x7f61080cf830 2024-03-06T07:27:25.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f61077fe640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 0x7f61081572a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f60f800e960 tx=0x7f60f800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8017070 con 0x7f61080cf830 2024-03-06T07:27:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f800ca00 con 0x7f61080cf830 2024-03-06T07:27:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f800cdd0 con 0x7f61080cf830 2024-03-06T07:27:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610815ea70 con 0x7f61080cf830 2024-03-06T07:27:25.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.074+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610815f030 con 0x7f61080cf830 2024-03-06T07:27:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.075+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610804f330 con 0x7f61080cf830 2024-03-06T07:27:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.076+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60f801a070 con 0x7f61080cf830 2024-03-06T07:27:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.076+0000 7f6104ff9640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e806b4d0 0x7f60e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.076+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60f808a330 con 0x7f61080cf830 2024-03-06T07:27:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.076+0000 7f60feffd640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e806b4d0 0x7f60e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.077+0000 7f60feffd640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e806b4d0 0x7f60e806d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f61081f6ff0 tx=0x7f60f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.081+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60f8058bf0 con 0x7f61080cf830 2024-03-06T07:27:25.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.259+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61080747a0 con 0x7f61080cf830 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.260+0000 7f6104ff9640 1 -- 172.21.15.122:0/643872033 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f60f805cc90 con 0x7f61080cf830 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:25.262 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":108,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:12.269669+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e806b4d0 msgr2=0x7f60e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e806b4d0 0x7f60e806d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f61081f6ff0 tx=0x7f60f0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 msgr2=0x7f61081572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 0x7f61081572a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f60f800e960 tx=0x7f60f800ee20 comp rx=0 tx=0).stop 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 shutdown_connections 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e806b4d0 0x7f60e806d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f610815dfe0 0x7f610815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f61080d29c0 0x7f61080b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 --2- 172.21.15.122:0/643872033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61080cf830 0x7f61081572a0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 >> 172.21.15.122:0/643872033 conn(0x7f61080d3cd0 msgr2=0x7f61080b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 shutdown_connections 2024-03-06T07:27:25.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.262+0000 7f610db1e640 1 -- 172.21.15.122:0/643872033 wait complete. 2024-03-06T07:27:25.275 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.843386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.579477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:25.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:25.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.377+0000 7fe550513640 1 Processor -- start 2024-03-06T07:27:25.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.377+0000 7fe550513640 1 -- start start 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe550513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe550513640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5481618b0 con 0x7fe5480d1910 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe550513640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480cf400 con 0x7fe5480d2480 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe550513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480cf6a0 con 0x7fe5481614d0 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe54daae640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe5480d1910 0x7fe5480d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42938/0 (socket says 172.21.15.122:42938) 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe54daae640 1 -- 172.21.15.122:0/175467987 learned_addr learned my addr 172.21.15.122:0/175467987 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.378+0000 7fe54e2af640 1 --2- 172.21.15.122:0/175467987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.379+0000 7fe54d2ad640 1 -- 172.21.15.122:0/175467987 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91557296 0 0) 0x7fe5481618b0 con 0x7fe5480d1910 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.379+0000 7fe54d2ad640 1 -- 172.21.15.122:0/175467987 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534003520 con 0x7fe5480d1910 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.379+0000 7fe54d2ad640 1 -- 172.21.15.122:0/175467987 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540080533 0 0) 0x7fe5480cf400 con 0x7fe5480d2480 2024-03-06T07:27:25.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.379+0000 7fe54e2af640 1 -- 172.21.15.122:0/175467987 >> v1:172.21.15.169:6789/0 conn(0x7fe5480d2480 legacy=0x7fe5480b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.379+0000 7fe54e2af640 1 -- 172.21.15.122:0/175467987 >> v1:172.21.15.122:6789/0 conn(0x7fe5480d1910 legacy=0x7fe5480d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.379+0000 7fe54e2af640 1 -- 172.21.15.122:0/175467987 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480cf980 con 0x7fe5481614d0 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.380+0000 7fe54e2af640 1 --2- 172.21.15.122:0/175467987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480d4860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe5440097d0 tx=0x7fe54402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeeeb804392dfdc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.380+0000 7fe54d2ad640 1 -- 172.21.15.122:0/175467987 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544035020 con 0x7fe5481614d0 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.380+0000 7fe54d2ad640 1 -- 172.21.15.122:0/175467987 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544031890 con 0x7fe5481614d0 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.380+0000 7fe54d2ad640 1 -- 172.21.15.122:0/175467987 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544031c40 con 0x7fe5481614d0 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 -- 172.21.15.122:0/175467987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 msgr2=0x7fe5480d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 --2- 172.21.15.122:0/175467987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480d4860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe5440097d0 tx=0x7fe54402b930 comp rx=0 tx=0).stop 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 -- 172.21.15.122:0/175467987 shutdown_connections 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 --2- 172.21.15.122:0/175467987 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480d4860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 -- 172.21.15.122:0/175467987 >> 172.21.15.122:0/175467987 conn(0x7fe5480d3d40 msgr2=0x7fe5480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 -- 172.21.15.122:0/175467987 shutdown_connections 2024-03-06T07:27:25.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.381+0000 7fe550513640 1 -- 172.21.15.122:0/175467987 wait complete. 2024-03-06T07:27:25.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.383+0000 7fe550513640 1 Processor -- start 2024-03-06T07:27:25.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.383+0000 7fe550513640 1 -- start start 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe550513640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5480d1910 0x7fe5480cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe550513640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 0x7fe5480c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe550513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe550513640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480b1540 con 0x7fe5480d2480 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe550513640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480b0910 con 0x7fe5480d1910 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe550513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480b16c0 con 0x7fe5481614d0 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 0x7fe5480c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 0x7fe5480c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37462/0 (socket says 172.21.15.122:37462) 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 -- 172.21.15.122:0/731944594 learned_addr learned my addr 172.21.15.122:0/731944594 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54e2af640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5480d1910 0x7fe5480cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54eab0640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 -- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 msgr2=0x7fe5480c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 -- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5480d1910 msgr2=0x7fe5480cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5480d1910 0x7fe5480cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54daae640 1 -- 172.21.15.122:0/731944594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480cbb90 con 0x7fe5480d2480 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.384+0000 7fe54e2af640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5480d1910 0x7fe5480cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.385+0000 7fe54daae640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 0x7fe5480c94f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe53800c430 tx=0x7fe53800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.385+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538017070 con 0x7fe5480d2480 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.385+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538005ce0 con 0x7fe5480d2480 2024-03-06T07:27:25.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.385+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5480cbe70 con 0x7fe5480d2480 2024-03-06T07:27:25.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.385+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380072f0 con 0x7fe5480d2480 2024-03-06T07:27:25.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.385+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5480cc430 con 0x7fe5480d2480 2024-03-06T07:27:25.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.386+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe548068ff0 con 0x7fe5480d2480 2024-03-06T07:27:25.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.386+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe53801a070 con 0x7fe5480d2480 2024-03-06T07:27:25.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.387+0000 7fe53effd640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe52c06b4d0 0x7fe52c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.387+0000 7fe54e2af640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe52c06b4d0 0x7fe52c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.387+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe538089620 con 0x7fe5480d2480 2024-03-06T07:27:25.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.387+0000 7fe54e2af640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe52c06b4d0 0x7fe52c06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe544036000 tx=0x7fe544008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.391+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe538058530 con 0x7fe5480d2480 2024-03-06T07:27:25.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.573+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5480670d0 con 0x7fe5480d2480 2024-03-06T07:27:25.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.574+0000 7fe53effd640 1 -- 172.21.15.122:0/731944594 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe538057e30 con 0x7fe5480d2480 2024-03-06T07:27:25.576 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:27:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe52c06b4d0 msgr2=0x7fe52c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe52c06b4d0 0x7fe52c06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe544036000 tx=0x7fe544008030 comp rx=0 tx=0).stop 2024-03-06T07:27:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 msgr2=0x7fe5480c94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 0x7fe5480c94f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe53800c430 tx=0x7fe53800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 shutdown_connections 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe52c06b4d0 0x7fe52c06d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe5481614d0 0x7fe5480c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480d2480 0x7fe5480c94f0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 --2- 172.21.15.122:0/731944594 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5480d1910 0x7fe5480cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 >> 172.21.15.122:0/731944594 conn(0x7fe5480d3d40 msgr2=0x7fe5480b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 shutdown_connections 2024-03-06T07:27:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.576+0000 7fe550513640 1 -- 172.21.15.122:0/731944594 wait complete. 2024-03-06T07:27:25.592 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:25.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.694+0000 7f2764f79640 1 Processor -- start 2024-03-06T07:27:25.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.694+0000 7f2764f79640 1 -- start start 2024-03-06T07:27:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f2764f79640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 0x7f27600cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f2764f79640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 0x7f27600ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f2764f79640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27600cd1e0 con 0x7f276015c050 2024-03-06T07:27:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f2764f79640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27600cd4b0 con 0x7f276006fd10 2024-03-06T07:27:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f2764f79640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27600cd790 con 0x7f27600d0af0 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f275ed9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 0x7f27600ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f275dd9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 0x7f27600cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f275ed9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 0x7f27600ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47874/0 (socket says 172.21.15.122:47874) 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f275ed9d640 1 -- 172.21.15.122:0/2203456436 learned_addr learned my addr 172.21.15.122:0/2203456436 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.695+0000 7f275dd9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 0x7f27600cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54712/0 (socket says 172.21.15.122:54712) 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.696+0000 7f275ed9d640 1 -- 172.21.15.122:0/2203456436 >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 msgr2=0x7f27600cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.696+0000 7f275ed9d640 1 --2- 172.21.15.122:0/2203456436 >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 0x7f27600cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.696+0000 7f275ed9d640 1 -- 172.21.15.122:0/2203456436 >> v1:172.21.15.169:6789/0 conn(0x7f276015c050 legacy=0x7f276006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.696+0000 7f275ed9d640 1 -- 172.21.15.122:0/2203456436 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27600cda70 con 0x7f27600d0af0 2024-03-06T07:27:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.696+0000 7f275dd9b640 1 --2- 172.21.15.122:0/2203456436 >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 0x7f27600cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.696+0000 7f275ed9d640 1 --2- 172.21.15.122:0/2203456436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 0x7f27600ccca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2758009e50 tx=0x7f275802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1298cc4490b26e07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f275d59a640 1 -- 172.21.15.122:0/2203456436 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275802ca70 con 0x7f27600d0af0 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f275d59a640 1 -- 172.21.15.122:0/2203456436 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275802cc10 con 0x7f27600d0af0 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f275d59a640 1 -- 172.21.15.122:0/2203456436 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27580343f0 con 0x7f27600d0af0 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 -- 172.21.15.122:0/2203456436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 msgr2=0x7f27600ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 --2- 172.21.15.122:0/2203456436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 0x7f27600ccca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2758009e50 tx=0x7f275802bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 -- 172.21.15.122:0/2203456436 shutdown_connections 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 --2- 172.21.15.122:0/2203456436 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f27600d0af0 0x7f27600ccca0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 --2- 172.21.15.122:0/2203456436 >> v2:172.21.15.169:3300/0 conn(0x7f276006fd10 0x7f27600cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 -- 172.21.15.122:0/2203456436 >> 172.21.15.122:0/2203456436 conn(0x7f27600b3800 msgr2=0x7f27600b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 -- 172.21.15.122:0/2203456436 shutdown_connections 2024-03-06T07:27:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.697+0000 7f2764f79640 1 -- 172.21.15.122:0/2203456436 wait complete. 2024-03-06T07:27:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.699+0000 7f2764f79640 1 Processor -- start 2024-03-06T07:27:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.699+0000 7f2764f79640 1 -- start start 2024-03-06T07:27:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.700+0000 7f2764f79640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f276006fd10 0x7f27601589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.700+0000 7f2764f79640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 0x7f276009baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.700+0000 7f2764f79640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f276015c050 0x7f27601576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.700+0000 7f2764f79640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760165c30 con 0x7f27600d0af0 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.700+0000 7f2764f79640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760165ab0 con 0x7f276006fd10 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.700+0000 7f2764f79640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2760165db0 con 0x7f276015c050 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 0x7f276009baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 0x7f276009baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37476/0 (socket says 172.21.15.122:37476) 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 -- 172.21.15.122:0/4228783633 learned_addr learned my addr 172.21.15.122:0/4228783633 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275ed9d640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f276015c050 0x7f27601576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275e59c640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f276006fd10 0x7f27601589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 -- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f276015c050 msgr2=0x7f27601576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f276015c050 0x7f27601576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 -- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f276006fd10 msgr2=0x7f27601589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f276006fd10 0x7f27601589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 -- 172.21.15.122:0/4228783633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27600b75f0 con 0x7f27600d0af0 2024-03-06T07:27:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f275dd9b640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 0x7f276009baf0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f275400e960 tx=0x7f275400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2754017070 con 0x7f27600d0af0 2024-03-06T07:27:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275400ca00 con 0x7f27600d0af0 2024-03-06T07:27:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27600b78d0 con 0x7f27600d0af0 2024-03-06T07:27:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400cdd0 con 0x7f27600d0af0 2024-03-06T07:27:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.701+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27601581f0 con 0x7f27600d0af0 2024-03-06T07:27:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.702+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276009fe60 con 0x7f27600d0af0 2024-03-06T07:27:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.703+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f275401a070 con 0x7f27600d0af0 2024-03-06T07:27:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.703+0000 7f274effd640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f274006b4d0 0x7f274006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.703+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f275408a3c0 con 0x7f27600d0af0 2024-03-06T07:27:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.703+0000 7f275e59c640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f274006b4d0 0x7f274006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.704+0000 7f275e59c640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f274006b4d0 0x7f274006d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2750001780 tx=0x7f2750009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.708+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2754058be0 con 0x7f27600d0af0 2024-03-06T07:27:25.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.883+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2760068ff0 con 0x7f27600d0af0 2024-03-06T07:27:25.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.884+0000 7f274effd640 1 -- 172.21.15.122:0/4228783633 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7f275405cc80 con 0x7f27600d0af0 2024-03-06T07:27:25.886 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.885+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f274006b4d0 msgr2=0x7f274006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.885+0000 7f2764f79640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f274006b4d0 0x7f274006d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2750001780 tx=0x7f2750009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.885+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 msgr2=0x7f276009baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.885+0000 7f2764f79640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 0x7f276009baf0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f275400e960 tx=0x7f275400ee20 comp rx=0 tx=0).stop 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.885+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 shutdown_connections 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f274006b4d0 0x7f274006d990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f276015c050 0x7f27601576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27600d0af0 0x7f276009baf0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 --2- 172.21.15.122:0/4228783633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f276006fd10 0x7f27601589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 >> 172.21.15.122:0/4228783633 conn(0x7f27600b3800 msgr2=0x7f27600aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 shutdown_connections 2024-03-06T07:27:25.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:25.886+0000 7f2764f79640 1 -- 172.21.15.122:0/4228783633 wait complete. 2024-03-06T07:27:25.898 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:26.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.001+0000 7f53f1615640 1 Processor -- start 2024-03-06T07:27:26.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.001+0000 7f53f1615640 1 -- start start 2024-03-06T07:27:26.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53f1615640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53ec0b1e40 0x7f53ec168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53f1615640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ec0d2730 con 0x7f53ec1614d0 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53f1615640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ec1618b0 con 0x7f53ec0d4860 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53f1615640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ec0d4cf0 con 0x7f53ec0b1e40 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53eb7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53ec0b1e40 0x7f53ec168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53eaffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f53ec1614d0 0x7f53ec0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42950/0 (socket says 172.21.15.122:42950) 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53eb7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f53ec0b1e40 0x7f53ec168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37480/0 (socket says 172.21.15.122:37480) 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.002+0000 7f53eaffd640 1 -- 172.21.15.122:0/1503282193 learned_addr learned my addr 172.21.15.122:0/1503282193 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53eb7fe640 1 -- 172.21.15.122:0/1503282193 >> v1:172.21.15.169:6789/0 conn(0x7f53ec0d4860 legacy=0x7f53ec0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53eb7fe640 1 -- 172.21.15.122:0/1503282193 >> v1:172.21.15.122:6789/0 conn(0x7f53ec1614d0 legacy=0x7f53ec0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53eb7fe640 1 -- 172.21.15.122:0/1503282193 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0cf400 con 0x7f53ec0b1e40 2024-03-06T07:27:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53ea7fc640 1 -- 172.21.15.122:0/1503282193 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627892822 0 0) 0x7f53ec0d2730 con 0x7f53ec1614d0 2024-03-06T07:27:26.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53eb7fe640 1 --2- 172.21.15.122:0/1503282193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0b1e40 0x7f53ec168b60 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0cf5d0 tx=0x7f53e402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f201b964d8d072 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53ea7fc640 1 -- 172.21.15.122:0/1503282193 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e4039070 con 0x7f53ec0b1e40 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53ea7fc640 1 -- 172.21.15.122:0/1503282193 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e402cbc0 con 0x7f53ec0b1e40 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.003+0000 7f53ea7fc640 1 -- 172.21.15.122:0/1503282193 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e4034440 con 0x7f53ec0b1e40 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 -- 172.21.15.122:0/1503282193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0b1e40 msgr2=0x7f53ec168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 --2- 172.21.15.122:0/1503282193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0b1e40 0x7f53ec168b60 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0cf5d0 tx=0x7f53e402be10 comp rx=0 tx=0).stop 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 -- 172.21.15.122:0/1503282193 shutdown_connections 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 --2- 172.21.15.122:0/1503282193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0b1e40 0x7f53ec168b60 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 -- 172.21.15.122:0/1503282193 >> 172.21.15.122:0/1503282193 conn(0x7f53ec0d3d40 msgr2=0x7f53ec0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 -- 172.21.15.122:0/1503282193 shutdown_connections 2024-03-06T07:27:26.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.004+0000 7f53f1615640 1 -- 172.21.15.122:0/1503282193 wait complete. 2024-03-06T07:27:26.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.005+0000 7f53f1615640 1 Processor -- start 2024-03-06T07:27:26.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.005+0000 7f53f1615640 1 -- start start 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53f1615640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53ec0b1e40 0x7f53ec14acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53f1615640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 0x7f53ec0cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53f1615640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53ec1614d0 0x7f53ec0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53f1615640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec0cfc60 con 0x7f53ec0d4860 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53f1615640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec0cf980 con 0x7f53ec1614d0 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53f1615640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec0b0910 con 0x7f53ec0b1e40 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 0x7f53ec0cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 0x7f53ec0cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37492/0 (socket says 172.21.15.122:37492) 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 -- 172.21.15.122:0/2080808449 learned_addr learned my addr 172.21.15.122:0/2080808449 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53eb7fe640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53ec1614d0 0x7f53ec0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53eaffd640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53ec0b1e40 0x7f53ec14acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 -- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53ec0b1e40 msgr2=0x7f53ec14acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53ec0b1e40 0x7f53ec14acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 -- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53ec1614d0 msgr2=0x7f53ec0cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53ec1614d0 0x7f53ec0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 -- 172.21.15.122:0/2080808449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0cae10 con 0x7f53ec0d4860 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.006+0000 7f53e3fff640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 0x7f53ec0cd5b0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f53d400c430 tx=0x7f53d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.007+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4011790 con 0x7f53ec0d4860 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.007+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d4017070 con 0x7f53ec0d4860 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.007+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ec0cb0f0 con 0x7f53ec0d4860 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.007+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4011bb0 con 0x7f53ec0d4860 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.007+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ec0ccdb0 con 0x7f53ec0d4860 2024-03-06T07:27:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.008+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ec068ff0 con 0x7f53ec0d4860 2024-03-06T07:27:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.008+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53d401c460 con 0x7f53ec0d4860 2024-03-06T07:27:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.008+0000 7f53e37fe640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53cc06b4d0 0x7f53cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.009+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53d408a130 con 0x7f53ec0d4860 2024-03-06T07:27:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.009+0000 7f53eaffd640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53cc06b4d0 0x7f53cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.009+0000 7f53eaffd640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53cc06b4d0 0x7f53cc06d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f53dc007fd0 tx=0x7f53dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.013+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53d4058fb0 con 0x7f53ec0d4860 2024-03-06T07:27:26.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.267+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53ec0b52c0 con 0x7f53ec0d4860 2024-03-06T07:27:26.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.267+0000 7f53e37fe640 1 -- 172.21.15.122:0/2080808449 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f53d40588b0 con 0x7f53ec0d4860 2024-03-06T07:27:26.269 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53cc06b4d0 msgr2=0x7f53cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53cc06b4d0 0x7f53cc06d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f53dc007fd0 tx=0x7f53dc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 msgr2=0x7f53ec0cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 0x7f53ec0cd5b0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f53d400c430 tx=0x7f53d400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 shutdown_connections 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f53cc06b4d0 0x7f53cc06d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f53ec1614d0 0x7f53ec0cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53ec0d4860 0x7f53ec0cd5b0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 --2- 172.21.15.122:0/2080808449 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f53ec0b1e40 0x7f53ec14acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 >> 172.21.15.122:0/2080808449 conn(0x7f53ec0d3d40 msgr2=0x7f53ec06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 shutdown_connections 2024-03-06T07:27:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.269+0000 7f53f1615640 1 -- 172.21.15.122:0/2080808449 wait complete. 2024-03-06T07:27:26.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.387+0000 7fe08898a640 1 Processor -- start 2024-03-06T07:27:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.387+0000 7fe08898a640 1 -- start start 2024-03-06T07:27:26.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe08898a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084162970 0x7fe0840d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe08898a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe084161cf0 0x7fe084162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe08898a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0840d22c0 con 0x7fe0840b4460 2024-03-06T07:27:26.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe08898a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe084161b00 con 0x7fe084161cf0 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe08898a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840cecc0 con 0x7fe084162970 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe084161cf0 0x7fe084162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe084161cf0 0x7fe084162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37498/0 (socket says 172.21.15.122:37498) 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 -- 172.21.15.122:0/972387296 learned_addr learned my addr 172.21.15.122:0/972387296 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe081d9b640 1 --2- 172.21.15.122:0/972387296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084162970 0x7fe0840d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 -- 172.21.15.122:0/972387296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084162970 msgr2=0x7fe0840d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 --2- 172.21.15.122:0/972387296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084162970 0x7fe0840d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 -- 172.21.15.122:0/972387296 >> v1:172.21.15.169:6789/0 conn(0x7fe0840b4460 legacy=0x7fe0840b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.388+0000 7fe082d9d640 1 -- 172.21.15.122:0/972387296 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840d25a0 con 0x7fe084161cf0 2024-03-06T07:27:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.389+0000 7fe082d9d640 1 --2- 172.21.15.122:0/972387296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe084161cf0 0x7fe084162180 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fe074009e50 tx=0x7fe07402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568b45a7c8e879ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.389+0000 7fe08159a640 1 -- 172.21.15.122:0/972387296 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07402cb00 con 0x7fe084161cf0 2024-03-06T07:27:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.389+0000 7fe08159a640 1 -- 172.21.15.122:0/972387296 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07402cca0 con 0x7fe084161cf0 2024-03-06T07:27:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.389+0000 7fe08159a640 1 -- 172.21.15.122:0/972387296 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074034610 con 0x7fe084161cf0 2024-03-06T07:27:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 -- 172.21.15.122:0/972387296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe084161cf0 msgr2=0x7fe084162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 --2- 172.21.15.122:0/972387296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe084161cf0 0x7fe084162180 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fe074009e50 tx=0x7fe07402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 -- 172.21.15.122:0/972387296 shutdown_connections 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 --2- 172.21.15.122:0/972387296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe084161cf0 0x7fe084162180 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 --2- 172.21.15.122:0/972387296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084162970 0x7fe0840d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 -- 172.21.15.122:0/972387296 >> 172.21.15.122:0/972387296 conn(0x7fe0840d3ce0 msgr2=0x7fe0840d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 -- 172.21.15.122:0/972387296 shutdown_connections 2024-03-06T07:27:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.390+0000 7fe08898a640 1 -- 172.21.15.122:0/972387296 wait complete. 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.391+0000 7fe08898a640 1 Processor -- start 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.391+0000 7fe08898a640 1 -- start start 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08898a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 0x7fe084156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08898a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084161cf0 0x7fe084157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08898a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe084162970 0x7fe0840b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08898a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840ce950 con 0x7fe0840b4460 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08898a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840d2b60 con 0x7fe084162970 2024-03-06T07:27:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08898a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840b4840 con 0x7fe084161cf0 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 0x7fe084156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 0x7fe084156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37512/0 (socket says 172.21.15.122:37512) 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 -- 172.21.15.122:0/1151875713 learned_addr learned my addr 172.21.15.122:0/1151875713 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe082d9d640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe084162970 0x7fe0840b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe081d9b640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084161cf0 0x7fe084157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 -- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084161cf0 msgr2=0x7fe084157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084161cf0 0x7fe084157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 -- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe084162970 msgr2=0x7fe0840b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe084162970 0x7fe0840b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 -- 172.21.15.122:0/1151875713 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0841588e0 con 0x7fe0840b4460 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.392+0000 7fe08259c640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 0x7fe084156230 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c3f0 tx=0x7fe07c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.393+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0052d0 con 0x7fe0840b4460 2024-03-06T07:27:26.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.393+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c005490 con 0x7fe0840b4460 2024-03-06T07:27:26.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.393+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c007260 con 0x7fe0840b4460 2024-03-06T07:27:26.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.393+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe084158bc0 con 0x7fe0840b4460 2024-03-06T07:27:26.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.393+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe084159180 con 0x7fe0840b4460 2024-03-06T07:27:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.394+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08404f330 con 0x7fe0840b4460 2024-03-06T07:27:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.394+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe07c007400 con 0x7fe0840b4460 2024-03-06T07:27:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.395+0000 7fe07affd640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe06406b4d0 0x7fe06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.395+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe07c089770 con 0x7fe0840b4460 2024-03-06T07:27:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.395+0000 7fe081d9b640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe06406b4d0 0x7fe06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.395+0000 7fe081d9b640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe06406b4d0 0x7fe06406d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe084157110 tx=0x7fe06c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.399+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe07c058560 con 0x7fe0840b4460 2024-03-06T07:27:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.623+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe08406b180 con 0x7fe0840b4460 2024-03-06T07:27:26.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.624+0000 7fe07affd640 1 -- 172.21.15.122:0/1151875713 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe07c057e60 con 0x7fe0840b4460 2024-03-06T07:27:26.626 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe06406b4d0 msgr2=0x7fe06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe06406b4d0 0x7fe06406d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe084157110 tx=0x7fe06c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 msgr2=0x7fe084156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 0x7fe084156230 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c3f0 tx=0x7fe07c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 shutdown_connections 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe06406b4d0 0x7fe06406d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe084162970 0x7fe0840b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe084161cf0 0x7fe084157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 --2- 172.21.15.122:0/1151875713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0840b4460 0x7fe084156230 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 >> 172.21.15.122:0/1151875713 conn(0x7fe0840d3ce0 msgr2=0x7fe0840b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 shutdown_connections 2024-03-06T07:27:26.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.626+0000 7fe08898a640 1 -- 172.21.15.122:0/1151875713 wait complete. 2024-03-06T07:27:26.638 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:27:26.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:26.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:26.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[73035]: fuse finished with error 0 and tester_r 0 2024-03-06T07:27:26.665 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:26.705 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:26.705 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:27:26.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:26.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.899+0000 7f5676c78640 1 Processor -- start 2024-03-06T07:27:26.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.899+0000 7f5676c78640 1 -- start start 2024-03-06T07:27:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f5676c78640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f56700d22b0 0x7f56700d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f5676c78640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f5676c78640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56700b3430 con 0x7f567015dfe0 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f5676c78640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56700d2c50 con 0x7f56700d22b0 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f5676c78640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56700b31e0 con 0x7f56700b41f0 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f56700d22b0 0x7f56700d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f56700d22b0 0x7f56700d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37526/0 (socket says 172.21.15.122:37526) 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 -- 172.21.15.122:0/409815130 learned_addr learned my addr 172.21.15.122:0/409815130 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 -- 172.21.15.122:0/409815130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 msgr2=0x7f56700b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 --2- 172.21.15.122:0/409815130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 -- 172.21.15.122:0/409815130 >> v1:172.21.15.169:6789/0 conn(0x7f567015dfe0 legacy=0x7f56700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.900+0000 7f566ffff640 1 -- 172.21.15.122:0/409815130 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56700b3710 con 0x7f56700d22b0 2024-03-06T07:27:26.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.901+0000 7f566ffff640 1 --2- 172.21.15.122:0/409815130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56700d22b0 0x7f56700d2710 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f565c009870 tx=0x7f565c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b5b45903d15a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.901+0000 7f566f7fe640 1 -- 172.21.15.122:0/409815130 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c035020 con 0x7f56700d22b0 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.901+0000 7f566f7fe640 1 -- 172.21.15.122:0/409815130 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c031640 con 0x7f56700d22b0 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.901+0000 7f566f7fe640 1 -- 172.21.15.122:0/409815130 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c02d040 con 0x7f56700d22b0 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 -- 172.21.15.122:0/409815130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56700d22b0 msgr2=0x7f56700d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 --2- 172.21.15.122:0/409815130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56700d22b0 0x7f56700d2710 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f565c009870 tx=0x7f565c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 -- 172.21.15.122:0/409815130 shutdown_connections 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 --2- 172.21.15.122:0/409815130 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 --2- 172.21.15.122:0/409815130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56700d22b0 0x7f56700d2710 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 -- 172.21.15.122:0/409815130 >> 172.21.15.122:0/409815130 conn(0x7f56700d3cd0 msgr2=0x7f56700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 -- 172.21.15.122:0/409815130 shutdown_connections 2024-03-06T07:27:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.902+0000 7f5676c78640 1 -- 172.21.15.122:0/409815130 wait complete. 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.903+0000 7f5676c78640 1 Processor -- start 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.903+0000 7f5676c78640 1 -- start start 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5676c78640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5676c78640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56700d22b0 0x7f56700cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5676c78640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 0x7f567014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5676c78640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56700cfa00 con 0x7f567015dfe0 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5676c78640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56700b3d60 con 0x7f56700d22b0 2024-03-06T07:27:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5676c78640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56700d1c70 con 0x7f56700b41f0 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5675215640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 0x7f567014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5675215640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 0x7f567014eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37534/0 (socket says 172.21.15.122:37534) 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5675215640 1 -- 172.21.15.122:0/1740358330 learned_addr learned my addr 172.21.15.122:0/1740358330 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f566ffff640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56700d22b0 0x7f56700cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.904+0000 7f5674a14640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5675215640 1 -- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 msgr2=0x7f56700cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5675215640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5675215640 1 -- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56700d22b0 msgr2=0x7f56700cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5675215640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56700d22b0 0x7f56700cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5675215640 1 -- 172.21.15.122:0/1740358330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56700cc910 con 0x7f567015dfe0 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5674a14640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5675215640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 0x7f567014eab0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f566800b970 tx=0x7f566800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668019070 con 0x7f567015dfe0 2024-03-06T07:27:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566800cb60 con 0x7f567015dfe0 2024-03-06T07:27:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56700ccbf0 con 0x7f567015dfe0 2024-03-06T07:27:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668014440 con 0x7f567015dfe0 2024-03-06T07:27:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.905+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56700cd1b0 con 0x7f567015dfe0 2024-03-06T07:27:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.906+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56700cd6a0 con 0x7f567015dfe0 2024-03-06T07:27:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.907+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f566801c070 con 0x7f567015dfe0 2024-03-06T07:27:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.907+0000 7f566d7fa640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f565406b4d0 0x7f565406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.907+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5668089df0 con 0x7f567015dfe0 2024-03-06T07:27:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.907+0000 7f5674a14640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f565406b4d0 0x7f565406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.908+0000 7f5674a14640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f565406b4d0 0x7f565406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5664006fd0 tx=0x7f5664008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:26.912+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56680586b0 con 0x7f567015dfe0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.195+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567014eff0 con 0x7f567015dfe0 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.196+0000 7f566d7fa640 1 -- 172.21.15.122:0/1740358330 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f566805c750 con 0x7f567015dfe0 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3571233755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3571233755},{"type":"v1","addr":"172.21.15.169:6834","nonce":3571233755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":7207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1631232650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1631232650},{"type":"v1","addr":"172.21.15.169:6835","nonce":1631232650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/2235394450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":2235394450},{"type":"v1","addr":"172.21.15.122:6835","nonce":2235394450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:12.635136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7962},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7962":{"gid":7962,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/386534812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":386534812},{"type":"v1","addr":"172.21.15.122:6837","nonce":386534812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7962,"qdb_cluster":[7962]},"id":8}]} 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 61 2024-03-06T07:27:27.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f565406b4d0 msgr2=0x7f565406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f565406b4d0 0x7f565406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5664006fd0 tx=0x7f5664008040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 msgr2=0x7f567014eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 0x7f567014eab0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f566800b970 tx=0x7f566800be30 comp rx=0 tx=0).stop 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 shutdown_connections 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f565406b4d0 0x7f565406d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f567015dfe0 0x7f567014eab0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56700d22b0 0x7f56700cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 --2- 172.21.15.122:0/1740358330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56700b41f0 0x7f56700cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 >> 172.21.15.122:0/1740358330 conn(0x7f56700d3cd0 msgr2=0x7f567006caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 shutdown_connections 2024-03-06T07:27:27.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.198+0000 7f5676c78640 1 -- 172.21.15.122:0/1740358330 wait complete. 2024-03-06T07:27:27.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:27:27.335 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:27.336 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:27.336 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:27.337 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:27.337 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:27.346 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:27.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:27:27.347 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:27.347 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:27:27.348 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:27.348 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:27:27.349 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:27.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:27:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.491+0000 7f82c7aa4640 1 Processor -- start 2024-03-06T07:27:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.491+0000 7f47abd7a640 1 Processor -- start 2024-03-06T07:27:27.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.491+0000 7f47abd7a640 1 -- start start 2024-03-06T07:27:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.491+0000 7f9be9381640 1 Processor -- start 2024-03-06T07:27:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.491+0000 7f82c7aa4640 1 -- start start 2024-03-06T07:27:27.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f9be9381640 1 -- start start 2024-03-06T07:27:27.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47abd7a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f47a40b4460 0x7f47a40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47abd7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a4162970 0x7f47a40b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47abd7a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a40d22c0 con 0x7f47a4161cf0 2024-03-06T07:27:27.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47abd7a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a4161b00 con 0x7f47a40b4460 2024-03-06T07:27:27.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47abd7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a40cecc0 con 0x7f47a4162970 2024-03-06T07:27:27.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f9be9381640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9be40d02b0 0x7f9be406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c7aa4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c01444c0 0x7f82c006ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c7aa4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f82c015b2d0 0x7f82c00708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c7aa4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c006d370 con 0x7f82c0070e10 2024-03-06T07:27:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c7aa4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c01448a0 con 0x7f82c015b2d0 2024-03-06T07:27:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c7aa4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0071260 con 0x7f82c01444c0 2024-03-06T07:27:27.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47aa317640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f47a4161cf0 0x7f47a40cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42964/0 (socket says 172.21.15.122:42964) 2024-03-06T07:27:27.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f47aa317640 1 -- 172.21.15.122:0/1997531108 learned_addr learned my addr 172.21.15.122:0/1997531108 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:27.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c503f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f82c015b2d0 0x7f82c00708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c503f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f82c015b2d0 0x7f82c00708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37548/0 (socket says 172.21.15.122:37548) 2024-03-06T07:27:27.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f82c503f640 1 -- 172.21.15.122:0/3103643426 learned_addr learned my addr 172.21.15.122:0/3103643426 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:27.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.492+0000 7f9be9381640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9be40b50d0 0x7f9be40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be9381640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be40b5c30 con 0x7f9be40670a0 2024-03-06T07:27:27.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be9381640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be40cd940 con 0x7f9be40b50d0 2024-03-06T07:27:27.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f82c5840640 1 --2- 172.21.15.122:0/3103643426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c01444c0 0x7f82c006ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be9381640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be40b5f10 con 0x7f9be40d02b0 2024-03-06T07:27:27.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be27fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9be40670a0 0x7f9be406b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42976/0 (socket says 172.21.15.122:42976) 2024-03-06T07:27:27.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f82c503f640 1 -- 172.21.15.122:0/3103643426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c01444c0 msgr2=0x7f82c006ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f82c503f640 1 --2- 172.21.15.122:0/3103643426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c01444c0 0x7f82c006ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be27fc640 1 -- 172.21.15.122:0/3693641812 learned_addr learned my addr 172.21.15.122:0/3693641812 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:27.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f82c503f640 1 -- 172.21.15.122:0/3103643426 >> v1:172.21.15.169:6789/0 conn(0x7f82c0070e10 legacy=0x7f82c015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f82c503f640 1 -- 172.21.15.122:0/3103643426 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c006d650 con 0x7f82c015b2d0 2024-03-06T07:27:27.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be37fe640 1 --2- 172.21.15.122:0/3693641812 >> v2:172.21.15.122:3300/0 conn(0x7f9be40b50d0 0x7f9be40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be2ffd640 1 --2- 172.21.15.122:0/3693641812 >> v2:172.21.15.169:3300/0 conn(0x7f9be40d02b0 0x7f9be406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a9b16640 1 --2- 172.21.15.122:0/1997531108 >> v2:172.21.15.122:3300/0 conn(0x7f47a40b4460 0x7f47a40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be37fe640 1 -- 172.21.15.122:0/3693641812 >> v2:172.21.15.169:3300/0 conn(0x7f9be40d02b0 msgr2=0x7f9be406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be37fe640 1 --2- 172.21.15.122:0/3693641812 >> v2:172.21.15.169:3300/0 conn(0x7f9be40d02b0 0x7f9be406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a9315640 1 --2- 172.21.15.122:0/1997531108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a4162970 0x7f47a40b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be37fe640 1 -- 172.21.15.122:0/3693641812 >> v1:172.21.15.122:6789/0 conn(0x7f9be40670a0 legacy=0x7f9be406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f9be37fe640 1 -- 172.21.15.122:0/3693641812 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be40cd5d0 con 0x7f9be40b50d0 2024-03-06T07:27:27.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a9315640 1 -- 172.21.15.122:0/1997531108 >> v2:172.21.15.122:3300/0 conn(0x7f47a40b4460 msgr2=0x7f47a40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a9315640 1 --2- 172.21.15.122:0/1997531108 >> v2:172.21.15.122:3300/0 conn(0x7f47a40b4460 0x7f47a40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a9315640 1 -- 172.21.15.122:0/1997531108 >> v1:172.21.15.122:6789/0 conn(0x7f47a4161cf0 legacy=0x7f47a40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a8b14640 1 -- 172.21.15.122:0/1997531108 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726865448 0 0) 0x7f47a40d22c0 con 0x7f47a4161cf0 2024-03-06T07:27:27.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.493+0000 7f47a9315640 1 -- 172.21.15.122:0/1997531108 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a40d25a0 con 0x7f47a4162970 2024-03-06T07:27:27.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f82c503f640 1 --2- 172.21.15.122:0/3103643426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c00708d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009870 tx=0x7f82b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5e9e50aa60b74f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f9be37fe640 1 --2- 172.21.15.122:0/3693641812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40b50d0 0x7f9be40ce5a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9be40cdd20 tx=0x7f9bdc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8584c9298b54b6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f82c483e640 1 -- 172.21.15.122:0/3103643426 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b0034070 con 0x7f82c015b2d0 2024-03-06T07:27:27.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f82c483e640 1 -- 172.21.15.122:0/3103643426 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b0030840 con 0x7f82c015b2d0 2024-03-06T07:27:27.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f82c483e640 1 -- 172.21.15.122:0/3103643426 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b0030c10 con 0x7f82c015b2d0 2024-03-06T07:27:27.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f47a9315640 1 --2- 172.21.15.122:0/1997531108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a4162970 0x7f47a40b3580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47940097a0 tx=0x7f479402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cee446fc9e577b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f9be1ffb640 1 -- 172.21.15.122:0/3693641812 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc039070 con 0x7f9be40b50d0 2024-03-06T07:27:27.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f9be1ffb640 1 -- 172.21.15.122:0/3693641812 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc02cbe0 con 0x7f9be40b50d0 2024-03-06T07:27:27.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f9be1ffb640 1 -- 172.21.15.122:0/3693641812 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc034440 con 0x7f9be40b50d0 2024-03-06T07:27:27.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f47a8b14640 1 -- 172.21.15.122:0/1997531108 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794035020 con 0x7f47a4162970 2024-03-06T07:27:27.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f47a8b14640 1 -- 172.21.15.122:0/1997531108 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4794031810 con 0x7f47a4162970 2024-03-06T07:27:27.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.494+0000 7f47a8b14640 1 -- 172.21.15.122:0/1997531108 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794031be0 con 0x7f47a4162970 2024-03-06T07:27:27.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9755fa2640 1 Processor -- start 2024-03-06T07:27:27.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 -- 172.21.15.122:0/3103643426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 msgr2=0x7f82c00708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/3103643426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c00708d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009870 tx=0x7f82b002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:27.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 -- 172.21.15.122:0/3693641812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40b50d0 msgr2=0x7f9be40ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 --2- 172.21.15.122:0/3693641812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40b50d0 0x7f9be40ce5a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9be40cdd20 tx=0x7f9bdc02be30 comp rx=0 tx=0).stop 2024-03-06T07:27:27.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 -- 172.21.15.122:0/3103643426 shutdown_connections 2024-03-06T07:27:27.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/3103643426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c00708d0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/3103643426 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c01444c0 0x7f82c006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 -- 172.21.15.122:0/3103643426 >> 172.21.15.122:0/3103643426 conn(0x7f82c00b1c80 msgr2=0x7f82c00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 -- 172.21.15.122:0/3693641812 shutdown_connections 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 --2- 172.21.15.122:0/3693641812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40b50d0 0x7f9be40ce5a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 --2- 172.21.15.122:0/3693641812 >> v2:172.21.15.169:3300/0 conn(0x7f9be40d02b0 0x7f9be406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 -- 172.21.15.122:0/3693641812 >> 172.21.15.122:0/3693641812 conn(0x7f9be4072520 msgr2=0x7f9be4072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 -- 172.21.15.122:0/3693641812 shutdown_connections 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 -- 172.21.15.122:0/3103643426 shutdown_connections 2024-03-06T07:27:27.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9be9381640 1 -- 172.21.15.122:0/3693641812 wait complete. 2024-03-06T07:27:27.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f82c7aa4640 1 -- 172.21.15.122:0/3103643426 wait complete. 2024-03-06T07:27:27.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f9755fa2640 1 -- start start 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f47abd7a640 1 -- 172.21.15.122:0/1997531108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a4162970 msgr2=0x7f47a40b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1997531108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a4162970 0x7f47a40b3580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47940097a0 tx=0x7f479402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.495+0000 7f47abd7a640 1 -- 172.21.15.122:0/1997531108 shutdown_connections 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1997531108 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a4162970 0x7f47a40b3580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1997531108 >> v2:172.21.15.122:3300/0 conn(0x7f47a40b4460 0x7f47a40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f47abd7a640 1 -- 172.21.15.122:0/1997531108 >> 172.21.15.122:0/1997531108 conn(0x7f47a40d3ce0 msgr2=0x7f47a40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f47abd7a640 1 -- 172.21.15.122:0/1997531108 shutdown_connections 2024-03-06T07:27:27.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f47abd7a640 1 -- 172.21.15.122:0/1997531108 wait complete. 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9755fa2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9750161ed0 0x7f97501622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9755fa2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f97501627f0 0x7f975006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9755fa2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97500afae0 0x7f975006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9754fa0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9750161ed0 0x7f97501622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f974ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f97501627f0 0x7f975006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9754fa0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9750161ed0 0x7f97501622b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37556/0 (socket says 172.21.15.122:37556) 2024-03-06T07:27:27.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f974ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f97501627f0 0x7f975006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54780/0 (socket says 172.21.15.122:54780) 2024-03-06T07:27:27.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f82c7aa4640 1 Processor -- start 2024-03-06T07:27:27.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9754fa0640 1 -- 172.21.15.122:0/3428350410 learned_addr learned my addr 172.21.15.122:0/3428350410 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:27.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f82c7aa4640 1 -- start start 2024-03-06T07:27:27.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f97557a1640 1 --2- 172.21.15.122:0/3428350410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97500afae0 0x7f975006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97500b3a90 con 0x7f9750161ed0 2024-03-06T07:27:27.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97500b3d70 con 0x7f97501627f0 2024-03-06T07:27:27.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.496+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97500b57a0 con 0x7f97500afae0 2024-03-06T07:27:27.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f9be9381640 1 Processor -- start 2024-03-06T07:27:27.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f9be9381640 1 -- start start 2024-03-06T07:27:27.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f97557a1640 1 -- 172.21.15.122:0/3428350410 >> v2:172.21.15.169:3300/0 conn(0x7f97501627f0 msgr2=0x7f975006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f97557a1640 1 --2- 172.21.15.122:0/3428350410 >> v2:172.21.15.169:3300/0 conn(0x7f97501627f0 0x7f975006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f97557a1640 1 -- 172.21.15.122:0/3428350410 >> v2:172.21.15.122:3300/0 conn(0x7f9750161ed0 msgr2=0x7f97501622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f97557a1640 1 --2- 172.21.15.122:0/3428350410 >> v2:172.21.15.122:3300/0 conn(0x7f9750161ed0 0x7f97501622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f97557a1640 1 -- 172.21.15.122:0/3428350410 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500b0270 con 0x7f97500afae0 2024-03-06T07:27:27.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c7aa4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 0x7f82c01f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c7aa4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82c01444c0 0x7f82c015a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c7aa4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c013a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c7aa4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c006def0 con 0x7f82c015b2d0 2024-03-06T07:27:27.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c7aa4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c006dc10 con 0x7f82c01444c0 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c7aa4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c006cf40 con 0x7f82c0070e10 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c5840640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 0x7f82c01f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c5840640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 0x7f82c01f2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47958/0 (socket says 172.21.15.122:47958) 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c5840640 1 -- 172.21.15.122:0/1844270731 learned_addr learned my addr 172.21.15.122:0/1844270731 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c503f640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82c01444c0 0x7f82c015a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.497+0000 7f82c6041640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c013a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f47abd7a640 1 Processor -- start 2024-03-06T07:27:27.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f47abd7a640 1 -- start start 2024-03-06T07:27:27.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c5840640 1 -- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82c01444c0 msgr2=0x7f82c015a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c5840640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82c01444c0 0x7f82c015a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c5840640 1 -- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 msgr2=0x7f82c013a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c5840640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c013a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c5840640 1 -- 172.21.15.122:0/1844270731 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c015aab0 con 0x7f82c0070e10 2024-03-06T07:27:27.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be9381640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40670a0 0x7f9be414f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be9381640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 0x7f9be414f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40670a0 0x7f9be414f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40670a0 0x7f9be414f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37570/0 (socket says 172.21.15.122:37570) 2024-03-06T07:27:27.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be9381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9be40d02b0 0x7f9be4098ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c5840640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 0x7f82c01f2bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00d800 tx=0x7f82bc00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 0x7f9be414f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 0x7f9be414f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54806/0 (socket says 172.21.15.122:54806) 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be2ffd640 1 -- 172.21.15.122:0/873779014 learned_addr learned my addr 172.21.15.122:0/873779014 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be37fe640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9be40d02b0 0x7f9be4098ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be406bd80 con 0x7f9be40670a0 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc00b6c0 con 0x7f82c0070e10 2024-03-06T07:27:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be406bc00 con 0x7f9be40b50d0 2024-03-06T07:27:27.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be40d0690 con 0x7f9be40d02b0 2024-03-06T07:27:27.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc019070 con 0x7f82c0070e10 2024-03-06T07:27:27.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c0139ad0 con 0x7f82c0070e10 2024-03-06T07:27:27.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f97557a1640 1 --2- 172.21.15.122:0/3428350410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97500afae0 0x7f975006b4f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9750071640 tx=0x7f974802be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0984656383fb1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc00bac0 con 0x7f82c0070e10 2024-03-06T07:27:27.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.498+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c013c390 con 0x7f82c0070e10 2024-03-06T07:27:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f974f7fe640 1 -- 172.21.15.122:0/3428350410 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748039070 con 0x7f97500afae0 2024-03-06T07:27:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f974f7fe640 1 -- 172.21.15.122:0/3428350410 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974802cbe0 con 0x7f97500afae0 2024-03-06T07:27:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be27fc640 1 -- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9be40d02b0 msgr2=0x7f9be4098ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be27fc640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9be40d02b0 0x7f9be4098ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be27fc640 1 -- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40670a0 msgr2=0x7f9be414f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be27fc640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40670a0 0x7f9be414f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be27fc640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be40c78c0 con 0x7f9be40b50d0 2024-03-06T07:27:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f974f7fe640 1 -- 172.21.15.122:0/3428350410 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748034440 con 0x7f97500afae0 2024-03-06T07:27:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be27fc640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 0x7f9be414f5d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8000f60 tx=0x7f9bd800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47abd7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a40b4460 0x7f47a4155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47abd7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 0x7f47a40c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8017070 con 0x7f9be40b50d0 2024-03-06T07:27:27.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd8005ce0 con 0x7f9be40b50d0 2024-03-06T07:27:27.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be40c3560 con 0x7f9be40b50d0 2024-03-06T07:27:27.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd80072f0 con 0x7f9be40b50d0 2024-03-06T07:27:27.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be40c7270 con 0x7f9be40b50d0 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47a9b16640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a40b4460 0x7f47a4155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47a9315640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 0x7f47a40c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47abd7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47a4162970 0x7f47a41520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47a9315640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 0x7f47a40c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37584/0 (socket says 172.21.15.122:37584) 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f47a9b16640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a40b4460 0x7f47a4155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47976/0 (socket says 172.21.15.122:47976) 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.499+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c004f330 con 0x7f82c0070e10 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47a9315640 1 -- 172.21.15.122:0/1882198993 learned_addr learned my addr 172.21.15.122:0/1882198993 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a40ce950 con 0x7f47a4161cf0 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a40d2b60 con 0x7f47a4162970 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a40b3ac0 con 0x7f47a40b4460 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97500afae0 msgr2=0x7f975006b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 --2- 172.21.15.122:0/3428350410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97500afae0 0x7f975006b4f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9750071640 tx=0x7f974802be30 comp rx=0 tx=0).stop 2024-03-06T07:27:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82bc00bc60 con 0x7f82c0070e10 2024-03-06T07:27:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 shutdown_connections 2024-03-06T07:27:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 --2- 172.21.15.122:0/3428350410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f97500afae0 0x7f975006b4f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 --2- 172.21.15.122:0/3428350410 >> v2:172.21.15.169:3300/0 conn(0x7f97501627f0 0x7f975006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 --2- 172.21.15.122:0/3428350410 >> v2:172.21.15.122:3300/0 conn(0x7f9750161ed0 0x7f97501622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 >> 172.21.15.122:0/3428350410 conn(0x7f975006c0c0 msgr2=0x7f975006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47aa317640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47a4162970 0x7f47a41520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47a9315640 1 -- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a40b4460 msgr2=0x7f47a4155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47a9315640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a40b4460 0x7f47a4155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47a9315640 1 -- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47a4162970 msgr2=0x7f47a41520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47a9315640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47a4162970 0x7f47a41520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47a9315640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a40ca440 con 0x7f47a4161cf0 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f82b67fc640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f82a406b4d0 0x7f82a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.500+0000 7f47aa317640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47a4162970 0x7f47a41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82bc08afa0 con 0x7f82c0070e10 2024-03-06T07:27:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f82c503f640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f82a406b4d0 0x7f82a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f47a9315640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 0x7f47a40c7810 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f47940098c0 tx=0x7f47940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 shutdown_connections 2024-03-06T07:27:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f9755fa2640 1 -- 172.21.15.122:0/3428350410 wait complete. 2024-03-06T07:27:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be404f330 con 0x7f9be40b50d0 2024-03-06T07:27:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f82c503f640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f82a406b4d0 0x7f82a406d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f82b0036000 tx=0x7f82b003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bd801a070 con 0x7f9be40b50d0 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.502+0000 7f9bcb7fe640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9bd006b4d0 0x7f9bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794041070 con 0x7f47a4161cf0 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479403ad00 con 0x7f47a4161cf0 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794035040 con 0x7f47a4161cf0 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a40ca720 con 0x7f47a4161cf0 2024-03-06T07:27:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.501+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a40b8030 con 0x7f47a4161cf0 2024-03-06T07:27:27.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.502+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bd80893c0 con 0x7f9be40b50d0 2024-03-06T07:27:27.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.502+0000 7f9be2ffd640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9bd006b4d0 0x7f9bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.502+0000 7f9be2ffd640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9bd006b4d0 0x7f9bd006d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009b60 tx=0x7f9bd4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.502+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a40b7b40 con 0x7f47a4161cf0 2024-03-06T07:27:27.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.503+0000 7f9755fa2640 1 Processor -- start 2024-03-06T07:27:27.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.503+0000 7f9755fa2640 1 -- start start 2024-03-06T07:27:27.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.503+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f479404c060 con 0x7f47a4161cf0 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.503+0000 7f479a7fc640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f478806b4d0 0x7f478806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.503+0000 7f47a9b16640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f478806b4d0 0x7f478806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.503+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47940abd80 con 0x7f47a4161cf0 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f47a9b16640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f478806b4d0 0x7f478806d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009aa0 tx=0x7f47a0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9755fa2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97500afae0 0x7f975015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9755fa2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9750161ed0 0x7f97500ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9755fa2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97501627f0 0x7f97500a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9755fa2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975006f410 con 0x7f97501627f0 2024-03-06T07:27:27.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f97557a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97501627f0 0x7f97500a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9755fa2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975006f290 con 0x7f97500afae0 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9754fa0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97500afae0 0x7f975015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9755fa2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975006f590 con 0x7f9750161ed0 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9754fa0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97500afae0 0x7f975015cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:54818/0 (socket says 172.21.15.122:54818) 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f9754fa0640 1 -- 172.21.15.122:0/1887011705 learned_addr learned my addr 172.21.15.122:0/1887011705 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.504+0000 7f974ffff640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9750161ed0 0x7f97500ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f97557a1640 1 -- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9750161ed0 msgr2=0x7f97500ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f97557a1640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9750161ed0 0x7f97500ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f97557a1640 1 -- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97500afae0 msgr2=0x7f975015cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f97557a1640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97500afae0 0x7f975015cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f97557a1640 1 -- 172.21.15.122:0/1887011705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975015d1a0 con 0x7f97501627f0 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f974ffff640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9750161ed0 0x7f97500ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82bc03c030 con 0x7f82c0070e10 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.505+0000 7f97557a1640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97501627f0 0x7f97500a9f00 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9748033000 tx=0x7f9748004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.506+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bd8057c80 con 0x7f9be40b50d0 2024-03-06T07:27:27.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.509+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748044020 con 0x7f97501627f0 2024-03-06T07:27:27.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.509+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9748035e50 con 0x7f97501627f0 2024-03-06T07:27:27.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.509+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748039040 con 0x7f97501627f0 2024-03-06T07:27:27.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.509+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97500cd0c0 con 0x7f97501627f0 2024-03-06T07:27:27.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.509+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f479407ac00 con 0x7f47a4161cf0 2024-03-06T07:27:27.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.509+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97500cd680 con 0x7f97501627f0 2024-03-06T07:27:27.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.511+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9750068f80 con 0x7f97501627f0 2024-03-06T07:27:27.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.511+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9748047020 con 0x7f97501627f0 2024-03-06T07:27:27.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.512+0000 7f974d7fa640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f973406b5a0 0x7f973406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.512+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97480aca90 con 0x7f97501627f0 2024-03-06T07:27:27.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.512+0000 7f9754fa0640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f973406b5a0 0x7f973406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.512+0000 7f9754fa0640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f973406b5a0 0x7f973406da60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9744006f40 tx=0x7f9744009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.516+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f974804e030 con 0x7f97501627f0 2024-03-06T07:27:27.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.759+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f9be406eeb0 con 0x7f9be40b50d0 2024-03-06T07:27:27.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.760+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f47a404f330 con 0x7f47a4161cf0 2024-03-06T07:27:27.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.779+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f82c00748c0 con 0x7f82c0070e10 2024-03-06T07:27:27.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:27.787+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f97500747a0 con 0x7f97501627f0 2024-03-06T07:27:28.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.490+0000 7f9bcb7fe640 1 -- 172.21.15.122:0/873779014 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7959 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7f9bd805bd20 con 0x7f9be40b50d0 2024-03-06T07:27:28.492 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 7959 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9bd006b4d0 msgr2=0x7f9bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9bd006b4d0 0x7f9bd006d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009b60 tx=0x7f9bd4009190 comp rx=0 tx=0).stop 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 msgr2=0x7f9be414f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 0x7f9be414f5d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8000f60 tx=0x7f9bd800c800 comp rx=0 tx=0).stop 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 shutdown_connections 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9bd006b4d0 0x7f9bd006d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9be40d02b0 0x7f9be4098ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9be40b50d0 0x7f9be414f5d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 --2- 172.21.15.122:0/873779014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9be40670a0 0x7f9be414f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 >> 172.21.15.122:0/873779014 conn(0x7f9be4072520 msgr2=0x7f9be40cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 shutdown_connections 2024-03-06T07:27:28.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.493+0000 7f9be9381640 1 -- 172.21.15.122:0/873779014 wait complete. 2024-03-06T07:27:28.500 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:27:28.498+0000 7fdd670e6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:28.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.500+0000 7f479a7fc640 1 -- 172.21.15.122:0/1882198993 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6518 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f479407a500 con 0x7f47a4161cf0 2024-03-06T07:27:28.502 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6518 2024-03-06T07:27:28.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.501+0000 7f974d7fa640 1 -- 172.21.15.122:0/1887011705 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7962 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f974807b3c0 con 0x7f97501627f0 2024-03-06T07:27:28.503 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 7962 2024-03-06T07:27:28.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f478806b4d0 msgr2=0x7f478806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f478806b4d0 0x7f478806d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009aa0 tx=0x7f47a0009190 comp rx=0 tx=0).stop 2024-03-06T07:27:28.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 msgr2=0x7f47a40c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 0x7f47a40c7810 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f47940098c0 tx=0x7f47940046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 shutdown_connections 2024-03-06T07:27:28.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f478806b4d0 0x7f478806d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47a4162970 0x7f47a41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a4161cf0 0x7f47a40c7810 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 --2- 172.21.15.122:0/1882198993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47a40b4460 0x7f47a4155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 >> 172.21.15.122:0/1882198993 conn(0x7f47a40d3ce0 msgr2=0x7f47a406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.504+0000 7f82b67fc640 1 -- 172.21.15.122:0/1844270731 <== mon.2 v2:172.21.15.169:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7207 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f82bc0598b0 con 0x7f82c0070e10 2024-03-06T07:27:28.507 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 7207 2024-03-06T07:27:28.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f973406b5a0 msgr2=0x7f973406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f973406b5a0 0x7f973406da60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9744006f40 tx=0x7f9744009240 comp rx=0 tx=0).stop 2024-03-06T07:27:28.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97501627f0 msgr2=0x7f97500a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97501627f0 0x7f97500a9f00 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9748033000 tx=0x7f9748004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:28.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 shutdown_connections 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f47abd7a640 1 -- 172.21.15.122:0/1882198993 wait complete. 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 shutdown_connections 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f973406b5a0 0x7f973406da60 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97501627f0 0x7f97500a9f00 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9750161ed0 0x7f97500ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 --2- 172.21.15.122:0/1887011705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97500afae0 0x7f975015cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 >> 172.21.15.122:0/1887011705 conn(0x7f975006c0c0 msgr2=0x7f975015db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 shutdown_connections 2024-03-06T07:27:28.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.505+0000 7f9755fa2640 1 -- 172.21.15.122:0/1887011705 wait complete. 2024-03-06T07:27:28.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f82a406b4d0 msgr2=0x7f82a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f82a406b4d0 0x7f82a406d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f82b0036000 tx=0x7f82b003b040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 msgr2=0x7f82c01f2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 0x7f82c01f2bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00d800 tx=0x7f82bc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 shutdown_connections 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f82a406b4d0 0x7f82a406d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c015b2d0 0x7f82c013a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82c01444c0 0x7f82c015a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 --2- 172.21.15.122:0/1844270731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f82c0070e10 0x7f82c01f2bf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.507+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 >> 172.21.15.122:0/1844270731 conn(0x7f82c00b1c80 msgr2=0x7f82c00b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.508+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 shutdown_connections 2024-03-06T07:27:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.508+0000 7f82c7aa4640 1 -- 172.21.15.122:0/1844270731 wait complete. 2024-03-06T07:27:28.512 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:28.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:28.515 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:28.515 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:28.526 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:28.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:28.528 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:28.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:28.529 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:28.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:28.530 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:28.530 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:28.532 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:28.532 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:28.571 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:28.571 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:28.571 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:28.596 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:27:28.611 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:27:28.612 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:27:28.648 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:27:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 Processor -- start 2024-03-06T07:27:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 -- start start 2024-03-06T07:27:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb52015dfe0 0x7fb5200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb5200cf830 0x7fb5200d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5200b3430 con 0x7fb5200d27f0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200d2c50 con 0x7fb52015dfe0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb526864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200b31e0 con 0x7fb5200cf830 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb51ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb52015dfe0 0x7fb5200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.673+0000 7fb51ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb52015dfe0 0x7fb5200cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37650/0 (socket says 172.21.15.122:37650) 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51ffff640 1 -- 172.21.15.122:0/1964542623 learned_addr learned my addr 172.21.15.122:0/1964542623 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51f7fe640 1 --2- 172.21.15.122:0/1964542623 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb5200cf830 0x7fb5200d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51ffff640 1 -- 172.21.15.122:0/1964542623 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb5200cf830 msgr2=0x7fb5200d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51ffff640 1 --2- 172.21.15.122:0/1964542623 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb5200cf830 0x7fb5200d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51ffff640 1 -- 172.21.15.122:0/1964542623 >> v1:172.21.15.169:6789/0 conn(0x7fb5200d27f0 legacy=0x7fb5200d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51ffff640 1 -- 172.21.15.122:0/1964542623 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5200b3710 con 0x7fb52015dfe0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.674+0000 7fb51ffff640 1 --2- 172.21.15.122:0/1964542623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb52015dfe0 0x7fb5200cf2f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb510009870 tx=0x7fb51002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0491530db3945c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb51effd640 1 -- 172.21.15.122:0/1964542623 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510035020 con 0x7fb52015dfe0 2024-03-06T07:27:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb51effd640 1 -- 172.21.15.122:0/1964542623 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb510031640 con 0x7fb52015dfe0 2024-03-06T07:27:28.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb51effd640 1 -- 172.21.15.122:0/1964542623 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51002d040 con 0x7fb52015dfe0 2024-03-06T07:27:28.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 -- 172.21.15.122:0/1964542623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb52015dfe0 msgr2=0x7fb5200cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 --2- 172.21.15.122:0/1964542623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb52015dfe0 0x7fb5200cf2f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb510009870 tx=0x7fb51002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 -- 172.21.15.122:0/1964542623 shutdown_connections 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 --2- 172.21.15.122:0/1964542623 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb5200cf830 0x7fb5200d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 --2- 172.21.15.122:0/1964542623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb52015dfe0 0x7fb5200cf2f0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 -- 172.21.15.122:0/1964542623 >> 172.21.15.122:0/1964542623 conn(0x7fb5200d3cd0 msgr2=0x7fb5200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 -- 172.21.15.122:0/1964542623 shutdown_connections 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.675+0000 7fb526864640 1 -- 172.21.15.122:0/1964542623 wait complete. 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.676+0000 7fb526864640 1 Processor -- start 2024-03-06T07:27:28.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 -- start start 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5200cf830 0x7fb52014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 0x7fb5200b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb52015dfe0 0x7fb520159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200d1c70 con 0x7fb5200d27f0 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200b3d60 con 0x7fb5200cf830 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb526864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200d1df0 con 0x7fb52015dfe0 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb51f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 0x7fb5200b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb51f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 0x7fb5200b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37660/0 (socket says 172.21.15.122:37660) 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb51f7fe640 1 -- 172.21.15.122:0/2328213212 learned_addr learned my addr 172.21.15.122:0/2328213212 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb51ffff640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5200cf830 0x7fb52014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.677+0000 7fb524e01640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb52015dfe0 0x7fb520159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51f7fe640 1 -- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb52015dfe0 msgr2=0x7fb520159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51f7fe640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb52015dfe0 0x7fb520159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51f7fe640 1 -- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5200cf830 msgr2=0x7fb52014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51f7fe640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5200cf830 0x7fb52014d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51f7fe640 1 -- 172.21.15.122:0/2328213212 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52014dff0 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51f7fe640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 0x7fb5200b68a0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fb51400c7f0 tx=0x7fb51400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514015070 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb514004220 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5201600e0 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514007620 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.678+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5201606a0 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.679+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5200ce3c0 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.679+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5140079d0 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.680+0000 7fb51cff9640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb50c06b500 0x7fb50c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.680+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb51408a220 con 0x7fb5200d27f0 2024-03-06T07:27:28.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.680+0000 7fb51ffff640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb50c06b500 0x7fb50c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.680+0000 7fb51ffff640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb50c06b500 0x7fb50c06d9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb510036000 tx=0x7fb51003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.684+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5200ce3c0 con 0x7fb5200d27f0 2024-03-06T07:27:28.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.892+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5200d0c80 con 0x7fb5200d27f0 2024-03-06T07:27:28.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.892+0000 7fb51cff9640 1 -- 172.21.15.122:0/2328213212 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1802 (secure 0 0 0) 0x7fb5140589f0 con 0x7fb5200d27f0 2024-03-06T07:27:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:28.895 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:28.499632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:28.895 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 63 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb50c06b500 msgr2=0x7fb50c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb50c06b500 0x7fb50c06d9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb510036000 tx=0x7fb51003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 msgr2=0x7fb5200b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 0x7fb5200b68a0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fb51400c7f0 tx=0x7fb51400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 shutdown_connections 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb50c06b500 0x7fb50c06d9c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb52015dfe0 0x7fb520159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5200d27f0 0x7fb5200b68a0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 --2- 172.21.15.122:0/2328213212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5200cf830 0x7fb52014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.894+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 >> 172.21.15.122:0/2328213212 conn(0x7fb5200d3cd0 msgr2=0x7fb52006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.895+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 shutdown_connections 2024-03-06T07:27:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:28.895+0000 7fb526864640 1 -- 172.21.15.122:0/2328213212 wait complete. 2024-03-06T07:27:29.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.604168+0000', 'modified': '2024-03-06T07:27:28.499632+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:29.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:27:30.310 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.425+0000 7f4a6dcb8640 1 Processor -- start 2024-03-06T07:27:30.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.425+0000 7f4a6dcb8640 1 -- start start 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a6dcb8640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4a680cce40 0x7f4a6815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a6dcb8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a68071f40 con 0x7f4a681515e0 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a6dcb8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a68070a70 con 0x7f4a680cc340 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a6dcb8640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a680cd2f0 con 0x7f4a680cce40 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a677fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4a681515e0 0x7f4a68071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42998/0 (socket says 172.21.15.122:42998) 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a677fe640 1 -- 172.21.15.122:0/2712135664 learned_addr learned my addr 172.21.15.122:0/2712135664 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a66ffd640 1 --2- 172.21.15.122:0/2712135664 >> v2:172.21.15.169:3300/0 conn(0x7f4a680cce40 0x7f4a6815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697753297 0 0) 0x7f4a68071f40 con 0x7f4a681515e0 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.426+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a50003520 con 0x7f4a681515e0 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238452645 0 0) 0x7f4a50003520 con 0x7f4a681515e0 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 >> v2:172.21.15.169:3300/0 conn(0x7f4a680cce40 msgr2=0x7f4a6815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 --2- 172.21.15.122:0/2712135664 >> v2:172.21.15.169:3300/0 conn(0x7f4a680cce40 0x7f4a6815b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 >> v1:172.21.15.169:6789/0 conn(0x7f4a680cc340 legacy=0x7f4a680cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68070d50 con 0x7f4a681515e0 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4a58001670 con 0x7f4a681515e0 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 >> v1:172.21.15.122:6789/0 conn(0x7f4a681515e0 legacy=0x7f4a68071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a500039b0 0x7f4a50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a500051c0 0x7f4a5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680cd2f0 con 0x7f4a50004800 2024-03-06T07:27:30.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5000c060 con 0x7f4a500051c0 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5000c260 con 0x7f4a500039b0 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a66ffd640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a67fff640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a500051c0 0x7f4a5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a677fe640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a500039b0 0x7f4a50003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a66ffd640 1 -- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a500039b0 msgr2=0x7f4a50003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a66ffd640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a500039b0 0x7f4a50003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a66ffd640 1 -- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a500051c0 msgr2=0x7f4a5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a66ffd640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a500051c0 0x7f4a5000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a66ffd640 1 -- 172.21.15.122:0/2712135664 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a50003520 con 0x7f4a50004800 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a67fff640 1 -- 172.21.15.122:0/2712135664 reap_dead start 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.427+0000 7f4a677fe640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a500039b0 0x7f4a50003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.428+0000 7f4a66ffd640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009ac0 tx=0x7f4a50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd78e6c82336203 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.428+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c039070 con 0x7f4a50004800 2024-03-06T07:27:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.428+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c02c6b0 con 0x7f4a50004800 2024-03-06T07:27:30.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.428+0000 7f4a667fc640 1 -- 172.21.15.122:0/2712135664 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c009690 con 0x7f4a50004800 2024-03-06T07:27:30.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/2712135664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a50004800 msgr2=0x7f4a50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009ac0 tx=0x7f4a50005830 comp rx=0 tx=0).stop 2024-03-06T07:27:30.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/2712135664 shutdown_connections 2024-03-06T07:27:30.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/2712135664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/2712135664 >> 172.21.15.122:0/2712135664 conn(0x7f4a680cf8a0 msgr2=0x7f4a680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/2712135664 shutdown_connections 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.429+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/2712135664 wait complete. 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.430+0000 7f4a6dcb8640 1 Processor -- start 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.430+0000 7f4a6dcb8640 1 -- start start 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a6dcb8640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a50004800 0x7f4a68163730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a6dcb8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a681523a0 0x7f4a68152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a6dcb8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 0x7f4a6814b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a6dcb8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680cc720 con 0x7f4a6814b1d0 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a6dcb8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68071310 con 0x7f4a681523a0 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a6dcb8640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680cc8a0 con 0x7f4a50004800 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 0x7f4a6814b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 0x7f4a6814b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37686/0 (socket says 172.21.15.122:37686) 2024-03-06T07:27:30.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 -- 172.21.15.122:0/1550804761 learned_addr learned my addr 172.21.15.122:0/1550804761 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:30.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a677fe640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a50004800 0x7f4a68163730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a66ffd640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a681523a0 0x7f4a68152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 -- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a50004800 msgr2=0x7f4a68163730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a50004800 0x7f4a68163730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 -- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a681523a0 msgr2=0x7f4a68152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a681523a0 0x7f4a68152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.431+0000 7f4a67fff640 1 -- 172.21.15.122:0/1550804761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6814bb90 con 0x7f4a6814b1d0 2024-03-06T07:27:30.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.432+0000 7f4a67fff640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 0x7f4a6814b650 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4a680b3d60 tx=0x7f4a6000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.432+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6001a070 con 0x7f4a6814b1d0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.432+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60006d60 con 0x7f4a6814b1d0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.432+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6815df60 con 0x7f4a6814b1d0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.432+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60003bb0 con 0x7f4a6814b1d0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.432+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6815e520 con 0x7f4a6814b1d0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.433+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a68068840 con 0x7f4a6814b1d0 2024-03-06T07:27:30.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.433+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a6001d070 con 0x7f4a6814b1d0 2024-03-06T07:27:30.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.434+0000 7f4a4ffff640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4a5406b4d0 0x7f4a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.434+0000 7f4a677fe640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4a5406b4d0 0x7f4a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.434+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4a60089950 con 0x7f4a6814b1d0 2024-03-06T07:27:30.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.434+0000 7f4a677fe640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4a5406b4d0 0x7f4a5406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4a680686a0 tx=0x7f4a58001450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.438+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a60058800 con 0x7f4a6814b1d0 2024-03-06T07:27:30.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.649+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a6804f330 con 0x7f4a6814b1d0 2024-03-06T07:27:30.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.650+0000 7f4a4ffff640 1 -- 172.21.15.122:0/1550804761 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f4a60058100 con 0x7f4a6814b1d0 2024-03-06T07:27:30.652 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:30.652 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:28.978919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:30.652 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 65 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.651+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4a5406b4d0 msgr2=0x7f4a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.651+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4a5406b4d0 0x7f4a5406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4a680686a0 tx=0x7f4a58001450 comp rx=0 tx=0).stop 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.651+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 msgr2=0x7f4a6814b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.651+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 0x7f4a6814b650 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4a680b3d60 tx=0x7f4a6000fc00 comp rx=0 tx=0).stop 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 shutdown_connections 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4a5406b4d0 0x7f4a5406d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a6814b1d0 0x7f4a6814b650 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a681523a0 0x7f4a68152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 --2- 172.21.15.122:0/1550804761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a50004800 0x7f4a68163730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 >> 172.21.15.122:0/1550804761 conn(0x7f4a680cf8a0 msgr2=0x7f4a680ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 shutdown_connections 2024-03-06T07:27:30.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:30.652+0000 7f4a6dcb8640 1 -- 172.21.15.122:0/1550804761 wait complete. 2024-03-06T07:27:31.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.604168+0000', 'modified': '2024-03-06T07:27:28.978919+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8283': {'gid': 8283, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.122:6837/2123619089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 2123619089}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2123619089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:31.067 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8283:up:reconnect 2024-03-06T07:27:32.068 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:32.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.170+0000 7fda696af640 1 Processor -- start 2024-03-06T07:27:32.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.170+0000 7fda696af640 1 -- start start 2024-03-06T07:27:32.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda696af640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda6415dfc0 0x7fda6406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda696af640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fda6406e480 0x7fda6406d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda696af640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6406bb60 con 0x7fda6406d7a0 2024-03-06T07:27:32.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda696af640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6415e3a0 con 0x7fda6415dfc0 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda696af640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6406be40 con 0x7fda6406e480 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda62ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda6415dfc0 0x7fda6406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda637fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fda6406d7a0 0x7fda640cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43008/0 (socket says 172.21.15.122:43008) 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda637fe640 1 -- 172.21.15.122:0/3492444256 learned_addr learned my addr 172.21.15.122:0/3492444256 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda62ffd640 1 -- 172.21.15.122:0/3492444256 >> v2:172.21.15.169:3300/0 conn(0x7fda6406e480 msgr2=0x7fda6406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda62ffd640 1 --2- 172.21.15.122:0/3492444256 >> v2:172.21.15.169:3300/0 conn(0x7fda6406e480 0x7fda6406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda62ffd640 1 -- 172.21.15.122:0/3492444256 >> v1:172.21.15.122:6789/0 conn(0x7fda6406d7a0 legacy=0x7fda640cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda62ffd640 1 -- 172.21.15.122:0/3492444256 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6406c120 con 0x7fda6415dfc0 2024-03-06T07:27:32.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.172+0000 7fda61ffb640 1 -- 172.21.15.122:0/3492444256 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242115341 0 0) 0x7fda6406bb60 con 0x7fda6406d7a0 2024-03-06T07:27:32.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda62ffd640 1 --2- 172.21.15.122:0/3492444256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6415dfc0 0x7fda6406df40 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fda640ccb10 tx=0x7fda5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d69b7269963b68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda61ffb640 1 -- 172.21.15.122:0/3492444256 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54039070 con 0x7fda6415dfc0 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda61ffb640 1 -- 172.21.15.122:0/3492444256 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5402cbf0 con 0x7fda6415dfc0 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda61ffb640 1 -- 172.21.15.122:0/3492444256 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54034440 con 0x7fda6415dfc0 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda696af640 1 -- 172.21.15.122:0/3492444256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6415dfc0 msgr2=0x7fda6406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda696af640 1 --2- 172.21.15.122:0/3492444256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6415dfc0 0x7fda6406df40 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fda640ccb10 tx=0x7fda5402be40 comp rx=0 tx=0).stop 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda696af640 1 -- 172.21.15.122:0/3492444256 shutdown_connections 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda696af640 1 --2- 172.21.15.122:0/3492444256 >> v2:172.21.15.169:3300/0 conn(0x7fda6406e480 0x7fda6406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda696af640 1 --2- 172.21.15.122:0/3492444256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6415dfc0 0x7fda6406df40 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.173+0000 7fda696af640 1 -- 172.21.15.122:0/3492444256 >> 172.21.15.122:0/3492444256 conn(0x7fda640d00e0 msgr2=0x7fda640d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.174+0000 7fda696af640 1 -- 172.21.15.122:0/3492444256 shutdown_connections 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.174+0000 7fda696af640 1 -- 172.21.15.122:0/3492444256 wait complete. 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.175+0000 7fda696af640 1 Processor -- start 2024-03-06T07:27:32.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.175+0000 7fda696af640 1 -- start start 2024-03-06T07:27:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda6406d7a0 0x7fda64158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 0x7fda640c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda6415dfc0 0x7fda640b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640ae320 con 0x7fda6406e480 2024-03-06T07:27:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6406dbb0 con 0x7fda6406d7a0 2024-03-06T07:27:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640ae4a0 con 0x7fda6415dfc0 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 0x7fda640c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 0x7fda640c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37700/0 (socket says 172.21.15.122:37700) 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 -- 172.21.15.122:0/2177195921 learned_addr learned my addr 172.21.15.122:0/2177195921 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda62ffd640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda6406d7a0 0x7fda64158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda637fe640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda6415dfc0 0x7fda640b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 -- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda6415dfc0 msgr2=0x7fda640b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda6415dfc0 0x7fda640b1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 -- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda6406d7a0 msgr2=0x7fda64158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda6406d7a0 0x7fda64158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 -- 172.21.15.122:0/2177195921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6415e660 con 0x7fda6406e480 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda637fe640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda6415dfc0 0x7fda640b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda627fc640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 0x7fda640c7260 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fda5800b410 tx=0x7fda5800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58015020 con 0x7fda6406e480 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda58011850 con 0x7fda6406e480 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6415e900 con 0x7fda6406e480 2024-03-06T07:27:32.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58011c40 con 0x7fda6406e480 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.176+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6415eec0 con 0x7fda6406e480 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.177+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6404f330 con 0x7fda6406e480 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.178+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda58018020 con 0x7fda6406e480 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.178+0000 7fda4b7fe640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda5006b4d0 0x7fda5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.178+0000 7fda62ffd640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda5006b4d0 0x7fda5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.178+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fda5808a110 con 0x7fda6406e480 2024-03-06T07:27:32.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.179+0000 7fda62ffd640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda5006b4d0 0x7fda5006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fda54036000 tx=0x7fda54030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.183+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda5801f020 con 0x7fda6406e480 2024-03-06T07:27:32.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.433+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda640d08b0 con 0x7fda6406e480 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.434+0000 7fda4b7fe640 1 -- 172.21.15.122:0/2177195921 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fda58058f30 con 0x7fda6406e480 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:27:32.436 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:27:32.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda5006b4d0 msgr2=0x7fda5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda5006b4d0 0x7fda5006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fda54036000 tx=0x7fda54030000 comp rx=0 tx=0).stop 2024-03-06T07:27:32.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 msgr2=0x7fda640c7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 0x7fda640c7260 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fda5800b410 tx=0x7fda5800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 shutdown_connections 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda5006b4d0 0x7fda5006d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda6415dfc0 0x7fda640b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda6406e480 0x7fda640c7260 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 --2- 172.21.15.122:0/2177195921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda6406d7a0 0x7fda64158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 >> 172.21.15.122:0/2177195921 conn(0x7fda640d00e0 msgr2=0x7fda640afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 shutdown_connections 2024-03-06T07:27:32.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.436+0000 7fda696af640 1 -- 172.21.15.122:0/2177195921 wait complete. 2024-03-06T07:27:32.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.604168+0000', 'modified': '2024-03-06T07:27:31.978819+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8283': {'gid': 8283, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.122:6837/2123619089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 2123619089}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2123619089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8283, 'qdb_cluster': [8283]} 2024-03-06T07:27:32.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:32.852 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:27:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.953+0000 7fb9cf2e1640 1 Processor -- start 2024-03-06T07:27:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.953+0000 7fb9cf2e1640 1 -- start start 2024-03-06T07:27:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cf2e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 0x7fb9c80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cf2e1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb9c8161cf0 0x7fb9c8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cf2e1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80d22c0 con 0x7fb9c80b4460 2024-03-06T07:27:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cf2e1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c8161b00 con 0x7fb9c8161cf0 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cf2e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80cecc0 con 0x7fb9c8162970 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cd07d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb9c80b4460 0x7fb9c80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43022/0 (socket says 172.21.15.122:43022) 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.954+0000 7fb9cd07d640 1 -- 172.21.15.122:0/1329610737 learned_addr learned my addr 172.21.15.122:0/1329610737 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 0x7fb9c80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1329610737 >> v2:172.21.15.169:3300/0 conn(0x7fb9c8161cf0 0x7fb9c8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269586676 0 0) 0x7fb9c80d22c0 con 0x7fb9c80b4460 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b4003520 con 0x7fb9c80b4460 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1076084179 0 0) 0x7fb9b4003520 con 0x7fb9c80b4460 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 msgr2=0x7fb9c80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 0x7fb9c80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 >> v2:172.21.15.169:3300/0 conn(0x7fb9c8161cf0 msgr2=0x7fb9c8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 --2- 172.21.15.122:0/1329610737 >> v2:172.21.15.169:3300/0 conn(0x7fb9c8161cf0 0x7fb9c8162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80d25a0 con 0x7fb9c80b4460 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb9c4001670 con 0x7fb9c80b4460 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 >> v1:172.21.15.122:6789/0 conn(0x7fb9c80b4460 legacy=0x7fb9c80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 0x7fb9b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9b4003f30 0x7fb9b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8161b00 con 0x7fb9b4003f30 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80cecc0 con 0x7fb9b4004d40 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b400b9c0 con 0x7fb9c8162970 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9b4003f30 0x7fb9b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 0x7fb9b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 -- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 msgr2=0x7fb9b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c8162970 0x7fb9b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 -- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4004d40 msgr2=0x7fb9b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cc87c640 1 -- 172.21.15.122:0/1329610737 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b4003520 con 0x7fb9b4003f30 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.955+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.956+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9b4003f30 0x7fb9b4004800 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009a90 tx=0x7fb9b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1f887277f3cefd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8039070 con 0x7fb9b4003f30 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b802c6b0 con 0x7fb9b4003f30 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9bffff640 1 -- 172.21.15.122:0/1329610737 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8009690 con 0x7fb9b4003f30 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/1329610737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9b4003f30 msgr2=0x7fb9b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/1329610737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9b4003f30 0x7fb9b4004800 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009a90 tx=0x7fb9b4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:32.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1329610737 reap_dead start 2024-03-06T07:27:32.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/1329610737 shutdown_connections 2024-03-06T07:27:32.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/1329610737 >> 172.21.15.122:0/1329610737 conn(0x7fb9c80d3ce0 msgr2=0x7fb9c80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/1329610737 shutdown_connections 2024-03-06T07:27:32.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.957+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/1329610737 wait complete. 2024-03-06T07:27:32.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.958+0000 7fb9cf2e1640 1 Processor -- start 2024-03-06T07:27:32.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.958+0000 7fb9cf2e1640 1 -- start start 2024-03-06T07:27:32.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cf2e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4003f30 0x7fb9c8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cf2e1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c815eb40 0x7fb9c815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cf2e1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 0x7fb9c80c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cf2e1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80ce950 con 0x7fb9c80c90c0 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cf2e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80d2b60 con 0x7fb9b4003f30 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cf2e1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80b4840 con 0x7fb9c815eb40 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 0x7fb9c80c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 0x7fb9c80c9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37722/0 (socket says 172.21.15.122:37722) 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 -- 172.21.15.122:0/76500524 learned_addr learned my addr 172.21.15.122:0/76500524 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c815eb40 0x7fb9c815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4003f30 0x7fb9c8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 -- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c815eb40 msgr2=0x7fb9c815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c815eb40 0x7fb9c815efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 -- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4003f30 msgr2=0x7fb9c8156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4003f30 0x7fb9c8156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.959+0000 7fb9cd87e640 1 -- 172.21.15.122:0/76500524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c81f4410 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.960+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 0x7fb9c80c9540 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80a1c60 tx=0x7fb9c0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.960+0000 7fb9bdffb640 1 -- 172.21.15.122:0/76500524 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c001a030 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.960+0000 7fb9bdffb640 1 -- 172.21.15.122:0/76500524 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c0010d80 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.960+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c81f46f0 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.960+0000 7fb9bdffb640 1 -- 172.21.15.122:0/76500524 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0003bd0 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.960+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c81f4cb0 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.961+0000 7fb9bdffb640 1 -- 172.21.15.122:0/76500524 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9c001d070 con 0x7fb9c80c90c0 2024-03-06T07:27:32.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.962+0000 7fb9bdffb640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb9ac06b4d0 0x7fb9ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.962+0000 7fb9bdffb640 1 -- 172.21.15.122:0/76500524 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9c008a710 con 0x7fb9c80c90c0 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.962+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb9ac06b4d0 0x7fb9ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.963+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb9ac06b4d0 0x7fb9ac06d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80d3740 tx=0x7fb9c40078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.963+0000 7fb9d1f9c740 1 Processor -- start 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.963+0000 7fb9d1f9c740 1 -- start start 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0a91870 0x55eba0a91c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0a958c0 0x55eba0a97360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 0x55eba0ac1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eba09fb6b0 con 0x55eba0ac1600 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eba09a4a60 con 0x55eba0a958c0 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eba09f7750 con 0x55eba0a91870 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 0x55eba0ac1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 0x55eba0ac1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37726/0 (socket says 172.21.15.122:37726) 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1689791697 learned_addr learned my addr 172.21.15.122:0/1689791697 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1689791697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0a91870 msgr2=0x55eba0a91c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0a91870 0x55eba0a91c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1689791697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0a958c0 msgr2=0x55eba0a97360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0a958c0 0x55eba0a97360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1689791697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eba09c2f30 con 0x55eba0ac1600 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 0x55eba0ac1a80 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0002230 tx=0x7fb9c00588e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a2df44c7574943 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.964+0000 7fb9b2ffd640 1 -- 172.21.15.122:0/1689791697 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c005d010 con 0x55eba0ac1600 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9b2ffd640 1 -- 172.21.15.122:0/1689791697 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c00615d0 con 0x55eba0ac1600 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9b2ffd640 1 -- 172.21.15.122:0/1689791697 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0055530 con 0x55eba0ac1600 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1689791697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 msgr2=0x55eba0ac1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 0x55eba0ac1a80 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0002230 tx=0x7fb9c00588e0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1689791697 shutdown_connections 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ac1600 0x55eba0ac1a80 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0a958c0 0x55eba0a97360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1689791697 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0a91870 0x55eba0a91c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1689791697 >> 172.21.15.122:0/1689791697 conn(0x55eba0a938f0 msgr2=0x55eba0a951b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1689791697 shutdown_connections 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.965+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1689791697 wait complete. 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.966+0000 7fb9d1f9c740 1 -- start start 2024-03-06T07:27:32.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.966+0000 7fb9d1f9c740 1 Processor -- start 2024-03-06T07:27:32.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.967+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ac27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0ac2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eba0a95d50 con 0x55eba0ade6b0 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eba09b7ea0 con 0x55eba0adea90 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eba0a96030 con 0x55eba0ac1600 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 10 client.0 _ll_register_callbacks cb 0x55eba09b7870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ac27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ac27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37738/0 (socket says 172.21.15.122:37738) 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 -- 172.21.15.122:0/258868480 learned_addr learned my addr 172.21.15.122:0/258868480 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0ac2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 -- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 msgr2=0x55eba0ad9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 -- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 msgr2=0x55eba0ac2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0ac2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 -- 172.21.15.122:0/258868480 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eba09b8270 con 0x55eba0ade6b0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0ac2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ac27a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000c00 tx=0x7fb9b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96262c34efe16e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9a7fff640 1 -- 172.21.15.122:0/258868480 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8039040 con 0x55eba0ade6b0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9a7fff640 1 -- 172.21.15.122:0/258868480 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9b802c8b0 con 0x55eba0ade6b0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9c00615d0 con 0x55eba0ade6b0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9a7fff640 1 -- 172.21.15.122:0/258868480 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9b802cad0 con 0x55eba0ade6b0 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 10 client.8316 fetch_fsmap learned FSMap version 67 2024-03-06T07:27:32.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.968+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55eba07102d0 con 0x55eba0ade6b0 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9a7fff640 1 -- 172.21.15.122:0/258868480 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c00615d0 con 0x55eba0ade6b0 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9a7fff640 1 -- 172.21.15.122:0/258868480 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb9b802cc70 con 0x55eba0ade6b0 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9d1f9c740 10 client.8316 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9d1f9c740 10 client.8316 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9d1f9c740 20 client.8316 populate_metadata read hostname 'smithi122' 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0adb170 0x55eba0afb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9d1f9c740 4 client.8316 mds_command: new command op to 8283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 --> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eba0a08bf0 con 0x55eba0adb170 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0adb170 0x55eba0afb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9a7fff640 10 client.8316 ms_handle_connect on v2:172.21.15.122:6836/2123619089 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.969+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0adb170 0x55eba0afb0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf3607128b6c451 server_cookie=9ebe739358345183 in_seq=0 out_seq=0 2024-03-06T07:27:32.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9a7fff640 1 -- 172.21.15.122:0/258868480 <== mds.0 v2:172.21.15.122:6836/2123619089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eba0a08bf0 con 0x55eba0adb170 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9a7fff640 10 client.8316 handle_command_reply: tid=0 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 client.8316 shutdown 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 msgr2=0x55eba0ac27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ac27a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000c00 tx=0x7fb9b8004730 comp rx=0 tx=0).stop 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 shutdown_connections 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0adb170 0x55eba0afb0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0ac2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ac27a0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9a7fff640 0 client.8316 ms_handle_reset on v2:172.21.15.122:6836/2123619089 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/258868480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 >> 172.21.15.122:0/258868480 conn(0x55eba0a91870 msgr2=0x55eba0ac0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 shutdown_connections 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/258868480 wait complete. 2024-03-06T07:27:32.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.970+0000 7fb9d1f9c740 20 client.8316 trim_cache size 0 max 16384 2024-03-06T07:27:32.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 Processor -- start 2024-03-06T07:27:32.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 -- start start 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0ac1600 0x55eba0ac0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ade6b0 0x55eba0ac2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 0x55eba0b42bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8200a50 con 0x55eba0adea90 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c81ffca0 con 0x55eba0ac1600 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80c8120 con 0x55eba0ade6b0 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.989+0000 7fb9cd87e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 0x55eba0b42bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 0x55eba0b42bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37754/0 (socket says 172.21.15.122:37754) 2024-03-06T07:27:32.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1557629751 learned_addr learned my addr 172.21.15.122:0/1557629751 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ade6b0 0x55eba0ac2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0ac1600 0x55eba0ac0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ade6b0 msgr2=0x55eba0ac2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ade6b0 0x55eba0ac2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0ac1600 msgr2=0x55eba0ac0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0ac1600 0x55eba0ac0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 -- 172.21.15.122:0/1557629751 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eba07da6b0 con 0x55eba0adea90 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 0x55eba0b42bd0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0058a20 tx=0x7fb9c0058950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4168ab7779cdd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9b27fc640 1 -- 172.21.15.122:0/1557629751 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00615d0 con 0x55eba0adea90 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9b27fc640 1 -- 172.21.15.122:0/1557629751 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c005d010 con 0x55eba0adea90 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.990+0000 7fb9b27fc640 1 -- 172.21.15.122:0/1557629751 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0055530 con 0x55eba0adea90 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1557629751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 msgr2=0x55eba0b42bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 0x55eba0b42bd0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0058a20 tx=0x7fb9c0058950 comp rx=0 tx=0).stop 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1557629751 shutdown_connections 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0adea90 0x55eba0b42bd0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ade6b0 0x55eba0ac2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/1557629751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0ac1600 0x55eba0ac0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1557629751 >> 172.21.15.122:0/1557629751 conn(0x55eba0a91870 msgr2=0x55eba0ad9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1557629751 shutdown_connections 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/1557629751 wait complete. 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 -- start start 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.991+0000 7fb9d1f9c740 1 Processor -- start 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ad8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0a95fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eba0adf450 con 0x55eba0ade6b0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80c8720 con 0x55eba0adea90 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eba0adfcf0 con 0x55eba0ac1600 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9d1f9c740 10 client.0 _ll_register_callbacks cb 0x7fb9c00095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9cc87c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ad8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9cc87c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ad8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37762/0 (socket says 172.21.15.122:37762) 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9cc87c640 1 -- 172.21.15.122:0/2810779421 learned_addr learned my addr 172.21.15.122:0/2810779421 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.992+0000 7fb9cd87e640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0a95fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9cc87c640 1 -- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 msgr2=0x55eba0ad7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9cc87c640 1 -- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 msgr2=0x55eba0a95fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0a95fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9cc87c640 1 -- 172.21.15.122:0/2810779421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eba0ac29d0 con 0x55eba0ade6b0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9cc87c640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ad8500 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000c00 tx=0x7fb9b8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe14604d5f63e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9a7fff640 1 -- 172.21.15.122:0/2810779421 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8039050 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9a7fff640 1 -- 172.21.15.122:0/2810779421 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9b8005b90 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9c005d010 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9a7fff640 1 -- 172.21.15.122:0/2810779421 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9b805e050 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 10 client.8322 fetch_fsmap learned FSMap version 67 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55eba0adbd30 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9a7fff640 1 -- 172.21.15.122:0/2810779421 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c005d010 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9a7fff640 1 -- 172.21.15.122:0/2810779421 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb9b8061020 con 0x55eba0ade6b0 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 10 client.8322 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 10 client.8322 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 20 client.8322 populate_metadata read hostname 'smithi122' 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0ab4fa0 0x55eba0ab5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 4 client.8322 mds_command: new command op to 8283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:33.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.993+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 --> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eba0a98cd0 con 0x55eba0ab4fa0 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0ab4fa0 0x55eba0ab5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9a7fff640 10 client.8322 ms_handle_connect on v2:172.21.15.122:6836/2123619089 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9cd07d640 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0ab4fa0 0x55eba0ab5380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a3ef50d2d99cf83 server_cookie=d1d1798dd7c434c2 in_seq=0 out_seq=0 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9a7fff640 1 -- 172.21.15.122:0/2810779421 <== mds.0 v2:172.21.15.122:6836/2123619089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55eba0a98cd0 con 0x55eba0ab4fa0 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9a7fff640 10 client.8322 handle_command_reply: tid=0 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 client.8322 shutdown 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 msgr2=0x55eba0ad8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ad8500 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000c00 tx=0x7fb9b8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 shutdown_connections 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:6836/2123619089,v1:172.21.15.122:6837/2123619089] conn(0x55eba0ab4fa0 0x55eba0ab5380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eba0adea90 0x55eba0a95fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9a7fff640 0 client.8322 ms_handle_reset on v2:172.21.15.122:6836/2123619089 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eba0ade6b0 0x55eba0ad8500 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 --2- 172.21.15.122:0/2810779421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55eba0ac1600 0x55eba0ad7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 >> 172.21.15.122:0/2810779421 conn(0x55eba0a91870 msgr2=0x55eba0ac0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 shutdown_connections 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.994+0000 7fb9d1f9c740 1 -- 172.21.15.122:0/2810779421 wait complete. 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9d1f9c740 20 client.8322 trim_cache size 0 max 16384 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 8283, 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:27:33.004 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 4.0261003860000004, 2024-03-06T07:27:33.005 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 67, 2024-03-06T07:27:33.005 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 112, 2024-03-06T07:27:33.005 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 112, 2024-03-06T07:27:33.005 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 4.3461952080000001 2024-03-06T07:27:33.005 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:27:33.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb9ac06b4d0 msgr2=0x7fb9ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb9ac06b4d0 0x7fb9ac06d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80d3740 tx=0x7fb9c40078f0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 msgr2=0x7fb9c80c9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 0x7fb9c80c9540 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80a1c60 tx=0x7fb9c0010590 comp rx=0 tx=0).stop 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 shutdown_connections 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb9ac06b4d0 0x7fb9ac06d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9c80c90c0 0x7fb9c80c9540 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9c815eb40 0x7fb9c815efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 --2- 172.21.15.122:0/76500524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4003f30 0x7fb9c8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 >> 172.21.15.122:0/76500524 conn(0x7fb9c80d3ce0 msgr2=0x7fb9c80b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 shutdown_connections 2024-03-06T07:27:33.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:32.995+0000 7fb9cf2e1640 1 -- 172.21.15.122:0/76500524 wait complete. 2024-03-06T07:27:33.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:33.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.107+0000 7fc001c64640 1 Processor -- start 2024-03-06T07:27:33.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fc001c64640 1 -- start start 2024-03-06T07:27:33.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fc001c64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 0x7fbffc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fc001c64640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbffc0cf830 0x7fbffc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fc001c64640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc0b3430 con 0x7fbffc0d27f0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fc001c64640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc0d2c50 con 0x7fbffc0cf830 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fc001c64640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0b31e0 con 0x7fbffc15dfe0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fbffbfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbffc0d27f0 0x7fbffc0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43028/0 (socket says 172.21.15.122:43028) 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.108+0000 7fbffbfff640 1 -- 172.21.15.122:0/399015154 learned_addr learned my addr 172.21.15.122:0/399015154 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffb7fe640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 0x7fbffc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffaffd640 1 --2- 172.21.15.122:0/399015154 >> v2:172.21.15.169:3300/0 conn(0x7fbffc0cf830 0x7fbffc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902900293 0 0) 0x7fbffc0b3430 con 0x7fbffc0d27f0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe4003520 con 0x7fbffc0d27f0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3006309288 0 0) 0x7fbfe4003520 con 0x7fbffc0d27f0 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 msgr2=0x7fbffc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 0x7fbffc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 >> v2:172.21.15.169:3300/0 conn(0x7fbffc0cf830 msgr2=0x7fbffc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 --2- 172.21.15.122:0/399015154 >> v2:172.21.15.169:3300/0 conn(0x7fbffc0cf830 0x7fbffc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc0b3710 con 0x7fbffc0d27f0 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fbff40019f0 con 0x7fbffc0d27f0 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 >> v1:172.21.15.122:6789/0 conn(0x7fbffc0d27f0 legacy=0x7fbffc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 0x7fbfe40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfe4003f30 0x7fbfe4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfe4004d40 0x7fbfe40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0b31e0 con 0x7fbfe4003f30 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe400b9c0 con 0x7fbfe4004d40 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe400bb40 con 0x7fbffc15dfe0 2024-03-06T07:27:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffaffd640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfe4003f30 0x7fbfe4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffb7fe640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 0x7fbfe40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.109+0000 7fbffbfff640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfe4004d40 0x7fbfe40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.110+0000 7fbffaffd640 1 -- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 msgr2=0x7fbfe40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.110+0000 7fbffaffd640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbffc15dfe0 0x7fbfe40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.110+0000 7fbffaffd640 1 -- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfe4004d40 msgr2=0x7fbfe40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.110+0000 7fbffaffd640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfe4004d40 0x7fbfe40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.110+0000 7fbffaffd640 1 -- 172.21.15.122:0/399015154 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4003520 con 0x7fbfe4003f30 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fbffaffd640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfe4003f30 0x7fbfe4004800 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009a90 tx=0x7fbfe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856ea720b184454e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0039070 con 0x7fbfe4003f30 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff002c6b0 con 0x7fbfe4003f30 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fbffa7fc640 1 -- 172.21.15.122:0/399015154 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0009690 con 0x7fbfe4003f30 2024-03-06T07:27:33.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fc001c64640 1 -- 172.21.15.122:0/399015154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfe4003f30 msgr2=0x7fbfe4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fc001c64640 1 --2- 172.21.15.122:0/399015154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfe4003f30 0x7fbfe4004800 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009a90 tx=0x7fbfe4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:33.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fbffbfff640 1 -- 172.21.15.122:0/399015154 reap_dead start 2024-03-06T07:27:33.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fc001c64640 1 -- 172.21.15.122:0/399015154 shutdown_connections 2024-03-06T07:27:33.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fc001c64640 1 -- 172.21.15.122:0/399015154 >> 172.21.15.122:0/399015154 conn(0x7fbffc0d3cd0 msgr2=0x7fbffc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fc001c64640 1 -- 172.21.15.122:0/399015154 shutdown_connections 2024-03-06T07:27:33.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.111+0000 7fc001c64640 1 -- 172.21.15.122:0/399015154 wait complete. 2024-03-06T07:27:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.113+0000 7fc001c64640 1 Processor -- start 2024-03-06T07:27:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.114+0000 7fc001c64640 1 -- start start 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfe4003f30 0x7fbffc159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 0x7fbffc0b9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbffc0ccd10 0x7fbffc0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0d1c70 con 0x7fbffc0b9380 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0b3d60 con 0x7fbffc0ccd10 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0d1df0 con 0x7fbfe4003f30 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 0x7fbffc0b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 0x7fbffc0b9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37768/0 (socket says 172.21.15.122:37768) 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 -- 172.21.15.122:0/3050277813 learned_addr learned my addr 172.21.15.122:0/3050277813 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffbfff640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbffc0ccd10 0x7fbffc0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffb7fe640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfe4003f30 0x7fbffc159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 -- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfe4003f30 msgr2=0x7fbffc159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfe4003f30 0x7fbffc159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 -- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbffc0ccd10 msgr2=0x7fbffc0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbffc0ccd10 0x7fbffc0cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 -- 172.21.15.122:0/3050277813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc0b7650 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffb7fe640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfe4003f30 0x7fbffc159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffbfff640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbffc0ccd10 0x7fbffc0cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbffaffd640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 0x7fbffc0b9800 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009a90 tx=0x7fbff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0039050 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0045070 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.115+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc0b7930 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.116+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0003a50 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.116+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc0b7ef0 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.117+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc0cf5f0 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.117+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbff0003bf0 con 0x7fbffc0b9380 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.117+0000 7fbfe3fff640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfe806b4d0 0x7fbfe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.117+0000 7fbffb7fe640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfe806b4d0 0x7fbfe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.118+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbff00abeb0 con 0x7fbffc0b9380 2024-03-06T07:27:33.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.118+0000 7fbffb7fe640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfe806b4d0 0x7fbfe806d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009ae0 tx=0x7fbfec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.122+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbffc0cf5f0 con 0x7fbffc0b9380 2024-03-06T07:27:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.446+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbffc04f330 con 0x7fbffc0b9380 2024-03-06T07:27:33.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.446+0000 7fbfe3fff640 1 -- 172.21.15.122:0/3050277813 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbff004e020 con 0x7fbffc0b9380 2024-03-06T07:27:33.448 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:33.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfe806b4d0 msgr2=0x7fbfe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfe806b4d0 0x7fbfe806d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009ae0 tx=0x7fbfec008040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 msgr2=0x7fbffc0b9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 0x7fbffc0b9800 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009a90 tx=0x7fbff0004730 comp rx=0 tx=0).stop 2024-03-06T07:27:33.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 shutdown_connections 2024-03-06T07:27:33.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfe806b4d0 0x7fbfe806d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbffc0ccd10 0x7fbffc0cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbffc0b9380 0x7fbffc0b9800 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 --2- 172.21.15.122:0/3050277813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfe4003f30 0x7fbffc159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 >> 172.21.15.122:0/3050277813 conn(0x7fbffc0d3cd0 msgr2=0x7fbffc0b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 shutdown_connections 2024-03-06T07:27:33.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:33.448+0000 7fc001c64640 1 -- 172.21.15.122:0/3050277813 wait complete. 2024-03-06T07:27:33.460 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:33.461 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:33.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.461 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:27:33.476 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:33.476 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42387sec preferred_lft 42387sec 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:33.478 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:33.479 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:27:33.479 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.479 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:33.542 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:27:33.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.543 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.599 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:27:33.599 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:27:33.599 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:33.599 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:33.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:33.600 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:27:33.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.694 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:27:33.708 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:33.708 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:33.708 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:33.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:27:33.776 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:33.776 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:33.776 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:33.831 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:33.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:33.831 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:33.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:27:33.951 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:33.976 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:27:33.974+0000 7fc172148640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.994 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:27:34.046 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:34.047 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:34.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:27:34.114 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:34.170 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:27:34.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:27:34.170 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:27:34.170 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:27:34.170 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:27:34.170 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.173 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:27:34.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:27:34.174+0000 7f54b9646480 -1 init, newargv = 0x55bdec687a80 newargc=15 2024-03-06T07:27:34.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[74706]: starting ceph client 2024-03-06T07:27:34.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[74706]: starting fuse 2024-03-06T07:27:34.192 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:27:34.192 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:34.192 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:27:34.194 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:34.194 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:27:34.194 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:34.194 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:27:34.314 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.74706.asok 2024-03-06T07:27:34.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:34.324 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74706.asok 2024-03-06T07:27:34.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74706.asok status 2024-03-06T07:27:34.461 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "74706", 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 8334, 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 8334 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 3864912020 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 3864912020 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.8334 192.168.0.1:0/3864912020", 2024-03-06T07:27:34.463 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/3864912020", 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 67, 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 113, 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:27:34.464 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:27:34.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.470 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:27:34.470 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.485 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:27:34.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.556 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:27:34.615 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:34.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.755+0000 7fc742955640 1 Processor -- start 2024-03-06T07:27:34.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.755+0000 7fc742955640 1 -- start start 2024-03-06T07:27:34.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc742955640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc73c0d22b0 0x7fc73c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc742955640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c0b3430 con 0x7fc73c15dfe0 2024-03-06T07:27:34.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc742955640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c0b3710 con 0x7fc73c0b41f0 2024-03-06T07:27:34.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc742955640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c0d2c50 con 0x7fc73c0d22b0 2024-03-06T07:27:34.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc73c15dfe0 0x7fc73c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44558/0 (socket says 172.21.15.122:44558) 2024-03-06T07:27:34.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73bfff640 1 -- 172.21.15.122:0/10168410 learned_addr learned my addr 172.21.15.122:0/10168410 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918351324 0 0) 0x7fc73c0b3430 con 0x7fc73c15dfe0 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73b7fe640 1 --2- 172.21.15.122:0/10168410 >> v2:172.21.15.169:3300/0 conn(0x7fc73c0d22b0 0x7fc73c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc724003520 con 0x7fc73c15dfe0 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2994004993 0 0) 0x7fc724003520 con 0x7fc73c15dfe0 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 >> v2:172.21.15.169:3300/0 conn(0x7fc73c0d22b0 msgr2=0x7fc73c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 --2- 172.21.15.122:0/10168410 >> v2:172.21.15.169:3300/0 conn(0x7fc73c0d22b0 0x7fc73c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 >> v1:172.21.15.169:6789/0 conn(0x7fc73c0b41f0 legacy=0x7fc73c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c0b39f0 con 0x7fc73c15dfe0 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fc72c001670 con 0x7fc73c15dfe0 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.757+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 >> v1:172.21.15.122:6789/0 conn(0x7fc73c15dfe0 legacy=0x7fc73c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73affd640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7240039b0 0x7fc724003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73affd640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc724004800 0x7fc724004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73affd640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc72400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72400c030 con 0x7fc7240051c0 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72400c1e0 con 0x7fc7240039b0 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72400c3e0 con 0x7fc724004800 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73bfff640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7240039b0 0x7fc724003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc740ef2640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc72400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73b7fe640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc724004800 0x7fc724004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc740ef2640 1 -- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc724004800 msgr2=0x7fc724004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc740ef2640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc724004800 0x7fc724004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc740ef2640 1 -- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7240039b0 msgr2=0x7fc724003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc740ef2640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7240039b0 0x7fc724003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc740ef2640 1 -- 172.21.15.122:0/10168410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724003520 con 0x7fc7240051c0 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.758+0000 7fc73bfff640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7240039b0 0x7fc724003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:34.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc740ef2640 1 -- 172.21.15.122:0/10168410 reap_dead start 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc740ef2640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc72400b9c0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fc734001e50 tx=0x7fc724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bcf30afb003a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c007d10 con 0x7fc7240051c0 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc734003bb0 con 0x7fc7240051c0 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc73affd640 1 -- 172.21.15.122:0/10168410 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734001a70 con 0x7fc7240051c0 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc742955640 1 -- 172.21.15.122:0/10168410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 msgr2=0x7fc72400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc742955640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc72400b9c0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fc734001e50 tx=0x7fc724005830 comp rx=0 tx=0).stop 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc742955640 1 -- 172.21.15.122:0/10168410 shutdown_connections 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc742955640 1 --2- 172.21.15.122:0/10168410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc72400b9c0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.759+0000 7fc742955640 1 -- 172.21.15.122:0/10168410 >> 172.21.15.122:0/10168410 conn(0x7fc73c0d3cd0 msgr2=0x7fc73c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.760+0000 7fc742955640 1 -- 172.21.15.122:0/10168410 shutdown_connections 2024-03-06T07:27:34.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.760+0000 7fc742955640 1 -- 172.21.15.122:0/10168410 wait complete. 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 Processor -- start 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 -- start start 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc73c0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc73c15a580 0x7fc73c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc73c15b4b0 0x7fc73c15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0b3d60 con 0x7fc7240051c0 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.761+0000 7fc742955640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0b31e0 con 0x7fc73c15a580 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc742955640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c15e3c0 con 0x7fc73c15b4b0 2024-03-06T07:27:34.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc73c0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc73c0cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49460/0 (socket says 172.21.15.122:49460) 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 -- 172.21.15.122:0/1090859102 learned_addr learned my addr 172.21.15.122:0/1090859102 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73b7fe640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc73c15a580 0x7fc73c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc740ef2640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc73c15b4b0 0x7fc73c15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 -- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc73c15b4b0 msgr2=0x7fc73c15c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc73c15b4b0 0x7fc73c15c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 -- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc73c15a580 msgr2=0x7fc73c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc73c15a580 0x7fc73c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 -- 172.21.15.122:0/1090859102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c0cc520 con 0x7fc7240051c0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc740ef2640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc73c15b4b0 0x7fc73c15c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc73bfff640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc73c0cd580 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0072c0 tx=0x7fc72c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00f960 con 0x7fc7240051c0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c018070 con 0x7fc7240051c0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c0cc7c0 con 0x7fc7240051c0 2024-03-06T07:27:34.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00fd60 con 0x7fc7240051c0 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.762+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c0cd060 con 0x7fc7240051c0 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.763+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c0cc990 con 0x7fc7240051c0 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.764+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc72c01d460 con 0x7fc7240051c0 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.764+0000 7fc738ff9640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72806b4d0 0x7fc72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.764+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc72c08a370 con 0x7fc7240051c0 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.764+0000 7fc73b7fe640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72806b4d0 0x7fc72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.765+0000 7fc73b7fe640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72806b4d0 0x7fc72806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc73c15a450 tx=0x7fc730008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.769+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc72c0591f0 con 0x7fc7240051c0 2024-03-06T07:27:34.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.983+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc73c04f330 con 0x7fc7240051c0 2024-03-06T07:27:34.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.984+0000 7fc738ff9640 1 -- 172.21.15.122:0/1090859102 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fc72c058af0 con 0x7fc7240051c0 2024-03-06T07:27:34.986 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:34.986 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:27:34.986 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:27:34.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72806b4d0 msgr2=0x7fc72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72806b4d0 0x7fc72806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc73c15a450 tx=0x7fc730008040 comp rx=0 tx=0).stop 2024-03-06T07:27:34.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 msgr2=0x7fc73c0cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc73c0cd580 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0072c0 tx=0x7fc72c007780 comp rx=0 tx=0).stop 2024-03-06T07:27:34.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 shutdown_connections 2024-03-06T07:27:34.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc72806b4d0 0x7fc72806d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc73c15b4b0 0x7fc73c15c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc73c15a580 0x7fc73c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 --2- 172.21.15.122:0/1090859102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7240051c0 0x7fc73c0cd580 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 >> 172.21.15.122:0/1090859102 conn(0x7fc73c0d3cd0 msgr2=0x7fc73c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 shutdown_connections 2024-03-06T07:27:34.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:34.986+0000 7fc742955640 1 -- 172.21.15.122:0/1090859102 wait complete. 2024-03-06T07:27:35.001 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:27:35.117 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:27:35.117 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:27:35.118 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:35.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.216+0000 7efce359e640 1 Processor -- start 2024-03-06T07:27:35.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.216+0000 7efce359e640 1 -- start start 2024-03-06T07:27:35.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce359e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efce40d20c0 0x7efce40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce359e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce359e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40d29e0 con 0x7efce40d01a0 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce359e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce40b2fa0 con 0x7efce40d20c0 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce359e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce40b3260 con 0x7efce40cf190 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efce40d20c0 0x7efce40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efce40d20c0 0x7efce40d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49476/0 (socket says 172.21.15.122:49476) 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 -- 172.21.15.122:0/340545786 learned_addr learned my addr 172.21.15.122:0/340545786 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce1d9b640 1 --2- 172.21.15.122:0/340545786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 -- 172.21.15.122:0/340545786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 msgr2=0x7efce40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 --2- 172.21.15.122:0/340545786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 -- 172.21.15.122:0/340545786 >> v1:172.21.15.169:6789/0 conn(0x7efce40d01a0 legacy=0x7efce40d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.217+0000 7efce259c640 1 -- 172.21.15.122:0/340545786 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40b3540 con 0x7efce40d20c0 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.218+0000 7efce259c640 1 --2- 172.21.15.122:0/340545786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d20c0 0x7efce40d24a0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009870 tx=0x7efcd802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cb1bac99dfa072 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.218+0000 7efce159a640 1 -- 172.21.15.122:0/340545786 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8034070 con 0x7efce40d20c0 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.218+0000 7efce159a640 1 -- 172.21.15.122:0/340545786 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd8030840 con 0x7efce40d20c0 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.218+0000 7efce159a640 1 -- 172.21.15.122:0/340545786 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8030c10 con 0x7efce40d20c0 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 -- 172.21.15.122:0/340545786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d20c0 msgr2=0x7efce40d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 --2- 172.21.15.122:0/340545786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d20c0 0x7efce40d24a0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009870 tx=0x7efcd802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 -- 172.21.15.122:0/340545786 shutdown_connections 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 --2- 172.21.15.122:0/340545786 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 --2- 172.21.15.122:0/340545786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d20c0 0x7efce40d24a0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 -- 172.21.15.122:0/340545786 >> 172.21.15.122:0/340545786 conn(0x7efce409ce20 msgr2=0x7efce40d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 -- 172.21.15.122:0/340545786 shutdown_connections 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.219+0000 7efce359e640 1 -- 172.21.15.122:0/340545786 wait complete. 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.221+0000 7efce359e640 1 Processor -- start 2024-03-06T07:27:35.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.221+0000 7efce359e640 1 -- start start 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.222+0000 7efce359e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce41582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.222+0000 7efce359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 0x7efce40c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.222+0000 7efce359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efce40d20c0 0x7efce40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.222+0000 7efce359e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce40d1af0 con 0x7efce40d01a0 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.222+0000 7efce359e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce40b3b00 con 0x7efce40d20c0 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.222+0000 7efce359e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce4161630 con 0x7efce40cf190 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 0x7efce40c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 0x7efce40c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49486/0 (socket says 172.21.15.122:49486) 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 -- 172.21.15.122:0/1758915614 learned_addr learned my addr 172.21.15.122:0/1758915614 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce259c640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce41582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 -- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 msgr2=0x7efce41582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce41582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 -- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efce40d20c0 msgr2=0x7efce40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce2d9d640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efce40d20c0 0x7efce40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efce40d20c0 0x7efce40b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 -- 172.21.15.122:0/1758915614 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40cd230 con 0x7efce40d01a0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce2d9d640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efce40d20c0 0x7efce40b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce1d9b640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 0x7efce40c75f0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7efcd400c7f0 tx=0x7efcd400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4015070 con 0x7efce40d01a0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd4004220 con 0x7efce40d01a0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce40b8f90 con 0x7efce40d01a0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4007620 con 0x7efce40d01a0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.223+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce40b7240 con 0x7efce40d01a0 2024-03-06T07:27:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.224+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce40b7440 con 0x7efce40d01a0 2024-03-06T07:27:35.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.225+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcd40079d0 con 0x7efce40d01a0 2024-03-06T07:27:35.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.225+0000 7efcd2ffd640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcc406b500 0x7efcc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.225+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7efcd408a1f0 con 0x7efce40d01a0 2024-03-06T07:27:35.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.225+0000 7efce259c640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcc406b500 0x7efcc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.225+0000 7efce259c640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcc406b500 0x7efcc406d9c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efcd8036000 tx=0x7efcd803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.229+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efce40b7440 con 0x7efce40d01a0 2024-03-06T07:27:35.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.501+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efce4068f80 con 0x7efce40d01a0 2024-03-06T07:27:35.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.501+0000 7efcd2ffd640 1 -- 172.21.15.122:0/1758915614 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7efcd4059070 con 0x7efce40d01a0 2024-03-06T07:27:35.503 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:27:35.504 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:27:35.504 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcc406b500 msgr2=0x7efcc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcc406b500 0x7efcc406d9c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efcd8036000 tx=0x7efcd803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 msgr2=0x7efce40c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 0x7efce40c75f0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7efcd400c7f0 tx=0x7efcd400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 shutdown_connections 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efcc406b500 0x7efcc406d9c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efce40d20c0 0x7efce40b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efce40d01a0 0x7efce40c75f0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 --2- 172.21.15.122:0/1758915614 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efce40cf190 0x7efce41582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 >> 172.21.15.122:0/1758915614 conn(0x7efce409ce20 msgr2=0x7efce40b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.503+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 shutdown_connections 2024-03-06T07:27:35.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:27:35.504+0000 7efce359e640 1 -- 172.21.15.122:0/1758915614 wait complete. 2024-03-06T07:27:35.517 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:27:35.633 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:27:35.633 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:27:54.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:54.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:05.634 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:05.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.673+0000 7ffb40cb2f40 1 Processor -- start 2024-03-06T07:28:05.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.673+0000 7ffb40cb2f40 1 -- start start 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb40cb2f40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad18390 0x55f9caccdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb40cb2f40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cad9f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb40cb2f40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 0x55f9cad9f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb40cb2f40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9cada5db0 con 0x55f9cad86fa0 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb40cb2f40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9cac75b70 con 0x55f9cad18770 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb40cb2f40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f9cada5f30 con 0x55f9cad18390 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb3efb4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 0x55f9cad9f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb3efb4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 0x55f9cad9f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59270/0 (socket says 172.21.15.122:59270) 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb3efb4640 1 -- 172.21.15.122:0/1720980455 learned_addr learned my addr 172.21.15.122:0/1720980455 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad18390 0x55f9caccdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cad9f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.674+0000 7ffb3efb4640 1 -- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad18390 msgr2=0x55f9caccdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad18390 0x55f9caccdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3efb4640 1 -- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 msgr2=0x55f9cad9f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cad9f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3efb4640 1 -- 172.21.15.122:0/1720980455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9cae31cb0 con 0x55f9cad86fa0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 0x55f9cad9f9b0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c4a0 tx=0x7ffb3800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391ede3d4ea6742b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3cfb0640 1 -- 172.21.15.122:0/1720980455 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380116b0 con 0x55f9cad86fa0 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3cfb0640 1 -- 172.21.15.122:0/1720980455 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb38017070 con 0x55f9cad86fa0 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb3cfb0640 1 -- 172.21.15.122:0/1720980455 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb38011ab0 con 0x55f9cad86fa0 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/1720980455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 msgr2=0x55f9cad9f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 0x55f9cad9f9b0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c4a0 tx=0x7ffb3800c960 comp rx=0 tx=0).stop 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/1720980455 shutdown_connections 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad86fa0 0x55f9cad9f9b0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cad9f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/1720980455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad18390 0x55f9caccdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/1720980455 >> 172.21.15.122:0/1720980455 conn(0x55f9cacf9780 msgr2=0x55f9cae31270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/1720980455 shutdown_connections 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.675+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/1720980455 wait complete. 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.676+0000 7ffb40cb2f40 1 Processor -- start 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.676+0000 7ffb40cb2f40 1 -- start start 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 0x55f9caccfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cacd0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad86fa0 0x55f9cad9cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9cad9fef0 con 0x55f9cad18390 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9cad9dd20 con 0x55f9cad18770 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f9cacccfc0 con 0x55f9cad86fa0 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 0x55f9caccfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 0x55f9caccfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59276/0 (socket says 172.21.15.122:59276) 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 learned_addr learned my addr 172.21.15.122:0/975139223 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cacd0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad86fa0 0x55f9cad9cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad86fa0 msgr2=0x55f9cad9cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad86fa0 0x55f9cad9cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 msgr2=0x55f9cacd0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f9cad18770 0x55f9cacd0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9cada3770 con 0x55f9cad18390 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f9cad86fa0 0x55f9cad9cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 0x55f9caccfe30 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ffb30046e10 tx=0x7ffb30002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb2f7fe640 1 -- 172.21.15.122:0/975139223 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30069070 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb2f7fe640 1 -- 172.21.15.122:0/975139223 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30058400 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.677+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f9cada3a50 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.678+0000 7ffb2f7fe640 1 -- 172.21.15.122:0/975139223 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb300587f0 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.678+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9cada2910 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.679+0000 7ffb2f7fe640 1 -- 172.21.15.122:0/975139223 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb3006c070 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.679+0000 7ffb2f7fe640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb20069ee0 0x7ffb2006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.679+0000 7ffb2f7fe640 1 -- 172.21.15.122:0/975139223 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ffb300ccd90 con 0x55f9cad18390 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.679+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb20069ee0 0x7ffb2006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.679+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x55f9cad17260 0x55f9cad17660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.680+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x55f9cad17260 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.680+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x55f9cad17260 0x55f9cad17660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.680+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb20069ee0 0x7ffb2006c3a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffb34001ed0 tx=0x7ffb34001930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.680+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x55f9cad17260 0x55f9cad17660 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7ffb38012990 0x7ffb3801d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb38012990 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7ffb38012990 0x7ffb3801d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7ffb38012990 0x7ffb3801d410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7ffb3006e060 0x7ffb3009a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.681+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3006e060 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.682+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7ffb3006e060 0x7ffb3009a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.682+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7ffb3006e060 0x7ffb3009a480 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.682+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.682+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7ffb340113f0 0x7ffb34011830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.682+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.682+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7ffb340113f0 0x7ffb34011830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.683+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7ffb340113f0 0x7ffb34011830 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.683+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.683+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x55f9cad17260 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.683+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.683+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3006e060 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 2 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x55f9cad17260 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 3 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.684+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3006e060 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.685+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.685+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3006e060 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.685+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.685+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ffb3401d8d0 0x7ffb3401a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.685+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3401d8d0 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.685+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ffb3401d8d0 0x7ffb3401a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.686+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ffb3401d8d0 0x7ffb3401a330 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.686+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.5 v2:172.21.15.169:6816/887889200 1 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3401d8d0 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.686+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x55f9cad17260 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.686+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.686+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7ffb3802d910 0x7ffb3801b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.686+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3802d910 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.687+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7ffb3802d910 0x7ffb3801b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.687+0000 7ffb3dfb2640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7ffb3802d910 0x7ffb3801b3f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.687+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.687+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb38012990 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.688+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.688+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.688+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 4 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.688+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3802d910 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.688+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.688+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3401d8d0 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.689+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.5 v2:172.21.15.169:6816/887889200 2 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3401d8d0 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.689+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3006e060 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.689+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.689+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x55f9cad17260 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.689+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.689+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x55f9cad17260 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.690+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.690+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3006e060 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.690+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.690+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.690+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 5 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.690+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb38012990 2024-03-06T07:28:05.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.691+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.691+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ffb3009b820 0x7ffb30058990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.691+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3009b820 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.691+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ffb3009b820 0x7ffb30058990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.691+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ffb3009b820 0x7ffb30058990 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.6 v2:172.21.15.169:6824/1645803606 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb3009b820 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 6 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3401d8d0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.5 v2:172.21.15.169:6816/887889200 3 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3401d8d0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb340113f0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 7 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ffb3803b960 0x7ffb3803a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.692+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3803b960 2024-03-06T07:28:05.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.693+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ffb3803b960 0x7ffb3803a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.693+0000 7ffb3e7b3640 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ffb3803b960 0x7ffb3803a3e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.693+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.2 v2:172.21.15.122:6816/1235809965 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3803b960 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.693+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f9cad17ba0 con 0x7ffb3009b820 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.693+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.6 v2:172.21.15.169:6824/1645803606 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb3009b820 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.694+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb38012990 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.694+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.694+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.694+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.694+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb340113f0 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb340113f0 2024-03-06T07:28:05.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3006e060 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb38012990 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3006e060 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.695+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.696+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.696+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.696+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3401d8d0 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.696+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.5 v2:172.21.15.169:6816/887889200 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3401d8d0 2024-03-06T07:28:05.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.696+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3401d8d0 2024-03-06T07:28:05.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.697+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.5 v2:172.21.15.169:6816/887889200 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3401d8d0 2024-03-06T07:28:05.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.697+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3009b820 2024-03-06T07:28:05.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.697+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.6 v2:172.21.15.169:6824/1645803606 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb3009b820 2024-03-06T07:28:05.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.697+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb38012990 2024-03-06T07:28:05.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.697+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.697+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3401d8d0 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.698+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.5 v2:172.21.15.169:6816/887889200 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3401d8d0 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.698+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.698+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.698+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x55f9cad17260 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x55f9cad17260 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x55f9cad17260 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x55f9cad17260 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.699+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.700+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.700+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.700+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.700+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.700+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3006e060 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.701+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.3 v2:172.21.15.122:6824/47863215 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3006e060 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.701+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x55f9cad17260 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.701+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.7 v2:172.21.15.169:6800/3079115781 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x55f9cad17260 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.701+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb340113f0 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.701+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.701+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.702+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.702+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.702+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.702+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3803b960 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.702+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.2 v2:172.21.15.122:6816/1235809965 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb3803b960 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.702+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb340113f0 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.703+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 11 ==== osd_op_reply(57 [pgnls start_epoch 114 out=81b] v108'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.703+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.703+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.703+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3009b820 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.704+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.6 v2:172.21.15.169:6824/1645803606 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb3009b820 2024-03-06T07:28:05.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.704+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb340113f0 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.704+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 12 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.704+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb38012990 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.704+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 <== osd.1 v2:172.21.15.122:6808/20360309 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb3006b030 con 0x7ffb38012990 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.704+0000 7ffb3e7b3640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3802d910 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.705+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 <== osd.4 v2:172.21.15.169:6804/1079551346 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=81b] v108'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ffb3400ea00 con 0x7ffb3802d910 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.705+0000 7ffb3dfb2640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb3009b820 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.705+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.6 v2:172.21.15.169:6824/1645803606 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb3009b820 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.705+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffb38047040 con 0x7ffb340113f0 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 <== osd.0 v2:172.21.15.122:6800/2293549780 13 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffb38011730 con 0x7ffb340113f0 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stdout:10000000001.00000000 2024-03-06T07:28:05.713 INFO:teuthology.orchestra.run.smithi122.stdout:10000000001.00000001 2024-03-06T07:28:05.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7ffb340113f0 msgr2=0x7ffb34011830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7ffb340113f0 0x7ffb34011830 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7ffb38012990 msgr2=0x7ffb3801d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7ffb38012990 0x7ffb3801d410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ffb3803b960 msgr2=0x7ffb3803a3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ffb3803b960 0x7ffb3803a3e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7ffb3006e060 msgr2=0x7ffb3009a480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7ffb3006e060 0x7ffb3009a480 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7ffb3802d910 msgr2=0x7ffb3801b3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7ffb3802d910 0x7ffb3801b3f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ffb3401d8d0 msgr2=0x7ffb3401a330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7ffb3401d8d0 0x7ffb3401a330 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 reap_dead start 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ffb3009b820 msgr2=0x7ffb30058990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ffb3009b820 0x7ffb30058990 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x55f9cad17260 msgr2=0x55f9cad17660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb3efb4640 1 -- 172.21.15.122:0/975139223 reap_dead start 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x55f9cad17260 0x55f9cad17660 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb20069ee0 msgr2=0x7ffb2006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb20069ee0 0x7ffb2006c3a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffb34001ed0 tx=0x7ffb34001930 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 msgr2=0x55f9caccfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 0x55f9caccfe30 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ffb30046e10 tx=0x7ffb30002fb0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 shutdown_connections 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x55f9cad17260 0x55f9cad17660 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb20069ee0 0x7ffb2006c3a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 --2- 172.21.15.122:0/975139223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9cad18390 0x55f9caccfe30 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 >> 172.21.15.122:0/975139223 conn(0x55f9cacf9780 msgr2=0x55f9caccdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 shutdown_connections 2024-03-06T07:28:05.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.706+0000 7ffb40cb2f40 1 -- 172.21.15.122:0/975139223 wait complete. 2024-03-06T07:28:05.718 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:28:05.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.856+0000 7f01303c7640 1 Processor -- start 2024-03-06T07:28:05.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.856+0000 7f01303c7640 1 -- start start 2024-03-06T07:28:05.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f01303c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f012815bf30 0x7f012815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f01303c7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012815f580 con 0x7f012813dc10 2024-03-06T07:28:05.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f01303c7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012815b450 con 0x7f01280b3060 2024-03-06T07:28:05.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f01303c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01280cf3f0 con 0x7f012815bf30 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012e163640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f012813dc10 0x7f01280b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52296/0 (socket says 172.21.15.122:52296) 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012e163640 1 -- 172.21.15.122:0/3711470572 learned_addr learned my addr 172.21.15.122:0/3711470572 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012e964640 1 --2- 172.21.15.122:0/3711470572 >> v2:172.21.15.169:3300/0 conn(0x7f012815bf30 0x7f012815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012d161640 1 -- 172.21.15.122:0/3711470572 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303161536 0 0) 0x7f012815f580 con 0x7f012813dc10 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012d161640 1 -- 172.21.15.122:0/3711470572 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0114003520 con 0x7f012813dc10 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012d161640 1 -- 172.21.15.122:0/3711470572 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1974985872 0 0) 0x7f0114003520 con 0x7f012813dc10 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012e964640 1 -- 172.21.15.122:0/3711470572 >> v1:172.21.15.169:6789/0 conn(0x7f01280b3060 legacy=0x7f012815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012e964640 1 -- 172.21.15.122:0/3711470572 >> v1:172.21.15.122:6789/0 conn(0x7f012813dc10 legacy=0x7f01280b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.857+0000 7f012e964640 1 -- 172.21.15.122:0/3711470572 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012815b730 con 0x7f012815bf30 2024-03-06T07:28:05.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.858+0000 7f012e964640 1 --2- 172.21.15.122:0/3711470572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012815bf30 0x7f012815ef60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0120009d90 tx=0x7f012002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc533b4e876c2526 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.858+0000 7f012d161640 1 -- 172.21.15.122:0/3711470572 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120039070 con 0x7f012815bf30 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.858+0000 7f012d161640 1 -- 172.21.15.122:0/3711470572 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012002c8b0 con 0x7f012815bf30 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.858+0000 7f012d161640 1 -- 172.21.15.122:0/3711470572 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124001670 con 0x7f012815bf30 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 -- 172.21.15.122:0/3711470572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012815bf30 msgr2=0x7f012815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 --2- 172.21.15.122:0/3711470572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012815bf30 0x7f012815ef60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0120009d90 tx=0x7f012002bd80 comp rx=0 tx=0).stop 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 -- 172.21.15.122:0/3711470572 shutdown_connections 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 --2- 172.21.15.122:0/3711470572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012815bf30 0x7f012815ef60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 -- 172.21.15.122:0/3711470572 >> 172.21.15.122:0/3711470572 conn(0x7f01280cf8a0 msgr2=0x7f01280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 -- 172.21.15.122:0/3711470572 shutdown_connections 2024-03-06T07:28:05.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.859+0000 7f01303c7640 1 -- 172.21.15.122:0/3711470572 wait complete. 2024-03-06T07:28:05.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.860+0000 7f01303c7640 1 Processor -- start 2024-03-06T07:28:05.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.860+0000 7f01303c7640 1 -- start start 2024-03-06T07:28:05.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f01303c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 0x7f01281443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f01303c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012813dc10 0x7f01280b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f01303c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f012815bf30 0x7f01280b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f01303c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01280718e0 con 0x7f01280b3060 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f01303c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012815b210 con 0x7f012813dc10 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f01303c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0128071a60 con 0x7f012815bf30 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 0x7f01281443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 0x7f01281443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59280/0 (socket says 172.21.15.122:59280) 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 -- 172.21.15.122:0/2805001053 learned_addr learned my addr 172.21.15.122:0/2805001053 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 -- 172.21.15.122:0/2805001053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f012815bf30 msgr2=0x7f01280b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f012815bf30 0x7f01280b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 -- 172.21.15.122:0/2805001053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012813dc10 msgr2=0x7f01280b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012813dc10 0x7f01280b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 -- 172.21.15.122:0/2805001053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128152760 con 0x7f01280b3060 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.861+0000 7f012e163640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 0x7f01281443f0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f012400c760 tx=0x7f012400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.862+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124017070 con 0x7f01280b3060 2024-03-06T07:28:05.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.862+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124005ad0 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.862+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0128152a40 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.862+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124007070 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.862+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0128153000 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.863+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012804f330 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.864+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f012401a070 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.864+0000 7f011effd640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.864+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0124089760 con 0x7f01280b3060 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.864+0000 7f012d962640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.864+0000 7f012d962640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f010c06b4d0 0x7f010c06d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0118000f60 tx=0x7f01180079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:05.872+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01240585e0 con 0x7f01280b3060 2024-03-06T07:28:06.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.083+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f012806eb00 con 0x7f01280b3060 2024-03-06T07:28:06.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.084+0000 7f011effd640 1 -- 172.21.15.122:0/2805001053 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f0124057ee0 con 0x7f01280b3060 2024-03-06T07:28:06.086 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:06.086 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:06.086 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:06.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f010c06b4d0 msgr2=0x7f010c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f010c06b4d0 0x7f010c06d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0118000f60 tx=0x7f01180079b0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 msgr2=0x7f01281443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 0x7f01281443f0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f012400c760 tx=0x7f012400cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 shutdown_connections 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f012815bf30 0x7f01280b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f012813dc10 0x7f01280b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 --2- 172.21.15.122:0/2805001053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01280b3060 0x7f01281443f0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 >> 172.21.15.122:0/2805001053 conn(0x7f01280cf8a0 msgr2=0x7f012806d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 shutdown_connections 2024-03-06T07:28:06.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.086+0000 7f01303c7640 1 -- 172.21.15.122:0/2805001053 wait complete. 2024-03-06T07:28:06.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:06.230 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:06.230 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:06.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.332+0000 7f711ca4e640 1 Processor -- start 2024-03-06T07:28:06.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.332+0000 7f711ca4e640 1 -- start start 2024-03-06T07:28:06.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f711ca4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b4170 0x7f71180b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f711ca4e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f71180d4760 0x7f71180d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f711ca4e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71180b4dd0 0x7f71180d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f711ca4e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71180d4bf0 con 0x7f71180b4dd0 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f711ca4e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71180ce9e0 con 0x7f71180d4760 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f711ca4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180d25d0 con 0x7f71180b4170 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71180b4dd0 0x7f71180d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71180b4dd0 0x7f71180d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59296/0 (socket says 172.21.15.122:59296) 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 -- 172.21.15.122:0/3515770462 learned_addr learned my addr 172.21.15.122:0/3515770462 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 -- 172.21.15.122:0/3515770462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b4170 msgr2=0x7f71180b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f710ffff640 1 --2- 172.21.15.122:0/3515770462 >> v2:172.21.15.169:3300/0 conn(0x7f71180d4760 0x7f71180d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 --2- 172.21.15.122:0/3515770462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b4170 0x7f71180b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 -- 172.21.15.122:0/3515770462 >> v2:172.21.15.169:3300/0 conn(0x7f71180d4760 msgr2=0x7f71180d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 --2- 172.21.15.122:0/3515770462 >> v2:172.21.15.169:3300/0 conn(0x7f71180d4760 0x7f71180d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.333+0000 7f7116d9d640 1 -- 172.21.15.122:0/3515770462 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180d2870 con 0x7f71180b4dd0 2024-03-06T07:28:06.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.334+0000 7f7116d9d640 1 --2- 172.21.15.122:0/3515770462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180d08b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f711808d600 tx=0x7f711002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475b7210ebcefbb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.334+0000 7f7115d9b640 1 -- 172.21.15.122:0/3515770462 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711002ca70 con 0x7f71180b4dd0 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.334+0000 7f7115d9b640 1 -- 172.21.15.122:0/3515770462 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711002cc10 con 0x7f71180b4dd0 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.334+0000 7f7115d9b640 1 -- 172.21.15.122:0/3515770462 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71100343f0 con 0x7f71180b4dd0 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 -- 172.21.15.122:0/3515770462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 msgr2=0x7f71180d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3515770462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180d08b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f711808d600 tx=0x7f711002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 -- 172.21.15.122:0/3515770462 shutdown_connections 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3515770462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180d08b0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3515770462 >> v2:172.21.15.169:3300/0 conn(0x7f71180d4760 0x7f71180d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3515770462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b4170 0x7f71180b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 -- 172.21.15.122:0/3515770462 >> 172.21.15.122:0/3515770462 conn(0x7f71180d3c40 msgr2=0x7f71180d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 -- 172.21.15.122:0/3515770462 shutdown_connections 2024-03-06T07:28:06.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.335+0000 7f711ca4e640 1 -- 172.21.15.122:0/3515770462 wait complete. 2024-03-06T07:28:06.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.337+0000 7f711ca4e640 1 Processor -- start 2024-03-06T07:28:06.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.337+0000 7f711ca4e640 1 -- start start 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711ca4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71180b4170 0x7f71181551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711ca4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711ca4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b0fe0 0x7f71180b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711ca4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71180b47d0 con 0x7f71180b4dd0 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711ca4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71180b4550 con 0x7f71180b4170 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711ca4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180b4950 con 0x7f71180b0fe0 2024-03-06T07:28:06.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59312/0 (socket says 172.21.15.122:59312) 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 -- 172.21.15.122:0/3561490490 learned_addr learned my addr 172.21.15.122:0/3561490490 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f7116d9d640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b0fe0 0x7f71180b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 -- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b0fe0 msgr2=0x7f71180b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b0fe0 0x7f71180b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711659c640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71180b4170 0x7f71181551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 -- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71180b4170 msgr2=0x7f71181551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71180b4170 0x7f71181551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 -- 172.21.15.122:0/3561490490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180b1470 con 0x7f71180b4dd0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f7116d9d640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b0fe0 0x7f71180b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f711659c640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71180b4170 0x7f71181551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.338+0000 7f710ffff640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180c8400 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f710000c7f0 tx=0x7f710000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.339+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100015070 con 0x7f71180b4dd0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.339+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7100004240 con 0x7f71180b4dd0 2024-03-06T07:28:06.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.339+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71180b1750 con 0x7f71180b4dd0 2024-03-06T07:28:06.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.339+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100003610 con 0x7f71180b4dd0 2024-03-06T07:28:06.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.339+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71180b1d10 con 0x7f71180b4dd0 2024-03-06T07:28:06.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.340+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711804f330 con 0x7f71180b4dd0 2024-03-06T07:28:06.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.340+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71000039c0 con 0x7f71180b4dd0 2024-03-06T07:28:06.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.341+0000 7f710effd640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70f806b4d0 0x7f70f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.341+0000 7f711659c640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70f806b4d0 0x7f70f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.341+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f710008a630 con 0x7f71180b4dd0 2024-03-06T07:28:06.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.341+0000 7f711659c640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70f806b4d0 0x7f70f806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f710800a870 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.345+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71000594b0 con 0x7f71180b4dd0 2024-03-06T07:28:06.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.623+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71180733b0 con 0x7f71180b4dd0 2024-03-06T07:28:06.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.624+0000 7f710effd640 1 -- 172.21.15.122:0/3561490490 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f710005de20 con 0x7f71180b4dd0 2024-03-06T07:28:06.626 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:06.626 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:06.627 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70f806b4d0 msgr2=0x7f70f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70f806b4d0 0x7f70f806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f710800a870 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 msgr2=0x7f71180c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180c8400 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f710000c7f0 tx=0x7f710000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 shutdown_connections 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70f806b4d0 0x7f70f806d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f71180b0fe0 0x7f71180b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71180b4dd0 0x7f71180c8400 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 --2- 172.21.15.122:0/3561490490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f71180b4170 0x7f71181551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 >> 172.21.15.122:0/3561490490 conn(0x7f71180d3c40 msgr2=0x7f71180b76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 shutdown_connections 2024-03-06T07:28:06.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.626+0000 7f711ca4e640 1 -- 172.21.15.122:0/3561490490 wait complete. 2024-03-06T07:28:06.639 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:06.768 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:28:06.768 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:06.768 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:06.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 Processor -- start 2024-03-06T07:28:06.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 -- start start 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 0x55f50d96fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b8f50 0x55f50d98efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50d85eb70 con 0x55f50d8b66c0 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50d862ad0 con 0x55f50d975570 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f09b5ff40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f50d8b7f20 con 0x55f50d8b8f50 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f07660640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 0x55f50d96fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f07660640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 0x55f50d96fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59332/0 (socket says 172.21.15.122:59332) 2024-03-06T07:28:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.808+0000 7f4f07660640 1 -- 172.21.15.122:0/1128154996 learned_addr learned my addr 172.21.15.122:0/1128154996 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07e61640 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f06e5f640 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b8f50 0x55f50d98efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07660640 1 -- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b8f50 msgr2=0x55f50d98efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07660640 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b8f50 0x55f50d98efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07660640 1 -- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 msgr2=0x55f50d8e6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07660640 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07660640 1 -- 172.21.15.122:0/1128154996 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f50d98c850 con 0x55f50d8b66c0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f06e5f640 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b8f50 0x55f50d98efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f07660640 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 0x55f50d96fd80 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8058040 tx=0x7f4ef8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221132ce25f8bfca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f05e5d640 1 -- 172.21.15.122:0/1128154996 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef804ed60 con 0x55f50d8b66c0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f05e5d640 1 -- 172.21.15.122:0/1128154996 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef8060430 con 0x55f50d8b66c0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f05e5d640 1 -- 172.21.15.122:0/1128154996 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8060820 con 0x55f50d8b66c0 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/1128154996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 msgr2=0x55f50d96fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 0x55f50d96fd80 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8058040 tx=0x7f4ef8051000 comp rx=0 tx=0).stop 2024-03-06T07:28:06.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/1128154996 shutdown_connections 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b8f50 0x55f50d98efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/1128154996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b66c0 0x55f50d96fd80 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.809+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/1128154996 >> 172.21.15.122:0/1128154996 conn(0x55f50d8e3350 msgr2=0x55f50d98dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.810+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/1128154996 shutdown_connections 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.810+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/1128154996 wait complete. 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.810+0000 7f4f09b5ff40 1 Processor -- start 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.810+0000 7f4f09b5ff40 1 -- start start 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f09b5ff40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b66c0 0x55f50d98b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f09b5ff40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b8f50 0x55f50d8f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f09b5ff40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f09b5ff40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50d9703a0 con 0x55f50d8b8f50 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f09b5ff40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50d973d70 con 0x55f50d975570 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f09b5ff40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f50d8f6b60 con 0x55f50d8b66c0 2024-03-06T07:28:06.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b8f50 0x55f50d8f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b8f50 0x55f50d8f94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59348/0 (socket says 172.21.15.122:59348) 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 -- 172.21.15.122:0/3136377995 learned_addr learned my addr 172.21.15.122:0/3136377995 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f07e61640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f07660640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b66c0 0x55f50d98b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 -- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b66c0 msgr2=0x55f50d98b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b66c0 0x55f50d98b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 -- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 msgr2=0x55f50d8e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f50d975570 0x55f50d8e6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 -- 172.21.15.122:0/3136377995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f50d973aa0 con 0x55f50d8b8f50 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f07660640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55f50d8b66c0 0x55f50d98b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f06e5f640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b8f50 0x55f50d8f94c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009760 tx=0x7f4efc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.811+0000 7f4f04e5b640 1 -- 172.21.15.122:0/3136377995 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc006b20 con 0x55f50d8b8f50 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.812+0000 7f4f04e5b640 1 -- 172.21.15.122:0/3136377995 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc006cc0 con 0x55f50d8b8f50 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.812+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f50d972e90 con 0x55f50d8b8f50 2024-03-06T07:28:06.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.812+0000 7f4f04e5b640 1 -- 172.21.15.122:0/3136377995 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc014680 con 0x55f50d8b8f50 2024-03-06T07:28:06.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.812+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f50d973450 con 0x55f50d8b8f50 2024-03-06T07:28:06.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.813+0000 7f4f04e5b640 1 -- 172.21.15.122:0/3136377995 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4efc01d440 con 0x55f50d8b8f50 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.813+0000 7f4f04e5b640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ef4069ee0 0x7f4ef406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.813+0000 7f4f04e5b640 1 -- 172.21.15.122:0/3136377995 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4efc089250 con 0x55f50d8b8f50 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.813+0000 7f4f07660640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ef4069ee0 0x7f4ef406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.813+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55f50d972990 0x55f50d8fb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55f50d8fbc70 con 0x55f50d972990 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f07e61640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55f50d972990 0x55f50d8fb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f07660640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ef4069ee0 0x7f4ef406c3a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8058040 tx=0x7f4ef8053000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f07e61640 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55f50d972990 0x55f50d8fb730 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f07e61640 1 -- 172.21.15.122:0/3136377995 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f4f00003450 con 0x55f50d972990 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55f50d972990 msgr2=0x55f50d8fb730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55f50d972990 0x55f50d8fb730 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ef4069ee0 msgr2=0x7f4ef406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4ef4069ee0 0x7f4ef406c3a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8058040 tx=0x7f4ef8053000 comp rx=0 tx=0).stop 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b8f50 msgr2=0x55f50d8f94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 --2- 172.21.15.122:0/3136377995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f50d8b8f50 0x55f50d8f94c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009760 tx=0x7f4efc009c20 comp rx=0 tx=0).stop 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f07e61640 1 -- 172.21.15.122:0/3136377995 reap_dead start 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 shutdown_connections 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.814+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 >> 172.21.15.122:0/3136377995 conn(0x55f50d8e3350 msgr2=0x55f50d8fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.815+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 shutdown_connections 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:06.815+0000 7f4f09b5ff40 1 -- 172.21.15.122:0/3136377995 wait complete. 2024-03-06T07:28:06.822 INFO:teuthology.orchestra.run.smithi122.stdout:] 10000000001 stray0 2024-03-06T07:28:06.822 DEBUG:teuthology.orchestra.run.smithi122:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:06.905 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 1099511627777, 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ancestors": [ 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1536, 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "10000000001", 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 11 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 256, 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "stray0", 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 5 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "pool": 17, 2024-03-06T07:28:06.906 INFO:teuthology.orchestra.run.smithi122.stdout: "old_pools": [] 2024-03-06T07:28:06.907 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:28:06.907 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:06.908 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:07.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.049+0000 7fa81d6aa640 1 Processor -- start 2024-03-06T07:28:07.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.049+0000 7fa81d6aa640 1 -- start start 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa81d6aa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815ab00 0x7fa8180b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa81d6aa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8180b48f0 0x7fa8180af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa81d6aa640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa8180afc00 0x7fa81806e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa81d6aa640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180b4e30 con 0x7fa8180b48f0 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa81d6aa640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180b5110 con 0x7fa8180afc00 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa81d6aa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81815b310 con 0x7fa81815ab00 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa8167fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8180b48f0 0x7fa8180af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa8167fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8180b48f0 0x7fa8180af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59352/0 (socket says 172.21.15.122:59352) 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.050+0000 7fa8167fc640 1 -- 172.21.15.122:0/1559554888 learned_addr learned my addr 172.21.15.122:0/1559554888 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa816ffd640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815ab00 0x7fa8180b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8167fc640 1 -- 172.21.15.122:0/1559554888 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815ab00 msgr2=0x7fa8180b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8167fc640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815ab00 0x7fa8180b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8177fe640 1 --2- 172.21.15.122:0/1559554888 >> v2:172.21.15.169:3300/0 conn(0x7fa8180afc00 0x7fa81806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8167fc640 1 -- 172.21.15.122:0/1559554888 >> v2:172.21.15.169:3300/0 conn(0x7fa8180afc00 msgr2=0x7fa81806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8167fc640 1 --2- 172.21.15.122:0/1559554888 >> v2:172.21.15.169:3300/0 conn(0x7fa8180afc00 0x7fa81806e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8167fc640 1 -- 172.21.15.122:0/1559554888 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81815b5f0 con 0x7fa8180b48f0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa816ffd640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815ab00 0x7fa8180b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8177fe640 1 --2- 172.21.15.122:0/1559554888 >> v2:172.21.15.169:3300/0 conn(0x7fa8180afc00 0x7fa81806e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa8167fc640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180b48f0 0x7fa8180af6c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fa81806e6a0 tx=0x7fa80c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0f2899922b089a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.051+0000 7fa815ffb640 1 -- 172.21.15.122:0/1559554888 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c03b070 con 0x7fa8180b48f0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa815ffb640 1 -- 172.21.15.122:0/1559554888 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80c036440 con 0x7fa8180b48f0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa815ffb640 1 -- 172.21.15.122:0/1559554888 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c036810 con 0x7fa8180b48f0 2024-03-06T07:28:07.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1559554888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180b48f0 msgr2=0x7fa8180af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180b48f0 0x7fa8180af6c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fa81806e6a0 tx=0x7fa80c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1559554888 shutdown_connections 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180b48f0 0x7fa8180af6c0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1559554888 >> v2:172.21.15.169:3300/0 conn(0x7fa8180afc00 0x7fa81806e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1559554888 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa81815ab00 0x7fa8180b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1559554888 >> 172.21.15.122:0/1559554888 conn(0x7fa81806c0c0 msgr2=0x7fa81806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1559554888 shutdown_connections 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.052+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1559554888 wait complete. 2024-03-06T07:28:07.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.054+0000 7fa81d6aa640 1 Processor -- start 2024-03-06T07:28:07.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.054+0000 7fa81d6aa640 1 -- start start 2024-03-06T07:28:07.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa81d6aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 0x7fa81814a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa81d6aa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8180b48f0 0x7fa8180d0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa81d6aa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815ab00 0x7fa8180a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa81d6aa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81815b060 con 0x7fa8180afc00 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa81d6aa640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81815aee0 con 0x7fa81815ab00 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa81d6aa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81806ecc0 con 0x7fa8180b48f0 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa816ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 0x7fa81814a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa816ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 0x7fa81814a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59364/0 (socket says 172.21.15.122:59364) 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa816ffd640 1 -- 172.21.15.122:0/1729174820 learned_addr learned my addr 172.21.15.122:0/1729174820 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa8167fc640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8180b48f0 0x7fa8180d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.055+0000 7fa8177fe640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815ab00 0x7fa8180a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa816ffd640 1 -- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8180b48f0 msgr2=0x7fa8180d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa816ffd640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8180b48f0 0x7fa8180d0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa816ffd640 1 -- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815ab00 msgr2=0x7fa8180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa816ffd640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815ab00 0x7fa8180a9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa816ffd640 1 -- 172.21.15.122:0/1729174820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180cf6c0 con 0x7fa8180afc00 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa816ffd640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 0x7fa81814a390 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fa80800c7f0 tx=0x7fa80800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808015070 con 0x7fa8180afc00 2024-03-06T07:28:07.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa808007820 con 0x7fa8180afc00 2024-03-06T07:28:07.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8180cf9a0 con 0x7fa8180afc00 2024-03-06T07:28:07.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808007bd0 con 0x7fa8180afc00 2024-03-06T07:28:07.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.056+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8180cff60 con 0x7fa8180afc00 2024-03-06T07:28:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.057+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8180d1800 con 0x7fa8180afc00 2024-03-06T07:28:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.058+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa808018070 con 0x7fa8180afc00 2024-03-06T07:28:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.058+0000 7fa7ff7fe640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.058+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa80808a5a0 con 0x7fa8180afc00 2024-03-06T07:28:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.058+0000 7fa8167fc640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.058+0000 7fa8167fc640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa80c038000 tx=0x7fa80c032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.067 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.066+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8180d1800 con 0x7fa8180afc00 2024-03-06T07:28:07.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.276+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa81806e700 con 0x7fa8180afc00 2024-03-06T07:28:07.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.276+0000 7fa7ff7fe640 1 -- 172.21.15.122:0/1729174820 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa808058eb0 con 0x7fa8180afc00 2024-03-06T07:28:07.278 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:07.279 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:07.279 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:07.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 msgr2=0x7fa80406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa80c038000 tx=0x7fa80c032000 comp rx=0 tx=0).stop 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 msgr2=0x7fa81814a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 0x7fa81814a390 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fa80800c7f0 tx=0x7fa80800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 shutdown_connections 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa80406b4d0 0x7fa80406d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa81815ab00 0x7fa8180a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8180b48f0 0x7fa8180d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 --2- 172.21.15.122:0/1729174820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8180afc00 0x7fa81814a390 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 >> 172.21.15.122:0/1729174820 conn(0x7fa81806c0c0 msgr2=0x7fa8180b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 shutdown_connections 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.278+0000 7fa81d6aa640 1 -- 172.21.15.122:0/1729174820 wait complete. 2024-03-06T07:28:07.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:07.409 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:28:07.410 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:28:07.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:07.410 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:28:07.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.574+0000 7fb4e6015640 1 Processor -- start 2024-03-06T07:28:07.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.574+0000 7fb4e6015640 1 -- start start 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4e6015640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 0x7fb4e00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4e6015640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e00b3430 con 0x7fb4e00b41f0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4e6015640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e00b3710 con 0x7fb4e015dfe0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4e6015640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00d2c50 con 0x7fb4e00d22b0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4dffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb4e00b41f0 0x7fb4e00b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52316/0 (socket says 172.21.15.122:52316) 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4dffff640 1 -- 172.21.15.122:0/407078074 learned_addr learned my addr 172.21.15.122:0/407078074 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774650894 0 0) 0x7fb4e00b3430 con 0x7fb4e00b41f0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c8003520 con 0x7fb4e00b41f0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4deffd640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 0x7fb4e00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3424786981 0 0) 0x7fb4c8003520 con 0x7fb4e00b41f0 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 msgr2=0x7fb4e00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 0x7fb4e00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 >> v1:172.21.15.169:6789/0 conn(0x7fb4e015dfe0 legacy=0x7fb4e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e00b39f0 con 0x7fb4e00b41f0 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb4d80019f0 con 0x7fb4e00b41f0 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.576+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 >> v1:172.21.15.122:6789/0 conn(0x7fb4e00b41f0 legacy=0x7fb4e00b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4de7fc640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 0x7fb4c80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4de7fc640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4c8003f60 0x7fb4c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4de7fc640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4c8004d40 0x7fb4c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c800ba70 con 0x7fb4c8004d40 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c800bc20 con 0x7fb4c8003f60 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c800be20 con 0x7fb4e00d22b0 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4dffff640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4c8004d40 0x7fb4c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4df7fe640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 0x7fb4c80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4deffd640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4c8003f60 0x7fb4c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4dffff640 1 -- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 msgr2=0x7fb4c80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4dffff640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00d22b0 0x7fb4c80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4dffff640 1 -- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4c8003f60 msgr2=0x7fb4c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4dffff640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4c8003f60 0x7fb4c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.577+0000 7fb4dffff640 1 -- 172.21.15.122:0/407078074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c8003520 con 0x7fb4c8004d40 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4dffff640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4c8004d40 0x7fb4c80051c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800a860 tx=0x7fb4c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cd7a51c8ffc64b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d800a480 con 0x7fb4c8004d40 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d8034630 con 0x7fb4c8004d40 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4de7fc640 1 -- 172.21.15.122:0/407078074 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8034aa0 con 0x7fb4c8004d40 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4e6015640 1 -- 172.21.15.122:0/407078074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4c8004d40 msgr2=0x7fb4c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4e6015640 1 --2- 172.21.15.122:0/407078074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4c8004d40 0x7fb4c80051c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800a860 tx=0x7fb4c8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4dffff640 1 -- 172.21.15.122:0/407078074 reap_dead start 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4e6015640 1 -- 172.21.15.122:0/407078074 shutdown_connections 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4e6015640 1 -- 172.21.15.122:0/407078074 >> 172.21.15.122:0/407078074 conn(0x7fb4e00d3cd0 msgr2=0x7fb4e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4e6015640 1 -- 172.21.15.122:0/407078074 shutdown_connections 2024-03-06T07:28:07.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.578+0000 7fb4e6015640 1 -- 172.21.15.122:0/407078074 wait complete. 2024-03-06T07:28:07.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 Processor -- start 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 -- start start 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00cdcb0 0x7fb4e014d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4e014d970 0x7fb4e0157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 0x7fb4e01f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00b3d60 con 0x7fb4e0157bb0 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00b31e0 con 0x7fb4e014d970 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.580+0000 7fb4e6015640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00d1c70 con 0x7fb4e00cdcb0 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 0x7fb4e01f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 0x7fb4e01f7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59374/0 (socket says 172.21.15.122:59374) 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 -- 172.21.15.122:0/1566957940 learned_addr learned my addr 172.21.15.122:0/1566957940 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4deffd640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4e014d970 0x7fb4e0157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4df7fe640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00cdcb0 0x7fb4e014d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 -- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00cdcb0 msgr2=0x7fb4e014d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00cdcb0 0x7fb4e014d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 -- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4e014d970 msgr2=0x7fb4e0157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4e014d970 0x7fb4e0157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 -- 172.21.15.122:0/1566957940 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0155c60 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4df7fe640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00cdcb0 0x7fb4e014d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4dffff640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 0x7fb4e01f7030 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800a980 tx=0x7fb4e00cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8045070 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d803a040 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e0155f40 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d800a480 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.581+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e0156500 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.582+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e0161670 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.583+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4d8048070 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.583+0000 7fb4c7fff640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb4cc06b500 0x7fb4cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.583+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb4d80abb00 con 0x7fb4e0157bb0 2024-03-06T07:28:07.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.583+0000 7fb4df7fe640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb4cc06b500 0x7fb4cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.583+0000 7fb4df7fe640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb4cc06b500 0x7fb4cc06d9c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00036b0 tx=0x7fb4d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.587+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4e0161670 con 0x7fb4e0157bb0 2024-03-06T07:28:07.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.804+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4e004f330 con 0x7fb4e0157bb0 2024-03-06T07:28:07.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.804+0000 7fb4c7fff640 1 -- 172.21.15.122:0/1566957940 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fb4d807a9b0 con 0x7fb4e0157bb0 2024-03-06T07:28:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:07.807 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:07.807 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb4cc06b500 msgr2=0x7fb4cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb4cc06b500 0x7fb4cc06d9c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00036b0 tx=0x7fb4d0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 msgr2=0x7fb4e01f7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 0x7fb4e01f7030 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800a980 tx=0x7fb4e00cfd40 comp rx=0 tx=0).stop 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 shutdown_connections 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb4cc06b500 0x7fb4cc06d9c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4e0157bb0 0x7fb4e01f7030 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4e014d970 0x7fb4e0157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 --2- 172.21.15.122:0/1566957940 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb4e00cdcb0 0x7fb4e014d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.806+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 >> 172.21.15.122:0/1566957940 conn(0x7fb4e00d3cd0 msgr2=0x7fb4e00cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.807+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 shutdown_connections 2024-03-06T07:28:07.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:07.807+0000 7fb4e6015640 1 -- 172.21.15.122:0/1566957940 wait complete. 2024-03-06T07:28:07.819 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:07.934 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:07.935 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:07.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:08.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.034+0000 7f9362d3a640 1 Processor -- start 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.034+0000 7f9362d3a640 1 -- start start 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9362d3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c15bb20 0x7f935c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9362d3a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f935c0cfff0 0x7f935c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9362d3a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f935c0ce4b0 0x7f935c0ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9362d3a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c161980 con 0x7f935c0cfff0 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9362d3a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c161c60 con 0x7f935c0ce4b0 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9362d3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c161f40 con 0x7f935c15bb20 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f935bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f935c0cfff0 0x7f935c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f935bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f935c0cfff0 0x7f935c0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59376/0 (socket says 172.21.15.122:59376) 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f935bfff640 1 -- 172.21.15.122:0/1587014135 learned_addr learned my addr 172.21.15.122:0/1587014135 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f93612d7640 1 --2- 172.21.15.122:0/1587014135 >> v2:172.21.15.169:3300/0 conn(0x7f935c0ce4b0 0x7f935c0ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.035+0000 7f9360ad6640 1 --2- 172.21.15.122:0/1587014135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c15bb20 0x7f935c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935bfff640 1 -- 172.21.15.122:0/1587014135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c15bb20 msgr2=0x7f935c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935bfff640 1 --2- 172.21.15.122:0/1587014135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c15bb20 0x7f935c0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935bfff640 1 -- 172.21.15.122:0/1587014135 >> v2:172.21.15.169:3300/0 conn(0x7f935c0ce4b0 msgr2=0x7f935c0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935bfff640 1 --2- 172.21.15.122:0/1587014135 >> v2:172.21.15.169:3300/0 conn(0x7f935c0ce4b0 0x7f935c0ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935bfff640 1 -- 172.21.15.122:0/1587014135 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c162220 con 0x7f935c0cfff0 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935bfff640 1 --2- 172.21.15.122:0/1587014135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c0d0470 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f935c0b13b0 tx=0x7f934402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294bbb4730ba252e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.036+0000 7f935b7fe640 1 -- 172.21.15.122:0/1587014135 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344035020 con 0x7f935c0cfff0 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f935b7fe640 1 -- 172.21.15.122:0/1587014135 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344031870 con 0x7f935c0cfff0 2024-03-06T07:28:08.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f935b7fe640 1 -- 172.21.15.122:0/1587014135 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344031c40 con 0x7f935c0cfff0 2024-03-06T07:28:08.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 -- 172.21.15.122:0/1587014135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 msgr2=0x7f935c0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 --2- 172.21.15.122:0/1587014135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c0d0470 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f935c0b13b0 tx=0x7f934402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:08.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 -- 172.21.15.122:0/1587014135 shutdown_connections 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 --2- 172.21.15.122:0/1587014135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c0d0470 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 --2- 172.21.15.122:0/1587014135 >> v2:172.21.15.169:3300/0 conn(0x7f935c0ce4b0 0x7f935c0ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 --2- 172.21.15.122:0/1587014135 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c15bb20 0x7f935c0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 -- 172.21.15.122:0/1587014135 >> 172.21.15.122:0/1587014135 conn(0x7f935c09a9f0 msgr2=0x7f935c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 -- 172.21.15.122:0/1587014135 shutdown_connections 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.037+0000 7f9362d3a640 1 -- 172.21.15.122:0/1587014135 wait complete. 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.038+0000 7f9362d3a640 1 Processor -- start 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.038+0000 7f9362d3a640 1 -- start start 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.039+0000 7f9362d3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c0ce4b0 0x7f935c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.039+0000 7f9362d3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c15f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.039+0000 7f9362d3a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f935c15bb20 0x7f935c15f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.039+0000 7f9362d3a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0d24b0 con 0x7f935c0cfff0 2024-03-06T07:28:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.039+0000 7f9362d3a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0d2330 con 0x7f935c15bb20 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.039+0000 7f9362d3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0d1980 con 0x7f935c0ce4b0 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c15f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c15f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59382/0 (socket says 172.21.15.122:59382) 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f9360ad6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c0ce4b0 0x7f935c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 -- 172.21.15.122:0/799892355 learned_addr learned my addr 172.21.15.122:0/799892355 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f93612d7640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f935c15bb20 0x7f935c15f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 -- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c0ce4b0 msgr2=0x7f935c06d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c0ce4b0 0x7f935c06d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 -- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f935c15bb20 msgr2=0x7f935c15f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f935c15bb20 0x7f935c15f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 -- 172.21.15.122:0/799892355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0c77a0 con 0x7f935c0cfff0 2024-03-06T07:28:08.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.040+0000 7f935bfff640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c15f330 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f9344000c00 tx=0x7f93440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.041+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344035040 con 0x7f935c0cfff0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.041+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344030d00 con 0x7f935c0cfff0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.041+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c0c7a80 con 0x7f935c0cfff0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.042+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440456a0 con 0x7f935c0cfff0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.042+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c0c8040 con 0x7f935c0cfff0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.043+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c04f330 con 0x7f935c0cfff0 2024-03-06T07:28:08.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.043+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9344038020 con 0x7f935c0cfff0 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.043+0000 7f93597fa640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f933c06b4d0 0x7f933c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.043+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f93440ac460 con 0x7f935c0cfff0 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.044+0000 7f9360ad6640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f933c06b4d0 0x7f933c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.044+0000 7f9360ad6640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f933c06b4d0 0x7f933c06d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9354011340 tx=0x7f935400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.052+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f934407ac90 con 0x7f935c0cfff0 2024-03-06T07:28:08.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.267+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f935c0b0bb0 con 0x7f935c0cfff0 2024-03-06T07:28:08.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.267+0000 7f93597fa640 1 -- 172.21.15.122:0/799892355 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f934407ed30 con 0x7f935c0cfff0 2024-03-06T07:28:08.270 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:08.270 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:08.270 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:08.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f933c06b4d0 msgr2=0x7f933c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f933c06b4d0 0x7f933c06d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9354011340 tx=0x7f935400b040 comp rx=0 tx=0).stop 2024-03-06T07:28:08.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 msgr2=0x7f935c15f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c15f330 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f9344000c00 tx=0x7f93440046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 shutdown_connections 2024-03-06T07:28:08.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f933c06b4d0 0x7f933c06d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f935c15bb20 0x7f935c15f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f935c0cfff0 0x7f935c15f330 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 --2- 172.21.15.122:0/799892355 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f935c0ce4b0 0x7f935c06d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 >> 172.21.15.122:0/799892355 conn(0x7f935c09a9f0 msgr2=0x7f935c0cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 shutdown_connections 2024-03-06T07:28:08.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.270+0000 7f9362d3a640 1 -- 172.21.15.122:0/799892355 wait complete. 2024-03-06T07:28:08.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:08.402 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:28:08.402 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:28:08.403 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.501+0000 7f31f044e640 1 Processor -- start 2024-03-06T07:28:08.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.501+0000 7f31f044e640 1 -- start start 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31f044e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e80cfff0 0x7f31e80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31f044e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f31e80cc8b0 0x7f31e80ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31f044e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e8161980 con 0x7f31e815bb20 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31f044e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e80d2500 con 0x7f31e80cc8b0 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31f044e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e8161c60 con 0x7f31e80cfff0 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31ee9eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f31e80cc8b0 0x7f31e80ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31ee9eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f31e80cc8b0 0x7f31e80ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59386/0 (socket says 172.21.15.122:59386) 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.502+0000 7f31ee9eb640 1 -- 172.21.15.122:0/2887939649 learned_addr learned my addr 172.21.15.122:0/2887939649 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.503+0000 7f31ed9e9640 1 --2- 172.21.15.122:0/2887939649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e80cfff0 0x7f31e80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.503+0000 7f31ee9eb640 1 -- 172.21.15.122:0/2887939649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e80cfff0 msgr2=0x7f31e80d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.503+0000 7f31ee9eb640 1 --2- 172.21.15.122:0/2887939649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e80cfff0 0x7f31e80d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.503+0000 7f31ee9eb640 1 -- 172.21.15.122:0/2887939649 >> v1:172.21.15.169:6789/0 conn(0x7f31e815bb20 legacy=0x7f31e80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.503+0000 7f31ee9eb640 1 -- 172.21.15.122:0/2887939649 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8161f40 con 0x7f31e80cc8b0 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31ee9eb640 1 --2- 172.21.15.122:0/2887939649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e80ccd40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009e50 tx=0x7f31e402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ff755b56e72b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31ed1e8640 1 -- 172.21.15.122:0/2887939649 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e402cb00 con 0x7f31e80cc8b0 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31ed1e8640 1 -- 172.21.15.122:0/2887939649 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e402cca0 con 0x7f31e80cc8b0 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31ed1e8640 1 -- 172.21.15.122:0/2887939649 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e4034610 con 0x7f31e80cc8b0 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 -- 172.21.15.122:0/2887939649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 msgr2=0x7f31e80ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 --2- 172.21.15.122:0/2887939649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e80ccd40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009e50 tx=0x7f31e402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 -- 172.21.15.122:0/2887939649 shutdown_connections 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 --2- 172.21.15.122:0/2887939649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e80ccd40 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 --2- 172.21.15.122:0/2887939649 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e80cfff0 0x7f31e80d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 -- 172.21.15.122:0/2887939649 >> 172.21.15.122:0/2887939649 conn(0x7f31e809a9f0 msgr2=0x7f31e809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 -- 172.21.15.122:0/2887939649 shutdown_connections 2024-03-06T07:28:08.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.504+0000 7f31f044e640 1 -- 172.21.15.122:0/2887939649 wait complete. 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.505+0000 7f31f044e640 1 Processor -- start 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 -- start start 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31e80cfff0 0x7f31e80c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e815bb20 0x7f31e815a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e80d1980 con 0x7f31e80cc8b0 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e8162500 con 0x7f31e80cfff0 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31f044e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e80d1b00 con 0x7f31e815bb20 2024-03-06T07:28:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31ee1ea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31ee1ea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e81554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59388/0 (socket says 172.21.15.122:59388) 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31ee1ea640 1 -- 172.21.15.122:0/4021745128 learned_addr learned my addr 172.21.15.122:0/4021745128 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31ed9e9640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31e80cfff0 0x7f31e80c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.506+0000 7f31ee9eb640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e815bb20 0x7f31e815a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee1ea640 1 -- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e815bb20 msgr2=0x7f31e815a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee1ea640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e815bb20 0x7f31e815a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee1ea640 1 -- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31e80cfff0 msgr2=0x7f31e80c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee1ea640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31e80cfff0 0x7f31e80c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee1ea640 1 -- 172.21.15.122:0/4021745128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e80c6020 con 0x7f31e80cc8b0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee9eb640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e815bb20 0x7f31e815a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31ee1ea640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e81554e0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f31e000c3c0 tx=0x7f31e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00052d0 con 0x7f31e80cc8b0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e0005490 con 0x7f31e80cc8b0 2024-03-06T07:28:08.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e80c6300 con 0x7f31e80cc8b0 2024-03-06T07:28:08.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0007260 con 0x7f31e80cc8b0 2024-03-06T07:28:08.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.507+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e80c68c0 con 0x7f31e80cc8b0 2024-03-06T07:28:08.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.508+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e804f330 con 0x7f31e80cc8b0 2024-03-06T07:28:08.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.509+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31e0007400 con 0x7f31e80cc8b0 2024-03-06T07:28:08.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.509+0000 7f31deffd640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f31cc06b4d0 0x7f31cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.509+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f31e00895f0 con 0x7f31e80cc8b0 2024-03-06T07:28:08.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.509+0000 7f31ed9e9640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f31cc06b4d0 0x7f31cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.509+0000 7f31ed9e9640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f31cc06b4d0 0x7f31cc06d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f31e806dfe0 tx=0x7f31d8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.513+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31e0057e20 con 0x7f31e80cc8b0 2024-03-06T07:28:08.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.731+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31e809b0b0 con 0x7f31e80cc8b0 2024-03-06T07:28:08.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.731+0000 7f31deffd640 1 -- 172.21.15.122:0/4021745128 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f31e005bec0 con 0x7f31e80cc8b0 2024-03-06T07:28:08.733 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:08.733 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:08.734 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:08.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f31cc06b4d0 msgr2=0x7f31cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f31cc06b4d0 0x7f31cc06d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f31e806dfe0 tx=0x7f31d8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:08.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 msgr2=0x7f31e81554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e81554e0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f31e000c3c0 tx=0x7f31e000c880 comp rx=0 tx=0).stop 2024-03-06T07:28:08.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 shutdown_connections 2024-03-06T07:28:08.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f31cc06b4d0 0x7f31cc06d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f31e815bb20 0x7f31e815a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31e80cfff0 0x7f31e80c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 --2- 172.21.15.122:0/4021745128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31e80cc8b0 0x7f31e81554e0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 >> 172.21.15.122:0/4021745128 conn(0x7f31e809a9f0 msgr2=0x7f31e80b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 shutdown_connections 2024-03-06T07:28:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.733+0000 7f31f044e640 1 -- 172.21.15.122:0/4021745128 wait complete. 2024-03-06T07:28:08.746 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:08.874 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:28:08.874 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:08.874 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:08.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 Processor -- start 2024-03-06T07:28:08.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 -- start start 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa2175240 0x55bfa22456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa21b23e0 0x55bfa2244280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa22447c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfa21bc0e0 con 0x55bfa224a080 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfa2119b70 con 0x55bfa2175240 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee657ff40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfa21bc260 con 0x55bfa21b23e0 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee4881640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa22447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee4881640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa22447c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59402/0 (socket says 172.21.15.122:59402) 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee4881640 1 -- 172.21.15.122:0/4139047886 learned_addr learned my addr 172.21.15.122:0/4139047886 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:08.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee4080640 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa2175240 0x55bfa22456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.915+0000 7f3ee387f640 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa21b23e0 0x55bfa2244280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4881640 1 -- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa21b23e0 msgr2=0x55bfa2244280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4881640 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa21b23e0 0x55bfa2244280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4881640 1 -- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa2175240 msgr2=0x55bfa22456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4881640 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa2175240 0x55bfa22456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4881640 1 -- 172.21.15.122:0/4139047886 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfa224e3c0 con 0x55bfa224a080 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4080640 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa2175240 0x55bfa22456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee4881640 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa22447c0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3edc001660 tx=0x7f3edc001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1652384a8b893 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee287d640 1 -- 172.21.15.122:0/4139047886 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc003650 con 0x55bfa224a080 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee287d640 1 -- 172.21.15.122:0/4139047886 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc003830 con 0x55bfa224a080 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee287d640 1 -- 172.21.15.122:0/4139047886 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc003ca0 con 0x55bfa224a080 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 -- 172.21.15.122:0/4139047886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 msgr2=0x55bfa22447c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa22447c0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3edc001660 tx=0x7f3edc001b20 comp rx=0 tx=0).stop 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 -- 172.21.15.122:0/4139047886 shutdown_connections 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa22447c0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa21b23e0 0x55bfa2244280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/4139047886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa2175240 0x55bfa22456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 -- 172.21.15.122:0/4139047886 >> 172.21.15.122:0/4139047886 conn(0x55bfa219ebd0 msgr2=0x55bfa21b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.916+0000 7f3ee657ff40 1 -- 172.21.15.122:0/4139047886 shutdown_connections 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.917+0000 7f3ee657ff40 1 -- 172.21.15.122:0/4139047886 wait complete. 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.917+0000 7f3ee657ff40 1 Processor -- start 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.917+0000 7f3ee657ff40 1 -- start start 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee657ff40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa2175240 0x55bfa21a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee657ff40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa21b23e0 0x55bfa223dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee657ff40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa21b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee657ff40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfa224e660 con 0x55bfa224a080 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee657ff40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfa219ccc0 con 0x55bfa21b23e0 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee657ff40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfa224b460 con 0x55bfa2175240 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa21b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa21b2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59406/0 (socket says 172.21.15.122:59406) 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 -- 172.21.15.122:0/3085630313 learned_addr learned my addr 172.21.15.122:0/3085630313 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4080640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa2175240 0x55bfa21a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee387f640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa21b23e0 0x55bfa223dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 -- 172.21.15.122:0/3085630313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa2175240 msgr2=0x55bfa21a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55bfa2175240 0x55bfa21a1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 -- 172.21.15.122:0/3085630313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa21b23e0 msgr2=0x55bfa223dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bfa21b23e0 0x55bfa223dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 -- 172.21.15.122:0/3085630313 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfa224b740 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee4881640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa21b2a90 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3edc001660 tx=0x7f3edc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.918+0000 7f3ee187b640 1 -- 172.21.15.122:0/3085630313 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc024070 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.919+0000 7f3ee187b640 1 -- 172.21.15.122:0/3085630313 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc019040 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.919+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bfa224ba20 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.919+0000 7f3ee187b640 1 -- 172.21.15.122:0/3085630313 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc013880 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.919+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfa223db10 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.920+0000 7f3ee187b640 1 -- 172.21.15.122:0/3085630313 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3edc010070 con 0x55bfa224a080 2024-03-06T07:28:08.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.920+0000 7f3ee187b640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ed0069eb0 0x7f3ed006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.920+0000 7f3ee187b640 1 -- 172.21.15.122:0/3085630313 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3edc089dc0 con 0x55bfa224a080 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.920+0000 7f3ee4080640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ed0069eb0 0x7f3ed006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55bfa21b2fd0 0x55bfa223ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55bfa223f8b0 con 0x55bfa21b2fd0 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee387f640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55bfa21b2fd0 0x55bfa223ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee4080640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ed0069eb0 0x7f3ed006c370 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8054000 tx=0x7f3ed804e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee387f640 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55bfa21b2fd0 0x55bfa223ca10 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee387f640 1 -- 172.21.15.122:0/3085630313 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f3ed4009970 con 0x55bfa21b2fd0 2024-03-06T07:28:08.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55bfa21b2fd0 msgr2=0x55bfa223ca10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55bfa21b2fd0 0x55bfa223ca10 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ed0069eb0 msgr2=0x7f3ed006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3ed0069eb0 0x7f3ed006c370 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8054000 tx=0x7f3ed804e000 comp rx=0 tx=0).stop 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 msgr2=0x55bfa21b2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 --2- 172.21.15.122:0/3085630313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bfa224a080 0x55bfa21b2a90 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3edc001660 tx=0x7f3edc001250 comp rx=0 tx=0).stop 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee4881640 1 -- 172.21.15.122:0/3085630313 reap_dead start 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 shutdown_connections 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.921+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 >> 172.21.15.122:0/3085630313 conn(0x55bfa219ebd0 msgr2=0x55bfa2241330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.922+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 shutdown_connections 2024-03-06T07:28:08.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:08.922+0000 7f3ee657ff40 1 -- 172.21.15.122:0/3085630313 wait complete. 2024-03-06T07:28:08.930 INFO:teuthology.orchestra.run.smithi122.stdout:W�file_cdir_2 2024-03-06T07:28:08.930 DEBUG:teuthology.orchestra.run.smithi122:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 1099511627777, 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "ancestors": [ 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1099511628278, 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "file_c", 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 15 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1, 2024-03-06T07:28:09.017 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "dir_2", 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 31 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout: "pool": 17, 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout: "old_pools": [] 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:28:09.018 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:09.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.019 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:28:09.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.078 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:28:09.136 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:09.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.278+0000 7f7c0ce4e640 1 Processor -- start 2024-03-06T07:28:09.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0ce4e640 1 -- start start 2024-03-06T07:28:09.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0ce4e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7c080d22b0 0x7f7c080d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0ce4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080b3430 con 0x7f7c0815dfe0 2024-03-06T07:28:09.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0ce4e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080b3710 con 0x7f7c080b41f0 2024-03-06T07:28:09.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0ce4e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c080d2c50 con 0x7f7c080d22b0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7c0815dfe0 0x7f7c080cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52326/0 (socket says 172.21.15.122:52326) 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.279+0000 7f7c0659c640 1 -- 172.21.15.122:0/1046454691 learned_addr learned my addr 172.21.15.122:0/1046454691 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599016729 0 0) 0x7f7c080b3430 con 0x7f7c0815dfe0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7bfffff640 1 --2- 172.21.15.122:0/1046454691 >> v2:172.21.15.169:3300/0 conn(0x7f7c080d22b0 0x7f7c080d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf4003520 con 0x7f7c0815dfe0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2934327369 0 0) 0x7f7bf4003520 con 0x7f7c0815dfe0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 >> v2:172.21.15.169:3300/0 conn(0x7f7c080d22b0 msgr2=0x7f7c080d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 --2- 172.21.15.122:0/1046454691 >> v2:172.21.15.169:3300/0 conn(0x7f7c080d22b0 0x7f7c080d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 >> v1:172.21.15.169:6789/0 conn(0x7f7c080b41f0 legacy=0x7f7c080b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c080b39f0 con 0x7f7c0815dfe0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7bf8001670 con 0x7f7c0815dfe0 2024-03-06T07:28:09.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 >> v1:172.21.15.122:6789/0 conn(0x7f7c0815dfe0 legacy=0x7f7c080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf40039b0 0x7f7bf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf4004800 0x7f7bf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7bf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf400c030 con 0x7f7bf40051c0 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf400c1e0 con 0x7f7bf40039b0 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf400c3e0 con 0x7f7bf4004800 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c0659c640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf40039b0 0x7f7bf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7c06d9d640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7bf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.280+0000 7f7bfffff640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf4004800 0x7f7bf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 -- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf4004800 msgr2=0x7f7bf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7bf4004800 0x7f7bf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 -- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf40039b0 msgr2=0x7f7bf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf40039b0 0x7f7bf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 -- 172.21.15.122:0/1046454691 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4003520 con 0x7f7bf40051c0 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c0659c640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bf40039b0 0x7f7bf4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 -- 172.21.15.122:0/1046454691 reap_dead start 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c06d9d640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7bf400b9c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f7c00001e50 tx=0x7f7bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c47b3c8ddba31c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8007d10 con 0x7f7bf40051c0 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00003bb0 con 0x7f7bf40051c0 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.281+0000 7f7c05d9b640 1 -- 172.21.15.122:0/1046454691 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00001a70 con 0x7f7bf40051c0 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/1046454691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 msgr2=0x7f7bf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7bf400b9c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f7c00001e50 tx=0x7f7bf4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/1046454691 shutdown_connections 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/1046454691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7bf400b9c0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/1046454691 >> 172.21.15.122:0/1046454691 conn(0x7f7c080d3cd0 msgr2=0x7f7c080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/1046454691 shutdown_connections 2024-03-06T07:28:09.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.282+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/1046454691 wait complete. 2024-03-06T07:28:09.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.283+0000 7f7c0ce4e640 1 Processor -- start 2024-03-06T07:28:09.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.283+0000 7f7c0ce4e640 1 -- start start 2024-03-06T07:28:09.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0ce4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7c080cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0ce4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c0815a580 0x7f7c081530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0ce4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c0815b4b0 0x7f7c0815c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0ce4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c080b3d60 con 0x7f7bf40051c0 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0ce4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c080b31e0 con 0x7f7c0815b4b0 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0ce4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0815e3c0 con 0x7f7c0815a580 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7c080cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7c080cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59416/0 (socket says 172.21.15.122:59416) 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 -- 172.21.15.122:0/2442202145 learned_addr learned my addr 172.21.15.122:0/2442202145 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7bfffff640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c0815a580 0x7f7c081530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c06d9d640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c0815b4b0 0x7f7c0815c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 -- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c0815a580 msgr2=0x7f7c081530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c0815a580 0x7f7c081530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 -- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c0815b4b0 msgr2=0x7f7c0815c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c0815b4b0 0x7f7c0815c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c0659c640 1 -- 172.21.15.122:0/2442202145 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c080cc520 con 0x7f7bf40051c0 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7bfffff640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c0815a580 0x7f7c081530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.284+0000 7f7c06d9d640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c0815b4b0 0x7f7c0815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.285+0000 7f7c0659c640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7c080cd580 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80072c0 tx=0x7f7bf8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.285+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf800f960 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.285+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf8018070 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.285+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c080cc7c0 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.285+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf800fd40 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.285+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c080cd060 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.286+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c080cc990 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.286+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bf801d460 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.287+0000 7f7bfeffd640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7be806b4d0 0x7f7be806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.287+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7bf808a3e0 con 0x7f7bf40051c0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.287+0000 7f7bfffff640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7be806b4d0 0x7f7be806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.287+0000 7f7bfffff640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7be806b4d0 0x7f7be806d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7c0815a450 tx=0x7f7bf0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.291+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bf8059260 con 0x7f7bf40051c0 2024-03-06T07:28:09.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.499+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0804f330 con 0x7f7bf40051c0 2024-03-06T07:28:09.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.500+0000 7f7bfeffd640 1 -- 172.21.15.122:0/2442202145 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f7bf8058b60 con 0x7f7bf40051c0 2024-03-06T07:28:09.502 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:09.502 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:09.502 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7be806b4d0 msgr2=0x7f7be806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7be806b4d0 0x7f7be806d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7c0815a450 tx=0x7f7bf0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 msgr2=0x7f7c080cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7c080cd580 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80072c0 tx=0x7f7bf8007780 comp rx=0 tx=0).stop 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 shutdown_connections 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7be806b4d0 0x7f7be806d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c0815b4b0 0x7f7c0815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7c0815a580 0x7f7c081530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 --2- 172.21.15.122:0/2442202145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf40051c0 0x7f7c080cd580 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 >> 172.21.15.122:0/2442202145 conn(0x7f7c080d3cd0 msgr2=0x7f7c0806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 shutdown_connections 2024-03-06T07:28:09.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:09.502+0000 7f7c0ce4e640 1 -- 172.21.15.122:0/2442202145 wait complete. 2024-03-06T07:28:09.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:09.630 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:28:09.631 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:28:09.631 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:28:14.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:14.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.730+0000 7f4b1f39a640 1 Processor -- start 2024-03-06T07:28:14.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1f39a640 1 -- start start 2024-03-06T07:28:14.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1f39a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4b1815bb20 0x7f4b180d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1f39a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180cfff0 0x7f4b180d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1f39a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b18161980 con 0x7f4b180ce4b0 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1f39a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b18161c60 con 0x7f4b1815bb20 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1f39a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18161f40 con 0x7f4b180cfff0 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.731+0000 7f4b1d136640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4b1815bb20 0x7f4b180d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4b1815bb20 0x7f4b180d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34772/0 (socket says 172.21.15.122:34772) 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 -- 172.21.15.122:0/584795645 learned_addr learned my addr 172.21.15.122:0/584795645 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1c935640 1 --2- 172.21.15.122:0/584795645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180cfff0 0x7f4b180d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 -- 172.21.15.122:0/584795645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180cfff0 msgr2=0x7f4b180d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 --2- 172.21.15.122:0/584795645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180cfff0 0x7f4b180d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 -- 172.21.15.122:0/584795645 >> v1:172.21.15.169:6789/0 conn(0x7f4b180ce4b0 legacy=0x7f4b180ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 -- 172.21.15.122:0/584795645 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18162220 con 0x7f4b1815bb20 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.732+0000 7f4b1d136640 1 --2- 172.21.15.122:0/584795645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b180d1df0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009870 tx=0x7f4b0802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca867426d850e881 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b17fff640 1 -- 172.21.15.122:0/584795645 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08035020 con 0x7f4b1815bb20 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b17fff640 1 -- 172.21.15.122:0/584795645 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b08031640 con 0x7f4b1815bb20 2024-03-06T07:28:14.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b17fff640 1 -- 172.21.15.122:0/584795645 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0802d040 con 0x7f4b1815bb20 2024-03-06T07:28:14.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 -- 172.21.15.122:0/584795645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 msgr2=0x7f4b180d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/584795645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b180d1df0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009870 tx=0x7f4b0802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 -- 172.21.15.122:0/584795645 shutdown_connections 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/584795645 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180cfff0 0x7f4b180d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/584795645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b180d1df0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 -- 172.21.15.122:0/584795645 >> 172.21.15.122:0/584795645 conn(0x7f4b1809a9f0 msgr2=0x7f4b1809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 -- 172.21.15.122:0/584795645 shutdown_connections 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.733+0000 7f4b1f39a640 1 -- 172.21.15.122:0/584795645 wait complete. 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.734+0000 7f4b1f39a640 1 Processor -- start 2024-03-06T07:28:14.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.734+0000 7f4b1f39a640 1 -- start start 2024-03-06T07:28:14.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1f39a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180ce4b0 0x7f4b18153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1f39a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b180cfff0 0x7f4b181e77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1f39a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b181e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1f39a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180d24b0 con 0x7f4b1815bb20 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1f39a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180d2330 con 0x7f4b180cfff0 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1f39a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180d1980 con 0x7f4b180ce4b0 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1d937640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b181e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1d937640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b181e7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34786/0 (socket says 172.21.15.122:34786) 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1d937640 1 -- 172.21.15.122:0/2516385178 learned_addr learned my addr 172.21.15.122:0/2516385178 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1c935640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b180cfff0 0x7f4b181e77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.735+0000 7f4b1d136640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180ce4b0 0x7f4b18153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1d937640 1 -- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180ce4b0 msgr2=0x7f4b18153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1d937640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180ce4b0 0x7f4b18153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1d937640 1 -- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b180cfff0 msgr2=0x7f4b181e77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1d937640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b180cfff0 0x7f4b181e77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1d937640 1 -- 172.21.15.122:0/2516385178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18154320 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1d937640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b181e7d20 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000c780 tx=0x7f4b1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10004b60 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b10004d00 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1815c150 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1000aa90 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.736+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1815c710 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.737+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1804f330 con 0x7f4b1815bb20 2024-03-06T07:28:14.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.738+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b1000ac30 con 0x7f4b1815bb20 2024-03-06T07:28:14.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.738+0000 7f4b15ffb640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b0406b4d0 0x7f4b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.738+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4b1008a3a0 con 0x7f4b1815bb20 2024-03-06T07:28:14.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.738+0000 7f4b1d136640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b0406b4d0 0x7f4b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.739+0000 7f4b1d136640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b0406b4d0 0x7f4b0406d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4b08036000 tx=0x7f4b0803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.743+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b10020070 con 0x7f4b1815bb20 2024-03-06T07:28:14.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.958+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b1806eb60 con 0x7f4b1815bb20 2024-03-06T07:28:14.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.958+0000 7f4b15ffb640 1 -- 172.21.15.122:0/2516385178 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f4b10058bd0 con 0x7f4b1815bb20 2024-03-06T07:28:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:14.961 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:14.961 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.960+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b0406b4d0 msgr2=0x7f4b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.960+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b0406b4d0 0x7f4b0406d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4b08036000 tx=0x7f4b0803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.960+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 msgr2=0x7f4b181e7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.960+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b181e7d20 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000c780 tx=0x7f4b1000cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 shutdown_connections 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4b0406b4d0 0x7f4b0406d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b1815bb20 0x7f4b181e7d20 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b180cfff0 0x7f4b181e77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 --2- 172.21.15.122:0/2516385178 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4b180ce4b0 0x7f4b18153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 >> 172.21.15.122:0/2516385178 conn(0x7f4b1809a9f0 msgr2=0x7f4b18070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 shutdown_connections 2024-03-06T07:28:14.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:14.961+0000 7f4b1f39a640 1 -- 172.21.15.122:0/2516385178 wait complete. 2024-03-06T07:28:14.976 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:15.095 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:15.095 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:15.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.194+0000 7f8bb0393640 1 Processor -- start 2024-03-06T07:28:15.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.194+0000 7f8bb0393640 1 -- start start 2024-03-06T07:28:15.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bb0393640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d4760 0x7f8ba80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bb0393640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8ba80d0a80 0x7f8ba80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bb0393640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba80d25d0 con 0x7f8ba80b4170 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bb0393640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba80d4bf0 con 0x7f8ba80d0a80 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bb0393640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80ce9e0 con 0x7f8ba80d4760 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bae930640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8ba80d0a80 0x7f8ba80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bae930640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8ba80d0a80 0x7f8ba80cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:58628/0 (socket says 172.21.15.122:58628) 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bad92e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d4760 0x7f8ba80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bae930640 1 -- 172.21.15.122:0/238496453 learned_addr learned my addr 172.21.15.122:0/238496453 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.196+0000 7f8bae12f640 1 --1- 172.21.15.122:0/238496453 >> v1:172.21.15.169:6789/0 conn(0x7f8ba80b4170 0x7f8ba80d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:45966/0 (socket says 172.21.15.122:45966) 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.197+0000 7f8bae930640 1 -- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d4760 msgr2=0x7f8ba80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.197+0000 7f8bae930640 1 --2- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d4760 0x7f8ba80b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.197+0000 7f8bae930640 1 -- 172.21.15.122:0/238496453 >> v1:172.21.15.169:6789/0 conn(0x7f8ba80b4170 legacy=0x7f8ba80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.197+0000 7f8bae930640 1 -- 172.21.15.122:0/238496453 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba80d2870 con 0x7f8ba80d0a80 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.197+0000 7f8bae930640 1 --2- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80d0a80 0x7f8ba80cee70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009e50 tx=0x7f8ba402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66ef25eea18bb445 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bad12d640 1 -- 172.21.15.122:0/238496453 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba402cb00 con 0x7f8ba80d0a80 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bad12d640 1 -- 172.21.15.122:0/238496453 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba402cca0 con 0x7f8ba80d0a80 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bad12d640 1 -- 172.21.15.122:0/238496453 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4034610 con 0x7f8ba80d0a80 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 -- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80d0a80 msgr2=0x7f8ba80cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 --2- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80d0a80 0x7f8ba80cee70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009e50 tx=0x7f8ba402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 -- 172.21.15.122:0/238496453 shutdown_connections 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 --2- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80d0a80 0x7f8ba80cee70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 --2- 172.21.15.122:0/238496453 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d4760 0x7f8ba80b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 -- 172.21.15.122:0/238496453 >> 172.21.15.122:0/238496453 conn(0x7f8ba80d3c40 msgr2=0x7f8ba80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 -- 172.21.15.122:0/238496453 shutdown_connections 2024-03-06T07:28:15.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.198+0000 7f8bb0393640 1 -- 172.21.15.122:0/238496453 wait complete. 2024-03-06T07:28:15.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.200+0000 7f8bb0393640 1 Processor -- start 2024-03-06T07:28:15.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.200+0000 7f8bb0393640 1 -- start start 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.201+0000 7f8bb0393640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80b4170 0x7f8ba8157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.201+0000 7f8bb0393640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d0a80 0x7f8ba80b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.201+0000 7f8bb0393640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 0x7f8ba81594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.201+0000 7f8bb0393640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80b4dd0 con 0x7f8ba8158ff0 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.201+0000 7f8bb0393640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80b4550 con 0x7f8ba80b4170 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.201+0000 7f8bb0393640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80b4f50 con 0x7f8ba80d0a80 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 0x7f8ba81594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 0x7f8ba81594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34790/0 (socket says 172.21.15.122:34790) 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 -- 172.21.15.122:0/1455576589 learned_addr learned my addr 172.21.15.122:0/1455576589 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bad92e640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d0a80 0x7f8ba80b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae12f640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80b4170 0x7f8ba8157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 -- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d0a80 msgr2=0x7f8ba80b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d0a80 0x7f8ba80b9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 -- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80b4170 msgr2=0x7f8ba8157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80b4170 0x7f8ba8157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 -- 172.21.15.122:0/1455576589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba80b7dd0 con 0x7f8ba8158ff0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bae930640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 0x7f8ba81594a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4036040 tx=0x7f8ba4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4043020 con 0x7f8ba8158ff0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba402ce50 con 0x7f8ba8158ff0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba80b80b0 con 0x7f8ba8158ff0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4009510 con 0x7f8ba8158ff0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.202+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba80b8670 con 0x7f8ba8158ff0 2024-03-06T07:28:15.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.203+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba804f330 con 0x7f8ba8158ff0 2024-03-06T07:28:15.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.204+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ba4046020 con 0x7f8ba8158ff0 2024-03-06T07:28:15.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.205+0000 7f8b9effd640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b8c06b500 0x7f8b8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.205+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8ba40ac6f0 con 0x7f8ba8158ff0 2024-03-06T07:28:15.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.205+0000 7f8bae12f640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b8c06b500 0x7f8b8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.205+0000 7f8bae12f640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b8c06b500 0x7f8b8c06d9c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0007fd0 tx=0x7f8ba0007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.209+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ba404d030 con 0x7f8ba8158ff0 2024-03-06T07:28:15.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.440+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ba80733b0 con 0x7f8ba8158ff0 2024-03-06T07:28:15.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.440+0000 7f8b9effd640 1 -- 172.21.15.122:0/1455576589 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f8ba407af30 con 0x7f8ba8158ff0 2024-03-06T07:28:15.442 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:15.443 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:15.443 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:15.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b8c06b500 msgr2=0x7f8b8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b8c06b500 0x7f8b8c06d9c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0007fd0 tx=0x7f8ba0007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 msgr2=0x7f8ba81594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 0x7f8ba81594a0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4036040 tx=0x7f8ba4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 shutdown_connections 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8b8c06b500 0x7f8b8c06d9c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ba8158ff0 0x7f8ba81594a0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ba80d0a80 0x7f8ba80b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 --2- 172.21.15.122:0/1455576589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80b4170 0x7f8ba8157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 >> 172.21.15.122:0/1455576589 conn(0x7f8ba80d3c40 msgr2=0x7f8ba80b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 shutdown_connections 2024-03-06T07:28:15.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.442+0000 7f8bb0393640 1 -- 172.21.15.122:0/1455576589 wait complete. 2024-03-06T07:28:15.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:15.571 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:28:15.572 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:28:15.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.670+0000 7f91d8fac640 1 Processor -- start 2024-03-06T07:28:15.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.670+0000 7f91d8fac640 1 -- start start 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d8fac640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f91d40d06b0 0x7f91d4166db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d8fac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40d0020 con 0x7f91d415c320 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d8fac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40b0d20 con 0x7f91d40b40c0 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d8fac640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d40b1000 con 0x7f91d40d06b0 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d259c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f91d415c320 0x7f91d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49952/0 (socket says 172.21.15.122:49952) 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d259c640 1 -- 172.21.15.122:0/578506639 learned_addr learned my addr 172.21.15.122:0/578506639 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d2d9d640 1 --2- 172.21.15.122:0/578506639 >> v2:172.21.15.169:3300/0 conn(0x7f91d40d06b0 0x7f91d4166db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725052880 0 0) 0x7f91d40d0020 con 0x7f91d415c320 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.671+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc003520 con 0x7f91d415c320 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922663249 0 0) 0x7f91d40b0d20 con 0x7f91d40b40c0 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d40d0020 con 0x7f91d40b40c0 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1403101157 0 0) 0x7f91bc003520 con 0x7f91d415c320 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 >> v2:172.21.15.169:3300/0 conn(0x7f91d40d06b0 msgr2=0x7f91d4166db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 --2- 172.21.15.122:0/578506639 >> v2:172.21.15.169:3300/0 conn(0x7f91d40d06b0 0x7f91d4166db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 >> v1:172.21.15.169:6789/0 conn(0x7f91d40b40c0 legacy=0x7f91d40b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d40cfa60 con 0x7f91d415c320 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f91c4001670 con 0x7f91d415c320 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 >> v1:172.21.15.122:6789/0 conn(0x7f91d415c320 legacy=0x7f91d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91bc003b90 0x7f91bc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc004800 0x7f91bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91bc005280 0x7f91bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40b1000 con 0x7f91bc003b90 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc00c1c0 con 0x7f91bc004800 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc00c3c0 con 0x7f91bc005280 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d259c640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91bc003b90 0x7f91bc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d1d9b640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc004800 0x7f91bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d2d9d640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91bc005280 0x7f91bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d259c640 1 -- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91bc005280 msgr2=0x7f91bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d259c640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91bc005280 0x7f91bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d259c640 1 -- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc004800 msgr2=0x7f91bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d259c640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc004800 0x7f91bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d259c640 1 -- 172.21.15.122:0/578506639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc003520 con 0x7f91bc003b90 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d2d9d640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91bc005280 0x7f91bc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.672+0000 7f91d2d9d640 1 -- 172.21.15.122:0/578506639 reap_dead start 2024-03-06T07:28:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.673+0000 7f91d259c640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91bc003b90 0x7f91bc003f90 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f91c400c0e0 tx=0x7f91bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd6e8905921ee00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.673+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4007c40 con 0x7f91bc003b90 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.673+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c4030070 con 0x7f91bc003b90 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.673+0000 7f91d159a640 1 -- 172.21.15.122:0/578506639 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4005d90 con 0x7f91bc003b90 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 -- 172.21.15.122:0/578506639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91bc003b90 msgr2=0x7f91bc003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91bc003b90 0x7f91bc003f90 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f91c400c0e0 tx=0x7f91bc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 -- 172.21.15.122:0/578506639 shutdown_connections 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 --2- 172.21.15.122:0/578506639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91bc003b90 0x7f91bc003f90 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 -- 172.21.15.122:0/578506639 >> 172.21.15.122:0/578506639 conn(0x7f91d40b35a0 msgr2=0x7f91d40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 -- 172.21.15.122:0/578506639 shutdown_connections 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.674+0000 7f91d8fac640 1 -- 172.21.15.122:0/578506639 wait complete. 2024-03-06T07:28:15.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.675+0000 7f91d8fac640 1 Processor -- start 2024-03-06T07:28:15.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.675+0000 7f91d8fac640 1 -- start start 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc003b90 0x7f91d40c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91d40b7460 0x7f91d4151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 0x7f91d40c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40cda00 con 0x7f91d40c6840 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40cd880 con 0x7f91bc003b90 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40b1be0 con 0x7f91d40b7460 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 0x7f91d40c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 0x7f91d40c6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34806/0 (socket says 172.21.15.122:34806) 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 -- 172.21.15.122:0/1758992263 learned_addr learned my addr 172.21.15.122:0/1758992263 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d1d9b640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91d40b7460 0x7f91d4151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d259c640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc003b90 0x7f91d40c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 -- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91d40b7460 msgr2=0x7f91d4151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91d40b7460 0x7f91d4151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 -- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc003b90 msgr2=0x7f91d40c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc003b90 0x7f91d40c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 -- 172.21.15.122:0/1758992263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d40b78c0 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d2d9d640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 0x7f91d40c6cc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f91d40b4640 tx=0x7f91cc00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc016620 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc01b070 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d40c9260 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.676+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc0169e0 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.677+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d40c8f80 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.678+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d404f330 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.678+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91cc005d10 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.679+0000 7f91c2ffd640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f91b406b500 0x7f91b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.679+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f91cc08aa70 con 0x7f91d40c6840 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.679+0000 7f91d259c640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f91b406b500 0x7f91b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.679+0000 7f91d259c640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f91b406b500 0x7f91b406d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f91c4034000 tx=0x7f91c4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.683+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91cc0592a0 con 0x7f91d40c6840 2024-03-06T07:28:15.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.890+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91d40b3d90 con 0x7f91d40c6840 2024-03-06T07:28:15.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.890+0000 7f91c2ffd640 1 -- 172.21.15.122:0/1758992263 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f91cc05d340 con 0x7f91d40c6840 2024-03-06T07:28:15.892 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:15.893 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:15.893 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f91b406b500 msgr2=0x7f91b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f91b406b500 0x7f91b406d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f91c4034000 tx=0x7f91c4009030 comp rx=0 tx=0).stop 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 msgr2=0x7f91d40c6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 0x7f91d40c6cc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f91d40b4640 tx=0x7f91cc00f520 comp rx=0 tx=0).stop 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 shutdown_connections 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f91b406b500 0x7f91b406d9c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91d40c6840 0x7f91d40c6cc0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f91d40b7460 0x7f91d4151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 --2- 172.21.15.122:0/1758992263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91bc003b90 0x7f91d40c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 >> 172.21.15.122:0/1758992263 conn(0x7f91d40b35a0 msgr2=0x7f91d406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 shutdown_connections 2024-03-06T07:28:15.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:15.892+0000 7f91d8fac640 1 -- 172.21.15.122:0/1758992263 wait complete. 2024-03-06T07:28:15.905 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:16.059 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:28:16.059 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:16.060 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.101+0000 7fcba061ef40 1 Processor -- start 2024-03-06T07:28:16.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.101+0000 7fcba061ef40 1 -- start start 2024-03-06T07:28:16.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcba061ef40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b8922e0 0x55de7b801940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcba061ef40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b805ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcba061ef40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b895130 0x55de7b806410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcba061ef40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de7b764b70 con 0x55de7b892c00 2024-03-06T07:28:16.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcba061ef40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de7b768ad0 con 0x55de7b895130 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcba061ef40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de7b8944b0 con 0x55de7b8922e0 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b805ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b805ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34810/0 (socket says 172.21.15.122:34810) 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 -- 172.21.15.122:0/2089801787 learned_addr learned my addr 172.21.15.122:0/2089801787 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9e920640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b895130 0x55de7b806410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9e11f640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b8922e0 0x55de7b801940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 -- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b8922e0 msgr2=0x55de7b801940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b8922e0 0x55de7b801940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 -- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b895130 msgr2=0x55de7b806410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b895130 0x55de7b806410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 -- 172.21.15.122:0/2089801787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de7b804000 con 0x55de7b892c00 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9e920640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b895130 0x55de7b806410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9e11f640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b8922e0 0x55de7b801940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9d91e640 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b805ed0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fcb94058040 tx=0x7fcb9404f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a933d83870b7604 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9c91c640 1 -- 172.21.15.122:0/2089801787 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94066070 con 0x55de7b892c00 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9c91c640 1 -- 172.21.15.122:0/2089801787 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9404db90 con 0x55de7b892c00 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.102+0000 7fcb9c91c640 1 -- 172.21.15.122:0/2089801787 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9405a040 con 0x55de7b892c00 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 -- 172.21.15.122:0/2089801787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 msgr2=0x55de7b805ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b805ed0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fcb94058040 tx=0x7fcb9404f000 comp rx=0 tx=0).stop 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 -- 172.21.15.122:0/2089801787 shutdown_connections 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b895130 0x55de7b806410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b805ed0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 --2- 172.21.15.122:0/2089801787 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b8922e0 0x55de7b801940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 -- 172.21.15.122:0/2089801787 >> 172.21.15.122:0/2089801787 conn(0x55de7b7e9bd0 msgr2=0x55de7b7c0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 -- 172.21.15.122:0/2089801787 shutdown_connections 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 -- 172.21.15.122:0/2089801787 wait complete. 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 Processor -- start 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.103+0000 7fcba061ef40 1 -- start start 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcba061ef40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b8922e0 0x55de7b7ec530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcba061ef40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b88a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcba061ef40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b895130 0x55de7b877870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcba061ef40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de7b886ef0 con 0x55de7b892c00 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcba061ef40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de7b7ed870 con 0x55de7b8922e0 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcba061ef40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de7b8042a0 con 0x55de7b895130 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcb9d91e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b88a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcb9d91e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b88a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34822/0 (socket says 172.21.15.122:34822) 2024-03-06T07:28:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcb9d91e640 1 -- 172.21.15.122:0/168965222 learned_addr learned my addr 172.21.15.122:0/168965222 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcb9e11f640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b8922e0 0x55de7b7ec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcb9e920640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b895130 0x55de7b877870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.104+0000 7fcb9d91e640 1 -- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b895130 msgr2=0x55de7b877870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb9d91e640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b895130 0x55de7b877870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb9d91e640 1 -- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b8922e0 msgr2=0x55de7b7ec530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb9d91e640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de7b8922e0 0x55de7b7ec530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb9d91e640 1 -- 172.21.15.122:0/168965222 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de7b923c80 con 0x55de7b892c00 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb9e920640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x55de7b895130 0x55de7b877870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb9d91e640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b88a620 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcb94064040 tx=0x7fcb94002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb8f7fe640 1 -- 172.21.15.122:0/168965222 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94073070 con 0x55de7b892c00 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb8f7fe640 1 -- 172.21.15.122:0/168965222 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb94067040 con 0x55de7b892c00 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55de7b923f90 con 0x55de7b892c00 2024-03-06T07:28:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcb8f7fe640 1 -- 172.21.15.122:0/168965222 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94066050 con 0x55de7b892c00 2024-03-06T07:28:16.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.105+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de7b7eca70 con 0x55de7b892c00 2024-03-06T07:28:16.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcb8f7fe640 1 -- 172.21.15.122:0/168965222 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb94076070 con 0x55de7b892c00 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcb8f7fe640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb80069eb0 0x7fcb8006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcb8f7fe640 1 -- 172.21.15.122:0/168965222 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcb940ce190 con 0x55de7b892c00 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcb9e11f640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb80069eb0 0x7fcb8006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcba061ef40 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55de7b88ab60 0x55de7b877db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55de7b7bbbf0 con 0x55de7b88ab60 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.107+0000 7fcb9e920640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55de7b88ab60 0x55de7b877db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcb9e11f640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb80069eb0 0x7fcb8006c370 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e4c0 tx=0x7fcb98003680 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcb9e920640 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55de7b88ab60 0x55de7b877db0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcb9e920640 1 -- 172.21.15.122:0/168965222 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fcb90001850 con 0x55de7b88ab60 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55de7b88ab60 msgr2=0x55de7b877db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x55de7b88ab60 0x55de7b877db0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb80069eb0 msgr2=0x7fcb8006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb80069eb0 0x7fcb8006c370 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e4c0 tx=0x7fcb98003680 comp rx=0 tx=0).stop 2024-03-06T07:28:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 msgr2=0x55de7b88a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 --2- 172.21.15.122:0/168965222 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de7b892c00 0x55de7b88a620 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcb94064040 tx=0x7fcb94002fb0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcb9e920640 1 -- 172.21.15.122:0/168965222 reap_dead start 2024-03-06T07:28:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 shutdown_connections 2024-03-06T07:28:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 >> 172.21.15.122:0/168965222 conn(0x55de7b7e9bd0 msgr2=0x55de7b7c0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 shutdown_connections 2024-03-06T07:28:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.108+0000 7fcba061ef40 1 -- 172.21.15.122:0/168965222 wait complete. 2024-03-06T07:28:16.117 INFO:teuthology.orchestra.run.smithi122.stdout:W�file_ddir_2* 2024-03-06T07:28:16.117 DEBUG:teuthology.orchestra.run.smithi122:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:16.204 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:28:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 1099511627777, 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "ancestors": [ 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1099511628278, 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "file_d", 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 25 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "dirino": 1, 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "dname": "dir_2", 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "version": 42 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "pool": 17, 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "old_pools": [] 2024-03-06T07:28:16.205 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:28:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:16.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:16.206 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:28:16.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:16.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.401+0000 7f13d8caf640 1 Processor -- start 2024-03-06T07:28:16.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.401+0000 7f13d8caf640 1 -- start start 2024-03-06T07:28:16.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d8caf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f13d415dfe0 0x7f13d40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d8caf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13d40cf830 0x7f13d40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d8caf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d40b3430 con 0x7f13d40d27f0 2024-03-06T07:28:16.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d8caf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d40b31e0 con 0x7f13d40cf830 2024-03-06T07:28:16.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d8caf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d40b3d60 con 0x7f13d415dfe0 2024-03-06T07:28:16.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13d40cf830 0x7f13d40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13d40cf830 0x7f13d40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34832/0 (socket says 172.21.15.122:34832) 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 -- 172.21.15.122:0/1948309115 learned_addr learned my addr 172.21.15.122:0/1948309115 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d259c640 1 --2- 172.21.15.122:0/1948309115 >> v2:172.21.15.169:3300/0 conn(0x7f13d415dfe0 0x7f13d40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 -- 172.21.15.122:0/1948309115 >> v2:172.21.15.169:3300/0 conn(0x7f13d415dfe0 msgr2=0x7f13d40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 --2- 172.21.15.122:0/1948309115 >> v2:172.21.15.169:3300/0 conn(0x7f13d415dfe0 0x7f13d40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 -- 172.21.15.122:0/1948309115 >> v1:172.21.15.169:6789/0 conn(0x7f13d40d27f0 legacy=0x7f13d40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.402+0000 7f13d1d9b640 1 -- 172.21.15.122:0/1948309115 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d40b3710 con 0x7f13d40cf830 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d1d9b640 1 --2- 172.21.15.122:0/1948309115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d40cf830 0x7f13d40d22b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009870 tx=0x7f13bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f972b9abd03b055 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d159a640 1 -- 172.21.15.122:0/1948309115 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc034070 con 0x7f13d40cf830 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d159a640 1 -- 172.21.15.122:0/1948309115 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13bc030860 con 0x7f13d40cf830 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d159a640 1 -- 172.21.15.122:0/1948309115 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc030c30 con 0x7f13d40cf830 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 -- 172.21.15.122:0/1948309115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d40cf830 msgr2=0x7f13d40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 --2- 172.21.15.122:0/1948309115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d40cf830 0x7f13d40d22b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009870 tx=0x7f13bc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 -- 172.21.15.122:0/1948309115 shutdown_connections 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 --2- 172.21.15.122:0/1948309115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d40cf830 0x7f13d40d22b0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 --2- 172.21.15.122:0/1948309115 >> v2:172.21.15.169:3300/0 conn(0x7f13d415dfe0 0x7f13d40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 -- 172.21.15.122:0/1948309115 >> 172.21.15.122:0/1948309115 conn(0x7f13d40d3cd0 msgr2=0x7f13d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 -- 172.21.15.122:0/1948309115 shutdown_connections 2024-03-06T07:28:16.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.404+0000 7f13d8caf640 1 -- 172.21.15.122:0/1948309115 wait complete. 2024-03-06T07:28:16.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 Processor -- start 2024-03-06T07:28:16.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 -- start start 2024-03-06T07:28:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d40cf830 0x7f13d40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d40d27f0 0x7f13d40b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 0x7f13d40b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40d1df0 con 0x7f13d415dfe0 2024-03-06T07:28:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40d1c70 con 0x7f13d40d27f0 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.406+0000 7f13d8caf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d415e3c0 con 0x7f13d40cf830 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 0x7f13d40b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 0x7f13d40b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34846/0 (socket says 172.21.15.122:34846) 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 -- 172.21.15.122:0/4131190516 learned_addr learned my addr 172.21.15.122:0/4131190516 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d1d9b640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d40d27f0 0x7f13d40b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d259c640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d40cf830 0x7f13d40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 -- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d40cf830 msgr2=0x7f13d40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d40cf830 0x7f13d40cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 -- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d40d27f0 msgr2=0x7f13d40b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d40d27f0 0x7f13d40b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 -- 172.21.15.122:0/4131190516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d40cc330 con 0x7f13d415dfe0 2024-03-06T07:28:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d259c640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d40cf830 0x7f13d40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d2d9d640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 0x7f13d40b6de0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00c400 tx=0x7f13cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc017070 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc0052d0 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d40cc5b0 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc003810 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.407+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d40ccb70 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.408+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d404f330 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.409+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13cc01a070 con 0x7f13d415dfe0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.409+0000 7f13caffd640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13b406b4d0 0x7f13b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.409+0000 7f13d259c640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13b406b4d0 0x7f13b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.409+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f13cc089550 con 0x7f13d415dfe0 2024-03-06T07:28:16.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.410+0000 7f13d259c640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13b406b4d0 0x7f13b406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009b20 tx=0x7f13c40039d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.414+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13d40cc330 con 0x7f13d415dfe0 2024-03-06T07:28:16.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.676+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13d415d220 con 0x7f13d415dfe0 2024-03-06T07:28:16.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.677+0000 7f13caffd640 1 -- 172.21.15.122:0/4131190516 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f13cc057cf0 con 0x7f13d415dfe0 2024-03-06T07:28:16.679 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:16.679 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:16.680 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:16.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13b406b4d0 msgr2=0x7f13b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13b406b4d0 0x7f13b406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009b20 tx=0x7f13c40039d0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 msgr2=0x7f13d40b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 0x7f13d40b6de0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00c400 tx=0x7f13cc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 shutdown_connections 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13b406b4d0 0x7f13b406d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d415dfe0 0x7f13d40b6de0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d40d27f0 0x7f13d40b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 --2- 172.21.15.122:0/4131190516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d40cf830 0x7f13d40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 >> 172.21.15.122:0/4131190516 conn(0x7f13d40d3cd0 msgr2=0x7f13d40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 shutdown_connections 2024-03-06T07:28:16.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:16.679+0000 7f13d8caf640 1 -- 172.21.15.122:0/4131190516 wait complete. 2024-03-06T07:28:16.693 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:16.811 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:16.811 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:28:21.812 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.915+0000 7f2d7894b640 1 Processor -- start 2024-03-06T07:28:21.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.915+0000 7f2d7894b640 1 -- start start 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7894b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2d740cdeb0 0x7f2d7415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7894b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d74071f40 con 0x7f2d740cce40 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7894b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d74070a70 con 0x7f2d741515e0 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7894b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d740cd2f0 con 0x7f2d740cdeb0 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d71d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2d740cce40 0x7f2d740cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49970/0 (socket says 172.21.15.122:49970) 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d71d9b640 1 -- 172.21.15.122:0/643293910 learned_addr learned my addr 172.21.15.122:0/643293910 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008513315 0 0) 0x7f2d74071f40 con 0x7f2d740cce40 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d60003520 con 0x7f2d740cce40 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d72d9d640 1 --2- 172.21.15.122:0/643293910 >> v2:172.21.15.169:3300/0 conn(0x7f2d740cdeb0 0x7f2d7415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1017933490 0 0) 0x7f2d60003520 con 0x7f2d740cce40 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 >> v2:172.21.15.169:3300/0 conn(0x7f2d740cdeb0 msgr2=0x7f2d7415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 --2- 172.21.15.122:0/643293910 >> v2:172.21.15.169:3300/0 conn(0x7f2d740cdeb0 0x7f2d7415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 >> v1:172.21.15.169:6789/0 conn(0x7f2d741515e0 legacy=0x7f2d74071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74070d50 con 0x7f2d740cce40 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.917+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f2d5c001690 con 0x7f2d740cce40 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 >> v1:172.21.15.122:6789/0 conn(0x7f2d740cce40 legacy=0x7f2d740cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d60004800 0x7f2d60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d600051c0 0x7f2d6000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740cd2f0 con 0x7f2d600039b0 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000e320 con 0x7f2d600051c0 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000e520 con 0x7f2d60004800 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7259c640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d72d9d640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d600051c0 0x7f2d6000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d71d9b640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d60004800 0x7f2d60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7259c640 1 -- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d60004800 msgr2=0x7f2d60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7259c640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d60004800 0x7f2d60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7259c640 1 -- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d600051c0 msgr2=0x7f2d6000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7259c640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d600051c0 0x7f2d6000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d7259c640 1 -- 172.21.15.122:0/643293910 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d60003520 con 0x7f2d600039b0 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d72d9d640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d600051c0 0x7f2d6000dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d71d9b640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d60004800 0x7f2d60004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.918+0000 7f2d72d9d640 1 -- 172.21.15.122:0/643293910 reap_dead start 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7259c640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d60003dd0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c001e50 tx=0x7f2d60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5d308836c8b16a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c001a70 con 0x7f2d600039b0 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c003b70 con 0x7f2d600039b0 2024-03-06T07:28:21.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7159a640 1 -- 172.21.15.122:0/643293910 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0050d0 con 0x7f2d600039b0 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7894b640 1 -- 172.21.15.122:0/643293910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 msgr2=0x7f2d60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7894b640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d60003dd0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c001e50 tx=0x7f2d60005830 comp rx=0 tx=0).stop 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7894b640 1 -- 172.21.15.122:0/643293910 shutdown_connections 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7894b640 1 --2- 172.21.15.122:0/643293910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d60003dd0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.919+0000 7f2d7894b640 1 -- 172.21.15.122:0/643293910 >> 172.21.15.122:0/643293910 conn(0x7f2d740cf8a0 msgr2=0x7f2d740cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.920+0000 7f2d7894b640 1 -- 172.21.15.122:0/643293910 shutdown_connections 2024-03-06T07:28:21.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.920+0000 7f2d7894b640 1 -- 172.21.15.122:0/643293910 wait complete. 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.921+0000 7f2d7894b640 1 Processor -- start 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.922+0000 7f2d7894b640 1 -- start start 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7894b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d741550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7894b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d74151a20 0x7f2d741412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7894b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d740717c0 0x7f2d74071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7894b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7415b430 con 0x7f2d600039b0 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7894b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74071310 con 0x7f2d740717c0 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7894b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7415b5b0 con 0x7f2d74151a20 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d741550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d741550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34856/0 (socket says 172.21.15.122:34856) 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 -- 172.21.15.122:0/711256921 learned_addr learned my addr 172.21.15.122:0/711256921 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d71d9b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d74151a20 0x7f2d741412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d72d9d640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d740717c0 0x7f2d74071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 -- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d74151a20 msgr2=0x7f2d741412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d74151a20 0x7f2d741412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 -- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d740717c0 msgr2=0x7f2d74071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d740717c0 0x7f2d74071c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d7259c640 1 -- 172.21.15.122:0/711256921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74151ea0 con 0x7f2d600039b0 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d72d9d640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d740717c0 0x7f2d74071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.923+0000 7f2d71d9b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d74151a20 0x7f2d741412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.924+0000 7f2d7259c640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d741550d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c001f70 tx=0x7f2d6c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.924+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c003970 con 0x7f2d600039b0 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.924+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c006b50 con 0x7f2d600039b0 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.924+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d74141800 con 0x7f2d600039b0 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.924+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c007030 con 0x7f2d600039b0 2024-03-06T07:28:21.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.924+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d74152120 con 0x7f2d600039b0 2024-03-06T07:28:21.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.925+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d74068840 con 0x7f2d600039b0 2024-03-06T07:28:21.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.925+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d6c0071d0 con 0x7f2d600039b0 2024-03-06T07:28:21.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.925+0000 7f2d6affd640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5406b4d0 0x7f2d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.926+0000 7f2d71d9b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5406b4d0 0x7f2d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.926+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2d6c0abee0 con 0x7f2d600039b0 2024-03-06T07:28:21.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.926+0000 7f2d71d9b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5406b4d0 0x7f2d5406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c001eb0 tx=0x7f2d5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:21.930+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d74151ea0 con 0x7f2d600039b0 2024-03-06T07:28:22.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.148+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d740b48b0 con 0x7f2d600039b0 2024-03-06T07:28:22.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.148+0000 7f2d6affd640 1 -- 172.21.15.122:0/711256921 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f2d740b48b0 con 0x7f2d600039b0 2024-03-06T07:28:22.150 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:22.150 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:22.151 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5406b4d0 msgr2=0x7f2d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5406b4d0 0x7f2d5406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c001eb0 tx=0x7f2d5c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 msgr2=0x7f2d741550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d741550d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c001f70 tx=0x7f2d6c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 shutdown_connections 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5406b4d0 0x7f2d5406d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d740717c0 0x7f2d74071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d74151a20 0x7f2d741412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 --2- 172.21.15.122:0/711256921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d600039b0 0x7f2d741550d0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 >> 172.21.15.122:0/711256921 conn(0x7f2d740cf8a0 msgr2=0x7f2d7414c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 shutdown_connections 2024-03-06T07:28:22.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.150+0000 7f2d7894b640 1 -- 172.21.15.122:0/711256921 wait complete. 2024-03-06T07:28:22.163 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:22.282 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:22.282 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:22.282 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:22.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.383+0000 7f9876513640 1 Processor -- start 2024-03-06T07:28:22.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.383+0000 7f9876513640 1 -- start start 2024-03-06T07:28:22.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f9876513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 0x7f987015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f9876513640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f987006b600 0x7f9870162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f9876513640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987015ea30 con 0x7f98700ce920 2024-03-06T07:28:22.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f9876513640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9870161b70 con 0x7f987006b600 2024-03-06T07:28:22.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f9876513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987015ed10 con 0x7f987006b220 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f9874ab0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f987006b600 0x7f9870162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f986ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 0x7f987015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f986f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f98700ce920 0x7f9870071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:46002/0 (socket says 172.21.15.122:46002) 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f986ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 0x7f987015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:45806/0 (socket says 172.21.15.122:45806) 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.384+0000 7f986f7fe640 1 -- 172.21.15.122:0/2917304671 learned_addr learned my addr 172.21.15.122:0/2917304671 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.385+0000 7f986ffff640 1 -- 172.21.15.122:0/2917304671 >> v2:172.21.15.169:3300/0 conn(0x7f987006b600 msgr2=0x7f9870162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.385+0000 7f986ffff640 1 --2- 172.21.15.122:0/2917304671 >> v2:172.21.15.169:3300/0 conn(0x7f987006b600 0x7f9870162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.385+0000 7f986ffff640 1 -- 172.21.15.122:0/2917304671 >> v1:172.21.15.169:6789/0 conn(0x7f98700ce920 legacy=0x7f9870071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.385+0000 7f986ffff640 1 -- 172.21.15.122:0/2917304671 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987015eff0 con 0x7f987006b220 2024-03-06T07:28:22.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.385+0000 7f986effd640 1 -- 172.21.15.122:0/2917304671 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490796975 0 0) 0x7f987015ea30 con 0x7f98700ce920 2024-03-06T07:28:22.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.385+0000 7f9874ab0640 1 --2- 172.21.15.122:0/2917304671 >> v2:172.21.15.169:3300/0 conn(0x7f987006b600 0x7f9870162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.386+0000 7f986ffff640 1 --2- 172.21.15.122:0/2917304671 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 0x7f987015e450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98580097a0 tx=0x7f985802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6e019f80a4de56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.386+0000 7f986effd640 1 -- 172.21.15.122:0/2917304671 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858035020 con 0x7f987006b220 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.386+0000 7f986effd640 1 -- 172.21.15.122:0/2917304671 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9858031830 con 0x7f987006b220 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.386+0000 7f986effd640 1 -- 172.21.15.122:0/2917304671 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858031c00 con 0x7f987006b220 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 -- 172.21.15.122:0/2917304671 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 msgr2=0x7f987015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 --2- 172.21.15.122:0/2917304671 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 0x7f987015e450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98580097a0 tx=0x7f985802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 -- 172.21.15.122:0/2917304671 shutdown_connections 2024-03-06T07:28:22.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 --2- 172.21.15.122:0/2917304671 >> v2:172.21.15.169:3300/0 conn(0x7f987006b600 0x7f9870162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 --2- 172.21.15.122:0/2917304671 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f987006b220 0x7f987015e450 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 -- 172.21.15.122:0/2917304671 >> 172.21.15.122:0/2917304671 conn(0x7f987006c0c0 msgr2=0x7f987006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 -- 172.21.15.122:0/2917304671 shutdown_connections 2024-03-06T07:28:22.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.387+0000 7f9876513640 1 -- 172.21.15.122:0/2917304671 wait complete. 2024-03-06T07:28:22.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.388+0000 7f9876513640 1 Processor -- start 2024-03-06T07:28:22.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.389+0000 7f9876513640 1 -- start start 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9876513640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987006b220 0x7f98701f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9876513640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 0x7f98701f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9876513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98700ce920 0x7f98701f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9876513640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9870072090 con 0x7f987006b600 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9876513640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700ce200 con 0x7f987006b220 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9876513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9870157260 con 0x7f98700ce920 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 0x7f98701f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 0x7f98701f77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34870/0 (socket says 172.21.15.122:34870) 2024-03-06T07:28:22.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 -- 172.21.15.122:0/1980524251 learned_addr learned my addr 172.21.15.122:0/1980524251 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986ffff640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987006b220 0x7f98701f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f9874ab0640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98700ce920 0x7f98701f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 -- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98700ce920 msgr2=0x7f98701f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98700ce920 0x7f98701f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 -- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987006b220 msgr2=0x7f98701f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987006b220 0x7f98701f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 -- 172.21.15.122:0/1980524251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98701e8410 con 0x7f987006b600 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986ffff640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987006b220 0x7f98701f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.390+0000 7f986f7fe640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 0x7f98701f77e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f986000c460 tx=0x7f986000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.391+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98600074e0 con 0x7f987006b600 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.391+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98600047e0 con 0x7f987006b600 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.391+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98701f40b0 con 0x7f987006b600 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.391+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860004bd0 con 0x7f987006b600 2024-03-06T07:28:22.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.391+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98701e8990 con 0x7f987006b600 2024-03-06T07:28:22.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.392+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987006be60 con 0x7f987006b600 2024-03-06T07:28:22.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.392+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9860011070 con 0x7f987006b600 2024-03-06T07:28:22.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.392+0000 7f986cff9640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.393+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9860089dc0 con 0x7f987006b600 2024-03-06T07:28:22.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.393+0000 7f986ffff640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.393+0000 7f986ffff640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9858036000 tx=0x7f9858008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.397+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f987006be60 con 0x7f987006b600 2024-03-06T07:28:22.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.612+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f987004f330 con 0x7f987006b600 2024-03-06T07:28:22.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.612+0000 7f986cff9640 1 -- 172.21.15.122:0/1980524251 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f986000e280 con 0x7f987006b600 2024-03-06T07:28:22.614 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:22.614 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:22.615 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f985006b4d0 msgr2=0x7f985006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9858036000 tx=0x7f9858008030 comp rx=0 tx=0).stop 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 msgr2=0x7f98701f77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 0x7f98701f77e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f986000c460 tx=0x7f986000c860 comp rx=0 tx=0).stop 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 shutdown_connections 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.614+0000 7f9876513640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98700ce920 0x7f98701f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.615+0000 7f9876513640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f987006b600 0x7f98701f77e0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.615+0000 7f9876513640 1 --2- 172.21.15.122:0/1980524251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987006b220 0x7f98701f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.615+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 >> 172.21.15.122:0/1980524251 conn(0x7f987006c0c0 msgr2=0x7f9870073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.615+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 shutdown_connections 2024-03-06T07:28:22.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.615+0000 7f9876513640 1 -- 172.21.15.122:0/1980524251 wait complete. 2024-03-06T07:28:22.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:28:22.744 INFO:teuthology.orchestra.run.smithi122.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:28:22.744 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:28:22.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:22.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.843+0000 7f60d674b640 1 Processor -- start 2024-03-06T07:28:22.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.843+0000 7f60d674b640 1 -- start start 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.844+0000 7f60d674b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60d015dfe0 0x7f60d00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.844+0000 7f60d674b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f60d00cf830 0x7f60d00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60d674b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d00d27f0 0x7f60d00d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60d674b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d00b31e0 con 0x7f60d015dfe0 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60d674b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d00b3d60 con 0x7f60d00cf830 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60d674b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00b3430 con 0x7f60d00d27f0 2024-03-06T07:28:22.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60d015dfe0 0x7f60d00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60d015dfe0 0x7f60d00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34872/0 (socket says 172.21.15.122:34872) 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 -- 172.21.15.122:0/3861251834 learned_addr learned my addr 172.21.15.122:0/3861251834 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60d4ce8640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d00d27f0 0x7f60d00d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cf7fe640 1 --2- 172.21.15.122:0/3861251834 >> v2:172.21.15.169:3300/0 conn(0x7f60d00cf830 0x7f60d00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 -- 172.21.15.122:0/3861251834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d00d27f0 msgr2=0x7f60d00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d00d27f0 0x7f60d00d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 -- 172.21.15.122:0/3861251834 >> v2:172.21.15.169:3300/0 conn(0x7f60d00cf830 msgr2=0x7f60d00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 --2- 172.21.15.122:0/3861251834 >> v2:172.21.15.169:3300/0 conn(0x7f60d00cf830 0x7f60d00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cffff640 1 -- 172.21.15.122:0/3861251834 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d00b3710 con 0x7f60d015dfe0 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60d4ce8640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d00d27f0 0x7f60d00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.845+0000 7f60cf7fe640 1 --2- 172.21.15.122:0/3861251834 >> v2:172.21.15.169:3300/0 conn(0x7f60d00cf830 0x7f60d00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.846+0000 7f60cffff640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d015dfe0 0x7f60d00cf2f0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f60d00d16b0 tx=0x7f60c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7e476c510ff860 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.846+0000 7f60ceffd640 1 -- 172.21.15.122:0/3861251834 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0035020 con 0x7f60d015dfe0 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.846+0000 7f60ceffd640 1 -- 172.21.15.122:0/3861251834 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c0031870 con 0x7f60d015dfe0 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.846+0000 7f60ceffd640 1 -- 172.21.15.122:0/3861251834 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0031c40 con 0x7f60d015dfe0 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 -- 172.21.15.122:0/3861251834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d015dfe0 msgr2=0x7f60d00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d015dfe0 0x7f60d00cf2f0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f60d00d16b0 tx=0x7f60c002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 -- 172.21.15.122:0/3861251834 shutdown_connections 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d00d27f0 0x7f60d00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 --2- 172.21.15.122:0/3861251834 >> v2:172.21.15.169:3300/0 conn(0x7f60d00cf830 0x7f60d00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 --2- 172.21.15.122:0/3861251834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d015dfe0 0x7f60d00cf2f0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 -- 172.21.15.122:0/3861251834 >> 172.21.15.122:0/3861251834 conn(0x7f60d00d3cd0 msgr2=0x7f60d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 -- 172.21.15.122:0/3861251834 shutdown_connections 2024-03-06T07:28:22.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.847+0000 7f60d674b640 1 -- 172.21.15.122:0/3861251834 wait complete. 2024-03-06T07:28:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.848+0000 7f60d674b640 1 Processor -- start 2024-03-06T07:28:22.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.848+0000 7f60d674b640 1 -- start start 2024-03-06T07:28:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60d674b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 0x7f60d015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60d674b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d00d27f0 0x7f60d01f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60d674b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d015dfe0 0x7f60d00b8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60d674b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00d1df0 con 0x7f60d00cf830 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60d674b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00d1c70 con 0x7f60d00d27f0 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60d674b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d015e3c0 con 0x7f60d015dfe0 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 0x7f60d015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 0x7f60d015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34882/0 (socket says 172.21.15.122:34882) 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 -- 172.21.15.122:0/178910440 learned_addr learned my addr 172.21.15.122:0/178910440 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 -- 172.21.15.122:0/178910440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d015dfe0 msgr2=0x7f60d00b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d015dfe0 0x7f60d00b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 -- 172.21.15.122:0/178910440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d00d27f0 msgr2=0x7f60d01f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d00d27f0 0x7f60d01f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 -- 172.21.15.122:0/178910440 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d01f7400 con 0x7f60d00cf830 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60cffff640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 0x7f60d015c280 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0035040 con 0x7f60d00cf830 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c0045480 con 0x7f60d00cf830 2024-03-06T07:28:22.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.849+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0045870 con 0x7f60d00cf830 2024-03-06T07:28:22.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.850+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d01f76e0 con 0x7f60d00cf830 2024-03-06T07:28:22.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.850+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d00cb220 con 0x7f60d00cf830 2024-03-06T07:28:22.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.851+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d004f330 con 0x7f60d00cf830 2024-03-06T07:28:22.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.851+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60c0038020 con 0x7f60d00cf830 2024-03-06T07:28:22.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.852+0000 7f60ccff9640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60bc06b4d0 0x7f60bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.852+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f60c00ac670 con 0x7f60d00cf830 2024-03-06T07:28:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.852+0000 7f60cf7fe640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60bc06b4d0 0x7f60bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.852+0000 7f60cf7fe640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60bc06b4d0 0x7f60bc06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f60d01f6980 tx=0x7f60c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:22.856+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60c007b4f0 con 0x7f60d00cf830 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.066+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d00747a0 con 0x7f60d00cf830 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.066+0000 7f60ccff9640 1 -- 172.21.15.122:0/178910440 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f60c007adf0 con 0x7f60d00cf830 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:23.069 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:23.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60bc06b4d0 msgr2=0x7f60bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60bc06b4d0 0x7f60bc06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f60d01f6980 tx=0x7f60c4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 msgr2=0x7f60d015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 0x7f60d015c280 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 shutdown_connections 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60bc06b4d0 0x7f60bc06d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60d015dfe0 0x7f60d00b8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d00d27f0 0x7f60d01f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 --2- 172.21.15.122:0/178910440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60d00cf830 0x7f60d015c280 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 >> 172.21.15.122:0/178910440 conn(0x7f60d00d3cd0 msgr2=0x7f60d006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 shutdown_connections 2024-03-06T07:28:23.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.068+0000 7f60d674b640 1 -- 172.21.15.122:0/178910440 wait complete. 2024-03-06T07:28:23.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2024-03-06T07:28:23.201 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:28:23.201 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:28:23.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.312+0000 7fcc233ac640 1 Processor -- start 2024-03-06T07:28:23.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc233ac640 1 -- start start 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc233ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcc1c06b220 0x7fcc1c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc233ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b600 0x7fcc1c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc233ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c15ea30 con 0x7fcc1c0ce920 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc233ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c161b70 con 0x7fcc1c06b220 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc233ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c15ed10 con 0x7fcc1c06b600 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc21148640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcc1c06b220 0x7fcc1c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc20947640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcc1c0ce920 0x7fcc1c071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49984/0 (socket says 172.21.15.122:49984) 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.313+0000 7fcc20947640 1 -- 172.21.15.122:0/3612592875 learned_addr learned my addr 172.21.15.122:0/3612592875 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc21949640 1 --2- 172.21.15.122:0/3612592875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b600 0x7fcc1c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc21148640 1 -- 172.21.15.122:0/3612592875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b600 msgr2=0x7fcc1c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc21148640 1 --2- 172.21.15.122:0/3612592875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b600 0x7fcc1c162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc21148640 1 -- 172.21.15.122:0/3612592875 >> v1:172.21.15.122:6789/0 conn(0x7fcc1c0ce920 legacy=0x7fcc1c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc21148640 1 -- 172.21.15.122:0/3612592875 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c15eff0 con 0x7fcc1c06b220 2024-03-06T07:28:23.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc1bfff640 1 -- 172.21.15.122:0/3612592875 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281694854 0 0) 0x7fcc1c15ea30 con 0x7fcc1c0ce920 2024-03-06T07:28:23.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.314+0000 7fcc21148640 1 --2- 172.21.15.122:0/3612592875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b220 0x7fcc1c15e450 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40917a6a1bb2e978 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc1bfff640 1 -- 172.21.15.122:0/3612592875 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c039070 con 0x7fcc1c06b220 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc1bfff640 1 -- 172.21.15.122:0/3612592875 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c02cbc0 con 0x7fcc1c06b220 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc1bfff640 1 -- 172.21.15.122:0/3612592875 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c034440 con 0x7fcc1c06b220 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 -- 172.21.15.122:0/3612592875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b220 msgr2=0x7fcc1c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 --2- 172.21.15.122:0/3612592875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b220 0x7fcc1c15e450 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02be10 comp rx=0 tx=0).stop 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 -- 172.21.15.122:0/3612592875 shutdown_connections 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 --2- 172.21.15.122:0/3612592875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b600 0x7fcc1c162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 --2- 172.21.15.122:0/3612592875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b220 0x7fcc1c15e450 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 -- 172.21.15.122:0/3612592875 >> 172.21.15.122:0/3612592875 conn(0x7fcc1c06c0c0 msgr2=0x7fcc1c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 -- 172.21.15.122:0/3612592875 shutdown_connections 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.315+0000 7fcc233ac640 1 -- 172.21.15.122:0/3612592875 wait complete. 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.316+0000 7fcc233ac640 1 Processor -- start 2024-03-06T07:28:23.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 -- start start 2024-03-06T07:28:23.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b220 0x7fcc1c1f8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 0x7fcc1c14efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc1c0ce920 0x7fcc1c1605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c072090 con 0x7fcc1c06b600 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0ce200 con 0x7fcc1c0ce920 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc233ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0aeaa0 con 0x7fcc1c06b220 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc20947640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 0x7fcc1c14efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc20947640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 0x7fcc1c14efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34904/0 (socket says 172.21.15.122:34904) 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc20947640 1 -- 172.21.15.122:0/2162602859 learned_addr learned my addr 172.21.15.122:0/2162602859 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc21148640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b220 0x7fcc1c1f8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.317+0000 7fcc21949640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc1c0ce920 0x7fcc1c1605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc20947640 1 -- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b220 msgr2=0x7fcc1c1f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc20947640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b220 0x7fcc1c1f8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc20947640 1 -- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc1c0ce920 msgr2=0x7fcc1c1605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc20947640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc1c0ce920 0x7fcc1c1605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc20947640 1 -- 172.21.15.122:0/2162602859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c1f8790 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc20947640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 0x7fcc1c14efb0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c4a0 tx=0x7fcc1000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10017070 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc10005ce0 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c0b1680 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc100072b0 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.318+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c0aecf0 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.319+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c04f330 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.320+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc1001a070 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.320+0000 7fcc19ffb640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc0806b4d0 0x7fcc0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.320+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcc100894d0 con 0x7fcc1c06b600 2024-03-06T07:28:23.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.320+0000 7fcc21148640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc0806b4d0 0x7fcc0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.321+0000 7fcc21148640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc0806b4d0 0x7fcc0806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036000 tx=0x7fcc0c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.325+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc1c1f8790 con 0x7fcc1c06b600 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.550+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc1c06f850 con 0x7fcc1c06b600 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.551+0000 7fcc19ffb640 1 -- 172.21.15.122:0/2162602859 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fcc10057d00 con 0x7fcc1c06b600 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc0806b4d0 msgr2=0x7fcc0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc0806b4d0 0x7fcc0806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036000 tx=0x7fcc0c030000 comp rx=0 tx=0).stop 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 msgr2=0x7fcc1c14efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 0x7fcc1c14efb0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c4a0 tx=0x7fcc1000c960 comp rx=0 tx=0).stop 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 shutdown_connections 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcc0806b4d0 0x7fcc0806d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc1c0ce920 0x7fcc1c1605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc1c06b600 0x7fcc1c14efb0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 --2- 172.21.15.122:0/2162602859 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcc1c06b220 0x7fcc1c1f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 >> 172.21.15.122:0/2162602859 conn(0x7fcc1c06c0c0 msgr2=0x7fcc1c06ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 shutdown_connections 2024-03-06T07:28:23.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.553+0000 7fcc233ac640 1 -- 172.21.15.122:0/2162602859 wait complete. 2024-03-06T07:28:23.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2024-03-06T07:28:23.705 INFO:teuthology.orchestra.run.smithi122.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:28:23.705 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:28:23.706 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.744+0000 7f9f07ab8f40 1 Processor -- start 2024-03-06T07:28:23.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.744+0000 7f9f07ab8f40 1 -- start start 2024-03-06T07:28:23.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f07ab8f40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399e8c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f07ab8f40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399e32f70 0x562399e2aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f07ab8f40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 0x562399de86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f07ab8f40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562399d90b70 con 0x562399ea8690 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f07ab8f40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562399e19b60 con 0x562399e32f70 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f07ab8f40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x562399ea1c60 con 0x562399deb810 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f05dba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 0x562399de86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f05dba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 0x562399de86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34924/0 (socket says 172.21.15.122:34924) 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f05dba640 1 -- 172.21.15.122:0/1597477692 learned_addr learned my addr 172.21.15.122:0/1597477692 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9f055b9640 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399e8c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.745+0000 7f9efffff640 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399e32f70 0x562399e2aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f05dba640 1 -- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 msgr2=0x562399e8c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f05dba640 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399e8c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f05dba640 1 -- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399e32f70 msgr2=0x562399e2aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f05dba640 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399e32f70 0x562399e2aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f05dba640 1 -- 172.21.15.122:0/1597477692 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562399ea5ec0 con 0x562399ea8690 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f05dba640 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 0x562399de86b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9f0001f5b0 tx=0x7f9f0004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefbfa1d05d6eaa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9eff7fe640 1 -- 172.21.15.122:0/1597477692 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0005a040 con 0x562399ea8690 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9eff7fe640 1 -- 172.21.15.122:0/1597477692 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f00064070 con 0x562399ea8690 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9eff7fe640 1 -- 172.21.15.122:0/1597477692 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0004dd00 con 0x562399ea8690 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/1597477692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 msgr2=0x562399de86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 0x562399de86b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9f0001f5b0 tx=0x7f9f0004f000 comp rx=0 tx=0).stop 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/1597477692 shutdown_connections 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399ea8690 0x562399de86b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399e32f70 0x562399e2aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/1597477692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399e8c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/1597477692 >> 172.21.15.122:0/1597477692 conn(0x562399e15340 msgr2=0x562399e29ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/1597477692 shutdown_connections 2024-03-06T07:28:23.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.746+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/1597477692 wait complete. 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.747+0000 7f9f07ab8f40 1 Processor -- start 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.747+0000 7f9f07ab8f40 1 -- start start 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399eb9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 0x562399e8e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399ea8690 0x562399e8e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562399ea6400 con 0x562399e32f70 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562399ea6160 con 0x562399ea8690 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x562399e8bbe0 con 0x562399deb810 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 0x562399e8e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 0x562399e8e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34932/0 (socket says 172.21.15.122:34932) 2024-03-06T07:28:23.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 learned_addr learned my addr 172.21.15.122:0/778656428 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399ea8690 0x562399e8e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399eb9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 msgr2=0x562399eb9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399eb9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399ea8690 msgr2=0x562399e8e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399ea8690 0x562399e8e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562399ea2ee0 con 0x562399e32f70 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562399ea8690 0x562399e8e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x562399deb810 0x562399eb9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 0x562399e8e2d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009700 tx=0x7f9ef0009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efe7fc640 1 -- 172.21.15.122:0/778656428 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0005ad0 con 0x562399e32f70 2024-03-06T07:28:23.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efe7fc640 1 -- 172.21.15.122:0/778656428 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef0005c90 con 0x562399e32f70 2024-03-06T07:28:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562399ea31c0 con 0x562399e32f70 2024-03-06T07:28:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9efe7fc640 1 -- 172.21.15.122:0/778656428 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0014680 con 0x562399e32f70 2024-03-06T07:28:23.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.748+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562399ea3780 con 0x562399e32f70 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.750+0000 7f9efe7fc640 1 -- 172.21.15.122:0/778656428 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ef001d440 con 0x562399e32f70 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.750+0000 7f9efe7fc640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee8069eb0 0x7f9ee806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.750+0000 7f9efe7fc640 1 -- 172.21.15.122:0/778656428 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9ef0089340 con 0x562399e32f70 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.750+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee8069eb0 0x7f9ee806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.750+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x562399ea3f80 0x562399ea4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.750+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.751+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee8069eb0 0x7f9ee806c370 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800a5e0 tx=0x7f9ef8006630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.751+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x562399ea3f80 0x562399ea4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.751+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x562399ea3f80 0x562399ea4360 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.751+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f9f00055030 0x7f9f0005fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f9f00055030 0x7f9f0005fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f9f00055030 0x7f9f0005fa90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f9ef8015930 0x7f9ef8013400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.752+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f9ef8015930 0x7f9ef8013400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.753+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f9ef8015930 0x7f9ef8013400 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.753+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.753+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f9ef0008030 0x7f9ef0056980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.753+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.753+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f9ef0008030 0x7f9ef0056980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.753+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f9ef0008030 0x7f9ef0056980 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.754+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.754+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.754+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.754+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.754+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.754+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.755+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 2 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.755+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.755+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.755+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.755+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 3 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.755+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7f9ef0057d20 0x7f9ef006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0057d20 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7f9ef0057d20 0x7f9ef006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.756+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7f9ef0057d20 0x7f9ef006ba80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.757+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.5 v2:172.21.15.169:6816/887889200 1 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9ef0057d20 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.757+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.757+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.757+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f9f000763f0 0x7f9f00076870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.757+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.757+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f9f000763f0 0x7f9f00076870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.758+0000 7f9efffff640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f9f000763f0 0x7f9f00076870 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.758+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.758+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.759+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.759+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.759+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 4 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.759+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.759+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.759+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0057d20 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.760+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.5 v2:172.21.15.169:6816/887889200 2 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9ef0057d20 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.760+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.760+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.760+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.760+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.760+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 5 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7f9ef8026040 0x7f9ef8026aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.761+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8026040 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.762+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7f9ef8026040 0x7f9ef8026aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.762+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7f9ef8026040 0x7f9ef8026aa0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.762+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.6 v2:172.21.15.169:6824/1645803606 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef8026040 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.762+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 6 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0057d20 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.5 v2:172.21.15.169:6816/887889200 3 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9ef0057d20 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 7 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f05dba640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f9f000683f0 0x7f9f00068870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000683f0 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.763+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f9f000683f0 0x7f9f00068870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.764+0000 7f9f055b9640 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f9f000683f0 0x7f9f00068870 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.764+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.2 v2:172.21.15.122:6816/1235809965 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f000683f0 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.764+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8026040 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.764+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.6 v2:172.21.15.169:6824/1645803606 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef8026040 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.764+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.765+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.766+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.766+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.766+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.766+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.766+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.766+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.767+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.767+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0057d20 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.767+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.5 v2:172.21.15.169:6816/887889200 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9ef0057d20 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.767+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0057d20 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.768+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.5 v2:172.21.15.169:6816/887889200 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9ef0057d20 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.768+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8026040 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.768+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.6 v2:172.21.15.169:6824/1645803606 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef8026040 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.768+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.768+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.768+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0057d20 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.769+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.5 v2:172.21.15.169:6816/887889200 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9ef0057d20 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.769+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.769+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.769+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.769+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.769+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.770+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.771+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.771+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8015930 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.771+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.3 v2:172.21.15.122:6824/47863215 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9ef8015930 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.771+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x562399ea3f80 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.771+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.7 v2:172.21.15.169:6800/3079115781 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x562399ea3f80 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.771+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.772+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.772+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.772+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.772+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.772+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.772+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000683f0 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.773+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.2 v2:172.21.15.122:6816/1235809965 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f000683f0 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.773+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.773+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 11 ==== osd_op_reply(57 [pgnls start_epoch 114 out=49b] v114'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.773+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.773+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.773+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8026040 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.774+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.6 v2:172.21.15.169:6824/1645803606 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef8026040 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.774+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.774+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 12 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.774+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f00055030 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.774+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 <== osd.1 v2:172.21.15.122:6808/20360309 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef8007880 con 0x7f9f00055030 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.774+0000 7f9f055b9640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9f000763f0 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 <== osd.4 v2:172.21.15.169:6804/1079551346 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=49b] v114'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ef0005040 con 0x7f9f000763f0 2024-03-06T07:28:23.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9efffff640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef8026040 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.6 v2:172.21.15.169:6824/1645803606 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef8026040 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x562399eba2f0 con 0x7f9ef0008030 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 <== osd.0 v2:172.21.15.122:6800/2293549780 13 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f0005d040 con 0x7f9ef0008030 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f9ef0008030 msgr2=0x7f9ef0056980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f9ef0008030 0x7f9ef0056980 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f9f00055030 msgr2=0x7f9f0005fa90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f9f00055030 0x7f9f0005fa90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f9f000683f0 msgr2=0x7f9f00068870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7f9f000683f0 0x7f9f00068870 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f9ef8015930 msgr2=0x7f9ef8013400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 reap_dead start 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.775+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f9ef8015930 0x7f9ef8013400 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f9f000763f0 msgr2=0x7f9f00076870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f9f000763f0 0x7f9f00076870 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7f9ef0057d20 msgr2=0x7f9ef006ba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7f9ef0057d20 0x7f9ef006ba80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7f9ef8026040 msgr2=0x7f9ef8026aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7f9ef8026040 0x7f9ef8026aa0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x562399ea3f80 msgr2=0x562399ea4360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x562399ea3f80 0x562399ea4360 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f05dba640 1 -- 172.21.15.122:0/778656428 reap_dead start 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee8069eb0 msgr2=0x7f9ee806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee8069eb0 0x7f9ee806c370 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800a5e0 tx=0x7f9ef8006630 comp rx=0 tx=0).stop 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 msgr2=0x562399e8e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 0x562399e8e2d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009700 tx=0x7f9ef0009bc0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 shutdown_connections 2024-03-06T07:28:23.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ee8069eb0 0x7f9ee806c370 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 --2- 172.21.15.122:0/778656428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562399e32f70 0x562399e8e2d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 >> 172.21.15.122:0/778656428 conn(0x562399e15340 msgr2=0x562399ea5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 shutdown_connections 2024-03-06T07:28:23.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.776+0000 7f9f07ab8f40 1 -- 172.21.15.122:0/778656428 wait complete. 2024-03-06T07:28:23.789 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:28:23.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.922+0000 7f1dd09e9640 1 Processor -- start 2024-03-06T07:28:23.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.922+0000 7f1dd09e9640 1 -- start start 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dd09e9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1dcc15bb20 0x7f1dcc0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dd09e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1dcc0cd8c0 0x7f1dcc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dd09e9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc161980 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dd09e9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dcc161c60 con 0x7f1dcc0cd8c0 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dd09e9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dcc161f40 con 0x7f1dcc15bb20 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1dcc0cd8c0 0x7f1dcc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1dcc0cd8c0 0x7f1dcc0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34942/0 (socket says 172.21.15.122:34942) 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 -- 172.21.15.122:0/951838766 learned_addr learned my addr 172.21.15.122:0/951838766 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dca59c640 1 --2- 172.21.15.122:0/951838766 >> v2:172.21.15.169:3300/0 conn(0x7f1dcc15bb20 0x7f1dcc0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 -- 172.21.15.122:0/951838766 >> v2:172.21.15.169:3300/0 conn(0x7f1dcc15bb20 msgr2=0x7f1dcc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 --2- 172.21.15.122:0/951838766 >> v2:172.21.15.169:3300/0 conn(0x7f1dcc15bb20 0x7f1dcc0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 -- 172.21.15.122:0/951838766 >> v1:172.21.15.169:6789/0 conn(0x7f1dcc0cfff0 legacy=0x7f1dcc0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.923+0000 7f1dcad9d640 1 -- 172.21.15.122:0/951838766 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc162220 con 0x7f1dcc0cd8c0 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.924+0000 7f1dcad9d640 1 --2- 172.21.15.122:0/951838766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0d04a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0d2bc0 tx=0x7f1dc402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b413eeba428a4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.924+0000 7f1dc959a640 1 -- 172.21.15.122:0/951838766 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc402cb20 con 0x7f1dcc0cd8c0 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.924+0000 7f1dc959a640 1 -- 172.21.15.122:0/951838766 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc402ccc0 con 0x7f1dcc0cd8c0 2024-03-06T07:28:23.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.924+0000 7f1dc959a640 1 -- 172.21.15.122:0/951838766 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4034610 con 0x7f1dcc0cd8c0 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 -- 172.21.15.122:0/951838766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cd8c0 msgr2=0x7f1dcc0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/951838766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0d04a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0d2bc0 tx=0x7f1dc402bf80 comp rx=0 tx=0).stop 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 -- 172.21.15.122:0/951838766 shutdown_connections 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/951838766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0d04a0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/951838766 >> v2:172.21.15.169:3300/0 conn(0x7f1dcc15bb20 0x7f1dcc0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 -- 172.21.15.122:0/951838766 >> 172.21.15.122:0/951838766 conn(0x7f1dcc09a9f0 msgr2=0x7f1dcc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 -- 172.21.15.122:0/951838766 shutdown_connections 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.925+0000 7f1dd09e9640 1 -- 172.21.15.122:0/951838766 wait complete. 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.926+0000 7f1dd09e9640 1 Processor -- start 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.926+0000 7f1dd09e9640 1 -- start start 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dd09e9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dd09e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 0x7f1dcc1f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dd09e9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1dcc15bb20 0x7f1dcc0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dd09e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc0d24b0 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dd09e9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc0d2330 con 0x7f1dcc15bb20 2024-03-06T07:28:23.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dd09e9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc0d1980 con 0x7f1dcc0cd8c0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 0x7f1dcc1f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 0x7f1dcc1f4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34946/0 (socket says 172.21.15.122:34946) 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 -- 172.21.15.122:0/2420370855 learned_addr learned my addr 172.21.15.122:0/2420370855 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dcad9d640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1dcc15bb20 0x7f1dcc0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dca59c640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 -- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1dcc0cd8c0 msgr2=0x7f1dcc0c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 -- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1dcc15bb20 msgr2=0x7f1dcc0c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1dcc15bb20 0x7f1dcc0c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.927+0000 7f1dc9d9b640 1 -- 172.21.15.122:0/2420370855 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc0c5e40 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.928+0000 7f1dc9d9b640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 0x7f1dcc1f4670 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00c400 tx=0x7f1dbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.928+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc0052d0 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.928+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dbc005490 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.928+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc0c6120 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.928+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc007260 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.928+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc06c070 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.929+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc04f330 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.929+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dbc007400 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.930+0000 7f1dbaffd640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dac06b4d0 0x7f1dac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.930+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1dbc089760 con 0x7f1dcc0cfff0 2024-03-06T07:28:23.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.930+0000 7f1dca59c640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dac06b4d0 0x7f1dac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.930+0000 7f1dca59c640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dac06b4d0 0x7f1dac06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009aa0 tx=0x7f1dc0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:23.934+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1dbc057f00 con 0x7f1dcc0cfff0 2024-03-06T07:28:24.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.150+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dcc0d0800 con 0x7f1dcc0cfff0 2024-03-06T07:28:24.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.151+0000 7f1dbaffd640 1 -- 172.21.15.122:0/2420370855 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f1dbc05bfa0 con 0x7f1dcc0cfff0 2024-03-06T07:28:24.153 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:24.153 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:24.153 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:24.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dac06b4d0 msgr2=0x7f1dac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dac06b4d0 0x7f1dac06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009aa0 tx=0x7f1dc0009190 comp rx=0 tx=0).stop 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 msgr2=0x7f1dcc1f4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 0x7f1dcc1f4670 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00c400 tx=0x7f1dbc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 shutdown_connections 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dac06b4d0 0x7f1dac06d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1dcc15bb20 0x7f1dcc0c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc0cfff0 0x7f1dcc1f4670 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 --2- 172.21.15.122:0/2420370855 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1dcc0cd8c0 0x7f1dcc0c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 >> 172.21.15.122:0/2420370855 conn(0x7f1dcc09a9f0 msgr2=0x7f1dcc06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 shutdown_connections 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.153+0000 7f1dd09e9640 1 -- 172.21.15.122:0/2420370855 wait complete. 2024-03-06T07:28:24.166 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:28:24.282 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:28:24.282 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:28:24.282 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:24.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.418+0000 7fa8e6181640 1 Processor -- start 2024-03-06T07:28:24.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8e6181640 1 -- start start 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8e6181640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa8e00d2710 0x7fa8e00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8e6181640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e00b3430 con 0x7fa8e00d22b0 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8e6181640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e00b3710 con 0x7fa8e015dfe0 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8e6181640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00b31e0 con 0x7fa8e00d2710 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8deffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa8e00d22b0 0x7fa8e00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49986/0 (socket says 172.21.15.122:49986) 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.419+0000 7fa8deffd640 1 -- 172.21.15.122:0/3129492455 learned_addr learned my addr 172.21.15.122:0/3129492455 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247636433 0 0) 0x7fa8e00b3430 con 0x7fa8e00d22b0 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c8003520 con 0x7fa8e00d22b0 2024-03-06T07:28:24.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8dffff640 1 --2- 172.21.15.122:0/3129492455 >> v2:172.21.15.169:3300/0 conn(0x7fa8e00d2710 0x7fa8e00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1115144007 0 0) 0x7fa8c8003520 con 0x7fa8e00d22b0 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 >> v2:172.21.15.169:3300/0 conn(0x7fa8e00d2710 msgr2=0x7fa8e00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 --2- 172.21.15.122:0/3129492455 >> v2:172.21.15.169:3300/0 conn(0x7fa8e00d2710 0x7fa8e00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 >> v1:172.21.15.169:6789/0 conn(0x7fa8e015dfe0 legacy=0x7fa8e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00b39f0 con 0x7fa8e00d22b0 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa8d4001670 con 0x7fa8e00d22b0 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 >> v1:172.21.15.122:6789/0 conn(0x7fa8e00d22b0 legacy=0x7fa8e00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8c80039e0 0x7fa8c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c8004800 0x7fa8c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8c80051c0 0x7fa8c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00b31e0 con 0x7fa8c80039e0 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c800c090 con 0x7fa8c8004800 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c800c290 con 0x7fa8c80051c0 2024-03-06T07:28:24.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8c80039e0 0x7fa8c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8deffd640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c8004800 0x7fa8c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8dffff640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8c80051c0 0x7fa8c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8df7fe640 1 -- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8c80051c0 msgr2=0x7fa8c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8c80051c0 0x7fa8c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8df7fe640 1 -- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c8004800 msgr2=0x7fa8c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c8004800 0x7fa8c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8dffff640 1 -- 172.21.15.122:0/3129492455 reap_dead start 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8df7fe640 1 -- 172.21.15.122:0/3129492455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8003520 con 0x7fa8c80039e0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.420+0000 7fa8deffd640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c8004800 0x7fa8c8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.421+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8c80039e0 0x7fa8c8003e00 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0001750 tx=0x7fa8c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1774af84fd8c2528 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.421+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0001370 con 0x7fa8c80039e0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.421+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d00052d0 con 0x7fa8c80039e0 2024-03-06T07:28:24.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.421+0000 7fa8de7fc640 1 -- 172.21.15.122:0/3129492455 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0003830 con 0x7fa8c80039e0 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 -- 172.21.15.122:0/3129492455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8c80039e0 msgr2=0x7fa8c8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8c80039e0 0x7fa8c8003e00 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0001750 tx=0x7fa8c8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 -- 172.21.15.122:0/3129492455 shutdown_connections 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 --2- 172.21.15.122:0/3129492455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8c80039e0 0x7fa8c8003e00 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 -- 172.21.15.122:0/3129492455 >> 172.21.15.122:0/3129492455 conn(0x7fa8e00d3cd0 msgr2=0x7fa8e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 -- 172.21.15.122:0/3129492455 shutdown_connections 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.422+0000 7fa8e6181640 1 -- 172.21.15.122:0/3129492455 wait complete. 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.423+0000 7fa8e6181640 1 Processor -- start 2024-03-06T07:28:24.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.423+0000 7fa8e6181640 1 -- start start 2024-03-06T07:28:24.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8e6181640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c80039e0 0x7fa8e00b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8e6181640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 0x7fa8e00b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8e6181640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8e00b1e60 0x7fa8e00b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8e6181640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00d1c70 con 0x7fa8e0153380 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8e6181640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00b3d60 con 0x7fa8c80039e0 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8e6181640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00d1df0 con 0x7fa8e00b1e60 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 0x7fa8e00b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 0x7fa8e00b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34960/0 (socket says 172.21.15.122:34960) 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 -- 172.21.15.122:0/2600919030 learned_addr learned my addr 172.21.15.122:0/2600919030 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c80039e0 0x7fa8e00b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8dffff640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8e00b1e60 0x7fa8e00b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 -- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8e00b1e60 msgr2=0x7fa8e00b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8e00b1e60 0x7fa8e00b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 -- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c80039e0 msgr2=0x7fa8e00b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c80039e0 0x7fa8e00b9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 -- 172.21.15.122:0/2600919030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cada0 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.424+0000 7fa8deffd640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 0x7fa8e00b1920 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400e780 tx=0x7fa8d400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.425+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d400ee30 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.425+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d4018070 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.425+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e00cb080 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.425+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d400ca70 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.425+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e00cb640 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.426+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e004f330 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.426+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8d400cc10 con 0x7fa8e0153380 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.426+0000 7fa8c7fff640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.427+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.427+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa8d4089ec0 con 0x7fa8e0153380 2024-03-06T07:28:24.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.427+0000 7fa8df7fe640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0033000 tx=0x7fa8d0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.431+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8d4058660 con 0x7fa8e0153380 2024-03-06T07:28:24.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.669+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8e015d5c0 con 0x7fa8e0153380 2024-03-06T07:28:24.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.669+0000 7fa8c7fff640 1 -- 172.21.15.122:0/2600919030 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa8d405c700 con 0x7fa8e0153380 2024-03-06T07:28:24.709 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:24.709 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:24.710 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:24.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8cc06b500 msgr2=0x7fa8cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0033000 tx=0x7fa8d0009030 comp rx=0 tx=0).stop 2024-03-06T07:28:24.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 msgr2=0x7fa8e00b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 0x7fa8e00b1920 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400e780 tx=0x7fa8d400ec40 comp rx=0 tx=0).stop 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 shutdown_connections 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8e00b1e60 0x7fa8e00b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8e0153380 0x7fa8e00b1920 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 --2- 172.21.15.122:0/2600919030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8c80039e0 0x7fa8e00b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 >> 172.21.15.122:0/2600919030 conn(0x7fa8e00d3cd0 msgr2=0x7fa8e00b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 shutdown_connections 2024-03-06T07:28:24.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:24.671+0000 7fa8e6181640 1 -- 172.21.15.122:0/2600919030 wait complete. 2024-03-06T07:28:24.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:24.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:24.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:24.830 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:24.830 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:24.830 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:24.830 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:24.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:25.052 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:25.053 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:28:25.053 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:25.069 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:25.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.211+0000 7fc6424eb640 1 Processor -- start 2024-03-06T07:28:25.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.211+0000 7fc6424eb640 1 -- start start 2024-03-06T07:28:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.212+0000 7fc6424eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc63c15dfe0 0x7fc63c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.212+0000 7fc6424eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc63c0cf830 0x7fc63c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.212+0000 7fc6424eb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63c0b3430 con 0x7fc63c0d27f0 2024-03-06T07:28:25.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.212+0000 7fc6424eb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c0b31e0 con 0x7fc63c0cf830 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.212+0000 7fc6424eb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c0b3d60 con 0x7fc63c15dfe0 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc63c0cf830 0x7fc63c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc63c0cf830 0x7fc63c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40920/0 (socket says 172.21.15.122:40920) 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 -- 172.21.15.122:0/2120310787 learned_addr learned my addr 172.21.15.122:0/2120310787 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63bfff640 1 --2- 172.21.15.122:0/2120310787 >> v2:172.21.15.169:3300/0 conn(0x7fc63c15dfe0 0x7fc63c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 -- 172.21.15.122:0/2120310787 >> v2:172.21.15.169:3300/0 conn(0x7fc63c15dfe0 msgr2=0x7fc63c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 --2- 172.21.15.122:0/2120310787 >> v2:172.21.15.169:3300/0 conn(0x7fc63c15dfe0 0x7fc63c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 -- 172.21.15.122:0/2120310787 >> v1:172.21.15.169:6789/0 conn(0x7fc63c0d27f0 legacy=0x7fc63c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.213+0000 7fc63b7fe640 1 -- 172.21.15.122:0/2120310787 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0b3710 con 0x7fc63c0cf830 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc63b7fe640 1 --2- 172.21.15.122:0/2120310787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c0cf830 0x7fc63c0d22b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009870 tx=0x7fc62c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9becec0b4f48894f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc63affd640 1 -- 172.21.15.122:0/2120310787 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c034070 con 0x7fc63c0cf830 2024-03-06T07:28:25.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc63affd640 1 -- 172.21.15.122:0/2120310787 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c030860 con 0x7fc63c0cf830 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc63affd640 1 -- 172.21.15.122:0/2120310787 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c030c30 con 0x7fc63c0cf830 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 -- 172.21.15.122:0/2120310787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c0cf830 msgr2=0x7fc63c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 --2- 172.21.15.122:0/2120310787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c0cf830 0x7fc63c0d22b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009870 tx=0x7fc62c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 -- 172.21.15.122:0/2120310787 shutdown_connections 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 --2- 172.21.15.122:0/2120310787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c0cf830 0x7fc63c0d22b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 --2- 172.21.15.122:0/2120310787 >> v2:172.21.15.169:3300/0 conn(0x7fc63c15dfe0 0x7fc63c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 -- 172.21.15.122:0/2120310787 >> 172.21.15.122:0/2120310787 conn(0x7fc63c0d3cd0 msgr2=0x7fc63c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 -- 172.21.15.122:0/2120310787 shutdown_connections 2024-03-06T07:28:25.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.214+0000 7fc6424eb640 1 -- 172.21.15.122:0/2120310787 wait complete. 2024-03-06T07:28:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.215+0000 7fc6424eb640 1 Processor -- start 2024-03-06T07:28:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 -- start start 2024-03-06T07:28:25.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c0cf830 0x7fc63c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc63c0d27f0 0x7fc63c0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 0x7fc63c0b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c0d1df0 con 0x7fc63c15dfe0 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c0d1c70 con 0x7fc63c0cf830 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc6424eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c15e3c0 con 0x7fc63c0d27f0 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc640a88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 0x7fc63c0b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc640a88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 0x7fc63c0b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40924/0 (socket says 172.21.15.122:40924) 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.216+0000 7fc640a88640 1 -- 172.21.15.122:0/1255789932 learned_addr learned my addr 172.21.15.122:0/1255789932 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc63bfff640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c0cf830 0x7fc63c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc63b7fe640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc63c0d27f0 0x7fc63c0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc640a88640 1 -- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc63c0d27f0 msgr2=0x7fc63c0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc640a88640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc63c0d27f0 0x7fc63c0b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc640a88640 1 -- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c0cf830 msgr2=0x7fc63c0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc640a88640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c0cf830 0x7fc63c0cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc640a88640 1 -- 172.21.15.122:0/1255789932 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0cc330 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc63bfff640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c0cf830 0x7fc63c0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc640a88640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 0x7fc63c0b6de0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc63400c7d0 tx=0x7fc63400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634003b70 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc634003d30 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c0cc5b0 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63400aa90 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.217+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c0ccb70 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.218+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63c04f330 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.218+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc63400ac30 con 0x7fc63c15dfe0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.219+0000 7fc638ff9640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc61c06b4d0 0x7fc61c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.219+0000 7fc63bfff640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc61c06b4d0 0x7fc61c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.219+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc63408a1c0 con 0x7fc63c15dfe0 2024-03-06T07:28:25.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.219+0000 7fc63bfff640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc61c06b4d0 0x7fc61c06d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc624009b90 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.224+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc63c0cc330 con 0x7fc63c15dfe0 2024-03-06T07:28:25.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.475+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc63c15d220 con 0x7fc63c15dfe0 2024-03-06T07:28:25.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.476+0000 7fc638ff9640 1 -- 172.21.15.122:0/1255789932 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16909 (secure 0 0 0) 0x7fc634058fb0 con 0x7fc63c15dfe0 2024-03-06T07:28:25.479 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:25.479 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":114,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:34.986316+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/386534812":"2024-03-07T07:27:28.499450+0000","172.21.15.122:6836/386534812":"2024-03-07T07:27:28.499450+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc61c06b4d0 msgr2=0x7fc61c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc61c06b4d0 0x7fc61c06d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc624009b90 tx=0x7fc624008040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 msgr2=0x7fc63c0b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 0x7fc63c0b6de0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc63400c7d0 tx=0x7fc63400cc90 comp rx=0 tx=0).stop 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 shutdown_connections 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc61c06b4d0 0x7fc61c06d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc63c15dfe0 0x7fc63c0b6de0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc63c0d27f0 0x7fc63c0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 --2- 172.21.15.122:0/1255789932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc63c0cf830 0x7fc63c0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.478+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 >> 172.21.15.122:0/1255789932 conn(0x7fc63c0d3cd0 msgr2=0x7fc63c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.479+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 shutdown_connections 2024-03-06T07:28:25.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.479+0000 7fc6424eb640 1 -- 172.21.15.122:0/1255789932 wait complete. 2024-03-06T07:28:25.491 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.843386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.579477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:25.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:25.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.593+0000 7fc8be15e640 1 Processor -- start 2024-03-06T07:28:25.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.593+0000 7fc8be15e640 1 -- start start 2024-03-06T07:28:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.594+0000 7fc8be15e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc8b806fd10 0x7fc8b80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.594+0000 7fc8be15e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.594+0000 7fc8be15e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80cd1e0 con 0x7fc8b815c050 2024-03-06T07:28:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.594+0000 7fc8be15e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80cd4b0 con 0x7fc8b806fd10 2024-03-06T07:28:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.594+0000 7fc8be15e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cd790 con 0x7fc8b80d0af0 2024-03-06T07:28:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc8b806fd10 0x7fc8b80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc8b806fd10 0x7fc8b80cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:35304/0 (socket says 172.21.15.122:35304) 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc8b815c050 0x7fc8b806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:54612/0 (socket says 172.21.15.122:54612) 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 -- 172.21.15.122:0/3252051948 learned_addr learned my addr 172.21.15.122:0/3252051948 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 -- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 msgr2=0x7fc8b80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 --2- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b80ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 -- 172.21.15.122:0/3252051948 >> v1:172.21.15.169:6789/0 conn(0x7fc8b815c050 legacy=0x7fc8b806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.595+0000 7fc8b6ffd640 1 -- 172.21.15.122:0/3252051948 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80cda70 con 0x7fc8b806fd10 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.596+0000 7fc8b6ffd640 1 --2- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b806fd10 0x7fc8b80cc760 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009870 tx=0x7fc8a002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b35509efdaad7484 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.596+0000 7fc8b67fc640 1 -- 172.21.15.122:0/3252051948 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0035020 con 0x7fc8b806fd10 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.596+0000 7fc8b67fc640 1 -- 172.21.15.122:0/3252051948 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a0031640 con 0x7fc8b806fd10 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.596+0000 7fc8b67fc640 1 -- 172.21.15.122:0/3252051948 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a002d040 con 0x7fc8b806fd10 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.596+0000 7fc8be15e640 1 -- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b806fd10 msgr2=0x7fc8b80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.596+0000 7fc8be15e640 1 --2- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b806fd10 0x7fc8b80cc760 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009870 tx=0x7fc8a002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.597+0000 7fc8be15e640 1 -- 172.21.15.122:0/3252051948 shutdown_connections 2024-03-06T07:28:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.597+0000 7fc8be15e640 1 --2- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b80ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.597+0000 7fc8be15e640 1 --2- 172.21.15.122:0/3252051948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b806fd10 0x7fc8b80cc760 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.597+0000 7fc8be15e640 1 -- 172.21.15.122:0/3252051948 >> 172.21.15.122:0/3252051948 conn(0x7fc8b80b3800 msgr2=0x7fc8b80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.597+0000 7fc8be15e640 1 -- 172.21.15.122:0/3252051948 shutdown_connections 2024-03-06T07:28:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.597+0000 7fc8be15e640 1 -- 172.21.15.122:0/3252051948 wait complete. 2024-03-06T07:28:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.598+0000 7fc8be15e640 1 Processor -- start 2024-03-06T07:28:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.598+0000 7fc8be15e640 1 -- start start 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8be15e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 0x7fc8b81547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8be15e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8be15e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b815c050 0x7fc8b81509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8be15e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8165c30 con 0x7fc8b806fd10 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8be15e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8165ab0 con 0x7fc8b815c050 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8be15e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8165db0 con 0x7fc8b80d0af0 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 0x7fc8b81547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 0x7fc8b81547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40926/0 (socket says 172.21.15.122:40926) 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 -- 172.21.15.122:0/2021005783 learned_addr learned my addr 172.21.15.122:0/2021005783 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b6ffd640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b7fff640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b815c050 0x7fc8b81509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 -- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 msgr2=0x7fc8b81576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b81576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 -- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b815c050 msgr2=0x7fc8b81509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b815c050 0x7fc8b81509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 -- 172.21.15.122:0/2021005783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b8152120 con 0x7fc8b806fd10 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8b77fe640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 0x7fc8b81547d0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c400 tx=0x7fc8a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80074e0 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.599+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a8007680 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.600+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8152400 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.600+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8004a00 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.600+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8154d10 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.600+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b809fe60 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.601+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8a801a020 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.602+0000 7fc8affff640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc89806b4d0 0x7fc89806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.602+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc8a8089b30 con 0x7fc8b806fd10 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.602+0000 7fc8b6ffd640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc89806b4d0 0x7fc89806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.602+0000 7fc8b6ffd640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc89806b4d0 0x7fc89806d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0036000 tx=0x7fc8a003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.608+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8a80588a0 con 0x7fc8b806fd10 2024-03-06T07:28:25.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.787+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8b8068ff0 con 0x7fc8b806fd10 2024-03-06T07:28:25.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.787+0000 7fc8affff640 1 -- 172.21.15.122:0/2021005783 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8a80581a0 con 0x7fc8b806fd10 2024-03-06T07:28:25.789 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc89806b4d0 msgr2=0x7fc89806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc89806b4d0 0x7fc89806d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0036000 tx=0x7fc8a003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 msgr2=0x7fc8b81547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 0x7fc8b81547d0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c400 tx=0x7fc8a800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 shutdown_connections 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc89806b4d0 0x7fc89806d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8b815c050 0x7fc8b81509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8b80d0af0 0x7fc8b81576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 --2- 172.21.15.122:0/2021005783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8b806fd10 0x7fc8b81547d0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 >> 172.21.15.122:0/2021005783 conn(0x7fc8b80b3800 msgr2=0x7fc8b815b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 shutdown_connections 2024-03-06T07:28:25.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.789+0000 7fc8be15e640 1 -- 172.21.15.122:0/2021005783 wait complete. 2024-03-06T07:28:25.806 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:25.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.943+0000 7f32db70d640 1 Processor -- start 2024-03-06T07:28:25.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.943+0000 7f32db70d640 1 -- start start 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32db70d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f32d406fd10 0x7f32d40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32db70d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d40d0af0 0x7f32d40ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32db70d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40cd1e0 con 0x7f32d415c050 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32db70d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d40cd4b0 con 0x7f32d406fd10 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32db70d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40cd790 con 0x7f32d40d0af0 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32d94a9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f32d415c050 0x7f32d406f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40992/0 (socket says 172.21.15.122:40992) 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32d8ca8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f32d406fd10 0x7f32d40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.944+0000 7f32d94a9640 1 -- 172.21.15.122:0/4171613762 learned_addr learned my addr 172.21.15.122:0/4171613762 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32d9caa640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d40d0af0 0x7f32d40ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32d8ca8640 1 -- 172.21.15.122:0/4171613762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d40d0af0 msgr2=0x7f32d40ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32d8ca8640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d40d0af0 0x7f32d40ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32d8ca8640 1 -- 172.21.15.122:0/4171613762 >> v1:172.21.15.122:6789/0 conn(0x7f32d415c050 legacy=0x7f32d406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32cbfff640 1 -- 172.21.15.122:0/4171613762 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288017649 0 0) 0x7f32d40cd1e0 con 0x7f32d415c050 2024-03-06T07:28:25.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32d8ca8640 1 -- 172.21.15.122:0/4171613762 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40cda70 con 0x7f32d406fd10 2024-03-06T07:28:25.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.945+0000 7f32d9caa640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d40d0af0 0x7f32d40ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.946+0000 7f32d8ca8640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d406fd10 0x7f32d40cc760 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f32c40097a0 tx=0x7f32c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54799c65a6308316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.946+0000 7f32cbfff640 1 -- 172.21.15.122:0/4171613762 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c40315a0 con 0x7f32d406fd10 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.946+0000 7f32cbfff640 1 -- 172.21.15.122:0/4171613762 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c4037070 con 0x7f32d406fd10 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.946+0000 7f32cbfff640 1 -- 172.21.15.122:0/4171613762 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4031960 con 0x7f32d406fd10 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 -- 172.21.15.122:0/4171613762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d406fd10 msgr2=0x7f32d40cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d406fd10 0x7f32d40cc760 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f32c40097a0 tx=0x7f32c402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 -- 172.21.15.122:0/4171613762 shutdown_connections 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d40d0af0 0x7f32d40ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 --2- 172.21.15.122:0/4171613762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d406fd10 0x7f32d40cc760 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 -- 172.21.15.122:0/4171613762 >> 172.21.15.122:0/4171613762 conn(0x7f32d40b3800 msgr2=0x7f32d40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 -- 172.21.15.122:0/4171613762 shutdown_connections 2024-03-06T07:28:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.947+0000 7f32db70d640 1 -- 172.21.15.122:0/4171613762 wait complete. 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 Processor -- start 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 -- start start 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32d406fd10 0x7f32d4159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 0x7f32d415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d415c050 0x7f32d41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4165c30 con 0x7f32d40d0af0 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4165ab0 con 0x7f32d406fd10 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.948+0000 7f32db70d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4165db0 con 0x7f32d415c050 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 0x7f32d415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 0x7f32d415ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40936/0 (socket says 172.21.15.122:40936) 2024-03-06T07:28:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 -- 172.21.15.122:0/3526126906 learned_addr learned my addr 172.21.15.122:0/3526126906 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d9caa640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d415c050 0x7f32d41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 -- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d415c050 msgr2=0x7f32d41545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d415c050 0x7f32d41545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 -- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32d406fd10 msgr2=0x7f32d4159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d94a9640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32d406fd10 0x7f32d4159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32d406fd10 0x7f32d4159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 -- 172.21.15.122:0/3526126906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d415b1b0 con 0x7f32d40d0af0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d94a9640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32d406fd10 0x7f32d4159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32d8ca8640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 0x7f32d415ac70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f32c40097a0 tx=0x7f32c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4004230 con 0x7f32d40d0af0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c40043f0 con 0x7f32d40d0af0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d409fb30 con 0x7f32d40d0af0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.949+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4003850 con 0x7f32d40d0af0 2024-03-06T07:28:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.950+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d4159eb0 con 0x7f32d40d0af0 2024-03-06T07:28:25.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.951+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d409fe60 con 0x7f32d40d0af0 2024-03-06T07:28:25.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.951+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32c40039f0 con 0x7f32d40d0af0 2024-03-06T07:28:25.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.951+0000 7f32c9ffb640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32b806b4d0 0x7f32b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.952+0000 7f32d94a9640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32b806b4d0 0x7f32b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.952+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f32c402f030 con 0x7f32d40d0af0 2024-03-06T07:28:25.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.952+0000 7f32d94a9640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32b806b4d0 0x7f32b806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f32cc000f60 tx=0x7f32cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:25.956+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32c4041070 con 0x7f32d40d0af0 2024-03-06T07:28:26.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.185+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32d4157ff0 con 0x7f32d40d0af0 2024-03-06T07:28:26.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.186+0000 7f32c9ffb640 1 -- 172.21.15.122:0/3526126906 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f32c407b2a0 con 0x7f32d40d0af0 2024-03-06T07:28:26.188 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:28:26.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32b806b4d0 msgr2=0x7f32b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32b806b4d0 0x7f32b806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f32cc000f60 tx=0x7f32cc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 msgr2=0x7f32d415ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 0x7f32d415ac70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f32c40097a0 tx=0x7f32c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 shutdown_connections 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f32b806b4d0 0x7f32b806d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32d415c050 0x7f32d41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32d40d0af0 0x7f32d415ac70 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 --2- 172.21.15.122:0/3526126906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32d406fd10 0x7f32d4159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 >> 172.21.15.122:0/3526126906 conn(0x7f32d40b3800 msgr2=0x7f32d40aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 shutdown_connections 2024-03-06T07:28:26.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.188+0000 7f32db70d640 1 -- 172.21.15.122:0/3526126906 wait complete. 2024-03-06T07:28:26.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.305+0000 7f744bd8c640 1 Processor -- start 2024-03-06T07:28:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.305+0000 7f744bd8c640 1 -- start start 2024-03-06T07:28:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744bd8c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f74441614d0 0x7f74440d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744bd8c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f74440d1910 0x7f74440d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744bd8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74441618b0 con 0x7f74440d22e0 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744bd8c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74440cf400 con 0x7f74440d1910 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744bd8c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74440d2710 con 0x7f74441614d0 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449327640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f74440d1910 0x7f74440d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744a329640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f74440d22e0 0x7f74440d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41002/0 (socket says 172.21.15.122:41002) 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449327640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f74440d1910 0x7f74440d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40942/0 (socket says 172.21.15.122:40942) 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f744a329640 1 -- 172.21.15.122:0/800296726 learned_addr learned my addr 172.21.15.122:0/800296726 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449b28640 1 --2- 172.21.15.122:0/800296726 >> v2:172.21.15.169:3300/0 conn(0x7f74441614d0 0x7f74440d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449327640 1 -- 172.21.15.122:0/800296726 >> v2:172.21.15.169:3300/0 conn(0x7f74441614d0 msgr2=0x7f74440d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449327640 1 --2- 172.21.15.122:0/800296726 >> v2:172.21.15.169:3300/0 conn(0x7f74441614d0 0x7f74440d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449327640 1 -- 172.21.15.122:0/800296726 >> v1:172.21.15.122:6789/0 conn(0x7f74440d22e0 legacy=0x7f74440d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7448b26640 1 -- 172.21.15.122:0/800296726 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570808232 0 0) 0x7f74441618b0 con 0x7f74440d22e0 2024-03-06T07:28:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449327640 1 -- 172.21.15.122:0/800296726 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440cf6a0 con 0x7f74440d1910 2024-03-06T07:28:26.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.306+0000 7f7449b28640 1 --2- 172.21.15.122:0/800296726 >> v2:172.21.15.169:3300/0 conn(0x7f74441614d0 0x7f74440d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.307+0000 7f7449327640 1 --2- 172.21.15.122:0/800296726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d1910 0x7f74440d1da0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f744408c470 tx=0x7f743402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff82247574740dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.307+0000 7f7448b26640 1 -- 172.21.15.122:0/800296726 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434035020 con 0x7f74440d1910 2024-03-06T07:28:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.307+0000 7f7448b26640 1 -- 172.21.15.122:0/800296726 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7434031830 con 0x7f74440d1910 2024-03-06T07:28:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.307+0000 7f7448b26640 1 -- 172.21.15.122:0/800296726 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434031c00 con 0x7f74440d1910 2024-03-06T07:28:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 -- 172.21.15.122:0/800296726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d1910 msgr2=0x7f74440d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 --2- 172.21.15.122:0/800296726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d1910 0x7f74440d1da0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f744408c470 tx=0x7f743402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 -- 172.21.15.122:0/800296726 shutdown_connections 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 --2- 172.21.15.122:0/800296726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d1910 0x7f74440d1da0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 --2- 172.21.15.122:0/800296726 >> v2:172.21.15.169:3300/0 conn(0x7f74441614d0 0x7f74440d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 -- 172.21.15.122:0/800296726 >> 172.21.15.122:0/800296726 conn(0x7f74440d3d40 msgr2=0x7f74440d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 -- 172.21.15.122:0/800296726 shutdown_connections 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.308+0000 7f744bd8c640 1 -- 172.21.15.122:0/800296726 wait complete. 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.309+0000 7f744bd8c640 1 Processor -- start 2024-03-06T07:28:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.309+0000 7f744bd8c640 1 -- start start 2024-03-06T07:28:26.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744bd8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74440d1910 0x7f74441596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744bd8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 0x7f74441f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744bd8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74441614d0 0x7f74440c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744bd8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440b0910 con 0x7f74440d22e0 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744bd8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440cfc60 con 0x7f74440d1910 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744bd8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74440b1e40 con 0x7f74441614d0 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 0x7f74441f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 0x7f74441f4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40950/0 (socket says 172.21.15.122:40950) 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 -- 172.21.15.122:0/1600422726 learned_addr learned my addr 172.21.15.122:0/1600422726 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f744a329640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74441614d0 0x7f74440c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449b28640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74440d1910 0x7f74441596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 -- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74441614d0 msgr2=0x7f74440c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74441614d0 0x7f74440c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 -- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74440d1910 msgr2=0x7f74441596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74440d1910 0x7f74441596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 -- 172.21.15.122:0/1600422726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440c7870 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.310+0000 7f7449327640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 0x7f74441f4330 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f7434009890 tx=0x7f74340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.311+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434041070 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.311+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743403aa30 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.311+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74441f4870 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.311+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434035040 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.311+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74440c7df0 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.311+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7444068ff0 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.312+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7434038020 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.312+0000 7f743a7fc640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f742806b4d0 0x7f742806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.313+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f74340abeb0 con 0x7f74440d22e0 2024-03-06T07:28:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.313+0000 7f7449b28640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f742806b4d0 0x7f742806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.318 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.313+0000 7f7449b28640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f742806b4d0 0x7f742806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f74440c7460 tx=0x7f7440008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.317+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f743407ac20 con 0x7f74440d22e0 2024-03-06T07:28:26.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.494+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74440a1c10 con 0x7f74440d22e0 2024-03-06T07:28:26.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.495+0000 7f743a7fc640 1 -- 172.21.15.122:0/1600422726 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f743407a520 con 0x7f74440d22e0 2024-03-06T07:28:26.497 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:28:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f742806b4d0 msgr2=0x7f742806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f742806b4d0 0x7f742806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f74440c7460 tx=0x7f7440008040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 msgr2=0x7f74441f4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 0x7f74441f4330 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f7434009890 tx=0x7f74340046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 shutdown_connections 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f742806b4d0 0x7f742806d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74441614d0 0x7f74440c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74440d22e0 0x7f74441f4330 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 --2- 172.21.15.122:0/1600422726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74440d1910 0x7f74441596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 >> 172.21.15.122:0/1600422726 conn(0x7f74440d3d40 msgr2=0x7f744406c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 shutdown_connections 2024-03-06T07:28:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.497+0000 7f744bd8c640 1 -- 172.21.15.122:0/1600422726 wait complete. 2024-03-06T07:28:26.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbca5815640 1 Processor -- start 2024-03-06T07:28:26.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbca5815640 1 -- start start 2024-03-06T07:28:26.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbca5815640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbca015bbf0 0x7fbca0071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbca5815640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca0072490 con 0x7fbca00cd7c0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbca5815640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca00d0090 con 0x7fbca00cebc0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbca5815640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca00d0370 con 0x7fbca015bbf0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbc9f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbca00cd7c0 0x7fbca00cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41012/0 (socket says 172.21.15.122:41012) 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.610+0000 7fbc9f7fe640 1 -- 172.21.15.122:0/2364975450 learned_addr learned my addr 172.21.15.122:0/2364975450 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9effd640 1 --2- 172.21.15.122:0/2364975450 >> v2:172.21.15.169:3300/0 conn(0x7fbca015bbf0 0x7fbca0071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240677568 0 0) 0x7fbca0072490 con 0x7fbca00cd7c0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8c003520 con 0x7fbca00cd7c0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862867392 0 0) 0x7fbc8c003520 con 0x7fbca00cd7c0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 >> v2:172.21.15.169:3300/0 conn(0x7fbca015bbf0 msgr2=0x7fbca0071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 --2- 172.21.15.122:0/2364975450 >> v2:172.21.15.169:3300/0 conn(0x7fbca015bbf0 0x7fbca0071e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 >> v1:172.21.15.169:6789/0 conn(0x7fbca00cebc0 legacy=0x7fbca015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca00d0650 con 0x7fbca00cd7c0 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fbc900019f0 con 0x7fbca00cd7c0 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 >> v1:172.21.15.122:6789/0 conn(0x7fbca00cd7c0 legacy=0x7fbca00cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c0039b0 0x7fbc8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc8c004800 0x7fbc8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc8c0051c0 0x7fbc8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00d0370 con 0x7fbc8c004800 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c00c060 con 0x7fbc8c0051c0 2024-03-06T07:28:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c00c260 con 0x7fbc8c0039b0 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc8c004800 0x7fbc8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9effd640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c0039b0 0x7fbc8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.611+0000 7fbc9f7fe640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc8c0051c0 0x7fbc8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c0039b0 msgr2=0x7fbc8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c0039b0 0x7fbc8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc8c0051c0 msgr2=0x7fbc8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc8c0051c0 0x7fbc8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/2364975450 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c003520 con 0x7fbc8c004800 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9f7fe640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc8c0051c0 0x7fbc8c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9f7fe640 1 -- 172.21.15.122:0/2364975450 reap_dead start 2024-03-06T07:28:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc8c004800 0x7fbc8c004c80 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fbc88001750 tx=0x7fbc8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce56cd6e1053318 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88001370 con 0x7fbc8c004800 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc880052d0 con 0x7fbc8c004800 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.612+0000 7fbc9dffb640 1 -- 172.21.15.122:0/2364975450 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88003830 con 0x7fbc8c004800 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 -- 172.21.15.122:0/2364975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc8c004800 msgr2=0x7fbc8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc8c004800 0x7fbc8c004c80 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fbc88001750 tx=0x7fbc8c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 -- 172.21.15.122:0/2364975450 shutdown_connections 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 --2- 172.21.15.122:0/2364975450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc8c004800 0x7fbc8c004c80 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 -- 172.21.15.122:0/2364975450 >> 172.21.15.122:0/2364975450 conn(0x7fbca00b1c80 msgr2=0x7fbca00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 -- 172.21.15.122:0/2364975450 shutdown_connections 2024-03-06T07:28:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.613+0000 7fbca5815640 1 -- 172.21.15.122:0/2364975450 wait complete. 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 Processor -- start 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 -- start start 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c004800 0x7fbca013e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 0x7fbca01595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbca015bfd0 0x7fbca0159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00b3800 con 0x7fbca013b100 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00b3680 con 0x7fbca015bfd0 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.615+0000 7fbca5815640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca0071e00 con 0x7fbc8c004800 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 0x7fbca01595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 0x7fbca01595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40968/0 (socket says 172.21.15.122:40968) 2024-03-06T07:28:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/1819988772 learned_addr learned my addr 172.21.15.122:0/1819988772 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9effd640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c004800 0x7fbca013e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9f7fe640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbca015bfd0 0x7fbca0159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c004800 msgr2=0x7fbca013e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c004800 0x7fbca013e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbca015bfd0 msgr2=0x7fbca0159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbca015bfd0 0x7fbca0159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 -- 172.21.15.122:0/1819988772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca013e650 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc9e7fc640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 0x7fbca01595c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fbc88000c00 tx=0x7fbc88001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88041020 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc88005480 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca015c550 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88007a30 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.616+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca013b9f0 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.617+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca004f330 con 0x7fbca013b100 2024-03-06T07:28:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.618+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc88044020 con 0x7fbca013b100 2024-03-06T07:28:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.618+0000 7fbc97fff640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc8006b4d0 0x7fbc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.618+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbc880abae0 con 0x7fbca013b100 2024-03-06T07:28:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.618+0000 7fbc9effd640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc8006b4d0 0x7fbc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.619+0000 7fbc9effd640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc8006b4d0 0x7fbc8006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009c40 tx=0x7fbc98009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.623+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc8804b030 con 0x7fbca013b100 2024-03-06T07:28:26.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.806+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbca0154a90 con 0x7fbca013b100 2024-03-06T07:28:26.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.807+0000 7fbc97fff640 1 -- 172.21.15.122:0/1819988772 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v114) v1 ==== 80+0+122 (secure 0 0 0) 0x7fbca013e650 con 0x7fbca013b100 2024-03-06T07:28:26.809 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122:6837/386534812 2024-03-07T07:27:28.499450+0000 2024-03-06T07:28:26.809 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122:6836/386534812 2024-03-07T07:27:28.499450+0000 2024-03-06T07:28:26.809 INFO:teuthology.orchestra.run.smithi122.stderr:listed 2 entries 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc8006b4d0 msgr2=0x7fbc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc8006b4d0 0x7fbc8006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009c40 tx=0x7fbc98009190 comp rx=0 tx=0).stop 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 msgr2=0x7fbca01595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 0x7fbca01595c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fbc88000c00 tx=0x7fbc88001120 comp rx=0 tx=0).stop 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 shutdown_connections 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc8006b4d0 0x7fbc8006d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbca015bfd0 0x7fbca0159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbca013b100 0x7fbca01595c0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 --2- 172.21.15.122:0/1819988772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc8c004800 0x7fbca013e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 >> 172.21.15.122:0/1819988772 conn(0x7fbca00b1c80 msgr2=0x7fbca0153fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 shutdown_connections 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:26.809+0000 7fbca5815640 1 -- 172.21.15.122:0/1819988772 wait complete. 2024-03-06T07:28:26.825 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:28:26.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:26.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:26.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[74706]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:26.852 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:28:26.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:26.852 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:27.017 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:28:27.017 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.036 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:27.036 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:28:27.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.036 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:27.094 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:28:27.094 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:28:27.095 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.110 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:27.110 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:28:27.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.110 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:27.164 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:28:27.165 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:28:27.165 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:27.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:28:27.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.183 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:27.240 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:28:27.241 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.339+0000 7fe3d8d7a640 1 Processor -- start 2024-03-06T07:28:27.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.339+0000 7fe3d8d7a640 1 -- start start 2024-03-06T07:28:27.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d8d7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe3d415dfe0 0x7fe3d40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d8d7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d8d7a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe3d40d27f0 0x7fe3d40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d8d7a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d40d2c50 con 0x7fe3d40d27f0 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d8d7a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d40b31e0 con 0x7fe3d415dfe0 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d8d7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40b3d60 con 0x7fe3d40cf830 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe3d40d27f0 0x7fe3d40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe3d40d27f0 0x7fe3d40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40980/0 (socket says 172.21.15.122:40980) 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1021806396 learned_addr learned my addr 172.21.15.122:0/1021806396 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1021806396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 msgr2=0x7fe3d40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 --2- 172.21.15.122:0/1021806396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1021806396 >> v2:172.21.15.169:3300/0 conn(0x7fe3d415dfe0 msgr2=0x7fe3d40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 --2- 172.21.15.122:0/1021806396 >> v2:172.21.15.169:3300/0 conn(0x7fe3d415dfe0 0x7fe3d40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.340+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1021806396 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40b3430 con 0x7fe3d40d27f0 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.341+0000 7fe3d2d9d640 1 --2- 172.21.15.122:0/1021806396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d40d27f0 0x7fe3d40d0300 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4067990 tx=0x7fe3cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7144ce96739c776f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.341+0000 7fe3d1d9b640 1 -- 172.21.15.122:0/1021806396 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc02ca70 con 0x7fe3d40d27f0 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.341+0000 7fe3d1d9b640 1 -- 172.21.15.122:0/1021806396 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc02cc10 con 0x7fe3d40d27f0 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.341+0000 7fe3d1d9b640 1 -- 172.21.15.122:0/1021806396 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc0343f0 con 0x7fe3d40d27f0 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1021806396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d40d27f0 msgr2=0x7fe3d40d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1021806396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d40d27f0 0x7fe3d40d0300 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4067990 tx=0x7fe3cc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1021806396 shutdown_connections 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1021806396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d40d27f0 0x7fe3d40d0300 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1021806396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1021806396 >> v2:172.21.15.169:3300/0 conn(0x7fe3d415dfe0 0x7fe3d40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1021806396 >> 172.21.15.122:0/1021806396 conn(0x7fe3d40d3cd0 msgr2=0x7fe3d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1021806396 shutdown_connections 2024-03-06T07:28:27.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.342+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1021806396 wait complete. 2024-03-06T07:28:27.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.343+0000 7fe3d8d7a640 1 Processor -- start 2024-03-06T07:28:27.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.343+0000 7fe3d8d7a640 1 -- start start 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d4158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3d40d27f0 0x7fe3d415fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 0x7fe3d415b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40d1c70 con 0x7fe3d415dfe0 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40b39f0 con 0x7fe3d40d27f0 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40d1df0 con 0x7fe3d40cf830 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 0x7fe3d415b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 0x7fe3d415b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40990/0 (socket says 172.21.15.122:40990) 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1104821259 learned_addr learned my addr 172.21.15.122:0/1104821259 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d259c640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d4158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3cbfff640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3d40d27f0 0x7fe3d415fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 msgr2=0x7fe3d4158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d4158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3d40d27f0 msgr2=0x7fe3d415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3d40d27f0 0x7fe3d415fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 -- 172.21.15.122:0/1104821259 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40ccbf0 con 0x7fe3d415dfe0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d2d9d640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 0x7fe3d415b9d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc036040 tx=0x7fe3cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc043020 con 0x7fe3d415dfe0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc03e450 con 0x7fe3d415dfe0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d40cbb50 con 0x7fe3d415dfe0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.344+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc03e820 con 0x7fe3d415dfe0 2024-03-06T07:28:27.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.345+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d40cc3f0 con 0x7fe3d415dfe0 2024-03-06T07:28:27.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.346+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d40cbf40 con 0x7fe3d415dfe0 2024-03-06T07:28:27.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.346+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3cc046020 con 0x7fe3d415dfe0 2024-03-06T07:28:27.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.346+0000 7fe3caffd640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.346+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe3cc0acc70 con 0x7fe3d415dfe0 2024-03-06T07:28:27.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.346+0000 7fe3d259c640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.347+0000 7fe3d259c640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe3b406b4d0 0x7fe3b406d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40ca2e0 tx=0x7fe3c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.351+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3cc04d030 con 0x7fe3d415dfe0 2024-03-06T07:28:27.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.566+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3d40ceb30 con 0x7fe3d415dfe0 2024-03-06T07:28:27.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.567+0000 7fe3caffd640 1 -- 172.21.15.122:0/1104821259 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fe3cc07ba60 con 0x7fe3d415dfe0 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe3b406b4d0 msgr2=0x7fe3b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe3b406b4d0 0x7fe3b406d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40ca2e0 tx=0x7fe3c4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 msgr2=0x7fe3d415b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 0x7fe3d415b9d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc036040 tx=0x7fe3cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 shutdown_connections 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3d415dfe0 0x7fe3d415b9d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3d40d27f0 0x7fe3d415fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 --2- 172.21.15.122:0/1104821259 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe3d40cf830 0x7fe3d4158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 >> 172.21.15.122:0/1104821259 conn(0x7fe3d40d3cd0 msgr2=0x7fe3d40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 shutdown_connections 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.570+0000 7fe3d8d7a640 1 -- 172.21.15.122:0/1104821259 wait complete. 2024-03-06T07:28:27.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:27.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.682+0000 7fb95c713640 1 Processor -- start 2024-03-06T07:28:27.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.683+0000 7fb95c713640 1 -- start start 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95c713640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb9540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95c713640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb9540b41f0 0x7fb9540b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95c713640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9540b3430 con 0x7fb95415dfe0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95c713640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9540d2c50 con 0x7fb9540b41f0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95c713640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540b31e0 con 0x7fb9540d22b0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95a4af640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb95415dfe0 0x7fb9540cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41024/0 (socket says 172.21.15.122:41024) 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95a4af640 1 -- 172.21.15.122:0/3899844999 learned_addr learned my addr 172.21.15.122:0/3899844999 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588053205 0 0) 0x7fb9540b3430 con 0x7fb95415dfe0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93c003520 con 0x7fb95415dfe0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb959cae640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb9540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.684+0000 7fb95acb0640 1 --2- 172.21.15.122:0/3899844999 >> v2:172.21.15.169:3300/0 conn(0x7fb9540b41f0 0x7fb9540b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 178976687 0 0) 0x7fb93c003520 con 0x7fb95415dfe0 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 msgr2=0x7fb9540d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb9540d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 >> v2:172.21.15.169:3300/0 conn(0x7fb9540b41f0 msgr2=0x7fb9540b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 --2- 172.21.15.122:0/3899844999 >> v2:172.21.15.169:3300/0 conn(0x7fb9540b41f0 0x7fb9540b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9540b3710 con 0x7fb95415dfe0 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb9440019f0 con 0x7fb95415dfe0 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 >> v1:172.21.15.122:6789/0 conn(0x7fb95415dfe0 legacy=0x7fb9540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb93c0039b0 0x7fb93c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb93c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb93c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540b31e0 con 0x7fb93c004800 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c00b9c0 con 0x7fb93c0039b0 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c00bb40 con 0x7fb9540d22b0 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb959cae640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb93c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb95a4af640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb93c0039b0 0x7fb93c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb95acb0640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb93c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb959cae640 1 -- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 msgr2=0x7fb93c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb959cae640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb93c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb959cae640 1 -- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb93c0039b0 msgr2=0x7fb93c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb959cae640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb93c0039b0 0x7fb93c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb959cae640 1 -- 172.21.15.122:0/3899844999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c003520 con 0x7fb93c004800 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.685+0000 7fb95acb0640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540d22b0 0x7fb93c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb959cae640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb93c004c80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fb948009ac0 tx=0x7fb93c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb130064cccaa71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948039070 con 0x7fb93c004800 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94802c6b0 con 0x7fb93c004800 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb9594ad640 1 -- 172.21.15.122:0/3899844999 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948009690 con 0x7fb93c004800 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb95c713640 1 -- 172.21.15.122:0/3899844999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 msgr2=0x7fb93c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb95c713640 1 --2- 172.21.15.122:0/3899844999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb93c004c80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fb948009ac0 tx=0x7fb93c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.686+0000 7fb95acb0640 1 -- 172.21.15.122:0/3899844999 reap_dead start 2024-03-06T07:28:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.687+0000 7fb95c713640 1 -- 172.21.15.122:0/3899844999 shutdown_connections 2024-03-06T07:28:27.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.687+0000 7fb95c713640 1 -- 172.21.15.122:0/3899844999 >> 172.21.15.122:0/3899844999 conn(0x7fb9540d3cd0 msgr2=0x7fb9540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.687+0000 7fb95c713640 1 -- 172.21.15.122:0/3899844999 shutdown_connections 2024-03-06T07:28:27.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.687+0000 7fb95c713640 1 -- 172.21.15.122:0/3899844999 wait complete. 2024-03-06T07:28:27.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.688+0000 7fb95c713640 1 Processor -- start 2024-03-06T07:28:27.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.688+0000 7fb95c713640 1 -- start start 2024-03-06T07:28:27.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.688+0000 7fb95c713640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb95415f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540b9600 0x7fb9540cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9540ccac0 0x7fb9540ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540cfa00 con 0x7fb93c004800 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540b3d60 con 0x7fb9540ccac0 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540d1c70 con 0x7fb9540b9600 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb95415f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb95415f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41004/0 (socket says 172.21.15.122:41004) 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 -- 172.21.15.122:0/1483582161 learned_addr learned my addr 172.21.15.122:0/1483582161 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb959cae640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540b9600 0x7fb9540cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95acb0640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9540ccac0 0x7fb9540ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 -- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540b9600 msgr2=0x7fb9540cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540b9600 0x7fb9540cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 -- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9540ccac0 msgr2=0x7fb9540ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9540ccac0 0x7fb9540ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 -- 172.21.15.122:0/1483582161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9540cd480 con 0x7fb93c004800 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95a4af640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb95415f410 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb944007a50 tx=0x7fb94400a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944017020 con 0x7fb93c004800 2024-03-06T07:28:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb944001a30 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb954160e20 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944013600 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.689+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb954160340 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.690+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95404f330 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.691+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb94401a020 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.691+0000 7fb952ffd640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb94006b500 0x7fb94006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.691+0000 7fb959cae640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb94006b500 0x7fb94006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.691+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb94408adb0 con 0x7fb93c004800 2024-03-06T07:28:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.692+0000 7fb959cae640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb94006b500 0x7fb94006d9c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb948036000 tx=0x7fb948030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.696+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb944021020 con 0x7fb93c004800 2024-03-06T07:28:27.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.876+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9540747a0 con 0x7fb93c004800 2024-03-06T07:28:27.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.877+0000 7fb952ffd640 1 -- 172.21.15.122:0/1483582161 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16909 (secure 0 0 0) 0x7fb944059ba0 con 0x7fb93c004800 2024-03-06T07:28:27.878 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:27.878 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":114,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:34.986316+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/386534812":"2024-03-07T07:27:28.499450+0000","172.21.15.122:6836/386534812":"2024-03-07T07:27:28.499450+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb94006b500 msgr2=0x7fb94006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb94006b500 0x7fb94006d9c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb948036000 tx=0x7fb948030000 comp rx=0 tx=0).stop 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 msgr2=0x7fb95415f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb95415f410 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb944007a50 tx=0x7fb94400a750 comp rx=0 tx=0).stop 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 shutdown_connections 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb94006b500 0x7fb94006d9c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9540ccac0 0x7fb9540ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9540b9600 0x7fb9540cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 --2- 172.21.15.122:0/1483582161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb93c004800 0x7fb95415f410 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 >> 172.21.15.122:0/1483582161 conn(0x7fb9540d3cd0 msgr2=0x7fb9540d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 shutdown_connections 2024-03-06T07:28:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.879+0000 7fb95c713640 1 -- 172.21.15.122:0/1483582161 wait complete. 2024-03-06T07:28:27.891 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.843386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.579477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:27.891 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.994+0000 7fb112381640 1 Processor -- start 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.994+0000 7fb112381640 1 -- start start 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb112381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c15def0 0x7fb10c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb112381640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb10c06cab0 0x7fb10c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb112381640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c06d030 con 0x7fb10c0ce880 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb112381640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c0cecf0 con 0x7fb10c06cab0 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb112381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0cf1c0 con 0x7fb10c15def0 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb10c06cab0 0x7fb10c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb10c06cab0 0x7fb10c0ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41020/0 (socket says 172.21.15.122:41020) 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 -- 172.21.15.122:0/3402150272 learned_addr learned my addr 172.21.15.122:0/3402150272 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10bfff640 1 --2- 172.21.15.122:0/3402150272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c15def0 0x7fb10c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 -- 172.21.15.122:0/3402150272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c15def0 msgr2=0x7fb10c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 --2- 172.21.15.122:0/3402150272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c15def0 0x7fb10c0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 -- 172.21.15.122:0/3402150272 >> v1:172.21.15.169:6789/0 conn(0x7fb10c0ce880 legacy=0x7fb10c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.995+0000 7fb10b7fe640 1 -- 172.21.15.122:0/3402150272 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c06d310 con 0x7fb10c06cab0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.996+0000 7fb10b7fe640 1 --2- 172.21.15.122:0/3402150272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0ce340 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb100009870 tx=0x7fb10002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c256d35e9ebc50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.996+0000 7fb10affd640 1 -- 172.21.15.122:0/3402150272 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb100034070 con 0x7fb10c06cab0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.996+0000 7fb10affd640 1 -- 172.21.15.122:0/3402150272 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb100030840 con 0x7fb10c06cab0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.996+0000 7fb10affd640 1 -- 172.21.15.122:0/3402150272 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb100030c10 con 0x7fb10c06cab0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 -- 172.21.15.122:0/3402150272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 msgr2=0x7fb10c0ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 --2- 172.21.15.122:0/3402150272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0ce340 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb100009870 tx=0x7fb10002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 -- 172.21.15.122:0/3402150272 shutdown_connections 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 --2- 172.21.15.122:0/3402150272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0ce340 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 --2- 172.21.15.122:0/3402150272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c15def0 0x7fb10c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 -- 172.21.15.122:0/3402150272 >> 172.21.15.122:0/3402150272 conn(0x7fb10c0b3690 msgr2=0x7fb10c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 -- 172.21.15.122:0/3402150272 shutdown_connections 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.997+0000 7fb112381640 1 -- 172.21.15.122:0/3402150272 wait complete. 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.998+0000 7fb112381640 1 Processor -- start 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.998+0000 7fb112381640 1 -- start start 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb112381640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb112381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c0ce880 0x7fb10c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb112381640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb10c15def0 0x7fb10c15c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb112381640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0cc3f0 con 0x7fb10c06cab0 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb112381640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0cc760 con 0x7fb10c15def0 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb112381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c15e2d0 con 0x7fb10c0ce880 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41036/0 (socket says 172.21.15.122:41036) 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 -- 172.21.15.122:0/4121599423 learned_addr learned my addr 172.21.15.122:0/4121599423 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10b7fe640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c0ce880 0x7fb10c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 -- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c0ce880 msgr2=0x7fb10c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c0ce880 0x7fb10c0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 -- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb10c15def0 msgr2=0x7fb10c15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb11091e640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb10c15def0 0x7fb10c15c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb10c15def0 0x7fb10c15c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10bfff640 1 -- 172.21.15.122:0/4121599423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c0cb9d0 con 0x7fb10c06cab0 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb10b7fe640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c0ce880 0x7fb10c0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:27.999+0000 7fb11091e640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb10c15def0 0x7fb10c15c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.000+0000 7fb10bfff640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0c8b90 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c7f0 tx=0x7fb0fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.000+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc015070 con 0x7fb10c06cab0 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.000+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc004220 con 0x7fb10c06cab0 2024-03-06T07:28:28.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.000+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c0b5190 con 0x7fb10c06cab0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.000+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc007620 con 0x7fb10c06cab0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.000+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c0b5750 con 0x7fb10c06cab0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.001+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c09fcf0 con 0x7fb10c06cab0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.001+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0fc0079d0 con 0x7fb10c06cab0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.001+0000 7fb108ff9640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0f806b4d0 0x7fb0f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.002+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb0fc08a5c0 con 0x7fb10c06cab0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.002+0000 7fb10b7fe640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0f806b4d0 0x7fb0f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.002+0000 7fb10b7fe640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0f806b4d0 0x7fb0f806d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb100036000 tx=0x7fb10003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.006+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0fc0593b0 con 0x7fb10c06cab0 2024-03-06T07:28:28.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.180+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb10c068ff0 con 0x7fb10c06cab0 2024-03-06T07:28:28.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.181+0000 7fb108ff9640 1 -- 172.21.15.122:0/4121599423 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0fc058cb0 con 0x7fb10c06cab0 2024-03-06T07:28:28.182 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.182+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0f806b4d0 msgr2=0x7fb0f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.182+0000 7fb112381640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0f806b4d0 0x7fb0f806d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb100036000 tx=0x7fb10003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.182+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 msgr2=0x7fb10c0c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.182+0000 7fb112381640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0c8b90 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c7f0 tx=0x7fb0fc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 shutdown_connections 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0f806b4d0 0x7fb0f806d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb10c15def0 0x7fb10c15c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb10c0ce880 0x7fb10c0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 --2- 172.21.15.122:0/4121599423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c06cab0 0x7fb10c0c8b90 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 >> 172.21.15.122:0/4121599423 conn(0x7fb10c0b3690 msgr2=0x7fb10c06f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 shutdown_connections 2024-03-06T07:28:28.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.183+0000 7fb112381640 1 -- 172.21.15.122:0/4121599423 wait complete. 2024-03-06T07:28:28.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.298+0000 7f5e36cb5640 1 Processor -- start 2024-03-06T07:28:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.298+0000 7f5e36cb5640 1 -- start start 2024-03-06T07:28:28.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e36cb5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e300b1e40 0x7f5e30168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e36cb5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e300d2730 con 0x7f5e300d4860 2024-03-06T07:28:28.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e36cb5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e301618b0 con 0x7f5e301614d0 2024-03-06T07:28:28.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e36cb5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e300d4cf0 con 0x7f5e300b1e40 2024-03-06T07:28:28.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e35252640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5e300b1e40 0x7f5e30168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e27fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5e300d4860 0x7f5e300d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41040/0 (socket says 172.21.15.122:41040) 2024-03-06T07:28:28.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.299+0000 7f5e27fff640 1 -- 172.21.15.122:0/1002438213 learned_addr learned my addr 172.21.15.122:0/1002438213 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.300+0000 7f5e35252640 1 -- 172.21.15.122:0/1002438213 >> v1:172.21.15.169:6789/0 conn(0x7f5e301614d0 legacy=0x7f5e300d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.300+0000 7f5e35252640 1 -- 172.21.15.122:0/1002438213 >> v1:172.21.15.122:6789/0 conn(0x7f5e300d4860 legacy=0x7f5e300d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.300+0000 7f5e35252640 1 -- 172.21.15.122:0/1002438213 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e300cf400 con 0x7f5e300b1e40 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.300+0000 7f5e277fe640 1 -- 172.21.15.122:0/1002438213 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564281593 0 0) 0x7f5e300d2730 con 0x7f5e300d4860 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e35252640 1 --2- 172.21.15.122:0/1002438213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300b1e40 0x7f5e30168b60 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5e300cf5d0 tx=0x7f5e2802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ee2a4a807f42c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e277fe640 1 -- 172.21.15.122:0/1002438213 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28039070 con 0x7f5e300b1e40 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e277fe640 1 -- 172.21.15.122:0/1002438213 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2802cc50 con 0x7f5e300b1e40 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e277fe640 1 -- 172.21.15.122:0/1002438213 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28034440 con 0x7f5e300b1e40 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 -- 172.21.15.122:0/1002438213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300b1e40 msgr2=0x7f5e30168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/1002438213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300b1e40 0x7f5e30168b60 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5e300cf5d0 tx=0x7f5e2802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 -- 172.21.15.122:0/1002438213 shutdown_connections 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/1002438213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300b1e40 0x7f5e30168b60 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 -- 172.21.15.122:0/1002438213 >> 172.21.15.122:0/1002438213 conn(0x7f5e300d3d40 msgr2=0x7f5e300d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 -- 172.21.15.122:0/1002438213 shutdown_connections 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.301+0000 7f5e36cb5640 1 -- 172.21.15.122:0/1002438213 wait complete. 2024-03-06T07:28:28.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.303+0000 7f5e36cb5640 1 Processor -- start 2024-03-06T07:28:28.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.303+0000 7f5e36cb5640 1 -- start start 2024-03-06T07:28:28.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e36cb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e300b1e40 0x7f5e300c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e36cb5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 0x7f5e30149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e36cb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e301614d0 0x7f5e300cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e36cb5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300cfc60 con 0x7f5e300d4860 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e36cb5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300cf980 con 0x7f5e301614d0 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e36cb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300b0910 con 0x7f5e300b1e40 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e27fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 0x7f5e30149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e27fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 0x7f5e30149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41048/0 (socket says 172.21.15.122:41048) 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e27fff640 1 -- 172.21.15.122:0/2344212820 learned_addr learned my addr 172.21.15.122:0/2344212820 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e34a51640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e300b1e40 0x7f5e300c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.304+0000 7f5e35252640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e301614d0 0x7f5e300cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e27fff640 1 -- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e300b1e40 msgr2=0x7f5e300c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e27fff640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e300b1e40 0x7f5e300c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e27fff640 1 -- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e301614d0 msgr2=0x7f5e300cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e27fff640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e301614d0 0x7f5e300cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e27fff640 1 -- 172.21.15.122:0/2344212820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3014a1f0 con 0x7f5e300d4860 2024-03-06T07:28:28.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e27fff640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 0x7f5e30149850 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000c4a0 tx=0x7f5e2000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20018070 con 0x7f5e300d4860 2024-03-06T07:28:28.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e200074e0 con 0x7f5e300d4860 2024-03-06T07:28:28.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3014a4d0 con 0x7f5e300d4860 2024-03-06T07:28:28.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20004a20 con 0x7f5e300d4860 2024-03-06T07:28:28.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.305+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3014aa90 con 0x7f5e300d4860 2024-03-06T07:28:28.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.306+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e300680d0 con 0x7f5e300d4860 2024-03-06T07:28:28.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.306+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e2001b070 con 0x7f5e300d4860 2024-03-06T07:28:28.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.307+0000 7f5e257fa640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.307+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5e20089510 con 0x7f5e300d4860 2024-03-06T07:28:28.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.307+0000 7f5e34a51640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.307+0000 7f5e34a51640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c000f60 tx=0x7f5e2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.311+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e300680d0 con 0x7f5e300d4860 2024-03-06T07:28:28.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.488+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e300a1c10 con 0x7f5e300d4860 2024-03-06T07:28:28.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.489+0000 7f5e257fa640 1 -- 172.21.15.122:0/2344212820 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e20057cb0 con 0x7f5e300d4860 2024-03-06T07:28:28.490 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:28:28.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e1c06b4d0 msgr2=0x7f5e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c000f60 tx=0x7f5e2c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 msgr2=0x7f5e30149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 0x7f5e30149850 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000c4a0 tx=0x7f5e2000c960 comp rx=0 tx=0).stop 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 shutdown_connections 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5e1c06b4d0 0x7f5e1c06d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e301614d0 0x7f5e300cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e300d4860 0x7f5e30149850 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 --2- 172.21.15.122:0/2344212820 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5e300b1e40 0x7f5e300c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 >> 172.21.15.122:0/2344212820 conn(0x7f5e300d3d40 msgr2=0x7f5e300d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 shutdown_connections 2024-03-06T07:28:28.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.491+0000 7f5e36cb5640 1 -- 172.21.15.122:0/2344212820 wait complete. 2024-03-06T07:28:28.503 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:28.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.607+0000 7fe2c6a4e640 1 Processor -- start 2024-03-06T07:28:28.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.607+0000 7fe2c6a4e640 1 -- start start 2024-03-06T07:28:28.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2c6a4e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2c00d10c0 0x7fe2c00d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2c6a4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c00d14a0 0x7fe2c015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2c6a4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00cfa50 con 0x7fe2c015eb00 2024-03-06T07:28:28.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2c6a4e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c015c050 con 0x7fe2c00d10c0 2024-03-06T07:28:28.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2c6a4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00d0060 con 0x7fe2c00d14a0 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2c00d10c0 0x7fe2c00d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2c00d10c0 0x7fe2c00d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41060/0 (socket says 172.21.15.122:41060) 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2c4feb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe2c015eb00 0x7fe2c00cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41050/0 (socket says 172.21.15.122:41050) 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 -- 172.21.15.122:0/772193479 learned_addr learned my addr 172.21.15.122:0/772193479 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bf7fe640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c00d14a0 0x7fe2c015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 -- 172.21.15.122:0/772193479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c00d14a0 msgr2=0x7fe2c015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c00d14a0 0x7fe2c015e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 -- 172.21.15.122:0/772193479 >> v1:172.21.15.122:6789/0 conn(0x7fe2c015eb00 legacy=0x7fe2c00cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2beffd640 1 -- 172.21.15.122:0/772193479 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277789365 0 0) 0x7fe2c00cfa50 con 0x7fe2c015eb00 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bffff640 1 -- 172.21.15.122:0/772193479 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c00d0a30 con 0x7fe2c00d10c0 2024-03-06T07:28:28.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.608+0000 7fe2bf7fe640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c00d14a0 0x7fe2c015e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.609+0000 7fe2bffff640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c00d04f0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe2c00d01e0 tx=0x7fe2b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb7cea7bbe2c835 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.609+0000 7fe2beffd640 1 -- 172.21.15.122:0/772193479 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0034070 con 0x7fe2c00d10c0 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.609+0000 7fe2beffd640 1 -- 172.21.15.122:0/772193479 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0030610 con 0x7fe2c00d10c0 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.609+0000 7fe2beffd640 1 -- 172.21.15.122:0/772193479 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00309e0 con 0x7fe2c00d10c0 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/772193479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 msgr2=0x7fe2c00d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c00d04f0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe2c00d01e0 tx=0x7fe2b002b960 comp rx=0 tx=0).stop 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/772193479 shutdown_connections 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c00d14a0 0x7fe2c015e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/772193479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c00d04f0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/772193479 >> 172.21.15.122:0/772193479 conn(0x7fe2c00b3800 msgr2=0x7fe2c00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/772193479 shutdown_connections 2024-03-06T07:28:28.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.610+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/772193479 wait complete. 2024-03-06T07:28:28.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.611+0000 7fe2c6a4e640 1 Processor -- start 2024-03-06T07:28:28.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.611+0000 7fe2c6a4e640 1 -- start start 2024-03-06T07:28:28.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c6a4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c014a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c6a4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c00d14a0 0x7fe2c00b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c6a4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c015eb00 0x7fe2c0157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c6a4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c015f440 con 0x7fe2c00d10c0 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c6a4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c015f160 con 0x7fe2c00d14a0 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c6a4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00cfcf0 con 0x7fe2c015eb00 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c014a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c014a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41070/0 (socket says 172.21.15.122:41070) 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 -- 172.21.15.122:0/1771998808 learned_addr learned my addr 172.21.15.122:0/1771998808 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 -- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c015eb00 msgr2=0x7fe2c0157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2c4feb640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c015eb00 0x7fe2c0157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bf7fe640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c00d14a0 0x7fe2c00b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c015eb00 0x7fe2c0157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 -- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c00d14a0 msgr2=0x7fe2c00b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c00d14a0 0x7fe2c00b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 -- 172.21.15.122:0/1771998808 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c00b68b0 con 0x7fe2c00d10c0 2024-03-06T07:28:28.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.612+0000 7fe2bffff640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c014a660 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0000c00 tx=0x7fe2b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.613+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0034040 con 0x7fe2c00d10c0 2024-03-06T07:28:28.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.613+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0004030 con 0x7fe2c00d10c0 2024-03-06T07:28:28.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.613+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0004420 con 0x7fe2c00d10c0 2024-03-06T07:28:28.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.613+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c00b6b90 con 0x7fe2c00d10c0 2024-03-06T07:28:28.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.613+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c00b1670 con 0x7fe2c00d10c0 2024-03-06T07:28:28.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.614+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c0068ff0 con 0x7fe2c00d10c0 2024-03-06T07:28:28.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.615+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2b0037070 con 0x7fe2c00d10c0 2024-03-06T07:28:28.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.615+0000 7fe2bcff9640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe2ac06b4d0 0x7fe2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.615+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe2b00abde0 con 0x7fe2c00d10c0 2024-03-06T07:28:28.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.615+0000 7fe2bf7fe640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe2ac06b4d0 0x7fe2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.615+0000 7fe2bf7fe640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe2ac06b4d0 0x7fe2ac06d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe2c00b3180 tx=0x7fe2b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.620+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2b003f070 con 0x7fe2c00d10c0 2024-03-06T07:28:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.835+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2c015e7d0 con 0x7fe2c00d10c0 2024-03-06T07:28:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.835+0000 7fe2bcff9640 1 -- 172.21.15.122:0/1771998808 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe2b007ac60 con 0x7fe2c00d10c0 2024-03-06T07:28:28.836 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:28:28.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe2ac06b4d0 msgr2=0x7fe2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe2ac06b4d0 0x7fe2ac06d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe2c00b3180 tx=0x7fe2b4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 msgr2=0x7fe2c014a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c014a660 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0000c00 tx=0x7fe2b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 shutdown_connections 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe2ac06b4d0 0x7fe2ac06d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2c015eb00 0x7fe2c0157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c00d14a0 0x7fe2c00b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 --2- 172.21.15.122:0/1771998808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c00d10c0 0x7fe2c014a660 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 >> 172.21.15.122:0/1771998808 conn(0x7fe2c00b3800 msgr2=0x7fe2c0070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 shutdown_connections 2024-03-06T07:28:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.837+0000 7fe2c6a4e640 1 -- 172.21.15.122:0/1771998808 wait complete. 2024-03-06T07:28:28.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:28.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.950+0000 7fb176447640 1 Processor -- start 2024-03-06T07:28:28.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.950+0000 7fb176447640 1 -- start start 2024-03-06T07:28:28.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.951+0000 7fb176447640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb17015dfe0 0x7fb1700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.951+0000 7fb176447640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700cf830 0x7fb1700d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.951+0000 7fb176447640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700b3430 con 0x7fb1700d27f0 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.951+0000 7fb176447640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1700d2c50 con 0x7fb17015dfe0 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.951+0000 7fb176447640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1700b31e0 con 0x7fb1700cf830 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb1749e4640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb1700d27f0 0x7fb1700d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41056/0 (socket says 172.21.15.122:41056) 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb1749e4640 1 -- 172.21.15.122:0/2335641989 learned_addr learned my addr 172.21.15.122:0/2335641989 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16ffff640 1 --2- 172.21.15.122:0/2335641989 >> v2:172.21.15.169:3300/0 conn(0x7fb17015dfe0 0x7fb1700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16f7fe640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700cf830 0x7fb1700d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16effd640 1 -- 172.21.15.122:0/2335641989 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873435743 0 0) 0x7fb1700b3430 con 0x7fb1700d27f0 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16effd640 1 -- 172.21.15.122:0/2335641989 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb158003520 con 0x7fb1700d27f0 2024-03-06T07:28:28.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16effd640 1 -- 172.21.15.122:0/2335641989 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834766379 0 0) 0x7fb158003520 con 0x7fb1700d27f0 2024-03-06T07:28:28.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16ffff640 1 -- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700cf830 msgr2=0x7fb1700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16ffff640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700cf830 0x7fb1700d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16ffff640 1 -- 172.21.15.122:0/2335641989 >> v1:172.21.15.122:6789/0 conn(0x7fb1700d27f0 legacy=0x7fb1700d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16ffff640 1 -- 172.21.15.122:0/2335641989 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1700b3710 con 0x7fb17015dfe0 2024-03-06T07:28:28.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.952+0000 7fb16f7fe640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700cf830 0x7fb1700d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:28.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb16ffff640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb17015dfe0 0x7fb1700cf2f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb1600097a0 tx=0x7fb16002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f225a61c42e66442 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb16effd640 1 -- 172.21.15.122:0/2335641989 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160037070 con 0x7fb17015dfe0 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb16effd640 1 -- 172.21.15.122:0/2335641989 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb160030510 con 0x7fb17015dfe0 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb16effd640 1 -- 172.21.15.122:0/2335641989 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1680019f0 con 0x7fb17015dfe0 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 -- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb17015dfe0 msgr2=0x7fb1700cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb17015dfe0 0x7fb1700cf2f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb1600097a0 tx=0x7fb16002b790 comp rx=0 tx=0).stop 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 -- 172.21.15.122:0/2335641989 shutdown_connections 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700cf830 0x7fb1700d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 --2- 172.21.15.122:0/2335641989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb17015dfe0 0x7fb1700cf2f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 -- 172.21.15.122:0/2335641989 >> 172.21.15.122:0/2335641989 conn(0x7fb1700d3cd0 msgr2=0x7fb1700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 -- 172.21.15.122:0/2335641989 shutdown_connections 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.954+0000 7fb176447640 1 -- 172.21.15.122:0/2335641989 wait complete. 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.955+0000 7fb176447640 1 Processor -- start 2024-03-06T07:28:28.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 -- start start 2024-03-06T07:28:28.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1700cf830 0x7fb1700cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700d27f0 0x7fb1700b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 0x7fb1701f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1700d1c70 con 0x7fb17015dfe0 2024-03-06T07:28:28.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1700b3d60 con 0x7fb1700cf830 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb176447640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1700d1df0 con 0x7fb1700d27f0 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb1749e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 0x7fb1701f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb1749e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 0x7fb1701f6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41086/0 (socket says 172.21.15.122:41086) 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.956+0000 7fb1749e4640 1 -- 172.21.15.122:0/3309074956 learned_addr learned my addr 172.21.15.122:0/3309074956 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb16f7fe640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700d27f0 0x7fb1700b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb16ffff640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1700cf830 0x7fb1700cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb1749e4640 1 -- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700d27f0 msgr2=0x7fb1700b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb1749e4640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700d27f0 0x7fb1700b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb1749e4640 1 -- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1700cf830 msgr2=0x7fb1700cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb1749e4640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1700cf830 0x7fb1700cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb1749e4640 1 -- 172.21.15.122:0/3309074956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1700b9160 con 0x7fb17015dfe0 2024-03-06T07:28:28.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb1749e4640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 0x7fb1701f6d60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fb168001890 tx=0x7fb16800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb168012400 con 0x7fb17015dfe0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1680125c0 con 0x7fb17015dfe0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1700b9440 con 0x7fb17015dfe0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1680129b0 con 0x7fb17015dfe0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.957+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1700b9a00 con 0x7fb17015dfe0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.958+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1700ce3c0 con 0x7fb17015dfe0 2024-03-06T07:28:28.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.959+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb168012b50 con 0x7fb17015dfe0 2024-03-06T07:28:28.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.959+0000 7fb16cff9640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb15c06b4d0 0x7fb15c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.959+0000 7fb16ffff640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb15c06b4d0 0x7fb15c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.959+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb16808a960 con 0x7fb17015dfe0 2024-03-06T07:28:28.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.960+0000 7fb16ffff640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb15c06b4d0 0x7fb15c06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb160034000 tx=0x7fb16002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:28.967+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1700ce3c0 con 0x7fb17015dfe0 2024-03-06T07:28:29.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.177+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17004f330 con 0x7fb17015dfe0 2024-03-06T07:28:29.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.178+0000 7fb16cff9640 1 -- 172.21.15.122:0/3309074956 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fb168059190 con 0x7fb17015dfe0 2024-03-06T07:28:29.179 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:29.179 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:29.179 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.179+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb15c06b4d0 msgr2=0x7fb15c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.179+0000 7fb176447640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb15c06b4d0 0x7fb15c06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb160034000 tx=0x7fb16002f000 comp rx=0 tx=0).stop 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.179+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 msgr2=0x7fb1701f6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.179+0000 7fb176447640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 0x7fb1701f6d60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fb168001890 tx=0x7fb16800a430 comp rx=0 tx=0).stop 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 shutdown_connections 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb15c06b4d0 0x7fb15c06d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb17015dfe0 0x7fb1701f6d60 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb1700d27f0 0x7fb1700b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 --2- 172.21.15.122:0/3309074956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1700cf830 0x7fb1700cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 >> 172.21.15.122:0/3309074956 conn(0x7fb1700d3cd0 msgr2=0x7fb1700cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 shutdown_connections 2024-03-06T07:28:29.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.180+0000 7fb176447640 1 -- 172.21.15.122:0/3309074956 wait complete. 2024-03-06T07:28:29.192 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:29.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.291+0000 7f3da7c2a640 1 Processor -- start 2024-03-06T07:28:29.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.291+0000 7f3da7c2a640 1 -- start start 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.292+0000 7f3da7c2a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3da015dfe0 0x7f3da00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.292+0000 7f3da7c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da00cf830 0x7f3da00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.292+0000 7f3da7c2a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da00b3430 con 0x7f3da00d27f0 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.292+0000 7f3da7c2a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da00d2c50 con 0x7f3da015dfe0 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.292+0000 7f3da7c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da00b31e0 con 0x7f3da00cf830 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da61c7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3da00d27f0 0x7f3da00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41072/0 (socket says 172.21.15.122:41072) 2024-03-06T07:28:29.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da61c7640 1 -- 172.21.15.122:0/2162868277 learned_addr learned my addr 172.21.15.122:0/2162868277 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da59c6640 1 --2- 172.21.15.122:0/2162868277 >> v2:172.21.15.169:3300/0 conn(0x7f3da015dfe0 0x7f3da00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da51c5640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da00cf830 0x7f3da00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da49c4640 1 -- 172.21.15.122:0/2162868277 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028505389 0 0) 0x7f3da00b3430 con 0x7f3da00d27f0 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da49c4640 1 -- 172.21.15.122:0/2162868277 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d8c003520 con 0x7f3da00d27f0 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da49c4640 1 -- 172.21.15.122:0/2162868277 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282868092 0 0) 0x7f3d8c003520 con 0x7f3da00d27f0 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da59c6640 1 -- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da00cf830 msgr2=0x7f3da00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da59c6640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da00cf830 0x7f3da00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da59c6640 1 -- 172.21.15.122:0/2162868277 >> v1:172.21.15.122:6789/0 conn(0x7f3da00d27f0 legacy=0x7f3da00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da59c6640 1 -- 172.21.15.122:0/2162868277 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da00b3710 con 0x7f3da015dfe0 2024-03-06T07:28:29.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.293+0000 7f3da51c5640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da00cf830 0x7f3da00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.294+0000 7f3da59c6640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da015dfe0 0x7f3da00cf2f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0097a0 tx=0x7f3d9c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a17acdc684cec1b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.294+0000 7f3da49c4640 1 -- 172.21.15.122:0/2162868277 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c037070 con 0x7f3da015dfe0 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.294+0000 7f3da49c4640 1 -- 172.21.15.122:0/2162868277 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c030510 con 0x7f3da015dfe0 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.294+0000 7f3da49c4640 1 -- 172.21.15.122:0/2162868277 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d980019f0 con 0x7f3da015dfe0 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da015dfe0 msgr2=0x7f3da00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da015dfe0 0x7f3da00cf2f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0097a0 tx=0x7f3d9c02b790 comp rx=0 tx=0).stop 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2162868277 shutdown_connections 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da00cf830 0x7f3da00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2162868277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da015dfe0 0x7f3da00cf2f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2162868277 >> 172.21.15.122:0/2162868277 conn(0x7f3da00d3cd0 msgr2=0x7f3da00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2162868277 shutdown_connections 2024-03-06T07:28:29.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.295+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2162868277 wait complete. 2024-03-06T07:28:29.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.296+0000 7f3da7c2a640 1 Processor -- start 2024-03-06T07:28:29.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.296+0000 7f3da7c2a640 1 -- start start 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 0x7f3da01f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da00d27f0 0x7f3da01510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da015dfe0 0x7f3da00b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da00d1c70 con 0x7f3da00cf830 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da00b3d60 con 0x7f3da00d27f0 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da00d1df0 con 0x7f3da015dfe0 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 0x7f3da01f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 0x7f3da01f7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41088/0 (socket says 172.21.15.122:41088) 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 -- 172.21.15.122:0/2350072366 learned_addr learned my addr 172.21.15.122:0/2350072366 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da51c5640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da00d27f0 0x7f3da01510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da61c7640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da015dfe0 0x7f3da00b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 -- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da015dfe0 msgr2=0x7f3da00b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da015dfe0 0x7f3da00b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 -- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da00d27f0 msgr2=0x7f3da01510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da00d27f0 0x7f3da01510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 -- 172.21.15.122:0/2350072366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da0160b80 con 0x7f3da00cf830 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da59c6640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 0x7f3da01f7d60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c02b7d0 tx=0x7f3d9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c042070 con 0x7f3da00cf830 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c03e870 con 0x7f3da00cf830 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c037040 con 0x7f3da00cf830 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.297+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da0160e60 con 0x7f3da00cf830 2024-03-06T07:28:29.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.298+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da00b8570 con 0x7f3da00cf830 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.299+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da006a190 con 0x7f3da00cf830 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.299+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d9c045070 con 0x7f3da00cf830 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.299+0000 7f3d967fc640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d8406b4d0 0x7f3d8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.299+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3d9c0abfb0 con 0x7f3da00cf830 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.299+0000 7f3da51c5640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d8406b4d0 0x7f3d8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.300+0000 7f3da51c5640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d8406b4d0 0x7f3d8406d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009560 tx=0x7f3d90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.304+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3da006a190 con 0x7f3da00cf830 2024-03-06T07:28:29.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.506+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3da00b09c0 con 0x7f3da00cf830 2024-03-06T07:28:29.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.508+0000 7f3d967fc640 1 -- 172.21.15.122:0/2350072366 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16909 (secure 0 0 0) 0x7f3d9c07af50 con 0x7f3da00cf830 2024-03-06T07:28:29.509 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:29.509 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":114,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:34.986316+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/386534812":"2024-03-07T07:27:28.499450+0000","172.21.15.122:6836/386534812":"2024-03-07T07:27:28.499450+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:29.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d8406b4d0 msgr2=0x7f3d8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d8406b4d0 0x7f3d8406d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009560 tx=0x7f3d90008040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 msgr2=0x7f3da01f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 0x7f3da01f7d60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c02b7d0 tx=0x7f3d9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 shutdown_connections 2024-03-06T07:28:29.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d8406b4d0 0x7f3d8406d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3da015dfe0 0x7f3da00b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3da00d27f0 0x7f3da01510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 --2- 172.21.15.122:0/2350072366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3da00cf830 0x7f3da01f7d60 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 >> 172.21.15.122:0/2350072366 conn(0x7f3da00d3cd0 msgr2=0x7f3da00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 shutdown_connections 2024-03-06T07:28:29.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.510+0000 7f3da7c2a640 1 -- 172.21.15.122:0/2350072366 wait complete. 2024-03-06T07:28:29.523 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:29.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:29.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.624+0000 7f25a3afb640 1 Processor -- start 2024-03-06T07:28:29.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.624+0000 7f25a3afb640 1 -- start start 2024-03-06T07:28:29.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a3afb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f259c0b3060 0x7f259c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a3afb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f259c15bf30 0x7f259c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a3afb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c15f2d0 con 0x7f259c13dc10 2024-03-06T07:28:29.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a3afb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0cf3f0 con 0x7f259c15bf30 2024-03-06T07:28:29.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a3afb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c15b210 con 0x7f259c0b3060 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f259c15bf30 0x7f259c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f259c15bf30 0x7f259c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41094/0 (socket says 172.21.15.122:41094) 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 -- 172.21.15.122:0/496644110 learned_addr learned my addr 172.21.15.122:0/496644110 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a1096640 1 --2- 172.21.15.122:0/496644110 >> v2:172.21.15.169:3300/0 conn(0x7f259c0b3060 0x7f259c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 -- 172.21.15.122:0/496644110 >> v2:172.21.15.169:3300/0 conn(0x7f259c0b3060 msgr2=0x7f259c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 --2- 172.21.15.122:0/496644110 >> v2:172.21.15.169:3300/0 conn(0x7f259c0b3060 0x7f259c15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 -- 172.21.15.122:0/496644110 >> v1:172.21.15.169:6789/0 conn(0x7f259c13dc10 legacy=0x7f259c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a2098640 1 -- 172.21.15.122:0/496644110 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c15f580 con 0x7f259c15bf30 2024-03-06T07:28:29.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.625+0000 7f25a1096640 1 --2- 172.21.15.122:0/496644110 >> v2:172.21.15.169:3300/0 conn(0x7f259c0b3060 0x7f259c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.626+0000 7f25a2098640 1 --2- 172.21.15.122:0/496644110 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c15bf30 0x7f259c15ed90 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f259c068700 tx=0x7f259402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd091c586e26dde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.626+0000 7f25a0895640 1 -- 172.21.15.122:0/496644110 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259402ea90 con 0x7f259c15bf30 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.626+0000 7f25a0895640 1 -- 172.21.15.122:0/496644110 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259402ec30 con 0x7f259c15bf30 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.626+0000 7f25a0895640 1 -- 172.21.15.122:0/496644110 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594009600 con 0x7f259c15bf30 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 -- 172.21.15.122:0/496644110 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c15bf30 msgr2=0x7f259c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 --2- 172.21.15.122:0/496644110 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c15bf30 0x7f259c15ed90 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f259c068700 tx=0x7f259402e0f0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 -- 172.21.15.122:0/496644110 shutdown_connections 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 --2- 172.21.15.122:0/496644110 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c15bf30 0x7f259c15ed90 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 --2- 172.21.15.122:0/496644110 >> v2:172.21.15.169:3300/0 conn(0x7f259c0b3060 0x7f259c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 -- 172.21.15.122:0/496644110 >> 172.21.15.122:0/496644110 conn(0x7f259c0cf8a0 msgr2=0x7f259c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 -- 172.21.15.122:0/496644110 shutdown_connections 2024-03-06T07:28:29.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.627+0000 7f25a3afb640 1 -- 172.21.15.122:0/496644110 wait complete. 2024-03-06T07:28:29.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.628+0000 7f25a3afb640 1 Processor -- start 2024-03-06T07:28:29.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.628+0000 7f25a3afb640 1 -- start start 2024-03-06T07:28:29.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.628+0000 7f25a3afb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f259c0b3060 0x7f259c15c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.628+0000 7f25a3afb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 0x7f259c1392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a3afb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f259c15bf30 0x7f259c13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a3afb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c13dff0 con 0x7f259c13dc10 2024-03-06T07:28:29.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a3afb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c15b730 con 0x7f259c15bf30 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a3afb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0cd7a0 con 0x7f259c0b3060 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 0x7f259c1392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 0x7f259c1392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41100/0 (socket says 172.21.15.122:41100) 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 -- 172.21.15.122:0/4142406309 learned_addr learned my addr 172.21.15.122:0/4142406309 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a2098640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f259c15bf30 0x7f259c13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 -- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f259c0b3060 msgr2=0x7f259c15c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1897640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f259c0b3060 0x7f259c15c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f259c0b3060 0x7f259c15c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 -- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f259c15bf30 msgr2=0x7f259c13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f259c15bf30 0x7f259c13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 -- 172.21.15.122:0/4142406309 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c13d7c0 con 0x7f259c13dc10 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a2098640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f259c15bf30 0x7f259c13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1897640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f259c0b3060 0x7f259c15c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a1096640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 0x7f259c1392d0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e810 tx=0x7f258c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00f040 con 0x7f259c13dc10 2024-03-06T07:28:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c004030 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c004400 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c15ca90 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.629+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c15d050 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.630+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c04f330 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.631+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f258c00c6d0 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.631+0000 7f25927fc640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f258006b4d0 0x7f258006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.631+0000 7f25a1897640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f258006b4d0 0x7f258006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.631+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f258c089fa0 con 0x7f259c13dc10 2024-03-06T07:28:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.632+0000 7f25a1897640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f258006b4d0 0x7f258006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2598001680 tx=0x7f2598009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.636+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f259c13d7c0 con 0x7f259c13dc10 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.844+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259c06eb00 con 0x7f259c13dc10 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.844+0000 7f25927fc640 1 -- 172.21.15.122:0/4142406309 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f258c017020 con 0x7f259c13dc10 2024-03-06T07:28:29.846 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:29.846 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:29.847 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:29.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f258006b4d0 msgr2=0x7f258006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f258006b4d0 0x7f258006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2598001680 tx=0x7f2598009040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 msgr2=0x7f259c1392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 0x7f259c1392d0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e810 tx=0x7f258c00ecd0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 shutdown_connections 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f258006b4d0 0x7f258006d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f259c15bf30 0x7f259c13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f259c13dc10 0x7f259c1392d0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.846+0000 7f25a3afb640 1 --2- 172.21.15.122:0/4142406309 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f259c0b3060 0x7f259c15c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.847+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 >> 172.21.15.122:0/4142406309 conn(0x7f259c0cf8a0 msgr2=0x7f259c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.847+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 shutdown_connections 2024-03-06T07:28:29.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.847+0000 7f25a3afb640 1 -- 172.21.15.122:0/4142406309 wait complete. 2024-03-06T07:28:29.859 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:29.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 Processor -- start 2024-03-06T07:28:29.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 -- start start 2024-03-06T07:28:29.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb91c15dfe0 0x7fb91c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c0cf830 0x7fb91c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91c0b3430 con 0x7fb91c0d27f0 2024-03-06T07:28:29.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91c0d2c50 con 0x7fb91c15dfe0 2024-03-06T07:28:29.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.976+0000 7fb92230c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0b31e0 con 0x7fb91c0cf830 2024-03-06T07:28:29.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb91c15dfe0 0x7fb91c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c0cf830 0x7fb91c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb91c15dfe0 0x7fb91c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:35524/0 (socket says 172.21.15.122:35524) 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 -- 172.21.15.122:0/1678478330 learned_addr learned my addr 172.21.15.122:0/1678478330 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 -- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c0cf830 msgr2=0x7fb91c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 --2- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c0cf830 0x7fb91c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 -- 172.21.15.122:0/1678478330 >> v1:172.21.15.169:6789/0 conn(0x7fb91c0d27f0 legacy=0x7fb91c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91bfff640 1 -- 172.21.15.122:0/1678478330 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c0b3710 con 0x7fb91c15dfe0 2024-03-06T07:28:29.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91affd640 1 -- 172.21.15.122:0/1678478330 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942294618 0 0) 0x7fb91c0b3430 con 0x7fb91c0d27f0 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.977+0000 7fb91b7fe640 1 --2- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c0cf830 0x7fb91c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.978+0000 7fb91bfff640 1 --2- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c15dfe0 0x7fb91c0cf2f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009870 tx=0x7fb90c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d14270657cd5383 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.978+0000 7fb91affd640 1 -- 172.21.15.122:0/1678478330 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c034070 con 0x7fb91c15dfe0 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.978+0000 7fb91affd640 1 -- 172.21.15.122:0/1678478330 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c030840 con 0x7fb91c15dfe0 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.978+0000 7fb91affd640 1 -- 172.21.15.122:0/1678478330 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c030c10 con 0x7fb91c15dfe0 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 -- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c15dfe0 msgr2=0x7fb91c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 --2- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c15dfe0 0x7fb91c0cf2f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009870 tx=0x7fb90c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:29.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 -- 172.21.15.122:0/1678478330 shutdown_connections 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 --2- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c0cf830 0x7fb91c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 --2- 172.21.15.122:0/1678478330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c15dfe0 0x7fb91c0cf2f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 -- 172.21.15.122:0/1678478330 >> 172.21.15.122:0/1678478330 conn(0x7fb91c0d3cd0 msgr2=0x7fb91c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 -- 172.21.15.122:0/1678478330 shutdown_connections 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.979+0000 7fb92230c640 1 -- 172.21.15.122:0/1678478330 wait complete. 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.980+0000 7fb92230c640 1 Processor -- start 2024-03-06T07:28:29.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.980+0000 7fb92230c640 1 -- start start 2024-03-06T07:28:29.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb92230c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 0x7fb91c160d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb92230c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c0d27f0 0x7fb91c15bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb92230c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c15dfe0 0x7fb91c0b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb92230c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0d1c70 con 0x7fb91c0cf830 2024-03-06T07:28:29.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb92230c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0b3d60 con 0x7fb91c0d27f0 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb92230c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0d1df0 con 0x7fb91c15dfe0 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 0x7fb91c160d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 0x7fb91c160d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41114/0 (socket says 172.21.15.122:41114) 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 -- 172.21.15.122:0/1677137666 learned_addr learned my addr 172.21.15.122:0/1677137666 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb9208a9640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c15dfe0 0x7fb91c0b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91b7fe640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c0d27f0 0x7fb91c15bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 -- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c15dfe0 msgr2=0x7fb91c0b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c15dfe0 0x7fb91c0b7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 -- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c0d27f0 msgr2=0x7fb91c15bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c0d27f0 0x7fb91c15bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 -- 172.21.15.122:0/1677137666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c15c1c0 con 0x7fb91c0cf830 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb9208a9640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c15dfe0 0x7fb91c0b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91b7fe640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c0d27f0 0x7fb91c15bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.981+0000 7fb91bfff640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 0x7fb91c160d70 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009870 tx=0x7fb90c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.982+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c034040 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.982+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c03ab90 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.982+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91c0b6e60 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.982+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c004030 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.982+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91c0b7420 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.983+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91c04f330 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.983+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb90c04b060 con 0x7fb91c0cf830 2024-03-06T07:28:29.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.983+0000 7fb918ff9640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb90806b4d0 0x7fb90806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.984+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb90c0abff0 con 0x7fb91c0cf830 2024-03-06T07:28:29.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.984+0000 7fb91b7fe640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb90806b4d0 0x7fb90806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.984+0000 7fb91b7fe640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb90806b4d0 0x7fb90806d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0a1630 tx=0x7fb910008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:29.988+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb91c15c1c0 con 0x7fb91c0cf830 2024-03-06T07:28:30.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.287+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb91c06a190 con 0x7fb91c0cf830 2024-03-06T07:28:30.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.288+0000 7fb918ff9640 1 -- 172.21.15.122:0/1677137666 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fb90c03f070 con 0x7fb91c0cf830 2024-03-06T07:28:30.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:30.290 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.604168+0000","modified":"2024-03-06T07:27:31.978819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6837/2123619089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2123619089},{"type":"v1","addr":"172.21.15.122:6837","nonce":2123619089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8283,"qdb_cluster":[8283]},"id":8}]} 2024-03-06T07:28:30.290 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 67 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb90806b4d0 msgr2=0x7fb90806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb90806b4d0 0x7fb90806d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0a1630 tx=0x7fb910008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 msgr2=0x7fb91c160d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 0x7fb91c160d70 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009870 tx=0x7fb90c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 shutdown_connections 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb90806b4d0 0x7fb90806d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb91c15dfe0 0x7fb91c0b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb91c0d27f0 0x7fb91c15bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 --2- 172.21.15.122:0/1677137666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb91c0cf830 0x7fb91c160d70 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 >> 172.21.15.122:0/1677137666 conn(0x7fb91c0d3cd0 msgr2=0x7fb91c0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 shutdown_connections 2024-03-06T07:28:30.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.290+0000 7fb92230c640 1 -- 172.21.15.122:0/1677137666 wait complete. 2024-03-06T07:28:30.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:30.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.411+0000 7f1223a46640 1 Processor -- start 2024-03-06T07:28:30.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.411+0000 7f1223a46640 1 -- start start 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1223a46640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c15bdd0 0x7f121c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1223a46640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f121c15fa10 0x7f121c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1223a46640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f121c0d16e0 0x7f121c0d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1223a46640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121c0cebe0 con 0x7f121c15fa10 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1223a46640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121c0ceec0 con 0x7f121c0d16e0 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1223a46640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c0cf1a0 con 0x7f121c15bdd0 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f121c15fa10 0x7f121c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f121c15fa10 0x7f121c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41120/0 (socket says 172.21.15.122:41120) 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 -- 172.21.15.122:0/986136942 learned_addr learned my addr 172.21.15.122:0/986136942 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1221fe3640 1 --2- 172.21.15.122:0/986136942 >> v2:172.21.15.169:3300/0 conn(0x7f121c0d16e0 0x7f121c0d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f12217e2640 1 --2- 172.21.15.122:0/986136942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c15bdd0 0x7f121c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 -- 172.21.15.122:0/986136942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c15bdd0 msgr2=0x7f121c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 --2- 172.21.15.122:0/986136942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c15bdd0 0x7f121c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 -- 172.21.15.122:0/986136942 >> v2:172.21.15.169:3300/0 conn(0x7f121c0d16e0 msgr2=0x7f121c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 --2- 172.21.15.122:0/986136942 >> v2:172.21.15.169:3300/0 conn(0x7f121c0d16e0 0x7f121c0d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.412+0000 7f1220fe1640 1 -- 172.21.15.122:0/986136942 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c0cc8e0 con 0x7f121c15fa10 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.413+0000 7f1220fe1640 1 --2- 172.21.15.122:0/986136942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0ce270 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f121c0729e0 tx=0x7f120c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161b45b2d88d1712 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.413+0000 7f1213fff640 1 -- 172.21.15.122:0/986136942 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c02e070 con 0x7f121c15fa10 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.413+0000 7f1213fff640 1 -- 172.21.15.122:0/986136942 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c037070 con 0x7f121c15fa10 2024-03-06T07:28:30.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.413+0000 7f1213fff640 1 -- 172.21.15.122:0/986136942 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c031a00 con 0x7f121c15fa10 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.413+0000 7f1223a46640 1 -- 172.21.15.122:0/986136942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 msgr2=0x7f121c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 --2- 172.21.15.122:0/986136942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0ce270 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f121c0729e0 tx=0x7f120c02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 -- 172.21.15.122:0/986136942 shutdown_connections 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 --2- 172.21.15.122:0/986136942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0ce270 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 --2- 172.21.15.122:0/986136942 >> v2:172.21.15.169:3300/0 conn(0x7f121c0d16e0 0x7f121c0d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 --2- 172.21.15.122:0/986136942 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c15bdd0 0x7f121c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 -- 172.21.15.122:0/986136942 >> 172.21.15.122:0/986136942 conn(0x7f121c09a9f0 msgr2=0x7f121c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 -- 172.21.15.122:0/986136942 shutdown_connections 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.414+0000 7f1223a46640 1 -- 172.21.15.122:0/986136942 wait complete. 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 Processor -- start 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 -- start start 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c0d16e0 0x7f121c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f121c15bdd0 0x7f121c09c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c0cc5d0 con 0x7f121c15fa10 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c0d2070 con 0x7f121c15bdd0 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.415+0000 7f1223a46640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c15c1b0 con 0x7f121c0d16e0 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0b6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41128/0 (socket says 172.21.15.122:41128) 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 -- 172.21.15.122:0/975505111 learned_addr learned my addr 172.21.15.122:0/975505111 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1220fe1640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f121c15bdd0 0x7f121c09c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f12217e2640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c0d16e0 0x7f121c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 -- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c0d16e0 msgr2=0x7f121c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c0d16e0 0x7f121c0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 -- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f121c15bdd0 msgr2=0x7f121c09c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f121c15bdd0 0x7f121c09c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 -- 172.21.15.122:0/975505111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c09cc00 con 0x7f121c15fa10 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f12217e2640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c0d16e0 0x7f121c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1221fe3640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0b6850 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f1214007e60 tx=0x7f121400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214014420 con 0x7f121c15fa10 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1214019070 con 0x7f121c15fa10 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c0b70a0 con 0x7f121c15fa10 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214014820 con 0x7f121c15fa10 2024-03-06T07:28:30.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.416+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c0c8e70 con 0x7f121c15fa10 2024-03-06T07:28:30.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.418+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c04f330 con 0x7f121c15fa10 2024-03-06T07:28:30.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.418+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12140149c0 con 0x7f121c15fa10 2024-03-06T07:28:30.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.418+0000 7f1211ffb640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f120006b4d0 0x7f120006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.418+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f121408a9d0 con 0x7f121c15fa10 2024-03-06T07:28:30.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.418+0000 7f12217e2640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f120006b4d0 0x7f120006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.419+0000 7f12217e2640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f120006b4d0 0x7f120006d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f121800b970 tx=0x7f1218008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.423+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1214059850 con 0x7f121c15fa10 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.600+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f121c09b110 con 0x7f121c15fa10 2024-03-06T07:28:30.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.601+0000 7f1211ffb640 1 -- 172.21.15.122:0/975505111 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16909 (secure 0 0 0) 0x7f1214059150 con 0x7f121c15fa10 2024-03-06T07:28:30.604 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:30.604 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":114,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:27:34.986316+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.843386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.579477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/386534812":"2024-03-07T07:27:28.499450+0000","172.21.15.122:6836/386534812":"2024-03-07T07:27:28.499450+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f120006b4d0 msgr2=0x7f120006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f120006b4d0 0x7f120006d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f121800b970 tx=0x7f1218008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 msgr2=0x7f121c0b6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0b6850 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f1214007e60 tx=0x7f121400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 shutdown_connections 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f120006b4d0 0x7f120006d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f121c15fa10 0x7f121c0b6850 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.603+0000 7f1223a46640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f121c15bdd0 0x7f121c09c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.604+0000 7f1223a46640 1 --2- 172.21.15.122:0/975505111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f121c0d16e0 0x7f121c0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.604+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 >> 172.21.15.122:0/975505111 conn(0x7f121c09a9f0 msgr2=0x7f121c06f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.604+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 shutdown_connections 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.604+0000 7f1223a46640 1 -- 172.21.15.122:0/975505111 wait complete. 2024-03-06T07:28:30.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:30.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.738+0000 7fbf61bc7640 1 Processor -- start 2024-03-06T07:28:30.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.739+0000 7fbf61bc7640 1 -- start start 2024-03-06T07:28:30.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.739+0000 7fbf61bc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 0x7fbf5c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.739+0000 7fbf61bc7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbf5c161cf0 0x7fbf5c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.739+0000 7fbf61bc7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0d22c0 con 0x7fbf5c0b4460 2024-03-06T07:28:30.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.739+0000 7fbf61bc7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c161b00 con 0x7fbf5c161cf0 2024-03-06T07:28:30.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.739+0000 7fbf61bc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0cecc0 con 0x7fbf5c162970 2024-03-06T07:28:30.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5bfff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbf5c161cf0 0x7fbf5c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5affd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 0x7fbf5c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5b7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fbf5c0b4460 0x7fbf5c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:54682/0 (socket says 172.21.15.122:54682) 2024-03-06T07:28:30.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5b7fe640 1 -- 172.21.15.122:0/3239930600 learned_addr learned my addr 172.21.15.122:0/3239930600 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5affd640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 0x7fbf5c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:39016/0 (socket says 172.21.15.122:39016) 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5bfff640 1 -- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 msgr2=0x7fbf5c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5bfff640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 0x7fbf5c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5bfff640 1 -- 172.21.15.122:0/3239930600 >> v1:172.21.15.169:6789/0 conn(0x7fbf5c0b4460 legacy=0x7fbf5c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5bfff640 1 -- 172.21.15.122:0/3239930600 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0d25a0 con 0x7fbf5c161cf0 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5a7fc640 1 -- 172.21.15.122:0/3239930600 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095391976 0 0) 0x7fbf5c0d22c0 con 0x7fbf5c0b4460 2024-03-06T07:28:30.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.740+0000 7fbf5affd640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 0x7fbf5c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.741+0000 7fbf5bfff640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c162180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbf540097a0 tx=0x7fbf5402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ffa25d2dfaca9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.741+0000 7fbf5a7fc640 1 -- 172.21.15.122:0/3239930600 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54035020 con 0x7fbf5c161cf0 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.741+0000 7fbf5a7fc640 1 -- 172.21.15.122:0/3239930600 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf54031830 con 0x7fbf5c161cf0 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.741+0000 7fbf5a7fc640 1 -- 172.21.15.122:0/3239930600 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54031c00 con 0x7fbf5c161cf0 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 -- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 msgr2=0x7fbf5c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c162180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbf540097a0 tx=0x7fbf5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 -- 172.21.15.122:0/3239930600 shutdown_connections 2024-03-06T07:28:30.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c162180 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/3239930600 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c162970 0x7fbf5c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 -- 172.21.15.122:0/3239930600 >> 172.21.15.122:0/3239930600 conn(0x7fbf5c0d3ce0 msgr2=0x7fbf5c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 -- 172.21.15.122:0/3239930600 shutdown_connections 2024-03-06T07:28:30.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.742+0000 7fbf61bc7640 1 -- 172.21.15.122:0/3239930600 wait complete. 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.743+0000 7fbf61bc7640 1 Processor -- start 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.743+0000 7fbf61bc7640 1 -- start start 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c0b4460 0x7fbf5c0b9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 0x7fbf5c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0ce950 con 0x7fbf5c162970 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0d2b60 con 0x7fbf5c161cf0 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b4840 con 0x7fbf5c0b4460 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 0x7fbf5c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 0x7fbf5c0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41138/0 (socket says 172.21.15.122:41138) 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 -- 172.21.15.122:0/1365222690 learned_addr learned my addr 172.21.15.122:0/1365222690 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:30.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5b7fe640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c0b4460 0x7fbf5c0b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5affd640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 -- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c0b4460 msgr2=0x7fbf5c0b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c0b4460 0x7fbf5c0b9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 -- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 msgr2=0x7fbf5c14f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c14f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 -- 172.21.15.122:0/1365222690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0cadb0 con 0x7fbf5c162970 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf5bfff640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 0x7fbf5c0b19f0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbf540097a0 tx=0x7fbf540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54040070 con 0x7fbf5c162970 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5403a830 con 0x7fbf5c162970 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c0cb050 con 0x7fbf5c162970 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.744+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54035040 con 0x7fbf5c162970 2024-03-06T07:28:30.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.745+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c0cb610 con 0x7fbf5c162970 2024-03-06T07:28:30.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.746+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c04f330 con 0x7fbf5c162970 2024-03-06T07:28:30.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.746+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf5404b060 con 0x7fbf5c162970 2024-03-06T07:28:30.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.746+0000 7fbf53fff640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.746+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbf540ab8e0 con 0x7fbf5c162970 2024-03-06T07:28:30.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.747+0000 7fbf5b7fe640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.747+0000 7fbf5b7fe640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0012e0 tx=0x7fbf4c001630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.751+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf54031800 con 0x7fbf5c162970 2024-03-06T07:28:30.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.960+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbf5c06b180 con 0x7fbf5c162970 2024-03-06T07:28:30.962 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:28:30.960+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:30.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.962+0000 7fbf53fff640 1 -- 172.21.15.122:0/1365222690 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbf5407a110 con 0x7fbf5c162970 2024-03-06T07:28:30.963 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:30.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbf3c06b4d0 msgr2=0x7fbf3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0012e0 tx=0x7fbf4c001630 comp rx=0 tx=0).stop 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 msgr2=0x7fbf5c0b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 0x7fbf5c0b19f0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbf540097a0 tx=0x7fbf540046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 shutdown_connections 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbf3c06b4d0 0x7fbf3c06d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf5c162970 0x7fbf5c0b19f0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf5c161cf0 0x7fbf5c14f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 --2- 172.21.15.122:0/1365222690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbf5c0b4460 0x7fbf5c0b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.964+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 >> 172.21.15.122:0/1365222690 conn(0x7fbf5c0d3ce0 msgr2=0x7fbf5c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.965+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 shutdown_connections 2024-03-06T07:28:30.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:30.965+0000 7fbf61bc7640 1 -- 172.21.15.122:0/1365222690 wait complete. 2024-03-06T07:28:30.970 INFO:tasks.ceph.mds.a.smithi122.stderr: -8739> 2024-03-06T07:27:33.974+0000 7fc172148640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:30.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:31.004 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.079+0000 7f475218a640 1 Processor -- start 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.079+0000 7f475218a640 1 -- start start 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f475218a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f475218a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c06b710 con 0x7f474c066d90 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f475218a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c06b150 con 0x7f474c0b55e0 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f475218a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c06b430 con 0x7f474c0cf9a0 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f474affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f474c066d90 0x7f474c0b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41084/0 (socket says 172.21.15.122:41084) 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f474affd640 1 -- 172.21.15.122:0/3012020389 learned_addr learned my addr 172.21.15.122:0/3012020389 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f474b7fe640 1 --2- 172.21.15.122:0/3012020389 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f474a7fc640 1 -- 172.21.15.122:0/3012020389 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784187816 0 0) 0x7f474c06b710 con 0x7f474c066d90 2024-03-06T07:28:31.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.080+0000 7f474a7fc640 1 -- 172.21.15.122:0/3012020389 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4734003520 con 0x7f474c066d90 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.081+0000 7f474b7fe640 1 -- 172.21.15.122:0/3012020389 >> v1:172.21.15.169:6789/0 conn(0x7f474c0b55e0 legacy=0x7f474c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.081+0000 7f474b7fe640 1 -- 172.21.15.122:0/3012020389 >> v1:172.21.15.122:6789/0 conn(0x7f474c066d90 legacy=0x7f474c0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.081+0000 7f474a7fc640 1 -- 172.21.15.122:0/3012020389 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432128936 0 0) 0x7f474c06b150 con 0x7f474c0b55e0 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.081+0000 7f474b7fe640 1 -- 172.21.15.122:0/3012020389 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0b0970 con 0x7f474c0cf9a0 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.081+0000 7f474b7fe640 1 --2- 172.21.15.122:0/3012020389 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c0b5ab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097d0 tx=0x7f473c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbbe7f98805f5d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f474a7fc640 1 -- 172.21.15.122:0/3012020389 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c035020 con 0x7f474c0cf9a0 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f474a7fc640 1 -- 172.21.15.122:0/3012020389 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c031890 con 0x7f474c0cf9a0 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f474a7fc640 1 -- 172.21.15.122:0/3012020389 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c031c40 con 0x7f474c0cf9a0 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 -- 172.21.15.122:0/3012020389 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 msgr2=0x7f474c0b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 --2- 172.21.15.122:0/3012020389 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c0b5ab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097d0 tx=0x7f473c02b930 comp rx=0 tx=0).stop 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 -- 172.21.15.122:0/3012020389 shutdown_connections 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 --2- 172.21.15.122:0/3012020389 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c0b5ab0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 -- 172.21.15.122:0/3012020389 >> 172.21.15.122:0/3012020389 conn(0x7f474c0b1d70 msgr2=0x7f474c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 -- 172.21.15.122:0/3012020389 shutdown_connections 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.082+0000 7f475218a640 1 -- 172.21.15.122:0/3012020389 wait complete. 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.083+0000 7f475218a640 1 Processor -- start 2024-03-06T07:28:31.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 -- start start 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 0x7f474c1f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f474c0b55e0 0x7f474c0727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0b6170 con 0x7f474c066d90 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0b5ff0 con 0x7f474c0b55e0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f475218a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0cfd80 con 0x7f474c0cf9a0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 0x7f474c1f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 0x7f474c1f79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41164/0 (socket says 172.21.15.122:41164) 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.084+0000 7f474b7fe640 1 -- 172.21.15.122:0/3003619162 learned_addr learned my addr 172.21.15.122:0/3003619162 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474bfff640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474affd640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f474c0b55e0 0x7f474c0727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474b7fe640 1 -- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 msgr2=0x7f474c156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474b7fe640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474b7fe640 1 -- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f474c0b55e0 msgr2=0x7f474c0727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474b7fe640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f474c0b55e0 0x7f474c0727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474b7fe640 1 -- 172.21.15.122:0/3003619162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0cafd0 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f474b7fe640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 0x7f474c1f79d0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f473c034040 tx=0x7f473c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c003820 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c0039c0 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c0cb2b0 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c003db0 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.085+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c0cb870 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.086+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c09e3d0 con 0x7f474c066d90 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.087+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f473c038020 con 0x7f474c066d90 2024-03-06T07:28:31.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.087+0000 7f4733fff640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473806b500 0x7f473806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.087+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f473c0ac8e0 con 0x7f474c066d90 2024-03-06T07:28:31.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.087+0000 7f474affd640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473806b500 0x7f473806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.087+0000 7f474affd640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473806b500 0x7f473806d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f474c1f6f20 tx=0x7f4740009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.091+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f473c02d040 con 0x7f474c066d90 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.307+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f474c068ff0 con 0x7f474c066d90 2024-03-06T07:28:31.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.963+0000 7f4733fff640 1 -- 172.21.15.122:0/3003619162 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7f473c07b030 con 0x7f474c066d90 2024-03-06T07:28:31.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473806b500 msgr2=0x7f473806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473806b500 0x7f473806d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f474c1f6f20 tx=0x7f4740009040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 msgr2=0x7f474c1f79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 0x7f474c1f79d0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f473c034040 tx=0x7f473c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 shutdown_connections 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473806b500 0x7f473806d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474c0cf9a0 0x7f474c156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f474c0b55e0 0x7f474c0727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 --2- 172.21.15.122:0/3003619162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474c066d90 0x7f474c1f79d0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 >> 172.21.15.122:0/3003619162 conn(0x7f474c0b1d70 msgr2=0x7f474c099870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 shutdown_connections 2024-03-06T07:28:31.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:31.967+0000 7f475218a640 1 -- 172.21.15.122:0/3003619162 wait complete. 2024-03-06T07:28:31.983 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:32.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.136+0000 7f02928df640 1 Processor -- start 2024-03-06T07:28:32.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.136+0000 7f02928df640 1 -- start start 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.136+0000 7f02928df640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f028c0d4630 0x7f028c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.136+0000 7f02928df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f028c0d2820 0x7f028c0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.136+0000 7f02928df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028c15e380 con 0x7f028c15dfa0 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.136+0000 7f02928df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c0d2c70 con 0x7f028c0d2820 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f02928df640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f028c0d2100 con 0x7f028c0d4630 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f028c0d2820 0x7f028c0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f028c0d2820 0x7f028c0cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41174/0 (socket says 172.21.15.122:41174) 2024-03-06T07:28:32.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 -- 172.21.15.122:0/1690971998 learned_addr learned my addr 172.21.15.122:0/1690971998 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 -- 172.21.15.122:0/1690971998 >> v2:172.21.15.169:3300/0 conn(0x7f028c0d4630 msgr2=0x7f028c0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 --2- 172.21.15.122:0/1690971998 >> v2:172.21.15.169:3300/0 conn(0x7f028c0d4630 0x7f028c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 -- 172.21.15.122:0/1690971998 >> v1:172.21.15.169:6789/0 conn(0x7f028c15dfa0 legacy=0x7f028c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.137+0000 7f0290e7c640 1 -- 172.21.15.122:0/1690971998 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0cf470 con 0x7f028c0d2820 2024-03-06T07:28:32.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f0290e7c640 1 --2- 172.21.15.122:0/1690971998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c0cb190 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f028c0b6910 tx=0x7f028402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cddc0bae9b990f97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f028affd640 1 -- 172.21.15.122:0/1690971998 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028402c9f0 con 0x7f028c0d2820 2024-03-06T07:28:32.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f028affd640 1 -- 172.21.15.122:0/1690971998 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028402cb90 con 0x7f028c0d2820 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f028affd640 1 -- 172.21.15.122:0/1690971998 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840343f0 con 0x7f028c0d2820 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 -- 172.21.15.122:0/1690971998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 msgr2=0x7f028c0cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 --2- 172.21.15.122:0/1690971998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c0cb190 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f028c0b6910 tx=0x7f028402be50 comp rx=0 tx=0).stop 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 -- 172.21.15.122:0/1690971998 shutdown_connections 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 --2- 172.21.15.122:0/1690971998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c0cb190 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 --2- 172.21.15.122:0/1690971998 >> v2:172.21.15.169:3300/0 conn(0x7f028c0d4630 0x7f028c0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 -- 172.21.15.122:0/1690971998 >> 172.21.15.122:0/1690971998 conn(0x7f028c09d000 msgr2=0x7f028c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 -- 172.21.15.122:0/1690971998 shutdown_connections 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.138+0000 7f02928df640 1 -- 172.21.15.122:0/1690971998 wait complete. 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 Processor -- start 2024-03-06T07:28:32.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 -- start start 2024-03-06T07:28:32.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c15a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f028c0d4630 0x7f028c0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f028c15dfa0 0x7f028c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0cb6d0 con 0x7f028c0d2820 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0cfa30 con 0x7f028c15dfa0 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.140+0000 7f02928df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0cb850 con 0x7f028c0d4630 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c15a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41176/0 (socket says 172.21.15.122:41176) 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 -- 172.21.15.122:0/1262986169 learned_addr learned my addr 172.21.15.122:0/1262986169 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f0290e7c640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f028c15dfa0 0x7f028c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028b7fe640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f028c0d4630 0x7f028c0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 -- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f028c0d4630 msgr2=0x7f028c0c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f028c0d4630 0x7f028c0c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 -- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f028c15dfa0 msgr2=0x7f028c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f028c15dfa0 0x7f028c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 -- 172.21.15.122:0/1262986169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0c7850 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028b7fe640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f028c0d4630 0x7f028c0c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f028bfff640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c15a220 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b3e0 tx=0x7f027c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c015020 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c011850 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c011c40 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028c0c7b30 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.141+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028c0c80f0 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.142+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028c04f330 con 0x7f028c0d2820 2024-03-06T07:28:32.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.143+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f027c018020 con 0x7f028c0d2820 2024-03-06T07:28:32.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.143+0000 7f0288ff9640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.143+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f027c08a1b0 con 0x7f028c0d2820 2024-03-06T07:28:32.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.143+0000 7f028b7fe640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.144+0000 7f028b7fe640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f027806b4d0 0x7f027806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f028c0c77a0 tx=0x7f0280008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.148+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f027c01f020 con 0x7f028c0d2820 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:32.323+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f028c09d3e0 con 0x7f028c0d2820 2024-03-06T07:28:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.113+0000 7f0288ff9640 1 -- 172.21.15.122:0/1262986169 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7f027c058f40 con 0x7f028c0d2820 2024-03-06T07:28:33.115 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f027806b4d0 msgr2=0x7f027806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f027806b4d0 0x7f027806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f028c0c77a0 tx=0x7f0280008040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 msgr2=0x7f028c15a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c15a220 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b3e0 tx=0x7f027c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 shutdown_connections 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f028c15dfa0 0x7f028c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f028c0d4630 0x7f028c0c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 --2- 172.21.15.122:0/1262986169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f028c0d2820 0x7f028c15a220 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 >> 172.21.15.122:0/1262986169 conn(0x7f028c09d000 msgr2=0x7f028c0cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 shutdown_connections 2024-03-06T07:28:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.116+0000 7f02928df640 1 -- 172.21.15.122:0/1262986169 wait complete. 2024-03-06T07:28:33.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:33.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.242+0000 7f963c288640 1 Processor -- start 2024-03-06T07:28:33.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.242+0000 7f963c288640 1 -- start start 2024-03-06T07:28:33.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963c288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96340d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963c288640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9634066d00 0x7f96340d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963c288640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96340d0f90 con 0x7f963415be30 2024-03-06T07:28:33.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963c288640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963406bb60 con 0x7f9634066d00 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963c288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963406be40 con 0x7f96340b5ca0 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963a024640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f963415be30 0x7f963415c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41096/0 (socket says 172.21.15.122:41096) 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963a024640 1 -- 172.21.15.122:0/1638013469 learned_addr learned my addr 172.21.15.122:0/1638013469 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f963a825640 1 --2- 172.21.15.122:0/1638013469 >> v2:172.21.15.169:3300/0 conn(0x7f9634066d00 0x7f96340d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639823640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96340d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979515032 0 0) 0x7f96340d0f90 con 0x7f963415be30 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620003520 con 0x7f963415be30 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1191097774 0 0) 0x7f9620003520 con 0x7f963415be30 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 msgr2=0x7f96340d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96340d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 >> v2:172.21.15.169:3300/0 conn(0x7f9634066d00 msgr2=0x7f96340d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 --2- 172.21.15.122:0/1638013469 >> v2:172.21.15.169:3300/0 conn(0x7f9634066d00 0x7f96340d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340b0990 con 0x7f963415be30 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f9630001670 con 0x7f963415be30 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 >> v1:172.21.15.122:6789/0 conn(0x7f963415be30 legacy=0x7f963415c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96200039b0 0x7f9620003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9620004800 0x7f9620004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963406bb60 con 0x7f9620004800 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963406be40 con 0x7f96200039b0 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962000b9c0 con 0x7f96340b5ca0 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.243+0000 7f9639823640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9620004800 0x7f9620004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f963a024640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96200039b0 0x7f9620003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f963a825640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f9639823640 1 -- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 msgr2=0x7f96200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f9639823640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f9639823640 1 -- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96200039b0 msgr2=0x7f9620003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f9639823640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96200039b0 0x7f9620003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f9639823640 1 -- 172.21.15.122:0/1638013469 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620003520 con 0x7f9620004800 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f963a825640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96340b5ca0 0x7f96200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.244+0000 7f9639823640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9620004800 0x7f9620004c80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f9624009ac0 tx=0x7f9620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb2cbd7a9ff8810 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624039070 con 0x7f9620004800 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962402c6b0 con 0x7f9620004800 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f9639022640 1 -- 172.21.15.122:0/1638013469 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624009690 con 0x7f9620004800 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963c288640 1 -- 172.21.15.122:0/1638013469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9620004800 msgr2=0x7f9620004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963c288640 1 --2- 172.21.15.122:0/1638013469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9620004800 0x7f9620004c80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f9624009ac0 tx=0x7f9620005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963a825640 1 -- 172.21.15.122:0/1638013469 reap_dead start 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963c288640 1 -- 172.21.15.122:0/1638013469 shutdown_connections 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963c288640 1 -- 172.21.15.122:0/1638013469 >> 172.21.15.122:0/1638013469 conn(0x7f96340b4ed0 msgr2=0x7f96340b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963c288640 1 -- 172.21.15.122:0/1638013469 shutdown_connections 2024-03-06T07:28:33.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.245+0000 7f963c288640 1 -- 172.21.15.122:0/1638013469 wait complete. 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.246+0000 7f963c288640 1 Processor -- start 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.246+0000 7f963c288640 1 -- start start 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f963c288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9620004800 0x7f963414d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f963c288640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 0x7f96341537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f963c288640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96341f2a00 0x7f96341f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f963c288640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963415c920 con 0x7f963414e270 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f963c288640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963406c120 con 0x7f96341f2a00 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f963c288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963415caa0 con 0x7f9620004800 2024-03-06T07:28:33.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 0x7f96341537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 0x7f96341537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41198/0 (socket says 172.21.15.122:41198) 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 -- 172.21.15.122:0/532455161 learned_addr learned my addr 172.21.15.122:0/532455161 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 -- 172.21.15.122:0/532455161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9620004800 msgr2=0x7f963414d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9620004800 0x7f963414d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 -- 172.21.15.122:0/532455161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96341f2a00 msgr2=0x7f96341f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96341f2a00 0x7f96341f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.247+0000 7f9639823640 1 -- 172.21.15.122:0/532455161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963414e7e0 con 0x7f963414e270 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.248+0000 7f9639823640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 0x7f96341537b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f9624037040 tx=0x7f9624004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.248+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624042020 con 0x7f963414e270 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.248+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962403a040 con 0x7f963414e270 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.248+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963414eac0 con 0x7f963414e270 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.248+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624039070 con 0x7f963414e270 2024-03-06T07:28:33.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.248+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963414f080 con 0x7f963414e270 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.249+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963404f330 con 0x7f963414e270 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.249+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9624045020 con 0x7f963414e270 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.249+0000 7f962affd640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f961806b4d0 0x7f961806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.250+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5501+0+0 (secure 0 0 0) 0x7f96240ab430 con 0x7f963414e270 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.250+0000 7f963a024640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f961806b4d0 0x7f961806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.250+0000 7f963a024640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f961806b4d0 0x7f961806d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9630001050 tx=0x7f9630000fe0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.254+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f962404c070 con 0x7f963414e270 2024-03-06T07:28:33.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:33.426+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f963406dad0 con 0x7f963414e270 2024-03-06T07:28:34.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.119+0000 7f962affd640 1 -- 172.21.15.122:0/532455161 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7f962407a380 con 0x7f963414e270 2024-03-06T07:28:34.121 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.124+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f961806b4d0 msgr2=0x7f961806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.124+0000 7f963c288640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f961806b4d0 0x7f961806d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9630001050 tx=0x7f9630000fe0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.124+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 msgr2=0x7f96341537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.124+0000 7f963c288640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 0x7f96341537b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f9624037040 tx=0x7f9624004730 comp rx=0 tx=0).stop 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 shutdown_connections 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f961806b4d0 0x7f961806d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96341f2a00 0x7f96341f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f963414e270 0x7f96341537b0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 --2- 172.21.15.122:0/532455161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9620004800 0x7f963414d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 >> 172.21.15.122:0/532455161 conn(0x7f96340b4ed0 msgr2=0x7f9634158840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 shutdown_connections 2024-03-06T07:28:34.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.125+0000 7f963c288640 1 -- 172.21.15.122:0/532455161 wait complete. 2024-03-06T07:28:34.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.248+0000 7f3fb7e39640 1 Processor -- start 2024-03-06T07:28:34.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.248+0000 7f3fb7e39640 1 -- start start 2024-03-06T07:28:34.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb7e39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb00b4970 0x7f3fb006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb7e39640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb015ff90 con 0x7f3fb006d5c0 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb7e39640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb0160270 con 0x7f3fb015bb40 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb7e39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb00b4dd0 con 0x7f3fb00b4970 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb63d6640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3fb006d5c0 0x7f3fb00cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41104/0 (socket says 172.21.15.122:41104) 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb63d6640 1 -- 172.21.15.122:0/2251872584 learned_addr learned my addr 172.21.15.122:0/2251872584 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.249+0000 7f3fb53d4640 1 --2- 172.21.15.122:0/2251872584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb00b4970 0x7f3fb006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510902375 0 0) 0x7f3fb015ff90 con 0x7f3fb006d5c0 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c003520 con 0x7f3fb006d5c0 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863160795 0 0) 0x7f3fb0160270 con 0x7f3fb015bb40 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb015ff90 con 0x7f3fb015bb40 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb53d4640 1 -- 172.21.15.122:0/2251872584 >> v1:172.21.15.169:6789/0 conn(0x7f3fb015bb40 legacy=0x7f3fb015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb53d4640 1 -- 172.21.15.122:0/2251872584 >> v1:172.21.15.122:6789/0 conn(0x7f3fb006d5c0 legacy=0x7f3fb00cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1311778265 0 0) 0x7f3f9c003520 con 0x7f3fb006d5c0 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.250+0000 7f3fb53d4640 1 -- 172.21.15.122:0/2251872584 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0160550 con 0x7f3fb00b4970 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb53d4640 1 --2- 172.21.15.122:0/2251872584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb00b4970 0x7f3fb006d080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00097d0 tx=0x7f3fa002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52b0d2c0632ad74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0037070 con 0x7f3fb00b4970 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa0030510 con 0x7f3fb00b4970 2024-03-06T07:28:34.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb4bd3640 1 -- 172.21.15.122:0/2251872584 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8001670 con 0x7f3fb00b4970 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 -- 172.21.15.122:0/2251872584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb00b4970 msgr2=0x7f3fb006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/2251872584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb00b4970 0x7f3fb006d080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00097d0 tx=0x7f3fa002b800 comp rx=0 tx=0).stop 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 -- 172.21.15.122:0/2251872584 shutdown_connections 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/2251872584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb00b4970 0x7f3fb006d080 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 -- 172.21.15.122:0/2251872584 >> 172.21.15.122:0/2251872584 conn(0x7f3fb006b190 msgr2=0x7f3fb006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 -- 172.21.15.122:0/2251872584 shutdown_connections 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.251+0000 7f3fb7e39640 1 -- 172.21.15.122:0/2251872584 wait complete. 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.252+0000 7f3fb7e39640 1 Processor -- start 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 -- start start 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb006d5c0 0x7f3fb014b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 0x7f3fb00af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fb015bb40 0x7f3fb00b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb015bf20 con 0x7f3fb00b4970 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb006c910 con 0x7f3fb015bb40 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb7e39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb015c0a0 con 0x7f3fb006d5c0 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb53d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 0x7f3fb00af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb53d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 0x7f3fb00af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:41206/0 (socket says 172.21.15.122:41206) 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb53d4640 1 -- 172.21.15.122:0/4003492318 learned_addr learned my addr 172.21.15.122:0/4003492318 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.253+0000 7f3fb5bd5640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb006d5c0 0x7f3fb014b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb63d6640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fb015bb40 0x7f3fb00b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb53d4640 1 -- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb006d5c0 msgr2=0x7f3fb014b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb53d4640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb006d5c0 0x7f3fb014b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb53d4640 1 -- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fb015bb40 msgr2=0x7f3fb00b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb53d4640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fb015bb40 0x7f3fb00b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb53d4640 1 -- 172.21.15.122:0/4003492318 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb015d660 con 0x7f3fb00b4970 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb63d6640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fb015bb40 0x7f3fb00b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb53d4640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 0x7f3fb00af4e0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0000c00 tx=0x7f3fa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0041020 con 0x7f3fb00b4970 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa003c870 con 0x7f3fb00b4970 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb00afd30 con 0x7f3fb00b4970 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0037040 con 0x7f3fb00b4970 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.254+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb00b02f0 con 0x7f3fb00b4970 2024-03-06T07:28:34.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.255+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb00732b0 con 0x7f3fb00b4970 2024-03-06T07:28:34.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.255+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fa0044020 con 0x7f3fb00b4970 2024-03-06T07:28:34.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.256+0000 7f3fa67fc640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f9406b4d0 0x7f3f9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.256+0000 7f3fb5bd5640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f9406b4d0 0x7f3f9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.256+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3fa00ab960 con 0x7f3fb00b4970 2024-03-06T07:28:34.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.256+0000 7f3fb5bd5640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f9406b4d0 0x7f3f9406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001560 tx=0x7f3fac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.260+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fa004b030 con 0x7f3fb00b4970 2024-03-06T07:28:34.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.433+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3fb0068ff0 con 0x7f3fb00b4970 2024-03-06T07:28:34.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.436+0000 7f3fa67fc640 1 -- 172.21.15.122:0/4003492318 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fa007ab10 con 0x7f3fb00b4970 2024-03-06T07:28:34.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f9406b4d0 msgr2=0x7f3f9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f9406b4d0 0x7f3f9406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001560 tx=0x7f3fac009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 msgr2=0x7f3fb00af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 0x7f3fb00af4e0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0000c00 tx=0x7f3fa00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 shutdown_connections 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f9406b4d0 0x7f3f9406d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fb015bb40 0x7f3fb00b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fb00b4970 0x7f3fb00af4e0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 --2- 172.21.15.122:0/4003492318 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fb006d5c0 0x7f3fb014b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 >> 172.21.15.122:0/4003492318 conn(0x7f3fb006b190 msgr2=0x7f3fb0154850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 shutdown_connections 2024-03-06T07:28:34.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.438+0000 7f3fb7e39640 1 -- 172.21.15.122:0/4003492318 wait complete. 2024-03-06T07:28:34.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:28:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.549+0000 7fcfc4f1e640 1 Processor -- start 2024-03-06T07:28:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.549+0000 7fcfc4f1e640 1 -- start start 2024-03-06T07:28:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfc4f1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcfc00d22b0 0x7fcfc00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfc4f1e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcfc00b41f0 0x7fcfc00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfc4f1e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc00b3430 con 0x7fcfc015dfe0 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfc4f1e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00d2c50 con 0x7fcfc00d22b0 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfc4f1e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00b31e0 con 0x7fcfc00b41f0 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfbdd9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcfc00d22b0 0x7fcfc00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfbe59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcfc015dfe0 0x7fcfc00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58284/0 (socket says 172.21.15.122:58284) 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.550+0000 7fcfbe59c640 1 -- 172.21.15.122:0/2298459746 learned_addr learned my addr 172.21.15.122:0/2298459746 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbdd9b640 1 -- 172.21.15.122:0/2298459746 >> v2:172.21.15.169:3300/0 conn(0x7fcfc00b41f0 msgr2=0x7fcfc00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbdd9b640 1 --2- 172.21.15.122:0/2298459746 >> v2:172.21.15.169:3300/0 conn(0x7fcfc00b41f0 0x7fcfc00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbdd9b640 1 -- 172.21.15.122:0/2298459746 >> v1:172.21.15.122:6789/0 conn(0x7fcfc015dfe0 legacy=0x7fcfc00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbdd9b640 1 -- 172.21.15.122:0/2298459746 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00b3710 con 0x7fcfc00d22b0 2024-03-06T07:28:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbd59a640 1 -- 172.21.15.122:0/2298459746 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956160845 0 0) 0x7fcfc00b3430 con 0x7fcfc015dfe0 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbdd9b640 1 --2- 172.21.15.122:0/2298459746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00d2710 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40097a0 tx=0x7fcfb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb2747f1336bfac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.551+0000 7fcfbd59a640 1 -- 172.21.15.122:0/2298459746 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4035020 con 0x7fcfc00d22b0 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfbd59a640 1 -- 172.21.15.122:0/2298459746 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4031830 con 0x7fcfc00d22b0 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfbd59a640 1 -- 172.21.15.122:0/2298459746 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4031c00 con 0x7fcfc00d22b0 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2298459746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc00d22b0 msgr2=0x7fcfc00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2298459746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00d2710 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40097a0 tx=0x7fcfb402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2298459746 shutdown_connections 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2298459746 >> v2:172.21.15.169:3300/0 conn(0x7fcfc00b41f0 0x7fcfc00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2298459746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00d2710 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2298459746 >> 172.21.15.122:0/2298459746 conn(0x7fcfc00d3cd0 msgr2=0x7fcfc00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2298459746 shutdown_connections 2024-03-06T07:28:34.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.552+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2298459746 wait complete. 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.553+0000 7fcfc4f1e640 1 Processor -- start 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.553+0000 7fcfc4f1e640 1 -- start start 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfc4f1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcfc00b41f0 0x7fcfc01f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfc4f1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfc4f1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 0x7fcfc00b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfc4f1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00cfa00 con 0x7fcfc015dfe0 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfc4f1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00b3d60 con 0x7fcfc00d22b0 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfc4f1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00d1c70 con 0x7fcfc00b41f0 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 0x7fcfc00b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 0x7fcfc00b1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46892/0 (socket says 172.21.15.122:46892) 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 -- 172.21.15.122:0/2551183964 learned_addr learned my addr 172.21.15.122:0/2551183964 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 -- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcfc00b41f0 msgr2=0x7fcfc01f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcfc00b41f0 0x7fcfc01f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 -- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcfc00d22b0 msgr2=0x7fcfc00b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbdd9b640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00b1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.554+0000 7fcfbed9d640 1 -- 172.21.15.122:0/2551183964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00b9b80 con 0x7fcfc015dfe0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.555+0000 7fcfbed9d640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 0x7fcfc00b1940 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009fd0 tx=0x7fcfb800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.555+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8011830 con 0x7fcfc015dfe0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.555+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8016070 con 0x7fcfc015dfe0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.555+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc015e4f0 con 0x7fcfc015dfe0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.555+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8011bf0 con 0x7fcfc015dfe0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.555+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc015eab0 con 0x7fcfc015dfe0 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.556+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc004f330 con 0x7fcfc015dfe0 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.556+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfb801b460 con 0x7fcfc015dfe0 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.557+0000 7fcfaeffd640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfa006b4d0 0x7fcfa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.557+0000 7fcfbe59c640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfa006b4d0 0x7fcfa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.557+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fcfb8089c30 con 0x7fcfc015dfe0 2024-03-06T07:28:34.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.557+0000 7fcfbe59c640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfa006b4d0 0x7fcfa006d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0007fd0 tx=0x7fcfb0007ce0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.561+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfb8058de0 con 0x7fcfc015dfe0 2024-03-06T07:28:34.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:34.798+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7fcfc00d2970 con 0x7fcfc015dfe0 2024-03-06T07:28:35.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.120+0000 7fcfaeffd640 1 -- 172.21.15.122:0/2551183964 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v359) v1 ==== 142+0+0 (secure 0 0 0) 0x7fcfb80586e0 con 0x7fcfc015dfe0 2024-03-06T07:28:35.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfa006b4d0 msgr2=0x7fcfa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfa006b4d0 0x7fcfa006d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0007fd0 tx=0x7fcfb0007ce0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 msgr2=0x7fcfc00b1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 0x7fcfc00b1940 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009fd0 tx=0x7fcfb800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 shutdown_connections 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcfa006b4d0 0x7fcfa006d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfc015dfe0 0x7fcfc00b1940 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcfc00d22b0 0x7fcfc00b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 --2- 172.21.15.122:0/2551183964 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcfc00b41f0 0x7fcfc01f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 >> 172.21.15.122:0/2551183964 conn(0x7fcfc00d3cd0 msgr2=0x7fcfc00d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 shutdown_connections 2024-03-06T07:28:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.122+0000 7fcfc4f1e640 1 -- 172.21.15.122:0/2551183964 wait complete. 2024-03-06T07:28:35.136 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:28:35.136 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:28:35.136 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:35.136 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.235+0000 7f92e9513640 1 Processor -- start 2024-03-06T07:28:35.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.235+0000 7f92e9513640 1 -- start start 2024-03-06T07:28:35.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e9513640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f92e40cfff0 0x7f92e40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e9513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e9513640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e4161980 con 0x7f92e415bb20 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e9513640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e40d2500 con 0x7f92e40cfff0 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e9513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4161c60 con 0x7f92e40cc8b0 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f92e40cfff0 0x7f92e40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f92e40cfff0 0x7f92e40d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46900/0 (socket says 172.21.15.122:46900) 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 -- 172.21.15.122:0/462012327 learned_addr learned my addr 172.21.15.122:0/462012327 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 -- 172.21.15.122:0/462012327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 msgr2=0x7f92e40ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 --2- 172.21.15.122:0/462012327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 -- 172.21.15.122:0/462012327 >> v1:172.21.15.169:6789/0 conn(0x7f92e415bb20 legacy=0x7f92e40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.236+0000 7f92e27fc640 1 -- 172.21.15.122:0/462012327 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4161f40 con 0x7f92e40cfff0 2024-03-06T07:28:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.237+0000 7f92e27fc640 1 --2- 172.21.15.122:0/462012327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40d04a0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009870 tx=0x7f92d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c895aa8691ac9e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.237+0000 7f92e1ffb640 1 -- 172.21.15.122:0/462012327 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8035020 con 0x7f92e40cfff0 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.237+0000 7f92e1ffb640 1 -- 172.21.15.122:0/462012327 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d8031640 con 0x7f92e40cfff0 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.237+0000 7f92e1ffb640 1 -- 172.21.15.122:0/462012327 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d802d040 con 0x7f92e40cfff0 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 -- 172.21.15.122:0/462012327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 msgr2=0x7f92e40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 --2- 172.21.15.122:0/462012327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40d04a0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009870 tx=0x7f92d802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 -- 172.21.15.122:0/462012327 shutdown_connections 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 --2- 172.21.15.122:0/462012327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e40ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 --2- 172.21.15.122:0/462012327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40d04a0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 -- 172.21.15.122:0/462012327 >> 172.21.15.122:0/462012327 conn(0x7f92e409a9f0 msgr2=0x7f92e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 -- 172.21.15.122:0/462012327 shutdown_connections 2024-03-06T07:28:35.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.238+0000 7f92e9513640 1 -- 172.21.15.122:0/462012327 wait complete. 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.239+0000 7f92e9513640 1 Processor -- start 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.239+0000 7f92e9513640 1 -- start start 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e9513640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e4155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e9513640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e9513640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92e415bb20 0x7f92e40c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e9513640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e40d1980 con 0x7f92e40cfff0 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e9513640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4162500 con 0x7f92e415bb20 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e9513640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e40d1b00 con 0x7f92e40cc8b0 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46904/0 (socket says 172.21.15.122:46904) 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 -- 172.21.15.122:0/1384823800 learned_addr learned my addr 172.21.15.122:0/1384823800 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e37fe640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92e415bb20 0x7f92e40c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e2ffd640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e4155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 -- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 msgr2=0x7f92e4155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e4155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 -- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92e415bb20 msgr2=0x7f92e40c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92e415bb20 0x7f92e40c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 -- 172.21.15.122:0/1384823800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e40cb2e0 con 0x7f92e40cfff0 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e2ffd640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e4155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e37fe640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92e415bb20 0x7f92e40c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.240+0000 7f92e27fc640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40cada0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f92d802b930 tx=0x7f92d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.241+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d803ac60 con 0x7f92e40cfff0 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.241+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d803ae20 con 0x7f92e40cfff0 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.241+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8044630 con 0x7f92e40cfff0 2024-03-06T07:28:35.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.241+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e40c9760 con 0x7f92e40cfff0 2024-03-06T07:28:35.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.241+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e40c9d20 con 0x7f92e40cfff0 2024-03-06T07:28:35.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.242+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e40706e0 con 0x7f92e40cfff0 2024-03-06T07:28:35.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.243+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92d8038020 con 0x7f92e40cfff0 2024-03-06T07:28:35.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.243+0000 7f92cb7fe640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f92d006b4d0 0x7f92d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.243+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f92d80abcb0 con 0x7f92e40cfff0 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.244+0000 7f92e2ffd640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f92d006b4d0 0x7f92d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.245+0000 7f92e2ffd640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f92d006b4d0 0x7f92d006d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007eb0 tx=0x7f92d40073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.250+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92e40706e0 con 0x7f92e40cfff0 2024-03-06T07:28:35.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.423+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92e404f330 con 0x7f92e40cfff0 2024-03-06T07:28:35.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.424+0000 7f92cb7fe640 1 -- 172.21.15.122:0/1384823800 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f92d803f070 con 0x7f92e40cfff0 2024-03-06T07:28:35.426 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:35.426 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":117,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:28:34.114883+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/2123619089":"2024-03-07T07:28:30.961460+0000","172.21.15.122:6836/2123619089":"2024-03-07T07:28:30.961460+0000","172.21.15.122:6837/386534812":"2024-03-07T07:27:28.499450+0000","172.21.15.122:6836/386534812":"2024-03-07T07:27:28.499450+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f92d006b4d0 msgr2=0x7f92d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f92d006b4d0 0x7f92d006d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007eb0 tx=0x7f92d40073c0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 msgr2=0x7f92e40cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40cada0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f92d802b930 tx=0x7f92d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 shutdown_connections 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f92d006b4d0 0x7f92d006d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92e415bb20 0x7f92e40c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92e40cfff0 0x7f92e40cada0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 --2- 172.21.15.122:0/1384823800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f92e40cc8b0 0x7f92e4155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 >> 172.21.15.122:0/1384823800 conn(0x7f92e409a9f0 msgr2=0x7f92e406fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 shutdown_connections 2024-03-06T07:28:35.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.426+0000 7f92e9513640 1 -- 172.21.15.122:0/1384823800 wait complete. 2024-03-06T07:28:35.439 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:35.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:35.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.541+0000 7f3985f4b640 1 Processor -- start 2024-03-06T07:28:35.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.541+0000 7f3985f4b640 1 -- start start 2024-03-06T07:28:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f3985f4b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3980145590 0x7f398006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f3985f4b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f398006b4f0 0x7f398006efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f3985f4b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f398006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f3985f4b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3980145970 con 0x7f3980145590 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f3985f4b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3980160110 con 0x7f398006b4f0 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f3985f4b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398006b970 con 0x7f398006f510 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f397f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3980145590 0x7f398006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f397f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3980145590 0x7f398006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46920/0 (socket says 172.21.15.122:46920) 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.542+0000 7f397f7fe640 1 -- 172.21.15.122:0/3835662531 learned_addr learned my addr 172.21.15.122:0/3835662531 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397f7fe640 1 -- 172.21.15.122:0/3835662531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 msgr2=0x7f398006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397ffff640 1 --2- 172.21.15.122:0/3835662531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f398006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397effd640 1 --2- 172.21.15.122:0/3835662531 >> v2:172.21.15.169:3300/0 conn(0x7f398006b4f0 0x7f398006efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397f7fe640 1 --2- 172.21.15.122:0/3835662531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f398006daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397f7fe640 1 -- 172.21.15.122:0/3835662531 >> v2:172.21.15.169:3300/0 conn(0x7f398006b4f0 msgr2=0x7f398006efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397f7fe640 1 --2- 172.21.15.122:0/3835662531 >> v2:172.21.15.169:3300/0 conn(0x7f398006b4f0 0x7f398006efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397f7fe640 1 -- 172.21.15.122:0/3835662531 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398015f7e0 con 0x7f3980145590 2024-03-06T07:28:35.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397effd640 1 --2- 172.21.15.122:0/3835662531 >> v2:172.21.15.169:3300/0 conn(0x7f398006b4f0 0x7f398006efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397f7fe640 1 --2- 172.21.15.122:0/3835662531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f398006afb0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f39800d08d0 tx=0x7f397002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e719e6f70eb0e796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.543+0000 7f397e7fc640 1 -- 172.21.15.122:0/3835662531 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970035020 con 0x7f3980145590 2024-03-06T07:28:35.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f397e7fc640 1 -- 172.21.15.122:0/3835662531 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970031870 con 0x7f3980145590 2024-03-06T07:28:35.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f397e7fc640 1 -- 172.21.15.122:0/3835662531 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970031c40 con 0x7f3980145590 2024-03-06T07:28:35.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 -- 172.21.15.122:0/3835662531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 msgr2=0x7f398006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 --2- 172.21.15.122:0/3835662531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f398006afb0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f39800d08d0 tx=0x7f397002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 -- 172.21.15.122:0/3835662531 shutdown_connections 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 --2- 172.21.15.122:0/3835662531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f398006daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 --2- 172.21.15.122:0/3835662531 >> v2:172.21.15.169:3300/0 conn(0x7f398006b4f0 0x7f398006efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 --2- 172.21.15.122:0/3835662531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f398006afb0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 -- 172.21.15.122:0/3835662531 >> 172.21.15.122:0/3835662531 conn(0x7f39800b2660 msgr2=0x7f39800b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 -- 172.21.15.122:0/3835662531 shutdown_connections 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.544+0000 7f3985f4b640 1 -- 172.21.15.122:0/3835662531 wait complete. 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.545+0000 7f3985f4b640 1 Processor -- start 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.545+0000 7f3985f4b640 1 -- start start 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f3985f4b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f398006b4f0 0x7f398015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f3985f4b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f39800b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f3985f4b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f39800c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f3985f4b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39801605a0 con 0x7f3980145590 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f3985f4b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398015fda0 con 0x7f398006b4f0 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f3985f4b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800d0510 con 0x7f398006f510 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f39800c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f39800c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46934/0 (socket says 172.21.15.122:46934) 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 -- 172.21.15.122:0/2697267551 learned_addr learned my addr 172.21.15.122:0/2697267551 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397f7fe640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f398006b4f0 0x7f398015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397effd640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f39800b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 -- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 msgr2=0x7f39800b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f39800b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 -- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f398006b4f0 msgr2=0x7f398015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f398006b4f0 0x7f398015b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.546+0000 7f397ffff640 1 -- 172.21.15.122:0/2697267551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800cf470 con 0x7f3980145590 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f397effd640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f39800b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f397ffff640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f39800c9ce0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f397800cd70 tx=0x7f397800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978003b90 con 0x7f3980145590 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3978003d30 con 0x7f3980145590 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39800cf750 con 0x7f3980145590 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978004240 con 0x7f3980145590 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.547+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39800cfff0 con 0x7f3980145590 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.548+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39800b2410 con 0x7f3980145590 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.548+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39780043e0 con 0x7f3980145590 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.549+0000 7f3967fff640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f396c06b4d0 0x7f396c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.549+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f397808aae0 con 0x7f3980145590 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.549+0000 7f397f7fe640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f396c06b4d0 0x7f396c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.549+0000 7f397f7fe640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f396c06b4d0 0x7f396c06d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3970036000 tx=0x7f397003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.553+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39800b2410 con 0x7f3980145590 2024-03-06T07:28:35.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.724+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39800d01f0 con 0x7f3980145590 2024-03-06T07:28:35.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.724+0000 7f3967fff640 1 -- 172.21.15.122:0/2697267551 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39800d01f0 con 0x7f3980145590 2024-03-06T07:28:35.726 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:28:35.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f396c06b4d0 msgr2=0x7f396c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f396c06b4d0 0x7f396c06d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3970036000 tx=0x7f397003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 msgr2=0x7f39800c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f39800c9ce0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f397800cd70 tx=0x7f397800a660 comp rx=0 tx=0).stop 2024-03-06T07:28:35.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 shutdown_connections 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f396c06b4d0 0x7f396c06d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3980145590 0x7f39800c9ce0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f398006f510 0x7f39800b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 --2- 172.21.15.122:0/2697267551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f398006b4f0 0x7f398015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 >> 172.21.15.122:0/2697267551 conn(0x7f39800b2660 msgr2=0x7f39800d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 shutdown_connections 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.726+0000 7f3985f4b640 1 -- 172.21.15.122:0/2697267551 wait complete. 2024-03-06T07:28:35.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:28:35.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.838+0000 7f96e6c88640 1 Processor -- start 2024-03-06T07:28:35.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e6c88640 1 -- start start 2024-03-06T07:28:35.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e6c88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e015dfe0 0x7f96e00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e6c88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f96e00cf830 0x7f96e00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e6c88640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e00b3430 con 0x7f96e00d27f0 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e6c88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e00d2c50 con 0x7f96e00cf830 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e6c88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e00b31e0 con 0x7f96e015dfe0 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96dffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f96e00cf830 0x7f96e00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e5225640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f96e00d27f0 0x7f96e00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58286/0 (socket says 172.21.15.122:58286) 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.839+0000 7f96e5225640 1 -- 172.21.15.122:0/1637348881 learned_addr learned my addr 172.21.15.122:0/1637348881 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96dffff640 1 -- 172.21.15.122:0/1637348881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e015dfe0 msgr2=0x7f96e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96e4a24640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e015dfe0 0x7f96e00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96df7fe640 1 -- 172.21.15.122:0/1637348881 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300983911 0 0) 0x7f96e00b3430 con 0x7f96e00d27f0 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96dffff640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e015dfe0 0x7f96e00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96dffff640 1 -- 172.21.15.122:0/1637348881 >> v1:172.21.15.122:6789/0 conn(0x7f96e00d27f0 legacy=0x7f96e00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96dffff640 1 -- 172.21.15.122:0/1637348881 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e00b3710 con 0x7f96e00cf830 2024-03-06T07:28:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.840+0000 7f96e4a24640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e015dfe0 0x7f96e00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96dffff640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e00cf830 0x7f96e00d22b0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0097a0 tx=0x7f96cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cdc0932305041d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96df7fe640 1 -- 172.21.15.122:0/1637348881 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc035020 con 0x7f96e00cf830 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96df7fe640 1 -- 172.21.15.122:0/1637348881 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc031830 con 0x7f96e00cf830 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96df7fe640 1 -- 172.21.15.122:0/1637348881 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc031c00 con 0x7f96e00cf830 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 -- 172.21.15.122:0/1637348881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e00cf830 msgr2=0x7f96e00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e00cf830 0x7f96e00d22b0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0097a0 tx=0x7f96cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 -- 172.21.15.122:0/1637348881 shutdown_connections 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e00cf830 0x7f96e00d22b0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1637348881 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e015dfe0 0x7f96e00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 -- 172.21.15.122:0/1637348881 >> 172.21.15.122:0/1637348881 conn(0x7f96e00d3cd0 msgr2=0x7f96e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 -- 172.21.15.122:0/1637348881 shutdown_connections 2024-03-06T07:28:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.841+0000 7f96e6c88640 1 -- 172.21.15.122:0/1637348881 wait complete. 2024-03-06T07:28:35.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.842+0000 7f96e6c88640 1 Processor -- start 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.842+0000 7f96e6c88640 1 -- start start 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e6c88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96e00cf830 0x7f96e014ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e6c88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e00d27f0 0x7f96e00cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e6c88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 0x7f96e00cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e6c88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e00d1c70 con 0x7f96e015dfe0 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e6c88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e00b3d60 con 0x7f96e00cf830 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e6c88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e00d1df0 con 0x7f96e00d27f0 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e5225640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 0x7f96e00cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e5225640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 0x7f96e00cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46954/0 (socket says 172.21.15.122:46954) 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.843+0000 7f96e5225640 1 -- 172.21.15.122:0/1040337428 learned_addr learned my addr 172.21.15.122:0/1040337428 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e5225640 1 -- 172.21.15.122:0/1040337428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e00d27f0 msgr2=0x7f96e00cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e5225640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e00d27f0 0x7f96e00cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e5225640 1 -- 172.21.15.122:0/1040337428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96e00cf830 msgr2=0x7f96e014ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e5225640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96e00cf830 0x7f96e014ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e5225640 1 -- 172.21.15.122:0/1040337428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e00cdd50 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e5225640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 0x7f96e00cd810 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f96d800c7a0 tx=0x7f96d800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8018070 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d800a640 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e00cbd90 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d800aa70 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.844+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e00cc350 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.845+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e004f330 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.845+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96d801b070 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.845+0000 7f96dd7fa640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f96c406b4d0 0x7f96c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.846+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f96d80895f0 con 0x7f96e015dfe0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.846+0000 7f96e4a24640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f96c406b4d0 0x7f96c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.846+0000 7f96e4a24640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f96c406b4d0 0x7f96c406d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006fd0 tx=0x7f96d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:35.850+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96d8058680 con 0x7f96e015dfe0 2024-03-06T07:28:36.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.085+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f96e00b3360 con 0x7f96e015dfe0 2024-03-06T07:28:36.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.127+0000 7f96dd7fa640 1 -- 172.21.15.122:0/1040337428 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v360) v1 ==== 143+0+0 (secure 0 0 0) 0x7f96d8057f80 con 0x7f96e015dfe0 2024-03-06T07:28:36.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f96c406b4d0 msgr2=0x7f96c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f96c406b4d0 0x7f96c406d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006fd0 tx=0x7f96d4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 msgr2=0x7f96e00cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 0x7f96e00cd810 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f96d800c7a0 tx=0x7f96d800cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 shutdown_connections 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f96c406b4d0 0x7f96c406d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e015dfe0 0x7f96e00cd810 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96e00d27f0 0x7f96e00cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 --2- 172.21.15.122:0/1040337428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96e00cf830 0x7f96e014ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 >> 172.21.15.122:0/1040337428 conn(0x7f96e00d3cd0 msgr2=0x7f96e00b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 shutdown_connections 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.129+0000 7f96e6c88640 1 -- 172.21.15.122:0/1040337428 wait complete. 2024-03-06T07:28:36.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:36.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.243+0000 7f014d9f6640 1 Processor -- start 2024-03-06T07:28:36.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.243+0000 7f014d9f6640 1 -- start start 2024-03-06T07:28:36.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f014d9f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f01481586e0 0x7f01480aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f014d9f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480aef80 0x7f01480cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f014d9f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f01480cffd0 0x7f014806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f014d9f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480d03f0 con 0x7f01480cffd0 2024-03-06T07:28:36.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f014d9f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0148158ac0 con 0x7f01481586e0 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f014d9f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014815b2a0 con 0x7f01480aef80 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f01480cffd0 0x7f014806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f01480cffd0 0x7f014806da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46968/0 (socket says 172.21.15.122:46968) 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 -- 172.21.15.122:0/1895504453 learned_addr learned my addr 172.21.15.122:0/1895504453 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 -- 172.21.15.122:0/1895504453 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480aef80 msgr2=0x7f01480cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 --2- 172.21.15.122:0/1895504453 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480aef80 0x7f01480cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 -- 172.21.15.122:0/1895504453 >> v2:172.21.15.169:3300/0 conn(0x7f01481586e0 msgr2=0x7f01480aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 --2- 172.21.15.122:0/1895504453 >> v2:172.21.15.169:3300/0 conn(0x7f01481586e0 0x7f01480aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.244+0000 7f01477fe640 1 -- 172.21.15.122:0/1895504453 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014815b540 con 0x7f01480cffd0 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.245+0000 7f01477fe640 1 --2- 172.21.15.122:0/1895504453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01480cffd0 0x7f014806da20 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f0148070d10 tx=0x7f014002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60d60d4586e3764 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.245+0000 7f0145ffb640 1 -- 172.21.15.122:0/1895504453 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014002f8e0 con 0x7f01480cffd0 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.245+0000 7f0145ffb640 1 -- 172.21.15.122:0/1895504453 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014002fa80 con 0x7f01480cffd0 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.245+0000 7f0145ffb640 1 -- 172.21.15.122:0/1895504453 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014002fe50 con 0x7f01480cffd0 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.245+0000 7f014d9f6640 1 -- 172.21.15.122:0/1895504453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01480cffd0 msgr2=0x7f014806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.245+0000 7f014d9f6640 1 --2- 172.21.15.122:0/1895504453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01480cffd0 0x7f014806da20 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f0148070d10 tx=0x7f014002db20 comp rx=0 tx=0).stop 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 -- 172.21.15.122:0/1895504453 shutdown_connections 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 --2- 172.21.15.122:0/1895504453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01480cffd0 0x7f014806da20 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 --2- 172.21.15.122:0/1895504453 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480aef80 0x7f01480cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 --2- 172.21.15.122:0/1895504453 >> v2:172.21.15.169:3300/0 conn(0x7f01481586e0 0x7f01480aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 -- 172.21.15.122:0/1895504453 >> 172.21.15.122:0/1895504453 conn(0x7f014806b190 msgr2=0x7f014806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 -- 172.21.15.122:0/1895504453 shutdown_connections 2024-03-06T07:28:36.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.246+0000 7f014d9f6640 1 -- 172.21.15.122:0/1895504453 wait complete. 2024-03-06T07:28:36.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.247+0000 7f014d9f6640 1 Processor -- start 2024-03-06T07:28:36.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.247+0000 7f014d9f6640 1 -- start start 2024-03-06T07:28:36.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.247+0000 7f014d9f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01480aef80 0x7f0148145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.247+0000 7f014d9f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480cffd0 0x7f01481f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.247+0000 7f014d9f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 0x7f01480ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f014d9f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014815bde0 con 0x7f01481586e0 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f014d9f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014815bb00 con 0x7f01480aef80 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f014d9f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0148071350 con 0x7f01480cffd0 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 0x7f01480ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 0x7f01480ae150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46972/0 (socket says 172.21.15.122:46972) 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 -- 172.21.15.122:0/424996240 learned_addr learned my addr 172.21.15.122:0/424996240 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 -- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480cffd0 msgr2=0x7f01481f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480cffd0 0x7f01481f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 -- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01480aef80 msgr2=0x7f0148145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f0146ffd640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01480aef80 0x7f0148145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01480aef80 0x7f0148145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 -- 172.21.15.122:0/424996240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01481efbb0 con 0x7f01481586e0 2024-03-06T07:28:36.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f0146ffd640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01480aef80 0x7f0148145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f01477fe640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 0x7f01480ae150 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f01400090b0 tx=0x7f01400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140047070 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0140040420 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01481f38d0 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140040810 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.248+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01481f3e90 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.249+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01480732b0 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.250+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f014004a070 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.250+0000 7f012ffff640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f013406b4d0 0x7f013406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.250+0000 7f0146ffd640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f013406b4d0 0x7f013406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.250+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f01400ae4c0 con 0x7f01481586e0 2024-03-06T07:28:36.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.251+0000 7f0146ffd640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f013406b4d0 0x7f013406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f01480cf4b0 tx=0x7f01380073e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.255+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01481efbb0 con 0x7f01481586e0 2024-03-06T07:28:36.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.428+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f01480a9b10 con 0x7f01481586e0 2024-03-06T07:28:36.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.432+0000 7f012ffff640 1 -- 172.21.15.122:0/424996240 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f014007d670 con 0x7f01481586e0 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.434+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f013406b4d0 msgr2=0x7f013406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f013406b4d0 0x7f013406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f01480cf4b0 tx=0x7f01380073e0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 msgr2=0x7f01480ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 0x7f01480ae150 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f01400090b0 tx=0x7f01400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 shutdown_connections 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f013406b4d0 0x7f013406d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01481586e0 0x7f01480ae150 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f01480cffd0 0x7f01481f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 --2- 172.21.15.122:0/424996240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01480aef80 0x7f0148145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 >> 172.21.15.122:0/424996240 conn(0x7f014806b190 msgr2=0x7f0148159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 shutdown_connections 2024-03-06T07:28:36.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.435+0000 7f014d9f6640 1 -- 172.21.15.122:0/424996240 wait complete. 2024-03-06T07:28:36.448 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:36.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:36.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:36.525 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:36.525 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:28:36.525 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:36.540 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:36.541 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:28:36.541 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:36.582 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:36.582 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:28:36.583 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:36.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:36.597 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:28:36.597 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:36.640 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:36.640 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:36.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:36.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.738+0000 7f6f45ff7640 1 Processor -- start 2024-03-06T07:28:36.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.738+0000 7f6f45ff7640 1 -- start start 2024-03-06T07:28:36.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f45ff7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4013dc10 0x7f6f400b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f45ff7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6f4015ef60 0x7f6f400b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f45ff7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4015f580 con 0x7f6f4015bf30 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f45ff7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f400cf3f0 con 0x7f6f4015ef60 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f45ff7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4015b210 con 0x7f6f4013dc10 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6f4015ef60 0x7f6f400b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6f4015bf30 0x7f6f4015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58302/0 (socket says 172.21.15.122:58302) 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3effd640 1 -- 172.21.15.122:0/30212039 learned_addr learned my addr 172.21.15.122:0/30212039 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3ffff640 1 -- 172.21.15.122:0/30212039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4013dc10 msgr2=0x7f6f400b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3ffff640 1 --2- 172.21.15.122:0/30212039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4013dc10 0x7f6f400b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3ffff640 1 -- 172.21.15.122:0/30212039 >> v1:172.21.15.122:6789/0 conn(0x7f6f4015bf30 legacy=0x7f6f4015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3e7fc640 1 -- 172.21.15.122:0/30212039 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582067330 0 0) 0x7f6f4015f580 con 0x7f6f4015bf30 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.739+0000 7f6f3ffff640 1 -- 172.21.15.122:0/30212039 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4015b450 con 0x7f6f4015ef60 2024-03-06T07:28:36.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.740+0000 7f6f3ffff640 1 --2- 172.21.15.122:0/30212039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4015ef60 0x7f6f400b2e90 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009d90 tx=0x7f6f3802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f34b5777e44535 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.740+0000 7f6f3e7fc640 1 -- 172.21.15.122:0/30212039 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f38039070 con 0x7f6f4015ef60 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.740+0000 7f6f3e7fc640 1 -- 172.21.15.122:0/30212039 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3802cbc0 con 0x7f6f4015ef60 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.740+0000 7f6f3e7fc640 1 -- 172.21.15.122:0/30212039 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f38034440 con 0x7f6f4015ef60 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 -- 172.21.15.122:0/30212039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4015ef60 msgr2=0x7f6f400b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/30212039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4015ef60 0x7f6f400b2e90 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009d90 tx=0x7f6f3802be10 comp rx=0 tx=0).stop 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 -- 172.21.15.122:0/30212039 shutdown_connections 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/30212039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4015ef60 0x7f6f400b2e90 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/30212039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4013dc10 0x7f6f400b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 -- 172.21.15.122:0/30212039 >> 172.21.15.122:0/30212039 conn(0x7f6f400cf8a0 msgr2=0x7f6f400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 -- 172.21.15.122:0/30212039 shutdown_connections 2024-03-06T07:28:36.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.741+0000 7f6f45ff7640 1 -- 172.21.15.122:0/30212039 wait complete. 2024-03-06T07:28:36.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.742+0000 7f6f45ff7640 1 Processor -- start 2024-03-06T07:28:36.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.742+0000 7f6f45ff7640 1 -- start start 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 0x7f6f40156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f4015bf30 0x7f6f40139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4015ef60 0x7f6f401f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400708d0 con 0x7f6f4013dc10 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4013dff0 con 0x7f6f4015bf30 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f40070a50 con 0x7f6f4015ef60 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 0x7f6f40156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 0x7f6f40156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:46984/0 (socket says 172.21.15.122:46984) 2024-03-06T07:28:36.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 -- 172.21.15.122:0/1422312803 learned_addr learned my addr 172.21.15.122:0/1422312803 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 -- 172.21.15.122:0/1422312803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4015ef60 msgr2=0x7f6f401f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4015ef60 0x7f6f401f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 -- 172.21.15.122:0/1422312803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f4015bf30 msgr2=0x7f6f40139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f4015bf30 0x7f6f40139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 -- 172.21.15.122:0/1422312803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f401f4960 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f3f7fe640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 0x7f6f40156530 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000b4a0 tx=0x7f6f3000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30014070 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f30010610 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f401f4c40 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30010a00 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.743+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40158da0 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.744+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4004f330 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.745+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f30017070 con 0x7f6f4013dc10 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.745+0000 7f6f27fff640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f2c06b4d0 0x7f6f2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.745+0000 7f6f3effd640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f2c06b4d0 0x7f6f2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.745+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6f300899c0 con 0x7f6f4013dc10 2024-03-06T07:28:36.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.746+0000 7f6f3effd640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f2c06b4d0 0x7f6f2c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6f34007fd0 tx=0x7f6f34007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.749+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f3008be20 con 0x7f6f4013dc10 2024-03-06T07:28:36.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.956+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f4006eb00 con 0x7f6f4013dc10 2024-03-06T07:28:36.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.956+0000 7f6f27fff640 1 -- 172.21.15.122:0/1422312803 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6f30058ae0 con 0x7f6f4013dc10 2024-03-06T07:28:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:36.958 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/848090849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":848090849},{"type":"v1","addr":"172.21.15.169:6834","nonce":848090849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/270140398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":270140398},{"type":"v1","addr":"172.21.15.122:6835","nonce":270140398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8277,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2773387039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2773387039},{"type":"v1","addr":"172.21.15.169:6835","nonce":2773387039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3826686101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3826686101},{"type":"v1","addr":"172.21.15.122:6837","nonce":3826686101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[]} 2024-03-06T07:28:36.958 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 69 2024-03-06T07:28:36.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f2c06b4d0 msgr2=0x7f6f2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f2c06b4d0 0x7f6f2c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6f34007fd0 tx=0x7f6f34007920 comp rx=0 tx=0).stop 2024-03-06T07:28:36.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 msgr2=0x7f6f40156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 0x7f6f40156530 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000b4a0 tx=0x7f6f3000b960 comp rx=0 tx=0).stop 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 shutdown_connections 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6f2c06b4d0 0x7f6f2c06d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6f4015ef60 0x7f6f401f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6f4015bf30 0x7f6f40139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 --2- 172.21.15.122:0/1422312803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f4013dc10 0x7f6f40156530 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 >> 172.21.15.122:0/1422312803 conn(0x7f6f400cf8a0 msgr2=0x7f6f400af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 shutdown_connections 2024-03-06T07:28:36.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:36.958+0000 7f6f45ff7640 1 -- 172.21.15.122:0/1422312803 wait complete. 2024-03-06T07:28:36.972 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:36.972 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:36.972 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:36.972 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.972 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:36.972 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:36.972 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:36.973 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.973 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:36.973 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:36.973 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:36.973 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.973 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:36.973 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:36.974 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:36.974 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.981 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:36.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:36.982 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:36.982 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:36.985 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:36.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:36.986 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:36.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:37.026 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:37.026 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.027 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:37.027 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.028 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:37.028 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.029 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:37.029 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:37.102 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:28:37.102 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:28:37.113 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:28:37.113 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:28:37.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.133+0000 7f63b955e640 1 Processor -- start 2024-03-06T07:28:37.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b955e640 1 -- start start 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b955e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63b40670a0 0x7f63b406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b955e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f63b40b50d0 0x7f63b40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b955e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b5c30 con 0x7f63b40d02b0 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b955e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b40cd940 con 0x7f63b40670a0 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b955e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b40b5f10 con 0x7f63b40b50d0 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63abfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63b40670a0 0x7f63b406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b2ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f63b40d02b0 0x7f63b406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58310/0 (socket says 172.21.15.122:58310) 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.134+0000 7f63b2ffd640 1 -- 172.21.15.122:0/1263623657 learned_addr learned my addr 172.21.15.122:0/1263623657 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63b37fe640 1 --2- 172.21.15.122:0/1263623657 >> v2:172.21.15.169:3300/0 conn(0x7f63b40b50d0 0x7f63b40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63abfff640 1 -- 172.21.15.122:0/1263623657 >> v2:172.21.15.169:3300/0 conn(0x7f63b40b50d0 msgr2=0x7f63b40ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63abfff640 1 --2- 172.21.15.122:0/1263623657 >> v2:172.21.15.169:3300/0 conn(0x7f63b40b50d0 0x7f63b40ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63abfff640 1 -- 172.21.15.122:0/1263623657 >> v1:172.21.15.122:6789/0 conn(0x7f63b40d02b0 legacy=0x7f63b406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63abfff640 1 -- 172.21.15.122:0/1263623657 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40cd5d0 con 0x7f63b40670a0 2024-03-06T07:28:37.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63b27fc640 1 -- 172.21.15.122:0/1263623657 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476980188 0 0) 0x7f63b40b5c30 con 0x7f63b40d02b0 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.135+0000 7f63abfff640 1 --2- 172.21.15.122:0/1263623657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40670a0 0x7f63b406b6c0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f63b406cf00 tx=0x7f639c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411dbd7a8a8fbe4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b27fc640 1 -- 172.21.15.122:0/1263623657 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c0315a0 con 0x7f63b40670a0 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b27fc640 1 -- 172.21.15.122:0/1263623657 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c037070 con 0x7f63b40670a0 2024-03-06T07:28:37.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b27fc640 1 -- 172.21.15.122:0/1263623657 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031960 con 0x7f63b40670a0 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 -- 172.21.15.122:0/1263623657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40670a0 msgr2=0x7f63b406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 --2- 172.21.15.122:0/1263623657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40670a0 0x7f63b406b6c0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f63b406cf00 tx=0x7f639c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 -- 172.21.15.122:0/1263623657 shutdown_connections 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 --2- 172.21.15.122:0/1263623657 >> v2:172.21.15.169:3300/0 conn(0x7f63b40b50d0 0x7f63b40ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 --2- 172.21.15.122:0/1263623657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40670a0 0x7f63b406b6c0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 -- 172.21.15.122:0/1263623657 >> 172.21.15.122:0/1263623657 conn(0x7f63b4072520 msgr2=0x7f63b4072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 -- 172.21.15.122:0/1263623657 shutdown_connections 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.136+0000 7f63b955e640 1 -- 172.21.15.122:0/1263623657 wait complete. 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.138+0000 7f63b955e640 1 Processor -- start 2024-03-06T07:28:37.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.138+0000 7f63b955e640 1 -- start start 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.138+0000 7f63b955e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63b40670a0 0x7f63b40c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.138+0000 7f63b955e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63b40b50d0 0x7f63b4151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.138+0000 7f63b955e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 0x7f63b409d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b955e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b406bd80 con 0x7f63b40d02b0 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b955e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b406bc00 con 0x7f63b40b50d0 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b955e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40d0690 con 0x7f63b40670a0 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 0x7f63b409d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 0x7f63b409d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47040/0 (socket says 172.21.15.122:47040) 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 -- 172.21.15.122:0/4229556051 learned_addr learned my addr 172.21.15.122:0/4229556051 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63abfff640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63b40b50d0 0x7f63b4151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b2ffd640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63b40670a0 0x7f63b40c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 -- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63b40670a0 msgr2=0x7f63b40c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63b40670a0 0x7f63b40c4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 -- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63b40b50d0 msgr2=0x7f63b4151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63b40b50d0 0x7f63b4151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 -- 172.21.15.122:0/4229556051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4152100 con 0x7f63b40d02b0 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63abfff640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63b40b50d0 0x7f63b4151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:37.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.139+0000 7f63b37fe640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 0x7f63b409d8a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00efc0 tx=0x7f63ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.141+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac018020 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.141+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac014480 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.141+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b41523e0 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.141+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac014870 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.141+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b4156980 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.142+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b404f330 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.142+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63ac005d10 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.143+0000 7f63ab7fe640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f639406b4d0 0x7f639406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.143+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f63ac08a950 con 0x7f63b40d02b0 2024-03-06T07:28:37.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.143+0000 7f63b2ffd640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f639406b4d0 0x7f639406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.143+0000 7f63b2ffd640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f639406b4d0 0x7f639406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f63a4007fd0 tx=0x7f63a4007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.149+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63ac022070 con 0x7f63b40d02b0 2024-03-06T07:28:37.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:37.334+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f63b40ae270 con 0x7f63b40d02b0 2024-03-06T07:28:38.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.159+0000 7f63ab7fe640 1 -- 172.21.15.122:0/4229556051 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v122) v1 ==== 97+0+0 (secure 0 0 0) 0x7f63ac010040 con 0x7f63b40d02b0 2024-03-06T07:28:38.162 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:28:38.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f639406b4d0 msgr2=0x7f639406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f639406b4d0 0x7f639406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f63a4007fd0 tx=0x7f63a4007de0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 msgr2=0x7f63b409d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 0x7f63b409d8a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00efc0 tx=0x7f63ac00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:38.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 shutdown_connections 2024-03-06T07:28:38.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f639406b4d0 0x7f639406d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63b40d02b0 0x7f63b409d8a0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63b40b50d0 0x7f63b4151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 --2- 172.21.15.122:0/4229556051 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63b40670a0 0x7f63b40c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 >> 172.21.15.122:0/4229556051 conn(0x7f63b4072520 msgr2=0x7f63b415f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 shutdown_connections 2024-03-06T07:28:38.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.162+0000 7f63b955e640 1 -- 172.21.15.122:0/4229556051 wait complete. 2024-03-06T07:28:38.176 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:38.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.277+0000 7fa262583640 1 Processor -- start 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.277+0000 7fa262583640 1 -- start start 2024-03-06T07:28:38.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa262583640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa25c15bb20 0x7fa25c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa262583640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0cfff0 0x7fa25c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa262583640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa25c0ce4b0 0x7fa25c0ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa262583640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c161980 con 0x7fa25c15bb20 2024-03-06T07:28:38.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa262583640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c161c60 con 0x7fa25c0ce4b0 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa262583640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c161f40 con 0x7fa25c0cfff0 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa25bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa25c15bb20 0x7fa25c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa25bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa25c15bb20 0x7fa25c0d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47046/0 (socket says 172.21.15.122:47046) 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.278+0000 7fa25bfff640 1 -- 172.21.15.122:0/2306745543 learned_addr learned my addr 172.21.15.122:0/2306745543 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25b7fe640 1 --2- 172.21.15.122:0/2306745543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0cfff0 0x7fa25c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa260b20640 1 --2- 172.21.15.122:0/2306745543 >> v2:172.21.15.169:3300/0 conn(0x7fa25c0ce4b0 0x7fa25c0ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25bfff640 1 -- 172.21.15.122:0/2306745543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0cfff0 msgr2=0x7fa25c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25bfff640 1 --2- 172.21.15.122:0/2306745543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0cfff0 0x7fa25c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25bfff640 1 -- 172.21.15.122:0/2306745543 >> v2:172.21.15.169:3300/0 conn(0x7fa25c0ce4b0 msgr2=0x7fa25c0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25bfff640 1 --2- 172.21.15.122:0/2306745543 >> v2:172.21.15.169:3300/0 conn(0x7fa25c0ce4b0 0x7fa25c0ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25bfff640 1 -- 172.21.15.122:0/2306745543 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c162220 con 0x7fa25c15bb20 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.279+0000 7fa25bfff640 1 --2- 172.21.15.122:0/2306745543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c15bb20 0x7fa25c0d1df0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0b13b0 tx=0x7fa24c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f56d272fccb16a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa25affd640 1 -- 172.21.15.122:0/2306745543 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c035020 con 0x7fa25c15bb20 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa25affd640 1 -- 172.21.15.122:0/2306745543 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c031870 con 0x7fa25c15bb20 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa25affd640 1 -- 172.21.15.122:0/2306745543 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c031c40 con 0x7fa25c15bb20 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 -- 172.21.15.122:0/2306745543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c15bb20 msgr2=0x7fa25c0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 --2- 172.21.15.122:0/2306745543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c15bb20 0x7fa25c0d1df0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0b13b0 tx=0x7fa24c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 -- 172.21.15.122:0/2306745543 shutdown_connections 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 --2- 172.21.15.122:0/2306745543 >> v2:172.21.15.169:3300/0 conn(0x7fa25c0ce4b0 0x7fa25c0ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 --2- 172.21.15.122:0/2306745543 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0cfff0 0x7fa25c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 --2- 172.21.15.122:0/2306745543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c15bb20 0x7fa25c0d1df0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 -- 172.21.15.122:0/2306745543 >> 172.21.15.122:0/2306745543 conn(0x7fa25c09a9f0 msgr2=0x7fa25c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 -- 172.21.15.122:0/2306745543 shutdown_connections 2024-03-06T07:28:38.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.280+0000 7fa262583640 1 -- 172.21.15.122:0/2306745543 wait complete. 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.281+0000 7fa262583640 1 Processor -- start 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.281+0000 7fa262583640 1 -- start start 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa262583640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0ce4b0 0x7fa25c1f16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa262583640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 0x7fa25c1f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa262583640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa25c15bb20 0x7fa25c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa262583640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0d24b0 con 0x7fa25c0cfff0 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa262583640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0d2330 con 0x7fa25c15bb20 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa262583640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0d1980 con 0x7fa25c0ce4b0 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa25b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 0x7fa25c1f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa25b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 0x7fa25c1f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47048/0 (socket says 172.21.15.122:47048) 2024-03-06T07:28:38.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa25b7fe640 1 -- 172.21.15.122:0/1421628517 learned_addr learned my addr 172.21.15.122:0/1421628517 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa25bfff640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0ce4b0 0x7fa25c1f16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.282+0000 7fa260b20640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa25c15bb20 0x7fa25c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa25b7fe640 1 -- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0ce4b0 msgr2=0x7fa25c1f16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa25b7fe640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0ce4b0 0x7fa25c1f16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa25b7fe640 1 -- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa25c15bb20 msgr2=0x7fa25c0c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa25b7fe640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa25c15bb20 0x7fa25c0c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa25b7fe640 1 -- 172.21.15.122:0/1421628517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0aeb40 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa25b7fe640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 0x7fa25c1f52c0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fa25000d800 tx=0x7fa25000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000b6c0 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa250019070 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c0aee20 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000ba80 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.283+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c0af3e0 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.284+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c0cc200 con 0x7fa25c0cfff0 2024-03-06T07:28:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.284+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa25000bc20 con 0x7fa25c0cfff0 2024-03-06T07:28:38.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.285+0000 7fa258ff9640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa24806b4d0 0x7fa24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.285+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa25008a390 con 0x7fa25c0cfff0 2024-03-06T07:28:38.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.285+0000 7fa25bfff640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa24806b4d0 0x7fa24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.285+0000 7fa25bfff640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa24806b4d0 0x7fa24806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa24c036000 tx=0x7fa24c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.289+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa250059560 con 0x7fa25c0cfff0 2024-03-06T07:28:38.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.524+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa25c04f330 con 0x7fa25c0cfff0 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.525+0000 7fa258ff9640 1 -- 172.21.15.122:0/1421628517 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa250058e60 con 0x7fa25c0cfff0 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.528 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:28:38.529 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.530 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:28:38.536 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:38.540 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:28:38.540 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa24806b4d0 msgr2=0x7fa24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa24806b4d0 0x7fa24806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa24c036000 tx=0x7fa24c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 msgr2=0x7fa25c1f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 0x7fa25c1f52c0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fa25000d800 tx=0x7fa25000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 shutdown_connections 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa24806b4d0 0x7fa24806d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa25c15bb20 0x7fa25c0c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa25c0cfff0 0x7fa25c1f52c0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 --2- 172.21.15.122:0/1421628517 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa25c0ce4b0 0x7fa25c1f16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 >> 172.21.15.122:0/1421628517 conn(0x7fa25c09a9f0 msgr2=0x7fa25c15c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 shutdown_connections 2024-03-06T07:28:38.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.527+0000 7fa262583640 1 -- 172.21.15.122:0/1421628517 wait complete. 2024-03-06T07:28:38.542 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:38.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.640+0000 7f00da9a2640 1 Processor -- start 2024-03-06T07:28:38.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.640+0000 7f00da9a2640 1 -- start start 2024-03-06T07:28:38.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00da9a2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00d415e990 0x7f00d415ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00da9a2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d406bfc0 0x7f00d406c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00da9a2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d41626b0 con 0x7f00d4162250 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00da9a2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d40af620 con 0x7f00d415e990 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00da9a2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40af900 con 0x7f00d406bfc0 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00d99a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00d415e990 0x7f00d415ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00d99a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00d415e990 0x7f00d415ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47058/0 (socket says 172.21.15.122:47058) 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.641+0000 7f00d99a0640 1 -- 172.21.15.122:0/1583717022 learned_addr learned my addr 172.21.15.122:0/1583717022 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00da1a1640 1 --2- 172.21.15.122:0/1583717022 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d406bfc0 0x7f00d406c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d99a0640 1 -- 172.21.15.122:0/1583717022 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d406bfc0 msgr2=0x7f00d406c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d99a0640 1 --2- 172.21.15.122:0/1583717022 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d406bfc0 0x7f00d406c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d99a0640 1 -- 172.21.15.122:0/1583717022 >> v1:172.21.15.169:6789/0 conn(0x7f00d4162250 legacy=0x7f00d40aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d99a0640 1 -- 172.21.15.122:0/1583717022 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d4161770 con 0x7f00d415e990 2024-03-06T07:28:38.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d99a0640 1 --2- 172.21.15.122:0/1583717022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d415e990 0x7f00d415ed70 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009870 tx=0x7f00d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1008af92945b09f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d899e640 1 -- 172.21.15.122:0/1583717022 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d0035020 con 0x7f00d415e990 2024-03-06T07:28:38.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d899e640 1 -- 172.21.15.122:0/1583717022 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d0031640 con 0x7f00d415e990 2024-03-06T07:28:38.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.642+0000 7f00d899e640 1 -- 172.21.15.122:0/1583717022 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d002d040 con 0x7f00d415e990 2024-03-06T07:28:38.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 -- 172.21.15.122:0/1583717022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d415e990 msgr2=0x7f00d415ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1583717022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d415e990 0x7f00d415ed70 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009870 tx=0x7f00d002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 -- 172.21.15.122:0/1583717022 shutdown_connections 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1583717022 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d406bfc0 0x7f00d406c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1583717022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d415e990 0x7f00d415ed70 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 -- 172.21.15.122:0/1583717022 >> 172.21.15.122:0/1583717022 conn(0x7f00d406ddf0 msgr2=0x7f00d406e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 -- 172.21.15.122:0/1583717022 shutdown_connections 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.643+0000 7f00da9a2640 1 -- 172.21.15.122:0/1583717022 wait complete. 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.644+0000 7f00da9a2640 1 Processor -- start 2024-03-06T07:28:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.644+0000 7f00da9a2640 1 -- start start 2024-03-06T07:28:38.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da9a2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d415e990 0x7f00d4160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da9a2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00d4162250 0x7f00d41f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da9a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 0x7f00d4152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da9a2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40717d0 con 0x7f00d41f77a0 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da9a2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4071650 con 0x7f00d4162250 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da9a2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4071950 con 0x7f00d415e990 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 0x7f00d4152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 0x7f00d4152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47070/0 (socket says 172.21.15.122:47070) 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 -- 172.21.15.122:0/1972384720 learned_addr learned my addr 172.21.15.122:0/1972384720 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00d99a0640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d415e990 0x7f00d4160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00d919f640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00d4162250 0x7f00d41f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 -- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d415e990 msgr2=0x7f00d4160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d415e990 0x7f00d4160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 -- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00d4162250 msgr2=0x7f00d41f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00d4162250 0x7f00d41f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 -- 172.21.15.122:0/1972384720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d41f7d20 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00d919f640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00d4162250 0x7f00d41f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.645+0000 7f00da1a1640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 0x7f00d4152320 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00cd70 tx=0x7f00cc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.646+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc003b90 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.646+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc003d30 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.646+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc004240 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.646+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d41f8000 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.646+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d4154040 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.647+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d4154240 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.647+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00cc0043e0 con 0x7f00d41f77a0 2024-03-06T07:28:38.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.647+0000 7f00ca7fc640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f00b806b4d0 0x7f00b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.647+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f00cc08a9f0 con 0x7f00d41f77a0 2024-03-06T07:28:38.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.647+0000 7f00d99a0640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f00b806b4d0 0x7f00b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.648+0000 7f00d99a0640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f00b806b4d0 0x7f00b806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f00d0036000 tx=0x7f00d003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.652+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00d4154240 con 0x7f00d41f77a0 2024-03-06T07:28:38.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.823+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00d4153b50 con 0x7f00d41f77a0 2024-03-06T07:28:38.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.824+0000 7f00ca7fc640 1 -- 172.21.15.122:0/1972384720 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+12901 (secure 0 0 0) 0x7f00cc01d340 con 0x7f00d41f77a0 2024-03-06T07:28:38.826 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:38.826 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":122,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:28:38.150905+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f00b806b4d0 msgr2=0x7f00b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f00b806b4d0 0x7f00b806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f00d0036000 tx=0x7f00d003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 msgr2=0x7f00d4152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 0x7f00d4152320 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00cd70 tx=0x7f00cc00a660 comp rx=0 tx=0).stop 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 shutdown_connections 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f00b806b4d0 0x7f00b806d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00d41f77a0 0x7f00d4152320 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00d4162250 0x7f00d41f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.825+0000 7f00da9a2640 1 --2- 172.21.15.122:0/1972384720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00d415e990 0x7f00d4160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.826+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 >> 172.21.15.122:0/1972384720 conn(0x7f00d406bfc0 msgr2=0x7f00d40b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.826+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 shutdown_connections 2024-03-06T07:28:38.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.826+0000 7f00da9a2640 1 -- 172.21.15.122:0/1972384720 wait complete. 2024-03-06T07:28:38.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:38.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.941+0000 7f3bd7dd0640 1 Processor -- start 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.941+0000 7f3bd7dd0640 1 -- start start 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd7dd0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd00cea50 0x7f3bd00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd7dd0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3bd006cab0 0x7f3bd00ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd7dd0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd006d030 con 0x7f3bd015def0 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd7dd0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd00cf1c0 con 0x7f3bd006cab0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd7dd0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00cc760 con 0x7f3bd00cea50 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd636d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3bd006cab0 0x7f3bd00ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd5b6c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3bd015def0 0x7f3bd00ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58318/0 (socket says 172.21.15.122:58318) 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd5b6c640 1 -- 172.21.15.122:0/656301811 learned_addr learned my addr 172.21.15.122:0/656301811 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd536b640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd00cea50 0x7f3bd00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd636d640 1 -- 172.21.15.122:0/656301811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd00cea50 msgr2=0x7f3bd00b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd636d640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd00cea50 0x7f3bd00b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd636d640 1 -- 172.21.15.122:0/656301811 >> v1:172.21.15.122:6789/0 conn(0x7f3bd015def0 legacy=0x7f3bd00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd636d640 1 -- 172.21.15.122:0/656301811 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd006d310 con 0x7f3bd006cab0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd4b6a640 1 -- 172.21.15.122:0/656301811 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136713687 0 0) 0x7f3bd006d030 con 0x7f3bd015def0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.942+0000 7f3bd536b640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd00cea50 0x7f3bd00b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.943+0000 7f3bd636d640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd006cab0 0x7f3bd00ccbf0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009d90 tx=0x7f3bcc02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394e6254acfaa9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.943+0000 7f3bd4b6a640 1 -- 172.21.15.122:0/656301811 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc039070 con 0x7f3bd006cab0 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.943+0000 7f3bd4b6a640 1 -- 172.21.15.122:0/656301811 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc02cce0 con 0x7f3bd006cab0 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.943+0000 7f3bd4b6a640 1 -- 172.21.15.122:0/656301811 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc034660 con 0x7f3bd006cab0 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.943+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/656301811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd006cab0 msgr2=0x7f3bd00ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.943+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd006cab0 0x7f3bd00ccbf0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009d90 tx=0x7f3bcc02bf30 comp rx=0 tx=0).stop 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.944+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/656301811 shutdown_connections 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.944+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd006cab0 0x7f3bd00ccbf0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.944+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/656301811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd00cea50 0x7f3bd00b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.944+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/656301811 >> 172.21.15.122:0/656301811 conn(0x7f3bd00b3690 msgr2=0x7f3bd00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.944+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/656301811 shutdown_connections 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.944+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/656301811 wait complete. 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 Processor -- start 2024-03-06T07:28:38.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 -- start start 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd006cab0 0x7f3bd00c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd00cea50 0x7f3bd00b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 0x7f3bd00c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd015e2d0 con 0x7f3bd015def0 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00cc3f0 con 0x7f3bd00cea50 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.945+0000 7f3bd7dd0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd015e450 con 0x7f3bd006cab0 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 0x7f3bd00c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 0x7f3bd00c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47094/0 (socket says 172.21.15.122:47094) 2024-03-06T07:28:38.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 -- 172.21.15.122:0/1196987851 learned_addr learned my addr 172.21.15.122:0/1196987851 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd5b6c640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd006cab0 0x7f3bd00c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 -- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd006cab0 msgr2=0x7f3bd00c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd536b640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd00cea50 0x7f3bd00b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd006cab0 0x7f3bd00c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 -- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd00cea50 msgr2=0x7f3bd00b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd00cea50 0x7f3bd00b7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 -- 172.21.15.122:0/1196987851 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd00b63b0 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd636d640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 0x7f3bd00c60f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009d90 tx=0x7f3bcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc044020 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc03f470 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd00b6690 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc039040 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.946+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd00b6c50 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.947+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd009fcf0 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.948+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bcc047020 con 0x7f3bd015def0 2024-03-06T07:28:38.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.948+0000 7f3bc67fc640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3bb406b4d0 0x7f3bb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.948+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3bcc0aca70 con 0x7f3bd015def0 2024-03-06T07:28:38.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.948+0000 7f3bd5b6c640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3bb406b4d0 0x7f3bb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.948+0000 7f3bd5b6c640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3bb406b4d0 0x7f3bb406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8000f60 tx=0x7f3bc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:38.952+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3bcc04e030 con 0x7f3bd015def0 2024-03-06T07:28:39.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.123+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bd015d710 con 0x7f3bd015def0 2024-03-06T07:28:39.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.123+0000 7f3bc67fc640 1 -- 172.21.15.122:0/1196987851 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3bcc07bcd0 con 0x7f3bd015def0 2024-03-06T07:28:39.125 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3bb406b4d0 msgr2=0x7f3bb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3bb406b4d0 0x7f3bb406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8000f60 tx=0x7f3bc8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 msgr2=0x7f3bd00c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 0x7f3bd00c60f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009d90 tx=0x7f3bcc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 shutdown_connections 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3bb406b4d0 0x7f3bb406d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bd015def0 0x7f3bd00c60f0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd00cea50 0x7f3bd00b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 --2- 172.21.15.122:0/1196987851 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3bd006cab0 0x7f3bd00c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 >> 172.21.15.122:0/1196987851 conn(0x7f3bd00b3690 msgr2=0x7f3bd01605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 shutdown_connections 2024-03-06T07:28:39.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.125+0000 7f3bd7dd0640 1 -- 172.21.15.122:0/1196987851 wait complete. 2024-03-06T07:28:39.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:39.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.237+0000 7efd8dc4e640 1 Processor -- start 2024-03-06T07:28:39.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.237+0000 7efd8dc4e640 1 -- start start 2024-03-06T07:28:39.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd8dc4e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd880b3060 0x7efd8815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd8dc4e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efd8815bf30 0x7efd8815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd8dc4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8815f2d0 con 0x7efd8813dc10 2024-03-06T07:28:39.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd8dc4e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd880cf3f0 con 0x7efd880b3060 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd8dc4e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8815b210 con 0x7efd8815bf30 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd86ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd880b3060 0x7efd8815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd877fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efd8813dc10 0x7efd880b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58324/0 (socket says 172.21.15.122:58324) 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd877fe640 1 -- 172.21.15.122:0/1114358828 learned_addr learned my addr 172.21.15.122:0/1114358828 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd87fff640 1 --2- 172.21.15.122:0/1114358828 >> v2:172.21.15.169:3300/0 conn(0x7efd8815bf30 0x7efd8815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd86ffd640 1 -- 172.21.15.122:0/1114358828 >> v2:172.21.15.169:3300/0 conn(0x7efd8815bf30 msgr2=0x7efd8815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd86ffd640 1 --2- 172.21.15.122:0/1114358828 >> v2:172.21.15.169:3300/0 conn(0x7efd8815bf30 0x7efd8815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd86ffd640 1 -- 172.21.15.122:0/1114358828 >> v1:172.21.15.122:6789/0 conn(0x7efd8813dc10 legacy=0x7efd880b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd86ffd640 1 -- 172.21.15.122:0/1114358828 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8815f580 con 0x7efd880b3060 2024-03-06T07:28:39.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.238+0000 7efd867fc640 1 -- 172.21.15.122:0/1114358828 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408206009 0 0) 0x7efd8815f2d0 con 0x7efd8813dc10 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.239+0000 7efd86ffd640 1 --2- 172.21.15.122:0/1114358828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd880b3060 0x7efd8815e850 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7efd880b4e00 tx=0x7efd7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6433ac3b6814f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.239+0000 7efd867fc640 1 -- 172.21.15.122:0/1114358828 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c0315a0 con 0x7efd880b3060 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.239+0000 7efd867fc640 1 -- 172.21.15.122:0/1114358828 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c037070 con 0x7efd880b3060 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.239+0000 7efd867fc640 1 -- 172.21.15.122:0/1114358828 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c031960 con 0x7efd880b3060 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1114358828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd880b3060 msgr2=0x7efd8815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1114358828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd880b3060 0x7efd8815e850 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7efd880b4e00 tx=0x7efd7c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1114358828 shutdown_connections 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1114358828 >> v2:172.21.15.169:3300/0 conn(0x7efd8815bf30 0x7efd8815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1114358828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd880b3060 0x7efd8815e850 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1114358828 >> 172.21.15.122:0/1114358828 conn(0x7efd880cf8a0 msgr2=0x7efd880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1114358828 shutdown_connections 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.240+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1114358828 wait complete. 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.241+0000 7efd8dc4e640 1 Processor -- start 2024-03-06T07:28:39.243 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.241+0000 7efd8dc4e640 1 -- start start 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd880b3060 0x7efd88144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 0x7efd8812f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd8815bf30 0x7efd88141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8813dff0 con 0x7efd8813dc10 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8815b730 con 0x7efd8815bf30 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880708d0 con 0x7efd880b3060 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 0x7efd8812f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 0x7efd8812f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47114/0 (socket says 172.21.15.122:47114) 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 -- 172.21.15.122:0/1012263056 learned_addr learned my addr 172.21.15.122:0/1012263056 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd87fff640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd8815bf30 0x7efd88141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 -- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd880b3060 msgr2=0x7efd88144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd877fe640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd880b3060 0x7efd88144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd880b3060 0x7efd88144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 -- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd8815bf30 msgr2=0x7efd88141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd8815bf30 0x7efd88141500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 -- 172.21.15.122:0/1012263056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8813c840 con 0x7efd8813dc10 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd87fff640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd8815bf30 0x7efd88141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd86ffd640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 0x7efd8812f9c0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7efd7c000c00 tx=0x7efd7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c032840 con 0x7efd8813dc10 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c032a00 con 0x7efd8813dc10 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd88153950 con 0x7efd8813dc10 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c032df0 con 0x7efd8813dc10 2024-03-06T07:28:39.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.242+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd88153f10 con 0x7efd8813dc10 2024-03-06T07:28:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.243+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8804f330 con 0x7efd8813dc10 2024-03-06T07:28:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.244+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd7c04c020 con 0x7efd8813dc10 2024-03-06T07:28:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.244+0000 7efd6ffff640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd7406b500 0x7efd7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.244+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efd7c02f020 con 0x7efd8813dc10 2024-03-06T07:28:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.244+0000 7efd877fe640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd7406b500 0x7efd7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.245+0000 7efd877fe640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd7406b500 0x7efd7406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efd78007fd0 tx=0x7efd78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.249+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd8813c840 con 0x7efd8813dc10 2024-03-06T07:28:39.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.424+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efd880b2e10 con 0x7efd8813dc10 2024-03-06T07:28:39.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.424+0000 7efd6ffff640 1 -- 172.21.15.122:0/1012263056 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v122) v1 ==== 84+0+65 (secure 0 0 0) 0x7efd7c040070 con 0x7efd8813dc10 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:39.427 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd7406b500 msgr2=0x7efd7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd7406b500 0x7efd7406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efd78007fd0 tx=0x7efd78009040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 msgr2=0x7efd8812f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 0x7efd8812f9c0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7efd7c000c00 tx=0x7efd7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 shutdown_connections 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd7406b500 0x7efd7406d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd8815bf30 0x7efd88141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd8813dc10 0x7efd8812f9c0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 --2- 172.21.15.122:0/1012263056 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd880b3060 0x7efd88144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.426+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 >> 172.21.15.122:0/1012263056 conn(0x7efd880cf8a0 msgr2=0x7efd88070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.427+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 shutdown_connections 2024-03-06T07:28:39.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.427+0000 7efd8dc4e640 1 -- 172.21.15.122:0/1012263056 wait complete. 2024-03-06T07:28:39.440 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:39.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.542+0000 7f0db2179640 1 Processor -- start 2024-03-06T07:28:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.542+0000 7f0db2179640 1 -- start start 2024-03-06T07:28:39.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0db2179640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac15bf30 0x7f0dac0d2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0db2179640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0dac0cd690 0x7f0dac0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0db2179640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0dac15f990 0x7f0dac15fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0db2179640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac15c310 con 0x7f0dac15f990 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0db2179640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac0cdaf0 con 0x7f0dac0cd690 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0db2179640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0cf3c0 con 0x7f0dac15bf30 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0dabfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0dac15f990 0x7f0dac15fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0dabfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0dac15f990 0x7f0dac15fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47128/0 (socket says 172.21.15.122:47128) 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.543+0000 7f0dabfff640 1 -- 172.21.15.122:0/3810193774 learned_addr learned my addr 172.21.15.122:0/3810193774 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dab7fe640 1 --2- 172.21.15.122:0/3810193774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac15bf30 0x7f0dac0d2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0daaffd640 1 --2- 172.21.15.122:0/3810193774 >> v2:172.21.15.169:3300/0 conn(0x7f0dac0cd690 0x7f0dac0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dabfff640 1 -- 172.21.15.122:0/3810193774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac15bf30 msgr2=0x7f0dac0d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dabfff640 1 --2- 172.21.15.122:0/3810193774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac15bf30 0x7f0dac0d2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dabfff640 1 -- 172.21.15.122:0/3810193774 >> v2:172.21.15.169:3300/0 conn(0x7f0dac0cd690 msgr2=0x7f0dac0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dabfff640 1 --2- 172.21.15.122:0/3810193774 >> v2:172.21.15.169:3300/0 conn(0x7f0dac0cd690 0x7f0dac0cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dabfff640 1 -- 172.21.15.122:0/3810193774 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac0cc8d0 con 0x7f0dac15f990 2024-03-06T07:28:39.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0daaffd640 1 --2- 172.21.15.122:0/3810193774 >> v2:172.21.15.169:3300/0 conn(0x7f0dac0cd690 0x7f0dac0cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0dabfff640 1 --2- 172.21.15.122:0/3810193774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac15fe20 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f0dac067ce0 tx=0x7f0da402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198112306b9e6f55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.544+0000 7f0daa7fc640 1 -- 172.21.15.122:0/3810193774 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da402ca70 con 0x7f0dac15f990 2024-03-06T07:28:39.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0daa7fc640 1 -- 172.21.15.122:0/3810193774 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da402cc10 con 0x7f0dac15f990 2024-03-06T07:28:39.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0daa7fc640 1 -- 172.21.15.122:0/3810193774 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da40343f0 con 0x7f0dac15f990 2024-03-06T07:28:39.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 -- 172.21.15.122:0/3810193774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 msgr2=0x7f0dac15fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 --2- 172.21.15.122:0/3810193774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac15fe20 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f0dac067ce0 tx=0x7f0da402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 -- 172.21.15.122:0/3810193774 shutdown_connections 2024-03-06T07:28:39.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 --2- 172.21.15.122:0/3810193774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac15fe20 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 --2- 172.21.15.122:0/3810193774 >> v2:172.21.15.169:3300/0 conn(0x7f0dac0cd690 0x7f0dac0cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 --2- 172.21.15.122:0/3810193774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac15bf30 0x7f0dac0d2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 -- 172.21.15.122:0/3810193774 >> 172.21.15.122:0/3810193774 conn(0x7f0dac0d1b20 msgr2=0x7f0dac0d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 -- 172.21.15.122:0/3810193774 shutdown_connections 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.545+0000 7f0db2179640 1 -- 172.21.15.122:0/3810193774 wait complete. 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.546+0000 7f0db2179640 1 Processor -- start 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.546+0000 7f0db2179640 1 -- start start 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0db2179640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac0cd690 0x7f0dac0af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0db2179640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dac15bf30 0x7f0dac0afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0db2179640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac0b6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0db2179640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0cd170 con 0x7f0dac15f990 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0db2179640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0cce90 con 0x7f0dac15bf30 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0db2179640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0b1480 con 0x7f0dac0cd690 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0dabfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac0b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0dabfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac0b6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47144/0 (socket says 172.21.15.122:47144) 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.547+0000 7f0dabfff640 1 -- 172.21.15.122:0/346852441 learned_addr learned my addr 172.21.15.122:0/346852441 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dab7fe640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac0cd690 0x7f0dac0af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0daaffd640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dac15bf30 0x7f0dac0afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dabfff640 1 -- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac0cd690 msgr2=0x7f0dac0af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dabfff640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac0cd690 0x7f0dac0af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dabfff640 1 -- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dac15bf30 msgr2=0x7f0dac0afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dabfff640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dac15bf30 0x7f0dac0afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dabfff640 1 -- 172.21.15.122:0/346852441 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac06d2e0 con 0x7f0dac15f990 2024-03-06T07:28:39.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0dabfff640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac0b6750 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0da4036040 tx=0x7f0da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4044020 con 0x7f0dac15f990 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4040670 con 0x7f0dac15f990 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac06d5c0 con 0x7f0dac15f990 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4009510 con 0x7f0dac15f990 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.548+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac06db80 con 0x7f0dac15f990 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.549+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dac04f330 con 0x7f0dac15f990 2024-03-06T07:28:39.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.549+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da4047020 con 0x7f0dac15f990 2024-03-06T07:28:39.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.550+0000 7f0d93fff640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9806b4d0 0x7f0d9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.550+0000 7f0dab7fe640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9806b4d0 0x7f0d9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.550+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0da40ac950 con 0x7f0dac15f990 2024-03-06T07:28:39.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.550+0000 7f0dab7fe640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9806b4d0 0x7f0d9806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006fd0 tx=0x7f0d9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.554+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da404e030 con 0x7f0dac15f990 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:39.726+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0dac0b7aa0 con 0x7f0dac15f990 2024-03-06T07:28:40.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.166+0000 7f0d93fff640 1 -- 172.21.15.122:0/346852441 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v123) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0da407bb20 con 0x7f0dac15f990 2024-03-06T07:28:40.168 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:40.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9806b4d0 msgr2=0x7f0d9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9806b4d0 0x7f0d9806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006fd0 tx=0x7f0d9c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 msgr2=0x7f0dac0b6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac0b6750 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0da4036040 tx=0x7f0da4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 shutdown_connections 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0d9806b4d0 0x7f0d9806d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dac15f990 0x7f0dac0b6750 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0dac15bf30 0x7f0dac0afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 --2- 172.21.15.122:0/346852441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0dac0cd690 0x7f0dac0af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 >> 172.21.15.122:0/346852441 conn(0x7f0dac0d1b20 msgr2=0x7f0dac06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 shutdown_connections 2024-03-06T07:28:40.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.168+0000 7f0db2179640 1 -- 172.21.15.122:0/346852441 wait complete. 2024-03-06T07:28:40.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:40.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.291+0000 7f5cba0c9640 1 Processor -- start 2024-03-06T07:28:40.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.292+0000 7f5cba0c9640 1 -- start start 2024-03-06T07:28:40.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.292+0000 7f5cba0c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5cb40d4860 0x7f5cb40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.292+0000 7f5cba0c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 0x7f5cb40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.292+0000 7f5cba0c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb41618b0 con 0x7f5cb41614d0 2024-03-06T07:28:40.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.292+0000 7f5cba0c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb40d4cf0 con 0x7f5cb40d4860 2024-03-06T07:28:40.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.292+0000 7f5cba0c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40cf400 con 0x7f5cb40b0e30 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb2ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5cb40d4860 0x7f5cb40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb3fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 0x7f5cb40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb3fff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 0x7f5cb40b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:41444/0 (socket says 172.21.15.122:41444) 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb2ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f5cb40d4860 0x7f5cb40d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:57718/0 (socket says 172.21.15.122:57718) 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb3fff640 1 -- 172.21.15.122:0/1818097546 learned_addr learned my addr 172.21.15.122:0/1818097546 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb2ffd640 1 -- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 msgr2=0x7f5cb40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb2ffd640 1 --2- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 0x7f5cb40b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb2ffd640 1 -- 172.21.15.122:0/1818097546 >> v1:172.21.15.169:6789/0 conn(0x7f5cb41614d0 legacy=0x7f5cb40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb2ffd640 1 -- 172.21.15.122:0/1818097546 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb40cf6a0 con 0x7f5cb40d4860 2024-03-06T07:28:40.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.293+0000 7f5cb3fff640 1 --2- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 0x7f5cb40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.294+0000 7f5cb2ffd640 1 --2- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb40d2020 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80097a0 tx=0x7f5ca802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a84c9c49da27ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.294+0000 7f5cb27fc640 1 -- 172.21.15.122:0/1818097546 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8035020 con 0x7f5cb40d4860 2024-03-06T07:28:40.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.294+0000 7f5cb27fc640 1 -- 172.21.15.122:0/1818097546 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca8031830 con 0x7f5cb40d4860 2024-03-06T07:28:40.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.294+0000 7f5cb27fc640 1 -- 172.21.15.122:0/1818097546 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8031c00 con 0x7f5cb40d4860 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 -- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 msgr2=0x7f5cb40d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb40d2020 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80097a0 tx=0x7f5ca802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 -- 172.21.15.122:0/1818097546 shutdown_connections 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb40b0e30 0x7f5cb40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/1818097546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb40d2020 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 -- 172.21.15.122:0/1818097546 >> 172.21.15.122:0/1818097546 conn(0x7f5cb40d3d40 msgr2=0x7f5cb40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 -- 172.21.15.122:0/1818097546 shutdown_connections 2024-03-06T07:28:40.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.295+0000 7f5cba0c9640 1 -- 172.21.15.122:0/1818097546 wait complete. 2024-03-06T07:28:40.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.296+0000 7f5cba0c9640 1 Processor -- start 2024-03-06T07:28:40.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.296+0000 7f5cba0c9640 1 -- start start 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 0x7f5cb40b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb414f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb41614d0 0x7f5cb40c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40b0910 con 0x7f5cb40b0e30 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40cfc60 con 0x7f5cb40d4860 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40d2560 con 0x7f5cb41614d0 2024-03-06T07:28:40.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 0x7f5cb40b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 0x7f5cb40b9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47154/0 (socket says 172.21.15.122:47154) 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 -- 172.21.15.122:0/821168202 learned_addr learned my addr 172.21.15.122:0/821168202 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb3fff640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb41614d0 0x7f5cb40c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb2ffd640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb414f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 -- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb41614d0 msgr2=0x7f5cb40c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb41614d0 0x7f5cb40c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 -- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 msgr2=0x7f5cb414f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb414f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 -- 172.21.15.122:0/821168202 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb40b7ee0 con 0x7f5cb40b0e30 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cb37fe640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 0x7f5cb40b9220 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c3c0 tx=0x7f5ca400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca40052d0 con 0x7f5cb40b0e30 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb40b8180 con 0x7f5cb40b0e30 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca4005490 con 0x7f5cb40b0e30 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4007260 con 0x7f5cb40b0e30 2024-03-06T07:28:40.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.297+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb40b8740 con 0x7f5cb40b0e30 2024-03-06T07:28:40.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.299+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb4068ff0 con 0x7f5cb40b0e30 2024-03-06T07:28:40.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.299+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ca4007400 con 0x7f5cb40b0e30 2024-03-06T07:28:40.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.299+0000 7f5c9bfff640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ca006b500 0x7f5ca006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.300+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5ca4089510 con 0x7f5cb40b0e30 2024-03-06T07:28:40.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.300+0000 7f5cb2ffd640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ca006b500 0x7f5ca006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.300+0000 7f5cb2ffd640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ca006b500 0x7f5ca006d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8036000 tx=0x7f5ca8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.304+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ca40584f0 con 0x7f5cb40b0e30 2024-03-06T07:28:40.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:40.478+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5cb40b48c0 con 0x7f5cb40b0e30 2024-03-06T07:28:41.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.168+0000 7f5c9bfff640 1 -- 172.21.15.122:0/821168202 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v124) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5ca4057df0 con 0x7f5cb40b0e30 2024-03-06T07:28:41.170 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ca006b500 msgr2=0x7f5ca006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ca006b500 0x7f5ca006d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8036000 tx=0x7f5ca8008030 comp rx=0 tx=0).stop 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 msgr2=0x7f5cb40b9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 0x7f5cb40b9220 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c3c0 tx=0x7f5ca400c880 comp rx=0 tx=0).stop 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 shutdown_connections 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5ca006b500 0x7f5ca006d9c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5cb41614d0 0x7f5cb40c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cb40d4860 0x7f5cb414f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 --2- 172.21.15.122:0/821168202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cb40b0e30 0x7f5cb40b9220 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 >> 172.21.15.122:0/821168202 conn(0x7f5cb40d3d40 msgr2=0x7f5cb40cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 shutdown_connections 2024-03-06T07:28:41.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.171+0000 7f5cba0c9640 1 -- 172.21.15.122:0/821168202 wait complete. 2024-03-06T07:28:41.192 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:41.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.294+0000 7f2c6c7eb640 1 Processor -- start 2024-03-06T07:28:41.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.294+0000 7f2c6c7eb640 1 -- start start 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6c7eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2c64158800 0x7f2c6415b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6c7eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2c640b3cd0 0x7f2c6415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6c7eb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6415bdf0 con 0x7f2c64158420 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6c7eb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6415e650 con 0x7f2c640b3cd0 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6c7eb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6415e8a0 con 0x7f2c64158800 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6a587640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2c64158420 0x7f2c640b35c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58326/0 (socket says 172.21.15.122:58326) 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6ad88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2c640b3cd0 0x7f2c6415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6a587640 1 -- 172.21.15.122:0/1791455171 learned_addr learned my addr 172.21.15.122:0/1791455171 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:41.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c69d86640 1 --2- 172.21.15.122:0/1791455171 >> v2:172.21.15.169:3300/0 conn(0x7f2c64158800 0x7f2c6415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6ad88640 1 -- 172.21.15.122:0/1791455171 >> v2:172.21.15.169:3300/0 conn(0x7f2c64158800 msgr2=0x7f2c6415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6ad88640 1 --2- 172.21.15.122:0/1791455171 >> v2:172.21.15.169:3300/0 conn(0x7f2c64158800 0x7f2c6415b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6ad88640 1 -- 172.21.15.122:0/1791455171 >> v1:172.21.15.122:6789/0 conn(0x7f2c64158420 legacy=0x7f2c640b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.295+0000 7f2c6ad88640 1 -- 172.21.15.122:0/1791455171 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6415eb80 con 0x7f2c640b3cd0 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.296+0000 7f2c6ad88640 1 --2- 172.21.15.122:0/1791455171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c6415b8b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f2c640cc690 tx=0x7f2c5c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992da25b2148299e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.296+0000 7f2c69585640 1 -- 172.21.15.122:0/1791455171 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0315a0 con 0x7f2c640b3cd0 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.296+0000 7f2c69585640 1 -- 172.21.15.122:0/1791455171 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c037070 con 0x7f2c640b3cd0 2024-03-06T07:28:41.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.296+0000 7f2c69585640 1 -- 172.21.15.122:0/1791455171 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c031960 con 0x7f2c640b3cd0 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/1791455171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 msgr2=0x7f2c6415b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/1791455171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c6415b8b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f2c640cc690 tx=0x7f2c5c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/1791455171 shutdown_connections 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/1791455171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c6415b8b0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/1791455171 >> v2:172.21.15.169:3300/0 conn(0x7f2c64158800 0x7f2c6415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/1791455171 >> 172.21.15.122:0/1791455171 conn(0x7f2c6406afb0 msgr2=0x7f2c6406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/1791455171 shutdown_connections 2024-03-06T07:28:41.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.297+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/1791455171 wait complete. 2024-03-06T07:28:41.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.298+0000 7f2c6c7eb640 1 Processor -- start 2024-03-06T07:28:41.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.298+0000 7f2c6c7eb640 1 -- start start 2024-03-06T07:28:41.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c641e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c64158420 0x7f2c6414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c64156b90 0x7f2c64157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64071a20 con 0x7f2c640b3cd0 2024-03-06T07:28:41.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6415f140 con 0x7f2c64158420 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64071ba0 con 0x7f2c64156b90 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c641e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c641e4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47174/0 (socket says 172.21.15.122:47174) 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 -- 172.21.15.122:0/221059474 learned_addr learned my addr 172.21.15.122:0/221059474 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 -- 172.21.15.122:0/221059474 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c64156b90 msgr2=0x7f2c64157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c64156b90 0x7f2c64157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 -- 172.21.15.122:0/221059474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c64158420 msgr2=0x7f2c6414dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c64158420 0x7f2c6414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 -- 172.21.15.122:0/221059474 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c641e6180 con 0x7f2c640b3cd0 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6a587640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c641e4e80 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000ede0 tx=0x7f2c6000c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60015070 con 0x7f2c640b3cd0 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c60011670 con 0x7f2c640b3cd0 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c641f0b20 con 0x7f2c640b3cd0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.299+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60011a80 con 0x7f2c640b3cd0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.300+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c641f10e0 con 0x7f2c640b3cd0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.301+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6404f330 con 0x7f2c640b3cd0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.301+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c60018070 con 0x7f2c640b3cd0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.301+0000 7f2c5affd640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4806b4d0 0x7f2c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.301+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2c6008cf00 con 0x7f2c640b3cd0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.301+0000 7f2c69d86640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4806b4d0 0x7f2c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.302+0000 7f2c69d86640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4806b4d0 0x7f2c4806d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2c54006fd0 tx=0x7f2c54008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.306+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c6005bde0 con 0x7f2c640b3cd0 2024-03-06T07:28:41.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:41.514+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2c640b2a30 con 0x7f2c640b3cd0 2024-03-06T07:28:42.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.197+0000 7f2c5affd640 1 -- 172.21.15.122:0/221059474 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2c6005b6e0 con 0x7f2c640b3cd0 2024-03-06T07:28:42.199 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:28:42.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.204+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4806b4d0 msgr2=0x7f2c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.204+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4806b4d0 0x7f2c4806d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2c54006fd0 tx=0x7f2c54008040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.204+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 msgr2=0x7f2c641e4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.204+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c641e4e80 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000ede0 tx=0x7f2c6000c640 comp rx=0 tx=0).stop 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.205+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 shutdown_connections 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.205+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2c4806b4d0 0x7f2c4806d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.205+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2c64156b90 0x7f2c64157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.205+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c64158420 0x7f2c6414dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.205+0000 7f2c6c7eb640 1 --2- 172.21.15.122:0/221059474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c640b3cd0 0x7f2c641e4e80 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.205+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 >> 172.21.15.122:0/221059474 conn(0x7f2c6406afb0 msgr2=0x7f2c64158bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.206+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 shutdown_connections 2024-03-06T07:28:42.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.207+0000 7f2c6c7eb640 1 -- 172.21.15.122:0/221059474 wait complete. 2024-03-06T07:28:42.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.325+0000 7f80242e0640 1 Processor -- start 2024-03-06T07:28:42.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.325+0000 7f80242e0640 1 -- start start 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.326+0000 7f80242e0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f801c13dc10 0x7f801c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.326+0000 7f80242e0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f801c15bf30 0x7f801c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.326+0000 7f80242e0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c15f580 con 0x7f801c15e850 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.326+0000 7f80242e0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c0cf3f0 con 0x7f801c13dc10 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.326+0000 7f80242e0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c15b210 con 0x7f801c15bf30 2024-03-06T07:28:42.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802207c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f801c13dc10 0x7f801c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802287d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f801c15e850 0x7f801c15ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58336/0 (socket says 172.21.15.122:58336) 2024-03-06T07:28:42.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802287d640 1 -- 172.21.15.122:0/1416488184 learned_addr learned my addr 172.21.15.122:0/1416488184 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:42.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802207c640 1 -- 172.21.15.122:0/1416488184 >> v2:172.21.15.169:3300/0 conn(0x7f801c15bf30 msgr2=0x7f801c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802207c640 1 --2- 172.21.15.122:0/1416488184 >> v2:172.21.15.169:3300/0 conn(0x7f801c15bf30 0x7f801c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802207c640 1 -- 172.21.15.122:0/1416488184 >> v1:172.21.15.122:6789/0 conn(0x7f801c15e850 legacy=0x7f801c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802107a640 1 -- 172.21.15.122:0/1416488184 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116723532 0 0) 0x7f801c15f580 con 0x7f801c15e850 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.327+0000 7f802207c640 1 -- 172.21.15.122:0/1416488184 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c15b450 con 0x7f801c13dc10 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f802207c640 1 --2- 172.21.15.122:0/1416488184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c13dc10 0x7f801c0b2950 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f801c06c790 tx=0x7f800c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd057930109bfaaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f802107a640 1 -- 172.21.15.122:0/1416488184 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c039070 con 0x7f801c13dc10 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f802107a640 1 -- 172.21.15.122:0/1416488184 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800c02cbf0 con 0x7f801c13dc10 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f802107a640 1 -- 172.21.15.122:0/1416488184 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c034440 con 0x7f801c13dc10 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f80242e0640 1 -- 172.21.15.122:0/1416488184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c13dc10 msgr2=0x7f801c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f80242e0640 1 --2- 172.21.15.122:0/1416488184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c13dc10 0x7f801c0b2950 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f801c06c790 tx=0x7f800c02be40 comp rx=0 tx=0).stop 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f80242e0640 1 -- 172.21.15.122:0/1416488184 shutdown_connections 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f80242e0640 1 --2- 172.21.15.122:0/1416488184 >> v2:172.21.15.169:3300/0 conn(0x7f801c15bf30 0x7f801c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f80242e0640 1 --2- 172.21.15.122:0/1416488184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c13dc10 0x7f801c0b2950 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.328+0000 7f80242e0640 1 -- 172.21.15.122:0/1416488184 >> 172.21.15.122:0/1416488184 conn(0x7f801c0cf8a0 msgr2=0x7f801c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.329+0000 7f80242e0640 1 -- 172.21.15.122:0/1416488184 shutdown_connections 2024-03-06T07:28:42.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.329+0000 7f80242e0640 1 -- 172.21.15.122:0/1416488184 wait complete. 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 Processor -- start 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 -- start start 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801c13dc10 0x7f801c141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 0x7f801c13fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801c15e850 0x7f801c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0708d0 con 0x7f801c15bf30 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c13dff0 con 0x7f801c15e850 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f80242e0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c070a50 con 0x7f801c13dc10 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f802187b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 0x7f801c13fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f802187b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 0x7f801c13fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47204/0 (socket says 172.21.15.122:47204) 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.330+0000 7f802187b640 1 -- 172.21.15.122:0/216303775 learned_addr learned my addr 172.21.15.122:0/216303775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802207c640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801c13dc10 0x7f801c141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802287d640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801c15e850 0x7f801c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802187b640 1 -- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801c13dc10 msgr2=0x7f801c141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802187b640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801c13dc10 0x7f801c141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802187b640 1 -- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801c15e850 msgr2=0x7f801c1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802187b640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801c15e850 0x7f801c1547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802187b640 1 -- 172.21.15.122:0/216303775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c142d00 con 0x7f801c15bf30 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802207c640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801c13dc10 0x7f801c141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f802187b640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 0x7f801c13fe20 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f801000b4a0 tx=0x7f801000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010014070 con 0x7f801c15bf30 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010010610 con 0x7f801c15bf30 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010010a00 con 0x7f801c15bf30 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c154ce0 con 0x7f801c15bf30 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.331+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c141f40 con 0x7f801c15bf30 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.332+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c04f330 con 0x7f801c15bf30 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.333+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8010017070 con 0x7f801c15bf30 2024-03-06T07:28:42.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.333+0000 7f801affd640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b500 0x7f800806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.333+0000 7f802207c640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b500 0x7f800806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.333+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8010089be0 con 0x7f801c15bf30 2024-03-06T07:28:42.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.333+0000 7f802207c640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b500 0x7f800806d9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f800c036000 tx=0x7f800c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.337+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8010058a60 con 0x7f801c15bf30 2024-03-06T07:28:42.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.520+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f801c15ad30 con 0x7f801c15bf30 2024-03-06T07:28:42.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.521+0000 7f801affd640 1 -- 172.21.15.122:0/216303775 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16692 (secure 0 0 0) 0x7f8010058360 con 0x7f801c15bf30 2024-03-06T07:28:42.524 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:42.524 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":125,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:28:41.515609+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b500 msgr2=0x7f800806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b500 0x7f800806d9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f800c036000 tx=0x7f800c030000 comp rx=0 tx=0).stop 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 msgr2=0x7f801c13fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 0x7f801c13fe20 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f801000b4a0 tx=0x7f801000b960 comp rx=0 tx=0).stop 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 shutdown_connections 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b500 0x7f800806d9c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f801c15e850 0x7f801c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f801c15bf30 0x7f801c13fe20 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 --2- 172.21.15.122:0/216303775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f801c13dc10 0x7f801c141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 >> 172.21.15.122:0/216303775 conn(0x7f801c0cf8a0 msgr2=0x7f801c0afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.523+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 shutdown_connections 2024-03-06T07:28:42.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.524+0000 7f80242e0640 1 -- 172.21.15.122:0/216303775 wait complete. 2024-03-06T07:28:42.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.635+0000 7fc51a3a2640 1 Processor -- start 2024-03-06T07:28:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.635+0000 7fc51a3a2640 1 -- start start 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc51a3a2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc5140d1210 0x7fc51406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc51a3a2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51406d090 con 0x7fc5140d1e70 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc51a3a2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51406d370 con 0x7fc51415de40 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc51a3a2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5140cfab0 con 0x7fc5140d1210 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc518b9f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc5140d1e70 0x7fc51406c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58346/0 (socket says 172.21.15.122:58346) 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc518b9f640 1 -- 172.21.15.122:0/3636655953 learned_addr learned my addr 172.21.15.122:0/3636655953 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474500762 0 0) 0x7fc51406d090 con 0x7fc5140d1e70 2024-03-06T07:28:42.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc500003520 con 0x7fc5140d1e70 2024-03-06T07:28:42.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3636655953 >> v2:172.21.15.169:3300/0 conn(0x7fc5140d1210 0x7fc51406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813817748 0 0) 0x7fc500003520 con 0x7fc5140d1e70 2024-03-06T07:28:42.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 >> v2:172.21.15.169:3300/0 conn(0x7fc5140d1210 msgr2=0x7fc51406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 --2- 172.21.15.122:0/3636655953 >> v2:172.21.15.169:3300/0 conn(0x7fc5140d1210 0x7fc51406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 >> v1:172.21.15.169:6789/0 conn(0x7fc51415de40 legacy=0x7fc5140d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51406d650 con 0x7fc5140d1e70 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fc504001670 con 0x7fc5140d1e70 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 >> v1:172.21.15.122:6789/0 conn(0x7fc5140d1e70 legacy=0x7fc51406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.636+0000 7fc50bfff640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc500003a20 0x7fc500003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc50bfff640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc50bfff640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc500005230 0x7fc50000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000c160 con 0x7fc500005230 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000c310 con 0x7fc500003a20 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000c510 con 0x7fc500004800 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc500005230 0x7fc50000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc5193a0640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc500003a20 0x7fc500003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc518b9f640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc519ba1640 1 -- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500004800 msgr2=0x7fc500004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc519ba1640 1 -- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc500003a20 msgr2=0x7fc500003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc500003a20 0x7fc500003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.637+0000 7fc519ba1640 1 -- 172.21.15.122:0/3636655953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc500003520 con 0x7fc500005230 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc519ba1640 1 -- 172.21.15.122:0/3636655953 reap_dead start 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc500005230 0x7fc50000b9c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009fc0 tx=0x7fc500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3490f457606a61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c009c10 con 0x7fc500005230 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c003bd0 con 0x7fc500005230 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc50bfff640 1 -- 172.21.15.122:0/3636655953 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c02c990 con 0x7fc500005230 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3636655953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc500005230 msgr2=0x7fc50000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc500005230 0x7fc50000b9c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009fc0 tx=0x7fc500005830 comp rx=0 tx=0).stop 2024-03-06T07:28:42.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3636655953 shutdown_connections 2024-03-06T07:28:42.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3636655953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc500005230 0x7fc50000b9c0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3636655953 >> 172.21.15.122:0/3636655953 conn(0x7fc51409a9f0 msgr2=0x7fc51409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3636655953 shutdown_connections 2024-03-06T07:28:42.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.638+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3636655953 wait complete. 2024-03-06T07:28:42.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 Processor -- start 2024-03-06T07:28:42.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 -- start start 2024-03-06T07:28:42.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500005230 0x7fc5140c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 0x7fc5140cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5140cd550 0x7fc5140cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140d1760 con 0x7fc5140b3cb0 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51415e220 con 0x7fc5140cd550 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc51a3a2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140d18e0 con 0x7fc500005230 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc518b9f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 0x7fc5140cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc518b9f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 0x7fc5140cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47232/0 (socket says 172.21.15.122:47232) 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc518b9f640 1 -- 172.21.15.122:0/3795445572 learned_addr learned my addr 172.21.15.122:0/3795445572 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.640+0000 7fc5193a0640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500005230 0x7fc5140c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5140cd550 0x7fc5140cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc518b9f640 1 -- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500005230 msgr2=0x7fc5140c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc518b9f640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500005230 0x7fc5140c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc518b9f640 1 -- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5140cd550 msgr2=0x7fc5140cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc518b9f640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5140cd550 0x7fc5140cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc518b9f640 1 -- 172.21.15.122:0/3795445572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140cb4a0 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc5193a0640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500005230 0x7fc5140c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc519ba1640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5140cd550 0x7fc5140cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc518b9f640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 0x7fc5140cd010 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc504000ff0 tx=0x7fc5040074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50400f7b0 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc504018070 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5140cb780 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50400fbb0 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.641+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5140cc060 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.642+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc514068f80 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.643+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc50400fd50 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.643+0000 7fc509ffb640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f806b500 0x7fc4f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.643+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc50408a180 con 0x7fc5140b3cb0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.643+0000 7fc5193a0640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f806b500 0x7fc4f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.643+0000 7fc5193a0640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f806b500 0x7fc4f806d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc510000c00 tx=0x7fc510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.647+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc504059000 con 0x7fc5140b3cb0 2024-03-06T07:28:42.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.819+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5140b77f0 con 0x7fc5140b3cb0 2024-03-06T07:28:42.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.821+0000 7fc509ffb640 1 -- 172.21.15.122:0/3795445572 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16692 (secure 0 0 0) 0x7fc504058900 con 0x7fc5140b3cb0 2024-03-06T07:28:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:42.823 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":125,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:28:41.515609+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f806b500 msgr2=0x7fc4f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f806b500 0x7fc4f806d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc510000c00 tx=0x7fc510009040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 msgr2=0x7fc5140cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 0x7fc5140cd010 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc504000ff0 tx=0x7fc5040074f0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 shutdown_connections 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc4f806b500 0x7fc4f806d9c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5140cd550 0x7fc5140cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5140b3cb0 0x7fc5140cd010 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 --2- 172.21.15.122:0/3795445572 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc500005230 0x7fc5140c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 >> 172.21.15.122:0/3795445572 conn(0x7fc51409a9f0 msgr2=0x7fc51415e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 shutdown_connections 2024-03-06T07:28:42.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.823+0000 7fc51a3a2640 1 -- 172.21.15.122:0/3795445572 wait complete. 2024-03-06T07:28:42.836 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:42.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.936+0000 7f9d9a0b5640 1 Processor -- start 2024-03-06T07:28:42.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.937+0000 7f9d9a0b5640 1 -- start start 2024-03-06T07:28:42.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.937+0000 7f9d9a0b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d941586e0 0x7f9d940aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.937+0000 7f9d9a0b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d94158ac0 con 0x7f9d940aef80 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.937+0000 7f9d9a0b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9415b2a0 con 0x7f9d940d01a0 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.937+0000 7f9d9a0b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9415b540 con 0x7f9d941586e0 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.937+0000 7f9d92ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9d940aef80 0x7f9d940cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58348/0 (socket says 172.21.15.122:58348) 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d92ffd640 1 -- 172.21.15.122:0/2032431126 learned_addr learned my addr 172.21.15.122:0/2032431126 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d937fe640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d941586e0 0x7f9d940aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514815130 0 0) 0x7f9d94158ac0 con 0x7f9d940aef80 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d7c003520 con 0x7f9d940aef80 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393633073 0 0) 0x7f9d7c003520 con 0x7f9d940aef80 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d941586e0 msgr2=0x7f9d940aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d941586e0 0x7f9d940aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 >> v1:172.21.15.169:6789/0 conn(0x7f9d940d01a0 legacy=0x7f9d940b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9415b820 con 0x7f9d940aef80 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f9d88001670 con 0x7f9d940aef80 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 >> v1:172.21.15.122:6789/0 conn(0x7f9d940aef80 legacy=0x7f9d940cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d7c0039e0 0x7f9d7c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c004390 0x7f9d7c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d7c005060 0x7f9d7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9415b540 con 0x7f9d7c0039e0 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c00bf30 con 0x7f9d7c004390 2024-03-06T07:28:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c00e340 con 0x7f9d7c005060 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d937fe640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d941586e0 0x7f9d940aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d937fe640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d7c0039e0 0x7f9d7c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d92ffd640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c004390 0x7f9d7c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.938+0000 7f9d93fff640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d7c005060 0x7f9d7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d937fe640 1 -- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d7c005060 msgr2=0x7f9d7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d937fe640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d7c005060 0x7f9d7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d937fe640 1 -- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c004390 msgr2=0x7f9d7c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d937fe640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c004390 0x7f9d7c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d937fe640 1 -- 172.21.15.122:0/2032431126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c003520 con 0x7f9d7c0039e0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d937fe640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d7c0039e0 0x7f9d7c003dc0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9d940b2600 tx=0x7f9d7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba15a560dde7cda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840097d0 con 0x7f9d7c0039e0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d84004030 con 0x7f9d7c0039e0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.939+0000 7f9d927fc640 1 -- 172.21.15.122:0/2032431126 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84004400 con 0x7f9d7c0039e0 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/2032431126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d7c0039e0 msgr2=0x7f9d7c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/2032431126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d7c0039e0 0x7f9d7c003dc0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9d940b2600 tx=0x7f9d7c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:42.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d93fff640 1 -- 172.21.15.122:0/2032431126 reap_dead start 2024-03-06T07:28:42.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/2032431126 shutdown_connections 2024-03-06T07:28:42.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/2032431126 >> 172.21.15.122:0/2032431126 conn(0x7f9d9406b190 msgr2=0x7f9d9406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/2032431126 shutdown_connections 2024-03-06T07:28:42.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.940+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/2032431126 wait complete. 2024-03-06T07:28:42.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.941+0000 7f9d9a0b5640 1 Processor -- start 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.941+0000 7f9d9a0b5640 1 -- start start 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d9a0b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c0039e0 0x7f9d94153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d9a0b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940a9450 0x7f9d9406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d9a0b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 0x7f9d94157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d9a0b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d94071350 con 0x7f9d941ef600 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d9a0b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9415bde0 con 0x7f9d7c0039e0 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d9a0b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940714d0 con 0x7f9d940a9450 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 0x7f9d94157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 0x7f9d94157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47250/0 (socket says 172.21.15.122:47250) 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 -- 172.21.15.122:0/1097808511 learned_addr learned my addr 172.21.15.122:0/1097808511 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d92ffd640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940a9450 0x7f9d9406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d937fe640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c0039e0 0x7f9d94153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 -- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940a9450 msgr2=0x7f9d9406dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940a9450 0x7f9d9406dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 -- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c0039e0 msgr2=0x7f9d94153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.942+0000 7f9d93fff640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c0039e0 0x7f9d94153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d93fff640 1 -- 172.21.15.122:0/1097808511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940ad290 con 0x7f9d941ef600 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d937fe640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c0039e0 0x7f9d94153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d92ffd640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940a9450 0x7f9d9406dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d93fff640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 0x7f9d94157390 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9d9406bf00 tx=0x7f9d8c00ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c003b80 con 0x7f9d941ef600 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c018070 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d941efb10 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00d770 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.943+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d941f00d0 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.944+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d940732b0 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.944+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d8c00d910 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.945+0000 7f9d7bfff640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d8006b500 0x7f9d8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.945+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d8c08a1a0 con 0x7f9d941ef600 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.945+0000 7f9d937fe640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d8006b500 0x7f9d8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.945+0000 7f9d937fe640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d8006b500 0x7f9d8006d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9d84035000 tx=0x7f9d84030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:42.949+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d940ad290 con 0x7f9d941ef600 2024-03-06T07:28:43.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.157+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9d940cfb60 con 0x7f9d941ef600 2024-03-06T07:28:43.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.198+0000 7f9d7bfff640 1 -- 172.21.15.122:0/1097808511 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9d8c0590b0 con 0x7f9d941ef600 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.200+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d8006b500 msgr2=0x7f9d8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d8006b500 0x7f9d8006d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9d84035000 tx=0x7f9d84030000 comp rx=0 tx=0).stop 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 msgr2=0x7f9d94157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 0x7f9d94157390 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9d9406bf00 tx=0x7f9d8c00ff70 comp rx=0 tx=0).stop 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 shutdown_connections 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9d8006b500 0x7f9d8006d9c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d941ef600 0x7f9d94157390 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d940a9450 0x7f9d9406dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 --2- 172.21.15.122:0/1097808511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d7c0039e0 0x7f9d94153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 >> 172.21.15.122:0/1097808511 conn(0x7f9d9406b190 msgr2=0x7f9d94158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 shutdown_connections 2024-03-06T07:28:43.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.201+0000 7f9d9a0b5640 1 -- 172.21.15.122:0/1097808511 wait complete. 2024-03-06T07:28:43.214 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:43.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:43.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.317+0000 7f4a0560e640 1 Processor -- start 2024-03-06T07:28:43.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.317+0000 7f4a0560e640 1 -- start start 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f4a0560e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a000d06e0 0x7f4a0015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f4a0560e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4a000cf200 0x7f4a000cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f4a0560e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000d47f0 con 0x7f4a000d0ac0 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f4a0560e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a000ced70 con 0x7f4a000d06e0 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f4a0560e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a000d1ab0 con 0x7f4a000cf200 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49feffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a000d06e0 0x7f4a0015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49f7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4a000d0ac0 0x7f4a000b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58360/0 (socket says 172.21.15.122:58360) 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49f7fff640 1 -- 172.21.15.122:0/2983461401 learned_addr learned my addr 172.21.15.122:0/2983461401 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49ff7fe640 1 --2- 172.21.15.122:0/2983461401 >> v2:172.21.15.169:3300/0 conn(0x7f4a000cf200 0x7f4a000cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49feffd640 1 -- 172.21.15.122:0/2983461401 >> v2:172.21.15.169:3300/0 conn(0x7f4a000cf200 msgr2=0x7f4a000cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49feffd640 1 --2- 172.21.15.122:0/2983461401 >> v2:172.21.15.169:3300/0 conn(0x7f4a000cf200 0x7f4a000cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49feffd640 1 -- 172.21.15.122:0/2983461401 >> v1:172.21.15.122:6789/0 conn(0x7f4a000d0ac0 legacy=0x7f4a000b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49feffd640 1 -- 172.21.15.122:0/2983461401 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000d1d90 con 0x7f4a000d06e0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.318+0000 7f49fe7fc640 1 -- 172.21.15.122:0/2983461401 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428686170 0 0) 0x7f4a000d47f0 con 0x7f4a000d0ac0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.319+0000 7f49feffd640 1 --2- 172.21.15.122:0/2983461401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d06e0 0x7f4a0015fc30 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f4a0008d160 tx=0x7f49f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66460a5f67ca072 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.319+0000 7f49fe7fc640 1 -- 172.21.15.122:0/2983461401 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0035020 con 0x7f4a000d06e0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.319+0000 7f49fe7fc640 1 -- 172.21.15.122:0/2983461401 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f0031830 con 0x7f4a000d06e0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.319+0000 7f49fe7fc640 1 -- 172.21.15.122:0/2983461401 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0031c00 con 0x7f4a000d06e0 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 -- 172.21.15.122:0/2983461401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d06e0 msgr2=0x7f4a0015fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 --2- 172.21.15.122:0/2983461401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d06e0 0x7f4a0015fc30 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f4a0008d160 tx=0x7f49f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 -- 172.21.15.122:0/2983461401 shutdown_connections 2024-03-06T07:28:43.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 --2- 172.21.15.122:0/2983461401 >> v2:172.21.15.169:3300/0 conn(0x7f4a000cf200 0x7f4a000cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 --2- 172.21.15.122:0/2983461401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d06e0 0x7f4a0015fc30 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 -- 172.21.15.122:0/2983461401 >> 172.21.15.122:0/2983461401 conn(0x7f4a000d3cd0 msgr2=0x7f4a000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 -- 172.21.15.122:0/2983461401 shutdown_connections 2024-03-06T07:28:43.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.320+0000 7f4a0560e640 1 -- 172.21.15.122:0/2983461401 wait complete. 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.321+0000 7f4a0560e640 1 Processor -- start 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.321+0000 7f4a0560e640 1 -- start start 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a000cf200 0x7f4a000b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a000d06e0 0x7f4a000b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 0x7f4a00155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a000b44d0 con 0x7f4a000d0ac0 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a000ce990 con 0x7f4a000cf200 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a000d0f30 con 0x7f4a000d06e0 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 0x7f4a00155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 0x7f4a00155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47268/0 (socket says 172.21.15.122:47268) 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 -- 172.21.15.122:0/1452064852 learned_addr learned my addr 172.21.15.122:0/1452064852 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 -- 172.21.15.122:0/1452064852 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a000d06e0 msgr2=0x7f4a000b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a000d06e0 0x7f4a000b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 -- 172.21.15.122:0/1452064852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a000cf200 msgr2=0x7f4a000b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a000cf200 0x7f4a000b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 -- 172.21.15.122:0/1452064852 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000b2910 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49ff7fe640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 0x7f4a00155310 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007e90 tx=0x7f49f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8014420 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f8019070 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8014800 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a000b0900 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.322+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a000b0ec0 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.323+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a000a1c80 con 0x7f4a000d0ac0 2024-03-06T07:28:43.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.324+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49f80149a0 con 0x7f4a000d0ac0 2024-03-06T07:28:43.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.324+0000 7f49f77fe640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e006b4d0 0x7f49e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.324+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49f808aa30 con 0x7f4a000d0ac0 2024-03-06T07:28:43.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.324+0000 7f49feffd640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e006b4d0 0x7f49e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.325+0000 7f49feffd640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e006b4d0 0x7f49e006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f49f0036000 tx=0x7f49f0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.329+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49f80598b0 con 0x7f4a000d0ac0 2024-03-06T07:28:43.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:43.539+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4a00068ff0 con 0x7f4a000d0ac0 2024-03-06T07:28:44.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.206+0000 7f49f77fe640 1 -- 172.21.15.122:0/1452064852 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f49f80591b0 con 0x7f4a000d0ac0 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e006b4d0 msgr2=0x7f49e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e006b4d0 0x7f49e006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f49f0036000 tx=0x7f49f0030000 comp rx=0 tx=0).stop 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 msgr2=0x7f4a00155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 0x7f4a00155310 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007e90 tx=0x7f49f800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 shutdown_connections 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49e006b4d0 0x7f49e006d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a000d0ac0 0x7f4a00155310 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a000d06e0 0x7f4a000b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 --2- 172.21.15.122:0/1452064852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a000cf200 0x7f4a000b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 >> 172.21.15.122:0/1452064852 conn(0x7f4a000d3cd0 msgr2=0x7f4a0006c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 shutdown_connections 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.209+0000 7f4a0560e640 1 -- 172.21.15.122:0/1452064852 wait complete. 2024-03-06T07:28:44.222 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:44.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.321+0000 7fc0ba555640 1 Processor -- start 2024-03-06T07:28:44.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.322+0000 7fc0ba555640 1 -- start start 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.322+0000 7fc0ba555640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 0x7fc0b40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.322+0000 7fc0ba555640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 0x7fc0b40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.322+0000 7fc0ba555640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40b3430 con 0x7fc0b415dfe0 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.322+0000 7fc0ba555640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b40d2c50 con 0x7fc0b40d22b0 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.322+0000 7fc0ba555640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40b31e0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 0x7fc0b40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b37fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 0x7fc0b40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b37fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 0x7fc0b40d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:57820/0 (socket says 172.21.15.122:57820) 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 0x7fc0b40b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:41528/0 (socket says 172.21.15.122:41528) 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b3fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc0b415dfe0 0x7fc0b40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:47186/0 (socket says 172.21.15.122:47186) 2024-03-06T07:28:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 -- 172.21.15.122:0/2697907111 learned_addr learned my addr 172.21.15.122:0/2697907111 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 -- 172.21.15.122:0/2697907111 >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 msgr2=0x7fc0b40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 --2- 172.21.15.122:0/2697907111 >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 0x7fc0b40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 -- 172.21.15.122:0/2697907111 >> v1:172.21.15.169:6789/0 conn(0x7fc0b415dfe0 legacy=0x7fc0b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b8af2640 1 -- 172.21.15.122:0/2697907111 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40b3710 con 0x7fc0b40b41f0 2024-03-06T07:28:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.323+0000 7fc0b37fe640 1 --2- 172.21.15.122:0/2697907111 >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 0x7fc0b40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.324+0000 7fc0b8af2640 1 --2- 172.21.15.122:0/2697907111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 0x7fc0b40b4680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009e50 tx=0x7fc0ac02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de15b5514e29e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.324+0000 7fc0b2ffd640 1 -- 172.21.15.122:0/2697907111 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac02cb00 con 0x7fc0b40b41f0 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.324+0000 7fc0b2ffd640 1 -- 172.21.15.122:0/2697907111 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac02cca0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.324+0000 7fc0b2ffd640 1 -- 172.21.15.122:0/2697907111 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac034610 con 0x7fc0b40b41f0 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 -- 172.21.15.122:0/2697907111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 msgr2=0x7fc0b40b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 --2- 172.21.15.122:0/2697907111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 0x7fc0b40b4680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009e50 tx=0x7fc0ac02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 -- 172.21.15.122:0/2697907111 shutdown_connections 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 --2- 172.21.15.122:0/2697907111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40b41f0 0x7fc0b40b4680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 --2- 172.21.15.122:0/2697907111 >> v2:172.21.15.169:3300/0 conn(0x7fc0b40d22b0 0x7fc0b40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 -- 172.21.15.122:0/2697907111 >> 172.21.15.122:0/2697907111 conn(0x7fc0b40d3cd0 msgr2=0x7fc0b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 -- 172.21.15.122:0/2697907111 shutdown_connections 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.325+0000 7fc0ba555640 1 -- 172.21.15.122:0/2697907111 wait complete. 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.326+0000 7fc0ba555640 1 Processor -- start 2024-03-06T07:28:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.327+0000 7fc0ba555640 1 -- start start 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0ba555640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 0x7fc0b4160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0ba555640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40d22b0 0x7fc0b40b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0ba555640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b415dfe0 0x7fc0b4157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0ba555640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40cfa00 con 0x7fc0b40b41f0 2024-03-06T07:28:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0ba555640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40b3d60 con 0x7fc0b415dfe0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0ba555640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40d1c70 con 0x7fc0b40d22b0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 0x7fc0b4160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 0x7fc0b4160900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47272/0 (socket says 172.21.15.122:47272) 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 -- 172.21.15.122:0/699638025 learned_addr learned my addr 172.21.15.122:0/699638025 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b8af2640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b415dfe0 0x7fc0b4157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b37fe640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40d22b0 0x7fc0b40b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 -- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40d22b0 msgr2=0x7fc0b40b96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40d22b0 0x7fc0b40b96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 -- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b415dfe0 msgr2=0x7fc0b4157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b415dfe0 0x7fc0b4157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.328+0000 7fc0b3fff640 1 -- 172.21.15.122:0/699638025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40b9be0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.329+0000 7fc0b3fff640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 0x7fc0b4160900 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800c3c0 tx=0x7fc0a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.329+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a80074e0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.329+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a8007680 con 0x7fc0b40b41f0 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.329+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8004a00 con 0x7fc0b40b41f0 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.329+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b415a7e0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.329+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b41603e0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.330+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b404f330 con 0x7fc0b40b41f0 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.330+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0a801a020 con 0x7fc0b40b41f0 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.331+0000 7fc0b0ff9640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc0a006b500 0x7fc0a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.331+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0a8089830 con 0x7fc0b40b41f0 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.331+0000 7fc0b37fe640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc0a006b500 0x7fc0a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.331+0000 7fc0b37fe640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc0a006b500 0x7fc0a006d9c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009560 tx=0x7fc0a4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.335+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0a8058060 con 0x7fc0b40b41f0 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.549+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0b415d3b0 con 0x7fc0b40b41f0 2024-03-06T07:28:44.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.550+0000 7fc0b0ff9640 1 -- 172.21.15.122:0/699638025 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4991 (secure 0 0 0) 0x7fc0a805c100 con 0x7fc0b40b41f0 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/1547452856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1547452856},{"type":"v1","addr":"172.21.15.169:6834","nonce":1547452856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1798905908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1798905908},{"type":"v1","addr":"172.21.15.169:6835","nonce":1798905908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:43.540750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:28:44.552 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 73 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc0a006b500 msgr2=0x7fc0a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc0a006b500 0x7fc0a006d9c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009560 tx=0x7fc0a4009190 comp rx=0 tx=0).stop 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 msgr2=0x7fc0b4160900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 0x7fc0b4160900 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800c3c0 tx=0x7fc0a800c880 comp rx=0 tx=0).stop 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 shutdown_connections 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc0a006b500 0x7fc0a006d9c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b415dfe0 0x7fc0b4157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc0b40d22b0 0x7fc0b40b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 --2- 172.21.15.122:0/699638025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0b40b41f0 0x7fc0b4160900 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 >> 172.21.15.122:0/699638025 conn(0x7fc0b40d3cd0 msgr2=0x7fc0b406bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 shutdown_connections 2024-03-06T07:28:44.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.552+0000 7fc0ba555640 1 -- 172.21.15.122:0/699638025 wait complete. 2024-03-06T07:28:44.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:44.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.670+0000 7f3b35081640 1 Processor -- start 2024-03-06T07:28:44.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.670+0000 7f3b35081640 1 -- start start 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b35081640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3b300d2710 0x7f3b300b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b35081640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b3430 con 0x7f3b300d22b0 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b35081640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b3710 con 0x7f3b3015dfe0 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b35081640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b300b31e0 con 0x7f3b300d2710 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2f59e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3b300d2710 0x7f3b300b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2e59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3b300d22b0 0x7f3b300b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41202/0 (socket says 172.21.15.122:41202) 2024-03-06T07:28:44.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2e59c640 1 -- 172.21.15.122:0/3521240784 learned_addr learned my addr 172.21.15.122:0/3521240784 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:44.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2f59e640 1 -- 172.21.15.122:0/3521240784 >> v1:172.21.15.169:6789/0 conn(0x7f3b3015dfe0 legacy=0x7f3b300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2f59e640 1 -- 172.21.15.122:0/3521240784 >> v1:172.21.15.122:6789/0 conn(0x7f3b300d22b0 legacy=0x7f3b300b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2dd9b640 1 -- 172.21.15.122:0/3521240784 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808127466 0 0) 0x7f3b300b3430 con 0x7f3b300d22b0 2024-03-06T07:28:44.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.671+0000 7f3b2f59e640 1 -- 172.21.15.122:0/3521240784 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300b39f0 con 0x7f3b300d2710 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.672+0000 7f3b2f59e640 1 --2- 172.21.15.122:0/3521240784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b300d2710 0x7f3b300b4900 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009dc0 tx=0x7f3b2802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb471678032272f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.672+0000 7f3b2dd9b640 1 -- 172.21.15.122:0/3521240784 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28039070 con 0x7f3b300d2710 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.672+0000 7f3b2dd9b640 1 -- 172.21.15.122:0/3521240784 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2802ccd0 con 0x7f3b300d2710 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.672+0000 7f3b2dd9b640 1 -- 172.21.15.122:0/3521240784 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28034660 con 0x7f3b300d2710 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 -- 172.21.15.122:0/3521240784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b300d2710 msgr2=0x7f3b300b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 --2- 172.21.15.122:0/3521240784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b300d2710 0x7f3b300b4900 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009dc0 tx=0x7f3b2802bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 -- 172.21.15.122:0/3521240784 shutdown_connections 2024-03-06T07:28:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 --2- 172.21.15.122:0/3521240784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b300d2710 0x7f3b300b4900 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 -- 172.21.15.122:0/3521240784 >> 172.21.15.122:0/3521240784 conn(0x7f3b300d3cd0 msgr2=0x7f3b300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 -- 172.21.15.122:0/3521240784 shutdown_connections 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.673+0000 7f3b35081640 1 -- 172.21.15.122:0/3521240784 wait complete. 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.674+0000 7f3b35081640 1 Processor -- start 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.674+0000 7f3b35081640 1 -- start start 2024-03-06T07:28:44.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b35081640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b300d22b0 0x7f3b3015e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b35081640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b300d2710 0x7f3b300b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b35081640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 0x7f3b301481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b35081640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300cfa00 con 0x7f3b3015dfe0 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b35081640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300b3d60 con 0x7f3b300d2710 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b35081640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300d1c70 con 0x7f3b300d22b0 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 0x7f3b301481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 0x7f3b301481d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39376/0 (socket says 172.21.15.122:39376) 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 -- 172.21.15.122:0/210861675 learned_addr learned my addr 172.21.15.122:0/210861675 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2ed9d640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b300d22b0 0x7f3b3015e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2e59c640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b300d2710 0x7f3b300b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 -- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b300d22b0 msgr2=0x7f3b3015e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b300d22b0 0x7f3b3015e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 -- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b300d2710 msgr2=0x7f3b300b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b300d2710 0x7f3b300b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.675+0000 7f3b2f59e640 1 -- 172.21.15.122:0/210861675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300b1990 con 0x7f3b3015dfe0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.676+0000 7f3b2f59e640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 0x7f3b301481d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3b28037040 tx=0x7f3b28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.676+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28039050 con 0x7f3b3015dfe0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.676+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b28045070 con 0x7f3b3015dfe0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.676+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2803e830 con 0x7f3b3015dfe0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.676+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b300b1c70 con 0x7f3b3015dfe0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.676+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b300b2230 con 0x7f3b3015dfe0 2024-03-06T07:28:44.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.677+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3004f330 con 0x7f3b3015dfe0 2024-03-06T07:28:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.677+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b2803e9d0 con 0x7f3b3015dfe0 2024-03-06T07:28:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.678+0000 7f3b277fe640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.678+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b280ad270 con 0x7f3b3015dfe0 2024-03-06T07:28:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.678+0000 7f3b2ed9d640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.678+0000 7f3b2ed9d640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3b1006b4d0 0x7f3b1006d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3b18000f60 tx=0x7f3b18009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.682+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b2807bbc0 con 0x7f3b3015dfe0 2024-03-06T07:28:44.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.854+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b300cea80 con 0x7f3b3015dfe0 2024-03-06T07:28:44.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.856+0000 7f3b277fe640 1 -- 172.21.15.122:0/210861675 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16679 (secure 0 0 0) 0x7f3b2807fc60 con 0x7f3b3015dfe0 2024-03-06T07:28:44.858 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:44.858 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":126,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:28:43.182048+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:44.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3b1006b4d0 msgr2=0x7f3b1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3b1006b4d0 0x7f3b1006d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3b18000f60 tx=0x7f3b18009040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 msgr2=0x7f3b301481d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 0x7f3b301481d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3b28037040 tx=0x7f3b28004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 shutdown_connections 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b3015dfe0 0x7f3b301481d0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b300d2710 0x7f3b300b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 --2- 172.21.15.122:0/210861675 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3b300d22b0 0x7f3b3015e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 >> 172.21.15.122:0/210861675 conn(0x7f3b300d3cd0 msgr2=0x7f3b300b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 shutdown_connections 2024-03-06T07:28:44.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.858+0000 7f3b35081640 1 -- 172.21.15.122:0/210861675 wait complete. 2024-03-06T07:28:44.871 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:44.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:44.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.970+0000 7f1288163640 1 Processor -- start 2024-03-06T07:28:44.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1288163640 1 -- start start 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1288163640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f128015dfe0 0x7f12800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1288163640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12800d29c0 0x7f12800b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1288163640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800b3430 con 0x7f12800cf830 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1288163640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12800b31e0 con 0x7f128015dfe0 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1288163640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12800b3d60 con 0x7f12800d29c0 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f1285eff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f128015dfe0 0x7f12800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f12856fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f12800cf830 0x7f12800d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41214/0 (socket says 172.21.15.122:41214) 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.971+0000 7f12856fe640 1 -- 172.21.15.122:0/644054736 learned_addr learned my addr 172.21.15.122:0/644054736 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1286700640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12800d29c0 0x7f12800b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1285eff640 1 -- 172.21.15.122:0/644054736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12800d29c0 msgr2=0x7f12800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1285eff640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12800d29c0 0x7f12800b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1285eff640 1 -- 172.21.15.122:0/644054736 >> v1:172.21.15.122:6789/0 conn(0x7f12800cf830 legacy=0x7f12800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1284efd640 1 -- 172.21.15.122:0/644054736 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129478400 0 0) 0x7f12800b3430 con 0x7f12800cf830 2024-03-06T07:28:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1285eff640 1 -- 172.21.15.122:0/644054736 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800b3710 con 0x7f128015dfe0 2024-03-06T07:28:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.972+0000 7f1286700640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12800d29c0 0x7f12800b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1285eff640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f128015dfe0 0x7f12800cf2f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f12700097a0 tx=0x7f127002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2600c099dec6e326 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1284efd640 1 -- 172.21.15.122:0/644054736 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270035020 con 0x7f128015dfe0 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1284efd640 1 -- 172.21.15.122:0/644054736 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1270031830 con 0x7f128015dfe0 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1284efd640 1 -- 172.21.15.122:0/644054736 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270031c00 con 0x7f128015dfe0 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 -- 172.21.15.122:0/644054736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f128015dfe0 msgr2=0x7f12800cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f128015dfe0 0x7f12800cf2f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f12700097a0 tx=0x7f127002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 -- 172.21.15.122:0/644054736 shutdown_connections 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f12800d29c0 0x7f12800b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 --2- 172.21.15.122:0/644054736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f128015dfe0 0x7f12800cf2f0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 -- 172.21.15.122:0/644054736 >> 172.21.15.122:0/644054736 conn(0x7f12800d3cd0 msgr2=0x7f12800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 -- 172.21.15.122:0/644054736 shutdown_connections 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.973+0000 7f1288163640 1 -- 172.21.15.122:0/644054736 wait complete. 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.974+0000 7f1288163640 1 Processor -- start 2024-03-06T07:28:44.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.974+0000 7f1288163640 1 -- start start 2024-03-06T07:28:44.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1288163640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 0x7f12800b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1288163640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12800d29c0 0x7f12800b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1288163640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f128015dfe0 0x7f128014c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1288163640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800d1df0 con 0x7f12800cf830 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1288163640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800d1c70 con 0x7f12800d29c0 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1288163640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128015e3c0 con 0x7f128015dfe0 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1285eff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 0x7f12800b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1285eff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 0x7f12800b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39398/0 (socket says 172.21.15.122:39398) 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1285eff640 1 -- 172.21.15.122:0/1086775272 learned_addr learned my addr 172.21.15.122:0/1086775272 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f1286700640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f128015dfe0 0x7f128014c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.975+0000 7f12856fe640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12800d29c0 0x7f12800b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1285eff640 1 -- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f128015dfe0 msgr2=0x7f128014c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1285eff640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f128015dfe0 0x7f128014c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1285eff640 1 -- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12800d29c0 msgr2=0x7f12800b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1285eff640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12800d29c0 0x7f12800b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1285eff640 1 -- 172.21.15.122:0/1086775272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800b8cf0 con 0x7f12800cf830 2024-03-06T07:28:44.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1285eff640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 0x7f12800b4c60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1270034040 tx=0x7f12700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270040070 con 0x7f12800cf830 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12700399a0 con 0x7f12800cf830 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12800b8fd0 con 0x7f12800cf830 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270035040 con 0x7f12800cf830 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.976+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12800b9590 con 0x7f12800cf830 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.977+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12800b9b70 con 0x7f12800cf830 2024-03-06T07:28:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.977+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1270038020 con 0x7f12800cf830 2024-03-06T07:28:44.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.978+0000 7f12767fc640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f126406b4d0 0x7f126406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.978+0000 7f12856fe640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f126406b4d0 0x7f126406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.978+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12700aba80 con 0x7f12800cf830 2024-03-06T07:28:44.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.978+0000 7f12856fe640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f126406b4d0 0x7f126406d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f12800ce160 tx=0x7f127c007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:44.982+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12800b9b70 con 0x7f12800cf830 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.252+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f128004f330 con 0x7f126406b4d0 2024-03-06T07:28:45.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.257+0000 7f12767fc640 1 -- 172.21.15.122:0/1086775272 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240259 (secure 0 0 0) 0x7f128004f330 con 0x7f126406b4d0 2024-03-06T07:28:45.262 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:45.263 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f126406b4d0 msgr2=0x7f126406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f126406b4d0 0x7f126406d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f12800ce160 tx=0x7f127c007b50 comp rx=0 tx=0).stop 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 msgr2=0x7f12800b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 0x7f12800b4c60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1270034040 tx=0x7f12700046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 shutdown_connections 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f126406b4d0 0x7f126406d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f128015dfe0 0x7f128014c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12800d29c0 0x7f12800b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 --2- 172.21.15.122:0/1086775272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12800cf830 0x7f12800b4c60 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.260+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 >> 172.21.15.122:0/1086775272 conn(0x7f12800d3cd0 msgr2=0x7f128006ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.261+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 shutdown_connections 2024-03-06T07:28:45.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.261+0000 7f1288163640 1 -- 172.21.15.122:0/1086775272 wait complete. 2024-03-06T07:28:45.274 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-03-06T07:28:43.411687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324968,"kb_used_data":13100,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754649752,"statfs":{"total":773094113280,"available":772761346048,"internally_reserved":0,"allocated":13414400,"data_stored":6470579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000577"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171856+0000","last_change":"2024-03-06T07:28:40.171856+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171856+0000","last_undegraded":"2024-03-06T07:28:40.171856+0000","last_fullsized":"2024-03-06T07:28:40.171856+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174362+0000","last_change":"2024-03-06T07:28:40.174362+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174362+0000","last_undegraded":"2024-03-06T07:28:40.174362+0000","last_fullsized":"2024-03-06T07:28:40.174362+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171671+0000","last_change":"2024-03-06T07:28:40.171671+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171671+0000","last_undegraded":"2024-03-06T07:28:40.171671+0000","last_fullsized":"2024-03-06T07:28:40.171671+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172699+0000","last_change":"2024-03-06T07:28:40.172699+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172699+0000","last_undegraded":"2024-03-06T07:28:40.172699+0000","last_fullsized":"2024-03-06T07:28:40.172699+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172706+0000","last_change":"2024-03-06T07:28:40.172706+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172706+0000","last_undegraded":"2024-03-06T07:28:40.172706+0000","last_fullsized":"2024-03-06T07:28:40.172706+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172579+0000","last_change":"2024-03-06T07:28:40.172579+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172579+0000","last_undegraded":"2024-03-06T07:28:40.172579+0000","last_fullsized":"2024-03-06T07:28:40.172579+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.169983+0000","last_change":"2024-03-06T07:28:40.169983+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.169983+0000","last_undegraded":"2024-03-06T07:28:40.169983+0000","last_fullsized":"2024-03-06T07:28:40.169983+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172274+0000","last_change":"2024-03-06T07:28:40.172274+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172274+0000","last_undegraded":"2024-03-06T07:28:40.172274+0000","last_fullsized":"2024-03-06T07:28:40.172274+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174890+0000","last_change":"2024-03-06T07:28:40.174890+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174890+0000","last_undegraded":"2024-03-06T07:28:40.174890+0000","last_fullsized":"2024-03-06T07:28:40.174890+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172439+0000","last_change":"2024-03-06T07:28:40.172439+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172439+0000","last_undegraded":"2024-03-06T07:28:40.172439+0000","last_fullsized":"2024-03-06T07:28:40.172439+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174333+0000","last_change":"2024-03-06T07:28:40.174333+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174333+0000","last_undegraded":"2024-03-06T07:28:40.174333+0000","last_fullsized":"2024-03-06T07:28:40.174333+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.169672+0000","last_change":"2024-03-06T07:28:40.169672+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.169672+0000","last_undegraded":"2024-03-06T07:28:40.169672+0000","last_fullsized":"2024-03-06T07:28:40.169672+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174547+0000","last_change":"2024-03-06T07:28:40.174547+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174547+0000","last_undegraded":"2024-03-06T07:28:40.174547+0000","last_fullsized":"2024-03-06T07:28:40.174547+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172271+0000","last_change":"2024-03-06T07:28:40.172271+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172271+0000","last_undegraded":"2024-03-06T07:28:40.172271+0000","last_fullsized":"2024-03-06T07:28:40.172271+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.169667+0000","last_change":"2024-03-06T07:28:40.169667+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.169667+0000","last_undegraded":"2024-03-06T07:28:40.169667+0000","last_fullsized":"2024-03-06T07:28:40.169667+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174348+0000","last_change":"2024-03-06T07:28:40.174348+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174348+0000","last_undegraded":"2024-03-06T07:28:40.174348+0000","last_fullsized":"2024-03-06T07:28:40.174348+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171184+0000","last_change":"2024-03-06T07:28:40.171184+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171184+0000","last_undegraded":"2024-03-06T07:28:40.171184+0000","last_fullsized":"2024-03-06T07:28:40.171184+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.169668+0000","last_change":"2024-03-06T07:28:40.169668+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.169668+0000","last_undegraded":"2024-03-06T07:28:40.169668+0000","last_fullsized":"2024-03-06T07:28:40.169668+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174278+0000","last_change":"2024-03-06T07:28:40.174278+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174278+0000","last_undegraded":"2024-03-06T07:28:40.174278+0000","last_fullsized":"2024-03-06T07:28:40.174278+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174613+0000","last_change":"2024-03-06T07:28:40.174613+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174613+0000","last_undegraded":"2024-03-06T07:28:40.174613+0000","last_fullsized":"2024-03-06T07:28:40.174613+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172173+0000","last_change":"2024-03-06T07:28:40.172173+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172173+0000","last_undegraded":"2024-03-06T07:28:40.172173+0000","last_fullsized":"2024-03-06T07:28:40.172173+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172527+0000","last_change":"2024-03-06T07:28:40.172527+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172527+0000","last_undegraded":"2024-03-06T07:28:40.172527+0000","last_fullsized":"2024-03-06T07:28:40.172527+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171146+0000","last_change":"2024-03-06T07:28:40.171146+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171146+0000","last_undegraded":"2024-03-06T07:28:40.171146+0000","last_fullsized":"2024-03-06T07:28:40.171146+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.174540+0000","last_change":"2024-03-06T07:28:40.174540+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.174540+0000","last_undegraded":"2024-03-06T07:28:40.174540+0000","last_fullsized":"2024-03-06T07:28:40.174540+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171883+0000","last_change":"2024-03-06T07:28:40.171883+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171883+0000","last_undegraded":"2024-03-06T07:28:40.171883+0000","last_fullsized":"2024-03-06T07:28:40.171883+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172758+0000","last_change":"2024-03-06T07:28:40.172758+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172758+0000","last_undegraded":"2024-03-06T07:28:40.172758+0000","last_fullsized":"2024-03-06T07:28:40.172758+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.170540+0000","last_change":"2024-03-06T07:28:40.170540+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.170540+0000","last_undegraded":"2024-03-06T07:28:40.170540+0000","last_fullsized":"2024-03-06T07:28:40.170540+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172317+0000","last_change":"2024-03-06T07:28:40.172317+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172317+0000","last_undegraded":"2024-03-06T07:28:40.172317+0000","last_fullsized":"2024-03-06T07:28:40.172317+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171952+0000","last_change":"2024-03-06T07:28:40.171952+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171952+0000","last_undegraded":"2024-03-06T07:28:40.171952+0000","last_fullsized":"2024-03-06T07:28:40.171952+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.172145+0000","last_change":"2024-03-06T07:28:40.172145+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.172145+0000","last_undegraded":"2024-03-06T07:28:40.172145+0000","last_fullsized":"2024-03-06T07:28:40.172145+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.170532+0000","last_change":"2024-03-06T07:28:40.170532+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.170532+0000","last_undegraded":"2024-03-06T07:28:40.170532+0000","last_fullsized":"2024-03-06T07:28:40.170532+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.167252+0000","last_change":"2024-03-06T07:28:41.167252+0000","last_active":"2024-03-06T07:28:41.167252+0000","last_peered":"2024-03-06T07:28:41.167252+0000","last_clean":"2024-03-06T07:28:41.167252+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.167252+0000","last_undegraded":"2024-03-06T07:28:41.167252+0000","last_fullsized":"2024-03-06T07:28:41.167252+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.171873+0000","last_change":"2024-03-06T07:28:40.171873+0000","last_active":"2024-03-06T07:28:40.162400+0000","last_peered":"2024-03-06T07:28:40.162400+0000","last_clean":"2024-03-06T07:28:40.162400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.171873+0000","last_undegraded":"2024-03-06T07:28:40.171873+0000","last_fullsized":"2024-03-06T07:28:40.171873+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":364,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.172499+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:28:40.172499+0000","last_peered":"2024-03-06T07:28:40.172499+0000","last_clean":"2024-03-06T07:28:40.172499+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:28:40.172499+0000","last_undegraded":"2024-03-06T07:28:40.172499+0000","last_fullsized":"2024-03-06T07:28:40.172499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37408,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334432,"statfs":{"total":96636764160,"available":96598458368,"internally_reserved":0,"allocated":2117632,"data_stored":1249417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1523712,"data_stored":659049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40988,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330852,"statfs":{"total":96636764160,"available":96594792448,"internally_reserved":0,"allocated":1523712,"data_stored":659049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40992,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330848,"statfs":{"total":96636764160,"available":96594788352,"internally_reserved":0,"allocated":1523712,"data_stored":659049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40996,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330844,"statfs":{"total":96636764160,"available":96594784256,"internally_reserved":0,"allocated":1523712,"data_stored":659049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":664866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":664866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574937,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1255234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.415+0000 7fc52cd3a640 1 Processor -- start 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.415+0000 7fc52cd3a640 1 -- start start 2024-03-06T07:28:45.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.415+0000 7fc52cd3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5280d4760 0x7fc5280b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.415+0000 7fc52cd3a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5280d25d0 con 0x7fc5280b4170 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.415+0000 7fc52cd3a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5280d2870 con 0x7fc5280d0a80 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.415+0000 7fc52cd3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280d4bf0 con 0x7fc5280d4760 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc52659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc5280b4170 0x7fc5280d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41222/0 (socket says 172.21.15.122:41222) 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc52659c640 1 -- 172.21.15.122:0/2154554720 learned_addr learned my addr 172.21.15.122:0/2154554720 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102888129 0 0) 0x7fc5280d25d0 con 0x7fc5280b4170 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc51ffff640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5280d4760 0x7fc5280b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc514003520 con 0x7fc5280b4170 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848462915 0 0) 0x7fc514003520 con 0x7fc5280b4170 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5280d4760 msgr2=0x7fc5280b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5280d4760 0x7fc5280b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 >> v1:172.21.15.169:6789/0 conn(0x7fc5280d0a80 legacy=0x7fc5280cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280d2b50 con 0x7fc5280b4170 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fc518001670 con 0x7fc5280b4170 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 >> v1:172.21.15.122:6789/0 conn(0x7fc5280b4170 legacy=0x7fc5280d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5140039e0 0x7fc514003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc514004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc514005020 0x7fc51400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280d4bf0 con 0x7fc514004340 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51400bf00 con 0x7fc514005020 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51400c0b0 con 0x7fc5140039e0 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc51ffff640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc514004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc52659c640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5140039e0 0x7fc514003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.416+0000 7fc526d9d640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc514005020 0x7fc51400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc51ffff640 1 -- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5140039e0 msgr2=0x7fc514003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc51ffff640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5140039e0 0x7fc514003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc51ffff640 1 -- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc514005020 msgr2=0x7fc51400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc51ffff640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc514005020 0x7fc51400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc51ffff640 1 -- 172.21.15.122:0/2154554720 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc514003520 con 0x7fc514004340 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc526d9d640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc514005020 0x7fc51400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc51ffff640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc514004820 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc510009ac0 tx=0x7fc514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3313cf4fe70c969 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510039070 con 0x7fc514004340 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.417+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51002c6b0 con 0x7fc514004340 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc525d9b640 1 -- 172.21.15.122:0/2154554720 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510009690 con 0x7fc514004340 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc52cd3a640 1 -- 172.21.15.122:0/2154554720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 msgr2=0x7fc514004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/2154554720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc514004820 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc510009ac0 tx=0x7fc514005830 comp rx=0 tx=0).stop 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc526d9d640 1 -- 172.21.15.122:0/2154554720 reap_dead start 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc52cd3a640 1 -- 172.21.15.122:0/2154554720 shutdown_connections 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc52cd3a640 1 -- 172.21.15.122:0/2154554720 >> 172.21.15.122:0/2154554720 conn(0x7fc5280d3c40 msgr2=0x7fc5280d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc52cd3a640 1 -- 172.21.15.122:0/2154554720 shutdown_connections 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.418+0000 7fc52cd3a640 1 -- 172.21.15.122:0/2154554720 wait complete. 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.420+0000 7fc52cd3a640 1 Processor -- start 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.420+0000 7fc52cd3a640 1 -- start start 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52cd3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc52815ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52cd3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc52815a530 0x7fc528157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52cd3a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5280b2980 0x7fc5280b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52cd3a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b4dd0 con 0x7fc514004340 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52cd3a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280ce9e0 con 0x7fc5280b2980 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52cd3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b4f50 con 0x7fc52815a530 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc52815ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.421+0000 7fc52659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc52815ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39420/0 (socket says 172.21.15.122:39420) 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 -- 172.21.15.122:0/1844253956 learned_addr learned my addr 172.21.15.122:0/1844253956 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc51ffff640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc52815a530 0x7fc528157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc526d9d640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5280b2980 0x7fc5280b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 -- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc52815a530 msgr2=0x7fc528157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc52815a530 0x7fc528157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 -- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5280b2980 msgr2=0x7fc5280b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5280b2980 0x7fc5280b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 -- 172.21.15.122:0/1844253956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52815ad20 con 0x7fc514004340 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52659c640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc52815ba70 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fc518007290 tx=0x7fc518007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51800f960 con 0x7fc514004340 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc518018070 con 0x7fc514004340 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52815b270 con 0x7fc514004340 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51800fd20 con 0x7fc514004340 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.422+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5280b2460 con 0x7fc514004340 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.423+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52804f330 con 0x7fc514004340 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.424+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc51801d460 con 0x7fc514004340 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.424+0000 7fc51effd640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc50806b500 0x7fc50806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.424+0000 7fc51ffff640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc50806b500 0x7fc50806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.424+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc51808a260 con 0x7fc514004340 2024-03-06T07:28:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.425+0000 7fc51ffff640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc50806b500 0x7fc50806d9c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc510036000 tx=0x7fc510030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.428+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5180590e0 con 0x7fc514004340 2024-03-06T07:28:45.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.603+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5280733b0 con 0x7fc50806b500 2024-03-06T07:28:45.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.609+0000 7fc51effd640 1 -- 172.21.15.122:0/1844253956 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7fc510082a50 con 0x7fc50806b500 2024-03-06T07:28:45.614 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:45.615 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc50806b500 msgr2=0x7fc50806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc50806b500 0x7fc50806d9c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc510036000 tx=0x7fc510030000 comp rx=0 tx=0).stop 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 msgr2=0x7fc52815ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc52815ba70 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fc518007290 tx=0x7fc518007750 comp rx=0 tx=0).stop 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 shutdown_connections 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc50806b500 0x7fc50806d9c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5280b2980 0x7fc5280b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc52815a530 0x7fc528157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 --2- 172.21.15.122:0/1844253956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc514004340 0x7fc52815ba70 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 >> 172.21.15.122:0/1844253956 conn(0x7fc5280d3c40 msgr2=0x7fc5280b8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 shutdown_connections 2024-03-06T07:28:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.612+0000 7fc52cd3a640 1 -- 172.21.15.122:0/1844253956 wait complete. 2024-03-06T07:28:45.626 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-03-06T07:28:45.412060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329520,"kb_used_data":13512,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645200,"statfs":{"total":773094113280,"available":772756684800,"internally_reserved":0,"allocated":13836288,"data_stored":6659284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.244417"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185514+0000","last_change":"2024-03-06T07:28:42.201426+0000","last_active":"2024-03-06T07:28:43.185514+0000","last_peered":"2024-03-06T07:28:43.185514+0000","last_clean":"2024-03-06T07:28:43.185514+0000","last_became_active":"2024-03-06T07:28:42.201184+0000","last_became_peered":"2024-03-06T07:28:42.201184+0000","last_unstale":"2024-03-06T07:28:43.185514+0000","last_undegraded":"2024-03-06T07:28:43.185514+0000","last_fullsized":"2024-03-06T07:28:43.185514+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:55.423456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230175+0000","last_change":"2024-03-06T07:28:42.200317+0000","last_active":"2024-03-06T07:28:43.230175+0000","last_peered":"2024-03-06T07:28:43.230175+0000","last_clean":"2024-03-06T07:28:43.230175+0000","last_became_active":"2024-03-06T07:28:42.200065+0000","last_became_peered":"2024-03-06T07:28:42.200065+0000","last_unstale":"2024-03-06T07:28:43.230175+0000","last_undegraded":"2024-03-06T07:28:43.230175+0000","last_fullsized":"2024-03-06T07:28:43.230175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:39.495663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187109+0000","last_change":"2024-03-06T07:28:42.195888+0000","last_active":"2024-03-06T07:28:43.187109+0000","last_peered":"2024-03-06T07:28:43.187109+0000","last_clean":"2024-03-06T07:28:43.187109+0000","last_became_active":"2024-03-06T07:28:42.195413+0000","last_became_peered":"2024-03-06T07:28:42.195413+0000","last_unstale":"2024-03-06T07:28:43.187109+0000","last_undegraded":"2024-03-06T07:28:43.187109+0000","last_fullsized":"2024-03-06T07:28:43.187109+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:39.157261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185276+0000","last_change":"2024-03-06T07:28:42.204776+0000","last_active":"2024-03-06T07:28:43.185276+0000","last_peered":"2024-03-06T07:28:43.185276+0000","last_clean":"2024-03-06T07:28:43.185276+0000","last_became_active":"2024-03-06T07:28:42.204545+0000","last_became_peered":"2024-03-06T07:28:42.204545+0000","last_unstale":"2024-03-06T07:28:43.185276+0000","last_undegraded":"2024-03-06T07:28:43.185276+0000","last_fullsized":"2024-03-06T07:28:43.185276+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:44.615652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186073+0000","last_change":"2024-03-06T07:28:42.201973+0000","last_active":"2024-03-06T07:28:43.186073+0000","last_peered":"2024-03-06T07:28:43.186073+0000","last_clean":"2024-03-06T07:28:43.186073+0000","last_became_active":"2024-03-06T07:28:42.201716+0000","last_became_peered":"2024-03-06T07:28:42.201716+0000","last_unstale":"2024-03-06T07:28:43.186073+0000","last_undegraded":"2024-03-06T07:28:43.186073+0000","last_fullsized":"2024-03-06T07:28:43.186073+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:56.655582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.210899+0000","last_change":"2024-03-06T07:28:42.203966+0000","last_active":"2024-03-06T07:28:43.210899+0000","last_peered":"2024-03-06T07:28:43.210899+0000","last_clean":"2024-03-06T07:28:43.210899+0000","last_became_active":"2024-03-06T07:28:42.203686+0000","last_became_peered":"2024-03-06T07:28:42.203686+0000","last_unstale":"2024-03-06T07:28:43.210899+0000","last_undegraded":"2024-03-06T07:28:43.210899+0000","last_fullsized":"2024-03-06T07:28:43.210899+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:34.502047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.210757+0000","last_change":"2024-03-06T07:28:42.207519+0000","last_active":"2024-03-06T07:28:43.210757+0000","last_peered":"2024-03-06T07:28:43.210757+0000","last_clean":"2024-03-06T07:28:43.210757+0000","last_became_active":"2024-03-06T07:28:42.207229+0000","last_became_peered":"2024-03-06T07:28:42.207229+0000","last_unstale":"2024-03-06T07:28:43.210757+0000","last_undegraded":"2024-03-06T07:28:43.210757+0000","last_fullsized":"2024-03-06T07:28:43.210757+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:29.186709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187200+0000","last_change":"2024-03-06T07:28:42.198278+0000","last_active":"2024-03-06T07:28:43.187200+0000","last_peered":"2024-03-06T07:28:43.187200+0000","last_clean":"2024-03-06T07:28:43.187200+0000","last_became_active":"2024-03-06T07:28:42.197948+0000","last_became_peered":"2024-03-06T07:28:42.197948+0000","last_unstale":"2024-03-06T07:28:43.187200+0000","last_undegraded":"2024-03-06T07:28:43.187200+0000","last_fullsized":"2024-03-06T07:28:43.187200+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:30.750314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187165+0000","last_change":"2024-03-06T07:28:42.200302+0000","last_active":"2024-03-06T07:28:43.187165+0000","last_peered":"2024-03-06T07:28:43.187165+0000","last_clean":"2024-03-06T07:28:43.187165+0000","last_became_active":"2024-03-06T07:28:42.200011+0000","last_became_peered":"2024-03-06T07:28:42.200011+0000","last_unstale":"2024-03-06T07:28:43.187165+0000","last_undegraded":"2024-03-06T07:28:43.187165+0000","last_fullsized":"2024-03-06T07:28:43.187165+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:18.472201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230305+0000","last_change":"2024-03-06T07:28:42.199622+0000","last_active":"2024-03-06T07:28:43.230305+0000","last_peered":"2024-03-06T07:28:43.230305+0000","last_clean":"2024-03-06T07:28:43.230305+0000","last_became_active":"2024-03-06T07:28:42.199360+0000","last_became_peered":"2024-03-06T07:28:42.199360+0000","last_unstale":"2024-03-06T07:28:43.230305+0000","last_undegraded":"2024-03-06T07:28:43.230305+0000","last_fullsized":"2024-03-06T07:28:43.230305+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:05.674362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230111+0000","last_change":"2024-03-06T07:28:42.194017+0000","last_active":"2024-03-06T07:28:43.230111+0000","last_peered":"2024-03-06T07:28:43.230111+0000","last_clean":"2024-03-06T07:28:43.230111+0000","last_became_active":"2024-03-06T07:28:42.193762+0000","last_became_peered":"2024-03-06T07:28:42.193762+0000","last_unstale":"2024-03-06T07:28:43.230111+0000","last_undegraded":"2024-03-06T07:28:43.230111+0000","last_fullsized":"2024-03-06T07:28:43.230111+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:27.077653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187363+0000","last_change":"2024-03-06T07:28:42.196923+0000","last_active":"2024-03-06T07:28:43.187363+0000","last_peered":"2024-03-06T07:28:43.187363+0000","last_clean":"2024-03-06T07:28:43.187363+0000","last_became_active":"2024-03-06T07:28:42.195727+0000","last_became_peered":"2024-03-06T07:28:42.195727+0000","last_unstale":"2024-03-06T07:28:43.187363+0000","last_undegraded":"2024-03-06T07:28:43.187363+0000","last_fullsized":"2024-03-06T07:28:43.187363+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:14.925873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185225+0000","last_change":"2024-03-06T07:28:42.207427+0000","last_active":"2024-03-06T07:28:43.185225+0000","last_peered":"2024-03-06T07:28:43.185225+0000","last_clean":"2024-03-06T07:28:43.185225+0000","last_became_active":"2024-03-06T07:28:42.207144+0000","last_became_peered":"2024-03-06T07:28:42.207144+0000","last_unstale":"2024-03-06T07:28:43.185225+0000","last_undegraded":"2024-03-06T07:28:43.185225+0000","last_fullsized":"2024-03-06T07:28:43.185225+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:16.880403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229811+0000","last_change":"2024-03-06T07:28:42.202574+0000","last_active":"2024-03-06T07:28:43.229811+0000","last_peered":"2024-03-06T07:28:43.229811+0000","last_clean":"2024-03-06T07:28:43.229811+0000","last_became_active":"2024-03-06T07:28:42.202300+0000","last_became_peered":"2024-03-06T07:28:42.202300+0000","last_unstale":"2024-03-06T07:28:43.229811+0000","last_undegraded":"2024-03-06T07:28:43.229811+0000","last_fullsized":"2024-03-06T07:28:43.229811+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:19.967407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185115+0000","last_change":"2024-03-06T07:28:42.207997+0000","last_active":"2024-03-06T07:28:43.185115+0000","last_peered":"2024-03-06T07:28:43.185115+0000","last_clean":"2024-03-06T07:28:43.185115+0000","last_became_active":"2024-03-06T07:28:42.207734+0000","last_became_peered":"2024-03-06T07:28:42.207734+0000","last_unstale":"2024-03-06T07:28:43.185115+0000","last_undegraded":"2024-03-06T07:28:43.185115+0000","last_fullsized":"2024-03-06T07:28:43.185115+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:30.366732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231019+0000","last_change":"2024-03-06T07:28:42.202351+0000","last_active":"2024-03-06T07:28:43.231019+0000","last_peered":"2024-03-06T07:28:43.231019+0000","last_clean":"2024-03-06T07:28:43.231019+0000","last_became_active":"2024-03-06T07:28:42.202092+0000","last_became_peered":"2024-03-06T07:28:42.202092+0000","last_unstale":"2024-03-06T07:28:43.231019+0000","last_undegraded":"2024-03-06T07:28:43.231019+0000","last_fullsized":"2024-03-06T07:28:43.231019+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:01.011834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.211086+0000","last_change":"2024-03-06T07:28:42.203816+0000","last_active":"2024-03-06T07:28:43.211086+0000","last_peered":"2024-03-06T07:28:43.211086+0000","last_clean":"2024-03-06T07:28:43.211086+0000","last_became_active":"2024-03-06T07:28:42.203537+0000","last_became_peered":"2024-03-06T07:28:42.203537+0000","last_unstale":"2024-03-06T07:28:43.211086+0000","last_undegraded":"2024-03-06T07:28:43.211086+0000","last_fullsized":"2024-03-06T07:28:43.211086+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:30.750314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231230+0000","last_change":"2024-03-06T07:28:42.205884+0000","last_active":"2024-03-06T07:28:43.231230+0000","last_peered":"2024-03-06T07:28:43.231230+0000","last_clean":"2024-03-06T07:28:43.231230+0000","last_became_active":"2024-03-06T07:28:42.205608+0000","last_became_peered":"2024-03-06T07:28:42.205608+0000","last_unstale":"2024-03-06T07:28:43.231230+0000","last_undegraded":"2024-03-06T07:28:43.231230+0000","last_fullsized":"2024-03-06T07:28:43.231230+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:01.508750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230235+0000","last_change":"2024-03-06T07:28:42.196464+0000","last_active":"2024-03-06T07:28:43.230235+0000","last_peered":"2024-03-06T07:28:43.230235+0000","last_clean":"2024-03-06T07:28:43.230235+0000","last_became_active":"2024-03-06T07:28:42.196121+0000","last_became_peered":"2024-03-06T07:28:42.196121+0000","last_unstale":"2024-03-06T07:28:43.230235+0000","last_undegraded":"2024-03-06T07:28:43.230235+0000","last_fullsized":"2024-03-06T07:28:43.230235+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:15.616038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231327+0000","last_change":"2024-03-06T07:28:42.203775+0000","last_active":"2024-03-06T07:28:43.231327+0000","last_peered":"2024-03-06T07:28:43.231327+0000","last_clean":"2024-03-06T07:28:43.231327+0000","last_became_active":"2024-03-06T07:28:42.203489+0000","last_became_peered":"2024-03-06T07:28:42.203489+0000","last_unstale":"2024-03-06T07:28:43.231327+0000","last_undegraded":"2024-03-06T07:28:43.231327+0000","last_fullsized":"2024-03-06T07:28:43.231327+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:39.931966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186192+0000","last_change":"2024-03-06T07:28:42.198479+0000","last_active":"2024-03-06T07:28:43.186192+0000","last_peered":"2024-03-06T07:28:43.186192+0000","last_clean":"2024-03-06T07:28:43.186192+0000","last_became_active":"2024-03-06T07:28:42.198220+0000","last_became_peered":"2024-03-06T07:28:42.198220+0000","last_unstale":"2024-03-06T07:28:43.186192+0000","last_undegraded":"2024-03-06T07:28:43.186192+0000","last_fullsized":"2024-03-06T07:28:43.186192+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:00.257786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229702+0000","last_change":"2024-03-06T07:28:42.193887+0000","last_active":"2024-03-06T07:28:43.229702+0000","last_peered":"2024-03-06T07:28:43.229702+0000","last_clean":"2024-03-06T07:28:43.229702+0000","last_became_active":"2024-03-06T07:28:42.193605+0000","last_became_peered":"2024-03-06T07:28:42.193605+0000","last_unstale":"2024-03-06T07:28:43.229702+0000","last_undegraded":"2024-03-06T07:28:43.229702+0000","last_fullsized":"2024-03-06T07:28:43.229702+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:32.121737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187282+0000","last_change":"2024-03-06T07:28:42.196387+0000","last_active":"2024-03-06T07:28:43.187282+0000","last_peered":"2024-03-06T07:28:43.187282+0000","last_clean":"2024-03-06T07:28:43.187282+0000","last_became_active":"2024-03-06T07:28:42.195952+0000","last_became_peered":"2024-03-06T07:28:42.195952+0000","last_unstale":"2024-03-06T07:28:43.187282+0000","last_undegraded":"2024-03-06T07:28:43.187282+0000","last_fullsized":"2024-03-06T07:28:43.187282+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:15.767489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229591+0000","last_change":"2024-03-06T07:28:42.197241+0000","last_active":"2024-03-06T07:28:43.229591+0000","last_peered":"2024-03-06T07:28:43.229591+0000","last_clean":"2024-03-06T07:28:43.229591+0000","last_became_active":"2024-03-06T07:28:42.195704+0000","last_became_peered":"2024-03-06T07:28:42.195704+0000","last_unstale":"2024-03-06T07:28:43.229591+0000","last_undegraded":"2024-03-06T07:28:43.229591+0000","last_fullsized":"2024-03-06T07:28:43.229591+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:57.816477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230014+0000","last_change":"2024-03-06T07:28:42.197358+0000","last_active":"2024-03-06T07:28:43.230014+0000","last_peered":"2024-03-06T07:28:43.230014+0000","last_clean":"2024-03-06T07:28:43.230014+0000","last_became_active":"2024-03-06T07:28:42.197095+0000","last_became_peered":"2024-03-06T07:28:42.197095+0000","last_unstale":"2024-03-06T07:28:43.230014+0000","last_undegraded":"2024-03-06T07:28:43.230014+0000","last_fullsized":"2024-03-06T07:28:43.230014+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:06.141317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231380+0000","last_change":"2024-03-06T07:28:42.204728+0000","last_active":"2024-03-06T07:28:43.231380+0000","last_peered":"2024-03-06T07:28:43.231380+0000","last_clean":"2024-03-06T07:28:43.231380+0000","last_became_active":"2024-03-06T07:28:42.204396+0000","last_became_peered":"2024-03-06T07:28:42.204396+0000","last_unstale":"2024-03-06T07:28:43.231380+0000","last_undegraded":"2024-03-06T07:28:43.231380+0000","last_fullsized":"2024-03-06T07:28:43.231380+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:26.015839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185466+0000","last_change":"2024-03-06T07:28:42.206077+0000","last_active":"2024-03-06T07:28:43.185466+0000","last_peered":"2024-03-06T07:28:43.185466+0000","last_clean":"2024-03-06T07:28:43.185466+0000","last_became_active":"2024-03-06T07:28:42.205796+0000","last_became_peered":"2024-03-06T07:28:42.205796+0000","last_unstale":"2024-03-06T07:28:43.185466+0000","last_undegraded":"2024-03-06T07:28:43.185466+0000","last_fullsized":"2024-03-06T07:28:43.185466+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:34.250303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229595+0000","last_change":"2024-03-06T07:28:42.201877+0000","last_active":"2024-03-06T07:28:43.229595+0000","last_peered":"2024-03-06T07:28:43.229595+0000","last_clean":"2024-03-06T07:28:43.229595+0000","last_became_active":"2024-03-06T07:28:42.201572+0000","last_became_peered":"2024-03-06T07:28:42.201572+0000","last_unstale":"2024-03-06T07:28:43.229595+0000","last_undegraded":"2024-03-06T07:28:43.229595+0000","last_fullsized":"2024-03-06T07:28:43.229595+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:48.581098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185289+0000","last_change":"2024-03-06T07:28:42.208181+0000","last_active":"2024-03-06T07:28:43.185289+0000","last_peered":"2024-03-06T07:28:43.185289+0000","last_clean":"2024-03-06T07:28:43.185289+0000","last_became_active":"2024-03-06T07:28:42.207915+0000","last_became_peered":"2024-03-06T07:28:42.207915+0000","last_unstale":"2024-03-06T07:28:43.185289+0000","last_undegraded":"2024-03-06T07:28:43.185289+0000","last_fullsized":"2024-03-06T07:28:43.185289+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:06.118730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185235+0000","last_change":"2024-03-06T07:28:42.208785+0000","last_active":"2024-03-06T07:28:43.185235+0000","last_peered":"2024-03-06T07:28:43.185235+0000","last_clean":"2024-03-06T07:28:43.185235+0000","last_became_active":"2024-03-06T07:28:42.208507+0000","last_became_peered":"2024-03-06T07:28:42.208507+0000","last_unstale":"2024-03-06T07:28:43.185235+0000","last_undegraded":"2024-03-06T07:28:43.185235+0000","last_fullsized":"2024-03-06T07:28:43.185235+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:48.772495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186378+0000","last_change":"2024-03-06T07:28:42.206049+0000","last_active":"2024-03-06T07:28:43.186378+0000","last_peered":"2024-03-06T07:28:43.186378+0000","last_clean":"2024-03-06T07:28:43.186378+0000","last_became_active":"2024-03-06T07:28:42.205638+0000","last_became_peered":"2024-03-06T07:28:42.205638+0000","last_unstale":"2024-03-06T07:28:43.186378+0000","last_undegraded":"2024-03-06T07:28:43.186378+0000","last_fullsized":"2024-03-06T07:28:43.186378+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:20.581625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.210698+0000","last_change":"2024-03-06T07:28:42.194922+0000","last_active":"2024-03-06T07:28:43.210698+0000","last_peered":"2024-03-06T07:28:43.210698+0000","last_clean":"2024-03-06T07:28:43.210698+0000","last_became_active":"2024-03-06T07:28:42.194525+0000","last_became_peered":"2024-03-06T07:28:42.194525+0000","last_unstale":"2024-03-06T07:28:43.210698+0000","last_undegraded":"2024-03-06T07:28:43.210698+0000","last_fullsized":"2024-03-06T07:28:43.210698+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:41.663661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231439+0000","last_change":"2024-03-06T07:28:42.202688+0000","last_active":"2024-03-06T07:28:43.231439+0000","last_peered":"2024-03-06T07:28:43.231439+0000","last_clean":"2024-03-06T07:28:43.231439+0000","last_became_active":"2024-03-06T07:28:42.202381+0000","last_became_peered":"2024-03-06T07:28:42.202381+0000","last_unstale":"2024-03-06T07:28:43.231439+0000","last_undegraded":"2024-03-06T07:28:43.231439+0000","last_fullsized":"2024-03-06T07:28:43.231439+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:40.428355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"125'4","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186495+0000","last_change":"2024-03-06T07:28:41.189708+0000","last_active":"2024-03-06T07:28:43.186495+0000","last_peered":"2024-03-06T07:28:43.186495+0000","last_clean":"2024-03-06T07:28:43.186495+0000","last_became_active":"2024-03-06T07:28:41.189410+0000","last_became_peered":"2024-03-06T07:28:41.189410+0000","last_unstale":"2024-03-06T07:28:43.186495+0000","last_undegraded":"2024-03-06T07:28:43.186495+0000","last_fullsized":"2024-03-06T07:28:43.186495+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.152894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187315+0000","last_change":"2024-03-06T07:28:42.201786+0000","last_active":"2024-03-06T07:28:43.187315+0000","last_peered":"2024-03-06T07:28:43.187315+0000","last_clean":"2024-03-06T07:28:43.187315+0000","last_became_active":"2024-03-06T07:28:42.201496+0000","last_became_peered":"2024-03-06T07:28:42.201496+0000","last_unstale":"2024-03-06T07:28:43.187315+0000","last_undegraded":"2024-03-06T07:28:43.187315+0000","last_fullsized":"2024-03-06T07:28:43.187315+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:31.141990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229633+0000","last_change":"2024-03-06T07:28:41.181499+0000","last_active":"2024-03-06T07:28:43.229633+0000","last_peered":"2024-03-06T07:28:43.229633+0000","last_clean":"2024-03-06T07:28:43.229633+0000","last_became_active":"2024-03-06T07:28:41.181181+0000","last_became_peered":"2024-03-06T07:28:41.181181+0000","last_unstale":"2024-03-06T07:28:43.229633+0000","last_undegraded":"2024-03-06T07:28:43.229633+0000","last_fullsized":"2024-03-06T07:28:43.229633+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:47.467422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186316+0000","last_change":"2024-03-06T07:28:42.204785+0000","last_active":"2024-03-06T07:28:43.186316+0000","last_peered":"2024-03-06T07:28:43.186316+0000","last_clean":"2024-03-06T07:28:43.186316+0000","last_became_active":"2024-03-06T07:28:42.204475+0000","last_became_peered":"2024-03-06T07:28:42.204475+0000","last_unstale":"2024-03-06T07:28:43.186316+0000","last_undegraded":"2024-03-06T07:28:43.186316+0000","last_fullsized":"2024-03-06T07:28:43.186316+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:34.016386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230279+0000","last_change":"2024-03-06T07:28:41.182845+0000","last_active":"2024-03-06T07:28:43.230279+0000","last_peered":"2024-03-06T07:28:43.230279+0000","last_clean":"2024-03-06T07:28:43.230279+0000","last_became_active":"2024-03-06T07:28:41.182580+0000","last_became_peered":"2024-03-06T07:28:41.182580+0000","last_unstale":"2024-03-06T07:28:43.230279+0000","last_undegraded":"2024-03-06T07:28:43.230279+0000","last_fullsized":"2024-03-06T07:28:43.230279+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:16.971330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186135+0000","last_change":"2024-03-06T07:28:42.201031+0000","last_active":"2024-03-06T07:28:43.186135+0000","last_peered":"2024-03-06T07:28:43.186135+0000","last_clean":"2024-03-06T07:28:43.186135+0000","last_became_active":"2024-03-06T07:28:42.200741+0000","last_became_peered":"2024-03-06T07:28:42.200741+0000","last_unstale":"2024-03-06T07:28:43.186135+0000","last_undegraded":"2024-03-06T07:28:43.186135+0000","last_fullsized":"2024-03-06T07:28:43.186135+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:28.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230351+0000","last_change":"2024-03-06T07:28:41.188986+0000","last_active":"2024-03-06T07:28:43.230351+0000","last_peered":"2024-03-06T07:28:43.230351+0000","last_clean":"2024-03-06T07:28:43.230351+0000","last_became_active":"2024-03-06T07:28:41.188698+0000","last_became_peered":"2024-03-06T07:28:41.188698+0000","last_unstale":"2024-03-06T07:28:43.230351+0000","last_undegraded":"2024-03-06T07:28:43.230351+0000","last_fullsized":"2024-03-06T07:28:43.230351+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:54.887989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231121+0000","last_change":"2024-03-06T07:28:42.205528+0000","last_active":"2024-03-06T07:28:43.231121+0000","last_peered":"2024-03-06T07:28:43.231121+0000","last_clean":"2024-03-06T07:28:43.231121+0000","last_became_active":"2024-03-06T07:28:42.205253+0000","last_became_peered":"2024-03-06T07:28:42.205253+0000","last_unstale":"2024-03-06T07:28:43.231121+0000","last_undegraded":"2024-03-06T07:28:43.231121+0000","last_fullsized":"2024-03-06T07:28:43.231121+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:41.807870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230075+0000","last_change":"2024-03-06T07:28:41.184022+0000","last_active":"2024-03-06T07:28:43.230075+0000","last_peered":"2024-03-06T07:28:43.230075+0000","last_clean":"2024-03-06T07:28:43.230075+0000","last_became_active":"2024-03-06T07:28:41.183745+0000","last_became_peered":"2024-03-06T07:28:41.183745+0000","last_unstale":"2024-03-06T07:28:43.230075+0000","last_undegraded":"2024-03-06T07:28:43.230075+0000","last_fullsized":"2024-03-06T07:28:43.230075+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:06.101713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.210929+0000","last_change":"2024-03-06T07:28:42.199647+0000","last_active":"2024-03-06T07:28:43.210929+0000","last_peered":"2024-03-06T07:28:43.210929+0000","last_clean":"2024-03-06T07:28:43.210929+0000","last_became_active":"2024-03-06T07:28:42.199388+0000","last_became_peered":"2024-03-06T07:28:42.199388+0000","last_unstale":"2024-03-06T07:28:43.210929+0000","last_undegraded":"2024-03-06T07:28:43.210929+0000","last_fullsized":"2024-03-06T07:28:43.210929+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:39.157261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231021+0000","last_change":"2024-03-06T07:28:41.190244+0000","last_active":"2024-03-06T07:28:43.231021+0000","last_peered":"2024-03-06T07:28:43.231021+0000","last_clean":"2024-03-06T07:28:43.231021+0000","last_became_active":"2024-03-06T07:28:41.189955+0000","last_became_peered":"2024-03-06T07:28:41.189955+0000","last_unstale":"2024-03-06T07:28:43.231021+0000","last_undegraded":"2024-03-06T07:28:43.231021+0000","last_fullsized":"2024-03-06T07:28:43.231021+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:40.142277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187262+0000","last_change":"2024-03-06T07:28:42.197581+0000","last_active":"2024-03-06T07:28:43.187262+0000","last_peered":"2024-03-06T07:28:43.187262+0000","last_clean":"2024-03-06T07:28:43.187262+0000","last_became_active":"2024-03-06T07:28:42.197267+0000","last_became_peered":"2024-03-06T07:28:42.197267+0000","last_unstale":"2024-03-06T07:28:43.187262+0000","last_undegraded":"2024-03-06T07:28:43.187262+0000","last_fullsized":"2024-03-06T07:28:43.187262+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:16.292841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.210607+0000","last_change":"2024-03-06T07:28:41.187109+0000","last_active":"2024-03-06T07:28:43.210607+0000","last_peered":"2024-03-06T07:28:43.210607+0000","last_clean":"2024-03-06T07:28:43.210607+0000","last_became_active":"2024-03-06T07:28:41.186804+0000","last_became_peered":"2024-03-06T07:28:41.186804+0000","last_unstale":"2024-03-06T07:28:43.210607+0000","last_undegraded":"2024-03-06T07:28:43.210607+0000","last_fullsized":"2024-03-06T07:28:43.210607+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:40.942258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229869+0000","last_change":"2024-03-06T07:28:42.194251+0000","last_active":"2024-03-06T07:28:43.229869+0000","last_peered":"2024-03-06T07:28:43.229869+0000","last_clean":"2024-03-06T07:28:43.229869+0000","last_became_active":"2024-03-06T07:28:42.193982+0000","last_became_peered":"2024-03-06T07:28:42.193982+0000","last_unstale":"2024-03-06T07:28:43.229869+0000","last_undegraded":"2024-03-06T07:28:43.229869+0000","last_fullsized":"2024-03-06T07:28:43.229869+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:25.239479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229788+0000","last_change":"2024-03-06T07:28:41.188760+0000","last_active":"2024-03-06T07:28:43.229788+0000","last_peered":"2024-03-06T07:28:43.229788+0000","last_clean":"2024-03-06T07:28:43.229788+0000","last_became_active":"2024-03-06T07:28:41.188496+0000","last_became_peered":"2024-03-06T07:28:41.188496+0000","last_unstale":"2024-03-06T07:28:43.229788+0000","last_undegraded":"2024-03-06T07:28:43.229788+0000","last_fullsized":"2024-03-06T07:28:43.229788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:37.457689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185406+0000","last_change":"2024-03-06T07:28:42.201280+0000","last_active":"2024-03-06T07:28:43.185406+0000","last_peered":"2024-03-06T07:28:43.185406+0000","last_clean":"2024-03-06T07:28:43.185406+0000","last_became_active":"2024-03-06T07:28:42.201016+0000","last_became_peered":"2024-03-06T07:28:42.201016+0000","last_unstale":"2024-03-06T07:28:43.185406+0000","last_undegraded":"2024-03-06T07:28:43.185406+0000","last_fullsized":"2024-03-06T07:28:43.185406+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:21.011222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229773+0000","last_change":"2024-03-06T07:28:41.181013+0000","last_active":"2024-03-06T07:28:43.229773+0000","last_peered":"2024-03-06T07:28:43.229773+0000","last_clean":"2024-03-06T07:28:43.229773+0000","last_became_active":"2024-03-06T07:28:41.180722+0000","last_became_peered":"2024-03-06T07:28:41.180722+0000","last_unstale":"2024-03-06T07:28:43.229773+0000","last_undegraded":"2024-03-06T07:28:43.229773+0000","last_fullsized":"2024-03-06T07:28:43.229773+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:51.742798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229697+0000","last_change":"2024-03-06T07:28:42.201348+0000","last_active":"2024-03-06T07:28:43.229697+0000","last_peered":"2024-03-06T07:28:43.229697+0000","last_clean":"2024-03-06T07:28:43.229697+0000","last_became_active":"2024-03-06T07:28:42.200855+0000","last_became_peered":"2024-03-06T07:28:42.200855+0000","last_unstale":"2024-03-06T07:28:43.229697+0000","last_undegraded":"2024-03-06T07:28:43.229697+0000","last_fullsized":"2024-03-06T07:28:43.229697+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:25.239479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231432+0000","last_change":"2024-03-06T07:28:41.188094+0000","last_active":"2024-03-06T07:28:43.231432+0000","last_peered":"2024-03-06T07:28:43.231432+0000","last_clean":"2024-03-06T07:28:43.231432+0000","last_became_active":"2024-03-06T07:28:41.187821+0000","last_became_peered":"2024-03-06T07:28:41.187821+0000","last_unstale":"2024-03-06T07:28:43.231432+0000","last_undegraded":"2024-03-06T07:28:43.231432+0000","last_fullsized":"2024-03-06T07:28:43.231432+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:29.488206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186480+0000","last_change":"2024-03-06T07:28:42.205367+0000","last_active":"2024-03-06T07:28:43.186480+0000","last_peered":"2024-03-06T07:28:43.186480+0000","last_clean":"2024-03-06T07:28:43.186480+0000","last_became_active":"2024-03-06T07:28:42.205067+0000","last_became_peered":"2024-03-06T07:28:42.205067+0000","last_unstale":"2024-03-06T07:28:43.186480+0000","last_undegraded":"2024-03-06T07:28:43.186480+0000","last_fullsized":"2024-03-06T07:28:43.186480+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:33.890212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"125'3","reported_seq":17,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229448+0000","last_change":"2024-03-06T07:28:41.184618+0000","last_active":"2024-03-06T07:28:43.229448+0000","last_peered":"2024-03-06T07:28:43.229448+0000","last_clean":"2024-03-06T07:28:43.229448+0000","last_became_active":"2024-03-06T07:28:41.184327+0000","last_became_peered":"2024-03-06T07:28:41.184327+0000","last_unstale":"2024-03-06T07:28:43.229448+0000","last_undegraded":"2024-03-06T07:28:43.229448+0000","last_fullsized":"2024-03-06T07:28:43.229448+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:41.989980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186283+0000","last_change":"2024-03-06T07:28:42.207667+0000","last_active":"2024-03-06T07:28:43.186283+0000","last_peered":"2024-03-06T07:28:43.186283+0000","last_clean":"2024-03-06T07:28:43.186283+0000","last_became_active":"2024-03-06T07:28:42.207229+0000","last_became_peered":"2024-03-06T07:28:42.207229+0000","last_unstale":"2024-03-06T07:28:43.186283+0000","last_undegraded":"2024-03-06T07:28:43.186283+0000","last_fullsized":"2024-03-06T07:28:43.186283+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:17.976182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"125'6","reported_seq":20,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187187+0000","last_change":"2024-03-06T07:28:41.189432+0000","last_active":"2024-03-06T07:28:43.187187+0000","last_peered":"2024-03-06T07:28:43.187187+0000","last_clean":"2024-03-06T07:28:43.187187+0000","last_became_active":"2024-03-06T07:28:41.188979+0000","last_became_peered":"2024-03-06T07:28:41.188979+0000","last_unstale":"2024-03-06T07:28:43.187187+0000","last_undegraded":"2024-03-06T07:28:43.187187+0000","last_fullsized":"2024-03-06T07:28:43.187187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:17.353313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185483+0000","last_change":"2024-03-06T07:28:42.199408+0000","last_active":"2024-03-06T07:28:43.185483+0000","last_peered":"2024-03-06T07:28:43.185483+0000","last_clean":"2024-03-06T07:28:43.185483+0000","last_became_active":"2024-03-06T07:28:42.199114+0000","last_became_peered":"2024-03-06T07:28:42.199114+0000","last_unstale":"2024-03-06T07:28:43.185483+0000","last_undegraded":"2024-03-06T07:28:43.185483+0000","last_fullsized":"2024-03-06T07:28:43.185483+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:27.341143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185309+0000","last_change":"2024-03-06T07:28:41.180122+0000","last_active":"2024-03-06T07:28:43.185309+0000","last_peered":"2024-03-06T07:28:43.185309+0000","last_clean":"2024-03-06T07:28:43.185309+0000","last_became_active":"2024-03-06T07:28:41.179857+0000","last_became_peered":"2024-03-06T07:28:41.179857+0000","last_unstale":"2024-03-06T07:28:43.185309+0000","last_undegraded":"2024-03-06T07:28:43.185309+0000","last_fullsized":"2024-03-06T07:28:43.185309+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:53.487575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229598+0000","last_change":"2024-03-06T07:28:42.191120+0000","last_active":"2024-03-06T07:28:43.229598+0000","last_peered":"2024-03-06T07:28:43.229598+0000","last_clean":"2024-03-06T07:28:43.229598+0000","last_became_active":"2024-03-06T07:28:42.190540+0000","last_became_peered":"2024-03-06T07:28:42.190540+0000","last_unstale":"2024-03-06T07:28:43.229598+0000","last_undegraded":"2024-03-06T07:28:43.229598+0000","last_fullsized":"2024-03-06T07:28:43.229598+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:45.395510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229952+0000","last_change":"2024-03-06T07:28:41.190545+0000","last_active":"2024-03-06T07:28:43.229952+0000","last_peered":"2024-03-06T07:28:43.229952+0000","last_clean":"2024-03-06T07:28:43.229952+0000","last_became_active":"2024-03-06T07:28:41.190261+0000","last_became_peered":"2024-03-06T07:28:41.190261+0000","last_unstale":"2024-03-06T07:28:43.229952+0000","last_undegraded":"2024-03-06T07:28:43.229952+0000","last_fullsized":"2024-03-06T07:28:43.229952+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.795431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187108+0000","last_change":"2024-03-06T07:28:42.196480+0000","last_active":"2024-03-06T07:28:43.187108+0000","last_peered":"2024-03-06T07:28:43.187108+0000","last_clean":"2024-03-06T07:28:43.187108+0000","last_became_active":"2024-03-06T07:28:42.196140+0000","last_became_peered":"2024-03-06T07:28:42.196140+0000","last_unstale":"2024-03-06T07:28:43.187108+0000","last_undegraded":"2024-03-06T07:28:43.187108+0000","last_fullsized":"2024-03-06T07:28:43.187108+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:38.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"125'8","reported_seq":22,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187278+0000","last_change":"2024-03-06T07:28:41.189229+0000","last_active":"2024-03-06T07:28:43.187278+0000","last_peered":"2024-03-06T07:28:43.187278+0000","last_clean":"2024-03-06T07:28:43.187278+0000","last_became_active":"2024-03-06T07:28:41.188946+0000","last_became_peered":"2024-03-06T07:28:41.188946+0000","last_unstale":"2024-03-06T07:28:43.187278+0000","last_undegraded":"2024-03-06T07:28:43.187278+0000","last_fullsized":"2024-03-06T07:28:43.187278+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:44.390658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.187075+0000","last_change":"2024-03-06T07:28:42.202597+0000","last_active":"2024-03-06T07:28:43.187075+0000","last_peered":"2024-03-06T07:28:43.187075+0000","last_clean":"2024-03-06T07:28:43.187075+0000","last_became_active":"2024-03-06T07:28:42.202218+0000","last_became_peered":"2024-03-06T07:28:42.202218+0000","last_unstale":"2024-03-06T07:28:43.187075+0000","last_undegraded":"2024-03-06T07:28:43.187075+0000","last_fullsized":"2024-03-06T07:28:43.187075+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:34.502047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229834+0000","last_change":"2024-03-06T07:28:41.187421+0000","last_active":"2024-03-06T07:28:43.229834+0000","last_peered":"2024-03-06T07:28:43.229834+0000","last_clean":"2024-03-06T07:28:43.229834+0000","last_became_active":"2024-03-06T07:28:41.187155+0000","last_became_peered":"2024-03-06T07:28:41.187155+0000","last_unstale":"2024-03-06T07:28:43.229834+0000","last_undegraded":"2024-03-06T07:28:43.229834+0000","last_fullsized":"2024-03-06T07:28:43.229834+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:46.911657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229725+0000","last_change":"2024-03-06T07:28:42.198642+0000","last_active":"2024-03-06T07:28:43.229725+0000","last_peered":"2024-03-06T07:28:43.229725+0000","last_clean":"2024-03-06T07:28:43.229725+0000","last_became_active":"2024-03-06T07:28:42.198387+0000","last_became_peered":"2024-03-06T07:28:42.198387+0000","last_unstale":"2024-03-06T07:28:43.229725+0000","last_undegraded":"2024-03-06T07:28:43.229725+0000","last_fullsized":"2024-03-06T07:28:43.229725+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:41.664169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229772+0000","last_change":"2024-03-06T07:28:41.186947+0000","last_active":"2024-03-06T07:28:43.229772+0000","last_peered":"2024-03-06T07:28:43.229772+0000","last_clean":"2024-03-06T07:28:43.229772+0000","last_became_active":"2024-03-06T07:28:41.186613+0000","last_became_peered":"2024-03-06T07:28:41.186613+0000","last_unstale":"2024-03-06T07:28:43.229772+0000","last_undegraded":"2024-03-06T07:28:43.229772+0000","last_fullsized":"2024-03-06T07:28:43.229772+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:59.252934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231372+0000","last_change":"2024-03-06T07:28:42.205342+0000","last_active":"2024-03-06T07:28:43.231372+0000","last_peered":"2024-03-06T07:28:43.231372+0000","last_clean":"2024-03-06T07:28:43.231372+0000","last_became_active":"2024-03-06T07:28:42.205069+0000","last_became_peered":"2024-03-06T07:28:42.205069+0000","last_unstale":"2024-03-06T07:28:43.231372+0000","last_undegraded":"2024-03-06T07:28:43.231372+0000","last_fullsized":"2024-03-06T07:28:43.231372+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:40.136261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.210961+0000","last_change":"2024-03-06T07:28:41.186910+0000","last_active":"2024-03-06T07:28:43.210961+0000","last_peered":"2024-03-06T07:28:43.210961+0000","last_clean":"2024-03-06T07:28:43.210961+0000","last_became_active":"2024-03-06T07:28:41.186583+0000","last_became_peered":"2024-03-06T07:28:41.186583+0000","last_unstale":"2024-03-06T07:28:43.210961+0000","last_undegraded":"2024-03-06T07:28:43.210961+0000","last_fullsized":"2024-03-06T07:28:43.210961+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:22.624787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230509+0000","last_change":"2024-03-06T07:28:42.197931+0000","last_active":"2024-03-06T07:28:43.230509+0000","last_peered":"2024-03-06T07:28:43.230509+0000","last_clean":"2024-03-06T07:28:43.230509+0000","last_became_active":"2024-03-06T07:28:42.197678+0000","last_became_peered":"2024-03-06T07:28:42.197678+0000","last_unstale":"2024-03-06T07:28:43.230509+0000","last_undegraded":"2024-03-06T07:28:43.230509+0000","last_fullsized":"2024-03-06T07:28:43.230509+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:29.186709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229487+0000","last_change":"2024-03-06T07:28:41.179940+0000","last_active":"2024-03-06T07:28:43.229487+0000","last_peered":"2024-03-06T07:28:43.229487+0000","last_clean":"2024-03-06T07:28:43.229487+0000","last_became_active":"2024-03-06T07:28:41.179652+0000","last_became_peered":"2024-03-06T07:28:41.179652+0000","last_unstale":"2024-03-06T07:28:43.229487+0000","last_undegraded":"2024-03-06T07:28:43.229487+0000","last_fullsized":"2024-03-06T07:28:43.229487+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:50.816471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185297+0000","last_change":"2024-03-06T07:28:42.206682+0000","last_active":"2024-03-06T07:28:43.185297+0000","last_peered":"2024-03-06T07:28:43.185297+0000","last_clean":"2024-03-06T07:28:43.185297+0000","last_became_active":"2024-03-06T07:28:42.206227+0000","last_became_peered":"2024-03-06T07:28:42.206227+0000","last_unstale":"2024-03-06T07:28:43.185297+0000","last_undegraded":"2024-03-06T07:28:43.185297+0000","last_fullsized":"2024-03-06T07:28:43.185297+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:42.776494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229472+0000","last_change":"2024-03-06T07:28:41.180697+0000","last_active":"2024-03-06T07:28:43.229472+0000","last_peered":"2024-03-06T07:28:43.229472+0000","last_clean":"2024-03-06T07:28:43.229472+0000","last_became_active":"2024-03-06T07:28:41.180401+0000","last_became_peered":"2024-03-06T07:28:41.180401+0000","last_unstale":"2024-03-06T07:28:43.229472+0000","last_undegraded":"2024-03-06T07:28:43.229472+0000","last_fullsized":"2024-03-06T07:28:43.229472+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:52.790131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.211026+0000","last_change":"2024-03-06T07:28:42.198890+0000","last_active":"2024-03-06T07:28:43.211026+0000","last_peered":"2024-03-06T07:28:43.211026+0000","last_clean":"2024-03-06T07:28:43.211026+0000","last_became_active":"2024-03-06T07:28:42.198625+0000","last_became_peered":"2024-03-06T07:28:42.198625+0000","last_unstale":"2024-03-06T07:28:43.211026+0000","last_undegraded":"2024-03-06T07:28:43.211026+0000","last_fullsized":"2024-03-06T07:28:43.211026+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:18.472201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231119+0000","last_change":"2024-03-06T07:28:41.186902+0000","last_active":"2024-03-06T07:28:43.231119+0000","last_peered":"2024-03-06T07:28:43.231119+0000","last_clean":"2024-03-06T07:28:43.231119+0000","last_became_active":"2024-03-06T07:28:41.186604+0000","last_became_peered":"2024-03-06T07:28:41.186604+0000","last_unstale":"2024-03-06T07:28:43.231119+0000","last_undegraded":"2024-03-06T07:28:43.231119+0000","last_fullsized":"2024-03-06T07:28:43.231119+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.381492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229738+0000","last_change":"2024-03-06T07:28:42.191360+0000","last_active":"2024-03-06T07:28:43.229738+0000","last_peered":"2024-03-06T07:28:43.229738+0000","last_clean":"2024-03-06T07:28:43.229738+0000","last_became_active":"2024-03-06T07:28:42.190847+0000","last_became_peered":"2024-03-06T07:28:42.190847+0000","last_unstale":"2024-03-06T07:28:43.229738+0000","last_undegraded":"2024-03-06T07:28:43.229738+0000","last_fullsized":"2024-03-06T07:28:43.229738+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:48.581098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230823+0000","last_change":"2024-03-06T07:28:41.184219+0000","last_active":"2024-03-06T07:28:43.230823+0000","last_peered":"2024-03-06T07:28:43.230823+0000","last_clean":"2024-03-06T07:28:43.230823+0000","last_became_active":"2024-03-06T07:28:41.183964+0000","last_became_peered":"2024-03-06T07:28:41.183964+0000","last_unstale":"2024-03-06T07:28:43.230823+0000","last_undegraded":"2024-03-06T07:28:43.230823+0000","last_fullsized":"2024-03-06T07:28:43.230823+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:56.076450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185951+0000","last_change":"2024-03-06T07:28:42.205926+0000","last_active":"2024-03-06T07:28:43.185951+0000","last_peered":"2024-03-06T07:28:43.185951+0000","last_clean":"2024-03-06T07:28:43.185951+0000","last_became_active":"2024-03-06T07:28:42.204385+0000","last_became_peered":"2024-03-06T07:28:42.204385+0000","last_unstale":"2024-03-06T07:28:43.185951+0000","last_undegraded":"2024-03-06T07:28:43.185951+0000","last_fullsized":"2024-03-06T07:28:43.185951+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:25.132217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229636+0000","last_change":"2024-03-06T07:28:41.189234+0000","last_active":"2024-03-06T07:28:43.229636+0000","last_peered":"2024-03-06T07:28:43.229636+0000","last_clean":"2024-03-06T07:28:43.229636+0000","last_became_active":"2024-03-06T07:28:41.188896+0000","last_became_peered":"2024-03-06T07:28:41.188896+0000","last_unstale":"2024-03-06T07:28:43.229636+0000","last_undegraded":"2024-03-06T07:28:43.229636+0000","last_fullsized":"2024-03-06T07:28:43.229636+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:33.011534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.211124+0000","last_change":"2024-03-06T07:28:42.198642+0000","last_active":"2024-03-06T07:28:43.211124+0000","last_peered":"2024-03-06T07:28:43.211124+0000","last_clean":"2024-03-06T07:28:43.211124+0000","last_became_active":"2024-03-06T07:28:42.198337+0000","last_became_peered":"2024-03-06T07:28:42.198337+0000","last_unstale":"2024-03-06T07:28:43.211124+0000","last_undegraded":"2024-03-06T07:28:43.211124+0000","last_fullsized":"2024-03-06T07:28:43.211124+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:16.292841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185544+0000","last_change":"2024-03-06T07:28:41.184071+0000","last_active":"2024-03-06T07:28:43.185544+0000","last_peered":"2024-03-06T07:28:43.185544+0000","last_clean":"2024-03-06T07:28:43.185544+0000","last_became_active":"2024-03-06T07:28:41.180788+0000","last_became_peered":"2024-03-06T07:28:41.180788+0000","last_unstale":"2024-03-06T07:28:43.185544+0000","last_undegraded":"2024-03-06T07:28:43.185544+0000","last_fullsized":"2024-03-06T07:28:43.185544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:09.354448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185570+0000","last_change":"2024-03-06T07:28:42.204169+0000","last_active":"2024-03-06T07:28:43.185570+0000","last_peered":"2024-03-06T07:28:43.185570+0000","last_clean":"2024-03-06T07:28:43.185570+0000","last_became_active":"2024-03-06T07:28:42.203913+0000","last_became_peered":"2024-03-06T07:28:42.203913+0000","last_unstale":"2024-03-06T07:28:43.185570+0000","last_undegraded":"2024-03-06T07:28:43.185570+0000","last_fullsized":"2024-03-06T07:28:43.185570+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:05.177446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"126'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186134+0000","last_change":"2024-03-06T07:28:41.186801+0000","last_active":"2024-03-06T07:28:43.186134+0000","last_peered":"2024-03-06T07:28:43.186134+0000","last_clean":"2024-03-06T07:28:43.186134+0000","last_became_active":"2024-03-06T07:28:41.186532+0000","last_became_peered":"2024-03-06T07:28:41.186532+0000","last_unstale":"2024-03-06T07:28:43.186134+0000","last_undegraded":"2024-03-06T07:28:43.186134+0000","last_fullsized":"2024-03-06T07:28:43.186134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:57.961981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230476+0000","last_change":"2024-03-06T07:28:42.197515+0000","last_active":"2024-03-06T07:28:43.230476+0000","last_peered":"2024-03-06T07:28:43.230476+0000","last_clean":"2024-03-06T07:28:43.230476+0000","last_became_active":"2024-03-06T07:28:42.197252+0000","last_became_peered":"2024-03-06T07:28:42.197252+0000","last_unstale":"2024-03-06T07:28:43.230476+0000","last_undegraded":"2024-03-06T07:28:43.230476+0000","last_fullsized":"2024-03-06T07:28:43.230476+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:41.663661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230381+0000","last_change":"2024-03-06T07:28:41.192211+0000","last_active":"2024-03-06T07:28:43.230381+0000","last_peered":"2024-03-06T07:28:43.230381+0000","last_clean":"2024-03-06T07:28:43.230381+0000","last_became_active":"2024-03-06T07:28:41.191920+0000","last_became_peered":"2024-03-06T07:28:41.191920+0000","last_unstale":"2024-03-06T07:28:43.230381+0000","last_undegraded":"2024-03-06T07:28:43.230381+0000","last_fullsized":"2024-03-06T07:28:43.230381+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:40.554076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.211008+0000","last_change":"2024-03-06T07:28:42.200085+0000","last_active":"2024-03-06T07:28:43.211008+0000","last_peered":"2024-03-06T07:28:43.211008+0000","last_clean":"2024-03-06T07:28:43.211008+0000","last_became_active":"2024-03-06T07:28:42.199801+0000","last_became_peered":"2024-03-06T07:28:42.199801+0000","last_unstale":"2024-03-06T07:28:43.211008+0000","last_undegraded":"2024-03-06T07:28:43.211008+0000","last_fullsized":"2024-03-06T07:28:43.211008+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:38.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230023+0000","last_change":"2024-03-06T07:28:41.191241+0000","last_active":"2024-03-06T07:28:43.230023+0000","last_peered":"2024-03-06T07:28:43.230023+0000","last_clean":"2024-03-06T07:28:43.230023+0000","last_became_active":"2024-03-06T07:28:41.190929+0000","last_became_peered":"2024-03-06T07:28:41.190929+0000","last_unstale":"2024-03-06T07:28:43.230023+0000","last_undegraded":"2024-03-06T07:28:43.230023+0000","last_fullsized":"2024-03-06T07:28:43.230023+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:30.250947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186271+0000","last_change":"2024-03-06T07:28:42.204997+0000","last_active":"2024-03-06T07:28:43.186271+0000","last_peered":"2024-03-06T07:28:43.186271+0000","last_clean":"2024-03-06T07:28:43.186271+0000","last_became_active":"2024-03-06T07:28:42.204727+0000","last_became_peered":"2024-03-06T07:28:42.204727+0000","last_unstale":"2024-03-06T07:28:43.186271+0000","last_undegraded":"2024-03-06T07:28:43.186271+0000","last_fullsized":"2024-03-06T07:28:43.186271+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:31.255799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230122+0000","last_change":"2024-03-06T07:28:41.189777+0000","last_active":"2024-03-06T07:28:43.230122+0000","last_peered":"2024-03-06T07:28:43.230122+0000","last_clean":"2024-03-06T07:28:43.230122+0000","last_became_active":"2024-03-06T07:28:41.189526+0000","last_became_peered":"2024-03-06T07:28:41.189526+0000","last_unstale":"2024-03-06T07:28:43.230122+0000","last_undegraded":"2024-03-06T07:28:43.230122+0000","last_fullsized":"2024-03-06T07:28:43.230122+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:18.311490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229620+0000","last_change":"2024-03-06T07:28:42.203103+0000","last_active":"2024-03-06T07:28:43.229620+0000","last_peered":"2024-03-06T07:28:43.229620+0000","last_clean":"2024-03-06T07:28:43.229620+0000","last_became_active":"2024-03-06T07:28:42.202218+0000","last_became_peered":"2024-03-06T07:28:42.202218+0000","last_unstale":"2024-03-06T07:28:43.229620+0000","last_undegraded":"2024-03-06T07:28:43.229620+0000","last_fullsized":"2024-03-06T07:28:43.229620+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:19.967407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186016+0000","last_change":"2024-03-06T07:28:41.190665+0000","last_active":"2024-03-06T07:28:43.186016+0000","last_peered":"2024-03-06T07:28:43.186016+0000","last_clean":"2024-03-06T07:28:43.186016+0000","last_became_active":"2024-03-06T07:28:41.190315+0000","last_became_peered":"2024-03-06T07:28:41.190315+0000","last_unstale":"2024-03-06T07:28:43.186016+0000","last_undegraded":"2024-03-06T07:28:43.186016+0000","last_fullsized":"2024-03-06T07:28:43.186016+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:52.502533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.231047+0000","last_change":"2024-03-06T07:28:42.203964+0000","last_active":"2024-03-06T07:28:43.231047+0000","last_peered":"2024-03-06T07:28:43.231047+0000","last_clean":"2024-03-06T07:28:43.231047+0000","last_became_active":"2024-03-06T07:28:42.203701+0000","last_became_peered":"2024-03-06T07:28:42.203701+0000","last_unstale":"2024-03-06T07:28:43.231047+0000","last_undegraded":"2024-03-06T07:28:43.231047+0000","last_fullsized":"2024-03-06T07:28:43.231047+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:14.957901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230877+0000","last_change":"2024-03-06T07:28:41.187024+0000","last_active":"2024-03-06T07:28:43.230877+0000","last_peered":"2024-03-06T07:28:43.230877+0000","last_clean":"2024-03-06T07:28:43.230877+0000","last_became_active":"2024-03-06T07:28:41.186724+0000","last_became_peered":"2024-03-06T07:28:41.186724+0000","last_unstale":"2024-03-06T07:28:43.230877+0000","last_undegraded":"2024-03-06T07:28:43.230877+0000","last_fullsized":"2024-03-06T07:28:43.230877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:15.748580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.230006+0000","last_change":"2024-03-06T07:28:42.197121+0000","last_active":"2024-03-06T07:28:43.230006+0000","last_peered":"2024-03-06T07:28:43.230006+0000","last_clean":"2024-03-06T07:28:43.230006+0000","last_became_active":"2024-03-06T07:28:42.196629+0000","last_became_peered":"2024-03-06T07:28:42.196629+0000","last_unstale":"2024-03-06T07:28:43.230006+0000","last_undegraded":"2024-03-06T07:28:43.230006+0000","last_fullsized":"2024-03-06T07:28:43.230006+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:41.664169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.229729+0000","last_change":"2024-03-06T07:28:41.183475+0000","last_active":"2024-03-06T07:28:43.229729+0000","last_peered":"2024-03-06T07:28:43.229729+0000","last_clean":"2024-03-06T07:28:43.229729+0000","last_became_active":"2024-03-06T07:28:41.183229+0000","last_became_peered":"2024-03-06T07:28:41.183229+0000","last_unstale":"2024-03-06T07:28:43.229729+0000","last_undegraded":"2024-03-06T07:28:43.229729+0000","last_fullsized":"2024-03-06T07:28:43.229729+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:19.576773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185837+0000","last_change":"2024-03-06T07:28:42.203579+0000","last_active":"2024-03-06T07:28:43.185837+0000","last_peered":"2024-03-06T07:28:43.185837+0000","last_clean":"2024-03-06T07:28:43.185837+0000","last_became_active":"2024-03-06T07:28:42.203180+0000","last_became_peered":"2024-03-06T07:28:42.203180+0000","last_unstale":"2024-03-06T07:28:43.185837+0000","last_undegraded":"2024-03-06T07:28:43.185837+0000","last_fullsized":"2024-03-06T07:28:43.185837+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.167252+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.167252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:38.418363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.186155+0000","last_change":"2024-03-06T07:28:41.190743+0000","last_active":"2024-03-06T07:28:43.186155+0000","last_peered":"2024-03-06T07:28:43.186155+0000","last_clean":"2024-03-06T07:28:43.186155+0000","last_became_active":"2024-03-06T07:28:41.190446+0000","last_became_peered":"2024-03-06T07:28:41.190446+0000","last_unstale":"2024-03-06T07:28:43.186155+0000","last_undegraded":"2024-03-06T07:28:43.186155+0000","last_fullsized":"2024-03-06T07:28:43.186155+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.162400+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.162400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:52.593136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":370,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.185652+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:28:43.185652+0000","last_peered":"2024-03-06T07:28:43.185652+0000","last_clean":"2024-03-06T07:28:43.185652+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:28:43.185652+0000","last_undegraded":"2024-03-06T07:28:43.185652+0000","last_fullsized":"2024-03-06T07:28:43.185652+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41644,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330196,"statfs":{"total":96636764160,"available":96594120704,"internally_reserved":0,"allocated":2191360,"data_stored":1276834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41048,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330792,"statfs":{"total":96636764160,"available":96594731008,"internally_reserved":0,"allocated":1585152,"data_stored":685828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574936,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41048,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330792,"statfs":{"total":96636764160,"available":96594731008,"internally_reserved":0,"allocated":1581056,"data_stored":684328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1576960,"data_stored":684286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1581056,"data_stored":684376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574937,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1581056,"data_stored":684832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41036,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330804,"statfs":{"total":96636764160,"available":96594743296,"internally_reserved":0,"allocated":1572864,"data_stored":684216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574938,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2166784,"data_stored":1274584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.630 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:45.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:45.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.772+0000 7efc9e60f640 1 Processor -- start 2024-03-06T07:28:45.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.772+0000 7efc9e60f640 1 -- start start 2024-03-06T07:28:45.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9e60f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815e0e0 0x7efc9806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9e60f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc9806b6c0 0x7efc980b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9e60f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc98161bd0 con 0x7efc9815dd00 2024-03-06T07:28:45.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9e60f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc98070220 con 0x7efc9806b6c0 2024-03-06T07:28:45.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9e60f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc98161e90 con 0x7efc9815e0e0 2024-03-06T07:28:45.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9cbac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc9806b6c0 0x7efc980b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc97fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efc9815dd00 0x7efc9806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41238/0 (socket says 172.21.15.122:41238) 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc97fff640 1 -- 172.21.15.122:0/4067506158 learned_addr learned my addr 172.21.15.122:0/4067506158 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc977fe640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815e0e0 0x7efc9806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9cbac640 1 -- 172.21.15.122:0/4067506158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815e0e0 msgr2=0x7efc9806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9cbac640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815e0e0 0x7efc9806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9cbac640 1 -- 172.21.15.122:0/4067506158 >> v1:172.21.15.122:6789/0 conn(0x7efc9815dd00 legacy=0x7efc9806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc9cbac640 1 -- 172.21.15.122:0/4067506158 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98162170 con 0x7efc9806b6c0 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc96ffd640 1 -- 172.21.15.122:0/4067506158 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424149338 0 0) 0x7efc98161bd0 con 0x7efc9815dd00 2024-03-06T07:28:45.776 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.773+0000 7efc977fe640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815e0e0 0x7efc9806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.774+0000 7efc9cbac640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9806b6c0 0x7efc980b42d0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efc90009d90 tx=0x7efc9002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3028f7b8e29bfb2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.774+0000 7efc96ffd640 1 -- 172.21.15.122:0/4067506158 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90039070 con 0x7efc9806b6c0 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.774+0000 7efc96ffd640 1 -- 172.21.15.122:0/4067506158 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9002cc50 con 0x7efc9806b6c0 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.774+0000 7efc96ffd640 1 -- 172.21.15.122:0/4067506158 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90034440 con 0x7efc9806b6c0 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 -- 172.21.15.122:0/4067506158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9806b6c0 msgr2=0x7efc980b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9806b6c0 0x7efc980b42d0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efc90009d90 tx=0x7efc9002bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 -- 172.21.15.122:0/4067506158 shutdown_connections 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9806b6c0 0x7efc980b42d0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 --2- 172.21.15.122:0/4067506158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815e0e0 0x7efc9806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 -- 172.21.15.122:0/4067506158 >> 172.21.15.122:0/4067506158 conn(0x7efc980b2660 msgr2=0x7efc980b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 -- 172.21.15.122:0/4067506158 shutdown_connections 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.775+0000 7efc9e60f640 1 -- 172.21.15.122:0/4067506158 wait complete. 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.776+0000 7efc9e60f640 1 Processor -- start 2024-03-06T07:28:45.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.776+0000 7efc9e60f640 1 -- start start 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc9806b6c0 0x7efc980c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 0x7efc981f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815bc30 0x7efc981606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980b0180 con 0x7efc9815dd00 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980b0000 con 0x7efc9806b6c0 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc980b0300 con 0x7efc9815bc30 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 0x7efc981f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 0x7efc981f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39432/0 (socket says 172.21.15.122:39432) 2024-03-06T07:28:45.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 -- 172.21.15.122:0/924213595 learned_addr learned my addr 172.21.15.122:0/924213595 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 -- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815bc30 msgr2=0x7efc981606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815bc30 0x7efc981606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 -- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc9806b6c0 msgr2=0x7efc980c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc97fff640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc9806b6c0 0x7efc980c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc9806b6c0 0x7efc980c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 -- 172.21.15.122:0/924213595 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc980c5f90 con 0x7efc9815dd00 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc977fe640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 0x7efc981f4130 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00c7f0 tx=0x7efc8c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c015070 con 0x7efc9815dd00 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c0079d0 con 0x7efc9815dd00 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc980c6270 con 0x7efc9815dd00 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c007da0 con 0x7efc9815dd00 2024-03-06T07:28:45.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.777+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc980c6830 con 0x7efc9815dd00 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.778+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc98073390 con 0x7efc9815dd00 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.779+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc8c018070 con 0x7efc9815dd00 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.779+0000 7efc94ff9640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc8406b4d0 0x7efc8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.779+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc8c08a170 con 0x7efc9815dd00 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.779+0000 7efc97fff640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc8406b4d0 0x7efc8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.780+0000 7efc97fff640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc8406b4d0 0x7efc8406d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efc88000f60 tx=0x7efc88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:45.788+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc8c058ff0 con 0x7efc9815dd00 2024-03-06T07:28:46.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.033+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efc98068ff0 con 0x7efc9815dd00 2024-03-06T07:28:46.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.035+0000 7efc94ff9640 1 -- 172.21.15.122:0/924213595 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7efc8c0588f0 con 0x7efc9815dd00 2024-03-06T07:28:46.037 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:28:46.037 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:28:46.037 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:28:46.037 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:28:46.037 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:46.038 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:28:46.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc8406b4d0 msgr2=0x7efc8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc8406b4d0 0x7efc8406d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efc88000f60 tx=0x7efc88009040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 msgr2=0x7efc981f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 0x7efc981f4130 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00c7f0 tx=0x7efc8c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 shutdown_connections 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc8406b4d0 0x7efc8406d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc9815bc30 0x7efc981606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc9815dd00 0x7efc981f4130 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 --2- 172.21.15.122:0/924213595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc9806b6c0 0x7efc980c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 >> 172.21.15.122:0/924213595 conn(0x7efc980b2660 msgr2=0x7efc980b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 shutdown_connections 2024-03-06T07:28:46.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.037+0000 7efc9e60f640 1 -- 172.21.15.122:0/924213595 wait complete. 2024-03-06T07:28:46.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:46.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 Processor -- start 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 -- start start 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 0x7f4e100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e100b3430 con 0x7f4e1015dfe0 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e100d2c50 con 0x7f4e100d22b0 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.148+0000 7f4e16d5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100b31e0 con 0x7f4e100b41f0 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e0ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 0x7f4e100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:33070/0 (socket says 172.21.15.122:33070) 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e0ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 0x7f4e100d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:50414/0 (socket says 172.21.15.122:50414) 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 -- 172.21.15.122:0/871777571 learned_addr learned my addr 172.21.15.122:0/871777571 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 -- 172.21.15.122:0/871777571 >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 msgr2=0x7f4e100d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 --2- 172.21.15.122:0/871777571 >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 0x7f4e100d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 -- 172.21.15.122:0/871777571 >> v1:172.21.15.169:6789/0 conn(0x7f4e1015dfe0 legacy=0x7f4e100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e152f9640 1 -- 172.21.15.122:0/871777571 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100b3710 con 0x7f4e100b41f0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e0f7fe640 1 -- 172.21.15.122:0/871777571 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627133442 0 0) 0x7f4e100b3430 con 0x7f4e1015dfe0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.149+0000 7f4e0ffff640 1 --2- 172.21.15.122:0/871777571 >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 0x7f4e100d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.150+0000 7f4e152f9640 1 --2- 172.21.15.122:0/871777571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100b4680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009e50 tx=0x7f4e0802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=814b7bcf00465e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e0f7fe640 1 -- 172.21.15.122:0/871777571 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0802cb00 con 0x7f4e100b41f0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e0f7fe640 1 -- 172.21.15.122:0/871777571 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0802cca0 con 0x7f4e100b41f0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e0f7fe640 1 -- 172.21.15.122:0/871777571 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08034610 con 0x7f4e100b41f0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 -- 172.21.15.122:0/871777571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 msgr2=0x7f4e100b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/871777571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100b4680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009e50 tx=0x7f4e0802bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 -- 172.21.15.122:0/871777571 shutdown_connections 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/871777571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100b4680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/871777571 >> v2:172.21.15.169:3300/0 conn(0x7f4e100d22b0 0x7f4e100d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 -- 172.21.15.122:0/871777571 >> 172.21.15.122:0/871777571 conn(0x7f4e100d3cd0 msgr2=0x7f4e100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 -- 172.21.15.122:0/871777571 shutdown_connections 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.151+0000 7f4e16d5c640 1 -- 172.21.15.122:0/871777571 wait complete. 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.152+0000 7f4e16d5c640 1 Processor -- start 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.152+0000 7f4e16d5c640 1 -- start start 2024-03-06T07:28:46.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e16d5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e16d5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 0x7f4e101f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e16d5c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e1015dfe0 0x7f4e101600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e16d5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100cfa00 con 0x7f4e100d22b0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e16d5c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100b3d60 con 0x7f4e1015dfe0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e16d5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100d1c70 con 0x7f4e100b41f0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e0ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 0x7f4e101f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e0ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 0x7f4e101f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39442/0 (socket says 172.21.15.122:39442) 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e0ffff640 1 -- 172.21.15.122:0/1277729733 learned_addr learned my addr 172.21.15.122:0/1277729733 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e152f9640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e1015dfe0 0x7f4e101600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.153+0000 7f4e14af8640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0ffff640 1 -- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 msgr2=0x7f4e100ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0ffff640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0ffff640 1 -- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e1015dfe0 msgr2=0x7f4e101600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0ffff640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e1015dfe0 0x7f4e101600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0ffff640 1 -- 172.21.15.122:0/1277729733 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100cbfc0 con 0x7f4e100d22b0 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e152f9640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e1015dfe0 0x7f4e101600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0ffff640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 0x7f4e101f3140 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00ebe0 tx=0x7f4dfc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc017070 con 0x7f4e100d22b0 2024-03-06T07:28:46.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc00cc30 con 0x7f4e100d22b0 2024-03-06T07:28:46.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e100cc260 con 0x7f4e100d22b0 2024-03-06T07:28:46.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc013470 con 0x7f4e100d22b0 2024-03-06T07:28:46.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.154+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e10153220 con 0x7f4e100d22b0 2024-03-06T07:28:46.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.155+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e100cc650 con 0x7f4e100d22b0 2024-03-06T07:28:46.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.155+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dfc01a070 con 0x7f4e100d22b0 2024-03-06T07:28:46.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.156+0000 7f4e0d7fa640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4df406b4d0 0x7f4df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.156+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dfc08a440 con 0x7f4e100d22b0 2024-03-06T07:28:46.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.156+0000 7f4e14af8640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4df406b4d0 0x7f4df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.156+0000 7f4e14af8640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4df406b4d0 0x7f4df406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f60 tx=0x7f4e04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.160+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dfc058c70 con 0x7f4e100d22b0 2024-03-06T07:28:46.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.459+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e1004f330 con 0x7f4e100d22b0 2024-03-06T07:28:46.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.460+0000 7f4e0d7fa640 1 -- 172.21.15.122:0/1277729733 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f4dfc05cd10 con 0x7f4e100d22b0 2024-03-06T07:28:46.462 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:46.462 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/1547452856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1547452856},{"type":"v1","addr":"172.21.15.169:6834","nonce":1547452856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1798905908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1798905908},{"type":"v1","addr":"172.21.15.169:6835","nonce":1798905908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:43.540750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:28:46.463 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 73 2024-03-06T07:28:46.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4df406b4d0 msgr2=0x7f4df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4df406b4d0 0x7f4df406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f60 tx=0x7f4e04009040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 msgr2=0x7f4e101f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 0x7f4e101f3140 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00ebe0 tx=0x7f4dfc00c520 comp rx=0 tx=0).stop 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 shutdown_connections 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4df406b4d0 0x7f4df406d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e1015dfe0 0x7f4e101600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e100d22b0 0x7f4e101f3140 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 --2- 172.21.15.122:0/1277729733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e100b41f0 0x7f4e100ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 >> 172.21.15.122:0/1277729733 conn(0x7f4e100d3cd0 msgr2=0x7f4e100b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 shutdown_connections 2024-03-06T07:28:46.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.462+0000 7f4e16d5c640 1 -- 172.21.15.122:0/1277729733 wait complete. 2024-03-06T07:28:46.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:28:43.540750+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8688}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:28:46.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:46.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:28:46.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.984+0000 7f60fd148640 1 Processor -- start 2024-03-06T07:28:46.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.984+0000 7f60fd148640 1 -- start start 2024-03-06T07:28:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60fd148640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f80b4fe0 0x7f60f80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60fd148640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60f815ba50 0x7f60f815be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60fd148640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f815d510 con 0x7f60f80b4c00 2024-03-06T07:28:46.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60fd148640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f815e0b0 con 0x7f60f815ba50 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60fd148640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f815d780 con 0x7f60f80b4fe0 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60f759e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60f815ba50 0x7f60f815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60f759e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f60f815ba50 0x7f60f815be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39446/0 (socket says 172.21.15.122:39446) 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.985+0000 7f60f759e640 1 -- 172.21.15.122:0/3495044962 learned_addr learned my addr 172.21.15.122:0/3495044962 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f659c640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f80b4fe0 0x7f60f80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f759e640 1 -- 172.21.15.122:0/3495044962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f80b4fe0 msgr2=0x7f60f80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f759e640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f80b4fe0 0x7f60f80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f759e640 1 -- 172.21.15.122:0/3495044962 >> v1:172.21.15.169:6789/0 conn(0x7f60f80b4c00 legacy=0x7f60f80b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f759e640 1 -- 172.21.15.122:0/3495044962 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f815da60 con 0x7f60f815ba50 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f659c640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f80b4fe0 0x7f60f80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.986+0000 7f60f759e640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f815ba50 0x7f60f815be90 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009e50 tx=0x7f60f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1df29ba9cf5928 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60f5d9b640 1 -- 172.21.15.122:0/3495044962 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f002cb00 con 0x7f60f815ba50 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60f5d9b640 1 -- 172.21.15.122:0/3495044962 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f002cca0 con 0x7f60f815ba50 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60f5d9b640 1 -- 172.21.15.122:0/3495044962 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0034610 con 0x7f60f815ba50 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 -- 172.21.15.122:0/3495044962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f815ba50 msgr2=0x7f60f815be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f815ba50 0x7f60f815be90 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009e50 tx=0x7f60f002bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 -- 172.21.15.122:0/3495044962 shutdown_connections 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f815ba50 0x7f60f815be90 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 --2- 172.21.15.122:0/3495044962 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f80b4fe0 0x7f60f80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 -- 172.21.15.122:0/3495044962 >> 172.21.15.122:0/3495044962 conn(0x7f60f809a9f0 msgr2=0x7f60f809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 -- 172.21.15.122:0/3495044962 shutdown_connections 2024-03-06T07:28:46.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.987+0000 7f60fd148640 1 -- 172.21.15.122:0/3495044962 wait complete. 2024-03-06T07:28:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.989+0000 7f60fd148640 1 Processor -- start 2024-03-06T07:28:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.989+0000 7f60fd148640 1 -- start start 2024-03-06T07:28:46.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.990+0000 7f60fd148640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 0x7f60f8152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.990+0000 7f60fd148640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60f80b4fe0 0x7f60f8152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.990+0000 7f60fd148640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f815ba50 0x7f60f8145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.990+0000 7f60fd148640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f815f6d0 con 0x7f60f80b4c00 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.990+0000 7f60fd148640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f815f550 con 0x7f60f80b4fe0 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.990+0000 7f60fd148640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f815c3d0 con 0x7f60f815ba50 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 0x7f60f8152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 0x7f60f8152c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39458/0 (socket says 172.21.15.122:39458) 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 -- 172.21.15.122:0/2714543531 learned_addr learned my addr 172.21.15.122:0/2714543531 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f759e640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f815ba50 0x7f60f8145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f659c640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60f80b4fe0 0x7f60f8152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 -- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f815ba50 msgr2=0x7f60f8145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f815ba50 0x7f60f8145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 -- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60f80b4fe0 msgr2=0x7f60f8152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60f80b4fe0 0x7f60f8152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 -- 172.21.15.122:0/2714543531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f81551a0 con 0x7f60f80b4c00 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f759e640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f815ba50 0x7f60f8145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 0x7f60f8152c00 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c460 tx=0x7f60e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60df7fe640 1 -- 172.21.15.122:0/2714543531 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8016020 con 0x7f60f80b4c00 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60df7fe640 1 -- 172.21.15.122:0/2714543531 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e8012870 con 0x7f60f80b4c00 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60df7fe640 1 -- 172.21.15.122:0/2714543531 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8012c60 con 0x7f60f80b4c00 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f8155480 con 0x7f60f80b4c00 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.991+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f8155a40 con 0x7f60f80b4c00 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.993+0000 7f60df7fe640 1 -- 172.21.15.122:0/2714543531 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60e8019020 con 0x7f60f80b4c00 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.994+0000 7f60df7fe640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e406b4d0 0x7f60e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.994+0000 7f60f659c640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e406b4d0 0x7f60e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.994+0000 7f60df7fe640 1 -- 172.21.15.122:0/2714543531 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60e8089e90 con 0x7f60f80b4c00 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.994+0000 7f60f659c640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e406b4d0 0x7f60e406d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f60f8151ff0 tx=0x7f60ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.994+0000 7f60ffe0d740 1 Processor -- start 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 -- start start 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5645119ec870 0x5645119ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 0x5645119f2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a1ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645119566b0 con 0x5645119f08c0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645118ffa60 con 0x564511a1c600 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x564511952750 con 0x5645119ec870 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60f659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 0x5645119f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60f659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 0x5645119f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39460/0 (socket says 172.21.15.122:39460) 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.995+0000 7f60f659c640 1 -- 172.21.15.122:0/1294081102 learned_addr learned my addr 172.21.15.122:0/1294081102 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5645119ec870 0x5645119ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f759e640 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a1ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f659c640 1 -- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5645119ec870 msgr2=0x5645119ecc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f659c640 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5645119ec870 0x5645119ecc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f659c640 1 -- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 msgr2=0x564511a1ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f659c640 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a1ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f659c640 1 -- 172.21.15.122:0/1294081102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56451191df30 con 0x5645119f08c0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60f659c640 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 0x5645119f2360 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00f9c0 tx=0x7f60ec012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24ff3c1f4db802c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60dcff9640 1 -- 172.21.15.122:0/1294081102 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec01b050 con 0x5645119f08c0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60dcff9640 1 -- 172.21.15.122:0/1294081102 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec025070 con 0x5645119f08c0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.996+0000 7f60dcff9640 1 -- 172.21.15.122:0/1294081102 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec014e00 con 0x5645119f08c0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1294081102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 msgr2=0x5645119f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 0x5645119f2360 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00f9c0 tx=0x7f60ec012000 comp rx=0 tx=0).stop 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1294081102 shutdown_connections 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a1ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5645119f08c0 0x5645119f2360 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1294081102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x5645119ec870 0x5645119ecc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1294081102 >> 172.21.15.122:0/1294081102 conn(0x5645119ee8f0 msgr2=0x5645119f01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1294081102 shutdown_connections 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1294081102 wait complete. 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.997+0000 7f60ffe0d740 1 -- start start 2024-03-06T07:28:47.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.998+0000 7f60ffe0d740 1 Processor -- start 2024-03-06T07:28:47.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511a1fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 0x564511a193d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645119f0d50 con 0x564511a18f50 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564511912ea0 con 0x564511a1c600 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x5645119f1030 con 0x564511a34c70 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60ffe0d740 10 client.0 _ll_register_callbacks cb 0x564511912870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 0x564511a193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 0x564511a193d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39472/0 (socket says 172.21.15.122:39472) 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 -- 172.21.15.122:0/1051368761 learned_addr learned my addr 172.21.15.122:0/1051368761 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f659c640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511a1fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 -- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 msgr2=0x564511a1fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511a1fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 -- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 msgr2=0x564511a483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 -- 172.21.15.122:0/1051368761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564511913270 con 0x564511a18f50 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f659c640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511a1fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60f759e640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 0x564511a193d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009e50 tx=0x7f60f0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7640bcd84f5b21a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:46.999+0000 7f60d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60d5ffb640 1 -- 172.21.15.122:0/1051368761 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00349a0 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60d5ffb640 1 -- 172.21.15.122:0/1051368761 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60f0034b40 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60ec025070 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60d5ffb640 1 -- 172.21.15.122:0/1051368761 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60f0034d60 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 10 client.8841 fetch_fsmap learned FSMap version 73 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56451166b2d0 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60d5ffb640 1 -- 172.21.15.122:0/1051368761 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec025070 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60d5ffb640 1 -- 172.21.15.122:0/1051368761 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 73) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f60f0050050 con 0x564511a18f50 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 10 client.8841 fetch_fsmap finished waiting for FSMap version 73 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 10 client.8841 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 20 client.8841 populate_metadata read hostname 'smithi122' 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1e070 0x564511a56c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 4 client.8841 mds_command: new command op to 8688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 --> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564511963bf0 con 0x564511a1e070 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.000+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1e070 0x564511a56c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1e070 0x564511a56c70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a560f6566c222fa server_cookie=e0b1078e8a0dd34e in_seq=0 out_seq=0 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60d5ffb640 10 client.8841 ms_handle_connect on v2:172.21.15.122:6835/2094222247 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60d5ffb640 1 -- 172.21.15.122:0/1051368761 <== mds.0 v2:172.21.15.122:6835/2094222247 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564511963bf0 con 0x564511a1e070 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60d5ffb640 10 client.8841 handle_command_reply: tid=0 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 client.8841 shutdown 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 msgr2=0x564511a193d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 0x564511a193d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009e50 tx=0x7f60f0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 shutdown_connections 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1e070 0x564511a56c70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a18f50 0x564511a193d0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511a1fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60d5ffb640 0 client.8841 ms_handle_reset on v2:172.21.15.122:6835/2094222247 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1051368761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a1c600 0x564511a483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 >> 172.21.15.122:0/1051368761 conn(0x5645119ec870 msgr2=0x5645119eb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 shutdown_connections 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1051368761 wait complete. 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.001+0000 7f60ffe0d740 20 client.8841 trim_cache size 0 max 16384 2024-03-06T07:28:47.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.018+0000 7f60ffe0d740 1 Processor -- start 2024-03-06T07:28:47.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.018+0000 7f60ffe0d740 1 -- start start 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a18380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a1c600 0x564511a1c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 0x564511a1cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f81fde80 con 0x564511a34c70 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f8150ec0 con 0x564511a18f50 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80b0e60 con 0x564511a1c600 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 0x564511a1cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 0x564511a1cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39476/0 (socket says 172.21.15.122:39476) 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 -- 172.21.15.122:0/1766785002 learned_addr learned my addr 172.21.15.122:0/1766785002 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 -- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a1c600 msgr2=0x564511a1c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f659c640 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a1c600 0x564511a1c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a18380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a1c600 0x564511a1c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 -- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 msgr2=0x564511a18380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a18380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 -- 172.21.15.122:0/1766785002 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645117356b0 con 0x564511a34c70 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60f759e640 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 0x564511a1cf20 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009b70 tx=0x7f60f0041070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6c878a14bdde0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60d7fff640 1 -- 172.21.15.122:0/1766785002 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0003b90 con 0x564511a34c70 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60d7fff640 1 -- 172.21.15.122:0/1766785002 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f002ce50 con 0x564511a34c70 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.019+0000 7f60d7fff640 1 -- 172.21.15.122:0/1766785002 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0034600 con 0x564511a34c70 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1766785002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 msgr2=0x564511a1cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 0x564511a1cf20 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009b70 tx=0x7f60f0041070 comp rx=0 tx=0).stop 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1766785002 shutdown_connections 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a34c70 0x564511a1cf20 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a1c600 0x564511a1c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/1766785002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a18380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1766785002 >> 172.21.15.122:0/1766785002 conn(0x5645119ec870 msgr2=0x564511a3b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1766785002 shutdown_connections 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 -- 172.21.15.122:0/1766785002 wait complete. 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 -- start start 2024-03-06T07:28:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.020+0000 7f60ffe0d740 1 Processor -- start 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a15ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 0x564511a1ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511aab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564511a12e60 con 0x564511a1c600 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f81508c0 con 0x564511a18f50 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x564511a13150 con 0x564511a34c70 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60ffe0d740 10 client.0 _ll_register_callbacks cb 0x7f60f002cd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:47.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60f659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 0x564511a1ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60f659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 0x564511a1ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39478/0 (socket says 172.21.15.122:39478) 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60f659c640 1 -- 172.21.15.122:0/802952480 learned_addr learned my addr 172.21.15.122:0/802952480 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60f759e640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511aab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.021+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a15ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60f659c640 1 -- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 msgr2=0x564511aab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60f659c640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511aab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60f659c640 1 -- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 msgr2=0x564511a15ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60f659c640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a15ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60f659c640 1 -- 172.21.15.122:0/802952480 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564511a1bf40 con 0x564511a1c600 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60f659c640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 0x564511a1ba00 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x564511a20cf0 tx=0x7f60ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1bc2f01ecba35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60d5ffb640 1 -- 172.21.15.122:0/802952480 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec005b90 con 0x564511a1c600 2024-03-06T07:28:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60d5ffb640 1 -- 172.21.15.122:0/802952480 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60ec005d30 con 0x564511a1c600 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60f002ce50 con 0x564511a1c600 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60d5ffb640 1 -- 172.21.15.122:0/802952480 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60ec01b020 con 0x564511a1c600 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 10 client.8847 fetch_fsmap learned FSMap version 73 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x564511a13f30 con 0x564511a1c600 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60d5ffb640 1 -- 172.21.15.122:0/802952480 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f002ce50 con 0x564511a1c600 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60d5ffb640 1 -- 172.21.15.122:0/802952480 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 73) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f60ec033070 con 0x564511a1c600 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 10 client.8847 fetch_fsmap finished waiting for FSMap version 73 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 10 client.8847 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 20 client.8847 populate_metadata read hostname 'smithi122' 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1cd80 0x564511a32fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 4 client.8847 mds_command: new command op to 8688 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.022+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 --> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56451195df40 con 0x564511a1cd80 2024-03-06T07:28:47.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1cd80 0x564511a32fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60d5ffb640 10 client.8847 ms_handle_connect on v2:172.21.15.122:6835/2094222247 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60f6d9d640 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1cd80 0x564511a32fc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c538970af7a9a4f8 server_cookie=7bba0ea479483035 in_seq=0 out_seq=0 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60d5ffb640 1 -- 172.21.15.122:0/802952480 <== mds.0 v2:172.21.15.122:6835/2094222247 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56451195df40 con 0x564511a1cd80 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60d5ffb640 10 client.8847 handle_command_reply: tid=0 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 client.8847 shutdown 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 msgr2=0x564511a1ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 0x564511a1ba00 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x564511a20cf0 tx=0x7f60ec004730 comp rx=0 tx=0).stop 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 shutdown_connections 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x564511a1cd80 0x564511a32fc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x564511a34c70 0x564511aab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564511a1c600 0x564511a1ba00 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60d5ffb640 0 client.8847 ms_handle_reset on v2:172.21.15.122:6835/2094222247 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 --2- 172.21.15.122:0/802952480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564511a18f50 0x564511a15ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 >> 172.21.15.122:0/802952480 conn(0x5645119ec870 msgr2=0x564511a126f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 shutdown_connections 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.023+0000 7f60ffe0d740 1 -- 172.21.15.122:0/802952480 wait complete. 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60ffe0d740 20 client.8847 trim_cache size 0 max 16384 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 8688, 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:28:47.032 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 4.8256079009999997, 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 73, 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 126, 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 126, 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 9.9213135440000002 2024-03-06T07:28:47.033 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e406b4d0 msgr2=0x7f60e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e406b4d0 0x7f60e406d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f60f8151ff0 tx=0x7f60ec008040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 msgr2=0x7f60f8152c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 0x7f60f8152c00 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c460 tx=0x7f60e800c920 comp rx=0 tx=0).stop 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 shutdown_connections 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f60e406b4d0 0x7f60e406d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f60f815ba50 0x7f60f8145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60f80b4fe0 0x7f60f8152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 --2- 172.21.15.122:0/2714543531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60f80b4c00 0x7f60f8152c00 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 >> 172.21.15.122:0/2714543531 conn(0x7f60f809a9f0 msgr2=0x7f60f806fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 shutdown_connections 2024-03-06T07:28:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.024+0000 7f60fd148640 1 -- 172.21.15.122:0/2714543531 wait complete. 2024-03-06T07:28:47.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:47.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.136+0000 7f0a5baaf640 1 Processor -- start 2024-03-06T07:28:47.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.136+0000 7f0a5baaf640 1 -- start start 2024-03-06T07:28:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5baaf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0a5415dfe0 0x7f0a540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5baaf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0a540d29c0 0x7f0a540b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5baaf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a540b3430 con 0x7f0a540cf830 2024-03-06T07:28:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5baaf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a540b31e0 con 0x7f0a5415dfe0 2024-03-06T07:28:47.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5baaf640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a540b3d60 con 0x7f0a540d29c0 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0a5415dfe0 0x7f0a540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0a5415dfe0 0x7f0a540cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39484/0 (socket says 172.21.15.122:39484) 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 -- 172.21.15.122:0/1914885922 learned_addr learned my addr 172.21.15.122:0/1914885922 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5a04c640 1 --2- 172.21.15.122:0/1914885922 >> v2:172.21.15.169:3300/0 conn(0x7f0a540d29c0 0x7f0a540b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 -- 172.21.15.122:0/1914885922 >> v2:172.21.15.169:3300/0 conn(0x7f0a540d29c0 msgr2=0x7f0a540b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 --2- 172.21.15.122:0/1914885922 >> v2:172.21.15.169:3300/0 conn(0x7f0a540d29c0 0x7f0a540b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 -- 172.21.15.122:0/1914885922 >> v1:172.21.15.169:6789/0 conn(0x7f0a540cf830 legacy=0x7f0a540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.137+0000 7f0a5984b640 1 -- 172.21.15.122:0/1914885922 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540b3710 con 0x7f0a5415dfe0 2024-03-06T07:28:47.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.138+0000 7f0a5984b640 1 --2- 172.21.15.122:0/1914885922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a5415dfe0 0x7f0a540cf2f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f0a540b3cd0 tx=0x7f0a5002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be828357dafb04d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.138+0000 7f0a58849640 1 -- 172.21.15.122:0/1914885922 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50034070 con 0x7f0a5415dfe0 2024-03-06T07:28:47.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.138+0000 7f0a58849640 1 -- 172.21.15.122:0/1914885922 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a50030860 con 0x7f0a5415dfe0 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.138+0000 7f0a58849640 1 -- 172.21.15.122:0/1914885922 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50030c30 con 0x7f0a5415dfe0 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 -- 172.21.15.122:0/1914885922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a5415dfe0 msgr2=0x7f0a540cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/1914885922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a5415dfe0 0x7f0a540cf2f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f0a540b3cd0 tx=0x7f0a5002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 -- 172.21.15.122:0/1914885922 shutdown_connections 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/1914885922 >> v2:172.21.15.169:3300/0 conn(0x7f0a540d29c0 0x7f0a540b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/1914885922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a5415dfe0 0x7f0a540cf2f0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 -- 172.21.15.122:0/1914885922 >> 172.21.15.122:0/1914885922 conn(0x7f0a540d3cd0 msgr2=0x7f0a540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 -- 172.21.15.122:0/1914885922 shutdown_connections 2024-03-06T07:28:47.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.139+0000 7f0a5baaf640 1 -- 172.21.15.122:0/1914885922 wait complete. 2024-03-06T07:28:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.140+0000 7f0a5baaf640 1 Processor -- start 2024-03-06T07:28:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.140+0000 7f0a5baaf640 1 -- start start 2024-03-06T07:28:47.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a540cf830 0x7f0a540cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 0x7f0a540b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5415dfe0 0x7f0a540c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a540d1df0 con 0x7f0a540d29c0 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a540d1c70 con 0x7f0a5415dfe0 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5415e3c0 con 0x7f0a540cf830 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 0x7f0a540b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 0x7f0a540b4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39498/0 (socket says 172.21.15.122:39498) 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 -- 172.21.15.122:0/3180570937 learned_addr learned my addr 172.21.15.122:0/3180570937 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5984b640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a540cf830 0x7f0a540cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5a04c640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5415dfe0 0x7f0a540c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 -- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a540cf830 msgr2=0x7f0a540cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a540cf830 0x7f0a540cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 -- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5415dfe0 msgr2=0x7f0a540c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5415dfe0 0x7f0a540c9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 -- 172.21.15.122:0/3180570937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5415fef0 con 0x7f0a540d29c0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5984b640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a540cf830 0x7f0a540cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5904a640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 0x7f0a540b4f50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400c440 tx=0x7f0a44001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a440052d0 con 0x7f0a540d29c0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a44005470 con 0x7f0a540d29c0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a540cdcf0 con 0x7f0a540d29c0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44007240 con 0x7f0a540d29c0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.141+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a54160190 con 0x7f0a540d29c0 2024-03-06T07:28:47.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.142+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5415fcc0 con 0x7f0a540d29c0 2024-03-06T07:28:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.143+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a440073e0 con 0x7f0a540d29c0 2024-03-06T07:28:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.143+0000 7f0a4a7fc640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a3806b4d0 0x7f0a3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.143+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a440896a0 con 0x7f0a540d29c0 2024-03-06T07:28:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.143+0000 7f0a5984b640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a3806b4d0 0x7f0a3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.144+0000 7f0a5984b640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a3806b4d0 0x7f0a3806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0a50036000 tx=0x7f0a5003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.148+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a44058520 con 0x7f0a540d29c0 2024-03-06T07:28:47.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.364+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0a5404f330 con 0x7f0a540d29c0 2024-03-06T07:28:47.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.364+0000 7f0a4a7fc640 1 -- 172.21.15.122:0/3180570937 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0a44057e20 con 0x7f0a540d29c0 2024-03-06T07:28:47.367 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:47.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a3806b4d0 msgr2=0x7f0a3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a3806b4d0 0x7f0a3806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0a50036000 tx=0x7f0a5003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 msgr2=0x7f0a540b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 0x7f0a540b4f50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400c440 tx=0x7f0a44001e70 comp rx=0 tx=0).stop 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 shutdown_connections 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a3806b4d0 0x7f0a3806d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5415dfe0 0x7f0a540c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a540d29c0 0x7f0a540b4f50 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 --2- 172.21.15.122:0/3180570937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a540cf830 0x7f0a540cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 >> 172.21.15.122:0/3180570937 conn(0x7f0a540d3cd0 msgr2=0x7f0a540cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 shutdown_connections 2024-03-06T07:28:47.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:47.367+0000 7f0a5baaf640 1 -- 172.21.15.122:0/3180570937 wait complete. 2024-03-06T07:28:47.383 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:47.383 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:47.384 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:47.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.384 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.400 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42313sec preferred_lft 42313sec 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:28:47.402 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.403 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:47.440 INFO:tasks.ceph.mds.c.smithi122.stderr:2024-03-06T07:28:47.438+0000 7f599c482640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:47.465 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:28:47.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.466 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:28:47.523 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:28:47.523 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:47.523 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:47.523 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:47.523 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:47.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.524 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:28:47.524 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:28:47.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:47.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:47.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:47.524 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:28:47.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.618 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:28:47.633 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:47.633 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:47.633 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:47.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.633 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:28:47.699 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:28:47.699 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:47.699 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:47.754 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:47.755 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:47.809 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:28:47.876 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:47.931 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:28:47.932 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:28:47.933 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:47.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.934 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:28:47.986 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:28:47.986 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:28:48.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:28:48.055 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:28:48.112 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:48.113 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.114 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:28:48.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:28:48.115+0000 7f3ab5837480 -1 init, newargv = 0x56507e420740 newargc=15 2024-03-06T07:28:48.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[78847]: starting ceph client 2024-03-06T07:28:48.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[78847]: starting fuse 2024-03-06T07:28:48.131 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:28:48.131 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:28:48.132 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:28:48.133 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:28:48.253 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.78847.asok 2024-03-06T07:28:48.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:48.263 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78847.asok 2024-03-06T07:28:48.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.263 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78847.asok status 2024-03-06T07:28:48.398 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "78847", 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 8859, 2024-03-06T07:28:48.399 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 8859 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1054696485 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:28:48.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1054696485 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.8859 192.168.0.1:0/1054696485", 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/1054696485", 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 73, 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 126, 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:28:48.401 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:28:48.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.407 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:28:48.407 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.422 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:28:48.422 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.491 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:28:48.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 Processor -- start 2024-03-06T07:28:48.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 -- start start 2024-03-06T07:28:48.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f73340d4860 0x7f73340d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f73340b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73341618b0 con 0x7f73341614d0 2024-03-06T07:28:48.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73340d4cf0 con 0x7f73340d4860 2024-03-06T07:28:48.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.635+0000 7f73394c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73340cf400 con 0x7f73340b0e30 2024-03-06T07:28:48.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7332ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f73341614d0 0x7f73340d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41254/0 (socket says 172.21.15.122:41254) 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7332ffd640 1 -- 172.21.15.122:0/3908023520 learned_addr learned my addr 172.21.15.122:0/3908023520 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f73337fe640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f73340b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f73327fc640 1 --2- 172.21.15.122:0/3908023520 >> v2:172.21.15.169:3300/0 conn(0x7f73340d4860 0x7f73340d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585936853 0 0) 0x7f73341618b0 con 0x7f73341614d0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7320003520 con 0x7f73341614d0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294822772 0 0) 0x7f7320003520 con 0x7f73341614d0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 msgr2=0x7f73340b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f73340b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 >> v2:172.21.15.169:3300/0 conn(0x7f73340d4860 msgr2=0x7f73340d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 --2- 172.21.15.122:0/3908023520 >> v2:172.21.15.169:3300/0 conn(0x7f73340d4860 0x7f73340d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73340cf6a0 con 0x7f73341614d0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7324001670 con 0x7f73341614d0 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f73337fe640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f73340b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f73327fc640 1 --2- 172.21.15.122:0/3908023520 >> v2:172.21.15.169:3300/0 conn(0x7f73340d4860 0x7f73340d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 >> v1:172.21.15.122:6789/0 conn(0x7f73341614d0 legacy=0x7f73340d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73200039b0 0x7f7320003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f7320004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7320004d40 0x7f73200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340d4cf0 con 0x7f7320004d40 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340cf400 con 0x7f73200039b0 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732000b9c0 con 0x7f73340b0e30 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f73337fe640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7320004d40 0x7f73200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f7332ffd640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73200039b0 0x7f7320003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.636+0000 7f73327fc640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f7320004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73337fe640 1 -- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 msgr2=0x7f7320004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73337fe640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f7320004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73337fe640 1 -- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73200039b0 msgr2=0x7f7320003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73337fe640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73200039b0 0x7f7320003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73337fe640 1 -- 172.21.15.122:0/3908023520 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320003520 con 0x7f7320004d40 2024-03-06T07:28:48.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73327fc640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340b0e30 0x7f7320004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f73337fe640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7320004d40 0x7f73200051c0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f732c009c90 tx=0x7f7320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555a77caf774dfd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c039070 con 0x7f7320004d40 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c02c8f0 con 0x7f7320004d40 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.637+0000 7f7331ffb640 1 -- 172.21.15.122:0/3908023520 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c009860 con 0x7f7320004d40 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73394c9640 1 -- 172.21.15.122:0/3908023520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7320004d40 msgr2=0x7f73200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73394c9640 1 --2- 172.21.15.122:0/3908023520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7320004d40 0x7f73200051c0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f732c009c90 tx=0x7f7320005830 comp rx=0 tx=0).stop 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73337fe640 1 -- 172.21.15.122:0/3908023520 reap_dead start 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73394c9640 1 -- 172.21.15.122:0/3908023520 shutdown_connections 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73394c9640 1 -- 172.21.15.122:0/3908023520 >> 172.21.15.122:0/3908023520 conn(0x7f73340d3d40 msgr2=0x7f73340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73394c9640 1 -- 172.21.15.122:0/3908023520 shutdown_connections 2024-03-06T07:28:48.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.638+0000 7f73394c9640 1 -- 172.21.15.122:0/3908023520 wait complete. 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.639+0000 7f73394c9640 1 Processor -- start 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.639+0000 7f73394c9640 1 -- start start 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73394c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73341551f0 0x7f733415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73394c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340c8280 0x7f73340cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73394c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 0x7f73340cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73394c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340b0910 con 0x7f73340cbed0 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73394c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340cfc60 con 0x7f73341551f0 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73394c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73340d2560 con 0x7f73340c8280 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 0x7f73340cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 0x7f73340cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39508/0 (socket says 172.21.15.122:39508) 2024-03-06T07:28:48.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73337fe640 1 -- 172.21.15.122:0/4145771522 learned_addr learned my addr 172.21.15.122:0/4145771522 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f7332ffd640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73341551f0 0x7f733415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.640+0000 7f73327fc640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340c8280 0x7f73340cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73337fe640 1 -- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340c8280 msgr2=0x7f73340cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73337fe640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340c8280 0x7f73340cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73337fe640 1 -- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73341551f0 msgr2=0x7f733415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73337fe640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73341551f0 0x7f733415bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73337fe640 1 -- 172.21.15.122:0/4145771522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73340c8960 con 0x7f73340cbed0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73337fe640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 0x7f73340cc330 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f732c009db0 tx=0x7f732c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c039030 con 0x7f73340cbed0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c03a040 con 0x7f73340cbed0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73340c8c40 con 0x7f73340cbed0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c009860 con 0x7f73340cbed0 2024-03-06T07:28:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.641+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73340cb190 con 0x7f73340cbed0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.642+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7334068ff0 con 0x7f73340cbed0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.642+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f732c045070 con 0x7f73340cbed0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.643+0000 7f732b7fe640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f731406b500 0x7f731406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.643+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f732c0aacd0 con 0x7f73340cbed0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.643+0000 7f7332ffd640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f731406b500 0x7f731406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.643+0000 7f7332ffd640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f731406b500 0x7f731406d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7324001550 tx=0x7f73240078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.651+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f732c079480 con 0x7f73340cbed0 2024-03-06T07:28:48.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.861+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f73340b4ee0 con 0x7f73340cbed0 2024-03-06T07:28:48.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.977+0000 7f732b7fe640 1 -- 172.21.15.122:0/4145771522 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v74) v1 ==== 105+0+0 (secure 0 0 0) 0x7f732c098020 con 0x7f73340cbed0 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.979+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f731406b500 msgr2=0x7f731406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.979+0000 7f73394c9640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f731406b500 0x7f731406d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7324001550 tx=0x7f73240078f0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.979+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 msgr2=0x7f73340cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.979+0000 7f73394c9640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 0x7f73340cc330 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f732c009db0 tx=0x7f732c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 shutdown_connections 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f731406b500 0x7f731406d9c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73340cbed0 0x7f73340cc330 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f73340c8280 0x7f73340cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 --2- 172.21.15.122:0/4145771522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73341551f0 0x7f733415bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 >> 172.21.15.122:0/4145771522 conn(0x7f73340d3d40 msgr2=0x7f73340b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 shutdown_connections 2024-03-06T07:28:48.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:48.980+0000 7f73394c9640 1 -- 172.21.15.122:0/4145771522 wait complete. 2024-03-06T07:28:48.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.096+0000 7f38f4e8d640 1 Processor -- start 2024-03-06T07:28:49.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.096+0000 7f38f4e8d640 1 -- start start 2024-03-06T07:28:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38f4e8d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f015dfe0 0x7f38f00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38f4e8d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f38f00d29c0 0x7f38f00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38f4e8d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f00b3430 con 0x7f38f00cf830 2024-03-06T07:28:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38f4e8d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f00b31e0 con 0x7f38f00d29c0 2024-03-06T07:28:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38f4e8d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f00b3d60 con 0x7f38f015dfe0 2024-03-06T07:28:49.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38edd9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f38f00cf830 0x7f38f00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41268/0 (socket says 172.21.15.122:41268) 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.097+0000 7f38edd9b640 1 -- 172.21.15.122:0/1297769129 learned_addr learned my addr 172.21.15.122:0/1297769129 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38eed9d640 1 --2- 172.21.15.122:0/1297769129 >> v2:172.21.15.169:3300/0 conn(0x7f38f00d29c0 0x7f38f00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ee59c640 1 --2- 172.21.15.122:0/1297769129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f015dfe0 0x7f38f00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ed59a640 1 -- 172.21.15.122:0/1297769129 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32328957 0 0) 0x7f38f00b3430 con 0x7f38f00cf830 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ed59a640 1 -- 172.21.15.122:0/1297769129 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d8003520 con 0x7f38f00cf830 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ee59c640 1 -- 172.21.15.122:0/1297769129 >> v2:172.21.15.169:3300/0 conn(0x7f38f00d29c0 msgr2=0x7f38f00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ee59c640 1 --2- 172.21.15.122:0/1297769129 >> v2:172.21.15.169:3300/0 conn(0x7f38f00d29c0 0x7f38f00b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ee59c640 1 -- 172.21.15.122:0/1297769129 >> v1:172.21.15.122:6789/0 conn(0x7f38f00cf830 legacy=0x7f38f00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ed59a640 1 -- 172.21.15.122:0/1297769129 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703536736 0 0) 0x7f38d8003520 con 0x7f38f00cf830 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38ee59c640 1 -- 172.21.15.122:0/1297769129 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f00b3710 con 0x7f38f015dfe0 2024-03-06T07:28:49.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.098+0000 7f38eed9d640 1 --2- 172.21.15.122:0/1297769129 >> v2:172.21.15.169:3300/0 conn(0x7f38f00d29c0 0x7f38f00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:49.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.099+0000 7f38ee59c640 1 --2- 172.21.15.122:0/1297769129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f015dfe0 0x7f38f00cf2f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f38e00097a0 tx=0x7f38e002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409cbebae2eba5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.099+0000 7f38ed59a640 1 -- 172.21.15.122:0/1297769129 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0035020 con 0x7f38f015dfe0 2024-03-06T07:28:49.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.099+0000 7f38ed59a640 1 -- 172.21.15.122:0/1297769129 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e0031830 con 0x7f38f015dfe0 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.099+0000 7f38ed59a640 1 -- 172.21.15.122:0/1297769129 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0031be0 con 0x7f38f015dfe0 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 -- 172.21.15.122:0/1297769129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f015dfe0 msgr2=0x7f38f00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/1297769129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f015dfe0 0x7f38f00cf2f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f38e00097a0 tx=0x7f38e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 -- 172.21.15.122:0/1297769129 shutdown_connections 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/1297769129 >> v2:172.21.15.169:3300/0 conn(0x7f38f00d29c0 0x7f38f00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/1297769129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f015dfe0 0x7f38f00cf2f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 -- 172.21.15.122:0/1297769129 >> 172.21.15.122:0/1297769129 conn(0x7f38f00d3cd0 msgr2=0x7f38f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 -- 172.21.15.122:0/1297769129 shutdown_connections 2024-03-06T07:28:49.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.100+0000 7f38f4e8d640 1 -- 172.21.15.122:0/1297769129 wait complete. 2024-03-06T07:28:49.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.101+0000 7f38f4e8d640 1 Processor -- start 2024-03-06T07:28:49.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.101+0000 7f38f4e8d640 1 -- start start 2024-03-06T07:28:49.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38f4e8d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f00cf830 0x7f38f014f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38f4e8d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f00d29c0 0x7f38f0160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38f4e8d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 0x7f38f00ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38f4e8d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f00d1df0 con 0x7f38f015dfe0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38f4e8d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f00d1c70 con 0x7f38f00cf830 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38f4e8d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f015e3c0 con 0x7f38f00d29c0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 0x7f38f00ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 0x7f38f00ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39524/0 (socket says 172.21.15.122:39524) 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 -- 172.21.15.122:0/3574741028 learned_addr learned my addr 172.21.15.122:0/3574741028 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38edd9b640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f00d29c0 0x7f38f0160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38ee59c640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f00cf830 0x7f38f014f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 -- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f00d29c0 msgr2=0x7f38f0160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f00d29c0 0x7f38f0160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 -- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f00cf830 msgr2=0x7f38f014f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f00cf830 0x7f38f014f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38eed9d640 1 -- 172.21.15.122:0/3574741028 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f015ff10 con 0x7f38f015dfe0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38edd9b640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f00d29c0 0x7f38f0160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.102+0000 7f38ee59c640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f00cf830 0x7f38f014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.103+0000 7f38eed9d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 0x7f38f00ccdb0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f38e800efc0 tx=0x7f38e8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.103+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8018030 con 0x7f38f015dfe0 2024-03-06T07:28:49.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.103+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e800cd80 con 0x7f38f015dfe0 2024-03-06T07:28:49.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.103+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8003bb0 con 0x7f38f015dfe0 2024-03-06T07:28:49.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.103+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f01601d0 con 0x7f38f015dfe0 2024-03-06T07:28:49.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.103+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f0160790 con 0x7f38f015dfe0 2024-03-06T07:28:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.104+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f004f330 con 0x7f38f015dfe0 2024-03-06T07:28:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.104+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38e8005d10 con 0x7f38f015dfe0 2024-03-06T07:28:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.105+0000 7f38deffd640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.105+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38e808ab60 con 0x7f38f015dfe0 2024-03-06T07:28:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.105+0000 7f38ee59c640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.105+0000 7f38ee59c640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f38e0036000 tx=0x7f38e003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.109+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38e8022070 con 0x7f38f015dfe0 2024-03-06T07:28:49.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.335+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38f00cf3a0 con 0x7f38f015dfe0 2024-03-06T07:28:49.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.335+0000 7f38deffd640 1 -- 172.21.15.122:0/3574741028 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5007 (secure 0 0 0) 0x7f38e8010040 con 0x7f38f015dfe0 2024-03-06T07:28:49.338 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:49.338 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/1547452856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1547452856},{"type":"v1","addr":"172.21.15.169:6834","nonce":1547452856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1798905908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1798905908},{"type":"v1","addr":"172.21.15.169:6835","nonce":1798905908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:48.977414+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:creating","state_seq":1,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:28:49.338 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 74 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 msgr2=0x7f38d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f38e0036000 tx=0x7f38e003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 msgr2=0x7f38f00ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 0x7f38f00ccdb0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f38e800efc0 tx=0x7f38e8009f90 comp rx=0 tx=0).stop 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 shutdown_connections 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f015dfe0 0x7f38f00ccdb0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f00d29c0 0x7f38f0160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 --2- 172.21.15.122:0/3574741028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f00cf830 0x7f38f014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 >> 172.21.15.122:0/3574741028 conn(0x7f38f00d3cd0 msgr2=0x7f38f006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.337+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 shutdown_connections 2024-03-06T07:28:49.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:49.338+0000 7f38f4e8d640 1 -- 172.21.15.122:0/3574741028 wait complete. 2024-03-06T07:28:49.350 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:28:52.440 INFO:tasks.ceph.mds.c.smithi122.stderr:2024-03-06T07:28:52.438+0000 7f599c482640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:28:54.351 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.452+0000 7fc724f20640 1 Processor -- start 2024-03-06T07:28:54.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.452+0000 7fc724f20640 1 -- start start 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.453+0000 7fc724f20640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7200d2710 0x7fc7200b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.453+0000 7fc724f20640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7200b3430 con 0x7fc72015dfe0 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.453+0000 7fc724f20640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7200b3710 con 0x7fc7200d22b0 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.453+0000 7fc724f20640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7200b31e0 con 0x7fc7200d2710 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.454+0000 7fc71ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7200d2710 0x7fc7200b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.454+0000 7fc71ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7200d2710 0x7fc7200b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39530/0 (socket says 172.21.15.122:39530) 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.454+0000 7fc71ed9d640 1 -- 172.21.15.122:0/1995752263 learned_addr learned my addr 172.21.15.122:0/1995752263 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.454+0000 7fc71ed9d640 1 -- 172.21.15.122:0/1995752263 >> v1:172.21.15.169:6789/0 conn(0x7fc7200d22b0 legacy=0x7fc7200b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.454+0000 7fc71ed9d640 1 -- 172.21.15.122:0/1995752263 >> v1:172.21.15.122:6789/0 conn(0x7fc72015dfe0 legacy=0x7fc7200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.454+0000 7fc71ed9d640 1 -- 172.21.15.122:0/1995752263 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200b39f0 con 0x7fc7200d2710 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc71ed9d640 1 --2- 172.21.15.122:0/1995752263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d2710 0x7fc7200b4900 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e80 tx=0x7fc71802bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a1e21ca58251b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc71d59a640 1 -- 172.21.15.122:0/1995752263 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71802cb80 con 0x7fc7200d2710 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc71d59a640 1 -- 172.21.15.122:0/1995752263 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71802cd20 con 0x7fc7200d2710 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc71d59a640 1 -- 172.21.15.122:0/1995752263 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718034610 con 0x7fc7200d2710 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 -- 172.21.15.122:0/1995752263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d2710 msgr2=0x7fc7200b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 --2- 172.21.15.122:0/1995752263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d2710 0x7fc7200b4900 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e80 tx=0x7fc71802bfe0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 -- 172.21.15.122:0/1995752263 shutdown_connections 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 --2- 172.21.15.122:0/1995752263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d2710 0x7fc7200b4900 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 -- 172.21.15.122:0/1995752263 >> 172.21.15.122:0/1995752263 conn(0x7fc7200d3cd0 msgr2=0x7fc7200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 -- 172.21.15.122:0/1995752263 shutdown_connections 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.455+0000 7fc724f20640 1 -- 172.21.15.122:0/1995752263 wait complete. 2024-03-06T07:28:54.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 Processor -- start 2024-03-06T07:28:54.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 -- start start 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 0x7fc7200cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7200d2710 0x7fc7200b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc72015dfe0 0x7fc7200b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200cfa00 con 0x7fc7200d22b0 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200b3d60 con 0x7fc72015dfe0 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.457+0000 7fc724f20640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200d1c70 con 0x7fc7200d2710 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 0x7fc7200cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 0x7fc7200cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39532/0 (socket says 172.21.15.122:39532) 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 -- 172.21.15.122:0/2420444243 learned_addr learned my addr 172.21.15.122:0/2420444243 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 -- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7200d2710 msgr2=0x7fc7200b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7200d2710 0x7fc7200b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 -- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc72015dfe0 msgr2=0x7fc7200b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71ed9d640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc72015dfe0 0x7fc7200b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc72015dfe0 0x7fc7200b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 -- 172.21.15.122:0/2420444243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200cc460 con 0x7fc7200d22b0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc71e59c640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 0x7fc7200cbdf0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c400 tx=0x7fc71000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710017070 con 0x7fc7200d22b0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7100052d0 con 0x7fc7200d22b0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7200cc740 con 0x7fc7200d22b0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710003830 con 0x7fc7200d22b0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.458+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7200ccd00 con 0x7fc7200d22b0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.459+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7200670d0 con 0x7fc7200d22b0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.460+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc71001a070 con 0x7fc7200d22b0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.460+0000 7fc70effd640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc70006b500 0x7fc70006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.460+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc710089560 con 0x7fc7200d22b0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.460+0000 7fc71dd9b640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc70006b500 0x7fc70006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.461+0000 7fc71dd9b640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc70006b500 0x7fc70006d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc714000f60 tx=0x7fc7140079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.469+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7200670d0 con 0x7fc7200d22b0 2024-03-06T07:28:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.693+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc72004f330 con 0x7fc7200d22b0 2024-03-06T07:28:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.694+0000 7fc70effd640 1 -- 172.21.15.122:0/2420444243 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5010 (secure 0 0 0) 0x7fc7100583e0 con 0x7fc7200d22b0 2024-03-06T07:28:54.696 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:54.696 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/1547452856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1547452856},{"type":"v1","addr":"172.21.15.169:6834","nonce":1547452856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1798905908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1798905908},{"type":"v1","addr":"172.21.15.169:6835","nonce":1798905908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:49.978484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8685,8688]},"id":9}]} 2024-03-06T07:28:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 75 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.695+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc70006b500 msgr2=0x7fc70006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.695+0000 7fc724f20640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc70006b500 0x7fc70006d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc714000f60 tx=0x7fc7140079d0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.695+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 msgr2=0x7fc7200cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.695+0000 7fc724f20640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 0x7fc7200cbdf0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c400 tx=0x7fc71000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 shutdown_connections 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc70006b500 0x7fc70006d9c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc72015dfe0 0x7fc7200b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7200d2710 0x7fc7200b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 --2- 172.21.15.122:0/2420444243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7200d22b0 0x7fc7200cbdf0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 >> 172.21.15.122:0/2420444243 conn(0x7fc7200d3cd0 msgr2=0x7fc7200d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 shutdown_connections 2024-03-06T07:28:54.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.696+0000 7fc724f20640 1 -- 172.21.15.122:0/2420444243 wait complete. 2024-03-06T07:28:54.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.830+0000 7fa62600e640 1 Processor -- start 2024-03-06T07:28:54.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.830+0000 7fa62600e640 1 -- start start 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.831+0000 7fa62600e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6200d22b0 0x7fa6200d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.831+0000 7fa62600e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200b41f0 0x7fa6200b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa62600e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200b3430 con 0x7fa62015dfe0 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa62600e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6200d2c50 con 0x7fa6200d22b0 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa62600e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200b31e0 con 0x7fa6200b41f0 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6200d22b0 0x7fa6200d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa6200d22b0 0x7fa6200d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54490/0 (socket says 172.21.15.122:54490) 2024-03-06T07:28:54.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 -- 172.21.15.122:0/2465071428 learned_addr learned my addr 172.21.15.122:0/2465071428 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:54.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61ffff640 1 --2- 172.21.15.122:0/2465071428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200b41f0 0x7fa6200b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 -- 172.21.15.122:0/2465071428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200b41f0 msgr2=0x7fa6200b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 --2- 172.21.15.122:0/2465071428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200b41f0 0x7fa6200b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 -- 172.21.15.122:0/2465071428 >> v1:172.21.15.169:6789/0 conn(0x7fa62015dfe0 legacy=0x7fa6200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.832+0000 7fa61effd640 1 -- 172.21.15.122:0/2465071428 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200b3710 con 0x7fa6200d22b0 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.833+0000 7fa61effd640 1 --2- 172.21.15.122:0/2465071428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6200d22b0 0x7fa6200d2710 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fa614009870 tx=0x7fa61402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6feaf38194b06ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.833+0000 7fa61e7fc640 1 -- 172.21.15.122:0/2465071428 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa614035020 con 0x7fa6200d22b0 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.833+0000 7fa61e7fc640 1 -- 172.21.15.122:0/2465071428 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa614031640 con 0x7fa6200d22b0 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.833+0000 7fa61e7fc640 1 -- 172.21.15.122:0/2465071428 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61402d040 con 0x7fa6200d22b0 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 -- 172.21.15.122:0/2465071428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6200d22b0 msgr2=0x7fa6200d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 --2- 172.21.15.122:0/2465071428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6200d22b0 0x7fa6200d2710 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fa614009870 tx=0x7fa61402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 -- 172.21.15.122:0/2465071428 shutdown_connections 2024-03-06T07:28:54.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 --2- 172.21.15.122:0/2465071428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200b41f0 0x7fa6200b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 --2- 172.21.15.122:0/2465071428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6200d22b0 0x7fa6200d2710 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 -- 172.21.15.122:0/2465071428 >> 172.21.15.122:0/2465071428 conn(0x7fa6200d3cd0 msgr2=0x7fa6200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 -- 172.21.15.122:0/2465071428 shutdown_connections 2024-03-06T07:28:54.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.834+0000 7fa62600e640 1 -- 172.21.15.122:0/2465071428 wait complete. 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.835+0000 7fa62600e640 1 Processor -- start 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.835+0000 7fa62600e640 1 -- start start 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa62600e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6200b41f0 0x7fa62015ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa62600e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200d22b0 0x7fa620147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa62600e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 0x7fa62014a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa62600e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200cfa00 con 0x7fa62015dfe0 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa62600e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200b3d60 con 0x7fa6200b41f0 2024-03-06T07:28:54.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa62600e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200d1c70 con 0x7fa6200d22b0 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 0x7fa62014a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 0x7fa62014a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54496/0 (socket says 172.21.15.122:54496) 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 -- 172.21.15.122:0/2836599748 learned_addr learned my addr 172.21.15.122:0/2836599748 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61effd640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200d22b0 0x7fa620147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61f7fe640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6200b41f0 0x7fa62015ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 -- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200d22b0 msgr2=0x7fa620147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200d22b0 0x7fa620147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 -- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6200b41f0 msgr2=0x7fa62015ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6200b41f0 0x7fa62015ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 -- 172.21.15.122:0/2836599748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620148430 con 0x7fa62015dfe0 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61effd640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200d22b0 0x7fa620147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:54.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61f7fe640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6200b41f0 0x7fa62015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.836+0000 7fa61ffff640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 0x7fa62014a110 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa618007e70 tx=0x7fa61800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.837+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618014620 con 0x7fa62015dfe0 2024-03-06T07:28:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.837+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618019070 con 0x7fa62015dfe0 2024-03-06T07:28:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.837+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618014a20 con 0x7fa62015dfe0 2024-03-06T07:28:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.837+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6200b9b90 con 0x7fa62015dfe0 2024-03-06T07:28:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.837+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6200b6c20 con 0x7fa62015dfe0 2024-03-06T07:28:54.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.838+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62004f330 con 0x7fa62015dfe0 2024-03-06T07:28:54.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.839+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa618014bc0 con 0x7fa62015dfe0 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.840+0000 7fa607fff640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa60c06b4d0 0x7fa60c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.840+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa61808ab00 con 0x7fa62015dfe0 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.840+0000 7fa61f7fe640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa60c06b4d0 0x7fa60c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.840+0000 7fa61f7fe640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa60c06b4d0 0x7fa60c06d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa61000a830 tx=0x7fa610008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:54.848+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa618059980 con 0x7fa62015dfe0 2024-03-06T07:28:54.930 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:54.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.103+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6200cea80 con 0x7fa62015dfe0 2024-03-06T07:28:55.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.103+0000 7fa607fff640 1 -- 172.21.15.122:0/2836599748 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5010 (secure 0 0 0) 0x7fa618059280 con 0x7fa62015dfe0 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/1547452856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1547452856},{"type":"v1","addr":"172.21.15.169:6834","nonce":1547452856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6896,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/1798905908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":1798905908},{"type":"v1","addr":"172.21.15.169:6835","nonce":1798905908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:49.978484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8685,8688]},"id":9}]} 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 75 2024-03-06T07:28:55.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa60c06b4d0 msgr2=0x7fa60c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa60c06b4d0 0x7fa60c06d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa61000a830 tx=0x7fa610008040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 msgr2=0x7fa62014a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 0x7fa62014a110 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa618007e70 tx=0x7fa61800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 shutdown_connections 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa60c06b4d0 0x7fa60c06d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa62015dfe0 0x7fa62014a110 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6200d22b0 0x7fa620147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 --2- 172.21.15.122:0/2836599748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6200b41f0 0x7fa62015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 >> 172.21.15.122:0/2836599748 conn(0x7fa6200d3cd0 msgr2=0x7fa6200b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 shutdown_connections 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.106+0000 7fa62600e640 1 -- 172.21.15.122:0/2836599748 wait complete. 2024-03-06T07:28:55.119 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and a 2024-03-06T07:28:55.120 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:55.120 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:55.131 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:55.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:28:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.262+0000 7fd9c77c7640 1 Processor -- start 2024-03-06T07:28:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.263+0000 7fd9c77c7640 1 -- start start 2024-03-06T07:28:55.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.263+0000 7fd9c77c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01613f0 0x7fd9c00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.263+0000 7fd9c77c7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 0x7fd9c006d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.263+0000 7fd9c77c7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c00b5ef0 con 0x7fd9c01617d0 2024-03-06T07:28:55.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.263+0000 7fd9c77c7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c006c0a0 con 0x7fd9c00b5320 2024-03-06T07:28:55.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.263+0000 7fd9c77c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00d0b50 con 0x7fd9c01613f0 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5d64640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 0x7fd9c006d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5563640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01613f0 0x7fd9c00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5d64640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 0x7fd9c006d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:48148/0 (socket says 172.21.15.122:48148) 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c4d62640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd9c01617d0 0x7fd9c00ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:50590/0 (socket says 172.21.15.122:50590) 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5d64640 1 -- 172.21.15.122:0/3403439677 learned_addr learned my addr 172.21.15.122:0/3403439677 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5563640 1 -- 172.21.15.122:0/3403439677 >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 msgr2=0x7fd9c006d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5563640 1 --2- 172.21.15.122:0/3403439677 >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 0x7fd9c006d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5563640 1 -- 172.21.15.122:0/3403439677 >> v1:172.21.15.169:6789/0 conn(0x7fd9c01617d0 legacy=0x7fd9c00ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5563640 1 -- 172.21.15.122:0/3403439677 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c00b5bf0 con 0x7fd9c01613f0 2024-03-06T07:28:55.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.264+0000 7fd9c5d64640 1 --2- 172.21.15.122:0/3403439677 >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 0x7fd9c006d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.265+0000 7fd9c5563640 1 --2- 172.21.15.122:0/3403439677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01613f0 0x7fd9c00cca20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0097a0 tx=0x7fd9bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c51954af6faba3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.265+0000 7fd9b7fff640 1 -- 172.21.15.122:0/3403439677 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc035020 con 0x7fd9c01613f0 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.265+0000 7fd9b7fff640 1 -- 172.21.15.122:0/3403439677 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc031830 con 0x7fd9c01613f0 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.265+0000 7fd9b7fff640 1 -- 172.21.15.122:0/3403439677 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc031c00 con 0x7fd9c01613f0 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3403439677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01613f0 msgr2=0x7fd9c00cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3403439677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01613f0 0x7fd9c00cca20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0097a0 tx=0x7fd9bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3403439677 shutdown_connections 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3403439677 >> v2:172.21.15.169:3300/0 conn(0x7fd9c00b5320 0x7fd9c006d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3403439677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01613f0 0x7fd9c00cca20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3403439677 >> 172.21.15.122:0/3403439677 conn(0x7fd9c0072520 msgr2=0x7fd9c0072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3403439677 shutdown_connections 2024-03-06T07:28:55.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.266+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3403439677 wait complete. 2024-03-06T07:28:55.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.268+0000 7fd9c77c7640 1 Processor -- start 2024-03-06T07:28:55.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.268+0000 7fd9c77c7640 1 -- start start 2024-03-06T07:28:55.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c77c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 0x7fd9c015d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c77c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9c01613f0 0x7fd9c015e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c77c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01617d0 0x7fd9c015e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c77c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00cd7f0 con 0x7fd9c00b5320 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c77c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00cd670 con 0x7fd9c01613f0 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c77c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c006de20 con 0x7fd9c01617d0 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c5563640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 0x7fd9c015d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c5563640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 0x7fd9c015d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54510/0 (socket says 172.21.15.122:54510) 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c5563640 1 -- 172.21.15.122:0/3101243106 learned_addr learned my addr 172.21.15.122:0/3101243106 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c5d64640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01617d0 0x7fd9c015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.269+0000 7fd9c4d62640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9c01613f0 0x7fd9c015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5563640 1 -- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01617d0 msgr2=0x7fd9c015e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5563640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01617d0 0x7fd9c015e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5563640 1 -- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9c01613f0 msgr2=0x7fd9c015e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5563640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9c01613f0 0x7fd9c015e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5563640 1 -- 172.21.15.122:0/3101243106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c00c8730 con 0x7fd9c00b5320 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5d64640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01617d0 0x7fd9c015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c4d62640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9c01613f0 0x7fd9c015e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c5563640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 0x7fd9c015d290 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009890 tx=0x7fd9bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc035040 con 0x7fd9c00b5320 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c00c8a10 con 0x7fd9c00b5320 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc03b920 con 0x7fd9c00b5320 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc03bd10 con 0x7fd9c00b5320 2024-03-06T07:28:55.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.270+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c00c8fd0 con 0x7fd9c00b5320 2024-03-06T07:28:55.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.271+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c004f330 con 0x7fd9c00b5320 2024-03-06T07:28:55.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.272+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9bc038020 con 0x7fd9c00b5320 2024-03-06T07:28:55.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.272+0000 7fd9b5ffb640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd9a406b4d0 0x7fd9a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.272+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9bc0ac3e0 con 0x7fd9c00b5320 2024-03-06T07:28:55.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.272+0000 7fd9c4d62640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd9a406b4d0 0x7fd9a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.273+0000 7fd9c4d62640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd9a406b4d0 0x7fd9a406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0007fd0 tx=0x7fd9b0007d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.277+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9bc044380 con 0x7fd9c00b5320 2024-03-06T07:28:55.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:55.542+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fd9c00c3d60 con 0x7fd9c00b5320 2024-03-06T07:28:56.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.242+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3101243106 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6896 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd9bc031800 con 0x7fd9c00b5320 2024-03-06T07:28:56.244 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6896 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd9a406b4d0 msgr2=0x7fd9a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd9a406b4d0 0x7fd9a406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0007fd0 tx=0x7fd9b0007d40 comp rx=0 tx=0).stop 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 msgr2=0x7fd9c015d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 0x7fd9c015d290 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009890 tx=0x7fd9bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 shutdown_connections 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd9a406b4d0 0x7fd9a406d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9c01617d0 0x7fd9c015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9c01613f0 0x7fd9c015e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 --2- 172.21.15.122:0/3101243106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9c00b5320 0x7fd9c015d290 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 >> 172.21.15.122:0/3101243106 conn(0x7fd9c0072520 msgr2=0x7fd9c015f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 shutdown_connections 2024-03-06T07:28:56.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.245+0000 7fd9c77c7640 1 -- 172.21.15.122:0/3101243106 wait complete. 2024-03-06T07:28:56.258 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:56.258 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:56.269 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:56.270 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:28:56.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.370+0000 7f1caceeb640 1 Processor -- start 2024-03-06T07:28:56.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.371+0000 7f1caceeb640 1 -- start start 2024-03-06T07:28:56.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.371+0000 7f1caceeb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1ca8070fe0 0x7f1ca8165b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1caceeb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca806cd60 con 0x7f1ca815b380 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1caceeb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca806d030 con 0x7f1ca8158880 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1caceeb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca806d310 con 0x7f1ca8070fe0 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca5d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1ca815b380 0x7f1ca80708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40594/0 (socket says 172.21.15.122:40594) 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/238205219 learned_addr learned my addr 172.21.15.122:0/238205219 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656451090 0 0) 0x7f1ca806cd60 con 0x7f1ca815b380 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca6d9d640 1 --2- 172.21.15.122:0/238205219 >> v2:172.21.15.169:3300/0 conn(0x7f1ca8070fe0 0x7f1ca8165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c90003520 con 0x7f1ca815b380 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1516606801 0 0) 0x7f1c90003520 con 0x7f1ca815b380 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 >> v2:172.21.15.169:3300/0 conn(0x7f1ca8070fe0 msgr2=0x7f1ca8165b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 --2- 172.21.15.122:0/238205219 >> v2:172.21.15.169:3300/0 conn(0x7f1ca8070fe0 0x7f1ca8165b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 >> v1:172.21.15.169:6789/0 conn(0x7f1ca8158880 legacy=0x7f1ca806c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca806d5f0 con 0x7f1ca815b380 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f1c9c001670 con 0x7f1ca815b380 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 >> v1:172.21.15.122:6789/0 conn(0x7f1ca815b380 legacy=0x7f1ca80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c900039b0 0x7f1c90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c90004800 0x7f1c90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c900051c0 0x7f1c9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca806d310 con 0x7f1c90004800 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9000e270 con 0x7f1c900039b0 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9000c030 con 0x7f1c900051c0 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.372+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c90004800 0x7f1c90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca659c640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c900039b0 0x7f1c90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c900051c0 msgr2=0x7f1c9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c900051c0 0x7f1c9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c900039b0 msgr2=0x7f1c90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c900039b0 0x7f1c90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/238205219 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90003520 con 0x7f1c90004800 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca6d9d640 1 -- 172.21.15.122:0/238205219 reap_dead start 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca659c640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c900039b0 0x7f1c90003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.373+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c90004800 0x7f1c90004c80 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c240 tx=0x7f1c90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc793d2f058757d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c007d10 con 0x7f1c90004800 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c030070 con 0x7f1c90004800 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1ca559a640 1 -- 172.21.15.122:0/238205219 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c005d90 con 0x7f1c90004800 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 -- 172.21.15.122:0/238205219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c90004800 msgr2=0x7f1c90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c90004800 0x7f1c90004c80 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c240 tx=0x7f1c90005830 comp rx=0 tx=0).stop 2024-03-06T07:28:56.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 -- 172.21.15.122:0/238205219 shutdown_connections 2024-03-06T07:28:56.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 --2- 172.21.15.122:0/238205219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c90004800 0x7f1c90004c80 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 -- 172.21.15.122:0/238205219 >> 172.21.15.122:0/238205219 conn(0x7f1ca80b1c80 msgr2=0x7f1ca80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 -- 172.21.15.122:0/238205219 shutdown_connections 2024-03-06T07:28:56.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.374+0000 7f1caceeb640 1 -- 172.21.15.122:0/238205219 wait complete. 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.376+0000 7f1caceeb640 1 Processor -- start 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.376+0000 7f1caceeb640 1 -- start start 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.377+0000 7f1caceeb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c90004800 0x7f1ca8146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.377+0000 7f1caceeb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 0x7f1ca815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.377+0000 7f1caceeb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ca815d1f0 0x7f1ca8148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.377+0000 7f1caceeb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca806c650 con 0x7f1ca80aa180 2024-03-06T07:28:56.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.377+0000 7f1caceeb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca806dbb0 con 0x7f1ca815d1f0 2024-03-06T07:28:56.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.377+0000 7f1caceeb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca806c7d0 con 0x7f1c90004800 2024-03-06T07:28:56.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 0x7f1ca815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 0x7f1ca815ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54528/0 (socket says 172.21.15.122:54528) 2024-03-06T07:28:56.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/967625104 learned_addr learned my addr 172.21.15.122:0/967625104 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:56.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca6d9d640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ca815d1f0 0x7f1ca8148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca659c640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c90004800 0x7f1ca8146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c90004800 msgr2=0x7f1ca8146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c90004800 0x7f1ca8146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ca815d1f0 msgr2=0x7f1ca8148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ca815d1f0 0x7f1ca8148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 -- 172.21.15.122:0/967625104 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca8149900 con 0x7f1ca80aa180 2024-03-06T07:28:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1ca5d9b640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 0x7f1ca815ccb0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c240 tx=0x7f1ca806d200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c045070 con 0x7f1ca80aa180 2024-03-06T07:28:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c004bf0 con 0x7f1ca80aa180 2024-03-06T07:28:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca80a96a0 con 0x7f1ca80aa180 2024-03-06T07:28:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c007d10 con 0x7f1ca80aa180 2024-03-06T07:28:56.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.378+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca80a9c60 con 0x7f1ca80aa180 2024-03-06T07:28:56.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.379+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca80ce660 con 0x7f1ca80aa180 2024-03-06T07:28:56.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.380+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c9c007070 con 0x7f1ca80aa180 2024-03-06T07:28:56.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.381+0000 7f1c96ffd640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c8806b4d0 0x7f1c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.381+0000 7f1ca659c640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c8806b4d0 0x7f1c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.381+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c9c0ab020 con 0x7f1ca80aa180 2024-03-06T07:28:56.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.381+0000 7f1ca659c640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c8806b4d0 0x7f1c8806d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1c98000c00 tx=0x7f1c98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.385+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ca80ce660 con 0x7f1ca80aa180 2024-03-06T07:28:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:56.634+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f1ca804f330 con 0x7f1ca80aa180 2024-03-06T07:28:57.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.243+0000 7f1c96ffd640 1 -- 172.21.15.122:0/967625104 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6893 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7f1c9c0799c0 con 0x7f1ca80aa180 2024-03-06T07:28:57.245 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 6893 2024-03-06T07:28:57.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c8806b4d0 msgr2=0x7f1c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c8806b4d0 0x7f1c8806d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1c98000c00 tx=0x7f1c98009040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 msgr2=0x7f1ca815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 0x7f1ca815ccb0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c240 tx=0x7f1ca806d200 comp rx=0 tx=0).stop 2024-03-06T07:28:57.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 shutdown_connections 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c8806b4d0 0x7f1c8806d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ca815d1f0 0x7f1ca8148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ca80aa180 0x7f1ca815ccb0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 --2- 172.21.15.122:0/967625104 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c90004800 0x7f1ca8146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 >> 172.21.15.122:0/967625104 conn(0x7f1ca80b1c80 msgr2=0x7f1ca80cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 shutdown_connections 2024-03-06T07:28:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.245+0000 7f1caceeb640 1 -- 172.21.15.122:0/967625104 wait complete. 2024-03-06T07:28:57.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:57.258 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:28:57.258 DEBUG:teuthology.orchestra.run.smithi122:> mkdir pin 2024-03-06T07:28:57.258 DEBUG:teuthology.orchestra.run.smithi122:> touch pin/placeholder 2024-03-06T07:28:57.258 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:28:57.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:57.283 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:28:57.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:57.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.480+0000 7fe240281640 1 Processor -- start 2024-03-06T07:28:57.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.480+0000 7fe240281640 1 -- start start 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe240281640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe2380d22b0 0x7fe2380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe240281640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2380b41f0 0x7fe2380b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe240281640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2380b3430 con 0x7fe23815dfe0 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe240281640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2380d2c50 con 0x7fe2380b41f0 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe240281640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2380b31e0 con 0x7fe2380d22b0 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe23e81e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2380b41f0 0x7fe2380b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.481+0000 7fe23e01d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe23815dfe0 0x7fe2380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40602/0 (socket says 172.21.15.122:40602) 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23e01d640 1 -- 172.21.15.122:0/759219149 learned_addr learned my addr 172.21.15.122:0/759219149 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23d81c640 1 --2- 172.21.15.122:0/759219149 >> v2:172.21.15.169:3300/0 conn(0x7fe2380d22b0 0x7fe2380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23e81e640 1 -- 172.21.15.122:0/759219149 >> v2:172.21.15.169:3300/0 conn(0x7fe2380d22b0 msgr2=0x7fe2380d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23e81e640 1 --2- 172.21.15.122:0/759219149 >> v2:172.21.15.169:3300/0 conn(0x7fe2380d22b0 0x7fe2380d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23e81e640 1 -- 172.21.15.122:0/759219149 >> v1:172.21.15.122:6789/0 conn(0x7fe23815dfe0 legacy=0x7fe2380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23e81e640 1 -- 172.21.15.122:0/759219149 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380b3710 con 0x7fe2380b41f0 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.482+0000 7fe23d01b640 1 -- 172.21.15.122:0/759219149 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059771392 0 0) 0x7fe2380b3430 con 0x7fe23815dfe0 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.483+0000 7fe23e81e640 1 --2- 172.21.15.122:0/759219149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe2380b4680 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fe230009d90 tx=0x7fe23002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7fd8c9e8e6d6ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.483+0000 7fe23d01b640 1 -- 172.21.15.122:0/759219149 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230039070 con 0x7fe2380b41f0 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.483+0000 7fe23d01b640 1 -- 172.21.15.122:0/759219149 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23002cc60 con 0x7fe2380b41f0 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.483+0000 7fe23d01b640 1 -- 172.21.15.122:0/759219149 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230034660 con 0x7fe2380b41f0 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 -- 172.21.15.122:0/759219149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 msgr2=0x7fe2380b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 --2- 172.21.15.122:0/759219149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe2380b4680 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fe230009d90 tx=0x7fe23002beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 -- 172.21.15.122:0/759219149 shutdown_connections 2024-03-06T07:28:57.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 --2- 172.21.15.122:0/759219149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe2380b4680 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 --2- 172.21.15.122:0/759219149 >> v2:172.21.15.169:3300/0 conn(0x7fe2380d22b0 0x7fe2380d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 -- 172.21.15.122:0/759219149 >> 172.21.15.122:0/759219149 conn(0x7fe2380d3cd0 msgr2=0x7fe2380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 -- 172.21.15.122:0/759219149 shutdown_connections 2024-03-06T07:28:57.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.484+0000 7fe240281640 1 -- 172.21.15.122:0/759219149 wait complete. 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.486+0000 7fe240281640 1 Processor -- start 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.486+0000 7fe240281640 1 -- start start 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe240281640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe23815b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe240281640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2380d22b0 0x7fe2380cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe240281640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe23815dfe0 0x7fe2380cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe240281640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380cfa00 con 0x7fe2380b41f0 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe240281640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380b3d60 con 0x7fe23815dfe0 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe240281640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380d1c70 con 0x7fe2380d22b0 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe23815b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe23815b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54542/0 (socket says 172.21.15.122:54542) 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 -- 172.21.15.122:0/2545999270 learned_addr learned my addr 172.21.15.122:0/2545999270 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23d81c640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2380d22b0 0x7fe2380cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e81e640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe23815dfe0 0x7fe2380cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 -- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2380d22b0 msgr2=0x7fe2380cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2380d22b0 0x7fe2380cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 -- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe23815dfe0 msgr2=0x7fe2380cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe23815dfe0 0x7fe2380cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.487+0000 7fe23e01d640 1 -- 172.21.15.122:0/2545999270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380cd9c0 con 0x7fe2380b41f0 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.488+0000 7fe23e01d640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe23815b400 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c460 tx=0x7fe23400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.488+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234018070 con 0x7fe2380b41f0 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.488+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234007500 con 0x7fe2380b41f0 2024-03-06T07:28:57.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.488+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234004a20 con 0x7fe2380b41f0 2024-03-06T07:28:57.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.488+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2380cd150 con 0x7fe2380b41f0 2024-03-06T07:28:57.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.488+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2380cce70 con 0x7fe2380b41f0 2024-03-06T07:28:57.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.489+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23804f330 con 0x7fe2380b41f0 2024-03-06T07:28:57.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.489+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe23401b070 con 0x7fe2380b41f0 2024-03-06T07:28:57.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.490+0000 7fe22effd640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe21c06b4d0 0x7fe21c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.490+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe234089320 con 0x7fe2380b41f0 2024-03-06T07:28:57.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.490+0000 7fe23d81c640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe21c06b4d0 0x7fe21c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.490+0000 7fe23d81c640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe21c06b4d0 0x7fe21c06d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe2380b2ce0 tx=0x7fe228008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.494+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2380cd9c0 con 0x7fe2380b41f0 2024-03-06T07:28:57.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.711+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2380d0c20 con 0x7fe2380b41f0 2024-03-06T07:28:57.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.712+0000 7fe22effd640 1 -- 172.21.15.122:0/2545999270 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+3423 (secure 0 0 0) 0x7fe234057be0 con 0x7fe2380b41f0 2024-03-06T07:28:57.714 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:28:57.714 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:49.978484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8685,8688]},"id":9}]} 2024-03-06T07:28:57.715 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 77 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe21c06b4d0 msgr2=0x7fe21c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe21c06b4d0 0x7fe21c06d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe2380b2ce0 tx=0x7fe228008040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 msgr2=0x7fe23815b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe23815b400 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c460 tx=0x7fe23400c920 comp rx=0 tx=0).stop 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 shutdown_connections 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe21c06b4d0 0x7fe21c06d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe23815dfe0 0x7fe2380cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe2380d22b0 0x7fe2380cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 --2- 172.21.15.122:0/2545999270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2380b41f0 0x7fe23815b400 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 >> 172.21.15.122:0/2545999270 conn(0x7fe2380d3cd0 msgr2=0x7fe23815d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 shutdown_connections 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:57.714+0000 7fe240281640 1 -- 172.21.15.122:0/2545999270 wait complete. 2024-03-06T07:28:57.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:28:57.843 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:28:57.843 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:28:59.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:59.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.943+0000 7f5737c1c640 1 Processor -- start 2024-03-06T07:28:59.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.943+0000 7f5737c1c640 1 -- start start 2024-03-06T07:28:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.944+0000 7f5737c1c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f57300cfff0 0x7f57300d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.944+0000 7f5737c1c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5730161980 con 0x7f57300cc8b0 2024-03-06T07:28:59.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.944+0000 7f5737c1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5730161c60 con 0x7f573015bb20 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.944+0000 7f5737c1c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57300d2500 con 0x7f57300cfff0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57361b9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f57300cc8b0 0x7f57300ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40614/0 (socket says 172.21.15.122:40614) 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57361b9640 1 -- 172.21.15.122:0/3016114251 learned_addr learned my addr 172.21.15.122:0/3016114251 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249681702 0 0) 0x7f5730161980 con 0x7f57300cc8b0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57351b7640 1 --2- 172.21.15.122:0/3016114251 >> v2:172.21.15.169:3300/0 conn(0x7f57300cfff0 0x7f57300d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c003520 con 0x7f57300cc8b0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 476830053 0 0) 0x7f571c003520 con 0x7f57300cc8b0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 >> v2:172.21.15.169:3300/0 conn(0x7f57300cfff0 msgr2=0x7f57300d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 --2- 172.21.15.122:0/3016114251 >> v2:172.21.15.169:3300/0 conn(0x7f57300cfff0 0x7f57300d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 >> v1:172.21.15.169:6789/0 conn(0x7f573015bb20 legacy=0x7f57300d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730161f40 con 0x7f57300cc8b0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f57280019f0 con 0x7f57300cc8b0 2024-03-06T07:28:59.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 >> v1:172.21.15.122:6789/0 conn(0x7f57300cc8b0 legacy=0x7f57300ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f571c004800 0x7f571c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f571c0051c0 0x7f571c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57300d2500 con 0x7f571c0039b0 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c00c060 con 0x7f571c0051c0 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c00c260 con 0x7f571c004800 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57359b8640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57351b7640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f571c004800 0x7f571c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.945+0000 7f57361b9640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f571c0051c0 0x7f571c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57359b8640 1 -- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f571c004800 msgr2=0x7f571c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57359b8640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f571c004800 0x7f571c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57359b8640 1 -- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f571c0051c0 msgr2=0x7f571c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57359b8640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f571c0051c0 0x7f571c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57359b8640 1 -- 172.21.15.122:0/3016114251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c003520 con 0x7f571c0039b0 2024-03-06T07:28:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57361b9640 1 -- 172.21.15.122:0/3016114251 reap_dead start 2024-03-06T07:28:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57359b8640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f5720001750 tx=0x7f571c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4ef21abeb6e10e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.946+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720001370 con 0x7f571c0039b0 2024-03-06T07:28:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57200052d0 con 0x7f571c0039b0 2024-03-06T07:28:59.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f57349b6640 1 -- 172.21.15.122:0/3016114251 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720003830 con 0x7f571c0039b0 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 -- 172.21.15.122:0/3016114251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 msgr2=0x7f571c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f5720001750 tx=0x7f571c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 -- 172.21.15.122:0/3016114251 shutdown_connections 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 --2- 172.21.15.122:0/3016114251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 -- 172.21.15.122:0/3016114251 >> 172.21.15.122:0/3016114251 conn(0x7f573009a9f0 msgr2=0x7f573009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 -- 172.21.15.122:0/3016114251 shutdown_connections 2024-03-06T07:28:59.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.947+0000 7f5737c1c640 1 -- 172.21.15.122:0/3016114251 wait complete. 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.948+0000 7f5737c1c640 1 Processor -- start 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.948+0000 7f5737c1c640 1 -- start start 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f5737c1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f573015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f5737c1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573006d6f0 0x7f57301f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f5737c1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f57301566c0 0x7f57301eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f5737c1c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57300d1980 con 0x7f571c0039b0 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f5737c1c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5730162500 con 0x7f573006d6f0 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f5737c1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57300d1b00 con 0x7f57301566c0 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57359b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f573015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57359b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f573015bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54568/0 (socket says 172.21.15.122:54568) 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57359b8640 1 -- 172.21.15.122:0/294356352 learned_addr learned my addr 172.21.15.122:0/294356352 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57351b7640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573006d6f0 0x7f57301f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57361b9640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f57301566c0 0x7f57301eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57359b8640 1 -- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f57301566c0 msgr2=0x7f57301eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.949+0000 7f57359b8640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f57301566c0 0x7f57301eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57359b8640 1 -- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573006d6f0 msgr2=0x7f57301f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57359b8640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573006d6f0 0x7f57301f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57359b8640 1 -- 172.21.15.122:0/294356352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57301f5260 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57351b7640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573006d6f0 0x7f57301f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57359b8640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f573015bf90 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5720005040 tx=0x7f5720001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720041020 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5720005480 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720007a30 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573006dd60 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.950+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57301f4520 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.951+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573006dad0 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.952+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5720044020 con 0x7f571c0039b0 2024-03-06T07:28:59.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.952+0000 7f57267fc640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f571406b4d0 0x7f571406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.952+0000 7f57351b7640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f571406b4d0 0x7f571406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.952+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57200abc40 con 0x7f571c0039b0 2024-03-06T07:28:59.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.953+0000 7f57351b7640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f571406b4d0 0x7f571406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f57300b0ae0 tx=0x7f572c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:28:59.956+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f572004b030 con 0x7f571c0039b0 2024-03-06T07:29:00.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.197+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f573004f330 con 0x7f571c0039b0 2024-03-06T07:29:00.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.198+0000 7f57267fc640 1 -- 172.21.15.122:0/294356352 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+3423 (secure 0 0 0) 0x7f57301f5260 con 0x7f571c0039b0 2024-03-06T07:29:00.200 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:00.200 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:49.978484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8685,8688]},"id":9}]} 2024-03-06T07:29:00.200 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 77 2024-03-06T07:29:00.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f571406b4d0 msgr2=0x7f571406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f571406b4d0 0x7f571406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f57300b0ae0 tx=0x7f572c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:00.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 msgr2=0x7f573015bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f573015bf90 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5720005040 tx=0x7f5720001120 comp rx=0 tx=0).stop 2024-03-06T07:29:00.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 shutdown_connections 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f571406b4d0 0x7f571406d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f57301566c0 0x7f57301eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573006d6f0 0x7f57301f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 --2- 172.21.15.122:0/294356352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f571c0039b0 0x7f573015bf90 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 >> 172.21.15.122:0/294356352 conn(0x7f573009a9f0 msgr2=0x7f57300cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 shutdown_connections 2024-03-06T07:29:00.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:00.200+0000 7f5737c1c640 1 -- 172.21.15.122:0/294356352 wait complete. 2024-03-06T07:29:00.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:00.377 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:00.378 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:00.378 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:02.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.496+0000 7f649795e640 1 Processor -- start 2024-03-06T07:29:02.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.496+0000 7f649795e640 1 -- start start 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f649795e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64900d22b0 0x7f64900d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f649795e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f64900b41f0 0x7f64900b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f649795e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900b3430 con 0x7f649015dfe0 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f649795e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64900d2c50 con 0x7f64900d22b0 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f649795e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64900b31e0 con 0x7f64900b41f0 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6494ef9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64900d22b0 0x7f64900d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f64956fa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f649015dfe0 0x7f64900cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40622/0 (socket says 172.21.15.122:40622) 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6494ef9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64900d22b0 0x7f64900d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54584/0 (socket says 172.21.15.122:54584) 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f64956fa640 1 -- 172.21.15.122:0/3374118955 learned_addr learned my addr 172.21.15.122:0/3374118955 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6495efb640 1 --2- 172.21.15.122:0/3374118955 >> v2:172.21.15.169:3300/0 conn(0x7f64900b41f0 0x7f64900b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6494ef9640 1 -- 172.21.15.122:0/3374118955 >> v2:172.21.15.169:3300/0 conn(0x7f64900b41f0 msgr2=0x7f64900b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6494ef9640 1 --2- 172.21.15.122:0/3374118955 >> v2:172.21.15.169:3300/0 conn(0x7f64900b41f0 0x7f64900b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6494ef9640 1 -- 172.21.15.122:0/3374118955 >> v1:172.21.15.122:6789/0 conn(0x7f649015dfe0 legacy=0x7f64900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.497+0000 7f6494ef9640 1 -- 172.21.15.122:0/3374118955 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900b3710 con 0x7f64900d22b0 2024-03-06T07:29:02.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.498+0000 7f6495efb640 1 --2- 172.21.15.122:0/3374118955 >> v2:172.21.15.169:3300/0 conn(0x7f64900b41f0 0x7f64900b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.498+0000 7f6494ef9640 1 --2- 172.21.15.122:0/3374118955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64900d22b0 0x7f64900d2710 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f64800097a0 tx=0x7f648002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9f5e34a949dc80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.498+0000 7f6487fff640 1 -- 172.21.15.122:0/3374118955 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480035020 con 0x7f64900d22b0 2024-03-06T07:29:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.498+0000 7f6487fff640 1 -- 172.21.15.122:0/3374118955 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6480031830 con 0x7f64900d22b0 2024-03-06T07:29:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.498+0000 7f6487fff640 1 -- 172.21.15.122:0/3374118955 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480031c00 con 0x7f64900d22b0 2024-03-06T07:29:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 -- 172.21.15.122:0/3374118955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64900d22b0 msgr2=0x7f64900d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 --2- 172.21.15.122:0/3374118955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64900d22b0 0x7f64900d2710 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f64800097a0 tx=0x7f648002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 -- 172.21.15.122:0/3374118955 shutdown_connections 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 --2- 172.21.15.122:0/3374118955 >> v2:172.21.15.169:3300/0 conn(0x7f64900b41f0 0x7f64900b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 --2- 172.21.15.122:0/3374118955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64900d22b0 0x7f64900d2710 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 -- 172.21.15.122:0/3374118955 >> 172.21.15.122:0/3374118955 conn(0x7f64900d3cd0 msgr2=0x7f64900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 -- 172.21.15.122:0/3374118955 shutdown_connections 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.499+0000 7f649795e640 1 -- 172.21.15.122:0/3374118955 wait complete. 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.500+0000 7f649795e640 1 Processor -- start 2024-03-06T07:29:02.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.500+0000 7f649795e640 1 -- start start 2024-03-06T07:29:02.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f649795e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64900b41f0 0x7f64901f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f649795e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64900d22b0 0x7f64900b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f649795e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 0x7f64900b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f649795e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64900cfa00 con 0x7f649015dfe0 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f649795e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64900b3d60 con 0x7f64900b41f0 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f649795e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64900d1c70 con 0x7f64900d22b0 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 0x7f64900b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 0x7f64900b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54594/0 (socket says 172.21.15.122:54594) 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 -- 172.21.15.122:0/2302749382 learned_addr learned my addr 172.21.15.122:0/2302749382 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6494ef9640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64900d22b0 0x7f64900b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f64956fa640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64900b41f0 0x7f64901f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 -- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64900d22b0 msgr2=0x7f64900b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64900d22b0 0x7f64900b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 -- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64900b41f0 msgr2=0x7f64901f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64900b41f0 0x7f64901f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 -- 172.21.15.122:0/2302749382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900b1fe0 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f64956fa640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64900b41f0 0x7f64901f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.501+0000 7f6495efb640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 0x7f64900b6de0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f648800cf20 tx=0x7f648800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.502+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648801a070 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.502+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648800d840 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.502+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64900b22c0 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.502+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800dc30 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.502+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64900b2880 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.503+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649004f330 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.503+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f648801d070 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.504+0000 7f6485ffb640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f647406b4d0 0x7f647406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.504+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f648808aa30 con 0x7f649015dfe0 2024-03-06T07:29:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.504+0000 7f64956fa640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f647406b4d0 0x7f647406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.504+0000 7f64956fa640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f647406b4d0 0x7f647406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f648c000f60 tx=0x7f648c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.512+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6488059940 con 0x7f649015dfe0 2024-03-06T07:29:02.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.728+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64900d0a10 con 0x7f649015dfe0 2024-03-06T07:29:02.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.728+0000 7f6485ffb640 1 -- 172.21.15.122:0/2302749382 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+3423 (secure 0 0 0) 0x7f6488059240 con 0x7f649015dfe0 2024-03-06T07:29:02.731 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:02.731 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:49.978484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8685,8688]},"id":9}]} 2024-03-06T07:29:02.731 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 77 2024-03-06T07:29:02.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f647406b4d0 msgr2=0x7f647406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f647406b4d0 0x7f647406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f648c000f60 tx=0x7f648c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 msgr2=0x7f64900b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 0x7f64900b6de0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f648800cf20 tx=0x7f648800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 shutdown_connections 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f647406b4d0 0x7f647406d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f649015dfe0 0x7f64900b6de0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64900d22b0 0x7f64900b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 --2- 172.21.15.122:0/2302749382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64900b41f0 0x7f64901f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 >> 172.21.15.122:0/2302749382 conn(0x7f64900d3cd0 msgr2=0x7f64900cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 shutdown_connections 2024-03-06T07:29:02.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:02.731+0000 7f649795e640 1 -- 172.21.15.122:0/2302749382 wait complete. 2024-03-06T07:29:02.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:02.859 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:02.860 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:02.860 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:04.860 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:04.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.959+0000 7f2d72a9a640 1 Processor -- start 2024-03-06T07:29:04.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.960+0000 7f2d72a9a640 1 -- start start 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d72a9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c15dfe0 0x7f2d6c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d72a9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d6c0cf830 0x7f2d6c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d72a9a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2d6c0d27f0 0x7f2d6c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d72a9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6c0d2c50 con 0x7f2d6c0cf830 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d72a9a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6c0b31e0 con 0x7f2d6c0d27f0 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d72a9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0b3d60 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d6c0cf830 0x7f2d6c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d6c0cf830 0x7f2d6c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58952/0 (socket says 172.21.15.122:58952) 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 -- 172.21.15.122:0/115633225 learned_addr learned my addr 172.21.15.122:0/115633225 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d70836640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c15dfe0 0x7f2d6c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d71037640 1 --2- 172.21.15.122:0/115633225 >> v2:172.21.15.169:3300/0 conn(0x7f2d6c0d27f0 0x7f2d6c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 -- 172.21.15.122:0/115633225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c15dfe0 msgr2=0x7f2d6c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c15dfe0 0x7f2d6c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 -- 172.21.15.122:0/115633225 >> v2:172.21.15.169:3300/0 conn(0x7f2d6c0d27f0 msgr2=0x7f2d6c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 --2- 172.21.15.122:0/115633225 >> v2:172.21.15.169:3300/0 conn(0x7f2d6c0d27f0 0x7f2d6c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d6bfff640 1 -- 172.21.15.122:0/115633225 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c0b3430 con 0x7f2d6c0cf830 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d70836640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c15dfe0 0x7f2d6c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.961+0000 7f2d71037640 1 --2- 172.21.15.122:0/115633225 >> v2:172.21.15.169:3300/0 conn(0x7f2d6c0d27f0 0x7f2d6c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:04.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.962+0000 7f2d6bfff640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c0cf830 0x7f2d6c0d22b0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c067ad0 tx=0x7f2d5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdbf795e5490e58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.962+0000 7f2d6b7fe640 1 -- 172.21.15.122:0/115633225 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58035020 con 0x7f2d6c0cf830 2024-03-06T07:29:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.962+0000 7f2d6b7fe640 1 -- 172.21.15.122:0/115633225 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d58031870 con 0x7f2d6c0cf830 2024-03-06T07:29:04.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.962+0000 7f2d6b7fe640 1 -- 172.21.15.122:0/115633225 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58031c40 con 0x7f2d6c0cf830 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 -- 172.21.15.122:0/115633225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c0cf830 msgr2=0x7f2d6c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c0cf830 0x7f2d6c0d22b0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c067ad0 tx=0x7f2d5802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 -- 172.21.15.122:0/115633225 shutdown_connections 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c0cf830 0x7f2d6c0d22b0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/115633225 >> v2:172.21.15.169:3300/0 conn(0x7f2d6c0d27f0 0x7f2d6c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/115633225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c15dfe0 0x7f2d6c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 -- 172.21.15.122:0/115633225 >> 172.21.15.122:0/115633225 conn(0x7f2d6c0d3cd0 msgr2=0x7f2d6c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 -- 172.21.15.122:0/115633225 shutdown_connections 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.963+0000 7f2d72a9a640 1 -- 172.21.15.122:0/115633225 wait complete. 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.964+0000 7f2d72a9a640 1 Processor -- start 2024-03-06T07:29:04.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.964+0000 7f2d72a9a640 1 -- start start 2024-03-06T07:29:04.969 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d72a9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c0cf830 0x7f2d6c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d72a9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d6c0d27f0 0x7f2d6c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d72a9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 0x7f2d6c0b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d72a9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0d1c70 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d72a9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0b39f0 con 0x7f2d6c0d27f0 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d72a9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0d1df0 con 0x7f2d6c0cf830 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 0x7f2d6c0b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 0x7f2d6c0b6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58958/0 (socket says 172.21.15.122:58958) 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 -- 172.21.15.122:0/3075471187 learned_addr learned my addr 172.21.15.122:0/3075471187 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d70836640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c0cf830 0x7f2d6c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d6bfff640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d6c0d27f0 0x7f2d6c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 -- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c0cf830 msgr2=0x7f2d6c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c0cf830 0x7f2d6c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 -- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d6c0d27f0 msgr2=0x7f2d6c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d6c0d27f0 0x7f2d6c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d71037640 1 -- 172.21.15.122:0/3075471187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c0cd780 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.965+0000 7f2d6bfff640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d6c0d27f0 0x7f2d6c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.966+0000 7f2d71037640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 0x7f2d6c0b6fd0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400cef0 tx=0x7f2d6400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.966+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6401a070 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.966+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6400d840 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.966+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c0cd4a0 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.966+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400dc30 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.966+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6c15f3a0 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.967+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6c04f330 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.967+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d6401d070 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.968+0000 7f2d697fa640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5006b4d0 0x7f2d5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.968+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d6408a940 con 0x7f2d6c15dfe0 2024-03-06T07:29:04.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.968+0000 7f2d70836640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5006b4d0 0x7f2d5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.968+0000 7f2d70836640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5006b4d0 0x7f2d5006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000a950 tx=0x7f2d60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:04.976+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d64059850 con 0x7f2d6c15dfe0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.196+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d6c0b0af0 con 0x7f2d6c15dfe0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.196+0000 7f2d697fa640 1 -- 172.21.15.122:0/3075471187 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+3423 (secure 0 0 0) 0x7f2d64059150 con 0x7f2d6c15dfe0 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:28:49.978484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8685,8688]},"id":9}]} 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 77 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5006b4d0 msgr2=0x7f2d5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5006b4d0 0x7f2d5006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000a950 tx=0x7f2d60008040 comp rx=0 tx=0).stop 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 msgr2=0x7f2d6c0b6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 0x7f2d6c0b6fd0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400cef0 tx=0x7f2d6400a710 comp rx=0 tx=0).stop 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 shutdown_connections 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2d5006b4d0 0x7f2d5006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d6c15dfe0 0x7f2d6c0b6fd0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d6c0d27f0 0x7f2d6c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 --2- 172.21.15.122:0/3075471187 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2d6c0cf830 0x7f2d6c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 >> 172.21.15.122:0/3075471187 conn(0x7f2d6c0d3cd0 msgr2=0x7f2d6c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 shutdown_connections 2024-03-06T07:29:05.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:05.199+0000 7f2d72a9a640 1 -- 172.21.15.122:0/3075471187 wait complete. 2024-03-06T07:29:05.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:05.336 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:05.337 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:05.337 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:07.338 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:07.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.441+0000 7f3a7258c640 1 Processor -- start 2024-03-06T07:29:07.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.441+0000 7f3a7258c640 1 -- start start 2024-03-06T07:29:07.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.442+0000 7f3a7258c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3a6c15dfe0 0x7f3a6c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.442+0000 7f3a7258c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 0x7f3a6c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.442+0000 7f3a7258c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c0b3430 con 0x7f3a6c0cf830 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.442+0000 7f3a7258c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c0b31e0 con 0x7f3a6c15dfe0 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.442+0000 7f3a7258c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0b3d60 con 0x7f3a6c0d29c0 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a70b29640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 0x7f3a6c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a6bfff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3a6c15dfe0 0x7f3a6c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a70b29640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 0x7f3a6c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:50728/0 (socket says 172.21.15.122:50728) 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a6b7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3a6c0cf830 0x7f3a6c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:39312/0 (socket says 172.21.15.122:39312) 2024-03-06T07:29:07.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a70b29640 1 -- 172.21.15.122:0/1024118494 learned_addr learned my addr 172.21.15.122:0/1024118494 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a6bfff640 1 -- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 msgr2=0x7f3a6c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a6bfff640 1 --2- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 0x7f3a6c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a6bfff640 1 -- 172.21.15.122:0/1024118494 >> v1:172.21.15.169:6789/0 conn(0x7f3a6c0cf830 legacy=0x7f3a6c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a6bfff640 1 -- 172.21.15.122:0/1024118494 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0b3710 con 0x7f3a6c15dfe0 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.443+0000 7f3a70b29640 1 --2- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 0x7f3a6c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.444+0000 7f3a6bfff640 1 --2- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cf2f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a05969181aaa8f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.444+0000 7f3a6affd640 1 -- 172.21.15.122:0/1024118494 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c035020 con 0x7f3a6c15dfe0 2024-03-06T07:29:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.444+0000 7f3a6affd640 1 -- 172.21.15.122:0/1024118494 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c031830 con 0x7f3a6c15dfe0 2024-03-06T07:29:07.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.444+0000 7f3a6affd640 1 -- 172.21.15.122:0/1024118494 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c031c00 con 0x7f3a6c15dfe0 2024-03-06T07:29:07.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 -- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c15dfe0 msgr2=0x7f3a6c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 --2- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cf2f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 -- 172.21.15.122:0/1024118494 shutdown_connections 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 --2- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c0d29c0 0x7f3a6c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 --2- 172.21.15.122:0/1024118494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cf2f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 -- 172.21.15.122:0/1024118494 >> 172.21.15.122:0/1024118494 conn(0x7f3a6c0d3cd0 msgr2=0x7f3a6c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 -- 172.21.15.122:0/1024118494 shutdown_connections 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.445+0000 7f3a7258c640 1 -- 172.21.15.122:0/1024118494 wait complete. 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.446+0000 7f3a7258c640 1 Processor -- start 2024-03-06T07:29:07.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.446+0000 7f3a7258c640 1 -- start start 2024-03-06T07:29:07.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a7258c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 0x7f3a6c150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a7258c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c0d29c0 0x7f3a6c09f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a7258c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a7258c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0d1df0 con 0x7f3a6c0cf830 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a7258c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0d1c70 con 0x7f3a6c0d29c0 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a7258c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c15e3c0 con 0x7f3a6c15dfe0 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a6bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 0x7f3a6c150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a6bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 0x7f3a6c150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58970/0 (socket says 172.21.15.122:58970) 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.447+0000 7f3a6bfff640 1 -- 172.21.15.122:0/4013983846 learned_addr learned my addr 172.21.15.122:0/4013983846 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a70b29640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6b7fe640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c0d29c0 0x7f3a6c09f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6bfff640 1 -- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c15dfe0 msgr2=0x7f3a6c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6bfff640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6bfff640 1 -- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c0d29c0 msgr2=0x7f3a6c09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6bfff640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c0d29c0 0x7f3a6c09f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6bfff640 1 -- 172.21.15.122:0/4013983846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0cb9b0 con 0x7f3a6c0cf830 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a6bfff640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 0x7f3a6c150000 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c034040 tx=0x7f3a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c040070 con 0x7f3a6c0cf830 2024-03-06T07:29:07.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c03a910 con 0x7f3a6c0cf830 2024-03-06T07:29:07.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c0cbc90 con 0x7f3a6c0cf830 2024-03-06T07:29:07.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c035040 con 0x7f3a6c0cf830 2024-03-06T07:29:07.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.448+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c0cc250 con 0x7f3a6c0cf830 2024-03-06T07:29:07.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.449+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c04f330 con 0x7f3a6c0cf830 2024-03-06T07:29:07.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.449+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a5c038020 con 0x7f3a6c0cf830 2024-03-06T07:29:07.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.450+0000 7f3a68ff9640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a5806b500 0x7f3a5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.450+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a5c0abac0 con 0x7f3a6c0cf830 2024-03-06T07:29:07.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.450+0000 7f3a6b7fe640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a5806b500 0x7f3a5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.450+0000 7f3a6b7fe640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a5806b500 0x7f3a5806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0caa70 tx=0x7f3a60007cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.458+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a5c031800 con 0x7f3a6c0cf830 2024-03-06T07:29:07.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.672+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a6c0b5460 con 0x7f3a6c0cf830 2024-03-06T07:29:07.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.672+0000 7f3a68ff9640 1 -- 172.21.15.122:0/4013983846 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+3424 (secure 0 0 0) 0x7f3a5c07a8b0 con 0x7f3a6c0cf830 2024-03-06T07:29:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:07.675 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:07.105747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688,8685]},"id":9}]} 2024-03-06T07:29:07.675 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 78 2024-03-06T07:29:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a5806b500 msgr2=0x7f3a5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a5806b500 0x7f3a5806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0caa70 tx=0x7f3a60007cc0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 msgr2=0x7f3a6c150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 0x7f3a6c150000 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c034040 tx=0x7f3a5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 shutdown_connections 2024-03-06T07:29:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3a5806b500 0x7f3a5806d9c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3a6c15dfe0 0x7f3a6c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a6c0d29c0 0x7f3a6c09f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 --2- 172.21.15.122:0/4013983846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a6c0cf830 0x7f3a6c150000 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 >> 172.21.15.122:0/4013983846 conn(0x7f3a6c0d3cd0 msgr2=0x7f3a6c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 shutdown_connections 2024-03-06T07:29:07.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:07.675+0000 7f3a7258c640 1 -- 172.21.15.122:0/4013983846 wait complete. 2024-03-06T07:29:07.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:07.821 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"10","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}}] 2024-03-06T07:29:07.822 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "10" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:29:07.822 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:29:07.822 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:29:07.822 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> stdin-killer -- python3 -c ' 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> import fcntl 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> import sys 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> import time 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:29:07.837 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> while True: 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> try: 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> if os.read(0, 4096) == b"": 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> break 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> except BlockingIOError: 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> pass 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> time.sleep(2) 2024-03-06T07:29:07.838 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:29:07.881 DEBUG:teuthology.orchestra.run.smithi122:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:07.897 INFO:teuthology.orchestra.run.smithi122.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:29:07.958 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:08.898 DEBUG:teuthology.orchestra.run.smithi122:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 29h/41d Inode: 2199023255552 Links: 2 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-06 07:29:07.950628079 +0000 2024-03-06T07:29:08.914 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-06 07:29:07.950628079 +0000 2024-03-06T07:29:08.915 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-06 07:29:07.950628079 +0000 2024-03-06T07:29:08.915 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-03-06T07:29:08.915 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:29:08.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:08.915 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:29:08.973 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:09.132 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:09.132 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:09.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:09.249 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:29:09.249 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:29:09.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:09.369 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:09.369 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:09.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:09.527 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:29:09.528 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:09.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json cache drop 2024-03-06T07:29:09.667 INFO:teuthology.orchestra.run.smithi122.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":328,"bytes":65087}},"duration":0.0055222550000000002} 2024-03-06T07:29:09.668 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65087, "items": 328 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.005522255, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:29:09.668 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:29:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.775+0000 7f62a6313640 1 Processor -- start 2024-03-06T07:29:09.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.775+0000 7f62a6313640 1 -- start start 2024-03-06T07:29:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f62a6313640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f62a015bbb0 0x7f62a006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f62a6313640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 0x7f62a0071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f62a6313640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a00cea90 con 0x7f62a006b4f0 2024-03-06T07:29:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f62a6313640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a0161970 con 0x7f62a015bbb0 2024-03-06T07:29:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f62a6313640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01624f0 con 0x7f62a015ee10 2024-03-06T07:29:09.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f629ffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f62a015bbb0 0x7f62a006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.776+0000 7f62a48b0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 0x7f62a0071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f62a48b0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 0x7f62a0071ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:50750/0 (socket says 172.21.15.122:50750) 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f62a48b0640 1 -- 172.21.15.122:0/1402889043 learned_addr learned my addr 172.21.15.122:0/1402889043 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f62a48b0640 1 -- 172.21.15.122:0/1402889043 >> v2:172.21.15.169:3300/0 conn(0x7f62a015bbb0 msgr2=0x7f62a006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f62a48b0640 1 --2- 172.21.15.122:0/1402889043 >> v2:172.21.15.169:3300/0 conn(0x7f62a015bbb0 0x7f62a006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f62a48b0640 1 -- 172.21.15.122:0/1402889043 >> v1:172.21.15.169:6789/0 conn(0x7f62a006b4f0 legacy=0x7f62a015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f62a48b0640 1 -- 172.21.15.122:0/1402889043 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a0161bc0 con 0x7f62a015ee10 2024-03-06T07:29:09.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.777+0000 7f629ffff640 1 --2- 172.21.15.122:0/1402889043 >> v2:172.21.15.169:3300/0 conn(0x7f62a015bbb0 0x7f62a006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:09.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.778+0000 7f62a48b0640 1 --2- 172.21.15.122:0/1402889043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 0x7f62a0071ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6298009e50 tx=0x7f629802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea82593e7d5c8a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.778+0000 7f629effd640 1 -- 172.21.15.122:0/1402889043 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629802cb00 con 0x7f62a015ee10 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.778+0000 7f629effd640 1 -- 172.21.15.122:0/1402889043 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629802cca0 con 0x7f62a015ee10 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.778+0000 7f629effd640 1 -- 172.21.15.122:0/1402889043 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298034610 con 0x7f62a015ee10 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 -- 172.21.15.122:0/1402889043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 msgr2=0x7f62a0071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 --2- 172.21.15.122:0/1402889043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 0x7f62a0071ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6298009e50 tx=0x7f629802bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 -- 172.21.15.122:0/1402889043 shutdown_connections 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 --2- 172.21.15.122:0/1402889043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ee10 0x7f62a0071ca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 --2- 172.21.15.122:0/1402889043 >> v2:172.21.15.169:3300/0 conn(0x7f62a015bbb0 0x7f62a006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 -- 172.21.15.122:0/1402889043 >> 172.21.15.122:0/1402889043 conn(0x7f62a00b2660 msgr2=0x7f62a00b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 -- 172.21.15.122:0/1402889043 shutdown_connections 2024-03-06T07:29:09.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.779+0000 7f62a6313640 1 -- 172.21.15.122:0/1402889043 wait complete. 2024-03-06T07:29:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.780+0000 7f62a6313640 1 Processor -- start 2024-03-06T07:29:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.780+0000 7f62a6313640 1 -- start start 2024-03-06T07:29:09.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a6313640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 0x7f62a014d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a6313640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62a015ee10 0x7f62a015a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a6313640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ac40 0x7f62a0156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a6313640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00721e0 con 0x7f62a006b4f0 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a6313640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0162180 con 0x7f62a015ee10 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a6313640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0072360 con 0x7f62a015ac40 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f629ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 0x7f62a014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f629ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 0x7f62a014d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58974/0 (socket says 172.21.15.122:58974) 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f629ffff640 1 -- 172.21.15.122:0/3927073669 learned_addr learned my addr 172.21.15.122:0/3927073669 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f629f7fe640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62a015ee10 0x7f62a015a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.781+0000 7f62a48b0640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ac40 0x7f62a0156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629ffff640 1 -- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ac40 msgr2=0x7f62a0156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629ffff640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ac40 0x7f62a0156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629ffff640 1 -- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62a015ee10 msgr2=0x7f62a015a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629ffff640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62a015ee10 0x7f62a015a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629ffff640 1 -- 172.21.15.122:0/3927073669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a0159720 con 0x7f62a006b4f0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629ffff640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 0x7f62a014d730 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f629000ebe0 tx=0x7f629000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290017070 con 0x7f62a006b4f0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629000cc30 con 0x7f62a006b4f0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a0159a00 con 0x7f62a006b4f0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290013470 con 0x7f62a006b4f0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.782+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a0159fc0 con 0x7f62a006b4f0 2024-03-06T07:29:09.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.783+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a015b050 con 0x7f62a006b4f0 2024-03-06T07:29:09.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.783+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f629001a070 con 0x7f62a006b4f0 2024-03-06T07:29:09.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.784+0000 7f629cff9640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f628c06b500 0x7f628c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.784+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f629008a400 con 0x7f62a006b4f0 2024-03-06T07:29:09.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.784+0000 7f629f7fe640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f628c06b500 0x7f628c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.784+0000 7f629f7fe640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f628c06b500 0x7f628c06d9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f62a006f7c0 tx=0x7f6294009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:09.788+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6290058c30 con 0x7f62a006b4f0 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:10.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.007+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f62a00733b0 con 0x7f62a006b4f0 2024-03-06T07:29:10.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.472+0000 7f629cff9640 1 -- 172.21.15.122:0/3927073669 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v79) v1 ==== 105+0+0 (secure 0 0 0) 0x7f629005ccd0 con 0x7f62a006b4f0 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f628c06b500 msgr2=0x7f628c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f628c06b500 0x7f628c06d9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f62a006f7c0 tx=0x7f6294009040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 msgr2=0x7f62a014d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 0x7f62a014d730 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f629000ebe0 tx=0x7f629000c520 comp rx=0 tx=0).stop 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 shutdown_connections 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f628c06b500 0x7f628c06d9c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f62a015ac40 0x7f62a0156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62a015ee10 0x7f62a015a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 --2- 172.21.15.122:0/3927073669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a006b4f0 0x7f62a014d730 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 >> 172.21.15.122:0/3927073669 conn(0x7f62a00b2660 msgr2=0x7f62a006dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 shutdown_connections 2024-03-06T07:29:10.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.475+0000 7f62a6313640 1 -- 172.21.15.122:0/3927073669 wait complete. 2024-03-06T07:29:10.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:10.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.586+0000 7fe70ac8d640 1 Processor -- start 2024-03-06T07:29:10.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.586+0000 7fe70ac8d640 1 -- start start 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe70ac8d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe70ac8d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7040cf830 0x7fe7040d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe70ac8d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe7040d27f0 0x7fe7040d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe70ac8d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7040d2c50 con 0x7fe7040cf830 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe70ac8d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7040b31e0 con 0x7fe7040d27f0 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe70ac8d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040b3d60 con 0x7fe70415dfe0 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.587+0000 7fe6fbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7040cf830 0x7fe7040d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7040cf830 0x7fe7040d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58978/0 (socket says 172.21.15.122:58978) 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 -- 172.21.15.122:0/3385738549 learned_addr learned my addr 172.21.15.122:0/3385738549 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe708a29640 1 --2- 172.21.15.122:0/3385738549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe70922a640 1 --2- 172.21.15.122:0/3385738549 >> v2:172.21.15.169:3300/0 conn(0x7fe7040d27f0 0x7fe7040d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 -- 172.21.15.122:0/3385738549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 msgr2=0x7fe7040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 --2- 172.21.15.122:0/3385738549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7040cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 -- 172.21.15.122:0/3385738549 >> v2:172.21.15.169:3300/0 conn(0x7fe7040d27f0 msgr2=0x7fe7040d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 --2- 172.21.15.122:0/3385738549 >> v2:172.21.15.169:3300/0 conn(0x7fe7040d27f0 0x7fe7040d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe6fbfff640 1 -- 172.21.15.122:0/3385738549 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040b3430 con 0x7fe7040cf830 2024-03-06T07:29:10.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.588+0000 7fe70922a640 1 --2- 172.21.15.122:0/3385738549 >> v2:172.21.15.169:3300/0 conn(0x7fe7040d27f0 0x7fe7040d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe6fbfff640 1 --2- 172.21.15.122:0/3385738549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe7040d22b0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe704067ad0 tx=0x7fe6f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31508c02f8eeb7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe6fb7fe640 1 -- 172.21.15.122:0/3385738549 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4035020 con 0x7fe7040cf830 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe6fb7fe640 1 -- 172.21.15.122:0/3385738549 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f4031870 con 0x7fe7040cf830 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe6fb7fe640 1 -- 172.21.15.122:0/3385738549 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4031c40 con 0x7fe7040cf830 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 -- 172.21.15.122:0/3385738549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 msgr2=0x7fe7040d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/3385738549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe7040d22b0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe704067ad0 tx=0x7fe6f402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 -- 172.21.15.122:0/3385738549 shutdown_connections 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/3385738549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe7040d22b0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/3385738549 >> v2:172.21.15.169:3300/0 conn(0x7fe7040d27f0 0x7fe7040d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/3385738549 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7040cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 -- 172.21.15.122:0/3385738549 >> 172.21.15.122:0/3385738549 conn(0x7fe7040d3cd0 msgr2=0x7fe7040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 -- 172.21.15.122:0/3385738549 shutdown_connections 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.589+0000 7fe70ac8d640 1 -- 172.21.15.122:0/3385738549 wait complete. 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.590+0000 7fe70ac8d640 1 Processor -- start 2024-03-06T07:29:10.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.590+0000 7fe70ac8d640 1 -- start start 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe70ac8d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe704160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe70ac8d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d27f0 0x7fe7041f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe70ac8d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7041f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe70ac8d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040d1c70 con 0x7fe7040cf830 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe70ac8d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040b39f0 con 0x7fe7040d27f0 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe70ac8d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040d1df0 con 0x7fe70415dfe0 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe708a29640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe704160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe708a29640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe704160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:58982/0 (socket says 172.21.15.122:58982) 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.591+0000 7fe708a29640 1 -- 172.21.15.122:0/1399423531 learned_addr learned my addr 172.21.15.122:0/1399423531 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe70922a640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7041f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe6fbfff640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d27f0 0x7fe7041f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe708a29640 1 -- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 msgr2=0x7fe7041f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe708a29640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7041f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe708a29640 1 -- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d27f0 msgr2=0x7fe7041f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe708a29640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d27f0 0x7fe7041f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe708a29640 1 -- 172.21.15.122:0/1399423531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70415f490 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe70922a640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7041f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe708a29640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe704160f90 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe70000e870 tx=0x7fe70000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700017070 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70000c940 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70415f770 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70000cd30 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.592+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7041614d0 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.593+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70404f330 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.594+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe70001a070 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.594+0000 7fe6f97fa640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6f006b4d0 0x7fe6f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.594+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe70008a3f0 con 0x7fe7040cf830 2024-03-06T07:29:10.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.594+0000 7fe6fbfff640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6f006b4d0 0x7fe6f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.594+0000 7fe6fbfff640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6f006b4d0 0x7fe6f006d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4036000 tx=0x7fe6f403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.603+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe700059300 con 0x7fe7040cf830 2024-03-06T07:29:10.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.816+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7040747a0 con 0x7fe7040cf830 2024-03-06T07:29:10.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.817+0000 7fe6f97fa640 1 -- 172.21.15.122:0/1399423531 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fe700058c00 con 0x7fe7040cf830 2024-03-06T07:29:10.819 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:10.819 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:10.475682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:10.819 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 80 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6f006b4d0 msgr2=0x7fe6f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6f006b4d0 0x7fe6f006d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4036000 tx=0x7fe6f403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 msgr2=0x7fe704160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe704160f90 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe70000e870 tx=0x7fe70000ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 shutdown_connections 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe6f006b4d0 0x7fe6f006d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe70415dfe0 0x7fe7041f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7040d27f0 0x7fe7041f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 --2- 172.21.15.122:0/1399423531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7040cf830 0x7fe704160f90 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 >> 172.21.15.122:0/1399423531 conn(0x7fe7040d3cd0 msgr2=0x7fe70406ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 shutdown_connections 2024-03-06T07:29:10.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:10.819+0000 7fe70ac8d640 1 -- 172.21.15.122:0/1399423531 wait complete. 2024-03-06T07:29:11.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:10.475682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:11.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:11.963 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:12.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.136+0000 7f3d754ed640 1 Processor -- start 2024-03-06T07:29:12.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.136+0000 7f3d754ed640 1 -- start start 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d754ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 0x7f3d700b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d754ed640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3d700d0a80 0x7f3d700cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d754ed640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d700d25d0 con 0x7f3d700d4760 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d754ed640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d700d4bf0 con 0x7f3d700d0a80 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d754ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700ce9e0 con 0x7f3d700b4170 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d67fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3d700d4760 0x7f3d700d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57464/0 (socket says 172.21.15.122:57464) 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d67fff640 1 -- 172.21.15.122:0/1065843586 learned_addr learned my addr 172.21.15.122:0/1065843586 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:12.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690206063 0 0) 0x7f3d700d25d0 con 0x7f3d700d4760 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c003520 con 0x7f3d700d4760 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d6effd640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 0x7f3d700b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.137+0000 7f3d6f7fe640 1 --2- 172.21.15.122:0/1065843586 >> v2:172.21.15.169:3300/0 conn(0x7f3d700d0a80 0x7f3d700cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588909408 0 0) 0x7f3d5c003520 con 0x7f3d700d4760 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 msgr2=0x7f3d700b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 0x7f3d700b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 >> v2:172.21.15.169:3300/0 conn(0x7f3d700d0a80 msgr2=0x7f3d700cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 --2- 172.21.15.122:0/1065843586 >> v2:172.21.15.169:3300/0 conn(0x7f3d700d0a80 0x7f3d700cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d700d2870 con 0x7f3d700d4760 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f3d58001670 con 0x7f3d700d4760 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6f7fe640 1 --2- 172.21.15.122:0/1065843586 >> v2:172.21.15.169:3300/0 conn(0x7f3d700d0a80 0x7f3d700cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 >> v1:172.21.15.122:6789/0 conn(0x7f3d700d4760 legacy=0x7f3d700d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 0x7f3d5c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d5c004d40 0x7f3d5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700d4bf0 con 0x7f3d5c003f30 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700ce9e0 con 0x7f3d5c004d40 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c00ba10 con 0x7f3d700b4170 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6effd640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 0x7f3d5c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d6f7fe640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d5c004d40 0x7f3d5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d67fff640 1 -- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 msgr2=0x7f3d5c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700b4170 0x7f3d5c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d67fff640 1 -- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d5c004d40 msgr2=0x7f3d5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d5c004d40 0x7f3d5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.138+0000 7f3d67fff640 1 -- 172.21.15.122:0/1065843586 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c003520 con 0x7f3d5c003f30 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.139+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d5c004800 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3d58001050 tx=0x7f3d5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a4f9c1b58195c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.139+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58007d70 con 0x7f3d5c003f30 2024-03-06T07:29:12.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.139+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58030070 con 0x7f3d5c003f30 2024-03-06T07:29:12.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.139+0000 7f3d6e7fc640 1 -- 172.21.15.122:0/1065843586 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5802cd80 con 0x7f3d5c003f30 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d754ed640 1 -- 172.21.15.122:0/1065843586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 msgr2=0x7f3d5c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1065843586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d5c004800 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3d58001050 tx=0x7f3d5c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d6f7fe640 1 -- 172.21.15.122:0/1065843586 reap_dead start 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d754ed640 1 -- 172.21.15.122:0/1065843586 shutdown_connections 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d754ed640 1 -- 172.21.15.122:0/1065843586 >> 172.21.15.122:0/1065843586 conn(0x7f3d700d3c40 msgr2=0x7f3d700d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d754ed640 1 -- 172.21.15.122:0/1065843586 shutdown_connections 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.140+0000 7f3d754ed640 1 -- 172.21.15.122:0/1065843586 wait complete. 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.141+0000 7f3d754ed640 1 Processor -- start 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.141+0000 7f3d754ed640 1 -- start start 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.141+0000 7f3d754ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d701f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700ccc30 0x7f3d700cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d70158200 0x7f3d701586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700b4dd0 con 0x7f3d5c003f30 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700b4550 con 0x7f3d70158200 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700b4f50 con 0x7f3d700ccc30 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d701f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d701f6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59002/0 (socket says 172.21.15.122:59002) 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 -- 172.21.15.122:0/1402277597 learned_addr learned my addr 172.21.15.122:0/1402277597 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700ccc30 0x7f3d700cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 -- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700ccc30 msgr2=0x7f3d700cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700ccc30 0x7f3d700cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 -- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d70158200 msgr2=0x7f3d701586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d70158200 0x7f3d701586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 -- 172.21.15.122:0/1402277597 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d700c9bb0 con 0x7f3d5c003f30 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d6effd640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d701f6ae0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f3d600118e0 tx=0x7f3d60011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000f950 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6000faf0 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d700cd7f0 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60014440 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.142+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d700b7360 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.144+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7004f330 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.144+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d60012040 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.144+0000 7f3d677fe640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d5006b500 0x7f3d5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.144+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d5006b500 0x7f3d5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.144+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d6008a540 con 0x7f3d5c003f30 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.145+0000 7f3d67fff640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d5006b500 0x7f3d5006d9c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3d58034000 tx=0x7f3d58007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.152+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d60019030 con 0x7f3d5c003f30 2024-03-06T07:29:12.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.363+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d700733b0 con 0x7f3d5c003f30 2024-03-06T07:29:12.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.364+0000 7f3d677fe640 1 -- 172.21.15.122:0/1402277597 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f3d60059450 con 0x7f3d5c003f30 2024-03-06T07:29:12.367 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:12.367 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:10.475682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 80 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d5006b500 msgr2=0x7f3d5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d5006b500 0x7f3d5006d9c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3d58034000 tx=0x7f3d58007000 comp rx=0 tx=0).stop 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 msgr2=0x7f3d701f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d701f6ae0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f3d600118e0 tx=0x7f3d60011da0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 shutdown_connections 2024-03-06T07:29:12.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3d5006b500 0x7f3d5006d9c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3d70158200 0x7f3d701586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3d700ccc30 0x7f3d700cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 --2- 172.21.15.122:0/1402277597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d5c003f30 0x7f3d701f6ae0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 >> 172.21.15.122:0/1402277597 conn(0x7f3d700d3c40 msgr2=0x7f3d700ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 shutdown_connections 2024-03-06T07:29:12.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:12.366+0000 7f3d754ed640 1 -- 172.21.15.122:0/1402277597 wait complete. 2024-03-06T07:29:12.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:10.475682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:12.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:13.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.682+0000 7f38f717a640 1 Processor -- start 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.682+0000 7f38f717a640 1 -- start start 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.682+0000 7f38f717a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 0x7f38f015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.682+0000 7f38f717a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f015ea30 con 0x7f38f006b600 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.682+0000 7f38f717a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f015ed10 con 0x7f38f00ce920 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.682+0000 7f38f717a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0161b70 con 0x7f38f006b220 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38f5717640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f38f006b600 0x7f38f0162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57468/0 (socket says 172.21.15.122:57468) 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38f5717640 1 -- 172.21.15.122:0/806402887 learned_addr learned my addr 172.21.15.122:0/806402887 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118289597 0 0) 0x7f38f015ea30 con 0x7f38f006b600 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc003520 con 0x7f38f006b600 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38f4f16640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 0x7f38f015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2457785391 0 0) 0x7f38dc003520 con 0x7f38f006b600 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 msgr2=0x7f38f015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 0x7f38f015e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 >> v1:172.21.15.169:6789/0 conn(0x7f38f00ce920 legacy=0x7f38f0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f015eff0 con 0x7f38f006b600 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f38e80019f0 con 0x7f38f006b600 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 >> v1:172.21.15.122:6789/0 conn(0x7f38f006b600 legacy=0x7f38f0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc0039e0 0x7f38dc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc004800 0x7f38dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 0x7f38dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0161b70 con 0x7f38dc0039e0 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc00ba70 con 0x7f38dc004800 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc00bc70 con 0x7f38f006b220 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38f4f16640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc0039e0 0x7f38dc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38effff640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc004800 0x7f38dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.683+0000 7f38f5717640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 0x7f38dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38f4f16640 1 -- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 msgr2=0x7f38dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38f4f16640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38f006b220 0x7f38dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38f4f16640 1 -- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc004800 msgr2=0x7f38dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38f4f16640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc004800 0x7f38dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38f4f16640 1 -- 172.21.15.122:0/806402887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc003520 con 0x7f38dc0039e0 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38f4f16640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc0039e0 0x7f38dc003e00 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009a90 tx=0x7f38dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a1dca2d2b44cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0039070 con 0x7f38dc0039e0 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e002c680 con 0x7f38dc0039e0 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.684+0000 7f38ef7fe640 1 -- 172.21.15.122:0/806402887 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0009690 con 0x7f38dc0039e0 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f717a640 1 -- 172.21.15.122:0/806402887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc0039e0 msgr2=0x7f38dc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f717a640 1 --2- 172.21.15.122:0/806402887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc0039e0 0x7f38dc003e00 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009a90 tx=0x7f38dc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f5717640 1 -- 172.21.15.122:0/806402887 reap_dead start 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f717a640 1 -- 172.21.15.122:0/806402887 shutdown_connections 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f717a640 1 -- 172.21.15.122:0/806402887 >> 172.21.15.122:0/806402887 conn(0x7f38f006c0c0 msgr2=0x7f38f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f717a640 1 -- 172.21.15.122:0/806402887 shutdown_connections 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.685+0000 7f38f717a640 1 -- 172.21.15.122:0/806402887 wait complete. 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.686+0000 7f38f717a640 1 Processor -- start 2024-03-06T07:29:13.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.686+0000 7f38f717a640 1 -- start start 2024-03-06T07:29:13.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f717a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38dc0039e0 0x7f38f0154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f717a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f01538c0 0x7f38f0156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f717a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 0x7f38f0156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f717a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0072090 con 0x7f38f0153d20 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f717a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f00ce200 con 0x7f38f01538c0 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f717a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0155540 con 0x7f38dc0039e0 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 0x7f38f0156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 0x7f38f0156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59024/0 (socket says 172.21.15.122:59024) 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 -- 172.21.15.122:0/1539763039 learned_addr learned my addr 172.21.15.122:0/1539763039 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f4f16640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38dc0039e0 0x7f38f0154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38effff640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f01538c0 0x7f38f0156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 -- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38dc0039e0 msgr2=0x7f38f0154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38dc0039e0 0x7f38f0154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 -- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f01538c0 msgr2=0x7f38f0156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f01538c0 0x7f38f0156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f5717640 1 -- 172.21.15.122:0/1539763039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f0155820 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38f4f16640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38dc0039e0 0x7f38f0154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.687+0000 7f38effff640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f01538c0 0x7f38f0156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.688+0000 7f38f5717640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 0x7f38f0156bc0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f38e80012e0 tx=0x7f38e800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.688+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e800ec10 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.688+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e8018070 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.688+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f0155b00 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.688+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8013420 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.688+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f0157100 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.689+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f004f330 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.689+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38e80135c0 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.690+0000 7f38ed7fa640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.690+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38e808a5b0 con 0x7f38f0153d20 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.690+0000 7f38f4f16640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.690+0000 7f38f4f16640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38e0036000 tx=0x7f38e0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.694+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38e8059430 con 0x7f38f0153d20 2024-03-06T07:29:13.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.909+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38f00cdf00 con 0x7f38f0153d20 2024-03-06T07:29:13.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.910+0000 7f38ed7fa640 1 -- 172.21.15.122:0/1539763039 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f38e8058d30 con 0x7f38f0153d20 2024-03-06T07:29:13.912 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:13.912 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:10.475682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:13.913 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 80 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 msgr2=0x7f38d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38e0036000 tx=0x7f38e0030000 comp rx=0 tx=0).stop 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 msgr2=0x7f38f0156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 0x7f38f0156bc0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f38e80012e0 tx=0x7f38e800e530 comp rx=0 tx=0).stop 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 shutdown_connections 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f38d006b4d0 0x7f38d006d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38f0153d20 0x7f38f0156bc0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38f01538c0 0x7f38f0156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 --2- 172.21.15.122:0/1539763039 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38dc0039e0 0x7f38f0154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.912+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 >> 172.21.15.122:0/1539763039 conn(0x7f38f006c0c0 msgr2=0x7f38f00cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.913+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 shutdown_connections 2024-03-06T07:29:13.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:13.913+0000 7f38f717a640 1 -- 172.21.15.122:0/1539763039 wait complete. 2024-03-06T07:29:13.964 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:14.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:10.475682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:14.127 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:15.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:15.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.227+0000 7f458ccfb640 1 Processor -- start 2024-03-06T07:29:15.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.227+0000 7f458ccfb640 1 -- start start 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458ccfb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f458815dfe0 0x7f45880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458ccfb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f45880cf830 0x7f45880d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458ccfb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45880d27f0 0x7f45880d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458ccfb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45880b31e0 con 0x7f458815dfe0 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458ccfb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45880b3d60 con 0x7f45880cf830 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458ccfb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45880b3430 con 0x7f45880d27f0 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f458815dfe0 0x7f45880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f458815dfe0 0x7f45880cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59924/0 (socket says 172.21.15.122:59924) 2024-03-06T07:29:15.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.228+0000 7f458659c640 1 -- 172.21.15.122:0/351588655 learned_addr learned my addr 172.21.15.122:0/351588655 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f457dd9b640 1 --2- 172.21.15.122:0/351588655 >> v2:172.21.15.169:3300/0 conn(0x7f45880cf830 0x7f45880d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f4586d9d640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45880d27f0 0x7f45880d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f458659c640 1 -- 172.21.15.122:0/351588655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45880d27f0 msgr2=0x7f45880d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f458659c640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45880d27f0 0x7f45880d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f458659c640 1 -- 172.21.15.122:0/351588655 >> v2:172.21.15.169:3300/0 conn(0x7f45880cf830 msgr2=0x7f45880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f458659c640 1 --2- 172.21.15.122:0/351588655 >> v2:172.21.15.169:3300/0 conn(0x7f45880cf830 0x7f45880d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f458659c640 1 -- 172.21.15.122:0/351588655 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45880b3710 con 0x7f458815dfe0 2024-03-06T07:29:15.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f4586d9d640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45880d27f0 0x7f45880d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.229+0000 7f458659c640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458815dfe0 0x7f45880cf2f0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f45880d16b0 tx=0x7f457802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5573ca2ef41a6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f4585d9b640 1 -- 172.21.15.122:0/351588655 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578035020 con 0x7f458815dfe0 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f4585d9b640 1 -- 172.21.15.122:0/351588655 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578031870 con 0x7f458815dfe0 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f4585d9b640 1 -- 172.21.15.122:0/351588655 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578031c40 con 0x7f458815dfe0 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 -- 172.21.15.122:0/351588655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458815dfe0 msgr2=0x7f45880cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458815dfe0 0x7f45880cf2f0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f45880d16b0 tx=0x7f457802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 -- 172.21.15.122:0/351588655 shutdown_connections 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45880d27f0 0x7f45880d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 --2- 172.21.15.122:0/351588655 >> v2:172.21.15.169:3300/0 conn(0x7f45880cf830 0x7f45880d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 --2- 172.21.15.122:0/351588655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458815dfe0 0x7f45880cf2f0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 -- 172.21.15.122:0/351588655 >> 172.21.15.122:0/351588655 conn(0x7f45880d3cd0 msgr2=0x7f45880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 -- 172.21.15.122:0/351588655 shutdown_connections 2024-03-06T07:29:15.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.230+0000 7f458ccfb640 1 -- 172.21.15.122:0/351588655 wait complete. 2024-03-06T07:29:15.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 Processor -- start 2024-03-06T07:29:15.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 -- start start 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45880cf830 0x7f45880b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 0x7f45880b8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f458815dfe0 0x7f45880b8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45880d1df0 con 0x7f45880d27f0 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45880d1c70 con 0x7f45880cf830 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.232+0000 7f458ccfb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458815e3c0 con 0x7f458815dfe0 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 0x7f45880b8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 0x7f45880b8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59936/0 (socket says 172.21.15.122:59936) 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 -- 172.21.15.122:0/2546460465 learned_addr learned my addr 172.21.15.122:0/2546460465 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f458659c640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45880cf830 0x7f45880b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f4586d9d640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f458815dfe0 0x7f45880b8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 -- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f458815dfe0 msgr2=0x7f45880b8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f458815dfe0 0x7f45880b8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 -- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45880cf830 msgr2=0x7f45880b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45880cf830 0x7f45880b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 -- 172.21.15.122:0/2546460465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45880caed0 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457dd9b640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 0x7f45880b8150 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f457000c820 tx=0x7f457000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570015070 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4570004020 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45880cb170 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570004410 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.233+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45880cb730 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.234+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458804f330 con 0x7f45880d27f0 2024-03-06T07:29:15.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.235+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4570007830 con 0x7f45880d27f0 2024-03-06T07:29:15.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.235+0000 7f457f7fe640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456806b4d0 0x7f456806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.235+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4570089e90 con 0x7f45880d27f0 2024-03-06T07:29:15.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.235+0000 7f458659c640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456806b4d0 0x7f456806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.236+0000 7f458659c640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456806b4d0 0x7f456806d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4578036000 tx=0x7f457803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.240+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4570058750 con 0x7f45880d27f0 2024-03-06T07:29:15.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.477+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45880ce9b0 con 0x7f45880d27f0 2024-03-06T07:29:15.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.477+0000 7f457f7fe640 1 -- 172.21.15.122:0/2546460465 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f457005c7f0 con 0x7f45880d27f0 2024-03-06T07:29:15.479 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:15.479 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:10.475682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:15.479 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 80 2024-03-06T07:29:15.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456806b4d0 msgr2=0x7f456806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456806b4d0 0x7f456806d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4578036000 tx=0x7f457803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.481 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 msgr2=0x7f45880b8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 0x7f45880b8150 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f457000c820 tx=0x7f457000cce0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 shutdown_connections 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456806b4d0 0x7f456806d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f458815dfe0 0x7f45880b8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45880d27f0 0x7f45880b8150 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 --2- 172.21.15.122:0/2546460465 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45880cf830 0x7f45880b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 >> 172.21.15.122:0/2546460465 conn(0x7f45880d3cd0 msgr2=0x7f458806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 shutdown_connections 2024-03-06T07:29:15.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:15.479+0000 7f458ccfb640 1 -- 172.21.15.122:0/2546460465 wait complete. 2024-03-06T07:29:15.695 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:10.475682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:15.695 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:15.967 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:16.696 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:16.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.795+0000 7f9d0ada0640 1 Processor -- start 2024-03-06T07:29:16.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.796+0000 7f9d0ada0640 1 -- start start 2024-03-06T07:29:16.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.796+0000 7f9d0ada0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d0415c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.796+0000 7f9d0ada0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d0415c990 0x7f9d040b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.796+0000 7f9d0ada0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040d2780 con 0x7f9d04152bc0 2024-03-06T07:29:16.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.796+0000 7f9d0ada0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d04152fa0 con 0x7f9d0415c990 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0ada0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d16e0 con 0x7f9d040b7540 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d0415c990 0x7f9d040b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d0415c990 0x7f9d040b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59952/0 (socket says 172.21.15.122:59952) 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 -- 172.21.15.122:0/1868727269 learned_addr learned my addr 172.21.15.122:0/1868727269 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0959d640 1 --2- 172.21.15.122:0/1868727269 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d0415c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 -- 172.21.15.122:0/1868727269 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 msgr2=0x7f9d0415c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 --2- 172.21.15.122:0/1868727269 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d0415c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 -- 172.21.15.122:0/1868727269 >> v1:172.21.15.169:6789/0 conn(0x7f9d04152bc0 legacy=0x7f9d040b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.797+0000 7f9d0a59f640 1 -- 172.21.15.122:0/1868727269 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040d19c0 con 0x7f9d0415c990 2024-03-06T07:29:16.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0a59f640 1 --2- 172.21.15.122:0/1868727269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d040b1950 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009e50 tx=0x7f9cfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bca5aa894e30e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d08d9c640 1 -- 172.21.15.122:0/1868727269 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc02ca70 con 0x7f9d0415c990 2024-03-06T07:29:16.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d08d9c640 1 -- 172.21.15.122:0/1868727269 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc02cc10 con 0x7f9d0415c990 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d08d9c640 1 -- 172.21.15.122:0/1868727269 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0343f0 con 0x7f9d0415c990 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 -- 172.21.15.122:0/1868727269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 msgr2=0x7f9d040b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/1868727269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d040b1950 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009e50 tx=0x7f9cfc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 -- 172.21.15.122:0/1868727269 shutdown_connections 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/1868727269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d040b1950 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/1868727269 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d0415c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 -- 172.21.15.122:0/1868727269 >> 172.21.15.122:0/1868727269 conn(0x7f9d0409a9f0 msgr2=0x7f9d0409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 -- 172.21.15.122:0/1868727269 shutdown_connections 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.798+0000 7f9d0ada0640 1 -- 172.21.15.122:0/1868727269 wait complete. 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 Processor -- start 2024-03-06T07:29:16.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 -- start start 2024-03-06T07:29:16.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d04149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d04152bc0 0x7f9d040d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d041f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d2260 con 0x7f9d0415c990 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d1f80 con 0x7f9d04152bc0 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0ada0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0415ce20 con 0x7f9d040b7540 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0a59f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d041f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0a59f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d041f4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59968/0 (socket says 172.21.15.122:59968) 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.800+0000 7f9d0a59f640 1 -- 172.21.15.122:0/3599566757 learned_addr learned my addr 172.21.15.122:0/3599566757 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0959d640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d04152bc0 0x7f9d040d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d09d9e640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d04149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0a59f640 1 -- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 msgr2=0x7f9d04149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0a59f640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d04149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0a59f640 1 -- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d04152bc0 msgr2=0x7f9d040d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0a59f640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d04152bc0 0x7f9d040d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0a59f640 1 -- 172.21.15.122:0/3599566757 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d041f5bf0 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0a59f640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d041f4dc0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009e50 tx=0x7f9cfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc043020 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc03e450 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc03e860 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d04150e50 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.801+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d04151410 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.802+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d04068f80 con 0x7f9d0415c990 2024-03-06T07:29:16.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.803+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cfc046020 con 0x7f9d0415c990 2024-03-06T07:29:16.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.803+0000 7f9cfa7fc640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ce806b4d0 0x7f9ce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.803+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cfc0acc10 con 0x7f9d0415c990 2024-03-06T07:29:16.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.803+0000 7f9d09d9e640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ce806b4d0 0x7f9ce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.804+0000 7f9d09d9e640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ce806b4d0 0x7f9ce806d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d00004ea0 tx=0x7f9d00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:16.807+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cfc04d030 con 0x7f9d0415c990 2024-03-06T07:29:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.021+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d040b1730 con 0x7f9d0415c990 2024-03-06T07:29:17.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.022+0000 7f9cfa7fc640 1 -- 172.21.15.122:0/3599566757 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f9d041f5bf0 con 0x7f9d0415c990 2024-03-06T07:29:17.024 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:17.024 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:10.475682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:17.024 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 80 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ce806b4d0 msgr2=0x7f9ce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ce806b4d0 0x7f9ce806d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d00004ea0 tx=0x7f9d00009040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 msgr2=0x7f9d041f4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d041f4dc0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009e50 tx=0x7f9cfc004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 shutdown_connections 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9ce806b4d0 0x7f9ce806d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0415c990 0x7f9d041f4dc0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d04152bc0 0x7f9d040d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 --2- 172.21.15.122:0/3599566757 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9d040b7540 0x7f9d04149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 >> 172.21.15.122:0/3599566757 conn(0x7f9d0409a9f0 msgr2=0x7f9d0414e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 shutdown_connections 2024-03-06T07:29:17.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:17.024+0000 7f9d0ada0640 1 -- 172.21.15.122:0/3599566757 wait complete. 2024-03-06T07:29:17.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:10.475682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:17.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:17.968 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:18.242 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:18.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 Processor -- start 2024-03-06T07:29:18.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 -- start start 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6b700d00c0 0x7f6b700d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 0x7f6b7006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b700d09e0 con 0x7f6b700cf850 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7006ca90 con 0x7f6b700d00c0 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.341+0000 7f6b7538a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7015e380 con 0x7f6b7006cf20 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 0x7f6b7006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6effd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6b700d00c0 0x7f6b700d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6f7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 0x7f6b7006d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:57404/0 (socket says 172.21.15.122:57404) 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6effd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6b700d00c0 0x7f6b700d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:50530/0 (socket says 172.21.15.122:50530) 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6f7fe640 1 -- 172.21.15.122:0/227657209 learned_addr learned my addr 172.21.15.122:0/227657209 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6e7fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f6b700cf850 0x7f6b700cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:50986/0 (socket says 172.21.15.122:50986) 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6dffb640 1 -- 172.21.15.122:0/227657209 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899567709 0 0) 0x7f6b700d09e0 con 0x7f6b700cf850 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6dffb640 1 -- 172.21.15.122:0/227657209 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b58003520 con 0x7f6b700cf850 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6effd640 1 -- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 msgr2=0x7f6b7006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6effd640 1 --2- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 0x7f6b7006d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6effd640 1 -- 172.21.15.122:0/227657209 >> v1:172.21.15.169:6789/0 conn(0x7f6b700cf850 legacy=0x7f6b700cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6effd640 1 -- 172.21.15.122:0/227657209 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b700cdc10 con 0x7f6b700d00c0 2024-03-06T07:29:18.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.342+0000 7f6b6f7fe640 1 --2- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 0x7f6b7006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.343+0000 7f6b6effd640 1 --2- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d00c0 0x7f6b700d04a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6b600097a0 tx=0x7f6b6002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1752ffa436fd121 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.343+0000 7f6b6dffb640 1 -- 172.21.15.122:0/227657209 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60035020 con 0x7f6b700d00c0 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.343+0000 7f6b6dffb640 1 -- 172.21.15.122:0/227657209 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60031830 con 0x7f6b700d00c0 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.343+0000 7f6b6dffb640 1 -- 172.21.15.122:0/227657209 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60031be0 con 0x7f6b700d00c0 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 -- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d00c0 msgr2=0x7f6b700d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 --2- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d00c0 0x7f6b700d04a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6b600097a0 tx=0x7f6b6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 -- 172.21.15.122:0/227657209 shutdown_connections 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 --2- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b7006cf20 0x7f6b7006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 --2- 172.21.15.122:0/227657209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b700d00c0 0x7f6b700d04a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 -- 172.21.15.122:0/227657209 >> 172.21.15.122:0/227657209 conn(0x7f6b700b35a0 msgr2=0x7f6b700b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 -- 172.21.15.122:0/227657209 shutdown_connections 2024-03-06T07:29:18.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.344+0000 7f6b7538a640 1 -- 172.21.15.122:0/227657209 wait complete. 2024-03-06T07:29:18.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.345+0000 7f6b7538a640 1 Processor -- start 2024-03-06T07:29:18.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.345+0000 7f6b7538a640 1 -- start start 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b7538a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b7006cf20 0x7f6b701495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b7538a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 0x7f6b7014bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b7538a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700d00c0 0x7f6b7014c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b7538a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7006c720 con 0x7f6b700cf850 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b7538a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7006c440 con 0x7f6b7006cf20 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b7538a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b700b1030 con 0x7f6b700d00c0 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 0x7f6b7014bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 0x7f6b7014bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59972/0 (socket says 172.21.15.122:59972) 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 -- 172.21.15.122:0/3988206783 learned_addr learned my addr 172.21.15.122:0/3988206783 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6f7fe640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700d00c0 0x7f6b7014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6effd640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b7006cf20 0x7f6b701495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 -- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700d00c0 msgr2=0x7f6b7014c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700d00c0 0x7f6b7014c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 -- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b7006cf20 msgr2=0x7f6b701495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b7006cf20 0x7f6b701495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 -- 172.21.15.122:0/3988206783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b701501e0 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b6e7fc640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 0x7f6b7014bed0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400c430 tx=0x7f6b6400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.346+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b640074e0 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.347+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b640047e0 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.347+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b700b68b0 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.347+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64004bd0 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.347+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b700b6e70 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.348+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7004f330 con 0x7f6b700cf850 2024-03-06T07:29:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.348+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b6401a020 con 0x7f6b700cf850 2024-03-06T07:29:18.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.349+0000 7f6b577fe640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5c06b4d0 0x7f6b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.349+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b640898d0 con 0x7f6b700cf850 2024-03-06T07:29:18.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.349+0000 7f6b6effd640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5c06b4d0 0x7f6b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.349+0000 7f6b6effd640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5c06b4d0 0x7f6b5c06d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6b60036000 tx=0x7f6b60008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.353+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b64058250 con 0x7f6b700cf850 2024-03-06T07:29:18.570 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.568+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b7006f790 con 0x7f6b700cf850 2024-03-06T07:29:18.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.569+0000 7f6b577fe640 1 -- 172.21.15.122:0/3988206783 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f6b6405c2f0 con 0x7f6b700cf850 2024-03-06T07:29:18.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:18.571 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:18.571 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5c06b4d0 msgr2=0x7f6b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5c06b4d0 0x7f6b5c06d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6b60036000 tx=0x7f6b60008030 comp rx=0 tx=0).stop 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 msgr2=0x7f6b7014bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 0x7f6b7014bed0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400c430 tx=0x7f6b6400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 shutdown_connections 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6b5c06b4d0 0x7f6b5c06d990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6b700d00c0 0x7f6b7014c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b700cf850 0x7f6b7014bed0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 --2- 172.21.15.122:0/3988206783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b7006cf20 0x7f6b701495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 >> 172.21.15.122:0/3988206783 conn(0x7f6b700b35a0 msgr2=0x7f6b700b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 shutdown_connections 2024-03-06T07:29:18.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:18.571+0000 7f6b7538a640 1 -- 172.21.15.122:0/3988206783 wait complete. 2024-03-06T07:29:18.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:18.785 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:19.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.886+0000 7f10bd981640 1 Processor -- start 2024-03-06T07:29:19.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.886+0000 7f10bd981640 1 -- start start 2024-03-06T07:29:19.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.886+0000 7f10bd981640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f10b815dfe0 0x7f10b80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.886+0000 7f10bd981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 0x7f10b80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10bd981640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80b3430 con 0x7f10b80d27f0 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10bd981640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b80d2c50 con 0x7f10b815dfe0 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10bd981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80b31e0 con 0x7f10b80cf830 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10affff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 0x7f10b80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f10b815dfe0 0x7f10b80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f10b815dfe0 0x7f10b80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:50542/0 (socket says 172.21.15.122:50542) 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f10b80d27f0 0x7f10b80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:50994/0 (socket says 172.21.15.122:50994) 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10affff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 0x7f10b80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:57428/0 (socket says 172.21.15.122:57428) 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 -- 172.21.15.122:0/107000367 learned_addr learned my addr 172.21.15.122:0/107000367 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b67fc640 1 -- 172.21.15.122:0/107000367 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780299135 0 0) 0x7f10b80b3430 con 0x7f10b80d27f0 2024-03-06T07:29:19.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b67fc640 1 -- 172.21.15.122:0/107000367 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a4003520 con 0x7f10b80d27f0 2024-03-06T07:29:19.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 -- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 msgr2=0x7f10b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 --2- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 0x7f10b80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 -- 172.21.15.122:0/107000367 >> v1:172.21.15.169:6789/0 conn(0x7f10b80d27f0 legacy=0x7f10b80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10b6ffd640 1 -- 172.21.15.122:0/107000367 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80b3710 con 0x7f10b815dfe0 2024-03-06T07:29:19.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.887+0000 7f10affff640 1 --2- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 0x7f10b80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.888+0000 7f10b6ffd640 1 --2- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80cf2f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009870 tx=0x7f10a802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b3869cafc7235b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10b67fc640 1 -- 172.21.15.122:0/107000367 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8034070 con 0x7f10b815dfe0 2024-03-06T07:29:19.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10b67fc640 1 -- 172.21.15.122:0/107000367 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8030840 con 0x7f10b815dfe0 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10b67fc640 1 -- 172.21.15.122:0/107000367 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8030c10 con 0x7f10b815dfe0 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 -- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 msgr2=0x7f10b80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 --2- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80cf2f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009870 tx=0x7f10a802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 -- 172.21.15.122:0/107000367 shutdown_connections 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 --2- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80cf830 0x7f10b80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 --2- 172.21.15.122:0/107000367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80cf2f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 -- 172.21.15.122:0/107000367 >> 172.21.15.122:0/107000367 conn(0x7f10b80d3cd0 msgr2=0x7f10b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 -- 172.21.15.122:0/107000367 shutdown_connections 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.889+0000 7f10bd981640 1 -- 172.21.15.122:0/107000367 wait complete. 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.890+0000 7f10bd981640 1 Processor -- start 2024-03-06T07:29:19.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.890+0000 7f10bd981640 1 -- start start 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10bd981640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 0x7f10b80b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10bd981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80d27f0 0x7f10b80b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10bd981640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10bd981640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80d1c70 con 0x7f10b80cf830 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10bd981640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80b3d60 con 0x7f10b815dfe0 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10bd981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80d1df0 con 0x7f10b80d27f0 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 0x7f10b80b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 0x7f10b80b9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59980/0 (socket says 172.21.15.122:59980) 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 -- 172.21.15.122:0/581562532 learned_addr learned my addr 172.21.15.122:0/581562532 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b77fe640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10affff640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80d27f0 0x7f10b80b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 -- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80d27f0 msgr2=0x7f10b80b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80d27f0 0x7f10b80b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 -- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 msgr2=0x7f10b80b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.891+0000 7f10b6ffd640 1 -- 172.21.15.122:0/581562532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80b7040 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.892+0000 7f10b6ffd640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 0x7f10b80b9220 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f10a802b9c0 tx=0x7f10a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.892+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8034040 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.892+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8039c90 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.892+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b815a8a0 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.892+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8043630 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.892+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b80b6d60 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.893+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b804f330 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.893+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10a804a060 con 0x7f10b80cf830 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.894+0000 7f10b4ff9640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f109806b4d0 0x7f109806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.894+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a80ac100 con 0x7f10b80cf830 2024-03-06T07:29:19.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.894+0000 7f10affff640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f109806b4d0 0x7f109806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.894+0000 7f10affff640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f109806b4d0 0x7f109806d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009560 tx=0x7f10a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:19.898+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10b815a8a0 con 0x7f10b80cf830 2024-03-06T07:29:19.970 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:20.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.114+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10b80b09c0 con 0x7f10b80cf830 2024-03-06T07:29:20.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.114+0000 7f10b4ff9640 1 -- 172.21.15.122:0/581562532 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f10b80b7040 con 0x7f10b80cf830 2024-03-06T07:29:20.116 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:20.117 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:20.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f109806b4d0 msgr2=0x7f109806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f109806b4d0 0x7f109806d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009560 tx=0x7f10a0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:20.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 msgr2=0x7f10b80b9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 0x7f10b80b9220 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f10a802b9c0 tx=0x7f10a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 shutdown_connections 2024-03-06T07:29:20.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f109806b4d0 0x7f109806d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10b815dfe0 0x7f10b80b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f10b80d27f0 0x7f10b80b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 --2- 172.21.15.122:0/581562532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10b80cf830 0x7f10b80b9220 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 >> 172.21.15.122:0/581562532 conn(0x7f10b80d3cd0 msgr2=0x7f10b80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 shutdown_connections 2024-03-06T07:29:20.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:20.116+0000 7f10bd981640 1 -- 172.21.15.122:0/581562532 wait complete. 2024-03-06T07:29:20.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:20.330 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:21.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:21.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.442+0000 7f4e0ef81640 1 Processor -- start 2024-03-06T07:29:21.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.442+0000 7f4e0ef81640 1 -- start start 2024-03-06T07:29:21.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.442+0000 7f4e0ef81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 0x7f4e080d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e0ef81640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080b3430 con 0x7f4e080b41f0 2024-03-06T07:29:21.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e0ef81640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080b3710 con 0x7f4e0815dfe0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e0ef81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080d2c50 con 0x7f4e080d22b0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e0d51e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4e080b41f0 0x7f4e080b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34098/0 (socket says 172.21.15.122:34098) 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e0d51e640 1 -- 172.21.15.122:0/2616589049 learned_addr learned my addr 172.21.15.122:0/2616589049 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e07fff640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 0x7f4e080d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930538932 0 0) 0x7f4e080b3430 con 0x7f4e080b41f0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df8003520 con 0x7f4e080b41f0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360980106 0 0) 0x7f4df8003520 con 0x7f4e080b41f0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 msgr2=0x7f4e080d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 0x7f4e080d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 >> v1:172.21.15.169:6789/0 conn(0x7f4e0815dfe0 legacy=0x7f4e080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e080b39f0 con 0x7f4e080b41f0 2024-03-06T07:29:21.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4e000019f0 con 0x7f4e080b41f0 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 >> v1:172.21.15.122:6789/0 conn(0x7f4e080b41f0 legacy=0x7f4e080b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 0x7f4df80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df8003f60 0x7f4df8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8004d40 0x7f4df80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df800ba70 con 0x7f4df8003f60 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df800bc20 con 0x7f4df8004d40 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.443+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df800be20 con 0x7f4e080d22b0 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e0cd1d640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 0x7f4df80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df8003f60 0x7f4df8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e0d51e640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8004d40 0x7f4df80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 -- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 msgr2=0x7f4df80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080d22b0 0x7f4df80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 -- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8004d40 msgr2=0x7f4df80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8004d40 0x7f4df80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 -- 172.21.15.122:0/2616589049 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8003520 con 0x7f4df8003f60 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.444+0000 7f4e07fff640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df8003f60 0x7f4df8004800 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009ac0 tx=0x7f4df8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73e5f6b82def470 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4039070 con 0x7f4df8003f60 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df402c6b0 con 0x7f4df8003f60 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e077fe640 1 -- 172.21.15.122:0/2616589049 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4009690 con 0x7f4df8003f60 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0ef81640 1 -- 172.21.15.122:0/2616589049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df8003f60 msgr2=0x7f4df8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/2616589049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df8003f60 0x7f4df8004800 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009ac0 tx=0x7f4df8005830 comp rx=0 tx=0).stop 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0d51e640 1 -- 172.21.15.122:0/2616589049 reap_dead start 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0ef81640 1 -- 172.21.15.122:0/2616589049 shutdown_connections 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0ef81640 1 -- 172.21.15.122:0/2616589049 >> 172.21.15.122:0/2616589049 conn(0x7f4e080d3cd0 msgr2=0x7f4e080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0ef81640 1 -- 172.21.15.122:0/2616589049 shutdown_connections 2024-03-06T07:29:21.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.445+0000 7f4e0ef81640 1 -- 172.21.15.122:0/2616589049 wait complete. 2024-03-06T07:29:21.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.446+0000 7f4e0ef81640 1 Processor -- start 2024-03-06T07:29:21.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.446+0000 7f4e0ef81640 1 -- start start 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0ef81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8003f60 0x7f4e080ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0ef81640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 0x7f4e080cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0ef81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080b6ff0 0x7f4e0815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0ef81640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080b3d60 con 0x7f4e080b6b90 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0ef81640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080b31e0 con 0x7f4df8003f60 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0ef81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080d1c70 con 0x7f4e080b6ff0 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e07fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 0x7f4e080cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e07fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 0x7f4e080cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59998/0 (socket says 172.21.15.122:59998) 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e07fff640 1 -- 172.21.15.122:0/3958169005 learned_addr learned my addr 172.21.15.122:0/3958169005 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0cd1d640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8003f60 0x7f4e080ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.447+0000 7f4e0d51e640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080b6ff0 0x7f4e0815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e07fff640 1 -- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080b6ff0 msgr2=0x7f4e0815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e07fff640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080b6ff0 0x7f4e0815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e07fff640 1 -- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8003f60 msgr2=0x7f4e080ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e07fff640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8003f60 0x7f4e080ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e07fff640 1 -- 172.21.15.122:0/3958169005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e080b5af0 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e07fff640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 0x7f4e080cd940 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f4df4037040 tx=0x7f4df4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4039050 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4045070 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e080b5dd0 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40044e0 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.448+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e080b6670 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.449+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0804f330 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.450+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4df4003850 con 0x7f4e080b6b90 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.450+0000 7f4e057fa640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4dec06b4d0 0x7f4dec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.450+0000 7f4e0cd1d640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4dec06b4d0 0x7f4dec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.450+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4df40abd10 con 0x7f4e080b6b90 2024-03-06T07:29:21.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.451+0000 7f4e0cd1d640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4dec06b4d0 0x7f4dec06d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0017e0 tx=0x7f4dfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.454+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4df404e020 con 0x7f4e080b6b90 2024-03-06T07:29:21.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.671+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e080b6160 con 0x7f4e080b6b90 2024-03-06T07:29:21.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.672+0000 7f4e057fa640 1 -- 172.21.15.122:0/3958169005 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f4df407a660 con 0x7f4e080b6b90 2024-03-06T07:29:21.674 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:21.674 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:21.674 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4dec06b4d0 msgr2=0x7f4dec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4dec06b4d0 0x7f4dec06d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0017e0 tx=0x7f4dfc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 msgr2=0x7f4e080cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 0x7f4e080cd940 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f4df4037040 tx=0x7f4df4004730 comp rx=0 tx=0).stop 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 shutdown_connections 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4dec06b4d0 0x7f4dec06d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e080b6ff0 0x7f4e0815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e080b6b90 0x7f4e080cd940 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 --2- 172.21.15.122:0/3958169005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4df8003f60 0x7f4e080ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 >> 172.21.15.122:0/3958169005 conn(0x7f4e080d3cd0 msgr2=0x7f4e081603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 shutdown_connections 2024-03-06T07:29:21.678 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:21.674+0000 7f4e0ef81640 1 -- 172.21.15.122:0/3958169005 wait complete. 2024-03-06T07:29:21.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:21.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:21.972 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:22.889 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.989+0000 7f8456c81640 1 Processor -- start 2024-03-06T07:29:22.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.989+0000 7f8456c81640 1 -- start start 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8456c81640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f845015bdd0 0x7f84500c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8456c81640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f84500d16e0 0x7f84500d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8456c81640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84500cebe0 con 0x7f845015fa10 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8456c81640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84500ce980 con 0x7f845015bdd0 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8456c81640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84500ceec0 con 0x7f84500d16e0 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8454a1d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f845015bdd0 0x7f84500c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8454a1d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f845015bdd0 0x7f84500c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60002/0 (socket says 172.21.15.122:60002) 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.990+0000 7f8454a1d640 1 -- 172.21.15.122:0/2879869045 learned_addr learned my addr 172.21.15.122:0/2879869045 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.991+0000 7f845521e640 1 --2- 172.21.15.122:0/2879869045 >> v2:172.21.15.169:3300/0 conn(0x7f84500d16e0 0x7f84500d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.991+0000 7f8454a1d640 1 -- 172.21.15.122:0/2879869045 >> v2:172.21.15.169:3300/0 conn(0x7f84500d16e0 msgr2=0x7f84500d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.991+0000 7f8454a1d640 1 --2- 172.21.15.122:0/2879869045 >> v2:172.21.15.169:3300/0 conn(0x7f84500d16e0 0x7f84500d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.991+0000 7f8454a1d640 1 -- 172.21.15.122:0/2879869045 >> v1:172.21.15.169:6789/0 conn(0x7f845015fa10 legacy=0x7f84500ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.991+0000 7f8454a1d640 1 -- 172.21.15.122:0/2879869045 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500cf1a0 con 0x7f845015bdd0 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.991+0000 7f8454a1d640 1 --2- 172.21.15.122:0/2879869045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015bdd0 0x7f84500c5610 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f84500c5520 tx=0x7f844402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87579dad7505b306 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f844f7fe640 1 -- 172.21.15.122:0/2879869045 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444035020 con 0x7f845015bdd0 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f844f7fe640 1 -- 172.21.15.122:0/2879869045 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444031870 con 0x7f845015bdd0 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f844f7fe640 1 -- 172.21.15.122:0/2879869045 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444031c40 con 0x7f845015bdd0 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 -- 172.21.15.122:0/2879869045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015bdd0 msgr2=0x7f84500c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 --2- 172.21.15.122:0/2879869045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015bdd0 0x7f84500c5610 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f84500c5520 tx=0x7f844402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 -- 172.21.15.122:0/2879869045 shutdown_connections 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 --2- 172.21.15.122:0/2879869045 >> v2:172.21.15.169:3300/0 conn(0x7f84500d16e0 0x7f84500d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 --2- 172.21.15.122:0/2879869045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015bdd0 0x7f84500c5610 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 -- 172.21.15.122:0/2879869045 >> 172.21.15.122:0/2879869045 conn(0x7f845009a9f0 msgr2=0x7f845009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 -- 172.21.15.122:0/2879869045 shutdown_connections 2024-03-06T07:29:22.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.992+0000 7f8456c81640 1 -- 172.21.15.122:0/2879869045 wait complete. 2024-03-06T07:29:22.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.993+0000 7f8456c81640 1 Processor -- start 2024-03-06T07:29:22.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 -- start start 2024-03-06T07:29:22.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d16e0 0x7f84500c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f845015bdd0 0x7f8450155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 0x7f84500b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84500cc5d0 con 0x7f845015fa10 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84500c5b50 con 0x7f84500d16e0 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.994+0000 7f8456c81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845015c1b0 con 0x7f845015bdd0 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 0x7f84500b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 0x7f84500b6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60016/0 (socket says 172.21.15.122:60016) 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 -- 172.21.15.122:0/2872238740 learned_addr learned my addr 172.21.15.122:0/2872238740 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f8454a1d640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d16e0 0x7f84500c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f844ffff640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f845015bdd0 0x7f8450155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 -- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f845015bdd0 msgr2=0x7f8450155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f845015bdd0 0x7f8450155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 -- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d16e0 msgr2=0x7f84500c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d16e0 0x7f84500c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 -- 172.21.15.122:0/2872238740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500b5680 con 0x7f845015fa10 2024-03-06T07:29:22.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f845521e640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 0x7f84500b6a00 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f8448007e90 tx=0x7f844800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448014420 con 0x7f845015fa10 2024-03-06T07:29:23.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8448019070 con 0x7f845015fa10 2024-03-06T07:29:23.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84500b5960 con 0x7f845015fa10 2024-03-06T07:29:23.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448014800 con 0x7f845015fa10 2024-03-06T07:29:23.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.995+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84500b5f20 con 0x7f845015fa10 2024-03-06T07:29:23.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.996+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8450160760 con 0x7f845015fa10 2024-03-06T07:29:23.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.997+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84480149a0 con 0x7f845015fa10 2024-03-06T07:29:23.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.997+0000 7f844d7fa640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.997+0000 7f8454a1d640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.997+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f844808aa30 con 0x7f845015fa10 2024-03-06T07:29:23.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:22.998+0000 7f8454a1d640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8444036000 tx=0x7f844403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.002+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8450160760 con 0x7f845015fa10 2024-03-06T07:29:23.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.286+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84500b7a40 con 0x7f845015fa10 2024-03-06T07:29:23.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.286+0000 7f844d7fa640 1 -- 172.21.15.122:0/2872238740 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f84480598b0 con 0x7f845015fa10 2024-03-06T07:29:23.288 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:23.289 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:23.289 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 msgr2=0x7f843406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8444036000 tx=0x7f844403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 msgr2=0x7f84500b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 0x7f84500b6a00 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f8448007e90 tx=0x7f844800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 shutdown_connections 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f845015fa10 0x7f84500b6a00 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f845015bdd0 0x7f8450155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 --2- 172.21.15.122:0/2872238740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84500d16e0 0x7f84500c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 >> 172.21.15.122:0/2872238740 conn(0x7f845009a9f0 msgr2=0x7f84500ccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 shutdown_connections 2024-03-06T07:29:23.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:23.288+0000 7f8456c81640 1 -- 172.21.15.122:0/2872238740 wait complete. 2024-03-06T07:29:23.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:23.502 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:23.974 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:24.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:24.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.607+0000 7f7859577640 1 Processor -- start 2024-03-06T07:29:24.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.607+0000 7f7859577640 1 -- start start 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f7859577640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f78540d22b0 0x7f78540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f7859577640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78540b41f0 0x7f78540b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f7859577640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78540b3430 con 0x7f785415dfe0 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f7859577640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78540d2c50 con 0x7f78540b41f0 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f7859577640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78540b31e0 con 0x7f78540d22b0 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78540b41f0 0x7f78540b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f78540b41f0 0x7f78540b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34032/0 (socket says 172.21.15.122:34032) 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 -- 172.21.15.122:0/34177505 learned_addr learned my addr 172.21.15.122:0/34177505 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f784bfff640 1 --2- 172.21.15.122:0/34177505 >> v2:172.21.15.169:3300/0 conn(0x7f78540d22b0 0x7f78540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 -- 172.21.15.122:0/34177505 >> v2:172.21.15.169:3300/0 conn(0x7f78540d22b0 msgr2=0x7f78540d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 --2- 172.21.15.122:0/34177505 >> v2:172.21.15.169:3300/0 conn(0x7f78540d22b0 0x7f78540d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 -- 172.21.15.122:0/34177505 >> v1:172.21.15.169:6789/0 conn(0x7f785415dfe0 legacy=0x7f78540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f78537fe640 1 -- 172.21.15.122:0/34177505 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540b3710 con 0x7f78540b41f0 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.608+0000 7f784bfff640 1 --2- 172.21.15.122:0/34177505 >> v2:172.21.15.169:3300/0 conn(0x7f78540d22b0 0x7f78540d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.609+0000 7f78537fe640 1 --2- 172.21.15.122:0/34177505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f78540b4680 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f784c009e50 tx=0x7f784c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cced35309ecc9436 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.609+0000 7f78527fc640 1 -- 172.21.15.122:0/34177505 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c02ca80 con 0x7f78540b41f0 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.609+0000 7f78527fc640 1 -- 172.21.15.122:0/34177505 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c02cc20 con 0x7f78540b41f0 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.609+0000 7f78527fc640 1 -- 172.21.15.122:0/34177505 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c0343f0 con 0x7f78540b41f0 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.609+0000 7f7859577640 1 -- 172.21.15.122:0/34177505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 msgr2=0x7f78540b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.609+0000 7f7859577640 1 --2- 172.21.15.122:0/34177505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f78540b4680 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f784c009e50 tx=0x7f784c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.610+0000 7f7859577640 1 -- 172.21.15.122:0/34177505 shutdown_connections 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.610+0000 7f7859577640 1 --2- 172.21.15.122:0/34177505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f78540b4680 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.610+0000 7f7859577640 1 --2- 172.21.15.122:0/34177505 >> v2:172.21.15.169:3300/0 conn(0x7f78540d22b0 0x7f78540d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.610+0000 7f7859577640 1 -- 172.21.15.122:0/34177505 >> 172.21.15.122:0/34177505 conn(0x7f78540d3cd0 msgr2=0x7f78540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.610+0000 7f7859577640 1 -- 172.21.15.122:0/34177505 shutdown_connections 2024-03-06T07:29:24.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.610+0000 7f7859577640 1 -- 172.21.15.122:0/34177505 wait complete. 2024-03-06T07:29:24.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 Processor -- start 2024-03-06T07:29:24.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 -- start start 2024-03-06T07:29:24.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f7854161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78540d22b0 0x7f78540cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f785415dfe0 0x7f78540b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540cfa00 con 0x7f78540b41f0 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540b3d60 con 0x7f785415dfe0 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7859577640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78540d1c70 con 0x7f78540d22b0 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.611+0000 7f7852ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f7854161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f7854161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34044/0 (socket says 172.21.15.122:34044) 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 -- 172.21.15.122:0/356775887 learned_addr learned my addr 172.21.15.122:0/356775887 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 -- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78540d22b0 msgr2=0x7f78540cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f784bfff640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78540d22b0 0x7f78540cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f78537fe640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f785415dfe0 0x7f78540b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78540d22b0 0x7f78540cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 -- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f785415dfe0 msgr2=0x7f78540b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f785415dfe0 0x7f78540b9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 -- 172.21.15.122:0/356775887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540cc250 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7852ffd640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f7854161150 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f784400b3f0 tx=0x7f784400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844015020 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7844011830 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78540cc530 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844011c20 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.612+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78540ccaf0 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.613+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785404f330 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.613+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7844018020 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.614+0000 7f784b7fe640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f783406b4d0 0x7f783406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.614+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7844089e00 con 0x7f78540b41f0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.614+0000 7f784bfff640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f783406b4d0 0x7f783406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.614+0000 7f784bfff640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f783406b4d0 0x7f783406d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f785406c980 tx=0x7f783c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.618+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f784401f030 con 0x7f78540b41f0 2024-03-06T07:29:24.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.827+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f785415d500 con 0x7f78540b41f0 2024-03-06T07:29:24.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.827+0000 7f784b7fe640 1 -- 172.21.15.122:0/356775887 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f78440172c0 con 0x7f78540b41f0 2024-03-06T07:29:24.829 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:24.829 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:24.830 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:24.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f783406b4d0 msgr2=0x7f783406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f783406b4d0 0x7f783406d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f785406c980 tx=0x7f783c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 msgr2=0x7f7854161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f7854161150 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f784400b3f0 tx=0x7f784400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 shutdown_connections 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f783406b4d0 0x7f783406d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f785415dfe0 0x7f78540b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78540d22b0 0x7f78540cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 --2- 172.21.15.122:0/356775887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78540b41f0 0x7f7854161150 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 >> 172.21.15.122:0/356775887 conn(0x7f78540d3cd0 msgr2=0x7f78540d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 shutdown_connections 2024-03-06T07:29:24.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:24.829+0000 7f7859577640 1 -- 172.21.15.122:0/356775887 wait complete. 2024-03-06T07:29:25.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:25.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:25.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:25.043 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:25.977 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:26.044 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:26.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.148+0000 7fc7bdbf7640 1 Processor -- start 2024-03-06T07:29:26.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.148+0000 7fc7bdbf7640 1 -- start start 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7bdbf7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7b815dfe0 0x7fc7b80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7bdbf7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b80b3430 con 0x7fc7b80d29c0 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7bdbf7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b80b3710 con 0x7fc7b80cf830 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7bdbf7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b80b31e0 con 0x7fc7b815dfe0 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7b815dfe0 0x7fc7b80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc7b815dfe0 0x7fc7b80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34060/0 (socket says 172.21.15.122:34060) 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b77fe640 1 -- 172.21.15.122:0/3562821995 learned_addr learned my addr 172.21.15.122:0/3562821995 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:26.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc7b80d29c0 0x7fc7b80b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46968/0 (socket says 172.21.15.122:46968) 2024-03-06T07:29:26.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b77fe640 1 -- 172.21.15.122:0/3562821995 >> v1:172.21.15.169:6789/0 conn(0x7fc7b80cf830 legacy=0x7fc7b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b77fe640 1 -- 172.21.15.122:0/3562821995 >> v1:172.21.15.122:6789/0 conn(0x7fc7b80d29c0 legacy=0x7fc7b80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b77fe640 1 -- 172.21.15.122:0/3562821995 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b80b39f0 con 0x7fc7b815dfe0 2024-03-06T07:29:26.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.149+0000 7fc7b67fc640 1 -- 172.21.15.122:0/3562821995 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88854794 0 0) 0x7fc7b80b3430 con 0x7fc7b80d29c0 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.150+0000 7fc7b77fe640 1 --2- 172.21.15.122:0/3562821995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b815dfe0 0x7fc7b80cf2f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80b3360 tx=0x7fc7a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167c321f8d8cf452 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.150+0000 7fc7b67fc640 1 -- 172.21.15.122:0/3562821995 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8034070 con 0x7fc7b815dfe0 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.150+0000 7fc7b67fc640 1 -- 172.21.15.122:0/3562821995 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a8030840 con 0x7fc7b815dfe0 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.150+0000 7fc7b67fc640 1 -- 172.21.15.122:0/3562821995 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8030c10 con 0x7fc7b815dfe0 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.150+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/3562821995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b815dfe0 msgr2=0x7fc7b80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.150+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/3562821995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b815dfe0 0x7fc7b80cf2f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80b3360 tx=0x7fc7a802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.151+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/3562821995 shutdown_connections 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.151+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/3562821995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b815dfe0 0x7fc7b80cf2f0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.151+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/3562821995 >> 172.21.15.122:0/3562821995 conn(0x7fc7b80d3cd0 msgr2=0x7fc7b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.151+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/3562821995 shutdown_connections 2024-03-06T07:29:26.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.151+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/3562821995 wait complete. 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.152+0000 7fc7bdbf7640 1 Processor -- start 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.152+0000 7fc7bdbf7640 1 -- start start 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7b80cf830 0x7fc7b8154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 0x7fc7b81f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b815dfe0 0x7fc7b81f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b815e3c0 con 0x7fc7b80d29c0 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b80b3d60 con 0x7fc7b815dfe0 2024-03-06T07:29:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b815e540 con 0x7fc7b80cf830 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 0x7fc7b81f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 0x7fc7b81f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34070/0 (socket says 172.21.15.122:34070) 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 -- 172.21.15.122:0/308953142 learned_addr learned my addr 172.21.15.122:0/308953142 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b77fe640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7b80cf830 0x7fc7b8154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b7fff640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b815dfe0 0x7fc7b81f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 -- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7b80cf830 msgr2=0x7fc7b8154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7b80cf830 0x7fc7b8154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 -- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b815dfe0 msgr2=0x7fc7b81f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b815dfe0 0x7fc7b81f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 -- 172.21.15.122:0/308953142 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b81f5950 con 0x7fc7b80d29c0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b7fff640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b815dfe0 0x7fc7b81f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7b6ffd640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 0x7fc7b81f4f60 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00c430 tx=0x7fc7ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac017070 con 0x7fc7b80d29c0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0035f0 con 0x7fc7b80d29c0 2024-03-06T07:29:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b81f5c30 con 0x7fc7b80d29c0 2024-03-06T07:29:26.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0039e0 con 0x7fc7b80d29c0 2024-03-06T07:29:26.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.153+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b81f61f0 con 0x7fc7b80d29c0 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.155+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b804f330 con 0x7fc7b80d29c0 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.155+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7ac003b80 con 0x7fc7b80d29c0 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.155+0000 7fc79ffff640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7a406b500 0x7fc7a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.156+0000 7fc7b77fe640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7a406b500 0x7fc7a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.156+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7ac089560 con 0x7fc7b80d29c0 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.156+0000 7fc7b77fe640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7a406b500 0x7fc7a406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8036000 tx=0x7fc7a803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.160+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7ac057d10 con 0x7fc7b80d29c0 2024-03-06T07:29:26.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.375+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7b80cf5b0 con 0x7fc7b80d29c0 2024-03-06T07:29:26.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.376+0000 7fc79ffff640 1 -- 172.21.15.122:0/308953142 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7fc7ac05bdb0 con 0x7fc7b80d29c0 2024-03-06T07:29:26.378 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:26.378 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:26.378 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7a406b500 msgr2=0x7fc7a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7a406b500 0x7fc7a406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8036000 tx=0x7fc7a803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 msgr2=0x7fc7b81f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 0x7fc7b81f4f60 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00c430 tx=0x7fc7ac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 shutdown_connections 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc7a406b500 0x7fc7a406d9c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7b815dfe0 0x7fc7b81f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7b80d29c0 0x7fc7b81f4f60 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 --2- 172.21.15.122:0/308953142 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc7b80cf830 0x7fc7b8154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 >> 172.21.15.122:0/308953142 conn(0x7fc7b80d3cd0 msgr2=0x7fc7b80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 shutdown_connections 2024-03-06T07:29:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:26.378+0000 7fc7bdbf7640 1 -- 172.21.15.122:0/308953142 wait complete. 2024-03-06T07:29:26.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:26.592 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:27.593 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 Processor -- start 2024-03-06T07:29:27.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 -- start start 2024-03-06T07:29:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f86dc1621c0 0x7f86dc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc0d4920 0x7f86dc0d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0ceac0 con 0x7f86dc0d2bf0 2024-03-06T07:29:27.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc0cf690 con 0x7f86dc1621c0 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.692+0000 7f86e1b3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0ced60 con 0x7f86dc0d4920 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f86dc1621c0 0x7f86dc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f86dc1621c0 0x7f86dc0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34072/0 (socket says 172.21.15.122:34072) 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 -- 172.21.15.122:0/369742834 learned_addr learned my addr 172.21.15.122:0/369742834 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86daffd640 1 --2- 172.21.15.122:0/369742834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc0d4920 0x7f86dc0d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 -- 172.21.15.122:0/369742834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc0d4920 msgr2=0x7f86dc0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 --2- 172.21.15.122:0/369742834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc0d4920 0x7f86dc0d26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 -- 172.21.15.122:0/369742834 >> v1:172.21.15.122:6789/0 conn(0x7f86dc0d2bf0 legacy=0x7f86dc15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.693+0000 7f86db7fe640 1 -- 172.21.15.122:0/369742834 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0cf040 con 0x7f86dc1621c0 2024-03-06T07:29:27.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86db7fe640 1 --2- 172.21.15.122:0/369742834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc1621c0 0x7f86dc0cfb20 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009870 tx=0x7f86d402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fc6e47d367c380 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86da7fc640 1 -- 172.21.15.122:0/369742834 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4034070 con 0x7f86dc1621c0 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86da7fc640 1 -- 172.21.15.122:0/369742834 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d4030840 con 0x7f86dc1621c0 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86da7fc640 1 -- 172.21.15.122:0/369742834 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4030c10 con 0x7f86dc1621c0 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 -- 172.21.15.122:0/369742834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc1621c0 msgr2=0x7f86dc0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/369742834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc1621c0 0x7f86dc0cfb20 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009870 tx=0x7f86d402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 -- 172.21.15.122:0/369742834 shutdown_connections 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/369742834 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc0d4920 0x7f86dc0d26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/369742834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc1621c0 0x7f86dc0cfb20 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 -- 172.21.15.122:0/369742834 >> 172.21.15.122:0/369742834 conn(0x7f86dc0d3d10 msgr2=0x7f86dc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 -- 172.21.15.122:0/369742834 shutdown_connections 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.694+0000 7f86e1b3a640 1 -- 172.21.15.122:0/369742834 wait complete. 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.695+0000 7f86e1b3a640 1 Processor -- start 2024-03-06T07:29:27.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 -- start start 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86dc0d2bf0 0x7f86dc15b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 0x7f86dc0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc1621c0 0x7f86dc0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc162720 con 0x7f86dc0d4920 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc1625a0 con 0x7f86dc0d2bf0 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86e1b3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc1628a0 con 0x7f86dc1621c0 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86daffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 0x7f86dc0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86daffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 0x7f86dc0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34088/0 (socket says 172.21.15.122:34088) 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.696+0000 7f86daffd640 1 -- 172.21.15.122:0/839677925 learned_addr learned my addr 172.21.15.122:0/839677925 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:27.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86db7fe640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86dc0d2bf0 0x7f86dc15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86dbfff640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc1621c0 0x7f86dc0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86daffd640 1 -- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc1621c0 msgr2=0x7f86dc0cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86daffd640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc1621c0 0x7f86dc0cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86daffd640 1 -- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86dc0d2bf0 msgr2=0x7f86dc15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86daffd640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86dc0d2bf0 0x7f86dc15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86daffd640 1 -- 172.21.15.122:0/839677925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc15dbb0 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86daffd640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 0x7f86dc0ccfb0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f86c400d800 tx=0x7f86c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c400b6c0 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c4019070 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c400ba80 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc15de90 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.697+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc15e450 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.698+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc04f330 con 0x7f86dc0d4920 2024-03-06T07:29:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.698+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86c400bc20 con 0x7f86dc0d4920 2024-03-06T07:29:27.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.699+0000 7f86d3fff640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86bc06b500 0x7f86bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.699+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86c408a620 con 0x7f86dc0d4920 2024-03-06T07:29:27.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.699+0000 7f86db7fe640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86bc06b500 0x7f86bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.699+0000 7f86db7fe640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86bc06b500 0x7f86bc06d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f86d4036000 tx=0x7f86d403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.703+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86c4059530 con 0x7f86dc0d4920 2024-03-06T07:29:27.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.943+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86dc073380 con 0x7f86dc0d4920 2024-03-06T07:29:27.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.944+0000 7f86d3fff640 1 -- 172.21.15.122:0/839677925 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f86c4058e30 con 0x7f86dc0d4920 2024-03-06T07:29:27.947 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:27.947 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:27.947 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:27.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86bc06b500 msgr2=0x7f86bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86bc06b500 0x7f86bc06d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f86d4036000 tx=0x7f86d403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 msgr2=0x7f86dc0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 0x7f86dc0ccfb0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f86c400d800 tx=0x7f86c400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 shutdown_connections 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86bc06b500 0x7f86bc06d9c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86dc1621c0 0x7f86dc0cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86dc0d4920 0x7f86dc0ccfb0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 --2- 172.21.15.122:0/839677925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86dc0d2bf0 0x7f86dc15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 >> 172.21.15.122:0/839677925 conn(0x7f86dc0d3d10 msgr2=0x7f86dc0b7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 shutdown_connections 2024-03-06T07:29:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:27.946+0000 7f86e1b3a640 1 -- 172.21.15.122:0/839677925 wait complete. 2024-03-06T07:29:27.978 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:28.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:28.161 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:29.163 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:29.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 Processor -- start 2024-03-06T07:29:29.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 -- start start 2024-03-06T07:29:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8b815bb20 0x7fa8b80d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b8161980 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b8161c60 con 0x7fa8b815bb20 2024-03-06T07:29:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.264+0000 7fa8be37a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8161f40 con 0x7fa8b80cfff0 2024-03-06T07:29:29.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa8b815bb20 0x7fa8b80d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8bc917640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa8b80ce4b0 0x7fa8b80ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46986/0 (socket says 172.21.15.122:46986) 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8bc917640 1 -- 172.21.15.122:0/3695741356 learned_addr learned my addr 172.21.15.122:0/3695741356 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b77fe640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b7fff640 1 -- 172.21.15.122:0/3695741356 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 msgr2=0x7fa8b80d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b7fff640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b80d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b7fff640 1 -- 172.21.15.122:0/3695741356 >> v1:172.21.15.122:6789/0 conn(0x7fa8b80ce4b0 legacy=0x7fa8b80ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b7fff640 1 -- 172.21.15.122:0/3695741356 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8162220 con 0x7fa8b815bb20 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b6ffd640 1 -- 172.21.15.122:0/3695741356 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256456259 0 0) 0x7fa8b8161980 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.265+0000 7fa8b77fe640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b80d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:29.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.266+0000 7fa8b7fff640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b815bb20 0x7fa8b80d1df0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8456bc2aa4fff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.266+0000 7fa8b6ffd640 1 -- 172.21.15.122:0/3695741356 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8035020 con 0x7fa8b815bb20 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.266+0000 7fa8b6ffd640 1 -- 172.21.15.122:0/3695741356 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8031830 con 0x7fa8b815bb20 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.266+0000 7fa8b6ffd640 1 -- 172.21.15.122:0/3695741356 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8031c00 con 0x7fa8b815bb20 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 -- 172.21.15.122:0/3695741356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b815bb20 msgr2=0x7fa8b80d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b815bb20 0x7fa8b80d1df0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 -- 172.21.15.122:0/3695741356 shutdown_connections 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b80d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 --2- 172.21.15.122:0/3695741356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b815bb20 0x7fa8b80d1df0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 -- 172.21.15.122:0/3695741356 >> 172.21.15.122:0/3695741356 conn(0x7fa8b809a9f0 msgr2=0x7fa8b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 -- 172.21.15.122:0/3695741356 shutdown_connections 2024-03-06T07:29:29.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.267+0000 7fa8be37a640 1 -- 172.21.15.122:0/3695741356 wait complete. 2024-03-06T07:29:29.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.268+0000 7fa8be37a640 1 Processor -- start 2024-03-06T07:29:29.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.269+0000 7fa8be37a640 1 -- start start 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8be37a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 0x7fa8b80c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8be37a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b81f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8be37a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8b815bb20 0x7fa8b8160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8be37a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80d24b0 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8be37a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80d2330 con 0x7fa8b815bb20 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8be37a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80d1980 con 0x7fa8b80cfff0 2024-03-06T07:29:29.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 0x7fa8b80c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 0x7fa8b80c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34098/0 (socket says 172.21.15.122:34098) 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 -- 172.21.15.122:0/2110171402 learned_addr learned my addr 172.21.15.122:0/2110171402 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 -- 172.21.15.122:0/2110171402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 msgr2=0x7fa8b81f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b81f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 -- 172.21.15.122:0/2110171402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8b815bb20 msgr2=0x7fa8b8160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8b815bb20 0x7fa8b8160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 -- 172.21.15.122:0/2110171402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b81f4880 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.270+0000 7fa8b7fff640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 0x7fa8b80c6210 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.271+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8035050 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.271+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8004230 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.271+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b81f4b20 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.271+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a803b800 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.271+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b81f50e0 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.272+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b804f330 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.272+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8a8038020 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.272+0000 7fa8b4ff9640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8a406b4d0 0x7fa8a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.272+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8a80abf80 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.273+0000 7fa8b77fe640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8a406b4d0 0x7fa8a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.273+0000 7fa8b77fe640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8a406b4d0 0x7fa8a406d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac006fd0 tx=0x7fa8ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.281+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8a8040070 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.497+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b80b30a0 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.497+0000 7fa8b4ff9640 1 -- 172.21.15.122:0/2110171402 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7fa8a807afb0 con 0x7fa8b80ce4b0 2024-03-06T07:29:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:29.500 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:29.500 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8a406b4d0 msgr2=0x7fa8a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8a406b4d0 0x7fa8a406d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac006fd0 tx=0x7fa8ac008040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 msgr2=0x7fa8b80c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 0x7fa8b80c6210 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 shutdown_connections 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa8a406b4d0 0x7fa8a406d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8b815bb20 0x7fa8b8160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa8b80cfff0 0x7fa8b81f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 --2- 172.21.15.122:0/2110171402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8b80ce4b0 0x7fa8b80c6210 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.499+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 >> 172.21.15.122:0/2110171402 conn(0x7fa8b809a9f0 msgr2=0x7fa8b806f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.500+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 shutdown_connections 2024-03-06T07:29:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:29.500+0000 7fa8be37a640 1 -- 172.21.15.122:0/2110171402 wait complete. 2024-03-06T07:29:29.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:29.714 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:29.980 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:30.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:30.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.815+0000 7f06ffd9a640 1 Processor -- start 2024-03-06T07:29:30.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.815+0000 7f06ffd9a640 1 -- start start 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06ffd9a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f06f81617e0 0x7f06f80cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06ffd9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06f80d2980 0x7f06f80b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06ffd9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b3840 con 0x7f06f80cf9a0 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06ffd9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80b35f0 con 0x7f06f80d2980 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06ffd9a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80b4170 con 0x7f06f81617e0 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06f80d2980 0x7f06f80b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06f80d2980 0x7f06f80b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34114/0 (socket says 172.21.15.122:34114) 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 -- 172.21.15.122:0/1527658551 learned_addr learned my addr 172.21.15.122:0/1527658551 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:30.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 -- 172.21.15.122:0/1527658551 >> v2:172.21.15.169:3300/0 conn(0x7f06f81617e0 msgr2=0x7f06f80cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fdb36640 1 --2- 172.21.15.122:0/1527658551 >> v2:172.21.15.169:3300/0 conn(0x7f06f81617e0 0x7f06f80cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 --2- 172.21.15.122:0/1527658551 >> v2:172.21.15.169:3300/0 conn(0x7f06f81617e0 0x7f06f80cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 -- 172.21.15.122:0/1527658551 >> v1:172.21.15.169:6789/0 conn(0x7f06f80cf9a0 legacy=0x7f06f80d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fe337640 1 -- 172.21.15.122:0/1527658551 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b3b20 con 0x7f06f80d2980 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.816+0000 7f06fdb36640 1 --2- 172.21.15.122:0/1527658551 >> v2:172.21.15.169:3300/0 conn(0x7f06f81617e0 0x7f06f80cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.817+0000 7f06fe337640 1 --2- 172.21.15.122:0/1527658551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80d2980 0x7f06f80b4600 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f06f808cbe0 tx=0x7f06f002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd377201518e0cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.817+0000 7f06fcb34640 1 -- 172.21.15.122:0/1527658551 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f002c9f0 con 0x7f06f80d2980 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.817+0000 7f06fcb34640 1 -- 172.21.15.122:0/1527658551 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f002cb90 con 0x7f06f80d2980 2024-03-06T07:29:30.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.817+0000 7f06fcb34640 1 -- 172.21.15.122:0/1527658551 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00343f0 con 0x7f06f80d2980 2024-03-06T07:29:30.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.817+0000 7f06ffd9a640 1 -- 172.21.15.122:0/1527658551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80d2980 msgr2=0x7f06f80b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.817+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/1527658551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80d2980 0x7f06f80b4600 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f06f808cbe0 tx=0x7f06f002be50 comp rx=0 tx=0).stop 2024-03-06T07:29:30.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.818+0000 7f06ffd9a640 1 -- 172.21.15.122:0/1527658551 shutdown_connections 2024-03-06T07:29:30.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.818+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/1527658551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80d2980 0x7f06f80b4600 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.818+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/1527658551 >> v2:172.21.15.169:3300/0 conn(0x7f06f81617e0 0x7f06f80cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.818+0000 7f06ffd9a640 1 -- 172.21.15.122:0/1527658551 >> 172.21.15.122:0/1527658551 conn(0x7f06f80d3c90 msgr2=0x7f06f80d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.818+0000 7f06ffd9a640 1 -- 172.21.15.122:0/1527658551 shutdown_connections 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.818+0000 7f06ffd9a640 1 -- 172.21.15.122:0/1527658551 wait complete. 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.819+0000 7f06ffd9a640 1 Processor -- start 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.819+0000 7f06ffd9a640 1 -- start start 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.819+0000 7f06ffd9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 0x7f06f81530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.819+0000 7f06ffd9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06f80d2980 0x7f06f806c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ffd9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06f81617e0 0x7f06f806c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ffd9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b0860 con 0x7f06f80cf9a0 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ffd9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80d4990 con 0x7f06f81617e0 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ffd9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b09e0 con 0x7f06f80d2980 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 0x7f06f81530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 0x7f06f81530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34122/0 (socket says 172.21.15.122:34122) 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 -- 172.21.15.122:0/2884318239 learned_addr learned my addr 172.21.15.122:0/2884318239 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fd335640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06f80d2980 0x7f06f806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fe337640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06f81617e0 0x7f06f806c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 -- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06f80d2980 msgr2=0x7f06f806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06f80d2980 0x7f06f806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 -- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06f81617e0 msgr2=0x7f06f806c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06f81617e0 0x7f06f806c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 -- 172.21.15.122:0/2884318239 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80c8800 con 0x7f06f80cf9a0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fe337640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06f81617e0 0x7f06f806c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06fdb36640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 0x7f06f81530b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f06f400c820 tx=0x7f06f400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4015070 con 0x7f06f80cf9a0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f4007a30 con 0x7f06f80cf9a0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4007e00 con 0x7f06f80cf9a0 2024-03-06T07:29:30.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80c8ae0 con 0x7f06f80cf9a0 2024-03-06T07:29:30.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.820+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80c90a0 con 0x7f06f80cf9a0 2024-03-06T07:29:30.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.821+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f808cce0 con 0x7f06f80cf9a0 2024-03-06T07:29:30.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.822+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06f4018070 con 0x7f06f80cf9a0 2024-03-06T07:29:30.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.823+0000 7f06ee7fc640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f06dc06b4d0 0x7f06dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.823+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06f408a1e0 con 0x7f06f80cf9a0 2024-03-06T07:29:30.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.823+0000 7f06fd335640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f06dc06b4d0 0x7f06dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.823+0000 7f06fd335640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f06dc06b4d0 0x7f06dc06d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f06e8006fd0 tx=0x7f06e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:30.827+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06f808cce0 con 0x7f06f80cf9a0 2024-03-06T07:29:31.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.038+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06f804f330 con 0x7f06f80cf9a0 2024-03-06T07:29:31.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.038+0000 7f06ee7fc640 1 -- 172.21.15.122:0/2884318239 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f06f4059060 con 0x7f06f80cf9a0 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:31.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f06dc06b4d0 msgr2=0x7f06dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f06dc06b4d0 0x7f06dc06d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f06e8006fd0 tx=0x7f06e8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:31.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 msgr2=0x7f06f81530b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 0x7f06f81530b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f06f400c820 tx=0x7f06f400cc20 comp rx=0 tx=0).stop 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 shutdown_connections 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f06dc06b4d0 0x7f06dc06d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06f81617e0 0x7f06f806c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06f80d2980 0x7f06f806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 --2- 172.21.15.122:0/2884318239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f80cf9a0 0x7f06f81530b0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 >> 172.21.15.122:0/2884318239 conn(0x7f06f80d3c90 msgr2=0x7f06f80d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 shutdown_connections 2024-03-06T07:29:31.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:31.041+0000 7f06ffd9a640 1 -- 172.21.15.122:0/2884318239 wait complete. 2024-03-06T07:29:31.260 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:31.260 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:31.982 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:32.260 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:32.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 Processor -- start 2024-03-06T07:29:32.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 -- start start 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff0e80d00c0 0x7ff0e80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff0e80cf850 0x7ff0e80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80d09e0 con 0x7ff0e80ce1b0 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e806ca90 con 0x7ff0e80d00c0 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0ed38c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e806df30 con 0x7ff0e80cf850 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.360+0000 7ff0e6ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff0e80d00c0 0x7ff0e80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e6ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff0e80d00c0 0x7ff0e80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34128/0 (socket says 172.21.15.122:34128) 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff0e80ce1b0 0x7ff0e80ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46988/0 (socket says 172.21.15.122:46988) 2024-03-06T07:29:32.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e6ffd640 1 -- 172.21.15.122:0/1071398934 learned_addr learned my addr 172.21.15.122:0/1071398934 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:32.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e67fc640 1 --2- 172.21.15.122:0/1071398934 >> v2:172.21.15.169:3300/0 conn(0x7ff0e80cf850 0x7ff0e80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e6ffd640 1 -- 172.21.15.122:0/1071398934 >> v2:172.21.15.169:3300/0 conn(0x7ff0e80cf850 msgr2=0x7ff0e80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e6ffd640 1 --2- 172.21.15.122:0/1071398934 >> v2:172.21.15.169:3300/0 conn(0x7ff0e80cf850 0x7ff0e80cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e6ffd640 1 -- 172.21.15.122:0/1071398934 >> v1:172.21.15.122:6789/0 conn(0x7ff0e80ce1b0 legacy=0x7ff0e80ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e6ffd640 1 -- 172.21.15.122:0/1071398934 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e815e380 con 0x7ff0e80d00c0 2024-03-06T07:29:32.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.361+0000 7ff0e5ffb640 1 -- 172.21.15.122:0/1071398934 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827419302 0 0) 0x7ff0e80d09e0 con 0x7ff0e80ce1b0 2024-03-06T07:29:32.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0e6ffd640 1 --2- 172.21.15.122:0/1071398934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80d00c0 0x7ff0e80d04a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8071de0 tx=0x7ff0dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab397337e493b303 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0e5ffb640 1 -- 172.21.15.122:0/1071398934 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc034070 con 0x7ff0e80d00c0 2024-03-06T07:29:32.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0e5ffb640 1 -- 172.21.15.122:0/1071398934 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc030860 con 0x7ff0e80d00c0 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0e5ffb640 1 -- 172.21.15.122:0/1071398934 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc030c30 con 0x7ff0e80d00c0 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 -- 172.21.15.122:0/1071398934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80d00c0 msgr2=0x7ff0e80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/1071398934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80d00c0 0x7ff0e80d04a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8071de0 tx=0x7ff0dc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 -- 172.21.15.122:0/1071398934 shutdown_connections 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/1071398934 >> v2:172.21.15.169:3300/0 conn(0x7ff0e80cf850 0x7ff0e80cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/1071398934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80d00c0 0x7ff0e80d04a0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 -- 172.21.15.122:0/1071398934 >> 172.21.15.122:0/1071398934 conn(0x7ff0e80b35a0 msgr2=0x7ff0e80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 -- 172.21.15.122:0/1071398934 shutdown_connections 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.362+0000 7ff0ed38c640 1 -- 172.21.15.122:0/1071398934 wait complete. 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 Processor -- start 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 -- start start 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e80ce1b0 0x7ff0e80cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 0x7ff0e80c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e80d00c0 0x7ff0e8153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e806c440 con 0x7ff0e80cf850 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e806c160 con 0x7ff0e80d00c0 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0ed38c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e806c720 con 0x7ff0e80ce1b0 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.364+0000 7ff0e67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 0x7ff0e80c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 0x7ff0e80c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34130/0 (socket says 172.21.15.122:34130) 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 -- 172.21.15.122:0/361704791 learned_addr learned my addr 172.21.15.122:0/361704791 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e6ffd640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e80ce1b0 0x7ff0e80cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e77fe640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e80d00c0 0x7ff0e8153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 -- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e80ce1b0 msgr2=0x7ff0e80cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e80ce1b0 0x7ff0e80cb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 -- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e80d00c0 msgr2=0x7ff0e8153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e80d00c0 0x7ff0e8153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 -- 172.21.15.122:0/361704791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80cbd50 con 0x7ff0e80cf850 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e6ffd640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e80ce1b0 0x7ff0e80cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e77fe640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e80d00c0 0x7ff0e8153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0e67fc640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 0x7ff0e80c8b40 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c7f0 tx=0x7ff0d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8015070 con 0x7ff0e80cf850 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8004240 con 0x7ff0e80cf850 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8007620 con 0x7ff0e80cf850 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e80c6a40 con 0x7ff0e80cf850 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.365+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e815d4f0 con 0x7ff0e80cf850 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.366+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e804f330 con 0x7ff0e80cf850 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.367+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0d80079d0 con 0x7ff0e80cf850 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.367+0000 7ff0cf7fe640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d406b500 0x7ff0d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.367+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0d808a270 con 0x7ff0e80cf850 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.367+0000 7ff0e6ffd640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d406b500 0x7ff0d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.367+0000 7ff0e6ffd640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d406b500 0x7ff0d406d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc036000 tx=0x7ff0dc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.371+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0d8059180 con 0x7ff0e80cf850 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.587+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0e80b3d60 con 0x7ff0e80cf850 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.587+0000 7ff0cf7fe640 1 -- 172.21.15.122:0/361704791 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7ff0d8058a80 con 0x7ff0e80cf850 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:32.590 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:32.590 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:32.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d406b500 msgr2=0x7ff0d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d406b500 0x7ff0d406d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc036000 tx=0x7ff0dc008030 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 msgr2=0x7ff0e80c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 0x7ff0e80c8b40 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c7f0 tx=0x7ff0d800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 shutdown_connections 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff0d406b500 0x7ff0d406d9c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0e80d00c0 0x7ff0e8153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0e80cf850 0x7ff0e80c8b40 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 --2- 172.21.15.122:0/361704791 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0e80ce1b0 0x7ff0e80cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 >> 172.21.15.122:0/361704791 conn(0x7ff0e80b35a0 msgr2=0x7ff0e80b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 shutdown_connections 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:32.589+0000 7ff0ed38c640 1 -- 172.21.15.122:0/361704791 wait complete. 2024-03-06T07:29:32.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:32.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:33.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:33.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.902+0000 7fb9a915c640 1 Processor -- start 2024-03-06T07:29:33.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.902+0000 7fb9a915c640 1 -- start start 2024-03-06T07:29:33.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a915c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb9a415fa10 0x7fb9a40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a915c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb9a40d16e0 0x7fb9a40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a915c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40cebe0 con 0x7fb9a415bdd0 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a915c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a40ceec0 con 0x7fb9a415fa10 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a915c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a40cf1a0 con 0x7fb9a40d16e0 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb9a415fa10 0x7fb9a40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a2d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb9a415bdd0 0x7fb9a40c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46998/0 (socket says 172.21.15.122:46998) 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a2d9d640 1 -- 172.21.15.122:0/2318981262 learned_addr learned my addr 172.21.15.122:0/2318981262 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a259c640 1 -- 172.21.15.122:0/2318981262 >> v2:172.21.15.169:3300/0 conn(0x7fb9a40d16e0 msgr2=0x7fb9a40d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a259c640 1 --2- 172.21.15.122:0/2318981262 >> v2:172.21.15.169:3300/0 conn(0x7fb9a40d16e0 0x7fb9a40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a259c640 1 -- 172.21.15.122:0/2318981262 >> v1:172.21.15.122:6789/0 conn(0x7fb9a415bdd0 legacy=0x7fb9a40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a1d9b640 1 -- 172.21.15.122:0/2318981262 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749849703 0 0) 0x7fb9a40cebe0 con 0x7fb9a415bdd0 2024-03-06T07:29:33.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.903+0000 7fb9a259c640 1 -- 172.21.15.122:0/2318981262 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40cc8e0 con 0x7fb9a415fa10 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.904+0000 7fb9a259c640 1 --2- 172.21.15.122:0/2318981262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415fa10 0x7fb9a40ce270 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fb9a40c5520 tx=0x7fb99802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddb0dbd995b897 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.904+0000 7fb9a1d9b640 1 -- 172.21.15.122:0/2318981262 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9980315a0 con 0x7fb9a415fa10 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.904+0000 7fb9a1d9b640 1 -- 172.21.15.122:0/2318981262 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb998037070 con 0x7fb9a415fa10 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.904+0000 7fb9a1d9b640 1 -- 172.21.15.122:0/2318981262 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998031960 con 0x7fb9a415fa10 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 -- 172.21.15.122:0/2318981262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415fa10 msgr2=0x7fb9a40ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2318981262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415fa10 0x7fb9a40ce270 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fb9a40c5520 tx=0x7fb99802b840 comp rx=0 tx=0).stop 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 -- 172.21.15.122:0/2318981262 shutdown_connections 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2318981262 >> v2:172.21.15.169:3300/0 conn(0x7fb9a40d16e0 0x7fb9a40d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2318981262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415fa10 0x7fb9a40ce270 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 -- 172.21.15.122:0/2318981262 >> 172.21.15.122:0/2318981262 conn(0x7fb9a409a9f0 msgr2=0x7fb9a409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 -- 172.21.15.122:0/2318981262 shutdown_connections 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.905+0000 7fb9a915c640 1 -- 172.21.15.122:0/2318981262 wait complete. 2024-03-06T07:29:33.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.906+0000 7fb9a915c640 1 Processor -- start 2024-03-06T07:29:33.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.906+0000 7fb9a915c640 1 -- start start 2024-03-06T07:29:33.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a915c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9a40d16e0 0x7fb9a40b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a915c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 0x7fb9a40cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a915c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9a415fa10 0x7fb9a40c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a915c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40d2030 con 0x7fb9a415bdd0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a915c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a415c1b0 con 0x7fb9a40d16e0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a915c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40ce7b0 con 0x7fb9a415fa10 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 0x7fb9a40cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 0x7fb9a40cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:34144/0 (socket says 172.21.15.122:34144) 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 -- 172.21.15.122:0/2847769951 learned_addr learned my addr 172.21.15.122:0/2847769951 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a359e640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9a415fa10 0x7fb9a40c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a2d9d640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9a40d16e0 0x7fb9a40b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 -- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9a415fa10 msgr2=0x7fb9a40c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9a415fa10 0x7fb9a40c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 -- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9a40d16e0 msgr2=0x7fb9a40b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9a40d16e0 0x7fb9a40b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a259c640 1 -- 172.21.15.122:0/2847769951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40c9d40 con 0x7fb9a415bdd0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a359e640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9a415fa10 0x7fb9a40c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.907+0000 7fb9a2d9d640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9a40d16e0 0x7fb9a40b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.908+0000 7fb9a259c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 0x7fb9a40cb740 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fb998000c00 tx=0x7fb9980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.908+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998004230 con 0x7fb9a415bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.908+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9980043d0 con 0x7fb9a415bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.908+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a40b5d00 con 0x7fb9a415bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.908+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998032860 con 0x7fb9a415bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.908+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a40b62c0 con 0x7fb9a415bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.909+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a404f330 con 0x7fb9a415bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.909+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb998032a00 con 0x7fb9a415bdd0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.910+0000 7fb98b7fe640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb99006b4d0 0x7fb99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.910+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb99802f030 con 0x7fb9a415bdd0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.910+0000 7fb9a2d9d640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb99006b4d0 0x7fb99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.910+0000 7fb9a2d9d640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb99006b4d0 0x7fb99006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb99c007fd0 tx=0x7fb99c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:33.914+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb998041070 con 0x7fb9a415bdd0 2024-03-06T07:29:33.983 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.128+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9a40b7a40 con 0x7fb9a415bdd0 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.128+0000 7fb98b7fe640 1 -- 172.21.15.122:0/2847769951 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7fb998044340 con 0x7fb9a415bdd0 2024-03-06T07:29:34.131 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:34.131 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:34.131 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:34.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb99006b4d0 msgr2=0x7fb99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb99006b4d0 0x7fb99006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb99c007fd0 tx=0x7fb99c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 msgr2=0x7fb9a40cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 0x7fb9a40cb740 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fb998000c00 tx=0x7fb9980046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 shutdown_connections 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb99006b4d0 0x7fb99006d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb9a415fa10 0x7fb9a40c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9a415bdd0 0x7fb9a40cb740 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 --2- 172.21.15.122:0/2847769951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9a40d16e0 0x7fb9a40b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 >> 172.21.15.122:0/2847769951 conn(0x7fb9a409a9f0 msgr2=0x7fb9a406f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 shutdown_connections 2024-03-06T07:29:34.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:34.130+0000 7fb9a915c640 1 -- 172.21.15.122:0/2847769951 wait complete. 2024-03-06T07:29:34.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:34.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:35.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:35.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.457+0000 7f5744088640 1 Processor -- start 2024-03-06T07:29:35.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5744088640 1 -- start start 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5744088640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f573c15dfe0 0x7f573c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5744088640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f573c0d29c0 0x7f573c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5744088640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0b3430 con 0x7f573c0cf830 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5744088640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0b31e0 con 0x7f573c0d29c0 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5744088640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0b3d60 con 0x7f573c15dfe0 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5742625640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f573c0d29c0 0x7f573c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5741623640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f573c0cf830 0x7f573c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47134/0 (socket says 172.21.15.122:47134) 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.458+0000 7f5741623640 1 -- 172.21.15.122:0/3263743875 learned_addr learned my addr 172.21.15.122:0/3263743875 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5741e24640 1 --2- 172.21.15.122:0/3263743875 >> v2:172.21.15.169:3300/0 conn(0x7f573c15dfe0 0x7f573c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5742625640 1 -- 172.21.15.122:0/3263743875 >> v2:172.21.15.169:3300/0 conn(0x7f573c15dfe0 msgr2=0x7f573c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5742625640 1 --2- 172.21.15.122:0/3263743875 >> v2:172.21.15.169:3300/0 conn(0x7f573c15dfe0 0x7f573c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5742625640 1 -- 172.21.15.122:0/3263743875 >> v1:172.21.15.122:6789/0 conn(0x7f573c0cf830 legacy=0x7f573c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5740e22640 1 -- 172.21.15.122:0/3263743875 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870942122 0 0) 0x7f573c0b3430 con 0x7f573c0cf830 2024-03-06T07:29:35.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5742625640 1 -- 172.21.15.122:0/3263743875 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0b3710 con 0x7f573c0d29c0 2024-03-06T07:29:35.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.459+0000 7f5741e24640 1 --2- 172.21.15.122:0/3263743875 >> v2:172.21.15.169:3300/0 conn(0x7f573c15dfe0 0x7f573c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:35.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5742625640 1 --2- 172.21.15.122:0/3263743875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c0d29c0 0x7f573c0b41f0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f573c067990 tx=0x7f573402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e411db5f10708e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5740e22640 1 -- 172.21.15.122:0/3263743875 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734039070 con 0x7f573c0d29c0 2024-03-06T07:29:35.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5740e22640 1 -- 172.21.15.122:0/3263743875 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573402cc80 con 0x7f573c0d29c0 2024-03-06T07:29:35.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5740e22640 1 -- 172.21.15.122:0/3263743875 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734034660 con 0x7f573c0d29c0 2024-03-06T07:29:35.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5744088640 1 -- 172.21.15.122:0/3263743875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c0d29c0 msgr2=0x7f573c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5744088640 1 --2- 172.21.15.122:0/3263743875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c0d29c0 0x7f573c0b41f0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f573c067990 tx=0x7f573402bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5744088640 1 -- 172.21.15.122:0/3263743875 shutdown_connections 2024-03-06T07:29:35.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5744088640 1 --2- 172.21.15.122:0/3263743875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c0d29c0 0x7f573c0b41f0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5744088640 1 --2- 172.21.15.122:0/3263743875 >> v2:172.21.15.169:3300/0 conn(0x7f573c15dfe0 0x7f573c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.460+0000 7f5744088640 1 -- 172.21.15.122:0/3263743875 >> 172.21.15.122:0/3263743875 conn(0x7f573c0d3cd0 msgr2=0x7f573c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.461+0000 7f5744088640 1 -- 172.21.15.122:0/3263743875 shutdown_connections 2024-03-06T07:29:35.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.461+0000 7f5744088640 1 -- 172.21.15.122:0/3263743875 wait complete. 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 Processor -- start 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 -- start start 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f573c0cf830 0x7f573c1f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573c0d29c0 0x7f573c1f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 0x7f573c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0d1df0 con 0x7f573c15dfe0 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0d1c70 con 0x7f573c0d29c0 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.462+0000 7f5744088640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c15e3c0 con 0x7f573c0cf830 2024-03-06T07:29:35.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 0x7f573c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 0x7f573c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37504/0 (socket says 172.21.15.122:37504) 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 -- 172.21.15.122:0/2023367682 learned_addr learned my addr 172.21.15.122:0/2023367682 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5741e24640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f573c0cf830 0x7f573c1f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5741623640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573c0d29c0 0x7f573c1f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 -- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f573c0cf830 msgr2=0x7f573c1f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f573c0cf830 0x7f573c1f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 -- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573c0d29c0 msgr2=0x7f573c1f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573c0d29c0 0x7f573c1f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 -- 172.21.15.122:0/2023367682 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c1f3700 con 0x7f573c15dfe0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5742625640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 0x7f573c0b68d0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f5734037040 tx=0x7f5734004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734039050 con 0x7f573c15dfe0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734045070 con 0x7f573c15dfe0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c1f4820 con 0x7f573c15dfe0 2024-03-06T07:29:35.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573403e830 con 0x7f573c15dfe0 2024-03-06T07:29:35.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.463+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c0b9080 con 0x7f573c15dfe0 2024-03-06T07:29:35.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.464+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c04f330 con 0x7f573c15dfe0 2024-03-06T07:29:35.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.465+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573403e9d0 con 0x7f573c15dfe0 2024-03-06T07:29:35.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.465+0000 7f57327fc640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f572006b4d0 0x7f572006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.465+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f57340ad260 con 0x7f573c15dfe0 2024-03-06T07:29:35.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.465+0000 7f5741e24640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f572006b4d0 0x7f572006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.466+0000 7f5741e24640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f572006b4d0 0x7f572006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5738006fd0 tx=0x7f5738008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.474+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f573c1f4820 con 0x7f573c15dfe0 2024-03-06T07:29:35.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.706+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f573c0b49c0 con 0x7f573c15dfe0 2024-03-06T07:29:35.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.706+0000 7f57327fc640 1 -- 172.21.15.122:0/2023367682 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f573407c1c0 con 0x7f573c15dfe0 2024-03-06T07:29:35.709 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:35.709 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:35.709 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f572006b4d0 msgr2=0x7f572006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f572006b4d0 0x7f572006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5738006fd0 tx=0x7f5738008040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 msgr2=0x7f573c0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 0x7f573c0b68d0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f5734037040 tx=0x7f5734004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 shutdown_connections 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f572006b4d0 0x7f572006d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f573c15dfe0 0x7f573c0b68d0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f573c0d29c0 0x7f573c1f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 --2- 172.21.15.122:0/2023367682 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f573c0cf830 0x7f573c1f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 >> 172.21.15.122:0/2023367682 conn(0x7f573c0d3cd0 msgr2=0x7f573c06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 shutdown_connections 2024-03-06T07:29:35.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:35.709+0000 7f5744088640 1 -- 172.21.15.122:0/2023367682 wait complete. 2024-03-06T07:29:35.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:35.922 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:35.985 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:36.923 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:37.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.023+0000 7fc2c11f6640 1 Processor -- start 2024-03-06T07:29:37.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.023+0000 7fc2c11f6640 1 -- start start 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2c11f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2bc15bdd0 0x7fc2bc0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2c11f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0cebe0 con 0x7fc2bc0d16e0 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2c11f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0ceec0 con 0x7fc2bc15fa10 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2c11f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0ce980 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2bad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2bc15bdd0 0x7fc2bc0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2bb59e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc2bc0d16e0 0x7fc2bc0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47136/0 (socket says 172.21.15.122:47136) 2024-03-06T07:29:37.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2bb59e640 1 -- 172.21.15.122:0/1151537050 learned_addr learned my addr 172.21.15.122:0/1151537050 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:37.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2bad9d640 1 -- 172.21.15.122:0/1151537050 >> v1:172.21.15.169:6789/0 conn(0x7fc2bc15fa10 legacy=0x7fc2bc0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2bad9d640 1 -- 172.21.15.122:0/1151537050 >> v1:172.21.15.122:6789/0 conn(0x7fc2bc0d16e0 legacy=0x7fc2bc0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2bad9d640 1 -- 172.21.15.122:0/1151537050 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0cf1a0 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.024+0000 7fc2b9d9b640 1 -- 172.21.15.122:0/1151537050 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606644849 0 0) 0x7fc2bc0cebe0 con 0x7fc2bc0d16e0 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.025+0000 7fc2bad9d640 1 --2- 172.21.15.122:0/1151537050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc0c5610 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40097a0 tx=0x7fc2b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfeae33d2c30fa54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.025+0000 7fc2b9d9b640 1 -- 172.21.15.122:0/1151537050 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b40315a0 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.025+0000 7fc2b9d9b640 1 -- 172.21.15.122:0/1151537050 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b4037070 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.025+0000 7fc2b9d9b640 1 -- 172.21.15.122:0/1151537050 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4031960 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 -- 172.21.15.122:0/1151537050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 msgr2=0x7fc2bc0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/1151537050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc0c5610 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40097a0 tx=0x7fc2b402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:37.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 -- 172.21.15.122:0/1151537050 shutdown_connections 2024-03-06T07:29:37.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/1151537050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc0c5610 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 -- 172.21.15.122:0/1151537050 >> 172.21.15.122:0/1151537050 conn(0x7fc2bc09a9f0 msgr2=0x7fc2bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 -- 172.21.15.122:0/1151537050 shutdown_connections 2024-03-06T07:29:37.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.026+0000 7fc2c11f6640 1 -- 172.21.15.122:0/1151537050 wait complete. 2024-03-06T07:29:37.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.028+0000 7fc2c11f6640 1 Processor -- start 2024-03-06T07:29:37.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.028+0000 7fc2c11f6640 1 -- start start 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2c11f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2bc0d16e0 0x7fc2bc15edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2c11f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc15c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2c11f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2bc15fa10 0x7fc2bc0c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2c11f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0cc5d0 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2c11f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0c5b50 con 0x7fc2bc0d16e0 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2c11f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc15c1b0 con 0x7fc2bc15fa10 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc15c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc15c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37520/0 (socket says 172.21.15.122:37520) 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 -- 172.21.15.122:0/3998397452 learned_addr learned my addr 172.21.15.122:0/3998397452 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2bad9d640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2bc0d16e0 0x7fc2bc15edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2bb59e640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2bc15fa10 0x7fc2bc0c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 -- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2bc15fa10 msgr2=0x7fc2bc0c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2bc15fa10 0x7fc2bc0c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 -- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2bc0d16e0 msgr2=0x7fc2bc15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.029+0000 7fc2ba59c640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2bc0d16e0 0x7fc2bc15edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2ba59c640 1 -- 172.21.15.122:0/3998397452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc15cd30 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2bad9d640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2bc0d16e0 0x7fc2bc15edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2ba59c640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc15c3c0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000b9c0 tx=0x7fc2b000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000cba0 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b000cd40 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc15d010 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0003bb0 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.030+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc15f300 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.031+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc04f330 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.031+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2b0014440 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.032+0000 7fc2a37fe640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2a806b4d0 0x7fc2a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.032+0000 7fc2bad9d640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2a806b4d0 0x7fc2a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.032+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc2b008a110 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.032+0000 7fc2bad9d640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2a806b4d0 0x7fc2a806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4036000 tx=0x7fc2b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.036+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2b0058f50 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.332+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2bc0b7a40 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.332+0000 7fc2a37fe640 1 -- 172.21.15.122:0/3998397452 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7fc2b0058850 con 0x7fc2bc15bdd0 2024-03-06T07:29:37.335 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:37.335 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:37.335 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:37.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2a806b4d0 msgr2=0x7fc2a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2a806b4d0 0x7fc2a806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4036000 tx=0x7fc2b4008030 comp rx=0 tx=0).stop 2024-03-06T07:29:37.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 msgr2=0x7fc2bc15c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc15c3c0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000b9c0 tx=0x7fc2b000be80 comp rx=0 tx=0).stop 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 shutdown_connections 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc2a806b4d0 0x7fc2a806d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc2bc15fa10 0x7fc2bc0c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2bc15bdd0 0x7fc2bc15c3c0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 --2- 172.21.15.122:0/3998397452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2bc0d16e0 0x7fc2bc15edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 >> 172.21.15.122:0/3998397452 conn(0x7fc2bc09a9f0 msgr2=0x7fc2bc06f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 shutdown_connections 2024-03-06T07:29:37.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:37.335+0000 7fc2c11f6640 1 -- 172.21.15.122:0/3998397452 wait complete. 2024-03-06T07:29:37.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:37.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:37.987 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:38.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.650+0000 7ff61ea39640 1 Processor -- start 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.650+0000 7ff61ea39640 1 -- start start 2024-03-06T07:29:38.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.651+0000 7ff61ea39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 0x7ff6180b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.651+0000 7ff61ea39640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6180b3430 con 0x7ff6180d22b0 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff61ea39640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6180b3710 con 0x7ff61815dfe0 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff61ea39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b31e0 con 0x7ff6180d2710 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff6177fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff6180d22b0 0x7ff6180b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47148/0 (socket says 172.21.15.122:47148) 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff6177fe640 1 -- 172.21.15.122:0/3597389061 learned_addr learned my addr 172.21.15.122:0/3597389061 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff61cfd6640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 0x7ff6180b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590404173 0 0) 0x7ff6180b3430 con 0x7ff6180d22b0 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff600003520 con 0x7ff6180d22b0 2024-03-06T07:29:38.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142621381 0 0) 0x7ff600003520 con 0x7ff6180d22b0 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 msgr2=0x7ff6180b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 0x7ff6180b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 >> v1:172.21.15.169:6789/0 conn(0x7ff61815dfe0 legacy=0x7ff6180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6180b39f0 con 0x7ff6180d22b0 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff60c001670 con 0x7ff6180d22b0 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 >> v1:172.21.15.122:6789/0 conn(0x7ff6180d22b0 legacy=0x7ff6180b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 0x7ff600003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff600004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff600004d40 0x7ff6000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b31e0 con 0x7ff600003f80 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60000bad0 con 0x7ff600004d40 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60000bcd0 con 0x7ff6180d2710 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff6177fe640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff600004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff617fff640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 0x7ff600003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.652+0000 7ff61cfd6640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff600004d40 0x7ff6000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff6177fe640 1 -- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 msgr2=0x7ff600003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff6177fe640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180d2710 0x7ff600003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff6177fe640 1 -- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff600004d40 msgr2=0x7ff6000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff6177fe640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff600004d40 0x7ff6000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff6177fe640 1 -- 172.21.15.122:0/3597389061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600003520 con 0x7ff600003f80 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff61cfd6640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff600004d40 0x7ff6000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:38.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.653+0000 7ff6177fe640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff600004800 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c210 tx=0x7ff600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf8428adfd11dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c007d10 con 0x7ff600003f80 2024-03-06T07:29:38.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c005ad0 con 0x7ff600003f80 2024-03-06T07:29:38.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff616ffd640 1 -- 172.21.15.122:0/3597389061 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c007070 con 0x7ff600003f80 2024-03-06T07:29:38.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61ea39640 1 -- 172.21.15.122:0/3597389061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 msgr2=0x7ff600004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61ea39640 1 --2- 172.21.15.122:0/3597389061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff600004800 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c210 tx=0x7ff600005830 comp rx=0 tx=0).stop 2024-03-06T07:29:38.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61cfd6640 1 -- 172.21.15.122:0/3597389061 reap_dead start 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61ea39640 1 -- 172.21.15.122:0/3597389061 shutdown_connections 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61ea39640 1 -- 172.21.15.122:0/3597389061 >> 172.21.15.122:0/3597389061 conn(0x7ff6180d3cd0 msgr2=0x7ff6180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61ea39640 1 -- 172.21.15.122:0/3597389061 shutdown_connections 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.654+0000 7ff61ea39640 1 -- 172.21.15.122:0/3597389061 wait complete. 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.656+0000 7ff61ea39640 1 Processor -- start 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.656+0000 7ff61ea39640 1 -- start start 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.657+0000 7ff61ea39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff618153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.657+0000 7ff61ea39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180cdab0 0x7ff6181608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.657+0000 7ff61ea39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff618160de0 0x7ff61815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.657+0000 7ff61ea39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180cfa00 con 0x7ff600003f80 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.657+0000 7ff61ea39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b3d60 con 0x7ff618160de0 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.657+0000 7ff61ea39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180d1c70 con 0x7ff6180cdab0 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff618153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff618153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37542/0 (socket says 172.21.15.122:37542) 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 -- 172.21.15.122:0/703141532 learned_addr learned my addr 172.21.15.122:0/703141532 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff6177fe640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180cdab0 0x7ff6181608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 -- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180cdab0 msgr2=0x7ff6181608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180cdab0 0x7ff6181608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 -- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff618160de0 msgr2=0x7ff61815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff61cfd6640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff618160de0 0x7ff61815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff618160de0 0x7ff61815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 -- 172.21.15.122:0/703141532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6181550c0 con 0x7ff600003f80 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff61cfd6640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff618160de0 0x7ff61815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff617fff640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff618153480 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7ff60800f880 tx=0x7ff60800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608001d20 con 0x7ff600003f80 2024-03-06T07:29:38.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608018070 con 0x7ff600003f80 2024-03-06T07:29:38.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6181553a0 con 0x7ff600003f80 2024-03-06T07:29:38.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60800d730 con 0x7ff600003f80 2024-03-06T07:29:38.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.658+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff618161260 con 0x7ff600003f80 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.659+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61804f330 con 0x7ff600003f80 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.660+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff60800d8d0 con 0x7ff600003f80 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.660+0000 7ff614ff9640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b4d0 0x7ff60406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.660+0000 7ff6177fe640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b4d0 0x7ff60406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.660+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff60808a050 con 0x7ff600003f80 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.661+0000 7ff6177fe640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b4d0 0x7ff60406d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff60c03c000 tx=0x7ff60c037000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.665+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff608058f20 con 0x7ff600003f80 2024-03-06T07:29:38.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.879+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6180b1bf0 con 0x7ff600003f80 2024-03-06T07:29:38.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.879+0000 7ff614ff9640 1 -- 172.21.15.122:0/703141532 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7ff608058820 con 0x7ff600003f80 2024-03-06T07:29:38.881 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:38.882 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:38.882 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b4d0 msgr2=0x7ff60406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b4d0 0x7ff60406d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff60c03c000 tx=0x7ff60c037000 comp rx=0 tx=0).stop 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 msgr2=0x7ff618153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff618153480 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7ff60800f880 tx=0x7ff60800fd40 comp rx=0 tx=0).stop 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 shutdown_connections 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff60406b4d0 0x7ff60406d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff618160de0 0x7ff61815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6180cdab0 0x7ff6181608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 --2- 172.21.15.122:0/703141532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff600003f80 0x7ff618153480 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 >> 172.21.15.122:0/703141532 conn(0x7ff6180d3cd0 msgr2=0x7ff6180b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 shutdown_connections 2024-03-06T07:29:38.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:38.881+0000 7ff61ea39640 1 -- 172.21.15.122:0/703141532 wait complete. 2024-03-06T07:29:39.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:39.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:39.990 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:40.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.195+0000 7f6448076640 1 Processor -- start 2024-03-06T07:29:40.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.195+0000 7f6448076640 1 -- start start 2024-03-06T07:29:40.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6448076640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f64400d2710 0x7f64400b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6448076640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400b3430 con 0x7f64400d22b0 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6448076640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400b3710 con 0x7f644015dfe0 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6448076640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64400b31e0 con 0x7f64400d2710 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6445611640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f64400d22b0 0x7f64400b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47162/0 (socket says 172.21.15.122:47162) 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6445611640 1 -- 172.21.15.122:0/627204199 learned_addr learned my addr 172.21.15.122:0/627204199 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6446613640 1 --2- 172.21.15.122:0/627204199 >> v2:172.21.15.169:3300/0 conn(0x7f64400d2710 0x7f64400b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637897624 0 0) 0x7f64400b3430 con 0x7f64400d22b0 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f642c003520 con 0x7f64400d22b0 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1018794234 0 0) 0x7f642c003520 con 0x7f64400d22b0 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 >> v2:172.21.15.169:3300/0 conn(0x7f64400d2710 msgr2=0x7f64400b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 --2- 172.21.15.122:0/627204199 >> v2:172.21.15.169:3300/0 conn(0x7f64400d2710 0x7f64400b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 >> v1:172.21.15.169:6789/0 conn(0x7f644015dfe0 legacy=0x7f64400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64400b39f0 con 0x7f64400d22b0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f6430001670 con 0x7f64400d22b0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 >> v1:172.21.15.122:6789/0 conn(0x7f64400d22b0 legacy=0x7f64400b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f642c0039e0 0x7f642c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c004800 0x7f642c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f642c0051c0 0x7f642c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64400b31e0 con 0x7f642c0051c0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c00c090 con 0x7f642c0039e0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.196+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c00c290 con 0x7f642c004800 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f642c0051c0 0x7f642c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6445e12640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f642c0039e0 0x7f642c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6445611640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c004800 0x7f642c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 -- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c004800 msgr2=0x7f642c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c004800 0x7f642c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 -- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f642c0039e0 msgr2=0x7f642c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f642c0039e0 0x7f642c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 -- 172.21.15.122:0/627204199 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c003520 con 0x7f642c0051c0 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.197+0000 7f6446613640 1 -- 172.21.15.122:0/627204199 reap_dead start 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6446613640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f642c0051c0 0x7f642c00b9c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f6438009fc0 tx=0x7f642c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc858610d26a0f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438009c10 con 0x7f642c0051c0 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643802cca0 con 0x7f642c0051c0 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6444e10640 1 -- 172.21.15.122:0/627204199 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438003bd0 con 0x7f642c0051c0 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 -- 172.21.15.122:0/627204199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f642c0051c0 msgr2=0x7f642c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f642c0051c0 0x7f642c00b9c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f6438009fc0 tx=0x7f642c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 -- 172.21.15.122:0/627204199 shutdown_connections 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 --2- 172.21.15.122:0/627204199 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f642c0051c0 0x7f642c00b9c0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 -- 172.21.15.122:0/627204199 >> 172.21.15.122:0/627204199 conn(0x7f64400d3cd0 msgr2=0x7f64400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 -- 172.21.15.122:0/627204199 shutdown_connections 2024-03-06T07:29:40.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.198+0000 7f6448076640 1 -- 172.21.15.122:0/627204199 wait complete. 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.199+0000 7f6448076640 1 Processor -- start 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 -- start start 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c0051c0 0x7f64400cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f644015ee10 0x7f644015b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 0x7f64400ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64400d1c70 con 0x7f64400ca160 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64400b3d60 con 0x7f644015ee10 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6448076640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64400d1df0 con 0x7f642c0051c0 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6446613640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 0x7f64400ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6446613640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 0x7f64400ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37562/0 (socket says 172.21.15.122:37562) 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6446613640 1 -- 172.21.15.122:0/3727988562 learned_addr learned my addr 172.21.15.122:0/3727988562 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6445e12640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c0051c0 0x7f64400cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.200+0000 7f6445611640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f644015ee10 0x7f644015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6446613640 1 -- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c0051c0 msgr2=0x7f64400cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6446613640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c0051c0 0x7f64400cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6446613640 1 -- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f644015ee10 msgr2=0x7f644015b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6446613640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f644015ee10 0x7f644015b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6446613640 1 -- 172.21.15.122:0/3727988562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644009ebd0 con 0x7f64400ca160 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6445611640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f644015ee10 0x7f644015b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6446613640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 0x7f64400ca5e0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f643802c6d0 tx=0x7f6438009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438003990 con 0x7f64400ca160 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6438044070 con 0x7f64400ca160 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64400cdb90 con 0x7f64400ca160 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438003d90 con 0x7f64400ca160 2024-03-06T07:29:40.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.201+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644015f690 con 0x7f64400ca160 2024-03-06T07:29:40.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.202+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644004f330 con 0x7f64400ca160 2024-03-06T07:29:40.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.203+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6438005d10 con 0x7f64400ca160 2024-03-06T07:29:40.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.203+0000 7f64367fc640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f642406b500 0x7f642406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.203+0000 7f6445e12640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f642406b500 0x7f642406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.203+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f64380ac950 con 0x7f64400ca160 2024-03-06T07:29:40.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.204+0000 7f6445e12640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f642406b500 0x7f642406d9c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f643c000c00 tx=0x7f643c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.207+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f644009ebd0 con 0x7f64400ca160 2024-03-06T07:29:40.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.436+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64400b2fc0 con 0x7f64400ca160 2024-03-06T07:29:40.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.437+0000 7f64367fc640 1 -- 172.21.15.122:0/3727988562 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f643807b790 con 0x7f64400ca160 2024-03-06T07:29:40.439 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:40.439 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:40.439 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:40.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f642406b500 msgr2=0x7f642406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f642406b500 0x7f642406d9c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f643c000c00 tx=0x7f643c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 msgr2=0x7f64400ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 0x7f64400ca5e0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f643802c6d0 tx=0x7f6438009750 comp rx=0 tx=0).stop 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 shutdown_connections 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f642406b500 0x7f642406d9c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64400ca160 0x7f64400ca5e0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f644015ee10 0x7f644015b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 --2- 172.21.15.122:0/3727988562 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f642c0051c0 0x7f64400cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 >> 172.21.15.122:0/3727988562 conn(0x7f64400d3cd0 msgr2=0x7f64400b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 shutdown_connections 2024-03-06T07:29:40.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:40.439+0000 7f6448076640 1 -- 172.21.15.122:0/3727988562 wait complete. 2024-03-06T07:29:40.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:40.653 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:41.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.754+0000 7fe77913a640 1 Processor -- start 2024-03-06T07:29:41.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.754+0000 7fe77913a640 1 -- start start 2024-03-06T07:29:41.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe77913a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7741621c0 0x7fe7740cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe77913a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740ceac0 con 0x7fe7741625a0 2024-03-06T07:29:41.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe77913a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740d2dc0 con 0x7fe7740d4920 2024-03-06T07:29:41.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe77913a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7740cf690 con 0x7fe7741621c0 2024-03-06T07:29:41.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe772d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7741621c0 0x7fe7740cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe77359e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe7741625a0 0x7fe77415dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:47166/0 (socket says 172.21.15.122:47166) 2024-03-06T07:29:41.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe772d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe7741621c0 0x7fe7740cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37572/0 (socket says 172.21.15.122:37572) 2024-03-06T07:29:41.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe77359e640 1 -- 172.21.15.122:0/3487762604 learned_addr learned my addr 172.21.15.122:0/3487762604 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:41.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe772d9d640 1 -- 172.21.15.122:0/3487762604 >> v1:172.21.15.169:6789/0 conn(0x7fe7740d4920 legacy=0x7fe7740d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe772d9d640 1 -- 172.21.15.122:0/3487762604 >> v1:172.21.15.122:6789/0 conn(0x7fe7741625a0 legacy=0x7fe77415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.755+0000 7fe772d9d640 1 -- 172.21.15.122:0/3487762604 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7740ced60 con 0x7fe7741621c0 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.756+0000 7fe772d9d640 1 --2- 172.21.15.122:0/3487762604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7741621c0 0x7fe7740cfb20 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0097a0 tx=0x7fe75c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e1ac9ddc92193c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.756+0000 7fe771d9b640 1 -- 172.21.15.122:0/3487762604 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0315a0 con 0x7fe7741621c0 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.756+0000 7fe771d9b640 1 -- 172.21.15.122:0/3487762604 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c037070 con 0x7fe7741621c0 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.756+0000 7fe771d9b640 1 -- 172.21.15.122:0/3487762604 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c031960 con 0x7fe7741621c0 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.756+0000 7fe77913a640 1 -- 172.21.15.122:0/3487762604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7741621c0 msgr2=0x7fe7740cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.756+0000 7fe77913a640 1 --2- 172.21.15.122:0/3487762604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7741621c0 0x7fe7740cfb20 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0097a0 tx=0x7fe75c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:41.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.757+0000 7fe77913a640 1 -- 172.21.15.122:0/3487762604 shutdown_connections 2024-03-06T07:29:41.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.757+0000 7fe77913a640 1 --2- 172.21.15.122:0/3487762604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7741621c0 0x7fe7740cfb20 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.757+0000 7fe77913a640 1 -- 172.21.15.122:0/3487762604 >> 172.21.15.122:0/3487762604 conn(0x7fe7740d3d10 msgr2=0x7fe7740d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.757+0000 7fe77913a640 1 -- 172.21.15.122:0/3487762604 shutdown_connections 2024-03-06T07:29:41.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.757+0000 7fe77913a640 1 -- 172.21.15.122:0/3487762604 wait complete. 2024-03-06T07:29:41.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.758+0000 7fe77913a640 1 Processor -- start 2024-03-06T07:29:41.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.758+0000 7fe77913a640 1 -- start start 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77913a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 0x7fe7740cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77913a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7741621c0 0x7fe77414bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77913a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7741625a0 0x7fe77414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77913a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77415e450 con 0x7fe7740d4920 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77913a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740cf320 con 0x7fe7741621c0 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77913a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77415e5d0 con 0x7fe7741625a0 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 0x7fe7740cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 0x7fe7740cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37588/0 (socket says 172.21.15.122:37588) 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 -- 172.21.15.122:0/2341542111 learned_addr learned my addr 172.21.15.122:0/2341542111 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77359e640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7741625a0 0x7fe77414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77259c640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7741621c0 0x7fe77414bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 -- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7741625a0 msgr2=0x7fe77414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7741625a0 0x7fe77414c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 -- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7741621c0 msgr2=0x7fe77414bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7741621c0 0x7fe77414bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 -- 172.21.15.122:0/2341542111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7741e7830 con 0x7fe7740d4920 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77359e640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7741625a0 0x7fe77414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe77259c640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7741621c0 0x7fe77414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.759+0000 7fe772d9d640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 0x7fe7740cc930 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fe75c034040 tx=0x7fe75c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.760+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c032840 con 0x7fe7740d4920 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.760+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c032a00 con 0x7fe7740d4920 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.760+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c032df0 con 0x7fe7740d4920 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.760+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7741e7b10 con 0x7fe7740d4920 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.760+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7741e80d0 con 0x7fe7740d4920 2024-03-06T07:29:41.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.761+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77404f330 con 0x7fe7740d4920 2024-03-06T07:29:41.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.761+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe75c03b3f0 con 0x7fe7740d4920 2024-03-06T07:29:41.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.761+0000 7fe76b7fe640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe75406b4d0 0x7fe75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.762+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe75c02f030 con 0x7fe7740d4920 2024-03-06T07:29:41.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.762+0000 7fe77259c640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe75406b4d0 0x7fe75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.762+0000 7fe77259c640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe75406b4d0 0x7fe75406d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe77406bab0 tx=0x7fe764009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.770+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe75c040070 con 0x7fe7740d4920 2024-03-06T07:29:41.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.984+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7740b4bb0 con 0x7fe7740d4920 2024-03-06T07:29:41.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.985+0000 7fe76b7fe640 1 -- 172.21.15.122:0/2341542111 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7fe75c07afc0 con 0x7fe7740d4920 2024-03-06T07:29:41.987 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:41.987 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:41.987 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.987+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe75406b4d0 msgr2=0x7fe75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe75406b4d0 0x7fe75406d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe77406bab0 tx=0x7fe764009040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 msgr2=0x7fe7740cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 0x7fe7740cc930 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fe75c034040 tx=0x7fe75c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 shutdown_connections 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe75406b4d0 0x7fe75406d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe7741625a0 0x7fe77414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7741621c0 0x7fe77414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 --2- 172.21.15.122:0/2341542111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7740d4920 0x7fe7740cc930 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 >> 172.21.15.122:0/2341542111 conn(0x7fe7740d3d10 msgr2=0x7fe7740cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 shutdown_connections 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:41.988+0000 7fe77913a640 1 -- 172.21.15.122:0/2341542111 wait complete. 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:42.201 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:42.201 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:43.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:43.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.339+0000 7f33d0071640 1 Processor -- start 2024-03-06T07:29:43.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.339+0000 7f33d0071640 1 -- start start 2024-03-06T07:29:43.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33d0071640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f33c80cce40 0x7f33c815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33d0071640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f33c80cd7a0 0x7f33c80cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33d0071640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c8071f40 con 0x7f33c81515e0 2024-03-06T07:29:43.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33d0071640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c80cd2f0 con 0x7f33c80cce40 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33d0071640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c8070a70 con 0x7f33c80cd7a0 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f33c80cce40 0x7f33c815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f33c80cce40 0x7f33c815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37600/0 (socket says 172.21.15.122:37600) 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 -- 172.21.15.122:0/2286722293 learned_addr learned my addr 172.21.15.122:0/2286722293 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33ce60e640 1 --2- 172.21.15.122:0/2286722293 >> v2:172.21.15.169:3300/0 conn(0x7f33c80cd7a0 0x7f33c80cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 -- 172.21.15.122:0/2286722293 >> v2:172.21.15.169:3300/0 conn(0x7f33c80cd7a0 msgr2=0x7f33c80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 --2- 172.21.15.122:0/2286722293 >> v2:172.21.15.169:3300/0 conn(0x7f33c80cd7a0 0x7f33c80cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 -- 172.21.15.122:0/2286722293 >> v1:172.21.15.169:6789/0 conn(0x7f33c81515e0 legacy=0x7f33c8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.340+0000 7f33cd60c640 1 -- 172.21.15.122:0/2286722293 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c8070d50 con 0x7f33c80cce40 2024-03-06T07:29:43.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.341+0000 7f33cd60c640 1 --2- 172.21.15.122:0/2286722293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c815b430 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f33c80cc340 tx=0x7f33b802ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b532585723e93f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.341+0000 7f33cce0b640 1 -- 172.21.15.122:0/2286722293 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b803a070 con 0x7f33c80cce40 2024-03-06T07:29:43.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33cce0b640 1 -- 172.21.15.122:0/2286722293 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b802dc30 con 0x7f33c80cce40 2024-03-06T07:29:43.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33cce0b640 1 -- 172.21.15.122:0/2286722293 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b800b040 con 0x7f33c80cce40 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 -- 172.21.15.122:0/2286722293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 msgr2=0x7f33c815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 --2- 172.21.15.122:0/2286722293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c815b430 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f33c80cc340 tx=0x7f33b802ce80 comp rx=0 tx=0).stop 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 -- 172.21.15.122:0/2286722293 shutdown_connections 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 --2- 172.21.15.122:0/2286722293 >> v2:172.21.15.169:3300/0 conn(0x7f33c80cd7a0 0x7f33c80cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 --2- 172.21.15.122:0/2286722293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c815b430 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 -- 172.21.15.122:0/2286722293 >> 172.21.15.122:0/2286722293 conn(0x7f33c80cf8a0 msgr2=0x7f33c80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 -- 172.21.15.122:0/2286722293 shutdown_connections 2024-03-06T07:29:43.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.342+0000 7f33d0071640 1 -- 172.21.15.122:0/2286722293 wait complete. 2024-03-06T07:29:43.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.344+0000 7f33d0071640 1 Processor -- start 2024-03-06T07:29:43.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.344+0000 7f33d0071640 1 -- start start 2024-03-06T07:29:43.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33d0071640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c8143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33d0071640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33c80cd7a0 0x7f33c806d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33d0071640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33c81515e0 0x7f33c806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33d0071640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80ce0c0 con 0x7f33c80cce40 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33d0071640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c8071310 con 0x7f33c81515e0 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33d0071640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80cc3a0 con 0x7f33c80cd7a0 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33cde0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c8143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33cde0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c8143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37608/0 (socket says 172.21.15.122:37608) 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33cde0d640 1 -- 172.21.15.122:0/3042505339 learned_addr learned my addr 172.21.15.122:0/3042505339 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33ce60e640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33c81515e0 0x7f33c806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.345+0000 7f33cd60c640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33c80cd7a0 0x7f33c806d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cde0d640 1 -- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33c80cd7a0 msgr2=0x7f33c806d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cde0d640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33c80cd7a0 0x7f33c806d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cde0d640 1 -- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33c81515e0 msgr2=0x7f33c806d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cde0d640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33c81515e0 0x7f33c806d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cde0d640 1 -- 172.21.15.122:0/3042505339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c8143790 con 0x7f33c80cce40 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cd60c640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33c80cd7a0 0x7f33c806d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33cde0d640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c8143250 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c340 tx=0x7f33c400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4017070 con 0x7f33c80cce40 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c40052d0 con 0x7f33c80cce40 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4003850 con 0x7f33c80cce40 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c81f5730 con 0x7f33c80cce40 2024-03-06T07:29:43.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.346+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c81f5cf0 con 0x7f33c80cce40 2024-03-06T07:29:43.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.347+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c8068840 con 0x7f33c80cce40 2024-03-06T07:29:43.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.347+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33c401a070 con 0x7f33c80cce40 2024-03-06T07:29:43.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.348+0000 7f33be7fc640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f33ac06b500 0x7f33ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.348+0000 7f33cd60c640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f33ac06b500 0x7f33ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.348+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f33c4089410 con 0x7f33c80cce40 2024-03-06T07:29:43.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.348+0000 7f33cd60c640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f33ac06b500 0x7f33ac06d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f33b8037000 tx=0x7f33b8031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.352+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33c4057c90 con 0x7f33c80cce40 2024-03-06T07:29:43.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.731+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33c804f330 con 0x7f33c80cce40 2024-03-06T07:29:43.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.732+0000 7f33be7fc640 1 -- 172.21.15.122:0/3042505339 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f33c405bd30 con 0x7f33c80cce40 2024-03-06T07:29:43.734 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:43.735 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f33ac06b500 msgr2=0x7f33ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f33ac06b500 0x7f33ac06d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f33b8037000 tx=0x7f33b8031000 comp rx=0 tx=0).stop 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 msgr2=0x7f33c8143250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c8143250 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c340 tx=0x7f33c400c800 comp rx=0 tx=0).stop 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 shutdown_connections 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f33ac06b500 0x7f33ac06d9c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33c81515e0 0x7f33c806d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f33c80cd7a0 0x7f33c806d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 --2- 172.21.15.122:0/3042505339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33c80cce40 0x7f33c8143250 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 >> 172.21.15.122:0/3042505339 conn(0x7f33c80cf8a0 msgr2=0x7f33c8072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 shutdown_connections 2024-03-06T07:29:43.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:43.734+0000 7f33d0071640 1 -- 172.21.15.122:0/3042505339 wait complete. 2024-03-06T07:29:43.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:43.948 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:43.993 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:44.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:45.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.050+0000 7f0540481640 1 Processor -- start 2024-03-06T07:29:45.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.050+0000 7f0540481640 1 -- start start 2024-03-06T07:29:45.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f0540481640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f05380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f0540481640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05380cf830 0x7f05380d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f0540481640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f05380d27f0 0x7f05380d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f0540481640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05380d2c50 con 0x7f05380cf830 2024-03-06T07:29:45.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f0540481640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05380b31e0 con 0x7f05380d27f0 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f0540481640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05380b3d60 con 0x7f053815dfe0 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05380cf830 0x7f05380d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05380cf830 0x7f05380d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38826/0 (socket says 172.21.15.122:38826) 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 -- 172.21.15.122:0/3131353640 learned_addr learned my addr 172.21.15.122:0/3131353640 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 -- 172.21.15.122:0/3131353640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 msgr2=0x7f05380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053e21d640 1 --2- 172.21.15.122:0/3131353640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f05380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053ea1e640 1 --2- 172.21.15.122:0/3131353640 >> v2:172.21.15.169:3300/0 conn(0x7f05380d27f0 0x7f05380d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 --2- 172.21.15.122:0/3131353640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f05380cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 -- 172.21.15.122:0/3131353640 >> v2:172.21.15.169:3300/0 conn(0x7f05380d27f0 msgr2=0x7f05380d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 --2- 172.21.15.122:0/3131353640 >> v2:172.21.15.169:3300/0 conn(0x7f05380d27f0 0x7f05380d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053da1c640 1 -- 172.21.15.122:0/3131353640 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05380b3430 con 0x7f05380cf830 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.051+0000 7f053ea1e640 1 --2- 172.21.15.122:0/3131353640 >> v2:172.21.15.169:3300/0 conn(0x7f05380d27f0 0x7f05380d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.052+0000 7f053da1c640 1 --2- 172.21.15.122:0/3131353640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380cf830 0x7f05380d22b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f0538067ad0 tx=0x7f052802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78f3d7e1be2ae05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.052+0000 7f053d21b640 1 -- 172.21.15.122:0/3131353640 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528035020 con 0x7f05380cf830 2024-03-06T07:29:45.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.052+0000 7f053d21b640 1 -- 172.21.15.122:0/3131353640 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528031870 con 0x7f05380cf830 2024-03-06T07:29:45.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.052+0000 7f053d21b640 1 -- 172.21.15.122:0/3131353640 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528031c40 con 0x7f05380cf830 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 -- 172.21.15.122:0/3131353640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380cf830 msgr2=0x7f05380d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 --2- 172.21.15.122:0/3131353640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380cf830 0x7f05380d22b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f0538067ad0 tx=0x7f052802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 -- 172.21.15.122:0/3131353640 shutdown_connections 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 --2- 172.21.15.122:0/3131353640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380cf830 0x7f05380d22b0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 --2- 172.21.15.122:0/3131353640 >> v2:172.21.15.169:3300/0 conn(0x7f05380d27f0 0x7f05380d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 --2- 172.21.15.122:0/3131353640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f05380cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 -- 172.21.15.122:0/3131353640 >> 172.21.15.122:0/3131353640 conn(0x7f05380d3cd0 msgr2=0x7f05380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 -- 172.21.15.122:0/3131353640 shutdown_connections 2024-03-06T07:29:45.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.053+0000 7f0540481640 1 -- 172.21.15.122:0/3131353640 wait complete. 2024-03-06T07:29:45.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.055+0000 7f0540481640 1 Processor -- start 2024-03-06T07:29:45.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.055+0000 7f0540481640 1 -- start start 2024-03-06T07:29:45.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f0540481640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05380cf830 0x7f05380b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f0540481640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 0x7f05381f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f0540481640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f0538149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f0540481640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05380d1c70 con 0x7f05380d27f0 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f0540481640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05380b39f0 con 0x7f05380cf830 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f0540481640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05380d1df0 con 0x7f053815dfe0 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f053da1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 0x7f05381f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f053da1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 0x7f05381f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38834/0 (socket says 172.21.15.122:38834) 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f053da1c640 1 -- 172.21.15.122:0/3751936431 learned_addr learned my addr 172.21.15.122:0/3751936431 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f053e21d640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05380cf830 0x7f05380b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.056+0000 7f053ea1e640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f0538149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f053da1c640 1 -- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 msgr2=0x7f0538149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f053da1c640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f0538149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f053da1c640 1 -- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05380cf830 msgr2=0x7f05380b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f053da1c640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05380cf830 0x7f05380b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f053da1c640 1 -- 172.21.15.122:0/3751936431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05380b8ca0 con 0x7f05380d27f0 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f053da1c640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 0x7f05381f3020 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f052802b950 tx=0x7f05280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528040070 con 0x7f05380d27f0 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052803a420 con 0x7f05380d27f0 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05380b8f20 con 0x7f05380d27f0 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528035040 con 0x7f05380d27f0 2024-03-06T07:29:45.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.057+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05380b94e0 con 0x7f05380d27f0 2024-03-06T07:29:45.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.058+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053804f330 con 0x7f05380d27f0 2024-03-06T07:29:45.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.059+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0528038020 con 0x7f05380d27f0 2024-03-06T07:29:45.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.059+0000 7f052effd640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f051c06b4d0 0x7f051c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.059+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05280ac1b0 con 0x7f05380d27f0 2024-03-06T07:29:45.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.059+0000 7f053e21d640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f051c06b4d0 0x7f051c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.060+0000 7f053e21d640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f051c06b4d0 0x7f051c06d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f05380a1d10 tx=0x7f0534008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.065 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.064+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05380b8ca0 con 0x7f05380d27f0 2024-03-06T07:29:45.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.363+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05380747a0 con 0x7f05380d27f0 2024-03-06T07:29:45.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.364+0000 7f052effd640 1 -- 172.21.15.122:0/3751936431 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f052807a9a0 con 0x7f05380d27f0 2024-03-06T07:29:45.366 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:45.366 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:45.366 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.366+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f051c06b4d0 msgr2=0x7f051c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.366+0000 7f0540481640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f051c06b4d0 0x7f051c06d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f05380a1d10 tx=0x7f0534008040 comp rx=0 tx=0).stop 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.366+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 msgr2=0x7f05381f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.366+0000 7f0540481640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 0x7f05381f3020 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f052802b950 tx=0x7f05280046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 shutdown_connections 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f051c06b4d0 0x7f051c06d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f053815dfe0 0x7f0538149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05380d27f0 0x7f05381f3020 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 --2- 172.21.15.122:0/3751936431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05380cf830 0x7f05380b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 >> 172.21.15.122:0/3751936431 conn(0x7f05380d3cd0 msgr2=0x7f05380d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 shutdown_connections 2024-03-06T07:29:45.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:45.367+0000 7f0540481640 1 -- 172.21.15.122:0/3751936431 wait complete. 2024-03-06T07:29:45.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:45.580 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:45.996 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:46.581 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.681+0000 7faf1a5d0640 1 Processor -- start 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.681+0000 7faf1a5d0640 1 -- start start 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf1a5d0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7faf1406da40 0x7faf140cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf1a5d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 0x7faf1415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf1a5d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf140cc980 con 0x7faf141424f0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf1a5d0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf140cd220 con 0x7faf1406da40 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf1a5d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf140ccc60 con 0x7faf140cd7a0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf13fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faf141424f0 0x7faf1406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41688/0 (socket says 172.21.15.122:41688) 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf13fff640 1 -- 172.21.15.122:0/1843872177 learned_addr learned my addr 172.21.15.122:0/1843872177 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77420047 0 0) 0x7faf140cc980 con 0x7faf141424f0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faefc003520 con 0x7faf141424f0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf18b6d640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 0x7faf1415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf137fe640 1 --2- 172.21.15.122:0/1843872177 >> v2:172.21.15.169:3300/0 conn(0x7faf1406da40 0x7faf140cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3325945630 0 0) 0x7faefc003520 con 0x7faf141424f0 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 msgr2=0x7faf1415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 0x7faf1415cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 >> v2:172.21.15.169:3300/0 conn(0x7faf1406da40 msgr2=0x7faf140cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 --2- 172.21.15.122:0/1843872177 >> v2:172.21.15.169:3300/0 conn(0x7faf1406da40 0x7faf140cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf140ccf40 con 0x7faf141424f0 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7faf04001670 con 0x7faf141424f0 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 >> v1:172.21.15.122:6789/0 conn(0x7faf141424f0 legacy=0x7faf1406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 0x7faefc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faefc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faefc004d40 0x7faefc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140cd220 con 0x7faefc003f80 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140ccc60 con 0x7faefc004d40 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faefc00b9c0 con 0x7faf140cd7a0 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.682+0000 7faf137fe640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faefc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf13fff640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 0x7faefc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf18b6d640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faefc004d40 0x7faefc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf137fe640 1 -- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 msgr2=0x7faefc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf137fe640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf140cd7a0 0x7faefc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf137fe640 1 -- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faefc004d40 msgr2=0x7faefc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf137fe640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faefc004d40 0x7faefc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf137fe640 1 -- 172.21.15.122:0/1843872177 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc003520 con 0x7faefc003f80 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.683+0000 7faf137fe640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faefc004800 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faf08009af0 tx=0x7faefc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d82e17c6140262 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf08039070 con 0x7faefc003f80 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0802c710 con 0x7faefc003f80 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf12ffd640 1 -- 172.21.15.122:0/1843872177 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf08009690 con 0x7faefc003f80 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf1a5d0640 1 -- 172.21.15.122:0/1843872177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 msgr2=0x7faefc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/1843872177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faefc004800 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faf08009af0 tx=0x7faefc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf18b6d640 1 -- 172.21.15.122:0/1843872177 reap_dead start 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf1a5d0640 1 -- 172.21.15.122:0/1843872177 shutdown_connections 2024-03-06T07:29:46.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf1a5d0640 1 -- 172.21.15.122:0/1843872177 >> 172.21.15.122:0/1843872177 conn(0x7faf140cf8a0 msgr2=0x7faf140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf1a5d0640 1 -- 172.21.15.122:0/1843872177 shutdown_connections 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.684+0000 7faf1a5d0640 1 -- 172.21.15.122:0/1843872177 wait complete. 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 Processor -- start 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 -- start start 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faf1413e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf141523c0 0x7faf14152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1413ff60 0x7faf141403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1415d070 con 0x7faefc003f80 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1415cef0 con 0x7faf1413ff60 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf1a5d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1415d1f0 con 0x7faf141523c0 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.686+0000 7faf13fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faf1413e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faf1413e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38850/0 (socket says 172.21.15.122:38850) 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 -- 172.21.15.122:0/3336717008 learned_addr learned my addr 172.21.15.122:0/3336717008 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf137fe640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf141523c0 0x7faf14152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf18b6d640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1413ff60 0x7faf141403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 -- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf141523c0 msgr2=0x7faf14152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf141523c0 0x7faf14152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 -- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1413ff60 msgr2=0x7faf141403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1413ff60 0x7faf141403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 -- 172.21.15.122:0/3336717008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1415f270 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf13fff640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faf1413e9b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7faf04001a10 tx=0x7faf040077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04001720 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf04018070 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0400fb80 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf14151600 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.687+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf14151bc0 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.688+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1404f330 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.689+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf0400fd20 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.689+0000 7faf10ff9640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf0006b670 0x7faf0006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.689+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf0408a2d0 con 0x7faefc003f80 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.689+0000 7faf137fe640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf0006b670 0x7faf0006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.690+0000 7faf137fe640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf0006b670 0x7faf0006db30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faf08036000 tx=0x7faf08030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.695+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf04059110 con 0x7faefc003f80 2024-03-06T07:29:46.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.905+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf140cc0d0 con 0x7faefc003f80 2024-03-06T07:29:46.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.905+0000 7faf10ff9640 1 -- 172.21.15.122:0/3336717008 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3422 (secure 0 0 0) 0x7faf04058a10 con 0x7faefc003f80 2024-03-06T07:29:46.908 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:46.908 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:17.105248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8688,"mds_1":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":1,"incarnation":74,"state":"up:stopping","state_seq":4,"addr":"172.21.15.122:6836/1727230917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1727230917},{"type":"v1","addr":"172.21.15.122:6836","nonce":1727230917}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:46.908 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 81 2024-03-06T07:29:46.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf0006b670 msgr2=0x7faf0006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf0006b670 0x7faf0006db30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faf08036000 tx=0x7faf08030000 comp rx=0 tx=0).stop 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 msgr2=0x7faf1413e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faf1413e9b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7faf04001a10 tx=0x7faf040077a0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 shutdown_connections 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf0006b670 0x7faf0006db30 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1413ff60 0x7faf141403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf141523c0 0x7faf14152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 --2- 172.21.15.122:0/3336717008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faefc003f80 0x7faf1413e9b0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 >> 172.21.15.122:0/3336717008 conn(0x7faf140cf8a0 msgr2=0x7faf1406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 shutdown_connections 2024-03-06T07:29:46.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:46.907+0000 7faf1a5d0640 1 -- 172.21.15.122:0/3336717008 wait complete. 2024-03-06T07:29:47.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:17.105248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8688, 'mds_1': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 1, 'incarnation': 74, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.122:6836/1727230917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 1727230917}, {'type': 'v1', 'addr': '172.21.15.122:6836', 'nonce': 1727230917}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:47.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:stopping 2024-03-06T07:29:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:48.071 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:29:48.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.225+0000 7f6471481640 1 Processor -- start 2024-03-06T07:29:48.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.225+0000 7f6471481640 1 -- start start 2024-03-06T07:29:48.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f6471481640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f646c0ae4c0 0x7f646c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f6471481640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 0x7f646c0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f6471481640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0b0160 con 0x7f646c15e110 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f6471481640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c0ae030 con 0x7f646c0ae4c0 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f6471481640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0ad9e0 con 0x7f646c06b4f0 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 0x7f646c0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646a7fc640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f646c0ae4c0 0x7f646c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 0x7f646c0b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:42206/0 (socket says 172.21.15.122:42206) 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646b7fe640 1 -- 172.21.15.122:0/3365417249 learned_addr learned my addr 172.21.15.122:0/3365417249 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646b7fe640 1 -- 172.21.15.122:0/3365417249 >> v2:172.21.15.169:3300/0 conn(0x7f646c0ae4c0 msgr2=0x7f646c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646b7fe640 1 --2- 172.21.15.122:0/3365417249 >> v2:172.21.15.169:3300/0 conn(0x7f646c0ae4c0 0x7f646c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.227+0000 7f646b7fe640 1 -- 172.21.15.122:0/3365417249 >> v1:172.21.15.169:6789/0 conn(0x7f646c15e110 legacy=0x7f646c0b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.228+0000 7f646b7fe640 1 -- 172.21.15.122:0/3365417249 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c0adcc0 con 0x7f646c06b4f0 2024-03-06T07:29:48.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.228+0000 7f646a7fc640 1 --2- 172.21.15.122:0/3365417249 >> v2:172.21.15.169:3300/0 conn(0x7f646c0ae4c0 0x7f646c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.228+0000 7f646b7fe640 1 --2- 172.21.15.122:0/3365417249 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 0x7f646c0b38f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f645c009e50 tx=0x7f645c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88ff75c4531daeb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.228+0000 7f6469ffb640 1 -- 172.21.15.122:0/3365417249 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c02cb90 con 0x7f646c06b4f0 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6469ffb640 1 -- 172.21.15.122:0/3365417249 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c02cd30 con 0x7f646c06b4f0 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6469ffb640 1 -- 172.21.15.122:0/3365417249 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c034610 con 0x7f646c06b4f0 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 -- 172.21.15.122:0/3365417249 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 msgr2=0x7f646c0b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 --2- 172.21.15.122:0/3365417249 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 0x7f646c0b38f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f645c009e50 tx=0x7f645c02bff0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 -- 172.21.15.122:0/3365417249 shutdown_connections 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 --2- 172.21.15.122:0/3365417249 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c06b4f0 0x7f646c0b38f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 --2- 172.21.15.122:0/3365417249 >> v2:172.21.15.169:3300/0 conn(0x7f646c0ae4c0 0x7f646c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 -- 172.21.15.122:0/3365417249 >> 172.21.15.122:0/3365417249 conn(0x7f646c0d0960 msgr2=0x7f646c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 -- 172.21.15.122:0/3365417249 shutdown_connections 2024-03-06T07:29:48.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.229+0000 7f6471481640 1 -- 172.21.15.122:0/3365417249 wait complete. 2024-03-06T07:29:48.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.231+0000 7f6471481640 1 Processor -- start 2024-03-06T07:29:48.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.231+0000 7f6471481640 1 -- start start 2024-03-06T07:29:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f6471481640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646c06b4f0 0x7f646c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f6471481640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c0ae4c0 0x7f646c15f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f6471481640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 0x7f646c147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f6471481640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0d17e0 con 0x7f646c15e110 2024-03-06T07:29:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f6471481640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0d1660 con 0x7f646c06b4f0 2024-03-06T07:29:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f6471481640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0afcf0 con 0x7f646c0ae4c0 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 0x7f646c147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 0x7f646c147480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38878/0 (socket says 172.21.15.122:38878) 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 -- 172.21.15.122:0/2566917582 learned_addr learned my addr 172.21.15.122:0/2566917582 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646a7fc640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c0ae4c0 0x7f646c15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646affd640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646c06b4f0 0x7f646c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 -- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c0ae4c0 msgr2=0x7f646c15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c0ae4c0 0x7f646c15f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 -- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646c06b4f0 msgr2=0x7f646c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646c06b4f0 0x7f646c0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 -- 172.21.15.122:0/2566917582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c15fe90 con 0x7f646c15e110 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.232+0000 7f646b7fe640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 0x7f646c147480 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f645c009f70 tx=0x7f645c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.233+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c043020 con 0x7f646c15e110 2024-03-06T07:29:48.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.233+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c03e450 con 0x7f646c15e110 2024-03-06T07:29:48.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.233+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c009510 con 0x7f646c15e110 2024-03-06T07:29:48.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.233+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c15f650 con 0x7f646c15e110 2024-03-06T07:29:48.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.233+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c15fbb0 con 0x7f646c15e110 2024-03-06T07:29:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.234+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c04f330 con 0x7f646c15e110 2024-03-06T07:29:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.234+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f645c046020 con 0x7f646c15e110 2024-03-06T07:29:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.235+0000 7f64637fe640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.235+0000 7f646affd640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.235+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f645c0ac7c0 con 0x7f646c15e110 2024-03-06T07:29:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.235+0000 7f646affd640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f644c06b4d0 0x7f644c06d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6464007fd0 tx=0x7f6464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.239+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f645c04d030 con 0x7f646c15e110 2024-03-06T07:29:48.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.455+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f646c0d06b0 con 0x7f646c15e110 2024-03-06T07:29:48.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.456+0000 7f64637fe640 1 -- 172.21.15.122:0/2566917582 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2612 (secure 0 0 0) 0x7f646c15f650 con 0x7f646c15e110 2024-03-06T07:29:48.458 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:48.458 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:47.105721+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:48.458 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 82 2024-03-06T07:29:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f644c06b4d0 msgr2=0x7f644c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f644c06b4d0 0x7f644c06d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6464007fd0 tx=0x7f6464009040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 msgr2=0x7f646c147480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 0x7f646c147480 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f645c009f70 tx=0x7f645c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 shutdown_connections 2024-03-06T07:29:48.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f646c15e110 0x7f646c147480 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f646c0ae4c0 0x7f646c15f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 --2- 172.21.15.122:0/2566917582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f646c06b4f0 0x7f646c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 >> 172.21.15.122:0/2566917582 conn(0x7f646c0d0960 msgr2=0x7f646c15d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 shutdown_connections 2024-03-06T07:29:48.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.458+0000 7f6471481640 1 -- 172.21.15.122:0/2566917582 wait complete. 2024-03-06T07:29:48.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:41.515900+0000', 'modified': '2024-03-06T07:29:47.105721+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8688}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8688': {'gid': 8688, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/2094222247', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6835', 'nonce': 2094222247}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2094222247}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8688, 'qdb_cluster': [8688]} 2024-03-06T07:29:48.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:48.672 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:29:48.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.772+0000 7fba648b5640 1 Processor -- start 2024-03-06T07:29:48.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.772+0000 7fba648b5640 1 -- start start 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.773+0000 7fba648b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba600ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.773+0000 7fba648b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba600d25e0 con 0x7fba6015e000 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.773+0000 7fba648b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba600d2880 con 0x7fba600d0d90 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.773+0000 7fba648b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba600c7c50 con 0x7fba600c90f0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.773+0000 7fba5e59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fba6015e000 0x7fba600d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41698/0 (socket says 172.21.15.122:41698) 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.773+0000 7fba5e59c640 1 -- 172.21.15.122:0/2205051587 learned_addr learned my addr 172.21.15.122:0/2205051587 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba600ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791189538 0 0) 0x7fba600d25e0 con 0x7fba6015e000 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c003520 con 0x7fba6015e000 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061198392 0 0) 0x7fba4c003520 con 0x7fba6015e000 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 msgr2=0x7fba600ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba600ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 >> v1:172.21.15.169:6789/0 conn(0x7fba600d0d90 legacy=0x7fba600ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba600d2b40 con 0x7fba6015e000 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fba50001670 con 0x7fba6015e000 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 >> v1:172.21.15.122:6789/0 conn(0x7fba6015e000 legacy=0x7fba600d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba4c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba4c004390 0x7fba4c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba4c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba600c7c50 con 0x7fba4c0039e0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c00b9f0 con 0x7fba4c004390 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c00bbf0 con 0x7fba600c90f0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5e59c640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba4c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba4c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba57fff640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba4c004390 0x7fba4c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5e59c640 1 -- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 msgr2=0x7fba4c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5e59c640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba4c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5e59c640 1 -- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba4c004390 msgr2=0x7fba4c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5e59c640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba4c004390 0x7fba4c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5e59c640 1 -- 172.21.15.122:0/2205051587 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c003520 con 0x7fba4c0039e0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.774+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba600c90f0 0x7fba4c0050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.775+0000 7fba5e59c640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba4c003dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fba5000c290 tx=0x7fba4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddfe16ab3145e02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50007d50 con 0x7fba4c0039e0 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba50030070 con 0x7fba4c0039e0 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba5dd9b640 1 -- 172.21.15.122:0/2205051587 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5002cd80 con 0x7fba4c0039e0 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba648b5640 1 -- 172.21.15.122:0/2205051587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 msgr2=0x7fba4c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba648b5640 1 --2- 172.21.15.122:0/2205051587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba4c003dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fba5000c290 tx=0x7fba4c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba5ed9d640 1 -- 172.21.15.122:0/2205051587 reap_dead start 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba648b5640 1 -- 172.21.15.122:0/2205051587 shutdown_connections 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba648b5640 1 -- 172.21.15.122:0/2205051587 >> 172.21.15.122:0/2205051587 conn(0x7fba600d3cf0 msgr2=0x7fba600d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba648b5640 1 -- 172.21.15.122:0/2205051587 shutdown_connections 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.776+0000 7fba648b5640 1 -- 172.21.15.122:0/2205051587 wait complete. 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 Processor -- start 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 -- start start 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba60156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba6006ca10 0x7fba6006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba60156b50 0x7fba60156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6015e3e0 con 0x7fba4c0039e0 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba600d22d0 con 0x7fba6006ca10 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.778+0000 7fba648b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba600d4810 con 0x7fba60156b50 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba60156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba57fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba6006ca10 0x7fba6006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba60156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38890/0 (socket says 172.21.15.122:38890) 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5ed9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba60156b50 0x7fba60156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 -- 172.21.15.122:0/3307437920 learned_addr learned my addr 172.21.15.122:0/3307437920 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 -- 172.21.15.122:0/3307437920 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba60156b50 msgr2=0x7fba60156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba60156b50 0x7fba60156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 -- 172.21.15.122:0/3307437920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba6006ca10 msgr2=0x7fba6006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba6006ca10 0x7fba6006ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 -- 172.21.15.122:0/3307437920 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba601578c0 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba5e59c640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba60156120 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fba5000c290 tx=0x7fba50001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba56ffd640 1 -- 172.21.15.122:0/3307437920 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50045070 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba56ffd640 1 -- 172.21.15.122:0/3307437920 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba50004c50 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba60157ba0 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba56ffd640 1 -- 172.21.15.122:0/3307437920 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50007d50 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.779+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6009dd70 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.781+0000 7fba56ffd640 1 -- 172.21.15.122:0/3307437920 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba50007070 con 0x7fba4c0039e0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.782+0000 7fba56ffd640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba4006b500 0x7fba4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.782+0000 7fba56ffd640 1 -- 172.21.15.122:0/3307437920 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba500ab2a0 con 0x7fba4c0039e0 2024-03-06T07:29:48.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.782+0000 7fba57fff640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba4006b500 0x7fba4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.782+0000 7fba57fff640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba4006b500 0x7fba4006d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fba6006d7b0 tx=0x7fba48001430 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.782+0000 7fba6757e740 1 Processor -- start 2024-03-06T07:29:48.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.782+0000 7fba6757e740 1 -- start start 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e167870 0x556b1e167c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba6757e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 0x556b1e16d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e197600 0x556b1e197a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba6757e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e0d16b0 con 0x556b1e16b8c0 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e07aa60 con 0x556b1e197600 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b1e0cd750 con 0x556b1e167870 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 0x556b1e16d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 0x556b1e16d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38894/0 (socket says 172.21.15.122:38894) 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 -- 172.21.15.122:0/3814584512 learned_addr learned my addr 172.21.15.122:0/3814584512 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba5e59c640 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e167870 0x556b1e167c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e197600 0x556b1e197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 -- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e167870 msgr2=0x556b1e167c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e167870 0x556b1e167c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 -- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e197600 msgr2=0x556b1e197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e197600 0x556b1e197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba57fff640 1 -- 172.21.15.122:0/3814584512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b1e098f30 con 0x556b1e16b8c0 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.783+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e197600 0x556b1e197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.784+0000 7fba57fff640 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 0x556b1e16d360 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fba48011990 tx=0x7fba48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163d085028e5f13c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.784+0000 7fba3ffff640 1 -- 172.21.15.122:0/3814584512 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48015bc0 con 0x556b1e16b8c0 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.784+0000 7fba3ffff640 1 -- 172.21.15.122:0/3814584512 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba48015d60 con 0x556b1e16b8c0 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.784+0000 7fba3ffff640 1 -- 172.21.15.122:0/3814584512 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48005020 con 0x556b1e16b8c0 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.784+0000 7fba6757e740 1 -- 172.21.15.122:0/3814584512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 msgr2=0x556b1e16d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.784+0000 7fba6757e740 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 0x556b1e16d360 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fba48011990 tx=0x7fba48014000 comp rx=0 tx=0).stop 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 -- 172.21.15.122:0/3814584512 shutdown_connections 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e197600 0x556b1e197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e16b8c0 0x556b1e16d360 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 --2- 172.21.15.122:0/3814584512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e167870 0x556b1e167c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 -- 172.21.15.122:0/3814584512 >> 172.21.15.122:0/3814584512 conn(0x556b1e1698f0 msgr2=0x556b1e16b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 -- 172.21.15.122:0/3814584512 shutdown_connections 2024-03-06T07:29:48.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 -- 172.21.15.122:0/3814584512 wait complete. 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 -- start start 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.785+0000 7fba6757e740 1 Processor -- start 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e1afae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e18df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e16bd50 con 0x556b1e197600 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e08dea0 con 0x556b1e18daf0 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b1e16c030 con 0x556b1e18cee0 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba6757e740 10 client.0 _ll_register_callbacks cb 0x556b1e08d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba5e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e1afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba5e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e1afae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38906/0 (socket says 172.21.15.122:38906) 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba5e59c640 1 -- 172.21.15.122:0/301202709 learned_addr learned my addr 172.21.15.122:0/301202709 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba57fff640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.786+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e18df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5e59c640 1 -- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 msgr2=0x556b1e1c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5e59c640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5e59c640 1 -- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 msgr2=0x556b1e18df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5e59c640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e18df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5e59c640 1 -- 172.21.15.122:0/301202709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b1e08e270 con 0x556b1e197600 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e18df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba5e59c640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e1afae0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fba50000f60 tx=0x7fba50079770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00384b58defee4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba3d7fa640 1 -- 172.21.15.122:0/301202709 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50079b70 con 0x556b1e197600 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba3d7fa640 1 -- 172.21.15.122:0/301202709 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba50081d40 con 0x556b1e197600 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba48015d60 con 0x556b1e197600 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba3d7fa640 1 -- 172.21.15.122:0/301202709 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba5007dc10 con 0x556b1e197600 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 10 client.9114 fetch_fsmap learned FSMap version 82 2024-03-06T07:29:48.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x556b1dde62d0 con 0x556b1e197600 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba3d7fa640 1 -- 172.21.15.122:0/301202709 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba48015d60 con 0x556b1e197600 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba3d7fa640 1 -- 172.21.15.122:0/301202709 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 82) v1 ==== 1597+0+0 (secure 0 0 0) 0x7fba5007eb30 con 0x556b1e197600 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 10 client.9114 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 10 client.9114 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 20 client.9114 populate_metadata read hostname 'smithi122' 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e1c3980 0x556b1e194350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 4 client.9114 mds_command: new command op to 8688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 --> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b1e0debf0 con 0x556b1e1c3980 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.787+0000 7fba57fff640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e1c3980 0x556b1e194350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba3d7fa640 10 client.9114 ms_handle_connect on v2:172.21.15.122:6835/2094222247 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba57fff640 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e1c3980 0x556b1e194350 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b8803f5af597c02 server_cookie=fe8ab5d60f292abd in_seq=0 out_seq=0 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba3d7fa640 1 -- 172.21.15.122:0/301202709 <== mds.0 v2:172.21.15.122:6835/2094222247 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556b1e0debf0 con 0x556b1e1c3980 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba3d7fa640 10 client.9114 handle_command_reply: tid=0 2024-03-06T07:29:48.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba6757e740 1 client.9114 shutdown 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 msgr2=0x556b1e1afae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.788+0000 7fba6757e740 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e1afae0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fba50000f60 tx=0x7fba50079770 comp rx=0 tx=0).stop 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 shutdown_connections 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e1c3980 0x556b1e194350 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e18df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba3d7fa640 0 client.9114 ms_handle_reset on v2:172.21.15.122:6835/2094222247 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 --2- 172.21.15.122:0/301202709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e1afae0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 >> 172.21.15.122:0/301202709 conn(0x556b1e167870 msgr2=0x556b1e166510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 shutdown_connections 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 1 -- 172.21.15.122:0/301202709 wait complete. 2024-03-06T07:29:48.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.789+0000 7fba6757e740 20 client.9114 trim_cache size 0 max 16384 2024-03-06T07:29:48.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.805+0000 7fba6757e740 1 Processor -- start 2024-03-06T07:29:48.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.805+0000 7fba6757e740 1 -- start start 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e197a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba6757e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e18f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba6757e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba601ff220 con 0x556b1e197600 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e16bd50 con 0x556b1e18daf0 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b1e18caa0 con 0x556b1e18cee0 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e18f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e18f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38908/0 (socket says 172.21.15.122:38908) 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 -- 172.21.15.122:0/3057820472 learned_addr learned my addr 172.21.15.122:0/3057820472 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5e59c640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba57fff640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 -- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 msgr2=0x556b1e1938e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1938e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 -- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 msgr2=0x556b1e197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 -- 172.21.15.122:0/3057820472 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b1deb06b0 con 0x556b1e197600 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba57fff640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5e59c640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.806+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e18f730 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fba58017680 tx=0x7fba58009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f2cecd9605aa64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba3f7fe640 1 -- 172.21.15.122:0/3057820472 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba580189b0 con 0x556b1e197600 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba3f7fe640 1 -- 172.21.15.122:0/3057820472 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5801c070 con 0x556b1e197600 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba3f7fe640 1 -- 172.21.15.122:0/3057820472 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58018db0 con 0x556b1e197600 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 -- 172.21.15.122:0/3057820472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 msgr2=0x556b1e18f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e18f730 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fba58017680 tx=0x7fba58009750 comp rx=0 tx=0).stop 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 -- 172.21.15.122:0/3057820472 shutdown_connections 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e197600 0x556b1e18f730 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 --2- 172.21.15.122:0/3057820472 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e18cee0 0x556b1e1938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 -- 172.21.15.122:0/3057820472 >> 172.21.15.122:0/3057820472 conn(0x556b1e167870 msgr2=0x556b1e18e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 -- 172.21.15.122:0/3057820472 shutdown_connections 2024-03-06T07:29:48.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.807+0000 7fba6757e740 1 -- 172.21.15.122:0/3057820472 wait complete. 2024-03-06T07:29:48.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.808+0000 7fba6757e740 1 -- start start 2024-03-06T07:29:48.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.808+0000 7fba6757e740 1 Processor -- start 2024-03-06T07:29:48.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 0x556b1e21cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e1cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e197600 0x556b1e1ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e191480 con 0x556b1e18cee0 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1e1cd5c0 con 0x556b1e18daf0 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b1e18c740 con 0x556b1e197600 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 10 client.0 _ll_register_callbacks cb 0x7fba50075f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 0x556b1e21cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 0x556b1e21cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38924/0 (socket says 172.21.15.122:38924) 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 -- 172.21.15.122:0/1917412372 learned_addr learned my addr 172.21.15.122:0/1917412372 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5ed9d640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e197600 0x556b1e1ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba57fff640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e1cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 -- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e197600 msgr2=0x556b1e1ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e197600 0x556b1e1ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 -- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 msgr2=0x556b1e1cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e1cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 -- 172.21.15.122:0/1917412372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b1e1b2170 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba5e59c640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 0x556b1e21cde0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fba50077e20 tx=0x7fba50042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27fe9eb0db0269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba3d7fa640 1 -- 172.21.15.122:0/1917412372 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50095070 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba3d7fa640 1 -- 172.21.15.122:0/1917412372 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba50081f50 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba5801c070 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba3d7fa640 1 -- 172.21.15.122:0/1917412372 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba5009d050 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 10 client.9120 fetch_fsmap learned FSMap version 82 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.809+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x556b1e1b2170 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba3d7fa640 1 -- 172.21.15.122:0/1917412372 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5801c070 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba3d7fa640 1 -- 172.21.15.122:0/1917412372 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 82) v1 ==== 1597+0+0 (secure 0 0 0) 0x7fba50098030 con 0x556b1e18cee0 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 10 client.9120 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:29:48.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 10 client.9120 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 20 client.9120 populate_metadata read hostname 'smithi122' 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e16dac0 0x556b1e222d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 4 client.9120 mds_command: new command op to 8688 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 --> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556b1e0d8f40 con 0x556b1e16dac0 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba57fff640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e16dac0 0x556b1e222d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba3d7fa640 10 client.9120 ms_handle_connect on v2:172.21.15.122:6835/2094222247 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba57fff640 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e16dac0 0x556b1e222d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f07e4dd6265acaae server_cookie=b1292c56c79a1910 in_seq=0 out_seq=0 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba3d7fa640 1 -- 172.21.15.122:0/1917412372 <== mds.0 v2:172.21.15.122:6835/2094222247 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x556b1e0d8f40 con 0x556b1e16dac0 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba3d7fa640 10 client.9120 handle_command_reply: tid=0 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 1 client.9120 shutdown 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 msgr2=0x556b1e21cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.810+0000 7fba6757e740 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 0x556b1e21cde0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fba50077e20 tx=0x7fba50042040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 shutdown_connections 2024-03-06T07:29:48.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:6835/2094222247,v1:172.21.15.122:6837/2094222247] conn(0x556b1e16dac0 0x556b1e222d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x556b1e197600 0x556b1e1ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556b1e18daf0 0x556b1e1cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba3d7fa640 0 client.9120 ms_handle_reset on v2:172.21.15.122:6835/2094222247 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 --2- 172.21.15.122:0/1917412372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556b1e18cee0 0x556b1e21cde0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 >> 172.21.15.122:0/1917412372 conn(0x556b1e167870 msgr2=0x556b1e18f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 shutdown_connections 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 1 -- 172.21.15.122:0/1917412372 wait complete. 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba6757e740 20 client.9120 trim_cache size 0 max 16384 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:29:48.819 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 8688, 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 66.612805304000005, 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 82, 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 130, 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 130, 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 71.708511728000005 2024-03-06T07:29:48.820 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba4006b500 msgr2=0x7fba4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba4006b500 0x7fba4006d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fba6006d7b0 tx=0x7fba48001430 comp rx=0 tx=0).stop 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 msgr2=0x7fba60156120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba60156120 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fba5000c290 tx=0x7fba50001a40 comp rx=0 tx=0).stop 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 shutdown_connections 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba4006b500 0x7fba4006d9c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba60156b50 0x7fba60156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba6006ca10 0x7fba6006ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 --2- 172.21.15.122:0/3307437920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba4c0039e0 0x7fba60156120 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 >> 172.21.15.122:0/3307437920 conn(0x7fba600d3cf0 msgr2=0x7fba600d1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 shutdown_connections 2024-03-06T07:29:48.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48.811+0000 7fba648b5640 1 -- 172.21.15.122:0/3307437920 wait complete. 2024-03-06T07:29:48.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:48 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:29:50.000 INFO:teuthology.orchestra.run.smithi122.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:50.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:50.010 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:50.010 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:50.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:50.131 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:50.131 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:29:50.131 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:50.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:50.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.285+0000 7f6ac38df640 1 Processor -- start 2024-03-06T07:29:50.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.285+0000 7f6ac38df640 1 -- start start 2024-03-06T07:29:50.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac38df640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 0x7f6abc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac38df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cc8b0 0x7f6abc0ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac38df640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc161980 con 0x7f6abc15bb20 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac38df640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0d2500 con 0x7f6abc0cfff0 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac38df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc161c60 con 0x7f6abc0cc8b0 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac0e7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 0x7f6abc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac1e7c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cc8b0 0x7f6abc0ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac167b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f6abc15bb20 0x7f6abc0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:36826/0 (socket says 172.21.15.122:36826) 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac167b640 1 -- 172.21.15.122:0/2059194598 learned_addr learned my addr 172.21.15.122:0/2059194598 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.286+0000 7f6ac0e7a640 1 --2- 172.21.15.122:0/2059194598 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 0x7f6abc0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:41044/0 (socket says 172.21.15.122:41044) 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.287+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2059194598 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 msgr2=0x7f6abc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.287+0000 7f6ac1e7c640 1 --2- 172.21.15.122:0/2059194598 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 0x7f6abc0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.287+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2059194598 >> v1:172.21.15.169:6789/0 conn(0x7f6abc15bb20 legacy=0x7f6abc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.287+0000 7f6ab3fff640 1 -- 172.21.15.122:0/2059194598 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931974038 0 0) 0x7f6abc161980 con 0x7f6abc15bb20 2024-03-06T07:29:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.287+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2059194598 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc161f40 con 0x7f6abc0cc8b0 2024-03-06T07:29:50.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.287+0000 7f6ac0e7a640 1 --2- 172.21.15.122:0/2059194598 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 0x7f6abc0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ac1e7c640 1 --2- 172.21.15.122:0/2059194598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cc8b0 0x7f6abc0ccd20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40097a0 tx=0x7f6ab402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e771ea1031d0781 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ab3fff640 1 -- 172.21.15.122:0/2059194598 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4035020 con 0x7f6abc0cc8b0 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ab3fff640 1 -- 172.21.15.122:0/2059194598 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4031830 con 0x7f6abc0cc8b0 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ab3fff640 1 -- 172.21.15.122:0/2059194598 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4031c00 con 0x7f6abc0cc8b0 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ac38df640 1 -- 172.21.15.122:0/2059194598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cc8b0 msgr2=0x7f6abc0ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2059194598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cc8b0 0x7f6abc0ccd20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40097a0 tx=0x7f6ab402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.288+0000 7f6ac38df640 1 -- 172.21.15.122:0/2059194598 shutdown_connections 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.289+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2059194598 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cc8b0 0x7f6abc0ccd20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.289+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2059194598 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cfff0 0x7f6abc0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.289+0000 7f6ac38df640 1 -- 172.21.15.122:0/2059194598 >> 172.21.15.122:0/2059194598 conn(0x7f6abc09a9f0 msgr2=0x7f6abc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.289+0000 7f6ac38df640 1 -- 172.21.15.122:0/2059194598 shutdown_connections 2024-03-06T07:29:50.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.289+0000 7f6ac38df640 1 -- 172.21.15.122:0/2059194598 wait complete. 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 Processor -- start 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 -- start start 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc0cc8b0 0x7f6abc0cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cfff0 0x7f6abc1e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 0x7f6abc1e9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0d1980 con 0x7f6abc15bb20 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc162500 con 0x7f6abc0cc8b0 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.290+0000 7f6ac38df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0d1b00 con 0x7f6abc0cfff0 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 0x7f6abc1e9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 0x7f6abc1e9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38932/0 (socket says 172.21.15.122:38932) 2024-03-06T07:29:50.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2374230904 learned_addr learned my addr 172.21.15.122:0/2374230904 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac167b640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc0cc8b0 0x7f6abc0cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac0e7a640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cfff0 0x7f6abc1e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cfff0 msgr2=0x7f6abc1e9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cfff0 0x7f6abc1e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc0cc8b0 msgr2=0x7f6abc0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc0cc8b0 0x7f6abc0cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 -- 172.21.15.122:0/2374230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc1e7980 con 0x7f6abc15bb20 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac1e7c640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 0x7f6abc1e9850 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4034040 tx=0x7f6ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4035040 con 0x7f6abc15bb20 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4004230 con 0x7f6abc15bb20 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc1ea060 con 0x7f6abc15bb20 2024-03-06T07:29:50.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab403a640 con 0x7f6abc15bb20 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.291+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc1e7f00 con 0x7f6abc15bb20 2024-03-06T07:29:50.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.293+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc04f330 con 0x7f6abc15bb20 2024-03-06T07:29:50.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.293+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ab4038020 con 0x7f6abc15bb20 2024-03-06T07:29:50.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.293+0000 7f6ab1ffb640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.293+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6ab40abf00 con 0x7f6abc15bb20 2024-03-06T07:29:50.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.293+0000 7f6ac167b640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.294+0000 7f6ac167b640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80012e0 tx=0x7f6ab8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.298+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ab407ad40 con 0x7f6abc15bb20 2024-03-06T07:29:50.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.477+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6abc0b2710 con 0x7f6abc15bb20 2024-03-06T07:29:50.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.478+0000 7f6ab1ffb640 1 -- 172.21.15.122:0/2374230904 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f6ab407a640 con 0x7f6abc15bb20 2024-03-06T07:29:50.481 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:50.481 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":130,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:29:35.994874+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 msgr2=0x7f6aa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80012e0 tx=0x7f6ab8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 msgr2=0x7f6abc1e9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 0x7f6abc1e9850 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4034040 tx=0x7f6ab40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 shutdown_connections 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b500 0x7f6aa006d9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc15bb20 0x7f6abc1e9850 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc0cfff0 0x7f6abc1e9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 --2- 172.21.15.122:0/2374230904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc0cc8b0 0x7f6abc0cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 >> 172.21.15.122:0/2374230904 conn(0x7f6abc09a9f0 msgr2=0x7f6abc09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.480+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 shutdown_connections 2024-03-06T07:29:50.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.481+0000 7f6ac38df640 1 -- 172.21.15.122:0/2374230904 wait complete. 2024-03-06T07:29:50.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:39.745958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:40.497682+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:50.497 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:50.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.600+0000 7f790b466640 1 Processor -- start 2024-03-06T07:29:50.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.600+0000 7f790b466640 1 -- start start 2024-03-06T07:29:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f790b466640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79041614d0 0x7f79040d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f790b466640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d1910 0x7f79040d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f790b466640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79040d2740 con 0x7f79040d22b0 2024-03-06T07:29:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f790b466640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79041618b0 con 0x7f79041614d0 2024-03-06T07:29:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f790b466640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79040cf400 con 0x7f79040d1910 2024-03-06T07:29:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79041614d0 0x7f79040d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f79041614d0 0x7f79040d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38940/0 (socket says 172.21.15.122:38940) 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 -- 172.21.15.122:0/3559966742 learned_addr learned my addr 172.21.15.122:0/3559966742 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7908a01640 1 --2- 172.21.15.122:0/3559966742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d1910 0x7f79040d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 -- 172.21.15.122:0/3559966742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d1910 msgr2=0x7f79040d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 --2- 172.21.15.122:0/3559966742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d1910 0x7f79040d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 -- 172.21.15.122:0/3559966742 >> v1:172.21.15.169:6789/0 conn(0x7f79040d22b0 legacy=0x7f79040d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.601+0000 7f7909202640 1 -- 172.21.15.122:0/3559966742 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040cf6a0 con 0x7f79041614d0 2024-03-06T07:29:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.602+0000 7f7909202640 1 --2- 172.21.15.122:0/3559966742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79041614d0 0x7f79040d4860 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7900009870 tx=0x7f790002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434b3b4c7318953d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.602+0000 7f78fbfff640 1 -- 172.21.15.122:0/3559966742 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900034070 con 0x7f79041614d0 2024-03-06T07:29:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.602+0000 7f78fbfff640 1 -- 172.21.15.122:0/3559966742 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7900030840 con 0x7f79041614d0 2024-03-06T07:29:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.602+0000 7f78fbfff640 1 -- 172.21.15.122:0/3559966742 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900030c10 con 0x7f79041614d0 2024-03-06T07:29:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 -- 172.21.15.122:0/3559966742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79041614d0 msgr2=0x7f79040d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 --2- 172.21.15.122:0/3559966742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79041614d0 0x7f79040d4860 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7900009870 tx=0x7f790002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 -- 172.21.15.122:0/3559966742 shutdown_connections 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 --2- 172.21.15.122:0/3559966742 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d1910 0x7f79040d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 --2- 172.21.15.122:0/3559966742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79041614d0 0x7f79040d4860 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 -- 172.21.15.122:0/3559966742 >> 172.21.15.122:0/3559966742 conn(0x7f79040d3d40 msgr2=0x7f79040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 -- 172.21.15.122:0/3559966742 shutdown_connections 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.603+0000 7f790b466640 1 -- 172.21.15.122:0/3559966742 wait complete. 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.604+0000 7f790b466640 1 Processor -- start 2024-03-06T07:29:50.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.604+0000 7f790b466640 1 -- start start 2024-03-06T07:29:50.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f790b466640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 0x7f7904151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f790b466640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d22b0 0x7f79040c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f790b466640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79041614d0 0x7f79040c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f790b466640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79040b0910 con 0x7f79040d1910 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f790b466640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79040cfc60 con 0x7f79041614d0 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f790b466640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79040b1e40 con 0x7f79040d22b0 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 0x7f7904151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 0x7f7904151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38954/0 (socket says 172.21.15.122:38954) 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 -- 172.21.15.122:0/1773170292 learned_addr learned my addr 172.21.15.122:0/1773170292 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7908a01640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d22b0 0x7f79040c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909a03640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79041614d0 0x7f79040c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 -- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d22b0 msgr2=0x7f79040c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d22b0 0x7f79040c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 -- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79041614d0 msgr2=0x7f79040c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79041614d0 0x7f79040c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.605+0000 7f7909202640 1 -- 172.21.15.122:0/1773170292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040ca220 con 0x7f79040d1910 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.606+0000 7f7909202640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 0x7f7904151220 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f7900000c00 tx=0x7f79000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.606+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900034040 con 0x7f79040d1910 2024-03-06T07:29:50.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.606+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7900039d90 con 0x7f79040d1910 2024-03-06T07:29:50.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.606+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79040ca4e0 con 0x7f79040d1910 2024-03-06T07:29:50.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.606+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900043610 con 0x7f79040d1910 2024-03-06T07:29:50.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.606+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79040caaa0 con 0x7f79040d1910 2024-03-06T07:29:50.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.607+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904068ff0 con 0x7f79040d1910 2024-03-06T07:29:50.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.607+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f790004a060 con 0x7f79040d1910 2024-03-06T07:29:50.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.608+0000 7f78f9ffb640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78e806b4d0 0x7f78e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.608+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f79000ac2f0 con 0x7f79040d1910 2024-03-06T07:29:50.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.608+0000 7f7908a01640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78e806b4d0 0x7f78e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.608+0000 7f7908a01640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78e806b4d0 0x7f78e806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f79040b6ec0 tx=0x7f78f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.612+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f790003e070 con 0x7f79040d1910 2024-03-06T07:29:50.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.783+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79040d0e70 con 0x7f79040d1910 2024-03-06T07:29:50.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.783+0000 7f78f9ffb640 1 -- 172.21.15.122:0/1773170292 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f790002d330 con 0x7f79040d1910 2024-03-06T07:29:50.785 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:29:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78e806b4d0 msgr2=0x7f78e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78e806b4d0 0x7f78e806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f79040b6ec0 tx=0x7f78f4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 msgr2=0x7f7904151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 0x7f7904151220 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f7900000c00 tx=0x7f79000046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 shutdown_connections 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78e806b4d0 0x7f78e806d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79041614d0 0x7f79040c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f79040d22b0 0x7f79040c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 --2- 172.21.15.122:0/1773170292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79040d1910 0x7f7904151220 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 >> 172.21.15.122:0/1773170292 conn(0x7f79040d3d40 msgr2=0x7f79040d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 shutdown_connections 2024-03-06T07:29:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.785+0000 7f790b466640 1 -- 172.21.15.122:0/1773170292 wait complete. 2024-03-06T07:29:50.798 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:50.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.900+0000 7f4289b56640 1 Processor -- start 2024-03-06T07:29:50.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.900+0000 7f4289b56640 1 -- start start 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f4289b56640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f4289b56640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f4289b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f4289b56640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4284158ac0 con 0x7f42841586e0 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f4289b56640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428415b2a0 con 0x7f42840aef80 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f4289b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428415b540 con 0x7f42840cffd0 2024-03-06T07:29:50.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38958/0 (socket says 172.21.15.122:38958) 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 -- 172.21.15.122:0/2870100329 learned_addr learned my addr 172.21.15.122:0/2870100329 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 -- 172.21.15.122:0/2870100329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840cffd0 msgr2=0x7f428406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 --2- 172.21.15.122:0/2870100329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 -- 172.21.15.122:0/2870100329 >> v2:172.21.15.169:3300/0 conn(0x7f42840aef80 msgr2=0x7f42840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 --2- 172.21.15.122:0/2870100329 >> v2:172.21.15.169:3300/0 conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.901+0000 7f42837fe640 1 -- 172.21.15.122:0/2870100329 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428415b820 con 0x7f42841586e0 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.902+0000 7f42837fe640 1 --2- 172.21.15.122:0/2870100329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42841586e0 0x7f42840aea40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4284072e30 tx=0x7f427402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a20064cf092ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.902+0000 7f42827fc640 1 -- 172.21.15.122:0/2870100329 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427403b070 con 0x7f42841586e0 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.902+0000 7f42827fc640 1 -- 172.21.15.122:0/2870100329 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4274036440 con 0x7f42841586e0 2024-03-06T07:29:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.902+0000 7f42827fc640 1 -- 172.21.15.122:0/2870100329 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274036810 con 0x7f42841586e0 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 -- 172.21.15.122:0/2870100329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42841586e0 msgr2=0x7f42840aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 --2- 172.21.15.122:0/2870100329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42841586e0 0x7f42840aea40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4284072e30 tx=0x7f427402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 -- 172.21.15.122:0/2870100329 shutdown_connections 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 --2- 172.21.15.122:0/2870100329 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42840cffd0 0x7f428406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 --2- 172.21.15.122:0/2870100329 >> v2:172.21.15.169:3300/0 conn(0x7f42840aef80 0x7f42840cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 --2- 172.21.15.122:0/2870100329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42841586e0 0x7f42840aea40 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 -- 172.21.15.122:0/2870100329 >> 172.21.15.122:0/2870100329 conn(0x7f428406b190 msgr2=0x7f428406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 -- 172.21.15.122:0/2870100329 shutdown_connections 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.903+0000 7f4289b56640 1 -- 172.21.15.122:0/2870100329 wait complete. 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.904+0000 7f4289b56640 1 Processor -- start 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.904+0000 7f4289b56640 1 -- start start 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4289b56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42840aef80 0x7f4284149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4289b56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428414f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4289b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42841586e0 0x7f428414db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4289b56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4284071350 con 0x7f42840cffd0 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4289b56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428415bde0 con 0x7f42840aef80 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4289b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42840714d0 con 0x7f42841586e0 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428414f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428414f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38960/0 (socket says 172.21.15.122:38960) 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 -- 172.21.15.122:0/3252272352 learned_addr learned my addr 172.21.15.122:0/3252272352 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4283fff640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42841586e0 0x7f428414db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f42837fe640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42840aef80 0x7f4284149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 -- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42841586e0 msgr2=0x7f428414db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42841586e0 0x7f428414db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 -- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42840aef80 msgr2=0x7f4284149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42840aef80 0x7f4284149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 -- 172.21.15.122:0/3252272352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428414fac0 con 0x7f42840cffd0 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4283fff640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42841586e0 0x7f428414db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f4282ffd640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428414f4b0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f426c00b410 tx=0x7f426c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c015020 con 0x7f42840cffd0 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c011850 con 0x7f42840cffd0 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.905+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c011c40 con 0x7f42840cffd0 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.906+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428414fda0 con 0x7f42840cffd0 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.906+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4284144f60 con 0x7f42840cffd0 2024-03-06T07:29:50.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.906+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42840b2630 con 0x7f42840cffd0 2024-03-06T07:29:50.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.907+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f426c018020 con 0x7f42840cffd0 2024-03-06T07:29:50.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.907+0000 7f427bfff640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426406b4d0 0x7f426406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.907+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f426c08a0d0 con 0x7f42840cffd0 2024-03-06T07:29:50.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.908+0000 7f42837fe640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426406b4d0 0x7f426406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.908+0000 7f42837fe640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426406b4d0 0x7f426406d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4274038000 tx=0x7f4274032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:50.912+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42840b2630 con 0x7f42840cffd0 2024-03-06T07:29:51.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.089+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4284142bf0 con 0x7f42840cffd0 2024-03-06T07:29:51.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.089+0000 7f427bfff640 1 -- 172.21.15.122:0/3252272352 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4284142bf0 con 0x7f42840cffd0 2024-03-06T07:29:51.091 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:29:51.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426406b4d0 msgr2=0x7f426406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426406b4d0 0x7f426406d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4274038000 tx=0x7f4274032000 comp rx=0 tx=0).stop 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 msgr2=0x7f428414f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428414f4b0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f426c00b410 tx=0x7f426c00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 shutdown_connections 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426406b4d0 0x7f426406d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42841586e0 0x7f428414db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42840cffd0 0x7f428414f4b0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 --2- 172.21.15.122:0/3252272352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42840aef80 0x7f4284149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 >> 172.21.15.122:0/3252272352 conn(0x7f428406b190 msgr2=0x7f42840722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 shutdown_connections 2024-03-06T07:29:51.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.091+0000 7f4289b56640 1 -- 172.21.15.122:0/3252272352 wait complete. 2024-03-06T07:29:51.104 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:51.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.204+0000 7f7034c9d640 1 Processor -- start 2024-03-06T07:29:51.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.204+0000 7f7034c9d640 1 -- start start 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f7034c9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f70301614d0 0x7f70300d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f7034c9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f70300d24b0 0x7f70300b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f7034c9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70301618b0 con 0x7f70300d1910 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f7034c9d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70300cf400 con 0x7f70300d24b0 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f7034c9d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70300cf6a0 con 0x7f70301614d0 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f70300d24b0 0x7f70300b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f70300d24b0 0x7f70300b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38966/0 (socket says 172.21.15.122:38966) 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 -- 172.21.15.122:0/3038849550 learned_addr learned my addr 172.21.15.122:0/3038849550 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702e59c640 1 --2- 172.21.15.122:0/3038849550 >> v2:172.21.15.169:3300/0 conn(0x7f70301614d0 0x7f70300d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 -- 172.21.15.122:0/3038849550 >> v2:172.21.15.169:3300/0 conn(0x7f70301614d0 msgr2=0x7f70300d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 --2- 172.21.15.122:0/3038849550 >> v2:172.21.15.169:3300/0 conn(0x7f70301614d0 0x7f70300d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 -- 172.21.15.122:0/3038849550 >> v1:172.21.15.169:6789/0 conn(0x7f70300d1910 legacy=0x7f70300d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.206+0000 7f702ed9d640 1 -- 172.21.15.122:0/3038849550 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70300cf980 con 0x7f70300d24b0 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.207+0000 7f702ed9d640 1 --2- 172.21.15.122:0/3038849550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d24b0 0x7f70300b0e30 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f703008c470 tx=0x7f702802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd84bc94f4f6b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.207+0000 7f702d59a640 1 -- 172.21.15.122:0/3038849550 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702802ca80 con 0x7f70300d24b0 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.207+0000 7f702d59a640 1 -- 172.21.15.122:0/3038849550 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702802cc20 con 0x7f70300d24b0 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.207+0000 7f702d59a640 1 -- 172.21.15.122:0/3038849550 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280343f0 con 0x7f70300d24b0 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 -- 172.21.15.122:0/3038849550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d24b0 msgr2=0x7f70300b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 --2- 172.21.15.122:0/3038849550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d24b0 0x7f70300b0e30 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f703008c470 tx=0x7f702802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 -- 172.21.15.122:0/3038849550 shutdown_connections 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 --2- 172.21.15.122:0/3038849550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d24b0 0x7f70300b0e30 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 --2- 172.21.15.122:0/3038849550 >> v2:172.21.15.169:3300/0 conn(0x7f70301614d0 0x7f70300d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 -- 172.21.15.122:0/3038849550 >> 172.21.15.122:0/3038849550 conn(0x7f70300d3d40 msgr2=0x7f70300d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 -- 172.21.15.122:0/3038849550 shutdown_connections 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.208+0000 7f7034c9d640 1 -- 172.21.15.122:0/3038849550 wait complete. 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.210+0000 7f7034c9d640 1 Processor -- start 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.210+0000 7f7034c9d640 1 -- start start 2024-03-06T07:29:51.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f7034c9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 0x7f70300c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f7034c9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70300d24b0 0x7f703014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f7034c9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70301614d0 0x7f70300ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f7034c9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70300b1370 con 0x7f70300d1910 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f7034c9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70300b0910 con 0x7f70301614d0 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f7034c9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70300b14f0 con 0x7f70300d24b0 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f702e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 0x7f70300c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f702e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 0x7f70300c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38976/0 (socket says 172.21.15.122:38976) 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f702e59c640 1 -- 172.21.15.122:0/2234981765 learned_addr learned my addr 172.21.15.122:0/2234981765 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f702dd9b640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70300d24b0 0x7f703014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.211+0000 7f702ed9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70301614d0 0x7f70300ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f702e59c640 1 -- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70300d24b0 msgr2=0x7f703014dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f702e59c640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70300d24b0 0x7f703014dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f702e59c640 1 -- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70301614d0 msgr2=0x7f70300ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f702e59c640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70301614d0 0x7f70300ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f702e59c640 1 -- 172.21.15.122:0/2234981765 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70300c9570 con 0x7f70300d1910 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f702e59c640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 0x7f70300c7870 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f702000c820 tx=0x7f702000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020015070 con 0x7f70300d1910 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7020004020 con 0x7f70300d1910 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70200043f0 con 0x7f70300d1910 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70300c9850 con 0x7f70300d1910 2024-03-06T07:29:51.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.212+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703014e210 con 0x7f70300d1910 2024-03-06T07:29:51.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.213+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70300ce170 con 0x7f70300d1910 2024-03-06T07:29:51.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.214+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7020007830 con 0x7f70300d1910 2024-03-06T07:29:51.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.214+0000 7f701effd640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f701006b4d0 0x7f701006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.214+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7020089f20 con 0x7f70300d1910 2024-03-06T07:29:51.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.214+0000 7f702dd9b640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f701006b4d0 0x7f701006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.214+0000 7f702dd9b640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f701006b4d0 0x7f701006d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f70301f73a0 tx=0x7f7024009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.222+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70200587a0 con 0x7f70300d1910 2024-03-06T07:29:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.393+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7030068ff0 con 0x7f70300d1910 2024-03-06T07:29:51.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.393+0000 7f701effd640 1 -- 172.21.15.122:0/2234981765 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7f702005c840 con 0x7f70300d1910 2024-03-06T07:29:51.396 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:29:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f701006b4d0 msgr2=0x7f701006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f701006b4d0 0x7f701006d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f70301f73a0 tx=0x7f7024009040 comp rx=0 tx=0).stop 2024-03-06T07:29:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 msgr2=0x7f70300c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 0x7f70300c7870 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f702000c820 tx=0x7f702000cce0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 shutdown_connections 2024-03-06T07:29:51.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f701006b4d0 0x7f701006d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70301614d0 0x7f70300ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70300d24b0 0x7f703014dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 --2- 172.21.15.122:0/2234981765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70300d1910 0x7f70300c7870 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 >> 172.21.15.122:0/2234981765 conn(0x7f70300d3d40 msgr2=0x7f70300d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 shutdown_connections 2024-03-06T07:29:51.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.396+0000 7f7034c9d640 1 -- 172.21.15.122:0/2234981765 wait complete. 2024-03-06T07:29:51.409 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:51.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.510+0000 7f161ec88640 1 Processor -- start 2024-03-06T07:29:51.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.510+0000 7f161ec88640 1 -- start start 2024-03-06T07:29:51.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f161ec88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16180d0d90 0x7f16180ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f161ec88640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16180d25e0 con 0x7f161815e000 2024-03-06T07:29:51.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f161ec88640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16180d2880 con 0x7f16180c90f0 2024-03-06T07:29:51.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f161ec88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16180c7c50 con 0x7f16180d0d90 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f1617fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16180d0d90 0x7f16180ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f1617fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f16180d0d90 0x7f16180ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38980/0 (socket says 172.21.15.122:38980) 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f1617fff640 1 -- 172.21.15.122:0/440564377 learned_addr learned my addr 172.21.15.122:0/440564377 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f161ca24640 1 --1- 172.21.15.122:0/440564377 >> v1:172.21.15.122:6789/0 conn(0x7f161815e000 0x7f16180d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41702/0 (socket says 172.21.15.122:41702) 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f1617fff640 1 -- 172.21.15.122:0/440564377 >> v1:172.21.15.169:6789/0 conn(0x7f16180c90f0 legacy=0x7f16180ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f1617fff640 1 -- 172.21.15.122:0/440564377 >> v1:172.21.15.122:6789/0 conn(0x7f161815e000 legacy=0x7f16180d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f1617fff640 1 -- 172.21.15.122:0/440564377 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180d2b40 con 0x7f16180d0d90 2024-03-06T07:29:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.511+0000 7f16177fe640 1 -- 172.21.15.122:0/440564377 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223747067 0 0) 0x7f16180d25e0 con 0x7f161815e000 2024-03-06T07:29:51.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.512+0000 7f1617fff640 1 --2- 172.21.15.122:0/440564377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d0d90 0x7f16180ce7c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f1604009870 tx=0x7f160402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d65960d7b26a835 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.512+0000 7f16177fe640 1 -- 172.21.15.122:0/440564377 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160402f840 con 0x7f16180d0d90 2024-03-06T07:29:51.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.512+0000 7f16177fe640 1 -- 172.21.15.122:0/440564377 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160402f9e0 con 0x7f16180d0d90 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.512+0000 7f16177fe640 1 -- 172.21.15.122:0/440564377 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160402fdb0 con 0x7f16180d0d90 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 -- 172.21.15.122:0/440564377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d0d90 msgr2=0x7f16180ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 --2- 172.21.15.122:0/440564377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d0d90 0x7f16180ce7c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f1604009870 tx=0x7f160402ba20 comp rx=0 tx=0).stop 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 -- 172.21.15.122:0/440564377 shutdown_connections 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 --2- 172.21.15.122:0/440564377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16180d0d90 0x7f16180ce7c0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 -- 172.21.15.122:0/440564377 >> 172.21.15.122:0/440564377 conn(0x7f16180d3cf0 msgr2=0x7f16180d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 -- 172.21.15.122:0/440564377 shutdown_connections 2024-03-06T07:29:51.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.513+0000 7f161ec88640 1 -- 172.21.15.122:0/440564377 wait complete. 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.514+0000 7f161ec88640 1 Processor -- start 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.514+0000 7f161ec88640 1 -- start start 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180c90f0 0x7f1618157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d0d90 0x7f16180b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 0x7f16180b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161815e3e0 con 0x7f161815e000 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16180d22d0 con 0x7f16180c90f0 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16180d4810 con 0x7f16180d0d90 2024-03-06T07:29:51.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 0x7f16180b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 0x7f16180b8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38994/0 (socket says 172.21.15.122:38994) 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 -- 172.21.15.122:0/237359960 learned_addr learned my addr 172.21.15.122:0/237359960 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 -- 172.21.15.122:0/237359960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d0d90 msgr2=0x7f16180b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d0d90 0x7f16180b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 -- 172.21.15.122:0/237359960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180c90f0 msgr2=0x7f1618157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180c90f0 0x7f1618157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 -- 172.21.15.122:0/237359960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618157890 con 0x7f161815e000 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161d225640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 0x7f16180b8a20 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f161000c7b0 tx=0x7f161000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610017070 con 0x7f161815e000 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1610003da0 con 0x7f161815e000 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.515+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16180a09e0 con 0x7f161815e000 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.516+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161000ce60 con 0x7f161815e000 2024-03-06T07:29:51.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.516+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16180a0fa0 con 0x7f161815e000 2024-03-06T07:29:51.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.517+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16180d1820 con 0x7f161815e000 2024-03-06T07:29:51.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.517+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f161001a070 con 0x7f161815e000 2024-03-06T07:29:51.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.517+0000 7f16157fa640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f15fc06b500 0x7f15fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.518+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1610089840 con 0x7f161815e000 2024-03-06T07:29:51.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.518+0000 7f161ca24640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f15fc06b500 0x7f15fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.518+0000 7f161ca24640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f15fc06b500 0x7f15fc06d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f160c007fd0 tx=0x7f160c007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.522+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1610058680 con 0x7f161815e000 2024-03-06T07:29:51.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.712+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f161804f330 con 0x7f161815e000 2024-03-06T07:29:51.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.713+0000 7f16157fa640 1 -- 172.21.15.122:0/237359960 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v130) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1610057f80 con 0x7f161815e000 2024-03-06T07:29:51.715 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f15fc06b500 msgr2=0x7f15fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f15fc06b500 0x7f15fc06d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f160c007fd0 tx=0x7f160c007960 comp rx=0 tx=0).stop 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 msgr2=0x7f16180b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 0x7f16180b8a20 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f161000c7b0 tx=0x7f161000cc70 comp rx=0 tx=0).stop 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 shutdown_connections 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f15fc06b500 0x7f15fc06d9c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f161815e000 0x7f16180b8a20 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f16180d0d90 0x7f16180b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 --2- 172.21.15.122:0/237359960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16180c90f0 0x7f1618157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 >> 172.21.15.122:0/237359960 conn(0x7f16180d3cf0 msgr2=0x7f16180b6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 shutdown_connections 2024-03-06T07:29:51.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:51.715+0000 7f161ec88640 1 -- 172.21.15.122:0/237359960 wait complete. 2024-03-06T07:29:51.728 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:29:51.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.728 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:51.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[78847]: fuse finished with error 0 and tester_r 0 2024-03-06T07:29:51.760 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:29:51.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.760 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:51.927 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:29:51.927 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.945 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:51.946 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:29:51.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.946 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:29:52.001 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:29:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:52.003 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:29:52.003 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:52.018 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:52.018 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:29:52.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:52.019 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:29:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:52.073 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:29:52.074 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:29:52.074 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:52.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:52.092 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:29:52.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:52.093 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:29:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:52.149 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:29:52.149 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:52.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.248+0000 7fd9066f4640 1 Processor -- start 2024-03-06T07:29:52.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.248+0000 7fd9066f4640 1 -- start start 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd9066f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd9000cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd9066f4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd90015b430 0x7fd900071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd9066f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90015b8b0 con 0x7fd9000cd7a0 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd9066f4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd900070a70 con 0x7fd90015b430 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd9066f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd900070d50 con 0x7fd9001515e0 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd8ff7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd90015b430 0x7fd900071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd904c91640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd9000cd7a0 0x7fd9000cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41712/0 (socket says 172.21.15.122:41712) 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd904c91640 1 -- 172.21.15.122:0/819978824 learned_addr learned my addr 172.21.15.122:0/819978824 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd8fffff640 1 --2- 172.21.15.122:0/819978824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd9000cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd8ff7fe640 1 -- 172.21.15.122:0/819978824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 msgr2=0x7fd9000cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd8ff7fe640 1 --2- 172.21.15.122:0/819978824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd9000cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.249+0000 7fd8ff7fe640 1 -- 172.21.15.122:0/819978824 >> v1:172.21.15.122:6789/0 conn(0x7fd9000cd7a0 legacy=0x7fd9000cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.250+0000 7fd8ff7fe640 1 -- 172.21.15.122:0/819978824 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900071030 con 0x7fd90015b430 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.250+0000 7fd8ff7fe640 1 --2- 172.21.15.122:0/819978824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd900071830 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00097a0 tx=0x7fd8f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6fd84efa052cb37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.250+0000 7fd8feffd640 1 -- 172.21.15.122:0/819978824 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0035020 con 0x7fd90015b430 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.250+0000 7fd8feffd640 1 -- 172.21.15.122:0/819978824 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0031830 con 0x7fd90015b430 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.250+0000 7fd8feffd640 1 -- 172.21.15.122:0/819978824 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0031c00 con 0x7fd90015b430 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 -- 172.21.15.122:0/819978824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 msgr2=0x7fd900071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 --2- 172.21.15.122:0/819978824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd900071830 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00097a0 tx=0x7fd8f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 -- 172.21.15.122:0/819978824 shutdown_connections 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 --2- 172.21.15.122:0/819978824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd900071830 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 --2- 172.21.15.122:0/819978824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd9000cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 -- 172.21.15.122:0/819978824 >> 172.21.15.122:0/819978824 conn(0x7fd9000cf8a0 msgr2=0x7fd9000cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 -- 172.21.15.122:0/819978824 shutdown_connections 2024-03-06T07:29:52.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.251+0000 7fd9066f4640 1 -- 172.21.15.122:0/819978824 wait complete. 2024-03-06T07:29:52.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.252+0000 7fd9066f4640 1 Processor -- start 2024-03-06T07:29:52.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.252+0000 7fd9066f4640 1 -- start start 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd9066f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9000cd7a0 0x7fd9000c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd9066f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd900167220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd9066f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd9001485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd9066f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd900071ef0 con 0x7fd90015b430 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd9066f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd900071d70 con 0x7fd9000cd7a0 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd9066f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000cc3a0 con 0x7fd9001515e0 2024-03-06T07:29:52.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd9001485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd9001485c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39016/0 (socket says 172.21.15.122:39016) 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 -- 172.21.15.122:0/3858637210 learned_addr learned my addr 172.21.15.122:0/3858637210 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd8ff7fe640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd900167220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd8fffff640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9000cd7a0 0x7fd9000c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 -- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 msgr2=0x7fd900167220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd900167220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 -- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9000cd7a0 msgr2=0x7fd9000c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9000cd7a0 0x7fd9000c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd904c91640 1 -- 172.21.15.122:0/3858637210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900167920 con 0x7fd90015b430 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.253+0000 7fd8fffff640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9000cd7a0 0x7fd9000c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.254+0000 7fd904c91640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd9001485c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800cbc0 tx=0x7fd8f800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.254+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8017070 con 0x7fd90015b430 2024-03-06T07:29:52.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.254+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f800ad40 con 0x7fd90015b430 2024-03-06T07:29:52.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.254+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd900167c00 con 0x7fd90015b430 2024-03-06T07:29:52.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.254+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8012600 con 0x7fd90015b430 2024-03-06T07:29:52.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.254+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd900148b00 con 0x7fd90015b430 2024-03-06T07:29:52.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.255+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd900068840 con 0x7fd90015b430 2024-03-06T07:29:52.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.256+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8f801a070 con 0x7fd90015b430 2024-03-06T07:29:52.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.256+0000 7fd8fcff9640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8e006b500 0x7fd8e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.256+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd8f8089e10 con 0x7fd90015b430 2024-03-06T07:29:52.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.256+0000 7fd8fffff640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8e006b500 0x7fd8e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.257+0000 7fd8fffff640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8e006b500 0x7fd8e006d9c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8006fd0 tx=0x7fd8e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.261+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8f8058690 con 0x7fd90015b430 2024-03-06T07:29:52.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.470+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd900168120 con 0x7fd90015b430 2024-03-06T07:29:52.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.470+0000 7fd8fcff9640 1 -- 172.21.15.122:0/3858637210 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3402 (secure 0 0 0) 0x7fd8f805c730 con 0x7fd90015b430 2024-03-06T07:29:52.472 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:52.473 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/696408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":696408149},{"type":"v1","addr":"172.21.15.122:6836","nonce":696408149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:47.105721+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:52.473 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 83 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8e006b500 msgr2=0x7fd8e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8e006b500 0x7fd8e006d9c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8006fd0 tx=0x7fd8e8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 msgr2=0x7fd9001485c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd9001485c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800cbc0 tx=0x7fd8f800a430 comp rx=0 tx=0).stop 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 shutdown_connections 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd8e006b500 0x7fd8e006d9c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90015b430 0x7fd9001485c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd9001515e0 0x7fd900167220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 --2- 172.21.15.122:0/3858637210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9000cd7a0 0x7fd9000c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 >> 172.21.15.122:0/3858637210 conn(0x7fd9000cf8a0 msgr2=0x7fd90014ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.472+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 shutdown_connections 2024-03-06T07:29:52.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.473+0000 7fd9066f4640 1 -- 172.21.15.122:0/3858637210 wait complete. 2024-03-06T07:29:52.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.588+0000 7fdf390d4640 1 Processor -- start 2024-03-06T07:29:52.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf390d4640 1 -- start start 2024-03-06T07:29:52.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf390d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf390d4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdf340d29c0 0x7fdf340b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf390d4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf340b3430 con 0x7fdf340cf830 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf390d4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf340b31e0 con 0x7fdf340d29c0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf390d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340b3d60 con 0x7fdf3415dfe0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf3359e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdf340d29c0 0x7fdf340b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf2bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdf340cf830 0x7fdf340d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41724/0 (socket says 172.21.15.122:41724) 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.589+0000 7fdf2bfff640 1 -- 172.21.15.122:0/3644957918 learned_addr learned my addr 172.21.15.122:0/3644957918 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf32d9d640 1 --2- 172.21.15.122:0/3644957918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf3359e640 1 -- 172.21.15.122:0/3644957918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 msgr2=0x7fdf340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf3359e640 1 --2- 172.21.15.122:0/3644957918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf3359e640 1 -- 172.21.15.122:0/3644957918 >> v1:172.21.15.122:6789/0 conn(0x7fdf340cf830 legacy=0x7fdf340d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf3259c640 1 -- 172.21.15.122:0/3644957918 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597171735 0 0) 0x7fdf340b3430 con 0x7fdf340cf830 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf3359e640 1 -- 172.21.15.122:0/3644957918 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340b3710 con 0x7fdf340d29c0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.590+0000 7fdf3359e640 1 --2- 172.21.15.122:0/3644957918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340b41f0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009d90 tx=0x7fdf2c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ed4bda9a6965d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf3259c640 1 -- 172.21.15.122:0/3644957918 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c039070 con 0x7fdf340d29c0 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf3259c640 1 -- 172.21.15.122:0/3644957918 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c02cc50 con 0x7fdf340d29c0 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf3259c640 1 -- 172.21.15.122:0/3644957918 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c034440 con 0x7fdf340d29c0 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 -- 172.21.15.122:0/3644957918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 msgr2=0x7fdf340b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 --2- 172.21.15.122:0/3644957918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340b41f0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009d90 tx=0x7fdf2c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 -- 172.21.15.122:0/3644957918 shutdown_connections 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 --2- 172.21.15.122:0/3644957918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340b41f0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 --2- 172.21.15.122:0/3644957918 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 -- 172.21.15.122:0/3644957918 >> 172.21.15.122:0/3644957918 conn(0x7fdf340d3cd0 msgr2=0x7fdf340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 -- 172.21.15.122:0/3644957918 shutdown_connections 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.591+0000 7fdf390d4640 1 -- 172.21.15.122:0/3644957918 wait complete. 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.592+0000 7fdf390d4640 1 Processor -- start 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.592+0000 7fdf390d4640 1 -- start start 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf390d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf340cf830 0x7fdf3415c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf390d4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf390d4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf390d4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340d1df0 con 0x7fdf340d29c0 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf390d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340d1c70 con 0x7fdf340cf830 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf390d4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3415e3c0 con 0x7fdf3415dfe0 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf2bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf2bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39022/0 (socket says 172.21.15.122:39022) 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf2bfff640 1 -- 172.21.15.122:0/417329377 learned_addr learned my addr 172.21.15.122:0/417329377 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf32d9d640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf340cf830 0x7fdf3415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.593+0000 7fdf3359e640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2bfff640 1 -- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 msgr2=0x7fdf340cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2bfff640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2bfff640 1 -- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf340cf830 msgr2=0x7fdf3415c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2bfff640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf340cf830 0x7fdf3415c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2bfff640 1 -- 172.21.15.122:0/417329377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340b74d0 con 0x7fdf340d29c0 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf3359e640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2bfff640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340cc550 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00c4b0 tx=0x7fdf1c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c018070 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c007500 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf340b77b0 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c004a20 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.594+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf340b7d70 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.595+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf340b8250 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.596+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf1c01b070 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.596+0000 7fdf2b7fe640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf1406b4d0 0x7fdf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.596+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf1c089500 con 0x7fdf340d29c0 2024-03-06T07:29:52.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.596+0000 7fdf32d9d640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf1406b4d0 0x7fdf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.597+0000 7fdf32d9d640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf1406b4d0 0x7fdf1406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009aa0 tx=0x7fdf24009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.601+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf340b8250 con 0x7fdf340d29c0 2024-03-06T07:29:52.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.848+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf340cd0f0 con 0x7fdf340d29c0 2024-03-06T07:29:52.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.849+0000 7fdf2b7fe640 1 -- 172.21.15.122:0/417329377 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16719 (secure 0 0 0) 0x7fdf1c057da0 con 0x7fdf340d29c0 2024-03-06T07:29:52.851 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:52.851 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":130,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:29:35.994874+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf1406b4d0 msgr2=0x7fdf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf1406b4d0 0x7fdf1406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009aa0 tx=0x7fdf24009190 comp rx=0 tx=0).stop 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 msgr2=0x7fdf340cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340cc550 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00c4b0 tx=0x7fdf1c00c970 comp rx=0 tx=0).stop 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 shutdown_connections 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdf1406b4d0 0x7fdf1406d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdf3415dfe0 0x7fdf340cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf340d29c0 0x7fdf340cc550 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 --2- 172.21.15.122:0/417329377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf340cf830 0x7fdf3415c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 >> 172.21.15.122:0/417329377 conn(0x7fdf340d3cd0 msgr2=0x7fdf340cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 shutdown_connections 2024-03-06T07:29:52.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.851+0000 7fdf390d4640 1 -- 172.21.15.122:0/417329377 wait complete. 2024-03-06T07:29:52.864 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:39.745958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:40.497682+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:52.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:52.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.966+0000 7f42a26f4640 1 Processor -- start 2024-03-06T07:29:52.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.966+0000 7f42a26f4640 1 -- start start 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.967+0000 7f42a26f4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f429c0d4860 0x7f429c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.967+0000 7f42a26f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429c1618b0 con 0x7f429c0b0e30 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.967+0000 7f42a26f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429c0cf400 con 0x7f429c1614d0 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.967+0000 7f42a26f4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429c0d4cf0 con 0x7f429c0d4860 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f42a0c91640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f429c0b0e30 0x7f429c0b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41736/0 (socket says 172.21.15.122:41736) 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f42a0c91640 1 -- 172.21.15.122:0/2768632146 learned_addr learned my addr 172.21.15.122:0/2768632146 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807284193 0 0) 0x7f429c1618b0 con 0x7f429c0b0e30 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4288003520 con 0x7f429c0b0e30 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f42937fe640 1 --2- 172.21.15.122:0/2768632146 >> v2:172.21.15.169:3300/0 conn(0x7f429c0d4860 0x7f429c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395089841 0 0) 0x7f4288003520 con 0x7f429c0b0e30 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 >> v2:172.21.15.169:3300/0 conn(0x7f429c0d4860 msgr2=0x7f429c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 --2- 172.21.15.122:0/2768632146 >> v2:172.21.15.169:3300/0 conn(0x7f429c0d4860 0x7f429c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 >> v1:172.21.15.169:6789/0 conn(0x7f429c1614d0 legacy=0x7f429c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0cf6a0 con 0x7f429c0b0e30 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4294001a10 con 0x7f429c0b0e30 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 >> v1:172.21.15.122:6789/0 conn(0x7f429c0b0e30 legacy=0x7f429c0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42880039b0 0x7f4288003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4288004800 0x7f4288004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42880051c0 0x7f428800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0d4cf0 con 0x7f4288004800 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428800c0f0 con 0x7f42880039b0 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428800c2f0 con 0x7f42880051c0 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f42937fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4288004800 0x7f4288004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f429bfff640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42880039b0 0x7f4288003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.968+0000 7f42a0c91640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42880051c0 0x7f428800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.969+0000 7f42937fe640 1 -- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42880051c0 msgr2=0x7f428800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.969+0000 7f42937fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42880051c0 0x7f428800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.969+0000 7f42937fe640 1 -- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42880039b0 msgr2=0x7f4288003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.969+0000 7f42937fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42880039b0 0x7f4288003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.969+0000 7f42a0c91640 1 -- 172.21.15.122:0/2768632146 reap_dead start 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.969+0000 7f42937fe640 1 -- 172.21.15.122:0/2768632146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4288003520 con 0x7f4288004800 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42937fe640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4288004800 0x7f4288004c80 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f428400ba00 tx=0x7f4288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2c717896eb2f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284036020 con 0x7f4288004800 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42840325f0 con 0x7f4288004800 2024-03-06T07:29:52.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f429b7fe640 1 -- 172.21.15.122:0/2768632146 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428400b600 con 0x7f4288004800 2024-03-06T07:29:52.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 -- 172.21.15.122:0/2768632146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4288004800 msgr2=0x7f4288004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4288004800 0x7f4288004c80 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f428400ba00 tx=0x7f4288005830 comp rx=0 tx=0).stop 2024-03-06T07:29:52.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 -- 172.21.15.122:0/2768632146 shutdown_connections 2024-03-06T07:29:52.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 --2- 172.21.15.122:0/2768632146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4288004800 0x7f4288004c80 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 -- 172.21.15.122:0/2768632146 >> 172.21.15.122:0/2768632146 conn(0x7f429c0d3d40 msgr2=0x7f429c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 -- 172.21.15.122:0/2768632146 shutdown_connections 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.970+0000 7f42a26f4640 1 -- 172.21.15.122:0/2768632146 wait complete. 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 Processor -- start 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 -- start start 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4288004800 0x7f429c159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f429c0c9050 0x7f429c0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 0x7f429c0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0b0910 con 0x7f429c0cc730 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0cfc60 con 0x7f4288004800 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.972+0000 7f42a26f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0d1910 con 0x7f429c0c9050 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 0x7f429c0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 0x7f429c0cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39052/0 (socket says 172.21.15.122:39052) 2024-03-06T07:29:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 -- 172.21.15.122:0/3496470864 learned_addr learned my addr 172.21.15.122:0/3496470864 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f429bfff640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4288004800 0x7f429c159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42937fe640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f429c0c9050 0x7f429c0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 -- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f429c0c9050 msgr2=0x7f429c0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f429c0c9050 0x7f429c0cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 -- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4288004800 msgr2=0x7f429c159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4288004800 0x7f429c159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 -- 172.21.15.122:0/3496470864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0b6e10 con 0x7f429c0cc730 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a0c91640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 0x7f429c0cd580 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f429c0b6900 tx=0x7f4294007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429400ab90 con 0x7f429c0cc730 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429400ad30 con 0x7f429c0cc730 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429c0b70f0 con 0x7f429c0cc730 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294013610 con 0x7f429c0cc730 2024-03-06T07:29:52.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.973+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429c0b76b0 con 0x7f429c0cc730 2024-03-06T07:29:52.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.975+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c161010 con 0x7f429c0cc730 2024-03-06T07:29:52.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.975+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42940137b0 con 0x7f429c0cc730 2024-03-06T07:29:52.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.975+0000 7f42997fa640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427c06b500 0x7f427c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.975+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f429408a550 con 0x7f429c0cc730 2024-03-06T07:29:52.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.975+0000 7f429bfff640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427c06b500 0x7f427c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.976+0000 7f429bfff640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427c06b500 0x7f427c06d9c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f429c0686a0 tx=0x7f428c007cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:52.983+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4294058dd0 con 0x7f429c0cc730 2024-03-06T07:29:53.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.156+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f429c0c71c0 con 0x7f429c0cc730 2024-03-06T07:29:53.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.157+0000 7f42997fa640 1 -- 172.21.15.122:0/3496470864 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f429405ce70 con 0x7f429c0cc730 2024-03-06T07:29:53.159 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427c06b500 msgr2=0x7f427c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427c06b500 0x7f427c06d9c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f429c0686a0 tx=0x7f428c007cc0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 msgr2=0x7f429c0cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 0x7f429c0cd580 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f429c0b6900 tx=0x7f4294007b20 comp rx=0 tx=0).stop 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 shutdown_connections 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f427c06b500 0x7f427c06d9c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f429c0cc730 0x7f429c0cd580 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f429c0c9050 0x7f429c0cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 --2- 172.21.15.122:0/3496470864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4288004800 0x7f429c159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 >> 172.21.15.122:0/3496470864 conn(0x7f429c0d3d40 msgr2=0x7f429c0b1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 shutdown_connections 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.159+0000 7f42a26f4640 1 -- 172.21.15.122:0/3496470864 wait complete. 2024-03-06T07:29:53.172 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:53.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f0364081640 1 Processor -- start 2024-03-06T07:29:53.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f0364081640 1 -- start start 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f0364081640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f035c1614d0 0x7f035c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f0364081640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c1618b0 con 0x7f035c0d24b0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f0364081640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0cf400 con 0x7f035c0d1910 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f0364081640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f035c0cf6a0 con 0x7f035c1614d0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f036261e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f035c0d24b0 0x7f035c0b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41740/0 (socket says 172.21.15.122:41740) 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.276+0000 7f036261e640 1 -- 172.21.15.122:0/452609438 learned_addr learned my addr 172.21.15.122:0/452609438 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 --2- 172.21.15.122:0/452609438 >> v2:172.21.15.169:3300/0 conn(0x7f035c1614d0 0x7f035c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684886910 0 0) 0x7f035c1618b0 con 0x7f035c0d24b0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0348003520 con 0x7f035c0d24b0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269211974 0 0) 0x7f0348003520 con 0x7f035c0d24b0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 >> v2:172.21.15.169:3300/0 conn(0x7f035c1614d0 msgr2=0x7f035c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 --2- 172.21.15.122:0/452609438 >> v2:172.21.15.169:3300/0 conn(0x7f035c1614d0 0x7f035c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 >> v1:172.21.15.169:6789/0 conn(0x7f035c0d1910 legacy=0x7f035c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0cf980 con 0x7f035c0d24b0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f03540019f0 con 0x7f035c0d24b0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 >> v1:172.21.15.122:6789/0 conn(0x7f035c0d24b0 legacy=0x7f035c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f0348003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0348004800 0x7f0348004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03480051c0 0x7f034800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0cf6a0 con 0x7f03480039b0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034800c060 con 0x7f0348004800 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034800c260 con 0x7f03480051c0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f0348003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f036161c640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0348004800 0x7f0348004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f036261e640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03480051c0 0x7f034800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 -- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03480051c0 msgr2=0x7f034800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f03480051c0 0x7f034800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 -- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0348004800 msgr2=0x7f0348004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0348004800 0x7f0348004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f0361e1d640 1 -- 172.21.15.122:0/452609438 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0348003520 con 0x7f03480039b0 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.277+0000 7f036261e640 1 -- 172.21.15.122:0/452609438 reap_dead start 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.278+0000 7f0361e1d640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f0348003dd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f034c009ac0 tx=0x7f0348005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed985f05a6d8990 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.278+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c039070 con 0x7f03480039b0 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.278+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c02c6d0 con 0x7f03480039b0 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.278+0000 7f0360e1b640 1 -- 172.21.15.122:0/452609438 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c009690 con 0x7f03480039b0 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 -- 172.21.15.122:0/452609438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 msgr2=0x7f0348003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f0348003dd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f034c009ac0 tx=0x7f0348005830 comp rx=0 tx=0).stop 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 -- 172.21.15.122:0/452609438 shutdown_connections 2024-03-06T07:29:53.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 --2- 172.21.15.122:0/452609438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f0348003dd0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 -- 172.21.15.122:0/452609438 >> 172.21.15.122:0/452609438 conn(0x7f035c0d3d40 msgr2=0x7f035c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 -- 172.21.15.122:0/452609438 shutdown_connections 2024-03-06T07:29:53.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.279+0000 7f0364081640 1 -- 172.21.15.122:0/452609438 wait complete. 2024-03-06T07:29:53.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.280+0000 7f0364081640 1 Processor -- start 2024-03-06T07:29:53.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.280+0000 7f0364081640 1 -- start start 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0364081640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f035c0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0364081640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f035c0b80f0 0x7f035c0b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0364081640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f035c151f90 0x7f035c0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0364081640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0d4860 con 0x7f03480039b0 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0364081640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0b0910 con 0x7f035c0b80f0 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0364081640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0d49e0 con 0x7f035c151f90 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f035c0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f035c0ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39076/0 (socket says 172.21.15.122:39076) 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 -- 172.21.15.122:0/3591558374 learned_addr learned my addr 172.21.15.122:0/3591558374 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f036261e640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f035c151f90 0x7f035c0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f036161c640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f035c0b80f0 0x7f035c0b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 -- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f035c151f90 msgr2=0x7f035c0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f035c151f90 0x7f035c0b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 -- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f035c0b80f0 msgr2=0x7f035c0b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f035c0b80f0 0x7f035c0b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 -- 172.21.15.122:0/3591558374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0b9c20 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f036261e640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f035c151f90 0x7f035c0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f036161c640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f035c0b80f0 0x7f035c0b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.281+0000 7f0361e1d640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f035c0ca510 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f034c037040 tx=0x7f034c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.282+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c039050 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.282+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c045070 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.282+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c0b8640 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.282+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c003a70 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.282+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035c0b8c00 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.283+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c06a1b0 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.283+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f034c003c10 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.284+0000 7f03527fc640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f034006b4d0 0x7f034006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.284+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f034c0abd20 con 0x7f03480039b0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.284+0000 7f036161c640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f034006b4d0 0x7f034006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.284+0000 7f036161c640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f034006b4d0 0x7f034006d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0358000c00 tx=0x7f0358009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.288+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f035c06a1b0 con 0x7f03480039b0 2024-03-06T07:29:53.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.464+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f035c0b1d60 con 0x7f03480039b0 2024-03-06T07:29:53.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.465+0000 7f03527fc640 1 -- 172.21.15.122:0/3591558374 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+11 (secure 0 0 0) 0x7f034c04e020 con 0x7f03480039b0 2024-03-06T07:29:53.467 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:29:53.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f034006b4d0 msgr2=0x7f034006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f034006b4d0 0x7f034006d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0358000c00 tx=0x7f0358009040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 msgr2=0x7f035c0ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f035c0ca510 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f034c037040 tx=0x7f034c004730 comp rx=0 tx=0).stop 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 shutdown_connections 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f034006b4d0 0x7f034006d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f035c151f90 0x7f035c0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f035c0b80f0 0x7f035c0b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 --2- 172.21.15.122:0/3591558374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03480039b0 0x7f035c0ca510 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 >> 172.21.15.122:0/3591558374 conn(0x7f035c0d3d40 msgr2=0x7f035c06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 shutdown_connections 2024-03-06T07:29:53.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.467+0000 7f0364081640 1 -- 172.21.15.122:0/3591558374 wait complete. 2024-03-06T07:29:53.480 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.582+0000 7efd36c5c640 1 Processor -- start 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.582+0000 7efd36c5c640 1 -- start start 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd36c5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd36c5c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd300d2480 0x7efd300b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd36c5c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd301618b0 con 0x7efd300d1910 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd36c5c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd300cf400 con 0x7efd300d2480 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd36c5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd300cf6a0 con 0x7efd301614d0 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd351f9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efd300d2480 0x7efd300b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd2ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efd300d1910 0x7efd300d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41746/0 (socket says 172.21.15.122:41746) 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd2ffff640 1 -- 172.21.15.122:0/905593905 learned_addr learned my addr 172.21.15.122:0/905593905 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd349f8640 1 --2- 172.21.15.122:0/905593905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd351f9640 1 -- 172.21.15.122:0/905593905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 msgr2=0x7efd300d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd351f9640 1 --2- 172.21.15.122:0/905593905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd351f9640 1 -- 172.21.15.122:0/905593905 >> v1:172.21.15.122:6789/0 conn(0x7efd300d1910 legacy=0x7efd300d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.583+0000 7efd351f9640 1 -- 172.21.15.122:0/905593905 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd300cf980 con 0x7efd300d2480 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.584+0000 7efd351f9640 1 --2- 172.21.15.122:0/905593905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d2480 0x7efd300b0e30 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7efd28009d90 tx=0x7efd2802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dba6641ba3617b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.584+0000 7efd2f7fe640 1 -- 172.21.15.122:0/905593905 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28039070 con 0x7efd300d2480 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.584+0000 7efd2f7fe640 1 -- 172.21.15.122:0/905593905 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2802cc50 con 0x7efd300d2480 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.584+0000 7efd2f7fe640 1 -- 172.21.15.122:0/905593905 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28034440 con 0x7efd300d2480 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 -- 172.21.15.122:0/905593905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d2480 msgr2=0x7efd300b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 --2- 172.21.15.122:0/905593905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d2480 0x7efd300b0e30 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7efd28009d90 tx=0x7efd2802bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 -- 172.21.15.122:0/905593905 shutdown_connections 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 --2- 172.21.15.122:0/905593905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d2480 0x7efd300b0e30 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 --2- 172.21.15.122:0/905593905 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 -- 172.21.15.122:0/905593905 >> 172.21.15.122:0/905593905 conn(0x7efd300d3d40 msgr2=0x7efd300d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 -- 172.21.15.122:0/905593905 shutdown_connections 2024-03-06T07:29:53.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.585+0000 7efd36c5c640 1 -- 172.21.15.122:0/905593905 wait complete. 2024-03-06T07:29:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.586+0000 7efd36c5c640 1 Processor -- start 2024-03-06T07:29:53.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.586+0000 7efd36c5c640 1 -- start start 2024-03-06T07:29:53.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 0x7efd3015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd300d2480 0x7efd301f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300b1370 con 0x7efd300d1910 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300b0910 con 0x7efd300d2480 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd300b14f0 con 0x7efd301614d0 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 0x7efd3015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 0x7efd3015f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39098/0 (socket says 172.21.15.122:39098) 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 -- 172.21.15.122:0/67223723 learned_addr learned my addr 172.21.15.122:0/67223723 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 -- 172.21.15.122:0/67223723 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 msgr2=0x7efd300b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 -- 172.21.15.122:0/67223723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd300d2480 msgr2=0x7efd301f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd300d2480 0x7efd301f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 -- 172.21.15.122:0/67223723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3015f5a0 con 0x7efd300d1910 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd349f8640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 0x7efd3015f060 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7efd2400d620 tx=0x7efd2400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2400b620 con 0x7efd300d1910 2024-03-06T07:29:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd24019070 con 0x7efd300d1910 2024-03-06T07:29:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd301f5210 con 0x7efd300d1910 2024-03-06T07:29:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2400b9e0 con 0x7efd300d1910 2024-03-06T07:29:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.587+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd300c7c80 con 0x7efd300d1910 2024-03-06T07:29:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.588+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd30068ff0 con 0x7efd300d1910 2024-03-06T07:29:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.589+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd2400bb80 con 0x7efd300d1910 2024-03-06T07:29:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.589+0000 7efd2d7fa640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd1406b4d0 0x7efd1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.589+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efd2408a5e0 con 0x7efd300d1910 2024-03-06T07:29:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.589+0000 7efd2ffff640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd1406b4d0 0x7efd1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.590+0000 7efd2ffff640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd1406b4d0 0x7efd1406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7efd1c000f60 tx=0x7efd1c007a80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.594+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd240594b0 con 0x7efd300d1910 2024-03-06T07:29:53.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.769+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efd300d49a0 con 0x7efd300d1910 2024-03-06T07:29:53.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.769+0000 7efd2d7fa640 1 -- 172.21.15.122:0/67223723 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7efd2405de20 con 0x7efd300d1910 2024-03-06T07:29:53.771 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:29:53.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd1406b4d0 msgr2=0x7efd1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd1406b4d0 0x7efd1406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7efd1c000f60 tx=0x7efd1c007a80 comp rx=0 tx=0).stop 2024-03-06T07:29:53.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 msgr2=0x7efd3015f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 0x7efd3015f060 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7efd2400d620 tx=0x7efd2400dae0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 shutdown_connections 2024-03-06T07:29:53.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efd1406b4d0 0x7efd1406d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efd301614d0 0x7efd300b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efd300d2480 0x7efd301f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 --2- 172.21.15.122:0/67223723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd300d1910 0x7efd3015f060 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 >> 172.21.15.122:0/67223723 conn(0x7efd300d3d40 msgr2=0x7efd300d1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 shutdown_connections 2024-03-06T07:29:53.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.771+0000 7efd36c5c640 1 -- 172.21.15.122:0/67223723 wait complete. 2024-03-06T07:29:53.784 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:53.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 Processor -- start 2024-03-06T07:29:53.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 -- start start 2024-03-06T07:29:53.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0b3430 con 0x7f93ec0cf830 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec0b31e0 con 0x7f93ec15dfe0 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.887+0000 7f93f25f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0b3d60 con 0x7f93ec0d29c0 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93ebfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93eb7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f93ec0cf830 0x7f93ec0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41750/0 (socket says 172.21.15.122:41750) 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93eb7fe640 1 -- 172.21.15.122:0/4105605502 learned_addr learned my addr 172.21.15.122:0/4105605502 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93f0b94640 1 --2- 172.21.15.122:0/4105605502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93ebfff640 1 -- 172.21.15.122:0/4105605502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 msgr2=0x7f93ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93ebfff640 1 --2- 172.21.15.122:0/4105605502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93ebfff640 1 -- 172.21.15.122:0/4105605502 >> v1:172.21.15.122:6789/0 conn(0x7f93ec0cf830 legacy=0x7f93ec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93ebfff640 1 -- 172.21.15.122:0/4105605502 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0b3710 con 0x7f93ec15dfe0 2024-03-06T07:29:53.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.888+0000 7f93eaffd640 1 -- 172.21.15.122:0/4105605502 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376476530 0 0) 0x7f93ec0b3430 con 0x7f93ec0cf830 2024-03-06T07:29:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93ebfff640 1 --2- 172.21.15.122:0/4105605502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0097a0 tx=0x7f93dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb25ac0a3ff5fee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93eaffd640 1 -- 172.21.15.122:0/4105605502 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc035020 con 0x7f93ec15dfe0 2024-03-06T07:29:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93eaffd640 1 -- 172.21.15.122:0/4105605502 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc031830 con 0x7f93ec15dfe0 2024-03-06T07:29:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93eaffd640 1 -- 172.21.15.122:0/4105605502 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc031c00 con 0x7f93ec15dfe0 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 -- 172.21.15.122:0/4105605502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 msgr2=0x7f93ec0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 --2- 172.21.15.122:0/4105605502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0097a0 tx=0x7f93dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 -- 172.21.15.122:0/4105605502 shutdown_connections 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 --2- 172.21.15.122:0/4105605502 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 --2- 172.21.15.122:0/4105605502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 -- 172.21.15.122:0/4105605502 >> 172.21.15.122:0/4105605502 conn(0x7f93ec0d3cd0 msgr2=0x7f93ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 -- 172.21.15.122:0/4105605502 shutdown_connections 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.889+0000 7f93f25f7640 1 -- 172.21.15.122:0/4105605502 wait complete. 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.890+0000 7f93f25f7640 1 Processor -- start 2024-03-06T07:29:53.895 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 -- start start 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93ec0cf830 0x7f93ec15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0d1df0 con 0x7f93ec15dfe0 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0d1c70 con 0x7f93ec0cf830 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f25f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec15e3c0 con 0x7f93ec0d29c0 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f0b94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f0b94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39102/0 (socket says 172.21.15.122:39102) 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.891+0000 7f93f0b94640 1 -- 172.21.15.122:0/3143923384 learned_addr learned my addr 172.21.15.122:0/3143923384 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93eb7fe640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93ebfff640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93ec0cf830 0x7f93ec15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f0b94640 1 -- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 msgr2=0x7f93ec0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f0b94640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f0b94640 1 -- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93ec0cf830 msgr2=0x7f93ec15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f0b94640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93ec0cf830 0x7f93ec15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f0b94640 1 -- 172.21.15.122:0/3143923384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0cd220 con 0x7f93ec15dfe0 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93ebfff640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93ec0cf830 0x7f93ec15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f0b94640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec160450 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f93e4007e40 tx=0x7f93e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4014620 con 0x7f93ec15dfe0 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e4019070 con 0x7f93ec15dfe0 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec0cd500 con 0x7f93ec15dfe0 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4014a20 con 0x7f93ec15dfe0 2024-03-06T07:29:53.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.892+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec0cdac0 con 0x7f93ec15dfe0 2024-03-06T07:29:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.894+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec0d1a90 con 0x7f93ec15dfe0 2024-03-06T07:29:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.894+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93e4014bc0 con 0x7f93ec15dfe0 2024-03-06T07:29:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.894+0000 7f93e8ff9640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93d806b500 0x7f93d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.894+0000 7f93ebfff640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93d806b500 0x7f93d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.894+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f93e408a990 con 0x7f93ec15dfe0 2024-03-06T07:29:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.895+0000 7f93ebfff640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93d806b500 0x7f93d806d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f93dc036000 tx=0x7f93dc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:53.903+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93ec0d1a90 con 0x7f93ec15dfe0 2024-03-06T07:29:54.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.126+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ec0cef60 con 0x7f93ec15dfe0 2024-03-06T07:29:54.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.127+0000 7f93e8ff9640 1 -- 172.21.15.122:0/3143923384 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3402 (secure 0 0 0) 0x7f93e4059860 con 0x7f93ec15dfe0 2024-03-06T07:29:54.129 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:54.129 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/696408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":696408149},{"type":"v1","addr":"172.21.15.122:6836","nonce":696408149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:47.105721+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:54.129 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 83 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93d806b500 msgr2=0x7f93d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93d806b500 0x7f93d806d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f93dc036000 tx=0x7f93dc008030 comp rx=0 tx=0).stop 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 msgr2=0x7f93ec160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec160450 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f93e4007e40 tx=0x7f93e400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 shutdown_connections 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f93d806b500 0x7f93d806d9c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93ec15dfe0 0x7f93ec160450 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93ec0d29c0 0x7f93ec0cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 --2- 172.21.15.122:0/3143923384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93ec0cf830 0x7f93ec15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 >> 172.21.15.122:0/3143923384 conn(0x7f93ec0d3cd0 msgr2=0x7f93ec15d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 shutdown_connections 2024-03-06T07:29:54.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.130+0000 7f93f25f7640 1 -- 172.21.15.122:0/3143923384 wait complete. 2024-03-06T07:29:54.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.258+0000 7f0a6b2b5640 1 Processor -- start 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a6b2b5640 1 -- start start 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a6b2b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0a6415ab00 0x7f0a640b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a6b2b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0a640afdd0 0x7f0a6415bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a6b2b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a640b4e30 con 0x7f0a640b48f0 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a6b2b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640b5110 con 0x7f0a6415ab00 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a6b2b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6415b310 con 0x7f0a640afdd0 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69051640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0a6415ab00 0x7f0a640b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a68850640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0a640b48f0 0x7f0a640af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41760/0 (socket says 172.21.15.122:41760) 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a68850640 1 -- 172.21.15.122:0/2921412684 learned_addr learned my addr 172.21.15.122:0/2921412684 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:54.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69852640 1 --2- 172.21.15.122:0/2921412684 >> v2:172.21.15.169:3300/0 conn(0x7f0a640afdd0 0x7f0a6415bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69051640 1 -- 172.21.15.122:0/2921412684 >> v2:172.21.15.169:3300/0 conn(0x7f0a640afdd0 msgr2=0x7f0a6415bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69051640 1 --2- 172.21.15.122:0/2921412684 >> v2:172.21.15.169:3300/0 conn(0x7f0a640afdd0 0x7f0a6415bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69051640 1 -- 172.21.15.122:0/2921412684 >> v1:172.21.15.122:6789/0 conn(0x7f0a640b48f0 legacy=0x7f0a640af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69051640 1 -- 172.21.15.122:0/2921412684 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6415b5f0 con 0x7f0a6415ab00 2024-03-06T07:29:54.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69852640 1 --2- 172.21.15.122:0/2921412684 >> v2:172.21.15.169:3300/0 conn(0x7f0a640afdd0 0x7f0a6415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.259+0000 7f0a69051640 1 --2- 172.21.15.122:0/2921412684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a640b43b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0a6406d070 tx=0x7f0a6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca379fd2eae08be8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a5bfff640 1 -- 172.21.15.122:0/2921412684 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60035020 con 0x7f0a6415ab00 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a5bfff640 1 -- 172.21.15.122:0/2921412684 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60031830 con 0x7f0a6415ab00 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a5bfff640 1 -- 172.21.15.122:0/2921412684 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60031c00 con 0x7f0a6415ab00 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/2921412684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 msgr2=0x7f0a640b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/2921412684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a640b43b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0a6406d070 tx=0x7f0a6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/2921412684 shutdown_connections 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/2921412684 >> v2:172.21.15.169:3300/0 conn(0x7f0a640afdd0 0x7f0a6415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/2921412684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a640b43b0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/2921412684 >> 172.21.15.122:0/2921412684 conn(0x7f0a6406c0c0 msgr2=0x7f0a6406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/2921412684 shutdown_connections 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/2921412684 wait complete. 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 Processor -- start 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.261+0000 7f0a6b2b5640 1 -- start start 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a6b2b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a640afdd0 0x7f0a640d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a6b2b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a640b48f0 0x7f0a641f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a6b2b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a641f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a6b2b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415b060 con 0x7f0a6415ab00 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a6b2b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415aee0 con 0x7f0a640b48f0 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a6b2b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415c050 con 0x7f0a640afdd0 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a69852640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a641f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a69852640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a641f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:39118/0 (socket says 172.21.15.122:39118) 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a69852640 1 -- 172.21.15.122:0/1715725775 learned_addr learned my addr 172.21.15.122:0/1715725775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:54.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a69051640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a640afdd0 0x7f0a640d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.263+0000 7f0a68850640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a640b48f0 0x7f0a641f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a69852640 1 -- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a640afdd0 msgr2=0x7f0a640d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a69852640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a640afdd0 0x7f0a640d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a69852640 1 -- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a640b48f0 msgr2=0x7f0a641f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a69852640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a640b48f0 0x7f0a641f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a69852640 1 -- 172.21.15.122:0/1715725775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a641f4720 con 0x7f0a6415ab00 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a69852640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a641f4130 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c007e90 tx=0x7f0a5c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c01a070 con 0x7f0a6415ab00 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c003bb0 con 0x7f0a6415ab00 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a641f4a00 con 0x7f0a6415ab00 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00de60 con 0x7f0a6415ab00 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.264+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a641f52a0 con 0x7f0a6415ab00 2024-03-06T07:29:54.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.265+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a5c01d070 con 0x7f0a6415ab00 2024-03-06T07:29:54.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.265+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6404f330 con 0x7f0a6415ab00 2024-03-06T07:29:54.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.266+0000 7f0a59ffb640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a4806b4d0 0x7f0a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.266+0000 7f0a69051640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a4806b4d0 0x7f0a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.266+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0a5c08a370 con 0x7f0a6415ab00 2024-03-06T07:29:54.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.266+0000 7f0a69051640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a4806b4d0 0x7f0a4806d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0a60036000 tx=0x7f0a6003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.270+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a5c010070 con 0x7f0a6415ab00 2024-03-06T07:29:54.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.471+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a640b2f10 con 0x7f0a6415ab00 2024-03-06T07:29:54.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.472+0000 7f0a59ffb640 1 -- 172.21.15.122:0/1715725775 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f0a5c058bf0 con 0x7f0a6415ab00 2024-03-06T07:29:54.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:54.474 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":130,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:29:35.994874+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:54.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a4806b4d0 msgr2=0x7f0a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a4806b4d0 0x7f0a4806d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0a60036000 tx=0x7f0a6003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 msgr2=0x7f0a641f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a641f4130 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c007e90 tx=0x7f0a5c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 shutdown_connections 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0a4806b4d0 0x7f0a4806d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a6415ab00 0x7f0a641f4130 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a640b48f0 0x7f0a641f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 --2- 172.21.15.122:0/1715725775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0a640afdd0 0x7f0a640d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 >> 172.21.15.122:0/1715725775 conn(0x7f0a6406c0c0 msgr2=0x7f0a640b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 shutdown_connections 2024-03-06T07:29:54.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.474+0000 7f0a6b2b5640 1 -- 172.21.15.122:0/1715725775 wait complete. 2024-03-06T07:29:54.488 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:54.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 Processor -- start 2024-03-06T07:29:54.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 -- start start 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff9e815dfe0 0x7ff9e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e80d29c0 0x7ff9e80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80b3430 con 0x7ff9e80cf830 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80b31e0 con 0x7ff9e815dfe0 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.588+0000 7ff9edef4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80b3d60 con 0x7ff9e80d29c0 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff9e815dfe0 0x7ff9e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e6ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff9e80cf830 0x7ff9e80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53168/0 (socket says 172.21.15.122:53168) 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e6ffd640 1 -- 172.21.15.122:0/2785009828 learned_addr learned my addr 172.21.15.122:0/2785009828 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e7fff640 1 --2- 172.21.15.122:0/2785009828 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e80d29c0 0x7ff9e80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e77fe640 1 -- 172.21.15.122:0/2785009828 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e80d29c0 msgr2=0x7ff9e80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e77fe640 1 --2- 172.21.15.122:0/2785009828 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e80d29c0 0x7ff9e80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e77fe640 1 -- 172.21.15.122:0/2785009828 >> v1:172.21.15.122:6789/0 conn(0x7ff9e80cf830 legacy=0x7ff9e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e77fe640 1 -- 172.21.15.122:0/2785009828 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80b3710 con 0x7ff9e815dfe0 2024-03-06T07:29:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.589+0000 7ff9e67fc640 1 -- 172.21.15.122:0/2785009828 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434292855 0 0) 0x7ff9e80b3430 con 0x7ff9e80cf830 2024-03-06T07:29:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9e77fe640 1 --2- 172.21.15.122:0/2785009828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e815dfe0 0x7ff9e80cf2f0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80097a0 tx=0x7ff9d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9a267804413797 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9e67fc640 1 -- 172.21.15.122:0/2785009828 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8035020 con 0x7ff9e815dfe0 2024-03-06T07:29:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9e67fc640 1 -- 172.21.15.122:0/2785009828 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8031830 con 0x7ff9e815dfe0 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9e67fc640 1 -- 172.21.15.122:0/2785009828 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8031c00 con 0x7ff9e815dfe0 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 -- 172.21.15.122:0/2785009828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e815dfe0 msgr2=0x7ff9e80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 --2- 172.21.15.122:0/2785009828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e815dfe0 0x7ff9e80cf2f0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80097a0 tx=0x7ff9d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 -- 172.21.15.122:0/2785009828 shutdown_connections 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 --2- 172.21.15.122:0/2785009828 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e80d29c0 0x7ff9e80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 --2- 172.21.15.122:0/2785009828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e815dfe0 0x7ff9e80cf2f0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 -- 172.21.15.122:0/2785009828 >> 172.21.15.122:0/2785009828 conn(0x7ff9e80d3cd0 msgr2=0x7ff9e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 -- 172.21.15.122:0/2785009828 shutdown_connections 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.590+0000 7ff9edef4640 1 -- 172.21.15.122:0/2785009828 wait complete. 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.591+0000 7ff9edef4640 1 Processor -- start 2024-03-06T07:29:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 -- start start 2024-03-06T07:29:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 0x7ff9e815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9e80d29c0 0x7ff9e80cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e815dfe0 0x7ff9e8160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80d1df0 con 0x7ff9e80cf830 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80d1c70 con 0x7ff9e80d29c0 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9edef4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e815e3c0 con 0x7ff9e815dfe0 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.592+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 0x7ff9e815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 0x7ff9e815c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55158/0 (socket says 172.21.15.122:55158) 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 -- 172.21.15.122:0/409300473 learned_addr learned my addr 172.21.15.122:0/409300473 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e6ffd640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9e80d29c0 0x7ff9e80cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e7fff640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e815dfe0 0x7ff9e8160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 -- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e815dfe0 msgr2=0x7ff9e8160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e815dfe0 0x7ff9e8160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 -- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9e80d29c0 msgr2=0x7ff9e80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9e80d29c0 0x7ff9e80cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 -- 172.21.15.122:0/409300473 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80cd220 con 0x7ff9e80cf830 2024-03-06T07:29:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e7fff640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e815dfe0 0x7ff9e8160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9e77fe640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 0x7ff9e815c0a0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80097a0 tx=0x7ff9d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8035040 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8004230 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e80cd500 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d803a700 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.593+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e80cdac0 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.594+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80d1a90 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.595+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9d8038020 con 0x7ff9e80cf830 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.595+0000 7ff9cffff640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff9d406b500 0x7ff9d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.595+0000 7ff9e6ffd640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff9d406b500 0x7ff9d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.595+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff9d80abeb0 con 0x7ff9e80cf830 2024-03-06T07:29:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.596+0000 7ff9e6ffd640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff9d406b500 0x7ff9d406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc000f60 tx=0x7ff9dc007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.600+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9e80d1a90 con 0x7ff9e80cf830 2024-03-06T07:29:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.907+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9e80cef60 con 0x7ff9e80cf830 2024-03-06T07:29:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.908+0000 7ff9cffff640 1 -- 172.21.15.122:0/409300473 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3402 (secure 0 0 0) 0x7ff9d807ad80 con 0x7ff9e80cf830 2024-03-06T07:29:54.909 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:54.909 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/696408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":696408149},{"type":"v1","addr":"172.21.15.122:6836","nonce":696408149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:47.105721+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:54.909 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 83 2024-03-06T07:29:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff9d406b500 msgr2=0x7ff9d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff9d406b500 0x7ff9d406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc000f60 tx=0x7ff9dc007d50 comp rx=0 tx=0).stop 2024-03-06T07:29:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 msgr2=0x7ff9e815c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 0x7ff9e815c0a0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80097a0 tx=0x7ff9d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 shutdown_connections 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff9d406b500 0x7ff9d406d9c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff9e815dfe0 0x7ff9e8160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9e80d29c0 0x7ff9e80cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 --2- 172.21.15.122:0/409300473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9e80cf830 0x7ff9e815c0a0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 >> 172.21.15.122:0/409300473 conn(0x7ff9e80d3cd0 msgr2=0x7ff9e815d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 shutdown_connections 2024-03-06T07:29:54.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:54.910+0000 7ff9edef4640 1 -- 172.21.15.122:0/409300473 wait complete. 2024-03-06T07:29:54.923 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:55.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.026+0000 7f89c22c9640 1 Processor -- start 2024-03-06T07:29:55.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.026+0000 7f89c22c9640 1 -- start start 2024-03-06T07:29:55.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c22c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c22c9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f89bc0cf830 0x7f89bc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c22c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f89bc0d27f0 0x7f89bc0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c22c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0d2c50 con 0x7f89bc0d27f0 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c22c9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0b31e0 con 0x7f89bc0cf830 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c22c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0b3d60 con 0x7f89bc15dfe0 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f89bc0d27f0 0x7f89bc0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f89bc0d27f0 0x7f89bc0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55160/0 (socket says 172.21.15.122:55160) 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 -- 172.21.15.122:0/1065186613 learned_addr learned my addr 172.21.15.122:0/1065186613 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89bbfff640 1 --2- 172.21.15.122:0/1065186613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89bb7fe640 1 --2- 172.21.15.122:0/1065186613 >> v2:172.21.15.169:3300/0 conn(0x7f89bc0cf830 0x7f89bc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 -- 172.21.15.122:0/1065186613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc15dfe0 msgr2=0x7f89bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 --2- 172.21.15.122:0/1065186613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 -- 172.21.15.122:0/1065186613 >> v2:172.21.15.169:3300/0 conn(0x7f89bc0cf830 msgr2=0x7f89bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 --2- 172.21.15.122:0/1065186613 >> v2:172.21.15.169:3300/0 conn(0x7f89bc0cf830 0x7f89bc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.027+0000 7f89c0866640 1 -- 172.21.15.122:0/1065186613 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0b3430 con 0x7f89bc0d27f0 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.028+0000 7f89c0866640 1 --2- 172.21.15.122:0/1065186613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0d0300 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f89bc08cba0 tx=0x7f89b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54909192ec49aad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.028+0000 7f89baffd640 1 -- 172.21.15.122:0/1065186613 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b402ca70 con 0x7f89bc0d27f0 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.028+0000 7f89baffd640 1 -- 172.21.15.122:0/1065186613 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b402cc10 con 0x7f89bc0d27f0 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.028+0000 7f89baffd640 1 -- 172.21.15.122:0/1065186613 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b40343f0 con 0x7f89bc0d27f0 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 -- 172.21.15.122:0/1065186613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 msgr2=0x7f89bc0d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 --2- 172.21.15.122:0/1065186613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0d0300 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f89bc08cba0 tx=0x7f89b402bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 -- 172.21.15.122:0/1065186613 shutdown_connections 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 --2- 172.21.15.122:0/1065186613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0d0300 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 --2- 172.21.15.122:0/1065186613 >> v2:172.21.15.169:3300/0 conn(0x7f89bc0cf830 0x7f89bc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 --2- 172.21.15.122:0/1065186613 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 -- 172.21.15.122:0/1065186613 >> 172.21.15.122:0/1065186613 conn(0x7f89bc0d3cd0 msgr2=0x7f89bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 -- 172.21.15.122:0/1065186613 shutdown_connections 2024-03-06T07:29:55.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.029+0000 7f89c22c9640 1 -- 172.21.15.122:0/1065186613 wait complete. 2024-03-06T07:29:55.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.030+0000 7f89c22c9640 1 Processor -- start 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.030+0000 7f89c22c9640 1 -- start start 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c22c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc0cf830 0x7f89bc1f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c22c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c22c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89bc15dfe0 0x7f89bc0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c22c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0d1c70 con 0x7f89bc0d27f0 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c22c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0b39f0 con 0x7f89bc15dfe0 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c22c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0d1df0 con 0x7f89bc0cf830 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55172/0 (socket says 172.21.15.122:55172) 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 -- 172.21.15.122:0/2915526778 learned_addr learned my addr 172.21.15.122:0/2915526778 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bbfff640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc0cf830 0x7f89bc1f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c0866640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89bc15dfe0 0x7f89bc0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 -- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc0cf830 msgr2=0x7f89bc1f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc0cf830 0x7f89bc1f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 -- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89bc15dfe0 msgr2=0x7f89bc0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89bc15dfe0 0x7f89bc0b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 -- 172.21.15.122:0/2915526778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0b8970 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89c0866640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89bc15dfe0 0x7f89bc0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89bb7fe640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0cbdf0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00d7d0 tx=0x7f89ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.031+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0096a0 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.032+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ac019070 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.032+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc1f7910 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.032+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac009a60 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.032+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc1f7ed0 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.032+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc04f330 con 0x7f89bc0d27f0 2024-03-06T07:29:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.033+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89ac009c00 con 0x7f89bc0d27f0 2024-03-06T07:29:55.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.033+0000 7f89b8ff9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f89a806b500 0x7f89a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.033+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f89ac08a5b0 con 0x7f89bc0d27f0 2024-03-06T07:29:55.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.033+0000 7f89bbfff640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f89a806b500 0x7f89a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.034+0000 7f89bbfff640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f89a806b500 0x7f89a806d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006fd0 tx=0x7f89b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.038+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89bc0b8970 con 0x7f89bc0d27f0 2024-03-06T07:29:55.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:55.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.256+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89bc0ce870 con 0x7f89bc0d27f0 2024-03-06T07:29:55.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.256+0000 7f89b8ff9640 1 -- 172.21.15.122:0/2915526778 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3402 (secure 0 0 0) 0x7f89ac059480 con 0x7f89bc0d27f0 2024-03-06T07:29:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/696408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":696408149},{"type":"v1","addr":"172.21.15.122:6836","nonce":696408149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:41.515900+0000","modified":"2024-03-06T07:29:47.105721+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8688},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8688":{"gid":8688,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/2094222247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2094222247},{"type":"v1","addr":"172.21.15.122:6837","nonce":2094222247}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8688,"qdb_cluster":[8688]},"id":9}]} 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 83 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f89a806b500 msgr2=0x7f89a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f89a806b500 0x7f89a806d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006fd0 tx=0x7f89b0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 msgr2=0x7f89bc0cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0cbdf0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00d7d0 tx=0x7f89ac00dc90 comp rx=0 tx=0).stop 2024-03-06T07:29:55.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 shutdown_connections 2024-03-06T07:29:55.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f89a806b500 0x7f89a806d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89bc15dfe0 0x7f89bc0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89bc0d27f0 0x7f89bc0cbdf0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 --2- 172.21.15.122:0/2915526778 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f89bc0cf830 0x7f89bc1f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 >> 172.21.15.122:0/2915526778 conn(0x7f89bc0d3cd0 msgr2=0x7f89bc0cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 shutdown_connections 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.258+0000 7f89c22c9640 1 -- 172.21.15.122:0/2915526778 wait complete. 2024-03-06T07:29:55.270 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:55.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.374+0000 7fd076d01640 1 Processor -- start 2024-03-06T07:29:55.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.374+0000 7fd076d01640 1 -- start start 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd076d01640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd07015dfe0 0x7fd0700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd076d01640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700cf830 0x7fd0700d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd076d01640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700b3430 con 0x7fd0700d27f0 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd076d01640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0700d2c50 con 0x7fd07015dfe0 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd076d01640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700b31e0 con 0x7fd0700cf830 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd074a9d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd07015dfe0 0x7fd0700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd06ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700cf830 0x7fd0700d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd07529e640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd0700d27f0 0x7fd0700d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:52390/0 (socket says 172.21.15.122:52390) 2024-03-06T07:29:55.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd07529e640 1 -- 172.21.15.122:0/3281464256 learned_addr learned my addr 172.21.15.122:0/3281464256 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:55.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd06ffff640 1 -- 172.21.15.122:0/3281464256 >> v2:172.21.15.169:3300/0 conn(0x7fd07015dfe0 msgr2=0x7fd0700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd06ffff640 1 --2- 172.21.15.122:0/3281464256 >> v2:172.21.15.169:3300/0 conn(0x7fd07015dfe0 0x7fd0700cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd06ffff640 1 -- 172.21.15.122:0/3281464256 >> v1:172.21.15.169:6789/0 conn(0x7fd0700d27f0 legacy=0x7fd0700d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd06ffff640 1 -- 172.21.15.122:0/3281464256 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0700b3710 con 0x7fd0700cf830 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.375+0000 7fd074a9d640 1 --2- 172.21.15.122:0/3281464256 >> v2:172.21.15.169:3300/0 conn(0x7fd07015dfe0 0x7fd0700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.376+0000 7fd06ffff640 1 --2- 172.21.15.122:0/3281464256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700cf830 0x7fd0700d22b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0097a0 tx=0x7fd05c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7874cb4695cd30d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.376+0000 7fd06f7fe640 1 -- 172.21.15.122:0/3281464256 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c035020 con 0x7fd0700cf830 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd06f7fe640 1 -- 172.21.15.122:0/3281464256 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c031830 con 0x7fd0700cf830 2024-03-06T07:29:55.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd06f7fe640 1 -- 172.21.15.122:0/3281464256 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c031c00 con 0x7fd0700cf830 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 -- 172.21.15.122:0/3281464256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700cf830 msgr2=0x7fd0700d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 --2- 172.21.15.122:0/3281464256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700cf830 0x7fd0700d22b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0097a0 tx=0x7fd05c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 -- 172.21.15.122:0/3281464256 shutdown_connections 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 --2- 172.21.15.122:0/3281464256 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700cf830 0x7fd0700d22b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 --2- 172.21.15.122:0/3281464256 >> v2:172.21.15.169:3300/0 conn(0x7fd07015dfe0 0x7fd0700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 -- 172.21.15.122:0/3281464256 >> 172.21.15.122:0/3281464256 conn(0x7fd0700d3cd0 msgr2=0x7fd0700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 -- 172.21.15.122:0/3281464256 shutdown_connections 2024-03-06T07:29:55.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.377+0000 7fd076d01640 1 -- 172.21.15.122:0/3281464256 wait complete. 2024-03-06T07:29:55.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.379+0000 7fd076d01640 1 Processor -- start 2024-03-06T07:29:55.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.379+0000 7fd076d01640 1 -- start start 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd076d01640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0700cf830 0x7fd0700b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd076d01640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700d27f0 0x7fd0701532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd076d01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 0x7fd07015e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd076d01640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700d1c70 con 0x7fd07015dfe0 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd076d01640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700b3d60 con 0x7fd0700cf830 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd076d01640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700d1df0 con 0x7fd0700d27f0 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd07529e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 0x7fd07015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd07529e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 0x7fd07015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55178/0 (socket says 172.21.15.122:55178) 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd07529e640 1 -- 172.21.15.122:0/3014946155 learned_addr learned my addr 172.21.15.122:0/3014946155 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.380+0000 7fd06ffff640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700d27f0 0x7fd0701532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd074a9d640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0700cf830 0x7fd0700b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd07529e640 1 -- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700d27f0 msgr2=0x7fd0701532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd07529e640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700d27f0 0x7fd0701532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd07529e640 1 -- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0700cf830 msgr2=0x7fd0700b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd07529e640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0700cf830 0x7fd0700b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd07529e640 1 -- 172.21.15.122:0/3014946155 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070153830 con 0x7fd07015dfe0 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd07529e640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 0x7fd07015e450 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fd06800cbc0 tx=0x7fd06800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068017070 con 0x7fd07015dfe0 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06800ad40 con 0x7fd07015dfe0 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07015e990 con 0x7fd07015dfe0 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068012600 con 0x7fd07015dfe0 2024-03-06T07:29:55.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.381+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07015ef10 con 0x7fd07015dfe0 2024-03-06T07:29:55.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.382+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07004f330 con 0x7fd07015dfe0 2024-03-06T07:29:55.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.382+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd06801a070 con 0x7fd07015dfe0 2024-03-06T07:29:55.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.383+0000 7fd06d7fa640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd05406b4d0 0x7fd05406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.383+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd068089e60 con 0x7fd07015dfe0 2024-03-06T07:29:55.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.383+0000 7fd074a9d640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd05406b4d0 0x7fd05406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.383+0000 7fd074a9d640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd05406b4d0 0x7fd05406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd064009a30 tx=0x7fd064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.387+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd070153830 con 0x7fd07015dfe0 2024-03-06T07:29:55.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.567+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0700b1110 con 0x7fd07015dfe0 2024-03-06T07:29:55.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.568+0000 7fd06d7fa640 1 -- 172.21.15.122:0/3014946155 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16719 (secure 0 0 0) 0x7fd068058650 con 0x7fd07015dfe0 2024-03-06T07:29:55.569 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:29:55.570 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":130,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:29:35.994874+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.745958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:40.497682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:55.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd05406b4d0 msgr2=0x7fd05406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd05406b4d0 0x7fd05406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd064009a30 tx=0x7fd064008040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 msgr2=0x7fd07015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 0x7fd07015e450 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fd06800cbc0 tx=0x7fd06800a430 comp rx=0 tx=0).stop 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 shutdown_connections 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd05406b4d0 0x7fd05406d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd07015dfe0 0x7fd07015e450 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0700d27f0 0x7fd0701532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 --2- 172.21.15.122:0/3014946155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0700cf830 0x7fd0700b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 >> 172.21.15.122:0/3014946155 conn(0x7fd0700d3cd0 msgr2=0x7fd07015d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 shutdown_connections 2024-03-06T07:29:55.572 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.570+0000 7fd076d01640 1 -- 172.21.15.122:0/3014946155 wait complete. 2024-03-06T07:29:55.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:55.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.687+0000 7fbfd5b56640 1 Processor -- start 2024-03-06T07:29:55.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.687+0000 7fbfd5b56640 1 -- start start 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfd5b56640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfd00b0240 0x7fbfd006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfd5b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd006b4f0 0x7fbfd00cd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfd5b56640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd0161cf0 con 0x7fbfd01586c0 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfd5b56640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd0158aa0 con 0x7fbfd00b0240 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfd5b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd006b960 con 0x7fbfd006b4f0 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfceffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfd00b0240 0x7fbfd006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfceffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfd00b0240 0x7fbfd006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55180/0 (socket says 172.21.15.122:55180) 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfcf7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbfd01586c0 0x7fbfd015efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53180/0 (socket says 172.21.15.122:53180) 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfceffd640 1 -- 172.21.15.122:0/2223146658 learned_addr learned my addr 172.21.15.122:0/2223146658 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:55.690 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.688+0000 7fbfcffff640 1 --2- 172.21.15.122:0/2223146658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd006b4f0 0x7fbfd00cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.689+0000 7fbfceffd640 1 -- 172.21.15.122:0/2223146658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd006b4f0 msgr2=0x7fbfd00cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.689+0000 7fbfceffd640 1 --2- 172.21.15.122:0/2223146658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd006b4f0 0x7fbfd00cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.689+0000 7fbfceffd640 1 -- 172.21.15.122:0/2223146658 >> v1:172.21.15.122:6789/0 conn(0x7fbfd01586c0 legacy=0x7fbfd015efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.689+0000 7fbfceffd640 1 -- 172.21.15.122:0/2223146658 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0161fd0 con 0x7fbfd00b0240 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.689+0000 7fbfce7fc640 1 -- 172.21.15.122:0/2223146658 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724875742 0 0) 0x7fbfd0161cf0 con 0x7fbfd01586c0 2024-03-06T07:29:55.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.689+0000 7fbfceffd640 1 --2- 172.21.15.122:0/2223146658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd00b0240 0x7fbfd006afb0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009870 tx=0x7fbfc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f00337dd080766 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfce7fc640 1 -- 172.21.15.122:0/2223146658 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0034070 con 0x7fbfd00b0240 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfce7fc640 1 -- 172.21.15.122:0/2223146658 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0030840 con 0x7fbfd00b0240 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfce7fc640 1 -- 172.21.15.122:0/2223146658 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0030c10 con 0x7fbfd00b0240 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 -- 172.21.15.122:0/2223146658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd00b0240 msgr2=0x7fbfd006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/2223146658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd00b0240 0x7fbfd006afb0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009870 tx=0x7fbfc002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 -- 172.21.15.122:0/2223146658 shutdown_connections 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/2223146658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd006b4f0 0x7fbfd00cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/2223146658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd00b0240 0x7fbfd006afb0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 -- 172.21.15.122:0/2223146658 >> 172.21.15.122:0/2223146658 conn(0x7fbfd00d09d0 msgr2=0x7fbfd00d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 -- 172.21.15.122:0/2223146658 shutdown_connections 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.690+0000 7fbfd5b56640 1 -- 172.21.15.122:0/2223146658 wait complete. 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.691+0000 7fbfd5b56640 1 Processor -- start 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.691+0000 7fbfd5b56640 1 -- start start 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfd5b56640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 0x7fbfd01f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfd5b56640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd00b0240 0x7fbfd0156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfd5b56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfd01586c0 0x7fbfd01e86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfd5b56640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0162870 con 0x7fbfd006b4f0 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfd5b56640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0162590 con 0x7fbfd01586c0 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfd5b56640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd00719e0 con 0x7fbfd00b0240 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfcf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 0x7fbfd01f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfcf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 0x7fbfd01f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55188/0 (socket says 172.21.15.122:55188) 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfcf7fe640 1 -- 172.21.15.122:0/3303013201 learned_addr learned my addr 172.21.15.122:0/3303013201 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfceffd640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd00b0240 0x7fbfd0156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.692+0000 7fbfcffff640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfd01586c0 0x7fbfd01e86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcf7fe640 1 -- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd00b0240 msgr2=0x7fbfd0156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcf7fe640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd00b0240 0x7fbfd0156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcf7fe640 1 -- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfd01586c0 msgr2=0x7fbfd01e86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcf7fe640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfd01586c0 0x7fbfd01e86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcf7fe640 1 -- 172.21.15.122:0/3303013201 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0161300 con 0x7fbfd006b4f0 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfceffd640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd00b0240 0x7fbfd0156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcffff640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfd01586c0 0x7fbfd01e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfcf7fe640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 0x7fbfd01f4120 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c430 tx=0x7fbfc400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40074e0 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd00b39d0 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc40047e0 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4004bd0 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.693+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd00b3f90 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.694+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd004f330 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.695+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfc4011070 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.695+0000 7fbfb7fff640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.695+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbfc4089e60 con 0x7fbfd006b4f0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.695+0000 7fbfceffd640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.695+0000 7fbfceffd640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0036000 tx=0x7fbfc003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.699+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfc400e280 con 0x7fbfd006b4f0 2024-03-06T07:29:55.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.913+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbfd00d17a0 con 0x7fbfd006b4f0 2024-03-06T07:29:55.915 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:29:55.914+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:55.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.915+0000 7fbfb7fff640 1 -- 172.21.15.122:0/3303013201 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbfc4058cb0 con 0x7fbfd006b4f0 2024-03-06T07:29:55.916 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 msgr2=0x7fbfbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0036000 tx=0x7fbfc003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 msgr2=0x7fbfd01f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 0x7fbfd01f4120 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c430 tx=0x7fbfc400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 shutdown_connections 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfd01586c0 0x7fbfd01e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfd00b0240 0x7fbfd0156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 --2- 172.21.15.122:0/3303013201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfd006b4f0 0x7fbfd01f4120 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 >> 172.21.15.122:0/3303013201 conn(0x7fbfd00d09d0 msgr2=0x7fbfd00b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.918+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 shutdown_connections 2024-03-06T07:29:55.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:55.919+0000 7fbfd5b56640 1 -- 172.21.15.122:0/3303013201 wait complete. 2024-03-06T07:29:55.922 INFO:tasks.ceph.mds.c.smithi122.stderr: -8513> 2024-03-06T07:28:47.438+0000 7f599c482640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:55.923 INFO:tasks.ceph.mds.c.smithi122.stderr: -8017> 2024-03-06T07:28:52.438+0000 7f599c482640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:29:55.931 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:55.955 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:29:56.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.032+0000 7f876c8f4640 1 Processor -- start 2024-03-06T07:29:56.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.032+0000 7f876c8f4640 1 -- start start 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f876c8f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cff40 0x7f87680b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f876c8f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8768158ac0 con 0x7f87680cfa90 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f876c8f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876815b2a0 con 0x7f87681586e0 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f876c8f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876815b540 con 0x7f87680cff40 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f8765d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f87680cfa90 0x7f87680b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53188/0 (socket says 172.21.15.122:53188) 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f8765d9b640 1 -- 172.21.15.122:0/1504659503 learned_addr learned my addr 172.21.15.122:0/1504659503 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.033+0000 7f8766d9d640 1 --2- 172.21.15.122:0/1504659503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cff40 0x7f87680b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942103332 0 0) 0x7f8768158ac0 con 0x7f87680cfa90 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8750003520 con 0x7f87680cfa90 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686382217 0 0) 0x7f876815b2a0 con 0x7f87681586e0 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768158ac0 con 0x7f87681586e0 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959872195 0 0) 0x7f8750003520 con 0x7f87680cfa90 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f8766d9d640 1 -- 172.21.15.122:0/1504659503 >> v1:172.21.15.169:6789/0 conn(0x7f87681586e0 legacy=0x7f87680aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f8766d9d640 1 -- 172.21.15.122:0/1504659503 >> v1:172.21.15.122:6789/0 conn(0x7f87680cfa90 legacy=0x7f87680b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.034+0000 7f8766d9d640 1 -- 172.21.15.122:0/1504659503 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876815b820 con 0x7f87680cff40 2024-03-06T07:29:56.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f8766d9d640 1 --2- 172.21.15.122:0/1504659503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cff40 0x7f87680b3ab0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8760009dc0 tx=0x7f876002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d51d9b1bc12a84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8760039070 con 0x7f87680cff40 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876002c910 con 0x7f87680cff40 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876559a640 1 -- 172.21.15.122:0/1504659503 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758001670 con 0x7f87680cff40 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 -- 172.21.15.122:0/1504659503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cff40 msgr2=0x7f87680b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 --2- 172.21.15.122:0/1504659503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cff40 0x7f87680b3ab0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8760009dc0 tx=0x7f876002bde0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 -- 172.21.15.122:0/1504659503 shutdown_connections 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 --2- 172.21.15.122:0/1504659503 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cff40 0x7f87680b3ab0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 -- 172.21.15.122:0/1504659503 >> 172.21.15.122:0/1504659503 conn(0x7f876806b190 msgr2=0x7f876806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 -- 172.21.15.122:0/1504659503 shutdown_connections 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.035+0000 7f876c8f4640 1 -- 172.21.15.122:0/1504659503 wait complete. 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.037+0000 7f876c8f4640 1 Processor -- start 2024-03-06T07:29:56.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.037+0000 7f876c8f4640 1 -- start start 2024-03-06T07:29:56.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.037+0000 7f876c8f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cfa90 0x7f876813e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.037+0000 7f876c8f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 0x7f8768140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f876c8f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87681586e0 0x7f87681e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f876c8f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680d0360 con 0x7f87680cff40 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f876c8f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876815bde0 con 0x7f87681586e0 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f876c8f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680af150 con 0x7f87680cfa90 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 0x7f8768140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 0x7f8768140a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55216/0 (socket says 172.21.15.122:55216) 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 -- 172.21.15.122:0/2200100510 learned_addr learned my addr 172.21.15.122:0/2200100510 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 -- 172.21.15.122:0/2200100510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cfa90 msgr2=0x7f876813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cfa90 0x7f876813e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 -- 172.21.15.122:0/2200100510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87681586e0 msgr2=0x7f87681e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87681586e0 0x7f87681e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 -- 172.21.15.122:0/2200100510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87681419a0 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8765d9b640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 0x7f8768140a10 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f875800c780 tx=0x7f875800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758017070 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758005ad0 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758007070 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8768141100 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.038+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87681416c0 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.039+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680cc170 con 0x7f87680cff40 2024-03-06T07:29:56.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.040+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f875801a070 con 0x7f87680cff40 2024-03-06T07:29:56.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.040+0000 7f8756ffd640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874806b500 0x7f874806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.040+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8758089730 con 0x7f87680cff40 2024-03-06T07:29:56.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.040+0000 7f876659c640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874806b500 0x7f874806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.041+0000 7f876659c640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874806b500 0x7f874806d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f875c0010e0 tx=0x7f875c007950 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.044+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87680cc170 con 0x7f87680cff40 2024-03-06T07:29:56.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.255+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f87680732b0 con 0x7f87680cff40 2024-03-06T07:29:56.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.916+0000 7f8756ffd640 1 -- 172.21.15.122:0/2200100510 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87580585b0 con 0x7f87680cff40 2024-03-06T07:29:56.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874806b500 msgr2=0x7f874806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874806b500 0x7f874806d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f875c0010e0 tx=0x7f875c007950 comp rx=0 tx=0).stop 2024-03-06T07:29:56.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 msgr2=0x7f8768140a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 0x7f8768140a10 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f875800c780 tx=0x7f875800cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:56.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 shutdown_connections 2024-03-06T07:29:56.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874806b500 0x7f874806d9c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87681586e0 0x7f87681e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680cff40 0x7f8768140a10 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 --2- 172.21.15.122:0/2200100510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680cfa90 0x7f876813e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 >> 172.21.15.122:0/2200100510 conn(0x7f876806b190 msgr2=0x7f87680cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 shutdown_connections 2024-03-06T07:29:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:56.919+0000 7f876c8f4640 1 -- 172.21.15.122:0/2200100510 wait complete. 2024-03-06T07:29:56.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:29:57.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 Processor -- start 2024-03-06T07:29:57.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 -- start start 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f194c15c5a0 0x7f194c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f194c0ce8c0 0x7f194c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c072a20 con 0x7f194c0ce8c0 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c0708b0 con 0x7f194c15c5a0 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.079+0000 7f1952d5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c070e70 con 0x7f194c15c980 2024-03-06T07:29:57.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f194c0ce8c0 0x7f194c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f194c0ce8c0 0x7f194c0cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55222/0 (socket says 172.21.15.122:55222) 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 -- 172.21.15.122:0/3918681499 learned_addr learned my addr 172.21.15.122:0/3918681499 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f19512fb640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f1950afa640 1 --2- 172.21.15.122:0/3918681499 >> v2:172.21.15.169:3300/0 conn(0x7f194c15c5a0 0x7f194c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 -- 172.21.15.122:0/3918681499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 msgr2=0x7f194c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 -- 172.21.15.122:0/3918681499 >> v2:172.21.15.169:3300/0 conn(0x7f194c15c5a0 msgr2=0x7f194c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 --2- 172.21.15.122:0/3918681499 >> v2:172.21.15.169:3300/0 conn(0x7f194c15c5a0 0x7f194c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f194bfff640 1 -- 172.21.15.122:0/3918681499 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c06ce50 con 0x7f194c0ce8c0 2024-03-06T07:29:57.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.080+0000 7f19512fb640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f194bfff640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c0ce8c0 0x7f194c0cc330 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f194c06bf40 tx=0x7f193802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39661cccd841f86f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f194b7fe640 1 -- 172.21.15.122:0/3918681499 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193802e070 con 0x7f194c0ce8c0 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f194b7fe640 1 -- 172.21.15.122:0/3918681499 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1938037070 con 0x7f194c0ce8c0 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f194b7fe640 1 -- 172.21.15.122:0/3918681499 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938031a00 con 0x7f194c0ce8c0 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 -- 172.21.15.122:0/3918681499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c0ce8c0 msgr2=0x7f194c0cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c0ce8c0 0x7f194c0cc330 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f194c06bf40 tx=0x7f193802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 -- 172.21.15.122:0/3918681499 shutdown_connections 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c0ce8c0 0x7f194c0cc330 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 --2- 172.21.15.122:0/3918681499 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 --2- 172.21.15.122:0/3918681499 >> v2:172.21.15.169:3300/0 conn(0x7f194c15c5a0 0x7f194c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 -- 172.21.15.122:0/3918681499 >> 172.21.15.122:0/3918681499 conn(0x7f194c0b1650 msgr2=0x7f194c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 -- 172.21.15.122:0/3918681499 shutdown_connections 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.081+0000 7f1952d5e640 1 -- 172.21.15.122:0/3918681499 wait complete. 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.082+0000 7f1952d5e640 1 Processor -- start 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.082+0000 7f1952d5e640 1 -- start start 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f1952d5e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f194c15c5a0 0x7f194c144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f1952d5e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c1f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f1952d5e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 0x7f194c158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f1952d5e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c070b00 con 0x7f194c144100 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f1952d5e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0cf030 con 0x7f194c15c5a0 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f1952d5e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0cc870 con 0x7f194c15c980 2024-03-06T07:29:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f19512fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 0x7f194c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f19512fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 0x7f194c158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55236/0 (socket says 172.21.15.122:55236) 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.083+0000 7f19512fb640 1 -- 172.21.15.122:0/1438323179 learned_addr learned my addr 172.21.15.122:0/1438323179 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f194bfff640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c1f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f1950afa640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f194c15c5a0 0x7f194c144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19512fb640 1 -- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 msgr2=0x7f194c1f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19512fb640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c1f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19512fb640 1 -- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f194c15c5a0 msgr2=0x7f194c144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19512fb640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f194c15c5a0 0x7f194c144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19512fb640 1 -- 172.21.15.122:0/1438323179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c158780 con 0x7f194c144100 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f1950afa640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f194c15c5a0 0x7f194c144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19512fb640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 0x7f194c158240 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f1944007ef0 tx=0x7f194400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194401a070 con 0x7f194c144100 2024-03-06T07:29:57.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1944003b90 con 0x7f194c144100 2024-03-06T07:29:57.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c0abda0 con 0x7f194c144100 2024-03-06T07:29:57.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944004030 con 0x7f194c144100 2024-03-06T07:29:57.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.084+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c0ac770 con 0x7f194c144100 2024-03-06T07:29:57.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.085+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c0687d0 con 0x7f194c144100 2024-03-06T07:29:57.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.085+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f194401d070 con 0x7f194c144100 2024-03-06T07:29:57.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.086+0000 7f19497fa640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f193006b4d0 0x7f193006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.086+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f194408a4d0 con 0x7f194c144100 2024-03-06T07:29:57.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.086+0000 7f1950afa640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f193006b4d0 0x7f193006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.086+0000 7f1950afa640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f193006b4d0 0x7f193006d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f1940006fd0 tx=0x7f1940008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.090+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1944010070 con 0x7f194c144100 2024-03-06T07:29:57.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:57.261+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f194c15bef0 con 0x7f194c144100 2024-03-06T07:29:58.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.077+0000 7f19497fa640 1 -- 172.21.15.122:0/1438323179 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1944058c70 con 0x7f194c144100 2024-03-06T07:29:58.079 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:29:58.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f193006b4d0 msgr2=0x7f193006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f193006b4d0 0x7f193006d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f1940006fd0 tx=0x7f1940008040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 msgr2=0x7f194c158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 0x7f194c158240 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f1944007ef0 tx=0x7f194400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 shutdown_connections 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f193006b4d0 0x7f193006d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194c144100 0x7f194c158240 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f194c15c980 0x7f194c1f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 --2- 172.21.15.122:0/1438323179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f194c15c5a0 0x7f194c144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 >> 172.21.15.122:0/1438323179 conn(0x7f194c0b1650 msgr2=0x7f194c0b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 shutdown_connections 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.083+0000 7f1952d5e640 1 -- 172.21.15.122:0/1438323179 wait complete. 2024-03-06T07:29:58.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.207+0000 7f54e2dd6640 1 Processor -- start 2024-03-06T07:29:58.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.207+0000 7f54e2dd6640 1 -- start start 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e2dd6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e2dd6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54dc070d10 0x7f54dc15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e2dd6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc15afe0 con 0x7f54dc0708b0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e2dd6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0cdc10 con 0x7f54dc070d10 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e2dd6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0cd000 con 0x7f54dc15ac00 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e1373640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54dc070d10 0x7f54dc15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e0b72640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54dbfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f54dc0708b0 0x7f54dc15e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:52410/0 (socket says 172.21.15.122:52410) 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e1373640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f54dc070d10 0x7f54dc15eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:48748/0 (socket says 172.21.15.122:48748) 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54dbfff640 1 -- 172.21.15.122:0/4150419811 learned_addr learned my addr 172.21.15.122:0/4150419811 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e1373640 1 -- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 msgr2=0x7f54dc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e1373640 1 --2- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e1373640 1 -- 172.21.15.122:0/4150419811 >> v1:172.21.15.169:6789/0 conn(0x7f54dc0708b0 legacy=0x7f54dc15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.208+0000 7f54e1373640 1 -- 172.21.15.122:0/4150419811 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0cd2e0 con 0x7f54dc070d10 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.209+0000 7f54e1373640 1 --2- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc070d10 0x7f54dc15eeb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009d90 tx=0x7f54d402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9494431ad1355fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.209+0000 7f54db7fe640 1 -- 172.21.15.122:0/4150419811 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4039070 con 0x7f54dc070d10 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.209+0000 7f54db7fe640 1 -- 172.21.15.122:0/4150419811 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d402cbc0 con 0x7f54dc070d10 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.209+0000 7f54db7fe640 1 -- 172.21.15.122:0/4150419811 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4034440 con 0x7f54dc070d10 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 -- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc070d10 msgr2=0x7f54dc15eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc070d10 0x7f54dc15eeb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009d90 tx=0x7f54d402be10 comp rx=0 tx=0).stop 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 -- 172.21.15.122:0/4150419811 shutdown_connections 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc070d10 0x7f54dc15eeb0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/4150419811 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 -- 172.21.15.122:0/4150419811 >> 172.21.15.122:0/4150419811 conn(0x7f54dc0b1650 msgr2=0x7f54dc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 -- 172.21.15.122:0/4150419811 shutdown_connections 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.210+0000 7f54e2dd6640 1 -- 172.21.15.122:0/4150419811 wait complete. 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.211+0000 7f54e2dd6640 1 Processor -- start 2024-03-06T07:29:58.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.211+0000 7f54e2dd6640 1 -- start start 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e2dd6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc0708b0 0x7f54dc15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e2dd6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 0x7f54dc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e2dd6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e2dd6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071190 con 0x7f54dc070d10 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e2dd6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0cd8a0 con 0x7f54dc0708b0 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e2dd6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc06ce50 con 0x7f54dc15ac00 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 0x7f54dc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 0x7f54dc06d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55252/0 (socket says 172.21.15.122:55252) 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 -- 172.21.15.122:0/3104442647 learned_addr learned my addr 172.21.15.122:0/3104442647 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e1373640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e0b72640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc0708b0 0x7f54dc15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 -- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 msgr2=0x7f54dc06d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 -- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc0708b0 msgr2=0x7f54dc15b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc0708b0 0x7f54dc15b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54dbfff640 1 -- 172.21.15.122:0/3104442647 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc15c0d0 con 0x7f54dc070d10 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.212+0000 7f54e0b72640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc0708b0 0x7f54dc15b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.213+0000 7f54dbfff640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 0x7f54dc06d1a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f54c800c470 tx=0x7f54c8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.213+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8017070 con 0x7f54dc070d10 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.213+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8005ce0 con 0x7f54dc070d10 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.213+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc15c3b0 con 0x7f54dc070d10 2024-03-06T07:29:58.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.213+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c80072f0 con 0x7f54dc070d10 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.213+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc14ff30 con 0x7f54dc070d10 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.214+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc0687d0 con 0x7f54dc070d10 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.214+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c801a070 con 0x7f54dc070d10 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.215+0000 7f54d97fa640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54c006b500 0x7f54c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.215+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f54c8089440 con 0x7f54dc070d10 2024-03-06T07:29:58.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.215+0000 7f54e0b72640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54c006b500 0x7f54c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.215+0000 7f54e0b72640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54c006b500 0x7f54c006d9c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009af0 tx=0x7f54d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.219+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54c8057d90 con 0x7f54dc070d10 2024-03-06T07:29:58.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:58.391+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54dc155ac0 con 0x7f54dc070d10 2024-03-06T07:29:59.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.079+0000 7f54d97fa640 1 -- 172.21.15.122:0/3104442647 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) v1 ==== 169+0+0 (secure 0 0 0) 0x7f54c805be30 con 0x7f54dc070d10 2024-03-06T07:29:59.081 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:29:59.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54c006b500 msgr2=0x7f54c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54c006b500 0x7f54c006d9c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009af0 tx=0x7f54d0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 msgr2=0x7f54dc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 0x7f54dc06d1a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f54c800c470 tx=0x7f54c8001e70 comp rx=0 tx=0).stop 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 shutdown_connections 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f54c006b500 0x7f54c006d9c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f54dc15ac00 0x7f54dc06d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54dc070d10 0x7f54dc06d1a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 --2- 172.21.15.122:0/3104442647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54dc0708b0 0x7f54dc15b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 >> 172.21.15.122:0/3104442647 conn(0x7f54dc0b1650 msgr2=0x7f54dc0b0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 shutdown_connections 2024-03-06T07:29:59.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.081+0000 7f54e2dd6640 1 -- 172.21.15.122:0/3104442647 wait complete. 2024-03-06T07:29:59.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:29:59.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.203+0000 7f553b65c640 1 Processor -- start 2024-03-06T07:29:59.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.204+0000 7f553b65c640 1 -- start start 2024-03-06T07:29:59.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.204+0000 7f553b65c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 0x7f55340cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.204+0000 7f553b65c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5534158ac0 con 0x7f55340b33a0 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.204+0000 7f553b65c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553415b2a0 con 0x7f55341586e0 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.204+0000 7f553b65c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553415b540 con 0x7f55340cfa90 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f5539bf9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f55340b33a0 0x7f55340b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53202/0 (socket says 172.21.15.122:53202) 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f5539bf9640 1 -- 172.21.15.122:0/2383856312 learned_addr learned my addr 172.21.15.122:0/2383856312 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f5538bf7640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 0x7f55340cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805221785 0 0) 0x7f5534158ac0 con 0x7f55340b33a0 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5520003520 con 0x7f55340b33a0 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270047554 0 0) 0x7f5520003520 con 0x7f55340b33a0 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 msgr2=0x7f55340cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 0x7f55340cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 >> v1:172.21.15.169:6789/0 conn(0x7f55341586e0 legacy=0x7f55340aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553415b820 con 0x7f55340b33a0 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f552c0019f0 con 0x7f55340b33a0 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 >> v1:172.21.15.122:6789/0 conn(0x7f55340b33a0 legacy=0x7f55340b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55200039e0 0x7f5520003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5520004390 0x7f5520004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 0x7f55200050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553415b540 con 0x7f5520004390 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552000b9f0 con 0x7f55200039e0 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552000de00 con 0x7f55340cfa90 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f5538bf7640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5520004390 0x7f5520004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f55393f8640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55200039e0 0x7f5520003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.205+0000 7f5539bf9640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 0x7f55200050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f5538bf7640 1 -- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 msgr2=0x7f55200050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.211 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f5538bf7640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f55340cfa90 0x7f55200050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f5538bf7640 1 -- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55200039e0 msgr2=0x7f5520003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f5538bf7640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55200039e0 0x7f5520003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f5538bf7640 1 -- 172.21.15.122:0/2383856312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520003520 con 0x7f5520004390 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f55393f8640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55200039e0 0x7f5520003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f5538bf7640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5520004390 0x7f5520004860 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5530009ac0 tx=0x7f5520005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb32f493101cd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530039070 con 0x7f5520004390 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553002c6b0 con 0x7f5520004390 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.206+0000 7f552bfff640 1 -- 172.21.15.122:0/2383856312 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530009690 con 0x7f5520004390 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f553b65c640 1 -- 172.21.15.122:0/2383856312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5520004390 msgr2=0x7f5520004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f553b65c640 1 --2- 172.21.15.122:0/2383856312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5520004390 0x7f5520004860 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5530009ac0 tx=0x7f5520005830 comp rx=0 tx=0).stop 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f5539bf9640 1 -- 172.21.15.122:0/2383856312 reap_dead start 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f553b65c640 1 -- 172.21.15.122:0/2383856312 shutdown_connections 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f553b65c640 1 -- 172.21.15.122:0/2383856312 >> 172.21.15.122:0/2383856312 conn(0x7f553406b190 msgr2=0x7f553406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f553b65c640 1 -- 172.21.15.122:0/2383856312 shutdown_connections 2024-03-06T07:29:59.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.207+0000 7f553b65c640 1 -- 172.21.15.122:0/2383856312 wait complete. 2024-03-06T07:29:59.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.208+0000 7f553b65c640 1 Processor -- start 2024-03-06T07:29:59.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.208+0000 7f553b65c640 1 -- start start 2024-03-06T07:29:59.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f553b65c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5520004390 0x7f553406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f553b65c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 0x7f5534144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f553b65c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55341567e0 0x7f5534156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f553b65c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55340af150 con 0x7f55341446d0 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f553b65c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553415bde0 con 0x7f55341567e0 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f553b65c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5534071350 con 0x7f5520004390 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 0x7f5534144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 0x7f5534144b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55266/0 (socket says 172.21.15.122:55266) 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 -- 172.21.15.122:0/286689995 learned_addr learned my addr 172.21.15.122:0/286689995 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f55393f8640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5520004390 0x7f553406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5539bf9640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55341567e0 0x7f5534156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 -- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5520004390 msgr2=0x7f553406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5520004390 0x7f553406d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 -- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55341567e0 msgr2=0x7f5534156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55341567e0 0x7f5534156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5538bf7640 1 -- 172.21.15.122:0/286689995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55341571a0 con 0x7f55341446d0 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.209+0000 7f5539bf9640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55341567e0 0x7f5534156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.210+0000 7f5538bf7640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 0x7f5534144b50 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5530037040 tx=0x7f5530004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.210+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530042020 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.210+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553003a040 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.210+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5534157430 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.210+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530039070 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.210+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55341579f0 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.211+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55340b4820 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.211+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5530045020 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.212+0000 7f5529ffb640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f551806b4d0 0x7f551806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.212+0000 7f55393f8640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f551806b4d0 0x7f551806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.212+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f55300ab4d0 con 0x7f55341446d0 2024-03-06T07:29:59.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.212+0000 7f55393f8640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f551806b4d0 0x7f551806d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5524007fd0 tx=0x7f5524009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.216+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55340b4820 con 0x7f55341446d0 2024-03-06T07:29:59.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.394+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f55340732b0 con 0x7f55341446d0 2024-03-06T07:29:59.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.399+0000 7f5529ffb640 1 -- 172.21.15.122:0/286689995 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f553004c070 con 0x7f55341446d0 2024-03-06T07:29:59.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f551806b4d0 msgr2=0x7f551806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f551806b4d0 0x7f551806d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5524007fd0 tx=0x7f5524009040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 msgr2=0x7f5534144b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 0x7f5534144b50 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5530037040 tx=0x7f5530004730 comp rx=0 tx=0).stop 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 shutdown_connections 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f551806b4d0 0x7f551806d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55341567e0 0x7f5534156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55341446d0 0x7f5534144b50 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 --2- 172.21.15.122:0/286689995 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5520004390 0x7f553406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 >> 172.21.15.122:0/286689995 conn(0x7f553406b190 msgr2=0x7f5534158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 shutdown_connections 2024-03-06T07:29:59.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.401+0000 7f553b65c640 1 -- 172.21.15.122:0/286689995 wait complete. 2024-03-06T07:29:59.414 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:29:59.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.512+0000 7f8be30c9640 1 Processor -- start 2024-03-06T07:29:59.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.512+0000 7f8be30c9640 1 -- start start 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8be30c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 0x7f8bdc15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8be30c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc071f40 con 0x7f8bdc1515e0 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8be30c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc0cd2c0 con 0x7f8bdc0cd7a0 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8be30c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc070a70 con 0x7f8bdc0cce40 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8be0e65640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8bdc1515e0 0x7f8bdc071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53210/0 (socket says 172.21.15.122:53210) 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8be0e65640 1 -- 172.21.15.122:0/1672795184 learned_addr learned my addr 172.21.15.122:0/1672795184 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982745507 0 0) 0x7f8bdc071f40 con 0x7f8bdc1515e0 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc003520 con 0x7f8bdc1515e0 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdbfff640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 0x7f8bdc15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102075328 0 0) 0x7f8bcc003520 con 0x7f8bdc1515e0 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 msgr2=0x7f8bdc15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 0x7f8bdc15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 >> v1:172.21.15.169:6789/0 conn(0x7f8bdc0cd7a0 legacy=0x7f8bdc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc070d50 con 0x7f8bdc1515e0 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f8bd0001670 con 0x7f8bdc1515e0 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 >> v1:172.21.15.122:6789/0 conn(0x7f8bdc1515e0 legacy=0x7f8bdc071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bcc0039e0 0x7f8bcc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc004800 0x7f8bcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 0x7f8bcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc070a70 con 0x7f8bcc0039e0 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc00cbb0 con 0x7f8bcc004800 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.513+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc00cdb0 con 0x7f8bdc0cce40 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bcc0039e0 0x7f8bcc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8bdbfff640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc004800 0x7f8bcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be1666640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 0x7f8bcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 -- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 msgr2=0x7f8bcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc0cce40 0x7f8bcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 -- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc004800 msgr2=0x7f8bcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc004800 0x7f8bcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 -- 172.21.15.122:0/1672795184 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc003520 con 0x7f8bcc0039e0 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8bdbfff640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc004800 0x7f8bcc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.514+0000 7f8be0e65640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bcc0039e0 0x7f8bcc003e00 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0001050 tx=0x7f8bcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb5180bb7a9e63a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0007d10 con 0x7f8bcc0039e0 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd0030070 con 0x7f8bcc0039e0 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8bdb7fe640 1 -- 172.21.15.122:0/1672795184 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0005d30 con 0x7f8bcc0039e0 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be30c9640 1 -- 172.21.15.122:0/1672795184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bcc0039e0 msgr2=0x7f8bcc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be30c9640 1 --2- 172.21.15.122:0/1672795184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bcc0039e0 0x7f8bcc003e00 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0001050 tx=0x7f8bcc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be1666640 1 -- 172.21.15.122:0/1672795184 reap_dead start 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be30c9640 1 -- 172.21.15.122:0/1672795184 shutdown_connections 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be30c9640 1 -- 172.21.15.122:0/1672795184 >> 172.21.15.122:0/1672795184 conn(0x7f8bdc0cf8a0 msgr2=0x7f8bdc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be30c9640 1 -- 172.21.15.122:0/1672795184 shutdown_connections 2024-03-06T07:29:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.515+0000 7f8be30c9640 1 -- 172.21.15.122:0/1672795184 wait complete. 2024-03-06T07:29:59.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.516+0000 7f8be30c9640 1 Processor -- start 2024-03-06T07:29:59.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.516+0000 7f8be30c9640 1 -- start start 2024-03-06T07:29:59.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be30c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc0039e0 0x7f8bdc1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be30c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc071c20 0x7f8bdc0720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be30c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 0x7f8bdc142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be30c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc15b430 con 0x7f8bdc142310 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be30c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc071310 con 0x7f8bcc0039e0 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be30c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc15b5b0 con 0x7f8bdc071c20 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be1666640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 0x7f8bdc142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be1666640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 0x7f8bdc142790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55286/0 (socket says 172.21.15.122:55286) 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be1666640 1 -- 172.21.15.122:0/3615036686 learned_addr learned my addr 172.21.15.122:0/3615036686 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8be0e65640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc0039e0 0x7f8bdc1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.517+0000 7f8bdbfff640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc071c20 0x7f8bdc0720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be1666640 1 -- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc071c20 msgr2=0x7f8bdc0720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be1666640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc071c20 0x7f8bdc0720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be1666640 1 -- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc0039e0 msgr2=0x7f8bdc1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be1666640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc0039e0 0x7f8bdc1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be1666640 1 -- 172.21.15.122:0/3615036686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc142cd0 con 0x7f8bdc142310 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be1666640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 0x7f8bdc142790 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0cc340 tx=0x7f8bd400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4003970 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd4018070 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc159270 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4003d50 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.518+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc159830 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.519+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc068840 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.519+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bd40050d0 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.520+0000 7f8bd97fa640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bc006b4d0 0x7f8bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.520+0000 7f8be0e65640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bc006b4d0 0x7f8bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.520+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8bd4089a20 con 0x7f8bdc142310 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.520+0000 7f8be0e65640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bc006b4d0 0x7f8bc006d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0034000 tx=0x7f8bd0007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.524+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bdc0cd2c0 con 0x7f8bdc142310 2024-03-06T07:29:59.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:29:59.763+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f8bdc04f330 con 0x7f8bdc142310 2024-03-06T07:30:00.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.094+0000 7f8bd97fa640 1 -- 172.21.15.122:0/3615036686 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v435) v1 ==== 140+0+0 (secure 0 0 0) 0x7f8bd4058550 con 0x7f8bdc142310 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bc006b4d0 msgr2=0x7f8bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bc006b4d0 0x7f8bc006d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0034000 tx=0x7f8bd0007000 comp rx=0 tx=0).stop 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 msgr2=0x7f8bdc142790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 0x7f8bdc142790 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0cc340 tx=0x7f8bd400eb90 comp rx=0 tx=0).stop 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 shutdown_connections 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8bc006b4d0 0x7f8bc006d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bdc142310 0x7f8bdc142790 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8bdc071c20 0x7f8bdc0720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 --2- 172.21.15.122:0/3615036686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bcc0039e0 0x7f8bdc1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 >> 172.21.15.122:0/3615036686 conn(0x7f8bdc0cf8a0 msgr2=0x7f8bdc0b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 shutdown_connections 2024-03-06T07:30:00.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.096+0000 7f8be30c9640 1 -- 172.21.15.122:0/3615036686 wait complete. 2024-03-06T07:30:00.109 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ok 2024-03-06T07:30:00.110 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:30:00.110 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:00.110 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:00.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.211+0000 7fc868b65640 1 Processor -- start 2024-03-06T07:30:00.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.211+0000 7fc868b65640 1 -- start start 2024-03-06T07:30:00.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc868b65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc8641424f0 0x7fc8640cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc868b65640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc86406da40 0x7fc86415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc868b65640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640cc980 con 0x7fc8640cd7a0 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc868b65640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640cd220 con 0x7fc8641424f0 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc868b65640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640ccc60 con 0x7fc86406da40 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc86259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc8641424f0 0x7fc8640cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc85bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc8640cd7a0 0x7fc86406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53224/0 (socket says 172.21.15.122:53224) 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc85bfff640 1 -- 172.21.15.122:0/3286162559 learned_addr learned my addr 172.21.15.122:0/3286162559 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc86259c640 1 -- 172.21.15.122:0/3286162559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc86406da40 msgr2=0x7fc86415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc86259c640 1 --2- 172.21.15.122:0/3286162559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc86406da40 0x7fc86415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc86259c640 1 -- 172.21.15.122:0/3286162559 >> v1:172.21.15.122:6789/0 conn(0x7fc8640cd7a0 legacy=0x7fc86406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc86259c640 1 -- 172.21.15.122:0/3286162559 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640ccf40 con 0x7fc8641424f0 2024-03-06T07:30:00.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.212+0000 7fc861d9b640 1 -- 172.21.15.122:0/3286162559 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770520134 0 0) 0x7fc8640cc980 con 0x7fc8640cd7a0 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.213+0000 7fc86259c640 1 --2- 172.21.15.122:0/3286162559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8641424f0 0x7fc8640cc2f0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097a0 tx=0x7fc85402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed53103c9b556c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.213+0000 7fc861d9b640 1 -- 172.21.15.122:0/3286162559 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854035020 con 0x7fc8641424f0 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.213+0000 7fc861d9b640 1 -- 172.21.15.122:0/3286162559 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854031830 con 0x7fc8641424f0 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.213+0000 7fc861d9b640 1 -- 172.21.15.122:0/3286162559 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854031c00 con 0x7fc8641424f0 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 -- 172.21.15.122:0/3286162559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8641424f0 msgr2=0x7fc8640cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 --2- 172.21.15.122:0/3286162559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8641424f0 0x7fc8640cc2f0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097a0 tx=0x7fc85402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 -- 172.21.15.122:0/3286162559 shutdown_connections 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 --2- 172.21.15.122:0/3286162559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc86406da40 0x7fc86415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 --2- 172.21.15.122:0/3286162559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8641424f0 0x7fc8640cc2f0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 -- 172.21.15.122:0/3286162559 >> 172.21.15.122:0/3286162559 conn(0x7fc8640cf8a0 msgr2=0x7fc8640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 -- 172.21.15.122:0/3286162559 shutdown_connections 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.214+0000 7fc868b65640 1 -- 172.21.15.122:0/3286162559 wait complete. 2024-03-06T07:30:00.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.215+0000 7fc868b65640 1 Processor -- start 2024-03-06T07:30:00.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.215+0000 7fc868b65640 1 -- start start 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc86406da40 0x7fc86414f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 0x7fc8640aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8641424f0 0x7fc8640abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864071ce0 con 0x7fc8640cd7a0 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864071b60 con 0x7fc86406da40 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864071e60 con 0x7fc8641424f0 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 0x7fc8640aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 0x7fc8640aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55306/0 (socket says 172.21.15.122:55306) 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 -- 172.21.15.122:0/3960754541 learned_addr learned my addr 172.21.15.122:0/3960754541 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc86259c640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc86406da40 0x7fc86414f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc862d9d640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8641424f0 0x7fc8640abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 -- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8641424f0 msgr2=0x7fc8640abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8641424f0 0x7fc8640abdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 -- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc86406da40 msgr2=0x7fc86414f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc86406da40 0x7fc86414f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 -- 172.21.15.122:0/3960754541 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86414f6a0 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc86259c640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc86406da40 0x7fc86414f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc862d9d640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8641424f0 0x7fc8640abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85bfff640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 0x7fc8640aa510 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c4d0 tx=0x7fc84c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c018070 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c007500 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8640aae10 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c004a20 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.216+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8640ab3d0 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.217+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86404f330 con 0x7fc8640cd7a0 2024-03-06T07:30:00.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.218+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc84c01b070 con 0x7fc8640cd7a0 2024-03-06T07:30:00.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.218+0000 7fc85affd640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc84406b4d0 0x7fc84406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.218+0000 7fc86259c640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc84406b4d0 0x7fc84406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.218+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc84c089200 con 0x7fc8640cd7a0 2024-03-06T07:30:00.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.219+0000 7fc86259c640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc84406b4d0 0x7fc84406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc854036000 tx=0x7fc854008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.223+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc84c058380 con 0x7fc8640cd7a0 2024-03-06T07:30:00.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.396+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8640d0620 con 0x7fc8640cd7a0 2024-03-06T07:30:00.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.397+0000 7fc85affd640 1 -- 172.21.15.122:0/3960754541 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+13032 (secure 0 0 0) 0x7fc84c057c80 con 0x7fc8640cd7a0 2024-03-06T07:30:00.399 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:00.399 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":133,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:29:59.078450+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6837/2094222247":"2024-03-07T07:29:55.914224+0000","172.21.15.122:6835/2094222247":"2024-03-07T07:29:55.914224+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc84406b4d0 msgr2=0x7fc84406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc84406b4d0 0x7fc84406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc854036000 tx=0x7fc854008030 comp rx=0 tx=0).stop 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 msgr2=0x7fc8640aa510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 0x7fc8640aa510 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c4d0 tx=0x7fc84c00c990 comp rx=0 tx=0).stop 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 shutdown_connections 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc84406b4d0 0x7fc84406d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc8641424f0 0x7fc8640abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8640cd7a0 0x7fc8640aa510 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.399+0000 7fc868b65640 1 --2- 172.21.15.122:0/3960754541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc86406da40 0x7fc86414f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.400+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 >> 172.21.15.122:0/3960754541 conn(0x7fc8640cf8a0 msgr2=0x7fc8640ce280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.400+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 shutdown_connections 2024-03-06T07:30:00.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.400+0000 7fc868b65640 1 -- 172.21.15.122:0/3960754541 wait complete. 2024-03-06T07:30:00.413 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:00.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:00.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.514+0000 7f43967d7640 1 Processor -- start 2024-03-06T07:30:00.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.515+0000 7f43967d7640 1 -- start start 2024-03-06T07:30:00.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.515+0000 7f43967d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 0x7f43900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.515+0000 7f43967d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43901618b0 con 0x7f43900d2480 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.515+0000 7f43967d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900cf400 con 0x7f43900d1910 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.515+0000 7f43967d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43900cf6a0 con 0x7f43901614d0 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f4394d74640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f43900d2480 0x7f43900b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53238/0 (socket says 172.21.15.122:53238) 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f4394d74640 1 -- 172.21.15.122:0/869002712 learned_addr learned my addr 172.21.15.122:0/869002712 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438ffff640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 0x7f43900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082056559 0 0) 0x7f43901618b0 con 0x7f43900d2480 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4378003520 con 0x7f43900d2480 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307067584 0 0) 0x7f43900cf400 con 0x7f43900d1910 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43901618b0 con 0x7f43900d1910 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1191194549 0 0) 0x7f4378003520 con 0x7f43900d2480 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 msgr2=0x7f43900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 0x7f43900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 >> v1:172.21.15.169:6789/0 conn(0x7f43900d1910 legacy=0x7f43900d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900cf980 con 0x7f43900d2480 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f43880019f0 con 0x7f43900d2480 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 >> v1:172.21.15.122:6789/0 conn(0x7f43900d2480 legacy=0x7f43900b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4378003b70 0x7f4378003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 0x7f4378004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4378004d40 0x7f43780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900cf6a0 con 0x7f4378004d40 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437800ba70 con 0x7f4378003b70 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437800bc20 con 0x7f43901614d0 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f4394d74640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4378004d40 0x7f43780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438ffff640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4378003b70 0x7f4378003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.516+0000 7f438f7fe640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 0x7f4378004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f4394d74640 1 -- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 msgr2=0x7f4378004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f4394d74640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f43901614d0 0x7f4378004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f4394d74640 1 -- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4378003b70 msgr2=0x7f4378003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f4394d74640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4378003b70 0x7f4378003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f4394d74640 1 -- 172.21.15.122:0/869002712 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378003520 con 0x7f4378004d40 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f4394d74640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4378004d40 0x7f43780051c0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f438800a7f0 tx=0x7f4378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9317717c83dbe5e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438800a450 con 0x7f4378004d40 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43880072d0 con 0x7f4378004d40 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.517+0000 7f438effd640 1 -- 172.21.15.122:0/869002712 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4388007700 con 0x7f4378004d40 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f43967d7640 1 -- 172.21.15.122:0/869002712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4378004d40 msgr2=0x7f43780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f43967d7640 1 --2- 172.21.15.122:0/869002712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4378004d40 0x7f43780051c0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f438800a7f0 tx=0x7f4378005830 comp rx=0 tx=0).stop 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f4394d74640 1 -- 172.21.15.122:0/869002712 reap_dead start 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f43967d7640 1 -- 172.21.15.122:0/869002712 shutdown_connections 2024-03-06T07:30:00.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f43967d7640 1 -- 172.21.15.122:0/869002712 >> 172.21.15.122:0/869002712 conn(0x7f43900d3d40 msgr2=0x7f43900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f43967d7640 1 -- 172.21.15.122:0/869002712 shutdown_connections 2024-03-06T07:30:00.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.518+0000 7f43967d7640 1 -- 172.21.15.122:0/869002712 wait complete. 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.519+0000 7f43967d7640 1 Processor -- start 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.519+0000 7f43967d7640 1 -- start start 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f43967d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4390148980 0x7f4390146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f43967d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 0x7f439014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f43967d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4390148d60 0x7f43901491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f43967d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900d4860 con 0x7f439014be10 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f43967d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900b0910 con 0x7f4390148d60 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f43967d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43900d49e0 con 0x7f4390148980 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 0x7f439014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 0x7f439014c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55326/0 (socket says 172.21.15.122:55326) 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 -- 172.21.15.122:0/3506979207 learned_addr learned my addr 172.21.15.122:0/3506979207 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438ffff640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4390148980 0x7f4390146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f4394d74640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4390148d60 0x7f43901491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 -- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4390148980 msgr2=0x7f4390146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4390148980 0x7f4390146920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 -- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4390148d60 msgr2=0x7f43901491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4390148d60 0x7f43901491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 -- 172.21.15.122:0/3506979207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439014c7b0 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.520+0000 7f438f7fe640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 0x7f439014c270 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f438400f7a0 tx=0x7f438400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.521+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400fe50 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.521+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4384018070 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.521+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43901f40e0 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.521+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400d6d0 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.521+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43900cb4e0 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.522+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43900680d0 con 0x7f439014be10 2024-03-06T07:30:00.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.522+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f438400d870 con 0x7f439014be10 2024-03-06T07:30:00.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.522+0000 7f438cff9640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f437c06b4d0 0x7f437c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.522+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4384089e10 con 0x7f439014be10 2024-03-06T07:30:00.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.522+0000 7f438ffff640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f437c06b4d0 0x7f437c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.523+0000 7f438ffff640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f437c06b4d0 0x7f437c06d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4380009b00 tx=0x7f4380008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.527+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43900680d0 con 0x7f439014be10 2024-03-06T07:30:00.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.698+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4390068ff0 con 0x7f439014be10 2024-03-06T07:30:00.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.698+0000 7f438cff9640 1 -- 172.21.15.122:0/3506979207 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) v1 ==== 93+0+10 (secure 0 0 0) 0x7f438401a310 con 0x7f439014be10 2024-03-06T07:30:00.700 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:30:00.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f437c06b4d0 msgr2=0x7f437c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f437c06b4d0 0x7f437c06d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4380009b00 tx=0x7f4380008040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 msgr2=0x7f439014c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 0x7f439014c270 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f438400f7a0 tx=0x7f438400fc60 comp rx=0 tx=0).stop 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 shutdown_connections 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f437c06b4d0 0x7f437c06d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4390148d60 0x7f43901491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f439014be10 0x7f439014c270 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 --2- 172.21.15.122:0/3506979207 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4390148980 0x7f4390146920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 >> 172.21.15.122:0/3506979207 conn(0x7f43900d3d40 msgr2=0x7f43900b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 shutdown_connections 2024-03-06T07:30:00.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.700+0000 7f43967d7640 1 -- 172.21.15.122:0/3506979207 wait complete. 2024-03-06T07:30:00.713 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:30:00.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.811+0000 7fbfd331e640 1 Processor -- start 2024-03-06T07:30:00.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.811+0000 7fbfd331e640 1 -- start start 2024-03-06T07:30:00.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd331e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfcc13dc10 0x7fbfcc0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd331e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc15ef60 0x7fbfcc0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd331e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfcc15f580 con 0x7fbfcc15bf30 2024-03-06T07:30:00.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd331e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfcc0cf3f0 con 0x7fbfcc13dc10 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd331e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc15b210 con 0x7fbfcc15ef60 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfcc13dc10 0x7fbfcc0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfcc13dc10 0x7fbfcc0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55336/0 (socket says 172.21.15.122:55336) 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 -- 172.21.15.122:0/1383422504 learned_addr learned my addr 172.21.15.122:0/1383422504 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd18bb640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc15ef60 0x7fbfcc0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 -- 172.21.15.122:0/1383422504 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc15ef60 msgr2=0x7fbfcc0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc15ef60 0x7fbfcc0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 -- 172.21.15.122:0/1383422504 >> v1:172.21.15.169:6789/0 conn(0x7fbfcc15bf30 legacy=0x7fbfcc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd10ba640 1 -- 172.21.15.122:0/1383422504 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc15b450 con 0x7fbfcc13dc10 2024-03-06T07:30:00.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.812+0000 7fbfd18bb640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc15ef60 0x7fbfcc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd10ba640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc13dc10 0x7fbfcc0b2950 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400bb10 tx=0x7fbfc402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529a4ee5d6fbdbd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfcbfff640 1 -- 172.21.15.122:0/1383422504 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc403b070 con 0x7fbfcc13dc10 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfcbfff640 1 -- 172.21.15.122:0/1383422504 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc402ede0 con 0x7fbfcc13dc10 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfcbfff640 1 -- 172.21.15.122:0/1383422504 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4036660 con 0x7fbfcc13dc10 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd331e640 1 -- 172.21.15.122:0/1383422504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc13dc10 msgr2=0x7fbfcc0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc13dc10 0x7fbfcc0b2950 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400bb10 tx=0x7fbfc402e030 comp rx=0 tx=0).stop 2024-03-06T07:30:00.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd331e640 1 -- 172.21.15.122:0/1383422504 shutdown_connections 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc15ef60 0x7fbfcc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1383422504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc13dc10 0x7fbfcc0b2950 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.813+0000 7fbfd331e640 1 -- 172.21.15.122:0/1383422504 >> 172.21.15.122:0/1383422504 conn(0x7fbfcc0cf8a0 msgr2=0x7fbfcc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.814+0000 7fbfd331e640 1 -- 172.21.15.122:0/1383422504 shutdown_connections 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.814+0000 7fbfd331e640 1 -- 172.21.15.122:0/1383422504 wait complete. 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 Processor -- start 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 -- start start 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc13dc10 0x7fbfcc06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 0x7fbfcc140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfcc15ef60 0x7fbfcc15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc0cd7a0 con 0x7fbfcc15bf30 2024-03-06T07:30:00.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc13dff0 con 0x7fbfcc15ef60 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd331e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc0cd920 con 0x7fbfcc13dc10 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd08b9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 0x7fbfcc140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd08b9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 0x7fbfcc140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55350/0 (socket says 172.21.15.122:55350) 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.815+0000 7fbfd08b9640 1 -- 172.21.15.122:0/1192113489 learned_addr learned my addr 172.21.15.122:0/1192113489 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd10ba640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc13dc10 0x7fbfcc06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd18bb640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfcc15ef60 0x7fbfcc15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd08b9640 1 -- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc13dc10 msgr2=0x7fbfcc06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd08b9640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc13dc10 0x7fbfcc06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd08b9640 1 -- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfcc15ef60 msgr2=0x7fbfcc15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd08b9640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfcc15ef60 0x7fbfcc15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd08b9640 1 -- 172.21.15.122:0/1192113489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc13aa60 con 0x7fbfcc15bf30 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd18bb640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfcc15ef60 0x7fbfcc15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd08b9640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 0x7fbfcc140880 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000c460 tx=0x7fbfc000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0016020 con 0x7fbfcc15bf30 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0012870 con 0x7fbfcc15bf30 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfcc13fda0 con 0x7fbfcc15bf30 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0012c60 con 0x7fbfcc15bf30 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.816+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfcc140360 con 0x7fbfcc15bf30 2024-03-06T07:30:00.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.817+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfcc04f330 con 0x7fbfcc15bf30 2024-03-06T07:30:00.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.818+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfc0019020 con 0x7fbfcc15bf30 2024-03-06T07:30:00.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.818+0000 7fbfc9ffb640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfb806b4d0 0x7fbfb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.818+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbfc0089af0 con 0x7fbfcc15bf30 2024-03-06T07:30:00.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.818+0000 7fbfd10ba640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfb806b4d0 0x7fbfb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.818+0000 7fbfd10ba640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfb806b4d0 0x7fbfb806d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4038000 tx=0x7fbfc4032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:00.822+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfc0020030 con 0x7fbfcc15bf30 2024-03-06T07:30:01.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.061+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7fbfcc06eb00 con 0x7fbfcc15bf30 2024-03-06T07:30:01.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.095+0000 7fbfc9ffb640 1 -- 172.21.15.122:0/1192113489 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v436) v1 ==== 144+0+0 (secure 0 0 0) 0x7fbfc0058d00 con 0x7fbfcc15bf30 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfb806b4d0 msgr2=0x7fbfb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfb806b4d0 0x7fbfb806d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4038000 tx=0x7fbfc4032000 comp rx=0 tx=0).stop 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 msgr2=0x7fbfcc140880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 0x7fbfcc140880 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000c460 tx=0x7fbfc000c920 comp rx=0 tx=0).stop 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 shutdown_connections 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfb806b4d0 0x7fbfb806d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfcc15ef60 0x7fbfcc15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfcc15bf30 0x7fbfcc140880 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 --2- 172.21.15.122:0/1192113489 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfcc13dc10 0x7fbfcc06e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 >> 172.21.15.122:0/1192113489 conn(0x7fbfcc0cf8a0 msgr2=0x7fbfcc0b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 shutdown_connections 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.097+0000 7fbfd331e640 1 -- 172.21.15.122:0/1192113489 wait complete. 2024-03-06T07:30:01.110 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:01.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.211+0000 7f4d7bc9a640 1 Processor -- start 2024-03-06T07:30:01.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 -- start start 2024-03-06T07:30:01.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d740aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4d740aef80 0x7f4d740cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4d740cffd0 0x7f4d7406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d740d03f0 con 0x7f4d740cffd0 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d74158ac0 con 0x7f4d740aef80 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7bc9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7415b2a0 con 0x7f4d741586e0 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.212+0000 7f4d7a237640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4d740cffd0 0x7f4d7406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4d740cffd0 0x7f4d7406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55362/0 (socket says 172.21.15.122:55362) 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 -- 172.21.15.122:0/4089963999 learned_addr learned my addr 172.21.15.122:0/4089963999 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d79235640 1 --2- 172.21.15.122:0/4089963999 >> v2:172.21.15.169:3300/0 conn(0x7f4d740aef80 0x7f4d740cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d79a36640 1 --2- 172.21.15.122:0/4089963999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d740aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 -- 172.21.15.122:0/4089963999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 msgr2=0x7f4d740aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 --2- 172.21.15.122:0/4089963999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d740aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 -- 172.21.15.122:0/4089963999 >> v2:172.21.15.169:3300/0 conn(0x7f4d740aef80 msgr2=0x7f4d740cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 --2- 172.21.15.122:0/4089963999 >> v2:172.21.15.169:3300/0 conn(0x7f4d740aef80 0x7f4d740cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 -- 172.21.15.122:0/4089963999 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7415b540 con 0x7f4d740cffd0 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d79235640 1 --2- 172.21.15.122:0/4089963999 >> v2:172.21.15.169:3300/0 conn(0x7f4d740aef80 0x7f4d740cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.213+0000 7f4d7a237640 1 --2- 172.21.15.122:0/4089963999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740cffd0 0x7f4d7406da20 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f4d740d0610 tx=0x7f4d6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a454e47ef18cb98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d78a34640 1 -- 172.21.15.122:0/4089963999 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c035020 con 0x7f4d740cffd0 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d78a34640 1 -- 172.21.15.122:0/4089963999 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6c031870 con 0x7f4d740cffd0 2024-03-06T07:30:01.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d78a34640 1 -- 172.21.15.122:0/4089963999 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c031c40 con 0x7f4d740cffd0 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/4089963999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740cffd0 msgr2=0x7f4d7406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/4089963999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740cffd0 0x7f4d7406da20 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f4d740d0610 tx=0x7f4d6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/4089963999 shutdown_connections 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/4089963999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740cffd0 0x7f4d7406da20 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/4089963999 >> v2:172.21.15.169:3300/0 conn(0x7f4d740aef80 0x7f4d740cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/4089963999 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d740aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/4089963999 >> 172.21.15.122:0/4089963999 conn(0x7f4d7406b190 msgr2=0x7f4d7406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/4089963999 shutdown_connections 2024-03-06T07:30:01.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.214+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/4089963999 wait complete. 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.215+0000 7f4d7bc9a640 1 Processor -- start 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.215+0000 7f4d7bc9a640 1 -- start start 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7bc9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 0x7f4d740cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7bc9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d740cffd0 0x7f4d74152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7bc9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d74144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7bc9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7415bde0 con 0x7f4d740aef80 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7bc9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7415bb00 con 0x7f4d740cffd0 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7bc9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d74071350 con 0x7f4d741586e0 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d79a36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 0x7f4d740cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d79a36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 0x7f4d740cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55364/0 (socket says 172.21.15.122:55364) 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d79a36640 1 -- 172.21.15.122:0/726240841 learned_addr learned my addr 172.21.15.122:0/726240841 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d79235640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d740cffd0 0x7f4d74152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d7a237640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d74144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.216+0000 7f4d79a36640 1 -- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 msgr2=0x7f4d74144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d79a36640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d74144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d79a36640 1 -- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d740cffd0 msgr2=0x7f4d74152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d79a36640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d740cffd0 0x7f4d74152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d79a36640 1 -- 172.21.15.122:0/726240841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d74153130 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d7a237640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d74144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d79a36640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 0x7f4d740cd500 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f4d70007e70 tx=0x7f4d7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70014620 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70019070 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70014a00 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d740cdc00 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.217+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d740ce1c0 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.218+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d740732b0 con 0x7f4d740aef80 2024-03-06T07:30:01.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.218+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d70014ba0 con 0x7f4d740aef80 2024-03-06T07:30:01.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.219+0000 7f4d6a7fc640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d5806b4d0 0x7f4d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.219+0000 7f4d79235640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d5806b4d0 0x7f4d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.219+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4d7008a620 con 0x7f4d740aef80 2024-03-06T07:30:01.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.219+0000 7f4d79235640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d5806b4d0 0x7f4d5806d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4d741580f0 tx=0x7f4d64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.223+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d70059830 con 0x7f4d740aef80 2024-03-06T07:30:01.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.394+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4d740b28c0 con 0x7f4d740aef80 2024-03-06T07:30:01.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.397+0000 7f4d6a7fc640 1 -- 172.21.15.122:0/726240841 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4d70059130 con 0x7f4d740aef80 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d5806b4d0 msgr2=0x7f4d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d5806b4d0 0x7f4d5806d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4d741580f0 tx=0x7f4d64008040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 msgr2=0x7f4d740cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 0x7f4d740cd500 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f4d70007e70 tx=0x7f4d7000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 shutdown_connections 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d5806b4d0 0x7f4d5806d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d741586e0 0x7f4d74144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d740cffd0 0x7f4d74152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 --2- 172.21.15.122:0/726240841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d740aef80 0x7f4d740cd500 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.399+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 >> 172.21.15.122:0/726240841 conn(0x7f4d7406b190 msgr2=0x7f4d7406d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.400+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 shutdown_connections 2024-03-06T07:30:01.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.400+0000 7f4d7bc9a640 1 -- 172.21.15.122:0/726240841 wait complete. 2024-03-06T07:30:01.413 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:01.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:01.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:01.529 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:01.529 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:30:01.529 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:01.544 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:01.544 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:30:01.544 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:01.561 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:01.562 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:30:01.562 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:01.602 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:01.602 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:30:01.602 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:01.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:01.619 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:01.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.741+0000 7f3fa825c640 1 Processor -- start 2024-03-06T07:30:01.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.741+0000 7f3fa825c640 1 -- start start 2024-03-06T07:30:01.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa825c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa825c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3fa015ef60 0x7f3fa00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa825c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa015f580 con 0x7f3fa015bf30 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa825c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa00cf3f0 con 0x7f3fa015ef60 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa825c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa015b210 con 0x7f3fa013dc10 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3fa015ef60 0x7f3fa00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3fa015ef60 0x7f3fa00b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55378/0 (socket says 172.21.15.122:55378) 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 -- 172.21.15.122:0/572031967 learned_addr learned my addr 172.21.15.122:0/572031967 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 -- 172.21.15.122:0/572031967 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 msgr2=0x7f3fa00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 --2- 172.21.15.122:0/572031967 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 -- 172.21.15.122:0/572031967 >> v1:172.21.15.169:6789/0 conn(0x7f3fa015bf30 legacy=0x7f3fa015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.742+0000 7f3fa67f9640 1 -- 172.21.15.122:0/572031967 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa015b450 con 0x7f3fa015ef60 2024-03-06T07:30:01.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.743+0000 7f3fa67f9640 1 --2- 172.21.15.122:0/572031967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa00b2e90 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009e50 tx=0x7f3f9802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb3c6ae00a2d65d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.743+0000 7f3fa4ff6640 1 -- 172.21.15.122:0/572031967 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9802c9e0 con 0x7f3fa015ef60 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.743+0000 7f3fa4ff6640 1 -- 172.21.15.122:0/572031967 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9802cb80 con 0x7f3fa015ef60 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.743+0000 7f3fa4ff6640 1 -- 172.21.15.122:0/572031967 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f980343f0 con 0x7f3fa015ef60 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.743+0000 7f3fa825c640 1 -- 172.21.15.122:0/572031967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 msgr2=0x7f3fa00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.743+0000 7f3fa825c640 1 --2- 172.21.15.122:0/572031967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa00b2e90 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009e50 tx=0x7f3f9802be40 comp rx=0 tx=0).stop 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.744+0000 7f3fa825c640 1 -- 172.21.15.122:0/572031967 shutdown_connections 2024-03-06T07:30:01.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.744+0000 7f3fa825c640 1 --2- 172.21.15.122:0/572031967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa00b2e90 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.744+0000 7f3fa825c640 1 --2- 172.21.15.122:0/572031967 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.744+0000 7f3fa825c640 1 -- 172.21.15.122:0/572031967 >> 172.21.15.122:0/572031967 conn(0x7f3fa00cf8a0 msgr2=0x7f3fa00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.744+0000 7f3fa825c640 1 -- 172.21.15.122:0/572031967 shutdown_connections 2024-03-06T07:30:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.744+0000 7f3fa825c640 1 -- 172.21.15.122:0/572031967 wait complete. 2024-03-06T07:30:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 Processor -- start 2024-03-06T07:30:01.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 -- start start 2024-03-06T07:30:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa015bf30 0x7f3fa00aa340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00708d0 con 0x7f3fa015ef60 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa013dff0 con 0x7f3fa015bf30 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.745+0000 7f3fa825c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa0070a50 con 0x7f3fa013dc10 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa006d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55388/0 (socket says 172.21.15.122:55388) 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 -- 172.21.15.122:0/3144082813 learned_addr learned my addr 172.21.15.122:0/3144082813 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa5ff8640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa57f7640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa015bf30 0x7f3fa00aa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 -- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 msgr2=0x7f3fa00aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 -- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa015bf30 msgr2=0x7f3fa00aa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa015bf30 0x7f3fa00aa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 -- 172.21.15.122:0/3144082813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00aa880 con 0x7f3fa015ef60 2024-03-06T07:30:01.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa57f7640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa015bf30 0x7f3fa00aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa67f9640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa006d850 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009e50 tx=0x7f3f98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98043020 con 0x7f3fa015ef60 2024-03-06T07:30:01.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9803e450 con 0x7f3fa015ef60 2024-03-06T07:30:01.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98009510 con 0x7f3fa015ef60 2024-03-06T07:30:01.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa00aab50 con 0x7f3fa015ef60 2024-03-06T07:30:01.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.746+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa00ab110 con 0x7f3fa015ef60 2024-03-06T07:30:01.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.747+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa004f330 con 0x7f3fa015ef60 2024-03-06T07:30:01.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.748+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f98005d10 con 0x7f3fa015ef60 2024-03-06T07:30:01.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.748+0000 7f3f967fc640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.748+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3f980aca30 con 0x7f3fa015ef60 2024-03-06T07:30:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.748+0000 7f3fa5ff8640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.748+0000 7f3fa5ff8640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f8406b4d0 0x7f3f8406d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009990 tx=0x7f3f9c009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.752+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f9807bbb0 con 0x7f3fa015ef60 2024-03-06T07:30:01.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.959+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fa004f330 con 0x7f3fa015ef60 2024-03-06T07:30:01.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.959+0000 7f3f967fc640 1 -- 172.21.15.122:0/3144082813 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2120 (secure 0 0 0) 0x7f3f9807b4b0 con 0x7f3fa015ef60 2024-03-06T07:30:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:01.962 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":85,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7196,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2610392853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6835","nonce":2610392853},{"type":"v1","addr":"172.21.15.122:6837","nonce":2610392853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6836/696408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":696408149},{"type":"v1","addr":"172.21.15.122:6836","nonce":696408149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[]} 2024-03-06T07:30:01.962 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 85 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f8406b4d0 msgr2=0x7f3f8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f8406b4d0 0x7f3f8406d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009990 tx=0x7f3f9c009110 comp rx=0 tx=0).stop 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 msgr2=0x7f3fa006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa006d850 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009e50 tx=0x7f3f98004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 shutdown_connections 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fa015ef60 0x7f3fa006d850 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fa015bf30 0x7f3fa00aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 --2- 172.21.15.122:0/3144082813 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3fa013dc10 0x7f3fa00aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 >> 172.21.15.122:0/3144082813 conn(0x7f3fa00cf8a0 msgr2=0x7f3fa00b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 shutdown_connections 2024-03-06T07:30:01.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:01.961+0000 7f3fa825c640 1 -- 172.21.15.122:0/3144082813 wait complete. 2024-03-06T07:30:01.975 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:01.975 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:01.975 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:01.975 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:01.975 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:01.975 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:01.975 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:01.976 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:01.976 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:01.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:01.976 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:01.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:01.979 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:01.979 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:01.980 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:01.980 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:01.986 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:01.986 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:01.988 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:01.988 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:02.029 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:02.029 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:02.030 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:02.030 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:02.030 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:02.066 INFO:tasks.ceph.mds.b.smithi169.stdout:starting mds.b at 2024-03-06T07:30:02.066 INFO:tasks.ceph.mds.d.smithi169.stdout:starting mds.d at 2024-03-06T07:30:02.107 INFO:tasks.ceph.mds.c.smithi122.stdout:starting mds.c at 2024-03-06T07:30:02.107 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:30:02.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.134+0000 7fd70b54b640 1 Processor -- start 2024-03-06T07:30:02.136 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.134+0000 7fd70b54b640 1 -- start start 2024-03-06T07:30:02.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd70b54b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd70415e010 0x7fd7040cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd70b54b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7040b3240 0x7fd7040b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd70b54b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd7040b2910 0x7fd7040d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd70b54b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7040b1f40 con 0x7fd7040b2910 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd70b54b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70415e3f0 con 0x7fd70415e010 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd70b54b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7040b1c10 con 0x7fd7040b3240 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd7040b2910 0x7fd7040d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd7040b2910 0x7fd7040d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55426/0 (socket says 172.21.15.122:55426) 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 -- 172.21.15.122:0/2446837594 learned_addr learned my addr 172.21.15.122:0/2446837594 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd7092e7640 1 --2- 172.21.15.122:0/2446837594 >> v2:172.21.15.169:3300/0 conn(0x7fd70415e010 0x7fd7040cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd708ae6640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7040b3240 0x7fd7040b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 -- 172.21.15.122:0/2446837594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7040b3240 msgr2=0x7fd7040b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7040b3240 0x7fd7040b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 -- 172.21.15.122:0/2446837594 >> v2:172.21.15.169:3300/0 conn(0x7fd70415e010 msgr2=0x7fd7040cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 --2- 172.21.15.122:0/2446837594 >> v2:172.21.15.169:3300/0 conn(0x7fd70415e010 0x7fd7040cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd709ae8640 1 -- 172.21.15.122:0/2446837594 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70406d1b0 con 0x7fd7040b2910 2024-03-06T07:30:02.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd7092e7640 1 --2- 172.21.15.122:0/2446837594 >> v2:172.21.15.169:3300/0 conn(0x7fd70415e010 0x7fd7040cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.135+0000 7fd708ae6640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7040b3240 0x7fd7040b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.136+0000 7fd709ae8640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b2910 0x7fd7040d1ad0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009e50 tx=0x7fd6fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f42060c324714b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.136+0000 7fd6fbfff640 1 -- 172.21.15.122:0/2446837594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc02c9e0 con 0x7fd7040b2910 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.136+0000 7fd6fbfff640 1 -- 172.21.15.122:0/2446837594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc02cb80 con 0x7fd7040b2910 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.136+0000 7fd6fbfff640 1 -- 172.21.15.122:0/2446837594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0343f0 con 0x7fd7040b2910 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 -- 172.21.15.122:0/2446837594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b2910 msgr2=0x7fd7040d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b2910 0x7fd7040d1ad0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009e50 tx=0x7fd6fc02be40 comp rx=0 tx=0).stop 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 -- 172.21.15.122:0/2446837594 shutdown_connections 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b2910 0x7fd7040d1ad0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2446837594 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd7040b3240 0x7fd7040b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2446837594 >> v2:172.21.15.169:3300/0 conn(0x7fd70415e010 0x7fd7040cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 -- 172.21.15.122:0/2446837594 >> 172.21.15.122:0/2446837594 conn(0x7fd70409a9f0 msgr2=0x7fd70409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 -- 172.21.15.122:0/2446837594 shutdown_connections 2024-03-06T07:30:02.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.137+0000 7fd70b54b640 1 -- 172.21.15.122:0/2446837594 wait complete. 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.138+0000 7fd70b54b640 1 Processor -- start 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.138+0000 7fd70b54b640 1 -- start start 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd70415e010 0x7fd7041573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 0x7fd7040b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd704157b80 0x7fd704158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7040cf310 con 0x7fd7040b6660 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70406ced0 con 0x7fd70415e010 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7040d2010 con 0x7fd704157b80 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 0x7fd7040b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 0x7fd7040b6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55430/0 (socket says 172.21.15.122:55430) 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 -- 172.21.15.122:0/2805631411 learned_addr learned my addr 172.21.15.122:0/2805631411 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:02.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd709ae8640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd704157b80 0x7fd704158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd7092e7640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd70415e010 0x7fd7041573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 -- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd704157b80 msgr2=0x7fd704158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd704157b80 0x7fd704158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 -- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd70415e010 msgr2=0x7fd7041573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd70415e010 0x7fd7041573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 -- 172.21.15.122:0/2805631411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7040b7100 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd708ae6640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 0x7fd7040b6ae0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400c820 tx=0x7fd6f400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4015070 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f4007a30 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7040c61e0 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.139+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4007e00 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.140+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7040c67a0 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.141+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70406d990 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.141+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6f4018070 con 0x7fd7040b6660 2024-03-06T07:30:02.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.141+0000 7fd6f9ffb640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e806b500 0x7fd6e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.141+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd6f4089d50 con 0x7fd7040b6660 2024-03-06T07:30:02.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.141+0000 7fd7092e7640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e806b500 0x7fd6e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.142+0000 7fd7092e7640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e806b500 0x7fd6e806d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd70000a950 tx=0x7fd700008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.146+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd70406d990 con 0x7fd7040b6660 2024-03-06T07:30:02.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.331+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd7040d2980 con 0x7fd7040b6660 2024-03-06T07:30:02.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.408+0000 7fd6f9ffb640 1 -- 172.21.15.122:0/2805631411 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd6f408d030 con 0x7fd7040b6660 2024-03-06T07:30:02.410 INFO:teuthology.orchestra.run.smithi122.stderr: removed all blocklist entries 2024-03-06T07:30:02.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e806b500 msgr2=0x7fd6e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e806b500 0x7fd6e806d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd70000a950 tx=0x7fd700008040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 msgr2=0x7fd7040b6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 0x7fd7040b6ae0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400c820 tx=0x7fd6f400cce0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 shutdown_connections 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6e806b500 0x7fd6e806d9c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd704157b80 0x7fd704158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7040b6660 0x7fd7040b6ae0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 --2- 172.21.15.122:0/2805631411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd70415e010 0x7fd7041573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 >> 172.21.15.122:0/2805631411 conn(0x7fd70409a9f0 msgr2=0x7fd7040d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 shutdown_connections 2024-03-06T07:30:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.411+0000 7fd70b54b640 1 -- 172.21.15.122:0/2805631411 wait complete. 2024-03-06T07:30:02.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:02.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 Processor -- start 2024-03-06T07:30:02.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 -- start start 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f919c15dfe0 0x7f919c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0cf830 0x7f919c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c0b3430 con 0x7f919c0d27f0 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919c0d2c50 con 0x7f919c15dfe0 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.523+0000 7f91a265c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c0b31e0 con 0x7f919c0cf830 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f919c15dfe0 0x7f919c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f919c15dfe0 0x7f919c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55432/0 (socket says 172.21.15.122:55432) 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 -- 172.21.15.122:0/1182565325 learned_addr learned my addr 172.21.15.122:0/1182565325 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f91a0bf9640 1 --1- 172.21.15.122:0/1182565325 >> v1:172.21.15.122:6789/0 conn(0x7f919c0d27f0 0x7f919c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53260/0 (socket says 172.21.15.122:53260) 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919b7fe640 1 --2- 172.21.15.122:0/1182565325 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0cf830 0x7f919c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 -- 172.21.15.122:0/1182565325 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0cf830 msgr2=0x7f919c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 --2- 172.21.15.122:0/1182565325 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0cf830 0x7f919c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 -- 172.21.15.122:0/1182565325 >> v1:172.21.15.122:6789/0 conn(0x7f919c0d27f0 legacy=0x7f919c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919bfff640 1 -- 172.21.15.122:0/1182565325 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0b3710 con 0x7f919c15dfe0 2024-03-06T07:30:02.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.524+0000 7f919affd640 1 -- 172.21.15.122:0/1182565325 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689801759 0 0) 0x7f919c0b3430 con 0x7f919c0d27f0 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f919bfff640 1 --2- 172.21.15.122:0/1182565325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0cf2f0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f918c009870 tx=0x7f918c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab53ce5368fd78c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f919affd640 1 -- 172.21.15.122:0/1182565325 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c034070 con 0x7f919c15dfe0 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f919affd640 1 -- 172.21.15.122:0/1182565325 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c030840 con 0x7f919c15dfe0 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f919affd640 1 -- 172.21.15.122:0/1182565325 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c030c10 con 0x7f919c15dfe0 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 -- 172.21.15.122:0/1182565325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 msgr2=0x7f919c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 --2- 172.21.15.122:0/1182565325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0cf2f0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f918c009870 tx=0x7f918c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 -- 172.21.15.122:0/1182565325 shutdown_connections 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 --2- 172.21.15.122:0/1182565325 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0cf830 0x7f919c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 --2- 172.21.15.122:0/1182565325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0cf2f0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 -- 172.21.15.122:0/1182565325 >> 172.21.15.122:0/1182565325 conn(0x7f919c0d3cd0 msgr2=0x7f919c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 -- 172.21.15.122:0/1182565325 shutdown_connections 2024-03-06T07:30:02.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.525+0000 7f91a265c640 1 -- 172.21.15.122:0/1182565325 wait complete. 2024-03-06T07:30:02.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.526+0000 7f91a265c640 1 Processor -- start 2024-03-06T07:30:02.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 -- start start 2024-03-06T07:30:02.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f919c0cf830 0x7f919c1f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0d27f0 0x7f919c160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c0d1c70 con 0x7f919c15dfe0 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c0b3d60 con 0x7f919c0cf830 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a265c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c0d1df0 con 0x7f919c0d27f0 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.527+0000 7f91a0bf9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0b9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55438/0 (socket says 172.21.15.122:55438) 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 -- 172.21.15.122:0/1420286288 learned_addr learned my addr 172.21.15.122:0/1420286288 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f919bfff640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f919c0cf830 0x7f919c1f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f919b7fe640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0d27f0 0x7f919c160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 -- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0d27f0 msgr2=0x7f919c160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0d27f0 0x7f919c160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 -- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f919c0cf830 msgr2=0x7f919c1f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f919c0cf830 0x7f919c1f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 -- 172.21.15.122:0/1420286288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c15f4a0 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a0bf9640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0b9390 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f919400c7a0 tx=0x7f919400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9194018070 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919400a640 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c15f780 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919400aab0 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.528+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c15fd40 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.529+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c04f330 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.530+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f919401b070 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.530+0000 7f9198ff9640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918806b500 0x7f918806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.530+0000 7f919bfff640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918806b500 0x7f918806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.530+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91940893b0 con 0x7f919c15dfe0 2024-03-06T07:30:02.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.530+0000 7f919bfff640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918806b500 0x7f918806d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f918c036000 tx=0x7f918c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.534+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9194057f30 con 0x7f919c15dfe0 2024-03-06T07:30:02.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.770+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f919c0d2350 con 0x7f919c15dfe0 2024-03-06T07:30:02.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.771+0000 7f9198ff9640 1 -- 172.21.15.122:0/1420286288 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f919405b4d0 con 0x7f919c15dfe0 2024-03-06T07:30:02.773 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:02.773 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:30:02.773 INFO:teuthology.orchestra.run.smithi122.stdout: "auth_dump": [ 2024-03-06T07:30:02.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.a", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlAoUnGxAA9MloMDMLIg81Dcm0fZBhoQ==", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.b", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmkeDJxAAZoyzHpWt5oi701zYvUXGUg==", 2024-03-06T07:30:02.774 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.c", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlddB/IhAAYhas7oq7bKAto9xcwH0Llw==", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:30:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mds.d", 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlmgXcLhAArbqcs4H12e0H+cRY5heTaw==", 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.776 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.0", 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBbGehltFcbExAANp/H+aKeYmYs5ZhDN427/w==", 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.1", 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBeGehl4drUEBAA8R3ZTSjvkfEZlk7Ow8QUWQ==", 2024-03-06T07:30:02.777 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.2", 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBhGehlaWHdDhAATX4YKFrti1LpboutmNee0A==", 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.3", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBkGehl+WBSDRAALIs6Jmc4r7RVqreVeXhBZg==", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.4", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBqGehlLQaIExAANQ1unPnhslLndGOr+vFASA==", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.779 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.5", 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBtGehlp7W/EhAAAXj/GLvZDxNK7TjwaI0Caw==", 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.780 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.6", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBwGehlC4JHERAAzt3ViMpY6NVljpYunhbRUA==", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "osd.7", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBzGehlMzTiDhAA/8F0/HCBrFAe+CQdqplodQ==", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.781 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.0", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA==", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow r", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.1", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlHHrHABAAM8ZfQAc7FYc+jlrcAU3Ftg==", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.782 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.2", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlIZb5OBAA4cz2YQETVrJy74X3SHD7GQ==", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.3", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBYGehlLCCjBhAAFGxjyc8zvacQdX9fcRn58w==", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow r", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow rw", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow rwx" 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "client.admin", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBWGehl5nX0ORAAfkLR2kSBN/zYt68vv8Vdaw==", 2024-03-06T07:30:02.784 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "mgr": "allow *", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow *", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.x", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlG5S+FxAAb+ECQeD48BQeyU7MsrcAjQ==", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: "entity": "mgr.y", 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: "key": "AQBXGehlql2YExAA64ya0Fq3081Zo3yhjtN/oA==", 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": { 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": "allow *", 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: "osd": "allow *" 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-06T07:30:02.786 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:30:02.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918806b500 msgr2=0x7f918806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918806b500 0x7f918806d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f918c036000 tx=0x7f918c03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 msgr2=0x7f919c0b9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0b9390 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f919400c7a0 tx=0x7f919400cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 shutdown_connections 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f918806b500 0x7f918806d9c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f919c15dfe0 0x7f919c0b9390 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f919c0d27f0 0x7f919c160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 --2- 172.21.15.122:0/1420286288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f919c0cf830 0x7f919c1f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 >> 172.21.15.122:0/1420286288 conn(0x7f919c0d3cd0 msgr2=0x7f919c0d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 shutdown_connections 2024-03-06T07:30:02.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.773+0000 7f91a265c640 1 -- 172.21.15.122:0/1420286288 wait complete. 2024-03-06T07:30:02.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:02.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.887+0000 7f32adac9640 1 Processor -- start 2024-03-06T07:30:02.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.887+0000 7f32adac9640 1 -- start start 2024-03-06T07:30:02.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32adac9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f32a80cce40 0x7f32a815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32adac9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a8071f40 con 0x7f32a81515e0 2024-03-06T07:30:02.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32adac9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a8070a70 con 0x7f32a80cd7a0 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32adac9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80cd2f0 con 0x7f32a80cce40 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f32a81515e0 0x7f32a8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53274/0 (socket says 172.21.15.122:53274) 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a77fe640 1 -- 172.21.15.122:0/1826233394 learned_addr learned my addr 172.21.15.122:0/1826233394 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a6ffd640 1 --2- 172.21.15.122:0/1826233394 >> v2:172.21.15.169:3300/0 conn(0x7f32a80cce40 0x7f32a815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a67fc640 1 -- 172.21.15.122:0/1826233394 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445904420 0 0) 0x7f32a8071f40 con 0x7f32a81515e0 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a67fc640 1 -- 172.21.15.122:0/1826233394 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3290003520 con 0x7f32a81515e0 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a67fc640 1 -- 172.21.15.122:0/1826233394 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3881458569 0 0) 0x7f3290003520 con 0x7f32a81515e0 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1826233394 >> v1:172.21.15.169:6789/0 conn(0x7f32a80cd7a0 legacy=0x7f32a80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1826233394 >> v1:172.21.15.122:6789/0 conn(0x7f32a81515e0 legacy=0x7f32a8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.888+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1826233394 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8070d50 con 0x7f32a80cce40 2024-03-06T07:30:02.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.889+0000 7f32a6ffd640 1 --2- 172.21.15.122:0/1826233394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a80cce40 0x7f32a815b430 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f329c00a940 tx=0x7f329c02ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53b9024cdfa96dbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.889+0000 7f32a67fc640 1 -- 172.21.15.122:0/1826233394 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c03a070 con 0x7f32a80cce40 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.889+0000 7f32a67fc640 1 -- 172.21.15.122:0/1826233394 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c02d820 con 0x7f32a80cce40 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.889+0000 7f32a67fc640 1 -- 172.21.15.122:0/1826233394 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298001690 con 0x7f32a80cce40 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 -- 172.21.15.122:0/1826233394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a80cce40 msgr2=0x7f32a815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 --2- 172.21.15.122:0/1826233394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a80cce40 0x7f32a815b430 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f329c00a940 tx=0x7f329c02ccf0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 -- 172.21.15.122:0/1826233394 shutdown_connections 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 --2- 172.21.15.122:0/1826233394 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a80cce40 0x7f32a815b430 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 -- 172.21.15.122:0/1826233394 >> 172.21.15.122:0/1826233394 conn(0x7f32a80cf8a0 msgr2=0x7f32a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 -- 172.21.15.122:0/1826233394 shutdown_connections 2024-03-06T07:30:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.890+0000 7f32adac9640 1 -- 172.21.15.122:0/1826233394 wait complete. 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.891+0000 7f32adac9640 1 Processor -- start 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.891+0000 7f32adac9640 1 -- start start 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32adac9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32a80cce40 0x7f32a81462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32adac9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 0x7f32a8154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32adac9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a81515e0 0x7f32a806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32adac9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80cc3a0 con 0x7f32a80cd7a0 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32adac9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8071310 con 0x7f32a81515e0 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32adac9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80cc520 con 0x7f32a80cce40 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 0x7f32a8154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 0x7f32a8154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55454/0 (socket says 172.21.15.122:55454) 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1739210709 learned_addr learned my addr 172.21.15.122:0/1739210709 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a77fe640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32a80cce40 0x7f32a81462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a7fff640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a81515e0 0x7f32a806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32a80cce40 msgr2=0x7f32a81462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32a80cce40 0x7f32a81462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a81515e0 msgr2=0x7f32a806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a81515e0 0x7f32a806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 -- 172.21.15.122:0/1739210709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8148580 con 0x7f32a80cd7a0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.892+0000 7f32a6ffd640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 0x7f32a8154570 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f329c000c00 tx=0x7f329c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.893+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c03a040 con 0x7f32a80cd7a0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.893+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c045070 con 0x7f32a80cd7a0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.893+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c00b040 con 0x7f32a80cd7a0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.893+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8152ca0 con 0x7f32a80cd7a0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.893+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a8153540 con 0x7f32a80cd7a0 2024-03-06T07:30:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.894+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a8068840 con 0x7f32a80cd7a0 2024-03-06T07:30:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.894+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f329c04a460 con 0x7f32a80cd7a0 2024-03-06T07:30:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.894+0000 7f328ffff640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f329406b4d0 0x7f329406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.894+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f329c0abdb0 con 0x7f32a80cd7a0 2024-03-06T07:30:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.894+0000 7f32a77fe640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f329406b4d0 0x7f329406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.895+0000 7f32a77fe640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f329406b4d0 0x7f329406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3298000fb0 tx=0x7f3298001e20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:02.899+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f329c07aa50 con 0x7f32a80cd7a0 2024-03-06T07:30:03.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.070+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32a804f330 con 0x7f32a80cd7a0 2024-03-06T07:30:03.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.071+0000 7f328ffff640 1 -- 172.21.15.122:0/1739210709 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+12901 (secure 0 0 0) 0x7f329c07eaf0 con 0x7f32a80cd7a0 2024-03-06T07:30:03.073 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:03.073 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":135,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:02.408402+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:03.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f329406b4d0 msgr2=0x7f329406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f329406b4d0 0x7f329406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3298000fb0 tx=0x7f3298001e20 comp rx=0 tx=0).stop 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 msgr2=0x7f32a8154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 0x7f32a8154570 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f329c000c00 tx=0x7f329c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 shutdown_connections 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f329406b4d0 0x7f329406d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32a81515e0 0x7f32a806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32a80cd7a0 0x7f32a8154570 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 --2- 172.21.15.122:0/1739210709 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f32a80cce40 0x7f32a81462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 >> 172.21.15.122:0/1739210709 conn(0x7f32a80cf8a0 msgr2=0x7f32a814fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 shutdown_connections 2024-03-06T07:30:03.076 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.073+0000 7f32adac9640 1 -- 172.21.15.122:0/1739210709 wait complete. 2024-03-06T07:30:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:03.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:03.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.188+0000 7f19e2464640 1 Processor -- start 2024-03-06T07:30:03.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.188+0000 7f19e2464640 1 -- start start 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19e2464640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc0cff40 0x7f19dc0b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19e2464640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc158ac0 con 0x7f19dc0cfa90 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19e2464640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc15b2a0 con 0x7f19dc1586e0 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19e2464640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc15b540 con 0x7f19dc0cff40 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19db7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f19dc0cfa90 0x7f19dc0b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53276/0 (socket says 172.21.15.122:53276) 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19db7fe640 1 -- 172.21.15.122:0/4014687149 learned_addr learned my addr 172.21.15.122:0/4014687149 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19e0a01640 1 --2- 172.21.15.122:0/4014687149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc0cff40 0x7f19dc0b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642573404 0 0) 0x7f19dc158ac0 con 0x7f19dc0cfa90 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.189+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c4003520 con 0x7f19dc0cfa90 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021259700 0 0) 0x7f19dc15b2a0 con 0x7f19dc1586e0 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19dc158ac0 con 0x7f19dc1586e0 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835925838 0 0) 0x7f19c4003520 con 0x7f19dc0cfa90 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19e0a01640 1 -- 172.21.15.122:0/4014687149 >> v1:172.21.15.169:6789/0 conn(0x7f19dc1586e0 legacy=0x7f19dc0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19e0a01640 1 -- 172.21.15.122:0/4014687149 >> v1:172.21.15.122:6789/0 conn(0x7f19dc0cfa90 legacy=0x7f19dc0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19e0a01640 1 -- 172.21.15.122:0/4014687149 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc15b820 con 0x7f19dc0cff40 2024-03-06T07:30:03.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.190+0000 7f19e0a01640 1 --2- 172.21.15.122:0/4014687149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc0cff40 0x7f19dc0b3ab0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009dc0 tx=0x7f19d402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b6768671655bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4039070 con 0x7f19dc0cff40 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d402c910 con 0x7f19dc0cff40 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19daffd640 1 -- 172.21.15.122:0/4014687149 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0001670 con 0x7f19dc0cff40 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 -- 172.21.15.122:0/4014687149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc0cff40 msgr2=0x7f19dc0b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 --2- 172.21.15.122:0/4014687149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc0cff40 0x7f19dc0b3ab0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009dc0 tx=0x7f19d402bde0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 -- 172.21.15.122:0/4014687149 shutdown_connections 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 --2- 172.21.15.122:0/4014687149 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc0cff40 0x7f19dc0b3ab0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 -- 172.21.15.122:0/4014687149 >> 172.21.15.122:0/4014687149 conn(0x7f19dc06b190 msgr2=0x7f19dc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 -- 172.21.15.122:0/4014687149 shutdown_connections 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.191+0000 7f19e2464640 1 -- 172.21.15.122:0/4014687149 wait complete. 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.192+0000 7f19e2464640 1 Processor -- start 2024-03-06T07:30:03.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 -- start start 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19dc0cfa90 0x7f19dc0ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 0x7f19dc0ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc1586e0 0x7f19dc0ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0d0360 con 0x7f19dc0cff40 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc15bde0 con 0x7f19dc0cfa90 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19e2464640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0af150 con 0x7f19dc1586e0 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 0x7f19dc0ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 0x7f19dc0ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55466/0 (socket says 172.21.15.122:55466) 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.193+0000 7f19db7fe640 1 -- 172.21.15.122:0/999843494 learned_addr learned my addr 172.21.15.122:0/999843494 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19dbfff640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19dc0cfa90 0x7f19dc0ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19e0a01640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc1586e0 0x7f19dc0ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19db7fe640 1 -- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc1586e0 msgr2=0x7f19dc0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19db7fe640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc1586e0 0x7f19dc0ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19db7fe640 1 -- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19dc0cfa90 msgr2=0x7f19dc0ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19db7fe640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19dc0cfa90 0x7f19dc0ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19db7fe640 1 -- 172.21.15.122:0/999843494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0ccd40 con 0x7f19dc0cff40 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19e0a01640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc1586e0 0x7f19dc0ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19db7fe640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 0x7f19dc0ce250 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f19d000c780 tx=0x7f19d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0017070 con 0x7f19dc0cff40 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0005af0 con 0x7f19dc0cff40 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0007070 con 0x7f19dc0cff40 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0cd020 con 0x7f19dc0cff40 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.194+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc0cd5e0 con 0x7f19dc0cff40 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.195+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0732b0 con 0x7f19dc0cff40 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.195+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19d001a070 con 0x7f19dc0cff40 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.196+0000 7f19d8ff9640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.196+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19d0089450 con 0x7f19dc0cff40 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.196+0000 7f19dbfff640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.196+0000 7f19dbfff640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19c806b4d0 0x7f19c806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0011e0 tx=0x7f19cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.200+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19d0058620 con 0x7f19dc0cff40 2024-03-06T07:30:03.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.370+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19dc068ff0 con 0x7f19dc0cff40 2024-03-06T07:30:03.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.370+0000 7f19d8ff9640 1 -- 172.21.15.122:0/999843494 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7f19d0057f20 con 0x7f19dc0cff40 2024-03-06T07:30:03.373 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19c806b4d0 msgr2=0x7f19c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19c806b4d0 0x7f19c806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0011e0 tx=0x7f19cc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 msgr2=0x7f19dc0ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 0x7f19dc0ce250 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f19d000c780 tx=0x7f19d000cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 shutdown_connections 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19c806b4d0 0x7f19c806d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19dc1586e0 0x7f19dc0ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19dc0cff40 0x7f19dc0ce250 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 --2- 172.21.15.122:0/999843494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19dc0cfa90 0x7f19dc0ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 >> 172.21.15.122:0/999843494 conn(0x7f19dc06b190 msgr2=0x7f19dc156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.372+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 shutdown_connections 2024-03-06T07:30:03.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.373+0000 7f19e2464640 1 -- 172.21.15.122:0/999843494 wait complete. 2024-03-06T07:30:03.386 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.484+0000 7f5b636b5640 1 Processor -- start 2024-03-06T07:30:03.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.485+0000 7f5b636b5640 1 -- start start 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.485+0000 7f5b636b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5b5c0b4170 0x7f5b5c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.485+0000 7f5b636b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0d25d0 con 0x7f5b5c0d4760 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.485+0000 7f5b636b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0d2870 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.485+0000 7f5b636b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5c0d4bf0 con 0x7f5b5c0b4170 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b61451640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5b5c0b4170 0x7f5b5c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b60c50640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5b5c0d4760 0x7f5b5c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53292/0 (socket says 172.21.15.122:53292) 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b61451640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5b5c0b4170 0x7f5b5c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55470/0 (socket says 172.21.15.122:55470) 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b60c50640 1 -- 172.21.15.122:0/1142709728 learned_addr learned my addr 172.21.15.122:0/1142709728 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:03.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b61451640 1 -- 172.21.15.122:0/1142709728 >> v1:172.21.15.169:6789/0 conn(0x7f5b5c0d0a80 legacy=0x7f5b5c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b61451640 1 -- 172.21.15.122:0/1142709728 >> v1:172.21.15.122:6789/0 conn(0x7f5b5c0d4760 legacy=0x7f5b5c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b61451640 1 -- 172.21.15.122:0/1142709728 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0d2b50 con 0x7f5b5c0b4170 2024-03-06T07:30:03.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.486+0000 7f5b53fff640 1 -- 172.21.15.122:0/1142709728 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702205831 0 0) 0x7f5b5c0d25d0 con 0x7f5b5c0d4760 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b61451640 1 --2- 172.21.15.122:0/1142709728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c0b6a90 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097a0 tx=0x7f5b4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b6a389fc7eb211 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b53fff640 1 -- 172.21.15.122:0/1142709728 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c035020 con 0x7f5b5c0b4170 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b53fff640 1 -- 172.21.15.122:0/1142709728 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c031830 con 0x7f5b5c0b4170 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b53fff640 1 -- 172.21.15.122:0/1142709728 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c031c00 con 0x7f5b5c0b4170 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 -- 172.21.15.122:0/1142709728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0b4170 msgr2=0x7f5b5c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1142709728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c0b6a90 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097a0 tx=0x7f5b4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 -- 172.21.15.122:0/1142709728 shutdown_connections 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1142709728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c0b6a90 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 -- 172.21.15.122:0/1142709728 >> 172.21.15.122:0/1142709728 conn(0x7f5b5c0d3c40 msgr2=0x7f5b5c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 -- 172.21.15.122:0/1142709728 shutdown_connections 2024-03-06T07:30:03.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.487+0000 7f5b636b5640 1 -- 172.21.15.122:0/1142709728 wait complete. 2024-03-06T07:30:03.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.488+0000 7f5b636b5640 1 Processor -- start 2024-03-06T07:30:03.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 -- start start 2024-03-06T07:30:03.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 0x7f5b5c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b5c14a030 0x7f5b5c14a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0b4dd0 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0ce9e0 con 0x7f5b5c0b4170 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b636b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0b4f50 con 0x7f5b5c14a030 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b60c50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 0x7f5b5c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b60c50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 0x7f5b5c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55476/0 (socket says 172.21.15.122:55476) 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b60c50640 1 -- 172.21.15.122:0/1363186190 learned_addr learned my addr 172.21.15.122:0/1363186190 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b61451640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.489+0000 7f5b61c52640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b5c14a030 0x7f5b5c14a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b60c50640 1 -- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b5c14a030 msgr2=0x7f5b5c14a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b60c50640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b5c14a030 0x7f5b5c14a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b60c50640 1 -- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b5c0b4170 msgr2=0x7f5b5c157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b60c50640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b60c50640 1 -- 172.21.15.122:0/1363186190 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c15dbe0 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b61c52640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b5c14a030 0x7f5b5c14a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b60c50640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 0x7f5b5c15a9b0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c460 tx=0x7f5b5800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58018070 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b58007500 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c15aef0 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58004a20 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.490+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c15d100 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.491+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c04f330 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.491+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b5801b070 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.492+0000 7f5b51ffb640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4006b500 0x7f5b4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.492+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b58089100 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.492+0000 7f5b61451640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4006b500 0x7f5b4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.492+0000 7f5b61451640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4006b500 0x7f5b4006d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c036000 tx=0x7f5b4c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.496+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b580582d0 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.667+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5b5c0733b0 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.668+0000 7f5b51ffb640 1 -- 172.21.15.122:0/1363186190 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5b58057bd0 con 0x7f5b5c0d0a80 2024-03-06T07:30:03.670 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:03.670 INFO:teuthology.orchestra.run.smithi122.stdout:[ 2024-03-06T07:30:03.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-06T07:30:03.670 INFO:teuthology.orchestra.run.smithi122.stdout: "poolnum": 1, 2024-03-06T07:30:03.671 INFO:teuthology.orchestra.run.smithi122.stdout: "poolname": ".mgr" 2024-03-06T07:30:03.671 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:03.671 INFO:teuthology.orchestra.run.smithi122.stdout:] 2024-03-06T07:30:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4006b500 msgr2=0x7f5b4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4006b500 0x7f5b4006d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c036000 tx=0x7f5b4c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 msgr2=0x7f5b5c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 0x7f5b5c15a9b0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c460 tx=0x7f5b5800c920 comp rx=0 tx=0).stop 2024-03-06T07:30:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 shutdown_connections 2024-03-06T07:30:03.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f5b4006b500 0x7f5b4006d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f5b5c14a030 0x7f5b5c14a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b5c0d0a80 0x7f5b5c15a9b0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 --2- 172.21.15.122:0/1363186190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b5c0b4170 0x7f5b5c157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 >> 172.21.15.122:0/1363186190 conn(0x7f5b5c0d3c40 msgr2=0x7f5b5c0b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 shutdown_connections 2024-03-06T07:30:03.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.670+0000 7f5b636b5640 1 -- 172.21.15.122:0/1363186190 wait complete. 2024-03-06T07:30:03.683 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:03.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:03.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.785+0000 7f6ae9c88640 1 Processor -- start 2024-03-06T07:30:03.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.785+0000 7f6ae9c88640 1 -- start start 2024-03-06T07:30:03.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae9c88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae40ce8c0 0x7f6ae40cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae9c88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ae415c980 0x7f6ae4071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae9c88640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae40ced50 con 0x7f6ae415c5a0 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae9c88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae4072a20 con 0x7f6ae415c980 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae9c88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40708b0 con 0x7f6ae40ce8c0 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ae415c980 0x7f6ae4071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6ae415c980 0x7f6ae4071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55486/0 (socket says 172.21.15.122:55486) 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae3fff640 1 -- 172.21.15.122:0/606248819 learned_addr learned my addr 172.21.15.122:0/606248819 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.786+0000 7f6ae2ffd640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae40ce8c0 0x7f6ae40cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae3fff640 1 -- 172.21.15.122:0/606248819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae40ce8c0 msgr2=0x7f6ae40cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae3fff640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae40ce8c0 0x7f6ae40cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae3fff640 1 -- 172.21.15.122:0/606248819 >> v1:172.21.15.169:6789/0 conn(0x7f6ae415c5a0 legacy=0x7f6ae406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae3fff640 1 -- 172.21.15.122:0/606248819 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae40cf030 con 0x7f6ae415c980 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae2ffd640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae40ce8c0 0x7f6ae40cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae3fff640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4071300 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009e50 tx=0x7f6adc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49714a3233eba42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae27fc640 1 -- 172.21.15.122:0/606248819 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc02ca70 con 0x7f6ae415c980 2024-03-06T07:30:03.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae27fc640 1 -- 172.21.15.122:0/606248819 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc02cc10 con 0x7f6ae415c980 2024-03-06T07:30:03.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.787+0000 7f6ae27fc640 1 -- 172.21.15.122:0/606248819 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0343f0 con 0x7f6ae415c980 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 -- 172.21.15.122:0/606248819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 msgr2=0x7f6ae4071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4071300 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009e50 tx=0x7f6adc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 -- 172.21.15.122:0/606248819 shutdown_connections 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4071300 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/606248819 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae40ce8c0 0x7f6ae40cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 -- 172.21.15.122:0/606248819 >> 172.21.15.122:0/606248819 conn(0x7f6ae40b1650 msgr2=0x7f6ae40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 -- 172.21.15.122:0/606248819 shutdown_connections 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.788+0000 7f6ae9c88640 1 -- 172.21.15.122:0/606248819 wait complete. 2024-03-06T07:30:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.789+0000 7f6ae9c88640 1 Processor -- start 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.789+0000 7f6ae9c88640 1 -- start start 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae9c88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ae415c5a0 0x7f6ae40b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae9c88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae9c88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae4158460 0x7f6ae4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae9c88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40cc870 con 0x7f6ae415c980 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae9c88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae4070e70 con 0x7f6ae415c5a0 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae9c88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae415ce30 con 0x7f6ae4158460 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55490/0 (socket says 172.21.15.122:55490) 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 -- 172.21.15.122:0/615075361 learned_addr learned my addr 172.21.15.122:0/615075361 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae3fff640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae4158460 0x7f6ae4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae37fe640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ae415c5a0 0x7f6ae40b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 -- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae4158460 msgr2=0x7f6ae4156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae4158460 0x7f6ae4156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 -- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ae415c5a0 msgr2=0x7f6ae40b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ae415c5a0 0x7f6ae40b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae2ffd640 1 -- 172.21.15.122:0/615075361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae41589d0 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.790+0000 7f6ae3fff640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae4158460 0x7f6ae4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.791+0000 7f6ae2ffd640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4156600 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800e7e0 tx=0x7f6ad800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.791+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad800f040 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.791+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad8004030 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.791+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8004400 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.791+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae4158cb0 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.791+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae4159270 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.792+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae40687d0 con 0x7f6ae415c980 2024-03-06T07:30:03.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.792+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ad800c6d0 con 0x7f6ae415c980 2024-03-06T07:30:03.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.793+0000 7f6acbfff640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ad006b4d0 0x7f6ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.793+0000 7f6ae37fe640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ad006b4d0 0x7f6ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.793+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ad8089d10 con 0x7f6ae415c980 2024-03-06T07:30:03.797 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.793+0000 7f6ae37fe640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ad006b4d0 0x7f6ad006d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40015a0 tx=0x7f6ad4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.797+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ad8017020 con 0x7f6ae415c980 2024-03-06T07:30:03.970 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:03.968+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6ae404f330 con 0x7f6ae415c980 2024-03-06T07:30:04.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.411+0000 7f6acbfff640 1 -- 172.21.15.122:0/615075361 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v136) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6ad8058e60 con 0x7f6ae415c980 2024-03-06T07:30:04.413 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ad006b4d0 msgr2=0x7f6ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ad006b4d0 0x7f6ad006d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40015a0 tx=0x7f6ad4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 msgr2=0x7f6ae4156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4156600 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800e7e0 tx=0x7f6ad800eca0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 shutdown_connections 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6ad006b4d0 0x7f6ad006d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6ae4158460 0x7f6ae4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ae415c980 0x7f6ae4156600 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 --2- 172.21.15.122:0/615075361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ae415c5a0 0x7f6ae40b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 >> 172.21.15.122:0/615075361 conn(0x7f6ae40b1650 msgr2=0x7f6ae415dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.415+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 shutdown_connections 2024-03-06T07:30:04.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.416+0000 7f6ae9c88640 1 -- 172.21.15.122:0/615075361 wait complete. 2024-03-06T07:30:04.431 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:04.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.537+0000 7fef9c6b8640 1 Processor -- start 2024-03-06T07:30:04.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.537+0000 7fef9c6b8640 1 -- start start 2024-03-06T07:30:04.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.538+0000 7fef9c6b8640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fef940b1e40 0x7fef94168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.538+0000 7fef9c6b8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef940d2730 con 0x7fef940d4860 2024-03-06T07:30:04.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.538+0000 7fef9c6b8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef941618b0 con 0x7fef941614d0 2024-03-06T07:30:04.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.538+0000 7fef9c6b8640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef940d4cf0 con 0x7fef940b1e40 2024-03-06T07:30:04.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.538+0000 7fef99c53640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fef940d4860 0x7fef940d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45198/0 (socket says 172.21.15.122:45198) 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 -- 172.21.15.122:0/1852493867 learned_addr learned my addr 172.21.15.122:0/1852493867 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef9ac55640 1 --2- 172.21.15.122:0/1852493867 >> v2:172.21.15.169:3300/0 conn(0x7fef940b1e40 0x7fef94168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955457691 0 0) 0x7fef940d2730 con 0x7fef940d4860 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef80003520 con 0x7fef940d4860 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3359978903 0 0) 0x7fef80003520 con 0x7fef940d4860 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 >> v2:172.21.15.169:3300/0 conn(0x7fef940b1e40 msgr2=0x7fef94168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 --2- 172.21.15.122:0/1852493867 >> v2:172.21.15.169:3300/0 conn(0x7fef940b1e40 0x7fef94168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 >> v1:172.21.15.169:6789/0 conn(0x7fef941614d0 legacy=0x7fef940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef940cf400 con 0x7fef940d4860 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fef84001670 con 0x7fef940d4860 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 >> v1:172.21.15.122:6789/0 conn(0x7fef940d4860 legacy=0x7fef940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef800039b0 0x7fef80003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef80004800 0x7fef80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80005210 0x7fef8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef940d4cf0 con 0x7fef80004800 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8000c140 con 0x7fef80005210 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8000c340 con 0x7fef800039b0 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef80004800 0x7fef80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef9a454640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef800039b0 0x7fef80003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef9ac55640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80005210 0x7fef8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 -- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef800039b0 msgr2=0x7fef80003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef800039b0 0x7fef80003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 -- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80005210 msgr2=0x7fef8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80005210 0x7fef8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef99c53640 1 -- 172.21.15.122:0/1852493867 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef80003520 con 0x7fef80004800 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.539+0000 7fef9ac55640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80005210 0x7fef8000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.540+0000 7fef9ac55640 1 -- 172.21.15.122:0/1852493867 reap_dead start 2024-03-06T07:30:04.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.540+0000 7fef99c53640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef80004800 0x7fef80004c80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fef8400c210 tx=0x7fef80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c4920e4b9320e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.540+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84007d30 con 0x7fef80004800 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.540+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef84030070 con 0x7fef80004800 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.540+0000 7fef99452640 1 -- 172.21.15.122:0/1852493867 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84005d70 con 0x7fef80004800 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 -- 172.21.15.122:0/1852493867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef80004800 msgr2=0x7fef80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef80004800 0x7fef80004c80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fef8400c210 tx=0x7fef80005830 comp rx=0 tx=0).stop 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 -- 172.21.15.122:0/1852493867 shutdown_connections 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/1852493867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef80004800 0x7fef80004c80 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 -- 172.21.15.122:0/1852493867 >> 172.21.15.122:0/1852493867 conn(0x7fef940d3d40 msgr2=0x7fef940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 -- 172.21.15.122:0/1852493867 shutdown_connections 2024-03-06T07:30:04.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.541+0000 7fef9c6b8640 1 -- 172.21.15.122:0/1852493867 wait complete. 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.542+0000 7fef9c6b8640 1 Processor -- start 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.542+0000 7fef9c6b8640 1 -- start start 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80004800 0x7fef941f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef9414e160 0x7fef940b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 0x7fef940b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef940cfc60 con 0x7fef940b6e70 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef940cf980 con 0x7fef80004800 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef940b0910 con 0x7fef9414e160 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 0x7fef940b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 0x7fef940b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60274/0 (socket says 172.21.15.122:60274) 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 -- 172.21.15.122:0/3710094515 learned_addr learned my addr 172.21.15.122:0/3710094515 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef99c53640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef9414e160 0x7fef940b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9a454640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80004800 0x7fef941f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 -- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef9414e160 msgr2=0x7fef940b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef9414e160 0x7fef940b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 -- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80004800 msgr2=0x7fef941f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80004800 0x7fef941f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 -- 172.21.15.122:0/3710094515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef940b98a0 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef99c53640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef9414e160 0x7fef940b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9a454640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80004800 0x7fef941f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9ac55640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 0x7fef940b8b10 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fef9408c4f0 tx=0x7fef8c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c010da0 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c003990 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef940c8300 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c003d80 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.543+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef940c88c0 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.544+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef94073fa0 con 0x7fef940b6e70 2024-03-06T07:30:04.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.545+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef8c01e020 con 0x7fef940b6e70 2024-03-06T07:30:04.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.545+0000 7fef8affd640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef7806b4d0 0x7fef7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.545+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fef8c08a770 con 0x7fef940b6e70 2024-03-06T07:30:04.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.545+0000 7fef9a454640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef7806b4d0 0x7fef7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.546+0000 7fef9a454640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef7806b4d0 0x7fef7806d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fef940ca2c0 tx=0x7fef90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.550+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef8c0597d0 con 0x7fef940b6e70 2024-03-06T07:30:04.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:04.726+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fef94068ff0 con 0x7fef940b6e70 2024-03-06T07:30:05.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.413+0000 7fef8affd640 1 -- 172.21.15.122:0/3710094515 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v137) v1 ==== 170+0+0 (secure 0 0 0) 0x7fef8c0590d0 con 0x7fef940b6e70 2024-03-06T07:30:05.415 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' created 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef7806b4d0 msgr2=0x7fef7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef7806b4d0 0x7fef7806d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fef940ca2c0 tx=0x7fef90009040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 msgr2=0x7fef940b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 0x7fef940b8b10 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fef9408c4f0 tx=0x7fef8c010590 comp rx=0 tx=0).stop 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 shutdown_connections 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef7806b4d0 0x7fef7806d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef940b6e70 0x7fef940b8b10 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef9414e160 0x7fef940b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 --2- 172.21.15.122:0/3710094515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef80004800 0x7fef941f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.416+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 >> 172.21.15.122:0/3710094515 conn(0x7fef940d3d40 msgr2=0x7fef940d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.417+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 shutdown_connections 2024-03-06T07:30:05.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.417+0000 7fef9c6b8640 1 -- 172.21.15.122:0/3710094515 wait complete. 2024-03-06T07:30:05.444 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:05.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 Processor -- start 2024-03-06T07:30:05.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 -- start start 2024-03-06T07:30:05.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 0x7fd09c15b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd09c0b3cd0 0x7fd09c15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c15bdf0 con 0x7fd09c158420 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c15e650 con 0x7fd09c0b3cd0 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.549+0000 7fd0a27e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c15e8a0 con 0x7fd09c158800 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 0x7fd09c15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd0a0d7f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd09c0b3cd0 0x7fd09c15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 0x7fd09c15b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:35070/0 (socket says 172.21.15.122:35070) 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 -- 172.21.15.122:0/1415962506 learned_addr learned my addr 172.21.15.122:0/1415962506 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09affd640 1 -- 172.21.15.122:0/1415962506 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118687944 0 0) 0x7fd09c15bdf0 con 0x7fd09c158420 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09affd640 1 -- 172.21.15.122:0/1415962506 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd084003520 con 0x7fd09c158420 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 -- 172.21.15.122:0/1415962506 >> v2:172.21.15.169:3300/0 conn(0x7fd09c0b3cd0 msgr2=0x7fd09c15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 --2- 172.21.15.122:0/1415962506 >> v2:172.21.15.169:3300/0 conn(0x7fd09c0b3cd0 0x7fd09c15b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 -- 172.21.15.122:0/1415962506 >> v1:172.21.15.169:6789/0 conn(0x7fd09c158420 legacy=0x7fd09c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.550+0000 7fd09b7fe640 1 -- 172.21.15.122:0/1415962506 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c15eb80 con 0x7fd09c158800 2024-03-06T07:30:05.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.551+0000 7fd09b7fe640 1 --2- 172.21.15.122:0/1415962506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 0x7fd09c15b370 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd090009870 tx=0x7fd09002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e19c31b3a498f1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.551+0000 7fd09affd640 1 -- 172.21.15.122:0/1415962506 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090035020 con 0x7fd09c158800 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.551+0000 7fd09affd640 1 -- 172.21.15.122:0/1415962506 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090031640 con 0x7fd09c158800 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.551+0000 7fd09affd640 1 -- 172.21.15.122:0/1415962506 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09002d040 con 0x7fd09c158800 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 -- 172.21.15.122:0/1415962506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 msgr2=0x7fd09c15b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/1415962506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 0x7fd09c15b370 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd090009870 tx=0x7fd09002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 -- 172.21.15.122:0/1415962506 shutdown_connections 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/1415962506 >> v2:172.21.15.169:3300/0 conn(0x7fd09c0b3cd0 0x7fd09c15b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/1415962506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158800 0x7fd09c15b370 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 -- 172.21.15.122:0/1415962506 >> 172.21.15.122:0/1415962506 conn(0x7fd09c06afb0 msgr2=0x7fd09c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 -- 172.21.15.122:0/1415962506 shutdown_connections 2024-03-06T07:30:05.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.552+0000 7fd0a27e2640 1 -- 172.21.15.122:0/1415962506 wait complete. 2024-03-06T07:30:05.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.553+0000 7fd0a27e2640 1 Processor -- start 2024-03-06T07:30:05.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.553+0000 7fd0a27e2640 1 -- start start 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a27e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd09c0b3cd0 0x7fd09c14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a27e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158420 0x7fd09c155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a27e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 0x7fd09c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a27e2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c06ca00 con 0x7fd09c153160 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a27e2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c15f140 con 0x7fd09c0b3cd0 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a27e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c06cb80 con 0x7fd09c158420 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 0x7fd09c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 0x7fd09c0ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60282/0 (socket says 172.21.15.122:60282) 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 -- 172.21.15.122:0/109534617 learned_addr learned my addr 172.21.15.122:0/109534617 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd09b7fe640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158420 0x7fd09c155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd09bfff640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd09c0b3cd0 0x7fd09c14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 -- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158420 msgr2=0x7fd09c155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158420 0x7fd09c155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 -- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd09c0b3cd0 msgr2=0x7fd09c14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd09c0b3cd0 0x7fd09c14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd0a0d7f640 1 -- 172.21.15.122:0/109534617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c155da0 con 0x7fd09c153160 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.554+0000 7fd09bfff640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd09c0b3cd0 0x7fd09c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.555+0000 7fd0a0d7f640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 0x7fd09c0ad860 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd09400be70 tx=0x7fd09400e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.555+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09401a070 con 0x7fd09c153160 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.555+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09400ebe0 con 0x7fd09c153160 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.555+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094003990 con 0x7fd09c153160 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.555+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c154440 con 0x7fd09c153160 2024-03-06T07:30:05.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.555+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c154a00 con 0x7fd09c153160 2024-03-06T07:30:05.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.556+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c04f330 con 0x7fd09c153160 2024-03-06T07:30:05.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.556+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd09401d070 con 0x7fd09c153160 2024-03-06T07:30:05.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.557+0000 7fd098ff9640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd08806b4d0 0x7fd08806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.557+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd09408a1c0 con 0x7fd09c153160 2024-03-06T07:30:05.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.557+0000 7fd09bfff640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd08806b4d0 0x7fd08806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.557+0000 7fd09bfff640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd08806b4d0 0x7fd08806d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0016b0 tx=0x7fd08c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.561+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd09c155da0 con 0x7fd09c153160 2024-03-06T07:30:05.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:05.768+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd09c155fd0 con 0x7fd09c153160 2024-03-06T07:30:06.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.415+0000 7fd098ff9640 1 -- 172.21.15.122:0/109534617 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd094059130 con 0x7fd09c153160 2024-03-06T07:30:06.417 INFO:teuthology.orchestra.run.smithi122.stderr:new fs with metadata pool 20 and data pool 21 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.420+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd08806b4d0 msgr2=0x7fd08806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.420+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd08806b4d0 0x7fd08806d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0016b0 tx=0x7fd08c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.420+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 msgr2=0x7fd09c0ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.420+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 0x7fd09c0ad860 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd09400be70 tx=0x7fd09400e490 comp rx=0 tx=0).stop 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.423+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 shutdown_connections 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.423+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd08806b4d0 0x7fd08806d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.423+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd09c153160 0x7fd09c0ad860 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.423+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd09c158420 0x7fd09c155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.423+0000 7fd0a27e2640 1 --2- 172.21.15.122:0/109534617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd09c0b3cd0 0x7fd09c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.423+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 >> 172.21.15.122:0/109534617 conn(0x7fd09c06afb0 msgr2=0x7fd09c06d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.427+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 shutdown_connections 2024-03-06T07:30:06.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.430+0000 7fd0a27e2640 1 -- 172.21.15.122:0/109534617 wait complete. 2024-03-06T07:30:06.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.558+0000 7f958c3f6640 1 Processor -- start 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.558+0000 7f958c3f6640 1 -- start start 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958c3f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f95841424f0 0x7f95840cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958c3f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f958406da40 0x7f958415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958c3f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95840cc980 con 0x7f95840cd7a0 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958c3f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95840cd220 con 0x7f958406da40 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958c3f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95840ccc60 con 0x7f95841424f0 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f958406da40 0x7f958415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f958406da40 0x7f958415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60292/0 (socket says 172.21.15.122:60292) 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 -- 172.21.15.122:0/3865249606 learned_addr learned my addr 172.21.15.122:0/3865249606 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 -- 172.21.15.122:0/3865249606 >> v2:172.21.15.169:3300/0 conn(0x7f95841424f0 msgr2=0x7f95840cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 --2- 172.21.15.122:0/3865249606 >> v2:172.21.15.169:3300/0 conn(0x7f95841424f0 0x7f95840cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 -- 172.21.15.122:0/3865249606 >> v1:172.21.15.169:6789/0 conn(0x7f95840cd7a0 legacy=0x7f958406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.559+0000 7f958a993640 1 -- 172.21.15.122:0/3865249606 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840ccf40 con 0x7f958406da40 2024-03-06T07:30:06.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.560+0000 7f958a993640 1 --2- 172.21.15.122:0/3865249606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958415cef0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f958406c6d0 tx=0x7f957c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102279ff4e65548f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.560+0000 7f9589190640 1 -- 172.21.15.122:0/3865249606 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c02ca80 con 0x7f958406da40 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.560+0000 7f9589190640 1 -- 172.21.15.122:0/3865249606 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c02cc20 con 0x7f958406da40 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.560+0000 7f9589190640 1 -- 172.21.15.122:0/3865249606 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c0343f0 con 0x7f958406da40 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.560+0000 7f958c3f6640 1 -- 172.21.15.122:0/3865249606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 msgr2=0x7f958415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.560+0000 7f958c3f6640 1 --2- 172.21.15.122:0/3865249606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958415cef0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f958406c6d0 tx=0x7f957c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.561+0000 7f958c3f6640 1 -- 172.21.15.122:0/3865249606 shutdown_connections 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.561+0000 7f958c3f6640 1 --2- 172.21.15.122:0/3865249606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958415cef0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.561+0000 7f958c3f6640 1 --2- 172.21.15.122:0/3865249606 >> v2:172.21.15.169:3300/0 conn(0x7f95841424f0 0x7f95840cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.561+0000 7f958c3f6640 1 -- 172.21.15.122:0/3865249606 >> 172.21.15.122:0/3865249606 conn(0x7f95840cf8a0 msgr2=0x7f95840cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.561+0000 7f958c3f6640 1 -- 172.21.15.122:0/3865249606 shutdown_connections 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.561+0000 7f958c3f6640 1 -- 172.21.15.122:0/3865249606 wait complete. 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 Processor -- start 2024-03-06T07:30:06.564 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 -- start start 2024-03-06T07:30:06.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958413f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95840cd7a0 0x7f95840c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95841424f0 0x7f95840c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584071ce0 con 0x7f958406da40 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584071b60 con 0x7f95840cd7a0 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.562+0000 7f958c3f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9584071e60 con 0x7f95841424f0 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958413f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60306/0 (socket says 172.21.15.122:60306) 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 -- 172.21.15.122:0/1276027687 learned_addr learned my addr 172.21.15.122:0/1276027687 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 -- 172.21.15.122:0/1276027687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95841424f0 msgr2=0x7f95840c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95841424f0 0x7f95840c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 -- 172.21.15.122:0/1276027687 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95840cd7a0 msgr2=0x7f95840c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95840cd7a0 0x7f95840c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 -- 172.21.15.122:0/1276027687 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840ad900 con 0x7f958406da40 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958a192640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958413f170 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f958000b440 tx=0x7f958000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580015020 con 0x7f958406da40 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9580011830 con 0x7f958406da40 2024-03-06T07:30:06.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95841599b0 con 0x7f958406da40 2024-03-06T07:30:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580011c20 con 0x7f958406da40 2024-03-06T07:30:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.563+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95840ade80 con 0x7f958406da40 2024-03-06T07:30:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.564+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958404f330 con 0x7f958406da40 2024-03-06T07:30:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.565+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9580018020 con 0x7f958406da40 2024-03-06T07:30:06.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.565+0000 7f957affd640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f956806b4d0 0x7f956806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.565+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f958008a110 con 0x7f958406da40 2024-03-06T07:30:06.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.565+0000 7f9589991640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f956806b4d0 0x7f956806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.569 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.566+0000 7f9589991640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f956806b4d0 0x7f956806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f95841f7a50 tx=0x7f9574007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.570+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f958001f020 con 0x7f958406da40 2024-03-06T07:30:06.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.740+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f958415acd0 con 0x7f958406da40 2024-03-06T07:30:06.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.742+0000 7f957affd640 1 -- 172.21.15.122:0/1276027687 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16692 (secure 0 0 0) 0x7f9580058f90 con 0x7f958406da40 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":138,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:05.770348+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f956806b4d0 msgr2=0x7f956806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f956806b4d0 0x7f956806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f95841f7a50 tx=0x7f9574007990 comp rx=0 tx=0).stop 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 msgr2=0x7f958413f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958413f170 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f958000b440 tx=0x7f958000b900 comp rx=0 tx=0).stop 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 shutdown_connections 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f956806b4d0 0x7f956806d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95841424f0 0x7f95840c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95840cd7a0 0x7f95840c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 --2- 172.21.15.122:0/1276027687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f958406da40 0x7f958413f170 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.743+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 >> 172.21.15.122:0/1276027687 conn(0x7f95840cf8a0 msgr2=0x7f958415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.744+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 shutdown_connections 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.744+0000 7f958c3f6640 1 -- 172.21.15.122:0/1276027687 wait complete. 2024-03-06T07:30:06.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.854+0000 7f7b7ceeb640 1 Processor -- start 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.855+0000 7f7b7ceeb640 1 -- start start 2024-03-06T07:30:06.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.855+0000 7f7b7ceeb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 0x7f7b780b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.855+0000 7f7b7ceeb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780b3430 con 0x7f7b7815dfe0 2024-03-06T07:30:06.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.855+0000 7f7b7ceeb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780b3710 con 0x7f7b780d22b0 2024-03-06T07:30:06.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.855+0000 7f7b7ceeb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780b31e0 con 0x7f7b780d2710 2024-03-06T07:30:06.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b7659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7b7815dfe0 0x7f7b780cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45206/0 (socket says 172.21.15.122:45206) 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b7659c640 1 -- 172.21.15.122:0/263802690 learned_addr learned my addr 172.21.15.122:0/263802690 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b76d9d640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 0x7f7b780b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396175531 0 0) 0x7f7b780b3430 con 0x7f7b7815dfe0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64003520 con 0x7f7b7815dfe0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145541928 0 0) 0x7f7b780b3710 con 0x7f7b780d22b0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b780b3430 con 0x7f7b780d22b0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348346069 0 0) 0x7f7b64003520 con 0x7f7b7815dfe0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 msgr2=0x7f7b780b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 0x7f7b780b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 >> v1:172.21.15.169:6789/0 conn(0x7f7b780d22b0 legacy=0x7f7b780b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780b39f0 con 0x7f7b7815dfe0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7b68001670 con 0x7f7b7815dfe0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 >> v1:172.21.15.122:6789/0 conn(0x7f7b7815dfe0 legacy=0x7f7b780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64003b70 0x7f7b64003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b64004800 0x7f7b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 0x7f7b640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780b31e0 con 0x7f7b64004800 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400bad0 con 0x7f7b64003b70 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400bcd0 con 0x7f7b780d2710 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b64004800 0x7f7b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b7659c640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64003b70 0x7f7b64003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.856+0000 7f7b76d9d640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 0x7f7b640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b6ffff640 1 -- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 msgr2=0x7f7b640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b780d2710 0x7f7b640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b6ffff640 1 -- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64003b70 msgr2=0x7f7b64003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64003b70 0x7f7b64003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b6ffff640 1 -- 172.21.15.122:0/263802690 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64003520 con 0x7f7b64004800 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b64004800 0x7f7b64004c80 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f7b600018a0 tx=0x7f7b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72401b01d45b9204 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037070 con 0x7f7b64004800 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b600052d0 con 0x7f7b64004800 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.857+0000 7f7b75d9b640 1 -- 172.21.15.122:0/263802690 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b600014f0 con 0x7f7b64004800 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/263802690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b64004800 msgr2=0x7f7b64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/263802690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b64004800 0x7f7b64004c80 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f7b600018a0 tx=0x7f7b64005830 comp rx=0 tx=0).stop 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b76d9d640 1 -- 172.21.15.122:0/263802690 reap_dead start 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/263802690 shutdown_connections 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/263802690 >> 172.21.15.122:0/263802690 conn(0x7f7b780d3cd0 msgr2=0x7f7b780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/263802690 shutdown_connections 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.858+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/263802690 wait complete. 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.859+0000 7f7b7ceeb640 1 Processor -- start 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.859+0000 7f7b7ceeb640 1 -- start start 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7ceeb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64004800 0x7f7b780cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7ceeb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 0x7f7b78160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7ceeb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b78160db0 0x7f7b780b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7ceeb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780cfa00 con 0x7f7b78159aa0 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7ceeb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780b3d60 con 0x7f7b64004800 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7ceeb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780d1c70 con 0x7f7b78160db0 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 0x7f7b78160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 0x7f7b78160870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60320/0 (socket says 172.21.15.122:60320) 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 -- 172.21.15.122:0/1736105632 learned_addr learned my addr 172.21.15.122:0/1736105632 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b7659c640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64004800 0x7f7b780cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b76d9d640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b78160db0 0x7f7b780b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 -- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b78160db0 msgr2=0x7f7b780b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b78160db0 0x7f7b780b8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 -- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64004800 msgr2=0x7f7b780cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64004800 0x7f7b780cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b6ffff640 1 -- 172.21.15.122:0/1736105632 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780cbe40 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.860+0000 7f7b76d9d640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b78160db0 0x7f7b780b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.861+0000 7f7b6ffff640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 0x7f7b78160870 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f7b60005040 tx=0x7f7b600012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.861+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60042020 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.861+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b60038040 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.861+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037070 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.861+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780cc120 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.861+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b780cc6e0 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.862+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7804f330 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.862+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b60045020 con 0x7f7b78159aa0 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.862+0000 7f7b6effd640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b5806b4d0 0x7f7b5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.862+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b600abb20 con 0x7f7b78159aa0 2024-03-06T07:30:06.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.862+0000 7f7b7659c640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b5806b4d0 0x7f7b5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.863+0000 7f7b7659c640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b5806b4d0 0x7f7b5806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7b68007b80 tx=0x7f7b68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:06.867+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b6004c030 con 0x7f7b78159aa0 2024-03-06T07:30:07.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.039+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b7804f330 con 0x7f7b78159aa0 2024-03-06T07:30:07.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.041+0000 7f7b6effd640 1 -- 172.21.15.122:0/1736105632 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16692 (secure 0 0 0) 0x7f7b6007aa30 con 0x7f7b78159aa0 2024-03-06T07:30:07.043 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:07.043 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":138,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:05.770348+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.042+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b5806b4d0 msgr2=0x7f7b5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.042+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b5806b4d0 0x7f7b5806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7b68007b80 tx=0x7f7b68009040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 msgr2=0x7f7b78160870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 0x7f7b78160870 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f7b60005040 tx=0x7f7b600012d0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 shutdown_connections 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7b5806b4d0 0x7f7b5806d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7b78160db0 0x7f7b780b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b78159aa0 0x7f7b78160870 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 --2- 172.21.15.122:0/1736105632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b64004800 0x7f7b780cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 >> 172.21.15.122:0/1736105632 conn(0x7f7b780d3cd0 msgr2=0x7f7b780b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 shutdown_connections 2024-03-06T07:30:07.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.043+0000 7f7b7ceeb640 1 -- 172.21.15.122:0/1736105632 wait complete. 2024-03-06T07:30:07.056 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:07.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.157+0000 7fc40046b640 1 Processor -- start 2024-03-06T07:30:07.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.157+0000 7fc40046b640 1 -- start start 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc40046b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc3f80cf0d0 0x7fc3f806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc40046b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80b3320 con 0x7fc3f815a220 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc40046b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80728d0 con 0x7fc3f80719e0 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc40046b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f80b3590 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fea08640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc3f80cf0d0 0x7fc3f806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fe207640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc3f815a220 0x7fc3f80712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45216/0 (socket says 172.21.15.122:45216) 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fea08640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc3f80cf0d0 0x7fc3f806d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60324/0 (socket says 172.21.15.122:60324) 2024-03-06T07:30:07.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fe207640 1 -- 172.21.15.122:0/2460498093 learned_addr learned my addr 172.21.15.122:0/2460498093 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:07.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fea08640 1 -- 172.21.15.122:0/2460498093 >> v1:172.21.15.169:6789/0 conn(0x7fc3f80719e0 legacy=0x7fc3f806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fea08640 1 -- 172.21.15.122:0/2460498093 >> v1:172.21.15.122:6789/0 conn(0x7fc3f815a220 legacy=0x7fc3f80712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fd205640 1 -- 172.21.15.122:0/2460498093 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937038172 0 0) 0x7fc3f80b3320 con 0x7fc3f815a220 2024-03-06T07:30:07.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.158+0000 7fc3fea08640 1 -- 172.21.15.122:0/2460498093 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80cf520 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.159+0000 7fc3fea08640 1 --2- 172.21.15.122:0/2460498093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f806d100 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f002cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ab78ff767f15a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.159+0000 7fc3fd205640 1 -- 172.21.15.122:0/2460498093 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f003a070 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.159+0000 7fc3fd205640 1 -- 172.21.15.122:0/2460498093 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f002dce0 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.159+0000 7fc3fd205640 1 -- 172.21.15.122:0/2460498093 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0035660 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.159+0000 7fc40046b640 1 -- 172.21.15.122:0/2460498093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 msgr2=0x7fc3f806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.159+0000 7fc40046b640 1 --2- 172.21.15.122:0/2460498093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f806d100 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f002cf30 comp rx=0 tx=0).stop 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.160+0000 7fc40046b640 1 -- 172.21.15.122:0/2460498093 shutdown_connections 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.160+0000 7fc40046b640 1 --2- 172.21.15.122:0/2460498093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f806d100 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.160+0000 7fc40046b640 1 -- 172.21.15.122:0/2460498093 >> 172.21.15.122:0/2460498093 conn(0x7fc3f806b190 msgr2=0x7fc3f806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.160+0000 7fc40046b640 1 -- 172.21.15.122:0/2460498093 shutdown_connections 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.160+0000 7fc40046b640 1 -- 172.21.15.122:0/2460498093 wait complete. 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 Processor -- start 2024-03-06T07:30:07.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 -- start start 2024-03-06T07:30:07.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3f80719e0 0x7fc3f80c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f8157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3f815a220 0x7fc3f8158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8072530 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8072030 con 0x7fc3f80719e0 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.161+0000 7fc40046b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80726b0 con 0x7fc3f815a220 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f8157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f8157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60336/0 (socket says 172.21.15.122:60336) 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 -- 172.21.15.122:0/2264980784 learned_addr learned my addr 172.21.15.122:0/2264980784 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 -- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3f815a220 msgr2=0x7fc3f8158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3f815a220 0x7fc3f8158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 -- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3f80719e0 msgr2=0x7fc3f80c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fe207640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3f80719e0 0x7fc3f80c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3f80719e0 0x7fc3f80c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 -- 172.21.15.122:0/2264980784 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80c4d30 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fe207640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3f80719e0 0x7fc3f80c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:07.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3fda06640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f8157c50 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c410 tx=0x7fc3e800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8017070 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e80035f0 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f80c5010 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8003a20 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.162+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f80c55d0 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.163+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f80732b0 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.164+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3e801a070 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.164+0000 7fc3eeffd640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.164+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3e8089650 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.164+0000 7fc3fe207640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.165+0000 7fc3fe207640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4000f60 tx=0x7fc3f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.168+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3e8058560 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.380+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc3f8068ff0 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.432+0000 7fc3eeffd640 1 -- 172.21.15.122:0/2264980784 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc3e8057e60 con 0x7fc3f80cf0d0 2024-03-06T07:30:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3dc06b4d0 msgr2=0x7fc3dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4000f60 tx=0x7fc3f4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 msgr2=0x7fc3f8157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f8157c50 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c410 tx=0x7fc3e800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 shutdown_connections 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc3dc06b4d0 0x7fc3dc06d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc3f815a220 0x7fc3f8158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f80cf0d0 0x7fc3f8157c50 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 --2- 172.21.15.122:0/2264980784 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3f80719e0 0x7fc3f80c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 >> 172.21.15.122:0/2264980784 conn(0x7fc3f806b190 msgr2=0x7fc3f806d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 shutdown_connections 2024-03-06T07:30:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.435+0000 7fc40046b640 1 -- 172.21.15.122:0/2264980784 wait complete. 2024-03-06T07:30:07.449 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:07.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.552+0000 7fbc0a1e2640 1 Processor -- start 2024-03-06T07:30:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.552+0000 7fbc0a1e2640 1 -- start start 2024-03-06T07:30:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc0a1e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc040d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc0a1e2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbc040d2480 0x7fbc040b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc0a1e2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc041618b0 con 0x7fbc040d1910 2024-03-06T07:30:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc0a1e2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc040cf400 con 0x7fbc040d2480 2024-03-06T07:30:07.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc0a1e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040cf6a0 con 0x7fbc041614d0 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbc040d2480 0x7fbc040b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc040d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbc040d2480 0x7fbc040b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:51158/0 (socket says 172.21.15.122:51158) 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 -- 172.21.15.122:0/1437055034 learned_addr learned my addr 172.21.15.122:0/1437055034 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 -- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 msgr2=0x7fbc040d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 --2- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc040d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 -- 172.21.15.122:0/1437055034 >> v1:172.21.15.169:6789/0 conn(0x7fbc040d1910 legacy=0x7fbc040d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.556 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.553+0000 7fbc03fff640 1 -- 172.21.15.122:0/1437055034 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040cf980 con 0x7fbc040d2480 2024-03-06T07:30:07.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.554+0000 7fbc03fff640 1 --2- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b0e30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009e50 tx=0x7fbbfc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aaf4876e8c0346c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.554+0000 7fbc027fc640 1 -- 172.21.15.122:0/1437055034 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc02cb00 con 0x7fbc040d2480 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.554+0000 7fbc027fc640 1 -- 172.21.15.122:0/1437055034 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc02cca0 con 0x7fbc040d2480 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc027fc640 1 -- 172.21.15.122:0/1437055034 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc034610 con 0x7fbc040d2480 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 msgr2=0x7fbc040b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b0e30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009e50 tx=0x7fbbfc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/1437055034 shutdown_connections 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b0e30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/1437055034 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc040d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/1437055034 >> 172.21.15.122:0/1437055034 conn(0x7fbc040d3d40 msgr2=0x7fbc040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/1437055034 shutdown_connections 2024-03-06T07:30:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.555+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/1437055034 wait complete. 2024-03-06T07:30:07.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.556+0000 7fbc0a1e2640 1 Processor -- start 2024-03-06T07:30:07.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.556+0000 7fbc0a1e2640 1 -- start start 2024-03-06T07:30:07.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc0a1e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 0x7fbc040a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc0a1e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc0a1e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc0415de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc0a1e2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040b1370 con 0x7fbc040d1910 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc0a1e2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040b0910 con 0x7fbc040d2480 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc0a1e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040b14f0 con 0x7fbc041614d0 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 0x7fbc040a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 0x7fbc040a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60348/0 (socket says 172.21.15.122:60348) 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc037fe640 1 -- 172.21.15.122:0/332205512 learned_addr learned my addr 172.21.15.122:0/332205512 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc02ffd640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc03fff640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc0415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc037fe640 1 -- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 msgr2=0x7fbc0415de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.557+0000 7fbc037fe640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc0415de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbc037fe640 1 -- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 msgr2=0x7fbc040b9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbc037fe640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbc037fe640 1 -- 172.21.15.122:0/332205512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040a18d0 con 0x7fbc040d1910 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbc037fe640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 0x7fbc040a1210 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00e870 tx=0x7fbbec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec017070 con 0x7fbc040d1910 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbec00c940 con 0x7fbc040d1910 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc040cae10 con 0x7fbc040d1910 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec00cd30 con 0x7fbc040d1910 2024-03-06T07:30:07.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.558+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc040cb3d0 con 0x7fbc040d1910 2024-03-06T07:30:07.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.559+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc04068ff0 con 0x7fbc040d1910 2024-03-06T07:30:07.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.559+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbec01a070 con 0x7fbc040d1910 2024-03-06T07:30:07.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.560+0000 7fbbfbfff640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbbe406b4d0 0x7fbbe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.560+0000 7fbc02ffd640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbbe406b4d0 0x7fbbe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.560+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbec08a450 con 0x7fbc040d1910 2024-03-06T07:30:07.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.560+0000 7fbc02ffd640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbbe406b4d0 0x7fbbe406d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbc040b9df0 tx=0x7fbbf4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.564+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbec058c00 con 0x7fbc040d1910 2024-03-06T07:30:07.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:07.775+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbc040d0a50 con 0x7fbc040d1910 2024-03-06T07:30:08.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.478+0000 7fbbfbfff640 1 -- 172.21.15.122:0/332205512 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v90) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbbec05cca0 con 0x7fbc040d1910 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.480+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbbe406b4d0 msgr2=0x7fbbe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.480+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbbe406b4d0 0x7fbbe406d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbc040b9df0 tx=0x7fbbf4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.480+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 msgr2=0x7fbc040a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.480+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 0x7fbc040a1210 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00e870 tx=0x7fbbec00ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.480+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 shutdown_connections 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.480+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbbe406b4d0 0x7fbbe406d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.481+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc041614d0 0x7fbc0415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.481+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc040d2480 0x7fbc040b9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.481+0000 7fbc0a1e2640 1 --2- 172.21.15.122:0/332205512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc040d1910 0x7fbc040a1210 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.481+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 >> 172.21.15.122:0/332205512 conn(0x7fbc040d3d40 msgr2=0x7fbc040d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.481+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 shutdown_connections 2024-03-06T07:30:08.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.481+0000 7fbc0a1e2640 1 -- 172.21.15.122:0/332205512 wait complete. 2024-03-06T07:30:08.494 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:08.494 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.592+0000 7f63cb393640 1 Processor -- start 2024-03-06T07:30:08.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.593+0000 7f63cb393640 1 -- start start 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.593+0000 7f63cb393640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63c40cce40 0x7f63c415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.593+0000 7f63cb393640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4071f40 con 0x7f63c41515e0 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.593+0000 7f63cb393640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4070a70 con 0x7f63c40cd7a0 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.593+0000 7f63cb393640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c40cd2f0 con 0x7f63c40cce40 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c892e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63c40cce40 0x7f63c415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c892e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63c40cce40 0x7f63c415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60360/0 (socket says 172.21.15.122:60360) 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c892e640 1 -- 172.21.15.122:0/793060594 learned_addr learned my addr 172.21.15.122:0/793060594 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c912f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f63c41515e0 0x7f63c4071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45232/0 (socket says 172.21.15.122:45232) 2024-03-06T07:30:08.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c892e640 1 -- 172.21.15.122:0/793060594 >> v1:172.21.15.169:6789/0 conn(0x7f63c40cd7a0 legacy=0x7f63c40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c3fff640 1 -- 172.21.15.122:0/793060594 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602202337 0 0) 0x7f63c4071f40 con 0x7f63c41515e0 2024-03-06T07:30:08.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c892e640 1 -- 172.21.15.122:0/793060594 >> v1:172.21.15.122:6789/0 conn(0x7f63c41515e0 legacy=0x7f63c4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.594+0000 7f63c892e640 1 -- 172.21.15.122:0/793060594 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4070d50 con 0x7f63c40cce40 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.595+0000 7f63c892e640 1 --2- 172.21.15.122:0/793060594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c415b430 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009870 tx=0x7f63b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edba729f20a8a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.595+0000 7f63c3fff640 1 -- 172.21.15.122:0/793060594 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8034070 con 0x7f63c40cce40 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.595+0000 7f63c3fff640 1 -- 172.21.15.122:0/793060594 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8030860 con 0x7f63c40cce40 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.595+0000 7f63c3fff640 1 -- 172.21.15.122:0/793060594 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8030c30 con 0x7f63c40cce40 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 -- 172.21.15.122:0/793060594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 msgr2=0x7f63c415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 --2- 172.21.15.122:0/793060594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c415b430 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009870 tx=0x7f63b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 -- 172.21.15.122:0/793060594 shutdown_connections 2024-03-06T07:30:08.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 --2- 172.21.15.122:0/793060594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c415b430 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 -- 172.21.15.122:0/793060594 >> 172.21.15.122:0/793060594 conn(0x7f63c40cf8a0 msgr2=0x7f63c40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 -- 172.21.15.122:0/793060594 shutdown_connections 2024-03-06T07:30:08.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.596+0000 7f63cb393640 1 -- 172.21.15.122:0/793060594 wait complete. 2024-03-06T07:30:08.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.597+0000 7f63cb393640 1 Processor -- start 2024-03-06T07:30:08.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.597+0000 7f63cb393640 1 -- start start 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c41f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63c40cd7a0 0x7f63c4147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63c41515e0 0x7f63c40cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40b3f90 con 0x7f63c40cce40 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4071310 con 0x7f63c40cd7a0 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40b4110 con 0x7f63c41515e0 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c41f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c41f3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60364/0 (socket says 172.21.15.122:60364) 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 -- 172.21.15.122:0/725625906 learned_addr learned my addr 172.21.15.122:0/725625906 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c892e640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63c40cd7a0 0x7f63c4147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c9930640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63c41515e0 0x7f63c40cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 -- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63c41515e0 msgr2=0x7f63c40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63c41515e0 0x7f63c40cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 -- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63c40cd7a0 msgr2=0x7f63c4147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63c40cd7a0 0x7f63c4147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 -- 172.21.15.122:0/725625906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4144fe0 con 0x7f63c40cce40 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c892e640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63c40cd7a0 0x7f63c4147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c912f640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c41f3730 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00c410 tx=0x7f63bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0074e0 con 0x7f63c40cce40 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc007680 con 0x7f63c40cce40 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c41452c0 con 0x7f63c40cce40 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.598+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc004a00 con 0x7f63c40cce40 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.599+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4145880 con 0x7f63c40cce40 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.600+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4068840 con 0x7f63c40cce40 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.600+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63bc01a020 con 0x7f63c40cce40 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.601+0000 7f63c1ffb640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63b006b4d0 0x7f63b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.601+0000 7f63c892e640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63b006b4d0 0x7f63b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.601+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63bc089a50 con 0x7f63c40cce40 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.601+0000 7f63c892e640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63b006b4d0 0x7f63b006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f63b8036000 tx=0x7f63b803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.605+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63bc010300 con 0x7f63c40cce40 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.820+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c404f330 con 0x7f63c40cce40 2024-03-06T07:30:08.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.820+0000 7f63c1ffb640 1 -- 172.21.15.122:0/725625906 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f63bc0588d0 con 0x7f63c40cce40 2024-03-06T07:30:08.823 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:08.823 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":90,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3471691317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3471691317},{"type":"v1","addr":"172.21.15.169:6834","nonce":3471691317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2660610501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2660610501},{"type":"v1","addr":"172.21.15.169:6835","nonce":2660610501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:07.777233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9321},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:30:08.823 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 90 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63b006b4d0 msgr2=0x7f63b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63b006b4d0 0x7f63b006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f63b8036000 tx=0x7f63b803b040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 msgr2=0x7f63c41f3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c41f3730 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00c410 tx=0x7f63bc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 shutdown_connections 2024-03-06T07:30:08.825 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63b006b4d0 0x7f63b006d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63c41515e0 0x7f63c40cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63c40cd7a0 0x7f63c4147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 --2- 172.21.15.122:0/725625906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c40cce40 0x7f63c41f3730 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 >> 172.21.15.122:0/725625906 conn(0x7f63c40cf8a0 msgr2=0x7f63c40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.822+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 shutdown_connections 2024-03-06T07:30:08.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.823+0000 7f63cb393640 1 -- 172.21.15.122:0/725625906 wait complete. 2024-03-06T07:30:08.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:08.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.938+0000 7f7ff2e9d640 1 Processor -- start 2024-03-06T07:30:08.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.938+0000 7f7ff2e9d640 1 -- start start 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff2e9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff2e9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7fec0d29c0 0x7f7fec0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff2e9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fec0b3430 con 0x7f7fec0cf830 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff2e9d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec0b31e0 con 0x7f7fec0d29c0 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff2e9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0b3d60 con 0x7f7fec15dfe0 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7fec0d29c0 0x7f7fec0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7fec0d29c0 0x7f7fec0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60378/0 (socket says 172.21.15.122:60378) 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 -- 172.21.15.122:0/2848017072 learned_addr learned my addr 172.21.15.122:0/2848017072 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff0c39640 1 --2- 172.21.15.122:0/2848017072 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 -- 172.21.15.122:0/2848017072 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 msgr2=0x7f7fec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 --2- 172.21.15.122:0/2848017072 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 -- 172.21.15.122:0/2848017072 >> v1:172.21.15.169:6789/0 conn(0x7f7fec0cf830 legacy=0x7f7fec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.939+0000 7f7ff143a640 1 -- 172.21.15.122:0/2848017072 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0b3710 con 0x7f7fec0d29c0 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.940+0000 7f7ff143a640 1 --2- 172.21.15.122:0/2848017072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0d29c0 0x7f7fec0b41f0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009870 tx=0x7f7fdc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa477272e83b7f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.940+0000 7f7feb7fe640 1 -- 172.21.15.122:0/2848017072 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc034070 con 0x7f7fec0d29c0 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.940+0000 7f7feb7fe640 1 -- 172.21.15.122:0/2848017072 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc030840 con 0x7f7fec0d29c0 2024-03-06T07:30:08.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.940+0000 7f7feb7fe640 1 -- 172.21.15.122:0/2848017072 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc030c10 con 0x7f7fec0d29c0 2024-03-06T07:30:08.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/2848017072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0d29c0 msgr2=0x7f7fec0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/2848017072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0d29c0 0x7f7fec0b41f0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009870 tx=0x7f7fdc02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/2848017072 shutdown_connections 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/2848017072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0d29c0 0x7f7fec0b41f0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/2848017072 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/2848017072 >> 172.21.15.122:0/2848017072 conn(0x7f7fec0d3cd0 msgr2=0x7f7fec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/2848017072 shutdown_connections 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.941+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/2848017072 wait complete. 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.942+0000 7f7ff2e9d640 1 Processor -- start 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.942+0000 7f7ff2e9d640 1 -- start start 2024-03-06T07:30:08.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff2e9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 0x7f7fec155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff2e9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fec0d29c0 0x7f7fec15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff2e9d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff2e9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0d1df0 con 0x7f7fec0cf830 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff2e9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0d1c70 con 0x7f7fec0d29c0 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff2e9d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec15e3c0 con 0x7f7fec15dfe0 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 0x7f7fec155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 0x7f7fec155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60382/0 (socket says 172.21.15.122:60382) 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 -- 172.21.15.122:0/1657606580 learned_addr learned my addr 172.21.15.122:0/1657606580 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff143a640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7febfff640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fec0d29c0 0x7f7fec15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 -- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 msgr2=0x7f7fec15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec15ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 -- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fec0d29c0 msgr2=0x7f7fec15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fec0d29c0 0x7f7fec15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.943+0000 7f7ff0c39640 1 -- 172.21.15.122:0/1657606580 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec1563b0 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.944+0000 7f7ff0c39640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 0x7f7fec155b90 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4007e40 tx=0x7f7fe400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.944+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4014620 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.944+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe4019070 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.944+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec0b9a50 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.944+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4014a00 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.944+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec15f5e0 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.945+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fec04f330 con 0x7f7fec0cf830 2024-03-06T07:30:08.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.945+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fe4014ba0 con 0x7f7fec0cf830 2024-03-06T07:30:08.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.946+0000 7f7fe97fa640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7fcc06b4d0 0x7f7fcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.946+0000 7f7febfff640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7fcc06b4d0 0x7f7fcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.946+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fe408aa00 con 0x7f7fec0cf830 2024-03-06T07:30:08.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.946+0000 7f7febfff640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7fcc06b4d0 0x7f7fcc06d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0b5030 tx=0x7f7fd4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:08.950+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fe4059910 con 0x7f7fec0cf830 2024-03-06T07:30:09.129 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.127+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fec15f7e0 con 0x7f7fec0cf830 2024-03-06T07:30:09.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.128+0000 7f7fe97fa640 1 -- 172.21.15.122:0/1657606580 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16679 (secure 0 0 0) 0x7f7fe4059210 con 0x7f7fec0cf830 2024-03-06T07:30:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:09.130 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":139,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:07.425954+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:09.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7fcc06b4d0 msgr2=0x7f7fcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7fcc06b4d0 0x7f7fcc06d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0b5030 tx=0x7f7fd4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:09.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 msgr2=0x7f7fec155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 0x7f7fec155b90 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4007e40 tx=0x7f7fe400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 shutdown_connections 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7fcc06b4d0 0x7f7fcc06d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7fec15dfe0 0x7f7fec15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fec0d29c0 0x7f7fec15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 --2- 172.21.15.122:0/1657606580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fec0cf830 0x7f7fec155b90 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 >> 172.21.15.122:0/1657606580 conn(0x7f7fec0d3cd0 msgr2=0x7f7fec06ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 shutdown_connections 2024-03-06T07:30:09.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.130+0000 7f7ff2e9d640 1 -- 172.21.15.122:0/1657606580 wait complete. 2024-03-06T07:30:09.143 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:09.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:09.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.242+0000 7fd0fbc76640 1 Processor -- start 2024-03-06T07:30:09.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 -- start start 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd0f415dfe0 0x7fd0f40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd0f40cf830 0x7fd0f40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f40b31e0 con 0x7fd0f40cf830 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f40b3d60 con 0x7fd0f415dfe0 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.243+0000 7fd0fbc76640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40b3430 con 0x7fd0f40d27f0 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd0f40cf830 0x7fd0f40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd0f40cf830 0x7fd0f40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60396/0 (socket says 172.21.15.122:60396) 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 -- 172.21.15.122:0/2400130647 learned_addr learned my addr 172.21.15.122:0/2400130647 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0fa213640 1 --2- 172.21.15.122:0/2400130647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 -- 172.21.15.122:0/2400130647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 msgr2=0x7fd0f40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9a12640 1 --2- 172.21.15.122:0/2400130647 >> v2:172.21.15.169:3300/0 conn(0x7fd0f415dfe0 0x7fd0f40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 --2- 172.21.15.122:0/2400130647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 -- 172.21.15.122:0/2400130647 >> v2:172.21.15.169:3300/0 conn(0x7fd0f415dfe0 msgr2=0x7fd0f40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 --2- 172.21.15.122:0/2400130647 >> v2:172.21.15.169:3300/0 conn(0x7fd0f415dfe0 0x7fd0f40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.244+0000 7fd0f9211640 1 -- 172.21.15.122:0/2400130647 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f40b3710 con 0x7fd0f40cf830 2024-03-06T07:30:09.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0f9211640 1 --2- 172.21.15.122:0/2400130647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f40cf830 0x7fd0f40d22b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4067990 tx=0x7fd0e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c063f696c420174 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0f8a10640 1 -- 172.21.15.122:0/2400130647 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4035020 con 0x7fd0f40cf830 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0f8a10640 1 -- 172.21.15.122:0/2400130647 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e4031870 con 0x7fd0f40cf830 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0f8a10640 1 -- 172.21.15.122:0/2400130647 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4031c40 con 0x7fd0f40cf830 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2400130647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f40cf830 msgr2=0x7fd0f40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2400130647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f40cf830 0x7fd0f40d22b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4067990 tx=0x7fd0e402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2400130647 shutdown_connections 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2400130647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f40cf830 0x7fd0f40d22b0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2400130647 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2400130647 >> v2:172.21.15.169:3300/0 conn(0x7fd0f415dfe0 0x7fd0f40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2400130647 >> 172.21.15.122:0/2400130647 conn(0x7fd0f40d3cd0 msgr2=0x7fd0f40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2400130647 shutdown_connections 2024-03-06T07:30:09.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.245+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2400130647 wait complete. 2024-03-06T07:30:09.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.247+0000 7fd0fbc76640 1 Processor -- start 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.247+0000 7fd0fbc76640 1 -- start start 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fbc76640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f40cf830 0x7fd0f40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fbc76640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fbc76640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 0x7fd0f40b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fbc76640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40d1df0 con 0x7fd0f415dfe0 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fbc76640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40d1c70 con 0x7fd0f40cf830 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fbc76640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f415e3c0 con 0x7fd0f40d27f0 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.248+0000 7fd0fa213640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 0x7fd0f40b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 0x7fd0f40b14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60410/0 (socket says 172.21.15.122:60410) 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 -- 172.21.15.122:0/2798121012 learned_addr learned my addr 172.21.15.122:0/2798121012 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0f9211640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0f9a12640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f40cf830 0x7fd0f40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 -- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 msgr2=0x7fd0f415a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f415a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 -- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f40cf830 msgr2=0x7fd0f40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f40cf830 0x7fd0f40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 -- 172.21.15.122:0/2798121012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f40cd610 con 0x7fd0f415dfe0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fa213640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 0x7fd0f40b14a0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00cd20 tx=0x7fd0ec00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec003b90 con 0x7fd0f415dfe0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec003d30 con 0x7fd0f415dfe0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f40cd8f0 con 0x7fd0f415dfe0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec004240 con 0x7fd0f415dfe0 2024-03-06T07:30:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.249+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f4161590 con 0x7fd0f415dfe0 2024-03-06T07:30:09.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.250+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f40b0da0 con 0x7fd0f415dfe0 2024-03-06T07:30:09.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.251+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0ec0043e0 con 0x7fd0f415dfe0 2024-03-06T07:30:09.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.251+0000 7fd0ea7fc640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd0d806b4d0 0x7fd0d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.251+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0ec08aca0 con 0x7fd0f415dfe0 2024-03-06T07:30:09.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.251+0000 7fd0f9a12640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd0d806b4d0 0x7fd0d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.251+0000 7fd0f9a12640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd0d806b4d0 0x7fd0d806d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0006fd0 tx=0x7fd0f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.255+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0f40b0da0 con 0x7fd0f415dfe0 2024-03-06T07:30:09.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.429+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0f404f330 con 0x7fd0d806b4d0 2024-03-06T07:30:09.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.435+0000 7fd0ea7fc640 1 -- 172.21.15.122:0/2798121012 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240258 (secure 0 0 0) 0x7fd0f404f330 con 0x7fd0d806b4d0 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd0d806b4d0 msgr2=0x7fd0d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd0d806b4d0 0x7fd0d806d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0006fd0 tx=0x7fd0f0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 msgr2=0x7fd0f40b14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 0x7fd0f40b14a0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00cd20 tx=0x7fd0ec00a660 comp rx=0 tx=0).stop 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 shutdown_connections 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd0d806b4d0 0x7fd0d806d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0f415dfe0 0x7fd0f40b14a0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd0f40d27f0 0x7fd0f415a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 --2- 172.21.15.122:0/2798121012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f40cf830 0x7fd0f40ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 >> 172.21.15.122:0/2798121012 conn(0x7fd0f40d3cd0 msgr2=0x7fd0f40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 shutdown_connections 2024-03-06T07:30:09.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.438+0000 7fd0fbc76640 1 -- 172.21.15.122:0/2798121012 wait complete. 2024-03-06T07:30:09.451 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-06T07:30:09.426932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330480,"kb_used_data":14460,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644240,"statfs":{"total":773094113280,"available":772755701760,"internally_reserved":0,"allocated":14807040,"data_stored":7110279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.014410"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417086+0000","last_change":"2024-03-06T07:30:04.417086+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417086+0000","last_undegraded":"2024-03-06T07:30:04.417086+0000","last_fullsized":"2024-03-06T07:30:04.417086+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417956+0000","last_change":"2024-03-06T07:30:04.417956+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417956+0000","last_undegraded":"2024-03-06T07:30:04.417956+0000","last_fullsized":"2024-03-06T07:30:04.417956+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420390+0000","last_change":"2024-03-06T07:30:04.420390+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420390+0000","last_undegraded":"2024-03-06T07:30:04.420390+0000","last_fullsized":"2024-03-06T07:30:04.420390+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420351+0000","last_change":"2024-03-06T07:30:04.420351+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420351+0000","last_undegraded":"2024-03-06T07:30:04.420351+0000","last_fullsized":"2024-03-06T07:30:04.420351+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416826+0000","last_change":"2024-03-06T07:30:04.416826+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416826+0000","last_undegraded":"2024-03-06T07:30:04.416826+0000","last_fullsized":"2024-03-06T07:30:04.416826+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417090+0000","last_change":"2024-03-06T07:30:04.417090+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417090+0000","last_undegraded":"2024-03-06T07:30:04.417090+0000","last_fullsized":"2024-03-06T07:30:04.417090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416358+0000","last_change":"2024-03-06T07:30:04.416358+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416358+0000","last_undegraded":"2024-03-06T07:30:04.416358+0000","last_fullsized":"2024-03-06T07:30:04.416358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420391+0000","last_change":"2024-03-06T07:30:04.420391+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420391+0000","last_undegraded":"2024-03-06T07:30:04.420391+0000","last_fullsized":"2024-03-06T07:30:04.420391+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417129+0000","last_change":"2024-03-06T07:30:04.417129+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417129+0000","last_undegraded":"2024-03-06T07:30:04.417129+0000","last_fullsized":"2024-03-06T07:30:04.417129+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416322+0000","last_change":"2024-03-06T07:30:04.416322+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416322+0000","last_undegraded":"2024-03-06T07:30:04.416322+0000","last_fullsized":"2024-03-06T07:30:04.416322+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416196+0000","last_change":"2024-03-06T07:30:04.416196+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416196+0000","last_undegraded":"2024-03-06T07:30:04.416196+0000","last_fullsized":"2024-03-06T07:30:04.416196+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414439+0000","last_change":"2024-03-06T07:30:04.414439+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414439+0000","last_undegraded":"2024-03-06T07:30:04.414439+0000","last_fullsized":"2024-03-06T07:30:04.414439+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415854+0000","last_change":"2024-03-06T07:30:04.415854+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415854+0000","last_undegraded":"2024-03-06T07:30:04.415854+0000","last_fullsized":"2024-03-06T07:30:04.415854+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417449+0000","last_change":"2024-03-06T07:30:04.417449+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417449+0000","last_undegraded":"2024-03-06T07:30:04.417449+0000","last_fullsized":"2024-03-06T07:30:04.417449+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417586+0000","last_change":"2024-03-06T07:30:04.417586+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417586+0000","last_undegraded":"2024-03-06T07:30:04.417586+0000","last_fullsized":"2024-03-06T07:30:04.417586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415865+0000","last_change":"2024-03-06T07:30:04.415865+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415865+0000","last_undegraded":"2024-03-06T07:30:04.415865+0000","last_fullsized":"2024-03-06T07:30:04.415865+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417256+0000","last_change":"2024-03-06T07:30:04.417256+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417256+0000","last_undegraded":"2024-03-06T07:30:04.417256+0000","last_fullsized":"2024-03-06T07:30:04.417256+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417074+0000","last_change":"2024-03-06T07:30:04.417074+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417074+0000","last_undegraded":"2024-03-06T07:30:04.417074+0000","last_fullsized":"2024-03-06T07:30:04.417074+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417249+0000","last_change":"2024-03-06T07:30:04.417249+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417249+0000","last_undegraded":"2024-03-06T07:30:04.417249+0000","last_fullsized":"2024-03-06T07:30:04.417249+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416617+0000","last_change":"2024-03-06T07:30:04.416617+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416617+0000","last_undegraded":"2024-03-06T07:30:04.416617+0000","last_fullsized":"2024-03-06T07:30:04.416617+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414450+0000","last_change":"2024-03-06T07:30:04.414450+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414450+0000","last_undegraded":"2024-03-06T07:30:04.414450+0000","last_fullsized":"2024-03-06T07:30:04.414450+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416361+0000","last_change":"2024-03-06T07:30:04.416361+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416361+0000","last_undegraded":"2024-03-06T07:30:04.416361+0000","last_fullsized":"2024-03-06T07:30:04.416361+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414123+0000","last_change":"2024-03-06T07:30:04.414123+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414123+0000","last_undegraded":"2024-03-06T07:30:04.414123+0000","last_fullsized":"2024-03-06T07:30:04.414123+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414132+0000","last_change":"2024-03-06T07:30:04.414132+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414132+0000","last_undegraded":"2024-03-06T07:30:04.414132+0000","last_fullsized":"2024-03-06T07:30:04.414132+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416640+0000","last_change":"2024-03-06T07:30:04.416640+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416640+0000","last_undegraded":"2024-03-06T07:30:04.416640+0000","last_fullsized":"2024-03-06T07:30:04.416640+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416321+0000","last_change":"2024-03-06T07:30:04.416321+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416321+0000","last_undegraded":"2024-03-06T07:30:04.416321+0000","last_fullsized":"2024-03-06T07:30:04.416321+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417160+0000","last_change":"2024-03-06T07:30:04.417160+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417160+0000","last_undegraded":"2024-03-06T07:30:04.417160+0000","last_fullsized":"2024-03-06T07:30:04.417160+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414459+0000","last_change":"2024-03-06T07:30:04.414459+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414459+0000","last_undegraded":"2024-03-06T07:30:04.414459+0000","last_fullsized":"2024-03-06T07:30:04.414459+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":391,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-03-06T07:30:04.417517+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:30:04.417517+0000","last_peered":"2024-03-06T07:30:04.417517+0000","last_clean":"2024-03-06T07:30:04.417517+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:30:04.417517+0000","last_undegraded":"2024-03-06T07:30:04.417517+0000","last_fullsized":"2024-03-06T07:30:04.417517+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574952,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574953,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":736103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:09.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:09.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.590+0000 7f805e17a640 1 Processor -- start 2024-03-06T07:30:09.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.590+0000 7f805e17a640 1 -- start start 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f805e17a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 0x7f80580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f805e17a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f80580b41f0 0x7f80580b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f805e17a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80580b3430 con 0x7f805815dfe0 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f805e17a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80580d2c50 con 0x7f80580b41f0 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f805e17a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80580b31e0 con 0x7f80580d22b0 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f8057fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f80580b41f0 0x7f80580b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f8056ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 0x7f80580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f80577fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f805815dfe0 0x7f80580cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:51958/0 (socket says 172.21.15.122:51958) 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f80577fe640 1 -- 172.21.15.122:0/2769842997 learned_addr learned my addr 172.21.15.122:0/2769842997 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f8057fff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f80580b41f0 0x7f80580b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:51202/0 (socket says 172.21.15.122:51202) 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.591+0000 7f8056ffd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 0x7f80580d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:35192/0 (socket says 172.21.15.122:35192) 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.592+0000 7f8057fff640 1 -- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 msgr2=0x7f80580d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.592+0000 7f8057fff640 1 --2- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 0x7f80580d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.592+0000 7f8057fff640 1 -- 172.21.15.122:0/2769842997 >> v1:172.21.15.169:6789/0 conn(0x7f805815dfe0 legacy=0x7f80580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.592+0000 7f8057fff640 1 -- 172.21.15.122:0/2769842997 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580b3710 con 0x7f80580b41f0 2024-03-06T07:30:09.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.592+0000 7f8056ffd640 1 --2- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 0x7f80580d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:09.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.592+0000 7f8057fff640 1 --2- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580b41f0 0x7f80580b4650 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8050009d90 tx=0x7f805002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=556bbb0a51dbcec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f80567fc640 1 -- 172.21.15.122:0/2769842997 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050039070 con 0x7f80580b41f0 2024-03-06T07:30:09.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f80567fc640 1 -- 172.21.15.122:0/2769842997 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805002cc50 con 0x7f80580b41f0 2024-03-06T07:30:09.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f80567fc640 1 -- 172.21.15.122:0/2769842997 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050034440 con 0x7f80580b41f0 2024-03-06T07:30:09.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 -- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580b41f0 msgr2=0x7f80580b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 --2- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580b41f0 0x7f80580b4650 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8050009d90 tx=0x7f805002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 -- 172.21.15.122:0/2769842997 shutdown_connections 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 --2- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580b41f0 0x7f80580b4650 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 --2- 172.21.15.122:0/2769842997 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80580d22b0 0x7f80580d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 -- 172.21.15.122:0/2769842997 >> 172.21.15.122:0/2769842997 conn(0x7f80580d3cd0 msgr2=0x7f80580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 -- 172.21.15.122:0/2769842997 shutdown_connections 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.593+0000 7f805e17a640 1 -- 172.21.15.122:0/2769842997 wait complete. 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.594+0000 7f805e17a640 1 Processor -- start 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 -- start start 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 0x7f805815a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580d22b0 0x7f8058160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f805815dfe0 0x7f8058161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80580cfa00 con 0x7f80580b41f0 2024-03-06T07:30:09.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80580b3d60 con 0x7f80580d22b0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f805e17a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80580d1c70 con 0x7f805815dfe0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f80577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 0x7f805815a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f80577fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 0x7f805815a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60420/0 (socket says 172.21.15.122:60420) 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.595+0000 7f80577fe640 1 -- 172.21.15.122:0/237463559 learned_addr learned my addr 172.21.15.122:0/237463559 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f8057fff640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f805815dfe0 0x7f8058161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f8056ffd640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580d22b0 0x7f8058160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f80577fe640 1 -- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f805815dfe0 msgr2=0x7f8058161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f80577fe640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f805815dfe0 0x7f8058161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f80577fe640 1 -- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580d22b0 msgr2=0x7f8058160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f80577fe640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580d22b0 0x7f8058160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f80577fe640 1 -- 172.21.15.122:0/237463559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580cca40 con 0x7f80580b41f0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f80577fe640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 0x7f805815a260 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f804c00c3f0 tx=0x7f804c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c017070 con 0x7f80580b41f0 2024-03-06T07:30:09.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80580ccd20 con 0x7f80580b41f0 2024-03-06T07:30:09.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c005ce0 con 0x7f80580b41f0 2024-03-06T07:30:09.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0072f0 con 0x7f80580b41f0 2024-03-06T07:30:09.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.596+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80580cd2e0 con 0x7f80580b41f0 2024-03-06T07:30:09.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.597+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805804f330 con 0x7f80580b41f0 2024-03-06T07:30:09.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.598+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f804c01a070 con 0x7f80580b41f0 2024-03-06T07:30:09.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.598+0000 7f803ffff640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f804406b4d0 0x7f804406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.598+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f804c089550 con 0x7f80580b41f0 2024-03-06T07:30:09.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.598+0000 7f8056ffd640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f804406b4d0 0x7f804406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.599+0000 7f8056ffd640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f804406b4d0 0x7f804406d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f80580b9d80 tx=0x7f8048009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.603+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f804c057d00 con 0x7f80580b41f0 2024-03-06T07:30:09.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.776+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80580ca540 con 0x7f804406b4d0 2024-03-06T07:30:09.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.786+0000 7f803ffff640 1 -- 172.21.15.122:0/237463559 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240258 (secure 0 0 0) 0x7f80580ca540 con 0x7f804406b4d0 2024-03-06T07:30:09.791 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:09.792 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f804406b4d0 msgr2=0x7f804406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f804406b4d0 0x7f804406d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f80580b9d80 tx=0x7f8048009190 comp rx=0 tx=0).stop 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 msgr2=0x7f805815a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 0x7f805815a260 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f804c00c3f0 tx=0x7f804c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 shutdown_connections 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f804406b4d0 0x7f804406d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f805815dfe0 0x7f8058161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80580d22b0 0x7f8058160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 --2- 172.21.15.122:0/237463559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80580b41f0 0x7f805815a260 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 >> 172.21.15.122:0/237463559 conn(0x7f80580d3cd0 msgr2=0x7f80580d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 shutdown_connections 2024-03-06T07:30:09.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.790+0000 7f805e17a640 1 -- 172.21.15.122:0/237463559 wait complete. 2024-03-06T07:30:09.803 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-06T07:30:09.426932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330480,"kb_used_data":14460,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644240,"statfs":{"total":773094113280,"available":772755701760,"internally_reserved":0,"allocated":14807040,"data_stored":7110279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.014410"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417086+0000","last_change":"2024-03-06T07:30:04.417086+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417086+0000","last_undegraded":"2024-03-06T07:30:04.417086+0000","last_fullsized":"2024-03-06T07:30:04.417086+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417956+0000","last_change":"2024-03-06T07:30:04.417956+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417956+0000","last_undegraded":"2024-03-06T07:30:04.417956+0000","last_fullsized":"2024-03-06T07:30:04.417956+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420390+0000","last_change":"2024-03-06T07:30:04.420390+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420390+0000","last_undegraded":"2024-03-06T07:30:04.420390+0000","last_fullsized":"2024-03-06T07:30:04.420390+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420351+0000","last_change":"2024-03-06T07:30:04.420351+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420351+0000","last_undegraded":"2024-03-06T07:30:04.420351+0000","last_fullsized":"2024-03-06T07:30:04.420351+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416826+0000","last_change":"2024-03-06T07:30:04.416826+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416826+0000","last_undegraded":"2024-03-06T07:30:04.416826+0000","last_fullsized":"2024-03-06T07:30:04.416826+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417090+0000","last_change":"2024-03-06T07:30:04.417090+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417090+0000","last_undegraded":"2024-03-06T07:30:04.417090+0000","last_fullsized":"2024-03-06T07:30:04.417090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416358+0000","last_change":"2024-03-06T07:30:04.416358+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416358+0000","last_undegraded":"2024-03-06T07:30:04.416358+0000","last_fullsized":"2024-03-06T07:30:04.416358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420391+0000","last_change":"2024-03-06T07:30:04.420391+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420391+0000","last_undegraded":"2024-03-06T07:30:04.420391+0000","last_fullsized":"2024-03-06T07:30:04.420391+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417129+0000","last_change":"2024-03-06T07:30:04.417129+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417129+0000","last_undegraded":"2024-03-06T07:30:04.417129+0000","last_fullsized":"2024-03-06T07:30:04.417129+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416322+0000","last_change":"2024-03-06T07:30:04.416322+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416322+0000","last_undegraded":"2024-03-06T07:30:04.416322+0000","last_fullsized":"2024-03-06T07:30:04.416322+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416196+0000","last_change":"2024-03-06T07:30:04.416196+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416196+0000","last_undegraded":"2024-03-06T07:30:04.416196+0000","last_fullsized":"2024-03-06T07:30:04.416196+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414439+0000","last_change":"2024-03-06T07:30:04.414439+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414439+0000","last_undegraded":"2024-03-06T07:30:04.414439+0000","last_fullsized":"2024-03-06T07:30:04.414439+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415854+0000","last_change":"2024-03-06T07:30:04.415854+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415854+0000","last_undegraded":"2024-03-06T07:30:04.415854+0000","last_fullsized":"2024-03-06T07:30:04.415854+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417449+0000","last_change":"2024-03-06T07:30:04.417449+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417449+0000","last_undegraded":"2024-03-06T07:30:04.417449+0000","last_fullsized":"2024-03-06T07:30:04.417449+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417586+0000","last_change":"2024-03-06T07:30:04.417586+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417586+0000","last_undegraded":"2024-03-06T07:30:04.417586+0000","last_fullsized":"2024-03-06T07:30:04.417586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415865+0000","last_change":"2024-03-06T07:30:04.415865+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415865+0000","last_undegraded":"2024-03-06T07:30:04.415865+0000","last_fullsized":"2024-03-06T07:30:04.415865+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417256+0000","last_change":"2024-03-06T07:30:04.417256+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417256+0000","last_undegraded":"2024-03-06T07:30:04.417256+0000","last_fullsized":"2024-03-06T07:30:04.417256+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417074+0000","last_change":"2024-03-06T07:30:04.417074+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417074+0000","last_undegraded":"2024-03-06T07:30:04.417074+0000","last_fullsized":"2024-03-06T07:30:04.417074+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417249+0000","last_change":"2024-03-06T07:30:04.417249+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417249+0000","last_undegraded":"2024-03-06T07:30:04.417249+0000","last_fullsized":"2024-03-06T07:30:04.417249+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416617+0000","last_change":"2024-03-06T07:30:04.416617+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416617+0000","last_undegraded":"2024-03-06T07:30:04.416617+0000","last_fullsized":"2024-03-06T07:30:04.416617+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414450+0000","last_change":"2024-03-06T07:30:04.414450+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414450+0000","last_undegraded":"2024-03-06T07:30:04.414450+0000","last_fullsized":"2024-03-06T07:30:04.414450+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416361+0000","last_change":"2024-03-06T07:30:04.416361+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416361+0000","last_undegraded":"2024-03-06T07:30:04.416361+0000","last_fullsized":"2024-03-06T07:30:04.416361+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414123+0000","last_change":"2024-03-06T07:30:04.414123+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414123+0000","last_undegraded":"2024-03-06T07:30:04.414123+0000","last_fullsized":"2024-03-06T07:30:04.414123+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414132+0000","last_change":"2024-03-06T07:30:04.414132+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414132+0000","last_undegraded":"2024-03-06T07:30:04.414132+0000","last_fullsized":"2024-03-06T07:30:04.414132+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416640+0000","last_change":"2024-03-06T07:30:04.416640+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416640+0000","last_undegraded":"2024-03-06T07:30:04.416640+0000","last_fullsized":"2024-03-06T07:30:04.416640+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416321+0000","last_change":"2024-03-06T07:30:04.416321+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416321+0000","last_undegraded":"2024-03-06T07:30:04.416321+0000","last_fullsized":"2024-03-06T07:30:04.416321+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417160+0000","last_change":"2024-03-06T07:30:04.417160+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417160+0000","last_undegraded":"2024-03-06T07:30:04.417160+0000","last_fullsized":"2024-03-06T07:30:04.417160+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414459+0000","last_change":"2024-03-06T07:30:04.414459+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414459+0000","last_undegraded":"2024-03-06T07:30:04.414459+0000","last_fullsized":"2024-03-06T07:30:04.414459+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":391,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-03-06T07:30:04.417517+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:30:04.417517+0000","last_peered":"2024-03-06T07:30:04.417517+0000","last_clean":"2024-03-06T07:30:04.417517+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:30:04.417517+0000","last_undegraded":"2024-03-06T07:30:04.417517+0000","last_fullsized":"2024-03-06T07:30:04.417517+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574952,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574953,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":736103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:09.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:09.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.946+0000 7f4acd72c640 1 Processor -- start 2024-03-06T07:30:09.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4acd72c640 1 -- start start 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4acd72c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ac81424f0 0x7f4ac80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4acd72c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80cc980 con 0x7f4ac806da40 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4acd72c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80ccc60 con 0x7f4ac80cd7a0 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4acd72c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac80cd220 con 0x7f4ac81424f0 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4ac77fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4ac806da40 0x7f4ac8071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45238/0 (socket says 172.21.15.122:45238) 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.947+0000 7f4ac77fe640 1 -- 172.21.15.122:0/2368281043 learned_addr learned my addr 172.21.15.122:0/2368281043 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac6ffd640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ac81424f0 0x7f4ac80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425166464 0 0) 0x7f4ac80cc980 con 0x7f4ac806da40 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab4003520 con 0x7f4ac806da40 2024-03-06T07:30:09.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598403552 0 0) 0x7f4ac80ccc60 con 0x7f4ac80cd7a0 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac80cc980 con 0x7f4ac80cd7a0 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285846999 0 0) 0x7f4ab4003520 con 0x7f4ac806da40 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ac81424f0 msgr2=0x7f4ac80cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ac81424f0 0x7f4ac80cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 >> v1:172.21.15.169:6789/0 conn(0x7f4ac80cd7a0 legacy=0x7f4ac806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac80ccf40 con 0x7f4ac806da40 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4ac0001670 con 0x7f4ac806da40 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 >> v1:172.21.15.122:6789/0 conn(0x7f4ac806da40 legacy=0x7f4ac8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4003b70 0x7f4ab4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab4004800 0x7f4ab4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ab40051c0 0x7f4ab400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac80cd220 con 0x7f4ab4004800 2024-03-06T07:30:09.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab400bfe0 con 0x7f4ab40051c0 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab400c1e0 con 0x7f4ab4003b70 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac6ffd640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ac81424f0 0x7f4ac80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab4004800 0x7f4ab4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac77fe640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ab40051c0 0x7f4ab400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.948+0000 7f4ac6ffd640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4003b70 0x7f4ab4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.949+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4003b70 msgr2=0x7f4ab4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.949+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4003b70 0x7f4ab4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.949+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ab40051c0 msgr2=0x7f4ab400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.949+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ab40051c0 0x7f4ab400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.949+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2368281043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4003520 con 0x7f4ab4004800 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.950+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab4004800 0x7f4ab4004c80 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00018c0 tx=0x7f4ab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea4fecdca177fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.950+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0037070 con 0x7f4ab4004800 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.950+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab00052f0 con 0x7f4ab4004800 2024-03-06T07:30:09.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.950+0000 7f4ac5ffb640 1 -- 172.21.15.122:0/2368281043 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab00014e0 con 0x7f4ab4004800 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4acd72c640 1 -- 172.21.15.122:0/2368281043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab4004800 msgr2=0x7f4ab4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2368281043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab4004800 0x7f4ab4004c80 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00018c0 tx=0x7f4ab4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4ac77fe640 1 -- 172.21.15.122:0/2368281043 reap_dead start 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4acd72c640 1 -- 172.21.15.122:0/2368281043 shutdown_connections 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4acd72c640 1 -- 172.21.15.122:0/2368281043 >> 172.21.15.122:0/2368281043 conn(0x7f4ac80cf8a0 msgr2=0x7f4ac80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4acd72c640 1 -- 172.21.15.122:0/2368281043 shutdown_connections 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.951+0000 7f4acd72c640 1 -- 172.21.15.122:0/2368281043 wait complete. 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.952+0000 7f4acd72c640 1 Processor -- start 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.952+0000 7f4acd72c640 1 -- start start 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4acd72c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4004800 0x7f4ac813e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4acd72c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 0x7f4ac815ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4acd72c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ac815d7d0 0x7f4ac813f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4acd72c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac80cc470 con 0x7f4ac813b110 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4acd72c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac80cc2f0 con 0x7f4ac815d7d0 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4acd72c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac80cc5f0 con 0x7f4ab4004800 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 0x7f4ac815ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 0x7f4ac815ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60446/0 (socket says 172.21.15.122:60446) 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2074318524 learned_addr learned my addr 172.21.15.122:0/2074318524 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac6ffd640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4004800 0x7f4ac813e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac77fe640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ac815d7d0 0x7f4ac813f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4004800 msgr2=0x7f4ac813e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4004800 0x7f4ac813e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ac815d7d0 msgr2=0x7f4ac813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ac815d7d0 0x7f4ac813f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 -- 172.21.15.122:0/2074318524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac815dd40 con 0x7f4ac813b110 2024-03-06T07:30:09.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4ac67fc640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 0x7f4ac815ede0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0008fd0 tx=0x7f4ab00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.953+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0042020 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.954+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab0038040 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.954+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac815e020 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.954+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0037070 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.954+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac815e5e0 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.955+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac804f330 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.955+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ab0045020 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.955+0000 7f4abf7fe640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4aa806b500 0x7f4aa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.955+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ab00ab920 con 0x7f4ac813b110 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.956+0000 7f4ac6ffd640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4aa806b500 0x7f4aa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.956+0000 7f4ac6ffd640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4aa806b500 0x7f4aa806d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009fd0 tx=0x7f4ab800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:09.960+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ab004c070 con 0x7f4ac813b110 2024-03-06T07:30:10.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.132+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ac80d0620 con 0x7f4aa806b500 2024-03-06T07:30:10.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.138+0000 7f4abf7fe640 1 -- 172.21.15.122:0/2074318524 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240258 (secure 0 0 0) 0x7f4ac80d0620 con 0x7f4aa806b500 2024-03-06T07:30:10.142 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:10.144 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:30:10.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4aa806b500 msgr2=0x7f4aa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4aa806b500 0x7f4aa806d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009fd0 tx=0x7f4ab800b040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 msgr2=0x7f4ac815ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 0x7f4ac815ede0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0008fd0 tx=0x7f4ab00012d0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 shutdown_connections 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4aa806b500 0x7f4aa806d9c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ac815d7d0 0x7f4ac813f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ac813b110 0x7f4ac815ede0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 --2- 172.21.15.122:0/2074318524 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4ab4004800 0x7f4ac813e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 >> 172.21.15.122:0/2074318524 conn(0x7f4ac80cf8a0 msgr2=0x7f4ac806cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 shutdown_connections 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.141+0000 7f4acd72c640 1 -- 172.21.15.122:0/2074318524 wait complete. 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-06T07:30:09.426932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330480,"kb_used_data":14460,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644240,"statfs":{"total":773094113280,"available":772755701760,"internally_reserved":0,"allocated":14807040,"data_stored":7110279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.014410"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417086+0000","last_change":"2024-03-06T07:30:04.417086+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417086+0000","last_undegraded":"2024-03-06T07:30:04.417086+0000","last_fullsized":"2024-03-06T07:30:04.417086+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417956+0000","last_change":"2024-03-06T07:30:04.417956+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417956+0000","last_undegraded":"2024-03-06T07:30:04.417956+0000","last_fullsized":"2024-03-06T07:30:04.417956+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420390+0000","last_change":"2024-03-06T07:30:04.420390+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420390+0000","last_undegraded":"2024-03-06T07:30:04.420390+0000","last_fullsized":"2024-03-06T07:30:04.420390+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420351+0000","last_change":"2024-03-06T07:30:04.420351+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420351+0000","last_undegraded":"2024-03-06T07:30:04.420351+0000","last_fullsized":"2024-03-06T07:30:04.420351+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416826+0000","last_change":"2024-03-06T07:30:04.416826+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416826+0000","last_undegraded":"2024-03-06T07:30:04.416826+0000","last_fullsized":"2024-03-06T07:30:04.416826+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417090+0000","last_change":"2024-03-06T07:30:04.417090+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417090+0000","last_undegraded":"2024-03-06T07:30:04.417090+0000","last_fullsized":"2024-03-06T07:30:04.417090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416358+0000","last_change":"2024-03-06T07:30:04.416358+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416358+0000","last_undegraded":"2024-03-06T07:30:04.416358+0000","last_fullsized":"2024-03-06T07:30:04.416358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420391+0000","last_change":"2024-03-06T07:30:04.420391+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420391+0000","last_undegraded":"2024-03-06T07:30:04.420391+0000","last_fullsized":"2024-03-06T07:30:04.420391+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417129+0000","last_change":"2024-03-06T07:30:04.417129+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417129+0000","last_undegraded":"2024-03-06T07:30:04.417129+0000","last_fullsized":"2024-03-06T07:30:04.417129+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416322+0000","last_change":"2024-03-06T07:30:04.416322+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416322+0000","last_undegraded":"2024-03-06T07:30:04.416322+0000","last_fullsized":"2024-03-06T07:30:04.416322+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416196+0000","last_change":"2024-03-06T07:30:04.416196+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416196+0000","last_undegraded":"2024-03-06T07:30:04.416196+0000","last_fullsized":"2024-03-06T07:30:04.416196+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414439+0000","last_change":"2024-03-06T07:30:04.414439+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414439+0000","last_undegraded":"2024-03-06T07:30:04.414439+0000","last_fullsized":"2024-03-06T07:30:04.414439+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415854+0000","last_change":"2024-03-06T07:30:04.415854+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415854+0000","last_undegraded":"2024-03-06T07:30:04.415854+0000","last_fullsized":"2024-03-06T07:30:04.415854+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417449+0000","last_change":"2024-03-06T07:30:04.417449+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417449+0000","last_undegraded":"2024-03-06T07:30:04.417449+0000","last_fullsized":"2024-03-06T07:30:04.417449+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417586+0000","last_change":"2024-03-06T07:30:04.417586+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417586+0000","last_undegraded":"2024-03-06T07:30:04.417586+0000","last_fullsized":"2024-03-06T07:30:04.417586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415865+0000","last_change":"2024-03-06T07:30:04.415865+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415865+0000","last_undegraded":"2024-03-06T07:30:04.415865+0000","last_fullsized":"2024-03-06T07:30:04.415865+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417256+0000","last_change":"2024-03-06T07:30:04.417256+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417256+0000","last_undegraded":"2024-03-06T07:30:04.417256+0000","last_fullsized":"2024-03-06T07:30:04.417256+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417074+0000","last_change":"2024-03-06T07:30:04.417074+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417074+0000","last_undegraded":"2024-03-06T07:30:04.417074+0000","last_fullsized":"2024-03-06T07:30:04.417074+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417249+0000","last_change":"2024-03-06T07:30:04.417249+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417249+0000","last_undegraded":"2024-03-06T07:30:04.417249+0000","last_fullsized":"2024-03-06T07:30:04.417249+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416617+0000","last_change":"2024-03-06T07:30:04.416617+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416617+0000","last_undegraded":"2024-03-06T07:30:04.416617+0000","last_fullsized":"2024-03-06T07:30:04.416617+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414450+0000","last_change":"2024-03-06T07:30:04.414450+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414450+0000","last_undegraded":"2024-03-06T07:30:04.414450+0000","last_fullsized":"2024-03-06T07:30:04.414450+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416361+0000","last_change":"2024-03-06T07:30:04.416361+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416361+0000","last_undegraded":"2024-03-06T07:30:04.416361+0000","last_fullsized":"2024-03-06T07:30:04.416361+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414123+0000","last_change":"2024-03-06T07:30:04.414123+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414123+0000","last_undegraded":"2024-03-06T07:30:04.414123+0000","last_fullsized":"2024-03-06T07:30:04.414123+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414132+0000","last_change":"2024-03-06T07:30:04.414132+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414132+0000","last_undegraded":"2024-03-06T07:30:04.414132+0000","last_fullsized":"2024-03-06T07:30:04.414132+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416640+0000","last_change":"2024-03-06T07:30:04.416640+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416640+0000","last_undegraded":"2024-03-06T07:30:04.416640+0000","last_fullsized":"2024-03-06T07:30:04.416640+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416321+0000","last_change":"2024-03-06T07:30:04.416321+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416321+0000","last_undegraded":"2024-03-06T07:30:04.416321+0000","last_fullsized":"2024-03-06T07:30:04.416321+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417160+0000","last_change":"2024-03-06T07:30:04.417160+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417160+0000","last_undegraded":"2024-03-06T07:30:04.417160+0000","last_fullsized":"2024-03-06T07:30:04.417160+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414459+0000","last_change":"2024-03-06T07:30:04.414459+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414459+0000","last_undegraded":"2024-03-06T07:30:04.414459+0000","last_fullsized":"2024-03-06T07:30:04.414459+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":391,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-03-06T07:30:04.417517+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:30:04.417517+0000","last_peered":"2024-03-06T07:30:04.417517+0000","last_clean":"2024-03-06T07:30:04.417517+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:30:04.417517+0000","last_undegraded":"2024-03-06T07:30:04.417517+0000","last_fullsized":"2024-03-06T07:30:04.417517+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574952,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574953,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":736103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:10.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3f is not active+clean 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3e is not active+clean 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3d is not active+clean 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3c is not active+clean 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3b is not active+clean 2024-03-06T07:30:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3a is not active+clean 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.39 is not active+clean 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.38 is not active+clean 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.37 is not active+clean 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.36 is not active+clean 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.35 is not active+clean 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.34 is not active+clean 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.33 is not active+clean 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.32 is not active+clean 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.31 is not active+clean 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.30 is not active+clean 2024-03-06T07:30:10.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2f is not active+clean 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2e is not active+clean 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2d is not active+clean 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2c is not active+clean 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2b is not active+clean 2024-03-06T07:30:10.161 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2a is not active+clean 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.29 is not active+clean 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.28 is not active+clean 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.27 is not active+clean 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.26 is not active+clean 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.25 is not active+clean 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.24 is not active+clean 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.23 is not active+clean 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.22 is not active+clean 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.21 is not active+clean 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.20 is not active+clean 2024-03-06T07:30:10.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1e is not active+clean 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1f is not active+clean 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417086+0000', 'last_change': '2024-03-06T07:30:04.417086+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417086+0000', 'last_undegraded': '2024-03-06T07:30:04.417086+0000', 'last_fullsized': '2024-03-06T07:30:04.417086+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1f is not active+clean 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1e is not active+clean 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417956+0000', 'last_change': '2024-03-06T07:30:04.417956+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417956+0000', 'last_undegraded': '2024-03-06T07:30:04.417956+0000', 'last_fullsized': '2024-03-06T07:30:04.417956+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1c is not active+clean 2024-03-06T07:30:10.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1d is not active+clean 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.420390+0000', 'last_change': '2024-03-06T07:30:04.420390+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.420390+0000', 'last_undegraded': '2024-03-06T07:30:04.420390+0000', 'last_fullsized': '2024-03-06T07:30:04.420390+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1d is not active+clean 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1c is not active+clean 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.420351+0000', 'last_change': '2024-03-06T07:30:04.420351+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.420351+0000', 'last_undegraded': '2024-03-06T07:30:04.420351+0000', 'last_fullsized': '2024-03-06T07:30:04.420351+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1a is not active+clean 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1b is not active+clean 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416826+0000', 'last_change': '2024-03-06T07:30:04.416826+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416826+0000', 'last_undegraded': '2024-03-06T07:30:04.416826+0000', 'last_fullsized': '2024-03-06T07:30:04.416826+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:30:10.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1b is not active+clean 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1a is not active+clean 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417090+0000', 'last_change': '2024-03-06T07:30:04.417090+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417090+0000', 'last_undegraded': '2024-03-06T07:30:04.417090+0000', 'last_fullsized': '2024-03-06T07:30:04.417090+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.18 is not active+clean 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.19 is not active+clean 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416358+0000', 'last_change': '2024-03-06T07:30:04.416358+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416358+0000', 'last_undegraded': '2024-03-06T07:30:04.416358+0000', 'last_fullsized': '2024-03-06T07:30:04.416358+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.19 is not active+clean 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.18 is not active+clean 2024-03-06T07:30:10.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.420391+0000', 'last_change': '2024-03-06T07:30:04.420391+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.420391+0000', 'last_undegraded': '2024-03-06T07:30:04.420391+0000', 'last_fullsized': '2024-03-06T07:30:04.420391+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.16 is not active+clean 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.17 is not active+clean 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417129+0000', 'last_change': '2024-03-06T07:30:04.417129+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417129+0000', 'last_undegraded': '2024-03-06T07:30:04.417129+0000', 'last_fullsized': '2024-03-06T07:30:04.417129+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.17 is not active+clean 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.16 is not active+clean 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416322+0000', 'last_change': '2024-03-06T07:30:04.416322+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416322+0000', 'last_undegraded': '2024-03-06T07:30:04.416322+0000', 'last_fullsized': '2024-03-06T07:30:04.416322+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.15 is not active+clean 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.14 is not active+clean 2024-03-06T07:30:10.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416196+0000', 'last_change': '2024-03-06T07:30:04.416196+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416196+0000', 'last_undegraded': '2024-03-06T07:30:04.416196+0000', 'last_fullsized': '2024-03-06T07:30:04.416196+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.12 is not active+clean 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.13 is not active+clean 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414439+0000', 'last_change': '2024-03-06T07:30:04.414439+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414439+0000', 'last_undegraded': '2024-03-06T07:30:04.414439+0000', 'last_fullsized': '2024-03-06T07:30:04.414439+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.13 is not active+clean 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.12 is not active+clean 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414431+0000', 'last_change': '2024-03-06T07:30:04.414431+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414431+0000', 'last_undegraded': '2024-03-06T07:30:04.414431+0000', 'last_fullsized': '2024-03-06T07:30:04.414431+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.10 is not active+clean 2024-03-06T07:30:10.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.11 is not active+clean 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414124+0000', 'last_change': '2024-03-06T07:30:04.414124+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414124+0000', 'last_undegraded': '2024-03-06T07:30:04.414124+0000', 'last_fullsized': '2024-03-06T07:30:04.414124+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.11 is not active+clean 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.10 is not active+clean 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.415854+0000', 'last_change': '2024-03-06T07:30:04.415854+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.415854+0000', 'last_undegraded': '2024-03-06T07:30:04.415854+0000', 'last_fullsized': '2024-03-06T07:30:04.415854+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.e is not active+clean 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.f is not active+clean 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417449+0000', 'last_change': '2024-03-06T07:30:04.417449+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417449+0000', 'last_undegraded': '2024-03-06T07:30:04.417449+0000', 'last_fullsized': '2024-03-06T07:30:04.417449+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:10.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.f is not active+clean 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.e is not active+clean 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417586+0000', 'last_change': '2024-03-06T07:30:04.417586+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417586+0000', 'last_undegraded': '2024-03-06T07:30:04.417586+0000', 'last_fullsized': '2024-03-06T07:30:04.417586+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.c is not active+clean 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.d is not active+clean 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.415865+0000', 'last_change': '2024-03-06T07:30:04.415865+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.415865+0000', 'last_undegraded': '2024-03-06T07:30:04.415865+0000', 'last_fullsized': '2024-03-06T07:30:04.415865+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.d is not active+clean 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.c is not active+clean 2024-03-06T07:30:10.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417256+0000', 'last_change': '2024-03-06T07:30:04.417256+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417256+0000', 'last_undegraded': '2024-03-06T07:30:04.417256+0000', 'last_fullsized': '2024-03-06T07:30:04.417256+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.a is not active+clean 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.b is not active+clean 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417074+0000', 'last_change': '2024-03-06T07:30:04.417074+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417074+0000', 'last_undegraded': '2024-03-06T07:30:04.417074+0000', 'last_fullsized': '2024-03-06T07:30:04.417074+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.b is not active+clean 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.a is not active+clean 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417249+0000', 'last_change': '2024-03-06T07:30:04.417249+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417249+0000', 'last_undegraded': '2024-03-06T07:30:04.417249+0000', 'last_fullsized': '2024-03-06T07:30:04.417249+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.8 is not active+clean 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.9 is not active+clean 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416617+0000', 'last_change': '2024-03-06T07:30:04.416617+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416617+0000', 'last_undegraded': '2024-03-06T07:30:04.416617+0000', 'last_fullsized': '2024-03-06T07:30:04.416617+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:10.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.9 is not active+clean 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.8 is not active+clean 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414450+0000', 'last_change': '2024-03-06T07:30:04.414450+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414450+0000', 'last_undegraded': '2024-03-06T07:30:04.414450+0000', 'last_fullsized': '2024-03-06T07:30:04.414450+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.6 is not active+clean 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.7 is not active+clean 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416361+0000', 'last_change': '2024-03-06T07:30:04.416361+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416361+0000', 'last_undegraded': '2024-03-06T07:30:04.416361+0000', 'last_fullsized': '2024-03-06T07:30:04.416361+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.7 is not active+clean 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.6 is not active+clean 2024-03-06T07:30:10.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414124+0000', 'last_change': '2024-03-06T07:30:04.414124+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414124+0000', 'last_undegraded': '2024-03-06T07:30:04.414124+0000', 'last_fullsized': '2024-03-06T07:30:04.414124+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.4 is not active+clean 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.5 is not active+clean 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414123+0000', 'last_change': '2024-03-06T07:30:04.414123+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414123+0000', 'last_undegraded': '2024-03-06T07:30:04.414123+0000', 'last_fullsized': '2024-03-06T07:30:04.414123+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.5 is not active+clean 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.4 is not active+clean 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414132+0000', 'last_change': '2024-03-06T07:30:04.414132+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414132+0000', 'last_undegraded': '2024-03-06T07:30:04.414132+0000', 'last_fullsized': '2024-03-06T07:30:04.414132+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2 is not active+clean 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3 is not active+clean 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416640+0000', 'last_change': '2024-03-06T07:30:04.416640+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416640+0000', 'last_undegraded': '2024-03-06T07:30:04.416640+0000', 'last_fullsized': '2024-03-06T07:30:04.416640+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3 is not active+clean 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2 is not active+clean 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.416321+0000', 'last_change': '2024-03-06T07:30:04.416321+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.416321+0000', 'last_undegraded': '2024-03-06T07:30:04.416321+0000', 'last_fullsized': '2024-03-06T07:30:04.416321+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.0 is not active+clean 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1 is not active+clean 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414431+0000', 'last_change': '2024-03-06T07:30:04.414431+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414431+0000', 'last_undegraded': '2024-03-06T07:30:04.414431+0000', 'last_fullsized': '2024-03-06T07:30:04.414431+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1 is not active+clean 2024-03-06T07:30:10.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.0 is not active+clean 2024-03-06T07:30:10.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.417160+0000', 'last_change': '2024-03-06T07:30:04.417160+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.417160+0000', 'last_undegraded': '2024-03-06T07:30:04.417160+0000', 'last_fullsized': '2024-03-06T07:30:04.417160+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:10.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.14 is not active+clean 2024-03-06T07:30:10.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:30:05.412087+0000', 'last_change': '2024-03-06T07:30:05.412087+0000', 'last_active': '2024-03-06T07:30:05.412087+0000', 'last_peered': '2024-03-06T07:30:05.412087+0000', 'last_clean': '2024-03-06T07:30:05.412087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:05.412087+0000', 'last_undegraded': '2024-03-06T07:30:05.412087+0000', 'last_fullsized': '2024-03-06T07:30:05.412087+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:05.412087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:30:10.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.15 is not active+clean 2024-03-06T07:30:10.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 136, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:04.414459+0000', 'last_change': '2024-03-06T07:30:04.414459+0000', 'last_active': '2024-03-06T07:30:04.408336+0000', 'last_peered': '2024-03-06T07:30:04.408336+0000', 'last_clean': '2024-03-06T07:30:04.408336+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:04.414459+0000', 'last_undegraded': '2024-03-06T07:30:04.414459+0000', 'last_fullsized': '2024-03-06T07:30:04.414459+0000', 'mapping_epoch': 136, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 136, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:04.408336+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:10.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:30:10.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.292+0000 7fdd6ffe9640 1 Processor -- start 2024-03-06T07:30:10.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6ffe9640 1 -- start start 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6ffe9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdd6806d3c0 0x7fdd680b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6ffe9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdd680b1850 0x7fdd680b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6ffe9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd680af4b0 con 0x7fdd681614a0 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6ffe9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd680af790 con 0x7fdd6806d3c0 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6ffe9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd680afa70 con 0x7fdd680b1850 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6d584640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdd6806d3c0 0x7fdd680b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.293+0000 7fdd6dd85640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdd681614a0 0x7fdd680b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45248/0 (socket says 172.21.15.122:45248) 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6d584640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fdd6806d3c0 0x7fdd680b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60458/0 (socket says 172.21.15.122:60458) 2024-03-06T07:30:10.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6dd85640 1 -- 172.21.15.122:0/250251294 learned_addr learned my addr 172.21.15.122:0/250251294 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6e586640 1 --2- 172.21.15.122:0/250251294 >> v2:172.21.15.169:3300/0 conn(0x7fdd680b1850 0x7fdd680b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6d584640 1 -- 172.21.15.122:0/250251294 >> v2:172.21.15.169:3300/0 conn(0x7fdd680b1850 msgr2=0x7fdd680b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6d584640 1 --2- 172.21.15.122:0/250251294 >> v2:172.21.15.169:3300/0 conn(0x7fdd680b1850 0x7fdd680b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6d584640 1 -- 172.21.15.122:0/250251294 >> v1:172.21.15.122:6789/0 conn(0x7fdd681614a0 legacy=0x7fdd680b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6d584640 1 -- 172.21.15.122:0/250251294 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680afd50 con 0x7fdd6806d3c0 2024-03-06T07:30:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.294+0000 7fdd6cd83640 1 -- 172.21.15.122:0/250251294 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189868115 0 0) 0x7fdd680af4b0 con 0x7fdd681614a0 2024-03-06T07:30:10.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6d584640 1 --2- 172.21.15.122:0/250251294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd6806d3c0 0x7fdd680b1310 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fdd680b34f0 tx=0x7fdd6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbbaf1c58dd9975 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6cd83640 1 -- 172.21.15.122:0/250251294 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64035020 con 0x7fdd6806d3c0 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6cd83640 1 -- 172.21.15.122:0/250251294 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64031830 con 0x7fdd6806d3c0 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6cd83640 1 -- 172.21.15.122:0/250251294 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64031c00 con 0x7fdd6806d3c0 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/250251294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd6806d3c0 msgr2=0x7fdd680b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/250251294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd6806d3c0 0x7fdd680b1310 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fdd680b34f0 tx=0x7fdd6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/250251294 shutdown_connections 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/250251294 >> v2:172.21.15.169:3300/0 conn(0x7fdd680b1850 0x7fdd680b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/250251294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd6806d3c0 0x7fdd680b1310 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/250251294 >> 172.21.15.122:0/250251294 conn(0x7fdd6809a9f0 msgr2=0x7fdd6809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/250251294 shutdown_connections 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.295+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/250251294 wait complete. 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.296+0000 7fdd6ffe9640 1 Processor -- start 2024-03-06T07:30:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.296+0000 7fdd6ffe9640 1 -- start start 2024-03-06T07:30:10.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6ffe9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd6806d3c0 0x7fdd68158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6ffe9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd680b1850 0x7fdd6815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6ffe9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 0x7fdd680ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6ffe9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680b1db0 con 0x7fdd681614a0 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6ffe9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680b1c30 con 0x7fdd6806d3c0 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6ffe9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680b1f30 con 0x7fdd680b1850 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6e586640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 0x7fdd680ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6e586640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 0x7fdd680ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60462/0 (socket says 172.21.15.122:60462) 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6e586640 1 -- 172.21.15.122:0/116220700 learned_addr learned my addr 172.21.15.122:0/116220700 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6dd85640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd6806d3c0 0x7fdd68158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.297+0000 7fdd6d584640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd680b1850 0x7fdd6815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6e586640 1 -- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd680b1850 msgr2=0x7fdd6815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6e586640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd680b1850 0x7fdd6815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6e586640 1 -- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd6806d3c0 msgr2=0x7fdd68158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6e586640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd6806d3c0 0x7fdd68158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6e586640 1 -- 172.21.15.122:0/116220700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680cad10 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6d584640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd680b1850 0x7fdd6815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6e586640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 0x7fdd680ccba0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fdd60007e60 tx=0x7fdd6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60014420 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd60019070 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60014820 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6815d4a0 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.298+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6815da60 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.299+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd680caee0 con 0x7fdd681614a0 2024-03-06T07:30:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.299+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd600149c0 con 0x7fdd681614a0 2024-03-06T07:30:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.300+0000 7fdd5a7fc640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.300+0000 7fdd6dd85640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.300+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd6008a8f0 con 0x7fdd681614a0 2024-03-06T07:30:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.300+0000 7fdd6dd85640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c000f60 tx=0x7fdd5c007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.304+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd60059800 con 0x7fdd681614a0 2024-03-06T07:30:10.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.614+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fdd6804f330 con 0x7fdd681614a0 2024-03-06T07:30:10.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.615+0000 7fdd5a7fc640 1 -- 172.21.15.122:0/116220700 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1274 (secure 0 0 0) 0x7fdd60059100 con 0x7fdd681614a0 2024-03-06T07:30:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:10.617 INFO:teuthology.orchestra.run.smithi122.stdout:{"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":521,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":139,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709700,"num_in_osds":8,"osd_in_since":1709709691,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":338411520,"bytes_avail":772755701760,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":90,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"a","status":"up:active","gid":9321}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-03-06T07:29:49.423524+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd4c06b4d0 msgr2=0x7fdd4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c000f60 tx=0x7fdd5c007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 msgr2=0x7fdd680ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 0x7fdd680ccba0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fdd60007e60 tx=0x7fdd6000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 shutdown_connections 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd681614a0 0x7fdd680ccba0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdd680b1850 0x7fdd6815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 --2- 172.21.15.122:0/116220700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd6806d3c0 0x7fdd68158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 >> 172.21.15.122:0/116220700 conn(0x7fdd6809a9f0 msgr2=0x7fdd681600c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 shutdown_connections 2024-03-06T07:30:10.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.617+0000 7fdd6ffe9640 1 -- 172.21.15.122:0/116220700 wait complete. 2024-03-06T07:30:10.630 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:30:10.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:10.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.729+0000 7fd21e36b640 1 Processor -- start 2024-03-06T07:30:10.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.729+0000 7fd21e36b640 1 -- start start 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21e36b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd21815dfe0 0x7fd2180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21e36b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2180cf830 0x7fd2180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21e36b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2180d27f0 0x7fd2180d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21e36b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2180b31e0 con 0x7fd2180cf830 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21e36b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2180b3d60 con 0x7fd21815dfe0 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21e36b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b3430 con 0x7fd2180d27f0 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2180cf830 0x7fd2180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd2180cf830 0x7fd2180d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60476/0 (socket says 172.21.15.122:60476) 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 -- 172.21.15.122:0/4056592131 learned_addr learned my addr 172.21.15.122:0/4056592131 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:10.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd21c908640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2180d27f0 0x7fd2180d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd217fff640 1 --2- 172.21.15.122:0/4056592131 >> v2:172.21.15.169:3300/0 conn(0x7fd21815dfe0 0x7fd2180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 -- 172.21.15.122:0/4056592131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2180d27f0 msgr2=0x7fd2180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2180d27f0 0x7fd2180d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 -- 172.21.15.122:0/4056592131 >> v2:172.21.15.169:3300/0 conn(0x7fd21815dfe0 msgr2=0x7fd2180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 --2- 172.21.15.122:0/4056592131 >> v2:172.21.15.169:3300/0 conn(0x7fd21815dfe0 0x7fd2180cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.730+0000 7fd2177fe640 1 -- 172.21.15.122:0/4056592131 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180b3710 con 0x7fd2180cf830 2024-03-06T07:30:10.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.731+0000 7fd21c908640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2180d27f0 0x7fd2180d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:10.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd2177fe640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180d22b0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fd218067990 tx=0x7fd20802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cd5c104d7b84e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd216ffd640 1 -- 172.21.15.122:0/4056592131 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208035020 con 0x7fd2180cf830 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd216ffd640 1 -- 172.21.15.122:0/4056592131 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd208031870 con 0x7fd2180cf830 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd216ffd640 1 -- 172.21.15.122:0/4056592131 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208031c40 con 0x7fd2180cf830 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 -- 172.21.15.122:0/4056592131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 msgr2=0x7fd2180d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180d22b0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fd218067990 tx=0x7fd20802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 -- 172.21.15.122:0/4056592131 shutdown_connections 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180d22b0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 --2- 172.21.15.122:0/4056592131 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd2180d27f0 0x7fd2180d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 --2- 172.21.15.122:0/4056592131 >> v2:172.21.15.169:3300/0 conn(0x7fd21815dfe0 0x7fd2180cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 -- 172.21.15.122:0/4056592131 >> 172.21.15.122:0/4056592131 conn(0x7fd2180d3cd0 msgr2=0x7fd2180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 -- 172.21.15.122:0/4056592131 shutdown_connections 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.732+0000 7fd21e36b640 1 -- 172.21.15.122:0/4056592131 wait complete. 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.733+0000 7fd21e36b640 1 Processor -- start 2024-03-06T07:30:10.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 -- start start 2024-03-06T07:30:10.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2180d27f0 0x7fd2180cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd21815dfe0 0x7fd2181f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180d1df0 con 0x7fd2180cf830 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180d1c70 con 0x7fd2180d27f0 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd21e36b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21815e3c0 con 0x7fd21815dfe0 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.734+0000 7fd217fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60486/0 (socket says 172.21.15.122:60486) 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 -- 172.21.15.122:0/2390683735 learned_addr learned my addr 172.21.15.122:0/2390683735 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd21c908640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd21815dfe0 0x7fd2181f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd2177fe640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2180d27f0 0x7fd2180cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 -- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd21815dfe0 msgr2=0x7fd2181f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd21815dfe0 0x7fd2181f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 -- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2180d27f0 msgr2=0x7fd2180cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2180d27f0 0x7fd2180cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 -- 172.21.15.122:0/2390683735 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180cbb80 con 0x7fd2180cf830 2024-03-06T07:30:10.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd217fff640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180cae10 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00c7f0 tx=0x7fd20c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c015070 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c004220 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c007620 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2180cbe60 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.735+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2180cc420 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.736+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21804f330 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.737+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd20c0079d0 con 0x7fd2180cf830 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.737+0000 7fd214ff9640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd20406b4d0 0x7fd20406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.738+0000 7fd2177fe640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd20406b4d0 0x7fd20406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.738+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd20c08a3f0 con 0x7fd2180cf830 2024-03-06T07:30:10.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.738+0000 7fd2177fe640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd20406b4d0 0x7fd20406d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd208036000 tx=0x7fd20803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.742+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd20c058cb0 con 0x7fd2180cf830 2024-03-06T07:30:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.915+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2180ce960 con 0x7fd20406b4d0 2024-03-06T07:30:10.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.920+0000 7fd214ff9640 1 -- 172.21.15.122:0/2390683735 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240258 (secure 0 0 0) 0x7fd208082aa0 con 0x7fd20406b4d0 2024-03-06T07:30:10.925 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:10.926 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:30:10.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd20406b4d0 msgr2=0x7fd20406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd20406b4d0 0x7fd20406d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd208036000 tx=0x7fd20803c040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 msgr2=0x7fd2180cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180cae10 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00c7f0 tx=0x7fd20c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 shutdown_connections 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd20406b4d0 0x7fd20406d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd21815dfe0 0x7fd2181f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd2180d27f0 0x7fd2180cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 --2- 172.21.15.122:0/2390683735 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2180cf830 0x7fd2180cae10 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.923+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 >> 172.21.15.122:0/2390683735 conn(0x7fd2180d3cd0 msgr2=0x7fd21806bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.924+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 shutdown_connections 2024-03-06T07:30:10.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:10.924+0000 7fd21e36b640 1 -- 172.21.15.122:0/2390683735 wait complete. 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-06T07:30:09.426932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330480,"kb_used_data":14460,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644240,"statfs":{"total":773094113280,"available":772755701760,"internally_reserved":0,"allocated":14807040,"data_stored":7110279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.014410"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417086+0000","last_change":"2024-03-06T07:30:04.417086+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417086+0000","last_undegraded":"2024-03-06T07:30:04.417086+0000","last_fullsized":"2024-03-06T07:30:04.417086+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417956+0000","last_change":"2024-03-06T07:30:04.417956+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417956+0000","last_undegraded":"2024-03-06T07:30:04.417956+0000","last_fullsized":"2024-03-06T07:30:04.417956+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420390+0000","last_change":"2024-03-06T07:30:04.420390+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420390+0000","last_undegraded":"2024-03-06T07:30:04.420390+0000","last_fullsized":"2024-03-06T07:30:04.420390+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420351+0000","last_change":"2024-03-06T07:30:04.420351+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420351+0000","last_undegraded":"2024-03-06T07:30:04.420351+0000","last_fullsized":"2024-03-06T07:30:04.420351+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416826+0000","last_change":"2024-03-06T07:30:04.416826+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416826+0000","last_undegraded":"2024-03-06T07:30:04.416826+0000","last_fullsized":"2024-03-06T07:30:04.416826+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417090+0000","last_change":"2024-03-06T07:30:04.417090+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417090+0000","last_undegraded":"2024-03-06T07:30:04.417090+0000","last_fullsized":"2024-03-06T07:30:04.417090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416358+0000","last_change":"2024-03-06T07:30:04.416358+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416358+0000","last_undegraded":"2024-03-06T07:30:04.416358+0000","last_fullsized":"2024-03-06T07:30:04.416358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.420391+0000","last_change":"2024-03-06T07:30:04.420391+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.420391+0000","last_undegraded":"2024-03-06T07:30:04.420391+0000","last_fullsized":"2024-03-06T07:30:04.420391+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417129+0000","last_change":"2024-03-06T07:30:04.417129+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417129+0000","last_undegraded":"2024-03-06T07:30:04.417129+0000","last_fullsized":"2024-03-06T07:30:04.417129+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416322+0000","last_change":"2024-03-06T07:30:04.416322+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416322+0000","last_undegraded":"2024-03-06T07:30:04.416322+0000","last_fullsized":"2024-03-06T07:30:04.416322+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416196+0000","last_change":"2024-03-06T07:30:04.416196+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416196+0000","last_undegraded":"2024-03-06T07:30:04.416196+0000","last_fullsized":"2024-03-06T07:30:04.416196+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414439+0000","last_change":"2024-03-06T07:30:04.414439+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414439+0000","last_undegraded":"2024-03-06T07:30:04.414439+0000","last_fullsized":"2024-03-06T07:30:04.414439+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415854+0000","last_change":"2024-03-06T07:30:04.415854+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415854+0000","last_undegraded":"2024-03-06T07:30:04.415854+0000","last_fullsized":"2024-03-06T07:30:04.415854+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417449+0000","last_change":"2024-03-06T07:30:04.417449+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417449+0000","last_undegraded":"2024-03-06T07:30:04.417449+0000","last_fullsized":"2024-03-06T07:30:04.417449+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417586+0000","last_change":"2024-03-06T07:30:04.417586+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417586+0000","last_undegraded":"2024-03-06T07:30:04.417586+0000","last_fullsized":"2024-03-06T07:30:04.417586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.415865+0000","last_change":"2024-03-06T07:30:04.415865+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.415865+0000","last_undegraded":"2024-03-06T07:30:04.415865+0000","last_fullsized":"2024-03-06T07:30:04.415865+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417256+0000","last_change":"2024-03-06T07:30:04.417256+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417256+0000","last_undegraded":"2024-03-06T07:30:04.417256+0000","last_fullsized":"2024-03-06T07:30:04.417256+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417074+0000","last_change":"2024-03-06T07:30:04.417074+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417074+0000","last_undegraded":"2024-03-06T07:30:04.417074+0000","last_fullsized":"2024-03-06T07:30:04.417074+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417249+0000","last_change":"2024-03-06T07:30:04.417249+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417249+0000","last_undegraded":"2024-03-06T07:30:04.417249+0000","last_fullsized":"2024-03-06T07:30:04.417249+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416617+0000","last_change":"2024-03-06T07:30:04.416617+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416617+0000","last_undegraded":"2024-03-06T07:30:04.416617+0000","last_fullsized":"2024-03-06T07:30:04.416617+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414450+0000","last_change":"2024-03-06T07:30:04.414450+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414450+0000","last_undegraded":"2024-03-06T07:30:04.414450+0000","last_fullsized":"2024-03-06T07:30:04.414450+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416361+0000","last_change":"2024-03-06T07:30:04.416361+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416361+0000","last_undegraded":"2024-03-06T07:30:04.416361+0000","last_fullsized":"2024-03-06T07:30:04.416361+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414124+0000","last_change":"2024-03-06T07:30:04.414124+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414124+0000","last_undegraded":"2024-03-06T07:30:04.414124+0000","last_fullsized":"2024-03-06T07:30:04.414124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414123+0000","last_change":"2024-03-06T07:30:04.414123+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414123+0000","last_undegraded":"2024-03-06T07:30:04.414123+0000","last_fullsized":"2024-03-06T07:30:04.414123+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414132+0000","last_change":"2024-03-06T07:30:04.414132+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414132+0000","last_undegraded":"2024-03-06T07:30:04.414132+0000","last_fullsized":"2024-03-06T07:30:04.414132+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416640+0000","last_change":"2024-03-06T07:30:04.416640+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416640+0000","last_undegraded":"2024-03-06T07:30:04.416640+0000","last_fullsized":"2024-03-06T07:30:04.416640+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.416321+0000","last_change":"2024-03-06T07:30:04.416321+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.416321+0000","last_undegraded":"2024-03-06T07:30:04.416321+0000","last_fullsized":"2024-03-06T07:30:04.416321+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414431+0000","last_change":"2024-03-06T07:30:04.414431+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414431+0000","last_undegraded":"2024-03-06T07:30:04.414431+0000","last_fullsized":"2024-03-06T07:30:04.414431+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.417160+0000","last_change":"2024-03-06T07:30:04.417160+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.417160+0000","last_undegraded":"2024-03-06T07:30:04.417160+0000","last_fullsized":"2024-03-06T07:30:04.417160+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:05.412087+0000","last_change":"2024-03-06T07:30:05.412087+0000","last_active":"2024-03-06T07:30:05.412087+0000","last_peered":"2024-03-06T07:30:05.412087+0000","last_clean":"2024-03-06T07:30:05.412087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:05.412087+0000","last_undegraded":"2024-03-06T07:30:05.412087+0000","last_fullsized":"2024-03-06T07:30:05.412087+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-03-06T07:30:04.414459+0000","last_change":"2024-03-06T07:30:04.414459+0000","last_active":"2024-03-06T07:30:04.408336+0000","last_peered":"2024-03-06T07:30:04.408336+0000","last_clean":"2024-03-06T07:30:04.408336+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:04.414459+0000","last_undegraded":"2024-03-06T07:30:04.414459+0000","last_fullsized":"2024-03-06T07:30:04.414459+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":391,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-03-06T07:30:04.417517+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:30:04.417517+0000","last_peered":"2024-03-06T07:30:04.417517+0000","last_clean":"2024-03-06T07:30:04.417517+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:30:04.417517+0000","last_undegraded":"2024-03-06T07:30:04.417517+0000","last_fullsized":"2024-03-06T07:30:04.417517+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574952,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41164,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330676,"statfs":{"total":96636764160,"available":96594612224,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574953,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":736103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41168,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330672,"statfs":{"total":96636764160,"available":96594608128,"internally_reserved":0,"allocated":1703936,"data_stored":741920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:12.419 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:30:12.417+0000 7fd232a43640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:13.941 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.041+0000 7f694879a640 1 Processor -- start 2024-03-06T07:30:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.041+0000 7f694879a640 1 -- start start 2024-03-06T07:30:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f694879a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f694015dfe0 0x7f69400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f694879a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69400cf830 0x7f69400d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f694879a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69400b3430 con 0x7f69400d27f0 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f694879a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400d2c50 con 0x7f69400cf830 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f694879a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69400b31e0 con 0x7f694015dfe0 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f6945d35640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69400cf830 0x7f69400d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f6945d35640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69400cf830 0x7f69400d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60496/0 (socket says 172.21.15.122:60496) 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f6945d35640 1 -- 172.21.15.122:0/3112409080 learned_addr learned my addr 172.21.15.122:0/3112409080 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.042+0000 7f6946536640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f694015dfe0 0x7f69400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945d35640 1 -- 172.21.15.122:0/3112409080 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f694015dfe0 msgr2=0x7f69400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945d35640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f694015dfe0 0x7f69400cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945d35640 1 -- 172.21.15.122:0/3112409080 >> v1:172.21.15.169:6789/0 conn(0x7f69400d27f0 legacy=0x7f69400d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945d35640 1 -- 172.21.15.122:0/3112409080 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400b3710 con 0x7f69400cf830 2024-03-06T07:30:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6946536640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f694015dfe0 0x7f69400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945d35640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69400cf830 0x7f69400d22b0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f6930009870 tx=0x7f693002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30cb063bff0edc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945534640 1 -- 172.21.15.122:0/3112409080 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930034070 con 0x7f69400cf830 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945534640 1 -- 172.21.15.122:0/3112409080 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6930030840 con 0x7f69400cf830 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.043+0000 7f6945534640 1 -- 172.21.15.122:0/3112409080 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930030c10 con 0x7f69400cf830 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 -- 172.21.15.122:0/3112409080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69400cf830 msgr2=0x7f69400d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69400cf830 0x7f69400d22b0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f6930009870 tx=0x7f693002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 -- 172.21.15.122:0/3112409080 shutdown_connections 2024-03-06T07:30:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69400cf830 0x7f69400d22b0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 --2- 172.21.15.122:0/3112409080 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f694015dfe0 0x7f69400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 -- 172.21.15.122:0/3112409080 >> 172.21.15.122:0/3112409080 conn(0x7f69400d3cd0 msgr2=0x7f69400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 -- 172.21.15.122:0/3112409080 shutdown_connections 2024-03-06T07:30:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.044+0000 7f694879a640 1 -- 172.21.15.122:0/3112409080 wait complete. 2024-03-06T07:30:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.045+0000 7f694879a640 1 Processor -- start 2024-03-06T07:30:14.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.045+0000 7f694879a640 1 -- start start 2024-03-06T07:30:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f694879a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69400cf830 0x7f694015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f694879a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69400d27f0 0x7f69400cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f694879a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 0x7f6940160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f694879a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69400d1c70 con 0x7f694015dfe0 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f694879a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69400b3d60 con 0x7f69400cf830 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f694879a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69400d1df0 con 0x7f69400d27f0 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 0x7f6940160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 0x7f6940160390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60502/0 (socket says 172.21.15.122:60502) 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 -- 172.21.15.122:0/2496481985 learned_addr learned my addr 172.21.15.122:0/2496481985 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6945d35640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69400d27f0 0x7f69400cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946536640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69400cf830 0x7f694015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 -- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69400d27f0 msgr2=0x7f69400cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69400d27f0 0x7f69400cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 -- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69400cf830 msgr2=0x7f694015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69400cf830 0x7f694015a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 -- 172.21.15.122:0/2496481985 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400b6c50 con 0x7f694015dfe0 2024-03-06T07:30:14.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946536640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69400cf830 0x7f694015a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6945d35640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69400d27f0 0x7f69400cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.046+0000 7f6946d37640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 0x7f6940160390 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c7d0 tx=0x7f693c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.047+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c003b70 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.047+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c003d30 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.047+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69400b6f30 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.047+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00aa90 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.047+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694015f730 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.047+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694004f330 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.048+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f693c00ac30 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.048+0000 7f6936ffd640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f692406b500 0x7f692406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.048+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f693c08a170 con 0x7f694015dfe0 2024-03-06T07:30:14.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.048+0000 7f6946536640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f692406b500 0x7f692406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.049+0000 7f6946536640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f692406b500 0x7f692406d9c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6938006fd0 tx=0x7f6938008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.053+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f693c058ff0 con 0x7f694015dfe0 2024-03-06T07:30:14.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.323+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f694015a830 con 0x7f692406b500 2024-03-06T07:30:14.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.329+0000 7f6936ffd640 1 -- 172.21.15.122:0/2496481985 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252488 (secure 0 0 0) 0x7f6938003e90 con 0x7f692406b500 2024-03-06T07:30:14.334 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:14.335 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f692406b500 msgr2=0x7f692406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f692406b500 0x7f692406d9c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6938006fd0 tx=0x7f6938008040 comp rx=0 tx=0).stop 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 msgr2=0x7f6940160390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 0x7f6940160390 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c7d0 tx=0x7f693c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 shutdown_connections 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f692406b500 0x7f692406d9c0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f694015dfe0 0x7f6940160390 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f69400d27f0 0x7f69400cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 --2- 172.21.15.122:0/2496481985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69400cf830 0x7f694015a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 >> 172.21.15.122:0/2496481985 conn(0x7f69400d3cd0 msgr2=0x7f69400cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 shutdown_connections 2024-03-06T07:30:14.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.332+0000 7f694879a640 1 -- 172.21.15.122:0/2496481985 wait complete. 2024-03-06T07:30:14.345 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-03-06T07:30:13.427628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330832,"kb_used_data":14824,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643888,"statfs":{"total":773094113280,"available":772755341312,"internally_reserved":0,"allocated":15179776,"data_stored":7275716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001347"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140882+0000","last_change":"2024-03-06T07:30:06.434937+0000","last_active":"2024-03-06T07:30:08.140882+0000","last_peered":"2024-03-06T07:30:08.140882+0000","last_clean":"2024-03-06T07:30:08.140882+0000","last_became_active":"2024-03-06T07:30:06.434648+0000","last_became_peered":"2024-03-06T07:30:06.434648+0000","last_unstale":"2024-03-06T07:30:08.140882+0000","last_undegraded":"2024-03-06T07:30:08.140882+0000","last_fullsized":"2024-03-06T07:30:08.140882+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:25.275095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429670+0000","last_change":"2024-03-06T07:30:06.429622+0000","last_active":"2024-03-06T07:30:07.429670+0000","last_peered":"2024-03-06T07:30:07.429670+0000","last_clean":"2024-03-06T07:30:07.429670+0000","last_became_active":"2024-03-06T07:30:06.427836+0000","last_became_peered":"2024-03-06T07:30:06.427836+0000","last_unstale":"2024-03-06T07:30:07.429670+0000","last_undegraded":"2024-03-06T07:30:07.429670+0000","last_fullsized":"2024-03-06T07:30:07.429670+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:36.226996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431440+0000","last_change":"2024-03-06T07:30:06.434647+0000","last_active":"2024-03-06T07:30:07.431440+0000","last_peered":"2024-03-06T07:30:07.431440+0000","last_clean":"2024-03-06T07:30:07.431440+0000","last_became_active":"2024-03-06T07:30:06.434229+0000","last_became_peered":"2024-03-06T07:30:06.434229+0000","last_unstale":"2024-03-06T07:30:07.431440+0000","last_undegraded":"2024-03-06T07:30:07.431440+0000","last_fullsized":"2024-03-06T07:30:07.431440+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:08.641996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431797+0000","last_change":"2024-03-06T07:30:06.436904+0000","last_active":"2024-03-06T07:30:07.431797+0000","last_peered":"2024-03-06T07:30:07.431797+0000","last_clean":"2024-03-06T07:30:07.431797+0000","last_became_active":"2024-03-06T07:30:06.436522+0000","last_became_peered":"2024-03-06T07:30:06.436522+0000","last_unstale":"2024-03-06T07:30:07.431797+0000","last_undegraded":"2024-03-06T07:30:07.431797+0000","last_fullsized":"2024-03-06T07:30:07.431797+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:23.597805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140225+0000","last_change":"2024-03-06T07:30:06.448335+0000","last_active":"2024-03-06T07:30:08.140225+0000","last_peered":"2024-03-06T07:30:08.140225+0000","last_clean":"2024-03-06T07:30:08.140225+0000","last_became_active":"2024-03-06T07:30:06.448078+0000","last_became_peered":"2024-03-06T07:30:06.448078+0000","last_unstale":"2024-03-06T07:30:08.140225+0000","last_undegraded":"2024-03-06T07:30:08.140225+0000","last_fullsized":"2024-03-06T07:30:08.140225+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:24.502621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140650+0000","last_change":"2024-03-06T07:30:06.436284+0000","last_active":"2024-03-06T07:30:08.140650+0000","last_peered":"2024-03-06T07:30:08.140650+0000","last_clean":"2024-03-06T07:30:08.140650+0000","last_became_active":"2024-03-06T07:30:06.435975+0000","last_became_peered":"2024-03-06T07:30:06.435975+0000","last_unstale":"2024-03-06T07:30:08.140650+0000","last_undegraded":"2024-03-06T07:30:08.140650+0000","last_fullsized":"2024-03-06T07:30:08.140650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:03.753224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141017+0000","last_change":"2024-03-06T07:30:06.442329+0000","last_active":"2024-03-06T07:30:08.141017+0000","last_peered":"2024-03-06T07:30:08.141017+0000","last_clean":"2024-03-06T07:30:08.141017+0000","last_became_active":"2024-03-06T07:30:06.441847+0000","last_became_peered":"2024-03-06T07:30:06.441847+0000","last_unstale":"2024-03-06T07:30:08.141017+0000","last_undegraded":"2024-03-06T07:30:08.141017+0000","last_fullsized":"2024-03-06T07:30:08.141017+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:40.567228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140010+0000","last_change":"2024-03-06T07:30:06.443554+0000","last_active":"2024-03-06T07:30:08.140010+0000","last_peered":"2024-03-06T07:30:08.140010+0000","last_clean":"2024-03-06T07:30:08.140010+0000","last_became_active":"2024-03-06T07:30:06.443255+0000","last_became_peered":"2024-03-06T07:30:06.443255+0000","last_unstale":"2024-03-06T07:30:08.140010+0000","last_undegraded":"2024-03-06T07:30:08.140010+0000","last_fullsized":"2024-03-06T07:30:08.140010+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:44.806317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429689+0000","last_change":"2024-03-06T07:30:06.432046+0000","last_active":"2024-03-06T07:30:07.429689+0000","last_peered":"2024-03-06T07:30:07.429689+0000","last_clean":"2024-03-06T07:30:07.429689+0000","last_became_active":"2024-03-06T07:30:06.431749+0000","last_became_peered":"2024-03-06T07:30:06.431749+0000","last_unstale":"2024-03-06T07:30:07.429689+0000","last_undegraded":"2024-03-06T07:30:07.429689+0000","last_fullsized":"2024-03-06T07:30:07.429689+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:52.862299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140921+0000","last_change":"2024-03-06T07:30:06.445698+0000","last_active":"2024-03-06T07:30:08.140921+0000","last_peered":"2024-03-06T07:30:08.140921+0000","last_clean":"2024-03-06T07:30:08.140921+0000","last_became_active":"2024-03-06T07:30:06.445390+0000","last_became_peered":"2024-03-06T07:30:06.445390+0000","last_unstale":"2024-03-06T07:30:08.140921+0000","last_undegraded":"2024-03-06T07:30:08.140921+0000","last_fullsized":"2024-03-06T07:30:08.140921+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:22.332084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140319+0000","last_change":"2024-03-06T07:30:06.443592+0000","last_active":"2024-03-06T07:30:08.140319+0000","last_peered":"2024-03-06T07:30:08.140319+0000","last_clean":"2024-03-06T07:30:08.140319+0000","last_became_active":"2024-03-06T07:30:06.443316+0000","last_became_peered":"2024-03-06T07:30:06.443316+0000","last_unstale":"2024-03-06T07:30:08.140319+0000","last_undegraded":"2024-03-06T07:30:08.140319+0000","last_fullsized":"2024-03-06T07:30:08.140319+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:22.061312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141071+0000","last_change":"2024-03-06T07:30:06.442047+0000","last_active":"2024-03-06T07:30:08.141071+0000","last_peered":"2024-03-06T07:30:08.141071+0000","last_clean":"2024-03-06T07:30:08.141071+0000","last_became_active":"2024-03-06T07:30:06.441753+0000","last_became_peered":"2024-03-06T07:30:06.441753+0000","last_unstale":"2024-03-06T07:30:08.141071+0000","last_undegraded":"2024-03-06T07:30:08.141071+0000","last_fullsized":"2024-03-06T07:30:08.141071+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:47.078323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140415+0000","last_change":"2024-03-06T07:30:06.447476+0000","last_active":"2024-03-06T07:30:08.140415+0000","last_peered":"2024-03-06T07:30:08.140415+0000","last_clean":"2024-03-06T07:30:08.140415+0000","last_became_active":"2024-03-06T07:30:06.447223+0000","last_became_peered":"2024-03-06T07:30:06.447223+0000","last_unstale":"2024-03-06T07:30:08.140415+0000","last_undegraded":"2024-03-06T07:30:08.140415+0000","last_fullsized":"2024-03-06T07:30:08.140415+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:42.221017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430611+0000","last_change":"2024-03-06T07:30:06.443748+0000","last_active":"2024-03-06T07:30:07.430611+0000","last_peered":"2024-03-06T07:30:07.430611+0000","last_clean":"2024-03-06T07:30:07.430611+0000","last_became_active":"2024-03-06T07:30:06.443317+0000","last_became_peered":"2024-03-06T07:30:06.443317+0000","last_unstale":"2024-03-06T07:30:07.430611+0000","last_undegraded":"2024-03-06T07:30:07.430611+0000","last_fullsized":"2024-03-06T07:30:07.430611+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:51.322488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140917+0000","last_change":"2024-03-06T07:30:06.435198+0000","last_active":"2024-03-06T07:30:08.140917+0000","last_peered":"2024-03-06T07:30:08.140917+0000","last_clean":"2024-03-06T07:30:08.140917+0000","last_became_active":"2024-03-06T07:30:06.434945+0000","last_became_peered":"2024-03-06T07:30:06.434945+0000","last_unstale":"2024-03-06T07:30:08.140917+0000","last_undegraded":"2024-03-06T07:30:08.140917+0000","last_fullsized":"2024-03-06T07:30:08.140917+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:01.737335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141042+0000","last_change":"2024-03-06T07:30:06.430521+0000","last_active":"2024-03-06T07:30:08.141042+0000","last_peered":"2024-03-06T07:30:08.141042+0000","last_clean":"2024-03-06T07:30:08.141042+0000","last_became_active":"2024-03-06T07:30:06.430267+0000","last_became_peered":"2024-03-06T07:30:06.430267+0000","last_unstale":"2024-03-06T07:30:08.141042+0000","last_undegraded":"2024-03-06T07:30:08.141042+0000","last_fullsized":"2024-03-06T07:30:08.141042+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:16.972342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141362+0000","last_change":"2024-03-06T07:30:06.442891+0000","last_active":"2024-03-06T07:30:08.141362+0000","last_peered":"2024-03-06T07:30:08.141362+0000","last_clean":"2024-03-06T07:30:08.141362+0000","last_became_active":"2024-03-06T07:30:06.442512+0000","last_became_peered":"2024-03-06T07:30:06.442512+0000","last_unstale":"2024-03-06T07:30:08.141362+0000","last_undegraded":"2024-03-06T07:30:08.141362+0000","last_fullsized":"2024-03-06T07:30:08.141362+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:09.228199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430339+0000","last_change":"2024-03-06T07:30:06.433546+0000","last_active":"2024-03-06T07:30:07.430339+0000","last_peered":"2024-03-06T07:30:07.430339+0000","last_clean":"2024-03-06T07:30:07.430339+0000","last_became_active":"2024-03-06T07:30:06.433174+0000","last_became_peered":"2024-03-06T07:30:06.433174+0000","last_unstale":"2024-03-06T07:30:07.430339+0000","last_undegraded":"2024-03-06T07:30:07.430339+0000","last_fullsized":"2024-03-06T07:30:07.430339+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:08.706684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140718+0000","last_change":"2024-03-06T07:30:06.438360+0000","last_active":"2024-03-06T07:30:08.140718+0000","last_peered":"2024-03-06T07:30:08.140718+0000","last_clean":"2024-03-06T07:30:08.140718+0000","last_became_active":"2024-03-06T07:30:06.437982+0000","last_became_peered":"2024-03-06T07:30:06.437982+0000","last_unstale":"2024-03-06T07:30:08.140718+0000","last_undegraded":"2024-03-06T07:30:08.140718+0000","last_fullsized":"2024-03-06T07:30:08.140718+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.851347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140317+0000","last_change":"2024-03-06T07:30:06.428705+0000","last_active":"2024-03-06T07:30:08.140317+0000","last_peered":"2024-03-06T07:30:08.140317+0000","last_clean":"2024-03-06T07:30:08.140317+0000","last_became_active":"2024-03-06T07:30:06.428433+0000","last_became_peered":"2024-03-06T07:30:06.428433+0000","last_unstale":"2024-03-06T07:30:08.140317+0000","last_undegraded":"2024-03-06T07:30:08.140317+0000","last_fullsized":"2024-03-06T07:30:08.140317+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:21.326137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140886+0000","last_change":"2024-03-06T07:30:06.441461+0000","last_active":"2024-03-06T07:30:08.140886+0000","last_peered":"2024-03-06T07:30:08.140886+0000","last_clean":"2024-03-06T07:30:08.140886+0000","last_became_active":"2024-03-06T07:30:06.441156+0000","last_became_peered":"2024-03-06T07:30:06.441156+0000","last_unstale":"2024-03-06T07:30:08.140886+0000","last_undegraded":"2024-03-06T07:30:08.140886+0000","last_fullsized":"2024-03-06T07:30:08.140886+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:31.906079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430635+0000","last_change":"2024-03-06T07:30:06.443559+0000","last_active":"2024-03-06T07:30:07.430635+0000","last_peered":"2024-03-06T07:30:07.430635+0000","last_clean":"2024-03-06T07:30:07.430635+0000","last_became_active":"2024-03-06T07:30:06.442908+0000","last_became_peered":"2024-03-06T07:30:06.442908+0000","last_unstale":"2024-03-06T07:30:07.430635+0000","last_undegraded":"2024-03-06T07:30:07.430635+0000","last_fullsized":"2024-03-06T07:30:07.430635+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:39.860873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431174+0000","last_change":"2024-03-06T07:30:06.435273+0000","last_active":"2024-03-06T07:30:07.431174+0000","last_peered":"2024-03-06T07:30:07.431174+0000","last_clean":"2024-03-06T07:30:07.431174+0000","last_became_active":"2024-03-06T07:30:06.434942+0000","last_became_peered":"2024-03-06T07:30:06.434942+0000","last_unstale":"2024-03-06T07:30:07.431174+0000","last_undegraded":"2024-03-06T07:30:07.431174+0000","last_fullsized":"2024-03-06T07:30:07.431174+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:25.275095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140244+0000","last_change":"2024-03-06T07:30:06.441614+0000","last_active":"2024-03-06T07:30:08.140244+0000","last_peered":"2024-03-06T07:30:08.140244+0000","last_clean":"2024-03-06T07:30:08.140244+0000","last_became_active":"2024-03-06T07:30:06.440989+0000","last_became_peered":"2024-03-06T07:30:06.440989+0000","last_unstale":"2024-03-06T07:30:08.140244+0000","last_undegraded":"2024-03-06T07:30:08.140244+0000","last_fullsized":"2024-03-06T07:30:08.140244+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:40.567228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140492+0000","last_change":"2024-03-06T07:30:06.432705+0000","last_active":"2024-03-06T07:30:08.140492+0000","last_peered":"2024-03-06T07:30:08.140492+0000","last_clean":"2024-03-06T07:30:08.140492+0000","last_became_active":"2024-03-06T07:30:06.432425+0000","last_became_peered":"2024-03-06T07:30:06.432425+0000","last_unstale":"2024-03-06T07:30:08.140492+0000","last_undegraded":"2024-03-06T07:30:08.140492+0000","last_fullsized":"2024-03-06T07:30:08.140492+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:40.998267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140464+0000","last_change":"2024-03-06T07:30:06.438112+0000","last_active":"2024-03-06T07:30:08.140464+0000","last_peered":"2024-03-06T07:30:08.140464+0000","last_clean":"2024-03-06T07:30:08.140464+0000","last_became_active":"2024-03-06T07:30:06.437805+0000","last_became_peered":"2024-03-06T07:30:06.437805+0000","last_unstale":"2024-03-06T07:30:08.140464+0000","last_undegraded":"2024-03-06T07:30:08.140464+0000","last_fullsized":"2024-03-06T07:30:08.140464+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:39.860873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140599+0000","last_change":"2024-03-06T07:30:06.432199+0000","last_active":"2024-03-06T07:30:08.140599+0000","last_peered":"2024-03-06T07:30:08.140599+0000","last_clean":"2024-03-06T07:30:08.140599+0000","last_became_active":"2024-03-06T07:30:06.431945+0000","last_became_peered":"2024-03-06T07:30:06.431945+0000","last_unstale":"2024-03-06T07:30:08.140599+0000","last_undegraded":"2024-03-06T07:30:08.140599+0000","last_fullsized":"2024-03-06T07:30:08.140599+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:05.391964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140885+0000","last_change":"2024-03-06T07:30:06.434119+0000","last_active":"2024-03-06T07:30:08.140885+0000","last_peered":"2024-03-06T07:30:08.140885+0000","last_clean":"2024-03-06T07:30:08.140885+0000","last_became_active":"2024-03-06T07:30:06.433815+0000","last_became_peered":"2024-03-06T07:30:06.433815+0000","last_unstale":"2024-03-06T07:30:08.140885+0000","last_undegraded":"2024-03-06T07:30:08.140885+0000","last_fullsized":"2024-03-06T07:30:08.140885+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:25.445019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140282+0000","last_change":"2024-03-06T07:30:06.435722+0000","last_active":"2024-03-06T07:30:08.140282+0000","last_peered":"2024-03-06T07:30:08.140282+0000","last_clean":"2024-03-06T07:30:08.140282+0000","last_became_active":"2024-03-06T07:30:06.435415+0000","last_became_peered":"2024-03-06T07:30:06.435415+0000","last_unstale":"2024-03-06T07:30:08.140282+0000","last_undegraded":"2024-03-06T07:30:08.140282+0000","last_fullsized":"2024-03-06T07:30:08.140282+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:21.354139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430795+0000","last_change":"2024-03-06T07:30:06.438462+0000","last_active":"2024-03-06T07:30:07.430795+0000","last_peered":"2024-03-06T07:30:07.430795+0000","last_clean":"2024-03-06T07:30:07.430795+0000","last_became_active":"2024-03-06T07:30:06.436606+0000","last_became_peered":"2024-03-06T07:30:06.436606+0000","last_unstale":"2024-03-06T07:30:07.430795+0000","last_undegraded":"2024-03-06T07:30:07.430795+0000","last_fullsized":"2024-03-06T07:30:07.430795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:08.706684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429473+0000","last_change":"2024-03-06T07:30:06.438779+0000","last_active":"2024-03-06T07:30:07.429473+0000","last_peered":"2024-03-06T07:30:07.429473+0000","last_clean":"2024-03-06T07:30:07.429473+0000","last_became_active":"2024-03-06T07:30:06.438479+0000","last_became_peered":"2024-03-06T07:30:06.438479+0000","last_unstale":"2024-03-06T07:30:07.429473+0000","last_undegraded":"2024-03-06T07:30:07.429473+0000","last_fullsized":"2024-03-06T07:30:07.429473+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:51.962542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141184+0000","last_change":"2024-03-06T07:30:06.431242+0000","last_active":"2024-03-06T07:30:08.141184+0000","last_peered":"2024-03-06T07:30:08.141184+0000","last_clean":"2024-03-06T07:30:08.141184+0000","last_became_active":"2024-03-06T07:30:06.430983+0000","last_became_peered":"2024-03-06T07:30:06.430983+0000","last_unstale":"2024-03-06T07:30:08.141184+0000","last_undegraded":"2024-03-06T07:30:08.141184+0000","last_fullsized":"2024-03-06T07:30:08.141184+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:35.380436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140974+0000","last_change":"2024-03-06T07:30:06.445957+0000","last_active":"2024-03-06T07:30:08.140974+0000","last_peered":"2024-03-06T07:30:08.140974+0000","last_clean":"2024-03-06T07:30:08.140974+0000","last_became_active":"2024-03-06T07:30:06.444738+0000","last_became_peered":"2024-03-06T07:30:06.444738+0000","last_unstale":"2024-03-06T07:30:08.140974+0000","last_undegraded":"2024-03-06T07:30:08.140974+0000","last_fullsized":"2024-03-06T07:30:08.140974+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:59.173946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141150+0000","last_change":"2024-03-06T07:30:05.436078+0000","last_active":"2024-03-06T07:30:08.141150+0000","last_peered":"2024-03-06T07:30:08.141150+0000","last_clean":"2024-03-06T07:30:08.141150+0000","last_became_active":"2024-03-06T07:30:05.435800+0000","last_became_peered":"2024-03-06T07:30:05.435800+0000","last_unstale":"2024-03-06T07:30:08.141150+0000","last_undegraded":"2024-03-06T07:30:08.141150+0000","last_fullsized":"2024-03-06T07:30:08.141150+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:51.858548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429778+0000","last_change":"2024-03-06T07:30:06.432086+0000","last_active":"2024-03-06T07:30:07.429778+0000","last_peered":"2024-03-06T07:30:07.429778+0000","last_clean":"2024-03-06T07:30:07.429778+0000","last_became_active":"2024-03-06T07:30:06.431808+0000","last_became_peered":"2024-03-06T07:30:06.431808+0000","last_unstale":"2024-03-06T07:30:07.429778+0000","last_undegraded":"2024-03-06T07:30:07.429778+0000","last_fullsized":"2024-03-06T07:30:07.429778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:22.311235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429703+0000","last_change":"2024-03-06T07:30:05.429646+0000","last_active":"2024-03-06T07:30:07.429703+0000","last_peered":"2024-03-06T07:30:07.429703+0000","last_clean":"2024-03-06T07:30:07.429703+0000","last_became_active":"2024-03-06T07:30:05.429368+0000","last_became_peered":"2024-03-06T07:30:05.429368+0000","last_unstale":"2024-03-06T07:30:07.429703+0000","last_undegraded":"2024-03-06T07:30:07.429703+0000","last_fullsized":"2024-03-06T07:30:07.429703+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:23.880930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140438+0000","last_change":"2024-03-06T07:30:06.444588+0000","last_active":"2024-03-06T07:30:08.140438+0000","last_peered":"2024-03-06T07:30:08.140438+0000","last_clean":"2024-03-06T07:30:08.140438+0000","last_became_active":"2024-03-06T07:30:06.444311+0000","last_became_peered":"2024-03-06T07:30:06.444311+0000","last_unstale":"2024-03-06T07:30:08.140438+0000","last_undegraded":"2024-03-06T07:30:08.140438+0000","last_fullsized":"2024-03-06T07:30:08.140438+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:56.366571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431661+0000","last_change":"2024-03-06T07:30:05.425268+0000","last_active":"2024-03-06T07:30:07.431661+0000","last_peered":"2024-03-06T07:30:07.431661+0000","last_clean":"2024-03-06T07:30:07.431661+0000","last_became_active":"2024-03-06T07:30:05.424889+0000","last_became_peered":"2024-03-06T07:30:05.424889+0000","last_unstale":"2024-03-06T07:30:07.431661+0000","last_undegraded":"2024-03-06T07:30:07.431661+0000","last_fullsized":"2024-03-06T07:30:07.431661+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.896666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140902+0000","last_change":"2024-03-06T07:30:06.443944+0000","last_active":"2024-03-06T07:30:08.140902+0000","last_peered":"2024-03-06T07:30:08.140902+0000","last_clean":"2024-03-06T07:30:08.140902+0000","last_became_active":"2024-03-06T07:30:06.443634+0000","last_became_peered":"2024-03-06T07:30:06.443634+0000","last_unstale":"2024-03-06T07:30:08.140902+0000","last_undegraded":"2024-03-06T07:30:08.140902+0000","last_fullsized":"2024-03-06T07:30:08.140902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:42.126432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431693+0000","last_change":"2024-03-06T07:30:05.430523+0000","last_active":"2024-03-06T07:30:07.431693+0000","last_peered":"2024-03-06T07:30:07.431693+0000","last_clean":"2024-03-06T07:30:07.431693+0000","last_became_active":"2024-03-06T07:30:05.430245+0000","last_became_peered":"2024-03-06T07:30:05.430245+0000","last_unstale":"2024-03-06T07:30:07.431693+0000","last_undegraded":"2024-03-06T07:30:07.431693+0000","last_fullsized":"2024-03-06T07:30:07.431693+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:23.498870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429583+0000","last_change":"2024-03-06T07:30:06.439009+0000","last_active":"2024-03-06T07:30:07.429583+0000","last_peered":"2024-03-06T07:30:07.429583+0000","last_clean":"2024-03-06T07:30:07.429583+0000","last_became_active":"2024-03-06T07:30:06.437823+0000","last_became_peered":"2024-03-06T07:30:06.437823+0000","last_unstale":"2024-03-06T07:30:07.429583+0000","last_undegraded":"2024-03-06T07:30:07.429583+0000","last_fullsized":"2024-03-06T07:30:07.429583+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:20.284944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140006+0000","last_change":"2024-03-06T07:30:05.431747+0000","last_active":"2024-03-06T07:30:08.140006+0000","last_peered":"2024-03-06T07:30:08.140006+0000","last_clean":"2024-03-06T07:30:08.140006+0000","last_became_active":"2024-03-06T07:30:05.431476+0000","last_became_peered":"2024-03-06T07:30:05.431476+0000","last_unstale":"2024-03-06T07:30:08.140006+0000","last_undegraded":"2024-03-06T07:30:08.140006+0000","last_fullsized":"2024-03-06T07:30:08.140006+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:57.743131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430930+0000","last_change":"2024-03-06T07:30:06.436707+0000","last_active":"2024-03-06T07:30:07.430930+0000","last_peered":"2024-03-06T07:30:07.430930+0000","last_clean":"2024-03-06T07:30:07.430930+0000","last_became_active":"2024-03-06T07:30:06.436365+0000","last_became_peered":"2024-03-06T07:30:06.436365+0000","last_unstale":"2024-03-06T07:30:07.430930+0000","last_undegraded":"2024-03-06T07:30:07.430930+0000","last_fullsized":"2024-03-06T07:30:07.430930+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.851347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140047+0000","last_change":"2024-03-06T07:30:05.429107+0000","last_active":"2024-03-06T07:30:08.140047+0000","last_peered":"2024-03-06T07:30:08.140047+0000","last_clean":"2024-03-06T07:30:08.140047+0000","last_became_active":"2024-03-06T07:30:05.428799+0000","last_became_peered":"2024-03-06T07:30:05.428799+0000","last_unstale":"2024-03-06T07:30:08.140047+0000","last_undegraded":"2024-03-06T07:30:08.140047+0000","last_fullsized":"2024-03-06T07:30:08.140047+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:38.198985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429474+0000","last_change":"2024-03-06T07:30:06.438432+0000","last_active":"2024-03-06T07:30:07.429474+0000","last_peered":"2024-03-06T07:30:07.429474+0000","last_clean":"2024-03-06T07:30:07.429474+0000","last_became_active":"2024-03-06T07:30:06.438130+0000","last_became_peered":"2024-03-06T07:30:06.438130+0000","last_unstale":"2024-03-06T07:30:07.429474+0000","last_undegraded":"2024-03-06T07:30:07.429474+0000","last_fullsized":"2024-03-06T07:30:07.429474+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:07.777780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140168+0000","last_change":"2024-03-06T07:30:05.428040+0000","last_active":"2024-03-06T07:30:08.140168+0000","last_peered":"2024-03-06T07:30:08.140168+0000","last_clean":"2024-03-06T07:30:08.140168+0000","last_became_active":"2024-03-06T07:30:05.427766+0000","last_became_peered":"2024-03-06T07:30:05.427766+0000","last_unstale":"2024-03-06T07:30:08.140168+0000","last_undegraded":"2024-03-06T07:30:08.140168+0000","last_fullsized":"2024-03-06T07:30:08.140168+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:28.915446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140349+0000","last_change":"2024-03-06T07:30:06.445238+0000","last_active":"2024-03-06T07:30:08.140349+0000","last_peered":"2024-03-06T07:30:08.140349+0000","last_clean":"2024-03-06T07:30:08.140349+0000","last_became_active":"2024-03-06T07:30:06.444940+0000","last_became_peered":"2024-03-06T07:30:06.444940+0000","last_unstale":"2024-03-06T07:30:08.140349+0000","last_undegraded":"2024-03-06T07:30:08.140349+0000","last_fullsized":"2024-03-06T07:30:08.140349+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:55.500634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430291+0000","last_change":"2024-03-06T07:30:05.431073+0000","last_active":"2024-03-06T07:30:07.430291+0000","last_peered":"2024-03-06T07:30:07.430291+0000","last_clean":"2024-03-06T07:30:07.430291+0000","last_became_active":"2024-03-06T07:30:05.430803+0000","last_became_peered":"2024-03-06T07:30:05.430803+0000","last_unstale":"2024-03-06T07:30:07.430291+0000","last_undegraded":"2024-03-06T07:30:07.430291+0000","last_fullsized":"2024-03-06T07:30:07.430291+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:51.242508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141066+0000","last_change":"2024-03-06T07:30:06.440003+0000","last_active":"2024-03-06T07:30:08.141066+0000","last_peered":"2024-03-06T07:30:08.141066+0000","last_clean":"2024-03-06T07:30:08.141066+0000","last_became_active":"2024-03-06T07:30:06.439695+0000","last_became_peered":"2024-03-06T07:30:06.439695+0000","last_unstale":"2024-03-06T07:30:08.141066+0000","last_undegraded":"2024-03-06T07:30:08.141066+0000","last_fullsized":"2024-03-06T07:30:08.141066+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:58.135047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429735+0000","last_change":"2024-03-06T07:30:05.427926+0000","last_active":"2024-03-06T07:30:07.429735+0000","last_peered":"2024-03-06T07:30:07.429735+0000","last_clean":"2024-03-06T07:30:07.429735+0000","last_became_active":"2024-03-06T07:30:05.427561+0000","last_became_peered":"2024-03-06T07:30:05.427561+0000","last_unstale":"2024-03-06T07:30:07.429735+0000","last_undegraded":"2024-03-06T07:30:07.429735+0000","last_fullsized":"2024-03-06T07:30:07.429735+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:10.869396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141186+0000","last_change":"2024-03-06T07:30:06.434140+0000","last_active":"2024-03-06T07:30:08.141186+0000","last_peered":"2024-03-06T07:30:08.141186+0000","last_clean":"2024-03-06T07:30:08.141186+0000","last_became_active":"2024-03-06T07:30:06.433792+0000","last_became_peered":"2024-03-06T07:30:06.433792+0000","last_unstale":"2024-03-06T07:30:08.141186+0000","last_undegraded":"2024-03-06T07:30:08.141186+0000","last_fullsized":"2024-03-06T07:30:08.141186+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:06.310028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141138+0000","last_change":"2024-03-06T07:30:05.434526+0000","last_active":"2024-03-06T07:30:08.141138+0000","last_peered":"2024-03-06T07:30:08.141138+0000","last_clean":"2024-03-06T07:30:08.141138+0000","last_became_active":"2024-03-06T07:30:05.434211+0000","last_became_peered":"2024-03-06T07:30:05.434211+0000","last_unstale":"2024-03-06T07:30:08.141138+0000","last_undegraded":"2024-03-06T07:30:08.141138+0000","last_fullsized":"2024-03-06T07:30:08.141138+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:21.307485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140731+0000","last_change":"2024-03-06T07:30:06.427120+0000","last_active":"2024-03-06T07:30:08.140731+0000","last_peered":"2024-03-06T07:30:08.140731+0000","last_clean":"2024-03-06T07:30:08.140731+0000","last_became_active":"2024-03-06T07:30:06.426721+0000","last_became_peered":"2024-03-06T07:30:06.426721+0000","last_unstale":"2024-03-06T07:30:08.140731+0000","last_undegraded":"2024-03-06T07:30:08.140731+0000","last_fullsized":"2024-03-06T07:30:08.140731+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:53.625998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":20,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141073+0000","last_change":"2024-03-06T07:30:05.428068+0000","last_active":"2024-03-06T07:30:08.141073+0000","last_peered":"2024-03-06T07:30:08.141073+0000","last_clean":"2024-03-06T07:30:08.141073+0000","last_became_active":"2024-03-06T07:30:05.427797+0000","last_became_peered":"2024-03-06T07:30:05.427797+0000","last_unstale":"2024-03-06T07:30:08.141073+0000","last_undegraded":"2024-03-06T07:30:08.141073+0000","last_fullsized":"2024-03-06T07:30:08.141073+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:55.746594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431407+0000","last_change":"2024-03-06T07:30:06.444899+0000","last_active":"2024-03-06T07:30:07.431407+0000","last_peered":"2024-03-06T07:30:07.431407+0000","last_clean":"2024-03-06T07:30:07.431407+0000","last_became_active":"2024-03-06T07:30:06.444629+0000","last_became_peered":"2024-03-06T07:30:06.444629+0000","last_unstale":"2024-03-06T07:30:07.431407+0000","last_undegraded":"2024-03-06T07:30:07.431407+0000","last_fullsized":"2024-03-06T07:30:07.431407+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:23.597805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430266+0000","last_change":"2024-03-06T07:30:05.428611+0000","last_active":"2024-03-06T07:30:07.430266+0000","last_peered":"2024-03-06T07:30:07.430266+0000","last_clean":"2024-03-06T07:30:07.430266+0000","last_became_active":"2024-03-06T07:30:05.427986+0000","last_became_peered":"2024-03-06T07:30:05.427986+0000","last_unstale":"2024-03-06T07:30:07.430266+0000","last_undegraded":"2024-03-06T07:30:07.430266+0000","last_fullsized":"2024-03-06T07:30:07.430266+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:43.822709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429606+0000","last_change":"2024-03-06T07:30:06.429341+0000","last_active":"2024-03-06T07:30:07.429606+0000","last_peered":"2024-03-06T07:30:07.429606+0000","last_clean":"2024-03-06T07:30:07.429606+0000","last_became_active":"2024-03-06T07:30:06.429065+0000","last_became_peered":"2024-03-06T07:30:06.429065+0000","last_unstale":"2024-03-06T07:30:07.429606+0000","last_undegraded":"2024-03-06T07:30:07.429606+0000","last_fullsized":"2024-03-06T07:30:07.429606+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:36.213074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431429+0000","last_change":"2024-03-06T07:30:05.427120+0000","last_active":"2024-03-06T07:30:07.431429+0000","last_peered":"2024-03-06T07:30:07.431429+0000","last_clean":"2024-03-06T07:30:07.431429+0000","last_became_active":"2024-03-06T07:30:05.426723+0000","last_became_peered":"2024-03-06T07:30:05.426723+0000","last_unstale":"2024-03-06T07:30:07.431429+0000","last_undegraded":"2024-03-06T07:30:07.431429+0000","last_fullsized":"2024-03-06T07:30:07.431429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:41.217266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140184+0000","last_change":"2024-03-06T07:30:06.448277+0000","last_active":"2024-03-06T07:30:08.140184+0000","last_peered":"2024-03-06T07:30:08.140184+0000","last_clean":"2024-03-06T07:30:08.140184+0000","last_became_active":"2024-03-06T07:30:06.448011+0000","last_became_peered":"2024-03-06T07:30:06.448011+0000","last_unstale":"2024-03-06T07:30:08.140184+0000","last_undegraded":"2024-03-06T07:30:08.140184+0000","last_fullsized":"2024-03-06T07:30:08.140184+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:20.900417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140082+0000","last_change":"2024-03-06T07:30:05.431252+0000","last_active":"2024-03-06T07:30:08.140082+0000","last_peered":"2024-03-06T07:30:08.140082+0000","last_clean":"2024-03-06T07:30:08.140082+0000","last_became_active":"2024-03-06T07:30:05.430910+0000","last_became_peered":"2024-03-06T07:30:05.430910+0000","last_unstale":"2024-03-06T07:30:08.140082+0000","last_undegraded":"2024-03-06T07:30:08.140082+0000","last_fullsized":"2024-03-06T07:30:08.140082+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:03.669439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430534+0000","last_change":"2024-03-06T07:30:06.425936+0000","last_active":"2024-03-06T07:30:07.430534+0000","last_peered":"2024-03-06T07:30:07.430534+0000","last_clean":"2024-03-06T07:30:07.430534+0000","last_became_active":"2024-03-06T07:30:06.425656+0000","last_became_peered":"2024-03-06T07:30:06.425656+0000","last_unstale":"2024-03-06T07:30:07.430534+0000","last_undegraded":"2024-03-06T07:30:07.430534+0000","last_fullsized":"2024-03-06T07:30:07.430534+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:03.753224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":22,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141239+0000","last_change":"2024-03-06T07:30:05.426995+0000","last_active":"2024-03-06T07:30:08.141239+0000","last_peered":"2024-03-06T07:30:08.141239+0000","last_clean":"2024-03-06T07:30:08.141239+0000","last_became_active":"2024-03-06T07:30:05.426583+0000","last_became_peered":"2024-03-06T07:30:05.426583+0000","last_unstale":"2024-03-06T07:30:08.141239+0000","last_undegraded":"2024-03-06T07:30:08.141239+0000","last_fullsized":"2024-03-06T07:30:08.141239+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:04.904745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141102+0000","last_change":"2024-03-06T07:30:06.433942+0000","last_active":"2024-03-06T07:30:08.141102+0000","last_peered":"2024-03-06T07:30:08.141102+0000","last_clean":"2024-03-06T07:30:08.141102+0000","last_became_active":"2024-03-06T07:30:06.433653+0000","last_became_peered":"2024-03-06T07:30:06.433653+0000","last_unstale":"2024-03-06T07:30:08.141102+0000","last_undegraded":"2024-03-06T07:30:08.141102+0000","last_fullsized":"2024-03-06T07:30:08.141102+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:31.848662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140553+0000","last_change":"2024-03-06T07:30:05.433085+0000","last_active":"2024-03-06T07:30:08.140553+0000","last_peered":"2024-03-06T07:30:08.140553+0000","last_clean":"2024-03-06T07:30:08.140553+0000","last_became_active":"2024-03-06T07:30:05.432752+0000","last_became_peered":"2024-03-06T07:30:05.432752+0000","last_unstale":"2024-03-06T07:30:08.140553+0000","last_undegraded":"2024-03-06T07:30:08.140553+0000","last_fullsized":"2024-03-06T07:30:08.140553+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:05.048954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140629+0000","last_change":"2024-03-06T07:30:06.433024+0000","last_active":"2024-03-06T07:30:08.140629+0000","last_peered":"2024-03-06T07:30:08.140629+0000","last_clean":"2024-03-06T07:30:08.140629+0000","last_became_active":"2024-03-06T07:30:06.432731+0000","last_became_peered":"2024-03-06T07:30:06.432731+0000","last_unstale":"2024-03-06T07:30:08.140629+0000","last_undegraded":"2024-03-06T07:30:08.140629+0000","last_fullsized":"2024-03-06T07:30:08.140629+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:28.631179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429769+0000","last_change":"2024-03-06T07:30:05.422150+0000","last_active":"2024-03-06T07:30:07.429769+0000","last_peered":"2024-03-06T07:30:07.429769+0000","last_clean":"2024-03-06T07:30:07.429769+0000","last_became_active":"2024-03-06T07:30:05.421869+0000","last_became_peered":"2024-03-06T07:30:05.421869+0000","last_unstale":"2024-03-06T07:30:07.429769+0000","last_undegraded":"2024-03-06T07:30:07.429769+0000","last_fullsized":"2024-03-06T07:30:07.429769+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:05.664021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431060+0000","last_change":"2024-03-06T07:30:06.443997+0000","last_active":"2024-03-06T07:30:07.431060+0000","last_peered":"2024-03-06T07:30:07.431060+0000","last_clean":"2024-03-06T07:30:07.431060+0000","last_became_active":"2024-03-06T07:30:06.443667+0000","last_became_peered":"2024-03-06T07:30:06.443667+0000","last_unstale":"2024-03-06T07:30:07.431060+0000","last_undegraded":"2024-03-06T07:30:07.431060+0000","last_fullsized":"2024-03-06T07:30:07.431060+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:25.275095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140780+0000","last_change":"2024-03-06T07:30:05.428760+0000","last_active":"2024-03-06T07:30:08.140780+0000","last_peered":"2024-03-06T07:30:08.140780+0000","last_clean":"2024-03-06T07:30:08.140780+0000","last_became_active":"2024-03-06T07:30:05.428516+0000","last_became_peered":"2024-03-06T07:30:05.428516+0000","last_unstale":"2024-03-06T07:30:08.140780+0000","last_undegraded":"2024-03-06T07:30:08.140780+0000","last_fullsized":"2024-03-06T07:30:08.140780+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:52.427793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140460+0000","last_change":"2024-03-06T07:30:06.436906+0000","last_active":"2024-03-06T07:30:08.140460+0000","last_peered":"2024-03-06T07:30:08.140460+0000","last_clean":"2024-03-06T07:30:08.140460+0000","last_became_active":"2024-03-06T07:30:06.436630+0000","last_became_peered":"2024-03-06T07:30:06.436630+0000","last_unstale":"2024-03-06T07:30:08.140460+0000","last_undegraded":"2024-03-06T07:30:08.140460+0000","last_fullsized":"2024-03-06T07:30:08.140460+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:51.322488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429690+0000","last_change":"2024-03-06T07:30:05.424132+0000","last_active":"2024-03-06T07:30:07.429690+0000","last_peered":"2024-03-06T07:30:07.429690+0000","last_clean":"2024-03-06T07:30:07.429690+0000","last_became_active":"2024-03-06T07:30:05.423850+0000","last_became_peered":"2024-03-06T07:30:05.423850+0000","last_unstale":"2024-03-06T07:30:07.429690+0000","last_undegraded":"2024-03-06T07:30:07.429690+0000","last_fullsized":"2024-03-06T07:30:07.429690+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:27.784139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430883+0000","last_change":"2024-03-06T07:30:06.438228+0000","last_active":"2024-03-06T07:30:07.430883+0000","last_peered":"2024-03-06T07:30:07.430883+0000","last_clean":"2024-03-06T07:30:07.430883+0000","last_became_active":"2024-03-06T07:30:06.436387+0000","last_became_peered":"2024-03-06T07:30:06.436387+0000","last_unstale":"2024-03-06T07:30:07.430883+0000","last_undegraded":"2024-03-06T07:30:07.430883+0000","last_fullsized":"2024-03-06T07:30:07.430883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:03.753224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141279+0000","last_change":"2024-03-06T07:30:05.432006+0000","last_active":"2024-03-06T07:30:08.141279+0000","last_peered":"2024-03-06T07:30:08.141279+0000","last_clean":"2024-03-06T07:30:08.141279+0000","last_became_active":"2024-03-06T07:30:05.431751+0000","last_became_peered":"2024-03-06T07:30:05.431751+0000","last_unstale":"2024-03-06T07:30:08.141279+0000","last_undegraded":"2024-03-06T07:30:08.141279+0000","last_fullsized":"2024-03-06T07:30:08.141279+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:24.602102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141235+0000","last_change":"2024-03-06T07:30:06.441766+0000","last_active":"2024-03-06T07:30:08.141235+0000","last_peered":"2024-03-06T07:30:08.141235+0000","last_clean":"2024-03-06T07:30:08.141235+0000","last_became_active":"2024-03-06T07:30:06.441497+0000","last_became_peered":"2024-03-06T07:30:06.441497+0000","last_unstale":"2024-03-06T07:30:08.141235+0000","last_undegraded":"2024-03-06T07:30:08.141235+0000","last_fullsized":"2024-03-06T07:30:08.141235+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:42.126432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140475+0000","last_change":"2024-03-06T07:30:05.434762+0000","last_active":"2024-03-06T07:30:08.140475+0000","last_peered":"2024-03-06T07:30:08.140475+0000","last_clean":"2024-03-06T07:30:08.140475+0000","last_became_active":"2024-03-06T07:30:05.434415+0000","last_became_peered":"2024-03-06T07:30:05.434415+0000","last_unstale":"2024-03-06T07:30:08.140475+0000","last_undegraded":"2024-03-06T07:30:08.140475+0000","last_fullsized":"2024-03-06T07:30:08.140475+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:30.353000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430236+0000","last_change":"2024-03-06T07:30:06.435882+0000","last_active":"2024-03-06T07:30:07.430236+0000","last_peered":"2024-03-06T07:30:07.430236+0000","last_clean":"2024-03-06T07:30:07.430236+0000","last_became_active":"2024-03-06T07:30:06.435602+0000","last_became_peered":"2024-03-06T07:30:06.435602+0000","last_unstale":"2024-03-06T07:30:07.430236+0000","last_undegraded":"2024-03-06T07:30:07.430236+0000","last_fullsized":"2024-03-06T07:30:07.430236+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:56.366571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140561+0000","last_change":"2024-03-06T07:30:05.434446+0000","last_active":"2024-03-06T07:30:08.140561+0000","last_peered":"2024-03-06T07:30:08.140561+0000","last_clean":"2024-03-06T07:30:08.140561+0000","last_became_active":"2024-03-06T07:30:05.434151+0000","last_became_peered":"2024-03-06T07:30:05.434151+0000","last_unstale":"2024-03-06T07:30:08.140561+0000","last_undegraded":"2024-03-06T07:30:08.140561+0000","last_fullsized":"2024-03-06T07:30:08.140561+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:35.209323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141166+0000","last_change":"2024-03-06T07:30:06.442118+0000","last_active":"2024-03-06T07:30:08.141166+0000","last_peered":"2024-03-06T07:30:08.141166+0000","last_clean":"2024-03-06T07:30:08.141166+0000","last_became_active":"2024-03-06T07:30:06.441544+0000","last_became_peered":"2024-03-06T07:30:06.441544+0000","last_unstale":"2024-03-06T07:30:08.141166+0000","last_undegraded":"2024-03-06T07:30:08.141166+0000","last_fullsized":"2024-03-06T07:30:08.141166+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:31.906079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430401+0000","last_change":"2024-03-06T07:30:05.432082+0000","last_active":"2024-03-06T07:30:07.430401+0000","last_peered":"2024-03-06T07:30:07.430401+0000","last_clean":"2024-03-06T07:30:07.430401+0000","last_became_active":"2024-03-06T07:30:05.431772+0000","last_became_peered":"2024-03-06T07:30:05.431772+0000","last_unstale":"2024-03-06T07:30:07.430401+0000","last_undegraded":"2024-03-06T07:30:07.430401+0000","last_fullsized":"2024-03-06T07:30:07.430401+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:32.843650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140912+0000","last_change":"2024-03-06T07:30:06.449028+0000","last_active":"2024-03-06T07:30:08.140912+0000","last_peered":"2024-03-06T07:30:08.140912+0000","last_clean":"2024-03-06T07:30:08.140912+0000","last_became_active":"2024-03-06T07:30:06.448717+0000","last_became_peered":"2024-03-06T07:30:06.448717+0000","last_unstale":"2024-03-06T07:30:08.140912+0000","last_undegraded":"2024-03-06T07:30:08.140912+0000","last_fullsized":"2024-03-06T07:30:08.140912+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:44.826460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141273+0000","last_change":"2024-03-06T07:30:05.434003+0000","last_active":"2024-03-06T07:30:08.141273+0000","last_peered":"2024-03-06T07:30:08.141273+0000","last_clean":"2024-03-06T07:30:08.141273+0000","last_became_active":"2024-03-06T07:30:05.433714+0000","last_became_peered":"2024-03-06T07:30:05.433714+0000","last_unstale":"2024-03-06T07:30:08.141273+0000","last_undegraded":"2024-03-06T07:30:08.141273+0000","last_fullsized":"2024-03-06T07:30:08.141273+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:21.133625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430990+0000","last_change":"2024-03-06T07:30:06.437969+0000","last_active":"2024-03-06T07:30:07.430990+0000","last_peered":"2024-03-06T07:30:07.430990+0000","last_clean":"2024-03-06T07:30:07.430990+0000","last_became_active":"2024-03-06T07:30:06.435764+0000","last_became_peered":"2024-03-06T07:30:06.435764+0000","last_unstale":"2024-03-06T07:30:07.430990+0000","last_undegraded":"2024-03-06T07:30:07.430990+0000","last_fullsized":"2024-03-06T07:30:07.430990+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.851347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"139'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140571+0000","last_change":"2024-03-06T07:30:05.433552+0000","last_active":"2024-03-06T07:30:08.140571+0000","last_peered":"2024-03-06T07:30:08.140571+0000","last_clean":"2024-03-06T07:30:08.140571+0000","last_became_active":"2024-03-06T07:30:05.433273+0000","last_became_peered":"2024-03-06T07:30:05.433273+0000","last_unstale":"2024-03-06T07:30:08.140571+0000","last_undegraded":"2024-03-06T07:30:08.140571+0000","last_fullsized":"2024-03-06T07:30:08.140571+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:51.396802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429650+0000","last_change":"2024-03-06T07:30:06.444627+0000","last_active":"2024-03-06T07:30:07.429650+0000","last_peered":"2024-03-06T07:30:07.429650+0000","last_clean":"2024-03-06T07:30:07.429650+0000","last_became_active":"2024-03-06T07:30:06.444338+0000","last_became_peered":"2024-03-06T07:30:06.444338+0000","last_unstale":"2024-03-06T07:30:07.429650+0000","last_undegraded":"2024-03-06T07:30:07.429650+0000","last_fullsized":"2024-03-06T07:30:07.429650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:31.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140651+0000","last_change":"2024-03-06T07:30:05.432338+0000","last_active":"2024-03-06T07:30:08.140651+0000","last_peered":"2024-03-06T07:30:08.140651+0000","last_clean":"2024-03-06T07:30:08.140651+0000","last_became_active":"2024-03-06T07:30:05.432047+0000","last_became_peered":"2024-03-06T07:30:05.432047+0000","last_unstale":"2024-03-06T07:30:08.140651+0000","last_undegraded":"2024-03-06T07:30:08.140651+0000","last_fullsized":"2024-03-06T07:30:08.140651+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:01.659447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140604+0000","last_change":"2024-03-06T07:30:06.447149+0000","last_active":"2024-03-06T07:30:08.140604+0000","last_peered":"2024-03-06T07:30:08.140604+0000","last_clean":"2024-03-06T07:30:08.140604+0000","last_became_active":"2024-03-06T07:30:06.446453+0000","last_became_peered":"2024-03-06T07:30:06.446453+0000","last_unstale":"2024-03-06T07:30:08.140604+0000","last_undegraded":"2024-03-06T07:30:08.140604+0000","last_fullsized":"2024-03-06T07:30:08.140604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:08.706684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140578+0000","last_change":"2024-03-06T07:30:05.431921+0000","last_active":"2024-03-06T07:30:08.140578+0000","last_peered":"2024-03-06T07:30:08.140578+0000","last_clean":"2024-03-06T07:30:08.140578+0000","last_became_active":"2024-03-06T07:30:05.431415+0000","last_became_peered":"2024-03-06T07:30:05.431415+0000","last_unstale":"2024-03-06T07:30:08.140578+0000","last_undegraded":"2024-03-06T07:30:08.140578+0000","last_fullsized":"2024-03-06T07:30:08.140578+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:42.591738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140096+0000","last_change":"2024-03-06T07:30:06.445510+0000","last_active":"2024-03-06T07:30:08.140096+0000","last_peered":"2024-03-06T07:30:08.140096+0000","last_clean":"2024-03-06T07:30:08.140096+0000","last_became_active":"2024-03-06T07:30:06.445211+0000","last_became_peered":"2024-03-06T07:30:06.445211+0000","last_unstale":"2024-03-06T07:30:08.140096+0000","last_undegraded":"2024-03-06T07:30:08.140096+0000","last_fullsized":"2024-03-06T07:30:08.140096+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:54.737893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141358+0000","last_change":"2024-03-06T07:30:05.425600+0000","last_active":"2024-03-06T07:30:08.141358+0000","last_peered":"2024-03-06T07:30:08.141358+0000","last_clean":"2024-03-06T07:30:08.141358+0000","last_became_active":"2024-03-06T07:30:05.425345+0000","last_became_peered":"2024-03-06T07:30:05.425345+0000","last_unstale":"2024-03-06T07:30:08.141358+0000","last_undegraded":"2024-03-06T07:30:08.141358+0000","last_fullsized":"2024-03-06T07:30:08.141358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:19.474761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140542+0000","last_change":"2024-03-06T07:30:06.437964+0000","last_active":"2024-03-06T07:30:08.140542+0000","last_peered":"2024-03-06T07:30:08.140542+0000","last_clean":"2024-03-06T07:30:08.140542+0000","last_became_active":"2024-03-06T07:30:06.437249+0000","last_became_peered":"2024-03-06T07:30:06.437249+0000","last_unstale":"2024-03-06T07:30:08.140542+0000","last_undegraded":"2024-03-06T07:30:08.140542+0000","last_fullsized":"2024-03-06T07:30:08.140542+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:58.261221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140621+0000","last_change":"2024-03-06T07:30:05.433413+0000","last_active":"2024-03-06T07:30:08.140621+0000","last_peered":"2024-03-06T07:30:08.140621+0000","last_clean":"2024-03-06T07:30:08.140621+0000","last_became_active":"2024-03-06T07:30:05.433106+0000","last_became_peered":"2024-03-06T07:30:05.433106+0000","last_unstale":"2024-03-06T07:30:08.140621+0000","last_undegraded":"2024-03-06T07:30:08.140621+0000","last_fullsized":"2024-03-06T07:30:08.140621+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:35.223245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140328+0000","last_change":"2024-03-06T07:30:06.432313+0000","last_active":"2024-03-06T07:30:08.140328+0000","last_peered":"2024-03-06T07:30:08.140328+0000","last_clean":"2024-03-06T07:30:08.140328+0000","last_became_active":"2024-03-06T07:30:06.432026+0000","last_became_peered":"2024-03-06T07:30:06.432026+0000","last_unstale":"2024-03-06T07:30:08.140328+0000","last_undegraded":"2024-03-06T07:30:08.140328+0000","last_fullsized":"2024-03-06T07:30:08.140328+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:03.230520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.430595+0000","last_change":"2024-03-06T07:30:05.429495+0000","last_active":"2024-03-06T07:30:07.430595+0000","last_peered":"2024-03-06T07:30:07.430595+0000","last_clean":"2024-03-06T07:30:07.430595+0000","last_became_active":"2024-03-06T07:30:05.429174+0000","last_became_peered":"2024-03-06T07:30:05.429174+0000","last_unstale":"2024-03-06T07:30:07.430595+0000","last_undegraded":"2024-03-06T07:30:07.430595+0000","last_fullsized":"2024-03-06T07:30:07.430595+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:57.131296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140212+0000","last_change":"2024-03-06T07:30:06.441006+0000","last_active":"2024-03-06T07:30:08.140212+0000","last_peered":"2024-03-06T07:30:08.140212+0000","last_clean":"2024-03-06T07:30:08.140212+0000","last_became_active":"2024-03-06T07:30:06.440737+0000","last_became_peered":"2024-03-06T07:30:06.440737+0000","last_unstale":"2024-03-06T07:30:08.140212+0000","last_undegraded":"2024-03-06T07:30:08.140212+0000","last_fullsized":"2024-03-06T07:30:08.140212+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:52.246259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.429430+0000","last_change":"2024-03-06T07:30:05.430432+0000","last_active":"2024-03-06T07:30:07.429430+0000","last_peered":"2024-03-06T07:30:07.429430+0000","last_clean":"2024-03-06T07:30:07.429430+0000","last_became_active":"2024-03-06T07:30:05.430155+0000","last_became_peered":"2024-03-06T07:30:05.430155+0000","last_unstale":"2024-03-06T07:30:07.429430+0000","last_undegraded":"2024-03-06T07:30:07.429430+0000","last_fullsized":"2024-03-06T07:30:07.429430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:06.402939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.141019+0000","last_change":"2024-03-06T07:30:06.444436+0000","last_active":"2024-03-06T07:30:08.141019+0000","last_peered":"2024-03-06T07:30:08.141019+0000","last_clean":"2024-03-06T07:30:08.141019+0000","last_became_active":"2024-03-06T07:30:06.444155+0000","last_became_peered":"2024-03-06T07:30:06.444155+0000","last_unstale":"2024-03-06T07:30:08.141019+0000","last_undegraded":"2024-03-06T07:30:08.141019+0000","last_fullsized":"2024-03-06T07:30:08.141019+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.412087+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.412087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:03.230520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.431502+0000","last_change":"2024-03-06T07:30:05.432091+0000","last_active":"2024-03-06T07:30:07.431502+0000","last_peered":"2024-03-06T07:30:07.431502+0000","last_clean":"2024-03-06T07:30:07.431502+0000","last_became_active":"2024-03-06T07:30:05.431797+0000","last_became_peered":"2024-03-06T07:30:05.431797+0000","last_unstale":"2024-03-06T07:30:07.431502+0000","last_undegraded":"2024-03-06T07:30:07.431502+0000","last_fullsized":"2024-03-06T07:30:07.431502+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.408336+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.408336+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:57.257470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":397,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:08.140470+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:30:08.140470+0000","last_peered":"2024-03-06T07:30:08.140470+0000","last_clean":"2024-03-06T07:30:08.140470+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:30:08.140470+0000","last_undegraded":"2024-03-06T07:30:08.140470+0000","last_fullsized":"2024-03-06T07:30:08.140470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574953,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2342912,"data_stored":1352290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574953,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41212,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330628,"statfs":{"total":96636764160,"available":96594563072,"internally_reserved":0,"allocated":1753088,"data_stored":761416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574953,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41208,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330632,"statfs":{"total":96636764160,"available":96594567168,"internally_reserved":0,"allocated":1744896,"data_stored":761292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41200,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330640,"statfs":{"total":96636764160,"available":96594575360,"internally_reserved":0,"allocated":1740800,"data_stored":761270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41204,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330636,"statfs":{"total":96636764160,"available":96594571264,"internally_reserved":0,"allocated":1744896,"data_stored":761360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41216,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330624,"statfs":{"total":96636764160,"available":96594558976,"internally_reserved":0,"allocated":1757184,"data_stored":762952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41204,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330636,"statfs":{"total":96636764160,"available":96594571264,"internally_reserved":0,"allocated":1744896,"data_stored":761852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2351104,"data_stored":1353284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:14.349 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:14.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:14.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.494+0000 7ff19c5a0640 1 Processor -- start 2024-03-06T07:30:14.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.494+0000 7ff19c5a0640 1 -- start start 2024-03-06T07:30:14.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19c5a0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff1940d4860 0x7ff1940d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19c5a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff1940b0e30 0x7ff1940b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19c5a0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1941618b0 con 0x7ff1941614d0 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19c5a0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1940d4cf0 con 0x7ff1940b0e30 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19c5a0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1940cf400 con 0x7ff1940d4860 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff1940b0e30 0x7ff1940b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff1940b0e30 0x7ff1940b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52140/0 (socket says 172.21.15.122:52140) 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19a33c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff1941614d0 0x7ff1940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37064/0 (socket says 172.21.15.122:37064) 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 -- 172.21.15.122:0/3513055917 learned_addr learned my addr 172.21.15.122:0/3513055917 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff199b3b640 1 --2- 172.21.15.122:0/3513055917 >> v2:172.21.15.169:3300/0 conn(0x7ff1940d4860 0x7ff1940d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 -- 172.21.15.122:0/3513055917 >> v2:172.21.15.169:3300/0 conn(0x7ff1940d4860 msgr2=0x7ff1940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 --2- 172.21.15.122:0/3513055917 >> v2:172.21.15.169:3300/0 conn(0x7ff1940d4860 0x7ff1940d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 -- 172.21.15.122:0/3513055917 >> v1:172.21.15.122:6789/0 conn(0x7ff1941614d0 legacy=0x7ff1940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff19ab3d640 1 -- 172.21.15.122:0/3513055917 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1940cf6a0 con 0x7ff1940b0e30 2024-03-06T07:30:14.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.495+0000 7ff199b3b640 1 --2- 172.21.15.122:0/3513055917 >> v2:172.21.15.169:3300/0 conn(0x7ff1940d4860 0x7ff1940d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.496+0000 7ff19ab3d640 1 --2- 172.21.15.122:0/3513055917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1940b0e30 0x7ff1940b1260 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff1940ce550 tx=0x7ff18c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2640389fef3f39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.496+0000 7ff19933a640 1 -- 172.21.15.122:0/3513055917 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c02cb20 con 0x7ff1940b0e30 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.496+0000 7ff19933a640 1 -- 172.21.15.122:0/3513055917 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18c02ccc0 con 0x7ff1940b0e30 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.496+0000 7ff19933a640 1 -- 172.21.15.122:0/3513055917 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c034610 con 0x7ff1940b0e30 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 -- 172.21.15.122:0/3513055917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1940b0e30 msgr2=0x7ff1940b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/3513055917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1940b0e30 0x7ff1940b1260 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff1940ce550 tx=0x7ff18c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 -- 172.21.15.122:0/3513055917 shutdown_connections 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/3513055917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1940b0e30 0x7ff1940b1260 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/3513055917 >> v2:172.21.15.169:3300/0 conn(0x7ff1940d4860 0x7ff1940d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 -- 172.21.15.122:0/3513055917 >> 172.21.15.122:0/3513055917 conn(0x7ff1940d3d40 msgr2=0x7ff1940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 -- 172.21.15.122:0/3513055917 shutdown_connections 2024-03-06T07:30:14.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.497+0000 7ff19c5a0640 1 -- 172.21.15.122:0/3513055917 wait complete. 2024-03-06T07:30:14.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.499+0000 7ff19c5a0640 1 Processor -- start 2024-03-06T07:30:14.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.499+0000 7ff19c5a0640 1 -- start start 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19c5a0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1940b0e30 0x7ff1941552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19c5a0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff1940d4860 0x7ff19415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19c5a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 0x7ff1940cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19c5a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940b0910 con 0x7ff1941614d0 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19c5a0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940cfc60 con 0x7ff1940b0e30 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19c5a0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940d2560 con 0x7ff1940d4860 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19ab3d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 0x7ff1940cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19ab3d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 0x7ff1940cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52150/0 (socket says 172.21.15.122:52150) 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19ab3d640 1 -- 172.21.15.122:0/459906772 learned_addr learned my addr 172.21.15.122:0/459906772 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19a33c640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1940b0e30 0x7ff1941552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff199b3b640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff1940d4860 0x7ff19415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19ab3d640 1 -- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff1940d4860 msgr2=0x7ff19415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.500+0000 7ff19ab3d640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff1940d4860 0x7ff19415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff19ab3d640 1 -- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1940b0e30 msgr2=0x7ff1941552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff19ab3d640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1940b0e30 0x7ff1941552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff19ab3d640 1 -- 172.21.15.122:0/459906772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1940caeb0 con 0x7ff1941614d0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff199b3b640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff1940d4860 0x7ff19415baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff19ab3d640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 0x7ff1940cb950 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ff18c036040 tx=0x7ff18c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c043020 con 0x7ff1941614d0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18c03e450 con 0x7ff1941614d0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19415c2d0 con 0x7ff1941614d0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c009510 con 0x7ff1941614d0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.501+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19415bfe0 con 0x7ff1941614d0 2024-03-06T07:30:14.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.502+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff194068ff0 con 0x7ff1941614d0 2024-03-06T07:30:14.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.502+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff18c046020 con 0x7ff1941614d0 2024-03-06T07:30:14.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.503+0000 7ff18affd640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff17806b4d0 0x7ff17806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.503+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff18c0acba0 con 0x7ff1941614d0 2024-03-06T07:30:14.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.503+0000 7ff19a33c640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff17806b4d0 0x7ff17806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.503+0000 7ff19a33c640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff17806b4d0 0x7ff17806d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff190000f60 tx=0x7ff190007c40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.507+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff18c04d030 con 0x7ff1941614d0 2024-03-06T07:30:14.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.928+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff1940d44a0 con 0x7ff1941614d0 2024-03-06T07:30:14.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.931+0000 7ff18affd640 1 -- 172.21.15.122:0/459906772 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff18c07b9a0 con 0x7ff1941614d0 2024-03-06T07:30:14.933 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-06T07:30:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBXGehlH78rMxAAqDGpa9jvBQOpiDoBNnSWVA== 2024-03-06T07:30:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: caps mds = "allow" 2024-03-06T07:30:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: caps mon = "allow r" 2024-03-06T07:30:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:14.933 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.0 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.933+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff17806b4d0 msgr2=0x7ff17806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.933+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff17806b4d0 0x7ff17806d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff190000f60 tx=0x7ff190007c40 comp rx=0 tx=0).stop 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.933+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 msgr2=0x7ff1940cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.933+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 0x7ff1940cb950 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ff18c036040 tx=0x7ff18c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 shutdown_connections 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff17806b4d0 0x7ff17806d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1941614d0 0x7ff1940cb950 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff1940d4860 0x7ff19415baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 --2- 172.21.15.122:0/459906772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1940b0e30 0x7ff1941552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 >> 172.21.15.122:0/459906772 conn(0x7ff1940d3d40 msgr2=0x7ff19406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 shutdown_connections 2024-03-06T07:30:14.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:14.934+0000 7ff19c5a0640 1 -- 172.21.15.122:0/459906772 wait complete. 2024-03-06T07:30:14.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:15.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.050+0000 7f3eb8a2a640 1 Processor -- start 2024-03-06T07:30:15.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.050+0000 7f3eb8a2a640 1 -- start start 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb8a2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3eb41515e0 0x7f3eb40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb8a2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb415b430 0x7f3eb4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb8a2a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb415b8b0 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb8a2a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb4070a70 con 0x7f3eb41515e0 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb8a2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4070d50 con 0x7f3eb415b430 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3eb41515e0 0x7f3eb40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb2d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3eb40cd7a0 0x7f3eb40cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37072/0 (socket says 172.21.15.122:37072) 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.051+0000 7f3eb2d9d640 1 -- 172.21.15.122:0/52292589 learned_addr learned my addr 172.21.15.122:0/52292589 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb259c640 1 -- 172.21.15.122:0/52292589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb415b430 msgr2=0x7f3eb4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb259c640 1 --2- 172.21.15.122:0/52292589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb415b430 0x7f3eb4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb259c640 1 -- 172.21.15.122:0/52292589 >> v1:172.21.15.122:6789/0 conn(0x7f3eb40cd7a0 legacy=0x7f3eb40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb159a640 1 -- 172.21.15.122:0/52292589 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137739847 0 0) 0x7f3eb415b8b0 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb259c640 1 -- 172.21.15.122:0/52292589 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4071030 con 0x7f3eb41515e0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb259c640 1 --2- 172.21.15.122:0/52292589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb41515e0 0x7f3eb40cce40 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097a0 tx=0x7f3ea402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79708ed7b3a39740 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.052+0000 7f3eb159a640 1 -- 172.21.15.122:0/52292589 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4035020 con 0x7f3eb41515e0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb159a640 1 -- 172.21.15.122:0/52292589 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4031830 con 0x7f3eb41515e0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb159a640 1 -- 172.21.15.122:0/52292589 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4031c00 con 0x7f3eb41515e0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/52292589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb41515e0 msgr2=0x7f3eb40cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/52292589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb41515e0 0x7f3eb40cce40 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097a0 tx=0x7f3ea402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/52292589 shutdown_connections 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/52292589 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb415b430 0x7f3eb4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/52292589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb41515e0 0x7f3eb40cce40 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/52292589 >> 172.21.15.122:0/52292589 conn(0x7f3eb40cf8a0 msgr2=0x7f3eb40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/52292589 shutdown_connections 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.053+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/52292589 wait complete. 2024-03-06T07:30:15.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.054+0000 7f3eb8a2a640 1 Processor -- start 2024-03-06T07:30:15.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 -- start start 2024-03-06T07:30:15.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 0x7f3eb40cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb41515e0 0x7f3eb415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3eb415b430 0x7f3eb4072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4071ef0 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4071d70 con 0x7f3eb415b430 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb8a2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b3f90 con 0x7f3eb41515e0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 0x7f3eb40cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 0x7f3eb40cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52176/0 (socket says 172.21.15.122:52176) 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb259c640 1 -- 172.21.15.122:0/1678754205 learned_addr learned my addr 172.21.15.122:0/1678754205 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.055+0000 7f3eb2d9d640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3eb415b430 0x7f3eb4072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb1d9b640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb41515e0 0x7f3eb415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb259c640 1 -- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb41515e0 msgr2=0x7f3eb415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb259c640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb41515e0 0x7f3eb415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb259c640 1 -- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3eb415b430 msgr2=0x7f3eb4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb259c640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3eb415b430 0x7f3eb4072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb259c640 1 -- 172.21.15.122:0/1678754205 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40cbc60 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb2d9d640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3eb415b430 0x7f3eb4072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb259c640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 0x7f3eb40cb720 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40098c0 tx=0x7f3ea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea403ba80 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea403bc20 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb40ca960 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40453f0 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.056+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb40caf20 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.057+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb4068840 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.057+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ea4038020 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.058+0000 7f3eaaffd640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e9406b4d0 0x7f3e9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.058+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ea40abfe0 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.058+0000 7f3eb1d9b640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e9406b4d0 0x7f3e9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.058+0000 7f3eb1d9b640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e9406b4d0 0x7f3e9406d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3eb41f19d0 tx=0x7f3e9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.062+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ea4040070 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.279+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb404f330 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.279+0000 7f3eaaffd640 1 -- 172.21.15.122:0/1678754205 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f3ea407aef0 con 0x7f3eb40cd7a0 2024-03-06T07:30:15.281 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:15.282 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":90,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3471691317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3471691317},{"type":"v1","addr":"172.21.15.169:6834","nonce":3471691317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2660610501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2660610501},{"type":"v1","addr":"172.21.15.169:6835","nonce":2660610501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:07.777233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9321},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:30:15.282 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 90 2024-03-06T07:30:15.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e9406b4d0 msgr2=0x7f3e9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e9406b4d0 0x7f3e9406d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3eb41f19d0 tx=0x7f3e9c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 msgr2=0x7f3eb40cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 0x7f3eb40cb720 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40098c0 tx=0x7f3ea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 shutdown_connections 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e9406b4d0 0x7f3e9406d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3eb415b430 0x7f3eb4072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3eb41515e0 0x7f3eb415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 --2- 172.21.15.122:0/1678754205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3eb40cd7a0 0x7f3eb40cb720 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 >> 172.21.15.122:0/1678754205 conn(0x7f3eb40cf8a0 msgr2=0x7f3eb415aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 shutdown_connections 2024-03-06T07:30:15.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.281+0000 7f3eb8a2a640 1 -- 172.21.15.122:0/1678754205 wait complete. 2024-03-06T07:30:15.696 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:30:07.777233+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9321}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:30:15.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:15.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:30:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.797+0000 7fbb7035e640 1 Processor -- start 2024-03-06T07:30:15.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.797+0000 7fbb7035e640 1 -- start start 2024-03-06T07:30:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb7035e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb680b4460 0x7fbb680d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb7035e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68162970 0x7fbb680b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb7035e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb680d22c0 con 0x7fbb68161cf0 2024-03-06T07:30:15.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb7035e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb68161b00 con 0x7fbb680b4460 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb7035e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680cecc0 con 0x7fbb68162970 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb680b4460 0x7fbb680d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb680b4460 0x7fbb680d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52188/0 (socket says 172.21.15.122:52188) 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e8fb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbb68161cf0 0x7fbb680cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37078/0 (socket says 172.21.15.122:37078) 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/3429373692 learned_addr learned my addr 172.21.15.122:0/3429373692 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/3429373692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68162970 0x7fbb680b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/3429373692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68162970 msgr2=0x7fbb680b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/3429373692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68162970 0x7fbb680b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/3429373692 >> v1:172.21.15.122:6789/0 conn(0x7fbb68161cf0 legacy=0x7fbb680cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/3429373692 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680d25a0 con 0x7fbb680b4460 2024-03-06T07:30:15.801 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.798+0000 7fbb6d0f8640 1 -- 172.21.15.122:0/3429373692 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356915349 0 0) 0x7fbb680d22c0 con 0x7fbb68161cf0 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.799+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/3429373692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb680b4460 0x7fbb680d4800 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009870 tx=0x7fbb6402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497016bb160994fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.799+0000 7fbb6d0f8640 1 -- 172.21.15.122:0/3429373692 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64034070 con 0x7fbb680b4460 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.799+0000 7fbb6d0f8640 1 -- 172.21.15.122:0/3429373692 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb64030840 con 0x7fbb680b4460 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.799+0000 7fbb6d0f8640 1 -- 172.21.15.122:0/3429373692 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64030c10 con 0x7fbb680b4460 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 -- 172.21.15.122:0/3429373692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb680b4460 msgr2=0x7fbb680d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 --2- 172.21.15.122:0/3429373692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb680b4460 0x7fbb680d4800 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009870 tx=0x7fbb6402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 -- 172.21.15.122:0/3429373692 shutdown_connections 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 --2- 172.21.15.122:0/3429373692 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68162970 0x7fbb680b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 --2- 172.21.15.122:0/3429373692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb680b4460 0x7fbb680d4800 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 -- 172.21.15.122:0/3429373692 >> 172.21.15.122:0/3429373692 conn(0x7fbb680d3ce0 msgr2=0x7fbb680d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.803 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 -- 172.21.15.122:0/3429373692 shutdown_connections 2024-03-06T07:30:15.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.800+0000 7fbb7035e640 1 -- 172.21.15.122:0/3429373692 wait complete. 2024-03-06T07:30:15.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.801+0000 7fbb7035e640 1 Processor -- start 2024-03-06T07:30:15.804 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.801+0000 7fbb7035e640 1 -- start start 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.801+0000 7fbb7035e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb680b4460 0x7fbb680cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.801+0000 7fbb7035e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68161cf0 0x7fbb681f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.801+0000 7fbb7035e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 0x7fbb6815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb7035e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680ce950 con 0x7fbb68162970 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb7035e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680d2b60 con 0x7fbb680b4460 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb7035e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680b3ac0 con 0x7fbb68161cf0 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 0x7fbb6815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 0x7fbb6815bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52202/0 (socket says 172.21.15.122:52202) 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/293670141 learned_addr learned my addr 172.21.15.122:0/293670141 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68161cf0 0x7fbb681f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb680b4460 0x7fbb680cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68161cf0 msgr2=0x7fbb681f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68161cf0 0x7fbb681f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb680b4460 msgr2=0x7fbb680cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb680b4460 0x7fbb680cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/293670141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb681f7e40 con 0x7fbb68162970 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb680b4460 0x7fbb680cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 0x7fbb6815bfd0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000c7a0 tx=0x7fbb6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb5effd640 1 -- 172.21.15.122:0/293670141 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60018070 con 0x7fbb68162970 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb5effd640 1 -- 172.21.15.122:0/293670141 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6000a640 con 0x7fbb68162970 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb681f8110 con 0x7fbb68162970 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb5effd640 1 -- 172.21.15.122:0/293670141 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6000aa70 con 0x7fbb68162970 2024-03-06T07:30:15.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.802+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb681f86d0 con 0x7fbb68162970 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.804+0000 7fbb5effd640 1 -- 172.21.15.122:0/293670141 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb6001b070 con 0x7fbb68162970 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.805+0000 7fbb5effd640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb4c06b4d0 0x7fbb4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.805+0000 7fbb5effd640 1 -- 172.21.15.122:0/293670141 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb60089700 con 0x7fbb68162970 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.805+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb4c06b4d0 0x7fbb4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.805+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb4c06b4d0 0x7fbb4c06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbb64036000 tx=0x7fbb6403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 Processor -- start 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 -- start start 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 0x560c6a5c2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5c68c0 0x560c6a5c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6a52c6b0 con 0x560c6a5c2870 2024-03-06T07:30:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6a4d5a60 con 0x560c6a5c68c0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.806+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c6a528750 con 0x560c6a5f2600 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 0x560c6a5c2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 0x560c6a5c2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52212/0 (socket says 172.21.15.122:52212) 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/884835272 learned_addr learned my addr 172.21.15.122:0/884835272 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5c68c0 0x560c6a5c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 msgr2=0x560c6a5f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5c68c0 msgr2=0x560c6a5c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5c68c0 0x560c6a5c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 -- 172.21.15.122:0/884835272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c6a4f3f30 con 0x560c6a5c2870 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5c68c0 0x560c6a5c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 0x560c6a5c2c50 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009960 tx=0x7fbb64030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977129632ee64826 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb4bfff640 1 -- 172.21.15.122:0/884835272 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64053070 con 0x560c6a5c2870 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb4bfff640 1 -- 172.21.15.122:0/884835272 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb64030d50 con 0x560c6a5c2870 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.807+0000 7fbb4bfff640 1 -- 172.21.15.122:0/884835272 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64034040 con 0x560c6a5c2870 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 -- 172.21.15.122:0/884835272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 msgr2=0x560c6a5c2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 0x560c6a5c2c50 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009960 tx=0x7fbb64030000 comp rx=0 tx=0).stop 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 -- 172.21.15.122:0/884835272 shutdown_connections 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5c68c0 0x560c6a5c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 --2- 172.21.15.122:0/884835272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5c2870 0x560c6a5c2c50 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 -- 172.21.15.122:0/884835272 >> 172.21.15.122:0/884835272 conn(0x560c6a5c48f0 msgr2=0x560c6a5c61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 -- 172.21.15.122:0/884835272 shutdown_connections 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.808+0000 7fbb73021740 1 -- 172.21.15.122:0/884835272 wait complete. 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.809+0000 7fbb73021740 1 -- start start 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.809+0000 7fbb73021740 1 Processor -- start 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.809+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5ec9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.809+0000 7fbb73021740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 0x560c6a60b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.809+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a60a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6a5c6d50 con 0x560c6a5ed620 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6a4e8ea0 con 0x560c6a608f40 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c6a5c7030 con 0x560c6a5f2600 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 10 client.0 _ll_register_callbacks cb 0x560c6a4e8870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 0x560c6a60b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 0x560c6a60b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52222/0 (socket says 172.21.15.122:52222) 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/4222426756 learned_addr learned my addr 172.21.15.122:0/4222426756 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/4222426756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 msgr2=0x560c6a5ec9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5ec9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/4222426756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 msgr2=0x560c6a60a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a60a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/4222426756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c6a4e9270 con 0x560c6a5ed620 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 0x560c6a60b6e0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fbb68162db0 tx=0x7fbb58011840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b096b5042ae69d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb497fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb497fa640 1 -- 172.21.15.122:0/4222426756 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5801a070 con 0x560c6a5ed620 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb497fa640 1 -- 172.21.15.122:0/4222426756 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb58004220 con 0x560c6a5ed620 2024-03-06T07:30:15.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb64030d50 con 0x560c6a5ed620 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb497fa640 1 -- 172.21.15.122:0/4222426756 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb58004420 con 0x560c6a5ed620 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 10 client.9480 fetch_fsmap learned FSMap version 90 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.810+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x560c6a2412d0 con 0x560c6a5ed620 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb497fa640 1 -- 172.21.15.122:0/4222426756 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb64030d50 con 0x560c6a5ed620 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb497fa640 1 -- 172.21.15.122:0/4222426756 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 90) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbb58027080 con 0x560c6a5ed620 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb73021740 10 client.9480 fetch_fsmap finished waiting for FSMap version 90 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb73021740 10 client.9480 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb73021740 20 client.9480 populate_metadata read hostname 'smithi122' 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb73021740 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a5f5b30 0x560c6a62c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb73021740 4 client.9480 mds_command: new command op to 9321 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 --> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c6a539bf0 con 0x560c6a5f5b30 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a5f5b30 0x560c6a62c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a5f5b30 0x560c6a62c880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51cc7ba44862458e server_cookie=e6a0fc110b894e9e in_seq=0 out_seq=0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.811+0000 7fbb497fa640 10 client.9480 ms_handle_connect on v2:172.21.15.122:6836/1416910179 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb497fa640 1 -- 172.21.15.122:0/4222426756 <== mds.0 v2:172.21.15.122:6836/1416910179 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c6a539bf0 con 0x560c6a5f5b30 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb497fa640 10 client.9480 handle_command_reply: tid=0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 client.9480 shutdown 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 msgr2=0x560c6a60b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 0x560c6a60b6e0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fbb68162db0 tx=0x7fbb58011840 comp rx=0 tx=0).stop 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 shutdown_connections 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a5f5b30 0x560c6a62c880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a60a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5ed620 0x560c6a60b6e0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb497fa640 0 client.9480 ms_handle_reset on v2:172.21.15.122:6836/1416910179 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 --2- 172.21.15.122:0/4222426756 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5f2600 0x560c6a5ec9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 >> 172.21.15.122:0/4222426756 conn(0x560c6a5c2870 msgr2=0x560c6a5f05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 shutdown_connections 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 1 -- 172.21.15.122:0/4222426756 wait complete. 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.812+0000 7fbb73021740 20 client.9480 trim_cache size 0 max 16384 2024-03-06T07:30:15.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 Processor -- start 2024-03-06T07:30:15.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 -- start start 2024-03-06T07:30:15.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a5f5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5f2600 0x560c6a607c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 0x560c6a608170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680b8b10 con 0x560c6a608f40 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680c7d80 con 0x560c6a5f2600 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.829+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680b8510 con 0x560c6a5ed620 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 0x560c6a608170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 0x560c6a608170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52236/0 (socket says 172.21.15.122:52236) 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/2888462323 learned_addr learned my addr 172.21.15.122:0/2888462323 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a5f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 msgr2=0x560c6a5f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a5f5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5f2600 0x560c6a607c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5f2600 msgr2=0x560c6a607c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5f2600 0x560c6a607c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 -- 172.21.15.122:0/2888462323 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c6a30b6b0 con 0x560c6a608f40 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 0x560c6a608170 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fbb60060060 tx=0x7fbb60057ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0eeed78a898b522 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb4b7fe640 1 -- 172.21.15.122:0/2888462323 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60060590 con 0x560c6a608f40 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb4b7fe640 1 -- 172.21.15.122:0/2888462323 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60056230 con 0x560c6a608f40 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.830+0000 7fbb4b7fe640 1 -- 172.21.15.122:0/2888462323 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6005e070 con 0x560c6a608f40 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 -- 172.21.15.122:0/2888462323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 msgr2=0x560c6a608170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 0x560c6a608170 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fbb60060060 tx=0x7fbb60057ff0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 -- 172.21.15.122:0/2888462323 shutdown_connections 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a608f40 0x560c6a608170 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a5f2600 0x560c6a607c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 --2- 172.21.15.122:0/2888462323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a5f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 -- 172.21.15.122:0/2888462323 >> 172.21.15.122:0/2888462323 conn(0x560c6a5c2870 msgr2=0x560c6a5bde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 -- 172.21.15.122:0/2888462323 shutdown_connections 2024-03-06T07:30:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.831+0000 7fbb73021740 1 -- 172.21.15.122:0/2888462323 wait complete. 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.832+0000 7fbb73021740 1 -- start start 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.832+0000 7fbb73021740 1 Processor -- start 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a677680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 0x560c6a6098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a609e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6a60bc20 con 0x560c6a5f2600 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6a608a70 con 0x560c6a608f40 2024-03-06T07:30:15.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c6a5ed2e0 con 0x560c6a5ed620 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb73021740 10 client.0 _ll_register_callbacks cb 0x7fbb5800acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb6d8f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 0x560c6a6098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb6d8f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 0x560c6a6098f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52246/0 (socket says 172.21.15.122:52246) 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/2568836793 learned_addr learned my addr 172.21.15.122:0/2568836793 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.833+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a677680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6e8fb640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a609e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 msgr2=0x560c6a677680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a677680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 msgr2=0x560c6a609e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a609e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6d8f9640 1 -- 172.21.15.122:0/2568836793 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c6a5f50c0 con 0x560c6a5f2600 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb6d8f9640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 0x560c6a6098f0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fbb58000c00 tx=0x7fbb580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35aa12118a68e7e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb497fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-03-06T07:30:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb497fa640 1 -- 172.21.15.122:0/2568836793 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5802a030 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb497fa640 1 -- 172.21.15.122:0/2568836793 <== mon.0 v2:172.21.15.122:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb58013730 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb60056230 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb497fa640 1 -- 172.21.15.122:0/2568836793 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb58033050 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 10 client.9486 fetch_fsmap learned FSMap version 90 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x560c6a5c4ce0 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb497fa640 1 -- 172.21.15.122:0/2568836793 <== mon.0 v2:172.21.15.122:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60056230 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb497fa640 1 -- 172.21.15.122:0/2568836793 <== mon.0 v2:172.21.15.122:3300/0 5 ==== fsmap(e 90) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbb58036020 con 0x560c6a5f2600 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 10 client.9486 fetch_fsmap finished waiting for FSMap version 90 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 10 client.9486 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 20 client.9486 populate_metadata read hostname 'smithi122' 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a681720 0x560c6a5ee190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 4 client.9486 mds_command: new command op to 9321 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.834+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 --> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560c6a533f40 con 0x560c6a681720 2024-03-06T07:30:15.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a681720 0x560c6a5ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb497fa640 10 client.9486 ms_handle_connect on v2:172.21.15.122:6836/1416910179 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb6e0fa640 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a681720 0x560c6a5ee190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36dc7f84b0fea705 server_cookie=7a2dfbb8d91c340c in_seq=0 out_seq=0 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb497fa640 1 -- 172.21.15.122:0/2568836793 <== mds.0 v2:172.21.15.122:6836/1416910179 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x560c6a533f40 con 0x560c6a681720 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb497fa640 10 client.9486 handle_command_reply: tid=0 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 client.9486 shutdown 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 msgr2=0x560c6a6098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 0x560c6a6098f0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fbb58000c00 tx=0x7fbb580046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 shutdown_connections 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:6836/1416910179,v1:172.21.15.122:6837/1416910179] conn(0x560c6a681720 0x560c6a5ee190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560c6a608f40 0x560c6a609e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560c6a5f2600 0x560c6a6098f0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 --2- 172.21.15.122:0/2568836793 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x560c6a5ed620 0x560c6a677680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb497fa640 0 client.9486 ms_handle_reset on v2:172.21.15.122:6836/1416910179 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 >> 172.21.15.122:0/2568836793 conn(0x560c6a5c2870 msgr2=0x560c6a5bde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 shutdown_connections 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.835+0000 7fbb73021740 1 -- 172.21.15.122:0/2568836793 wait complete. 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb73021740 20 client.9486 trim_cache size 0 max 16384 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stdout: "cluster_fsid": "12542aa0-5a71-4218-ae26-0a8da740ed3f", 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stdout: "whoami": 0, 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 9321, 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stdout: "want_state": "up:active", 2024-03-06T07:30:15.844 INFO:teuthology.orchestra.run.smithi122.stdout: "state": "up:active", 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs", 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: "rank_uptime": 9.4198370199999992, 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: "mdsmap_epoch": 90, 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch": 139, 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_epoch_barrier": 139, 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: "uptime": 13.727725144000001 2024-03-06T07:30:15.845 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb4c06b4d0 msgr2=0x7fbb4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb4c06b4d0 0x7fbb4c06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbb64036000 tx=0x7fbb6403b040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 msgr2=0x7fbb6815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 0x7fbb6815bfd0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000c7a0 tx=0x7fbb6000cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 shutdown_connections 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb4c06b4d0 0x7fbb4c06d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb68162970 0x7fbb6815bfd0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb68161cf0 0x7fbb681f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 --2- 172.21.15.122:0/293670141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb680b4460 0x7fbb680cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 >> 172.21.15.122:0/293670141 conn(0x7fbb680d3ce0 msgr2=0x7fbb680d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 shutdown_connections 2024-03-06T07:30:15.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.836+0000 7fbb7035e640 1 -- 172.21.15.122:0/293670141 wait complete. 2024-03-06T07:30:15.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:15.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.951+0000 7fe0cd33a640 1 Processor -- start 2024-03-06T07:30:15.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.951+0000 7fe0cd33a640 1 -- start start 2024-03-06T07:30:15.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0cd33a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe0c814bb80 0x7fe0c80b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0cd33a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe0c806dba0 0x7fe0c80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0cd33a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c80b3df0 con 0x7fe0c80cc330 2024-03-06T07:30:15.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0cd33a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c80b4a00 con 0x7fe0c806dba0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0cd33a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c80b40d0 con 0x7fe0c814bb80 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe0c806dba0 0x7fe0c80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe0c806dba0 0x7fe0c80cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52254/0 (socket says 172.21.15.122:52254) 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 -- 172.21.15.122:0/3058381988 learned_addr learned my addr 172.21.15.122:0/3058381988 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/3058381988 >> v2:172.21.15.169:3300/0 conn(0x7fe0c814bb80 0x7fe0c80b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 -- 172.21.15.122:0/3058381988 >> v2:172.21.15.169:3300/0 conn(0x7fe0c814bb80 msgr2=0x7fe0c80b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/3058381988 >> v2:172.21.15.169:3300/0 conn(0x7fe0c814bb80 0x7fe0c80b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 -- 172.21.15.122:0/3058381988 >> v1:172.21.15.169:6789/0 conn(0x7fe0c80cc330 legacy=0x7fe0c806d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.952+0000 7fe0c77fe640 1 -- 172.21.15.122:0/3058381988 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80b43b0 con 0x7fe0c806dba0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.953+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/3058381988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c80cd830 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8073220 tx=0x7fe0c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f09a9ae2ad6bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0c67fc640 1 -- 172.21.15.122:0/3058381988 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c002ca80 con 0x7fe0c806dba0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0c67fc640 1 -- 172.21.15.122:0/3058381988 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c002cc20 con 0x7fe0c806dba0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0c67fc640 1 -- 172.21.15.122:0/3058381988 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c00343f0 con 0x7fe0c806dba0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 -- 172.21.15.122:0/3058381988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 msgr2=0x7fe0c80cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/3058381988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c80cd830 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8073220 tx=0x7fe0c002bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 -- 172.21.15.122:0/3058381988 shutdown_connections 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/3058381988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c80cd830 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/3058381988 >> v2:172.21.15.169:3300/0 conn(0x7fe0c814bb80 0x7fe0c80b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 -- 172.21.15.122:0/3058381988 >> 172.21.15.122:0/3058381988 conn(0x7fe0c80cf8a0 msgr2=0x7fe0c80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 -- 172.21.15.122:0/3058381988 shutdown_connections 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.954+0000 7fe0cd33a640 1 -- 172.21.15.122:0/3058381988 wait complete. 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.956+0000 7fe0cd33a640 1 Processor -- start 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.956+0000 7fe0cd33a640 1 -- start start 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0cd33a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c81fc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0cd33a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cc330 0x7fe0c8144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0cd33a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c814bb80 0x7fe0c80c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0cd33a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80cd4c0 con 0x7fe0c806dba0 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0cd33a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80cd340 con 0x7fe0c80cc330 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0cd33a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80cdd70 con 0x7fe0c814bb80 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0c6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c81fc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.957+0000 7fe0c6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c81fc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52256/0 (socket says 172.21.15.122:52256) 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 -- 172.21.15.122:0/2419285731 learned_addr learned my addr 172.21.15.122:0/2419285731 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c814bb80 0x7fe0c80c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0bffff640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cc330 0x7fe0c8144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 -- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c814bb80 msgr2=0x7fe0c80c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c814bb80 0x7fe0c80c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 -- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cc330 msgr2=0x7fe0c8144a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cc330 0x7fe0c8144a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 -- 172.21.15.122:0/2419285731 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80c5dd0 con 0x7fe0c806dba0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c81fc730 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800c7f0 tx=0x7fe0b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8015070 con 0x7fe0c806dba0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b8004220 con 0x7fe0c806dba0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c80c60b0 con 0x7fe0c806dba0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8007620 con 0x7fe0c806dba0 2024-03-06T07:30:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.958+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c80c6670 con 0x7fe0c806dba0 2024-03-06T07:30:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.959+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c804f330 con 0x7fe0c806dba0 2024-03-06T07:30:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.960+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0b80079d0 con 0x7fe0c806dba0 2024-03-06T07:30:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.960+0000 7fe0bf7fe640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0a806b500 0x7fe0a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.960+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0b808a1d0 con 0x7fe0c806dba0 2024-03-06T07:30:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.960+0000 7fe0bffff640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0a806b500 0x7fe0a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.961+0000 7fe0bffff640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0a806b500 0x7fe0a806d9c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe0c81fcc70 tx=0x7fe0b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:15.964+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0b8059050 con 0x7fe0c806dba0 2024-03-06T07:30:16.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.181+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe0c80c7170 con 0x7fe0c806dba0 2024-03-06T07:30:16.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.181+0000 7fe0bf7fe640 1 -- 172.21.15.122:0/2419285731 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v90) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe0b8058950 con 0x7fe0c806dba0 2024-03-06T07:30:16.183 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:16.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0a806b500 msgr2=0x7fe0a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0a806b500 0x7fe0a806d9c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe0c81fcc70 tx=0x7fe0b0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 msgr2=0x7fe0c81fc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c81fc730 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800c7f0 tx=0x7fe0b800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 shutdown_connections 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0a806b500 0x7fe0a806d9c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c814bb80 0x7fe0c80c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cc330 0x7fe0c8144a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 --2- 172.21.15.122:0/2419285731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0c806dba0 0x7fe0c81fc730 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 >> 172.21.15.122:0/2419285731 conn(0x7fe0c80cf8a0 msgr2=0x7fe0c8161b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 shutdown_connections 2024-03-06T07:30:16.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:16.183+0000 7fe0cd33a640 1 -- 172.21.15.122:0/2419285731 wait complete. 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:30:16.196 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:16.197 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:16.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.197 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.212 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:30:16.213 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:30:16.213 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:16.214 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42225sec preferred_lft 42225sec 2024-03-06T07:30:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:30:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.215 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:16.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.217 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:16.277 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:30:16.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.279 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:30:16.334 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:30:16.334 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:16.334 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:16.334 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:16.334 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:16.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.335 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:30:16.335 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:30:16.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:16.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:16.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:16.335 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:30:16.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.431 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:30:16.446 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:16.446 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:16.446 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:16.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:30:16.513 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:30:16.513 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:16.513 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:16.569 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:16.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:30:16.690 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.749 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:30:16.801 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:30:16.801 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:30:16.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:30:16.871 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:16.926 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:30:16.926 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:30:16.926 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:30:16.926 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:30:16.927 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:16.928 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:16.929 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:30:16.929 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:16.929 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:16.929 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:16.929 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:16.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:16.930 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:30:16.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:30:16.934+0000 7fe2f720d480 -1 init, newargv = 0x55bbc71c7860 newargc=15 2024-03-06T07:30:16.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[83659]: starting ceph client 2024-03-06T07:30:16.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[83659]: starting fuse 2024-03-06T07:30:16.953 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:30:16.953 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:30:16.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:30:16.953 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:30:16.953 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:30:16.953 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:30:16.953 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:30:16.954 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:30:16.955 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:30:16.955 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:30:16.955 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:30:16.955 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:16.955 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:30:16.955 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:30:17.074 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.83659.asok 2024-03-06T07:30:17.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:17.084 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83659.asok 2024-03-06T07:30:17.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:17.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83659.asok status 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "83659", 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 9498, 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 9498 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 497265647 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 497265647 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.9498 192.168.0.1:0/497265647", 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/497265647", 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 90, 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 139, 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:17.229 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:30:17.230 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:17.245 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:30:17.245 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:17.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:17.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:17.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:30:17.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.455+0000 7f642b30c640 1 Processor -- start 2024-03-06T07:30:17.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.456+0000 7f642b30c640 1 -- start start 2024-03-06T07:30:17.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.456+0000 7f642b30c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f64240b0f70 0x7f64240b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.456+0000 7f642b30c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240b1790 con 0x7f642406d450 2024-03-06T07:30:17.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.456+0000 7f642b30c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240b14b0 con 0x7f642415e1b0 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.456+0000 7f642b30c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64240b2470 con 0x7f64240b0f70 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f64298a9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f642406d450 0x7f64240ae920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37094/0 (socket says 172.21.15.122:37094) 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f64298a9640 1 -- 172.21.15.122:0/3278374790 learned_addr learned my addr 172.21.15.122:0/3278374790 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f64288a7640 1 --2- 172.21.15.122:0/3278374790 >> v2:172.21.15.169:3300/0 conn(0x7f64240b0f70 0x7f64240b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533575246 0 0) 0x7f64240b1790 con 0x7f642406d450 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6410003520 con 0x7f642406d450 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1040586753 0 0) 0x7f6410003520 con 0x7f642406d450 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 >> v2:172.21.15.169:3300/0 conn(0x7f64240b0f70 msgr2=0x7f64240b64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 --2- 172.21.15.122:0/3278374790 >> v2:172.21.15.169:3300/0 conn(0x7f64240b0f70 0x7f64240b64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 >> v1:172.21.15.169:6789/0 conn(0x7f642415e1b0 legacy=0x7f642406cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64240b2750 con 0x7f642406d450 2024-03-06T07:30:17.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f641c0019f0 con 0x7f642406d450 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 >> v1:172.21.15.122:6789/0 conn(0x7f642406d450 legacy=0x7f64240ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64100039b0 0x7f6410003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6410004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64100051c0 0x7f641000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64240b2470 con 0x7f6410004800 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641000c160 con 0x7f64100051c0 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641000c360 con 0x7f64100039b0 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f64288a7640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6410004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f64298a9640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64100051c0 0x7f641000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.457+0000 7f64290a8640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64100039b0 0x7f6410003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64288a7640 1 -- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64100039b0 msgr2=0x7f6410003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64288a7640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64100039b0 0x7f6410003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64288a7640 1 -- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64100051c0 msgr2=0x7f641000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64288a7640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64100051c0 0x7f641000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64298a9640 1 -- 172.21.15.122:0/3278374790 reap_dead start 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64288a7640 1 -- 172.21.15.122:0/3278374790 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410003520 con 0x7f6410004800 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64290a8640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64100039b0 0x7f6410003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f64288a7640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6410004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f64240b6a10 tx=0x7f6410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4972f6d710011e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414039070 con 0x7f6410004800 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641402c660 con 0x7f6410004800 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.458+0000 7f641bfff640 1 -- 172.21.15.122:0/3278374790 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414009690 con 0x7f6410004800 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 -- 172.21.15.122:0/3278374790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 msgr2=0x7f6410004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6410004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f64240b6a10 tx=0x7f6410005830 comp rx=0 tx=0).stop 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 -- 172.21.15.122:0/3278374790 shutdown_connections 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 --2- 172.21.15.122:0/3278374790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6410004c80 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 -- 172.21.15.122:0/3278374790 >> 172.21.15.122:0/3278374790 conn(0x7f64240b6ea0 msgr2=0x7f64240b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 -- 172.21.15.122:0/3278374790 shutdown_connections 2024-03-06T07:30:17.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.459+0000 7f642b30c640 1 -- 172.21.15.122:0/3278374790 wait complete. 2024-03-06T07:30:17.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.461+0000 7f642b30c640 1 Processor -- start 2024-03-06T07:30:17.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.461+0000 7f642b30c640 1 -- start start 2024-03-06T07:30:17.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f642b30c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6424161090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f642b30c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6424152490 0x7f6424157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f642b30c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6424155d90 0x7f6424153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f642b30c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642406c270 con 0x7f6410004800 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f642b30c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642406c0f0 con 0x7f6424155d90 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f642b30c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64240b2f50 con 0x7f6424152490 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6424161090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6424161090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52270/0 (socket says 172.21.15.122:52270) 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 -- 172.21.15.122:0/825362235 learned_addr learned my addr 172.21.15.122:0/825362235 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64288a7640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6424152490 0x7f6424157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64298a9640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6424155d90 0x7f6424153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 -- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6424152490 msgr2=0x7f6424157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6424152490 0x7f6424157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 -- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6424155d90 msgr2=0x7f6424153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.462+0000 7f64290a8640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6424155d90 0x7f6424153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f64290a8640 1 -- 172.21.15.122:0/825362235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64241583c0 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f64288a7640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6424152490 0x7f6424157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f64298a9640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6424155d90 0x7f6424153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f64290a8640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6424161090 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f642000e480 tx=0x7f642000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64200052e0 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6420018070 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6424152c30 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420003810 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.463+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642415e910 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.464+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6424068ff0 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.464+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64200039b0 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.465+0000 7f6419ffb640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f640806b4d0 0x7f640806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.465+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6420089e30 con 0x7f6410004800 2024-03-06T07:30:17.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.465+0000 7f64288a7640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f640806b4d0 0x7f640806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.465+0000 7f64288a7640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f640806b4d0 0x7f640806d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6414036000 tx=0x7f6414030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.473+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6420058d40 con 0x7f6410004800 2024-03-06T07:30:17.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:17.768+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f642415d9b0 con 0x7f6410004800 2024-03-06T07:30:18.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.014+0000 7f6419ffb640 1 -- 172.21.15.122:0/825362235 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v91) v1 ==== 105+0+0 (secure 0 0 0) 0x7f6420058640 con 0x7f6410004800 2024-03-06T07:30:18.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f640806b4d0 msgr2=0x7f640806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f640806b4d0 0x7f640806d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6414036000 tx=0x7f6414030000 comp rx=0 tx=0).stop 2024-03-06T07:30:18.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 msgr2=0x7f6424161090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6424161090 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f642000e480 tx=0x7f642000e940 comp rx=0 tx=0).stop 2024-03-06T07:30:18.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 shutdown_connections 2024-03-06T07:30:18.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f640806b4d0 0x7f640806d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6424155d90 0x7f6424153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6424152490 0x7f6424157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 --2- 172.21.15.122:0/825362235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6410004800 0x7f6424161090 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 >> 172.21.15.122:0/825362235 conn(0x7f64240b6ea0 msgr2=0x7f64240b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.016+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 shutdown_connections 2024-03-06T07:30:18.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.017+0000 7f642b30c640 1 -- 172.21.15.122:0/825362235 wait complete. 2024-03-06T07:30:18.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:18.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.132+0000 7f14d7839640 1 Processor -- start 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.132+0000 7f14d7839640 1 -- start start 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d7839640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d7839640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f14d00cf830 0x7f14d00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d7839640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d00b3430 con 0x7f14d00d27f0 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d7839640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d00d2c50 con 0x7f14d00cf830 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d7839640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00b31e0 con 0x7f14d015dfe0 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f14d00cf830 0x7f14d00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f14d00cf830 0x7f14d00d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52278/0 (socket says 172.21.15.122:52278) 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 -- 172.21.15.122:0/243861037 learned_addr learned my addr 172.21.15.122:0/243861037 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 -- 172.21.15.122:0/243861037 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 msgr2=0x7f14d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d55d5640 1 --2- 172.21.15.122:0/243861037 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 --2- 172.21.15.122:0/243861037 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 -- 172.21.15.122:0/243861037 >> v1:172.21.15.169:6789/0 conn(0x7f14d00d27f0 legacy=0x7f14d00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.133+0000 7f14d4dd4640 1 -- 172.21.15.122:0/243861037 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00b3710 con 0x7f14d00cf830 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.134+0000 7f14d4dd4640 1 --2- 172.21.15.122:0/243861037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00cf830 0x7f14d00d22b0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009870 tx=0x7f14cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e96884d9f451aa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.134+0000 7f14c7fff640 1 -- 172.21.15.122:0/243861037 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc035020 con 0x7f14d00cf830 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.134+0000 7f14c7fff640 1 -- 172.21.15.122:0/243861037 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc031640 con 0x7f14d00cf830 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.134+0000 7f14c7fff640 1 -- 172.21.15.122:0/243861037 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc02d040 con 0x7f14d00cf830 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 -- 172.21.15.122:0/243861037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00cf830 msgr2=0x7f14d00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 --2- 172.21.15.122:0/243861037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00cf830 0x7f14d00d22b0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009870 tx=0x7f14cc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 -- 172.21.15.122:0/243861037 shutdown_connections 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 --2- 172.21.15.122:0/243861037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00cf830 0x7f14d00d22b0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 --2- 172.21.15.122:0/243861037 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 -- 172.21.15.122:0/243861037 >> 172.21.15.122:0/243861037 conn(0x7f14d00d3cd0 msgr2=0x7f14d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 -- 172.21.15.122:0/243861037 shutdown_connections 2024-03-06T07:30:18.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.135+0000 7f14d7839640 1 -- 172.21.15.122:0/243861037 wait complete. 2024-03-06T07:30:18.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.136+0000 7f14d7839640 1 Processor -- start 2024-03-06T07:30:18.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.136+0000 7f14d7839640 1 -- start start 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d7839640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14d00cf830 0x7f14d00cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d7839640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 0x7f14d0157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d7839640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d7839640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00d1c70 con 0x7f14d00d27f0 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d7839640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00b3d60 con 0x7f14d00cf830 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d7839640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00d1df0 con 0x7f14d015dfe0 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 0x7f14d0157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 0x7f14d0157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52284/0 (socket says 172.21.15.122:52284) 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 -- 172.21.15.122:0/4031141842 learned_addr learned my addr 172.21.15.122:0/4031141842 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d55d5640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14d00cf830 0x7f14d00cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d5dd6640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 -- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 msgr2=0x7f14d00b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00b22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 -- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14d00cf830 msgr2=0x7f14d00cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14d00cf830 0x7f14d00cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 -- 172.21.15.122:0/4031141842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00cc280 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d55d5640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14d00cf830 0x7f14d00cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.137+0000 7f14d4dd4640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 0x7f14d0157330 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02b930 tx=0x7f14cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.138+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc040070 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.138+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc039c10 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.138+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d00cc560 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.138+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc035040 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.138+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d00ccb20 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.139+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d004f330 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.139+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14cc038020 con 0x7f14d00d27f0 2024-03-06T07:30:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.140+0000 7f14c5ffb640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.140+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14cc0abdb0 con 0x7f14d00d27f0 2024-03-06T07:30:18.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.140+0000 7f14d55d5640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.140+0000 7f14d55d5640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f14c0006fd0 tx=0x7f14c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.148+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14cc07ac30 con 0x7f14d00d27f0 2024-03-06T07:30:18.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.363+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14d00747a0 con 0x7f14d00d27f0 2024-03-06T07:30:18.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.364+0000 7f14c5ffb640 1 -- 172.21.15.122:0/4031141842 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5006 (secure 0 0 0) 0x7f14cc07a530 con 0x7f14d00d27f0 2024-03-06T07:30:18.366 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:18.366 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":92,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3471691317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3471691317},{"type":"v1","addr":"172.21.15.169:6834","nonce":3471691317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2660610501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2660610501},{"type":"v1","addr":"172.21.15.169:6835","nonce":2660610501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:18.017462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:creating","state_seq":1,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:30:18.366 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 92 2024-03-06T07:30:18.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f14b406b4d0 msgr2=0x7f14b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f14c0006fd0 tx=0x7f14c0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 msgr2=0x7f14d0157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 0x7f14d0157330 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f14cc02b930 tx=0x7f14cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 shutdown_connections 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14d015dfe0 0x7f14d00b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14d00d27f0 0x7f14d0157330 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 --2- 172.21.15.122:0/4031141842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14d00cf830 0x7f14d00cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 >> 172.21.15.122:0/4031141842 conn(0x7f14d00d3cd0 msgr2=0x7f14d006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 shutdown_connections 2024-03-06T07:30:18.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:18.366+0000 7f14d7839640 1 -- 172.21.15.122:0/4031141842 wait complete. 2024-03-06T07:30:18.379 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:30:22.419 INFO:tasks.ceph.mds.a.smithi122.stderr:2024-03-06T07:30:22.417+0000 7fd232a43640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:30:23.380 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.481+0000 7f41e361c640 1 Processor -- start 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.481+0000 7f41e361c640 1 -- start start 2024-03-06T07:30:23.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e361c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f41dc15dfe0 0x7f41dc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e361c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f41dc0cf830 0x7f41dc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e361c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0d27f0 0x7f41dc0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e361c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc0b31e0 con 0x7f41dc15dfe0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e361c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc0b3d60 con 0x7f41dc0cf830 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e361c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0b3430 con 0x7f41dc0d27f0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f41dc15dfe0 0x7f41dc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f41dc15dfe0 0x7f41dc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52296/0 (socket says 172.21.15.122:52296) 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 -- 172.21.15.122:0/2976110730 learned_addr learned my addr 172.21.15.122:0/2976110730 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e1bb9640 1 --2- 172.21.15.122:0/2976110730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0d27f0 0x7f41dc0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e0bb7640 1 --2- 172.21.15.122:0/2976110730 >> v2:172.21.15.169:3300/0 conn(0x7f41dc0cf830 0x7f41dc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 -- 172.21.15.122:0/2976110730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0d27f0 msgr2=0x7f41dc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 --2- 172.21.15.122:0/2976110730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0d27f0 0x7f41dc0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 -- 172.21.15.122:0/2976110730 >> v2:172.21.15.169:3300/0 conn(0x7f41dc0cf830 msgr2=0x7f41dc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 --2- 172.21.15.122:0/2976110730 >> v2:172.21.15.169:3300/0 conn(0x7f41dc0cf830 0x7f41dc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.483+0000 7f41e13b8640 1 -- 172.21.15.122:0/2976110730 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc0b3710 con 0x7f41dc15dfe0 2024-03-06T07:30:23.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.484+0000 7f41e13b8640 1 --2- 172.21.15.122:0/2976110730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0cf2f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0d16b0 tx=0x7f41d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d63c3f3ac1df756 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.484+0000 7f41d3fff640 1 -- 172.21.15.122:0/2976110730 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8035020 con 0x7f41dc15dfe0 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.484+0000 7f41d3fff640 1 -- 172.21.15.122:0/2976110730 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d8031870 con 0x7f41dc15dfe0 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.484+0000 7f41d3fff640 1 -- 172.21.15.122:0/2976110730 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8031c40 con 0x7f41dc15dfe0 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 -- 172.21.15.122:0/2976110730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 msgr2=0x7f41dc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 --2- 172.21.15.122:0/2976110730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0cf2f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0d16b0 tx=0x7f41d802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 -- 172.21.15.122:0/2976110730 shutdown_connections 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 --2- 172.21.15.122:0/2976110730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0d27f0 0x7f41dc0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 --2- 172.21.15.122:0/2976110730 >> v2:172.21.15.169:3300/0 conn(0x7f41dc0cf830 0x7f41dc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 --2- 172.21.15.122:0/2976110730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0cf2f0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 -- 172.21.15.122:0/2976110730 >> 172.21.15.122:0/2976110730 conn(0x7f41dc0d3cd0 msgr2=0x7f41dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 -- 172.21.15.122:0/2976110730 shutdown_connections 2024-03-06T07:30:23.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.485+0000 7f41e361c640 1 -- 172.21.15.122:0/2976110730 wait complete. 2024-03-06T07:30:23.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.486+0000 7f41e361c640 1 Processor -- start 2024-03-06T07:30:23.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.486+0000 7f41e361c640 1 -- start start 2024-03-06T07:30:23.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0cf830 0x7f41dc0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41dc0d27f0 0x7f41dc0a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0d1df0 con 0x7f41dc15dfe0 2024-03-06T07:30:23.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0d1c70 con 0x7f41dc0d27f0 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc15e3c0 con 0x7f41dc0cf830 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0b1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52310/0 (socket says 172.21.15.122:52310) 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 -- 172.21.15.122:0/1493930736 learned_addr learned my addr 172.21.15.122:0/1493930736 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e0bb7640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41dc0d27f0 0x7f41dc0a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e13b8640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0cf830 0x7f41dc0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 -- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0cf830 msgr2=0x7f41dc0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0cf830 0x7f41dc0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 -- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41dc0d27f0 msgr2=0x7f41dc0a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41dc0d27f0 0x7f41dc0a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 -- 172.21.15.122:0/1493930736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc1f48f0 con 0x7f41dc15dfe0 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e0bb7640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41dc0d27f0 0x7f41dc0a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e13b8640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0cf830 0x7f41dc0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e1bb9640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0b1320 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f41d400cda0 tx=0x7f41d400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4003b90 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4003d30 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc1f4610 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.487+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4004240 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.488+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc0cc170 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.489+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc04f330 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.489+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41d40043e0 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.489+0000 7f41d1ffb640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f41c006b500 0x7f41c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.489+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41d408ac50 con 0x7f41dc15dfe0 2024-03-06T07:30:23.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.489+0000 7f41e13b8640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f41c006b500 0x7f41c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.490+0000 7f41e13b8640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f41c006b500 0x7f41c006d9c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f41d8036000 tx=0x7f41d8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.498+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41d4090020 con 0x7f41dc15dfe0 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.739+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41dc15d570 con 0x7f41dc15dfe0 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.739+0000 7f41d1ffb640 1 -- 172.21.15.122:0/1493930736 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5009 (secure 0 0 0) 0x7f41d4059ad0 con 0x7f41dc15dfe0 2024-03-06T07:30:23.742 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:23.742 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":93,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3471691317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3471691317},{"type":"v1","addr":"172.21.15.169:6834","nonce":3471691317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2660610501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2660610501},{"type":"v1","addr":"172.21.15.169:6835","nonce":2660610501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:19.020885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9318,9321]},"id":10}]} 2024-03-06T07:30:23.742 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 93 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f41c006b500 msgr2=0x7f41c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f41c006b500 0x7f41c006d9c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f41d8036000 tx=0x7f41d8008030 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 msgr2=0x7f41dc0b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0b1320 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f41d400cda0 tx=0x7f41d400a660 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 shutdown_connections 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f41c006b500 0x7f41c006d9c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41dc15dfe0 0x7f41dc0b1320 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41dc0d27f0 0x7f41dc0a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 --2- 172.21.15.122:0/1493930736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f41dc0cf830 0x7f41dc0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 >> 172.21.15.122:0/1493930736 conn(0x7f41dc0d3cd0 msgr2=0x7f41dc0b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 shutdown_connections 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.742+0000 7f41e361c640 1 -- 172.21.15.122:0/1493930736 wait complete. 2024-03-06T07:30:23.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.857+0000 7f93a8b1e640 1 Processor -- start 2024-03-06T07:30:23.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.857+0000 7f93a8b1e640 1 -- start start 2024-03-06T07:30:23.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a8b1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a8b1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f93a40b41f0 0x7f93a40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a8b1e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a40b3430 con 0x7f93a415dfe0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a8b1e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a40d2c50 con 0x7f93a40b41f0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a8b1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40b31e0 con 0x7f93a40d22b0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f93a40b41f0 0x7f93a40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a259c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f93a415dfe0 0x7f93a40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37110/0 (socket says 172.21.15.122:37110) 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a259c640 1 -- 172.21.15.122:0/1932966733 learned_addr learned my addr 172.21.15.122:0/1932966733 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a1d9b640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a2d9d640 1 -- 172.21.15.122:0/1932966733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 msgr2=0x7f93a40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a2d9d640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a2d9d640 1 -- 172.21.15.122:0/1932966733 >> v1:172.21.15.122:6789/0 conn(0x7f93a415dfe0 legacy=0x7f93a40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a2d9d640 1 -- 172.21.15.122:0/1932966733 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a40b3710 con 0x7f93a40b41f0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a159a640 1 -- 172.21.15.122:0/1932966733 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735512465 0 0) 0x7f93a40b3430 con 0x7f93a415dfe0 2024-03-06T07:30:23.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.858+0000 7f93a1d9b640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.859+0000 7f93a2d9d640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a40b4650 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f939c009d90 tx=0x7f939c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85306da4481cf191 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a159a640 1 -- 172.21.15.122:0/1932966733 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c039070 con 0x7f93a40b41f0 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a159a640 1 -- 172.21.15.122:0/1932966733 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c02cc50 con 0x7f93a40b41f0 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a159a640 1 -- 172.21.15.122:0/1932966733 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c034440 con 0x7f93a40b41f0 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 -- 172.21.15.122:0/1932966733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 msgr2=0x7f93a40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a40b4650 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f939c009d90 tx=0x7f939c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 -- 172.21.15.122:0/1932966733 shutdown_connections 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a40b4650 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/1932966733 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 -- 172.21.15.122:0/1932966733 >> 172.21.15.122:0/1932966733 conn(0x7f93a40d3cd0 msgr2=0x7f93a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 -- 172.21.15.122:0/1932966733 shutdown_connections 2024-03-06T07:30:23.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.860+0000 7f93a8b1e640 1 -- 172.21.15.122:0/1932966733 wait complete. 2024-03-06T07:30:23.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 Processor -- start 2024-03-06T07:30:23.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 -- start start 2024-03-06T07:30:23.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a414f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40b6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a415dfe0 0x7f93a40cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40cfa00 con 0x7f93a40b41f0 2024-03-06T07:30:23.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40b3d60 con 0x7f93a415dfe0 2024-03-06T07:30:23.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a8b1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40d1c70 con 0x7f93a40d22b0 2024-03-06T07:30:23.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a414f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a414f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52320/0 (socket says 172.21.15.122:52320) 2024-03-06T07:30:23.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a259c640 1 -- 172.21.15.122:0/2195509424 learned_addr learned my addr 172.21.15.122:0/2195509424 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:23.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.862+0000 7f93a1d9b640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40b6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a2d9d640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a415dfe0 0x7f93a40cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a259c640 1 -- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 msgr2=0x7f93a40b6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a259c640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40b6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a259c640 1 -- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a415dfe0 msgr2=0x7f93a40cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a259c640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a415dfe0 0x7f93a40cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a259c640 1 -- 172.21.15.122:0/2195509424 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a41f7380 con 0x7f93a40b41f0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a2d9d640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a415dfe0 0x7f93a40cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a259c640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a414f090 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f939400c460 tx=0x7f939400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394018070 con 0x7f93a40b41f0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394007500 con 0x7f93a40b41f0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a41f7660 con 0x7f93a40b41f0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394004a20 con 0x7f93a40b41f0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.863+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a41f7c20 con 0x7f93a40b41f0 2024-03-06T07:30:23.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.864+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a404f330 con 0x7f93a40b41f0 2024-03-06T07:30:23.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.865+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f939401b070 con 0x7f93a40b41f0 2024-03-06T07:30:23.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.865+0000 7f9392ffd640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.865+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9394089320 con 0x7f93a40b41f0 2024-03-06T07:30:23.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.865+0000 7f93a1d9b640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.866+0000 7f93a1d9b640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f938406b4d0 0x7f938406d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f93a415cc50 tx=0x7f9398008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:23.869+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9394057be0 con 0x7f93a40b41f0 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.087+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a406bc10 con 0x7f93a40b41f0 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.087+0000 7f9392ffd640 1 -- 172.21.15.122:0/2195509424 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5009 (secure 0 0 0) 0x7f939405bc80 con 0x7f93a40b41f0 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:24.090 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":93,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6834/3471691317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3471691317},{"type":"v1","addr":"172.21.15.169:6834","nonce":3471691317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":9309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/2660610501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6833","nonce":2660610501},{"type":"v1","addr":"172.21.15.169:6835","nonce":2660610501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:19.020885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9318,9321]},"id":10}]} 2024-03-06T07:30:24.090 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 93 2024-03-06T07:30:24.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f938406b4d0 msgr2=0x7f938406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f938406b4d0 0x7f938406d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f93a415cc50 tx=0x7f9398008040 comp rx=0 tx=0).stop 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 msgr2=0x7f93a414f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a414f090 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f939400c460 tx=0x7f939400c920 comp rx=0 tx=0).stop 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 shutdown_connections 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a415dfe0 0x7f93a40cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f93a40d22b0 0x7f93a40b6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 --2- 172.21.15.122:0/2195509424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93a40b41f0 0x7f93a414f090 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 >> 172.21.15.122:0/2195509424 conn(0x7f93a40d3cd0 msgr2=0x7f93a40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 shutdown_connections 2024-03-06T07:30:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.089+0000 7f93a8b1e640 1 -- 172.21.15.122:0/2195509424 wait complete. 2024-03-06T07:30:24.102 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-03-06T07:30:24.103 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:24.103 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:24.114 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:24.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:30:24.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.214+0000 7ff0afb9a640 1 Processor -- start 2024-03-06T07:30:24.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.214+0000 7ff0afb9a640 1 -- start start 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0afb9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a80d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0afb9a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 0x7ff0a80d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0afb9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a80d25e0 con 0x7ff0a80c80e0 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0afb9a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a80c7c50 con 0x7ff0a80d4810 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0afb9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80d2880 con 0x7ff0a815e000 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0ad936640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a80d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0ad135640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 0x7ff0a80d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0ad936640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a80d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:33626/0 (socket says 172.21.15.122:33626) 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0ad936640 1 -- 172.21.15.122:0/171565417 learned_addr learned my addr 172.21.15.122:0/171565417 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.215+0000 7ff0ad135640 1 --2- 172.21.15.122:0/171565417 >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 0x7ff0a80d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:59858/0 (socket says 172.21.15.122:59858) 2024-03-06T07:30:24.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ac934640 1 -- 172.21.15.122:0/171565417 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110424523 0 0) 0x7ff0a80d25e0 con 0x7ff0a80c80e0 2024-03-06T07:30:24.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ac934640 1 -- 172.21.15.122:0/171565417 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff094003520 con 0x7ff0a80c80e0 2024-03-06T07:30:24.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ad936640 1 -- 172.21.15.122:0/171565417 >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 msgr2=0x7ff0a80d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ad936640 1 --2- 172.21.15.122:0/171565417 >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 0x7ff0a80d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ad936640 1 -- 172.21.15.122:0/171565417 >> v1:172.21.15.169:6789/0 conn(0x7ff0a80c80e0 legacy=0x7ff0a80c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ad936640 1 -- 172.21.15.122:0/171565417 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80d2b40 con 0x7ff0a815e000 2024-03-06T07:30:24.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.216+0000 7ff0ad135640 1 --2- 172.21.15.122:0/171565417 >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 0x7ff0a80d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.217+0000 7ff0ad936640 1 --2- 172.21.15.122:0/171565417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a80d0680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009870 tx=0x7ff0a002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3224cd0c539161d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.217+0000 7ff0ac934640 1 -- 172.21.15.122:0/171565417 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a002f840 con 0x7ff0a815e000 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.217+0000 7ff0ac934640 1 -- 172.21.15.122:0/171565417 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a002f9e0 con 0x7ff0a815e000 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.217+0000 7ff0ac934640 1 -- 172.21.15.122:0/171565417 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a002fd90 con 0x7ff0a815e000 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 -- 172.21.15.122:0/171565417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 msgr2=0x7ff0a80d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/171565417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a80d0680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009870 tx=0x7ff0a002ba20 comp rx=0 tx=0).stop 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 -- 172.21.15.122:0/171565417 shutdown_connections 2024-03-06T07:30:24.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/171565417 >> v2:172.21.15.169:3300/0 conn(0x7ff0a80d4810 0x7ff0a80d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/171565417 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a80d0680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 -- 172.21.15.122:0/171565417 >> 172.21.15.122:0/171565417 conn(0x7ff0a80d3cf0 msgr2=0x7ff0a80d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 -- 172.21.15.122:0/171565417 shutdown_connections 2024-03-06T07:30:24.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.218+0000 7ff0afb9a640 1 -- 172.21.15.122:0/171565417 wait complete. 2024-03-06T07:30:24.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.220+0000 7ff0afb9a640 1 Processor -- start 2024-03-06T07:30:24.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.220+0000 7ff0afb9a640 1 -- start start 2024-03-06T07:30:24.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0afb9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0a80c80e0 0x7ff0a815b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0afb9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 0x7ff0a80b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0afb9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a8160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0afb9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a815e3e0 con 0x7ff0a80d4810 2024-03-06T07:30:24.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0afb9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80d22d0 con 0x7ff0a80c80e0 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0afb9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80cf7d0 con 0x7ff0a815e000 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 0x7ff0a80b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 0x7ff0a80b8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52332/0 (socket says 172.21.15.122:52332) 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 -- 172.21.15.122:0/836149245 learned_addr learned my addr 172.21.15.122:0/836149245 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad936640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0a80c80e0 0x7ff0a815b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ae137640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a8160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 -- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 msgr2=0x7ff0a8160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a8160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 -- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0a80c80e0 msgr2=0x7ff0a815b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0a80c80e0 0x7ff0a815b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.221+0000 7ff0ad135640 1 -- 172.21.15.122:0/836149245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80b7b00 con 0x7ff0a80d4810 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.222+0000 7ff0ad135640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 0x7ff0a80b8ba0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff09800ebe0 tx=0x7ff09800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.222+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098017070 con 0x7ff0a80d4810 2024-03-06T07:30:24.225 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.222+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09800cc30 con 0x7ff0a80d4810 2024-03-06T07:30:24.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.222+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a80b7de0 con 0x7ff0a80d4810 2024-03-06T07:30:24.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.222+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098013470 con 0x7ff0a80d4810 2024-03-06T07:30:24.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.222+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a80b83a0 con 0x7ff0a80d4810 2024-03-06T07:30:24.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.223+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a804f330 con 0x7ff0a80d4810 2024-03-06T07:30:24.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.224+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff09801a070 con 0x7ff0a80d4810 2024-03-06T07:30:24.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.224+0000 7ff09e7fc640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.224+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff09808a470 con 0x7ff0a80d4810 2024-03-06T07:30:24.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.224+0000 7ff0ad936640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.224+0000 7ff0ad936640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff08c06b4d0 0x7ff08c06d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0036000 tx=0x7ff0a0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.232+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff098059380 con 0x7ff0a80d4810 2024-03-06T07:30:24.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:24.484+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7ff0a80d0730 con 0x7ff0a80d4810 2024-03-06T07:30:25.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.046+0000 7ff09e7fc640 1 -- 172.21.15.122:0/836149245 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 9309 v94) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff098058c80 con 0x7ff0a80d4810 2024-03-06T07:30:25.049 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 9309 2024-03-06T07:30:25.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff08c06b4d0 msgr2=0x7ff08c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff08c06b4d0 0x7ff08c06d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0036000 tx=0x7ff0a0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 msgr2=0x7ff0a80b8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 0x7ff0a80b8ba0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff09800ebe0 tx=0x7ff09800c520 comp rx=0 tx=0).stop 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 shutdown_connections 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff0a815e000 0x7ff0a8160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0a80d4810 0x7ff0a80b8ba0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 --2- 172.21.15.122:0/836149245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0a80c80e0 0x7ff0a815b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 >> 172.21.15.122:0/836149245 conn(0x7ff0a80d3cf0 msgr2=0x7ff0a806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 shutdown_connections 2024-03-06T07:30:25.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.049+0000 7ff0afb9a640 1 -- 172.21.15.122:0/836149245 wait complete. 2024-03-06T07:30:25.062 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:25.062 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:25.072 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:25.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:30:25.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.173+0000 7f9af5381640 1 Processor -- start 2024-03-06T07:30:25.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.173+0000 7f9af5381640 1 -- start start 2024-03-06T07:30:25.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9af5381640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9af006d3c0 0x7f9af00b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9af5381640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00af4b0 con 0x7f9af01614a0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9af5381640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00af790 con 0x7f9af00b1850 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9af5381640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af00afa70 con 0x7f9af006d3c0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aeeffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9af01614a0 0x7f9af00b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52926/0 (socket says 172.21.15.122:52926) 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aeeffd640 1 -- 172.21.15.122:0/4216156493 learned_addr learned my addr 172.21.15.122:0/4216156493 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578994117 0 0) 0x7f9af00af4b0 con 0x7f9af01614a0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad8003520 con 0x7f9af01614a0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aee7fc640 1 --2- 172.21.15.122:0/4216156493 >> v2:172.21.15.169:3300/0 conn(0x7f9af006d3c0 0x7f9af00b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 347383653 0 0) 0x7f9ad8003520 con 0x7f9af01614a0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.174+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 >> v2:172.21.15.169:3300/0 conn(0x7f9af006d3c0 msgr2=0x7f9af00b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 --2- 172.21.15.122:0/4216156493 >> v2:172.21.15.169:3300/0 conn(0x7f9af006d3c0 0x7f9af00b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 >> v1:172.21.15.169:6789/0 conn(0x7f9af00b1850 legacy=0x7f9af00b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00afd50 con 0x7f9af01614a0 2024-03-06T07:30:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f9ae4001670 con 0x7f9af01614a0 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 >> v1:172.21.15.122:6789/0 conn(0x7f9af01614a0 legacy=0x7f9af00b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad8004800 0x7f9ad8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ad80051c0 0x7f9ad800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00afa70 con 0x7f9ad80039b0 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800c060 con 0x7f9ad80051c0 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800c260 con 0x7f9ad8004800 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aee7fc640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad8004800 0x7f9ad8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aef7fe640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ad80051c0 0x7f9ad800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aeeffd640 1 -- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad8004800 msgr2=0x7f9ad8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad8004800 0x7f9ad8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aeeffd640 1 -- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ad80051c0 msgr2=0x7f9ad800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ad80051c0 0x7f9ad800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aef7fe640 1 -- 172.21.15.122:0/4216156493 reap_dead start 2024-03-06T07:30:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.175+0000 7f9aeeffd640 1 -- 172.21.15.122:0/4216156493 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8003520 con 0x7f9ad80039b0 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.176+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c210 tx=0x7f9ad8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615c3f71d923176b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.176+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4007d30 con 0x7f9ad80039b0 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.176+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae4030070 con 0x7f9ad80039b0 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.176+0000 7f9aedffb640 1 -- 172.21.15.122:0/4216156493 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae402cda0 con 0x7f9ad80039b0 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 -- 172.21.15.122:0/4216156493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ad80039b0 msgr2=0x7f9ad8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c210 tx=0x7f9ad8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 -- 172.21.15.122:0/4216156493 shutdown_connections 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 --2- 172.21.15.122:0/4216156493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 -- 172.21.15.122:0/4216156493 >> 172.21.15.122:0/4216156493 conn(0x7f9af009a9f0 msgr2=0x7f9af009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 -- 172.21.15.122:0/4216156493 shutdown_connections 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.177+0000 7f9af5381640 1 -- 172.21.15.122:0/4216156493 wait complete. 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.178+0000 7f9af5381640 1 Processor -- start 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.178+0000 7f9af5381640 1 -- start start 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9af5381640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad80039b0 0x7f9af01f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9af5381640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 0x7f9af00b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9af5381640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9af01477f0 0x7f9af00b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9af5381640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b1db0 con 0x7f9af0147370 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9af5381640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b1c30 con 0x7f9af01477f0 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9af5381640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b1f30 con 0x7f9ad80039b0 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 0x7f9af00b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 0x7f9af00b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48570/0 (socket says 172.21.15.122:48570) 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 -- 172.21.15.122:0/2112837266 learned_addr learned my addr 172.21.15.122:0/2112837266 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad80039b0 0x7f9af01f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aef7fe640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9af01477f0 0x7f9af00b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 -- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad80039b0 msgr2=0x7f9af01f77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad80039b0 0x7f9af01f77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 -- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9af01477f0 msgr2=0x7f9af00b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9af01477f0 0x7f9af00b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.179+0000 7f9aee7fc640 1 -- 172.21.15.122:0/2112837266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af009e560 con 0x7f9af0147370 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.180+0000 7f9aee7fc640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 0x7f9af00b20b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0011a70 tx=0x7f9ae0011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.180+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae000fb50 con 0x7f9af0147370 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.180+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae000fcf0 con 0x7f9af0147370 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.180+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae000fea0 con 0x7f9af0147370 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.180+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af01493d0 con 0x7f9af0147370 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.180+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af01f4020 con 0x7f9af0147370 2024-03-06T07:30:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.181+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af004f330 con 0x7f9af0147370 2024-03-06T07:30:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.181+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ae0012040 con 0x7f9af0147370 2024-03-06T07:30:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.182+0000 7f9ad77fe640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.182+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ae008a660 con 0x7f9af0147370 2024-03-06T07:30:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.182+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.182+0000 7f9aeeffd640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9adc06b4d0 0x7f9adc06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4034000 tx=0x7f9ae4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.186+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ae0019030 con 0x7f9af0147370 2024-03-06T07:30:25.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:25.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:25.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:25.456+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f9af006e8e0 con 0x7f9af0147370 2024-03-06T07:30:26.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.047+0000 7f9ad77fe640 1 -- 172.21.15.122:0/2112837266 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 8107 v95) v1 ==== 95+0+0 (secure 0 0 0) 0x7f9ae0058f20 con 0x7f9af0147370 2024-03-06T07:30:26.049 INFO:teuthology.orchestra.run.smithi122.stderr:failed mds gid 8107 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.049+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9adc06b4d0 msgr2=0x7f9adc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.049+0000 7f9af5381640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9adc06b4d0 0x7f9adc06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4034000 tx=0x7f9ae4009030 comp rx=0 tx=0).stop 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.049+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 msgr2=0x7f9af00b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.049+0000 7f9af5381640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 0x7f9af00b20b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0011a70 tx=0x7f9ae0011f30 comp rx=0 tx=0).stop 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 shutdown_connections 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9af01477f0 0x7f9af00b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0147370 0x7f9af00b20b0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 --2- 172.21.15.122:0/2112837266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ad80039b0 0x7f9af01f77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 >> 172.21.15.122:0/2112837266 conn(0x7f9af009a9f0 msgr2=0x7f9af00b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 shutdown_connections 2024-03-06T07:30:26.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.050+0000 7f9af5381640 1 -- 172.21.15.122:0/2112837266 wait complete. 2024-03-06T07:30:26.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:26.066 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:30:26.066 DEBUG:teuthology.orchestra.run.smithi122:> mkdir dir_1 dir_2 2024-03-06T07:30:26.066 DEBUG:teuthology.orchestra.run.smithi122:> touch dir_1/original 2024-03-06T07:30:26.066 DEBUG:teuthology.orchestra.run.smithi122:> ln dir_1/original dir_2/linkto 2024-03-06T07:30:26.066 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:30:26.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:26.100 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 dir_1) 2024-03-06T07:30:26.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.298+0000 7fbc2b037640 1 Processor -- start 2024-03-06T07:30:26.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.298+0000 7fbc2b037640 1 -- start start 2024-03-06T07:30:26.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.298+0000 7fbc2b037640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbc240d2710 0x7fbc240b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.298+0000 7fbc2b037640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc240b3430 con 0x7fbc2415dfe0 2024-03-06T07:30:26.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.298+0000 7fbc2b037640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc240b3710 con 0x7fbc240d22b0 2024-03-06T07:30:26.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.298+0000 7fbc2b037640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc240b31e0 con 0x7fbc240d2710 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc28dd3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbc2415dfe0 0x7fbc240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52940/0 (socket says 172.21.15.122:52940) 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc28dd3640 1 -- 172.21.15.122:0/2195318748 learned_addr learned my addr 172.21.15.122:0/2195318748 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc295d4640 1 --2- 172.21.15.122:0/2195318748 >> v2:172.21.15.169:3300/0 conn(0x7fbc240d2710 0x7fbc240b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc1b7fe640 1 -- 172.21.15.122:0/2195318748 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327321337 0 0) 0x7fbc240b3430 con 0x7fbc2415dfe0 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc1b7fe640 1 -- 172.21.15.122:0/2195318748 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c003520 con 0x7fbc2415dfe0 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc295d4640 1 -- 172.21.15.122:0/2195318748 >> v1:172.21.15.169:6789/0 conn(0x7fbc240d22b0 legacy=0x7fbc240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc1b7fe640 1 -- 172.21.15.122:0/2195318748 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848530259 0 0) 0x7fbc240b3710 con 0x7fbc240d22b0 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc295d4640 1 -- 172.21.15.122:0/2195318748 >> v1:172.21.15.122:6789/0 conn(0x7fbc2415dfe0 legacy=0x7fbc240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.299+0000 7fbc295d4640 1 -- 172.21.15.122:0/2195318748 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc240b39f0 con 0x7fbc240d2710 2024-03-06T07:30:26.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.300+0000 7fbc295d4640 1 --2- 172.21.15.122:0/2195318748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc240b4900 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009dc0 tx=0x7fbc1c02bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23ceb3e71fcfddcd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.300+0000 7fbc1b7fe640 1 -- 172.21.15.122:0/2195318748 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c039070 con 0x7fbc240d2710 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.300+0000 7fbc1b7fe640 1 -- 172.21.15.122:0/2195318748 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c02c890 con 0x7fbc240d2710 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.300+0000 7fbc1b7fe640 1 -- 172.21.15.122:0/2195318748 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20001670 con 0x7fbc240d2710 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.300+0000 7fbc2b037640 1 -- 172.21.15.122:0/2195318748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 msgr2=0x7fbc240b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.300+0000 7fbc2b037640 1 --2- 172.21.15.122:0/2195318748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc240b4900 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009dc0 tx=0x7fbc1c02bd60 comp rx=0 tx=0).stop 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.301+0000 7fbc2b037640 1 -- 172.21.15.122:0/2195318748 shutdown_connections 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.301+0000 7fbc2b037640 1 --2- 172.21.15.122:0/2195318748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc240b4900 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.301+0000 7fbc2b037640 1 -- 172.21.15.122:0/2195318748 >> 172.21.15.122:0/2195318748 conn(0x7fbc240d3cd0 msgr2=0x7fbc240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.301+0000 7fbc2b037640 1 -- 172.21.15.122:0/2195318748 shutdown_connections 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.301+0000 7fbc2b037640 1 -- 172.21.15.122:0/2195318748 wait complete. 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.302+0000 7fbc2b037640 1 Processor -- start 2024-03-06T07:30:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.302+0000 7fbc2b037640 1 -- start start 2024-03-06T07:30:26.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc2b037640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 0x7fbc24150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc2b037640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc241f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc2b037640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc2415dfe0 0x7fbc241f6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc2b037640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc240cfa00 con 0x7fbc240d22b0 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc2b037640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc240b3d60 con 0x7fbc240d2710 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc2b037640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc240d1c70 con 0x7fbc2415dfe0 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc28dd3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 0x7fbc24150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc28dd3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 0x7fbc24150d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48582/0 (socket says 172.21.15.122:48582) 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc28dd3640 1 -- 172.21.15.122:0/842084844 learned_addr learned my addr 172.21.15.122:0/842084844 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc295d4640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc2415dfe0 0x7fbc241f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.303+0000 7fbc1bfff640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc241f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc28dd3640 1 -- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc2415dfe0 msgr2=0x7fbc241f6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc28dd3640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc2415dfe0 0x7fbc241f6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc28dd3640 1 -- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 msgr2=0x7fbc241f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc28dd3640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc241f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc28dd3640 1 -- 172.21.15.122:0/842084844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2414ce00 con 0x7fbc240d22b0 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc295d4640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc2415dfe0 0x7fbc241f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc28dd3640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 0x7fbc24150d00 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c780 tx=0x7fbc2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20018070 con 0x7fbc240d22b0 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2414d0e0 con 0x7fbc240d22b0 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20007500 con 0x7fbc240d22b0 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20004a20 con 0x7fbc240d22b0 2024-03-06T07:30:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.304+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2414d6a0 con 0x7fbc240d22b0 2024-03-06T07:30:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.305+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2404f330 con 0x7fbc240d22b0 2024-03-06T07:30:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.306+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc20004bc0 con 0x7fbc240d22b0 2024-03-06T07:30:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.307+0000 7fbc197fa640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1006b4d0 0x7fbc1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.307+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc20089790 con 0x7fbc240d22b0 2024-03-06T07:30:26.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.307+0000 7fbc1bfff640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1006b4d0 0x7fbc1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.308+0000 7fbc1bfff640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1006b4d0 0x7fbc1006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbc240b2340 tx=0x7fbc14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.311+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc20057f30 con 0x7fbc240d22b0 2024-03-06T07:30:26.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.591+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc240cea80 con 0x7fbc240d22b0 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.592+0000 7fbc197fa640 1 -- 172.21.15.122:0/842084844 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+3422 (secure 0 0 0) 0x7fbc2005bfd0 con 0x7fbc240d22b0 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:19.020885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9318,9321]},"id":10}]} 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 95 2024-03-06T07:30:26.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1006b4d0 msgr2=0x7fbc1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1006b4d0 0x7fbc1006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbc240b2340 tx=0x7fbc14009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 msgr2=0x7fbc24150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 0x7fbc24150d00 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c780 tx=0x7fbc2000cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 shutdown_connections 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1006b4d0 0x7fbc1006d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc2415dfe0 0x7fbc241f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc240d2710 0x7fbc241f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 --2- 172.21.15.122:0/842084844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc240d22b0 0x7fbc24150d00 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 >> 172.21.15.122:0/842084844 conn(0x7fbc240d3cd0 msgr2=0x7fbc240d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 shutdown_connections 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:26.594+0000 7fbc2b037640 1 -- 172.21.15.122:0/842084844 wait complete. 2024-03-06T07:30:26.607 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:26.723 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:26.724 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:26.724 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:28.725 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.824+0000 7f2003288640 1 Processor -- start 2024-03-06T07:30:28.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.824+0000 7f2003288640 1 -- start start 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.825+0000 7f2003288640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1ffc15dfe0 0x7f1ffc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.825+0000 7f2003288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc0d29c0 0x7f1ffc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.825+0000 7f2003288640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc0b3430 con 0x7f1ffc0cf830 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.825+0000 7f2003288640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0b31e0 con 0x7f1ffc15dfe0 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.825+0000 7f2003288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0b3d60 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.825+0000 7f2001024640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1ffc15dfe0 0x7f1ffc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001024640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1ffc15dfe0 0x7f1ffc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48594/0 (socket says 172.21.15.122:48594) 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001024640 1 -- 172.21.15.122:0/1595909221 learned_addr learned my addr 172.21.15.122:0/1595909221 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:28.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001825640 1 --2- 172.21.15.122:0/1595909221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc0d29c0 0x7f1ffc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001024640 1 -- 172.21.15.122:0/1595909221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc0d29c0 msgr2=0x7f1ffc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001024640 1 --2- 172.21.15.122:0/1595909221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc0d29c0 0x7f1ffc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001024640 1 -- 172.21.15.122:0/1595909221 >> v1:172.21.15.169:6789/0 conn(0x7f1ffc0cf830 legacy=0x7f1ffc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.826+0000 7f2001024640 1 -- 172.21.15.122:0/1595909221 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0b3710 con 0x7f1ffc15dfe0 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.827+0000 7f2001024640 1 --2- 172.21.15.122:0/1595909221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc15dfe0 0x7f1ffc0cf2f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009870 tx=0x7f1ff802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e8c60b74a373a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.827+0000 7f1ff3fff640 1 -- 172.21.15.122:0/1595909221 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8035020 con 0x7f1ffc15dfe0 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.827+0000 7f1ff3fff640 1 -- 172.21.15.122:0/1595909221 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8031640 con 0x7f1ffc15dfe0 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.827+0000 7f1ff3fff640 1 -- 172.21.15.122:0/1595909221 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff802d040 con 0x7f1ffc15dfe0 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.827+0000 7f2003288640 1 -- 172.21.15.122:0/1595909221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc15dfe0 msgr2=0x7f1ffc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.827+0000 7f2003288640 1 --2- 172.21.15.122:0/1595909221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc15dfe0 0x7f1ffc0cf2f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009870 tx=0x7f1ff802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.828+0000 7f2003288640 1 -- 172.21.15.122:0/1595909221 shutdown_connections 2024-03-06T07:30:28.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.828+0000 7f2003288640 1 --2- 172.21.15.122:0/1595909221 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc0d29c0 0x7f1ffc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.828+0000 7f2003288640 1 --2- 172.21.15.122:0/1595909221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc15dfe0 0x7f1ffc0cf2f0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.828+0000 7f2003288640 1 -- 172.21.15.122:0/1595909221 >> 172.21.15.122:0/1595909221 conn(0x7f1ffc0d3cd0 msgr2=0x7f1ffc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.828+0000 7f2003288640 1 -- 172.21.15.122:0/1595909221 shutdown_connections 2024-03-06T07:30:28.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.828+0000 7f2003288640 1 -- 172.21.15.122:0/1595909221 wait complete. 2024-03-06T07:30:28.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 Processor -- start 2024-03-06T07:30:28.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 -- start start 2024-03-06T07:30:28.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc0cf830 0x7f1ffc0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 0x7f1ffc1520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc15dfe0 0x7f1ffc15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0d1df0 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0d1c70 con 0x7f1ffc0cf830 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.829+0000 7f2003288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc15e3c0 con 0x7f1ffc15dfe0 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 0x7f1ffc1520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 0x7f1ffc1520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48608/0 (socket says 172.21.15.122:48608) 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 -- 172.21.15.122:0/1947344518 learned_addr learned my addr 172.21.15.122:0/1947344518 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2001024640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc0cf830 0x7f1ffc0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2001825640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc15dfe0 0x7f1ffc15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 -- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc15dfe0 msgr2=0x7f1ffc15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc15dfe0 0x7f1ffc15bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 -- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc0cf830 msgr2=0x7f1ffc0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc0cf830 0x7f1ffc0cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 -- 172.21.15.122:0/1947344518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc15ba70 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2001825640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc15dfe0 0x7f1ffc15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2000823640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 0x7f1ffc1520c0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c400 tx=0x7f1fec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec011790 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec017070 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc0ce010 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.830+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec011bb0 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.831+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc1f7580 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.832+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc04f330 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.832+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fec01c460 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.832+0000 7f1ff1ffb640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1fe006b4d0 0x7f1fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.833+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fec089fc0 con 0x7f1ffc0d29c0 2024-03-06T07:30:28.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.833+0000 7f2001024640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1fe006b4d0 0x7f1fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.833+0000 7f2001024640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1fe006b4d0 0x7f1fe006d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8036000 tx=0x7f1ff803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:28.837+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fec058ed0 con 0x7f1ffc0d29c0 2024-03-06T07:30:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.051+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ffc0ca6b0 con 0x7f1ffc0d29c0 2024-03-06T07:30:29.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.051+0000 7f1ff1ffb640 1 -- 172.21.15.122:0/1947344518 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f1fec0587d0 con 0x7f1ffc0d29c0 2024-03-06T07:30:29.054 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:29.054 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:19.020885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9318,9321]},"id":10}]} 2024-03-06T07:30:29.054 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 95 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1fe006b4d0 msgr2=0x7f1fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1fe006b4d0 0x7f1fe006d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8036000 tx=0x7f1ff803c040 comp rx=0 tx=0).stop 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 msgr2=0x7f1ffc1520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 0x7f1ffc1520c0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c400 tx=0x7f1fec00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 shutdown_connections 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1fe006b4d0 0x7f1fe006d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ffc15dfe0 0x7f1ffc15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ffc0d29c0 0x7f1ffc1520c0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 --2- 172.21.15.122:0/1947344518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ffc0cf830 0x7f1ffc0cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.053+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 >> 172.21.15.122:0/1947344518 conn(0x7f1ffc0d3cd0 msgr2=0x7f1ffc06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.054+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 shutdown_connections 2024-03-06T07:30:29.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:29.054+0000 7f2003288640 1 -- 172.21.15.122:0/1947344518 wait complete. 2024-03-06T07:30:29.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:29.183 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:29.183 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:29.183 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:31.184 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:31.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.320+0000 7f5471565640 1 Processor -- start 2024-03-06T07:30:31.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.320+0000 7f5471565640 1 -- start start 2024-03-06T07:30:31.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.321+0000 7f5471565640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f546c0cf470 0x7f546c0cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.321+0000 7f5471565640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546c15b160 con 0x7f546c0730d0 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.321+0000 7f5471565640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546c0b16f0 con 0x7f546c06bb70 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.321+0000 7f5471565640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c0b4f70 con 0x7f546c0cf470 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.321+0000 7f546a7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f546c0730d0 0x7f546c0b59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52948/0 (socket says 172.21.15.122:52948) 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f546a7fc640 1 -- 172.21.15.122:0/1760770217 learned_addr learned my addr 172.21.15.122:0/1760770217 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f546affd640 1 --2- 172.21.15.122:0/1760770217 >> v2:172.21.15.169:3300/0 conn(0x7f546c0cf470 0x7f546c0cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504187843 0 0) 0x7f546c15b160 con 0x7f546c0730d0 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5454003520 con 0x7f546c0730d0 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225106607 0 0) 0x7f546c0b16f0 con 0x7f546c06bb70 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f546c15b160 con 0x7f546c06bb70 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f546affd640 1 -- 172.21.15.122:0/1760770217 >> v1:172.21.15.169:6789/0 conn(0x7f546c06bb70 legacy=0x7f546c0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244533385 0 0) 0x7f5454003520 con 0x7f546c0730d0 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f546affd640 1 -- 172.21.15.122:0/1760770217 >> v1:172.21.15.122:6789/0 conn(0x7f546c0730d0 legacy=0x7f546c0b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.322+0000 7f546affd640 1 -- 172.21.15.122:0/1760770217 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c0b19d0 con 0x7f546c0cf470 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.323+0000 7f546affd640 1 --2- 172.21.15.122:0/1760770217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0cf470 0x7f546c0cfb70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a0b6bec7f88ece2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.323+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c038070 con 0x7f546c0cf470 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.323+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c031500 con 0x7f546c0cf470 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.323+0000 7f5469ffb640 1 -- 172.21.15.122:0/1760770217 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5460001670 con 0x7f546c0cf470 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 -- 172.21.15.122:0/1760770217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0cf470 msgr2=0x7f546c0cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 --2- 172.21.15.122:0/1760770217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0cf470 0x7f546c0cfb70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02b820 comp rx=0 tx=0).stop 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 -- 172.21.15.122:0/1760770217 shutdown_connections 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 --2- 172.21.15.122:0/1760770217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0cf470 0x7f546c0cfb70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 -- 172.21.15.122:0/1760770217 >> 172.21.15.122:0/1760770217 conn(0x7f546c0724d0 msgr2=0x7f546c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 -- 172.21.15.122:0/1760770217 shutdown_connections 2024-03-06T07:30:31.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.324+0000 7f5471565640 1 -- 172.21.15.122:0/1760770217 wait complete. 2024-03-06T07:30:31.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.326+0000 7f5471565640 1 Processor -- start 2024-03-06T07:30:31.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.326+0000 7f5471565640 1 -- start start 2024-03-06T07:30:31.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f5471565640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f546c06bb70 0x7f546c1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f5471565640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0730d0 0x7f546c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f5471565640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 0x7f546c14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f5471565640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0b1330 con 0x7f546c0cf470 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f5471565640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0b54a0 con 0x7f546c0730d0 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f5471565640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0b14b0 con 0x7f546c06bb70 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f546b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 0x7f546c14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f546b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 0x7f546c14ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48610/0 (socket says 172.21.15.122:48610) 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f546b7fe640 1 -- 172.21.15.122:0/3948076354 learned_addr learned my addr 172.21.15.122:0/3948076354 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f546a7fc640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0730d0 0x7f546c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.327+0000 7f546affd640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f546c06bb70 0x7f546c1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546b7fe640 1 -- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f546c06bb70 msgr2=0x7f546c1504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546b7fe640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f546c06bb70 0x7f546c1504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546b7fe640 1 -- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0730d0 msgr2=0x7f546c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546b7fe640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0730d0 0x7f546c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546b7fe640 1 -- 172.21.15.122:0/3948076354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c1e4190 con 0x7f546c0cf470 2024-03-06T07:30:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546a7fc640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0730d0 0x7f546c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:31.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f546b7fe640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 0x7f546c14ed50 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f546400c790 tx=0x7f546400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464003b70 con 0x7f546c0cf470 2024-03-06T07:30:31.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5464003d10 con 0x7f546c0cf470 2024-03-06T07:30:31.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546c15be30 con 0x7f546c0cf470 2024-03-06T07:30:31.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546400aa50 con 0x7f546c0cf470 2024-03-06T07:30:31.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.328+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c15c3f0 con 0x7f546c0cf470 2024-03-06T07:30:31.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.329+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f546400abf0 con 0x7f546c0cf470 2024-03-06T07:30:31.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.329+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c04f330 con 0x7f546c0cf470 2024-03-06T07:30:31.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.330+0000 7f54537fe640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f545806b4d0 0x7f545806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.330+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f546408a080 con 0x7f546c0cf470 2024-03-06T07:30:31.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.330+0000 7f546affd640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f545806b4d0 0x7f545806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.330+0000 7f546affd640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f545806b4d0 0x7f545806d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f545c035000 tx=0x7f545c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.334+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54640588b0 con 0x7f546c0cf470 2024-03-06T07:30:31.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.623+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f546c15a1d0 con 0x7f546c0cf470 2024-03-06T07:30:31.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.623+0000 7f54537fe640 1 -- 172.21.15.122:0/3948076354 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+3422 (secure 0 0 0) 0x7f546405c950 con 0x7f546c0cf470 2024-03-06T07:30:31.625 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:31.625 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:19.020885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9318,9321]},"id":10}]} 2024-03-06T07:30:31.626 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 95 2024-03-06T07:30:31.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f545806b4d0 msgr2=0x7f545806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f545806b4d0 0x7f545806d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f545c035000 tx=0x7f545c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 msgr2=0x7f546c14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 0x7f546c14ed50 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f546400c790 tx=0x7f546400cc50 comp rx=0 tx=0).stop 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 shutdown_connections 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f545806b4d0 0x7f545806d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0cf470 0x7f546c14ed50 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f546c0730d0 0x7f546c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 --2- 172.21.15.122:0/3948076354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f546c06bb70 0x7f546c1504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 >> 172.21.15.122:0/3948076354 conn(0x7f546c0724d0 msgr2=0x7f546c159440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 shutdown_connections 2024-03-06T07:30:31.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:31.625+0000 7f5471565640 1 -- 172.21.15.122:0/3948076354 wait complete. 2024-03-06T07:30:31.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:31.757 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:31.757 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:31.757 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:33.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:33.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.857+0000 7fcf06ceb640 1 Processor -- start 2024-03-06T07:30:33.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.857+0000 7fcf06ceb640 1 -- start start 2024-03-06T07:30:33.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf06ceb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fcf000d22b0 0x7fcf000d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf06ceb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcf000b41f0 0x7fcf000b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf06ceb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf000b3430 con 0x7fcf0015dfe0 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf06ceb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000d2c50 con 0x7fcf000b41f0 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf06ceb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000b31e0 con 0x7fcf000d22b0 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcf000b41f0 0x7fcf000b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcf000b41f0 0x7fcf000b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48622/0 (socket says 172.21.15.122:48622) 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 -- 172.21.15.122:0/1576130845 learned_addr learned my addr 172.21.15.122:0/1576130845 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 -- 172.21.15.122:0/1576130845 >> v2:172.21.15.169:3300/0 conn(0x7fcf000d22b0 msgr2=0x7fcf000d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 --2- 172.21.15.122:0/1576130845 >> v2:172.21.15.169:3300/0 conn(0x7fcf000d22b0 0x7fcf000d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 -- 172.21.15.122:0/1576130845 >> v1:172.21.15.169:6789/0 conn(0x7fcf0015dfe0 legacy=0x7fcf000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.858+0000 7fcf05288640 1 -- 172.21.15.122:0/1576130845 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000b3710 con 0x7fcf000b41f0 2024-03-06T07:30:33.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.859+0000 7fcf05288640 1 --2- 172.21.15.122:0/1576130845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000b41f0 0x7fcf000b4680 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009e50 tx=0x7fcef802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fa1809f499c1bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.859+0000 7fceff7fe640 1 -- 172.21.15.122:0/1576130845 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef802ca80 con 0x7fcf000b41f0 2024-03-06T07:30:33.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.859+0000 7fceff7fe640 1 -- 172.21.15.122:0/1576130845 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef802cc20 con 0x7fcf000b41f0 2024-03-06T07:30:33.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.859+0000 7fceff7fe640 1 -- 172.21.15.122:0/1576130845 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80343f0 con 0x7fcf000b41f0 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1576130845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000b41f0 msgr2=0x7fcf000b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1576130845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000b41f0 0x7fcf000b4680 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009e50 tx=0x7fcef802bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1576130845 shutdown_connections 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1576130845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000b41f0 0x7fcf000b4680 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1576130845 >> v2:172.21.15.169:3300/0 conn(0x7fcf000d22b0 0x7fcf000d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1576130845 >> 172.21.15.122:0/1576130845 conn(0x7fcf000d3cd0 msgr2=0x7fcf000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1576130845 shutdown_connections 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.860+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1576130845 wait complete. 2024-03-06T07:30:33.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.861+0000 7fcf06ceb640 1 Processor -- start 2024-03-06T07:30:33.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.861+0000 7fcf06ceb640 1 -- start start 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcf06ceb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf000b41f0 0x7fcf0015e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcf06ceb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 0x7fcf000b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcf06ceb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf0015dfe0 0x7fcf00150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcf06ceb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000cfa00 con 0x7fcf000d22b0 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcf06ceb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000b3d60 con 0x7fcf0015dfe0 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcf06ceb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000d1c70 con 0x7fcf000b41f0 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcefffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 0x7fcf000b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.862+0000 7fcefffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 0x7fcf000b7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48626/0 (socket says 172.21.15.122:48626) 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 -- 172.21.15.122:0/1057541584 learned_addr learned my addr 172.21.15.122:0/1057541584 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcf05288640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf0015dfe0 0x7fcf00150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcf04a87640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf000b41f0 0x7fcf0015e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 -- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf000b41f0 msgr2=0x7fcf0015e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf000b41f0 0x7fcf0015e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 -- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf0015dfe0 msgr2=0x7fcf00150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf0015dfe0 0x7fcf00150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 -- 172.21.15.122:0/1057541584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0015eaa0 con 0x7fcf000d22b0 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefffff640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 0x7fcf000b7a40 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fceec00b4a0 tx=0x7fceec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec014070 con 0x7fcf000d22b0 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec010610 con 0x7fcf000d22b0 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec010a00 con 0x7fcf000d22b0 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0015ed80 con 0x7fcf000d22b0 2024-03-06T07:30:33.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.863+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0015f340 con 0x7fcf000d22b0 2024-03-06T07:30:33.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.864+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0004f330 con 0x7fcf000d22b0 2024-03-06T07:30:33.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.865+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceec017070 con 0x7fcf000d22b0 2024-03-06T07:30:33.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.865+0000 7fcefd7fa640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcee406b500 0x7fcee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.865+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceec089ca0 con 0x7fcf000d22b0 2024-03-06T07:30:33.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.865+0000 7fcf04a87640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcee406b500 0x7fcee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.865+0000 7fcf04a87640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcee406b500 0x7fcee406d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcef4006fd0 tx=0x7fcef4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:33.870+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fceec058aa0 con 0x7fcf000d22b0 2024-03-06T07:30:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.089+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf000d4940 con 0x7fcf000d22b0 2024-03-06T07:30:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.089+0000 7fcefd7fa640 1 -- 172.21.15.122:0/1057541584 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3423 (secure 0 0 0) 0x7fceec0583a0 con 0x7fcf000d22b0 2024-03-06T07:30:34.091 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:34.091 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":96,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:32.111604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321,9318]},"id":10}]} 2024-03-06T07:30:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 96 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcee406b500 msgr2=0x7fcee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcee406b500 0x7fcee406d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcef4006fd0 tx=0x7fcef4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 msgr2=0x7fcf000b7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 0x7fcf000b7a40 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fceec00b4a0 tx=0x7fceec00b960 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 shutdown_connections 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcee406b500 0x7fcee406d9c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf0015dfe0 0x7fcf00150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf000d22b0 0x7fcf000b7a40 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 --2- 172.21.15.122:0/1057541584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcf000b41f0 0x7fcf0015e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 >> 172.21.15.122:0/1057541584 conn(0x7fcf000d3cd0 msgr2=0x7fcf0015d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 shutdown_connections 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.091+0000 7fcf06ceb640 1 -- 172.21.15.122:0/1057541584 wait complete. 2024-03-06T07:30:34.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:34.227 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"16","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:30:34.228 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:34.228 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2024-03-06T07:30:34.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.228 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 dir_2) 2024-03-06T07:30:34.244 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.403+0000 7fed68925640 1 Processor -- start 2024-03-06T07:30:34.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.403+0000 7fed68925640 1 -- start start 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed68925640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed6415dfe0 0x7fed640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed68925640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fed640d29c0 0x7fed640b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed68925640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640b3430 con 0x7fed640cf830 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed68925640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed640b31e0 con 0x7fed640d29c0 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed68925640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640b3d60 con 0x7fed6415dfe0 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed62d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fed640d29c0 0x7fed640b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed62d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fed640d29c0 0x7fed640b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48640/0 (socket says 172.21.15.122:48640) 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.404+0000 7fed62d9d640 1 -- 172.21.15.122:0/1189399159 learned_addr learned my addr 172.21.15.122:0/1189399159 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.405+0000 7fed6259c640 1 --2- 172.21.15.122:0/1189399159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed6415dfe0 0x7fed640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.405+0000 7fed62d9d640 1 -- 172.21.15.122:0/1189399159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed6415dfe0 msgr2=0x7fed640cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.405+0000 7fed62d9d640 1 --2- 172.21.15.122:0/1189399159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed6415dfe0 0x7fed640cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.405+0000 7fed62d9d640 1 -- 172.21.15.122:0/1189399159 >> v1:172.21.15.122:6789/0 conn(0x7fed640cf830 legacy=0x7fed640d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.405+0000 7fed62d9d640 1 -- 172.21.15.122:0/1189399159 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed640b3710 con 0x7fed640d29c0 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.406+0000 7fed62d9d640 1 --2- 172.21.15.122:0/1189399159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640d29c0 0x7fed640b41f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fed54009870 tx=0x7fed5402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25e4274ab4386ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.406+0000 7fed6159a640 1 -- 172.21.15.122:0/1189399159 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed54034070 con 0x7fed640d29c0 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.406+0000 7fed6159a640 1 -- 172.21.15.122:0/1189399159 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed54030840 con 0x7fed640d29c0 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.406+0000 7fed6159a640 1 -- 172.21.15.122:0/1189399159 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed54030c10 con 0x7fed640d29c0 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 -- 172.21.15.122:0/1189399159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640d29c0 msgr2=0x7fed640b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 --2- 172.21.15.122:0/1189399159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640d29c0 0x7fed640b41f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fed54009870 tx=0x7fed5402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 -- 172.21.15.122:0/1189399159 shutdown_connections 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 --2- 172.21.15.122:0/1189399159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640d29c0 0x7fed640b41f0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 --2- 172.21.15.122:0/1189399159 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed6415dfe0 0x7fed640cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 -- 172.21.15.122:0/1189399159 >> 172.21.15.122:0/1189399159 conn(0x7fed640d3cd0 msgr2=0x7fed640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 -- 172.21.15.122:0/1189399159 shutdown_connections 2024-03-06T07:30:34.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.407+0000 7fed68925640 1 -- 172.21.15.122:0/1189399159 wait complete. 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.408+0000 7fed68925640 1 Processor -- start 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.408+0000 7fed68925640 1 -- start start 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 0x7fed640a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed640d29c0 0x7fed640cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed6415dfe0 0x7fed6415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed640d1df0 con 0x7fed640cf830 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed640d1c70 con 0x7fed6415dfe0 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6415e3c0 con 0x7fed640d29c0 2024-03-06T07:30:34.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 0x7fed640a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 0x7fed640a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48642/0 (socket says 172.21.15.122:48642) 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 -- 172.21.15.122:0/3050751192 learned_addr learned my addr 172.21.15.122:0/3050751192 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed62d9d640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed6415dfe0 0x7fed6415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed61d9b640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed640d29c0 0x7fed640cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 -- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed640d29c0 msgr2=0x7fed640cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed640d29c0 0x7fed640cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 -- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed6415dfe0 msgr2=0x7fed6415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed6415dfe0 0x7fed6415be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 -- 172.21.15.122:0/3050751192 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed640caf80 con 0x7fed640cf830 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed6259c640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 0x7fed640a1690 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00d5f0 tx=0x7fed5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0125f0 con 0x7fed640cf830 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c019070 con 0x7fed640cf830 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed640cb260 con 0x7fed640cf830 2024-03-06T07:30:34.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.409+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0129d0 con 0x7fed640cf830 2024-03-06T07:30:34.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.410+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed640cd7a0 con 0x7fed640cf830 2024-03-06T07:30:34.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.411+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6404f330 con 0x7fed640cf830 2024-03-06T07:30:34.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.411+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed5c012b70 con 0x7fed640cf830 2024-03-06T07:30:34.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.411+0000 7fed52ffd640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed4406b4d0 0x7fed4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.412+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed5c08a580 con 0x7fed640cf830 2024-03-06T07:30:34.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.412+0000 7fed61d9b640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed4406b4d0 0x7fed4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.412+0000 7fed61d9b640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed4406b4d0 0x7fed4406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fed58007fd0 tx=0x7fed58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.416+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed5c059490 con 0x7fed640cf830 2024-03-06T07:30:34.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.789+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed640ca380 con 0x7fed640cf830 2024-03-06T07:30:34.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.790+0000 7fed52ffd640 1 -- 172.21.15.122:0/3050751192 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3423 (secure 0 0 0) 0x7fed5c058d90 con 0x7fed640cf830 2024-03-06T07:30:34.792 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:34.792 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":96,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:32.111604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321,9318]},"id":10}]} 2024-03-06T07:30:34.793 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 96 2024-03-06T07:30:34.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed4406b4d0 msgr2=0x7fed4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed4406b4d0 0x7fed4406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fed58007fd0 tx=0x7fed58009040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 msgr2=0x7fed640a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 0x7fed640a1690 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00d5f0 tx=0x7fed5c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 shutdown_connections 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fed4406b4d0 0x7fed4406d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed6415dfe0 0x7fed6415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fed640d29c0 0x7fed640cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 --2- 172.21.15.122:0/3050751192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed640cf830 0x7fed640a1690 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 >> 172.21.15.122:0/3050751192 conn(0x7fed640d3cd0 msgr2=0x7fed640b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 shutdown_connections 2024-03-06T07:30:34.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:34.792+0000 7fed68925640 1 -- 172.21.15.122:0/3050751192 wait complete. 2024-03-06T07:30:34.808 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:34.925 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:34.926 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:34.926 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:36.927 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.031+0000 7f80cbb79640 1 Processor -- start 2024-03-06T07:30:37.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.031+0000 7f80cbb79640 1 -- start start 2024-03-06T07:30:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80cbb79640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80c40cce40 0x7f80c415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80cbb79640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cd7a0 0x7f80c40cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80cbb79640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c4071f40 con 0x7f80c41515e0 2024-03-06T07:30:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80cbb79640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c40cd2f0 con 0x7f80c40cce40 2024-03-06T07:30:37.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80cbb79640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c4070a70 con 0x7f80c40cd7a0 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80c40cce40 0x7f80c415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80c40cce40 0x7f80c415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51322/0 (socket says 172.21.15.122:51322) 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 -- 172.21.15.122:0/78344568 learned_addr learned my addr 172.21.15.122:0/78344568 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80ca116640 1 --2- 172.21.15.122:0/78344568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cd7a0 0x7f80c40cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 -- 172.21.15.122:0/78344568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cd7a0 msgr2=0x7f80c40cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 --2- 172.21.15.122:0/78344568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cd7a0 0x7f80c40cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 -- 172.21.15.122:0/78344568 >> v1:172.21.15.169:6789/0 conn(0x7f80c41515e0 legacy=0x7f80c4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.032+0000 7f80c9114640 1 -- 172.21.15.122:0/78344568 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c4070d50 con 0x7f80c40cce40 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.033+0000 7f80c9114640 1 --2- 172.21.15.122:0/78344568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cce40 0x7f80c415b430 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f80c40cc110 tx=0x7f80b402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a611f71d32ddd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.033+0000 7f80c8913640 1 -- 172.21.15.122:0/78344568 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b403b070 con 0x7f80c40cce40 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.033+0000 7f80c8913640 1 -- 172.21.15.122:0/78344568 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b402ee50 con 0x7f80c40cce40 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.033+0000 7f80c8913640 1 -- 172.21.15.122:0/78344568 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b4036660 con 0x7f80c40cce40 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 -- 172.21.15.122:0/78344568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cce40 msgr2=0x7f80c415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 --2- 172.21.15.122:0/78344568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cce40 0x7f80c415b430 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f80c40cc110 tx=0x7f80b402e0a0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 -- 172.21.15.122:0/78344568 shutdown_connections 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 --2- 172.21.15.122:0/78344568 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cd7a0 0x7f80c40cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 --2- 172.21.15.122:0/78344568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cce40 0x7f80c415b430 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 -- 172.21.15.122:0/78344568 >> 172.21.15.122:0/78344568 conn(0x7f80c40cf8a0 msgr2=0x7f80c40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 -- 172.21.15.122:0/78344568 shutdown_connections 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.034+0000 7f80cbb79640 1 -- 172.21.15.122:0/78344568 wait complete. 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.035+0000 7f80cbb79640 1 Processor -- start 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.035+0000 7f80cbb79640 1 -- start start 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80cbb79640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cce40 0x7f80c406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80cbb79640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 0x7f80c414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80cbb79640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80c41515e0 0x7f80c40c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80cbb79640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40ce0f0 con 0x7f80c40cd7a0 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80cbb79640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c4071310 con 0x7f80c41515e0 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80cbb79640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40cc3a0 con 0x7f80c40cce40 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 0x7f80c414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 0x7f80c414ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51324/0 (socket says 172.21.15.122:51324) 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 -- 172.21.15.122:0/1600783266 learned_addr learned my addr 172.21.15.122:0/1600783266 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:37.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9915640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cce40 0x7f80c406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80ca116640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80c41515e0 0x7f80c40c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 -- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cce40 msgr2=0x7f80c406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cce40 0x7f80c406d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 -- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80c41515e0 msgr2=0x7f80c40c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80c41515e0 0x7f80c40c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 -- 172.21.15.122:0/1600783266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c406d890 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80c9114640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 0x7f80c414ace0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f80b402e0e0 tx=0x7f80b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.036+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b4046070 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.037+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b403f550 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.037+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b403b070 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.037+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c40cb120 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.037+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c40cb6e0 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.038+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c4068840 con 0x7f80c40cd7a0 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.038+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80b4049070 con 0x7f80c40cd7a0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.039+0000 7f80ba7fc640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f80a806b4d0 0x7f80a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.039+0000 7f80c9915640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f80a806b4d0 0x7f80a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.039+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f80b40abfb0 con 0x7f80c40cd7a0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.039+0000 7f80c9915640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f80a806b4d0 0x7f80a806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f80c000a750 tx=0x7f80c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.047+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80b407ae10 con 0x7f80c40cd7a0 2024-03-06T07:30:37.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.265+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80c404f330 con 0x7f80c40cd7a0 2024-03-06T07:30:37.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.265+0000 7f80ba7fc640 1 -- 172.21.15.122:0/1600783266 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3423 (secure 0 0 0) 0x7f80b407a710 con 0x7f80c40cd7a0 2024-03-06T07:30:37.268 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:37.268 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":96,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:32.111604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321,9318]},"id":10}]} 2024-03-06T07:30:37.268 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 96 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f80a806b4d0 msgr2=0x7f80a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f80a806b4d0 0x7f80a806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f80c000a750 tx=0x7f80c0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 msgr2=0x7f80c414ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 0x7f80c414ace0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f80b402e0e0 tx=0x7f80b4004730 comp rx=0 tx=0).stop 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 shutdown_connections 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f80a806b4d0 0x7f80a806d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80c41515e0 0x7f80c40c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80c40cd7a0 0x7f80c414ace0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 --2- 172.21.15.122:0/1600783266 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80c40cce40 0x7f80c406d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 >> 172.21.15.122:0/1600783266 conn(0x7f80c40cf8a0 msgr2=0x7f80c406df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 shutdown_connections 2024-03-06T07:30:37.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:37.267+0000 7f80cbb79640 1 -- 172.21.15.122:0/1600783266 wait complete. 2024-03-06T07:30:37.280 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:37.396 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:37.396 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:37.397 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:39.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.502+0000 7f4fe89ac640 1 Processor -- start 2024-03-06T07:30:39.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.502+0000 7f4fe89ac640 1 -- start start 2024-03-06T07:30:39.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe89ac640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4fe415baf0 0x7f4fe4072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe89ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4fe406b6c0 0x7f4fe40ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe89ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40d0610 con 0x7f4fe40cd390 2024-03-06T07:30:39.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe89ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe4072c40 con 0x7f4fe406b6c0 2024-03-06T07:30:39.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe89ac640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe40d08f0 con 0x7f4fe415baf0 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4fe406b6c0 0x7f4fe40ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4fe406b6c0 0x7f4fe40ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51340/0 (socket says 172.21.15.122:51340) 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 -- 172.21.15.122:0/1849025475 learned_addr learned my addr 172.21.15.122:0/1849025475 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe259c640 1 --2- 172.21.15.122:0/1849025475 >> v2:172.21.15.169:3300/0 conn(0x7f4fe415baf0 0x7f4fe4072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 -- 172.21.15.122:0/1849025475 >> v2:172.21.15.169:3300/0 conn(0x7f4fe415baf0 msgr2=0x7f4fe4072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 --2- 172.21.15.122:0/1849025475 >> v2:172.21.15.169:3300/0 conn(0x7f4fe415baf0 0x7f4fe4072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 -- 172.21.15.122:0/1849025475 >> v1:172.21.15.122:6789/0 conn(0x7f4fe40cd390 legacy=0x7f4fe406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.504+0000 7f4fe2d9d640 1 -- 172.21.15.122:0/1849025475 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40d0bd0 con 0x7f4fe406b6c0 2024-03-06T07:30:39.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.505+0000 7f4fe2d9d640 1 --2- 172.21.15.122:0/1849025475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe40ced90 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4073720 tx=0x7f4fdc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14dbfaf8992fa4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.505+0000 7f4fe159a640 1 -- 172.21.15.122:0/1849025475 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc02cb20 con 0x7f4fe406b6c0 2024-03-06T07:30:39.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.505+0000 7f4fe159a640 1 -- 172.21.15.122:0/1849025475 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc02ccc0 con 0x7f4fe406b6c0 2024-03-06T07:30:39.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.505+0000 7f4fe159a640 1 -- 172.21.15.122:0/1849025475 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc034610 con 0x7f4fe406b6c0 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 -- 172.21.15.122:0/1849025475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 msgr2=0x7f4fe40ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/1849025475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe40ced90 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4073720 tx=0x7f4fdc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 -- 172.21.15.122:0/1849025475 shutdown_connections 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/1849025475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe40ced90 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/1849025475 >> v2:172.21.15.169:3300/0 conn(0x7f4fe415baf0 0x7f4fe4072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 -- 172.21.15.122:0/1849025475 >> 172.21.15.122:0/1849025475 conn(0x7f4fe406c0c0 msgr2=0x7f4fe406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 -- 172.21.15.122:0/1849025475 shutdown_connections 2024-03-06T07:30:39.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.506+0000 7f4fe89ac640 1 -- 172.21.15.122:0/1849025475 wait complete. 2024-03-06T07:30:39.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.508+0000 7f4fe89ac640 1 Processor -- start 2024-03-06T07:30:39.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.508+0000 7f4fe89ac640 1 -- start start 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe89ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe4137360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe89ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fe40cd390 0x7f4fe415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe89ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe415baf0 0x7f4fe415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe89ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe415bed0 con 0x7f4fe406b6c0 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe89ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40d1190 con 0x7f4fe415baf0 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe89ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40d16b0 con 0x7f4fe40cd390 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe4137360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.509+0000 7f4fe259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe4137360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51356/0 (socket says 172.21.15.122:51356) 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 -- 172.21.15.122:0/2064440816 learned_addr learned my addr 172.21.15.122:0/2064440816 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe2d9d640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe415baf0 0x7f4fe415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe1d9b640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fe40cd390 0x7f4fe415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 -- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fe40cd390 msgr2=0x7f4fe415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fe40cd390 0x7f4fe415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 -- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe415baf0 msgr2=0x7f4fe415cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe415baf0 0x7f4fe415cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 -- 172.21.15.122:0/2064440816 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe415ed60 con 0x7f4fe406b6c0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe259c640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe4137360 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400e8a0 tx=0x7f4fd400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4017070 con 0x7f4fe406b6c0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd400c940 con 0x7f4fe406b6c0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400cd30 con 0x7f4fe406b6c0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe415d5d0 con 0x7f4fe406b6c0 2024-03-06T07:30:39.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.510+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe4137b30 con 0x7f4fe406b6c0 2024-03-06T07:30:39.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.511+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe404f330 con 0x7f4fe406b6c0 2024-03-06T07:30:39.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.512+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fd401a070 con 0x7f4fe406b6c0 2024-03-06T07:30:39.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.512+0000 7f4fd2ffd640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fc406b4d0 0x7f4fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.512+0000 7f4fe1d9b640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fc406b4d0 0x7f4fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.513+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4fd408a450 con 0x7f4fe406b6c0 2024-03-06T07:30:39.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.513+0000 7f4fe1d9b640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fc406b4d0 0x7f4fc406d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8007fd0 tx=0x7f4fd8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.518+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fd40592b0 con 0x7f4fe406b6c0 2024-03-06T07:30:39.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.734+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fe4071f00 con 0x7f4fe406b6c0 2024-03-06T07:30:39.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.735+0000 7f4fd2ffd640 1 -- 172.21.15.122:0/2064440816 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3423 (secure 0 0 0) 0x7f4fd4058bb0 con 0x7f4fe406b6c0 2024-03-06T07:30:39.736 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:39.736 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":96,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:32.111604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321,9318]},"id":10}]} 2024-03-06T07:30:39.737 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 96 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fc406b4d0 msgr2=0x7f4fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fc406b4d0 0x7f4fc406d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8007fd0 tx=0x7f4fd8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 msgr2=0x7f4fe4137360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe4137360 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400e8a0 tx=0x7f4fd400ed60 comp rx=0 tx=0).stop 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 shutdown_connections 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4fc406b4d0 0x7f4fc406d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe415baf0 0x7f4fe415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4fe40cd390 0x7f4fe415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 --2- 172.21.15.122:0/2064440816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fe406b6c0 0x7f4fe4137360 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 >> 172.21.15.122:0/2064440816 conn(0x7f4fe406c0c0 msgr2=0x7f4fe40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 shutdown_connections 2024-03-06T07:30:39.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:39.737+0000 7f4fe89ac640 1 -- 172.21.15.122:0/2064440816 wait complete. 2024-03-06T07:30:39.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:39.869 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:39.870 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:39.870 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:41.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.009+0000 7f8e227ac640 1 Processor -- start 2024-03-06T07:30:42.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.009+0000 7f8e227ac640 1 -- start start 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e227ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 0x7f8e1c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e227ac640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8e1c0d4760 0x7f8e1c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e227ac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0d25d0 con 0x7f8e1c0b4dd0 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e227ac640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c0d4bf0 con 0x7f8e1c0d4760 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e227ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0ce9e0 con 0x7f8e1c0b4170 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 0x7f8e1c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8e1c0d4760 0x7f8e1c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 0x7f8e1c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:60282/0 (socket says 172.21.15.122:60282) 2024-03-06T07:30:42.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 -- 172.21.15.122:0/1851451373 learned_addr learned my addr 172.21.15.122:0/1851451373 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:42.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 -- 172.21.15.122:0/1851451373 >> v2:172.21.15.169:3300/0 conn(0x7f8e1c0d4760 msgr2=0x7f8e1c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 --2- 172.21.15.122:0/1851451373 >> v2:172.21.15.169:3300/0 conn(0x7f8e1c0d4760 0x7f8e1c0d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1affd640 1 -- 172.21.15.122:0/1851451373 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340937689 0 0) 0x7f8e1c0d25d0 con 0x7f8e1c0b4dd0 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 -- 172.21.15.122:0/1851451373 >> v1:172.21.15.169:6789/0 conn(0x7f8e1c0b4dd0 legacy=0x7f8e1c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.011+0000 7f8e1bfff640 1 -- 172.21.15.122:0/1851451373 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0d2870 con 0x7f8e1c0b4170 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.012+0000 7f8e1bfff640 1 --2- 172.21.15.122:0/1851451373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 0x7f8e1c0b6a90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009870 tx=0x7f8e0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63010a122675cd00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.012+0000 7f8e1affd640 1 -- 172.21.15.122:0/1851451373 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c034070 con 0x7f8e1c0b4170 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.012+0000 7f8e1affd640 1 -- 172.21.15.122:0/1851451373 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c030840 con 0x7f8e1c0b4170 2024-03-06T07:30:42.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e1affd640 1 -- 172.21.15.122:0/1851451373 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c030c10 con 0x7f8e1c0b4170 2024-03-06T07:30:42.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 -- 172.21.15.122:0/1851451373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 msgr2=0x7f8e1c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1851451373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 0x7f8e1c0b6a90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009870 tx=0x7f8e0c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:42.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 -- 172.21.15.122:0/1851451373 shutdown_connections 2024-03-06T07:30:42.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1851451373 >> v2:172.21.15.169:3300/0 conn(0x7f8e1c0d4760 0x7f8e1c0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1851451373 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4170 0x7f8e1c0b6a90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 -- 172.21.15.122:0/1851451373 >> 172.21.15.122:0/1851451373 conn(0x7f8e1c0d3c40 msgr2=0x7f8e1c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 -- 172.21.15.122:0/1851451373 shutdown_connections 2024-03-06T07:30:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.013+0000 7f8e227ac640 1 -- 172.21.15.122:0/1851451373 wait complete. 2024-03-06T07:30:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.014+0000 7f8e227ac640 1 Processor -- start 2024-03-06T07:30:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.014+0000 7f8e227ac640 1 -- start start 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e227ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e1c0b4170 0x7f8e1c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e227ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 0x7f8e1c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e227ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e1c158e30 0x7f8e1c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e227ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0b47d0 con 0x7f8e1c158e30 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e227ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0b4550 con 0x7f8e1c0b4170 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e227ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c0b4950 con 0x7f8e1c0b4dd0 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e1b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 0x7f8e1c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e20d49640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e1c158e30 0x7f8e1c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e1c0b4170 0x7f8e1c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e1b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 0x7f8e1c1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:60294/0 (socket says 172.21.15.122:60294) 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.015+0000 7f8e1b7fe640 1 -- 172.21.15.122:0/1959456639 learned_addr learned my addr 172.21.15.122:0/1959456639 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e20d49640 1 -- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 msgr2=0x7f8e1c1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e20d49640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 0x7f8e1c1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e20d49640 1 -- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e1c0b4170 msgr2=0x7f8e1c15b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e20d49640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e1c0b4170 0x7f8e1c15b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e20d49640 1 -- 172.21.15.122:0/1959456639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0b0e40 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e1b7fe640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 0x7f8e1c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e20d49640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e1c158e30 0x7f8e1c0b2e00 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400c6d0 tx=0x7f8e1400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14003b70 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c0b0b60 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14003d30 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e140052b0 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.016+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c0b1550 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.017+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c0cca10 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.018+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e14005450 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.018+0000 7f8e18ff9640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e0806b4d0 0x7f8e0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.018+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8e1408a030 con 0x7f8e1c158e30 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.018+0000 7f8e1bfff640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e0806b4d0 0x7f8e0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.019+0000 7f8e1bfff640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e0806b4d0 0x7f8e0806d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c036000 tx=0x7f8e0c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.023+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e1c0cca10 con 0x7f8e1c158e30 2024-03-06T07:30:42.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.269+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e1c04f330 con 0x7f8e1c158e30 2024-03-06T07:30:42.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.269+0000 7f8e18ff9640 1 -- 172.21.15.122:0/1959456639 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3423 (secure 0 0 0) 0x7f8e140587c0 con 0x7f8e1c158e30 2024-03-06T07:30:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:42.272 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":96,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:30:32.111604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321,9318]},"id":10}]} 2024-03-06T07:30:42.272 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 96 2024-03-06T07:30:42.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e0806b4d0 msgr2=0x7f8e0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e0806b4d0 0x7f8e0806d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c036000 tx=0x7f8e0c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:42.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e1c158e30 msgr2=0x7f8e1c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e1c158e30 0x7f8e1c0b2e00 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400c6d0 tx=0x7f8e1400cb90 comp rx=0 tx=0).stop 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 shutdown_connections 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8e0806b4d0 0x7f8e0806d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e1c158e30 0x7f8e1c0b2e00 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8e1c0b4dd0 0x7f8e1c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 --2- 172.21.15.122:0/1959456639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e1c0b4170 0x7f8e1c15b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 >> 172.21.15.122:0/1959456639 conn(0x7f8e1c0d3c40 msgr2=0x7f8e1c06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 shutdown_connections 2024-03-06T07:30:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.272+0000 7f8e227ac640 1 -- 172.21.15.122:0/1959456639 wait complete. 2024-03-06T07:30:42.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi122.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-06T07:30:42.438 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:42.439 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2024-03-06T07:30:42.439 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:30:42.439 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:42.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:42.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.592+0000 7f07a56b5640 1 Processor -- start 2024-03-06T07:30:42.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f07a56b5640 1 -- start start 2024-03-06T07:30:42.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f07a56b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f07a00d22b0 0x7f07a00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f07a56b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a00b3430 con 0x7f07a00b41f0 2024-03-06T07:30:42.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f07a56b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a00b3710 con 0x7f07a015dfe0 2024-03-06T07:30:42.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f07a56b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a00d2c50 con 0x7f07a00d22b0 2024-03-06T07:30:42.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f079f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f07a00b41f0 0x7f07a00b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57594/0 (socket says 172.21.15.122:57594) 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.593+0000 7f079f7fe640 1 -- 172.21.15.122:0/2781719061 learned_addr learned my addr 172.21.15.122:0/2781719061 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079e7fc640 1 --2- 172.21.15.122:0/2781719061 >> v2:172.21.15.169:3300/0 conn(0x7f07a00d22b0 0x7f07a00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744655882 0 0) 0x7f07a00b3430 con 0x7f07a00b41f0 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0788003520 con 0x7f07a00b41f0 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4131140490 0 0) 0x7f0788003520 con 0x7f07a00b41f0 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 >> v2:172.21.15.169:3300/0 conn(0x7f07a00d22b0 msgr2=0x7f07a00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 --2- 172.21.15.122:0/2781719061 >> v2:172.21.15.169:3300/0 conn(0x7f07a00d22b0 0x7f07a00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 >> v1:172.21.15.169:6789/0 conn(0x7f07a015dfe0 legacy=0x7f07a00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00b39f0 con 0x7f07a00b41f0 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f07980019f0 con 0x7f07a00b41f0 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 >> v1:172.21.15.122:6789/0 conn(0x7f07a00b41f0 legacy=0x7f07a00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07880039b0 0x7f0788003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0788004800 0x7f0788004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07880051c0 0x7f078800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078800c030 con 0x7f07880039b0 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078800c1e0 con 0x7f0788004800 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078800c3e0 con 0x7f07880051c0 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079effd640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07880039b0 0x7f0788003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079e7fc640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0788004800 0x7f0788004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079f7fe640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07880051c0 0x7f078800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079effd640 1 -- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07880051c0 msgr2=0x7f078800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079effd640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07880051c0 0x7f078800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079effd640 1 -- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0788004800 msgr2=0x7f0788004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079effd640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0788004800 0x7f0788004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079effd640 1 -- 172.21.15.122:0/2781719061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788003520 con 0x7f07880039b0 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.594+0000 7f079f7fe640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07880051c0 0x7f078800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.595+0000 7f079f7fe640 1 -- 172.21.15.122:0/2781719061 reap_dead start 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.595+0000 7f079effd640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07880039b0 0x7f0788003dd0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f0790001750 tx=0x7f0788005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caa0de14e43cd55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.595+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790001370 con 0x7f07880039b0 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.595+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07900052d0 con 0x7f07880039b0 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.595+0000 7f079dffb640 1 -- 172.21.15.122:0/2781719061 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790003830 con 0x7f07880039b0 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 -- 172.21.15.122:0/2781719061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07880039b0 msgr2=0x7f0788003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07880039b0 0x7f0788003dd0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f0790001750 tx=0x7f0788005830 comp rx=0 tx=0).stop 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 -- 172.21.15.122:0/2781719061 shutdown_connections 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2781719061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07880039b0 0x7f0788003dd0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 -- 172.21.15.122:0/2781719061 >> 172.21.15.122:0/2781719061 conn(0x7f07a00d3cd0 msgr2=0x7f07a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 -- 172.21.15.122:0/2781719061 shutdown_connections 2024-03-06T07:30:42.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.596+0000 7f07a56b5640 1 -- 172.21.15.122:0/2781719061 wait complete. 2024-03-06T07:30:42.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.598+0000 7f07a56b5640 1 Processor -- start 2024-03-06T07:30:42.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.598+0000 7f07a56b5640 1 -- start start 2024-03-06T07:30:42.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.598+0000 7f07a56b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07880039b0 0x7f07a00caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07a56b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07a01f3bd0 0x7f07a00b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07a56b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 0x7f07a014ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07a56b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00b3d60 con 0x7f07a015fbc0 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07a56b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00b31e0 con 0x7f07880039b0 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07a56b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a015e3c0 con 0x7f07a01f3bd0 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 0x7f07a014ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 0x7f07a014ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51384/0 (socket says 172.21.15.122:51384) 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 -- 172.21.15.122:0/2166575767 learned_addr learned my addr 172.21.15.122:0/2166575767 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079e7fc640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07a01f3bd0 0x7f07a00b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079effd640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07880039b0 0x7f07a00caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 -- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07a01f3bd0 msgr2=0x7f07a00b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07a01f3bd0 0x7f07a00b82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 -- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07880039b0 msgr2=0x7f07a00caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07880039b0 0x7f07a00caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 -- 172.21.15.122:0/2166575767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a01f4340 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f079f7fe640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 0x7f07a014ec80 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f07a008c940 tx=0x7f079800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079800eb90 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798018070 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798013420 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.599+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a01f4620 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.600+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a015e8d0 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.601+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a004f330 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.601+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07980135c0 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.602+0000 7f07877fe640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f078c06b500 0x7f078c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.602+0000 7f079effd640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f078c06b500 0x7f078c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.602+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f079808a290 con 0x7f07a015fbc0 2024-03-06T07:30:42.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.602+0000 7f079effd640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f078c06b500 0x7f078c06d9c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0790033000 tx=0x7f0790009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.606+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07980591b0 con 0x7f07a015fbc0 2024-03-06T07:30:42.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.787+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07a015eaa0 con 0x7f07a015fbc0 2024-03-06T07:30:42.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.789+0000 7f07877fe640 1 -- 172.21.15.122:0/2166575767 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f0798058ab0 con 0x7f07a015fbc0 2024-03-06T07:30:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:42.791 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.790+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f078c06b500 msgr2=0x7f078c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.790+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f078c06b500 0x7f078c06d9c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0790033000 tx=0x7f0790009030 comp rx=0 tx=0).stop 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.790+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 msgr2=0x7f07a014ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.790+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 0x7f07a014ec80 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f07a008c940 tx=0x7f079800e530 comp rx=0 tx=0).stop 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 shutdown_connections 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f078c06b500 0x7f078c06d9c0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07a015fbc0 0x7f07a014ec80 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f07a01f3bd0 0x7f07a00b82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 --2- 172.21.15.122:0/2166575767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07880039b0 0x7f07a00caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 >> 172.21.15.122:0/2166575767 conn(0x7f07a00d3cd0 msgr2=0x7f07a00ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 shutdown_connections 2024-03-06T07:30:42.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.791+0000 7f07a56b5640 1 -- 172.21.15.122:0/2166575767 wait complete. 2024-03-06T07:30:42.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.987708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.745253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:42.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.913+0000 7faa51c2a640 1 Processor -- start 2024-03-06T07:30:42.915 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.913+0000 7faa51c2a640 1 -- start start 2024-03-06T07:30:42.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa51c2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7faa4c1552c0 0x7faa4c066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.916 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa51c2a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7faa4c0b0710 0x7faa4c0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa51c2a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4c0b6230 con 0x7faa4c0b5db0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa51c2a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4c0b2460 con 0x7faa4c1552c0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa51c2a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4c1556a0 con 0x7faa4c0b0710 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7faa4c1552c0 0x7faa4c066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7faa4c1552c0 0x7faa4c066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51396/0 (socket says 172.21.15.122:51396) 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 -- 172.21.15.122:0/661386993 learned_addr learned my addr 172.21.15.122:0/661386993 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4bfff640 1 --2- 172.21.15.122:0/661386993 >> v2:172.21.15.169:3300/0 conn(0x7faa4c0b0710 0x7faa4c0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 -- 172.21.15.122:0/661386993 >> v2:172.21.15.169:3300/0 conn(0x7faa4c0b0710 msgr2=0x7faa4c0b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 --2- 172.21.15.122:0/661386993 >> v2:172.21.15.169:3300/0 conn(0x7faa4c0b0710 0x7faa4c0b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 -- 172.21.15.122:0/661386993 >> v1:172.21.15.122:6789/0 conn(0x7faa4c0b5db0 legacy=0x7faa4c0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4b7fe640 1 -- 172.21.15.122:0/661386993 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c15bdb0 con 0x7faa4c1552c0 2024-03-06T07:30:42.917 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.914+0000 7faa4bfff640 1 --2- 172.21.15.122:0/661386993 >> v2:172.21.15.169:3300/0 conn(0x7faa4c0b0710 0x7faa4c0b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.915+0000 7faa4b7fe640 1 --2- 172.21.15.122:0/661386993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c1552c0 0x7faa4c066d00 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7faa4c072110 tx=0x7faa3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38369de93298def5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.915+0000 7faa4a7fc640 1 -- 172.21.15.122:0/661386993 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c034070 con 0x7faa4c1552c0 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.915+0000 7faa4a7fc640 1 -- 172.21.15.122:0/661386993 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c030860 con 0x7faa4c1552c0 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.915+0000 7faa4a7fc640 1 -- 172.21.15.122:0/661386993 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c030c30 con 0x7faa4c1552c0 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 -- 172.21.15.122:0/661386993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c1552c0 msgr2=0x7faa4c066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 --2- 172.21.15.122:0/661386993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c1552c0 0x7faa4c066d00 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7faa4c072110 tx=0x7faa3c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 -- 172.21.15.122:0/661386993 shutdown_connections 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 --2- 172.21.15.122:0/661386993 >> v2:172.21.15.169:3300/0 conn(0x7faa4c0b0710 0x7faa4c0b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 --2- 172.21.15.122:0/661386993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c1552c0 0x7faa4c066d00 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 -- 172.21.15.122:0/661386993 >> 172.21.15.122:0/661386993 conn(0x7faa4c0b4fe0 msgr2=0x7faa4c0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 -- 172.21.15.122:0/661386993 shutdown_connections 2024-03-06T07:30:42.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.916+0000 7faa51c2a640 1 -- 172.21.15.122:0/661386993 wait complete. 2024-03-06T07:30:42.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.918+0000 7faa51c2a640 1 Processor -- start 2024-03-06T07:30:42.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.918+0000 7faa51c2a640 1 -- start start 2024-03-06T07:30:42.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa51c2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 0x7faa4c159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa51c2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa4c0b5db0 0x7faa4c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa51c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa4c1552c0 0x7faa4c0d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa51c2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c0b20f0 con 0x7faa4c0b0710 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa51c2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c0b1e10 con 0x7faa4c0b5db0 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa51c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c06afb0 con 0x7faa4c1552c0 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 0x7faa4c159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 0x7faa4c159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51398/0 (socket says 172.21.15.122:51398) 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.919+0000 7faa4b7fe640 1 -- 172.21.15.122:0/1703554396 learned_addr learned my addr 172.21.15.122:0/1703554396 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4bfff640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa4c1552c0 0x7faa4c0d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4affd640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa4c0b5db0 0x7faa4c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4b7fe640 1 -- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa4c1552c0 msgr2=0x7faa4c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4b7fe640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa4c1552c0 0x7faa4c0d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4b7fe640 1 -- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa4c0b5db0 msgr2=0x7faa4c0d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4b7fe640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa4c0b5db0 0x7faa4c0d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4b7fe640 1 -- 172.21.15.122:0/1703554396 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c151820 con 0x7faa4c0b0710 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4affd640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa4c0b5db0 0x7faa4c0d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa4b7fe640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 0x7faa4c159f00 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7faa3c02b9e0 tx=0x7faa3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c034040 con 0x7faa4c0b0710 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c039d90 con 0x7faa4c0b0710 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c0d0d00 con 0x7faa4c0b0710 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c043610 con 0x7faa4c0b0710 2024-03-06T07:30:42.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.920+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c1e40e0 con 0x7faa4c0b0710 2024-03-06T07:30:42.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.921+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c09e4e0 con 0x7faa4c0b0710 2024-03-06T07:30:42.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.922+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa3c04a060 con 0x7faa4c0b0710 2024-03-06T07:30:42.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.922+0000 7faa33fff640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa3806b4d0 0x7faa3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.922+0000 7faa4affd640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa3806b4d0 0x7faa3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.922+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faa3c0ac1d0 con 0x7faa4c0b0710 2024-03-06T07:30:42.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.923+0000 7faa4affd640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa3806b4d0 0x7faa3806d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0cfc10 tx=0x7faa40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:42.928+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa3c03e070 con 0x7faa4c0b0710 2024-03-06T07:30:43.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.187+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa4c068ff0 con 0x7faa4c0b0710 2024-03-06T07:30:43.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.188+0000 7faa33fff640 1 -- 172.21.15.122:0/1703554396 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa3c07b0c0 con 0x7faa4c0b0710 2024-03-06T07:30:43.190 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:30:43.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa3806b4d0 msgr2=0x7faa3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa3806b4d0 0x7faa3806d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0cfc10 tx=0x7faa40009040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 msgr2=0x7faa4c159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 0x7faa4c159f00 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7faa3c02b9e0 tx=0x7faa3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 shutdown_connections 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa3806b4d0 0x7faa3806d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa4c1552c0 0x7faa4c0d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa4c0b5db0 0x7faa4c0d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 --2- 172.21.15.122:0/1703554396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa4c0b0710 0x7faa4c159f00 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 >> 172.21.15.122:0/1703554396 conn(0x7faa4c0b4fe0 msgr2=0x7faa4c15c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 shutdown_connections 2024-03-06T07:30:43.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.190+0000 7faa51c2a640 1 -- 172.21.15.122:0/1703554396 wait complete. 2024-03-06T07:30:43.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:43.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.305+0000 7f44b06a4640 1 Processor -- start 2024-03-06T07:30:43.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.305+0000 7f44b06a4640 1 -- start start 2024-03-06T07:30:43.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.306+0000 7f44b06a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.306+0000 7f44b06a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f44a80d1910 0x7f44a80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44b06a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80d2740 con 0x7f44a80d22b0 2024-03-06T07:30:43.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44b06a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a81618b0 con 0x7f44a80d1910 2024-03-06T07:30:43.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44b06a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80cf400 con 0x7f44a81614d0 2024-03-06T07:30:43.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f44a80d1910 0x7f44a80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f44a80d1910 0x7f44a80d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51404/0 (socket says 172.21.15.122:51404) 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 -- 172.21.15.122:0/606974506 learned_addr learned my addr 172.21.15.122:0/606974506 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44ae440640 1 --2- 172.21.15.122:0/606974506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 -- 172.21.15.122:0/606974506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 msgr2=0x7f44a80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 --2- 172.21.15.122:0/606974506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 -- 172.21.15.122:0/606974506 >> v1:172.21.15.122:6789/0 conn(0x7f44a80d22b0 legacy=0x7f44a80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.307+0000 7f44adc3f640 1 -- 172.21.15.122:0/606974506 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80cf6a0 con 0x7f44a80d1910 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.308+0000 7f44adc3f640 1 --2- 172.21.15.122:0/606974506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80d1d70 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4498009870 tx=0x7f449802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32233ee2075e8a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.308+0000 7f44ad43e640 1 -- 172.21.15.122:0/606974506 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498034070 con 0x7f44a80d1910 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.308+0000 7f44ad43e640 1 -- 172.21.15.122:0/606974506 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4498030840 con 0x7f44a80d1910 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.308+0000 7f44ad43e640 1 -- 172.21.15.122:0/606974506 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498030c10 con 0x7f44a80d1910 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.308+0000 7f44b06a4640 1 -- 172.21.15.122:0/606974506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 msgr2=0x7f44a80d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.308+0000 7f44b06a4640 1 --2- 172.21.15.122:0/606974506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80d1d70 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4498009870 tx=0x7f449802b980 comp rx=0 tx=0).stop 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.309+0000 7f44b06a4640 1 -- 172.21.15.122:0/606974506 shutdown_connections 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.309+0000 7f44b06a4640 1 --2- 172.21.15.122:0/606974506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80d1d70 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.309+0000 7f44b06a4640 1 --2- 172.21.15.122:0/606974506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.309+0000 7f44b06a4640 1 -- 172.21.15.122:0/606974506 >> 172.21.15.122:0/606974506 conn(0x7f44a80d3d40 msgr2=0x7f44a80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.309+0000 7f44b06a4640 1 -- 172.21.15.122:0/606974506 shutdown_connections 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.309+0000 7f44b06a4640 1 -- 172.21.15.122:0/606974506 wait complete. 2024-03-06T07:30:43.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 Processor -- start 2024-03-06T07:30:43.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 -- start start 2024-03-06T07:30:43.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44a80d22b0 0x7f44a80ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a814fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80b0910 con 0x7f44a80d1910 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80cfc60 con 0x7f44a80d22b0 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.310+0000 7f44b06a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80b1e40 con 0x7f44a81614d0 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51418/0 (socket says 172.21.15.122:51418) 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 -- 172.21.15.122:0/166215115 learned_addr learned my addr 172.21.15.122:0/166215115 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44aec41640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a814fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44adc3f640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44a80d22b0 0x7f44a80ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 -- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 msgr2=0x7f44a814fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a814fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 -- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44a80d22b0 msgr2=0x7f44a80ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44a80d22b0 0x7f44a80ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 -- 172.21.15.122:0/166215115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80b8460 con 0x7f44a80d1910 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44ae440640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80b42a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f44a400d620 tx=0x7f44a400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a400b620 con 0x7f44a80d1910 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a4019070 con 0x7f44a80d1910 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a80b8740 con 0x7f44a80d1910 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a400b9e0 con 0x7f44a80d1910 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.311+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a80b8d00 con 0x7f44a80d1910 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.312+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8068ff0 con 0x7f44a80d1910 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.313+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44a400bb80 con 0x7f44a80d1910 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.314+0000 7f449effd640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f448c06b4d0 0x7f448c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.314+0000 7f44adc3f640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f448c06b4d0 0x7f448c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.314+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f44a408a530 con 0x7f44a80d1910 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.314+0000 7f44adc3f640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f448c06b4d0 0x7f448c06d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4498036000 tx=0x7f449803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.318+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44a4059420 con 0x7f44a80d1910 2024-03-06T07:30:43.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.496+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44a80b9b10 con 0x7f44a80d1910 2024-03-06T07:30:43.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.496+0000 7f449effd640 1 -- 172.21.15.122:0/166215115 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44a4058d20 con 0x7f44a80d1910 2024-03-06T07:30:43.499 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:30:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f448c06b4d0 msgr2=0x7f448c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f448c06b4d0 0x7f448c06d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4498036000 tx=0x7f449803b040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 msgr2=0x7f44a80b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80b42a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f44a400d620 tx=0x7f44a400dae0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 shutdown_connections 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f448c06b4d0 0x7f448c06d990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44a81614d0 0x7f44a814fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44a80d22b0 0x7f44a80ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 --2- 172.21.15.122:0/166215115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44a80d1910 0x7f44a80b42a0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 >> 172.21.15.122:0/166215115 conn(0x7f44a80d3d40 msgr2=0x7f44a806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 shutdown_connections 2024-03-06T07:30:43.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.499+0000 7f44b06a4640 1 -- 172.21.15.122:0/166215115 wait complete. 2024-03-06T07:30:43.512 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.630+0000 7f13b8081640 1 Processor -- start 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.630+0000 7f13b8081640 1 -- start start 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b8081640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13b00d4860 0x7f13b00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b8081640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f13b00b0e30 0x7f13b00b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b8081640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b01618b0 con 0x7f13b01614d0 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b8081640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b00d4cf0 con 0x7f13b00d4860 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b8081640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b00cf400 con 0x7f13b00b0e30 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13b00d4860 0x7f13b00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13b00d4860 0x7f13b00d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51434/0 (socket says 172.21.15.122:51434) 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 -- 172.21.15.122:0/70942873 learned_addr learned my addr 172.21.15.122:0/70942873 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b661e640 1 --2- 172.21.15.122:0/70942873 >> v2:172.21.15.169:3300/0 conn(0x7f13b00b0e30 0x7f13b00b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 -- 172.21.15.122:0/70942873 >> v2:172.21.15.169:3300/0 conn(0x7f13b00b0e30 msgr2=0x7f13b00b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 --2- 172.21.15.122:0/70942873 >> v2:172.21.15.169:3300/0 conn(0x7f13b00b0e30 0x7f13b00b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 -- 172.21.15.122:0/70942873 >> v1:172.21.15.169:6789/0 conn(0x7f13b01614d0 legacy=0x7f13b00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b561c640 1 -- 172.21.15.122:0/70942873 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b00cf6a0 con 0x7f13b00d4860 2024-03-06T07:30:43.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.631+0000 7f13b661e640 1 --2- 172.21.15.122:0/70942873 >> v2:172.21.15.169:3300/0 conn(0x7f13b00b0e30 0x7f13b00b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:43.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b561c640 1 --2- 172.21.15.122:0/70942873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b00d4860 0x7f13b00d2020 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f13b00b2a70 tx=0x7f13a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74fec6343113e8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b4e1b640 1 -- 172.21.15.122:0/70942873 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0034070 con 0x7f13b00d4860 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b4e1b640 1 -- 172.21.15.122:0/70942873 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0030860 con 0x7f13b00d4860 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b4e1b640 1 -- 172.21.15.122:0/70942873 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0030c30 con 0x7f13b00d4860 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b8081640 1 -- 172.21.15.122:0/70942873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b00d4860 msgr2=0x7f13b00d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b8081640 1 --2- 172.21.15.122:0/70942873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b00d4860 0x7f13b00d2020 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f13b00b2a70 tx=0x7f13a002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b8081640 1 -- 172.21.15.122:0/70942873 shutdown_connections 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b8081640 1 --2- 172.21.15.122:0/70942873 >> v2:172.21.15.169:3300/0 conn(0x7f13b00b0e30 0x7f13b00b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b8081640 1 --2- 172.21.15.122:0/70942873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b00d4860 0x7f13b00d2020 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.633+0000 7f13b8081640 1 -- 172.21.15.122:0/70942873 >> 172.21.15.122:0/70942873 conn(0x7f13b00d3d40 msgr2=0x7f13b00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.634+0000 7f13b8081640 1 -- 172.21.15.122:0/70942873 shutdown_connections 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.634+0000 7f13b8081640 1 -- 172.21.15.122:0/70942873 wait complete. 2024-03-06T07:30:43.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.635+0000 7f13b8081640 1 Processor -- start 2024-03-06T07:30:43.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.635+0000 7f13b8081640 1 -- start start 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.636+0000 7f13b8081640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13b00b0e30 0x7f13b014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13b00d4860 0x7f13b00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 0x7f13b00c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00b0910 con 0x7f13b01614d0 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00cfc60 con 0x7f13b00b0e30 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00d2560 con 0x7f13b00d4860 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 0x7f13b00c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 0x7f13b00c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51436/0 (socket says 172.21.15.122:51436) 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 -- 172.21.15.122:0/4078921176 learned_addr learned my addr 172.21.15.122:0/4078921176 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b561c640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13b00d4860 0x7f13b00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b5e1d640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13b00b0e30 0x7f13b014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 -- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13b00d4860 msgr2=0x7f13b00cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13b00d4860 0x7f13b00cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 -- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13b00b0e30 msgr2=0x7f13b014f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13b00b0e30 0x7f13b014f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 -- 172.21.15.122:0/4078921176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b00c7cc0 con 0x7f13b01614d0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b661e640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 0x7f13b00c8d60 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009fa0 tx=0x7f13a800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a801a070 con 0x7f13b01614d0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8003bb0 con 0x7f13b01614d0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b00c7fa0 con 0x7f13b01614d0 2024-03-06T07:30:43.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8004030 con 0x7f13b01614d0 2024-03-06T07:30:43.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.637+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b00c8560 con 0x7f13b01614d0 2024-03-06T07:30:43.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.638+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b0068ff0 con 0x7f13b01614d0 2024-03-06T07:30:43.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.639+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13a801d070 con 0x7f13b01614d0 2024-03-06T07:30:43.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.639+0000 7f13a67fc640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f139406b4d0 0x7f139406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.639+0000 7f13b5e1d640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f139406b4d0 0x7f139406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.639+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f13a808a590 con 0x7f13b01614d0 2024-03-06T07:30:43.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.640+0000 7f13b5e1d640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f139406b4d0 0x7f139406d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f13ac000f60 tx=0x7f13ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.644+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13a8010070 con 0x7f13b01614d0 2024-03-06T07:30:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.821+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13b0069fc0 con 0x7f13b01614d0 2024-03-06T07:30:43.824 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.822+0000 7f13a67fc640 1 -- 172.21.15.122:0/4078921176 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13a80593f0 con 0x7f13b01614d0 2024-03-06T07:30:43.824 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f139406b4d0 msgr2=0x7f139406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f139406b4d0 0x7f139406d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f13ac000f60 tx=0x7f13ac009040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 msgr2=0x7f13b00c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 0x7f13b00c8d60 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009fa0 tx=0x7f13a800d560 comp rx=0 tx=0).stop 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 shutdown_connections 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f139406b4d0 0x7f139406d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13b01614d0 0x7f13b00c8d60 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13b00d4860 0x7f13b00cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 --2- 172.21.15.122:0/4078921176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13b00b0e30 0x7f13b014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 >> 172.21.15.122:0/4078921176 conn(0x7f13b00d3d40 msgr2=0x7f13b00b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 shutdown_connections 2024-03-06T07:30:43.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.824+0000 7f13b8081640 1 -- 172.21.15.122:0/4078921176 wait complete. 2024-03-06T07:30:43.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:43.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.942+0000 7fe913e81640 1 Processor -- start 2024-03-06T07:30:43.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.942+0000 7fe913e81640 1 -- start start 2024-03-06T07:30:43.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe913e81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1613f0 0x7fe90c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe913e81640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe90c1617d0 0x7fe90c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe913e81640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe90c0b5320 0x7fe90c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe913e81640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c06c0a0 con 0x7fe90c0b5320 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe913e81640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c0b5ef0 con 0x7fe90c1617d0 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe913e81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0d0b50 con 0x7fe90c1613f0 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe90c0b5320 0x7fe90c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe90c0b5320 0x7fe90c0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51446/0 (socket says 172.21.15.122:51446) 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 -- 172.21.15.122:0/4106023902 learned_addr learned my addr 172.21.15.122:0/4106023902 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe911c1d640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1613f0 0x7fe90c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 -- 172.21.15.122:0/4106023902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1613f0 msgr2=0x7fe90c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91141c640 1 --2- 172.21.15.122:0/4106023902 >> v2:172.21.15.169:3300/0 conn(0x7fe90c1617d0 0x7fe90c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1613f0 0x7fe90c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 -- 172.21.15.122:0/4106023902 >> v2:172.21.15.169:3300/0 conn(0x7fe90c1617d0 msgr2=0x7fe90c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 --2- 172.21.15.122:0/4106023902 >> v2:172.21.15.169:3300/0 conn(0x7fe90c1617d0 0x7fe90c0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.943+0000 7fe91241e640 1 -- 172.21.15.122:0/4106023902 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c0b5bf0 con 0x7fe90c0b5320 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.944+0000 7fe911c1d640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1613f0 0x7fe90c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.944+0000 7fe91141c640 1 --2- 172.21.15.122:0/4106023902 >> v2:172.21.15.169:3300/0 conn(0x7fe90c1617d0 0x7fe90c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.944+0000 7fe91241e640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c0b5320 0x7fe90c0cd4a0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0cdcb0 tx=0x7fe90402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97e44069b51b86d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.944+0000 7fe910c1b640 1 -- 172.21.15.122:0/4106023902 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90402ca70 con 0x7fe90c0b5320 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.944+0000 7fe910c1b640 1 -- 172.21.15.122:0/4106023902 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90402cc10 con 0x7fe90c0b5320 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.944+0000 7fe910c1b640 1 -- 172.21.15.122:0/4106023902 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9040343f0 con 0x7fe90c0b5320 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 -- 172.21.15.122:0/4106023902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c0b5320 msgr2=0x7fe90c0cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c0b5320 0x7fe90c0cd4a0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0cdcb0 tx=0x7fe90402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 -- 172.21.15.122:0/4106023902 shutdown_connections 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c0b5320 0x7fe90c0cd4a0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 --2- 172.21.15.122:0/4106023902 >> v2:172.21.15.169:3300/0 conn(0x7fe90c1617d0 0x7fe90c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 --2- 172.21.15.122:0/4106023902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1613f0 0x7fe90c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 -- 172.21.15.122:0/4106023902 >> 172.21.15.122:0/4106023902 conn(0x7fe90c072520 msgr2=0x7fe90c072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 -- 172.21.15.122:0/4106023902 shutdown_connections 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.945+0000 7fe913e81640 1 -- 172.21.15.122:0/4106023902 wait complete. 2024-03-06T07:30:43.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.946+0000 7fe913e81640 1 Processor -- start 2024-03-06T07:30:43.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.946+0000 7fe913e81640 1 -- start start 2024-03-06T07:30:43.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe913e81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe90c0b5320 0x7fe90c14bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe913e81640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 0x7fe90c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe913e81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1617d0 0x7fe90c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe913e81640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c06b130 con 0x7fe90c1613f0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe913e81640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c06afb0 con 0x7fe90c0b5320 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe913e81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c06b2b0 con 0x7fe90c1617d0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 0x7fe90c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 0x7fe90c0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51460/0 (socket says 172.21.15.122:51460) 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 -- 172.21.15.122:0/2934267887 learned_addr learned my addr 172.21.15.122:0/2934267887 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91241e640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1617d0 0x7fe90c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe911c1d640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe90c0b5320 0x7fe90c14bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 -- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1617d0 msgr2=0x7fe90c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1617d0 0x7fe90c0ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 -- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe90c0b5320 msgr2=0x7fe90c14bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe90c0b5320 0x7fe90c14bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.947+0000 7fe91141c640 1 -- 172.21.15.122:0/2934267887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c0b29a0 con 0x7fe90c1613f0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.948+0000 7fe91141c640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 0x7fe90c0cdfa0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00c7f0 tx=0x7fe8fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.948+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc015070 con 0x7fe90c1613f0 2024-03-06T07:30:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.948+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc007820 con 0x7fe90c1613f0 2024-03-06T07:30:43.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.948+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0c67c0 con 0x7fe90c1613f0 2024-03-06T07:30:43.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.948+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc007bd0 con 0x7fe90c1613f0 2024-03-06T07:30:43.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.948+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c14c500 con 0x7fe90c1613f0 2024-03-06T07:30:43.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.949+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c04f330 con 0x7fe90c1613f0 2024-03-06T07:30:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.950+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8fc018070 con 0x7fe90c1613f0 2024-03-06T07:30:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.950+0000 7fe9027fc640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.950+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe8fc08a060 con 0x7fe90c1613f0 2024-03-06T07:30:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.950+0000 7fe911c1d640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.951+0000 7fe911c1d640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe90800ab60 tx=0x7fe908008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:43.955+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8fc0587e0 con 0x7fe90c1613f0 2024-03-06T07:30:44.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.146+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe90c072aa0 con 0x7fe90c1613f0 2024-03-06T07:30:44.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.146+0000 7fe9027fc640 1 -- 172.21.15.122:0/2934267887 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe8fc05c880 con 0x7fe90c1613f0 2024-03-06T07:30:44.148 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:30:44.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 msgr2=0x7fe8f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe90800ab60 tx=0x7fe908008040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 msgr2=0x7fe90c0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 0x7fe90c0cdfa0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00c7f0 tx=0x7fe8fc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 shutdown_connections 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe8f006b4d0 0x7fe8f006d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe90c1617d0 0x7fe90c0ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe90c1613f0 0x7fe90c0cdfa0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 --2- 172.21.15.122:0/2934267887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe90c0b5320 0x7fe90c14bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 >> 172.21.15.122:0/2934267887 conn(0x7fe90c072520 msgr2=0x7fe90c06d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 shutdown_connections 2024-03-06T07:30:44.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.148+0000 7fe913e81640 1 -- 172.21.15.122:0/2934267887 wait complete. 2024-03-06T07:30:44.161 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:30:44.161 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:44.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f871d647640 1 Processor -- start 2024-03-06T07:30:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f871d647640 1 -- start start 2024-03-06T07:30:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f871d647640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f87180d00c0 0x7f87180d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f871d647640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180d09e0 con 0x7f87180cf850 2024-03-06T07:30:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f871d647640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871815e380 con 0x7f871806cf20 2024-03-06T07:30:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f871d647640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871806ca90 con 0x7f87180d00c0 2024-03-06T07:30:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.323+0000 7f870ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f87180cf850 0x7f87180cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57620/0 (socket says 172.21.15.122:57620) 2024-03-06T07:30:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f870ffff640 1 -- 172.21.15.122:0/3117709477 learned_addr learned my addr 172.21.15.122:0/3117709477 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:44.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f8716ffd640 1 --2- 172.21.15.122:0/3117709477 >> v2:172.21.15.169:3300/0 conn(0x7f87180d00c0 0x7f87180d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035907432 0 0) 0x7f87180d09e0 con 0x7f87180cf850 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8704003520 con 0x7f87180cf850 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437532331 0 0) 0x7f8704003520 con 0x7f87180cf850 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 >> v2:172.21.15.169:3300/0 conn(0x7f87180d00c0 msgr2=0x7f87180d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 --2- 172.21.15.122:0/3117709477 >> v2:172.21.15.169:3300/0 conn(0x7f87180d00c0 0x7f87180d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 >> v1:172.21.15.169:6789/0 conn(0x7f871806cf20 legacy=0x7f871806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180cdc10 con 0x7f87180cf850 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f8700001670 con 0x7f87180cf850 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 >> v1:172.21.15.122:6789/0 conn(0x7f87180cf850 legacy=0x7f87180cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8704004800 0x7f8704004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871806ca90 con 0x7f87040039b0 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870400c060 con 0x7f87040051c0 2024-03-06T07:30:44.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870400c260 con 0x7f8704004800 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f8716ffd640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f870ffff640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8704004800 0x7f8704004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.324+0000 7f87177fe640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f8716ffd640 1 -- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8704004800 msgr2=0x7f8704004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f8716ffd640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8704004800 0x7f8704004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f8716ffd640 1 -- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040051c0 msgr2=0x7f870400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f8716ffd640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f8716ffd640 1 -- 172.21.15.122:0/3117709477 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8704003520 con 0x7f87040039b0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f87177fe640 1 -- 172.21.15.122:0/3117709477 reap_dead start 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.325+0000 7f8716ffd640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f8708009ac0 tx=0x7f8704005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13fa2ecaccaeae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708039070 con 0x7f87040039b0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870802c6b0 con 0x7f87040039b0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f87167fc640 1 -- 172.21.15.122:0/3117709477 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708009690 con 0x7f87040039b0 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 -- 172.21.15.122:0/3117709477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87040039b0 msgr2=0x7f8704003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f8708009ac0 tx=0x7f8704005830 comp rx=0 tx=0).stop 2024-03-06T07:30:44.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 -- 172.21.15.122:0/3117709477 shutdown_connections 2024-03-06T07:30:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 --2- 172.21.15.122:0/3117709477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 -- 172.21.15.122:0/3117709477 >> 172.21.15.122:0/3117709477 conn(0x7f87180b35a0 msgr2=0x7f87180b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 -- 172.21.15.122:0/3117709477 shutdown_connections 2024-03-06T07:30:44.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.326+0000 7f871d647640 1 -- 172.21.15.122:0/3117709477 wait complete. 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.328+0000 7f871d647640 1 Processor -- start 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.328+0000 7f871d647640 1 -- start start 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040039b0 0x7f871814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 0x7f87181f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87180d04a0 0x7f87180d0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871806c720 con 0x7f8718152be0 2024-03-06T07:30:44.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871806c440 con 0x7f87040039b0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b1030 con 0x7f87180d04a0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 0x7f87181f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 0x7f87181f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51480/0 (socket says 172.21.15.122:51480) 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 -- 172.21.15.122:0/1109663272 learned_addr learned my addr 172.21.15.122:0/1109663272 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f87177fe640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87180d04a0 0x7f87180d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f8716ffd640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040039b0 0x7f871814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 -- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87180d04a0 msgr2=0x7f87180d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87180d04a0 0x7f87180d0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 -- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040039b0 msgr2=0x7f871814f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040039b0 0x7f871814f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 -- 172.21.15.122:0/1109663272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871814fc20 con 0x7f8718152be0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f8716ffd640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040039b0 0x7f871814f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f87177fe640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87180d04a0 0x7f87180d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870ffff640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 0x7f87181f6c80 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f87000072c0 tx=0x7f8700007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870001a070 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870000f960 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.329+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87180b6c30 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.330+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000fd50 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.330+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87180b71b0 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.331+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871804f330 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.331+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f870001d070 con 0x7f8718152be0 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.332+0000 7f870f7fe640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86f806b500 0x7f86f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.332+0000 7f8716ffd640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86f806b500 0x7f86f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.332+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8700089d10 con 0x7f8718152be0 2024-03-06T07:30:44.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.332+0000 7f8716ffd640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86f806b500 0x7f86f806d9c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8708036000 tx=0x7f8708030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.336+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f871814fc20 con 0x7f8718152be0 2024-03-06T07:30:44.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.524+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f871815dec0 con 0x7f8718152be0 2024-03-06T07:30:44.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.525+0000 7f870f7fe640 1 -- 172.21.15.122:0/1109663272 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f8700058c00 con 0x7f8718152be0 2024-03-06T07:30:44.527 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:44.527 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86f806b500 msgr2=0x7f86f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86f806b500 0x7f86f806d9c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8708036000 tx=0x7f8708030000 comp rx=0 tx=0).stop 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 msgr2=0x7f87181f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 0x7f87181f6c80 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f87000072c0 tx=0x7f8700007780 comp rx=0 tx=0).stop 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 shutdown_connections 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86f806b500 0x7f86f806d9c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87180d04a0 0x7f87180d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8718152be0 0x7f87181f6c80 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 --2- 172.21.15.122:0/1109663272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040039b0 0x7f871814f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 >> 172.21.15.122:0/1109663272 conn(0x7f87180b35a0 msgr2=0x7f87180b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 shutdown_connections 2024-03-06T07:30:44.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.528+0000 7f871d647640 1 -- 172.21.15.122:0/1109663272 wait complete. 2024-03-06T07:30:44.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.987708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.745253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:44.541 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:44.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 Processor -- start 2024-03-06T07:30:44.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 -- start start 2024-03-06T07:30:44.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2be80b0e30 0x7f2be80b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be81618b0 con 0x7f2be81614d0 2024-03-06T07:30:44.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80d4cf0 con 0x7f2be80d4860 2024-03-06T07:30:44.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.644+0000 7f2bed7a1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80cf400 con 0x7f2be80b0e30 2024-03-06T07:30:44.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35122/0 (socket says 172.21.15.122:35122) 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 -- 172.21.15.122:0/1039475772 learned_addr learned my addr 172.21.15.122:0/1039475772 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2be77fe640 1 --2- 172.21.15.122:0/1039475772 >> v2:172.21.15.169:3300/0 conn(0x7f2be80b0e30 0x7f2be80b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 -- 172.21.15.122:0/1039475772 >> v2:172.21.15.169:3300/0 conn(0x7f2be80b0e30 msgr2=0x7f2be80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 --2- 172.21.15.122:0/1039475772 >> v2:172.21.15.169:3300/0 conn(0x7f2be80b0e30 0x7f2be80b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 -- 172.21.15.122:0/1039475772 >> v1:172.21.15.169:6789/0 conn(0x7f2be81614d0 legacy=0x7f2be80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.645+0000 7f2bdffff640 1 -- 172.21.15.122:0/1039475772 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80cf6a0 con 0x7f2be80d4860 2024-03-06T07:30:44.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.646+0000 7f2bdffff640 1 --2- 172.21.15.122:0/1039475772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be80d2020 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f2be80b2a70 tx=0x7f2bd002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee88893ad1f0e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.646+0000 7f2be67fc640 1 -- 172.21.15.122:0/1039475772 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0034070 con 0x7f2be80d4860 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.646+0000 7f2be67fc640 1 -- 172.21.15.122:0/1039475772 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd0030860 con 0x7f2be80d4860 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.646+0000 7f2be67fc640 1 -- 172.21.15.122:0/1039475772 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0030c30 con 0x7f2be80d4860 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1039475772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 msgr2=0x7f2be80d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1039475772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be80d2020 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f2be80b2a70 tx=0x7f2bd002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1039475772 shutdown_connections 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1039475772 >> v2:172.21.15.169:3300/0 conn(0x7f2be80b0e30 0x7f2be80b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1039475772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1039475772 >> 172.21.15.122:0/1039475772 conn(0x7f2be80d3d40 msgr2=0x7f2be80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1039475772 shutdown_connections 2024-03-06T07:30:44.650 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.647+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1039475772 wait complete. 2024-03-06T07:30:44.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.648+0000 7f2bed7a1640 1 Processor -- start 2024-03-06T07:30:44.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.648+0000 7f2bed7a1640 1 -- start start 2024-03-06T07:30:44.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bed7a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2be80b0e30 0x7f2be80b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bed7a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be8159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bed7a1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2be81614d0 0x7f2be81e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bed7a1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80b0910 con 0x7f2be80d4860 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bed7a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80cfc60 con 0x7f2be80b0e30 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bed7a1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80d2560 con 0x7f2be81614d0 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be8159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be8159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35134/0 (socket says 172.21.15.122:35134) 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 -- 172.21.15.122:0/1873796835 learned_addr learned my addr 172.21.15.122:0/1873796835 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2be6ffd640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2be80b0e30 0x7f2be80b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2be77fe640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2be81614d0 0x7f2be81e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 -- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2be81614d0 msgr2=0x7f2be81e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2be81614d0 0x7f2be81e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 -- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2be80b0e30 msgr2=0x7f2be80b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2be80b0e30 0x7f2be80b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 -- 172.21.15.122:0/1873796835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80cb310 con 0x7f2be80d4860 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.649+0000 7f2bdffff640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be8159df0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f2bd002b9e0 tx=0x7f2bd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.650+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0048070 con 0x7f2be80d4860 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.650+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd0030660 con 0x7f2be80d4860 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.650+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be80cb5f0 con 0x7f2be80d4860 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.650+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0034040 con 0x7f2be80d4860 2024-03-06T07:30:44.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.650+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be80cbbb0 con 0x7f2be80d4860 2024-03-06T07:30:44.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.651+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be8068ff0 con 0x7f2be80d4860 2024-03-06T07:30:44.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.651+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bd004b070 con 0x7f2be80d4860 2024-03-06T07:30:44.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.652+0000 7f2bdf7fe640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2bc806b4d0 0x7f2bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.652+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2bd00ac2a0 con 0x7f2be80d4860 2024-03-06T07:30:44.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.652+0000 7f2be6ffd640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2bc806b4d0 0x7f2bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.652+0000 7f2be6ffd640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2bc806b4d0 0x7f2bc806d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2be815e160 tx=0x7f2bd8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.660+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bd007b100 con 0x7f2be80d4860 2024-03-06T07:30:44.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.970+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2be80cbdf0 con 0x7f2be80d4860 2024-03-06T07:30:44.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.971+0000 7f2bdf7fe640 1 -- 172.21.15.122:0/1873796835 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bd007aa00 con 0x7f2be80d4860 2024-03-06T07:30:44.973 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2bc806b4d0 msgr2=0x7f2bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2bc806b4d0 0x7f2bc806d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2be815e160 tx=0x7f2bd8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 msgr2=0x7f2be8159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be8159df0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f2bd002b9e0 tx=0x7f2bd0004730 comp rx=0 tx=0).stop 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 shutdown_connections 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2bc806b4d0 0x7f2bc806d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2be81614d0 0x7f2be81e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2be80d4860 0x7f2be8159df0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 --2- 172.21.15.122:0/1873796835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2be80b0e30 0x7f2be80b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 >> 172.21.15.122:0/1873796835 conn(0x7f2be80d3d40 msgr2=0x7f2be80cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 shutdown_connections 2024-03-06T07:30:44.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:44.973+0000 7f2bed7a1640 1 -- 172.21.15.122:0/1873796835 wait complete. 2024-03-06T07:30:44.985 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:45.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.095+0000 7f677f38c640 1 Processor -- start 2024-03-06T07:30:45.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.095+0000 7f677f38c640 1 -- start start 2024-03-06T07:30:45.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677f38c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f67780cf200 0x7f67780cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677f38c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67781600a0 con 0x7f67780d06e0 2024-03-06T07:30:45.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677f38c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780d47f0 con 0x7f677815fc30 2024-03-06T07:30:45.098 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677f38c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67780ced70 con 0x7f67780cf200 2024-03-06T07:30:45.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677d929640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f67780cf200 0x7f67780cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677d929640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f67780cf200 0x7f67780cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35136/0 (socket says 172.21.15.122:35136) 2024-03-06T07:30:45.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677d929640 1 -- 172.21.15.122:0/3796615042 learned_addr learned my addr 172.21.15.122:0/3796615042 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:45.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677d929640 1 -- 172.21.15.122:0/3796615042 >> v1:172.21.15.169:6789/0 conn(0x7f677815fc30 legacy=0x7f67780b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677d929640 1 -- 172.21.15.122:0/3796615042 >> v1:172.21.15.122:6789/0 conn(0x7f67780d06e0 legacy=0x7f67780d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.096+0000 7f677d929640 1 -- 172.21.15.122:0/3796615042 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780d1ab0 con 0x7f67780cf200 2024-03-06T07:30:45.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.097+0000 7f677d929640 1 --2- 172.21.15.122:0/3796615042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67780cf200 0x7f67780cf670 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6770009e50 tx=0x7f677002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf30be1d3562de55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.097+0000 7f676ffff640 1 -- 172.21.15.122:0/3796615042 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677002cb20 con 0x7f67780cf200 2024-03-06T07:30:45.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.097+0000 7f676ffff640 1 -- 172.21.15.122:0/3796615042 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677002ccc0 con 0x7f67780cf200 2024-03-06T07:30:45.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.097+0000 7f676ffff640 1 -- 172.21.15.122:0/3796615042 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770034610 con 0x7f67780cf200 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 -- 172.21.15.122:0/3796615042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67780cf200 msgr2=0x7f67780cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 --2- 172.21.15.122:0/3796615042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67780cf200 0x7f67780cf670 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6770009e50 tx=0x7f677002bf80 comp rx=0 tx=0).stop 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 -- 172.21.15.122:0/3796615042 shutdown_connections 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 --2- 172.21.15.122:0/3796615042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67780cf200 0x7f67780cf670 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 -- 172.21.15.122:0/3796615042 >> 172.21.15.122:0/3796615042 conn(0x7f67780d3cd0 msgr2=0x7f67780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 -- 172.21.15.122:0/3796615042 shutdown_connections 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.098+0000 7f677f38c640 1 -- 172.21.15.122:0/3796615042 wait complete. 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.099+0000 7f677f38c640 1 Processor -- start 2024-03-06T07:30:45.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.099+0000 7f677f38c640 1 -- start start 2024-03-06T07:30:45.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677f38c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67780cf200 0x7f6778159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677f38c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f67780d06e0 0x7f67780cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677f38c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 0x7f677815b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677f38c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67780ce990 con 0x7f677815fc30 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677f38c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67780d4ab0 con 0x7f67780cf200 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677f38c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67780b44d0 con 0x7f67780d06e0 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677d929640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 0x7f677815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677d929640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 0x7f677815b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35142/0 (socket says 172.21.15.122:35142) 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.100+0000 7f677d929640 1 -- 172.21.15.122:0/1560876533 learned_addr learned my addr 172.21.15.122:0/1560876533 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d128640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67780cf200 0x7f6778159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677c927640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f67780d06e0 0x7f67780cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d929640 1 -- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f67780d06e0 msgr2=0x7f67780cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d929640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f67780d06e0 0x7f67780cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d929640 1 -- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67780cf200 msgr2=0x7f6778159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d929640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67780cf200 0x7f6778159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d929640 1 -- 172.21.15.122:0/1560876533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677815c370 con 0x7f677815fc30 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d128640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67780cf200 0x7f6778159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677d929640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 0x7f677815b320 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f6770009e50 tx=0x7f6770004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770043020 con 0x7f677815fc30 2024-03-06T07:30:45.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677003e450 con 0x7f677815fc30 2024-03-06T07:30:45.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677003e820 con 0x7f677815fc30 2024-03-06T07:30:45.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677815c650 con 0x7f677815fc30 2024-03-06T07:30:45.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.101+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677815cc10 con 0x7f677815fc30 2024-03-06T07:30:45.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.103+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67780a1c80 con 0x7f677815fc30 2024-03-06T07:30:45.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.103+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6770046020 con 0x7f677815fc30 2024-03-06T07:30:45.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.103+0000 7f676dffb640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f675c06b4d0 0x7f675c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.103+0000 7f677d128640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f675c06b4d0 0x7f675c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.104+0000 7f677d128640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f675c06b4d0 0x7f675c06d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6774007fd0 tx=0x7f6774009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.106+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6770079340 con 0x7f677815fc30 2024-03-06T07:30:45.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.111+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f677004d030 con 0x7f677815fc30 2024-03-06T07:30:45.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.298+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6778068ff0 con 0x7f677815fc30 2024-03-06T07:30:45.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.298+0000 7f676dffb640 1 -- 172.21.15.122:0/1560876533 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f677007b9a0 con 0x7f677815fc30 2024-03-06T07:30:45.300 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:30:45.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f675c06b4d0 msgr2=0x7f675c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f675c06b4d0 0x7f675c06d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6774007fd0 tx=0x7f6774009040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 msgr2=0x7f677815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 0x7f677815b320 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f6770009e50 tx=0x7f6770004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 shutdown_connections 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f675c06b4d0 0x7f675c06d990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677815fc30 0x7f677815b320 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f67780d06e0 0x7f67780cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 --2- 172.21.15.122:0/1560876533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67780cf200 0x7f6778159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 >> 172.21.15.122:0/1560876533 conn(0x7f67780d3cd0 msgr2=0x7f67780b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 shutdown_connections 2024-03-06T07:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.300+0000 7f677f38c640 1 -- 172.21.15.122:0/1560876533 wait complete. 2024-03-06T07:30:45.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:45.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.418+0000 7f84aa58a640 1 Processor -- start 2024-03-06T07:30:45.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.419+0000 7f84aa58a640 1 -- start start 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.419+0000 7f84aa58a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84a415bb40 0x7f84a40b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.419+0000 7f84aa58a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a415ff90 con 0x7f84a40cfa90 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.419+0000 7f84aa58a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a4160270 con 0x7f84a415f7d0 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.419+0000 7f84aa58a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a406c910 con 0x7f84a415bb40 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.419+0000 7f84a3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84a415bb40 0x7f84a40b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.420+0000 7f84a3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f84a415bb40 0x7f84a40b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35152/0 (socket says 172.21.15.122:35152) 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.420+0000 7f84a3fff640 1 -- 172.21.15.122:0/1653363995 learned_addr learned my addr 172.21.15.122:0/1653363995 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.420+0000 7f84a3fff640 1 -- 172.21.15.122:0/1653363995 >> v1:172.21.15.169:6789/0 conn(0x7f84a415f7d0 legacy=0x7f84a406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.420+0000 7f84a3fff640 1 -- 172.21.15.122:0/1653363995 >> v1:172.21.15.122:6789/0 conn(0x7f84a40cfa90 legacy=0x7f84a40cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.420+0000 7f84a3fff640 1 -- 172.21.15.122:0/1653363995 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4160550 con 0x7f84a415bb40 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84a3fff640 1 --2- 172.21.15.122:0/1653363995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a40b4970 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f84a406ca90 tx=0x7f849402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc4b40c0300ff50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84a2ffd640 1 -- 172.21.15.122:0/1653363995 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494034070 con 0x7f84a415bb40 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84a2ffd640 1 -- 172.21.15.122:0/1653363995 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8494030840 con 0x7f84a415bb40 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84a2ffd640 1 -- 172.21.15.122:0/1653363995 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494030c10 con 0x7f84a415bb40 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84aa58a640 1 -- 172.21.15.122:0/1653363995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 msgr2=0x7f84a40b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84aa58a640 1 --2- 172.21.15.122:0/1653363995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a40b4970 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f84a406ca90 tx=0x7f849402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.421+0000 7f84aa58a640 1 -- 172.21.15.122:0/1653363995 shutdown_connections 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.422+0000 7f84aa58a640 1 --2- 172.21.15.122:0/1653363995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a40b4970 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.422+0000 7f84aa58a640 1 -- 172.21.15.122:0/1653363995 >> 172.21.15.122:0/1653363995 conn(0x7f84a406b190 msgr2=0x7f84a406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.422+0000 7f84aa58a640 1 -- 172.21.15.122:0/1653363995 shutdown_connections 2024-03-06T07:30:45.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.422+0000 7f84aa58a640 1 -- 172.21.15.122:0/1653363995 wait complete. 2024-03-06T07:30:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.423+0000 7f84aa58a640 1 Processor -- start 2024-03-06T07:30:45.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.423+0000 7f84aa58a640 1 -- start start 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a40cfa90 0x7f84a41f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a41f7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a415f7d0 0x7f84a40add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a406d790 con 0x7f84a415bb40 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a415fc30 con 0x7f84a40cfa90 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a415bf20 con 0x7f84a415f7d0 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a41f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a41f7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35160/0 (socket says 172.21.15.122:35160) 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 -- 172.21.15.122:0/745594322 learned_addr learned my addr 172.21.15.122:0/745594322 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a3fff640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a40cfa90 0x7f84a41f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a8b27640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a415f7d0 0x7f84a40add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 -- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a415f7d0 msgr2=0x7f84a40add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a415f7d0 0x7f84a40add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 -- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a40cfa90 msgr2=0x7f84a41f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a40cfa90 0x7f84a41f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 -- 172.21.15.122:0/745594322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4070bd0 con 0x7f84a415bb40 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a37fe640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a41f7960 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f849800b3f0 tx=0x7f849800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498015020 con 0x7f84a415bb40 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498011830 con 0x7f84a415bb40 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.424+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4070eb0 con 0x7f84a415bb40 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.425+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4071470 con 0x7f84a415bb40 2024-03-06T07:30:45.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.425+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498011c20 con 0x7f84a415bb40 2024-03-06T07:30:45.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.426+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a40732b0 con 0x7f84a415bb40 2024-03-06T07:30:45.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.426+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8498018020 con 0x7f84a415bb40 2024-03-06T07:30:45.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.426+0000 7f84a0ff9640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.427+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8498089d90 con 0x7f84a415bb40 2024-03-06T07:30:45.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.427+0000 7f84a3fff640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.427+0000 7f84a3fff640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8494036000 tx=0x7f849403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.434+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f849801f030 con 0x7f84a415bb40 2024-03-06T07:30:45.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.611+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84a4068ff0 con 0x7f84a415bb40 2024-03-06T07:30:45.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.612+0000 7f84a0ff9640 1 -- 172.21.15.122:0/745594322 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84980172c0 con 0x7f84a415bb40 2024-03-06T07:30:45.614 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:30:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 msgr2=0x7f849006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8494036000 tx=0x7f849403b040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 msgr2=0x7f84a41f7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a41f7960 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f849800b3f0 tx=0x7f849800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 shutdown_connections 2024-03-06T07:30:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f849006b4d0 0x7f849006d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84a415f7d0 0x7f84a40add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84a415bb40 0x7f84a41f7960 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 --2- 172.21.15.122:0/745594322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84a40cfa90 0x7f84a41f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 >> 172.21.15.122:0/745594322 conn(0x7f84a406b190 msgr2=0x7f84a40af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 shutdown_connections 2024-03-06T07:30:45.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.614+0000 7f84aa58a640 1 -- 172.21.15.122:0/745594322 wait complete. 2024-03-06T07:30:45.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:45.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.728+0000 7f64c9681640 1 Processor -- start 2024-03-06T07:30:45.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c9681640 1 -- start start 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c9681640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64c40b3240 0x7f64c40b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c9681640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40b2910 0x7f64c40d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c9681640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c415e3f0 con 0x7f64c415e010 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c9681640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c40b1f40 con 0x7f64c40b3240 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c9681640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40b1c10 con 0x7f64c40b2910 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.729+0000 7f64c27fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64c40b3240 0x7f64c40b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c27fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f64c40b3240 0x7f64c40b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35168/0 (socket says 172.21.15.122:35168) 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c27fc640 1 -- 172.21.15.122:0/1441793769 learned_addr learned my addr 172.21.15.122:0/1441793769 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c37fe640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40b2910 0x7f64c40d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c27fc640 1 -- 172.21.15.122:0/1441793769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40b2910 msgr2=0x7f64c40d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c27fc640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40b2910 0x7f64c40d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c27fc640 1 -- 172.21.15.122:0/1441793769 >> v1:172.21.15.169:6789/0 conn(0x7f64c415e010 legacy=0x7f64c40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c27fc640 1 -- 172.21.15.122:0/1441793769 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c406d1b0 con 0x7f64c40b3240 2024-03-06T07:30:45.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.730+0000 7f64c37fe640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40b2910 0x7f64c40d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c27fc640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c40b3240 0x7f64c40b23d0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009870 tx=0x7f64b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e852927568d6c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c1ffb640 1 -- 172.21.15.122:0/1441793769 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8035020 con 0x7f64c40b3240 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c1ffb640 1 -- 172.21.15.122:0/1441793769 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8031640 con 0x7f64c40b3240 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c1ffb640 1 -- 172.21.15.122:0/1441793769 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b802d040 con 0x7f64c40b3240 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 -- 172.21.15.122:0/1441793769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c40b3240 msgr2=0x7f64c40b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c40b3240 0x7f64c40b23d0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009870 tx=0x7f64b802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 -- 172.21.15.122:0/1441793769 shutdown_connections 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40b2910 0x7f64c40d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 --2- 172.21.15.122:0/1441793769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c40b3240 0x7f64c40b23d0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 -- 172.21.15.122:0/1441793769 >> 172.21.15.122:0/1441793769 conn(0x7f64c409a9f0 msgr2=0x7f64c409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 -- 172.21.15.122:0/1441793769 shutdown_connections 2024-03-06T07:30:45.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.731+0000 7f64c9681640 1 -- 172.21.15.122:0/1441793769 wait complete. 2024-03-06T07:30:45.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.733+0000 7f64c9681640 1 Processor -- start 2024-03-06T07:30:45.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.733+0000 7f64c9681640 1 -- start start 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 0x7f64c4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64c40c7f20 0x7f64c4158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40ce8a0 0x7f64c40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d2010 con 0x7f64c415e010 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c406ced0 con 0x7f64c40c7f20 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d2190 con 0x7f64c40ce8a0 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 0x7f64c4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 0x7f64c4152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35176/0 (socket says 172.21.15.122:35176) 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 -- 172.21.15.122:0/3357780556 learned_addr learned my addr 172.21.15.122:0/3357780556 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c37fe640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40ce8a0 0x7f64c40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c27fc640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64c40c7f20 0x7f64c4158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 -- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40ce8a0 msgr2=0x7f64c40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40ce8a0 0x7f64c40ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 -- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64c40c7f20 msgr2=0x7f64c4158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64c40c7f20 0x7f64c4158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 -- 172.21.15.122:0/3357780556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4159210 con 0x7f64c415e010 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c27fc640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64c40c7f20 0x7f64c4158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c2ffd640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 0x7f64c4152510 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f64b400c3c0 tx=0x7f64b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b40052d0 con 0x7f64c415e010 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b4005490 con 0x7f64c415e010 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c40b4920 con 0x7f64c415e010 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.735+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4007260 con 0x7f64c415e010 2024-03-06T07:30:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.736+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c40b4ee0 con 0x7f64c415e010 2024-03-06T07:30:45.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.736+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c404f330 con 0x7f64c415e010 2024-03-06T07:30:45.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.737+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64b4007400 con 0x7f64c415e010 2024-03-06T07:30:45.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.738+0000 7f64ab7fe640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64b006b500 0x7f64b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.738+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f64b4089700 con 0x7f64c415e010 2024-03-06T07:30:45.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.738+0000 7f64c27fc640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64b006b500 0x7f64b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.738+0000 7f64c27fc640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64b006b500 0x7f64b006d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f64b8036000 tx=0x7f64b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.742+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64b4058550 con 0x7f64c415e010 2024-03-06T07:30:45.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.948+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64c40d2980 con 0x7f64c415e010 2024-03-06T07:30:45.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.948+0000 7f64ab7fe640 1 -- 172.21.15.122:0/3357780556 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7f64b4057e50 con 0x7f64c415e010 2024-03-06T07:30:45.950 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:30:45.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64b006b500 msgr2=0x7f64b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64b006b500 0x7f64b006d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f64b8036000 tx=0x7f64b8008030 comp rx=0 tx=0).stop 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 msgr2=0x7f64c4152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 0x7f64c4152510 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f64b400c3c0 tx=0x7f64b400c880 comp rx=0 tx=0).stop 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 shutdown_connections 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64b006b500 0x7f64b006d9c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64c40ce8a0 0x7f64c40ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64c40c7f20 0x7f64c4158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 --2- 172.21.15.122:0/3357780556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64c415e010 0x7f64c4152510 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 >> 172.21.15.122:0/3357780556 conn(0x7f64c409a9f0 msgr2=0x7f64c415d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 shutdown_connections 2024-03-06T07:30:45.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:45.950+0000 7f64c9681640 1 -- 172.21.15.122:0/3357780556 wait complete. 2024-03-06T07:30:45.963 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:30:45.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:45.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[83659]: fuse finished with error 0 and tester_r 0 2024-03-06T07:30:45.990 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:52.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:30:52.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:52.091 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:52.106 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:30:52.300 INFO:teuthology.orchestra.run.smithi122.stdout:{"message":"","return_code":0} 2024-03-06T07:30:52.301 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:52.301 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:30:52.421 INFO:teuthology.orchestra.run.smithi122.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":241,"bytes":58528}},"duration":0.0058266680000000001} 2024-03-06T07:30:52.421 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 58528, "items": 241 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.005826668, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:30:52.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.539+0000 7fe256e88640 1 Processor -- start 2024-03-06T07:30:52.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.539+0000 7fe256e88640 1 -- start start 2024-03-06T07:30:52.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe256e88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe25013dc10 0x7fe2500b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe256e88640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25015f580 con 0x7fe25015bf30 2024-03-06T07:30:52.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe256e88640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25015b450 con 0x7fe25015ef60 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe256e88640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2500cf3f0 con 0x7fe25013dc10 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe254c24640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe25013dc10 0x7fe2500b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe247fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe25015bf30 0x7fe25015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58110/0 (socket says 172.21.15.122:58110) 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe247fff640 1 -- 172.21.15.122:0/2520579486 learned_addr learned my addr 172.21.15.122:0/2520579486 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe254c24640 1 -- 172.21.15.122:0/2520579486 >> v1:172.21.15.169:6789/0 conn(0x7fe25015ef60 legacy=0x7fe2500708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe254c24640 1 -- 172.21.15.122:0/2520579486 >> v1:172.21.15.122:6789/0 conn(0x7fe25015bf30 legacy=0x7fe25015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe2477fe640 1 -- 172.21.15.122:0/2520579486 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43583486 0 0) 0x7fe25015f580 con 0x7fe25015bf30 2024-03-06T07:30:52.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.540+0000 7fe254c24640 1 -- 172.21.15.122:0/2520579486 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25015b730 con 0x7fe25013dc10 2024-03-06T07:30:52.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.541+0000 7fe254c24640 1 --2- 172.21.15.122:0/2520579486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe2500b2950 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0097a0 tx=0x7fe24c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd25fa401c0c275b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.541+0000 7fe2477fe640 1 -- 172.21.15.122:0/2520579486 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0315a0 con 0x7fe25013dc10 2024-03-06T07:30:52.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.541+0000 7fe2477fe640 1 -- 172.21.15.122:0/2520579486 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c037070 con 0x7fe25013dc10 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.541+0000 7fe2477fe640 1 -- 172.21.15.122:0/2520579486 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c031960 con 0x7fe25013dc10 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 -- 172.21.15.122:0/2520579486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 msgr2=0x7fe2500b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 --2- 172.21.15.122:0/2520579486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe2500b2950 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0097a0 tx=0x7fe24c02b840 comp rx=0 tx=0).stop 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 -- 172.21.15.122:0/2520579486 shutdown_connections 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 --2- 172.21.15.122:0/2520579486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe2500b2950 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 -- 172.21.15.122:0/2520579486 >> 172.21.15.122:0/2520579486 conn(0x7fe2500cf8a0 msgr2=0x7fe2500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 -- 172.21.15.122:0/2520579486 shutdown_connections 2024-03-06T07:30:52.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.542+0000 7fe256e88640 1 -- 172.21.15.122:0/2520579486 wait complete. 2024-03-06T07:30:52.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.544+0000 7fe256e88640 1 Processor -- start 2024-03-06T07:30:52.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.544+0000 7fe256e88640 1 -- start start 2024-03-06T07:30:52.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe256e88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe250154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe256e88640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe25015bf30 0x7fe250138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe256e88640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe25015ef60 0x7fe25006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe256e88640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500b2e90 con 0x7fe25013dc10 2024-03-06T07:30:52.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe256e88640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25015b210 con 0x7fe25015bf30 2024-03-06T07:30:52.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe256e88640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500b3010 con 0x7fe25015ef60 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe250154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe250154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35202/0 (socket says 172.21.15.122:35202) 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 -- 172.21.15.122:0/2846624235 learned_addr learned my addr 172.21.15.122:0/2846624235 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe255425640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe25015ef60 0x7fe25006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe247fff640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe25015bf30 0x7fe250138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 -- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe25015ef60 msgr2=0x7fe25006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe25015ef60 0x7fe25006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 -- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe25015bf30 msgr2=0x7fe250138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe25015bf30 0x7fe250138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 -- 172.21.15.122:0/2846624235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250137300 con 0x7fe25013dc10 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe255425640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe25015ef60 0x7fe25006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe247fff640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe25015bf30 0x7fe250138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:52.549 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe254c24640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe250154830 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0097a0 tx=0x7fe24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.545+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c032840 con 0x7fe25013dc10 2024-03-06T07:30:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.546+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe250137570 con 0x7fe25013dc10 2024-03-06T07:30:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.546+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c0329e0 con 0x7fe25013dc10 2024-03-06T07:30:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.546+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c032dd0 con 0x7fe25013dc10 2024-03-06T07:30:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.546+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe250137b30 con 0x7fe25013dc10 2024-03-06T07:30:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.547+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25004f330 con 0x7fe25013dc10 2024-03-06T07:30:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.547+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe24c04d3f0 con 0x7fe25013dc10 2024-03-06T07:30:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.548+0000 7fe2457fa640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23c06b4d0 0x7fe23c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.548+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe24c02f030 con 0x7fe25013dc10 2024-03-06T07:30:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.548+0000 7fe247fff640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23c06b4d0 0x7fe23c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.548+0000 7fe247fff640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23c06b4d0 0x7fe23c06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe25013aaf0 tx=0x7fe240009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.553+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe24c040070 con 0x7fe25013dc10 2024-03-06T07:30:52.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.928+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe25006eb00 con 0x7fe25013dc10 2024-03-06T07:30:52.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.929+0000 7fe2457fa640 1 -- 172.21.15.122:0/2846624235 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v96) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe24c07af00 con 0x7fe25013dc10 2024-03-06T07:30:52.931 INFO:teuthology.orchestra.run.smithi122.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23c06b4d0 msgr2=0x7fe23c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23c06b4d0 0x7fe23c06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe25013aaf0 tx=0x7fe240009040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 msgr2=0x7fe250154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe250154830 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0097a0 tx=0x7fe24c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 shutdown_connections 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe23c06b4d0 0x7fe23c06d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe25015ef60 0x7fe25006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe25015bf30 0x7fe250138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 --2- 172.21.15.122:0/2846624235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe25013dc10 0x7fe250154830 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.930+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 >> 172.21.15.122:0/2846624235 conn(0x7fe2500cf8a0 msgr2=0x7fe250070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.931+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 shutdown_connections 2024-03-06T07:30:52.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:52.931+0000 7fe256e88640 1 -- 172.21.15.122:0/2846624235 wait complete. 2024-03-06T07:30:52.943 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:30:52.943 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:52.943 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-06T07:30:52.943 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:52.943 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:52.944 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:30:52.944 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:30:52.944 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:52.944 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:52.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:52.944 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42188sec preferred_lft 42188sec 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::409c:ffff:fed5:87ef/64 scope link 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether c6:06:9d:f8:eb:79 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c406:9dff:fef8:eb79/64 scope link 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether f2:c8:fb:47:e1:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::f0c8:fbff:fe47:e1b7/64 scope link 2024-03-06T07:30:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:52.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:52.963 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.027 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-06T07:30:53.083 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-06T07:30:53.084 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:53.084 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:53.084 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:53.084 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:53.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.084 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2024-03-06T07:30:53.084 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-06T07:30:53.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:53.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:53.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:53.085 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-06T07:30:53.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.179 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:30:53.194 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:53.194 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:53.194 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:53.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-06T07:30:53.260 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:30:53.261 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:53.261 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:53.316 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:53.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:53.317 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:53.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:30:53.439 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:53.495 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:30:53.495 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:30:53.495 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:30:53.495 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:30:53.495 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:30:53.495 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:30:53.496 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:30:53.497 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:53.498 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:53.498 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:53.498 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:53.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.498 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:30:53.551 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:30:53.551 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:30:53.594 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-06T07:30:53.622 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-06T07:30:53.678 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.680 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-06T07:30:53.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:2024-03-06T07:30:53.687+0000 7f4bc6d87480 -1 init, newargv = 0x55caa668a8c0 newargc=15 2024-03-06T07:30:53.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[85432]: starting ceph client 2024-03-06T07:30:53.696 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[85432]: starting fuse 2024-03-06T07:30:53.704 INFO:teuthology.orchestra.run.smithi122.stdout:41 2024-03-06T07:30:53.704 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:30:53.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:30:53.704 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-06T07:30:53.704 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:30:53.705 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.0")) 2024-03-06T07:30:53.706 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-06T07:30:53.827 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.0.85432.asok 2024-03-06T07:30:53.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:53.837 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85432.asok 2024-03-06T07:30:53.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.837 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85432.asok status 2024-03-06T07:30:53.975 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "0", 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "85432", 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 9660, 2024-03-06T07:30:53.976 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 9660 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1966308205 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1966308205 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-06T07:30:53.977 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.9660 192.168.0.1:0/1966308205", 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/1966308205", 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 96, 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 143, 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-06T07:30:53.978 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:30:53.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.984 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-06T07:30:53.984 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:53.998 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-06T07:30:53.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:53.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.999 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:54.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:54.066 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/original) 2024-03-06T07:30:54.129 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:30:54.129 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:54.184 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:54.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.322+0000 7f5621a15640 1 Processor -- start 2024-03-06T07:30:54.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f5621a15640 1 -- start start 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f5621a15640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f561c15fa10 0x7f561c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f5621a15640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0cebe0 con 0x7f561c0d16e0 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f5621a15640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0ceec0 con 0x7f561c15bdd0 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f5621a15640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561c0cf1a0 con 0x7f561c15fa10 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f561b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f561c0d16e0 0x7f561c0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58120/0 (socket says 172.21.15.122:58120) 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.323+0000 7f561b7fe640 1 -- 172.21.15.122:0/147399245 learned_addr learned my addr 172.21.15.122:0/147399245 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.324+0000 7f561a7fc640 1 --2- 172.21.15.122:0/147399245 >> v2:172.21.15.169:3300/0 conn(0x7f561c15fa10 0x7f561c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.324+0000 7f5619ffb640 1 -- 172.21.15.122:0/147399245 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104734480 0 0) 0x7f561c0cebe0 con 0x7f561c0d16e0 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.324+0000 7f5619ffb640 1 -- 172.21.15.122:0/147399245 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5604003520 con 0x7f561c0d16e0 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.324+0000 7f561a7fc640 1 -- 172.21.15.122:0/147399245 >> v1:172.21.15.169:6789/0 conn(0x7f561c15bdd0 legacy=0x7f561c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.324+0000 7f561a7fc640 1 -- 172.21.15.122:0/147399245 >> v1:172.21.15.122:6789/0 conn(0x7f561c0d16e0 legacy=0x7f561c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.324+0000 7f561a7fc640 1 -- 172.21.15.122:0/147399245 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0cc8e0 con 0x7f561c15fa10 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f561a7fc640 1 --2- 172.21.15.122:0/147399245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c0ce270 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5610009d90 tx=0x7f561002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a893500f1c8f8b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5619ffb640 1 -- 172.21.15.122:0/147399245 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610039070 con 0x7f561c15fa10 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5619ffb640 1 -- 172.21.15.122:0/147399245 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561002cc60 con 0x7f561c15fa10 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5619ffb640 1 -- 172.21.15.122:0/147399245 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610034660 con 0x7f561c15fa10 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5621a15640 1 -- 172.21.15.122:0/147399245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 msgr2=0x7f561c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5621a15640 1 --2- 172.21.15.122:0/147399245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c0ce270 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5610009d90 tx=0x7f561002beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5621a15640 1 -- 172.21.15.122:0/147399245 shutdown_connections 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5621a15640 1 --2- 172.21.15.122:0/147399245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c0ce270 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5621a15640 1 -- 172.21.15.122:0/147399245 >> 172.21.15.122:0/147399245 conn(0x7f561c09a9f0 msgr2=0x7f561c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.325+0000 7f5621a15640 1 -- 172.21.15.122:0/147399245 shutdown_connections 2024-03-06T07:30:54.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.326+0000 7f5621a15640 1 -- 172.21.15.122:0/147399245 wait complete. 2024-03-06T07:30:54.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.327+0000 7f5621a15640 1 Processor -- start 2024-03-06T07:30:54.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.327+0000 7f5621a15640 1 -- start start 2024-03-06T07:30:54.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f561c0d16e0 0x7f561c149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 0x7f561c0b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0ce7b0 con 0x7f561c15bdd0 2024-03-06T07:30:54.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c15c1b0 con 0x7f561c15fa10 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0ce930 con 0x7f561c0d16e0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 0x7f561c0b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 0x7f561c0b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35216/0 (socket says 172.21.15.122:35216) 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 -- 172.21.15.122:0/545298323 learned_addr learned my addr 172.21.15.122:0/545298323 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561affd640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f561c0d16e0 0x7f561c149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 -- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f561c0d16e0 msgr2=0x7f561c149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f561c0d16e0 0x7f561c149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 -- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 msgr2=0x7f561c06eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561b7fe640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c06eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 -- 172.21.15.122:0/545298323 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c15ea40 con 0x7f561c15bdd0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f561a7fc640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 0x7f561c0b6f30 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5610009d90 tx=0x7f5610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610039050 con 0x7f561c15bdd0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5610045070 con 0x7f561c15bdd0 2024-03-06T07:30:54.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561c15ed20 con 0x7f561c15bdd0 2024-03-06T07:30:54.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.329+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561003e850 con 0x7f561c15bdd0 2024-03-06T07:30:54.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.330+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c15f2e0 con 0x7f561c15bdd0 2024-03-06T07:30:54.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.331+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561c04f330 con 0x7f561c15bdd0 2024-03-06T07:30:54.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.331+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f561003e9f0 con 0x7f561c15bdd0 2024-03-06T07:30:54.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.332+0000 7f5603fff640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f560806b4d0 0x7f560806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.332+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f56100ad060 con 0x7f561c15bdd0 2024-03-06T07:30:54.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.332+0000 7f561affd640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f560806b4d0 0x7f560806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.332+0000 7f561affd640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f560806b4d0 0x7f560806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f56140015a0 tx=0x7f5614009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.336+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f561007bfe0 con 0x7f561c15bdd0 2024-03-06T07:30:54.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.530+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f561c0747a0 con 0x7f561c15bdd0 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.532+0000 7f5603fff640 1 -- 172.21.15.122:0/545298323 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f561007b8e0 con 0x7f561c15bdd0 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:54.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f560806b4d0 msgr2=0x7f560806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f560806b4d0 0x7f560806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f56140015a0 tx=0x7f5614009040 comp rx=0 tx=0).stop 2024-03-06T07:30:54.536 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 msgr2=0x7f561c0b6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 0x7f561c0b6f30 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5610009d90 tx=0x7f5610004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 shutdown_connections 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f560806b4d0 0x7f560806d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f561c15fa10 0x7f561c06eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f561c15bdd0 0x7f561c0b6f30 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 --2- 172.21.15.122:0/545298323 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f561c0d16e0 0x7f561c149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 >> 172.21.15.122:0/545298323 conn(0x7f561c09a9f0 msgr2=0x7f561c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 shutdown_connections 2024-03-06T07:30:54.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.534+0000 7f5621a15640 1 -- 172.21.15.122:0/545298323 wait complete. 2024-03-06T07:30:54.547 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.987708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.745253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:54.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.656+0000 7feecc181640 1 Processor -- start 2024-03-06T07:30:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.656+0000 7feecc181640 1 -- start start 2024-03-06T07:30:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.657+0000 7feecc181640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feec41614d0 0x7feec40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.657+0000 7feecc181640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7feec40d1910 0x7feec40d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feecc181640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec41618b0 con 0x7feec40d22e0 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feecc181640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec40cf400 con 0x7feec41614d0 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feecc181640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec40d2710 con 0x7feec40d1910 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feec41614d0 0x7feec40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feec41614d0 0x7feec40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54854/0 (socket says 172.21.15.122:54854) 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 -- 172.21.15.122:0/3621570215 learned_addr learned my addr 172.21.15.122:0/3621570215 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 -- 172.21.15.122:0/3621570215 >> v2:172.21.15.169:3300/0 conn(0x7feec40d1910 msgr2=0x7feec40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec971c640 1 --2- 172.21.15.122:0/3621570215 >> v2:172.21.15.169:3300/0 conn(0x7feec40d1910 0x7feec40d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 --2- 172.21.15.122:0/3621570215 >> v2:172.21.15.169:3300/0 conn(0x7feec40d1910 0x7feec40d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 -- 172.21.15.122:0/3621570215 >> v1:172.21.15.169:6789/0 conn(0x7feec40d22e0 legacy=0x7feec40d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec9f1d640 1 -- 172.21.15.122:0/3621570215 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec40cf6a0 con 0x7feec41614d0 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.658+0000 7feec971c640 1 --2- 172.21.15.122:0/3621570215 >> v2:172.21.15.169:3300/0 conn(0x7feec40d1910 0x7feec40d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.659+0000 7feec9f1d640 1 --2- 172.21.15.122:0/3621570215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec41614d0 0x7feec40d4860 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7feec408c470 tx=0x7feec002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9070107098d0b1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.659+0000 7feec8f1b640 1 -- 172.21.15.122:0/3621570215 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0035020 con 0x7feec41614d0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.659+0000 7feec8f1b640 1 -- 172.21.15.122:0/3621570215 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec0031640 con 0x7feec41614d0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.659+0000 7feec8f1b640 1 -- 172.21.15.122:0/3621570215 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec002d040 con 0x7feec41614d0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 -- 172.21.15.122:0/3621570215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec41614d0 msgr2=0x7feec40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 --2- 172.21.15.122:0/3621570215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec41614d0 0x7feec40d4860 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7feec408c470 tx=0x7feec002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 -- 172.21.15.122:0/3621570215 shutdown_connections 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 --2- 172.21.15.122:0/3621570215 >> v2:172.21.15.169:3300/0 conn(0x7feec40d1910 0x7feec40d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 --2- 172.21.15.122:0/3621570215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec41614d0 0x7feec40d4860 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 -- 172.21.15.122:0/3621570215 >> 172.21.15.122:0/3621570215 conn(0x7feec40d3d40 msgr2=0x7feec40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 -- 172.21.15.122:0/3621570215 shutdown_connections 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.660+0000 7feecc181640 1 -- 172.21.15.122:0/3621570215 wait complete. 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.661+0000 7feecc181640 1 Processor -- start 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.661+0000 7feecc181640 1 -- start start 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feecc181640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 0x7feec40c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feecc181640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feec40d22e0 0x7feec40b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feecc181640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feec41614d0 0x7feec40b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feecc181640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec40b0910 con 0x7feec40d1910 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feecc181640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec40cfc60 con 0x7feec41614d0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feecc181640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec40b1e40 con 0x7feec40d22e0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 0x7feec40c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 0x7feec40c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54868/0 (socket says 172.21.15.122:54868) 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 -- 172.21.15.122:0/3962863710 learned_addr learned my addr 172.21.15.122:0/3962863710 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec971c640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feec40d22e0 0x7feec40b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feeca71e640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feec41614d0 0x7feec40b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 -- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feec40d22e0 msgr2=0x7feec40b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feec40d22e0 0x7feec40b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 -- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feec41614d0 msgr2=0x7feec40b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feec41614d0 0x7feec40b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 -- 172.21.15.122:0/3962863710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec40ca200 con 0x7feec40d1910 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.662+0000 7feec9f1d640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 0x7feec40c92d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7feec002b930 tx=0x7feec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.663+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0040070 con 0x7feec40d1910 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.663+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec0039c10 con 0x7feec40d1910 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.663+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec40c9c40 con 0x7feec40d1910 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.663+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0035040 con 0x7feec40d1910 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.663+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec40cb780 con 0x7feec40d1910 2024-03-06T07:30:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.664+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec40d05b0 con 0x7feec40d1910 2024-03-06T07:30:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.664+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feec0038020 con 0x7feec40d1910 2024-03-06T07:30:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.664+0000 7feeba7fc640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feea806b4d0 0x7feea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.665+0000 7feec971c640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feea806b4d0 0x7feea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.665+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feec00abd70 con 0x7feec40d1910 2024-03-06T07:30:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.665+0000 7feec971c640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feea806b4d0 0x7feea806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feec415db60 tx=0x7feeb4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.673+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feec40d05b0 con 0x7feec40d1910 2024-03-06T07:30:54.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.896+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feec4068ff0 con 0x7feec40d1910 2024-03-06T07:30:54.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.897+0000 7feeba7fc640 1 -- 172.21.15.122:0/3962863710 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7feec0031610 con 0x7feec40d1910 2024-03-06T07:30:54.899 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feea806b4d0 msgr2=0x7feea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feea806b4d0 0x7feea806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feec415db60 tx=0x7feeb4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 msgr2=0x7feec40c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 0x7feec40c92d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7feec002b930 tx=0x7feec00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 shutdown_connections 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feea806b4d0 0x7feea806d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feec41614d0 0x7feec40b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feec40d22e0 0x7feec40b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 --2- 172.21.15.122:0/3962863710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec40d1910 0x7feec40c92d0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 >> 172.21.15.122:0/3962863710 conn(0x7feec40d3d40 msgr2=0x7feec40cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 shutdown_connections 2024-03-06T07:30:54.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:54.899+0000 7feecc181640 1 -- 172.21.15.122:0/3962863710 wait complete. 2024-03-06T07:30:54.912 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:55.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.038+0000 7f95f3765640 1 Processor -- start 2024-03-06T07:30:55.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.038+0000 7f95f3765640 1 -- start start 2024-03-06T07:30:55.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f3765640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95ec15def0 0x7f95ec0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f3765640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0cea50 0x7f95ec0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f3765640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec06d030 con 0x7f95ec06cab0 2024-03-06T07:30:55.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f3765640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0cf1c0 con 0x7f95ec15def0 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f3765640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0cc760 con 0x7f95ec0cea50 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95ec15def0 0x7f95ec0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95ec15def0 0x7f95ec0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54878/0 (socket says 172.21.15.122:54878) 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 -- 172.21.15.122:0/3339992305 learned_addr learned my addr 172.21.15.122:0/3339992305 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f0d00640 1 --1- 172.21.15.122:0/3339992305 >> v1:172.21.15.122:6789/0 conn(0x7f95ec06cab0 0x7f95ec0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45570/0 (socket says 172.21.15.122:45570) 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1d02640 1 --2- 172.21.15.122:0/3339992305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0cea50 0x7f95ec0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 -- 172.21.15.122:0/3339992305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0cea50 msgr2=0x7f95ec0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 --2- 172.21.15.122:0/3339992305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0cea50 0x7f95ec0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 -- 172.21.15.122:0/3339992305 >> v1:172.21.15.122:6789/0 conn(0x7f95ec06cab0 legacy=0x7f95ec0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95e3fff640 1 -- 172.21.15.122:0/3339992305 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564569921 0 0) 0x7f95ec06d030 con 0x7f95ec06cab0 2024-03-06T07:30:55.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.039+0000 7f95f1501640 1 -- 172.21.15.122:0/3339992305 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec06d310 con 0x7f95ec15def0 2024-03-06T07:30:55.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.040+0000 7f95f1501640 1 --2- 172.21.15.122:0/3339992305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15def0 0x7f95ec0b41b0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009870 tx=0x7f95e802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34f5ddc51fc70e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.040+0000 7f95e3fff640 1 -- 172.21.15.122:0/3339992305 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e802f840 con 0x7f95ec15def0 2024-03-06T07:30:55.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95e3fff640 1 -- 172.21.15.122:0/3339992305 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e802f9e0 con 0x7f95ec15def0 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95e3fff640 1 -- 172.21.15.122:0/3339992305 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e802fdb0 con 0x7f95ec15def0 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 -- 172.21.15.122:0/3339992305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15def0 msgr2=0x7f95ec0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 --2- 172.21.15.122:0/3339992305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15def0 0x7f95ec0b41b0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009870 tx=0x7f95e802ba20 comp rx=0 tx=0).stop 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 -- 172.21.15.122:0/3339992305 shutdown_connections 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 --2- 172.21.15.122:0/3339992305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec0cea50 0x7f95ec0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 --2- 172.21.15.122:0/3339992305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec15def0 0x7f95ec0b41b0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 -- 172.21.15.122:0/3339992305 >> 172.21.15.122:0/3339992305 conn(0x7f95ec0b3690 msgr2=0x7f95ec0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 -- 172.21.15.122:0/3339992305 shutdown_connections 2024-03-06T07:30:55.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.041+0000 7f95f3765640 1 -- 172.21.15.122:0/3339992305 wait complete. 2024-03-06T07:30:55.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.042+0000 7f95f3765640 1 Processor -- start 2024-03-06T07:30:55.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.042+0000 7f95f3765640 1 -- start start 2024-03-06T07:30:55.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f3765640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec06cab0 0x7f95ec0b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f3765640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 0x7f95ec15d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f3765640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec15def0 0x7f95ec0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f3765640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec15e2d0 con 0x7f95ec0cea50 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f3765640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0cc3f0 con 0x7f95ec06cab0 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f3765640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec15e450 con 0x7f95ec15def0 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f0d00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 0x7f95ec15d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f0d00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 0x7f95ec15d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54892/0 (socket says 172.21.15.122:54892) 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f0d00640 1 -- 172.21.15.122:0/3085500522 learned_addr learned my addr 172.21.15.122:0/3085500522 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f1d02640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec15def0 0x7f95ec0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f1501640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec06cab0 0x7f95ec0b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.043+0000 7f95f0d00640 1 -- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec15def0 msgr2=0x7f95ec0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f0d00640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec15def0 0x7f95ec0c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f0d00640 1 -- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec06cab0 msgr2=0x7f95ec0b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f0d00640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec06cab0 0x7f95ec0b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f0d00640 1 -- 172.21.15.122:0/3085500522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec15d550 con 0x7f95ec0cea50 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f1501640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec06cab0 0x7f95ec0b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f0d00640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 0x7f95ec15d010 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00c4d0 tx=0x7f95dc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc018070 con 0x7f95ec0cea50 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc007500 con 0x7f95ec0cea50 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec0b4c60 con 0x7f95ec0cea50 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc004a20 con 0x7f95ec0cea50 2024-03-06T07:30:55.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.044+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0b5220 con 0x7f95ec0cea50 2024-03-06T07:30:55.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.045+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95dc01b070 con 0x7f95ec0cea50 2024-03-06T07:30:55.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.045+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec09fcf0 con 0x7f95ec0cea50 2024-03-06T07:30:55.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.046+0000 7f95e1ffb640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95d006b500 0x7f95d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.046+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f95dc0894c0 con 0x7f95ec0cea50 2024-03-06T07:30:55.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.046+0000 7f95f1501640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95d006b500 0x7f95d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.046+0000 7f95f1501640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95d006b500 0x7f95d006d9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f95e8036000 tx=0x7f95e8039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.054+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95ec15d550 con 0x7f95ec0cea50 2024-03-06T07:30:55.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.275+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95ec068ff0 con 0x7f95ec0cea50 2024-03-06T07:30:55.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.275+0000 7f95e1ffb640 1 -- 172.21.15.122:0/3085500522 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95dc0583b0 con 0x7f95ec0cea50 2024-03-06T07:30:55.277 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:30:55.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95d006b500 msgr2=0x7f95d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95d006b500 0x7f95d006d9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f95e8036000 tx=0x7f95e8039040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 msgr2=0x7f95ec15d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 0x7f95ec15d010 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00c4d0 tx=0x7f95dc00c990 comp rx=0 tx=0).stop 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 shutdown_connections 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95d006b500 0x7f95d006d9c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95ec15def0 0x7f95ec0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95ec0cea50 0x7f95ec15d010 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 --2- 172.21.15.122:0/3085500522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95ec06cab0 0x7f95ec0b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 >> 172.21.15.122:0/3085500522 conn(0x7f95ec0b3690 msgr2=0x7f95ec0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 shutdown_connections 2024-03-06T07:30:55.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.277+0000 7f95f3765640 1 -- 172.21.15.122:0/3085500522 wait complete. 2024-03-06T07:30:55.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:55.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:55.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:55.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.394+0000 7f6702f3e640 1 Processor -- start 2024-03-06T07:30:55.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.394+0000 7f6702f3e640 1 -- start start 2024-03-06T07:30:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6702f3e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f66fc06b9a0 0x7f66fc0ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6702f3e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc072e90 con 0x7f66fc0b5e20 2024-03-06T07:30:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6702f3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc073170 con 0x7f66fc06bd80 2024-03-06T07:30:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6702f3e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc1619c0 con 0x7f66fc06b9a0 2024-03-06T07:30:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6700cda640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f66fc06b9a0 0x7f66fc0ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f66fbfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f66fc0b5e20 0x7f66fc072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45580/0 (socket says 172.21.15.122:45580) 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f66fbfff640 1 -- 172.21.15.122:0/4248015787 learned_addr learned my addr 172.21.15.122:0/4248015787 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6700cda640 1 -- 172.21.15.122:0/4248015787 >> v1:172.21.15.169:6789/0 conn(0x7f66fc06bd80 legacy=0x7f66fc06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6700cda640 1 -- 172.21.15.122:0/4248015787 >> v1:172.21.15.122:6789/0 conn(0x7f66fc0b5e20 legacy=0x7f66fc072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f66fb7fe640 1 -- 172.21.15.122:0/4248015787 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193328237 0 0) 0x7f66fc072e90 con 0x7f66fc0b5e20 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.396+0000 7f6700cda640 1 -- 172.21.15.122:0/4248015787 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc073450 con 0x7f66fc06b9a0 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.397+0000 7f6700cda640 1 --2- 172.21.15.122:0/4248015787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0ad160 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097a0 tx=0x7f66f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9bf6212009be23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.397+0000 7f66fb7fe640 1 -- 172.21.15.122:0/4248015787 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f00315a0 con 0x7f66fc06b9a0 2024-03-06T07:30:55.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.397+0000 7f66fb7fe640 1 -- 172.21.15.122:0/4248015787 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0037070 con 0x7f66fc06b9a0 2024-03-06T07:30:55.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.397+0000 7f66fb7fe640 1 -- 172.21.15.122:0/4248015787 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0031960 con 0x7f66fc06b9a0 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 -- 172.21.15.122:0/4248015787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06b9a0 msgr2=0x7f66fc0ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 --2- 172.21.15.122:0/4248015787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0ad160 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097a0 tx=0x7f66f002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 -- 172.21.15.122:0/4248015787 shutdown_connections 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 --2- 172.21.15.122:0/4248015787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0ad160 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 -- 172.21.15.122:0/4248015787 >> 172.21.15.122:0/4248015787 conn(0x7f66fc0b4fe0 msgr2=0x7f66fc0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 -- 172.21.15.122:0/4248015787 shutdown_connections 2024-03-06T07:30:55.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.398+0000 7f6702f3e640 1 -- 172.21.15.122:0/4248015787 wait complete. 2024-03-06T07:30:55.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.400+0000 7f6702f3e640 1 Processor -- start 2024-03-06T07:30:55.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.400+0000 7f6702f3e640 1 -- start start 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f6702f3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f6702f3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 0x7f66fc14fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f6702f3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f66fc0b5e20 0x7f66fc0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f6702f3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0ad6a0 con 0x7f66fc06bd80 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f6702f3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0cfed0 con 0x7f66fc06b9a0 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f6702f3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0cf520 con 0x7f66fc0b5e20 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.401+0000 7f66fbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 0x7f66fc14fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 0x7f66fc14fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54906/0 (socket says 172.21.15.122:54906) 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 -- 172.21.15.122:0/866083062 learned_addr learned my addr 172.21.15.122:0/866083062 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f6700cda640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f67014db640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f66fc0b5e20 0x7f66fc0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 -- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f66fc0b5e20 msgr2=0x7f66fc0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f66fc0b5e20 0x7f66fc0cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 -- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66fc06b9a0 msgr2=0x7f66fc0cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 -- 172.21.15.122:0/866083062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc14ffa0 con 0x7f66fc06bd80 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f67014db640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f66fc0b5e20 0x7f66fc0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66fbfff640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 0x7f66fc14fa60 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f66e800c4b0 tx=0x7f66e800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8018070 con 0x7f66fc06bd80 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e8007500 con 0x7f66fc06bd80 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8004a20 con 0x7f66fc06bd80 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc0c95a0 con 0x7f66fc06bd80 2024-03-06T07:30:55.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.402+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc0c9e40 con 0x7f66fc06bd80 2024-03-06T07:30:55.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.404+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc0c9990 con 0x7f66fc06bd80 2024-03-06T07:30:55.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.404+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66e801b070 con 0x7f66fc06bd80 2024-03-06T07:30:55.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.405+0000 7f66f97fa640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f66e006b4d0 0x7f66e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.405+0000 7f6700cda640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f66e006b4d0 0x7f66e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.405+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f66e8089410 con 0x7f66fc06bd80 2024-03-06T07:30:55.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.405+0000 7f6700cda640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f66e006b4d0 0x7f66e006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f66f0036000 tx=0x7f66f003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.409+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66e8057d70 con 0x7f66fc06bd80 2024-03-06T07:30:55.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.588+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66fc09e4e0 con 0x7f66fc06bd80 2024-03-06T07:30:55.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.589+0000 7f66f97fa640 1 -- 172.21.15.122:0/866083062 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f66e805be10 con 0x7f66fc06bd80 2024-03-06T07:30:55.591 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f66e006b4d0 msgr2=0x7f66e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f66e006b4d0 0x7f66e006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f66f0036000 tx=0x7f66f003d040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 msgr2=0x7f66fc14fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 0x7f66fc14fa60 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f66e800c4b0 tx=0x7f66e800c970 comp rx=0 tx=0).stop 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 shutdown_connections 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f66e006b4d0 0x7f66e006d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f66fc0b5e20 0x7f66fc0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66fc06bd80 0x7f66fc14fa60 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 --2- 172.21.15.122:0/866083062 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66fc06b9a0 0x7f66fc0cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 >> 172.21.15.122:0/866083062 conn(0x7f66fc0b4fe0 msgr2=0x7f66fc06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 shutdown_connections 2024-03-06T07:30:55.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.592+0000 7f6702f3e640 1 -- 172.21.15.122:0/866083062 wait complete. 2024-03-06T07:30:55.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:55.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.745+0000 7f70ea237640 1 Processor -- start 2024-03-06T07:30:55.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.745+0000 7f70ea237640 1 -- start start 2024-03-06T07:30:55.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70ea237640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f70e40b4460 0x7f70e40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70ea237640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f70e4162970 0x7f70e40b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70ea237640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e40d22c0 con 0x7f70e4161cf0 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70ea237640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e4161b00 con 0x7f70e4162970 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70ea237640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e40cecc0 con 0x7f70e40b4460 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70dbfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f70e4162970 0x7f70e40b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70e3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f70e4161cf0 0x7f70e40cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45596/0 (socket says 172.21.15.122:45596) 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70e3fff640 1 -- 172.21.15.122:0/183017734 learned_addr learned my addr 172.21.15.122:0/183017734 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70e37fe640 1 --2- 172.21.15.122:0/183017734 >> v2:172.21.15.169:3300/0 conn(0x7f70e40b4460 0x7f70e40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70dbfff640 1 -- 172.21.15.122:0/183017734 >> v2:172.21.15.169:3300/0 conn(0x7f70e40b4460 msgr2=0x7f70e40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70dbfff640 1 --2- 172.21.15.122:0/183017734 >> v2:172.21.15.169:3300/0 conn(0x7f70e40b4460 0x7f70e40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70dbfff640 1 -- 172.21.15.122:0/183017734 >> v1:172.21.15.122:6789/0 conn(0x7f70e4161cf0 legacy=0x7f70e40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.746+0000 7f70dbfff640 1 -- 172.21.15.122:0/183017734 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e40d25a0 con 0x7f70e4162970 2024-03-06T07:30:55.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.747+0000 7f70dbfff640 1 --2- 172.21.15.122:0/183017734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4162970 0x7f70e40b3580 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f70e408ddd0 tx=0x7f70cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbaafdf1ff79bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.747+0000 7f70e2ffd640 1 -- 172.21.15.122:0/183017734 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc035020 con 0x7f70e4162970 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.747+0000 7f70e2ffd640 1 -- 172.21.15.122:0/183017734 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc031830 con 0x7f70e4162970 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.747+0000 7f70e2ffd640 1 -- 172.21.15.122:0/183017734 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc031c00 con 0x7f70e4162970 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 -- 172.21.15.122:0/183017734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4162970 msgr2=0x7f70e40b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 --2- 172.21.15.122:0/183017734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4162970 0x7f70e40b3580 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f70e408ddd0 tx=0x7f70cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 -- 172.21.15.122:0/183017734 shutdown_connections 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 --2- 172.21.15.122:0/183017734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4162970 0x7f70e40b3580 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 --2- 172.21.15.122:0/183017734 >> v2:172.21.15.169:3300/0 conn(0x7f70e40b4460 0x7f70e40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 -- 172.21.15.122:0/183017734 >> 172.21.15.122:0/183017734 conn(0x7f70e40d3ce0 msgr2=0x7f70e40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 -- 172.21.15.122:0/183017734 shutdown_connections 2024-03-06T07:30:55.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.748+0000 7f70ea237640 1 -- 172.21.15.122:0/183017734 wait complete. 2024-03-06T07:30:55.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.750+0000 7f70ea237640 1 Processor -- start 2024-03-06T07:30:55.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.750+0000 7f70ea237640 1 -- start start 2024-03-06T07:30:55.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70ea237640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70e40b4460 0x7f70e40b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70ea237640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 0x7f70e40cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70ea237640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e4162970 0x7f70e40ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70ea237640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40ce950 con 0x7f70e4161cf0 2024-03-06T07:30:55.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70ea237640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40d2b60 con 0x7f70e4162970 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70ea237640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40b3ac0 con 0x7f70e40b4460 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 0x7f70e40cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 0x7f70e40cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54916/0 (socket says 172.21.15.122:54916) 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 -- 172.21.15.122:0/762924165 learned_addr learned my addr 172.21.15.122:0/762924165 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70e37fe640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70e40b4460 0x7f70e40b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70e3fff640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e4162970 0x7f70e40ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 -- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70e40b4460 msgr2=0x7f70e40b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70e40b4460 0x7f70e40b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 -- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e4162970 msgr2=0x7f70e40ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e4162970 0x7f70e40ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70dbfff640 1 -- 172.21.15.122:0/762924165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e40cc840 con 0x7f70e4161cf0 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70e3fff640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e4162970 0x7f70e40ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.751+0000 7f70e37fe640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70e40b4460 0x7f70e40b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.752+0000 7f70dbfff640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 0x7f70e40cc140 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009890 tx=0x7f70cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.752+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc041070 con 0x7f70e4161cf0 2024-03-06T07:30:55.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.752+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc03aa30 con 0x7f70e4161cf0 2024-03-06T07:30:55.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.752+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e40ccb20 con 0x7f70e4161cf0 2024-03-06T07:30:55.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.752+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc035040 con 0x7f70e4161cf0 2024-03-06T07:30:55.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.752+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e40b6be0 con 0x7f70e4161cf0 2024-03-06T07:30:55.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.753+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e404f330 con 0x7f70e4161cf0 2024-03-06T07:30:55.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.753+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70cc038020 con 0x7f70e4161cf0 2024-03-06T07:30:55.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.754+0000 7f70e0ff9640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70c406b4d0 0x7f70c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.754+0000 7f70e37fe640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70c406b4d0 0x7f70c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.754+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f70cc0abca0 con 0x7f70e4161cf0 2024-03-06T07:30:55.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.754+0000 7f70e37fe640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70c406b4d0 0x7f70c406d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f70d4006fd0 tx=0x7f70d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.761+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70cc07aa70 con 0x7f70e4161cf0 2024-03-06T07:30:55.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.959+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f70e406b180 con 0x7f70e4161cf0 2024-03-06T07:30:55.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.959+0000 7f70e0ff9640 1 -- 172.21.15.122:0/762924165 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7f70cc07a370 con 0x7f70e4161cf0 2024-03-06T07:30:55.961 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70c406b4d0 msgr2=0x7f70c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70c406b4d0 0x7f70c406d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f70d4006fd0 tx=0x7f70d4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 msgr2=0x7f70e40cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 0x7f70e40cc140 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009890 tx=0x7f70cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 shutdown_connections 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f70c406b4d0 0x7f70c406d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e4162970 0x7f70e40ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70e4161cf0 0x7f70e40cc140 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 --2- 172.21.15.122:0/762924165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f70e40b4460 0x7f70e40b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 >> 172.21.15.122:0/762924165 conn(0x7f70e40d3ce0 msgr2=0x7f70e41612e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 shutdown_connections 2024-03-06T07:30:55.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:55.961+0000 7f70ea237640 1 -- 172.21.15.122:0/762924165 wait complete. 2024-03-06T07:30:55.974 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:30:55.974 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:55.989 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.128+0000 7f3503cb1640 1 Processor -- start 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.128+0000 7f3503cb1640 1 -- start start 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3503cb1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f34fc15e090 0x7f34fc0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3503cb1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f34fc0d2920 0x7f34fc0b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3503cb1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34fc15e470 con 0x7f34fc0b3b20 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3503cb1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34fc0d18f0 con 0x7f34fc15e090 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3503cb1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34fc0d1b90 con 0x7f34fc0d2920 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3501a4d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f34fc15e090 0x7f34fc0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3501a4d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f34fc15e090 0x7f34fc0d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54932/0 (socket says 172.21.15.122:54932) 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f350224e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f34fc0b3b20 0x7f34fc0d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45610/0 (socket says 172.21.15.122:45610) 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f3501a4d640 1 -- 172.21.15.122:0/1373058145 learned_addr learned my addr 172.21.15.122:0/1373058145 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.129+0000 7f350124c640 1 --2- 172.21.15.122:0/1373058145 >> v2:172.21.15.169:3300/0 conn(0x7f34fc0d2920 0x7f34fc0b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3501a4d640 1 -- 172.21.15.122:0/1373058145 >> v2:172.21.15.169:3300/0 conn(0x7f34fc0d2920 msgr2=0x7f34fc0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3501a4d640 1 --2- 172.21.15.122:0/1373058145 >> v2:172.21.15.169:3300/0 conn(0x7f34fc0d2920 0x7f34fc0b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3501a4d640 1 -- 172.21.15.122:0/1373058145 >> v1:172.21.15.122:6789/0 conn(0x7f34fc0b3b20 legacy=0x7f34fc0d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3501a4d640 1 -- 172.21.15.122:0/1373058145 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0ceb50 con 0x7f34fc15e090 2024-03-06T07:30:56.133 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3500a4b640 1 -- 172.21.15.122:0/1373058145 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535957146 0 0) 0x7f34fc15e470 con 0x7f34fc0b3b20 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3501a4d640 1 --2- 172.21.15.122:0/1373058145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc15e090 0x7f34fc0d47a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0d1700 tx=0x7f34f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7edc75c97c36437 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3500a4b640 1 -- 172.21.15.122:0/1373058145 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8034070 con 0x7f34fc15e090 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3500a4b640 1 -- 172.21.15.122:0/1373058145 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8030860 con 0x7f34fc15e090 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.130+0000 7f3500a4b640 1 -- 172.21.15.122:0/1373058145 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8030c30 con 0x7f34fc15e090 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 -- 172.21.15.122:0/1373058145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc15e090 msgr2=0x7f34fc0d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1373058145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc15e090 0x7f34fc0d47a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0d1700 tx=0x7f34f802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 -- 172.21.15.122:0/1373058145 shutdown_connections 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1373058145 >> v2:172.21.15.169:3300/0 conn(0x7f34fc0d2920 0x7f34fc0b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1373058145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc15e090 0x7f34fc0d47a0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 -- 172.21.15.122:0/1373058145 >> 172.21.15.122:0/1373058145 conn(0x7f34fc0d3c80 msgr2=0x7f34fc0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 -- 172.21.15.122:0/1373058145 shutdown_connections 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.131+0000 7f3503cb1640 1 -- 172.21.15.122:0/1373058145 wait complete. 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 Processor -- start 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 -- start start 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34fc0d2920 0x7f34fc15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34fc15e090 0x7f34fc0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 0x7f34fc0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0ce800 con 0x7f34fc0b8f60 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0b2df0 con 0x7f34fc15e090 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f3503cb1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0d0d60 con 0x7f34fc0d2920 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f350224e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 0x7f34fc0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f350224e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 0x7f34fc0b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54946/0 (socket says 172.21.15.122:54946) 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.133+0000 7f350224e640 1 -- 172.21.15.122:0/1098669214 learned_addr learned my addr 172.21.15.122:0/1098669214 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f3501a4d640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34fc0d2920 0x7f34fc15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350124c640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34fc15e090 0x7f34fc0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350224e640 1 -- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34fc0d2920 msgr2=0x7f34fc15a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350224e640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34fc0d2920 0x7f34fc15a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350224e640 1 -- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34fc15e090 msgr2=0x7f34fc0b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350224e640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34fc15e090 0x7f34fc0b8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350224e640 1 -- 172.21.15.122:0/1098669214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0b94c0 con 0x7f34fc0b8f60 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350124c640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34fc15e090 0x7f34fc0b8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f350224e640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 0x7f34fc0b68a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f34f400c780 tx=0x7f34f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f4018070 con 0x7f34fc0b8f60 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f400a640 con 0x7f34fc0b8f60 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34fc15eeb0 con 0x7f34fc0b8f60 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f400aab0 con 0x7f34fc0b8f60 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.134+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc15f470 con 0x7f34fc0b8f60 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.135+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34fc15af30 con 0x7f34fc0b8f60 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.136+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34f401b070 con 0x7f34fc0b8f60 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.137+0000 7f34ee7fc640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e006b4d0 0x7f34e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.137+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f34f40897b0 con 0x7f34fc0b8f60 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.137+0000 7f3501a4d640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e006b4d0 0x7f34e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.137+0000 7f3501a4d640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e006b4d0 0x7f34e006d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f34f8036000 tx=0x7f34f803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.141+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34fc15af30 con 0x7f34fc0b8f60 2024-03-06T07:30:56.320 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.319+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34fc04f330 con 0x7f34fc0b8f60 2024-03-06T07:30:56.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.320+0000 7f34ee7fc640 1 -- 172.21.15.122:0/1098669214 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f34f40586a0 con 0x7f34fc0b8f60 2024-03-06T07:30:56.323 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:30:56.323 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:56.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e006b4d0 msgr2=0x7f34e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e006b4d0 0x7f34e006d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f34f8036000 tx=0x7f34f803b040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 msgr2=0x7f34fc0b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 0x7f34fc0b68a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f34f400c780 tx=0x7f34f400cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 shutdown_connections 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f34e006b4d0 0x7f34e006d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34fc0b8f60 0x7f34fc0b68a0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34fc15e090 0x7f34fc0b8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 --2- 172.21.15.122:0/1098669214 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34fc0d2920 0x7f34fc15a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 >> 172.21.15.122:0/1098669214 conn(0x7f34fc0d3c80 msgr2=0x7f34fc06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 shutdown_connections 2024-03-06T07:30:56.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.323+0000 7f3503cb1640 1 -- 172.21.15.122:0/1098669214 wait complete. 2024-03-06T07:30:56.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.987708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.745253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:56.338 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:56.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 Processor -- start 2024-03-06T07:30:56.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 -- start start 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc6600b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc66006cab0 0x7fc6600ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006d030 con 0x7fc6600ce880 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600cecf0 con 0x7fc66006cab0 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.441+0000 7fc665ca4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600cf1c0 con 0x7fc66015def0 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc66006cab0 0x7fc6600ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc66006cab0 0x7fc6600ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54948/0 (socket says 172.21.15.122:54948) 2024-03-06T07:30:56.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 -- 172.21.15.122:0/2782072788 learned_addr learned my addr 172.21.15.122:0/2782072788 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:56.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65f7fe640 1 --2- 172.21.15.122:0/2782072788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc6600b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 -- 172.21.15.122:0/2782072788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 msgr2=0x7fc6600b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 --2- 172.21.15.122:0/2782072788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc6600b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 -- 172.21.15.122:0/2782072788 >> v1:172.21.15.169:6789/0 conn(0x7fc6600ce880 legacy=0x7fc6600cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.442+0000 7fc65effd640 1 -- 172.21.15.122:0/2782072788 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66006d310 con 0x7fc66006cab0 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.443+0000 7fc65effd640 1 --2- 172.21.15.122:0/2782072788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600ce340 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc654009870 tx=0x7fc65402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba644f1d35b9838 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.443+0000 7fc65e7fc640 1 -- 172.21.15.122:0/2782072788 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65402f840 con 0x7fc66006cab0 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.443+0000 7fc65e7fc640 1 -- 172.21.15.122:0/2782072788 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65402f9e0 con 0x7fc66006cab0 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.443+0000 7fc65e7fc640 1 -- 172.21.15.122:0/2782072788 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65402fdb0 con 0x7fc66006cab0 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.443+0000 7fc665ca4640 1 -- 172.21.15.122:0/2782072788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 msgr2=0x7fc6600ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2782072788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600ce340 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc654009870 tx=0x7fc65402ba20 comp rx=0 tx=0).stop 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 -- 172.21.15.122:0/2782072788 shutdown_connections 2024-03-06T07:30:56.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2782072788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600ce340 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2782072788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc6600b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 -- 172.21.15.122:0/2782072788 >> 172.21.15.122:0/2782072788 conn(0x7fc6600b3690 msgr2=0x7fc6600b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 -- 172.21.15.122:0/2782072788 shutdown_connections 2024-03-06T07:30:56.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.444+0000 7fc665ca4640 1 -- 172.21.15.122:0/2782072788 wait complete. 2024-03-06T07:30:56.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.445+0000 7fc665ca4640 1 Processor -- start 2024-03-06T07:30:56.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.446+0000 7fc665ca4640 1 -- start start 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc665ca4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc665ca4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6600ce880 0x7fc6600cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc665ca4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc66015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc665ca4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600cc3f0 con 0x7fc66006cab0 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc665ca4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600cc760 con 0x7fc6600ce880 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc665ca4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015e2d0 con 0x7fc66015def0 2024-03-06T07:30:56.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54964/0 (socket says 172.21.15.122:54964) 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 -- 172.21.15.122:0/2174744960 learned_addr learned my addr 172.21.15.122:0/2174744960 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65effd640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6600ce880 0x7fc6600cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65ffff640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc66015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 -- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 msgr2=0x7fc66015ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc66015ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 -- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6600ce880 msgr2=0x7fc6600cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6600ce880 0x7fc6600cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 -- 172.21.15.122:0/2174744960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600c9640 con 0x7fc66006cab0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65ffff640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc66015ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.447+0000 7fc65f7fe640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600c7870 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fc65000d620 tx=0x7fc65000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.448+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65000b620 con 0x7fc66006cab0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.448+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6600ca670 con 0x7fc66006cab0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.448+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc650019070 con 0x7fc66006cab0 2024-03-06T07:30:56.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.448+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6600c9e70 con 0x7fc66006cab0 2024-03-06T07:30:56.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.448+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65000ba00 con 0x7fc66006cab0 2024-03-06T07:30:56.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.449+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66006dc80 con 0x7fc66006cab0 2024-03-06T07:30:56.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.450+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc65000bba0 con 0x7fc66006cab0 2024-03-06T07:30:56.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.450+0000 7fc647fff640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc64c06b4d0 0x7fc64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.450+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc65008a640 con 0x7fc66006cab0 2024-03-06T07:30:56.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.450+0000 7fc65effd640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc64c06b4d0 0x7fc64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.451+0000 7fc65effd640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc64c06b4d0 0x7fc64c06d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc654036000 tx=0x7fc654008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.455+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc66006dc80 con 0x7fc66006cab0 2024-03-06T07:30:56.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.631+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc66009fcf0 con 0x7fc66006cab0 2024-03-06T07:30:56.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.632+0000 7fc647fff640 1 -- 172.21.15.122:0/2174744960 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc650059420 con 0x7fc66006cab0 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:30:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc64c06b4d0 msgr2=0x7fc64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc64c06b4d0 0x7fc64c06d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc654036000 tx=0x7fc654008030 comp rx=0 tx=0).stop 2024-03-06T07:30:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 msgr2=0x7fc6600c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600c7870 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fc65000d620 tx=0x7fc65000dae0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 shutdown_connections 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc64c06b4d0 0x7fc64c06d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc66015def0 0x7fc66015ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6600ce880 0x7fc6600cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 --2- 172.21.15.122:0/2174744960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc66006cab0 0x7fc6600c7870 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 >> 172.21.15.122:0/2174744960 conn(0x7fc6600b3690 msgr2=0x7fc6600d23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 shutdown_connections 2024-03-06T07:30:56.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.634+0000 7fc665ca4640 1 -- 172.21.15.122:0/2174744960 wait complete. 2024-03-06T07:30:56.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:56.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.780+0000 7fce2b93e640 1 Processor -- start 2024-03-06T07:30:56.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.780+0000 7fce2b93e640 1 -- start start 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce2b93e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fce240d10c0 0x7fce240d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce2b93e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240cfa50 con 0x7fce2415ecd0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce2b93e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240d0a30 con 0x7fce240d14a0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce2b93e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2415c050 con 0x7fce240d10c0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce29edb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fce2415ecd0 0x7fce240cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45612/0 (socket says 172.21.15.122:45612) 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce29edb640 1 -- 172.21.15.122:0/256646127 learned_addr learned my addr 172.21.15.122:0/256646127 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce296da640 1 --2- 172.21.15.122:0/256646127 >> v2:172.21.15.169:3300/0 conn(0x7fce240d10c0 0x7fce240d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce1bfff640 1 -- 172.21.15.122:0/256646127 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348439588 0 0) 0x7fce240cfa50 con 0x7fce2415ecd0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.781+0000 7fce1bfff640 1 -- 172.21.15.122:0/256646127 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce10003520 con 0x7fce2415ecd0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.782+0000 7fce296da640 1 -- 172.21.15.122:0/256646127 >> v1:172.21.15.169:6789/0 conn(0x7fce240d14a0 legacy=0x7fce2415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.782+0000 7fce1bfff640 1 -- 172.21.15.122:0/256646127 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664796790 0 0) 0x7fce240d0a30 con 0x7fce240d14a0 2024-03-06T07:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.782+0000 7fce296da640 1 -- 172.21.15.122:0/256646127 >> v1:172.21.15.122:6789/0 conn(0x7fce2415ecd0 legacy=0x7fce240cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.782+0000 7fce296da640 1 -- 172.21.15.122:0/256646127 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240b45c0 con 0x7fce240d10c0 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.782+0000 7fce296da640 1 --2- 172.21.15.122:0/256646127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240d10c0 0x7fce240d04f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009d90 tx=0x7fce1c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd3f044933a36fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce1bfff640 1 -- 172.21.15.122:0/256646127 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c039070 con 0x7fce240d10c0 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce1bfff640 1 -- 172.21.15.122:0/256646127 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c02cd00 con 0x7fce240d10c0 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce1bfff640 1 -- 172.21.15.122:0/256646127 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c034660 con 0x7fce240d10c0 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 -- 172.21.15.122:0/256646127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240d10c0 msgr2=0x7fce240d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 --2- 172.21.15.122:0/256646127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240d10c0 0x7fce240d04f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009d90 tx=0x7fce1c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 -- 172.21.15.122:0/256646127 shutdown_connections 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 --2- 172.21.15.122:0/256646127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240d10c0 0x7fce240d04f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 -- 172.21.15.122:0/256646127 >> 172.21.15.122:0/256646127 conn(0x7fce240b3800 msgr2=0x7fce240b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 -- 172.21.15.122:0/256646127 shutdown_connections 2024-03-06T07:30:56.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.783+0000 7fce2b93e640 1 -- 172.21.15.122:0/256646127 wait complete. 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.784+0000 7fce2b93e640 1 Processor -- start 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.784+0000 7fce2b93e640 1 -- start start 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce2b93e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 0x7fce2414c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce2b93e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce240d14a0 0x7fce240b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce2b93e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240b6e20 0x7fce241606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce2b93e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2415f440 con 0x7fce240d10c0 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce2b93e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce240d0060 con 0x7fce240b6e20 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce2b93e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce240cfcf0 con 0x7fce240d14a0 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce296da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 0x7fce2414c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce296da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 0x7fce2414c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54970/0 (socket says 172.21.15.122:54970) 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce296da640 1 -- 172.21.15.122:0/3169731016 learned_addr learned my addr 172.21.15.122:0/3169731016 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce28ed9640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce240d14a0 0x7fce240b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.785+0000 7fce29edb640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240b6e20 0x7fce241606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce296da640 1 -- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce240d14a0 msgr2=0x7fce240b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce296da640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce240d14a0 0x7fce240b68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce296da640 1 -- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240b6e20 msgr2=0x7fce241606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce296da640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240b6e20 0x7fce241606d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce296da640 1 -- 172.21.15.122:0/3169731016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240b4e60 con 0x7fce240d10c0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce29edb640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240b6e20 0x7fce241606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce296da640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 0x7fce2414c7a0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009eb0 tx=0x7fce1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c039050 con 0x7fce240d10c0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c045070 con 0x7fce240d10c0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce240b5140 con 0x7fce240d10c0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c03e830 con 0x7fce240d10c0 2024-03-06T07:30:56.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.786+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce240b5700 con 0x7fce240d10c0 2024-03-06T07:30:56.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.787+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce24069e80 con 0x7fce240d10c0 2024-03-06T07:30:56.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.788+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce1c03e9d0 con 0x7fce240d10c0 2024-03-06T07:30:56.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.788+0000 7fce19ffb640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce0806b4d0 0x7fce0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.788+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce1c0ad160 con 0x7fce240d10c0 2024-03-06T07:30:56.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.788+0000 7fce28ed9640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce0806b4d0 0x7fce0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.789+0000 7fce28ed9640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce0806b4d0 0x7fce0806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fce240b0130 tx=0x7fce14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:56.797+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce24069e80 con 0x7fce240d10c0 2024-03-06T07:30:57.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.019+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce2409fe60 con 0x7fce240d10c0 2024-03-06T07:30:57.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.020+0000 7fce19ffb640 1 -- 172.21.15.122:0/3169731016 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce1c07c170 con 0x7fce240d10c0 2024-03-06T07:30:57.022 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce0806b4d0 msgr2=0x7fce0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce0806b4d0 0x7fce0806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fce240b0130 tx=0x7fce14009040 comp rx=0 tx=0).stop 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 msgr2=0x7fce2414c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 0x7fce2414c7a0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009eb0 tx=0x7fce1c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 shutdown_connections 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fce0806b4d0 0x7fce0806d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce240b6e20 0x7fce241606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce240d14a0 0x7fce240b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 --2- 172.21.15.122:0/3169731016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce240d10c0 0x7fce2414c7a0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 >> 172.21.15.122:0/3169731016 conn(0x7fce240b3800 msgr2=0x7fce240cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 shutdown_connections 2024-03-06T07:30:57.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.023+0000 7fce2b93e640 1 -- 172.21.15.122:0/3169731016 wait complete. 2024-03-06T07:30:57.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:57.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.143+0000 7f880d481640 1 Processor -- start 2024-03-06T07:30:57.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.143+0000 7f880d481640 1 -- start start 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.144+0000 7f880d481640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080d14a0 0x7f880815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.144+0000 7f880d481640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f880815eb00 0x7f88080cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.144+0000 7f880d481640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080cfa50 con 0x7f88080d10c0 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.144+0000 7f880d481640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f880815c050 con 0x7f880815eb00 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.144+0000 7f880d481640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88080d0060 con 0x7f88080d14a0 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88077fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f880815eb00 0x7f88080cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f8806ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f88080d10c0 0x7f88080d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45624/0 (socket says 172.21.15.122:45624) 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f8806ffd640 1 -- 172.21.15.122:0/3626469602 learned_addr learned my addr 172.21.15.122:0/3626469602 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:30:57.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88077fe640 1 --2- 172.21.15.122:0/3626469602 >> v2:172.21.15.122:3300/0 conn(0x7f880815eb00 0x7f88080cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54974/0 (socket says 172.21.15.122:54974) 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88067fc640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080d14a0 0x7f880815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88077fe640 1 -- 172.21.15.122:0/3626469602 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080d14a0 msgr2=0x7f880815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88077fe640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080d14a0 0x7f880815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88077fe640 1 -- 172.21.15.122:0/3626469602 >> v1:172.21.15.122:6789/0 conn(0x7f88080d10c0 legacy=0x7f88080d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88077fe640 1 -- 172.21.15.122:0/3626469602 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080b45c0 con 0x7f880815eb00 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f8805ffb640 1 -- 172.21.15.122:0/3626469602 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372363404 0 0) 0x7f88080cfa50 con 0x7f88080d10c0 2024-03-06T07:30:57.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.145+0000 7f88067fc640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080d14a0 0x7f880815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f88077fe640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815eb00 0x7f88080cae80 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f8800009d90 tx=0x7f880002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2174a43e5c570c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f8805ffb640 1 -- 172.21.15.122:0/3626469602 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800039070 con 0x7f880815eb00 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f8805ffb640 1 -- 172.21.15.122:0/3626469602 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880002cc50 con 0x7f880815eb00 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f8805ffb640 1 -- 172.21.15.122:0/3626469602 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800034440 con 0x7f880815eb00 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 -- 172.21.15.122:0/3626469602 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815eb00 msgr2=0x7f88080cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815eb00 0x7f88080cae80 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f8800009d90 tx=0x7f880002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 -- 172.21.15.122:0/3626469602 shutdown_connections 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815eb00 0x7f88080cae80 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 --2- 172.21.15.122:0/3626469602 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080d14a0 0x7f880815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 -- 172.21.15.122:0/3626469602 >> 172.21.15.122:0/3626469602 conn(0x7f88080b3800 msgr2=0x7f88080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 -- 172.21.15.122:0/3626469602 shutdown_connections 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.146+0000 7f880d481640 1 -- 172.21.15.122:0/3626469602 wait complete. 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.148+0000 7f880d481640 1 Processor -- start 2024-03-06T07:30:57.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.148+0000 7f880d481640 1 -- start start 2024-03-06T07:30:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 0x7f88080b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080d14a0 0x7f880814e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f880815eb00 0x7f88080b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88080cfcf0 con 0x7f88080d10c0 2024-03-06T07:30:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880815f440 con 0x7f88080d14a0 2024-03-06T07:30:57.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88080cb3c0 con 0x7f880815eb00 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 0x7f88080b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 0x7f88080b7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54984/0 (socket says 172.21.15.122:54984) 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 -- 172.21.15.122:0/3732424024 learned_addr learned my addr 172.21.15.122:0/3732424024 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f88077fe640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f880815eb00 0x7f88080b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f88067fc640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080d14a0 0x7f880814e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 -- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f880815eb00 msgr2=0x7f88080b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f880815eb00 0x7f88080b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 -- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080d14a0 msgr2=0x7f880814e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080d14a0 0x7f880814e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 -- 172.21.15.122:0/3732424024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880815adf0 con 0x7f88080d10c0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f88077fe640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f880815eb00 0x7f88080b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f8806ffd640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 0x7f88080b7640 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f87f000c4d0 tx=0x7f87f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0018070 con 0x7f88080d10c0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0007500 con 0x7f88080d10c0 2024-03-06T07:30:57.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880814ef60 con 0x7f88080d10c0 2024-03-06T07:30:57.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.150+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0004a20 con 0x7f88080d10c0 2024-03-06T07:30:57.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.151+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88080b4d00 con 0x7f88080d10c0 2024-03-06T07:30:57.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.152+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880809fe60 con 0x7f88080d10c0 2024-03-06T07:30:57.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.152+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87f001b070 con 0x7f88080d10c0 2024-03-06T07:30:57.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.152+0000 7f87ff7fe640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e806b4d0 0x7f87e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.152+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f87f00894a0 con 0x7f88080d10c0 2024-03-06T07:30:57.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.152+0000 7f88067fc640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e806b4d0 0x7f87e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.153+0000 7f88067fc640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e806b4d0 0x7f87e806d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f88081521a0 tx=0x7f87f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.161+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87f0058390 con 0x7f88080d10c0 2024-03-06T07:30:57.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.339+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8808068ff0 con 0x7f88080d10c0 2024-03-06T07:30:57.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.340+0000 7f87ff7fe640 1 -- 172.21.15.122:0/3732424024 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87f0057c90 con 0x7f88080d10c0 2024-03-06T07:30:57.342 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e806b4d0 msgr2=0x7f87e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e806b4d0 0x7f87e806d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f88081521a0 tx=0x7f87f8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 msgr2=0x7f88080b7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 0x7f88080b7640 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f87f000c4d0 tx=0x7f87f000c990 comp rx=0 tx=0).stop 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 shutdown_connections 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e806b4d0 0x7f87e806d990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f880815eb00 0x7f88080b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080d14a0 0x7f880814e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 --2- 172.21.15.122:0/3732424024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88080d10c0 0x7f88080b7640 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 >> 172.21.15.122:0/3732424024 conn(0x7f88080b3800 msgr2=0x7f880806c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 shutdown_connections 2024-03-06T07:30:57.345 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.341+0000 7f880d481640 1 -- 172.21.15.122:0/3732424024 wait complete. 2024-03-06T07:30:57.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:57.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.458+0000 7fc20038a640 1 Processor -- start 2024-03-06T07:30:57.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.458+0000 7fc20038a640 1 -- start start 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc20038a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc1f80b4460 0x7fc1f80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc20038a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc1f8162970 0x7fc1f80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc20038a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f80d22c0 con 0x7fc1f8161cf0 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc20038a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f8161b00 con 0x7fc1f80b4460 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc20038a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f80cecc0 con 0x7fc1f8162970 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc1fe126640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc1f80b4460 0x7fc1f80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc1fe126640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc1f80b4460 0x7fc1f80d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54996/0 (socket says 172.21.15.122:54996) 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc1fe927640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc1f8161cf0 0x7fc1f80cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45640/0 (socket says 172.21.15.122:45640) 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.459+0000 7fc1fe126640 1 -- 172.21.15.122:0/1693202775 learned_addr learned my addr 172.21.15.122:0/1693202775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:57.462 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.460+0000 7fc1fd925640 1 --2- 172.21.15.122:0/1693202775 >> v2:172.21.15.169:3300/0 conn(0x7fc1f8162970 0x7fc1f80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.460+0000 7fc1fe126640 1 -- 172.21.15.122:0/1693202775 >> v2:172.21.15.169:3300/0 conn(0x7fc1f8162970 msgr2=0x7fc1f80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.460+0000 7fc1fe126640 1 --2- 172.21.15.122:0/1693202775 >> v2:172.21.15.169:3300/0 conn(0x7fc1f8162970 0x7fc1f80b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.460+0000 7fc1fe126640 1 -- 172.21.15.122:0/1693202775 >> v1:172.21.15.122:6789/0 conn(0x7fc1f8161cf0 legacy=0x7fc1f80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.460+0000 7fc1fd124640 1 -- 172.21.15.122:0/1693202775 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672941784 0 0) 0x7fc1f80d22c0 con 0x7fc1f8161cf0 2024-03-06T07:30:57.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.460+0000 7fc1fe126640 1 -- 172.21.15.122:0/1693202775 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f80d25a0 con 0x7fc1f80b4460 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc1fe126640 1 --2- 172.21.15.122:0/1693202775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f80b4460 0x7fc1f80d4800 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80d17e0 tx=0x7fc1ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2308ca982036ff2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc1fd124640 1 -- 172.21.15.122:0/1693202775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec034070 con 0x7fc1f80b4460 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc1fd124640 1 -- 172.21.15.122:0/1693202775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec030860 con 0x7fc1f80b4460 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc1fd124640 1 -- 172.21.15.122:0/1693202775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec030c30 con 0x7fc1f80b4460 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 -- 172.21.15.122:0/1693202775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f80b4460 msgr2=0x7fc1f80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 --2- 172.21.15.122:0/1693202775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f80b4460 0x7fc1f80d4800 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80d17e0 tx=0x7fc1ec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 -- 172.21.15.122:0/1693202775 shutdown_connections 2024-03-06T07:30:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 --2- 172.21.15.122:0/1693202775 >> v2:172.21.15.169:3300/0 conn(0x7fc1f8162970 0x7fc1f80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 --2- 172.21.15.122:0/1693202775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f80b4460 0x7fc1f80d4800 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 -- 172.21.15.122:0/1693202775 >> 172.21.15.122:0/1693202775 conn(0x7fc1f80d3ce0 msgr2=0x7fc1f80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 -- 172.21.15.122:0/1693202775 shutdown_connections 2024-03-06T07:30:57.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.461+0000 7fc20038a640 1 -- 172.21.15.122:0/1693202775 wait complete. 2024-03-06T07:30:57.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.463+0000 7fc20038a640 1 Processor -- start 2024-03-06T07:30:57.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.463+0000 7fc20038a640 1 -- start start 2024-03-06T07:30:57.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.464+0000 7fc20038a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1f80b4460 0x7fc1f80cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.464+0000 7fc20038a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1f8161cf0 0x7fc1f80c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.464+0000 7fc20038a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 0x7fc1f80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.464+0000 7fc20038a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80ce950 con 0x7fc1f8162970 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.464+0000 7fc20038a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80d2b60 con 0x7fc1f8161cf0 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.464+0000 7fc20038a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80b3ac0 con 0x7fc1f80b4460 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 0x7fc1f80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 0x7fc1f80b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55010/0 (socket says 172.21.15.122:55010) 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 -- 172.21.15.122:0/1791406618 learned_addr learned my addr 172.21.15.122:0/1791406618 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe126640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1f80b4460 0x7fc1f80cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 -- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1f80b4460 msgr2=0x7fc1f80cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1f80b4460 0x7fc1f80cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 -- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1f8161cf0 msgr2=0x7fc1f80c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1f8161cf0 0x7fc1f80c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 -- 172.21.15.122:0/1791406618 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f80c9610 con 0x7fc1f8162970 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1fe927640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 0x7fc1f80b2170 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000c7a0 tx=0x7fc1f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0018070 con 0x7fc1f8162970 2024-03-06T07:30:57.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f000a640 con 0x7fc1f8162970 2024-03-06T07:30:57.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f80c98f0 con 0x7fc1f8162970 2024-03-06T07:30:57.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f000aab0 con 0x7fc1f8162970 2024-03-06T07:30:57.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.465+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f80c9eb0 con 0x7fc1f8162970 2024-03-06T07:30:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.467+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f804f330 con 0x7fc1f8162970 2024-03-06T07:30:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.467+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1f001b070 con 0x7fc1f8162970 2024-03-06T07:30:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.467+0000 7fc1eaffd640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.467+0000 7fc1fe126640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.467+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc1f0089860 con 0x7fc1f8162970 2024-03-06T07:30:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.468+0000 7fc1fe126640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec036000 tx=0x7fc1ec008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.471+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1f00586c0 con 0x7fc1f8162970 2024-03-06T07:30:57.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.687+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc1f806b180 con 0x7fc1f8162970 2024-03-06T07:30:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.687+0000 7fc1eaffd640 1 -- 172.21.15.122:0/1791406618 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc1f0057fc0 con 0x7fc1f8162970 2024-03-06T07:30:57.689 INFO:teuthology.orchestra.run.smithi122.stderr:listed 0 entries 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1dc06b4d0 msgr2=0x7fc1dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec036000 tx=0x7fc1ec008030 comp rx=0 tx=0).stop 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 msgr2=0x7fc1f80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 0x7fc1f80b2170 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000c7a0 tx=0x7fc1f000cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 shutdown_connections 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1dc06b4d0 0x7fc1dc06d990 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1f8162970 0x7fc1f80b2170 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1f8161cf0 0x7fc1f80c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 --2- 172.21.15.122:0/1791406618 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1f80b4460 0x7fc1f80cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 >> 172.21.15.122:0/1791406618 conn(0x7fc1f80d3ce0 msgr2=0x7fc1f80d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 shutdown_connections 2024-03-06T07:30:57.692 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:30:57.689+0000 7fc20038a640 1 -- 172.21.15.122:0/1791406618 wait complete. 2024-03-06T07:30:57.702 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2024-03-06T07:30:57.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:57.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:57.727 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi122.stderr:ceph-fuse[85432]: fuse finished with error 0 and tester_r 0 2024-03-06T07:30:57.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:03.830 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:31:03.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:03.830 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:03.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:04.002 INFO:teuthology.orchestra.run.smithi122.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:31:04.002 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:31:04.003 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:31:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.114+0000 7fd631d0d640 1 Processor -- start 2024-03-06T07:31:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.114+0000 7fd631d0d640 1 -- start start 2024-03-06T07:31:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd631d0d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd62c1586e0 0x7fd62c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd631d0d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd62c0aef80 0x7fd62c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd631d0d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c158ac0 con 0x7fd62c0cffd0 2024-03-06T07:31:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd631d0d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62c15b2a0 con 0x7fd62c1586e0 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd631d0d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62c15b540 con 0x7fd62c0aef80 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd62c1586e0 0x7fd62c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd62c1586e0 0x7fd62c0aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55026/0 (socket says 172.21.15.122:55026) 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 -- 172.21.15.122:0/1824877775 learned_addr learned my addr 172.21.15.122:0/1824877775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62affd640 1 --2- 172.21.15.122:0/1824877775 >> v2:172.21.15.169:3300/0 conn(0x7fd62c0aef80 0x7fd62c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 -- 172.21.15.122:0/1824877775 >> v2:172.21.15.169:3300/0 conn(0x7fd62c0aef80 msgr2=0x7fd62c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 --2- 172.21.15.122:0/1824877775 >> v2:172.21.15.169:3300/0 conn(0x7fd62c0aef80 0x7fd62c0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 -- 172.21.15.122:0/1824877775 >> v1:172.21.15.169:6789/0 conn(0x7fd62c0cffd0 legacy=0x7fd62c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62b7fe640 1 -- 172.21.15.122:0/1824877775 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c15b820 con 0x7fd62c1586e0 2024-03-06T07:31:04.118 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.115+0000 7fd62affd640 1 --2- 172.21.15.122:0/1824877775 >> v2:172.21.15.169:3300/0 conn(0x7fd62c0aef80 0x7fd62c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:04.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.116+0000 7fd62b7fe640 1 --2- 172.21.15.122:0/1824877775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c1586e0 0x7fd62c0aea40 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fd62c072e30 tx=0x7fd61c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0369d1ae268e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.116+0000 7fd62a7fc640 1 -- 172.21.15.122:0/1824877775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c035020 con 0x7fd62c1586e0 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.116+0000 7fd62a7fc640 1 -- 172.21.15.122:0/1824877775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61c031870 con 0x7fd62c1586e0 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.116+0000 7fd62a7fc640 1 -- 172.21.15.122:0/1824877775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c031c40 con 0x7fd62c1586e0 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 -- 172.21.15.122:0/1824877775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c1586e0 msgr2=0x7fd62c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 --2- 172.21.15.122:0/1824877775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c1586e0 0x7fd62c0aea40 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fd62c072e30 tx=0x7fd61c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 -- 172.21.15.122:0/1824877775 shutdown_connections 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 --2- 172.21.15.122:0/1824877775 >> v2:172.21.15.169:3300/0 conn(0x7fd62c0aef80 0x7fd62c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 --2- 172.21.15.122:0/1824877775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c1586e0 0x7fd62c0aea40 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 -- 172.21.15.122:0/1824877775 >> 172.21.15.122:0/1824877775 conn(0x7fd62c06b190 msgr2=0x7fd62c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 -- 172.21.15.122:0/1824877775 shutdown_connections 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.117+0000 7fd631d0d640 1 -- 172.21.15.122:0/1824877775 wait complete. 2024-03-06T07:31:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.118+0000 7fd631d0d640 1 Processor -- start 2024-03-06T07:31:04.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.118+0000 7fd631d0d640 1 -- start start 2024-03-06T07:31:04.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd631d0d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd62c0aef80 0x7fd62c13ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd631d0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 0x7fd62c14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd631d0d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd62c1586e0 0x7fd62c145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd631d0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c071350 con 0x7fd62c0cffd0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd631d0d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c15bde0 con 0x7fd62c0aef80 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd631d0d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c0714d0 con 0x7fd62c1586e0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 0x7fd62c14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 0x7fd62c14f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55038/0 (socket says 172.21.15.122:55038) 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 -- 172.21.15.122:0/3889845626 learned_addr learned my addr 172.21.15.122:0/3889845626 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 -- 172.21.15.122:0/3889845626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd62c1586e0 msgr2=0x7fd62c145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd62c1586e0 0x7fd62c145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 -- 172.21.15.122:0/3889845626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd62c0aef80 msgr2=0x7fd62c13ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd62c0aef80 0x7fd62c13ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 -- 172.21.15.122:0/3889845626 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c14e7a0 con 0x7fd62c0cffd0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd62affd640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 0x7fd62c14f800 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fd61400c820 tx=0x7fd61400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.119+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614015070 con 0x7fd62c0cffd0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.120+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd614007a30 con 0x7fd62c0cffd0 2024-03-06T07:31:04.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.120+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c14ea40 con 0x7fd62c0cffd0 2024-03-06T07:31:04.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.120+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614007e00 con 0x7fd62c0cffd0 2024-03-06T07:31:04.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.120+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c14f000 con 0x7fd62c0cffd0 2024-03-06T07:31:04.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.121+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c0732b0 con 0x7fd62c0cffd0 2024-03-06T07:31:04.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.121+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd614018070 con 0x7fd62c0cffd0 2024-03-06T07:31:04.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.121+0000 7fd623fff640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd60c06b4d0 0x7fd60c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.122+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd61408a200 con 0x7fd62c0cffd0 2024-03-06T07:31:04.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.122+0000 7fd62b7fe640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd60c06b4d0 0x7fd60c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.122+0000 7fd62b7fe640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd60c06b4d0 0x7fd60c06d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd61c036000 tx=0x7fd61c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.126+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd614059060 con 0x7fd62c0cffd0 2024-03-06T07:31:04.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:04.356+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fd62c068ff0 con 0x7fd62c0cffd0 2024-03-06T07:31:05.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.138+0000 7fd623fff640 1 -- 172.21.15.122:0/3889845626 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd614058960 con 0x7fd62c0cffd0 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd60c06b4d0 msgr2=0x7fd60c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd60c06b4d0 0x7fd60c06d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd61c036000 tx=0x7fd61c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 msgr2=0x7fd62c14f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 0x7fd62c14f800 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fd61400c820 tx=0x7fd61400cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 shutdown_connections 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd60c06b4d0 0x7fd60c06d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd62c1586e0 0x7fd62c145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd62c0cffd0 0x7fd62c14f800 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 --2- 172.21.15.122:0/3889845626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd62c0aef80 0x7fd62c13ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 >> 172.21.15.122:0/3889845626 conn(0x7fd62c06b190 msgr2=0x7fd62c0af430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 shutdown_connections 2024-03-06T07:31:05.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.141+0000 7fd631d0d640 1 -- 172.21.15.122:0/3889845626 wait complete. 2024-03-06T07:31:05.155 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.253+0000 7f45b45d7640 1 Processor -- start 2024-03-06T07:31:05.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.253+0000 7f45b45d7640 1 -- start start 2024-03-06T07:31:05.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b45d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0b4170 0x7f45ac0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b45d7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45ac0d4760 0x7f45ac0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b45d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0d25d0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b45d7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac0d4bf0 con 0x7f45ac0d4760 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b45d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0ce9e0 con 0x7f45ac0b4170 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b1b72640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45ac0d4760 0x7f45ac0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b1b72640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45ac0d4760 0x7f45ac0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40494/0 (socket says 172.21.15.122:40494) 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.254+0000 7f45b1b72640 1 -- 172.21.15.122:0/4204936067 learned_addr learned my addr 172.21.15.122:0/4204936067 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b2373640 1 --2- 172.21.15.122:0/4204936067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0b4170 0x7f45ac0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b1b72640 1 -- 172.21.15.122:0/4204936067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0b4170 msgr2=0x7f45ac0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b1b72640 1 --2- 172.21.15.122:0/4204936067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0b4170 0x7f45ac0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b1b72640 1 -- 172.21.15.122:0/4204936067 >> v1:172.21.15.169:6789/0 conn(0x7f45ac0b4dd0 legacy=0x7f45ac0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b1b72640 1 -- 172.21.15.122:0/4204936067 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0d2870 con 0x7f45ac0d4760 2024-03-06T07:31:05.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b1b72640 1 --2- 172.21.15.122:0/4204936067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0d4760 0x7f45ac0d0370 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009870 tx=0x7f45a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1ad5789b0274b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.255+0000 7f45b1371640 1 -- 172.21.15.122:0/4204936067 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0035020 con 0x7f45ac0d4760 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b1371640 1 -- 172.21.15.122:0/4204936067 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0031640 con 0x7f45ac0d4760 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b1371640 1 -- 172.21.15.122:0/4204936067 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a002d040 con 0x7f45ac0d4760 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 -- 172.21.15.122:0/4204936067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0d4760 msgr2=0x7f45ac0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 --2- 172.21.15.122:0/4204936067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0d4760 0x7f45ac0d0370 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009870 tx=0x7f45a002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 -- 172.21.15.122:0/4204936067 shutdown_connections 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 --2- 172.21.15.122:0/4204936067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0d4760 0x7f45ac0d0370 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 --2- 172.21.15.122:0/4204936067 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0b4170 0x7f45ac0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 -- 172.21.15.122:0/4204936067 >> 172.21.15.122:0/4204936067 conn(0x7f45ac0d3c40 msgr2=0x7f45ac0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 -- 172.21.15.122:0/4204936067 shutdown_connections 2024-03-06T07:31:05.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.256+0000 7f45b45d7640 1 -- 172.21.15.122:0/4204936067 wait complete. 2024-03-06T07:31:05.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.257+0000 7f45b45d7640 1 Processor -- start 2024-03-06T07:31:05.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.257+0000 7f45b45d7640 1 -- start start 2024-03-06T07:31:05.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b45d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45ac0b4170 0x7f45ac156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b45d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 0x7f45ac15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b45d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0c9ba0 0x7f45ac0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b45d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0b47d0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b45d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0b4550 con 0x7f45ac0b4170 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b45d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0b4950 con 0x7f45ac0c9ba0 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b1b72640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 0x7f45ac15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b1b72640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 0x7f45ac15dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40510/0 (socket says 172.21.15.122:40510) 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b1b72640 1 -- 172.21.15.122:0/3110060679 learned_addr learned my addr 172.21.15.122:0/3110060679 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b2b74640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0c9ba0 0x7f45ac0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.258+0000 7f45b2373640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45ac0b4170 0x7f45ac156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b1b72640 1 -- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0c9ba0 msgr2=0x7f45ac0ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b1b72640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0c9ba0 0x7f45ac0ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b1b72640 1 -- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45ac0b4170 msgr2=0x7f45ac156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b1b72640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45ac0b4170 0x7f45ac156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b1b72640 1 -- 172.21.15.122:0/3110060679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0cd4e0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b2373640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45ac0b4170 0x7f45ac156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:05.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b1b72640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 0x7f45ac15dff0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f45a002b930 tx=0x7f45a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0040070 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a00399f0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac0cd7c0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0035040 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.259+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac0b1980 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.260+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac04f330 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.261+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45a0038020 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.261+0000 7f459effd640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f459006b4d0 0x7f459006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.261+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f45a00aba50 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.261+0000 7f45b2373640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f459006b4d0 0x7f459006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.262+0000 7f45b2373640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f459006b4d0 0x7f459006d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f45a8006fd0 tx=0x7f45a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.266+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45a007a2f0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.508+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45ac0733b0 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.508+0000 7f459effd640 1 -- 172.21.15.122:0/3110060679 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f45a007e390 con 0x7f45ac0b4dd0 2024-03-06T07:31:05.511 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:05.511 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:05.511 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f459006b4d0 msgr2=0x7f459006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f459006b4d0 0x7f459006d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f45a8006fd0 tx=0x7f45a8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 msgr2=0x7f45ac15dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 0x7f45ac15dff0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f45a002b930 tx=0x7f45a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 shutdown_connections 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f459006b4d0 0x7f459006d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.510+0000 7f45b45d7640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45ac0c9ba0 0x7f45ac0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.511+0000 7f45b45d7640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45ac0b4dd0 0x7f45ac15dff0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.511+0000 7f45b45d7640 1 --2- 172.21.15.122:0/3110060679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45ac0b4170 0x7f45ac156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.511+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 >> 172.21.15.122:0/3110060679 conn(0x7f45ac0d3c40 msgr2=0x7f45ac0b7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.511+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 shutdown_connections 2024-03-06T07:31:05.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:05.511+0000 7f45b45d7640 1 -- 172.21.15.122:0/3110060679 wait complete. 2024-03-06T07:31:05.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:05.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:06.726 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:06.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.827+0000 7fdb8eb6c640 1 Processor -- start 2024-03-06T07:31:06.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8eb6c640 1 -- start start 2024-03-06T07:31:06.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8eb6c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fdb8806c6d0 0x7fdb8815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8eb6c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880cd450 con 0x7fdb880b4f70 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8eb6c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880cd730 con 0x7fdb88066d90 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8eb6c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8815e9f0 con 0x7fdb8806c6d0 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8c908640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdb880b4f70 0x7fdb880b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49044/0 (socket says 172.21.15.122:49044) 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.828+0000 7fdb8c908640 1 -- 172.21.15.122:0/1324720573 learned_addr learned my addr 172.21.15.122:0/1324720573 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25422370 0 0) 0x7fdb880cd450 con 0x7fdb880b4f70 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb74003520 con 0x7fdb880b4f70 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb87fff640 1 --2- 172.21.15.122:0/1324720573 >> v2:172.21.15.169:3300/0 conn(0x7fdb8806c6d0 0x7fdb8815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1935829807 0 0) 0x7fdb74003520 con 0x7fdb880b4f70 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 >> v2:172.21.15.169:3300/0 conn(0x7fdb8806c6d0 msgr2=0x7fdb8815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 --2- 172.21.15.122:0/1324720573 >> v2:172.21.15.169:3300/0 conn(0x7fdb8806c6d0 0x7fdb8815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 >> v1:172.21.15.169:6789/0 conn(0x7fdb88066d90 legacy=0x7fdb880ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8815e680 con 0x7fdb880b4f70 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fdb800019f0 con 0x7fdb880b4f70 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb87fff640 1 --2- 172.21.15.122:0/1324720573 >> v2:172.21.15.169:3300/0 conn(0x7fdb8806c6d0 0x7fdb8815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 >> v1:172.21.15.122:6789/0 conn(0x7fdb880b4f70 legacy=0x7fdb880b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8815e9f0 con 0x7fdb740051c0 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7400c110 con 0x7fdb74004800 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7400c310 con 0x7fdb740039b0 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8d109640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8c908640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb87fff640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8d109640 1 -- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb740039b0 msgr2=0x7fdb74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8d109640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb740039b0 0x7fdb74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8d109640 1 -- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb74004800 msgr2=0x7fdb74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8d109640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb74004800 0x7fdb74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.829+0000 7fdb8d109640 1 -- 172.21.15.122:0/1324720573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb74003520 con 0x7fdb740051c0 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.830+0000 7fdb8d109640 1 -- 172.21.15.122:0/1324720573 reap_dead start 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.830+0000 7fdb8d109640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fdb880ca740 tx=0x7fdb74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63891a9bee41f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.830+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78001370 con 0x7fdb740051c0 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.830+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb780052d0 con 0x7fdb740051c0 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.830+0000 7fdb877fe640 1 -- 172.21.15.122:0/1324720573 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78003830 con 0x7fdb740051c0 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/1324720573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb740051c0 msgr2=0x7fdb7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fdb880ca740 tx=0x7fdb74005830 comp rx=0 tx=0).stop 2024-03-06T07:31:06.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/1324720573 shutdown_connections 2024-03-06T07:31:06.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/1324720573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb740051c0 0x7fdb7400b9c0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/1324720573 >> 172.21.15.122:0/1324720573 conn(0x7fdb880724d0 msgr2=0x7fdb880728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/1324720573 shutdown_connections 2024-03-06T07:31:06.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.831+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/1324720573 wait complete. 2024-03-06T07:31:06.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.832+0000 7fdb8eb6c640 1 Processor -- start 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.832+0000 7fdb8eb6c640 1 -- start start 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8eb6c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb740051c0 0x7fdb881466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8eb6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 0x7fdb880c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8eb6c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb880c99d0 0x7fdb8815fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8eb6c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb880b56e0 con 0x7fdb880caf10 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8eb6c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb880b5560 con 0x7fdb740051c0 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8eb6c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb880b5860 con 0x7fdb880c99d0 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 0x7fdb880c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 0x7fdb880c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40534/0 (socket says 172.21.15.122:40534) 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 -- 172.21.15.122:0/3965161276 learned_addr learned my addr 172.21.15.122:0/3965161276 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8d109640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb880c99d0 0x7fdb8815fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb8c908640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb740051c0 0x7fdb881466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 -- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb880c99d0 msgr2=0x7fdb8815fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb880c99d0 0x7fdb8815fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 -- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb740051c0 msgr2=0x7fdb881466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb740051c0 0x7fdb881466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.833+0000 7fdb87fff640 1 -- 172.21.15.122:0/3965161276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88146c10 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.834+0000 7fdb87fff640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 0x7fdb880c9490 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fdb70011590 tx=0x7fdb70011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.834+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70019070 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.834+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb70004220 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.834+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7000f040 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.834+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb880c9e70 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.834+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb880ca430 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.835+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8804f330 con 0x7fdb880caf10 2024-03-06T07:31:06.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.835+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb70015670 con 0x7fdb880caf10 2024-03-06T07:31:06.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.836+0000 7fdb857fa640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdb6806b500 0x7fdb6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.836+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdb7008a440 con 0x7fdb880caf10 2024-03-06T07:31:06.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.836+0000 7fdb8c908640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdb6806b500 0x7fdb6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.836+0000 7fdb8c908640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdb6806b500 0x7fdb6806d9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdb80002230 tx=0x7fdb80006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:06.840+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb88146c10 con 0x7fdb880caf10 2024-03-06T07:31:07.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.055+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb880b1860 con 0x7fdb880caf10 2024-03-06T07:31:07.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.055+0000 7fdb857fa640 1 -- 172.21.15.122:0/3965161276 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fdb70059330 con 0x7fdb880caf10 2024-03-06T07:31:07.058 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:07.058 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:07.058 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:07.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdb6806b500 msgr2=0x7fdb6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdb6806b500 0x7fdb6806d9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdb80002230 tx=0x7fdb80006040 comp rx=0 tx=0).stop 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 msgr2=0x7fdb880c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 0x7fdb880c9490 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fdb70011590 tx=0x7fdb70011a50 comp rx=0 tx=0).stop 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 shutdown_connections 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fdb6806b500 0x7fdb6806d9c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fdb880c99d0 0x7fdb8815fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb880caf10 0x7fdb880c9490 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 --2- 172.21.15.122:0/3965161276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb740051c0 0x7fdb881466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 >> 172.21.15.122:0/3965161276 conn(0x7fdb880724d0 msgr2=0x7fdb8815bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 shutdown_connections 2024-03-06T07:31:07.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:07.058+0000 7fdb8eb6c640 1 -- 172.21.15.122:0/3965161276 wait complete. 2024-03-06T07:31:07.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:07.272 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:08.273 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:08.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.373+0000 7faa6d5c7640 1 Processor -- start 2024-03-06T07:31:08.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.373+0000 7faa6d5c7640 1 -- start start 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.374+0000 7faa6d5c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa680b4170 0x7faa680b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.374+0000 7faa6d5c7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa680d25d0 con 0x7faa680d0a80 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.374+0000 7faa6d5c7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa680d2870 con 0x7faa680d4760 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.374+0000 7faa6d5c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680d4bf0 con 0x7faa680b4170 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa677fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faa680d0a80 0x7faa680cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49056/0 (socket says 172.21.15.122:49056) 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa677fe640 1 -- 172.21.15.122:0/3834640833 learned_addr learned my addr 172.21.15.122:0/3834640833 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa66ffd640 1 --2- 172.21.15.122:0/3834640833 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa680b4170 0x7faa680b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa667fc640 1 -- 172.21.15.122:0/3834640833 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633071708 0 0) 0x7faa680d25d0 con 0x7faa680d0a80 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa667fc640 1 -- 172.21.15.122:0/3834640833 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa54003520 con 0x7faa680d0a80 2024-03-06T07:31:08.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa667fc640 1 -- 172.21.15.122:0/3834640833 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454250848 0 0) 0x7faa680d2870 con 0x7faa680d4760 2024-03-06T07:31:08.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa66ffd640 1 -- 172.21.15.122:0/3834640833 >> v1:172.21.15.169:6789/0 conn(0x7faa680d4760 legacy=0x7faa680d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa66ffd640 1 -- 172.21.15.122:0/3834640833 >> v1:172.21.15.122:6789/0 conn(0x7faa680d0a80 legacy=0x7faa680cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.375+0000 7faa66ffd640 1 -- 172.21.15.122:0/3834640833 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680d2b50 con 0x7faa680b4170 2024-03-06T07:31:08.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.376+0000 7faa66ffd640 1 --2- 172.21.15.122:0/3834640833 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa680b4170 0x7faa680b6a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faa580097d0 tx=0x7faa5802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca3b9e1cf76369c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.376+0000 7faa667fc640 1 -- 172.21.15.122:0/3834640833 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58035020 con 0x7faa680b4170 2024-03-06T07:31:08.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.376+0000 7faa667fc640 1 -- 172.21.15.122:0/3834640833 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa58031890 con 0x7faa680b4170 2024-03-06T07:31:08.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.376+0000 7faa667fc640 1 -- 172.21.15.122:0/3834640833 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58031c40 con 0x7faa680b4170 2024-03-06T07:31:08.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 -- 172.21.15.122:0/3834640833 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa680b4170 msgr2=0x7faa680b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/3834640833 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa680b4170 0x7faa680b6a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faa580097d0 tx=0x7faa5802b930 comp rx=0 tx=0).stop 2024-03-06T07:31:08.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 -- 172.21.15.122:0/3834640833 shutdown_connections 2024-03-06T07:31:08.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/3834640833 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa680b4170 0x7faa680b6a90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 -- 172.21.15.122:0/3834640833 >> 172.21.15.122:0/3834640833 conn(0x7faa680d3c40 msgr2=0x7faa680d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 -- 172.21.15.122:0/3834640833 shutdown_connections 2024-03-06T07:31:08.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.377+0000 7faa6d5c7640 1 -- 172.21.15.122:0/3834640833 wait complete. 2024-03-06T07:31:08.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.379+0000 7faa6d5c7640 1 Processor -- start 2024-03-06T07:31:08.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.379+0000 7faa6d5c7640 1 -- start start 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa6d5c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa680b4170 0x7faa681f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa6d5c7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 0x7faa68149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa6d5c7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa68159500 0x7faa6814a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa6d5c7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa680b4dd0 con 0x7faa680d0a80 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa6d5c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa680ce9e0 con 0x7faa680b4170 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa6d5c7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680b4f50 con 0x7faa68159500 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 0x7faa68149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 0x7faa68149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40546/0 (socket says 172.21.15.122:40546) 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 -- 172.21.15.122:0/2305969636 learned_addr learned my addr 172.21.15.122:0/2305969636 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:08.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa66ffd640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa680b4170 0x7faa681f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa677fe640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa68159500 0x7faa6814a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 -- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa68159500 msgr2=0x7faa6814a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa68159500 0x7faa6814a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 -- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa680b4170 msgr2=0x7faa681f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa680b4170 0x7faa681f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa5ffff640 1 -- 172.21.15.122:0/2305969636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680cdce0 con 0x7faa680d0a80 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.380+0000 7faa677fe640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa68159500 0x7faa6814a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.381+0000 7faa5ffff640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 0x7faa68149d60 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7faa5000c410 tx=0x7faa5000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.381+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50017070 con 0x7faa680d0a80 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.381+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa500035f0 con 0x7faa680d0a80 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.381+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa500039e0 con 0x7faa680d0a80 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.381+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6815ab30 con 0x7faa680d0a80 2024-03-06T07:31:08.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.381+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa68159d70 con 0x7faa680d0a80 2024-03-06T07:31:08.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.382+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6804f330 con 0x7faa680d0a80 2024-03-06T07:31:08.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.382+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa5001a070 con 0x7faa680d0a80 2024-03-06T07:31:08.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.383+0000 7faa5f7fe640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa4806b4d0 0x7faa4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.383+0000 7faa66ffd640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa4806b4d0 0x7faa4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.383+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faa50089610 con 0x7faa680d0a80 2024-03-06T07:31:08.385 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.383+0000 7faa66ffd640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa4806b4d0 0x7faa4806d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7faa58036000 tx=0x7faa58008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.389 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.387+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa50058470 con 0x7faa680d0a80 2024-03-06T07:31:08.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.602+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa6815ad90 con 0x7faa680d0a80 2024-03-06T07:31:08.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.603+0000 7faa5f7fe640 1 -- 172.21.15.122:0/2305969636 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7faa50057d70 con 0x7faa680d0a80 2024-03-06T07:31:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:08.605 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:08.605 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:08.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa4806b4d0 msgr2=0x7faa4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa4806b4d0 0x7faa4806d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7faa58036000 tx=0x7faa58008030 comp rx=0 tx=0).stop 2024-03-06T07:31:08.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 msgr2=0x7faa68149d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 0x7faa68149d60 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7faa5000c410 tx=0x7faa5000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 shutdown_connections 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faa4806b4d0 0x7faa4806d990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faa68159500 0x7faa6814a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa680d0a80 0x7faa68149d60 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 --2- 172.21.15.122:0/2305969636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa680b4170 0x7faa681f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 >> 172.21.15.122:0/2305969636 conn(0x7faa680d3c40 msgr2=0x7faa680b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 shutdown_connections 2024-03-06T07:31:08.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:08.605+0000 7faa6d5c7640 1 -- 172.21.15.122:0/2305969636 wait complete. 2024-03-06T07:31:08.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:08.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:09.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:09.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.922+0000 7f155f555640 1 Processor -- start 2024-03-06T07:31:09.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.922+0000 7f155f555640 1 -- start start 2024-03-06T07:31:09.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155f555640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f15580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155f555640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f15580b41f0 0x7f15580b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155f555640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15580b3430 con 0x7f155815dfe0 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155f555640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580d2c50 con 0x7f15580b41f0 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155f555640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580b31e0 con 0x7f15580d22b0 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f15580b41f0 0x7f15580b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f15580b41f0 0x7f15580b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40554/0 (socket says 172.21.15.122:40554) 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 -- 172.21.15.122:0/3601908026 learned_addr learned my addr 172.21.15.122:0/3601908026 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155caf0640 1 --2- 172.21.15.122:0/3601908026 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f15580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 -- 172.21.15.122:0/3601908026 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 msgr2=0x7f15580d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 --2- 172.21.15.122:0/3601908026 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f15580d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 -- 172.21.15.122:0/3601908026 >> v1:172.21.15.169:6789/0 conn(0x7f155815dfe0 legacy=0x7f15580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.923+0000 7f155daf2640 1 -- 172.21.15.122:0/3601908026 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580b3710 con 0x7f15580b41f0 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.924+0000 7f155daf2640 1 --2- 172.21.15.122:0/3601908026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15580b41f0 0x7f15580b4650 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f1550009e50 tx=0x7f155002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98b95f55019691a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.924+0000 7f154ffff640 1 -- 172.21.15.122:0/3601908026 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155002ca70 con 0x7f15580b41f0 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.924+0000 7f154ffff640 1 -- 172.21.15.122:0/3601908026 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155002cc10 con 0x7f15580b41f0 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.924+0000 7f154ffff640 1 -- 172.21.15.122:0/3601908026 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15500343f0 con 0x7f15580b41f0 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 -- 172.21.15.122:0/3601908026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15580b41f0 msgr2=0x7f15580b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 --2- 172.21.15.122:0/3601908026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15580b41f0 0x7f15580b4650 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f1550009e50 tx=0x7f155002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 -- 172.21.15.122:0/3601908026 shutdown_connections 2024-03-06T07:31:09.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 --2- 172.21.15.122:0/3601908026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15580b41f0 0x7f15580b4650 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 --2- 172.21.15.122:0/3601908026 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f15580d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 -- 172.21.15.122:0/3601908026 >> 172.21.15.122:0/3601908026 conn(0x7f15580d3cd0 msgr2=0x7f15580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 -- 172.21.15.122:0/3601908026 shutdown_connections 2024-03-06T07:31:09.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.925+0000 7f155f555640 1 -- 172.21.15.122:0/3601908026 wait complete. 2024-03-06T07:31:09.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.926+0000 7f155f555640 1 Processor -- start 2024-03-06T07:31:09.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.926+0000 7f155f555640 1 -- start start 2024-03-06T07:31:09.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155f555640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15580b41f0 0x7f15580cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155f555640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f1558160e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155f555640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 0x7f155815e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155f555640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580cfa00 con 0x7f155815dfe0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155f555640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580b3d60 con 0x7f15580b41f0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155f555640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580d1c70 con 0x7f15580d22b0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 0x7f155815e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 0x7f155815e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40560/0 (socket says 172.21.15.122:40560) 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 -- 172.21.15.122:0/2032706873 learned_addr learned my addr 172.21.15.122:0/2032706873 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155caf0640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f1558160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155d2f1640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15580b41f0 0x7f15580cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 -- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 msgr2=0x7f1558160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f1558160e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 -- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15580b41f0 msgr2=0x7f15580cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15580b41f0 0x7f15580cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 -- 172.21.15.122:0/2032706873 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580cbfb0 con 0x7f155815dfe0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155caf0640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f1558160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f155daf2640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 0x7f155815e840 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f1550036040 tx=0x7f1550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.927+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550043020 con 0x7f155815dfe0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.928+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155003e450 con 0x7f155815dfe0 2024-03-06T07:31:09.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.928+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15580cc270 con 0x7f155815dfe0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.928+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550009510 con 0x7f155815dfe0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.928+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15580cc830 con 0x7f155815dfe0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.929+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155804f330 con 0x7f155815dfe0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.929+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1550046020 con 0x7f155815dfe0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.930+0000 7f154dffb640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f153c06b4d0 0x7f153c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.930+0000 7f155d2f1640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f153c06b4d0 0x7f153c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.930+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f15500aca30 con 0x7f155815dfe0 2024-03-06T07:31:09.937 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.930+0000 7f155d2f1640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f153c06b4d0 0x7f153c06d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:09.938+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f155004d030 con 0x7f155815dfe0 2024-03-06T07:31:10.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.212+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15580747a0 con 0x7f155815dfe0 2024-03-06T07:31:10.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.212+0000 7f154dffb640 1 -- 172.21.15.122:0/2032706873 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f155007b890 con 0x7f155815dfe0 2024-03-06T07:31:10.215 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:10.215 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:10.215 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f153c06b4d0 msgr2=0x7f153c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f153c06b4d0 0x7f153c06d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548008040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 msgr2=0x7f155815e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 0x7f155815e840 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f1550036040 tx=0x7f1550004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 shutdown_connections 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f153c06b4d0 0x7f153c06d990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155815dfe0 0x7f155815e840 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f15580d22b0 0x7f1558160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 --2- 172.21.15.122:0/2032706873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15580b41f0 0x7f15580cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 >> 172.21.15.122:0/2032706873 conn(0x7f15580d3cd0 msgr2=0x7f155806ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 shutdown_connections 2024-03-06T07:31:10.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:10.215+0000 7f155f555640 1 -- 172.21.15.122:0/2032706873 wait complete. 2024-03-06T07:31:10.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:10.429 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:11.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:11.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.528+0000 7efc41647640 1 Processor -- start 2024-03-06T07:31:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.528+0000 7efc41647640 1 -- start start 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc41647640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc3c0cce40 0x7efc3c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc41647640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc41647640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3c071f40 con 0x7efc3c1515e0 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc41647640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3c0cd2f0 con 0x7efc3c0cce40 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc41647640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c070a70 con 0x7efc3c0cd7a0 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc3a7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc3c0cce40 0x7efc3c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc3a7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc3c0cce40 0x7efc3c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40562/0 (socket says 172.21.15.122:40562) 2024-03-06T07:31:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc3affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efc3c1515e0 0x7efc3c071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49068/0 (socket says 172.21.15.122:49068) 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.529+0000 7efc3a7fc640 1 -- 172.21.15.122:0/1501737231 learned_addr learned my addr 172.21.15.122:0/1501737231 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.530+0000 7efc3b7fe640 1 --2- 172.21.15.122:0/1501737231 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.530+0000 7efc3a7fc640 1 -- 172.21.15.122:0/1501737231 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 msgr2=0x7efc3c0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.530+0000 7efc3a7fc640 1 --2- 172.21.15.122:0/1501737231 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.530+0000 7efc3a7fc640 1 -- 172.21.15.122:0/1501737231 >> v1:172.21.15.122:6789/0 conn(0x7efc3c1515e0 legacy=0x7efc3c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.530+0000 7efc3a7fc640 1 -- 172.21.15.122:0/1501737231 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c070d50 con 0x7efc3c0cce40 2024-03-06T07:31:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.530+0000 7efc39ffb640 1 -- 172.21.15.122:0/1501737231 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983153934 0 0) 0x7efc3c071f40 con 0x7efc3c1515e0 2024-03-06T07:31:11.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc3a7fc640 1 --2- 172.21.15.122:0/1501737231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c15b430 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00bb10 tx=0x7efc2c02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb725121aeb480f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc39ffb640 1 -- 172.21.15.122:0/1501737231 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c03b070 con 0x7efc3c0cce40 2024-03-06T07:31:11.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc39ffb640 1 -- 172.21.15.122:0/1501737231 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c036440 con 0x7efc3c0cce40 2024-03-06T07:31:11.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc39ffb640 1 -- 172.21.15.122:0/1501737231 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c036810 con 0x7efc3c0cce40 2024-03-06T07:31:11.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 -- 172.21.15.122:0/1501737231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 msgr2=0x7efc3c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 --2- 172.21.15.122:0/1501737231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c15b430 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00bb10 tx=0x7efc2c02e150 comp rx=0 tx=0).stop 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 -- 172.21.15.122:0/1501737231 shutdown_connections 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 --2- 172.21.15.122:0/1501737231 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 --2- 172.21.15.122:0/1501737231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c15b430 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 -- 172.21.15.122:0/1501737231 >> 172.21.15.122:0/1501737231 conn(0x7efc3c0cf8a0 msgr2=0x7efc3c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 -- 172.21.15.122:0/1501737231 shutdown_connections 2024-03-06T07:31:11.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.531+0000 7efc41647640 1 -- 172.21.15.122:0/1501737231 wait complete. 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 Processor -- start 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 -- start start 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0aa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc3c1515e0 0x7efc3c0c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0ce0f0 con 0x7efc3c0cce40 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c071310 con 0x7efc3c1515e0 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.533+0000 7efc41647640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0cc3a0 con 0x7efc3c0cd7a0 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40568/0 (socket says 172.21.15.122:40568) 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 -- 172.21.15.122:0/4053079146 learned_addr learned my addr 172.21.15.122:0/4053079146 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:11.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3a7fc640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3b7fe640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc3c1515e0 0x7efc3c0c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 -- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 msgr2=0x7efc3c0aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0aa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 -- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc3c1515e0 msgr2=0x7efc3c0c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc3c1515e0 0x7efc3c0c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 -- 172.21.15.122:0/4053079146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0ca310 con 0x7efc3c0cce40 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3a7fc640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3b7fe640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc3c1515e0 0x7efc3c0c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc3affd640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c06c800 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7efc3000c440 tx=0x7efc3000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc300074e0 con 0x7efc3c0cce40 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc300047e0 con 0x7efc3c0cce40 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c0a97c0 con 0x7efc3c0cce40 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30004bd0 con 0x7efc3c0cce40 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.534+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c0ca5d0 con 0x7efc3c0cce40 2024-03-06T07:31:11.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.535+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c068840 con 0x7efc3c0cce40 2024-03-06T07:31:11.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.536+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc3001a020 con 0x7efc3c0cce40 2024-03-06T07:31:11.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.536+0000 7efc237fe640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc2806b4d0 0x7efc2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.536+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efc300899c0 con 0x7efc3c0cce40 2024-03-06T07:31:11.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.536+0000 7efc3a7fc640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc2806b4d0 0x7efc2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.537+0000 7efc3a7fc640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc2806b4d0 0x7efc2806d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7efc2c038000 tx=0x7efc2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.545+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc3008be20 con 0x7efc3c0cce40 2024-03-06T07:31:11.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.762+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc3c072bc0 con 0x7efc3c0cce40 2024-03-06T07:31:11.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.762+0000 7efc237fe640 1 -- 172.21.15.122:0/4053079146 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7efc30010300 con 0x7efc3c0cce40 2024-03-06T07:31:11.764 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:11.764 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:11.765 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc2806b4d0 msgr2=0x7efc2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc2806b4d0 0x7efc2806d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7efc2c038000 tx=0x7efc2c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 msgr2=0x7efc3c06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c06c800 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7efc3000c440 tx=0x7efc3000c900 comp rx=0 tx=0).stop 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 shutdown_connections 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc2806b4d0 0x7efc2806d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc3c1515e0 0x7efc3c0c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc3c0cd7a0 0x7efc3c0aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 --2- 172.21.15.122:0/4053079146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0cce40 0x7efc3c06c800 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 >> 172.21.15.122:0/4053079146 conn(0x7efc3c0cf8a0 msgr2=0x7efc3c14e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 shutdown_connections 2024-03-06T07:31:11.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:11.764+0000 7efc41647640 1 -- 172.21.15.122:0/4053079146 wait complete. 2024-03-06T07:31:11.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:11.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:12.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:13.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.079+0000 7fae69bd7640 1 Processor -- start 2024-03-06T07:31:13.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.079+0000 7fae69bd7640 1 -- start start 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae69bd7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae6415dfe0 0x7fae640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae69bd7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640cf830 0x7fae640d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae69bd7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640b3430 con 0x7fae640d27f0 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae69bd7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae640d2c50 con 0x7fae6415dfe0 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae69bd7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b31e0 con 0x7fae640cf830 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae6415dfe0 0x7fae640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae6415dfe0 0x7fae640cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40578/0 (socket says 172.21.15.122:40578) 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 -- 172.21.15.122:0/3980217755 learned_addr learned my addr 172.21.15.122:0/3980217755 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae5affd640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640cf830 0x7fae640d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 -- 172.21.15.122:0/3980217755 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640cf830 msgr2=0x7fae640d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640cf830 0x7fae640d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 -- 172.21.15.122:0/3980217755 >> v1:172.21.15.169:6789/0 conn(0x7fae640d27f0 legacy=0x7fae640d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae637fe640 1 -- 172.21.15.122:0/3980217755 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640b3710 con 0x7fae6415dfe0 2024-03-06T07:31:13.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.080+0000 7fae5affd640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640cf830 0x7fae640d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.081+0000 7fae637fe640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cf2f0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fae54009870 tx=0x7fae5402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4aec2cb88352770 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.081+0000 7fae62ffd640 1 -- 172.21.15.122:0/3980217755 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54035020 con 0x7fae6415dfe0 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae62ffd640 1 -- 172.21.15.122:0/3980217755 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae54031640 con 0x7fae6415dfe0 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae62ffd640 1 -- 172.21.15.122:0/3980217755 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5402d040 con 0x7fae6415dfe0 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 -- 172.21.15.122:0/3980217755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 msgr2=0x7fae640cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cf2f0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fae54009870 tx=0x7fae5402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 -- 172.21.15.122:0/3980217755 shutdown_connections 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640cf830 0x7fae640d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3980217755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cf2f0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 -- 172.21.15.122:0/3980217755 >> 172.21.15.122:0/3980217755 conn(0x7fae640d3cd0 msgr2=0x7fae640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 -- 172.21.15.122:0/3980217755 shutdown_connections 2024-03-06T07:31:13.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.082+0000 7fae69bd7640 1 -- 172.21.15.122:0/3980217755 wait complete. 2024-03-06T07:31:13.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.083+0000 7fae69bd7640 1 Processor -- start 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.083+0000 7fae69bd7640 1 -- start start 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae69bd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae640cf830 0x7fae64159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae69bd7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640d27f0 0x7fae640cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae69bd7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae69bd7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640d1c70 con 0x7fae6415dfe0 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae69bd7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b3d60 con 0x7fae640cf830 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae69bd7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640d1df0 con 0x7fae640d27f0 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40584/0 (socket says 172.21.15.122:40584) 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 -- 172.21.15.122:0/3949409253 learned_addr learned my addr 172.21.15.122:0/3949409253 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 -- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640d27f0 msgr2=0x7fae640cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640d27f0 0x7fae640cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 -- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae640cf830 msgr2=0x7fae64159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae637fe640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae640cf830 0x7fae64159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae640cf830 0x7fae64159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.084+0000 7fae63fff640 1 -- 172.21.15.122:0/3949409253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae641f6f80 con 0x7fae6415dfe0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.085+0000 7fae63fff640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cc040 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00c780 tx=0x7fae5c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.085+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c004b60 con 0x7fae6415dfe0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.085+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c004d00 con 0x7fae6415dfe0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.085+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae640cc6e0 con 0x7fae6415dfe0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.085+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c00aa90 con 0x7fae6415dfe0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.085+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641f7500 con 0x7fae6415dfe0 2024-03-06T07:31:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.086+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6404f330 con 0x7fae6415dfe0 2024-03-06T07:31:13.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.087+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae5c00ac30 con 0x7fae6415dfe0 2024-03-06T07:31:13.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.087+0000 7fae60ff9640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae4806b500 0x7fae4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.087+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fae5c08a430 con 0x7fae6415dfe0 2024-03-06T07:31:13.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.087+0000 7fae637fe640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae4806b500 0x7fae4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.088+0000 7fae637fe640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae4806b500 0x7fae4806d9c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fae54036000 tx=0x7fae54008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.092+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae5c020070 con 0x7fae6415dfe0 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.307+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae640747a0 con 0x7fae6415dfe0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.308+0000 7fae60ff9640 1 -- 172.21.15.122:0/3949409253 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fae5c059290 con 0x7fae6415dfe0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.309+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae4806b500 msgr2=0x7fae4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.309+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae4806b500 0x7fae4806d9c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fae54036000 tx=0x7fae54008030 comp rx=0 tx=0).stop 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 msgr2=0x7fae640cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cc040 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00c780 tx=0x7fae5c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 shutdown_connections 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae4806b500 0x7fae4806d9c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae6415dfe0 0x7fae640cc040 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae640d27f0 0x7fae640cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 --2- 172.21.15.122:0/3949409253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae640cf830 0x7fae64159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 >> 172.21.15.122:0/3949409253 conn(0x7fae640d3cd0 msgr2=0x7fae640b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 shutdown_connections 2024-03-06T07:31:13.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:13.310+0000 7fae69bd7640 1 -- 172.21.15.122:0/3949409253 wait complete. 2024-03-06T07:31:13.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:13.524 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:14.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:14.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.624+0000 7f6ac40a4640 1 Processor -- start 2024-03-06T07:31:14.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.624+0000 7f6ac40a4640 1 -- start start 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac40a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6abc06da40 0x7f6abc0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac40a4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cd7a0 0x7f6abc15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac40a4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0cc980 con 0x7f6abc1424f0 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac40a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0cd220 con 0x7f6abc06da40 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac40a4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0ccc60 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6abc06da40 0x7f6abc0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6abc06da40 0x7f6abc0cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36086/0 (socket says 172.21.15.122:36086) 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 -- 172.21.15.122:0/1510626011 learned_addr learned my addr 172.21.15.122:0/1510626011 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac2641640 1 --2- 172.21.15.122:0/1510626011 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cd7a0 0x7f6abc15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 -- 172.21.15.122:0/1510626011 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cd7a0 msgr2=0x7f6abc15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 --2- 172.21.15.122:0/1510626011 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cd7a0 0x7f6abc15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 -- 172.21.15.122:0/1510626011 >> v1:172.21.15.169:6789/0 conn(0x7f6abc1424f0 legacy=0x7f6abc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.625+0000 7f6ac163f640 1 -- 172.21.15.122:0/1510626011 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0ccf40 con 0x7f6abc06da40 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.626+0000 7f6ac163f640 1 --2- 172.21.15.122:0/1510626011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc06da40 0x7f6abc0cc2f0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f6abc068700 tx=0x7f6ab802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5dbbe7d84021082 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.626+0000 7f6ac0e3e640 1 -- 172.21.15.122:0/1510626011 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8035020 con 0x7f6abc06da40 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.626+0000 7f6ac0e3e640 1 -- 172.21.15.122:0/1510626011 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8031870 con 0x7f6abc06da40 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.626+0000 7f6ac0e3e640 1 -- 172.21.15.122:0/1510626011 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8031c40 con 0x7f6abc06da40 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 -- 172.21.15.122:0/1510626011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc06da40 msgr2=0x7f6abc0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/1510626011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc06da40 0x7f6abc0cc2f0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f6abc068700 tx=0x7f6ab802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 -- 172.21.15.122:0/1510626011 shutdown_connections 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/1510626011 >> v2:172.21.15.169:3300/0 conn(0x7f6abc0cd7a0 0x7f6abc15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/1510626011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc06da40 0x7f6abc0cc2f0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 -- 172.21.15.122:0/1510626011 >> 172.21.15.122:0/1510626011 conn(0x7f6abc0cf8a0 msgr2=0x7f6abc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 -- 172.21.15.122:0/1510626011 shutdown_connections 2024-03-06T07:31:14.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.627+0000 7f6ac40a4640 1 -- 172.21.15.122:0/1510626011 wait complete. 2024-03-06T07:31:14.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.629+0000 7f6ac40a4640 1 Processor -- start 2024-03-06T07:31:14.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.629+0000 7f6ac40a4640 1 -- start start 2024-03-06T07:31:14.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.629+0000 7f6ac40a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc06da40 0x7f6abc158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.629+0000 7f6ac40a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 0x7f6abc137120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac40a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc1424f0 0x7f6abc154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac40a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc071ce0 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac40a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc071b60 con 0x7f6abc1424f0 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac40a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc071e60 con 0x7f6abc06da40 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 0x7f6abc137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 0x7f6abc137120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36090/0 (socket says 172.21.15.122:36090) 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 -- 172.21.15.122:0/958194899 learned_addr learned my addr 172.21.15.122:0/958194899 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac1e40640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc06da40 0x7f6abc158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac2641640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc1424f0 0x7f6abc154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 -- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc06da40 msgr2=0x7f6abc158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc06da40 0x7f6abc158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 -- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc1424f0 msgr2=0x7f6abc154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc1424f0 0x7f6abc154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 -- 172.21.15.122:0/958194899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc137660 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac1e40640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc06da40 0x7f6abc158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac163f640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 0x7f6abc137120 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f6ab802b950 tx=0x7f6ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8040070 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8030d00 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc0b1080 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8035040 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.630+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc0b4510 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.631+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0b3750 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.632+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ab804b060 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.633+0000 7f6ab27fc640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b4d0 0x7f6aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.633+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6ab80abf90 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.633+0000 7f6ac1e40640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b4d0 0x7f6aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.633+0000 7f6ac1e40640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b4d0 0x7f6aa006d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00a830 tx=0x7f6aac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.637+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ab807ae80 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.845+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6abc04f330 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.846+0000 7f6ab27fc640 1 -- 172.21.15.122:0/958194899 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f6ab807a780 con 0x7f6abc0cd7a0 2024-03-06T07:31:14.848 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:14.848 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:14.848 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b4d0 msgr2=0x7f6aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b4d0 0x7f6aa006d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00a830 tx=0x7f6aac008040 comp rx=0 tx=0).stop 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 msgr2=0x7f6abc137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 0x7f6abc137120 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f6ab802b950 tx=0x7f6ab80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 shutdown_connections 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6aa006b4d0 0x7f6aa006d990 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6abc1424f0 0x7f6abc154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6abc0cd7a0 0x7f6abc137120 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 --2- 172.21.15.122:0/958194899 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6abc06da40 0x7f6abc158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.847+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 >> 172.21.15.122:0/958194899 conn(0x7f6abc0cf8a0 msgr2=0x7f6abc0721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.848+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 shutdown_connections 2024-03-06T07:31:14.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:14.848+0000 7f6ac40a4640 1 -- 172.21.15.122:0/958194899 wait complete. 2024-03-06T07:31:15.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:15.063 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:16.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:16.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.162+0000 7f1a3110d640 1 Processor -- start 2024-03-06T07:31:16.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.162+0000 7f1a3110d640 1 -- start start 2024-03-06T07:31:16.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a3110d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a2c0b4170 0x7f1a2c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a3110d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1a2c0d4760 0x7f1a2c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a3110d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0d25d0 con 0x7f1a2c0b4dd0 2024-03-06T07:31:16.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a3110d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0d4bf0 con 0x7f1a2c0b4170 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a3110d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0ce9e0 con 0x7f1a2c0d4760 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a2ad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a2c0b4170 0x7f1a2c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a2ad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1a2c0b4170 0x7f1a2c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36104/0 (socket says 172.21.15.122:36104) 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a2ad9d640 1 -- 172.21.15.122:0/1769254565 learned_addr learned my addr 172.21.15.122:0/1769254565 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.163+0000 7f1a2a59c640 1 --2- 172.21.15.122:0/1769254565 >> v2:172.21.15.169:3300/0 conn(0x7f1a2c0d4760 0x7f1a2c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a2ad9d640 1 -- 172.21.15.122:0/1769254565 >> v2:172.21.15.169:3300/0 conn(0x7f1a2c0d4760 msgr2=0x7f1a2c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a2ad9d640 1 --2- 172.21.15.122:0/1769254565 >> v2:172.21.15.169:3300/0 conn(0x7f1a2c0d4760 0x7f1a2c0d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a2ad9d640 1 -- 172.21.15.122:0/1769254565 >> v1:172.21.15.169:6789/0 conn(0x7f1a2c0b4dd0 legacy=0x7f1a2c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a2ad9d640 1 -- 172.21.15.122:0/1769254565 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0d2870 con 0x7f1a2c0b4170 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a2a59c640 1 --2- 172.21.15.122:0/1769254565 >> v2:172.21.15.169:3300/0 conn(0x7f1a2c0d4760 0x7f1a2c0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a2ad9d640 1 --2- 172.21.15.122:0/1769254565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c0b4170 0x7f1a2c0b6a90 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c08d0c0 tx=0x7f1a1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abaf4288cfb7d240 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a29d9b640 1 -- 172.21.15.122:0/1769254565 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c035020 con 0x7f1a2c0b4170 2024-03-06T07:31:16.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a29d9b640 1 -- 172.21.15.122:0/1769254565 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c031870 con 0x7f1a2c0b4170 2024-03-06T07:31:16.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.164+0000 7f1a29d9b640 1 -- 172.21.15.122:0/1769254565 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c031c40 con 0x7f1a2c0b4170 2024-03-06T07:31:16.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 -- 172.21.15.122:0/1769254565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c0b4170 msgr2=0x7f1a2c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 --2- 172.21.15.122:0/1769254565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c0b4170 0x7f1a2c0b6a90 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c08d0c0 tx=0x7f1a1c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:16.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 -- 172.21.15.122:0/1769254565 shutdown_connections 2024-03-06T07:31:16.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 --2- 172.21.15.122:0/1769254565 >> v2:172.21.15.169:3300/0 conn(0x7f1a2c0d4760 0x7f1a2c0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 --2- 172.21.15.122:0/1769254565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c0b4170 0x7f1a2c0b6a90 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 -- 172.21.15.122:0/1769254565 >> 172.21.15.122:0/1769254565 conn(0x7f1a2c0d3c40 msgr2=0x7f1a2c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 -- 172.21.15.122:0/1769254565 shutdown_connections 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.165+0000 7f1a3110d640 1 -- 172.21.15.122:0/1769254565 wait complete. 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.166+0000 7f1a3110d640 1 Processor -- start 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.166+0000 7f1a3110d640 1 -- start start 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a3110d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c0b4170 0x7f1a2c1f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a3110d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c0b4dd0 0x7f1a2c1f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a3110d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 0x7f1a2c15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a3110d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0b47d0 con 0x7f1a2c1571a0 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a3110d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0b4550 con 0x7f1a2c0b4dd0 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a3110d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0b4950 con 0x7f1a2c0b4170 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a2b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 0x7f1a2c15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a2b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 0x7f1a2c15b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36108/0 (socket says 172.21.15.122:36108) 2024-03-06T07:31:16.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a2a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c0b4dd0 0x7f1a2c1f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a2b59e640 1 -- 172.21.15.122:0/735773224 learned_addr learned my addr 172.21.15.122:0/735773224 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.167+0000 7f1a2ad9d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c0b4170 0x7f1a2c1f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a2b59e640 1 -- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c0b4170 msgr2=0x7f1a2c1f77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a2b59e640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c0b4170 0x7f1a2c1f77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a2b59e640 1 -- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c0b4dd0 msgr2=0x7f1a2c1f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a2b59e640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c0b4dd0 0x7f1a2c1f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a2b59e640 1 -- 172.21.15.122:0/735773224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c1f86c0 con 0x7f1a2c1571a0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a2b59e640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 0x7f1a2c15b520 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c7b0 tx=0x7f1a2400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24004b60 con 0x7f1a2c1571a0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c157650 con 0x7f1a2c1571a0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24004d00 con 0x7f1a2c1571a0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400aa90 con 0x7f1a2c1571a0 2024-03-06T07:31:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.168+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c1f8100 con 0x7f1a2c1571a0 2024-03-06T07:31:16.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.169+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c06b3a0 con 0x7f1a2c1571a0 2024-03-06T07:31:16.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.169+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a2400ac30 con 0x7f1a2c1571a0 2024-03-06T07:31:16.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.170+0000 7f1a137fe640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.170+0000 7f1a2ad9d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.170+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1a2408a410 con 0x7f1a2c1571a0 2024-03-06T07:31:16.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.170+0000 7f1a2ad9d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c036000 tx=0x7f1a1c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.174+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a2c06b3a0 con 0x7f1a2c1571a0 2024-03-06T07:31:16.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.392+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a2c04f330 con 0x7f1a2c1571a0 2024-03-06T07:31:16.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.392+0000 7f1a137fe640 1 -- 172.21.15.122:0/735773224 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f1a24020070 con 0x7f1a2c1571a0 2024-03-06T07:31:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:16.395 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:16.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 msgr2=0x7f1a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c036000 tx=0x7f1a1c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 msgr2=0x7f1a2c15b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 0x7f1a2c15b520 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c7b0 tx=0x7f1a2400cc70 comp rx=0 tx=0).stop 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 shutdown_connections 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1a1806b4d0 0x7f1a1806d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a2c1571a0 0x7f1a2c15b520 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a2c0b4dd0 0x7f1a2c1f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 --2- 172.21.15.122:0/735773224 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1a2c0b4170 0x7f1a2c1f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 >> 172.21.15.122:0/735773224 conn(0x7f1a2c0d3c40 msgr2=0x7f1a2c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 shutdown_connections 2024-03-06T07:31:16.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:16.394+0000 7f1a3110d640 1 -- 172.21.15.122:0/735773224 wait complete. 2024-03-06T07:31:16.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:16.612 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:17.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:17.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.712+0000 7f083c581640 1 Processor -- start 2024-03-06T07:31:17.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.712+0000 7f083c581640 1 -- start start 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083c581640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f083415dfe0 0x7f08340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083c581640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340cf830 0x7f08340d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083c581640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08340b3430 con 0x7f08340d27f0 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083c581640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08340d2c50 con 0x7f083415dfe0 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083c581640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08340b31e0 con 0x7f08340cf830 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f083415dfe0 0x7f08340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f083415dfe0 0x7f08340cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36124/0 (socket says 172.21.15.122:36124) 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 -- 172.21.15.122:0/607523986 learned_addr learned my addr 172.21.15.122:0/607523986 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f0839b1c640 1 --2- 172.21.15.122:0/607523986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340cf830 0x7f08340d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 -- 172.21.15.122:0/607523986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340cf830 msgr2=0x7f08340d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 --2- 172.21.15.122:0/607523986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340cf830 0x7f08340d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 -- 172.21.15.122:0/607523986 >> v1:172.21.15.169:6789/0 conn(0x7f08340d27f0 legacy=0x7f08340d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.713+0000 7f083a31d640 1 -- 172.21.15.122:0/607523986 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08340b3710 con 0x7f083415dfe0 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.714+0000 7f083a31d640 1 --2- 172.21.15.122:0/607523986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340cf2f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f0824009870 tx=0x7f082402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92877715ad4b87d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.714+0000 7f083931b640 1 -- 172.21.15.122:0/607523986 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0824034070 con 0x7f083415dfe0 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.714+0000 7f083931b640 1 -- 172.21.15.122:0/607523986 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0824030840 con 0x7f083415dfe0 2024-03-06T07:31:17.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.714+0000 7f083931b640 1 -- 172.21.15.122:0/607523986 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0824030c10 con 0x7f083415dfe0 2024-03-06T07:31:17.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 -- 172.21.15.122:0/607523986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 msgr2=0x7f08340cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 --2- 172.21.15.122:0/607523986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340cf2f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f0824009870 tx=0x7f082402b980 comp rx=0 tx=0).stop 2024-03-06T07:31:17.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 -- 172.21.15.122:0/607523986 shutdown_connections 2024-03-06T07:31:17.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 --2- 172.21.15.122:0/607523986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340cf830 0x7f08340d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 --2- 172.21.15.122:0/607523986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340cf2f0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 -- 172.21.15.122:0/607523986 >> 172.21.15.122:0/607523986 conn(0x7f08340d3cd0 msgr2=0x7f08340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 -- 172.21.15.122:0/607523986 shutdown_connections 2024-03-06T07:31:17.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.715+0000 7f083c581640 1 -- 172.21.15.122:0/607523986 wait complete. 2024-03-06T07:31:17.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.717+0000 7f083c581640 1 Processor -- start 2024-03-06T07:31:17.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.717+0000 7f083c581640 1 -- start start 2024-03-06T07:31:17.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083c581640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08340cf830 0x7f08341609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083c581640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340d27f0 0x7f08340b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083c581640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083c581640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340d1c70 con 0x7f083415dfe0 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083c581640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340b3d60 con 0x7f08340cf830 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083c581640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08340d1df0 con 0x7f08340d27f0 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083ab1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083ab1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36138/0 (socket says 172.21.15.122:36138) 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083ab1e640 1 -- 172.21.15.122:0/2920302174 learned_addr learned my addr 172.21.15.122:0/2920302174 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f0839b1c640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340d27f0 0x7f08340b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.718+0000 7f083a31d640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08340cf830 0x7f08341609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083ab1e640 1 -- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340d27f0 msgr2=0x7f08340b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083ab1e640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340d27f0 0x7f08340b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083ab1e640 1 -- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08340cf830 msgr2=0x7f08341609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083ab1e640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08340cf830 0x7f08341609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083ab1e640 1 -- 172.21.15.122:0/2920302174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083415f910 con 0x7f083415dfe0 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083ab1e640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340b6de0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f082c00c7d0 tx=0x7f082c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c003b70 con 0x7f083415dfe0 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c003d30 con 0x7f083415dfe0 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083415fbf0 con 0x7f083415dfe0 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00aa90 con 0x7f083415dfe0 2024-03-06T07:31:17.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.719+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08341601b0 con 0x7f083415dfe0 2024-03-06T07:31:17.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.720+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0834160ef0 con 0x7f083415dfe0 2024-03-06T07:31:17.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.721+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f082c00ac30 con 0x7f083415dfe0 2024-03-06T07:31:17.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.721+0000 7f082affd640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f081806b4d0 0x7f081806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.722+0000 7f083a31d640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f081806b4d0 0x7f081806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.722+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f082c089ff0 con 0x7f083415dfe0 2024-03-06T07:31:17.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.722+0000 7f083a31d640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f081806b4d0 0x7f081806d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f0824036000 tx=0x7f082403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:17.726+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f082c058800 con 0x7f083415dfe0 2024-03-06T07:31:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.035+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f083404f330 con 0x7f083415dfe0 2024-03-06T07:31:18.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.035+0000 7f082affd640 1 -- 172.21.15.122:0/2920302174 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f082c05c8a0 con 0x7f083415dfe0 2024-03-06T07:31:18.037 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:18.037 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f081806b4d0 msgr2=0x7f081806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f081806b4d0 0x7f081806d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f0824036000 tx=0x7f082403b040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 msgr2=0x7f08340b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340b6de0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f082c00c7d0 tx=0x7f082c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 shutdown_connections 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f081806b4d0 0x7f081806d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f083415dfe0 0x7f08340b6de0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f08340d27f0 0x7f08340b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 --2- 172.21.15.122:0/2920302174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08340cf830 0x7f08341609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 >> 172.21.15.122:0/2920302174 conn(0x7f08340d3cd0 msgr2=0x7f083406bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 shutdown_connections 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:18.037+0000 7f083c581640 1 -- 172.21.15.122:0/2920302174 wait complete. 2024-03-06T07:31:18.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:18.256 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:19.257 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:19.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 Processor -- start 2024-03-06T07:31:19.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 -- start start 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb0d80b4170 0x7fb0d80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb0d80d4760 0x7fb0d80d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80d25d0 con 0x7fb0d80b4dd0 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d80d4bf0 con 0x7fb0d80d4760 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.357+0000 7fb0ddb72640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d80ce9e0 con 0x7fb0d80b4170 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb0d80d4760 0x7fb0d80d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb0d80d4760 0x7fb0d80d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36148/0 (socket says 172.21.15.122:36148) 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 -- 172.21.15.122:0/1740043914 learned_addr learned my addr 172.21.15.122:0/1740043914 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d7fff640 1 --1- 172.21.15.122:0/1740043914 >> v1:172.21.15.122:6789/0 conn(0x7fb0d80b4dd0 0x7fb0d80cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53890/0 (socket says 172.21.15.122:53890) 2024-03-06T07:31:19.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d77fe640 1 --2- 172.21.15.122:0/1740043914 >> v2:172.21.15.169:3300/0 conn(0x7fb0d80b4170 0x7fb0d80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 -- 172.21.15.122:0/1740043914 >> v2:172.21.15.169:3300/0 conn(0x7fb0d80b4170 msgr2=0x7fb0d80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 --2- 172.21.15.122:0/1740043914 >> v2:172.21.15.169:3300/0 conn(0x7fb0d80b4170 0x7fb0d80b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 -- 172.21.15.122:0/1740043914 >> v1:172.21.15.122:6789/0 conn(0x7fb0d80b4dd0 legacy=0x7fb0d80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d67fc640 1 -- 172.21.15.122:0/1740043914 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473804465 0 0) 0x7fb0d80d25d0 con 0x7fb0d80b4dd0 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.358+0000 7fb0d6ffd640 1 -- 172.21.15.122:0/1740043914 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d80d2870 con 0x7fb0d80d4760 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.359+0000 7fb0d6ffd640 1 --2- 172.21.15.122:0/1740043914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d80d4760 0x7fb0d80d0370 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fb0d806e030 tx=0x7fb0cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16a0e658e09e39b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.359+0000 7fb0d67fc640 1 -- 172.21.15.122:0/1740043914 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc034070 con 0x7fb0d80d4760 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.359+0000 7fb0d67fc640 1 -- 172.21.15.122:0/1740043914 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc030860 con 0x7fb0d80d4760 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.359+0000 7fb0d67fc640 1 -- 172.21.15.122:0/1740043914 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc030c30 con 0x7fb0d80d4760 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 -- 172.21.15.122:0/1740043914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d80d4760 msgr2=0x7fb0d80d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/1740043914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d80d4760 0x7fb0d80d0370 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fb0d806e030 tx=0x7fb0cc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 -- 172.21.15.122:0/1740043914 shutdown_connections 2024-03-06T07:31:19.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/1740043914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d80d4760 0x7fb0d80d0370 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/1740043914 >> v2:172.21.15.169:3300/0 conn(0x7fb0d80b4170 0x7fb0d80b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 -- 172.21.15.122:0/1740043914 >> 172.21.15.122:0/1740043914 conn(0x7fb0d80d3c40 msgr2=0x7fb0d80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 -- 172.21.15.122:0/1740043914 shutdown_connections 2024-03-06T07:31:19.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.360+0000 7fb0ddb72640 1 -- 172.21.15.122:0/1740043914 wait complete. 2024-03-06T07:31:19.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.361+0000 7fb0ddb72640 1 Processor -- start 2024-03-06T07:31:19.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.361+0000 7fb0ddb72640 1 -- start start 2024-03-06T07:31:19.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0d80b4170 0x7fb0d8157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb0d80b4dd0 0x7fb0d80b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 0x7fb0d80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80b47d0 con 0x7fb0d81583b0 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80b4550 con 0x7fb0d80b4170 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80b4950 con 0x7fb0d80b4dd0 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 0x7fb0d80b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 0x7fb0d80b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36150/0 (socket says 172.21.15.122:36150) 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 -- 172.21.15.122:0/2013044428 learned_addr learned my addr 172.21.15.122:0/2013044428 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d77fe640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0d80b4170 0x7fb0d8157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d6ffd640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb0d80b4dd0 0x7fb0d80b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 -- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb0d80b4dd0 msgr2=0x7fb0d80b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb0d80b4dd0 0x7fb0d80b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 -- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0d80b4170 msgr2=0x7fb0d8157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0d80b4170 0x7fb0d8157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 -- 172.21.15.122:0/2013044428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d80b3710 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d6ffd640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb0d80b4dd0 0x7fb0d80b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0d7fff640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 0x7fb0d80b2380 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c7d0 tx=0x7fb0d000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0018070 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d000a640 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.362+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d8158060 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.363+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000aa70 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.363+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d80b3430 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.364+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d804f330 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.364+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0d001b070 con 0x7fb0d81583b0 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.364+0000 7fb0bffff640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0c406b500 0x7fb0c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.365+0000 7fb0d77fe640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0c406b500 0x7fb0c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.365+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb0d00896d0 con 0x7fb0d81583b0 2024-03-06T07:31:19.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.365+0000 7fb0d77fe640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0c406b500 0x7fb0c406d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8006fd0 tx=0x7fb0c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.369+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0d0057f70 con 0x7fb0d81583b0 2024-03-06T07:31:19.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.584+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d80733b0 con 0x7fb0d81583b0 2024-03-06T07:31:19.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.585+0000 7fb0bffff640 1 -- 172.21.15.122:0/2013044428 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fb0d005c010 con 0x7fb0d81583b0 2024-03-06T07:31:19.587 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:19.587 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:19.588 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0c406b500 msgr2=0x7fb0c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0c406b500 0x7fb0c406d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8006fd0 tx=0x7fb0c8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 msgr2=0x7fb0d80b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 0x7fb0d80b2380 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c7d0 tx=0x7fb0d000cc90 comp rx=0 tx=0).stop 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 shutdown_connections 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb0c406b500 0x7fb0c406d9c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0d81583b0 0x7fb0d80b2380 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb0d80b4dd0 0x7fb0d80b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 --2- 172.21.15.122:0/2013044428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0d80b4170 0x7fb0d8157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.586+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 >> 172.21.15.122:0/2013044428 conn(0x7fb0d80d3c40 msgr2=0x7fb0d80b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.587+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 shutdown_connections 2024-03-06T07:31:19.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:19.587+0000 7fb0ddb72640 1 -- 172.21.15.122:0/2013044428 wait complete. 2024-03-06T07:31:19.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:19.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:20.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:20.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.901+0000 7f387e8b5640 1 Processor -- start 2024-03-06T07:31:20.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.901+0000 7f387e8b5640 1 -- start start 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387e8b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f387813dc10 0x7f38780b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387e8b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f387815ef60 0x7f38780b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387e8b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387815f580 con 0x7f387815bf30 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387e8b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38780cf3f0 con 0x7f387815ef60 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387e8b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387815b210 con 0x7f387813dc10 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f387815ef60 0x7f38780b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f387815ef60 0x7f38780b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36152/0 (socket says 172.21.15.122:36152) 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 -- 172.21.15.122:0/2448914228 learned_addr learned my addr 172.21.15.122:0/2448914228 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f3877fff640 1 --2- 172.21.15.122:0/2448914228 >> v2:172.21.15.169:3300/0 conn(0x7f387813dc10 0x7f38780b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 -- 172.21.15.122:0/2448914228 >> v2:172.21.15.169:3300/0 conn(0x7f387813dc10 msgr2=0x7f38780b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 --2- 172.21.15.122:0/2448914228 >> v2:172.21.15.169:3300/0 conn(0x7f387813dc10 0x7f38780b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 -- 172.21.15.122:0/2448914228 >> v1:172.21.15.169:6789/0 conn(0x7f387815bf30 legacy=0x7f387815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.902+0000 7f387ce52640 1 -- 172.21.15.122:0/2448914228 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387815b450 con 0x7f387815ef60 2024-03-06T07:31:20.906 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.903+0000 7f3877fff640 1 --2- 172.21.15.122:0/2448914228 >> v2:172.21.15.169:3300/0 conn(0x7f387813dc10 0x7f38780b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:20.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.903+0000 7f387ce52640 1 --2- 172.21.15.122:0/2448914228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387815ef60 0x7f38780b2e90 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f38780731a0 tx=0x7f387002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e87c62c2de6af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.903+0000 7f3876ffd640 1 -- 172.21.15.122:0/2448914228 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387002ca80 con 0x7f387815ef60 2024-03-06T07:31:20.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.903+0000 7f3876ffd640 1 -- 172.21.15.122:0/2448914228 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387002cc20 con 0x7f387815ef60 2024-03-06T07:31:20.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.903+0000 7f3876ffd640 1 -- 172.21.15.122:0/2448914228 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38700343f0 con 0x7f387815ef60 2024-03-06T07:31:20.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 -- 172.21.15.122:0/2448914228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387815ef60 msgr2=0x7f38780b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 --2- 172.21.15.122:0/2448914228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387815ef60 0x7f38780b2e90 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f38780731a0 tx=0x7f387002bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 -- 172.21.15.122:0/2448914228 shutdown_connections 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 --2- 172.21.15.122:0/2448914228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387815ef60 0x7f38780b2e90 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 --2- 172.21.15.122:0/2448914228 >> v2:172.21.15.169:3300/0 conn(0x7f387813dc10 0x7f38780b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 -- 172.21.15.122:0/2448914228 >> 172.21.15.122:0/2448914228 conn(0x7f38780cf8a0 msgr2=0x7f38780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 -- 172.21.15.122:0/2448914228 shutdown_connections 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.904+0000 7f387e8b5640 1 -- 172.21.15.122:0/2448914228 wait complete. 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.905+0000 7f387e8b5640 1 Processor -- start 2024-03-06T07:31:20.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.905+0000 7f387e8b5640 1 -- start start 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f387e8b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 0x7f3878154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f387e8b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f387815bf30 0x7f38781436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f387e8b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387815ef60 0x7f3878143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f387e8b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38780708d0 con 0x7f387813dc10 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f387e8b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387813dff0 con 0x7f387815ef60 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f387e8b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3878070a50 con 0x7f387815bf30 2024-03-06T07:31:20.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 0x7f3878154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 0x7f3878154730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36168/0 (socket says 172.21.15.122:36168) 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 -- 172.21.15.122:0/1497798018 learned_addr learned my addr 172.21.15.122:0/1497798018 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 -- 172.21.15.122:0/1497798018 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f387815bf30 msgr2=0x7f38781436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f387815bf30 0x7f38781436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 -- 172.21.15.122:0/1497798018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387815ef60 msgr2=0x7f3878143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387815ef60 0x7f3878143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 -- 172.21.15.122:0/1497798018 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38781f43b0 con 0x7f387813dc10 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.906+0000 7f3877fff640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 0x7f3878154730 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f386c00e7e0 tx=0x7f386c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.907+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c00f040 con 0x7f387813dc10 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.907+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386c004030 con 0x7f387813dc10 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.907+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38781f4690 con 0x7f387813dc10 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.907+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c004400 con 0x7f387813dc10 2024-03-06T07:31:20.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.907+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38781f4080 con 0x7f387813dc10 2024-03-06T07:31:20.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.908+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387804f330 con 0x7f387813dc10 2024-03-06T07:31:20.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.908+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f386c00c6d0 con 0x7f387813dc10 2024-03-06T07:31:20.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.909+0000 7f3874ff9640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.912 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.909+0000 7f38777fe640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.909+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f386c089fc0 con 0x7f387813dc10 2024-03-06T07:31:20.913 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.909+0000 7f38777fe640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f386406b4d0 0x7f386406d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f3868000f60 tx=0x7f38680079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.914 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:20.913+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f386c017020 con 0x7f387813dc10 2024-03-06T07:31:21.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.126+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f387806eb00 con 0x7f387813dc10 2024-03-06T07:31:21.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.126+0000 7f3874ff9640 1 -- 172.21.15.122:0/1497798018 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f386c058eb0 con 0x7f387813dc10 2024-03-06T07:31:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:21.129 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:21.129 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f386406b4d0 msgr2=0x7f386406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f386406b4d0 0x7f386406d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f3868000f60 tx=0x7f38680079b0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 msgr2=0x7f3878154730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 0x7f3878154730 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f386c00e7e0 tx=0x7f386c00eca0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 shutdown_connections 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f387815ef60 0x7f3878143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f387815bf30 0x7f38781436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 --2- 172.21.15.122:0/1497798018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f387813dc10 0x7f3878154730 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 >> 172.21.15.122:0/1497798018 conn(0x7f38780cf8a0 msgr2=0x7f38780af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 shutdown_connections 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:21.129+0000 7f387e8b5640 1 -- 172.21.15.122:0/1497798018 wait complete. 2024-03-06T07:31:21.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:21.343 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:22.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.449+0000 7f0ada083640 1 Processor -- start 2024-03-06T07:31:22.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ada083640 1 -- start start 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ada083640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ad40d2710 0x7f0ad40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ada083640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40b3430 con 0x7f0ad40d22b0 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ada083640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40b3710 con 0x7f0ad415dfe0 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ada083640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad40b31e0 con 0x7f0ad40d2710 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ad3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ad40d2710 0x7f0ad40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0ad3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ad40d2710 0x7f0ad40b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36180/0 (socket says 172.21.15.122:36180) 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.451+0000 7f0ad3fff640 1 -- 172.21.15.122:0/4246192876 learned_addr learned my addr 172.21.15.122:0/4246192876 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.450+0000 7f0acbfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0ad40d22b0 0x7f0ad40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53904/0 (socket says 172.21.15.122:53904) 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.451+0000 7f0ad3fff640 1 -- 172.21.15.122:0/4246192876 >> v1:172.21.15.169:6789/0 conn(0x7f0ad415dfe0 legacy=0x7f0ad40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.451+0000 7f0ad3fff640 1 -- 172.21.15.122:0/4246192876 >> v1:172.21.15.122:6789/0 conn(0x7f0ad40d22b0 legacy=0x7f0ad40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.451+0000 7f0ad3fff640 1 -- 172.21.15.122:0/4246192876 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40b39f0 con 0x7f0ad40d2710 2024-03-06T07:31:22.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.451+0000 7f0ad2ffd640 1 -- 172.21.15.122:0/4246192876 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198009114 0 0) 0x7f0ad40b3430 con 0x7f0ad40d22b0 2024-03-06T07:31:22.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.452+0000 7f0ad3fff640 1 --2- 172.21.15.122:0/4246192876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad40d2710 0x7f0ad40b4900 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009e80 tx=0x7f0acc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8232739e73c196c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.452+0000 7f0ad2ffd640 1 -- 172.21.15.122:0/4246192876 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc02caf0 con 0x7f0ad40d2710 2024-03-06T07:31:22.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.452+0000 7f0ad2ffd640 1 -- 172.21.15.122:0/4246192876 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc02cc90 con 0x7f0ad40d2710 2024-03-06T07:31:22.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.452+0000 7f0ad2ffd640 1 -- 172.21.15.122:0/4246192876 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc034610 con 0x7f0ad40d2710 2024-03-06T07:31:22.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 -- 172.21.15.122:0/4246192876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad40d2710 msgr2=0x7f0ad40b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 --2- 172.21.15.122:0/4246192876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad40d2710 0x7f0ad40b4900 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009e80 tx=0x7f0acc02bf50 comp rx=0 tx=0).stop 2024-03-06T07:31:22.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 -- 172.21.15.122:0/4246192876 shutdown_connections 2024-03-06T07:31:22.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 --2- 172.21.15.122:0/4246192876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad40d2710 0x7f0ad40b4900 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 -- 172.21.15.122:0/4246192876 >> 172.21.15.122:0/4246192876 conn(0x7f0ad40d3cd0 msgr2=0x7f0ad40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 -- 172.21.15.122:0/4246192876 shutdown_connections 2024-03-06T07:31:22.455 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.453+0000 7f0ada083640 1 -- 172.21.15.122:0/4246192876 wait complete. 2024-03-06T07:31:22.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.454+0000 7f0ada083640 1 Processor -- start 2024-03-06T07:31:22.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.454+0000 7f0ada083640 1 -- start start 2024-03-06T07:31:22.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ad40d22b0 0x7f0ad40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ad40d2710 0x7f0ad4160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 0x7f0ad4152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40cfa00 con 0x7f0ad415dfe0 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40b3d60 con 0x7f0ad40d22b0 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40d1c70 con 0x7f0ad40d2710 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 0x7f0ad4152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 0x7f0ad4152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36190/0 (socket says 172.21.15.122:36190) 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 -- 172.21.15.122:0/2372906953 learned_addr learned my addr 172.21.15.122:0/2372906953 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0acbfff640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ad40d2710 0x7f0ad4160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad37fe640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ad40d22b0 0x7f0ad40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 -- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ad40d2710 msgr2=0x7f0ad4160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ad40d2710 0x7f0ad4160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 -- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ad40d22b0 msgr2=0x7f0ad40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ad40d22b0 0x7f0ad40b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 -- 172.21.15.122:0/2372906953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad4153020 con 0x7f0ad415dfe0 2024-03-06T07:31:22.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad3fff640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 0x7f0ad4152ae0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009fa0 tx=0x7f0acc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc043020 con 0x7f0ad415dfe0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc03e450 con 0x7f0ad415dfe0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad40b6fd0 con 0x7f0ad415dfe0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc009510 con 0x7f0ad415dfe0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.455+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad40b7590 con 0x7f0ad415dfe0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.456+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad415db40 con 0x7f0ad415dfe0 2024-03-06T07:31:22.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.457+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0acc046020 con 0x7f0ad415dfe0 2024-03-06T07:31:22.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.457+0000 7f0ad0ff9640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ab406b500 0x7f0ab406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.457+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0acc0ac870 con 0x7f0ad415dfe0 2024-03-06T07:31:22.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.458+0000 7f0ad37fe640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ab406b500 0x7f0ab406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.458+0000 7f0ad37fe640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ab406b500 0x7f0ab406d9c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400a730 tx=0x7f0ac4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.462+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ad415db40 con 0x7f0ad415dfe0 2024-03-06T07:31:22.693 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.692+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad40671f0 con 0x7f0ad415dfe0 2024-03-06T07:31:22.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.693+0000 7f0ad0ff9640 1 -- 172.21.15.122:0/2372906953 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f0ad40671f0 con 0x7f0ad415dfe0 2024-03-06T07:31:22.694 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:22.694 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:22.694 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ab406b500 msgr2=0x7f0ab406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ab406b500 0x7f0ab406d9c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400a730 tx=0x7f0ac4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 msgr2=0x7f0ad4152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 0x7f0ad4152ae0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009fa0 tx=0x7f0acc004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 shutdown_connections 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0ab406b500 0x7f0ab406d9c0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad415dfe0 0x7f0ad4152ae0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ad40d2710 0x7f0ad4160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.695+0000 7f0ada083640 1 --2- 172.21.15.122:0/2372906953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ad40d22b0 0x7f0ad40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.696+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 >> 172.21.15.122:0/2372906953 conn(0x7f0ad40d3cd0 msgr2=0x7f0ad40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.696+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 shutdown_connections 2024-03-06T07:31:22.698 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:22.696+0000 7f0ada083640 1 -- 172.21.15.122:0/2372906953 wait complete. 2024-03-06T07:31:22.913 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:22.913 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:23.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.014+0000 7fbdc41f7640 1 Processor -- start 2024-03-06T07:31:24.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.014+0000 7fbdc41f7640 1 -- start start 2024-03-06T07:31:24.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc41f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 0x7fbdbc1603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc41f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc0cfc90 con 0x7fbdbc15c040 2024-03-06T07:31:24.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc41f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc15c530 con 0x7fbdbc0cf6f0 2024-03-06T07:31:24.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc41f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1608e0 con 0x7fbdbc0d05e0 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc1f93640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbdbc15c040 0x7fbdbc0cfed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53912/0 (socket says 172.21.15.122:53912) 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc1f93640 1 -- 172.21.15.122:0/1792711658 learned_addr learned my addr 172.21.15.122:0/1792711658 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc1792640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 0x7fbdbc1603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245034652 0 0) 0x7fbdbc0cfc90 con 0x7fbdbc15c040 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda8003520 con 0x7fbdbc15c040 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663916375 0 0) 0x7fbda8003520 con 0x7fbdbc15c040 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 msgr2=0x7fbdbc1603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 0x7fbdbc1603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 >> v1:172.21.15.169:6789/0 conn(0x7fbdbc0cf6f0 legacy=0x7fbdbc0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc15c810 con 0x7fbdbc15c040 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fbdb8001670 con 0x7fbdbc15c040 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 >> v1:172.21.15.122:6789/0 conn(0x7fbdbc15c040 legacy=0x7fbdbc0cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda80039e0 0x7fbda8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda8004390 0x7fbda8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 0x7fbda80050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1608e0 con 0x7fbda80039e0 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda800b9c0 con 0x7fbda8004390 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.015+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda800bbc0 con 0x7fbdbc0d05e0 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda80039e0 0x7fbda8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc2794640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 0x7fbda80050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1792640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda8004390 0x7fbda8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1f93640 1 -- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 msgr2=0x7fbda80050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbdbc0d05e0 0x7fbda80050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1f93640 1 -- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda8004390 msgr2=0x7fbda8004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda8004390 0x7fbda8004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.016+0000 7fbdc1f93640 1 -- 172.21.15.122:0/1792711658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda8003520 con 0x7fbda80039e0 2024-03-06T07:31:24.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda80039e0 0x7fbda8003dc0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800c290 tx=0x7fbda8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5647fc5fc6f572da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8007d50 con 0x7fbda80039e0 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb8030070 con 0x7fbda80039e0 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc0f91640 1 -- 172.21.15.122:0/1792711658 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb802cd80 con 0x7fbda80039e0 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc41f7640 1 -- 172.21.15.122:0/1792711658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda80039e0 msgr2=0x7fbda8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/1792711658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda80039e0 0x7fbda8003dc0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800c290 tx=0x7fbda8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc2794640 1 -- 172.21.15.122:0/1792711658 reap_dead start 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc41f7640 1 -- 172.21.15.122:0/1792711658 shutdown_connections 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc41f7640 1 -- 172.21.15.122:0/1792711658 >> 172.21.15.122:0/1792711658 conn(0x7fbdbc0d1b20 msgr2=0x7fbdbc0d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc41f7640 1 -- 172.21.15.122:0/1792711658 shutdown_connections 2024-03-06T07:31:24.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.017+0000 7fbdc41f7640 1 -- 172.21.15.122:0/1792711658 wait complete. 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.018+0000 7fbdc41f7640 1 Processor -- start 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.018+0000 7fbdc41f7640 1 -- start start 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc41f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda80039e0 0x7fbdbc15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc41f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 0x7fbdbc14cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc41f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbdbc0b63e0 0x7fbdbc0b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc41f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc15d2f0 con 0x7fbdbc14cad0 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc41f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc15cdd0 con 0x7fbdbc0b63e0 2024-03-06T07:31:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc41f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1603a0 con 0x7fbda80039e0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 0x7fbdbc14cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 0x7fbdbc14cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36210/0 (socket says 172.21.15.122:36210) 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 -- 172.21.15.122:0/2936485694 learned_addr learned my addr 172.21.15.122:0/2936485694 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda80039e0 0x7fbdbc15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc2794640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbdbc0b63e0 0x7fbdbc0b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 -- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda80039e0 msgr2=0x7fbdbc15ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda80039e0 0x7fbdbc15ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 -- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbdbc0b63e0 msgr2=0x7fbdbc0b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbdbc0b63e0 0x7fbdbc0b6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.019+0000 7fbdc1792640 1 -- 172.21.15.122:0/2936485694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc14d490 con 0x7fbdbc14cad0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.020+0000 7fbdc1792640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 0x7fbdbc14cf50 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fbdac011910 tx=0x7fbdac011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.020+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac00fa90 con 0x7fbdbc14cad0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.020+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac00fc30 con 0x7fbdbc14cad0 2024-03-06T07:31:24.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.020+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc0b47e0 con 0x7fbdbc14cad0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.020+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac014440 con 0x7fbdbc14cad0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.020+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc0b4da0 con 0x7fbdbc14cad0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.021+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc04f330 con 0x7fbdbc14cad0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.021+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdac012040 con 0x7fbdbc14cad0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.021+0000 7fbdb27fc640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbda006b4d0 0x7fbda006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.022+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbdac08a750 con 0x7fbdbc14cad0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.022+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbda006b4d0 0x7fbda006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.022+0000 7fbdc1f93640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbda006b4d0 0x7fbda006d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800c290 tx=0x7fbdb8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.026+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbdac019030 con 0x7fbdbc14cad0 2024-03-06T07:31:24.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.239+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdbc06e850 con 0x7fbdbc14cad0 2024-03-06T07:31:24.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.240+0000 7fbdb27fc640 1 -- 172.21.15.122:0/2936485694 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fbdac0595b0 con 0x7fbdbc14cad0 2024-03-06T07:31:24.242 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:24.242 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:24.243 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:24.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbda006b4d0 msgr2=0x7fbda006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbda006b4d0 0x7fbda006d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800c290 tx=0x7fbdb8009030 comp rx=0 tx=0).stop 2024-03-06T07:31:24.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 msgr2=0x7fbdbc14cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 0x7fbdbc14cf50 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fbdac011910 tx=0x7fbdac011dd0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 shutdown_connections 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbda006b4d0 0x7fbda006d990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbdbc0b63e0 0x7fbdbc0b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbdbc14cad0 0x7fbdbc14cf50 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 --2- 172.21.15.122:0/2936485694 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda80039e0 0x7fbdbc15ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 >> 172.21.15.122:0/2936485694 conn(0x7fbdbc0d1b20 msgr2=0x7fbdbc15d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 shutdown_connections 2024-03-06T07:31:24.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:24.242+0000 7fbdc41f7640 1 -- 172.21.15.122:0/2936485694 wait complete. 2024-03-06T07:31:24.456 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:24.456 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:25.389 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:25.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:25.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:25.557 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 Processor -- start 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 -- start start 2024-03-06T07:31:25.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa20813dc10 0x7fa2080b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815bf30 0x7fa2080b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20815f580 con 0x7fa20815e850 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2080cf3f0 con 0x7fa20813dc10 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.556+0000 7fa20d97a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20815b210 con 0x7fa20815bf30 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa20813dc10 0x7fa2080b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa20813dc10 0x7fa2080b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47262/0 (socket says 172.21.15.122:47262) 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 -- 172.21.15.122:0/3306745106 learned_addr learned my addr 172.21.15.122:0/3306745106 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa2067fc640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815bf30 0x7fa2080b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 -- 172.21.15.122:0/3306745106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815bf30 msgr2=0x7fa2080b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815bf30 0x7fa2080b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 -- 172.21.15.122:0/3306745106 >> v1:172.21.15.169:6789/0 conn(0x7fa20815e850 legacy=0x7fa20815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa206ffd640 1 -- 172.21.15.122:0/3306745106 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20815b450 con 0x7fa20813dc10 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.557+0000 7fa2067fc640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815bf30 0x7fa2080b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa206ffd640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa2080b2950 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c020b42a45bef58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa205ffb640 1 -- 172.21.15.122:0/3306745106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80317c0 con 0x7fa20813dc10 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa205ffb640 1 -- 172.21.15.122:0/3306745106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8037070 con 0x7fa20813dc10 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa205ffb640 1 -- 172.21.15.122:0/3306745106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8031b80 con 0x7fa20813dc10 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 -- 172.21.15.122:0/3306745106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 msgr2=0x7fa2080b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa2080b2950 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f802b860 comp rx=0 tx=0).stop 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 -- 172.21.15.122:0/3306745106 shutdown_connections 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815bf30 0x7fa2080b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 --2- 172.21.15.122:0/3306745106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa2080b2950 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 -- 172.21.15.122:0/3306745106 >> 172.21.15.122:0/3306745106 conn(0x7fa2080cf8a0 msgr2=0x7fa2080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 -- 172.21.15.122:0/3306745106 shutdown_connections 2024-03-06T07:31:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.558+0000 7fa20d97a640 1 -- 172.21.15.122:0/3306745106 wait complete. 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.560+0000 7fa20d97a640 1 Processor -- start 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.560+0000 7fa20d97a640 1 -- start start 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa20d97a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa20813ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa20d97a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa20815bf30 0x7fa2081edd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa20d97a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815e850 0x7fa2081f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa20d97a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080708d0 con 0x7fa20813dc10 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa20d97a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20813dff0 con 0x7fa20815bf30 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa20d97a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa208070a50 con 0x7fa20815e850 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa206ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa20813ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa206ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa20813ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47264/0 (socket says 172.21.15.122:47264) 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.561+0000 7fa206ffd640 1 -- 172.21.15.122:0/1936814362 learned_addr learned my addr 172.21.15.122:0/1936814362 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa2067fc640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa20815bf30 0x7fa2081edd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa2077fe640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815e850 0x7fa2081f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa206ffd640 1 -- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815e850 msgr2=0x7fa2081f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa206ffd640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815e850 0x7fa2081f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa206ffd640 1 -- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa20815bf30 msgr2=0x7fa2081edd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa206ffd640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa20815bf30 0x7fa2081edd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa206ffd640 1 -- 172.21.15.122:0/1936814362 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2081ee260 con 0x7fa20813dc10 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa2077fe640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815e850 0x7fa2081f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa2067fc640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa20815bf30 0x7fa2081edd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa206ffd640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa20813ef20 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8032860 con 0x7fa20813dc10 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8032a00 con 0x7fa20813dc10 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20815c560 con 0x7fa20813dc10 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8032df0 con 0x7fa20813dc10 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.562+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2081f4860 con 0x7fa20813dc10 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.563+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20804f330 con 0x7fa20813dc10 2024-03-06T07:31:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.564+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1f804c060 con 0x7fa20813dc10 2024-03-06T07:31:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.564+0000 7fa1effff640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1f406b4d0 0x7fa1f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.564+0000 7fa2067fc640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1f406b4d0 0x7fa1f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.564+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa1f80abac0 con 0x7fa20813dc10 2024-03-06T07:31:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.565+0000 7fa2067fc640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1f406b4d0 0x7fa1f406d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa20815a9a0 tx=0x7fa1fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.571 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.569+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1f8040070 con 0x7fa20813dc10 2024-03-06T07:31:25.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.777+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa20806eb00 con 0x7fa20813dc10 2024-03-06T07:31:25.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.778+0000 7fa1effff640 1 -- 172.21.15.122:0/1936814362 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fa1f807a9b0 con 0x7fa20813dc10 2024-03-06T07:31:25.780 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:25.780 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:25.780 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1f406b4d0 msgr2=0x7fa1f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1f406b4d0 0x7fa1f406d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa20815a9a0 tx=0x7fa1fc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 msgr2=0x7fa20813ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa20813ef20 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 shutdown_connections 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1f406b4d0 0x7fa1f406d990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa20815e850 0x7fa2081f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa20815bf30 0x7fa2081edd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 --2- 172.21.15.122:0/1936814362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa20813dc10 0x7fa20813ef20 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 >> 172.21.15.122:0/1936814362 conn(0x7fa2080cf8a0 msgr2=0x7fa2080b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 shutdown_connections 2024-03-06T07:31:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:25.780+0000 7fa20d97a640 1 -- 172.21.15.122:0/1936814362 wait complete. 2024-03-06T07:31:25.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:25.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:26.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:27.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.094+0000 7f1671ecf640 1 Processor -- start 2024-03-06T07:31:27.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.094+0000 7f1671ecf640 1 -- start start 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f1671ecf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 0x7f166c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f1671ecf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c0b3430 con 0x7f166c15dfe0 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f1671ecf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c0b3710 con 0x7f166c0d22b0 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f1671ecf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b31e0 con 0x7f166c0d2710 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f166c15dfe0 0x7f166c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37044/0 (socket says 172.21.15.122:37044) 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166b7fe640 1 -- 172.21.15.122:0/3738772273 learned_addr learned my addr 172.21.15.122:0/3738772273 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166bfff640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 0x7f166c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472170016 0 0) 0x7f166c0b3430 con 0x7f166c15dfe0 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c003520 con 0x7f166c15dfe0 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 517097926 0 0) 0x7f164c003520 con 0x7f166c15dfe0 2024-03-06T07:31:27.100 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 msgr2=0x7f166c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 0x7f166c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 >> v1:172.21.15.169:6789/0 conn(0x7f166c0d22b0 legacy=0x7f166c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c0b39f0 con 0x7f166c15dfe0 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.095+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f165c001670 con 0x7f166c15dfe0 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 >> v1:172.21.15.122:6789/0 conn(0x7f166c15dfe0 legacy=0x7f166c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0039e0 0x7f164c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f164c0043b0 0x7f164c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 0x7f164c0050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b31e0 con 0x7f164c0043b0 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c00ba20 con 0x7f164c0039e0 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164c00bc20 con 0x7f166c0d2710 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f164c0043b0 0x7f164c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.101 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166b7fe640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0039e0 0x7f164c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f166bfff640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 0x7f164c0050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f1662ffd640 1 -- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 msgr2=0x7f164c0050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0d2710 0x7f164c0050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f1662ffd640 1 -- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0039e0 msgr2=0x7f164c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0039e0 0x7f164c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.096+0000 7f1662ffd640 1 -- 172.21.15.122:0/3738772273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c003520 con 0x7f164c0043b0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f164c0043b0 0x7f164c004880 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f1654001860 tx=0x7f164c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d0e873ed5795fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654038070 con 0x7f164c0043b0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654033410 con 0x7f164c0043b0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f166affd640 1 -- 172.21.15.122:0/3738772273 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654001430 con 0x7f164c0043b0 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1671ecf640 1 -- 172.21.15.122:0/3738772273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f164c0043b0 msgr2=0x7f164c004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3738772273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f164c0043b0 0x7f164c004880 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f1654001860 tx=0x7f164c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:27.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f166bfff640 1 -- 172.21.15.122:0/3738772273 reap_dead start 2024-03-06T07:31:27.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1671ecf640 1 -- 172.21.15.122:0/3738772273 shutdown_connections 2024-03-06T07:31:27.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1671ecf640 1 -- 172.21.15.122:0/3738772273 >> 172.21.15.122:0/3738772273 conn(0x7f166c0d3cd0 msgr2=0x7f166c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1671ecf640 1 -- 172.21.15.122:0/3738772273 shutdown_connections 2024-03-06T07:31:27.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.097+0000 7f1671ecf640 1 -- 172.21.15.122:0/3738772273 wait complete. 2024-03-06T07:31:27.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.099+0000 7f1671ecf640 1 Processor -- start 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.099+0000 7f1671ecf640 1 -- start start 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1671ecf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0043b0 0x7f166c0b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1671ecf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 0x7f166c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1671ecf640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0b1390 0x7f166c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1671ecf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0cfa00 con 0x7f166c15c2d0 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1671ecf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b3d60 con 0x7f164c0043b0 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1671ecf640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0d1c70 con 0x7f166c0b1390 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 0x7f166c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1662ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 0x7f166c0b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47296/0 (socket says 172.21.15.122:47296) 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f1662ffd640 1 -- 172.21.15.122:0/3623246789 learned_addr learned my addr 172.21.15.122:0/3623246789 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f166b7fe640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0043b0 0x7f166c0b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.100+0000 7f166bfff640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0b1390 0x7f166c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1662ffd640 1 -- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0b1390 msgr2=0x7f166c0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0b1390 0x7f166c0b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1662ffd640 1 -- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0043b0 msgr2=0x7f166c0b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0043b0 0x7f166c0b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1662ffd640 1 -- 172.21.15.122:0/3623246789 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c0b75c0 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1662ffd640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 0x7f166c0b0e50 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f1654001980 tx=0x7f1654001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654041020 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654006e00 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654038040 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c0b78a0 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.101+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c0b7e60 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.102+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c04f330 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.102+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1654044020 con 0x7f166c15c2d0 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.103+0000 7f1668ff9640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f165006b500 0x7f165006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.103+0000 7f166b7fe640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f165006b500 0x7f165006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.103+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f16540ab3a0 con 0x7f166c15c2d0 2024-03-06T07:31:27.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.103+0000 7f166b7fe640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f165006b500 0x7f165006d9c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f165c000f60 tx=0x7f165c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.107+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f165404b030 con 0x7f166c15c2d0 2024-03-06T07:31:27.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.349+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f166c15d7e0 con 0x7f166c15c2d0 2024-03-06T07:31:27.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.350+0000 7f1668ff9640 1 -- 172.21.15.122:0/3623246789 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f1654079c40 con 0x7f166c15c2d0 2024-03-06T07:31:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:27.352 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:27.352 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.351+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f165006b500 msgr2=0x7f165006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.351+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f165006b500 0x7f165006d9c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f165c000f60 tx=0x7f165c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.351+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 msgr2=0x7f166c0b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.351+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 0x7f166c0b0e50 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f1654001980 tx=0x7f1654001230 comp rx=0 tx=0).stop 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.351+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 shutdown_connections 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.351+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f165006b500 0x7f165006d9c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.352+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f166c0b1390 0x7f166c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.352+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f166c15c2d0 0x7f166c0b0e50 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.352+0000 7f1671ecf640 1 --2- 172.21.15.122:0/3623246789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f164c0043b0 0x7f166c0b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.352+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 >> 172.21.15.122:0/3623246789 conn(0x7f166c0d3cd0 msgr2=0x7f166c0b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.352+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 shutdown_connections 2024-03-06T07:31:27.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:27.352+0000 7f1671ecf640 1 -- 172.21.15.122:0/3623246789 wait complete. 2024-03-06T07:31:27.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:27.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:28.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:28.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 Processor -- start 2024-03-06T07:31:28.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 -- start start 2024-03-06T07:31:28.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbfdc0b3060 0x7fbfdc15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbfdc15bf30 0x7fbfdc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc15f2d0 con 0x7fbfdc13dc10 2024-03-06T07:31:28.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc0cf3f0 con 0x7fbfdc15bf30 2024-03-06T07:31:28.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.666+0000 7fbfe1356640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc15b210 con 0x7fbfdc0b3060 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdaffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbfdc13dc10 0x7fbfdc0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37058/0 (socket says 172.21.15.122:37058) 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdaffd640 1 -- 172.21.15.122:0/548037515 learned_addr learned my addr 172.21.15.122:0/548037515 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdb7fe640 1 --2- 172.21.15.122:0/548037515 >> v2:172.21.15.122:3300/0 conn(0x7fbfdc15bf30 0x7fbfdc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfd3fff640 1 --2- 172.21.15.122:0/548037515 >> v2:172.21.15.169:3300/0 conn(0x7fbfdc0b3060 0x7fbfdc15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/548037515 >> v2:172.21.15.169:3300/0 conn(0x7fbfdc0b3060 msgr2=0x7fbfdc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdb7fe640 1 --2- 172.21.15.122:0/548037515 >> v2:172.21.15.169:3300/0 conn(0x7fbfdc0b3060 0x7fbfdc15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/548037515 >> v1:172.21.15.122:6789/0 conn(0x7fbfdc13dc10 legacy=0x7fbfdc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/548037515 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc15f580 con 0x7fbfdc15bf30 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfda7fc640 1 -- 172.21.15.122:0/548037515 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945112115 0 0) 0x7fbfdc15f2d0 con 0x7fbfdc13dc10 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.667+0000 7fbfd3fff640 1 --2- 172.21.15.122:0/548037515 >> v2:172.21.15.169:3300/0 conn(0x7fbfdc0b3060 0x7fbfdc15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:28.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfdb7fe640 1 --2- 172.21.15.122:0/548037515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc15ed90 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc068700 tx=0x7fbfd402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21cf7884f0d2ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfda7fc640 1 -- 172.21.15.122:0/548037515 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4039070 con 0x7fbfdc15bf30 2024-03-06T07:31:28.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfda7fc640 1 -- 172.21.15.122:0/548037515 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd402cbf0 con 0x7fbfdc15bf30 2024-03-06T07:31:28.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfda7fc640 1 -- 172.21.15.122:0/548037515 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4034440 con 0x7fbfdc15bf30 2024-03-06T07:31:28.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 -- 172.21.15.122:0/548037515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 msgr2=0x7fbfdc15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 --2- 172.21.15.122:0/548037515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc15ed90 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc068700 tx=0x7fbfd402be40 comp rx=0 tx=0).stop 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 -- 172.21.15.122:0/548037515 shutdown_connections 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 --2- 172.21.15.122:0/548037515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc15ed90 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 --2- 172.21.15.122:0/548037515 >> v2:172.21.15.169:3300/0 conn(0x7fbfdc0b3060 0x7fbfdc15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 -- 172.21.15.122:0/548037515 >> 172.21.15.122:0/548037515 conn(0x7fbfdc0cf8a0 msgr2=0x7fbfdc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 -- 172.21.15.122:0/548037515 shutdown_connections 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.668+0000 7fbfe1356640 1 -- 172.21.15.122:0/548037515 wait complete. 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.670+0000 7fbfe1356640 1 Processor -- start 2024-03-06T07:31:28.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.670+0000 7fbfe1356640 1 -- start start 2024-03-06T07:31:28.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfe1356640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfdc0b3060 0x7fbfdc1423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfe1356640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfdc13dc10 0x7fbfdc142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfe1356640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc13a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfe1356640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc13dff0 con 0x7fbfdc15bf30 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfe1356640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc15b730 con 0x7fbfdc13dc10 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfe1356640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0708d0 con 0x7fbfdc0b3060 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfdb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc13a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfdb7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc13a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47316/0 (socket says 172.21.15.122:47316) 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.671+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/1716367243 learned_addr learned my addr 172.21.15.122:0/1716367243 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdaffd640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfdc0b3060 0x7fbfdc1423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfd3fff640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfdc13dc10 0x7fbfdc142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfdc0b3060 msgr2=0x7fbfdc1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdb7fe640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfdc0b3060 0x7fbfdc1423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfdc13dc10 msgr2=0x7fbfdc142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdb7fe640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfdc13dc10 0x7fbfdc142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdb7fe640 1 -- 172.21.15.122:0/1716367243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc141c70 con 0x7fbfdc15bf30 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfdb7fe640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc13a0e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009e80 tx=0x7fbfd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4039050 con 0x7fbfdc15bf30 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4045070 con 0x7fbfdc15bf30 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc141990 con 0x7fbfdc15bf30 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd403e830 con 0x7fbfdc15bf30 2024-03-06T07:31:28.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.672+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc1537a0 con 0x7fbfdc15bf30 2024-03-06T07:31:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.673+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc04f330 con 0x7fbfdc15bf30 2024-03-06T07:31:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.674+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfd403e9d0 con 0x7fbfdc15bf30 2024-03-06T07:31:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.674+0000 7fbfd37fe640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.674+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbfd40ad3b0 con 0x7fbfdc15bf30 2024-03-06T07:31:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.674+0000 7fbfdaffd640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.677 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.674+0000 7fbfdaffd640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc000f60 tx=0x7fbfcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.678+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfd407c330 con 0x7fbfdc15bf30 2024-03-06T07:31:28.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.923+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfdc06eb00 con 0x7fbfdc15bf30 2024-03-06T07:31:28.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.923+0000 7fbfd37fe640 1 -- 172.21.15.122:0/1716367243 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fbfd407bc30 con 0x7fbfdc15bf30 2024-03-06T07:31:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:28.926 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:28.926 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:28.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 msgr2=0x7fbfbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc000f60 tx=0x7fbfcc009040 comp rx=0 tx=0).stop 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 msgr2=0x7fbfdc13a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc13a0e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009e80 tx=0x7fbfd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 shutdown_connections 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbfdc15bf30 0x7fbfdc13a0e0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfdc13dc10 0x7fbfdc142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 --2- 172.21.15.122:0/1716367243 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbfdc0b3060 0x7fbfdc1423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 >> 172.21.15.122:0/1716367243 conn(0x7fbfdc0cf8a0 msgr2=0x7fbfdc0b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 shutdown_connections 2024-03-06T07:31:28.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:28.926+0000 7fbfe1356640 1 -- 172.21.15.122:0/1716367243 wait complete. 2024-03-06T07:31:29.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:29.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:30.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:30.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.243+0000 7f802c2b5640 1 Processor -- start 2024-03-06T07:31:30.245 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.243+0000 7f802c2b5640 1 -- start start 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802c2b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80240b1e10 0x7f802406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802c2b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802406e400 con 0x7f8024151d10 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802c2b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802406b6c0 con 0x7f802406dcc0 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802c2b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802406c2d0 con 0x7f80240b1e10 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802a051640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8024151d10 0x7f802406d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37068/0 (socket says 172.21.15.122:37068) 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802a051640 1 -- 172.21.15.122:0/2298692144 learned_addr learned my addr 172.21.15.122:0/2298692144 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802a852640 1 --2- 172.21.15.122:0/2298692144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80240b1e10 0x7f802406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637592151 0 0) 0x7f802406e400 con 0x7f8024151d10 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.244+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8010003520 con 0x7f8024151d10 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.245+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458414876 0 0) 0x7f802406b6c0 con 0x7f802406dcc0 2024-03-06T07:31:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.245+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802406e400 con 0x7f802406dcc0 2024-03-06T07:31:30.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.245+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737225283 0 0) 0x7f8010003520 con 0x7f8024151d10 2024-03-06T07:31:30.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.245+0000 7f802a852640 1 -- 172.21.15.122:0/2298692144 >> v1:172.21.15.169:6789/0 conn(0x7f802406dcc0 legacy=0x7f802415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.245+0000 7f802a852640 1 -- 172.21.15.122:0/2298692144 >> v1:172.21.15.122:6789/0 conn(0x7f8024151d10 legacy=0x7f802406d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.245+0000 7f802a852640 1 -- 172.21.15.122:0/2298692144 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802406b9a0 con 0x7f80240b1e10 2024-03-06T07:31:30.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802a852640 1 --2- 172.21.15.122:0/2298692144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80240b1e10 0x7f802406afb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f801c009dc0 tx=0x7f801c02bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab81149da0df4b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c039070 con 0x7f80240b1e10 2024-03-06T07:31:30.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c02c910 con 0x7f80240b1e10 2024-03-06T07:31:30.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802904f640 1 -- 172.21.15.122:0/2298692144 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8020001670 con 0x7f80240b1e10 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 -- 172.21.15.122:0/2298692144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80240b1e10 msgr2=0x7f802406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 --2- 172.21.15.122:0/2298692144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80240b1e10 0x7f802406afb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f801c009dc0 tx=0x7f801c02bde0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 -- 172.21.15.122:0/2298692144 shutdown_connections 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 --2- 172.21.15.122:0/2298692144 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f80240b1e10 0x7f802406afb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 -- 172.21.15.122:0/2298692144 >> 172.21.15.122:0/2298692144 conn(0x7f80240b4fe0 msgr2=0x7f80240b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 -- 172.21.15.122:0/2298692144 shutdown_connections 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.246+0000 7f802c2b5640 1 -- 172.21.15.122:0/2298692144 wait complete. 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.247+0000 7f802c2b5640 1 Processor -- start 2024-03-06T07:31:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.247+0000 7f802c2b5640 1 -- start start 2024-03-06T07:31:30.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802c2b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802406dcc0 0x7f80240d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802c2b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80240b1e10 0x7f802414dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802c2b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 0x7f80241574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802c2b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80241520f0 con 0x7f8024151d10 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802c2b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802406bf60 con 0x7f80240b1e10 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802c2b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802406c760 con 0x7f802406dcc0 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802a852640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 0x7f80241574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802a852640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 0x7f80241574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47324/0 (socket says 172.21.15.122:47324) 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802a051640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802406dcc0 0x7f80240d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f802a852640 1 -- 172.21.15.122:0/1252200023 learned_addr learned my addr 172.21.15.122:0/1252200023 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.248+0000 7f8029850640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80240b1e10 0x7f802414dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802a852640 1 -- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802406dcc0 msgr2=0x7f80240d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802a852640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802406dcc0 0x7f80240d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802a852640 1 -- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80240b1e10 msgr2=0x7f802414dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802a852640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80240b1e10 0x7f802414dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802a852640 1 -- 172.21.15.122:0/1252200023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80241f3ad0 con 0x7f8024151d10 2024-03-06T07:31:30.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802a852640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 0x7f80241574c0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f801c037040 tx=0x7f801c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c039050 con 0x7f8024151d10 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c045070 con 0x7f8024151d10 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80241584c0 con 0x7f8024151d10 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c009450 con 0x7f8024151d10 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.249+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80240ae140 con 0x7f8024151d10 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.250+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802404f330 con 0x7f8024151d10 2024-03-06T07:31:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.250+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f801c040690 con 0x7f8024151d10 2024-03-06T07:31:30.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.251+0000 7f801affd640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b4d0 0x7f800806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.251+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f801c0ac990 con 0x7f8024151d10 2024-03-06T07:31:30.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.251+0000 7f802a051640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b4d0 0x7f800806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.251+0000 7f802a051640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b4d0 0x7f800806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f80200011e0 tx=0x7f8020001710 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.255+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f801c07b390 con 0x7f8024151d10 2024-03-06T07:31:30.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.503+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f802415c9b0 con 0x7f8024151d10 2024-03-06T07:31:30.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.503+0000 7f801affd640 1 -- 172.21.15.122:0/1252200023 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f801c07f430 con 0x7f8024151d10 2024-03-06T07:31:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:30.506 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:30.506 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:30.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b4d0 msgr2=0x7f800806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b4d0 0x7f800806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f80200011e0 tx=0x7f8020001710 comp rx=0 tx=0).stop 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 msgr2=0x7f80241574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 0x7f80241574c0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f801c037040 tx=0x7f801c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 shutdown_connections 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f800806b4d0 0x7f800806d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8024151d10 0x7f80241574c0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80240b1e10 0x7f802414dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 --2- 172.21.15.122:0/1252200023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f802406dcc0 0x7f80240d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 >> 172.21.15.122:0/1252200023 conn(0x7f80240b4fe0 msgr2=0x7f802415d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 shutdown_connections 2024-03-06T07:31:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:30.506+0000 7f802c2b5640 1 -- 172.21.15.122:0/1252200023 wait complete. 2024-03-06T07:31:30.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:30.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:31.725 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:31.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.825+0000 7faf375f7640 1 Processor -- start 2024-03-06T07:31:31.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.825+0000 7faf375f7640 1 -- start start 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.826+0000 7faf375f7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7faf3015dfe0 0x7faf300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.826+0000 7faf375f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300b3430 con 0x7faf300cf830 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.826+0000 7faf375f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300b3710 con 0x7faf300d29c0 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.826+0000 7faf375f7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf300b31e0 con 0x7faf3015dfe0 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.826+0000 7faf34b92640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faf300cf830 0x7faf300d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37072/0 (socket says 172.21.15.122:37072) 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.826+0000 7faf34b92640 1 -- 172.21.15.122:0/2567885314 learned_addr learned my addr 172.21.15.122:0/2567885314 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:31.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 --2- 172.21.15.122:0/2567885314 >> v2:172.21.15.169:3300/0 conn(0x7faf3015dfe0 0x7faf300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322329332 0 0) 0x7faf300b3430 con 0x7faf300cf830 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c003520 con 0x7faf300cf830 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246491893 0 0) 0x7faf1c003520 con 0x7faf300cf830 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 >> v2:172.21.15.169:3300/0 conn(0x7faf3015dfe0 msgr2=0x7faf300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 --2- 172.21.15.122:0/2567885314 >> v2:172.21.15.169:3300/0 conn(0x7faf3015dfe0 0x7faf300cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 >> v1:172.21.15.169:6789/0 conn(0x7faf300d29c0 legacy=0x7faf300b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300b39f0 con 0x7faf300cf830 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7faf2c001670 con 0x7faf300cf830 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 >> v1:172.21.15.122:6789/0 conn(0x7faf300cf830 legacy=0x7faf300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf1c0039b0 0x7faf1c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf1c004800 0x7faf1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0051c0 0x7faf1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf300b31e0 con 0x7faf1c0039b0 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c00c090 con 0x7faf1c0051c0 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c00c290 con 0x7faf1c004800 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf1c0039b0 0x7faf1c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf34b92640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf1c004800 0x7faf1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35b94640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0051c0 0x7faf1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 -- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf1c004800 msgr2=0x7faf1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf1c004800 0x7faf1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 -- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0051c0 msgr2=0x7faf1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0051c0 0x7faf1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35393640 1 -- 172.21.15.122:0/2567885314 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c003520 con 0x7faf1c0039b0 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf35b94640 1 -- 172.21.15.122:0/2567885314 reap_dead start 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.827+0000 7faf34b92640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf1c004800 0x7faf1c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:31.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.828+0000 7faf35393640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf1c0039b0 0x7faf1c003df0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7faf20009ac0 tx=0x7faf1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbba1cf1c0b5b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.828+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20039070 con 0x7faf1c0039b0 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.828+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2002c6b0 con 0x7faf1c0039b0 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.828+0000 7faf27fff640 1 -- 172.21.15.122:0/2567885314 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20009690 con 0x7faf1c0039b0 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 -- 172.21.15.122:0/2567885314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf1c0039b0 msgr2=0x7faf1c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf1c0039b0 0x7faf1c003df0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7faf20009ac0 tx=0x7faf1c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 -- 172.21.15.122:0/2567885314 shutdown_connections 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 --2- 172.21.15.122:0/2567885314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf1c0039b0 0x7faf1c003df0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 -- 172.21.15.122:0/2567885314 >> 172.21.15.122:0/2567885314 conn(0x7faf300d3cd0 msgr2=0x7faf300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 -- 172.21.15.122:0/2567885314 shutdown_connections 2024-03-06T07:31:31.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.829+0000 7faf375f7640 1 -- 172.21.15.122:0/2567885314 wait complete. 2024-03-06T07:31:31.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.831+0000 7faf375f7640 1 Processor -- start 2024-03-06T07:31:31.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.831+0000 7faf375f7640 1 -- start start 2024-03-06T07:31:31.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf375f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0039b0 0x7faf301521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf375f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 0x7faf300b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf375f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf300cb200 0x7faf300b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf375f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3015e3c0 con 0x7faf300cada0 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf375f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf300b3d60 con 0x7faf1c0039b0 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf375f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3015e540 con 0x7faf300cb200 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 0x7faf300b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 0x7faf300b4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47344/0 (socket says 172.21.15.122:47344) 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 -- 172.21.15.122:0/3735659788 learned_addr learned my addr 172.21.15.122:0/3735659788 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf35b94640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf300cb200 0x7faf300b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf35393640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0039b0 0x7faf301521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 -- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf300cb200 msgr2=0x7faf300b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf300cb200 0x7faf300b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 -- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0039b0 msgr2=0x7faf301521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0039b0 0x7faf301521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf34b92640 1 -- 172.21.15.122:0/3735659788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30152700 con 0x7faf300cada0 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf35b94640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf300cb200 0x7faf300b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:31.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.832+0000 7faf35393640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0039b0 0x7faf301521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.833+0000 7faf34b92640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 0x7faf300b4b90 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00e780 tx=0x7faf2c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.833+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c00c660 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.833+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c018070 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.833+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf300cb780 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.833+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c00ca20 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.833+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf300cbd40 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.834+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3004f330 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.834+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf2c00cbc0 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.835+0000 7faf25ffb640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf1406b500 0x7faf1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.835+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faf2c089d10 con 0x7faf300cada0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.835+0000 7faf35393640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf1406b500 0x7faf1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.839 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.835+0000 7faf35393640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf1406b500 0x7faf1406d9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7faf20009ac0 tx=0x7faf20030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:31.839+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf2c058b70 con 0x7faf300cada0 2024-03-06T07:31:32.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.054+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf3006c200 con 0x7faf300cada0 2024-03-06T07:31:32.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.054+0000 7faf25ffb640 1 -- 172.21.15.122:0/3735659788 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7faf2c058470 con 0x7faf300cada0 2024-03-06T07:31:32.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:32.056 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:32.057 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf1406b500 msgr2=0x7faf1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf1406b500 0x7faf1406d9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7faf20009ac0 tx=0x7faf20030000 comp rx=0 tx=0).stop 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 msgr2=0x7faf300b4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 0x7faf300b4b90 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00e780 tx=0x7faf2c00ec40 comp rx=0 tx=0).stop 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 shutdown_connections 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7faf1406b500 0x7faf1406d9c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7faf300cb200 0x7faf300b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf300cada0 0x7faf300b4b90 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 --2- 172.21.15.122:0/3735659788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faf1c0039b0 0x7faf301521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 >> 172.21.15.122:0/3735659788 conn(0x7faf300d3cd0 msgr2=0x7faf3006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 shutdown_connections 2024-03-06T07:31:32.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:32.057+0000 7faf375f7640 1 -- 172.21.15.122:0/3735659788 wait complete. 2024-03-06T07:31:32.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:32.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:33.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:33.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.376+0000 7f1b9028f640 1 Processor -- start 2024-03-06T07:31:33.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.376+0000 7f1b9028f640 1 -- start start 2024-03-06T07:31:33.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b9028f640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1b8815dfe0 0x7f1b880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b9028f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1b880cf830 0x7f1b880d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b9028f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b880b3430 con 0x7f1b880d27f0 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b9028f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b880b31e0 con 0x7f1b880cf830 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b9028f640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b880b3d60 con 0x7f1b8815dfe0 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b8d82a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1b880cf830 0x7f1b880d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b8d82a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1b880cf830 0x7f1b880d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47358/0 (socket says 172.21.15.122:47358) 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.377+0000 7f1b8d82a640 1 -- 172.21.15.122:0/1007721386 learned_addr learned my addr 172.21.15.122:0/1007721386 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8e02b640 1 --2- 172.21.15.122:0/1007721386 >> v2:172.21.15.169:3300/0 conn(0x7f1b8815dfe0 0x7f1b880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8d82a640 1 -- 172.21.15.122:0/1007721386 >> v2:172.21.15.169:3300/0 conn(0x7f1b8815dfe0 msgr2=0x7f1b880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8d82a640 1 --2- 172.21.15.122:0/1007721386 >> v2:172.21.15.169:3300/0 conn(0x7f1b8815dfe0 0x7f1b880cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8d82a640 1 -- 172.21.15.122:0/1007721386 >> v1:172.21.15.169:6789/0 conn(0x7f1b880d27f0 legacy=0x7f1b880d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8d82a640 1 -- 172.21.15.122:0/1007721386 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880b3710 con 0x7f1b880cf830 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8e02b640 1 --2- 172.21.15.122:0/1007721386 >> v2:172.21.15.169:3300/0 conn(0x7f1b8815dfe0 0x7f1b880cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.378+0000 7f1b8d82a640 1 --2- 172.21.15.122:0/1007721386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b880d22b0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009870 tx=0x7f1b7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6021e8d3242054a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b8d029640 1 -- 172.21.15.122:0/1007721386 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78035020 con 0x7f1b880cf830 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b8d029640 1 -- 172.21.15.122:0/1007721386 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b78031870 con 0x7f1b880cf830 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b8d029640 1 -- 172.21.15.122:0/1007721386 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78031c40 con 0x7f1b880cf830 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b9028f640 1 -- 172.21.15.122:0/1007721386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 msgr2=0x7f1b880d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b9028f640 1 --2- 172.21.15.122:0/1007721386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b880d22b0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009870 tx=0x7f1b7802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b9028f640 1 -- 172.21.15.122:0/1007721386 shutdown_connections 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b9028f640 1 --2- 172.21.15.122:0/1007721386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b880d22b0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b9028f640 1 --2- 172.21.15.122:0/1007721386 >> v2:172.21.15.169:3300/0 conn(0x7f1b8815dfe0 0x7f1b880cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.379+0000 7f1b9028f640 1 -- 172.21.15.122:0/1007721386 >> 172.21.15.122:0/1007721386 conn(0x7f1b880d3cd0 msgr2=0x7f1b880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.380+0000 7f1b9028f640 1 -- 172.21.15.122:0/1007721386 shutdown_connections 2024-03-06T07:31:33.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.380+0000 7f1b9028f640 1 -- 172.21.15.122:0/1007721386 wait complete. 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 Processor -- start 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 -- start start 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b8814e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b880d27f0 0x7f1b880ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b8815dfe0 0x7f1b880cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880d1df0 con 0x7f1b880cf830 2024-03-06T07:31:33.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880d1c70 con 0x7f1b880d27f0 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b9028f640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8815e3c0 con 0x7f1b8815dfe0 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b8e02b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b8814e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b8e02b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b8814e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47360/0 (socket says 172.21.15.122:47360) 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.381+0000 7f1b8e02b640 1 -- 172.21.15.122:0/3711895975 learned_addr learned my addr 172.21.15.122:0/3711895975 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8d82a640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b880d27f0 0x7f1b880ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e82c640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b8815dfe0 0x7f1b880cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e02b640 1 -- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b8815dfe0 msgr2=0x7f1b880cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e02b640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b8815dfe0 0x7f1b880cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e02b640 1 -- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b880d27f0 msgr2=0x7f1b880ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e02b640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b880d27f0 0x7f1b880ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e02b640 1 -- 172.21.15.122:0/3711895975 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8814e750 con 0x7f1b880cf830 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b8e02b640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b8814e210 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c7f0 tx=0x7f1b8400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84015070 con 0x7f1b880cf830 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b84007a30 con 0x7f1b880cf830 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b880cc0f0 con 0x7f1b880cf830 2024-03-06T07:31:33.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84007e00 con 0x7f1b880cf830 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.382+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8815f360 con 0x7f1b880cf830 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.383+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8804f330 con 0x7f1b880cf830 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.384+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b84018070 con 0x7f1b880cf830 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.384+0000 7f1b7effd640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b6c06b4d0 0x7f1b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.384+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1b8408a030 con 0x7f1b880cf830 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.384+0000 7f1b8d82a640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b6c06b4d0 0x7f1b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.385+0000 7f1b8d82a640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b6c06b4d0 0x7f1b6c06d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1b78036000 tx=0x7f1b7803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.389+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b840588d0 con 0x7f1b880cf830 2024-03-06T07:31:33.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.607+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b880b09c0 con 0x7f1b880cf830 2024-03-06T07:31:33.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.608+0000 7f1b7effd640 1 -- 172.21.15.122:0/3711895975 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f1b8405c970 con 0x7f1b880cf830 2024-03-06T07:31:33.610 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:33.610 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:05.142162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:33.610 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 99 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b6c06b4d0 msgr2=0x7f1b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b6c06b4d0 0x7f1b6c06d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1b78036000 tx=0x7f1b7803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 msgr2=0x7f1b8814e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b8814e210 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c7f0 tx=0x7f1b8400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 shutdown_connections 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1b6c06b4d0 0x7f1b6c06d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1b8815dfe0 0x7f1b880cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b880d27f0 0x7f1b880ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 --2- 172.21.15.122:0/3711895975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b880cf830 0x7f1b8814e210 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 >> 172.21.15.122:0/3711895975 conn(0x7f1b880d3cd0 msgr2=0x7f1b8806c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 shutdown_connections 2024-03-06T07:31:33.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:33.610+0000 7f1b9028f640 1 -- 172.21.15.122:0/3711895975 wait complete. 2024-03-06T07:31:33.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:05.142162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:33.828 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:34.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:34.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.928+0000 7fe17395c640 1 Processor -- start 2024-03-06T07:31:34.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 -- start start 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c15dfe0 0x7fe16c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe16c0cf830 0x7fe16c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d27f0 0x7fe16c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c0d2c50 con 0x7fe16c0cf830 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c0b31e0 con 0x7fe16c0d27f0 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe17395c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0b3d60 con 0x7fe16c15dfe0 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.929+0000 7fe170ef7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe16c0cf830 0x7fe16c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe16c0cf830 0x7fe16c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45642/0 (socket says 172.21.15.122:45642) 2024-03-06T07:31:34.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 -- 172.21.15.122:0/2017499316 learned_addr learned my addr 172.21.15.122:0/2017499316 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe171ef9640 1 --2- 172.21.15.122:0/2017499316 >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d27f0 0x7fe16c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe1716f8640 1 --2- 172.21.15.122:0/2017499316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c15dfe0 0x7fe16c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 -- 172.21.15.122:0/2017499316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c15dfe0 msgr2=0x7fe16c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 --2- 172.21.15.122:0/2017499316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c15dfe0 0x7fe16c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 -- 172.21.15.122:0/2017499316 >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d27f0 msgr2=0x7fe16c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 --2- 172.21.15.122:0/2017499316 >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d27f0 0x7fe16c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.930+0000 7fe170ef7640 1 -- 172.21.15.122:0/2017499316 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0b3430 con 0x7fe16c0cf830 2024-03-06T07:31:34.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe170ef7640 1 --2- 172.21.15.122:0/2017499316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0cf830 0x7fe16c0d22b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe16c067ad0 tx=0x7fe15c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707f44d9a508fc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe163fff640 1 -- 172.21.15.122:0/2017499316 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c035020 con 0x7fe16c0cf830 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe163fff640 1 -- 172.21.15.122:0/2017499316 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c031870 con 0x7fe16c0cf830 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe163fff640 1 -- 172.21.15.122:0/2017499316 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c031c40 con 0x7fe16c0cf830 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 -- 172.21.15.122:0/2017499316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0cf830 msgr2=0x7fe16c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 --2- 172.21.15.122:0/2017499316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0cf830 0x7fe16c0d22b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe16c067ad0 tx=0x7fe15c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 -- 172.21.15.122:0/2017499316 shutdown_connections 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 --2- 172.21.15.122:0/2017499316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c0cf830 0x7fe16c0d22b0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 --2- 172.21.15.122:0/2017499316 >> v2:172.21.15.169:3300/0 conn(0x7fe16c0d27f0 0x7fe16c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 --2- 172.21.15.122:0/2017499316 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c15dfe0 0x7fe16c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.931+0000 7fe17395c640 1 -- 172.21.15.122:0/2017499316 >> 172.21.15.122:0/2017499316 conn(0x7fe16c0d3cd0 msgr2=0x7fe16c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.932+0000 7fe17395c640 1 -- 172.21.15.122:0/2017499316 shutdown_connections 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.932+0000 7fe17395c640 1 -- 172.21.15.122:0/2017499316 wait complete. 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 Processor -- start 2024-03-06T07:31:34.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 -- start start 2024-03-06T07:31:34.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0cf830 0x7fe16c1f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c0d27f0 0x7fe16c1f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 0x7fe16c149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0d1c70 con 0x7fe16c15dfe0 2024-03-06T07:31:34.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0b39f0 con 0x7fe16c0cf830 2024-03-06T07:31:34.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.933+0000 7fe17395c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0d1df0 con 0x7fe16c0d27f0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 0x7fe16c149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 0x7fe16c149cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45650/0 (socket says 172.21.15.122:45650) 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 -- 172.21.15.122:0/870164462 learned_addr learned my addr 172.21.15.122:0/870164462 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe170ef7640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c0d27f0 0x7fe16c1f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe1716f8640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0cf830 0x7fe16c1f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 -- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c0d27f0 msgr2=0x7fe16c1f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c0d27f0 0x7fe16c1f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 -- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0cf830 msgr2=0x7fe16c1f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0cf830 0x7fe16c1f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 -- 172.21.15.122:0/870164462 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c161450 con 0x7fe16c15dfe0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe171ef9640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 0x7fe16c149cf0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe16400d830 tx=0x7fe16400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164018020 con 0x7fe16c15dfe0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1640096c0 con 0x7fe16c15dfe0 2024-03-06T07:31:34.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c0cc980 con 0x7fe16c15dfe0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164009a90 con 0x7fe16c15dfe0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.934+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c14dcc0 con 0x7fe16c15dfe0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.935+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c0cc7a0 con 0x7fe16c15dfe0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.936+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe16401b020 con 0x7fe16c15dfe0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.936+0000 7fe161ffb640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe15006b4d0 0x7fe15006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.936+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe16408a3e0 con 0x7fe16c15dfe0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.936+0000 7fe1716f8640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe15006b4d0 0x7fe15006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.937+0000 7fe1716f8640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe15006b4d0 0x7fe15006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe168003aa0 tx=0x7fe168009830 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:34.941+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe16c0cc7a0 con 0x7fe16c15dfe0 2024-03-06T07:31:35.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.159+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe16c04f330 con 0x7fe16c15dfe0 2024-03-06T07:31:35.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.159+0000 7fe161ffb640 1 -- 172.21.15.122:0/870164462 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fe164022030 con 0x7fe16c15dfe0 2024-03-06T07:31:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:35.162 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:35.162 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.161+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe15006b4d0 msgr2=0x7fe15006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.161+0000 7fe17395c640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe15006b4d0 0x7fe15006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe168003aa0 tx=0x7fe168009830 comp rx=0 tx=0).stop 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.161+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 msgr2=0x7fe16c149cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.161+0000 7fe17395c640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 0x7fe16c149cf0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe16400d830 tx=0x7fe16400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 shutdown_connections 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe15006b4d0 0x7fe15006d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe16c15dfe0 0x7fe16c149cf0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe16c0d27f0 0x7fe16c1f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 --2- 172.21.15.122:0/870164462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe16c0cf830 0x7fe16c1f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 >> 172.21.15.122:0/870164462 conn(0x7fe16c0d3cd0 msgr2=0x7fe16c06ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 shutdown_connections 2024-03-06T07:31:35.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:35.162+0000 7fe17395c640 1 -- 172.21.15.122:0/870164462 wait complete. 2024-03-06T07:31:35.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:35.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:36.377 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:36.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.491+0000 7fa4b693a640 1 Processor -- start 2024-03-06T07:31:36.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4b693a640 1 -- start start 2024-03-06T07:31:36.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4b693a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa4b00d06b0 0x7fa4b0166db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4b693a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00d0020 con 0x7fa4b015c320 2024-03-06T07:31:36.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4b693a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00b0d20 con 0x7fa4b00b40c0 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4b693a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00b1000 con 0x7fa4b00d06b0 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4affff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa4b015c320 0x7fa4b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58618/0 (socket says 172.21.15.122:58618) 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.492+0000 7fa4affff640 1 -- 172.21.15.122:0/2585308770 learned_addr learned my addr 172.21.15.122:0/2585308770 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4b4ed7640 1 --2- 172.21.15.122:0/2585308770 >> v2:172.21.15.169:3300/0 conn(0x7fa4b00d06b0 0x7fa4b0166db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713841066 0 0) 0x7fa4b00d0020 con 0x7fa4b015c320 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498003520 con 0x7fa4b015c320 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 557392309 0 0) 0x7fa498003520 con 0x7fa4b015c320 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 >> v2:172.21.15.169:3300/0 conn(0x7fa4b00d06b0 msgr2=0x7fa4b0166db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 --2- 172.21.15.122:0/2585308770 >> v2:172.21.15.169:3300/0 conn(0x7fa4b00d06b0 0x7fa4b0166db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 >> v1:172.21.15.169:6789/0 conn(0x7fa4b00b40c0 legacy=0x7fa4b00b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00cfa60 con 0x7fa4b015c320 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa4a4001670 con 0x7fa4b015c320 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 >> v1:172.21.15.122:6789/0 conn(0x7fa4b015c320 legacy=0x7fa4b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4980039b0 0x7fa498003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa49800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00b1000 con 0x7fa4980052e0 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800c1b0 con 0x7fa4980039b0 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800c3b0 con 0x7fa498004800 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4b4ed7640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa49800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4affff640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4980039b0 0x7fa498003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.493+0000 7fa4af7fe640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 -- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa498004800 msgr2=0x7fa498004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 -- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4980039b0 msgr2=0x7fa498003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4980039b0 0x7fa498003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 -- 172.21.15.122:0/2585308770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498003520 con 0x7fa4980052e0 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 -- 172.21.15.122:0/2585308770 reap_dead start 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4b4ed7640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa49800b9c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009f90 tx=0x7fa498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab72ecb25e5a8027 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8009c10 con 0x7fa4980052e0 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a802cc20 con 0x7fa4980052e0 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.494+0000 7fa4aeffd640 1 -- 172.21.15.122:0/2585308770 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8003990 con 0x7fa4980052e0 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 -- 172.21.15.122:0/2585308770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 msgr2=0x7fa49800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa49800b9c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009f90 tx=0x7fa498005830 comp rx=0 tx=0).stop 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 -- 172.21.15.122:0/2585308770 shutdown_connections 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 --2- 172.21.15.122:0/2585308770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa49800b9c0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 -- 172.21.15.122:0/2585308770 >> 172.21.15.122:0/2585308770 conn(0x7fa4b00b35a0 msgr2=0x7fa4b00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 -- 172.21.15.122:0/2585308770 shutdown_connections 2024-03-06T07:31:36.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.495+0000 7fa4b693a640 1 -- 172.21.15.122:0/2585308770 wait complete. 2024-03-06T07:31:36.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.497+0000 7fa4b693a640 1 Processor -- start 2024-03-06T07:31:36.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.497+0000 7fa4b693a640 1 -- start start 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4b693a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa4b014ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4b693a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4b00b6f50 0x7fa4b00c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4b693a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa4b00cb140 0x7fa4b00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4b693a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00cda00 con 0x7fa4980052e0 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4b693a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00cd880 con 0x7fa4b00b6f50 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4b693a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00b1be0 con 0x7fa4b00cb140 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4affff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa4b014ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4affff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa4b014ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45676/0 (socket says 172.21.15.122:45676) 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.498+0000 7fa4affff640 1 -- 172.21.15.122:0/1804439885 learned_addr learned my addr 172.21.15.122:0/1804439885 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4af7fe640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4b00b6f50 0x7fa4b00c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4b4ed7640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa4b00cb140 0x7fa4b00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4affff640 1 -- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa4b00cb140 msgr2=0x7fa4b00b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4affff640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa4b00cb140 0x7fa4b00b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4affff640 1 -- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4b00b6f50 msgr2=0x7fa4b00c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4affff640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4b00b6f50 0x7fa4b00c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4affff640 1 -- 172.21.15.122:0/1804439885 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00c6e10 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4affff640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa4b014ff20 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4007290 tx=0x7fa4a4007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4001640 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a4018070 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400fba0 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b00c70f0 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.499+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b00c76b0 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.500+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b00b7330 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.501+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4a400fd40 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.501+0000 7fa4acff9640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa49c06b500 0x7fa49c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.502+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa4a408a160 con 0x7fa4980052e0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.502+0000 7fa4af7fe640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa49c06b500 0x7fa49c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.502+0000 7fa4af7fe640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa49c06b500 0x7fa49c06d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0000c00 tx=0x7fa4a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.512 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.510+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4a4058a00 con 0x7fa4980052e0 2024-03-06T07:31:36.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.725+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4b004f330 con 0x7fa4980052e0 2024-03-06T07:31:36.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.725+0000 7fa4acff9640 1 -- 172.21.15.122:0/1804439885 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fa4a405caa0 con 0x7fa4980052e0 2024-03-06T07:31:36.727 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:36.728 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:36.728 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa49c06b500 msgr2=0x7fa49c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa49c06b500 0x7fa49c06d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0000c00 tx=0x7fa4a0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 msgr2=0x7fa4b014ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa4b014ff20 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4007290 tx=0x7fa4a4007750 comp rx=0 tx=0).stop 2024-03-06T07:31:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 shutdown_connections 2024-03-06T07:31:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa49c06b500 0x7fa49c06d9c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa4b00cb140 0x7fa4b00b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4b00b6f50 0x7fa4b00c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 --2- 172.21.15.122:0/1804439885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4980052e0 0x7fa4b014ff20 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 >> 172.21.15.122:0/1804439885 conn(0x7fa4b00b35a0 msgr2=0x7fa4b00b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 shutdown_connections 2024-03-06T07:31:36.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:36.728+0000 7fa4b693a640 1 -- 172.21.15.122:0/1804439885 wait complete. 2024-03-06T07:31:36.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:36.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:37.943 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:38.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 Processor -- start 2024-03-06T07:31:38.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 -- start start 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007015dfe0 0x7f00700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00700d29c0 0x7f00700b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00700b3430 con 0x7f00700cf830 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00700b31e0 con 0x7f00700d29c0 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.042+0000 7f0074c1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00700b3d60 con 0x7f007015dfe0 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00700d29c0 0x7f00700b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00700d29c0 0x7f00700b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45684/0 (socket says 172.21.15.122:45684) 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f0065d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f00700cf830 0x7f00700d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58622/0 (socket says 172.21.15.122:58622) 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 -- 172.21.15.122:0/3674854225 learned_addr learned my addr 172.21.15.122:0/3674854225 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006e59c640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007015dfe0 0x7f00700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 -- 172.21.15.122:0/3674854225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007015dfe0 msgr2=0x7f00700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007015dfe0 0x7f00700cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 -- 172.21.15.122:0/3674854225 >> v1:172.21.15.122:6789/0 conn(0x7f00700cf830 legacy=0x7f00700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006ed9d640 1 -- 172.21.15.122:0/3674854225 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700b3710 con 0x7f00700d29c0 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006dd9b640 1 -- 172.21.15.122:0/3674854225 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24489253 0 0) 0x7f00700b3430 con 0x7f00700cf830 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.043+0000 7f006e59c640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007015dfe0 0x7f00700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f006ed9d640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00700d29c0 0x7f00700b41f0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f0068009e50 tx=0x7f006802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8873780887f971d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f006dd9b640 1 -- 172.21.15.122:0/3674854225 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006802cb00 con 0x7f00700d29c0 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f006dd9b640 1 -- 172.21.15.122:0/3674854225 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006802cca0 con 0x7f00700d29c0 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f006dd9b640 1 -- 172.21.15.122:0/3674854225 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068034610 con 0x7f00700d29c0 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 -- 172.21.15.122:0/3674854225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00700d29c0 msgr2=0x7f00700b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00700d29c0 0x7f00700b41f0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f0068009e50 tx=0x7f006802bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 -- 172.21.15.122:0/3674854225 shutdown_connections 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00700d29c0 0x7f00700b41f0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3674854225 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007015dfe0 0x7f00700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 -- 172.21.15.122:0/3674854225 >> 172.21.15.122:0/3674854225 conn(0x7f00700d3cd0 msgr2=0x7f00700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 -- 172.21.15.122:0/3674854225 shutdown_connections 2024-03-06T07:31:38.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.044+0000 7f0074c1c640 1 -- 172.21.15.122:0/3674854225 wait complete. 2024-03-06T07:31:38.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.046+0000 7f0074c1c640 1 Processor -- start 2024-03-06T07:31:38.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.046+0000 7f0074c1c640 1 -- start start 2024-03-06T07:31:38.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0074c1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00700cf830 0x7f00701f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0074c1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00700d29c0 0x7f0070152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0074c1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 0x7f00700b9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0074c1c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00700d1df0 con 0x7f007015dfe0 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0074c1c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00700d1c70 con 0x7f00700cf830 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0074c1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007015e3c0 con 0x7f00700d29c0 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 0x7f00700b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 0x7f00700b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45690/0 (socket says 172.21.15.122:45690) 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 -- 172.21.15.122:0/3471015326 learned_addr learned my addr 172.21.15.122:0/3471015326 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f0065d9b640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00700d29c0 0x7f0070152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 -- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00700d29c0 msgr2=0x7f0070152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00700d29c0 0x7f0070152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006e59c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00700cf830 0x7f00701f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 -- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00700cf830 msgr2=0x7f00701f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00700cf830 0x7f00701f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 -- 172.21.15.122:0/3471015326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007015b540 con 0x7f007015dfe0 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f006ed9d640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 0x7f00700b9380 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f0068036040 tx=0x7f0068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068043030 con 0x7f007015dfe0 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006802ce50 con 0x7f007015dfe0 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.048+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006803e450 con 0x7f007015dfe0 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.049+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00700ccbc0 con 0x7f007015dfe0 2024-03-06T07:31:38.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.049+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00700cd180 con 0x7f007015dfe0 2024-03-06T07:31:38.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.050+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00700b3360 con 0x7f007015dfe0 2024-03-06T07:31:38.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.050+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0068046070 con 0x7f007015dfe0 2024-03-06T07:31:38.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.050+0000 7f00677fe640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005006b4d0 0x7f005006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.050+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f00680ac640 con 0x7f007015dfe0 2024-03-06T07:31:38.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.051+0000 7f006e59c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005006b4d0 0x7f005006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.051+0000 7f006e59c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005006b4d0 0x7f005006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f006000b840 tx=0x7f0060008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.055+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00700b3360 con 0x7f007015dfe0 2024-03-06T07:31:38.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.273+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00700cd520 con 0x7f007015dfe0 2024-03-06T07:31:38.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.273+0000 7f00677fe640 1 -- 172.21.15.122:0/3471015326 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f006804d070 con 0x7f007015dfe0 2024-03-06T07:31:38.276 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:38.276 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:38.276 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005006b4d0 msgr2=0x7f005006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005006b4d0 0x7f005006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f006000b840 tx=0x7f0060008040 comp rx=0 tx=0).stop 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 msgr2=0x7f00700b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 0x7f00700b9380 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f0068036040 tx=0x7f0068004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 shutdown_connections 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f005006b4d0 0x7f005006d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007015dfe0 0x7f00700b9380 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f00700d29c0 0x7f0070152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 --2- 172.21.15.122:0/3471015326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00700cf830 0x7f00701f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 >> 172.21.15.122:0/3471015326 conn(0x7f00700d3cd0 msgr2=0x7f00700b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.275+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 shutdown_connections 2024-03-06T07:31:38.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:38.276+0000 7f0074c1c640 1 -- 172.21.15.122:0/3471015326 wait complete. 2024-03-06T07:31:38.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:38.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:39.491 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:39.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 Processor -- start 2024-03-06T07:31:39.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 -- start start 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a815bb20 0x7f74a80d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f74a80cfff0 0x7f74a80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f74a80ce4b0 0x7f74a80ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a8161980 con 0x7f74a80cfff0 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a8161c60 con 0x7f74a80ce4b0 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.591+0000 7f74adb4b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a8161f40 con 0x7f74a815bb20 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f74a80cfff0 0x7f74a80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f74a80cfff0 0x7f74a80d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45696/0 (socket says 172.21.15.122:45696) 2024-03-06T07:31:39.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 -- 172.21.15.122:0/3184851116 learned_addr learned my addr 172.21.15.122:0/3184851116 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f74a77fe640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a815bb20 0x7f74a80d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 -- 172.21.15.122:0/3184851116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a815bb20 msgr2=0x7f74a80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a815bb20 0x7f74a80d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f74a7fff640 1 --2- 172.21.15.122:0/3184851116 >> v2:172.21.15.169:3300/0 conn(0x7f74a80ce4b0 0x7f74a80ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 -- 172.21.15.122:0/3184851116 >> v2:172.21.15.169:3300/0 conn(0x7f74a80ce4b0 msgr2=0x7f74a80ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 --2- 172.21.15.122:0/3184851116 >> v2:172.21.15.169:3300/0 conn(0x7f74a80ce4b0 0x7f74a80ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f749ffff640 1 -- 172.21.15.122:0/3184851116 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8162220 con 0x7f74a80cfff0 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f74a77fe640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a815bb20 0x7f74a80d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.592+0000 7f74a7fff640 1 --2- 172.21.15.122:0/3184851116 >> v2:172.21.15.169:3300/0 conn(0x7f74a80ce4b0 0x7f74a80ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f749ffff640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80cfff0 0x7f74a80d0470 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f74a80b13b0 tx=0x7f749002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153406ddc08ded93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74a6ffd640 1 -- 172.21.15.122:0/3184851116 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490035020 con 0x7f74a80cfff0 2024-03-06T07:31:39.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74a6ffd640 1 -- 172.21.15.122:0/3184851116 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490031870 con 0x7f74a80cfff0 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74a6ffd640 1 -- 172.21.15.122:0/3184851116 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490031c40 con 0x7f74a80cfff0 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 -- 172.21.15.122:0/3184851116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80cfff0 msgr2=0x7f74a80d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80cfff0 0x7f74a80d0470 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f74a80b13b0 tx=0x7f749002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 -- 172.21.15.122:0/3184851116 shutdown_connections 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80cfff0 0x7f74a80d0470 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3184851116 >> v2:172.21.15.169:3300/0 conn(0x7f74a80ce4b0 0x7f74a80ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3184851116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a815bb20 0x7f74a80d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 -- 172.21.15.122:0/3184851116 >> 172.21.15.122:0/3184851116 conn(0x7f74a809a9f0 msgr2=0x7f74a809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 -- 172.21.15.122:0/3184851116 shutdown_connections 2024-03-06T07:31:39.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.593+0000 7f74adb4b640 1 -- 172.21.15.122:0/3184851116 wait complete. 2024-03-06T07:31:39.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 Processor -- start 2024-03-06T07:31:39.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 -- start start 2024-03-06T07:31:39.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 0x7f74a815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a80cfff0 0x7f74a806da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74a815bb20 0x7f74a80c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80d24b0 con 0x7f74a80ce4b0 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80d2330 con 0x7f74a815bb20 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74adb4b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80d1980 con 0x7f74a80cfff0 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74a77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 0x7f74a815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74a77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 0x7f74a815aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45712/0 (socket says 172.21.15.122:45712) 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.595+0000 7f74a77fe640 1 -- 172.21.15.122:0/3072102377 learned_addr learned my addr 172.21.15.122:0/3072102377 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a77fe640 1 -- 172.21.15.122:0/3072102377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a80cfff0 msgr2=0x7f74a806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a77fe640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a80cfff0 0x7f74a806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a77fe640 1 -- 172.21.15.122:0/3072102377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74a815bb20 msgr2=0x7f74a80c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a77fe640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74a815bb20 0x7f74a80c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a77fe640 1 -- 172.21.15.122:0/3072102377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a815e3e0 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a77fe640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 0x7f74a815aba0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f749800c7f0 tx=0x7f749800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498015070 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7498007820 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a815e6c0 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498007bd0 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.596+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a815ec80 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.597+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a804f330 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.598+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7498018070 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.598+0000 7f74a4ff9640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f748806b4d0 0x7f748806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.598+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f749808a180 con 0x7f74a80ce4b0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.598+0000 7f749ffff640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f748806b4d0 0x7f748806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.599+0000 7f749ffff640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f748806b4d0 0x7f748806d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7490036000 tx=0x7f7490030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.603+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7498058fe0 con 0x7f74a80ce4b0 2024-03-06T07:31:39.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.817+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74a80b1820 con 0x7f74a80ce4b0 2024-03-06T07:31:39.820 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.818+0000 7f74a4ff9640 1 -- 172.21.15.122:0/3072102377 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f74980588e0 con 0x7f74a80ce4b0 2024-03-06T07:31:39.820 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:39.820 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:39.820 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:39.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f748806b4d0 msgr2=0x7f748806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f748806b4d0 0x7f748806d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7490036000 tx=0x7f7490030000 comp rx=0 tx=0).stop 2024-03-06T07:31:39.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 msgr2=0x7f74a815aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 0x7f74a815aba0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f749800c7f0 tx=0x7f749800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 shutdown_connections 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f748806b4d0 0x7f748806d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f74a815bb20 0x7f74a80c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f74a80cfff0 0x7f74a806da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 --2- 172.21.15.122:0/3072102377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ce4b0 0x7f74a815aba0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 >> 172.21.15.122:0/3072102377 conn(0x7f74a809a9f0 msgr2=0x7f74a806fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 shutdown_connections 2024-03-06T07:31:39.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:39.820+0000 7f74adb4b640 1 -- 172.21.15.122:0/3072102377 wait complete. 2024-03-06T07:31:40.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:40.034 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:41.035 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.135+0000 7f99b613a640 1 Processor -- start 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99b613a640 1 -- start start 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99b613a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99b015dfe0 0x7f99b00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99b613a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00cf830 0x7f99b00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99b613a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00b3430 con 0x7f99b00d27f0 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99b613a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00d2c50 con 0x7f99b015dfe0 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99b613a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00b31e0 con 0x7f99b00cf830 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99af7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99b015dfe0 0x7f99b00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99affff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f99b00d27f0 0x7f99b00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58626/0 (socket says 172.21.15.122:58626) 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.136+0000 7f99affff640 1 -- 172.21.15.122:0/673884047 learned_addr learned my addr 172.21.15.122:0/673884047 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99aeffd640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00cf830 0x7f99b00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99af7fe640 1 -- 172.21.15.122:0/673884047 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00cf830 msgr2=0x7f99b00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99af7fe640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00cf830 0x7f99b00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99af7fe640 1 -- 172.21.15.122:0/673884047 >> v1:172.21.15.122:6789/0 conn(0x7f99b00d27f0 legacy=0x7f99b00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99af7fe640 1 -- 172.21.15.122:0/673884047 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00b3710 con 0x7f99b015dfe0 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99ae7fc640 1 -- 172.21.15.122:0/673884047 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930757561 0 0) 0x7f99b00b3430 con 0x7f99b00d27f0 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.137+0000 7f99aeffd640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00cf830 0x7f99b00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99af7fe640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b00cf2f0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f99a00097a0 tx=0x7f99a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a80bf4a0b2e039 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99ae7fc640 1 -- 172.21.15.122:0/673884047 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0035020 con 0x7f99b015dfe0 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99ae7fc640 1 -- 172.21.15.122:0/673884047 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0031830 con 0x7f99b015dfe0 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99ae7fc640 1 -- 172.21.15.122:0/673884047 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0031c00 con 0x7f99b015dfe0 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 -- 172.21.15.122:0/673884047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 msgr2=0x7f99b00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b00cf2f0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f99a00097a0 tx=0x7f99a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 -- 172.21.15.122:0/673884047 shutdown_connections 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00cf830 0x7f99b00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 --2- 172.21.15.122:0/673884047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b00cf2f0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 -- 172.21.15.122:0/673884047 >> 172.21.15.122:0/673884047 conn(0x7f99b00d3cd0 msgr2=0x7f99b00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 -- 172.21.15.122:0/673884047 shutdown_connections 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.138+0000 7f99b613a640 1 -- 172.21.15.122:0/673884047 wait complete. 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.139+0000 7f99b613a640 1 Processor -- start 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 -- start start 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99b00cf830 0x7f99b01f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00d27f0 0x7f99b00cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00d1c70 con 0x7f99b015dfe0 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00b3d60 con 0x7f99b00cf830 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99b613a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00d1df0 con 0x7f99b00d27f0 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99affff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99affff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b014e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45734/0 (socket says 172.21.15.122:45734) 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.140+0000 7f99affff640 1 -- 172.21.15.122:0/2027905897 learned_addr learned my addr 172.21.15.122:0/2027905897 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99aeffd640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00d27f0 0x7f99b00cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99af7fe640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99b00cf830 0x7f99b01f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99affff640 1 -- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00d27f0 msgr2=0x7f99b00cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99affff640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00d27f0 0x7f99b00cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99affff640 1 -- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99b00cf830 msgr2=0x7f99b01f2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99affff640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99b00cf830 0x7f99b01f2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99affff640 1 -- 172.21.15.122:0/2027905897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b014f280 con 0x7f99b015dfe0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99affff640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b014e8a0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c7e0 tx=0x7f99a800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8018070 con 0x7f99b015dfe0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a800a640 con 0x7f99b015dfe0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b014f560 con 0x7f99b015dfe0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800aa30 con 0x7f99b015dfe0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.141+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b00cb220 con 0x7f99b015dfe0 2024-03-06T07:31:41.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.142+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b00cb420 con 0x7f99b015dfe0 2024-03-06T07:31:41.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.143+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99a801b070 con 0x7f99b015dfe0 2024-03-06T07:31:41.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.143+0000 7f9997fff640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f999c06b4d0 0x7f999c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.143+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f99a80897c0 con 0x7f99b015dfe0 2024-03-06T07:31:41.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.143+0000 7f99af7fe640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f999c06b4d0 0x7f999c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.144+0000 7f99af7fe640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f999c06b4d0 0x7f999c06d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f99a0036000 tx=0x7f99a0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.152+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99b00cb420 con 0x7f99b015dfe0 2024-03-06T07:31:41.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.390+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99b004f330 con 0x7f99b015dfe0 2024-03-06T07:31:41.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.391+0000 7f9997fff640 1 -- 172.21.15.122:0/2027905897 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f99a8057fd0 con 0x7f99b015dfe0 2024-03-06T07:31:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:41.393 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:41.393 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f999c06b4d0 msgr2=0x7f999c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f999c06b4d0 0x7f999c06d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f99a0036000 tx=0x7f99a0008030 comp rx=0 tx=0).stop 2024-03-06T07:31:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 msgr2=0x7f99b014e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b014e8a0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c7e0 tx=0x7f99a800cca0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 shutdown_connections 2024-03-06T07:31:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f999c06b4d0 0x7f999c06d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99b015dfe0 0x7f99b014e8a0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99b00d27f0 0x7f99b00cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 --2- 172.21.15.122:0/2027905897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99b00cf830 0x7f99b01f2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 >> 172.21.15.122:0/2027905897 conn(0x7f99b00d3cd0 msgr2=0x7f99b00d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 shutdown_connections 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:41.393+0000 7f99b613a640 1 -- 172.21.15.122:0/2027905897 wait complete. 2024-03-06T07:31:41.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:41.607 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:42.608 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:42.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.708+0000 7fd3374b5640 1 Processor -- start 2024-03-06T07:31:42.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 -- start start 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33013dc10 0x7fd3300b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd33015bf30 0x7fd3300b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd33015e850 0x7fd33015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3300cf3f0 con 0x7fd33015e850 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33015b210 con 0x7fd33015bf30 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd3374b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33015f580 con 0x7fd33013dc10 2024-03-06T07:31:42.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.709+0000 7fd335a52640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd33015e850 0x7fd33015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd33015e850 0x7fd33015eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45748/0 (socket says 172.21.15.122:45748) 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 -- 172.21.15.122:0/1026130837 learned_addr learned my addr 172.21.15.122:0/1026130837 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335251640 1 --2- 172.21.15.122:0/1026130837 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33013dc10 0x7fd3300b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd334a50640 1 --2- 172.21.15.122:0/1026130837 >> v2:172.21.15.169:3300/0 conn(0x7fd33015bf30 0x7fd3300b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 -- 172.21.15.122:0/1026130837 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33013dc10 msgr2=0x7fd3300b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 --2- 172.21.15.122:0/1026130837 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33013dc10 0x7fd3300b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 -- 172.21.15.122:0/1026130837 >> v2:172.21.15.169:3300/0 conn(0x7fd33015bf30 msgr2=0x7fd3300b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 --2- 172.21.15.122:0/1026130837 >> v2:172.21.15.169:3300/0 conn(0x7fd33015bf30 0x7fd3300b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.710+0000 7fd335a52640 1 -- 172.21.15.122:0/1026130837 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33015b450 con 0x7fd33015e850 2024-03-06T07:31:42.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd335a52640 1 --2- 172.21.15.122:0/1026130837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015e850 0x7fd33015eca0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fd330068890 tx=0x7fd32802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2964a5c577eca959 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd327fff640 1 -- 172.21.15.122:0/1026130837 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32802c9e0 con 0x7fd33015e850 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd327fff640 1 -- 172.21.15.122:0/1026130837 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32802cb80 con 0x7fd33015e850 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd327fff640 1 -- 172.21.15.122:0/1026130837 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3280343f0 con 0x7fd33015e850 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 -- 172.21.15.122:0/1026130837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015e850 msgr2=0x7fd33015eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 --2- 172.21.15.122:0/1026130837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015e850 0x7fd33015eca0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fd330068890 tx=0x7fd32802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 -- 172.21.15.122:0/1026130837 shutdown_connections 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 --2- 172.21.15.122:0/1026130837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015e850 0x7fd33015eca0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 --2- 172.21.15.122:0/1026130837 >> v2:172.21.15.169:3300/0 conn(0x7fd33015bf30 0x7fd3300b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 --2- 172.21.15.122:0/1026130837 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33013dc10 0x7fd3300b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 -- 172.21.15.122:0/1026130837 >> 172.21.15.122:0/1026130837 conn(0x7fd3300cf8a0 msgr2=0x7fd3300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 -- 172.21.15.122:0/1026130837 shutdown_connections 2024-03-06T07:31:42.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.711+0000 7fd3374b5640 1 -- 172.21.15.122:0/1026130837 wait complete. 2024-03-06T07:31:42.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 Processor -- start 2024-03-06T07:31:42.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 -- start start 2024-03-06T07:31:42.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd33013dc10 0x7fd330154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 0x7fd3301f35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33015e850 0x7fd330071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33015f1e0 con 0x7fd33015bf30 2024-03-06T07:31:42.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33013dff0 con 0x7fd33013dc10 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd3374b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33015f360 con 0x7fd33015e850 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd334a50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 0x7fd3301f35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd334a50640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 0x7fd3301f35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45750/0 (socket says 172.21.15.122:45750) 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.713+0000 7fd334a50640 1 -- 172.21.15.122:0/2547758374 learned_addr learned my addr 172.21.15.122:0/2547758374 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd335a52640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33015e850 0x7fd330071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd335251640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd33013dc10 0x7fd330154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd334a50640 1 -- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33015e850 msgr2=0x7fd330071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd334a50640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33015e850 0x7fd330071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd334a50640 1 -- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd33013dc10 msgr2=0x7fd330154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd334a50640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd33013dc10 0x7fd330154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd334a50640 1 -- 172.21.15.122:0/2547758374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33013f420 con 0x7fd33015bf30 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd335251640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd33013dc10 0x7fd330154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd334a50640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 0x7fd3301f35f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd32000d800 tx=0x7fd32000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32000b6c0 con 0x7fd33015bf30 2024-03-06T07:31:42.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd320019070 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32000ba80 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3301f77f0 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.714+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3301f7db0 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.715+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33004f330 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.716+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd32000bc20 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.716+0000 7fd325ffb640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd31406b4d0 0x7fd31406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.716+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd32008a630 con 0x7fd33015bf30 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.717+0000 7fd335251640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd31406b4d0 0x7fd31406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.717+0000 7fd335251640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd31406b4d0 0x7fd31406d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00ba50 tx=0x7fd32c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.725+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd320059490 con 0x7fd33015bf30 2024-03-06T07:31:42.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.938+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3300af940 con 0x7fd33015bf30 2024-03-06T07:31:42.941 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.939+0000 7fd325ffb640 1 -- 172.21.15.122:0/2547758374 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fd320058d90 con 0x7fd33015bf30 2024-03-06T07:31:42.941 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:42.941 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:42.941 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.940+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd31406b4d0 msgr2=0x7fd31406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.940+0000 7fd3374b5640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd31406b4d0 0x7fd31406d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00ba50 tx=0x7fd32c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.940+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 msgr2=0x7fd3301f35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.940+0000 7fd3374b5640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 0x7fd3301f35f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd32000d800 tx=0x7fd32000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 shutdown_connections 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd31406b4d0 0x7fd31406d990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd33015e850 0x7fd330071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd33015bf30 0x7fd3301f35f0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 --2- 172.21.15.122:0/2547758374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd33013dc10 0x7fd330154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 >> 172.21.15.122:0/2547758374 conn(0x7fd3300cf8a0 msgr2=0x7fd33015c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 shutdown_connections 2024-03-06T07:31:42.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:42.941+0000 7fd3374b5640 1 -- 172.21.15.122:0/2547758374 wait complete. 2024-03-06T07:31:43.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:43.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:44.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:44.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.260+0000 7f78fcd3e640 1 Processor -- start 2024-03-06T07:31:44.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.260+0000 7f78fcd3e640 1 -- start start 2024-03-06T07:31:44.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78fcd3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 0x7f78f80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78fcd3e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 0x7f78f80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78fcd3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f815f580 con 0x7f78f815e850 2024-03-06T07:31:44.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78fcd3e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f80cf3f0 con 0x7f78f815bf30 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78fcd3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f815b210 con 0x7f78f813dc10 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f5d9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 0x7f78f80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f659c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 0x7f78f80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f5d9b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 0x7f78f80b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:34478/0 (socket says 172.21.15.122:34478) 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f659c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 0x7f78f80b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:55666/0 (socket says 172.21.15.122:55666) 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f5d9b640 1 -- 172.21.15.122:0/158595575 learned_addr learned my addr 172.21.15.122:0/158595575 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f6d9d640 1 --1- 172.21.15.122:0/158595575 >> v1:172.21.15.169:6789/0 conn(0x7f78f815e850 0x7f78f815eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:59066/0 (socket says 172.21.15.122:59066) 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f659c640 1 -- 172.21.15.122:0/158595575 >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 msgr2=0x7f78f80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f659c640 1 --2- 172.21.15.122:0/158595575 >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 0x7f78f80b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f659c640 1 -- 172.21.15.122:0/158595575 >> v1:172.21.15.169:6789/0 conn(0x7f78f815e850 legacy=0x7f78f815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f659c640 1 -- 172.21.15.122:0/158595575 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f815b450 con 0x7f78f813dc10 2024-03-06T07:31:44.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.262+0000 7f78f5d9b640 1 --2- 172.21.15.122:0/158595575 >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 0x7f78f80b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.263+0000 7f78f659c640 1 --2- 172.21.15.122:0/158595575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 0x7f78f80b2950 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009d90 tx=0x7f78e802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=978b36958b2eaeb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.264+0000 7f78f559a640 1 -- 172.21.15.122:0/158595575 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8039070 con 0x7f78f813dc10 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.264+0000 7f78f559a640 1 -- 172.21.15.122:0/158595575 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e802cbc0 con 0x7f78f813dc10 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.264+0000 7f78f559a640 1 -- 172.21.15.122:0/158595575 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8034440 con 0x7f78f813dc10 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 -- 172.21.15.122:0/158595575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 msgr2=0x7f78f80b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/158595575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 0x7f78f80b2950 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009d90 tx=0x7f78e802be10 comp rx=0 tx=0).stop 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 -- 172.21.15.122:0/158595575 shutdown_connections 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/158595575 >> v2:172.21.15.169:3300/0 conn(0x7f78f815bf30 0x7f78f80b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/158595575 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f813dc10 0x7f78f80b2950 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 -- 172.21.15.122:0/158595575 >> 172.21.15.122:0/158595575 conn(0x7f78f80cf8a0 msgr2=0x7f78f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 -- 172.21.15.122:0/158595575 shutdown_connections 2024-03-06T07:31:44.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.265+0000 7f78fcd3e640 1 -- 172.21.15.122:0/158595575 wait complete. 2024-03-06T07:31:44.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.266+0000 7f78fcd3e640 1 Processor -- start 2024-03-06T07:31:44.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.266+0000 7f78fcd3e640 1 -- start start 2024-03-06T07:31:44.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78fcd3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78f813dc10 0x7f78f813a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78fcd3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f815bf30 0x7f78f815a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78fcd3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 0x7f78f813c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78fcd3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80cd7a0 con 0x7f78f815e850 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78fcd3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f813dff0 con 0x7f78f813dc10 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78fcd3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80cd920 con 0x7f78f815bf30 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 0x7f78f813c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 0x7f78f813c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45754/0 (socket says 172.21.15.122:45754) 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 -- 172.21.15.122:0/27809119 learned_addr learned my addr 172.21.15.122:0/27809119 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f5d9b640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f815bf30 0x7f78f815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f659c640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78f813dc10 0x7f78f813a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 -- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f815bf30 msgr2=0x7f78f815a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f815bf30 0x7f78f815a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 -- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78f813dc10 msgr2=0x7f78f813a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78f813dc10 0x7f78f813a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 -- 172.21.15.122:0/27809119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f81597c0 con 0x7f78f815e850 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.267+0000 7f78f6d9d640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 0x7f78f813c400 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f78f000e730 tx=0x7f78f000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.268+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f001a070 con 0x7f78f815e850 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.268+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f0006bf0 con 0x7f78f815e850 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.268+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000c640 con 0x7f78f815e850 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.268+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f815a760 con 0x7f78f815e850 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.268+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f8158a00 con 0x7f78f815e850 2024-03-06T07:31:44.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.269+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f815d370 con 0x7f78f815e850 2024-03-06T07:31:44.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.269+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78f001d070 con 0x7f78f815e850 2024-03-06T07:31:44.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.269+0000 7f78e6ffd640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b4d0 0x7f78d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.270+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f78f008b470 con 0x7f78f815e850 2024-03-06T07:31:44.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.270+0000 7f78f659c640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b4d0 0x7f78d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.273 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.270+0000 7f78f659c640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b4d0 0x7f78d806d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f78e8036000 tx=0x7f78e8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.274+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78f815d370 con 0x7f78f815e850 2024-03-06T07:31:44.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.540+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f806eb00 con 0x7f78f815e850 2024-03-06T07:31:44.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.540+0000 7f78e6ffd640 1 -- 172.21.15.122:0/27809119 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f78f81597c0 con 0x7f78f815e850 2024-03-06T07:31:44.543 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:44.543 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:44.543 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:44.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b4d0 msgr2=0x7f78d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b4d0 0x7f78d806d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f78e8036000 tx=0x7f78e8030000 comp rx=0 tx=0).stop 2024-03-06T07:31:44.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 msgr2=0x7f78f813c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 0x7f78f813c400 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f78f000e730 tx=0x7f78f000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 shutdown_connections 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f78d806b4d0 0x7f78d806d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78f815e850 0x7f78f813c400 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f78f815bf30 0x7f78f815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 --2- 172.21.15.122:0/27809119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f78f813dc10 0x7f78f813a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 >> 172.21.15.122:0/27809119 conn(0x7f78f80cf8a0 msgr2=0x7f78f813ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 shutdown_connections 2024-03-06T07:31:44.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:44.543+0000 7f78fcd3e640 1 -- 172.21.15.122:0/27809119 wait complete. 2024-03-06T07:31:44.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:44.757 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:45.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:45.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.857+0000 7f9b299d7640 1 Processor -- start 2024-03-06T07:31:45.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.857+0000 7f9b299d7640 1 -- start start 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b299d7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9b241515e0 0x7f9b240cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b299d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b24071f40 con 0x7f9b240cd7a0 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b299d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2415b8e0 con 0x7f9b2415b430 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b299d7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b24070a70 con 0x7f9b241515e0 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b237fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9b240cd7a0 0x7f9b240cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37480/0 (socket says 172.21.15.122:37480) 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b237fe640 1 -- 172.21.15.122:0/2221548017 learned_addr learned my addr 172.21.15.122:0/2221548017 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851315952 0 0) 0x7f9b24071f40 con 0x7f9b240cd7a0 2024-03-06T07:31:45.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/2221548017 >> v2:172.21.15.169:3300/0 conn(0x7f9b241515e0 0x7f9b240cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b0c003520 con 0x7f9b240cd7a0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587806971 0 0) 0x7f9b0c003520 con 0x7f9b240cd7a0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 >> v2:172.21.15.169:3300/0 conn(0x7f9b241515e0 msgr2=0x7f9b240cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b21ffb640 1 --2- 172.21.15.122:0/2221548017 >> v2:172.21.15.169:3300/0 conn(0x7f9b241515e0 0x7f9b240cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.858+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 >> v1:172.21.15.169:6789/0 conn(0x7f9b2415b430 legacy=0x7f9b24071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b24070d50 con 0x7f9b240cd7a0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f9b1c001a10 con 0x7f9b240cd7a0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 >> v1:172.21.15.122:6789/0 conn(0x7f9b240cd7a0 legacy=0x7f9b240cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b0c004800 0x7f9b0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b0c0051c0 0x7f9b0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24070a70 con 0x7f9b0c0039b0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c00c060 con 0x7f9b0c0051c0 2024-03-06T07:31:45.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c00c260 con 0x7f9b0c004800 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b237fe640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b0c0051c0 0x7f9b0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b227fc640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b0c004800 0x7f9b0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b22ffd640 1 -- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b0c004800 msgr2=0x7f9b0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b0c004800 0x7f9b0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b22ffd640 1 -- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b0c0051c0 msgr2=0x7f9b0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b0c0051c0 0x7f9b0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b237fe640 1 -- 172.21.15.122:0/2221548017 reap_dead start 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.859+0000 7f9b22ffd640 1 -- 172.21.15.122:0/2221548017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c003520 con 0x7f9b0c0039b0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.860+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b0c003dd0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f9b180093c0 tx=0x7f9b0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36da3b6e877f7501 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.860+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1803a070 con 0x7f9b0c0039b0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.860+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1802d7c0 con 0x7f9b0c0039b0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.860+0000 7f9b21ffb640 1 -- 172.21.15.122:0/2221548017 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18009690 con 0x7f9b0c0039b0 2024-03-06T07:31:45.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 -- 172.21.15.122:0/2221548017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 msgr2=0x7f9b0c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b0c003dd0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f9b180093c0 tx=0x7f9b0c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:45.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 -- 172.21.15.122:0/2221548017 shutdown_connections 2024-03-06T07:31:45.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 --2- 172.21.15.122:0/2221548017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b0c003dd0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 -- 172.21.15.122:0/2221548017 >> 172.21.15.122:0/2221548017 conn(0x7f9b240cf8a0 msgr2=0x7f9b240cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 -- 172.21.15.122:0/2221548017 shutdown_connections 2024-03-06T07:31:45.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.861+0000 7f9b299d7640 1 -- 172.21.15.122:0/2221548017 wait complete. 2024-03-06T07:31:45.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.862+0000 7f9b299d7640 1 Processor -- start 2024-03-06T07:31:45.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.862+0000 7f9b299d7640 1 -- start start 2024-03-06T07:31:45.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b299d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b2406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b299d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b241e58b0 0x7f9b241e3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b299d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b241e4300 0x7f9b24144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b299d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240cce40 con 0x7f9b0c0039b0 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b299d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24071310 con 0x7f9b241e58b0 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b299d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240ccfc0 con 0x7f9b241e4300 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b2406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b2406dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40198/0 (socket says 172.21.15.122:40198) 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 -- 172.21.15.122:0/3715228902 learned_addr learned my addr 172.21.15.122:0/3715228902 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b237fe640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b241e4300 0x7f9b24144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b227fc640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b241e58b0 0x7f9b241e3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 -- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b241e4300 msgr2=0x7f9b24144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b241e4300 0x7f9b24144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 -- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b241e58b0 msgr2=0x7f9b241e3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b241e58b0 0x7f9b241e3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.863+0000 7f9b22ffd640 1 -- 172.21.15.122:0/3715228902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b241e4810 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.864+0000 7f9b22ffd640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b2406dac0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f9b18038040 tx=0x7f9b18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.864+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1803a040 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.864+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b18045070 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.864+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18009690 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.864+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b241e4b50 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.864+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b241e5390 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.865+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b24068840 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.865+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b18003850 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.866+0000 7f9b0bfff640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b1006b500 0x7f9b1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.866+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9b180aba20 con 0x7f9b0c0039b0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.866+0000 7f9b227fc640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b1006b500 0x7f9b1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.866+0000 7f9b227fc640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b1006b500 0x7f9b1006d9c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9b14000c00 tx=0x7f9b14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.872 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:45.870+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b1804e020 con 0x7f9b0c0039b0 2024-03-06T07:31:46.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.086+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b2404f330 con 0x7f9b0c0039b0 2024-03-06T07:31:46.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.086+0000 7f9b0bfff640 1 -- 172.21.15.122:0/3715228902 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f9b241e4b50 con 0x7f9b0c0039b0 2024-03-06T07:31:46.089 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:46.089 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:46.089 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b1006b500 msgr2=0x7f9b1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b1006b500 0x7f9b1006d9c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9b14000c00 tx=0x7f9b14009040 comp rx=0 tx=0).stop 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 msgr2=0x7f9b2406dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b2406dac0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f9b18038040 tx=0x7f9b18004730 comp rx=0 tx=0).stop 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 shutdown_connections 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9b1006b500 0x7f9b1006d9c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9b241e4300 0x7f9b24144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b241e58b0 0x7f9b241e3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 --2- 172.21.15.122:0/3715228902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b0c0039b0 0x7f9b2406dac0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.088+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 >> 172.21.15.122:0/3715228902 conn(0x7f9b240cf8a0 msgr2=0x7f9b24072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.089+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 shutdown_connections 2024-03-06T07:31:46.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:46.089+0000 7f9b299d7640 1 -- 172.21.15.122:0/3715228902 wait complete. 2024-03-06T07:31:46.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:46.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:47.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.402+0000 7f38087d0640 1 Processor -- start 2024-03-06T07:31:47.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.402+0000 7f38087d0640 1 -- start start 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f38087d0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f380015dfe0 0x7f38000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f38087d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38000d29c0 0x7f38000b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f38087d0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38000b3430 con 0x7f38000cf830 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f38087d0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38000b31e0 con 0x7f380015dfe0 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f38087d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38000b3d60 con 0x7f38000d29c0 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f380656c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f380015dfe0 0x7f38000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f380656c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f380015dfe0 0x7f38000cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40212/0 (socket says 172.21.15.122:40212) 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.403+0000 7f380656c640 1 -- 172.21.15.122:0/1881732803 learned_addr learned my addr 172.21.15.122:0/1881732803 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.404+0000 7f380656c640 1 -- 172.21.15.122:0/1881732803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38000d29c0 msgr2=0x7f38000b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.404+0000 7f380656c640 1 --2- 172.21.15.122:0/1881732803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38000d29c0 0x7f38000b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.404+0000 7f380656c640 1 -- 172.21.15.122:0/1881732803 >> v1:172.21.15.169:6789/0 conn(0x7f38000cf830 legacy=0x7f38000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.404+0000 7f380656c640 1 -- 172.21.15.122:0/1881732803 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000b3710 con 0x7f380015dfe0 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.404+0000 7f380656c640 1 --2- 172.21.15.122:0/1881732803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380015dfe0 0x7f38000cf2f0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009e50 tx=0x7f37f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985b07219ef8dda7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f380556a640 1 -- 172.21.15.122:0/1881732803 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f802ca70 con 0x7f380015dfe0 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f380556a640 1 -- 172.21.15.122:0/1881732803 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f802cc10 con 0x7f380015dfe0 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f380556a640 1 -- 172.21.15.122:0/1881732803 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80343f0 con 0x7f380015dfe0 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f38087d0640 1 -- 172.21.15.122:0/1881732803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380015dfe0 msgr2=0x7f38000cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f38087d0640 1 --2- 172.21.15.122:0/1881732803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380015dfe0 0x7f38000cf2f0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009e50 tx=0x7f37f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f38087d0640 1 -- 172.21.15.122:0/1881732803 shutdown_connections 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f38087d0640 1 --2- 172.21.15.122:0/1881732803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f38000d29c0 0x7f38000b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f38087d0640 1 --2- 172.21.15.122:0/1881732803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f380015dfe0 0x7f38000cf2f0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.405+0000 7f38087d0640 1 -- 172.21.15.122:0/1881732803 >> 172.21.15.122:0/1881732803 conn(0x7f38000d3cd0 msgr2=0x7f38000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.406+0000 7f38087d0640 1 -- 172.21.15.122:0/1881732803 shutdown_connections 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.406+0000 7f38087d0640 1 -- 172.21.15.122:0/1881732803 wait complete. 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.407+0000 7f38087d0640 1 Processor -- start 2024-03-06T07:31:47.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.407+0000 7f38087d0640 1 -- start start 2024-03-06T07:31:47.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f38087d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38000cf830 0x7f3800152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f38087d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 0x7f3800160ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f38087d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380015dfe0 0x7f38001610e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f38087d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38000d1df0 con 0x7f38000d29c0 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f38087d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38000d1c70 con 0x7f38000cf830 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f38087d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380015e3c0 con 0x7f380015dfe0 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f3805d6b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 0x7f3800160ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f3805d6b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 0x7f3800160ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40226/0 (socket says 172.21.15.122:40226) 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f3805d6b640 1 -- 172.21.15.122:0/2400352804 learned_addr learned my addr 172.21.15.122:0/2400352804 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f380656c640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38000cf830 0x7f3800152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f3806d6d640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380015dfe0 0x7f38001610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f3805d6b640 1 -- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380015dfe0 msgr2=0x7f38001610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.408+0000 7f3805d6b640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380015dfe0 0x7f38001610e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f3805d6b640 1 -- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38000cf830 msgr2=0x7f3800152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f3805d6b640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38000cf830 0x7f3800152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f3805d6b640 1 -- 172.21.15.122:0/2400352804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38001f36a0 con 0x7f38000d29c0 2024-03-06T07:31:47.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f3806d6d640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380015dfe0 0x7f38001610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f3805d6b640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 0x7f3800160ba0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00c430 tx=0x7f37fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc017070 con 0x7f38000d29c0 2024-03-06T07:31:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc0035f0 con 0x7f38000d29c0 2024-03-06T07:31:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3800153020 con 0x7f38000d29c0 2024-03-06T07:31:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0039e0 con 0x7f38000d29c0 2024-03-06T07:31:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.409+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38000cbe60 con 0x7f38000d29c0 2024-03-06T07:31:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.410+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38000cc060 con 0x7f38000d29c0 2024-03-06T07:31:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.411+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37fc01a070 con 0x7f38000d29c0 2024-03-06T07:31:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.411+0000 7f37f2ffd640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37e406b4d0 0x7f37e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.411+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f37fc0894a0 con 0x7f38000d29c0 2024-03-06T07:31:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.411+0000 7f380656c640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37e406b4d0 0x7f37e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.411+0000 7f380656c640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37e406b4d0 0x7f37e406d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f37f8035000 tx=0x7f37f8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.415+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38000cc060 con 0x7f38000d29c0 2024-03-06T07:31:47.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.714+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f380004f330 con 0x7f38000d29c0 2024-03-06T07:31:47.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.715+0000 7f37f2ffd640 1 -- 172.21.15.122:0/2400352804 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f37fc0010c0 con 0x7f38000d29c0 2024-03-06T07:31:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:47.717 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:47.717 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37e406b4d0 msgr2=0x7f37e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37e406b4d0 0x7f37e406d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f37f8035000 tx=0x7f37f8030000 comp rx=0 tx=0).stop 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 msgr2=0x7f3800160ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 0x7f3800160ba0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00c430 tx=0x7f37fc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 shutdown_connections 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f37e406b4d0 0x7f37e406d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f380015dfe0 0x7f38001610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38000d29c0 0x7f3800160ba0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 --2- 172.21.15.122:0/2400352804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38000cf830 0x7f3800152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 >> 172.21.15.122:0/2400352804 conn(0x7f38000d3cd0 msgr2=0x7f38000d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 shutdown_connections 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:47.717+0000 7f38087d0640 1 -- 172.21.15.122:0/2400352804 wait complete. 2024-03-06T07:31:47.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:47.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:48.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:49.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.031+0000 7f29f5f63640 1 Processor -- start 2024-03-06T07:31:49.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29f5f63640 1 -- start start 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29f5f63640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29f015dfe0 0x7f29f00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29f5f63640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00d29c0 0x7f29f00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29f5f63640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f00b3430 con 0x7f29f00cf830 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29f5f63640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f00b31e0 con 0x7f29f015dfe0 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29f5f63640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f00b3d60 con 0x7f29f00d29c0 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29ef7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29f015dfe0 0x7f29f00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29ef7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29f015dfe0 0x7f29f00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40242/0 (socket says 172.21.15.122:40242) 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29ef7fe640 1 -- 172.21.15.122:0/2602175452 learned_addr learned my addr 172.21.15.122:0/2602175452 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:49.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.032+0000 7f29eeffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f29f00cf830 0x7f29f00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37490/0 (socket says 172.21.15.122:37490) 2024-03-06T07:31:49.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.033+0000 7f29effff640 1 --2- 172.21.15.122:0/2602175452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00d29c0 0x7f29f00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.033+0000 7f29ef7fe640 1 -- 172.21.15.122:0/2602175452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00d29c0 msgr2=0x7f29f00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.033+0000 7f29ef7fe640 1 --2- 172.21.15.122:0/2602175452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00d29c0 0x7f29f00b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.033+0000 7f29ef7fe640 1 -- 172.21.15.122:0/2602175452 >> v1:172.21.15.122:6789/0 conn(0x7f29f00cf830 legacy=0x7f29f00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.033+0000 7f29ef7fe640 1 -- 172.21.15.122:0/2602175452 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f00b3710 con 0x7f29f015dfe0 2024-03-06T07:31:49.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.033+0000 7f29ef7fe640 1 --2- 172.21.15.122:0/2602175452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00cf2f0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009e50 tx=0x7f29e002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2e4c850a8df3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29ee7fc640 1 -- 172.21.15.122:0/2602175452 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e002cb00 con 0x7f29f015dfe0 2024-03-06T07:31:49.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29ee7fc640 1 -- 172.21.15.122:0/2602175452 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e002cca0 con 0x7f29f015dfe0 2024-03-06T07:31:49.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29ee7fc640 1 -- 172.21.15.122:0/2602175452 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0034610 con 0x7f29f015dfe0 2024-03-06T07:31:49.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 -- 172.21.15.122:0/2602175452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 msgr2=0x7f29f00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 --2- 172.21.15.122:0/2602175452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00cf2f0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009e50 tx=0x7f29e002bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 -- 172.21.15.122:0/2602175452 shutdown_connections 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 --2- 172.21.15.122:0/2602175452 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00d29c0 0x7f29f00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 --2- 172.21.15.122:0/2602175452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00cf2f0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 -- 172.21.15.122:0/2602175452 >> 172.21.15.122:0/2602175452 conn(0x7f29f00d3cd0 msgr2=0x7f29f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 -- 172.21.15.122:0/2602175452 shutdown_connections 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.034+0000 7f29f5f63640 1 -- 172.21.15.122:0/2602175452 wait complete. 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.035+0000 7f29f5f63640 1 Processor -- start 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 -- start start 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00cf830 0x7f29f015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f00d29c0 0x7f29f015e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f00d1df0 con 0x7f29f015dfe0 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f00d1c70 con 0x7f29f00d29c0 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29f5f63640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f015e3c0 con 0x7f29f00cf830 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40250/0 (socket says 172.21.15.122:40250) 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29effff640 1 -- 172.21.15.122:0/1705198519 learned_addr learned my addr 172.21.15.122:0/1705198519 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.036+0000 7f29ef7fe640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00cf830 0x7f29f015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29eeffd640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f00d29c0 0x7f29f015e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29effff640 1 -- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00cf830 msgr2=0x7f29f015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29effff640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00cf830 0x7f29f015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29effff640 1 -- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f00d29c0 msgr2=0x7f29f015e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29effff640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f00d29c0 0x7f29f015e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29effff640 1 -- 172.21.15.122:0/1705198519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f015b5f0 con 0x7f29f015dfe0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29effff640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00b88b0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f29e800c820 tx=0x7f29e800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8015070 con 0x7f29f015dfe0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e8004020 con 0x7f29f015dfe0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8004410 con 0x7f29f015dfe0 2024-03-06T07:31:49.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f015f250 con 0x7f29f015dfe0 2024-03-06T07:31:49.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.037+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f015ffa0 con 0x7f29f015dfe0 2024-03-06T07:31:49.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.038+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f004f330 con 0x7f29f015dfe0 2024-03-06T07:31:49.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.039+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29e8007830 con 0x7f29f015dfe0 2024-03-06T07:31:49.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.039+0000 7f29e7fff640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29d006b500 0x7f29d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.039+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f29e8089e90 con 0x7f29f015dfe0 2024-03-06T07:31:49.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.039+0000 7f29ef7fe640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29d006b500 0x7f29d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.039+0000 7f29ef7fe640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29d006b500 0x7f29d006d9c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f29e0035000 tx=0x7f29e0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.047+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29e8058730 con 0x7f29f015dfe0 2024-03-06T07:31:49.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.256+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f00747a0 con 0x7f29f015dfe0 2024-03-06T07:31:49.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.257+0000 7f29e7fff640 1 -- 172.21.15.122:0/1705198519 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f29e805c7d0 con 0x7f29f015dfe0 2024-03-06T07:31:49.259 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:49.259 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:49.259 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29d006b500 msgr2=0x7f29d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29d006b500 0x7f29d006d9c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f29e0035000 tx=0x7f29e0030000 comp rx=0 tx=0).stop 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 msgr2=0x7f29f00b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00b88b0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f29e800c820 tx=0x7f29e800cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 shutdown_connections 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29d006b500 0x7f29d006d9c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f015dfe0 0x7f29f00b88b0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f00d29c0 0x7f29f015e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.261 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 --2- 172.21.15.122:0/1705198519 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f00cf830 0x7f29f015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 >> 172.21.15.122:0/1705198519 conn(0x7f29f00d3cd0 msgr2=0x7f29f006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 shutdown_connections 2024-03-06T07:31:49.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:49.259+0000 7f29f5f63640 1 -- 172.21.15.122:0/1705198519 wait complete. 2024-03-06T07:31:49.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:49.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:50.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:50.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 Processor -- start 2024-03-06T07:31:50.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 -- start start 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63d015dfe0 0x7f63d00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d00b3430 con 0x7f63d00cf830 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00b31e0 con 0x7f63d015dfe0 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.573+0000 7f63d51a1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00b3d60 con 0x7f63d00d29c0 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63d015dfe0 0x7f63d00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f63d015dfe0 0x7f63d00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40258/0 (socket says 172.21.15.122:40258) 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 -- 172.21.15.122:0/700742677 learned_addr learned my addr 172.21.15.122:0/700742677 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 -- 172.21.15.122:0/700742677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 msgr2=0x7f63d00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 --2- 172.21.15.122:0/700742677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 -- 172.21.15.122:0/700742677 >> v1:172.21.15.169:6789/0 conn(0x7f63d00cf830 legacy=0x7f63d00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.574+0000 7f63ced9d640 1 -- 172.21.15.122:0/700742677 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00b3710 con 0x7f63d015dfe0 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63ced9d640 1 --2- 172.21.15.122:0/700742677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d015dfe0 0x7f63d00cf2f0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009e50 tx=0x7f63c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92066bd17f43b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63cdd9b640 1 -- 172.21.15.122:0/700742677 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c802ca70 con 0x7f63d015dfe0 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63cdd9b640 1 -- 172.21.15.122:0/700742677 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c802cc10 con 0x7f63d015dfe0 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63cdd9b640 1 -- 172.21.15.122:0/700742677 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c80343f0 con 0x7f63d015dfe0 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 -- 172.21.15.122:0/700742677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d015dfe0 msgr2=0x7f63d00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 --2- 172.21.15.122:0/700742677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d015dfe0 0x7f63d00cf2f0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009e50 tx=0x7f63c802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 -- 172.21.15.122:0/700742677 shutdown_connections 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 --2- 172.21.15.122:0/700742677 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 --2- 172.21.15.122:0/700742677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d015dfe0 0x7f63d00cf2f0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 -- 172.21.15.122:0/700742677 >> 172.21.15.122:0/700742677 conn(0x7f63d00d3cd0 msgr2=0x7f63d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 -- 172.21.15.122:0/700742677 shutdown_connections 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.575+0000 7f63d51a1640 1 -- 172.21.15.122:0/700742677 wait complete. 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.577+0000 7f63d51a1640 1 Processor -- start 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.577+0000 7f63d51a1640 1 -- start start 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.577+0000 7f63d51a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 0x7f63d0152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d0160ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63d015dfe0 0x7f63d01610e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00d1df0 con 0x7f63d00cf830 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00d1c70 con 0x7f63d015dfe0 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d015e3c0 con 0x7f63d00d29c0 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 0x7f63d0152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 0x7f63d0152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40266/0 (socket says 172.21.15.122:40266) 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 -- 172.21.15.122:0/923304596 learned_addr learned my addr 172.21.15.122:0/923304596 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ce59c640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d0160ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63cf59e640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63d015dfe0 0x7f63d01610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 -- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 msgr2=0x7f63d0160ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d0160ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 -- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63d015dfe0 msgr2=0x7f63d01610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63d015dfe0 0x7f63d01610e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 -- 172.21.15.122:0/923304596 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d01f36a0 con 0x7f63d00cf830 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63ced9d640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 0x7f63d0152ae0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f63c8036040 tx=0x7f63c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8043020 con 0x7f63d00cf830 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c803e450 con 0x7f63d00cf830 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d0153020 con 0x7f63d00cf830 2024-03-06T07:31:50.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c803e840 con 0x7f63d00cf830 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.578+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d00cbe60 con 0x7f63d00cf830 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.579+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d004f330 con 0x7f63d00cf830 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.580+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63c8046020 con 0x7f63d00cf830 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.581+0000 7f63b77fe640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63bc06b4d0 0x7f63bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.581+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f63c80acb00 con 0x7f63d00cf830 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.581+0000 7f63ce59c640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63bc06b4d0 0x7f63bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.581+0000 7f63ce59c640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63bc06b4d0 0x7f63bc06d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f63d0160a60 tx=0x7f63c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.585+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63c804d030 con 0x7f63d00cf830 2024-03-06T07:31:50.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.793+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63d00747a0 con 0x7f63d00cf830 2024-03-06T07:31:50.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.794+0000 7f63b77fe640 1 -- 172.21.15.122:0/923304596 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f63c807b9f0 con 0x7f63d00cf830 2024-03-06T07:31:50.796 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:50.796 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:50.796 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63bc06b4d0 msgr2=0x7f63bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63bc06b4d0 0x7f63bc06d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f63d0160a60 tx=0x7f63c4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 msgr2=0x7f63d0152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 0x7f63d0152ae0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f63c8036040 tx=0x7f63c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 shutdown_connections 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f63bc06b4d0 0x7f63bc06d990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63d015dfe0 0x7f63d01610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f63d00d29c0 0x7f63d0160ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 --2- 172.21.15.122:0/923304596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63d00cf830 0x7f63d0152ae0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 >> 172.21.15.122:0/923304596 conn(0x7f63d00d3cd0 msgr2=0x7f63d00d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 shutdown_connections 2024-03-06T07:31:50.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:50.796+0000 7f63d51a1640 1 -- 172.21.15.122:0/923304596 wait complete. 2024-03-06T07:31:51.009 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:51.009 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:52.011 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:52.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.110+0000 7f346a833640 1 Processor -- start 2024-03-06T07:31:52.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.110+0000 7f346a833640 1 -- start start 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f346a833640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f34640d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f346a833640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3464162400 0x7f34640cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f346a833640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640b1070 con 0x7f3464162a00 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f346a833640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640ce9d0 con 0x7f3464162400 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f346a833640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640d2da0 con 0x7f3464162020 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f34637fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3464162a00 0x7f34640b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37506/0 (socket says 172.21.15.122:37506) 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f34637fe640 1 -- 172.21.15.122:0/3372929892 learned_addr learned my addr 172.21.15.122:0/3372929892 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.111+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611635310 0 0) 0x7f34640b1070 con 0x7f3464162a00 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f344c003520 con 0x7f3464162a00 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3463fff640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f34640d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3372929892 >> v2:172.21.15.169:3300/0 conn(0x7f3464162400 0x7f34640cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394236037 0 0) 0x7f344c003520 con 0x7f3464162a00 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 msgr2=0x7f34640d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f34640d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 >> v2:172.21.15.169:3300/0 conn(0x7f3464162400 msgr2=0x7f34640cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 --2- 172.21.15.122:0/3372929892 >> v2:172.21.15.169:3300/0 conn(0x7f3464162400 0x7f34640cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640d2500 con 0x7f3464162a00 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f3454001670 con 0x7f3464162a00 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 >> v1:172.21.15.122:6789/0 conn(0x7f3464162a00 legacy=0x7f34640b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344c0039b0 0x7f344c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f344c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344c004d40 0x7f344c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640d2da0 con 0x7f344c004d40 2024-03-06T07:31:52.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c00b9c0 con 0x7f344c0039b0 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c00bb40 con 0x7f3464162020 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344c004d40 0x7f344c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3463fff640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344c0039b0 0x7f344c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f34637fe640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f344c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 -- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 msgr2=0x7f344c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f344c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 -- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344c0039b0 msgr2=0x7f344c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f344c0039b0 0x7f344c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f3468dd0640 1 -- 172.21.15.122:0/3372929892 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c003520 con 0x7f344c004d40 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.112+0000 7f34637fe640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3464162020 0x7f344c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.113+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344c004d40 0x7f344c0051c0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f345c009fc0 tx=0x7f344c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35f8a68dd74d499 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.113+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c039070 con 0x7f344c004d40 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.113+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c02cc60 con 0x7f344c004d40 2024-03-06T07:31:52.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.113+0000 7f3462ffd640 1 -- 172.21.15.122:0/3372929892 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c009c10 con 0x7f344c004d40 2024-03-06T07:31:52.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f346a833640 1 -- 172.21.15.122:0/3372929892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344c004d40 msgr2=0x7f344c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f346a833640 1 --2- 172.21.15.122:0/3372929892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f344c004d40 0x7f344c0051c0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f345c009fc0 tx=0x7f344c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:52.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f3468dd0640 1 -- 172.21.15.122:0/3372929892 reap_dead start 2024-03-06T07:31:52.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f346a833640 1 -- 172.21.15.122:0/3372929892 shutdown_connections 2024-03-06T07:31:52.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f346a833640 1 -- 172.21.15.122:0/3372929892 >> 172.21.15.122:0/3372929892 conn(0x7f34640d3d10 msgr2=0x7f34640d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f346a833640 1 -- 172.21.15.122:0/3372929892 shutdown_connections 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.114+0000 7f346a833640 1 -- 172.21.15.122:0/3372929892 wait complete. 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.115+0000 7f346a833640 1 Processor -- start 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.115+0000 7f346a833640 1 -- start start 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f346a833640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 0x7f34640b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f346a833640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34640b44a0 0x7f34640b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f346a833640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34640b8bf0 0x7f34640b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f346a833640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640cec00 con 0x7f3464155c60 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f346a833640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640d2ac0 con 0x7f34640b44a0 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f346a833640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640d4830 con 0x7f34640b8bf0 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 0x7f34640b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 0x7f34640b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40286/0 (socket says 172.21.15.122:40286) 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 -- 172.21.15.122:0/3337743063 learned_addr learned my addr 172.21.15.122:0/3337743063 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34640b8bf0 0x7f34640b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f34637fe640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34640b44a0 0x7f34640b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 -- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34640b8bf0 msgr2=0x7f34640b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34640b8bf0 0x7f34640b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 -- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34640b44a0 msgr2=0x7f34640b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34640b44a0 0x7f34640b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 -- 172.21.15.122:0/3337743063 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640b49b0 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f34637fe640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34640b44a0 0x7f34640b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3468dd0640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34640b8bf0 0x7f34640b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.116+0000 7f3463fff640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 0x7f34640b3f60 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f3458011940 tx=0x7f3458011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.117+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345800fa90 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.117+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345800fc30 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.117+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34640b4c90 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.117+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458014440 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.117+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34640b5250 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.118+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346404f330 con 0x7f3464155c60 2024-03-06T07:31:52.122 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.118+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3458012040 con 0x7f3464155c60 2024-03-06T07:31:52.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.119+0000 7f3460ff9640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f345006b500 0x7f345006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.119+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f345808a720 con 0x7f3464155c60 2024-03-06T07:31:52.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.119+0000 7f34637fe640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f345006b500 0x7f345006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.119+0000 7f34637fe640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f345006b500 0x7f345006d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3464161760 tx=0x7f3454009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.123+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3458019030 con 0x7f3464155c60 2024-03-06T07:31:52.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.337+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34640b8fd0 con 0x7f3464155c60 2024-03-06T07:31:52.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.338+0000 7f3460ff9640 1 -- 172.21.15.122:0/3337743063 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f3458059610 con 0x7f3464155c60 2024-03-06T07:31:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:52.340 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:52.340 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.339+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f345006b500 msgr2=0x7f345006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.339+0000 7f346a833640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f345006b500 0x7f345006d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3464161760 tx=0x7f3454009040 comp rx=0 tx=0).stop 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.339+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 msgr2=0x7f34640b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.339+0000 7f346a833640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 0x7f34640b3f60 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f3458011940 tx=0x7f3458011e00 comp rx=0 tx=0).stop 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 shutdown_connections 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f345006b500 0x7f345006d9c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f34640b8bf0 0x7f34640b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34640b44a0 0x7f34640b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 --2- 172.21.15.122:0/3337743063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3464155c60 0x7f34640b3f60 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 >> 172.21.15.122:0/3337743063 conn(0x7f34640d3d10 msgr2=0x7f34640d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 shutdown_connections 2024-03-06T07:31:52.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:52.340+0000 7f346a833640 1 -- 172.21.15.122:0/3337743063 wait complete. 2024-03-06T07:31:52.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:52.554 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:53.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:53.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.655+0000 7fd64484e640 1 Processor -- start 2024-03-06T07:31:53.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.655+0000 7fd64484e640 1 -- start start 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd64484e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6401424f0 0x7fd6400cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd64484e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6400cd7a0 0x7fd64015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd64484e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6400cc980 con 0x7fd64006d330 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd64484e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400cd220 con 0x7fd6400cd7a0 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd64484e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400ccc60 con 0x7fd6401424f0 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd63dd9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6400cd7a0 0x7fd64015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd63dd9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6400cd7a0 0x7fd64015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40290/0 (socket says 172.21.15.122:40290) 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd63dd9b640 1 -- 172.21.15.122:0/2215571481 learned_addr learned my addr 172.21.15.122:0/2215571481 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.656+0000 7fd63e59c640 1 --2- 172.21.15.122:0/2215571481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6401424f0 0x7fd6400cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.657+0000 7fd63dd9b640 1 -- 172.21.15.122:0/2215571481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6401424f0 msgr2=0x7fd6400cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.657+0000 7fd63dd9b640 1 --2- 172.21.15.122:0/2215571481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6401424f0 0x7fd6400cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.657+0000 7fd63dd9b640 1 -- 172.21.15.122:0/2215571481 >> v1:172.21.15.169:6789/0 conn(0x7fd64006d330 legacy=0x7fd640071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.657+0000 7fd63dd9b640 1 -- 172.21.15.122:0/2215571481 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400ccf40 con 0x7fd6400cd7a0 2024-03-06T07:31:53.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.657+0000 7fd63dd9b640 1 --2- 172.21.15.122:0/2215571481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400cd7a0 0x7fd64015cef0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fd630009870 tx=0x7fd63002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599e34d8dcc7b440 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.657+0000 7fd63d59a640 1 -- 172.21.15.122:0/2215571481 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630034070 con 0x7fd6400cd7a0 2024-03-06T07:31:53.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd63d59a640 1 -- 172.21.15.122:0/2215571481 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630030840 con 0x7fd6400cd7a0 2024-03-06T07:31:53.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd63d59a640 1 -- 172.21.15.122:0/2215571481 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630030c10 con 0x7fd6400cd7a0 2024-03-06T07:31:53.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 -- 172.21.15.122:0/2215571481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400cd7a0 msgr2=0x7fd64015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 --2- 172.21.15.122:0/2215571481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400cd7a0 0x7fd64015cef0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fd630009870 tx=0x7fd63002b980 comp rx=0 tx=0).stop 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 -- 172.21.15.122:0/2215571481 shutdown_connections 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 --2- 172.21.15.122:0/2215571481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400cd7a0 0x7fd64015cef0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 --2- 172.21.15.122:0/2215571481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6401424f0 0x7fd6400cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 -- 172.21.15.122:0/2215571481 >> 172.21.15.122:0/2215571481 conn(0x7fd6400cf8a0 msgr2=0x7fd6400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 -- 172.21.15.122:0/2215571481 shutdown_connections 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.658+0000 7fd64484e640 1 -- 172.21.15.122:0/2215571481 wait complete. 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.659+0000 7fd64484e640 1 Processor -- start 2024-03-06T07:31:53.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 -- start start 2024-03-06T07:31:53.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64006d330 0x7fd640158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6400cd7a0 0x7fd6400b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 0x7fd6400b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64006d940 con 0x7fd6401424f0 2024-03-06T07:31:53.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64006d7c0 con 0x7fd6400cd7a0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd64484e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64006dac0 con 0x7fd64006d330 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd63ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 0x7fd6400b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd63ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 0x7fd6400b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40300/0 (socket says 172.21.15.122:40300) 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd63ed9d640 1 -- 172.21.15.122:0/983321357 learned_addr learned my addr 172.21.15.122:0/983321357 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.660+0000 7fd63e59c640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64006d330 0x7fd640158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63dd9b640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6400cd7a0 0x7fd6400b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63ed9d640 1 -- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64006d330 msgr2=0x7fd640158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63ed9d640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64006d330 0x7fd640158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63ed9d640 1 -- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6400cd7a0 msgr2=0x7fd6400b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63ed9d640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6400cd7a0 0x7fd6400b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63ed9d640 1 -- 172.21.15.122:0/983321357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400b43e0 con 0x7fd6401424f0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63dd9b640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6400cd7a0 0x7fd6400b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd63ed9d640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 0x7fd6400b3a50 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c7a0 tx=0x7fd63800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638003b70 con 0x7fd6401424f0 2024-03-06T07:31:53.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638003d30 con 0x7fd6401424f0 2024-03-06T07:31:53.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6400ae310 con 0x7fd6401424f0 2024-03-06T07:31:53.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63800aa90 con 0x7fd6401424f0 2024-03-06T07:31:53.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.661+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6400ae8d0 con 0x7fd6401424f0 2024-03-06T07:31:53.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.662+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64004f330 con 0x7fd6401424f0 2024-03-06T07:31:53.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.663+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd63800ac30 con 0x7fd6401424f0 2024-03-06T07:31:53.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.663+0000 7fd62effd640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62006b4d0 0x7fd62006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.663+0000 7fd63e59c640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62006b4d0 0x7fd62006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.664+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd63808a0d0 con 0x7fd6401424f0 2024-03-06T07:31:53.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.664+0000 7fd63e59c640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62006b4d0 0x7fd62006d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd634009aa0 tx=0x7fd634009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.668+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd638058f30 con 0x7fd6401424f0 2024-03-06T07:31:53.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.882+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6400d0620 con 0x7fd6401424f0 2024-03-06T07:31:53.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.882+0000 7fd62effd640 1 -- 172.21.15.122:0/983321357 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fd638058830 con 0x7fd6401424f0 2024-03-06T07:31:53.884 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:53.885 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:53.885 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62006b4d0 msgr2=0x7fd62006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62006b4d0 0x7fd62006d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd634009aa0 tx=0x7fd634009190 comp rx=0 tx=0).stop 2024-03-06T07:31:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 msgr2=0x7fd6400b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 0x7fd6400b3a50 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c7a0 tx=0x7fd63800cc60 comp rx=0 tx=0).stop 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 shutdown_connections 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd62006b4d0 0x7fd62006d990 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6401424f0 0x7fd6400b3a50 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6400cd7a0 0x7fd6400b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 --2- 172.21.15.122:0/983321357 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd64006d330 0x7fd640158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 >> 172.21.15.122:0/983321357 conn(0x7fd6400cf8a0 msgr2=0x7fd64015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 shutdown_connections 2024-03-06T07:31:53.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:53.885+0000 7fd64484e640 1 -- 172.21.15.122:0/983321357 wait complete. 2024-03-06T07:31:54.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:54.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:55.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.199+0000 7f95d71d0640 1 Processor -- start 2024-03-06T07:31:55.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.199+0000 7f95d71d0640 1 -- start start 2024-03-06T07:31:55.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d71d0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95d00d2710 0x7f95d00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d71d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d00b3430 con 0x7f95d00d22b0 2024-03-06T07:31:55.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d71d0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d00b3710 con 0x7f95d015dfe0 2024-03-06T07:31:55.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d71d0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d00b31e0 con 0x7f95d00d2710 2024-03-06T07:31:55.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d576d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f95d00d2710 0x7f95d00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95cffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f95d00d22b0 0x7f95d00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59190/0 (socket says 172.21.15.122:59190) 2024-03-06T07:31:55.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95cffff640 1 -- 172.21.15.122:0/825407846 learned_addr learned my addr 172.21.15.122:0/825407846 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:55.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d576d640 1 -- 172.21.15.122:0/825407846 >> v1:172.21.15.169:6789/0 conn(0x7f95d015dfe0 legacy=0x7f95d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d576d640 1 -- 172.21.15.122:0/825407846 >> v1:172.21.15.122:6789/0 conn(0x7f95d00d22b0 legacy=0x7f95d00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95cf7fe640 1 -- 172.21.15.122:0/825407846 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302176635 0 0) 0x7f95d00b3430 con 0x7f95d00d22b0 2024-03-06T07:31:55.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.200+0000 7f95d576d640 1 -- 172.21.15.122:0/825407846 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00b39f0 con 0x7f95d00d2710 2024-03-06T07:31:55.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.201+0000 7f95d576d640 1 --2- 172.21.15.122:0/825407846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d2710 0x7f95d00b4900 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009dc0 tx=0x7f95c802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f281cfe486f6126 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.201+0000 7f95cf7fe640 1 -- 172.21.15.122:0/825407846 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8039070 con 0x7f95d00d2710 2024-03-06T07:31:55.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.201+0000 7f95cf7fe640 1 -- 172.21.15.122:0/825407846 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c802cc40 con 0x7f95d00d2710 2024-03-06T07:31:55.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.201+0000 7f95cf7fe640 1 -- 172.21.15.122:0/825407846 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8034440 con 0x7f95d00d2710 2024-03-06T07:31:55.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 -- 172.21.15.122:0/825407846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d2710 msgr2=0x7f95d00b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 --2- 172.21.15.122:0/825407846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d2710 0x7f95d00b4900 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009dc0 tx=0x7f95c802be90 comp rx=0 tx=0).stop 2024-03-06T07:31:55.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 -- 172.21.15.122:0/825407846 shutdown_connections 2024-03-06T07:31:55.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 --2- 172.21.15.122:0/825407846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d2710 0x7f95d00b4900 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 -- 172.21.15.122:0/825407846 >> 172.21.15.122:0/825407846 conn(0x7f95d00d3cd0 msgr2=0x7f95d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 -- 172.21.15.122:0/825407846 shutdown_connections 2024-03-06T07:31:55.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.202+0000 7f95d71d0640 1 -- 172.21.15.122:0/825407846 wait complete. 2024-03-06T07:31:55.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.203+0000 7f95d71d0640 1 Processor -- start 2024-03-06T07:31:55.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.203+0000 7f95d71d0640 1 -- start start 2024-03-06T07:31:55.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d71d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 0x7f95d00b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d71d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95d00d2710 0x7f95d01f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d71d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95d015dfe0 0x7f95d0159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d71d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00cfa00 con 0x7f95d00d22b0 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d71d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00b3d60 con 0x7f95d00d2710 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d71d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00d1c70 con 0x7f95d015dfe0 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 0x7f95d00b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 0x7f95d00b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36372/0 (socket says 172.21.15.122:36372) 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 -- 172.21.15.122:0/3524260668 learned_addr learned my addr 172.21.15.122:0/3524260668 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95cffff640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95d00d2710 0x7f95d01f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d576d640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95d015dfe0 0x7f95d0159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 -- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95d015dfe0 msgr2=0x7f95d0159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95d015dfe0 0x7f95d0159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 -- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95d00d2710 msgr2=0x7f95d01f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95d00d2710 0x7f95d01f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 -- 172.21.15.122:0/3524260668 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00b68d0 con 0x7f95d00d22b0 2024-03-06T07:31:55.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d576d640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95d015dfe0 0x7f95d0159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95d4f6c640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 0x7f95d00b8db0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f95c400b3f0 tx=0x7f95c400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4015020 con 0x7f95d00d22b0 2024-03-06T07:31:55.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.204+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c4011830 con 0x7f95d00d22b0 2024-03-06T07:31:55.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.205+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d01f7a90 con 0x7f95d00d22b0 2024-03-06T07:31:55.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.205+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4011c20 con 0x7f95d00d22b0 2024-03-06T07:31:55.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.205+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d00b6e50 con 0x7f95d00d22b0 2024-03-06T07:31:55.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.206+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d00b7050 con 0x7f95d00d22b0 2024-03-06T07:31:55.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.206+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95c4018020 con 0x7f95d00d22b0 2024-03-06T07:31:55.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.207+0000 7f95cd7fa640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95b406b4d0 0x7f95b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.207+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f95c4089e40 con 0x7f95d00d22b0 2024-03-06T07:31:55.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.207+0000 7f95cffff640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95b406b4d0 0x7f95b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.207+0000 7f95cffff640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95b406b4d0 0x7f95b406d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f95d015c9e0 tx=0x7f95bc007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.211+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95d00b7050 con 0x7f95d00d22b0 2024-03-06T07:31:55.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.420+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95d00cec80 con 0x7f95d00d22b0 2024-03-06T07:31:55.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.421+0000 7f95cd7fa640 1 -- 172.21.15.122:0/3524260668 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f95c401f030 con 0x7f95d00d22b0 2024-03-06T07:31:55.423 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:55.423 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:55.423 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95b406b4d0 msgr2=0x7f95b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95b406b4d0 0x7f95b406d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f95d015c9e0 tx=0x7f95bc007b90 comp rx=0 tx=0).stop 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 msgr2=0x7f95d00b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 0x7f95d00b8db0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f95c400b3f0 tx=0x7f95c400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 shutdown_connections 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f95b406b4d0 0x7f95b406d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f95d015dfe0 0x7f95d0159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95d00d2710 0x7f95d01f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 --2- 172.21.15.122:0/3524260668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95d00d22b0 0x7f95d00b8db0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 >> 172.21.15.122:0/3524260668 conn(0x7f95d00d3cd0 msgr2=0x7f95d00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 shutdown_connections 2024-03-06T07:31:55.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:55.423+0000 7f95d71d0640 1 -- 172.21.15.122:0/3524260668 wait complete. 2024-03-06T07:31:55.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:55.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:55.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:55.637 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:56.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:56.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.737+0000 7ffb3b4a4640 1 Processor -- start 2024-03-06T07:31:56.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.737+0000 7ffb3b4a4640 1 -- start start 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb3b4a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb340d22b0 0x7ffb340d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb3b4a4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ffb340b41f0 0x7ffb340b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb3b4a4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb340b3430 con 0x7ffb3415dfe0 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb3b4a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340d2c50 con 0x7ffb340d22b0 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb3b4a4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340b31e0 con 0x7ffb340b41f0 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb38a3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb340d22b0 0x7ffb340d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb38a3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffb340d22b0 0x7ffb340d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36386/0 (socket says 172.21.15.122:36386) 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb38a3f640 1 -- 172.21.15.122:0/638419379 learned_addr learned my addr 172.21.15.122:0/638419379 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.738+0000 7ffb39a41640 1 --2- 172.21.15.122:0/638419379 >> v2:172.21.15.169:3300/0 conn(0x7ffb340b41f0 0x7ffb340b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb38a3f640 1 -- 172.21.15.122:0/638419379 >> v2:172.21.15.169:3300/0 conn(0x7ffb340b41f0 msgr2=0x7ffb340b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb38a3f640 1 --2- 172.21.15.122:0/638419379 >> v2:172.21.15.169:3300/0 conn(0x7ffb340b41f0 0x7ffb340b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb38a3f640 1 -- 172.21.15.122:0/638419379 >> v1:172.21.15.169:6789/0 conn(0x7ffb3415dfe0 legacy=0x7ffb340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb38a3f640 1 -- 172.21.15.122:0/638419379 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340b3710 con 0x7ffb340d22b0 2024-03-06T07:31:56.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb39a41640 1 --2- 172.21.15.122:0/638419379 >> v2:172.21.15.169:3300/0 conn(0x7ffb340b41f0 0x7ffb340b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:56.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb38a3f640 1 --2- 172.21.15.122:0/638419379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb340d22b0 0x7ffb340d2710 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009870 tx=0x7ffb3002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247cd012ea8e4592 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb2bfff640 1 -- 172.21.15.122:0/638419379 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30034070 con 0x7ffb340d22b0 2024-03-06T07:31:56.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.739+0000 7ffb2bfff640 1 -- 172.21.15.122:0/638419379 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30030860 con 0x7ffb340d22b0 2024-03-06T07:31:56.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb2bfff640 1 -- 172.21.15.122:0/638419379 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30030c30 con 0x7ffb340d22b0 2024-03-06T07:31:56.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/638419379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb340d22b0 msgr2=0x7ffb340d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/638419379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb340d22b0 0x7ffb340d2710 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009870 tx=0x7ffb3002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/638419379 shutdown_connections 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/638419379 >> v2:172.21.15.169:3300/0 conn(0x7ffb340b41f0 0x7ffb340b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/638419379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb340d22b0 0x7ffb340d2710 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/638419379 >> 172.21.15.122:0/638419379 conn(0x7ffb340d3cd0 msgr2=0x7ffb340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/638419379 shutdown_connections 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.740+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/638419379 wait complete. 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.741+0000 7ffb3b4a4640 1 Processor -- start 2024-03-06T07:31:56.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.741+0000 7ffb3b4a4640 1 -- start start 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb3b4a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb340b41f0 0x7ffb3414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb3b4a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb340d22b0 0x7ffb340cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb3b4a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 0x7ffb34160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb3b4a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340cfa00 con 0x7ffb3415dfe0 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb3b4a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340b3d60 con 0x7ffb340d22b0 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb3b4a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d1c70 con 0x7ffb340b41f0 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 0x7ffb34160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 0x7ffb34160400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36390/0 (socket says 172.21.15.122:36390) 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 -- 172.21.15.122:0/115185421 learned_addr learned my addr 172.21.15.122:0/115185421 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb38a3f640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb340d22b0 0x7ffb340cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39240640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb340b41f0 0x7ffb3414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 -- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb340b41f0 msgr2=0x7ffb3414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb340b41f0 0x7ffb3414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 -- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb340d22b0 msgr2=0x7ffb340cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb340d22b0 0x7ffb340cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39a41640 1 -- 172.21.15.122:0/115185421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb341f4a20 con 0x7ffb3415dfe0 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb38a3f640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb340d22b0 0x7ffb340cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.742+0000 7ffb39240640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb340b41f0 0x7ffb3414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.743+0000 7ffb39a41640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 0x7ffb34160400 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c007ef0 tx=0x7ffb2c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.743+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c01a070 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.743+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c003b90 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.743+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb340b8680 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.743+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c00de20 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.743+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3414f740 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.744+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3404f330 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.744+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb2c01d070 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.745+0000 7ffb29ffb640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb1806b4d0 0x7ffb1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.745+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffb2c08a3d0 con 0x7ffb3415dfe0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.745+0000 7ffb39240640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb1806b4d0 0x7ffb1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.745+0000 7ffb39240640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb1806b4d0 0x7ffb1806d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ffb240015e0 tx=0x7ffb24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:56.749+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb2c010070 con 0x7ffb3415dfe0 2024-03-06T07:31:57.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.048+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb340b0af0 con 0x7ffb3415dfe0 2024-03-06T07:31:57.050 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.048+0000 7ffb29ffb640 1 -- 172.21.15.122:0/115185421 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7ffb2c058be0 con 0x7ffb3415dfe0 2024-03-06T07:31:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:57.051 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:57.051 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:57.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb1806b4d0 msgr2=0x7ffb1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb1806b4d0 0x7ffb1806d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ffb240015e0 tx=0x7ffb24009040 comp rx=0 tx=0).stop 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 msgr2=0x7ffb34160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 0x7ffb34160400 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c007ef0 tx=0x7ffb2c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 shutdown_connections 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ffb1806b4d0 0x7ffb1806d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb3415dfe0 0x7ffb34160400 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb340d22b0 0x7ffb340cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 --2- 172.21.15.122:0/115185421 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ffb340b41f0 0x7ffb3414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 >> 172.21.15.122:0/115185421 conn(0x7ffb340d3cd0 msgr2=0x7ffb340b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 shutdown_connections 2024-03-06T07:31:57.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:57.050+0000 7ffb3b4a4640 1 -- 172.21.15.122:0/115185421 wait complete. 2024-03-06T07:31:57.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:57.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:58.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 Processor -- start 2024-03-06T07:31:58.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 -- start start 2024-03-06T07:31:58.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f55980cf830 0x7f55980d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55980b3430 con 0x7f55980d27f0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55980d2c50 con 0x7f55980cf830 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f559d26d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55980b31e0 con 0x7f559815dfe0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.366+0000 7f55977fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f55980d27f0 0x7f55980d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59204/0 (socket says 172.21.15.122:59204) 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55977fe640 1 -- 172.21.15.122:0/1930412402 learned_addr learned my addr 172.21.15.122:0/1930412402 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f5596ffd640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f558ffff640 1 --2- 172.21.15.122:0/1930412402 >> v2:172.21.15.169:3300/0 conn(0x7f55980cf830 0x7f55980d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337762753 0 0) 0x7f55980b3430 con 0x7f55980d27f0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5584003520 con 0x7f55980d27f0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437953202 0 0) 0x7f5584003520 con 0x7f55980d27f0 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 msgr2=0x7f55980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55980cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 >> v2:172.21.15.169:3300/0 conn(0x7f55980cf830 msgr2=0x7f55980d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 --2- 172.21.15.122:0/1930412402 >> v2:172.21.15.169:3300/0 conn(0x7f55980cf830 0x7f55980d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55980b3710 con 0x7f55980d27f0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f55900019f0 con 0x7f55980d27f0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f5596ffd640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55980cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 >> v1:172.21.15.122:6789/0 conn(0x7f55980d27f0 legacy=0x7f55980d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5584003ef0 0x7f5584004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584004d40 0x7f55840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55980b31e0 con 0x7f5584003ef0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558400b9c0 con 0x7f5584004d40 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558400bb40 con 0x7f559815dfe0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f558ffff640 1 --2- 172.21.15.122:0/1930412402 >> v2:172.21.15.169:3300/0 conn(0x7f55980cf830 0x7f55980d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f558ffff640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5584003ef0 0x7f5584004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f5596ffd640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.367+0000 7f55977fe640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584004d40 0x7f55840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f558ffff640 1 -- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 msgr2=0x7f55840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f558ffff640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559815dfe0 0x7f55840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f558ffff640 1 -- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584004d40 msgr2=0x7f55840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f558ffff640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584004d40 0x7f55840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f558ffff640 1 -- 172.21.15.122:0/1930412402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5584003520 con 0x7f5584003ef0 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f558ffff640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5584003ef0 0x7f5584004800 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f5580009c40 tx=0x7f5584005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fe55261f1a1b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580009840 con 0x7f5584003ef0 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5580004020 con 0x7f5584003ef0 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.368+0000 7f55967fc640 1 -- 172.21.15.122:0/1930412402 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580004450 con 0x7f5584003ef0 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f559d26d640 1 -- 172.21.15.122:0/1930412402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5584003ef0 msgr2=0x7f5584004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f559d26d640 1 --2- 172.21.15.122:0/1930412402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5584003ef0 0x7f5584004800 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f5580009c40 tx=0x7f5584005830 comp rx=0 tx=0).stop 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f55977fe640 1 -- 172.21.15.122:0/1930412402 reap_dead start 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f559d26d640 1 -- 172.21.15.122:0/1930412402 shutdown_connections 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f559d26d640 1 -- 172.21.15.122:0/1930412402 >> 172.21.15.122:0/1930412402 conn(0x7f55980d3cd0 msgr2=0x7f55980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f559d26d640 1 -- 172.21.15.122:0/1930412402 shutdown_connections 2024-03-06T07:31:58.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.369+0000 7f559d26d640 1 -- 172.21.15.122:0/1930412402 wait complete. 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.370+0000 7f559d26d640 1 Processor -- start 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.370+0000 7f559d26d640 1 -- start start 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f559d26d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584003ef0 0x7f559815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f559d26d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559806c410 0x7f559806c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f559d26d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 0x7f55980b74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f559d26d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55980d1c70 con 0x7f55980b7030 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f559d26d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55980b3d60 con 0x7f5584003ef0 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f559d26d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55980d1df0 con 0x7f559806c410 2024-03-06T07:31:58.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 0x7f55980b74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 0x7f55980b74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36410/0 (socket says 172.21.15.122:36410) 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 -- 172.21.15.122:0/3015117496 learned_addr learned my addr 172.21.15.122:0/3015117496 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f558ffff640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559806c410 0x7f559806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f5596ffd640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584003ef0 0x7f559815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 -- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559806c410 msgr2=0x7f559806c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559806c410 0x7f559806c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 -- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584003ef0 msgr2=0x7f559815b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584003ef0 0x7f559815b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 -- 172.21.15.122:0/3015117496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559806cf80 con 0x7f55980b7030 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.371+0000 7f55977fe640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 0x7f55980b74c0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f55980b3360 tx=0x7f5590007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.372+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559000abe0 con 0x7f55980b7030 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.372+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590019070 con 0x7f55980b7030 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.372+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55900133f0 con 0x7f55980b7030 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.372+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559806d260 con 0x7f55980b7030 2024-03-06T07:31:58.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.372+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55981f4f60 con 0x7f55980b7030 2024-03-06T07:31:58.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.373+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559804f330 con 0x7f55980b7030 2024-03-06T07:31:58.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.373+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5590013590 con 0x7f55980b7030 2024-03-06T07:31:58.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.373+0000 7f558f7fe640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f557806b4d0 0x7f557806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.374+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f559008a850 con 0x7f55980b7030 2024-03-06T07:31:58.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.374+0000 7f5596ffd640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f557806b4d0 0x7f557806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.374+0000 7f5596ffd640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f557806b4d0 0x7f557806d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5588009560 tx=0x7f5588008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.378+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55900596b0 con 0x7f55980b7030 2024-03-06T07:31:58.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.598+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55980b1400 con 0x7f55980b7030 2024-03-06T07:31:58.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.598+0000 7f558f7fe640 1 -- 172.21.15.122:0/3015117496 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f5590058fb0 con 0x7f55980b7030 2024-03-06T07:31:58.600 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:31:58.601 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:31:58.601 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:31:58.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f557806b4d0 msgr2=0x7f557806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f557806b4d0 0x7f557806d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5588009560 tx=0x7f5588008040 comp rx=0 tx=0).stop 2024-03-06T07:31:58.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 msgr2=0x7f55980b74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 0x7f55980b74c0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f55980b3360 tx=0x7f5590007b00 comp rx=0 tx=0).stop 2024-03-06T07:31:58.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 shutdown_connections 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f557806b4d0 0x7f557806d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55980b7030 0x7f55980b74c0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f559806c410 0x7f559806c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 --2- 172.21.15.122:0/3015117496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5584003ef0 0x7f559815b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 >> 172.21.15.122:0/3015117496 conn(0x7f55980d3cd0 msgr2=0x7f55980cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 shutdown_connections 2024-03-06T07:31:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:58.601+0000 7f559d26d640 1 -- 172.21.15.122:0/3015117496 wait complete. 2024-03-06T07:31:58.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:31:58.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:31:59.817 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:59.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.917+0000 7f9e65325640 1 Processor -- start 2024-03-06T07:31:59.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.917+0000 7f9e65325640 1 -- start start 2024-03-06T07:31:59.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e65325640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e6015dfe0 0x7f9e600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e65325640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9e600cf830 0x7f9e600d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e65325640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e600d27f0 0x7f9e600d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e65325640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e600d2c50 con 0x7f9e600d27f0 2024-03-06T07:31:59.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e65325640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e600b31e0 con 0x7f9e600cf830 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e65325640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600b3d60 con 0x7f9e6015dfe0 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e600d27f0 0x7f9e600d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e600d27f0 0x7f9e600d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36414/0 (socket says 172.21.15.122:36414) 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/3477774441 learned_addr learned my addr 172.21.15.122:0/3477774441 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e57fff640 1 --2- 172.21.15.122:0/3477774441 >> v2:172.21.15.169:3300/0 conn(0x7f9e600cf830 0x7f9e600d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5effd640 1 --2- 172.21.15.122:0/3477774441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e6015dfe0 0x7f9e600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/3477774441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e6015dfe0 msgr2=0x7f9e600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 --2- 172.21.15.122:0/3477774441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e6015dfe0 0x7f9e600cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/3477774441 >> v2:172.21.15.169:3300/0 conn(0x7f9e600cf830 msgr2=0x7f9e600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 --2- 172.21.15.122:0/3477774441 >> v2:172.21.15.169:3300/0 conn(0x7f9e600cf830 0x7f9e600d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.918+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/3477774441 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600b3430 con 0x7f9e600d27f0 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.919+0000 7f9e5f7fe640 1 --2- 172.21.15.122:0/3477774441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e600d27f0 0x7f9e600d0300 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f9e6008cba0 tx=0x7f9e5802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e83cb17a34290e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.919+0000 7f9e5e7fc640 1 -- 172.21.15.122:0/3477774441 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5802ca70 con 0x7f9e600d27f0 2024-03-06T07:31:59.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.919+0000 7f9e5e7fc640 1 -- 172.21.15.122:0/3477774441 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5802cc10 con 0x7f9e600d27f0 2024-03-06T07:31:59.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.919+0000 7f9e5e7fc640 1 -- 172.21.15.122:0/3477774441 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e580343f0 con 0x7f9e600d27f0 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 -- 172.21.15.122:0/3477774441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e600d27f0 msgr2=0x7f9e600d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 --2- 172.21.15.122:0/3477774441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e600d27f0 0x7f9e600d0300 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f9e6008cba0 tx=0x7f9e5802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 -- 172.21.15.122:0/3477774441 shutdown_connections 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 --2- 172.21.15.122:0/3477774441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e600d27f0 0x7f9e600d0300 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 --2- 172.21.15.122:0/3477774441 >> v2:172.21.15.169:3300/0 conn(0x7f9e600cf830 0x7f9e600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 --2- 172.21.15.122:0/3477774441 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e6015dfe0 0x7f9e600cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 -- 172.21.15.122:0/3477774441 >> 172.21.15.122:0/3477774441 conn(0x7f9e600d3cd0 msgr2=0x7f9e600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 -- 172.21.15.122:0/3477774441 shutdown_connections 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.920+0000 7f9e65325640 1 -- 172.21.15.122:0/3477774441 wait complete. 2024-03-06T07:31:59.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.921+0000 7f9e65325640 1 Processor -- start 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.921+0000 7f9e65325640 1 -- start start 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.921+0000 7f9e65325640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e600cf830 0x7f9e60161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e600d27f0 0x7f9e600b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 0x7f9e60149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600d1c70 con 0x7f9e6015dfe0 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600b39f0 con 0x7f9e600d27f0 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600d1df0 con 0x7f9e600cf830 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 0x7f9e60149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 0x7f9e60149f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36426/0 (socket says 172.21.15.122:36426) 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/2080082722 learned_addr learned my addr 172.21.15.122:0/2080082722 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5effd640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e600cf830 0x7f9e60161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e57fff640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e600d27f0 0x7f9e600b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e600cf830 msgr2=0x7f9e60161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e600cf830 0x7f9e60161190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e600d27f0 msgr2=0x7f9e600b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e600d27f0 0x7f9e600b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 -- 172.21.15.122:0/2080082722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600b8ee0 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e5f7fe640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 0x7f9e60149f30 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f9e58036040 tx=0x7f9e58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58043020 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5803e450 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e600b91c0 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58009510 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.922+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e600b9780 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.923+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6004f330 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.924+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e58046020 con 0x7f9e6015dfe0 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.924+0000 7f9e577fe640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e4006b4d0 0x7f9e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.924+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9e580ac920 con 0x7f9e6015dfe0 2024-03-06T07:31:59.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.924+0000 7f9e5effd640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e4006b4d0 0x7f9e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.925+0000 7f9e5effd640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e4006b4d0 0x7f9e4006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9e500079a0 tx=0x7f9e50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:31:59.929+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e5804d030 con 0x7f9e6015dfe0 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.142+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e6015d4c0 con 0x7f9e6015dfe0 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.143+0000 7f9e577fe640 1 -- 172.21.15.122:0/2080082722 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f9e5807b130 con 0x7f9e6015dfe0 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e4006b4d0 msgr2=0x7f9e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e4006b4d0 0x7f9e4006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9e500079a0 tx=0x7f9e50008040 comp rx=0 tx=0).stop 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 msgr2=0x7f9e60149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 0x7f9e60149f30 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f9e58036040 tx=0x7f9e58004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 shutdown_connections 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e4006b4d0 0x7f9e4006d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6015dfe0 0x7f9e60149f30 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e600d27f0 0x7f9e600b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 --2- 172.21.15.122:0/2080082722 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e600cf830 0x7f9e60161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 >> 172.21.15.122:0/2080082722 conn(0x7f9e600d3cd0 msgr2=0x7f9e6006cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 shutdown_connections 2024-03-06T07:32:00.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:00.145+0000 7f9e65325640 1 -- 172.21.15.122:0/2080082722 wait complete. 2024-03-06T07:32:00.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:00.359 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:01.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:01.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.472+0000 7f45001f6640 1 Processor -- start 2024-03-06T07:32:01.475 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f45001f6640 1 -- start start 2024-03-06T07:32:01.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f45001f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f44f80d22b0 0x7f44f80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f45001f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80b41f0 0x7f44f80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f45001f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f80b3430 con 0x7f44f815dfe0 2024-03-06T07:32:01.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f45001f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f80d2c50 con 0x7f44f80d22b0 2024-03-06T07:32:01.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f45001f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80b31e0 con 0x7f44f80b41f0 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f44fd791640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f44f80d22b0 0x7f44f80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f44fdf92640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f44f815dfe0 0x7f44f80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59206/0 (socket says 172.21.15.122:59206) 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f44fd791640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f44f80d22b0 0x7f44f80d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36430/0 (socket says 172.21.15.122:36430) 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.473+0000 7f44fd791640 1 -- 172.21.15.122:0/944631639 learned_addr learned my addr 172.21.15.122:0/944631639 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fe793640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80b41f0 0x7f44f80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fd791640 1 -- 172.21.15.122:0/944631639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80b41f0 msgr2=0x7f44f80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fd791640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80b41f0 0x7f44f80b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fd791640 1 -- 172.21.15.122:0/944631639 >> v1:172.21.15.122:6789/0 conn(0x7f44f815dfe0 legacy=0x7f44f80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fcf90640 1 -- 172.21.15.122:0/944631639 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331911277 0 0) 0x7f44f80b3430 con 0x7f44f815dfe0 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fd791640 1 -- 172.21.15.122:0/944631639 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f80b3710 con 0x7f44f80d22b0 2024-03-06T07:32:01.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.474+0000 7f44fe793640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80b41f0 0x7f44f80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:01.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.475+0000 7f44fd791640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80d22b0 0x7f44f80d2710 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009870 tx=0x7f44e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79138af5ffefab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.475+0000 7f44fcf90640 1 -- 172.21.15.122:0/944631639 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8034070 con 0x7f44f80d22b0 2024-03-06T07:32:01.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.475+0000 7f44fcf90640 1 -- 172.21.15.122:0/944631639 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8030840 con 0x7f44f80d22b0 2024-03-06T07:32:01.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.475+0000 7f44fcf90640 1 -- 172.21.15.122:0/944631639 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8030c10 con 0x7f44f80d22b0 2024-03-06T07:32:01.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 -- 172.21.15.122:0/944631639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80d22b0 msgr2=0x7f44f80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80d22b0 0x7f44f80d2710 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009870 tx=0x7f44e802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 -- 172.21.15.122:0/944631639 shutdown_connections 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80b41f0 0x7f44f80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 --2- 172.21.15.122:0/944631639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80d22b0 0x7f44f80d2710 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 -- 172.21.15.122:0/944631639 >> 172.21.15.122:0/944631639 conn(0x7f44f80d3cd0 msgr2=0x7f44f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 -- 172.21.15.122:0/944631639 shutdown_connections 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.476+0000 7f45001f6640 1 -- 172.21.15.122:0/944631639 wait complete. 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.477+0000 7f45001f6640 1 Processor -- start 2024-03-06T07:32:01.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.478+0000 7f45001f6640 1 -- start start 2024-03-06T07:32:01.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f45001f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 0x7f44f80cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f45001f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80d22b0 0x7f44f8160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f45001f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f815dfe0 0x7f44f8159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.482 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f45001f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80cfa00 con 0x7f44f80b41f0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f45001f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80b3d60 con 0x7f44f815dfe0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f45001f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80d1c70 con 0x7f44f80d22b0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 0x7f44f80cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 0x7f44f80cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36440/0 (socket says 172.21.15.122:36440) 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 -- 172.21.15.122:0/3471675875 learned_addr learned my addr 172.21.15.122:0/3471675875 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fd791640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80d22b0 0x7f44f8160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 -- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80d22b0 msgr2=0x7f44f8160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80d22b0 0x7f44f8160640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 -- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f815dfe0 msgr2=0x7f44f8159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fe793640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f815dfe0 0x7f44f8159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f815dfe0 0x7f44f8159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 -- 172.21.15.122:0/3471675875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f80cc9e0 con 0x7f44f80b41f0 2024-03-06T07:32:01.483 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fe793640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f815dfe0 0x7f44f8159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44fdf92640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 0x7f44f80cc180 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c430 tx=0x7f44f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.479+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f40074e0 con 0x7f44f80b41f0 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.480+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f40047e0 con 0x7f44f80b41f0 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.480+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f80ccf70 con 0x7f44f80b41f0 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.480+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f4004bd0 con 0x7f44f80b41f0 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.480+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f80cd530 con 0x7f44f80b41f0 2024-03-06T07:32:01.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.480+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f804f330 con 0x7f44f80b41f0 2024-03-06T07:32:01.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.481+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44f4011070 con 0x7f44f80b41f0 2024-03-06T07:32:01.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.482+0000 7f44ee7fc640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f44dc06b4d0 0x7f44dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.482+0000 7f44fd791640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f44dc06b4d0 0x7f44dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.482+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f44f4089e40 con 0x7f44f80b41f0 2024-03-06T07:32:01.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.482+0000 7f44fd791640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f44dc06b4d0 0x7f44dc06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f44e8036000 tx=0x7f44e803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.486+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44f4058ca0 con 0x7f44f80b41f0 2024-03-06T07:32:01.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.697+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44f806d150 con 0x7f44f80b41f0 2024-03-06T07:32:01.699 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.697+0000 7f44ee7fc640 1 -- 172.21.15.122:0/3471675875 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f44f40585a0 con 0x7f44f80b41f0 2024-03-06T07:32:01.699 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:01.700 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:01.700 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f44dc06b4d0 msgr2=0x7f44dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f44dc06b4d0 0x7f44dc06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f44e8036000 tx=0x7f44e803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 msgr2=0x7f44f80cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 0x7f44f80cc180 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c430 tx=0x7f44f400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 shutdown_connections 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f44dc06b4d0 0x7f44dc06d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f815dfe0 0x7f44f8159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f44f80d22b0 0x7f44f8160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 --2- 172.21.15.122:0/3471675875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f80b41f0 0x7f44f80cc180 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 >> 172.21.15.122:0/3471675875 conn(0x7f44f80d3cd0 msgr2=0x7f44f806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 shutdown_connections 2024-03-06T07:32:01.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:01.699+0000 7f45001f6640 1 -- 172.21.15.122:0/3471675875 wait complete. 2024-03-06T07:32:01.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:01.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:02.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.013+0000 7f4a164b5640 1 Processor -- start 2024-03-06T07:32:03.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.013+0000 7f4a164b5640 1 -- start start 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a164b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 0x7f4a100cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a164b5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a100b3430 con 0x7f4a100cf830 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a164b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a100b3710 con 0x7f4a100d29c0 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a164b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100b31e0 con 0x7f4a1015dfe0 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a0f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4a100cf830 0x7f4a100d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59208/0 (socket says 172.21.15.122:59208) 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/4276886760 learned_addr learned my addr 172.21.15.122:0/4276886760 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a0ffff640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 0x7f4a100cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168065947 0 0) 0x7f4a100b3430 con 0x7f4a100cf830 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.014+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f8003520 con 0x7f4a100cf830 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938772760 0 0) 0x7f49f8003520 con 0x7f4a100cf830 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 msgr2=0x7f4a100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 0x7f4a100cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 >> v1:172.21.15.169:6789/0 conn(0x7f4a100d29c0 legacy=0x7f4a100b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100b39f0 con 0x7f4a100cf830 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4a04001670 con 0x7f4a100cf830 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 >> v1:172.21.15.122:6789/0 conn(0x7f4a100cf830 legacy=0x7f4a100d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49f80039e0 0x7f49f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8004800 0x7f49f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 0x7f49f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100b31e0 con 0x7f49f8004800 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f800ba70 con 0x7f49f80039e0 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f800bc70 con 0x7f4a1015dfe0 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8004800 0x7f49f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0ffff640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49f80039e0 0x7f49f8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a14a52640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 0x7f49f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 msgr2=0x7f49f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4a1015dfe0 0x7f49f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49f80039e0 msgr2=0x7f49f8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49f80039e0 0x7f49f8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.015+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/4276886760 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f8003520 con 0x7f49f8004800 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.016+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8004800 0x7f49f8004c80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f4a100cfc70 tx=0x7f49f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acdbd926f7ae78c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.016+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04007d10 con 0x7f49f8004800 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.016+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04030070 con 0x7f49f8004800 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.016+0000 7f4a0effd640 1 -- 172.21.15.122:0/4276886760 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04005d30 con 0x7f49f8004800 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.016+0000 7f4a164b5640 1 -- 172.21.15.122:0/4276886760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8004800 msgr2=0x7f49f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.016+0000 7f4a164b5640 1 --2- 172.21.15.122:0/4276886760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8004800 0x7f49f8004c80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f4a100cfc70 tx=0x7f49f8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.017+0000 7f4a14a52640 1 -- 172.21.15.122:0/4276886760 reap_dead start 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.017+0000 7f4a164b5640 1 -- 172.21.15.122:0/4276886760 shutdown_connections 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.017+0000 7f4a164b5640 1 -- 172.21.15.122:0/4276886760 >> 172.21.15.122:0/4276886760 conn(0x7f4a100d3cd0 msgr2=0x7f4a100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.017+0000 7f4a164b5640 1 -- 172.21.15.122:0/4276886760 shutdown_connections 2024-03-06T07:32:03.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.017+0000 7f4a164b5640 1 -- 172.21.15.122:0/4276886760 wait complete. 2024-03-06T07:32:03.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.018+0000 7f4a164b5640 1 Processor -- start 2024-03-06T07:32:03.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.019+0000 7f4a164b5640 1 -- start start 2024-03-06T07:32:03.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a164b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49f8004800 0x7f4a1015bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a164b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 0x7f4a100cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a164b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a100a11b0 0x7f4a100a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a164b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1015e3c0 con 0x7f4a1015c430 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a164b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100b3d60 con 0x7f4a100a11b0 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a164b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1015e540 con 0x7f49f8004800 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 0x7f4a100cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 0x7f4a100cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:36456/0 (socket says 172.21.15.122:36456) 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/3548677305 learned_addr learned my addr 172.21.15.122:0/3548677305 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/3548677305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49f8004800 msgr2=0x7f4a1015bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49f8004800 0x7f4a1015bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/3548677305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a100a11b0 msgr2=0x7f4a100a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a100a11b0 0x7f4a100a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 -- 172.21.15.122:0/3548677305 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100cb630 con 0x7f4a1015c430 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0f7fe640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 0x7f4a100cada0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c300 tx=0x7f4a100cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04007d10 con 0x7f4a1015c430 2024-03-06T07:32:03.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04004c10 con 0x7f4a1015c430 2024-03-06T07:32:03.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.020+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04007070 con 0x7f4a1015c430 2024-03-06T07:32:03.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.021+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a100cb910 con 0x7f4a1015c430 2024-03-06T07:32:03.025 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.021+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a100cbed0 con 0x7f4a1015c430 2024-03-06T07:32:03.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.022+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1004f330 con 0x7f4a1015c430 2024-03-06T07:32:03.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.022+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a04007210 con 0x7f4a1015c430 2024-03-06T07:32:03.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.022+0000 7f4a0cff9640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49fc06b4d0 0x7f49fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.022+0000 7f4a0ffff640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49fc06b4d0 0x7f49fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.023+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4a040ab900 con 0x7f4a1015c430 2024-03-06T07:32:03.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.023+0000 7f4a0ffff640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49fc06b4d0 0x7f49fc06d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009ab0 tx=0x7f4a000095d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.027+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a0407a760 con 0x7f4a1015c430 2024-03-06T07:32:03.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.434+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a1004f330 con 0x7f4a1015c430 2024-03-06T07:32:03.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.435+0000 7f4a0cff9640 1 -- 172.21.15.122:0/3548677305 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f4a0407a060 con 0x7f4a1015c430 2024-03-06T07:32:03.437 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:03.437 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:03.437 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49fc06b4d0 msgr2=0x7f49fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49fc06b4d0 0x7f49fc06d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009ab0 tx=0x7f4a000095d0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 msgr2=0x7f4a100cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 0x7f4a100cada0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c300 tx=0x7f4a100cfd40 comp rx=0 tx=0).stop 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 shutdown_connections 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f49fc06b4d0 0x7f49fc06d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a100a11b0 0x7f4a100a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a1015c430 0x7f4a100cada0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 --2- 172.21.15.122:0/3548677305 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49f8004800 0x7f4a1015bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 >> 172.21.15.122:0/3548677305 conn(0x7f4a100d3cd0 msgr2=0x7f4a100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 shutdown_connections 2024-03-06T07:32:03.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:03.437+0000 7f4a164b5640 1 -- 172.21.15.122:0/3548677305 wait complete. 2024-03-06T07:32:03.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:03.651 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:04.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.752+0000 7f4719a64640 1 Processor -- start 2024-03-06T07:32:04.753 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.752+0000 7f4719a64640 1 -- start start 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f4719a64640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f471415dfe0 0x7f47140cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f4719a64640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47140b3430 con 0x7f47140d29c0 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f4719a64640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47140b3710 con 0x7f47140cf830 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f4719a64640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47140b31e0 con 0x7f471415dfe0 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47137fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f471415dfe0 0x7f47140cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47137fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f471415dfe0 0x7f47140cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47530/0 (socket says 172.21.15.122:47530) 2024-03-06T07:32:04.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47137fe640 1 -- 172.21.15.122:0/4192261451 learned_addr learned my addr 172.21.15.122:0/4192261451 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f4713fff640 1 --1- 172.21.15.122:0/4192261451 >> v1:172.21.15.122:6789/0 conn(0x7f47140d29c0 0x7f47140b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53608/0 (socket says 172.21.15.122:53608) 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47137fe640 1 -- 172.21.15.122:0/4192261451 >> v1:172.21.15.169:6789/0 conn(0x7f47140cf830 legacy=0x7f47140d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47137fe640 1 -- 172.21.15.122:0/4192261451 >> v1:172.21.15.122:6789/0 conn(0x7f47140d29c0 legacy=0x7f47140b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47127fc640 1 -- 172.21.15.122:0/4192261451 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552809937 0 0) 0x7f47140b3430 con 0x7f47140d29c0 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.753+0000 7f47137fe640 1 -- 172.21.15.122:0/4192261451 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47140b39f0 con 0x7f471415dfe0 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.754+0000 7f47137fe640 1 --2- 172.21.15.122:0/4192261451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f471415dfe0 0x7f47140cf2f0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f47140b3360 tx=0x7f470402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389fd1ca2690505 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.754+0000 7f47127fc640 1 -- 172.21.15.122:0/4192261451 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470402caf0 con 0x7f471415dfe0 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.754+0000 7f47127fc640 1 -- 172.21.15.122:0/4192261451 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470402cc90 con 0x7f471415dfe0 2024-03-06T07:32:04.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.754+0000 7f47127fc640 1 -- 172.21.15.122:0/4192261451 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704034610 con 0x7f471415dfe0 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 -- 172.21.15.122:0/4192261451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f471415dfe0 msgr2=0x7f47140cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 --2- 172.21.15.122:0/4192261451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f471415dfe0 0x7f47140cf2f0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f47140b3360 tx=0x7f470402bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 -- 172.21.15.122:0/4192261451 shutdown_connections 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 --2- 172.21.15.122:0/4192261451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f471415dfe0 0x7f47140cf2f0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 -- 172.21.15.122:0/4192261451 >> 172.21.15.122:0/4192261451 conn(0x7f47140d3cd0 msgr2=0x7f47140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 -- 172.21.15.122:0/4192261451 shutdown_connections 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.755+0000 7f4719a64640 1 -- 172.21.15.122:0/4192261451 wait complete. 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.756+0000 7f4719a64640 1 Processor -- start 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.756+0000 7f4719a64640 1 -- start start 2024-03-06T07:32:04.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 0x7f4714150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47140d29c0 0x7f47141500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f471415dfe0 0x7f47140b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471415e3c0 con 0x7f47140cf830 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47140b3d60 con 0x7f471415dfe0 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471415e540 con 0x7f47140d29c0 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 0x7f4714150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 0x7f4714150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47544/0 (socket says 172.21.15.122:47544) 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 -- 172.21.15.122:0/1647750173 learned_addr learned my addr 172.21.15.122:0/1647750173 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4713fff640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f471415dfe0 0x7f47140b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4712ffd640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47140d29c0 0x7f47141500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 -- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47140d29c0 msgr2=0x7f47141500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47140d29c0 0x7f47141500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 -- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f471415dfe0 msgr2=0x7f47140b7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f471415dfe0 0x7f47140b7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 -- 172.21.15.122:0/1647750173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471415eb70 con 0x7f47140cf830 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f47137fe640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 0x7f4714150950 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f4704036040 tx=0x7f4704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704043020 con 0x7f47140cf830 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471415ee50 con 0x7f47140cf830 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470403e450 con 0x7f47140cf830 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470403e840 con 0x7f47140cf830 2024-03-06T07:32:04.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.757+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471415f410 con 0x7f47140cf830 2024-03-06T07:32:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.758+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471404f330 con 0x7f47140cf830 2024-03-06T07:32:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.759+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4704046020 con 0x7f47140cf830 2024-03-06T07:32:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.759+0000 7f470bfff640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f46f406b4d0 0x7f46f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.759+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f47040acc70 con 0x7f47140cf830 2024-03-06T07:32:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.759+0000 7f4712ffd640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f46f406b4d0 0x7f46f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.760+0000 7f4712ffd640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f46f406b4d0 0x7f46f406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f47140d2320 tx=0x7f46fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.764+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f470404d030 con 0x7f47140cf830 2024-03-06T07:32:04.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.977+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47140b2c30 con 0x7f47140cf830 2024-03-06T07:32:04.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.978+0000 7f470bfff640 1 -- 172.21.15.122:0/1647750173 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f470407bb70 con 0x7f47140cf830 2024-03-06T07:32:04.979 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:04.979 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:04.979 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f46f406b4d0 msgr2=0x7f46f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f46f406b4d0 0x7f46f406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f47140d2320 tx=0x7f46fc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 msgr2=0x7f4714150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 0x7f4714150950 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f4704036040 tx=0x7f4704004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 shutdown_connections 2024-03-06T07:32:04.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f46f406b4d0 0x7f46f406d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f471415dfe0 0x7f47140b7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f47140d29c0 0x7f47141500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 --2- 172.21.15.122:0/1647750173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47140cf830 0x7f4714150950 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 >> 172.21.15.122:0/1647750173 conn(0x7f47140d3cd0 msgr2=0x7f47140cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 shutdown_connections 2024-03-06T07:32:04.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:04.980+0000 7f4719a64640 1 -- 172.21.15.122:0/1647750173 wait complete. 2024-03-06T07:32:05.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:05.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:06.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:06.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.298+0000 7f4257b02640 1 Processor -- start 2024-03-06T07:32:06.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.298+0000 7f4257b02640 1 -- start start 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f4257b02640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f425015dfe0 0x7f42500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f4257b02640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42500cf830 0x7f42500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f4257b02640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42500b3430 con 0x7f42500d27f0 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f4257b02640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42500b31e0 con 0x7f42500cf830 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f4257b02640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42500b3d60 con 0x7f425015dfe0 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f425509d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42500cf830 0x7f42500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f425509d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42500cf830 0x7f42500d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47556/0 (socket says 172.21.15.122:47556) 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.299+0000 7f425509d640 1 -- 172.21.15.122:0/1422275653 learned_addr learned my addr 172.21.15.122:0/1422275653 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:06.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425589e640 1 --2- 172.21.15.122:0/1422275653 >> v2:172.21.15.169:3300/0 conn(0x7f425015dfe0 0x7f42500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425509d640 1 -- 172.21.15.122:0/1422275653 >> v2:172.21.15.169:3300/0 conn(0x7f425015dfe0 msgr2=0x7f42500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425509d640 1 --2- 172.21.15.122:0/1422275653 >> v2:172.21.15.169:3300/0 conn(0x7f425015dfe0 0x7f42500cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425509d640 1 -- 172.21.15.122:0/1422275653 >> v1:172.21.15.169:6789/0 conn(0x7f42500d27f0 legacy=0x7f42500d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425509d640 1 -- 172.21.15.122:0/1422275653 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42500b3710 con 0x7f42500cf830 2024-03-06T07:32:06.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425589e640 1 --2- 172.21.15.122:0/1422275653 >> v2:172.21.15.169:3300/0 conn(0x7f425015dfe0 0x7f42500cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.300+0000 7f425509d640 1 --2- 172.21.15.122:0/1422275653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500cf830 0x7f42500d22b0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f424c009870 tx=0x7f424c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b82ef9e1478d432 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f425489c640 1 -- 172.21.15.122:0/1422275653 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c034070 con 0x7f42500cf830 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f425489c640 1 -- 172.21.15.122:0/1422275653 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c030860 con 0x7f42500cf830 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f425489c640 1 -- 172.21.15.122:0/1422275653 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c030c30 con 0x7f42500cf830 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 -- 172.21.15.122:0/1422275653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500cf830 msgr2=0x7f42500d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 --2- 172.21.15.122:0/1422275653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500cf830 0x7f42500d22b0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f424c009870 tx=0x7f424c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 -- 172.21.15.122:0/1422275653 shutdown_connections 2024-03-06T07:32:06.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 --2- 172.21.15.122:0/1422275653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500cf830 0x7f42500d22b0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 --2- 172.21.15.122:0/1422275653 >> v2:172.21.15.169:3300/0 conn(0x7f425015dfe0 0x7f42500cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 -- 172.21.15.122:0/1422275653 >> 172.21.15.122:0/1422275653 conn(0x7f42500d3cd0 msgr2=0x7f42500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.301+0000 7f4257b02640 1 -- 172.21.15.122:0/1422275653 shutdown_connections 2024-03-06T07:32:06.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.302+0000 7f4257b02640 1 -- 172.21.15.122:0/1422275653 wait complete. 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 Processor -- start 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 -- start start 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42500cf830 0x7f42500b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 0x7f42500b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f425015dfe0 0x7f42500cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42500d1df0 con 0x7f42500d27f0 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42500d1c70 con 0x7f42500cf830 2024-03-06T07:32:06.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f4257b02640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425015e3c0 con 0x7f425015dfe0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f425509d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 0x7f42500b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f425509d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 0x7f42500b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47566/0 (socket says 172.21.15.122:47566) 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.303+0000 7f425509d640 1 -- 172.21.15.122:0/2747943606 learned_addr learned my addr 172.21.15.122:0/2747943606 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425589e640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42500cf830 0x7f42500b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425609f640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f425015dfe0 0x7f42500cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425509d640 1 -- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f425015dfe0 msgr2=0x7f42500cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425509d640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f425015dfe0 0x7f42500cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425509d640 1 -- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42500cf830 msgr2=0x7f42500b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425509d640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42500cf830 0x7f42500b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425509d640 1 -- 172.21.15.122:0/2747943606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250157050 con 0x7f42500d27f0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f425509d640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 0x7f42500b90a0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f424c02b9e0 tx=0x7f424c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c034040 con 0x7f42500d27f0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c0433f0 con 0x7f42500d27f0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425015a9c0 con 0x7f42500d27f0 2024-03-06T07:32:06.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c0437e0 con 0x7f42500d27f0 2024-03-06T07:32:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.304+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42500b71b0 con 0x7f42500d27f0 2024-03-06T07:32:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.305+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425004f330 con 0x7f42500d27f0 2024-03-06T07:32:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.306+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f424c04a060 con 0x7f42500d27f0 2024-03-06T07:32:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.306+0000 7f42467fc640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f423406b500 0x7f423406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.306+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f424c0ac330 con 0x7f42500d27f0 2024-03-06T07:32:06.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.306+0000 7f425589e640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f423406b500 0x7f423406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.307+0000 7f425589e640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f423406b500 0x7f423406d9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f424000ab60 tx=0x7f4240008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.311 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.311+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4250157050 con 0x7f42500d27f0 2024-03-06T07:32:06.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.536+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42500b09c0 con 0x7f42500d27f0 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.537+0000 7f42467fc640 1 -- 172.21.15.122:0/2747943606 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f424c03e070 con 0x7f42500d27f0 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f423406b500 msgr2=0x7f423406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f423406b500 0x7f423406d9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f424000ab60 tx=0x7f4240008040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 msgr2=0x7f42500b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 0x7f42500b90a0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f424c02b9e0 tx=0x7f424c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 shutdown_connections 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f423406b500 0x7f423406d9c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f425015dfe0 0x7f42500cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42500d27f0 0x7f42500b90a0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 --2- 172.21.15.122:0/2747943606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42500cf830 0x7f42500b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 >> 172.21.15.122:0/2747943606 conn(0x7f42500d3cd0 msgr2=0x7f425006bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 shutdown_connections 2024-03-06T07:32:06.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:06.539+0000 7f4257b02640 1 -- 172.21.15.122:0/2747943606 wait complete. 2024-03-06T07:32:06.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:06.752 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:07.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:07.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.851+0000 7fab5fd55640 1 Processor -- start 2024-03-06T07:32:07.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5fd55640 1 -- start start 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5fd55640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fab5806da40 0x7fab580cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5fd55640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab5815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5fd55640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab580cc980 con 0x7fab581424f0 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5fd55640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab580cd220 con 0x7fab5806da40 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5fd55640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab580ccc60 con 0x7fab580cd7a0 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5d2f0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fab5806da40 0x7fab580cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5daf1640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fab581424f0 0x7fab5806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53622/0 (socket says 172.21.15.122:53622) 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.852+0000 7fab5daf1640 1 -- 172.21.15.122:0/1296325396 learned_addr learned my addr 172.21.15.122:0/1296325396 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:07.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.853+0000 7fab5e2f2640 1 --2- 172.21.15.122:0/1296325396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab5815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.853+0000 7fab5d2f0640 1 -- 172.21.15.122:0/1296325396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 msgr2=0x7fab5815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.853+0000 7fab5d2f0640 1 --2- 172.21.15.122:0/1296325396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab5815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.853+0000 7fab5d2f0640 1 -- 172.21.15.122:0/1296325396 >> v1:172.21.15.122:6789/0 conn(0x7fab581424f0 legacy=0x7fab5806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.853+0000 7fab5caef640 1 -- 172.21.15.122:0/1296325396 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995359408 0 0) 0x7fab580cc980 con 0x7fab581424f0 2024-03-06T07:32:07.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.853+0000 7fab5d2f0640 1 -- 172.21.15.122:0/1296325396 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab580ccf40 con 0x7fab5806da40 2024-03-06T07:32:07.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5d2f0640 1 --2- 172.21.15.122:0/1296325396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580cc2f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fab480097a0 tx=0x7fab4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cbe4fbe320f31b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5caef640 1 -- 172.21.15.122:0/1296325396 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48035020 con 0x7fab5806da40 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5caef640 1 -- 172.21.15.122:0/1296325396 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48031830 con 0x7fab5806da40 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5caef640 1 -- 172.21.15.122:0/1296325396 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48031c00 con 0x7fab5806da40 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 -- 172.21.15.122:0/1296325396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 msgr2=0x7fab580cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 --2- 172.21.15.122:0/1296325396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580cc2f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fab480097a0 tx=0x7fab4802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 -- 172.21.15.122:0/1296325396 shutdown_connections 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 --2- 172.21.15.122:0/1296325396 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab5815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 --2- 172.21.15.122:0/1296325396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580cc2f0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 -- 172.21.15.122:0/1296325396 >> 172.21.15.122:0/1296325396 conn(0x7fab580cf8a0 msgr2=0x7fab580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 -- 172.21.15.122:0/1296325396 shutdown_connections 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.854+0000 7fab5fd55640 1 -- 172.21.15.122:0/1296325396 wait complete. 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.855+0000 7fab5fd55640 1 Processor -- start 2024-03-06T07:32:07.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 -- start start 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab58157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab581424f0 0x7fab580cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab58071ce0 con 0x7fab5806da40 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab58071b60 con 0x7fab581424f0 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5fd55640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab58071e60 con 0x7fab580cd7a0 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5daf1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5daf1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47578/0 (socket says 172.21.15.122:47578) 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.856+0000 7fab5daf1640 1 -- 172.21.15.122:0/483069550 learned_addr learned my addr 172.21.15.122:0/483069550 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:07.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5d2f0640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab58157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5e2f2640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab581424f0 0x7fab580cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5daf1640 1 -- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 msgr2=0x7fab58157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5daf1640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab58157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5daf1640 1 -- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab581424f0 msgr2=0x7fab580cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5daf1640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab581424f0 0x7fab580cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5daf1640 1 -- 172.21.15.122:0/483069550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab580b4a40 con 0x7fab5806da40 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5e2f2640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab581424f0 0x7fab580cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5daf1640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580b4500 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fab5400c7f0 tx=0x7fab5400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54018070 con 0x7fab5806da40 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5400a640 con 0x7fab5806da40 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab580ae310 con 0x7fab5806da40 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5400aa30 con 0x7fab5806da40 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.857+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab580ae8d0 con 0x7fab5806da40 2024-03-06T07:32:07.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.858+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5804f330 con 0x7fab5806da40 2024-03-06T07:32:07.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.859+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab5401b070 con 0x7fab5806da40 2024-03-06T07:32:07.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.859+0000 7fab4e7fc640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab3c06b4d0 0x7fab3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.859+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fab54089b80 con 0x7fab5806da40 2024-03-06T07:32:07.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.859+0000 7fab5d2f0640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab3c06b4d0 0x7fab3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.860+0000 7fab5d2f0640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab3c06b4d0 0x7fab3c06d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fab48036000 tx=0x7fab48008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:07.863+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab54058ac0 con 0x7fab5806da40 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.075+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab5806d600 con 0x7fab5806da40 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.075+0000 7fab4e7fc640 1 -- 172.21.15.122:0/483069550 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fab540583c0 con 0x7fab5806da40 2024-03-06T07:32:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:08.078 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:08.078 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab3c06b4d0 msgr2=0x7fab3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab3c06b4d0 0x7fab3c06d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fab48036000 tx=0x7fab48008030 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 msgr2=0x7fab580b4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580b4500 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fab5400c7f0 tx=0x7fab5400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 shutdown_connections 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fab3c06b4d0 0x7fab3c06d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab581424f0 0x7fab580cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fab580cd7a0 0x7fab58157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 --2- 172.21.15.122:0/483069550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab5806da40 0x7fab580b4500 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 >> 172.21.15.122:0/483069550 conn(0x7fab580cf8a0 msgr2=0x7fab5806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 shutdown_connections 2024-03-06T07:32:08.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:08.077+0000 7fab5fd55640 1 -- 172.21.15.122:0/483069550 wait complete. 2024-03-06T07:32:08.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:08.292 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:09.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:09.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.391+0000 7f11915ac640 1 Processor -- start 2024-03-06T07:32:09.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.392+0000 7f11915ac640 1 -- start start 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.392+0000 7f11915ac640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f118c15bdd0 0x7f118c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.392+0000 7f11915ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0cebe0 con 0x7f118c0d16e0 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.392+0000 7f11915ac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0ceec0 con 0x7f118c15fa10 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.392+0000 7f11915ac640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c0ce980 con 0x7f118c15bdd0 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.392+0000 7f118b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f118c0d16e0 0x7f118c0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53638/0 (socket says 172.21.15.122:53638) 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118b7fe640 1 -- 172.21.15.122:0/614263676 learned_addr learned my addr 172.21.15.122:0/614263676 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060726023 0 0) 0x7f118c0cebe0 con 0x7f118c0d16e0 2024-03-06T07:32:09.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118affd640 1 --2- 172.21.15.122:0/614263676 >> v2:172.21.15.169:3300/0 conn(0x7f118c15bdd0 0x7f118c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1178003520 con 0x7f118c0d16e0 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2159303276 0 0) 0x7f1178003520 con 0x7f118c0d16e0 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 >> v2:172.21.15.169:3300/0 conn(0x7f118c15bdd0 msgr2=0x7f118c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 --2- 172.21.15.122:0/614263676 >> v2:172.21.15.169:3300/0 conn(0x7f118c15bdd0 0x7f118c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 >> v1:172.21.15.169:6789/0 conn(0x7f118c15fa10 legacy=0x7f118c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0cf1a0 con 0x7f118c0d16e0 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f1184001a10 con 0x7f118c0d16e0 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 >> v1:172.21.15.122:6789/0 conn(0x7f118c0d16e0 legacy=0x7f118c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11780039b0 0x7f1178003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1178004800 0x7f1178004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11780051c0 0x7f117800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0ce980 con 0x7f1178004800 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117800c160 con 0x7f11780039b0 2024-03-06T07:32:09.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117800c360 con 0x7f11780051c0 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118affd640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11780039b0 0x7f1178003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f11827fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1178004800 0x7f1178004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.393+0000 7f118b7fe640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11780051c0 0x7f117800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f11827fc640 1 -- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11780051c0 msgr2=0x7f117800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f11827fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f11780051c0 0x7f117800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f11827fc640 1 -- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11780039b0 msgr2=0x7f1178003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f11827fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11780039b0 0x7f1178003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f118b7fe640 1 -- 172.21.15.122:0/614263676 reap_dead start 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f11827fc640 1 -- 172.21.15.122:0/614263676 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1178003520 con 0x7f1178004800 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f11827fc640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1178004800 0x7f1178004c80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f1174001720 tx=0x7f1178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed91053fe1c22c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174001370 con 0x7f1178004800 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11740052d0 con 0x7f1178004800 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.394+0000 7f118a7fc640 1 -- 172.21.15.122:0/614263676 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174003830 con 0x7f1178004800 2024-03-06T07:32:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 -- 172.21.15.122:0/614263676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1178004800 msgr2=0x7f1178004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1178004800 0x7f1178004c80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f1174001720 tx=0x7f1178005830 comp rx=0 tx=0).stop 2024-03-06T07:32:09.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 -- 172.21.15.122:0/614263676 shutdown_connections 2024-03-06T07:32:09.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 --2- 172.21.15.122:0/614263676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1178004800 0x7f1178004c80 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 -- 172.21.15.122:0/614263676 >> 172.21.15.122:0/614263676 conn(0x7f118c09a9f0 msgr2=0x7f118c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 -- 172.21.15.122:0/614263676 shutdown_connections 2024-03-06T07:32:09.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.395+0000 7f11915ac640 1 -- 172.21.15.122:0/614263676 wait complete. 2024-03-06T07:32:09.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.397+0000 7f11915ac640 1 Processor -- start 2024-03-06T07:32:09.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.397+0000 7f11915ac640 1 -- start start 2024-03-06T07:32:09.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11915ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1178004800 0x7f118c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11915ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 0x7f118c0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11915ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f118c0c8b90 0x7f118c0b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11915ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c15c1b0 con 0x7f118c15b350 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11915ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0cc5d0 con 0x7f118c0c8b90 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11915ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0ce270 con 0x7f1178004800 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11827fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 0x7f118c0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11827fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 0x7f118c0b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47588/0 (socket says 172.21.15.122:47588) 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f11827fc640 1 -- 172.21.15.122:0/88750370 learned_addr learned my addr 172.21.15.122:0/88750370 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f118affd640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1178004800 0x7f118c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.398+0000 7f118b7fe640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f118c0c8b90 0x7f118c0b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11827fc640 1 -- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1178004800 msgr2=0x7f118c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11827fc640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1178004800 0x7f118c15aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11827fc640 1 -- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f118c0c8b90 msgr2=0x7f118c0b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11827fc640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f118c0c8b90 0x7f118c0b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11827fc640 1 -- 172.21.15.122:0/88750370 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c15b0e0 con 0x7f118c15b350 2024-03-06T07:32:09.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11827fc640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 0x7f118c0b68a0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1174000c00 tx=0x7f1174001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174041020 con 0x7f118c15b350 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174005480 con 0x7f118c15b350 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c0c9090 con 0x7f118c15b350 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174001370 con 0x7f118c15b350 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.399+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c0c9650 con 0x7f118c15b350 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.400+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c04f330 con 0x7f118c15b350 2024-03-06T07:32:09.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.400+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1174044020 con 0x7f118c15b350 2024-03-06T07:32:09.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.401+0000 7f1183fff640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f116c06b500 0x7f116c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.401+0000 7f118affd640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f116c06b500 0x7f116c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.401+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f11740abba0 con 0x7f118c15b350 2024-03-06T07:32:09.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.402+0000 7f118affd640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f116c06b500 0x7f116c06d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f118c09fa80 tx=0x7f117c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.405+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f117404b030 con 0x7f118c15b350 2024-03-06T07:32:09.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.623+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f118c0b7a40 con 0x7f118c15b350 2024-03-06T07:32:09.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.623+0000 7f1183fff640 1 -- 172.21.15.122:0/88750370 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f118c15b0e0 con 0x7f118c15b350 2024-03-06T07:32:09.625 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:09.625 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f116c06b500 msgr2=0x7f116c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f116c06b500 0x7f116c06d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f118c09fa80 tx=0x7f117c009190 comp rx=0 tx=0).stop 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 msgr2=0x7f118c0b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 0x7f118c0b68a0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1174000c00 tx=0x7f1174001120 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 shutdown_connections 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f116c06b500 0x7f116c06d9c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f118c0c8b90 0x7f118c0b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f118c15b350 0x7f118c0b68a0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 --2- 172.21.15.122:0/88750370 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1178004800 0x7f118c15aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 >> 172.21.15.122:0/88750370 conn(0x7f118c09a9f0 msgr2=0x7f118c06f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 shutdown_connections 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:09.625+0000 7f11915ac640 1 -- 172.21.15.122:0/88750370 wait complete. 2024-03-06T07:32:09.839 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:09.839 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:10.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:10.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.940+0000 7fb65fb3e640 1 Processor -- start 2024-03-06T07:32:10.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.940+0000 7fb65fb3e640 1 -- start start 2024-03-06T07:32:10.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65fb3e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb6580d22b0 0x7fb6580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65fb3e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6580b41f0 0x7fb6580b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65fb3e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6580b3430 con 0x7fb65815dfe0 2024-03-06T07:32:10.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65fb3e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580d2c50 con 0x7fb6580b41f0 2024-03-06T07:32:10.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65fb3e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580b31e0 con 0x7fb6580d22b0 2024-03-06T07:32:10.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6580b41f0 0x7fb6580b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6580b41f0 0x7fb6580b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47592/0 (socket says 172.21.15.122:47592) 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 -- 172.21.15.122:0/2254381918 learned_addr learned my addr 172.21.15.122:0/2254381918 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65d0d9640 1 --2- 172.21.15.122:0/2254381918 >> v2:172.21.15.169:3300/0 conn(0x7fb6580d22b0 0x7fb6580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 -- 172.21.15.122:0/2254381918 >> v2:172.21.15.169:3300/0 conn(0x7fb6580d22b0 msgr2=0x7fb6580d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 --2- 172.21.15.122:0/2254381918 >> v2:172.21.15.169:3300/0 conn(0x7fb6580d22b0 0x7fb6580d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 -- 172.21.15.122:0/2254381918 >> v1:172.21.15.169:6789/0 conn(0x7fb65815dfe0 legacy=0x7fb6580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.941+0000 7fb65e0db640 1 -- 172.21.15.122:0/2254381918 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580b3710 con 0x7fb6580b41f0 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.942+0000 7fb65e0db640 1 --2- 172.21.15.122:0/2254381918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580b41f0 0x7fb6580b4680 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb650009e50 tx=0x7fb65002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66570e142192008 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.942+0000 7fb65c8d8640 1 -- 172.21.15.122:0/2254381918 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65002ca80 con 0x7fb6580b41f0 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.942+0000 7fb65c8d8640 1 -- 172.21.15.122:0/2254381918 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65002cc20 con 0x7fb6580b41f0 2024-03-06T07:32:10.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.942+0000 7fb65c8d8640 1 -- 172.21.15.122:0/2254381918 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500343f0 con 0x7fb6580b41f0 2024-03-06T07:32:10.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2254381918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580b41f0 msgr2=0x7fb6580b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2254381918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580b41f0 0x7fb6580b4680 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb650009e50 tx=0x7fb65002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2254381918 shutdown_connections 2024-03-06T07:32:10.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2254381918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580b41f0 0x7fb6580b4680 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2254381918 >> v2:172.21.15.169:3300/0 conn(0x7fb6580d22b0 0x7fb6580d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2254381918 >> 172.21.15.122:0/2254381918 conn(0x7fb6580d3cd0 msgr2=0x7fb6580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2254381918 shutdown_connections 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.943+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2254381918 wait complete. 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.944+0000 7fb65fb3e640 1 Processor -- start 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.944+0000 7fb65fb3e640 1 -- start start 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6580b41f0 0x7fb6580b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 0x7fb6580b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb65815dfe0 0x7fb6580b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580cfa00 con 0x7fb6580d22b0 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b3d60 con 0x7fb65815dfe0 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580d1c70 con 0x7fb6580b41f0 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 0x7fb6580b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 0x7fb6580b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47600/0 (socket says 172.21.15.122:47600) 2024-03-06T07:32:10.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 -- 172.21.15.122:0/2650490917 learned_addr learned my addr 172.21.15.122:0/2650490917 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d8da640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6580b41f0 0x7fb6580b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 -- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6580b41f0 msgr2=0x7fb6580b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65e0db640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb65815dfe0 0x7fb6580b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6580b41f0 0x7fb6580b8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 -- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb65815dfe0 msgr2=0x7fb6580b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb65815dfe0 0x7fb6580b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 -- 172.21.15.122:0/2650490917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580b9be0 con 0x7fb6580d22b0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d8da640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6580b41f0 0x7fb6580b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65e0db640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb65815dfe0 0x7fb6580b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65d0d9640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 0x7fb6580b68a0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb64800d830 tx=0x7fb64800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800b6c0 con 0x7fb6580d22b0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648019070 con 0x7fb6580d22b0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6580b9060 con 0x7fb6580d22b0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800ba80 con 0x7fb6580d22b0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.945+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6580b9620 con 0x7fb6580d22b0 2024-03-06T07:32:10.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.946+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65804f330 con 0x7fb6580d22b0 2024-03-06T07:32:10.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.947+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb64800bc20 con 0x7fb6580d22b0 2024-03-06T07:32:10.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.947+0000 7fb64e7fc640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63c06b500 0x7fb63c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.947+0000 7fb65d8da640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63c06b500 0x7fb63c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.947+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb64808a680 con 0x7fb6580d22b0 2024-03-06T07:32:10.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.948+0000 7fb65d8da640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63c06b500 0x7fb63c06d9c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb65400a830 tx=0x7fb654008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:10.952+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb64801c360 con 0x7fb6580d22b0 2024-03-06T07:32:11.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.166+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6580cea80 con 0x7fb6580d22b0 2024-03-06T07:32:11.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.167+0000 7fb64e7fc640 1 -- 172.21.15.122:0/2650490917 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fb6480594e0 con 0x7fb6580d22b0 2024-03-06T07:32:11.169 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:11.169 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:11.169 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63c06b500 msgr2=0x7fb63c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63c06b500 0x7fb63c06d9c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb65400a830 tx=0x7fb654008040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 msgr2=0x7fb6580b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 0x7fb6580b68a0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb64800d830 tx=0x7fb64800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 shutdown_connections 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb63c06b500 0x7fb63c06d9c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb65815dfe0 0x7fb6580b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6580d22b0 0x7fb6580b68a0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 --2- 172.21.15.122:0/2650490917 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb6580b41f0 0x7fb6580b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.168+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 >> 172.21.15.122:0/2650490917 conn(0x7fb6580d3cd0 msgr2=0x7fb6580d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.169+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 shutdown_connections 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:11.169+0000 7fb65fb3e640 1 -- 172.21.15.122:0/2650490917 wait complete. 2024-03-06T07:32:11.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:11.383 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:12.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:12.485 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.483+0000 7ff2a43f7640 1 Processor -- start 2024-03-06T07:32:12.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.483+0000 7ff2a43f7640 1 -- start start 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.484+0000 7ff2a43f7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff29c0d2710 0x7ff29c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.484+0000 7ff2a43f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0b3430 con 0x7ff29c15dfe0 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.484+0000 7ff2a43f7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0b3710 con 0x7ff29c0d22b0 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.484+0000 7ff2a43f7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29c0b31e0 con 0x7ff29c0d2710 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.484+0000 7ff2a2994640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff29c0d2710 0x7ff29c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.485+0000 7ff2a2193640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff29c15dfe0 0x7ff29c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53652/0 (socket says 172.21.15.122:53652) 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.485+0000 7ff2a2193640 1 -- 172.21.15.122:0/4073660336 learned_addr learned my addr 172.21.15.122:0/4073660336 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.485+0000 7ff2a2994640 1 -- 172.21.15.122:0/4073660336 >> v1:172.21.15.169:6789/0 conn(0x7ff29c0d22b0 legacy=0x7ff29c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.485+0000 7ff2a2994640 1 -- 172.21.15.122:0/4073660336 >> v1:172.21.15.122:6789/0 conn(0x7ff29c15dfe0 legacy=0x7ff29c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.485+0000 7ff2a2994640 1 -- 172.21.15.122:0/4073660336 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c0b39f0 con 0x7ff29c0d2710 2024-03-06T07:32:12.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.485+0000 7ff2a1191640 1 -- 172.21.15.122:0/4073660336 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392795277 0 0) 0x7ff29c0b3430 con 0x7ff29c15dfe0 2024-03-06T07:32:12.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a2994640 1 --2- 172.21.15.122:0/4073660336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d2710 0x7ff29c0b4900 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7ff294009dc0 tx=0x7ff29402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656727ea9940398b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a1191640 1 -- 172.21.15.122:0/4073660336 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294039070 con 0x7ff29c0d2710 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a1191640 1 -- 172.21.15.122:0/4073660336 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29402ccd0 con 0x7ff29c0d2710 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a1191640 1 -- 172.21.15.122:0/4073660336 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294034660 con 0x7ff29c0d2710 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4073660336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d2710 msgr2=0x7ff29c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4073660336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d2710 0x7ff29c0b4900 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7ff294009dc0 tx=0x7ff29402bf20 comp rx=0 tx=0).stop 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4073660336 shutdown_connections 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4073660336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d2710 0x7ff29c0b4900 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4073660336 >> 172.21.15.122:0/4073660336 conn(0x7ff29c0d3cd0 msgr2=0x7ff29c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4073660336 shutdown_connections 2024-03-06T07:32:12.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.486+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4073660336 wait complete. 2024-03-06T07:32:12.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.487+0000 7ff2a43f7640 1 Processor -- start 2024-03-06T07:32:12.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.487+0000 7ff2a43f7640 1 -- start start 2024-03-06T07:32:12.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a43f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 0x7ff29c15bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a43f7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff29c0d2710 0x7ff29c15c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a43f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff29c15dfe0 0x7ff29c0b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a43f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c0cfa00 con 0x7ff29c0d22b0 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a43f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c0b3d60 con 0x7ff29c15dfe0 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a43f7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c0d1c70 con 0x7ff29c0d2710 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a2193640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 0x7ff29c15bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a2193640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 0x7ff29c15bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47622/0 (socket says 172.21.15.122:47622) 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a2193640 1 -- 172.21.15.122:0/4096341630 learned_addr learned my addr 172.21.15.122:0/4096341630 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a2994640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff29c15dfe0 0x7ff29c0b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.488+0000 7ff2a1992640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff29c0d2710 0x7ff29c15c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a2193640 1 -- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff29c0d2710 msgr2=0x7ff29c15c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a2193640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff29c0d2710 0x7ff29c15c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a2193640 1 -- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff29c15dfe0 msgr2=0x7ff29c0b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a2193640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff29c15dfe0 0x7ff29c0b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a2193640 1 -- 172.21.15.122:0/4096341630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c15ef00 con 0x7ff29c0d22b0 2024-03-06T07:32:12.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a2193640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 0x7ff29c15bde0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c3f0 tx=0x7ff29800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2980074e0 con 0x7ff29c0d22b0 2024-03-06T07:32:12.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2980047e0 con 0x7ff29c0d22b0 2024-03-06T07:32:12.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c15f1e0 con 0x7ff29c0d22b0 2024-03-06T07:32:12.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298004c10 con 0x7ff29c0d22b0 2024-03-06T07:32:12.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.489+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c15f7a0 con 0x7ff29c0d22b0 2024-03-06T07:32:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.490+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c04f330 con 0x7ff29c0d22b0 2024-03-06T07:32:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.490+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff298011070 con 0x7ff29c0d22b0 2024-03-06T07:32:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.491+0000 7ff292ffd640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff28006b4d0 0x7ff28006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.491+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff298089c30 con 0x7ff29c0d22b0 2024-03-06T07:32:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.491+0000 7ff2a1992640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff28006b4d0 0x7ff28006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.491+0000 7ff2a1992640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff28006b4d0 0x7ff28006d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff29c1600b0 tx=0x7ff28c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.499+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2980584d0 con 0x7ff29c0d22b0 2024-03-06T07:32:12.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.715+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff29c06cd70 con 0x7ff29c0d22b0 2024-03-06T07:32:12.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.715+0000 7ff292ffd640 1 -- 172.21.15.122:0/4096341630 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7ff29805c570 con 0x7ff29c0d22b0 2024-03-06T07:32:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:12.717 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:12.718 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:12.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff28006b4d0 msgr2=0x7ff28006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff28006b4d0 0x7ff28006d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff29c1600b0 tx=0x7ff28c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:12.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 msgr2=0x7ff29c15bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 0x7ff29c15bde0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c3f0 tx=0x7ff29800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 shutdown_connections 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff28006b4d0 0x7ff28006d990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff29c15dfe0 0x7ff29c0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff29c0d2710 0x7ff29c15c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 --2- 172.21.15.122:0/4096341630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff29c0d22b0 0x7ff29c15bde0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 >> 172.21.15.122:0/4096341630 conn(0x7ff29c0d3cd0 msgr2=0x7ff29c06cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 shutdown_connections 2024-03-06T07:32:12.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:12.717+0000 7ff2a43f7640 1 -- 172.21.15.122:0/4096341630 wait complete. 2024-03-06T07:32:12.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:12.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:13.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:14.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.033+0000 7ff53729a640 1 Processor -- start 2024-03-06T07:32:14.036 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.034+0000 7ff53729a640 1 -- start start 2024-03-06T07:32:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.034+0000 7ff53729a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5301424f0 0x7ff5300cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.034+0000 7ff53729a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff53006da40 0x7ff53015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.034+0000 7ff53729a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5300cc980 con 0x7ff5300cd7a0 2024-03-06T07:32:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.034+0000 7ff53729a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300cd220 con 0x7ff5301424f0 2024-03-06T07:32:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.034+0000 7ff53729a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300ccc60 con 0x7ff53006da40 2024-03-06T07:32:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535036640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5301424f0 0x7ff5300cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff534835640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff5300cd7a0 0x7ff53006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53666/0 (socket says 172.21.15.122:53666) 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535036640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff5301424f0 0x7ff5300cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47636/0 (socket says 172.21.15.122:47636) 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff534835640 1 -- 172.21.15.122:0/2122669259 learned_addr learned my addr 172.21.15.122:0/2122669259 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535837640 1 --2- 172.21.15.122:0/2122669259 >> v2:172.21.15.169:3300/0 conn(0x7ff53006da40 0x7ff53015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535036640 1 -- 172.21.15.122:0/2122669259 >> v2:172.21.15.169:3300/0 conn(0x7ff53006da40 msgr2=0x7ff53015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535036640 1 --2- 172.21.15.122:0/2122669259 >> v2:172.21.15.169:3300/0 conn(0x7ff53006da40 0x7ff53015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535036640 1 -- 172.21.15.122:0/2122669259 >> v1:172.21.15.122:6789/0 conn(0x7ff5300cd7a0 legacy=0x7ff53006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff535036640 1 -- 172.21.15.122:0/2122669259 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300ccf40 con 0x7ff5301424f0 2024-03-06T07:32:14.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.035+0000 7ff527fff640 1 -- 172.21.15.122:0/2122669259 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147444290 0 0) 0x7ff5300cc980 con 0x7ff5300cd7a0 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff535036640 1 --2- 172.21.15.122:0/2122669259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff5300cc2f0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7ff530068700 tx=0x7ff52c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc171a823158956 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff527fff640 1 -- 172.21.15.122:0/2122669259 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c035020 con 0x7ff5301424f0 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff527fff640 1 -- 172.21.15.122:0/2122669259 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c031830 con 0x7ff5301424f0 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff527fff640 1 -- 172.21.15.122:0/2122669259 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c031c00 con 0x7ff5301424f0 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 -- 172.21.15.122:0/2122669259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 msgr2=0x7ff5300cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 --2- 172.21.15.122:0/2122669259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff5300cc2f0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7ff530068700 tx=0x7ff52c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 -- 172.21.15.122:0/2122669259 shutdown_connections 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 --2- 172.21.15.122:0/2122669259 >> v2:172.21.15.169:3300/0 conn(0x7ff53006da40 0x7ff53015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 --2- 172.21.15.122:0/2122669259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff5300cc2f0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 -- 172.21.15.122:0/2122669259 >> 172.21.15.122:0/2122669259 conn(0x7ff5300cf8a0 msgr2=0x7ff5300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.036+0000 7ff53729a640 1 -- 172.21.15.122:0/2122669259 shutdown_connections 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.037+0000 7ff53729a640 1 -- 172.21.15.122:0/2122669259 wait complete. 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 Processor -- start 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 -- start start 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff53006da40 0x7ff53013e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5300cd7a0 0x7ff5300b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff530158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff530071ce0 con 0x7ff5301424f0 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff530071b60 con 0x7ff53006da40 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff53729a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff530071e60 con 0x7ff5300cd7a0 2024-03-06T07:32:14.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff535837640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff530158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff535837640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff530158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:47642/0 (socket says 172.21.15.122:47642) 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.038+0000 7ff535837640 1 -- 172.21.15.122:0/2299848010 learned_addr learned my addr 172.21.15.122:0/2299848010 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535036640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff53006da40 0x7ff53013e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff534835640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5300cd7a0 0x7ff5300b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535837640 1 -- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5300cd7a0 msgr2=0x7ff5300b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535837640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5300cd7a0 0x7ff5300b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535837640 1 -- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff53006da40 msgr2=0x7ff53013e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535837640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff53006da40 0x7ff53013e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535837640 1 -- 172.21.15.122:0/2299848010 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53013f400 con 0x7ff5301424f0 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535036640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff53006da40 0x7ff53013e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff535837640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff530158ef0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7ff528007de0 tx=0x7ff52800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528014620 con 0x7ff5301424f0 2024-03-06T07:32:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528019070 con 0x7ff5301424f0 2024-03-06T07:32:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53013f6a0 con 0x7ff5301424f0 2024-03-06T07:32:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528014a20 con 0x7ff5301424f0 2024-03-06T07:32:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.039+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53013f940 con 0x7ff5301424f0 2024-03-06T07:32:14.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.040+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53004f330 con 0x7ff5301424f0 2024-03-06T07:32:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.041+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff528014bc0 con 0x7ff5301424f0 2024-03-06T07:32:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.041+0000 7ff525ffb640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff51406b4d0 0x7ff51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.041+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff52808ab00 con 0x7ff5301424f0 2024-03-06T07:32:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.041+0000 7ff535036640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff51406b4d0 0x7ff51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.042+0000 7ff535036640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff51406b4d0 0x7ff51406d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff52c036000 tx=0x7ff52c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.046+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5280599f0 con 0x7ff5301424f0 2024-03-06T07:32:14.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.261+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5300d0620 con 0x7ff5301424f0 2024-03-06T07:32:14.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.262+0000 7ff525ffb640 1 -- 172.21.15.122:0/2299848010 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7ff5280592f0 con 0x7ff5301424f0 2024-03-06T07:32:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:14.264 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:14.264 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:14.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff51406b4d0 msgr2=0x7ff51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff51406b4d0 0x7ff51406d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff52c036000 tx=0x7ff52c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:14.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 msgr2=0x7ff530158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff530158ef0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7ff528007de0 tx=0x7ff52800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 shutdown_connections 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff51406b4d0 0x7ff51406d990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5301424f0 0x7ff530158ef0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff5300cd7a0 0x7ff5300b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 --2- 172.21.15.122:0/2299848010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff53006da40 0x7ff53013e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 >> 172.21.15.122:0/2299848010 conn(0x7ff5300cf8a0 msgr2=0x7ff5300cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 shutdown_connections 2024-03-06T07:32:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:14.264+0000 7ff53729a640 1 -- 172.21.15.122:0/2299848010 wait complete. 2024-03-06T07:32:14.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:14.478 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:15.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:15.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.579+0000 7fa21f822640 1 Processor -- start 2024-03-06T07:32:15.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.579+0000 7fa21f822640 1 -- start start 2024-03-06T07:32:15.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21f822640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21f822640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa2180d29c0 0x7fa2180b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21f822640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180b3430 con 0x7fa2180cf830 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21f822640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2180b31e0 con 0x7fa2180d29c0 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21f822640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180b3d60 con 0x7fa21815dfe0 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21ddbf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa2180d29c0 0x7fa2180b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21ddbf640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa2180d29c0 0x7fa2180b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:49378/0 (socket says 172.21.15.122:49378) 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21d5be640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.580+0000 7fa21ddbf640 1 -- 172.21.15.122:0/523742245 learned_addr learned my addr 172.21.15.122:0/523742245 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.581+0000 7fa21ddbf640 1 -- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 msgr2=0x7fa2180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.581+0000 7fa21ddbf640 1 --2- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.581+0000 7fa21ddbf640 1 -- 172.21.15.122:0/523742245 >> v1:172.21.15.169:6789/0 conn(0x7fa2180cf830 legacy=0x7fa2180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.581+0000 7fa21ddbf640 1 -- 172.21.15.122:0/523742245 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180b3710 con 0x7fa2180d29c0 2024-03-06T07:32:15.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.581+0000 7fa20ffff640 1 -- 172.21.15.122:0/523742245 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456809749 0 0) 0x7fa2180b3430 con 0x7fa2180cf830 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.581+0000 7fa21ddbf640 1 --2- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180d29c0 0x7fa2180b41f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa210009e50 tx=0x7fa21002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c11434412697b454 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa20ffff640 1 -- 172.21.15.122:0/523742245 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21002cb00 con 0x7fa2180d29c0 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa20ffff640 1 -- 172.21.15.122:0/523742245 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21002cca0 con 0x7fa2180d29c0 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa20ffff640 1 -- 172.21.15.122:0/523742245 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210034610 con 0x7fa2180d29c0 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 -- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180d29c0 msgr2=0x7fa2180b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 --2- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180d29c0 0x7fa2180b41f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa210009e50 tx=0x7fa21002bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:15.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 -- 172.21.15.122:0/523742245 shutdown_connections 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 --2- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180d29c0 0x7fa2180b41f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 --2- 172.21.15.122:0/523742245 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 -- 172.21.15.122:0/523742245 >> 172.21.15.122:0/523742245 conn(0x7fa2180d3cd0 msgr2=0x7fa2180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 -- 172.21.15.122:0/523742245 shutdown_connections 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.582+0000 7fa21f822640 1 -- 172.21.15.122:0/523742245 wait complete. 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.583+0000 7fa21f822640 1 Processor -- start 2024-03-06T07:32:15.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.583+0000 7fa21f822640 1 -- start start 2024-03-06T07:32:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21f822640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180cf830 0x7fa2180b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21f822640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 0x7fa21814fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21f822640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21f822640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180d1df0 con 0x7fa2180d29c0 2024-03-06T07:32:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21f822640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180d1c70 con 0x7fa2180cf830 2024-03-06T07:32:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21f822640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21815e3c0 con 0x7fa21815dfe0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 0x7fa21814fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 0x7fa21814fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38760/0 (socket says 172.21.15.122:38760) 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 -- 172.21.15.122:0/4251861151 learned_addr learned my addr 172.21.15.122:0/4251861151 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21ddbf640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21d5be640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180cf830 0x7fa2180b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 -- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 msgr2=0x7fa2180cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 -- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180cf830 msgr2=0x7fa2180b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180cf830 0x7fa2180b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.584+0000 7fa21cdbd640 1 -- 172.21.15.122:0/4251861151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180cc3f0 con 0x7fa2180d29c0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.585+0000 7fa21cdbd640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 0x7fa21814fc00 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c520 tx=0x7fa20800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.585+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208017070 con 0x7fa2180d29c0 2024-03-06T07:32:15.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.585+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2080035f0 con 0x7fa2180d29c0 2024-03-06T07:32:15.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.585+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2180cc6d0 con 0x7fa2180d29c0 2024-03-06T07:32:15.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.585+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2080039e0 con 0x7fa2180d29c0 2024-03-06T07:32:15.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.585+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2180ccf70 con 0x7fa2180d29c0 2024-03-06T07:32:15.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.586+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2180d3770 con 0x7fa2180d29c0 2024-03-06T07:32:15.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.586+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x7fa2180d29c0 2024-03-06T07:32:15.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.587+0000 7fa20dffb640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1fc06b4d0 0x7fa1fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.587+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa2080894c0 con 0x7fa2180d29c0 2024-03-06T07:32:15.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.587+0000 7fa21d5be640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1fc06b4d0 0x7fa1fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.587+0000 7fa21d5be640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1fc06b4d0 0x7fa1fc06d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa214009aa0 tx=0x7fa214009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.591+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2180d3770 con 0x7fa2180d29c0 2024-03-06T07:32:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.833+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2180cd170 con 0x7fa2180d29c0 2024-03-06T07:32:15.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.833+0000 7fa20dffb640 1 -- 172.21.15.122:0/4251861151 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fa2180cd170 con 0x7fa2180d29c0 2024-03-06T07:32:15.836 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:15.836 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:15.836 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:15.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1fc06b4d0 msgr2=0x7fa1fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1fc06b4d0 0x7fa1fc06d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa214009aa0 tx=0x7fa214009190 comp rx=0 tx=0).stop 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 msgr2=0x7fa21814fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 0x7fa21814fc00 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c520 tx=0x7fa20800c9e0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 shutdown_connections 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa1fc06b4d0 0x7fa1fc06d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa21815dfe0 0x7fa2180cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2180d29c0 0x7fa21814fc00 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 --2- 172.21.15.122:0/4251861151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2180cf830 0x7fa2180b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 >> 172.21.15.122:0/4251861151 conn(0x7fa2180d3cd0 msgr2=0x7fa21806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 shutdown_connections 2024-03-06T07:32:15.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:15.835+0000 7fa21f822640 1 -- 172.21.15.122:0/4251861151 wait complete. 2024-03-06T07:32:16.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:16.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:17.053 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.153+0000 7fbd8215e640 1 Processor -- start 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.153+0000 7fbd8215e640 1 -- start start 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.154+0000 7fbd8215e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c15bdd0 0x7fbd7c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.154+0000 7fbd8215e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbd7c15fa10 0x7fbd7c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.154+0000 7fbd8215e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c0cebe0 con 0x7fbd7c0d16e0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.154+0000 7fbd8215e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7c0ceec0 con 0x7fbd7c15fa10 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.154+0000 7fbd8215e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c0cf1a0 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbd7c15fa10 0x7fbd7c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7b7fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c15bdd0 0x7fbd7c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbd7c15fa10 0x7fbd7c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:49388/0 (socket says 172.21.15.122:49388) 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 -- 172.21.15.122:0/1648001400 learned_addr learned my addr 172.21.15.122:0/1648001400 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 -- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c15bdd0 msgr2=0x7fbd7c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 --2- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c15bdd0 0x7fbd7c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 -- 172.21.15.122:0/1648001400 >> v1:172.21.15.169:6789/0 conn(0x7fbd7c0d16e0 legacy=0x7fbd7c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7affd640 1 -- 172.21.15.122:0/1648001400 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c0cc8e0 con 0x7fbd7c15fa10 2024-03-06T07:32:17.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.155+0000 7fbd7b7fe640 1 --2- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c15bdd0 0x7fbd7c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.156+0000 7fbd7affd640 1 --2- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c0ce270 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009870 tx=0x7fbd6c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4dcc75f21d9f4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.156+0000 7fbd7a7fc640 1 -- 172.21.15.122:0/1648001400 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c035020 con 0x7fbd7c15fa10 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.156+0000 7fbd7a7fc640 1 -- 172.21.15.122:0/1648001400 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c031640 con 0x7fbd7c15fa10 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.156+0000 7fbd7a7fc640 1 -- 172.21.15.122:0/1648001400 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c02d040 con 0x7fbd7c15fa10 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.156+0000 7fbd8215e640 1 -- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 msgr2=0x7fbd7c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.156+0000 7fbd8215e640 1 --2- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c0ce270 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009870 tx=0x7fbd6c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.157+0000 7fbd8215e640 1 -- 172.21.15.122:0/1648001400 shutdown_connections 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.157+0000 7fbd8215e640 1 --2- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c0ce270 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.157+0000 7fbd8215e640 1 --2- 172.21.15.122:0/1648001400 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c15bdd0 0x7fbd7c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.157+0000 7fbd8215e640 1 -- 172.21.15.122:0/1648001400 >> 172.21.15.122:0/1648001400 conn(0x7fbd7c09a9f0 msgr2=0x7fbd7c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.157+0000 7fbd8215e640 1 -- 172.21.15.122:0/1648001400 shutdown_connections 2024-03-06T07:32:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.157+0000 7fbd8215e640 1 -- 172.21.15.122:0/1648001400 wait complete. 2024-03-06T07:32:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.158+0000 7fbd8215e640 1 Processor -- start 2024-03-06T07:32:17.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.158+0000 7fbd8215e640 1 -- start start 2024-03-06T07:32:17.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c0d16e0 0x7fbd7c0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 0x7fbd7c0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c15c1b0 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c0cc5d0 con 0x7fbd7c15fa10 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c0ce7b0 con 0x7fbd7c0d16e0 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 0x7fbd7c0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 0x7fbd7c0ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38762/0 (socket says 172.21.15.122:38762) 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 -- 172.21.15.122:0/2821053043 learned_addr learned my addr 172.21.15.122:0/2821053043 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7b7fe640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c0d16e0 0x7fbd7c0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7bfff640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 -- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c0d16e0 msgr2=0x7fbd7c0c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c0d16e0 0x7fbd7c0c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 -- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 msgr2=0x7fbd7c152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 -- 172.21.15.122:0/2821053043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c1f5430 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd7affd640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 0x7fbd7c0ca000 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c02b930 tx=0x7fbd6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c040070 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c03e070 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c035040 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7c1f4820 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.159+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7c1f4de0 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.160+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7c04f330 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.161+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd6c038020 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.161+0000 7fbd63fff640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd6806b4d0 0x7fbd6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.161+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbd6c0abb20 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.161+0000 7fbd7b7fe640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd6806b4d0 0x7fbd6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.162+0000 7fbd7b7fe640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd6806b4d0 0x7fbd6806d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009560 tx=0x7fbd70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.166+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd6c07a980 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.374+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd7c0b7a40 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.374+0000 7fbd63fff640 1 -- 172.21.15.122:0/2821053043 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fbd6c07a280 con 0x7fbd7c15bdd0 2024-03-06T07:32:17.376 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:17.377 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:17.377 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:17.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd6806b4d0 msgr2=0x7fbd6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd6806b4d0 0x7fbd6806d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009560 tx=0x7fbd70008040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 msgr2=0x7fbd7c0ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 0x7fbd7c0ca000 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c02b930 tx=0x7fbd6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 shutdown_connections 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd6806b4d0 0x7fbd6806d990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd7c15fa10 0x7fbd7c152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c15bdd0 0x7fbd7c0ca000 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 --2- 172.21.15.122:0/2821053043 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbd7c0d16e0 0x7fbd7c0c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 >> 172.21.15.122:0/2821053043 conn(0x7fbd7c09a9f0 msgr2=0x7fbd7c15e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 shutdown_connections 2024-03-06T07:32:17.379 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:17.376+0000 7fbd8215e640 1 -- 172.21.15.122:0/2821053043 wait complete. 2024-03-06T07:32:17.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:17.590 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:18.591 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:18.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.689+0000 7f19c878a640 1 Processor -- start 2024-03-06T07:32:18.691 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.689+0000 7f19c878a640 1 -- start start 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.690+0000 7f19c878a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f19c00d2710 0x7f19c00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.690+0000 7f19c878a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c00b3430 con 0x7f19c015dfe0 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.690+0000 7f19c878a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c00b3710 con 0x7f19c00d22b0 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.690+0000 7f19c878a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c00b31e0 con 0x7f19c00d2710 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c6526640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f19c015dfe0 0x7f19c00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40218/0 (socket says 172.21.15.122:40218) 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c6526640 1 -- 172.21.15.122:0/3336752928 learned_addr learned my addr 172.21.15.122:0/3336752928 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094537415 0 0) 0x7f19c00b3430 con 0x7f19c015dfe0 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c6d27640 1 --2- 172.21.15.122:0/3336752928 >> v2:172.21.15.169:3300/0 conn(0x7f19c00d2710 0x7f19c00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac003520 con 0x7f19c015dfe0 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233389183 0 0) 0x7f19ac003520 con 0x7f19c015dfe0 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 >> v2:172.21.15.169:3300/0 conn(0x7f19c00d2710 msgr2=0x7f19c00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.695 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 --2- 172.21.15.122:0/3336752928 >> v2:172.21.15.169:3300/0 conn(0x7f19c00d2710 0x7f19c00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 >> v1:172.21.15.169:6789/0 conn(0x7f19c00d22b0 legacy=0x7f19c00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00b39f0 con 0x7f19c015dfe0 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f19bc001670 con 0x7f19c015dfe0 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 >> v1:172.21.15.122:6789/0 conn(0x7f19c015dfe0 legacy=0x7f19c00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19ac0039e0 0x7f19ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19ac004800 0x7f19ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00b31e0 con 0x7f19ac0051c0 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac00c090 con 0x7f19ac004800 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac00c290 con 0x7f19ac0039e0 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c6d27640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c5d25640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19ac004800 0x7f19ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.691+0000 7f19c6526640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19ac0039e0 0x7f19ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.692+0000 7f19c6d27640 1 -- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19ac0039e0 msgr2=0x7f19ac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.692+0000 7f19c6d27640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19ac0039e0 0x7f19ac003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.692+0000 7f19c6d27640 1 -- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19ac004800 msgr2=0x7f19ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.692+0000 7f19c6d27640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19ac004800 0x7f19ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.692+0000 7f19c6d27640 1 -- 172.21.15.122:0/3336752928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac003520 con 0x7f19ac0051c0 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.692+0000 7f19c6526640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19ac0039e0 0x7f19ac003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c6d27640 1 -- 172.21.15.122:0/3336752928 reap_dead start 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c6d27640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19ac00b9c0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009fc0 tx=0x7f19ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4104560f034bd6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8009c10 con 0x7f19ac0051c0 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b802cca0 con 0x7f19ac0051c0 2024-03-06T07:32:18.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c5524640 1 -- 172.21.15.122:0/3336752928 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8003bd0 con 0x7f19ac0051c0 2024-03-06T07:32:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 -- 172.21.15.122:0/3336752928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 msgr2=0x7f19ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19ac00b9c0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009fc0 tx=0x7f19ac005830 comp rx=0 tx=0).stop 2024-03-06T07:32:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 -- 172.21.15.122:0/3336752928 shutdown_connections 2024-03-06T07:32:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 --2- 172.21.15.122:0/3336752928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19ac00b9c0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.701 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 -- 172.21.15.122:0/3336752928 >> 172.21.15.122:0/3336752928 conn(0x7f19c00d3cd0 msgr2=0x7f19c00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 -- 172.21.15.122:0/3336752928 shutdown_connections 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.693+0000 7f19c878a640 1 -- 172.21.15.122:0/3336752928 wait complete. 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 Processor -- start 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 -- start start 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19c0156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c015fc10 0x7f19c0160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19c0159880 0x7f19c0159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00d1c70 con 0x7f19ac0051c0 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00b3d60 con 0x7f19c015fc10 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c878a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00d1df0 con 0x7f19c0159880 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c6526640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19c0156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c6526640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19c0156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38788/0 (socket says 172.21.15.122:38788) 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.695+0000 7f19c6526640 1 -- 172.21.15.122:0/2007901434 learned_addr learned my addr 172.21.15.122:0/2007901434 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6d27640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19c0159880 0x7f19c0159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c5d25640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c015fc10 0x7f19c0160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6526640 1 -- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19c0159880 msgr2=0x7f19c0159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6526640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19c0159880 0x7f19c0159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6526640 1 -- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c015fc10 msgr2=0x7f19c0160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6526640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c015fc10 0x7f19c0160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6526640 1 -- 172.21.15.122:0/2007901434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c01f7360 con 0x7f19ac0051c0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c6526640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19c0156750 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00e830 tx=0x7f19bc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00c6c0 con 0x7f19ac0051c0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc018070 con 0x7f19ac0051c0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c01f7640 con 0x7f19ac0051c0 2024-03-06T07:32:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00ca80 con 0x7f19ac0051c0 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.696+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c01f7c00 con 0x7f19ac0051c0 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.697+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c004f330 con 0x7f19ac0051c0 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.698+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19bc01d460 con 0x7f19ac0051c0 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.698+0000 7f19b2ffd640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19a406b4d0 0x7f19a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.698+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f19bc08a200 con 0x7f19ac0051c0 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.698+0000 7f19c5d25640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19a406b4d0 0x7f19a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.698+0000 7f19c5d25640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19a406b4d0 0x7f19a406d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f19b4000c00 tx=0x7f19b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.706+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19bc059060 con 0x7f19ac0051c0 2024-03-06T07:32:18.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.933+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19c00b2a40 con 0x7f19ac0051c0 2024-03-06T07:32:18.936 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.934+0000 7f19b2ffd640 1 -- 172.21.15.122:0/2007901434 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f19bc058960 con 0x7f19ac0051c0 2024-03-06T07:32:18.936 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:18.936 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:18.936 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19a406b4d0 msgr2=0x7f19a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19a406b4d0 0x7f19a406d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f19b4000c00 tx=0x7f19b4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 msgr2=0x7f19c0156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19c0156750 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00e830 tx=0x7f19bc00ecf0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 shutdown_connections 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f19a406b4d0 0x7f19a406d990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f19c0159880 0x7f19c0159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19c015fc10 0x7f19c0160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 --2- 172.21.15.122:0/2007901434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0051c0 0x7f19c0156750 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 >> 172.21.15.122:0/2007901434 conn(0x7f19c00d3cd0 msgr2=0x7f19c00b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 shutdown_connections 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:18.936+0000 7f19c878a640 1 -- 172.21.15.122:0/2007901434 wait complete. 2024-03-06T07:32:19.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:19.150 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:20.151 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:20.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.267+0000 7f86097eb640 1 Processor -- start 2024-03-06T07:32:20.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.267+0000 7f86097eb640 1 -- start start 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86097eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f86040d00c0 0x7f86040d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86097eb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f86040cf850 0x7f86040cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86097eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040ce1b0 0x7f86040ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86097eb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860406ca90 con 0x7f86040cf850 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86097eb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040d09e0 con 0x7f86040d00c0 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86097eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860415e380 con 0x7f86040ce1b0 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f86040cf850 0x7f86040cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f86040cf850 0x7f86040cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38796/0 (socket says 172.21.15.122:38796) 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 -- 172.21.15.122:0/2819087732 learned_addr learned my addr 172.21.15.122:0/2819087732 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86037fe640 1 --2- 172.21.15.122:0/2819087732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040ce1b0 0x7f86040ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f8602ffd640 1 --2- 172.21.15.122:0/2819087732 >> v2:172.21.15.169:3300/0 conn(0x7f86040d00c0 0x7f86040d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 -- 172.21.15.122:0/2819087732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040ce1b0 msgr2=0x7f86040ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 --2- 172.21.15.122:0/2819087732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040ce1b0 0x7f86040ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 -- 172.21.15.122:0/2819087732 >> v2:172.21.15.169:3300/0 conn(0x7f86040d00c0 msgr2=0x7f86040d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 --2- 172.21.15.122:0/2819087732 >> v2:172.21.15.169:3300/0 conn(0x7f86040d00c0 0x7f86040d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f86027fc640 1 -- 172.21.15.122:0/2819087732 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040cdc10 con 0x7f86040cf850 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.268+0000 7f8602ffd640 1 --2- 172.21.15.122:0/2819087732 >> v2:172.21.15.169:3300/0 conn(0x7f86040d00c0 0x7f86040d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.269+0000 7f86027fc640 1 --2- 172.21.15.122:0/2819087732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040cf850 0x7f86040cd3c0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f86040d1a40 tx=0x7f85ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6006545f7f02ab33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.269+0000 7f8601ffb640 1 -- 172.21.15.122:0/2819087732 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec035020 con 0x7f86040cf850 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.269+0000 7f8601ffb640 1 -- 172.21.15.122:0/2819087732 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec031870 con 0x7f86040cf850 2024-03-06T07:32:20.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.269+0000 7f8601ffb640 1 -- 172.21.15.122:0/2819087732 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec031c40 con 0x7f86040cf850 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.269+0000 7f86097eb640 1 -- 172.21.15.122:0/2819087732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040cf850 msgr2=0x7f86040cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.269+0000 7f86097eb640 1 --2- 172.21.15.122:0/2819087732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040cf850 0x7f86040cd3c0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f86040d1a40 tx=0x7f85ec02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 -- 172.21.15.122:0/2819087732 shutdown_connections 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 --2- 172.21.15.122:0/2819087732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040cf850 0x7f86040cd3c0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 --2- 172.21.15.122:0/2819087732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040ce1b0 0x7f86040ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 --2- 172.21.15.122:0/2819087732 >> v2:172.21.15.169:3300/0 conn(0x7f86040d00c0 0x7f86040d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 -- 172.21.15.122:0/2819087732 >> 172.21.15.122:0/2819087732 conn(0x7f86040b35a0 msgr2=0x7f86040b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 -- 172.21.15.122:0/2819087732 shutdown_connections 2024-03-06T07:32:20.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.270+0000 7f86097eb640 1 -- 172.21.15.122:0/2819087732 wait complete. 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.271+0000 7f86097eb640 1 Processor -- start 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.271+0000 7f86097eb640 1 -- start start 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 0x7f860415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86040cf850 0x7f860415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040d00c0 0x7f86040af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860406c720 con 0x7f86040ce1b0 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860406c440 con 0x7f86040cf850 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86040cd900 con 0x7f86040d00c0 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 0x7f860415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 0x7f860415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38812/0 (socket says 172.21.15.122:38812) 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 -- 172.21.15.122:0/3794838803 learned_addr learned my addr 172.21.15.122:0/3794838803 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86037fe640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040d00c0 0x7f86040af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86027fc640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86040cf850 0x7f860415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 -- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040d00c0 msgr2=0x7f86040af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040d00c0 0x7f86040af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 -- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86040cf850 msgr2=0x7f860415a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86040cf850 0x7f860415a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 -- 172.21.15.122:0/3794838803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860415a7f0 con 0x7f86040ce1b0 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f8602ffd640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 0x7f860415e5c0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00ce00 tx=0x7f85fc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00f890 con 0x7f86040ce1b0 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc00fa30 con 0x7f86040ce1b0 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8604161260 con 0x7f86040ce1b0 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00fe00 con 0x7f86040ce1b0 2024-03-06T07:32:20.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.272+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86040b6020 con 0x7f86040ce1b0 2024-03-06T07:32:20.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.273+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86040b5b30 con 0x7f86040ce1b0 2024-03-06T07:32:20.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.274+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85fc018400 con 0x7f86040ce1b0 2024-03-06T07:32:20.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.275+0000 7f85fb7fe640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f85e406b4d0 0x7f85e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.275+0000 7f86027fc640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f85e406b4d0 0x7f85e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.275+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f85fc08b0e0 con 0x7f86040ce1b0 2024-03-06T07:32:20.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.275+0000 7f86027fc640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f85e406b4d0 0x7f85e406d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f85ec036000 tx=0x7f85ec03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.279+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85fc014030 con 0x7f86040ce1b0 2024-03-06T07:32:20.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.528+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860404f330 con 0x7f86040ce1b0 2024-03-06T07:32:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.528+0000 7f85fb7fe640 1 -- 172.21.15.122:0/3794838803 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f85fc00ae20 con 0x7f86040ce1b0 2024-03-06T07:32:20.530 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:20.531 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:20.531 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f85e406b4d0 msgr2=0x7f85e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f85e406b4d0 0x7f85e406d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f85ec036000 tx=0x7f85ec03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 msgr2=0x7f860415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 0x7f860415e5c0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00ce00 tx=0x7f85fc00a650 comp rx=0 tx=0).stop 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 shutdown_connections 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f85e406b4d0 0x7f85e406d990 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86040d00c0 0x7f86040af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86040cf850 0x7f860415a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 --2- 172.21.15.122:0/3794838803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040ce1b0 0x7f860415e5c0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 >> 172.21.15.122:0/3794838803 conn(0x7f86040b35a0 msgr2=0x7f86040d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 shutdown_connections 2024-03-06T07:32:20.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:20.530+0000 7f86097eb640 1 -- 172.21.15.122:0/3794838803 wait complete. 2024-03-06T07:32:20.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:20.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:21.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:21.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.849+0000 7fd8b3535640 1 Processor -- start 2024-03-06T07:32:21.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.849+0000 7fd8b3535640 1 -- start start 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b3535640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 0x7fd8ac0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b3535640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ac0cebe0 con 0x7fd8ac0d16e0 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b3535640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ac0ceec0 con 0x7fd8ac15fa10 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b3535640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0ce980 con 0x7fd8ac15bdd0 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b1ad2640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd8ac0d16e0 0x7fd8ac0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40224/0 (socket says 172.21.15.122:40224) 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b1ad2640 1 -- 172.21.15.122:0/2377940215 learned_addr learned my addr 172.21.15.122:0/2377940215 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138525592 0 0) 0x7fd8ac0cebe0 con 0x7fd8ac0d16e0 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd898003520 con 0x7fd8ac0d16e0 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8b12d1640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 0x7fd8ac0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3563734871 0 0) 0x7fd898003520 con 0x7fd8ac0d16e0 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 msgr2=0x7fd8ac0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 0x7fd8ac0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 >> v1:172.21.15.169:6789/0 conn(0x7fd8ac15fa10 legacy=0x7fd8ac0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac0cf1a0 con 0x7fd8ac0d16e0 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.850+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fd8a40019f0 con 0x7fd8ac0d16e0 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 >> v1:172.21.15.122:6789/0 conn(0x7fd8ac0d16e0 legacy=0x7fd8ac0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8980039e0 0x7fd898003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 0x7fd898004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd898004d40 0x7fd8980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0ce980 con 0x7fd898004d40 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800baf0 con 0x7fd8980039e0 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800bcf0 con 0x7fd8ac15bdd0 2024-03-06T07:32:21.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b1ad2640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd898004d40 0x7fd8980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b12d1640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8980039e0 0x7fd898003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b0ad0640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 0x7fd898004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b1ad2640 1 -- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 msgr2=0x7fd898004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b1ad2640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac15bdd0 0x7fd898004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b1ad2640 1 -- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8980039e0 msgr2=0x7fd898003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b1ad2640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8980039e0 0x7fd898003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.851+0000 7fd8b1ad2640 1 -- 172.21.15.122:0/2377940215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898003520 con 0x7fd898004d40 2024-03-06T07:32:21.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.852+0000 7fd8b1ad2640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd898004d40 0x7fd8980051c0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400a8d0 tx=0x7fd898005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db65caee824825fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.852+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400a4a0 con 0x7fd898004d40 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.852+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a4034650 con 0x7fd898004d40 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.852+0000 7fd8a3fff640 1 -- 172.21.15.122:0/2377940215 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4034a40 con 0x7fd898004d40 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b3535640 1 -- 172.21.15.122:0/2377940215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd898004d40 msgr2=0x7fd8980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b3535640 1 --2- 172.21.15.122:0/2377940215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd898004d40 0x7fd8980051c0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400a8d0 tx=0x7fd898005830 comp rx=0 tx=0).stop 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b1ad2640 1 -- 172.21.15.122:0/2377940215 reap_dead start 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b3535640 1 -- 172.21.15.122:0/2377940215 shutdown_connections 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b3535640 1 -- 172.21.15.122:0/2377940215 >> 172.21.15.122:0/2377940215 conn(0x7fd8ac09a9f0 msgr2=0x7fd8ac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b3535640 1 -- 172.21.15.122:0/2377940215 shutdown_connections 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.853+0000 7fd8b3535640 1 -- 172.21.15.122:0/2377940215 wait complete. 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.854+0000 7fd8b3535640 1 Processor -- start 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.854+0000 7fd8b3535640 1 -- start start 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b3535640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 0x7fd8ac0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b3535640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac0b2d00 0x7fd8ac06f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b3535640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8ac06c0c0 0x7fd8ac06c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b3535640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac15c1b0 con 0x7fd8ac06f340 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b3535640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0cc5d0 con 0x7fd8ac06c0c0 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b3535640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0c5610 con 0x7fd8ac0b2d00 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b12d1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 0x7fd8ac0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b12d1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 0x7fd8ac0b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38826/0 (socket says 172.21.15.122:38826) 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b12d1640 1 -- 172.21.15.122:0/1582968339 learned_addr learned my addr 172.21.15.122:0/1582968339 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b0ad0640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac0b2d00 0x7fd8ac06f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b1ad2640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8ac06c0c0 0x7fd8ac06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.855+0000 7fd8b12d1640 1 -- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac0b2d00 msgr2=0x7fd8ac06f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b12d1640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac0b2d00 0x7fd8ac06f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b12d1640 1 -- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8ac06c0c0 msgr2=0x7fd8ac06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b12d1640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8ac06c0c0 0x7fd8ac06c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b12d1640 1 -- 172.21.15.122:0/1582968339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac06e0f0 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b0ad0640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac0b2d00 0x7fd8ac06f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b12d1640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 0x7fd8ac0b3650 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fd8a80119d0 tx=0x7fd8a8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a800fb50 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a800fcf0 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac06fc60 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a800fea0 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.856+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac15e110 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.857+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac15e350 con 0x7fd8ac06f340 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.858+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8a8012040 con 0x7fd8ac06f340 2024-03-06T07:32:21.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.858+0000 7fd8a1ffb640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd89006b500 0x7fd89006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.858+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd8a808a760 con 0x7fd8ac06f340 2024-03-06T07:32:21.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.858+0000 7fd8b0ad0640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd89006b500 0x7fd89006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.858+0000 7fd8b0ad0640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd89006b500 0x7fd89006d9c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac15e920 tx=0x7fd89c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:21.862+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8ac06e0f0 con 0x7fd8ac06f340 2024-03-06T07:32:22.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.077+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8ac04f330 con 0x7fd8ac06f340 2024-03-06T07:32:22.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.078+0000 7fd8a1ffb640 1 -- 172.21.15.122:0/1582968339 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fd8a8019030 con 0x7fd8ac06f340 2024-03-06T07:32:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:22.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd89006b500 msgr2=0x7fd89006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd89006b500 0x7fd89006d9c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac15e920 tx=0x7fd89c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 msgr2=0x7fd8ac0b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 0x7fd8ac0b3650 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fd8a80119d0 tx=0x7fd8a8011e90 comp rx=0 tx=0).stop 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 shutdown_connections 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd89006b500 0x7fd89006d9c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8ac06c0c0 0x7fd8ac06c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd8ac0b2d00 0x7fd8ac06f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 --2- 172.21.15.122:0/1582968339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8ac06f340 0x7fd8ac0b3650 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 >> 172.21.15.122:0/1582968339 conn(0x7fd8ac09a9f0 msgr2=0x7fd8ac0c6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 shutdown_connections 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:22.080+0000 7fd8b3535640 1 -- 172.21.15.122:0/1582968339 wait complete. 2024-03-06T07:32:22.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:22.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:23.295 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:23.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.412+0000 7fc55ff7a640 1 Processor -- start 2024-03-06T07:32:23.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.413+0000 7fc55ff7a640 1 -- start start 2024-03-06T07:32:23.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.413+0000 7fc55ff7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc55815dfe0 0x7fc5580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.413+0000 7fc55ff7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 0x7fc5580b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.413+0000 7fc55ff7a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580b3430 con 0x7fc5580cf830 2024-03-06T07:32:23.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.413+0000 7fc55ff7a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580b31e0 con 0x7fc55815dfe0 2024-03-06T07:32:23.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55ff7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580b3d60 con 0x7fc5580d29c0 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc55815dfe0 0x7fc5580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55e517640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 0x7fc5580b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc55815dfe0 0x7fc5580cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:49482/0 (socket says 172.21.15.122:49482) 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 -- 172.21.15.122:0/2973771342 learned_addr learned my addr 172.21.15.122:0/2973771342 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55e517640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 0x7fc5580b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:47986/0 (socket says 172.21.15.122:47986) 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55d515640 1 --1- 172.21.15.122:0/2973771342 >> v1:172.21.15.169:6789/0 conn(0x7fc5580cf830 0x7fc5580d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:37914/0 (socket says 172.21.15.122:37914) 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 -- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 msgr2=0x7fc5580b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 --2- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 0x7fc5580b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 -- 172.21.15.122:0/2973771342 >> v1:172.21.15.169:6789/0 conn(0x7fc5580cf830 legacy=0x7fc5580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55dd16640 1 -- 172.21.15.122:0/2973771342 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580b3710 con 0x7fc55815dfe0 2024-03-06T07:32:23.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.414+0000 7fc55e517640 1 --2- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 0x7fc5580b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:23.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.415+0000 7fc55dd16640 1 --2- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cf2f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc548009870 tx=0x7fc54802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd0d58743a22a269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.415+0000 7fc55cd14640 1 -- 172.21.15.122:0/2973771342 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548034070 con 0x7fc55815dfe0 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.415+0000 7fc55cd14640 1 -- 172.21.15.122:0/2973771342 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc548030840 con 0x7fc55815dfe0 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.415+0000 7fc55cd14640 1 -- 172.21.15.122:0/2973771342 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548030c10 con 0x7fc55815dfe0 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 -- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 msgr2=0x7fc5580cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cf2f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc548009870 tx=0x7fc54802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 -- 172.21.15.122:0/2973771342 shutdown_connections 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580d29c0 0x7fc5580b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/2973771342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cf2f0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 -- 172.21.15.122:0/2973771342 >> 172.21.15.122:0/2973771342 conn(0x7fc5580d3cd0 msgr2=0x7fc5580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 -- 172.21.15.122:0/2973771342 shutdown_connections 2024-03-06T07:32:23.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.416+0000 7fc55ff7a640 1 -- 172.21.15.122:0/2973771342 wait complete. 2024-03-06T07:32:23.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.418+0000 7fc55ff7a640 1 Processor -- start 2024-03-06T07:32:23.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.418+0000 7fc55ff7a640 1 -- start start 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55ff7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580cf830 0x7fc5580ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55ff7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 0x7fc5581f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55ff7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55ff7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580d1df0 con 0x7fc5580d29c0 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55ff7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580d1c70 con 0x7fc55815dfe0 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55ff7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55815e3c0 con 0x7fc5580cf830 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 0x7fc5581f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 0x7fc5581f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:38836/0 (socket says 172.21.15.122:38836) 2024-03-06T07:32:23.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 -- 172.21.15.122:0/1539808247 learned_addr learned my addr 172.21.15.122:0/1539808247 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55dd16640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580cf830 0x7fc5580ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55e517640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 -- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580cf830 msgr2=0x7fc5580ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580cf830 0x7fc5580ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 -- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 msgr2=0x7fc5580cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 -- 172.21.15.122:0/1539808247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5581f3b90 con 0x7fc5580d29c0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55dd16640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580cf830 0x7fc5580ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55e517640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.419+0000 7fc55d515640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 0x7fc5581f6f90 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c440 tx=0x7fc554001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.420+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5540074e0 con 0x7fc5580d29c0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.420+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc554007680 con 0x7fc5580d29c0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.420+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5581f7670 con 0x7fc5580d29c0 2024-03-06T07:32:23.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.420+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554004a20 con 0x7fc5580d29c0 2024-03-06T07:32:23.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.420+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5581f7c30 con 0x7fc5580d29c0 2024-03-06T07:32:23.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.421+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5581f3d60 con 0x7fc5580d29c0 2024-03-06T07:32:23.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.421+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc55401a020 con 0x7fc5580d29c0 2024-03-06T07:32:23.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.421+0000 7fc54e7fc640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc53c06b4d0 0x7fc53c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.421+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc5540899b0 con 0x7fc5580d29c0 2024-03-06T07:32:23.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.421+0000 7fc55dd16640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc53c06b4d0 0x7fc53c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.422+0000 7fc55dd16640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc53c06b4d0 0x7fc53c06d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc548036000 tx=0x7fc54803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.430+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc554010300 con 0x7fc5580d29c0 2024-03-06T07:32:23.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.643+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc55804f330 con 0x7fc5580d29c0 2024-03-06T07:32:23.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.643+0000 7fc54e7fc640 1 -- 172.21.15.122:0/1539808247 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fc5540588a0 con 0x7fc5580d29c0 2024-03-06T07:32:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:23.646 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:23.646 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:23.647 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc53c06b4d0 msgr2=0x7fc53c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc53c06b4d0 0x7fc53c06d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc548036000 tx=0x7fc54803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 msgr2=0x7fc5581f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 0x7fc5581f6f90 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c440 tx=0x7fc554001e70 comp rx=0 tx=0).stop 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 shutdown_connections 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc53c06b4d0 0x7fc53c06d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc55815dfe0 0x7fc5580cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5580d29c0 0x7fc5581f6f90 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 --2- 172.21.15.122:0/1539808247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc5580cf830 0x7fc5580ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 >> 172.21.15.122:0/1539808247 conn(0x7fc5580d3cd0 msgr2=0x7fc55806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 shutdown_connections 2024-03-06T07:32:23.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:23.645+0000 7fc55ff7a640 1 -- 172.21.15.122:0/1539808247 wait complete. 2024-03-06T07:32:23.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:23.859 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:24.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.959+0000 7f8aec81c640 1 Processor -- start 2024-03-06T07:32:24.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.959+0000 7f8aec81c640 1 -- start start 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8aec81c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f8ae80cce40 0x7f8ae815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8aec81c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 0x7f8ae80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8aec81c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae8071f40 con 0x7f8ae81515e0 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8aec81c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae80cd2f0 con 0x7f8ae80cce40 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8aec81c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8070a70 con 0x7f8ae80cd7a0 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8ae659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8ae81515e0 0x7f8ae8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56718/0 (socket says 172.21.15.122:56718) 2024-03-06T07:32:24.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.960+0000 7f8ae659c640 1 -- 172.21.15.122:0/3095481788 learned_addr learned my addr 172.21.15.122:0/3095481788 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186350185 0 0) 0x7f8ae8071f40 con 0x7f8ae81515e0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae6d9d640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 0x7f8ae80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0003520 con 0x7f8ae81515e0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae5d9b640 1 --2- 172.21.15.122:0/3095481788 >> v2:172.21.15.169:3300/0 conn(0x7f8ae80cce40 0x7f8ae815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3668867720 0 0) 0x7f8ad0003520 con 0x7f8ae81515e0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 msgr2=0x7f8ae80cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 0x7f8ae80cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 >> v2:172.21.15.169:3300/0 conn(0x7f8ae80cce40 msgr2=0x7f8ae815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 --2- 172.21.15.122:0/3095481788 >> v2:172.21.15.169:3300/0 conn(0x7f8ae80cce40 0x7f8ae815b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae8070d50 con 0x7f8ae81515e0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f8adc001690 con 0x7f8ae81515e0 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 >> v1:172.21.15.122:6789/0 conn(0x7f8ae81515e0 legacy=0x7f8ae8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad00039b0 0x7f8ad0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 0x7f8ad0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.966 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ad0004d40 0x7f8ad00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80cd2f0 con 0x7f8ad00039b0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8070a70 con 0x7f8ad0004d40 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad000b9c0 con 0x7f8ae80cd7a0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae659c640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad00039b0 0x7f8ad0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae5d9b640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 0x7f8ad0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.961+0000 7f8ae6d9d640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ad0004d40 0x7f8ad00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae659c640 1 -- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 msgr2=0x7f8ad0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae659c640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ae80cd7a0 0x7f8ad0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae659c640 1 -- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ad0004d40 msgr2=0x7f8ad00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae659c640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ad0004d40 0x7f8ad00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae659c640 1 -- 172.21.15.122:0/3095481788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0003520 con 0x7f8ad00039b0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae659c640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad00039b0 0x7f8ad0003dd0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f8adc007d20 tx=0x7f8ad0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df83b7d50331d3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc007d60 con 0x7f8ad00039b0 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc032070 con 0x7f8ad00039b0 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.962+0000 7f8ae559a640 1 -- 172.21.15.122:0/3095481788 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0058d0 con 0x7f8ad00039b0 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8aec81c640 1 -- 172.21.15.122:0/3095481788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad00039b0 msgr2=0x7f8ad0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8aec81c640 1 --2- 172.21.15.122:0/3095481788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad00039b0 0x7f8ad0003dd0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f8adc007d20 tx=0x7f8ad0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8ae6d9d640 1 -- 172.21.15.122:0/3095481788 reap_dead start 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8aec81c640 1 -- 172.21.15.122:0/3095481788 shutdown_connections 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8aec81c640 1 -- 172.21.15.122:0/3095481788 >> 172.21.15.122:0/3095481788 conn(0x7f8ae80cf8a0 msgr2=0x7f8ae80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8aec81c640 1 -- 172.21.15.122:0/3095481788 shutdown_connections 2024-03-06T07:32:24.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.963+0000 7f8aec81c640 1 -- 172.21.15.122:0/3095481788 wait complete. 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.964+0000 7f8aec81c640 1 Processor -- start 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.964+0000 7f8aec81c640 1 -- start start 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8aec81c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ad00039b0 0x7f8ae814da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8aec81c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ae8143b20 0x7f8ae806de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8aec81c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 0x7f8ae81433b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8aec81c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80cdbb0 con 0x7f8ae8142f30 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8aec81c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8071310 con 0x7f8ae8143b20 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8aec81c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80cdd30 con 0x7f8ad00039b0 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 0x7f8ae81433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 0x7f8ae81433b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59682/0 (socket says 172.21.15.122:59682) 2024-03-06T07:32:24.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 -- 172.21.15.122:0/2560064711 learned_addr learned my addr 172.21.15.122:0/2560064711 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae659c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ad00039b0 0x7f8ae814da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae5d9b640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ae8143b20 0x7f8ae806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 -- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ad00039b0 msgr2=0x7f8ae814da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ad00039b0 0x7f8ae814da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 -- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ae8143b20 msgr2=0x7f8ae806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ae8143b20 0x7f8ae806de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.965+0000 7f8ae6d9d640 1 -- 172.21.15.122:0/2560064711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae81438f0 con 0x7f8ae8142f30 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.966+0000 7f8ae6d9d640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 0x7f8ae81433b0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f8ae80731f0 tx=0x7f8ae000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.966+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae000fde0 con 0x7f8ae8142f30 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.966+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae001d070 con 0x7f8ae8142f30 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.966+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae8143fb0 con 0x7f8ae8142f30 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.966+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0015670 con 0x7f8ae8142f30 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.966+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae8144570 con 0x7f8ae8142f30 2024-03-06T07:32:24.972 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.967+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae8068840 con 0x7f8ae8142f30 2024-03-06T07:32:24.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.967+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ae0015810 con 0x7f8ae8142f30 2024-03-06T07:32:24.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.967+0000 7f8ad6ffd640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8ac806b4d0 0x7f8ac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.967+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8ae008aba0 con 0x7f8ae8142f30 2024-03-06T07:32:24.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.967+0000 7f8ae659c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8ac806b4d0 0x7f8ac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.968+0000 7f8ae659c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8ac806b4d0 0x7f8ac806d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8adc036000 tx=0x7f8adc007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:24.972+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ae0010040 con 0x7f8ae8142f30 2024-03-06T07:32:25.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.202+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae804f330 con 0x7f8ae8142f30 2024-03-06T07:32:25.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.202+0000 7f8ad6ffd640 1 -- 172.21.15.122:0/2560064711 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f8ae0059a00 con 0x7f8ae8142f30 2024-03-06T07:32:25.204 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:25.205 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:25.205 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8ac806b4d0 msgr2=0x7f8ac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8ac806b4d0 0x7f8ac806d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8adc036000 tx=0x7f8adc007000 comp rx=0 tx=0).stop 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 msgr2=0x7f8ae81433b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 0x7f8ae81433b0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f8ae80731f0 tx=0x7f8ae000f580 comp rx=0 tx=0).stop 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 shutdown_connections 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8ac806b4d0 0x7f8ac806d990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ae8142f30 0x7f8ae81433b0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ae8143b20 0x7f8ae806de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 --2- 172.21.15.122:0/2560064711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8ad00039b0 0x7f8ae814da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 >> 172.21.15.122:0/2560064711 conn(0x7f8ae80cf8a0 msgr2=0x7f8ae81475f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 shutdown_connections 2024-03-06T07:32:25.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:25.204+0000 7f8aec81c640 1 -- 172.21.15.122:0/2560064711 wait complete. 2024-03-06T07:32:25.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:25.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:25.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:25.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:26.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 Processor -- start 2024-03-06T07:32:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 -- start start 2024-03-06T07:32:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7f041424f0 0x7f7f040cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f040cd7a0 0x7f7f0415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f040cc980 con 0x7f7f0406d330 2024-03-06T07:32:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040cd220 con 0x7f7f040cd7a0 2024-03-06T07:32:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.519+0000 7f7f090ed640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040ccc60 con 0x7f7f041424f0 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f040cd7a0 0x7f7f0415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7f040cd7a0 0x7f7f0415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59698/0 (socket says 172.21.15.122:59698) 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 -- 172.21.15.122:0/595644785 learned_addr learned my addr 172.21.15.122:0/595644785 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 -- 172.21.15.122:0/595644785 >> v2:172.21.15.169:3300/0 conn(0x7f7f041424f0 msgr2=0x7f7f040cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 --2- 172.21.15.122:0/595644785 >> v2:172.21.15.169:3300/0 conn(0x7f7f041424f0 0x7f7f040cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 -- 172.21.15.122:0/595644785 >> v1:172.21.15.169:6789/0 conn(0x7f7f0406d330 legacy=0x7f7f04071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.520+0000 7f7f0259c640 1 -- 172.21.15.122:0/595644785 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040ccf40 con 0x7f7f040cd7a0 2024-03-06T07:32:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f0259c640 1 --2- 172.21.15.122:0/595644785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f0415cef0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f7f0406c6d0 tx=0x7f7ef802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7259d943085537b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f01d9b640 1 -- 172.21.15.122:0/595644785 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef802ca80 con 0x7f7f040cd7a0 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f01d9b640 1 -- 172.21.15.122:0/595644785 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef802cc20 con 0x7f7f040cd7a0 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f01d9b640 1 -- 172.21.15.122:0/595644785 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80343f0 con 0x7f7f040cd7a0 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 -- 172.21.15.122:0/595644785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 msgr2=0x7f7f0415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 --2- 172.21.15.122:0/595644785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f0415cef0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f7f0406c6d0 tx=0x7f7ef802bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 -- 172.21.15.122:0/595644785 shutdown_connections 2024-03-06T07:32:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 --2- 172.21.15.122:0/595644785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f0415cef0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 --2- 172.21.15.122:0/595644785 >> v2:172.21.15.169:3300/0 conn(0x7f7f041424f0 0x7f7f040cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 -- 172.21.15.122:0/595644785 >> 172.21.15.122:0/595644785 conn(0x7f7f040cf8a0 msgr2=0x7f7f040cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 -- 172.21.15.122:0/595644785 shutdown_connections 2024-03-06T07:32:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.521+0000 7f7f090ed640 1 -- 172.21.15.122:0/595644785 wait complete. 2024-03-06T07:32:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.523+0000 7f7f090ed640 1 Processor -- start 2024-03-06T07:32:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.523+0000 7f7f090ed640 1 -- start start 2024-03-06T07:32:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f0406d330 0x7f7f0413f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f04153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f041424f0 0x7f7f04155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0406d920 con 0x7f7f040cd7a0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0406d7a0 con 0x7f7f0406d330 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0406daa0 con 0x7f7f041424f0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f04153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f04153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59708/0 (socket says 172.21.15.122:59708) 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 -- 172.21.15.122:0/3559328324 learned_addr learned my addr 172.21.15.122:0/3559328324 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0359e640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f041424f0 0x7f7f04155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f02d9d640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f0406d330 0x7f7f0413f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 -- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f041424f0 msgr2=0x7f7f04155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f041424f0 0x7f7f04155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 -- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f0406d330 msgr2=0x7f7f0413f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f0406d330 0x7f7f0413f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 -- 172.21.15.122:0/3559328324 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f04153e30 con 0x7f7f040cd7a0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f0259c640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f04153450 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80093e0 tx=0x7f7ef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8043020 con 0x7f7f040cd7a0 2024-03-06T07:32:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef803e450 con 0x7f7f040cd7a0 2024-03-06T07:32:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef803e820 con 0x7f7f040cd7a0 2024-03-06T07:32:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f04151810 con 0x7f7f040cd7a0 2024-03-06T07:32:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.525+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0413f620 con 0x7f7f040cd7a0 2024-03-06T07:32:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.527+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0404f330 con 0x7f7f040cd7a0 2024-03-06T07:32:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.527+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ef8005d10 con 0x7f7f040cd7a0 2024-03-06T07:32:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.527+0000 7f7eeb7fe640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.527+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7ef80acf40 con 0x7f7f040cd7a0 2024-03-06T07:32:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.528+0000 7f7f02d9d640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.528+0000 7f7f02d9d640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ef006b4d0 0x7f7ef006d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f7f040d0570 tx=0x7f7efc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.532+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ef807bda0 con 0x7f7f040cd7a0 2024-03-06T07:32:26.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.897+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f040d0620 con 0x7f7f040cd7a0 2024-03-06T07:32:26.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.897+0000 7f7eeb7fe640 1 -- 172.21.15.122:0/3559328324 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f7ef807b6a0 con 0x7f7f040cd7a0 2024-03-06T07:32:26.899 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:26.899 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:26.900 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ef006b4d0 msgr2=0x7f7ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ef006b4d0 0x7f7ef006d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f7f040d0570 tx=0x7f7efc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 msgr2=0x7f7f04153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f04153450 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80093e0 tx=0x7f7ef8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 shutdown_connections 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7f041424f0 0x7f7f04155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f040cd7a0 0x7f7f04153450 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 --2- 172.21.15.122:0/3559328324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f0406d330 0x7f7f0413f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 >> 172.21.15.122:0/3559328324 conn(0x7f7f040cf8a0 msgr2=0x7f7f0415d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 shutdown_connections 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:26.899+0000 7f7f090ed640 1 -- 172.21.15.122:0/3559328324 wait complete. 2024-03-06T07:32:27.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:27.114 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:28.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:28.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.214+0000 7fe61cb8a640 1 Processor -- start 2024-03-06T07:32:28.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.214+0000 7fe61cb8a640 1 -- start start 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61cb8a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe61815dfe0 0x7fe6180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61cb8a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe6180cf830 0x7fe6180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61cb8a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61cb8a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6180b31e0 con 0x7fe61815dfe0 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61cb8a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6180b3d60 con 0x7fe6180cf830 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61cb8a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180b3430 con 0x7fe6180d27f0 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe61815dfe0 0x7fe6180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61659c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe61815dfe0 0x7fe6180cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59712/0 (socket says 172.21.15.122:59712) 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.215+0000 7fe61659c640 1 -- 172.21.15.122:0/3556528183 learned_addr learned my addr 172.21.15.122:0/3556528183 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe61659c640 1 -- 172.21.15.122:0/3556528183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 msgr2=0x7fe6180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe616d9d640 1 --2- 172.21.15.122:0/3556528183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe60ffff640 1 --2- 172.21.15.122:0/3556528183 >> v2:172.21.15.169:3300/0 conn(0x7fe6180cf830 0x7fe6180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe61659c640 1 --2- 172.21.15.122:0/3556528183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe61659c640 1 -- 172.21.15.122:0/3556528183 >> v2:172.21.15.169:3300/0 conn(0x7fe6180cf830 msgr2=0x7fe6180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe61659c640 1 --2- 172.21.15.122:0/3556528183 >> v2:172.21.15.169:3300/0 conn(0x7fe6180cf830 0x7fe6180d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe61659c640 1 -- 172.21.15.122:0/3556528183 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180b3710 con 0x7fe61815dfe0 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.216+0000 7fe61659c640 1 --2- 172.21.15.122:0/3556528183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180cf2f0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fe6180d16b0 tx=0x7fe60802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2081c821d18c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe615d9b640 1 -- 172.21.15.122:0/3556528183 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608035020 con 0x7fe61815dfe0 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe615d9b640 1 -- 172.21.15.122:0/3556528183 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe608031870 con 0x7fe61815dfe0 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe615d9b640 1 -- 172.21.15.122:0/3556528183 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608031c40 con 0x7fe61815dfe0 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 -- 172.21.15.122:0/3556528183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 msgr2=0x7fe6180cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/3556528183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180cf2f0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fe6180d16b0 tx=0x7fe60802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 -- 172.21.15.122:0/3556528183 shutdown_connections 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/3556528183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/3556528183 >> v2:172.21.15.169:3300/0 conn(0x7fe6180cf830 0x7fe6180d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/3556528183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180cf2f0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 -- 172.21.15.122:0/3556528183 >> 172.21.15.122:0/3556528183 conn(0x7fe6180d3cd0 msgr2=0x7fe6180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 -- 172.21.15.122:0/3556528183 shutdown_connections 2024-03-06T07:32:28.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.217+0000 7fe61cb8a640 1 -- 172.21.15.122:0/3556528183 wait complete. 2024-03-06T07:32:28.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.218+0000 7fe61cb8a640 1 Processor -- start 2024-03-06T07:32:28.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.218+0000 7fe61cb8a640 1 -- start start 2024-03-06T07:32:28.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe61cb8a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6180cf830 0x7fe6180cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe61cb8a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe61cb8a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe61cb8a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180d1df0 con 0x7fe61815dfe0 2024-03-06T07:32:28.226 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe61cb8a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180d1c70 con 0x7fe6180cf830 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe61cb8a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61815e3c0 con 0x7fe6180d27f0 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe616d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe616d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180b1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59714/0 (socket says 172.21.15.122:59714) 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.219+0000 7fe616d9d640 1 -- 172.21.15.122:0/4241263487 learned_addr learned my addr 172.21.15.122:0/4241263487 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe60ffff640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe61659c640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6180cf830 0x7fe6180cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe616d9d640 1 -- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 msgr2=0x7fe6180a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe616d9d640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe616d9d640 1 -- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6180cf830 msgr2=0x7fe6180cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe616d9d640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6180cf830 0x7fe6180cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe616d9d640 1 -- 172.21.15.122:0/4241263487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6181f48f0 con 0x7fe61815dfe0 2024-03-06T07:32:28.227 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe60ffff640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe616d9d640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180b1320 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fe61000cda0 tx=0x7fe61000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610003b90 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe610003d30 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610004240 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181f4610 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.220+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6180cc170 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.221+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61804f330 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.221+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6100043e0 con 0x7fe61815dfe0 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.222+0000 7fe60effd640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe5f806b500 0x7fe5f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.228 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.222+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe61008ad10 con 0x7fe61815dfe0 2024-03-06T07:32:28.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.222+0000 7fe61659c640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe5f806b500 0x7fe5f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.222+0000 7fe61659c640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe5f806b500 0x7fe5f806d9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe608036000 tx=0x7fe608008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.230+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe610090020 con 0x7fe61815dfe0 2024-03-06T07:32:28.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.511+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe61815d570 con 0x7fe61815dfe0 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.512+0000 7fe60effd640 1 -- 172.21.15.122:0/4241263487 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fe610059ad0 con 0x7fe61815dfe0 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe5f806b500 msgr2=0x7fe5f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe5f806b500 0x7fe5f806d9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe608036000 tx=0x7fe608008030 comp rx=0 tx=0).stop 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 msgr2=0x7fe6180b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180b1320 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fe61000cda0 tx=0x7fe61000a660 comp rx=0 tx=0).stop 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 shutdown_connections 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe5f806b500 0x7fe5f806d9c0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe61815dfe0 0x7fe6180b1320 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6180d27f0 0x7fe6180a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 --2- 172.21.15.122:0/4241263487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6180cf830 0x7fe6180cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 >> 172.21.15.122:0/4241263487 conn(0x7fe6180d3cd0 msgr2=0x7fe6180b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 shutdown_connections 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:28.514+0000 7fe61cb8a640 1 -- 172.21.15.122:0/4241263487 wait complete. 2024-03-06T07:32:28.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:28.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:29.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:29.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.831+0000 7f4bf3e4e640 1 Processor -- start 2024-03-06T07:32:29.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.831+0000 7f4bf3e4e640 1 -- start start 2024-03-06T07:32:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.832+0000 7f4bf3e4e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4bec0cfd20 0x7f4bec06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf3e4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec066e30 con 0x7f4bec15afd0 2024-03-06T07:32:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf3e4e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec06bba0 con 0x7f4bec15abf0 2024-03-06T07:32:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf3e4e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec06be80 con 0x7f4bec0cfd20 2024-03-06T07:32:29.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf13e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4bec0cfd20 0x7f4bec06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf13e9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4bec0cfd20 0x7f4bec06d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59726/0 (socket says 172.21.15.122:59726) 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf23eb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4bec15afd0 0x7f4bec06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56732/0 (socket says 172.21.15.122:56732) 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf13e9640 1 -- 172.21.15.122:0/390056642 learned_addr learned my addr 172.21.15.122:0/390056642 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf13e9640 1 -- 172.21.15.122:0/390056642 >> v1:172.21.15.169:6789/0 conn(0x7f4bec15abf0 legacy=0x7f4bec0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf13e9640 1 -- 172.21.15.122:0/390056642 >> v1:172.21.15.122:6789/0 conn(0x7f4bec15afd0 legacy=0x7f4bec06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf13e9640 1 -- 172.21.15.122:0/390056642 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec06eae0 con 0x7f4bec0cfd20 2024-03-06T07:32:29.836 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.833+0000 7f4bf0be8640 1 -- 172.21.15.122:0/390056642 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898914231 0 0) 0x7f4bec066e30 con 0x7f4bec15afd0 2024-03-06T07:32:29.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.834+0000 7f4bf13e9640 1 --2- 172.21.15.122:0/390056642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec0cfd20 0x7f4bec06d160 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009870 tx=0x7f4be002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de950610994879d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.834+0000 7f4bf0be8640 1 -- 172.21.15.122:0/390056642 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0035020 con 0x7f4bec0cfd20 2024-03-06T07:32:29.837 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.834+0000 7f4bf0be8640 1 -- 172.21.15.122:0/390056642 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be0031870 con 0x7f4bec0cfd20 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.834+0000 7f4bf0be8640 1 -- 172.21.15.122:0/390056642 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0031c40 con 0x7f4bec0cfd20 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/390056642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec0cfd20 msgr2=0x7f4bec06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/390056642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec0cfd20 0x7f4bec06d160 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009870 tx=0x7f4be002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/390056642 shutdown_connections 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/390056642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec0cfd20 0x7f4bec06d160 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/390056642 >> 172.21.15.122:0/390056642 conn(0x7f4bec0b1d00 msgr2=0x7f4bec0b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/390056642 shutdown_connections 2024-03-06T07:32:29.838 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.835+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/390056642 wait complete. 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.836+0000 7f4bf3e4e640 1 Processor -- start 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.836+0000 7f4bf3e4e640 1 -- start start 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bec0cfd20 0x7f4bec0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 0x7f4bec1f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bec15afd0 0x7f4bec0ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec0732b0 con 0x7f4bec15abf0 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec073130 con 0x7f4bec15afd0 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec06f000 con 0x7f4bec0cfd20 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 0x7f4bec1f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 0x7f4bec1f3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59732/0 (socket says 172.21.15.122:59732) 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 -- 172.21.15.122:0/2451378542 learned_addr learned my addr 172.21.15.122:0/2451378542 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf23eb640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bec15afd0 0x7f4bec0ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf1bea640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bec0cfd20 0x7f4bec0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 -- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bec0cfd20 msgr2=0x7f4bec0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bec0cfd20 0x7f4bec0cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 -- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bec15afd0 msgr2=0x7f4bec0ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bec15afd0 0x7f4bec0ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 -- 172.21.15.122:0/2451378542 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec1f4460 con 0x7f4bec15abf0 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf23eb640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bec15afd0 0x7f4bec0ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf1bea640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bec0cfd20 0x7f4bec0cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:29.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf13e9640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 0x7f4bec1f3af0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f4be00090b0 tx=0x7f4be00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0040070 con 0x7f4bec15abf0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be0004030 con 0x7f4bec15abf0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0035040 con 0x7f4bec15abf0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.837+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec1f4740 con 0x7f4bec15abf0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.838+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec1f4d00 con 0x7f4bec15abf0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.839+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec04f330 con 0x7f4bec15abf0 2024-03-06T07:32:29.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.839+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4be0038020 con 0x7f4bec15abf0 2024-03-06T07:32:29.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.839+0000 7f4bde7fc640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bd006b4d0 0x7f4bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.839+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4be00ac1a0 con 0x7f4bec15abf0 2024-03-06T07:32:29.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.839+0000 7f4bf1bea640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bd006b4d0 0x7f4bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.840+0000 7f4bf1bea640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bd006b4d0 0x7f4bd006d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4be8007fd0 tx=0x7f4be8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:29.844+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4be0042350 con 0x7f4bec15abf0 2024-03-06T07:32:30.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.054+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bec15a470 con 0x7f4bec15abf0 2024-03-06T07:32:30.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.055+0000 7f4bde7fc640 1 -- 172.21.15.122:0/2451378542 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f4be007b000 con 0x7f4bec15abf0 2024-03-06T07:32:30.057 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:30.057 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:30.057 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.057+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bd006b4d0 msgr2=0x7f4bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.057+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bd006b4d0 0x7f4bd006d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4be8007fd0 tx=0x7f4be8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.057+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 msgr2=0x7f4bec1f3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.057+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 0x7f4bec1f3af0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f4be00090b0 tx=0x7f4be00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 shutdown_connections 2024-03-06T07:32:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bd006b4d0 0x7f4bd006d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4bec15afd0 0x7f4bec0ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bec15abf0 0x7f4bec1f3af0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 --2- 172.21.15.122:0/2451378542 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4bec0cfd20 0x7f4bec0cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 >> 172.21.15.122:0/2451378542 conn(0x7f4bec0b1d00 msgr2=0x7f4bec0acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 shutdown_connections 2024-03-06T07:32:30.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:30.058+0000 7f4bf3e4e640 1 -- 172.21.15.122:0/2451378542 wait complete. 2024-03-06T07:32:30.271 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:30.272 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:31.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:31.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.372+0000 7f1488e8d640 1 Processor -- start 2024-03-06T07:32:31.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.372+0000 7f1488e8d640 1 -- start start 2024-03-06T07:32:31.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1488e8d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14840d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1488e8d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f14840b41f0 0x7f14840b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1488e8d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840b3430 con 0x7f148415dfe0 2024-03-06T07:32:31.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1488e8d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14840d2c50 con 0x7f14840b41f0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1488e8d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14840b31e0 con 0x7f14840d22b0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f148259c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f148415dfe0 0x7f14840cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56734/0 (socket says 172.21.15.122:56734) 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f148259c640 1 -- 172.21.15.122:0/3793670218 learned_addr learned my addr 172.21.15.122:0/3793670218 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1481d9b640 1 --2- 172.21.15.122:0/3793670218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14840d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1482d9d640 1 --2- 172.21.15.122:0/3793670218 >> v2:172.21.15.169:3300/0 conn(0x7f14840b41f0 0x7f14840b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f148159a640 1 -- 172.21.15.122:0/3793670218 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644673770 0 0) 0x7f14840b3430 con 0x7f148415dfe0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f148159a640 1 -- 172.21.15.122:0/3793670218 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146c003520 con 0x7f148415dfe0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1481d9b640 1 -- 172.21.15.122:0/3793670218 >> v2:172.21.15.169:3300/0 conn(0x7f14840b41f0 msgr2=0x7f14840b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1481d9b640 1 --2- 172.21.15.122:0/3793670218 >> v2:172.21.15.169:3300/0 conn(0x7f14840b41f0 0x7f14840b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1481d9b640 1 -- 172.21.15.122:0/3793670218 >> v1:172.21.15.122:6789/0 conn(0x7f148415dfe0 legacy=0x7f14840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1481d9b640 1 -- 172.21.15.122:0/3793670218 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14840b3710 con 0x7f14840d22b0 2024-03-06T07:32:31.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.373+0000 7f1482d9d640 1 --2- 172.21.15.122:0/3793670218 >> v2:172.21.15.169:3300/0 conn(0x7f14840b41f0 0x7f14840b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:31.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.374+0000 7f1481d9b640 1 --2- 172.21.15.122:0/3793670218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14840d2710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f14780097a0 tx=0x7f147802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd5553a2047975f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.374+0000 7f148159a640 1 -- 172.21.15.122:0/3793670218 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478035020 con 0x7f14840d22b0 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.374+0000 7f148159a640 1 -- 172.21.15.122:0/3793670218 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1478031830 con 0x7f14840d22b0 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.374+0000 7f148159a640 1 -- 172.21.15.122:0/3793670218 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478031be0 con 0x7f14840d22b0 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 -- 172.21.15.122:0/3793670218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 msgr2=0x7f14840d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 --2- 172.21.15.122:0/3793670218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14840d2710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f14780097a0 tx=0x7f147802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 -- 172.21.15.122:0/3793670218 shutdown_connections 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 --2- 172.21.15.122:0/3793670218 >> v2:172.21.15.169:3300/0 conn(0x7f14840b41f0 0x7f14840b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 --2- 172.21.15.122:0/3793670218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14840d2710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 -- 172.21.15.122:0/3793670218 >> 172.21.15.122:0/3793670218 conn(0x7f14840d3cd0 msgr2=0x7f14840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 -- 172.21.15.122:0/3793670218 shutdown_connections 2024-03-06T07:32:31.378 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.375+0000 7f1488e8d640 1 -- 172.21.15.122:0/3793670218 wait complete. 2024-03-06T07:32:31.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.376+0000 7f1488e8d640 1 Processor -- start 2024-03-06T07:32:31.380 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.376+0000 7f1488e8d640 1 -- start start 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1488e8d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14840b41f0 0x7f14840b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1488e8d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14841f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1488e8d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 0x7f14841f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1488e8d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14840cfa00 con 0x7f148415dfe0 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1488e8d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14840b3d60 con 0x7f14840b41f0 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1488e8d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14840d1c70 con 0x7f14840d22b0 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 0x7f14841f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 0x7f14841f7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59748/0 (socket says 172.21.15.122:59748) 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 -- 172.21.15.122:0/84851609 learned_addr learned my addr 172.21.15.122:0/84851609 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1481d9b640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14841f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f148259c640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14840b41f0 0x7f14840b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 -- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 msgr2=0x7f14841f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.381 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14841f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 -- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14840b41f0 msgr2=0x7f14840b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14840b41f0 0x7f14840b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 -- 172.21.15.122:0/84851609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484161470 con 0x7f148415dfe0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1481d9b640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14841f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.377+0000 7f1482d9d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 0x7f14841f7240 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f147c00efc0 tx=0x7f147c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.378+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c018030 con 0x7f148415dfe0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.378+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148415fcb0 con 0x7f148415dfe0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.378+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c00cd90 con 0x7f148415dfe0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.378+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c003bb0 con 0x7f148415dfe0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.378+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1484160270 con 0x7f148415dfe0 2024-03-06T07:32:31.382 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.379+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1484160470 con 0x7f148415dfe0 2024-03-06T07:32:31.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.379+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f147c01b070 con 0x7f148415dfe0 2024-03-06T07:32:31.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.380+0000 7f1472ffd640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f146406b500 0x7f146406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.380+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f147c08ab70 con 0x7f148415dfe0 2024-03-06T07:32:31.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.380+0000 7f148259c640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f146406b500 0x7f146406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.383 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.380+0000 7f148259c640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f146406b500 0x7f146406d9c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f1474005880 tx=0x7f1474009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.384+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1484160470 con 0x7f148415dfe0 2024-03-06T07:32:31.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.630+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f148404f330 con 0x7f148415dfe0 2024-03-06T07:32:31.632 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.630+0000 7f1472ffd640 1 -- 172.21.15.122:0/84851609 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f147c010040 con 0x7f148415dfe0 2024-03-06T07:32:31.633 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:31.633 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:31.633 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f146406b500 msgr2=0x7f146406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f146406b500 0x7f146406d9c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f1474005880 tx=0x7f1474009040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 msgr2=0x7f14841f7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 0x7f14841f7240 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f147c00efc0 tx=0x7f147c00c490 comp rx=0 tx=0).stop 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 shutdown_connections 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f146406b500 0x7f146406d9c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f148415dfe0 0x7f14841f7240 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14840d22b0 0x7f14841f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 --2- 172.21.15.122:0/84851609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14840b41f0 0x7f14840b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.632+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 >> 172.21.15.122:0/84851609 conn(0x7f14840d3cd0 msgr2=0x7f148406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.633+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 shutdown_connections 2024-03-06T07:32:31.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:31.633+0000 7f1488e8d640 1 -- 172.21.15.122:0/84851609 wait complete. 2024-03-06T07:32:31.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:31.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:32.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:32.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.948+0000 7fd516781640 1 Processor -- start 2024-03-06T07:32:32.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.948+0000 7fd516781640 1 -- start start 2024-03-06T07:32:32.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd516781640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd5100d22b0 0x7fd5100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd516781640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5100b3430 con 0x7fd5100b41f0 2024-03-06T07:32:32.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd516781640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5100b3710 con 0x7fd51015dfe0 2024-03-06T07:32:32.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd516781640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5100d2c50 con 0x7fd5100d22b0 2024-03-06T07:32:32.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd514d1e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd5100b41f0 0x7fd5100b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56740/0 (socket says 172.21.15.122:56740) 2024-03-06T07:32:32.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd514d1e640 1 -- 172.21.15.122:0/298232561 learned_addr learned my addr 172.21.15.122:0/298232561 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50f7fe640 1 --2- 172.21.15.122:0/298232561 >> v2:172.21.15.169:3300/0 conn(0x7fd5100d22b0 0x7fd5100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160006043 0 0) 0x7fd5100b3430 con 0x7fd5100b41f0 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8003520 con 0x7fd5100b41f0 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735234573 0 0) 0x7fd4f8003520 con 0x7fd5100b41f0 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 >> v2:172.21.15.169:3300/0 conn(0x7fd5100d22b0 msgr2=0x7fd5100d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50effd640 1 --2- 172.21.15.122:0/298232561 >> v2:172.21.15.169:3300/0 conn(0x7fd5100d22b0 0x7fd5100d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.949+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 >> v1:172.21.15.169:6789/0 conn(0x7fd51015dfe0 legacy=0x7fd5100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5100b39f0 con 0x7fd5100b41f0 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fd5080019f0 con 0x7fd5100b41f0 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 >> v1:172.21.15.122:6789/0 conn(0x7fd5100b41f0 legacy=0x7fd5100b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f8003a00 0x7fd4f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f8004800 0x7fd4f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4f80051c0 0x7fd4f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f800c090 con 0x7fd4f80051c0 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f800c240 con 0x7fd4f8004800 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f800c440 con 0x7fd4f8003a00 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd514d1e640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4f80051c0 0x7fd4f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50ffff640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f8003a00 0x7fd4f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50f7fe640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f8004800 0x7fd4f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd514d1e640 1 -- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f8003a00 msgr2=0x7fd4f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd514d1e640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f8003a00 0x7fd4f8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd514d1e640 1 -- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f8004800 msgr2=0x7fd4f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd514d1e640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f8004800 0x7fd4f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd514d1e640 1 -- 172.21.15.122:0/298232561 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8003520 con 0x7fd4f80051c0 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.950+0000 7fd50f7fe640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f8004800 0x7fd4f8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.951+0000 7fd514d1e640 1 -- 172.21.15.122:0/298232561 reap_dead start 2024-03-06T07:32:32.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.951+0000 7fd514d1e640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4f80051c0 0x7fd4f800b9c0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fd50800a790 tx=0x7fd4f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd8055d7f312a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.951+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50800a3b0 con 0x7fd4f80051c0 2024-03-06T07:32:32.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.951+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508034670 con 0x7fd4f80051c0 2024-03-06T07:32:32.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.951+0000 7fd50effd640 1 -- 172.21.15.122:0/298232561 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508034a60 con 0x7fd4f80051c0 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 -- 172.21.15.122:0/298232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4f80051c0 msgr2=0x7fd4f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4f80051c0 0x7fd4f800b9c0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fd50800a790 tx=0x7fd4f8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 -- 172.21.15.122:0/298232561 shutdown_connections 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 --2- 172.21.15.122:0/298232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4f80051c0 0x7fd4f800b9c0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 -- 172.21.15.122:0/298232561 >> 172.21.15.122:0/298232561 conn(0x7fd5100d3cd0 msgr2=0x7fd5100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 -- 172.21.15.122:0/298232561 shutdown_connections 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.952+0000 7fd516781640 1 -- 172.21.15.122:0/298232561 wait complete. 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.953+0000 7fd516781640 1 Processor -- start 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.953+0000 7fd516781640 1 -- start start 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd516781640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f80051c0 0x7fd510144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd516781640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 0x7fd51006c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd516781640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5100caf60 0x7fd51006ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd516781640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100b3d60 con 0x7fd5100b9470 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd516781640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5100b31e0 con 0x7fd5100caf60 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd516781640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51015e3c0 con 0x7fd4f80051c0 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 0x7fd51006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 0x7fd51006c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59756/0 (socket says 172.21.15.122:59756) 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 -- 172.21.15.122:0/3360364736 learned_addr learned my addr 172.21.15.122:0/3360364736 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd514d1e640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5100caf60 0x7fd51006ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50ffff640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f80051c0 0x7fd510144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 -- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f80051c0 msgr2=0x7fd510144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f80051c0 0x7fd510144690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 -- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5100caf60 msgr2=0x7fd51006ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5100caf60 0x7fd51006ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 -- 172.21.15.122:0/3360364736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5100cc700 con 0x7fd5100b9470 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.954+0000 7fd50f7fe640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 0x7fd51006c910 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fd5040115a0 tx=0x7fd504011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.955+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504015870 con 0x7fd5100b9470 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.955+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504015a10 con 0x7fd5100b9470 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.955+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd510144bd0 con 0x7fd5100b9470 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.955+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504015dc0 con 0x7fd5100b9470 2024-03-06T07:32:32.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.955+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5100ccc80 con 0x7fd5100b9470 2024-03-06T07:32:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.956+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51004f330 con 0x7fd5100b9470 2024-03-06T07:32:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.956+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd50401d400 con 0x7fd5100b9470 2024-03-06T07:32:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.956+0000 7fd50cff9640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.957+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd50408aa40 con 0x7fd5100b9470 2024-03-06T07:32:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.957+0000 7fd50ffff640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.957+0000 7fd50ffff640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd500000c00 tx=0x7fd500009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:32.961+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd504019030 con 0x7fd5100b9470 2024-03-06T07:32:33.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.175+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5100b9b90 con 0x7fd5100b9470 2024-03-06T07:32:33.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.175+0000 7fd50cff9640 1 -- 172.21.15.122:0/3360364736 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fd50408ae20 con 0x7fd5100b9470 2024-03-06T07:32:33.178 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:33.178 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:33.178 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.177+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4fc06b500 msgr2=0x7fd4fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.177+0000 7fd516781640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd500000c00 tx=0x7fd500009040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.177+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 msgr2=0x7fd51006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.177+0000 7fd516781640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 0x7fd51006c910 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fd5040115a0 tx=0x7fd504011a60 comp rx=0 tx=0).stop 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 shutdown_connections 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5100caf60 0x7fd51006ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5100b9470 0x7fd51006c910 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 --2- 172.21.15.122:0/3360364736 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4f80051c0 0x7fd510144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 >> 172.21.15.122:0/3360364736 conn(0x7fd5100d3cd0 msgr2=0x7fd5100d0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 shutdown_connections 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:33.178+0000 7fd516781640 1 -- 172.21.15.122:0/3360364736 wait complete. 2024-03-06T07:32:33.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:33.392 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:34.393 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:34.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.492+0000 7f4587f13640 1 Processor -- start 2024-03-06T07:32:34.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.493+0000 7f4587f13640 1 -- start start 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.493+0000 7f4587f13640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45800cf100 0x7f45800cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.493+0000 7f4587f13640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45800ccc70 con 0x7f458006e270 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.493+0000 7f4587f13640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458015e440 con 0x7f45800cc810 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.493+0000 7f4587f13640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458006e680 con 0x7f45800cf100 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4585caf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45800cf100 0x7f45800cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4585caf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f45800cf100 0x7f45800cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48062/0 (socket says 172.21.15.122:48062) 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4585caf640 1 -- 172.21.15.122:0/3560920618 learned_addr learned my addr 172.21.15.122:0/3560920618 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f45864b0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f458006e270 0x7f45800ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48656/0 (socket says 172.21.15.122:48656) 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4585caf640 1 -- 172.21.15.122:0/3560920618 >> v1:172.21.15.169:6789/0 conn(0x7f45800cc810 legacy=0x7f458006db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4585caf640 1 -- 172.21.15.122:0/3560920618 >> v1:172.21.15.122:6789/0 conn(0x7f458006e270 legacy=0x7f45800ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4585caf640 1 -- 172.21.15.122:0/3560920618 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458015e700 con 0x7f45800cf100 2024-03-06T07:32:34.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.494+0000 7f4584cad640 1 -- 172.21.15.122:0/3560920618 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191602275 0 0) 0x7f45800ccc70 con 0x7f458006e270 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4585caf640 1 --2- 172.21.15.122:0/3560920618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45800cf100 0x7f45800cc2d0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f457802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a8b677562701b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4584cad640 1 -- 172.21.15.122:0/3560920618 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578035020 con 0x7f45800cf100 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4584cad640 1 -- 172.21.15.122:0/3560920618 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578031870 con 0x7f45800cf100 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4584cad640 1 -- 172.21.15.122:0/3560920618 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578031c40 con 0x7f45800cf100 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 -- 172.21.15.122:0/3560920618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45800cf100 msgr2=0x7f45800cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 --2- 172.21.15.122:0/3560920618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45800cf100 0x7f45800cc2d0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f457802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 -- 172.21.15.122:0/3560920618 shutdown_connections 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 --2- 172.21.15.122:0/3560920618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45800cf100 0x7f45800cc2d0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 -- 172.21.15.122:0/3560920618 >> 172.21.15.122:0/3560920618 conn(0x7f45800cf8a0 msgr2=0x7f45800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 -- 172.21.15.122:0/3560920618 shutdown_connections 2024-03-06T07:32:34.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.495+0000 7f4587f13640 1 -- 172.21.15.122:0/3560920618 wait complete. 2024-03-06T07:32:34.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.497+0000 7f4587f13640 1 Processor -- start 2024-03-06T07:32:34.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.497+0000 7f4587f13640 1 -- start start 2024-03-06T07:32:34.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4587f13640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 0x7f45800b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4587f13640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45800cc810 0x7f4580145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4587f13640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45800cf100 0x7f4580160f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4587f13640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458015efa0 con 0x7f458006e270 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4587f13640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458015ecc0 con 0x7f45800cc810 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4587f13640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800cf4e0 con 0x7f45800cf100 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.498+0000 7f4585caf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 0x7f45800b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 0x7f45800b0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48078/0 (socket says 172.21.15.122:48078) 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 -- 172.21.15.122:0/1782377636 learned_addr learned my addr 172.21.15.122:0/1782377636 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f45864b0640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45800cf100 0x7f4580160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f45854ae640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45800cc810 0x7f4580145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 -- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45800cf100 msgr2=0x7f4580160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45800cf100 0x7f4580160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 -- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45800cc810 msgr2=0x7f4580145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45800cc810 0x7f4580145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 -- 172.21.15.122:0/1782377636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45801459d0 con 0x7f458006e270 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4585caf640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 0x7f45800b0680 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f45780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457803ad60 con 0x7f458006e270 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45780443f0 con 0x7f458006e270 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45800af5e0 con 0x7f458006e270 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45780447e0 con 0x7f458006e270 2024-03-06T07:32:34.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.499+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45800afba0 con 0x7f458006e270 2024-03-06T07:32:34.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.500+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458004f330 con 0x7f458006e270 2024-03-06T07:32:34.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.501+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f457804b060 con 0x7f458006e270 2024-03-06T07:32:34.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.501+0000 7f457e7fc640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456c06b4d0 0x7f456c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.501+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f45780ac3e0 con 0x7f458006e270 2024-03-06T07:32:34.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.501+0000 7f45854ae640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456c06b4d0 0x7f456c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.501+0000 7f45854ae640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456c06b4d0 0x7f456c06d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f4580160cb0 tx=0x7f4574009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.505+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45801459d0 con 0x7f458006e270 2024-03-06T07:32:34.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.735+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4580154130 con 0x7f458006e270 2024-03-06T07:32:34.737 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.735+0000 7f457e7fc640 1 -- 172.21.15.122:0/1782377636 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f457803f070 con 0x7f458006e270 2024-03-06T07:32:34.737 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:34.737 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:34.738 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:34.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456c06b4d0 msgr2=0x7f456c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456c06b4d0 0x7f456c06d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f4580160cb0 tx=0x7f4574009040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 msgr2=0x7f45800b0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 0x7f45800b0680 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f45780046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 shutdown_connections 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f456c06b4d0 0x7f456c06d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f45800cf100 0x7f4580160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45800cc810 0x7f4580145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 --2- 172.21.15.122:0/1782377636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f458006e270 0x7f45800b0680 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 >> 172.21.15.122:0/1782377636 conn(0x7f45800cf8a0 msgr2=0x7f45800cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 shutdown_connections 2024-03-06T07:32:34.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:34.737+0000 7f4587f13640 1 -- 172.21.15.122:0/1782377636 wait complete. 2024-03-06T07:32:34.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:34.951 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:35.952 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:36.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 Processor -- start 2024-03-06T07:32:36.053 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 -- start start 2024-03-06T07:32:36.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f046015dfe0 0x7f04600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f04600d27f0 0x7f04600d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04600d2c50 con 0x7f046015dfe0 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04600b31e0 con 0x7f04600d27f0 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f04660df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04600b3d60 con 0x7f04600cf830 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.051+0000 7f045f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f046015dfe0 0x7f04600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f046015dfe0 0x7f04600cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48090/0 (socket says 172.21.15.122:48090) 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 -- 172.21.15.122:0/3908831440 learned_addr learned my addr 172.21.15.122:0/3908831440 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045effd640 1 --2- 172.21.15.122:0/3908831440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 -- 172.21.15.122:0/3908831440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 msgr2=0x7f04600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 --2- 172.21.15.122:0/3908831440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045ffff640 1 --2- 172.21.15.122:0/3908831440 >> v2:172.21.15.169:3300/0 conn(0x7f04600d27f0 0x7f04600d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 -- 172.21.15.122:0/3908831440 >> v2:172.21.15.169:3300/0 conn(0x7f04600d27f0 msgr2=0x7f04600d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 --2- 172.21.15.122:0/3908831440 >> v2:172.21.15.169:3300/0 conn(0x7f04600d27f0 0x7f04600d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.052+0000 7f045f7fe640 1 -- 172.21.15.122:0/3908831440 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04600b3430 con 0x7f046015dfe0 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f045f7fe640 1 --2- 172.21.15.122:0/3908831440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046015dfe0 0x7f04600cf2f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f0460067ad0 tx=0x7f045002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1864a5078d8ae557 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f045e7fc640 1 -- 172.21.15.122:0/3908831440 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450035020 con 0x7f046015dfe0 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f045e7fc640 1 -- 172.21.15.122:0/3908831440 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450031870 con 0x7f046015dfe0 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f045e7fc640 1 -- 172.21.15.122:0/3908831440 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450031c40 con 0x7f046015dfe0 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 -- 172.21.15.122:0/3908831440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046015dfe0 msgr2=0x7f04600cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 --2- 172.21.15.122:0/3908831440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046015dfe0 0x7f04600cf2f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f0460067ad0 tx=0x7f045002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 -- 172.21.15.122:0/3908831440 shutdown_connections 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 --2- 172.21.15.122:0/3908831440 >> v2:172.21.15.169:3300/0 conn(0x7f04600d27f0 0x7f04600d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 --2- 172.21.15.122:0/3908831440 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 --2- 172.21.15.122:0/3908831440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046015dfe0 0x7f04600cf2f0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 -- 172.21.15.122:0/3908831440 >> 172.21.15.122:0/3908831440 conn(0x7f04600d3cd0 msgr2=0x7f04600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 -- 172.21.15.122:0/3908831440 shutdown_connections 2024-03-06T07:32:36.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.053+0000 7f04660df640 1 -- 172.21.15.122:0/3908831440 wait complete. 2024-03-06T07:32:36.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.055+0000 7f04660df640 1 Processor -- start 2024-03-06T07:32:36.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.055+0000 7f04660df640 1 -- start start 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.056+0000 7f04660df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.056+0000 7f04660df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 0x7f04600cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.056+0000 7f04660df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046015dfe0 0x7f04600cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f04660df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04600d1c70 con 0x7f04600d27f0 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f04660df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04600b39f0 con 0x7f046015dfe0 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f04660df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04600d1df0 con 0x7f04600cf830 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 0x7f04600cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 0x7f04600cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48106/0 (socket says 172.21.15.122:48106) 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 -- 172.21.15.122:0/1315558880 learned_addr learned my addr 172.21.15.122:0/1315558880 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045f7fe640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 -- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 msgr2=0x7f04600cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045ffff640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046015dfe0 0x7f04600cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 -- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046015dfe0 msgr2=0x7f04600cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046015dfe0 0x7f04600cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 -- 172.21.15.122:0/1315558880 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04600cd5c0 con 0x7f04600d27f0 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045f7fe640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045ffff640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046015dfe0 0x7f04600cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f045effd640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 0x7f04600cada0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f045400c7f0 tx=0x7f045400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454015070 con 0x7f04600d27f0 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0454004220 con 0x7f04600d27f0 2024-03-06T07:32:36.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04600cd8a0 con 0x7f04600d27f0 2024-03-06T07:32:36.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454007620 con 0x7f04600d27f0 2024-03-06T07:32:36.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.057+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04600cde60 con 0x7f04600d27f0 2024-03-06T07:32:36.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.058+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04600b5200 con 0x7f04600d27f0 2024-03-06T07:32:36.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.059+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04540077c0 con 0x7f04600d27f0 2024-03-06T07:32:36.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.060+0000 7f0447fff640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044c06b4d0 0x7f044c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.060+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f045408a0e0 con 0x7f04600d27f0 2024-03-06T07:32:36.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.060+0000 7f045f7fe640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044c06b4d0 0x7f044c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.060+0000 7f045f7fe640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044c06b4d0 0x7f044c06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0450036000 tx=0x7f0450008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.066 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.064+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04600b5200 con 0x7f04600d27f0 2024-03-06T07:32:36.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.280+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04600d0b60 con 0x7f04600d27f0 2024-03-06T07:32:36.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.280+0000 7f0447fff640 1 -- 172.21.15.122:0/1315558880 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f0454058eb0 con 0x7f04600d27f0 2024-03-06T07:32:36.282 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:36.282 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:31:34.001145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:36.282 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 100 2024-03-06T07:32:36.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044c06b4d0 msgr2=0x7f044c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044c06b4d0 0x7f044c06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0450036000 tx=0x7f0450008030 comp rx=0 tx=0).stop 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 msgr2=0x7f04600cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 0x7f04600cada0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f045400c7f0 tx=0x7f045400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 shutdown_connections 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044c06b4d0 0x7f044c06d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046015dfe0 0x7f04600cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04600d27f0 0x7f04600cada0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 --2- 172.21.15.122:0/1315558880 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f04600cf830 0x7f04600cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 >> 172.21.15.122:0/1315558880 conn(0x7f04600d3cd0 msgr2=0x7f046006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 shutdown_connections 2024-03-06T07:32:36.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:36.282+0000 7f04660df640 1 -- 172.21.15.122:0/1315558880 wait complete. 2024-03-06T07:32:36.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:31:34.001145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:36.500 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:37.501 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:37.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 Processor -- start 2024-03-06T07:32:37.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 -- start start 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f474415dfe0 0x7f47440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f47440cf830 0x7f47440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47440b3430 con 0x7f47440d27f0 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440b31e0 con 0x7f474415dfe0 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.601+0000 7f474af19640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440b3d60 con 0x7f47440cf830 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f474415dfe0 0x7f47440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f474415dfe0 0x7f47440cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48116/0 (socket says 172.21.15.122:48116) 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 -- 172.21.15.122:0/430758724 learned_addr learned my addr 172.21.15.122:0/430758724 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f473bfff640 1 --2- 172.21.15.122:0/430758724 >> v2:172.21.15.169:3300/0 conn(0x7f47440cf830 0x7f47440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 -- 172.21.15.122:0/430758724 >> v2:172.21.15.169:3300/0 conn(0x7f47440cf830 msgr2=0x7f47440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 --2- 172.21.15.122:0/430758724 >> v2:172.21.15.169:3300/0 conn(0x7f47440cf830 0x7f47440d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 -- 172.21.15.122:0/430758724 >> v1:172.21.15.122:6789/0 conn(0x7f47440d27f0 legacy=0x7f47440d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f4748cb5640 1 -- 172.21.15.122:0/430758724 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440b3710 con 0x7f474415dfe0 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f473b7fe640 1 -- 172.21.15.122:0/430758724 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560320558 0 0) 0x7f47440b3430 con 0x7f47440d27f0 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.602+0000 7f473bfff640 1 --2- 172.21.15.122:0/430758724 >> v2:172.21.15.169:3300/0 conn(0x7f47440cf830 0x7f47440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f4748cb5640 1 --2- 172.21.15.122:0/430758724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474415dfe0 0x7f47440cf2f0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f4740009870 tx=0x7f474002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d10a543cb63ac40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f473b7fe640 1 -- 172.21.15.122:0/430758724 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740034070 con 0x7f474415dfe0 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f473b7fe640 1 -- 172.21.15.122:0/430758724 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4740030860 con 0x7f474415dfe0 2024-03-06T07:32:37.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f473b7fe640 1 -- 172.21.15.122:0/430758724 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740030c30 con 0x7f474415dfe0 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 -- 172.21.15.122:0/430758724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474415dfe0 msgr2=0x7f47440cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 --2- 172.21.15.122:0/430758724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474415dfe0 0x7f47440cf2f0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f4740009870 tx=0x7f474002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 -- 172.21.15.122:0/430758724 shutdown_connections 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 --2- 172.21.15.122:0/430758724 >> v2:172.21.15.169:3300/0 conn(0x7f47440cf830 0x7f47440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 --2- 172.21.15.122:0/430758724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f474415dfe0 0x7f47440cf2f0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 -- 172.21.15.122:0/430758724 >> 172.21.15.122:0/430758724 conn(0x7f47440d3cd0 msgr2=0x7f47440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 -- 172.21.15.122:0/430758724 shutdown_connections 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.603+0000 7f474af19640 1 -- 172.21.15.122:0/430758724 wait complete. 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.604+0000 7f474af19640 1 Processor -- start 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 -- start start 2024-03-06T07:32:37.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47440cf830 0x7f47440b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 0x7f47440b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474415dfe0 0x7f47440b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440d1df0 con 0x7f47440d27f0 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440d1c70 con 0x7f47440cf830 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.605+0000 7f474af19640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474415e3c0 con 0x7f474415dfe0 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 0x7f47440b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 0x7f47440b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48122/0 (socket says 172.21.15.122:48122) 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 -- 172.21.15.122:0/2367088680 learned_addr learned my addr 172.21.15.122:0/2367088680 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f47494b6640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474415dfe0 0x7f47440b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f4748cb5640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47440cf830 0x7f47440b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 -- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474415dfe0 msgr2=0x7f47440b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474415dfe0 0x7f47440b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 -- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47440cf830 msgr2=0x7f47440b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47440cf830 0x7f47440b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 -- 172.21.15.122:0/2367088680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440b8430 con 0x7f47440d27f0 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f473bfff640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 0x7f47440b4610 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f473400c7f0 tx=0x7f473400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734015070 con 0x7f47440d27f0 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47440b8710 con 0x7f47440d27f0 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.606+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47440b8cd0 con 0x7f47440d27f0 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.608+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474404f330 con 0x7f47440d27f0 2024-03-06T07:32:37.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.609+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734007a30 con 0x7f47440d27f0 2024-03-06T07:32:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.609+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473401f060 con 0x7f47440d27f0 2024-03-06T07:32:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.610+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4734018070 con 0x7f47440d27f0 2024-03-06T07:32:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.610+0000 7f47397fa640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473006b4d0 0x7f473006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.611+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f473408a800 con 0x7f47440d27f0 2024-03-06T07:32:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.611+0000 7f4748cb5640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473006b4d0 0x7f473006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.611+0000 7f4748cb5640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473006b4d0 0x7f473006d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4740036000 tx=0x7f474003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.614+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4734059660 con 0x7f47440d27f0 2024-03-06T07:32:37.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.826+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f474406a1b0 con 0x7f47440d27f0 2024-03-06T07:32:37.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.826+0000 7f47397fa640 1 -- 172.21.15.122:0/2367088680 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f4734058f60 con 0x7f47440d27f0 2024-03-06T07:32:37.829 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:37.829 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:37.829 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473006b4d0 msgr2=0x7f473006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473006b4d0 0x7f473006d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4740036000 tx=0x7f474003b040 comp rx=0 tx=0).stop 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 msgr2=0x7f47440b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 0x7f47440b4610 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f473400c7f0 tx=0x7f473400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 shutdown_connections 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f473006b4d0 0x7f473006d990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f474415dfe0 0x7f47440b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47440d27f0 0x7f47440b4610 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 --2- 172.21.15.122:0/2367088680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47440cf830 0x7f47440b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 >> 172.21.15.122:0/2367088680 conn(0x7f47440d3cd0 msgr2=0x7f47440d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 shutdown_connections 2024-03-06T07:32:37.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:37.829+0000 7f474af19640 1 -- 172.21.15.122:0/2367088680 wait complete. 2024-03-06T07:32:38.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:38.043 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:39.044 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:39.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.148+0000 7f7a5059a640 1 Processor -- start 2024-03-06T07:32:39.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.148+0000 7f7a5059a640 1 -- start start 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a5059a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7a4806da40 0x7f7a480cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a5059a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7a480cd7a0 0x7f7a4815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a5059a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a480cc980 con 0x7f7a481424f0 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a5059a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a480cd220 con 0x7f7a480cd7a0 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a5059a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a480ccc60 con 0x7f7a4806da40 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4eb37640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7a480cd7a0 0x7f7a4815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4e336640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7a481424f0 0x7f7a4806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48684/0 (socket says 172.21.15.122:48684) 2024-03-06T07:32:39.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4e336640 1 -- 172.21.15.122:0/1289793070 learned_addr learned my addr 172.21.15.122:0/1289793070 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:39.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4db35640 1 --2- 172.21.15.122:0/1289793070 >> v2:172.21.15.169:3300/0 conn(0x7f7a4806da40 0x7f7a480cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4eb37640 1 -- 172.21.15.122:0/1289793070 >> v2:172.21.15.169:3300/0 conn(0x7f7a4806da40 msgr2=0x7f7a480cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4eb37640 1 --2- 172.21.15.122:0/1289793070 >> v2:172.21.15.169:3300/0 conn(0x7f7a4806da40 0x7f7a480cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4eb37640 1 -- 172.21.15.122:0/1289793070 >> v1:172.21.15.122:6789/0 conn(0x7f7a481424f0 legacy=0x7f7a4806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.152 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.149+0000 7f7a4eb37640 1 -- 172.21.15.122:0/1289793070 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a480ccf40 con 0x7f7a480cd7a0 2024-03-06T07:32:39.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a4eb37640 1 --2- 172.21.15.122:0/1289793070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480cd7a0 0x7f7a4815cef0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f7a480b2a00 tx=0x7f7a4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0d692c36b448ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a4d334640 1 -- 172.21.15.122:0/1289793070 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40039070 con 0x7f7a480cd7a0 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a4d334640 1 -- 172.21.15.122:0/1289793070 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4002cc80 con 0x7f7a480cd7a0 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a4d334640 1 -- 172.21.15.122:0/1289793070 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40034660 con 0x7f7a480cd7a0 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a5059a640 1 -- 172.21.15.122:0/1289793070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480cd7a0 msgr2=0x7f7a4815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a5059a640 1 --2- 172.21.15.122:0/1289793070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480cd7a0 0x7f7a4815cef0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f7a480b2a00 tx=0x7f7a4002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.150+0000 7f7a5059a640 1 -- 172.21.15.122:0/1289793070 shutdown_connections 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.151+0000 7f7a5059a640 1 --2- 172.21.15.122:0/1289793070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a480cd7a0 0x7f7a4815cef0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.151+0000 7f7a5059a640 1 --2- 172.21.15.122:0/1289793070 >> v2:172.21.15.169:3300/0 conn(0x7f7a4806da40 0x7f7a480cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.151+0000 7f7a5059a640 1 -- 172.21.15.122:0/1289793070 >> 172.21.15.122:0/1289793070 conn(0x7f7a480cf8a0 msgr2=0x7f7a480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.151+0000 7f7a5059a640 1 -- 172.21.15.122:0/1289793070 shutdown_connections 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.151+0000 7f7a5059a640 1 -- 172.21.15.122:0/1289793070 wait complete. 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 Processor -- start 2024-03-06T07:32:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 -- start start 2024-03-06T07:32:39.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 0x7f7a4813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480cd7a0 0x7f7a481f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481424f0 0x7f7a481f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48071ce0 con 0x7f7a4806da40 2024-03-06T07:32:39.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48071b60 con 0x7f7a480cd7a0 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a5059a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48071e60 con 0x7f7a481424f0 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a4e336640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 0x7f7a4813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a4e336640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 0x7f7a4813f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48132/0 (socket says 172.21.15.122:48132) 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.152+0000 7f7a4e336640 1 -- 172.21.15.122:0/677808573 learned_addr learned my addr 172.21.15.122:0/677808573 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4eb37640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481424f0 0x7f7a481f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4db35640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480cd7a0 0x7f7a481f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4e336640 1 -- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481424f0 msgr2=0x7f7a481f5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4e336640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481424f0 0x7f7a481f5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4e336640 1 -- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480cd7a0 msgr2=0x7f7a481f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4e336640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480cd7a0 0x7f7a481f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4e336640 1 -- 172.21.15.122:0/677808573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a481f2350 con 0x7f7a4806da40 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4eb37640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481424f0 0x7f7a481f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a4e336640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 0x7f7a4813f4c0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c3c0 tx=0x7f7a3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38017070 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38005ce0 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a481f2630 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380072f0 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.153+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a481f5400 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.154+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a481f29d0 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.154+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a3801a070 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.155+0000 7f7a3effd640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.155+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7a380895c0 con 0x7f7a4806da40 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.155+0000 7f7a4db35640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.155+0000 7f7a4db35640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7a4813fa00 tx=0x7f7a44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.159+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a38058420 con 0x7f7a4806da40 2024-03-06T07:32:39.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.538+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a4804f330 con 0x7f7a4806da40 2024-03-06T07:32:39.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.539+0000 7f7a3effd640 1 -- 172.21.15.122:0/677808573 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f7a38057d20 con 0x7f7a4806da40 2024-03-06T07:32:39.541 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:39.541 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:39.541 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:39.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2c06b4d0 msgr2=0x7f7a2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7a4813fa00 tx=0x7f7a44008040 comp rx=0 tx=0).stop 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 msgr2=0x7f7a4813f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 0x7f7a4813f4c0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c3c0 tx=0x7f7a3800c880 comp rx=0 tx=0).stop 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 shutdown_connections 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7a481424f0 0x7f7a481f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7a480cd7a0 0x7f7a481f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 --2- 172.21.15.122:0/677808573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a4806da40 0x7f7a4813f4c0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 >> 172.21.15.122:0/677808573 conn(0x7f7a480cf8a0 msgr2=0x7f7a48072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 shutdown_connections 2024-03-06T07:32:39.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:39.541+0000 7f7a5059a640 1 -- 172.21.15.122:0/677808573 wait complete. 2024-03-06T07:32:39.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:39.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:40.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:40.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.855+0000 7f1ceee39640 1 Processor -- start 2024-03-06T07:32:40.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.855+0000 7f1ceee39640 1 -- start start 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.856+0000 7f1ceee39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d2710 0x7f1ce80b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.856+0000 7f1ceee39640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b3430 con 0x7f1ce815dfe0 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.856+0000 7f1ceee39640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b3710 con 0x7f1ce80d22b0 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.856+0000 7f1ceee39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80b31e0 con 0x7f1ce80d2710 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.856+0000 7f1cecbd5640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1ce815dfe0 0x7f1ce80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48700/0 (socket says 172.21.15.122:48700) 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.856+0000 7f1cecbd5640 1 -- 172.21.15.122:0/575814139 learned_addr learned my addr 172.21.15.122:0/575814139 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:40.861 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ced3d6640 1 --2- 172.21.15.122:0/575814139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d2710 0x7f1ce80b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104038003 0 0) 0x7f1ce80b3430 con 0x7f1ce815dfe0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8003520 con 0x7f1ce815dfe0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240904453 0 0) 0x7f1ce80b3710 con 0x7f1ce80d22b0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce80b3430 con 0x7f1ce80d22b0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541820197 0 0) 0x7f1cd8003520 con 0x7f1ce815dfe0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ced3d6640 1 -- 172.21.15.122:0/575814139 >> v1:172.21.15.169:6789/0 conn(0x7f1ce80d22b0 legacy=0x7f1ce80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ced3d6640 1 -- 172.21.15.122:0/575814139 >> v1:172.21.15.122:6789/0 conn(0x7f1ce815dfe0 legacy=0x7f1ce80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.857+0000 7f1ced3d6640 1 -- 172.21.15.122:0/575814139 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80b39f0 con 0x7f1ce80d2710 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.858+0000 7f1ced3d6640 1 --2- 172.21.15.122:0/575814139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d2710 0x7f1ce80b4900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80b3360 tx=0x7f1ce002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd879b0d2d8e9099 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.858+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0039070 con 0x7f1ce80d2710 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.858+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce002c910 con 0x7f1ce80d2710 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.858+0000 7f1ce77fe640 1 -- 172.21.15.122:0/575814139 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc001670 con 0x7f1ce80d2710 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 -- 172.21.15.122:0/575814139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d2710 msgr2=0x7f1ce80b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 --2- 172.21.15.122:0/575814139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d2710 0x7f1ce80b4900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80b3360 tx=0x7f1ce002bde0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 -- 172.21.15.122:0/575814139 shutdown_connections 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 --2- 172.21.15.122:0/575814139 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d2710 0x7f1ce80b4900 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 -- 172.21.15.122:0/575814139 >> 172.21.15.122:0/575814139 conn(0x7f1ce80d3cd0 msgr2=0x7f1ce80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 -- 172.21.15.122:0/575814139 shutdown_connections 2024-03-06T07:32:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.859+0000 7f1ceee39640 1 -- 172.21.15.122:0/575814139 wait complete. 2024-03-06T07:32:40.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.860+0000 7f1ceee39640 1 Processor -- start 2024-03-06T07:32:40.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.861+0000 7f1ceee39640 1 -- start start 2024-03-06T07:32:40.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ceee39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d22b0 0x7f1ce81f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ceee39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ce80d2710 0x7f1ce8150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ceee39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 0x7f1ce80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ceee39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80cfa00 con 0x7f1ce815dfe0 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ceee39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80b3d60 con 0x7f1ce80d2710 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ceee39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80d1c70 con 0x7f1ce80d22b0 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 0x7f1ce80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 0x7f1ce80ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48142/0 (socket says 172.21.15.122:48142) 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 -- 172.21.15.122:0/1003943086 learned_addr learned my addr 172.21.15.122:0/1003943086 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1cecbd5640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d22b0 0x7f1ce81f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ce7fff640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ce80d2710 0x7f1ce8150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 -- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d22b0 msgr2=0x7f1ce81f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d22b0 0x7f1ce81f7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 -- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ce80d2710 msgr2=0x7f1ce8150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ce80d2710 0x7f1ce8150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.862+0000 7f1ced3d6640 1 -- 172.21.15.122:0/1003943086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce81f7650 con 0x7f1ce815dfe0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.863+0000 7f1ced3d6640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 0x7f1ce80ccdb0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009d90 tx=0x7f1ce0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.863+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0039050 con 0x7f1ce815dfe0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.863+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce003a040 con 0x7f1ce815dfe0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.863+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce81f78f0 con 0x7f1ce815dfe0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.863+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0009450 con 0x7f1ce815dfe0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.863+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce81f7eb0 con 0x7f1ce815dfe0 2024-03-06T07:32:40.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.864+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce804f330 con 0x7f1ce815dfe0 2024-03-06T07:32:40.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.864+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ce0047020 con 0x7f1ce815dfe0 2024-03-06T07:32:40.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.865+0000 7f1ce57fa640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ccc06b4d0 0x7f1ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.865+0000 7f1cecbd5640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ccc06b4d0 0x7f1ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.865+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1ce00abfb0 con 0x7f1ce815dfe0 2024-03-06T07:32:40.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.865+0000 7f1cecbd5640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ccc06b4d0 0x7f1ccc06d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc000fd0 tx=0x7f1cdc001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:40.873+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ce004e020 con 0x7f1ce815dfe0 2024-03-06T07:32:41.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.086+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce80b5080 con 0x7f1ce815dfe0 2024-03-06T07:32:41.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.087+0000 7f1ce57fa640 1 -- 172.21.15.122:0/1003943086 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f1ce007a7b0 con 0x7f1ce815dfe0 2024-03-06T07:32:41.092 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:41.092 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ccc06b4d0 msgr2=0x7f1ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ccc06b4d0 0x7f1ccc06d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc000fd0 tx=0x7f1cdc001150 comp rx=0 tx=0).stop 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 msgr2=0x7f1ce80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 0x7f1ce80ccdb0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009d90 tx=0x7f1ce0004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 shutdown_connections 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1ccc06b4d0 0x7f1ccc06d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ce815dfe0 0x7f1ce80ccdb0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ce80d2710 0x7f1ce8150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 --2- 172.21.15.122:0/1003943086 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1ce80d22b0 0x7f1ce81f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.088+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 >> 172.21.15.122:0/1003943086 conn(0x7f1ce80d3cd0 msgr2=0x7f1ce80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.089+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 shutdown_connections 2024-03-06T07:32:41.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:41.089+0000 7f1ceee39640 1 -- 172.21.15.122:0/1003943086 wait complete. 2024-03-06T07:32:41.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:41.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:42.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:42.405 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.403+0000 7f84dd5a1640 1 Processor -- start 2024-03-06T07:32:42.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.404+0000 7f84dd5a1640 1 -- start start 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.404+0000 7f84dd5a1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f84d8072730 0x7f84d815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.404+0000 7f84dd5a1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.404+0000 7f84dd5a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d815eb40 con 0x7f84d8158600 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.404+0000 7f84dd5a1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d8161ba0 con 0x7f84d8072730 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.404+0000 7f84dd5a1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8072be0 con 0x7f84d8162030 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84cffff640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f84d8072730 0x7f84d815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:52308/0 (socket says 172.21.15.122:52308) 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 -- 172.21.15.122:0/3377829218 learned_addr learned my addr 172.21.15.122:0/3377829218 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 -- 172.21.15.122:0/3377829218 >> v2:172.21.15.169:3300/0 conn(0x7f84d8072730 msgr2=0x7f84d815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 --2- 172.21.15.122:0/3377829218 >> v2:172.21.15.169:3300/0 conn(0x7f84d8072730 0x7f84d815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 -- 172.21.15.122:0/3377829218 >> v1:172.21.15.169:6789/0 conn(0x7f84d8158600 legacy=0x7f84d806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84d77fe640 1 -- 172.21.15.122:0/3377829218 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d815ee20 con 0x7f84d8162030 2024-03-06T07:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.405+0000 7f84cffff640 1 --2- 172.21.15.122:0/3377829218 >> v2:172.21.15.169:3300/0 conn(0x7f84d8072730 0x7f84d815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:42.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.406+0000 7f84d77fe640 1 --2- 172.21.15.122:0/3377829218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81624c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009e50 tx=0x7f84d002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d3211c48f925824 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.406+0000 7f84d67fc640 1 -- 172.21.15.122:0/3377829218 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d002ca70 con 0x7f84d8162030 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.406+0000 7f84d67fc640 1 -- 172.21.15.122:0/3377829218 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d002cc10 con 0x7f84d8162030 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.406+0000 7f84d67fc640 1 -- 172.21.15.122:0/3377829218 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00343f0 con 0x7f84d8162030 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 -- 172.21.15.122:0/3377829218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 msgr2=0x7f84d81624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/3377829218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81624c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009e50 tx=0x7f84d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 -- 172.21.15.122:0/3377829218 shutdown_connections 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/3377829218 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81624c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/3377829218 >> v2:172.21.15.169:3300/0 conn(0x7f84d8072730 0x7f84d815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 -- 172.21.15.122:0/3377829218 >> 172.21.15.122:0/3377829218 conn(0x7f84d806c0c0 msgr2=0x7f84d806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 -- 172.21.15.122:0/3377829218 shutdown_connections 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.407+0000 7f84dd5a1640 1 -- 172.21.15.122:0/3377829218 wait complete. 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.408+0000 7f84dd5a1640 1 Processor -- start 2024-03-06T07:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.408+0000 7f84dd5a1640 1 -- start start 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.408+0000 7f84dd5a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 0x7f84d80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84d8158600 0x7f84d80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81f8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d806e080 con 0x7f84d8072730 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d815e8e0 con 0x7f84d8158600 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81f8850 con 0x7f84d8162030 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 0x7f84d80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 0x7f84d80aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48146/0 (socket says 172.21.15.122:48146) 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 -- 172.21.15.122:0/2738103042 learned_addr learned my addr 172.21.15.122:0/2738103042 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d77fe640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84cffff640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84d8158600 0x7f84d80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 -- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 msgr2=0x7f84d81f8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81f8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 -- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84d8158600 msgr2=0x7f84d80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84d8158600 0x7f84d80ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 -- 172.21.15.122:0/2738103042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80af260 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84d6ffd640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 0x7f84d80aeb80 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f84c800c430 tx=0x7f84c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c80052d0 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c8005490 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d80af540 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8007260 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.409+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d81f3ac0 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.410+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d804f330 con 0x7f84d8072730 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.411+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84c8007400 con 0x7f84d8072730 2024-03-06T07:32:42.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.411+0000 7f84cf7fe640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84b806b4d0 0x7f84b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.411+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f84c80895d0 con 0x7f84d8072730 2024-03-06T07:32:42.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.411+0000 7f84cffff640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84b806b4d0 0x7f84b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.412+0000 7f84cffff640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84b806b4d0 0x7f84b806d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009560 tx=0x7f84c0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.416+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84c8057e70 con 0x7f84d8072730 2024-03-06T07:32:42.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.632+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d80d06f0 con 0x7f84d8072730 2024-03-06T07:32:42.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.632+0000 7f84cf7fe640 1 -- 172.21.15.122:0/2738103042 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f84c805bf10 con 0x7f84d8072730 2024-03-06T07:32:42.634 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:42.634 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:42.634 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:42.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84b806b4d0 msgr2=0x7f84b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84b806b4d0 0x7f84b806d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009560 tx=0x7f84c0009190 comp rx=0 tx=0).stop 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 msgr2=0x7f84d80aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 0x7f84d80aeb80 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f84c800c430 tx=0x7f84c800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 shutdown_connections 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f84b806b4d0 0x7f84b806d990 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f84d8162030 0x7f84d81f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84d8158600 0x7f84d80ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 --2- 172.21.15.122:0/2738103042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84d8072730 0x7f84d80aeb80 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 >> 172.21.15.122:0/2738103042 conn(0x7f84d806c0c0 msgr2=0x7f84d80b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 shutdown_connections 2024-03-06T07:32:42.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:42.634+0000 7f84dd5a1640 1 -- 172.21.15.122:0/2738103042 wait complete. 2024-03-06T07:32:42.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:42.848 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:43.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.949+0000 7fda5df1e640 1 Processor -- start 2024-03-06T07:32:43.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.949+0000 7fda5df1e640 1 -- start start 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.950+0000 7fda5df1e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda5815dfe0 0x7fda580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.950+0000 7fda5df1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.950+0000 7fda5df1e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580b3430 con 0x7fda580d27f0 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.950+0000 7fda5df1e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580d2c50 con 0x7fda5815dfe0 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.950+0000 7fda5df1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580b31e0 con 0x7fda580cf830 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.950+0000 7fda577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda5815dfe0 0x7fda580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda577fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fda5815dfe0 0x7fda580cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48154/0 (socket says 172.21.15.122:48154) 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda577fe640 1 -- 172.21.15.122:0/3551815029 learned_addr learned my addr 172.21.15.122:0/3551815029 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda577fe640 1 -- 172.21.15.122:0/3551815029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 msgr2=0x7fda580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda56ffd640 1 --2- 172.21.15.122:0/3551815029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda577fe640 1 --2- 172.21.15.122:0/3551815029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda577fe640 1 -- 172.21.15.122:0/3551815029 >> v1:172.21.15.169:6789/0 conn(0x7fda580d27f0 legacy=0x7fda580d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.951+0000 7fda577fe640 1 -- 172.21.15.122:0/3551815029 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580b3710 con 0x7fda5815dfe0 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda577fe640 1 --2- 172.21.15.122:0/3551815029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cf2f0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fda48009e50 tx=0x7fda4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca61bb50835769c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda567fc640 1 -- 172.21.15.122:0/3551815029 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4802ca70 con 0x7fda5815dfe0 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda567fc640 1 -- 172.21.15.122:0/3551815029 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4802cc10 con 0x7fda5815dfe0 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda567fc640 1 -- 172.21.15.122:0/3551815029 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda480343f0 con 0x7fda5815dfe0 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 -- 172.21.15.122:0/3551815029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 msgr2=0x7fda580cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 --2- 172.21.15.122:0/3551815029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cf2f0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fda48009e50 tx=0x7fda4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 -- 172.21.15.122:0/3551815029 shutdown_connections 2024-03-06T07:32:43.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 --2- 172.21.15.122:0/3551815029 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 --2- 172.21.15.122:0/3551815029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cf2f0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 -- 172.21.15.122:0/3551815029 >> 172.21.15.122:0/3551815029 conn(0x7fda580d3cd0 msgr2=0x7fda580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.952+0000 7fda5df1e640 1 -- 172.21.15.122:0/3551815029 shutdown_connections 2024-03-06T07:32:43.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.953+0000 7fda5df1e640 1 -- 172.21.15.122:0/3551815029 wait complete. 2024-03-06T07:32:43.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.954+0000 7fda5df1e640 1 Processor -- start 2024-03-06T07:32:43.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.954+0000 7fda5df1e640 1 -- start start 2024-03-06T07:32:43.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.954+0000 7fda5df1e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.954+0000 7fda5df1e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580d27f0 0x7fda580cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.954+0000 7fda5df1e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda5df1e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d1c70 con 0x7fda5815dfe0 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda5df1e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580b3d60 con 0x7fda580d27f0 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda5df1e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d1df0 con 0x7fda580cf830 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48168/0 (socket says 172.21.15.122:48168) 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 -- 172.21.15.122:0/2786236380 learned_addr learned my addr 172.21.15.122:0/2786236380 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda577fe640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda56ffd640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580d27f0 0x7fda580cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 -- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 msgr2=0x7fda580b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 -- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580d27f0 msgr2=0x7fda580cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580d27f0 0x7fda580cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 -- 172.21.15.122:0/2786236380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580ccfb0 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda577fe640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda56ffd640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580d27f0 0x7fda580cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda57fff640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cba10 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c390 tx=0x7fda5000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50017070 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda500035f0 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda580cc470 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500039e0 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.955+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda580cc9f0 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.956+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda580ce3c0 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.957+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda50003b80 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.957+0000 7fda4ffff640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.957+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fda50089630 con 0x7fda5815dfe0 2024-03-06T07:32:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.957+0000 7fda577fe640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.962 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.958+0000 7fda577fe640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fda48035000 tx=0x7fda48030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:43.962+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda580ce3c0 con 0x7fda5815dfe0 2024-03-06T07:32:44.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.171+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda5815f000 con 0x7fda5815dfe0 2024-03-06T07:32:44.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.171+0000 7fda4ffff640 1 -- 172.21.15.122:0/2786236380 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fda50058410 con 0x7fda5815dfe0 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda3806b4d0 msgr2=0x7fda3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fda48035000 tx=0x7fda48030000 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 msgr2=0x7fda580cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cba10 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c390 tx=0x7fda5000c850 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 shutdown_connections 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda5815dfe0 0x7fda580cba10 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda580d27f0 0x7fda580cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 --2- 172.21.15.122:0/2786236380 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fda580cf830 0x7fda580b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.173+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 >> 172.21.15.122:0/2786236380 conn(0x7fda580d3cd0 msgr2=0x7fda5806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.174+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 shutdown_connections 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:44.174+0000 7fda5df1e640 1 -- 172.21.15.122:0/2786236380 wait complete. 2024-03-06T07:32:44.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:44.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:45.389 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:45.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.491+0000 7f6d3de71640 1 Processor -- start 2024-03-06T07:32:45.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.491+0000 7f6d3de71640 1 -- start start 2024-03-06T07:32:45.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d3de71640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f6d3815bb20 0x7f6d380d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d3de71640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d38161980 con 0x7f6d380cfff0 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d3de71640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d38161c60 con 0x7f6d380cd8c0 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d3de71640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d38161f40 con 0x7f6d3815bb20 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d36ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6d380cfff0 0x7f6d380ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54406/0 (socket says 172.21.15.122:54406) 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d36ffd640 1 -- 172.21.15.122:0/375726418 learned_addr learned my addr 172.21.15.122:0/375726418 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d377fe640 1 --2- 172.21.15.122:0/375726418 >> v2:172.21.15.169:3300/0 conn(0x7f6d3815bb20 0x7f6d380d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870282913 0 0) 0x7f6d38161980 con 0x7f6d380cfff0 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d20003520 con 0x7f6d380cfff0 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422790300 0 0) 0x7f6d20003520 con 0x7f6d380cfff0 2024-03-06T07:32:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 >> v2:172.21.15.169:3300/0 conn(0x7f6d3815bb20 msgr2=0x7f6d380d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 --2- 172.21.15.122:0/375726418 >> v2:172.21.15.169:3300/0 conn(0x7f6d3815bb20 0x7f6d380d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 >> v1:172.21.15.169:6789/0 conn(0x7f6d380cd8c0 legacy=0x7f6d380cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d38162220 con 0x7f6d380cfff0 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f6d28001670 con 0x7f6d380cfff0 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.492+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 >> v1:172.21.15.122:6789/0 conn(0x7f6d380cfff0 legacy=0x7f6d380ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d367fc640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d200039b0 0x7f6d20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d367fc640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d20004800 0x7f6d20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d367fc640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d200051c0 0x7f6d2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38161f40 con 0x7f6d200051c0 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2000c060 con 0x7f6d200039b0 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2000c260 con 0x7f6d20004800 2024-03-06T07:32:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d37fff640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d200051c0 0x7f6d2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d377fe640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d200039b0 0x7f6d20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d36ffd640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d20004800 0x7f6d20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d37fff640 1 -- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d20004800 msgr2=0x7f6d20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d37fff640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d20004800 0x7f6d20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d37fff640 1 -- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d200039b0 msgr2=0x7f6d20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d37fff640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d200039b0 0x7f6d20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d37fff640 1 -- 172.21.15.122:0/375726418 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d20003520 con 0x7f6d200051c0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.493+0000 7f6d36ffd640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d20004800 0x7f6d20004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d37fff640 1 -- 172.21.15.122:0/375726418 reap_dead start 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d37fff640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d200051c0 0x7f6d2000b9c0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f6d30001e50 tx=0x7f6d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48eb84a1991d960 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30001a70 con 0x7f6d200051c0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30003bb0 con 0x7f6d200051c0 2024-03-06T07:32:45.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d367fc640 1 -- 172.21.15.122:0/375726418 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d300050d0 con 0x7f6d200051c0 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 -- 172.21.15.122:0/375726418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d200051c0 msgr2=0x7f6d2000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d200051c0 0x7f6d2000b9c0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f6d30001e50 tx=0x7f6d20005830 comp rx=0 tx=0).stop 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 -- 172.21.15.122:0/375726418 shutdown_connections 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 --2- 172.21.15.122:0/375726418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d200051c0 0x7f6d2000b9c0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 -- 172.21.15.122:0/375726418 >> 172.21.15.122:0/375726418 conn(0x7f6d3809a9f0 msgr2=0x7f6d3809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 -- 172.21.15.122:0/375726418 shutdown_connections 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.494+0000 7f6d3de71640 1 -- 172.21.15.122:0/375726418 wait complete. 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.495+0000 7f6d3de71640 1 Processor -- start 2024-03-06T07:32:45.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 -- start start 2024-03-06T07:32:45.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d200051c0 0x7f6d38156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380c8b90 0x7f6d380b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 0x7f6d380b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380d1b00 con 0x7f6d380b70d0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380d1980 con 0x7f6d380c8b90 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d3de71640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380d04a0 con 0x7f6d200051c0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d37fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 0x7f6d380b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d37fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 0x7f6d380b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54550/0 (socket says 172.21.15.122:54550) 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.496+0000 7f6d37fff640 1 -- 172.21.15.122:0/1329002658 learned_addr learned my addr 172.21.15.122:0/1329002658 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d37fff640 1 -- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d200051c0 msgr2=0x7f6d38156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d37fff640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d200051c0 0x7f6d38156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d36ffd640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380c8b90 0x7f6d380b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d37fff640 1 -- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380c8b90 msgr2=0x7f6d380b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d37fff640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380c8b90 0x7f6d380b4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d37fff640 1 -- 172.21.15.122:0/1329002658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380aeda0 con 0x7f6d380b70d0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d37fff640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 0x7f6d380b50b0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f6d30005040 tx=0x7f6d300014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30006d60 con 0x7f6d380b70d0 2024-03-06T07:32:45.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30007030 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d380af080 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d300073e0 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.497+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d380af640 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.498+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3804f330 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.499+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d30007580 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.499+0000 7f6d1ffff640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d2406b500 0x7f6d2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.499+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6d300ab790 con 0x7f6d380b70d0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.499+0000 7f6d377fe640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d2406b500 0x7f6d2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.499+0000 7f6d377fe640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d2406b500 0x7f6d2406d9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6d3806df50 tx=0x7f6d2c009320 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.503+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d30033030 con 0x7f6d380b70d0 2024-03-06T07:32:45.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.712+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d380b1820 con 0x7f6d380b70d0 2024-03-06T07:32:45.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.712+0000 7f6d1ffff640 1 -- 172.21.15.122:0/1329002658 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f6d3007a030 con 0x7f6d380b70d0 2024-03-06T07:32:45.714 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:45.714 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:45.715 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:45.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d2406b500 msgr2=0x7f6d2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d2406b500 0x7f6d2406d9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6d3806df50 tx=0x7f6d2c009320 comp rx=0 tx=0).stop 2024-03-06T07:32:45.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 msgr2=0x7f6d380b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 0x7f6d380b50b0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f6d30005040 tx=0x7f6d300014a0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 shutdown_connections 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6d2406b500 0x7f6d2406d9c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d380b70d0 0x7f6d380b50b0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d380c8b90 0x7f6d380b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 --2- 172.21.15.122:0/1329002658 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6d200051c0 0x7f6d38156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 >> 172.21.15.122:0/1329002658 conn(0x7f6d3809a9f0 msgr2=0x7f6d380b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 shutdown_connections 2024-03-06T07:32:45.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:45.714+0000 7f6d3de71640 1 -- 172.21.15.122:0/1329002658 wait complete. 2024-03-06T07:32:45.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:45.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:46.929 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 Processor -- start 2024-03-06T07:32:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 -- start start 2024-03-06T07:32:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f183c15dfe0 0x7f183c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f183c0cf830 0x7f183c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183c0b3430 con 0x7f183c0d27f0 2024-03-06T07:32:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c0b31e0 con 0x7f183c0cf830 2024-03-06T07:32:47.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f18424b5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c0b3d60 con 0x7f183c15dfe0 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f183b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f183c0cf830 0x7f183c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f183b7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f183c0cf830 0x7f183c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54558/0 (socket says 172.21.15.122:54558) 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.029+0000 7f183b7fe640 1 -- 172.21.15.122:0/1611114492 learned_addr learned my addr 172.21.15.122:0/1611114492 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.030+0000 7f183b7fe640 1 -- 172.21.15.122:0/1611114492 >> v2:172.21.15.169:3300/0 conn(0x7f183c15dfe0 msgr2=0x7f183c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.030+0000 7f183b7fe640 1 --2- 172.21.15.122:0/1611114492 >> v2:172.21.15.169:3300/0 conn(0x7f183c15dfe0 0x7f183c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.030+0000 7f183b7fe640 1 -- 172.21.15.122:0/1611114492 >> v1:172.21.15.169:6789/0 conn(0x7f183c0d27f0 legacy=0x7f183c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.030+0000 7f183b7fe640 1 -- 172.21.15.122:0/1611114492 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c0b3710 con 0x7f183c0cf830 2024-03-06T07:32:47.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.030+0000 7f183b7fe640 1 --2- 172.21.15.122:0/1611114492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0cf830 0x7f183c0d22b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f1830009870 tx=0x7f183002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f68d02ed6d4ee84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f183affd640 1 -- 172.21.15.122:0/1611114492 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830035020 con 0x7f183c0cf830 2024-03-06T07:32:47.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f183affd640 1 -- 172.21.15.122:0/1611114492 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1830031870 con 0x7f183c0cf830 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f183affd640 1 -- 172.21.15.122:0/1611114492 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830031c40 con 0x7f183c0cf830 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 -- 172.21.15.122:0/1611114492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0cf830 msgr2=0x7f183c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 --2- 172.21.15.122:0/1611114492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0cf830 0x7f183c0d22b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f1830009870 tx=0x7f183002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 -- 172.21.15.122:0/1611114492 shutdown_connections 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 --2- 172.21.15.122:0/1611114492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0cf830 0x7f183c0d22b0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 --2- 172.21.15.122:0/1611114492 >> v2:172.21.15.169:3300/0 conn(0x7f183c15dfe0 0x7f183c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 -- 172.21.15.122:0/1611114492 >> 172.21.15.122:0/1611114492 conn(0x7f183c0d3cd0 msgr2=0x7f183c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 -- 172.21.15.122:0/1611114492 shutdown_connections 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.031+0000 7f18424b5640 1 -- 172.21.15.122:0/1611114492 wait complete. 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.032+0000 7f18424b5640 1 Processor -- start 2024-03-06T07:32:47.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.032+0000 7f18424b5640 1 -- start start 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f18424b5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f183c0cf830 0x7f183c14e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f18424b5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 0x7f183c0b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f18424b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f183c15dfe0 0x7f183c0b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f18424b5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0d1df0 con 0x7f183c0d27f0 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f18424b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0d1c70 con 0x7f183c15dfe0 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f18424b5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c15e3c0 con 0x7f183c0cf830 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f183b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 0x7f183c0b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f183b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 0x7f183c0b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54574/0 (socket says 172.21.15.122:54574) 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f183b7fe640 1 -- 172.21.15.122:0/423292227 learned_addr learned my addr 172.21.15.122:0/423292227 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:47.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f183bfff640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f183c0cf830 0x7f183c14e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.033+0000 7f1840a52640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f183c15dfe0 0x7f183c0b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183b7fe640 1 -- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f183c0cf830 msgr2=0x7f183c14e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183b7fe640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f183c0cf830 0x7f183c14e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183b7fe640 1 -- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f183c15dfe0 msgr2=0x7f183c0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183b7fe640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f183c15dfe0 0x7f183c0b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183b7fe640 1 -- 172.21.15.122:0/423292227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c14e7e0 con 0x7f183c0d27f0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183bfff640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f183c0cf830 0x7f183c14e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f183b7fe640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 0x7f183c0b1f70 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f1830009870 tx=0x7f18300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183003ad60 con 0x7f183c0d27f0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18300443f0 con 0x7f183c0d27f0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c15f1b0 con 0x7f183c0d27f0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18300447e0 con 0x7f183c0d27f0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.034+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c15f770 con 0x7f183c0d27f0 2024-03-06T07:32:47.039 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.035+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c04f330 con 0x7f183c0d27f0 2024-03-06T07:32:47.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.035+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f183004b060 con 0x7f183c0d27f0 2024-03-06T07:32:47.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.036+0000 7f1838ff9640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f182806b4d0 0x7f182806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.036+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f18300ac3e0 con 0x7f183c0d27f0 2024-03-06T07:32:47.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.036+0000 7f183bfff640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f182806b4d0 0x7f182806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.036+0000 7f183bfff640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f182806b4d0 0x7f182806d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f182c009930 tx=0x7f182c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.044+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f183003f070 con 0x7f183c0d27f0 2024-03-06T07:32:47.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.261+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f183c0b2f60 con 0x7f183c0d27f0 2024-03-06T07:32:47.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.261+0000 7f1838ff9640 1 -- 172.21.15.122:0/423292227 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f183002f320 con 0x7f183c0d27f0 2024-03-06T07:32:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:47.263 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:47.264 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:47.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f182806b4d0 msgr2=0x7f182806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f182806b4d0 0x7f182806d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f182c009930 tx=0x7f182c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 msgr2=0x7f183c0b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 0x7f183c0b1f70 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f1830009870 tx=0x7f18300046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 shutdown_connections 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f182806b4d0 0x7f182806d990 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f183c15dfe0 0x7f183c0b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f183c0d27f0 0x7f183c0b1f70 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 --2- 172.21.15.122:0/423292227 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f183c0cf830 0x7f183c14e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 >> 172.21.15.122:0/423292227 conn(0x7f183c0d3cd0 msgr2=0x7f183c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 shutdown_connections 2024-03-06T07:32:47.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:47.263+0000 7f18424b5640 1 -- 172.21.15.122:0/423292227 wait complete. 2024-03-06T07:32:47.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:47.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:48.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:48.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.604+0000 7f143f11e640 1 Processor -- start 2024-03-06T07:32:48.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.604+0000 7f143f11e640 1 -- start start 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143f11e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14381515e0 0x7f14380cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143f11e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f143815b430 0x7f1438071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143f11e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f14380cd7a0 0x7f14380cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143f11e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143815b8b0 con 0x7f143815b430 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143f11e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1438070a70 con 0x7f14380cd7a0 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143f11e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1438070d50 con 0x7f14381515e0 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f1437fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f143815b430 0x7f1438071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f1437fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f143815b430 0x7f1438071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54586/0 (socket says 172.21.15.122:54586) 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f1437fff640 1 -- 172.21.15.122:0/1458452571 learned_addr learned my addr 172.21.15.122:0/1458452571 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.605+0000 7f143ceba640 1 --2- 172.21.15.122:0/1458452571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14381515e0 0x7f14380cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f143d6bb640 1 --2- 172.21.15.122:0/1458452571 >> v2:172.21.15.169:3300/0 conn(0x7f14380cd7a0 0x7f14380cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f1437fff640 1 -- 172.21.15.122:0/1458452571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14381515e0 msgr2=0x7f14380cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f1437fff640 1 --2- 172.21.15.122:0/1458452571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14381515e0 0x7f14380cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f1437fff640 1 -- 172.21.15.122:0/1458452571 >> v2:172.21.15.169:3300/0 conn(0x7f14380cd7a0 msgr2=0x7f14380cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f1437fff640 1 --2- 172.21.15.122:0/1458452571 >> v2:172.21.15.169:3300/0 conn(0x7f14380cd7a0 0x7f14380cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f1437fff640 1 -- 172.21.15.122:0/1458452571 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438071030 con 0x7f143815b430 2024-03-06T07:32:48.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f1437fff640 1 --2- 172.21.15.122:0/1458452571 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f143815b430 0x7f1438071830 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f14380b3d60 tx=0x7f142c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47faa8de1306b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.606+0000 7f14377fe640 1 -- 172.21.15.122:0/1458452571 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c035020 con 0x7f143815b430 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f14377fe640 1 -- 172.21.15.122:0/1458452571 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c031870 con 0x7f143815b430 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f14377fe640 1 -- 172.21.15.122:0/1458452571 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c031c40 con 0x7f143815b430 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 -- 172.21.15.122:0/1458452571 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f143815b430 msgr2=0x7f1438071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 --2- 172.21.15.122:0/1458452571 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f143815b430 0x7f1438071830 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f14380b3d60 tx=0x7f142c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 -- 172.21.15.122:0/1458452571 shutdown_connections 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 --2- 172.21.15.122:0/1458452571 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f143815b430 0x7f1438071830 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 --2- 172.21.15.122:0/1458452571 >> v2:172.21.15.169:3300/0 conn(0x7f14380cd7a0 0x7f14380cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 --2- 172.21.15.122:0/1458452571 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f14381515e0 0x7f14380cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 -- 172.21.15.122:0/1458452571 >> 172.21.15.122:0/1458452571 conn(0x7f14380cf8a0 msgr2=0x7f14380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 -- 172.21.15.122:0/1458452571 shutdown_connections 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.607+0000 7f143f11e640 1 -- 172.21.15.122:0/1458452571 wait complete. 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.608+0000 7f143f11e640 1 Processor -- start 2024-03-06T07:32:48.610 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.608+0000 7f143f11e640 1 -- start start 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143f11e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14380cd7a0 0x7f14381f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143f11e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 0x7f14380720d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143f11e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f143815b430 0x7f1438072610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143f11e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1438071d70 con 0x7f14381515e0 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143f11e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14380ce110 con 0x7f14380cd7a0 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143f11e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1438071ef0 con 0x7f143815b430 2024-03-06T07:32:48.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 0x7f14380720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 0x7f14380720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54590/0 (socket says 172.21.15.122:54590) 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 -- 172.21.15.122:0/1036267684 learned_addr learned my addr 172.21.15.122:0/1036267684 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143d6bb640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f143815b430 0x7f1438072610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f143ceba640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14380cd7a0 0x7f14381f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 -- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f143815b430 msgr2=0x7f1438072610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f143815b430 0x7f1438072610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 -- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14380cd7a0 msgr2=0x7f14381f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.609+0000 7f1437fff640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14380cd7a0 0x7f14381f1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f1437fff640 1 -- 172.21.15.122:0/1036267684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438154f70 con 0x7f14381515e0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f1437fff640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 0x7f14380720d0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f142c02b950 tx=0x7f142c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c040070 con 0x7f14381515e0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c030d00 con 0x7f14381515e0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143813e490 con 0x7f14381515e0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c035040 con 0x7f14381515e0 2024-03-06T07:32:48.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.610+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143813ea50 con 0x7f14381515e0 2024-03-06T07:32:48.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.611+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1438068840 con 0x7f14381515e0 2024-03-06T07:32:48.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.611+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f142c04b060 con 0x7f14381515e0 2024-03-06T07:32:48.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.612+0000 7f14357fa640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f142406b500 0x7f142406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.612+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f142c0abf90 con 0x7f14381515e0 2024-03-06T07:32:48.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.612+0000 7f143ceba640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f142406b500 0x7f142406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.612+0000 7f143ceba640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f142406b500 0x7f142406d9c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1428006fd0 tx=0x7f1428008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.616+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f142c07adf0 con 0x7f14381515e0 2024-03-06T07:32:48.907 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.906+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f143804f330 con 0x7f14381515e0 2024-03-06T07:32:48.908 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.907+0000 7f14357fa640 1 -- 172.21.15.122:0/1036267684 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f142c07a6f0 con 0x7f14381515e0 2024-03-06T07:32:48.909 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:48.909 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:48.909 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:48.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f142406b500 msgr2=0x7f142406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f142406b500 0x7f142406d9c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1428006fd0 tx=0x7f1428008040 comp rx=0 tx=0).stop 2024-03-06T07:32:48.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 msgr2=0x7f14380720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 0x7f14380720d0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f142c02b950 tx=0x7f142c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.910 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 shutdown_connections 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f142406b500 0x7f142406d9c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f143815b430 0x7f1438072610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14381515e0 0x7f14380720d0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 --2- 172.21.15.122:0/1036267684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14380cd7a0 0x7f14381f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 >> 172.21.15.122:0/1036267684 conn(0x7f14380cf8a0 msgr2=0x7f143814fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 shutdown_connections 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:48.909+0000 7f143f11e640 1 -- 172.21.15.122:0/1036267684 wait complete. 2024-03-06T07:32:49.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:49.122 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:50.123 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:50.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde928c9640 1 Processor -- start 2024-03-06T07:32:50.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde928c9640 1 -- start start 2024-03-06T07:32:50.229 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde928c9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fde8c15dfe0 0x7fde8c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde928c9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c0b3430 con 0x7fde8c0cf830 2024-03-06T07:32:50.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde928c9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c0b3710 con 0x7fde8c0d29c0 2024-03-06T07:32:50.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde928c9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8c0b31e0 con 0x7fde8c15dfe0 2024-03-06T07:32:50.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.224+0000 7fde8bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fde8c15dfe0 0x7fde8c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.230 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fde8c0cf830 0x7fde8c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54422/0 (socket says 172.21.15.122:54422) 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fde8c15dfe0 0x7fde8c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54592/0 (socket says 172.21.15.122:54592) 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8b7fe640 1 -- 172.21.15.122:0/303983917 learned_addr learned my addr 172.21.15.122:0/303983917 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8bfff640 1 -- 172.21.15.122:0/303983917 >> v1:172.21.15.169:6789/0 conn(0x7fde8c0d29c0 legacy=0x7fde8c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8bfff640 1 -- 172.21.15.122:0/303983917 >> v1:172.21.15.122:6789/0 conn(0x7fde8c0cf830 legacy=0x7fde8c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8affd640 1 -- 172.21.15.122:0/303983917 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636131632 0 0) 0x7fde8c0b3430 con 0x7fde8c0cf830 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.225+0000 7fde8bfff640 1 -- 172.21.15.122:0/303983917 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0b39f0 con 0x7fde8c15dfe0 2024-03-06T07:32:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde8bfff640 1 --2- 172.21.15.122:0/303983917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c15dfe0 0x7fde8c0cf2f0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0b3360 tx=0x7fde7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602ff07b55ec1f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde8affd640 1 -- 172.21.15.122:0/303983917 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c035020 con 0x7fde8c15dfe0 2024-03-06T07:32:50.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde8affd640 1 -- 172.21.15.122:0/303983917 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c031830 con 0x7fde8c15dfe0 2024-03-06T07:32:50.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde8affd640 1 -- 172.21.15.122:0/303983917 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c031c00 con 0x7fde8c15dfe0 2024-03-06T07:32:50.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde928c9640 1 -- 172.21.15.122:0/303983917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c15dfe0 msgr2=0x7fde8c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde928c9640 1 --2- 172.21.15.122:0/303983917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c15dfe0 0x7fde8c0cf2f0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0b3360 tx=0x7fde7c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.232 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde928c9640 1 -- 172.21.15.122:0/303983917 shutdown_connections 2024-03-06T07:32:50.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde928c9640 1 --2- 172.21.15.122:0/303983917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c15dfe0 0x7fde8c0cf2f0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.226+0000 7fde928c9640 1 -- 172.21.15.122:0/303983917 >> 172.21.15.122:0/303983917 conn(0x7fde8c0d3cd0 msgr2=0x7fde8c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.227+0000 7fde928c9640 1 -- 172.21.15.122:0/303983917 shutdown_connections 2024-03-06T07:32:50.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.227+0000 7fde928c9640 1 -- 172.21.15.122:0/303983917 wait complete. 2024-03-06T07:32:50.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 Processor -- start 2024-03-06T07:32:50.233 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 -- start start 2024-03-06T07:32:50.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde8c0cf830 0x7fde8c150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 0x7fde8c09fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fde8c15dfe0 0x7fde8c0a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c15e3c0 con 0x7fde8c0d29c0 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0b3d60 con 0x7fde8c0cf830 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.228+0000 7fde928c9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c15e540 con 0x7fde8c15dfe0 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 0x7fde8c09fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 0x7fde8c09fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54604/0 (socket says 172.21.15.122:54604) 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 -- 172.21.15.122:0/3846761522 learned_addr learned my addr 172.21.15.122:0/3846761522 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 -- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fde8c15dfe0 msgr2=0x7fde8c0a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde90e66640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fde8c15dfe0 0x7fde8c0a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fde8c15dfe0 0x7fde8c0a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 -- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde8c0cf830 msgr2=0x7fde8c150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde8c0cf830 0x7fde8c150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 -- 172.21.15.122:0/3846761522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0a0850 con 0x7fde8c0d29c0 2024-03-06T07:32:50.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde8b7fe640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 0x7fde8c09fad0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fde8000c4b0 tx=0x7fde8000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80018070 con 0x7fde8c0d29c0 2024-03-06T07:32:50.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80007500 con 0x7fde8c0d29c0 2024-03-06T07:32:50.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80004a20 con 0x7fde8c0d29c0 2024-03-06T07:32:50.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c0a0b30 con 0x7fde8c0d29c0 2024-03-06T07:32:50.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.229+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c0a10f0 con 0x7fde8c0d29c0 2024-03-06T07:32:50.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.230+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c04f330 con 0x7fde8c0d29c0 2024-03-06T07:32:50.241 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.231+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde8001b070 con 0x7fde8c0d29c0 2024-03-06T07:32:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.231+0000 7fde88ff9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fde7806b4d0 0x7fde7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.231+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fde80089450 con 0x7fde8c0d29c0 2024-03-06T07:32:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.231+0000 7fde8bfff640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fde7806b4d0 0x7fde7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.231+0000 7fde8bfff640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fde7806b4d0 0x7fde7806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fde7c036000 tx=0x7fde7c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.242 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.235+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde80058400 con 0x7fde8c0d29c0 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.463+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8c06c3f0 con 0x7fde8c0d29c0 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.463+0000 7fde88ff9640 1 -- 172.21.15.122:0/3846761522 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fde80006e20 con 0x7fde8c0d29c0 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:50.466 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fde7806b4d0 msgr2=0x7fde7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fde7806b4d0 0x7fde7806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fde7c036000 tx=0x7fde7c030000 comp rx=0 tx=0).stop 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 msgr2=0x7fde8c09fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 0x7fde8c09fad0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fde8000c4b0 tx=0x7fde8000c970 comp rx=0 tx=0).stop 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 shutdown_connections 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fde7806b4d0 0x7fde7806d990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fde8c15dfe0 0x7fde8c0a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.468 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde8c0d29c0 0x7fde8c09fad0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 --2- 172.21.15.122:0/3846761522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde8c0cf830 0x7fde8c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 >> 172.21.15.122:0/3846761522 conn(0x7fde8c0d3cd0 msgr2=0x7fde8c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 shutdown_connections 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:50.466+0000 7fde928c9640 1 -- 172.21.15.122:0/3846761522 wait complete. 2024-03-06T07:32:50.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:50.680 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:51.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.781+0000 7fb28adb5640 1 Processor -- start 2024-03-06T07:32:51.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.781+0000 7fb28adb5640 1 -- start start 2024-03-06T07:32:51.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb28adb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 0x7fb2840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb28adb5640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fb2840cf830 0x7fb2840d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb28adb5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2840b3430 con 0x7fb2840d27f0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb28adb5640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2840d2c50 con 0x7fb2840cf830 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb28adb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840b31e0 con 0x7fb28415dfe0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb289352640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb2840d27f0 0x7fb2840d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54438/0 (socket says 172.21.15.122:54438) 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.782+0000 7fb289352640 1 -- 172.21.15.122:0/3102893640 learned_addr learned my addr 172.21.15.122:0/3102893640 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb288b51640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 0x7fb2840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 --2- 172.21.15.122:0/3102893640 >> v2:172.21.15.169:3300/0 conn(0x7fb2840cf830 0x7fb2840d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952273725 0 0) 0x7fb2840b3430 con 0x7fb2840d27f0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb274003520 con 0x7fb2840d27f0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3907178948 0 0) 0x7fb274003520 con 0x7fb2840d27f0 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 msgr2=0x7fb2840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 0x7fb2840cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 >> v2:172.21.15.169:3300/0 conn(0x7fb2840cf830 msgr2=0x7fb2840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 --2- 172.21.15.122:0/3102893640 >> v2:172.21.15.169:3300/0 conn(0x7fb2840cf830 0x7fb2840d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840b3710 con 0x7fb2840d27f0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fb27c0019f0 con 0x7fb2840d27f0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 >> v1:172.21.15.122:6789/0 conn(0x7fb2840d27f0 legacy=0x7fb2840d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2740039b0 0x7fb274003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb274004800 0x7fb274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 0x7fb2740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840b31e0 con 0x7fb274004800 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27400b9c0 con 0x7fb2740039b0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27400bb40 con 0x7fb28415dfe0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb274004800 0x7fb274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb288b51640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2740039b0 0x7fb274003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb289352640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 0x7fb2740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 -- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 msgr2=0x7fb2740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415dfe0 0x7fb2740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 -- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2740039b0 msgr2=0x7fb274003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2740039b0 0x7fb274003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.783+0000 7fb283fff640 1 -- 172.21.15.122:0/3102893640 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274003520 con 0x7fb274004800 2024-03-06T07:32:51.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.784+0000 7fb283fff640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb274004800 0x7fb274004c80 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fb270009a90 tx=0x7fb274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4162108714cf25b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.784+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270039070 con 0x7fb274004800 2024-03-06T07:32:51.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.784+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27002c6b0 con 0x7fb274004800 2024-03-06T07:32:51.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.784+0000 7fb2837fe640 1 -- 172.21.15.122:0/3102893640 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270009690 con 0x7fb274004800 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb28adb5640 1 -- 172.21.15.122:0/3102893640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb274004800 msgr2=0x7fb274004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb28adb5640 1 --2- 172.21.15.122:0/3102893640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb274004800 0x7fb274004c80 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fb270009a90 tx=0x7fb274005830 comp rx=0 tx=0).stop 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb289352640 1 -- 172.21.15.122:0/3102893640 reap_dead start 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb28adb5640 1 -- 172.21.15.122:0/3102893640 shutdown_connections 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb28adb5640 1 -- 172.21.15.122:0/3102893640 >> 172.21.15.122:0/3102893640 conn(0x7fb2840d3cd0 msgr2=0x7fb2840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb28adb5640 1 -- 172.21.15.122:0/3102893640 shutdown_connections 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.785+0000 7fb28adb5640 1 -- 172.21.15.122:0/3102893640 wait complete. 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.786+0000 7fb28adb5640 1 Processor -- start 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.786+0000 7fb28adb5640 1 -- start start 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb28adb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb274004800 0x7fb28415e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb28adb5640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415fb60 0x7fb28415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb28adb5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 0x7fb2841f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb28adb5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840d1c70 con 0x7fb284160b30 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb28adb5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840b3d60 con 0x7fb274004800 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb28adb5640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840d1df0 con 0x7fb28415fb60 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 0x7fb2841f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 0x7fb2841f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54624/0 (socket says 172.21.15.122:54624) 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 -- 172.21.15.122:0/547693393 learned_addr learned my addr 172.21.15.122:0/547693393 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb288b51640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb274004800 0x7fb28415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb283fff640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415fb60 0x7fb28415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 -- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415fb60 msgr2=0x7fb28415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415fb60 0x7fb28415c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 -- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb274004800 msgr2=0x7fb28415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb274004800 0x7fb28415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.787+0000 7fb289352640 1 -- 172.21.15.122:0/547693393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28415eac0 con 0x7fb284160b30 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb288b51640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb274004800 0x7fb28415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:51.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb283fff640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415fb60 0x7fb28415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb289352640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 0x7fb2841f6fc0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fb2840cff40 tx=0x7fb27c007ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c017020 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c00abe0 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28415eda0 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0133f0 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.788+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28415f360 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.789+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28404f330 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.789+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb27c01a020 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.790+0000 7fb2817fa640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb26806b500 0x7fb26806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.790+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb27c08a940 con 0x7fb284160b30 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.790+0000 7fb288b51640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb26806b500 0x7fb26806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.795 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.790+0000 7fb288b51640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb26806b500 0x7fb26806d9c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb2780099e0 tx=0x7fb278008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:51.794+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb27c021020 con 0x7fb284160b30 2024-03-06T07:32:52.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.008+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2840b4500 con 0x7fb284160b30 2024-03-06T07:32:52.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.009+0000 7fb2817fa640 1 -- 172.21.15.122:0/547693393 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fb27c059830 con 0x7fb284160b30 2024-03-06T07:32:52.011 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:52.011 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:52.011 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:52.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb26806b500 msgr2=0x7fb26806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb26806b500 0x7fb26806d9c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb2780099e0 tx=0x7fb278008040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 msgr2=0x7fb2841f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 0x7fb2841f6fc0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fb2840cff40 tx=0x7fb27c007ae0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 shutdown_connections 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fb26806b500 0x7fb26806d9c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb284160b30 0x7fb2841f6fc0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fb28415fb60 0x7fb28415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 --2- 172.21.15.122:0/547693393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb274004800 0x7fb28415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 >> 172.21.15.122:0/547693393 conn(0x7fb2840d3cd0 msgr2=0x7fb2840b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 shutdown_connections 2024-03-06T07:32:52.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:52.011+0000 7fb28adb5640 1 -- 172.21.15.122:0/547693393 wait complete. 2024-03-06T07:32:52.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:52.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:53.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:53.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.330+0000 7f4284b3a640 1 Processor -- start 2024-03-06T07:32:53.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.330+0000 7f4284b3a640 1 -- start start 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.330+0000 7f4284b3a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f428015dfe0 0x7f42800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.330+0000 7f4284b3a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f42800cf830 0x7f42800d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.330+0000 7f4284b3a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42800b3430 con 0x7f42800d27f0 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f4284b3a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42800b31e0 con 0x7f428015dfe0 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f4284b3a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42800b3d60 con 0x7f42800cf830 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f428015dfe0 0x7f42800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f428015dfe0 0x7f42800cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54636/0 (socket says 172.21.15.122:54636) 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 -- 172.21.15.122:0/2907739158 learned_addr learned my addr 172.21.15.122:0/2907739158 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427dd9b640 1 --2- 172.21.15.122:0/2907739158 >> v2:172.21.15.169:3300/0 conn(0x7f42800cf830 0x7f42800d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 -- 172.21.15.122:0/2907739158 >> v2:172.21.15.169:3300/0 conn(0x7f42800cf830 msgr2=0x7f42800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 --2- 172.21.15.122:0/2907739158 >> v2:172.21.15.169:3300/0 conn(0x7f42800cf830 0x7f42800d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 -- 172.21.15.122:0/2907739158 >> v1:172.21.15.169:6789/0 conn(0x7f42800d27f0 legacy=0x7f42800d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.331+0000 7f427e59c640 1 -- 172.21.15.122:0/2907739158 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800b3710 con 0x7f428015dfe0 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.332+0000 7f427e59c640 1 --2- 172.21.15.122:0/2907739158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f428015dfe0 0x7f42800cf2f0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f4274009870 tx=0x7f427402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69160545251188fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.332+0000 7f427d59a640 1 -- 172.21.15.122:0/2907739158 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274035020 con 0x7f428015dfe0 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.332+0000 7f427d59a640 1 -- 172.21.15.122:0/2907739158 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4274031870 con 0x7f428015dfe0 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.332+0000 7f427d59a640 1 -- 172.21.15.122:0/2907739158 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274031c40 con 0x7f428015dfe0 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 -- 172.21.15.122:0/2907739158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f428015dfe0 msgr2=0x7f42800cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2907739158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f428015dfe0 0x7f42800cf2f0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f4274009870 tx=0x7f427402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 -- 172.21.15.122:0/2907739158 shutdown_connections 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2907739158 >> v2:172.21.15.169:3300/0 conn(0x7f42800cf830 0x7f42800d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2907739158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f428015dfe0 0x7f42800cf2f0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 -- 172.21.15.122:0/2907739158 >> 172.21.15.122:0/2907739158 conn(0x7f42800d3cd0 msgr2=0x7f42800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 -- 172.21.15.122:0/2907739158 shutdown_connections 2024-03-06T07:32:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.333+0000 7f4284b3a640 1 -- 172.21.15.122:0/2907739158 wait complete. 2024-03-06T07:32:53.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.334+0000 7f4284b3a640 1 Processor -- start 2024-03-06T07:32:53.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 -- start start 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42800cf830 0x7f42800b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 0x7f428014de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428015dfe0 0x7f4280161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42800d1df0 con 0x7f42800d27f0 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42800d1c70 con 0x7f428015dfe0 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f4284b3a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428015e3c0 con 0x7f42800cf830 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f427dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 0x7f428014de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f427dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 0x7f428014de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54648/0 (socket says 172.21.15.122:54648) 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f427dd9b640 1 -- 172.21.15.122:0/2255861510 learned_addr learned my addr 172.21.15.122:0/2255861510 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f427e59c640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42800cf830 0x7f42800b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.335+0000 7f427ed9d640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428015dfe0 0x7f4280161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f427dd9b640 1 -- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42800cf830 msgr2=0x7f42800b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f427dd9b640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42800cf830 0x7f42800b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f427dd9b640 1 -- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428015dfe0 msgr2=0x7f4280161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f427dd9b640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428015dfe0 0x7f4280161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f427dd9b640 1 -- 172.21.15.122:0/2255861510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800b9070 con 0x7f42800d27f0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f427dd9b640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 0x7f428014de40 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f427000c7c0 tx=0x7f427000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270015070 con 0x7f42800d27f0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4270007820 con 0x7f42800d27f0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42800b9310 con 0x7f42800d27f0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270007bd0 con 0x7f42800d27f0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.336+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42800b98d0 con 0x7f42800d27f0 2024-03-06T07:32:53.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.337+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428004f330 con 0x7f42800d27f0 2024-03-06T07:32:53.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.338+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4270018070 con 0x7f42800d27f0 2024-03-06T07:32:53.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.338+0000 7f426effd640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426006b4d0 0x7f426006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.338+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4270089f90 con 0x7f42800d27f0 2024-03-06T07:32:53.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.338+0000 7f427e59c640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426006b4d0 0x7f426006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.339+0000 7f427e59c640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426006b4d0 0x7f426006d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4274036000 tx=0x7f427403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.344 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.343+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42700587a0 con 0x7f42800d27f0 2024-03-06T07:32:53.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.603+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f428015d470 con 0x7f42800d27f0 2024-03-06T07:32:53.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.603+0000 7f426effd640 1 -- 172.21.15.122:0/2255861510 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f427005c840 con 0x7f42800d27f0 2024-03-06T07:32:53.606 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:53.606 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:53.606 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:53.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426006b4d0 msgr2=0x7f426006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426006b4d0 0x7f426006d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4274036000 tx=0x7f427403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 msgr2=0x7f428014de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 0x7f428014de40 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f427000c7c0 tx=0x7f427000cc80 comp rx=0 tx=0).stop 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 shutdown_connections 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f426006b4d0 0x7f426006d990 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428015dfe0 0x7f4280161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42800d27f0 0x7f428014de40 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 --2- 172.21.15.122:0/2255861510 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f42800cf830 0x7f42800b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 >> 172.21.15.122:0/2255861510 conn(0x7f42800d3cd0 msgr2=0x7f42800cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 shutdown_connections 2024-03-06T07:32:53.608 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:53.605+0000 7f4284b3a640 1 -- 172.21.15.122:0/2255861510 wait complete. 2024-03-06T07:32:53.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:53.820 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:54.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:54.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.919+0000 7f4e04a95640 1 Processor -- start 2024-03-06T07:32:54.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4e04a95640 1 -- start start 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4e04a95640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4e0015dfe0 0x7f4e000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4e04a95640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4e000d29c0 0x7f4e000b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4e04a95640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e000b3430 con 0x7f4e000cf830 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4e04a95640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e000b31e0 con 0x7f4e0015dfe0 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4e04a95640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e000b3d60 con 0x7f4e000d29c0 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4dfe59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4e0015dfe0 0x7f4e000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4df7fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4e000cf830 0x7f4e000d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52458/0 (socket says 172.21.15.122:52458) 2024-03-06T07:32:54.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.920+0000 7f4df7fff640 1 -- 172.21.15.122:0/1675746632 learned_addr learned my addr 172.21.15.122:0/1675746632 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:32:54.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.921+0000 7f4dfed9d640 1 --2- 172.21.15.122:0/1675746632 >> v2:172.21.15.169:3300/0 conn(0x7f4e000d29c0 0x7f4e000b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.921+0000 7f4dfe59c640 1 -- 172.21.15.122:0/1675746632 >> v2:172.21.15.169:3300/0 conn(0x7f4e000d29c0 msgr2=0x7f4e000b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.921+0000 7f4dfe59c640 1 --2- 172.21.15.122:0/1675746632 >> v2:172.21.15.169:3300/0 conn(0x7f4e000d29c0 0x7f4e000b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.921+0000 7f4dfe59c640 1 -- 172.21.15.122:0/1675746632 >> v1:172.21.15.122:6789/0 conn(0x7f4e000cf830 legacy=0x7f4e000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.921+0000 7f4dfe59c640 1 -- 172.21.15.122:0/1675746632 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000b3710 con 0x7f4e0015dfe0 2024-03-06T07:32:54.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.921+0000 7f4dfdd9b640 1 -- 172.21.15.122:0/1675746632 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328391146 0 0) 0x7f4e000b3430 con 0x7f4e000cf830 2024-03-06T07:32:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.922+0000 7f4dfe59c640 1 --2- 172.21.15.122:0/1675746632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0015dfe0 0x7f4e000cf2f0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f4e000b3cd0 tx=0x7f4de802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc5673c9ee5b72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.922+0000 7f4dfdd9b640 1 -- 172.21.15.122:0/1675746632 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de8035020 con 0x7f4e0015dfe0 2024-03-06T07:32:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.922+0000 7f4dfdd9b640 1 -- 172.21.15.122:0/1675746632 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de8031830 con 0x7f4e0015dfe0 2024-03-06T07:32:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.922+0000 7f4dfdd9b640 1 -- 172.21.15.122:0/1675746632 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de8031c00 con 0x7f4e0015dfe0 2024-03-06T07:32:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 -- 172.21.15.122:0/1675746632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0015dfe0 msgr2=0x7f4e000cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1675746632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0015dfe0 0x7f4e000cf2f0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f4e000b3cd0 tx=0x7f4de802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 -- 172.21.15.122:0/1675746632 shutdown_connections 2024-03-06T07:32:54.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1675746632 >> v2:172.21.15.169:3300/0 conn(0x7f4e000d29c0 0x7f4e000b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1675746632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0015dfe0 0x7f4e000cf2f0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 -- 172.21.15.122:0/1675746632 >> 172.21.15.122:0/1675746632 conn(0x7f4e000d3cd0 msgr2=0x7f4e000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 -- 172.21.15.122:0/1675746632 shutdown_connections 2024-03-06T07:32:54.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.923+0000 7f4e04a95640 1 -- 172.21.15.122:0/1675746632 wait complete. 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.924+0000 7f4e04a95640 1 Processor -- start 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.924+0000 7f4e04a95640 1 -- start start 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4e04a95640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e000cf830 0x7f4e0015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4e04a95640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 0x7f4e001f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4e04a95640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e0015dfe0 0x7f4e000b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4e04a95640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000d1df0 con 0x7f4e000d29c0 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4e04a95640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000d1c70 con 0x7f4e000cf830 2024-03-06T07:32:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4e04a95640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0015e3c0 con 0x7f4e0015dfe0 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 0x7f4e001f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 0x7f4e001f4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51020/0 (socket says 172.21.15.122:51020) 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 -- 172.21.15.122:0/1122673989 learned_addr learned my addr 172.21.15.122:0/1122673989 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4dfe59c640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e000cf830 0x7f4e0015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4dfed9d640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e0015dfe0 0x7f4e000b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 -- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e0015dfe0 msgr2=0x7f4e000b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e0015dfe0 0x7f4e000b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 -- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e000cf830 msgr2=0x7f4e0015bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e000cf830 0x7f4e0015bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 -- 172.21.15.122:0/1122673989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e001f7280 con 0x7f4e000d29c0 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4dfed9d640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e0015dfe0 0x7f4e000b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4dfe59c640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e000cf830 0x7f4e0015bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:54.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.925+0000 7f4df7fff640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 0x7f4e001f4300 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f4df000c4b0 tx=0x7f4df000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.926+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0018070 con 0x7f4e000d29c0 2024-03-06T07:32:54.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.926+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df0007500 con 0x7f4e000d29c0 2024-03-06T07:32:54.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.926+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0004a20 con 0x7f4e000d29c0 2024-03-06T07:32:54.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.926+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e001f7560 con 0x7f4e000d29c0 2024-03-06T07:32:54.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.926+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e001f7b20 con 0x7f4e000d29c0 2024-03-06T07:32:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.927+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e001f7d60 con 0x7f4e000d29c0 2024-03-06T07:32:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.927+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4df001b070 con 0x7f4e000d29c0 2024-03-06T07:32:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.927+0000 7f4df6ffd640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4de006b500 0x7f4de006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.927+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4df00893e0 con 0x7f4e000d29c0 2024-03-06T07:32:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.927+0000 7f4dfe59c640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4de006b500 0x7f4de006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.928+0000 7f4dfe59c640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4de006b500 0x7f4de006d9c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4de8036000 tx=0x7f4de8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:54.932+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e001f7d60 con 0x7f4e000d29c0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.142+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0004f330 con 0x7f4e000d29c0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.142+0000 7f4df6ffd640 1 -- 172.21.15.122:0/1122673989 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f4df0058390 con 0x7f4e000d29c0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:55.145 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:55.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4de006b500 msgr2=0x7f4de006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4de006b500 0x7f4de006d9c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4de8036000 tx=0x7f4de8008030 comp rx=0 tx=0).stop 2024-03-06T07:32:55.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 msgr2=0x7f4e001f4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 0x7f4e001f4300 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f4df000c4b0 tx=0x7f4df000c970 comp rx=0 tx=0).stop 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 shutdown_connections 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4de006b500 0x7f4de006d9c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4e0015dfe0 0x7f4e000b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e000d29c0 0x7f4e001f4300 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 --2- 172.21.15.122:0/1122673989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e000cf830 0x7f4e0015bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 >> 172.21.15.122:0/1122673989 conn(0x7f4e000d3cd0 msgr2=0x7f4e000ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 shutdown_connections 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:55.144+0000 7f4e04a95640 1 -- 172.21.15.122:0/1122673989 wait complete. 2024-03-06T07:32:55.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:55.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:55.855 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:55.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:56.359 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:56.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.461+0000 7f6c9364e640 1 Processor -- start 2024-03-06T07:32:56.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.461+0000 7f6c9364e640 1 -- start start 2024-03-06T07:32:56.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c9364e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c0b3060 0x7f6c8c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c9364e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6c8c15bf30 0x7f6c8c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c9364e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c15f2d0 con 0x7f6c8c13dc10 2024-03-06T07:32:56.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c9364e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0cf3f0 con 0x7f6c8c15bf30 2024-03-06T07:32:56.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c9364e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c15b210 con 0x7f6c8c0b3060 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6c8c15bf30 0x7f6c8c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6c8c15bf30 0x7f6c8c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51030/0 (socket says 172.21.15.122:51030) 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 -- 172.21.15.122:0/1595610777 learned_addr learned my addr 172.21.15.122:0/1595610777 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c90be9640 1 --2- 172.21.15.122:0/1595610777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c0b3060 0x7f6c8c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 -- 172.21.15.122:0/1595610777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c0b3060 msgr2=0x7f6c8c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 --2- 172.21.15.122:0/1595610777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c0b3060 0x7f6c8c15e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 -- 172.21.15.122:0/1595610777 >> v1:172.21.15.169:6789/0 conn(0x7f6c8c13dc10 legacy=0x7f6c8c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.462+0000 7f6c91beb640 1 -- 172.21.15.122:0/1595610777 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c15f580 con 0x7f6c8c15bf30 2024-03-06T07:32:56.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c91beb640 1 --2- 172.21.15.122:0/1595610777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c15ed90 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009e50 tx=0x7f6c8402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11a3e1dff6307b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c83fff640 1 -- 172.21.15.122:0/1595610777 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8402c9e0 con 0x7f6c8c15bf30 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c83fff640 1 -- 172.21.15.122:0/1595610777 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8402cb80 con 0x7f6c8c15bf30 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c83fff640 1 -- 172.21.15.122:0/1595610777 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c840343f0 con 0x7f6c8c15bf30 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c9364e640 1 -- 172.21.15.122:0/1595610777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c15bf30 msgr2=0x7f6c8c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c9364e640 1 --2- 172.21.15.122:0/1595610777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c15ed90 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009e50 tx=0x7f6c8402be40 comp rx=0 tx=0).stop 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c9364e640 1 -- 172.21.15.122:0/1595610777 shutdown_connections 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c9364e640 1 --2- 172.21.15.122:0/1595610777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c15ed90 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c9364e640 1 --2- 172.21.15.122:0/1595610777 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c0b3060 0x7f6c8c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.463+0000 7f6c9364e640 1 -- 172.21.15.122:0/1595610777 >> 172.21.15.122:0/1595610777 conn(0x7f6c8c0cf8a0 msgr2=0x7f6c8c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.464+0000 7f6c9364e640 1 -- 172.21.15.122:0/1595610777 shutdown_connections 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.464+0000 7f6c9364e640 1 -- 172.21.15.122:0/1595610777 wait complete. 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 Processor -- start 2024-03-06T07:32:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 -- start start 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 0x7f6c8c1f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c13dc10 0x7f6c8c13eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c13c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c13dff0 con 0x7f6c8c0b3060 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c15b730 con 0x7f6c8c15bf30 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c9364e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0708d0 con 0x7f6c8c13dc10 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c913ea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 0x7f6c8c1f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c913ea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 0x7f6c8c1f6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51036/0 (socket says 172.21.15.122:51036) 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.465+0000 7f6c913ea640 1 -- 172.21.15.122:0/2021412990 learned_addr learned my addr 172.21.15.122:0/2021412990 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c90be9640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c13dc10 0x7f6c8c13eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c91beb640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c13c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c913ea640 1 -- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c13dc10 msgr2=0x7f6c8c13eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c913ea640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c13dc10 0x7f6c8c13eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c913ea640 1 -- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c8c15bf30 msgr2=0x7f6c8c13c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c913ea640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c13c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c913ea640 1 -- 172.21.15.122:0/2021412990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c13af60 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c913ea640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 0x7f6c8c1f6e00 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c410 tx=0x7f6c7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c017070 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c0035f0 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c13b240 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c003a20 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.466+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c13b800 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.467+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c7c01a070 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.468+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c04f330 con 0x7f6c8c0b3060 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.468+0000 7f6c81ffb640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c7006b4d0 0x7f6c7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.468+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6c7c089670 con 0x7f6c8c0b3060 2024-03-06T07:32:56.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.468+0000 7f6c90be9640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c7006b4d0 0x7f6c7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.472 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.468+0000 7f6c90be9640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c7006b4d0 0x7f6c7006d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0ce720 tx=0x7f6c88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.473+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c7c058560 con 0x7f6c8c0b3060 2024-03-06T07:32:56.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.875+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8c06eb00 con 0x7f6c8c0b3060 2024-03-06T07:32:56.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.876+0000 7f6c81ffb640 1 -- 172.21.15.122:0/2021412990 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f6c7c0010c0 con 0x7f6c8c0b3060 2024-03-06T07:32:56.878 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:56.879 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:56.879 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:56.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c7006b4d0 msgr2=0x7f6c7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c7006b4d0 0x7f6c7006d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0ce720 tx=0x7f6c88008040 comp rx=0 tx=0).stop 2024-03-06T07:32:56.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 msgr2=0x7f6c8c1f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 0x7f6c8c1f6e00 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c410 tx=0x7f6c7c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 shutdown_connections 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f6c7006b4d0 0x7f6c7006d990 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c13c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f6c8c13dc10 0x7f6c8c13eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 --2- 172.21.15.122:0/2021412990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c8c0b3060 0x7f6c8c1f6e00 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 >> 172.21.15.122:0/2021412990 conn(0x7f6c8c0cf8a0 msgr2=0x7f6c8c0b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 shutdown_connections 2024-03-06T07:32:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:56.879+0000 7f6c9364e640 1 -- 172.21.15.122:0/2021412990 wait complete. 2024-03-06T07:32:57.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:57.093 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:58.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.193+0000 7f880e53e640 1 Processor -- start 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.193+0000 7f880e53e640 1 -- start start 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.194+0000 7f880e53e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 0x7f88080d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.194+0000 7f880e53e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f88080cc8b0 0x7f88080ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.194+0000 7f880e53e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8808161980 con 0x7f880815bb20 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.194+0000 7f880e53e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88080d2500 con 0x7f88080cc8b0 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.194+0000 7f880e53e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8808161c60 con 0x7f88080cfff0 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 0x7f88080d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f880cadb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f88080cc8b0 0x7f88080ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 0x7f88080d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:58156/0 (socket says 172.21.15.122:58156) 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 -- 172.21.15.122:0/1913619113 learned_addr learned my addr 172.21.15.122:0/1913619113 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 -- 172.21.15.122:0/1913619113 >> v2:172.21.15.169:3300/0 conn(0x7f88080cc8b0 msgr2=0x7f88080ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 --2- 172.21.15.122:0/1913619113 >> v2:172.21.15.169:3300/0 conn(0x7f88080cc8b0 0x7f88080ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 -- 172.21.15.122:0/1913619113 >> v1:172.21.15.169:6789/0 conn(0x7f880815bb20 legacy=0x7f88080d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f88077fe640 1 -- 172.21.15.122:0/1913619113 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8808161f40 con 0x7f88080cfff0 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.195+0000 7f880cadb640 1 --2- 172.21.15.122:0/1913619113 >> v2:172.21.15.169:3300/0 conn(0x7f88080cc8b0 0x7f88080ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.196+0000 7f88077fe640 1 --2- 172.21.15.122:0/1913619113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 0x7f88080d0470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009870 tx=0x7f87fc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf59f777d9a6c228 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.196+0000 7f8806ffd640 1 -- 172.21.15.122:0/1913619113 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc034070 con 0x7f88080cfff0 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.196+0000 7f8806ffd640 1 -- 172.21.15.122:0/1913619113 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87fc030840 con 0x7f88080cfff0 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.196+0000 7f8806ffd640 1 -- 172.21.15.122:0/1913619113 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc030c10 con 0x7f88080cfff0 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 -- 172.21.15.122:0/1913619113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 msgr2=0x7f88080d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 --2- 172.21.15.122:0/1913619113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 0x7f88080d0470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009870 tx=0x7f87fc02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 -- 172.21.15.122:0/1913619113 shutdown_connections 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 --2- 172.21.15.122:0/1913619113 >> v2:172.21.15.169:3300/0 conn(0x7f88080cc8b0 0x7f88080ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 --2- 172.21.15.122:0/1913619113 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cfff0 0x7f88080d0470 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 -- 172.21.15.122:0/1913619113 >> 172.21.15.122:0/1913619113 conn(0x7f880809a9f0 msgr2=0x7f880809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 -- 172.21.15.122:0/1913619113 shutdown_connections 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.197+0000 7f880e53e640 1 -- 172.21.15.122:0/1913619113 wait complete. 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.198+0000 7f880e53e640 1 Processor -- start 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.198+0000 7f880e53e640 1 -- start start 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880e53e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cc8b0 0x7f88081f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880e53e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080cfff0 0x7f8808145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880e53e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 0x7f88081461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880e53e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88080d1980 con 0x7f880815bb20 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880e53e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8808162500 con 0x7f88080cfff0 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880e53e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88080d1b00 con 0x7f88080cc8b0 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 0x7f88081461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 0x7f88081461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51038/0 (socket says 172.21.15.122:51038) 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 -- 172.21.15.122:0/317186433 learned_addr learned my addr 172.21.15.122:0/317186433 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f88077fe640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080cfff0 0x7f8808145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f8807fff640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cc8b0 0x7f88081f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 -- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cc8b0 msgr2=0x7f88081f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cc8b0 0x7f88081f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 -- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080cfff0 msgr2=0x7f8808145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080cfff0 0x7f8808145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.199+0000 7f880cadb640 1 -- 172.21.15.122:0/317186433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080c5920 con 0x7f880815bb20 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.200+0000 7f880cadb640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 0x7f88081461e0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f880000efc0 tx=0x7f880000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.200+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800018030 con 0x7f880815bb20 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.200+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880000cdf0 con 0x7f880815bb20 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.200+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880806c9f0 con 0x7f880815bb20 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.200+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800003bb0 con 0x7f880815bb20 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.200+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880806cfb0 con 0x7f880815bb20 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.201+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8808069e80 con 0x7f880815bb20 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.201+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8800005d10 con 0x7f880815bb20 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.202+0000 7f8804ff9640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87f406b500 0x7f87f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.202+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f880008aa40 con 0x7f880815bb20 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.202+0000 7f8807fff640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87f406b500 0x7f87f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.202+0000 7f8807fff640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87f406b500 0x7f87f406d9c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f87f80016e0 tx=0x7f87f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.206+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8808069e80 con 0x7f880815bb20 2024-03-06T07:32:58.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.414+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f880804f330 con 0x7f880815bb20 2024-03-06T07:32:58.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.414+0000 7f8804ff9640 1 -- 172.21.15.122:0/317186433 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f880008ae20 con 0x7f880815bb20 2024-03-06T07:32:58.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:58.417 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:58.417 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:58.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87f406b500 msgr2=0x7f87f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87f406b500 0x7f87f406d9c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f87f80016e0 tx=0x7f87f8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 msgr2=0x7f88081461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 0x7f88081461e0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f880000efc0 tx=0x7f880000c490 comp rx=0 tx=0).stop 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 shutdown_connections 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87f406b500 0x7f87f406d9c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f880815bb20 0x7f88081461e0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88080cfff0 0x7f8808145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 --2- 172.21.15.122:0/317186433 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f88080cc8b0 0x7f88081f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 >> 172.21.15.122:0/317186433 conn(0x7f880809a9f0 msgr2=0x7f880806f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 shutdown_connections 2024-03-06T07:32:58.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:58.417+0000 7f880e53e640 1 -- 172.21.15.122:0/317186433 wait complete. 2024-03-06T07:32:58.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:32:58.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:32:59.633 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:59.735 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.733+0000 7f2081f93640 1 Processor -- start 2024-03-06T07:32:59.736 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.733+0000 7f2081f93640 1 -- start start 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f2081f93640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f207c0cfff0 0x7f207c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f2081f93640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f207c0cc8b0 0x7f207c0ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f2081f93640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c161980 con 0x7f207c15bb20 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f2081f93640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c0d2500 con 0x7f207c0cfff0 2024-03-06T07:32:59.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f2081f93640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c161c60 con 0x7f207c0cc8b0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f207affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f207c0cfff0 0x7f207c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f207affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f207c0cfff0 0x7f207c0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51054/0 (socket says 172.21.15.122:51054) 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.734+0000 7f207affd640 1 -- 172.21.15.122:0/2805507033 learned_addr learned my addr 172.21.15.122:0/2805507033 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207bfff640 1 --2- 172.21.15.122:0/2805507033 >> v2:172.21.15.169:3300/0 conn(0x7f207c0cc8b0 0x7f207c0ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207affd640 1 -- 172.21.15.122:0/2805507033 >> v2:172.21.15.169:3300/0 conn(0x7f207c0cc8b0 msgr2=0x7f207c0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207affd640 1 --2- 172.21.15.122:0/2805507033 >> v2:172.21.15.169:3300/0 conn(0x7f207c0cc8b0 0x7f207c0ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207affd640 1 -- 172.21.15.122:0/2805507033 >> v1:172.21.15.169:6789/0 conn(0x7f207c15bb20 legacy=0x7f207c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207affd640 1 -- 172.21.15.122:0/2805507033 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c161f40 con 0x7f207c0cfff0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207affd640 1 --2- 172.21.15.122:0/2805507033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cfff0 0x7f207c0d04a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f207c1618c0 tx=0x7f207002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7cdb77f66a616 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.735+0000 7f207a7fc640 1 -- 172.21.15.122:0/2805507033 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070035020 con 0x7f207c0cfff0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f207a7fc640 1 -- 172.21.15.122:0/2805507033 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2070031870 con 0x7f207c0cfff0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f207a7fc640 1 -- 172.21.15.122:0/2805507033 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070031c40 con 0x7f207c0cfff0 2024-03-06T07:32:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 -- 172.21.15.122:0/2805507033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cfff0 msgr2=0x7f207c0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 --2- 172.21.15.122:0/2805507033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cfff0 0x7f207c0d04a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f207c1618c0 tx=0x7f207002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 -- 172.21.15.122:0/2805507033 shutdown_connections 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 --2- 172.21.15.122:0/2805507033 >> v2:172.21.15.169:3300/0 conn(0x7f207c0cc8b0 0x7f207c0ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 --2- 172.21.15.122:0/2805507033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cfff0 0x7f207c0d04a0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 -- 172.21.15.122:0/2805507033 >> 172.21.15.122:0/2805507033 conn(0x7f207c09a9f0 msgr2=0x7f207c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 -- 172.21.15.122:0/2805507033 shutdown_connections 2024-03-06T07:32:59.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.736+0000 7f2081f93640 1 -- 172.21.15.122:0/2805507033 wait complete. 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.737+0000 7f2081f93640 1 Processor -- start 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.737+0000 7f2081f93640 1 -- start start 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f2081f93640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 0x7f207c155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f2081f93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f207c0cfff0 0x7f207c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f2081f93640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f207c15bb20 0x7f207c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f2081f93640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0d1980 con 0x7f207c0cc8b0 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f2081f93640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c162500 con 0x7f207c0cfff0 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f2081f93640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0d1b00 con 0x7f207c15bb20 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f207b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 0x7f207c155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f207b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 0x7f207c155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51058/0 (socket says 172.21.15.122:51058) 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f207b7fe640 1 -- 172.21.15.122:0/73752106 learned_addr learned my addr 172.21.15.122:0/73752106 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f207affd640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f207c0cfff0 0x7f207c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.738+0000 7f207bfff640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f207c15bb20 0x7f207c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207b7fe640 1 -- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f207c15bb20 msgr2=0x7f207c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207b7fe640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f207c15bb20 0x7f207c06d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207b7fe640 1 -- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f207c0cfff0 msgr2=0x7f207c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207b7fe640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f207c0cfff0 0x7f207c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207b7fe640 1 -- 172.21.15.122:0/73752106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c0c5790 con 0x7f207c0cc8b0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207affd640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f207c0cfff0 0x7f207c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f207b7fe640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 0x7f207c155000 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f206c00b420 tx=0x7f206c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c011830 con 0x7f207c0cc8b0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c016070 con 0x7f207c0cc8b0 2024-03-06T07:32:59.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207c0c5a70 con 0x7f207c0cc8b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c011c10 con 0x7f207c0cc8b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.739+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207c155540 con 0x7f207c0cc8b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.740+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207c04f330 con 0x7f207c0cc8b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.741+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f206c01b460 con 0x7f207c0cc8b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.741+0000 7f2063fff640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f206806b500 0x7f206806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.741+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f206c089c90 con 0x7f207c0cc8b0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.741+0000 7f207affd640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f206806b500 0x7f206806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.741+0000 7f207affd640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f206806b500 0x7f206806d9c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2070036000 tx=0x7f207003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.750+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f206c020020 con 0x7f207c0cc8b0 2024-03-06T07:32:59.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.961+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f207c0b1fe0 con 0x7f207c0cc8b0 2024-03-06T07:32:59.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.962+0000 7f2063fff640 1 -- 172.21.15.122:0/73752106 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f206c058b80 con 0x7f207c0cc8b0 2024-03-06T07:32:59.964 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:32:59.964 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:32:59.965 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f206806b500 msgr2=0x7f206806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f206806b500 0x7f206806d9c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2070036000 tx=0x7f207003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 msgr2=0x7f207c155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 0x7f207c155000 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f206c00b420 tx=0x7f206c00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 shutdown_connections 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f206806b500 0x7f206806d9c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f207c15bb20 0x7f207c06d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f207c0cfff0 0x7f207c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 --2- 172.21.15.122:0/73752106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f207c0cc8b0 0x7f207c155000 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 >> 172.21.15.122:0/73752106 conn(0x7f207c09a9f0 msgr2=0x7f207c06f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.964+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 shutdown_connections 2024-03-06T07:32:59.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:32:59.965+0000 7f2081f93640 1 -- 172.21.15.122:0/73752106 wait complete. 2024-03-06T07:33:00.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:00.179 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:01.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 Processor -- start 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 -- start start 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fce0c0b5030 0x7fce0c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0d29e0 con 0x7fce0c0cf190 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c0b2fa0 con 0x7fce0c0b5030 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.279+0000 7fce11fa0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0b3260 con 0x7fce0c0d20c0 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fce0c0b5030 0x7fce0c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fce0c0b5030 0x7fce0c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51066/0 (socket says 172.21.15.122:51066) 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 -- 172.21.15.122:0/2467150617 learned_addr learned my addr 172.21.15.122:0/2467150617 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 -- 172.21.15.122:0/2467150617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 msgr2=0x7fce0c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 --2- 172.21.15.122:0/2467150617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 -- 172.21.15.122:0/2467150617 >> v1:172.21.15.169:6789/0 conn(0x7fce0c0cf190 legacy=0x7fce0c0d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.280+0000 7fce1179f640 1 -- 172.21.15.122:0/2467150617 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c0b3540 con 0x7fce0c0b5030 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce1179f640 1 --2- 172.21.15.122:0/2467150617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c0cf590 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fce04009e50 tx=0x7fce0402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3699cfcb7dcbeea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce0b7fe640 1 -- 172.21.15.122:0/2467150617 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0402cb00 con 0x7fce0c0b5030 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce0b7fe640 1 -- 172.21.15.122:0/2467150617 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0402cca0 con 0x7fce0c0b5030 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce0b7fe640 1 -- 172.21.15.122:0/2467150617 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04034610 con 0x7fce0c0b5030 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 -- 172.21.15.122:0/2467150617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 msgr2=0x7fce0c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2467150617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c0cf590 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fce04009e50 tx=0x7fce0402bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 -- 172.21.15.122:0/2467150617 shutdown_connections 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2467150617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c0cf590 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2467150617 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 -- 172.21.15.122:0/2467150617 >> 172.21.15.122:0/2467150617 conn(0x7fce0c09ce20 msgr2=0x7fce0c0d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 -- 172.21.15.122:0/2467150617 shutdown_connections 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.281+0000 7fce11fa0640 1 -- 172.21.15.122:0/2467150617 wait complete. 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.283+0000 7fce11fa0640 1 Processor -- start 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.283+0000 7fce11fa0640 1 -- start start 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.284+0000 7fce11fa0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c15dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.284+0000 7fce11fa0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce0c0cf190 0x7fce0c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.284+0000 7fce11fa0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.284+0000 7fce11fa0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0d1af0 con 0x7fce0c0b5030 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.284+0000 7fce11fa0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0b3b00 con 0x7fce0c0cf190 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.284+0000 7fce11fa0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c161630 con 0x7fce0c0d20c0 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c15dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c15dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51072/0 (socket says 172.21.15.122:51072) 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 -- 172.21.15.122:0/2515017753 learned_addr learned my addr 172.21.15.122:0/2515017753 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce1179f640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce0bfff640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce0c0cf190 0x7fce0c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 -- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 msgr2=0x7fce0c0cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 -- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce0c0cf190 msgr2=0x7fce0c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce0c0cf190 0x7fce0c0cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 -- 172.21.15.122:0/2515017753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c0ca0a0 con 0x7fce0c0b5030 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce10f9e640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c15dc40 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b4d0 tx=0x7fce0000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00014070 con 0x7fce0c0b5030 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00010870 con 0x7fce0c0b5030 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c0c9800 con 0x7fce0c0b5030 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00010c60 con 0x7fce0c0b5030 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.285+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c0c9dc0 con 0x7fce0c0b5030 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.286+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c068f80 con 0x7fce0c0b5030 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.287+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce00017070 con 0x7fce0c0b5030 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.287+0000 7fce097fa640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcdf006b4d0 0x7fcdf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.287+0000 7fce0bfff640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcdf006b4d0 0x7fcdf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.287+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce00089f50 con 0x7fce0c0b5030 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.288+0000 7fce0bfff640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcdf006b4d0 0x7fcdf006d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8006fd0 tx=0x7fcdf8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.292+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce00058e40 con 0x7fce0c0b5030 2024-03-06T07:33:01.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.574+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce0c0d2530 con 0x7fce0c0b5030 2024-03-06T07:33:01.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.574+0000 7fce097fa640 1 -- 172.21.15.122:0/2515017753 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fce00058740 con 0x7fce0c0b5030 2024-03-06T07:33:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:01.577 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:01.577 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.576+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcdf006b4d0 msgr2=0x7fcdf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.576+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcdf006b4d0 0x7fcdf006d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8006fd0 tx=0x7fcdf8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.576+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 msgr2=0x7fce0c15dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.576+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c15dc40 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b4d0 tx=0x7fce0000b990 comp rx=0 tx=0).stop 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 shutdown_connections 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcdf006b4d0 0x7fcdf006d990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fce0c0d20c0 0x7fce0c0cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fce0c0cf190 0x7fce0c0cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 --2- 172.21.15.122:0/2515017753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce0c0b5030 0x7fce0c15dc40 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 >> 172.21.15.122:0/2515017753 conn(0x7fce0c09ce20 msgr2=0x7fce0c0cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 shutdown_connections 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:01.577+0000 7fce11fa0640 1 -- 172.21.15.122:0/2515017753 wait complete. 2024-03-06T07:33:01.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:01.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:02.791 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:02.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.890+0000 7f2aa7f7a640 1 Processor -- start 2024-03-06T07:33:02.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa7f7a640 1 -- start start 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa7f7a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2aa015dfe0 0x7f2aa00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa7f7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 0x7f2aa00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa7f7a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00b3430 con 0x7f2aa00d27f0 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa7f7a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00d2c50 con 0x7f2aa015dfe0 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa7f7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b31e0 con 0x7f2aa00cf830 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa6517640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2aa00d27f0 0x7f2aa00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52474/0 (socket says 172.21.15.122:52474) 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.891+0000 7f2aa6517640 1 -- 172.21.15.122:0/2111844129 learned_addr learned my addr 172.21.15.122:0/2111844129 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 0x7f2aa00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa5d16640 1 --2- 172.21.15.122:0/2111844129 >> v2:172.21.15.169:3300/0 conn(0x7f2aa015dfe0 0x7f2aa00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661317360 0 0) 0x7f2aa00b3430 con 0x7f2aa00d27f0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c003520 con 0x7f2aa00d27f0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404290055 0 0) 0x7f2a8c003520 con 0x7f2aa00d27f0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 msgr2=0x7f2aa00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 0x7f2aa00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 >> v2:172.21.15.169:3300/0 conn(0x7f2aa015dfe0 msgr2=0x7f2aa00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 --2- 172.21.15.122:0/2111844129 >> v2:172.21.15.169:3300/0 conn(0x7f2aa015dfe0 0x7f2aa00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00b3710 con 0x7f2aa00d27f0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f2a980019f0 con 0x7f2aa00d27f0 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 >> v1:172.21.15.122:6789/0 conn(0x7f2aa00d27f0 legacy=0x7f2aa00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a8c0039b0 0x7f2a8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 0x7f2a8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b31e0 con 0x7f2a8c004800 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c00b9c0 con 0x7f2a8c0039b0 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c00bb40 con 0x7f2aa00cf830 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa5d16640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a8c0039b0 0x7f2a8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.892+0000 7f2aa6517640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 0x7f2a8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa5515640 1 -- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 msgr2=0x7f2a8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa00cf830 0x7f2a8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa5515640 1 -- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a8c0039b0 msgr2=0x7f2a8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a8c0039b0 0x7f2a8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa5515640 1 -- 172.21.15.122:0/2111844129 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c003520 con 0x7f2a8c004800 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009a90 tx=0x7f2a8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80d1016c304e392 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c039070 con 0x7f2a8c004800 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c02c680 con 0x7f2a8c004800 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.893+0000 7f2aa4d14640 1 -- 172.21.15.122:0/2111844129 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c009690 con 0x7f2a8c004800 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2111844129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 msgr2=0x7f2a8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2111844129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009a90 tx=0x7f2a8c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa6517640 1 -- 172.21.15.122:0/2111844129 reap_dead start 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2111844129 shutdown_connections 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2111844129 >> 172.21.15.122:0/2111844129 conn(0x7f2aa00d3cd0 msgr2=0x7f2aa00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2111844129 shutdown_connections 2024-03-06T07:33:02.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.894+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2111844129 wait complete. 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.895+0000 7f2aa7f7a640 1 Processor -- start 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.895+0000 7f2aa7f7a640 1 -- start start 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa7f7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2aa014c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa7f7a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa01f7e40 0x7f2aa00cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa7f7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2aa01f7210 0x7f2aa01f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa7f7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d1c70 con 0x7f2a8c004800 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa7f7a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b3d60 con 0x7f2aa01f7210 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa7f7a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d1df0 con 0x7f2aa01f7e40 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2aa014c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2aa014c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51080/0 (socket says 172.21.15.122:51080) 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 -- 172.21.15.122:0/2688474181 learned_addr learned my addr 172.21.15.122:0/2688474181 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa6517640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2aa01f7210 0x7f2aa01f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa01f7e40 0x7f2aa00cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 -- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa01f7e40 msgr2=0x7f2aa00cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa01f7e40 0x7f2aa00cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 -- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2aa01f7210 msgr2=0x7f2aa01f7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2aa01f7210 0x7f2aa01f7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 -- 172.21.15.122:0/2688474181 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa014cd50 con 0x7f2a8c004800 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.896+0000 7f2aa5d16640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2aa014c810 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f2a90011960 tx=0x7f2a90011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.897+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000faf0 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.897+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9000fc90 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.897+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90014440 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.897+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa00cc620 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.897+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa00ccbe0 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.898+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa004f330 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.898+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a90012040 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.898+0000 7f2a967fc640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2a8406b500 0x7f2a8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.899+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2a9008a7f0 con 0x7f2a8c004800 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.899+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2a8406b500 0x7f2a8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.899+0000 7f2aa5515640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2a8406b500 0x7f2a8406d9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c036000 tx=0x7f2a9c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.905 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:02.903+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a90019030 con 0x7f2a8c004800 2024-03-06T07:33:03.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.118+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa006a190 con 0x7f2a8c004800 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.119+0000 7f2a967fc640 1 -- 172.21.15.122:0/2688474181 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2a90059090 con 0x7f2a8c004800 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2a8406b500 msgr2=0x7f2a8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2a8406b500 0x7f2a8406d9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c036000 tx=0x7f2a9c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 msgr2=0x7f2aa014c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2aa014c810 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f2a90011960 tx=0x7f2a90011e20 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 shutdown_connections 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2a8406b500 0x7f2a8406d9c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2aa01f7210 0x7f2aa01f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2aa01f7e40 0x7f2aa00cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 --2- 172.21.15.122:0/2688474181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a8c004800 0x7f2aa014c810 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 >> 172.21.15.122:0/2688474181 conn(0x7f2aa00d3cd0 msgr2=0x7f2aa006ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 shutdown_connections 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:03.121+0000 7f2aa7f7a640 1 -- 172.21.15.122:0/2688474181 wait complete. 2024-03-06T07:33:03.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:03.335 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:04.336 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:04.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.435+0000 7f569ac39640 1 Processor -- start 2024-03-06T07:33:04.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.435+0000 7f569ac39640 1 -- start start 2024-03-06T07:33:04.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f569ac39640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f569415dfe0 0x7f56940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f569ac39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56940d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f569ac39640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f56940d27f0 0x7f56940d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f569ac39640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56940d2c50 con 0x7f569415dfe0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f569ac39640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56940b31e0 con 0x7f56940d27f0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f569ac39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56940b3d60 con 0x7f56940cf830 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f569415dfe0 0x7f56940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f569415dfe0 0x7f56940cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51086/0 (socket says 172.21.15.122:51086) 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 -- 172.21.15.122:0/1221422576 learned_addr learned my addr 172.21.15.122:0/1221422576 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f5693fff640 1 --2- 172.21.15.122:0/1221422576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56940d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56991d6640 1 --2- 172.21.15.122:0/1221422576 >> v2:172.21.15.169:3300/0 conn(0x7f56940d27f0 0x7f56940d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 -- 172.21.15.122:0/1221422576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 msgr2=0x7f56940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 --2- 172.21.15.122:0/1221422576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56940d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 -- 172.21.15.122:0/1221422576 >> v2:172.21.15.169:3300/0 conn(0x7f56940d27f0 msgr2=0x7f56940d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 --2- 172.21.15.122:0/1221422576 >> v2:172.21.15.169:3300/0 conn(0x7f56940d27f0 0x7f56940d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.436+0000 7f56989d5640 1 -- 172.21.15.122:0/1221422576 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940b3430 con 0x7f569415dfe0 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.437+0000 7f56989d5640 1 --2- 172.21.15.122:0/1221422576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f56940cf2f0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f5694067ad0 tx=0x7f568802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580f8006e2fff7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.437+0000 7f56937fe640 1 -- 172.21.15.122:0/1221422576 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688035020 con 0x7f569415dfe0 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.437+0000 7f56937fe640 1 -- 172.21.15.122:0/1221422576 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688031870 con 0x7f569415dfe0 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.437+0000 7f56937fe640 1 -- 172.21.15.122:0/1221422576 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688031c40 con 0x7f569415dfe0 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 -- 172.21.15.122:0/1221422576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 msgr2=0x7f56940cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 --2- 172.21.15.122:0/1221422576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f56940cf2f0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f5694067ad0 tx=0x7f568802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 -- 172.21.15.122:0/1221422576 shutdown_connections 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 --2- 172.21.15.122:0/1221422576 >> v2:172.21.15.169:3300/0 conn(0x7f56940d27f0 0x7f56940d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 --2- 172.21.15.122:0/1221422576 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56940d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 --2- 172.21.15.122:0/1221422576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f56940cf2f0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 -- 172.21.15.122:0/1221422576 >> 172.21.15.122:0/1221422576 conn(0x7f56940d3cd0 msgr2=0x7f56940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 -- 172.21.15.122:0/1221422576 shutdown_connections 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.438+0000 7f569ac39640 1 -- 172.21.15.122:0/1221422576 wait complete. 2024-03-06T07:33:04.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.439+0000 7f569ac39640 1 Processor -- start 2024-03-06T07:33:04.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.439+0000 7f569ac39640 1 -- start start 2024-03-06T07:33:04.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f569ac39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56941f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f569ac39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56940d27f0 0x7f5694160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f569ac39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f569415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f569ac39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56940d1c70 con 0x7f569415dfe0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f569ac39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56940b39f0 con 0x7f56940d27f0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f569ac39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56940d1df0 con 0x7f56940cf830 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f569415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f569415fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51102/0 (socket says 172.21.15.122:51102) 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 -- 172.21.15.122:0/2688431066 learned_addr learned my addr 172.21.15.122:0/2688431066 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 -- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 msgr2=0x7f56941f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56989d5640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56941f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f5693fff640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56940d27f0 0x7f5694160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56941f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 -- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56940d27f0 msgr2=0x7f5694160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56940d27f0 0x7f5694160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.440+0000 7f56991d6640 1 -- 172.21.15.122:0/2688431066 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940b9740 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.441+0000 7f56991d6640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f569415fcb0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f568c007e40 tx=0x7f568c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.441+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c014620 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.441+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c019070 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.441+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c014a20 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.441+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56940b9a20 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.441+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56940cceb0 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.442+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56940b5200 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.442+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f568c014bc0 con 0x7f569415dfe0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.443+0000 7f56917fa640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f567806b4d0 0x7f567806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.443+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f568c08aaf0 con 0x7f569415dfe0 2024-03-06T07:33:04.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.443+0000 7f56989d5640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f567806b4d0 0x7f567806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.443+0000 7f56989d5640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f567806b4d0 0x7f567806d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5688036000 tx=0x7f5688008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.452+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56940b5200 con 0x7f569415dfe0 2024-03-06T07:33:04.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.661+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f569404f330 con 0x7f569415dfe0 2024-03-06T07:33:04.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.661+0000 7f56917fa640 1 -- 172.21.15.122:0/2688431066 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f568c059950 con 0x7f569415dfe0 2024-03-06T07:33:04.663 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:04.664 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:04.664 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:04.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f567806b4d0 msgr2=0x7f567806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f567806b4d0 0x7f567806d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5688036000 tx=0x7f5688008030 comp rx=0 tx=0).stop 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 msgr2=0x7f569415fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f569415fcb0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f568c007e40 tx=0x7f568c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 shutdown_connections 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f567806b4d0 0x7f567806d990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f569415dfe0 0x7f569415fcb0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56940d27f0 0x7f5694160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 --2- 172.21.15.122:0/2688431066 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56940cf830 0x7f56941f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 >> 172.21.15.122:0/2688431066 conn(0x7f56940d3cd0 msgr2=0x7f569406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 shutdown_connections 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:04.663+0000 7f569ac39640 1 -- 172.21.15.122:0/2688431066 wait complete. 2024-03-06T07:33:04.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:04.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:05.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:05.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.977+0000 7efef5177640 1 Processor -- start 2024-03-06T07:33:05.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efef5177640 1 -- start start 2024-03-06T07:33:05.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efef5177640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efef00d22b0 0x7efef00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efef5177640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00b41f0 0x7efef00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efef5177640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef00b3430 con 0x7efef015dfe0 2024-03-06T07:33:05.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efef5177640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef00d2c50 con 0x7efef00d22b0 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efef5177640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef00b31e0 con 0x7efef00b41f0 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efeee59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efef00d22b0 0x7efef00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efeee59c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efef00d22b0 0x7efef00d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43234/0 (socket says 172.21.15.122:43234) 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.978+0000 7efeee59c640 1 -- 172.21.15.122:0/1830955306 learned_addr learned my addr 172.21.15.122:0/1830955306 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.979+0000 7efeef59e640 1 --2- 172.21.15.122:0/1830955306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00b41f0 0x7efef00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.979+0000 7efeee59c640 1 -- 172.21.15.122:0/1830955306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00b41f0 msgr2=0x7efef00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.979+0000 7efeee59c640 1 --2- 172.21.15.122:0/1830955306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00b41f0 0x7efef00b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.979+0000 7efeee59c640 1 -- 172.21.15.122:0/1830955306 >> v1:172.21.15.122:6789/0 conn(0x7efef015dfe0 legacy=0x7efef00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.979+0000 7efeedd9b640 1 -- 172.21.15.122:0/1830955306 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595015602 0 0) 0x7efef00b3430 con 0x7efef015dfe0 2024-03-06T07:33:05.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.979+0000 7efeee59c640 1 -- 172.21.15.122:0/1830955306 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef00b3710 con 0x7efef00d22b0 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.980+0000 7efeee59c640 1 --2- 172.21.15.122:0/1830955306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00d22b0 0x7efef00d2710 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7efee0009870 tx=0x7efee002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f512959f5a7c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.980+0000 7efeedd9b640 1 -- 172.21.15.122:0/1830955306 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0034070 con 0x7efef00d22b0 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.980+0000 7efeedd9b640 1 -- 172.21.15.122:0/1830955306 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee0030840 con 0x7efef00d22b0 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.980+0000 7efeedd9b640 1 -- 172.21.15.122:0/1830955306 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0030c10 con 0x7efef00d22b0 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.980+0000 7efef5177640 1 -- 172.21.15.122:0/1830955306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00d22b0 msgr2=0x7efef00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.980+0000 7efef5177640 1 --2- 172.21.15.122:0/1830955306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00d22b0 0x7efef00d2710 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7efee0009870 tx=0x7efee002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.981+0000 7efef5177640 1 -- 172.21.15.122:0/1830955306 shutdown_connections 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.981+0000 7efef5177640 1 --2- 172.21.15.122:0/1830955306 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00b41f0 0x7efef00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.981+0000 7efef5177640 1 --2- 172.21.15.122:0/1830955306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00d22b0 0x7efef00d2710 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.981+0000 7efef5177640 1 -- 172.21.15.122:0/1830955306 >> 172.21.15.122:0/1830955306 conn(0x7efef00d3cd0 msgr2=0x7efef00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.981+0000 7efef5177640 1 -- 172.21.15.122:0/1830955306 shutdown_connections 2024-03-06T07:33:05.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.981+0000 7efef5177640 1 -- 172.21.15.122:0/1830955306 wait complete. 2024-03-06T07:33:05.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.982+0000 7efef5177640 1 Processor -- start 2024-03-06T07:33:05.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.982+0000 7efef5177640 1 -- start start 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efef5177640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 0x7efef00cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efef5177640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00d22b0 0x7efef00b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efef5177640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efef015dfe0 0x7efef00b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efef5177640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cfa00 con 0x7efef00b41f0 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efef5177640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00b3d60 con 0x7efef015dfe0 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efef5177640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef00d1c70 con 0x7efef00d22b0 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 0x7efef00cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 0x7efef00cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43246/0 (socket says 172.21.15.122:43246) 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 -- 172.21.15.122:0/770479412 learned_addr learned my addr 172.21.15.122:0/770479412 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeee59c640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00d22b0 0x7efef00b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 -- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00d22b0 msgr2=0x7efef00b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00d22b0 0x7efef00b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 -- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efef015dfe0 msgr2=0x7efef00b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efef015dfe0 0x7efef00b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 -- 172.21.15.122:0/770479412 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef015cf10 con 0x7efef00b41f0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efeeed9d640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 0x7efef00cd870 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7efed800c440 tx=0x7efed800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.983+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8017070 con 0x7efef00b41f0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.984+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed8005480 con 0x7efef00b41f0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.984+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef009ddd0 con 0x7efef00b41f0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.984+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8003830 con 0x7efef00b41f0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.984+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef00cb8d0 con 0x7efef00b41f0 2024-03-06T07:33:05.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.985+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef00cbad0 con 0x7efef00b41f0 2024-03-06T07:33:05.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.985+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efed801a070 con 0x7efef00b41f0 2024-03-06T07:33:05.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.986+0000 7efee77fe640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efed006b4d0 0x7efed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.986+0000 7efeee59c640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efed006b4d0 0x7efed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.986+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efed8089470 con 0x7efef00b41f0 2024-03-06T07:33:05.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.986+0000 7efeee59c640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efed006b4d0 0x7efed006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efee0036000 tx=0x7efee003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:05.990+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efef00cbad0 con 0x7efef00b41f0 2024-03-06T07:33:06.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.204+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef00cb270 con 0x7efef00b41f0 2024-03-06T07:33:06.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.205+0000 7efee77fe640 1 -- 172.21.15.122:0/770479412 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7efef009ddd0 con 0x7efef00b41f0 2024-03-06T07:33:06.207 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:06.207 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:06.208 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efed006b4d0 msgr2=0x7efed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efed006b4d0 0x7efed006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efee0036000 tx=0x7efee003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 msgr2=0x7efef00cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 0x7efef00cd870 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7efed800c440 tx=0x7efed800c900 comp rx=0 tx=0).stop 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 shutdown_connections 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efed006b4d0 0x7efed006d990 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efef015dfe0 0x7efef00b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.206+0000 7efef5177640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efef00d22b0 0x7efef00b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.207+0000 7efef5177640 1 --2- 172.21.15.122:0/770479412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef00b41f0 0x7efef00cd870 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.207+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 >> 172.21.15.122:0/770479412 conn(0x7efef00d3cd0 msgr2=0x7efef00b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.207+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 shutdown_connections 2024-03-06T07:33:06.210 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:06.207+0000 7efef5177640 1 -- 172.21.15.122:0/770479412 wait complete. 2024-03-06T07:33:06.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:06.424 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:07.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:07.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.525+0000 7f64d1a84640 1 Processor -- start 2024-03-06T07:33:07.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.525+0000 7f64d1a84640 1 -- start start 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64d1a84640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 0x7f64cc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64d1a84640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f64cc0d29c0 0x7f64cc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64d1a84640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc0b3430 con 0x7f64cc0cf830 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64d1a84640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc0b31e0 con 0x7f64cc0d29c0 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64d1a84640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0b3d60 con 0x7f64cc15dfe0 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64caffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f64cc0cf830 0x7f64cc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53760/0 (socket says 172.21.15.122:53760) 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64caffd640 1 -- 172.21.15.122:0/9160930 learned_addr learned my addr 172.21.15.122:0/9160930 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64cb7fe640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 0x7f64cc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64cbfff640 1 --2- 172.21.15.122:0/9160930 >> v2:172.21.15.169:3300/0 conn(0x7f64cc0d29c0 0x7f64cc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861061389 0 0) 0x7f64cc0b3430 con 0x7f64cc0cf830 2024-03-06T07:33:07.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b8003520 con 0x7f64cc0cf830 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980190290 0 0) 0x7f64b8003520 con 0x7f64cc0cf830 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 msgr2=0x7f64cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 0x7f64cc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 >> v2:172.21.15.169:3300/0 conn(0x7f64cc0d29c0 msgr2=0x7f64cc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 --2- 172.21.15.122:0/9160930 >> v2:172.21.15.169:3300/0 conn(0x7f64cc0d29c0 0x7f64cc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc0b3710 con 0x7f64cc0cf830 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f64b4001670 con 0x7f64cc0cf830 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 >> v1:172.21.15.122:6789/0 conn(0x7f64cc0cf830 legacy=0x7f64cc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.526+0000 7f64ca7fc640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 0x7f64b80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64ca7fc640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b8003f30 0x7f64b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64ca7fc640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b8004d40 0x7f64b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0b31e0 con 0x7f64b8004d40 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0b3d60 con 0x7f64b8003f30 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b800b9c0 con 0x7f64cc15dfe0 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cbfff640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b8004d40 0x7f64b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cb7fe640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 0x7f64b80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64caffd640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b8003f30 0x7f64b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cbfff640 1 -- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 msgr2=0x7f64b80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cbfff640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc15dfe0 0x7f64b80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cbfff640 1 -- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b8003f30 msgr2=0x7f64b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cbfff640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b8003f30 0x7f64b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64cbfff640 1 -- 172.21.15.122:0/9160930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8003520 con 0x7f64b8004d40 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.527+0000 7f64caffd640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64b8003f30 0x7f64b8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64cbfff640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b8004d40 0x7f64b80051c0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009ac0 tx=0x7f64b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df454528fc6e750 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4039070 con 0x7f64b8004d40 2024-03-06T07:33:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c402c6d0 con 0x7f64b8004d40 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64ca7fc640 1 -- 172.21.15.122:0/9160930 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4009690 con 0x7f64b8004d40 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64d1a84640 1 -- 172.21.15.122:0/9160930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b8004d40 msgr2=0x7f64b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64d1a84640 1 --2- 172.21.15.122:0/9160930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64b8004d40 0x7f64b80051c0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009ac0 tx=0x7f64b8005830 comp rx=0 tx=0).stop 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64cbfff640 1 -- 172.21.15.122:0/9160930 reap_dead start 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64d1a84640 1 -- 172.21.15.122:0/9160930 shutdown_connections 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64d1a84640 1 -- 172.21.15.122:0/9160930 >> 172.21.15.122:0/9160930 conn(0x7f64cc0d3cd0 msgr2=0x7f64cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64d1a84640 1 -- 172.21.15.122:0/9160930 shutdown_connections 2024-03-06T07:33:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.528+0000 7f64d1a84640 1 -- 172.21.15.122:0/9160930 wait complete. 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.529+0000 7f64d1a84640 1 Processor -- start 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 -- start start 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc14f120 0x7f64cc161250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64cc14f500 0x7f64cc15fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 0x7f64cc0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0d1df0 con 0x7f64cc0b68d0 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0d1c70 con 0x7f64cc14f500 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.530+0000 7f64d1a84640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15e3c0 con 0x7f64cc14f120 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 0x7f64cc0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 0x7f64cc0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43270/0 (socket says 172.21.15.122:43270) 2024-03-06T07:33:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 -- 172.21.15.122:0/1521816058 learned_addr learned my addr 172.21.15.122:0/1521816058 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cb7fe640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc14f120 0x7f64cc161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64caffd640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64cc14f500 0x7f64cc15fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 -- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc14f120 msgr2=0x7f64cc161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc14f120 0x7f64cc161250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 -- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64cc14f500 msgr2=0x7f64cc15fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64cc14f500 0x7f64cc15fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 -- 172.21.15.122:0/1521816058 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc1601b0 con 0x7f64cc0b68d0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64cbfff640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 0x7f64cc0b1fd0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009be0 tx=0x7f64c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4039070 con 0x7f64cc0b68d0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4043070 con 0x7f64cc0b68d0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc160490 con 0x7f64cc0b68d0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4009690 con 0x7f64cc0b68d0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.531+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc160a50 con 0x7f64cc0b68d0 2024-03-06T07:33:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.532+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc04f330 con 0x7f64cc0b68d0 2024-03-06T07:33:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.533+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64c4003850 con 0x7f64cc0b68d0 2024-03-06T07:33:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.533+0000 7f64c3fff640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64ac06b4d0 0x7f64ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.533+0000 7f64cb7fe640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64ac06b4d0 0x7f64ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.533+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f64c40ab190 con 0x7f64cc0b68d0 2024-03-06T07:33:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.534+0000 7f64cb7fe640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64ac06b4d0 0x7f64ac06d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f64bc014510 tx=0x7f64bc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.538+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64c404c020 con 0x7f64cc0b68d0 2024-03-06T07:33:07.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.753+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64cc15c990 con 0x7f64cc0b68d0 2024-03-06T07:33:07.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.753+0000 7f64c3fff640 1 -- 172.21.15.122:0/1521816058 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f64c40799a0 con 0x7f64cc0b68d0 2024-03-06T07:33:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:07.756 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:07.756 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:07.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64ac06b4d0 msgr2=0x7f64ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64ac06b4d0 0x7f64ac06d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f64bc014510 tx=0x7f64bc00b040 comp rx=0 tx=0).stop 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 msgr2=0x7f64cc0b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 0x7f64cc0b1fd0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009be0 tx=0x7f64c4004730 comp rx=0 tx=0).stop 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 shutdown_connections 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f64ac06b4d0 0x7f64ac06d990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64cc0b68d0 0x7f64cc0b1fd0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64cc14f500 0x7f64cc15fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 --2- 172.21.15.122:0/1521816058 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f64cc14f120 0x7f64cc161250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 >> 172.21.15.122:0/1521816058 conn(0x7f64cc0d3cd0 msgr2=0x7f64cc06cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 shutdown_connections 2024-03-06T07:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:07.755+0000 7f64d1a84640 1 -- 172.21.15.122:0/1521816058 wait complete. 2024-03-06T07:33:07.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:07.969 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:08.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:09.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.069+0000 7f4bee901640 1 Processor -- start 2024-03-06T07:33:09.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.069+0000 7f4bee901640 1 -- start start 2024-03-06T07:33:09.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bee901640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4be806c680 0x7f4be806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bee901640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be815e180 0x7f4be80d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bee901640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4be80b0aa0 0x7f4be806ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bee901640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be80d06c0 con 0x7f4be806c680 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bee901640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be80d1090 con 0x7f4be80b0aa0 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bee901640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be806cfa0 con 0x7f4be815e180 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4be806c680 0x7f4be806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4be806c680 0x7f4be806c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43278/0 (socket says 172.21.15.122:43278) 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 -- 172.21.15.122:0/3877439497 learned_addr learned my addr 172.21.15.122:0/3877439497 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be77fe640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be815e180 0x7f4be80d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 -- 172.21.15.122:0/3877439497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be815e180 msgr2=0x7f4be80d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bece9e640 1 --2- 172.21.15.122:0/3877439497 >> v2:172.21.15.169:3300/0 conn(0x7f4be80b0aa0 0x7f4be806ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be815e180 0x7f4be80d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 -- 172.21.15.122:0/3877439497 >> v2:172.21.15.169:3300/0 conn(0x7f4be80b0aa0 msgr2=0x7f4be806ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 --2- 172.21.15.122:0/3877439497 >> v2:172.21.15.169:3300/0 conn(0x7f4be80b0aa0 0x7f4be806ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.074 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be7fff640 1 -- 172.21.15.122:0/3877439497 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be815f380 con 0x7f4be806c680 2024-03-06T07:33:09.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4be77fe640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be815e180 0x7f4be80d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:09.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.070+0000 7f4bece9e640 1 --2- 172.21.15.122:0/3877439497 >> v2:172.21.15.169:3300/0 conn(0x7f4be80b0aa0 0x7f4be806ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:09.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.071+0000 7f4be7fff640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be806c680 0x7f4be806c0c0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f4be806e820 tx=0x7f4bd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326e680d132a787 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.071+0000 7f4be6ffd640 1 -- 172.21.15.122:0/3877439497 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8035020 con 0x7f4be806c680 2024-03-06T07:33:09.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.071+0000 7f4be6ffd640 1 -- 172.21.15.122:0/3877439497 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8031870 con 0x7f4be806c680 2024-03-06T07:33:09.075 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.071+0000 7f4be6ffd640 1 -- 172.21.15.122:0/3877439497 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8031c40 con 0x7f4be806c680 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 -- 172.21.15.122:0/3877439497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be806c680 msgr2=0x7f4be806c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be806c680 0x7f4be806c0c0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f4be806e820 tx=0x7f4bd802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 -- 172.21.15.122:0/3877439497 shutdown_connections 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 --2- 172.21.15.122:0/3877439497 >> v2:172.21.15.169:3300/0 conn(0x7f4be80b0aa0 0x7f4be806ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be815e180 0x7f4be80d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 --2- 172.21.15.122:0/3877439497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be806c680 0x7f4be806c0c0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 -- 172.21.15.122:0/3877439497 >> 172.21.15.122:0/3877439497 conn(0x7f4be80d1b20 msgr2=0x7f4be80d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 -- 172.21.15.122:0/3877439497 shutdown_connections 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.072+0000 7f4bee901640 1 -- 172.21.15.122:0/3877439497 wait complete. 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.073+0000 7f4bee901640 1 Processor -- start 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.074+0000 7f4bee901640 1 -- start start 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.074+0000 7f4bee901640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be806c680 0x7f4be81f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.078 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.074+0000 7f4bee901640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4be80b0aa0 0x7f4be80b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.074+0000 7f4bee901640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 0x7f4be81f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.074+0000 7f4bee901640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80cfcb0 con 0x7f4be815e180 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bee901640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80d0350 con 0x7f4be80b0aa0 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bee901640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80cfe30 con 0x7f4be806c680 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 0x7f4be81f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 0x7f4be81f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43280/0 (socket says 172.21.15.122:43280) 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 -- 172.21.15.122:0/3979110210 learned_addr learned my addr 172.21.15.122:0/3979110210 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4be7fff640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be806c680 0x7f4be81f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4be77fe640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4be80b0aa0 0x7f4be80b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 -- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be806c680 msgr2=0x7f4be81f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be806c680 0x7f4be81f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.079 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 -- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4be80b0aa0 msgr2=0x7f4be80b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4be80b0aa0 0x7f4be80b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 -- 172.21.15.122:0/3979110210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be81f5340 con 0x7f4be815e180 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4be77fe640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4be80b0aa0 0x7f4be80b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bece9e640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 0x7f4be81f16f0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f4be000cf20 tx=0x7f4be000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be001a070 con 0x7f4be815e180 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be000d840 con 0x7f4be815e180 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be81f5620 con 0x7f4be815e180 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be000dc30 con 0x7f4be815e180 2024-03-06T07:33:09.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.075+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be809db10 con 0x7f4be815e180 2024-03-06T07:33:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.076+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be804f330 con 0x7f4be815e180 2024-03-06T07:33:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.077+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4be001d070 con 0x7f4be815e180 2024-03-06T07:33:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.077+0000 7f4be4ff9640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.077+0000 7f4be7fff640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.077+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4be008a9a0 con 0x7f4be815e180 2024-03-06T07:33:09.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.078+0000 7f4be7fff640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8036000 tx=0x7f4bd803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.082+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4be0059240 con 0x7f4be815e180 2024-03-06T07:33:09.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.297+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be80b2c80 con 0x7f4be815e180 2024-03-06T07:33:09.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.298+0000 7f4be4ff9640 1 -- 172.21.15.122:0/3979110210 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f4be005d2e0 con 0x7f4be815e180 2024-03-06T07:33:09.300 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:09.300 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:09.301 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:09.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 msgr2=0x7f4bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8036000 tx=0x7f4bd803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:09.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 msgr2=0x7f4be81f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 0x7f4be81f16f0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f4be000cf20 tx=0x7f4be000a7d0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 shutdown_connections 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be815e180 0x7f4be81f16f0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4be80b0aa0 0x7f4be80b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 --2- 172.21.15.122:0/3979110210 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4be806c680 0x7f4be81f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 >> 172.21.15.122:0/3979110210 conn(0x7f4be80d1b20 msgr2=0x7f4be806eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 shutdown_connections 2024-03-06T07:33:09.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:09.300+0000 7f4bee901640 1 -- 172.21.15.122:0/3979110210 wait complete. 2024-03-06T07:33:09.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:09.514 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:10.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:10.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.619+0000 7f9a7ec4e640 1 Processor -- start 2024-03-06T07:33:10.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.619+0000 7f9a7ec4e640 1 -- start start 2024-03-06T07:33:10.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.619+0000 7f9a7ec4e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9a7815dfe0 0x7f9a780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.619+0000 7f9a7ec4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780b3430 con 0x7f9a780cf830 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.619+0000 7f9a7ec4e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780b3710 con 0x7f9a780d29c0 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.619+0000 7f9a7ec4e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a780b31e0 con 0x7f9a7815dfe0 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a77fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9a780cf830 0x7f9a780d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53762/0 (socket says 172.21.15.122:53762) 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a77fff640 1 -- 172.21.15.122:0/661013867 learned_addr learned my addr 172.21.15.122:0/661013867 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a7c9ea640 1 --2- 172.21.15.122:0/661013867 >> v2:172.21.15.169:3300/0 conn(0x7f9a7815dfe0 0x7f9a780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a777fe640 1 -- 172.21.15.122:0/661013867 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501497943 0 0) 0x7f9a780b3430 con 0x7f9a780cf830 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a777fe640 1 -- 172.21.15.122:0/661013867 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a68003520 con 0x7f9a780cf830 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a777fe640 1 -- 172.21.15.122:0/661013867 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290992384 0 0) 0x7f9a68003520 con 0x7f9a780cf830 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/661013867 >> v1:172.21.15.169:6789/0 conn(0x7f9a780d29c0 legacy=0x7f9a780b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/661013867 >> v1:172.21.15.122:6789/0 conn(0x7f9a780cf830 legacy=0x7f9a780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.620+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/661013867 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780b39f0 con 0x7f9a7815dfe0 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.621+0000 7f9a7c9ea640 1 --2- 172.21.15.122:0/661013867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815dfe0 0x7f9a780cf2f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9a780b3360 tx=0x7f9a6c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=466d30b2bb7cb9ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.621+0000 7f9a777fe640 1 -- 172.21.15.122:0/661013867 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c039070 con 0x7f9a7815dfe0 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.621+0000 7f9a777fe640 1 -- 172.21.15.122:0/661013867 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c0314f0 con 0x7f9a7815dfe0 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.621+0000 7f9a777fe640 1 -- 172.21.15.122:0/661013867 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64001670 con 0x7f9a7815dfe0 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/661013867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815dfe0 msgr2=0x7f9a780cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/661013867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815dfe0 0x7f9a780cf2f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9a780b3360 tx=0x7f9a6c02b810 comp rx=0 tx=0).stop 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/661013867 shutdown_connections 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/661013867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815dfe0 0x7f9a780cf2f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/661013867 >> 172.21.15.122:0/661013867 conn(0x7f9a780d3cd0 msgr2=0x7f9a780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/661013867 shutdown_connections 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.622+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/661013867 wait complete. 2024-03-06T07:33:10.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.623+0000 7f9a7ec4e640 1 Processor -- start 2024-03-06T07:33:10.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.623+0000 7f9a7ec4e640 1 -- start start 2024-03-06T07:33:10.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7ec4e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 0x7f9a7814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7ec4e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a780d29c0 0x7f9a780b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7ec4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815b740 0x7f9a7814f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7ec4e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7815e3c0 con 0x7f9a780cf830 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7ec4e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a780b3d60 con 0x7f9a7815b740 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7ec4e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7815e540 con 0x7f9a780d29c0 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 0x7f9a7814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 0x7f9a7814fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43290/0 (socket says 172.21.15.122:43290) 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/3470594772 learned_addr learned my addr 172.21.15.122:0/3470594772 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a77fff640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a780d29c0 0x7f9a780b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7d1eb640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815b740 0x7f9a7814f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a780d29c0 msgr2=0x7f9a780b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a780d29c0 0x7f9a780b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815b740 msgr2=0x7f9a7814f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815b740 0x7f9a7814f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 -- 172.21.15.122:0/3470594772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780b6e10 con 0x7f9a780cf830 2024-03-06T07:33:10.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7d1eb640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815b740 0x7f9a7814f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.624+0000 7f9a7c9ea640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 0x7f9a7814fb70 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009890 tx=0x7f9a6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.625+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c044070 con 0x7f9a780cf830 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.625+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c004020 con 0x7f9a780cf830 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.625+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c039040 con 0x7f9a780cf830 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.625+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a780cc520 con 0x7f9a780cf830 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.625+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a780ccae0 con 0x7f9a780cf830 2024-03-06T07:33:10.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.626+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7804f330 con 0x7f9a780cf830 2024-03-06T07:33:10.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.626+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a6c03c040 con 0x7f9a780cf830 2024-03-06T07:33:10.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.627+0000 7f9a757fa640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.627+0000 7f9a77fff640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.627+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a6c0abba0 con 0x7f9a780cf830 2024-03-06T07:33:10.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.627+0000 7f9a77fff640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9a7815bc90 tx=0x7f9a64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.631+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a780b6e10 con 0x7f9a780cf830 2024-03-06T07:33:10.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.847+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a780ca160 con 0x7f9a780cf830 2024-03-06T07:33:10.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.847+0000 7f9a757fa640 1 -- 172.21.15.122:0/3470594772 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9a6c07ab20 con 0x7f9a780cf830 2024-03-06T07:33:10.849 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:10.849 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:10.850 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a5c06b4d0 msgr2=0x7f9a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9a7815bc90 tx=0x7f9a64009040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 msgr2=0x7f9a7814fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 0x7f9a7814fb70 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009890 tx=0x7f9a6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 shutdown_connections 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a7815b740 0x7f9a7814f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9a780d29c0 0x7f9a780b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 --2- 172.21.15.122:0/3470594772 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a780cf830 0x7f9a7814fb70 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 >> 172.21.15.122:0/3470594772 conn(0x7f9a780d3cd0 msgr2=0x7f9a780b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 shutdown_connections 2024-03-06T07:33:10.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:10.849+0000 7f9a7ec4e640 1 -- 172.21.15.122:0/3470594772 wait complete. 2024-03-06T07:33:11.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:11.064 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:12.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:12.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.164+0000 7f1c3944e640 1 Processor -- start 2024-03-06T07:33:12.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c3944e640 1 -- start start 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c3944e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1c340b4170 0x7f1c340b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c3944e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1c340d0a80 0x7f1c340cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c3944e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340d25d0 con 0x7f1c340d4760 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c3944e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c340d4bf0 con 0x7f1c340b4170 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c3944e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c340ce9e0 con 0x7f1c340d0a80 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c32ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1c340b4170 0x7f1c340b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c327fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1c340d4760 0x7f1c340d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53770/0 (socket says 172.21.15.122:53770) 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c327fc640 1 -- 172.21.15.122:0/3803037375 learned_addr learned my addr 172.21.15.122:0/3803037375 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.165+0000 7f1c32ffd640 1 --2- 172.21.15.122:0/3803037375 >> v2:172.21.15.122:3300/0 conn(0x7f1c340b4170 0x7f1c340b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43294/0 (socket says 172.21.15.122:43294) 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c337fe640 1 --2- 172.21.15.122:0/3803037375 >> v2:172.21.15.169:3300/0 conn(0x7f1c340d0a80 0x7f1c340cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c32ffd640 1 -- 172.21.15.122:0/3803037375 >> v2:172.21.15.169:3300/0 conn(0x7f1c340d0a80 msgr2=0x7f1c340cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c32ffd640 1 --2- 172.21.15.122:0/3803037375 >> v2:172.21.15.169:3300/0 conn(0x7f1c340d0a80 0x7f1c340cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c32ffd640 1 -- 172.21.15.122:0/3803037375 >> v1:172.21.15.122:6789/0 conn(0x7f1c340d4760 legacy=0x7f1c340d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c31ffb640 1 -- 172.21.15.122:0/3803037375 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261267453 0 0) 0x7f1c340d25d0 con 0x7f1c340d4760 2024-03-06T07:33:12.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c32ffd640 1 -- 172.21.15.122:0/3803037375 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340d2870 con 0x7f1c340b4170 2024-03-06T07:33:12.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.166+0000 7f1c32ffd640 1 --2- 172.21.15.122:0/3803037375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c340b4170 0x7f1c340b6a90 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f1c3408df00 tx=0x7f1c2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645195a2bf2aa19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c31ffb640 1 -- 172.21.15.122:0/3803037375 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c039070 con 0x7f1c340b4170 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c31ffb640 1 -- 172.21.15.122:0/3803037375 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c02cc80 con 0x7f1c340b4170 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c31ffb640 1 -- 172.21.15.122:0/3803037375 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c034660 con 0x7f1c340b4170 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 -- 172.21.15.122:0/3803037375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c340b4170 msgr2=0x7f1c340b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 --2- 172.21.15.122:0/3803037375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c340b4170 0x7f1c340b6a90 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f1c3408df00 tx=0x7f1c2c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 -- 172.21.15.122:0/3803037375 shutdown_connections 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 --2- 172.21.15.122:0/3803037375 >> v2:172.21.15.169:3300/0 conn(0x7f1c340d0a80 0x7f1c340cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 --2- 172.21.15.122:0/3803037375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c340b4170 0x7f1c340b6a90 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 -- 172.21.15.122:0/3803037375 >> 172.21.15.122:0/3803037375 conn(0x7f1c340d3c40 msgr2=0x7f1c340d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 -- 172.21.15.122:0/3803037375 shutdown_connections 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.167+0000 7f1c3944e640 1 -- 172.21.15.122:0/3803037375 wait complete. 2024-03-06T07:33:12.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.169+0000 7f1c3944e640 1 Processor -- start 2024-03-06T07:33:12.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.169+0000 7f1c3944e640 1 -- start start 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.169+0000 7f1c3944e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c340b4170 0x7f1c34156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c340d0a80 0x7f1c3415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 0x7f1c340b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c340b4dd0 con 0x7f1c3415d810 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c340b4550 con 0x7f1c340d0a80 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c340b4f50 con 0x7f1c340b4170 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 0x7f1c340b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 0x7f1c340b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43308/0 (socket says 172.21.15.122:43308) 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 -- 172.21.15.122:0/4272127824 learned_addr learned my addr 172.21.15.122:0/4272127824 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c32ffd640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c340b4170 0x7f1c34156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c327fc640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c340d0a80 0x7f1c3415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 -- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c340b4170 msgr2=0x7f1c34156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c340b4170 0x7f1c34156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 -- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c340d0a80 msgr2=0x7f1c3415d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c340d0a80 0x7f1c3415d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 -- 172.21.15.122:0/4272127824 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3415c230 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c337fe640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 0x7f1c340b32a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400d800 tx=0x7f1c2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400b6c0 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c24019070 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3415c510 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400bac0 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.170+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3415cad0 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.171+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3404f330 con 0x7f1c3415d810 2024-03-06T07:33:12.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.172+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c2400bc60 con 0x7f1c3415d810 2024-03-06T07:33:12.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.172+0000 7f1c2b7fe640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c1406b500 0x7f1c1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.173+0000 7f1c32ffd640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c1406b500 0x7f1c1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.173+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1c2408a470 con 0x7f1c3415d810 2024-03-06T07:33:12.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.173+0000 7f1c32ffd640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c1406b500 0x7f1c1406d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c036000 tx=0x7f1c2c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.177+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c24058d10 con 0x7f1c3415d810 2024-03-06T07:33:12.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.440+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c340733b0 con 0x7f1c3415d810 2024-03-06T07:33:12.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.441+0000 7f1c2b7fe640 1 -- 172.21.15.122:0/4272127824 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f1c2405cdb0 con 0x7f1c3415d810 2024-03-06T07:33:12.443 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:12.443 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:12.443 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:12.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.442+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c1406b500 msgr2=0x7f1c1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.442+0000 7f1c3944e640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c1406b500 0x7f1c1406d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c036000 tx=0x7f1c2c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:12.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.442+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 msgr2=0x7f1c340b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.442+0000 7f1c3944e640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 0x7f1c340b32a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400d800 tx=0x7f1c2400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 shutdown_connections 2024-03-06T07:33:12.445 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c1406b500 0x7f1c1406d9c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c3415d810 0x7f1c340b32a0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c340d0a80 0x7f1c3415d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 --2- 172.21.15.122:0/4272127824 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c340b4170 0x7f1c34156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 >> 172.21.15.122:0/4272127824 conn(0x7f1c340d3c40 msgr2=0x7f1c340b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 shutdown_connections 2024-03-06T07:33:12.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:12.443+0000 7f1c3944e640 1 -- 172.21.15.122:0/4272127824 wait complete. 2024-03-06T07:33:12.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:12.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:13.658 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:13.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.758+0000 7f0d0d9ac640 1 Processor -- start 2024-03-06T07:33:13.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.758+0000 7f0d0d9ac640 1 -- start start 2024-03-06T07:33:13.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.758+0000 7f0d0d9ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 0x7f0d080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d0d9ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080b3430 con 0x7f0d080d29c0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d0d9ac640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080b3710 con 0x7f0d080cf830 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d0d9ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b31e0 con 0x7f0d0815dfe0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d077fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0d080d29c0 0x7f0d080b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53778/0 (socket says 172.21.15.122:53778) 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d077fe640 1 -- 172.21.15.122:0/288274354 learned_addr learned my addr 172.21.15.122:0/288274354 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790403316 0 0) 0x7f0d080b3430 con 0x7f0d080d29c0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d06ffd640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 0x7f0d080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0003520 con 0x7f0d080d29c0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3232508646 0 0) 0x7f0cf0003520 con 0x7f0d080d29c0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 msgr2=0x7f0d080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 0x7f0d080cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 >> v1:172.21.15.169:6789/0 conn(0x7f0d080cf830 legacy=0x7f0d080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080b39f0 con 0x7f0d080d29c0 2024-03-06T07:33:13.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f0d000019f0 con 0x7f0d080d29c0 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 >> v1:172.21.15.122:6789/0 conn(0x7f0d080d29c0 legacy=0x7f0d080b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 0x7f0cf00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0cf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b31e0 con 0x7f0cf0003f60 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf000baa0 con 0x7f0cf0004d40 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf000bca0 con 0x7f0d0815dfe0 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.759+0000 7f0d067fc640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0cf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d06ffd640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 0x7f0cf00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d077fe640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d067fc640 1 -- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 msgr2=0x7f0cf00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d067fc640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d0815dfe0 0x7f0cf00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d067fc640 1 -- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf0004d40 msgr2=0x7f0cf00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d067fc640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d067fc640 1 -- 172.21.15.122:0/288274354 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0003520 con 0x7f0cf0003f60 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.760+0000 7f0d067fc640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0cf0004800 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc001720 tx=0x7f0cf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af3be3f64d1d6ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc001370 con 0x7f0cf0003f60 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc0052d0 con 0x7f0cf0003f60 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d05ffb640 1 -- 172.21.15.122:0/288274354 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc003830 con 0x7f0cf0003f60 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/288274354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 msgr2=0x7f0cf0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/288274354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0cf0004800 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc001720 tx=0x7f0cf0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d077fe640 1 -- 172.21.15.122:0/288274354 reap_dead start 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/288274354 shutdown_connections 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/288274354 >> 172.21.15.122:0/288274354 conn(0x7f0d080d3cd0 msgr2=0x7f0d080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/288274354 shutdown_connections 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.761+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/288274354 wait complete. 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.762+0000 7f0d0d9ac640 1 Processor -- start 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.762+0000 7f0d0d9ac640 1 -- start start 2024-03-06T07:33:13.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d0d9ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0d0806c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d0d9ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d080b68d0 0x7f0d080b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d0d9ac640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d081f6a60 0x7f0d081f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d0d9ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815e3c0 con 0x7f0cf0003f60 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d0d9ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b3d60 con 0x7f0d080b68d0 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d0d9ac640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815e540 con 0x7f0d081f6a60 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d06ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0d0806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d06ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0d0806c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43328/0 (socket says 172.21.15.122:43328) 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d06ffd640 1 -- 172.21.15.122:0/3342044102 learned_addr learned my addr 172.21.15.122:0/3342044102 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d067fc640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d080b68d0 0x7f0d080b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.763+0000 7f0d077fe640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d081f6a60 0x7f0d081f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d06ffd640 1 -- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d081f6a60 msgr2=0x7f0d081f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d06ffd640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d081f6a60 0x7f0d081f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d06ffd640 1 -- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d080b68d0 msgr2=0x7f0d080b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d06ffd640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d080b68d0 0x7f0d080b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d06ffd640 1 -- 172.21.15.122:0/3342044102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080b6dd0 con 0x7f0cf0003f60 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d06ffd640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0d0806c850 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80119a0 tx=0x7f0cf8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800fa30 con 0x7f0cf0003f60 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf800fbd0 con 0x7f0cf0003f60 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d080b70b0 con 0x7f0cf0003f60 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8014440 con 0x7f0cf0003f60 2024-03-06T07:33:13.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.764+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d080b7670 con 0x7f0cf0003f60 2024-03-06T07:33:13.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.765+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0804f330 con 0x7f0cf0003f60 2024-03-06T07:33:13.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.765+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cf8012040 con 0x7f0cf0003f60 2024-03-06T07:33:13.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.766+0000 7f0ceffff640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0cf406b4d0 0x7f0cf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.766+0000 7f0d067fc640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0cf406b4d0 0x7f0cf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.766+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0cf808a8d0 con 0x7f0cf0003f60 2024-03-06T07:33:13.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.766+0000 7f0d067fc640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0cf406b4d0 0x7f0cf406d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc033000 tx=0x7f0cfc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.770+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cf8019030 con 0x7f0cf0003f60 2024-03-06T07:33:13.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.978+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d080b5530 con 0x7f0cf0003f60 2024-03-06T07:33:13.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.978+0000 7f0ceffff640 1 -- 172.21.15.122:0/3342044102 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0cf8059730 con 0x7f0cf0003f60 2024-03-06T07:33:13.980 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:13.980 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:13.981 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:13.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0cf406b4d0 msgr2=0x7f0cf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0cf406b4d0 0x7f0cf406d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc033000 tx=0x7f0cfc009030 comp rx=0 tx=0).stop 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 msgr2=0x7f0d0806c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0d0806c850 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80119a0 tx=0x7f0cf8011da0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 shutdown_connections 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0cf406b4d0 0x7f0cf406d990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0d081f6a60 0x7f0d081f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d080b68d0 0x7f0d080b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 --2- 172.21.15.122:0/3342044102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cf0003f60 0x7f0d0806c850 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 >> 172.21.15.122:0/3342044102 conn(0x7f0d080d3cd0 msgr2=0x7f0d080b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 shutdown_connections 2024-03-06T07:33:13.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:13.980+0000 7f0d0d9ac640 1 -- 172.21.15.122:0/3342044102 wait complete. 2024-03-06T07:33:14.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:14.194 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:15.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:15.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.294+0000 7fd1570e0640 1 Processor -- start 2024-03-06T07:33:15.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.294+0000 7fd1570e0640 1 -- start start 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd1570e0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 0x7fd1500d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd1570e0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1500b3430 con 0x7fd15015dfe0 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd1570e0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1500b3710 con 0x7fd1500b41f0 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd1570e0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1500d2c50 con 0x7fd1500d22b0 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd154e7c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd15015dfe0 0x7fd1500cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44950/0 (socket says 172.21.15.122:44950) 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd154e7c640 1 -- 172.21.15.122:0/1741157132 learned_addr learned my addr 172.21.15.122:0/1741157132 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd14ffff640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 0x7fd1500d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668353181 0 0) 0x7fd1500b3430 con 0x7fd15015dfe0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.295+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd140003520 con 0x7fd15015dfe0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787011490 0 0) 0x7fd140003520 con 0x7fd15015dfe0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 msgr2=0x7fd1500d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 0x7fd1500d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 >> v1:172.21.15.169:6789/0 conn(0x7fd1500b41f0 legacy=0x7fd1500b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1500b39f0 con 0x7fd15015dfe0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fd144001670 con 0x7fd15015dfe0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 >> v1:172.21.15.122:6789/0 conn(0x7fd15015dfe0 legacy=0x7fd1500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1400039e0 0x7fd140003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd140004390 0x7fd140004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 0x7fd1400050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14000b9c0 con 0x7fd140004390 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14000bb40 con 0x7fd1400039e0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14000bd40 con 0x7fd1500d22b0 2024-03-06T07:33:15.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14ffff640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd140004390 0x7fd140004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd154e7c640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1400039e0 0x7fd140003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd15567d640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 0x7fd1400050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14ffff640 1 -- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 msgr2=0x7fd1400050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14ffff640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd1500d22b0 0x7fd1400050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14ffff640 1 -- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1400039e0 msgr2=0x7fd140003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14ffff640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1400039e0 0x7fd140003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.296+0000 7fd14ffff640 1 -- 172.21.15.122:0/1741157132 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd140003520 con 0x7fd140004390 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.297+0000 7fd14ffff640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd140004390 0x7fd140004860 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009ac0 tx=0x7fd140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0fe046dc5de735 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.297+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039070 con 0x7fd140004390 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.297+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c02c6b0 con 0x7fd140004390 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.297+0000 7fd14f7fe640 1 -- 172.21.15.122:0/1741157132 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c009690 con 0x7fd140004390 2024-03-06T07:33:15.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.297+0000 7fd1570e0640 1 -- 172.21.15.122:0/1741157132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd140004390 msgr2=0x7fd140004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.298+0000 7fd1570e0640 1 --2- 172.21.15.122:0/1741157132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd140004390 0x7fd140004860 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009ac0 tx=0x7fd140005830 comp rx=0 tx=0).stop 2024-03-06T07:33:15.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.298+0000 7fd15567d640 1 -- 172.21.15.122:0/1741157132 reap_dead start 2024-03-06T07:33:15.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.298+0000 7fd1570e0640 1 -- 172.21.15.122:0/1741157132 shutdown_connections 2024-03-06T07:33:15.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.298+0000 7fd1570e0640 1 -- 172.21.15.122:0/1741157132 >> 172.21.15.122:0/1741157132 conn(0x7fd1500d3cd0 msgr2=0x7fd1500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.298+0000 7fd1570e0640 1 -- 172.21.15.122:0/1741157132 shutdown_connections 2024-03-06T07:33:15.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.298+0000 7fd1570e0640 1 -- 172.21.15.122:0/1741157132 wait complete. 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.299+0000 7fd1570e0640 1 Processor -- start 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.299+0000 7fd1570e0640 1 -- start start 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140004390 0x7fd1501f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 0x7fd1500b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15015c0f0 0x7fd15015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1500b3d60 con 0x7fd1500b72e0 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1500b31e0 con 0x7fd140004390 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1500d1c70 con 0x7fd15015c0f0 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 0x7fd1500b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 0x7fd1500b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33088/0 (socket says 172.21.15.122:33088) 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 -- 172.21.15.122:0/3114332397 learned_addr learned my addr 172.21.15.122:0/3114332397 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd154e7c640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140004390 0x7fd1501f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd15567d640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15015c0f0 0x7fd15015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 -- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15015c0f0 msgr2=0x7fd15015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15015c0f0 0x7fd15015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 -- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140004390 msgr2=0x7fd1501f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140004390 0x7fd1501f6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 -- 172.21.15.122:0/3114332397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15015ca90 con 0x7fd1500b72e0 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14ffff640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 0x7fd1500b7760 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c043020 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c02cd90 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1500b7da0 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039040 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.300+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1500cafb0 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.301+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15004f330 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.302+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd13c046020 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.302+0000 7fd14d7fa640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13406b500 0x7fd13406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.303+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd13c0ab410 con 0x7fd1500b72e0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.303+0000 7fd154e7c640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13406b500 0x7fd13406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.303+0000 7fd154e7c640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13406b500 0x7fd13406d9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd1440011e0 tx=0x7fd144000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.307+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd13c079e90 con 0x7fd1500b72e0 2024-03-06T07:33:15.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.612+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd15006bb70 con 0x7fd1500b72e0 2024-03-06T07:33:15.615 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.613+0000 7fd14d7fa640 1 -- 172.21.15.122:0/3114332397 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd13c07df30 con 0x7fd1500b72e0 2024-03-06T07:33:15.615 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:15.615 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:15.615 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.614+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13406b500 msgr2=0x7fd13406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.614+0000 7fd1570e0640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13406b500 0x7fd13406d9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd1440011e0 tx=0x7fd144000ed0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.614+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 msgr2=0x7fd1500b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.614+0000 7fd1570e0640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 0x7fd1500b7760 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c004730 comp rx=0 tx=0).stop 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.614+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 shutdown_connections 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd13406b500 0x7fd13406d9c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd15015c0f0 0x7fd15015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1500b72e0 0x7fd1500b7760 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 --2- 172.21.15.122:0/3114332397 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140004390 0x7fd1501f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 >> 172.21.15.122:0/3114332397 conn(0x7fd1500d3cd0 msgr2=0x7fd1500cee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 shutdown_connections 2024-03-06T07:33:15.618 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:15.615+0000 7fd1570e0640 1 -- 172.21.15.122:0/3114332397 wait complete. 2024-03-06T07:33:15.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:15.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:16.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.968+0000 7f8765da2640 1 Processor -- start 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.969+0000 7f8765da2640 1 -- start start 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8765da2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f87600ae1d0 0x7f876006c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8765da2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87600ce2a0 0x7f87600ace70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8765da2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876015bff0 con 0x7f8760066d00 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8765da2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87600ae5b0 con 0x7f87600ce2a0 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8765da2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876009ddd0 con 0x7f87600ae1d0 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f87655a1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87600ce2a0 0x7f87600ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f875ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8760066d00 0x7f87600cdb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44966/0 (socket says 172.21.15.122:44966) 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f875ffff640 1 -- 172.21.15.122:0/2755896176 learned_addr learned my addr 172.21.15.122:0/2755896176 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8764da0640 1 --2- 172.21.15.122:0/2755896176 >> v2:172.21.15.169:3300/0 conn(0x7f87600ae1d0 0x7f876006c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f87655a1640 1 -- 172.21.15.122:0/2755896176 >> v2:172.21.15.169:3300/0 conn(0x7f87600ae1d0 msgr2=0x7f876006c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f87655a1640 1 --2- 172.21.15.122:0/2755896176 >> v2:172.21.15.169:3300/0 conn(0x7f87600ae1d0 0x7f876006c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f87655a1640 1 -- 172.21.15.122:0/2755896176 >> v1:172.21.15.122:6789/0 conn(0x7f8760066d00 legacy=0x7f87600cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f87655a1640 1 -- 172.21.15.122:0/2755896176 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876006b150 con 0x7f87600ce2a0 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f875f7fe640 1 -- 172.21.15.122:0/2755896176 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807953376 0 0) 0x7f876015bff0 con 0x7f8760066d00 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.970+0000 7f8764da0640 1 --2- 172.21.15.122:0/2755896176 >> v2:172.21.15.169:3300/0 conn(0x7f87600ae1d0 0x7f876006c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.971+0000 7f87655a1640 1 --2- 172.21.15.122:0/2755896176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87600ace70 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f87600cc890 tx=0x7f875802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2bfdd31b9dbbf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.971+0000 7f875f7fe640 1 -- 172.21.15.122:0/2755896176 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758039070 con 0x7f87600ce2a0 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.971+0000 7f875f7fe640 1 -- 172.21.15.122:0/2755896176 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875802cc80 con 0x7f87600ce2a0 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.971+0000 7f875f7fe640 1 -- 172.21.15.122:0/2755896176 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758034660 con 0x7f87600ce2a0 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 -- 172.21.15.122:0/2755896176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 msgr2=0x7f87600ace70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 --2- 172.21.15.122:0/2755896176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87600ace70 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f87600cc890 tx=0x7f875802bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 -- 172.21.15.122:0/2755896176 shutdown_connections 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 --2- 172.21.15.122:0/2755896176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87600ace70 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 --2- 172.21.15.122:0/2755896176 >> v2:172.21.15.169:3300/0 conn(0x7f87600ae1d0 0x7f876006c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 -- 172.21.15.122:0/2755896176 >> 172.21.15.122:0/2755896176 conn(0x7f87600cffc0 msgr2=0x7f87600d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 -- 172.21.15.122:0/2755896176 shutdown_connections 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.972+0000 7f8765da2640 1 -- 172.21.15.122:0/2755896176 wait complete. 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.973+0000 7f8765da2640 1 Processor -- start 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.973+0000 7f8765da2640 1 -- start start 2024-03-06T07:33:16.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8765da2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8760066d00 0x7f87600c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8765da2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87600ae1d0 0x7f876009b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8765da2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87601597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8765da2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600731d0 con 0x7f87600ce2a0 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8765da2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876006bc80 con 0x7f87600ae1d0 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8765da2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876006c6e0 con 0x7f8760066d00 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87601597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87601597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33092/0 (socket says 172.21.15.122:33092) 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 -- 172.21.15.122:0/3202605407 learned_addr learned my addr 172.21.15.122:0/3202605407 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8764da0640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8760066d00 0x7f87600c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f875ffff640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87600ae1d0 0x7f876009b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 -- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8760066d00 msgr2=0x7f87600c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8760066d00 0x7f87600c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 -- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87600ae1d0 msgr2=0x7f876009b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87600ae1d0 0x7f876009b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f87655a1640 1 -- 172.21.15.122:0/3202605407 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876009bac0 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f8764da0640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8760066d00 0x7f87600c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.974+0000 7f875ffff640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87600ae1d0 0x7f876009b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.975+0000 7f87655a1640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87601597a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f8758037040 tx=0x7f8758004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.975+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758044020 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.975+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875803a040 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.975+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87600c3b90 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.975+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758039050 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.975+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87600c4150 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.976+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8760068f80 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.976+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8758003b90 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.976+0000 7f875d7fa640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874406b4d0 0x7f874406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.977+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f87580aca00 con 0x7f87600ce2a0 2024-03-06T07:33:16.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.977+0000 7f8764da0640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874406b4d0 0x7f874406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.977+0000 7f8764da0640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874406b4d0 0x7f874406d990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8754006fd0 tx=0x7f8754008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:16.981+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f875804e020 con 0x7f87600ce2a0 2024-03-06T07:33:17.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.195+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8760070fe0 con 0x7f87600ce2a0 2024-03-06T07:33:17.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.196+0000 7f875d7fa640 1 -- 172.21.15.122:0/3202605407 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f875807b8f0 con 0x7f87600ce2a0 2024-03-06T07:33:17.198 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:17.198 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:17.198 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:17.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874406b4d0 msgr2=0x7f874406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874406b4d0 0x7f874406d990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8754006fd0 tx=0x7f8754008040 comp rx=0 tx=0).stop 2024-03-06T07:33:17.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 msgr2=0x7f87601597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87601597a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f8758037040 tx=0x7f8758004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 shutdown_connections 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f874406b4d0 0x7f874406d990 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87600ce2a0 0x7f87601597a0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87600ae1d0 0x7f876009b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 --2- 172.21.15.122:0/3202605407 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8760066d00 0x7f87600c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 >> 172.21.15.122:0/3202605407 conn(0x7f87600cffc0 msgr2=0x7f87600d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 shutdown_connections 2024-03-06T07:33:17.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:17.198+0000 7f8765da2640 1 -- 172.21.15.122:0/3202605407 wait complete. 2024-03-06T07:33:17.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:17.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:18.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:18.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.518+0000 7ff59ba15640 1 Processor -- start 2024-03-06T07:33:18.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.518+0000 7ff59ba15640 1 -- start start 2024-03-06T07:33:18.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff59ba15640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff5941515e0 0x7ff5940cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff59ba15640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff594071f40 con 0x7ff59415b430 2024-03-06T07:33:18.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff59ba15640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59415b8e0 con 0x7ff5940cd7a0 2024-03-06T07:33:18.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff59ba15640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff594070a70 con 0x7ff5941515e0 2024-03-06T07:33:18.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff598fb0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff59415b430 0x7ff594071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44974/0 (socket says 172.21.15.122:44974) 2024-03-06T07:33:18.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff598fb0640 1 -- 172.21.15.122:0/1037183073 learned_addr learned my addr 172.21.15.122:0/1037183073 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff5997b1640 1 --2- 172.21.15.122:0/1037183073 >> v2:172.21.15.169:3300/0 conn(0x7ff5941515e0 0x7ff5940cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295652803 0 0) 0x7ff594071f40 con 0x7ff59415b430 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff580003520 con 0x7ff59415b430 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2374877396 0 0) 0x7ff580003520 con 0x7ff59415b430 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 >> v2:172.21.15.169:3300/0 conn(0x7ff5941515e0 msgr2=0x7ff5940cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 --2- 172.21.15.122:0/1037183073 >> v2:172.21.15.169:3300/0 conn(0x7ff5941515e0 0x7ff5940cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 >> v1:172.21.15.169:6789/0 conn(0x7ff5940cd7a0 legacy=0x7ff5940cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594070d50 con 0x7ff59415b430 2024-03-06T07:33:18.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff584001690 con 0x7ff59415b430 2024-03-06T07:33:18.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 >> v1:172.21.15.122:6789/0 conn(0x7ff59415b430 legacy=0x7ff594071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800039b0 0x7ff580003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff580004800 0x7ff580004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800051c0 0x7ff58000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff594070a70 con 0x7ff5800051c0 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58000c060 con 0x7ff5800039b0 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58000c260 con 0x7ff580004800 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.519+0000 7ff599fb2640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800051c0 0x7ff58000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff5997b1640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800039b0 0x7ff580003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff598fb0640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff580004800 0x7ff580004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 -- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff580004800 msgr2=0x7ff580004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff580004800 0x7ff580004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 -- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800039b0 msgr2=0x7ff580003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800039b0 0x7ff580003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 -- 172.21.15.122:0/1037183073 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580003520 con 0x7ff5800051c0 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff598fb0640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff580004800 0x7ff580004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 -- 172.21.15.122:0/1037183073 reap_dead start 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.520+0000 7ff599fb2640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800051c0 0x7ff58000b9c0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7ff58c001e50 tx=0x7ff580005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a497908925212e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c001a70 con 0x7ff5800051c0 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c003bb0 con 0x7ff5800051c0 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff58bfff640 1 -- 172.21.15.122:0/1037183073 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0050d0 con 0x7ff5800051c0 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 -- 172.21.15.122:0/1037183073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800051c0 msgr2=0x7ff58000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800051c0 0x7ff58000b9c0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7ff58c001e50 tx=0x7ff580005830 comp rx=0 tx=0).stop 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 -- 172.21.15.122:0/1037183073 shutdown_connections 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 --2- 172.21.15.122:0/1037183073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5800051c0 0x7ff58000b9c0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 -- 172.21.15.122:0/1037183073 >> 172.21.15.122:0/1037183073 conn(0x7ff5940cf8a0 msgr2=0x7ff5940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 -- 172.21.15.122:0/1037183073 shutdown_connections 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.521+0000 7ff59ba15640 1 -- 172.21.15.122:0/1037183073 wait complete. 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.523+0000 7ff59ba15640 1 Processor -- start 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.523+0000 7ff59ba15640 1 -- start start 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff59ba15640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800051c0 0x7ff594147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff59ba15640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff594153700 0x7ff594153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff59ba15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 0x7ff59406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff59ba15640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5940cce40 con 0x7ff59406dce0 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff59ba15640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff594071310 con 0x7ff5800051c0 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff59ba15640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5940ccfc0 con 0x7ff594153700 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff599fb2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 0x7ff59406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff599fb2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 0x7ff59406e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33098/0 (socket says 172.21.15.122:33098) 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.524+0000 7ff599fb2640 1 -- 172.21.15.122:0/3158787732 learned_addr learned my addr 172.21.15.122:0/3158787732 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff5997b1640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800051c0 0x7ff594147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.542 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff598fb0640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff594153700 0x7ff594153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff599fb2640 1 -- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff594153700 msgr2=0x7ff594153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff599fb2640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff594153700 0x7ff594153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff599fb2640 1 -- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800051c0 msgr2=0x7ff594147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff599fb2640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800051c0 0x7ff594147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff599fb2640 1 -- 172.21.15.122:0/3158787732 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594159dc0 con 0x7ff59406dce0 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff598fb0640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff594153700 0x7ff594153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff599fb2640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 0x7ff59406e190 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02c060 tx=0x7ff58c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c007db0 con 0x7ff59406dce0 2024-03-06T07:33:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c003970 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c037560 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff594155130 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.525+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5941556f0 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.526+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff594068840 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.527+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff58c037700 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.527+0000 7ff589ffb640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff57806b4d0 0x7ff57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.527+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff58c0adb20 con 0x7ff59406dce0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.527+0000 7ff5997b1640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff57806b4d0 0x7ff57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.528+0000 7ff5997b1640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff57806b4d0 0x7ff57806d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff5940ccc00 tx=0x7ff590009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.531+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff58c049060 con 0x7ff59406dce0 2024-03-06T07:33:18.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.783+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff59404f330 con 0x7ff59406dce0 2024-03-06T07:33:18.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.784+0000 7ff589ffb640 1 -- 172.21.15.122:0/3158787732 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7ff58c033030 con 0x7ff59406dce0 2024-03-06T07:33:18.786 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:18.786 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:18.786 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:18.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.785+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff57806b4d0 msgr2=0x7ff57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.785+0000 7ff59ba15640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff57806b4d0 0x7ff57806d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff5940ccc00 tx=0x7ff590009190 comp rx=0 tx=0).stop 2024-03-06T07:33:18.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 msgr2=0x7ff59406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 0x7ff59406e190 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02c060 tx=0x7ff58c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 shutdown_connections 2024-03-06T07:33:18.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff57806b4d0 0x7ff57806d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff59406dce0 0x7ff59406e190 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff594153700 0x7ff594153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 --2- 172.21.15.122:0/3158787732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5800051c0 0x7ff594147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 >> 172.21.15.122:0/3158787732 conn(0x7ff5940cf8a0 msgr2=0x7ff594071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 shutdown_connections 2024-03-06T07:33:18.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:18.786+0000 7ff59ba15640 1 -- 172.21.15.122:0/3158787732 wait complete. 2024-03-06T07:33:19.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:19.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:20.001 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:20.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.100+0000 7fc4782df640 1 Processor -- start 2024-03-06T07:33:20.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.100+0000 7fc4782df640 1 -- start start 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc4782df640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fc470158600 0x7fc470072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc4782df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc47015e3a0 0x7fc47006d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc4782df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47015eb40 con 0x7fc47006deb0 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc4782df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc470161ba0 con 0x7fc47015e3a0 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc4782df640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47015ee20 con 0x7fc470158600 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc47587a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc47015e3a0 0x7fc47006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc47687c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc47006deb0 0x7fc470162030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:44984/0 (socket says 172.21.15.122:44984) 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc47587a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc47015e3a0 0x7fc47006d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33102/0 (socket says 172.21.15.122:33102) 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc47687c640 1 -- 172.21.15.122:0/69554628 learned_addr learned my addr 172.21.15.122:0/69554628 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:20.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.101+0000 7fc47607b640 1 --2- 172.21.15.122:0/69554628 >> v2:172.21.15.169:3300/0 conn(0x7fc470158600 0x7fc470072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc47587a640 1 -- 172.21.15.122:0/69554628 >> v2:172.21.15.169:3300/0 conn(0x7fc470158600 msgr2=0x7fc470072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc47587a640 1 --2- 172.21.15.122:0/69554628 >> v2:172.21.15.169:3300/0 conn(0x7fc470158600 0x7fc470072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc47587a640 1 -- 172.21.15.122:0/69554628 >> v1:172.21.15.122:6789/0 conn(0x7fc47006deb0 legacy=0x7fc470162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc475079640 1 -- 172.21.15.122:0/69554628 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770925209 0 0) 0x7fc47015eb40 con 0x7fc47006deb0 2024-03-06T07:33:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc47587a640 1 -- 172.21.15.122:0/69554628 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47015f100 con 0x7fc47015e3a0 2024-03-06T07:33:20.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc47607b640 1 --2- 172.21.15.122:0/69554628 >> v2:172.21.15.169:3300/0 conn(0x7fc470158600 0x7fc470072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc47587a640 1 --2- 172.21.15.122:0/69554628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47015e3a0 0x7fc47006d970 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fc470161d20 tx=0x7fc46c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98643d32584ceff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc475079640 1 -- 172.21.15.122:0/69554628 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c038070 con 0x7fc47015e3a0 2024-03-06T07:33:20.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc475079640 1 -- 172.21.15.122:0/69554628 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c031700 con 0x7fc47015e3a0 2024-03-06T07:33:20.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.102+0000 7fc475079640 1 -- 172.21.15.122:0/69554628 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c031ad0 con 0x7fc47015e3a0 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 -- 172.21.15.122:0/69554628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47015e3a0 msgr2=0x7fc47006d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 --2- 172.21.15.122:0/69554628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47015e3a0 0x7fc47006d970 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fc470161d20 tx=0x7fc46c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 -- 172.21.15.122:0/69554628 shutdown_connections 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 --2- 172.21.15.122:0/69554628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc47015e3a0 0x7fc47006d970 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 --2- 172.21.15.122:0/69554628 >> v2:172.21.15.169:3300/0 conn(0x7fc470158600 0x7fc470072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 -- 172.21.15.122:0/69554628 >> 172.21.15.122:0/69554628 conn(0x7fc47006c0c0 msgr2=0x7fc47006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 -- 172.21.15.122:0/69554628 shutdown_connections 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.103+0000 7fc4782df640 1 -- 172.21.15.122:0/69554628 wait complete. 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.104+0000 7fc4782df640 1 Processor -- start 2024-03-06T07:33:20.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.104+0000 7fc4782df640 1 -- start start 2024-03-06T07:33:20.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc4782df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47006deb0 0x7fc4701443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc4782df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 0x7fc47015a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc4782df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc47015e3a0 0x7fc470152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc4782df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4701628c0 con 0x7fc470158600 2024-03-06T07:33:20.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc4782df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc470162740 con 0x7fc47006deb0 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc4782df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47006afb0 con 0x7fc47015e3a0 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 0x7fc47015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 0x7fc47015a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33104/0 (socket says 172.21.15.122:33104) 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 -- 172.21.15.122:0/691638064 learned_addr learned my addr 172.21.15.122:0/691638064 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47607b640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47006deb0 0x7fc4701443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 -- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc47015e3a0 msgr2=0x7fc470152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47687c640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc47015e3a0 0x7fc470152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc47015e3a0 0x7fc470152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 -- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47006deb0 msgr2=0x7fc4701443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47006deb0 0x7fc4701443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 -- 172.21.15.122:0/691638064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc470144920 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.105+0000 7fc47587a640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 0x7fc47015a950 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fc46c000c00 tx=0x7fc46c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.106+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c042070 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.106+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c033a30 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.106+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c038040 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.106+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4701537a0 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.106+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc470153d60 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.107+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47004f330 con 0x7fc470158600 2024-03-06T07:33:20.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.107+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc46c045070 con 0x7fc470158600 2024-03-06T07:33:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.108+0000 7fc466ffd640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45406b500 0x7fc45406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.108+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc46c0ac170 con 0x7fc470158600 2024-03-06T07:33:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.108+0000 7fc47607b640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45406b500 0x7fc45406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.108+0000 7fc47607b640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45406b500 0x7fc45406d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc460008850 tx=0x7fc460008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.112+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc470144920 con 0x7fc470158600 2024-03-06T07:33:20.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.327+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4700af780 con 0x7fc470158600 2024-03-06T07:33:20.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.327+0000 7fc466ffd640 1 -- 172.21.15.122:0/691638064 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fc46c030070 con 0x7fc470158600 2024-03-06T07:33:20.329 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:20.329 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:20.330 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45406b500 msgr2=0x7fc45406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45406b500 0x7fc45406d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc460008850 tx=0x7fc460008040 comp rx=0 tx=0).stop 2024-03-06T07:33:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 msgr2=0x7fc47015a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 0x7fc47015a950 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fc46c000c00 tx=0x7fc46c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 shutdown_connections 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc45406b500 0x7fc45406d9c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc47015e3a0 0x7fc470152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc470158600 0x7fc47015a950 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 --2- 172.21.15.122:0/691638064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47006deb0 0x7fc4701443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 >> 172.21.15.122:0/691638064 conn(0x7fc47006c0c0 msgr2=0x7fc47006efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 shutdown_connections 2024-03-06T07:33:20.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:20.329+0000 7fc4782df640 1 -- 172.21.15.122:0/691638064 wait complete. 2024-03-06T07:33:20.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:20.545 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:21.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.646+0000 7fa9efcd4640 1 Processor -- start 2024-03-06T07:33:21.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.647+0000 7fa9efcd4640 1 -- start start 2024-03-06T07:33:21.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.647+0000 7fa9efcd4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 0x7fa9e80b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.647+0000 7fa9efcd4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80b3430 con 0x7fa9e815dfe0 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.647+0000 7fa9efcd4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80b3710 con 0x7fa9e80d22b0 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.647+0000 7fa9efcd4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80b31e0 con 0x7fa9e80d2710 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eda70640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa9e815dfe0 0x7fa9e80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45000/0 (socket says 172.21.15.122:45000) 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eda70640 1 -- 172.21.15.122:0/57744800 learned_addr learned my addr 172.21.15.122:0/57744800 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9ee271640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 0x7fa9e80b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944507978 0 0) 0x7fa9e80b3430 con 0x7fa9e815dfe0 2024-03-06T07:33:21.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0003520 con 0x7fa9e815dfe0 2024-03-06T07:33:21.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3185553503 0 0) 0x7fa9d0003520 con 0x7fa9e815dfe0 2024-03-06T07:33:21.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 msgr2=0x7fa9e80b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 0x7fa9e80b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 >> v1:172.21.15.169:6789/0 conn(0x7fa9e80d22b0 legacy=0x7fa9e80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80b39f0 con 0x7fa9e815dfe0 2024-03-06T07:33:21.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa9d8001670 con 0x7fa9e815dfe0 2024-03-06T07:33:21.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 >> v1:172.21.15.122:6789/0 conn(0x7fa9e815dfe0 legacy=0x7fa9e80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 0x7fa9d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9d0003f40 0x7fa9d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d0004d40 0x7fa9d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80b31e0 con 0x7fa9d0004d40 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d000ba70 con 0x7fa9d0003f40 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d000bc40 con 0x7fa9e80d2710 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9ee271640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d0004d40 0x7fa9d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9eda70640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 0x7fa9d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.648+0000 7fa9ed26f640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9d0003f40 0x7fa9d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.649+0000 7fa9ee271640 1 -- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 msgr2=0x7fa9d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.649+0000 7fa9ee271640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80d2710 0x7fa9d00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.649+0000 7fa9ee271640 1 -- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9d0003f40 msgr2=0x7fa9d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.649+0000 7fa9ee271640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9d0003f40 0x7fa9d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.649+0000 7fa9ee271640 1 -- 172.21.15.122:0/57744800 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0003520 con 0x7fa9d0004d40 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.649+0000 7fa9ee271640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d0004d40 0x7fa9d00051c0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000e170 tx=0x7fa9d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b37811205194bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0039070 con 0x7fa9d0004d40 2024-03-06T07:33:21.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e002cbf0 con 0x7fa9d0004d40 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9eca6e640 1 -- 172.21.15.122:0/57744800 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0009c10 con 0x7fa9d0004d40 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9efcd4640 1 -- 172.21.15.122:0/57744800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d0004d40 msgr2=0x7fa9d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/57744800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d0004d40 0x7fa9d00051c0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000e170 tx=0x7fa9d0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9ee271640 1 -- 172.21.15.122:0/57744800 reap_dead start 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9efcd4640 1 -- 172.21.15.122:0/57744800 shutdown_connections 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9efcd4640 1 -- 172.21.15.122:0/57744800 >> 172.21.15.122:0/57744800 conn(0x7fa9e80d3cd0 msgr2=0x7fa9e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9efcd4640 1 -- 172.21.15.122:0/57744800 shutdown_connections 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.650+0000 7fa9efcd4640 1 -- 172.21.15.122:0/57744800 wait complete. 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.651+0000 7fa9efcd4640 1 Processor -- start 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.651+0000 7fa9efcd4640 1 -- start start 2024-03-06T07:33:21.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9efcd4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9e80cc290 0x7fa9e814b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9efcd4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 0x7fa9e80ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9efcd4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80cd260 0x7fa9e80b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9efcd4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80cfa00 con 0x7fa9e80b68d0 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9efcd4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80b3d60 con 0x7fa9e80cc290 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9efcd4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80d1c70 con 0x7fa9e80cd260 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 0x7fa9e80ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 0x7fa9e80ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33136/0 (socket says 172.21.15.122:33136) 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 -- 172.21.15.122:0/1807209650 learned_addr learned my addr 172.21.15.122:0/1807209650 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:21.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ee271640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80cd260 0x7fa9e80b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9eda70640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9e80cc290 0x7fa9e814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 -- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80cd260 msgr2=0x7fa9e80b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80cd260 0x7fa9e80b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 -- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9e80cc290 msgr2=0x7fa9e814b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9e80cc290 0x7fa9e814b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 -- 172.21.15.122:0/1807209650 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80cc670 con 0x7fa9e80b68d0 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9eda70640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9e80cc290 0x7fa9e814b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:21.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.652+0000 7fa9ed26f640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 0x7fa9e80ccd20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00f8b0 tx=0x7fa9dc00fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.653+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0052e0 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.653+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc018070 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.653+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e80b9490 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.653+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc00d910 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.653+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e80b9a50 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.654+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e804f330 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.654+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9dc00dab0 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.654+0000 7fa9e67fc640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa9d406b4d0 0x7fa9d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.654+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa9dc08a0d0 con 0x7fa9e80b68d0 2024-03-06T07:33:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.654+0000 7fa9eda70640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa9d406b4d0 0x7fa9d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.655+0000 7fa9eda70640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa9d406b4d0 0x7fa9d406d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8004840 tx=0x7fa9d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.659+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9dc058fc0 con 0x7fa9e80b68d0 2024-03-06T07:33:21.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.868+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9e814b780 con 0x7fa9e80b68d0 2024-03-06T07:33:21.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.869+0000 7fa9e67fc640 1 -- 172.21.15.122:0/1807209650 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa9dc0588c0 con 0x7fa9e80b68d0 2024-03-06T07:33:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:21.871 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:21.872 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:21.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa9d406b4d0 msgr2=0x7fa9d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa9d406b4d0 0x7fa9d406d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8004840 tx=0x7fa9d8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 msgr2=0x7fa9e80ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 0x7fa9e80ccd20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00f8b0 tx=0x7fa9dc00fd70 comp rx=0 tx=0).stop 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 shutdown_connections 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa9d406b4d0 0x7fa9d406d990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa9e80cd260 0x7fa9e80b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9e80b68d0 0x7fa9e80ccd20 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 --2- 172.21.15.122:0/1807209650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9e80cc290 0x7fa9e814b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 >> 172.21.15.122:0/1807209650 conn(0x7fa9e80d3cd0 msgr2=0x7fa9e80b5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 shutdown_connections 2024-03-06T07:33:21.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:21.871+0000 7fa9efcd4640 1 -- 172.21.15.122:0/1807209650 wait complete. 2024-03-06T07:33:22.086 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:22.086 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:23.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.185+0000 7fae1e8a4640 1 Processor -- start 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.185+0000 7fae1e8a4640 1 -- start start 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1e8a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 0x7fae180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1e8a4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fae180cf830 0x7fae180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1e8a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae180b3430 con 0x7fae180d27f0 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1e8a4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae180d2c50 con 0x7fae180cf830 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1e8a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae180b31e0 con 0x7fae1815dfe0 2024-03-06T07:33:23.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1ce41640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fae180d27f0 0x7fae180d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45008/0 (socket says 172.21.15.122:45008) 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae1ce41640 1 -- 172.21.15.122:0/3239167023 learned_addr learned my addr 172.21.15.122:0/3239167023 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae17fff640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 0x7fae180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae177fe640 1 --2- 172.21.15.122:0/3239167023 >> v2:172.21.15.169:3300/0 conn(0x7fae180cf830 0x7fae180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866554714 0 0) 0x7fae180b3430 con 0x7fae180d27f0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae00003520 con 0x7fae180d27f0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003546925 0 0) 0x7fae00003520 con 0x7fae180d27f0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 msgr2=0x7fae180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 0x7fae180cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 >> v2:172.21.15.169:3300/0 conn(0x7fae180cf830 msgr2=0x7fae180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 --2- 172.21.15.122:0/3239167023 >> v2:172.21.15.169:3300/0 conn(0x7fae180cf830 0x7fae180d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae180b3710 con 0x7fae180d27f0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.186+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fae100019f0 con 0x7fae180d27f0 2024-03-06T07:33:23.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 >> v1:172.21.15.122:6789/0 conn(0x7fae180d27f0 legacy=0x7fae180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae000039b0 0x7fae00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 0x7fae00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae00004d40 0x7fae000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180b31e0 con 0x7fae00004d40 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000b9c0 con 0x7fae000039b0 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000bb40 con 0x7fae1815dfe0 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae1ce41640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae00004d40 0x7fae000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae17fff640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae000039b0 0x7fae00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae177fe640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 0x7fae00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae1ce41640 1 -- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 msgr2=0x7fae00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae1ce41640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae1815dfe0 0x7fae00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae1ce41640 1 -- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae000039b0 msgr2=0x7fae00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae1ce41640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae000039b0 0x7fae00003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.187+0000 7fae1ce41640 1 -- 172.21.15.122:0/3239167023 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00003520 con 0x7fae00004d40 2024-03-06T07:33:23.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.188+0000 7fae1ce41640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae00004d40 0x7fae000051c0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fae1000a870 tx=0x7fae00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13a2197da00eedc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.188+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000a4c0 con 0x7fae00004d40 2024-03-06T07:33:23.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.188+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae10034630 con 0x7fae00004d40 2024-03-06T07:33:23.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.188+0000 7fae16ffd640 1 -- 172.21.15.122:0/3239167023 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10034a20 con 0x7fae00004d40 2024-03-06T07:33:23.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1e8a4640 1 -- 172.21.15.122:0/3239167023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae00004d40 msgr2=0x7fae000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/3239167023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae00004d40 0x7fae000051c0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fae1000a870 tx=0x7fae00005830 comp rx=0 tx=0).stop 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1ce41640 1 -- 172.21.15.122:0/3239167023 reap_dead start 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1e8a4640 1 -- 172.21.15.122:0/3239167023 shutdown_connections 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1e8a4640 1 -- 172.21.15.122:0/3239167023 >> 172.21.15.122:0/3239167023 conn(0x7fae180d3cd0 msgr2=0x7fae180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1e8a4640 1 -- 172.21.15.122:0/3239167023 shutdown_connections 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.189+0000 7fae1e8a4640 1 -- 172.21.15.122:0/3239167023 wait complete. 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.190+0000 7fae1e8a4640 1 Processor -- start 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.190+0000 7fae1e8a4640 1 -- start start 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18153260 0x7fae180b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 0x7fae180b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae180b57c0 0x7fae180b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180d1c70 con 0x7fae180b7440 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180b3d60 con 0x7fae18153260 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae180d1df0 con 0x7fae180b57c0 2024-03-06T07:33:23.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 0x7fae180b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 0x7fae180b83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33148/0 (socket says 172.21.15.122:33148) 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 -- 172.21.15.122:0/1553407584 learned_addr learned my addr 172.21.15.122:0/1553407584 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae17fff640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18153260 0x7fae180b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 -- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae180b57c0 msgr2=0x7fae180b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1ce41640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae180b57c0 0x7fae180b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae180b57c0 0x7fae180b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 -- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18153260 msgr2=0x7fae180b7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18153260 0x7fae180b7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 -- 172.21.15.122:0/1553407584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1815c6c0 con 0x7fae180b7440 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae17fff640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18153260 0x7fae180b7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1ce41640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae180b57c0 0x7fae180b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae177fe640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 0x7fae180b83d0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fae0c0119d0 tx=0x7fae0c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c019070 con 0x7fae180b7440 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c004240 con 0x7fae180b7440 2024-03-06T07:33:23.198 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c00f940 con 0x7fae180b7440 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.191+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae180b63a0 con 0x7fae180b7440 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.192+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae180b6960 con 0x7fae180b7440 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.192+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1804f330 con 0x7fae180b7440 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.193+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae0c01c070 con 0x7fae180b7440 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.193+0000 7fae14ff9640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0406b500 0x7fae0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.194+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fae0c08a670 con 0x7fae180b7440 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.194+0000 7fae17fff640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0406b500 0x7fae0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.194+0000 7fae17fff640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0406b500 0x7fae0406d9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fae08009ab0 tx=0x7fae08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.198+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae1815c6c0 con 0x7fae180b7440 2024-03-06T07:33:23.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.463+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae180cdd20 con 0x7fae180b7440 2024-03-06T07:33:23.466 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.464+0000 7fae14ff9640 1 -- 172.21.15.122:0/1553407584 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fae0c059560 con 0x7fae180b7440 2024-03-06T07:33:23.466 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:23.467 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:23.467 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0406b500 msgr2=0x7fae0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0406b500 0x7fae0406d9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fae08009ab0 tx=0x7fae08008040 comp rx=0 tx=0).stop 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 msgr2=0x7fae180b83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 0x7fae180b83d0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fae0c0119d0 tx=0x7fae0c011e90 comp rx=0 tx=0).stop 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 shutdown_connections 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae0406b500 0x7fae0406d9c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae180b57c0 0x7fae180b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae180b7440 0x7fae180b83d0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 --2- 172.21.15.122:0/1553407584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae18153260 0x7fae180b7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.469 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 >> 172.21.15.122:0/1553407584 conn(0x7fae180d3cd0 msgr2=0x7fae180ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 shutdown_connections 2024-03-06T07:33:23.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:23.466+0000 7fae1e8a4640 1 -- 172.21.15.122:0/1553407584 wait complete. 2024-03-06T07:33:23.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:23.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:24.682 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:24.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 Processor -- start 2024-03-06T07:33:24.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 -- start start 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 0x7f339c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f339c0d29c0 0x7f339c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c0b3430 con 0x7f339c0cf830 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0b31e0 con 0x7f339c0d29c0 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.781+0000 7f33a1593640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0b3d60 con 0x7f339c15dfe0 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f339c0d29c0 0x7f339c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339affd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 0x7f339c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339affd640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 0x7f339c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:52762/0 (socket says 172.21.15.122:52762) 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339a7fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f339c0cf830 0x7f339c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:38276/0 (socket says 172.21.15.122:38276) 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339b7fe640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f339c0d29c0 0x7f339c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:44016/0 (socket says 172.21.15.122:44016) 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339affd640 1 -- 172.21.15.122:0/1703046378 learned_addr learned my addr 172.21.15.122:0/1703046378 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339b7fe640 1 -- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 msgr2=0x7f339c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339b7fe640 1 --2- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 0x7f339c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339b7fe640 1 -- 172.21.15.122:0/1703046378 >> v1:172.21.15.169:6789/0 conn(0x7f339c0cf830 legacy=0x7f339c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339b7fe640 1 -- 172.21.15.122:0/1703046378 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0b3710 con 0x7f339c0d29c0 2024-03-06T07:33:24.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.782+0000 7f339affd640 1 --2- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 0x7f339c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:24.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.783+0000 7f339b7fe640 1 --2- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c0d29c0 0x7f339c0b41f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097a0 tx=0x7f338c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ed184ab1f747d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.783+0000 7f3399ffb640 1 -- 172.21.15.122:0/1703046378 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c035020 con 0x7f339c0d29c0 2024-03-06T07:33:24.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.783+0000 7f3399ffb640 1 -- 172.21.15.122:0/1703046378 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c031830 con 0x7f339c0d29c0 2024-03-06T07:33:24.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.783+0000 7f3399ffb640 1 -- 172.21.15.122:0/1703046378 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c031c00 con 0x7f339c0d29c0 2024-03-06T07:33:24.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 -- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c0d29c0 msgr2=0x7f339c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 --2- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c0d29c0 0x7f339c0b41f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097a0 tx=0x7f338c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 -- 172.21.15.122:0/1703046378 shutdown_connections 2024-03-06T07:33:24.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 --2- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c0d29c0 0x7f339c0b41f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 --2- 172.21.15.122:0/1703046378 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c15dfe0 0x7f339c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 -- 172.21.15.122:0/1703046378 >> 172.21.15.122:0/1703046378 conn(0x7f339c0d3cd0 msgr2=0x7f339c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 -- 172.21.15.122:0/1703046378 shutdown_connections 2024-03-06T07:33:24.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.784+0000 7f33a1593640 1 -- 172.21.15.122:0/1703046378 wait complete. 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.785+0000 7f33a1593640 1 Processor -- start 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.785+0000 7f33a1593640 1 -- start start 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f33a1593640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c0cf830 0x7f339c1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f33a1593640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 0x7f339c0b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f33a1593640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c15dfe0 0x7f339c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f33a1593640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0d1df0 con 0x7f339c0d29c0 2024-03-06T07:33:24.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f33a1593640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0d1c70 con 0x7f339c15dfe0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f33a1593640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c15e3c0 con 0x7f339c0cf830 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 0x7f339c0b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 0x7f339c0b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43696/0 (socket says 172.21.15.122:43696) 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 -- 172.21.15.122:0/2875276943 learned_addr learned my addr 172.21.15.122:0/2875276943 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339affd640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c0cf830 0x7f339c1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339b7fe640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c15dfe0 0x7f339c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 -- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c0cf830 msgr2=0x7f339c1572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c0cf830 0x7f339c1572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 -- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c15dfe0 msgr2=0x7f339c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c15dfe0 0x7f339c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 -- 172.21.15.122:0/2875276943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c15e790 con 0x7f339c0d29c0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.786+0000 7f339a7fc640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 0x7f339c0b72e0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f339000c440 tx=0x7f3390001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.787+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33900074e0 con 0x7f339c0d29c0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.787+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3390007680 con 0x7f339c0d29c0 2024-03-06T07:33:24.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.787+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c15ea70 con 0x7f339c0d29c0 2024-03-06T07:33:24.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.787+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390004a20 con 0x7f339c0d29c0 2024-03-06T07:33:24.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.787+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c15f030 con 0x7f339c0d29c0 2024-03-06T07:33:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.788+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c04f330 con 0x7f339c0d29c0 2024-03-06T07:33:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.788+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f339001a020 con 0x7f339c0d29c0 2024-03-06T07:33:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.789+0000 7f33837fe640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f338806b4d0 0x7f338806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.789+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3390089a00 con 0x7f339c0d29c0 2024-03-06T07:33:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.789+0000 7f339affd640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f338806b4d0 0x7f338806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.789+0000 7f339affd640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f338806b4d0 0x7f338806d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3394009ce0 tx=0x7f339400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.796 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:24.793+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3390010300 con 0x7f339c0d29c0 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.007+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f339c0747a0 con 0x7f339c0d29c0 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.007+0000 7f33837fe640 1 -- 172.21.15.122:0/2875276943 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f3390058870 con 0x7f339c0d29c0 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:25.010 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:25.010 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:25.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f338806b4d0 msgr2=0x7f338806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f338806b4d0 0x7f338806d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3394009ce0 tx=0x7f339400b040 comp rx=0 tx=0).stop 2024-03-06T07:33:25.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 msgr2=0x7f339c0b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 0x7f339c0b72e0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f339000c440 tx=0x7f3390001e70 comp rx=0 tx=0).stop 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 shutdown_connections 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f338806b4d0 0x7f338806d990 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f339c15dfe0 0x7f339c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f339c0d29c0 0x7f339c0b72e0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 --2- 172.21.15.122:0/2875276943 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f339c0cf830 0x7f339c1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 >> 172.21.15.122:0/2875276943 conn(0x7f339c0d3cd0 msgr2=0x7f339c0b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 shutdown_connections 2024-03-06T07:33:25.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:25.009+0000 7f33a1593640 1 -- 172.21.15.122:0/2875276943 wait complete. 2024-03-06T07:33:25.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:25.226 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:26.077 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:26.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:26.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:26.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.329+0000 7f49aa79a640 1 Processor -- start 2024-03-06T07:33:26.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.329+0000 7f49aa79a640 1 -- start start 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49aa79a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f49a40d22b0 0x7f49a40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49aa79a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f49a40b41f0 0x7f49a40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49aa79a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a40b3430 con 0x7f49a415dfe0 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49aa79a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a40d2c50 con 0x7f49a40d22b0 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49aa79a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a40b31e0 con 0x7f49a40b41f0 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f49a40d22b0 0x7f49a40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f49a415dfe0 0x7f49a40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38278/0 (socket says 172.21.15.122:38278) 2024-03-06T07:33:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a3fff640 1 -- 172.21.15.122:0/1099680295 learned_addr learned my addr 172.21.15.122:0/1099680295 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f49a40d22b0 0x7f49a40d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43712/0 (socket says 172.21.15.122:43712) 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a37fe640 1 -- 172.21.15.122:0/1099680295 >> v2:172.21.15.169:3300/0 conn(0x7f49a40b41f0 msgr2=0x7f49a40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a37fe640 1 --2- 172.21.15.122:0/1099680295 >> v2:172.21.15.169:3300/0 conn(0x7f49a40b41f0 0x7f49a40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a37fe640 1 -- 172.21.15.122:0/1099680295 >> v1:172.21.15.122:6789/0 conn(0x7f49a415dfe0 legacy=0x7f49a40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a2ffd640 1 -- 172.21.15.122:0/1099680295 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251114155 0 0) 0x7f49a40b3430 con 0x7f49a415dfe0 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.330+0000 7f49a37fe640 1 -- 172.21.15.122:0/1099680295 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a40b3710 con 0x7f49a40d22b0 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.331+0000 7f49a37fe640 1 --2- 172.21.15.122:0/1099680295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40d22b0 0x7f49a40d2710 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f49940097a0 tx=0x7f499402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79bf1561f963282 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.331+0000 7f49a2ffd640 1 -- 172.21.15.122:0/1099680295 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994035020 con 0x7f49a40d22b0 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.331+0000 7f49a2ffd640 1 -- 172.21.15.122:0/1099680295 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4994031830 con 0x7f49a40d22b0 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.331+0000 7f49a2ffd640 1 -- 172.21.15.122:0/1099680295 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994031c00 con 0x7f49a40d22b0 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 -- 172.21.15.122:0/1099680295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40d22b0 msgr2=0x7f49a40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1099680295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40d22b0 0x7f49a40d2710 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f49940097a0 tx=0x7f499402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 -- 172.21.15.122:0/1099680295 shutdown_connections 2024-03-06T07:33:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1099680295 >> v2:172.21.15.169:3300/0 conn(0x7f49a40b41f0 0x7f49a40b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1099680295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40d22b0 0x7f49a40d2710 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 -- 172.21.15.122:0/1099680295 >> 172.21.15.122:0/1099680295 conn(0x7f49a40d3cd0 msgr2=0x7f49a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 -- 172.21.15.122:0/1099680295 shutdown_connections 2024-03-06T07:33:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.332+0000 7f49aa79a640 1 -- 172.21.15.122:0/1099680295 wait complete. 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.333+0000 7f49aa79a640 1 Processor -- start 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.333+0000 7f49aa79a640 1 -- start start 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49aa79a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 0x7f49a41f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49aa79a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49a40d22b0 0x7f49a40b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49aa79a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49a415dfe0 0x7f49a40b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49aa79a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40cfa00 con 0x7f49a40b41f0 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49aa79a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40b3d60 con 0x7f49a415dfe0 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49aa79a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40d1c70 con 0x7f49a40d22b0 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 0x7f49a41f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 0x7f49a41f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43714/0 (socket says 172.21.15.122:43714) 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 -- 172.21.15.122:0/1585304240 learned_addr learned my addr 172.21.15.122:0/1585304240 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 -- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49a40d22b0 msgr2=0x7f49a40b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a37fe640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49a40d22b0 0x7f49a40b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49a40d22b0 0x7f49a40b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a8d37640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49a415dfe0 0x7f49a40b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 -- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49a415dfe0 msgr2=0x7f49a40b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49a415dfe0 0x7f49a40b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a3fff640 1 -- 172.21.15.122:0/1585304240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a40b9b80 con 0x7f49a40b41f0 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a37fe640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49a40d22b0 0x7f49a40b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.334+0000 7f49a8d37640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49a415dfe0 0x7f49a40b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.335+0000 7f49a3fff640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 0x7f49a41f3250 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f499800c430 tx=0x7f499800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.335+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49980074e0 con 0x7f49a40b41f0 2024-03-06T07:33:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.335+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49980047e0 con 0x7f49a40b41f0 2024-03-06T07:33:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.335+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a415e4f0 con 0x7f49a40b41f0 2024-03-06T07:33:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.335+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998004bd0 con 0x7f49a40b41f0 2024-03-06T07:33:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.335+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a415eab0 con 0x7f49a40b41f0 2024-03-06T07:33:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.336+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a404f330 con 0x7f49a40b41f0 2024-03-06T07:33:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.336+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4998011070 con 0x7f49a40b41f0 2024-03-06T07:33:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.337+0000 7f49a0ff9640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f499006b4d0 0x7f499006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.337+0000 7f49a37fe640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f499006b4d0 0x7f499006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.337+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4998089e00 con 0x7f49a40b41f0 2024-03-06T07:33:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.338+0000 7f49a37fe640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f499006b4d0 0x7f499006d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4994036000 tx=0x7f4994008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.343 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.341+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4998058cf0 con 0x7f49a40b41f0 2024-03-06T07:33:26.560 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.558+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a40d2970 con 0x7f49a40b41f0 2024-03-06T07:33:26.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.558+0000 7f49a0ff9640 1 -- 172.21.15.122:0/1585304240 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f49980585f0 con 0x7f49a40b41f0 2024-03-06T07:33:26.561 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:26.561 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:26.561 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f499006b4d0 msgr2=0x7f499006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f499006b4d0 0x7f499006d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4994036000 tx=0x7f4994008030 comp rx=0 tx=0).stop 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 msgr2=0x7f49a41f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 0x7f49a41f3250 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f499800c430 tx=0x7f499800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 shutdown_connections 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f499006b4d0 0x7f499006d990 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49a415dfe0 0x7f49a40b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f49a40d22b0 0x7f49a40b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 --2- 172.21.15.122:0/1585304240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49a40b41f0 0x7f49a41f3250 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 >> 172.21.15.122:0/1585304240 conn(0x7f49a40d3cd0 msgr2=0x7f49a40d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 shutdown_connections 2024-03-06T07:33:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:26.560+0000 7f49aa79a640 1 -- 172.21.15.122:0/1585304240 wait complete. 2024-03-06T07:33:26.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:26.775 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:27.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:27.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.875+0000 7fa06365c640 1 Processor -- start 2024-03-06T07:33:27.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.875+0000 7fa06365c640 1 -- start start 2024-03-06T07:33:27.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa06365c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa05c15e090 0x7fa05c0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa06365c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d0650 0x7fa05c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa06365c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c15e470 con 0x7fa05c0d2920 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa06365c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0d18f0 con 0x7fa05c15e090 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa06365c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0b3cf0 con 0x7fa05c0d0650 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa0613f8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa05c15e090 0x7fa05c0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa0613f8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa05c15e090 0x7fa05c0d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43718/0 (socket says 172.21.15.122:43718) 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa060bf7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa05c0d2920 0x7fa05c0b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38284/0 (socket says 172.21.15.122:38284) 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.876+0000 7fa0613f8640 1 -- 172.21.15.122:0/245377123 learned_addr learned my addr 172.21.15.122:0/245377123 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa061bf9640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d0650 0x7fa05c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa0613f8640 1 -- 172.21.15.122:0/245377123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d0650 msgr2=0x7fa05c0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa0613f8640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d0650 0x7fa05c0d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa0613f8640 1 -- 172.21.15.122:0/245377123 >> v1:172.21.15.122:6789/0 conn(0x7fa05c0d2920 legacy=0x7fa05c0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa0613f8640 1 -- 172.21.15.122:0/245377123 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c0d1b90 con 0x7fa05c15e090 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa053fff640 1 -- 172.21.15.122:0/245377123 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550135747 0 0) 0x7fa05c15e470 con 0x7fa05c0d2920 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa061bf9640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d0650 0x7fa05c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.877+0000 7fa0613f8640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c0d47a0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009870 tx=0x7fa04c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596398454db6f6e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa053fff640 1 -- 172.21.15.122:0/245377123 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c034070 con 0x7fa05c15e090 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa053fff640 1 -- 172.21.15.122:0/245377123 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c030840 con 0x7fa05c15e090 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa053fff640 1 -- 172.21.15.122:0/245377123 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c030c10 con 0x7fa05c15e090 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 -- 172.21.15.122:0/245377123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 msgr2=0x7fa05c0d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c0d47a0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009870 tx=0x7fa04c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:27.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 -- 172.21.15.122:0/245377123 shutdown_connections 2024-03-06T07:33:27.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d0650 0x7fa05c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 --2- 172.21.15.122:0/245377123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c0d47a0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 -- 172.21.15.122:0/245377123 >> 172.21.15.122:0/245377123 conn(0x7fa05c0d3c80 msgr2=0x7fa05c0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 -- 172.21.15.122:0/245377123 shutdown_connections 2024-03-06T07:33:27.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.878+0000 7fa06365c640 1 -- 172.21.15.122:0/245377123 wait complete. 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.879+0000 7fa06365c640 1 Processor -- start 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.879+0000 7fa06365c640 1 -- start start 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa06365c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa05c0d0650 0x7fa05c15fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa06365c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d2920 0x7fa05c156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa06365c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c1572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa06365c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0b2df0 con 0x7fa05c15e090 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa06365c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0cee30 con 0x7fa05c0d0650 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa06365c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0ce800 con 0x7fa05c0d2920 2024-03-06T07:33:27.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c1572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c1572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43722/0 (socket says 172.21.15.122:43722) 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 -- 172.21.15.122:0/3805441437 learned_addr learned my addr 172.21.15.122:0/3805441437 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa060bf7640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d2920 0x7fa05c156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa0613f8640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa05c0d0650 0x7fa05c15fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 -- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d2920 msgr2=0x7fa05c156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d2920 0x7fa05c156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 -- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa05c0d0650 msgr2=0x7fa05c15fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa05c0d0650 0x7fa05c15fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa061bf9640 1 -- 172.21.15.122:0/3805441437 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c15e9d0 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa060bf7640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d2920 0x7fa05c156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.880+0000 7fa0613f8640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa05c0d0650 0x7fa05c15fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.881+0000 7fa061bf9640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c1572f0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa05400cd50 tx=0x7fa05400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.881+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400f680 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.881+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05400f840 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.881+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c15ecb0 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.881+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400fc30 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.881+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c15f270 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.882+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c0cab00 con 0x7fa05c15e090 2024-03-06T07:33:27.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.883+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0540213f0 con 0x7fa05c15e090 2024-03-06T07:33:27.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.883+0000 7fa051ffb640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa04006b4d0 0x7fa04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.883+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa05408aff0 con 0x7fa05c15e090 2024-03-06T07:33:27.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.883+0000 7fa0613f8640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa04006b4d0 0x7fa04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.884+0000 7fa0613f8640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa04006b4d0 0x7fa04006d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa04c036000 tx=0x7fa04c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:27.887+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa05c0cab00 con 0x7fa05c15e090 2024-03-06T07:33:28.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.102+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa05c0d1230 con 0x7fa05c15e090 2024-03-06T07:33:28.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.103+0000 7fa051ffb640 1 -- 172.21.15.122:0/3805441437 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa054014030 con 0x7fa05c15e090 2024-03-06T07:33:28.105 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:28.105 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:28.105 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.104+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa04006b4d0 msgr2=0x7fa04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.104+0000 7fa06365c640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa04006b4d0 0x7fa04006d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa04c036000 tx=0x7fa04c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.104+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 msgr2=0x7fa05c1572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.104+0000 7fa06365c640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c1572f0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa05400cd50 tx=0x7fa05400a650 comp rx=0 tx=0).stop 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 shutdown_connections 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa04006b4d0 0x7fa04006d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa05c15e090 0x7fa05c1572f0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa05c0d2920 0x7fa05c156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 --2- 172.21.15.122:0/3805441437 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa05c0d0650 0x7fa05c15fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 >> 172.21.15.122:0/3805441437 conn(0x7fa05c0d3c80 msgr2=0x7fa05c0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 shutdown_connections 2024-03-06T07:33:28.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:28.105+0000 7fa06365c640 1 -- 172.21.15.122:0/3805441437 wait complete. 2024-03-06T07:33:28.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:28.320 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:29.321 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:29.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.420+0000 7f1c1ef8c640 1 Processor -- start 2024-03-06T07:33:29.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.420+0000 7f1c1ef8c640 1 -- start start 2024-03-06T07:33:29.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.421+0000 7f1c1ef8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c180d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.421+0000 7f1c1ef8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180b3430 con 0x7f1c180b41f0 2024-03-06T07:33:29.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.421+0000 7f1c1ef8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180b3710 con 0x7f1c1815dfe0 2024-03-06T07:33:29.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.421+0000 7f1c1ef8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180d2c50 con 0x7f1c180d22b0 2024-03-06T07:33:29.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c1d529640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1c180b41f0 0x7f1c180b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38290/0 (socket says 172.21.15.122:38290) 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c1d529640 1 -- 172.21.15.122:0/2801935744 learned_addr learned my addr 172.21.15.122:0/2801935744 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c17fff640 1 --2- 172.21.15.122:0/2801935744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c180d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879465533 0 0) 0x7f1c180b3430 con 0x7f1c180b41f0 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c00003520 con 0x7f1c180b41f0 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188539490 0 0) 0x7f1c180b3710 con 0x7f1c1815dfe0 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c180b3430 con 0x7f1c1815dfe0 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c17fff640 1 -- 172.21.15.122:0/2801935744 >> v1:172.21.15.169:6789/0 conn(0x7f1c1815dfe0 legacy=0x7f1c180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358646249 0 0) 0x7f1c00003520 con 0x7f1c180b41f0 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c17fff640 1 -- 172.21.15.122:0/2801935744 >> v1:172.21.15.122:6789/0 conn(0x7f1c180b41f0 legacy=0x7f1c180b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.422+0000 7f1c17fff640 1 -- 172.21.15.122:0/2801935744 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c180b39f0 con 0x7f1c180d22b0 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.423+0000 7f1c17fff640 1 --2- 172.21.15.122:0/2801935744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c180d2710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0097d0 tx=0x7f1c0c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a91a5e685ff17e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.423+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c037070 con 0x7f1c180d22b0 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.423+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c030510 con 0x7f1c180d22b0 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.423+0000 7f1c177fe640 1 -- 172.21.15.122:0/2801935744 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08001670 con 0x7f1c180d22b0 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2801935744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 msgr2=0x7f1c180d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2801935744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c180d2710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0097d0 tx=0x7f1c0c02b800 comp rx=0 tx=0).stop 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2801935744 shutdown_connections 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2801935744 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c180d2710 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2801935744 >> 172.21.15.122:0/2801935744 conn(0x7f1c180d3cd0 msgr2=0x7f1c180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2801935744 shutdown_connections 2024-03-06T07:33:29.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.424+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2801935744 wait complete. 2024-03-06T07:33:29.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.425+0000 7f1c1ef8c640 1 Processor -- start 2024-03-06T07:33:29.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.425+0000 7f1c1ef8c640 1 -- start start 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1ef8c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 0x7f1c1814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1ef8c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c1815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1ef8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c1815dfe0 0x7f1c180b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1ef8c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180b3d60 con 0x7f1c180b41f0 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1ef8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180b31e0 con 0x7f1c1815dfe0 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1ef8c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180d1c70 con 0x7f1c180d22b0 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 0x7f1c1814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 0x7f1c1814f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43728/0 (socket says 172.21.15.122:43728) 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 -- 172.21.15.122:0/2894988721 learned_addr learned my addr 172.21.15.122:0/2894988721 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1d529640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c1815dfe0 0x7f1c180b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c17fff640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c1815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 -- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 msgr2=0x7f1c1815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c1815e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 -- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c1815dfe0 msgr2=0x7f1c180b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c1815dfe0 0x7f1c180b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 -- 172.21.15.122:0/2894988721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1815ed30 con 0x7f1c180b41f0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c17fff640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c1815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1d529640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c1815dfe0 0x7f1c180b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c1cd28640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 0x7f1c1814f0b0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c7a0 tx=0x7f1c1000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10003b70 con 0x7f1c180b41f0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.426+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c10003d10 con 0x7f1c180b41f0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.427+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1000aa90 con 0x7f1c180b41f0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.427+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1815f010 con 0x7f1c180b41f0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.427+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c18160240 con 0x7f1c180b41f0 2024-03-06T07:33:29.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.428+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1804f330 con 0x7f1c180b41f0 2024-03-06T07:33:29.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.428+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c1000ac30 con 0x7f1c180b41f0 2024-03-06T07:33:29.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.429+0000 7f1c157fa640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.429+0000 7f1c17fff640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.429+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1c1008a150 con 0x7f1c180b41f0 2024-03-06T07:33:29.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.429+0000 7f1c17fff640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c0406b4d0 0x7f1c0406d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c034000 tx=0x7f1c0c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.437+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c10059040 con 0x7f1c180b41f0 2024-03-06T07:33:29.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.646+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c1814f5f0 con 0x7f1c180b41f0 2024-03-06T07:33:29.649 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.647+0000 7f1c157fa640 1 -- 172.21.15.122:0/2894988721 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f1c10058940 con 0x7f1c180b41f0 2024-03-06T07:33:29.649 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:29.649 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:29.649 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:29.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c0406b4d0 msgr2=0x7f1c0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c0406b4d0 0x7f1c0406d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c034000 tx=0x7f1c0c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 msgr2=0x7f1c1814f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 0x7f1c1814f0b0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c7a0 tx=0x7f1c1000cc60 comp rx=0 tx=0).stop 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 shutdown_connections 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c1815dfe0 0x7f1c180b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c180d22b0 0x7f1c1815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 --2- 172.21.15.122:0/2894988721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c180b41f0 0x7f1c1814f0b0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 >> 172.21.15.122:0/2894988721 conn(0x7f1c180d3cd0 msgr2=0x7f1c1806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 shutdown_connections 2024-03-06T07:33:29.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:29.649+0000 7f1c1ef8c640 1 -- 172.21.15.122:0/2894988721 wait complete. 2024-03-06T07:33:29.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:29.864 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:30.864 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:30.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.978+0000 7f4f5419a640 1 Processor -- start 2024-03-06T07:33:30.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.979+0000 7f4f5419a640 1 -- start start 2024-03-06T07:33:30.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.979+0000 7f4f5419a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 0x7f4f4c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.979+0000 7f4f5419a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c071f40 con 0x7f4f4c0cd7a0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.979+0000 7f4f5419a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c15b8b0 con 0x7f4f4c15b430 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.979+0000 7f4f5419a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c070a70 con 0x7f4f4c1515e0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.979+0000 7f4f52737640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4f4c0cd7a0 0x7f4f4c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38296/0 (socket says 172.21.15.122:38296) 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f52737640 1 -- 172.21.15.122:0/3222872608 learned_addr learned my addr 172.21.15.122:0/3222872608 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677652311 0 0) 0x7f4f4c071f40 con 0x7f4f4c0cd7a0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f51f36640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 0x7f4f4c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f38003520 con 0x7f4f4c0cd7a0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356095625 0 0) 0x7f4f38003520 con 0x7f4f4c0cd7a0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 msgr2=0x7f4f4c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 0x7f4f4c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 >> v1:172.21.15.169:6789/0 conn(0x7f4f4c15b430 legacy=0x7f4f4c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c070d50 con 0x7f4f4c0cd7a0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4f440019f0 con 0x7f4f4c0cd7a0 2024-03-06T07:33:30.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 >> v1:172.21.15.122:6789/0 conn(0x7f4f4c0cd7a0 legacy=0x7f4f4c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f380039e0 0x7f4f38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 0x7f4f38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f38004d40 0x7f4f380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c070a70 con 0x7f4f38004d40 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3800cb80 con 0x7f4f380039e0 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3800cd80 con 0x7f4f4c1515e0 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f52737640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f38004d40 0x7f4f380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f51f36640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f380039e0 0x7f4f38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.980+0000 7f4f51735640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 0x7f4f38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f52737640 1 -- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 msgr2=0x7f4f38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f52737640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1515e0 0x7f4f38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f52737640 1 -- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f380039e0 msgr2=0x7f4f38003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f52737640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f380039e0 0x7f4f38003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f52737640 1 -- 172.21.15.122:0/3222872608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f38003520 con 0x7f4f38004d40 2024-03-06T07:33:30.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f52737640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f38004d40 0x7f4f380051c0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f4f44002230 tx=0x7f4f38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1afde9c43c057da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400a4a0 con 0x7f4f38004d40 2024-03-06T07:33:30.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f44034650 con 0x7f4f38004d40 2024-03-06T07:33:30.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.981+0000 7f4f50f34640 1 -- 172.21.15.122:0/3222872608 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44034a40 con 0x7f4f38004d40 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f5419a640 1 -- 172.21.15.122:0/3222872608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f38004d40 msgr2=0x7f4f380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f5419a640 1 --2- 172.21.15.122:0/3222872608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f38004d40 0x7f4f380051c0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f4f44002230 tx=0x7f4f38005830 comp rx=0 tx=0).stop 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f52737640 1 -- 172.21.15.122:0/3222872608 reap_dead start 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f5419a640 1 -- 172.21.15.122:0/3222872608 shutdown_connections 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f5419a640 1 -- 172.21.15.122:0/3222872608 >> 172.21.15.122:0/3222872608 conn(0x7f4f4c0cf8a0 msgr2=0x7f4f4c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f5419a640 1 -- 172.21.15.122:0/3222872608 shutdown_connections 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.982+0000 7f4f5419a640 1 -- 172.21.15.122:0/3222872608 wait complete. 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.984+0000 7f4f5419a640 1 Processor -- start 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.984+0000 7f4f5419a640 1 -- start start 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f5419a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1431b0 0x7f4f4c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f5419a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 0x7f4f4c06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.989 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f5419a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f4c15d150 0x7f4f4c06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f5419a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0cce40 con 0x7f4f4c143590 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f5419a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c071310 con 0x7f4f4c15d150 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f5419a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0ccfc0 con 0x7f4f4c1431b0 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51735640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 0x7f4f4c06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51735640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 0x7f4f4c06d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43754/0 (socket says 172.21.15.122:43754) 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51735640 1 -- 172.21.15.122:0/1729441172 learned_addr learned my addr 172.21.15.122:0/1729441172 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51f36640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1431b0 0x7f4f4c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f52737640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f4c15d150 0x7f4f4c06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51735640 1 -- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1431b0 msgr2=0x7f4f4c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51735640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1431b0 0x7f4f4c15bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.985+0000 7f4f51735640 1 -- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f4c15d150 msgr2=0x7f4f4c06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f51735640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f4c15d150 0x7f4f4c06dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f51735640 1 -- 172.21.15.122:0/1729441172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c15c250 con 0x7f4f4c143590 2024-03-06T07:33:30.990 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f52737640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f4c15d150 0x7f4f4c06dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f51735640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 0x7f4f4c06d850 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800f820 tx=0x7f4f4800fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f480050d0 con 0x7f4f4c143590 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f48018070 con 0x7f4f4c143590 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c15c530 con 0x7f4f4c143590 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4800d730 con 0x7f4f4c143590 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.986+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c1f0190 con 0x7f4f4c143590 2024-03-06T07:33:30.991 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.987+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c068840 con 0x7f4f4c143590 2024-03-06T07:33:30.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.987+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f4800d8d0 con 0x7f4f4c143590 2024-03-06T07:33:30.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.988+0000 7f4f427fc640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f3006b4d0 0x7f4f3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.988+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4f48089f30 con 0x7f4f4c143590 2024-03-06T07:33:30.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.988+0000 7f4f51f36640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f3006b4d0 0x7f4f3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.988+0000 7f4f51f36640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f3006b4d0 0x7f4f3006d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009bc0 tx=0x7f4f3c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:30.997+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f48058e20 con 0x7f4f4c143590 2024-03-06T07:33:31.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.212+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f4c15b1e0 con 0x7f4f4c143590 2024-03-06T07:33:31.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.212+0000 7f4f427fc640 1 -- 172.21.15.122:0/1729441172 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f4f4c15b1e0 con 0x7f4f4c143590 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:31.215 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:31.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f3006b4d0 msgr2=0x7f4f3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f3006b4d0 0x7f4f3006d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009bc0 tx=0x7f4f3c009190 comp rx=0 tx=0).stop 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 msgr2=0x7f4f4c06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 0x7f4f4c06d850 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800f820 tx=0x7f4f4800fc20 comp rx=0 tx=0).stop 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 shutdown_connections 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4f3006b4d0 0x7f4f3006d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4f4c15d150 0x7f4f4c06dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f4c143590 0x7f4f4c06d850 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 --2- 172.21.15.122:0/1729441172 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4f4c1431b0 0x7f4f4c15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 >> 172.21.15.122:0/1729441172 conn(0x7f4f4c0cf8a0 msgr2=0x7f4f4c14e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 shutdown_connections 2024-03-06T07:33:31.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:31.214+0000 7f4f5419a640 1 -- 172.21.15.122:0/1729441172 wait complete. 2024-03-06T07:33:31.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:31.433 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:32.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:32.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.534+0000 7f1860c23640 1 Processor -- start 2024-03-06T07:33:32.535 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.534+0000 7f1860c23640 1 -- start start 2024-03-06T07:33:32.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1860c23640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f185c0d2710 0x7f185c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1860c23640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0b3430 con 0x7f185c0d22b0 2024-03-06T07:33:32.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1860c23640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0b3710 con 0x7f185c15dfe0 2024-03-06T07:33:32.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1860c23640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c0b31e0 con 0x7f185c0d2710 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1853fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f185c0d22b0 0x7f185c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38302/0 (socket says 172.21.15.122:38302) 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1853fff640 1 -- 172.21.15.122:0/2661718515 learned_addr learned my addr 172.21.15.122:0/2661718515 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2661718515 >> v2:172.21.15.169:3300/0 conn(0x7f185c0d2710 0x7f185c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908759274 0 0) 0x7f185c0b3430 con 0x7f185c0d22b0 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848003520 con 0x7f185c0d22b0 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 867042829 0 0) 0x7f1848003520 con 0x7f185c0d22b0 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 >> v2:172.21.15.169:3300/0 conn(0x7f185c0d2710 msgr2=0x7f185c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1859d9b640 1 --2- 172.21.15.122:0/2661718515 >> v2:172.21.15.169:3300/0 conn(0x7f185c0d2710 0x7f185c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.535+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 >> v1:172.21.15.169:6789/0 conn(0x7f185c15dfe0 legacy=0x7f185c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0b39f0 con 0x7f185c0d22b0 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f1844001670 con 0x7f185c0d22b0 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 >> v1:172.21.15.122:6789/0 conn(0x7f185c0d22b0 legacy=0x7f185c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.539 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480039e0 0x7f1848003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1848004800 0x7f1848004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18480051c0 0x7f184800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0b31e0 con 0x7f18480051c0 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184800c090 con 0x7f1848004800 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184800c290 con 0x7f18480039e0 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18480051c0 0x7f184800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185a59c640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480039e0 0x7f1848003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f1853fff640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1848004800 0x7f1848004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185ad9d640 1 -- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480039e0 msgr2=0x7f1848003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480039e0 0x7f1848003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185ad9d640 1 -- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1848004800 msgr2=0x7f1848004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1848004800 0x7f1848004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.536+0000 7f185ad9d640 1 -- 172.21.15.122:0/2661718515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848003520 con 0x7f18480051c0 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f185ad9d640 1 -- 172.21.15.122:0/2661718515 reap_dead start 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18480051c0 0x7f184800b9c0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f1854009fc0 tx=0x7f1848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f59956ba795692 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854009c10 con 0x7f18480051c0 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185402cca0 con 0x7f18480051c0 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1859d9b640 1 -- 172.21.15.122:0/2661718515 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854003bd0 con 0x7f18480051c0 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 -- 172.21.15.122:0/2661718515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18480051c0 msgr2=0x7f184800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18480051c0 0x7f184800b9c0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f1854009fc0 tx=0x7f1848005830 comp rx=0 tx=0).stop 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 -- 172.21.15.122:0/2661718515 shutdown_connections 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 --2- 172.21.15.122:0/2661718515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18480051c0 0x7f184800b9c0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 -- 172.21.15.122:0/2661718515 >> 172.21.15.122:0/2661718515 conn(0x7f185c0d3cd0 msgr2=0x7f185c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 -- 172.21.15.122:0/2661718515 shutdown_connections 2024-03-06T07:33:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.537+0000 7f1860c23640 1 -- 172.21.15.122:0/2661718515 wait complete. 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.539+0000 7f1860c23640 1 Processor -- start 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.539+0000 7f1860c23640 1 -- start start 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.539+0000 7f1860c23640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480051c0 0x7f185c15c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.539+0000 7f1860c23640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f185c09e220 0x7f185c0b78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1860c23640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 0x7f185c0b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1860c23640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0d1c70 con 0x7f185c09e6a0 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1860c23640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0b3d60 con 0x7f185c09e220 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1860c23640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0d1df0 con 0x7f18480051c0 2024-03-06T07:33:32.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 0x7f185c0b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 0x7f185c0b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43774/0 (socket says 172.21.15.122:43774) 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 -- 172.21.15.122:0/2799239986 learned_addr learned my addr 172.21.15.122:0/2799239986 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185a59c640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480051c0 0x7f185c15c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 -- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480051c0 msgr2=0x7f185c15c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480051c0 0x7f185c15c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 -- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f185c09e220 msgr2=0x7f185c0b78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f185c09e220 0x7f185c0b78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 -- 172.21.15.122:0/2799239986 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c15c680 con 0x7f185c09e6a0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f185ad9d640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 0x7f185c0b7de0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f1854035d50 tx=0x7f1854009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854042020 con 0x7f185c09e6a0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185402ce50 con 0x7f185c09e6a0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c0b6ae0 con 0x7f185c09e6a0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854003bf0 con 0x7f185c09e6a0 2024-03-06T07:33:32.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.540+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c0b70a0 con 0x7f185c09e6a0 2024-03-06T07:33:32.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.541+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c0d4c10 con 0x7f185c09e6a0 2024-03-06T07:33:32.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.542+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1854045020 con 0x7f185c09e6a0 2024-03-06T07:33:32.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.543+0000 7f1852ffd640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f183c06b500 0x7f183c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.543+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f18540acb70 con 0x7f185c09e6a0 2024-03-06T07:33:32.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.543+0000 7f185a59c640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f183c06b500 0x7f183c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.543+0000 7f185a59c640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f183c06b500 0x7f183c06d9c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f184c000c00 tx=0x7f184c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.547+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f185c0d4c10 con 0x7f185c09e6a0 2024-03-06T07:33:32.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.762+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f185c04f330 con 0x7f185c09e6a0 2024-03-06T07:33:32.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.762+0000 7f1852ffd640 1 -- 172.21.15.122:0/2799239986 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f185404c020 con 0x7f185c09e6a0 2024-03-06T07:33:32.763 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:32.763 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:32.764 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:32.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f183c06b500 msgr2=0x7f183c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f183c06b500 0x7f183c06d9c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f184c000c00 tx=0x7f184c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 msgr2=0x7f185c0b7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 0x7f185c0b7de0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f1854035d50 tx=0x7f1854009750 comp rx=0 tx=0).stop 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 shutdown_connections 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f183c06b500 0x7f183c06d9c0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f185c09e6a0 0x7f185c0b7de0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f185c09e220 0x7f185c0b78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 --2- 172.21.15.122:0/2799239986 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f18480051c0 0x7f185c15c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 >> 172.21.15.122:0/2799239986 conn(0x7f185c0d3cd0 msgr2=0x7f185c0b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 shutdown_connections 2024-03-06T07:33:32.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:32.764+0000 7f1860c23640 1 -- 172.21.15.122:0/2799239986 wait complete. 2024-03-06T07:33:32.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:32.978 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:33.979 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:34.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.081+0000 7fe0ce022640 1 Processor -- start 2024-03-06T07:33:34.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.081+0000 7fe0ce022640 1 -- start start 2024-03-06T07:33:34.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.082+0000 7fe0ce022640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe0c815dfe0 0x7fe0c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.082+0000 7fe0ce022640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c80b3430 con 0x7fe0c80cf830 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.082+0000 7fe0ce022640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c80b3710 con 0x7fe0c80d29c0 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.082+0000 7fe0ce022640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c80b31e0 con 0x7fe0c815dfe0 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c6ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe0c80cf830 0x7fe0c80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38310/0 (socket says 172.21.15.122:38310) 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c6ffd640 1 -- 172.21.15.122:0/690026963 learned_addr learned my addr 172.21.15.122:0/690026963 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933305070 0 0) 0x7fe0c80b3430 con 0x7fe0c80cf830 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b0003520 con 0x7fe0c80cf830 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/690026963 >> v2:172.21.15.169:3300/0 conn(0x7fe0c815dfe0 0x7fe0c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2712344615 0 0) 0x7fe0b0003520 con 0x7fe0c80cf830 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 >> v2:172.21.15.169:3300/0 conn(0x7fe0c815dfe0 msgr2=0x7fe0c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 --2- 172.21.15.122:0/690026963 >> v2:172.21.15.169:3300/0 conn(0x7fe0c815dfe0 0x7fe0c80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 >> v1:172.21.15.169:6789/0 conn(0x7fe0c80d29c0 legacy=0x7fe0c80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80b39f0 con 0x7fe0c80cf830 2024-03-06T07:33:34.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fe0bc001670 con 0x7fe0c80cf830 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 >> v1:172.21.15.122:6789/0 conn(0x7fe0c80cf830 legacy=0x7fe0c80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0b0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0b0004800 0x7fe0b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0b0005210 0x7fe0b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80b31e0 con 0x7fe0b00039b0 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b000c140 con 0x7fe0b0005210 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b000c340 con 0x7fe0b0004800 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0b0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0b0004800 0x7fe0b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.083+0000 7fe0c7fff640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0b0005210 0x7fe0b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c77fe640 1 -- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0b0004800 msgr2=0x7fe0b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0b0004800 0x7fe0b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c77fe640 1 -- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0b0005210 msgr2=0x7fe0b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0b0005210 0x7fe0b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c77fe640 1 -- 172.21.15.122:0/690026963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b0003520 con 0x7fe0b00039b0 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0b0004800 0x7fe0b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c7fff640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0b0005210 0x7fe0b000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c7fff640 1 -- 172.21.15.122:0/690026963 reap_dead start 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0b0003df0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800ba30 tx=0x7fe0b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecc4e0372fbbf59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8036020 con 0x7fe0b00039b0 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b8032640 con 0x7fe0b00039b0 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.084+0000 7fe0c67fc640 1 -- 172.21.15.122:0/690026963 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b800b600 con 0x7fe0b00039b0 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 -- 172.21.15.122:0/690026963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 msgr2=0x7fe0b0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0b0003df0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800ba30 tx=0x7fe0b0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:34.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 -- 172.21.15.122:0/690026963 shutdown_connections 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 --2- 172.21.15.122:0/690026963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0b0003df0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 -- 172.21.15.122:0/690026963 >> 172.21.15.122:0/690026963 conn(0x7fe0c80d3cd0 msgr2=0x7fe0c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 -- 172.21.15.122:0/690026963 shutdown_connections 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.085+0000 7fe0ce022640 1 -- 172.21.15.122:0/690026963 wait complete. 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.086+0000 7fe0ce022640 1 Processor -- start 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.086+0000 7fe0ce022640 1 -- start start 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0ce022640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0c81f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0ce022640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cdb30 0x7fe0c8158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0ce022640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c806c080 0x7fe0c806c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0ce022640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c815e3c0 con 0x7fe0b00039b0 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0ce022640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80b3d60 con 0x7fe0c80cdb30 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0ce022640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c815e540 con 0x7fe0c806c080 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0c81f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0c81f6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:43784/0 (socket says 172.21.15.122:43784) 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 -- 172.21.15.122:0/1885503719 learned_addr learned my addr 172.21.15.122:0/1885503719 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c7fff640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c806c080 0x7fe0c806c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cdb30 0x7fe0c8158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 -- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c806c080 msgr2=0x7fe0c806c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c806c080 0x7fe0c806c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 -- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cdb30 msgr2=0x7fe0c8158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cdb30 0x7fe0c8158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c77fe640 1 -- 172.21.15.122:0/1885503719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80b7de0 con 0x7fe0b00039b0 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.087+0000 7fe0c7fff640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c806c080 0x7fe0c806c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.088+0000 7fe0c77fe640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0c81f6e90 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800bb50 tx=0x7fe0b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.088+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8041070 con 0x7fe0b00039b0 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.088+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b8030a50 con 0x7fe0b00039b0 2024-03-06T07:33:34.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.088+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8036040 con 0x7fe0b00039b0 2024-03-06T07:33:34.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.088+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c81f7470 con 0x7fe0b00039b0 2024-03-06T07:33:34.096 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.088+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c81f7a30 con 0x7fe0b00039b0 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.089+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c804f330 con 0x7fe0b00039b0 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.089+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0b804c060 con 0x7fe0b00039b0 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.090+0000 7fe0affff640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0b406b4d0 0x7fe0b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.090+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0b406b4d0 0x7fe0b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.090+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe0b80abec0 con 0x7fe0b00039b0 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.090+0000 7fe0c6ffd640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0b406b4d0 0x7fe0b406d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc000ff0 tx=0x7fe0bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.097 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.094+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0b807ad20 con 0x7fe0b00039b0 2024-03-06T07:33:34.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.454+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0c80b2da0 con 0x7fe0b00039b0 2024-03-06T07:33:34.457 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.455+0000 7fe0affff640 1 -- 172.21.15.122:0/1885503719 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fe0b807a620 con 0x7fe0b00039b0 2024-03-06T07:33:34.457 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:34.457 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:34.458 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0b406b4d0 msgr2=0x7fe0b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0b406b4d0 0x7fe0b406d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc000ff0 tx=0x7fe0bc009040 comp rx=0 tx=0).stop 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 msgr2=0x7fe0c81f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0c81f6e90 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800bb50 tx=0x7fe0b8004730 comp rx=0 tx=0).stop 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 shutdown_connections 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe0b406b4d0 0x7fe0b406d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe0c806c080 0x7fe0c806c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0c80cdb30 0x7fe0c8158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 --2- 172.21.15.122:0/1885503719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0b00039b0 0x7fe0c81f6e90 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 >> 172.21.15.122:0/1885503719 conn(0x7fe0c80d3cd0 msgr2=0x7fe0c80b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 shutdown_connections 2024-03-06T07:33:34.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:34.457+0000 7fe0ce022640 1 -- 172.21.15.122:0/1885503719 wait complete. 2024-03-06T07:33:34.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:34.672 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:35.673 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:35.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.773+0000 7f81bac1c640 1 Processor -- start 2024-03-06T07:33:35.775 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.773+0000 7f81bac1c640 1 -- start start 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81bac1c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f81b415dfe0 0x7f81b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81bac1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81bac1c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81b40d27f0 0x7f81b40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81bac1c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40d2c50 con 0x7f81b40d27f0 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81bac1c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40b31e0 con 0x7f81b415dfe0 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81bac1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40b3d60 con 0x7f81b40cf830 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81b40d27f0 0x7f81b40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81b40d27f0 0x7f81b40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48036/0 (socket says 172.21.15.122:48036) 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 -- 172.21.15.122:0/647547601 learned_addr learned my addr 172.21.15.122:0/647547601 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b3fff640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b89b8640 1 --2- 172.21.15.122:0/647547601 >> v2:172.21.15.169:3300/0 conn(0x7f81b415dfe0 0x7f81b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 -- 172.21.15.122:0/647547601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 msgr2=0x7f81b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 -- 172.21.15.122:0/647547601 >> v2:172.21.15.169:3300/0 conn(0x7f81b415dfe0 msgr2=0x7f81b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 --2- 172.21.15.122:0/647547601 >> v2:172.21.15.169:3300/0 conn(0x7f81b415dfe0 0x7f81b40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b91b9640 1 -- 172.21.15.122:0/647547601 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40b3430 con 0x7f81b40d27f0 2024-03-06T07:33:35.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.774+0000 7f81b3fff640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:35.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.775+0000 7f81b91b9640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40d0300 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f81b4067990 tx=0x7f81ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c28b8533a19ca3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.775+0000 7f81b37fe640 1 -- 172.21.15.122:0/647547601 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac02ca70 con 0x7f81b40d27f0 2024-03-06T07:33:35.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.775+0000 7f81b37fe640 1 -- 172.21.15.122:0/647547601 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac02cc10 con 0x7f81b40d27f0 2024-03-06T07:33:35.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.775+0000 7f81b37fe640 1 -- 172.21.15.122:0/647547601 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0343f0 con 0x7f81b40d27f0 2024-03-06T07:33:35.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 -- 172.21.15.122:0/647547601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 msgr2=0x7f81b40d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40d0300 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f81b4067990 tx=0x7f81ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.779 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 -- 172.21.15.122:0/647547601 shutdown_connections 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40d0300 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 --2- 172.21.15.122:0/647547601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 --2- 172.21.15.122:0/647547601 >> v2:172.21.15.169:3300/0 conn(0x7f81b415dfe0 0x7f81b40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 -- 172.21.15.122:0/647547601 >> 172.21.15.122:0/647547601 conn(0x7f81b40d3cd0 msgr2=0x7f81b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 -- 172.21.15.122:0/647547601 shutdown_connections 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.776+0000 7f81bac1c640 1 -- 172.21.15.122:0/647547601 wait complete. 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.777+0000 7f81bac1c640 1 Processor -- start 2024-03-06T07:33:35.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.777+0000 7f81bac1c640 1 -- start start 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81bac1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b4157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81bac1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81bac1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81b415dfe0 0x7f81b415f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81bac1c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40d1c70 con 0x7f81b40d27f0 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81bac1c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40b39f0 con 0x7f81b415dfe0 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81bac1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40d1df0 con 0x7f81b40cf830 2024-03-06T07:33:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40b7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48044/0 (socket says 172.21.15.122:48044) 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 -- 172.21.15.122:0/2531915825 learned_addr learned my addr 172.21.15.122:0/2531915825 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b89b8640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b4157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b91b9640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81b415dfe0 0x7f81b415f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 -- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 msgr2=0x7f81b4157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b4157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 -- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81b415dfe0 msgr2=0x7f81b415f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81b415dfe0 0x7f81b415f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 -- 172.21.15.122:0/2531915825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40cc6b0 con 0x7f81b40d27f0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.778+0000 7f81b3fff640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40b7f10 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f81a000e7e0 tx=0x7f81a000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.779+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a000f040 con 0x7f81b40d27f0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.779+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a0004030 con 0x7f81b40d27f0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.779+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b40cc920 con 0x7f81b40d27f0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.779+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a00043e0 con 0x7f81b40d27f0 2024-03-06T07:33:35.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.779+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b40ccee0 con 0x7f81b40d27f0 2024-03-06T07:33:35.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.780+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b404f330 con 0x7f81b40d27f0 2024-03-06T07:33:35.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.780+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81a000c6d0 con 0x7f81b40d27f0 2024-03-06T07:33:35.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.781+0000 7f81b17fa640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f819806b4d0 0x7f819806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.781+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f81a0089da0 con 0x7f81b40d27f0 2024-03-06T07:33:35.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.781+0000 7f81b89b8640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f819806b4d0 0x7f819806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.781+0000 7f81b89b8640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f819806b4d0 0x7f819806d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f81a80079a0 tx=0x7f81a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:35.785+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81a0017020 con 0x7f81b40d27f0 2024-03-06T07:33:36.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.001+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b40cea10 con 0x7f81b40d27f0 2024-03-06T07:33:36.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.001+0000 7f81b17fa640 1 -- 172.21.15.122:0/2531915825 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f81a00585b0 con 0x7f81b40d27f0 2024-03-06T07:33:36.003 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:36.004 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:36.004 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:36.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f819806b4d0 msgr2=0x7f819806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f819806b4d0 0x7f819806d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f81a80079a0 tx=0x7f81a8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:36.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 msgr2=0x7f81b40b7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40b7f10 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f81a000e7e0 tx=0x7f81a000eca0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 shutdown_connections 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f819806b4d0 0x7f819806d990 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81b415dfe0 0x7f81b415f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81b40d27f0 0x7f81b40b7f10 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 --2- 172.21.15.122:0/2531915825 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f81b40cf830 0x7f81b4157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 >> 172.21.15.122:0/2531915825 conn(0x7f81b40d3cd0 msgr2=0x7f81b40b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 shutdown_connections 2024-03-06T07:33:36.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:36.004+0000 7f81bac1c640 1 -- 172.21.15.122:0/2531915825 wait complete. 2024-03-06T07:33:36.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:36.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:37.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:37.321 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.319+0000 7f239d756640 1 Processor -- start 2024-03-06T07:33:37.322 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.319+0000 7f239d756640 1 -- start start 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f239d756640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23980b4170 0x7f23980b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f239d756640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980d25d0 con 0x7f23980d4760 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f239d756640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980d2870 con 0x7f23980d0a80 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f239d756640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980d4bf0 con 0x7f23980b4170 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f2396ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23980b4170 0x7f23980b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f2396ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23980b4170 0x7f23980b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48052/0 (socket says 172.21.15.122:48052) 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.320+0000 7f2396ffd640 1 -- 172.21.15.122:0/2343843292 learned_addr learned my addr 172.21.15.122:0/2343843292 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.321+0000 7f2396ffd640 1 -- 172.21.15.122:0/2343843292 >> v1:172.21.15.169:6789/0 conn(0x7f23980d0a80 legacy=0x7f23980cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.321+0000 7f2396ffd640 1 -- 172.21.15.122:0/2343843292 >> v1:172.21.15.122:6789/0 conn(0x7f23980d4760 legacy=0x7f23980d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.321+0000 7f2396ffd640 1 -- 172.21.15.122:0/2343843292 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980d2b50 con 0x7f23980b4170 2024-03-06T07:33:37.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.321+0000 7f23967fc640 1 -- 172.21.15.122:0/2343843292 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918876289 0 0) 0x7f23980d25d0 con 0x7f23980d4760 2024-03-06T07:33:37.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.321+0000 7f2396ffd640 1 --2- 172.21.15.122:0/2343843292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980b4170 0x7f23980b6a90 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f2388009870 tx=0x7f238802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758acf7282494f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f23967fc640 1 -- 172.21.15.122:0/2343843292 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238802f840 con 0x7f23980b4170 2024-03-06T07:33:37.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f23967fc640 1 -- 172.21.15.122:0/2343843292 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238802f9e0 con 0x7f23980b4170 2024-03-06T07:33:37.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f23967fc640 1 -- 172.21.15.122:0/2343843292 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238802fdb0 con 0x7f23980b4170 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 -- 172.21.15.122:0/2343843292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980b4170 msgr2=0x7f23980b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 --2- 172.21.15.122:0/2343843292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980b4170 0x7f23980b6a90 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f2388009870 tx=0x7f238802ba20 comp rx=0 tx=0).stop 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 -- 172.21.15.122:0/2343843292 shutdown_connections 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 --2- 172.21.15.122:0/2343843292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980b4170 0x7f23980b6a90 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 -- 172.21.15.122:0/2343843292 >> 172.21.15.122:0/2343843292 conn(0x7f23980d3c40 msgr2=0x7f23980d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 -- 172.21.15.122:0/2343843292 shutdown_connections 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.322+0000 7f239d756640 1 -- 172.21.15.122:0/2343843292 wait complete. 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.323+0000 7f239d756640 1 Processor -- start 2024-03-06T07:33:37.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.323+0000 7f239d756640 1 -- start start 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f239d756640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23980b4170 0x7f239815e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f239d756640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 0x7f2398157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f239d756640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f239815be50 0x7f239815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f239d756640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980b4dd0 con 0x7f23980d0a80 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f239d756640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980ce9e0 con 0x7f239815be50 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f239d756640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23980b4f50 con 0x7f23980b4170 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 0x7f2398157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 0x7f2398157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48056/0 (socket says 172.21.15.122:48056) 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 -- 172.21.15.122:0/1600830209 learned_addr learned my addr 172.21.15.122:0/1600830209 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f2396ffd640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23980b4170 0x7f239815e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f23977fe640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f239815be50 0x7f239815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 -- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23980b4170 msgr2=0x7f239815e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23980b4170 0x7f239815e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 -- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f239815be50 msgr2=0x7f239815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.324+0000 7f238ffff640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f239815be50 0x7f239815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f238ffff640 1 -- 172.21.15.122:0/1600830209 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239815d0f0 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f238ffff640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 0x7f2398157d30 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f238000c3c0 tx=0x7f238000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2380017070 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23800035f0 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239815bc40 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800039e0 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.325+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239815f000 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.326+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239804f330 con 0x7f23980d0a80 2024-03-06T07:33:37.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.327+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f238001a070 con 0x7f23980d0a80 2024-03-06T07:33:37.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.327+0000 7f238f7fe640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f237806b4d0 0x7f237806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.327+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f23800894f0 con 0x7f23980d0a80 2024-03-06T07:33:37.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.327+0000 7f2396ffd640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f237806b4d0 0x7f237806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.328+0000 7f2396ffd640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f237806b4d0 0x7f237806d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f2388036000 tx=0x7f2388008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.332+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2380057d00 con 0x7f23980d0a80 2024-03-06T07:33:37.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.541+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23980733b0 con 0x7f23980d0a80 2024-03-06T07:33:37.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.542+0000 7f238f7fe640 1 -- 172.21.15.122:0/1600830209 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f238005bda0 con 0x7f23980d0a80 2024-03-06T07:33:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:37.544 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:37.545 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:37.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f237806b4d0 msgr2=0x7f237806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f237806b4d0 0x7f237806d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f2388036000 tx=0x7f2388008030 comp rx=0 tx=0).stop 2024-03-06T07:33:37.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 msgr2=0x7f2398157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 0x7f2398157d30 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f238000c3c0 tx=0x7f238000c880 comp rx=0 tx=0).stop 2024-03-06T07:33:37.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 shutdown_connections 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f237806b4d0 0x7f237806d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f239815be50 0x7f239815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23980d0a80 0x7f2398157d30 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 --2- 172.21.15.122:0/1600830209 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f23980b4170 0x7f239815e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 >> 172.21.15.122:0/1600830209 conn(0x7f23980d3c40 msgr2=0x7f239806b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 shutdown_connections 2024-03-06T07:33:37.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:37.544+0000 7f239d756640 1 -- 172.21.15.122:0/1600830209 wait complete. 2024-03-06T07:33:37.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:37.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:38.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 Processor -- start 2024-03-06T07:33:38.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 -- start start 2024-03-06T07:33:38.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f99c815bdd0 0x7f99c80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.862 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99c80d16e0 0x7f99c80d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c80cebe0 con 0x7f99c815fa10 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c80ce980 con 0x7f99c80d16e0 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99ce15c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c80ceec0 con 0x7f99c815bdd0 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99c7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99c80d16e0 0x7f99c80d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99c6ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f99c815fa10 0x7f99c80ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52758/0 (socket says 172.21.15.122:52758) 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99c7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f99c80d16e0 0x7f99c80d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48072/0 (socket says 172.21.15.122:48072) 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.858+0000 7f99c6ffd640 1 -- 172.21.15.122:0/1574434106 learned_addr learned my addr 172.21.15.122:0/1574434106 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.859+0000 7f99c77fe640 1 --2- 172.21.15.122:0/1574434106 >> v2:172.21.15.169:3300/0 conn(0x7f99c815bdd0 0x7f99c80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.859+0000 7f99c7fff640 1 -- 172.21.15.122:0/1574434106 >> v2:172.21.15.169:3300/0 conn(0x7f99c815bdd0 msgr2=0x7f99c80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.863 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.859+0000 7f99c7fff640 1 --2- 172.21.15.122:0/1574434106 >> v2:172.21.15.169:3300/0 conn(0x7f99c815bdd0 0x7f99c80c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.859+0000 7f99c7fff640 1 -- 172.21.15.122:0/1574434106 >> v1:172.21.15.122:6789/0 conn(0x7f99c815fa10 legacy=0x7f99c80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.859+0000 7f99c67fc640 1 -- 172.21.15.122:0/1574434106 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637056291 0 0) 0x7f99c80cebe0 con 0x7f99c815fa10 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.859+0000 7f99c7fff640 1 -- 172.21.15.122:0/1574434106 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c80cf1a0 con 0x7f99c80d16e0 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99c7fff640 1 --2- 172.21.15.122:0/1574434106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80d1af0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f99c806e3c0 tx=0x7f99c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6fd4fb7c510584 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99c67fc640 1 -- 172.21.15.122:0/1574434106 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0039070 con 0x7f99c80d16e0 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99c67fc640 1 -- 172.21.15.122:0/1574434106 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c002cbf0 con 0x7f99c80d16e0 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99c67fc640 1 -- 172.21.15.122:0/1574434106 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0034440 con 0x7f99c80d16e0 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 -- 172.21.15.122:0/1574434106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 msgr2=0x7f99c80d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 --2- 172.21.15.122:0/1574434106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80d1af0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f99c806e3c0 tx=0x7f99c002be40 comp rx=0 tx=0).stop 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 -- 172.21.15.122:0/1574434106 shutdown_connections 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 --2- 172.21.15.122:0/1574434106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80d1af0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 --2- 172.21.15.122:0/1574434106 >> v2:172.21.15.169:3300/0 conn(0x7f99c815bdd0 0x7f99c80c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 -- 172.21.15.122:0/1574434106 >> 172.21.15.122:0/1574434106 conn(0x7f99c809a9f0 msgr2=0x7f99c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 -- 172.21.15.122:0/1574434106 shutdown_connections 2024-03-06T07:33:38.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.860+0000 7f99ce15c640 1 -- 172.21.15.122:0/1574434106 wait complete. 2024-03-06T07:33:38.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.861+0000 7f99ce15c640 1 Processor -- start 2024-03-06T07:33:38.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 -- start start 2024-03-06T07:33:38.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99c815bdd0 0x7f99c80c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99c815fa10 0x7f99c80c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.868 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c80cc5d0 con 0x7f99c80d16e0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c80c5b50 con 0x7f99c815bdd0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99ce15c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c815c1b0 con 0x7f99c815fa10 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80cac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48078/0 (socket says 172.21.15.122:48078) 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.862+0000 7f99c77fe640 1 -- 172.21.15.122:0/4175814124 learned_addr learned my addr 172.21.15.122:0/4175814124 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c7fff640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99c815fa10 0x7f99c80c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c6ffd640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99c815bdd0 0x7f99c80c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c77fe640 1 -- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99c815fa10 msgr2=0x7f99c80c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c77fe640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99c815fa10 0x7f99c80c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c77fe640 1 -- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99c815bdd0 msgr2=0x7f99c80c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c77fe640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99c815bdd0 0x7f99c80c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c77fe640 1 -- 172.21.15.122:0/4175814124 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c80cbd00 con 0x7f99c80d16e0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99c77fe640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80cac60 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f99b800d5d0 tx=0x7f99b800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.869 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80125f0 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b8019070 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c80cb1a0 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80129f0 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.863+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c80cb740 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.864+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c804f330 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.865+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99b8012b90 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.865+0000 7f99affff640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99b406b4d0 0x7f99b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.865+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f99b808a600 con 0x7f99c80d16e0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.865+0000 7f99c6ffd640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99b406b4d0 0x7f99b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.870 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.866+0000 7f99c6ffd640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99b406b4d0 0x7f99b406d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f99c80b6f40 tx=0x7f99bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.871 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:38.869+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99b80594f0 con 0x7f99c80d16e0 2024-03-06T07:33:39.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.165+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99c806f200 con 0x7f99c80d16e0 2024-03-06T07:33:39.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.166+0000 7f99affff640 1 -- 172.21.15.122:0/4175814124 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f99b8058df0 con 0x7f99c80d16e0 2024-03-06T07:33:39.168 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:39.168 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:39.168 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99b406b4d0 msgr2=0x7f99b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99b406b4d0 0x7f99b406d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f99c80b6f40 tx=0x7f99bc009040 comp rx=0 tx=0).stop 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 msgr2=0x7f99c80cac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80cac60 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f99b800d5d0 tx=0x7f99b800da90 comp rx=0 tx=0).stop 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 shutdown_connections 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f99b406b4d0 0x7f99b406d990 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f99c815fa10 0x7f99c80c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99c815bdd0 0x7f99c80c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 --2- 172.21.15.122:0/4175814124 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99c80d16e0 0x7f99c80cac60 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.167+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 >> 172.21.15.122:0/4175814124 conn(0x7f99c809a9f0 msgr2=0x7f99c806f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.168+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 shutdown_connections 2024-03-06T07:33:39.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:39.168+0000 7f99ce15c640 1 -- 172.21.15.122:0/4175814124 wait complete. 2024-03-06T07:33:39.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:39.384 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:40.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:40.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.485+0000 7f9e137f6640 1 Processor -- start 2024-03-06T07:33:40.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 -- start start 2024-03-06T07:33:40.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e0c1515e0 0x7f9e0c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9e0c15b430 0x7f9e0c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c0cd7a0 0x7f9e0c0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0c15b8e0 con 0x7f9e0c1515e0 2024-03-06T07:33:40.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0c070a70 con 0x7f9e0c15b430 2024-03-06T07:33:40.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e137f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c070d50 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e11592640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e0c1515e0 0x7f9e0c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e11592640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e0c1515e0 0x7f9e0c0cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48082/0 (socket says 172.21.15.122:48082) 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.486+0000 7f9e11592640 1 -- 172.21.15.122:0/2386395132 learned_addr learned my addr 172.21.15.122:0/2386395132 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11d93640 1 --2- 172.21.15.122:0/2386395132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c0cd7a0 0x7f9e0c0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e10d91640 1 --2- 172.21.15.122:0/2386395132 >> v2:172.21.15.169:3300/0 conn(0x7f9e0c15b430 0x7f9e0c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11592640 1 -- 172.21.15.122:0/2386395132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c0cd7a0 msgr2=0x7f9e0c0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11592640 1 --2- 172.21.15.122:0/2386395132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c0cd7a0 0x7f9e0c0cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11592640 1 -- 172.21.15.122:0/2386395132 >> v2:172.21.15.169:3300/0 conn(0x7f9e0c15b430 msgr2=0x7f9e0c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11592640 1 --2- 172.21.15.122:0/2386395132 >> v2:172.21.15.169:3300/0 conn(0x7f9e0c15b430 0x7f9e0c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11592640 1 -- 172.21.15.122:0/2386395132 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c071030 con 0x7f9e0c1515e0 2024-03-06T07:33:40.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e10d91640 1 --2- 172.21.15.122:0/2386395132 >> v2:172.21.15.169:3300/0 conn(0x7f9e0c15b430 0x7f9e0c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:40.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.487+0000 7f9e11592640 1 --2- 172.21.15.122:0/2386395132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c0cce40 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0cc110 tx=0x7f9dfc02cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f763d5ce19480771 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e0bfff640 1 -- 172.21.15.122:0/2386395132 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc03a070 con 0x7f9e0c1515e0 2024-03-06T07:33:40.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e0bfff640 1 -- 172.21.15.122:0/2386395132 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc02dba0 con 0x7f9e0c1515e0 2024-03-06T07:33:40.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e0bfff640 1 -- 172.21.15.122:0/2386395132 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc00b040 con 0x7f9e0c1515e0 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 -- 172.21.15.122:0/2386395132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c1515e0 msgr2=0x7f9e0c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2386395132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c0cce40 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0cc110 tx=0x7f9dfc02cdf0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 -- 172.21.15.122:0/2386395132 shutdown_connections 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2386395132 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c0cd7a0 0x7f9e0c0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2386395132 >> v2:172.21.15.169:3300/0 conn(0x7f9e0c15b430 0x7f9e0c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2386395132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c0cce40 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 -- 172.21.15.122:0/2386395132 >> 172.21.15.122:0/2386395132 conn(0x7f9e0c0cf8a0 msgr2=0x7f9e0c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 -- 172.21.15.122:0/2386395132 shutdown_connections 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.488+0000 7f9e137f6640 1 -- 172.21.15.122:0/2386395132 wait complete. 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.489+0000 7f9e137f6640 1 Processor -- start 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.489+0000 7f9e137f6640 1 -- start start 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e137f6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 0x7f9e0c143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e137f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e137f6640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c15b430 0x7f9e0c1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e137f6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c071d70 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e137f6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c0ce0f0 con 0x7f9e0c1515e0 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e137f6640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c071ef0 con 0x7f9e0c15b430 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e11592640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 0x7f9e0c143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e11592640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 0x7f9e0c143d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48096/0 (socket says 172.21.15.122:48096) 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e11592640 1 -- 172.21.15.122:0/2717509599 learned_addr learned my addr 172.21.15.122:0/2717509599 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e10d91640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.490+0000 7f9e11d93640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c15b430 0x7f9e0c1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e11592640 1 -- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c15b430 msgr2=0x7f9e0c1508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e11592640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c15b430 0x7f9e0c1508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e11592640 1 -- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e0c1515e0 msgr2=0x7f9e0c150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e11592640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e11592640 1 -- 172.21.15.122:0/2717509599 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c15e9d0 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e11592640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 0x7f9e0c143d10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc038040 tx=0x7f9dfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc043070 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc02d9a0 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c151b50 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc03a030 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.491+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c152110 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.492+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c068840 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.493+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dfc046070 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.493+0000 7f9e09ffb640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9df806b4d0 0x7f9df806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.493+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9dfc0ab420 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.493+0000 7f9e10d91640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9df806b4d0 0x7f9df806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.494+0000 7f9e10d91640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9df806b4d0 0x7f9df806d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9e00003fd0 tx=0x7f9e000039f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.498+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dfc07a310 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.728+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e0c04f330 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.729+0000 7f9e09ffb640 1 -- 172.21.15.122:0/2717509599 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9dfc079c10 con 0x7f9e0c0cd7a0 2024-03-06T07:33:40.731 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:40.731 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:40.731 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.730+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9df806b4d0 msgr2=0x7f9df806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.730+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9df806b4d0 0x7f9df806d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9e00003fd0 tx=0x7f9e000039f0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.730+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 msgr2=0x7f9e0c143d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.730+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 0x7f9e0c143d10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc038040 tx=0x7f9dfc004730 comp rx=0 tx=0).stop 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 shutdown_connections 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9df806b4d0 0x7f9df806d990 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9e0c15b430 0x7f9e0c1508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e0c1515e0 0x7f9e0c150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 --2- 172.21.15.122:0/2717509599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e0c0cd7a0 0x7f9e0c143d10 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 >> 172.21.15.122:0/2717509599 conn(0x7f9e0c0cf8a0 msgr2=0x7f9e0c1593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 shutdown_connections 2024-03-06T07:33:40.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:40.731+0000 7f9e137f6640 1 -- 172.21.15.122:0/2717509599 wait complete. 2024-03-06T07:33:40.946 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:40.946 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:41.947 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:42.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.046+0000 7fea4dde2640 1 Processor -- start 2024-03-06T07:33:42.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.046+0000 7fea4dde2640 1 -- start start 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea4dde2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea48072730 0x7fea4815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea4dde2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4815eb40 con 0x7fea48158600 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea4dde2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4815ee20 con 0x7fea48162030 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea4dde2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea48161ba0 con 0x7fea48072730 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea46ffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea48072730 0x7fea4815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea477fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fea48158600 0x7fea4806d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52760/0 (socket says 172.21.15.122:52760) 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea477fe640 1 -- 172.21.15.122:0/398205330 learned_addr learned my addr 172.21.15.122:0/398205330 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea46ffd640 1 -- 172.21.15.122:0/398205330 >> v1:172.21.15.169:6789/0 conn(0x7fea48162030 legacy=0x7fea4806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea46ffd640 1 -- 172.21.15.122:0/398205330 >> v1:172.21.15.122:6789/0 conn(0x7fea48158600 legacy=0x7fea4806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea46ffd640 1 -- 172.21.15.122:0/398205330 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4815f100 con 0x7fea48072730 2024-03-06T07:33:42.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.047+0000 7fea467fc640 1 -- 172.21.15.122:0/398205330 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824281326 0 0) 0x7fea4815eb40 con 0x7fea48158600 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.048+0000 7fea46ffd640 1 --2- 172.21.15.122:0/398205330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4815e3a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fea300097a0 tx=0x7fea3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b76822934de5153 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.048+0000 7fea467fc640 1 -- 172.21.15.122:0/398205330 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea300315a0 con 0x7fea48072730 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.048+0000 7fea467fc640 1 -- 172.21.15.122:0/398205330 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30037070 con 0x7fea48072730 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.048+0000 7fea467fc640 1 -- 172.21.15.122:0/398205330 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30031960 con 0x7fea48072730 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.048+0000 7fea4dde2640 1 -- 172.21.15.122:0/398205330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 msgr2=0x7fea4815e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.048+0000 7fea4dde2640 1 --2- 172.21.15.122:0/398205330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4815e3a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fea300097a0 tx=0x7fea3002b840 comp rx=0 tx=0).stop 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.049+0000 7fea4dde2640 1 -- 172.21.15.122:0/398205330 shutdown_connections 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.049+0000 7fea4dde2640 1 --2- 172.21.15.122:0/398205330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4815e3a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.049+0000 7fea4dde2640 1 -- 172.21.15.122:0/398205330 >> 172.21.15.122:0/398205330 conn(0x7fea4806c0c0 msgr2=0x7fea4806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.049+0000 7fea4dde2640 1 -- 172.21.15.122:0/398205330 shutdown_connections 2024-03-06T07:33:42.052 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.049+0000 7fea4dde2640 1 -- 172.21.15.122:0/398205330 wait complete. 2024-03-06T07:33:42.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.050+0000 7fea4dde2640 1 Processor -- start 2024-03-06T07:33:42.054 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.050+0000 7fea4dde2640 1 -- start start 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4814fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea48158600 0x7fea4815f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea48162030 0x7fea4814b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4815e8e0 con 0x7fea48072730 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48072be0 con 0x7fea48158600 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4806b6c0 con 0x7fea48162030 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4814fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4814fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48116/0 (socket says 172.21.15.122:48116) 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 -- 172.21.15.122:0/2353545494 learned_addr learned my addr 172.21.15.122:0/2353545494 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 -- 172.21.15.122:0/2353545494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea48162030 msgr2=0x7fea4814b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea48162030 0x7fea4814b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 -- 172.21.15.122:0/2353545494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea48158600 msgr2=0x7fea4815f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea48158600 0x7fea4815f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 -- 172.21.15.122:0/2353545494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea48152c60 con 0x7fea48072730 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea477fe640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4814fb90 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fea3800c4d0 tx=0x7fea3800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38018070 con 0x7fea48072730 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea38007500 con 0x7fea48072730 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38004a20 con 0x7fea48072730 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea48152f40 con 0x7fea48072730 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.051+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea481500d0 con 0x7fea48072730 2024-03-06T07:33:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.052+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4804f330 con 0x7fea48072730 2024-03-06T07:33:42.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.053+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea3801b070 con 0x7fea48072730 2024-03-06T07:33:42.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.053+0000 7fea3ffff640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea2806b4d0 0x7fea2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.053+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fea380894d0 con 0x7fea48072730 2024-03-06T07:33:42.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.053+0000 7fea46ffd640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea2806b4d0 0x7fea2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.054+0000 7fea46ffd640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea2806b4d0 0x7fea2806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fea30036000 tx=0x7fea30030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.062+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea380583c0 con 0x7fea48072730 2024-03-06T07:33:42.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.274+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea480d06f0 con 0x7fea48072730 2024-03-06T07:33:42.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.275+0000 7fea3ffff640 1 -- 172.21.15.122:0/2353545494 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fea38006e20 con 0x7fea48072730 2024-03-06T07:33:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:42.277 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:42.277 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:42.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea2806b4d0 msgr2=0x7fea2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea2806b4d0 0x7fea2806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fea30036000 tx=0x7fea30030000 comp rx=0 tx=0).stop 2024-03-06T07:33:42.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 msgr2=0x7fea4814fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4814fb90 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fea3800c4d0 tx=0x7fea3800c990 comp rx=0 tx=0).stop 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 shutdown_connections 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea2806b4d0 0x7fea2806d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea48162030 0x7fea4814b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea48158600 0x7fea4815f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 --2- 172.21.15.122:0/2353545494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48072730 0x7fea4814fb90 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 >> 172.21.15.122:0/2353545494 conn(0x7fea4806c0c0 msgr2=0x7fea4806d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 shutdown_connections 2024-03-06T07:33:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:42.277+0000 7fea4dde2640 1 -- 172.21.15.122:0/2353545494 wait complete. 2024-03-06T07:33:42.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:42.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:43.494 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:43.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.595+0000 7fbc3d864640 1 Processor -- start 2024-03-06T07:33:43.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.595+0000 7fbc3d864640 1 -- start start 2024-03-06T07:33:43.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc3d864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380d22b0 0x7fbc380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc3d864640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbc380b41f0 0x7fbc380b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc3d864640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380b3430 con 0x7fbc3815dfe0 2024-03-06T07:33:43.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc3d864640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380d2c50 con 0x7fbc380b41f0 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc3d864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b31e0 con 0x7fbc380d22b0 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc377fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbc380b41f0 0x7fbc380b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc36ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbc3815dfe0 0x7fbc380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52776/0 (socket says 172.21.15.122:52776) 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc36ffd640 1 -- 172.21.15.122:0/3330637761 learned_addr learned my addr 172.21.15.122:0/3330637761 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc377fe640 1 -- 172.21.15.122:0/3330637761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380d22b0 msgr2=0x7fbc380d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc2ffff640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380d22b0 0x7fbc380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc377fe640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380d22b0 0x7fbc380d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc367fc640 1 -- 172.21.15.122:0/3330637761 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462303246 0 0) 0x7fbc380b3430 con 0x7fbc3815dfe0 2024-03-06T07:33:43.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc377fe640 1 -- 172.21.15.122:0/3330637761 >> v1:172.21.15.122:6789/0 conn(0x7fbc3815dfe0 legacy=0x7fbc380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc377fe640 1 -- 172.21.15.122:0/3330637761 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380b3710 con 0x7fbc380b41f0 2024-03-06T07:33:43.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.596+0000 7fbc2ffff640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380d22b0 0x7fbc380d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.597+0000 7fbc377fe640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380b41f0 0x7fbc380b4650 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009d90 tx=0x7fbc3002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff44f446132bf92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.597+0000 7fbc367fc640 1 -- 172.21.15.122:0/3330637761 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30039070 con 0x7fbc380b41f0 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.597+0000 7fbc367fc640 1 -- 172.21.15.122:0/3330637761 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3002cc50 con 0x7fbc380b41f0 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.597+0000 7fbc367fc640 1 -- 172.21.15.122:0/3330637761 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30034440 con 0x7fbc380b41f0 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.597+0000 7fbc3d864640 1 -- 172.21.15.122:0/3330637761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380b41f0 msgr2=0x7fbc380b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.597+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380b41f0 0x7fbc380b4650 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009d90 tx=0x7fbc3002bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.598+0000 7fbc3d864640 1 -- 172.21.15.122:0/3330637761 shutdown_connections 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.598+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380b41f0 0x7fbc380b4650 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.598+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3330637761 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380d22b0 0x7fbc380d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.598+0000 7fbc3d864640 1 -- 172.21.15.122:0/3330637761 >> 172.21.15.122:0/3330637761 conn(0x7fbc380d3cd0 msgr2=0x7fbc380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.598+0000 7fbc3d864640 1 -- 172.21.15.122:0/3330637761 shutdown_connections 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.598+0000 7fbc3d864640 1 -- 172.21.15.122:0/3330637761 wait complete. 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 Processor -- start 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 -- start start 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380b41f0 0x7fbc381f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 0x7fbc381f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc3815dfe0 0x7fbc381605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cfa00 con 0x7fbc380d22b0 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b3d60 con 0x7fbc3815dfe0 2024-03-06T07:33:43.604 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.599+0000 7fbc3d864640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380d1c70 con 0x7fbc380b41f0 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 0x7fbc381f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 0x7fbc381f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48146/0 (socket says 172.21.15.122:48146) 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 -- 172.21.15.122:0/3473670296 learned_addr learned my addr 172.21.15.122:0/3473670296 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc377fe640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc3815dfe0 0x7fbc381605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 -- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380b41f0 msgr2=0x7fbc381f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc36ffd640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380b41f0 0x7fbc381f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380b41f0 0x7fbc381f42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 -- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc3815dfe0 msgr2=0x7fbc381605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc3815dfe0 0x7fbc381605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 -- 172.21.15.122:0/3473670296 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc381f4830 con 0x7fbc380d22b0 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc377fe640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc3815dfe0 0x7fbc381605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc36ffd640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380b41f0 0x7fbc381f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:43.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc2ffff640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 0x7fbc381f0cb0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c7f0 tx=0x7fbc2000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20015070 con 0x7fbc380d22b0 2024-03-06T07:33:43.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20007a30 con 0x7fbc380d22b0 2024-03-06T07:33:43.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3815f540 con 0x7fbc380d22b0 2024-03-06T07:33:43.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20007e00 con 0x7fbc380d22b0 2024-03-06T07:33:43.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.600+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3815fb00 con 0x7fbc380d22b0 2024-03-06T07:33:43.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.601+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3804f330 con 0x7fbc380d22b0 2024-03-06T07:33:43.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.602+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc20018070 con 0x7fbc380d22b0 2024-03-06T07:33:43.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.602+0000 7fbc34ff9640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1806b4d0 0x7fbc1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.602+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbc2008a100 con 0x7fbc380d22b0 2024-03-06T07:33:43.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.602+0000 7fbc36ffd640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1806b4d0 0x7fbc1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.607 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.602+0000 7fbc36ffd640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1806b4d0 0x7fbc1806d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbc28000f60 tx=0x7fbc28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.612 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.610+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc381f4830 con 0x7fbc380d22b0 2024-03-06T07:33:43.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.825+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc3806bfe0 con 0x7fbc380d22b0 2024-03-06T07:33:43.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.826+0000 7fbc34ff9640 1 -- 172.21.15.122:0/3473670296 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fbc20058ff0 con 0x7fbc380d22b0 2024-03-06T07:33:43.828 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:43.828 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:43.828 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:43.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1806b4d0 msgr2=0x7fbc1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1806b4d0 0x7fbc1806d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbc28000f60 tx=0x7fbc28009040 comp rx=0 tx=0).stop 2024-03-06T07:33:43.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 msgr2=0x7fbc381f0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 0x7fbc381f0cb0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c7f0 tx=0x7fbc2000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 shutdown_connections 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbc1806b4d0 0x7fbc1806d990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc3815dfe0 0x7fbc381605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc380d22b0 0x7fbc381f0cb0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 --2- 172.21.15.122:0/3473670296 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbc380b41f0 0x7fbc381f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 >> 172.21.15.122:0/3473670296 conn(0x7fbc380d3cd0 msgr2=0x7fbc380d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 shutdown_connections 2024-03-06T07:33:43.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:43.828+0000 7fbc3d864640 1 -- 172.21.15.122:0/3473670296 wait complete. 2024-03-06T07:33:44.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:44.042 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:45.043 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:45.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.144+0000 7f88032af640 1 Processor -- start 2024-03-06T07:33:45.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.144+0000 7f88032af640 1 -- start start 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f88032af640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f87fc1515e0 0x7f87fc0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f88032af640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87fc15b430 0x7f87fc071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f88032af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc0cd7a0 0x7f87fc0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f88032af640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc15b8e0 con 0x7f87fc15b430 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f88032af640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc070a70 con 0x7f87fc1515e0 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f88032af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc070d50 con 0x7f87fc0cd7a0 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87fc15b430 0x7f87fc071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87fc15b430 0x7f87fc071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53160/0 (socket says 172.21.15.122:53160) 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 -- 172.21.15.122:0/1490614332 learned_addr learned my addr 172.21.15.122:0/1490614332 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880104b640 1 --2- 172.21.15.122:0/1490614332 >> v2:172.21.15.169:3300/0 conn(0x7f87fc1515e0 0x7f87fc0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880184c640 1 --2- 172.21.15.122:0/1490614332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc0cd7a0 0x7f87fc0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 -- 172.21.15.122:0/1490614332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc0cd7a0 msgr2=0x7f87fc0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 --2- 172.21.15.122:0/1490614332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc0cd7a0 0x7f87fc0cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 -- 172.21.15.122:0/1490614332 >> v2:172.21.15.169:3300/0 conn(0x7f87fc1515e0 msgr2=0x7f87fc0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 --2- 172.21.15.122:0/1490614332 >> v2:172.21.15.169:3300/0 conn(0x7f87fc1515e0 0x7f87fc0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.145+0000 7f880084a640 1 -- 172.21.15.122:0/1490614332 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc071030 con 0x7f87fc15b430 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.146+0000 7f880084a640 1 --2- 172.21.15.122:0/1490614332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc15b430 0x7f87fc071830 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0d0460 tx=0x7f87f802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fb5be99ab2bed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.146+0000 7f87f3fff640 1 -- 172.21.15.122:0/1490614332 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f803b070 con 0x7f87fc15b430 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.146+0000 7f87f3fff640 1 -- 172.21.15.122:0/1490614332 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f8036440 con 0x7f87fc15b430 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.146+0000 7f87f3fff640 1 -- 172.21.15.122:0/1490614332 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8036810 con 0x7f87fc15b430 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 -- 172.21.15.122:0/1490614332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc15b430 msgr2=0x7f87fc071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 --2- 172.21.15.122:0/1490614332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc15b430 0x7f87fc071830 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0d0460 tx=0x7f87f802e0c0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 -- 172.21.15.122:0/1490614332 shutdown_connections 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 --2- 172.21.15.122:0/1490614332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc15b430 0x7f87fc071830 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 --2- 172.21.15.122:0/1490614332 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc0cd7a0 0x7f87fc0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 --2- 172.21.15.122:0/1490614332 >> v2:172.21.15.169:3300/0 conn(0x7f87fc1515e0 0x7f87fc0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 -- 172.21.15.122:0/1490614332 >> 172.21.15.122:0/1490614332 conn(0x7f87fc0cf8a0 msgr2=0x7f87fc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 -- 172.21.15.122:0/1490614332 shutdown_connections 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.147+0000 7f88032af640 1 -- 172.21.15.122:0/1490614332 wait complete. 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.148+0000 7f88032af640 1 Processor -- start 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.148+0000 7f88032af640 1 -- start start 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f88032af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87fc0cd7a0 0x7f87fc15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f88032af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 0x7f87fc143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f88032af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc15b430 0x7f87fc147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f88032af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc071d70 con 0x7f87fc1515e0 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f88032af640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0ce0f0 con 0x7f87fc0cd7a0 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f88032af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc071ef0 con 0x7f87fc15b430 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 0x7f87fc143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 0x7f87fc143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53164/0 (socket says 172.21.15.122:53164) 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 -- 172.21.15.122:0/2717241978 learned_addr learned my addr 172.21.15.122:0/2717241978 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880184c640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc15b430 0x7f87fc147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880104b640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87fc0cd7a0 0x7f87fc15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 -- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc15b430 msgr2=0x7f87fc147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc15b430 0x7f87fc147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 -- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87fc0cd7a0 msgr2=0x7f87fc15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87fc0cd7a0 0x7f87fc15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 -- 172.21.15.122:0/2717241978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc147dd0 con 0x7f87fc1515e0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f880084a640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 0x7f87fc143690 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f87f802e100 tx=0x7f87f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.149+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8045020 con 0x7f87fc1515e0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.150+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f80346c0 con 0x7f87fc1515e0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.150+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f803b040 con 0x7f87fc1515e0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.150+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc148070 con 0x7f87fc1515e0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.150+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc1451b0 con 0x7f87fc1515e0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.151+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc068840 con 0x7f87fc1515e0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.151+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87f8048020 con 0x7f87fc1515e0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.152+0000 7f87f1ffb640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e006b4d0 0x7f87e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.152+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f87f80aec70 con 0x7f87fc1515e0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.152+0000 7f880104b640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e006b4d0 0x7f87e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.152+0000 7f880104b640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e006b4d0 0x7f87e006d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.156+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87f804f030 con 0x7f87fc1515e0 2024-03-06T07:33:45.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.397+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87fc04f330 con 0x7f87fc1515e0 2024-03-06T07:33:45.400 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.398+0000 7f87f1ffb640 1 -- 172.21.15.122:0/2717241978 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f87f807dad0 con 0x7f87fc1515e0 2024-03-06T07:33:45.400 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:45.400 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:45.401 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:45.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e006b4d0 msgr2=0x7f87e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e006b4d0 0x7f87e006d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec008040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 msgr2=0x7f87fc143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 0x7f87fc143690 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f87f802e100 tx=0x7f87f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.402 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 shutdown_connections 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f87e006b4d0 0x7f87e006d990 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87fc15b430 0x7f87fc147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87fc1515e0 0x7f87fc143690 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 --2- 172.21.15.122:0/2717241978 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87fc0cd7a0 0x7f87fc15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 >> 172.21.15.122:0/2717241978 conn(0x7f87fc0cf8a0 msgr2=0x7f87fc14d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 shutdown_connections 2024-03-06T07:33:45.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:45.400+0000 7f88032af640 1 -- 172.21.15.122:0/2717241978 wait complete. 2024-03-06T07:33:45.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:45.615 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:46.616 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.715+0000 7fa6cc74b640 1 Processor -- start 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.715+0000 7fa6cc74b640 1 -- start start 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6cc74b640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa6c40d22b0 0x7fa6c40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6cc74b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 0x7fa6c40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6cc74b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40b3430 con 0x7fa6c415dfe0 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6cc74b640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40d2c50 con 0x7fa6c40d22b0 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6cc74b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b31e0 con 0x7fa6c40b41f0 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6ca4e7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa6c415dfe0 0x7fa6c40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34060/0 (socket says 172.21.15.122:34060) 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.716+0000 7fa6ca4e7640 1 -- 172.21.15.122:0/3515829993 learned_addr learned my addr 172.21.15.122:0/3515829993 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 0x7fa6c40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c9ce6640 1 --2- 172.21.15.122:0/3515829993 >> v2:172.21.15.169:3300/0 conn(0x7fa6c40d22b0 0x7fa6c40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533946525 0 0) 0x7fa6c40b3430 con 0x7fa6c415dfe0 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0003520 con 0x7fa6c415dfe0 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 762182393 0 0) 0x7fa6b0003520 con 0x7fa6c415dfe0 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 msgr2=0x7fa6c40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 0x7fa6c40b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 >> v2:172.21.15.169:3300/0 conn(0x7fa6c40d22b0 msgr2=0x7fa6c40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 --2- 172.21.15.122:0/3515829993 >> v2:172.21.15.169:3300/0 conn(0x7fa6c40d22b0 0x7fa6c40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40b3710 con 0x7fa6c415dfe0 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa6c0001670 con 0x7fa6c415dfe0 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c9ce6640 1 --2- 172.21.15.122:0/3515829993 >> v2:172.21.15.169:3300/0 conn(0x7fa6c40d22b0 0x7fa6c40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 >> v1:172.21.15.122:6789/0 conn(0x7fa6c415dfe0 legacy=0x7fa6c40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 0x7fa6b00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b0003f30 0x7fa6b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b0004d40 0x7fa6b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b31e0 con 0x7fa6b0004d40 2024-03-06T07:33:46.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000b9c0 con 0x7fa6b0003f30 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000bb40 con 0x7fa6c40b41f0 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b0004d40 0x7fa6b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6ca4e7640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 0x7fa6b00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6c9ce6640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b0003f30 0x7fa6b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 -- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 msgr2=0x7fa6b00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6c40b41f0 0x7fa6b00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 -- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b0003f30 msgr2=0x7fa6b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b0003f30 0x7fa6b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.717+0000 7fa6cace8640 1 -- 172.21.15.122:0/3515829993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0003520 con 0x7fa6b0004d40 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.718+0000 7fa6cace8640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b0004d40 0x7fa6b00051c0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009fc0 tx=0x7fa6b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433fae33068dbf52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.718+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc039070 con 0x7fa6b0004d40 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.718+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc02cc00 con 0x7fa6b0004d40 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.718+0000 7fa6c94e5640 1 -- 172.21.15.122:0/3515829993 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc009c10 con 0x7fa6b0004d40 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cc74b640 1 -- 172.21.15.122:0/3515829993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b0004d40 msgr2=0x7fa6b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/3515829993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6b0004d40 0x7fa6b00051c0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009fc0 tx=0x7fa6b0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:46.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cace8640 1 -- 172.21.15.122:0/3515829993 reap_dead start 2024-03-06T07:33:46.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cc74b640 1 -- 172.21.15.122:0/3515829993 shutdown_connections 2024-03-06T07:33:46.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cc74b640 1 -- 172.21.15.122:0/3515829993 >> 172.21.15.122:0/3515829993 conn(0x7fa6c40d3cd0 msgr2=0x7fa6c40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cc74b640 1 -- 172.21.15.122:0/3515829993 shutdown_connections 2024-03-06T07:33:46.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.719+0000 7fa6cc74b640 1 -- 172.21.15.122:0/3515829993 wait complete. 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.720+0000 7fa6cc74b640 1 Processor -- start 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.720+0000 7fa6cc74b640 1 -- start start 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cc74b640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b0004d40 0x7fa6c41595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cc74b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 0x7fa6c41e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cc74b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c415a3d0 0x7fa6c41e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cc74b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40cfa00 con 0x7fa6c41e6770 2024-03-06T07:33:46.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cc74b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b3d60 con 0x7fa6c415a3d0 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cc74b640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40d1c70 con 0x7fa6b0004d40 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 0x7fa6c41e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 0x7fa6c41e7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53174/0 (socket says 172.21.15.122:53174) 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 -- 172.21.15.122:0/1266903019 learned_addr learned my addr 172.21.15.122:0/1266903019 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6ca4e7640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b0004d40 0x7fa6c41595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6cace8640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c415a3d0 0x7fa6c41e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 -- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b0004d40 msgr2=0x7fa6c41595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b0004d40 0x7fa6c41595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 -- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c415a3d0 msgr2=0x7fa6c41e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c415a3d0 0x7fa6c41e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 -- 172.21.15.122:0/1266903019 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c41e6c00 con 0x7fa6c41e6770 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6ca4e7640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b0004d40 0x7fa6c41595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.721+0000 7fa6c9ce6640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 0x7fa6c41e7ca0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4011cd0 tx=0x7fa6b400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.722+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b400fd20 con 0x7fa6c41e6770 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.722+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b4004030 con 0x7fa6c41e6770 2024-03-06T07:33:46.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.722+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c41e6ee0 con 0x7fa6c41e6770 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.722+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4004420 con 0x7fa6c41e6770 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.722+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c41e7780 con 0x7fa6c41e6770 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.722+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c404f330 con 0x7fa6c41e6770 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.723+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6b4012040 con 0x7fa6c41e6770 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.724+0000 7fa6baffd640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.724+0000 7fa6ca4e7640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.724+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa6b408a7f0 con 0x7fa6c41e6770 2024-03-06T07:33:46.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.724+0000 7fa6ca4e7640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6a806b4d0 0x7fa6a806d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007f10 tx=0x7fa6c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.728+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6b4019070 con 0x7fa6c41e6770 2024-03-06T07:33:46.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.942+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6c40cebb0 con 0x7fa6c41e6770 2024-03-06T07:33:46.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.942+0000 7fa6baffd640 1 -- 172.21.15.122:0/1266903019 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa6b4059650 con 0x7fa6c41e6770 2024-03-06T07:33:46.944 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:46.944 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:46.945 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:46.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6a806b4d0 msgr2=0x7fa6a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6a806b4d0 0x7fa6a806d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007f10 tx=0x7fa6c0009040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 msgr2=0x7fa6c41e7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 0x7fa6c41e7ca0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4011cd0 tx=0x7fa6b400f610 comp rx=0 tx=0).stop 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 shutdown_connections 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c415a3d0 0x7fa6c41e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6c41e6770 0x7fa6c41e7ca0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 --2- 172.21.15.122:0/1266903019 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa6b0004d40 0x7fa6c41595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 >> 172.21.15.122:0/1266903019 conn(0x7fa6c40d3cd0 msgr2=0x7fa6c415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 shutdown_connections 2024-03-06T07:33:46.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:46.944+0000 7fa6cc74b640 1 -- 172.21.15.122:0/1266903019 wait complete. 2024-03-06T07:33:47.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:47.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:48.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:48.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.258+0000 7fbda7bd0640 1 Processor -- start 2024-03-06T07:33:48.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.259+0000 7fbda7bd0640 1 -- start start 2024-03-06T07:33:48.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.259+0000 7fbda7bd0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 0x7fbda00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.259+0000 7fbda7bd0640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbda00d29c0 0x7fbda00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.262 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.259+0000 7fbda7bd0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda00b3430 con 0x7fbda00cf830 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.259+0000 7fbda7bd0640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda00b31e0 con 0x7fbda00d29c0 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.259+0000 7fbda7bd0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00b3d60 con 0x7fbda015dfe0 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda596c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 0x7fbda00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda596c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 0x7fbda00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:39664/0 (socket says 172.21.15.122:39664) 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda616d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fbda00d29c0 0x7fbda00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda516b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fbda00cf830 0x7fbda00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:34778/0 (socket says 172.21.15.122:34778) 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda596c640 1 -- 172.21.15.122:0/3473235715 learned_addr learned my addr 172.21.15.122:0/3473235715 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda616d640 1 -- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 msgr2=0x7fbda00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda616d640 1 --2- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 0x7fbda00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda616d640 1 -- 172.21.15.122:0/3473235715 >> v1:172.21.15.169:6789/0 conn(0x7fbda00cf830 legacy=0x7fbda00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda616d640 1 -- 172.21.15.122:0/3473235715 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00b3710 con 0x7fbda00d29c0 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda496a640 1 -- 172.21.15.122:0/3473235715 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579546880 0 0) 0x7fbda00b3430 con 0x7fbda00cf830 2024-03-06T07:33:48.263 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.260+0000 7fbda596c640 1 --2- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 0x7fbda00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.261+0000 7fbda616d640 1 --2- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda00b41f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009d90 tx=0x7fbd9802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83905270e1e9b883 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.261+0000 7fbda496a640 1 -- 172.21.15.122:0/3473235715 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98039070 con 0x7fbda00d29c0 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.261+0000 7fbda496a640 1 -- 172.21.15.122:0/3473235715 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9802cc50 con 0x7fbda00d29c0 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.261+0000 7fbda496a640 1 -- 172.21.15.122:0/3473235715 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98034440 con 0x7fbda00d29c0 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 -- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 msgr2=0x7fbda00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda00b41f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009d90 tx=0x7fbd9802bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 -- 172.21.15.122:0/3473235715 shutdown_connections 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda00b41f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/3473235715 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda015dfe0 0x7fbda00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 -- 172.21.15.122:0/3473235715 >> 172.21.15.122:0/3473235715 conn(0x7fbda00d3cd0 msgr2=0x7fbda00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 -- 172.21.15.122:0/3473235715 shutdown_connections 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.262+0000 7fbda7bd0640 1 -- 172.21.15.122:0/3473235715 wait complete. 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.263+0000 7fbda7bd0640 1 Processor -- start 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.263+0000 7fbda7bd0640 1 -- start start 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda00cf830 0x7fbda015cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda009e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 0x7fbda00cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00d1df0 con 0x7fbda015dfe0 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00d1c70 con 0x7fbda00d29c0 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda015e3c0 con 0x7fbda00cf830 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 0x7fbda00cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 0x7fbda00cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53186/0 (socket says 172.21.15.122:53186) 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 -- 172.21.15.122:0/2534488246 learned_addr learned my addr 172.21.15.122:0/2534488246 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda596c640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda00cf830 0x7fbda015cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda516b640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda009e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 -- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda00cf830 msgr2=0x7fbda015cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda00cf830 0x7fbda015cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 -- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 msgr2=0x7fbda009e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda009e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 -- 172.21.15.122:0/2534488246 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda015e990 con 0x7fbda015dfe0 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda616d640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 0x7fbda00cce50 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009d90 tx=0x7fbd98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98044020 con 0x7fbda015dfe0 2024-03-06T07:33:48.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9803a040 con 0x7fbda015dfe0 2024-03-06T07:33:48.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda015ec70 con 0x7fbda015dfe0 2024-03-06T07:33:48.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.265+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98039050 con 0x7fbda015dfe0 2024-03-06T07:33:48.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.266+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda015f230 con 0x7fbda015dfe0 2024-03-06T07:33:48.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.266+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda004f330 con 0x7fbda015dfe0 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.267+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd98003990 con 0x7fbda015dfe0 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.267+0000 7fbd967fc640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.267+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbd980ac470 con 0x7fbda015dfe0 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.267+0000 7fbda596c640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.268+0000 7fbda596c640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006ca0 tx=0x7fbd9c009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.272+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd9804e060 con 0x7fbda015dfe0 2024-03-06T07:33:48.487 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.485+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbda006a190 con 0x7fbda015dfe0 2024-03-06T07:33:48.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.486+0000 7fbd967fc640 1 -- 172.21.15.122:0/2534488246 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fbd9807ad10 con 0x7fbda015dfe0 2024-03-06T07:33:48.488 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:48.488 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:48.488 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:48.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 msgr2=0x7fbd8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006ca0 tx=0x7fbd9c009240 comp rx=0 tx=0).stop 2024-03-06T07:33:48.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 msgr2=0x7fbda00cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 0x7fbda00cce50 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009d90 tx=0x7fbd98004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 shutdown_connections 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbd8406b4d0 0x7fbd8406d990 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbda015dfe0 0x7fbda00cce50 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbda00d29c0 0x7fbda009e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 --2- 172.21.15.122:0/2534488246 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbda00cf830 0x7fbda015cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 >> 172.21.15.122:0/2534488246 conn(0x7fbda00d3cd0 msgr2=0x7fbda00b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 shutdown_connections 2024-03-06T07:33:48.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:48.488+0000 7fbda7bd0640 1 -- 172.21.15.122:0/2534488246 wait complete. 2024-03-06T07:33:48.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:48.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:49.703 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:49.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.803+0000 7f234ea64640 1 Processor -- start 2024-03-06T07:33:49.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.803+0000 7f234ea64640 1 -- start start 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.804+0000 7f234ea64640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f234806e760 0x7f234806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.804+0000 7f234ea64640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f23480acf50 0x7f23480ad360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.804+0000 7f234ea64640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23480ad8a0 con 0x7f234815a480 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.804+0000 7f234ea64640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23480adc90 con 0x7f23480acf50 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.804+0000 7f234ea64640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23480cae40 con 0x7f234806e760 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234c800640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f234815a480 0x7f23480ca730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34070/0 (socket says 172.21.15.122:34070) 2024-03-06T07:33:49.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234c800640 1 -- 172.21.15.122:0/396996592 learned_addr learned my addr 172.21.15.122:0/396996592 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234d001640 1 --2- 172.21.15.122:0/396996592 >> v2:172.21.15.122:3300/0 conn(0x7f23480acf50 0x7f23480ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f2347fff640 1 --2- 172.21.15.122:0/396996592 >> v2:172.21.15.169:3300/0 conn(0x7f234806e760 0x7f234806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f23477fe640 1 -- 172.21.15.122:0/396996592 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278119480 0 0) 0x7f233c001110 con 0x7f234815a480 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f23477fe640 1 -- 172.21.15.122:0/396996592 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2338003520 con 0x7f234815a480 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234d001640 1 -- 172.21.15.122:0/396996592 >> v2:172.21.15.169:3300/0 conn(0x7f234806e760 msgr2=0x7f234806ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234d001640 1 --2- 172.21.15.122:0/396996592 >> v2:172.21.15.169:3300/0 conn(0x7f234806e760 0x7f234806ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234d001640 1 -- 172.21.15.122:0/396996592 >> v1:172.21.15.122:6789/0 conn(0x7f234815a480 legacy=0x7f23480ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.808 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.805+0000 7f234d001640 1 -- 172.21.15.122:0/396996592 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480adee0 con 0x7f23480acf50 2024-03-06T07:33:49.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.806+0000 7f234d001640 1 --2- 172.21.15.122:0/396996592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f23480ad360 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f23480afb70 tx=0x7f234002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb68081d083c9084 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.806+0000 7f23477fe640 1 -- 172.21.15.122:0/396996592 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2340039070 con 0x7f23480acf50 2024-03-06T07:33:49.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.806+0000 7f23477fe640 1 -- 172.21.15.122:0/396996592 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234002cc80 con 0x7f23480acf50 2024-03-06T07:33:49.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.806+0000 7f23477fe640 1 -- 172.21.15.122:0/396996592 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2340034660 con 0x7f23480acf50 2024-03-06T07:33:49.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 -- 172.21.15.122:0/396996592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 msgr2=0x7f23480ad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 --2- 172.21.15.122:0/396996592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f23480ad360 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f23480afb70 tx=0x7f234002bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 -- 172.21.15.122:0/396996592 shutdown_connections 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 --2- 172.21.15.122:0/396996592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f23480ad360 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 --2- 172.21.15.122:0/396996592 >> v2:172.21.15.169:3300/0 conn(0x7f234806e760 0x7f234806ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 -- 172.21.15.122:0/396996592 >> 172.21.15.122:0/396996592 conn(0x7f23480d00e0 msgr2=0x7f23480d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 -- 172.21.15.122:0/396996592 shutdown_connections 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.807+0000 7f234ea64640 1 -- 172.21.15.122:0/396996592 wait complete. 2024-03-06T07:33:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.809+0000 7f234ea64640 1 Processor -- start 2024-03-06T07:33:49.811 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.809+0000 7f234ea64640 1 -- start start 2024-03-06T07:33:49.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f234ea64640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f234806e760 0x7f234814cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.813 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f234ea64640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f234806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f234ea64640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f234815a480 0x7f234806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f234ea64640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234806d2e0 con 0x7f23480acf50 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f234ea64640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23480ae4a0 con 0x7f234806e760 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f234ea64640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234815a860 con 0x7f234815a480 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f2347fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f234806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f2347fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f234806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53216/0 (socket says 172.21.15.122:53216) 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.810+0000 7f2347fff640 1 -- 172.21.15.122:0/1538438241 learned_addr learned my addr 172.21.15.122:0/1538438241 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f234c800640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f234806e760 0x7f234814cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f234d001640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f234815a480 0x7f234806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f2347fff640 1 -- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f234815a480 msgr2=0x7f234806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f2347fff640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f234815a480 0x7f234806b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f2347fff640 1 -- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f234806e760 msgr2=0x7f234814cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f2347fff640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f234806e760 0x7f234814cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.814 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f2347fff640 1 -- 172.21.15.122:0/1538438241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480c98f0 con 0x7f23480acf50 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f234c800640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f234806e760 0x7f234814cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f2347fff640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f234806afb0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f233400e9b0 tx=0x7f233400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334017070 con 0x7f23480acf50 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233400c9a0 con 0x7f23480acf50 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23480c9bd0 con 0x7f23480acf50 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233400cd90 con 0x7f23480acf50 2024-03-06T07:33:49.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.811+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23480ca190 con 0x7f23480acf50 2024-03-06T07:33:49.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.812+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234804f330 con 0x7f23480acf50 2024-03-06T07:33:49.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.812+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f233401a070 con 0x7f23480acf50 2024-03-06T07:33:49.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.813+0000 7f23457fa640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f232c06b4d0 0x7f232c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.813+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f233408a600 con 0x7f23480acf50 2024-03-06T07:33:49.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.813+0000 7f234c800640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f232c06b4d0 0x7f232c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.813+0000 7f234c800640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f232c06b4d0 0x7f232c06d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f233c0012b0 tx=0x7f233c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:49.821+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2334059460 con 0x7f23480acf50 2024-03-06T07:33:50.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.120+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23480ce720 con 0x7f23480acf50 2024-03-06T07:33:50.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.121+0000 7f23457fa640 1 -- 172.21.15.122:0/1538438241 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2334058d60 con 0x7f23480acf50 2024-03-06T07:33:50.123 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:50.123 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:50.123 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:50.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f232c06b4d0 msgr2=0x7f232c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f232c06b4d0 0x7f232c06d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f233c0012b0 tx=0x7f233c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.125 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 msgr2=0x7f234806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f234806afb0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f233400e9b0 tx=0x7f233400ee70 comp rx=0 tx=0).stop 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 shutdown_connections 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f232c06b4d0 0x7f232c06d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f234815a480 0x7f234806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23480acf50 0x7f234806afb0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 --2- 172.21.15.122:0/1538438241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f234806e760 0x7f234814cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 >> 172.21.15.122:0/1538438241 conn(0x7f23480d00e0 msgr2=0x7f23480d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 shutdown_connections 2024-03-06T07:33:50.126 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:50.123+0000 7f234ea64640 1 -- 172.21.15.122:0/1538438241 wait complete. 2024-03-06T07:33:50.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:50.342 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:51.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:51.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.442+0000 7f0c08981640 1 Processor -- start 2024-03-06T07:33:51.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.442+0000 7f0c08981640 1 -- start start 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c08981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 0x7f0c040cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c08981640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040ceac0 con 0x7f0c041621c0 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c08981640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040d2dc0 con 0x7f0c041625a0 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c08981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cf690 con 0x7f0c040d4920 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c0259c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0c041621c0 0x7f0c040d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34086/0 (socket says 172.21.15.122:34086) 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c0259c640 1 -- 172.21.15.122:0/109189546 learned_addr learned my addr 172.21.15.122:0/109189546 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0bfbfff640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 0x7f0c040cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752494707 0 0) 0x7f0c040ceac0 con 0x7f0c041621c0 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf0003520 con 0x7f0c041621c0 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880051062 0 0) 0x7f0bf0003520 con 0x7f0c041621c0 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 msgr2=0x7f0c040cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 0x7f0c040cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 >> v1:172.21.15.169:6789/0 conn(0x7f0c041625a0 legacy=0x7f0c0415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.447 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c040ced60 con 0x7f0c041621c0 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f0bf4001670 con 0x7f0c041621c0 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 >> v1:172.21.15.122:6789/0 conn(0x7f0c041621c0 legacy=0x7f0c040d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf00039e0 0x7f0bf0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 0x7f0bf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bf0004d40 0x7f0bf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cf690 con 0x7f0bf0004d40 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf000ba70 con 0x7f0bf00039e0 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.443+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf000bc70 con 0x7f0c040d4920 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c02d9d640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bf0004d40 0x7f0bf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c0259c640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf00039e0 0x7f0bf0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0bfbfff640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 0x7f0bf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c02d9d640 1 -- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 msgr2=0x7f0bf0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c02d9d640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c040d4920 0x7f0bf0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c02d9d640 1 -- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf00039e0 msgr2=0x7f0bf0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.448 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c02d9d640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf00039e0 0x7f0bf0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.444+0000 7f0c02d9d640 1 -- 172.21.15.122:0/109189546 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0003520 con 0x7f0bf0004d40 2024-03-06T07:33:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c02d9d640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bf0004d40 0x7f0bf00051c0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc001e20 tx=0x7f0bf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86185fd93c42daf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc001a70 con 0x7f0bf0004d40 2024-03-06T07:33:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc003b70 con 0x7f0bf0004d40 2024-03-06T07:33:51.449 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c01d9b640 1 -- 172.21.15.122:0/109189546 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0050d0 con 0x7f0bf0004d40 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c08981640 1 -- 172.21.15.122:0/109189546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bf0004d40 msgr2=0x7f0bf00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c08981640 1 --2- 172.21.15.122:0/109189546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bf0004d40 0x7f0bf00051c0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc001e20 tx=0x7f0bf0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c02d9d640 1 -- 172.21.15.122:0/109189546 reap_dead start 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c08981640 1 -- 172.21.15.122:0/109189546 shutdown_connections 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c08981640 1 -- 172.21.15.122:0/109189546 >> 172.21.15.122:0/109189546 conn(0x7f0c040d3d10 msgr2=0x7f0c040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c08981640 1 -- 172.21.15.122:0/109189546 shutdown_connections 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.445+0000 7f0c08981640 1 -- 172.21.15.122:0/109189546 wait complete. 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 Processor -- start 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 -- start start 2024-03-06T07:33:51.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c04151dd0 0x7f0c04154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 0x7f0c041551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c04155700 0x7f0c041f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cfb20 con 0x7f0c041524b0 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cf320 con 0x7f0c04155700 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c08981640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cfca0 con 0x7f0c04151dd0 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0bfbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 0x7f0c041551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0bfbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 0x7f0c041551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53226/0 (socket says 172.21.15.122:53226) 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0bfbfff640 1 -- 172.21.15.122:0/3284804272 learned_addr learned my addr 172.21.15.122:0/3284804272 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.447+0000 7f0c0259c640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c04151dd0 0x7f0c04154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0c02d9d640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c04155700 0x7f0c041f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfbfff640 1 -- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c04151dd0 msgr2=0x7f0c04154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfbfff640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c04151dd0 0x7f0c04154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfbfff640 1 -- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c04155700 msgr2=0x7f0c041f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfbfff640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c04155700 0x7f0c041f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfbfff640 1 -- 172.21.15.122:0/3284804272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c041521b0 con 0x7f0c041524b0 2024-03-06T07:33:51.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0c02d9d640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c04155700 0x7f0c041f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfbfff640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 0x7f0c041551c0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0119d0 tx=0x7f0bec011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00faf0 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec00fc90 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c04153ec0 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec014440 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.448+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04154480 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.449+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0404f330 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.450+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bec012040 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.450+0000 7f0bfaffd640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.450+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0bec08a6a0 con 0x7f0c041524b0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.450+0000 7f0c0259c640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.454 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.450+0000 7f0c0259c640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f0c040ca470 tx=0x7f0bf4000f80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.456 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.454+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bec019030 con 0x7f0c041524b0 2024-03-06T07:33:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.662+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c04073380 con 0x7f0c041524b0 2024-03-06T07:33:51.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.662+0000 7f0bfaffd640 1 -- 172.21.15.122:0/3284804272 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0bec059590 con 0x7f0c041524b0 2024-03-06T07:33:51.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:51.665 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:51.665 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0be406b4d0 msgr2=0x7f0be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f0c040ca470 tx=0x7f0bf4000f80 comp rx=0 tx=0).stop 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 msgr2=0x7f0c041551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 0x7f0c041551c0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0119d0 tx=0x7f0bec011e90 comp rx=0 tx=0).stop 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 shutdown_connections 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c04155700 0x7f0c041f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c041524b0 0x7f0c041551c0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 --2- 172.21.15.122:0/3284804272 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c04151dd0 0x7f0c04154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 >> 172.21.15.122:0/3284804272 conn(0x7f0c040d3d10 msgr2=0x7f0c040b7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 shutdown_connections 2024-03-06T07:33:51.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:51.664+0000 7f0c08981640 1 -- 172.21.15.122:0/3284804272 wait complete. 2024-03-06T07:33:51.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:51.879 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:52.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:53.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:52.999+0000 7fa5782fa640 1 Processor -- start 2024-03-06T07:33:53.001 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:52.999+0000 7fa5782fa640 1 -- start start 2024-03-06T07:33:53.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa5782fa640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fa57015bf30 0x7fa57015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa5782fa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57015f580 con 0x7fa57013dc10 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa5782fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57015b450 con 0x7fa5700b3060 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa5782fa640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5700cf3f0 con 0x7fa57015bf30 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa576096640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa57013dc10 0x7fa5700b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34096/0 (socket says 172.21.15.122:34096) 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa576096640 1 -- 172.21.15.122:0/2226122601 learned_addr learned my addr 172.21.15.122:0/2226122601 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343499116 0 0) 0x7fa57015f580 con 0x7fa57013dc10 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c003520 con 0x7fa57013dc10 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa576897640 1 --2- 172.21.15.122:0/2226122601 >> v2:172.21.15.169:3300/0 conn(0x7fa57015bf30 0x7fa57015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1535031489 0 0) 0x7fa55c003520 con 0x7fa57013dc10 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 >> v2:172.21.15.169:3300/0 conn(0x7fa57015bf30 msgr2=0x7fa57015ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 --2- 172.21.15.122:0/2226122601 >> v2:172.21.15.169:3300/0 conn(0x7fa57015bf30 0x7fa57015ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 >> v1:172.21.15.169:6789/0 conn(0x7fa5700b3060 legacy=0x7fa57015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57015b730 con 0x7fa57013dc10 2024-03-06T07:33:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fa560001670 con 0x7fa57013dc10 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 >> v1:172.21.15.122:6789/0 conn(0x7fa57013dc10 legacy=0x7fa5700b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c0039b0 0x7fa55c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa55c004310 0x7fa55c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa55c004fe0 0x7fa55c00ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700cf3f0 con 0x7fa55c004fe0 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c010510 con 0x7fa55c004310 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c010710 con 0x7fa55c0039b0 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.000+0000 7fa576897640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa55c004fe0 0x7fa55c00ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa575895640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa55c004310 0x7fa55c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576096640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c0039b0 0x7fa55c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 -- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c0039b0 msgr2=0x7fa55c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c0039b0 0x7fa55c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 -- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa55c004310 msgr2=0x7fa55c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa55c004310 0x7fa55c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 -- 172.21.15.122:0/2226122601 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c003520 con 0x7fa55c004fe0 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 -- 172.21.15.122:0/2226122601 reap_dead start 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.001+0000 7fa576897640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa55c004fe0 0x7fa55c00ffd0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fa568009fc0 tx=0x7fa55c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e33f1b74fdb1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568009c10 con 0x7fa55c004fe0 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56802cc60 con 0x7fa55c004fe0 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa575094640 1 -- 172.21.15.122:0/2226122601 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568003bd0 con 0x7fa55c004fe0 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 -- 172.21.15.122:0/2226122601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa55c004fe0 msgr2=0x7fa55c00ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa55c004fe0 0x7fa55c00ffd0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fa568009fc0 tx=0x7fa55c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 -- 172.21.15.122:0/2226122601 shutdown_connections 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 --2- 172.21.15.122:0/2226122601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa55c004fe0 0x7fa55c00ffd0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 -- 172.21.15.122:0/2226122601 >> 172.21.15.122:0/2226122601 conn(0x7fa5700cf8a0 msgr2=0x7fa5700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 -- 172.21.15.122:0/2226122601 shutdown_connections 2024-03-06T07:33:53.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.002+0000 7fa5782fa640 1 -- 172.21.15.122:0/2226122601 wait complete. 2024-03-06T07:33:53.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.003+0000 7fa5782fa640 1 Processor -- start 2024-03-06T07:33:53.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.003+0000 7fa5782fa640 1 -- start start 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa5782fa640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c004fe0 0x7fa57015cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa5782fa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 0x7fa57013f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa5782fa640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa57015d770 0x7fa57013cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa5782fa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700b2950 con 0x7fa57013e200 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa5782fa640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57015b210 con 0x7fa57015d770 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa5782fa640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700b2ad0 con 0x7fa55c004fe0 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 0x7fa57013f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 0x7fa57013f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:53232/0 (socket says 172.21.15.122:53232) 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 -- 172.21.15.122:0/1804252554 learned_addr learned my addr 172.21.15.122:0/1804252554 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa576096640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c004fe0 0x7fa57015cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa576897640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa57015d770 0x7fa57013cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 -- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c004fe0 msgr2=0x7fa57015cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c004fe0 0x7fa57015cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 -- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa57015d770 msgr2=0x7fa57013cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa57015d770 0x7fa57013cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa575895640 1 -- 172.21.15.122:0/1804252554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57015e370 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.004+0000 7fa576897640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa57015d770 0x7fa57013cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.005+0000 7fa575895640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 0x7fa57013f8f0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00f5f0 tx=0x7fa56c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.005+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0052e0 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.005+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c018070 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.005+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c003810 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.005+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57013e850 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.005+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57013ee10 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.006+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57004f330 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.006+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa56c0039b0 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.007+0000 7fa566ffd640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa55406b500 0x7fa55406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.007+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa56c089ed0 con 0x7fa57013e200 2024-03-06T07:33:53.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.007+0000 7fa576096640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa55406b500 0x7fa55406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.007+0000 7fa576096640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa55406b500 0x7fa55406d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa560001470 tx=0x7fa5600078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.011+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa56c058dc0 con 0x7fa57013e200 2024-03-06T07:33:53.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.333+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa57006eb00 con 0x7fa57013e200 2024-03-06T07:33:53.335 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.333+0000 7fa566ffd640 1 -- 172.21.15.122:0/1804252554 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa56c0586c0 con 0x7fa57013e200 2024-03-06T07:33:53.335 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:53.335 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:53.336 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:53.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa55406b500 msgr2=0x7fa55406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa55406b500 0x7fa55406d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa560001470 tx=0x7fa5600078f0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 msgr2=0x7fa57013f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 0x7fa57013f8f0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00f5f0 tx=0x7fa56c00fab0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 shutdown_connections 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa55406b500 0x7fa55406d9c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa57015d770 0x7fa57013cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa57013e200 0x7fa57013f8f0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 --2- 172.21.15.122:0/1804252554 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa55c004fe0 0x7fa57015cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 >> 172.21.15.122:0/1804252554 conn(0x7fa5700cf8a0 msgr2=0x7fa5700cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 shutdown_connections 2024-03-06T07:33:53.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:53.335+0000 7fa5782fa640 1 -- 172.21.15.122:0/1804252554 wait complete. 2024-03-06T07:33:53.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:53.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:54.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.650+0000 7f0058e37640 1 Processor -- start 2024-03-06T07:33:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.650+0000 7f0058e37640 1 -- start start 2024-03-06T07:33:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f0058e37640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f005415dfe0 0x7f00540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f0058e37640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00540cf830 0x7f00540d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f0058e37640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00540b3430 con 0x7f00540d27f0 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f0058e37640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00540b31e0 con 0x7f00540cf830 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f0058e37640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00540b3d60 con 0x7f005415dfe0 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00540cf830 0x7f00540d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f00540cf830 0x7f00540d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60568/0 (socket says 172.21.15.122:60568) 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 -- 172.21.15.122:0/441771716 learned_addr learned my addr 172.21.15.122:0/441771716 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f0052d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f00540d27f0 0x7f00540d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35606/0 (socket says 172.21.15.122:35606) 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 -- 172.21.15.122:0/441771716 >> v2:172.21.15.169:3300/0 conn(0x7f005415dfe0 msgr2=0x7f00540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 --2- 172.21.15.122:0/441771716 >> v2:172.21.15.169:3300/0 conn(0x7f005415dfe0 0x7f00540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 -- 172.21.15.122:0/441771716 >> v1:172.21.15.122:6789/0 conn(0x7f00540d27f0 legacy=0x7f00540d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.652+0000 7f004bfff640 1 -- 172.21.15.122:0/441771716 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540b3710 con 0x7f00540cf830 2024-03-06T07:33:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f004bfff640 1 --2- 172.21.15.122:0/441771716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f00540d22b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f003c009870 tx=0x7f003c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1d1064ddcde18b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0051d9b640 1 -- 172.21.15.122:0/441771716 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c034070 con 0x7f00540cf830 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0051d9b640 1 -- 172.21.15.122:0/441771716 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c030860 con 0x7f00540cf830 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0051d9b640 1 -- 172.21.15.122:0/441771716 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c030c30 con 0x7f00540cf830 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0058e37640 1 -- 172.21.15.122:0/441771716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 msgr2=0x7f00540d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0058e37640 1 --2- 172.21.15.122:0/441771716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f00540d22b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f003c009870 tx=0x7f003c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0058e37640 1 -- 172.21.15.122:0/441771716 shutdown_connections 2024-03-06T07:33:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0058e37640 1 --2- 172.21.15.122:0/441771716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f00540d22b0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0058e37640 1 --2- 172.21.15.122:0/441771716 >> v2:172.21.15.169:3300/0 conn(0x7f005415dfe0 0x7f00540cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.653+0000 7f0058e37640 1 -- 172.21.15.122:0/441771716 >> 172.21.15.122:0/441771716 conn(0x7f00540d3cd0 msgr2=0x7f00540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.654+0000 7f0058e37640 1 -- 172.21.15.122:0/441771716 shutdown_connections 2024-03-06T07:33:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.654+0000 7f0058e37640 1 -- 172.21.15.122:0/441771716 wait complete. 2024-03-06T07:33:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.655+0000 7f0058e37640 1 Processor -- start 2024-03-06T07:33:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.655+0000 7f0058e37640 1 -- start start 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.655+0000 7f0058e37640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f0054156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.655+0000 7f0058e37640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00540d27f0 0x7f0054147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f0058e37640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f005415dfe0 0x7f005415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f0058e37640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00540d1df0 con 0x7f00540cf830 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f0058e37640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00540d1c70 con 0x7f00540d27f0 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f0058e37640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005415e3c0 con 0x7f005415dfe0 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f0054156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f0054156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60574/0 (socket says 172.21.15.122:60574) 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 -- 172.21.15.122:0/2665912127 learned_addr learned my addr 172.21.15.122:0/2665912127 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 -- 172.21.15.122:0/2665912127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f005415dfe0 msgr2=0x7f005415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f005415dfe0 0x7f005415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 -- 172.21.15.122:0/2665912127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00540d27f0 msgr2=0x7f0054147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00540d27f0 0x7f0054147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 -- 172.21.15.122:0/2665912127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005415aeb0 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f005259c640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f0054156ff0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f004400b4d0 tx=0x7f004400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0044014070 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0044010870 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005415f870 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0044010c60 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.656+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005415fe30 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.657+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005404f330 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.658+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0044017070 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.658+0000 7f004affd640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f003406b4d0 0x7f003406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.658+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0044089ff0 con 0x7f00540cf830 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.658+0000 7f004bfff640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f003406b4d0 0x7f003406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.659+0000 7f004bfff640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f003406b4d0 0x7f003406d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f003c036000 tx=0x7f003c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.663+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0044058e60 con 0x7f00540cf830 2024-03-06T07:33:54.879 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.877+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f005415d600 con 0x7f00540cf830 2024-03-06T07:33:54.880 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.878+0000 7f004affd640 1 -- 172.21.15.122:0/2665912127 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0044058760 con 0x7f00540cf830 2024-03-06T07:33:54.880 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:54.880 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:54.880 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f003406b4d0 msgr2=0x7f003406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f003406b4d0 0x7f003406d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f003c036000 tx=0x7f003c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 msgr2=0x7f0054156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f0054156ff0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f004400b4d0 tx=0x7f004400b990 comp rx=0 tx=0).stop 2024-03-06T07:33:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 shutdown_connections 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f003406b4d0 0x7f003406d990 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f005415dfe0 0x7f005415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00540d27f0 0x7f0054147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 --2- 172.21.15.122:0/2665912127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00540cf830 0x7f0054156ff0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 >> 172.21.15.122:0/2665912127 conn(0x7f00540d3cd0 msgr2=0x7f005406bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 shutdown_connections 2024-03-06T07:33:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:54.880+0000 7f0058e37640 1 -- 172.21.15.122:0/2665912127 wait complete. 2024-03-06T07:33:55.094 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:55.094 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:56.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:56.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.194+0000 7ff6df6f4640 1 Processor -- start 2024-03-06T07:33:56.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.194+0000 7ff6df6f4640 1 -- start start 2024-03-06T07:33:56.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.195+0000 7ff6df6f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 0x7ff6d80d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.195+0000 7ff6df6f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80cebe0 con 0x7ff6d815fa10 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.195+0000 7ff6df6f4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80ceec0 con 0x7ff6d815bdd0 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.195+0000 7ff6df6f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80ce980 con 0x7ff6d80d16e0 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.195+0000 7ff6dcc8f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff6d815fa10 0x7ff6d80ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35622/0 (socket says 172.21.15.122:35622) 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.195+0000 7ff6dcc8f640 1 -- 172.21.15.122:0/2033919479 learned_addr learned my addr 172.21.15.122:0/2033919479 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6ddc91640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 0x7ff6d80d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947660756 0 0) 0x7ff6d80cebe0 con 0x7ff6d815fa10 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c4003520 con 0x7ff6d815fa10 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098615569 0 0) 0x7ff6c4003520 con 0x7ff6d815fa10 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 msgr2=0x7ff6d80d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 0x7ff6d80d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 >> v1:172.21.15.169:6789/0 conn(0x7ff6d815bdd0 legacy=0x7ff6d80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d80cf1a0 con 0x7ff6d815fa10 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff6c8001670 con 0x7ff6d815fa10 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 >> v1:172.21.15.122:6789/0 conn(0x7ff6d815fa10 legacy=0x7ff6d80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 0x7ff6c40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6c4004d40 0x7ff6c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80ce980 con 0x7ff6c4003f60 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c400bb50 con 0x7ff6c4004d40 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c400bd50 con 0x7ff6d80d16e0 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dd490640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 0x7ff6c40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6ddc91640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6c4004d40 0x7ff6c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dcc8f640 1 -- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 msgr2=0x7ff6c40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d80d16e0 0x7ff6c40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dcc8f640 1 -- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6c4004d40 msgr2=0x7ff6c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6c4004d40 0x7ff6c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.196+0000 7ff6dcc8f640 1 -- 172.21.15.122:0/2033919479 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c4003520 con 0x7ff6c4003f60 2024-03-06T07:33:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.197+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6c4004800 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80d28c0 tx=0x7ff6c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be111a9f319420c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.197+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8007d30 con 0x7ff6c4003f60 2024-03-06T07:33:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.197+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c8030070 con 0x7ff6c4003f60 2024-03-06T07:33:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.197+0000 7ff6cffff640 1 -- 172.21.15.122:0/2033919479 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8005d30 con 0x7ff6c4003f60 2024-03-06T07:33:56.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6df6f4640 1 -- 172.21.15.122:0/2033919479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 msgr2=0x7ff6c4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/2033919479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6c4004800 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80d28c0 tx=0x7ff6c4005830 comp rx=0 tx=0).stop 2024-03-06T07:33:56.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6ddc91640 1 -- 172.21.15.122:0/2033919479 reap_dead start 2024-03-06T07:33:56.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6df6f4640 1 -- 172.21.15.122:0/2033919479 shutdown_connections 2024-03-06T07:33:56.205 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6df6f4640 1 -- 172.21.15.122:0/2033919479 >> 172.21.15.122:0/2033919479 conn(0x7ff6d809a9f0 msgr2=0x7ff6d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6df6f4640 1 -- 172.21.15.122:0/2033919479 shutdown_connections 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.198+0000 7ff6df6f4640 1 -- 172.21.15.122:0/2033919479 wait complete. 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.199+0000 7ff6df6f4640 1 Processor -- start 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.199+0000 7ff6df6f4640 1 -- start start 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6df6f4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6d8152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6df6f4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d815ef80 0x7ff6d80c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6df6f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6d815ca20 0x7ff6d815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6df6f4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d815c1b0 con 0x7ff6c4003f60 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6df6f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80cc5d0 con 0x7ff6d815ca20 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6df6f4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80d1b10 con 0x7ff6d815ef80 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6d8152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6d8152f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60590/0 (socket says 172.21.15.122:60590) 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 -- 172.21.15.122:0/4111291488 learned_addr learned my addr 172.21.15.122:0/4111291488 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6ddc91640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6d815ca20 0x7ff6d815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d815ef80 0x7ff6d80c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 -- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d815ef80 msgr2=0x7ff6d80c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d815ef80 0x7ff6d80c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 -- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6d815ca20 msgr2=0x7ff6d815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6d815ca20 0x7ff6d815cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dd490640 1 -- 172.21.15.122:0/4111291488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d81534b0 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.200+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d815ef80 0x7ff6d80c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.201+0000 7ff6dd490640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6d8152f70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400f820 tx=0x7ff6d400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.201+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40050d0 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.201+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d4018070 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.201+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d80cb560 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.201+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4005490 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.201+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d80ca7a0 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.202+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d804f330 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.202+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6d400d6d0 con 0x7ff6c4003f60 2024-03-06T07:33:56.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.202+0000 7ff6cdffb640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff6bc06b4d0 0x7ff6bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.203+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff6d408a000 con 0x7ff6c4003f60 2024-03-06T07:33:56.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.203+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff6bc06b4d0 0x7ff6bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.203+0000 7ff6dcc8f640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff6bc06b4d0 0x7ff6bc06d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80d28c0 tx=0x7ff6c8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.207+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6d81534b0 con 0x7ff6c4003f60 2024-03-06T07:33:56.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:56.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.423+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6d80b7a40 con 0x7ff6c4003f60 2024-03-06T07:33:56.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.424+0000 7ff6cdffb640 1 -- 172.21.15.122:0/4111291488 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7ff6d4058e60 con 0x7ff6c4003f60 2024-03-06T07:33:56.426 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:56.426 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:56.426 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.425+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff6bc06b4d0 msgr2=0x7ff6bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.425+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff6bc06b4d0 0x7ff6bc06d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80d28c0 tx=0x7ff6c8009030 comp rx=0 tx=0).stop 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.425+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 msgr2=0x7ff6d8152f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6d8152f70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400f820 tx=0x7ff6d400fce0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 shutdown_connections 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff6bc06b4d0 0x7ff6bc06d990 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6d815ca20 0x7ff6d815cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff6d815ef80 0x7ff6d80c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 --2- 172.21.15.122:0/4111291488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6c4003f60 0x7ff6d8152f70 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 >> 172.21.15.122:0/4111291488 conn(0x7ff6d809a9f0 msgr2=0x7ff6d806f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 shutdown_connections 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:56.426+0000 7ff6df6f4640 1 -- 172.21.15.122:0/4111291488 wait complete. 2024-03-06T07:33:56.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:56.643 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:57.644 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:57.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.743+0000 7fcb35581640 1 Processor -- start 2024-03-06T07:33:57.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.743+0000 7fcb35581640 1 -- start start 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb35581640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 0x7fcb300b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb35581640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300d25d0 con 0x7fcb300d4760 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb35581640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300d2870 con 0x7fcb300d0a80 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb35581640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300d4bf0 con 0x7fcb300b4170 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb2e7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcb300d4760 0x7fcb300d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35636/0 (socket says 172.21.15.122:35636) 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb2e7fc640 1 -- 172.21.15.122:0/1934147525 learned_addr learned my addr 172.21.15.122:0/1934147525 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23946551 0 0) 0x7fcb300d25d0 con 0x7fcb300d4760 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb18003520 con 0x7fcb300d4760 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.744+0000 7fcb2effd640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 0x7fcb300b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039441782 0 0) 0x7fcb18003520 con 0x7fcb300d4760 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 msgr2=0x7fcb300b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 0x7fcb300b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 >> v1:172.21.15.169:6789/0 conn(0x7fcb300d0a80 legacy=0x7fcb300cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300d2b50 con 0x7fcb300d4760 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fcb20001670 con 0x7fcb300d4760 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 >> v1:172.21.15.122:6789/0 conn(0x7fcb300d4760 legacy=0x7fcb300d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb180039e0 0x7fcb18003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb18004800 0x7fcb18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 0x7fcb18005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300d4bf0 con 0x7fcb18004800 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1800baf0 con 0x7fcb180039e0 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1800bcf0 con 0x7fcb300b4170 2024-03-06T07:33:57.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2e7fc640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb18004800 0x7fcb18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2effd640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb180039e0 0x7fcb18003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2f7fe640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 0x7fcb18005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2e7fc640 1 -- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 msgr2=0x7fcb18005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2e7fc640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b4170 0x7fcb18005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2e7fc640 1 -- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb180039e0 msgr2=0x7fcb18003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2e7fc640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb180039e0 0x7fcb18003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.745+0000 7fcb2e7fc640 1 -- 172.21.15.122:0/1934147525 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb18003520 con 0x7fcb18004800 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb2e7fc640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb18004800 0x7fcb18004c80 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000c210 tx=0x7fcb18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fd0d5cd0ffec2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20007d30 con 0x7fcb18004800 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20030070 con 0x7fcb18004800 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb2dffb640 1 -- 172.21.15.122:0/1934147525 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2002cd60 con 0x7fcb18004800 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb35581640 1 -- 172.21.15.122:0/1934147525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb18004800 msgr2=0x7fcb18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb35581640 1 --2- 172.21.15.122:0/1934147525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb18004800 0x7fcb18004c80 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000c210 tx=0x7fcb18005830 comp rx=0 tx=0).stop 2024-03-06T07:33:57.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.746+0000 7fcb2f7fe640 1 -- 172.21.15.122:0/1934147525 reap_dead start 2024-03-06T07:33:57.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.747+0000 7fcb35581640 1 -- 172.21.15.122:0/1934147525 shutdown_connections 2024-03-06T07:33:57.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.747+0000 7fcb35581640 1 -- 172.21.15.122:0/1934147525 >> 172.21.15.122:0/1934147525 conn(0x7fcb300d3c40 msgr2=0x7fcb300d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.747+0000 7fcb35581640 1 -- 172.21.15.122:0/1934147525 shutdown_connections 2024-03-06T07:33:57.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.747+0000 7fcb35581640 1 -- 172.21.15.122:0/1934147525 wait complete. 2024-03-06T07:33:57.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.748+0000 7fcb35581640 1 Processor -- start 2024-03-06T07:33:57.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.748+0000 7fcb35581640 1 -- start start 2024-03-06T07:33:57.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb18004800 0x7fcb300b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b9850 0x7fcb301f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 0x7fcb3015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300b4dd0 con 0x7fcb301f7680 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300ce9e0 con 0x7fcb18004800 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300b4f50 con 0x7fcb300b9850 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 0x7fcb3015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 0x7fcb3015a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60606/0 (socket says 172.21.15.122:60606) 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 -- 172.21.15.122:0/2720258667 learned_addr learned my addr 172.21.15.122:0/2720258667 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 -- 172.21.15.122:0/2720258667 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b9850 msgr2=0x7fcb301f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b9850 0x7fcb301f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 -- 172.21.15.122:0/2720258667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb18004800 msgr2=0x7fcb300b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb18004800 0x7fcb300b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 -- 172.21.15.122:0/2720258667 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3015adf0 con 0x7fcb301f7680 2024-03-06T07:33:57.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb2f7fe640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 0x7fcb3015a0c0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fcb3008df00 tx=0x7fcb2800fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2801a070 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28006d60 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3015b0d0 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800fea0 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.749+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb301f4d10 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.750+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3004f330 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.751+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb2801d070 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.751+0000 7fcb177fe640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb1c06b4d0 0x7fcb1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.751+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcb28089900 con 0x7fcb301f7680 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.751+0000 7fcb2effd640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb1c06b4d0 0x7fcb1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.752+0000 7fcb2effd640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb1c06b4d0 0x7fcb1c06d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcb24006a90 tx=0x7fcb24006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:57.755+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb280587f0 con 0x7fcb301f7680 2024-03-06T07:33:58.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.004+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb300733b0 con 0x7fcb301f7680 2024-03-06T07:33:58.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.004+0000 7fcb177fe640 1 -- 172.21.15.122:0/2720258667 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fcb280580f0 con 0x7fcb301f7680 2024-03-06T07:33:58.006 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:58.007 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:58.007 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb1c06b4d0 msgr2=0x7fcb1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb1c06b4d0 0x7fcb1c06d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcb24006a90 tx=0x7fcb24006a20 comp rx=0 tx=0).stop 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 msgr2=0x7fcb3015a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 0x7fcb3015a0c0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fcb3008df00 tx=0x7fcb2800fc50 comp rx=0 tx=0).stop 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 shutdown_connections 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fcb1c06b4d0 0x7fcb1c06d990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb301f7680 0x7fcb3015a0c0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fcb300b9850 0x7fcb301f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 --2- 172.21.15.122:0/2720258667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb18004800 0x7fcb300b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 >> 172.21.15.122:0/2720258667 conn(0x7fcb300d3c40 msgr2=0x7fcb300b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 shutdown_connections 2024-03-06T07:33:58.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:58.006+0000 7fcb35581640 1 -- 172.21.15.122:0/2720258667 wait complete. 2024-03-06T07:33:58.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:58.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:33:59.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.322+0000 7f9ea54b1640 1 Processor -- start 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.322+0000 7f9ea54b1640 1 -- start start 2024-03-06T07:33:59.325 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9ea54b1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea006b220 0x7f9ea015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9ea54b1640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9ea00ce920 0x7f9ea0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9ea54b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ea0071ec0 0x7f9ea0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9ea54b1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea0161b70 con 0x7f9ea0071ec0 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9ea54b1640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea015ea30 con 0x7f9ea00ce920 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9ea54b1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea015ed10 con 0x7f9ea006b220 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ea0071ec0 0x7f9ea0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9ea0071ec0 0x7f9ea0162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60612/0 (socket says 172.21.15.122:60612) 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 -- 172.21.15.122:0/455735481 learned_addr learned my addr 172.21.15.122:0/455735481 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e97fff640 1 --2- 172.21.15.122:0/455735481 >> v2:172.21.15.169:3300/0 conn(0x7f9ea00ce920 0x7f9ea0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9effd640 1 --2- 172.21.15.122:0/455735481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea006b220 0x7f9ea015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 -- 172.21.15.122:0/455735481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea006b220 msgr2=0x7f9ea015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 --2- 172.21.15.122:0/455735481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea006b220 0x7f9ea015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 -- 172.21.15.122:0/455735481 >> v2:172.21.15.169:3300/0 conn(0x7f9ea00ce920 msgr2=0x7f9ea0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 --2- 172.21.15.122:0/455735481 >> v2:172.21.15.169:3300/0 conn(0x7f9ea00ce920 0x7f9ea0071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e9f7fe640 1 -- 172.21.15.122:0/455735481 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea015eff0 con 0x7f9ea0071ec0 2024-03-06T07:33:59.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.323+0000 7f9e97fff640 1 --2- 172.21.15.122:0/455735481 >> v2:172.21.15.169:3300/0 conn(0x7f9ea00ce920 0x7f9ea0071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:59.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.324+0000 7f9e9f7fe640 1 --2- 172.21.15.122:0/455735481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea0071ec0 0x7f9ea0162000 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f9ea006cf90 tx=0x7f9e9802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d4876e26bec397 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.324+0000 7f9e9e7fc640 1 -- 172.21.15.122:0/455735481 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9802ca70 con 0x7f9ea0071ec0 2024-03-06T07:33:59.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.324+0000 7f9e9e7fc640 1 -- 172.21.15.122:0/455735481 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9802cc10 con 0x7f9ea0071ec0 2024-03-06T07:33:59.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.324+0000 7f9e9e7fc640 1 -- 172.21.15.122:0/455735481 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e980343f0 con 0x7f9ea0071ec0 2024-03-06T07:33:59.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 -- 172.21.15.122:0/455735481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea0071ec0 msgr2=0x7f9ea0162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/455735481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea0071ec0 0x7f9ea0162000 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f9ea006cf90 tx=0x7f9e9802bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 -- 172.21.15.122:0/455735481 shutdown_connections 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/455735481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea0071ec0 0x7f9ea0162000 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/455735481 >> v2:172.21.15.169:3300/0 conn(0x7f9ea00ce920 0x7f9ea0071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/455735481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea006b220 0x7f9ea015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 -- 172.21.15.122:0/455735481 >> 172.21.15.122:0/455735481 conn(0x7f9ea006c0c0 msgr2=0x7f9ea006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 -- 172.21.15.122:0/455735481 shutdown_connections 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.325+0000 7f9ea54b1640 1 -- 172.21.15.122:0/455735481 wait complete. 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.326+0000 7f9ea54b1640 1 Processor -- start 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.326+0000 7f9ea54b1640 1 -- start start 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9ea54b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 0x7f9ea01f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9ea54b1640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea0071ec0 0x7f9ea0149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9ea54b1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ea00ce920 0x7f9ea0156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9ea54b1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00ce380 con 0x7f9ea006b220 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9ea54b1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00ce200 con 0x7f9ea00ce920 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9ea54b1640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea006b600 con 0x7f9ea0071ec0 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 0x7f9ea01f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 0x7f9ea01f3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60624/0 (socket says 172.21.15.122:60624) 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 -- 172.21.15.122:0/2442852116 learned_addr learned my addr 172.21.15.122:0/2442852116 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9f7fe640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ea00ce920 0x7f9ea0156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e97fff640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea0071ec0 0x7f9ea0149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 -- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea0071ec0 msgr2=0x7f9ea0149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea0071ec0 0x7f9ea0149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 -- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ea00ce920 msgr2=0x7f9ea0156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ea00ce920 0x7f9ea0156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 -- 172.21.15.122:0/2442852116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea01578a0 con 0x7f9ea006b220 2024-03-06T07:33:59.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e9effd640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 0x7f9ea01f3e00 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c7f0 tx=0x7f9e9000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.327+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90015070 con 0x7f9ea006b220 2024-03-06T07:33:59.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.328+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e90004240 con 0x7f9ea006b220 2024-03-06T07:33:59.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.328+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea0157b40 con 0x7f9ea006b220 2024-03-06T07:33:59.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.328+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90007620 con 0x7f9ea006b220 2024-03-06T07:33:59.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.328+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea01583e0 con 0x7f9ea006b220 2024-03-06T07:33:59.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.329+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea0157f30 con 0x7f9ea006b220 2024-03-06T07:33:59.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.329+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e900079d0 con 0x7f9ea006b220 2024-03-06T07:33:59.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.329+0000 7f9e977fe640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e8006b4d0 0x7f9e8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.329+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9e9008a1f0 con 0x7f9ea006b220 2024-03-06T07:33:59.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.329+0000 7f9e97fff640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e8006b4d0 0x7f9e8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.330+0000 7f9e97fff640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e8006b4d0 0x7f9e8006d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0153dd0 tx=0x7f9e88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.334+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e900590e0 con 0x7f9ea006b220 2024-03-06T07:33:59.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.556+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea004f330 con 0x7f9ea006b220 2024-03-06T07:33:59.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.556+0000 7f9e977fe640 1 -- 172.21.15.122:0/2442852116 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9e900589e0 con 0x7f9ea006b220 2024-03-06T07:33:59.559 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:33:59.559 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:33:59.559 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e8006b4d0 msgr2=0x7f9e8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e8006b4d0 0x7f9e8006d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0153dd0 tx=0x7f9e88008040 comp rx=0 tx=0).stop 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 msgr2=0x7f9ea01f3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 0x7f9ea01f3e00 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c7f0 tx=0x7f9e9000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 shutdown_connections 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9e8006b4d0 0x7f9e8006d990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9ea00ce920 0x7f9ea0156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9ea0071ec0 0x7f9ea0149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 --2- 172.21.15.122:0/2442852116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ea006b220 0x7f9ea01f3e00 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 >> 172.21.15.122:0/2442852116 conn(0x7f9ea006c0c0 msgr2=0x7f9ea006ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 shutdown_connections 2024-03-06T07:33:59.561 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:33:59.558+0000 7f9ea54b1640 1 -- 172.21.15.122:0/2442852116 wait complete. 2024-03-06T07:33:59.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:33:59.773 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:00.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:00.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.872+0000 7efc6704e640 1 Processor -- start 2024-03-06T07:34:00.874 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.872+0000 7efc6704e640 1 -- start start 2024-03-06T07:34:00.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc6704e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc6704e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efc600b41f0 0x7efc600b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc6704e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b3430 con 0x7efc6015dfe0 2024-03-06T07:34:00.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc6704e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc600d2c50 con 0x7efc600b41f0 2024-03-06T07:34:00.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc6704e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600b31e0 con 0x7efc600d22b0 2024-03-06T07:34:00.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc5ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc655eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7efc600b41f0 0x7efc600b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc5ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:39252/0 (socket says 172.21.15.122:39252) 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.873+0000 7efc5ffff640 1 -- 172.21.15.122:0/2879171603 learned_addr learned my addr 172.21.15.122:0/2879171603 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.874+0000 7efc5ffff640 1 -- 172.21.15.122:0/2879171603 >> v2:172.21.15.169:3300/0 conn(0x7efc600b41f0 msgr2=0x7efc600b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.874+0000 7efc5ffff640 1 --2- 172.21.15.122:0/2879171603 >> v2:172.21.15.169:3300/0 conn(0x7efc600b41f0 0x7efc600b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.874+0000 7efc5ffff640 1 -- 172.21.15.122:0/2879171603 >> v1:172.21.15.169:6789/0 conn(0x7efc6015dfe0 legacy=0x7efc600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.874+0000 7efc5ffff640 1 -- 172.21.15.122:0/2879171603 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b3710 con 0x7efc600d22b0 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.874+0000 7efc655eb640 1 --2- 172.21.15.122:0/2879171603 >> v2:172.21.15.169:3300/0 conn(0x7efc600b41f0 0x7efc600b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.874+0000 7efc5ffff640 1 --2- 172.21.15.122:0/2879171603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600d2710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009870 tx=0x7efc4c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91fbd76d920baf70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2879171603 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c034070 con 0x7efc600d22b0 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2879171603 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c030840 con 0x7efc600d22b0 2024-03-06T07:34:00.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2879171603 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c030c10 con 0x7efc600d22b0 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 -- 172.21.15.122:0/2879171603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 msgr2=0x7efc600d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 --2- 172.21.15.122:0/2879171603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600d2710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009870 tx=0x7efc4c02b980 comp rx=0 tx=0).stop 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 -- 172.21.15.122:0/2879171603 shutdown_connections 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 --2- 172.21.15.122:0/2879171603 >> v2:172.21.15.169:3300/0 conn(0x7efc600b41f0 0x7efc600b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 --2- 172.21.15.122:0/2879171603 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600d2710 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 -- 172.21.15.122:0/2879171603 >> 172.21.15.122:0/2879171603 conn(0x7efc600d3cd0 msgr2=0x7efc600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 -- 172.21.15.122:0/2879171603 shutdown_connections 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.875+0000 7efc6704e640 1 -- 172.21.15.122:0/2879171603 wait complete. 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.876+0000 7efc6704e640 1 Processor -- start 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.876+0000 7efc6704e640 1 -- start start 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc6704e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc600b41f0 0x7efc600b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc6704e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc6704e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 0x7efc600ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc6704e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc600cfa00 con 0x7efc6015dfe0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc6704e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc600b3d60 con 0x7efc600b41f0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc6704e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600d1c70 con 0x7efc600d22b0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc655eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 0x7efc600ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc5ffff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc655eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 0x7efc600ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60626/0 (socket says 172.21.15.122:60626) 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc655eb640 1 -- 172.21.15.122:0/442324432 learned_addr learned my addr 172.21.15.122:0/442324432 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.877+0000 7efc64dea640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc600b41f0 0x7efc600b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc655eb640 1 -- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 msgr2=0x7efc600b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc655eb640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc655eb640 1 -- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc600b41f0 msgr2=0x7efc600b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc655eb640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc600b41f0 0x7efc600b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc655eb640 1 -- 172.21.15.122:0/442324432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b90e0 con 0x7efc6015dfe0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc655eb640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 0x7efc600ccf70 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7efc5800efc0 tx=0x7efc5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc58018030 con 0x7efc6015dfe0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5800cdf0 con 0x7efc6015dfe0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc600b93c0 con 0x7efc6015dfe0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc58003bb0 con 0x7efc6015dfe0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.878+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc600b8cd0 con 0x7efc6015dfe0 2024-03-06T07:34:00.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.879+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6004f330 con 0x7efc6015dfe0 2024-03-06T07:34:00.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.880+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc58005d10 con 0x7efc6015dfe0 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.880+0000 7efc5d7fa640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc4406b4d0 0x7efc4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.880+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efc5808aa10 con 0x7efc6015dfe0 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.880+0000 7efc64dea640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc4406b4d0 0x7efc4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.880+0000 7efc64dea640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc4406b4d0 0x7efc4406d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7efc54001780 tx=0x7efc54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:00.884+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc58022070 con 0x7efc6015dfe0 2024-03-06T07:34:01.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.097+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc600b9b90 con 0x7efc6015dfe0 2024-03-06T07:34:01.099 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.098+0000 7efc5d7fa640 1 -- 172.21.15.122:0/442324432 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7efc58010040 con 0x7efc6015dfe0 2024-03-06T07:34:01.100 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:01.100 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:01.100 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.099+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc4406b4d0 msgr2=0x7efc4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.099+0000 7efc6704e640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc4406b4d0 0x7efc4406d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7efc54001780 tx=0x7efc54009040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.099+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 msgr2=0x7efc600ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.099+0000 7efc6704e640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 0x7efc600ccf70 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7efc5800efc0 tx=0x7efc5800c490 comp rx=0 tx=0).stop 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 shutdown_connections 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7efc4406b4d0 0x7efc4406d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc6015dfe0 0x7efc600ccf70 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7efc600d22b0 0x7efc600b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.102 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 --2- 172.21.15.122:0/442324432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efc600b41f0 0x7efc600b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 >> 172.21.15.122:0/442324432 conn(0x7efc600d3cd0 msgr2=0x7efc600b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 shutdown_connections 2024-03-06T07:34:01.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:01.100+0000 7efc6704e640 1 -- 172.21.15.122:0/442324432 wait complete. 2024-03-06T07:34:01.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:01.314 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:02.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:02.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 Processor -- start 2024-03-06T07:34:02.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 -- start start 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feb2c15dfe0 0x7feb2c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7feb2c0cf830 0x7feb2c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0b3430 con 0x7feb2c0d27f0 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0b31e0 con 0x7feb2c15dfe0 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.414+0000 7feb314a4640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0b3d60 con 0x7feb2c0cf830 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feb2c15dfe0 0x7feb2c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7feb2c0d27f0 0x7feb2c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35638/0 (socket says 172.21.15.122:35638) 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2b7fe640 1 -- 172.21.15.122:0/2480093828 learned_addr learned my addr 172.21.15.122:0/2480093828 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2a7fc640 1 --2- 172.21.15.122:0/2480093828 >> v2:172.21.15.169:3300/0 conn(0x7feb2c0cf830 0x7feb2c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2affd640 1 -- 172.21.15.122:0/2480093828 >> v2:172.21.15.169:3300/0 conn(0x7feb2c0cf830 msgr2=0x7feb2c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2affd640 1 --2- 172.21.15.122:0/2480093828 >> v2:172.21.15.169:3300/0 conn(0x7feb2c0cf830 0x7feb2c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb29ffb640 1 -- 172.21.15.122:0/2480093828 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929703877 0 0) 0x7feb2c0b3430 con 0x7feb2c0d27f0 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2affd640 1 -- 172.21.15.122:0/2480093828 >> v1:172.21.15.122:6789/0 conn(0x7feb2c0d27f0 legacy=0x7feb2c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.415+0000 7feb2affd640 1 -- 172.21.15.122:0/2480093828 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0b3710 con 0x7feb2c15dfe0 2024-03-06T07:34:02.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb2affd640 1 --2- 172.21.15.122:0/2480093828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0cf2f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7feb24009d90 tx=0x7feb2402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b40ddbf527514c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb29ffb640 1 -- 172.21.15.122:0/2480093828 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24039070 con 0x7feb2c15dfe0 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb29ffb640 1 -- 172.21.15.122:0/2480093828 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2402cc60 con 0x7feb2c15dfe0 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb29ffb640 1 -- 172.21.15.122:0/2480093828 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24034660 con 0x7feb2c15dfe0 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 -- 172.21.15.122:0/2480093828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 msgr2=0x7feb2c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 --2- 172.21.15.122:0/2480093828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0cf2f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7feb24009d90 tx=0x7feb2402beb0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 -- 172.21.15.122:0/2480093828 shutdown_connections 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 --2- 172.21.15.122:0/2480093828 >> v2:172.21.15.169:3300/0 conn(0x7feb2c0cf830 0x7feb2c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 --2- 172.21.15.122:0/2480093828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0cf2f0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 -- 172.21.15.122:0/2480093828 >> 172.21.15.122:0/2480093828 conn(0x7feb2c0d3cd0 msgr2=0x7feb2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 -- 172.21.15.122:0/2480093828 shutdown_connections 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.416+0000 7feb314a4640 1 -- 172.21.15.122:0/2480093828 wait complete. 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 Processor -- start 2024-03-06T07:34:02.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 -- start start 2024-03-06T07:34:02.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb2c0cf830 0x7feb2c0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feb2c0d27f0 0x7feb2c0b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0d1df0 con 0x7feb2c15dfe0 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0d1c70 con 0x7feb2c0cf830 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb314a4640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c15e3c0 con 0x7feb2c0d27f0 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb2b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb2b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60644/0 (socket says 172.21.15.122:60644) 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.418+0000 7feb2b7fe640 1 -- 172.21.15.122:0/2056069630 learned_addr learned my addr 172.21.15.122:0/2056069630 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb2b7fe640 1 -- 172.21.15.122:0/2056069630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feb2c0d27f0 msgr2=0x7feb2c0b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb2b7fe640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feb2c0d27f0 0x7feb2c0b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb2b7fe640 1 -- 172.21.15.122:0/2056069630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb2c0cf830 msgr2=0x7feb2c0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb2b7fe640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb2c0cf830 0x7feb2c0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb2b7fe640 1 -- 172.21.15.122:0/2056069630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0b6be0 con 0x7feb2c15dfe0 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb2b7fe640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0b84b0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c450 tx=0x7feb1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c018070 con 0x7feb2c15dfe0 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c007500 con 0x7feb2c15dfe0 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c004a20 con 0x7feb2c15dfe0 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c0b6ec0 con 0x7feb2c15dfe0 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.419+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c0b7760 con 0x7feb2c15dfe0 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.420+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c04f330 con 0x7feb2c15dfe0 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.420+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb1c01b070 con 0x7feb2c15dfe0 2024-03-06T07:34:02.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.421+0000 7feb237fe640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb0c06b4d0 0x7feb0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.421+0000 7feb2affd640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb0c06b4d0 0x7feb0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.421+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feb1c0894a0 con 0x7feb2c15dfe0 2024-03-06T07:34:02.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.421+0000 7feb2affd640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb0c06b4d0 0x7feb0c06d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb2402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.425+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb1c058300 con 0x7feb2c15dfe0 2024-03-06T07:34:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.787+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb2c0ce930 con 0x7feb2c15dfe0 2024-03-06T07:34:02.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.788+0000 7feb237fe640 1 -- 172.21.15.122:0/2056069630 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7feb1c006e20 con 0x7feb2c15dfe0 2024-03-06T07:34:02.790 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:02.790 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:02.790 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb0c06b4d0 msgr2=0x7feb0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb0c06b4d0 0x7feb0c06d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb2402e000 comp rx=0 tx=0).stop 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 msgr2=0x7feb2c0b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0b84b0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c450 tx=0x7feb1c00c910 comp rx=0 tx=0).stop 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 shutdown_connections 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7feb0c06b4d0 0x7feb0c06d990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb2c15dfe0 0x7feb2c0b84b0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7feb2c0d27f0 0x7feb2c0b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 --2- 172.21.15.122:0/2056069630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb2c0cf830 0x7feb2c0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 >> 172.21.15.122:0/2056069630 conn(0x7feb2c0d3cd0 msgr2=0x7feb2c0b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 shutdown_connections 2024-03-06T07:34:02.794 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:02.791+0000 7feb314a4640 1 -- 172.21.15.122:0/2056069630 wait complete. 2024-03-06T07:34:03.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:03.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:04.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 Processor -- start 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 -- start start 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c0cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fe72c0ce0d0 0x7fe72c15c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c0d0a70 con 0x7fe72c06afb0 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c0d0850 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.108 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.106+0000 7fe732464640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0d0d10 con 0x7fe72c0b4420 2024-03-06T07:34:04.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe72c06afb0 0x7fe72c06b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35650/0 (socket says 172.21.15.122:35650) 2024-03-06T07:34:04.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72bfff640 1 -- 172.21.15.122:0/3451918340 learned_addr learned my addr 172.21.15.122:0/3451918340 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:04.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72b7fe640 1 --2- 172.21.15.122:0/3451918340 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c0cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe730a01640 1 --2- 172.21.15.122:0/3451918340 >> v2:172.21.15.169:3300/0 conn(0x7fe72c0ce0d0 0x7fe72c15c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72affd640 1 -- 172.21.15.122:0/3451918340 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284413414 0 0) 0x7fe72c0d0a70 con 0x7fe72c06afb0 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72affd640 1 -- 172.21.15.122:0/3451918340 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe714003520 con 0x7fe72c06afb0 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72b7fe640 1 -- 172.21.15.122:0/3451918340 >> v2:172.21.15.169:3300/0 conn(0x7fe72c0ce0d0 msgr2=0x7fe72c15c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72b7fe640 1 --2- 172.21.15.122:0/3451918340 >> v2:172.21.15.169:3300/0 conn(0x7fe72c0ce0d0 0x7fe72c15c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72b7fe640 1 -- 172.21.15.122:0/3451918340 >> v1:172.21.15.122:6789/0 conn(0x7fe72c06afb0 legacy=0x7fe72c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe72b7fe640 1 -- 172.21.15.122:0/3451918340 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c0d0ff0 con 0x7fe72c0b4420 2024-03-06T07:34:04.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.107+0000 7fe730a01640 1 --2- 172.21.15.122:0/3451918340 >> v2:172.21.15.169:3300/0 conn(0x7fe72c0ce0d0 0x7fe72c15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:04.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.108+0000 7fe72b7fe640 1 --2- 172.21.15.122:0/3451918340 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c0cdb90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0097a0 tx=0x7fe71c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfffa7fad2ded1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe72affd640 1 -- 172.21.15.122:0/3451918340 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0315a0 con 0x7fe72c0b4420 2024-03-06T07:34:04.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe72affd640 1 -- 172.21.15.122:0/3451918340 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c037070 con 0x7fe72c0b4420 2024-03-06T07:34:04.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe72affd640 1 -- 172.21.15.122:0/3451918340 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c031940 con 0x7fe72c0b4420 2024-03-06T07:34:04.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 -- 172.21.15.122:0/3451918340 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 msgr2=0x7fe72c0cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 --2- 172.21.15.122:0/3451918340 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c0cdb90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0097a0 tx=0x7fe71c02b840 comp rx=0 tx=0).stop 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 -- 172.21.15.122:0/3451918340 shutdown_connections 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 --2- 172.21.15.122:0/3451918340 >> v2:172.21.15.169:3300/0 conn(0x7fe72c0ce0d0 0x7fe72c15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 --2- 172.21.15.122:0/3451918340 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c0cdb90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 -- 172.21.15.122:0/3451918340 >> 172.21.15.122:0/3451918340 conn(0x7fe72c06c0c0 msgr2=0x7fe72c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 -- 172.21.15.122:0/3451918340 shutdown_connections 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.109+0000 7fe732464640 1 -- 172.21.15.122:0/3451918340 wait complete. 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.110+0000 7fe732464640 1 Processor -- start 2024-03-06T07:34:04.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 -- start start 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe72c06afb0 0x7fe72c144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c1557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 0x7fe72c144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0ceb90 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c145840 con 0x7fe72c06afb0 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe732464640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c15c9d0 con 0x7fe72c0b4420 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe730a01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 0x7fe72c144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe730a01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 0x7fe72c144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60646/0 (socket says 172.21.15.122:60646) 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe730a01640 1 -- 172.21.15.122:0/3048096183 learned_addr learned my addr 172.21.15.122:0/3048096183 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe72b7fe640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c1557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.111+0000 7fe72bfff640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe72c06afb0 0x7fe72c144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe730a01640 1 -- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 msgr2=0x7fe72c1557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe730a01640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c1557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe730a01640 1 -- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe72c06afb0 msgr2=0x7fe72c144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe730a01640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe72c06afb0 0x7fe72c144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe730a01640 1 -- 172.21.15.122:0/3048096183 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c144fe0 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe730a01640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 0x7fe72c144aa0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fe72400efc0 tx=0x7fe72400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724018030 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72400cdf0 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c156b80 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724003bb0 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.112+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c157140 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.113+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c04f330 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.114+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe724005d10 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.114+0000 7fe728ff9640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe71806b500 0x7fe71806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.114+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe72408a980 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.114+0000 7fe72bfff640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe71806b500 0x7fe71806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.114+0000 7fe72bfff640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe71806b500 0x7fe71806d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe720005880 tx=0x7fe720009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.118+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe724022070 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.334+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe72c0b5cb0 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.334+0000 7fe728ff9640 1 -- 172.21.15.122:0/3048096183 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fe724010040 con 0x7fe72c0ce0d0 2024-03-06T07:34:04.337 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:04.337 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:04.337 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:04.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe71806b500 msgr2=0x7fe71806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe71806b500 0x7fe71806d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe720005880 tx=0x7fe720009040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 msgr2=0x7fe72c144aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 0x7fe72c144aa0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fe72400efc0 tx=0x7fe72400c490 comp rx=0 tx=0).stop 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 shutdown_connections 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe71806b500 0x7fe71806d9c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe72c0ce0d0 0x7fe72c144aa0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe72c0b4420 0x7fe72c1557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 --2- 172.21.15.122:0/3048096183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe72c06afb0 0x7fe72c144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 >> 172.21.15.122:0/3048096183 conn(0x7fe72c06c0c0 msgr2=0x7fe72c15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 shutdown_connections 2024-03-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:04.336+0000 7fe732464640 1 -- 172.21.15.122:0/3048096183 wait complete. 2024-03-06T07:34:04.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:04.554 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:05.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:05.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.653+0000 7f0819a84640 1 Processor -- start 2024-03-06T07:34:05.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0819a84640 1 -- start start 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0819a84640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f08140d22b0 0x7f08140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0819a84640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f08140b41f0 0x7f08140b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0819a84640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140b3430 con 0x7f081415dfe0 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0819a84640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08140d2c50 con 0x7f08140b41f0 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0819a84640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08140b31e0 con 0x7f08140d22b0 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.654+0000 7f0813fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f08140b41f0 0x7f08140b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0813fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f08140b41f0 0x7f08140b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37316/0 (socket says 172.21.15.122:37316) 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0813fff640 1 -- 172.21.15.122:0/4077886807 learned_addr learned my addr 172.21.15.122:0/4077886807 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0812ffd640 1 --2- 172.21.15.122:0/4077886807 >> v2:172.21.15.169:3300/0 conn(0x7f08140d22b0 0x7f08140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0813fff640 1 -- 172.21.15.122:0/4077886807 >> v2:172.21.15.169:3300/0 conn(0x7f08140d22b0 msgr2=0x7f08140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0813fff640 1 --2- 172.21.15.122:0/4077886807 >> v2:172.21.15.169:3300/0 conn(0x7f08140d22b0 0x7f08140d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0813fff640 1 -- 172.21.15.122:0/4077886807 >> v1:172.21.15.169:6789/0 conn(0x7f081415dfe0 legacy=0x7f08140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.655+0000 7f0813fff640 1 -- 172.21.15.122:0/4077886807 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140b3710 con 0x7f08140b41f0 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0813fff640 1 --2- 172.21.15.122:0/4077886807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140b41f0 0x7f08140b4680 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f080c009e50 tx=0x7f080c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7479685dce613b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f08127fc640 1 -- 172.21.15.122:0/4077886807 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c02ca80 con 0x7f08140b41f0 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f08127fc640 1 -- 172.21.15.122:0/4077886807 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080c02cc20 con 0x7f08140b41f0 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f08127fc640 1 -- 172.21.15.122:0/4077886807 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c0343f0 con 0x7f08140b41f0 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 -- 172.21.15.122:0/4077886807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140b41f0 msgr2=0x7f08140b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 --2- 172.21.15.122:0/4077886807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140b41f0 0x7f08140b4680 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f080c009e50 tx=0x7f080c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 -- 172.21.15.122:0/4077886807 shutdown_connections 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 --2- 172.21.15.122:0/4077886807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140b41f0 0x7f08140b4680 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 --2- 172.21.15.122:0/4077886807 >> v2:172.21.15.169:3300/0 conn(0x7f08140d22b0 0x7f08140d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 -- 172.21.15.122:0/4077886807 >> 172.21.15.122:0/4077886807 conn(0x7f08140d3cd0 msgr2=0x7f08140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 -- 172.21.15.122:0/4077886807 shutdown_connections 2024-03-06T07:34:05.661 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.656+0000 7f0819a84640 1 -- 172.21.15.122:0/4077886807 wait complete. 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.657+0000 7f0819a84640 1 Processor -- start 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 -- start start 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08140b41f0 0x7f08140b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 0x7f08140b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f081415dfe0 0x7f08140b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08140cfa00 con 0x7f08140d22b0 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08140b3d60 con 0x7f08140b41f0 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0819a84640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08140d1c70 con 0x7f081415dfe0 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0812ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 0x7f08140b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0812ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 0x7f08140b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37322/0 (socket says 172.21.15.122:37322) 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.658+0000 7f0812ffd640 1 -- 172.21.15.122:0/3411116991 learned_addr learned my addr 172.21.15.122:0/3411116991 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:05.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0813fff640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f081415dfe0 0x7f08140b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0812ffd640 1 -- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f081415dfe0 msgr2=0x7f08140b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0812ffd640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f081415dfe0 0x7f08140b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0812ffd640 1 -- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08140b41f0 msgr2=0x7f08140b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0812ffd640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08140b41f0 0x7f08140b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0812ffd640 1 -- 172.21.15.122:0/3411116991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140b9be0 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0812ffd640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 0x7f08140b68a0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f080800d830 tx=0x7f080800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800b6c0 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808019070 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800ba80 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08140b9060 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.659+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08140b9620 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.660+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081404f330 con 0x7f08140d22b0 2024-03-06T07:34:05.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.661+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f080801e460 con 0x7f08140d22b0 2024-03-06T07:34:05.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.661+0000 7f07fbfff640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f080006b500 0x7f080006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.661+0000 7f08137fe640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f080006b500 0x7f080006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.661+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f080808a8e0 con 0x7f08140d22b0 2024-03-06T07:34:05.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.662+0000 7f08137fe640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f080006b500 0x7f080006d9c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0804006fd0 tx=0x7f0804006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.665+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0808059740 con 0x7f08140d22b0 2024-03-06T07:34:05.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.894+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08140cea80 con 0x7f08140d22b0 2024-03-06T07:34:05.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.894+0000 7f07fbfff640 1 -- 172.21.15.122:0/3411116991 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0808059040 con 0x7f08140d22b0 2024-03-06T07:34:05.896 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:05.897 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:05.897 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f080006b500 msgr2=0x7f080006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f080006b500 0x7f080006d9c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0804006fd0 tx=0x7f0804006bc0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 msgr2=0x7f08140b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 0x7f08140b68a0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f080800d830 tx=0x7f080800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 shutdown_connections 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f080006b500 0x7f080006d9c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f081415dfe0 0x7f08140b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140d22b0 0x7f08140b68a0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 --2- 172.21.15.122:0/3411116991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08140b41f0 0x7f08140b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 >> 172.21.15.122:0/3411116991 conn(0x7f08140d3cd0 msgr2=0x7f08140d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 shutdown_connections 2024-03-06T07:34:05.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:05.897+0000 7f0819a84640 1 -- 172.21.15.122:0/3411116991 wait complete. 2024-03-06T07:34:06.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:06.111 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:07.112 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:07.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.211+0000 7fae99047640 1 Processor -- start 2024-03-06T07:34:07.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.211+0000 7fae99047640 1 -- start start 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae99047640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fae9406c6d0 0x7fae9415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae99047640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae94066d90 0x7fae940ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae99047640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae940cd450 con 0x7fae940b4f70 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae99047640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9415e9f0 con 0x7fae94066d90 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae99047640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae940cda10 con 0x7fae9406c6d0 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae94066d90 0x7fae940ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae94066d90 0x7fae940ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37328/0 (socket says 172.21.15.122:37328) 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 -- 172.21.15.122:0/1647427064 learned_addr learned my addr 172.21.15.122:0/1647427064 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 -- 172.21.15.122:0/1647427064 >> v2:172.21.15.169:3300/0 conn(0x7fae9406c6d0 msgr2=0x7fae9415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 --2- 172.21.15.122:0/1647427064 >> v2:172.21.15.169:3300/0 conn(0x7fae9406c6d0 0x7fae9415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 -- 172.21.15.122:0/1647427064 >> v1:172.21.15.169:6789/0 conn(0x7fae940b4f70 legacy=0x7fae940b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.212+0000 7fae9359e640 1 -- 172.21.15.122:0/1647427064 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae940cd730 con 0x7fae94066d90 2024-03-06T07:34:07.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.213+0000 7fae9359e640 1 --2- 172.21.15.122:0/1647427064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae94066d90 0x7fae940ccb50 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fae940ccb00 tx=0x7fae8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07d0d98e21eaffc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.213+0000 7fae9259c640 1 -- 172.21.15.122:0/1647427064 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c02ca80 con 0x7fae94066d90 2024-03-06T07:34:07.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.213+0000 7fae9259c640 1 -- 172.21.15.122:0/1647427064 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c02cc20 con 0x7fae94066d90 2024-03-06T07:34:07.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.213+0000 7fae9259c640 1 -- 172.21.15.122:0/1647427064 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c0343f0 con 0x7fae94066d90 2024-03-06T07:34:07.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 -- 172.21.15.122:0/1647427064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae94066d90 msgr2=0x7fae940ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.217 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 --2- 172.21.15.122:0/1647427064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae94066d90 0x7fae940ccb50 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fae940ccb00 tx=0x7fae8c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 -- 172.21.15.122:0/1647427064 shutdown_connections 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 --2- 172.21.15.122:0/1647427064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae94066d90 0x7fae940ccb50 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 --2- 172.21.15.122:0/1647427064 >> v2:172.21.15.169:3300/0 conn(0x7fae9406c6d0 0x7fae9415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 -- 172.21.15.122:0/1647427064 >> 172.21.15.122:0/1647427064 conn(0x7fae940724d0 msgr2=0x7fae940728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 -- 172.21.15.122:0/1647427064 shutdown_connections 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.214+0000 7fae99047640 1 -- 172.21.15.122:0/1647427064 wait complete. 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.215+0000 7fae99047640 1 Processor -- start 2024-03-06T07:34:07.218 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.215+0000 7fae99047640 1 -- start start 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae94066d90 0x7fae941f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 0x7fae9406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae940d0500 0x7fae940d0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940cd090 con 0x7fae940b4f70 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415f3c0 con 0x7fae940d0500 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940cd210 con 0x7fae94066d90 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 0x7fae9406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 0x7fae9406dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37344/0 (socket says 172.21.15.122:37344) 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 -- 172.21.15.122:0/4241982403 learned_addr learned my addr 172.21.15.122:0/4241982403 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae92d9d640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae94066d90 0x7fae941f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae9359e640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae940d0500 0x7fae940d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 -- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae94066d90 msgr2=0x7fae941f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae94066d90 0x7fae941f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 -- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae940d0500 msgr2=0x7fae940d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae940d0500 0x7fae940d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 -- 172.21.15.122:0/4241982403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae941f51a0 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae9359e640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae940d0500 0x7fae940d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8bfff640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 0x7fae9406dbd0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00b4d0 tx=0x7fae7c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c014070 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c010870 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae940cb2c0 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c010c60 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.216+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae940c98c0 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.218+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9404f330 con 0x7fae940b4f70 2024-03-06T07:34:07.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.218+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae7c017070 con 0x7fae940b4f70 2024-03-06T07:34:07.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.218+0000 7fae8b7fe640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae7406b4d0 0x7fae7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.218+0000 7fae92d9d640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae7406b4d0 0x7fae7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.218+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fae7c08a040 con 0x7fae940b4f70 2024-03-06T07:34:07.223 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.219+0000 7fae92d9d640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae7406b4d0 0x7fae7406d990 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fae84007fd0 tx=0x7fae84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.224 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.223+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae941f51a0 con 0x7fae940b4f70 2024-03-06T07:34:07.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.435+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae940b1860 con 0x7fae940b4f70 2024-03-06T07:34:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.435+0000 7fae8b7fe640 1 -- 172.21.15.122:0/4241982403 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fae7c058ea0 con 0x7fae940b4f70 2024-03-06T07:34:07.437 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:07.437 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae7406b4d0 msgr2=0x7fae7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae7406b4d0 0x7fae7406d990 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fae84007fd0 tx=0x7fae84009040 comp rx=0 tx=0).stop 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 msgr2=0x7fae9406dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 0x7fae9406dbd0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00b4d0 tx=0x7fae7c00b990 comp rx=0 tx=0).stop 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 shutdown_connections 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fae7406b4d0 0x7fae7406d990 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae940d0500 0x7fae940d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae940b4f70 0x7fae9406dbd0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 --2- 172.21.15.122:0/4241982403 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fae94066d90 0x7fae941f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 >> 172.21.15.122:0/4241982403 conn(0x7fae940724d0 msgr2=0x7fae9406bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 shutdown_connections 2024-03-06T07:34:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:07.437+0000 7fae99047640 1 -- 172.21.15.122:0/4241982403 wait complete. 2024-03-06T07:34:07.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:07.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:08.652 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:08.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.750+0000 7ff562dc9640 1 Processor -- start 2024-03-06T07:34:08.752 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.750+0000 7ff562dc9640 1 -- start start 2024-03-06T07:34:08.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff562dc9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff55c15dfe0 0x7ff55c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff562dc9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff55c0d29c0 0x7ff55c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.754 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff562dc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55c0b3430 con 0x7ff55c0cf830 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff562dc9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c0b31e0 con 0x7ff55c15dfe0 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff562dc9640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c0b3d60 con 0x7ff55c0d29c0 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff560b65640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff55c15dfe0 0x7ff55c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff55bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff55c0cf830 0x7ff55c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35600/0 (socket says 172.21.15.122:35600) 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff55bfff640 1 -- 172.21.15.122:0/3594341333 learned_addr learned my addr 172.21.15.122:0/3594341333 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff561366640 1 --2- 172.21.15.122:0/3594341333 >> v2:172.21.15.169:3300/0 conn(0x7ff55c0d29c0 0x7ff55c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff560b65640 1 -- 172.21.15.122:0/3594341333 >> v2:172.21.15.169:3300/0 conn(0x7ff55c0d29c0 msgr2=0x7ff55c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff560b65640 1 --2- 172.21.15.122:0/3594341333 >> v2:172.21.15.169:3300/0 conn(0x7ff55c0d29c0 0x7ff55c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.751+0000 7ff560b65640 1 -- 172.21.15.122:0/3594341333 >> v1:172.21.15.122:6789/0 conn(0x7ff55c0cf830 legacy=0x7ff55c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.752+0000 7ff55b7fe640 1 -- 172.21.15.122:0/3594341333 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293030678 0 0) 0x7ff55c0b3430 con 0x7ff55c0cf830 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.752+0000 7ff560b65640 1 -- 172.21.15.122:0/3594341333 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c0b3710 con 0x7ff55c15dfe0 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.752+0000 7ff560b65640 1 --2- 172.21.15.122:0/3594341333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0cf2f0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0b3cd0 tx=0x7ff55002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c026e56ff2a9b651 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.755 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.752+0000 7ff55b7fe640 1 -- 172.21.15.122:0/3594341333 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550035020 con 0x7ff55c15dfe0 2024-03-06T07:34:08.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.752+0000 7ff55b7fe640 1 -- 172.21.15.122:0/3594341333 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550031830 con 0x7ff55c15dfe0 2024-03-06T07:34:08.756 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.752+0000 7ff55b7fe640 1 -- 172.21.15.122:0/3594341333 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550031c00 con 0x7ff55c15dfe0 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 -- 172.21.15.122:0/3594341333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 msgr2=0x7ff55c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 --2- 172.21.15.122:0/3594341333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0cf2f0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0b3cd0 tx=0x7ff55002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 -- 172.21.15.122:0/3594341333 shutdown_connections 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 --2- 172.21.15.122:0/3594341333 >> v2:172.21.15.169:3300/0 conn(0x7ff55c0d29c0 0x7ff55c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 --2- 172.21.15.122:0/3594341333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0cf2f0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 -- 172.21.15.122:0/3594341333 >> 172.21.15.122:0/3594341333 conn(0x7ff55c0d3cd0 msgr2=0x7ff55c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 -- 172.21.15.122:0/3594341333 shutdown_connections 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.753+0000 7ff562dc9640 1 -- 172.21.15.122:0/3594341333 wait complete. 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.754+0000 7ff562dc9640 1 Processor -- start 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.754+0000 7ff562dc9640 1 -- start start 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff562dc9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff55c0cf830 0x7ff55c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff562dc9640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff55c0d29c0 0x7ff55c1f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff562dc9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff562dc9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0d1df0 con 0x7ff55c15dfe0 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff562dc9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0d1c70 con 0x7ff55c0cf830 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff562dc9640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c15e3c0 con 0x7ff55c0d29c0 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37372/0 (socket says 172.21.15.122:37372) 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 -- 172.21.15.122:0/1050505140 learned_addr learned my addr 172.21.15.122:0/1050505140 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff560b65640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff55c0cf830 0x7ff55c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff55bfff640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff55c0d29c0 0x7ff55c1f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 -- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff55c0d29c0 msgr2=0x7ff55c1f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff55c0d29c0 0x7ff55c1f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 -- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff55c0cf830 msgr2=0x7ff55c15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff55c0cf830 0x7ff55c15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.755+0000 7ff561366640 1 -- 172.21.15.122:0/1050505140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c1f7280 con 0x7ff55c15dfe0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.756+0000 7ff561366640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0b68d0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7ff554007e30 tx=0x7ff55400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.756+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554014620 con 0x7ff55c15dfe0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.756+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff554019070 con 0x7ff55c15dfe0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.756+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c1f7560 con 0x7ff55c15dfe0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.756+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554014a00 con 0x7ff55c15dfe0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.756+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c1f7b20 con 0x7ff55c15dfe0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.757+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c04f330 con 0x7ff55c15dfe0 2024-03-06T07:34:08.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.757+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff554014ba0 con 0x7ff55c15dfe0 2024-03-06T07:34:08.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.758+0000 7ff5597fa640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff54006b500 0x7ff54006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.758+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff55408a9d0 con 0x7ff55c15dfe0 2024-03-06T07:34:08.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.758+0000 7ff560b65640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff54006b500 0x7ff54006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.758+0000 7ff560b65640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff54006b500 0x7ff54006d9c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff550036000 tx=0x7ff550008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.762+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff554011260 con 0x7ff55c15dfe0 2024-03-06T07:34:08.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.977+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff55c1f7d20 con 0x7ff55c15dfe0 2024-03-06T07:34:08.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.977+0000 7ff5597fa640 1 -- 172.21.15.122:0/1050505140 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7ff5540598c0 con 0x7ff55c15dfe0 2024-03-06T07:34:08.980 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:08.980 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:08.980 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff54006b500 msgr2=0x7ff54006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff54006b500 0x7ff54006d9c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff550036000 tx=0x7ff550008030 comp rx=0 tx=0).stop 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 msgr2=0x7ff55c0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0b68d0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7ff554007e30 tx=0x7ff55400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 shutdown_connections 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff54006b500 0x7ff54006d9c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff55c15dfe0 0x7ff55c0b68d0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff55c0d29c0 0x7ff55c1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 --2- 172.21.15.122:0/1050505140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff55c0cf830 0x7ff55c15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 >> 172.21.15.122:0/1050505140 conn(0x7ff55c0d3cd0 msgr2=0x7ff55c0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 shutdown_connections 2024-03-06T07:34:08.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:08.979+0000 7ff562dc9640 1 -- 172.21.15.122:0/1050505140 wait complete. 2024-03-06T07:34:09.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:09.194 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:10.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:10.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.293+0000 7fe65c688640 1 Processor -- start 2024-03-06T07:34:10.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.293+0000 7fe65c688640 1 -- start start 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.294+0000 7fe65c688640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe65415dfe0 0x7fe6540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.294+0000 7fe65c688640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe6540d29c0 0x7fe6540b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.294+0000 7fe65c688640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6540b3430 con 0x7fe6540cf830 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.294+0000 7fe65c688640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6540b31e0 con 0x7fe6540d29c0 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.294+0000 7fe65c688640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540b3d60 con 0x7fe65415dfe0 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe6540d29c0 0x7fe6540b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe6540d29c0 0x7fe6540b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37378/0 (socket says 172.21.15.122:37378) 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 -- 172.21.15.122:0/2525548966 learned_addr learned my addr 172.21.15.122:0/2525548966 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:10.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65a424640 1 --2- 172.21.15.122:0/2525548966 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe65415dfe0 0x7fe6540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 -- 172.21.15.122:0/2525548966 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe65415dfe0 msgr2=0x7fe6540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 --2- 172.21.15.122:0/2525548966 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe65415dfe0 0x7fe6540cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 -- 172.21.15.122:0/2525548966 >> v1:172.21.15.169:6789/0 conn(0x7fe6540cf830 legacy=0x7fe6540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.295+0000 7fe65ac25640 1 -- 172.21.15.122:0/2525548966 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540b3710 con 0x7fe6540d29c0 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65ac25640 1 --2- 172.21.15.122:0/2525548966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6540d29c0 0x7fe6540b41f0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009e50 tx=0x7fe64c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab2b299890f8d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe659422640 1 -- 172.21.15.122:0/2525548966 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c02cb00 con 0x7fe6540d29c0 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe659422640 1 -- 172.21.15.122:0/2525548966 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c02cca0 con 0x7fe6540d29c0 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe659422640 1 -- 172.21.15.122:0/2525548966 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c034610 con 0x7fe6540d29c0 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 -- 172.21.15.122:0/2525548966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6540d29c0 msgr2=0x7fe6540b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 --2- 172.21.15.122:0/2525548966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6540d29c0 0x7fe6540b41f0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009e50 tx=0x7fe64c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 -- 172.21.15.122:0/2525548966 shutdown_connections 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 --2- 172.21.15.122:0/2525548966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6540d29c0 0x7fe6540b41f0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 --2- 172.21.15.122:0/2525548966 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe65415dfe0 0x7fe6540cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 -- 172.21.15.122:0/2525548966 >> 172.21.15.122:0/2525548966 conn(0x7fe6540d3cd0 msgr2=0x7fe6540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 -- 172.21.15.122:0/2525548966 shutdown_connections 2024-03-06T07:34:10.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.296+0000 7fe65c688640 1 -- 172.21.15.122:0/2525548966 wait complete. 2024-03-06T07:34:10.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 Processor -- start 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 -- start start 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6540cf830 0x7fe6540a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6540d29c0 0x7fe6540cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 0x7fe65415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540d1df0 con 0x7fe65415dfe0 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540d1c70 con 0x7fe6540d29c0 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65c688640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65415e3c0 con 0x7fe6540cf830 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65ac25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 0x7fe65415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65ac25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 0x7fe65415be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37388/0 (socket says 172.21.15.122:37388) 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65ac25640 1 -- 172.21.15.122:0/1947653963 learned_addr learned my addr 172.21.15.122:0/1947653963 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.298+0000 7fe65a424640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6540cf830 0x7fe6540a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe659c23640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6540d29c0 0x7fe6540cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65ac25640 1 -- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6540cf830 msgr2=0x7fe6540a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65ac25640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6540cf830 0x7fe6540a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65ac25640 1 -- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6540d29c0 msgr2=0x7fe6540cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65ac25640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6540d29c0 0x7fe6540cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65ac25640 1 -- 172.21.15.122:0/1947653963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540caf80 con 0x7fe65415dfe0 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65ac25640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 0x7fe65415be30 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009e50 tx=0x7fe64c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c043020 con 0x7fe65415dfe0 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c02ce50 con 0x7fe65415dfe0 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6540cb260 con 0x7fe65415dfe0 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c009510 con 0x7fe65415dfe0 2024-03-06T07:34:10.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.299+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6540cd7a0 con 0x7fe65415dfe0 2024-03-06T07:34:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.300+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65404f330 con 0x7fe65415dfe0 2024-03-06T07:34:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.301+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe64c046020 con 0x7fe65415dfe0 2024-03-06T07:34:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.301+0000 7fe64affd640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe63806b4d0 0x7fe63806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.301+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe64c0aca70 con 0x7fe65415dfe0 2024-03-06T07:34:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.301+0000 7fe65a424640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe63806b4d0 0x7fe63806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.302+0000 7fe65a424640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe63806b4d0 0x7fe63806d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe6500079a0 tx=0x7fe650008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.307 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.305+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe64c04d030 con 0x7fe65415dfe0 2024-03-06T07:34:10.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.523+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6540ca380 con 0x7fe65415dfe0 2024-03-06T07:34:10.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.524+0000 7fe64affd640 1 -- 172.21.15.122:0/1947653963 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fe64c07b310 con 0x7fe65415dfe0 2024-03-06T07:34:10.526 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:10.526 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:10.526 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:10.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe63806b4d0 msgr2=0x7fe63806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe63806b4d0 0x7fe63806d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe6500079a0 tx=0x7fe650008040 comp rx=0 tx=0).stop 2024-03-06T07:34:10.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 msgr2=0x7fe65415be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 0x7fe65415be30 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009e50 tx=0x7fe64c004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:10.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 shutdown_connections 2024-03-06T07:34:10.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fe63806b4d0 0x7fe63806d990 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe65415dfe0 0x7fe65415be30 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6540d29c0 0x7fe6540cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 --2- 172.21.15.122:0/1947653963 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fe6540cf830 0x7fe6540a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 >> 172.21.15.122:0/1947653963 conn(0x7fe6540d3cd0 msgr2=0x7fe6540b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 shutdown_connections 2024-03-06T07:34:10.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:10.526+0000 7fe65c688640 1 -- 172.21.15.122:0/1947653963 wait complete. 2024-03-06T07:34:10.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:10.740 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:11.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:11.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.840+0000 7f2a0ec0e640 1 Processor -- start 2024-03-06T07:34:11.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0ec0e640 1 -- start start 2024-03-06T07:34:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0ec0e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f2a080b4170 0x7f2a080b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0ec0e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080d25d0 con 0x7f2a080d0a80 2024-03-06T07:34:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0ec0e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080d2870 con 0x7f2a080d4760 2024-03-06T07:34:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0ec0e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a080d4bf0 con 0x7f2a080b4170 2024-03-06T07:34:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0d1ab640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2a080d0a80 0x7f2a080cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35614/0 (socket says 172.21.15.122:35614) 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.841+0000 7f2a0d1ab640 1 -- 172.21.15.122:0/577900840 learned_addr learned my addr 172.21.15.122:0/577900840 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725222921 0 0) 0x7f2a080d25d0 con 0x7f2a080d0a80 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f4003520 con 0x7f2a080d0a80 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/577900840 >> v2:172.21.15.169:3300/0 conn(0x7f2a080b4170 0x7f2a080b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638612974 0 0) 0x7f29f4003520 con 0x7f2a080d0a80 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 >> v2:172.21.15.169:3300/0 conn(0x7f2a080b4170 msgr2=0x7f2a080b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 --2- 172.21.15.122:0/577900840 >> v2:172.21.15.169:3300/0 conn(0x7f2a080b4170 0x7f2a080b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 >> v1:172.21.15.169:6789/0 conn(0x7f2a080d4760 legacy=0x7f2a080d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a080d2b50 con 0x7f2a080d0a80 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f29f8001670 con 0x7f2a080d0a80 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 >> v1:172.21.15.122:6789/0 conn(0x7f2a080d0a80 legacy=0x7f2a080cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f29f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f4004800 0x7f29f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f40051c0 0x7f29f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080d4bf0 con 0x7f29f40039b0 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f400c110 con 0x7f29f40051c0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f400c310 con 0x7f29f4004800 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f29f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a07fff640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f4004800 0x7f29f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0d1ab640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f40051c0 0x7f29f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f4004800 msgr2=0x7f29f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f29f4004800 0x7f29f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f40051c0 msgr2=0x7f29f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f40051c0 0x7f29f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/577900840 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f4003520 con 0x7f29f40039b0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0d1ab640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29f40051c0 0x7f29f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.842+0000 7f2a0d1ab640 1 -- 172.21.15.122:0/577900840 reap_dead start 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.843+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f29f4003dd0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f2a00007360 tx=0x7f29f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55b545d8894f255 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.843+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00039070 con 0x7f29f40039b0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.843+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0002ccd0 con 0x7f29f40039b0 2024-03-06T07:34:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.843+0000 7f2a077fe640 1 -- 172.21.15.122:0/577900840 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00009be0 con 0x7f29f40039b0 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/577900840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 msgr2=0x7f29f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f29f4003dd0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f2a00007360 tx=0x7f29f4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/577900840 shutdown_connections 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/577900840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f29f4003dd0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/577900840 >> 172.21.15.122:0/577900840 conn(0x7f2a080d3c40 msgr2=0x7f2a080d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/577900840 shutdown_connections 2024-03-06T07:34:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.844+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/577900840 wait complete. 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.845+0000 7f2a0ec0e640 1 Processor -- start 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.845+0000 7f2a0ec0e640 1 -- start start 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0ec0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f2a0815a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0ec0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2a081589a0 0x7f2a080c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0ec0e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a0815df50 0x7f2a0815e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0ec0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080b4dd0 con 0x7f29f40039b0 2024-03-06T07:34:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0ec0e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080ce9e0 con 0x7f2a0815df50 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0ec0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080b4f50 con 0x7f2a081589a0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f2a0815a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f2a0815a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37414/0 (socket says 172.21.15.122:37414) 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/2347890696 learned_addr learned my addr 172.21.15.122:0/2347890696 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2a081589a0 msgr2=0x7f2a080c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a07fff640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2a081589a0 0x7f2a080c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2a081589a0 0x7f2a080c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a0815df50 msgr2=0x7f2a0815e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a0815df50 0x7f2a0815e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 -- 172.21.15.122:0/2347890696 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0815aec0 con 0x7f29f40039b0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.846+0000 7f2a0c9aa640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f2a0815a570 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f2a00007480 tx=0x7f2a00004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.847+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00042030 con 0x7f29f40039b0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.847+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0003a040 con 0x7f29f40039b0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.847+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00039070 con 0x7f29f40039b0 2024-03-06T07:34:11.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.847+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0815b1a0 con 0x7f29f40039b0 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.847+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0815aab0 con 0x7f29f40039b0 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.848+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0804f330 con 0x7f29f40039b0 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.848+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a00045070 con 0x7f29f40039b0 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.848+0000 7f2a057fa640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29e806b4d0 0x7f29e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.849+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2a000ab740 con 0x7f29f40039b0 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.849+0000 7f2a07fff640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29e806b4d0 0x7f29e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.849+0000 7f2a07fff640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29e806b4d0 0x7f29e806d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f29f0000c00 tx=0x7f29f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:11.853+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a0007a630 con 0x7f29f40039b0 2024-03-06T07:34:12.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.067+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a080733b0 con 0x7f29f40039b0 2024-03-06T07:34:12.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.068+0000 7f2a057fa640 1 -- 172.21.15.122:0/2347890696 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2a00079f30 con 0x7f29f40039b0 2024-03-06T07:34:12.070 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:12.070 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:12.070 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:12.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29e806b4d0 msgr2=0x7f29e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29e806b4d0 0x7f29e806d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f29f0000c00 tx=0x7f29f0009040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 msgr2=0x7f2a0815a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.072 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f2a0815a570 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f2a00007480 tx=0x7f2a00004ab0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 shutdown_connections 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f29e806b4d0 0x7f29e806d990 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a0815df50 0x7f2a0815e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2a081589a0 0x7f2a080c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 --2- 172.21.15.122:0/2347890696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29f40039b0 0x7f2a0815a570 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 >> 172.21.15.122:0/2347890696 conn(0x7f2a080d3c40 msgr2=0x7f2a080b1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 shutdown_connections 2024-03-06T07:34:12.073 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:12.070+0000 7f2a0ec0e640 1 -- 172.21.15.122:0/2347890696 wait complete. 2024-03-06T07:34:12.284 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:12.284 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:13.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:13.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.420+0000 7f0caf9d7640 1 Processor -- start 2024-03-06T07:34:13.422 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.420+0000 7f0caf9d7640 1 -- start start 2024-03-06T07:34:13.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0caf9d7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ca80d0df0 0x7f0ca80c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0caf9d7640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0ca80d47f0 0x7f0ca80b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0caf9d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca8162b20 con 0x7f0ca80b51d0 2024-03-06T07:34:13.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0caf9d7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca80b6b20 con 0x7f0ca80d0df0 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0caf9d7640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca80cf400 con 0x7f0ca80d47f0 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ca80d0df0 0x7f0ca80c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0ca80d0df0 0x7f0ca80c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37420/0 (socket says 172.21.15.122:37420) 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cad773640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0ca80b51d0 0x7f0ca80d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:35624/0 (socket says 172.21.15.122:35624) 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 -- 172.21.15.122:0/2463748107 learned_addr learned my addr 172.21.15.122:0/2463748107 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 -- 172.21.15.122:0/2463748107 >> v2:172.21.15.169:3300/0 conn(0x7f0ca80d47f0 msgr2=0x7f0ca80b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cadf74640 1 --2- 172.21.15.122:0/2463748107 >> v2:172.21.15.169:3300/0 conn(0x7f0ca80d47f0 0x7f0ca80b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 --2- 172.21.15.122:0/2463748107 >> v2:172.21.15.169:3300/0 conn(0x7f0ca80d47f0 0x7f0ca80b6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0c9ffff640 1 -- 172.21.15.122:0/2463748107 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722283637 0 0) 0x7f0ca8162b20 con 0x7f0ca80b51d0 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 -- 172.21.15.122:0/2463748107 >> v1:172.21.15.122:6789/0 conn(0x7f0ca80b51d0 legacy=0x7f0ca80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.422+0000 7f0cacf72640 1 -- 172.21.15.122:0/2463748107 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca80d4c00 con 0x7f0ca80d0df0 2024-03-06T07:34:13.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.423+0000 7f0cacf72640 1 --2- 172.21.15.122:0/2463748107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c6cb0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80ce530 tx=0x7f0ca402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cf8b68e455bec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.423+0000 7f0c9ffff640 1 -- 172.21.15.122:0/2463748107 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4035020 con 0x7f0ca80d0df0 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.423+0000 7f0c9ffff640 1 -- 172.21.15.122:0/2463748107 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca4031870 con 0x7f0ca80d0df0 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.423+0000 7f0c9ffff640 1 -- 172.21.15.122:0/2463748107 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4031c40 con 0x7f0ca80d0df0 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 -- 172.21.15.122:0/2463748107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d0df0 msgr2=0x7f0ca80c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/2463748107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c6cb0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80ce530 tx=0x7f0ca402b910 comp rx=0 tx=0).stop 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 -- 172.21.15.122:0/2463748107 shutdown_connections 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/2463748107 >> v2:172.21.15.169:3300/0 conn(0x7f0ca80d47f0 0x7f0ca80b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/2463748107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c6cb0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 -- 172.21.15.122:0/2463748107 >> 172.21.15.122:0/2463748107 conn(0x7f0ca80d3cd0 msgr2=0x7f0ca80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 -- 172.21.15.122:0/2463748107 shutdown_connections 2024-03-06T07:34:13.427 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.424+0000 7f0caf9d7640 1 -- 172.21.15.122:0/2463748107 wait complete. 2024-03-06T07:34:13.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.426+0000 7f0caf9d7640 1 Processor -- start 2024-03-06T07:34:13.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.426+0000 7f0caf9d7640 1 -- start start 2024-03-06T07:34:13.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0caf9d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ca80b51d0 0x7f0ca8165b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0caf9d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0caf9d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 0x7f0ca815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0caf9d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca80cf980 con 0x7f0ca80d47f0 2024-03-06T07:34:13.430 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0caf9d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca80cf6a0 con 0x7f0ca80d0df0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0caf9d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca80cfc60 con 0x7f0ca80b51d0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 0x7f0ca815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 0x7f0ca815f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37432/0 (socket says 172.21.15.122:37432) 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 -- 172.21.15.122:0/1328570165 learned_addr learned my addr 172.21.15.122:0/1328570165 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cacf72640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cad773640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ca80b51d0 0x7f0ca8165b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 -- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ca80b51d0 msgr2=0x7f0ca8165b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ca80b51d0 0x7f0ca8165b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 -- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ca80d0df0 msgr2=0x7f0ca80c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.427+0000 7f0cadf74640 1 -- 172.21.15.122:0/1328570165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca815f8d0 con 0x7f0ca80d47f0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.428+0000 7f0cadf74640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 0x7f0ca815f390 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000cc30 tx=0x7f0ca000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.428+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca000ce40 con 0x7f0ca80d47f0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.428+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0003990 con 0x7f0ca80d47f0 2024-03-06T07:34:13.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.428+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca81fb5b0 con 0x7f0ca80d47f0 2024-03-06T07:34:13.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.428+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0003d40 con 0x7f0ca80d47f0 2024-03-06T07:34:13.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.428+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca81fbb70 con 0x7f0ca80d47f0 2024-03-06T07:34:13.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.429+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca804f330 con 0x7f0ca80d47f0 2024-03-06T07:34:13.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.430+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ca0003ee0 con 0x7f0ca80d47f0 2024-03-06T07:34:13.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.430+0000 7f0c9dffb640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c8c06b500 0x7f0c8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.430+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0ca008a580 con 0x7f0ca80d47f0 2024-03-06T07:34:13.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.430+0000 7f0cad773640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c8c06b500 0x7f0c8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.431+0000 7f0cad773640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c8c06b500 0x7f0c8c06d9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0c98000f60 tx=0x7f0c98007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.436+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ca815f8d0 con 0x7f0ca80d47f0 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.712+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ca81625c0 con 0x7f0ca80d47f0 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.712+0000 7f0c9dffb640 1 -- 172.21.15.122:0/1328570165 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0ca0014030 con 0x7f0ca80d47f0 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c8c06b500 msgr2=0x7f0c8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c8c06b500 0x7f0c8c06d9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0c98000f60 tx=0x7f0c98007d50 comp rx=0 tx=0).stop 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 msgr2=0x7f0ca815f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 0x7f0ca815f390 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000cc30 tx=0x7f0ca000a5c0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 shutdown_connections 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c8c06b500 0x7f0c8c06d9c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ca80d47f0 0x7f0ca815f390 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ca80d0df0 0x7f0ca80c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 --2- 172.21.15.122:0/1328570165 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0ca80b51d0 0x7f0ca8165b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 >> 172.21.15.122:0/1328570165 conn(0x7f0ca80d3cd0 msgr2=0x7f0ca80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 shutdown_connections 2024-03-06T07:34:13.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:13.714+0000 7f0caf9d7640 1 -- 172.21.15.122:0/1328570165 wait complete. 2024-03-06T07:34:13.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.770682+0000', 'modified': '2024-03-06T07:32:37.113279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9321, 'mds_1': 9318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9318': {'gid': 9318, 'name': 'c', 'rank': 1, 'incarnation': 92, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.122:6835/138214324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 138214324}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 138214324}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:34.001145+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9321': {'gid': 9321, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6837/1416910179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 1416910179}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 1416910179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9321, 'qdb_cluster': [9321]} 2024-03-06T07:34:13.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9318:up:stopping 2024-03-06T07:34:14.929 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:30:05.770682+0000", "damaged": [], "data_pools": [ 21 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9318": { "addr": "172.21.15.122:6835/138214324", "addrs": { "addrvec": [ { "addr": "172.21.15.122:6834", "nonce": 138214324, "type": "v2" }, { "addr": "172.21.15.122:6835", "nonce": 138214324, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540701547738038271, "flags": 0, "gid": 9318, "incarnation": 92, "join_fscid": -1, "laggy_since": "2024-03-06T07:31:34.001145+0000", "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_9321": { "addr": "172.21.15.122:6837/1416910179", "addrs": { "addrvec": [ { "addr": "172.21.15.122:6836", "nonce": 1416910179, "type": "v2" }, { "addr": "172.21.15.122:6837", "nonce": 1416910179, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9321, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 20, "modified": "2024-03-06T07:32:37.113279+0000", "qdb_cluster": [ 9321 ], "qdb_leader": 9321, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9321, "mds_1": 9318 } } } ], "standbys": [] } 2024-03-06T07:34:14.930 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:34:14.930 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:14.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:15.018 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:34:15.018 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:34:15.018 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:15.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:15.033 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:34:15.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.033 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:15.088 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-06T07:34:15.088 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:34:15.089 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:15.107 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:15.107 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.107 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:34:15.164 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:34:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:15.165 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:34:15.165 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:15.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:15.181 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:34:15.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.181 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:34:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:15.235 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:34:15.235 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:34:15.235 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:15.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:15.253 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:34:15.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.253 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:34:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:15.311 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:34:15.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:15.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.410+0000 7f025333e640 1 Processor -- start 2024-03-06T07:34:15.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f025333e640 1 -- start start 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f025333e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f024c15dfe0 0x7f024c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f025333e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f025333e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0b3430 con 0x7f024c0d27f0 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f025333e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024c0d2c50 con 0x7f024c15dfe0 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f025333e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0b31e0 con 0x7f024c0cf830 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f02510da640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f024c15dfe0 0x7f024c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f02518db640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f024c0d27f0 0x7f024c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51052/0 (socket says 172.21.15.122:51052) 2024-03-06T07:34:15.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.411+0000 7f02518db640 1 -- 172.21.15.122:0/2005826574 learned_addr learned my addr 172.21.15.122:0/2005826574 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02508d9640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02510da640 1 -- 172.21.15.122:0/2005826574 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 msgr2=0x7f024c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02510da640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02510da640 1 -- 172.21.15.122:0/2005826574 >> v1:172.21.15.122:6789/0 conn(0x7f024c0d27f0 legacy=0x7f024c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f0243fff640 1 -- 172.21.15.122:0/2005826574 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369393745 0 0) 0x7f024c0b3430 con 0x7f024c0d27f0 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02510da640 1 -- 172.21.15.122:0/2005826574 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0b3710 con 0x7f024c15dfe0 2024-03-06T07:34:15.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02508d9640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:15.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.412+0000 7f02510da640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c0cf2f0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f02480097a0 tx=0x7f024802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596400fb036a1bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f0243fff640 1 -- 172.21.15.122:0/2005826574 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248035020 con 0x7f024c15dfe0 2024-03-06T07:34:15.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f0243fff640 1 -- 172.21.15.122:0/2005826574 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0248031830 con 0x7f024c15dfe0 2024-03-06T07:34:15.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f0243fff640 1 -- 172.21.15.122:0/2005826574 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248031c00 con 0x7f024c15dfe0 2024-03-06T07:34:15.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 -- 172.21.15.122:0/2005826574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 msgr2=0x7f024c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c0cf2f0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f02480097a0 tx=0x7f024802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 -- 172.21.15.122:0/2005826574 shutdown_connections 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 --2- 172.21.15.122:0/2005826574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c0cf2f0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 -- 172.21.15.122:0/2005826574 >> 172.21.15.122:0/2005826574 conn(0x7f024c0d3cd0 msgr2=0x7f024c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 -- 172.21.15.122:0/2005826574 shutdown_connections 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.413+0000 7f025333e640 1 -- 172.21.15.122:0/2005826574 wait complete. 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.415+0000 7f025333e640 1 Processor -- start 2024-03-06T07:34:15.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.415+0000 7f025333e640 1 -- start start 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f025333e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c15c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f025333e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0d27f0 0x7f024c1f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f025333e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c1f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f025333e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0d1c70 con 0x7f024c15dfe0 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f025333e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0b3d60 con 0x7f024c0d27f0 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f025333e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0d1df0 con 0x7f024c0cf830 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f02518db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c1f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f02518db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c1f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60424/0 (socket says 172.21.15.122:60424) 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.416+0000 7f02518db640 1 -- 172.21.15.122:0/2488023956 learned_addr learned my addr 172.21.15.122:0/2488023956 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02510da640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02508d9640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0d27f0 0x7f024c1f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02518db640 1 -- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 msgr2=0x7f024c15c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02518db640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c15c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02518db640 1 -- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0d27f0 msgr2=0x7f024c1f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02518db640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0d27f0 0x7f024c1f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02518db640 1 -- 172.21.15.122:0/2488023956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0cae40 con 0x7f024c15dfe0 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f02518db640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c1f7500 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f024400c7d0 tx=0x7f024400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244018070 con 0x7f024c15dfe0 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024400a640 con 0x7f024c15dfe0 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c0cb0b0 con 0x7f024c15dfe0 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024400aa70 con 0x7f024c15dfe0 2024-03-06T07:34:15.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.417+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c0cb670 con 0x7f024c15dfe0 2024-03-06T07:34:15.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.418+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c0cb870 con 0x7f024c15dfe0 2024-03-06T07:34:15.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.419+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f024401b070 con 0x7f024c15dfe0 2024-03-06T07:34:15.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.419+0000 7f0241ffb640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f023006b4d0 0x7f023006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.419+0000 7f02510da640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f023006b4d0 0x7f023006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.419+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0244089820 con 0x7f024c15dfe0 2024-03-06T07:34:15.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.420+0000 7f02510da640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f023006b4d0 0x7f023006d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f0248036000 tx=0x7f024803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.424+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f024c0cb870 con 0x7f024c15dfe0 2024-03-06T07:34:15.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.640+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f024c04f330 con 0x7f024c15dfe0 2024-03-06T07:34:15.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.641+0000 7f0241ffb640 1 -- 172.21.15.122:0/2488023956 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0244058680 con 0x7f024c15dfe0 2024-03-06T07:34:15.642 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:15.642 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:15.642 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f023006b4d0 msgr2=0x7f023006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f023006b4d0 0x7f023006d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f0248036000 tx=0x7f024803c040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 msgr2=0x7f024c1f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c1f7500 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f024400c7d0 tx=0x7f024400cc90 comp rx=0 tx=0).stop 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 shutdown_connections 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f023006b4d0 0x7f023006d990 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f024c15dfe0 0x7f024c1f7500 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0d27f0 0x7f024c1f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 --2- 172.21.15.122:0/2488023956 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f024c0cf830 0x7f024c15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 >> 172.21.15.122:0/2488023956 conn(0x7f024c0d3cd0 msgr2=0x7f024c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 shutdown_connections 2024-03-06T07:34:15.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.643+0000 7f025333e640 1 -- 172.21.15.122:0/2488023956 wait complete. 2024-03-06T07:34:15.655 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:15.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.755+0000 7f13dfa47640 1 Processor -- start 2024-03-06T07:34:15.757 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.756+0000 7f13dfa47640 1 -- start start 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.756+0000 7f13dfa47640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13d80cfff0 0x7f13d80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.756+0000 7f13dfa47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cc8b0 0x7f13d80ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.756+0000 7f13dfa47640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d8161980 con 0x7f13d815bb20 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.756+0000 7f13dfa47640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80d2500 con 0x7f13d80cfff0 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.756+0000 7f13dfa47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d8161c60 con 0x7f13d80cc8b0 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13d80cfff0 0x7f13d80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f13d80cfff0 0x7f13d80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60432/0 (socket says 172.21.15.122:60432) 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 -- 172.21.15.122:0/1556624175 learned_addr learned my addr 172.21.15.122:0/1556624175 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13ddfe4640 1 --2- 172.21.15.122:0/1556624175 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cc8b0 0x7f13d80ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 -- 172.21.15.122:0/1556624175 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cc8b0 msgr2=0x7f13d80ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 --2- 172.21.15.122:0/1556624175 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cc8b0 0x7f13d80ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 -- 172.21.15.122:0/1556624175 >> v1:172.21.15.169:6789/0 conn(0x7f13d815bb20 legacy=0x7f13d80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.757+0000 7f13dcfe2640 1 -- 172.21.15.122:0/1556624175 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d8161f40 con 0x7f13d80cfff0 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dcfe2640 1 --2- 172.21.15.122:0/1556624175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cfff0 0x7f13d80d04a0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009870 tx=0x7f13c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b807141ed164369 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13cffff640 1 -- 172.21.15.122:0/1556624175 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8034070 con 0x7f13d80cfff0 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13cffff640 1 -- 172.21.15.122:0/1556624175 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c8030840 con 0x7f13d80cfff0 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13cffff640 1 -- 172.21.15.122:0/1556624175 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8030c10 con 0x7f13d80cfff0 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 -- 172.21.15.122:0/1556624175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cfff0 msgr2=0x7f13d80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 --2- 172.21.15.122:0/1556624175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cfff0 0x7f13d80d04a0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009870 tx=0x7f13c802b980 comp rx=0 tx=0).stop 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 -- 172.21.15.122:0/1556624175 shutdown_connections 2024-03-06T07:34:15.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 --2- 172.21.15.122:0/1556624175 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cc8b0 0x7f13d80ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 --2- 172.21.15.122:0/1556624175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cfff0 0x7f13d80d04a0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 -- 172.21.15.122:0/1556624175 >> 172.21.15.122:0/1556624175 conn(0x7f13d809a9f0 msgr2=0x7f13d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 -- 172.21.15.122:0/1556624175 shutdown_connections 2024-03-06T07:34:15.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.758+0000 7f13dfa47640 1 -- 172.21.15.122:0/1556624175 wait complete. 2024-03-06T07:34:15.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.760+0000 7f13dfa47640 1 Processor -- start 2024-03-06T07:34:15.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.760+0000 7f13dfa47640 1 -- start start 2024-03-06T07:34:15.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dfa47640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 0x7f13d8152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dfa47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cfff0 0x7f13d8160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dfa47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d815bb20 0x7f13d806d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dfa47640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80d1980 con 0x7f13d80cc8b0 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dfa47640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d8162500 con 0x7f13d815bb20 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dfa47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80d1b00 con 0x7f13d80cfff0 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 0x7f13d8152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 0x7f13d8152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60448/0 (socket says 172.21.15.122:60448) 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 -- 172.21.15.122:0/3766301605 learned_addr learned my addr 172.21.15.122:0/3766301605 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dcfe2640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cfff0 0x7f13d8160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13ddfe4640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d815bb20 0x7f13d806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 -- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cfff0 msgr2=0x7f13d8160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cfff0 0x7f13d8160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 -- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d815bb20 msgr2=0x7f13d806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d815bb20 0x7f13d806d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.764 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13dd7e3640 1 -- 172.21.15.122:0/3766301605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80c6de0 con 0x7f13d80cc8b0 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.761+0000 7f13ddfe4640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d815bb20 0x7f13d806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.762+0000 7f13dd7e3640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 0x7f13d8152060 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f13d400c3f0 tx=0x7f13d400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.762+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d40052d0 con 0x7f13d80cc8b0 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.762+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4005490 con 0x7f13d80cc8b0 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.762+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d8160950 con 0x7f13d80cc8b0 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.762+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4007260 con 0x7f13d80cc8b0 2024-03-06T07:34:15.765 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.762+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d8160f10 con 0x7f13d80cc8b0 2024-03-06T07:34:15.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.763+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d804f330 con 0x7f13d80cc8b0 2024-03-06T07:34:15.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.763+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13d4007400 con 0x7f13d80cc8b0 2024-03-06T07:34:15.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.764+0000 7f13cdffb640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13bc06b4d0 0x7f13bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.764+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f13d40895c0 con 0x7f13d80cc8b0 2024-03-06T07:34:15.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.764+0000 7f13dcfe2640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13bc06b4d0 0x7f13bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.766 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.764+0000 7f13dcfe2640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13bc06b4d0 0x7f13bc06d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f13c8036000 tx=0x7f13c803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.768+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13d4057e60 con 0x7f13d80cc8b0 2024-03-06T07:34:15.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.959+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13d80b7650 con 0x7f13d80cc8b0 2024-03-06T07:34:15.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.960+0000 7f13cdffb640 1 -- 172.21.15.122:0/3766301605 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f13d405ce20 con 0x7f13d80cc8b0 2024-03-06T07:34:15.961 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:15.961 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13bc06b4d0 msgr2=0x7f13bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13bc06b4d0 0x7f13bc06d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f13c8036000 tx=0x7f13c803b040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 msgr2=0x7f13d8152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 0x7f13d8152060 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f13d400c3f0 tx=0x7f13d400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 shutdown_connections 2024-03-06T07:34:15.963 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f13bc06b4d0 0x7f13bc06d990 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13d815bb20 0x7f13d806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f13d80cfff0 0x7f13d8160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 --2- 172.21.15.122:0/3766301605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13d80cc8b0 0x7f13d8152060 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 >> 172.21.15.122:0/3766301605 conn(0x7f13d809a9f0 msgr2=0x7f13d80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 shutdown_connections 2024-03-06T07:34:15.964 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:15.962+0000 7f13dfa47640 1 -- 172.21.15.122:0/3766301605 wait complete. 2024-03-06T07:34:15.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.370683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.987708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.745253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:15.975 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:16.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.080+0000 7fc177116640 1 Processor -- start 2024-03-06T07:34:16.081 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.080+0000 7fc177116640 1 -- start start 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc177116640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc1701614d0 0x7fc1700d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc177116640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1701618b0 con 0x7fc1700d24b0 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc177116640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1700cf400 con 0x7fc1700d1910 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc177116640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700cf6a0 con 0x7fc1701614d0 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc174eb2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc1701614d0 0x7fc1700d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc1756b3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc1700d24b0 0x7fc1700b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51054/0 (socket says 172.21.15.122:51054) 2024-03-06T07:34:16.082 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc1756b3640 1 -- 172.21.15.122:0/510372325 learned_addr learned my addr 172.21.15.122:0/510372325 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:16.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc174eb2640 1 -- 172.21.15.122:0/510372325 >> v1:172.21.15.169:6789/0 conn(0x7fc1700d1910 legacy=0x7fc1700d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc174eb2640 1 -- 172.21.15.122:0/510372325 >> v1:172.21.15.122:6789/0 conn(0x7fc1700d24b0 legacy=0x7fc1700b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc16f7fe640 1 -- 172.21.15.122:0/510372325 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330166408 0 0) 0x7fc1701618b0 con 0x7fc1700d24b0 2024-03-06T07:34:16.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.081+0000 7fc174eb2640 1 -- 172.21.15.122:0/510372325 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700cf980 con 0x7fc1701614d0 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.082+0000 7fc174eb2640 1 --2- 172.21.15.122:0/510372325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1701614d0 0x7fc1700d4860 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fc17008c470 tx=0x7fc16402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fd2d3a09577ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.082+0000 7fc16f7fe640 1 -- 172.21.15.122:0/510372325 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164035020 con 0x7fc1701614d0 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.082+0000 7fc16f7fe640 1 -- 172.21.15.122:0/510372325 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc164031800 con 0x7fc1701614d0 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.082+0000 7fc16f7fe640 1 -- 172.21.15.122:0/510372325 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164031bd0 con 0x7fc1701614d0 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 -- 172.21.15.122:0/510372325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1701614d0 msgr2=0x7fc1700d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 --2- 172.21.15.122:0/510372325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1701614d0 0x7fc1700d4860 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fc17008c470 tx=0x7fc16402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 -- 172.21.15.122:0/510372325 shutdown_connections 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 --2- 172.21.15.122:0/510372325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1701614d0 0x7fc1700d4860 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 -- 172.21.15.122:0/510372325 >> 172.21.15.122:0/510372325 conn(0x7fc1700d3d40 msgr2=0x7fc1700d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 -- 172.21.15.122:0/510372325 shutdown_connections 2024-03-06T07:34:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.083+0000 7fc177116640 1 -- 172.21.15.122:0/510372325 wait complete. 2024-03-06T07:34:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.084+0000 7fc177116640 1 Processor -- start 2024-03-06T07:34:16.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.084+0000 7fc177116640 1 -- start start 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1700d1910 0x7fc1700c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 0x7fc170160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1701614d0 0x7fc17015f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b1540 con 0x7fc1700d24b0 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b0910 con 0x7fc1701614d0 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b16c0 con 0x7fc1700d1910 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 0x7fc170160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 0x7fc170160190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60468/0 (socket says 172.21.15.122:60468) 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 -- 172.21.15.122:0/3205288367 learned_addr learned my addr 172.21.15.122:0/3205288367 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc174eb2640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1700d1910 0x7fc1700c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc1756b3640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1701614d0 0x7fc17015f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 -- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1700d1910 msgr2=0x7fc1700c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1700d1910 0x7fc1700c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 -- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1701614d0 msgr2=0x7fc17015f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1701614d0 0x7fc17015f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 -- 172.21.15.122:0/3205288367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17015ff70 con 0x7fc1700d24b0 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16ffff640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 0x7fc170160190 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c430 tx=0x7fc15c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c011790 con 0x7fc1700d24b0 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c017070 con 0x7fc1700d24b0 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1700ccd60 con 0x7fc1700d24b0 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c011bb0 con 0x7fc1700d24b0 2024-03-06T07:34:16.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.085+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1700b6fe0 con 0x7fc1700d24b0 2024-03-06T07:34:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.086+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170068ff0 con 0x7fc1700d24b0 2024-03-06T07:34:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.087+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc15c01c460 con 0x7fc1700d24b0 2024-03-06T07:34:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.087+0000 7fc16d7fa640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc15406b4d0 0x7fc15406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.087+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc15c08ab40 con 0x7fc1700d24b0 2024-03-06T07:34:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.088+0000 7fc174eb2640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc15406b4d0 0x7fc15406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.088+0000 7fc174eb2640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc15406b4d0 0x7fc15406d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc164036000 tx=0x7fc16403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.091+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc17015ff70 con 0x7fc1700d24b0 2024-03-06T07:34:16.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.290+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1700d09e0 con 0x7fc1700d24b0 2024-03-06T07:34:16.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.290+0000 7fc16d7fa640 1 -- 172.21.15.122:0/3205288367 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc15c059a80 con 0x7fc1700d24b0 2024-03-06T07:34:16.291 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.292+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc15406b4d0 msgr2=0x7fc15406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.292+0000 7fc177116640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc15406b4d0 0x7fc15406d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc164036000 tx=0x7fc16403c040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.292+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 msgr2=0x7fc170160190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.292+0000 7fc177116640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 0x7fc170160190 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c430 tx=0x7fc15c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 shutdown_connections 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc15406b4d0 0x7fc15406d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1701614d0 0x7fc17015f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1700d24b0 0x7fc170160190 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 --2- 172.21.15.122:0/3205288367 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1700d1910 0x7fc1700c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 >> 172.21.15.122:0/3205288367 conn(0x7fc1700d3d40 msgr2=0x7fc1700cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 shutdown_connections 2024-03-06T07:34:16.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.293+0000 7fc177116640 1 -- 172.21.15.122:0/3205288367 wait complete. 2024-03-06T07:34:16.305 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 Processor -- start 2024-03-06T07:34:16.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 -- start start 2024-03-06T07:34:16.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 0x7f06680d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f06680d1910 0x7f06680d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06680d2740 con 0x7f06680d22b0 2024-03-06T07:34:16.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06681618b0 con 0x7f06680d1910 2024-03-06T07:34:16.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066eb43640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06680cf400 con 0x7f06681614d0 2024-03-06T07:34:16.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.408+0000 7f066d0e0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f06680d22b0 0x7f06680d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51068/0 (socket says 172.21.15.122:51068) 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f066d0e0640 1 -- 172.21.15.122:0/2834342719 learned_addr learned my addr 172.21.15.122:0/2834342719 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f066c8df640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 0x7f06680d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f0667fff640 1 --2- 172.21.15.122:0/2834342719 >> v2:172.21.15.169:3300/0 conn(0x7f06680d1910 0x7f06680d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541278430 0 0) 0x7f0658001110 con 0x7f06680d22b0 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0650003520 con 0x7f06680d22b0 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 108839343 0 0) 0x7f0650003520 con 0x7f06680d22b0 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 msgr2=0x7f06680d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 0x7f06680d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 >> v2:172.21.15.169:3300/0 conn(0x7f06680d1910 msgr2=0x7f06680d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 --2- 172.21.15.122:0/2834342719 >> v2:172.21.15.169:3300/0 conn(0x7f06680d1910 0x7f06680d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06680cf6a0 con 0x7f06680d22b0 2024-03-06T07:34:16.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f0658001840 con 0x7f06680d22b0 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 >> v1:172.21.15.122:6789/0 conn(0x7f06680d22b0 legacy=0x7f06680d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 0x7f06500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0650003f80 0x7f0650004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f06500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681618b0 con 0x7f0650004d40 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06680cf400 con 0x7f0650003f80 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065000b9c0 con 0x7f06681614d0 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f066d0e0640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f06500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f066c8df640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 0x7f06500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.409+0000 7f0667fff640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0650003f80 0x7f0650004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f066d0e0640 1 -- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 msgr2=0x7f06500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f066d0e0640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06681614d0 0x7f06500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f066d0e0640 1 -- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0650003f80 msgr2=0x7f0650004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f066d0e0640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0650003f80 0x7f0650004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f066d0e0640 1 -- 172.21.15.122:0/2834342719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650003520 con 0x7f0650004d40 2024-03-06T07:34:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f0667fff640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0650003f80 0x7f0650004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f066d0e0640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f06500051c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f065800c320 tx=0x7f0650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ffbdab67fd002b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658007e60 con 0x7f0650004d40 2024-03-06T07:34:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0658030070 con 0x7f0650004d40 2024-03-06T07:34:16.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.410+0000 7f06677fe640 1 -- 172.21.15.122:0/2834342719 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065802ce30 con 0x7f0650004d40 2024-03-06T07:34:16.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066eb43640 1 -- 172.21.15.122:0/2834342719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 msgr2=0x7f06500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066eb43640 1 --2- 172.21.15.122:0/2834342719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f06500051c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f065800c320 tx=0x7f0650005830 comp rx=0 tx=0).stop 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066d0e0640 1 -- 172.21.15.122:0/2834342719 reap_dead start 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066eb43640 1 -- 172.21.15.122:0/2834342719 shutdown_connections 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066eb43640 1 -- 172.21.15.122:0/2834342719 >> 172.21.15.122:0/2834342719 conn(0x7f06680d3d40 msgr2=0x7f06680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066eb43640 1 -- 172.21.15.122:0/2834342719 shutdown_connections 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.411+0000 7f066eb43640 1 -- 172.21.15.122:0/2834342719 wait complete. 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.412+0000 7f066eb43640 1 Processor -- start 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.412+0000 7f066eb43640 1 -- start start 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066eb43640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f0668153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066eb43640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06680b7600 0x7f06680b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066eb43640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06680b83b0 0x7f06680b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066eb43640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06680b0910 con 0x7f0650004d40 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066eb43640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06680cfc60 con 0x7f06680b83b0 2024-03-06T07:34:16.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066eb43640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06680b1e40 con 0x7f06680b7600 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f0668153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f0668153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60476/0 (socket says 172.21.15.122:60476) 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 -- 172.21.15.122:0/2387796240 learned_addr learned my addr 172.21.15.122:0/2387796240 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f0667fff640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06680b7600 0x7f06680b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066d0e0640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06680b83b0 0x7f06680b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 -- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06680b7600 msgr2=0x7f06680b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06680b7600 0x7f06680b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 -- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06680b83b0 msgr2=0x7f06680b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06680b83b0 0x7f06680b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.413+0000 7f066c8df640 1 -- 172.21.15.122:0/2387796240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06680b6560 con 0x7f0650004d40 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.414+0000 7f066c8df640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f0668153260 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f0660011e30 tx=0x7f066000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.414+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660015630 con 0x7f0650004d40 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.414+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06600157d0 con 0x7f0650004d40 2024-03-06T07:34:16.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.414+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06680b6840 con 0x7f0650004d40 2024-03-06T07:34:16.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.414+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660015ba0 con 0x7f0650004d40 2024-03-06T07:34:16.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.414+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06680b6e00 con 0x7f0650004d40 2024-03-06T07:34:16.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.415+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06680b8790 con 0x7f0650004d40 2024-03-06T07:34:16.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.416+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f066001d400 con 0x7f0650004d40 2024-03-06T07:34:16.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.416+0000 7f06657fa640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f065406b500 0x7f065406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.416+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f066008b250 con 0x7f0650004d40 2024-03-06T07:34:16.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.416+0000 7f0667fff640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f065406b500 0x7f065406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.420 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.416+0000 7f0667fff640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f065406b500 0x7f065406d9c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f06680ca200 tx=0x7f065c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.425 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.424+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0660019030 con 0x7f0650004d40 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.601+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06680c7400 con 0x7f0650004d40 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.602+0000 7f06657fa640 1 -- 172.21.15.122:0/2387796240 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0660010040 con 0x7f0650004d40 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 32 2024-03-06T07:34:16.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f065406b500 msgr2=0x7f065406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f065406b500 0x7f065406d9c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f06680ca200 tx=0x7f065c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 msgr2=0x7f0668153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f0668153260 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f0660011e30 tx=0x7f066000f580 comp rx=0 tx=0).stop 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 shutdown_connections 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f065406b500 0x7f065406d9c0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06680b83b0 0x7f06680b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f06680b7600 0x7f06680b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 --2- 172.21.15.122:0/2387796240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0650004d40 0x7f0668153260 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 >> 172.21.15.122:0/2387796240 conn(0x7f06680d3d40 msgr2=0x7f06680b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 shutdown_connections 2024-03-06T07:34:16.606 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.604+0000 7f066eb43640 1 -- 172.21.15.122:0/2387796240 wait complete. 2024-03-06T07:34:16.616 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:16.718 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.717+0000 7fef2d0df640 1 Processor -- start 2024-03-06T07:34:16.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2d0df640 1 -- start start 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2d0df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d10c0 0x7fef280d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2d0df640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fef280d14a0 0x7fef2815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2d0df640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280cfa50 con 0x7fef2815eb00 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2d0df640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2815c050 con 0x7fef280d14a0 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2d0df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef280d0060 con 0x7fef280d10c0 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef1ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fef280d14a0 0x7fef2815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2759e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fef2815eb00 0x7fef280cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51078/0 (socket says 172.21.15.122:51078) 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef1ffff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fef280d14a0 0x7fef2815e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60480/0 (socket says 172.21.15.122:60480) 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.718+0000 7fef2759e640 1 -- 172.21.15.122:0/397067167 learned_addr learned my addr 172.21.15.122:0/397067167 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef26d9d640 1 --2- 172.21.15.122:0/397067167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d10c0 0x7fef280d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef1ffff640 1 -- 172.21.15.122:0/397067167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d10c0 msgr2=0x7fef280d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef1ffff640 1 --2- 172.21.15.122:0/397067167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d10c0 0x7fef280d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef1ffff640 1 -- 172.21.15.122:0/397067167 >> v1:172.21.15.122:6789/0 conn(0x7fef2815eb00 legacy=0x7fef280cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef1ffff640 1 -- 172.21.15.122:0/397067167 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280d0a30 con 0x7fef280d14a0 2024-03-06T07:34:16.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef2659c640 1 -- 172.21.15.122:0/397067167 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360501135 0 0) 0x7fef280cfa50 con 0x7fef2815eb00 2024-03-06T07:34:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.719+0000 7fef1ffff640 1 --2- 172.21.15.122:0/397067167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef280d14a0 0x7fef2815e5c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fef280d01e0 tx=0x7fef1002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d98282d8a8b97b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2659c640 1 -- 172.21.15.122:0/397067167 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef10035020 con 0x7fef280d14a0 2024-03-06T07:34:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2659c640 1 -- 172.21.15.122:0/397067167 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef10031800 con 0x7fef280d14a0 2024-03-06T07:34:16.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2659c640 1 -- 172.21.15.122:0/397067167 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef10031bd0 con 0x7fef280d14a0 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 -- 172.21.15.122:0/397067167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef280d14a0 msgr2=0x7fef2815e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 --2- 172.21.15.122:0/397067167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef280d14a0 0x7fef2815e5c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fef280d01e0 tx=0x7fef1002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 -- 172.21.15.122:0/397067167 shutdown_connections 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 --2- 172.21.15.122:0/397067167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef280d14a0 0x7fef2815e5c0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 --2- 172.21.15.122:0/397067167 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d10c0 0x7fef280d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 -- 172.21.15.122:0/397067167 >> 172.21.15.122:0/397067167 conn(0x7fef280b3800 msgr2=0x7fef280b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 -- 172.21.15.122:0/397067167 shutdown_connections 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.720+0000 7fef2d0df640 1 -- 172.21.15.122:0/397067167 wait complete. 2024-03-06T07:34:16.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.722+0000 7fef2d0df640 1 Processor -- start 2024-03-06T07:34:16.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.722+0000 7fef2d0df640 1 -- start start 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.723+0000 7fef2d0df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef280d10c0 0x7fef28152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.723+0000 7fef2d0df640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d14a0 0x7fef280ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2d0df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 0x7fef280b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2d0df640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2815f440 con 0x7fef2815eb00 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2d0df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2815f160 con 0x7fef280d10c0 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2d0df640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef280cfcf0 con 0x7fef280d14a0 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 0x7fef280b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 0x7fef280b6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60492/0 (socket says 172.21.15.122:60492) 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 -- 172.21.15.122:0/258711947 learned_addr learned my addr 172.21.15.122:0/258711947 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef1ffff640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d14a0 0x7fef280ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.726 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef26d9d640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef280d10c0 0x7fef28152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 -- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d14a0 msgr2=0x7fef280ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d14a0 0x7fef280ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 -- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef280d10c0 msgr2=0x7fef28152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef280d10c0 0x7fef28152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 -- 172.21.15.122:0/258711947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280b73a0 con 0x7fef2815eb00 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef1ffff640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d14a0 0x7fef280ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2759e640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 0x7fef280b6920 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c7a0 tx=0x7fef2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef20018070 con 0x7fef2815eb00 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2000a640 con 0x7fef2815eb00 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef280b7680 con 0x7fef2815eb00 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000aa70 con 0x7fef2815eb00 2024-03-06T07:34:16.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.724+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef280cc500 con 0x7fef2815eb00 2024-03-06T07:34:16.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.725+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2809fe60 con 0x7fef2815eb00 2024-03-06T07:34:16.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.726+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef2001b070 con 0x7fef2815eb00 2024-03-06T07:34:16.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.726+0000 7fef1f7fe640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef0806b4d0 0x7fef0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.726+0000 7fef26d9d640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef0806b4d0 0x7fef0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.726+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fef20089840 con 0x7fef2815eb00 2024-03-06T07:34:16.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.727+0000 7fef26d9d640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef0806b4d0 0x7fef0806d990 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fef18006fd0 tx=0x7fef18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.731+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef200586a0 con 0x7fef2815eb00 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.900+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef28068ff0 con 0x7fef2815eb00 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.900+0000 7fef1f7fe640 1 -- 172.21.15.122:0/258711947 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef20057fa0 con 0x7fef2815eb00 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 64 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef0806b4d0 msgr2=0x7fef0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef0806b4d0 0x7fef0806d990 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fef18006fd0 tx=0x7fef18008040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 msgr2=0x7fef280b6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 0x7fef280b6920 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c7a0 tx=0x7fef2000cc60 comp rx=0 tx=0).stop 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 shutdown_connections 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fef0806b4d0 0x7fef0806d990 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef2815eb00 0x7fef280b6920 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fef280d14a0 0x7fef280ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 --2- 172.21.15.122:0/258711947 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef280d10c0 0x7fef28152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 >> 172.21.15.122:0/258711947 conn(0x7fef280b3800 msgr2=0x7fef280b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 shutdown_connections 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:16.902+0000 7fef2d0df640 1 -- 172.21.15.122:0/258711947 wait complete. 2024-03-06T07:34:16.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:17.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.011+0000 7f9006b5c640 1 Processor -- start 2024-03-06T07:34:17.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.011+0000 7f9006b5c640 1 -- start start 2024-03-06T07:34:17.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f9006b5c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f900015dfe0 0x7f90000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f9006b5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000cf830 0x7f90000d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f9006b5c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90000b3430 con 0x7f90000d27f0 2024-03-06T07:34:17.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f9006b5c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90000d2c50 con 0x7f900015dfe0 2024-03-06T07:34:17.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f9006b5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90000b31e0 con 0x7f90000cf830 2024-03-06T07:34:17.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f90048f8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f900015dfe0 0x7f90000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f90048f8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f900015dfe0 0x7f90000cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60500/0 (socket says 172.21.15.122:60500) 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f90048f8640 1 -- 172.21.15.122:0/2398591330 learned_addr learned my addr 172.21.15.122:0/2398591330 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.012+0000 7f8ffffff640 1 --2- 172.21.15.122:0/2398591330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000cf830 0x7f90000d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.013+0000 7f90048f8640 1 -- 172.21.15.122:0/2398591330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000cf830 msgr2=0x7f90000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.013+0000 7f90048f8640 1 --2- 172.21.15.122:0/2398591330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000cf830 0x7f90000d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.013+0000 7f90048f8640 1 -- 172.21.15.122:0/2398591330 >> v1:172.21.15.169:6789/0 conn(0x7f90000d27f0 legacy=0x7f90000d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.013+0000 7f90048f8640 1 -- 172.21.15.122:0/2398591330 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90000b3710 con 0x7f900015dfe0 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.014+0000 7f90048f8640 1 --2- 172.21.15.122:0/2398591330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f90000cf2f0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009870 tx=0x7f8ff402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6f27baaf794c92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.014+0000 7f8fff7fe640 1 -- 172.21.15.122:0/2398591330 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4034070 con 0x7f900015dfe0 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.014+0000 7f8fff7fe640 1 -- 172.21.15.122:0/2398591330 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4030840 con 0x7f900015dfe0 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.014+0000 7f8fff7fe640 1 -- 172.21.15.122:0/2398591330 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4030c10 con 0x7f900015dfe0 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 -- 172.21.15.122:0/2398591330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 msgr2=0x7f90000cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2398591330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f90000cf2f0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009870 tx=0x7f8ff402b980 comp rx=0 tx=0).stop 2024-03-06T07:34:17.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 -- 172.21.15.122:0/2398591330 shutdown_connections 2024-03-06T07:34:17.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2398591330 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000cf830 0x7f90000d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2398591330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f90000cf2f0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 -- 172.21.15.122:0/2398591330 >> 172.21.15.122:0/2398591330 conn(0x7f90000d3cd0 msgr2=0x7f90000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 -- 172.21.15.122:0/2398591330 shutdown_connections 2024-03-06T07:34:17.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.015+0000 7f9006b5c640 1 -- 172.21.15.122:0/2398591330 wait complete. 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.016+0000 7f9006b5c640 1 Processor -- start 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.016+0000 7f9006b5c640 1 -- start start 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90000cf830 0x7f900014af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000d27f0 0x7f900015f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f9000160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90000d1c70 con 0x7f900015dfe0 2024-03-06T07:34:17.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90000b3d60 con 0x7f90000cf830 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90000d1df0 con 0x7f90000d27f0 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f9000160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f9000160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60508/0 (socket says 172.21.15.122:60508) 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 -- 172.21.15.122:0/2973799797 learned_addr learned my addr 172.21.15.122:0/2973799797 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f8ffffff640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000d27f0 0x7f900015f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 -- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000d27f0 msgr2=0x7f900015f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000d27f0 0x7f900015f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 -- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90000cf830 msgr2=0x7f900014af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90048f8640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90000cf830 0x7f900014af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90000cf830 0x7f900014af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 -- 172.21.15.122:0/2973799797 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900015a330 con 0x7f900015dfe0 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90048f8640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90000cf830 0x7f900014af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f90050f9640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f9000160490 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800c7a0 tx=0x7f8ff800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.021 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8003b70 con 0x7f900015dfe0 2024-03-06T07:34:17.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff8003d30 con 0x7f900015dfe0 2024-03-06T07:34:17.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.017+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90000cda60 con 0x7f900015dfe0 2024-03-06T07:34:17.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.018+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff800aa90 con 0x7f900015dfe0 2024-03-06T07:34:17.022 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.018+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90000ce020 con 0x7f900015dfe0 2024-03-06T07:34:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.019+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90000cde50 con 0x7f900015dfe0 2024-03-06T07:34:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.019+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ff800ac30 con 0x7f900015dfe0 2024-03-06T07:34:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.019+0000 7f8ffd7fa640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.019+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8ff808a160 con 0x7f900015dfe0 2024-03-06T07:34:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.020+0000 7f90048f8640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.023 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.020+0000 7f90048f8640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fe406b4d0 0x7f8fe406d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4036000 tx=0x7f8ff403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.024+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ff8058fc0 con 0x7f900015dfe0 2024-03-06T07:34:17.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.234+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90000cdc30 con 0x7f900015dfe0 2024-03-06T07:34:17.236 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.234+0000 7f8ffd7fa640 1 -- 172.21.15.122:0/2973799797 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f8ff80588c0 con 0x7f900015dfe0 2024-03-06T07:34:17.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:17.237 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:17.237 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fe406b4d0 msgr2=0x7f8fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fe406b4d0 0x7f8fe406d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4036000 tx=0x7f8ff403b040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 msgr2=0x7f9000160490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f9000160490 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800c7a0 tx=0x7f8ff800cc60 comp rx=0 tx=0).stop 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 shutdown_connections 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f900015dfe0 0x7f9000160490 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f90000d27f0 0x7f900015f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 --2- 172.21.15.122:0/2973799797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90000cf830 0x7f900014af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 >> 172.21.15.122:0/2973799797 conn(0x7f90000d3cd0 msgr2=0x7f900006d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 shutdown_connections 2024-03-06T07:34:17.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.236+0000 7f9006b5c640 1 -- 172.21.15.122:0/2973799797 wait complete. 2024-03-06T07:34:17.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:17.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.351+0000 7f0c578eb640 1 Processor -- start 2024-03-06T07:34:17.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.351+0000 7f0c578eb640 1 -- start start 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.352+0000 7f0c578eb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0c500d0a80 0x7f0c500b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.352+0000 7f0c578eb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c500d25d0 con 0x7f0c500d4760 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.352+0000 7f0c578eb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c500d2870 con 0x7f0c500b4170 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.352+0000 7f0c578eb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c500d4bf0 con 0x7f0c500d0a80 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.352+0000 7f0c54e86640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0c500d4760 0x7f0c500cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51080/0 (socket says 172.21.15.122:51080) 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.352+0000 7f0c54e86640 1 -- 172.21.15.122:0/1240541735 learned_addr learned my addr 172.21.15.122:0/1240541735 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c55e88640 1 --2- 172.21.15.122:0/1240541735 >> v2:172.21.15.169:3300/0 conn(0x7f0c500d0a80 0x7f0c500b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574701936 0 0) 0x7f0c500d25d0 con 0x7f0c500d4760 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c3c003520 con 0x7f0c500d4760 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431842102 0 0) 0x7f0c500d2870 con 0x7f0c500b4170 2024-03-06T07:34:17.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c500d25d0 con 0x7f0c500b4170 2024-03-06T07:34:17.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c55e88640 1 -- 172.21.15.122:0/1240541735 >> v1:172.21.15.169:6789/0 conn(0x7f0c500b4170 legacy=0x7f0c500d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647116379 0 0) 0x7f0c3c003520 con 0x7f0c500d4760 2024-03-06T07:34:17.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c55e88640 1 -- 172.21.15.122:0/1240541735 >> v1:172.21.15.122:6789/0 conn(0x7f0c500d4760 legacy=0x7f0c500cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.353+0000 7f0c55e88640 1 -- 172.21.15.122:0/1240541735 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c500d2b50 con 0x7f0c500d0a80 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.354+0000 7f0c55e88640 1 --2- 172.21.15.122:0/1240541735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b6a90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009d90 tx=0x7f0c4802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de03e6e22c4a1932 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.354+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48039070 con 0x7f0c500d0a80 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.354+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4802c940 con 0x7f0c500d0a80 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.354+0000 7f0c47fff640 1 -- 172.21.15.122:0/1240541735 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c40001670 con 0x7f0c500d0a80 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 -- 172.21.15.122:0/1240541735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 msgr2=0x7f0c500b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 --2- 172.21.15.122:0/1240541735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b6a90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009d90 tx=0x7f0c4802be10 comp rx=0 tx=0).stop 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 -- 172.21.15.122:0/1240541735 shutdown_connections 2024-03-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 --2- 172.21.15.122:0/1240541735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b6a90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 -- 172.21.15.122:0/1240541735 >> 172.21.15.122:0/1240541735 conn(0x7f0c500d3c40 msgr2=0x7f0c500d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 -- 172.21.15.122:0/1240541735 shutdown_connections 2024-03-06T07:34:17.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.355+0000 7f0c578eb640 1 -- 172.21.15.122:0/1240541735 wait complete. 2024-03-06T07:34:17.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.356+0000 7f0c578eb640 1 Processor -- start 2024-03-06T07:34:17.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.356+0000 7f0c578eb640 1 -- start start 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c578eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c500b4170 0x7f0c500b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c578eb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c578eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 0x7f0c5015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c578eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c500b4dd0 con 0x7f0c5015b8c0 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c578eb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c500ce9e0 con 0x7f0c500d0a80 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c578eb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c500b4f50 con 0x7f0c500b4170 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 0x7f0c5015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 0x7f0c5015bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60524/0 (socket says 172.21.15.122:60524) 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 -- 172.21.15.122:0/3849549161 learned_addr learned my addr 172.21.15.122:0/3849549161 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55687640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c500b4170 0x7f0c500b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c54e86640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 -- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c500b4170 msgr2=0x7f0c500b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c500b4170 0x7f0c500b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 -- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 msgr2=0x7f0c500b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 -- 172.21.15.122:0/3849549161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5015c720 con 0x7f0c5015b8c0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55687640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c500b4170 0x7f0c500b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c54e86640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c55e88640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 0x7f0c5015bd50 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f0c48037040 tx=0x7f0c48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.357+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48039050 con 0x7f0c5015b8c0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.358+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4803a040 con 0x7f0c5015b8c0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.358+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c500b1a90 con 0x7f0c5015b8c0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.358+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48009450 con 0x7f0c5015b8c0 2024-03-06T07:34:17.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.358+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c500cbd60 con 0x7f0c5015b8c0 2024-03-06T07:34:17.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.359+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5004f330 con 0x7f0c5015b8c0 2024-03-06T07:34:17.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.359+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c48047020 con 0x7f0c5015b8c0 2024-03-06T07:34:17.362 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.360+0000 7f0c45ffb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c3406b4d0 0x7f0c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.360+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0c480acb30 con 0x7f0c5015b8c0 2024-03-06T07:34:17.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.360+0000 7f0c55687640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c3406b4d0 0x7f0c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.363 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.360+0000 7f0c55687640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c3406b4d0 0x7f0c3406d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0011e0 tx=0x7f0c4c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.364+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c4804e020 con 0x7f0c5015b8c0 2024-03-06T07:34:17.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.538+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c500733b0 con 0x7f0c5015b8c0 2024-03-06T07:34:17.541 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.539+0000 7f0c45ffb640 1 -- 172.21.15.122:0/3849549161 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f0c5015c720 con 0x7f0c5015b8c0 2024-03-06T07:34:17.542 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:17.542 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c3406b4d0 msgr2=0x7f0c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c3406b4d0 0x7f0c3406d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0011e0 tx=0x7f0c4c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 msgr2=0x7f0c5015bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 0x7f0c5015bd50 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f0c48037040 tx=0x7f0c48004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 shutdown_connections 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0c3406b4d0 0x7f0c3406d990 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c5015b8c0 0x7f0c5015bd50 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c500d0a80 0x7f0c500b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 --2- 172.21.15.122:0/3849549161 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0c500b4170 0x7f0c500b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 >> 172.21.15.122:0/3849549161 conn(0x7f0c500d3c40 msgr2=0x7f0c5015cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.541+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 shutdown_connections 2024-03-06T07:34:17.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.542+0000 7f0c578eb640 1 -- 172.21.15.122:0/3849549161 wait complete. 2024-03-06T07:34:17.559 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:17.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:17.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.661+0000 7fd6ec35c640 1 Processor -- start 2024-03-06T07:34:17.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 -- start start 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fd6e415dfe0 0x7fd6e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6e40d27f0 0x7fd6e40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e40d2c50 con 0x7fd6e40d27f0 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e40b31e0 con 0x7fd6e415dfe0 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.662+0000 7fd6ec35c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b3d60 con 0x7fd6e40cf830 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6e40d27f0 0x7fd6e40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd6e40d27f0 0x7fd6e40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60536/0 (socket says 172.21.15.122:60536) 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/3330470946 learned_addr learned my addr 172.21.15.122:0/3330470946 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6e98f7640 1 --2- 172.21.15.122:0/3330470946 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea0f8640 1 --2- 172.21.15.122:0/3330470946 >> v2:172.21.15.169:3300/0 conn(0x7fd6e415dfe0 0x7fd6e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/3330470946 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 msgr2=0x7fd6e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 --2- 172.21.15.122:0/3330470946 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/3330470946 >> v2:172.21.15.169:3300/0 conn(0x7fd6e415dfe0 msgr2=0x7fd6e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 --2- 172.21.15.122:0/3330470946 >> v2:172.21.15.169:3300/0 conn(0x7fd6e415dfe0 0x7fd6e40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/3330470946 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40b3430 con 0x7fd6e40d27f0 2024-03-06T07:34:17.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.663+0000 7fd6ea0f8640 1 --2- 172.21.15.122:0/3330470946 >> v2:172.21.15.169:3300/0 conn(0x7fd6e415dfe0 0x7fd6e40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ea8f9640 1 --2- 172.21.15.122:0/3330470946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e40d27f0 0x7fd6e40d0300 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4067990 tx=0x7fd6dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47dd51bb070067d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6e90f6640 1 -- 172.21.15.122:0/3330470946 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc02ca70 con 0x7fd6e40d27f0 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6e90f6640 1 -- 172.21.15.122:0/3330470946 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc02cc10 con 0x7fd6e40d27f0 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6e90f6640 1 -- 172.21.15.122:0/3330470946 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0343f0 con 0x7fd6e40d27f0 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 -- 172.21.15.122:0/3330470946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e40d27f0 msgr2=0x7fd6e40d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/3330470946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e40d27f0 0x7fd6e40d0300 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4067990 tx=0x7fd6dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 -- 172.21.15.122:0/3330470946 shutdown_connections 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/3330470946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e40d27f0 0x7fd6e40d0300 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/3330470946 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/3330470946 >> v2:172.21.15.169:3300/0 conn(0x7fd6e415dfe0 0x7fd6e40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.664+0000 7fd6ec35c640 1 -- 172.21.15.122:0/3330470946 >> 172.21.15.122:0/3330470946 conn(0x7fd6e40d3cd0 msgr2=0x7fd6e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.665+0000 7fd6ec35c640 1 -- 172.21.15.122:0/3330470946 shutdown_connections 2024-03-06T07:34:17.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.665+0000 7fd6ec35c640 1 -- 172.21.15.122:0/3330470946 wait complete. 2024-03-06T07:34:17.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 Processor -- start 2024-03-06T07:34:17.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 -- start start 2024-03-06T07:34:17.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e414f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6e40d27f0 0x7fd6e415e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.672 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 0x7fd6e415bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40d1c70 con 0x7fd6e415dfe0 2024-03-06T07:34:17.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b39f0 con 0x7fd6e40d27f0 2024-03-06T07:34:17.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.666+0000 7fd6ec35c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40d1df0 con 0x7fd6e40cf830 2024-03-06T07:34:17.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 0x7fd6e415bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 0x7fd6e415bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60550/0 (socket says 172.21.15.122:60550) 2024-03-06T07:34:17.673 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/4084231644 learned_addr learned my addr 172.21.15.122:0/4084231644 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:17.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea0f8640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e414f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6e98f7640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6e40d27f0 0x7fd6e415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 msgr2=0x7fd6e414f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e414f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6e40d27f0 msgr2=0x7fd6e415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6e40d27f0 0x7fd6e415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 -- 172.21.15.122:0/4084231644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40cada0 con 0x7fd6e415dfe0 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea0f8640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e414f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ea8f9640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 0x7fd6e415bf70 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009f40 tx=0x7fd6dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc043020 con 0x7fd6e415dfe0 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc03e450 con 0x7fd6e415dfe0 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e40cb080 con 0x7fd6e415dfe0 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc009510 con 0x7fd6e415dfe0 2024-03-06T07:34:17.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.667+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e40cb640 con 0x7fd6e415dfe0 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.668+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e404f330 con 0x7fd6e415dfe0 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.669+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6dc046020 con 0x7fd6e415dfe0 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.669+0000 7fd6daffd640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6c806b4d0 0x7fd6c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.669+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd6dc0ac9b0 con 0x7fd6e415dfe0 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.669+0000 7fd6ea0f8640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6c806b4d0 0x7fd6c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.670+0000 7fd6ea0f8640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6c806b4d0 0x7fd6c806d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0006fd0 tx=0x7fd6e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.676 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.673+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6dc04d030 con 0x7fd6e415dfe0 2024-03-06T07:34:17.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.887+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e40b9b90 con 0x7fd6e415dfe0 2024-03-06T07:34:17.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.887+0000 7fd6daffd640 1 -- 172.21.15.122:0/4084231644 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd6dc07b810 con 0x7fd6e415dfe0 2024-03-06T07:34:17.889 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:17.890 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:17.890 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:17.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6c806b4d0 msgr2=0x7fd6c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6c806b4d0 0x7fd6c806d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0006fd0 tx=0x7fd6e0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 msgr2=0x7fd6e415bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 0x7fd6e415bf70 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009f40 tx=0x7fd6dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 shutdown_connections 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd6c806b4d0 0x7fd6c806d990 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6e415dfe0 0x7fd6e415bf70 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6e40d27f0 0x7fd6e415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 --2- 172.21.15.122:0/4084231644 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd6e40cf830 0x7fd6e414f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 >> 172.21.15.122:0/4084231644 conn(0x7fd6e40d3cd0 msgr2=0x7fd6e40cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 shutdown_connections 2024-03-06T07:34:17.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:17.889+0000 7fd6ec35c640 1 -- 172.21.15.122:0/4084231644 wait complete. 2024-03-06T07:34:17.902 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:18.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.001+0000 7fd4eb4e0640 1 Processor -- start 2024-03-06T07:34:18.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.001+0000 7fd4eb4e0640 1 -- start start 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4eb4e0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 0x7fd4e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4eb4e0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e40b3430 con 0x7fd4e40cf830 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4eb4e0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e40b3710 con 0x7fd4e40d29c0 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4eb4e0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b31e0 con 0x7fd4e415dfe0 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4e8a7b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd4e40cf830 0x7fd4e40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51088/0 (socket says 172.21.15.122:51088) 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4e8a7b640 1 -- 172.21.15.122:0/909865516 learned_addr learned my addr 172.21.15.122:0/909865516 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4e927c640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 0x7fd4e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412357576 0 0) 0x7fd4e40b3430 con 0x7fd4e40cf830 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d0003520 con 0x7fd4e40cf830 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.002+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525458106 0 0) 0x7fd4d0003520 con 0x7fd4e40cf830 2024-03-06T07:34:18.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 msgr2=0x7fd4e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 0x7fd4e40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 >> v1:172.21.15.169:6789/0 conn(0x7fd4e40d29c0 legacy=0x7fd4e40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40b39f0 con 0x7fd4e40cf830 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fd4e0001670 con 0x7fd4e40cf830 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 >> v1:172.21.15.122:6789/0 conn(0x7fd4e40cf830 legacy=0x7fd4e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4d00039e0 0x7fd4d0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4d0004800 0x7fd4d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 0x7fd4d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b31e0 con 0x7fd4d0004800 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d000ba70 con 0x7fd4d00039e0 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d000bc70 con 0x7fd4e415dfe0 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e8a7b640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4d0004800 0x7fd4d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e927c640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4d00039e0 0x7fd4d0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e9a7d640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 0x7fd4d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e8a7b640 1 -- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 msgr2=0x7fd4d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e8a7b640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4e415dfe0 0x7fd4d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e8a7b640 1 -- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4d00039e0 msgr2=0x7fd4d0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e8a7b640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4d00039e0 0x7fd4d0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.003+0000 7fd4e8a7b640 1 -- 172.21.15.122:0/909865516 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0003520 con 0x7fd4d0004800 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4e8a7b640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4d0004800 0x7fd4d0004c80 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40cfc70 tx=0x7fd4d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefb6620f9f1cd57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0007d10 con 0x7fd4d0004800 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e0030070 con 0x7fd4d0004800 2024-03-06T07:34:18.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4dbfff640 1 -- 172.21.15.122:0/909865516 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0005d30 con 0x7fd4d0004800 2024-03-06T07:34:18.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/909865516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4d0004800 msgr2=0x7fd4d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/909865516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4d0004800 0x7fd4d0004c80 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40cfc70 tx=0x7fd4d0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:18.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4e9a7d640 1 -- 172.21.15.122:0/909865516 reap_dead start 2024-03-06T07:34:18.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/909865516 shutdown_connections 2024-03-06T07:34:18.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.004+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/909865516 >> 172.21.15.122:0/909865516 conn(0x7fd4e40d3cd0 msgr2=0x7fd4e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.005+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/909865516 shutdown_connections 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.005+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/909865516 wait complete. 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 Processor -- start 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 -- start start 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4d0004800 0x7fd4e415bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4e415c430 0x7fd4e40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 0x7fd4e40a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e415e3c0 con 0x7fd4e40a11b0 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b3d60 con 0x7fd4e415c430 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.006+0000 7fd4eb4e0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e415e540 con 0x7fd4d0004800 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 0x7fd4e40a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 0x7fd4e40a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60570/0 (socket says 172.21.15.122:60570) 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 -- 172.21.15.122:0/79994470 learned_addr learned my addr 172.21.15.122:0/79994470 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e8a7b640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4e415c430 0x7fd4e40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e927c640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4d0004800 0x7fd4e415bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 -- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4d0004800 msgr2=0x7fd4e415bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4d0004800 0x7fd4e415bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 -- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4e415c430 msgr2=0x7fd4e40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4e415c430 0x7fd4e40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 -- 172.21.15.122:0/79994470 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40cb630 con 0x7fd4e40a11b0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e927c640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4d0004800 0x7fd4e415bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4e9a7d640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 0x7fd4e40a1630 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b5770 tx=0x7fd4dc00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc01a070 con 0x7fd4e40a11b0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc006d60 con 0x7fd4e40a11b0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc003b90 con 0x7fd4e40a11b0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e40cb910 con 0x7fd4e40a11b0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.007+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e40cbed0 con 0x7fd4e40a11b0 2024-03-06T07:34:18.012 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.008+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e404f330 con 0x7fd4e40a11b0 2024-03-06T07:34:18.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.009+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4dc01d070 con 0x7fd4e40a11b0 2024-03-06T07:34:18.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.009+0000 7fd4d9ffb640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4c806b4d0 0x7fd4c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.009+0000 7fd4e927c640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4c806b4d0 0x7fd4c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.009+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd4dc0897c0 con 0x7fd4e40a11b0 2024-03-06T07:34:18.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.010+0000 7fd4e927c640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4c806b4d0 0x7fd4c806d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009ab0 tx=0x7fd4d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.015 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.014+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4dc058620 con 0x7fd4e40a11b0 2024-03-06T07:34:18.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.232+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e40b09c0 con 0x7fd4e40a11b0 2024-03-06T07:34:18.234 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.233+0000 7fd4d9ffb640 1 -- 172.21.15.122:0/79994470 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd4dc057f20 con 0x7fd4e40a11b0 2024-03-06T07:34:18.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:18.235 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":101,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.770682+0000","modified":"2024-03-06T07:32:37.113279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9321,"mds_1":9318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9318":{"gid":9318,"name":"c","rank":1,"incarnation":92,"state":"up:stopping","state_seq":5,"addr":"172.21.15.122:6835/138214324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":138214324},{"type":"v1","addr":"172.21.15.122:6835","nonce":138214324}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:34.001145+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9321":{"gid":9321,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6837/1416910179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1416910179},{"type":"v1","addr":"172.21.15.122:6837","nonce":1416910179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9321,"qdb_cluster":[9321]},"id":10}]} 2024-03-06T07:34:18.235 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 101 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4c806b4d0 msgr2=0x7fd4c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4c806b4d0 0x7fd4c806d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009ab0 tx=0x7fd4d4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 msgr2=0x7fd4e40a1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 0x7fd4e40a1630 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b5770 tx=0x7fd4dc00fd40 comp rx=0 tx=0).stop 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 shutdown_connections 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fd4c806b4d0 0x7fd4c806d990 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.237 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd4e40a11b0 0x7fd4e40a1630 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4e415c430 0x7fd4e40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.234+0000 7fd4eb4e0640 1 --2- 172.21.15.122:0/79994470 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fd4d0004800 0x7fd4e415bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.235+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 >> 172.21.15.122:0/79994470 conn(0x7fd4e40d3cd0 msgr2=0x7fd4e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.235+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 shutdown_connections 2024-03-06T07:34:18.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.235+0000 7fd4eb4e0640 1 -- 172.21.15.122:0/79994470 wait complete. 2024-03-06T07:34:18.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:18.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.346+0000 7f40c42af640 1 Processor -- start 2024-03-06T07:34:18.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.346+0000 7f40c42af640 1 -- start start 2024-03-06T07:34:18.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c42af640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 0x7f40bc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c42af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 0x7f40bc0ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c42af640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc161980 con 0x7f40bc15bb20 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c42af640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc0d2500 con 0x7f40bc0cfff0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c42af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc161c60 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 0x7f40bc0ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c184a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 0x7f40bc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 0x7f40bc0ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:48684/0 (socket says 172.21.15.122:48684) 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c184a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 0x7f40bc0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:59040/0 (socket says 172.21.15.122:59040) 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 -- 172.21.15.122:0/1973545604 learned_addr learned my addr 172.21.15.122:0/1973545604 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c204b640 1 --1- 172.21.15.122:0/1973545604 >> v1:172.21.15.169:6789/0 conn(0x7f40bc15bb20 0x7f40bc0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.122:58318/0 (socket says 172.21.15.122:58318) 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 -- 172.21.15.122:0/1973545604 >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 msgr2=0x7f40bc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 --2- 172.21.15.122:0/1973545604 >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 0x7f40bc0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 -- 172.21.15.122:0/1973545604 >> v1:172.21.15.169:6789/0 conn(0x7f40bc15bb20 legacy=0x7f40bc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c284c640 1 -- 172.21.15.122:0/1973545604 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc161f40 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.351 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.347+0000 7f40c184a640 1 --2- 172.21.15.122:0/1973545604 >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 0x7f40bc0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:18.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.348+0000 7f40c284c640 1 --2- 172.21.15.122:0/1973545604 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 0x7f40bc0ccd20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009e50 tx=0x7f40b402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7be0da75e18bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.354 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.348+0000 7f40c1049640 1 -- 172.21.15.122:0/1973545604 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b402cb00 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.348+0000 7f40c1049640 1 -- 172.21.15.122:0/1973545604 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b402cca0 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.348+0000 7f40c1049640 1 -- 172.21.15.122:0/1973545604 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4034610 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 -- 172.21.15.122:0/1973545604 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 msgr2=0x7f40bc0ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 --2- 172.21.15.122:0/1973545604 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 0x7f40bc0ccd20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009e50 tx=0x7f40b402bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 -- 172.21.15.122:0/1973545604 shutdown_connections 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 --2- 172.21.15.122:0/1973545604 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc0cc8b0 0x7f40bc0ccd20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 --2- 172.21.15.122:0/1973545604 >> v2:172.21.15.169:3300/0 conn(0x7f40bc0cfff0 0x7f40bc0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 -- 172.21.15.122:0/1973545604 >> 172.21.15.122:0/1973545604 conn(0x7f40bc09a9f0 msgr2=0x7f40bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 -- 172.21.15.122:0/1973545604 shutdown_connections 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.349+0000 7f40c42af640 1 -- 172.21.15.122:0/1973545604 wait complete. 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.350+0000 7f40c42af640 1 Processor -- start 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.350+0000 7f40c42af640 1 -- start start 2024-03-06T07:34:18.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c42af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 0x7f40bc155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c42af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f40bc0cfff0 0x7f40bc0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c42af640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc15bb20 0x7f40bc0af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c42af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0d1980 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c42af640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc162500 con 0x7f40bc0cfff0 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c42af640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0d1b00 con 0x7f40bc15bb20 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 0x7f40bc155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 0x7f40bc155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60584/0 (socket says 172.21.15.122:60584) 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 -- 172.21.15.122:0/3099633933 learned_addr learned my addr 172.21.15.122:0/3099633933 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c184a640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f40bc0cfff0 0x7f40bc0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c284c640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc15bb20 0x7f40bc0af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 -- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc15bb20 msgr2=0x7f40bc0af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc15bb20 0x7f40bc0af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 -- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f40bc0cfff0 msgr2=0x7f40bc0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f40bc0cfff0 0x7f40bc0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.351+0000 7f40c204b640 1 -- 172.21.15.122:0/3099633933 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0c6b30 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.352+0000 7f40c204b640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 0x7f40bc155320 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00c3c0 tx=0x7f40ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.352+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac0074e0 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.352+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac007680 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.352+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac004a00 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.352+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc0c94e0 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.357 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.352+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc0c9aa0 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.353+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc0b3060 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.353+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40ac01a020 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.354+0000 7f40b2ffd640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f40a006b4d0 0x7f40a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.354+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f40ac0898a0 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.354+0000 7f40c184a640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f40a006b4d0 0x7f40a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.358 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.354+0000 7f40c184a640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f40a006b4d0 0x7f40a006d990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009560 tx=0x7f40b8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.358+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40bc0b3060 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.538+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40bc04f330 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.539+0000 7f40b2ffd640 1 -- 172.21.15.122:0/3099633933 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f40ac058030 con 0x7f40bc0cc8b0 2024-03-06T07:34:18.551 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:18.551 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":143,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:30:36.073519+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.987708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.745253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f40a006b4d0 msgr2=0x7f40a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f40a006b4d0 0x7f40a006d990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009560 tx=0x7f40b8009190 comp rx=0 tx=0).stop 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 msgr2=0x7f40bc155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 0x7f40bc155320 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00c3c0 tx=0x7f40ac00c880 comp rx=0 tx=0).stop 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 shutdown_connections 2024-03-06T07:34:18.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f40a006b4d0 0x7f40a006d990 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f40bc15bb20 0x7f40bc0af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f40bc0cfff0 0x7f40bc0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 --2- 172.21.15.122:0/3099633933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40bc0cc8b0 0x7f40bc155320 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 >> 172.21.15.122:0/3099633933 conn(0x7f40bc09a9f0 msgr2=0x7f40bc09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 shutdown_connections 2024-03-06T07:34:18.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.541+0000 7f40c42af640 1 -- 172.21.15.122:0/3099633933 wait complete. 2024-03-06T07:34:18.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:34:18.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.658+0000 7f964f488640 1 Processor -- start 2024-03-06T07:34:18.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.659+0000 7f964f488640 1 -- start start 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.659+0000 7f964f488640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.659+0000 7f964f488640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9648161cf0 0x7f9648162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.659+0000 7f964f488640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480d22c0 con 0x7f9648162970 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.659+0000 7f964f488640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9648161b00 con 0x7f9648161cf0 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.659+0000 7f964f488640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480cecc0 con 0x7f96480b4460 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964d224640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9648161cf0 0x7f9648162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9648161cf0 0x7f9648162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.122:59066/0 (socket says 172.21.15.122:59066) 2024-03-06T07:34:18.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964d224640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:48700/0 (socket says 172.21.15.122:48700) 2024-03-06T07:34:18.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 -- 172.21.15.122:0/522784668 learned_addr learned my addr 172.21.15.122:0/522784668 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:18.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 -- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 msgr2=0x7f96480d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 --2- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 -- 172.21.15.122:0/522784668 >> v1:172.21.15.169:6789/0 conn(0x7f9648162970 legacy=0x7f96480b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.660+0000 7f964da25640 1 -- 172.21.15.122:0/522784668 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480d25a0 con 0x7f9648161cf0 2024-03-06T07:34:18.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.661+0000 7f964da25640 1 --2- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f9648162180 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9640009e50 tx=0x7f964002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f0453354e0243a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.661+0000 7f9647fff640 1 -- 172.21.15.122:0/522784668 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964002cb00 con 0x7f9648161cf0 2024-03-06T07:34:18.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.661+0000 7f9647fff640 1 -- 172.21.15.122:0/522784668 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964002cca0 con 0x7f9648161cf0 2024-03-06T07:34:18.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.661+0000 7f9647fff640 1 -- 172.21.15.122:0/522784668 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640034610 con 0x7f9648161cf0 2024-03-06T07:34:18.664 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 -- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 msgr2=0x7f9648162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 --2- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f9648162180 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9640009e50 tx=0x7f964002bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 -- 172.21.15.122:0/522784668 shutdown_connections 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 --2- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f9648162180 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 --2- 172.21.15.122:0/522784668 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 -- 172.21.15.122:0/522784668 >> 172.21.15.122:0/522784668 conn(0x7f96480d3ce0 msgr2=0x7f96480d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 -- 172.21.15.122:0/522784668 shutdown_connections 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.662+0000 7f964f488640 1 -- 172.21.15.122:0/522784668 wait complete. 2024-03-06T07:34:18.665 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.664+0000 7f964f488640 1 Processor -- start 2024-03-06T07:34:18.666 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.664+0000 7f964f488640 1 -- start start 2024-03-06T07:34:18.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964f488640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964f488640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f96480cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964f488640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 0x7f9648155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964f488640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480ce950 con 0x7f9648162970 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964f488640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480d2b60 con 0x7f9648161cf0 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964f488640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480b4840 con 0x7f96480b4460 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 0x7f9648155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 0x7f9648155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60594/0 (socket says 172.21.15.122:60594) 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 -- 172.21.15.122:0/1604509925 learned_addr learned my addr 172.21.15.122:0/1604509925 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964d224640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964ca23640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f96480cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 -- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 msgr2=0x7f96480c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 -- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 msgr2=0x7f96480cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f96480cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964da25640 1 -- 172.21.15.122:0/1604509925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480ccf80 con 0x7f9648162970 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.665+0000 7f964ca23640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f96480cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.666+0000 7f964da25640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 0x7f9648155fe0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f9640036040 tx=0x7f9640004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.666+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640043020 con 0x7f9648162970 2024-03-06T07:34:18.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.666+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964002ce50 con 0x7f9648162970 2024-03-06T07:34:18.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.666+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96480c9010 con 0x7f9648162970 2024-03-06T07:34:18.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.666+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640009510 con 0x7f9648162970 2024-03-06T07:34:18.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.666+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96480c95d0 con 0x7f9648162970 2024-03-06T07:34:18.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.667+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96480b7160 con 0x7f9648162970 2024-03-06T07:34:18.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.667+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9640046020 con 0x7f9648162970 2024-03-06T07:34:18.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.668+0000 7f9645ffb640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f963406b4d0 0x7f963406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.668+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f96400ac720 con 0x7f9648162970 2024-03-06T07:34:18.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.668+0000 7f964d224640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f963406b4d0 0x7f963406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.671 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.668+0000 7f964d224640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f963406b4d0 0x7f963406d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9638009aa0 tx=0x7f9638009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:18.673+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96480b7160 con 0x7f9648162970 2024-03-06T07:34:19.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.081+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f964804f330 con 0x7f9648162970 2024-03-06T07:34:19.083 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-06T07:34:19.081+0000 7fdd698eb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:19.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.082+0000 7f9645ffb640 1 -- 172.21.15.122:0/1604509925 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7f964004d030 con 0x7f9648162970 2024-03-06T07:34:19.084 INFO:teuthology.orchestra.run.smithi122.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.084+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f963406b4d0 msgr2=0x7f963406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.084+0000 7f964f488640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f963406b4d0 0x7f963406d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9638009aa0 tx=0x7f9638009190 comp rx=0 tx=0).stop 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.084+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 msgr2=0x7f9648155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.084+0000 7f964f488640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 0x7f9648155fe0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f9640036040 tx=0x7f9640004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 shutdown_connections 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f963406b4d0 0x7f963406d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9648162970 0x7f9648155fe0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9648161cf0 0x7f96480cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 --2- 172.21.15.122:0/1604509925 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f96480b4460 0x7f96480c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 >> 172.21.15.122:0/1604509925 conn(0x7f96480d3ce0 msgr2=0x7f9648161250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 shutdown_connections 2024-03-06T07:34:19.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.085+0000 7f964f488640 1 -- 172.21.15.122:0/1604509925 wait complete. 2024-03-06T07:34:19.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:34:19.127 INFO:tasks.ceph.mds.a.smithi122.stdout:starting mds.a at 2024-03-06T07:34:19.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.200+0000 7f0098288640 1 Processor -- start 2024-03-06T07:34:19.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.200+0000 7f0098288640 1 -- start start 2024-03-06T07:34:19.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.201+0000 7f0098288640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f009015ecd0 0x7f00900cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.201+0000 7f0098288640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00900cfa50 con 0x7f00900d14a0 2024-03-06T07:34:19.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.201+0000 7f0098288640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00900b45c0 con 0x7f00900d10c0 2024-03-06T07:34:19.206 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.201+0000 7f0098288640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009015c050 con 0x7f009015ecd0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.201+0000 7f0095823640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f00900d14a0 0x7f009015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51096/0 (socket says 172.21.15.122:51096) 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.201+0000 7f0095823640 1 -- 172.21.15.122:0/3188308250 learned_addr learned my addr 172.21.15.122:0/3188308250 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767366663 0 0) 0x7f00900cfa50 con 0x7f00900d14a0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f007c003520 con 0x7f00900d14a0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 --2- 172.21.15.122:0/3188308250 >> v2:172.21.15.169:3300/0 conn(0x7f009015ecd0 0x7f00900cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131850222 0 0) 0x7f007c003520 con 0x7f00900d14a0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 >> v2:172.21.15.169:3300/0 conn(0x7f009015ecd0 msgr2=0x7f00900cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 --2- 172.21.15.122:0/3188308250 >> v2:172.21.15.169:3300/0 conn(0x7f009015ecd0 0x7f00900cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 >> v1:172.21.15.169:6789/0 conn(0x7f00900d10c0 legacy=0x7f00900d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009015f160 con 0x7f00900d14a0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f0080001670 con 0x7f00900d14a0 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 >> v1:172.21.15.122:6789/0 conn(0x7f00900d14a0 legacy=0x7f009015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f007c0039b0 0x7f007c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c004800 0x7f007c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.207 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007c0051c0 0x7f007c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c00c0e0 con 0x7f007c0051c0 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c00c290 con 0x7f007c0039b0 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c00c490 con 0x7f007c004800 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 --2- 172.21.15.122:0/3188308250 >> v2:172.21.15.169:3300/0 conn(0x7f009015ecd0 0x7f00900cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007c0051c0 0x7f007c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096024640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f007c0039b0 0x7f007c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0095823640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c004800 0x7f007c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 -- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c004800 msgr2=0x7f007c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c004800 0x7f007c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 -- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f007c0039b0 msgr2=0x7f007c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f007c0039b0 0x7f007c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.208 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.202+0000 7f0096825640 1 -- 172.21.15.122:0/3188308250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c003520 con 0x7f007c0051c0 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.203+0000 7f0096825640 1 -- 172.21.15.122:0/3188308250 reap_dead start 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.203+0000 7f0096825640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007c0051c0 0x7f007c00b9c0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f00880098a0 tx=0x7f007c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2686ff14378da299 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.203+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008800f040 con 0x7f007c0051c0 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.203+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088037600 con 0x7f007c0051c0 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.203+0000 7f0095022640 1 -- 172.21.15.122:0/3188308250 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00880379b0 con 0x7f007c0051c0 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 -- 172.21.15.122:0/3188308250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007c0051c0 msgr2=0x7f007c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007c0051c0 0x7f007c00b9c0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f00880098a0 tx=0x7f007c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 -- 172.21.15.122:0/3188308250 shutdown_connections 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 --2- 172.21.15.122:0/3188308250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f007c0051c0 0x7f007c00b9c0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 -- 172.21.15.122:0/3188308250 >> 172.21.15.122:0/3188308250 conn(0x7f00900b3800 msgr2=0x7f00900b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 -- 172.21.15.122:0/3188308250 shutdown_connections 2024-03-06T07:34:19.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.204+0000 7f0098288640 1 -- 172.21.15.122:0/3188308250 wait complete. 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.205+0000 7f0098288640 1 Processor -- start 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.205+0000 7f0098288640 1 -- start start 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c0051c0 0x7f009014d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f009014de10 0x7f00900b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 0x7f00900ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900cfcf0 con 0x7f00900b0610 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900d0060 con 0x7f009014de10 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00900d04f0 con 0x7f007c0051c0 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 0x7f00900ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 0x7f00900ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60648/0 (socket says 172.21.15.122:60648) 2024-03-06T07:34:19.212 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 -- 172.21.15.122:0/2119828158 learned_addr learned my addr 172.21.15.122:0/2119828158 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096024640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c0051c0 0x7f009014d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0095823640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f009014de10 0x7f00900b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 -- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c0051c0 msgr2=0x7f009014d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c0051c0 0x7f009014d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 -- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f009014de10 msgr2=0x7f00900b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f009014de10 0x7f00900b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 -- 172.21.15.122:0/2119828158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00900b48f0 con 0x7f00900b0610 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096024640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c0051c0 0x7f009014d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0095823640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f009014de10 0x7f00900b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0096825640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 0x7f00900ce0e0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f00880099c0 tx=0x7f0088004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008800f000 con 0x7f00900b0610 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088047070 con 0x7f00900b0610 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00900cecc0 con 0x7f00900b0610 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.206+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008802fc50 con 0x7f00900b0610 2024-03-06T07:34:19.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.207+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00900cf280 con 0x7f00900b0610 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.207+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009009fe60 con 0x7f00900b0610 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.208+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f008804c460 con 0x7f00900b0610 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.208+0000 7f0086ffd640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f007406b500 0x7f007406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.208+0000 7f0096024640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f007406b500 0x7f007406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.208+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f00880ac4b0 con 0x7f00900b0610 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.209+0000 7f0096024640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f007406b500 0x7f007406d9c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f008c000f80 tx=0x7f008c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.213+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0088033030 con 0x7f00900b0610 2024-03-06T07:34:19.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:19.422+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0090068ff0 con 0x7f00900b0610 2024-03-06T07:34:20.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.084+0000 7f0086ffd640 1 -- 172.21.15.122:0/2119828158 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v103) v1 ==== 104+0+0 (secure 0 0 0) 0x7f008802e370 con 0x7f00900b0610 2024-03-06T07:34:20.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f007406b500 msgr2=0x7f007406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f007406b500 0x7f007406d9c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f008c000f80 tx=0x7f008c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 msgr2=0x7f00900ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 0x7f00900ce0e0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f00880099c0 tx=0x7f0088004730 comp rx=0 tx=0).stop 2024-03-06T07:34:20.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 shutdown_connections 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f007406b500 0x7f007406d9c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00900b0610 0x7f00900ce0e0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f009014de10 0x7f00900b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 --2- 172.21.15.122:0/2119828158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f007c0051c0 0x7f009014d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 >> 172.21.15.122:0/2119828158 conn(0x7f00900b3800 msgr2=0x7f0090070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 shutdown_connections 2024-03-06T07:34:20.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.087+0000 7f0098288640 1 -- 172.21.15.122:0/2119828158 wait complete. 2024-03-06T07:34:20.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:34:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.246+0000 7f1de5c2a640 1 Processor -- start 2024-03-06T07:34:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.246+0000 7f1de5c2a640 1 -- start start 2024-03-06T07:34:20.249 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1de5c2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1de015ac00 0x7f1de006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1de5c2a640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1de00708b0 0x7f1de0070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1de5c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015e7a0 0x7f1de015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1de5c2a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de0071250 con 0x7f1de015ac00 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1de5c2a640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00cdc10 con 0x7f1de00708b0 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1de5c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015afe0 con 0x7f1de015e7a0 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1de015ac00 0x7f1de006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1de015ac00 0x7f1de006c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60650/0 (socket says 172.21.15.122:60650) 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 -- 172.21.15.122:0/1024284228 learned_addr learned my addr 172.21.15.122:0/1024284228 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 -- 172.21.15.122:0/1024284228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015e7a0 msgr2=0x7f1de015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/1024284228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015e7a0 0x7f1de015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 -- 172.21.15.122:0/1024284228 >> v2:172.21.15.169:3300/0 conn(0x7f1de00708b0 msgr2=0x7f1de0070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddeffd640 1 --2- 172.21.15.122:0/1024284228 >> v2:172.21.15.169:3300/0 conn(0x7f1de00708b0 0x7f1de0070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/1024284228 >> v2:172.21.15.169:3300/0 conn(0x7f1de00708b0 0x7f1de0070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.247+0000 7f1ddf7fe640 1 -- 172.21.15.122:0/1024284228 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00cd000 con 0x7f1de015ac00 2024-03-06T07:34:20.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.248+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/1024284228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015ac00 0x7f1de006c910 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f1de0074f80 tx=0x7f1dd002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ef174bf20c193c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.252 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.248+0000 7f1dde7fc640 1 -- 172.21.15.122:0/1024284228 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd002e070 con 0x7f1de015ac00 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.248+0000 7f1dde7fc640 1 -- 172.21.15.122:0/1024284228 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0037070 con 0x7f1de015ac00 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.248+0000 7f1dde7fc640 1 -- 172.21.15.122:0/1024284228 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0031a00 con 0x7f1de015ac00 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 -- 172.21.15.122:0/1024284228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015ac00 msgr2=0x7f1de006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/1024284228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015ac00 0x7f1de006c910 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f1de0074f80 tx=0x7f1dd002b880 comp rx=0 tx=0).stop 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 -- 172.21.15.122:0/1024284228 shutdown_connections 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/1024284228 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015e7a0 0x7f1de015ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/1024284228 >> v2:172.21.15.169:3300/0 conn(0x7f1de00708b0 0x7f1de0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/1024284228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015ac00 0x7f1de006c910 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 -- 172.21.15.122:0/1024284228 >> 172.21.15.122:0/1024284228 conn(0x7f1de00b1650 msgr2=0x7f1de00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 -- 172.21.15.122:0/1024284228 shutdown_connections 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.249+0000 7f1de5c2a640 1 -- 172.21.15.122:0/1024284228 wait complete. 2024-03-06T07:34:20.253 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.250+0000 7f1de5c2a640 1 Processor -- start 2024-03-06T07:34:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.250+0000 7f1de5c2a640 1 -- start start 2024-03-06T07:34:20.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1de5c2a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de00708b0 0x7f1de0159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1de5c2a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015ac00 0x7f1de0143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1de5c2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 0x7f1de014d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1de5c2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00cd8a0 con 0x7f1de015e7a0 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1de5c2a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00cd5c0 con 0x7f1de00708b0 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1de5c2a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de006ce50 con 0x7f1de015ac00 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 0x7f1de014d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 0x7f1de014d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60662/0 (socket says 172.21.15.122:60662) 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 -- 172.21.15.122:0/3304454143 learned_addr learned my addr 172.21.15.122:0/3304454143 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddeffd640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015ac00 0x7f1de0143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 -- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015ac00 msgr2=0x7f1de0143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015ac00 0x7f1de0143560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 -- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de00708b0 msgr2=0x7f1de0159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de00708b0 0x7f1de0159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de00708b0 0x7f1de0159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 -- 172.21.15.122:0/3304454143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de014da40 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de00708b0 0x7f1de0159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.251+0000 7f1ddffff640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 0x7f1de014d1f0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b970 tx=0x7f1dd800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.252+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8019070 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.252+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd800cb60 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.252+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8014440 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.252+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de014dd20 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.252+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de014e2e0 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.253+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de00687d0 con 0x7f1de015e7a0 2024-03-06T07:34:20.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.253+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dd801c070 con 0x7f1de015e7a0 2024-03-06T07:34:20.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.253+0000 7f1dc7fff640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dcc06b4d0 0x7f1dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.254+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f1dd8089f80 con 0x7f1de015e7a0 2024-03-06T07:34:20.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.254+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dcc06b4d0 0x7f1dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.254+0000 7f1ddf7fe640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dcc06b4d0 0x7f1dcc06d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0036000 tx=0x7f1dd0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.258+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1dd8058dc0 con 0x7f1de015e7a0 2024-03-06T07:34:20.432 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:20.430+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1de00b13c0 con 0x7f1de015e7a0 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.148+0000 7f1dc7fff640 1 -- 172.21.15.122:0/3304454143 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1dd80586c0 con 0x7f1de015e7a0 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.153+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dcc06b4d0 msgr2=0x7f1dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.153+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dcc06b4d0 0x7f1dcc06d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0036000 tx=0x7f1dd0008030 comp rx=0 tx=0).stop 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.153+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 msgr2=0x7f1de014d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.153+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 0x7f1de014d1f0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b970 tx=0x7f1dd800be30 comp rx=0 tx=0).stop 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.153+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 shutdown_connections 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1dcc06b4d0 0x7f1dcc06d990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1de015e7a0 0x7f1de014d1f0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1de015ac00 0x7f1de0143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 --2- 172.21.15.122:0/3304454143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1de00708b0 0x7f1de0159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 >> 172.21.15.122:0/3304454143 conn(0x7f1de00b1650 msgr2=0x7f1de00af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 shutdown_connections 2024-03-06T07:34:21.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.154+0000 7f1de5c2a640 1 -- 172.21.15.122:0/3304454143 wait complete. 2024-03-06T07:34:21.171 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:34:21.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.279+0000 7f0f628fb640 1 Processor -- start 2024-03-06T07:34:21.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.279+0000 7f0f628fb640 1 -- start start 2024-03-06T07:34:21.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f628fb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f0f5c15c5a0 0x7f0f5c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f628fb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c0ce8c0 0x7f0f5c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f628fb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0f5c15c980 0x7f0f5c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f628fb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c072a20 con 0x7f0f5c15c980 2024-03-06T07:34:21.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f628fb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c0708b0 con 0x7f0f5c15c5a0 2024-03-06T07:34:21.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f628fb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c070e70 con 0x7f0f5c0ce8c0 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0f5c15c980 0x7f0f5c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0f5c15c980 0x7f0f5c071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60678/0 (socket says 172.21.15.122:60678) 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 -- 172.21.15.122:0/476568094 learned_addr learned my addr 172.21.15.122:0/476568094 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 -- 172.21.15.122:0/476568094 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c0ce8c0 msgr2=0x7f0f5c0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 --2- 172.21.15.122:0/476568094 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c0ce8c0 0x7f0f5c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 -- 172.21.15.122:0/476568094 >> v2:172.21.15.169:3300/0 conn(0x7f0f5c15c5a0 msgr2=0x7f0f5c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f5bfff640 1 --2- 172.21.15.122:0/476568094 >> v2:172.21.15.169:3300/0 conn(0x7f0f5c15c5a0 0x7f0f5c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 --2- 172.21.15.122:0/476568094 >> v2:172.21.15.169:3300/0 conn(0x7f0f5c15c5a0 0x7f0f5c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f60e98640 1 -- 172.21.15.122:0/476568094 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c06ce50 con 0x7f0f5c15c980 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.280+0000 7f0f5bfff640 1 --2- 172.21.15.122:0/476568094 >> v2:172.21.15.169:3300/0 conn(0x7f0f5c15c5a0 0x7f0f5c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.281+0000 7f0f60e98640 1 --2- 172.21.15.122:0/476568094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c071300 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c06bf40 tx=0x7f0f5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee4143be44c64ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.281+0000 7f0f5affd640 1 -- 172.21.15.122:0/476568094 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5402c9e0 con 0x7f0f5c15c980 2024-03-06T07:34:21.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.281+0000 7f0f5affd640 1 -- 172.21.15.122:0/476568094 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5402cb80 con 0x7f0f5c15c980 2024-03-06T07:34:21.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.281+0000 7f0f5affd640 1 -- 172.21.15.122:0/476568094 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f540343f0 con 0x7f0f5c15c980 2024-03-06T07:34:21.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 -- 172.21.15.122:0/476568094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 msgr2=0x7f0f5c071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 --2- 172.21.15.122:0/476568094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c071300 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c06bf40 tx=0x7f0f5402be40 comp rx=0 tx=0).stop 2024-03-06T07:34:21.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 -- 172.21.15.122:0/476568094 shutdown_connections 2024-03-06T07:34:21.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 --2- 172.21.15.122:0/476568094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c071300 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 --2- 172.21.15.122:0/476568094 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c0ce8c0 0x7f0f5c0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 --2- 172.21.15.122:0/476568094 >> v2:172.21.15.169:3300/0 conn(0x7f0f5c15c5a0 0x7f0f5c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 -- 172.21.15.122:0/476568094 >> 172.21.15.122:0/476568094 conn(0x7f0f5c0b1650 msgr2=0x7f0f5c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 -- 172.21.15.122:0/476568094 shutdown_connections 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.282+0000 7f0f628fb640 1 -- 172.21.15.122:0/476568094 wait complete. 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.283+0000 7f0f628fb640 1 Processor -- start 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.283+0000 7f0f628fb640 1 -- start start 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c15c5a0 0x7f0f5c13d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c0ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f5c0aecc0 0x7f0f5c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c070b00 con 0x7f0f5c15c980 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0cf030 con 0x7f0f5c0aecc0 2024-03-06T07:34:21.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0cc870 con 0x7f0f5c15c5a0 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c0ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c0ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60690/0 (socket says 172.21.15.122:60690) 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 -- 172.21.15.122:0/1510823989 learned_addr learned my addr 172.21.15.122:0/1510823989 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f60e98640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f5c0aecc0 0x7f0f5c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5bfff640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c15c5a0 0x7f0f5c13d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 -- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c15c5a0 msgr2=0x7f0f5c13d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c15c5a0 0x7f0f5c13d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 -- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f5c0aecc0 msgr2=0x7f0f5c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f5c0aecc0 0x7f0f5c0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 -- 172.21.15.122:0/1510823989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c0af1b0 con 0x7f0f5c15c980 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f60e98640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f5c0aecc0 0x7f0f5c0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5bfff640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c15c5a0 0x7f0f5c13d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f5b7fe640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c0ae780 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f0f5000b440 tx=0x7f0f5000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50015020 con 0x7f0f5c15c980 2024-03-06T07:34:21.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f50011890 con 0x7f0f5c15c980 2024-03-06T07:34:21.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c156730 con 0x7f0f5c15c980 2024-03-06T07:34:21.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.284+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50011c80 con 0x7f0f5c15c980 2024-03-06T07:34:21.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.285+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c156cb0 con 0x7f0f5c15c980 2024-03-06T07:34:21.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.286+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c0687d0 con 0x7f0f5c15c980 2024-03-06T07:34:21.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.286+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f50018020 con 0x7f0f5c15c980 2024-03-06T07:34:21.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.286+0000 7f0f58ff9640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f4806b4d0 0x7f0f4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.286+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5472+0+0 (secure 0 0 0) 0x7f0f5008a170 con 0x7f0f5c15c980 2024-03-06T07:34:21.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.286+0000 7f0f5bfff640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f4806b4d0 0x7f0f4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.287+0000 7f0f5bfff640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f4806b4d0 0x7f0f4806d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c006fd0 tx=0x7f0f4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.291+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f5001f020 con 0x7f0f5c15c980 2024-03-06T07:34:21.465 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:21.463+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0f5c0cc090 con 0x7f0f5c15c980 2024-03-06T07:34:22.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.162+0000 7f0f58ff9640 1 -- 172.21.15.122:0/1510823989 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0f5000d040 con 0x7f0f5c15c980 2024-03-06T07:34:22.165 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'cephfs_data' removed 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.166+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f4806b4d0 msgr2=0x7f0f4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.166+0000 7f0f628fb640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f4806b4d0 0x7f0f4806d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c006fd0 tx=0x7f0f4c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.166+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 msgr2=0x7f0f5c0ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c0ae780 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f0f5000b440 tx=0x7f0f5000b900 comp rx=0 tx=0).stop 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 shutdown_connections 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f0f4806b4d0 0x7f0f4806d990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f5c0aecc0 0x7f0f5c0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f5c15c980 0x7f0f5c0ae780 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 --2- 172.21.15.122:0/1510823989 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f0f5c15c5a0 0x7f0f5c13d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 >> 172.21.15.122:0/1510823989 conn(0x7f0f5c0b1650 msgr2=0x7f0f5c06d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 shutdown_connections 2024-03-06T07:34:22.170 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.167+0000 7f0f628fb640 1 -- 172.21.15.122:0/1510823989 wait complete. 2024-03-06T07:34:22.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:34:22.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.282+0000 7f876ebbc640 1 Processor -- start 2024-03-06T07:34:22.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876ebbc640 1 -- start start 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876ebbc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87680d4860 0x7f87680d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876ebbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680b0e30 0x7f87680b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876ebbc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87681618b0 con 0x7f87681614d0 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876ebbc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87680d4cf0 con 0x7f87680d4860 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876ebbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680cf400 con 0x7f87680b0e30 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f8767fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f87680d4860 0x7f87680d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876c958640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f87681614d0 0x7f87680d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51106/0 (socket says 172.21.15.122:51106) 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.283+0000 7f876c958640 1 -- 172.21.15.122:0/2247661351 learned_addr learned my addr 172.21.15.122:0/2247661351 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:22.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.284+0000 7f8767fff640 1 -- 172.21.15.122:0/2247661351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680b0e30 msgr2=0x7f87680b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.284+0000 7f8767fff640 1 --2- 172.21.15.122:0/2247661351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680b0e30 0x7f87680b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.284+0000 7f8767fff640 1 -- 172.21.15.122:0/2247661351 >> v1:172.21.15.122:6789/0 conn(0x7f87681614d0 legacy=0x7f87680d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.284+0000 7f8767fff640 1 -- 172.21.15.122:0/2247661351 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680cf6a0 con 0x7f87680d4860 2024-03-06T07:34:22.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.284+0000 7f87677fe640 1 -- 172.21.15.122:0/2247661351 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955711909 0 0) 0x7f87681618b0 con 0x7f87681614d0 2024-03-06T07:34:22.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.284+0000 7f8767fff640 1 --2- 172.21.15.122:0/2247661351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680d2020 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2905b5e9c7c0919 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f87677fe640 1 -- 172.21.15.122:0/2247661351 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c035020 con 0x7f87680d4860 2024-03-06T07:34:22.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f87677fe640 1 -- 172.21.15.122:0/2247661351 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c031830 con 0x7f87680d4860 2024-03-06T07:34:22.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f87677fe640 1 -- 172.21.15.122:0/2247661351 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c031c00 con 0x7f87680d4860 2024-03-06T07:34:22.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 -- 172.21.15.122:0/2247661351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 msgr2=0x7f87680d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 --2- 172.21.15.122:0/2247661351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680d2020 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 -- 172.21.15.122:0/2247661351 shutdown_connections 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 --2- 172.21.15.122:0/2247661351 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87680b0e30 0x7f87680b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 --2- 172.21.15.122:0/2247661351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680d2020 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 -- 172.21.15.122:0/2247661351 >> 172.21.15.122:0/2247661351 conn(0x7f87680d3d40 msgr2=0x7f87680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 -- 172.21.15.122:0/2247661351 shutdown_connections 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.285+0000 7f876ebbc640 1 -- 172.21.15.122:0/2247661351 wait complete. 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.286+0000 7f876ebbc640 1 Processor -- start 2024-03-06T07:34:22.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.286+0000 7f876ebbc640 1 -- start start 2024-03-06T07:34:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876ebbc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87680b0e30 0x7f87680cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876ebbc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876ebbc640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87681614d0 0x7f87680b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876ebbc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680b0910 con 0x7f87680d4860 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876ebbc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680cfc60 con 0x7f87680b0e30 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876ebbc640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680d2560 con 0x7f87681614d0 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f8767fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f8767fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60704/0 (socket says 172.21.15.122:60704) 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f8767fff640 1 -- 172.21.15.122:0/3857885711 learned_addr learned my addr 172.21.15.122:0/3857885711 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.287+0000 7f876d159640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87681614d0 0x7f87680b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f876c958640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87680b0e30 0x7f87680cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f8767fff640 1 -- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87681614d0 msgr2=0x7f87680b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f8767fff640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87681614d0 0x7f87680b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f8767fff640 1 -- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87680b0e30 msgr2=0x7f87680cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f8767fff640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87680b0e30 0x7f87680cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f8767fff640 1 -- 172.21.15.122:0/3857885711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680b9800 con 0x7f87680d4860 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f876d159640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87681614d0 0x7f87680b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:22.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f876c958640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87680b0e30 0x7f87680cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f8767fff640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680b6930 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f875c0098c0 tx=0x7f875c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c004230 con 0x7f87680d4860 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c0043d0 con 0x7f87680d4860 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87680b78a0 con 0x7f87680d4860 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c03cad0 con 0x7f87680d4860 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.288+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87680b92e0 con 0x7f87680d4860 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.289+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680b94b0 con 0x7f87680d4860 2024-03-06T07:34:22.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.289+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f875c038020 con 0x7f87680d4860 2024-03-06T07:34:22.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.290+0000 7f87657fa640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f875406b4d0 0x7f875406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.290+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f875c0ac210 con 0x7f87680d4860 2024-03-06T07:34:22.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.290+0000 7f876c958640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f875406b4d0 0x7f875406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.290+0000 7f876c958640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f875406b4d0 0x7f875406d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8758000f60 tx=0x7f8758009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.294+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87680b78a0 con 0x7f87680d4860 2024-03-06T07:34:22.470 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.468+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8768068ff0 con 0x7f87680d4860 2024-03-06T07:34:22.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.471+0000 7f87657fa640 1 -- 172.21.15.122:0/3857885711 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f875c041070 con 0x7f87680d4860 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.473+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f875406b4d0 msgr2=0x7f875406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.473+0000 7f876ebbc640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f875406b4d0 0x7f875406d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8758000f60 tx=0x7f8758009040 comp rx=0 tx=0).stop 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.473+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 msgr2=0x7f87680b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.473+0000 7f876ebbc640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680b6930 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f875c0098c0 tx=0x7f875c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 shutdown_connections 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f875406b4d0 0x7f875406d990 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f87681614d0 0x7f87680b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87680d4860 0x7f87680b6930 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 --2- 172.21.15.122:0/3857885711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87680b0e30 0x7f87680cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 >> 172.21.15.122:0/3857885711 conn(0x7f87680d3d40 msgr2=0x7f8768160360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 shutdown_connections 2024-03-06T07:34:22.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.474+0000 7f876ebbc640 1 -- 172.21.15.122:0/3857885711 wait complete. 2024-03-06T07:34:22.487 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:34:22.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.584+0000 7f86f5bf7640 1 Processor -- start 2024-03-06T07:34:22.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.584+0000 7f86f5bf7640 1 -- start start 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86f5bf7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 0x7f86f015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86f5bf7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f015f580 con 0x7f86f013dc10 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86f5bf7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f015b450 con 0x7f86f00b3060 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86f5bf7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00cf3f0 con 0x7f86f015bf30 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86ef7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f86f013dc10 0x7f86f00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51110/0 (socket says 172.21.15.122:51110) 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86ef7fe640 1 -- 172.21.15.122:0/3755781775 learned_addr learned my addr 172.21.15.122:0/3755781775 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86effff640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 0x7f86f015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173719563 0 0) 0x7f86f015f580 con 0x7f86f013dc10 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.585+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d8003520 con 0x7f86f013dc10 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1282803700 0 0) 0x7f86d8003520 con 0x7f86f013dc10 2024-03-06T07:34:22.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 msgr2=0x7f86f015ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 0x7f86f015ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 >> v1:172.21.15.169:6789/0 conn(0x7f86f00b3060 legacy=0x7f86f015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f015b730 con 0x7f86f013dc10 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f86e0001670 con 0x7f86f013dc10 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 >> v1:172.21.15.122:6789/0 conn(0x7f86f013dc10 legacy=0x7f86f00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86d80039e0 0x7f86d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 0x7f86d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86d8004d40 0x7f86d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00cf3f0 con 0x7f86d8004d40 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d800baf0 con 0x7f86d80039e0 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d800bcf0 con 0x7f86f015bf30 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86effff640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86d8004d40 0x7f86d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86ef7fe640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86d80039e0 0x7f86d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86eeffd640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 0x7f86d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86effff640 1 -- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 msgr2=0x7f86d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86effff640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f015bf30 0x7f86d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86effff640 1 -- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86d80039e0 msgr2=0x7f86d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86effff640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86d80039e0 0x7f86d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.586+0000 7f86effff640 1 -- 172.21.15.122:0/3755781775 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d8003520 con 0x7f86d8004d40 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.587+0000 7f86effff640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86d8004d40 0x7f86d80051c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009fc0 tx=0x7f86d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9fe7e157e2c3f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.587+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8039070 con 0x7f86d8004d40 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.587+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e802cc20 con 0x7f86d8004d40 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.587+0000 7f86ee7fc640 1 -- 172.21.15.122:0/3755781775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8009c10 con 0x7f86d8004d40 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86f5bf7640 1 -- 172.21.15.122:0/3755781775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86d8004d40 msgr2=0x7f86d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/3755781775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86d8004d40 0x7f86d80051c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009fc0 tx=0x7f86d8005830 comp rx=0 tx=0).stop 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86effff640 1 -- 172.21.15.122:0/3755781775 reap_dead start 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86f5bf7640 1 -- 172.21.15.122:0/3755781775 shutdown_connections 2024-03-06T07:34:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86f5bf7640 1 -- 172.21.15.122:0/3755781775 >> 172.21.15.122:0/3755781775 conn(0x7f86f00cf8a0 msgr2=0x7f86f00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86f5bf7640 1 -- 172.21.15.122:0/3755781775 shutdown_connections 2024-03-06T07:34:22.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.588+0000 7f86f5bf7640 1 -- 172.21.15.122:0/3755781775 wait complete. 2024-03-06T07:34:22.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.589+0000 7f86f5bf7640 1 Processor -- start 2024-03-06T07:34:22.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.589+0000 7f86f5bf7640 1 -- start start 2024-03-06T07:34:22.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f0154530 0x7f86f0143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 0x7f86f0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86f015a270 0x7f86f0142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f015ef60 con 0x7f86f01f7870 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f015b210 con 0x7f86f015a270 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f015f0e0 con 0x7f86f0154530 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 0x7f86f0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 0x7f86f0159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60720/0 (socket says 172.21.15.122:60720) 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 -- 172.21.15.122:0/1989198701 learned_addr learned my addr 172.21.15.122:0/1989198701 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86ef7fe640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f0154530 0x7f86f0143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 -- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f0154530 msgr2=0x7f86f0143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f0154530 0x7f86f0143ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86effff640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86f015a270 0x7f86f0142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 -- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86f015a270 msgr2=0x7f86f0142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86f015a270 0x7f86f0142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 -- 172.21.15.122:0/1989198701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f01f4300 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86eeffd640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 0x7f86f0159d30 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f86e400f820 tx=0x7f86e400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e40050d0 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e4018070 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.590+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f01f7d10 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.591+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e400d730 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.591+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f01f82d0 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.591+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f015a680 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.592+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86e400d8d0 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.592+0000 7f86d7fff640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86dc06b500 0x7f86dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.592+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f86e4089d80 con 0x7f86f01f7870 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.592+0000 7f86ef7fe640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86dc06b500 0x7f86dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.593+0000 7f86ef7fe640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86dc06b500 0x7f86dc06d9c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f86e0007c70 tx=0x7f86e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.597+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86f015a680 con 0x7f86f01f7870 2024-03-06T07:34:22.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:22.839+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7f86f004f330 con 0x7f86f01f7870 2024-03-06T07:34:23.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.164+0000 7f86d7fff640 1 -- 172.21.15.122:0/1989198701 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v653) v1 ==== 141+0+0 (secure 0 0 0) 0x7f86e401a310 con 0x7f86f01f7870 2024-03-06T07:34:23.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86dc06b500 msgr2=0x7f86dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86dc06b500 0x7f86dc06d9c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f86e0007c70 tx=0x7f86e0009040 comp rx=0 tx=0).stop 2024-03-06T07:34:23.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 msgr2=0x7f86f0159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 0x7f86f0159d30 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f86e400f820 tx=0x7f86e400fce0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 shutdown_connections 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f86dc06b500 0x7f86dc06d9c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86f015a270 0x7f86f0142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86f01f7870 0x7f86f0159d30 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 --2- 172.21.15.122:0/1989198701 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f86f0154530 0x7f86f0143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 >> 172.21.15.122:0/1989198701 conn(0x7f86f00cf8a0 msgr2=0x7f86f01402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 shutdown_connections 2024-03-06T07:34:23.169 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:23.166+0000 7f86f5bf7640 1 -- 172.21.15.122:0/1989198701 wait complete. 2024-03-06T07:34:23.180 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays ... ERROR 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:23.181 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:Ran 9 tests in 719.731s 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:34:23.182 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:34:23.183 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:23.183 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:23.370 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=92d5f02d6a8f48f9825ab2f8d6141c29 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:23.372 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:34:23.381 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-06T07:34:23.406 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-06T07:34:23.407 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:34:23.407 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:23.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:23.426 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-06T07:34:23.426 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:23.482 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15661 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:23.482 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:34:23.482 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:23.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:23.501 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-06T07:34:23.501 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:23.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19660 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:23.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.559 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-06T07:34:23.621 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:34:23.621 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:34:23.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:34:23.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:34:23.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link delete ceph-brx 2024-03-06T07:34:23.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.865 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-06T07:34:23.880 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:23.881 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:23.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:23.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:23.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:24.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:24.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link delete ceph-brx 2024-03-06T07:34:24.113 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:34:24.188 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:34:24.189 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.288+0000 7f4d5d0bb640 1 Processor -- start 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.288+0000 7f4d5d0bb640 1 -- start start 2024-03-06T07:34:24.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5d0bb640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f4d5815bf30 0x7f4d5815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5d0bb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5815f580 con 0x7f4d580b3060 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5d0bb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5815b450 con 0x7f4d5813dc10 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5d0bb640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d580cf3f0 con 0x7f4d5815bf30 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4d580b3060 0x7f4d5815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51120/0 (socket says 172.21.15.122:51120) 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5659c640 1 -- 172.21.15.122:0/1682448983 learned_addr learned my addr 172.21.15.122:0/1682448983 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d5759e640 1 --2- 172.21.15.122:0/1682448983 >> v2:172.21.15.169:3300/0 conn(0x7f4d5815bf30 0x7f4d5815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377763905 0 0) 0x7f4d5815f580 con 0x7f4d580b3060 2024-03-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.289+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d40005730 con 0x7f4d580b3060 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683834529 0 0) 0x7f4d40005730 con 0x7f4d580b3060 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 >> v2:172.21.15.169:3300/0 conn(0x7f4d5815bf30 msgr2=0x7f4d5815ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 --2- 172.21.15.122:0/1682448983 >> v2:172.21.15.169:3300/0 conn(0x7f4d5815bf30 0x7f4d5815ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 >> v1:172.21.15.169:6789/0 conn(0x7f4d5813dc10 legacy=0x7f4d580b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5815b730 con 0x7f4d580b3060 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f4d4c001670 con 0x7f4d580b3060 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 >> v1:172.21.15.122:6789/0 conn(0x7f4d580b3060 legacy=0x7f4d5815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d40005bc0 0x7f4d40005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d40006a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d40007330 0x7f4d4000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580cf3f0 con 0x7f4d400065c0 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4000e270 con 0x7f4d40005bc0 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4000e470 con 0x7f4d40007330 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d40006a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d56d9d640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d40005bc0 0x7f4d40005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5759e640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d40007330 0x7f4d4000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5659c640 1 -- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d40007330 msgr2=0x7f4d4000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d40007330 0x7f4d4000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5659c640 1 -- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d40005bc0 msgr2=0x7f4d40005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d40005bc0 0x7f4d40005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5659c640 1 -- 172.21.15.122:0/1682448983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40005730 con 0x7f4d400065c0 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5759e640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d40007330 0x7f4d4000dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.290+0000 7f4d5759e640 1 -- 172.21.15.122:0/1682448983 reap_dead start 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d40006a90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c240 tx=0x7f4d40007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d5bb1810861461 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c007d10 con 0x7f4d400065c0 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c030070 con 0x7f4d400065c0 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d55d9b640 1 -- 172.21.15.122:0/1682448983 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c005d70 con 0x7f4d400065c0 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1682448983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 msgr2=0x7f4d40006a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d40006a90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c240 tx=0x7f4d40007a40 comp rx=0 tx=0).stop 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1682448983 shutdown_connections 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1682448983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d40006a90 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1682448983 >> 172.21.15.122:0/1682448983 conn(0x7f4d580cf8a0 msgr2=0x7f4d580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1682448983 shutdown_connections 2024-03-06T07:34:24.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.291+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1682448983 wait complete. 2024-03-06T07:34:24.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.293+0000 7f4d5d0bb640 1 Processor -- start 2024-03-06T07:34:24.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.293+0000 7f4d5d0bb640 1 -- start start 2024-03-06T07:34:24.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d58138df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d5813fd60 0x7f4d5813e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d5813e740 0x7f4d58071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580b2950 con 0x7f4d400065c0 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5815b210 con 0x7f4d5813fd60 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580b2ad0 con 0x7f4d5813e740 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d58138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d58138df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60734/0 (socket says 172.21.15.122:60734) 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 -- 172.21.15.122:0/1309116809 learned_addr learned my addr 172.21.15.122:0/1309116809 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5759e640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d5813e740 0x7f4d58071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d5813fd60 0x7f4d5813e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 -- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d5813e740 msgr2=0x7f4d58071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d5813e740 0x7f4d58071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 -- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d5813fd60 msgr2=0x7f4d5813e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d5813fd60 0x7f4d5813e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 -- 172.21.15.122:0/1309116809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5813ecc0 con 0x7f4d400065c0 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d56d9d640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d58138df0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800f7c0 tx=0x7f4d4800fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800fe70 con 0x7f4d400065c0 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d48018070 con 0x7f4d400065c0 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5813efa0 con 0x7f4d400065c0 2024-03-06T07:34:24.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800d730 con 0x7f4d400065c0 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.294+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5813f560 con 0x7f4d400065c0 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.295+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5815e350 con 0x7f4d400065c0 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.296+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d4800d8d0 con 0x7f4d400065c0 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.296+0000 7f4d3f7fe640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d4406b4d0 0x7f4d4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.296+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4d48089d70 con 0x7f4d400065c0 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.296+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d4406b4d0 0x7f4d4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.297+0000 7f4d5659c640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d4406b4d0 0x7f4d4406d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c034000 tx=0x7f4d4c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.300+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d5815e350 con 0x7f4d400065c0 2024-03-06T07:34:24.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.472+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d5815a850 con 0x7f4d4406b4d0 2024-03-06T07:34:24.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.472+0000 7f4d3f7fe640 1 -- 172.21.15.122:0/1309116809 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13433 (secure 0 0 0) 0x7f4d4c048490 con 0x7f4d4406b4d0 2024-03-06T07:34:24.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:24.475 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:34:24.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.474+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d4406b4d0 msgr2=0x7f4d4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.474+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d4406b4d0 0x7f4d4406d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c034000 tx=0x7f4d4c007000 comp rx=0 tx=0).stop 2024-03-06T07:34:24.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.474+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 msgr2=0x7f4d58138df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.474+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d58138df0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800f7c0 tx=0x7f4d4800fc80 comp rx=0 tx=0).stop 2024-03-06T07:34:24.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 shutdown_connections 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f4d4406b4d0 0x7f4d4406d990 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f4d5813e740 0x7f4d58071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d5813fd60 0x7f4d5813e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 --2- 172.21.15.122:0/1309116809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d400065c0 0x7f4d58138df0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 >> 172.21.15.122:0/1309116809 conn(0x7f4d580cf8a0 msgr2=0x7f4d580ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 shutdown_connections 2024-03-06T07:34:24.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.475+0000 7f4d5d0bb640 1 -- 172.21.15.122:0/1309116809 wait complete. 2024-03-06T07:34:24.487 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-03-06T07:34:23.472017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331408,"kb_used_data":15408,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643312,"statfs":{"total":773094113280,"available":772754751488,"internally_reserved":0,"allocated":15777792,"data_stored":7584088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-141,"num_write_kb":-85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001112"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:34:19.249226+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:34:19.249226+0000","last_peered":"2024-03-06T07:34:19.249226+0000","last_clean":"2024-03-06T07:34:19.249226+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:34:19.249226+0000","last_undegraded":"2024-03-06T07:34:19.249226+0000","last_fullsized":"2024-03-06T07:34:19.249226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575003,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2424832,"data_stored":1393697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41264,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330576,"statfs":{"total":96636764160,"available":96594509824,"internally_reserved":0,"allocated":1806336,"data_stored":791305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1802240,"data_stored":791283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41316,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330524,"statfs":{"total":96636764160,"available":96594456576,"internally_reserved":0,"allocated":1859584,"data_stored":826337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41272,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330568,"statfs":{"total":96636764160,"available":96594501632,"internally_reserved":0,"allocated":1814528,"data_stored":791977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575005,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2433024,"data_stored":1405343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:24.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:24.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.585+0000 7f794f1d0640 1 Processor -- start 2024-03-06T07:34:24.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.585+0000 7f794f1d0640 1 -- start start 2024-03-06T07:34:24.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794f1d0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f794813dc10 0x7f79480b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794f1d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794815ef60 0x7f79480b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794f1d0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794815f580 con 0x7f794815bf30 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794f1d0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79480cf3f0 con 0x7f794813dc10 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794f1d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794815b210 con 0x7f794815ef60 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f794813dc10 0x7f79480b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f794813dc10 0x7f79480b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44934/0 (socket says 172.21.15.122:44934) 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 -- 172.21.15.122:0/587591258 learned_addr learned my addr 172.21.15.122:0/587591258 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:24.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794d76d640 1 --2- 172.21.15.122:0/587591258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794815ef60 0x7f79480b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 -- 172.21.15.122:0/587591258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794815ef60 msgr2=0x7f79480b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 --2- 172.21.15.122:0/587591258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794815ef60 0x7f79480b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 -- 172.21.15.122:0/587591258 >> v1:172.21.15.169:6789/0 conn(0x7f794815bf30 legacy=0x7f794815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.586+0000 7f794cf6c640 1 -- 172.21.15.122:0/587591258 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794815b450 con 0x7f794813dc10 2024-03-06T07:34:24.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.587+0000 7f794cf6c640 1 --2- 172.21.15.122:0/587591258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794813dc10 0x7f79480b2950 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f793000bb10 tx=0x7f793002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363c543c0c8bc846 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.587+0000 7f79477fe640 1 -- 172.21.15.122:0/587591258 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793003b070 con 0x7f794813dc10 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.587+0000 7f79477fe640 1 -- 172.21.15.122:0/587591258 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793002ede0 con 0x7f794813dc10 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.587+0000 7f79477fe640 1 -- 172.21.15.122:0/587591258 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930036660 con 0x7f794813dc10 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 -- 172.21.15.122:0/587591258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794813dc10 msgr2=0x7f79480b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 --2- 172.21.15.122:0/587591258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794813dc10 0x7f79480b2950 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f793000bb10 tx=0x7f793002e030 comp rx=0 tx=0).stop 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 -- 172.21.15.122:0/587591258 shutdown_connections 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 --2- 172.21.15.122:0/587591258 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794815ef60 0x7f79480b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 --2- 172.21.15.122:0/587591258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794813dc10 0x7f79480b2950 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 -- 172.21.15.122:0/587591258 >> 172.21.15.122:0/587591258 conn(0x7f79480cf8a0 msgr2=0x7f79480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 -- 172.21.15.122:0/587591258 shutdown_connections 2024-03-06T07:34:24.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.588+0000 7f794f1d0640 1 -- 172.21.15.122:0/587591258 wait complete. 2024-03-06T07:34:24.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.589+0000 7f794f1d0640 1 Processor -- start 2024-03-06T07:34:24.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.589+0000 7f794f1d0640 1 -- start start 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794f1d0640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794813dc10 0x7f7948159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794f1d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 0x7f7948154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794f1d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f794815ef60 0x7f79481552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794f1d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79480cd7a0 con 0x7f794815bf30 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794f1d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794813dff0 con 0x7f794815ef60 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794f1d0640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79480cd920 con 0x7f794813dc10 2024-03-06T07:34:24.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 0x7f7948154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 0x7f7948154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44936/0 (socket says 172.21.15.122:44936) 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 -- 172.21.15.122:0/119395278 learned_addr learned my addr 172.21.15.122:0/119395278 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 -- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794813dc10 msgr2=0x7f7948159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794cf6c640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794813dc10 0x7f7948159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f794d76d640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f794815ef60 0x7f79481552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794813dc10 0x7f7948159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 -- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f794815ef60 msgr2=0x7f79481552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f794815ef60 0x7f79481552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 -- 172.21.15.122:0/119395278 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794813e210 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f7947fff640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 0x7f7948154d60 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f793800c430 tx=0x7f793800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.590+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938011790 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.591+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7938017070 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.591+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794813e4f0 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.591+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938011bb0 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.591+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794813eab0 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.592+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794804f330 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.592+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f793801c460 con 0x7f794815bf30 2024-03-06T07:34:24.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.592+0000 7f79457fa640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f792806b4d0 0x7f792806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.592+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7938089e20 con 0x7f794815bf30 2024-03-06T07:34:24.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.592+0000 7f794cf6c640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f792806b4d0 0x7f792806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.593+0000 7f794cf6c640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f792806b4d0 0x7f792806d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7930038000 tx=0x7f7930008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.597+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7938058f40 con 0x7f794815bf30 2024-03-06T07:34:24.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.767+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79480cd1d0 con 0x7f792806b4d0 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.768+0000 7f79457fa640 1 -- 172.21.15.122:0/119395278 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13433 (secure 0 0 0) 0x7f7930048490 con 0x7f792806b4d0 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:24.770 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:34:24.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f792806b4d0 msgr2=0x7f792806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f792806b4d0 0x7f792806d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7930038000 tx=0x7f7930008040 comp rx=0 tx=0).stop 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 msgr2=0x7f7948154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 0x7f7948154d60 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f793800c430 tx=0x7f793800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 shutdown_connections 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f792806b4d0 0x7f792806d990 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f794815ef60 0x7f79481552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794815bf30 0x7f7948154d60 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 --2- 172.21.15.122:0/119395278 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f794813dc10 0x7f7948159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 >> 172.21.15.122:0/119395278 conn(0x7f79480cf8a0 msgr2=0x7f794815d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 shutdown_connections 2024-03-06T07:34:24.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.770+0000 7f794f1d0640 1 -- 172.21.15.122:0/119395278 wait complete. 2024-03-06T07:34:24.783 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-03-06T07:34:23.472017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331408,"kb_used_data":15408,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643312,"statfs":{"total":773094113280,"available":772754751488,"internally_reserved":0,"allocated":15777792,"data_stored":7584088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-141,"num_write_kb":-85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001112"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:34:19.249226+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:34:19.249226+0000","last_peered":"2024-03-06T07:34:19.249226+0000","last_clean":"2024-03-06T07:34:19.249226+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:34:19.249226+0000","last_undegraded":"2024-03-06T07:34:19.249226+0000","last_fullsized":"2024-03-06T07:34:19.249226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575003,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2424832,"data_stored":1393697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41264,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330576,"statfs":{"total":96636764160,"available":96594509824,"internally_reserved":0,"allocated":1806336,"data_stored":791305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1802240,"data_stored":791283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41316,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330524,"statfs":{"total":96636764160,"available":96594456576,"internally_reserved":0,"allocated":1859584,"data_stored":826337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41272,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330568,"statfs":{"total":96636764160,"available":96594501632,"internally_reserved":0,"allocated":1814528,"data_stored":791977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575005,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2433024,"data_stored":1405343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:24.783 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:34:24.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.880+0000 7fea9869d640 1 Processor -- start 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.880+0000 7fea9869d640 1 -- start start 2024-03-06T07:34:24.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea9869d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea9006da40 0x7fea900cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea9869d640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fea900cd7a0 0x7fea9015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea9869d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea900cc980 con 0x7fea901424f0 2024-03-06T07:34:24.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea9869d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea900cd220 con 0x7fea9006da40 2024-03-06T07:34:24.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea9869d640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea900ccc60 con 0x7fea900cd7a0 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea9006da40 0x7fea900cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fea9006da40 0x7fea900cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44946/0 (socket says 172.21.15.122:44946) 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 -- 172.21.15.122:0/2589112775 learned_addr learned my addr 172.21.15.122:0/2589112775 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea96c3a640 1 --2- 172.21.15.122:0/2589112775 >> v2:172.21.15.169:3300/0 conn(0x7fea900cd7a0 0x7fea9015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 -- 172.21.15.122:0/2589112775 >> v2:172.21.15.169:3300/0 conn(0x7fea900cd7a0 msgr2=0x7fea9015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 --2- 172.21.15.122:0/2589112775 >> v2:172.21.15.169:3300/0 conn(0x7fea900cd7a0 0x7fea9015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 -- 172.21.15.122:0/2589112775 >> v1:172.21.15.169:6789/0 conn(0x7fea901424f0 legacy=0x7fea9006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.881+0000 7fea95c38640 1 -- 172.21.15.122:0/2589112775 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900ccf40 con 0x7fea9006da40 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.882+0000 7fea95c38640 1 --2- 172.21.15.122:0/2589112775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea9006da40 0x7fea900cc2f0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fea90068700 tx=0x7fea8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8cb5d52dba33b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.882+0000 7fea95437640 1 -- 172.21.15.122:0/2589112775 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80035020 con 0x7fea9006da40 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.882+0000 7fea95437640 1 -- 172.21.15.122:0/2589112775 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80031870 con 0x7fea9006da40 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.882+0000 7fea95437640 1 -- 172.21.15.122:0/2589112775 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80031c40 con 0x7fea9006da40 2024-03-06T07:34:24.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 -- 172.21.15.122:0/2589112775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea9006da40 msgr2=0x7fea900cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.889 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 --2- 172.21.15.122:0/2589112775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea9006da40 0x7fea900cc2f0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fea90068700 tx=0x7fea8002b910 comp rx=0 tx=0).stop 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 -- 172.21.15.122:0/2589112775 shutdown_connections 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 --2- 172.21.15.122:0/2589112775 >> v2:172.21.15.169:3300/0 conn(0x7fea900cd7a0 0x7fea9015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 --2- 172.21.15.122:0/2589112775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea9006da40 0x7fea900cc2f0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 -- 172.21.15.122:0/2589112775 >> 172.21.15.122:0/2589112775 conn(0x7fea900cf8a0 msgr2=0x7fea900cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 -- 172.21.15.122:0/2589112775 shutdown_connections 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.883+0000 7fea9869d640 1 -- 172.21.15.122:0/2589112775 wait complete. 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.884+0000 7fea9869d640 1 Processor -- start 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.884+0000 7fea9869d640 1 -- start start 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.884+0000 7fea9869d640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea9006da40 0x7fea900b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.884+0000 7fea9869d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 0x7fea90155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea9869d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea901424f0 0x7fea9013bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea9869d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea90071ce0 con 0x7fea900cd7a0 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea9869d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea90071b60 con 0x7fea901424f0 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea9869d640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea90071e60 con 0x7fea9006da40 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 0x7fea90155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 0x7fea90155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44950/0 (socket says 172.21.15.122:44950) 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 -- 172.21.15.122:0/566751686 learned_addr learned my addr 172.21.15.122:0/566751686 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:24.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea96c3a640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea901424f0 0x7fea9013bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea96439640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea9006da40 0x7fea900b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 -- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea9006da40 msgr2=0x7fea900b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea9006da40 0x7fea900b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 -- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea901424f0 msgr2=0x7fea9013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea901424f0 0x7fea9013bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 -- 172.21.15.122:0/566751686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900b12c0 con 0x7fea900cd7a0 2024-03-06T07:34:24.892 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea96c3a640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea901424f0 0x7fea9013bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea95c38640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 0x7fea90155840 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fea8002b950 tx=0x7fea800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80040070 con 0x7fea900cd7a0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80030d00 con 0x7fea900cd7a0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea90158230 con 0x7fea900cd7a0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80035040 con 0x7fea900cd7a0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.885+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea901587f0 con 0x7fea900cd7a0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.886+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9004f330 con 0x7fea900cd7a0 2024-03-06T07:34:24.893 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.887+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea8004b060 con 0x7fea900cd7a0 2024-03-06T07:34:24.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.887+0000 7fea86ffd640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea7406b4d0 0x7fea7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.887+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fea800abd50 con 0x7fea900cd7a0 2024-03-06T07:34:24.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.887+0000 7fea96439640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea7406b4d0 0x7fea7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.888+0000 7fea96439640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea7406b4d0 0x7fea7406d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fea8c006fd0 tx=0x7fea8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.894 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:24.891+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea800af030 con 0x7fea900cd7a0 2024-03-06T07:34:25.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.058+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea900d0620 con 0x7fea7406b4d0 2024-03-06T07:34:25.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.059+0000 7fea86ffd640 1 -- 172.21.15.122:0/566751686 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13433 (secure 0 0 0) 0x7fea900d0620 con 0x7fea7406b4d0 2024-03-06T07:34:25.061 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:25.061 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:34:25.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea7406b4d0 msgr2=0x7fea7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea7406b4d0 0x7fea7406d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fea8c006fd0 tx=0x7fea8c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.063 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 msgr2=0x7fea90155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 0x7fea90155840 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fea8002b950 tx=0x7fea800046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 shutdown_connections 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fea7406b4d0 0x7fea7406d990 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea901424f0 0x7fea9013bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea900cd7a0 0x7fea90155840 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 --2- 172.21.15.122:0/566751686 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fea9006da40 0x7fea900b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 >> 172.21.15.122:0/566751686 conn(0x7fea900cf8a0 msgr2=0x7fea900721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 shutdown_connections 2024-03-06T07:34:25.064 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.061+0000 7fea9869d640 1 -- 172.21.15.122:0/566751686 wait complete. 2024-03-06T07:34:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-03-06T07:34:23.472017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331408,"kb_used_data":15408,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643312,"statfs":{"total":773094113280,"available":772754751488,"internally_reserved":0,"allocated":15777792,"data_stored":7584088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-141,"num_write_kb":-85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001112"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:34:19.249226+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:34:19.249226+0000","last_peered":"2024-03-06T07:34:19.249226+0000","last_clean":"2024-03-06T07:34:19.249226+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:34:19.249226+0000","last_undegraded":"2024-03-06T07:34:19.249226+0000","last_fullsized":"2024-03-06T07:34:19.249226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575003,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2424832,"data_stored":1393697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41264,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330576,"statfs":{"total":96636764160,"available":96594509824,"internally_reserved":0,"allocated":1806336,"data_stored":791305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41260,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1802240,"data_stored":791283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41316,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330524,"statfs":{"total":96636764160,"available":96594456576,"internally_reserved":0,"allocated":1859584,"data_stored":826337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41272,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330568,"statfs":{"total":96636764160,"available":96594501632,"internally_reserved":0,"allocated":1814528,"data_stored":791977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575005,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2433024,"data_stored":1405343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:25.074 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:25.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.173+0000 7f09ff565640 1 Processor -- start 2024-03-06T07:34:25.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.173+0000 7f09ff565640 1 -- start start 2024-03-06T07:34:25.177 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09ff565640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f09f8158600 0x7f09f8072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09ff565640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f815eb40 con 0x7f09f815e3a0 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09ff565640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f815ee20 con 0x7f09f8162030 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09ff565640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f8161ba0 con 0x7f09f8158600 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09fcb00640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f09f815e3a0 0x7f09f806d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58298/0 (socket says 172.21.15.122:58298) 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09fcb00640 1 -- 172.21.15.122:0/2734650902 learned_addr learned my addr 172.21.15.122:0/2734650902 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09fd301640 1 --2- 172.21.15.122:0/2734650902 >> v2:172.21.15.169:3300/0 conn(0x7f09f8158600 0x7f09f8072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173482213 0 0) 0x7f09f815eb40 con 0x7f09f815e3a0 2024-03-06T07:34:25.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4003520 con 0x7f09f815e3a0 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2813655836 0 0) 0x7f09e4003520 con 0x7f09f815e3a0 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 >> v2:172.21.15.169:3300/0 conn(0x7f09f8158600 msgr2=0x7f09f8072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 --2- 172.21.15.122:0/2734650902 >> v2:172.21.15.169:3300/0 conn(0x7f09f8158600 0x7f09f8072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 >> v1:172.21.15.169:6789/0 conn(0x7f09f8162030 legacy=0x7f09f806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f815f100 con 0x7f09f815e3a0 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f09e8001670 con 0x7f09f815e3a0 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 >> v1:172.21.15.122:6789/0 conn(0x7f09f815e3a0 legacy=0x7f09f806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09e40039b0 0x7f09e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e4004800 0x7f09e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e40051c0 0x7f09e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8161ba0 con 0x7f09e4004800 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e400c160 con 0x7f09e40051c0 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e400c360 con 0x7f09e40039b0 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.174+0000 7f09fcb00640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e4004800 0x7f09e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fd301640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09e40039b0 0x7f09e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fdb02640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e40051c0 0x7f09e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fcb00640 1 -- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09e40039b0 msgr2=0x7f09e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fcb00640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09e40039b0 0x7f09e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.180 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fcb00640 1 -- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e40051c0 msgr2=0x7f09e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fcb00640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e40051c0 0x7f09e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fdb02640 1 -- 172.21.15.122:0/2734650902 reap_dead start 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fcb00640 1 -- 172.21.15.122:0/2734650902 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4003520 con 0x7f09e4004800 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.175+0000 7f09fcb00640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e4004800 0x7f09e4004c80 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c240 tx=0x7f09e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3a5bec43eaf66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8007cf0 con 0x7f09e4004800 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e8030070 con 0x7f09e4004800 2024-03-06T07:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09effff640 1 -- 172.21.15.122:0/2734650902 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8005d70 con 0x7f09e4004800 2024-03-06T07:34:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 -- 172.21.15.122:0/2734650902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e4004800 msgr2=0x7f09e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e4004800 0x7f09e4004c80 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c240 tx=0x7f09e4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 -- 172.21.15.122:0/2734650902 shutdown_connections 2024-03-06T07:34:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 --2- 172.21.15.122:0/2734650902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e4004800 0x7f09e4004c80 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 -- 172.21.15.122:0/2734650902 >> 172.21.15.122:0/2734650902 conn(0x7f09f806c0c0 msgr2=0x7f09f806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 -- 172.21.15.122:0/2734650902 shutdown_connections 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.176+0000 7f09ff565640 1 -- 172.21.15.122:0/2734650902 wait complete. 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.177+0000 7f09ff565640 1 Processor -- start 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.177+0000 7f09ff565640 1 -- start start 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09ff565640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e4004800 0x7f09f81f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09ff565640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 0x7f09f81e9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09ff565640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f81590d0 0x7f09f815ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09ff565640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8162600 con 0x7f09f8144450 2024-03-06T07:34:25.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09ff565640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8162480 con 0x7f09e4004800 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09ff565640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8162780 con 0x7f09f81590d0 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 0x7f09f81e9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 0x7f09f81e9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44966/0 (socket says 172.21.15.122:44966) 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 -- 172.21.15.122:0/1352554506 learned_addr learned my addr 172.21.15.122:0/1352554506 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fd301640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e4004800 0x7f09f81f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fdb02640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f81590d0 0x7f09f815ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 -- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f81590d0 msgr2=0x7f09f815ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f81590d0 0x7f09f815ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 -- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e4004800 msgr2=0x7f09f81f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e4004800 0x7f09f81f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fcb00640 1 -- 172.21.15.122:0/1352554506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f81ea120 con 0x7f09f8144450 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.178+0000 7f09fd301640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e4004800 0x7f09f81f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.179+0000 7f09fcb00640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 0x7f09f81e9be0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c240 tx=0x7f09f815ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.179+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8046070 con 0x7f09f8144450 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.179+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e8000ed0 con 0x7f09f8144450 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.179+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8004e10 con 0x7f09f8144450 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.179+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f81448d0 con 0x7f09f8144450 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.179+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f81f6080 con 0x7f09f8144450 2024-03-06T07:34:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.180+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f804f330 con 0x7f09f8144450 2024-03-06T07:34:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.180+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09e8007070 con 0x7f09f8144450 2024-03-06T07:34:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.181+0000 7f09edffb640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09dc06b500 0x7f09dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.181+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f09e80ab1f0 con 0x7f09f8144450 2024-03-06T07:34:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.181+0000 7f09fd301640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09dc06b500 0x7f09dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.181+0000 7f09fd301640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09dc06b500 0x7f09dc06d9c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009c40 tx=0x7f09f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.185+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09e8079c20 con 0x7f09f8144450 2024-03-06T07:34:25.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.359+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09f80723f0 con 0x7f09f8144450 2024-03-06T07:34:25.361 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.359+0000 7f09edffb640 1 -- 172.21.15.122:0/1352554506 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f09e807dcc0 con 0x7f09f8144450 2024-03-06T07:34:25.362 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:25.362 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":146,"fsid":"12542aa0-5a71-4218-ae26-0a8da740ed3f","created":"2024-03-06T07:21:29.172741+0000","modified":"2024-03-06T07:34:22.156043+0000","last_up_change":"2024-03-06T07:21:40.252216+0000","last_in_change":"2024-03-06T07:21:31.414268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.370683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d8c299b-b0ee-4721-950f-7e9b86285b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6801","nonce":2293549780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6803","nonce":2293549780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6807","nonce":2293549780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2293549780},{"type":"v1","addr":"172.21.15.122:6805","nonce":2293549780}]},"public_addr":"172.21.15.122:6801/2293549780","cluster_addr":"172.21.15.122:6803/2293549780","heartbeat_back_addr":"172.21.15.122:6807/2293549780","heartbeat_front_addr":"172.21.15.122:6805/2293549780","state":["exists","up"]},{"osd":1,"uuid":"49c5e29b-a496-4025-a0fa-d8122f00ae45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6809","nonce":20360309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6811","nonce":20360309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6815","nonce":20360309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":20360309},{"type":"v1","addr":"172.21.15.122:6813","nonce":20360309}]},"public_addr":"172.21.15.122:6809/20360309","cluster_addr":"172.21.15.122:6811/20360309","heartbeat_back_addr":"172.21.15.122:6815/20360309","heartbeat_front_addr":"172.21.15.122:6813/20360309","state":["exists","up"]},{"osd":2,"uuid":"d3fb0805-c3a0-4ee3-b49e-a11f3bd7e2db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6817","nonce":1235809965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6819","nonce":1235809965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6823","nonce":1235809965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1235809965},{"type":"v1","addr":"172.21.15.122:6821","nonce":1235809965}]},"public_addr":"172.21.15.122:6817/1235809965","cluster_addr":"172.21.15.122:6819/1235809965","heartbeat_back_addr":"172.21.15.122:6823/1235809965","heartbeat_front_addr":"172.21.15.122:6821/1235809965","state":["exists","up"]},{"osd":3,"uuid":"17483d4d-a374-4a1a-b0cf-6765bcc28ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6825","nonce":47863215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6827","nonce":47863215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6831","nonce":47863215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":47863215},{"type":"v1","addr":"172.21.15.122:6829","nonce":47863215}]},"public_addr":"172.21.15.122:6825/47863215","cluster_addr":"172.21.15.122:6827/47863215","heartbeat_back_addr":"172.21.15.122:6831/47863215","heartbeat_front_addr":"172.21.15.122:6829/47863215","state":["exists","up"]},{"osd":4,"uuid":"a0dacb4c-2138-4ddc-a686-2f38ac64f9dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6806","nonce":1079551346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6810","nonce":1079551346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6815","nonce":1079551346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1079551346},{"type":"v1","addr":"172.21.15.169:6813","nonce":1079551346}]},"public_addr":"172.21.15.169:6806/1079551346","cluster_addr":"172.21.15.169:6810/1079551346","heartbeat_back_addr":"172.21.15.169:6815/1079551346","heartbeat_front_addr":"172.21.15.169:6813/1079551346","state":["exists","up"]},{"osd":5,"uuid":"eeb05427-e45f-42bd-9e82-bcf250302eef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6817","nonce":887889200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6819","nonce":887889200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6823","nonce":887889200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":887889200},{"type":"v1","addr":"172.21.15.169:6821","nonce":887889200}]},"public_addr":"172.21.15.169:6817/887889200","cluster_addr":"172.21.15.169:6819/887889200","heartbeat_back_addr":"172.21.15.169:6823/887889200","heartbeat_front_addr":"172.21.15.169:6821/887889200","state":["exists","up"]},{"osd":6,"uuid":"60c966b2-ab66-42b5-826b-8282fb747a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6825","nonce":1645803606}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6827","nonce":1645803606}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6831","nonce":1645803606}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1645803606},{"type":"v1","addr":"172.21.15.169:6829","nonce":1645803606}]},"public_addr":"172.21.15.169:6825/1645803606","cluster_addr":"172.21.15.169:6827/1645803606","heartbeat_back_addr":"172.21.15.169:6831/1645803606","heartbeat_front_addr":"172.21.15.169:6829/1645803606","state":["exists","up"]},{"osd":7,"uuid":"1fd2b82a-556e-4d70-bde1-39ac144dca46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6801","nonce":3079115781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6803","nonce":3079115781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6811","nonce":3079115781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":3079115781},{"type":"v1","addr":"172.21.15.169:6807","nonce":3079115781}]},"public_addr":"172.21.15.169:6801/3079115781","cluster_addr":"172.21.15.169:6803/3079115781","heartbeat_back_addr":"172.21.15.169:6811/3079115781","heartbeat_front_addr":"172.21.15.169:6807/3079115781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.531430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.572873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.535701+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.613619+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.548811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.658843+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.670812+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:36.647783+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6835/138214324":"2024-03-07T07:34:19.082389+0000","172.21.15.122:6834/138214324":"2024-03-07T07:34:19.082389+0000","172.21.15.122:6837/1416910179":"2024-03-07T07:34:19.082399+0000","172.21.15.122:6836/1416910179":"2024-03-07T07:34:19.082399+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:25.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09dc06b500 msgr2=0x7f09dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09dc06b500 0x7f09dc06d9c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009c40 tx=0x7f09f4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 msgr2=0x7f09f81e9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 0x7f09f81e9be0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c240 tx=0x7f09f815ed10 comp rx=0 tx=0).stop 2024-03-06T07:34:25.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 shutdown_connections 2024-03-06T07:34:25.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f09dc06b500 0x7f09dc06d9c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f09f81590d0 0x7f09f815ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09f8144450 0x7f09f81e9be0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 --2- 172.21.15.122:0/1352554506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f09e4004800 0x7f09f81f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.361+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 >> 172.21.15.122:0/1352554506 conn(0x7f09f806c0c0 msgr2=0x7f09f80b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.362+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 shutdown_connections 2024-03-06T07:34:25.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:25.362+0000 7f09ff565640 1 -- 172.21.15.122:0/1352554506 wait complete. 2024-03-06T07:34:26.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:26.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:26.375 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:34:26.375 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:26.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.496+0000 7f56b1b81640 1 Processor -- start 2024-03-06T07:34:26.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.496+0000 7f56b1b81640 1 -- start start 2024-03-06T07:34:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56b1b81640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f56ac0d1d90 0x7f56ac0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56b1b81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 0x7f56ac0b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56b1b81640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac0cfb30 con 0x7f56ac15dff0 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56b1b81640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ac0b3320 con 0x7f56ac0d1d90 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56b1b81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac0b3600 con 0x7f56ac0d2760 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 0x7f56ac0b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 0x7f56ac0b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:51646/0 (socket says 172.21.15.122:51646) 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 -- 172.21.15.122:0/3215745070 learned_addr learned my addr 172.21.15.122:0/3215745070 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56aaffd640 1 --2- 172.21.15.122:0/3215745070 >> v2:172.21.15.169:3300/0 conn(0x7f56ac0d1d90 0x7f56ac0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 -- 172.21.15.122:0/3215745070 >> v2:172.21.15.169:3300/0 conn(0x7f56ac0d1d90 msgr2=0x7f56ac0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 --2- 172.21.15.122:0/3215745070 >> v2:172.21.15.169:3300/0 conn(0x7f56ac0d1d90 0x7f56ac0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 -- 172.21.15.122:0/3215745070 >> v1:172.21.15.169:6789/0 conn(0x7f56ac15dff0 legacy=0x7f56ac0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.497+0000 7f56abfff640 1 -- 172.21.15.122:0/3215745070 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac0b38e0 con 0x7f56ac0d2760 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.498+0000 7f56abfff640 1 --2- 172.21.15.122:0/3215745070 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 0x7f56ac0b3e00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009e50 tx=0x7f56a402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6817b9463aac1340 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.498+0000 7f56aa7fc640 1 -- 172.21.15.122:0/3215745070 <== mon.2 v2:172.21.15.169:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a402cb90 con 0x7f56ac0d2760 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.498+0000 7f56aa7fc640 1 -- 172.21.15.122:0/3215745070 <== mon.2 v2:172.21.15.169:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a402cd30 con 0x7f56ac0d2760 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.498+0000 7f56aa7fc640 1 -- 172.21.15.122:0/3215745070 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4034610 con 0x7f56ac0d2760 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 -- 172.21.15.122:0/3215745070 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 msgr2=0x7f56ac0b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 --2- 172.21.15.122:0/3215745070 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 0x7f56ac0b3e00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009e50 tx=0x7f56a402bff0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 -- 172.21.15.122:0/3215745070 shutdown_connections 2024-03-06T07:34:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 --2- 172.21.15.122:0/3215745070 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d2760 0x7f56ac0b3e00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 --2- 172.21.15.122:0/3215745070 >> v2:172.21.15.169:3300/0 conn(0x7f56ac0d1d90 0x7f56ac0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 -- 172.21.15.122:0/3215745070 >> 172.21.15.122:0/3215745070 conn(0x7f56ac0d3d50 msgr2=0x7f56ac0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 -- 172.21.15.122:0/3215745070 shutdown_connections 2024-03-06T07:34:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.499+0000 7f56b1b81640 1 -- 172.21.15.122:0/3215745070 wait complete. 2024-03-06T07:34:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.500+0000 7f56b1b81640 1 Processor -- start 2024-03-06T07:34:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.500+0000 7f56b1b81640 1 -- start start 2024-03-06T07:34:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56b1b81640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d1d90 0x7f56ac154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56b1b81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac0d2760 0x7f56ac155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56b1b81640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56ac15dff0 0x7f56ac0b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56b1b81640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac15e550 con 0x7f56ac15dff0 2024-03-06T07:34:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56b1b81640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac15e3d0 con 0x7f56ac0d2760 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56b1b81640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac0b4340 con 0x7f56ac0d1d90 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56ac15dff0 0x7f56ac0b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56ac15dff0 0x7f56ac0b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44982/0 (socket says 172.21.15.122:44982) 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 -- 172.21.15.122:0/1389799476 learned_addr learned my addr 172.21.15.122:0/1389799476 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56ab7fe640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d1d90 0x7f56ac154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56aaffd640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac0d2760 0x7f56ac155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 -- 172.21.15.122:0/1389799476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d1d90 msgr2=0x7f56ac154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f56ac0d1d90 0x7f56ac154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 -- 172.21.15.122:0/1389799476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac0d2760 msgr2=0x7f56ac155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac0d2760 0x7f56ac155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.501+0000 7f56abfff640 1 -- 172.21.15.122:0/1389799476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac15e980 con 0x7f56ac15dff0 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.502+0000 7f56abfff640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56ac15dff0 0x7f56ac0b6970 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009f70 tx=0x7f56a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.502+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4043020 con 0x7f56ac15dff0 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.502+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a403e450 con 0x7f56ac15dff0 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.502+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac15ec60 con 0x7f56ac15dff0 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.502+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4009510 con 0x7f56ac15dff0 2024-03-06T07:34:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.502+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ac0b6eb0 con 0x7f56ac15dff0 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.503+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f56ac15e730 con 0x7f56ac15dff0 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.503+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56a4046020 con 0x7f56ac15dff0 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.503+0000 7f5693fff640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f56a40ac460 con 0x7f56ac15dff0 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f56ab7fe640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f5693fff640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f5698071430 0x7f5698073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56a404ee50 con 0x7f5698071430 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a40b3050 con 0x7f56ac15dff0 2024-03-06T07:34:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f56aaffd640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f5698071430 0x7f5698073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f56ab7fe640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569806b4d0 0x7f569806d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f56a00016e0 tx=0x7f56a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.504+0000 7f56aaffd640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f5698071430 0x7f5698073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.505+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== osd.0 v2:172.21.15.122:6800/2293549780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f56a404ee50 con 0x7f5698071430 2024-03-06T07:34:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.518+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 --> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f56ac1f3640 con 0x7f5698071430 2024-03-06T07:34:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.518+0000 7f5693fff640 1 -- 172.21.15.122:0/1389799476 <== osd.0 v2:172.21.15.122:6800/2293549780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f569c009d50 con 0x7f5698071430 2024-03-06T07:34:26.521 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:26.521 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:26.521 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.518+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f5698071430 msgr2=0x7f5698073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.518+0000 7f56b1b81640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6800/2293549780,v1:172.21.15.122:6801/2293549780] conn(0x7f5698071430 0x7f5698073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569806b4d0 msgr2=0x7f569806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f569806b4d0 0x7f569806d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f56a00016e0 tx=0x7f56a0009040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56ac15dff0 msgr2=0x7f56ac0b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 --2- 172.21.15.122:0/1389799476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56ac15dff0 0x7f56ac0b6970 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009f70 tx=0x7f56a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56abfff640 1 -- 172.21.15.122:0/1389799476 reap_dead start 2024-03-06T07:34:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 shutdown_connections 2024-03-06T07:34:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 >> 172.21.15.122:0/1389799476 conn(0x7f56ac0d3d50 msgr2=0x7f56ac0b57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 shutdown_connections 2024-03-06T07:34:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.519+0000 7f56b1b81640 1 -- 172.21.15.122:0/1389799476 wait complete. 2024-03-06T07:34:26.531 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:34:26.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.631+0000 7fa717e47640 1 Processor -- start 2024-03-06T07:34:26.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.632+0000 7fa717e47640 1 -- start start 2024-03-06T07:34:26.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.632+0000 7fa717e47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b4c00 0x7fa7100b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.632+0000 7fa717e47640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa7100b28a0 0x7fa7100b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa717e47640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71015d510 con 0x7fa71015e540 2024-03-06T07:34:26.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa717e47640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa71015e0b0 con 0x7fa7100b28a0 2024-03-06T07:34:26.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa717e47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71015d780 con 0x7fa7100b4c00 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa7100b28a0 0x7fa7100b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa7100b28a0 0x7fa7100b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44994/0 (socket says 172.21.15.122:44994) 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 -- 172.21.15.122:0/3904666431 learned_addr learned my addr 172.21.15.122:0/3904666431 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa715be3640 1 --2- 172.21.15.122:0/3904666431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b4c00 0x7fa7100b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 -- 172.21.15.122:0/3904666431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b4c00 msgr2=0x7fa7100b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 --2- 172.21.15.122:0/3904666431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b4c00 0x7fa7100b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 -- 172.21.15.122:0/3904666431 >> v1:172.21.15.169:6789/0 conn(0x7fa71015e540 legacy=0x7fa71015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.633+0000 7fa7153e2640 1 -- 172.21.15.122:0/3904666431 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71015da60 con 0x7fa7100b28a0 2024-03-06T07:34:26.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.634+0000 7fa7153e2640 1 --2- 172.21.15.122:0/3904666431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b28a0 0x7fa7100b2ce0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009870 tx=0x7fa70c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddb87b1b7e81f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.634+0000 7fa714be1640 1 -- 172.21.15.122:0/3904666431 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c02f840 con 0x7fa7100b28a0 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.634+0000 7fa714be1640 1 -- 172.21.15.122:0/3904666431 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c02f9e0 con 0x7fa7100b28a0 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.634+0000 7fa714be1640 1 -- 172.21.15.122:0/3904666431 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c02fdb0 con 0x7fa7100b28a0 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 -- 172.21.15.122:0/3904666431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b28a0 msgr2=0x7fa7100b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 --2- 172.21.15.122:0/3904666431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b28a0 0x7fa7100b2ce0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009870 tx=0x7fa70c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 -- 172.21.15.122:0/3904666431 shutdown_connections 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 --2- 172.21.15.122:0/3904666431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b28a0 0x7fa7100b2ce0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 --2- 172.21.15.122:0/3904666431 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b4c00 0x7fa7100b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 -- 172.21.15.122:0/3904666431 >> 172.21.15.122:0/3904666431 conn(0x7fa71009a9f0 msgr2=0x7fa71009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 -- 172.21.15.122:0/3904666431 shutdown_connections 2024-03-06T07:34:26.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.635+0000 7fa717e47640 1 -- 172.21.15.122:0/3904666431 wait complete. 2024-03-06T07:34:26.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.636+0000 7fa717e47640 1 Processor -- start 2024-03-06T07:34:26.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.636+0000 7fa717e47640 1 -- start start 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa717e47640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b28a0 0x7fa710144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa717e47640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 0x7fa7100afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa717e47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa71015e540 0x7fa7101533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa717e47640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b33a0 con 0x7fa7100b4c00 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa717e47640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b3220 con 0x7fa71015e540 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa717e47640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71015f090 con 0x7fa7100b28a0 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 0x7fa7100afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 0x7fa7100afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45006/0 (socket says 172.21.15.122:45006) 2024-03-06T07:34:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 -- 172.21.15.122:0/319276774 learned_addr learned my addr 172.21.15.122:0/319276774 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa715be3640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b28a0 0x7fa710144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7163e4640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa71015e540 0x7fa7101533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 -- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b28a0 msgr2=0x7fa710144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b28a0 0x7fa710144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 -- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa71015e540 msgr2=0x7fa7101533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa71015e540 0x7fa7101533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 -- 172.21.15.122:0/319276774 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100b05c0 con 0x7fa7100b4c00 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa715be3640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b28a0 0x7fa710144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7153e2640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 0x7fa7100afb30 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fa70c000c00 tx=0x7fa70c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.637+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c037bc0 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.638+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c046070 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.638+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7100b08a0 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.638+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c041400 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.638+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa710151720 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.639+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7100b5ad0 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.639+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa70c040070 con 0x7fa7100b4c00 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.639+0000 7fa7067fc640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f406b500 0x7fa6f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.640+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa70c0ac550 con 0x7fa7100b4c00 2024-03-06T07:34:26.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.640+0000 7fa715be3640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f406b500 0x7fa6f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.640+0000 7fa715be3640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f406b500 0x7fa6f406d9c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa700009b90 tx=0x7fa700008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.646 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.644+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7100b5ad0 con 0x7fa7100b4c00 2024-03-06T07:34:26.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.824+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa71004f330 con 0x7fa6f406b500 2024-03-06T07:34:26.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.824+0000 7fa7067fc640 1 -- 172.21.15.122:0/319276774 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa71004f330 con 0x7fa6f406b500 2024-03-06T07:34:26.826 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:34:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f406b500 msgr2=0x7fa6f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f406b500 0x7fa6f406d9c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa700009b90 tx=0x7fa700008040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 msgr2=0x7fa7100afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 0x7fa7100afb30 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fa70c000c00 tx=0x7fa70c004730 comp rx=0 tx=0).stop 2024-03-06T07:34:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 shutdown_connections 2024-03-06T07:34:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fa6f406b500 0x7fa6f406d9c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa71015e540 0x7fa7101533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7100b4c00 0x7fa7100afb30 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 --2- 172.21.15.122:0/319276774 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fa7100b28a0 0x7fa710144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 >> 172.21.15.122:0/319276774 conn(0x7fa71009a9f0 msgr2=0x7fa7100703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 shutdown_connections 2024-03-06T07:34:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.826+0000 7fa717e47640 1 -- 172.21.15.122:0/319276774 wait complete. 2024-03-06T07:34:26.839 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:34:26.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:26.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.941+0000 7f0730c40640 1 Processor -- start 2024-03-06T07:34:26.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.941+0000 7f0730c40640 1 -- start start 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.942+0000 7f0730c40640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c144690 0x7f072c0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.942+0000 7f0730c40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f072c0b3a40 0x7f072c0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.942+0000 7f0730c40640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c06d760 con 0x7f072c0cd960 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.942+0000 7f0730c40640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0d0830 con 0x7f072c0b3a40 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.942+0000 7f0730c40640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c144a70 con 0x7f072c144690 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f072c0b3a40 0x7f072c0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f072c0b3a40 0x7f072c0cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45022/0 (socket says 172.21.15.122:45022) 2024-03-06T07:34:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 -- 172.21.15.122:0/1556445746 learned_addr learned my addr 172.21.15.122:0/1556445746 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:26.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 -- 172.21.15.122:0/1556445746 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c144690 msgr2=0x7f072c0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 --2- 172.21.15.122:0/1556445746 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c144690 0x7f072c0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 -- 172.21.15.122:0/1556445746 >> v1:172.21.15.169:6789/0 conn(0x7f072c0cd960 legacy=0x7f072c0ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 -- 172.21.15.122:0/1556445746 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c15f660 con 0x7f072c0b3a40 2024-03-06T07:34:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.943+0000 7f0729d9b640 1 --2- 172.21.15.122:0/1556445746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c0b3a40 0x7f072c0cd420 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f071c009870 tx=0x7f071c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d34c0e06ff30201 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f072959a640 1 -- 172.21.15.122:0/1556445746 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c035020 con 0x7f072c0b3a40 2024-03-06T07:34:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f072959a640 1 -- 172.21.15.122:0/1556445746 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c031640 con 0x7f072c0b3a40 2024-03-06T07:34:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f072959a640 1 -- 172.21.15.122:0/1556445746 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c02d040 con 0x7f072c0b3a40 2024-03-06T07:34:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 -- 172.21.15.122:0/1556445746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c0b3a40 msgr2=0x7f072c0cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 --2- 172.21.15.122:0/1556445746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c0b3a40 0x7f072c0cd420 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f071c009870 tx=0x7f071c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 -- 172.21.15.122:0/1556445746 shutdown_connections 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 --2- 172.21.15.122:0/1556445746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c0b3a40 0x7f072c0cd420 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 --2- 172.21.15.122:0/1556445746 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c144690 0x7f072c0d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 -- 172.21.15.122:0/1556445746 >> 172.21.15.122:0/1556445746 conn(0x7f072c0d09b0 msgr2=0x7f072c0d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 -- 172.21.15.122:0/1556445746 shutdown_connections 2024-03-06T07:34:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.944+0000 7f0730c40640 1 -- 172.21.15.122:0/1556445746 wait complete. 2024-03-06T07:34:26.950 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 Processor -- start 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 -- start start 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c0b3a40 0x7f072c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f072c0cd960 0x7f072c1e74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c144690 0x7f072c0ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c15f280 con 0x7f072c144690 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c06dd00 con 0x7f072c0cd960 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.946+0000 7f0730c40640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c15f400 con 0x7f072c0b3a40 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c144690 0x7f072c0ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c144690 0x7f072c0ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45030/0 (socket says 172.21.15.122:45030) 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 -- 172.21.15.122:0/3976682134 learned_addr learned my addr 172.21.15.122:0/3976682134 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072a59c640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c0b3a40 0x7f072c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f0729d9b640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f072c0cd960 0x7f072c1e74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 -- 172.21.15.122:0/3976682134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c0b3a40 msgr2=0x7f072c15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f072c0b3a40 0x7f072c15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 -- 172.21.15.122:0/3976682134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f072c0cd960 msgr2=0x7f072c1e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f072c0cd960 0x7f072c1e74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 -- 172.21.15.122:0/3976682134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c15dd80 con 0x7f072c144690 2024-03-06T07:34:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f072ad9d640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c144690 0x7f072c0ced40 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f072400c7b0 tx=0x7f072400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724017070 con 0x7f072c144690 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724003da0 con 0x7f072c144690 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c0b08e0 con 0x7f072c144690 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072400ce60 con 0x7f072c144690 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.947+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c0b0ea0 con 0x7f072c144690 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.948+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f072c04f330 con 0x7f072c144690 2024-03-06T07:34:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f072401a070 con 0x7f072c144690 2024-03-06T07:34:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f071affd640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f070c06b4d0 0x7f070c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f07240894a0 con 0x7f072c144690 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f072a59c640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f070c06b4d0 0x7f070c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f071affd640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f070c071430 0x7f070c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072400ee50 con 0x7f070c071430 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0724090050 con 0x7f072c144690 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f0729d9b640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f070c071430 0x7f070c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.949+0000 7f072a59c640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f070c06b4d0 0x7f070c06d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f072c0b2200 tx=0x7f0720008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.950+0000 7f0729d9b640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f070c071430 0x7f070c073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.950+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== osd.1 v2:172.21.15.122:6808/20360309 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f072400ee50 con 0x7f070c071430 2024-03-06T07:34:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.963+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 --> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f072c0b1600 con 0x7f070c071430 2024-03-06T07:34:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.963+0000 7f071affd640 1 -- 172.21.15.122:0/3976682134 <== osd.1 v2:172.21.15.122:6808/20360309 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f072c0b1600 con 0x7f070c071430 2024-03-06T07:34:26.966 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:26.966 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:26.966 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f070c071430 msgr2=0x7f070c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6808/20360309,v1:172.21.15.122:6809/20360309] conn(0x7f070c071430 0x7f070c073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f070c06b4d0 msgr2=0x7f070c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f070c06b4d0 0x7f070c06d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f072c0b2200 tx=0x7f0720008040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c144690 msgr2=0x7f072c0ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 --2- 172.21.15.122:0/3976682134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f072c144690 0x7f072c0ced40 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f072400c7b0 tx=0x7f072400cc70 comp rx=0 tx=0).stop 2024-03-06T07:34:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f072ad9d640 1 -- 172.21.15.122:0/3976682134 reap_dead start 2024-03-06T07:34:26.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 shutdown_connections 2024-03-06T07:34:26.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 >> 172.21.15.122:0/3976682134 conn(0x7f072c0d09b0 msgr2=0x7f072c0721c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 shutdown_connections 2024-03-06T07:34:26.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:26.964+0000 7f0730c40640 1 -- 172.21.15.122:0/3976682134 wait complete. 2024-03-06T07:34:26.976 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:34:27.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.075+0000 7f1c8775c640 1 Processor -- start 2024-03-06T07:34:27.077 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.075+0000 7f1c8775c640 1 -- start start 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.076+0000 7f1c8775c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 0x7f1c800d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.083 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.076+0000 7f1c8775c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800d22c0 con 0x7f1c80162970 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.076+0000 7f1c8775c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800d25a0 con 0x7f1c800b4460 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.076+0000 7f1c8775c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80161b00 con 0x7f1c80162400 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.076+0000 7f1c84cf7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1c80162970 0x7f1c80161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58306/0 (socket says 172.21.15.122:58306) 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.076+0000 7f1c84cf7640 1 -- 172.21.15.122:0/2565028898 learned_addr learned my addr 172.21.15.122:0/2565028898 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c85cf9640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 0x7f1c800d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028762457 0 0) 0x7f1c800d22c0 con 0x7f1c80162970 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c003520 con 0x7f1c80162970 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1181172394 0 0) 0x7f1c6c003520 con 0x7f1c80162970 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 msgr2=0x7f1c800d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 0x7f1c800d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 >> v1:172.21.15.169:6789/0 conn(0x7f1c800b4460 legacy=0x7f1c800b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800d2880 con 0x7f1c80162970 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f1c7c001670 con 0x7f1c80162970 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 >> v1:172.21.15.122:6789/0 conn(0x7f1c80162970 legacy=0x7f1c80161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c6c0039e0 0x7f1c6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 0x7f1c6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c004d40 0x7f1c6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80161b00 con 0x7f1c6c0039e0 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c00b9f0 con 0x7f1c6c004d40 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c00bbf0 con 0x7f1c80162400 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c6c0039e0 0x7f1c6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c84cf7640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 0x7f1c6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c85cf9640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c004d40 0x7f1c6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c854f8640 1 -- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 msgr2=0x7f1c6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c80162400 0x7f1c6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c854f8640 1 -- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c004d40 msgr2=0x7f1c6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c004d40 0x7f1c6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.077+0000 7f1c854f8640 1 -- 172.21.15.122:0/2565028898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c003520 con 0x7f1c6c0039e0 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.078+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c6c0039e0 0x7f1c6c003dc0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f1c70001860 tx=0x7f1c6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8ec5bf2e799da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.078+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70038070 con 0x7f1c6c0039e0 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.078+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c70033660 con 0x7f1c6c0039e0 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.078+0000 7f1c77fff640 1 -- 172.21.15.122:0/2565028898 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70001430 con 0x7f1c6c0039e0 2024-03-06T07:34:27.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c8775c640 1 -- 172.21.15.122:0/2565028898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c6c0039e0 msgr2=0x7f1c6c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2565028898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c6c0039e0 0x7f1c6c003dc0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f1c70001860 tx=0x7f1c6c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:27.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c85cf9640 1 -- 172.21.15.122:0/2565028898 reap_dead start 2024-03-06T07:34:27.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c8775c640 1 -- 172.21.15.122:0/2565028898 shutdown_connections 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c8775c640 1 -- 172.21.15.122:0/2565028898 >> 172.21.15.122:0/2565028898 conn(0x7f1c800d3ce0 msgr2=0x7f1c800d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c8775c640 1 -- 172.21.15.122:0/2565028898 shutdown_connections 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.079+0000 7f1c8775c640 1 -- 172.21.15.122:0/2565028898 wait complete. 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.080+0000 7f1c8775c640 1 Processor -- start 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.080+0000 7f1c8775c640 1 -- start start 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c8775c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c0039e0 0x7f1c800b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c8775c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 0x7f1c801562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c8775c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c800b1f10 0x7f1c800b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c8775c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800ce950 con 0x7f1c800c7810 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c8775c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800cecc0 con 0x7f1c6c0039e0 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c8775c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800b4840 con 0x7f1c800b1f10 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 0x7f1c801562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 0x7f1c801562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45052/0 (socket says 172.21.15.122:45052) 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 -- 172.21.15.122:0/2961871919 learned_addr learned my addr 172.21.15.122:0/2961871919 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c85cf9640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c800b1f10 0x7f1c800b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c0039e0 0x7f1c800b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 -- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c800b1f10 msgr2=0x7f1c800b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c800b1f10 0x7f1c800b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 -- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c0039e0 msgr2=0x7f1c800b8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c0039e0 0x7f1c800b8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.081+0000 7f1c84cf7640 1 -- 172.21.15.122:0/2961871919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800cb350 con 0x7f1c800c7810 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.082+0000 7f1c84cf7640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 0x7f1c801562a0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0073a0 tx=0x7f1c7c007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.082+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c01a070 con 0x7f1c800c7810 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.082+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c00f940 con 0x7f1c800c7810 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.082+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c800c7c90 con 0x7f1c800c7810 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.082+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00fd10 con 0x7f1c800c7810 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.082+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c800c8200 con 0x7f1c800c7810 2024-03-06T07:34:27.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.083+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8004f330 con 0x7f1c800c7810 2024-03-06T07:34:27.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.084+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c7c01d070 con 0x7f1c800c7810 2024-03-06T07:34:27.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.084+0000 7f1c75ffb640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c6406b500 0x7f1c6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.084+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1c7c08a530 con 0x7f1c800c7810 2024-03-06T07:34:27.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.084+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c6406b500 0x7f1c6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.085+0000 7f1c854f8640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c6406b500 0x7f1c6406d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1c70035000 tx=0x7f1c70009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.094 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.089+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c7c058330 con 0x7f1c800c7810 2024-03-06T07:34:27.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.265+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c8006b180 con 0x7f1c6406b500 2024-03-06T07:34:27.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.265+0000 7f1c75ffb640 1 -- 172.21.15.122:0/2961871919 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1c8006b180 con 0x7f1c6406b500 2024-03-06T07:34:27.267 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:34:27.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c6406b500 msgr2=0x7f1c6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c6406b500 0x7f1c6406d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1c70035000 tx=0x7f1c70009030 comp rx=0 tx=0).stop 2024-03-06T07:34:27.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 msgr2=0x7f1c801562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 0x7f1c801562a0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0073a0 tx=0x7f1c7c007860 comp rx=0 tx=0).stop 2024-03-06T07:34:27.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 shutdown_connections 2024-03-06T07:34:27.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1c6406b500 0x7f1c6406d9c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1c800b1f10 0x7f1c800b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c800c7810 0x7f1c801562a0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 --2- 172.21.15.122:0/2961871919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c6c0039e0 0x7f1c800b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 >> 172.21.15.122:0/2961871919 conn(0x7f1c800d3ce0 msgr2=0x7f1c801610b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 shutdown_connections 2024-03-06T07:34:27.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.267+0000 7f1c8775c640 1 -- 172.21.15.122:0/2961871919 wait complete. 2024-03-06T07:34:27.280 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:34:27.280 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:27.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.382+0000 7ff27ca39640 1 Processor -- start 2024-03-06T07:34:27.384 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.382+0000 7ff27ca39640 1 -- start start 2024-03-06T07:34:27.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27ca39640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff2780cdb30 0x7ff2780d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27ca39640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff2780b3a40 0x7ff2780ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27ca39640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27806d760 con 0x7ff278144690 2024-03-06T07:34:27.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27ca39640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780d0830 con 0x7ff2780cdb30 2024-03-06T07:34:27.386 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27ca39640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff278144a70 con 0x7ff2780b3a40 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff275d9b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff2780cdb30 0x7ff2780d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff278144690 0x7ff2780cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58308/0 (socket says 172.21.15.122:58308) 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27659c640 1 -- 172.21.15.122:0/2509797746 learned_addr learned my addr 172.21.15.122:0/2509797746 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff276d9d640 1 --2- 172.21.15.122:0/2509797746 >> v2:172.21.15.169:3300/0 conn(0x7ff2780b3a40 0x7ff2780ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff275d9b640 1 -- 172.21.15.122:0/2509797746 >> v2:172.21.15.169:3300/0 conn(0x7ff2780b3a40 msgr2=0x7ff2780ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff275d9b640 1 --2- 172.21.15.122:0/2509797746 >> v2:172.21.15.169:3300/0 conn(0x7ff2780b3a40 0x7ff2780ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff275d9b640 1 -- 172.21.15.122:0/2509797746 >> v1:172.21.15.122:6789/0 conn(0x7ff278144690 legacy=0x7ff2780cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.387 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff275d9b640 1 -- 172.21.15.122:0/2509797746 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27815f660 con 0x7ff2780cdb30 2024-03-06T07:34:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff27559a640 1 -- 172.21.15.122:0/2509797746 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018526174 0 0) 0x7ff27806d760 con 0x7ff278144690 2024-03-06T07:34:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.383+0000 7ff276d9d640 1 --2- 172.21.15.122:0/2509797746 >> v2:172.21.15.169:3300/0 conn(0x7ff2780b3a40 0x7ff2780ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.384+0000 7ff275d9b640 1 --2- 172.21.15.122:0/2509797746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2780cdb30 0x7ff2780d02f0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff27806f9f0 tx=0x7ff26c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928a2bf2106febcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.384+0000 7ff27559a640 1 -- 172.21.15.122:0/2509797746 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c035020 con 0x7ff2780cdb30 2024-03-06T07:34:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.384+0000 7ff27559a640 1 -- 172.21.15.122:0/2509797746 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c031830 con 0x7ff2780cdb30 2024-03-06T07:34:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.384+0000 7ff27559a640 1 -- 172.21.15.122:0/2509797746 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c031c00 con 0x7ff2780cdb30 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.384+0000 7ff27ca39640 1 -- 172.21.15.122:0/2509797746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2780cdb30 msgr2=0x7ff2780d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.384+0000 7ff27ca39640 1 --2- 172.21.15.122:0/2509797746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2780cdb30 0x7ff2780d02f0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff27806f9f0 tx=0x7ff26c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.385+0000 7ff27ca39640 1 -- 172.21.15.122:0/2509797746 shutdown_connections 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.385+0000 7ff27ca39640 1 --2- 172.21.15.122:0/2509797746 >> v2:172.21.15.169:3300/0 conn(0x7ff2780b3a40 0x7ff2780ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.385+0000 7ff27ca39640 1 --2- 172.21.15.122:0/2509797746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2780cdb30 0x7ff2780d02f0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.385+0000 7ff27ca39640 1 -- 172.21.15.122:0/2509797746 >> 172.21.15.122:0/2509797746 conn(0x7ff2780d09b0 msgr2=0x7ff2780d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.393 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.385+0000 7ff27ca39640 1 -- 172.21.15.122:0/2509797746 shutdown_connections 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.385+0000 7ff27ca39640 1 -- 172.21.15.122:0/2509797746 wait complete. 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.386+0000 7ff27ca39640 1 Processor -- start 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.386+0000 7ff27ca39640 1 -- start start 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27ca39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2780b3a40 0x7ff27813bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27ca39640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2780cdb30 0x7ff2781565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27ca39640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff278144690 0x7ff278139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27ca39640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27815f280 con 0x7ff278144690 2024-03-06T07:34:27.394 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27ca39640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806dd00 con 0x7ff2780b3a40 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27ca39640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27815f400 con 0x7ff2780cdb30 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff278144690 0x7ff278139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff278144690 0x7ff278139190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45068/0 (socket says 172.21.15.122:45068) 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 -- 172.21.15.122:0/427434343 learned_addr learned my addr 172.21.15.122:0/427434343 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff275d9b640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2780cdb30 0x7ff2781565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 -- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2780cdb30 msgr2=0x7ff2781565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2780cdb30 0x7ff2781565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.395 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27659c640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2780b3a40 0x7ff27813bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 -- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2780b3a40 msgr2=0x7ff27813bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2780b3a40 0x7ff27813bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 -- 172.21.15.122:0/427434343 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27813ccb0 con 0x7ff278144690 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff275d9b640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff2780cdb30 0x7ff2781565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff27659c640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2780b3a40 0x7ff27813bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff276d9d640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff278144690 0x7ff278139190 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7ff26800ce40 tx=0x7ff26800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26801a070 con 0x7ff278144690 2024-03-06T07:34:27.396 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.387+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26800d840 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.388+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27813cf90 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.388+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26800dc30 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.388+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27813d550 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.388+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff27813d930 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.389+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff26801d070 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.389+0000 7ff266ffd640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff25806b4d0 0x7ff25806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff26808ae40 con 0x7ff278144690 2024-03-06T07:34:27.397 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff27659c640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff25806b4d0 0x7ff25806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff266ffd640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ff258071430 0x7ff258073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff268011e50 con 0x7ff258071430 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff26808b2d0 con 0x7ff278144690 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff275d9b640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ff258071430 0x7ff258073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff27659c640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff25806b4d0 0x7ff25806d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff270000f60 tx=0x7ff270009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.390+0000 7ff275d9b640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ff258071430 0x7ff258073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.398 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.391+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== osd.2 v2:172.21.15.122:6816/1235809965 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff268011e50 con 0x7ff258071430 2024-03-06T07:34:27.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 --> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff27804f330 con 0x7ff258071430 2024-03-06T07:34:27.406 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff266ffd640 1 -- 172.21.15.122:0/427434343 <== osd.2 v2:172.21.15.122:6816/1235809965 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff27804f330 con 0x7ff258071430 2024-03-06T07:34:27.406 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:27.407 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:27.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ff258071430 msgr2=0x7ff258073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6816/1235809965,v1:172.21.15.122:6817/1235809965] conn(0x7ff258071430 0x7ff258073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff25806b4d0 msgr2=0x7ff25806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff25806b4d0 0x7ff25806d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff270000f60 tx=0x7ff270009040 comp rx=0 tx=0).stop 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff278144690 msgr2=0x7ff278139190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 --2- 172.21.15.122:0/427434343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff278144690 0x7ff278139190 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7ff26800ce40 tx=0x7ff26800a710 comp rx=0 tx=0).stop 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff276d9d640 1 -- 172.21.15.122:0/427434343 reap_dead start 2024-03-06T07:34:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 shutdown_connections 2024-03-06T07:34:27.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 >> 172.21.15.122:0/427434343 conn(0x7ff2780d09b0 msgr2=0x7ff2780d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 shutdown_connections 2024-03-06T07:34:27.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.404+0000 7ff27ca39640 1 -- 172.21.15.122:0/427434343 wait complete. 2024-03-06T07:34:27.417 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:34:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.516+0000 7fc257948640 1 Processor -- start 2024-03-06T07:34:27.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.516+0000 7fc257948640 1 -- start start 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc257948640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2500b4460 0x7fc2500d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc257948640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250161cf0 0x7fc250162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc257948640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2500d22c0 con 0x7fc250162970 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc257948640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc250161b00 con 0x7fc2500b4460 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc257948640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500cecc0 con 0x7fc250161cf0 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc2556e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc2500b4460 0x7fc2500d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.520 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc254ee3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc250162970 0x7fc2500b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58318/0 (socket says 172.21.15.122:58318) 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc254ee3640 1 -- 172.21.15.122:0/3130457818 learned_addr learned my addr 172.21.15.122:0/3130457818 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc255ee5640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250161cf0 0x7fc250162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc2556e4640 1 -- 172.21.15.122:0/3130457818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250161cf0 msgr2=0x7fc250162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc2556e4640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250161cf0 0x7fc250162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.517+0000 7fc2556e4640 1 -- 172.21.15.122:0/3130457818 >> v1:172.21.15.122:6789/0 conn(0x7fc250162970 legacy=0x7fc2500b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc243fff640 1 -- 172.21.15.122:0/3130457818 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255349315 0 0) 0x7fc2500d22c0 con 0x7fc250162970 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc2556e4640 1 -- 172.21.15.122:0/3130457818 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500d25a0 con 0x7fc2500b4460 2024-03-06T07:34:27.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc255ee5640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250161cf0 0x7fc250162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc2556e4640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2500d4800 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fc2440097a0 tx=0x7fc24402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebd34043451c52c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc243fff640 1 -- 172.21.15.122:0/3130457818 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244035020 con 0x7fc2500b4460 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc243fff640 1 -- 172.21.15.122:0/3130457818 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc244031830 con 0x7fc2500b4460 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.518+0000 7fc243fff640 1 -- 172.21.15.122:0/3130457818 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244031c00 con 0x7fc2500b4460 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 -- 172.21.15.122:0/3130457818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 msgr2=0x7fc2500d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2500d4800 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fc2440097a0 tx=0x7fc24402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 -- 172.21.15.122:0/3130457818 shutdown_connections 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250161cf0 0x7fc250162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 --2- 172.21.15.122:0/3130457818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2500d4800 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 -- 172.21.15.122:0/3130457818 >> 172.21.15.122:0/3130457818 conn(0x7fc2500d3ce0 msgr2=0x7fc2500d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 -- 172.21.15.122:0/3130457818 shutdown_connections 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.519+0000 7fc257948640 1 -- 172.21.15.122:0/3130457818 wait complete. 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.520+0000 7fc257948640 1 Processor -- start 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.520+0000 7fc257948640 1 -- start start 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc257948640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2501558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc257948640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250161cf0 0x7fc2500b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc257948640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250162970 0x7fc25015ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc257948640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500ce950 con 0x7fc2500b4460 2024-03-06T07:34:27.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc257948640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500d2b60 con 0x7fc250161cf0 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc257948640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500b4840 con 0x7fc250162970 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2501558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2501558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45082/0 (socket says 172.21.15.122:45082) 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 -- 172.21.15.122:0/1356774730 learned_addr learned my addr 172.21.15.122:0/1356774730 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc255ee5640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250162970 0x7fc25015ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc254ee3640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250161cf0 0x7fc2500b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 -- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250162970 msgr2=0x7fc25015ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250162970 0x7fc25015ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 -- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250161cf0 msgr2=0x7fc2500b88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250161cf0 0x7fc2500b88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.521+0000 7fc2556e4640 1 -- 172.21.15.122:0/1356774730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc250155df0 con 0x7fc2500b4460 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.522+0000 7fc2556e4640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2501558b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fc244000c00 tx=0x7fc2440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.522+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244041070 con 0x7fc2500b4460 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.522+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc244004230 con 0x7fc2500b4460 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.522+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2500b93d0 con 0x7fc2500b4460 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.522+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244035040 con 0x7fc2500b4460 2024-03-06T07:34:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.522+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25015a620 con 0x7fc2500b4460 2024-03-06T07:34:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.523+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25015a140 con 0x7fc2500b4460 2024-03-06T07:34:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.523+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc244038020 con 0x7fc2500b4460 2024-03-06T07:34:27.529 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.524+0000 7fc241ffb640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc23406b4d0 0x7fc23406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.524+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc2440ab8b0 con 0x7fc2500b4460 2024-03-06T07:34:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.524+0000 7fc254ee3640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc23406b4d0 0x7fc23406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.524+0000 7fc254ee3640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc23406b4d0 0x7fc23406d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc250160ee0 tx=0x7fc24c007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.532+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc24407a410 con 0x7fc2500b4460 2024-03-06T07:34:27.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.722+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc25004f330 con 0x7fc23406b4d0 2024-03-06T07:34:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.723+0000 7fc241ffb640 1 -- 172.21.15.122:0/1356774730 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc25004f330 con 0x7fc23406b4d0 2024-03-06T07:34:27.725 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.724+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc23406b4d0 msgr2=0x7fc23406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.724+0000 7fc257948640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc23406b4d0 0x7fc23406d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc250160ee0 tx=0x7fc24c007de0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.724+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 msgr2=0x7fc2501558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.724+0000 7fc257948640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2501558b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fc244000c00 tx=0x7fc2440046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 shutdown_connections 2024-03-06T07:34:27.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc23406b4d0 0x7fc23406d990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc250162970 0x7fc25015ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc250161cf0 0x7fc2500b88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 --2- 172.21.15.122:0/1356774730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2500b4460 0x7fc2501558b0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 >> 172.21.15.122:0/1356774730 conn(0x7fc2500d3ce0 msgr2=0x7fc2501602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 shutdown_connections 2024-03-06T07:34:27.728 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.725+0000 7fc257948640 1 -- 172.21.15.122:0/1356774730 wait complete. 2024-03-06T07:34:27.737 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:34:27.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:27.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.840+0000 7f3e5824e640 1 Processor -- start 2024-03-06T07:34:27.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.840+0000 7f3e5824e640 1 -- start start 2024-03-06T07:34:27.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e5824e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3e500708b0 0x7f3e50070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e5824e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 0x7f3e500ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e5824e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5015afe0 con 0x7f3e5015ac00 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e5824e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e50071250 con 0x7f3e500708b0 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e5824e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cdc10 con 0x7f3e500ce0a0 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e567eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 0x7f3e500ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e557e9640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f3e500708b0 0x7f3e50070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e567eb640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 0x7f3e500ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3301/0 says I am v2:172.21.15.122:51736/0 (socket says 172.21.15.122:51736) 2024-03-06T07:34:27.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e567eb640 1 -- 172.21.15.122:0/930705955 learned_addr learned my addr 172.21.15.122:0/930705955 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e557e9640 1 -- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 msgr2=0x7f3e500ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e557e9640 1 --2- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 0x7f3e500ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e557e9640 1 -- 172.21.15.122:0/930705955 >> v1:172.21.15.169:6789/0 conn(0x7f3e5015ac00 legacy=0x7f3e5006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e557e9640 1 -- 172.21.15.122:0/930705955 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500cd000 con 0x7f3e500708b0 2024-03-06T07:34:27.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.841+0000 7f3e567eb640 1 --2- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 0x7f3e500ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:27.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.842+0000 7f3e557e9640 1 --2- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500708b0 0x7f3e50070d10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3e400097a0 tx=0x7f3e4002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25423f472b88490a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.842+0000 7f3e54fe8640 1 -- 172.21.15.122:0/930705955 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400315a0 con 0x7f3e500708b0 2024-03-06T07:34:27.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.842+0000 7f3e54fe8640 1 -- 172.21.15.122:0/930705955 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e40037070 con 0x7f3e500708b0 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.842+0000 7f3e54fe8640 1 -- 172.21.15.122:0/930705955 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40031960 con 0x7f3e500708b0 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 -- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500708b0 msgr2=0x7f3e50070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 --2- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500708b0 0x7f3e50070d10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3e400097a0 tx=0x7f3e4002b840 comp rx=0 tx=0).stop 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 -- 172.21.15.122:0/930705955 shutdown_connections 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 --2- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e500ce0a0 0x7f3e500ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 --2- 172.21.15.122:0/930705955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500708b0 0x7f3e50070d10 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 -- 172.21.15.122:0/930705955 >> 172.21.15.122:0/930705955 conn(0x7f3e500b1650 msgr2=0x7f3e500b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 -- 172.21.15.122:0/930705955 shutdown_connections 2024-03-06T07:34:27.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.843+0000 7f3e5824e640 1 -- 172.21.15.122:0/930705955 wait complete. 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.844+0000 7f3e5824e640 1 Processor -- start 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.844+0000 7f3e5824e640 1 -- start start 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e5824e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e500708b0 0x7f3e501402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e5824e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500ce0a0 0x7f3e50165c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e5824e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e5015ac00 0x7f3e5014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e5824e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cd8a0 con 0x7f3e500708b0 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e5824e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cd5c0 con 0x7f3e500ce0a0 2024-03-06T07:34:27.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e5824e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cf5d0 con 0x7f3e5015ac00 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e500708b0 0x7f3e501402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e500708b0 0x7f3e501402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45084/0 (socket says 172.21.15.122:45084) 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 -- 172.21.15.122:0/1422775054 learned_addr learned my addr 172.21.15.122:0/1422775054 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e567eb640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e5015ac00 0x7f3e5014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e557e9640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500ce0a0 0x7f3e50165c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 -- 172.21.15.122:0/1422775054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e5015ac00 msgr2=0x7f3e5014d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f3e5015ac00 0x7f3e5014d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 -- 172.21.15.122:0/1422775054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500ce0a0 msgr2=0x7f3e50165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e500ce0a0 0x7f3e50165c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.845+0000 7f3e55fea640 1 -- 172.21.15.122:0/1422775054 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e501666d0 con 0x7f3e500708b0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e55fea640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e500708b0 0x7f3e501402f0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00c400 tx=0x7f3e4c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0052d0 con 0x7f3e500708b0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c005490 con 0x7f3e500708b0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e501669b0 con 0x7f3e500708b0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c007260 con 0x7f3e500708b0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e50166f70 con 0x7f3e500708b0 2024-03-06T07:34:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.846+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e50167350 con 0x7f3e500708b0 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.847+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e4c007400 con 0x7f3e500708b0 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.847+0000 7f3e467fc640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e3406b4d0 0x7f3e3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3e4c089370 con 0x7f3e500708b0 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e557e9640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e3406b4d0 0x7f3e3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e467fc640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f3e34071430 0x7f3e34073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e4c009e50 con 0x7f3e34071430 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e4c090050 con 0x7f3e500708b0 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e567eb640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f3e34071430 0x7f3e34073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e557e9640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e3406b4d0 0x7f3e3406d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3e40036000 tx=0x7f3e4003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.848+0000 7f3e567eb640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f3e34071430 0x7f3e34073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.849+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== osd.3 v2:172.21.15.122:6824/47863215 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3e4c009e50 con 0x7f3e34071430 2024-03-06T07:34:27.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.861+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 --> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3e5004f330 con 0x7f3e34071430 2024-03-06T07:34:27.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e467fc640 1 -- 172.21.15.122:0/1422775054 <== osd.3 v2:172.21.15.122:6824/47863215 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3e5004f330 con 0x7f3e34071430 2024-03-06T07:34:27.864 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:27.864 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:27.865 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:27.865 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f3e34071430 msgr2=0x7f3e34073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6824/47863215,v1:172.21.15.122:6825/47863215] conn(0x7f3e34071430 0x7f3e34073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e3406b4d0 msgr2=0x7f3e3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f3e3406b4d0 0x7f3e3406d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3e40036000 tx=0x7f3e4003d040 comp rx=0 tx=0).stop 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e500708b0 msgr2=0x7f3e501402f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 --2- 172.21.15.122:0/1422775054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e500708b0 0x7f3e501402f0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00c400 tx=0x7f3e4c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e567eb640 1 -- 172.21.15.122:0/1422775054 reap_dead start 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 shutdown_connections 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 >> 172.21.15.122:0/1422775054 conn(0x7f3e500b1650 msgr2=0x7f3e5014e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.866 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 shutdown_connections 2024-03-06T07:34:27.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.862+0000 7f3e5824e640 1 -- 172.21.15.122:0/1422775054 wait complete. 2024-03-06T07:34:27.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:34:27.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.973+0000 7f1d9411e640 1 Processor -- start 2024-03-06T07:34:27.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.973+0000 7f1d9411e640 1 -- start start 2024-03-06T07:34:27.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d9411e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f1d8c0b0240 0x7f1d8c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d9411e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c161cf0 con 0x7f1d8c1586c0 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d9411e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c161fd0 con 0x7f1d8c06b4f0 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d9411e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d8c158aa0 con 0x7f1d8c0b0240 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d91eba640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1d8c1586c0 0x7f1d8c15efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58324/0 (socket says 172.21.15.122:58324) 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d91eba640 1 -- 172.21.15.122:0/1169049257 learned_addr learned my addr 172.21.15.122:0/1169049257 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d916b9640 1 --2- 172.21.15.122:0/1169049257 >> v2:172.21.15.169:3300/0 conn(0x7f1d8c0b0240 0x7f1d8c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d90eb8640 1 -- 172.21.15.122:0/1169049257 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281545000 0 0) 0x7f1d8c161cf0 con 0x7f1d8c1586c0 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d90eb8640 1 -- 172.21.15.122:0/1169049257 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d78003520 con 0x7f1d8c1586c0 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d916b9640 1 -- 172.21.15.122:0/1169049257 >> v1:172.21.15.169:6789/0 conn(0x7f1d8c06b4f0 legacy=0x7f1d8c0cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d90eb8640 1 -- 172.21.15.122:0/1169049257 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1644247273 0 0) 0x7f1d78003520 con 0x7f1d8c1586c0 2024-03-06T07:34:27.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d916b9640 1 -- 172.21.15.122:0/1169049257 >> v1:172.21.15.122:6789/0 conn(0x7f1d8c1586c0 legacy=0x7f1d8c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.974+0000 7f1d916b9640 1 -- 172.21.15.122:0/1169049257 --> v2:172.21.15.169:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c1622b0 con 0x7f1d8c0b0240 2024-03-06T07:34:27.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.975+0000 7f1d916b9640 1 --2- 172.21.15.122:0/1169049257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c06afb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097a0 tx=0x7f1d7c02b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9444e0ef1b05b5b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.975+0000 7f1d90eb8640 1 -- 172.21.15.122:0/1169049257 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c037070 con 0x7f1d8c0b0240 2024-03-06T07:34:27.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.975+0000 7f1d90eb8640 1 -- 172.21.15.122:0/1169049257 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c030510 con 0x7f1d8c0b0240 2024-03-06T07:34:27.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.975+0000 7f1d90eb8640 1 -- 172.21.15.122:0/1169049257 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88001670 con 0x7f1d8c0b0240 2024-03-06T07:34:27.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 -- 172.21.15.122:0/1169049257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c0b0240 msgr2=0x7f1d8c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 --2- 172.21.15.122:0/1169049257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c06afb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097a0 tx=0x7f1d7c02b680 comp rx=0 tx=0).stop 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 -- 172.21.15.122:0/1169049257 shutdown_connections 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 --2- 172.21.15.122:0/1169049257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c06afb0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 -- 172.21.15.122:0/1169049257 >> 172.21.15.122:0/1169049257 conn(0x7f1d8c0d09d0 msgr2=0x7f1d8c0d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 -- 172.21.15.122:0/1169049257 shutdown_connections 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.976+0000 7f1d9411e640 1 -- 172.21.15.122:0/1169049257 wait complete. 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.977+0000 7f1d9411e640 1 Processor -- start 2024-03-06T07:34:27.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.977+0000 7f1d9411e640 1 -- start start 2024-03-06T07:34:27.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c06b4f0 0x7f1d8c0b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1d8c1586c0 0x7f1d8c0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c162870 con 0x7f1d8c0b0240 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c06b960 con 0x7f1d8c06b4f0 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c0719e0 con 0x7f1d8c1586c0 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c0b5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45100/0 (socket says 172.21.15.122:45100) 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 -- 172.21.15.122:0/3692217158 learned_addr learned my addr 172.21.15.122:0/3692217158 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:27.983 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d91eba640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c06b4f0 0x7f1d8c0b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d926bb640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1d8c1586c0 0x7f1d8c0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 -- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1d8c1586c0 msgr2=0x7f1d8c0ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1d8c1586c0 0x7f1d8c0ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 -- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c06b4f0 msgr2=0x7f1d8c0b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c06b4f0 0x7f1d8c0b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 -- 172.21.15.122:0/3692217158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c0b44e0 con 0x7f1d8c0b0240 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d916b9640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c0b5580 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c035040 tx=0x7f1d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c042070 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c004220 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c0b47c0 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.978+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c037040 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.979+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c0b4d80 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.979+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c04f330 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.980+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d7c03a040 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.980+0000 7f1d827fc640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d7006b4d0 0x7f1d7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.980+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1d7c0ab860 con 0x7f1d8c0b0240 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.980+0000 7f1d91eba640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d7006b4d0 0x7f1d7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.985 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.981+0000 7f1d91eba640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d7006b4d0 0x7f1d7006d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f1d880017e0 tx=0x7f1d88001230 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:27.985+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d7c07a2b0 con 0x7f1d8c0b0240 2024-03-06T07:34:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.155+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d8c0d17a0 con 0x7f1d7006b4d0 2024-03-06T07:34:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.155+0000 7f1d827fc640 1 -- 172.21.15.122:0/3692217158 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1d8c0d17a0 con 0x7f1d7006b4d0 2024-03-06T07:34:28.157 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:34:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d7006b4d0 msgr2=0x7f1d7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d7006b4d0 0x7f1d7006d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f1d880017e0 tx=0x7f1d88001230 comp rx=0 tx=0).stop 2024-03-06T07:34:28.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 msgr2=0x7f1d8c0b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c0b5580 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c035040 tx=0x7f1d7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 shutdown_connections 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f1d7006b4d0 0x7f1d7006d990 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f1d8c1586c0 0x7f1d8c0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d8c0b0240 0x7f1d8c0b5580 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 --2- 172.21.15.122:0/3692217158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d8c06b4f0 0x7f1d8c0b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 >> 172.21.15.122:0/3692217158 conn(0x7f1d8c0d09d0 msgr2=0x7f1d8c0ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 shutdown_connections 2024-03-06T07:34:28.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.157+0000 7f1d9411e640 1 -- 172.21.15.122:0/3692217158 wait complete. 2024-03-06T07:34:28.169 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:34:28.170 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:28.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.272+0000 7f02f6681640 1 Processor -- start 2024-03-06T07:34:28.274 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.272+0000 7f02f6681640 1 -- start start 2024-03-06T07:34:28.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02f6681640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f02f00708b0 0x7f02f0070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02f6681640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f02f00ce0a0 0x7f02f00ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02f6681640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f015afe0 con 0x7f02f015ac00 2024-03-06T07:34:28.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02f6681640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f0071250 con 0x7f02f00708b0 2024-03-06T07:34:28.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02f6681640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f00cdc10 con 0x7f02f00ce0a0 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f02f00708b0 0x7f02f0070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f02f00708b0 0x7f02f0070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45116/0 (socket says 172.21.15.122:45116) 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 -- 172.21.15.122:0/3297684645 learned_addr learned my addr 172.21.15.122:0/3297684645 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02f4c1e640 1 --2- 172.21.15.122:0/3297684645 >> v2:172.21.15.169:3300/0 conn(0x7f02f00ce0a0 0x7f02f00ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 -- 172.21.15.122:0/3297684645 >> v2:172.21.15.169:3300/0 conn(0x7f02f00ce0a0 msgr2=0x7f02f00ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 --2- 172.21.15.122:0/3297684645 >> v2:172.21.15.169:3300/0 conn(0x7f02f00ce0a0 0x7f02f00ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 -- 172.21.15.122:0/3297684645 >> v1:172.21.15.169:6789/0 conn(0x7f02f015ac00 legacy=0x7f02f006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.273+0000 7f02ef7fe640 1 -- 172.21.15.122:0/3297684645 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f00cd000 con 0x7f02f00708b0 2024-03-06T07:34:28.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.274+0000 7f02ef7fe640 1 --2- 172.21.15.122:0/3297684645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0070d10 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f02f00cc0a0 tx=0x7f02e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13452d9878548d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.274+0000 7f02eeffd640 1 -- 172.21.15.122:0/3297684645 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e4035020 con 0x7f02f00708b0 2024-03-06T07:34:28.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.274+0000 7f02eeffd640 1 -- 172.21.15.122:0/3297684645 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e4031870 con 0x7f02f00708b0 2024-03-06T07:34:28.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.274+0000 7f02eeffd640 1 -- 172.21.15.122:0/3297684645 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e4031c40 con 0x7f02f00708b0 2024-03-06T07:34:28.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.274+0000 7f02f6681640 1 -- 172.21.15.122:0/3297684645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 msgr2=0x7f02f0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.274+0000 7f02f6681640 1 --2- 172.21.15.122:0/3297684645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0070d10 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f02f00cc0a0 tx=0x7f02e402b910 comp rx=0 tx=0).stop 2024-03-06T07:34:28.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.275+0000 7f02f6681640 1 -- 172.21.15.122:0/3297684645 shutdown_connections 2024-03-06T07:34:28.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.275+0000 7f02f6681640 1 --2- 172.21.15.122:0/3297684645 >> v2:172.21.15.169:3300/0 conn(0x7f02f00ce0a0 0x7f02f00ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.275+0000 7f02f6681640 1 --2- 172.21.15.122:0/3297684645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0070d10 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.281 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.275+0000 7f02f6681640 1 -- 172.21.15.122:0/3297684645 >> 172.21.15.122:0/3297684645 conn(0x7f02f00b1650 msgr2=0x7f02f00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.275+0000 7f02f6681640 1 -- 172.21.15.122:0/3297684645 shutdown_connections 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.275+0000 7f02f6681640 1 -- 172.21.15.122:0/3297684645 wait complete. 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.276+0000 7f02f6681640 1 Processor -- start 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.276+0000 7f02f6681640 1 -- start start 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f6681640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f6681640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f02f00ce0a0 0x7f02f0159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f6681640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f015ac00 0x7f02f0155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f6681640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f00cd8a0 con 0x7f02f00708b0 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f6681640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f00cd5c0 con 0x7f02f015ac00 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f6681640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f00cf5d0 con 0x7f02f00ce0a0 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45130/0 (socket says 172.21.15.122:45130) 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 -- 172.21.15.122:0/2516052247 learned_addr learned my addr 172.21.15.122:0/2516052247 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02f4c1e640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f015ac00 0x7f02f0155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02ef7fe640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f02f00ce0a0 0x7f02f0159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 -- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f02f00ce0a0 msgr2=0x7f02f0159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f02f00ce0a0 0x7f02f0159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 -- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f015ac00 msgr2=0x7f02f0155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f015ac00 0x7f02f0155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 -- 172.21.15.122:0/2516052247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0158420 con 0x7f02f00708b0 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.277+0000 7f02effff640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0152fe0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c3c0 tx=0x7f02e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.278+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e00052d0 con 0x7f02f00708b0 2024-03-06T07:34:28.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.278+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e0005490 con 0x7f02f00708b0 2024-03-06T07:34:28.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.278+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0007260 con 0x7f02f00708b0 2024-03-06T07:34:28.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.278+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f01586c0 con 0x7f02f00708b0 2024-03-06T07:34:28.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.278+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f0158c80 con 0x7f02f00708b0 2024-03-06T07:34:28.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.278+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02f0070d10 con 0x7f02f00708b0 2024-03-06T07:34:28.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.279+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02e0007400 con 0x7f02f00708b0 2024-03-06T07:34:28.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.279+0000 7f02ecff9640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f02dc06b4d0 0x7f02dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.279+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f02e0089300 con 0x7f02f00708b0 2024-03-06T07:34:28.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02ef7fe640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f02dc06b4d0 0x7f02dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02ecff9640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f02dc071430 0x7f02dc073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02e0009e50 con 0x7f02dc071430 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02e0090050 con 0x7f02f00708b0 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02f4c1e640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f02dc071430 0x7f02dc073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02ef7fe640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f02dc06b4d0 0x7f02dc06d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f02e4036000 tx=0x7f02e403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.280+0000 7f02f4c1e640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f02dc071430 0x7f02dc073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.281+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== osd.4 v2:172.21.15.169:6804/1079551346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02e0009e50 con 0x7f02dc071430 2024-03-06T07:34:28.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.294+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 --> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f02f004f330 con 0x7f02dc071430 2024-03-06T07:34:28.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.294+0000 7f02ecff9640 1 -- 172.21.15.122:0/2516052247 <== osd.4 v2:172.21.15.169:6804/1079551346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f02f004f330 con 0x7f02dc071430 2024-03-06T07:34:28.297 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:28.297 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:28.297 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:28.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f02dc071430 msgr2=0x7f02dc073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.169:6804/1079551346,v1:172.21.15.169:6806/1079551346] conn(0x7f02dc071430 0x7f02dc073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f02dc06b4d0 msgr2=0x7f02dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f02dc06b4d0 0x7f02dc06d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f02e4036000 tx=0x7f02e403c040 comp rx=0 tx=0).stop 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 msgr2=0x7f02f0152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 --2- 172.21.15.122:0/2516052247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f02f00708b0 0x7f02f0152fe0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c3c0 tx=0x7f02e000c880 comp rx=0 tx=0).stop 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f4c1e640 1 -- 172.21.15.122:0/2516052247 reap_dead start 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 shutdown_connections 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 >> 172.21.15.122:0/2516052247 conn(0x7f02f00b1650 msgr2=0x7f02f00af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 shutdown_connections 2024-03-06T07:34:28.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.295+0000 7f02f6681640 1 -- 172.21.15.122:0/2516052247 wait complete. 2024-03-06T07:34:28.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:34:28.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.405+0000 7fbb7fa9a640 1 Processor -- start 2024-03-06T07:34:28.407 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.405+0000 7fbb7fa9a640 1 -- start start 2024-03-06T07:34:28.408 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7fa9a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb78158880 0x7fbb7806c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7fa9a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7806d030 con 0x7fbb7815b380 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7fa9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7806d310 con 0x7fbb7806cb90 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7fa9a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7806d5f0 con 0x7fbb78158880 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7d836640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fbb78158880 0x7fbb7806c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7d035640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fbb7815b380 0x7fbb780708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58330/0 (socket says 172.21.15.122:58330) 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7d035640 1 -- 172.21.15.122:0/3945128986 learned_addr learned my addr 172.21.15.122:0/3945128986 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7d836640 1 -- 172.21.15.122:0/3945128986 >> v1:172.21.15.169:6789/0 conn(0x7fbb7806cb90 legacy=0x7fbb780cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.409 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7d836640 1 -- 172.21.15.122:0/3945128986 >> v1:172.21.15.122:6789/0 conn(0x7fbb7815b380 legacy=0x7fbb780708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7d836640 1 -- 172.21.15.122:0/3945128986 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7806d8d0 con 0x7fbb78158880 2024-03-06T07:34:28.410 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.406+0000 7fbb7c834640 1 -- 172.21.15.122:0/3945128986 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699203752 0 0) 0x7fbb7806d030 con 0x7fbb7815b380 2024-03-06T07:34:28.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.407+0000 7fbb7d836640 1 --2- 172.21.15.122:0/3945128986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb78158880 0x7fbb7806c650 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009d90 tx=0x7fbb6802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d195ac502c8c419 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.407+0000 7fbb7c834640 1 -- 172.21.15.122:0/3945128986 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68039070 con 0x7fbb78158880 2024-03-06T07:34:28.411 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.407+0000 7fbb7c834640 1 -- 172.21.15.122:0/3945128986 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6802cd00 con 0x7fbb78158880 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.407+0000 7fbb7c834640 1 -- 172.21.15.122:0/3945128986 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68034660 con 0x7fbb78158880 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/3945128986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb78158880 msgr2=0x7fbb7806c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/3945128986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb78158880 0x7fbb7806c650 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009d90 tx=0x7fbb6802bf50 comp rx=0 tx=0).stop 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/3945128986 shutdown_connections 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/3945128986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb78158880 0x7fbb7806c650 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/3945128986 >> 172.21.15.122:0/3945128986 conn(0x7fbb780b1c80 msgr2=0x7fbb780cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/3945128986 shutdown_connections 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.408+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/3945128986 wait complete. 2024-03-06T07:34:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.409+0000 7fbb7fa9a640 1 Processor -- start 2024-03-06T07:34:28.413 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.409+0000 7fbb7fa9a640 1 -- start start 2024-03-06T07:34:28.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 0x7fbb781513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb78158880 0x7fbb7815d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb7815b380 0x7fbb7815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78071160 con 0x7fbb7806cb90 2024-03-06T07:34:28.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78070fe0 con 0x7fbb78158880 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb780cded0 con 0x7fbb7815b380 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 0x7fbb781513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 0x7fbb781513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45146/0 (socket says 172.21.15.122:45146) 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 -- 172.21.15.122:0/2648738327 learned_addr learned my addr 172.21.15.122:0/2648738327 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d035640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb78158880 0x7fbb7815d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7e037640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb7815b380 0x7fbb7815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 -- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb7815b380 msgr2=0x7fbb7815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.416 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb7815b380 0x7fbb7815d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 -- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb78158880 msgr2=0x7fbb7815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb78158880 0x7fbb7815d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 -- 172.21.15.122:0/2648738327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78151920 con 0x7fbb7806cb90 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7d836640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 0x7fbb781513e0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fbb68037040 tx=0x7fbb68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68039050 con 0x7fbb7806cb90 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68045070 con 0x7fbb7806cb90 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb780c5780 con 0x7fbb7806cb90 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.410+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6803ea50 con 0x7fbb7806cb90 2024-03-06T07:34:28.417 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.411+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb780c5d40 con 0x7fbb7806cb90 2024-03-06T07:34:28.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.412+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7804f330 con 0x7fbb7806cb90 2024-03-06T07:34:28.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.412+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb6802cbc0 con 0x7fbb7806cb90 2024-03-06T07:34:28.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.412+0000 7fbb767fc640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb6406b4d0 0x7fbb6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.412+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbb680ace10 con 0x7fbb7806cb90 2024-03-06T07:34:28.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.412+0000 7fbb7d035640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb6406b4d0 0x7fbb6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.418 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.413+0000 7fbb7d035640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb6406b4d0 0x7fbb6406d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fbb780b3ef0 tx=0x7fbb6c007d00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.419 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.417+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb6807b8e0 con 0x7fbb7806cb90 2024-03-06T07:34:28.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.589+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb780c6080 con 0x7fbb6406b4d0 2024-03-06T07:34:28.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.589+0000 7fbb767fc640 1 -- 172.21.15.122:0/2648738327 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbb780c6080 con 0x7fbb6406b4d0 2024-03-06T07:34:28.592 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:34:28.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb6406b4d0 msgr2=0x7fbb6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb6406b4d0 0x7fbb6406d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fbb780b3ef0 tx=0x7fbb6c007d00 comp rx=0 tx=0).stop 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 msgr2=0x7fbb781513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 0x7fbb781513e0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fbb68037040 tx=0x7fbb68004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 shutdown_connections 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fbb6406b4d0 0x7fbb6406d990 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fbb7815b380 0x7fbb7815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb78158880 0x7fbb7815d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 --2- 172.21.15.122:0/2648738327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb7806cb90 0x7fbb781513e0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 >> 172.21.15.122:0/2648738327 conn(0x7fbb780b1c80 msgr2=0x7fbb780708d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 shutdown_connections 2024-03-06T07:34:28.595 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.591+0000 7fbb7fa9a640 1 -- 172.21.15.122:0/2648738327 wait complete. 2024-03-06T07:34:28.604 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:34:28.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:28.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.706+0000 7fc1d25d7640 1 Processor -- start 2024-03-06T07:34:28.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.706+0000 7fc1d25d7640 1 -- start start 2024-03-06T07:34:28.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1d25d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 0x7fc1cc0b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1d25d7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc068b00 con 0x7fc1cc15f9d0 2024-03-06T07:34:28.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1d25d7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc15e4e0 con 0x7fc1cc0b5ca0 2024-03-06T07:34:28.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1d25d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc15edf0 con 0x7fc1cc06be80 2024-03-06T07:34:28.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1d0b74640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc1cc15f9d0 0x7fc1cc15fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58332/0 (socket says 172.21.15.122:58332) 2024-03-06T07:34:28.711 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1d0b74640 1 -- 172.21.15.122:0/3018714191 learned_addr learned my addr 172.21.15.122:0/3018714191 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293489074 0 0) 0x7fc1cc068b00 con 0x7fc1cc15f9d0 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b4003520 con 0x7fc1cc15f9d0 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.707+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 0x7fc1cc0b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414089231 0 0) 0x7fc1b4003520 con 0x7fc1cc15f9d0 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 msgr2=0x7fc1cc0b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 0x7fc1cc0b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 >> v1:172.21.15.169:6789/0 conn(0x7fc1cc0b5ca0 legacy=0x7fc1cc06c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0b1900 con 0x7fc1cc15f9d0 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fc1c40019f0 con 0x7fc1cc15f9d0 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 >> v1:172.21.15.122:6789/0 conn(0x7fc1cc15f9d0 legacy=0x7fc1cc15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1b4004800 0x7fc1b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 0x7fc1b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc15edf0 con 0x7fc1b40039e0 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4005520 con 0x7fc1b4004800 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b400bf00 con 0x7fc1cc06be80 2024-03-06T07:34:28.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1d0b74640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 0x7fc1b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cb7fe640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1b4004800 0x7fc1b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cbfff640 1 -- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 msgr2=0x7fc1b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06be80 0x7fc1b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cbfff640 1 -- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1b4004800 msgr2=0x7fc1b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1b4004800 0x7fc1b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.708+0000 7fc1cbfff640 1 -- 172.21.15.122:0/3018714191 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4003520 con 0x7fc1b40039e0 2024-03-06T07:34:28.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1b4003e00 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00ba30 tx=0x7fc1b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf759016a0ba8f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc036020 con 0x7fc1b40039e0 2024-03-06T07:34:28.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc032640 con 0x7fc1b40039e0 2024-03-06T07:34:28.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1caffd640 1 -- 172.21.15.122:0/3018714191 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00b600 con 0x7fc1b40039e0 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1d25d7640 1 -- 172.21.15.122:0/3018714191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 msgr2=0x7fc1b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1d25d7640 1 --2- 172.21.15.122:0/3018714191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1b4003e00 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00ba30 tx=0x7fc1b4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1d0b74640 1 -- 172.21.15.122:0/3018714191 reap_dead start 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1d25d7640 1 -- 172.21.15.122:0/3018714191 shutdown_connections 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.709+0000 7fc1d25d7640 1 -- 172.21.15.122:0/3018714191 >> 172.21.15.122:0/3018714191 conn(0x7fc1cc0b4ed0 msgr2=0x7fc1cc0b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.710+0000 7fc1d25d7640 1 -- 172.21.15.122:0/3018714191 shutdown_connections 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.710+0000 7fc1d25d7640 1 -- 172.21.15.122:0/3018714191 wait complete. 2024-03-06T07:34:28.719 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 Processor -- start 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 -- start start 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1cc151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc15a800 0x7fc1cc1f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06dfd0 0x7fc1cc06e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc15ea80 con 0x7fc1b40039e0 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1618a0 con 0x7fc1cc15a800 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.711+0000 7fc1d25d7640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc15f280 con 0x7fc1cc06dfd0 2024-03-06T07:34:28.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1cc151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1cc151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45168/0 (socket says 172.21.15.122:45168) 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 -- 172.21.15.122:0/2897429719 learned_addr learned my addr 172.21.15.122:0/2897429719 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1d0b74640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06dfd0 0x7fc1cc06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cb7fe640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc15a800 0x7fc1cc1f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 -- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06dfd0 msgr2=0x7fc1cc06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fc1cc06dfd0 0x7fc1cc06e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 -- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc15a800 msgr2=0x7fc1cc1f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc15a800 0x7fc1cc1f74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 -- 172.21.15.122:0/2897429719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc151a80 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cb7fe640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc15a800 0x7fc1cc1f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1cbfff640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1cc151540 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc035040 tx=0x7fc1bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc041070 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc030a50 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc036040 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc1f7de0 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.712+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc1f83a0 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.713+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc1cc04f330 con 0x7fc1b40039e0 2024-03-06T07:34:28.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.713+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1bc039020 con 0x7fc1b40039e0 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1c8ff9640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1b806b500 0x7fc1b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc1bc0ac170 con 0x7fc1b40039e0 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1cb7fe640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1b806b500 0x7fc1b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1c8ff9640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7fc1b8071460 0x7fc1b8073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1bc03be50 con 0x7fc1b8071460 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1bc0b3050 con 0x7fc1b40039e0 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1d0b74640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7fc1b8071460 0x7fc1b8073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.714+0000 7fc1cb7fe640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1b806b500 0x7fc1b806d9c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0d0d50 tx=0x7fc1c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.715+0000 7fc1d0b74640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7fc1b8071460 0x7fc1b8073880 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.715+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== osd.5 v2:172.21.15.169:6816/887889200 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc1bc03be50 con 0x7fc1b8071460 2024-03-06T07:34:28.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.728+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 --> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc1cc0af770 con 0x7fc1b8071460 2024-03-06T07:34:28.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1c8ff9640 1 -- 172.21.15.122:0/2897429719 <== osd.5 v2:172.21.15.169:6816/887889200 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc1cc0af770 con 0x7fc1b8071460 2024-03-06T07:34:28.731 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:28.731 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:28.731 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:28.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7fc1b8071460 msgr2=0x7fc1b8073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.732 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.169:6816/887889200,v1:172.21.15.169:6817/887889200] conn(0x7fc1b8071460 0x7fc1b8073880 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1b806b500 msgr2=0x7fc1b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fc1b806b500 0x7fc1b806d9c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0d0d50 tx=0x7fc1c0009040 comp rx=0 tx=0).stop 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 msgr2=0x7fc1cc151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 --2- 172.21.15.122:0/2897429719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1b40039e0 0x7fc1cc151540 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc035040 tx=0x7fc1bc004730 comp rx=0 tx=0).stop 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d0b74640 1 -- 172.21.15.122:0/2897429719 reap_dead start 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 shutdown_connections 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 >> 172.21.15.122:0/2897429719 conn(0x7fc1cc0b4ed0 msgr2=0x7fc1cc0af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.733 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 shutdown_connections 2024-03-06T07:34:28.734 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.729+0000 7fc1d25d7640 1 -- 172.21.15.122:0/2897429719 wait complete. 2024-03-06T07:34:28.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:34:28.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.840+0000 7f7659e5c640 1 Processor -- start 2024-03-06T07:34:28.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.840+0000 7f7659e5c640 1 -- start start 2024-03-06T07:34:28.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f7659e5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 0x7f76540d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f7659e5c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f7654161cf0 0x7f7654162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f7659e5c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540d22c0 con 0x7f76540b4460 2024-03-06T07:34:28.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f7659e5c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7654161b00 con 0x7f7654161cf0 2024-03-06T07:34:28.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f7659e5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76540cecc0 con 0x7f7654162970 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f76537fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f76540b4460 0x7f76540b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58334/0 (socket says 172.21.15.122:58334) 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.841+0000 7f76537fe640 1 -- 172.21.15.122:0/2262668878 learned_addr learned my addr 172.21.15.122:0/2262668878 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f764affd640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 0x7f76540d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 --2- 172.21.15.122:0/2262668878 >> v2:172.21.15.169:3300/0 conn(0x7f7654161cf0 0x7f7654162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104939371 0 0) 0x7f76540d22c0 con 0x7f76540b4460 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7640003520 con 0x7f76540b4460 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1644363677 0 0) 0x7f7640003520 con 0x7f76540b4460 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 msgr2=0x7f76540d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 0x7f76540d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 >> v2:172.21.15.169:3300/0 conn(0x7f7654161cf0 msgr2=0x7f7654162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 --2- 172.21.15.122:0/2262668878 >> v2:172.21.15.169:3300/0 conn(0x7f7654161cf0 0x7f7654162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76540d25a0 con 0x7f76540b4460 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f7644001670 con 0x7f76540b4460 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 >> v1:172.21.15.122:6789/0 conn(0x7f76540b4460 legacy=0x7f76540b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76400039b0 0x7f7640003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 0x7f7640004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7654161b00 con 0x7f7640004d40 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540cecc0 con 0x7f76400039b0 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764000b9c0 con 0x7f7654162970 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f76537fe640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76400039b0 0x7f7640003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f764affd640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 0x7f7640004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 -- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 msgr2=0x7f7640004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654162970 0x7f7640004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 -- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76400039b0 msgr2=0x7f7640003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76400039b0 0x7f7640003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.842+0000 7f7653fff640 1 -- 172.21.15.122:0/2262668878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7640003520 con 0x7f7640004d40 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.843+0000 7f7653fff640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76400051c0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f764c009fc0 tx=0x7f7640005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d204f15f3d18b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.843+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c039070 con 0x7f7640004d40 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.843+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c02cc60 con 0x7f7640004d40 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.843+0000 7f7652ffd640 1 -- 172.21.15.122:0/2262668878 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c009c10 con 0x7f7640004d40 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7659e5c640 1 -- 172.21.15.122:0/2262668878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 msgr2=0x7f76400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7659e5c640 1 --2- 172.21.15.122:0/2262668878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76400051c0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f764c009fc0 tx=0x7f7640005830 comp rx=0 tx=0).stop 2024-03-06T07:34:28.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7653fff640 1 -- 172.21.15.122:0/2262668878 reap_dead start 2024-03-06T07:34:28.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7659e5c640 1 -- 172.21.15.122:0/2262668878 shutdown_connections 2024-03-06T07:34:28.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7659e5c640 1 -- 172.21.15.122:0/2262668878 >> 172.21.15.122:0/2262668878 conn(0x7f76540d3ce0 msgr2=0x7f76540d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7659e5c640 1 -- 172.21.15.122:0/2262668878 shutdown_connections 2024-03-06T07:34:28.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.844+0000 7f7659e5c640 1 -- 172.21.15.122:0/2262668878 wait complete. 2024-03-06T07:34:28.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.845+0000 7f7659e5c640 1 Processor -- start 2024-03-06T07:34:28.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.845+0000 7f7659e5c640 1 -- start start 2024-03-06T07:34:28.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7659e5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76540cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7659e5c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654155d90 0x7f7654156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7659e5c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76540c8db0 0x7f76540c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7659e5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540ce950 con 0x7f7640004d40 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7659e5c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540d2b60 con 0x7f76540c8db0 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7659e5c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76540b4840 con 0x7f7654155d90 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76540cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76540cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45194/0 (socket says 172.21.15.122:45194) 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 -- 172.21.15.122:0/55428937 learned_addr learned my addr 172.21.15.122:0/55428937 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7653fff640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76540c8db0 0x7f76540c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f764affd640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654155d90 0x7f7654156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 -- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654155d90 msgr2=0x7f7654156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654155d90 0x7f7654156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 -- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76540c8db0 msgr2=0x7f76540c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76540c8db0 0x7f76540c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 -- 172.21.15.122:0/55428937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654156750 con 0x7f7640004d40 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f764affd640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654155d90 0x7f7654156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7653fff640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76540c8db0 0x7f76540c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f76537fe640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76540cd310 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f7644007340 tx=0x7f7644007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.846+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400f9c0 con 0x7f7640004d40 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.847+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644018070 con 0x7f7640004d40 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.847+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76540c9ae0 con 0x7f7640004d40 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.847+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400fda0 con 0x7f7640004d40 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.847+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76540caf50 con 0x7f7640004d40 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.848+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765404f330 con 0x7f7640004d40 2024-03-06T07:34:28.856 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.848+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f764401d460 con 0x7f7640004d40 2024-03-06T07:34:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.848+0000 7f7650ff9640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f763406b500 0x7f763406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.848+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f764408a0d0 con 0x7f7640004d40 2024-03-06T07:34:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.848+0000 7f764affd640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f763406b500 0x7f763406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.849+0000 7f764affd640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f763406b500 0x7f763406d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f76540a1be0 tx=0x7f763c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:28.853+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7654156750 con 0x7f7640004d40 2024-03-06T07:34:29.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.024+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f76540cad90 con 0x7f763406b500 2024-03-06T07:34:29.026 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.024+0000 7f7650ff9640 1 -- 172.21.15.122:0/55428937 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f763c0139f0 con 0x7f763406b500 2024-03-06T07:34:29.026 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:34:29.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f763406b500 msgr2=0x7f763406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f763406b500 0x7f763406d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f76540a1be0 tx=0x7f763c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:29.028 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 msgr2=0x7f76540cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76540cd310 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f7644007340 tx=0x7f7644007800 comp rx=0 tx=0).stop 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 shutdown_connections 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f763406b500 0x7f763406d9c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76540c8db0 0x7f76540c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f7654155d90 0x7f7654156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 --2- 172.21.15.122:0/55428937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7640004d40 0x7f76540cd310 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 >> 172.21.15.122:0/55428937 conn(0x7f76540d3ce0 msgr2=0x7f76540d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 shutdown_connections 2024-03-06T07:34:29.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.026+0000 7f7659e5c640 1 -- 172.21.15.122:0/55428937 wait complete. 2024-03-06T07:34:29.038 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:34:29.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:29.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.140+0000 7ff74c95e640 1 Processor -- start 2024-03-06T07:34:29.142 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.140+0000 7ff74c95e640 1 -- start start 2024-03-06T07:34:29.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff74c95e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7ff74815dfa0 0x7ff7480d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff74c95e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74815e380 con 0x7ff7480d29f0 2024-03-06T07:34:29.146 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff74c95e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480d1e30 con 0x7ff7480d4630 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff74c95e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7480d2100 con 0x7ff74815dfa0 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff746d9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff7480d29f0 0x7ff7480cb190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58348/0 (socket says 172.21.15.122:58348) 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff746d9d640 1 -- 172.21.15.122:0/2540648481 learned_addr learned my addr 172.21.15.122:0/2540648481 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.141+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646390804 0 0) 0x7ff74815e380 con 0x7ff7480d29f0 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff730003520 con 0x7ff7480d29f0 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 --2- 172.21.15.122:0/2540648481 >> v2:172.21.15.169:3300/0 conn(0x7ff74815dfa0 0x7ff7480d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124068380 0 0) 0x7ff730003520 con 0x7ff7480d29f0 2024-03-06T07:34:29.147 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 >> v2:172.21.15.169:3300/0 conn(0x7ff74815dfa0 msgr2=0x7ff7480d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 --2- 172.21.15.122:0/2540648481 >> v2:172.21.15.169:3300/0 conn(0x7ff74815dfa0 0x7ff7480d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 >> v1:172.21.15.169:6789/0 conn(0x7ff7480d4630 legacy=0x7ff7480d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7480cf470 con 0x7ff7480d29f0 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7ff740001670 con 0x7ff7480d29f0 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 >> v1:172.21.15.122:6789/0 conn(0x7ff7480d29f0 legacy=0x7ff7480cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7300039b0 0x7ff730003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300051c0 0x7ff73000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480d2100 con 0x7ff7300039b0 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000c160 con 0x7ff730004800 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000c360 con 0x7ff7300051c0 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7300039b0 0x7ff730003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff746d9d640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300051c0 0x7ff73000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff745d9b640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 -- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300051c0 msgr2=0x7ff73000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300051c0 0x7ff73000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 -- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff730004800 msgr2=0x7ff730004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff74659c640 1 -- 172.21.15.122:0/2540648481 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730003520 con 0x7ff7300039b0 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff746d9d640 1 -- 172.21.15.122:0/2540648481 reap_dead start 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.142+0000 7ff745d9b640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.143+0000 7ff74659c640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7300039b0 0x7ff730003dd0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7ff738009fa0 tx=0x7ff730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4983718983198040 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.143+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738039070 con 0x7ff7300039b0 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.143+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73802ccb0 con 0x7ff7300039b0 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.143+0000 7ff74559a640 1 -- 172.21.15.122:0/2540648481 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738009bf0 con 0x7ff7300039b0 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 -- 172.21.15.122:0/2540648481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7300039b0 msgr2=0x7ff730003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.150 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7300039b0 0x7ff730003dd0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7ff738009fa0 tx=0x7ff730005830 comp rx=0 tx=0).stop 2024-03-06T07:34:29.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 -- 172.21.15.122:0/2540648481 shutdown_connections 2024-03-06T07:34:29.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 --2- 172.21.15.122:0/2540648481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7300039b0 0x7ff730003dd0 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 -- 172.21.15.122:0/2540648481 >> 172.21.15.122:0/2540648481 conn(0x7ff74809d000 msgr2=0x7ff7480d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 -- 172.21.15.122:0/2540648481 shutdown_connections 2024-03-06T07:34:29.151 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.144+0000 7ff74c95e640 1 -- 172.21.15.122:0/2540648481 wait complete. 2024-03-06T07:34:29.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.145+0000 7ff74c95e640 1 Processor -- start 2024-03-06T07:34:29.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.145+0000 7ff74c95e640 1 -- start start 2024-03-06T07:34:29.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74c95e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300039b0 0x7ff7480cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74c95e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7480c99a0 0x7ff7480cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.154 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74c95e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff74815bf80 0x7ff7480b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74c95e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480d18f0 con 0x7ff74815bf80 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74c95e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480cfa30 con 0x7ff7480c99a0 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74c95e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480d1a70 con 0x7ff7300039b0 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff74815bf80 0x7ff7480b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff74815bf80 0x7ff7480b8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45218/0 (socket says 172.21.15.122:45218) 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 -- 172.21.15.122:0/2202210506 learned_addr learned my addr 172.21.15.122:0/2202210506 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74659c640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300039b0 0x7ff7480cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff745d9b640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7480c99a0 0x7ff7480cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.155 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 -- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300039b0 msgr2=0x7ff7480cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300039b0 0x7ff7480cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 -- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7480c99a0 msgr2=0x7ff7480cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7480c99a0 0x7ff7480cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 -- 172.21.15.122:0/2202210506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7480c8900 con 0x7ff74815bf80 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff74659c640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7ff7300039b0 0x7ff7480cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff746d9d640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff74815bf80 0x7ff7480b8ae0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7ff7480d18a0 tx=0x7ff740007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.146+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74000f620 con 0x7ff74815bf80 2024-03-06T07:34:29.156 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.147+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff740018070 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.147+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74000fa00 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.147+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7480c8be0 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.147+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7480c91a0 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.147+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff74804f330 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.148+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff74000fba0 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.148+0000 7ff736ffd640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff72806b4d0 0x7ff72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.148+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff740089c00 con 0x7ff74815bf80 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.148+0000 7ff74659c640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff72806b4d0 0x7ff72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.149+0000 7ff736ffd640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ff728071430 0x7ff728073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.157 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.149+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff740009e50 con 0x7ff728071430 2024-03-06T07:34:29.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.149+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff74008c300 con 0x7ff74815bf80 2024-03-06T07:34:29.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.149+0000 7ff745d9b640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ff728071430 0x7ff728073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.149+0000 7ff74659c640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff72806b4d0 0x7ff72806d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff738036000 tx=0x7ff73802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.149+0000 7ff745d9b640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ff728071430 0x7ff728073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.158 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.150+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== osd.6 v2:172.21.15.169:6824/1645803606 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff740009e50 con 0x7ff728071430 2024-03-06T07:34:29.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.163+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 --> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff7480c93e0 con 0x7ff728071430 2024-03-06T07:34:29.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.163+0000 7ff736ffd640 1 -- 172.21.15.122:0/2202210506 <== osd.6 v2:172.21.15.169:6824/1645803606 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff7480c93e0 con 0x7ff728071430 2024-03-06T07:34:29.166 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:29.166 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:29.166 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ff728071430 msgr2=0x7ff728073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.169:6824/1645803606,v1:172.21.15.169:6825/1645803606] conn(0x7ff728071430 0x7ff728073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff72806b4d0 msgr2=0x7ff72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7ff72806b4d0 0x7ff72806d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff738036000 tx=0x7ff73802e000 comp rx=0 tx=0).stop 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff74815bf80 msgr2=0x7ff7480b8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 --2- 172.21.15.122:0/2202210506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff74815bf80 0x7ff7480b8ae0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7ff7480d18a0 tx=0x7ff740007430 comp rx=0 tx=0).stop 2024-03-06T07:34:29.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff746d9d640 1 -- 172.21.15.122:0/2202210506 reap_dead start 2024-03-06T07:34:29.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 shutdown_connections 2024-03-06T07:34:29.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 >> 172.21.15.122:0/2202210506 conn(0x7ff74809d000 msgr2=0x7ff7481605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 shutdown_connections 2024-03-06T07:34:29.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.164+0000 7ff74c95e640 1 -- 172.21.15.122:0/2202210506 wait complete. 2024-03-06T07:34:29.176 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:34:29.276 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.274+0000 7f98bbc0e640 1 Processor -- start 2024-03-06T07:34:29.277 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.274+0000 7f98bbc0e640 1 -- start start 2024-03-06T07:34:29.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98bbc0e640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f98b41614a0 0x7f98b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.278 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98bbc0e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f98b40b1a20 0x7f98b40b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98bbc0e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b40af4b0 con 0x7f98b40b0270 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98bbc0e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40af790 con 0x7f98b40b1a20 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98bbc0e640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40afa70 con 0x7f98b41614a0 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98ba1ab640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f98b40b1a20 0x7f98b40b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98ba1ab640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f98b40b1a20 0x7f98b40b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45226/0 (socket says 172.21.15.122:45226) 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.275+0000 7f98ba1ab640 1 -- 172.21.15.122:0/689002094 learned_addr learned my addr 172.21.15.122:0/689002094 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.276+0000 7f98ba1ab640 1 -- 172.21.15.122:0/689002094 >> v2:172.21.15.169:3300/0 conn(0x7f98b41614a0 msgr2=0x7f98b406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.276+0000 7f98ba1ab640 1 --2- 172.21.15.122:0/689002094 >> v2:172.21.15.169:3300/0 conn(0x7f98b41614a0 0x7f98b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.279 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.276+0000 7f98ba1ab640 1 -- 172.21.15.122:0/689002094 >> v1:172.21.15.169:6789/0 conn(0x7f98b40b0270 legacy=0x7f98b40b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.276+0000 7f98ba1ab640 1 -- 172.21.15.122:0/689002094 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40afd50 con 0x7f98b40b1a20 2024-03-06T07:34:29.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.276+0000 7f98ba1ab640 1 --2- 172.21.15.122:0/689002094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b1a20 0x7f98b40b5510 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f98b40730b0 tx=0x7f98ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1de8c7ccc7172a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.276+0000 7f98b89a8640 1 -- 172.21.15.122:0/689002094 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac02e070 con 0x7f98b40b1a20 2024-03-06T07:34:29.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98b89a8640 1 -- 172.21.15.122:0/689002094 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac037070 con 0x7f98b40b1a20 2024-03-06T07:34:29.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98b89a8640 1 -- 172.21.15.122:0/689002094 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac031a00 con 0x7f98b40b1a20 2024-03-06T07:34:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 -- 172.21.15.122:0/689002094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b1a20 msgr2=0x7f98b40b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/689002094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b1a20 0x7f98b40b5510 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f98b40730b0 tx=0x7f98ac02b880 comp rx=0 tx=0).stop 2024-03-06T07:34:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 -- 172.21.15.122:0/689002094 shutdown_connections 2024-03-06T07:34:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/689002094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b1a20 0x7f98b40b5510 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/689002094 >> v2:172.21.15.169:3300/0 conn(0x7f98b41614a0 0x7f98b406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.283 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 -- 172.21.15.122:0/689002094 >> 172.21.15.122:0/689002094 conn(0x7f98b409a9f0 msgr2=0x7f98b409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 -- 172.21.15.122:0/689002094 shutdown_connections 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.277+0000 7f98bbc0e640 1 -- 172.21.15.122:0/689002094 wait complete. 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.278+0000 7f98bbc0e640 1 Processor -- start 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.278+0000 7f98bbc0e640 1 -- start start 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98bbc0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 0x7f98b4147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98bbc0e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b1a20 0x7f98b41475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98bbc0e640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b41614a0 0x7f98b4158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98bbc0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40b1e00 con 0x7f98b40b0270 2024-03-06T07:34:29.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98bbc0e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40b06e0 con 0x7f98b40b1a20 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98bbc0e640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40b1f80 con 0x7f98b41614a0 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 0x7f98b4147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 0x7f98b4147fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45228/0 (socket says 172.21.15.122:45228) 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 -- 172.21.15.122:0/4167589993 learned_addr learned my addr 172.21.15.122:0/4167589993 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 -- 172.21.15.122:0/4167589993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b41614a0 msgr2=0x7f98b4158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b41614a0 0x7f98b4158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 -- 172.21.15.122:0/4167589993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b1a20 msgr2=0x7f98b41475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b1a20 0x7f98b41475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 -- 172.21.15.122:0/4167589993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b4159710 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.279+0000 7f98b99aa640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 0x7f98b4147fa0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f98b000ba80 tx=0x7f98b000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.280+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0019070 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.280+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b000cc20 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.280+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b40cc140 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.280+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0014440 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.280+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b40cc700 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.281+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b404f330 con 0x7f98b40b0270 2024-03-06T07:34:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.281+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98b001c070 con 0x7f98b40b0270 2024-03-06T07:34:29.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.281+0000 7f98aa7fc640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f989806b4d0 0x7f989806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.282+0000 7f98b91a9640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f989806b4d0 0x7f989806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.282+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f98b0089ba0 con 0x7f98b40b0270 2024-03-06T07:34:29.287 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.282+0000 7f98b91a9640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f989806b4d0 0x7f989806d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f98b415c060 tx=0x7f98a4006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.288 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.286+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98b0058670 con 0x7f98b40b0270 2024-03-06T07:34:29.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.458+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f98b406e8e0 con 0x7f989806b4d0 2024-03-06T07:34:29.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.459+0000 7f98aa7fc640 1 -- 172.21.15.122:0/4167589993 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f98b406e8e0 con 0x7f989806b4d0 2024-03-06T07:34:29.461 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:34:29.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.460+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f989806b4d0 msgr2=0x7f989806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.460+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f989806b4d0 0x7f989806d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f98b415c060 tx=0x7f98a4006ba0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.460+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 msgr2=0x7f98b4147fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.460+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 0x7f98b4147fa0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f98b000ba80 tx=0x7f98b000bf40 comp rx=0 tx=0).stop 2024-03-06T07:34:29.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 shutdown_connections 2024-03-06T07:34:29.463 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f989806b4d0 0x7f989806d990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f98b41614a0 0x7f98b4158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98b40b1a20 0x7f98b41475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 --2- 172.21.15.122:0/4167589993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98b40b0270 0x7f98b4147fa0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 >> 172.21.15.122:0/4167589993 conn(0x7f98b409a9f0 msgr2=0x7f98b40b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 shutdown_connections 2024-03-06T07:34:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.461+0000 7f98bbc0e640 1 -- 172.21.15.122:0/4167589993 wait complete. 2024-03-06T07:34:29.473 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:34:29.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:29.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.575+0000 7f9f584a3640 1 Processor -- start 2024-03-06T07:34:29.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.575+0000 7f9f584a3640 1 -- start start 2024-03-06T07:34:29.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f584a3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015ac00 0x7f9f5006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.578 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f584a3640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f9f500708b0 0x7f9f50070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f584a3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9f5015e7a0 0x7f9f5015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f584a3640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f50071250 con 0x7f9f5015e7a0 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f584a3640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f500cdc10 con 0x7f9f500708b0 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f584a3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5015afe0 con 0x7f9f5015ac00 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9f5015e7a0 0x7f9f5015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9f5015e7a0 0x7f9f5015ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45234/0 (socket says 172.21.15.122:45234) 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 -- 172.21.15.122:0/3010532839 learned_addr learned my addr 172.21.15.122:0/3010532839 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f5623f640 1 --2- 172.21.15.122:0/3010532839 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015ac00 0x7f9f5006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f55a3e640 1 --2- 172.21.15.122:0/3010532839 >> v2:172.21.15.169:3300/0 conn(0x7f9f500708b0 0x7f9f50070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 -- 172.21.15.122:0/3010532839 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015ac00 msgr2=0x7f9f5006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3010532839 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015ac00 0x7f9f5006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 -- 172.21.15.122:0/3010532839 >> v2:172.21.15.169:3300/0 conn(0x7f9f500708b0 msgr2=0x7f9f50070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3010532839 >> v2:172.21.15.169:3300/0 conn(0x7f9f500708b0 0x7f9f50070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.576+0000 7f9f56a40640 1 -- 172.21.15.122:0/3010532839 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f500cd000 con 0x7f9f5015e7a0 2024-03-06T07:34:29.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.577+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3010532839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015e7a0 0x7f9f5015ec20 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f9f5006c860 tx=0x7f9f4802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f668371f0e589ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.577+0000 7f9f5523d640 1 -- 172.21.15.122:0/3010532839 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4802c9e0 con 0x7f9f5015e7a0 2024-03-06T07:34:29.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.577+0000 7f9f5523d640 1 -- 172.21.15.122:0/3010532839 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4802cb80 con 0x7f9f5015e7a0 2024-03-06T07:34:29.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.577+0000 7f9f5523d640 1 -- 172.21.15.122:0/3010532839 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f480343f0 con 0x7f9f5015e7a0 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 -- 172.21.15.122:0/3010532839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015e7a0 msgr2=0x7f9f5015ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3010532839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015e7a0 0x7f9f5015ec20 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f9f5006c860 tx=0x7f9f4802be40 comp rx=0 tx=0).stop 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 -- 172.21.15.122:0/3010532839 shutdown_connections 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3010532839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015e7a0 0x7f9f5015ec20 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3010532839 >> v2:172.21.15.169:3300/0 conn(0x7f9f500708b0 0x7f9f50070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3010532839 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015ac00 0x7f9f5006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 -- 172.21.15.122:0/3010532839 >> 172.21.15.122:0/3010532839 conn(0x7f9f500b1650 msgr2=0x7f9f500b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 -- 172.21.15.122:0/3010532839 shutdown_connections 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.578+0000 7f9f584a3640 1 -- 172.21.15.122:0/3010532839 wait complete. 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.579+0000 7f9f584a3640 1 Processor -- start 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.579+0000 7f9f584a3640 1 -- start start 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f500708b0 0x7f9f5015c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015ac00 0x7f9f501423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015e7a0 0x7f9f500b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500cd8a0 con 0x7f9f5015ac00 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500cd5c0 con 0x7f9f500708b0 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5006ce50 con 0x7f9f5015e7a0 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015ac00 0x7f9f501423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015ac00 0x7f9f501423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45248/0 (socket says 172.21.15.122:45248) 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 -- 172.21.15.122:0/3129406078 learned_addr learned my addr 172.21.15.122:0/3129406078 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f5623f640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f500708b0 0x7f9f5015c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015e7a0 0x7f9f500b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 -- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015e7a0 msgr2=0x7f9f500b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015e7a0 0x7f9f500b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 -- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f500708b0 msgr2=0x7f9f5015c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f500708b0 0x7f9f5015c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 -- 172.21.15.122:0/3129406078 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5014eb80 con 0x7f9f5015ac00 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f9f5015e7a0 0x7f9f500b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f5623f640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9f500708b0 0x7f9f5015c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f55a3e640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015ac00 0x7f9f501423c0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000d7d0 tx=0x7f9f4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40018030 con 0x7f9f5015ac00 2024-03-06T07:34:29.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f40004020 con 0x7f9f5015ac00 2024-03-06T07:34:29.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f501e7270 con 0x7f9f5015ac00 2024-03-06T07:34:29.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40004230 con 0x7f9f5015ac00 2024-03-06T07:34:29.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.580+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f501e7830 con 0x7f9f5015ac00 2024-03-06T07:34:29.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.581+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f5006c910 con 0x7f9f5015ac00 2024-03-06T07:34:29.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f4001b070 con 0x7f9f5015ac00 2024-03-06T07:34:29.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f46ffd640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.589 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f5623f640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9f4008a140 con 0x7f9f5015ac00 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f46ffd640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f9f34071430 0x7f9f34073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f40001d30 con 0x7f9f34071430 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f40091050 con 0x7f9f5015ac00 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f9f34071430 0x7f9f34073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.582+0000 7f9f5623f640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9f3406b4d0 0x7f9f3406d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c006fd0 tx=0x7f9f4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.583+0000 7f9f56a40640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f9f34071430 0x7f9f34073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.584+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== osd.7 v2:172.21.15.169:6800/3079115781 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9f40001d30 con 0x7f9f34071430 2024-03-06T07:34:29.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 --> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9f501e6f20 con 0x7f9f34071430 2024-03-06T07:34:29.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f46ffd640 1 -- 172.21.15.122:0/3129406078 <== osd.7 v2:172.21.15.169:6800/3079115781 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9f501e6f20 con 0x7f9f34071430 2024-03-06T07:34:29.599 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-06T07:34:29.599 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:29.600 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-06T07:34:29.600 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f9f34071430 msgr2=0x7f9f34073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.169:6800/3079115781,v1:172.21.15.169:6801/3079115781] conn(0x7f9f34071430 0x7f9f34073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9f3406b4d0 msgr2=0x7f9f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f9f3406b4d0 0x7f9f3406d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c006fd0 tx=0x7f9f4c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015ac00 msgr2=0x7f9f501423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.597+0000 7f9f584a3640 1 --2- 172.21.15.122:0/3129406078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f5015ac00 0x7f9f501423c0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000d7d0 tx=0x7f9f4000dc90 comp rx=0 tx=0).stop 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.598+0000 7f9f56a40640 1 -- 172.21.15.122:0/3129406078 reap_dead start 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.598+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 shutdown_connections 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.598+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 >> 172.21.15.122:0/3129406078 conn(0x7f9f500b1650 msgr2=0x7f9f501e92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.601 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.598+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 shutdown_connections 2024-03-06T07:34:29.602 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.598+0000 7f9f584a3640 1 -- 172.21.15.122:0/3129406078 wait complete. 2024-03-06T07:34:29.610 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:34:29.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.708+0000 7f2b24f13640 1 Processor -- start 2024-03-06T07:34:29.710 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.708+0000 7f2b24f13640 1 -- start start 2024-03-06T07:34:29.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b24f13640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 0x7f2b2006ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b24f13640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2006d370 con 0x7f2b20070fe0 2024-03-06T07:34:29.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b24f13640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2006d650 con 0x7f2b2015b2d0 2024-03-06T07:34:29.713 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b24f13640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b201448a0 con 0x7f2b201444c0 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b1ed9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2b20070fe0 0x7f2b200cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58364/0 (socket says 172.21.15.122:58364) 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/1029802404 learned_addr learned my addr 172.21.15.122:0/1029802404 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.709+0000 7f2b1e59c640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 0x7f2b2006ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96925331 0 0) 0x7f2b2006d370 con 0x7f2b20070fe0 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08003520 con 0x7f2b20070fe0 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158242422 0 0) 0x7f2b08003520 con 0x7f2b20070fe0 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 msgr2=0x7f2b2006ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 0x7f2b2006ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 >> v1:172.21.15.169:6789/0 conn(0x7f2b2015b2d0 legacy=0x7f2b200708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.714 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2006d930 con 0x7f2b20070fe0 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f2b18001670 con 0x7f2b20070fe0 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 >> v1:172.21.15.122:6789/0 conn(0x7f2b20070fe0 legacy=0x7f2b200cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 0x7f2b080039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b08003f60 0x7f2b08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b08004d40 0x7f2b080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b201448a0 con 0x7f2b08004d40 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0800baa0 con 0x7f2b08003f60 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0800bca0 con 0x7f2b201444c0 2024-03-06T07:34:29.715 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b08004d40 0x7f2b080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1e59c640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 0x7f2b080039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1dd9b640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b08003f60 0x7f2b08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 msgr2=0x7f2b080039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b201444c0 0x7f2b080039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b08003f60 msgr2=0x7f2b08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b08003f60 0x7f2b08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.710+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/1029802404 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08003520 con 0x7f2b08004d40 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.711+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b08004d40 0x7f2b080051c0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800c240 tx=0x7f2b08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a84f6d834c28e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.716 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.711+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18007d10 con 0x7f2b08004d40 2024-03-06T07:34:29.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.711+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18030070 con 0x7f2b08004d40 2024-03-06T07:34:29.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.711+0000 7f2b1d59a640 1 -- 172.21.15.122:0/1029802404 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1802cdc0 con 0x7f2b08004d40 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b24f13640 1 -- 172.21.15.122:0/1029802404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b08004d40 msgr2=0x7f2b080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b24f13640 1 --2- 172.21.15.122:0/1029802404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b08004d40 0x7f2b080051c0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800c240 tx=0x7f2b08005830 comp rx=0 tx=0).stop 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/1029802404 reap_dead start 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b24f13640 1 -- 172.21.15.122:0/1029802404 shutdown_connections 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b24f13640 1 -- 172.21.15.122:0/1029802404 >> 172.21.15.122:0/1029802404 conn(0x7f2b200b1c80 msgr2=0x7f2b200cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b24f13640 1 -- 172.21.15.122:0/1029802404 shutdown_connections 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.712+0000 7f2b24f13640 1 -- 172.21.15.122:0/1029802404 wait complete. 2024-03-06T07:34:29.720 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.713+0000 7f2b24f13640 1 Processor -- start 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.713+0000 7f2b24f13640 1 -- start start 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b24f13640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b08004d40 0x7f2b200aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b24f13640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b200adfc0 0x7f2b200ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b24f13640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 0x7f2b200ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b24f13640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2006ca00 con 0x7f2b2015a3f0 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b24f13640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2006def0 con 0x7f2b200adfc0 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b24f13640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2006cb80 con 0x7f2b08004d40 2024-03-06T07:34:29.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 0x7f2b200ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 0x7f2b200ae600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45266/0 (socket says 172.21.15.122:45266) 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/3111211153 learned_addr learned my addr 172.21.15.122:0/3111211153 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b08004d40 msgr2=0x7f2b200aa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1e59c640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b08004d40 0x7f2b200aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1dd9b640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b200adfc0 0x7f2b200ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b08004d40 0x7f2b200aa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b200adfc0 msgr2=0x7f2b200ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b200adfc0 0x7f2b200ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 -- 172.21.15.122:0/3111211153 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200acc80 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b1ed9d640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 0x7f2b200ae600 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f2b18001970 tx=0x7f2b2006d540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18007d10 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.714+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18000ed0 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.715+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b200acf20 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.715+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180049e0 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.715+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b200ad4e0 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.716+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2004f330 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.716+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b1803f030 con 0x7f2b2015a3f0 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.716+0000 7f2b0effd640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.723 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.716+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2b180aa6b0 con 0x7f2b2015a3f0 2024-03-06T07:34:29.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.716+0000 7f2b1e59c640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.717+0000 7f2b1e59c640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2b0006b4d0 0x7f2b0006d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f2b10007440 tx=0x7f2b1000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.724 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.721+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b18079860 con 0x7f2b2015a3f0 2024-03-06T07:34:29.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.894+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b200748c0 con 0x7f2b0006b4d0 2024-03-06T07:34:29.896 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.894+0000 7f2b0effd640 1 -- 172.21.15.122:0/3111211153 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2b200748c0 con 0x7f2b0006b4d0 2024-03-06T07:34:29.897 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:34:29.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2b0006b4d0 msgr2=0x7f2b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2b0006b4d0 0x7f2b0006d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f2b10007440 tx=0x7f2b1000b040 comp rx=0 tx=0).stop 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 msgr2=0x7f2b200ae600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 0x7f2b200ae600 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f2b18001970 tx=0x7f2b2006d540 comp rx=0 tx=0).stop 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 shutdown_connections 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b2015a3f0 0x7f2b200ae600 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b200adfc0 0x7f2b200ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 --2- 172.21.15.122:0/3111211153 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f2b08004d40 0x7f2b200aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 >> 172.21.15.122:0/3111211153 conn(0x7f2b200b1c80 msgr2=0x7f2b200cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 shutdown_connections 2024-03-06T07:34:29.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:29.896+0000 7f2b24f13640 1 -- 172.21.15.122:0/3111211153 wait complete. 2024-03-06T07:34:29.909 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:30.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.007+0000 7f8f711e2640 1 Processor -- start 2024-03-06T07:34:30.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.007+0000 7f8f711e2640 1 -- start start 2024-03-06T07:34:30.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.007+0000 7f8f711e2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f6c13dc10 0x7f8f6c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.007+0000 7f8f711e2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c15f580 con 0x7f8f6c15bf30 2024-03-06T07:34:30.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f711e2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c15b450 con 0x7f8f6c15ef60 2024-03-06T07:34:30.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f711e2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cf3f0 con 0x7f8f6c13dc10 2024-03-06T07:34:30.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f6ad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8f6c13dc10 0x7f8f6c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f6a59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8f6c15bf30 0x7f8f6c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58368/0 (socket says 172.21.15.122:58368) 2024-03-06T07:34:30.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f6a59c640 1 -- 172.21.15.122:0/2272464441 learned_addr learned my addr 172.21.15.122:0/2272464441 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:30.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/2272464441 >> v1:172.21.15.169:6789/0 conn(0x7f8f6c15ef60 legacy=0x7f8f6c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/2272464441 >> v1:172.21.15.122:6789/0 conn(0x7f8f6c15bf30 legacy=0x7f8f6c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/2272464441 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c15b730 con 0x7f8f6c13dc10 2024-03-06T07:34:30.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.008+0000 7f8f69d9b640 1 -- 172.21.15.122:0/2272464441 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737996999 0 0) 0x7f8f6c15f580 con 0x7f8f6c15bf30 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f6ad9d640 1 --2- 172.21.15.122:0/2272464441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c0b2950 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0097a0 tx=0x7f8f5c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d79efa66b0eba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f69d9b640 1 -- 172.21.15.122:0/2272464441 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0315a0 con 0x7f8f6c13dc10 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f69d9b640 1 -- 172.21.15.122:0/2272464441 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c037070 con 0x7f8f6c13dc10 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f69d9b640 1 -- 172.21.15.122:0/2272464441 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c031960 con 0x7f8f6c13dc10 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 -- 172.21.15.122:0/2272464441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 msgr2=0x7f8f6c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 --2- 172.21.15.122:0/2272464441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c0b2950 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0097a0 tx=0x7f8f5c02b840 comp rx=0 tx=0).stop 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 -- 172.21.15.122:0/2272464441 shutdown_connections 2024-03-06T07:34:30.013 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 --2- 172.21.15.122:0/2272464441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c0b2950 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 -- 172.21.15.122:0/2272464441 >> 172.21.15.122:0/2272464441 conn(0x7f8f6c0cf8a0 msgr2=0x7f8f6c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 -- 172.21.15.122:0/2272464441 shutdown_connections 2024-03-06T07:34:30.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.009+0000 7f8f711e2640 1 -- 172.21.15.122:0/2272464441 wait complete. 2024-03-06T07:34:30.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 Processor -- start 2024-03-06T07:34:30.014 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 -- start start 2024-03-06T07:34:30.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.016 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f6c15bf30 0x7f8f6c13b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f6c15ef60 0x7f8f6c139d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0b2e90 con 0x7f8f6c13dc10 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c15b210 con 0x7f8f6c15ef60 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.011+0000 7f8f711e2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0b3010 con 0x7f8f6c15bf30 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c1547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45278/0 (socket says 172.21.15.122:45278) 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/3355460079 learned_addr learned my addr 172.21.15.122:0/3355460079 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/3355460079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f6c15bf30 msgr2=0x7f8f6c13b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.017 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f6c15bf30 0x7f8f6c13b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/3355460079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f6c15ef60 msgr2=0x7f8f6c139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f6c15ef60 0x7f8f6c139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 -- 172.21.15.122:0/3355460079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c13a800 con 0x7f8f6c13dc10 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f6ad9d640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c1547c0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0097a0 tx=0x7f8f5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c004030 con 0x7f8f6c13dc10 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0041f0 con 0x7f8f6c13dc10 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c13aae0 con 0x7f8f6c13dc10 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c032640 con 0x7f8f6c13dc10 2024-03-06T07:34:30.018 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.012+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c13b0a0 con 0x7f8f6c13dc10 2024-03-06T07:34:30.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.013+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c04f330 con 0x7f8f6c13dc10 2024-03-06T07:34:30.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.014+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f5c0327e0 con 0x7f8f6c13dc10 2024-03-06T07:34:30.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.014+0000 7f8f537fe640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f5806b4d0 0x7f8f5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.014+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8f5c02f030 con 0x7f8f6c13dc10 2024-03-06T07:34:30.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.014+0000 7f8f6a59c640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f5806b4d0 0x7f8f5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.019 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.014+0000 7f8f6a59c640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f5806b4d0 0x7f8f5806d990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8f600011e0 tx=0x7f8f60007940 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.020 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.018+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f5c041070 con 0x7f8f6c13dc10 2024-03-06T07:34:30.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.187+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f6c06eb00 con 0x7f8f5806b4d0 2024-03-06T07:34:30.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.188+0000 7f8f537fe640 1 -- 172.21.15.122:0/3355460079 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13489 (secure 0 0 0) 0x7f8f600194d0 con 0x7f8f5806b4d0 2024-03-06T07:34:30.190 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:30.191 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f5806b4d0 msgr2=0x7f8f5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f5806b4d0 0x7f8f5806d990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8f600011e0 tx=0x7f8f60007940 comp rx=0 tx=0).stop 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 msgr2=0x7f8f6c1547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c1547c0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0097a0 tx=0x7f8f5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 shutdown_connections 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f8f5806b4d0 0x7f8f5806d990 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8f6c15ef60 0x7f8f6c139d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f8f6c15bf30 0x7f8f6c13b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 --2- 172.21.15.122:0/3355460079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f6c13dc10 0x7f8f6c1547c0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 >> 172.21.15.122:0/3355460079 conn(0x7f8f6c0cf8a0 msgr2=0x7f8f6c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 shutdown_connections 2024-03-06T07:34:30.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:30.190+0000 7f8f711e2640 1 -- 172.21.15.122:0/3355460079 wait complete. 2024-03-06T07:34:30.203 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-03-06T07:34:29.472895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331516,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643204,"statfs":{"total":773094113280,"available":772754640896,"internally_reserved":0,"allocated":15867904,"data_stored":7605288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.322751"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":412,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:22.783280+0000","last_change":"2024-03-06T07:21:42.572070+0000","last_active":"2024-03-06T07:34:22.783280+0000","last_peered":"2024-03-06T07:34:22.783280+0000","last_clean":"2024-03-06T07:34:22.783280+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:34:22.783280+0000","last_undegraded":"2024-03-06T07:34:22.783280+0000","last_fullsized":"2024-03-06T07:34:22.783280+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:41.489091+0000","last_clean_scrub_stamp":"2024-03-06T07:21:41.489091+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:10.191854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575004,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1393437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575005,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2428928,"data_stored":1393437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:30.205 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:21:41.489091+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=21, tm_sec=41, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=34, tm_sec=25, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:34:30.206 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:34:50.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:50.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.321+0000 7f584d0ed640 1 Processor -- start 2024-03-06T07:34:50.323 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.321+0000 7f584d0ed640 1 -- start start 2024-03-06T07:34:50.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584d0ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 0x7f58480b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584d0ed640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f584815ef60 0x7f58480b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584d0ed640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584815f580 con 0x7f584815bf30 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584d0ed640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58480cf3f0 con 0x7f584815ef60 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584d0ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584815b210 con 0x7f584813dc10 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f584815bf30 0x7f584815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57226/0 (socket says 172.21.15.122:57226) 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584659c640 1 -- 172.21.15.122:0/2052658702 learned_addr learned my addr 172.21.15.122:0/2052658702 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f584759e640 1 --2- 172.21.15.122:0/2052658702 >> v2:172.21.15.169:3300/0 conn(0x7f584815ef60 0x7f58480b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f5846d9d640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 0x7f58480b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174709696 0 0) 0x7f584815f580 con 0x7f584815bf30 2024-03-06T07:34:50.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.322+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5830003520 con 0x7f584815bf30 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4231527220 0 0) 0x7f5830003520 con 0x7f584815bf30 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 msgr2=0x7f58480b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 0x7f58480b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 >> v2:172.21.15.169:3300/0 conn(0x7f584815ef60 msgr2=0x7f58480b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 --2- 172.21.15.122:0/2052658702 >> v2:172.21.15.169:3300/0 conn(0x7f584815ef60 0x7f58480b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584815b450 con 0x7f584815bf30 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7f583c001670 con 0x7f584815bf30 2024-03-06T07:34:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 >> v1:172.21.15.122:6789/0 conn(0x7f584815bf30 legacy=0x7f584815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 0x7f58300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5830003f30 0x7f5830004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58480cf3f0 con 0x7f5830003f30 2024-03-06T07:34:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584815b210 con 0x7f5830004d40 2024-03-06T07:34:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583000ba10 con 0x7f584813dc10 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584759e640 1 --2- 172.21.15.122:0/2052658702 >> v2:172.21.15.169:3300/0 conn(0x7f584815ef60 0x7f58480b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584659c640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5830003f30 0x7f5830004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f5846d9d640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 0x7f58300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584759e640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584659c640 1 -- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 msgr2=0x7f58300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584659c640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f584813dc10 0x7f58300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584659c640 1 -- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830004d40 msgr2=0x7f58300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584659c640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584659c640 1 -- 172.21.15.122:0/2052658702 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830003520 con 0x7f5830003f30 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.323+0000 7f584759e640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830004d40 0x7f58300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.324+0000 7f584659c640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5830003f30 0x7f5830004800 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c2c0 tx=0x7f5830005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f758bbd9e7db561 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.324+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c007d50 con 0x7f5830003f30 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.324+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c030070 con 0x7f5830003f30 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.324+0000 7f5845d9b640 1 -- 172.21.15.122:0/2052658702 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c02cda0 con 0x7f5830003f30 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584d0ed640 1 -- 172.21.15.122:0/2052658702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5830003f30 msgr2=0x7f5830004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584d0ed640 1 --2- 172.21.15.122:0/2052658702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5830003f30 0x7f5830004800 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c2c0 tx=0x7f5830005830 comp rx=0 tx=0).stop 2024-03-06T07:34:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584759e640 1 -- 172.21.15.122:0/2052658702 reap_dead start 2024-03-06T07:34:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584d0ed640 1 -- 172.21.15.122:0/2052658702 shutdown_connections 2024-03-06T07:34:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584d0ed640 1 -- 172.21.15.122:0/2052658702 >> 172.21.15.122:0/2052658702 conn(0x7f58480cf8a0 msgr2=0x7f58480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584d0ed640 1 -- 172.21.15.122:0/2052658702 shutdown_connections 2024-03-06T07:34:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.325+0000 7f584d0ed640 1 -- 172.21.15.122:0/2052658702 wait complete. 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.326+0000 7f584d0ed640 1 Processor -- start 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 -- start start 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830003f30 0x7f58481542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 0x7f58481e7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58481e7a70 0x7f584812cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58480b2950 con 0x7f584812ecd0 2024-03-06T07:34:50.336 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584813dff0 con 0x7f5830003f30 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584d0ed640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58480b2ad0 con 0x7f58481e7a70 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 0x7f58481e7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 0x7f58481e7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35360/0 (socket says 172.21.15.122:35360) 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.327+0000 7f584659c640 1 -- 172.21.15.122:0/3466753150 learned_addr learned my addr 172.21.15.122:0/3466753150 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f5846d9d640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830003f30 0x7f58481542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584759e640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58481e7a70 0x7f584812cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.337 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584659c640 1 -- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58481e7a70 msgr2=0x7f584812cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584659c640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58481e7a70 0x7f584812cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584659c640 1 -- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830003f30 msgr2=0x7f58481542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584659c640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830003f30 0x7f58481542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584659c640 1 -- 172.21.15.122:0/3466753150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584812dc30 con 0x7f584812ecd0 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584659c640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 0x7f58481e7530 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c2c0 tx=0x7f583c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c001c80 con 0x7f584812ecd0 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c004bf0 con 0x7f584812ecd0 2024-03-06T07:34:50.338 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584812df10 con 0x7f584812ecd0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c007070 con 0x7f584812ecd0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.328+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584812e4d0 con 0x7f584812ecd0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.329+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584804f330 con 0x7f584812ecd0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.330+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f583c048020 con 0x7f584812ecd0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.330+0000 7f582f7fe640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f583406b4d0 0x7f583406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.330+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f583c0ab190 con 0x7f584812ecd0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.330+0000 7f5846d9d640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f583406b4d0 0x7f583406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.339 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.331+0000 7f5846d9d640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f583406b4d0 0x7f583406d990 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5838009ae0 tx=0x7f5838009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.335+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f583c07a2b0 con 0x7f584812ecd0 2024-03-06T07:34:50.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.504+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 --> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f584813d5d0 con 0x7f583406b4d0 2024-03-06T07:34:50.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.505+0000 7f582f7fe640 1 -- 172.21.15.122:0/3466753150 <== mgr.4104 v2:172.21.15.122:6832/602544710 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7f584813d5d0 con 0x7f583406b4d0 2024-03-06T07:34:50.507 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:50.507 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-03-06T07:34:50.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f583406b4d0 msgr2=0x7f583406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f583406b4d0 0x7f583406d990 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5838009ae0 tx=0x7f5838009190 comp rx=0 tx=0).stop 2024-03-06T07:34:50.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 msgr2=0x7f58481e7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 0x7f58481e7530 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c2c0 tx=0x7f583c001a40 comp rx=0 tx=0).stop 2024-03-06T07:34:50.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 shutdown_connections 2024-03-06T07:34:50.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f583406b4d0 0x7f583406d990 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f58481e7a70 0x7f584812cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f584812ecd0 0x7f58481e7530 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 --2- 172.21.15.122:0/3466753150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830003f30 0x7f58481542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 >> 172.21.15.122:0/3466753150 conn(0x7f58480cf8a0 msgr2=0x7f58480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 shutdown_connections 2024-03-06T07:34:50.511 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.507+0000 7f584d0ed640 1 -- 172.21.15.122:0/3466753150 wait complete. 2024-03-06T07:34:50.519 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2024-03-06T07:34:49.476760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331504,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643216,"statfs":{"total":773094113280,"available":772754653184,"internally_reserved":0,"allocated":15867904,"data_stored":7605288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":323472944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002467"},"pg_stats":[{"pgid":"1.0","version":"146'119","reported_seq":423,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:30.480089+0000","last_change":"2024-03-06T07:34:30.480126+0000","last_active":"2024-03-06T07:34:30.480089+0000","last_peered":"2024-03-06T07:34:30.480089+0000","last_clean":"2024-03-06T07:34:30.480089+0000","last_became_active":"2024-03-06T07:21:42.571059+0000","last_became_peered":"2024-03-06T07:21:42.571059+0000","last_unstale":"2024-03-06T07:34:30.480089+0000","last_undegraded":"2024-03-06T07:34:30.480089+0000","last_fullsized":"2024-03-06T07:34:30.480089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'119","last_scrub_stamp":"2024-03-06T07:34:30.480005+0000","last_deep_scrub":"146'119","last_deep_scrub_stamp":"2024-03-06T07:34:30.480005+0000","last_clean_scrub_stamp":"2024-03-06T07:34:30.480005+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:31.340966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2428928,"data_stored":1393437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":803069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2428928,"data_stored":1393437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:50.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:34:50.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.618+0000 7fba99ba2640 1 Processor -- start 2024-03-06T07:34:50.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.618+0000 7fba99ba2640 1 -- start start 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba99ba2640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7fba940cfb80 0x7fba94161910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba99ba2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba940d0020 con 0x7fba94160720 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba99ba2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba940d0300 con 0x7fba940b0d30 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba99ba2640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba940b1110 con 0x7fba940cfb80 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba993a1640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fba94160720 0x7fba9415d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57238/0 (socket says 172.21.15.122:57238) 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba993a1640 1 -- 172.21.15.122:0/3034254867 learned_addr learned my addr 172.21.15.122:0/3034254867 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba93fff640 1 --2- 172.21.15.122:0/3034254867 >> v2:172.21.15.169:3300/0 conn(0x7fba940cfb80 0x7fba94161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225077994 0 0) 0x7fba940d0020 con 0x7fba94160720 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.619+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c003520 con 0x7fba94160720 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3620969375 0 0) 0x7fba7c003520 con 0x7fba94160720 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 >> v2:172.21.15.169:3300/0 conn(0x7fba940cfb80 msgr2=0x7fba94161910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 --2- 172.21.15.122:0/3034254867 >> v2:172.21.15.169:3300/0 conn(0x7fba940cfb80 0x7fba94161910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 >> v1:172.21.15.169:6789/0 conn(0x7fba940b0d30 legacy=0x7fba9406f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba940d05e0 con 0x7fba94160720 2024-03-06T07:34:50.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2468749825 0 0) 0x7fba8c0019f0 con 0x7fba94160720 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 >> v1:172.21.15.122:6789/0 conn(0x7fba94160720 legacy=0x7fba9415d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0039b0 0x7fba7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba7c004800 0x7fba7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7c0051c0 0x7fba7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba940b1110 con 0x7fba7c0051c0 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c00c060 con 0x7fba7c004800 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c00c260 con 0x7fba7c0039b0 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba993a1640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7c0051c0 0x7fba7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba98ba0640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0039b0 0x7fba7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba93fff640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba7c004800 0x7fba7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba993a1640 1 -- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0039b0 msgr2=0x7fba7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba993a1640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0039b0 0x7fba7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba993a1640 1 -- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba7c004800 msgr2=0x7fba7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba993a1640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba7c004800 0x7fba7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba993a1640 1 -- 172.21.15.122:0/3034254867 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c003520 con 0x7fba7c0051c0 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.620+0000 7fba93fff640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba7c004800 0x7fba7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba993a1640 1 -- 172.21.15.122:0/3034254867 reap_dead start 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba993a1640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7c0051c0 0x7fba7c00b9c0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fba94160bb0 tx=0x7fba7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca28f9c267aa1994 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c00a4a0 con 0x7fba7c0051c0 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c034650 con 0x7fba7c0051c0 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba937fe640 1 -- 172.21.15.122:0/3034254867 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c034a40 con 0x7fba7c0051c0 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba99ba2640 1 -- 172.21.15.122:0/3034254867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7c0051c0 msgr2=0x7fba7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.621+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7c0051c0 0x7fba7c00b9c0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fba94160bb0 tx=0x7fba7c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.622+0000 7fba99ba2640 1 -- 172.21.15.122:0/3034254867 shutdown_connections 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.622+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3034254867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba7c0051c0 0x7fba7c00b9c0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.622+0000 7fba99ba2640 1 -- 172.21.15.122:0/3034254867 >> 172.21.15.122:0/3034254867 conn(0x7fba940d1850 msgr2=0x7fba940d1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.622+0000 7fba99ba2640 1 -- 172.21.15.122:0/3034254867 shutdown_connections 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.622+0000 7fba99ba2640 1 -- 172.21.15.122:0/3034254867 wait complete. 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.623+0000 7fba99ba2640 1 Processor -- start 2024-03-06T07:34:50.630 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.623+0000 7fba99ba2640 1 -- start start 2024-03-06T07:34:50.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0051c0 0x7fba941f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba940b6590 0x7fba940b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 0x7fba9406d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.633 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9406f610 con 0x7fba9406d3b0 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba940d0c30 con 0x7fba940b6590 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9406f790 con 0x7fba7c0051c0 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 0x7fba9406d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 0x7fba9406d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35364/0 (socket says 172.21.15.122:35364) 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 -- 172.21.15.122:0/3076665349 learned_addr learned my addr 172.21.15.122:0/3076665349 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba98ba0640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0051c0 0x7fba941f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 -- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0051c0 msgr2=0x7fba941f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.634 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba93fff640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba940b6590 0x7fba940b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0051c0 0x7fba941f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 -- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba940b6590 msgr2=0x7fba940b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba940b6590 0x7fba940b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 -- 172.21.15.122:0/3076665349 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba940b4740 con 0x7fba9406d3b0 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba98ba0640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0051c0 0x7fba941f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba93fff640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba940b6590 0x7fba940b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba993a1640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 0x7fba9406d830 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fba8c02cc40 tx=0x7fba8c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c044070 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c0017b0 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba940b49e0 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c039040 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.624+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba940b4fa0 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.625+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba94068f10 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.626+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba8c047070 con 0x7fba9406d3b0 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.626+0000 7fba917fa640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba8006b4d0 0x7fba8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.626+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fba8c0ab5e0 con 0x7fba9406d3b0 2024-03-06T07:34:50.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.626+0000 7fba98ba0640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba8006b4d0 0x7fba8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.627+0000 7fba98ba0640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba8006b4d0 0x7fba8006d990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fba84000c00 tx=0x7fba84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.630+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba8c07a0b0 con 0x7fba9406d3b0 2024-03-06T07:34:50.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.800+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fba940cf550 con 0x7fba9406d3b0 2024-03-06T07:34:50.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.804+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v36)=0 v36) v1 ==== 123+0+0 (secure 0 0 0) 0x7fba8c07e150 con 0x7fba9406d3b0 2024-03-06T07:34:50.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.804+0000 7fba917fa640 1 -- 172.21.15.122:0/3076665349 <== mon.0 v2:172.21.15.122:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fba8c082710 con 0x7fba9406d3b0 2024-03-06T07:34:50.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba8006b4d0 msgr2=0x7fba8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba8006b4d0 0x7fba8006d990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fba84000c00 tx=0x7fba84009040 comp rx=0 tx=0).stop 2024-03-06T07:34:50.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 msgr2=0x7fba9406d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 0x7fba9406d830 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fba8c02cc40 tx=0x7fba8c001dc0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 shutdown_connections 2024-03-06T07:34:50.809 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7fba8006b4d0 0x7fba8006d990 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba9406d3b0 0x7fba9406d830 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba940b6590 0x7fba940b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 --2- 172.21.15.122:0/3076665349 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7fba7c0051c0 0x7fba941f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 >> 172.21.15.122:0/3076665349 conn(0x7fba940d1850 msgr2=0x7fba940afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 shutdown_connections 2024-03-06T07:34:50.810 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.806+0000 7fba99ba2640 1 -- 172.21.15.122:0/3076665349 wait complete. 2024-03-06T07:34:50.820 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:50.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:50.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.917+0000 7f046ba1c640 1 Processor -- start 2024-03-06T07:34:50.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.917+0000 7f046ba1c640 1 -- start start 2024-03-06T07:34:50.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f046ba1c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f046413dc10 0x7f04640b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f046ba1c640 1 --2- >> v2:172.21.15.169:3300/0 conn(0x7f046415ef60 0x7f04640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f046ba1c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046415f580 con 0x7f046415bf30 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f046ba1c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640cf3f0 con 0x7f046413dc10 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f046ba1c640 1 -- --> v2:172.21.15.169:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046415b210 con 0x7f046415ef60 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f04697b8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f046413dc10 0x7f04640b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f0468fb7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f046415bf30 0x7f046415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57248/0 (socket says 172.21.15.122:57248) 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f0468fb7640 1 -- 172.21.15.122:0/2148773115 learned_addr learned my addr 172.21.15.122:0/2148773115 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f0469fb9640 1 --2- 172.21.15.122:0/2148773115 >> v2:172.21.15.169:3300/0 conn(0x7f046415ef60 0x7f04640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f04697b8640 1 -- 172.21.15.122:0/2148773115 >> v2:172.21.15.169:3300/0 conn(0x7f046415ef60 msgr2=0x7f04640b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f04697b8640 1 --2- 172.21.15.122:0/2148773115 >> v2:172.21.15.169:3300/0 conn(0x7f046415ef60 0x7f04640b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f04697b8640 1 -- 172.21.15.122:0/2148773115 >> v1:172.21.15.122:6789/0 conn(0x7f046415bf30 legacy=0x7f046415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f045bfff640 1 -- 172.21.15.122:0/2148773115 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203866904 0 0) 0x7f046415f580 con 0x7f046415bf30 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f04697b8640 1 -- 172.21.15.122:0/2148773115 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046415b450 con 0x7f046413dc10 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.918+0000 7f0469fb9640 1 --2- 172.21.15.122:0/2148773115 >> v2:172.21.15.169:3300/0 conn(0x7f046415ef60 0x7f04640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.919+0000 7f04697b8640 1 --2- 172.21.15.122:0/2148773115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046413dc10 0x7f04640b2950 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f0464068700 tx=0x7f046002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1449968d05500238 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.919+0000 7f045bfff640 1 -- 172.21.15.122:0/2148773115 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460039070 con 0x7f046413dc10 2024-03-06T07:34:50.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.919+0000 7f045bfff640 1 -- 172.21.15.122:0/2148773115 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f046002cbf0 con 0x7f046413dc10 2024-03-06T07:34:50.924 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.919+0000 7f045bfff640 1 -- 172.21.15.122:0/2148773115 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460034440 con 0x7f046413dc10 2024-03-06T07:34:50.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 -- 172.21.15.122:0/2148773115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046413dc10 msgr2=0x7f04640b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2148773115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046413dc10 0x7f04640b2950 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f0464068700 tx=0x7f046002be40 comp rx=0 tx=0).stop 2024-03-06T07:34:50.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 -- 172.21.15.122:0/2148773115 shutdown_connections 2024-03-06T07:34:50.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2148773115 >> v2:172.21.15.169:3300/0 conn(0x7f046415ef60 0x7f04640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.927 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2148773115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046413dc10 0x7f04640b2950 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 -- 172.21.15.122:0/2148773115 >> 172.21.15.122:0/2148773115 conn(0x7f04640cf8a0 msgr2=0x7f04640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 -- 172.21.15.122:0/2148773115 shutdown_connections 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.920+0000 7f046ba1c640 1 -- 172.21.15.122:0/2148773115 wait complete. 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.921+0000 7f046ba1c640 1 Processor -- start 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.921+0000 7f046ba1c640 1 -- start start 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f046ba1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046413dc10 0x7f0464137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f046ba1c640 1 --2- >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f046415bf30 0x7f046413c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f046ba1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 0x7f046413c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f046ba1c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640708d0 con 0x7f046415ef60 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f046ba1c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046413dff0 con 0x7f046413dc10 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f046ba1c640 1 -- --> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0464070a50 con 0x7f046415bf30 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 0x7f046413c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 0x7f046413c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:35384/0 (socket says 172.21.15.122:35384) 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 -- 172.21.15.122:0/2947184099 learned_addr learned my addr 172.21.15.122:0/2947184099 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f04697b8640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046413dc10 0x7f0464137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0468fb7640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f046415bf30 0x7f046413c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 -- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f046415bf30 msgr2=0x7f046413c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f046415bf30 0x7f046413c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 -- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046413dc10 msgr2=0x7f0464137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046413dc10 0x7f0464137320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 -- 172.21.15.122:0/2947184099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464137860 con 0x7f046415ef60 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0469fb9640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 0x7f046413c600 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f045c00c7c0 tx=0x7f045c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.922+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c015020 con 0x7f046415ef60 2024-03-06T07:34:50.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.923+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f045c004230 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.923+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c007620 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.923+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046413e3c0 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.923+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046413e980 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.924+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046404f330 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.924+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f045c018020 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.925+0000 7f0459ffb640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044806b4d0 0x7f044806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.925+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f045c08a090 con 0x7f046415ef60 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.925+0000 7f04697b8640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044806b4d0 0x7f044806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:50.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.925+0000 7f04697b8640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044806b4d0 0x7f044806d990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0460036000 tx=0x7f0460030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:50.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:50.929+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f045c01f020 con 0x7f046415ef60 2024-03-06T07:34:51.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.135+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f046406eb00 con 0x7f046415ef60 2024-03-06T07:34:51.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.135+0000 7f0459ffb640 1 -- 172.21.15.122:0/2947184099 <== mon.0 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+1332 (secure 0 0 0) 0x7f045c059270 con 0x7f046415ef60 2024-03-06T07:34:51.138 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-06T07:34:51.138 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":103,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3229859338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3229859338},{"type":"v1","addr":"172.21.15.122:6837","nonce":3229859338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[]} 2024-03-06T07:34:51.138 INFO:teuthology.orchestra.run.smithi122.stderr:dumped fsmap epoch 103 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044806b4d0 msgr2=0x7f044806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044806b4d0 0x7f044806d990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0460036000 tx=0x7f0460030000 comp rx=0 tx=0).stop 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 msgr2=0x7f046413c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 0x7f046413c600 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f045c00c7c0 tx=0x7f045c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 shutdown_connections 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:6832/602544710,v1:172.21.15.122:6833/602544710] conn(0x7f044806b4d0 0x7f044806d990 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.140 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f046415ef60 0x7f046413c600 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] conn(0x7f046415bf30 0x7f046413c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 --2- 172.21.15.122:0/2947184099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f046413dc10 0x7f0464137320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.137+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 >> 172.21.15.122:0/2947184099 conn(0x7f04640cf8a0 msgr2=0x7f04640b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:51.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.138+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 shutdown_connections 2024-03-06T07:34:51.141 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-06T07:34:51.138+0000 7f046ba1c640 1 -- 172.21.15.122:0/2947184099 wait complete. 2024-03-06T07:34:51.150 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:34:51.150 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:34:51.151 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:51.151 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.159 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:51.159 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:51.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.210 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:51.210 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-03-06T07:34:51.213 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-03-06T07:34:51.216 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:34:51.216 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:34:51.217 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.286 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:34:51.286 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:34:51.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.351 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:34:51.351 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:34:51.351 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.426 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:34:51.427 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:34:51.427 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.500 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:34:51.500 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.531 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:34:51.531 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:34:51.531 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.609 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:34:51.609 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:34:51.609 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.676 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:34:51.676 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:34:51.677 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.743 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:34:51.743 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:34:51.744 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:34:51.744 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.770 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:34:51.771 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:34:51.771 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.811 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:34:51.811 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:34:51.811 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:34:51.812 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.823 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:34:51.823 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:34:51.823 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.863 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:34:51.863 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:34:51.864 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:51.915 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:34:51.916 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:34:51.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:34:51.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:34:51.947 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:34:52.241 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:34:52.241 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:34:52.315 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:34:52.315 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:34:52.372 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:34:52.373 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:34:52.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:34:52.424 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:34:52.585 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:34:52.585 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:34:52.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:34:52.664 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:34:52.712 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:34:52.713 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:34:52.764 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:34:52.765 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/data/mon.a.tgz 2024-03-06T07:34:52.765 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-06T07:34:52.780 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.P1dqiUln2Q 2024-03-06T07:34:52.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.P1dqiUln2Q 2024-03-06T07:34:52.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.P1dqiUln2Q 2024-03-06T07:34:53.008 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.P1dqiUln2Q is 380KB 2024-03-06T07:34:53.106 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.P1dqiUln2Q 2024-03-06T07:34:53.120 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/data/mon.b.tgz 2024-03-06T07:34:53.120 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-06T07:34:53.135 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.DIsn4dh3oo 2024-03-06T07:34:53.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DIsn4dh3oo 2024-03-06T07:34:53.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.DIsn4dh3oo 2024-03-06T07:34:53.383 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.DIsn4dh3oo is 382KB 2024-03-06T07:34:53.478 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.DIsn4dh3oo 2024-03-06T07:34:53.492 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/data/mon.c.tgz 2024-03-06T07:34:53.492 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-06T07:34:53.546 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.zjRJ089qT9 2024-03-06T07:34:53.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zjRJ089qT9 2024-03-06T07:34:53.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.zjRJ089qT9 2024-03-06T07:34:53.763 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.zjRJ089qT9 is 382KB 2024-03-06T07:34:53.860 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.zjRJ089qT9 2024-03-06T07:34:53.875 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:34:53.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:34:53.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:34:53.960 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:34:53.997 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:34:53.997 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:34:53.999 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/remote/smithi122/crash 2024-03-06T07:34:53.999 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:34:54.028 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/remote/smithi169/crash 2024-03-06T07:34:54.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:34:54.062 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:34:54.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:54.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:54.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:54.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:54.210 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:34:54.210 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:34:54.226 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:34:54.305 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:34:54.305 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38010.log 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.38250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44359.log 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38010.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gz 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.1.44359.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38010.log.gz 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44825.log 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose 89.9% -- /var/log/ceph/ceph-client.admin.38130.log -- replaced with /var/log/ceph/ceph-client.1.44359.log.gz 2024-03-06T07:34:54.306 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.3.44825.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38130.log.gz 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.3.44825.log.gz 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38170.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.7.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38290.log 2024-03-06T07:34:54.307 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38210.log 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38290.log.gz 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38210.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38210.log.gz 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:34:54.308 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38050.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-03-06T07:34:54.309 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38090.log 2024-03-06T07:34:54.309 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mon.b.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:34:54.309 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:34:54.309 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.38090.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38090.log.gz 2024-03-06T07:34:54.309 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:34:54.309 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mds.d.log: 94.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:34:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mon.c.log: 91.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:34:54.590 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57191.log 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81102.log 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49342.log 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57191.log: /var/log/ceph/ceph-client.admin.81102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67721.log 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57191.log.gz 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56450.log 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81102.log.gz 2024-03-06T07:34:54.591 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67721.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49342.log.gz 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76922.log 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67721.log.gz 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47821.log 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78354.log 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56450.log.gz 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47821.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.76922.log.gz 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr: /var/log/ceph/ceph-client.admin.55669.log 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3%/var/log/ceph/ceph-client.admin.78354.log: -- replaced with /var/log/ceph/ceph-client.admin.47821.log.gz 2024-03-06T07:34:54.592 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70397.log 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78354.log.gz 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82473.log 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74053.log 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55669.log.gz 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70397.log.gz 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80602.log 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82473.log.gz 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53501.log 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49012.log 2024-03-06T07:34:54.593 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74053.log.gz 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53501.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80602.log.gz 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45338.log 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49012.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53501.log.gz 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82006.log 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45338.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49012.log.gz 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51562.log 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82006.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45338.log.gz 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49552.log 2024-03-06T07:34:54.594 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51562.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82006.log.gz 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87723.log 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51562.log.gz 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49552.log.gz 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62387.log 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51392.log 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50434.log 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87723.log.gz 2024-03-06T07:34:54.595 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51392.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62387.log.gz 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51392.log.gz 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70439.log 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80038.log 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50434.log.gz 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70008.log 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78588.log 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70439.log.gz 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8%/var/log/ceph/ceph-client.admin.70008.log: -- replaced with /var/log/ceph/ceph-client.admin.80038.log.gz 2024-03-06T07:34:54.596 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80206.log 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78588.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70008.log.gz 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88497.log 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78588.log.gz 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68433.log 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7%/var/log/ceph/ceph-client.admin.88497.log: -- replaced with /var/log/ceph/ceph-client.admin.80206.log.gz 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89086.log 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68433.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.88497.log.gz --verbose 2024-03-06T07:34:54.597 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.54087.log 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89086.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68433.log.gz 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57107.log 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89086.log.gz 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88624.log 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82515.log 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54087.log.gz 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88624.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57107.log.gz 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54568.log 2024-03-06T07:34:54.598 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88624.log.gz 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54400.log 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54568.log: gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.82515.log.gz --verbose 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.66893.log 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54400.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54568.log.gz 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81507.log 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54400.log.gz 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66389.log 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66893.log.gz 2024-03-06T07:34:54.599 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64990.log 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66389.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81507.log.gz 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56981.log 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64990.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66389.log.gz 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47779.log 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64990.log.gz 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73686.log 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47779.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56981.log.gz 2024-03-06T07:34:54.600 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58208.log 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73686.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47779.log.gz 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76475.log 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73686.log.gz 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58038.log 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76475.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58208.log.gz -5 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81186.log 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5%/var/log/ceph/ceph-client.admin.58038.log: -- replaced with /var/log/ceph/ceph-client.admin.76475.log.gz 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50770.log 2024-03-06T07:34:54.601 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81186.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58038.log.gz 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48786.log 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81186.log.gz 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88353.log 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74052.log 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50770.log.gz 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88353.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48786.log.gz 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51929.log 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88353.log.gz 2024-03-06T07:34:54.602 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46252.log 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61574.log 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74052.log.gz 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46252.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51929.log.gz 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67584.log 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46252.log.gz 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64168.log 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66935.log 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61574.log.gz 2024-03-06T07:34:54.603 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67584.log.gz 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58292.log 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64168.log.gz 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82739.log 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66935.log: /var/log/ceph/ceph-client.admin.58292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44740.log 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66935.log.gz 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82739.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58292.log.gz 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85909.log 2024-03-06T07:34:54.604 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82739.log.gz 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63553.log 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85909.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.54869.log -- replaced with /var/log/ceph/ceph-client.admin.44740.log.gz 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63553.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85909.log.gz 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85867.log 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63553.log.gz 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43920.log 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81144.log 2024-03-06T07:34:54.605 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54869.log.gz 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85867.log.gz 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93043.log 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47669.log 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81144.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43920.log.gz 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.93043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55119.log 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81144.log.gz 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47669.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93043.log.gz 2024-03-06T07:34:54.606 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45330.log 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55119.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47669.log.gz 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82911.log 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45330.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55119.log.gz 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52353.log 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45330.log.gz 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88455.log 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52353.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82911.log.gz 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52353.log.gz 2024-03-06T07:34:54.607 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88750.log 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89230.log 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88750.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.88455.log.gz 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61868.log 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88750.log.gz 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56319.log 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89230.log: /var/log/ceph/ceph-client.admin.61868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70778.log 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89230.log.gz 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56319.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61868.log.gz 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70694.log 2024-03-06T07:34:54.608 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56319.log.gz 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60541.log 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70778.log: /var/log/ceph/ceph-client.admin.70694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60499.log 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70778.log.gz 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60541.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70694.log.gz 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60937.log 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60541.log.gz 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69946.log 2024-03-06T07:34:54.609 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90760.log 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60499.log.gz 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69946.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60937.log.gz 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50350.log 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69946.log.gz 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49710.log 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87537.log 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90760.log.gz 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49710.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50350.log.gz 2024-03-06T07:34:54.610 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55753.log 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49710.log.gz 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84397.log 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87537.log.gz 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84397.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55753.log.gz 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53682.log 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84397.log.gz 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90046.log 2024-03-06T07:34:54.611 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76754.log 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90046.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53682.log.gz 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47011.log 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90046.log.gz 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5/var/log/ceph/ceph-client.admin.76754.log: --verbose -- /var/log/ceph/ceph-client.admin.66614.log 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.47011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86637.log 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76754.log.gz 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88792.log 2024-03-06T07:34:54.612 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66614.log.gz 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87933.log 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.47011.log.gz 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86637.log.gz 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.88792.log: -- /var/log/ceph/ceph-client.admin.81837.log 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87933.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88792.log.gz 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44088.log 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87933.log.gz 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78847.log 2024-03-06T07:34:54.613 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89440.log 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81837.log.gz 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.78847.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44088.log.gz 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70256.log 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56576.log 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70256.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.89440.log.gz 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80164.log 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51148.log 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56576.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70256.log.gz 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80164.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56576.log.gz -5 2024-03-06T07:34:54.614 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.92833.log 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose 89.0% -- -- replaced with /var/log/ceph/ceph-client.admin.80164.log.gz /var/log/ceph/ceph-client.admin.92102.log 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51148.log: /var/log/ceph/ceph-client.admin.92833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69157.log 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51148.log.gz 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92833.log.gz 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69286.log 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81675.log 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69157.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92102.log.gz 2024-03-06T07:34:54.615 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44046.log 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69157.log.gz 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69286.log.gz 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49054.log 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67883.log 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81675.log.gz 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49054.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.44046.log.gz --verbose 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.92341.log 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49054.log.gz 2024-03-06T07:34:54.616 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92875.log 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92341.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67883.log.gz 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79100.log 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92341.log.gz 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.78847.log.gz 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53961.log 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92875.log: /var/log/ceph/ceph-client.admin.79100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57485.log 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92875.log.gz 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53961.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79100.log.gz 2024-03-06T07:34:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53039.log 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57485.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53961.log.gz 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49180.log 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53039.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.90340.log 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57485.log.gz 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49180.log: 89.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53039.log.gz -5 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55795.log 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90340.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49180.log.gz 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63832.log 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90340.log.gz 2024-03-06T07:34:54.618 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75775.log 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91831.log 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55795.log.gz 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75775.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63832.log.gz 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92018.log 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75775.log.gz 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69569.log 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60347.log 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91831.log.gz 2024-03-06T07:34:54.619 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.92018.log.gz 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75483.log 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69569.log.gz 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80560.log 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54442.log 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60347.log.gz 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80560.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.75483.log.gz 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57664.log 2024-03-06T07:34:54.620 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80560.log.gz 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82049.log 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81633.log 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54442.log.gz 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82049.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57664.log.gz 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72458.log 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82049.log.gz 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44004.log 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55291.log 2024-03-06T07:34:54.621 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81633.log.gz 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44004.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72458.log.gz 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57443.log 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44004.log.gz 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89272.log 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57233.log 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55291.log.gz 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57443.log.gz 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71470.log 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89272.log.gz 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61698.log 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85173.log 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57233.log.gz 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71470.log.gz 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72332.log 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61698.log.gz 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57994.log 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65571.log 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85173.log.gz 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57994.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.72332.log.gz 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69227.log 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57994.log.gz 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84318.log 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90382.log 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65571.log.gz 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84318.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69227.log.gz 2024-03-06T07:34:54.624 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44278.log 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84318.log.gz 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80854.log 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63255.log 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90382.log.gz 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80854.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44278.log.gz 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71124.log 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80854.log.gz 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57317.log 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64355.log 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63255.log.gz 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57317.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71124.log.gz 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51064.log 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57317.log.gz 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91030.log/var/log/ceph/ceph-client.admin.64355.log: 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69884.log 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64355.log.gz 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51064.log.gz 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87411.log 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69884.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91030.log.gz 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45669.log 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69884.log.gz/var/log/ceph/ceph-client.admin.87411.log: 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90718.log 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45669.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87411.log.gz 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49836.log 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45669.log.gz 2024-03-06T07:34:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72248.log 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65255.log 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90718.log.gz 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72248.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49836.log.gz 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85665.log 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72248.log.gz 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61494.log 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51818.log 2024-03-06T07:34:54.628 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65255.log.gz 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61494.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85665.log.gz 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79333.log 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61494.log.gz 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44996.log 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57527.log 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51818.log.gz 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44996.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79333.log.gz 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62135.log 2024-03-06T07:34:54.629 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44996.log.gz 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79256.log 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66572.log 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57527.log.gz 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79256.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62135.log.gz 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79055.log 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79256.log.gz 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71072.log 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46084.log 2024-03-06T07:34:54.630 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66572.log.gz 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71072.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79055.log.gz 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89482.log 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71072.log.gz 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50938.log 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58502.log 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46084.log.gz 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50938.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89482.log.gz 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60835.log 2024-03-06T07:34:54.631 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50938.log.gz 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56618.log 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63874.log 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58502.log.gz 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56618.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60835.log.gz 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89608.log 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56618.log.gz 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61656.log 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89608.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.63874.log.gz 2024-03-06T07:34:54.632 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75316.log 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61656.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89608.log.gz 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92749.log 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61656.log.gz 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84194.log 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64607.log 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75316.log.gz 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84194.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92749.log.gz 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77955.log 2024-03-06T07:34:54.633 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84194.log.gz 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79143.log 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51646.log 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64607.log.gz 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79143.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77955.log.gz 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50392.log 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79143.log.gz 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93212.log 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47863.log 88.2% 2024-03-06T07:34:54.634 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51646.log.gz 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50392.log.gz 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.93212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88582.log 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93212.log.gz 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79004.log 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87201.log 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47863.log.gz 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79004.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88582.log.gz 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49426.log 2024-03-06T07:34:54.635 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79004.log.gz 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55711.log 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49426.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.87201.log.gz 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.82781.log 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55711.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49426.log.gz 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53640.log 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55711.log.gz 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59178.log 2024-03-06T07:34:54.636 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53640.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82781.log.gz 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52804.log 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59178.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53640.log.gz 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86991.log 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59178.log.gz 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.52804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93254.log 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67626.log 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.93254.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86991.log.gz 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72626.log 2024-03-06T07:34:54.637 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93254.log.gz 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65739.log 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64000.log 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72626.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67626.log.gz 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65739.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72626.log.gzgzip 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70090.log 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69527.log 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65739.log.gz 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64000.log: /var/log/ceph/ceph-client.admin.70090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77235.log 2024-03-06T07:34:54.638 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.64000.log.gz 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84583.log 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70090.log.gz 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69527.log: /var/log/ceph/ceph-client.admin.77235.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69527.log.gz 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44656.log 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77235.log.gz 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75004.log 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84583.log: /var/log/ceph/ceph-client.admin.44656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54129.log 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84583.log.gz 2024-03-06T07:34:54.639 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44656.log.gz 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64824.log 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48157.log/var/log/ceph/ceph-client.admin.54129.log: 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81591.log 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54129.log.gz 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48157.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64824.log.gz 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67061.log 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75004.log.gz 2024-03-06T07:34:54.640 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48157.log.gz 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78396.log 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49978.log 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81591.log.gz 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78396.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67061.log.gz 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74219.log 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49978.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78396.log.gz 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62513.log 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49978.log.gz 2024-03-06T07:34:54.641 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92383.log 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74219.log: /var/log/ceph/ceph-client.admin.62513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82133.log 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74219.log.gz 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92383.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62513.log.gz 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72668.log 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87681.log 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82133.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92383.log.gz 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68475.log 88.7% 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82133.log.gz 2024-03-06T07:34:54.642 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72668.log.gz 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88413.log 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87681.log: /var/log/ceph/ceph-client.admin.68475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73442.log 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.87681.log.gz 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68475.log.gz 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71428.log 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88413.log.gz 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72118.log 2024-03-06T07:34:54.643 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73442.log: /var/log/ceph/ceph-client.admin.71428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57910.log 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73442.log.gz 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71428.log.gz 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48954.log 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72118.log.gz 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50644.log 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57910.log.gz 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87033.log 2024-03-06T07:34:54.644 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50644.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48954.log.gz 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62880.log 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50644.log.gz 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90214.log 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.87033.log.gz 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87075.log 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.62880.log: /var/log/ceph/ceph-client.admin.90214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61189.log 2024-03-06T07:34:54.645 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87075.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90214.log.gz 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91453.log 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61189.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87075.log.gz 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55585.log 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61189.log.gz 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81964.log 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73854.log 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55585.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91453.log.gz 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90988.log 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55585.log.gz 2024-03-06T07:34:54.646 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.73854.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53724.log 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81964.log.gz 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90988.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73854.log.gz 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48478.log 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61147.log 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53724.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90988.log.gz 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48478.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53724.log.gz 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90946.log 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48478.log.gz 2024-03-06T07:34:54.647 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71853.log 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61147.log: /var/log/ceph/ceph-client.admin.90946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44530.log 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61147.log.gz 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90946.log.gz 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77277.log 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74706.log 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71853.log.gz 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44132.log 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44530.log.gz 2024-03-06T07:34:54.648 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.77277.log.gz 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80770.log 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.74706.log: /var/log/ceph/ceph-client.admin.44132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80392.log 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80770.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76316.log 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44132.log.gz 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80770.log.gz 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51520.log 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76316.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50602.log 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80392.log.gz 2024-03-06T07:34:54.649 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51520.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73932.log 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76316.log.gz 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50602.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51520.log.gz 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70820.log 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73932.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50602.log.gz 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53174.log 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73932.log.gz 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77487.log 2024-03-06T07:34:54.650 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.53174.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.73354.log 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70820.log.gz 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82869.log 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53174.log.gz 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77487.log.gz 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59220.log 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73354.log: /var/log/ceph/ceph-client.admin.82869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82270.log 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.73354.log.gz/var/log/ceph/ceph-client.admin.59220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55543.log 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82869.log.gz 2024-03-06T07:34:54.651 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59220.log.gz 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84929.log 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82270.log.gz 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55543.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.75165.log 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84929.log: gzip -5 --verbose 87.5% -- /var/log/ceph/ceph-client.admin.77999.log 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55543.log.gz 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84929.log.gz/var/log/ceph/ceph-client.admin.75165.log: 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52269.log 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75165.log.gz 2024-03-06T07:34:54.652 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64948.log 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77999.log.gz 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52269.log: -5 --verbose 93.5% -- replaced with /var/log/ceph/ceph-client.0.74706.log.gz 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.57569.log 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64948.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52269.log.gz 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54789.log 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64948.log.gz 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79828.log 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54789.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57569.log.gz 2024-03-06T07:34:54.653 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54341.log 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79828.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54789.log.gz 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65074.log 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6%/var/log/ceph/ceph-client.admin.54341.log: -- replaced with /var/log/ceph/ceph-client.admin.79828.log.gz 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80308.log 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54341.log.gz 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70946.log 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65074.log.gz 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83331.log 2024-03-06T07:34:54.654 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70946.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80308.log.gz 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78140.log 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70946.log.gz 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78228.log 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78140.log: gzip 91.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.83331.log.gz 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65445.log 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78140.log.gz 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.78228.log: -- /var/log/ceph/ceph-client.admin.67500.log 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65445.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78228.log.gz 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68223.log 2024-03-06T07:34:54.655 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75969.log 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67500.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65445.log.gz 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68391.log 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67500.log.gz 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68223.log.gz 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.75969.log: /var/log/ceph/ceph-client.admin.65889.log 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72067.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75969.log.gz 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.656 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68391.log.gz 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38536.log 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65889.log.gz 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85993.log 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38536.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72067.log.gz 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80518.log 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38536.log.gz 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45006.log 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85993.log.gz 2024-03-06T07:34:54.657 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84829.log 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80518.log.gz 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83289.log 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84829.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45006.log.gz 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92622.log 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.84829.log.gz 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45004.log 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92622.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83289.log.gz 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44572.log 2024-03-06T07:34:54.658 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92622.log.gz 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52059.log 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45004.log.gz 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71895.log 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr: 87.4%/var/log/ceph/ceph-client.admin.52059.log: -- replaced with /var/log/ceph/ceph-client.admin.44572.log.gz 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81880.log 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71895.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52059.log.gz 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73311.log 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81880.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71895.log.gz 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66530.log 2024-03-06T07:34:54.659 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81880.log.gz/var/log/ceph/ceph-client.admin.73311.log: 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78095.log 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73311.log.gz 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48073.log 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.0.62880.log.gz 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66530.log.gz 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66851.log 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48073.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78095.log.gz 2024-03-06T07:34:54.660 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77006.log 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48073.log.gz 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53766.log 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66851.log.gz 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51774.log 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77006.log: /var/log/ceph/ceph-client.admin.53766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84745.log 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77006.log.gz 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53766.log.gz 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65032.log 2024-03-06T07:34:54.661 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55459.log 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51774.log.gz 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84745.log: /var/log/ceph/ceph-client.admin.65032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47133.log 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84745.log.gz 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73812.log 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55459.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65032.log.gz 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45341.log 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55459.log.gz 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46126.log 2024-03-06T07:34:54.662 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73812.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47133.log.gz 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45341.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73812.log.gz 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55163.log 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45341.log.gz 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70541.log 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46126.log: /var/log/ceph/ceph-client.admin.55163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77048.log 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46126.log.gz 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61105.log 2024-03-06T07:34:54.663 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55163.log.gz 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.70541.log.gz 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64481.log 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77048.log.gz 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61105.log: /var/log/ceph/ceph-client.admin.64481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83813.log 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.audit.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61105.log.gz 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64481.log.gz 2024-03-06T07:34:54.664 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57826.log 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44362.log 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83813.log: /var/log/ceph/ceph-client.admin.57826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64313.log 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83813.log.gz 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57826.log.gz 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68349.log 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.64313.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44362.log.gz 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78182.log 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88143.log 2024-03-06T07:34:54.665 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64313.log.gz 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68349.log.gz 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61399.log 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.88143.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91156.log 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78182.log.gz 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61315.log 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88143.log.gz 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61399.log.gz 2024-03-06T07:34:54.666 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62555.log 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74051.log 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61315.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91156.log.gz 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88059.log 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61315.log.gz 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84035.log 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88059.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74051.log.gz 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81339.log 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.62555.log.gz 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4%/var/log/ceph/ceph-client.admin.84035.log: -- replaced with /var/log/ceph/ceph-client.admin.88059.log.gz 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89314.log 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81339.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84035.log.gz 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49794.log 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67374.log 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89314.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81339.log.gz 2024-03-06T07:34:54.668 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49794.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89314.log.gz 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71344.log 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49794.log.gz 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82688.log 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67374.log: /var/log/ceph/ceph-client.admin.71344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58166.log 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67374.log.gz 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71344.log.gz 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63511.log 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46504.log 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58166.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82688.log.gz 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84703.log 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58166.log.gz 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63511.log.gz 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76559.log 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46504.log: /var/log/ceph/ceph-client.admin.84703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76031.log 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51478.log 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84703.log.gz 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.46504.log.gz 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76031.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76559.log.gz 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54610.log 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51478.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76031.log.gz 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73628.log 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51478.log.gz 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86949.log 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54610.log.gz 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68181.log 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89860.log 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3%/var/log/ceph/ceph-client.admin.86949.log: -- replaced with /var/log/ceph/ceph-client.admin.73628.log.gz 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68181.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86949.log.gz 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77403.log 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68181.log.gz 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88918.log 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52143.log 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89860.log.gz 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88918.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.77403.log.gz -5 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69465.log 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52143.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88918.log.gz 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51228.log 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52143.log.gz 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83079.log 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69465.log: /var/log/ceph/ceph-client.admin.51228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86679.log 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83079.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69465.log.gz 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51228.log.gz -5 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91663.log 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83079.log.gz 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52395.log 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63193.log 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86679.log.gz 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52395.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.91663.log.gz 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83121.log 2024-03-06T07:34:54.674 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73568.log 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63193.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52395.log.gz 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50728.log 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63193.log.gz 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83121.log.gz 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87975.log 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73568.log: /var/log/ceph/ceph-client.admin.50728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84256.log 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73568.log.gz 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50728.log.gz 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48031.log 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49752.log 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84256.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87975.log.gz 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48031.log: gzip -5 --verbose 87.7% -- -- replaced with /var/log/ceph/ceph-client.admin.84256.log.gz /var/log/ceph/ceph-client.admin.53598.log 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48031.log.gz 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89776.log 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63097.log 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49752.log.gz 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89776.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53598.log.gz 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87495.log 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63097.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89776.log.gz 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69285.log 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63097.log.gz 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66656.log 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87495.log.gz 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59572.log 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55417.log 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69285.log.gz/var/log/ceph/ceph-client.admin.66656.log: 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59572.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66656.log.gz 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44997.log 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59572.log.gz 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48912.log 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55417.log: /var/log/ceph/ceph-client.admin.44997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46000.log 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55417.log.gz 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74054.log 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44997.log.gz 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48912.log.gz 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61273.log 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46000.log.gz 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52479.log 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62621.log 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61273.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74054.log.gz 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51436.log 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61273.log.gz 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48241.log 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62621.log: /var/log/ceph/ceph-client.admin.51436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58082.log 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.52479.log.gz 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62621.log.gz 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51436.log.gz 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64268.log 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58082.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48241.log.gz 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54484.log 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58082.log.gz 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72416.log 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64268.log.gz 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53236.log 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54484.log: /var/log/ceph/ceph-client.admin.72416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55501.log 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54484.log.gz 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53236.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72416.log.gz 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65118.log 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53236.log.gz 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45670.log 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55501.log.gz 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60793.log 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65118.log.gz 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65613.log 2024-03-06T07:34:54.682 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45670.log: /var/log/ceph/ceph-client.admin.60793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66698.log 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45670.log.gz 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65613.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60793.log.gz 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59446.log 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65613.log.gz 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86385.log 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81549.log 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59446.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66698.log.gz 2024-03-06T07:34:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56771.log 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59446.log.gz 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86385.log.gz 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90130.log 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81549.log: /var/log/ceph/ceph-client.admin.56771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54951.log 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81549.log.gz 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90130.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56771.log.gz 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55627.log 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90130.log.gz 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60583.log 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59404.log 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55627.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54951.log.gz 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60583.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55627.log.gz -5 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.0.59993.log 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60583.log.gz 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62345.log 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59404.log: /var/log/ceph/ceph-client.0.59993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46606.log 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59404.log.gz 2024-03-06T07:34:54.685 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77090.log 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62345.log.gz 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86865.log 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50980.log 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77090.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46606.log.gz 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79583.log 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77090.log.gz 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76601.log 2024-03-06T07:34:54.686 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50980.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86865.log.gz 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50980.log.gz 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76254.log 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90256.log 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79583.log.gz 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67542.log 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76601.log.gz 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76254.log.gz/var/log/ceph/ceph-client.admin.90256.log: 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83397.log 2024-03-06T07:34:54.687 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76712.log 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90256.log.gz 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67542.log.gz 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91621.log 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50104.log 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83397.log.gz 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76712.log: /var/log/ceph/ceph-client.admin.91621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60625.log 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76712.log.gz 2024-03-06T07:34:54.688 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50104.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91621.log.gz 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76198.log 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50104.log.gz 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88269.log 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90844.log 88.6% 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60625.log.gz 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76198.log: /var/log/ceph/ceph-client.admin.88269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70499.log 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48607.log 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88269.log.gz 2024-03-06T07:34:54.689 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70499.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90844.log.gz 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 92.6% -- replaced with /var/log/ceph/ceph-client.admin.76198.log.gz 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70499.log.gz 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58460.log 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48607.log.gz 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53112.log 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2024-03-06T07:34:54.690 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79996.log 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61740.log 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58460.log.gz 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89524.log 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53112.log.gz 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61740.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79996.log.gz 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47947.log 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92486.log 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89524.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61740.log.gz 2024-03-06T07:34:54.691 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85055.log 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89524.log.gz 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92486.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47947.log.gz 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63379.log 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85055.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92486.log.gz 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71175.log 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63379.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85055.log.gz 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63379.log.gz 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75651.log 2024-03-06T07:34:54.692 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67103.log 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71175.log: /var/log/ceph/ceph-client.admin.75651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61357.log 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71175.log.gz 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75651.log.gz 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59734.log 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67103.log.gz 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92791.log 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61357.log.gz 2024-03-06T07:34:54.693 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44404.log 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48428.log 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92791.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59734.log.gz 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45676.log 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92791.log.gz 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44404.log.gz 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89692.log 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52185.log 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45676.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48428.log.gz 2024-03-06T07:34:54.694 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89692.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45676.log.gz 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52185.log: gzip -5 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89692.log.gz 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54526.log 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52185.log.gz 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62177.log 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88185.log 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54526.log.gz 2024-03-06T07:34:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58887.log 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88185.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62177.log.gz 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45916.log 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88185.log.gz 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.58887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49468.log 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88227.log 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49468.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45916.log.gz 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63637.log 2024-03-06T07:34:54.696 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49468.log.gz 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56492.log 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88227.log: /var/log/ceph/ceph-client.admin.63637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71386.log 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56492.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88227.log.gz 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63637.log.gz 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59262.log 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56492.log.gz 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66451.log 2024-03-06T07:34:54.697 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71386.log: /var/log/ceph/ceph-client.admin.59262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90802.log 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71386.log.gz 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66451.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59262.log.gz 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66451.log.gz 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70652.log 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77132.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90802.log.gz 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80080.log 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70652.log.gz 2024-03-06T07:34:54.698 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79206.log 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2%/var/log/ceph/ceph-client.admin.80080.log: -- replaced with /var/log/ceph/ceph-client.admin.77132.log.gz 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44698.log 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80080.log.gz 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57359.log 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79206.log: /var/log/ceph/ceph-client.admin.44698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52101.log 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57359.log: 88.7% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44698.log.gz 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79206.log.gz 2024-03-06T07:34:54.699 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57359.log.gz 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44236.log 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.52101.log: /var/log/ceph/ceph-client.admin.44782.log 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61021.log 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52101.log.gz 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44236.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68139.log 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44782.log: /var/log/ceph/ceph-client.admin.61021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75713.log 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68139.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44782.log.gz 2024-03-06T07:34:54.700 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61021.log.gz 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68139.log.gz 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.52227.log 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81228.log 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75713.log.gz 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60667.log 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52017.log 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52227.log.gz 2024-03-06T07:34:54.701 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81228.log: /var/log/ceph/ceph-client.admin.60667.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81228.log.gz 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44824.log 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91579.log 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60667.log.gz 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44824.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52017.log.gz 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59650.log 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44824.log.gz 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63037.log 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91579.log: /var/log/ceph/ceph-client.admin.59650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77829.log 2024-03-06T07:34:54.702 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63037.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59650.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91579.log.gz 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63037.log.gz 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71937.log 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83951.log 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71937.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77829.log.gz 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71937.log.gz 2024-03-06T07:34:54.703 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46294.log 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83951.log: /var/log/ceph/ceph-client.admin.45342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72160.log 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46294.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83951.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46294.log.gz 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92664.log 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.72160.log: -- /var/log/ceph/ceph-client.admin.54171.log 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92664.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86301.log 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72160.log.gz 2024-03-06T07:34:54.704 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92664.log.gz 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61231.log 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50266.log 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61231.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54171.log.gz 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86301.log.gz 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62471.log 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61231.log.gz 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86739.log 2024-03-06T07:34:54.705 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88708.log 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50266.log.gz 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62471.log: /var/log/ceph/ceph-client.admin.86739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64523.log 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0%/var/log/ceph/ceph-client.admin.88708.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86739.log.gz 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62471.log.gz 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88708.log.gz 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45538.log 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84971.log 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45538.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64523.log.gz 2024-03-06T07:34:54.706 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44488.log 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45538.log.gz 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46462.log 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67458.log 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44488.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84971.log.gz 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46462.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44488.log.gz 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46462.log.gz 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62005.log 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87765.log 2024-03-06T07:34:54.707 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62005.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67458.log.gz 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86511.log 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48828.log 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62005.log.gz/var/log/ceph/ceph-client.admin.87765.log: 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49258.log 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87765.log.gz 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86511.log.gz 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58334.log 2024-03-06T07:34:54.708 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48828.log.gz 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79457.log 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45674.log 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58334.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49258.log.gz 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79457.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58334.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60305.log 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45674.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79457.log.gz 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45674.log.gz 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90904.log 2024-03-06T07:34:54.709 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60305.log: /var/log/ceph/ceph-client.admin.90904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85809.log 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68265.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60305.log.gz 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90904.log.gz 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87285.log 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92060.log 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85809.log.gz 2024-03-06T07:34:54.710 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74943.log 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0%/var/log/ceph/ceph-client.admin.92060.log: -- replaced with /var/log/ceph/ceph-client.admin.87285.log.gz 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88539.log 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92060.log.gz 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81423.log 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74943.log: /var/log/ceph/ceph-client.admin.88539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43754.log 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81423.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74943.log.gz 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88539.log.gz 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.47505.log 2024-03-06T07:34:54.711 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81423.log.gz 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70194.log 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43754.log: /var/log/ceph/ceph-client.2.47505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63679.log 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70194.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43754.log.gz 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75589.log 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr: 93.3% 89.3% -- replaced with /var/log/ceph/ceph-client.0.58887.log.gz 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70194.log.gz 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.2.47505.log.gz/var/log/ceph/ceph-client.admin.63679.log: 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82350.log 2024-03-06T07:34:54.712 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63679.log.gz 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53459.log 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.82350.log: -5 --verbose 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75589.log.gz 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.51106.log 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53459.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.82350.log.gz 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80686.log 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53459.log.gz 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56162.log 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.80686.log: -5 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51106.log.gz 2024-03-06T07:34:54.713 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86035.log 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.56162.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80686.log.gz 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91976.log 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89944.log 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86035.log.gz 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91705.log 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91976.log.gz 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83037.log 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89944.log: /var/log/ceph/ceph-client.admin.91705.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89944.log.gz 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91705.log.gz 2024-03-06T07:34:54.714 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64565.log 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88876.log 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83037.log: gzip/var/log/ceph/ceph-client.admin.64565.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.76880.log 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88876.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64565.log.gz 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83037.log.gz 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88876.log.gz 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64042.log 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91114.log 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65385.log 2024-03-06T07:34:54.715 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64042.log: 88.5%/var/log/ceph/ceph-client.admin.91114.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76880.log.gz 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64042.log.gz 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91114.log.gz 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50812.log 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67801.log 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56813.log 88.2% 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65385.log.gz 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50812.log: /var/log/ceph/ceph-client.admin.67801.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50812.log.gz 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67801.log.gz 2024-03-06T07:34:54.716 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84521.log 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61063.log 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84521.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56813.log.gz 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84521.log.gz 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89566.log 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71217.log 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50224.log: 93.2% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61063.log.gz 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.0.59993.log.gz 2024-03-06T07:34:54.717 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89566.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90088.log 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.89566.log.gz 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64649.log 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71217.log.gz 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45958.log 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64649.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90088.log.gz 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84132.log 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64649.log.gz 2024-03-06T07:34:54.718 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55035.log 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45958.log.gz 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84132.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84132.log.gz 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87891.log 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-osd.3.log: 87.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.75907.log 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55035.log.gz 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70736.log 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75907.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87891.log.gz 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61824.log 2024-03-06T07:34:54.719 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7%/var/log/ceph/ceph-client.admin.70736.log: -- replaced with /var/log/ceph/ceph-client.admin.75907.log.gz 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70736.log.gz 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr: /var/log/ceph/ceph-client.0.66148.log 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61824.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61824.log.gz 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.52311.log 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.66148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85432.log 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49594.log 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52311.log.gz 2024-03-06T07:34:54.720 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.85432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85707.log 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49594.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49594.log.gz 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91537.log 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85707.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85707.log.gz 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.85432.log.gz 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74339.log 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70862.log 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91537.log.gz 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65487.log 2024-03-06T07:34:54.721 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70862.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74339.log.gz 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86175.log 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70862.log.gz 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55077.log 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86175.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65487.log.gz 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86175.log.gz 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71729.log 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91072.log 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55077.log.gz/var/log/ceph/ceph-client.admin.71729.log: 2024-03-06T07:34:54.722 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48870.log 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91072.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71729.log.gz 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71811.log 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91072.log.gz 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60389.log 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71811.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48870.log.gz 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71811.log.gz 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86427.log 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82953.log 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60389.log.gz 2024-03-06T07:34:54.723 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75440.log 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82953.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86427.log.gz 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62303.log 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82953.log.gz 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67332.log 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62303.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75440.log.gz 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62303.log.gz 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81465.log 2024-03-06T07:34:54.724 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80914.log 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81465.log: /var/log/ceph/ceph-client.admin.67332.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67332.log.gz 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81465.log.gz 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91495.log 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88834.log 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80914.log.gz 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60877.log 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88834.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91495.log.gz 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr: 95.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62429.log 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.0.66148.log.gz 2024-03-06T07:34:54.725 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88834.log.gz 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59086.log 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54993.log 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60877.log.gz 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62429.log.gz 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90424.log 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65781.log 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59086.log.gz/var/log/ceph/ceph-client.admin.90424.log: 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54993.log.gz 2024-03-06T07:34:54.726 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90424.log.gz 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54045.log 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87597.log 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr: 93.4%/var/log/ceph/ceph-client.admin.54045.log: -- replaced with /var/log/ceph/ceph-client.0.56162.log.gz 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65781.log.gz 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76093.log 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54045.log.gz 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.87597.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.74381.log 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76093.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87597.log.gz 2024-03-06T07:34:54.727 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.80956.log 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2%/var/log/ceph/ceph-client.admin.74381.log: -- replaced with /var/log/ceph/ceph-client.admin.76093.log.gz 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54652.log 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.74381.log.gz 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67229.log 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54652.log.gz 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86469.log 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80956.log.gz 2024-03-06T07:34:54.728 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53360.log: /var/log/ceph/ceph-client.admin.67229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68009.log 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz/var/log/ceph/ceph-client.admin.86469.log: 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67229.log.gz 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 88.7% --verbose -- /var/log/ceph/ceph-client.admin.86343.log 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86469.log.gz 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69760.log 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68009.log.gz 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90634.log 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.69760.log: -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86343.log.gz 2024-03-06T07:34:54.729 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83163.log 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90634.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69760.log.gz 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.80122.log 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90634.log.gz 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66977.log 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83163.log.gz 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89356.log 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66977.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80122.log.gz 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82600.log 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66977.log.gz 2024-03-06T07:34:54.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43658.log 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89356.log.gz 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83247.log 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82600.log.gz 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86553.log 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83247.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43658.log.gz 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45874.log 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86553.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83247.log.gz 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63958.log 2024-03-06T07:34:54.731 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86553.log.gz 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82558.log 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45874.log.gz 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72776.log 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 88.7%/var/log/ceph/ceph-client.admin.82558.log: -- replaced with /var/log/ceph/ceph-client.admin.63958.log.gz 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.74881.log 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72776.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82558.log.gz 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67925.log 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72776.log.gz 2024-03-06T07:34:54.732 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77787.log 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74881.log.gz 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61963.log 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77787.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67925.log.gz 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91933.log 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77787.log.gz 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.61963.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63595.log 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9%/var/log/ceph/ceph-client.admin.91933.log: -- replaced with /var/log/ceph/ceph-client.admin.61963.log.gz 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75818.log 2024-03-06T07:34:54.733 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91933.log.gz 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5/var/log/ceph/ceph-client.admin.75818.log: 88.8% --verbose -- /var/log/ceph/ceph-client.admin.82175.log 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63595.log.gz 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84661.log 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75818.log.gz 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81735.log 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82175.log: /var/log/ceph/ceph-client.admin.84661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60263.log 2024-03-06T07:34:54.734 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82175.log.gz/var/log/ceph/ceph-client.admin.81735.log: 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84661.log.gz 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51350.log 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81735.log.gz 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44995.log 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60263.log.gz 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43962.log 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.44995.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62219.log 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51350.log.gz 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88666.log 2024-03-06T07:34:54.735 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44995.log.gz 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71649.log 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43962.log.gz 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62219.log: /var/log/ceph/ceph-client.admin.88666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80812.log 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62219.log.gz 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55837.log 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.88666.log.gz 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71649.log: /var/log/ceph/ceph-client.admin.80812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57868.log 2024-03-06T07:34:54.736 INFO:teuthology.orchestra.run.smithi122.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.80812.log.gz 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76155.log 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71649.log.gz 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64439.log 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57868.log.gz/var/log/ceph/ceph-client.admin.76155.log: 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55837.log.gz 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57065.log 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2%/var/log/ceph/ceph-client.admin.64439.log: -- replaced with /var/log/ceph/ceph-client.admin.76155.log.gz 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86781.log 2024-03-06T07:34:54.737 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64439.log.gz 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87327.log 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57065.log.gz 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.86781.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86259.log 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87327.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86781.log.gz 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87327.log.gz 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr: /var/log/ceph/ceph-client.admin.57744.log 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91747.log 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86259.log.gz 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82431.log 2024-03-06T07:34:54.738 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3%/var/log/ceph/ceph-client.admin.91747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92536.log 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57744.log.gz 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph-client.admin.82431.log: 89.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.44999.log 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91747.log.gz 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92536.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82431.log.gzgzip 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92536.log.gz 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60221.log 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44999.log: /var/log/ceph/ceph-mgr.y.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44999.log.gz 2024-03-06T07:34:54.739 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50062.log 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87117.log 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr: 88.1%/var/log/ceph/ceph-client.admin.50062.log: -- replaced with /var/log/ceph/ceph-client.admin.60221.log.gz 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56855.log 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50062.log.gz 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91282.log 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87117.log: /var/log/ceph/ceph-client.admin.56855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65655.log 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87117.log.gz 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91282.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56855.log.gz 2024-03-06T07:34:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72374.log 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65655.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91282.log.gz 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63916.log 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65655.log.gz 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72374.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72374.log.gz 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83205.log 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91789.log 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83205.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63916.log.gz 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54003.log 2024-03-06T07:34:54.741 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83205.log.gz 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91789.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91789.log.gz 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80476.log 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72584.log 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80476.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54003.log.gz 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48522.log 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80476.log.gz 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72584.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72584.log.gz 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60709.log 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87159.log 2024-03-06T07:34:54.742 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60709.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48522.log.gz 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48199.log 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60709.log.gz 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87159.log: /var/log/ceph/ceph-client.admin.48199.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48199.log.gz 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87159.log.gz 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86595.log 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93172.log 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86595.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2024-03-06T07:34:54.743 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65162.log 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86595.log.gz 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50854.log 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.93172.log: /var/log/ceph/ceph-client.admin.65162.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65162.log.gz 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93172.log.gz 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82995.log 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52437.log 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82995.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50854.log.gz 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56729.log 2024-03-06T07:34:54.744 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82995.log.gz 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52437.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52437.log.gz 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70988.log 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73728.log 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70988.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56729.log.gz 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65529.log 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70988.log.gz 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73728.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73728.log.gz 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55903.log 2024-03-06T07:34:54.745 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73526.log 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55903.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65529.log.gz 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87453.log 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55903.log.gz 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73526.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73526.log.gz 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49138.log 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77624.log 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49138.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.87453.log.gz 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59488.log 2024-03-06T07:34:54.746 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49138.log.gz 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59304.log 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77624.log: /var/log/ceph/ceph-client.admin.59488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78312.log 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59304.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77624.log.gz 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59488.log.gz 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59304.log.gz 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46210.log 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43566.log 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78312.log.gz/var/log/ceph/ceph-client.admin.46210.log: 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64084.log 2024-03-06T07:34:54.747 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46210.log.gz 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78480.log 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43566.log.gz 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50686.log 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64084.log.gz/var/log/ceph/ceph-client.admin.78480.log: 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43877.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78480.log.gz 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50686.log: gzip -5 88.3% --verbose -- /var/log/ceph/ceph-client.admin.87807.log 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50686.log.gz 2024-03-06T07:34:54.748 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79912.log 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43877.log.gz/var/log/ceph/ceph-client.admin.87807.log: 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91198.log 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87807.log.gz 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77192.log 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79912.log: /var/log/ceph/ceph-client.admin.91198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48691.log 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79912.log.gz 88.2% 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77192.log: -- replaced with /var/log/ceph/ceph-client.admin.91198.log.gz 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89170.log 2024-03-06T07:34:54.749 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77192.log.gz 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73770.log 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67416.log 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48691.log.gz 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73770.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89170.log.gz 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49096.log 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81922.log 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73770.log.gz 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67416.log.gz 2024-03-06T07:34:54.750 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49096.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49096.log.gz 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85623.log 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89902.log 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85623.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81922.log.gz 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77361.log 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85623.log.gz 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59346.log 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89902.log: /var/log/ceph/ceph-client.admin.77361.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77361.log.gz 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89902.log.gz 2024-03-06T07:34:54.751 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76517.log 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51730.log 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76517.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59346.log.gz 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93085.log 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76517.log.gz 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69822.log 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51730.log: /var/log/ceph/ceph-client.admin.93085.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51730.log.gz 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93085.log.gz 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47989.log 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76796.log 2024-03-06T07:34:54.752 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47989.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69822.log.gz 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64210.log 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47989.log.gz/var/log/ceph/ceph-client.admin.76796.log: 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72202.log 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76796.log.gz 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64210.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64210.log.gz 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.90508.log 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80434.log 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72202.log.gz/var/log/ceph/ceph-client.admin.90508.log: 2024-03-06T07:34:54.753 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65297.log 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90508.log.gz 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65697.log 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90298.log 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65297.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80434.log.gz 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65697.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.65297.log.gz 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53919.log 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65697.log.gz 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49878.log 2024-03-06T07:34:54.754 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93127.log 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90298.log.gz 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44994.log 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53919.log.gz 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49878.log: /var/log/ceph/ceph-client.admin.93127.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49878.log.gz 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93127.log.gz 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44994.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44994.log.gz 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56897.log 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2024-03-06T07:34:54.755 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58376.log 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56897.log.gz 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57952.log 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58376.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74293.log 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58376.log.gz 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57952.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57952.log.gz 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55249.log 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67019.log 2024-03-06T07:34:54.756 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74293.log.gz 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91240.log 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr: 88.7%/var/log/ceph/ceph-client.admin.67019.log: -- replaced with /var/log/ceph/ceph-client.admin.55249.log.gz 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67019.log.gz 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48649.log 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90550.log 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91240.log: /var/log/ceph/ceph-client.admin.48649.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91240.log.gz 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48649.log.gz 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68517.log 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52545.log 2024-03-06T07:34:54.757 INFO:teuthology.orchestra.run.smithi122.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90550.log.gz 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52979.log 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52545.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68517.log.gz 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52545.log.gz 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88101.log 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91873.log 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.52979.log: /var/log/ceph/ceph-client.admin.88101.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52979.log.gz 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88101.log.gz 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90676.log 2024-03-06T07:34:54.758 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91873.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91873.log.gz 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88017.log 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50308.log 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90676.log: /var/log/ceph/ceph-client.admin.88017.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90676.log.gz 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88017.log.gz 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54694.log 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50308.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50308.log.gz 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53298.log 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77319.log 2024-03-06T07:34:54.759 INFO:teuthology.orchestra.run.smithi122.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54694.log.gz 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55333.log 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77319.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53298.log.gz 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87369.log 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77319.log.gz 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72542.log 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87369.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55333.log.gz 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87369.log.gz 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73035.log 2024-03-06T07:34:54.760 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84459.log 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72542.log: /var/log/ceph/ceph-client.0.73035.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72542.log.gz 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86823.log 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84459.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84459.log.gz 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76838.log 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86823.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86823.log.gz 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77529.log 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76838.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76838.log.gz 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51604.log 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77529.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77529.log.gz 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79519.log 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51604.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51604.log.gz 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46336.log 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79519.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.79519.log.gz 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75122.log 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46336.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46336.log.gz 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.73035.log.gz 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64906.log 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86907.log 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75122.log: /var/log/ceph/ceph-client.admin.64906.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64906.log.gz 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75122.log.gz 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68944.log 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58562.log 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86907.log: /var/log/ceph/ceph-client.0.68944.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86907.log.gz 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64126.log 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58562.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.58562.log.gz 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79870.log 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64126.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64126.log.gz 2024-03-06T07:34:54.763 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83909.log 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79870.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.79870.log.gz 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54297.log 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83909.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.83909.log.gz 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73484.log 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54297.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54297.log.gz 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50560.log 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.73484.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73484.log.gz 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80248.log 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50560.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50560.log.gz 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68307.log 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80248.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80248.log.gz 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68307.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68307.log.gz 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44176.log 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69287.log 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44176.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44176.log.gz 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70904.log 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69287.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69287.log.gz 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50896.log 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70904.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70904.log.gz 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84887.log 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50896.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50896.log.gz 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78043.log 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84887.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84887.log.gz 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47627.log 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78043.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78043.log.gz 2024-03-06T07:34:54.766 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45668.log 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47627.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47627.log.gz 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92707.log 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45668.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45668.log.gz 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54213.log 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92707.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92707.log.gz 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68051.log 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54213.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54213.log.gz 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86217.log 2024-03-06T07:34:54.767 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68051.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68051.log.gz 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80350.log 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86217.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86217.log.gz 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72290.log 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80350.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80350.log.gz 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55375.log 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72290.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72290.log.gz 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55375.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55375.log.gz 2024-03-06T07:34:54.768 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45001.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43835.log 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65823.log 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.43835.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43835.log.gz 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76433.log 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65823.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65823.log.gz 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55207.log 2024-03-06T07:34:54.769 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76433.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76433.log.gz 2024-03-06T07:34:54.770 INFO:teuthology.orchestra.run.smithi169.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:34:54.771 INFO:teuthology.orchestra.run.smithi169.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:34:54.771 INFO:teuthology.orchestra.run.smithi169.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:34:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53877.log 2024-03-06T07:34:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.55207.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55207.log.gz 2024-03-06T07:34:54.814 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85013.log 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53877.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53877.log.gz 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77445.log 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85013.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85013.log.gz 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89986.log 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77445.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.77445.log.gz 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45329.log 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89986.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89986.log.gz 2024-03-06T07:34:54.815 INFO:teuthology.orchestra.run.smithi122.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45329.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45329.log.gz 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63317.log 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57023.log 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.63317.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63317.log.gz 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62093.log 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57023.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57023.log.gz 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49510.log 2024-03-06T07:34:54.816 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62093.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62093.log.gz 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72710.log 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49510.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49510.log.gz 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92959.log 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72710.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.72710.log.gz 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79395.log 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92959.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92959.log.gz 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50020.log 2024-03-06T07:34:54.817 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79395.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79395.log.gz 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87639.log 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50020.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50020.log.gz 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58628.log 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87639.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.87639.log.gz 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44320.log 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58628.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58628.log.gz 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59044.log 2024-03-06T07:34:54.818 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44320.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44320.log.gz 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80644.log 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59044.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59044.log.gz 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91324.log 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80644.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80644.log.gz 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76964.log 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.91324.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91324.log.gz 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59530.log 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.76964.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76964.log.gz 2024-03-06T07:34:54.819 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82091.log 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59530.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59530.log.gz 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71554.log 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82091.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82091.log.gz 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48325.log 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71554.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71554.log.gz 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92578.log 2024-03-06T07:34:54.820 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48325.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48325.log.gz 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67187.log 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92578.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92578.log.gz 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46168.log 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67187.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67187.log.gz 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48564.log 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46168.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46168.log.gz 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87849.log 2024-03-06T07:34:54.821 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48564.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48564.log.gz 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87849.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87849.log.gz 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75378.log 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38429.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83864.log 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75378.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75378.log.gz 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57401.log 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83864.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83864.log.gz 2024-03-06T07:34:54.822 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71512.log 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57401.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57401.log.gz 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70355.log 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71512.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71512.log.gz 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56939.log 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70355.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70355.log.gz 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67967.log 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56939.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56939.log.gz 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47905.log 2024-03-06T07:34:54.823 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67967.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67967.log.gz 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70070.log 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.47905.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47905.log.gz 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62047.log 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.70070.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70070.log.gz 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88311.log 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62047.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62047.log.gz 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84787.log 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88311.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88311.log.gz 2024-03-06T07:34:54.824 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67145.log 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.84787.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84787.log.gz 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81777.log 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67145.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67145.log.gz 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45678.log 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81777.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81777.log.gz 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45677.log 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45678.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45678.log.gz 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67289.log 2024-03-06T07:34:54.825 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45677.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45677.log.gz 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48115.log 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.67289.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67289.log.gz 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83659.log 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48115.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48115.log.gz 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66809.log 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.83659.log: 92.3% -- replaced with /var/log/ceph/ceph-client.0.83659.log.gz 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89650.log 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.66809.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66809.log.gz 2024-03-06T07:34:54.826 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89734.log 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89650.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89650.log.gz 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56534.log 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89734.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89734.log.gz 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68095.log 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.56534.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56534.log.gz 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85951.log 2024-03-06T07:34:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68095.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.68095.log.gz 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89818.log 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85951.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85951.log.gz 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88960.log 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89818.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89818.log.gz 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81381.log 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.88960.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88960.log.gz 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58418.log 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81381.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81381.log.gz 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.32103.log 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.58418.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58418.log.gz 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71979.log 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.tmp-client.admin.32103.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.32103.log.gz 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78438.log 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71979.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71979.log.gz 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49384.log 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78438.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78438.log.gz 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72023.log 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49384.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49384.log.gz 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68619.log 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72023.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72023.log.gz 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72500.log 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68619.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.68619.log.gz 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49300.log 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.72500.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72500.log.gz 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89044.log 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49300.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49300.log.gz 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85749.log 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89044.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89044.log.gz 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49936.log 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.85749.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85749.log.gz 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77704.log 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49936.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49936.log.gz 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44614.log 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77704.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77704.log.gz 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51869.log 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44614.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44614.log.gz 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77871.log 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51869.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51869.log.gz 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87243.log 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77871.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77871.log.gz 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89398.log 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.87243.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87243.log.gz 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61782.log 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89398.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89398.log.gz 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90466.log 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61782.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61782.log.gz 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69616.log 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90466.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90466.log.gz 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48283.log 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69616.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69616.log.gz 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82823.log 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48283.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48283.log.gz 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64397.log 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82823.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82823.log.gz 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81060.log 2024-03-06T07:34:54.863 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64397.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64397.log.gz 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92917.log 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.81060.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81060.log.gz 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92917.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.92917.log.gz 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92299.log 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45339.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49636.log 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92299.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92299.log.gz 2024-03-06T07:34:54.864 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45340.log 2024-03-06T07:34:54.865 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.49636.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49636.log.gz 2024-03-06T07:34:54.865 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89002.log 2024-03-06T07:34:54.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45340.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45340.log.gz 2024-03-06T07:34:54.881 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65343.log 2024-03-06T07:34:54.881 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89002.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89002.log.gz 2024-03-06T07:34:54.881 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54255.log 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65343.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65343.log.gz 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74447.log 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.54255.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54255.log.gz 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60979.log 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.74447.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.74447.log.gz 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38162.log 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60979.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60979.log.gz 2024-03-06T07:34:54.882 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44446.log 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38162.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38162.log.gz 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83993.log 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.44446.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44446.log.gz 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68559.log 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.83993.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83993.log.gz 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46378.log 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68559.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68559.log.gz 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45334.log 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.46378.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46378.log.gz 2024-03-06T07:34:54.883 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57275.log 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.45334.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45334.log.gz 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79954.log 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57275.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57275.log.gz 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59692.log 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.79954.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79954.log.gz 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65213.log 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.59692.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59692.log.gz 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75060.log 2024-03-06T07:34:54.884 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.65213.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65213.log.gz 2024-03-06T07:34:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51308.log 2024-03-06T07:34:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75060.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75060.log.gz 2024-03-06T07:34:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77913.log 2024-03-06T07:34:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51308.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51308.log.gz 2024-03-06T07:34:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51971.log 2024-03-06T07:34:54.885 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.77913.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77913.log.gz 2024-03-06T07:34:54.892 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89128.log 2024-03-06T07:34:54.892 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51971.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51971.log.gz 2024-03-06T07:34:54.892 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82644.log 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.89128.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89128.log.gz 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75237.log 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.82644.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82644.log.gz 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57149.log 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.75237.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75237.log.gz 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90172.log 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.57149.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57149.log.gz 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86133.log 2024-03-06T07:34:54.893 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90172.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90172.log.gz 2024-03-06T07:34:54.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:34:54.894 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.86133.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86133.log.gz 2024-03-06T07:34:54.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48367.log 2024-03-06T07:34:54.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50518.log 2024-03-06T07:34:54.905 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.48367.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48367.log.gz 2024-03-06T07:34:54.918 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53403.log 2024-03-06T07:34:54.919 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50518.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50518.log.gz 2024-03-06T07:34:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62261.log 2024-03-06T07:34:54.925 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.53403.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53403.log.gz 2024-03-06T07:34:54.946 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60751.log 2024-03-06T07:34:54.946 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.62261.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62261.log.gz 2024-03-06T07:34:54.957 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93001.log 2024-03-06T07:34:54.957 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.60751.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60751.log.gz 2024-03-06T07:34:54.966 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50182.log 2024-03-06T07:34:54.966 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.93001.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93001.log.gz 2024-03-06T07:34:54.983 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64744.log 2024-03-06T07:34:54.984 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50182.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50182.log.gz 2024-03-06T07:34:54.992 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71030.log 2024-03-06T07:34:54.992 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.64744.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64744.log.gz 2024-03-06T07:34:55.007 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68685.log 2024-03-06T07:34:55.007 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.71030.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.71030.log.gz 2024-03-06T07:34:55.019 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51688.log 2024-03-06T07:34:55.019 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.68685.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68685.log.gz 2024-03-06T07:34:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90592.log 2024-03-06T07:34:55.036 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51688.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51688.log.gz 2024-03-06T07:34:55.052 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50476.log 2024-03-06T07:34:55.052 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.90592.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90592.log.gz 2024-03-06T07:34:55.064 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61912.log 2024-03-06T07:34:55.064 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.50476.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50476.log.gz 2024-03-06T07:34:55.077 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92257.log 2024-03-06T07:34:55.077 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.61912.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61912.log.gz 2024-03-06T07:34:55.086 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78270.log 2024-03-06T07:34:55.086 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92257.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.92257.log.gz 2024-03-06T07:34:55.101 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78522.log 2024-03-06T07:34:55.102 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78270.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78270.log.gz 2024-03-06T07:34:55.115 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51022.log 2024-03-06T07:34:55.116 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.78522.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.78522.log.gz 2024-03-06T07:34:55.132 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80728.log 2024-03-06T07:34:55.133 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.51022.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51022.log.gz 2024-03-06T07:34:55.143 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69658.log 2024-03-06T07:34:55.143 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.80728.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80728.log.gz 2024-03-06T07:34:55.152 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92444.log 2024-03-06T07:34:55.153 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.69658.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.69658.log.gz 2024-03-06T07:34:55.161 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.92444.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92444.log.gz 2024-03-06T07:34:55.176 INFO:teuthology.orchestra.run.smithi122.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.52804.log.gz 2024-03-06T07:34:55.257 INFO:teuthology.orchestra.run.smithi169.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:34:55.375 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:34:55.379 INFO:teuthology.orchestra.run.smithi169.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi169.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:34:55.500 INFO:teuthology.orchestra.run.smithi169.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:34:55.502 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-06T07:34:55.502 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m1.232s 2024-03-06T07:34:55.502 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m5.412s 2024-03-06T07:34:55.502 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.208s 2024-03-06T07:34:55.759 INFO:teuthology.orchestra.run.smithi122.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:34:55.801 INFO:teuthology.orchestra.run.smithi122.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:34:55.992 INFO:teuthology.orchestra.run.smithi122.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:34:56.004 INFO:teuthology.orchestra.run.smithi122.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:34:56.714 INFO:teuthology.orchestra.run.smithi122.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.68944.log.gz 2024-03-06T07:35:03.397 INFO:teuthology.orchestra.run.smithi122.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:35:03.399 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-06T07:35:03.399 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m9.161s 2024-03-06T07:35:03.400 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m18.592s 2024-03-06T07:35:03.400 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m1.084s 2024-03-06T07:35:03.400 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:35:03.401 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/remote/smithi122/log 2024-03-06T07:35:03.402 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:35:06.505 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/remote/smithi169/log 2024-03-06T07:35:06.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:35:07.221 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:35:07.230 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:35:07.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:35:07.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:35:07.318 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:35:07.328 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:35:07.328 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:35:07.331 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:35:07.344 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:35:07.346 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:35:07.348 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:35:07.349 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-06T07:35:07.349 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +1815us[+1815us] +/- 100ms 2024-03-06T07:35:07.349 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 7 377 34 -1066us[-1072us] +/- 55ms 2024-03-06T07:35:07.349 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +14us[ +14us] +/- 91ms 2024-03-06T07:35:07.349 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:35:07.350 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:35:07.350 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-06T07:35:07.350 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1908us[+1908us] +/- 100ms 2024-03-06T07:35:07.351 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -1011us[-1008us] +/- 55ms 2024-03-06T07:35:07.351 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +62us[ +62us] +/- 91ms 2024-03-06T07:35:07.351 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:35:07.352 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:35:07.360 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:35:07.361 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:35:07.369 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:35:07.488 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:35:07.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:35:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:07.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:35:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:07.623 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:35:07.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:35:07.641 INFO:teuthology.task.internal:Duration was 1262.685921 seconds 2024-03-06T07:35:07.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:35:07.650 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:35:07.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:35:07.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:35:07.693 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:35:07.707 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:35:07.972 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:35:07.973 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-03-06T07:35:07.973 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:35:08.032 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-06T07:35:08.032 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:35:08.052 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:35:08.052 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:35:08.074 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:35:08.120 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:35:08.120 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:35:08.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:35:08.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:35:08.293 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:35:08.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:35:08.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:35:08.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:35:08.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:08.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:08.385 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-03-06T07:35:08.412 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-06T07:35:08.428 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:08.457 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:08.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:35:08.492 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:35:08.493 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/remote/smithi122 2024-03-06T07:35:08.493 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:35:08.535 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063/remote/smithi169 2024-03-06T07:35:08.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:35:08.570 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:35:08.570 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:35:08.573 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:35:08.619 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:35:08.630 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:35:08.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:35:08.693 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:35:08.693 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:35:08.695 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:35:08.707 INFO:teuthology.orchestra.run.smithi122.stdout: 265589 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:35 /home/ubuntu/cephtest 2024-03-06T07:35:08.709 INFO:teuthology.orchestra.run.smithi169.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:35 /home/ubuntu/cephtest 2024-03-06T07:35:08.711 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:35:08.722 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:35:08.743 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:35:08.769 DEBUG:teuthology.nuke:shortname: smithi122 2024-03-06T07:35:08.770 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:35:08.795 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.776244', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDw56Hh3KyspNyvaqv217HKBRutd3W/q/7CoyAjsp0tCJ6aA5Dm6IB+gQXPBBW+Zp/Gm+67VWFbbWwOFy5KGYOw='} 2024-03-06T07:35:08.798 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-06T07:35:08.799 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:35:08.819 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.777318', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNzfSidiz0rkDkJfLzZf7THIJ4NstMbuBEpwm7P8rMqb9+nhELtBXuADDX0j2ZoCV4097QnzClIx3awlnqAeAeQ='} 2024-03-06T07:35:08.847 INFO:teuthology.orchestra.console.smithi122:Power off 2024-03-06T07:35:08.848 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:35:08.872 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-03-06T07:35:08.872 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:08.889 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-06T07:35:08.889 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:35:08.899 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-06T07:35:08.913 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-06T07:35:08.913 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:08.935 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-06T07:35:12.901 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:12.936 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:13.011 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-06T07:35:13.046 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-06T07:35:17.013 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:17.047 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:17.124 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-06T07:35:17.158 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-06T07:35:17.258 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-06T07:35:21.129 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:21.239 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-03-06T07:35:21.340 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-03-06T07:35:21.444 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1262.685920715332 failure_reason: 'Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=92d5f02d6a8f48f9825ab2f8d6141c29 status: fail success: false 2024-03-06T07:35:21.444 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:35:21.522 INFO:teuthology.run:FAIL